---
title: "Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad"
description: "Benchmarks such as MathArena show reasoning LLMs performing at near-elite levels on… 14 references, cited by 30 papers here."
authors: ["Ivo Petrov", "Jasper Dekoninck", "Lyuben Baltadzhiev", "Maria Drencheva", "Kristian Minchev", "Mislav Balunović", "Nikola Jovanović", "Martin Vechev"]
year: 2025
venue: "arXiv:2503.21934"
source: "https://arxiv.org/abs/2503.21934"
doi: "10.48550/arxiv.2503.21934"
arxiv: "2503.21934"
licence: "CC BY 4.0 (arXiv)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/petrov-2025-proof-or-bluff-evaluating-llms/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/petrov-2025-proof-or-bluff-evaluating-llms/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/4/) / Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad

# Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad

**Authors:** [Ivo Petrov](https://md.research.elamur.ai/free/en/authors/ivo-petrov/), [Jasper Dekoninck](https://md.research.elamur.ai/free/en/authors/jasper-dekoninck/), Lyuben Baltadzhiev, [Maria Drencheva](https://md.research.elamur.ai/free/en/authors/maria-drencheva/), [Kristian Minchev](https://md.research.elamur.ai/free/en/authors/kristian-minchev/), [Mislav Balunović](https://md.research.elamur.ai/free/en/authors/mislav-balunovic/), [Nikola Jovanović](https://md.research.elamur.ai/free/en/authors/nikola-jovanovic/) and [Martin Vechev](https://md.research.elamur.ai/free/en/authors/martin-vechev/) · **Year:** 2025 · **Venue:** arXiv:2503.21934 · **Licence:** [CC BY 4.0 (arXiv)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2503.21934) · [PDF](https://arxiv.org/pdf/2503.21934v5) · [DOI 10.48550/arxiv.2503.21934](https://doi.org/10.48550/arxiv.2503.21934) · [arXiv 2503.21934](https://arxiv.org/abs/2503.21934) · [OpenAlex](https://openalex.org/W4416420775)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [30](https://md.research.elamur.ai/free/en/papers/petrov-2025-proof-or-bluff-evaluating-llms/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Benchmarks such as MathArena show reasoning LLMs performing at near-elite levels on answer-based contests like AIME and HMMT. However, these tests only check a final number, not whether the argument is valid. The authors test whether current models can write rigorous natural-language proofs. They used the six problems of USAMO 2025, which were uncontaminated because evaluation began within hours of the contest's release. Eight reasoning models each attempted every problem four times. Four experienced olympiad graders scored the anonymized LaTeX solutions on the standard 7-point scale, with two graders per problem and rubrics built from verified AoPS solutions. Results were poor. Gemini-2.5-Pro averaged about 24%, and every other model stayed under 5%. Every model presented its answers as complete solutions, even when they were wrong. The study also classifies errors and documents training-induced quirks. These findings mean that LLM-generated mathematics still requires careful human verification, and that answer-only benchmarks overstate models' reasoning ability.

## Key points
- Models tested: Gemini-2.5-Pro, R1, Grok 3, Flash-Thinking, Claude 3.7, QwQ, o1-pro, o3-mini.
- Average totals out of 42 points:
  - Gemini-2.5-Pro: 10.1 (about 24.4%), with 6 of its 24 attempts scoring 6/7 or higher.
  - R1 and Grok 3: 2.0 each.
  - o3-mini: 0.9.
  - o1-pro: 1.2, at a cost of about $203.
- Across roughly 175 solutions from the models other than Gemini, the only full 7/7 score was a single Grok 3 attempt on Problem 1.
- Graders labeled the first error in each solution as logic, assumption, creativity, or algebra/arithmetic. Flawed logic was the most common category. Models also often dismissed key steps as "trivial," which was especially frequent for o3-mini.
- Observed artifacts include:
  - placing answers in boxes when no single answer was appropriate (QwQ reduced "all even k" to 2);
  - generalizing from a single example;
  - Gemini inventing citations to papers that do not exist.
- An attempt to grade solutions automatically with an LLM judge, given the rubric and reference solution, was unsuccessful.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 496 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,948 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,394 works

## Cite

Cite as: Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislav Balunović, Nikola Jovanović and Martin Vechev. 2025. Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad. arXiv:2503.21934.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{petrov2025proof,
  title = {{Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad}},
  author = {Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislav Balunović, Nikola Jovanović and Martin Vechev},
  year = {2025},
  doi = {10.48550/arxiv.2503.21934},
  eprint = {2503.21934},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.21934}
}
```

- [← previous Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models](https://md.research.elamur.ai/free/en/papers/wang-2025-lost-multilinguality-dissecting-cross-lingual/)
- [next → RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinement](https://md.research.elamur.ai/free/en/papers/jiang-2025-rag-star-enhancing-deliberative-reasoning/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/4/)

## References

14 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. Claude 3.7 sonnet system card, March 2025. URL [https://assets.anthropic.com/](https://assets.anthropic.com/) m/785e231869ea8b3b/original/claude-3-7-sonnet-system-card.pdf . [↗](https://assets.anthropic.com/)
2. Mislav Balunovic, Jasper Dekoninck, Nikola Jovanovic, Ivo Petrov, and Martin T. Vechev. Math-construct: Challenging LLM reasoning with constructive proofs. CoRR, abs/2502.10197, 2025. doi: 10.48550/ARXIV.2502.10197. URL https://doi.org/10.48550/arXiv.2502.10197 . — [doi:10.48550/arxiv.2502.10197](https://doi.org/10.48550/arxiv.2502.10197) [↗](https://doi.org/10.48550/arXiv.2502.10197)
3. Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. [Matharena: Evaluating llms on uncontaminated math competitions](https://md.research.elamur.ai/free/en/papers/balunovic-2025-matharena-evaluating-llms-uncontaminated-math/), February 2025. URL [https://matharena.ai/](https://matharena.ai/) . [↗](https://matharena.ai/)
4. Simon Frieder, Luca Pinchetti, Alexis Chevalier, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Petersen, and Julius Berner. [Mathematical capabilities of chatgpt](https://md.research.elamur.ai/free/en/papers/frieder-2023-mathematical-capabilities-chatgpt/). In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neu…
5. Google DeepMind. Gemini Pro, March 2025. URL [https://deepmind.google/technologies/](https://deepmind.google/technologies/) gemini/pro/ . Accessed: 2025-04-03. [↗](https://deepmind.google/technologies/)
6. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948, 2025.
7. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, An… — [OpenAI o1 System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/)
8. Chengwu Liu, Jianhao Shen, Huajian Xin, Zhengying Liu, Ye Yuan, Haiming Wang, Wei Ju, Chuanyang Zheng, Yichun Yin, Lin Li, Ming Zhang, and Qun Liu. [FIMO: A challenge formal dataset for automated theorem proving](https://md.research.elamur.ai/free/en/papers/liu-2023-fimo-challenge-formal-dataset-automated/). CoRR, abs/2309.04295, 2023.
9. OpenAI. Openai o3-mini system card, January 2025. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Openai+o3%2Dmini+system+card%2C+January+2025%2E)
10. URL [https://cdn.openai.com/o3-mini-system-card-feb10.pdf](https://cdn.openai.com/o3-mini-system-card-feb10.pdf) . Qwen. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwenlm.github.io/blog/qwq-32b/ . [↗](https://cdn.openai.com/o3-mini-system-card-feb10.pdf)
11. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean-Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sotti… — [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/)
12. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). CoRR, abs/2402.03300, 2024. doi: 10.48550/ARXIV.2402.03300. URL https://doi.org/10.48550/arXiv.2402.03300… [↗](https://doi.org/10.48550/arXiv.2402.03300…)
13. George Tsoukalas, Jasper Lee, John Jennings, Jimmy Xin, Michelle Ding, Michael Jennings, Ami-tayush Thakur, and Swarat Chaudhuri. [Putnambench: Evaluating neural theorem-provers on the putnam mathematical competition](https://md.research.elamur.ai/free/en/papers/tsoukalas-2024-putnambench-evaluating-neural-theorem-provers/). CoRR, abs/2407.11214, 2024. xAI. Grok 3 beta — the age of reasoning agents, Februar…
14. Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. [minif2f: a cross-system benchmark for formal olympiad-level mathematics](https://md.research.elamur.ai/free/en/papers/zheng-2021-minif2f-cross-system-benchmark-formal/). In ICLR. OpenReview.net, 2022.

*Updated 3 November 2026*