---
title: "FrontierMath — paper by Glazer et al. (2024)"
description: "FrontierMath therefore offers a scalable, leakage-resistant way to track progress toward… 28 references, cited by 88 papers here."
authors: ["Elliot Glazer", "Ege Erdil", "Tamay Besiroglu", "Diego Chicharro", "Evan Chen", "Alex Gunning", "Caroline Falkman Olsson", "Jean-Stanislas Denain", "Anson Ho", "Emily de Oliveira Santos", "Olli Järviniemi", "Matthew Barnett", "Robert Sandler", "Matej Vrzala", "Jaime Sevilla", "Qiuyu Ren", "Elizabeth Pratt", "Lionel Levine", "Grant Barkley", "Natalie Stewart", "Bogdan Grechuk", "Tetiana Grechuk", "Shreepranav Varma Enugandla", "Mark Wildon"]
year: 2024
venue: "arXiv:2411.04872"
source: "https://arxiv.org/abs/2411.04872"
doi: "10.48550/arxiv.2411.04872"
arxiv: "2411.04872"
licence: "CC BY 4.0 (arXiv)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/glazer-2024-frontiermath-benchmark-evaluating-advanced-mathematical/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/glazer-2024-frontiermath-benchmark-evaluating-advanced-mathematical/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI

# FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI

**Authors:** Elliot Glazer, [Ege Erdil](https://md.research.elamur.ai/free/en/authors/ege-erdil/), [Tamay Besiroglu](https://md.research.elamur.ai/free/en/authors/tamay-besiroglu/), Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, [Anson Ho](https://md.research.elamur.ai/free/en/authors/anson-ho/), Emily de Oliveira Santos, Olli Järviniemi, [Matthew Barnett](https://md.research.elamur.ai/free/en/authors/matthew-barnett/), Robert Sandler, Matej Vrzala, Jaime Sevilla, Qiuyu Ren, Elizabeth Pratt, Lionel Levine, Grant Barkley, Natalie Stewart, Bogdan Grechuk, Tetiana Grechuk, Shreepranav Varma Enugandla and Mark Wildon · **Year:** 2024 · **Venue:** arXiv:2411.04872 · **Licence:** [CC BY 4.0 (arXiv)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2411.04872) · [PDF](https://arxiv.org/pdf/2411.04872v7) · [DOI 10.48550/arxiv.2411.04872](https://doi.org/10.48550/arxiv.2411.04872) · [arXiv 2411.04872](https://arxiv.org/abs/2411.04872) · [OpenAlex](https://openalex.org/W4404397121)

**Citations:** 9 citations (OpenAlex, 2026-10-14) · [88](https://md.research.elamur.ai/free/en/papers/glazer-2024-frontiermath-benchmark-evaluating-advanced-mathematical/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Popular math benchmarks such as MATH and GSM8K mostly test high-school or early-undergraduate skills, and leading models now score near the ceiling on them. Fresh competition sets such as IMO and AIME avoid training-data leakage, but they contain few problems and are slow to grade. FrontierMath, built by Epoch AI with more than 60 mathematicians, addresses both problems. It contains several hundred newly written, unpublished problems, ranging from hard competition-style tasks to questions drawn from current research, and covers about 70% of the top-level MSC2020 subjects. Every problem has one definitive answer, usually a large integer or a SymPy object, which a script checks automatically. A domain expert peer-reviews each item, and each comes with difficulty ratings. A specialist typically needs hours to solve one problem, and the hardest take days. When the paper was written, state-of-the-art models solved under 2% of the problems, while they score close to the maximum on other benchmarks. FrontierMath therefore offers a scalable, leakage-resistant way to track progress toward research-level mathematical reasoning, although it does not test proof writing.

## Key points
- The problems come from more than 60 contributors in over a dozen countries, from graduate students to faculty. Together they hold 14 IMO gold medals, and one is a Fields Medalist. Terence Tao contributed several problems.
- Problems had to meet four requirements: originality, automatic verifiability, guessproofness (no more than about a 1% chance of a lucky guess) and computational tractability (reference scripts must run in under a minute on ordinary hardware).
- Models answer by submitting Python code that saves a result object. Integer answers must match exactly. Symbolic answers are accepted when SymPy reduces the difference from the reference answer to zero.
- Frontier models solved fewer than 2% of the problems, even with multiple attempts, so more than 98% stay unsolved. MATH, GSM8K, AIME, MMLU, MathVista and Omni-Math are much closer to saturation.
- Fields Medalists Tao, Gowers and Borcherds, plus IMO coach Evan Chen, reviewed sample problems and called them exceptionally hard. The authors list these limitations: answers must be numerical, there is no proof writing, and problems take hours rather than the months typical of real research.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 209 works · 244,782 citations
- [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) — University, Cambridge, United States · 243 works · 76,506 citations
- [Cornell University](https://md.research.elamur.ai/free/en/universities/cornell-university/) — University, Ithaca, United States · 134 works · 65,159 citations
- [Harvard University](https://md.research.elamur.ai/free/en/universities/harvard-university/) — University, Cambridge, United States · 126 works · 45,130 citations
- [Rutgers, The State University of New Jersey](https://md.research.elamur.ai/free/en/universities/rutgers-the-state-university-of-new-jersey/) — University, New Brunswick, United States · 40 works · 12,486 citations
- [King's College London](https://md.research.elamur.ai/free/en/universities/king-s-college-london/) — University, London, United Kingdom · 46 works · 1,714 citations
- [University of Bristol](https://md.research.elamur.ai/free/en/universities/university-of-bristol/) — University, Bristol, United Kingdom · 18 works · 822 citations
- [University of Siegen](https://md.research.elamur.ai/free/en/universities/university-of-siegen/) — University, Siegen, Germany · 2 works · 103 citations

Also on the byline: [University of Leicester](https://ror.org/04h699437).

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 321 works

🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works

## Cite

Cite as: Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, Olli Järviniemi, Matthew Barnett, Robert Sandler, Matej Vrzala, Jaime Sevilla, Qiuyu Ren, Elizabeth Pratt, Lionel Levine, Grant Barkley, Natalie Stewart, Bogdan Grechuk, Tetiana Grechuk, Shreepranav Varma Enugandla and Mark Wildon. 2024. FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI. arXiv:2411.04872.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{glazer2024frontiermath,
  title = {{FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI}},
  author = {Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily {de Oliveira Santos}, Olli Järviniemi, Matthew Barnett, Robert Sandler, Matej Vrzala, Jaime Sevilla, Qiuyu Ren, Elizabeth Pratt, Lionel Levine, Grant Barkley, Natalie Stewart, Bogdan Grechuk, Tetiana Grechuk, Shreepranav Varma Enugandla and Mark Wildon},
  year = {2024},
  doi = {10.48550/arxiv.2411.04872},
  eprint = {2411.04872},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2411.04872}
}
```

- [← previous Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs](https://md.research.elamur.ai/free/en/papers/lai-2024-step-dpo-step-wise-preference/)
- [next → Jailbreak Attacks and Defenses Against Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/yi-2024-jailbreak-attacks-defenses-against-large/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

28 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic (2024a). Claude 3.5 Sonnet Model Card Addendum. URL : https://www - cdn . anthropic . com / fed9cc193a14b84131812372d8d5857f8f304c52/Model_Card_Claude_3_Addendum.pdf. – (2024b). Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku. URL : https://www . anthropic.c… [find](https://scholar.google.com/scholar?q=Anthropic+%282024a%29%2E+Claude+3%2E5+Sonnet+Model+Card+Addendum%2E+URL+%3A+https%3A%2F%2Fwww+%2D+cdn+%2E+anthropic+%2E+com+%2F+fed9cc193a14b84131812372d8d5857f8f304c52%2FModel%5FCard%5FClaude%5F3%5FAddendum%2Epdf%2E+%E2%80%93+%282024b%29%2E+Introducing+)
2. Burnside, W. (1911). Theory of Groups of Finite Order. Cambridge Univ. Press, 2nd Ed. [find](https://scholar.google.com/scholar?q=Burnside%2C+W%2E+%281911%29%2E+Theory+of+Groups+of+Finite+Order%2E+Cambridge+Univ%2E+Press%2C+2nd+Ed%2E)
3. Chen, Evan (2024a). Math Olympiad Hardness Scale (MOHS). Accessed: October 29, 2024. URL : [https://web.evanchen.cc/upload/MOHS-hardness.pdf](https://web.evanchen.cc/upload/MOHS-hardness.pdf). – (2024b). MOHS was a mistake. Accessed: October 29, 2024. URL : https://blog.evanchen.cc/2024/01/05/ mohs-was-a-mistake/. [↗](https://web.evanchen.cc/upload/MOHS-hardness.pdf)
4. Cobbe, Karl et al. (2021). “[Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/), 2021”. In: URL https://arxiv.org/abs/2110.14168. [↗](https://arxiv.org/abs/2110.14168)
5. Deng, Chunyuan et al. (2023). “[Investigating data contamination in modern benchmarks for large language models](https://md.research.elamur.ai/free/en/papers/deng-2024-investigating-data-contamination-modern-benchmarks/)”. In: arXiv preprint arXiv:2311.09783.
6. Frieder, Simon et al. (2024). “[Mathematical capabilities of chatgpt](https://md.research.elamur.ai/free/en/papers/frieder-2023-mathematical-capabilities-chatgpt/)”. In: Advances in neural information processing systems 36.
7. Gao, Bofei et al. (2024). “[Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models](https://md.research.elamur.ai/free/en/papers/gao-2024-omni-math-universal-olympiad-level/)”. In: arXiv preprint arXiv:2410.07985.
8. Garcia, Arnaldo (2001). “Curves over Finite Fields Attaining the Hasse-Weil Upper Bound”. In: European Congress of Mathematics. Ed. by Carles Casacuberta et al. Basel: Birkhäuser Basel, pp. 199–205. ISBN : 978-3-0348-8266-8. [find](https://scholar.google.com/scholar?q=Garcia%2C+Arnaldo+%282001%29%2E+%E2%80%9CCurves+over+Finite+Fields+Attaining+the+Hasse%2DWeil+Upper+Bound%E2%80%9D%2E+In%3A+European+Congress+of+Mathematics%2E+Ed%2E+by+Carles+Casacuberta+et+al%2E+Basel%3A+Birkh%C3%A4user+Basel%2C+pp%2E+199%E2%80%93205%2E+I)
9. Gema, Aryo Pradipta et al. (2024). “Are We Done with MMLU?” In: arXiv preprint arXiv:2406.04127. [find](https://scholar.google.com/scholar?q=Gema%2C+Aryo+Pradipta+et+al%2E+%282024%29%2E+%E2%80%9CAre+We+Done+with+MMLU%3F%E2%80%9D+In%3A+arXiv+preprint+arXiv%3A2406%2E04127%2E)
10. GoogleAI (2024). Release notes. URL : [https://ai.google.dev/gemini-api/docs/changelog](https://ai.google.dev/gemini-api/docs/changelog). [↗](https://ai.google.dev/gemini-api/docs/changelog)
11. Gulati, Aryan et al. (2024). “Putnam-AXIOM: A Functional and Static Benchmark for Measuring Higher Level Mathematical Reasoning”. In: The 4th Workshop on Mathematical Reasoning and AI at NeurIPS’24. URL : [https://openreview.net/forum?id=YXnwlZe0yf](https://openreview.net/forum?id=YXnwlZe0yf). [↗](https://openreview.net/forum?id=YXnwlZe0yf)
12. He, Chaoqun et al. (2024). “[Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems](https://md.research.elamur.ai/free/en/papers/he-2024-olympiadbench-challenging-benchmark-promoting-agi/)”. In: arXiv preprint arXiv:2402.14008.
13. Hendrycks, Dan, Collin Burns, Steven Basart, et al. (2021). [Measuring Massive Multitask Language Understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv: 2009.03300 [cs.CY]. URL : https://arxiv.org/abs/2009.03300. [↗](https://arxiv.org/abs/2009.03300)
14. Hendrycks, Dan, Collin Burns, Saurav Kadavath, et al. (2021). “[Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/)”. In: arXiv preprint arXiv:2103.03874.
15. Järviniemi, Olli (2020). Equality of orders of a set of integers modulo a prime. arXiv: 1912.02554 [math.NT]. URL : https://arxiv.org/abs/1912.02554. — [arXiv:1912.02554](https://arxiv.org/abs/1912.02554) [↗](https://arxiv.org/abs/1912.02554)
16. Ji, Caleb (2021). The Weil Conjectures for Curves. URL : [https://www.math.columbia.edu/~calebji/RH-curves.pdf](https://www.math.columbia.edu/~calebji/RH-curves.pdf). [↗](https://www.math.columbia.edu/~calebji/RH-curves.pdf)
17. Lean Community (2024). Missing undergraduate mathematics in mathlib. URL : [https://leanprover-community.github.io/undergrad_todo.html](https://leanprover-community.github.io/undergrad_todo.html). [↗](https://leanprover-community.github.io/undergrad_todo.html)
18. Lu, Pan et al. (2024). [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). arXiv: 2310.02255 [cs.CV]. URL : https://arxiv.org/abs/2310.02255. [↗](https://arxiv.org/abs/2310.02255)
19. Northcutt, Curtis G., Anish Athalye, and Jonas Mueller (2021). Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks. arXiv: 2103.14749 [stat.ML]. URL : https://arxiv.org/abs/2103.14749. [↗](https://arxiv.org/abs/2103.14749)
20. OpenAI (2024a). GPT-4o. URL : [https://platform.openai.com/docs/models](https://platform.openai.com/docs/models). – (2024b). Introducing OpenAI o1-preview. URL : https://openai.com/index/introducing- openai- o1-preview/. – (2024c). Learning to Reason with LLMs. URL : https://openai.com/index/learning-to-reason-with-llms/. – (2024d). OpenAI o… [↗](https://platform.openai.com/docs/models)
21. Romera-Paredes, Bernardino et al. (2024). “[Mathematical discoveries from program search with large language models](https://md.research.elamur.ai/free/en/papers/paredes-2023-mathematical-discoveries-program-search-large/)”. In: Nature 625.7995, pp. 468–475.
22. Sawada, Tomohiro et al. (2023). “[Arb: Advanced reasoning benchmark for large language models](https://md.research.elamur.ai/free/en/papers/sawada-2023-arb-advanced-reasoning-benchmark-large/)”. In: arXiv preprint arXiv:2307.13692.
23. Trinh, Trieu H et al. (2024). “[Solving olympiad geometry without human demonstrations](https://md.research.elamur.ai/free/en/papers/trinh-2024-solving-olympiad-geometry-without-human/)”. In: Nature 625.7995, pp. 476–482.
24. Tsoukalas, George et al. (2024). “[PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competition](https://md.research.elamur.ai/free/en/papers/tsoukalas-2024-putnambench-evaluating-neural-theorem-provers/)”. In: arXiv preprint arXiv:2407.11214.
25. XAI (2024). Grok-2 Beta Release. URL : [https://x.ai/blog/grok-2](https://x.ai/blog/grok-2). [↗](https://x.ai/blog/grok-2)
26. XTX Investments (2024). AI Mathematical Olympiad - Progress Prize 1. Accessed April 2024. Kaggle. URL : [https://kaggle.com/competitions/ai-mathematical-olympiad-prize](https://kaggle.com/competitions/ai-mathematical-olympiad-prize). [↗](https://kaggle.com/competitions/ai-mathematical-olympiad-prize)
27. Xu, Cheng et al. (2024). “[Benchmark Data Contamination of Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/xu-2024-benchmark-data-contamination-large/)”. In: arXiv preprint arXiv:2406.04244.
28. Zheng, Kunhao, Jesse Michael Han, and Stanislas Polu (2021). “[Minif2f: a cross-system benchmark for formal olympiad-level mathematics](https://md.research.elamur.ai/free/en/papers/zheng-2021-minif2f-cross-system-benchmark-formal/)”. In: arXiv preprint arXiv:2109.00110.

*Updated 2 November 2026*