---
title: "MathOdyssey — paper by Fang et al. (2024)"
description: "To address this, the authors release MathOdyssey, a set of 387 original problems. 20 references, cited by 32 papers here."
authors: ["Meng Fang", "Xiangpeng Wan", "Fei Lu", "Fei Xing", "Kai Zou"]
year: 2024
venue: "arXiv:2406.18321"
source: "https://arxiv.org/abs/2406.18321"
doi: "10.48550/arxiv.2406.18321"
arxiv: "2406.18321"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/fang-2024-mathodyssey-benchmarking-mathematical-problem-solving/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/fang-2024-mathodyssey-benchmarking-mathematical-problem-solving/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/) / MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Data

# MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Data

**Authors:** [Meng Fang](https://md.research.elamur.ai/free/en/authors/meng-fang/), Xiangpeng Wan, Fei Lu, Fei Xing and [Kai Zou](https://md.research.elamur.ai/free/en/authors/kai-zou/) · **Year:** 2024 · **Venue:** [arXiv:2406.18321](https://md.research.elamur.ai/free/en/venues/scientific-data/) ([Scientific Data 2024](https://md.research.elamur.ai/free/en/venues/scientific-data/2024/)) · **Publisher:** [Nature Portfolio](https://md.research.elamur.ai/free/en/publishers/nature-portfolio/) · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2406.18321) · [PDF](https://arxiv.org/pdf/2406.18321) · [DOI 10.48550/arxiv.2406.18321](https://doi.org/10.48550/arxiv.2406.18321) · [arXiv 2406.18321](https://arxiv.org/abs/2406.18321) · [OpenAlex](https://openalex.org/W4413054423)

**Citations:** 14 citations (OpenAlex, 2026-10-14) · [32](https://md.research.elamur.ai/free/en/papers/fang-2024-mathodyssey-benchmarking-mathematical-problem-solving/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Leading LLMs now score very high on GSM8K and MATH, and those benchmarks may have leaked into training data. As a result, they reveal little about how models handle truly difficult mathematics. To address this, the authors release MathOdyssey, a set of 387 original problems. High-school teachers, researchers and university professors wrote them at the invitation of the GAIC Math organization (AGI Odyssey). The items fall into three tiers (Olympiad-level, High School and University-level) and cover topics from algebra and number theory to differential equations and statistics. Each problem has an objective answer and a worked solution.

The authors tested Llama-3-70B, DBRX-Instruct and several GPT-series models with chain-of-thought prompting, and they cite published Gemini and Claude figures for comparison. Models handle school-level questions reasonably well but fail on Olympiad items, where the best accuracy is only about 10%. Open models nearly equal GPT-4 Turbo on that hardest tier but fall far behind on easier levels. The data and code are public, which gives the field a fresh, contamination-resistant test of mathematical reasoning.

## Key points
- MathOdyssey contains 387 expert-written problems in three difficulty tiers. The subject list runs from algebra, number theory, geometry, combinatorics and pre-calculus through calculus with analysis, abstract as well as linear algebra, and differential equations, to probability and statistics. The problems mix multiple-choice and open-answer formats.
- With chain-of-thought prompting, GPT-4 Turbo had the best overall result among the directly tested models at 47.03%:
  - Olympiad-level: 10.14%
  - High School: 84.78%
  - University-level: 49.50%
- Overall scores for the other directly tested models were GPT-4 37.21%, Llama-3-70B 27.91%, DBRX-Instruct 23.77% and GPT-3.5 Turbo 19.64%. The overall figures taken from the Gemini 1.5 report were Gemini 1.5 Pro 45.0%, Math-Specialized 1.5 Pro 55.8% and Claude 3 Opus 40.6%.
- On Olympiad problems, Llama-3-70B (9.46%) came close to GPT-4 Turbo (10.14%). On High School problems the gap was more than 30 points (52.17% vs. 84.78%). Llama-3-70B had the top Olympiad Number Theory score (25%).
- Stated limitations are incomplete coverage of reasoning types and the absence of proofs, visual tasks and interactive tasks, which the authors list as future extensions. Resources are at mathodyssey.github.io.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Johns Hopkins University](https://md.research.elamur.ai/free/en/universities/johns-hopkins-university/) — University, Baltimore, United States · 254 works · 106,461 citations
- [University of Liverpool](https://md.research.elamur.ai/free/en/universities/university-of-liverpool/) — University, Liverpool, United Kingdom · 19 works · 840 citations

Also on the byline: [Mathematica Policy Research](https://ror.org/02403vr89).

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [➗ Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/) · 233 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 496 works · [➗ Applied Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/applied-mathematics/) · 73 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,948 works

## Cite

Cite as: Meng Fang, Xiangpeng Wan, Fei Lu, Fei Xing and Kai Zou. 2024. MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Data. arXiv:2406.18321.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{fang2024mathodyssey,
  title = {{MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Data}},
  author = {Meng Fang, Xiangpeng Wan, Fei Lu, Fei Xing and Kai Zou},
  year = {2024},
  doi = {10.48550/arxiv.2406.18321},
  eprint = {2406.18321},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.18321}
}
```

- [← previous Language models scale reliably with over-training and on downstream tasks](https://md.research.elamur.ai/free/en/papers/gadre-2024-language-models-scale-reliably-over/)
- [next → Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages](https://md.research.elamur.ai/free/en/papers/yue-2024-pangea-fully-open-multilingual-multimodal/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/)

## References

20 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
2. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint…
3. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.05530,…
4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
5. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). NeurIPS, 2021a.
6. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
7. Iddo Drori, Sarah Zhang, Zad Chin, Reece Shuttleworth, Albert Lu, Linda Chen, Bereket Birbo, Michele He, Pedro Lantigua, Sunny Tran, et al. A dataset for learning university stem courses at scale and generating questions at a human level. In Proceedings of the AAAI Conference on Artificial Intellige… [find](https://scholar.google.com/scholar?q=Iddo+Drori%2C+Sarah+Zhang%2C+Zad+Chin%2C+Reece+Shuttleworth%2C+Albert+Lu%2C+Linda+Chen%2C+Bereket+Birbo%2C+Michele+He%2C+Pedro+Lantigua%2C+Sunny+Tran%2C+et+al%2E+A+dataset+for+learning+university+stem+courses+at+scale+and+)
8. Ernest Davis and Scott Aaronson. Testing gpt-4 with wolfram alpha and code interpreter plug-ins on math and science problems. arXiv preprint arXiv:2308.05713, 2023. [find](https://scholar.google.com/scholar?q=Ernest+Davis+and+Scott+Aaronson%2E+Testing+gpt%2D4+with+wolfram+alpha+and+code+interpreter+plug%2Dins+on+math+and+science+problems%2E+arXiv+preprint+arXiv%3A2308%2E05713%2C+2023%2E)
9. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). NeurIPS, 2021b.
10. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824–24837, 2022.
11. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171, 2022.
12. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/). arXiv preprint arXiv:2205.10625, 2022.
13. Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W. Ayers, Dragomir R. Radev, and Jeremy Avigad. [Proofnet: Autoformalizing and formally proving undergraduate-level mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-proofnet-autoformalizing-formally-proving-undergraduate/). ArXiv, abs/2302.12433, 2023a.
14. Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). In Alic…
15. Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. [MiniF2F: A cross-system benchmark for formal Olympiad-level mathematics](https://md.research.elamur.ai/free/en/papers/zheng-2021-minif2f-cross-system-benchmark-formal/), 2022.
16. Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He He, and Thang Luong. [Solving olympiad geometry without human demonstrations](https://md.research.elamur.ai/free/en/papers/trinh-2024-solving-olympiad-geometry-without-human/). Nature, 625(7995):476–482, 2024. ISSN 0028-0836, 1476-4687. doi: 10.1038/s41586-023-06747-5.
17. Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. [Llemma: An Open Language Model For Mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/), 2023b.
18. Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Petersen, and Julius Berner. [Mathematical capabilities of chatgpt](https://md.research.elamur.ai/free/en/papers/frieder-2023-mathematical-capabilities-chatgpt/). Advances in Neural Information Processing Systems, 36, 2024.
19. Anthropic. The claude 3 model family: Opus, sonnet, haiku. 2024. [find](https://scholar.google.com/scholar?q=Anthropic%2E+The+claude+3+model+family%3A+Opus%2C+sonnet%2C+haiku%2E+2024%2E)
20. Mosaic. Introducing dbrx: A new state-of-the-art open llm, March 2024. URL [https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm](https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm). [↗](https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm)

*Updated 3 November 2026*