---
title: "U-MATH — paper by Chernyshev et al. (2024)"
description: "Most math benchmarks for LLMs target school-level or Olympiad problems, use… 90.1% F1. 57 references, cited by 12 papers here."
authors: ["Konstantin Chernyshev", "Vitaliy Polshkov", "Ekaterina Artemova", "Alex Myasnikov", "Vlad Stepanov", "Alexei Miasnikov", "Sergei Tilga"]
year: 2024
venue: "arXiv:2412.03205"
source: "https://arxiv.org/abs/2412.03205"
doi: "10.48550/arxiv.2412.03205"
arxiv: "2412.03205"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chernyshev-2024-u-math-university-level-benchmark/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chernyshev-2024-u-math-university-level-benchmark/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/) / U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMs

# U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMs

**Authors:** Konstantin Chernyshev, Vitaliy Polshkov, [Ekaterina Artemova](https://md.research.elamur.ai/free/en/authors/ekaterina-artemova/), Alex Myasnikov, Vlad Stepanov, Alexei Miasnikov and [Sergei Tilga](https://md.research.elamur.ai/free/en/authors/sergei-tilga/) · **Year:** 2024 · **Venue:** arXiv:2412.03205 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2412.03205) · [PDF](https://arxiv.org/pdf/2412.03205) · [DOI 10.48550/arxiv.2412.03205](https://doi.org/10.48550/arxiv.2412.03205) · [arXiv 2412.03205](https://arxiv.org/abs/2412.03205) · [OpenAlex](https://openalex.org/W4405090047)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [12](https://md.research.elamur.ai/free/en/papers/chernyshev-2024-u-math-university-level-benchmark/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most math benchmarks for LLMs target school-level or Olympiad problems, use multiple-choice formats, are small, or draw on public material that may have leaked into training data. Visual math and the reliability of LLM-based grading are also poorly tested. The authors, from Toloka AI with Gradarius and Stevens Institute of Technology, built U-MATH. It contains 1,100 previously unpublished, open-ended problems from current US university courses across six core subjects, and about a fifth of them include an image. The authors filtered the problems to favor reasoning over heavy arithmetic and to keep only the hardest items. Answers are free-form, so an LLM judge grades them. To check how reliable such judges are, the team also built µ-MATH, a set of labeled model solutions for meta-evaluation. The strongest models reach 93.1% on text problems but only 58.5% on visual ones. Judging is hard too: the best judge reaches 90.1% F1. The resource offers a leakage-resistant, university-level test and a way to measure grader error.

## Key points
- **U-MATH:** 1,100 problems from Gradarius coursework, with expert-written reference solutions and roughly 20% visual tasks (one image each). The authors removed multiple-choice, calculator-based and short-solution items, and used small models to screen for difficulty.
- **µ-MATH:** 1,084 meta-evaluation tasks covering about 25% of U-MATH problems. Each problem has solutions from four strong models with ground-truth correctness labels. Only text problems are included.
- **Problem-solving results:** the top accuracy is 93.1% on text and 58.5% on visual problems. Proprietary models clearly beat open-source ones on visual tasks, while the two are nearly equal on text.
- **Judging results:** o1 reaches the best macro F1 of 90.1%. Most judges show bias and unstable behavior, and judging ability turns out to be a separate skill from solving problems.
- **Release and limitations:** the data and evaluation code are open-sourced at github.com/toloka/u-math. The text part is nearing saturation for reasoning models, and the authors suggest partial-credit scoring as future work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 252 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Konstantin Chernyshev, Vitaliy Polshkov, Ekaterina Artemova, Alex Myasnikov, Vlad Stepanov, Alexei Miasnikov and Sergei Tilga. 2024. U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMs. arXiv:2412.03205.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chernyshev2024u,
  title = {{U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMs}},
  author = {Konstantin Chernyshev, Vitaliy Polshkov, Ekaterina Artemova, Alex Myasnikov, Vlad Stepanov, Alexei Miasnikov and Sergei Tilga},
  year = {2024},
  doi = {10.48550/arxiv.2412.03205},
  eprint = {2412.03205},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.03205}
}
```

- [← previous Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix" Cycle](https://md.research.elamur.ai/free/en/papers/haider-2024-phi-3-safety-post-training/)
- [next → A Challenge Dataset and Effective Models for Conversational Stance Detection](https://md.research.elamur.ai/free/en/papers/niu-2024-challenge-dataset-effective-models-conversational/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/)

## References

57 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. [Large language models for mathematical reasoning: Progresses and challenges](https://md.research.elamur.ai/free/en/papers/ahn-2024-large-language-models-mathematical-reasoning/). Preprint, arXiv:2402.00157.
3. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. [MathQA: Towards interpretable math word problem solving with operation-based formalisms](https://md.research.elamur.ai/free/en/papers/amini-2019-mathqa-interpretable-math-word-problem/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computation…
4. Anthropic. 2024. Introducing claude 3.5 sonnet. [https://www.anthropic.com/news/claud](https://www.anthropic.com/news/claud) e-3-5-sonnet. Accessed: 2024-11-20. [↗](https://www.anthropic.com/news/claud)
5. Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W Ayers, Dragomir Radev, and Jeremy Avigad. 2023. [Proofnet: Autoformalizing and formally proving undergraduate-level mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-proofnet-autoformalizing-formally-proving-undergraduate/). arXiv preprint arXiv:2302.12433.
6. Edward Beeching, Shengyi Costa Huang, Albert Jiang, Jia Li, Benjamin Lipkin, Zihan Qina, Kashif Rasul, Ziju Shen, Roman Soletskyi, and Lewis Tunstall. 2024. Numinamath 7b cot. https://huggin gface.co/AI-MO/NuminaMath-7B-CoT. [find](https://scholar.google.com/scholar?q=Edward+Beeching%2C+Shengyi+Costa+Huang%2C+Albert+Jiang%2C+Jia+Li%2C+Benjamin+Lipkin%2C+Zihan+Qina%2C+Kashif+Rasul%2C+Ziju+Shen%2C+Roman+Soletskyi%2C+and+Lewis+Tunstall%2E+2024%2E+Numinamath+7b+cot%2E+https%3A%2F%2Fhuggin+gface%2Eco%2F)
7. Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. 2022a. [UniGeo: Unifying geometry logical reasoning via reformulating mathematical expression](https://md.research.elamur.ai/free/en/papers/chen-2022-unigeo-unifying-geometry-logical-reasoning/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3313–3323, Abu Dhabi, U…
8. Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P. Xing, and Liang Lin. 2022b. [Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning](https://md.research.elamur.ai/free/en/papers/chen-2021-geoqa-geometric-question-answering-benchmark/). Preprint, arXiv:2105.14517.
9. Nuo Chen, Ning Wu, Jianhui Chang, and Jia Li. 2024. Controlmath: Controllable data generation promotes math generalist models. Preprint, arXiv:2409.15376. [find](https://scholar.google.com/scholar?q=Nuo+Chen%2C+Ning+Wu%2C+Jianhui+Chang%2C+and+Jia+Li%2E+2024%2E+Controlmath%3A+Controllable+data+generation+promotes+math+generalist+models%2E+Preprint%2C+arXiv%3A2409%2E15376%2E)
10. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
11. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chengg… — [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/)
12. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H Zhang, Han Bao, Ha… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+Dejian+Yang%2C+Deli+Chen%2C+Dongjie+Ji%2C+Erhan)
13. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
14. Meng Fang, Xiangpeng Wan, Fei Lu, Fei Xing, and Kai Zou. 2024. Mathodyssey: Benchmarking mathematical problem-solving skills in large language models using odyssey math data. arXiv preprint arXiv:2406.18321. [find](https://scholar.google.com/scholar?q=Meng+Fang%2C+Xiangpeng+Wan%2C+Fei+Lu%2C+Fei+Xing%2C+and+Kai+Zou%2E+2024%2E+Mathodyssey%3A+Benchmarking+mathematical+problem%2Dsolving+skills+in+large+language+models+using+odyssey+math+data%2E+arXiv+preprint+arXiv%3A2406)
15. Google. 2024. gemini2-flash-thinking. https://de epmind.google/technologies/gemini/ flash-thinking/. Accessed: 2024-10-01. [find](https://scholar.google.com/scholar?q=Google%2E+2024%2E+gemini2%2Dflash%2Dthinking%2E+https%3A%2F%2Fde+epmind%2Egoogle%2Ftechnologies%2Fgemini%2F+flash%2Dthinking%2F%2E+Accessed%3A+2024%2D10%2D01%2E)
16. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300.
17. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874.
18. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
19. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). Preprint, arXiv:2…
20. Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. 2024a. Common 7b language models already possess strong math capabilities. Preprint, arXiv:2403.04706. [find](https://scholar.google.com/scholar?q=Chen+Li%2C+Weiqi+Wang%2C+Jingcheng+Hu%2C+Yixuan+Wei%2C+Nanning+Zheng%2C+Han+Hu%2C+Zheng+Zhang%2C+and+Houwen+Peng%2E+2024a%2E+Common+7b+language+models+already+possess+strong+math+capabilities%2E+Preprint%2C+arXiv%3A2403%2E0470)
21. Wangyue Li, Liangzhi Li, Tong Xiang, Xiao Liu, Wei Deng, and Noa Garcia. 2024b. [Can multiple-choice questions really be useful in detecting the abilities of llms?](https://md.research.elamur.ai/free/en/papers/li-2024-can-multiple-choice-questions-really/) Preprint, arXiv:2403.17752.
22. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). arXiv preprint arXiv:2305.20050.
23. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. [Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). arXiv preprint arXiv:2310.02255.
24. Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. 2021. [Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning](https://md.research.elamur.ai/free/en/papers/lu-2021-inter-gps-interpretable-geometry-problem/). Preprint, arXiv:2105.04165.
25. Yujun Mao, Yoon Kim, and Yilun Zhou. 2024. Champ: A competition-level dataset for fine-grained analyses of llms’ mathematical reasoning capabilities. arXiv preprint arXiv:2401.06961. [find](https://scholar.google.com/scholar?q=Yujun+Mao%2C+Yoon+Kim%2C+and+Yilun+Zhou%2E+2024%2E+Champ%3A+A+competition%2Dlevel+dataset+for+fine%2Dgrained+analyses+of+llms%E2%80%99+mathematical+reasoning+capabilities%2E+arXiv+preprint+arXiv%3A2401%2E06961%2E)
26. Meta AI. 2024. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. [https://ai.meta.com/blog/llama-3-2-connect](https://ai.meta.com/blog/llama-3-2-connect) -2024-vision-edge-mobile-devices/. Accessed: 2024-11-15. [↗](https://ai.meta.com/blog/llama-3-2-connect)
27. Mistral AI. 2024. Announsing pixtral-12b. [https://mistral.ai/news/pixtral-12b/](https://mistral.ai/news/pixtral-12b/). Accessed: 2024-10-01. [↗](https://mistral.ai/news/pixtral-12b/)
28. Mistral.ai. 2024a. Ministral. [https://mistral.ai/en/news/ministraux](https://mistral.ai/en/news/ministraux). Accessed: 2024-10-01. [↗](https://mistral.ai/en/news/ministraux)
29. Mistral.ai. 2024b. Mistral large 2. https://mist ral.ai/en/news/mistral-large-2407. Accessed: 2024-10-01. [find](https://scholar.google.com/scholar?q=Mistral%2Eai%2E+2024b%2E+Mistral+large+2%2E+https%3A%2F%2Fmist+ral%2Eai%2Fen%2Fnews%2Fmistral%2Dlarge%2D2407%2E+Accessed%3A+2024%2D10%2D01%2E)
30. Mistral.ai. 2024c. Mistral small 3. https://mi stral.ai/en/news/mistral-small-3. Accessed: 2024-10-01. [find](https://scholar.google.com/scholar?q=Mistral%2Eai%2E+2024c%2E+Mistral+small+3%2E+https%3A%2F%2Fmi+stral%2Eai%2Fen%2Fnews%2Fmistral%2Dsmall%2D3%2E+Accessed%3A+2024%2D10%2D01%2E)
31. Nexusflow. 2024. Introducing athene-v2: Advancing beyond the limits of scaling with targeted post-training. [https://nexusflow.ai/blogs/a](https://nexusflow.ai/blogs/a) thene-v2. Accessed: 2024-11-15. [↗](https://nexusflow.ai/blogs/a)
32. OpenAI. 2024a. Hello gpt-4o. [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/). Accessed: 2024-10-01. [↗](https://openai.com/index/hello-gpt-4o/)
33. OpenAI. 2024b. o1. [https://openai.com/i](https://openai.com/i) ndex/learning-to-reason-with-llms/. Accessed: 2024-10-01. [↗](https://openai.com/i)
34. OpenAI. 2024c. o1-mini. [https://openai.com/index/openai-o1-mini-advancing](https://openai.com/index/openai-o1-mini-advancing) -cost-efficient-reasoning/. Accessed: 2024-10-01. [↗](https://openai.com/index/openai-o1-mini-advancing)
35. OpenAI. 2024d. o3-mini. [https://openai.com/index/openai-o3-mini/](https://openai.com/index/openai-o3-mini/). Accessed: 2024-10-01. [↗](https://openai.com/index/openai-o3-mini/)
36. Pouya Pezeshkpour and Estevam Hruschka. 2023. [Large language models sensitivity to the order of options in multiple-choice questions](https://md.research.elamur.ai/free/en/papers/pezeshkpour-2023-large-language-models-sensitivity-order/). Preprint, arXiv:2308.11483.
37. Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. 2024. Fine-tuning enhances existing mechanisms: A case study on entity tracking. Preprint, arXiv:2402.14811. — [arXiv:2402.14811](https://arxiv.org/abs/2402.14811) [find](https://scholar.google.com/scholar?q=Nikhil+Prakash%2C+Tamar+Rott+Shaham%2C+Tal+Haklay%2C+Yonatan+Belinkov%2C+and+David+Bau%2E+2024%2E+Fine%2Dtuning+enhances+existing+mechanisms%3A+A+case+study+on+entity+tracking%2E+Preprint%2C+arXiv%3A2402%2E14811%2E+%E2%80%94+%5BarXiv%3A24)
38. Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. 2024. We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01284. [find](https://scholar.google.com/scholar?q=Runqi+Qiao%2C+Qiuna+Tan%2C+Guanting+Dong%2C+Minhui+Wu%2C+Chong+Sun%2C+Xiaoshuai+Song%2C+Zhuoma+GongQue%2C+Shanglin+Lei%2C+Zhe+Wei%2C+Miaoxuan+Zhang%2C+et+al%2E+2024%2E+We%2Dmath%3A+Does+your+large+multimodal+model+achieve+human%2D)
39. QwenLM. 2024a. Qvq 72b preview. https://qwen lm.github.io/blog/qvq-72b-preview/. Accessed: 2024-10-01. [find](https://scholar.google.com/scholar?q=QwenLM%2E+2024a%2E+Qvq+72b+preview%2E+https%3A%2F%2Fqwen+lm%2Egithub%2Eio%2Fblog%2Fqvq%2D72b%2Dpreview%2F%2E+Accessed%3A+2024%2D10%2D01%2E)
40. QwenLM. 2024b. Qwq 32b preview. https://qw enlm.github.io/blog/qwq-32b-previ ew/. Accessed: 2024-10-01. [find](https://scholar.google.com/scholar?q=QwenLM%2E+2024b%2E+Qwq+32b+preview%2E+https%3A%2F%2Fqw+enlm%2Egithub%2Eio%2Fblog%2Fqwq%2D32b%2Dprevi+ew%2F%2E+Accessed%3A+2024%2D10%2D01%2E)
41. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2022. [Language models are multilingual chain-of-thought reasoners](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/). arXiv preprint arXiv:2210.03057.
42. Andreas Stephan, Dawei Zhu, Matthias Aßenmacher, Xiaoyu Shen, and Benjamin Roth. 2024. [From calculation to adjudication: Examining llm judges on mathematical reasoning tasks](https://md.research.elamur.ai/free/en/papers/stephan-2024-calculation-adjudication-examining-llm-judges/). Preprint, arXiv:2409.04168.
43. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, Andrea Tacchetti, Colin… — [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/)
44. Qwen Team. 2024. Qwen2.5: A party of foundation models. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+2024%2E+Qwen2%2E5%3A+A+party+of+foundation+models%2E)
45. Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. 2024a. [Measuring multimodal mathematical reasoning with math-vision dataset](https://md.research.elamur.ai/free/en/papers/wang-2024-measuring-multimodal-mathematical-reasoning-math/). arXiv preprint arXiv:2402.14804.
46. Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. 2024b. Helpsteer2-preference: Complementing ratings with preferences. Preprint, arXiv:2410.01257. [find](https://scholar.google.com/scholar?q=Zhilin+Wang%2C+Alexander+Bukharin%2C+Olivier+Delalleau%2C+Daniel+Egert%2C+Gerald+Shen%2C+Jiaqi+Zeng%2C+Oleksii+Kuchaiev%2C+and+Yi+Dong%2E+2024b%2E+Helpsteer2%2Dpreference%3A+Complementing+ratings+with+preferences%2E+Preprint)
47. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824–24837.
48. Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, and Pengfei Liu. 2024. [Evaluating mathematical reasoning beyond accuracy](https://md.research.elamur.ai/free/en/papers/xia-2024-evaluating-mathematical-reasoning-accuracy/). arXiv preprint arXiv:2404.05692.
49. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Juny… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Zhou%2C+Chengpeng+Li%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Guanting+Dong%2C+Haoran+Wei%2C+Huan+Lin%2C+Jialong+Tang%2C+Jialin+Wang%2C+Jian+Yang%2C+Jian)
50. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024b. [Qwen2.5-math technical report: Toward mathematical expert model via self-improvement](https://md.research.elamur.ai/free/en/papers/yang-2024-qwen2-5-math-technical-report/).…
51. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2023. [Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/). In Proceedings of the IEEE/CVF Conference on Computer Vision an…
52. Zhongshen Zeng, Pengguang Chen, Shu Liu, Haiyun Jiang, and Jiaya Jia. 2023. Mr-gsm8k: A meta-reasoning benchmark for large language model evaluation. CoRR, abs/2312.17080. [find](https://scholar.google.com/scholar?q=Zhongshen+Zeng%2C+Pengguang+Chen%2C+Shu+Liu%2C+Haiyun+Jiang%2C+and+Jiaya+Jia%2E+2023%2E+Mr%2Dgsm8k%3A+A+meta%2Dreasoning+benchmark+for+large+language+model+evaluation%2E+CoRR%2C+abs%2F2312%2E17080%2E)
53. Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. 2024. [Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?](https://md.research.elamur.ai/free/en/papers/zhang-2024-mathverse-multi-modal-llm-truly/) arXiv preprint arXiv:2403.14624.
54. Yiran Zhao, Jinghan Zhang, I Chern, Siyang Gao, Pengfei Liu, Junxian He, et al. 2024. Felm: Benchmarking factuality evaluation of large language models. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Yiran+Zhao%2C+Jinghan+Zhang%2C+I+Chern%2C+Siyang+Gao%2C+Pengfei+Liu%2C+Junxian+He%2C+et+al%2E+2024%2E+Felm%3A+Benchmarking+factuality+evaluation+of+large+language+models%2E+Advances+in+Neural+Information+Processing+Syste)
55. Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. 2021. [Minif2f: a cross-system benchmark for formal olympiad-level mathematics](https://md.research.elamur.ai/free/en/papers/zheng-2021-minif2f-cross-system-benchmark-formal/). arXiv preprint arXiv:2109.00110.
56. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Preprint, arXiv:2306.05685. *Figure 4: Examples of text-on…
57. U-MATH Sample Problem with Reference Solution Example: Differential Calculus. [find](https://scholar.google.com/scholar?q=U%2DMATH+Sample+Problem+with+Reference+Solution+Example%3A+Differential+Calculus%2E)

*Updated 31 October 2026*