---
title: "UGPhysics — paper by Xu et al. (2025) — summary and ideas"
description: "LLMs have become strong at mathematical reasoning, but their physics skills at university level have… 77 references, cited by 18 papers here."
authors: ["Xin Xu", "Qiyun Xu", "Tong Xiao", "Tianhao Chen", "Yuchen Yan", "Jiaxin Zhang", "Shizhe Diao", "Can Yang", "Yang Wang"]
year: 2025
venue: "arXiv:2502.00334"
source: "https://arxiv.org/abs/2502.00334"
doi: "10.48550/arxiv.2502.00334"
arxiv: "2502.00334"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2025-ugphysics-comprehensive-benchmark-undergraduate-physics/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2025-ugphysics-comprehensive-benchmark-undergraduate-physics/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/) / UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models

# UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models

**Authors:** [Xin Xu](https://md.research.elamur.ai/free/en/authors/xin-xu/), Qiyun Xu, [Tong Xiao](https://md.research.elamur.ai/free/en/authors/tong-xiao/), Tianhao Chen, [Yuchen Yan](https://md.research.elamur.ai/free/en/authors/yuchen-yan/), [Jiaxin Zhang](https://md.research.elamur.ai/free/en/authors/jiaxin-zhang/), [Shizhe Diao](https://md.research.elamur.ai/free/en/authors/shizhe-diao/), Can Yang and [Yang Wang](https://md.research.elamur.ai/free/en/authors/yang-wang/) · **Year:** 2025 · **Venue:** arXiv:2502.00334 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2502.00334) · [PDF](https://arxiv.org/pdf/2502.00334) · [DOI 10.48550/arxiv.2502.00334](https://doi.org/10.48550/arxiv.2502.00334) · [arXiv 2502.00334](https://arxiv.org/abs/2502.00334) · [OpenAlex](https://openalex.org/W4407123504)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [18](https://md.research.elamur.ai/free/en/papers/xu-2025-ugphysics-comprehensive-benchmark-undergraduate-physics/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLMs have become strong at mathematical reasoning, but their physics skills at university level have been tested much less. Earlier physics benchmarks mostly target school-level material, rely on multiple-choice formats, or offer only a small slice of undergraduate topics. UGPhysics fills this gap. It contains 5,520 undergraduate physics problems, collected, cleaned and checked for data leakage, with English translations added for bilingual testing. Each problem is tagged by domain, subject, answer type and the kind of physics reasoning it requires. The authors also built MARJ (Model-Assistant Rule-based Judgment), a grading pipeline. MARJ uses rule-based checks for precise numerical comparison and calls on an LLM when answers are complex or written in equivalent forms. Human evaluation supports its reliability. The authors tested 31 LLMs, including proprietary, general open-source, math-specialized and o1-like reasoning models. The strongest model, OpenAI-o1-mini, solved fewer than half of the problems. This suggests that strong math ability does not automatically carry over to physics, and that better physics training data is needed.

## Key points
- **Scale and structure:** The dataset has 5,520 problems, or 11,040 counting both English and Chinese versions. These span 3 domains (Mechanics & Thermodynamics, Electromagnetism, Modern Physics), 13 subjects and 59 topics, with 7 answer types and 4 reasoning-skill categories.
- **Headline results:** OpenAI-o1-mini scored highest at 49.8% overall accuracy, and DeepSeek-R1-Distill-Llama-70B ranked second. Math-specialized models such as NuminaMath and Qwen2.5-Math improved only slightly on their general-purpose counterparts.
- **Difficulty compared with other benchmarks:** GPT-4o reached 38.67% on UGPhysics, versus 53.60% on GPQA and over 68% on the MMLU physics subsets.
- **Error analysis:** Human annotators reviewed 100 incorrect o1-mini answers. The main failure causes were flawed reasoning, missing knowledge and misapplied laws, not calculation slips.
- **Leakage and cost:** A 5-gram accuracy check found at most about 0.75% contaminated samples. DeepSeek-R1 used roughly 5,555 tokens per problem on average. Code and data are available at github.com/YangLabHKUST/UGPhysics (ICML 2025).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 439 works · 94,783 citations
- [Hong Kong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/hong-kong-university-of-science-and-technology/) — University, Hong Kong, China · 183 works · 27,600 citations
- [University of Science and Technology of China](https://md.research.elamur.ai/free/en/universities/university-of-science-and-technology-of-china/) — University, Hefei, China · 152 works · 114,627 citations
- [Zhejiang University](https://md.research.elamur.ai/free/en/universities/zhejiang-university/) — University, Hangzhou, China · 236 works · 14,072 citations
- [Nvidia](https://md.research.elamur.ai/free/en/companies/nvidia/) — Company, Santa Clara, United States · 40 works · 12,732 citations
- [Nvidia (United Kingdom)](https://md.research.elamur.ai/free/en/companies/nvidia-united-kingdom/) — Company, Reading, United Kingdom · 30 works · 11,217 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,621 works

🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,948 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,394 works

## Cite

Cite as: Xin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen, Yuchen Yan, Jiaxin Zhang, Shizhe Diao, Can Yang and Yang Wang. 2025. UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models. arXiv:2502.00334.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{xu2025ugphysics,
  title = {{UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models}},
  author = {Xin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen, Yuchen Yan, Jiaxin Zhang, Shizhe Diao, Can Yang and Yang Wang},
  year = {2025},
  doi = {10.48550/arxiv.2502.00334},
  eprint = {2502.00334},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.00334}
}
```

- [← previous Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains](https://md.research.elamur.ai/free/en/papers/tan-2025-think-silently-think-fast-dynamic/)
- [next → Unlocking General Long Chain-of-Thought Reasoning Capabilities of Large Language Models via Representation Engineering](https://md.research.elamur.ai/free/en/papers/tang-2025-unlocking-general-long-chain-thought/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/)

## References

77 references, 42 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 01-AI, :, Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., Yu, K., Liu, P., Liu, Q., Yue, S., Yang, S., Yang, S., Yu, T., Xie, W., Huang, W., Hu, X., Ren, X., Niu, X., Nie, P., Xu, Y., Liu, Y., Wang, Y., Cai, Y., Gu, Z., Liu, Z., and Dai, Z. Yi: Ope… — [Yi: Open Foundation Models by 01.AI](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/)
2. Arora, D., Singh, H. G., et al. [Have llms advanced enough? a challenging problem solving benchmark for large language models](https://md.research.elamur.ai/free/en/papers/arora-2023-have-llms-advanced-enough-challenging/). arXiv preprint arXiv:2305.15074, 2023.
3. Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S., Jiang, A. Q., Deng, J., Biderman, S., and Welleck, S. [Llemma: An open language model for mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/). ArXiv preprint, abs/2310.10631, 2023. URL https://arxiv.org/abs/2310.10631. [↗](https://arxiv.org/abs/2310.10631)
4. Bakhtin, A., van der Maaten, L., Johnson, J., Gustafson, L., and Girshick, R. [Phyre: A new benchmark for physical reasoning](https://md.research.elamur.ai/free/en/papers/bakhtin-2019-phyre-new-benchmark-physical-reasoning/). Advances in Neural Information Processing Systems, 32, 2019.
5. Beeching, E., Huang, S. C., Jiang, A., Li, J., Lipkin, B., Qina, Z., Rasul, K., Shen, Z., Soletskyi, R., and Tunstall, L. Numinamath 7b cot. [https://huggingface.co/AI-MO/NuminaMath-7B-CoT](https://huggingface.co/AI-MO/NuminaMath-7B-CoT), 2024. [↗](https://huggingface.co/AI-MO/NuminaMath-7B-CoT)
6. Bisk, Y., Zellers, R., Le Bras, R., Gao, J., and Choi, Y. Reasoning about physical commonsense in natural language, 2019. [find](https://scholar.google.com/scholar?q=Bisk%2C+Y%2E%2C+Zellers%2C+R%2E%2C+Le+Bras%2C+R%2E%2C+Gao%2C+J%2E%2C+and+Choi%2C+Y%2E+Reasoning+about+physical+commonsense+in+natural+language%2C+2019%2E)
7. Cao, J. and Xiao, J. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings of the 29th International Conference on Computational Linguistics, pp. 1511–1520, 2022. [find](https://scholar.google.com/scholar?q=Cao%2C+J%2E+and+Xiao%2C+J%2E+An+augmented+benchmark+dataset+for+geometric+question+answering+through+dual+parallel+text+encoding%2E+In+Proceedings+of+the+29th+International+Conference+on+Computational+Linguisti)
8. Chen, J., Tang, J., Qin, J., Liang, X., Liu, L., Xing, E. P., and Lin, L. [Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning](https://md.research.elamur.ai/free/en/papers/chen-2021-geoqa-geometric-question-answering-benchmark/). arXiv preprint arXiv:2105.14517, 2021.
9. Chen, W., Ma, X., Wang, X., and Cohen, W. W. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. ArXiv preprint, abs/2211.12588, 2022. URL https://arxiv.org/abs/2211.12588. [↗](https://arxiv.org/abs/2211.12588)
10. Chen, W., Yin, M., Ku, M., Lu, P., Wan, Y., Ma, X., Xu, J., Wang, X., and Xia, T. [Theoremqa: A theorem-driven question answering dataset](https://md.research.elamur.ai/free/en/papers/chen-2023-theoremqa-theorem-driven-question-answering/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7889–7901, 2023.
11. Chen, X., Xu, J., Liang, T., He, Z., Pang, J., Yu, D., Song, L., Liu, Q., Zhou, M., Zhang, Z., et al. Do not think that much for 2+ 3=? on the overthinking of o1-like llms. arXiv preprint arXiv:2412.[21187, 2024](https://md.research.elamur.ai/free/en/papers/chen-2024-think-that-much-2-3/).
12. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). ArXiv preprint, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.14168. [↗](https://arxiv.org/abs/2110.14168)
13. Dai, D., Deng, C., Zhao, C., Xu, R. X., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., Xie, Z., Li, Y. K., Huang, P., Luo, F., Ruan, C., Sui, Z., and Liang, W. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. CoRR, abs/2401.06066, 2024. URL https://arx… [find](https://scholar.google.com/scholar?q=Dai%2C+D%2E%2C+Deng%2C+C%2E%2C+Zhao%2C+C%2E%2C+Xu%2C+R%2E+X%2E%2C+Gao%2C+H%2E%2C+Chen%2C+D%2E%2C+Li%2C+J%2E%2C+Zeng%2C+W%2E%2C+Yu%2C+X%2E%2C+Wu%2C+Y%2E%2C+Xie%2C+Z%2E%2C+Li%2C+Y%2E+K%2E%2C+Huang%2C+P%2E%2C+Luo%2C+F%2E%2C+Ruan%2C+C%2E%2C+Sui%2C+Z%2E%2C+and+Liang%2C+W%2E+Deepseekmoe%3A+Towards+ultimate+expe)
14. DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024. [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2E+Deepseek%2Dv2%3A+A+strong%2C+economical%2C+and+efficient+mixture%2Dof%2Dexperts+language+model%2C+2024%2E)
15. DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Chen, D., Ji, D… — [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/)
16. Didolkar, A., Goyal, A., Ke, N. R., Guo, S., Valko, M., Lillicrap, T., Jimenez Rezende, D., Bengio, Y., Mozer, M. C., and Arora, S. [Metacognitive capabilities of llms: An exploration in mathematical problem solving](https://md.research.elamur.ai/free/en/papers/didolkar-2024-metacognitive-capabilities-llms-exploration-mathematical/). Advances in Neural Information Processing Systems, 37: 19783–19812, 2024.
17. Ding, J., Cen, Y., and Wei, X. Using large language model to solve and explain physics word problems approaching human level. arXiv preprint arXiv:2309.08182, 2023. — [arXiv:2309.08182](https://arxiv.org/abs/2309.08182) [find](https://scholar.google.com/scholar?q=Ding%2C+J%2E%2C+Cen%2C+Y%2E%2C+and+Wei%2C+X%2E+Using+large+language+model+to+solve+and+explain+physics+word+problems+approaching+human+level%2E+arXiv+preprint+arXiv%3A2309%2E08182%2C+2023%2E+%E2%80%94+%5BarXiv%3A2309%2E08182%5D%28https%3A%2F%2Farxiv%2E)
18. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783, 2024.
19. Gao, B., Song, F., Yang, Z., Cai, Z., Miao, Y., Dong, Q., Li, L., Ma, C., Chen, L., Xu, R., et al. [Omni-math: A universal olympiad level mathematic benchmark for large language models](https://md.research.elamur.ai/free/en/papers/gao-2024-omni-math-universal-olympiad-level/). arXiv preprint arXiv:2410.07985, 2024.
20. Giancoli, D. C. Physics for Scientists and Engineers. Prentice Hall, 2000. [find](https://scholar.google.com/scholar?q=Giancoli%2C+D%2E+C%2E+Physics+for+Scientists+and+Engineers%2E+Prentice+Hall%2C+2000%2E)
21. Hawking, S. A Brief History of Time: From the Big Bang to Black Holes. Bantam Books, 1988. [find](https://scholar.google.com/scholar?q=Hawking%2C+S%2E+A+Brief+History+of+Time%3A+From+the+Big+Bang+to+Black+Holes%2E+Bantam+Books%2C+1988%2E)
22. He, C., Luo, R., Bai, Y., Hu, S., Thai, Z. L., Shen, J., Hu, J., Han, X., Huang, Y., Zhang, Y., et al. [Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems](https://md.research.elamur.ai/free/en/papers/he-2024-olympiadbench-challenging-benchmark-promoting-agi/). arXiv preprint arXiv:2402.14008, 2024.
23. Heller, P., Keith, R., and Anderson, S. Teaching problem solving through cooperative grouping. part 1: Group versus individual problem solving. American Journal of Physics, 60(7):627–636, 1992. [find](https://scholar.google.com/scholar?q=Heller%2C+P%2E%2C+Keith%2C+R%2E%2C+and+Anderson%2C+S%2E+Teaching+problem+solving+through+cooperative+grouping%2E+part+1%3A+Group+versus+individual+problem+solving%2E+American+Journal+of+Physics%2C+60%287%29%3A627%E2%80%93636%2C+1992%2E)
24. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300, 2020.
25. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). ArXiv preprint, abs/2103.03874, 2021. URL https://arxiv.org/abs/2103.03874. [↗](https://arxiv.org/abs/2103.03874)
26. Hou, J., Ao, C., Wu, H., Kong, X., Zheng, Z., Tang, D., Li, C., Hu, X., Xu, R., Ni, S., et al. [E-eval: A comprehensive chinese k-12 education evaluation benchmark for large language models](https://md.research.elamur.ai/free/en/papers/hou-2024-e-eval-comprehensive-chinese/). arXiv preprint arXiv:2401.15927, 2024.
27. Huang, Y., Bai, Y., Zhu, Z., Zhang, J., Zhang, J., Su, T., Liu, J., Lv, C., Zhang, Y., Fu, Y., et al. [C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/). Advances in Neural Information Processing Systems, 36, 2024a.
28. Huang, Z., Wang, Z., Xia, S., Li, X., Zou, H., Xu, R., Fan, R.-Z., Ye, L., Chern, E., Ye, Y., et al. [Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai](https://md.research.elamur.ai/free/en/papers/huang-2024-olympicarena-benchmarking-multi-discipline-cognitive/). arXiv preprint arXiv:2406.12753, 2024b.
29. Jaiswal, R., Jain, D., Popat, H. P., Anand, A., Dharmadhikari, A., Marathe, A., and Shah, R. R. [Improving physics reasoning in large language models using mixture of refinement agents](https://md.research.elamur.ai/free/en/papers/jaiswal-2024-improving-physics-reasoning-large-language/). arXiv preprint arXiv:2412.00821, 2024a.
30. Jaiswal, R., Jain, D., Popat, H. P., Anand, A., Dharmadhikari, A., Marathe, A., and Shah, R. R. [Improving physics reasoning in large language models using mixture of refinement agents](https://md.research.elamur.ai/free/en/papers/jaiswal-2024-improving-physics-reasoning-large-language/). arXiv preprint arXiv:2412.00821, 2024b.
31. Klahr, P. and Waterman, D. A. Applications of Artificial Intelligence to Physics. AAAI Press, 1986. [find](https://scholar.google.com/scholar?q=Klahr%2C+P%2E+and+Waterman%2C+D%2E+A%2E+Applications+of+Artificial+Intelligence+to+Physics%2E+AAAI+Press%2C+1986%2E)
32. Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787, 2024. [find](https://scholar.google.com/scholar?q=Lambert%2C+N%2E%2C+Pyatkin%2C+V%2E%2C+Morrison%2C+J%2E%2C+Miranda%2C+L%2E%2C+Lin%2C+B%2E+Y%2E%2C+Chandu%2C+K%2E%2C+Dziri%2C+N%2E%2C+Kumar%2C+S%2E%2C+Zick%2C+T%2E%2C+Choi%2C+Y%2E%2C+et+al%2E+Rewardbench%3A+Evaluating+reward+models+for+language+modeling%2E+arXiv+preprin)
33. Larkin, J. H., McDermott, J., Simon, D. P., and Simon, H. A. Expert and novice performance in solving physics problems. Science, 208(4450):1335–1342, 1980. [find](https://scholar.google.com/scholar?q=Larkin%2C+J%2E+H%2E%2C+McDermott%2C+J%2E%2C+Simon%2C+D%2E+P%2E%2C+and+Simon%2C+H%2E+A%2E+Expert+and+novice+performance+in+solving+physics+problems%2E+Science%2C+208%284450%29%3A1335%E2%80%931342%2C+1980%2E)
34. Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). Advances in Neural Information Processing Systems, 35:3843–3857, 2022.
35. Li, H., Zhang, Y., Koto, F., Yang, Y., Zhao, H., Gong, Y., Duan, N., and Baldwin, T. [cmmlu: Measuring massive multitask language understanding in chinese](https://md.research.elamur.ai/free/en/papers/li-2023-cmmlu-measuring-massive-multitask-language/). arXiv preprint arXiv:2306.09212, 2023.
36. Liu, H., Zheng, Z., Qiao, Y., Duan, H., Fei, Z., Zhou, F., Zhang, W., Zhang, S., Lin, D., and Chen, K. [Mathbench: Evaluating the theory and application proficiency of llms with a hierarchical mathematics benchmark](https://md.research.elamur.ai/free/en/papers/liu-2024-mathbench-evaluating-theory-application-proficiency/). arXiv preprint arXiv:2405.12209, 2024.
37. Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022. [find](https://scholar.google.com/scholar?q=Lu%2C+P%2E%2C+Mishra%2C+S%2E%2C+Xia%2C+T%2E%2C+Qiu%2C+L%2E%2C+Chang%2C+K%2E%2DW%2E%2C+Zhu%2C+S%2E%2DC%2E%2C+Tafjord%2C+O%2E%2C+Clark%2C+P%2E%2C+and+Kalyan%2C+A%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+science+question+answering%2E+Advance)
38. Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. [Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). arXiv preprint arXiv:2310.02255, 2023.
39. Ma, Y., Gou, Z., Hao, J., Xu, R., Wang, S., Pan, L., Yang, Y., Cao, Y., Sun, A., Awadalla, H., et al. [Sciagent: Tool-augmented language models for scientific reasoning](https://md.research.elamur.ai/free/en/papers/ma-2024-sciagent-tool-augmented-language-models/). arXiv preprint arXiv:2402.11451, 2024.
40. McDermott, L. C. and Redish, E. F. Resource letter on physics education research. American Journal of Physics, 67(9):755–767, 1999. [find](https://scholar.google.com/scholar?q=McDermott%2C+L%2E+C%2E+and+Redish%2C+E%2E+F%2E+Resource+letter+on+physics+education+research%2E+American+Journal+of+Physics%2C+67%289%29%3A755%E2%80%93767%2C+1999%2E)
41. Mendelson, E. and Zelinski, D. E. The Physics Problem Solver. 1984. [find](https://scholar.google.com/scholar?q=Mendelson%2C+E%2E+and+Zelinski%2C+D%2E+E%2E+The+Physics+Problem+Solver%2E+1984%2E)
42. Mistral. Mathstral. [https://mistral.ai/news/mathstral/](https://mistral.ai/news/mathstral/), 2023. Accessed: 2024-09-23. [↗](https://mistral.ai/news/mathstral/)
43. Mistral. Mistral large 2. =https://mistral.ai/news/mistral-large-2407/, 2024a. [find](https://scholar.google.com/scholar?q=Mistral%2E+Mistral+large+2%2E+%3Dhttps%3A%2F%2Fmistral%2Eai%2Fnews%2Fmistral%2Dlarge%2D2407%2F%2C+2024a%2E)
44. Mistral. The future of ai: Trends and predictions. [https://mistral.ai/news/mistral-nemo/](https://mistral.ai/news/mistral-nemo/), 2024b. Accessed: September 29, 2024. [↗](https://mistral.ai/news/mistral-nemo/)
45. Mistral. mistralai/mistral-small-instruct-2409. [https://huggingface.co/mistralai/Mistral-Small-Instruct-2409](https://huggingface.co/mistralai/Mistral-Small-Instruct-2409), 2024c. [↗](https://huggingface.co/mistralai/Mistral-Small-Instruct-2409)
46. MistralAI. Ministral model card, 2024. URL [https://huggingface.co/mistralai/Ministral-8B-Instruct-2410](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410). [↗](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410)
47. Mudur, N., Cui, H., Venugopalan, S., Raccuglia, P., Brenner, M., and Norgaard, P. C. Feabench: Evaluating language models on real world physics reasoning ability. In NeurIPS 2024 Workshop on Open-World Agents, 2024. [find](https://scholar.google.com/scholar?q=Mudur%2C+N%2E%2C+Cui%2C+H%2E%2C+Venugopalan%2C+S%2E%2C+Raccuglia%2C+P%2E%2C+Brenner%2C+M%2E%2C+and+Norgaard%2C+P%2E+C%2E+Feabench%3A+Evaluating+language+models+on+real+world+physics+reasoning+ability%2E+In+NeurIPS+2024+Workshop+on+Open%2DWorl)
48. OpenAI. Hello gpt-4o. [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/), 2024a. [↗](https://openai.com/index/hello-gpt-4o/)
49. OpenAI. Learning to reason with llms. [https://openai.com/index/learning-to-reason-with-llms/](https://openai.com/index/learning-to-reason-with-llms/), 2024b. Accessed: 2024-09-23. [↗](https://openai.com/index/learning-to-reason-with-llms/)
50. Pang, X., Hong, R., Zhou, Z., Lv, F., Yang, X., Liang, Z., Han, B., and Zhang, C. Physics reasoner: Knowledge-augmented reasoning for solving physics problems with large language models. arXiv preprint arXiv:2412.13791, 2024. [find](https://scholar.google.com/scholar?q=Pang%2C+X%2E%2C+Hong%2C+R%2E%2C+Zhou%2C+Z%2E%2C+Lv%2C+F%2E%2C+Yang%2C+X%2E%2C+Liang%2C+Z%2E%2C+Han%2C+B%2E%2C+and+Zhang%2C+C%2E+Physics+reasoner%3A+Knowledge%2Daugmented+reasoning+for+solving+physics+problems+with+large+language+models%2E+arXiv+preprin)
51. Planck, M. Scientific Autobiography and Other Papers. 1949. [find](https://scholar.google.com/scholar?q=Planck%2C+M%2E+Scientific+Autobiography+and+Other+Papers%2E+1949%2E)
52. Qian, K., Wan, S., Tang, C., Wang, Y., Zhang, X., Chen, M., and Yu, Z. Varbench: Robust language model benchmarking through dynamic variable perturbation. arXiv preprint arXiv:2406.17681, 2024. [find](https://scholar.google.com/scholar?q=Qian%2C+K%2E%2C+Wan%2C+S%2E%2C+Tang%2C+C%2E%2C+Wang%2C+Y%2E%2C+Zhang%2C+X%2E%2C+Chen%2C+M%2E%2C+and+Yu%2C+Z%2E+Varbench%3A+Robust+language+model+benchmarking+through+dynamic+variable+perturbation%2E+arXiv+preprint+arXiv%3A2406%2E17681%2C+2024%2E)
53. QwQ-Team. Qwq-32b-preview. =https://qwenlm.github.io/blog/qwq-32b-preview/, 2024. [find](https://scholar.google.com/scholar?q=QwQ%2DTeam%2E+Qwq%2D32b%2Dpreview%2E+%3Dhttps%3A%2F%2Fqwenlm%2Egithub%2Eio%2Fblog%2Fqwq%2D32b%2Dpreview%2F%2C+2024%2E)
54. Redish, E. F. Teaching Physics with the Physics Suite. Wiley, 2003. [find](https://scholar.google.com/scholar?q=Redish%2C+E%2E+F%2E+Teaching+Physics+with+the+Physics+Suite%2E+Wiley%2C+2003%2E)
55. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. [Gpqa: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). arXiv preprint arXiv:2311.12022, 2023.
56. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Zhang, M., Li, Y., Wu, Y., and Guo, D. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). ArXiv preprint, abs/2402.03300, 2024. URL https://arxiv.org/abs/2402.03300. [↗](https://arxiv.org/abs/2402.03300)
57. Skywork. Skywork-o1 model card, 2024. [https://huggingface.co/Skywork/Skywork-o1-Open-Llama-3.1-8B](https://huggingface.co/Skywork/Skywork-o1-Open-Llama-3.1-8B). [↗](https://huggingface.co/Skywork/Skywork-o1-Open-Llama-3.1-8B)
58. URL Srivastava, S., PV, A., Menon, S., Sukumar, A., Philipose, A., Prince, S., Thomas, S., et al. [Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap](https://md.research.elamur.ai/free/en/papers/srivastava-2024-functional-benchmarks-robust-evaluation-reasoning/). arXiv preprint arXiv:2402.19450, 2024.
59. Tang, Z., Zhang, X., Wan, B., and Wei, F. [Mathscale: Scaling instruction tuning for mathematical reasoning](https://md.research.elamur.ai/free/en/papers/tang-2024-mathscale-scaling-instruction-tuning-mathematical/). arXiv preprint arXiv:2403.02884, 2024.
60. Tong, Y., Zhang, X., Wang, R., Wu, R., and He, J. [Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving](https://md.research.elamur.ai/free/en/papers/tong-2024-dart-math-difficulty-aware-rejection/). arXiv preprint arXiv:2407.13690, 2024.
61. Toshniwal, S., Du, W., Moshkov, I., Kisacanin, B., Ayrapetyan, A., and Gitman, I. [Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data](https://md.research.elamur.ai/free/en/papers/toshniwal-2024-openmathinstruct-2-accelerating-ai-math/). arXiv preprint arXiv:2410.01560, 2024.
62. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). ArXiv preprint, abs/2203.11171, 2022. URL https://arxiv.org/abs/2203.11171. [↗](https://arxiv.org/abs/2203.11171)
63. Wang, X., Hu, Z., Lu, P., Zhu, Y., Zhang, J., Subramaniam, S., Loomba, A. R., Zhang, S., Sun, Y., and Wang, W. [Scibench: Evaluating college-level scientific problem-solving abilities of large language models](https://md.research.elamur.ai/free/en/papers/wang-2023-scibench-evaluating-college-level-scientific/). arXiv preprint arXiv:2307.10635, 2023a.
64. Wang, Y. R., Duan, J., Fox, D., and Srinivasa, S. [Newton: Are large language models capable of physical reasoning?](https://md.research.elamur.ai/free/en/papers/wang-2023-newton-are-large-language-models/) arXiv preprint arXiv:2310.07018, 2023b.
65. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35: 24824–24837, 2022.
66. Welbl, J., Liu, N. F., and Gardner, M. [Crowdsourcing multiple choice science questions](https://md.research.elamur.ai/free/en/papers/welbl-2017-crowdsourcing-multiple-choice-science-questions/). arXiv preprint arXiv:1707.06209, 2017.
67. White, C., Dooley, S., Roberts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv, R., Jain, N., Saifullah, K., Naidu, S., et al. Livebench: A challenging, contamination-free llm benchmark. arXiv preprint arXiv:2406.19314, 2024. [find](https://scholar.google.com/scholar?q=White%2C+C%2E%2C+Dooley%2C+S%2E%2C+Roberts%2C+M%2E%2C+Pal%2C+A%2E%2C+Feuer%2C+B%2E%2C+Jain%2C+S%2E%2C+Shwartz%2DZiv%2C+R%2E%2C+Jain%2C+N%2E%2C+Saifullah%2C+K%2E%2C+Naidu%2C+S%2E%2C+et+al%2E+Livebench%3A+A+challenging%2C+contamination%2Dfree+llm+benchmark%2E+arXiv+preprint)
68. Xu, R., Wang, Z., Fan, R.-Z., and Liu, P. [Benchmarking benchmark leakage in large language models](https://md.research.elamur.ai/free/en/papers/xu-2024-benchmarking-benchmark-leakage-large-language/). arXiv preprint arXiv:2404.18824, 2024a.
69. Xu, X., Diao, S., Yang, C., and Wang, Y. Can we verify step by step for incorrect answer detection? arXiv preprint arXiv:2402.10528, 2024b. — [arXiv:2402.10528](https://arxiv.org/abs/2402.10528) [find](https://scholar.google.com/scholar?q=Xu%2C+X%2E%2C+Diao%2C+S%2E%2C+Yang%2C+C%2E%2C+and+Wang%2C+Y%2E+Can+we+verify+step+by+step+for+incorrect+answer+detection%3F+arXiv+preprint+arXiv%3A2402%2E10528%2C+2024b%2E+%E2%80%94+%5BarXiv%3A2402%2E10528%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E10528%29)
70. Xu, X., Xiao, T., Chao, Z., Huang, Z., Yang, C., and Wang, Y. Can llms solve longer math word problems better? arXiv preprint arXiv:2405.14804, 2024c. — [arXiv:2405.14804](https://arxiv.org/abs/2405.14804) [find](https://scholar.google.com/scholar?q=Xu%2C+X%2E%2C+Xiao%2C+T%2E%2C+Chao%2C+Z%2E%2C+Huang%2C+Z%2E%2C+Yang%2C+C%2E%2C+and+Wang%2C+Y%2E+Can+llms+solve+longer+math+word+problems+better%3F+arXiv+preprint+arXiv%3A2405%2E14804%2C+2024c%2E+%E2%80%94+%5BarXiv%3A2405%2E14804%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2405%2E1)
71. Xu, X., Zhang, J., Chen, T., Chao, Z., Hu, J., and Yang, C. Ugmathbench: A diverse and dynamic benchmark for undergraduate-level mathematical reasoning with large language models. arXiv preprint arXiv:2501.13766, 2025. — [arXiv:2501.13766](https://arxiv.org/abs/2501.13766) [find](https://scholar.google.com/scholar?q=Xu%2C+X%2E%2C+Zhang%2C+J%2E%2C+Chen%2C+T%2E%2C+Chao%2C+Z%2E%2C+Hu%2C+J%2E%2C+and+Yang%2C+C%2E+Ugmathbench%3A+A+diverse+and+dynamic+benchmark+for+undergraduate%2Dlevel+mathematical+reasoning+with+large+language+models%2E+arXiv+preprint+arXiv)
72. Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024a. [find](https://scholar.google.com/scholar?q=Yang%2C+A%2E%2C+Yang%2C+B%2E%2C+Zhang%2C+B%2E%2C+Hui%2C+B%2E%2C+Zheng%2C+B%2E%2C+Yu%2C+B%2E%2C+Li%2C+C%2E%2C+Liu%2C+D%2E%2C+Huang%2C+F%2E%2C+Wei%2C+H%2E%2C+et+al%2E+Qwen2%2E+5+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E15115%2C+2024a%2E)
73. Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.[12122, 2024b](https://md.research.elamur.ai/free/en/papers/yang-2024-qwen2-5-math-technical-report/).
74. Zhang, H., Da, J., Lee, D., Robinson, V., Wu, C., Song, W., Zhao, T., Raja, P., Slack, D., Lyu, Q., et al. [A careful examination of large language model performance on grade school arithmetic](https://md.research.elamur.ai/free/en/papers/zhang-2024-careful-examination-large-language-model/). arXiv preprint arXiv:2405.00332, 2024.
75. Zhang, X., Li, C., Zong, Y., Ying, Z., He, L., and Qiu, X. [Evaluating the performance of large language models on gaokao benchmark](https://md.research.elamur.ai/free/en/papers/zhang-2023-evaluating-performance-large-language/). arXiv preprint arXiv:2305.12474, 2023.
76. Zheng, C., Zhang, Z., Zhang, B., Lin, R., Lu, K., Yu, B., Liu, D., Zhou, J., and Lin, J. [Processbench: Identifying process errors in mathematical reasoning](https://md.research.elamur.ai/free/en/papers/zheng-2025-processbench-identifying-process-errors-mathematical/). arXiv preprint arXiv:2412.06559, 2024.
77. Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., and Duan, N. [Agieval: A human-centric benchmark for evaluating foundation models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/). arXiv preprint arXiv:2304.06364, 2023.

*Updated 3 November 2026*