---
title: "InternLM-Math — paper by Ying et al. (2024)"
description: "InternLM-Math is a family of open math-specialised LLMs from Shanghai AI Laboratory and partner… 75 references, cited by 87 papers here."
authors: ["Huaiyuan Ying", "Shuo Zhang", "Linyang Li", "Zhejian Zhou", "Yunfan Shao", "Zhaoye Fei", "Yichuan Ma", "Jiawei Hong", "Kuikun Liu", "Ziyi Wang", "Yudong Wang", "Zijian Wu", "Shuaibin Li", "Fengzhe Zhou", "Hongwei Liu", "Songyang Zhang", "Wenwei Zhang", "Hang Yan", "Xipeng Qiu", "Jiayu Wang", "Kai Chen", "Dahua Lin"]
year: 2024
venue: "arXiv:2402.06332"
source: "https://arxiv.org/abs/2402.06332"
doi: "10.48550/arxiv.2402.06332"
arxiv: "2402.06332"
licence: "CC BY-NC-ND 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-nd/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ying-2024-internlm-math-open-math-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ying-2024-internlm-math-open-math-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning

# InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning

**Authors:** [Huaiyuan Ying](https://md.research.elamur.ai/free/en/authors/huaiyuan-ying/), [Shuo Zhang](https://md.research.elamur.ai/free/en/authors/shuo-zhang/), [Linyang Li](https://md.research.elamur.ai/free/en/authors/linyang-li/), [Zhejian Zhou](https://md.research.elamur.ai/free/en/authors/zhejian-zhou/), [Yunfan Shao](https://md.research.elamur.ai/free/en/authors/yunfan-shao/), [Zhaoye Fei](https://md.research.elamur.ai/free/en/authors/zhaoye-fei/), [Yichuan Ma](https://md.research.elamur.ai/free/en/authors/yichuan-ma/), Jiawei Hong, [Kuikun Liu](https://md.research.elamur.ai/free/en/authors/kuikun-liu/), [Ziyi Wang](https://md.research.elamur.ai/free/en/authors/ziyi-wang/), [Yudong Wang](https://md.research.elamur.ai/free/en/authors/yudong-wang/), [Zijian Wu](https://md.research.elamur.ai/free/en/authors/zijian-wu/), Shuaibin Li, [Fengzhe Zhou](https://md.research.elamur.ai/free/en/authors/fengzhe-zhou/), [Hongwei Liu](https://md.research.elamur.ai/free/en/authors/hongwei-liu/), [Songyang Zhang](https://md.research.elamur.ai/free/en/authors/songyang-zhang/), [Wenwei Zhang](https://md.research.elamur.ai/free/en/authors/wenwei-zhang/), [Hang Yan](https://md.research.elamur.ai/free/en/authors/hang-yan/), [Xipeng Qiu](https://md.research.elamur.ai/free/en/authors/xipeng-qiu/), [Jiayu Wang](https://md.research.elamur.ai/free/en/authors/jiayu-wang/), [Kai Chen](https://md.research.elamur.ai/free/en/authors/kai-chen/) and [Dahua Lin](https://md.research.elamur.ai/free/en/authors/dahua-lin/) · **Year:** 2024 · **Venue:** arXiv:2402.06332 · **Licence:** [CC BY-NC-ND 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-nd/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.06332) · [PDF](https://arxiv.org/pdf/2402.06332) · [DOI 10.48550/arxiv.2402.06332](https://doi.org/10.48550/arxiv.2402.06332) · [arXiv 2402.06332](https://arxiv.org/abs/2402.06332) · [OpenAlex](https://openalex.org/W4391766676)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [87](https://md.research.elamur.ai/free/en/papers/ying-2024-internlm-math-open-math-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
InternLM-Math is a family of open math-specialised LLMs from Shanghai AI Laboratory and partner universities. Its long-term goal is mathematical reasoning whose results can be checked. The authors continued pre-training InternLM2-Base on about 31.2B unique tokens of math data, which came to roughly 125B tokens seen over training. The data had three parts: math pages retrieved from Common Crawl with Query of CC, curated domain sources (open-web-math, algebraic-stack, arXiv, forums, books) and synthetic arithmetic data. The data was deduplicated and checked for overlap with the MATH test set. Supervised fine-tuning then puts several skills into one seq2seq query–response format: chain-of-thought solving, interleaved reasoning and Python execution (RICO), outcome and process reward modelling, LEAN-based formal proving and translation, and generating new problems. The resulting 7B and 20B models were the strongest open models at the time on GSM8K, MATH, the Hungarian exam, MathBench-ZH and MiniF2F. The paper reports they reach over 90% of GPT-4's level. Because one model can generate, verify and formalise solutions, it can serve as a base for self-improving math LLMs.

## Key points
- Pre-training data: about 20B tokens retrieved via Query of CC, 11B tokens of domain-specific math data and 0.2B tokens of synthetic numerical data. No LLM-generated text was used. Most sources were repeated for 4 epochs, and MinHash-LSH removed near-duplicates above 0.7 similarity.
- Without any fine-tuning, the base model scores 30.3 on the MiniF2F test set, ahead of Llemma and Minerva in few-shot settings.
- InternLM-Math-7B reaches 34.6 on MATH and 78.1 on GSM8K. An ablation shows that adding chain-of-thought and code-interpreter data on top of MetaMath gives small gains.
- In a manual check of the 20B model, 25 MATH answers with correct final results had sound reasoning in only 20 cases. Errors grew with difficulty level, which shows the limits of scoring by final answer alone.
- Limitations: formal SFT data is in LEAN 3 rather than LEAN 4, process reward modelling is weak, the model is sensitive to prompt wording, and AlgebraicStack was not checked for MiniF2F contamination. Code and data are at github.com/InternLM/InternLM-Math.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 321 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, Yudong Wang, Zijian Wu, Shuaibin Li, Fengzhe Zhou, Hongwei Liu, Songyang Zhang, Wenwei Zhang, Hang Yan, Xipeng Qiu, Jiayu Wang, Kai Chen and Dahua Lin. 2024. InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning. arXiv:2402.06332.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ying2024internlm,
  title = {{InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning}},
  author = {Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, Yudong Wang, Zijian Wu, Shuaibin Li, Fengzhe Zhou, Hongwei Liu, Songyang Zhang, Wenwei Zhang, Hang Yan, Xipeng Qiu, Jiayu Wang, Kai Chen and Dahua Lin},
  year = {2024},
  doi = {10.48550/arxiv.2402.06332},
  eprint = {2402.06332},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.06332}
}
```

- [← previous CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark](https://md.research.elamur.ai/free/en/papers/romero-2024-cvqa-culturally-diverse-multilingual-visual/)
- [next → Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling](https://md.research.elamur.ai/free/en/papers/maini-2024-rephrasing-web-recipe-compute-data/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

75 references, 50 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alibaba. Qwen technical report, 2023. [find](https://scholar.google.com/scholar?q=Alibaba%2E+Qwen+technical+report%2C+2023%2E)
2. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403, 2023.
3. Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W. Ayers, Dragomir Radev, and Jeremy Avigad. [Proofnet: Autoformalizing and formally proving undergraduate-level mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-proofnet-autoformalizing-formally-proving-undergraduate/), 2023a.
4. Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. [Llemma: An open language model for mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/), 2023b.
5. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. [find](https://scholar.google.com/scholar?q=Xiao+Bi%2C+Deli+Chen%2C+Guanting+Chen%2C+Shanhuang+Chen%2C+Damai+Dai%2C+Chengqi+Deng%2C+Honghui+Ding%2C+Kai+Dong%2C+Qiushi+Du%2C+Zhe+Fu%2C+et+al%2E+Deepseek+llm%3A+Scaling+open%2Dsource+language+models+with+longtermism%2E+arXiv+)
6. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, C… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
8. Nuo Chen, Zinan Zheng, Ning Wu, Linjun Shou, Ming Gong, Yangqiu Song, Dongmei Zhang, and Jia Li. [Breaking language barriers in multilingual mathematical reasoning: Insights and observations](https://md.research.elamur.ai/free/en/papers/chen-2024-breaking-language-barriers-multilingual-mathematical/), 2023a.
9. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023b. [find](https://scholar.google.com/scholar?q=Wenhu+Chen%2C+Xueguang+Ma%2C+Xinyi+Wang%2C+and+William+W%2E+Cohen%2E+Program+of+thoughts+prompting%3A+Disentangling+computation+from+reasoning+for+numerical+reasoning+tasks%2E+Transactions+on+Machine+Learning+Resea)
10. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
11. OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. [https://github.com/open-compass/opencompass](https://github.com/open-compass/opencompass), 2023. [↗](https://github.com/open-compass/opencompass)
12. Tri Dao. [FlashAttention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). 2023.
13. Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber. Implicit chain of thought reasoning via knowledge distillation. ArXiv, abs/2311.01460, 2023. URL [https://api.semanticscholar.org/CorpusID:264935229](https://api.semanticscholar.org/CorpusID:264935229). [↗](https://api.semanticscholar.org/CorpusID:264935229)
14. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. [Glm: General language model pretraining with autoregressive blank infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 320–335, 2022.
15. Zhaoye Fei, Yunfan Shao, Linyang Li, Zhiyuan Zeng, Hang Yan, Xipeng Qiu, and Dahua Lin. Query of CC: [Unearthing Large Scale Domain-Specific Knowledge from Public Corpora](https://md.research.elamur.ai/free/en/papers/fei-2024-unearthing-large-scale-domain-specific/). arXiv e-prints, art. arXiv:2401.14624, January 2024. doi: 10.48550/arXiv.2401.14624.
16. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models, 2023. [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Pengfei+Liu%2C+Yiming+Yang%2C+Jamie+Callan%2C+and+Graham+Neubig%2E+Pal%3A+Program%2Daided+language+models%2C+2023%2E)
17. Fabian Gloeckle, Baptiste Roziere, Amaury Hayat, and Gabriel Synnaeve. Temperature-scaled large language models for lean proofstep prediction. In The 3rd Workshop on Mathematical Reasoning and AI at NeurIPS’23, 2023. [find](https://scholar.google.com/scholar?q=Fabian+Gloeckle%2C+Baptiste+Roziere%2C+Amaury+Hayat%2C+and+Gabriel+Synnaeve%2E+Temperature%2Dscaled+large+language+models+for+lean+proofstep+prediction%2E+In+The+3rd+Workshop+on+Mathematical+Reasoning+and+AI+at+N)
18. Google. [Gemini: A family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/), 2023.
19. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. [Tora: A tool-integrated reasoning agent for mathematical problem solving](https://md.research.elamur.ai/free/en/papers/gou-2023-tora-tool-integrated-reasoning-agent/), 2023.
20. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming – the rise of code intelligence, 2024. [find](https://scholar.google.com/scholar?q=Daya+Guo%2C+Qihao+Zhu%2C+Dejian+Yang%2C+Zhenda+Xie%2C+Kai+Dong%2C+Wentao+Zhang%2C+Guanting+Chen%2C+Xiao+Bi%2C+Y%2E+Wu%2C+Y%2E+K%2E+Li%2C+Fuli+Luo%2C+Yingfei+Xiong%2C+and+Wenfeng+Liang%2E+Deepseek%2Dcoder%3A+When+the+large+language+model)
21. Jesse Michael Han, Jason Rute, Yuhuai Wu, Edward W Ayers, and Stanislas Polu. [Proof artifact co-training for theorem proving with language models](https://md.research.elamur.ai/free/en/papers/han-2021-proof-artifact-co-training-theorem/). arXiv preprint arXiv:2102.06203, 2021.
22. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the MATH dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). 2021a.
23. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874, 2021b.
24. Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. [Minicpm: Unveiling the potential of small language models with scalable training strategies](https://md.research.elamur.ai/free/en/papers/hu-2024-minicpm-unveiling-potential-small-language/). arXiv preprint arXiv:2404.06395, 2024.
25. Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. [Large language models can self-improve](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/), 2022.
26. InternLM. Internlm: A multilingual language model with progressively enhanced capabilities. [https://github.com/InternLM/InternLM](https://github.com/InternLM/InternLM), 2023. [↗](https://github.com/InternLM/InternLM)
27. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+Mistral+7b)
28. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+et+al%2E+Mixtral+of+expe)
29. Khan. Khan-exercises. [https://github.com/Khan/khan-exercises](https://github.com/Khan/khan-exercises), 2021. [↗](https://github.com/Khan/khan-exercises)
30. Guillaume Lample, Timothee Lacroix, Marie-Anne Lachaux, Aurelien Rodriguez, Amaury Hayat, Thibaut Lavril, Gabriel Ebner, and Xavier Martinet. [Hypertree proof search for neural theorem proving](https://md.research.elamur.ai/free/en/papers/lample-2022-hypertree-proof-search-neural-theorem/). Advances in neural information processing systems, 35:26337–26349, 2022.
31. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/), 2022a.
32. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). In NeurIPS, 2022b.
33. Chengpeng Li, Zheng Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang, and Chang Zhou. Query and response augmentation cannot help out-of-domain math reasoning generalization. arXiv preprint arXiv:2310.05506, 2023. [find](https://scholar.google.com/scholar?q=Chengpeng+Li%2C+Zheng+Yuan%2C+Guanting+Dong%2C+Keming+Lu%2C+Jiancan+Wu%2C+Chuanqi+Tan%2C+Xiang+Wang%2C+and+Chang+Zhou%2E+Query+and+response+augmentation+cannot+help+out%2Dof%2Ddomain+math+reasoning+generalization%2E+arXiv+)
34. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.[20050, 2023](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/).
35. Haoxiong Liu and Andrew Chi-Chih Yao. [Augmenting math word problems via iterative question composing](https://md.research.elamur.ai/free/en/papers/liu-2024-augmenting-math-word-problems-iterative/), 2024.
36. Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou, Wenwei Zhang, Songyang Zhang, Dahua Lin, and Kai Chen. [Mathbench: Evaluating the theory and application proficiency of llms with a hierarchical mathematics benchmark](https://md.research.elamur.ai/free/en/papers/liu-2024-mathbench-evaluating-theory-application-proficiency/), 2024.
37. Tiedong Liu and Bryan Kian Hsiang Low. [Goat: Fine-tuned llama outperforms gpt-4 on arithmetic tasks](https://md.research.elamur.ai/free/en/papers/liu-2023-goat-fine-tuned-llama-outperforms/). arXiv preprint arXiv:2305.14201, 2023.
38. Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. arXiv e-prints, art. arXiv:1711.05101, November 2017. doi: 10.48550/arXiv.1711.05101. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+Decoupled+Weight+Decay+Regularization%2E+arXiv+e%2Dprints%2C+art%2E+arXiv%3A1711%2E05101%2C+November+2017%2E+doi%3A+10%2E48550%2FarXiv%2E1711%2E05101%2E)
39. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. [Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/), 2023.
40. Math-eval. TAL-SCQ5K. [https://github.com/math-eval/TAL-SCQ5K](https://github.com/math-eval/TAL-SCQ5K), 2023. [↗](https://github.com/math-eval/TAL-SCQ5K)
41. The mathlib Community. [The lean mathematical library](https://md.research.elamur.ai/free/en/papers/community-2019-lean-mathematical-library/). In Proceedings of the 9th ACM SIGPLAN International Conference on Certified Programs and Proofs, POPL ’20. ACM, January 2020. doi: 10.1145/3372885.3373824. URL http://dx.doi.org/10.1145/3372885.3373824. [↗](http://dx.doi.org/10.1145/3372885.3373824)
42. Leonardo de Moura and Sebastian Ullrich. The lean 4 theorem prover and programming language. In André Platzer and Geoff Sutcliffe (eds.), Automated Deduction – CADE 28, pp. 625–635, Cham, 2021. [Springer International Publishing](https://md.research.elamur.ai/free/en/papers/moura-2021-lean-4-theorem-prover-programming/). ISBN 978-3-030-79876-5.
43. Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models, 2023. [find](https://scholar.google.com/scholar?q=Niklas+Muennighoff%2C+Alexander+M%2E+Rush%2C+Boaz+Barak%2C+Teven+Le+Scao%2C+Aleksandra+Piktus%2C+Nouamane+Tazi%2C+Sampo+Pyysalo%2C+Thomas+Wolf%2C+and+Colin+Raffel%2E+Scaling+data%2Dconstrained+language+models%2C+2023%2E)
44. Maxwell Nye, Anders Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. [Show your work: Scratchpads for intermediate computation with language models](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/). ArXiv, abs/2112.00114, 2021. URL ht…
45. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023.
46. Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. [Openwebmath: An open dataset of high-quality mathematical web text](https://md.research.elamur.ai/free/en/papers/paster-2023-openwebmath-open-dataset-high-quality/). arXiv preprint arXiv:2310.06786, 2023.
47. Lawrence C. Paulson. Isabelle: The next 700 theorem provers, 2000. [find](https://scholar.google.com/scholar?q=Lawrence+C%2E+Paulson%2E+Isabelle%3A+The+next+700+theorem+provers%2C+2000%2E)
48. Stanislas Polu, Jesse Michael Han, Kunhao Zheng, Mantas Baksys, Igor Babuschkin, and Ilya Sutskever. [Formal mathematics statement curriculum learning](https://md.research.elamur.ai/free/en/papers/polu-2022-formal-mathematics-statement-curriculum-learning/), 2022.
49. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Fai… — [Code Llama: Open Foundation Models for Code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
50. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/), 2024.
51. Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Je… — [Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models](https://md.research.elamur.ai/free/en/papers/singh-2023-human-data-scaling-self-training/)
52. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. [Galactica: A large language model for science](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/). 2023.
53. The Coq Development Team. The Coq reference manual – release 8.18.0. [https://coq.inria.fr/doc/V8.18.0/refman](https://coq.inria.fr/doc/V8.18.0/refman), 2023. [↗](https://coq.inria.fr/doc/V8.18.0/refman)
54. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
55. Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. [Solving math word problems with process- and outcome-based feedback](https://md.research.elamur.ai/free/en/papers/uesato-2022-solving-math-word-problems-process/), 2022.
56. Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. [Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning](https://md.research.elamur.ai/free/en/papers/wang-2023-mathcoder-seamless-code-integration-llms/), 2023a.
57. Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. [Math-shepherd: Verify and reinforce llms step-by-step without human annotations](https://md.research.elamur.ai/free/en/papers/wang-2024-math-shepherd-verify-reinforce-llms/). CoRR, abs/2312.08935, 2023b.
58. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations, 2023c. URL [https://openreview.net/forum?id…](https://openreview.net/forum?id…) [↗](https://openreview.net/forum?id…)
59. Yan Wang, Xiaojiang Liu, and Shuming Shi. Deep neural solver for math word problems. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 845–854, Copenhagen, Denmark, September 2017. Association for C… [find](https://scholar.google.com/scholar?q=Yan+Wang%2C+Xiaojiang+Liu%2C+and+Shuming+Shi%2E+Deep+neural+solver+for+math+word+problems%2E+In+Martha+Palmer%2C+Rebecca+Hwa%2C+and+Sebastian+Riedel+%28eds%2E%29%2C+Proceedings+of+the+2017+Conference+on+Empirical+Methods)
60. Zengzhi Wang, Rui Xia, and Liu Pengfei. Generative ai for math: Part i – mathpile: A billion-token-scale pretraining corpus for math. arXiv preprint arXiv:2312.17120, 2023d. [find](https://scholar.google.com/scholar?q=Zengzhi+Wang%2C+Rui+Xia%2C+and+Liu+Pengfei%2E+Generative+ai+for+math%3A+Part+i+%E2%80%93+mathpile%3A+A+billion%2Dtoken%2Dscale+pretraining+corpus+for+math%2E+arXiv+preprint+arXiv%3A2312%2E17120%2C+2023d%2E)
61. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
62. Sean Welleck and Rahul Saha. Llmstep: Llm proofstep suggestions in lean. arXiv preprint arXiv:2310.18457, 2023. [find](https://scholar.google.com/scholar?q=Sean+Welleck+and+Rahul+Saha%2E+Llmstep%3A+Llm+proofstep+suggestions+in+lean%2E+arXiv+preprint+arXiv%3A2310%2E18457%2C+2023%2E)
63. Kaiyu Yang, Aidan M Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar. [Leandojo: Theorem proving with retrieval-augmented language models](https://md.research.elamur.ai/free/en/papers/yang-2023-leandojo-theorem-proving-retrieval-augmented/). arXiv preprint arXiv:2306.15626, 2023a.
64. Zhen Yang, Ming Ding, Qingsong Lv, Zhihuan Jiang, Zehai He, Yuyi Guo, Jinfeng Bai, and Jie Tang. Gpt can solve mathematical problems without a calculator. arXiv preprint arXiv:2309.03241, 2023b. [find](https://scholar.google.com/scholar?q=Zhen+Yang%2C+Ming+Ding%2C+Qingsong+Lv%2C+Zhihuan+Jiang%2C+Zehai+He%2C+Yuyi+Guo%2C+Jinfeng+Bai%2C+and+Jie+Tang%2E+Gpt+can+solve+mathematical+problems+without+a+calculator%2E+arXiv+preprint+arXiv%3A2309%2E03241%2C+2023b%2E)
65. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate problem solving with large language models, 2023. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L%2E+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+Tree+of+Thoughts%3A+Deliberate+problem+solving+with+large+language+models%2C+2023%2E)
66. Fei Yu, Anningzhe Gao, and Benyou Wang. Outcome-supervised verifiers for planning in mathematical reasoning, 2023a. [find](https://scholar.google.com/scholar?q=Fei+Yu%2C+Anningzhe+Gao%2C+and+Benyou+Wang%2E+Outcome%2Dsupervised+verifiers+for+planning+in+mathematical+reasoning%2C+2023a%2E)
67. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/), 2023b.
68. Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. [Advancing llm reasoning generalists with preference trees](https://md.research.elamur.ai/free/en/papers/yuan-2024-advancing-llm-reasoning-generalists-preference/), 2024.
69. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. [Scaling relationship on learning mathematical reasoning with large language models](https://md.research.elamur.ai/free/en/papers/yuan-2023-scaling-relationship-learning-mathematical-reasoning/), 2023.
70. Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. [MAmmoTH: Building math generalist models through hybrid instruction tuning](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/). arXiv preprint arXiv:2309.05653, 2023.
71. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. [STar: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL [https://openreview.net/forum?id=](https://openreview.net/forum?id=) 3ELRdg2sgI. [↗](https://openreview.net/forum?id=)
72. Chuanyang Zheng, Haiming Wang, Enze Xie, Zhengying Liu, Jiankai Sun, Huajian Xin, Jianhao Shen, Zhenguo Li, and Yu Li. Lyra: Orchestrating dual correction in automated theorem proving, 2023. [find](https://scholar.google.com/scholar?q=Chuanyang+Zheng%2C+Haiming+Wang%2C+Enze+Xie%2C+Zhengying+Liu%2C+Jiankai+Sun%2C+Huajian+Xin%2C+Jianhao+Shen%2C+Zhenguo+Li%2C+and+Yu+Li%2E+Lyra%3A+Orchestrating+dual+correction+in+automated+theorem+proving%2C+2023%2E)
73. Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. [Minif2f: a cross-system benchmark for formal olympiad-level mathematics](https://md.research.elamur.ai/free/en/papers/zheng-2021-minif2f-cross-system-benchmark-formal/). arXiv preprint arXiv:2109.00110, 2021.
74. Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, and Hongsheng Li. [Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification](https://md.research.elamur.ai/free/en/papers/zhou-2023-solving-challenging-math-word-problems/), 2023.
75. Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang. [Solving math word problems via cooperative reasoning induced language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-solving-math-word-problems-cooperative/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),…

*Updated 2 November 2026*