---
title: "PyraMathBench — paper by Ouyang et al. (2026)"
description: "To address these gaps, the authors propose the SOLVE module and the IRPO reinforcement-learning algorithm. 5.0 points. 55 references."
authors: ["Zetian Ouyang", "Linlin Wang", "Gerard de Melo", "Liang He"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1869/"
doi: "10.18653/v1/2026.findings-acl.1869"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ouyang-2026-pyramathbench-evaluating-improving-mathematical-capability/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ouyang-2026-pyramathbench-evaluating-improving-mathematical-capability/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/40/) / PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models

# PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models

**Authors:** Zetian Ouyang, [Linlin Wang](https://md.research.elamur.ai/free/en/authors/linlin-wang/), [Gerard de Melo](https://md.research.elamur.ai/free/en/authors/gerard-de-melo/) and [Liang He](https://md.research.elamur.ai/free/en/authors/liang-he/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1869/) · [PDF](https://aclanthology.org/2026.findings-acl.1869.pdf) · [DOI 10.18653/v1/2026.findings-acl.1869](https://doi.org/10.18653/v1/2026.findings-acl.1869) · [OpenAlex](https://openalex.org/W7166845561)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Most math benchmarks for LLMs only check the final answer. When a model fails, they cannot show whether the cause was bad arithmetic, misreading numbers, or flawed reasoning. PyraMathBench (PMB) addresses this by organizing math skills as a pyramid. Real word problems and visual reasoning problems are broken into simpler subtasks, which can be scored separately and together. This shows how weak basic skills affect performance on harder problems. Six annotators with high-school-level math expertise built the subtasks from six existing datasets.

Testing 11 LLMs and MLLMs, the authors found recurring weaknesses in:
- abstracting a problem into mathematical form,
- solving equations,
- recalling facts,
- extracting information from images.

To address these gaps, the authors propose the SOLVE module and the IRPO reinforcement-learning algorithm. SOLVE makes tool calling more robust through fuzzy matching and by rejecting low-quality calls. IRPO optimizes multi-round tool use. Together they raised Qwen-2.5 by about 5.0 points. The benchmark is useful for diagnosing where math failures come from, not just how often they occur.

## Key points
- **Size and structure:** PMB has 32,505 questions drawn from 7,404 math word problems. They cover 2 modalities and 14 subtasks grouped into four levels: Complex Reasoning, Understanding, Calculation and Numerical Parsing.
- **Data sources:** ASDiv, alg514, Dolphin 18K, SVAMP, TAT-QA and MathVista. Tables from TAT-QA were turned into images to add multimodal items.
- **Quality control:** Questions were dropped if cross-annotator validation scored below 90. Floating-point answers are rounded to six decimals.
- **Models evaluated:** GPT-4o, GPT-4o-mini, Claude-opus-4.6, LLaVA 13B, DeepSeek-R1, DeepSeek-Math 7B, Qwen-2.5 14B, Qwen-2.5-Math 7B, Llama3.1 8B, Gemma2 9B and Mistral 7B. Each question was sampled three times, and the median score was reported.
- **Limitations:** The benchmark is English-only. Subtask decomposition is not unique. SOLVE and IRPO may help less outside word-problem-style tasks, such as proofs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Potsdam](https://md.research.elamur.ai/free/en/universities/university-of-potsdam/) — University, Potsdam, Germany · 44 works · 5,305 citations
- [East China Normal University](https://md.research.elamur.ai/free/en/universities/east-china-normal-university/) — University, Shanghai, China · 50 works · 2,174 citations
- [Hasso Plattner Institute](https://md.research.elamur.ai/free/en/labs/hasso-plattner-institute/) — Research institute, Potsdam, Germany · 12 works · 4,998 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [➗ Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/) · 177 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 252 works · [➗ Applied Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/applied-mathematics/) · 46 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Zetian Ouyang, Linlin Wang, Gerard de Melo and Liang He. 2026. PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ouyang2026pyramathbench,
  title = {{PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models}},
  author = {Zetian Ouyang, Linlin Wang, Gerard {de Melo} and Liang He},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1869},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1869}
}
```

- [← previous Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering](https://md.research.elamur.ai/free/en/papers/wang-2026-putting-captions-test-evaluating-video/)
- [next → Q2EI: Query-to-Entity Inference for Semantic Condensation in Domain-Specific Retrieval](https://md.research.elamur.ai/free/en/papers/sun-2026-q2ei-query-entity-inference-semantic/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/40/)

## References

55 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mubashara Akhtar, Abhilash Shankarampeta, Vivek Gupta, Arpit Patil, Oana Cocarascu, and Elena Simperl. 2023. [Exploring the numerical reasoning capabilities of language models: A comprehensive analysis on tabular data](https://md.research.elamur.ai/free/en/papers/akhtar-2023-exploring-numerical-reasoning-capabilities-language/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15…
2. Li Cai, Kilchan Choi, Mark Hansen, and Lauren Harrell. 2016. Item response theory. Annual Review of Statistics and Its Application, 3(1):297–321. [find](https://scholar.google.com/scholar?q=Li+Cai%2C+Kilchan+Choi%2C+Mark+Hansen%2C+and+Lauren+Harrell%2E+2016%2E+Item+response+theory%2E+Annual+Review+of+Statistics+and+Its+Application%2C+3%281%29%3A297%E2%80%93321%2E)
3. A universal olympiad level mathematic benchmark for large language models. In The Thirteenth International Conference on Learning Representations. Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de O… [find](https://scholar.google.com/scholar?q=A+universal+olympiad+level+mathematic+benchmark+for+large+language+models%2E+In+The+Thirteenth+International+Conference+on+Learning+Representations%2E+Elliot+Glazer%2C+Ege+Erdil%2C+Tamay+Besiroglu%2C+Diego+Chic)
4. Anna Goldie, Azalia Mirhoseini, Hao Zhou, Irene Cai, and Christopher D Manning. 2025. Synthetic data generation & multi-step rl for reasoning & tool use. arXiv preprint arXiv:2504.04736. [find](https://scholar.google.com/scholar?q=Anna+Goldie%2C+Azalia+Mirhoseini%2C+Hao+Zhou%2C+Irene+Cai%2C+and+Christopher+D+Manning%2E+2025%2E+Synthetic+data+generation+%26+multi%2Dstep+rl+for+reasoning+%26+tool+use%2E+arXiv+preprint+arXiv%3A2504%2E04736%2E)
5. Aaron Grattafiori, Abhimanyu Dubey, and Abhinav Jauhri et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). Preprint, arXiv:2407.21783.
6. Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. 2023. [Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings](https://md.research.elamur.ai/free/en/papers/hao-2023-toolkengpt-augmenting-frozen-language-models/). Advances in neural information processing systems, 36:45870–45894.
7. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, and 1 others. 2024. [OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems](https://md.research.elamur.ai/free/en/papers/he-2024-olympiadbench-challenging-benchmark-promoting-agi/). In Proceedings of the 62nd Annual Me…
8. Anthony Chen, Panupong Pasupat, Sameer Singh, Hongrae Lee, and Kelvin Guu. 2023. [Purr: Efficiently editing language model hallucinations by denoising language model corruptions](https://md.research.elamur.ai/free/en/papers/chen-2023-purr-efficiently-editing-language-model/). arXiv preprint arXiv:2305.14908.
9. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the MATH dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks.
10. Chung-Chi Chen, Hen-Hsen Huang, Hiroya Takamura, and Hsin-Hsi Chen. 2019. Numeracy-600k: Learning numeracy for detecting exaggerated information in market comments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6307–6313. [find](https://scholar.google.com/scholar?q=Chung%2DChi+Chen%2C+Hen%2DHsen+Huang%2C+Hiroya+Takamura%2C+and+Hsin%2DHsi+Chen%2E+2019%2E+Numeracy%2D600k%3A+Learning+numeracy+for+detecting+exaggerated+information+in+market+comments%2E+In+Proceedings+of+the+57th+Annual+M)
11. Cheng-Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister. 2023. Tool documentation enables zero-shot tool-usage with large language models. arXiv preprint arXiv:2308.00675. [find](https://scholar.google.com/scholar?q=Cheng%2DYu+Hsieh%2C+Si%2DAn+Chen%2C+Chun%2DLiang+Li%2C+Yasuhisa+Fujii%2C+Alexander+Ratner%2C+Chen%2DYu+Lee%2C+Ranjay+Krishna%2C+and+Tomas+Pfister%2E+2023%2E+Tool+documentation+enables+zero%2Dshot+tool%2Dusage+with+large+language+m)
12. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
13. DeepSeek-AI, Daya Guo, and Dejian Yang et al. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). Preprint, arXiv:2501.12948.
14. Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, and 1 others. 2025. [find](https://scholar.google.com/scholar?q=Bofei+Gao%2C+Feifan+Song%2C+Zhe+Yang%2C+Zefan+Cai%2C+Yibo+Miao%2C+Qingxiu+Dong%2C+Lei+Li%2C+Chenghao+Ma%2C+Liang+Chen%2C+Runxin+Xu%2C+and+1+others%2E+2025%2E)
15. Omni-MATH: Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. [Survey of hallucination in natural language generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/). ACM Computing Surveys, 55(12):1–38.
16. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, and 1 others. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+and+1+others%2E+202)
17. Chengyue Jiang, Zhonglin Nian, Kaihao Guo, Shanbo Chu, Yinggong Zhao, Libin Shen, and Kewei Tu. 2020. Learning numeral embedding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2586–2599, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chengyue+Jiang%2C+Zhonglin+Nian%2C+Kaihao+Guo%2C+Shanbo+Chu%2C+Yinggong+Zhao%2C+Libin+Shen%2C+and+Kewei+Tu%2E+2020%2E+Learning+numeral+embedding%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+20)
18. Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014. Learning to automatically solve algebra word problems. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 271–281, Baltimore, Maryland. Association for Comput… [find](https://scholar.google.com/scholar?q=Nate+Kushman%2C+Yoav+Artzi%2C+Luke+Zettlemoyer%2C+and+Regina+Barzilay%2E+2014%2E+Learning+to+automatically+solve+algebra+word+problems%2E+In+Proceedings+of+the+52nd+Annual+Meeting+of+the+Association+for+Computati)
19. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/).
20. Tiedong Liu and Bryan Kian Hsiang Low. 2023. GOAT: Fine-tuned LLaMA outperforms GPT-4 on arithmetic tasks. arXiv preprint arXiv:2305.14201. [find](https://scholar.google.com/scholar?q=Tiedong+Liu+and+Bryan+Kian+Hsiang+Low%2E+2023%2E+GOAT%3A+Fine%2Dtuned+LLaMA+outperforms+GPT%2D4+on+arithmetic+tasks%2E+arXiv+preprint+arXiv%3A2305%2E14201%2E)
21. Zuyan Liu, Yuhao Dong, Yongming Rao, Jie Zhou, and Jiwen Lu. 2024. Chain-of-spot: Interactive reasoning improves large vision-language models. Preprint, arXiv:2403.12966. — [arXiv:2403.12966](https://arxiv.org/abs/2403.12966) [find](https://scholar.google.com/scholar?q=Zuyan+Liu%2C+Yuhao+Dong%2C+Yongming+Rao%2C+Jie+Zhou%2C+and+Jiwen+Lu%2E+2024%2E+Chain%2Dof%2Dspot%3A+Interactive+reasoning+improves+large+vision%2Dlanguage+models%2E+Preprint%2C+arXiv%3A2403%2E12966%2E+%E2%80%94+%5BarXiv%3A2403%2E12966%5D%28https%3A%2F%2F)
22. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. [MathVista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). In The Twelfth International Conference on Learning Representations.
23. Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. [Chameleon: Plug-and-play compositional reasoning with large language models](https://md.research.elamur.ai/free/en/papers/lu-2023-chameleon-plug-play-compositional-reasoning/). Advances in Neural Information Processing Systems, 36:43447–43478.
24. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. [Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/). arXiv preprint arXiv:2308.09583.
25. Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th annual meeting of the Association for Computational Linguistics, pages 975–984. [find](https://scholar.google.com/scholar?q=Shen%2DYun+Miao%2C+Chao%2DChun+Liang%2C+and+Keh%2DYih+Su%2E+2020%2E+A+diverse+corpus+for+evaluating+and+developing+english+math+word+problem+solvers%2E+In+Proceedings+of+the+58th+annual+meeting+of+the+Association+for)
26. Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. 2022. [LILA: A unified benchmark for mathematical reasoning](https://md.research.elamur.ai/free/en/papers/mishra-2022-lila-unified-benchmark-mathematical-reasoning/). In Proceedings of the 2022 Conference on Empirical Methods in Natural…
27. Aaron Parisi, Yao Zhao, and Noah Fiedel. 2022. [Talm: Tool augmented language models](https://md.research.elamur.ai/free/en/papers/parisi-2022-talm-tool-augmented-language-models/). arXiv preprint arXiv:2205.12255.
28. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. [Are NLP models really able to solve simple math word problems?](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/) In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, Online. Associat…
29. Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2024. [Gorilla: Large language model connected with massive apis](https://md.research.elamur.ai/free/en/papers/patil-2023-gorilla-large-language-model-connected/). Advances in Neural Information Processing Systems, 37:126544–126565.
30. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, and 1 others. 2024. [ToolLLM: Facilitating large language models to master 16000+ real-world APIs](https://md.research.elamur.ai/free/en/papers/qin-2023-toolllm-facilitating-large-language-models/). In The Twelfth International Conference on Learning Representations.
31. Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J Nay, Kshitij Gupta, and Aran Komatsuzaki. 2023. Arb: Advanced reasoning benchmark for large language models. arXiv preprint arXiv:2307.13692. [find](https://scholar.google.com/scholar?q=Tomohiro+Sawada%2C+Daniel+Paleka%2C+Alexander+Havrilla%2C+Pranav+Tadepalli%2C+Paula+Vidas%2C+Alexander+Kranias%2C+John+J+Nay%2C+Kshitij+Gupta%2C+and+Aran+Komatsuzaki%2E+2023%2E+Arb%3A+Advanced+reasoning+benchmark+for+large)
32. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. [Toolformer: Language models can teach themselves to use tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/). Advances in Neural Information Processing Systems, 36:68539–68551.
33. John Schulman. 2020. Approximating kl divergence. John Schulman’s Homepage. [find](https://scholar.google.com/scholar?q=John+Schulman%2E+2020%2E+Approximating+kl+divergence%2E+John+Schulman%E2%80%99s+Homepage%2E)
34. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, and 1 others. 2025. DeepSeek-Math: Pushing the limits of mathematical reasoning in open language models. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhihong+Shao%2C+Peiyi+Wang%2C+Qihao+Zhu%2C+Runxin+Xu%2C+Junxiao+Song%2C+Xiao+Bi%2C+Haowei+Zhang%2C+Mingchuan+Zhang%2C+YK+Li%2C+Y+Wu%2C+and+1+others%2E+2025%2E+DeepSeek%2DMath%3A+Pushing+the+limits+of+mathematical+reasoning+in+op)
35. Jia Tracy Shen, Michiharu Yamashita, Ethan Prihar, Neil Heffernan, Xintao Wu, Ben Graff, and Dongwon Lee. 2021. Mathbert: A pre-trained language model for general nlp tasks in mathematics education. arXiv preprint arXiv:2106.07340. — [arXiv:2106.07340](https://arxiv.org/abs/2106.07340) [find](https://scholar.google.com/scholar?q=Jia+Tracy+Shen%2C+Michiharu+Yamashita%2C+Ethan+Prihar%2C+Neil+Heffernan%2C+Xintao+Wu%2C+Ben+Graff%2C+and+Dongwon+Lee%2E+2021%2E+Mathbert%3A+A+pre%2Dtrained+language+model+for+general+nlp+tasks+in+mathematics+education%2E+a)
36. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. [Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face](https://md.research.elamur.ai/free/en/papers/shen-2023-hugginggpt-solving-ai-tasks-chatgpt/). Advances in Neural Information Processing Systems, 36:38154–38180.
37. Shuming Shi, Yuehui Wang, Chin-Yew Lin, Xiaojiang Liu, and Yong Rui. 2015. Automatically solving number word problems by semantic parsing and reasoning. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1132–1142, Lisbon, Portugal. Association for Compu… [find](https://scholar.google.com/scholar?q=Shuming+Shi%2C+Yuehui+Wang%2C+Chin%2DYew+Lin%2C+Xiaojiang+Liu%2C+and+Yong+Rui%2E+2015%2E+Automatically+solving+number+word+problems+by+semantic+parsing+and+reasoning%2E+In+Proceedings+of+the+2015+Conference+on+Empiri)
38. Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang Huang, Cheng Li, Ke Wang, Rong Yao, and 1 others. 2023. [Restgpt: Connecting large language models with real-world restful apis](https://md.research.elamur.ai/free/en/papers/song-2023-restgpt-connecting-large-language-models/). arXiv preprint arXiv:2306.06624.
39. Georgios Spithourakis and Sebastian Riedel. 2018. Numeracy for language models: Evaluating and improving their ability to predict numbers. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2104–2115, Melbourne, Australia. Associ… [find](https://scholar.google.com/scholar?q=Georgios+Spithourakis+and+Sebastian+Riedel%2E+2018%2E+Numeracy+for+language+models%3A+Evaluating+and+improving+their+ability+to+predict+numbers%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+)
40. Dhanasekar Sundararaman, Shijing Si, Vivek Subramanian, Guoyin Wang, Devamanyu Hazarika, and Lawrence Carin. 2020. Methods for numeracy-preserving word embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4742–4753. [find](https://scholar.google.com/scholar?q=Dhanasekar+Sundararaman%2C+Shijing+Si%2C+Vivek+Subramanian%2C+Guoyin+Wang%2C+Devamanyu+Hazarika%2C+and+Lawrence+Carin%2E+2020%2E+Methods+for+numeracy%2Dpreserving+word+embeddings%2E+In+Proceedings+of+the+2020+Conferenc)
41. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Morgane+Riviere%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+L%C3%A9onard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Ram%C3%A9%2C+and+1+others%2E+2024%2E+Gemma+2%3A+Impr)
42. George Tsoukalas, Jasper Lee, John Jennings, Jimmy Xin, Michelle Ding, Michael Jennings, Amitayush Thakur, and Swarat Chaudhuri. 2024. PutnamBench: Evaluating neural theorem-provers on the putnam mathematical competition. In Advances in Neural Information Processing Systems, volume 37. [find](https://scholar.google.com/scholar?q=George+Tsoukalas%2C+Jasper+Lee%2C+John+Jennings%2C+Jimmy+Xin%2C+Michelle+Ding%2C+Michael+Jennings%2C+Amitayush+Thakur%2C+and+Swarat+Chaudhuri%2E+2024%2E+PutnamBench%3A+Evaluating+neural+theorem%2Dprovers+on+the+putnam+math)
43. Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024. [Math-Shepherd: Verify and reinforce LLMs step-by-step without human annotations](https://md.research.elamur.ai/free/en/papers/wang-2024-math-shepherd-verify-reinforce-llms/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
44. Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 845–854. [find](https://scholar.google.com/scholar?q=Yan+Wang%2C+Xiaojiang+Liu%2C+and+Shuming+Shi%2E+2017%2E+Deep+neural+solver+for+math+word+problems%2E+In+Proceedings+of+the+2017+conference+on+empirical+methods+in+natural+language+processing%2C+pages+845%E2%80%93854%2E)
45. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824– 24837.
46. Jialiang Xu, Mengyu Zhou, Xinyi He, Shi Han, and Dongmei Zhang. 2022. Towards robust numerical question answering: Diagnosing numerical capabilities of NLP systems. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Jialiang+Xu%2C+Mengyu+Zhou%2C+Xinyi+He%2C+Shi+Han%2C+and+Dongmei+Zhang%2E+2022%2E+Towards+robust+numerical+question+answering%3A+Diagnosing+numerical+capabilities+of+NLP+systems%2E+In+Proceedings+of+the+2022+Conferen)
47. Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. 2023. [On the tool manipulation capability of open-source large language models](https://md.research.elamur.ai/free/en/papers/xu-2023-tool-manipulation-capability-open-source/). arXiv preprint arXiv:2305.16504.
48. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 22 others. 2024a. Qwen2.5 technical report. arXiv preprint arX… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+Yang%2C+Jin)
49. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, and 1 others. 2024b. [Qwen2.5-Math technical report: Toward mathematical expert model via self-improvement](https://md.research.elamur.ai/free/en/papers/yang-2024-qwen2-5-math-technical-report/). In Advances in Neural Information Processing Systems, volume 37.
50. Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Ren Kan, Dongsheng Li, and Deqing Yang. 2025. EasyTool: Enhancing LLM-based agents with concise tool instruction. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics: Human… [find](https://scholar.google.com/scholar?q=Siyu+Yuan%2C+Kaitao+Song%2C+Jiangjie+Chen%2C+Xu+Tan%2C+Yongliang+Shen%2C+Ren+Kan%2C+Dongsheng+Li%2C+and+Deqing+Yang%2E+2025%2E+EasyTool%3A+Enhancing+LLM%2Dbased+agents+with+concise+tool+instruction%2E+In+Proceedings+of+the+2)
51. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. 2023. How well do large language models perform in arithmetic tasks? arXiv preprint arXiv:2304.02015. [find](https://scholar.google.com/scholar?q=Zheng+Yuan%2C+Hongyi+Yuan%2C+Chuanqi+Tan%2C+Wei+Wang%2C+and+Songfang+Huang%2E+2023%2E+How+well+do+large+language+models+perform+in+arithmetic+tasks%3F+arXiv+preprint+arXiv%3A2304%2E02015%2E)
52. Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, and 1 others. 2024. [Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?](https://md.research.elamur.ai/free/en/papers/zhang-2024-mathverse-multi-modal-llm-truly/) In European Conference on Computer Vision, pages 169–186. Springer.
53. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, and 1 others. 2023. [A survey of large language models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/). arXiv preprint arXiv:2303.18223.
54. Wei Zhao, Mingyue Shang, Yang Liu, Liang Wang, and Jingming Liu. 2020. Ape210k: A large-scale and template-rich dataset of math word problems. arXiv preprint arXiv:2009.11506. [find](https://scholar.google.com/scholar?q=Wei+Zhao%2C+Mingyue+Shang%2C+Yang+Liu%2C+Liang+Wang%2C+and+Jingming+Liu%2E+2020%2E+Ape210k%3A+A+large%2Dscale+and+template%2Drich+dataset+of+math+word+problems%2E+arXiv+preprint+arXiv%3A2009%2E11506%2E)
55. Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. [TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance](https://md.research.elamur.ai/free/en/papers/zhu-2021-tat-qa-question-answering-benchmark/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics…

*Updated 1 November 2026*