---
title: "MR-GSM8K — paper by Zeng et al. (2023) — summary and ideas"
description: "The authors propose testing models as graders instead of solvers, a setup they call… 80% accuracy. 40 references, cited by 35 papers here."
authors: ["Zhongshen Zeng", "Pengguang Chen", "Shu Liu", "Haiyun Jiang", "Jiaya Jia"]
year: 2023
venue: "arXiv:2312.17080"
source: "https://arxiv.org/abs/2312.17080"
doi: "10.48550/arxiv.2312.17080"
arxiv: "2312.17080"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zeng-2023-mr-gsm8k-meta-reasoning-benchmark/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zeng-2023-mr-gsm8k-meta-reasoning-benchmark/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/) / MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation

# MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation

**Authors:** Zhongshen Zeng, Pengguang Chen, [Shu Liu](https://md.research.elamur.ai/free/en/authors/shu-liu/), [Haiyun Jiang](https://md.research.elamur.ai/free/en/authors/haiyun-jiang/) and [Jiaya Jia](https://md.research.elamur.ai/free/en/authors/jiaya-jia/) · **Year:** 2023 · **Venue:** arXiv:2312.17080 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2312.17080) · [PDF](https://arxiv.org/pdf/2312.17080) · [DOI 10.48550/arxiv.2312.17080](https://doi.org/10.48550/arxiv.2312.17080) · [arXiv 2312.17080](https://arxiv.org/abs/2312.17080) · [OpenAlex](https://openalex.org/W4390437991)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [35](https://md.research.elamur.ai/free/en/papers/zeng-2023-mr-gsm8k-meta-reasoning-benchmark/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Math word-problem benchmarks such as GSM8K are close to saturation. Leading LLMs exceed 80% accuracy, scoring checks only the final answer, and test-set contamination may inflate results. The authors propose testing models as graders instead of solvers, a setup they call meta-reasoning. They built MR-GSM8K from three GSM8K-derived question variants: the original problems, code-oriented "Program of Thought" versions, and reversed-reasoning versions in which one input is hidden. Step-by-step solutions were sampled from MetaMath-7B so that roughly half are correct. Expert annotators then recorded three things for each solution: whether its reasoning is genuinely sound, where the first faulty step occurs, and why it is wrong. Evaluated models must reproduce these judgments, which are combined into a new metric, MR-Score. Open and commercial models that look alike on GSM8K diverge sharply on MR-GSM8K, revealing weaknesses that answer-only tests miss. The recipe can be applied to other existing benchmarks, giving harder evaluation of the reasoning process that is more resistant to contamination.

## Key points
- Models receive a question-solution pair and must judge correctness, locate the first error step, and explain the error. These outputs feed the MR-Score metric.
- Deepseek-v2 and Claude3-Sonnet nearly match GPT-4 on GSM8K, yet trail it by more than 20 absolute points on MR-GSM8K. Across all tested models, the gaps reach more than tenfold.
- Math-specialized fine-tuned models (e.g., MetaMath, WizardMath) generalize poorly to the grading task, whether given instructions or few-shot examples. The small Phi-3-Mini outperforms models about twenty times its size.
- The three-shot prompts contained two correct solutions and one incorrect solution. This setting raised true positive rates and lowered true negative rates, so models appear swayed by the label mix in the examples rather than by reasoning. An ablation on the number of correct examples supports this explanation.
- Case studies expose several recurring failures: the reversal curse, confused handling of quantity units (weak ontological understanding), and insensitivity to arithmetic mistakes.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tencent](https://md.research.elamur.ai/free/en/companies/tencent/) — Company, Shenzhen, China · 174 works · 21,355 citations
- [Chinese University of Hong Kong](https://md.research.elamur.ai/free/en/universities/chinese-university-of-hong-kong/) — University, Hong Kong, China · 176 works · 17,536 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,621 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,948 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,394 works

## Cite

Cite as: Zhongshen Zeng, Pengguang Chen, Shu Liu, Haiyun Jiang and Jiaya Jia. 2023. MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation. arXiv:2312.17080.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zeng2023mr,
  title = {{MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation}},
  author = {Zhongshen Zeng, Pengguang Chen, Shu Liu, Haiyun Jiang and Jiaya Jia},
  year = {2023},
  doi = {10.48550/arxiv.2312.17080},
  eprint = {2312.17080},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.17080}
}
```

- [← previous Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-teacher-coach-measuring/)
- [next → The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASR](https://md.research.elamur.ai/free/en/papers/sanabria-2023-edinburgh-international-accents-english-corpus/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/)

## References

40 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. Introducing claude, 2023. introducing-claude. [find](https://scholar.google.com/scholar?q=Anthropic%2E+Introducing+claude%2C+2023%2E+introducing%2Dclaude%2E)
2. Konstantine Arkoudas. GPT-4 can’t reason. CoRR, abs/2308.[03762, 2023](https://md.research.elamur.ai/free/en/papers/arkoudas-2023-gpt-4-t-reason/). doi: 10.48550/ARXIV.2308.03762. URL https://doi.org/10.48550/arXiv.2308.03762. [↗](https://doi.org/10.48550/arXiv.2308.03762)
3. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… — [Qwen Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
4. Simone Balloccu, Patr’icia Schmidtov’a, Mateusz Lango, and Ondvrej Duvsek. [Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source llms](https://md.research.elamur.ai/free/en/papers/balloccu-2024-leak-cheat-repeat-data-contamination/). In Annual Meeting of the Association for Computational Linguistics, 2024. URL [https://api.semanticscholar.org/CorpusID:267499939](https://api.semanticscholar.org/CorpusID:267499939). [↗](https://api.semanticscholar.org/CorpusID:267499939)
5. Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on "a is b" fail to learn "b is a". ArXiv, abs/2309.12288, 2023. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 262083829. [↗](https://api.semanticscholar.org/CorpusID:)
6. Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on "a is b" fail to learn "b is a". CoRR, abs/2309.12288, 2023. doi: 10.48550/ARXIV.2309.12288. URL https://doi.org/10.48550/arXiv.2309.12288. [↗](https://doi.org/10.48550/arXiv.2309.12288)
7. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li,… [find](https://scholar.google.com/scholar?q=Xiao+Bi%2C+Deli+Chen%2C+Guanting+Chen%2C+Shanhuang+Chen%2C+Damai+Dai%2C+Chengqi+Deng%2C+Honghui+Ding%2C+Kai+Dong%2C+Qiushi+Du%2C+Zhe+Fu%2C+Huazuo+Gao%2C+Kaige+Gao%2C+Wenjun+Gao%2C+Ruiqi+Ge%2C+Kang+Guan%2C+Daya+Guo%2C+Jianzhong+Guo%2C+)
8. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, John A. Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuan-Fang Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). ArXiv, abs/23…
9. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. ArXiv, abs/2211.12588, 2022. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 253801709. [↗](https://api.semanticscholar.org/CorpusID:)
10. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/), 2021. URL [https://www.anthropic.com/index/](https://www.anthropic.com/index/) [↗](https://www.anthropic.com/index/)
11. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. [Qlora: Efficient finetuning of quantized llms](https://md.research.elamur.ai/free/en/papers/dettmers-2023-qlora-efficient-finetuning-quantized-llms/). ArXiv, abs/2305.14314, 2023. URL [https://api.semanticscholar.org/CorpusID:258841328](https://api.semanticscholar.org/CorpusID:258841328). [↗](https://api.semanticscholar.org/CorpusID:258841328)
12. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jian, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaïd Harchaoui, and Yejin Choi. [Faith and fate: Limits of transformers on compositionality](https://md.research.elamur.ai/free/en/papers/dziri-2023-faith-fate-limits-transformers-compositionality/).…
13. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. [Tora: A tool-integrated reasoning agent for mathematical problem solving](https://md.research.elamur.ai/free/en/papers/gou-2023-tora-tool-integrated-reasoning-agent/). ArXiv, abs/2309.17452, 2023. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 263310365. [↗](https://api.semanticscholar.org/CorpusID:)
14. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi… [find](https://scholar.google.com/scholar?q=Suriya+Gunasekar%2C+Yi+Zhang%2C+Jyoti+Aneja%2C+Caio+C%C3%A9sar+Teodoro+Mendes%2C+Allie+Del+Giorno%2C+Sivakanth+Gopi%2C+Mojan+Javaheripi%2C+Piero+Kauffmann%2C+Gustavo+de+Rosa%2C+Olli+Saarikivi%2C+Adil+Salim%2C+Shital+Shah%2C+Harki)
15. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/), 2021.
16. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. [Large language models cannot self-correct reasoning yet](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-cannot-self/). ArXiv, abs/2310.01798, 2023. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 263609132. [↗](https://api.semanticscholar.org/CorpusID:)
17. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. Mawps: A math word problem repository. In North American Chapter of the Association for Computational Linguistics, 2016. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 2228719. [↗](https://api.semanticscholar.org/CorpusID:)
18. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. ArXiv, abs/2305.[20050, 2023](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 258987659. [↗](https://api.semanticscholar.org/CorpusID:)
19. Xiao Liu, Xuanyu Lei, Sheng-Ping Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Tam, Xiaohan Zhang, Lichao Sun, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, and Jie Tang. [Alignbench: Benchmarking chinese alignment of large language models](https://md.research.elamur.ai/free/en/papers/liu-2023-alignbench-benchmarking-chinese-alignment/). ArXiv, abs/2311.18743,…
20. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. [Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/). arXiv preprint arXiv:2308.09583, 2023.
21. Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. [Language models of code are few-shot commonsense learners](https://md.research.elamur.ai/free/en/papers/madaan-2022-language-models-code-few-shot/). ArXiv, abs/2210.07128, 2022. URL [https://api.semanticscholar.org/CorpusID:252873120](https://api.semanticscholar.org/CorpusID:252873120). [↗](https://api.semanticscholar.org/CorpusID:252873120)
22. Brian W. Matthews. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et biophysica acta, 405 2:442–51, 1975. URL [https://api.semanticscholar.org/CorpusID:44596673](https://api.semanticscholar.org/CorpusID:44596673). [↗](https://api.semanticscholar.org/CorpusID:44596673)
23. Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. In Annual Meeting of the Association for Computational Linguistics, 2020. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 220047831. [↗](https://api.semanticscholar.org/CorpusID:)
24. OpenAI. Chatgpt: Optimizing language models for dialogue, 2022. URL [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt). [↗](https://openai.com/blog/chatgpt)
25. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). ArXiv, abs/2303.08774, 2023. semanticscholar.org/CorpusID:257532815.
26. Keiran Paster. Testing language models on a held-out high school national finals exam. [https://huggingface.co/datasets/keirp/hungarian_national_hs_finals_exam](https://huggingface.co/datasets/keirp/hungarian_national_hs_finals_exam), 2023. [↗](https://huggingface.co/datasets/keirp/hungarian_national_hs_finals_exam)
27. Stanislas Polu and Ilya Sutskever. [Generative language modeling for automated theorem proving](https://md.research.elamur.ai/free/en/papers/polu-2020-generative-language-modeling-automated-theorem/). ArXiv, abs/2009.03393, 2020. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 221535103. [↗](https://api.semanticscholar.org/CorpusID:)
28. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). CoRR, abs/2402.03300, 2024. doi: 10.48550/ARXIV.2402.03300. URL https://doi.org/10.48550/arXiv.2402.03300. [↗](https://doi.org/10.48550/arXiv.2402.03300)
29. Mike Sharples, David C. Hogg, Chris Hutchinson, Steve Torrance, and David J. Young. Computers and thought: A practical introduction to artificial intelligence. In Proceedings of the Conference on Innovative Applications of Artificial Intelligence, 1989. URL [https://api.semanticscholar.org/CorpusID:6…](https://api.semanticscholar.org/CorpusID:6…) [↗](https://api.semanticscholar.org/CorpusID:6…)
30. Christian Szegedy. A promising path towards autoformalization and general artificial intelligence. In International Conference on Intelligent Computer Mathematics, 2020. URL [https://api.semanticscholar.org/CorpusID:220729524](https://api.semanticscholar.org/CorpusID:220729524). [↗](https://api.semanticscholar.org/CorpusID:220729524)
31. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
32. Vernon Toh, Ratish Puduppully, and Nancy F. Chen. Veritymath: Advancing mathematical reasoning by self-verification through unit consistency. ArXiv, abs/2311.07172, 2023. URL [https://api.semanticscholar.org/CorpusID:265149887](https://api.semanticscholar.org/CorpusID:265149887). — [arXiv:2311.07172](https://arxiv.org/abs/2311.07172) [↗](https://api.semanticscholar.org/CorpusID:265149887)
33. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
34. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.[11903, 2022](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 246411621. [↗](https://api.semanticscholar.org/CorpusID:)
35. Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica. [Rethinking benchmark and contamination for language models with rephrased samples](https://md.research.elamur.ai/free/en/papers/yang-2023-rethinking-benchmark-contamination-language-models/). ArXiv, abs/2311.04850, 2023. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 265050721. [↗](https://api.semanticscholar.org/CorpusID:)
36. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. ArXiv, abs/2305.10601, 2023. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 258762525. [↗](https://api.semanticscholar.org/CorpusID:)
37. An-Zi Yen and Wei-Ling Hsu. Three questions concerning the use of large language models to facilitate mathematics learning. ArXiv, abs/2310.13615, 2023. URL [https://api.semanticscholar.org/CorpusID:264405766](https://api.semanticscholar.org/CorpusID:264405766). [↗](https://api.semanticscholar.org/CorpusID:264405766)
38. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). arXiv preprint arXiv:2309.12284, 2023. URL https://api.
39. Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. [Mammoth: Building math generalist models through hybrid instruction tuning](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/). ArXiv, abs/2309.05653, 2023. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 261696697. [↗](https://api.semanticscholar.org/CorpusID:)
40. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph Gonzalez, and Ion Stoica. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). ArXiv, abs/2306.05685, 2023. URL [https://api.semanticscholar.or…](https://api.semanticscholar.or…) [↗](https://api.semanticscholar.or…)

*Updated 3 November 2026*