---
title: "MuggleMath — paper by Li et al. (2024) — summary and ideas"
description: "Open-source LLMs still trail proprietary systems at math word problems. 75 references, cited by 37 papers here."
authors: ["Chengpeng Li", "Zheng Yuan", "Hongyi Yuan", "Guanting Dong", "Keming Lu", "Jiancan Wu", "Chuanqi Tan", "Xiang Wang", "Chang Zhou"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.551/"
doi: "10.18653/v1/2024.acl-long.551"
arxiv: "2310.05506"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2024-mugglemath-assessing-impact-query-response/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2024-mugglemath-assessing-impact-query-response/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/) / MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning

# MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning

**Authors:** [Chengpeng Li](https://md.research.elamur.ai/free/en/authors/chengpeng-li/), [Zheng Yuan](https://md.research.elamur.ai/free/en/authors/zheng-yuan/), [Hongyi Yuan](https://md.research.elamur.ai/free/en/authors/hongyi-yuan/), [Guanting Dong](https://md.research.elamur.ai/free/en/authors/guanting-dong/), [Keming Lu](https://md.research.elamur.ai/free/en/authors/keming-lu/), [Jiancan Wu](https://md.research.elamur.ai/free/en/authors/jiancan-wu/), [Chuanqi Tan](https://md.research.elamur.ai/free/en/authors/chuanqi-tan/), [Xiang Wang](https://md.research.elamur.ai/free/en/authors/xiang-wang/) and [Chang Zhou](https://md.research.elamur.ai/free/en/authors/chang-zhou/) · **Year:** 2024 · **Venue:** [Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.551/) · [PDF](https://aclanthology.org/2024.acl-long.551.pdf) · [DOI 10.18653/v1/2024.acl-long.551](https://doi.org/10.18653/v1/2024.acl-long.551) · [arXiv 2310.05506](https://arxiv.org/abs/2310.05506) · [OpenAlex](https://openalex.org/W4402671688)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [37](https://md.research.elamur.ai/free/en/papers/li-2024-mugglemath-assessing-impact-query-response/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Open-source LLMs still trail proprietary systems at math word problems. A common remedy is supervised fine-tuning on synthetic data. This paper asks which augmentation strategies matter, how accuracy grows with the amount of synthetic data, and whether gains carry over to other math domains. The authors used gpt-3.5-turbo-0613 and gpt-4-0613 for two kinds of augmentation. First, they rewrote GSM8K and MATH questions in five human-inspired ways, mostly producing variants of equal or higher difficulty. Second, they sampled several solution chains for each new question. This produced two datasets, AugGSM8K and AugMATH. Fully fine-tuning LLaMA and LLaMA-2 models on them gives the MuggleMath family, which reached state-of-the-art results on both benchmarks. On GSM8K, accuracy rises log-linearly with the volume of synthetic data. On MATH, the relationship is piecewise log-linear. However, improvements barely transfer between the two benchmarks. Embedding visualizations show that the augmented questions remain close to their source distribution. For anyone generating synthetic training data, the takeaway is that diversity of the seed questions matters for broad reasoning ability, not just volume.

## Key points
- **Base models and training:** LLaMA-1 7B and LLaMA-2 7B/13B/70B were fully fine-tuned for 3 epochs at learning rate 2e-5 on 32 NVIDIA A100 GPUs. GSM8K provides 7,473 training and 1,319 test problems.
- **Scaling behavior:** On GSM8K, the scaling slope for synthetic data resembles the slope for adding new human-written examples.
- **Which problems to augment:** Augmenting hard problems (more than three reasoning steps) helps more than augmenting easy, medium, or random ones. Augmenting problems the model gets wrong also beats random selection.
- **Gains by difficulty:** For LLaMA-2 7B, test accuracy rose from 0.55/0.42/0.21 to 0.73/0.70/0.64 on easy/medium/hard problems. The largest improvement came on hard items.
- **Cross-domain transfer:** Training on AugGSM8K lifts MATH accuracy only marginally, staying in single digits for 7B models, and the reverse direction is similarly weak. MuggleMath does stay robust on perturbed GSM8K test sets (Change-Test, Aug-Test).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) — Company, Hangzhou, China · 207 works · 17,449 citations
- [University of Science and Technology of China](https://md.research.elamur.ai/free/en/universities/university-of-science-and-technology-of-china/) — University, Hefei, China · 152 works · 114,627 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 496 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,948 works

## Cite

Cite as: Chengpeng Li, Zheng Yuan, Hongyi Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang and Chang Zhou. 2024. MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{li2024mugglemath,
  title = {{MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning}},
  author = {Chengpeng Li, Zheng Yuan, Hongyi Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang and Chang Zhou},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.551},
  eprint = {2310.05506},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.acl-long.551}
}
```

- [← previous MLR-Copilot: Autonomous Machine Learning Research based on Large Language Models Agents](https://md.research.elamur.ai/free/en/papers/li-2024-mlr-copilot-autonomous-machine/)
- [next → Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision](https://md.research.elamur.ai/free/en/papers/wang-2024-multi-step-problem-solving-through/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/)

## References

75 references, 40 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
2. Alibaba. 2023. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609.
3. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403.
4. BaichuanInc. 2023. [Baichuan 2](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). technical report. arXiv preprint arXiv:2309.16609.
5. Tiffany Tianhui Cai, Hongseok Namkoong, and Steve Yadlowsky. 2023. Diagnosing model performance under distribution shift. [find](https://scholar.google.com/scholar?q=Tiffany+Tianhui+Cai%2C+Hongseok+Namkoong%2C+and+Steve+Yadlowsky%2E+2023%2E+Diagnosing+model+performance+under+distribution+shift%2E)
6. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
7. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
8. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Elliot Creager, Jörn-Henrik Jacobsen, and Richard S. Zemel. 2021. Environment inference for invariant learning. pages 2189–2200. PMLR. [find](https://scholar.google.com/scholar?q=Karl+Cobbe%2C+Vineet+Kosaraju%2C+Mohammad+Bavarian%2C+Mark+Chen%2C+Heewoo+Jun%2C+Lukasz+Kaiser%2C+Matthias+Elliot+Creager%2C+J%C3%B6rn%2DHenrik+Jacobsen%2C+and+Richard+S%2E+Zemel%2E+2021%2E+Environment+inference+for+invariant+lea)
9. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. [Enhancing chat language models by scaling high-quality instructional conversations](https://md.research.elamur.ai/free/en/papers/ding-2023-enhancing-chat-language-models-scaling/). arXiv preprint arXiv:2305.14233.
10. Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. 2024. [How abilities in large language models are affected by supervised fine-tuning data composition](https://md.research.elamur.ai/free/en/papers/dong-2024-how-abilities-large-language-models/).
11. Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021. [A survey of data augmentation approaches for NLP](https://md.research.elamur.ai/free/en/papers/feng-2021-survey-data-augmentation-approaches-nlp/). In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 968–988, Online. Association for Computational Lin…
12. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: program-aided language models. In ICML 2023. [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Pengfei+Liu%2C+Yiming+Yang%2C+Jamie+Callan%2C+and+Graham+Neubig%2E+PAL%3A+program%2Daided+language+models%2E+In+ICML+2023%2E)
13. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021a. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874.
14. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. [Measuring mathematical problem solving with the MATH dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/).
15. Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. [Large language models can self-improve](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/).
16. Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018. Progressive growing of gans for improved quality, stability, and variation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. [find](https://scholar.google.com/scholar?q=Tero+Karras%2C+Timo+Aila%2C+Samuli+Laine%2C+and+Jaakko+Lehtinen%2E+2018%2E+Progressive+growing+of+gans+for+improved+quality%2C+stability%2C+and+variation%2E+In+6th+International+Conference+on+Learning+Representations)
17. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. MAWPS: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1152–1157, San D… [find](https://scholar.google.com/scholar?q=Rik+Koncel%2DKedziorski%2C+Subhro+Roy%2C+Aida+Amini%2C+Nate+Kushman%2C+and+Hannaneh+Hajishirzi%2E+2016%2E+MAWPS%3A+A+math+word+problem+repository%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+o)
18. Kun Kuang, Ruoxuan Xiong, Peng Cui, Susan Athey, and Bo Li. 2020. Stable prediction with model misspecification and agnostic distribution shift. pages 4485–4492. AAAI Press. [find](https://scholar.google.com/scholar?q=Kun+Kuang%2C+Ruoxuan+Xiong%2C+Peng+Cui%2C+Susan+Athey%2C+and+Bo+Li%2E+2020%2E+Stable+prediction+with+model+misspecification+and+agnostic+distribution+shift%2E+pages+4485%E2%80%934492%2E+AAAI+Press%2E)
19. Ariel N. Lee, Cole J. Hunter, and Nataniel Ruiz. 2023. [Platypus: Quick, cheap, and powerful refinement of llms](https://md.research.elamur.ai/free/en/papers/lee-2023-platypus-quick-cheap-powerful-refinement/).
20. Shanglin Lei, Guanting Dong, Xiaoping Wang, Keheng Wang, and Sirui Wang. 2024. Instructerc: Reforming emotion recognition in conversation with a retrieval multi-task llms framework. [find](https://scholar.google.com/scholar?q=Shanglin+Lei%2C+Guanting+Dong%2C+Xiaoping+Wang%2C+Keheng+Wang%2C+and+Sirui+Wang%2E+2024%2E+Instructerc%3A+Reforming+emotion+recognition+in+conversation+with+a+retrieval+multi%2Dtask+llms+framework%2E)
21. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022a. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/).
22. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022b. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). In NeurIPS.
23. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023a. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). arXiv preprint arXiv:2305.20050.
24. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023b. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/).
25. Zachary C. Lipton, Yu-Xiang Wang, and Alexander J. Smola. 2018. Detecting and correcting for label shift with black box predictors. pages 3128–3136. PMLR. [find](https://scholar.google.com/scholar?q=Zachary+C%2E+Lipton%2C+Yu%2DXiang+Wang%2C+and+Alexander+J%2E+Smola%2E+2018%2E+Detecting+and+correcting+for+label+shift+with+black+box+predictors%2E+pages+3128%E2%80%933136%2E+PMLR%2E)
26. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023. [The flan collection: Designing data and methods for effective instruction tuning](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/). arXiv preprint arXiv:2301.13688.
27. Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. 2023. [Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning](https://md.research.elamur.ai/free/en/papers/lu-2022-dynamic-prompt-learning-policy-gradient/). In ICLR.
28. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023a. [Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/).
29. Haoran Luo, Haihong E, Zichen Tang, Shiyao Peng, Yikai Guo, Wentai Zhang, Chenghao Ma, Guanting Dong, Meina Song, Wei Lin, Yifan Zhu, and Luu Anh Tuan. 2024. Chatkbqa: A generate-then-retrieve framework for knowledge base question answering with fine-tuned large language models. [find](https://scholar.google.com/scholar?q=Haoran+Luo%2C+Haihong+E%2C+Zichen+Tang%2C+Shiyao+Peng%2C+Yikai+Guo%2C+Wentai+Zhang%2C+Chenghao+Ma%2C+Guanting+Dong%2C+Meina+Song%2C+Wei+Lin%2C+Yifan+Zhu%2C+and+Luu+Anh+Tuan%2E+2024%2E+Chatkbqa%3A+A+generate%2Dthen%2Dretrieve+framewo)
30. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023b. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568. [find](https://scholar.google.com/scholar?q=Ziyang+Luo%2C+Can+Xu%2C+Pu+Zhao%2C+Qingfeng+Sun%2C+Xiubo+Geng%2C+Wenxiang+Hu%2C+Chongyang+Tao%2C+Jing+Ma%2C+Qingwei+Lin%2C+and+Daxin+Jiang%2E+2023b%2E+Wizardcoder%3A+Empowering+code+large+language+models+with+evol%2Dinstruct%2E+)
31. Yuzhou Mao, Liu Yu, Yi Yang, Fan Zhou, and Ting Zhong. 2023. Debiasing intrinsic bias and application bias jointly via invariant risk minimization (student abstract). AAAI Press. [find](https://scholar.google.com/scholar?q=Yuzhou+Mao%2C+Liu+Yu%2C+Yi+Yang%2C+Fan+Zhou%2C+and+Ting+Zhong%2E+2023%2E+Debiasing+intrinsic+bias+and+application+bias+jointly+via+invariant+risk+minimization+%28student+abstract%29%2E+AAAI+Press%2E)
32. Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. A diverse corpus for evaluating and developing english math word problem solvers. In ACL. [find](https://scholar.google.com/scholar?q=Shen%2DYun+Miao%2C+Chao%2DChun+Liang%2C+and+Keh%2DYih+Su%2E+2020%2E+A+diverse+corpus+for+evaluating+and+developing+english+math+word+problem+solvers%2E+In+ACL%2E)
33. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. [Orca: Progressive learning from complex explanation traces of gpt-4](https://md.research.elamur.ai/free/en/papers/mukherjee-2023-orca-progressive-learning-complex-explanation/). arXiv preprint arXiv:2306.02707.
34. Ansong Ni, Jeevana Priya Inala, Chenglong Wang, Alex Polozov, Christopher Meek, Dragomir Radev, and Jianfeng Gao. 2023. [Learning math reasoning from self-sampled correct and partially-correct solutions](https://md.research.elamur.ai/free/en/papers/ni-2022-learning-math-reasoning-self-sampled/). In The Eleventh International Conference on Learning Representations.
35. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
36. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. [Train…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
37. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. [Are NLP models really able to solve simple math word problems?](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/) In NAACL-HLT.
38. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon LLM: outperforming curated corpora with web data, and web data only. [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Alessandro+Cappelli%2C+Hamza+Alobeidli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+2023%2E+The+refinedweb+dataset+for+f)
39. Ru Peng, Qiuyang Duan, Haobo Wang, Jiachen Ma, Yanbo Jiang, Yongjun Tu, Xiu Jiang, and Junbo Zhao. 2023. Came: Contrastive automated model evaluation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20121– 20132. [find](https://scholar.google.com/scholar?q=Ru+Peng%2C+Qiuyang+Duan%2C+Haobo+Wang%2C+Jiachen+Ma%2C+Yanbo+Jiang%2C+Yongjun+Tu%2C+Xiu+Jiang%2C+and+Junbo+Zhao%2E+2023%2E+Came%3A+Contrastive+automated+model+evaluation%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Conf)
40. Ru Peng, Heming Zou, Haobo Wang, Yawen Zeng, Zenan Huang, and Junbo Zhao. 2024. Energy-based automated model evaluation. arXiv preprint arXiv:2401.12689. — [arXiv:2401.12689](https://arxiv.org/abs/2401.12689) [find](https://scholar.google.com/scholar?q=Ru+Peng%2C+Heming+Zou%2C+Haobo+Wang%2C+Yawen+Zeng%2C+Zenan+Huang%2C+and+Junbo+Zhao%2E+2024%2E+Energy%2Dbased+automated+model+evaluation%2E+arXiv+preprint+arXiv%3A2401%2E12689%2E+%E2%80%94+%5BarXiv%3A2401%2E12689%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F240)
41. Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. 2015. Causal inference using invariant prediction: identification and confidence intervals. [find](https://scholar.google.com/scholar?q=Jonas+Peters%2C+Peter+B%C3%BChlmann%2C+and+Nicolai+Meinshausen%2E+2015%2E+Causal+inference+using+invariant+prediction%3A+identification+and+confidence+intervals%2E)
42. Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqi… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
43. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal… — [Code Llama: Open Foundation Models for Code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
44. Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris M. Mooij. 2012. On causal and anticausal learning. [find](https://scholar.google.com/scholar?q=Bernhard+Sch%C3%B6lkopf%2C+Dominik+Janzing%2C+Jonas+Peters%2C+Eleni+Sgouritsa%2C+Kun+Zhang%2C+and+Joris+M%2E+Mooij%2E+2012%2E+On+causal+and+anticausal+learning%2E)
45. Zheyan Shen, Peng Cui, Tong Zhang, and Kun Kuang. 2020. Stable learning via sample reweighting. pages 5692–5699. AAAI Press. [find](https://scholar.google.com/scholar?q=Zheyan+Shen%2C+Peng+Cui%2C+Tong+Zhang%2C+and+Kun+Kuang%2E+2020%2E+Stable+learning+via+sample+reweighting%2E+pages+5692%E2%80%935699%2E+AAAI+Press%2E)
46. Xiaoshuai Song, Keqing He, Pei Wang, Guanting Dong, Yutao Mou, Jingang Wang, Yunsen Xian, Xunliang Cai, and Weiran Xu. 2023. Large language models meet open-world intent discovery and recognition: An evaluation of chatgpt. [find](https://scholar.google.com/scholar?q=Xiaoshuai+Song%2C+Keqing+He%2C+Pei+Wang%2C+Guanting+Dong%2C+Yutao+Mou%2C+Jingang+Wang%2C+Yunsen+Xian%2C+Xunliang+Cai%2C+and+Weiran+Xu%2E+2023%2E+Large+language+models+meet+open%2Dworld+intent+discovery+and+recognition%3A+An+)
47. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca). [↗](https://github.com/tatsu-lab/stanford_alpaca)
48. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. [Galactica: A large language model for science](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/).
49. InternLM Team. 2023a. Internlm: A multilingual language model with progressively enhanced capabilities. [https://github.com/InternLM/InternLM](https://github.com/InternLM/InternLM). [↗](https://github.com/InternLM/InternLM)
50. MosaicML NLP Team. 2023b. Introducing mpt-7b: A new standard for open-source, commercially usable llms. Accessed: 2023-05-05. [find](https://scholar.google.com/scholar?q=MosaicML+NLP+Team%2E+2023b%2E+Introducing+mpt%2D7b%3A+A+new+standard+for+open%2Dsource%2C+commercially+usable+llms%2E+Accessed%3A+2023%2D05%2D05%2E)
51. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/).
52. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+Dan+Bikel%2C+Lukas+Blecher%2C+Cristian+Canton+Fe)
53. Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miłoś. 2023. Focused transformer: Contrastive training for context scaling. arXiv preprint arXiv:2307.03170. — [arXiv:2307.03170](https://arxiv.org/abs/2307.03170) [find](https://scholar.google.com/scholar?q=Szymon+Tworkowski%2C+Konrad+Staniszewski%2C+Miko%C5%82aj+Pacek%2C+Yuhuai+Wu%2C+Henryk+Michalewski%2C+and+Piotr+Mi%C5%82os%CC%81%2E+2023%2E+Focused+transformer%3A+Contrastive+training+for+context+scaling%2E+arXiv+preprint+arXiv%3A2307%2E0)
54. Z. Lin Y. Sheng Z. Wu H. Zhang L. Zheng S. Zhuang Y. Zhuang J. Gonzalez I. Stoica W. Chiang, Z. Li and E. Xing. 2023. icuna: An open-source chatbot impressing gpt-4 with 90 Technical Report. [find](https://scholar.google.com/scholar?q=Z%2E+Lin+Y%2E+Sheng+Z%2E+Wu+H%2E+Zhang+L%2E+Zheng+S%2E+Zhuang+Y%2E+Zhuang+J%2E+Gonzalez+I%2E+Stoica+W%2E+Chiang%2C+Z%2E+Li+and+E%2E+Xing%2E+2023%2E+icuna%3A+An+open%2Dsource+chatbot+impressing+gpt%2D4+with+90+Technical+Report%2E)
55. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [https://github.com/kingoflolz/](https://github.com/kingoflolz/) mesh-transformer-jax. [↗](https://github.com/kingoflolz/)
56. Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, and Tao Qin. 2021. Generalizing to unseen domains: A survey on domain generalization. pages 4627–4635. [find](https://scholar.google.com/scholar?q=Jindong+Wang%2C+Cuiling+Lan%2C+Chang+Liu%2C+Yidong+Ouyang%2C+and+Tao+Qin%2E+2021%2E+Generalizing+to+unseen+domains%3A+A+survey+on+domain+generalization%2E+pages+4627%E2%80%934635%2E)
57. Pei Wang, Yejie Wang, Muxi Diao, Keqing He, Guanting Dong, and Weiran Xu. 2024a. Multi-perspective consistency enhances confidence estimation in large language models. arXiv preprint arXiv:2402.11279. — [arXiv:2402.11279](https://arxiv.org/abs/2402.11279) [find](https://scholar.google.com/scholar?q=Pei+Wang%2C+Yejie+Wang%2C+Muxi+Diao%2C+Keqing+He%2C+Guanting+Dong%2C+and+Weiran+Xu%2E+2024a%2E+Multi%2Dperspective+consistency+enhances+confidence+estimation+in+large+language+models%2E+arXiv+preprint+arXiv%3A2402%2E11279%2E)
58. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023a. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations.
59. Yejie Wang, Keqing He, Guanting Dong, Pei Wang, Weihao Zeng, Muxi Diao, Yutao Mou, Mengdi Zhang, Jingang Wang, Xunliang Cai, and Weiran Xu. 2024b. Dolphcoder: Echo-locating code large language models with diverse and multi-objective instruction tuning. [find](https://scholar.google.com/scholar?q=Yejie+Wang%2C+Keqing+He%2C+Guanting+Dong%2C+Pei+Wang%2C+Weihao+Zeng%2C+Muxi+Diao%2C+Yutao+Mou%2C+Mengdi+Zhang%2C+Jingang+Wang%2C+Xunliang+Cai%2C+and+Weiran+Xu%2E+2024b%2E+Dolphcoder%3A+Echo%2Dlocating+code+large+language+models+)
60. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A%2E+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2023b%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+In+Pro)
61. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837.
62. Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023. Magicoder: Source code is all you need. [find](https://scholar.google.com/scholar?q=Yuxiang+Wei%2C+Zhe+Wang%2C+Jiawei+Liu%2C+Yifeng+Ding%2C+and+Lingming+Zhang%2E+2023%2E+Magicoder%3A+Source+code+is+all+you+need%2E)
63. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. [find](https://scholar.google.com/scholar?q=Can+Xu%2C+Qingfeng+Sun%2C+Kai+Zheng%2C+Xiubo+Geng%2C+Pu+Zhao%2C+Jiazhan+Feng%2C+Chongyang+Tao%2C+and+Daxin+Jiang%2E+2023%2E+Wizardlm%3A+Empowering+large+language+models+to+follow+complex+instructions%2E)
64. Mingfeng Xue, Dayiheng Liu, Kexin Yang, Guanting Dong, Wenqiang Lei, Zheng Yuan, Chang Zhou, and Jingren Zhou. 2023. Occuquest: Mitigating occupational bias for inclusive large language models. [find](https://scholar.google.com/scholar?q=Mingfeng+Xue%2C+Dayiheng+Liu%2C+Kexin+Yang%2C+Guanting+Dong%2C+Wenqiang+Lei%2C+Zheng+Yuan%2C+Chang+Zhou%2C+and+Jingren+Zhou%2E+2023%2E+Occuquest%3A+Mitigating+occupational+bias+for+inclusive+large+language+models%2E)
65. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023a. [Instruction-following evaluation for large language models](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/).
66. Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. 2023b. Domain generalization: A survey. IEEE Trans. Pattern Anal. Mach. Intell., pages 4396–4415. [find](https://scholar.google.com/scholar?q=Kaiyang+Zhou%2C+Ziwei+Liu%2C+Yu+Qiao%2C+Tao+Xiang%2C+and+Chen+Change+Loy%2E+2023b%2E+Domain+generalization%3A+A+survey%2E+IEEE+Trans%2E+Pattern+Anal%2E+Mach%2E+Intell%2E%2C+pages+4396%E2%80%934415%2E)
67. Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang. 2023a. [Solving math word problems via cooperative reasoning induced language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-solving-math-word-problems-cooperative/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P…
68. Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang. 2023b. [Solving math word problems via cooperative reasoning induced language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-solving-math-word-problems-cooperative/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P…
69. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/).
70. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023a. [Scaling relationship on learning mathematical reasoning with large language models](https://md.research.elamur.ai/free/en/papers/yuan-2023-scaling-relationship-learning-mathematical-reasoning/).
71. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. 2023b. [How well do large language models perform in arithmetic tasks?](https://md.research.elamur.ai/free/en/papers/yuan-2023-well-large-language-models-perform/) arXiv preprint arXiv:2304.02015.
72. Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2023a. Mammoth: Building math generalist models through hybrid instruction. [find](https://scholar.google.com/scholar?q=Xiang+Yue%2C+Xingwei+Qu%2C+Ge+Zhang%2C+Yao+Fu%2C+Wenhao+Huang%2C+Huan+Sun%2C+Yu+Su%2C+and+Wenhu+Chen%2E+2023a%2E+Mammoth%3A+Building+math+generalist+models+through+hybrid+instruction%2E)
73. Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2023b. [MAmmoTH: Building math generalist models through hybrid instruction tuning](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/). arXiv preprint arXiv:2309.05653.
74. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. [STar: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). In Advances in Neural Information Processing Systems.
75. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. [Glm-130b: An open bilingual pre-trained model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/). arXiv preprint arXiv:2210.02414.

*Updated 3 November 2026*