---
title: "MathCoder: Seamless Code Integration in LLMs for — Wang 2023"
description: "Open-source LLMs do much worse than proprietary systems such as GPT-4 Code Interpreter on school and competition math."
authors: ["Ke Wang", "Houxing Ren", "Aojun Zhou", "Zimu Lu", "Sichun Luo", "Weikang Shi", "Renrui Zhang", "Linqi Song", "Mingjie Zhan", "Hongsheng Li"]
year: 2023
venue: "arXiv:2310.03731"
source: "https://arxiv.org/abs/2310.03731"
doi: "10.48550/arxiv.2310.03731"
arxiv: "2310.03731"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2023-mathcoder-seamless-code-integration-llms/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2023-mathcoder-seamless-code-integration-llms/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning

# MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning

**Authors:** [Ke Wang](https://md.research.elamur.ai/free/en/authors/ke-wang/), Houxing Ren, [Aojun Zhou](https://md.research.elamur.ai/free/en/authors/aojun-zhou/), [Zimu Lu](https://md.research.elamur.ai/free/en/authors/zimu-lu/), [Sichun Luo](https://md.research.elamur.ai/free/en/authors/sichun-luo/), [Weikang Shi](https://md.research.elamur.ai/free/en/authors/weikang-shi/), [Renrui Zhang](https://md.research.elamur.ai/free/en/authors/renrui-zhang/), [Linqi Song](https://md.research.elamur.ai/free/en/authors/linqi-song/), [Mingjie Zhan](https://md.research.elamur.ai/free/en/authors/mingjie-zhan/), [Hongsheng Li](https://md.research.elamur.ai/free/en/authors/hongsheng-li/) · **Year:** 2023 · **Venue:** arXiv:2310.03731 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.03731) · [PDF](https://arxiv.org/pdf/2310.03731) · [DOI 10.48550/arxiv.2310.03731](https://doi.org/10.48550/arxiv.2310.03731) · [arXiv 2310.03731](https://arxiv.org/abs/2310.03731) · [OpenAlex](https://openalex.org/W4387436656)

**Citations:** 4 citations (openalex-snapshot · 2026-10-14) · [94](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Open-source LLMs do much worse than proprietary systems such as GPT-4 Code Interpreter on school and competition math. This paper tries to give open models the Code Interpreter way of working: the model switches between reasoning in natural language, writing code, running it and reading the output. The authors build MathCodeInstruct, an 80k-example instruction dataset. First, GPT-4 wrote solutions in this mixed format for about 49k GSM8K and MATH training problems. A CodeLlama-34B model tuned on that data (MathCoder-Initial) then solved 31k new problems created by "problem interpolation", a prompt that produces questions harder than GSM8K but easier than MATH. Special tokens mark text, code and execution output. At inference, each code block runs in a Jupyter kernel and its result is added back to the context so generation can continue. The resulting MathCoder models, fine-tuned from Llama-2 and CodeLlama, score 83.9% on GSM8K and 45.2% on MATH. These were the best open-source results at the time, ahead of ChatGPT-3.5 and PaLM-2, and above GPT-4 on MATH. The work shows that open models can learn this tool-assisted way of reasoning.

## Key points
- MathCodeInstruct has 80k problem–solution pairs: about 49k GPT-4 solutions for GSM8K and MATH training problems, plus 31k interpolated problems solved by MathCoder-Initial (CodeLlama-34B).
- The solution format is called LCE: natural-language reasoning, code and execution results in one sequence. The special tokens are `<|text|>`, `<|code|>` and `<|execution|>`.
- There are two model families: MathCoder-L (Llama-2 7B/13B/70B) and MathCoder-CL (CodeLlama 7B/13B/34B).
- Best scores are 45.2% on MATH and 83.9% on GSM8K. For comparison, Llama-2-70B gets 13.5% and 56.8%, and GPT-4 Code Interpreter gets 69.7% and 97%.
- The models also do well on out-of-domain sets (Mathematics, SimulEq). The code and data are at github.com/mathllm/MathCoder.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/)

## Cited by

Cited by

94 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 87 not yet.

- [AlphaMath Almost Zero: Process Supervision without Process](https://md.research.elamur.ai/free/en/papers/chen-2024-alphamath-almost-zero-process-supervision/)
- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2025-towards-reasoning-era-survey-long/)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://md.research.elamur.ai/free/en/papers/ge-2024-scaling-synthetic-data-creation-1/)
- [rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking](https://md.research.elamur.ai/free/en/papers/guan-2025-rstar-math-small-llms-can/)
- [ReFT: Reasoning with Reinforced Fine-Tuning](https://md.research.elamur.ai/free/en/papers/trung-2024-reft-reasoning-reinforced-fine-tuning/)
- [Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset](https://md.research.elamur.ai/free/en/papers/wang-2024-measuring-multimodal-mathematical-reasoning-math/)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://md.research.elamur.ai/free/en/papers/zhang-2024-mathverse-multi-modal-llm-truly/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wang2023mathcoder,
  title = {{MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning}},
  author = {Wang, Ke and Ren, Houxing and Zhou, Aojun and Lu, Zimu and Luo, Sichun and Shi, Weikang and Zhang, Renrui and Song, Linqi and Zhan, Mingjie and Li, Hongsheng},
  year = {2023},
  doi = {10.48550/arxiv.2310.03731},
  eprint = {2310.03731},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.03731}
}
```

- [← previous Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models](https://md.research.elamur.ai/free/en/papers/cheng-2023-marked-personas-natural-language-prompts/)
- [next → MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems](https://md.research.elamur.ai/free/en/papers/macina-2023-mathdial-dialogue-tutoring-dataset-rich/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

47 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816, 2020. — [arXiv:2012.09816](https://arxiv.org/abs/2012.09816) [find](https://scholar.google.com/scholar?q=Zeyuan+Allen%2DZhu+and+Yuanzhi+Li%2E+Towards+understanding+ensemble%2C+knowledge+distillation+and+self%2Ddistillation+in+deep+learning%2E+arXiv+preprint+arXiv%3A2012%2E09816%2C+2020%2E+%E2%80%94+%5BarXiv%3A2012%2E09816%5D%28https%3A%2F%2Farxi)
2. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. [find](https://scholar.google.com/scholar?q=Rohan+Anil%2C+Andrew+M+Dai%2C+Orhan+Firat%2C+Melvin+Johnson%2C+Dmitry+Lepikhin%2C+Alexandre+Passos%2C+Siamak+Shakeri%2C+Emanuel+Taropa%2C+Paige+Bailey%2C+Zhifeng+Chen%2C+et+al%2E+Palm+2+technical+report%2E+arXiv+preprint+arX) [our card](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/)
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+)
4. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022. [find](https://scholar.google.com/scholar?q=Wenhu+Chen%2C+Xueguang+Ma%2C+Xinyi+Wang%2C+and+William+W+Cohen%2E+Program+of+thoughts+prompting%3A+Disentangling+computation+from+reasoning+for+numerical+reasoning+tasks%2E+arXiv+preprint+arXiv%3A2211%2E12588%2C+2022%2E)
5. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXiv+prep)
6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [find](https://scholar.google.com/scholar?q=Karl+Cobbe%2C+Vineet+Kosaraju%2C+Mohammad+Bavarian%2C+Mark+Chen%2C+Heewoo+Jun%2C+Lukasz+Kaiser%2C+Matthias+Plappert%2C+Jerry+Tworek%2C+Jacob+Hilton%2C+Reiichiro+Nakano%2C+et+al%2E+Training+verifiers+to+solve+math+word+prob) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
7. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Daniel+Y%2E+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2C+2022%2E)
8. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. A survey on in-context learning, 2023. [find](https://scholar.google.com/scholar?q=Qingxiu+Dong%2C+Lei+Li%2C+Damai+Dai%2C+Ce+Zheng%2C+Zhiyong+Wu%2C+Baobao+Chang%2C+Xu+Sun%2C+Jingjing+Xu%2C+Lei+Li%2C+and+Zhifang+Sui%2E+A+survey+on+in%2Dcontext+learning%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/)
9. Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting for multi-step reasoning. arXiv preprint arXiv:2210.00720, 2022. [find](https://scholar.google.com/scholar?q=Yao+Fu%2C+Hao+Peng%2C+Ashish+Sabharwal%2C+Peter+Clark%2C+and+Tushar+Khot%2E+Complexity%2Dbased+prompting+for+multi%2Dstep+reasoning%2E+arXiv+preprint+arXiv%3A2210%2E00720%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/fu-2022-complexity-based-prompting-multi-step/)
10. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pp. 10764–10799. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Pengfei+Liu%2C+Yiming+Yang%2C+Jamie+Callan%2C+and+Graham+Neubig%2E+Pal%3A+Program%2Daided+language+models%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+10764%E2%80%93)
11. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ArXiv, abs/2009.03300, 2020. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 221516475. [↗](https://api.semanticscholar.org/CorpusID:) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
12. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Collin+Burns%2C+Saurav+Kadavath%2C+Akul+Arora%2C+Steven+Basart%2C+Eric+Tang%2C+Dawn+Song%2C+and+Jacob+Steinhardt%2E+Measuring+mathematical+problem+solving+with+the+math+dataset%2E+arXiv+preprint+arXiv%3A) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/)
13. Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531, 2015. URL [https://api.semanticscholar.org/CorpusID:7200347](https://api.semanticscholar.org/CorpusID:7200347). [↗](https://api.semanticscholar.org/CorpusID:7200347) [our card](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/)
14. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2023. [find](https://scholar.google.com/scholar?q=Takeshi+Kojima%2C+Shixiang+Shane+Gu%2C+Machel+Reid%2C+Yutaka+Matsuo%2C+and+Yusuke+Iwasawa%2E+Large+language+models+are+zero%2Dshot+reasoners%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)
15. Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. Learning to automatically solve algebra word problems. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 271–281, 2014. [find](https://scholar.google.com/scholar?q=Nate+Kushman%2C+Yoav+Artzi%2C+Luke+Zettlemoyer%2C+and+Regina+Barzilay%2E+Learning+to+automatically+solve+algebra+word+problems%2E+In+Proceedings+of+the+52nd+Annual+Meeting+of+the+Association+for+Computational+L)
16. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843–3857, 2022. [find](https://scholar.google.com/scholar?q=Aitor+Lewkowycz%2C+Anders+Andreassen%2C+David+Dohan%2C+Ethan+Dyer%2C+Henryk+Michalewski%2C+Vinay+Ramasesh%2C+Ambrose+Slone%2C+Cem+Anil%2C+Imanol+Schlag%2C+Theo+Gutman%2DSolo%2C+et+al%2E+Solving+quantitative+reasoning+problem) [our card](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/)
17. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas… [find](https://scholar.google.com/scholar?q=Raymond+Li%2C+Loubna+Ben+Allal%2C+Yangtian+Zi%2C+Niklas+Muennighoff%2C+Denis+Kocetkov%2C+Chenghao+Mou%2C+Marc+Marone%2C+Christopher+Akiki%2C+Jia+Li%2C+Jenny+Chim%2C+Qian+Liu%2C+Evgenii+Zheltonozhskii%2C+Terry+Yue+Zhuo%2C+Thoma)
18. Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. ArXiv, abs/2308.06259, 2023b. URL [https://api.semanticscholar.org/CorpusID:260866107](https://api.semanticscholar.org/CorpusID:260866107). [↗](https://api.semanticscholar.org/CorpusID:260866107) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-self-alignment-instruction-backtranslation/)
19. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 158–167, Vancouver, Canada… [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Dani+Yogatama%2C+Chris+Dyer%2C+and+Phil+Blunsom%2E+Program+induction+by+rationale+generation%3A+Learning+to+solve+and+explain+algebraic+word+problems%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+t)
20. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688, 2023. [find](https://scholar.google.com/scholar?q=Shayne+Longpre%2C+Le+Hou%2C+Tu+Vu%2C+Albert+Webson%2C+Hyung+Won+Chung%2C+Yi+Tay%2C+Denny+Zhou%2C+Quoc+V+Le%2C+Barret+Zoph%2C+Jason+Wei%2C+et+al%2E+The+flan+collection%3A+Designing+data+and+methods+for+effective+instruction+t) [our card](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/)
21. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583, 2023. [find](https://scholar.google.com/scholar?q=Haipeng+Luo%2C+Qingfeng+Sun%2C+Can+Xu%2C+Pu+Zhao%2C+Jianguang+Lou%2C+Chongyang+Tao%2C+Xiubo+Geng%2C+Qingwei+Lin%2C+Shifeng+Chen%2C+and+Dongmei+Zhang%2E+Wizardmath%3A+Empowering+mathematical+reasoning+for+large+language+mod) [our card](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/)
22. OpenAI. Gpt-4 technical report, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
23. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191, 2021. [find](https://scholar.google.com/scholar?q=Arkil+Patel%2C+Satwik+Bhattamishra%2C+and+Navin+Goyal%2E+Are+nlp+models+really+able+to+solve+simple+math+word+problems%3F+arXiv+preprint+arXiv%3A2103%2E07191%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/)
24. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116… [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Alessandro+Cappelli%2C+Hamza+Alobeidli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+The+refinedweb+dataset+for+falcon+)
25. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023. [find](https://scholar.google.com/scholar?q=Baolin+Peng%2C+Chunyuan+Li%2C+Pengcheng+He%2C+Michel+Galley%2C+and+Jianfeng+Gao%2E+Instruction+tuning+with+gpt%2D4%2E+arXiv+preprint+arXiv%3A2304%2E03277%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/peng-2023-instruction-tuning-gpt-4/)
26. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2020. [find](https://scholar.google.com/scholar?q=Samyam+Rajbhandari%2C+Jeff+Rasley%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+Zero%3A+Memory+optimizations+toward+training+trillion+parameter+models%2C+2020%2E)
27. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal… [find](https://scholar.google.com/scholar?q=Baptiste+Rozi%C3%A8re%2C+Jonas+Gehring%2C+Fabian+Gloeckle%2C+Sten+Sootla%2C+Itai+Gat%2C+Xiaoqing+Ellen+Tan%2C+Yossi+Adi%2C+Jingyu+Liu%2C+Tal+Remez%2C+J%C3%A9r%C3%A9my+Rapin%2C+Artyom+Kozhevnikov%2C+Ivan+Evtimov%2C+Joanna+Bitton%2C+Manish+Bha) [our card](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
28. Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207, 2021. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Albert+Webson%2C+Colin+Raffel%2C+Stephen+H+Bach%2C+Lintang+Sutawika%2C+Zaid+Alyafeai%2C+Antoine+Chaffin%2C+Arnaud+Stiegler%2C+Teven+Le+Scao%2C+Arun+Raja%2C+et+al%2E+Multitask+prompted+training+enables+zero%2Ds) [our card](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
29. David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. arXiv preprint arXiv:1904.01557, 2019. [find](https://scholar.google.com/scholar?q=David+Saxton%2C+Edward+Grefenstette%2C+Felix+Hill%2C+and+Pushmeet+Kohli%2E+Analysing+mathematical+reasoning+abilities+of+neural+models%2E+arXiv+preprint+arXiv%3A1904%2E01557%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/saxton-2019-analysing-mathematical-reasoning-abilities-neural/)
30. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
31. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085, 2022. [find](https://scholar.google.com/scholar?q=Ross+Taylor%2C+Marcin+Kardas%2C+Guillem+Cucurull%2C+Thomas+Scialom%2C+Anthony+Hartshorn%2C+Elvis+Saravia%2C+Andrew+Poulton%2C+Viktor+Kerkez%2C+and+Robert+Stojnic%2E+Galactica%3A+A+large+language+model+for+science%2E+arXiv+) [our card](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/)
32. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtun)
33. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023a. URL [https://openreview.net/forum?id…](https://openreview.net/forum?id…) [↗](https://openreview.net/forum?id…) [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/)
34. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022a. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+arXiv+preprint+)
35. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. arXiv preprint arXiv:2204.07705,… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Anjana+Arunkumar%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Atharva+Naik%2C+David+Stap%2C+et+al%2E+Super%2Dnaturalinstructio)
36. Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. arXiv preprint arXiv:2306.04751, 2023b. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Hamish+Ivison%2C+Pradeep+Dasigi%2C+Jack+Hessel%2C+Tushar+Khot%2C+Khyathi+Raghavi+Chandu%2C+David+Wadden%2C+Kelsey+MacMillan%2C+Noah+A+Smith%2C+Iz+Beltagy%2C+et+al%2E+How+far+can+camels+go%3F+exploring+the+sta) [our card](https://md.research.elamur.ai/free/en/papers/wang-2023-far-camels-go-exploring-state/)
37. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Bosma%2C+Vincent+Y+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+M+Dai%2C+and+Quoc+V+Le%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+arXiv+preprint+arXiv%3A2109%2E01652)
38. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Proce… [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Maarten+Bosma%2C+brian+ichter%2C+Fei+Xia%2C+Ed+H%2E+Chi%2C+Quoc+V+Le%2C+and+Denny+Zhou%2E+Chain+of+thought+prompting+elicits+reasoning+in+large+language+models%2E+In+Alice+H%2E+) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
39. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023. [find](https://scholar.google.com/scholar?q=Can+Xu%2C+Qingfeng+Sun%2C+Kai+Zheng%2C+Xiubo+Geng%2C+Pu+Zhao%2C+Jiazhan+Feng%2C+Chongyang+Tao%2C+and+Daxin+Jiang%2E+Wizardlm%3A+Empowering+large+language+models+to+follow+complex+instructions%2E+arXiv+preprint+arXiv%3A2304) [our card](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/)
40. Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. Crossfit: A few-shot learning challenge for cross-task generalization in nlp. arXiv preprint arXiv:2104.08835, 2021. [find](https://scholar.google.com/scholar?q=Qinyuan+Ye%2C+Bill+Yuchen+Lin%2C+and+Xiang+Ren%2E+Crossfit%3A+A+few%2Dshot+learning+challenge+for+cross%2Dtask+generalization+in+nlp%2E+arXiv+preprint+arXiv%3A2104%2E08835%2C+2021%2E)
41. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023. [find](https://scholar.google.com/scholar?q=Zheng+Yuan%2C+Hongyi+Yuan%2C+Chengpeng+Li%2C+Guanting+Dong%2C+Chuanqi+Tan%2C+and+Chang+Zhou%2E+Scaling+relationship+on+learning+mathematical+reasoning+with+large+language+models%2E+arXiv+preprint+arXiv%3A2308%2E01825%2C+) [our card](https://md.research.elamur.ai/free/en/papers/yuan-2023-scaling-relationship-learning-mathematical-reasoning/)
42. Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning, 2023. [find](https://scholar.google.com/scholar?q=Xiang+Yue%2C+Xingwei+Qu%2C+Ge+Zhang%2C+Yao+Fu%2C+Wenhao+Huang%2C+Huan+Sun%2C+Yu+Su%2C+and+Wenhu+Chen%2E+Mammoth%3A+Building+math+generalist+models+through+hybrid+instruction+tuning%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/)
43. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. [find](https://scholar.google.com/scholar?q=Susan+Zhang%2C+Stephen+Roller%2C+Naman+Goyal%2C+Mikel+Artetxe%2C+Moya+Chen%2C+Shuohui+Chen%2C+Christopher+Dewan%2C+Mona+Diab%2C+Xian+Li%2C+Xi+Victoria+Lin%2C+et+al%2E+Opt%3A+Open+pre%2Dtrained+transformer+language+models%2E+arXi)
44. Xu Zhao, Yuxi Xie, Kenji Kawaguchi, Junxian He, and Qizhe Xie. Automatic model selection with large language models for reasoning, 2023. [find](https://scholar.google.com/scholar?q=Xu+Zhao%2C+Yuxi+Xie%2C+Kenji+Kawaguchi%2C+Junxian+He%2C+and+Qizhe+Xie%2E+Automatic+model+selection+with+large+language+models+for+reasoning%2C+2023%2E)
45. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric%2E+P+Xing%2C+Hao+Zhang%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+Judging+llm%2Das%2Da%2Djud)
46. Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. arXiv preprint arXiv:2308.07921, 2023a. [find](https://scholar.google.com/scholar?q=Aojun+Zhou%2C+Ke+Wang%2C+Zimu+Lu%2C+Weikang+Shi%2C+Sichun+Luo%2C+Zipeng+Qin%2C+Shaoqing+Lu%2C+Anya+Jia%2C+Linqi+Song%2C+Mingjie+Zhan%2C+et+al%2E+Solving+challenging+math+word+problems+using+gpt%2D4+code+interpreter+with+code) [our card](https://md.research.elamur.ai/free/en/papers/zhou-2023-solving-challenging-math-word-problems/)
47. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023b. [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Pengfei+Liu%2C+Puxin+Xu%2C+Srini+Iyer%2C+Jiao+Sun%2C+Yuning+Mao%2C+Xuezhe+Ma%2C+Avia+Efrat%2C+Ping+Yu%2C+Lili+Yu%2C+et+al%2E+Lima%3A+Less+is+more+for+alignment%2E+arXiv+preprint+arXiv%3A2305%2E11206%2C+2023b%2E)