---
title: "MathGenie — paper by Lu et al. (2024) — summary and ideas"
description: "Open-source LLMs still trail closed systems like GPT-4 at math reasoning, and building training data… 35 references, cited by 44 papers here."
authors: ["Zimu Lu", "Aojun Zhou", "Houxing Ren", "Ke Wang", "Weikang Shi", "Junting Pan", "Mingjie Zhan", "Hongsheng Li"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.151/"
doi: "10.18653/v1/2024.acl-long.151"
arxiv: "2402.16352"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lu-2024-mathgenie-generating-synthetic-data-question/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lu-2024-mathgenie-generating-synthetic-data-question/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs

# MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs

**Authors:** [Zimu Lu](https://md.research.elamur.ai/free/en/authors/zimu-lu/), [Aojun Zhou](https://md.research.elamur.ai/free/en/authors/aojun-zhou/), [Houxing Ren](https://md.research.elamur.ai/free/en/authors/houxing-ren/), [Ke Wang](https://md.research.elamur.ai/free/en/authors/ke-wang/), [Weikang Shi](https://md.research.elamur.ai/free/en/authors/weikang-shi/), [Junting Pan](https://md.research.elamur.ai/free/en/authors/junting-pan/), [Mingjie Zhan](https://md.research.elamur.ai/free/en/authors/mingjie-zhan/) and [Hongsheng Li](https://md.research.elamur.ai/free/en/authors/hongsheng-li/) · **Year:** 2024 · **Venue:** [Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.151/) · [PDF](https://aclanthology.org/2024.acl-long.151.pdf) · [DOI 10.18653/v1/2024.acl-long.151](https://doi.org/10.18653/v1/2024.acl-long.151) · [arXiv 2402.16352](https://arxiv.org/abs/2402.16352) · [OpenAlex](https://openalex.org/W4402684157)

**Citations:** 14 citations (OpenAlex, 2026-10-14) · [44](https://md.research.elamur.ai/free/en/papers/lu-2024-mathgenie-generating-synthetic-data-question/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Open-source LLMs still trail closed systems like GPT-4 at math reasoning, and building training data with GPT-4 is costly to scale. MathGenie is a pipeline that uses open models to produce large amounts of synthetic math training data from a small seed set of about 15K GSM8K and MATH problems with human-written solutions. It works in three steps:

- **Solution augmentation:** a model repeatedly rewrites the seed solutions into new variants.
- **Question back-translation:** a separate model turns each new solution into a matching question. Because questions are derived from solutions, the logical constraints of the problem stay intact.
- **Verification-based filtering:** a Llama-2 70B model writes code-integrated solutions for the new questions. It then checks them with verification rationales that mix natural language and code, and only solutions judged correct are kept.

Fine-tuning base models from 7B to 70B on the resulting MathGenieData produces the MathGenieLM family, which beats earlier open-source models on five benchmarks. This shows that cheap, open-model data generation can narrow the gap with proprietary systems.

## Key points
- The best single model, MathGenieLM-InternLM2, reaches 87.7% on GSM8K and 55.7% on MATH. MathGenieLM-Llama-2-70B with 10-path majority voting reaches 91.7% and 63.3%.
- The evaluation covers MATH, GSM8K, SVAMP, Simuleq and Mathematics, across six pretrained base models.
- The solution generator M_code (Llama-2 70B) was trained on 80K GPT-4 Code Interpreter samples containing solutions and verification rationales. On its own, it scores 86.4% on GSM8K and 49.5% on MATH.
- In an ablation, back-translation averaged 72.0, compared with 70.0 for MetaMath-style data and 69.5–70.6 for direct question augmentation.
- Verified inference (up to two verification rounds) raises the 70B model's average from 78.5 to 81.1 at about 2.3× generation cost. This nearly matches 3-path voting (81.4) at a lower cost.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Chinese University of Hong Kong](https://md.research.elamur.ai/free/en/universities/chinese-university-of-hong-kong/) — University, Hong Kong, China · 159 works · 16,154 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan and Hongsheng Li. 2024. MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{lu2024mathgenie,
  title = {{MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs}},
  author = {Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan and Hongsheng Li},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.151},
  eprint = {2402.16352},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.acl-long.151}
}
```

- [← previous MaLA-500: Massive Language Adaptation of Large Language Models](https://md.research.elamur.ai/free/en/papers/lin-2024-mala-500-massive-language-adaptation/)
- [next → MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance](https://md.research.elamur.ai/free/en/papers/pi-2024-mllm-protector-ensuring-mllm-s/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

35 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403.
2. Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. 2023. [find](https://scholar.google.com/scholar?q=Zhangir+Azerbayev%2C+Hailey+Schoelkopf%2C+Keiran+Paster%2C+Marco+Dos+Santos%2C+Stephen+McAleer%2C+Albert+Q+Jiang%2C+Jia+Deng%2C+Stella+Biderman%2C+and+Sean+Welleck%2E+2023%2E)
3. [Llemma: An open language model for mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/). arXiv preprint arXiv:2310.10631.
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
5. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2022. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588. [find](https://scholar.google.com/scholar?q=Wenhu+Chen%2C+Xueguang+Ma%2C+Xinyi+Wang%2C+and+William+W+Cohen%2E+2022%2E+Program+of+thoughts+prompting%3A+Disentangling+computation+from+reasoning+for+numerical+reasoning+tasks%2E+arXiv+preprint+arXiv%3A2211%2E12588%2E)
6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
7. Alex Davies, Petar Velickovic, Lars Buesing, Sam Blackwell, Daniel Zheng, Nenad Tomašev, Richard Tanburn, Peter W. Battaglia, Charles Blundell, András Juhász, Marc Lackenby, Geordie Williamson, Demis Hassabis, and Pushmeet Kohli. 2021. Advancing mathematics by guiding human intuition with ai. Nature… [find](https://scholar.google.com/scholar?q=Alex+Davies%2C+Petar+Velickovic%2C+Lars+Buesing%2C+Sam+Blackwell%2C+Daniel+Zheng%2C+Nenad+Toma%C5%A1ev%2C+Richard+Tanburn%2C+Peter+W%2E+Battaglia%2C+Charles+Blundell%2C+Andr%C3%A1s+Juh%C3%A1sz%2C+Marc+Lackenby%2C+Geordie+Williamson%2C+Demis+)
8. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. 2023. [Tora: A tool-integrated reasoning agent for mathematical problem solving](https://md.research.elamur.ai/free/en/papers/gou-2023-tora-tool-integrated-reasoning-agent/). arXiv preprint arXiv:2309.17452.
9. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874.
10. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023%2E+Mist)
11. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+et+al%2E+2024%2E+Mixtral+o)
12. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. Mawps: A math word problem repository. In North American Chapter of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rik+Koncel%2DKedziorski%2C+Subhro+Roy%2C+Aida+Amini%2C+Nate+Kushman%2C+and+Hannaneh+Hajishirzi%2E+2016%2E+Mawps%3A+A+math+word+problem+repository%2E+In+North+American+Chapter+of+the+Association+for+Computational+Lingui)
13. Chengpeng Li, Zheng Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang, and Chang Zhou. 2023a. Query and response augmentation cannot help out-of-domain math reasoning generalization. arXiv preprint arXiv:2310.05506. [find](https://scholar.google.com/scholar?q=Chengpeng+Li%2C+Zheng+Yuan%2C+Guanting+Dong%2C+Keming+Lu%2C+Jiancan+Wu%2C+Chuanqi+Tan%2C+Xiang+Wang%2C+and+Chang+Zhou%2E+2023a%2E+Query+and+response+augmentation+cannot+help+out%2Dof%2Ddomain+math+reasoning+generalization%2E)
14. Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. 2023b. [Self-alignment with instruction back-translation](https://md.research.elamur.ai/free/en/papers/li-2023-self-alignment-instruction-backtranslation/). arXiv preprint arXiv:2308.06259.
15. InternLM Team. 2023. Internlm: A multilingual language model with progressively enhanced capabilities. [https://github.com/InternLM/](https://github.com/InternLM/) InternLM-techreport. [↗](https://github.com/InternLM/)
16. Haoxiong Liu and Andrew Chi-Chih Yao. 2024. [Augmenting math word problems via iterative question composing](https://md.research.elamur.ai/free/en/papers/liu-2024-augmenting-math-word-problems-iterative/). arXiv preprint arXiv:2401.09003.
17. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
18. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2017%2E+Decoupled+weight+decay+regularization%2E+arXiv+preprint+arXiv%3A1711%2E05101%2E)
19. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. [Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). arXiv preprint arXiv:2310.02255.
20. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. [Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/). arXiv preprint arXiv:2308.09583.
21. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. [Orca: Progressive learning from complex explanation traces of gpt-4](https://md.research.elamur.ai/free/en/papers/mukherjee-2023-orca-progressive-learning-complex-explanation/). arXiv preprint arXiv:2306.02707.
22. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). abs/2303.08774.
23. ArXiv, Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. [Are nlp models really able to solve simple math word problems?](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/) arXiv preprint arXiv:2103.07191.
24. Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. 2023. [Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning](https://md.research.elamur.ai/free/en/papers/wang-2023-mathcoder-seamless-code-integration-llms/). In International Conference on Learning Representations.
25. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171.
26. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837.
27. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. [Wizardlm: Empowering large language models to follow complex instructions](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/). arXiv preprint arXiv:2304.12244.
28. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. [Instruction tuning with gpt-4](https://md.research.elamur.ai/free/en/papers/peng-2023-instruction-tuning-gpt-4/). arXiv preprint arXiv:2304.03277.
29. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). arXiv preprint arXiv:2309.12284.
30. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950.
31. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. 2023. [Scaling relationship on learning mathematical reasoning with large language models](https://md.research.elamur.ai/free/en/papers/yuan-2023-scaling-relationship-learning-mathematical-reasoning/). arXiv preprint arXiv:2308.01825.
32. Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, et al. 2023. [Beyond human data: Scaling self-training for problem-solving with language models](https://md.research.elamur.ai/free/en/papers/singh-2023-human-data-scaling-self-training/). arXiv preprint arXiv:2312.06585.
33. Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2023. [Mammoth: Building math generalist models through hybrid instruction tuning](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/). arXiv preprint arXiv:2309.05653.
34. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca). [↗](https://github.com/tatsu-lab/stanford_alpaca)
35. Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, and Hongsheng Li. 2023. [Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification](https://md.research.elamur.ai/free/en/papers/zhou-2023-solving-challenging-math-word-problems/). In International Conference on Learning Representations.

*Updated 2 November 2026*