---
title: "MathScale: Scaling Instruction Tuning for Mathematical Reasoning"
description: "The authors present this as similar to how people learn math: first compressing concepts, then… 32 references, cited by 81 papers here."
authors: ["Zhengyang Tang", "Xingxing Zhang", "Benyou Wang", "Furu Wei"]
year: 2024
venue: "arXiv:2403.02884"
source: "https://arxiv.org/abs/2403.02884"
doi: "10.48550/arxiv.2403.02884"
arxiv: "2403.02884"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tang-2024-mathscale-scaling-instruction-tuning-mathematical/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tang-2024-mathscale-scaling-instruction-tuning-mathematical/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / MathScale: Scaling Instruction Tuning for Mathematical Reasoning

# MathScale: Scaling Instruction Tuning for Mathematical Reasoning

**Authors:** [Zhengyang Tang](https://md.research.elamur.ai/free/en/authors/zhengyang-tang/), [Xingxing Zhang](https://md.research.elamur.ai/free/en/authors/xingxing-zhang/), [Benyou Wang](https://md.research.elamur.ai/free/en/authors/benyou-wang/) and [Furu Wei](https://md.research.elamur.ai/free/en/authors/furu-wei/) · **Year:** 2024 · **Venue:** arXiv:2403.02884 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2403.02884) · [PDF](https://arxiv.org/pdf/2403.02884) · [DOI 10.48550/arxiv.2403.02884](https://doi.org/10.48550/arxiv.2403.02884) · [arXiv 2403.02884](https://arxiv.org/abs/2403.02884) · [OpenAlex](https://openalex.org/W4392538380)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [81](https://md.research.elamur.ai/free/en/papers/tang-2024-mathscale-scaling-instruction-tuning-mathematical/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLMs still have trouble with multi-step math problems. One reason is that the main training sets are small: GSM8K and MATH each have only about 7.5K examples. Earlier augmentation methods such as WizardMath and MetaMath rewrite existing questions, so their output stays close to the seed questions and is hard to scale. MathScale takes a different route. GPT-3.5 first reduces about 20K seed questions to abstract topics and knowledge points. These concepts are linked in a weighted co-occurrence graph, and new problems are generated from combinations of concepts sampled from the graph. The authors present this as similar to how people learn math: first compressing concepts, then making connections between them. The pipeline produced MathScaleQA, which has two million question–answer pairs. LLaMA-2 and Mistral were fine-tuned on it. For a fairer comparison, the authors also built MWPBench, which combines ten datasets from primary school to college level and evaluates them all the same way: zero-shot, greedy decoding and accuracy. MathScale-7B was the strongest model of its size on every MWPBench dataset. This suggests that generating data from concepts scales better than rewriting existing questions.

## Key points
- About 2K topics and 8K knowledge points were extracted from the MWPBench training split; concepts that appeared only once were removed. The graph has three edge types: topic–topic, topic–knowledge point and knowledge point–knowledge point.
- On MWPBench, MathScale-7B reached 35.0% micro and 37.5% macro average accuracy. That is a relative gain of 42.9% and 43.7% over the best models of the same size.
- MWPBench adds a new set called CollegeMath, built from nine textbooks with permissive licences (1,281 training and 2,818 test items). Chinese datasets (Math23k, Ape210k, Gaokao) were translated into English, and multiple-choice items were turned into open-answer word problems.
- GPT-4 judged 26% of a 5K sample of GPT-3.5 solutions to be wrong. Replacing those solutions did not improve the fine-tuned model, so the validation step was removed from the pipeline.
- On Fresh-GaokaoMath-2023 (30 Gaokao questions from 2023, unlikely to be in any training data), MathScale-7B scored 30.0. WizardMath-7B scored 13.3, MetaMath-7B 16.6, GPT-3.5-Turbo 40.0 and GPT-4 43.3.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [➗ Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/) · 189 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 321 works · [➗ Applied Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/applied-mathematics/) · 52 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Zhengyang Tang, Xingxing Zhang, Benyou Wang and Furu Wei. 2024. MathScale: Scaling Instruction Tuning for Mathematical Reasoning. arXiv:2403.02884.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{tang2024mathscale,
  title = {{MathScale: Scaling Instruction Tuning for Mathematical Reasoning}},
  author = {Zhengyang Tang, Xingxing Zhang, Benyou Wang and Furu Wei},
  year = {2024},
  doi = {10.48550/arxiv.2403.02884},
  eprint = {2403.02884},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.02884}
}
```

- [← previous Large Language Model Instruction Following: A Survey of Progresses and Challenges](https://md.research.elamur.ai/free/en/papers/lou-2024-large-language-model-instruction-following/)
- [next → Mitigating the Alignment Tax of RLHF](https://md.research.elamur.ai/free/en/papers/lin-2024-mitigating-alignment-tax-rlhf/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

32 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Chen, W., Ma, X., Wang, X., and Cohen, W. W. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022. [find](https://scholar.google.com/scholar?q=Chen%2C+W%2E%2C+Ma%2C+X%2E%2C+Wang%2C+X%2E%2C+and+Cohen%2C+W%2E+W%2E+Program+of+thoughts+prompting%3A+Disentangling+computation+from+reasoning+for+numerical+reasoning+tasks%2E+arXiv+preprint+arXiv%3A2211%2E12588%2C+2022%2E)
2. Chern, E., Zou, H., Li, X., Hu, J., Feng, K., Li, J., and Liu, P. Generative ai for math: Abel. [https://github.com/GAIR-NLP/abel](https://github.com/GAIR-NLP/abel), 2023. [↗](https://github.com/GAIR-NLP/abel)
3. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
4. Corral, M. CORRAL’S VECTOR CALCULUS. 2008. [find](https://scholar.google.com/scholar?q=Corral%2C+M%2E+CORRAL%E2%80%99S+VECTOR+CALCULUS%2E+2008%2E)
5. Evans, M. J. and Rosenthal, J. S. Probability and statistics: The science of uncertainty. Macmillan, 2004. [find](https://scholar.google.com/scholar?q=Evans%2C+M%2E+J%2E+and+Rosenthal%2C+J%2E+S%2E+Probability+and+statistics%3A+The+science+of+uncertainty%2E+Macmillan%2C+2004%2E)
6. Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. PAL: Program-aided language models. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of P… [find](https://scholar.google.com/scholar?q=Gao%2C+L%2E%2C+Madaan%2C+A%2E%2C+Zhou%2C+S%2E%2C+Alon%2C+U%2E%2C+Liu%2C+P%2E%2C+Yang%2C+Y%2E%2C+Callan%2C+J%2E%2C+and+Neubig%2C+G%2E+PAL%3A+Program%2Daided+language+models%2E+In+Krause%2C+A%2E%2C+Brunskill%2C+E%2E%2C+Cho%2C+K%2E%2C+Engelhardt%2C+B%2E%2C+Sabato%2C+S%2E%2C+and+Scarle)
7. Gou, Z., Shao, Z., Gong, Y., Yang, Y., Huang, M., Duan, N., Chen, W., et al. [Tora: A tool-integrated reasoning agent for mathematical problem solving](https://md.research.elamur.ai/free/en/papers/gou-2023-tora-tool-integrated-reasoning-agent/). arXiv preprint arXiv:2309.17452, 2023.
8. Grinstead, C. M. and Snell, J. L. Grinstead and Snell’s introduction to probability. Chance Project, 2006. [find](https://scholar.google.com/scholar?q=Grinstead%2C+C%2E+M%2E+and+Snell%2C+J%2E+L%2E+Grinstead+and+Snell%E2%80%99s+introduction+to+probability%2E+Chance+Project%2C+2006%2E)
9. Guichard, D. Calculus. 2009. [find](https://scholar.google.com/scholar?q=Guichard%2C+D%2E+Calculus%2E+2009%2E)
10. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874, 2021.
11. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E%2C+Sablayrolles%2C+A%2E%2C+Mensch%2C+A%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2E+S%2E%2C+Casas%2C+D%2E+d%2E+l%2E%2C+Bressand%2C+F%2E%2C+Lengyel%2C+G%2E%2C+Lample%2C+G%2E%2C+Saulnier%2C+L%2E%2C+et+al%2E+Mistral+7b%2E+arXiv+preprint+arXiv%3A2310%2E06825%2C+2023%2E)
12. Kim, Y. and Rush, A. M. [Sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/). In Su, J., Duh, K., and Carreras, X. (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1317–1327, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1…
13. Kuttler, K. and Farah, I. A First Course in Linear Algebra, 2017A version (Lyryx). Lyryx, 2017. [find](https://scholar.google.com/scholar?q=Kuttler%2C+K%2E+and+Farah%2C+I%2E+A+First+Course+in+Linear+Algebra%2C+2017A+version+%28Lyryx%29%2E+Lyryx%2C+2017%2E)
14. Luo, H., Sun, Q., Xu, C., Zhao, P., Lou, J., Tao, C., Geng, X., Lin, Q., Chen, S., and Zhang, D. [Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/). arXiv preprint arXiv:2308.09583, 2023.
15. Selinger, P. Matrix theory and linear algebra, 2018. URL [https://www.mathstat.dal.ca/˜selinger/linear-algebra/](https://www.mathstat.dal.ca/˜selinger/linear-algebra/). An introduction to linear algebra for first or second year university students. Licensed under Creative Commons CC BY 4.0 License. Last updated on October 26, 2018. [↗](https://www.mathstat.dal.ca/˜selinger/linear-algebra/)
16. Stitz, C. and Zeager, J. Precalculus. Stitz Zeager Open Source Mathematics, 2013. [find](https://scholar.google.com/scholar?q=Stitz%2C+C%2E+and+Zeager%2C+J%2E+Precalculus%2E+Stitz+Zeager+Open+Source+Mathematics%2C+2013%2E)
17. TAL. Tal-scq5k, 2023. URL [https://github.com/math-eval/TAL-SCQ5K](https://github.com/math-eval/TAL-SCQ5K). GitHub repository. [↗](https://github.com/math-eval/TAL-SCQ5K)
18. Tall, D. How humans learn to think mathematically: Exploring the three worlds of mathematics. Cambridge University Press, 2013. [find](https://scholar.google.com/scholar?q=Tall%2C+D%2E+How+humans+learn+to+think+mathematically%3A+Exploring+the+three+worlds+of+mathematics%2E+Cambridge+University+Press%2C+2013%2E)
19. Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
20. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
21. Trench, W. F. Elementary Differential Equations. Brooks/-Cole Thomson Learning, San Antonio, Texas, USA, 2001. URL [http://ramanujan.math.trinity.edu/wtrench/texts/TRENCH_DIFF_EQNS_I.PDF](http://ramanujan.math.trinity.edu/wtrench/texts/TRENCH_DIFF_EQNS_I.PDF). Free Edition 1.01 (December 2013). [↗](http://ramanujan.math.trinity.edu/wtrench/texts/TRENCH_DIFF_EQNS_I.PDF)
22. Wallace, T. Beginning and intermediate algebra. 2010. [find](https://scholar.google.com/scholar?q=Wallace%2C+T%2E+Beginning+and+intermediate+algebra%2E+2010%2E)
23. Wang, Y., Liu, X., and Shi, S. Deep neural solver for math word problems. In Proceedings of the 2017 conference on empirical methods in natural language processing, pp. 845–854, 2017. [find](https://scholar.google.com/scholar?q=Wang%2C+Y%2E%2C+Liu%2C+X%2E%2C+and+Shi%2C+S%2E+Deep+neural+solver+for+math+word+problems%2E+In+Proceedings+of+the+2017+conference+on+empirical+methods+in+natural+language+processing%2C+pp%2E+845%E2%80%93854%2C+2017%2E)
24. Wang, Y., Ivison, H., Dasigi, P., Hessel, J., Khot, T., Chandu, K. R., Wadden, D., MacMillan, K., Smith, N. A., Beltagy, I., et al. [How far can camels go? exploring the state of instruction tuning on open resources](https://md.research.elamur.ai/free/en/papers/wang-2023-far-camels-go-exploring-state/). arXiv preprint arXiv:2306.04751, 2023.
25. Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). arXiv preprint arXiv:2109.01652, 2021.
26. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35: 24824–24837, 2022.
27. Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.[12244, 2023](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/).
28. Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). arXiv preprint arXiv:2309.12284, 2023.
29. Yue, X., Qu, X., Zhang, G., Fu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. [Mammoth: Building math generalist models through hybrid instruction tuning](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/). arXiv preprint arXiv:2309.05653, 2023.
30. Zhang, X., Li, C., Zong, Y., Ying, Z., He, L., and Qiu, X. [Evaluating the performance of large language models on gaokao benchmark](https://md.research.elamur.ai/free/en/papers/zhang-2023-evaluating-performance-large-language/). 2023.
31. Zhao, W., Shang, M., Liu, Y., Wang, L., and Liu, J. Ape210k: A large-scale and template-rich dataset of math word problems. arXiv preprint arXiv:2009.11506, 2020. [find](https://scholar.google.com/scholar?q=Zhao%2C+W%2E%2C+Shang%2C+M%2E%2C+Liu%2C+Y%2E%2C+Wang%2C+L%2E%2C+and+Liu%2C+J%2E+Ape210k%3A+A+large%2Dscale+and+template%2Drich+dataset+of+math+word+problems%2E+arXiv+preprint+arXiv%3A2009%2E11506%2C+2020%2E)
32. Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., and Duan, N. [Agieval: A human-centric benchmark for evaluating foundation models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/). arXiv preprint arXiv:2304.06364, 2023.

*Updated 2 November 2026*