---
title: "MAmmoTH2: Scaling Instructions from the Web — Yue 2024"
description: "Instruction tuning makes LLMs better at reasoning, but good instruction data is hard to get."
authors: ["Xiang Yue", "Tuney Zheng", "Ge Zhang", "Wenhu Chen"]
year: 2024
venue: "arXiv:2405.03548"
source: "https://arxiv.org/abs/2405.03548"
doi: "10.48550/arxiv.2405.03548"
arxiv: "2405.03548"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yue-2024-mammoth2-scaling-instructions-web/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yue-2024-mammoth2-scaling-instructions-web/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / MAmmoTH2: Scaling Instructions from the Web

# MAmmoTH2: Scaling Instructions from the Web

**Authors:** [Xiang Yue](https://md.research.elamur.ai/free/en/authors/xiang-yue/), Tuney Zheng, [Ge Zhang](https://md.research.elamur.ai/free/en/authors/ge-zhang/), [Wenhu Chen](https://md.research.elamur.ai/free/en/authors/wenhu-chen/) · **Year:** 2024 · **Venue:** arXiv:2405.03548 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2405.03548) · [PDF](https://arxiv.org/pdf/2405.03548) · [DOI 10.48550/arxiv.2405.03548](https://doi.org/10.48550/arxiv.2405.03548) · [arXiv 2405.03548](https://arxiv.org/abs/2405.03548) · [OpenAlex](https://openalex.org/W4415796482)

**Citations:** 2 citations (openalex-snapshot · 2026-10-14) · [92](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Instruction tuning makes LLMs better at reasoning, but good instruction data is hard to get. Human annotation is costly and usually small in scale. Data distilled from GPT-4 tends to be narrow and can contain hallucinations. The authors argue that web pre-training corpora already contain plenty of question–answer material, such as exam sites, educational pages and forums. They build a three-stage pipeline to mine it from Common Crawl:
1. **Recall:** a fastText classifier, seeded with crawled quiz-site problems and retrained after GPT-3.5/GPT-4 domain filtering, finds about 18M relevant documents.
2. **Extract:** after rule-based HTML cleanup, Qwen-72B pulls out candidate Q-A pairs.
3. **Refine:** Mixtral-8×7B and Qwen-72B rewrite the pairs, fixing formatting and adding missing step-by-step solutions.

The result is WebInstruct, about 10M pairs built without crowdsourcing or GPT-4 distillation. Fine-tuning Mistral-7B, Llama3-8B, Mixtral-8×7B and Yi-34B on it produces the MAmmoTH2 models. They improve strongly on held-out reasoning benchmarks even though they were trained on no in-domain data. This makes web mining a cheap, scalable source of instruction data.

## Key points
- MAmmoTH2-7B (Mistral) raises MATH from 11.2% to 36.7% and GSM8K from 36% to 68.4%. Across seven benchmarks (TheoremQA, GSM8K, MATH, ARC-C, MMLU-STEM, GPQA, BBH), it gains about 14 points on average; Yi-34B gains 5.8.
- MAmmoTH2-Plus adds tuning on OpenHermes2.5, Code-Feedback and Math-plus. The 8B-Plus model beats Llama-3-8B-Instruct by about 6 points on reasoning, though both used 10M examples and only Llama's were human-annotated. The 8×7B-Plus model matches Qwen-1.5-110B on reasoning with 13B active parameters and leads on AlpacaEval 2.0 and Arena Hard.
- Ablations:
  - SFT loss works better than LM loss.
  - Merging data refined by both LLMs beats either one alone.
  - Data from education sites beats data from forums.
  - A mix of domains generalizes better than math alone.
- Humans checked 50 refined samples: 78% were improved, and 10% gained hallucinations.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/)

## Cited by

Cited by

92 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 82 not yet.

- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2025-towards-reasoning-era-survey-long/)
- [Process Reinforcement through Implicit Rewards](https://md.research.elamur.ai/free/en/papers/cui-2025-process-reinforcement-through-implicit-rewards/)
- [OpenThoughts: Data Recipes for Reasoning Models](https://md.research.elamur.ai/free/en/papers/guha-2025-openthoughts-data-recipes-reasoning-models/)
- [DataComp-LM: In search of the next generation of training sets for language models](https://md.research.elamur.ai/free/en/papers/li-2024-datacomp-lm-search-next-generation/)
- [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/)
- [Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing](https://md.research.elamur.ai/free/en/papers/xu-2024-magpie-alignment-data-synthesis-scratch/)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://md.research.elamur.ai/free/en/papers/yang-2024-qwen2-5-math-technical-report/)
- [LIMO: Less is More for Reasoning](https://md.research.elamur.ai/free/en/papers/ye-2025-limo-less-more-reasoning/)
- [Demystifying Long Chain-of-Thought Reasoning in LLMs](https://md.research.elamur.ai/free/en/papers/yeo-2025-demystifying-long-chain-thought-reasoning/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{yue2024mammoth2,
  title = {{MAmmoTH2: Scaling Instructions from the Web}},
  author = {Yue, Xiang and Zheng, Tuney and Zhang, Ge and Chen, Wenhu},
  year = {2024},
  doi = {10.48550/arxiv.2405.03548},
  eprint = {2405.03548},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.03548}
}
```

- [← previous Major Entity Identification: A Generalizable Alternative to Coreference Resolution](https://md.research.elamur.ai/free/en/papers/manikantan-2024-major-entity-identification-generalizable-alternative/)
- [next → Many-Shot In-Context Learning](https://md.research.elamur.ai/free/en/papers/agarwal-2024-many-shot-context-learning/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

69 references, 36 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. ArXiv preprint, abs/2303.08774, 2023. URL https://arxiv.org/abs/2303.08774. [↗](https://arxiv.org/abs/2303.08774) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
2. A. Amini, S. Gabriel, S. Lin, R. Koncel-Kedziorski, Y. Choi, and H. Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag… [find](https://scholar.google.com/scholar?q=A%2E+Amini%2C+S%2E+Gabriel%2C+S%2E+Lin%2C+R%2E+Koncel%2DKedziorski%2C+Y%2E+Choi%2C+and+H%2E+Hajishirzi%2E+MathQA%3A+Towards+interpretable+math+word+problem+solving+with+operation%2Dbased+formalisms%2E+In+Proceedings+of+the+2019+Conf) [our card](https://md.research.elamur.ai/free/en/papers/amini-2019-mathqa-interpretable-math-word-problem/)
3. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. ArXiv preprint, abs/2108.07732, 2021. URL https://arxiv.org/abs/2108.07732. [↗](https://arxiv.org/abs/2108.07732) [our card](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/)
4. Z. Azerbayev, H. Schoelkopf, K. Paster, M. Dos Santos, S. M. McAleer, A. Q. Jiang, J. Deng, S. Biderman, and S. Welleck. Llemma: An open language model for mathematics. In The Twelfth International Conference on Learning Representations, 2023. URL [https://openreview.net/forum?id=4WnqRR915j](https://openreview.net/forum?id=4WnqRR915j). [↗](https://openreview.net/forum?id=4WnqRR915j) [our card](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/)
5. J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen technical report. ArXiv preprint, abs/2309.16609, 2023. URL https://arxiv.org/abs/2309.16609. [↗](https://arxiv.org/abs/2309.16609) [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
6. L. Ben Allal, A. Lozhkov, G. Penedo, T. Wolf, and L. von Werra. Cosmopedia, 2024. URL [https://huggingface.co/datasets/HuggingFaceTB/cosmopedia](https://huggingface.co/datasets/HuggingFaceTB/cosmopedia). [↗](https://huggingface.co/datasets/HuggingFaceTB/cosmopedia)
7. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Ber… [find](https://scholar.google.com/scholar?q=T%2E+B%2E+Brown%2C+B%2E+Mann%2C+N%2E+Ryder%2C+M%2E+Subbiah%2C+J%2E+Kaplan%2C+P%2E+Dhariwal%2C+A%2E+Neelakantan%2C+P%2E+Shyam%2C+G%2E+Sastry%2C+A%2E+Askell%2C+S%2E+Agarwal%2C+A%2E+Herbert%2DVoss%2C+G%2E+Krueger%2C+T%2E+Henighan%2C+R%2E+Child%2C+A%2E+Ramesh%2C+D%2E+M%2E+Zie)
8. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. ArXiv preprint, abs/2107.03374, 2021. URL https://arxiv.org/abs/2107.03374. [↗](https://arxiv.org/abs/2107.03374) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
9. W. Chen, X. Ma, X. Wang, and W. W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023a. URL [https://openreview.net/forum?id=YfZ4ZPt8zd](https://openreview.net/forum?id=YfZ4ZPt8zd). [↗](https://openreview.net/forum?id=YfZ4ZPt8zd)
10. W. Chen, M. Yin, M. Ku, P. Lu, Y. Wan, X. Ma, J. Xu, X. Wang, and T. Xia. Theoremqa: A theorem-driven question answering dataset. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023b. URL https://aclanthology.org/2023.emnlp-main.489/. [↗](https://aclanthology.org/2023.emnlp-main.489/) [our card](https://md.research.elamur.ai/free/en/papers/chen-2023-theoremqa-theorem-driven-question-answering/)
11. H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024. URL [https://arxiv.org/pdf/2210.11416](https://arxiv.org/pdf/2210.11416). [↗](https://arxiv.org/pdf/2210.11416)
12. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv preprint, abs/1803.05457, 2018. URL https://arxiv.org/abs/1803.05457. [↗](https://arxiv.org/abs/1803.05457)
13. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. ArXiv preprint, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.14168. [↗](https://arxiv.org/abs/2110.14168) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
14. T. Computer. Redpajama: an open dataset for training large language models, 2023. URL [https://github.com/togethercomputer/RedPajama-Data](https://github.com/togethercomputer/RedPajama-Data). [↗](https://github.com/togethercomputer/RedPajama-Data) [our card](https://md.research.elamur.ai/free/en/papers/weber-2024-redpajama-open-dataset-training-large/)
15. L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764–10799. PMLR, 2023. URL [https://arxiv.org/pdf/2211.10435](https://arxiv.org/pdf/2211.10435). [↗](https://arxiv.org/pdf/2211.10435)
16. Z. Gou, Z. Shao, Y. Gong, Y. Yang, M. Huang, N. Duan, W. Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. ArXiv preprint, abs/2309.17452, 2023. URL https://arxiv.org/abs/2309.17452. [↗](https://arxiv.org/abs/2309.17452) [our card](https://md.research.elamur.ai/free/en/papers/gou-2023-tora-tool-integrated-reasoning-agent/)
17. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021a. URL [https://openreview.net/forum?id=d7KBjmI3Gm…](https://openreview.net/forum?id=d7KBjmI3Gm…) [↗](https://openreview.net/forum?id=d7KBjmI3Gm…) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
18. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021b. URL [https://openreview.net/…](https://openreview.net/…) [↗](https://openreview.net/…) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/)
19. H. Ivison, Y. Wang, V. Pyatkin, N. Lambert, M. Peters, P. Dasigi, J. Jang, D. Wadden, N. A. Smith, I. Beltagy, et al. Camels in a changing climate: Enhancing lm adaptation with tulu 2. ArXiv preprint, abs/2311.10702, 2023. URL [https://arxiv.org/pdf/2311.10702](https://arxiv.org/pdf/2311.10702). [↗](https://arxiv.org/pdf/2311.10702) [our card](https://md.research.elamur.ai/free/en/papers/ivison-2023-camels-changing-climate-enhancing-lm/)
20. A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. ArXiv preprint, abs/2310.06825, 2023. URL https://arxiv.org/abs/2310.06825. [↗](https://arxiv.org/abs/2310.06825)
21. A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al. Mixtral of experts. ArXiv preprint, abs/2401.04088, 2024. URL https://arxiv.org/abs/2401.04088. [↗](https://arxiv.org/abs/2401.04088)
22. A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jégou, and T. Mikolov. Fasttext. zip: Compressing text classification models. ArXiv preprint, abs/1612.03651, 2016. URL https://arxiv.org/abs/1612.03651. [↗](https://arxiv.org/abs/1612.03651) [our card](https://md.research.elamur.ai/free/en/papers/joulin-2016-fasttext-zip-compressing-text-classification/)
23. A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843–3857, 2022. URL [https://openreview.net/forum?id=…](https://openreview.net/forum?id=…) [↗](https://openreview.net/forum?id=…) [our card](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/)
24. C. Li, W. Wang, J. Hu, Y. Wei, N. Zheng, H. Hu, Z. Zhang, and H. Peng. Common 7b language models already possess strong math capabilities. ArXiv preprint, abs/2403.04706, 2024a. URL https://arxiv.org/abs/2403.04706. [↗](https://arxiv.org/abs/2403.04706)
25. T. Li, W.-L. Chiang, L. D. Evan Frick, B. Zhu, J. E. Gonzalez, and I. Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024b. URL [https://lmsys.org/blog/2024-04-19-arena-hard/](https://lmsys.org/blog/2024-04-19-arena-hard/). [↗](https://lmsys.org/blog/2024-04-19-arena-hard/)
26. X. Li, T. Zhang, Y. Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval), 2023. [↗](https://github.com/tatsu-lab/alpaca_eval)
27. Z. Lin, Z. Gou, T. Liang, R. Luo, H. Liu, and Y. Yang. Criticbench: Benchmarking llms for critique-correct reasoning. ArXiv preprint, abs/2402.14809, 2024. URL https://arxiv.org/abs/2402.14809. [↗](https://arxiv.org/abs/2402.14809)
28. J. Liu, C. S. Xia, Y. Wang, and L. Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024. URL [https://openreview.net/pdf?id=1qvx610Cu7](https://openreview.net/pdf?id=1qvx610Cu7). [↗](https://openreview.net/pdf?id=1qvx610Cu7) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/)
29. H. Luo, Q. Sun, C. Xu, P. Zhao, J. Lou, C. Tao, X. Geng, Q. Lin, S. Chen, and D. Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. ArXiv preprint, abs/2308.09583, 2023. URL https://arxiv.org/abs/2308.09583. [↗](https://arxiv.org/abs/2308.09583) [our card](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/)
30. Meta. Introducing meta llama 3: The most capable openly available llm to date. [https://ai.meta.com/blog/meta-llama-3/](https://ai.meta.com/blog/meta-llama-3/), April 2024. [↗](https://ai.meta.com/blog/meta-llama-3/)
31. A. Mitra, H. Khanpour, C. Rosset, and A. Awadallah. Orca-math: Unlocking the potential of slms in grade school math. ArXiv preprint, abs/2402.14830, 2024. URL https://arxiv.org/abs/2402.14830. [↗](https://arxiv.org/abs/2402.14830) [our card](https://md.research.elamur.ai/free/en/papers/mitra-2024-orca-math-unlocking-potential-slms/)
32. M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, et al. Show your work: Scratchpads for intermediate computation with language models. In Deep Learning for Code Workshop, 2022. URL [https://openreview.net/forum?id=iedYJm92o0a](https://openreview.net/forum?id=iedYJm92o0a). [↗](https://openreview.net/forum?id=iedYJm92o0a) [our card](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/)
33. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730– 27744, 2022. URL [https://proceedings.neurips.cc/paper_fi…](https://proceedings.neurips.cc/paper_fi…) [↗](https://proceedings.neurips.cc/paper_fi…)
34. K. Paster, M. Dos Santos, Z. Azerbayev, and J. Ba. Openwebmath: An open dataset of high-quality mathematical web text. In The Twelfth International Conference on Learning Representations, 2023. URL [https://openreview.net/pdf?id=jKHmjlpViu](https://openreview.net/pdf?id=jKHmjlpViu). [↗](https://openreview.net/pdf?id=jKHmjlpViu) [our card](https://md.research.elamur.ai/free/en/papers/paster-2023-openwebmath-open-dataset-high-quality/)
35. B. Peng, C. Li, P. He, M. Galley, and J. Gao. Instruction tuning with gpt-4. ArXiv preprint, abs/2304.03277, 2023. URL https://arxiv.org/abs/2304.03277. [↗](https://arxiv.org/abs/2304.03277) [our card](https://md.research.elamur.ai/free/en/papers/peng-2023-instruction-tuning-gpt-4/)
36. J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In R. Gupta, Y. Liu, J. Tang, and B. A. Prakash, editors, KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Ev… — [doi:10.1145/3394486.3406703](https://doi.org/10.1145/3394486.3406703) [find](https://scholar.google.com/scholar?q=J%2E+Rasley%2C+S%2E+Rajbhandari%2C+O%2E+Ruwase%2C+and+Y%2E+He%2E+Deepspeed%3A+System+optimizations+enable+training+deep+learning+models+with+over+100+billion+parameters%2E+In+R%2E+Gupta%2C+Y%2E+Liu%2C+J%2E+Tang%2C+and+B%2E+A%2E+Prakash%2C)
37. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. ArXiv preprint, abs/2311.12022, 2023. URL https://arxiv.org/abs/2311.12022. [↗](https://arxiv.org/abs/2311.12022) [our card](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/)
38. V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, A. Raja, M. Dey, M. S. Bari, C. Xu, U. Thakker, S. S. Sharma, E. Szczechla, T. Kim, G. Chhablani, N. V. Nayak, D. Datta, J. Chang, M. T. Jiang, H. Wang, M. Manica, S. Shen, Z. X. Yong, H. Pandey, R. Bawden,… [find](https://scholar.google.com/scholar?q=V%2E+Sanh%2C+A%2E+Webson%2C+C%2E+Raffel%2C+S%2E+H%2E+Bach%2C+L%2E+Sutawika%2C+Z%2E+Alyafeai%2C+A%2E+Chaffin%2C+A%2E+Stiegler%2C+A%2E+Raja%2C+M%2E+Dey%2C+M%2E+S%2E+Bari%2C+C%2E+Xu%2C+U%2E+Thakker%2C+S%2E+S%2E+Sharma%2C+E%2E+Szczechla%2C+T%2E+Kim%2C+G%2E+Chhablani%2C+N%2E+V%2E+Na) [our card](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
39. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. Li, Y. Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv preprint, abs/2402.03300, 2024. URL https://arxiv.org/abs/2402.03300. [↗](https://arxiv.org/abs/2402.03300) [our card](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/)
40. A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. URL [https://openreview.net…](https://openreview.net…) [↗](https://openreview.net…) [our card](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
41. L. Sun, Y. Han, Z. Zhao, D. Ma, Z. Shen, B. Chen, L. Chen, and K. Yu. Scieval: A multi-level large language model evaluation benchmark for scientific research. ArXiv preprint, abs/2308.13149, 2023. URL https://arxiv.org/abs/2308.13149. [↗](https://arxiv.org/abs/2308.13149)
42. M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, , and J. Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. ArXiv preprint, abs/2210.09261, 2022. URL https://arxiv.org/abs/2210.09261. [↗](https://arxiv.org/abs/2210.09261) [our card](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/)
43. Z. Tang, X. Zhang, B. Wan, and F. Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. ArXiv preprint, abs/2403.02884, 2024. URL https://arxiv.org/abs/2403.02884. [↗](https://arxiv.org/abs/2403.02884)
44. R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
45. R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, and R. Stojnic. Galactica: A large language model for science. ArXiv preprint, abs/2211.09085, 2022. URL https://arxiv.org/abs/2211.09085. [↗](https://arxiv.org/abs/2211.09085) [our card](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/)
46. G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: a family of highly capable multimodal models. ArXiv preprint, abs/2312.11805, 2023. URL https://arxiv.org/abs/2312.11805. [↗](https://arxiv.org/abs/2312.11805) [our card](https://md.research.elamur.ai/free/en/papers/gemini-2023-team-gemini-family-highly-capable-multimodal/)
47. Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023. URL [https://huggingface.co/datasets/teknium/OpenHermes-2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5). [↗](https://huggingface.co/datasets/teknium/OpenHermes-2.5)
48. TIGER-Lab. Mmlu professional dataset. Hugging Face Dataset Hub, 2024. URL [https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro). [↗](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro)
49. S. Toshniwal, I. Moshkov, S. Narenthiran, D. Gitman, F. Jia, and I. Gitman. Openmathinstruct-1: A 1.8 million math instruction tuning dataset. ArXiv preprint, abs/2402.10176, 2024. URL https://arxiv.org/abs/2402.10176. [↗](https://arxiv.org/abs/2402.10176)
50. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint, abs/2302.13971, 2023a. URL https://arxiv.org/abs/2302.13971. [↗](https://arxiv.org/abs/2302.13971)
51. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, abs/2307.09288, 2023b. URL https://arxiv.org/abs/2307.09288. [↗](https://arxiv.org/abs/2307.09288)
52. X. Wang, Z. Hu, P. Lu, Y. Zhu, J. Zhang, S. Subramaniam, A. Loomba, S. Zhang, Y. Sun, and W. Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. In The 3rd Workshop on Mathematical Reasoning and AI at NeurIPS’23, 2023a. URL [https://openreview.net/f…](https://openreview.net/f…) [↗](https://openreview.net/f…) [our card](https://md.research.elamur.ai/free/en/papers/wang-2023-scibench-evaluating-college-level-scientific/)
53. Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran, A. Arunkumar, D. Stap, E. Pathak, G. Karamanolakis, H. Lai, I. Purohit, I. Mondal, J. Anderson, K. Kuznia, K. Doshi, K. K. Pal, M. Patel, M. Moradshahi, M. Parmar, M. Purohit, N. Varshney, P. R. Kaz… [find](https://scholar.google.com/scholar?q=Y%2E+Wang%2C+S%2E+Mishra%2C+P%2E+Alipoormolabashi%2C+Y%2E+Kordi%2C+A%2E+Mirzaei%2C+A%2E+Naik%2C+A%2E+Ashok%2C+A%2E+S%2E+Dhanasekaran%2C+A%2E+Arunkumar%2C+D%2E+Stap%2C+E%2E+Pathak%2C+G%2E+Karamanolakis%2C+H%2E+Lai%2C+I%2E+Purohit%2C+I%2E+Mondal%2C+J%2E+Anderson%2C+K%2E)
54. Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484– 13508, 2023b.… [find](https://scholar.google.com/scholar?q=Y%2E+Wang%2C+Y%2E+Kordi%2C+S%2E+Mishra%2C+A%2E+Liu%2C+N%2E+A%2E+Smith%2C+D%2E+Khashabi%2C+and+H%2E+Hajishirzi%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+In+Proceedings+of+the+61st+Annual+Meeting+o)
55. Z. Wang, R. Xia, and P. Liu. Generative ai for math: Part i–mathpile: A billion-token-scale pretraining corpus for math. ArXiv preprint, abs/2312.17120, 2023c. URL https://arxiv.org/abs/2312.17120. [↗](https://arxiv.org/abs/2312.17120)
56. J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022a. URL [https://openreview.net/forum?id=gEZ…](https://openreview.net/forum?id=gEZ…) [↗](https://openreview.net/forum?id=gEZ…)
57. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022b. URL [https://openreview.net/pdf?id=_VjQlMeSB_J](https://openreview.net/pdf?id=_VjQlMeSB_J). [↗](https://openreview.net/pdf?id=_VjQlMeSB_J) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
58. C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, and D. Jiang. Wizardlm: Empowering large language models to follow complex instructions. ArXiv preprint, abs/2304.12244, 2023. URL https://arxiv.org/abs/2304.12244. [↗](https://arxiv.org/abs/2304.12244) [our card](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/)
59. H. Ying, S. Zhang, L. Li, Z. Zhou, Y. Shao, Z. Fei, Y. Ma, J. Hong, K. Liu, Z. Wang, et al. Internlm-math: Open math large language models toward verifiable reasoning. ArXiv preprint, abs/2402.06332, 2024. URL https://arxiv.org/abs/2402.06332. [↗](https://arxiv.org/abs/2402.06332)
60. A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang, et al. Yi: Open foundation models by 01. ai. ArXiv preprint, abs/2403.04652, 2024. URL https://arxiv.org/abs/2403.04652. [↗](https://arxiv.org/abs/2403.04652) [our card](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/)
61. L. Yu, W. Jiang, H. Shi, Y. Jincheng, Z. Liu, Y. Zhang, J. Kwok, Z. Li, A. Weller, and W. Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2023. URL [https://openreview.net/forum?id=N8N0hgNDRt](https://openreview.net/forum?id=N8N0hgNDRt). [↗](https://openreview.net/forum?id=N8N0hgNDRt) [our card](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/)
62. L. Yuan, G. Cui, H. Wang, N. Ding, X. Wang, J. Deng, B. Shan, H. Chen, R. Xie, Y. Lin, et al. Advancing llm reasoning generalists with preference trees. ArXiv preprint, abs/2404.02078, 2024. URL https://arxiv.org/abs/2404.02078. [↗](https://arxiv.org/abs/2404.02078)
63. X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. ArXiv preprint, abs/2311.16502, 2023a. URL https://arxiv.org/abs/2311.16502. [↗](https://arxiv.org/abs/2311.16502) [our card](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/)
64. X. Yue, X. Qu, G. Zhang, Y. Fu, W. Huang, H. Sun, Y. Su, and W. Chen. Mammoth: Building math generalist models through hybrid instruction tuning. In The Twelfth International Conference on Learning Representations, 2023b. URL [https://openreview.net/forum?id=yLClGs770I](https://openreview.net/forum?id=yLClGs770I). [↗](https://openreview.net/forum?id=yLClGs770I) [our card](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/)
65. Y. Zhang, Y. Luo, Y. Yuan, and A. C.-C. Yao. Automathtext: Autonomous data selection with language models for mathematical texts. ArXiv preprint, abs/2402.07625, 2024. URL https://arxiv.org/abs/2402.07625. — [arXiv:2402.07625](https://arxiv.org/abs/2402.07625) [↗](https://arxiv.org/abs/2402.07625)
66. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024a. URL [https://openreview.net/forum?id=uccHPGDlao](https://openreview.net/forum?id=uccHPGDlao). [↗](https://openreview.net/forum?id=uccHPGDlao)
67. T. Zheng, S. Guo, X. Qu, J. Guo, W. Zhang, X. Du, C. Lin, W. Huang, W. Chen, J. Fu, et al. Kun: Answer polishment for chinese self-alignment with instruction back-translation. ArXiv preprint, abs/2401.06477, 2024b. URL https://arxiv.org/abs/2401.06477. [↗](https://arxiv.org/abs/2401.06477)
68. T. Zheng, G. Zhang, T. Shen, X. Liu, B. Y. Lin, J. Fu, W. Chen, and X. Yue. Opencodeinterpreter: Integrating code generation with execution and refinement. ArXiv preprint, abs/2402.14658, 2024c. URL https://arxiv.org/abs/2402.14658. [↗](https://arxiv.org/abs/2402.14658)
69. Y. Zheng, R. Zhang, J. Zhang, Y. Ye, Z. Luo, and Y. Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. ArXiv preprint, abs/2403.13372, 2024d. URL https://arxiv.org/abs/2403.13372. [↗](https://arxiv.org/abs/2403.13372) [our card](https://md.research.elamur.ai/free/en/papers/zheng-2024-llamafactory-unified-efficient-fine-tuning/)