---
title: "Rho-1: Not All Tokens Are What You Need (2024)"
description: "They propose Selective Language Modeling (SLM), used to train the Rho-1 models, in three steps… 123 references, cited by 57 papers here."
authors: ["Zhenghao Lin", "Zhibin Gou", "Yeyun Gong", "Xiao Liu", "Yelong Shen", "Ruochen Xu", "Chen Lin", "Yujiu Yang", "Jian Jiao", "Nan Duan", "Weizhu Chen"]
year: 2024
venue: "arXiv:2404.07965"
source: "https://arxiv.org/abs/2404.07965"
doi: "10.48550/arxiv.2404.07965"
arxiv: "2404.07965"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lin-2024-rho-1-all-tokens-need/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lin-2024-rho-1-all-tokens-need/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / Rho-1: Not All Tokens Are What You Need

# Rho-1: Not All Tokens Are What You Need

**Authors:** [Zhenghao Lin](https://md.research.elamur.ai/free/en/authors/zhenghao-lin/), [Zhibin Gou](https://md.research.elamur.ai/free/en/authors/zhibin-gou/), [Yeyun Gong](https://md.research.elamur.ai/free/en/authors/yeyun-gong/), [Xiao Liu](https://md.research.elamur.ai/free/en/authors/xiao-liu/), [Yelong Shen](https://md.research.elamur.ai/free/en/authors/yelong-shen/), [Ruochen Xu](https://md.research.elamur.ai/free/en/authors/ruochen-xu/), [Chen Lin](https://md.research.elamur.ai/free/en/authors/chen-lin/), [Yujiu Yang](https://md.research.elamur.ai/free/en/authors/yujiu-yang/), [Jian Jiao](https://md.research.elamur.ai/free/en/authors/jian-jiao/), [Nan Duan](https://md.research.elamur.ai/free/en/authors/nan-duan/) and [Weizhu Chen](https://md.research.elamur.ai/free/en/authors/weizhu-chen/) · **Year:** 2024 · **Venue:** arXiv:2404.07965 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2404.07965) · [PDF](https://arxiv.org/pdf/2404.07965) · [DOI 10.48550/arxiv.2404.07965](https://doi.org/10.48550/arxiv.2404.07965) · [arXiv 2404.07965](https://arxiv.org/abs/2404.07965) · [OpenAlex](https://openalex.org/W4394782324)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [57](https://md.research.elamur.ai/free/en/papers/lin-2024-rho-1-all-tokens-need/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Standard LLM pre-training applies the next-token loss uniformly, even though filtered corpora still contain noisy, irrelevant, or already-mastered tokens. The authors tracked per-token loss while continuing to pre-train Tinyllama-1B on OpenWebMath. Only about a quarter of tokens actually improved; most were already learned, and many fluctuated without converging.

They propose Selective Language Modeling (SLM), used to train the Rho-1 models, in three steps:
- train a reference model on curated high-quality data,
- score every token of the pretraining corpus with this reference model,
- backpropagate loss only for the top-k% of tokens, ranked by excess loss (current model loss minus reference loss).

The full sequence is still fed as context.

On 15B OpenWebMath tokens, Rho-1 at 1B and 7B scale beats causal-LM baselines by over 16% on GSM8k/MATH and reaches baseline accuracy 5–10x sooner. On 80B general tokens, it also gains across many benchmarks. The work shows that selecting data at the token level, not just the document level, can significantly improve both data efficiency and final quality.

## Key points
- **Token dynamics:** On a validation set of roughly 320K tokens, 51% stayed low-loss (L→L), 26% decreased (H→L), 11% stayed high (H→H), and 12% unexpectedly rose (L→H).
- **Math results:** After fine-tuning, Rho-1-1B scores 40.6% and Rho-1-7B scores 51.8% on MATH. The 7B model matches DeepSeekMath-7B using 15B tokens instead of 500B. Few-shot gains reach up to 30% across 9 math tasks.
- **General pre-training:** Continual pre-training of Tinyllama-1B on 80B tokens gives a 6.8% average gain over 15 benchmarks, with more than 10% improvement on code and math.
- **Self-referencing:** Without external high-quality data, SLM still adds up to 3.3% while training on about 40% fewer tokens.
- **Selection ratio:** Keeping around 60% of tokens worked best. Selected tokens are mostly math-related, and their loss correlates with downstream accuracy. Published at NeurIPS 2024.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 378 works · 161,630 citations
- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 152 works · 13,166 citations
- [Xiamen University](https://md.research.elamur.ai/free/en/universities/xiamen-university/) — University, Xiamen, China · 24 works · 2,317 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan and Weizhu Chen. 2024. Rho-1: Not All Tokens Are What You Need. arXiv:2404.07965.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{lin2024rho,
  title = {{Rho-1: Not All Tokens Are What You Need}},
  author = {Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan and Weizhu Chen},
  year = {2024},
  doi = {10.48550/arxiv.2404.07965},
  eprint = {2404.07965},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.07965}
}
```

- [← previous NADI 2024: The Fifth Nuanced Arabic Dialect Identification Shared Task](https://md.research.elamur.ai/free/en/papers/mageed-2024-nadi-2024-fifth-nuanced-arabic/)
- [next → Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling](https://md.research.elamur.ai/free/en/papers/ren-2024-samba-simple-hybrid-state-space/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

123 references, 65 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361, 2020.
2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
3. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023.
4. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
5. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. [Ccnet: Extracting high quality monolingual datasets from web crawl data](https://md.research.elamur.ai/free/en/papers/wenzek-2020-ccnet-extracting-high-quality-monolingual/). arXiv preprint arXiv:1911.00359, 2019.
6. Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. [Challenges in detoxifying language models](https://md.research.elamur.ai/free/en/papers/welbl-2021-challenges-detoxifying-language-models/). In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2447–2469, 20…
7. Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Niklas+Muennighoff%2C+Alexander+Rush%2C+Boaz+Barak%2C+Teven+Le+Scao%2C+Nouamane+Tazi%2C+Aleksandra+Piktus%2C+Sampo+Pyysalo%2C+Thomas+Wolf%2C+and+Colin+A+Raffel%2E+Scaling+data%2Dconstrained+language+models%2E+Advances+in+N)
8. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process… [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Maarten+Sap%2C+Ana+Marasovic%CC%81%2C+William+Agnew%2C+Gabriel+Ilharco%2C+Dirk+Groeneveld%2C+Margaret+Mitchell%2C+and+Matt+Gardner%2E+Documenting+large+webtext+corpora%3A+A+case+study+on+the+colossal+clean+cr)
9. Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. arXiv preprint arXiv:2305.[13169, 2023](https://md.research.elamur.ai/free/en/papers/longpre-2023-pretrainer-s-guide-training-data/).
10. Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. [Scaling laws vs model architectures: How does inductive bias influence scaling?](https://md.research.elamur.ai/free/en/papers/tay-2023-scaling-laws-vs-model-architectures/) arXiv preprint arXiv:2207.10551, 2022.
11. Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. Should you mask 15% in masked language modeling? In Andreas Vlachos and Isabelle Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2985–3000, Dubrovnik, C… [find](https://scholar.google.com/scholar?q=Alexander+Wettig%2C+Tianyu+Gao%2C+Zexuan+Zhong%2C+and+Danqi+Chen%2E+Should+you+mask+15%25+in+masked+language+modeling%3F+In+Andreas+Vlachos+and+Isabelle+Augenstein%2C+editors%2C+Proceedings+of+the+17th+Conference+of+)
12. Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learning, 110(3):457–506, 2021. [find](https://scholar.google.com/scholar?q=Eyke+H%C3%BCllermeier+and+Willem+Waegeman%2E+Aleatoric+and+epistemic+uncertainty+in+machine+learning%3A+An+introduction+to+concepts+and+methods%2E+Machine+learning%2C+110%283%29%3A457%E2%80%93506%2C+2021%2E)
13. Longhui Yu, Weisen Jiang, Han Shi, YU Jincheng, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). In ICLR, 2024.
14. Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-point-driven data synthesis with its enhancement on mathematical reasoning. arXiv preprint arXiv:2403.02333, 2024. [find](https://scholar.google.com/scholar?q=Yiming+Huang%2C+Xiao+Liu%2C+Yeyun+Gong%2C+Zhibin+Gou%2C+Yelong+Shen%2C+Nan+Duan%2C+and+Weizhu+Chen%2E+Key%2Dpoint%2Ddriven+data+synthesis+with+its+enhancement+on+mathematical+reasoning%2E+arXiv+preprint+arXiv%3A2403%2E02333%2C)
15. Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. [Mammoth: Building math generalist models through hybrid instruction tuning](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/). In ICLR, 2024.
16. Xinzhe Ni, Yeyun Gong, Zhibin Gou, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Exploring the mystery of influential data for mathematical reasoning, 2024. [find](https://scholar.google.com/scholar?q=Xinzhe+Ni%2C+Yeyun+Gong%2C+Zhibin+Gou%2C+Yelong+Shen%2C+Yujiu+Yang%2C+Nan+Duan%2C+and+Weizhu+Chen%2E+Exploring+the+mystery+of+influential+data+for+mathematical+reasoning%2C+2024%2E)
17. Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. [Camels in a changing climate: Enhancing lm adaptation with tulu 2](https://md.research.elamur.ai/free/en/papers/ivison-2023-camels-changing-climate-enhancing-lm/). arXiv preprint arXiv:2311.10702, 2023.
18. Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023. URL [https://huggingface.co/datasets/teknium/OpenHermes-2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5). [↗](https://huggingface.co/datasets/teknium/OpenHermes-2.5)
19. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.[20050, 2023](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/).
20. Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. [Openwebmath: An open dataset of high-quality mathematical web text](https://md.research.elamur.ai/free/en/papers/paster-2023-openwebmath-open-dataset-high-quality/), 2023.
21. Soboleva Daria, Al-Khateeb Faisal, Myers Robert Steeves Jacob R, Hestness Joel, and Dey Nolan. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. [https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama](https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama), 2023. URL https://huggingface.co… [↗](https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama)
22. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas… [find](https://scholar.google.com/scholar?q=Raymond+Li%2C+Loubna+Ben+Allal%2C+Yangtian+Zi%2C+Niklas+Muennighoff%2C+Denis+Kocetkov%2C+Chenghao+Mou%2C+Marc+Marone%2C+Christopher+Akiki%2C+Jia+Li%2C+Jenny+Chim%2C+Qian+Liu%2C+Evgenii+Zheltonozhskii%2C+Terry+Yue+Zhuo%2C+Thoma)
23. Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. [Tinyllama: An open-source small language model](https://md.research.elamur.ai/free/en/papers/zhang-2024-tinyllama-open-source-small-language/), 2024.
24. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+Mistral+7b)
25. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Thomas+Mesnard%2C+Cassidy+Hardin%2C+Robert+Dadashi%2C+Surya+Bhupatiraju%2C+Shreya+Pathak%2C+Laurent+Sifre%2C+Morgane+Rivi%C3%A8re%2C+Mihir+Sanjay+Kale%2C+Juliette+Love%2C+et+al%2E+Gemma%3A+Open+models+based+on+gemin)
26. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609, 2023.
27. Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. [Textbooks are all you need ii: phi-1.5 technical report](https://md.research.elamur.ai/free/en/papers/li-2023-textbooks-all-need-ii-phi/), 2023b.
28. DeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. URL [https://github.com/deepseek-ai/DeepSeek-LLM](https://github.com/deepseek-ai/DeepSeek-LLM). [↗](https://github.com/deepseek-ai/DeepSeek-LLM)
29. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Y Wu, and Daya Guo. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300, 2024.
30. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950, 2023.
31. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). Advances in Neural Information Processing Systems, 35:3843–3857, 2022.
32. Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. [Llemma: An open language model for mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/). arXiv preprint arXiv:2310.10631, 2023.
33. Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, et al. [Internlm-math: Open math large language models toward verifiable reasoning](https://md.research.elamur.ai/free/en/papers/ying-2024-internlm-math-open-math-large/). arXiv preprint arXiv:2402.06332, 2024.
34. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. [Tora: A tool-integrated reasoning agent for mathematical problem solving](https://md.research.elamur.ai/free/en/papers/gou-2023-tora-tool-integrated-reasoning-agent/). In ICLR, 2024.
35. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Ani… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Alain+Le+Noac%E2%80%99h%2C+Haonan+Li%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Chris+Oci)
36. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
37. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In NIPS, volume 35, pages 24824–24837, 2022a.
38. Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Alexandros G. Dimakis, Gabriel Ilharco, Shuran Song, Thomas Kollar, Yair Carmon, Ach… — [Language models scale reliably with over-training and on downstream tasks](https://md.research.elamur.ai/free/en/papers/gadre-2024-language-models-scale-reliably-over/)
39. Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003, 2021. [find](https://scholar.google.com/scholar?q=Preetum+Nakkiran%2C+Gal+Kaplun%2C+Yamini+Bansal%2C+Tristan+Yang%2C+Boaz+Barak%2C+and+Ilya+Sutskever%2E+Deep+double+descent%3A+Where+bigger+models+and+more+data+hurt%2E+Journal+of+Statistical+Mechanics%3A+Theory+and+Exp)
40. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In NAACL-HLT (1), pages 4171–4186. Association for Computational Linguistics, 2019.
41. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. [Roberta: A robustly optimized BERT pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). CoRR, abs/1907.11692, 2019.
42. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of machine learning research, 21(140):1–67, 2020.
43. Stanislas Polu and Ilya Sutskever. [Generative language modeling for automated theorem proving](https://md.research.elamur.ai/free/en/papers/polu-2020-generative-language-modeling-automated-theorem/). arXiv preprint arXiv:2009.03393, 2020.
44. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023. [find](https://scholar.google.com/scholar?q=Suriya+Gunasekar%2C+Yi+Zhang%2C+Jyoti+Aneja%2C+Caio+C%C3%A9sar+Teodoro+Mendes%2C+Allie+Del+Giorno%2C+Sivakanth+Gopi%2C+Mojan+Javaheripi%2C+Piero+Kauffmann%2C+Gustavo+de+Rosa%2C+Olli+Saarikivi%2C+et+al%2E+Textbooks+are+all+you+n)
45. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021. [find](https://scholar.google.com/scholar?q=Katherine+Lee%2C+Daphne+Ippolito%2C+Andrew+Nystrom%2C+Chiyuan+Zhang%2C+Douglas+Eck%2C+Chris+Callison%2DBurch%2C+and+Nicholas+Carlini%2E+Deduplicating+training+data+makes+language+models+better%2E+arXiv+preprint+arXiv%3A2)
46. Nikhil Kandpal, Eric Wallace, and Colin Raffel. [Deduplicating training data mitigates privacy risks in language models](https://md.research.elamur.ai/free/en/papers/kandpal-2022-deduplicating-training-data-mitigates-privacy/). In International Conference on Machine Learning, pages 10697–10707. PMLR, 2022.
47. Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. [D4: Improving llm pretraining via document de-duplication and diversification](https://md.research.elamur.ai/free/en/papers/tirumala-2023-d4-improving-llm-pretraining-document/). In NIPS, volume 36, 2023.
48. Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36, 2024a. [find](https://scholar.google.com/scholar?q=Sang+Michael+Xie%2C+Shibani+Santurkar%2C+Tengyu+Ma%2C+and+Percy+S+Liang%2E+Data+selection+for+language+models+via+importance+resampling%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2C+2024a%2E)
49. Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. [A survey on data selection for language models](https://md.research.elamur.ai/free/en/papers/albalak-2024-survey-data-selection-language-models/), 2024.
50. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36, 2024b. [find](https://scholar.google.com/scholar?q=Sang+Michael+Xie%2C+Hieu+Pham%2C+Xuanyi+Dong%2C+Nan+Du%2C+Hanxiao+Liu%2C+Yifeng+Lu%2C+Percy+S+Liang%2C+Quoc+V+Le%2C+Tengyu+Ma%2C+and+Adams+Wei+Yu%2E+Doremi%3A+Optimizing+data+mixtures+speeds+up+language+model+pretraining%2E+)
51. Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. Skill-it! a data-driven skills framework for understanding and training language models. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Mayee+Chen%2C+Nicholas+Roberts%2C+Kush+Bhatia%2C+Jue+Wang%2C+Ce+Zhang%2C+Frederic+Sala%2C+and+Christopher+R%C3%A9%2E+Skill%2Dit%21+a+data%2Ddriven+skills+framework+for+understanding+and+training+language+models%2E+Advances+in+N)
52. Yingwei Ma, Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shanshan Li. At which training stage does code data help LLMs reasoning? In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=KIPJKST4gw](https://openreview.net/forum?id=KIPJKST4gw). [↗](https://openreview.net/forum?id=KIPJKST4gw)
53. Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. [From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning](https://md.research.elamur.ai/free/en/papers/li-2023-quantity-quality-boosting-llm-performance/). arXiv preprint arXiv:2308.12032, 2023c.
54. Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. [What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-makes-good-data-alignment-comprehensive/). In ICLR, 2024.
55. Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang, Min Yang, Lei Zhang, Shuzheng Si, Junhao Liu, Tongliang Liu, Fei Huang, et al. One shot learning as instruction data prospector for large language models. arXiv preprint arXiv:2312.[10302, 2023d](https://md.research.elamur.ai/free/en/papers/li-2024-one-shot-learning-as-instruction/).
56. Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. [Less: Selecting influential data for targeted instruction tuning](https://md.research.elamur.ai/free/en/papers/xia-2024-less-selecting-influential-data-targeted/). arXiv preprint arXiv:2402.04333, 2024.
57. Feiyang Kang, Hoang Anh Just, Yifan Sun, Himanshu Jahagirdar, Yuanzhi Zhang, Rongxing Du, Anit Kumar Sahu, and Ruoxi Jia. Get more for less: Principled data selection for warming up fine-tuning in LLMs. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview… [find](https://scholar.google.com/scholar?q=Feiyang+Kang%2C+Hoang+Anh+Just%2C+Yifan+Sun%2C+Himanshu+Jahagirdar%2C+Yuanzhi+Zhang%2C+Rongxing+Du%2C+Anit+Kumar+Sahu%2C+and+Ruoxi+Jia%2E+Get+more+for+less%3A+Principled+data+selection+for+warming+up+fine%2Dtuning+in+LLM)
58. Ziheng Qin, Kai Wang, Zangwei Zheng, Jianyang Gu, Xiangyu Peng, Zhaopan Xu, Daquan Zhou, Lei Shang, Baigui Sun, Xuansong Xie, and Yang You. Infobatch: Lossless training speed up by unbiased dynamic data pruning. In The Twelfth International Conference on Learning Representations, 2024. URL https://… [find](https://scholar.google.com/scholar?q=Ziheng+Qin%2C+Kai+Wang%2C+Zangwei+Zheng%2C+Jianyang+Gu%2C+Xiangyu+Peng%2C+Zhaopan+Xu%2C+Daquan+Zhou%2C+Lei+Shang%2C+Baigui+Sun%2C+Xuansong+Xie%2C+and+Yang+You%2E+Infobatch%3A+Lossless+training+speed+up+by+unbiased+dynamic+da)
59. Together Computer. [Redpajama: an open dataset for training large language models](https://md.research.elamur.ai/free/en/papers/weber-2024-redpajama-open-dataset-training-large/), 10 2023. URL [https://github.com/togethercomputer/RedPajama-Data](https://github.com/togethercomputer/RedPajama-Data). [↗](https://github.com/togethercomputer/RedPajama-Data)
60. Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489, 2017. [find](https://scholar.google.com/scholar?q=Ozan+Sener+and+Silvio+Savarese%2E+Active+learning+for+convolutional+neural+networks%3A+A+core%2Dset+approach%2E+arXiv+preprint+arXiv%3A1708%2E00489%2C+2017%2E)
61. Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh Iyer. Retrieve: Coreset selection for efficient and robust semi-supervised learning. Advances in neural information processing systems, 34:14488–14501, 2021. [find](https://scholar.google.com/scholar?q=Krishnateja+Killamsetty%2C+Xujiang+Zhao%2C+Feng+Chen%2C+and+Rishabh+Iyer%2E+Retrieve%3A+Coreset+selection+for+efficient+and+robust+semi%2Dsupervised+learning%2E+Advances+in+neural+information+processing+systems%2C+34)
62. Ilya Loshchilov and Frank Hutter. Online batch selection for faster training of neural networks. arXiv preprint arXiv:1511.06343, 2015. — [arXiv:1511.06343](https://arxiv.org/abs/1511.06343) [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+Online+batch+selection+for+faster+training+of+neural+networks%2E+arXiv+preprint+arXiv%3A1511%2E06343%2C+2015%2E+%E2%80%94+%5BarXiv%3A1511%2E06343%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1511%2E06343%29)
63. Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015. [find](https://scholar.google.com/scholar?q=Tom+Schaul%2C+John+Quan%2C+Ioannis+Antonoglou%2C+and+David+Silver%2E+Prioritized+experience+replay%2E+arXiv+preprint+arXiv%3A1511%2E05952%2C+2015%2E)
64. Haw-Shiuan Chang, Erik Learned-Miller, and Andrew McCallum. Active bias: Training more accurate neural networks by emphasizing high variance samples. Advances in Neural Information Processing Systems, 30, 2017. [find](https://scholar.google.com/scholar?q=Haw%2DShiuan+Chang%2C+Erik+Learned%2DMiller%2C+and+Andrew+McCallum%2E+Active+bias%3A+Training+more+accurate+neural+networks+by+emphasizing+high+variance+samples%2E+Advances+in+Neural+Information+Processing+Systems%2C)
65. Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning, pages 2525–2534. PMLR, 2018. [find](https://scholar.google.com/scholar?q=Angelos+Katharopoulos+and+Fran%C3%A7ois+Fleuret%2E+Not+all+samples+are+created+equal%3A+Deep+learning+with+importance+sampling%2E+In+International+conference+on+machine+learning%2C+pages+2525%E2%80%932534%2E+PMLR%2C+2018%2E)
66. Angela H Jiang, Daniel L-K Wong, Giulio Zhou, David G Andersen, Jeffrey Dean, Gregory R Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary C Lipton, et al. Accelerating deep learning by focusing on the biggest losers. arXiv preprint arXiv:1910.00762, 2019. — [arXiv:1910.00762](https://arxiv.org/abs/1910.00762) [find](https://scholar.google.com/scholar?q=Angela+H+Jiang%2C+Daniel+L%2DK+Wong%2C+Giulio+Zhou%2C+David+G+Andersen%2C+Jeffrey+Dean%2C+Gregory+R+Ganger%2C+Gauri+Joshi%2C+Michael+Kaminksy%2C+Michael+Kozuch%2C+Zachary+C+Lipton%2C+et+al%2E+Accelerating+deep+learning+by+fo)
67. Hwanjun Song, Minseok Kim, Sundong Kim, and Jae-Gil Lee. Carpe diem, seize the samples uncertain" at the moment" for adaptive batch selection. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 1385–1394, 2020. [find](https://scholar.google.com/scholar?q=Hwanjun+Song%2C+Minseok+Kim%2C+Sundong+Kim%2C+and+Jae%2DGil+Lee%2E+Carpe+diem%2C+seize+the+samples+uncertain%22+at+the+moment%22+for+adaptive+batch+selection%2E+In+Proceedings+of+the+29th+ACM+International+Conference+o)
68. Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. Prioritized training on points that are learnable, worth learning, and not yet learnt. In Kamalika Chaudhuri, Stefanie Je… [find](https://scholar.google.com/scholar?q=S%C3%B6ren+Mindermann%2C+Jan+M+Brauner%2C+Muhammed+T+Razzak%2C+Mrinank+Sharma%2C+Andreas+Kirsch%2C+Winnie+Xu%2C+Benedikt+H%C3%B6ltgen%2C+Aidan+N+Gomez%2C+Adrien+Morisot%2C+Sebastian+Farquhar%2C+and+Yarin+Gal%2E+Prioritized+training+)
69. Simin Fan and Martin Jaggi. arXiv:2310.15389, 2023. — [arXiv:2310.15389](https://arxiv.org/abs/2310.15389) [find](https://scholar.google.com/scholar?q=Simin+Fan+and+Martin+Jaggi%2E+arXiv%3A2310%2E15389%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E15389%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E15389%29)
70. Irreducible curriculum for language model pretraining. Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, and Matt Kusner. No train no gain: Revisiting efficient training algorithms for transformer-based language models. In Thirty-seventh Conference on Neural Information Processing Systems,… [find](https://scholar.google.com/scholar?q=Irreducible+curriculum+for+language+model+pretraining%2E+Jean+Kaddour%2C+Oscar+Key%2C+Piotr+Nawrot%2C+Pasquale+Minervini%2C+and+Matt+Kusner%2E+No+train+no+gain%3A+Revisiting+efficient+training+algorithms+for+transf)
71. Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. arXiv preprint arXiv:1906.11829, 2019. [find](https://scholar.google.com/scholar?q=Cody+Coleman%2C+Christopher+Yeh%2C+Stephen+Mussmann%2C+Baharan+Mirzasoleiman%2C+Peter+Bailis%2C+Percy+Liang%2C+Jure+Leskovec%2C+and+Matei+Zaharia%2E+Selection+via+proxy%3A+Efficient+data+selection+for+deep+learning%2E+ar)
72. Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels. arXiv preprint arXiv:2401.12926, 2024. [find](https://scholar.google.com/scholar?q=Logan+Engstrom%2C+Axel+Feldmann%2C+and+Aleksander+Madry%2E+Dsdm%3A+Model%2Daware+dataset+selection+with+datamodels%2E+arXiv+preprint+arXiv%3A2401%2E12926%2C+2024%2E)
73. Yonatan Oren, Shiori Sagawa, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust language modeling. arXiv preprint arXiv:1909.02060, 2019. [find](https://scholar.google.com/scholar?q=Yonatan+Oren%2C+Shiori+Sagawa%2C+Tatsunori+B+Hashimoto%2C+and+Percy+Liang%2E+Distributionally+robust+language+modeling%2E+arXiv+preprint+arXiv%3A1909%2E02060%2C+2019%2E)
74. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. [find](https://scholar.google.com/scholar?q=Yann+LeCun%2C+L%C3%A9on+Bottou%2C+Yoshua+Bengio%2C+and+Patrick+Haffner%2E+Gradient%2Dbased+learning+applied+to+document+recognition%2E+Proceedings+of+the+IEEE%2C+86%2811%29%3A2278%E2%80%932324%2C+1998%2E)
75. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. [Recursive deep models for semantic compositionality over a sentiment treebank](https://md.research.elamur.ai/free/en/papers/socher-2013-recursive-deep-models-semantic-compositionality/). In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642,…
76. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805, 2018.
77. Yuxian Gu, Zhengyan Zhang, Xiaozhi Wang, Zhiyuan Liu, and Maosong Sun. Train no evil: Selective masking for task-guided pre-training. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages… [find](https://scholar.google.com/scholar?q=Yuxian+Gu%2C+Zhengyan+Zhang%2C+Xiaozhi+Wang%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+Train+no+evil%3A+Selective+masking+for+task%2Dguided+pre%2Dtraining%2E+In+Bonnie+Webber%2C+Trevor+Cohn%2C+Yulan+He%2C+and+Yang+Liu%2C+editors%2C+Pr)
78. Tanish Lad, Himanshu Maheshwari, Shreyas Kottukkal, and Radhika Mamidi. Using selective masking as a bridge between pre-training and fine-tuning. arXiv preprint arXiv:2211.13815, 2022. — [arXiv:2211.13815](https://arxiv.org/abs/2211.13815) [find](https://scholar.google.com/scholar?q=Tanish+Lad%2C+Himanshu+Maheshwari%2C+Shreyas+Kottukkal%2C+and+Radhika+Mamidi%2E+Using+selective+masking+as+a+bridge+between+pre%2Dtraining+and+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A2211%2E13815%2C+2022%2E+%E2%80%94+%5BarXiv%3A2211%2E1)
79. Qihuang Zhong, Liang Ding, Juhua Liu, Xuebo Liu, Min Zhang, Bo Du, and Dacheng Tao. Revisiting token dropping strategy in efficient BERT pretraining. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Qihuang+Zhong%2C+Liang+Ding%2C+Juhua+Liu%2C+Xuebo+Liu%2C+Min+Zhang%2C+Bo+Du%2C+and+Dacheng+Tao%2E+Revisiting+token+dropping+strategy+in+efficient+BERT+pretraining%2E+In+Anna+Rogers%2C+Jordan+Boyd%2DGraber%2C+and+Naoaki+Oka)
80. Le Hou, Richard Yuanzhe Pang, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, and Denny Zhou. Token dropping for efficient BERT pretraining. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Le+Hou%2C+Richard+Yuanzhe+Pang%2C+Tianyi+Zhou%2C+Yuexin+Wu%2C+Xinying+Song%2C+Xiaodan+Song%2C+and+Denny+Zhou%2E+Token+dropping+for+efficient+BERT+pretraining%2E+In+Smaranda+Muresan%2C+Preslav+Nakov%2C+and+Aline+Villavice)
81. Tianjian Li, Haoran Xu, Philipp Koehn, Daniel Khashabi, and Kenton Murray. Error norm truncation: Robust training in the presence of data noise for text generation models. arXiv preprint arXiv:2310.00840, 2023e. — [arXiv:2310.00840](https://arxiv.org/abs/2310.00840) [find](https://scholar.google.com/scholar?q=Tianjian+Li%2C+Haoran+Xu%2C+Philipp+Koehn%2C+Daniel+Khashabi%2C+and+Kenton+Murray%2E+Error+norm+truncation%3A+Robust+training+in+the+presence+of+data+noise+for+text+generation+models%2E+arXiv+preprint+arXiv%3A2310%2E00)
82. Jessica Rumbelow and Matthew Watkins. Solidgoldmagikarp (plus, prompt generation). LessWrong, 2023. URL [https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation](https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation). [↗](https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation)
83. Sander Land and Max Bartolo. Fishing for magikarp: Automatically detecting under-trained tokens in large language models. arXiv preprint arXiv:2405.05417, 2024. arXiv preprint Naomi Saphra and Adam Lopez. [Understanding learning dynamics of language models with svcca](https://md.research.elamur.ai/free/en/papers/saphra-2019-understanding-learning-dynamics-language-models/). arXiv preprint arXiv:1811.00225,…
84. Leshem Choshen, Guy Hacohen, Daphna Weinshall, and Omri Abend. The grammar-learning trajectories of neural language models. arXiv preprint arXiv:2109.06096, 2021. [find](https://scholar.google.com/scholar?q=Leshem+Choshen%2C+Guy+Hacohen%2C+Daphna+Weinshall%2C+and+Omri+Abend%2E+The+grammar%2Dlearning+trajectories+of+neural+language+models%2E+arXiv+preprint+arXiv%3A2109%2E06096%2C+2021%2E)
85. Leo Z Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A Smith. [Probing across time: What does roberta know and when?](https://md.research.elamur.ai/free/en/papers/liu-2021-probing-across-time-what-does/) arXiv preprint arXiv:2104.07885, 2021.
86. Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. [Grokking: Generalization beyond overfitting on small algorithmic datasets](https://md.research.elamur.ai/free/en/papers/power-2022-grokking-generalization-overfitting-small-algorithmic/). arXiv preprint arXiv:2201.02177, 2022.
87. Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Ves Stoyanov. Training trajectories of language models across scales. arXiv preprint arXiv:2212.09803, 2022. [find](https://scholar.google.com/scholar?q=Mengzhou+Xia%2C+Mikel+Artetxe%2C+Chunting+Zhou%2C+Xi+Victoria+Lin%2C+Ramakanth+Pasunuru%2C+Danqi+Chen%2C+Luke+Zettlemoyer%2C+and+Ves+Stoyanov%2E+Training+trajectories+of+language+models+across+scales%2E+arXiv+preprint+)
88. Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. [Scaling laws for transfer](https://md.research.elamur.ai/free/en/papers/hernandez-2021-scaling-laws-transfer/). arXiv preprint arXiv:2102.01293, 2021.
89. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). arXiv preprint arXiv:2203.15556, 2022.
90. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). arXiv preprint arXiv:2206.07682, 2022b.
91. Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. Scaling laws for downstream task performance of large language models. arXiv preprint arXiv:2402.04177, 2024. — [arXiv:2402.04177](https://arxiv.org/abs/2402.04177) [find](https://scholar.google.com/scholar?q=Berivan+Isik%2C+Natalia+Ponomareva%2C+Hussein+Hazimeh%2C+Dimitris+Paparas%2C+Sergei+Vassilvitskii%2C+and+Sanmi+Koyejo%2E+Scaling+laws+for+downstream+task+performance+of+large+language+models%2E+arXiv+preprint+arXiv)
92. Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. [Memorization without overfitting: Analyzing the training dynamics of large language models](https://md.research.elamur.ai/free/en/papers/tirumala-2022-memorization-overfitting-analyzing-training-dynamics/). Advances in Neural Information Processing Systems, 35:38274–38290, 2022.
93. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022. [find](https://scholar.google.com/scholar?q=Nicholas+Carlini%2C+Daphne+Ippolito%2C+Matthew+Jagielski%2C+Katherine+Lee%2C+Florian+Tramer%2C+and+Chiyuan+Zhang%2E+Quantifying+memorization+across+neural+language+models%2E+arXiv+preprint+arXiv%3A2202%2E07646%2C+2022%2E)
94. Tom Henighan, Shan Carter, Tristan Hume, Nelson Elhage, Robert Lasenby, Stanislav Fort, Nicholas Schiefer, and Christopher Olah. Superposition, memorization, and double descent. Transformer Circuits Thread, 2023. [find](https://scholar.google.com/scholar?q=Tom+Henighan%2C+Shan+Carter%2C+Tristan+Hume%2C+Nelson+Elhage%2C+Robert+Lasenby%2C+Stanislav+Fort%2C+Nicholas+Schiefer%2C+and+Christopher+Olah%2E+Superposition%2C+memorization%2C+and+double+descent%2E+Transformer+Circuits+T)
95. Stella Biderman, USVSN PRASHANTH, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. [Emergent and predictable memorization in large language models](https://md.research.elamur.ai/free/en/papers/biderman-2023-emergent-predictable-memorization-large-language/). Advances in Neural Information Processing Systems, 36, 2024.
96. Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. [Scaling laws and interpretability of learning from repeated data](https://md.research.elamur.ai/free/en/papers/hernandez-2022-scaling-laws-interpretability-learning-repeated/). arXiv preprint arXiv:2205.10487, 2022.
97. Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, and Yang You. To repeat or not to repeat: Insights from scaling llm under token-crisis. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Fuzhao+Xue%2C+Yao+Fu%2C+Wangchunshu+Zhou%2C+Zangwei+Zheng%2C+and+Yang+You%2E+To+repeat+or+not+to+repeat%3A+Insights+from+scaling+llm+under+token%2Dcrisis%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2C+2024)
98. Charles AE Goodhart and CAE Goodhart. Problems of monetary management: the UK experience. Springer, 1984. [find](https://scholar.google.com/scholar?q=Charles+AE+Goodhart+and+CAE+Goodhart%2E+Problems+of+monetary+management%3A+the+UK+experience%2E+Springer%2C+1984%2E)
99. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 35:27730–27744, 2022.
100. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/), 2021. URL https://arxiv.org/abs/2110.14168. [↗](https://arxiv.org/abs/2110.14168)
101. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In NIPS, 2021.
102. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. arXiv preprint arXiv:2211.10435, 2022. [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Pengfei+Liu%2C+Yiming+Yang%2C+Jamie+Callan%2C+and+Graham+Neubig%2E+Pal%3A+Program%2Daided+language+models%2E+arXiv+preprint+arXiv%3A2211%2E10435%2C+2022%2E)
103. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. [Are NLP models really able to solve simple math word problems?](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/) In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, Online, June 2021. Ass…
104. Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing English math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975–984, Online, July 2020. Association for Computational Linguistics. d… [find](https://scholar.google.com/scholar?q=Shen%2Dyun+Miao%2C+Chao%2DChun+Liang%2C+and+Keh%2DYih+Su%2E+A+diverse+corpus+for+evaluating+and+developing+English+math+word+problem+solvers%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Compu)
105. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1152–1157, San Diego,… [find](https://scholar.google.com/scholar?q=Rik+Koncel%2DKedziorski%2C+Subhro+Roy%2C+Aida+Amini%2C+Nate+Kushman%2C+and+Hannaneh+Hajishirzi%2E+MAWPS%3A+A+math+word+problem+repository%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of+the+)
106. Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. [Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning](https://md.research.elamur.ai/free/en/papers/lu-2022-dynamic-prompt-learning-policy-gradient/). In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview…
107. Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. [Mathqa: Towards interpretable math word problem solving with operation-based formalisms](https://md.research.elamur.ai/free/en/papers/amini-2019-mathqa-interpretable-math-word-problem/). arXiv preprint arXiv:1905.13319, 2019.
108. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300, 2020.
109. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. [Challenging big-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). arXiv preprint arXiv:2210.09261, 2022.
110. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. [Agieval: A human-centric benchmark for evaluating foundation models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/). arXiv preprint arXiv:2304.06364, 2023b.
111. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457, 2018.
112. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). arXiv preprint arXiv:1905.10044, 2019.
113. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020.
114. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2C+2019%2E)
115. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2C+2021%2E)
116. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+arXiv+preprint+arXiv%3A1809%2E02789%2C+2018%2E)
117. Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, et al. [Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x](https://md.research.elamur.ai/free/en/papers/zheng-2023-codegeex-pre-trained-model-code/). In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining…
118. Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. Tydi qa: A benchmark for information-seeking question answering in ty pologically di verse languages. Transactions of the Association for Computational Linguistics, 8:454–470, 202… [find](https://scholar.google.com/scholar?q=Jonathan+H+Clark%2C+Eunsol+Choi%2C+Michael+Collins%2C+Dan+Garrette%2C+Tom+Kwiatkowski%2C+Vitaly+Nikolaev%2C+and+Jennimaria+Palomaki%2E+Tydi+qa%3A+A+benchmark+for+information%2Dseeking+question+answering+in+ty+pological)
119. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732, 2021.
120. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024. [find](https://scholar.google.com/scholar?q=Daya+Guo%2C+Qihao+Zhu%2C+Dejian+Yang%2C+Zhenda+Xie%2C+Kai+Dong%2C+Wentao+Zhang%2C+Guanting+Chen%2C+Xiao+Bi%2C+Y+Wu%2C+YK+Li%2C+et+al%2E+Deepseek%2Dcoder%3A+When+the+large+language+model+meets+programming%E2%80%93the+rise+of+code+intel)
121. Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. [Contrastive decoding: Open-ended text generation as optimization](https://md.research.elamur.ai/free/en/papers/li-2022-contrastive-decoding-open-ended-text/). In ACL (1), pages 12286–12312. Association for Computational Linguistics, 2023f.
122. Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. [Knowledge fusion of large language models](https://md.research.elamur.ai/free/en/papers/wan-2024-knowledge-fusion-large-language-models/). In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=jiDsk12qcz](https://openreview.net/forum?id=jiDsk12qcz). [↗](https://openreview.net/forum?id=jiDsk12qcz)
123. Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. [Specializing smaller language models towards multi-step reasoning](https://md.research.elamur.ai/free/en/papers/fu-2023-specializing-smaller-language-models-towards/). In International Conference on Machine Learning, pages 10421–10430. PMLR, 2023.

*Updated 2 November 2026*