---
title: "Efficient Online Data Mixing For Language Model Pre-Training"
description: "The authors propose Online Data Mixing (ODM), which treats each of The Pile's 22 domains as an… 21 references, cited by 34 papers here."
authors: ["Alon Albalak", "Liangming Pan", "Colin Raffel", "William Yang Wang"]
year: 2023
venue: "arXiv:2312.02406"
source: "https://arxiv.org/abs/2312.02406"
doi: "10.48550/arxiv.2312.02406"
arxiv: "2312.02406"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/albalak-2023-efficient-online-data-mixing-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/albalak-2023-efficient-online-data-mixing-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/11/) / Efficient Online Data Mixing For Language Model Pre-Training

# Efficient Online Data Mixing For Language Model Pre-Training

**Authors:** [Alon Albalak](https://md.research.elamur.ai/free/en/authors/alon-albalak/), [Liangming Pan](https://md.research.elamur.ai/free/en/authors/liangming-pan/), [Colin Raffel](https://md.research.elamur.ai/free/en/authors/colin-raffel/) and [William Yang Wang](https://md.research.elamur.ai/free/en/authors/william-yang-wang/) · **Year:** 2023 · **Venue:** arXiv:2312.02406 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2312.02406) · [PDF](https://arxiv.org/pdf/2312.02406) · [DOI 10.48550/arxiv.2312.02406](https://doi.org/10.48550/arxiv.2312.02406) · [arXiv 2312.02406](https://arxiv.org/abs/2312.02406) · [OpenAlex](https://openalex.org/W4389469774)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [34](https://md.research.elamur.ai/free/en/papers/albalak-2023-efficient-online-data-mixing-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

When pretraining an LLM, practitioners must decide how often to draw from each data domain. This choice is usually fixed in advance, either by hand (as with The Pile's suggested weights) or through costly extra training runs (DoReMi, which needs separate reference and proxy models). Static mixtures cannot respond as the model's needs change during training. The authors propose Online Data Mixing (ODM), which treats each of The Pile's 22 domains as an arm in a multi-armed bandit based on Exp3. The reward is the per-domain training loss, smoothed with a moving average. This shifts sampling toward domains the model still finds hard and requires no extra forward or backward passes. They test it on a 1B-parameter model with the Pythia configuration, trained on 50B tokens with GPT-NeoX, against The Pile Weights and two DoReMi variants. ODM reaches the best baseline's final validation perplexity in 19% fewer steps and scores highest on 5-shot MMLU, at almost no runtime cost. The work shows that adaptive mixing can be nearly free, and that precomputed DoReMi weights may not transfer across tokenizers.

## Key points

- **Setup:** 100,000 steps with a global batch of 480 sequences of 1024 tokens (about 50B tokens). ODM starts from The Pile Weights and keeps its policy frozen during a warmup of 1% of steps.
- **Efficiency:** ODM matches the final perplexity of The Pile Weights with 30% fewer iterations and of DoReMi-50k with 19% fewer. Its final validation perplexity is lower by 4.8% (vs. The Pile Weights), 2.4% (vs. DoReMi-50k) and 4.9% (vs. DoReMi-256k). The added wall-clock overhead is about 0.000007%.
- **MMLU (57 tasks, 5-shot) average accuracy:**
  - ODM: 0.284
  - DoReMi-50k: 0.279
  - DoReMi-256k: 0.276
  - The Pile Weights: 0.275
  
  This is roughly a 1.9% relative gain over DoReMi-50k and about 3% over The Pile Weights.
- **Tokenizer transfer:** The published DoReMi-256k weights, derived with a 256k-vocabulary tokenizer, gave 0.7% worse test perplexity than The Pile Weights under the 50k tokenizer used here. Weights recomputed for that tokenizer (DoReMi-50k) improved on The Pile Weights by 2.6%.
- **Learned mixture:**
  - The learned mixture drifts toward uniform. PhilPapers, HackerNews and BookCorpus2 gain the most share, while Github, ArXiv and PubMed_Central lose the most.
  - The mixture mostly settles by about 40k steps.
  - ODM is best on 9 of 22 domains and worst on 4.
  - Its perplexity is higher early in training, which the authors attribute to single-domain micro-batches.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Toronto](https://md.research.elamur.ai/free/en/universities/university-of-toronto/) — University, Toronto, Canada · 110 works · 179,061 citations
- [University of California, Santa Barbara](https://md.research.elamur.ai/free/en/universities/university-of-california-santa-barbara/) — University, Santa Barbara, United States · 67 works · 10,550 citations
- [Vector Institute](https://md.research.elamur.ai/free/en/labs/vector-institute/) — Research institute, Toronto, Canada · 37 works · 5,703 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Alon Albalak, Liangming Pan, Colin Raffel and William Yang Wang. 2023. Efficient Online Data Mixing For Language Model Pre-Training. arXiv:2312.02406.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{albalak2023efficient,
  title = {{Efficient Online Data Mixing For Language Model Pre-Training}},
  author = {Alon Albalak, Liangming Pan, Colin Raffel and William Yang Wang},
  year = {2023},
  doi = {10.48550/arxiv.2312.02406},
  eprint = {2312.02406},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.02406}
}
```

- [← previous Dolphin: A Challenging and Diverse Benchmark for Arabic NLG](https://md.research.elamur.ai/free/en/papers/nagoudi-2023-dolphin-challenging-diverse-benchmark-arabic/)
- [next → Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Models](https://md.research.elamur.ai/free/en/papers/feng-2023-knowledge-card-filling-llms-knowledge/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/11/)

## References

21 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+Shawn+Presser%2C+and+Connor+Leahy%2E+The+pile%3A+An+800gb+dataset+of+)
2. Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open llm leaderboard. [https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), 2023. [↗](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
4. Yuval Kirstain, Patrick Lewis, Sebastian Riedel, and Omer Levy. A few more examples may be worth billions of parameters. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1017–1029, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Yuval+Kirstain%2C+Patrick+Lewis%2C+Sebastian+Riedel%2C+and+Omer+Levy%2E+A+few+more+examples+may+be+worth+billions+of+parameters%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2022%2C+pages)
5. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Pengfei+Liu%2C+Puxin+Xu%2C+Srini+Iyer%2C+Jiao+Sun%2C+Yuning+Mao%2C+Xuezhe+Ma%2C+Avia+Efrat%2C+Ping+Yu%2C+LILI+YU%2C+Susan+Zhang%2C+Gargi+Ghosh%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Omer+Levy%2E+LIMA%3A+Less+is+mo)
6. Alon Albalak, Colin Raffel, and William Yang Wang. Improving few-shot generalization by exploring and exploiting auxiliary data. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. [find](https://scholar.google.com/scholar?q=Alon+Albalak%2C+Colin+Raffel%2C+and+William+Yang+Wang%2E+Improving+few%2Dshot+generalization+by+exploring+and+exploiting+auxiliary+data%2E+In+Thirty%2Dseventh+Conference+on+Neural+Information+Processing+Systems%2C+)
7. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). In International Conference on Learning Representations, 2022.
8. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020. [find](https://scholar.google.com/scholar?q=Daniel+M%2E+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B%2E+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+Fine%2Dtuning+language+models+from+human+preferences%2C+2020%2E)
9. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. [find](https://scholar.google.com/scholar?q=Sang+Michael+Xie%2C+Hieu+Pham%2C+Xuanyi+Dong%2C+Nan+Du%2C+Hanxiao+Liu%2C+Yifeng+Lu%2C+Percy+Liang%2C+Quoc+V+Le%2C+Tengyu+Ma%2C+and+Adams+Wei+Yu%2E+Doremi%3A+Optimizing+data+mixtures+speeds+up+language+model+pretraining%2E+In)
10. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). Proceedings of the International Conference on Learning Representations (ICLR), 2021.
11. Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. Scaling expert language models with unsupervised domain discovery, 2023. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Margaret+Li%2C+Mike+Lewis%2C+Weijia+Shi%2C+Tim+Althoff%2C+Noah+A%2E+Smith%2C+and+Luke+Zettlemoyer%2E+Scaling+expert+language+models+with+unsupervised+domain+discovery%2C+2023%2E)
12. RICHARD BELLMAN. A markovian decision process. Journal of Mathematics and Mechanics, 6(5):679–684, 1957. [find](https://scholar.google.com/scholar?q=RICHARD+BELLMAN%2E+A+markovian+decision+process%2E+Journal+of+Mathematics+and+Mechanics%2C+6%285%29%3A679%E2%80%93684%2C+1957%2E)
13. Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing, 32(1):48–77, 2002. [find](https://scholar.google.com/scholar?q=Peter+Auer%2C+Nicolo+Cesa%2DBianchi%2C+Yoav+Freund%2C+and+Robert+E+Schapire%2E+The+nonstochastic+multiarmed+bandit+problem%2E+SIAM+journal+on+computing%2C+32%281%29%3A48%E2%80%9377%2C+2002%2E)
14. Yevgeny Seldin, Csaba Szepesvári, Peter Auer, and Yasin Abbasi-Yadkori. Evaluation and analysis of the performance of the exp3 algorithm in stochastic environments. In Marc Peter Deisenroth, Csaba Szepesvári, and Jan Peters, editors, Proceedings of the Tenth European Workshop on Reinforcement Learni… [find](https://scholar.google.com/scholar?q=Yevgeny+Seldin%2C+Csaba+Szepesv%C3%A1ri%2C+Peter+Auer%2C+and+Yasin+Abbasi%2DYadkori%2E+Evaluation+and+analysis+of+the+performance+of+the+exp3+algorithm+in+stochastic+environments%2E+In+Marc+Peter+Deisenroth%2C+Csaba+Sze)
15. Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Shivanshu Purohit, Tri Songz, Wang Phil, and Samuel Weinbach. GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorc… [find](https://scholar.google.com/scholar?q=Alex+Andonian%2C+Quentin+Anthony%2C+Stella+Biderman%2C+Sid+Black%2C+Preetham+Gali%2C+Leo+Gao%2C+Eric+Hallahan%2C+Josh+Levy%2DKramer%2C+Connor+Leahy%2C+Lucas+Nestler%2C+Kip+Parker%2C+Michael+Pieler%2C+Shivanshu+Purohit%2C+Tri+Son)
16. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on M… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Gregory+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+et+al%2E+Pythia%3A+A+sui)
17. Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashch… [find](https://scholar.google.com/scholar?q=Adam+Roberts%2C+Hyung+Won+Chung%2C+Anselm+Levskaya%2C+Gaurav+Mishra%2C+James+Bradbury%2C+Daniel+Andor%2C+Sharan+Narang%2C+Brian+Lester%2C+Colin+Gaffney%2C+Afroz+Mohiuddin%2C+Curtis+Hawthorne%2C+Aitor+Lewkowycz%2C+Alex+Salcia)
18. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/), 2022.
19. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Daniel+Y%2E+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+FlashAttention%3A+Fast+and+memory%2Defficient+exact+attention+with+IO%2Dawareness%2E+In+Advances+in+Neural+Information+Processing+Systems%2C)
20. Diederik P. Kingma and Jimmy Ba. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
21. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20B: An open-source autoregressive… — [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/)

*Updated 30 October 2026*