---
title: "D4: Improving LLM Pretraining via Document De-Duplication and Diversification — Tirumala 2023"
description: "The authors propose D4, which works in a pre-trained embedding space. 61 references, cited by 59 papers here."
authors: ["Kushal Tirumala", "Daniel Simig", "Armen Aghajanyan", "Ari S. Morcos"]
year: 2023
venue: "arXiv:2308.12284"
source: "https://arxiv.org/abs/2308.12284"
doi: "10.48550/arxiv.2308.12284"
arxiv: "2308.12284"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tirumala-2023-d4-improving-llm-pretraining-document/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tirumala-2023-d4-improving-llm-pretraining-document/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / D4: Improving LLM Pretraining via Document De-Duplication and Diversification

# D4: Improving LLM Pretraining via Document De-Duplication and Diversification

**Authors:** [Kushal Tirumala](https://md.research.elamur.ai/free/en/authors/kushal-tirumala/), [Daniel Simig](https://md.research.elamur.ai/free/en/authors/daniel-simig/), [Armen Aghajanyan](https://md.research.elamur.ai/free/en/authors/armen-aghajanyan/) and [Ari S. Morcos](https://md.research.elamur.ai/free/en/authors/ari-s-morcos/) · **Year:** 2023 · **Venue:** arXiv:2308.12284 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2308.12284) · [PDF](https://arxiv.org/pdf/2308.12284) · [DOI 10.48550/arxiv.2308.12284](https://doi.org/10.48550/arxiv.2308.12284) · [arXiv 2308.12284](https://arxiv.org/abs/2308.12284) · [OpenAlex](https://openalex.org/W7133241490)

**Citations:** 20 citations (OpenAlex, 2026-10-14) · [59](https://md.research.elamur.ai/free/en/papers/tirumala-2023-d4-improving-llm-pretraining-document/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Most LLM pretraining pipelines sample web text at random after basic filtering and near-duplicate removal such as MinHash. Because scaling laws follow a power law, adding more random data brings smaller and smaller gains. This paper asks whether smarter document selection can help. The authors propose D4, which works in a pre-trained embedding space. It first applies SemDeDup to strip out semantic near-duplicates. It then re-clusters the remaining documents and applies SSL Prototypes to diversify the selection. The re-clustering step matters because clusters dominated by duplicates otherwise distort prototype-based pruning.

They train OPT models from scratch, ranging from 125M to 6.7B parameters, on CC-dedup. CC-dedup is a CCNet-processed CommonCrawl corpus with an extra MinHash de-duplication pass. At 6.7B parameters, D4 reaches baseline quality with roughly 20% fewer tokens. It also improves average zero-shot accuracy on 16 tasks by about 2%, and the efficiency gain grows with model size. The work challenges the habit of single-epoch training on maximal data and points to curation as a way past the limits of web data.

## Key points
- **Setup:** OPT models are trained with MetaSeq at three scales:
  - 125M parameters on 3B tokens
  - 1.3B parameters on 40B tokens
  - 6.7B parameters on 100B tokens

  All models are compared against random document selection at a given selection ratio R.
- **Main result:** At 6.7B, D4 gives about 18–22% efficiency gains in validation perplexity. It also adds roughly 2% mean zero-shot accuracy on the 16 NLP tasks from the OPT evaluation suite, which includes ARC, HellaSwag, PIQA, Winogrande and SuperGLUE tasks.
- **Repeating data:** When data runs out, epoching over a D4-selected subset beats training on fresh random data. Repeating randomly chosen data does worse than the baseline.
- **Evaluation:** Perplexity on web-snapshot validation sets (C4, CommonCrawl, CC-dedup) correlates slightly negatively with downstream accuracy. Perplexity on Instructions + Answers, sampled from the OPT-IML tuning data covering over 1500 tasks, correlates positively (Pearson ≈ 0.3). The authors therefore treat it as a more reliable quality signal.
- **Ablation:** Dropping the re-clustering step between SemDeDup and SSL Prototypes clearly hurts perplexity. Analysis shows that SemDeDup removes very dense, duplicate-driven clusters near centroids.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

## Cite

Cite as: Kushal Tirumala, Daniel Simig, Armen Aghajanyan and Ari S. Morcos. 2023. D4: Improving LLM Pretraining via Document De-Duplication and Diversification. arXiv:2308.12284.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{tirumala2023d4,
  title = {{D4: Improving LLM Pretraining via Document De-Duplication and Diversification}},
  author = {Kushal Tirumala, Daniel Simig, Armen Aghajanyan and Ari S. Morcos},
  year = {2023},
  doi = {10.48550/arxiv.2308.12284},
  eprint = {2308.12284},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.12284}
}
```

- [← previous NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://md.research.elamur.ai/free/en/papers/dhole-2023-nl-augmenter-framework-task-sensitive/)
- [next → Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?](https://md.research.elamur.ai/free/en/papers/tay-2023-scaling-laws-vs-model-architectures/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

61 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amro Abbas, Kushal Tirumala, Daniel Simig, Surya Ganguli, and Ari S. Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication. ArXiv, abs/2303.09540, 2023. [our card](https://md.research.elamur.ai/free/en/papers/abbas-2023-semdedup-data-efficient-learning-web/)
2. Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, et al. Efficient large scale language modeling with mixtures of experts. arXiv preprint arXiv:2112.10684, 2021. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Shruti+Bhosale%2C+Naman+Goyal%2C+Todor+Mihaylov%2C+Myle+Ott%2C+Sam+Shleifer%2C+Xi+Victoria+Lin%2C+Jingfei+Du%2C+Srinivasan+Iyer%2C+Ramakanth+Pasunuru%2C+et+al%2E+Efficient+large+scale+language+modeling+wit)
3. Stephen H. Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Févry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged S. Al-shaib… [our card](https://md.research.elamur.ai/free/en/papers/bach-2022-promptsource-integrated-development-environment-repository/)
4. Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. The pushshift reddit dataset. In Proceedings of the international AAAI conference on web and social media, volume 14, pages 830–839, 2020. [find](https://scholar.google.com/scholar?q=Jason+Baumgartner%2C+Savvas+Zannettou%2C+Brian+Keegan%2C+Megan+Squire%2C+and+Jeremy+Blackburn%2E+The+pushshift+reddit+dataset%2E+In+Proceedings+of+the+international+AAAI+conference+on+web+and+social+media%2C+volume)
5. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373, 2023. [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+et+al%2E+Pythia%3A+A+suite+for+a)
6. Vighnesh Birodkar, Hossein Mobahi, and Samy Bengio. Semantic redundancies in image-classification datasets: The 10% you don’t need. arXiv preprint arXiv:1901.11409, 2019. — [arXiv:1901.11409](https://arxiv.org/abs/1901.11409) [find](https://scholar.google.com/scholar?q=Vighnesh+Birodkar%2C+Hossein+Mobahi%2C+and+Samy+Bengio%2E+Semantic+redundancies+in+image%2Dclassification+datasets%3A+The+10%25+you+don%E2%80%99t+need%2E+arXiv+preprint+arXiv%3A1901%2E11409%2C+2019%2E+%E2%80%94+%5BarXiv%3A1901%2E11409%5D%28https%3A%2F%2F)
7. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020. [our card](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/)
8. Andrei Z Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pages 21–29. IEEE, 1997. [find](https://scholar.google.com/scholar?q=Andrei+Z+Broder%2E+On+the+resemblance+and+containment+of+documents%2E+In+Proceedings%2E+Compression+and+Complexity+of+SEQUENCES+1997+%28Cat%2E+No%2E+97TB100171%29%2C+pages+21%E2%80%9329%2E+IEEE%2C+1997%2E)
9. George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4750–4759, 2022. [find](https://scholar.google.com/scholar?q=George+Cazenavette%2C+Tongzhou+Wang%2C+Antonio+Torralba%2C+Alexei+A+Efros%2C+and+Jun%2DYan+Zhu%2E+Dataset+distillation+by+matching+training+trajectories%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Visi)
10. Kashyap Chitta, José M Álvarez, Elmar Haussmann, and Clément Farabet. Training data subset search with ensemble active learning. IEEE Transactions on Intelligent Transportation Systems, 23(9):14741–14752, 2021. [find](https://scholar.google.com/scholar?q=Kashyap+Chitta%2C+Jos%C3%A9+M+%C3%81lvarez%2C+Elmar+Haussmann%2C+and+Cl%C3%A9ment+Farabet%2E+Training+data+subset+search+with+ensemble+active+learning%2E+IEEE+Transactions+on+Intelligent+Transportation+Systems%2C+23%289%29%3A14741%E2%80%9314)
11. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. [our card](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
12. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. [find](https://scholar.google.com/scholar?q=Peter+Clark%2C+Isaac+Cowhey%2C+Oren+Etzioni%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+Carissa+Schoenick%2C+and+Oyvind+Tafjord%2E+Think+you+have+solved+question+answering%3F+try+arc%2C+the+ai2+reasoning+challenge%2E+arXiv+pre)
13. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339, 2022. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Mike+Lewis%2C+Younes+Belkada%2C+and+Luke+Zettlemoyer%2E+Llm%2E+int8+%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2E+arXiv+preprint+arXiv%3A2208%2E07339%2C+2022%2E)
14. Bo Dong, Cristian Lumezanu, Yuncong Chen, Dongjin Song, Takehiko Mizoguchi, Haifeng Chen, and Latifur Khan. At the speed of sound: Efficient audio scene classification. In Proceedings of the 2020 International Conference on Multimedia Retrieval, ICMR ’20, page 301–305, New York, NY, USA, 2020. Assoc… — [doi:10.1145/3372278.3390730](https://doi.org/10.1145/3372278.3390730) [find](https://scholar.google.com/scholar?q=Bo+Dong%2C+Cristian+Lumezanu%2C+Yuncong+Chen%2C+Dongjin+Song%2C+Takehiko+Mizoguchi%2C+Haifeng+Chen%2C+and+Latifur+Khan%2E+At+the+speed+of+sound%3A+Efficient+audio+scene+classification%2E+In+Proceedings+of+the+2020+Inte)
15. Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33: 2881–2891, 2020. [find](https://scholar.google.com/scholar?q=Vitaly+Feldman+and+Chiyuan+Zhang%2E+What+neural+networks+memorize+and+why%3A+Discovering+the+long+tail+via+influence+estimation%2E+Advances+in+Neural+Information+Processing+Systems%2C+33%3A+2881%E2%80%932891%2C+2020%2E)
16. Leo Gao, Stella Rose Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. ArXiv, abs/2101.00027, 2020. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Rose+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+Shawn+Presser%2C+and+Connor+Leahy%2E+The+pile%3A+An+800gb+datase)
17. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don’t stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964, 2020. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Ana+Marasovic%CC%81%2C+Swabha+Swayamdipta%2C+Kyle+Lo%2C+Iz+Beltagy%2C+Doug+Downey%2C+and+Noah+A+Smith%2E+Don%E2%80%99t+stop+pretraining%3A+Adapt+language+models+to+domains+and+tasks%2E+arXiv+preprint+arXiv%3A2004)
18. Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks+and+Kevin+Gimpel%2E+Gaussian+error+linear+units+%28gelus%29%2E+arXiv+preprint+arXiv%3A1606%2E08415%2C+2016%2E)
19. Danny Hernandez, Tom B. Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, T. J. Henighan, Tristan Hume, Scott Johnston, Benjamin Mann, Christopher Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, and Sam McCandlish. Scaling laws and… [our card](https://md.research.elamur.ai/free/en/papers/hernandez-2022-scaling-laws-interpretability-learning-repeated/)
20. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… [our card](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
21. Srinivas Iyer, Xiaojuan Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O’Horo, Gabriel Pereyra, Jeff Wang, Christopher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, and Veselin Stoyanov. Opt-iml: Scaling language mo… [our card](https://md.research.elamur.ai/free/en/papers/iyer-2022-opt-iml-scaling-language-model/)
22. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118, 2021. [our card](https://md.research.elamur.ai/free/en/papers/izacard-2021-unsupervised-dense-information-retrieval-contrastive/)
23. Angela H Jiang, Daniel L-K Wong, Giulio Zhou, David G Andersen, Jeffrey Dean, Gregory R Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary C Lipton, et al. Accelerating deep learning by focusing on the biggest losers. arXiv preprint arXiv:1910.00762, 2019. — [arXiv:1910.00762](https://arxiv.org/abs/1910.00762) [find](https://scholar.google.com/scholar?q=Angela+H+Jiang%2C+Daniel+L%2DK+Wong%2C+Giulio+Zhou%2C+David+G+Andersen%2C+Jeffrey+Dean%2C+Gregory+R+Ganger%2C+Gauri+Joshi%2C+Michael+Kaminksy%2C+Michael+Kozuch%2C+Zachary+C+Lipton%2C+et+al%2E+Accelerating+deep+learning+by+fo)
24. Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547, 2019. [our card](https://md.research.elamur.ai/free/en/papers/johnson-2017-billion-scale-similarity-search-gpus/)
25. Jared Kaplan, Sam McCandlish, T. J. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv, abs/2001.08361, 2020. [our card](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)
26. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+Adam%3A+A+method+for+stochastic+optimization%2E+arXiv+preprint+arXiv%3A1412%2E6980%2C+2014%2E)
27. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Annual Meeting of the Association for Computational Linguistics, 2021. [find](https://scholar.google.com/scholar?q=Katherine+Lee%2C+Daphne+Ippolito%2C+Andrew+Nystrom%2C+Chiyuan+Zhang%2C+Douglas+Eck%2C+Chris+Callison%2DBurch%2C+and+Nicholas+Carlini%2E+Deduplicating+training+data+makes+language+models+better%2E+In+Annual+Meeting+of+t)
28. Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+The+winograd+schema+challenge%2E+In+Thirteenth+international+conference+on+the+principles+of+knowledge+representation+and+reasoning%2C+2012%2E)
29. Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient llms at inference time, 2023. [find](https://scholar.google.com/scholar?q=Zichang+Liu%2C+Jue+Wang%2C+Tri+Dao%2C+Tianyi+Zhou%2C+Binhang+Yuan%2C+Zhao+Song%2C+Anshumali+Shrivastava%2C+Ce+Zhang%2C+Yuandong+Tian%2C+Christopher+Re%2C+et+al%2E+Deja+vu%3A+Contextual+sparsity+for+efficient+llms+at+inferenc)
30. S. Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David M. Mimno, and Daphne Ippolito. A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. ArXiv, abs/2305.13169, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Longpre%2C+Gregory+Yauney%2C+Emily+Reif%2C+Katherine+Lee%2C+Adam+Roberts%2C+Barret+Zoph%2C+Denny+Zhou%2C+Jason+Wei%2C+Kevin+Robinson%2C+David+M%2E+Mimno%2C+and+Daphne+Ippolito%2E+A+pretrainer%E2%80%99s+guide+to+training+data%3A+Mea)
31. Kristof Meding, Luca M Schulze Buschoff, Robert Geirhos, and Felix A Wichmann. Trivial or impossible–dichotomous data difficulty masks model differences (on imagenet and beyond). arXiv preprint arXiv:2110.05922, 2021. — [arXiv:2110.05922](https://arxiv.org/abs/2110.05922) [find](https://scholar.google.com/scholar?q=Kristof+Meding%2C+Luca+M+Schulze+Buschoff%2C+Robert+Geirhos%2C+and+Felix+A+Wichmann%2E+Trivial+or+impossible%E2%80%93dichotomous+data+difficulty+masks+model+differences+%28on+imagenet+and+beyond%29%2E+arXiv+preprint+arXiv%3A)
32. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. [our card](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/)
33. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+arXiv+preprint+arXiv%3A1809%2E02789%2C+2018%2E)
34. Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Lea… [find](https://scholar.google.com/scholar?q=S%C3%B6ren+Mindermann%2C+Jan+M+Brauner%2C+Muhammed+T+Razzak%2C+Mrinank+Sharma%2C+Andreas+Kirsch%2C+Winnie+Xu%2C+Benedikt+H%C3%B6ltgen%2C+Aidan+N+Gomez%2C+Adrien+Morisot%2C+Sebastian+Farquhar%2C+et+al%2E+Prioritized+training+on+point)
35. Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pages 6950–6960. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Baharan+Mirzasoleiman%2C+Jeff+Bilmes%2C+and+Jure+Leskovec%2E+Coresets+for+data%2Defficient+training+of+machine+learning+models%2E+In+International+Conference+on+Machine+Learning%2C+pages+6950%E2%80%936960%2E+PMLR%2C+2020%2E)
36. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and evaluation framework for deeper understanding of commonsense stories. arXiv preprint arXiv:1604.01696, 2016. [find](https://scholar.google.com/scholar?q=Nasrin+Mostafazadeh%2C+Nathanael+Chambers%2C+Xiaodong+He%2C+Devi+Parikh%2C+Dhruv+Batra%2C+Lucy+Vanderwende%2C+Pushmeet+Kohli%2C+and+James+Allen%2E+A+corpus+and+evaluation+framework+for+deeper+understanding+of+commons)
37. Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. 2023. [find](https://scholar.google.com/scholar?q=Niklas+Muennighoff%2C+Alexander+M%2E+Rush%2C+Boaz+Barak%2C+Teven+Le+Scao%2C+Aleksandra+Piktus%2C+Nouamane+Tazi%2C+Sampo+Pyysalo%2C+Thomas+Wolf%2C+and+Colin+Raffel%2E+Scaling+data%2Dconstrained+language+models%2E+2023%2E)
38. Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34:20596–20607, 2021. [find](https://scholar.google.com/scholar?q=Mansheej+Paul%2C+Surya+Ganguli%2C+and+Gintare+Karolina+Dziugaite%2E+Deep+learning+on+a+data+diet%3A+Finding+important+examples+early+in+training%2E+Advances+in+Neural+Information+Processing+Systems%2C+34%3A20596%E2%80%9320)
39. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only. [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Alessandro+Cappelli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+The+refinedweb+dataset+for+falcon+llm%3A+Outperformin)
40. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+Language+models+are+unsupervised+multitask+learners%2E+2019%2E)
41. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020. [our card](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)
42. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2C+2021%2E)
43. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? arXiv preprint arXiv:2304.15004, 2023. [our card](https://md.research.elamur.ai/free/en/papers/schaeffer-2023-emergent-abilities-large-language-models/)
44. Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489, 2017. [find](https://scholar.google.com/scholar?q=Ozan+Sener+and+Silvio+Savarese%2E+Active+learning+for+convolutional+neural+networks%3A+A+core%2Dset+approach%2E+arXiv+preprint+arXiv%3A1708%2E00489%2C+2017%2E)
45. Mohammad Shoeybi, M Patwary, R Puri, P LeGresley, J Casper, B Megatron-LM Catanzaro, et al. Training multi-billion parameter language models using model parallelism. arXiv preprint cs.CL/1909.08053, 2019. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+M+Patwary%2C+R+Puri%2C+P+LeGresley%2C+J+Casper%2C+B+Megatron%2DLM+Catanzaro%2C+et+al%2E+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+arXiv+preprint+cs%2ECL%2F1909%2E08053%2C+2)
46. Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:22… [our card](https://md.research.elamur.ai/free/en/papers/smith-2022-using-deepspeed-megatron-train-megatron/)
47. Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. ArXiv, abs/2206.14486, 2022. [our card](https://md.research.elamur.ai/free/en/papers/sorscher-2022-neural-scaling-laws-beating-power/)
48. Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274–38290, 2022. [our card](https://md.research.elamur.ai/free/en/papers/tirumala-2022-memorization-overfitting-analyzing-training-dynamics/)
49. Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159, 2018. — [arXiv:1812.05159](https://arxiv.org/abs/1812.05159) [find](https://scholar.google.com/scholar?q=Mariya+Toneva%2C+Alessandro+Sordoni%2C+Remi+Tachet+des+Combes%2C+Adam+Trischler%2C+Yoshua+Bengio%2C+and+Geoffrey+J+Gordon%2E+An+empirical+study+of+example+forgetting+during+deep+neural+network+learning%2E+arXiv+pre)
50. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur’elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, abs/23… [our card](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)
51. Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. Will we run out of data? an analysis of the limits of scaling datasets in machine learning, 2022. [find](https://scholar.google.com/scholar?q=Pablo+Villalobos%2C+Jaime+Sevilla%2C+Lennart+Heim%2C+Tamay+Besiroglu%2C+Marius+Hobbhahn%2C+and+Anson+Ho%2E+Will+we+run+out+of+data%3F+an+analysis+of+the+limits+of+scaling+datasets+in+machine+learning%2C+2022%2E)
52. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019. [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
53. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Ma… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Anjana+Arunkumar%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Atharva+Naik%2C+David+Stap%2C+Eshaan+Pathak%2C+Giannis+Karaman)
54. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzm’an, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. ArXiv, abs/1911.00359, 2019. [our card](https://md.research.elamur.ai/free/en/papers/wenzek-2020-ccnet-extracting-high-quality-monolingual/)
55. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V. Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. ArXiv, abs/2305.10429, 2023. [find](https://scholar.google.com/scholar?q=Sang+Michael+Xie%2C+Hieu+Pham%2C+Xuanyi+Dong%2C+Nan+Du%2C+Hanxiao+Liu%2C+Yifeng+Lu%2C+Percy+Liang%2C+Quoc+V%2E+Le%2C+Tengyu+Ma%2C+and+Adams+Wei+Yu%2E+Doremi%3A+Optimizing+data+mixtures+speeds+up+language+model+pretraining%2E+A)
56. Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. ArXiv, abs/2302.03169, 2023. [find](https://scholar.google.com/scholar?q=Sang+Michael+Xie%2C+Shibani+Santurkar%2C+Tengyu+Ma%2C+and+Percy+Liang%2E+Data+selection+for+language+models+via+importance+resampling%2E+ArXiv%2C+abs%2F2302%2E03169%2C+2023%2E)
57. Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, and Yang You. To repeat or not to repeat: Insights from scaling llm under token-crisis. arXiv preprint arXiv:2305.13230, 2023. [find](https://scholar.google.com/scholar?q=Fuzhao+Xue%2C+Yao+Fu%2C+Wangchunshu+Zhou%2C+Zangwei+Zheng%2C+and+Yang+You%2E+To+repeat+or+not+to+repeat%3A+Insights+from+scaling+llm+under+token%2Dcrisis%2E+arXiv+preprint+arXiv%3A2305%2E13230%2C+2023%2E)
58. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2C+2019%2E)
59. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained tr… [our card](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/)
60. Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint arXiv:2006.05929, 2020. [find](https://scholar.google.com/scholar?q=Bo+Zhao%2C+Konda+Reddy+Mopuri%2C+and+Hakan+Bilen%2E+Dataset+condensation+with+gradient+matching%2E+arXiv+preprint+arXiv%3A2006%2E05929%2C+2020%2E)
61. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–27, 20… [our card](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/)