---
title: "A Pretrainer's Guide to Training Data: Measuring the Effects of"
description: "The authors present this as the largest public study of its kind, giving developers measured… 114 references, cited by 98 papers here."
authors: ["Shayne Longpre", "Gregory Yauney", "Emily Reif", "Katherine Lee", "Adam Roberts", "Barret Zoph", "Denny Zhou", "Jason Wei", "Kevin Robinson", "David Mimno", "Daphne Ippolito"]
year: 2023
venue: "arXiv:2305.13169"
source: "https://arxiv.org/abs/2305.13169"
doi: "10.48550/arxiv.2305.13169"
arxiv: "2305.13169"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/longpre-2023-pretrainer-s-guide-training-data/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/longpre-2023-pretrainer-s-guide-training-data/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity

# A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity

**Authors:** [Shayne Longpre](https://md.research.elamur.ai/free/en/authors/shayne-longpre/), [Gregory Yauney](https://md.research.elamur.ai/free/en/authors/gregory-yauney/), [Emily Reif](https://md.research.elamur.ai/free/en/authors/emily-reif/), [Katherine Lee](https://md.research.elamur.ai/free/en/authors/katherine-lee/), [Adam Roberts](https://md.research.elamur.ai/free/en/authors/adam-roberts/), [Barret Zoph](https://md.research.elamur.ai/free/en/authors/barret-zoph/), [Denny Zhou](https://md.research.elamur.ai/free/en/authors/denny-zhou/), [Jason Wei](https://md.research.elamur.ai/free/en/authors/jason-wei/), Kevin Robinson, [David Mimno](https://md.research.elamur.ai/free/en/authors/david-mimno/) and [Daphne Ippolito](https://md.research.elamur.ai/free/en/authors/daphne-ippolito/) · **Year:** 2023 · **Venue:** arXiv:2305.13169 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2305.13169) · [PDF](https://arxiv.org/pdf/2305.13169) · [DOI 10.48550/arxiv.2305.13169](https://doi.org/10.48550/arxiv.2305.13169) · [arXiv 2305.13169](https://arxiv.org/abs/2305.13169) · [OpenAlex](https://openalex.org/W4377865190)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [98](https://md.research.elamur.ai/free/en/papers/longpre-2023-pretrainer-s-guide-training-data/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Decisions about pretraining data for language models are seldom documented or justified, so developers often fall back on untested habits. This study measures how three such decisions affect downstream behaviour: how old the data is, whether content is filtered for quality or toxicity, and which domains are included. The authors pretrained 28 decoder-only models of 1.5B parameters each. Every model used a version of C4 or the Pile that differed along a single dimension, such as a Common Crawl snapshot from another year, a classifier-based filter, or a removed Pile source. They then evaluated each model on many downstream tasks, including toxicity identification and generation. Three patterns emerged. When pretraining and evaluation data come from different periods, performance drops, and finetuning does not fix it. Quality and toxicity filtering pull performance in opposing directions. Varied sources like Books and Web help most. The authors present this as the largest public study of its kind, giving developers measured trade-offs to replace guesswork.

## Key points
- **Setup:** 28 decoder-only models at 1.5B parameters. Data came from C4 (a 2019 Common Crawl snapshot, regenerated for other years) and the Pile (800GB, 22 sources). Both corpora were deduplicated with the approximate method of Lee et al. (2022).
- **Data age:** Performance falls whenever evaluation data is older or newer than the pretraining data. Substantial finetuning does not remove this gap, and larger models suffer more. This also complicates comparisons between older and newer models.
- **Quality filtering:** Removing low-quality text improved downstream results despite shrinking the training data. However, it also increased toxic generation, and its benefits could not be predicted from text characteristics.
- **Toxicity filtering:** Removing toxic text reduced toxic outputs but weakened generalization. Taking the opposite approach, inverse toxicity filtering that discards the cleanest documents, gave targeted gains, so no single filter suits every goal.
- **Domain composition:** Books and Web data helped most, though they also contributed most to toxicity. The best models kept every source, so the authors advise collecting more books and diverse web text.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 492 works · 440,462 citations
- [Cornell University](https://md.research.elamur.ai/free/en/universities/cornell-university/) — University, Ithaca, United States · 134 works · 65,159 citations
- [OpenAI](https://md.research.elamur.ai/free/en/companies/openai/) — Company, San Francisco, United States · 23 works · 27,874 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 587 works

## Cite

Cite as: Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno and Daphne Ippolito. 2023. A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity. arXiv:2305.13169.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{longpre2023pretrainer,
  title = {{A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, \& Toxicity}},
  author = {Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno and Daphne Ippolito},
  year = {2023},
  doi = {10.48550/arxiv.2305.13169},
  eprint = {2305.13169},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.13169}
}
```

- [← previous Zero-Shot Listwise Document Reranking with a Large Language Model](https://md.research.elamur.ai/free/en/papers/ma-2023-zero-shot-listwise-document-reranking/)
- [next → LENS: A Learnable Evaluation Metric for Text Simplification](https://md.research.elamur.ai/free/en/papers/maddela-2023-lens-learnable-evaluation-metric-text/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

114 references, 53 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. [TensorFlow: Large-scale machine learning on heterogeneous distributed systems](https://md.research.elamur.ai/free/en/papers/abadi-2016-tensorflow-large-scale-machine-learning/). arXiv preprint arXiv:1603.04467, 2016.
2. Oshin Agarwal and Ani Nenkova. [Temporal effects on pre-trained models for language processing tasks](https://md.research.elamur.ai/free/en/papers/agarwal-2022-temporal-effects-pre-trained-models/). Transactions of the Association for Computational Linguistics, 10:904–921, 2022.
3. Alon Albalak, Colin Raffel, and William Yang Wang. Improving few-shot generalization by exploring and exploiting auxiliary data. arXiv preprint arXiv:2302.00674, 2023. [find](https://scholar.google.com/scholar?q=Alon+Albalak%2C+Colin+Raffel%2C+and+William+Yang+Wang%2E+Improving+few%2Dshot+generalization+by+exploring+and+exploiting+auxiliary+data%2E+arXiv+preprint+arXiv%3A2302%2E00674%2C+2023%2E)
4. Eduardo G Altmann, Janet B Pierrehumbert, and Adilson E Motter. Beyond word frequency: Bursts, lulls, and scaling in the temporal distributions of words. PLOS one, 4(11):e7678, 2009. [find](https://scholar.google.com/scholar?q=Eduardo+G+Altmann%2C+Janet+B+Pierrehumbert%2C+and+Adilson+E+Motter%2E+Beyond+word+frequency%3A+Bursts%2C+lulls%2C+and+scaling+in+the+temporal+distributions+of+words%2E+PLOS+one%2C+4%2811%29%3Ae7678%2C+2009%2E)
5. Anthropic AI. Introducing Claude, 2023. URL [https://www.anthropic.com/index/introducing-claude](https://www.anthropic.com/index/introducing-claude). [↗](https://www.anthropic.com/index/introducing-claude)
6. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. [Constitutional AI: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073, 2022.
7. Jack Bandy and Nicholas Vincent. Addressing “documentation debt” in machine learning research: A retrospective datasheet for bookcorpus. arXiv preprint arXiv:2105.05241, 2021. — [arXiv:2105.05241](https://arxiv.org/abs/2105.05241) [find](https://scholar.google.com/scholar?q=Jack+Bandy+and+Nicholas+Vincent%2E+Addressing+%E2%80%9Cdocumentation+debt%E2%80%9D+in+machine+learning+research%3A+A+retrospective+datasheet+for+bookcorpus%2E+arXiv+preprint+arXiv%3A2105%2E05241%2C+2021%2E+%E2%80%94+%5BarXiv%3A2105%2E05241%5D%28htt)
8. Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. The Pushshift Reddit dataset. In Proceedings of the international AAAI conference on web and social media, volume 14, pages 830–839, 2020. [find](https://scholar.google.com/scholar?q=Jason+Baumgartner%2C+Savvas+Zannettou%2C+Brian+Keegan%2C+Megan+Squire%2C+and+Jeremy+Blackburn%2E+The+Pushshift+Reddit+dataset%2E+In+Proceedings+of+the+international+AAAI+conference+on+web+and+social+media%2C+volume)
9. Emily M Bender. Linguistic fundamentals for natural language processing: 100 essentials from morphology and syntax. Synthesis lectures on human language technologies, 6(3):1–184, 2013. [find](https://scholar.google.com/scholar?q=Emily+M+Bender%2E+Linguistic+fundamentals+for+natural+language+processing%3A+100+essentials+from+morphology+and+syntax%2E+Synthesis+lectures+on+human+language+technologies%2C+6%283%29%3A1%E2%80%93184%2C+2013%2E)
10. Emily M. Bender and Batya Friedman. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6: 587–604, 2018. doi: 10.1162/tacl_a_00041. URL https://aclanthology.org/Q18-1041. [↗](https://aclanthology.org/Q18-1041)
11. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373, 2023. [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+et+al%2E+Pythia%3A+A+suite+for+a)
12. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. [GPT-NeoX-20B: An open-source autoregressive language model](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/). Challenges & Perspectives in Creating Large Language Models, page 95, 2022.
13. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
14. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. [One billion word benchmark for measuring progress in statistical language modeling](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/). arXiv preprint arXiv:1312.3005, 2013.
15. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021. URL https://arxiv.org/abs/2107.03374. [↗](https://arxiv.org/abs/2107.03374)
16. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, et al. [PaLM: Scaling language modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311, 2022. URL https://arxiv.org/abs/2204.02311. [↗](https://arxiv.org/abs/2204.02311)
17. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXiv+prep)
18. Hyung Won Chung, Xavier Garcia, Adam Roberts, Yi Tay, Orhan Firat, Sharan Narang, and Noah Constant. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining. In The Eleventh International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Xavier+Garcia%2C+Adam+Roberts%2C+Yi+Tay%2C+Orhan+Firat%2C+Sharan+Narang%2C+and+Noah+Constant%2E+Unimax%3A+Fairer+and+more+effective+language+sampling+for+large%2Dscale+multilingual+pretraining%2E+In+Th)
19. Cohere AI. Cohere command nightly, 2023. URL [https://docs.cohere.com/docs/command-beta](https://docs.cohere.com/docs/command-beta). [↗](https://docs.cohere.com/docs/command-beta)
20. Corinna Cortes and Neil D Lawrence. Inconsistency in conference peer review: Revisiting the 2014 NeurIPS experiment. arXiv preprint arXiv:2109.09774, 2021. — [arXiv:2109.09774](https://arxiv.org/abs/2109.09774) [find](https://scholar.google.com/scholar?q=Corinna+Cortes+and+Neil+D+Lawrence%2E+Inconsistency+in+conference+peer+review%3A+Revisiting+the+2014+NeurIPS+experiment%2E+arXiv+preprint+arXiv%3A2109%2E09774%2C+2021%2E+%E2%80%94+%5BarXiv%3A2109%2E09774%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
21. Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. [Plug and play language models: A simple approach to controlled text generation](https://md.research.elamur.ai/free/en/papers/dathathri-2019-plug-play-language-models-simple/). In International Conference on Learning Representations, 2020.
22. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). NAACL, 2019. URL https://aclanthology.org/N19-1423. [↗](https://aclanthology.org/N19-1423)
23. Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. [Time-aware language models as temporal knowledge bases](https://md.research.elamur.ai/free/en/papers/dhingra-2022-time-aware-language-models-as/). Transactions of the Association for Computational Linguistics, 10:257–273, 2022.
24. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the Colossal Clean Crawled Corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processi… [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Maarten+Sap%2C+Ana+Marasovi%C4%87%2C+William+Agnew%2C+Gabriel+Ilharco%2C+Dirk+Groeneveld%2C+Margaret+Mitchell%2C+and+Matt+Gardner%2E+Documenting+large+webtext+corpora%3A+A+case+study+on+the+Colossal+Clean+Cra)
25. Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas… — [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/)
26. Jacob Eisenstein, Brendan O’Connor, Noah A Smith, and Eric P Xing. Diffusion of lexical change in social media. PloS one, 9(11):e113114, 2014. [find](https://scholar.google.com/scholar?q=Jacob+Eisenstein%2C+Brendan+O%E2%80%99Connor%2C+Noah+A+Smith%2C+and+Eric+P+Xing%2E+Diffusion+of+lexical+change+in+social+media%2E+PloS+one%2C+9%2811%29%3Ae113114%2C+2014%2E)
27. István Endrédy and Attila Novák. More effective boilerplate removal-the goldminer algorithm. Polibits, 48: 79–83, 2013. [find](https://scholar.google.com/scholar?q=Istv%C3%A1n+Endr%C3%A9dy+and+Attila+Nov%C3%A1k%2E+More+effective+boilerplate+removal%2Dthe+goldminer+algorithm%2E+Polibits%2C+48%3A+79%E2%80%9383%2C+2013%2E)
28. Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. [Mrqa 2019 shared task: Evaluating generalization in reading comprehension](https://md.research.elamur.ai/free/en/papers/fisch-2019-mrqa-2019-shared-task-evaluating/). In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 1–13, 2019.
29. Paul Friedl. Dis/similarities in the design and development of legal and algorithmic normative systems: the case of perspective api. Law, Innovation and Technology, pages 1–35, 2023. [find](https://scholar.google.com/scholar?q=Paul+Friedl%2E+Dis%2Fsimilarities+in+the+design+and+development+of+legal+and+algorithmic+normative+systems%3A+the+case+of+perspective+api%2E+Law%2C+Innovation+and+Technology%2C+pages+1%E2%80%9335%2C+2023%2E)
30. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+et+al%2E+The+Pile%3A+An+800GB+dataset+of+diverse+text+for+language+)
31. Tianyu Gao, Adam Fisch, and Danqi Chen. [Making pre-trained language models better few-shot learners](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/). ACL, 2021. doi: 10.18653/v1/2021.acl-long.295. URL https://aclanthology.org/2021.acl-long.295. [↗](https://aclanthology.org/2021.acl-long.295)
32. Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. Evaluating models’ local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP 2020, [pages 130…](https://md.research.elamur.ai/free/en/papers/gardner-2020-evaluating-models-local-decision-boundaries/)
33. SK Gargee, Pranav Bhargav Gopinath, Shridhar Reddy SR Kancharla, CR Anand, and Anoop S Babu. Analyzing and addressing the difference in toxicity prediction between different comments with same semantic meaning in Google’s Perspective API. In ICT Systems and Sustainability: Proceedings of ICT4SD 2022… [find](https://scholar.google.com/scholar?q=SK+Gargee%2C+Pranav+Bhargav+Gopinath%2C+Shridhar+Reddy+SR+Kancharla%2C+CR+Anand%2C+and+Anoop+S+Babu%2E+Analyzing+and+addressing+the+difference+in+toxicity+prediction+between+different+comments+with+same+semanti)
34. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III au2, and Kate Crawford. Datasheets for datasets, 2021. [find](https://scholar.google.com/scholar?q=Timnit+Gebru%2C+Jamie+Morgenstern%2C+Briana+Vecchione%2C+Jennifer+Wortman+Vaughan%2C+Hanna+Wallach%2C+Hal+Daum%C3%A9+III+au2%2C+and+Kate+Crawford%2E+Datasheets+for+datasets%2C+2021%2E)
35. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. [RealToxicityPrompts: Evaluating neural toxic degeneration in language models](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, 2020.
36. Aaron Gokaslan*, Vanya Cohen*, Ellie Pavlick, and Stefanie Tellex. OpenWebText corpus, 2019. URL [http://Skylion007.github.io/OpenWebTextCorpus](http://Skylion007.github.io/OpenWebTextCorpus). [↗](http://Skylion007.github.io/OpenWebTextCorpus)
37. Google. [PaLM 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/), 2023. URL [https://ai.google/static/documents/palm2techreport](https://ai.google/static/documents/palm2techreport). pdf. [↗](https://ai.google/static/documents/palm2techreport)
38. Google Cloud NLP. Google Cloud infotype detector, 2023a. URL [https://cloud.google.com/dlp/docs/infotypes-reference](https://cloud.google.com/dlp/docs/infotypes-reference). [↗](https://cloud.google.com/dlp/docs/infotypes-reference)
39. Google Cloud NLP. Google Cloud analyzing sentiment, 2023b. URL [https://cloud.google.com/natural-language/docs/analyzing-sentiment](https://cloud.google.com/natural-language/docs/analyzing-sentiment). [↗](https://cloud.google.com/natural-language/docs/analyzing-sentiment)
40. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, 2020. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Ana+Marasovi%C4%87%2C+Swabha+Swayamdipta%2C+Kyle+Lo%2C+Iz+Beltagy%2C+Doug+Downey%2C+and+Noah+A+Smith%2E+Don%E2%80%99t+stop+pretraining%3A+Adapt+language+models+to+domains+and+tasks%2E+In+Proceedings+of+the+58th)
41. Suchin Gururangan, Dallas Card, Sarah K Drier, Emily K Gade, Leroy Z Wang, Zeyu Wang, Luke Zettlemoyer, and Noah A Smith. Whose language counts as high quality? Measuring language ideologies in text data selection. arXiv preprint arXiv:2201.10474, 2022. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Dallas+Card%2C+Sarah+K+Drier%2C+Emily+K+Gade%2C+Leroy+Z+Wang%2C+Zeyu+Wang%2C+Luke+Zettlemoyer%2C+and+Noah+A+Smith%2E+Whose+language+counts+as+high+quality%3F+Measuring+language+ideologies+in+text+d)
42. Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. [Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection](https://md.research.elamur.ai/free/en/papers/hartvigsen-2022-toxigen-large-scale-machine-generated/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long…
43. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). arXiv preprint arXiv:2203.15556, 2022.
44. Dan Iter and David Grangier. On the complementarity of data selection and fine tuning for domain adaptation. arXiv preprint arXiv:2109.07591, 2021. — [arXiv:2109.07591](https://arxiv.org/abs/2109.07591) [find](https://scholar.google.com/scholar?q=Dan+Iter+and+David+Grangier%2E+On+the+complementarity+of+data+selection+and+fine+tuning+for+domain+adaptation%2E+arXiv+preprint+arXiv%3A2109%2E07591%2C+2021%2E+%E2%80%94+%5BarXiv%3A2109%2E07591%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2109%2E0759)
45. Kokil Jaidka, Niyati Chhaya, and Lyle Ungar. Diachronic degradation of language models: Insights from social media. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 195–200, Melbourne, Australia, July 2018. Association for Com… [find](https://scholar.google.com/scholar?q=Kokil+Jaidka%2C+Niyati+Chhaya%2C+and+Lyle+Ungar%2E+Diachronic+degradation+of+language+models%3A+Insights+from+social+media%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Lingu)
46. Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. TemporalWiki: A lifelong benchmark for training and evaluating ever-evolving language models. arXiv preprint arXiv:2204.14211, 2022. [find](https://scholar.google.com/scholar?q=Joel+Jang%2C+Seonghyeon+Ye%2C+Changho+Lee%2C+Sohee+Yang%2C+Joongbo+Shin%2C+Janghoon+Han%2C+Gyeonghun+Kim%2C+and+Minjoon+Seo%2E+TemporalWiki%3A+A+lifelong+benchmark+for+training+and+evaluating+ever%2Devolving+language+mod)
47. Jean Kaddour. [The minipile challenge for data-efficient language models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-minipile-challenge-data-efficient-language/). arXiv preprint arXiv:2304.08442, 2023.
48. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361, 2020. URL https://arxiv.org/abs/2001.08361. [↗](https://arxiv.org/abs/2001.08361)
49. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. [UnifiedQA: Crossing format boundaries with a single QA system](https://md.research.elamur.ai/free/en/papers/khashabi-2020-unifiedqa-crossing-format-boundaries-single/). In Findings of the Association for Computational Linguistics: EMNLP 2020, 2020. URL https://aclanthology.org/2020.findings-e… [↗](https://aclanthology.org/2020.findings-e…)
50. J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, Naval Technical Training Command Millington TN Research Branch,… [find](https://scholar.google.com/scholar?q=J+Peter+Kincaid%2C+Robert+P+Fishburne+Jr%2C+Richard+L+Rogers%2C+and+Brad+S+Chissom%2E+Derivation+of+new+readability+formulas+%28automated+readability+index%2C+fog+count+and+flesch+reading+ease+formula%29+for+navy+e)
51. Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al. [Quality at a glance: An audit of web-crawled multilingual datasets](https://md.research.elamur.ai/free/en/papers/kreutzer-2021-quality-glance-audit-web-crawled/). Transactions of the Association for Computational Linguistic…
52. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. [Natural Questions: a benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Linguistics, 7:4…
53. William Labov. Principles of linguistic change, volume 3: Cognitive and cultural factors, volume 3. John Wiley & Sons, 2011. [find](https://scholar.google.com/scholar?q=William+Labov%2E+Principles+of+linguistic+change%2C+volume+3%3A+Cognitive+and+cultural+factors%2C+volume+3%2E+John+Wiley+%26+Sons%2C+2011%2E)
54. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. [ALBERT: A lite BERT for self-supervised learning of language representations](https://md.research.elamur.ai/free/en/papers/lan-2019-albert-lite-bert-self-supervised/). In International Conference on Learning Representations, 2020.
55. Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. [The BigScience ROOTS Corpus: A 1.6TB composite multilingual dataset](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2023-bigscience-roots-corpus-1-6tb/). In Thirty-sixth Conference on Neural Infor…
56. Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomas Kocisky, Sebastian Ruder, et al. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural Information Processing Systems,… [find](https://scholar.google.com/scholar?q=Angeliki+Lazaridou%2C+Adhi+Kuncoro%2C+Elena+Gribovskaya%2C+Devang+Agrawal%2C+Adam+Liska%2C+Tayfun+Terzi%2C+Mai+Gimenez%2C+Cyprien+de+Masson+d%E2%80%99Autume%2C+Tomas+Kocisky%2C+Sebastian+Ruder%2C+et+al%2E+Mind+the+gap%3A+Assessing+t)
57. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages… [find](https://scholar.google.com/scholar?q=Katherine+Lee%2C+Daphne+Ippolito%2C+Andrew+Nystrom%2C+Chiyuan+Zhang%2C+Douglas+Eck%2C+Chris+Callison%2DBurch%2C+and+Nicholas+Carlini%2E+Deduplicating+training+data+makes+language+models+better%2E+In+Proceedings+of+the+)
58. Alyssa Lees, Vinh Q Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman. [A new generation of Perspective API: Efficient multilingual character-level transformers](https://md.research.elamur.ai/free/en/papers/lees-2022-new-generation-perspective-api/). arXiv preprint arXiv:2202.11176, 2022. URL [http://perspectiveapi.com.com](http://perspectiveapi.com.com). [↗](http://perspectiveapi.com.com)
59. Adam Liska, Tomas Kocisky, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien De Masson D’Autume, Tim Scholtes, Manzil Zaheer, Susannah Young, Ellen Gilsenan-Mcmahon, Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. StreamingQA: A benchmark for adaptation to new knowledge ove… [find](https://scholar.google.com/scholar?q=Adam+Liska%2C+Tomas+Kocisky%2C+Elena+Gribovskaya%2C+Tayfun+Terzi%2C+Eren+Sezener%2C+Devang+Agrawal%2C+Cyprien+De+Masson+D%E2%80%99Autume%2C+Tim+Scholtes%2C+Manzil+Zaheer%2C+Susannah+Young%2C+Ellen+Gilsenan%2DMcmahon%2C+Sophia+Austin)
60. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. [RoBERTa: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692, 2019.
61. Shayne Longpre, Yi Lu, Zhucheng Tu, and Chris DuBois. An exploration of data augmentation and sampling techniques for domain-agnostic question answering. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 220–227, 2019. [find](https://scholar.google.com/scholar?q=Shayne+Longpre%2C+Yi+Lu%2C+Zhucheng+Tu%2C+and+Chris+DuBois%2E+An+exploration+of+data+augmentation+and+sampling+techniques+for+domain%2Dagnostic+question+answering%2E+In+Proceedings+of+the+2nd+Workshop+on+Machine+)
62. Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. [Entity-based knowledge conflicts in question answering](https://md.research.elamur.ai/free/en/papers/longpre-2021-entity-based-knowledge-conflicts-question/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7052–7063, 2021.
63. Shayne Longpre, Julia Rachel Reisler, Edward Greg Huang, Yi Lu, Andrew Frank, Nikhil Ramesh, and Christopher DuBois. Active learning over multiple domains in natural language tasks. In NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications, 2022. [find](https://scholar.google.com/scholar?q=Shayne+Longpre%2C+Julia+Rachel+Reisler%2C+Edward+Greg+Huang%2C+Yi+Lu%2C+Andrew+Frank%2C+Nikhil+Ramesh%2C+and+Christopher+DuBois%2E+Active+learning+over+multiple+domains+in+natural+language+tasks%2E+In+NeurIPS+2022+Wo)
64. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. [The Flan collection: Designing data and methods for effective instruction tuning](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/). arXiv preprint arXiv:2301.13688, 2023.
65. Alexandra Sasha Luccioni and Joseph D Viviano. What’s in the box? a preliminary analysis of undesirable content in the Common Crawl corpus. arXiv preprint arXiv:2105.02732, 2021. — [arXiv:2105.02732](https://arxiv.org/abs/2105.02732) [find](https://scholar.google.com/scholar?q=Alexandra+Sasha+Luccioni+and+Joseph+D+Viviano%2E+What%E2%80%99s+in+the+box%3F+a+preliminary+analysis+of+undesirable+content+in+the+Common+Crawl+corpus%2E+arXiv+preprint+arXiv%3A2105%2E02732%2C+2021%2E+%E2%80%94+%5BarXiv%3A2105%2E02732%5D%28)
66. Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Karishma Mandyam, and Noah A Smith. [Time waits for no one! analysis and challenges of temporal misalignment](https://md.research.elamur.ai/free/en/papers/luu-2022-time-waits-no-one-analysis/). arXiv preprint arXiv:2111.07408, 2021.
67. Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. [Language models of code are few-shot commonsense learners](https://md.research.elamur.ai/free/en/papers/madaan-2022-language-models-code-few-shot/), 2022.
68. Nicholas Meade, Elinor Poole-Dayan, and Siva Reddy. [An empirical survey of the effectiveness of debiasing techniques for pre-trained language models](https://md.research.elamur.ai/free/en/papers/meade-2022-empirical-survey-effectiveness-debiasing-techniques/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1878–1898, 2022.
69. Rajiv Movva, Jinhao Lei, Shayne Longpre, Ajay Gupta, and Chris DuBois. Combining compressions for multiplicative size scaling on natural language tasks. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2861–2872, 2022. [find](https://scholar.google.com/scholar?q=Rajiv+Movva%2C+Jinhao+Lei%2C+Shayne+Longpre%2C+Ajay+Gupta%2C+and+Chris+DuBois%2E+Combining+compressions+for+multiplicative+size+scaling+on+natural+language+tasks%2E+In+Proceedings+of+the+29th+International+Confer)
70. Alex Nichol. DALL-E 2 pre-training mitigations, 2022. URL [https://openai.com/research/dall-e-2-pre-training-mitigations](https://openai.com/research/dall-e-2-pre-training-mitigations). [↗](https://openai.com/research/dall-e-2-pre-training-mitigations)
71. Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, pages 16784–16804. PMLR, 2… [find](https://scholar.google.com/scholar?q=Alexander+Quinn+Nichol%2C+Prafulla+Dhariwal%2C+Aditya+Ramesh%2C+Pranav+Shyam%2C+Pamela+Mishkin%2C+Bob+Mcgrew%2C+Ilya+Sutskever%2C+and+Mark+Chen%2E+Glide%3A+Towards+photorealistic+image+generation+and+editing+with+text%2D)
72. Nostalgebraist. Chinchilla’s wild implications. AI Alignment Forum, 2022. URL [https://www.alignmentforum.org/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications](https://www.alignmentforum.org/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications). [↗](https://www.alignmentforum.org/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications)
73. The Open Team NYT. To apply machine learning responsibly, we use it in moderation, 2020. URL [https://open.nytimes.com/to-apply-machine-learning-responsibly-we-use-it-in-moderation-d001f49e0644](https://open.nytimes.com/to-apply-machine-learning-responsibly-we-use-it-in-moderation-d001f49e0644). [↗](https://open.nytimes.com/to-apply-machine-learning-responsibly-we-use-it-in-moderation-d001f49e0644)
74. OpenAI. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arxiv:2303.08774, 2023. URL [https://arxiv.org/pdf/2303.08774.pdf](https://arxiv.org/pdf/2303.08774.pdf). [↗](https://arxiv.org/pdf/2303.08774.pdf)
75. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). arXiv preprint arXiv:2203.02155, 2022. URL https://arxiv.org/abs/2203.02155. [↗](https://arxiv.org/abs/2203.02155)
76. Isabel Papadimitriou and Dan Jurafsky. Learning music helps you read: Using transfer to study linguistic structure in language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6829–6839, 2020. [find](https://scholar.google.com/scholar?q=Isabel+Papadimitriou+and+Dan+Jurafsky%2E+Learning+music+helps+you+read%3A+Using+transfer+to+study+linguistic+structure+in+language+models%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Nat)
77. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). NAACL, 2018. URL https://aclanthology.org/N18-1202. [↗](https://aclanthology.org/N18-1202)
78. Luiza Amador Pozzobon, Beyza Ermis, Patrick Lewis, and Sara Hooker. On the challenges of using black-box APIs for toxicity evaluation in research. In ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models, 2023. [find](https://scholar.google.com/scholar?q=Luiza+Amador+Pozzobon%2C+Beyza+Ermis%2C+Patrick+Lewis%2C+and+Sara+Hooker%2E+On+the+challenges+of+using+black%2Dbox+APIs+for+toxicity+evaluation+in+research%2E+In+ICLR+2023+Workshop+on+Trustworthy+and+Reliable+Lar)
79. Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel Bowman. [Intermediate-task transfer learning with pretrained language models: When and why does it work?](https://md.research.elamur.ai/free/en/papers/pruksachatkun-2020-intermediate-task-transfer-learning-pretrained/) In Proceedings of the 58th Annual Meeting of the Association…
80. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. URL [https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf). [↗](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)
81. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training Gopher. arXiv preprint arXiv:2112.11446, 2021. URL [https://arxiv.org/abs…](https://arxiv.org/abs…) [↗](https://arxiv.org/abs…)
82. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21:1–67, 2020. URL https://arxiv.org/abs/1910.10683. [↗](https://arxiv.org/abs/1910.10683)
83. Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashch… — [Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$](https://md.research.elamur.ai/free/en/papers/roberts-2022-scaling-up-models-data-texttt/)
84. Anna Rogers. Changing the world by changing the data. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2182–2194, 2021. [find](https://scholar.google.com/scholar?q=Anna+Rogers%2E+Changing+the+world+by+changing+the+data%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11th+International+Joint+Conference+on+Natural+)
85. Paul Röttger and Janet Pierrehumbert. Temporal adaptation of BERT and performance on downstream document classification: Insights from social media. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2400–2412, Punta Cana, Dominican Republic, November 2021. Association f… [find](https://scholar.google.com/scholar?q=Paul+R%C3%B6ttger+and+Janet+Pierrehumbert%2E+Temporal+adaptation+of+BERT+and+performance+on+downstream+document+classification%3A+Insights+from+social+media%2E+In+Findings+of+the+Association+for+Computational+Li)
86. Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. “Everyone wants to do the model work, not the data work”: Data cascades in high-stakes AI. In CHI, CHI ’21, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450380966. doi: 10… — [doi:10.1145/3411764.3445518](https://doi.org/10.1145/3411764.3445518) [find](https://scholar.google.com/scholar?q=Nithya+Sambasivan%2C+Shivani+Kapania%2C+Hannah+Highfill%2C+Diana+Akrong%2C+Praveen+Paritosh%2C+and+Lora+M+Aroyo%2E+%E2%80%9CEveryone+wants+to+do+the+model+work%2C+not+the+data+work%E2%80%9D%3A+Data+cascades+in+high%2Dstakes+AI%2E+In+CHI)
87. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. [Social bias frames: Reasoning about social and power implications of language](https://md.research.elamur.ai/free/en/papers/sap-2020-social-bias-frames-reasoning-about/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477–5490, Online, July 2020. Associ…
88. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. BLOOM: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ili%C4%87%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+et+al%2E+BLOOM%3A+A+176b%2Dparameter+open%2Da)
89. John Schulman. Reinforcement learning from human feedback: Progress and challenges. Berkeley EECS, 2023. URL [https://www.youtube.com/watch?v=hhiLw5Q_UFg](https://www.youtube.com/watch?v=hhiLw5Q_UFg). [↗](https://www.youtube.com/watch?v=hhiLw5Q_UFg)
90. Thibault Sellam, Steve Yadlowsky, Ian Tenney, Jason Wei, Naomi Saphra, Alexander D’Amour, Tal Linzen, Jasmijn Bastings, Iulia Raluca Turc, Jacob Eisenstein, et al. The MultiBERTs: BERT reproductions for robustness analysis. In International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Steve+Yadlowsky%2C+Ian+Tenney%2C+Jason+Wei%2C+Naomi+Saphra%2C+Alexander+D%E2%80%99Amour%2C+Tal+Linzen%2C+Jasmijn+Bastings%2C+Iulia+Raluca+Turc%2C+Jacob+Eisenstein%2C+et+al%2E+The+MultiBERTs%3A+BERT+reproductions+f)
91. Ayush Singh and John E Ortega. Addressing distribution shift at test time in pre-trained language models. arXiv preprint arXiv:2212.02384, 2022. — [arXiv:2212.02384](https://arxiv.org/abs/2212.02384) [find](https://scholar.google.com/scholar?q=Ayush+Singh+and+John+E+Ortega%2E+Addressing+distribution+shift+at+test+time+in+pre%2Dtrained+language+models%2E+arXiv+preprint+arXiv%3A2212%2E02384%2C+2022%2E+%E2%80%94+%5BarXiv%3A2212%2E02384%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2212%2E02384%29)
92. Spandana Singh. Everything in moderation: An analysis of how internet platforms are using artificial intelligence to moderate user-generated content. New America, 2019. URL [https://www.newamerica.org/oti/reports/everything-moderation-analysis-how-internet-platforms-are-using-artificial-intelligence-…](https://www.newamerica.org/oti/reports/everything-moderation-analysis-how-internet-platforms-are-using-artificial-intelligence-…) [↗](https://www.newamerica.org/oti/reports/everything-moderation-analysis-how-internet-platforms-are-using-artificial-intelligence-…)
93. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: An instruction-following LLaMA model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
94. Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. [Scaling laws vs model architectures: How does inductive bias influence scaling?](https://md.research.elamur.ai/free/en/papers/tay-2023-scaling-laws-vs-model-architectures/) arXiv preprint arXiv:2207.10551, 2022.
95. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. [LaMDA: Language models for dialog applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/). arXiv preprint arXiv:2201.08239, 2022. URL https://arxiv.org/abs/2201.08239. [↗](https://arxiv.org/abs/2201.08239)
96. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [LLaMA: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
97. Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. [Learning from the worst: Dynamically generated datasets to improve online hate detection](https://md.research.elamur.ai/free/en/papers/vidgen-2021-learning-worst-dynamically-generated-datasets/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural L…
98. Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. Will we run out of data? an analysis of the limits of scaling datasets in machine learning. arXiv preprint arXiv:2211.[04325, 2022](https://md.research.elamur.ai/free/en/papers/villalobos-2022-will-run-out-data-limits/).
99. Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective work best for zero-shot generalization? ICML, 2022. URL https://arxiv.org/abs/2204.05832. [↗](https://arxiv.org/abs/2204.05832)
100. Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. Balancing training for multilingual neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8526–8537, 2020. [find](https://scholar.google.com/scholar?q=Xinyi+Wang%2C+Yulia+Tsvetkov%2C+and+Graham+Neubig%2E+Balancing+training+for+multilingual+neural+machine+translation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistic)
101. Alex Warstadt, Yian Zhang, Haau-Sing Li, Haokun Liu, and Samuel R Bowman. Learning which features matter: RoBERTa acquires a preference for linguistic generalizations (eventually). arXiv preprint arXiv:2010.05358, 2020. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Yian+Zhang%2C+Haau%2DSing+Li%2C+Haokun+Liu%2C+and+Samuel+R+Bowman%2E+Learning+which+features+matter%3A+RoBERTa+acquires+a+preference+for+linguistic+generalizations+%28eventually%29%2E+arXiv+preprint+arXi)
102. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). TMLR, 2022. URL [https://openreview.net/forum?id=yzkSU5zdwD](https://openreview.net/forum?id=yzkSU5zdwD). [↗](https://openreview.net/forum?id=yzkSU5zdwD)
103. Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. [Challenges in detoxifying language models](https://md.research.elamur.ai/free/en/papers/welbl-2021-challenges-detoxifying-language-models/). In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2447–2469, 20…
104. Yuhuai Wu, Felix Li, and Percy Liang. Insights into pre-training via simpler synthetic tasks. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Yuhuai+Wu%2C+Felix+Li%2C+and+Percy+Liang%2E+Insights+into+pre%2Dtraining+via+simpler+synthetic+tasks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2022%2E)
105. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. DoReMi: Optimizing data mixtures speeds up language model pretraining. arXiv preprint arXiv:2305.10429, 2023a. [find](https://scholar.google.com/scholar?q=Sang+Michael+Xie%2C+Hieu+Pham%2C+Xuanyi+Dong%2C+Nan+Du%2C+Hanxiao+Liu%2C+Yifeng+Lu%2C+Percy+Liang%2C+Quoc+V+Le%2C+Tengyu+Ma%2C+and+Adams+Wei+Yu%2E+DoReMi%3A+Optimizing+data+mixtures+speeds+up+language+model+pretraining%2E+ar)
106. Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. arXiv preprint arXiv:2302.03169, 2023b. [find](https://scholar.google.com/scholar?q=Sang+Michael+Xie%2C+Shibani+Santurkar%2C+Tengyu+Ma%2C+and+Percy+Liang%2E+Data+selection+for+language+models+via+importance+resampling%2E+arXiv+preprint+arXiv%3A2302%2E03169%2C+2023b%2E)
107. Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, and Dan Klein. [Detoxifying language models risks marginalizing minority voices](https://md.research.elamur.ai/free/en/papers/xu-2021-detoxifying-language-models-risks-marginalizing/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p…
108. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. [mT5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistic…
109. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. [XLNet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). Advances in neural information processing systems, 32, 2019.
110. Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei Koh, and Chelsea Finn. Wild-Time: A benchmark of in-the-wild distribution shift over time. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. [find](https://scholar.google.com/scholar?q=Huaxiu+Yao%2C+Caroline+Choi%2C+Bochuan+Cao%2C+Yoonho+Lee%2C+Pang+Wei+Koh%2C+and+Chelsea+Finn%2E+Wild%2DTime%3A+A+benchmark+of+in%2Dthe%2Dwild+distribution+shift+over+time%2E+In+Thirty%2Dsixth+Conference+on+Neural+Information)
111. Michael Zhang and Eunsol Choi. SituatedQA: Incorporating extra-linguistic contexts into qa. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7371–7387, 2021. [find](https://scholar.google.com/scholar?q=Michael+Zhang+and+Eunsol+Choi%2E+SituatedQA%3A+Incorporating+extra%2Dlinguistic+contexts+into+qa%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+7371%E2%80%937387%2C+)
112. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. [OPT: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068, 2022.
113. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. [A survey of large language models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/). arXiv preprint arXiv:2303.18223, 2023.
114. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. [Aligning books and movies: Towards story-like visual explanations by watching movies and reading books](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/). In Proceedings of the IEEE international conference on computer vision, pages 19–27, 20…

*Updated 2 November 2026*