---
title: "What Language Model to Train if You Have One Million GPU"
description: "The BigScience collaboration had a fixed allowance of roughly one million A100 GPU-hours to build… 80 references, cited by 67 papers here."
authors: ["Teven Le Scao", "Thomas Wang", "Daniel Hesslow", "Lucile Saulnier", "Stas Bekman", "M Saiful Bari", "Stella Biderman", "Hady Elsahar", "Niklas Muennighoff", "Jason Phang", "Ofir Press", "Colin Raffel", "Victor Sanh", "Sheng Shen", "Lintang Sutawika", "Jaesung Tae", "Zheng Xin Yong", "Julien Launay", "Iz Beltagy"]
year: 2022
venue: "arXiv:2210.15424"
source: "https://arxiv.org/abs/2210.15424"
doi: "10.48550/arxiv.2210.15424"
arxiv: "2210.15424"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/scao-2022-language-model-train-if-have/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/scao-2022-language-model-train-if-have/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/) / What Language Model to Train if You Have One Million GPU Hours?

# What Language Model to Train if You Have One Million GPU Hours?

**Authors:** [Teven Le Scao](https://md.research.elamur.ai/free/en/authors/teven-le-scao/), [Thomas Wang](https://md.research.elamur.ai/free/en/authors/thomas-wang/), [Daniel Hesslow](https://md.research.elamur.ai/free/en/authors/daniel-hesslow/), [Lucile Saulnier](https://md.research.elamur.ai/free/en/authors/lucile-saulnier/), [Stas Bekman](https://md.research.elamur.ai/free/en/authors/stas-bekman/), [M Saiful Bari](https://md.research.elamur.ai/free/en/authors/m-saiful-bari/), [Stella Biderman](https://md.research.elamur.ai/free/en/authors/stella-biderman/), [Hady Elsahar](https://md.research.elamur.ai/free/en/authors/hady-elsahar/), [Niklas Muennighoff](https://md.research.elamur.ai/free/en/authors/niklas-muennighoff/), [Jason Phang](https://md.research.elamur.ai/free/en/authors/jason-phang/), [Ofir Press](https://md.research.elamur.ai/free/en/authors/ofir-press/), [Colin Raffel](https://md.research.elamur.ai/free/en/authors/colin-raffel/), [Victor Sanh](https://md.research.elamur.ai/free/en/authors/victor-sanh/), [Sheng Shen](https://md.research.elamur.ai/free/en/authors/sheng-shen/), [Lintang Sutawika](https://md.research.elamur.ai/free/en/authors/lintang-sutawika/), Jaesung Tae, [Zheng Xin Yong](https://md.research.elamur.ai/free/en/authors/zheng-xin-yong/), [Julien Launay](https://md.research.elamur.ai/free/en/authors/julien-launay/) and [Iz Beltagy](https://md.research.elamur.ai/free/en/authors/iz-beltagy/) · **Year:** 2022 · **Venue:** arXiv:2210.15424 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2210.15424) · [PDF](https://arxiv.org/pdf/2210.15424) · [DOI 10.48550/arxiv.2210.15424](https://doi.org/10.48550/arxiv.2210.15424) · [arXiv 2210.15424](https://arxiv.org/abs/2210.15424) · [OpenAlex](https://openalex.org/W4307536954)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [67](https://md.research.elamur.ai/free/en/papers/scao-2022-language-model-train-if-have/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The BigScience collaboration had a fixed allowance of roughly one million A100 GPU-hours to build BLOOM, a massively multilingual model with more than 100B parameters. Because large-scale trial and error was unaffordable, the team ran controlled ablations on smaller decoder-only, GPT-2-style models. Most runs used 1.3B parameters trained on 112B tokens, and each run was judged by zero-shot accuracy on 27 tasks from the EleutherAI Evaluation Harness.

The experiments compared three pretraining corpora (OSCAR v1, C4 and The Pile), several positional embedding schemes, activation functions, embedding normalization, and multilingual versus English-only training. They also fitted scaling laws, both overall and per language, to choose the size and shape of the final model.

Key findings:
- Curated, cross-domain data beats filtered web crawl alone.
- ALiBi is the best positional option, and SwiGLU looks promising.
- Embedding normalization improves stability but can hurt zero-shot quality.
- Multilingual models fall behind English-only ones on English benchmarks.

These results shaped the 176B BLOOM design. All models, code and meeting notes were released openly, which makes the work a practical guide for teams training models from 1B to 100B+ parameters.

## Key points
- **Budget and setup:** 1,000,000 A100-hours in total. Ablation models had 1.3B parameters and were trained on 112B tokens with GPT-3 hyperparameters, full attention and a 2,048-token context. Evaluation used 27 EAI harness tasks, where random chance is 33.3%.
- **Corpus comparison at 112B tokens (average accuracy):** The Pile 42.79, C4 42.77, OSCAR v1 41.72. A 13B model trained on OSCAR for 300B tokens still trailed OpenAI's Curie (47.09 vs 49.28), which the authors attribute to the weak quality of OSCAR's English text.
- **Choice of metric:** Zero-shot accuracy served as the main signal, supported by a Pearson correlation of 0.93 between zero-shot and few-shot scores in GPT-3 results.
- **Scaling behaviour:** Loss followed a compute power law with α_C ≈ 0.046, consistent with Kaplan et al. Per-language scaling curves were similar across languages and differed mostly in their loss offset.
- **Final BLOOM shape:** 176B parameters, 70 layers, hidden size 14,336 and 112 attention heads of dimension 128. It was chosen from 20 benchmarked configurations for its higher throughput, about 150 TFLOPs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 402 works · 212,763 citations
- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 304 works · 106,137 citations
- [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 219 works · 103,487 citations
- [Brown University](https://md.research.elamur.ai/free/en/universities/brown-university/) — University, Providence, United States · 80 works · 26,036 citations
- [Nanyang Technological University](https://md.research.elamur.ai/free/en/universities/nanyang-technological-university/) — University, Singapore, Singapore · 184 works · 27,484 citations

Also on the byline: [Laboratoire de Physique de l'Ecole Normale Supérieure](https://ror.org/03a26mh11).

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 882 works · [⚡ GPU & parallel computing](https://md.research.elamur.ai/free/en/topics/gpu-parallel-computing/) · 41 works

## Cite

Cite as: Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Niklas Muennighoff, Jason Phang, Ofir Press, Colin Raffel, Victor Sanh, Sheng Shen, Lintang Sutawika, Jaesung Tae, Zheng Xin Yong, Julien Launay and Iz Beltagy. 2022. What Language Model to Train if You Have One Million GPU Hours?. arXiv:2210.15424.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{scao2022what,
  title = {{What Language Model to Train if You Have One Million GPU Hours?}},
  author = {Teven {Le Scao}, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Niklas Muennighoff, Jason Phang, Ofir Press, Colin Raffel, Victor Sanh, Sheng Shen, Lintang Sutawika, Jaesung Tae, Zheng Xin Yong, Julien Launay and Iz Beltagy},
  year = {2022},
  doi = {10.48550/arxiv.2210.15424},
  eprint = {2210.15424},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2210.15424}
}
```

- [← previous Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation](https://md.research.elamur.ai/free/en/papers/wang-2022-expanding-pretrained-models-thousands-more/)
- [next → Wojood: Nested Arabic Named Entity Corpus and Recognition using BERT](https://md.research.elamur.ai/free/en/papers/jarrar-2022-wojood-nested-arabic-named-entity/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/)

## References

80 references, 48 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. [MathQA: Towards interpretable math word problem solving with operation-based formalisms](https://md.research.elamur.ai/free/en/papers/amini-2019-mathqa-interpretable-math-word-problem/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computation…
2. Stéphane Aroca-Ouellette, Cory Paik, Alessandro Roncone, and Katharina Kann. 2021. PROST: Physical reasoning about objects through space and time. In Budget [PF-days] [find](https://scholar.google.com/scholar?q=St%C3%A9phane+Aroca%2DOuellette%2C+Cory+Paik%2C+Alessandro+Roncone%2C+and+Katharina+Kann%2E+2021%2E+PROST%3A+Physical+reasoning+about+objects+through+space+and+time%2E+In+Budget+%5BPF%2Ddays%5D)
3. Layers Hidden dim. Attention heads num. dim. 432 300 300 42 180 300 270 4,106 3,646 3,708 604 3,063 4,313 9,938 64 96 76 64 76 80 105 8,192 12,288 13,824 16,384 16,384 16,384 20,480 128 96 96 128 64 128 144 128 128 128 128 160 Attention heads num. dim. Memory [GB] Performance [sec/iter.] [TFLOPs] 64… [find](https://scholar.google.com/scholar?q=Layers+Hidden+dim%2E+Attention+heads+num%2E+dim%2E+432+300+300+42+180+300+270+4%2C106+3%2C646+3%2C708+604+3%2C063+4%2C313+9%2C938+64+96+76+64+76+80+105+8%2C192+12%2C288+13%2C824+16%2C384+16%2C384+16%2C384+20%2C480+128+96+96+128+64+1)
4. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. [Layer normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/). arXiv preprint arXiv:1607.06450.
5. Qiang Ning Ben Zhou, Daniel Khashabi and Dan Roth. 2019. “going on a vacation” takes longer than “going for a walk”: A study of temporal commonsense understanding. In EMNLP. [find](https://scholar.google.com/scholar?q=Qiang+Ning+Ben+Zhou%2C+Daniel+Khashabi+and+Dan+Roth%2E+2019%2E+%E2%80%9Cgoing+on+a+vacation%E2%80%9D+takes+longer+than+%E2%80%9Cgoing+for+a+walk%E2%80%9D%3A+A+study+of+temporal+commonsense+understanding%2E+In+EMNLP%2E)
6. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. [Semantic parsing on freebase from question-answer pairs](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/). In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544.
7. [@BlancheMinerva] Stella Biderman. 2021. You: Gee stella, \#eleutherai sure hypes rotary embeddings a lot. are you sure that they’re that good? me:. Twitter. [find](https://scholar.google.com/scholar?q=%5B%40BlancheMinerva%5D+Stella+Biderman%2E+2021%2E+You%3A+Gee+stella%2C+%5C%23eleutherai+sure+hypes+rotary+embeddings+a+lot%2E+are+you+sure+that+they%E2%80%99re+that+good%3F+me%3A%2E+Twitter%2E)
8. BigScience Workshop. 2022. 4ab0472). [find](https://scholar.google.com/scholar?q=BigScience+Workshop%2E+2022%2E+4ab0472%29%2E)
9. Bloom (revision Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Thirty-Fourth AAAI Conference on Artificial Intelligence.
10. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. If you use this software, please cite it using these metadata. [find](https://scholar.google.com/scholar?q=Sid+Black%2C+Leo+Gao%2C+Phil+Wang%2C+Connor+Leahy%2C+and+Stella+Biderman%2E+2021%2E+GPT%2DNeo%3A+Large+Scale+Autoregressive+Language+Modeling+with+Mesh%2DTensorflow%2E+If+you+use+this+software%2C+please+cite+it+using+these)
11. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
12. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. [Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/). URL [https://openai.com/blog/sparse-transformers](https://openai.com/blog/sparse-transformers). [↗](https://openai.com/blog/sparse-transformers)
13. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In NAACL.
14. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the AI2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). CoRR, abs/1803.05457.
15. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computation… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Adina+Williams%2C+Samuel+R%2E+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+Xnli%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+)
16. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177–190. Springer. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2005%2E+The+pascal+recognising+textual+entailment+challenge%2E+In+Machine+Learning+Challenges+Workshop%2C+pages+177%E2%80%93190%2E+Springer%2E)
17. Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2016. [Language modeling with gated convolutional networks](https://md.research.elamur.ai/free/en/papers/dauphin-2016-language-modeling-gated-convolutional-networks/). CoRR, abs/1612.08083.
18. Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2021. [8-bit optimizers via block-wise quantization](https://md.research.elamur.ai/free/en/papers/dettmers-2021-8-bit-optimizers-block-wise/).
19. William B Dolan and Chris Brockett. 2005. [Automatically constructing a corpus of sentential paraphrases](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/). In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
20. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. [Unified language model pre-training for natural language understanding and generation](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/). Advances in Neural Information Processing Systems, 32.
21. Deep Ganguli, Danny Hernandez, Liane Lovitt, Nova DasSarma, Tom Henighan, Andy Jones, Nicholas Joseph, Jackson Kernion, Ben Mann, Amanda Askell, et al. 2022. [Predictability and surprise in large generative models](https://md.research.elamur.ai/free/en/papers/ganguli-2022-predictability-surprise-large-generative-models/). arXiv preprint arXiv:2202.07785.
22. Leo Gao. 2021. On the sizes of openai api models. [find](https://scholar.google.com/scholar?q=Leo+Gao%2E+2021%2E+On+the+sizes+of+openai+api+models%2E)
23. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: an 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+Shawn+Presser%2C+and+Connor+Leahy%2E+2020%2E+The+Pile%3A+an+800GB+datas)
24. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Jason+Phang%2C+Laria+Reynolds%2C+Eric+Tang%2C+Anish+Thite)
25. Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012. [SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning](https://md.research.elamur.ai/free/en/papers/gordon-2012-semeval-2012-task-7-choice/). In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the s…
26. Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks+and+Kevin+Gimpel%2E+2016%2E+Gaussian+error+linear+units+%28gelus%29%2E+arXiv+preprint+arXiv%3A1606%2E08415%2E)
27. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). arXiv preprint arXiv:2203.15556.
28. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. [Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization](https://md.research.elamur.ai/free/en/papers/hu-2020-xtreme-massively-multilingual-multi-task/). ArXiv, abs/2003.11080.
29. Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. 2017. First quora dataset release: Question pairs. [find](https://scholar.google.com/scholar?q=Shankar+Iyer%2C+Nikhil+Dandekar%2C+and+Kornel+Csernai%2E+2017%2E+First+quora+dataset+release%3A+Question+pairs%2E)
30. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. [Pubmedqa: A dataset for biomedical research question answering](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce…
31. Matt Gardner Johannes Welbl, Nelson F. Liu. 2017. [Crowdsourcing multiple choice science questions](https://md.research.elamur.ai/free/en/papers/welbl-2017-crowdsourcing-multiple-choice-science-questions/).
32. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. [Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada. Association for Computationa…
33. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361.
34. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. [Looking beyond the surface:a challenge set for reading comprehension over multiple sentences](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/). In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL).
35. Boseop Kim, Hyoungseok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Dong Hyeon Jeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dong Hyung Seo, Heungsub Lee, Minyoung Jeong, Sungjae Lee, Minsub Kim, SukHyun Ko, Seokhun Kim, Taeyong Park, Jinuk Kim, Soyoung Kang, Na-Hyeon Ryu, Kang Min Yoo, Minsuk Ch… [find](https://scholar.google.com/scholar?q=Boseop+Kim%2C+Hyoungseok+Kim%2C+Sang%2DWoo+Lee%2C+Gichang+Lee%2C+Donghyun+Kwak%2C+Dong+Hyeon+Jeon%2C+Sunghyun+Park%2C+Sungju+Kim%2C+Seonhoon+Kim%2C+Dong+Hyung+Seo%2C+Heungsub+Lee%2C+Minyoung+Jeong%2C+Sungjae+Lee%2C+Minsub+Kim%2C+S)
36. Diederik P Kingma and Jimmy Ba. 2014. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980.
37. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. [Race: Large-scale reading comprehension dataset from examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/). arXiv preprint arXiv:1704.04683.
38. Ahmad Lashgar, Amirali Baniasadi, and Ahmad Khon-sari. 2013. Warp size impact in gpus: large or small? In GPGPU@ASPLOS. [find](https://scholar.google.com/scholar?q=Ahmad+Lashgar%2C+Amirali+Baniasadi%2C+and+Ahmad+Khon%2Dsari%2E+2013%2E+Warp+size+impact+in+gpus%3A+large+or+small%3F+In+GPGPU%40ASPLOS%2E)
39. Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gérard Dupont, Stella Biderman, Anna Rogers, Loub… — [The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2023-bigscience-roots-corpus-1-6tb/)
40. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). arXiv preprint arXiv:2104.08691.
41. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+winograd+schema+challenge%2E+In+Thirteenth+International+Conference+on+the+Principles+of+Knowledge+Representation+and+Reasoning%2E)
42. Yoav Levine, Noam Wies, Or Sharir, Hofit Bata, and Amnon Shashua. 2020. Limits to depth efficiencies of self-attention. Advances in Neural Information Processing Systems, 33:22640–22651. [find](https://scholar.google.com/scholar?q=Yoav+Levine%2C+Noam+Wies%2C+Or+Sharir%2C+Hofit+Bata%2C+and+Amnon+Shashua%2E+2020%2E+Limits+to+depth+efficiencies+of+self%2Dattention%2E+Advances+in+Neural+Information+Processing+Systems%2C+33%3A22640%E2%80%9322651%2E)
43. Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. 2021. Jurassic-1: Technical details and evaluation. Technical report, AI21 Labs. [find](https://scholar.google.com/scholar?q=Opher+Lieber%2C+Or+Sharir%2C+Barak+Lenz%2C+and+Yoav+Shoham%2E+2021%2E+Jurassic%2D1%3A+Technical+details+and+evaluation%2E+Technical+report%2C+AI21+Labs%2E)
44. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Ves Stoyanov, a… — [Few-shot Learning with Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
45. Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. [Logiqa: A challenge dataset for machine reading comprehension with logical reasoning](https://md.research.elamur.ai/free/en/papers/liu-2020-logiqa-challenge-dataset-machine-reading/). CoRR, abs/2007.08124.
46. Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. [Generating wikipedia by summarizing long sequences](https://md.research.elamur.ai/free/en/papers/liu-2018-generating-wikipedia-summarizing-long-sequences/). In International Conference on Learning Representations.
47. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+In+EMNLP%2E)
48. Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. 2021. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the Interna… [find](https://scholar.google.com/scholar?q=Deepak+Narayanan%2C+Mohammad+Shoeybi%2C+Jared+Casper%2C+Patrick+LeGresley%2C+Mostofa+Patwary%2C+Vijay+Korthikanti%2C+Dmitri+Vainbrand%2C+Prethvi+Kashinkunti%2C+Julie+Bernauer%2C+Bryan+Catanzaro%2C+et+al%2E+2021%2E+Efficient+)
49. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 – 16, Mannheim… [find](https://scholar.google.com/scholar?q=Pedro+Javier+Ortiz+Su%C3%A1rez%2C+Beno%C3%AEt+Sagot%2C+and+Laurent+Romary%2E+2019%2E+Asynchronous+pipelines+for+processing+huge+corpora+on+medium+to+low+resource+infrastructures%2E+Proceedings+of+the+Workshop+on+Challeng)
50. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/). In Proceedings of the 54th Annual Meeting of the Association fo…
51. Mohammad Taher Pilehvar and os’e Camacho-Collados. 2018. [Wic: 10, 000 example pairs for evaluating context-sensitive representations](https://md.research.elamur.ai/free/en/papers/pilehvar-2019-wic-word-context-dataset-evaluating/). CoRR, abs/1808.09121.
52. Ofir Press, Noah Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Noah+Smith%2C+and+Mike+Lewis%2E+2022%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+In+International+Conference+on+Learning+Representations%2E)
53. Ofir Press and Lior Wolf. 2017. [Using the output embedding to improve language models](https://md.research.elamur.ai/free/en/papers/press-2017-output-embedding-improve-language-models/). In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 157–163, Valencia, Spain. Association for Computational Linguistics.
54. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
55. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. [find](https://scholar.google.com/scholar?q=Jack+W+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+et+al%2E+2021%2E+Scaling+language+models%3A+Methods%2C+a)
56. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). CoRR, abs/1910.10683.
57. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). arXiv preprint arXiv:1606.05250.
58. Prajit Ramachandran, Barret Zoph, and Quoc V Le. 2017. Searching for activation functions. arXiv preprint arXiv:1710.05941. [find](https://scholar.google.com/scholar?q=Prajit+Ramachandran%2C+Barret+Zoph%2C+and+Quoc+V+Le%2E+2017%2E+Searching+for+activation+functions%2E+arXiv+preprint+arXiv%3A1710%2E05941%2E)
59. Corby Rosset. 2020. Turing-nlg: A 17-billion-parameter language model by microsoft. [find](https://scholar.google.com/scholar?q=Corby+Rosset%2E+2020%2E+Turing%2Dnlg%3A+A+17%2Dbillion%2Dparameter+language+model+by+microsoft%2E)
60. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2019%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+arXiv+preprint+arXiv%3A1907%2E10641%2E)
61. Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang A. Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M SAIFUL BARI, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajy… — [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
62. Noam Shazeer. 2020. former. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+2020%2E+former%2E)
63. Glu variants improve trans-Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022. [Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative langua…](https://md.research.elamur.ai/free/en/papers/smith-2022-using-deepspeed-megatron-train-megatron/)
64. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. [Recursive deep models for semantic compositionality over a sentiment treebank](https://md.research.elamur.ai/free/en/papers/socher-2013-recursive-deep-models-semantic-compositionality/). In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1…
65. Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). arXiv preprint arXiv:2104.09864.
66. Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2020. [Long range arena: A benchmark for efficient transformers](https://md.research.elamur.ai/free/en/papers/tay-2020-long-range-arena-benchmark-efficient/). arXiv preprint arXiv:2011.04006.
67. Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. 2021. Scale efficiently: Insights from pre-training and fine-tuning transformers. ArXiv, abs/2109.10686. [find](https://scholar.google.com/scholar?q=Yi+Tay%2C+Mostafa+Dehghani%2C+Jinfeng+Rao%2C+William+Fedus%2C+Samira+Abnar%2C+Hyung+Won+Chung%2C+Sharan+Narang%2C+Dani+Yogatama%2C+Ashish+Vaswani%2C+and+Donald+Metzler%2E+2021%2E+Scale+efficiently%3A+Insights+from+pre%2Dtraini)
68. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. [Lamda: Language models for dialog applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/). arXiv preprint arXiv:2201.08239.
69. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in neural information processing systems, pages 5998–6008.
70. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In the Proceedings of ICLR.
71. Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. 2022. What language model architecture and pretraining objective work best for zero-shot generalization? arXiv preprint arXiv:2204.05832. [find](https://scholar.google.com/scholar?q=Thomas+Wang%2C+Adam+Roberts%2C+Daniel+Hesslow%2C+Teven+Le+Scao%2C+Hyung+Won+Chung%2C+Iz+Beltagy%2C+Julien+Launay%2C+and+Colin+Raffel%2E+2022%2E+What+language+model+architecture+and+pretraining+objective+work+best+for+z)
72. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). arXiv preprint arXiv:2109.01652.
73. Shaohua Wu, Xudong Zhao, Tong Yu, Rongguo Zhang, Chong Shen, Hongli Liu, Feng Li, Hong Zhu, Jiangang Luo, Liang Xu, et al. 2021. Yuan 1.0: Large-scale pre-trained language model in zero-shot and few-shot learning. arXiv preprint arXiv:2110.04725. [find](https://scholar.google.com/scholar?q=Shaohua+Wu%2C+Xudong+Zhao%2C+Tong+Yu%2C+Rongguo+Zhang%2C+Chong+Shen%2C+Hongli+Liu%2C+Feng+Li%2C+Hong+Zhu%2C+Jiangang+Luo%2C+Liang+Xu%2C+et+al%2E+2021%2E+Yuan+1%2E0%3A+Large%2Dscale+pre%2Dtrained+language+model+in+zero%2Dshot+and+few%2Ds)
74. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. [mt5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). arXiv preprint arXiv:2010.11934.
75. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. [mt5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In NAACL.
76. Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. [Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models](https://md.research.elamur.ai/free/en/papers/ben-zaken-2021-bitfit-simple-parameter-efficient-fine/). arXiv preprint arXiv:2106.10199.
77. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791– 4800, Florence, Italy. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Com)
78. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. [Glm-130b: An open bilingual pre-trained model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/). arXiv preprint arXiv:2210.02414.
79. Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, et al. 2021. [Pangu-α: Large-scale autoregressive pretrained chinese language models with auto-parallel computation](https://md.research.elamur.ai/free/en/papers/zeng-2021-pangu-alpha-large-scale/). arXiv preprint arXiv:2104.12369.
80. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068.

*Updated 2 November 2026*