---
title: "Training Compute-Optimal Large Language Models — Hoffmann 2022"
description: "The authors from DeepMind investigate how best to split a fixed compute budget (FLOPs) between two things: the… 61 references, cited by 1,110 papers here."
authors: ["Jordan Hoffmann", "Sebastian Borgeaud", "Arthur Mensch", "Elena Buchatskaya", "Trevor Cai", "Eliza Rutherford", "Diego de Las Casas", "Lisa Anne Hendricks", "Johannes Welbl", "Aidan Clark", "Tom Hennigan", "Eric Noland", "Katie Millican", "George van den Driessche", "Bogdan Damoc", "Aurelia Guy", "Simon Osindero", "Karen Simonyan", "Erich Elsen", "Jack W. Rae", "Oriol Vinyals", "Laurent Sifre"]
year: 2022
venue: "arXiv:2203.15556"
source: "https://arxiv.org/abs/2203.15556"
doi: "10.48550/arxiv.2203.15556"
arxiv: "2203.15556"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/) / Training Compute-Optimal Large Language Models

# Training Compute-Optimal Large Language Models

**Authors:** Jordan Hoffmann, [Sebastian Borgeaud](https://md.research.elamur.ai/free/en/authors/sebastian-borgeaud/), [Arthur Mensch](https://md.research.elamur.ai/free/en/authors/arthur-mensch/), Elena Buchatskaya, [Trevor Cai](https://md.research.elamur.ai/free/en/authors/trevor-cai/), [Eliza Rutherford](https://md.research.elamur.ai/free/en/authors/eliza-rutherford/), Diego de Las Casas, [Lisa Anne Hendricks](https://md.research.elamur.ai/free/en/authors/lisa-anne-hendricks/), [Johannes Welbl](https://md.research.elamur.ai/free/en/authors/johannes-welbl/), [Aidan Clark](https://md.research.elamur.ai/free/en/authors/aidan-clark/), Tom Hennigan, Eric Noland, [Katie Millican](https://md.research.elamur.ai/free/en/authors/katie-millican/), George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, [Karen Simonyan](https://md.research.elamur.ai/free/en/authors/karen-simonyan/), [Erich Elsen](https://md.research.elamur.ai/free/en/authors/erich-elsen/), Jack W. Rae, [Oriol Vinyals](https://md.research.elamur.ai/free/en/authors/oriol-vinyals/) and Laurent Sifre · **Year:** 2022 · **Venue:** arXiv:2203.15556 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2203.15556) · [PDF](https://arxiv.org/pdf/2203.15556) · [DOI 10.48550/arxiv.2203.15556](https://doi.org/10.48550/arxiv.2203.15556) · [arXiv 2203.15556](https://arxiv.org/abs/2203.15556)

**Citations:** [1110](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
The authors from DeepMind investigate how best to split a fixed compute budget (FLOPs) between two things: the size of a transformer language model and the number of tokens it is trained on. They trained over 400 models ranging from roughly 70M to over 16B parameters on 5 to 500B tokens. They then used three independent approaches to estimate the optimal ratio. All three approaches gave a similar answer: the number of parameters and the amount of data should be scaled equally. If the model becomes twice as large, the number of training tokens should also double. Consequently, most LLMs of the time, including Gopher, GPT-3 and MT-NLG, are too large for their budget and trained on too little data. To test this conclusion, the authors built Chinchilla, a 70B-parameter model trained on 1.4 trillion tokens. It used the same amount of compute as Gopher. Chinchilla consistently outperformed much larger models on a wide range of tasks, and its fine-tuning and inference are cheaper. After this work, the field began to pay more attention to the amount and quality of training data, not just to model size.

## Key points
- Disagreement with Kaplan et al. (2020). They recommended that, for a 10-fold increase in budget, the model should be enlarged 5.5 times and the data only 1.8 times. Here it is shown that both quantities should grow in equal proportion. The reason for the discrepancy is that Kaplan et al. used the same learning rate schedule for all models, regardless of training duration.
- The optimal model for the Gopher budget would be four times smaller and trained on four times as many tokens. For comparison: Gopher has 280B parameters and saw 300B tokens, while Chinchilla has 70B parameters and 1.4T tokens.
- On MMLU, Chinchilla reaches 67.5%, the best result at the time and more than 7 points higher than Gopher. It also outperforms GPT-3 (175B), Jurassic-1 (178B) and Megatron-Turing NLG (530B).
- Toxicity was evaluated on 25,000 generated texts using PerspectiveAPI. The mean score barely differs: 0.087 for Chinchilla versus 0.081 for Gopher. On Winogender, Chinchilla resolves pronouns correctly more often: 78.3% versus 71.4%. However, the differences between groups indicate that bias is still present in the model.
- Limitations of the work: there are only two comparable runs at large scale, the estimate relies on a power-law assumption, and each model was trained for less than one epoch. Because of this, the optimal size of large models may still be overestimated.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

## Cite

Cite as: Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals and Laurent Sifre. 2022. Training Compute-Optimal Large Language Models. arXiv:2203.15556.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hoffmann2022training,
  title = {{Training Compute-Optimal Large Language Models}},
  author = {Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego {de Las Casas}, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George {van den Driessche}, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals and Laurent Sifre},
  year = {2022},
  doi = {10.48550/arxiv.2203.15556},
  eprint = {2203.15556},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2203.15556}
}
```

- [← previous Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/)
- [next → MTEB: Massive Text Embedding Benchmark](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-mteb-massive-text-embedding-benchmark/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/)

## References

61 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O’Horo, J. Wang, L. Zettlemoyer, M. Diab, Z. Kozareva, and V. Stoyanov. Efficient Large Scale Language… [find](https://scholar.google.com/scholar?q=M%2E+Artetxe%2C+S%2E+Bhosale%2C+N%2E+Goyal%2C+T%2E+Mihaylov%2C+M%2E+Ott%2C+S%2E+Shleifer%2C+X%2E+V%2E+Lin%2C+J%2E+Du%2C+S%2E+Iyer%2C+R%2E+Pasunuru%2C+G%2E+Anantharaman%2C+X%2E+Li%2C+S%2E+Chen%2C+H%2E+Akin%2C+M%2E+Baines%2C+L%2E+Martin%2C+X%2E+Zhou%2C+P%2E+S%2E+Koura%2C+B%2E+O%E2%80%99H)
2. E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, 2021. [find](https://scholar.google.com/scholar?q=E%2E+M%2E+Bender%2C+T%2E+Gebru%2C+A%2E+McMillan%2DMajor%2C+and+S%2E+Shmitchell%2E+On+the+dangers+of+stochastic+parrots%3A+Can+language+models+be+too+big%3F+In+Proceedings+of+the+2021+ACM+Conference+on+Fairness%2C+Accountabilit)
3. BIG-bench collaboration. Beyond the imitation game: Measuring and extrapolating the capabilities of language models. In preparation, 2021. URL [https://github.com/google/BIG-bench/](https://github.com/google/BIG-bench/) . [↗](https://github.com/google/BIG-bench/)
4. Y. Bisk, R. Zellers, J. Gao, Y. Choi, et al. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7432–7439, 2020. [our card](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/)
5. S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. van den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, D. de Las Casas, A. Guy, J. Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. Irving, O. Vinyals, S. Osindero, K. Simo… [find](https://scholar.google.com/scholar?q=S%2E+Borgeaud%2C+A%2E+Mensch%2C+J%2E+Hoffmann%2C+T%2E+Cai%2C+E%2E+Rutherford%2C+K%2E+Millican%2C+G%2E+van+den+Driessche%2C+J%2E%2DB%2E+Lespiau%2C+B%2E+Damoc%2C+A%2E+Clark%2C+D%2E+de+Las+Casas%2C+A%2E+Guy%2C+J%2E+Menick%2C+R%2E+Ring%2C+T%2E+Hennigan%2C+S%2E+Huang%2C+L%2E)
6. J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs. 2018. URL [http://github.com/google/jax](http://github.com/google/jax) . [↗](http://github.com/google/jax)
7. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner… [find](https://scholar.google.com/scholar?q=T%2E+Brown%2C+B%2E+Mann%2C+N%2E+Ryder%2C+M%2E+Subbiah%2C+J%2E+D%2E+Kaplan%2C+P%2E+Dhariwal%2C+A%2E+Neelakantan%2C+P%2E+Shyam%2C+G%2E+Sastry%2C+A%2E+Askell%2C+S%2E+Agarwal%2C+A%2E+Herbert%2DVoss%2C+G%2E+Krueger%2C+T%2E+Henighan%2C+R%2E+Child%2C+A%2E+Ramesh%2C+D%2E+Ziegle)
8. Curran Associates, Inc., 2020. URL [https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb49674](https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb49674) 18bfb8ac142f64a-Paper.pdf . [↗](https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb49674)
9. S. Bubeck. Convex Optimization: Algorithms and Complexity. Foundations and Trends in Machine Learning, 8(3-4):231–357, 2015. URL [http://www.nowpublishers.com/article/Detail](http://www.nowpublishers.com/article/Detail) s/MAL-050 . [↗](http://www.nowpublishers.com/article/Detail)
10. A. Clark, D. d. l. Casas, A. Guy, A. Mensch, M. Paganini, J. Hoffmann, B. Damoc, B. Hechtman, T. Cai, S. Borgeaud, G. v. d. Driessche, E. Rutherford, T. Hennigan, M. Johnson, K. Millican, A. Cassirer, C. Jones, E. Buchatskaya, D. Budden, L. Sifre, S. Osindero, O. Vinyals, J. Rae, E. Elsen, K. Kavukc… — [arXiv:2202.01169](https://arxiv.org/abs/2202.01169) [find](https://scholar.google.com/scholar?q=A%2E+Clark%2C+D%2E+d%2E+l%2E+Casas%2C+A%2E+Guy%2C+A%2E+Mensch%2C+M%2E+Paganini%2C+J%2E+Hoffmann%2C+B%2E+Damoc%2C+B%2E+Hechtman%2C+T%2E+Cai%2C+S%2E+Borgeaud%2C+G%2E+v%2E+d%2E+Driessche%2C+E%2E+Rutherford%2C+T%2E+Hennigan%2C+M%2E+Johnson%2C+K%2E+Millican%2C+A%2E+Cassirer%2C)
11. C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volu… [our card](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/)
12. N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scali… [our card](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/)
13. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961, 2021. [find](https://scholar.google.com/scholar?q=W%2E+Fedus%2C+B%2E+Zoph%2C+and+N%2E+Shazeer%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+arXiv+preprint+arXiv%3A2101%2E03961%2C+2021%2E)
14. L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. [find](https://scholar.google.com/scholar?q=L%2E+Gao%2C+S%2E+Biderman%2C+S%2E+Black%2C+L%2E+Golding%2C+T%2E+Hoppe%2C+C%2E+Foster%2C+J%2E+Phang%2C+H%2E+He%2C+A%2E+Thite%2C+N%2E+Nabeshima%2C+S%2E+Presser%2C+and+C%2E+Leahy%2E+The+Pile%3A+An+800GB+dataset+of+diverse+text+for+language+modeling%2E+arX)
15. S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online, Nov. 2020. Association for Computational Linguistics. doi: 10.1… [our card](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/)
16. K. Guu, K. Lee, Z. Tung, P. Pasupat, and M.-W. Chang. REALM: Retrieval-augmented language model pre-training, 2020. [find](https://scholar.google.com/scholar?q=K%2E+Guu%2C+K%2E+Lee%2C+Z%2E+Tung%2C+P%2E+Pasupat%2C+and+M%2E%2DW%2E+Chang%2E+REALM%3A+Retrieval%2Daugmented+language+model+pre%2Dtraining%2C+2020%2E)
17. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
18. T. Hennigan, T. Cai, T. Norman, and I. Babuschkin. Haiku: Sonnet for JAX. 2020. URL [http://github.com/deepmind/dm-haiku](http://github.com/deepmind/dm-haiku) . [↗](http://github.com/deepmind/dm-haiku)
19. D. Hernandez, J. Kaplan, T. Henighan, and S. McCandlish. Scaling laws for transfer, 2021. [our card](https://md.research.elamur.ai/free/en/papers/hernandez-2021-scaling-laws-transfer/)
20. P. J. Huber. Robust Estimation of a Location Parameter. The Annals of Mathematical Statistics, 35 (1):73–101, Mar. 1964. ISSN 0003-4851, 2168-8990. doi: 10.1214/aoms/1177703732. URL [https://projecteuclid.org/journals/annals-of-mathematical-statistics/vol](https://projecteuclid.org/journals/annals-of-mathematical-statistics/vol) ume-35/issue-1/Robust-Estimation-of-a-Locatio… — [doi:10.1214/aoms/1177703732](https://doi.org/10.1214/aoms/1177703732) [↗](https://projecteuclid.org/journals/annals-of-mathematical-statistics/vol)
21. G. Izacard and E. Grave. Distilling knowledge from reader to retriever for question answering, 2020. [find](https://scholar.google.com/scholar?q=G%2E+Izacard+and+E%2E+Grave%2E+Distilling+knowledge+from+reader+to+retriever+for+question+answering%2C+2020%2E)
22. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017. [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
23. N. P. Jouppi, C. Young, N. Patil, D. Patterson, G. Agrawal, R. Bajwa, S. Bates, S. Bhatia, N. Boden, A. Borchers, R. Boyle, P.-l. Cantin, C. Chao, C. Clark, J. Coriell, M. Daley, M. Dau, J. Dean, B. Gelb, T. V. Ghaemmaghami, R. Gottipati, W. Gulland, R. Hagmann, C. R. Ho, D. Hogberg, J. Hu, R. Hundt… [find](https://scholar.google.com/scholar?q=N%2E+P%2E+Jouppi%2C+C%2E+Young%2C+N%2E+Patil%2C+D%2E+Patterson%2C+G%2E+Agrawal%2C+R%2E+Bajwa%2C+S%2E+Bates%2C+S%2E+Bhatia%2C+N%2E+Boden%2C+A%2E+Borchers%2C+R%2E+Boyle%2C+P%2E%2Dl%2E+Cantin%2C+C%2E+Chao%2C+C%2E+Clark%2C+J%2E+Coriell%2C+M%2E+Daley%2C+M%2E+Dau%2C+J%2E+Dean%2C+B%2E+G)
24. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. [our card](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)
25. D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. [find](https://scholar.google.com/scholar?q=D%2E+P%2E+Kingma+and+J%2E+Ba%2E+Adam%3A+A+method+for+stochastic+optimization%2E+arXiv+preprint+arXiv%3A1412%2E6980%2C+2014%2E)
26. T. Kudo and J. Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018. [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
27. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research. Transactions of th… [our card](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
28. G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794, Copenhagen, Denmark, Sept. 2017. Association for Computational Linguistics. doi:… [our card](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/)
29. Y. Levine, N. Wies, O. Sharir, H. Bata, and A. Shashua. The depth-to-width interplay in self-attention. arXiv preprint arXiv:2006.12467, 2020. — [arXiv:2006.12467](https://arxiv.org/abs/2006.12467) [find](https://scholar.google.com/scholar?q=Y%2E+Levine%2C+N%2E+Wies%2C+O%2E+Sharir%2C+H%2E+Bata%2C+and+A%2E+Shashua%2E+The+depth%2Dto%2Dwidth+interplay+in+self%2Dattention%2E+arXiv+preprint+arXiv%3A2006%2E12467%2C+2020%2E+%E2%80%94+%5BarXiv%3A2006%2E12467%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2006%2E12467%29)
30. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel, and D. Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474, 2020. [find](https://scholar.google.com/scholar?q=P%2E+Lewis%2C+E%2E+Perez%2C+A%2E+Piktus%2C+F%2E+Petroni%2C+V%2E+Karpukhin%2C+N%2E+Goyal%2C+H%2E+K%C3%BCttler%2C+M%2E+Lewis%2C+W%2E%2Dt%2E+Yih%2C+T%2E+Rockt%C3%A4schel%2C+S%2E+Riedel%2C+and+D%2E+Kiela%2E+Retrieval%2Daugmented+generation+for+knowledge%2Dintensive+nlp+)
31. O. Lieber, O. Sharir, B. Lenz, and Y. Shoham. Jurassic-1: Technical details and evaluation. White Paper. AI21 Labs, 2021. [find](https://scholar.google.com/scholar?q=O%2E+Lieber%2C+O%2E+Sharir%2C+B%2E+Lenz%2C+and+Y%2E+Shoham%2E+Jurassic%2D1%3A+Technical+details+and+evaluation%2E+White+Paper%2E+AI21+Labs%2C+2021%2E)
32. S. Lin, J. Hilton, and O. Evans. TruthfulQA: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021. [our card](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/)
33. I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL [https://openreview.net/forum?id=Bkg6RiCqY7](https://openreview.net/forum?id=Bkg6RiCqY7) . [↗](https://openreview.net/forum?id=Bkg6RiCqY7)
34. S. McCandlish, J. Kaplan, D. Amodei, and O. D. Team. An empirical model of large-batch training, 2018. [find](https://scholar.google.com/scholar?q=S%2E+McCandlish%2C+J%2E+Kaplan%2C+D%2E+Amodei%2C+and+O%2E+D%2E+Team%2E+An+empirical+model+of+large%2Dbatch+training%2C+2018%2E)
35. S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models. International Conference on Learning Representations, 2017. [our card](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/)
36. M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, and T. Gebru. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pages 220–229, 2019. [find](https://scholar.google.com/scholar?q=M%2E+Mitchell%2C+S%2E+Wu%2C+A%2E+Zaldivar%2C+P%2E+Barnes%2C+L%2E+Vasserman%2C+B%2E+Hutchinson%2C+E%2E+Spitzer%2C+I%2E+D%2E+Raji%2C+and+T%2E+Gebru%2E+Model+cards+for+model+reporting%2E+In+Proceedings+of+the+conference+on+fairness%2C+accountabi)
37. J. Nocedal. Updating Quasi-Newton Matrices with Limited Storage. Mathematics of Computation, 35(151):773–782, 1980. ISSN 0025-5718. doi: 10.2307/2006193. URL [https://www.jstor.org/stable/2006193](https://www.jstor.org/stable/2006193) . [↗](https://www.jstor.org/stable/2006193)
38. D. Paperno, G. Kruszewski, A. Lazaridou, Q. N. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, and R. Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context, 2016. [our card](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/)
39. J. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P.-S. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor… [find](https://scholar.google.com/scholar?q=J%2E+Rae%2C+S%2E+Borgeaud%2C+T%2E+Cai%2C+K%2E+Millican%2C+J%2E+Hoffmann%2C+F%2E+Song%2C+J%2E+Aslanides%2C+S%2E+Henderson%2C+R%2E+Ring%2C+S%2E+Young%2C+E%2E+Rutherford%2C+T%2E+Hennigan%2C+J%2E+Menick%2C+A%2E+Cassirer%2C+R%2E+Powell%2C+G%2E+van+den+Driessche%2C+L%2E+A)
40. J. W. Rae, A. Potapenko, S. M. Jayakumar, T. P. Lillicrap, K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, et al. Compressive transformers for long-range sequence modelling. Advances in Neural Information Processing Systems, 33:6154–6158, 2020. [our card](https://md.research.elamur.ai/free/en/papers/rae-2019-compressive-transformers-long-range-sequence/)
41. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020a. URL [http://jmlr.org/papers/v21/20-074.html](http://jmlr.org/papers/v21/20-074.html) . [↗](http://jmlr.org/papers/v21/20-074.html) [our card](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)
42. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020b. [our card](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)
43. S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020. [find](https://scholar.google.com/scholar?q=S%2E+Rajbhandari%2C+J%2E+Rasley%2C+O%2E+Ruwase%2C+and+Y%2E+He%2E+Zero%3A+Memory+optimizations+toward+training+trillion+parameter+models%2E+In+SC20%3A+International+Conference+for+High+Performance+Computing%2C+Networking%2C+Sto)
44. H. Robbins and S. Monro. A Stochastic Approximation Method. The Annals of Mathematical Statistics, 22(3):400–407, Sept. 1951. [find](https://scholar.google.com/scholar?q=H%2E+Robbins+and+S%2E+Monro%2E+A+Stochastic+Approximation+Method%2E+The+Annals+of+Mathematical+Statistics%2C+22%283%29%3A400%E2%80%93407%2C+Sept%2E+1951%2E)
45. R. Rudinger, J. Naradowsky, B. Leonard, and B. Van Durme. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, Louisiana, June 2018. Association for Compu… [our card](https://md.research.elamur.ai/free/en/papers/rudinger-2018-gender-bias-coreference-resolution/)
46. K. Sakaguchi, R. Le Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020. [find](https://scholar.google.com/scholar?q=K%2E+Sakaguchi%2C+R%2E+Le+Bras%2C+C%2E+Bhagavatula%2C+and+Y%2E+Choi%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+34%2C+pages)
47. M. Sap, H. Rashkin, D. Chen, R. LeBras, and Y. Choi. SocialIQA: Commonsense reasoning about social interactions. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 2019. [our card](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/)
48. C. J. Shallue, J. Lee, J. Antognini, J. Sohl-Dickstein, R. Frostig, and G. E. Dahl. Measuring the effects of data parallelism on neural network training. arXiv preprint arXiv:1811.03600, 2018. — [arXiv:1811.03600](https://arxiv.org/abs/1811.03600) [find](https://scholar.google.com/scholar?q=C%2E+J%2E+Shallue%2C+J%2E+Lee%2C+J%2E+Antognini%2C+J%2E+Sohl%2DDickstein%2C+R%2E+Frostig%2C+and+G%2E+E%2E+Dahl%2E+Measuring+the+effects+of+data+parallelism+on+neural+network+training%2E+arXiv+preprint+arXiv%3A1811%2E03600%2C+2018%2E+%E2%80%94+%5BarXi)
49. J. W. Siegel and J. Xu. Approximation rates for neural networks with general activation functions. Neural Networks, 128:313–321, Aug. 2020. URL [https://www.sciencedirect.com/scienc](https://www.sciencedirect.com/scienc) e/article/pii/S0893608020301891 . [↗](https://www.sciencedirect.com/scienc)
50. S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti, E. Zhang, R. Child, R. Y. Aminabadi, J. Bernauer, X. Song, M. Shoeybi, Y. He, M. Houston, S. Tiwary, and B. Catanzaro. Using Deepspeed and Megatron to Train Megatron-turing NL… [our card](https://md.research.elamur.ai/free/en/papers/smith-2022-using-deepspeed-megatron-train-megatron/)
51. J. Steinhardt. Updates and lessons from AI forecasting, 2021. URL [https://bounded-regret.g](https://bounded-regret.g) host.io/ai-forecasting/ . [↗](https://bounded-regret.g)
52. Y. Tay, M. Dehghani, J. Rao, W. Fedus, S. Abnar, H. W. Chung, S. Narang, D. Yogatama, A. Vaswani, and D. Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers, 2021. [find](https://scholar.google.com/scholar?q=Y%2E+Tay%2C+M%2E+Dehghani%2C+J%2E+Rao%2C+W%2E+Fedus%2C+S%2E+Abnar%2C+H%2E+W%2E+Chung%2C+S%2E+Narang%2C+D%2E+Yogatama%2C+A%2E+Vaswani%2C+and+D%2E+Metzler%2E+Scale+efficiently%3A+Insights+from+pre%2Dtraining+and+fine%2Dtuning+transformers%2C+2021%2E)
53. R. Thoppilan, D. D. Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, Y. Li, H. Lee, H. S. Zheng, A. Ghafouri, M. Menegali, Y. Huang, M. Krikun, D. Lepikhin, J. Qin, D. Chen, Y. Xu, Z. Chen, A. Roberts, M. Bosma, Y. Zhou, C.-C. Chang, I. Krivokon, W. Rusch,… [our card](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/)
54. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
55. L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh, Z. Kenton, S. Brown, W. Hawkins, T. Stepleton, C. Biles, A. Birhane, J. Haas, L. Rimell, L. A. Hendricks, W. Isaac, S. Legassick, G. Irving, and I. Gabriel. Ethical and social risks of… [our card](https://md.research.elamur.ai/free/en/papers/weidinger-2021-ethical-social-risks-harm-language/)
56. J. Welbl, A. Glaese, J. Uesato, S. Dathathri, J. Mellor, L. A. Hendricks, K. Anderson, P. Kohli, B. Coppin, and P.-S. Huang. Challenges in detoxifying language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2447–2469, Punta Cana, Dominican Republic, Nov. 2021… [our card](https://md.research.elamur.ai/free/en/papers/welbl-2021-challenges-detoxifying-language-models/)
57. A. Xu, E. Pathak, E. Wallace, S. Gururangan, M. Sap, and D. Klein. Detoxifying language models risks marginalizing minority voices. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2390–2397, Onli… [find](https://scholar.google.com/scholar?q=A%2E+Xu%2C+E%2E+Pathak%2C+E%2E+Wallace%2C+S%2E+Gururangan%2C+M%2E+Sap%2C+and+D%2E+Klein%2E+Detoxifying+language+models+risks+marginalizing+minority+voices%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+)
58. G. Yang, E. J. Hu, I. Babuschkin, S. Sidor, X. Liu, D. Farhi, N. Ryder, J. Pachocki, W. Chen, and J. Gao. Tuning large neural networks via zero-shot hyperparameter transfer. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021.… [find](https://scholar.google.com/scholar?q=G%2E+Yang%2C+E%2E+J%2E+Hu%2C+I%2E+Babuschkin%2C+S%2E+Sidor%2C+X%2E+Liu%2C+D%2E+Farhi%2C+N%2E+Ryder%2C+J%2E+Pachocki%2C+W%2E+Chen%2C+and+J%2E+Gao%2E+Tuning+large+neural+networks+via+zero%2Dshot+hyperparameter+transfer%2E+In+A%2E+Beygelzimer%2C+Y%2E+Daup)
59. R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. [find](https://scholar.google.com/scholar?q=R%2E+Zellers%2C+A%2E+Holtzman%2C+Y%2E+Bisk%2C+A%2E+Farhadi%2C+and+Y%2E+Choi%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguist)
60. G. Zhang, L. Li, Z. Nado, J. Martens, S. Sachdeva, G. Dahl, C. Shallue, and R. B. Grosse. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural In… [find](https://scholar.google.com/scholar?q=G%2E+Zhang%2C+L%2E+Li%2C+Z%2E+Nado%2C+J%2E+Martens%2C+S%2E+Sachdeva%2C+G%2E+Dahl%2C+C%2E+Shallue%2C+and+R%2E+B%2E+Grosse%2E+Which+algorithmic+choices+matter+at+which+batch+sizes%3F+insights+from+a+noisy+quadratic+model%2E+In+H%2E+Wallach%2C+H)
61. B. Zoph, I. Bello, S. Kumar, N. Du, Y. Huang, J. Dean, N. Shazeer, and W. Fedus. Designing effective sparse expert models, 2022. [find](https://scholar.google.com/scholar?q=B%2E+Zoph%2C+I%2E+Bello%2C+S%2E+Kumar%2C+N%2E+Du%2C+Y%2E+Huang%2C+J%2E+Dean%2C+N%2E+Shazeer%2C+and+W%2E+Fedus%2E+Designing+effective+sparse+expert+models%2C+2022%2E)