---
title: "Resolving Discrepancies in Compute-Optimal Scaling of Language Models — Porian 2024"
description: "Two influential compute-optimal scaling laws disagree. 63 references, cited by 13 papers here."
authors: ["Tomer Porian", "Mitchell Wortsman", "Jenia Jitsev", "Ludwig Schmidt", "Yair Carmon"]
year: 2024
venue: "arXiv:2406.19146"
source: "https://arxiv.org/abs/2406.19146"
doi: "10.48550/arxiv.2406.19146"
arxiv: "2406.19146"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/porian-2024-resolving-discrepancies-compute-optimal-scaling/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/porian-2024-resolving-discrepancies-compute-optimal-scaling/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Resolving Discrepancies in Compute-Optimal Scaling of Language Models

# Resolving Discrepancies in Compute-Optimal Scaling of Language Models

**Authors:** Tomer Porian, [Mitchell Wortsman](https://md.research.elamur.ai/free/en/authors/mitchell-wortsman/), [Jenia Jitsev](https://md.research.elamur.ai/free/en/authors/jenia-jitsev/), [Ludwig Schmidt](https://md.research.elamur.ai/free/en/authors/ludwig-schmidt/) and [Yair Carmon](https://md.research.elamur.ai/free/en/authors/yair-carmon/) · **Year:** 2024 · **Venue:** arXiv:2406.19146 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2406.19146) · [PDF](https://arxiv.org/pdf/2406.19146) · [DOI 10.48550/arxiv.2406.19146](https://doi.org/10.48550/arxiv.2406.19146) · [arXiv 2406.19146](https://arxiv.org/abs/2406.19146) · [OpenAlex](https://openalex.org/W4415796118)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [13](https://md.research.elamur.ai/free/en/papers/porian-2024-resolving-discrepancies-compute-optimal-scaling/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Two influential compute-optimal scaling laws disagree. Kaplan et al. found that, as compute grows, model size should grow faster than training tokens. Hoffmann et al. ("Chinchilla") found that tokens per parameter should stay roughly fixed. Porian et al. investigate why. They reproduce the Kaplan-style result with Llama-like decoder-only Transformers trained in OpenLM, using 16 models from 5M to 901M parameters. The data are RefinedWeb and OpenWebText2. They estimate optimal model sizes from IsoFLOP curves and fit power laws with bootstrap uncertainty, across more than 900 runs. Three factors close the gap:

* counting the FLOPs of the output head,
* shortening warmup for small models,
* tuning learning rate, batch size and AdamW β2 per model size.

With these corrections, the fitted exponent matches Chinchilla. Contrary to Hoffmann et al.'s explanation, a learning-rate decay schedule matched to each token budget is not required, and even a constant learning rate works. The study shows how to get reliable scaling predictions on modest, academic-scale budgets.

## Key points
- The fitted exponent a in N⋆ ∝ C^a falls from 0.835 (Kaplan reproduction) to 0.497 (Chinchilla-like) once the factors are corrected. Intermediate values are 0.706 with head FLOPs counted and 0.602 with scaled warmup. Cosine decay alone gives 0.571, while per-size tuning with no decay reaches 0.497.
- At Chinchilla's compute budget of 5.88e23 FLOPs, the predicted optimal model shrinks from about 3T to about 77B parameters.
- Datasets: RefinedWeb (~600B tokens, CommonCrawl) and OpenWebText2 (~30B tokens, Reddit). Hyperparameters were tuned only on RefinedWeb and transferred well to OpenWebText2.
- Secondary result: power-law fits describing how the best learning rate and best batch size scale. The common AdamW setting β2 = 0.95 performs poorly at batch sizes of 128 or below.
- Code and data are released on GitHub (formll/resolving-scaling-law-discrepancies), with checkpoints on Hugging Face. Using a constant learning rate saves roughly as much compute as the hyperparameter sweep costs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

## Cite

Cite as: Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt and Yair Carmon. 2024. Resolving Discrepancies in Compute-Optimal Scaling of Language Models. arXiv:2406.19146.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{porian2024resolving,
  title = {{Resolving Discrepancies in Compute-Optimal Scaling of Language Models}},
  author = {Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt and Yair Carmon},
  year = {2024},
  doi = {10.48550/arxiv.2406.19146},
  eprint = {2406.19146},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.19146}
}
```

- [← previous Alignment at Pre-training! Towards Native Alignment for Arabic LLMs](https://md.research.elamur.ai/free/en/papers/liang-2024-alignment-pre-training-towards-native/)
- [next → SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corpora](https://md.research.elamur.ai/free/en/papers/qarah-2024-saudibert-large-language-model-pretrained/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

63 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Common Crawl. [https://commoncrawl.org](https://commoncrawl.org). [↗](https://commoncrawl.org)
2. New scaling laws for large language models, 2023. URL [https://www.lesswrong.com/](https://www.lesswrong.com/) posts/midXmMb2Xg37F2Kgn/new-scaling-laws-for-large-language-models. Accessed: 2024-08-19. [↗](https://www.lesswrong.com/)
3. H. Akima. A new method of interpolation and smooth curve fitting based on local procedures. Journal of the ACM (JACM), 1970. [find](https://scholar.google.com/scholar?q=H%2E+Akima%2E+A+new+method+of+interpolation+and+smooth+curve+fitting+based+on+local+procedures%2E+Journal+of+the+ACM+%28JACM%29%2C+1970%2E)
4. I. Alabdulmohsin, X. Zhai, A. Kolesnikov, and L. Beyer. Getting ViT in shape: Scaling laws for compute-optimal model design. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [find](https://scholar.google.com/scholar?q=I%2E+Alabdulmohsin%2C+X%2E+Zhai%2C+A%2E+Kolesnikov%2C+and+L%2E+Beyer%2E+Getting+ViT+in+shape%3A+Scaling+laws+for+compute%2Doptimal+model+design%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2C+2023%2E)
5. A. Atanasov, B. Bordelon, S. Sainathan, and C. Pehlevan. The onset of variance-limited behavior for networks in the lazy and rich regimes. In International Conference on Learning Representations (ICLR), 2023. [find](https://scholar.google.com/scholar?q=A%2E+Atanasov%2C+B%2E+Bordelon%2C+S%2E+Sainathan%2C+and+C%2E+Pehlevan%2E+The+onset+of+variance%2Dlimited+behavior+for+networks+in+the+lazy+and+rich+regimes%2E+In+International+Conference+on+Learning+Representations+%28ICLR)
6. J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv:1607.06450, 2016. [our card](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/)
7. G. Bachmann, S. Anagnostidis, and T. Hofmann. Scaling mlps: A tale of inductive bias. Advances in Neural Information Processing Systems (NeurIPS), 2024. [find](https://scholar.google.com/scholar?q=G%2E+Bachmann%2C+S%2E+Anagnostidis%2C+and+T%2E+Hofmann%2E+Scaling+mlps%3A+A+tale+of+inductive+bias%2E+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2C+2024%2E)
8. M. Bellagente, J. Tow, D. Mahan, D. Phung, M. Zhuravinskyi, R. Adithyan, J. Baicoianu, B. Brooks, N. Cooper, A. Datta, et al. Stable lm 2 1.6 b technical report. arXiv:2402.17834, 2024. [find](https://scholar.google.com/scholar?q=M%2E+Bellagente%2C+J%2E+Tow%2C+D%2E+Mahan%2C+D%2E+Phung%2C+M%2E+Zhuravinskyi%2C+R%2E+Adithyan%2C+J%2E+Baicoianu%2C+B%2E+Brooks%2C+N%2E+Cooper%2C+A%2E+Datta%2C+et+al%2E+Stable+lm+2+1%2E6+b+technical+report%2E+arXiv%3A2402%2E17834%2C+2024%2E)
9. T. Besiroglu, E. Erdil, M. Barnett, and J. You. Chinchilla scaling: A replication attempt. arXiv:2404.10102, 2024. [our card](https://md.research.elamur.ai/free/en/papers/besiroglu-2024-chinchilla-scaling-replication-attempt/)
10. S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, et al. Gpt-neox-20b: An open-source autoregressive language model. arXiv:2204.06745, 2022. [our card](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/)
11. B. Bordelon, A. Atanasov, and C. Pand Pehlevan. A dynamical model of neural scaling laws. In International Conference on Machine Learning (ICML), 2024. [find](https://scholar.google.com/scholar?q=B%2E+Bordelon%2C+A%2E+Atanasov%2C+and+C%2E+Pand+Pehlevan%2E+A+dynamical+model+of+neural+scaling+laws%2E+In+International+Conference+on+Machine+Learning+%28ICML%29%2C+2024%2E)
12. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems (NeurIPS), 2020. [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
13. M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev. Reproducible scaling laws for contrastive language-image learning. In Conference on computer vision and pattern recognition (CVPR), 2023. [our card](https://md.research.elamur.ai/free/en/papers/cherti-2022-reproducible-scaling-laws-contrastive-language/)
14. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113, 2023. [our card](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
15. DeepSeek. Deepseek LLM: Scaling open-source language models with longtermism. arXiv:2401.02954, 2024. [find](https://scholar.google.com/scholar?q=DeepSeek%2E+Deepseek+LLM%3A+Scaling+open%2Dsource+language+models+with+longtermism%2E+arXiv%3A2401%2E02954%2C+2024%2E)
16. M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. P. Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning (ICML), 2023. [find](https://scholar.google.com/scholar?q=M%2E+Dehghani%2C+J%2E+Djolonga%2C+B%2E+Mustafa%2C+P%2E+Padlewski%2C+J%2E+Heek%2C+J%2E+Gilmer%2C+A%2E+P%2E+Steiner%2C+M%2E+Caron%2C+R%2E+Geirhos%2C+I%2E+Alabdulmohsin%2C+et+al%2E+Scaling+vision+transformers+to+22+billion+parameters%2E+In+Internati)
17. S. Y. Gadre, G. Smyrnis, V. Shankar, S. Gururangan, M. Wortsman, R. Shao, J. Mercat, A. Fang, J. Li, S. Keh, R. Xin, M. Nezhurina, I. Vasiljevic, J. Jitsev, A. G. Dimakis, G. Ilharco, S. Song, T. Kollar, Y. Carmon, A. Dave, R. Heckel, N. Muennighoff, and L. Schmidt. Language models scale reliably wi… [find](https://scholar.google.com/scholar?q=S%2E+Y%2E+Gadre%2C+G%2E+Smyrnis%2C+V%2E+Shankar%2C+S%2E+Gururangan%2C+M%2E+Wortsman%2C+R%2E+Shao%2C+J%2E+Mercat%2C+A%2E+Fang%2C+J%2E+Li%2C+S%2E+Keh%2C+R%2E+Xin%2C+M%2E+Nezhurina%2C+I%2E+Vasiljevic%2C+J%2E+Jitsev%2C+A%2E+G%2E+Dimakis%2C+G%2E+Ilharco%2C+S%2E+Song%2C+T%2E+Koll)
18. L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv:2101.00027, 2020. [find](https://scholar.google.com/scholar?q=L%2E+Gao%2C+S%2E+Biderman%2C+S%2E+Black%2C+L%2E+Golding%2C+T%2E+Hoppe%2C+C%2E+Foster%2C+J%2E+Phang%2C+H%2E+He%2C+A%2E+Thite%2C+N%2E+Nabeshima%2C+S%2E+Presser%2C+and+C%2E+Leahy%2E+The+Pile%3A+An+800gb+dataset+of+diverse+text+for+language+modeling%2E+arX)
19. P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv:1706.02677, 2017. [find](https://scholar.google.com/scholar?q=P%2E+Goyal%2C+P%2E+Doll%C3%A1r%2C+R%2E+Girshick%2C+P%2E+Noordhuis%2C+L%2E+Wesolowski%2C+A%2E+Kyrola%2C+A%2E+Tulloch%2C+Y%2E+Jia%2C+and+K%2E+He%2E+Accurate%2C+large+minibatch+SGD%3A+Training+imagenet+in+1+hour%2E+arXiv%3A1706%2E02677%2C+2017%2E)
20. S. Goyal, P. Maini, Z. C. Lipton, A. Raghunathan, and J. Z. Kolter. Scaling laws for data filtering–data curation cannot be compute agnostic. In Conference on computer vision and pattern recognition (CVPR), 2024. [find](https://scholar.google.com/scholar?q=S%2E+Goyal%2C+P%2E+Maini%2C+Z%2E+C%2E+Lipton%2C+A%2E+Raghunathan%2C+and+J%2E+Z%2E+Kolter%2E+Scaling+laws+for+data+filtering%E2%80%93data+curation+cannot+be+compute+agnostic%2E+In+Conference+on+computer+vision+and+pattern+recognition+%28)
21. S. Gururangan, M. Wortsman, S. Y. Gadre, A. Dave, M. Kilian, W. Shi, J. Mercat, G. Smyrnis, G. Ilharco, M. Jordan, R. Heckel, A. Dimakis, A. Farhadi, V. Shankar, and L. Schmidt. OpenLM: a minimal but performative language modeling (lm) repository, 2023. [https://github.com/](https://github.com/) mlfoundations/open_lm. [↗](https://github.com/)
22. A. Hägele, E. Bakouch, A. Kosson, L. B. Allal, L. Von Werra, and M. Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. arXiv:2405.18392, 2024. [our card](https://md.research.elamur.ai/free/en/papers/hagele-2024-scaling-laws-compute-optimal-training/)
23. T. Henighan, J. Kaplan, M. Katz, M. Chen, C. Hesse, J. Jackson, H. Jun, T. B. Brown, P. Dhariwal, S. Gray, et al. Scaling laws for autoregressive generative modeling. arXiv:2010.14701, 2020. [our card](https://md.research.elamur.ai/free/en/papers/henighan-2020-scaling-laws-autoregressive-generative-modeling/)
24. J. Hestness, S. Narang, N. Ardalani, G. F. Diamos, H. Jun, H. Kianinejad, M. M. A. Patwary, Y. Yang, and Y. Zhou. Deep learning scaling is predictable, empirically. arXiv:1712.00409, 2017. [our card](https://md.research.elamur.ai/free/en/papers/hestness-2017-deep-learning-scaling-predictable-empirically/)
25. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems (NeurIPS), 2022. [find](https://scholar.google.com/scholar?q=J%2E+Hoffmann%2C+S%2E+Borgeaud%2C+A%2E+Mensch%2C+E%2E+Buchatskaya%2C+T%2E+Cai%2C+E%2E+Rutherford%2C+D%2E+de+Las+Casas%2C+L%2E+A%2E+Hendricks%2C+J%2E+Welbl%2C+A%2E+Clark%2C+et+al%2E+An+empirical+analysis+of+compute%2Doptimal+large+language+model+t)
26. S. Hu, Y. Tu, X. Han, C. He, G. Cui, X. Long, Z. Zheng, Y. Fang, Y. Huang, W. Zhao, et al. MiniCPM: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024. [our card](https://md.research.elamur.ai/free/en/papers/hu-2024-minicpm-unveiling-potential-small-language/)
27. M. Ivgi, Y. Carmon, and J. Berant. Scaling laws under the microscope: Predicting transformer performance from small scale experiments. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022. [find](https://scholar.google.com/scholar?q=M%2E+Ivgi%2C+Y%2E+Carmon%2C+and+J%2E+Berant%2E+Scaling+laws+under+the+microscope%3A+Predicting+transformer+performance+from+small+scale+experiments%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing)
28. H. J. Jeon and B. Van Roy. arXiv:2407.01456, 2024. Information-theoretic foundations for neural scaling laws. — [arXiv:2407.01456](https://arxiv.org/abs/2407.01456) [find](https://scholar.google.com/scholar?q=H%2E+J%2E+Jeon+and+B%2E+Van+Roy%2E+arXiv%3A2407%2E01456%2C+2024%2E+Information%2Dtheoretic+foundations+for+neural+scaling+laws%2E+%E2%80%94+%5BarXiv%3A2407%2E01456%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2407%2E01456%29)
29. A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, F. B. Diego de las Casas, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mistral 7B. arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Q%2E+Jiang%2C+A%2E+Sablayrolles%2C+A%2E+Mensch%2C+C%2E+Bamford%2C+D%2E+S%2E+Chaplot%2C+F%2E+B%2E+Diego+de+las+Casas%2C+G%2E+Lengyel%2C+G%2E+Lample%2C+L%2E+Saulnier%2C+L%2E+R%2E+Lavaud%2C+M%2E%2DA%2E+Lachaux%2C+P%2E+Stock%2C+T%2E+L%2E+Scao%2C+T%2E+Lavril%2C+T%2E+Wang%2C)
30. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv:2001.08361, 2020. [our card](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)
31. B. Lefaudeux, F. Massa, D. Liskovich, W. Xiong, V. Caggiano, S. Naren, M. Xu, J. Hu, M. Tintore, S. Zhang, P. Labatut, D. Haziza, L. Wehrstedt, J. Reizenstein, and G. Sizov. xformers: A modular and hackable transformer modelling library. [https://github.com/](https://github.com/) facebookresearch/xformers, 2022. [↗](https://github.com/)
32. O. Lieber, O. Sharir, B. Lenz, and Y. Shoham. Jurassic-1: Technical details and evaluation, 2021. [find](https://scholar.google.com/scholar?q=O%2E+Lieber%2C+O%2E+Sharir%2C+B%2E+Lenz%2C+and+Y%2E+Shoham%2E+Jurassic%2D1%3A+Technical+details+and+evaluation%2C+2021%2E)
33. L. Lin, J. Wu, S. M. Kakade, P. L. Bartlett, and J. D. Lee. Scaling laws in linear regression: Compute, parameters, and data. arXiv:2406.08466, 2024. — [arXiv:2406.08466](https://arxiv.org/abs/2406.08466) [find](https://scholar.google.com/scholar?q=L%2E+Lin%2C+J%2E+Wu%2C+S%2E+M%2E+Kakade%2C+P%2E+L%2E+Bartlett%2C+and+J%2E+D%2E+Lee%2E+Scaling+laws+in+linear+regression%3A+Compute%2C+parameters%2C+and+data%2E+arXiv%3A2406%2E08466%2C+2024%2E+%E2%80%94+%5BarXiv%3A2406%2E08466%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2406%2E08)
34. I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv:1711.05101, 2017. [find](https://scholar.google.com/scholar?q=I%2E+Loshchilov+and+F%2E+Hutter%2E+Decoupled+weight+decay+regularization%2E+arXiv%3A1711%2E05101%2C+2017%2E)
35. A. Maloney, D. A. Roberts, and J. Sully. arXiv:2210.16859, 2022. A solvable model of neural scaling laws. — [arXiv:2210.16859](https://arxiv.org/abs/2210.16859) [find](https://scholar.google.com/scholar?q=A%2E+Maloney%2C+D%2E+A%2E+Roberts%2C+and+J%2E+Sully%2E+arXiv%3A2210%2E16859%2C+2022%2E+A+solvable+model+of+neural+scaling+laws%2E+%E2%80%94+%5BarXiv%3A2210%2E16859%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2210%2E16859%29)
36. S. McCandlish, J. Kaplan, D. Amodei, and The OpenAI Dota Team. An empirical model of large-batch training. arXiv:1812.06162, 2018. [find](https://scholar.google.com/scholar?q=S%2E+McCandlish%2C+J%2E+Kaplan%2C+D%2E+Amodei%2C+and+The+OpenAI+Dota+Team%2E+An+empirical+model+of+large%2Dbatch+training%2E+arXiv%3A1812%2E06162%2C+2018%2E)
37. N. Muennighoff, A. Rush, B. Barak, T. Le Scao, N. Tazi, A. Piktus, S. Pyysalo, T. Wolf, and C. A. Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems (NeurIPS), 2024. [find](https://scholar.google.com/scholar?q=N%2E+Muennighoff%2C+A%2E+Rush%2C+B%2E+Barak%2C+T%2E+Le+Scao%2C+N%2E+Tazi%2C+A%2E+Piktus%2C+S%2E+Pyysalo%2C+T%2E+Wolf%2C+and+C%2E+A%2E+Raffel%2E+Scaling+data%2Dconstrained+language+models%2E+Advances+in+Neural+Information+Processing+Systems+%28N)
38. E. Paquette, C. Paquette, L. Xiao, and J. Pennington. 4+3 phases of compute-optimal neural scaling laws. arXiv:2405.15074, 2024. — [arXiv:2405.15074](https://arxiv.org/abs/2405.15074) [find](https://scholar.google.com/scholar?q=E%2E+Paquette%2C+C%2E+Paquette%2C+L%2E+Xiao%2C+and+J%2E+Pennington%2E+4%2B3+phases+of+compute%2Doptimal+neural+scaling+laws%2E+arXiv%3A2405%2E15074%2C+2024%2E+%E2%80%94+%5BarXiv%3A2405%2E15074%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2405%2E15074%29)
39. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. 2019. [our card](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
40. T. Pearce and J. Song. Reconciling Kaplan and Chinchilla scaling laws. arXiv:2406.12907, 2024. — [arXiv:2406.12907](https://arxiv.org/abs/2406.12907) [find](https://scholar.google.com/scholar?q=T%2E+Pearce+and+J%2E+Song%2E+Reconciling+Kaplan+and+Chinchilla+scaling+laws%2E+arXiv%3A2406%2E12907%2C+2024%2E+%E2%80%94+%5BarXiv%3A2406%2E12907%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2406%2E12907%29)
41. G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, H. Alobeidli, A. Cappelli, B. Pannier, E. Almazrouei, and J. Launay. The refinedweb dataset for falcon LLM: Outperforming curated corpora with web data only. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [find](https://scholar.google.com/scholar?q=G%2E+Penedo%2C+Q%2E+Malartic%2C+D%2E+Hesslow%2C+R%2E+Cojocaru%2C+H%2E+Alobeidli%2C+A%2E+Cappelli%2C+B%2E+Pannier%2C+E%2E+Almazrouei%2C+and+J%2E+Launay%2E+The+refinedweb+dataset+for+falcon+LLM%3A+Outperforming+curated+corpora+with+web+data)
42. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners, 2019. [find](https://scholar.google.com/scholar?q=A%2E+Radford%2C+J%2E+Wu%2C+R%2E+Child%2C+D%2E+Luan%2C+D%2E+Amodei%2C+I%2E+Sutskever%2C+et+al%2E+Language+models+are+unsupervised+multitask+learners%2C+2019%2E)
43. J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv:2112.11446, 2021. [find](https://scholar.google.com/scholar?q=J%2E+W%2E+Rae%2C+S%2E+Borgeaud%2C+T%2E+Cai%2C+K%2E+Millican%2C+J%2E+Hoffmann%2C+F%2E+Song%2C+J%2E+Aslanides%2C+S%2E+Henderson%2C+R%2E+Ring%2C+S%2E+Young%2C+et+al%2E+Scaling+language+models%3A+Methods%2C+analysis+%26+insights+from+training+gopher%2E+arX)
44. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 21(140):1–67, 2020. [our card](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)
45. J. S. Rosenfeld, A. Rosenfeld, Y. Belinkov, and N. Shavit. A constructive prediction of the generalization error across scales. arXiv:1909.12673, 2019. [find](https://scholar.google.com/scholar?q=J%2E+S%2E+Rosenfeld%2C+A%2E+Rosenfeld%2C+Y%2E+Belinkov%2C+and+N%2E+Shavit%2E+A+constructive+prediction+of+the+generalization+error+across+scales%2E+arXiv%3A1909%2E12673%2C+2019%2E)
46. N. Sardana and J. Frankle. Beyond Chinchilla-optimal: Accounting for inference in language model scaling laws. arXiv:2401.00448, 2023. [our card](https://md.research.elamur.ai/free/en/papers/sardana-2024-chinchilla-optimal-accounting-inference-language/)
47. T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, et al. BLOOM: A 176b-parameter open-access multilingual language model. arXiv:2211.05100, 2022. [find](https://scholar.google.com/scholar?q=T%2E+L%2E+Scao%2C+A%2E+Fan%2C+C%2E+Akiki%2C+E%2E+Pavlick%2C+S%2E+Ilic%CC%81%2C+D%2E+Hesslow%2C+R%2E+Castagn%C3%A9%2C+A%2E+S%2E+Luccioni%2C+F%2E+Yvon%2C+et+al%2E+BLOOM%3A+A+176b%2Dparameter+open%2Daccess+multilingual+language+model%2E+arXiv%3A2211%2E05100%2C+2022%2E)
48. R. Schaeffer, B. Miranda, and S. Koyejo. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems (NeurIPS), 2024. [our card](https://md.research.elamur.ai/free/en/papers/schaeffer-2023-emergent-abilities-large-language-models/)
49. C. J. Shallue, J. Lee, J. Antognini, J. Sohl-Dickstein, R. Frostig, and G. E. Dahl. Measuring the effects of data parallelism on neural network training. Journal of Machine Learning Research (JMLR), 20(112):1–49, 2019. [find](https://scholar.google.com/scholar?q=C%2E+J%2E+Shallue%2C+J%2E+Lee%2C+J%2E+Antognini%2C+J%2E+Sohl%2DDickstein%2C+R%2E+Frostig%2C+and+G%2E+E%2E+Dahl%2E+Measuring+the+effects+of+data+parallelism+on+neural+network+training%2E+Journal+of+Machine+Learning+Research+%28JMLR%29%2C+2)
50. N. Shazeer. Glu variants improve transformer. arXiv:2002.05202, 2020. [find](https://scholar.google.com/scholar?q=N%2E+Shazeer%2E+Glu+variants+improve+transformer%2E+arXiv%3A2002%2E05202%2C+2020%2E)
51. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv:1909.08053, 2019. [find](https://scholar.google.com/scholar?q=M%2E+Shoeybi%2C+M%2E+Patwary%2C+R%2E+Puri%2C+P%2E+LeGresley%2C+J%2E+Casper%2C+and+B%2E+Catanzaro%2E+Megatron%2DLM%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+arXiv%3A1909%2E08053%2C+2019%2E)
52. S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti, et al. Using DeepSpeed and Megatron to train Megatron-Turing NLG 530B, a large-scale generative language model. rXiv:2201.11990, 2022. [our card](https://md.research.elamur.ai/free/en/papers/smith-2022-using-deepspeed-megatron-train-megatron/)
53. J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. [our card](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/)
54. Y. Tay, M. Dehghani, J. Rao, W. Fedus, S. Abnar, H. W. Chung, S. Narang, D. Yogatama, A. Vaswani, and D. Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers. In International Conference on Learning Representations (ICLR), 2022. [find](https://scholar.google.com/scholar?q=Y%2E+Tay%2C+M%2E+Dehghani%2C+J%2E+Rao%2C+W%2E+Fedus%2C+S%2E+Abnar%2C+H%2E+W%2E+Chung%2C+S%2E+Narang%2C+D%2E+Yogatama%2C+A%2E+Vaswani%2C+and+D%2E+Metzler%2E+Scale+efficiently%3A+Insights+from+pre%2Dtraining+and+fine%2Dtuning+transformers%2E+In+Interna)
55. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971, 2023. [our card](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)
56. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan,… [find](https://scholar.google.com/scholar?q=H%2E+Touvron%2C+L%2E+Martin%2C+K%2E+Stone%2C+P%2E+Albert%2C+A%2E+Almahairi%2C+Y%2E+Babaei%2C+N%2E+Bashlykov%2C+S%2E+Batra%2C+P%2E+Bhargava%2C+S%2E+Bhosale%2C+D%2E+Bikel%2C+L%2E+Blecher%2C+C%2E+C%2E+Ferrer%2C+M%2E+Chen%2C+G%2E+Cucurull%2C+D%2E+Esiobu%2C+J%2E+Fernandes%2C)
57. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. 2017. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
58. M. Wortsman, P. J. Liu, L. Xiao, K. E. Everett, A. A. Alemi, B. Adlam, J. D. Co-Reyes, I. Gur, A. Kumar, R. Novak, J. Pennington, J. Sohl-Dickstein, K. Xu, J. Lee, J. Gilmer, and S. Kornblith. Small-scale proxies for large-scale transformer training instabilities. In International Conference on Lear… [find](https://scholar.google.com/scholar?q=M%2E+Wortsman%2C+P%2E+J%2E+Liu%2C+L%2E+Xiao%2C+K%2E+E%2E+Everett%2C+A%2E+A%2E+Alemi%2C+B%2E+Adlam%2C+J%2E+D%2E+Co%2DReyes%2C+I%2E+Gur%2C+A%2E+Kumar%2C+R%2E+Novak%2C+J%2E+Pennington%2C+J%2E+Sohl%2DDickstein%2C+K%2E+Xu%2C+J%2E+Lee%2C+J%2E+Gilmer%2C+and+S%2E+Kornblith%2E+Small%2Ds)
59. X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer. Scaling vision transformers. In Conference on computer vision and pattern recognition (CVPR), 2022. [find](https://scholar.google.com/scholar?q=X%2E+Zhai%2C+A%2E+Kolesnikov%2C+N%2E+Houlsby%2C+and+L%2E+Beyer%2E+Scaling+vision+transformers%2E+In+Conference+on+computer+vision+and+pattern+recognition+%28CVPR%29%2C+2022%2E)
60. B. Zhang, I. Titov, and R. Sennrich. Improving deep transformer with depth-scaled initialization and merged attention. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019. [find](https://scholar.google.com/scholar?q=B%2E+Zhang%2C+I%2E+Titov%2C+and+R%2E+Sennrich%2E+Improving+deep+transformer+with+depth%2Dscaled+initialization+and+merged+attention%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+2019%2E)
61. G. Zhang, L. Li, Z. Nado, J. Martens, S. Sachdeva, G. Dahl, C. Shallue, and R. B. Grosse. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. In Advances in Neural Information Processing Systems (NeuIPS), 2019. [find](https://scholar.google.com/scholar?q=G%2E+Zhang%2C+L%2E+Li%2C+Z%2E+Nado%2C+J%2E+Martens%2C+S%2E+Sachdeva%2C+G%2E+Dahl%2C+C%2E+Shallue%2C+and+R%2E+B%2E+Grosse%2E+Which+algorithmic+choices+matter+at+which+batch+sizes%3F+insights+from+a+noisy+quadratic+model%2E+In+Advances+in+N)
62. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. OPT: Open pre-trained transformer language models. arXiv:2205.01068, 2022. [our card](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/)
63. Y. Zhang, C. Chen, N. Shi, R. Sun, and Z.-Q. Luo. Adam can converge without any modification on update rules. Advances in neural information processing systems, 35:28386–28399, 2022. [find](https://scholar.google.com/scholar?q=Y%2E+Zhang%2C+C%2E+Chen%2C+N%2E+Shi%2C+R%2E+Sun%2C+and+Z%2E%2DQ%2E+Luo%2E+Adam+can+converge+without+any+modification+on+update+rules%2E+Advances+in+neural+information+processing+systems%2C+35%3A28386%E2%80%9328399%2C+2022%2E)