---
title: "Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws — Sardana 2024"
description: "The Chinchilla scaling laws suggest how to divide a compute budget between the number of parameters and the amount… 69 references, cited by 47 papers here."
authors: ["Nikhil Sardana", "Jacob Portes", "Sasha Doubov", "Jonathan Frankle"]
year: 2024
venue: "arXiv:2401.00448"
source: "https://arxiv.org/abs/2401.00448"
doi: "10.48550/arxiv.2401.00448"
arxiv: "2401.00448"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sardana-2024-chinchilla-optimal-accounting-inference-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sardana-2024-chinchilla-optimal-accounting-inference-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws

# Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws

**Authors:** Nikhil Sardana, Jacob Portes, Sasha Doubov and Jonathan Frankle · **Year:** 2024 · **Venue:** arXiv:2401.00448 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2401.00448) · [PDF](https://arxiv.org/pdf/2401.00448) · [DOI 10.48550/arxiv.2401.00448](https://doi.org/10.48550/arxiv.2401.00448) · [arXiv 2401.00448](https://arxiv.org/abs/2401.00448)

**Citations:** [47](https://md.research.elamur.ai/free/en/papers/sardana-2024-chinchilla-optimal-accounting-inference-language/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary

The Chinchilla scaling laws suggest how to divide a compute budget between the number of parameters and the amount of training data. However, they account only for training cost and ignore the cost of inference. The authors from Databricks MosaicML change this formulation. They fix a target model quality (pre-training loss) and the expected volume of requests over the model's entire lifetime, and then search for the model size and number of tokens that minimize the combined cost of training and inference. The problem has no analytical solution, so the optimum is found numerically using Newton's method. Costs are estimated twice: in FLOPs and in actual money, taking into account hardware utilization (MFU) and GPU prices. The conclusion: if around a billion requests or more are expected, it is more cost-effective to train a smaller model for longer than Chinchilla recommends. To test this in practice, the authors trained 47 models. Quality kept improving even with extremely long training, and they observed no signs of saturation. The work explains why models such as Llama 2 and Llama 3 were trained on far more tokens than is "optimal" according to Chinchilla.

## Key points

- **Experiments:** 47 models ranging from 150M to 6B parameters. Between 10 and 10,000 tokens were used per parameter, and data were not repeated (a single epoch). Quality was evaluated on the MosaicML Evaluation Gauntlet (MMLU, ARC, PIQA, COPA, etc.).
- **Example in FLOPs:** a developer plans a 13B model and expects 2 trillion inference tokens. If a 7B model is trained on more data instead, total compute decreases by 1.7×10²² FLOPs, i.e. by roughly 17%. For quality at the level of 30B-Chinchilla and 10¹³ inference tokens, the optimum is a 13.6B model with 2.84× more data; this saves 28% of FLOPs.
- **Example in money:** for quality at the level of 30B-Chinchilla and 1.5B requests, a 16B model trained on 3.35T tokens is 17% cheaper.
- **Why money and FLOPs diverge:** for Chinchilla-70B with 2T inference tokens, the overspend in FLOPs is only 1.3%, but in money it is already 36%. The reason is that generating output tokens has roughly 50 times lower MFU than training.
- **Ablation:** if the Chinchilla coefficients are fitted only on typical ratios (≤100 tokens per parameter), the law overestimates the benefit of additional data for very long training.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

## Cite

Cite as: Nikhil Sardana, Jacob Portes, Sasha Doubov and Jonathan Frankle. 2024. Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws. arXiv:2401.00448.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{sardana2024beyond,
  title = {{Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws}},
  author = {Nikhil Sardana, Jacob Portes, Sasha Doubov and Jonathan Frankle},
  year = {2024},
  doi = {10.48550/arxiv.2401.00448},
  eprint = {2401.00448},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.00448}
}
```

- [← previous ArabicaQA: A Comprehensive Dataset for Arabic Question Answering](https://md.research.elamur.ai/free/en/papers/abdallah-2024-arabicaqa-comprehensive-dataset-arabic-question/)
- [next → Metaphor Understanding Challenge Dataset for LLMs](https://md.research.elamur.ai/free/en/papers/tong-2024-metaphor-understanding-challenge-dataset-llms/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

69 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abnar, S., Dehghani, M., Neyshabur, B., and Sedghi, H. Exploring the limits of large scale pre-training. arXiv preprint arXiv:2110.02095, 2021. — [arXiv:2110.02095](https://arxiv.org/abs/2110.02095) [find](https://scholar.google.com/scholar?q=Abnar%2C+S%2E%2C+Dehghani%2C+M%2E%2C+Neyshabur%2C+B%2E%2C+and+Sedghi%2C+H%2E+Exploring+the+limits+of+large+scale+pre%2Dtraining%2E+arXiv+preprint+arXiv%3A2110%2E02095%2C+2021%2E+%E2%80%94+%5BarXiv%3A2110%2E02095%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2110%2E02095%29)
2. AI@Meta. Llama 3 model card, 2024. URL [https://github.com/meta-llama/llama3/blob/main/MODEL](https://github.com/meta-llama/llama3/blob/main/MODEL) CARD.md. [↗](https://github.com/meta-llama/llama3/blob/main/MODEL)
3. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023. [our card](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/)
4. Amini, A., Gabriel, S., Lin, S., Koncel-Kedziorski, R., Choi, Y., and Hajishirzi, H. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. CoRR, abs/1905.13319, 2019. URL http://arxiv.org/abs/1905.13319. [↗](http://arxiv.org/abs/1905.13319) [our card](https://md.research.elamur.ai/free/en/papers/amini-2019-mathqa-interpretable-math-word-problem/)
5. Barton, T. Calibrating the mosaic evaluation gauntlet. [https://www.databricks.com/blog/calibrating-mosaic-evaluation-gauntlet](https://www.databricks.com/blog/calibrating-mosaic-evaluation-gauntlet), April 2024. Blog post. [↗](https://www.databricks.com/blog/calibrating-mosaic-evaluation-gauntlet)
6. Besiroglu, T., Erdil, E., Barnett, M., and You, J. Chinchilla scaling: A replication attempt. arXiv preprint arXiv:2404.10102, 2024. [our card](https://md.research.elamur.ai/free/en/papers/besiroglu-2024-chinchilla-scaling-replication-attempt/)
7. Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432–7439, 2020. [our card](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/)
8. Bordelon, B., Atanasov, A., and Pehlevan, C. A dynamical model of neural scaling laws. arXiv preprint arXiv:2402.01092, 2024. — [arXiv:2402.01092](https://arxiv.org/abs/2402.01092) [find](https://scholar.google.com/scholar?q=Bordelon%2C+B%2E%2C+Atanasov%2C+A%2E%2C+and+Pehlevan%2C+C%2E+A+dynamical+model+of+neural+scaling+laws%2E+arXiv+preprint+arXiv%3A2402%2E01092%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E01092%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E01092%29)
9. Caballero, E., Gupta, K., Rish, I., and Krueger, D. Broken neural scaling laws. arXiv preprint arXiv:2210.14891, 2022. — [arXiv:2210.14891](https://arxiv.org/abs/2210.14891) [find](https://scholar.google.com/scholar?q=Caballero%2C+E%2E%2C+Gupta%2C+K%2E%2C+Rish%2C+I%2E%2C+and+Krueger%2C+D%2E+Broken+neural+scaling+laws%2E+arXiv+preprint+arXiv%3A2210%2E14891%2C+2022%2E+%E2%80%94+%5BarXiv%3A2210%2E14891%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2210%2E14891%29)
10. Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., Lu, Y., and Le, Q. V. Symbolic discovery of optimization algorithms, 2023. [find](https://scholar.google.com/scholar?q=Chen%2C+X%2E%2C+Liang%2C+C%2E%2C+Huang%2C+D%2E%2C+Real%2C+E%2E%2C+Wang%2C+K%2E%2C+Liu%2C+Y%2E%2C+Pham%2C+H%2E%2C+Dong%2C+X%2E%2C+Luong%2C+T%2E%2C+Hsieh%2C+C%2E%2DJ%2E%2C+Lu%2C+Y%2E%2C+and+Le%2C+Q%2E+V%2E+Symbolic+discovery+of+optimization+algorithms%2C+2023%2E)
11. Clark, C., Lee, K., Ming-Wei Chang, T. K., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019. [our card](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/)
12. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018. [find](https://scholar.google.com/scholar?q=Clark%2C+P%2E%2C+Cowhey%2C+I%2E%2C+Etzioni%2C+O%2E%2C+Khot%2C+T%2E%2C+Sabharwal%2C+A%2E%2C+Schoenick%2C+C%2E%2C+and+Tafjord%2C+O%2E+Think+you+have+solved+question+answering%3F+try+arc%2C+the+ai2+reasoning+challenge%2E+arXiv%3A1803%2E05457v1%2C+2018%2E)
13. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
14. De Vries, H. Go smol or go home, 2023. [https://www.harmdevries.com/post/model-size-vs-compute-overhead/](https://www.harmdevries.com/post/model-size-vs-compute-overhead/). [↗](https://www.harmdevries.com/post/model-size-vs-compute-overhead/)
15. URL Dey, N., Gosal, G., Zhiming, Chen, Khachane, H., Marshall, W., Pathria, R., Tom, M., and Hestness, J. Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster, 2023. [find](https://scholar.google.com/scholar?q=URL+Dey%2C+N%2E%2C+Gosal%2C+G%2E%2C+Zhiming%2C+Chen%2C+Khachane%2C+H%2E%2C+Marshall%2C+W%2E%2C+Pathria%2C+R%2E%2C+Tom%2C+M%2E%2C+and+Hestness%2C+J%2E+Cerebras%2Dgpt%3A+Open+compute%2Doptimal+language+models+trained+on+the+cerebras+wafer%2Dscale+cluster)
16. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023. [find](https://scholar.google.com/scholar?q=Frantar%2C+E%2E%2C+Ashkboos%2C+S%2E%2C+Hoefler%2C+T%2E%2C+and+Alistarh%2C+D%2E+Gptq%3A+Accurate+post%2Dtraining+quantization+for+generative+pre%2Dtrained+transformers%2C+2023%2E)
17. Gadre, S. Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J., Keh, S., et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.08540, 2024. [find](https://scholar.google.com/scholar?q=Gadre%2C+S%2E+Y%2E%2C+Smyrnis%2C+G%2E%2C+Shankar%2C+V%2E%2C+Gururangan%2C+S%2E%2C+Wortsman%2C+M%2E%2C+Shao%2C+R%2E%2C+Mercat%2C+J%2E%2C+Fang%2C+A%2E%2C+Li%2C+J%2E%2C+Keh%2C+S%2E%2C+et+al%2E+Language+models+scale+reliably+with+over%2Dtraining+and+on+downstream+tasks%2E)
18. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
19. Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020. [our card](https://md.research.elamur.ai/free/en/papers/henighan-2020-scaling-laws-autoregressive-generative-modeling/)
20. Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. Scaling laws for transfer. arXiv preprint arXiv:2102.01293, 2021. [our card](https://md.research.elamur.ai/free/en/papers/hernandez-2021-scaling-laws-transfer/)
21. Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017. [our card](https://md.research.elamur.ai/free/en/papers/hestness-2017-deep-learning-scaling-predictable-empirically/)
22. Hestness, J., Ardalani, N., and Diamos, G. Beyond human-level accuracy: Computational challenges in deep learning. In Proceedings of the 24th symposium on principles and practice of parallel programming, pp. 1–14, 2019. [find](https://scholar.google.com/scholar?q=Hestness%2C+J%2E%2C+Ardalani%2C+N%2E%2C+and+Diamos%2C+G%2E+Beyond+human%2Dlevel+accuracy%3A+Computational+challenges+in+deep+learning%2E+In+Proceedings+of+the+24th+symposium+on+principles+and+practice+of+parallel+programmi)
23. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L.… [our card](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
24. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models, 2020. [our card](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)
25. Knight, W. Openai’s ceo says the age of giant ai models is already over. Wired, 2023. ISSN 1059-1028. URL [https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/](https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/). [↗](https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/)
26. Korthikanti, V., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B. Reducing activation recomputation in large transformer models, 2022. [our card](https://md.research.elamur.ai/free/en/papers/korthikanti-2022-reducing-activation-recomputation-large-transformer/)
27. Krajewski, J., Ludziejewski, J., Adamczewski, K., Pióro, M., Krutul, M., Antoniak, S., Ciebiera, K., Król, K., Odrzygóźdź, T., Sankowski, P., et al. Scaling laws for fine-grained mixture of experts. arXiv preprint arXiv:2402.07871, 2024. — [arXiv:2402.07871](https://arxiv.org/abs/2402.07871) [find](https://scholar.google.com/scholar?q=Krajewski%2C+J%2E%2C+Ludziejewski%2C+J%2E%2C+Adamczewski%2C+K%2E%2C+Pio%CC%81ro%2C+M%2E%2C+Krutul%2C+M%2E%2C+Antoniak%2C+S%2E%2C+Ciebiera%2C+K%2E%2C+Kro%CC%81l%2C+K%2E%2C+Odrzygo%CC%81z%CC%81dz%CC%81%2C+T%2E%2C+Sankowski%2C+P%2E%2C+et+al%2E+Scaling+laws+for+fine%2Dgrained+mixture+of+exper)
28. Labs, L. Gpu cloud - vms for deep learning. [https://lambdalabs.com/service/gpu-cloud](https://lambdalabs.com/service/gpu-cloud), 2023. Accessed 2023-10-02. [↗](https://lambdalabs.com/service/gpu-cloud)
29. Levesque, H., Davis, E., and Morgenstern, L. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. Citeseer, 2012. [find](https://scholar.google.com/scholar?q=Levesque%2C+H%2E%2C+Davis%2C+E%2E%2C+and+Morgenstern%2C+L%2E+The+winograd+schema+challenge%2E+In+Thirteenth+International+Conference+on+the+Principles+of+Knowledge+Representation+and+Reasoning%2E+Citeseer%2C+2012%2E)
30. Michaud, E., Liu, Z., Girit, U., and Tegmark, M. The quantization model of neural scaling. Advances in Neural Information Processing Systems, 36, 2024. [our card](https://md.research.elamur.ai/free/en/papers/michaud-2023-quantization-model-neural-scaling/)
31. Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Conference on Empirical Methods in Natural Language Processing, 2018. URL [https://api.semanticscholar.org/CorpusID:52183757](https://api.semanticscholar.org/CorpusID:52183757). [↗](https://api.semanticscholar.org/CorpusID:52183757)
32. Mikami, H., Fukumizu, K., Murai, S., Suzuki, S., Kikuchi, Y., Suzuki, T., Maeda, S.-i., and Hayashi, K. A scaling law for synthetic-to-real transfer: How much is your pre-training effective? arXiv preprint arXiv:2108.11018, 2021. — [arXiv:2108.11018](https://arxiv.org/abs/2108.11018) [find](https://scholar.google.com/scholar?q=Mikami%2C+H%2E%2C+Fukumizu%2C+K%2E%2C+Murai%2C+S%2E%2C+Suzuki%2C+S%2E%2C+Kikuchi%2C+Y%2E%2C+Suzuki%2C+T%2E%2C+Maeda%2C+S%2E%2Di%2E%2C+and+Hayashi%2C+K%2E+A+scaling+law+for+synthetic%2Dto%2Dreal+transfer%3A+How+much+is+your+pre%2Dtraining+effective%3F+arXiv+pre)
33. MosaicML. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023. URL www.mosaicml.com/blog/mpt-7b. Accessed: 2024-01-28. [find](https://scholar.google.com/scholar?q=MosaicML%2E+Introducing+mpt%2D7b%3A+A+new+standard+for+open%2Dsource%2C+commercially+usable+llms%2C+2023%2E+URL+www%2Emosaicml%2Ecom%2Fblog%2Fmpt%2D7b%2E+Accessed%3A+2024%2D01%2D28%2E)
34. MosaicML NLP Team. Llm evaluation scores. [https://www.mosaicml.com/llm-evaluation](https://www.mosaicml.com/llm-evaluation), 2023. Accessed: 2023-09-28. [↗](https://www.mosaicml.com/llm-evaluation)
35. Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., and Raffel, C. Scaling data-constrained language models, 2023. [find](https://scholar.google.com/scholar?q=Muennighoff%2C+N%2E%2C+Rush%2C+A%2E+M%2E%2C+Barak%2C+B%2E%2C+Scao%2C+T%2E+L%2E%2C+Piktus%2C+A%2E%2C+Tazi%2C+N%2E%2C+Pyysalo%2C+S%2E%2C+Wolf%2C+T%2E%2C+and+Raffel%2C+C%2E+Scaling+data%2Dconstrained+language+models%2C+2023%2E)
36. NVIDIA. Nvidia a100 datasheet, 2021. URL [https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf](https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf). [↗](https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf)
37. OpenAI and Pilipiszyn, A. Gpt-3 powers the next generation of apps, Mar 2021. URL [https://openai.com/blog/gpt-3-apps](https://openai.com/blog/gpt-3-apps). [↗](https://openai.com/blog/gpt-3-apps)
38. Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016. [our card](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/)
39. Paquette, E., Paquette, C., Xiao, L., and Pennington, J. 4+ 3 phases of compute-optimal neural scaling laws. arXiv preprint arXiv:2405.15074, 2024. — [arXiv:2405.15074](https://arxiv.org/abs/2405.15074) [find](https://scholar.google.com/scholar?q=Paquette%2C+E%2E%2C+Paquette%2C+C%2E%2C+Xiao%2C+L%2E%2C+and+Pennington%2C+J%2E+4%2B+3+phases+of+compute%2Doptimal+neural+scaling+laws%2E+arXiv+preprint+arXiv%3A2405%2E15074%2C+2024%2E+%E2%80%94+%5BarXiv%3A2405%2E15074%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2405%2E1507)
40. Patel, A., Bhattamishra, S., and Goyal, N. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2080–2094, Online, June 2021. Association f… [our card](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/)
41. Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Levskaya, A., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference, 2022. [find](https://scholar.google.com/scholar?q=Pope%2C+R%2E%2C+Douglas%2C+S%2E%2C+Chowdhery%2C+A%2E%2C+Devlin%2C+J%2E%2C+Bradbury%2C+J%2E%2C+Levskaya%2C+A%2E%2C+Heek%2C+J%2E%2C+Xiao%2C+K%2E%2C+Agrawal%2C+S%2E%2C+and+Dean%2C+J%2E+Efficiently+scaling+transformer+inference%2C+2022%2E)
42. Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., and Carmon, Y. Resolving discrepancies in compute-optimal scaling of language models. arXiv preprint arXiv:2406.19146, 2024. [our card](https://md.research.elamur.ai/free/en/papers/porian-2024-resolving-discrepancies-compute-optimal-scaling/)
43. Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation, 2022. [find](https://scholar.google.com/scholar?q=Press%2C+O%2E%2C+Smith%2C+N%2E+A%2E%2C+and+Lewis%2C+M%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2C+2022%2E)
44. Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., van den Driessche, G., Hendricks, L. A., Rauh, M., Huang, P.-S., Glaese, A., Welbl, J., Dathathri, S., Huang… [find](https://scholar.google.com/scholar?q=Rae%2C+J%2E+W%2E%2C+Borgeaud%2C+S%2E%2C+Cai%2C+T%2E%2C+Millican%2C+K%2E%2C+Hoffmann%2C+J%2E%2C+Song%2C+F%2E%2C+Aslanides%2C+J%2E%2C+Henderson%2C+S%2E%2C+Ring%2C+R%2E%2C+Young%2C+S%2E%2C+Rutherford%2C+E%2E%2C+Hennigan%2C+T%2E%2C+Menick%2C+J%2E%2C+Cassirer%2C+A%2E%2C+Powell%2C+R%2E%2C+van+den+)
45. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. [find](https://scholar.google.com/scholar?q=Raffel%2C+C%2E%2C+Shazeer%2C+N%2E%2C+Roberts%2C+A%2E%2C+Lee%2C+K%2E%2C+Narang%2C+S%2E%2C+Matena%2C+M%2E%2C+Zhou%2C+Y%2E%2C+Li%2C+W%2E%2C+and+Liu%2C+P%2E+J%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext+transformer%2C+2023%2E)
46. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv e-prints, art. arXiv:1606.05250, 2016. [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
47. Reddy, S., Chen, D., and Manning, C. D. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266, 2019. doi: 10.1162/tacl a 00266. URL https://aclanthology.org/Q19-1016. [↗](https://aclanthology.org/Q19-1016) [our card](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/)
48. Roemmele, M., Beja, C. A., and Gordon, A. S. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. Papers from the 2011 AAAI Spring Symposium, 2011. [find](https://scholar.google.com/scholar?q=Roemmele%2C+M%2E%2C+Beja%2C+C%2E+A%2E%2C+and+Gordon%2C+A%2E+S%2E+Choice+of+plausible+alternatives%3A+An+evaluation+of+commonsense+causal+reasoning%2E+Papers+from+the+2011+AAAI+Spring+Symposium%2C+2011%2E)
49. Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N. A constructive prediction of the generalization error across scales. arXiv preprint arXiv:1909.12673, 2019. [find](https://scholar.google.com/scholar?q=Rosenfeld%2C+J%2E+S%2E%2C+Rosenfeld%2C+A%2E%2C+Belinkov%2C+Y%2E%2C+and+Shavit%2C+N%2E+A+constructive+prediction+of+the+generalization+error+across+scales%2E+arXiv+preprint+arXiv%3A1909%2E12673%2C+2019%2E)
50. Ruan, Y., Maddison, C. J., and Hashimoto, T. Observational scaling laws and the predictability of language model performance. arXiv preprint arXiv:2405.10938, 2024. [our card](https://md.research.elamur.ai/free/en/papers/ruan-2024-observational-scaling-laws-predictability-language/)
51. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. WINOGRANDE: an adversarial winograd schema challenge at scale. CoRR, abs/1907.10641, 2019. URL http://arxiv.org/abs/1907.10641. [↗](http://arxiv.org/abs/1907.10641)
52. Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019. [our card](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/)
53. Sardana, N. and Frankle, J. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. arXiv preprint arXiv:2401.00448, 2023. — [arXiv:2401.00448](https://arxiv.org/abs/2401.00448) [find](https://scholar.google.com/scholar?q=Sardana%2C+N%2E+and+Frankle%2C+J%2E+Beyond+chinchilla%2Doptimal%3A+Accounting+for+inference+in+language+model+scaling+laws%2E+arXiv+preprint+arXiv%3A2401%2E00448%2C+2023%2E+%E2%80%94+%5BarXiv%3A2401%2E00448%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2401%2E0)
54. Shazeer, N. and Freitas, D. d. Introducing character, Dec 2022. URL [https://blog.character.ai/introducing-character/](https://blog.character.ai/introducing-character/). [↗](https://blog.character.ai/introducing-character/)
55. Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Morcos, A. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523–19536, 2022. [our card](https://md.research.elamur.ai/free/en/papers/sorscher-2022-neural-scaling-laws-beating-power/)
56. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022. [our card](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
57. Talmor, A., Herzig, J., Lourie, N., and Berant, J. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937, 2018. [our card](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/)
58. Tay, Y., Dehghani, M., Abnar, S., Chung, H. W., Fedus, W., Rao, J., Narang, S., Tran, V. Q., Yogatama, D., and Metzler, D. Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207.10551, 2022. [our card](https://md.research.elamur.ai/free/en/papers/tay-2023-scaling-laws-vs-model-architectures/)
59. The Mosaic ML Team. composer. [https://github.com/mosaicml/composer/](https://github.com/mosaicml/composer/), 2021. [↗](https://github.com/mosaicml/composer/)
60. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models, 2023a. [our card](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)
61. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hossei… [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%2C+Martin%2C+L%2E%2C+Stone%2C+K%2E%2C+Albert%2C+P%2E%2C+Almahairi%2C+A%2E%2C+Babaei%2C+Y%2E%2C+Bashlykov%2C+N%2E%2C+Batra%2C+S%2E%2C+Bhargava%2C+P%2E%2C+Bhosale%2C+S%2E%2C+Bikel%2C+D%2E%2C+Blecher%2C+L%2E%2C+Ferrer%2C+C%2E+C%2E%2C+Chen%2C+M%2E%2C+Cucurull%2C+G%2E%2C+Esiobu%2C+D)
62. Tow, J., Bellagente, M., Mahan, D., and Riquelme, C. Stablelm 3b 4e1t, 2023. URL [https://huggingface.co/stabilityai/stablelm-3b-4e1t](https://huggingface.co/stabilityai/stablelm-3b-4e1t). [↗](https://huggingface.co/stabilityai/stablelm-3b-4e1t)
63. Villalobos, P. Scaling laws literature review, 2023. URL [https://epochai.org/blog/scaling-laws-literature-review](https://epochai.org/blog/scaling-laws-literature-review). [↗](https://epochai.org/blog/scaling-laws-literature-review)
64. Villalobos, P. and Atkinson, D. Trading off compute in training and inference, 2023. URL [https://epochai.org/blog/trading-off-compute-in-training-and-inference](https://epochai.org/blog/trading-off-compute-in-training-and-inference). Accessed: 2023-9-26. [↗](https://epochai.org/blog/trading-off-compute-in-training-and-inference)
65. Wolfe, T., Tunstall, L., and von Platen, P. Jeopardy dataset on hugging face hub. [https://huggingface.co/datasets/jeopardy](https://huggingface.co/datasets/jeopardy), 2022. [↗](https://huggingface.co/datasets/jeopardy)
66. Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models, 2023. [find](https://scholar.google.com/scholar?q=Xiao%2C+G%2E%2C+Lin%2C+J%2E%2C+Seznec%2C+M%2E%2C+Wu%2C+H%2E%2C+Demouth%2C+J%2E%2C+and+Han%2C+S%2E+Smoothquant%3A+Accurate+and+efficient+post%2Dtraining+quantization+for+large+language+models%2C+2023%2E)
67. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. [find](https://scholar.google.com/scholar?q=Zellers%2C+R%2E%2C+Holtzman%2C+A%2E%2C+Bisk%2C+Y%2E%2C+Farhadi%2C+A%2E%2C+and+Choi%2C+Y%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2C+2019%2E)
68. Zheng, L., Chiang, W.-L., Sheng, Y., Li, T., Zhuang, S., Wu, Z., Zhuang, Y., Li, Z., Lin, Z., Xing, E. P., Gonzalez, J. E., Stoica, I., and Zhang, H. Lmsys-chat-1m: A large-scale real-world llm conversation dataset, 2023. [our card](https://md.research.elamur.ai/free/en/papers/zheng-2023-lmsys-chat-1m-large-scale/)
69. Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., and Duan, N. Agieval: A human-centric benchmark for evaluating foundation models, 2023. [our card](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/)