--- title: "Chinchilla Scaling: A replication attempt — Besiroglu 2024" description: "The original parameters describe the recovered data poorly, and the reported confidence intervals are far too… 9 references, cited by 12 papers here." authors: ["Tamay Besiroglu", "Ege Erdil", "Matthew Barnett", "Josh You"] year: 2024 venue: "arXiv:2404.10102" source: "https://arxiv.org/abs/2404.10102" doi: "10.48550/arxiv.2404.10102" arxiv: "2404.10102" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/besiroglu-2024-chinchilla-scaling-replication-attempt/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/besiroglu-2024-chinchilla-scaling-replication-attempt/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Chinchilla Scaling: A replication attempt # Chinchilla Scaling: A replication attempt **Authors:** Tamay Besiroglu, Ege Erdil, Matthew Barnett and Josh You · **Year:** 2024 · **Venue:** arXiv:2404.10102 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2404.10102) · [PDF](https://arxiv.org/pdf/2404.10102) · [DOI 10.48550/arxiv.2404.10102](https://doi.org/10.48550/arxiv.2404.10102) · [arXiv 2404.10102](https://arxiv.org/abs/2404.10102) · [OpenAlex](https://openalex.org/W4394907038) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [12](https://md.research.elamur.ai/free/en/papers/besiroglu-2024-chinchilla-scaling-replication-attempt/cited-by/) in our library, of the 140,884 papers whose bibliographies we read ## Summary Hoffmann et al. (2022), the Chinchilla paper, estimated compute-optimal scaling for language models in three ways. The third, Approach 3, fits a parametric loss L(N, D) = E + A/N^α + B/D^β, where N is parameter count and D is training tokens. Epoch AI researchers checked whether this fit holds up. The original data were never released, so they recovered about 240 training runs from the SVG version of the paper's Figure 4. Point positions gave model size and compute, and the colour bar gave loss. They then refit the formula using Hoffmann et al.'s own procedure (Huber loss with δ = 10⁻³ and a grid of initialisations) and bootstrapped the standard errors. Their estimates differ significantly from the published ones, especially for E and β. The original parameters describe the recovered data poorly, and the reported confidence intervals are far too narrow for the number of experiments. The authors trace these problems to an optimizer that stopped before converging and to rounding of the printed coefficients. Their refit agrees with Hoffmann et al.'s other two approaches. This matters because widely used models, including Google's Gemini, rely on these scaling laws. ## Key points - **Refit vs. Hoffmann et al.:** - Refit: E = 1.8172, A = 482.01, B = 2085.43, α = 0.3478, β = 0.3658. - Hoffmann et al. (TeX-source values): E = 1.6934, A = 406.4, B = 410.7, α = 0.3392, β = 0.2849. - A χ² test on bootstrapped covariance gives p \< 10⁻⁵¹, and a likelihood ratio test gives p \< 10⁻²³⁵. - **Implausible confidence intervals:** Hoffmann et al.'s intervals would require over 600,000 experiments, but they likely ran fewer than 500. - **Rounding bias:** Rounding β from 0.2849 to 0.28 biases the data term upward by about 13% at D = 10¹¹ tokens. - **Inconsistent tokens-per-parameter ratio:** - The published fit implies about 70 tokens per parameter, which contradicts the ~20 actually used for the 70B Chinchilla model. - The refit is consistent with roughly 20 (25.6 in the variant that keeps five outlier runs). - At 1e26 FLOP and above, the refit remains compatible with anywhere from 4 to 40 tokens per parameter. - **Code availability:** Analysis code is published at epochai.org/code/analyzing-chinchilla-repo. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/) ## Cite Cite as: Tamay Besiroglu, Ege Erdil, Matthew Barnett and Josh You. 2024. Chinchilla Scaling: A replication attempt. arXiv:2404.10102. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{besiroglu2024chinchilla, title = {{Chinchilla Scaling: A replication attempt}}, author = {Tamay Besiroglu, Ege Erdil, Matthew Barnett and Josh You}, year = {2024}, doi = {10.48550/arxiv.2404.10102}, eprint = {2404.10102}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2404.10102} } ``` - [← previous Amharic LLaMA and LLaVA: Multimodal LLMs for Low Resource Languages](https://md.research.elamur.ai/free/en/papers/andersland-2024-amharic-llama-llava-multimodal-llms/) - [next → Discourse Relation Prediction and Discourse Parsing in Dialogues with Minimal Supervision](https://md.research.elamur.ai/free/en/papers/li-2024-discourse-relation-prediction-discourse-parsing/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) ## References 9 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. (2023). Palm 2 technical report. arXiv preprint arXiv:2305.10403. [our card](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/) 2. Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al. (2024). Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. [find](https://scholar.google.com/scholar?q=Bi%2C+X%2E%2C+Chen%2C+D%2E%2C+Chen%2C+G%2E%2C+Chen%2C+S%2E%2C+Dai%2C+D%2E%2C+Deng%2C+C%2E%2C+Ding%2C+H%2E%2C+Dong%2C+K%2E%2C+Du%2C+Q%2E%2C+Fu%2C+Z%2E%2C+et+al%2E+%282024%29%2E+Deepseek+llm%3A+Scaling+open%2Dsource+language+models+with+longtermism%2E+arXiv+preprint+arXiv%3A240) 3. Bordelon, B., Atanasov, A., and Pehlevan, C. (2024). A dynamical model of neural scaling laws. arXiv preprint arXiv:2402.01092. — [arXiv:2402.01092](https://arxiv.org/abs/2402.01092) [find](https://scholar.google.com/scholar?q=Bordelon%2C+B%2E%2C+Atanasov%2C+A%2E%2C+and+Pehlevan%2C+C%2E+%282024%29%2E+A+dynamical+model+of+neural+scaling+laws%2E+arXiv+preprint+arXiv%3A2402%2E01092%2E+%E2%80%94+%5BarXiv%3A2402%2E01092%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E01092%29) 4. Borgeaud, S. (2024). Twitter. Twitter thread involving discussion on the paper’s methodologies and data replication issues. [find](https://scholar.google.com/scholar?q=Borgeaud%2C+S%2E+%282024%29%2E+Twitter%2E+Twitter+thread+involving+discussion+on+the+paper%E2%80%99s+methodologies+and+data+replication+issues%2E) 5. Google, G. T. (2023). Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. [our card](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/) 6. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022). Training compute-optimal large language models. arXiv preprint arXiv:2203.15556. [our card](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/) 7. Michaud, E., Liu, Z., Girit, U., and Tegmark, M. (2024). The quantization model of neural scaling. Advances in Neural Information Processing Systems, 36. [our card](https://md.research.elamur.ai/free/en/papers/michaud-2023-quantization-model-neural-scaling/) 8. Sardana, N. and Frankle, J. (2023). Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. arXiv preprint arXiv:2401.00448. [our card](https://md.research.elamur.ai/free/en/papers/sardana-2024-chinchilla-optimal-accounting-inference-language/) 9. Wilks, S. S. (1938). The large-sample distribution of the likelihood ratio for testing composite hypotheses. The annals of mathematical statistics, 9(1):60–62. [find](https://scholar.google.com/scholar?q=Wilks%2C+S%2E+S%2E+%281938%29%2E+The+large%2Dsample+distribution+of+the+likelihood+ratio+for+testing+composite+hypotheses%2E+The+annals+of+mathematical+statistics%2C+9%281%29%3A60%E2%80%9362%2E)