---
title: "Scaling laws — definition and key papers"
description: "Scaling laws are empirical relationships, usually power laws, that describe how a neural network's performance, often its test loss, changes predictably…"
canonical: "https://research.elamur.ai/free/en/concepts/scaling-laws/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/concepts/scaling-laws/)

[home](https://md.research.elamur.ai/free/en/) / [concepts](https://md.research.elamur.ai/free/en/concepts/) / Scaling laws

# 📈 Scaling laws

Concept · 30 key papers · [all concepts](https://md.research.elamur.ai/free/en/concepts/)

Scaling laws are empirical relationships, usually power laws, that describe how a neural network's performance, often its test loss, changes predictably as model size, training data or compute grows. They let researchers forecast large-model behaviour from small experiments and plan how to split a budget.

Also: neural scaling laws, compute-optimal scaling, Chinchilla scaling, Kaplan scaling laws

## What it is

A scaling law summarises a regularity seen across many training runs. When one resource is increased and the others are not bottlenecks, loss falls along a smooth curve. That resource can be parameters, training examples or floating-point operations. On log-log axes the curve is close to a straight line, which is the signature of a power law.

The practical value is extrapolation. Researchers fit the curve on small, cheap models and then predict the loss of a much larger run before paying for it. They also compare architectures, datasets or objectives by the slope of their curves rather than by a single point.

The idea was worked out most fully for autoregressive language models ([Kaplan et al. 2020](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)). It was then extended to other generative domains ([Henighan et al. 2020](https://md.research.elamur.ai/free/en/papers/henighan-2020-scaling-laws-autoregressive-generative-modeling/)), to contrastive image-text models ([Cherti et al. 2022](https://md.research.elamur.ai/free/en/papers/cherti-2022-reproducible-scaling-laws-contrastive-language/)), to mixtures of modalities ([Aghajanyan et al. 2023](https://md.research.elamur.ai/free/en/papers/aghajanyan-2023-scaling-laws-generative-mixed-modal/)) and to source code ([Luo et al. 2026](https://md.research.elamur.ai/free/en/papers/luo-2026-scaling-laws-code-more-data/)).

## The key idea

The core mechanism is a power-law decline of loss as resources grow; in the fuller form, the decline approaches a floor. Two standard forms are shown below. The first is the single-variable form associated with Kaplan and colleagues. The second is the joint parametric form fitted in the Chinchilla work.

```
L(X) ≈ (X_c / X)^α_X        with X = N, D or C
L(N, D) = E + A / N^α + B / D^β
```

In the first line, L is the test cross-entropy loss. X is the resource being scaled: N is the number of model parameters, D is the number of training tokens, and C is the training compute. X_c is a fitted constant that sets the scale. The exponent α_X says how fast loss falls as that resource grows. This form has no irreducible term, so taken literally it falls toward zero rather than toward a floor.

Kaplan and colleagues described separate laws of this kind, L(N), L(D) and L(C). Each holds when the other two resources do not limit training ([Kaplan et al. 2020](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)).

The second line adds the floor explicitly. E is the irreducible loss that remains even with infinite model and data. A, B, α and β are fitted constants. The two fractions are the excess loss due to finite model size and finite data ([Hoffmann et al. 2022](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/); [Besiroglu et al. 2024](https://md.research.elamur.ai/free/en/papers/besiroglu-2024-chinchilla-scaling-replication-attempt/)).

Training compute for a dense transformer is roughly proportional to N times D. Minimising L(N, D) under a fixed compute budget therefore tells you how to divide that budget between a bigger model and more tokens. Hoffmann and colleagues concluded that parameters and tokens should grow roughly in step. This gives the widely quoted rule of about 20 training tokens per parameter. That is the ratio actually used for the 70B Chinchilla model, and it agrees with two of the paper's three estimation methods ([Hoffmann et al. 2022](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/); [Besiroglu et al. 2024](https://md.research.elamur.ai/free/en/papers/besiroglu-2024-chinchilla-scaling-replication-attempt/)).

The earlier Kaplan analysis had instead suggested that model size should outpace data as compute increases ([Kaplan et al. 2020](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/); [Porian et al. 2024](https://md.research.elamur.ai/free/en/papers/porian-2024-resolving-discrepancies-compute-optimal-scaling/)).

A power law also implies diminishing returns. Each constant reduction in loss needs a multiplicative increase in resources. This is why test error falling only as a slow power of dataset size is seen as costly ([Sorscher et al. 2022](https://md.research.elamur.ai/free/en/papers/sorscher-2022-neural-scaling-laws-beating-power/)).

## History and founding papers

The modern programme starts with the 2020 study of neural language models. That study measured loss across several orders of magnitude of size, data and compute, and it drew allocation advice from the fits ([Kaplan et al. 2020](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)). The same year, GPT-3 demonstrated what scaling a language model up could buy in few-shot ability ([Brown et al. 2020](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)).

Follow-up work found similar laws for images, video and other autoregressive generative tasks ([Henighan et al. 2020](https://md.research.elamur.ai/free/en/papers/henighan-2020-scaling-laws-autoregressive-generative-modeling/)). Another line expressed the benefit of pretraining in units of data. When fine-tuning data is scarce, the "effective data transferred" follows its own power law. For text-to-Python transfer, a tenfold larger model is worth roughly a hundredfold more fine-tuning data ([Hernandez et al. 2021](https://md.research.elamur.ai/free/en/papers/hernandez-2021-scaling-laws-transfer/)). A 2022 study then examined how repeated data distorts scaling. It found that even a small share of repeated data can markedly hurt performance, and it linked this harm to damage to the induction heads that support copying ([Hernandez et al. 2022](https://md.research.elamur.ai/free/en/papers/hernandez-2022-scaling-laws-interpretability-learning-repeated/)).

The 2022 Chinchilla paper shifted the field toward smaller models trained on more tokens ([Hoffmann et al. 2022](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)). Its numbers were later scrutinised.

- A replication rebuilt about 240 runs from the paper's figure. As published, the parametric fit of the third method implies about 70 tokens per parameter, at odds with the ratio used in practice. The refit is consistent with roughly 20, in line with the paper's other two estimation methods ([Besiroglu et al. 2024](https://md.research.elamur.ai/free/en/papers/besiroglu-2024-chinchilla-scaling-replication-attempt/)).
- Another study traced the Kaplan–Chinchilla gap to three details. The first was uncounted output-head FLOPs. The second was warmup that was too long for small models. The third was learning rate, batch size and AdamW β2 left untuned per model size. In the authors' own reproduction of the Kaplan setup, the exponent of optimal size against compute was 0.835. Correcting the three details lowered it to 0.497, which matches the Chinchilla result ([Porian et al. 2024](https://md.research.elamur.ai/free/en/papers/porian-2024-resolving-discrepancies-compute-optimal-scaling/)).

A theoretical account, the quantization model, proposes that networks acquire discrete skills ("quanta") in order of how often each skill is useful. If those frequencies follow a Zipf-like distribution, smooth power-law loss emerges as an average of many abrupt steps ([Michaud et al. 2023](https://md.research.elamur.ai/free/en/papers/michaud-2023-quantization-model-neural-scaling/)).

## How it is used today

Scaling laws now guide decisions far beyond choosing N and D for pretraining. Typical uses include the following.

| Use | Representative finding |
| --- | --- |
| Inference-aware training | Counting expected serving cost moves the optimum away from Chinchilla ratios ([Sardana et al. 2024](https://md.research.elamur.ai/free/en/papers/sardana-2024-chinchilla-optimal-accounting-inference-language/)) |
| Quantization | Across over 35,000 runs, 4-bit weights gave the best accuracy per model bit ([Dettmers & Zettlemoyer 2022](https://md.research.elamur.ai/free/en/papers/dettmers-2022-case-4-bit-precision-k/)) |
| Vocabulary size | Optimal vocabulary parameters grow with exponent about 0.83 in non-vocabulary parameters ([Tao et al. 2024](https://md.research.elamur.ai/free/en/papers/tao-2024-scaling-laws-vocabulary-larger-models/)) |
| Distillation | Student loss depends on the teacher only through the teacher's loss ([Busbridge et al. 2025](https://md.research.elamur.ai/free/en/papers/busbridge-2025-distillation-scaling-laws/)) |
| Code | Across 117 training runs, the compute-optimal tokens-per-parameter ratio is much larger for code than for natural language, and the gap widens with budget ([Luo et al. 2026](https://md.research.elamur.ai/free/en/papers/luo-2026-scaling-laws-code-more-data/)) |

Training schedules have also been revisited so that laws can be fitted without a separate run for each amount of training data ([Hägele et al. 2024](https://md.research.elamur.ai/free/en/papers/hagele-2024-scaling-laws-compute-optimal-training/)). Observational scaling laws go further: they predict capabilities from many existing public models rather than from new training runs ([Ruan et al. 2024](https://md.research.elamur.ai/free/en/papers/ruan-2024-observational-scaling-laws-predictability-language/)).

A fast-growing branch studies compute spent after training. Accuracy from repeated sampling and voting rises quickly but approaches a ceiling. On MATH-500, counted by number of samples, Llemma-7B beats Llemma-34B until roughly 128 samples are drawn. At matched compute, the 7B model with the REBASE tree search comes close to the 34B model's accuracy while spending about half the FLOPs ([Wu et al. 2024](https://md.research.elamur.ai/free/en/papers/wu-2024-inference-scaling-laws-empirical-analysis/)). With a well-chosen search strategy and reward model, a 1B model can surpass a 405B model on MATH-500 ([Liu et al. 2025](https://md.research.elamur.ai/free/en/papers/liu-2025-1b-llm-surpass-405b-llm/)).

When verification is priced fairly, simple majority voting wins at most budgets. Generative verification pays off only when compute is very large. On MATH it needed up to 8× more compute merely to equal majority voting, and about 128× more to beat it by 3.8%. In that high-budget regime, the optimal number of solutions grows as compute to the power 0.57, and the optimal number of verifications as compute to the power 0.39 ([Singhi et al. 2025](https://md.research.elamur.ai/free/en/papers/singhi-2025-solve-verify-compute-optimal-problem/)). For generating synthetic training data, samples from a weaker but cheaper model can beat samples from a stronger one at matched compute ([Bansal et al. 2024](https://md.research.elamur.ai/free/en/papers/bansal-2024-smaller-weaker-yet-better-training/)).

Alignment has scaling laws too. Optimising a policy hard against a learned reward model eventually lowers true quality, and this overoptimization follows regular curves ([Gao et al. 2022](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/)).

## Open questions

Several questions remain open.

- How far do fitted laws carry over between datasets? CLIP trained on different datasets scales at different rates on different tasks ([Cherti et al. 2022](https://md.research.elamur.ai/free/en/papers/cherti-2022-reproducible-scaling-laws-contrastive-language/)). In one setting, 7B models fine-tuned on Skywork-MathQA, a synthetic set of 2.5M problem–solution pairs, kept gaining math accuracy as data grew. Whether that lack of saturation holds for other models and corpora is unknown ([Zeng et al. 2024](https://md.research.elamur.ai/free/en/papers/zeng-2024-skywork-math-data-scaling-laws/)).
- How much does architecture change the slope? Among ten families the vanilla Transformer scaled best, and comparisons at a single model size can mislead ([Tay et al. 2023](https://md.research.elamur.ai/free/en/papers/tay-2023-scaling-laws-vs-model-architectures/)).
- How far can training objectives move the curve? A short extra phase with a mixture-of-denoisers objective cost about 0.1–1% of the original FLOPs. After it, the 540B model reached the final quality of PaLM 540B using roughly half the compute ([Tay et al. 2022](https://md.research.elamur.ai/free/en/papers/tay-2022-transcending-scaling-laws-0-1/)).
- Can data selection change the slope itself? In theory, with a high-quality pruning metric, error can fall exponentially rather than as a power law in the size of the pruned dataset. In practice, a self-supervised metric removed 20% of ImageNet without losing accuracy ([Sorscher et al. 2022](https://md.research.elamur.ai/free/en/papers/sorscher-2022-neural-scaling-laws-beating-power/)).
- How should context length be planned? Proposed laws relate the rotary-embedding base to how far a model can extrapolate beyond its training length ([Liu et al. 2023](https://md.research.elamur.ai/free/en/papers/liu-2023-scaling-laws-rope-based/)).

Smooth loss curves also do not guarantee better behaviour. A prize contest received 99 submissions and selected 11 winning tasks that showed robust inverse scaling, where larger models did worse. The study traced this to four causes. Models may repeat memorised patterns instead of following the instructions in the prompt. They may imitate undesirable patterns in training data. They may be pulled off course by easy distractor tasks, or misled by unhelpful demonstrations. Some of these trends turned U-shaped at the largest scale, so extrapolating from small models is risky ([McKenzie et al. 2023](https://md.research.elamur.ai/free/en/papers/mckenzie-2023-inverse-scaling-bigger-isn-t/)).

Capabilities that appear only at large scale also complicate forecasting. Chinchilla 70B can map a known answer onto its multiple-choice letter, a skill its 7B sibling lacks ([Lieberum et al. 2023](https://md.research.elamur.ai/free/en/papers/lieberum-2023-circuit-analysis-interpretability-scale-evidence/)). Whether such capabilities can be predicted from aggregate loss remains a central unresolved question. So does how long power-law returns will continue as high-quality data runs short.

## Key papers

The works this page cites, the most cited first; citations as counted by OpenAlex plus the papers of this library that cite the work, read 2026-10-23.

- [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/) — Tom B. Brown, Benjamin Mann, Nick Ryder et al. · 2020 · 10,782 citations
- [Scaling Laws for Neural Language Models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/) — Jared Kaplan, Sam McCandlish, Tom Henighan et al. · 2020 · 1,814 citations
- [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/) — Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch et al. · 2022 · 1,110 citations
- [Reproducible scaling laws for contrastive language-image learning](https://md.research.elamur.ai/free/en/papers/cherti-2022-reproducible-scaling-laws-contrastive-language/) — Mehdi Cherti, Romain Beaumont, Ross Wightman et al. · 2022 · 750 citations
- [Scaling Laws for Reward Model Overoptimization](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/) — Leo Gao, John Schulman, Jacob Hilton · 2022 · 266 citations
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://md.research.elamur.ai/free/en/papers/sorscher-2022-neural-scaling-laws-beating-power/) — Ben Sorscher, Robert Geirhos, Shashank Shekhar et al. · 2022 · 220 citations
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://md.research.elamur.ai/free/en/papers/dettmers-2022-case-4-bit-precision-k/) — Tim Dettmers, Luke Zettlemoyer · 2022 · 119 citations
- [Scaling Laws for Autoregressive Generative Modeling](https://md.research.elamur.ai/free/en/papers/henighan-2020-scaling-laws-autoregressive-generative-modeling/) — Tom Henighan, Jared Kaplan, Mor Katz et al. · 2020 · 115 citations
- [Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models](https://md.research.elamur.ai/free/en/papers/wu-2024-inference-scaling-laws-empirical-analysis/) — Yangzhen Wu, Zhiqing Sun, Shanda Li et al. · 2024 · 114 citations
- [Scaling Laws for Transfer](https://md.research.elamur.ai/free/en/papers/hernandez-2021-scaling-laws-transfer/) — Danny Hernandez, Jared Kaplan, Tom Henighan et al. · 2021 · 102 citations
- [Inverse Scaling: When Bigger Isn't Better](https://md.research.elamur.ai/free/en/papers/mckenzie-2023-inverse-scaling-bigger-isn-t/) — Ian R. McKenzie, Alexander Lyzhov, Michael Pieler et al. · 2023 · 99 citations
- [Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?](https://md.research.elamur.ai/free/en/papers/tay-2023-scaling-laws-vs-model-architectures/) — Yi Tay, Mostafa Dehghani, Samira Abnar et al. · 2023 · 79 citations
- [Scaling Laws and Interpretability of Learning from Repeated Data](https://md.research.elamur.ai/free/en/papers/hernandez-2022-scaling-laws-interpretability-learning-repeated/) — Danny Hernandez, Tom Brown, Tom Conerly et al. · 2022 · 62 citations
- [Scaling Laws of RoPE-based Extrapolation](https://md.research.elamur.ai/free/en/papers/liu-2023-scaling-laws-rope-based/) — Xiaoran Liu, Hang Yan, Shuo Zhang et al. · 2023 · 53 citations
- [Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla](https://md.research.elamur.ai/free/en/papers/lieberum-2023-circuit-analysis-interpretability-scale-evidence/) — Tom Lieberum, Matthew Rahtz, Janos Kramar et al. · 2023 · 51 citations
- [Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws](https://md.research.elamur.ai/free/en/papers/sardana-2024-chinchilla-optimal-accounting-inference-language/) — Nikhil Sardana, Jacob Portes, Sasha Doubov et al. · 2024 · 47 citations
- [Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies](https://md.research.elamur.ai/free/en/papers/tao-2024-scaling-laws-vocabulary-larger-models/) — Chaofan Tao, Qian Liu, Longxu Dou et al. · 2024 · 42 citations
- [Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling](https://md.research.elamur.ai/free/en/papers/liu-2025-1b-llm-surpass-405b-llm/) — Runze Liu, Junqi Gao, Jian Zhao et al. · 2025 · 41 citations
- [Transcending Scaling Laws with 0.1% Extra Compute](https://md.research.elamur.ai/free/en/papers/tay-2022-transcending-scaling-laws-0-1/) — Yi Tay, Jason Wei, Hyung Won Chung et al. · 2022 · 40 citations
- [Scaling Laws for Generative Mixed-Modal Language Models](https://md.research.elamur.ai/free/en/papers/aghajanyan-2023-scaling-laws-generative-mixed-modal/) — Armen Aghajanyan, Lili Yu, Alexis Conneau et al. · 2023 · 39 citations
- [Observational Scaling Laws and the Predictability of Language Model Performance](https://md.research.elamur.ai/free/en/papers/ruan-2024-observational-scaling-laws-predictability-language/) — Yangjun Ruan, Chris J. Maddison, Tatsunori Hashimoto · 2024 · 28 citations
- [The Quantization Model of Neural Scaling](https://md.research.elamur.ai/free/en/papers/michaud-2023-quantization-model-neural-scaling/) — Eric J. Michaud, Ziming Liu, Uzay Girit et al. · 2023 · 27 citations
- [Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling](https://md.research.elamur.ai/free/en/papers/bansal-2024-smaller-weaker-yet-better-training/) — Hritik Bansal, Arian Hosseini, Rishabh Agarwal et al. · 2024 · 26 citations
- [Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations](https://md.research.elamur.ai/free/en/papers/hagele-2024-scaling-laws-compute-optimal-training/) — Alexander Hägele, Elie Bakouch, Atli Kosson et al. · 2024 · 24 citations
- [Distillation Scaling Laws](https://md.research.elamur.ai/free/en/papers/busbridge-2025-distillation-scaling-laws/) — Dan Busbridge, Amitis Shidani, Floris Weers et al. · 2025 · 23 citations
- [Resolving Discrepancies in Compute-Optimal Scaling of Language Models](https://md.research.elamur.ai/free/en/papers/porian-2024-resolving-discrepancies-compute-optimal-scaling/) — Tomer Porian, Mitchell Wortsman, Jenia Jitsev et al. · 2024 · 14 citations
- [Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models — The Story Goes On](https://md.research.elamur.ai/free/en/papers/zeng-2024-skywork-math-data-scaling-laws/) — Liang Zeng, Liangjun Zhong, Liang Zhao et al. · 2024 · 14 citations
- [When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoning](https://md.research.elamur.ai/free/en/papers/singhi-2025-solve-verify-compute-optimal-problem/) — Nishad Singhi, Hritik Bansal, Arian Hosseini et al. · 2025 · 12 citations
- [Chinchilla Scaling: A replication attempt](https://md.research.elamur.ai/free/en/papers/besiroglu-2024-chinchilla-scaling-replication-attempt/) — Tamay Besiroglu, Ege Erdil, Matthew Barnett et al. · 2024 · 12 citations
- [Scaling Laws for Code: A More Data-Hungry Regime](https://md.research.elamur.ai/free/en/papers/luo-2026-scaling-laws-code-more-data/) — Xianzhen Luo, Wenzhen Zheng, Qingfu Zhu et al. · 2026 · 3 citations

## Related concepts

- [Language model](https://md.research.elamur.ai/free/en/concepts/language-model/) — A language model is a probability distribution over sequences of tokens (words, subwords or characters). It is usually factored into…
- [Transformer and attention](https://md.research.elamur.ai/free/en/concepts/transformer/) — The Transformer is a neural network architecture that represents a sequence by letting each element gather information from all others…
- [Minimum description length](https://md.research.elamur.ai/free/en/concepts/minimum-description-length/) — Minimum description length (MDL) is a model-selection principle rooted in information theory. It says the best explanation of a dataset is…
- [Subword tokenization](https://md.research.elamur.ai/free/en/concepts/subword-tokenization/) — Subword tokenization is a way of splitting text into units smaller than words but usually larger than single characters. A vocabulary of…

## More papers

Topics: [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/)

Shelves: [🤖 AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/) · [💬 Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

- [← previous concept Psycholinguistics](https://md.research.elamur.ai/free/en/concepts/psycholinguistics/)
- [next concept → Semantic parsing](https://md.research.elamur.ai/free/en/concepts/semantic-parsing/)
- [↑ back to concepts](https://md.research.elamur.ai/free/en/concepts/)