---
title: "Grokking — paper by Power et al. (2022) — summary and ideas"
description: "Power and colleagues offer a controlled testbed: small synthetic binary-operation tables of the form… 20 references, cited by 100 papers here."
authors: ["Alethea Power", "Yuri Burda", "Harri Edwards", "Igor Babuschkin", "Vedant Misra"]
year: 2022
venue: "arXiv:2201.02177"
source: "https://arxiv.org/abs/2201.02177"
doi: "10.48550/arxiv.2201.02177"
arxiv: "2201.02177"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/power-2022-grokking-generalization-overfitting-small-algorithmic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/power-2022-grokking-generalization-overfitting-small-algorithmic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) / Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

# Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

**Authors:** Alethea Power, [Yuri Burda](https://md.research.elamur.ai/free/en/authors/yuri-burda/), [Harri Edwards](https://md.research.elamur.ai/free/en/authors/harri-edwards/), [Igor Babuschkin](https://md.research.elamur.ai/free/en/authors/igor-babuschkin/) and [Vedant Misra](https://md.research.elamur.ai/free/en/authors/vedant-misra/) · **Year:** 2022 · **Venue:** arXiv:2201.02177 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2201.02177) · [PDF](https://arxiv.org/pdf/2201.02177) · [DOI 10.48550/arxiv.2201.02177](https://doi.org/10.48550/arxiv.2201.02177) · [arXiv 2201.02177](https://arxiv.org/abs/2201.02177) · [OpenAlex](https://openalex.org/W4226434736)

**Citations:** 77 citations (OpenAlex, 2026-10-14) · [100](https://md.research.elamur.ai/free/en/papers/power-2022-grokking-generalization-overfitting-small-algorithmic/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Why heavily overparameterized networks generalize instead of simply memorizing their training data is still poorly understood. With natural datasets, the effect is also hard to isolate. Power and colleagues offer a controlled testbed: small synthetic binary-operation tables of the form a ◦ b = c. Each element is an opaque token, so the model gets no digits, permutation notation or other internal structure. A small transformer is trained on part of a table and must fill in the missing cells. The tasks include modular arithmetic mod 97, composition in the permutation group S5 and various polynomials. The main finding, which the authors call "grokking," is that validation accuracy can stay at chance level long after the training set is fit perfectly, then suddenly climb to 100%. As the training fraction shrinks, the number of optimization steps needed to generalize grows steeply. Weight decay is the most effective intervention for data efficiency. The experiments run on a single GPU, which makes them a cheap testbed for theories of generalization, such as the link between flat minima and good generalization.

## Key points
- On division mod 97 with half the table used for training, training accuracy is near-perfect within 10^3 steps. Validation accuracy barely moves until about 10^5 steps and reaches near-perfect only around 10^6.
- For the S5 product, at about 25–30% training data, removing 1% more data raises the median steps to 99% validation accuracy by 40–50%. Training accuracy still reaches 99% within 10^3–10^4 steps.
- Weight decay cuts the data needed by more than half compared with most other interventions. Decay toward zero works better than decay toward the initial weights. Optimization noise also helps, and the learning rate only works within about one order of magnitude.
- Operations that are symmetric in their operands (x+y, x·y) need less data than asymmetric ones. x−y and x/y need similar amounts of data because they are the same task up to relabeling. x³+xy²+y mod 97 never generalized, even with 95% of the data.
- t-SNE plots of the output-layer weights show circular structure for modular addition and coset clusters for S5. Sharpness of the minimum is negatively correlated with validation accuracy on S5 (Spearman ≈ −0.80).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works

## Cite

Cite as: Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin and Vedant Misra. 2022. Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets. arXiv:2201.02177.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{power2022grokking,
  title = {{Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets}},
  author = {Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin and Vedant Misra},
  year = {2022},
  doi = {10.48550/arxiv.2201.02177},
  eprint = {2201.02177},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2201.02177}
}
```

- [← previous Findings of the IWSLT 2022 Evaluation Campaign](https://md.research.elamur.ai/free/en/papers/anastasopoulos-2022-findings-iwslt-2022-evaluation-campaign/)
- [next → What Artificial Neural Networks Can Tell Us About Human Language Acquisition](https://md.research.elamur.ai/free/en/papers/warstadt-2022-artificial-neural-networks-tell-us/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/)

## References

20 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning practice and the bias-variance trade-off. arXiv preprint arXiv:1812.11118, 2018. — [arXiv:1812.11118](https://arxiv.org/abs/1812.11118) [find](https://scholar.google.com/scholar?q=Mikhail+Belkin%2C+Daniel+Hsu%2C+Siyuan+Ma%2C+and+Soumik+Mandal%2E+Reconciling+modern+machine+learning+practice+and+the+bias%2Dvariance+trade%2Doff%2E+arXiv+preprint+arXiv%3A1812%2E11118%2C+2018%2E+%E2%80%94+%5BarXiv%3A1812%2E11118%5D%28http)
2. Stéphane d’Ascoli, Levent Sagun, and Giulio Biroli. Triple descent and the two kinds of overfitting: Where & why do they appear? arXiv preprint arXiv:2006.03509, 2020. — [arXiv:2006.03509](https://arxiv.org/abs/2006.03509) [find](https://scholar.google.com/scholar?q=Ste%CC%81phane+d%E2%80%99Ascoli%2C+Levent+Sagun%2C+and+Giulio+Biroli%2E+Triple+descent+and+the+two+kinds+of+overfitting%3A+Where+%26+why+do+they+appear%3F+arXiv+preprint+arXiv%3A2006%2E03509%2C+2020%2E+%E2%80%94+%5BarXiv%3A2006%2E03509%5D%28https%3A%2F%2Far)
3. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03819, 2018. [find](https://scholar.google.com/scholar?q=Mostafa+Dehghani%2C+Stephan+Gouws%2C+Oriol+Vinyals%2C+Jakob+Uszkoreit%2C+and+%C5%81ukasz+Kaiser%2E+Universal+transformers%2E+arXiv+preprint+arXiv%3A1807%2E03819%2C+2018%2E)
4. Alex Graves. Adaptive computation time for recurrent neural networks. arXiv:1603.08983, 2016. [find](https://scholar.google.com/scholar?q=Alex+Graves%2E+Adaptive+computation+time+for+recurrent+neural+networks%2E+arXiv%3A1603%2E08983%2C+2016%2E)
5. Alex Graves, Greg Wayne, and Ivo Danihelka. arXiv:1410.5401, 2014. [find](https://scholar.google.com/scholar?q=Alex+Graves%2C+Greg+Wayne%2C+and+Ivo+Danihelka%2E+arXiv%3A1410%2E5401%2C+2014%2E)
6. Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. Learning to transduce with unbounded memory. Advances in neural information processing systems, 28: 1828–1836, 2015. [find](https://scholar.google.com/scholar?q=Edward+Grefenstette%2C+Karl+Moritz+Hermann%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+Learning+to+transduce+with+unbounded+memory%2E+Advances+in+neural+information+processing+systems%2C+28%3A+1828%E2%80%931836%2C+2015%2E)
7. Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1):1–42, 1997. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+Flat+minima%2E+Neural+computation%2C+9%281%29%3A1%E2%80%9342%2C+1997%2E)
8. Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. arXiv preprint arXiv:1912.02178, 2019. — [arXiv:1912.02178](https://arxiv.org/abs/1912.02178) [find](https://scholar.google.com/scholar?q=Yiding+Jiang%2C+Behnam+Neyshabur%2C+Hossein+Mobahi%2C+Dilip+Krishnan%2C+and+Samy+Bengio%2E+Fantastic+generalization+measures+and+where+to+find+them%2E+arXiv+preprint+arXiv%3A1912%2E02178%2C+2019%2E+%E2%80%94+%5BarXiv%3A1912%2E02178%5D%28h)
9. Łukasz Kaiser and Ilya Sutskever. Neural gpus learn algorithms. arXiv preprint arXiv:1511.08228, 2015. [find](https://scholar.google.com/scholar?q=%C5%81ukasz+Kaiser+and+Ilya+Sutskever%2E+Neural+gpus+learn+algorithms%2E+arXiv+preprint+arXiv%3A1511%2E08228%2C+2015%2E)
10. Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. CoRR, abs/1609.04836, 2016. URL http://arxiv.org/abs/1609.04836. [↗](http://arxiv.org/abs/1609.04836)
11. Ilya Loshchilov and Frank Hutter. arXiv:1711.05101, 2017. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+arXiv%3A1711%2E05101%2C+2017%2E)
12. Decoupled weight decay regularization. Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. arXiv preprint arXiv:1912.02292, 2019. [find](https://scholar.google.com/scholar?q=Decoupled+weight+decay+regularization%2E+Preetum+Nakkiran%2C+Gal+Kaplun%2C+Yamini+Bansal%2C+Tristan+Yang%2C+Boaz+Barak%2C+and+Ilya+Sutskever%2E+Deep+double+descent%3A+Where+bigger+models+and+more+data+hurt%2E+arXiv+pre)
13. Arvind Neelakantan, Luke Vilnis, Quoc V Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens. Adding gradient noise improves learning for very deep networks. arXiv preprint arXiv:1511.06807, 2015. [find](https://scholar.google.com/scholar?q=Arvind+Neelakantan%2C+Luke+Vilnis%2C+Quoc+V+Le%2C+Ilya+Sutskever%2C+Lukasz+Kaiser%2C+Karol+Kurach%2C+and+James+Martens%2E+Adding+gradient+noise+improves+learning+for+very+deep+networks%2E+arXiv+preprint+arXiv%3A1511%2E06)
14. Scott Reed and Nando De Freitas. Neural programmer-interpreters. arXiv preprint arXiv:1511.06279, 2015. [find](https://scholar.google.com/scholar?q=Scott+Reed+and+Nando+De+Freitas%2E+Neural+programmer%2Dinterpreters%2E+arXiv+preprint+arXiv%3A1511%2E06279%2C+2015%2E)
15. David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. [Analysing mathematical reasoning abilities of neural models](https://md.research.elamur.ai/free/en/papers/saxton-2019-analysing-mathematical-reasoning-abilities-neural/). arXiv preprint arXiv:1904.01557, 2019.
16. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. [Dropout: a simple way to prevent neural networks from overfitting](https://md.research.elamur.ai/free/en/papers/srivastava-2014-dropout-simple-way-prevent/). The journal of machine learning research, 15(1):1929–1958, 2014.
17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). arXiv preprint arXiv:1706.03762, 2017.
18. Jason Weston, Sumit Chopra, and Antoine Bordes. arXiv:1410.3916, 2014. arXiv preprint Neural turing machines. arXiv preprint arXiv preprint Memory networks. arXiv preprint Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards a… [find](https://scholar.google.com/scholar?q=Jason+Weston%2C+Sumit+Chopra%2C+and+Antoine+Bordes%2E+arXiv%3A1410%2E3916%2C+2014%2E+arXiv+preprint+Neural+turing+machines%2E+arXiv+preprint+arXiv+preprint+Memory+networks%2E+arXiv+preprint+Jason+Weston%2C+Antoine+Bordes)
19. Wojciech Zaremba and Ilya Sutskever. Reinforcement learning neural turing machines-revised. arXiv preprint arXiv:1505.[00521, 2015](https://md.research.elamur.ai/free/en/papers/zaremba-2015-reinforcement-learning-neural-turing-machines/).
20. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016. [find](https://scholar.google.com/scholar?q=Chiyuan+Zhang%2C+Samy+Bengio%2C+Moritz+Hardt%2C+Benjamin+Recht%2C+and+Oriol+Vinyals%2E+Understanding+deep+learning+requires+rethinking+generalization%2E+arXiv+preprint+arXiv%3A1611%2E03530%2C+2016%2E)

*Updated 2 November 2026*