---
title: "Progress measures for grokking via mechanistic — Nanda 2023"
description: "Deep networks sometimes gain new abilities abruptly, which makes their behavior hard to anticipate."
authors: ["Neel Nanda", "Lawrence Chan", "Tom Lieberum", "Jess Smith", "Jacob Steinhardt"]
year: 2023
venue: "arXiv:2301.05217"
source: "https://arxiv.org/abs/2301.05217"
doi: "10.48550/arxiv.2301.05217"
arxiv: "2301.05217"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nanda-2023-progress-measures-grokking-mechanistic-interpretability/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nanda-2023-progress-measures-grokking-mechanistic-interpretability/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Progress measures for grokking via mechanistic interpretability

# Progress measures for grokking via mechanistic interpretability

**Authors:** [Neel Nanda](https://md.research.elamur.ai/free/en/authors/neel-nanda/), Lawrence Chan, Tom Lieberum, Jess Smith, [Jacob Steinhardt](https://md.research.elamur.ai/free/en/authors/jacob-steinhardt/) · **Year:** 2023 · **Venue:** arXiv:2301.05217 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2301.05217) · [PDF](https://arxiv.org/pdf/2301.05217) · [DOI 10.48550/arxiv.2301.05217](https://doi.org/10.48550/arxiv.2301.05217) · [arXiv 2301.05217](https://arxiv.org/abs/2301.05217) · [OpenAlex](https://openalex.org/W4316135772)

**Citations:** 51 citations (openalex-snapshot · 2026-10-14) · [156](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Deep networks sometimes gain new abilities abruptly, which makes their behavior hard to anticipate. This work asks whether such jumps hide smooth underlying progress, and whether interpretability can reveal it. The test case is grokking: small transformers learning addition modulo a prime first memorize the training set and only generalize much later. The authors fully reverse-engineer a one-layer transformer trained on this task. The model embeds each input as sines and cosines at a few key frequencies, combines them with trigonometric identities in its attention and MLP layers, and reads out the answer by making waves interfere constructively at the correct sum. They call this algorithm Fourier multiplication. They support it with periodic structure in weights and activations, approximations of individual neurons, and ablations in Fourier space. Building on this mechanism, they define two progress measures, restricted loss and excluded loss. Both improve steadily well before test accuracy jumps. This shows that grokking is gradual circuit formation followed by weight-decay-driven removal of memorization, rather than a sudden discovery.

## Key points
- Setup: P = 113, a one-layer ReLU transformer (d = 128, 4 heads of size 32, 512 MLP units, no LayerNorm), trained on 30% of all 113·113 input pairs. Training used full-batch AdamW (lr 0.001, weight decay 1) for 40,000 epochs, and generalization appeared around epoch 10,000.
- The neuron-logit map W_L is roughly rank 10, covering the sine and cosine of 5 key frequencies. Most neurons and attention heads fit well as degree-2 trigonometric polynomials of a single frequency.
- Removing the key frequencies drops accuracy to chance level. Removing the remaining ~95% of frequencies slightly improves performance.
- Training splits into three phases: memorization, circuit formation, and cleanup. The sharp rise in test accuracy happens during cleanup, and grokking did not appear without regularization.
- Code and interactive figures are available at neelnanda.io/grokking-paper.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/)

## Cited by

Cited by

156 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 152 not yet.

- [How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://md.research.elamur.ai/free/en/papers/hanna-2023-gpt-2-compute-greater-than/)
- [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/)
- [Large Language Model Alignment: A Survey](https://md.research.elamur.ai/free/en/papers/shen-2023-large-language-model-alignment-survey/)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://md.research.elamur.ai/free/en/papers/wendler-2024-do-llamas-work-english-latent/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{nanda2023progress,
  title = {{Progress measures for grokking via mechanistic interpretability}},
  author = {Nanda, Neel and Chan, Lawrence and Lieberum, Tom and Smith, Jess and Steinhardt, Jacob},
  year = {2023},
  doi = {10.48550/arxiv.2301.05217},
  eprint = {2301.05217},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2301.05217}
}
```

- [← previous Proceedings of the First International Workshop on Construction Grammars and NLP (CxGs+NLP, GURT/SyntaxFest 2023)](https://md.research.elamur.ai/free/en/papers/anon-2023-proceedings-first-international-workshop-construction/)
- [next → Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://md.research.elamur.ai/free/en/papers/kim-2023-prometheus-inducing-fine-grained-evaluation/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

26 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Boaz Barak, Benjamin L Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit. arXiv preprint arXiv:2207.08799, 2022. [find](https://scholar.google.com/scholar?q=Boaz+Barak%2C+Benjamin+L+Edelman%2C+Surbhi+Goel%2C+Sham+Kakade%2C+Eran+Malach%2C+and+Cyril+Zhang%2E+Hidden+progress+in+deep+learning%3A+Sgd+learns+parities+near+the+computational+limit%2E+arXiv+preprint+arXiv%3A2207%2E08)
2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+)
3. Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schubert, Chelsea Voss, Ben Egan, and Swee Kiat Lim. Thread: Circuits. Distill, 2020. doi: 10.23915/distill.00024. [https://distill.pub/2020/circuits](https://distill.pub/2020/circuits). — [doi:10.23915/distill.00024](https://doi.org/10.23915/distill.00024) [↗](https://distill.pub/2020/circuits)
4. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Ja… [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Neel+Nanda%2C+Catherine+Olsson%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Zac+Hatfield%2DDodds%2C+Da)
5. Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018. [find](https://scholar.google.com/scholar?q=Jonathan+Frankle+and+Michael+Carbin%2E+The+lottery+ticket+hypothesis%3A+Finding+sparse%2C+trainable+neural+networks%2E+arXiv+preprint+arXiv%3A1803%2E03635%2C+2018%2E)
6. Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, et al. Predictability and surprise in large generative models. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 1747–1764, 2022. [find](https://scholar.google.com/scholar?q=Deep+Ganguli%2C+Danny+Hernandez%2C+Liane+Lovitt%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+Nova+Dassarma%2C+Dawn+Drain%2C+Nelson+Elhage%2C+et+al%2E+Predictability+and+surprise+in+large+generative+models%2E)
7. Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Rı́o, Mar… [find](https://scholar.google.com/scholar?q=Charles+R%2E+Harris%2C+K%2E+Jarrod+Millman%2C+Ste%CC%81fan+J%2E+van+der+Walt%2C+Ralf+Gommers%2C+Pauli+Virtanen%2C+David+Cournapeau%2C+Eric+Wieser%2C+Julian+Taylor%2C+Sebastian+Berg%2C+Nathaniel+J%2E+Smith%2C+Robert+Kern%2C+Matti+Picus%2C) [our card](https://md.research.elamur.ai/free/en/papers/harris-2020-array-programming-numpy/)
8. Niall Hurley and Scott Rickard. Comparing measures of sparsity. IEEE Transactions on Information Theory, 55(10):4723–4741, 2009. [find](https://scholar.google.com/scholar?q=Niall+Hurley+and+Scott+Rickard%2E+Comparing+measures+of+sparsity%2E+IEEE+Transactions+on+Information+Theory%2C+55%2810%29%3A4723%E2%80%934741%2C+2009%2E)
9. Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. arXiv preprint arXiv:2205.10343, 2022. [find](https://scholar.google.com/scholar?q=Ziming+Liu%2C+Ouail+Kitouni%2C+Niklas+Nolte%2C+Eric+J+Michaud%2C+Max+Tegmark%2C+and+Mike+Williams%2E+Towards+understanding+grokking%3A+An+effective+theory+of+representation+learning%2E+arXiv+preprint+arXiv%3A2205%2E10343)
10. Ilya Loshchilov and Frank Hutter. arXiv:1711.05101, 2017. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+arXiv%3A1711%2E05101%2C+2017%2E)
11. Decoupled weight decay regularization. Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. arXiv preprint arXiv:2111.09259, 2021. — [arXiv:2111.09259](https://arxiv.org/abs/2111.09259) [find](https://scholar.google.com/scholar?q=Decoupled+weight+decay+regularization%2E+Thomas+McGrath%2C+Andrei+Kapishnikov%2C+Nenad+Tomas%CC%8Cev%2C+Adam+Pearce%2C+Demis+Hassabis%2C+Been+Kim%2C+Ulrich+Paquet%2C+and+Vladimir+Kramnik%2E+Acquisition+of+chess+knowledge+in)
12. Beren Millidge. Grokking ’grokking’, 2022. 2022-01-11-Grokking-Grokking/. [find](https://scholar.google.com/scholar?q=Beren+Millidge%2E+Grokking+%E2%80%99grokking%E2%80%99%2C+2022%2E+2022%2D01%2D11%2DGrokking%2DGrokking%2F%2E)
13. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amode… [find](https://scholar.google.com/scholar?q=Catherine+Olsson%2C+Nelson+Elhage%2C+Neel+Nanda%2C+Nicholas+Joseph%2C+Nova+DasSarma%2C+Tom+Henighan%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Zac+Hatfield%2DDodds%2C+Da)
14. Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint arXiv:2201.03544, 2022. — [arXiv:2201.03544](https://arxiv.org/abs/2201.03544) [find](https://scholar.google.com/scholar?q=Alexander+Pan%2C+Kush+Bhatia%2C+and+Jacob+Steinhardt%2E+The+effects+of+reward+misspecification%3A+Mapping+and+mitigating+misaligned+models%2E+arXiv+preprint+arXiv%3A2201%2E03544%2C+2022%2E+%E2%80%94+%5BarXiv%3A2201%2E03544%5D%28https%3A%2F%2F)
15. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019. [find](https://scholar.google.com/scholar?q=Adam+Paszke%2C+Sam+Gross%2C+Francisco+Massa%2C+Adam+Lerer%2C+James+Bradbury%2C+Gregory+Chanan%2C+Trevor+Killeen%2C+Zeming+Lin%2C+Natalia+Gimelshein%2C+Luca+Antiga%2C+et+al%2E+Pytorch%3A+An+imperative+style%2C+high%2Dperformance+) [our card](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
16. Plotly Technologies Inc. Collaborative data science, 2015. URL [https://plot.ly.arXiv](https://plot.ly.arXiv) preprint URL https://www.beren.io/ [↗](https://plot.ly.arXiv)
17. Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022. [find](https://scholar.google.com/scholar?q=Alethea+Power%2C+Yuri+Burda%2C+Harri+Edwards%2C+Igor+Babuschkin%2C+and+Vedant+Misra%2E+Grokking%3A+Generalization+beyond+overfitting+on+small+algorithmic+datasets%2E+arXiv+preprint+arXiv%3A2201%2E02177%2C+2022%2E)
18. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2C+2019%2E)
19. Alex Rogozhnikov. Einops: Clear and reliable tensor manipulations with einstein-like notation. In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?id=oapKSVM2bcj](https://openreview.net/forum?id=oapKSVM2bcj). [↗](https://openreview.net/forum?id=oapKSVM2bcj)
20. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958, 2014. [find](https://scholar.google.com/scholar?q=Nitish+Srivastava%2C+Geoffrey+Hinton%2C+Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Ruslan+Salakhutdinov%2E+Dropout%3A+a+simple+way+to+prevent+neural+networks+from+overfitting%2E+The+journal+of+machine+learning+resear)
21. Jacob Steinhardt. More is different for ai, Feb 2022. URL [https://bounded-regret.ghost.io/more-is-different-for-ai/](https://bounded-regret.ghost.io/more-is-different-for-ai/). [↗](https://bounded-regret.ghost.io/more-is-different-for-ai/)
22. Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua Susskind. The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon. arXiv preprint arXiv:2206.04817, 2022. — [arXiv:2206.04817](https://arxiv.org/abs/2206.04817) [find](https://scholar.google.com/scholar?q=Vimal+Thilak%2C+Etai+Littwin%2C+Shuangfei+Zhai%2C+Omid+Saremi%2C+Roni+Paiss%2C+and+Joshua+Susskind%2E+The+slingshot+mechanism%3A+An+empirical+study+of+adaptive+optimizers+and+the+grokking+phenomenon%2E+arXiv+preprint)
23. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022. [find](https://scholar.google.com/scholar?q=Kevin+Wang%2C+Alexandre+Variengien%2C+Arthur+Conmy%2C+Buck+Shlegeris%2C+and+Jacob+Steinhardt%2E+Interpretability+in+the+wild%3A+a+circuit+for+indirect+object+identification+in+gpt%2D2+small%2E+arXiv+preprint+arXiv%3A22)
24. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022a. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Yi+Tay%2C+Rishi+Bommasani%2C+Colin+Raffel%2C+Barret+Zoph%2C+Sebastian+Borgeaud%2C+Dani+Yogatama%2C+Maarten+Bosma%2C+Denny+Zhou%2C+Donald+Metzler%2C+et+al%2E+Emergent+abilities+of+large+language+models%2E+arXiv+p) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/)
25. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022b. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Maarten+Bosma%2C+Ed+Chi%2C+Quoc+Le%2C+and+Denny+Zhou%2E+Chain+of+thought+prompting+elicits+reasoning+in+large+language+models%2E+arXiv+preprint+arXiv%3A2201%2E11903%2C+2022b%2E) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
26. Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman (eds.), Proceedings of the 9th Python in Science Conference, pp. 56 – 61, 2010. doi: 10.25080/Majora-92bf1922-00a. [find](https://scholar.google.com/scholar?q=Wes+McKinney%2E+Data+Structures+for+Statistical+Computing+in+Python%2E+In+Ste%CC%81fan+van+der+Walt+and+Jarrod+Millman+%28eds%2E%29%2C+Proceedings+of+the+9th+Python+in+Science+Conference%2C+pp%2E+56+%E2%80%93+61%2C+2010%2E+doi%3A+10%2E25)