---
title: "Emergent Linear Representations in World Models of"
description: "The authors argue that the model does not track absolute tile colours (BLACK/WHITE/EMPTY). 49 references, cited by 81 papers here."
authors: ["Neel Nanda", "Andrew Lee", "Martin Wattenberg"]
year: 2023
venue: "arXiv:2309.00941"
source: "https://arxiv.org/abs/2309.00941"
doi: "10.48550/arxiv.2309.00941"
arxiv: "2309.00941"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nanda-2023-emergent-linear-representations-world-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nanda-2023-emergent-linear-representations-world-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / Emergent Linear Representations in World Models of Self-Supervised Sequence Models

# Emergent Linear Representations in World Models of Self-Supervised Sequence Models

**Authors:** [Neel Nanda](https://md.research.elamur.ai/free/en/authors/neel-nanda/), [Andrew Lee](https://md.research.elamur.ai/free/en/authors/andrew-lee/) and [Martin Wattenberg](https://md.research.elamur.ai/free/en/authors/martin-wattenberg/) · **Year:** 2023 · **Venue:** arXiv:2309.00941 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2309.00941) · [PDF](https://arxiv.org/pdf/2309.00941) · [DOI 10.48550/arxiv.2309.00941](https://doi.org/10.48550/arxiv.2309.00941) · [arXiv 2309.00941](https://arxiv.org/abs/2309.00941) · [OpenAlex](https://openalex.org/W4389518382)

**Citations:** 35 citations (OpenAlex, 2026-10-14) · [81](https://md.research.elamur.ai/free/en/papers/nanda-2023-emergent-linear-representations-world-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This paper asks whether the board-state "world model" learned by OthelloGPT is encoded linearly. Li et al. (2023a) found that only non-linear probes could read it. The authors argue that the model does not track absolute tile colours (BLACK/WHITE/EMPTY). Instead, it tracks colours relative to whoever is moving at the current step (MINE/YOURS/EMPTY).

They train one linear probe per layer on the residual stream of the released 8-layer synthetic OthelloGPT using this relative labelling. Accuracy is close to perfect from the middle layers onward. They then test causality by adding probe directions directly to the residual stream, a single vector addition rather than the gradient-based edits of earlier work. This steering changes which moves the model predicts as legal, matching or beating the earlier method.

Further linear analyses show how empty tiles are computed and reveal a linear "flipped" feature. They also find signs that features are refined across layers. Together, the results give concrete support for the linear representation hypothesis and show that understanding representations makes models easier to control.

## Key points
- **Model:** OthelloGPT is an 8-layer GPT with 8 heads per layer, 512-dimensional hidden states and a 60-token move vocabulary. It was trained only to predict uniformly sampled legal next moves, with no knowledge of the rules.
- **Probe data:** linear probes were trained on 3,500,000 game sequences, with 512 validation games and 1,000 held-out test games. They converged after roughly 100,000 samples.
- **Probe accuracy:** the relative (MINE/YOURS/EMPTY) linear probe peaks around 99.6% (layer 6). A linear probe using absolute colours stays near 75%, and the earlier non-linear probe tops out near 98.7%.
- **Interventions:** on a 1,000-case benchmark, flipping tile colours by vector addition left 0.10 average errors, versus 0.12 for gradient-based non-linear edits and 2.723 with no intervention. Erasing tiles with the EMPTY direction left 0.02 errors, versus 0.11 for the non-linear edits.
- **Code:** released at github.com/ajyl/mech_int_othelloGPT; the experiments use the TransformerLens library.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Harvard University](https://md.research.elamur.ai/free/en/universities/harvard-university/) — University, Cambridge, United States · 126 works · 45,130 citations
- [University of Michigan](https://md.research.elamur.ai/free/en/universities/university-of-michigan/) — University, Ann Arbor, United States · 86 works · 15,493 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works

🧭 Topics (1): [🔬 Interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/) · 149 works

## Cite

Cite as: Neel Nanda, Andrew Lee and Martin Wattenberg. 2023. Emergent Linear Representations in World Models of Self-Supervised Sequence Models. arXiv:2309.00941.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{nanda2023emergent,
  title = {{Emergent Linear Representations in World Models of Self-Supervised Sequence Models}},
  author = {Neel Nanda, Andrew Lee and Martin Wattenberg},
  year = {2023},
  doi = {10.48550/arxiv.2309.00941},
  eprint = {2309.00941},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.00941}
}
```

- [← previous Platypus: Quick, Cheap, and Powerful Refinement of LLMs](https://md.research.elamur.ai/free/en/papers/lee-2023-platypus-quick-cheap-powerful-refinement/)
- [next → GlotLID: Language Identification for Low-Resource Languages](https://md.research.elamur.ai/free/en/papers/kargaran-2023-glotlid-language-identification-low-resource/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

49 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Omer Antverg and Yonatan Belinkov. 2021. On the pitfalls of analyzing individual neurons in language models. arXiv preprint arXiv:2110.07483. — [arXiv:2110.07483](https://arxiv.org/abs/2110.07483) [find](https://scholar.google.com/scholar?q=Omer+Antverg+and+Yonatan+Belinkov%2E+2021%2E+On+the+pitfalls+of+analyzing+individual+neurons+in+language+models%2E+arXiv+preprint+arXiv%3A2110%2E07483%2E+%E2%80%94+%5BarXiv%3A2110%2E07483%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2110%2E07483%29)
2. Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2018. [Linear algebraic structure of word senses, with applications to polysemy](https://md.research.elamur.ai/free/en/papers/arora-2018-linear-algebraic-structure-word-senses/). Transactions of the Association for Computational Linguistics, 6:483–495.
3. David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. 2020. Understanding the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences. [find](https://scholar.google.com/scholar?q=David+Bau%2C+Jun%2DYan+Zhu%2C+Hendrik+Strobelt%2C+Agata+Lapedriza%2C+Bolei+Zhou%2C+and+Antonio+Torralba%2E+2020%2E+Understanding+the+role+of+individual+units+in+a+deep+neural+network%2E+Proceedings+of+the+National+Acad)
4. Yonatan Belinkov. 2022a. [Probing classifiers: Promises, shortcomings, and advances](https://md.research.elamur.ai/free/en/papers/belinkov-2022-probing-classifiers-promises-shortcomings-advances/). Computational Linguistics, 48(1):207–219.
5. Yonatan Belinkov. 2022b. [Probing classifiers: Promises, shortcomings, and advances](https://md.research.elamur.ai/free/en/papers/belinkov-2022-probing-classifiers-promises-shortcomings-advances/). Computational Linguistics, 48(1):207–219.
6. Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. [Eliciting latent predictions from transformers with the tuned lens](https://md.research.elamur.ai/free/en/papers/belrose-2023-eliciting-latent-predictions-transformers-tuned/). arXiv preprint arXiv:2303.08112.
7. Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2022. Discovering latent knowledge in language models without supervision. ArXiV. [find](https://scholar.google.com/scholar?q=Collin+Burns%2C+Haotian+Ye%2C+Dan+Klein%2C+and+Jacob+Steinhardt%2E+2022%2E+Discovering+latent+knowledge+in+language+models+without+supervision%2E+ArXiV%2E)
8. Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. 2020. Generative pretraining from pixels. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1691–1703. PMLR. [find](https://scholar.google.com/scholar?q=Mark+Chen%2C+Alec+Radford%2C+Rewon+Child%2C+Jeffrey+Wu%2C+Heewoo+Jun%2C+David+Luan%2C+and+Ilya+Sutskever%2E+2020%2E+Generative+pretraining+from+pixels%2E+In+Proceedings+of+the+37th+International+Conference+on+Machine+L)
9. Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018. [What you can cram into a single $&!\#* vector: Probing sentence embeddings for linguistic properties](https://md.research.elamur.ai/free/en/papers/conneau-2018-what-you-can-cram-into/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long…
10. Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer ElShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Andy Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatfield-Dodds, Jackson Kern… [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Tristan+Hume%2C+Catherine+Olsson%2C+Neel+Nanda%2C+Tom+Henighan%2C+Scott+Johnston%2C+Sheer+ElShowk%2C+Nicholas+Joseph%2C+Nova+DasSarma%2C+Ben+Mann%2C+Danny+Hernandez%2C+Amanda+Askell%2C+Kamal+Ndousse%2C+Andy+Jo)
11. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022b. Toy models of superposition. Transforme… [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Tristan+Hume%2C+Catherine+Olsson%2C+Nicholas+Schiefer%2C+Tom+Henighan%2C+Shauna+Kravec%2C+Zac+Hatfield%2DDodds%2C+Robert+Lasenby%2C+Dawn+Drain%2C+Carol+Chen%2C+Roger+Grosse%2C+Sam+McCandlish%2C+Jared+Kaplan%2C+D)
12. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Ja… [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Neel+Nanda%2C+Catherine+Olsson%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Zac+Hatfield%2DDodds%2C+Da)
13. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational… [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Roei+Schuster%2C+Jonathan+Berant%2C+and+Omer+Levy%2E+2021%2E+Transformer+feed%2Dforward+layers+are+key%2Dvalue+memories%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Pr)
14. Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neura… [find](https://scholar.google.com/scholar?q=Mario+Giulianelli%2C+Jack+Harding%2C+Florian+Mohnert%2C+Dieuwke+Hupkes%2C+and+Willem+Zuidema%2E+2018%2E+Under+the+hood%3A+Using+diagnostic+classifiers+to+investigate+and+improve+how+language+models+track+agreement+)
15. Gabriel Goh, Nick Cammarata †, Chelsea Voss †, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. 2021. Multimodal neurons in artificial neural networks. Distill. Https://distill.pub/2021/multimodal-neurons. [find](https://scholar.google.com/scholar?q=Gabriel+Goh%2C+Nick+Cammarata+%E2%80%A0%2C+Chelsea+Voss+%E2%80%A0%2C+Shan+Carter%2C+Michael+Petrov%2C+Ludwig+Schubert%2C+Alec+Radford%2C+and+Chris+Olah%2E+2021%2E+Multimodal+neurons+in+artificial+neural+networks%2E+Distill%2E+Https%3A%2F%2Fdist)
16. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. 2023. [Finding neurons in a haystack: Case studies with sparse probing](https://md.research.elamur.ai/free/en/papers/gurnee-2023-finding-neurons-haystack-case-studies/). arXiv preprint arXiv:2305.01610.
17. Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. 2023. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. arXiv preprint arXiv:2301.04213. [find](https://scholar.google.com/scholar?q=Peter+Hase%2C+Mohit+Bansal%2C+Been+Kim%2C+and+Asma+Ghandeharioun%2E+2023%2E+Does+localization+inform+editing%3F+surprising+differences+in+causality%2Dbased+localization+vs%2E+knowledge+editing+in+language+models%2E+arX)
18. Evan Hernandez, Belinda Z Li, and Jacob Andreas. 2023. [Measuring and manipulating knowledge representations in language models](https://md.research.elamur.ai/free/en/papers/hernandez-2023-inspecting-editing-knowledge-representations-language/). arXiv preprint arXiv:2304.00740.
19. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022. [Editing models with task arithmetic](https://md.research.elamur.ai/free/en/papers/ilharco-2022-editing-models-task-arithmetic/). arXiv preprint arXiv:2212.04089.
20. Stanisław Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. 2018. Residual connections encourage iterative inference. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Stanis%C5%82aw+Jastrzebski%2C+Devansh+Arpit%2C+Nicolas+Ballas%2C+Vikas+Verma%2C+Tong+Che%2C+and+Yoshua+Bengio%2E+2018%2E+Residual+connections+encourage+iterative+inference%2E+In+International+Conference+on+Learning+Repres)
21. Belinda Z. Li, Maxwell Nye, and Jacob Andreas. 2021. Implicit representations of meaning in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pa… [find](https://scholar.google.com/scholar?q=Belinda+Z%2E+Li%2C+Maxwell+Nye%2C+and+Jacob+Andreas%2E+2021%2E+Implicit+representations+of+meaning+in+neural+language+models%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Lingu)
22. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022a. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+Andonian%2C+and+Yonatan+Belinkov%2E+2022a%2E+Locating+and+editing+factual+associations+in+GPT%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2E)
23. Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2022b. Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+Arnab+Sen+Sharma%2C+Alex+Andonian%2C+Yonatan+Belinkov%2C+and+David+Bau%2E+2022b%2E+Mass%2Dediting+memory+in+a+transformer%2E+arXiv+preprint+arXiv%3A2210%2E07229%2E)
24. Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. 2022. Linearly mapping from image to text space. arXiv preprint arXiv:2209.15162. [find](https://scholar.google.com/scholar?q=Jack+Merullo%2C+Louis+Castricato%2C+Carsten+Eickhoff%2C+and+Ellie+Pavlick%2E+2022%2E+Linearly+mapping+from+image+to+text+space%2E+arXiv+preprint+arXiv%3A2209%2E15162%2E)
25. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). arXiv preprint arXiv:1301.3781.
26. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.
27. Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013c. [Linguistic regularities in continuous space word representations](https://md.research.elamur.ai/free/en/papers/mikolov-2013-linguistic-regularities-continuous-space-word/). In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 746–751.
28. Jesse Mu and Jacob Andreas. 2020. Compositional explanations of neurons. Advances in Neural Information Processing Systems, 33:17153–17163. [find](https://scholar.google.com/scholar?q=Jesse+Mu+and+Jacob+Andreas%2E+2020%2E+Compositional+explanations+of+neurons%2E+Advances+in+Neural+Information+Processing+Systems%2C+33%3A17153%E2%80%9317163%2E)
29. Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023a. Emergent world representations: Exploring a sequence model trained on a synthetic task. In The Eleventh International Conference on Learning Representations. nostalgebraist. 2020. interpreting g… [find](https://scholar.google.com/scholar?q=Kenneth+Li%2C+Aspen+K+Hopkins%2C+David+Bau%2C+Fernanda+Vi%C3%A9gas%2C+Hanspeter+Pfister%2C+and+Martin+Wattenberg%2E+2023a%2E+Emergent+world+representations%3A+Exploring+a+sequence+model+trained+on+a+synthetic+task%2E+In+The)
30. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023b. [Inference-time intervention: Eliciting truthful answers from a language model](https://md.research.elamur.ai/free/en/papers/li-2023-inference-time-intervention-eliciting-truthful/). arXiv preprint arXiv:2306.03341.
31. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. Zoom in: An introduction to circuits. Distill. Https://distill.pub/2020/circuits/zoom-in. [find](https://scholar.google.com/scholar?q=Chris+Olah%2C+Nick+Cammarata%2C+Ludwig+Schubert%2C+Gabriel+Goh%2C+Michael+Petrov%2C+and+Shan+Carter%2E+2020%2E+Zoom+in%3A+An+introduction+to+circuits%2E+Distill%2E+Https%3A%2F%2Fdistill%2Epub%2F2020%2Fcircuits%2Fzoom%2Din%2E)
32. Charles Lovering, Jessica Forde, George Konidaris, Ellie Pavlick, and Michael Littman. 2022. Evaluation beyond task performance: Analyzing concepts in alphazero in hex. In Advances in Neural Information Processing Systems, volume 35, pages 25992–26006. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Charles+Lovering%2C+Jessica+Forde%2C+George+Konidaris%2C+Ellie+Pavlick%2C+and+Michael+Littman%2E+2022%2E+Evaluation+beyond+task+performance%3A+Analyzing+concepts+in+alphazero+in+hex%2E+In+Advances+in+Neural+Informati)
33. Roma Patel and Ellie Pavlick. 2022. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Roma+Patel+and+Ellie+Pavlick%2E+2022%2E+Mapping+language+models+to+grounded+conceptual+spaces%2E+In+International+Conference+on+Learning+Representations%2E)
34. Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. 2022. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, 119(47):e2206625119. [find](https://scholar.google.com/scholar?q=Thomas+McGrath%2C+Andrei+Kapishnikov%2C+Nenad+Toma%C5%A1ev%2C+Adam+Pearce%2C+Martin+Wattenberg%2C+Demis+Hassabis%2C+Been+Kim%2C+Ulrich+Paquet%2C+and+Vladimir+Kramnik%2E+2022%2E+Acquisition+of+chess+knowledge+in+alphazero%2E+Pro)
35. Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. 2023. The hydra effect: Emergent self-repair in language model computations. arXiv preprint arXiv:2307.15771. [find](https://scholar.google.com/scholar?q=Thomas+McGrath%2C+Matthew+Rahtz%2C+Janos+Kramar%2C+Vladimir+Mikulik%2C+and+Shane+Legg%2E+2023%2E+The+hydra+effect%3A+Emergent+self%2Drepair+in+language+model+computations%2E+arXiv+preprint+arXiv%3A2307%2E15771%2E)
36. Neel Nanda, Lawrence Chan, Tom Liberum, Jess Smith, and Jacob Steinhardt. 2023. [Progress measures for grokking via mechanistic interpretability](https://md.research.elamur.ai/free/en/papers/nanda-2023-progress-measures-grokking-mechanistic-interpretability/). arXiv preprint arXiv:2301.05217.
37. Tiago Pimentel, Naomi Saphra, Adina Williams, and Ryan Cotterell. 2020a. Pareto probing: Trading off accuracy for complexity. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3138–3153, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tiago+Pimentel%2C+Naomi+Saphra%2C+Adina+Williams%2C+and+Ryan+Cotterell%2E+2020a%2E+Pareto+probing%3A+Trading+off+accuracy+for+complexity%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Lang)
38. Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020b. [Information-theoretic probing for linguistic structure](https://md.research.elamur.ai/free/en/papers/pimentel-2020-information-theoretic-probing-linguistic-structure/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4609–4622, Online. Association for Com…
39. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
40. Naomi Saphra and Adam Lopez. 2019. [Understanding learning dynamics of language models with SVCCA](https://md.research.elamur.ai/free/en/papers/saphra-2019-understanding-learning-dynamics-language-models/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3257–3267, Minn…
41. Adam Scherlis, Kshitij Sachan, Adam S Jermyn, Joe Benton, and Buck Shlegeris. 2022. Polysemanticity and capacity in neural networks. arXiv preprint arXiv:2210.01892. — [arXiv:2210.01892](https://arxiv.org/abs/2210.01892) [find](https://scholar.google.com/scholar?q=Adam+Scherlis%2C+Kshitij+Sachan%2C+Adam+S+Jermyn%2C+Joe+Benton%2C+and+Buck+Shlegeris%2E+2022%2E+Polysemanticity+and+capacity+in+neural+networks%2E+arXiv+preprint+arXiv%3A2210%2E01892%2E+%E2%80%94+%5BarXiv%3A2210%2E01892%5D%28https%3A%2F%2Farxiv)
42. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. 2018. A general reinforcement learning algorithm that masters chess, shogi, and go thr… [find](https://scholar.google.com/scholar?q=David+Silver%2C+Thomas+Hubert%2C+Julian+Schrittwieser%2C+Ioannis+Antonoglou%2C+Matthew+Lai%2C+Arthur+Guez%2C+Marc+Lanctot%2C+Laurent+Sifre%2C+Dharshan+Kumaran%2C+Thore+Graepel%2C+Timothy+Lillicrap%2C+Karen+Simonyan%2C+and+De)
43. Nishant Subramani, Nivedita Suresh, and Matthew Peters. 2022. [Extracting latent steering vectors from pretrained language models](https://md.research.elamur.ai/free/en/papers/subramani-2022-extracting-latent-steering-vectors-pretrained/). In Findings of the Association for Computational Linguistics: ACL 2022, pages 566–581, Dublin, Ireland. Association for Computational Linguistics.
44. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. [BERT rediscovers the classical NLP pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics.
45. Mycal Tucker, Peng Qian, and Roger Levy. 2021. What if this modified that? syntactic interventions with counterfactual embeddings. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 862–875, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mycal+Tucker%2C+Peng+Qian%2C+and+Roger+Levy%2E+2021%2E+What+if+this+modified+that%3F+syntactic+interventions+with+counterfactual+embeddings%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL%2DIJC)
46. Alex Turner, Monte MacDiarmid, David Udell, lisathier-gart, and Ulisse Mini. 2023. Steering gpt-2-xl by adding an activation vector - ai alignment forum. [find](https://scholar.google.com/scholar?q=Alex+Turner%2C+Monte+MacDiarmid%2C+David+Udell%2C+lisathier%2Dgart%2C+and+Ulisse+Mini%2E+2023%2E+Steering+gpt%2D2%2Dxl+by+adding+an+activation+vector+%2D+ai+alignment+forum%2E)
47. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
48. Andreas Veit, Michael J Wilber, and Serge Belongie. 2016. Residual networks behave like ensembles of relatively shallow networks. Advances in neural information processing systems, 29. [find](https://scholar.google.com/scholar?q=Andreas+Veit%2C+Michael+J+Wilber%2C+and+Serge+Belongie%2E+2016%2E+Residual+networks+behave+like+ensembles+of+relatively+shallow+networks%2E+Advances+in+neural+information+processing+systems%2C+29%2E)
49. Hyperparameter Value Optimizer Learning Rate Weight Decay Betas Validation Step Validation Size Validation Patience AdamW 1e-2 1e-2 0.9, 0.99 200 512 10 *Table 4: Hyperparameters used for our linear probes.* [find](https://scholar.google.com/scholar?q=Hyperparameter+Value+Optimizer+Learning+Rate+Weight+Decay+Betas+Validation+Step+Validation+Size+Validation+Patience+AdamW+1e%2D2+1e%2D2+0%2E9%2C+0%2E99+200+512+10+%2ATable+4%3A+Hyperparameters+used+for+our+linear+p)

*Updated 2 November 2026*