---
title: "A Practical Review of Mechanistic Interpretability for"
description: "They build the taxonomy around three core objects of study: features, circuits, and universality. 211 references, cited by 48 papers here."
authors: ["Daking Rai", "Yilun Zhou", "Shi Feng", "Abulhair Saparov", "Ziyu Yao"]
year: 2024
venue: "arXiv:2407.02646"
source: "https://arxiv.org/abs/2407.02646"
doi: "10.48550/arxiv.2407.02646"
arxiv: "2407.02646"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rai-2024-practical-review-mechanistic-interpretability-transformer/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rai-2024-practical-review-mechanistic-interpretability-transformer/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models

# A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models

**Authors:** [Daking Rai](https://md.research.elamur.ai/free/en/authors/daking-rai/), [Yilun Zhou](https://md.research.elamur.ai/free/en/authors/yilun-zhou/), [Shi Feng](https://md.research.elamur.ai/free/en/authors/shi-feng/), [Abulhair Saparov](https://md.research.elamur.ai/free/en/authors/abulhair-saparov/) and [Ziyu Yao](https://md.research.elamur.ai/free/en/authors/ziyu-yao/) · **Year:** 2024 · **Venue:** arXiv:2407.02646 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2407.02646) · [PDF](https://arxiv.org/pdf/2407.02646) · [DOI 10.48550/arxiv.2407.02646](https://doi.org/10.48550/arxiv.2407.02646) · [arXiv 2407.02646](https://arxiv.org/abs/2407.02646) · [OpenAlex](https://openalex.org/W4400375635)

**Citations:** 10 citations (OpenAlex, 2026-10-14) · [48](https://md.research.elamur.ai/free/en/papers/rai-2024-practical-review-mechanistic-interpretability-transformer/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This survey targets a gap in mechanistic interpretability (MI), the line of work that tries to explain transformer-based language models by reverse-engineering their internal computations into mechanisms people can understand. The field has produced many findings, but newcomers lack a structured entry point. Instead of organizing the literature by technique, the authors group it by research question. They build the taxonomy around three core objects of study: features, circuits, and universality. For each one they give a step-by-step workflow, called the Beginner's Roadmap, that runs from forming a hypothesis to running experiments. They also describe the main toolkits, including vocabulary projection, interventions, sparse autoencoders, probing, and visualization, with each tool's limitations and how it is evaluated. Case studies show how earlier papers fit the workflows. The review then covers findings on model capabilities, training dynamics, applications, and benchmarks. It closes with open problems such as weak evaluation standards and the reliance on manual hypothesis-building. The work matters as a practical guide for new researchers and as a planning aid for experienced ones.

## Key points
- The authors compare their survey with six related surveys, among them Ferrando et al. (2024), Bereska & Gavves (2024), and Sharkey et al. (2025). They claim theirs is the only one that is both task-centric and covers techniques, LM findings, applications, and future directions.
- Covered findings include in-context learning, reasoning, knowledge mechanisms, phase changes during training, and the effects of post-training. Covered applications include model enhancement and AI safety.
- The authors judge current benchmarks insufficient. For example, Tracr compiles RASP programs into transformers with known ground truth, but results on these synthetic models may not carry over to naturally trained models.
- Proposed future directions include:
  - automated hypothesis generation;
  - decoding higher-level propositions, such as the binding vectors of Feng & Steinhardt (2023);
  - human-AI collaboration built on mechanistic explanations, as in SHIFT (Marks et al., 2024b).
- An accompanying paper list is maintained on GitHub (awesome-mechanistic-interpretability-lm-papers). The version reviewed here is v4, dated October 2025.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [George Washington University](https://md.research.elamur.ai/free/en/universities/george-washington-university/) — University, Washington, United States · 52 works · 8,206 citations
- [Purdue University West Lafayette](https://md.research.elamur.ai/free/en/universities/purdue-university-west-lafayette/) — University, West Lafayette, United States · 28 works · 2,141 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works · [🔬 Interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/) · 156 works

## Cite

Cite as: Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov and Ziyu Yao. 2024. A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models. arXiv:2407.02646.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{rai2024practical,
  title = {{A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models}},
  author = {Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov and Ziyu Yao},
  year = {2024},
  doi = {10.48550/arxiv.2407.02646},
  eprint = {2407.02646},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.02646}
}
```

- [← previous The Mamba in the Llama: Distilling and Accelerating Hybrid Models](https://md.research.elamur.ai/free/en/papers/wang-2024-mamba-llama-distilling-accelerating-hybrid/)
- [next → Bias in Large Language Models: Origin, Evaluation, and Mitigation](https://md.research.elamur.ai/free/en/papers/guo-2024-bias-large-language-models-origin/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

211 references, 68 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
2. Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375, 2018. — [arXiv:1803.08375](https://arxiv.org/abs/1803.08375) [find](https://scholar.google.com/scholar?q=Abien+Fred+Agarap%2E+Deep+learning+using+rectified+linear+units+%28relu%29%2E+arXiv+preprint+arXiv%3A1803%2E08375%2C+2018%2E+%E2%80%94+%5BarXiv%3A1803%2E08375%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1803%2E08375%29)
3. Evan Anders and Joseph Bloom. Examining language model performance with reconstructed activations using sparse autoencoders. [https://www.lesswrong.com/posts/8QRH8wKcnKGhpAu2o/](https://www.lesswrong.com/posts/8QRH8wKcnKGhpAu2o/) examining-language-model-performance-with-reconstructed, 2024. [↗](https://www.lesswrong.com/posts/8QRH8wKcnKGhpAu2o/)
4. Evan Anders, Clement Neo, Jason Hoelscher-Obermaier, and Jessica N Howard. Sparse autoencoders find composed features in small toy models, 2024. [find](https://scholar.google.com/scholar?q=Evan+Anders%2C+Clement+Neo%2C+Jason+Hoelscher%2DObermaier%2C+and+Jessica+N+Howard%2E+Sparse+autoencoders+find+composed+features+in+small+toy+models%2C+2024%2E)
5. Omer Antverg and Yonatan Belinkov. On the pitfalls of analyzing individual neurons in language models. In International Conference on Learning Representations, 2021. [find](https://scholar.google.com/scholar?q=Omer+Antverg+and+Yonatan+Belinkov%2E+On+the+pitfalls+of+analyzing+individual+neurons+in+language+models%2E+In+International+Conference+on+Learning+Representations%2C+2021%2E)
6. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. [Refusal in language models is mediated by a single direction](https://md.research.elamur.ai/free/en/papers/arditi-2024-refusal-language-models-mediated-single/). arXiv preprint arXiv:2406.11717, 2024.
7. Aryaman Arora, Dan Jurafsky, and Christopher Potts. [CausalGym: Benchmarking causal interpretability methods on linguistic tasks](https://md.research.elamur.ai/free/en/papers/arora-2024-causalgym-benchmarking-causal-interpretability-methods/). In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p…
8. Tomer Ashuach, Martin Tutek, and Yonatan Belinkov. Revs: Unlearning sensitive information in language models via rank editing in the vocabulary space. arXiv preprint arXiv:2406.09325, 2024. [find](https://scholar.google.com/scholar?q=Tomer+Ashuach%2C+Martin+Tutek%2C+and+Yonatan+Belinkov%2E+Revs%3A+Unlearning+sensitive+information+in+language+models+via+rank+editing+in+the+vocabulary+space%2E+arXiv+preprint+arXiv%3A2406%2E09325%2C+2024%2E)
9. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. arXiv:1607.[06450, 2016](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/).
10. Layer normalization. arXiv preprint Sarah Ball, Frauke Kreuter, and Nina Panickssery. Understanding jailbreak success: A study of latent space dynamics in large language models, 2024. [find](https://scholar.google.com/scholar?q=Layer+normalization%2E+arXiv+preprint+Sarah+Ball%2C+Frauke+Kreuter%2C+and+Nina+Panickssery%2E+Understanding+jailbreak+success%3A+A+study+of+latent+space+dynamics+in+large+language+models%2C+2024%2E)
11. Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. Does the whole exceed its parts? the effect of ai explanations on complementary team performance, 2021. [find](https://scholar.google.com/scholar?q=Gagan+Bansal%2C+Tongshuang+Wu%2C+Joyce+Zhou%2C+Raymond+Fok%2C+Besmira+Nushi%2C+Ece+Kamar%2C+Marco+Tulio+Ribeiro%2C+and+Daniel+Weld%2E+Does+the+whole+exceed+its+parts%3F+the+effect+of+ai+explanations+on+complementary+te)
12. Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff, and Dan Roth. Rethinking the role of scale for in-context learning: An interpretability-based case study at 66 billion scale. In Proceedings of the 61st Annual Meeting of the Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Hritik+Bansal%2C+Karthik+Gopalakrishnan%2C+Saket+Dingliwal%2C+Sravan+Bodapati%2C+Katrin+Kirchhoff%2C+and+Dan+Roth%2E+Rethinking+the+role+of+scale+for+in%2Dcontext+learning%3A+An+interpretability%2Dbased+case+study+at+6)
13. Yonatan Belinkov. [Probing classifiers: Promises, shortcomings, and advances](https://md.research.elamur.ai/free/en/papers/belinkov-2022-probing-classifiers-promises-shortcomings-advances/). Computational Linguistics, 48(1):207–219, 2022.
14. Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. [Eliciting latent predictions from transformers with the tuned lens](https://md.research.elamur.ai/free/en/papers/belrose-2023-eliciting-latent-predictions-transformers-tuned/). arXiv preprint arXiv:2303.08112, 2023.
15. Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Trevor Darrell, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, et al. Managing extreme ai risks amid rapid progress. Science, 384(6698):842–845, 2024. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+Geoffrey+Hinton%2C+Andrew+Yao%2C+Dawn+Song%2C+Pieter+Abbeel%2C+Trevor+Darrell%2C+Yuval+Noah+Harari%2C+Ya%2DQin+Zhang%2C+Lan+Xue%2C+Shai+Shalev%2DShwartz%2C+et+al%2E+Managing+extreme+ai+risks+amid+rapid+progres)
16. Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety–a review. arXiv preprint arXiv:2404.[14082, 2024](https://md.research.elamur.ai/free/en/papers/bereska-2024-mechanistic-interpretability-ai-safety-review/).
17. Adithya Bhaskar, Alexander Wettig, Dan Friedman, and Danqi Chen. Finding transformer circuits with edge pruning. arXiv preprint arXiv:2406.16778, 2024. [find](https://scholar.google.com/scholar?q=Adithya+Bhaskar%2C+Alexander+Wettig%2C+Dan+Friedman%2C+and+Danqi+Chen%2E+Finding+transformer+circuits+with+edge+pruning%2E+arXiv+preprint+arXiv%3A2406%2E16778%2C+2024%2E)
18. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on M… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Gregory+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+et+al%2E+Pythia%3A+A+sui)
19. Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou. Birth of a transformer: A memory viewpoint. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Alberto+Bietti%2C+Vivien+Cabannes%2C+Diane+Bouchacourt%2C+Herve+Jegou%2C+and+Leon+Bottou%2E+Birth+of+a+transformer%3A+A+memory+viewpoint%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2C+2024%2E)
20. Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. [https://openai.com/index/language-models-can-explain-neurons-in-language-models/](https://openai.com/index/language-models-can-explain-neurons-in-language-models/), 2023. [↗](https://openai.com/index/language-models-can-explain-neurons-in-language-models/)
21. Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, and Amir Globerson. [Hopping too late: Exploring the limitations of large language models on multi-hop queries](https://md.research.elamur.ai/free/en/papers/biran-2024-hopping-too-late-exploring-limitations/). arXiv preprint arXiv:2406.12775, 2024.
22. Dan Braun, Jordan Taylor, Nicholas Goldowsky-Dill, and Lee Sharkey. Identifying functionally important features with end-to-end sparse dictionary learning. arXiv preprint arXiv:2405.12241, 2024. — [arXiv:2405.12241](https://arxiv.org/abs/2405.12241) [find](https://scholar.google.com/scholar?q=Dan+Braun%2C+Jordan+Taylor%2C+Nicholas+Goldowsky%2DDill%2C+and+Lee+Sharkey%2E+Identifying+functionally+important+features+with+end%2Dto%2Dend+sparse+dictionary+learning%2E+arXiv+preprint+arXiv%3A2405%2E12241%2C+2024%2E+%E2%80%94+%5Bar)
23. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah… [find](https://scholar.google.com/scholar?q=Trenton+Bricken%2C+Adly+Templeton%2C+Joshua+Batson%2C+Brian+Chen%2C+Adam+Jermyn%2C+Tom+Conerly%2C+Nick+Turner%2C+Cem+Anil%2C+Carson+Denison%2C+Amanda+Askell%2C+Robert+Lasenby%2C+Yifan+Wu%2C+Shauna+Kravec%2C+Nicholas+Schiefer%2C+)
24. Jannik Brinkmann, Abhay Sheshadri, Victor Levoso, Paul Swoboda, and Christian Bartelt. A mechanistic analysis of a transformer trained on a symbolic multi-step reasoning task. arXiv preprint arXiv:2402.11917, 2024. [find](https://scholar.google.com/scholar?q=Jannik+Brinkmann%2C+Abhay+Sheshadri%2C+Victor+Levoso%2C+Paul+Swoboda%2C+and+Christian+Bartelt%2E+A+mechanistic+analysis+of+a+transformer+trained+on+a+symbolic+multi%2Dstep+reasoning+task%2E+arXiv+preprint+arXiv%3A240)
25. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
26. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712, 2023.
27. Lennart Bürger, Fred A Hamprecht, and Boaz Nadler. Truth is universal: Robust detection of lies in llms. arXiv preprint arXiv:2407.12831, 2024. — [arXiv:2407.12831](https://arxiv.org/abs/2407.12831) [find](https://scholar.google.com/scholar?q=Lennart+B%C3%BCrger%2C+Fred+A+Hamprecht%2C+and+Boaz+Nadler%2E+Truth+is+universal%3A+Robust+detection+of+lies+in+llms%2E+arXiv+preprint+arXiv%3A2407%2E12831%2C+2024%2E+%E2%80%94+%5BarXiv%3A2407%2E12831%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2407%2E12831%29)
28. Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda. Learning multi-level features with matryoshka sparse autoencoders. In Forty-second International Conference on Machine Learning, 2025. URL [https://openreview.net/forum?id=m25T5rAy43](https://openreview.net/forum?id=m25T5rAy43). [↗](https://openreview.net/forum?id=m25T5rAy43)
29. Nick Cammarata, Gabriel Goh, Shan Carter, Ludwig Schubert, Michael Petrov, and Chris Olah. Curve detectors. Distill, 5(6):e00024–003, 2020. [find](https://scholar.google.com/scholar?q=Nick+Cammarata%2C+Gabriel+Goh%2C+Shan+Carter%2C+Ludwig+Schubert%2C+Michael+Petrov%2C+and+Chris+Olah%2E+Curve+detectors%2E+Distill%2C+5%286%29%3Ae00024%E2%80%93003%2C+2020%2E)
30. Nicola Cancedda. Spectral filters, dark signals, and attention sinks. arXiv preprint arXiv:2402.09221, 2024. [find](https://scholar.google.com/scholar?q=Nicola+Cancedda%2E+Spectral+filters%2C+dark+signals%2C+and+attention+sinks%2E+arXiv+preprint+arXiv%3A2402%2E09221%2C+2024%2E)
31. Samuel Carton, Qiaozhu Mei, and Paul Resnick. Feature-based explanations don’t help people detect misclassifications of online toxicity, 2020. [find](https://scholar.google.com/scholar?q=Samuel+Carton%2C+Qiaozhu+Mei%2C+and+Paul+Resnick%2E+Feature%2Dbased+explanations+don%E2%80%99t+help+people+detect+misclassifications+of+online+toxicity%2C+2020%2E)
32. Stephen Casper. The engineer’s interpretability sequence. a6ne2ve5uturEEQK7, 2023a. [https://www.lesswrong.com/s/](https://www.lesswrong.com/s/) [↗](https://www.lesswrong.com/s/)
33. Stephen Casper. Is mechanistic interpretability about to be practically useful? [https://www.lesswrong.com/s/a6ne2ve5uturEEQK7/p/Es2qzCxhJ8QYsckaA](https://www.lesswrong.com/s/a6ne2ve5uturEEQK7/p/Es2qzCxhJ8QYsckaA), 2023b. [↗](https://www.lesswrong.com/s/a6ne2ve5uturEEQK7/p/Es2qzCxhJ8QYsckaA)
34. Stephen Casper, Tong Bu, Yuxiao Li, Jiawei Li, Kevin Zhang, Kaivalya Hariharan, and Dylan Hadfield-Menell. Red teaming deep neural networks with feature synthesis tools. Advances in Neural Information Processing Systems, 36:80470–80516, 2023a. [find](https://scholar.google.com/scholar?q=Stephen+Casper%2C+Tong+Bu%2C+Yuxiao+Li%2C+Jiawei+Li%2C+Kevin+Zhang%2C+Kaivalya+Hariharan%2C+and+Dylan+Hadfield%2DMenell%2E+Red+teaming+deep+neural+networks+with+feature+synthesis+tools%2E+Advances+in+Neural+Information)
35. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023b. [find](https://scholar.google.com/scholar?q=Stephen+Casper%2C+Xander+Davies%2C+Claudia+Shi%2C+Thomas+Krendl+Gilbert%2C+J%C3%A9r%C3%A9my+Scheurer%2C+Javier+Rando%2C+Rachel+Freedman%2C+Tomasz+Korbak%2C+David+Lindner%2C+Pedro+Freire%2C+et+al%2E+Open+problems+and+fundamental+limi)
36. David J Chalmers. Propositional interpretability in artificial intelligence. arXiv preprint arXiv:2501.15740, 2025. — [arXiv:2501.15740](https://arxiv.org/abs/2501.15740) [find](https://scholar.google.com/scholar?q=David+J+Chalmers%2E+Propositional+interpretability+in+artificial+intelligence%2E+arXiv+preprint+arXiv%3A2501%2E15740%2C+2025%2E+%E2%80%94+%5BarXiv%3A2501%2E15740%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2501%2E15740%29)
37. Lawrence Chan, Adrià Garriga-Alonso, Nicholas Goldwosky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. Causal scrubbing, a method for rigorously testing interpretability hypotheses. AI Alignment Forum, 2022a. [https://www.alignmentforum.org/](https://www.alignmentforum.org/) posts/JvZ… [↗](https://www.alignmentforum.org/)
38. Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers. Advances in Neural Information Processing Systems, 35:18878–18891, 2022b. [find](https://scholar.google.com/scholar?q=Stephanie+Chan%2C+Adam+Santoro%2C+Andrew+Lampinen%2C+Jane+Wang%2C+Aaditya+Singh%2C+Pierre+Richemond%2C+James+McClelland%2C+and+Felix+Hill%2E+Data+distributional+properties+drive+emergent+in%2Dcontext+learning+in+transf)
39. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. [A survey on evaluation of large language models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/). ACM Transactions on Intelligent Systems and Technology, 15(3):1–45, 2024.
40. David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, and Joseph Bloom. A is for absorption: Studying feature splitting and absorption in sparse autoencoders. arXiv preprint arXiv:2409.14507, 2024. — [arXiv:2409.14507](https://arxiv.org/abs/2409.14507) [find](https://scholar.google.com/scholar?q=David+Chanin%2C+James+Wilken%2DSmith%2C+Tom%C3%A1%C5%A1+Dulka%2C+Hardik+Bhatnagar%2C+and+Joseph+Bloom%2E+A+is+for+absorption%3A+Studying+feature+splitting+and+absorption+in+sparse+autoencoders%2E+arXiv+preprint+arXiv%3A2409%2E1450)
41. Angelica Chen, Ravid Schwartz-Ziv, Kyunghyun Cho, Matthew L. Leavitt, and Naomi Saphra. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in mlms. In The Twelfth International Conference on Learning Representations, 2024. [find](https://scholar.google.com/scholar?q=Angelica+Chen%2C+Ravid+Schwartz%2DZiv%2C+Kyunghyun+Cho%2C+Matthew+L%2E+Leavitt%2C+and+Naomi+Saphra%2E+Sudden+drops+in+the+loss%3A+Syntax+acquisition%2C+phase+transitions%2C+and+simplicity+bias+in+mlms%2E+In+The+Twelfth+Int)
42. Valerie Chen, Q Vera Liao, Jennifer Wortman Vaughan, and Gagan Bansal. Understanding the role of human intuition on reliance in human-ai decision-making with explanations, 2023. [find](https://scholar.google.com/scholar?q=Valerie+Chen%2C+Q+Vera+Liao%2C+Jennifer+Wortman+Vaughan%2C+and+Gagan+Bansal%2E+Understanding+the+role+of+human+intuition+on+reliance+in+human%2Dai+decision%2Dmaking+with+explanations%2C+2023%2E)
43. Abhijith Chintam, Rahel Beloch, Willem Zuidema, Michael Hanna, and Oskar van der Wal. [Identifying and adapting transformer-components responsible for gender bias in an English language model](https://md.research.elamur.ai/free/en/papers/chintam-2023-identifying-adapting-transformer-components-responsible/). In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Najoung Kim, Arya McCarthy, and Hosein Mohebbi (eds.), Procee…
44. Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations. In International Conference on Machine Learning, pp. 6243–6267. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Bilal+Chughtai%2C+Lawrence+Chan%2C+and+Neel+Nanda%2E+A+toy+model+of+universality%3A+Reverse+engineering+how+networks+learn+group+operations%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+6243%E2%80%936267%2E+PML)
45. Bilal Chughtai, Alan Cooney, and Neel Nanda. Summing up the facts: Additive mechanisms behind factual recall in llms. arXiv preprint arXiv:2402.07321, 2024. — [arXiv:2402.07321](https://arxiv.org/abs/2402.07321) [find](https://scholar.google.com/scholar?q=Bilal+Chughtai%2C+Alan+Cooney%2C+and+Neel+Nanda%2E+Summing+up+the+facts%3A+Additive+mechanisms+behind+factual+recall+in+llms%2E+arXiv+preprint+arXiv%3A2402%2E07321%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E07321%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
46. Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. [Crawling the internal knowledge-base of language models](https://md.research.elamur.ai/free/en/papers/cohen-2023-crawling-internal-knowledge-base-language/). In Findings of the Association for Computational Linguistics: EACL 2023, pp. 1856– 1869, 2023.
47. Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models. Transactions of the Association for Computational Linguistics, 12: 283–298, 2024. [find](https://scholar.google.com/scholar?q=Roi+Cohen%2C+Eden+Biran%2C+Ori+Yoran%2C+Amir+Globerson%2C+and+Mor+Geva%2E+Evaluating+the+ripple+effects+of+knowledge+editing+in+language+models%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+12)
48. Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 36:16318–16352, 2023. [find](https://scholar.google.com/scholar?q=Arthur+Conmy%2C+Augustine+Mavor%2DParker%2C+Aengus+Lynch%2C+Stefan+Heimersheim%2C+and+Adri%C3%A0+Garriga%2DAlonso%2E+Towards+automated+circuit+discovery+for+mechanistic+interpretability%2E+Advances+in+Neural+Information+P)
49. Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.… — [What you can cram into a single \$&!\#* vector: Probing sentence embeddings for linguistic properties](https://md.research.elamur.ai/free/en/papers/conneau-2018-what-you-can-cram-into/)
50. Alan Cooney and Neel Nanda. Circuitsvis. [https://github.com/TransformerLensOrg/CircuitsVis](https://github.com/TransformerLensOrg/CircuitsVis), 2023. [↗](https://github.com/TransformerLensOrg/CircuitsVis)
51. Róbert Csordás, Christopher Potts, Christopher D Manning, and Atticus Geiger. Recurrent neural networks learn to store and generate sequences using non-linear representations. arXiv preprint arXiv:2408.10920, 2024. [find](https://scholar.google.com/scholar?q=R%C3%B3bert+Csord%C3%A1s%2C+Christopher+Potts%2C+Christopher+D+Manning%2C+and+Atticus+Geiger%2E+Recurrent+neural+networks+learn+to+store+and+generate+sequences+using+non%2Dlinear+representations%2E+arXiv+preprint+arXiv%3A240)
52. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023. [find](https://scholar.google.com/scholar?q=Hoagy+Cunningham%2C+Aidan+Ewart%2C+Logan+Riggs%2C+Robert+Huben%2C+and+Lee+Sharkey%2E+Sparse+autoencoders+find+highly+interpretable+features+in+language+models%2E+arXiv+preprint+arXiv%3A2309%2E08600%2C+2023%2E)
53. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8493–8502, 2022. [find](https://scholar.google.com/scholar?q=Damai+Dai%2C+Li+Dong%2C+Yaru+Hao%2C+Zhifang+Sui%2C+Baobao+Chang%2C+and+Furu+Wei%2E+Knowledge+neurons+in+pretrained+transformers%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Ling)
54. Yunkai Dang, Kaichen Huang, Jiahao Huo, Yibo Yan, Sirui Huang, Dongrui Liu, Mengxi Gao, Jie Zhang, Chen Qian, Kun Wang, et al. Explainable and interpretable multimodal large language models: A comprehensive survey. arXiv preprint arXiv:2412.02104, 2024. — [arXiv:2412.02104](https://arxiv.org/abs/2412.02104) [find](https://scholar.google.com/scholar?q=Yunkai+Dang%2C+Kaichen+Huang%2C+Jiahao+Huo%2C+Yibo+Yan%2C+Sirui+Huang%2C+Dongrui+Liu%2C+Mengxi+Gao%2C+Jie+Zhang%2C+Chen+Qian%2C+Kun+Wang%2C+et+al%2E+Explainable+and+interpretable+multimodal+large+language+models%3A+A+compreh)
55. Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. [Analyzing transformers in embedding space](https://md.research.elamur.ai/free/en/papers/dar-2023-analyzing-transformers-embedding-space/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16124–16170, 2023.
56. Aghyad Deeb and Fabien Roger. Do unlearning methods remove information from language model weights? arXiv preprint arXiv:2410.08827, 2024. — [arXiv:2410.08827](https://arxiv.org/abs/2410.08827) [find](https://scholar.google.com/scholar?q=Aghyad+Deeb+and+Fabien+Roger%2E+Do+unlearning+methods+remove+information+from+language+model+weights%3F+arXiv+preprint+arXiv%3A2410%2E08827%2C+2024%2E+%E2%80%94+%5BarXiv%3A2410%2E08827%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2410%2E08827%29)
57. Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. Jump to conclusions: Short-cutting transformers with linear transformations. arXiv preprint arXiv:2303.09435, 2023. [find](https://scholar.google.com/scholar?q=Alexander+Yom+Din%2C+Taelin+Karidi%2C+Leshem+Choshen%2C+and+Mor+Geva%2E+Jump+to+conclusions%3A+Short%2Dcutting+transformers+with+linear+transformations%2E+arXiv+preprint+arXiv%3A2303%2E09435%2C+2023%2E)
58. Finale Doshi-Velez and Been Kim. [Towards a rigorous science of interpretable machine learning](https://md.research.elamur.ai/free/en/papers/doshi-velez-2017-towards-rigorous-science-interpretable-machine/). arXiv preprint arXiv:1702.08608, 2017.
59. Subhabrata Dutta, Joykirat Singh, Soumen Chakrabarti, and Tanmoy Chakraborty. How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning. arXiv preprint arXiv:2402.18312, 2024. [find](https://scholar.google.com/scholar?q=Subhabrata+Dutta%2C+Joykirat+Singh%2C+Soumen+Chakrabarti%2C+and+Tanmoy+Chakraborty%2E+How+to+think+step%2Dby%2Dstep%3A+A+mechanistic+understanding+of+chain%2Dof%2Dthought+reasoning%2E+arXiv+preprint+arXiv%3A2402%2E18312%2C+202)
60. Benjamin L Edelman, Ezra Edelman, Surbhi Goel, Eran Malach, and Nikolaos Tsilivis. The evolution of statistical induction heads: In-context learning markov chains. arXiv preprint arXiv:2402.11004, 2024. — [arXiv:2402.11004](https://arxiv.org/abs/2402.11004) [find](https://scholar.google.com/scholar?q=Benjamin+L+Edelman%2C+Ezra+Edelman%2C+Surbhi+Goel%2C+Eran+Malach%2C+and+Nikolaos+Tsilivis%2E+The+evolution+of+statistical+induction+heads%3A+In%2Dcontext+learning+markov+chains%2E+arXiv+preprint+arXiv%3A2402%2E11004%2C+202)
61. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Ja… [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Neel+Nanda%2C+Catherine+Olsson%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Zac+Hatfield%2DDodds%2C+Da)
62. Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer ElShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Andy Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatfield-Dodds, Jackson Kern… [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Tristan+Hume%2C+Catherine+Olsson%2C+Neel+Nanda%2C+Tom+Henighan%2C+Scott+Johnston%2C+Sheer+ElShowk%2C+Nicholas+Joseph%2C+Nova+DasSarma%2C+Ben+Mann%2C+Danny+Hernandez%2C+Amanda+Askell%2C+Kamal+Ndousse%2C+Andy+Jo)
63. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circu… [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Tristan+Hume%2C+Catherine+Olsson%2C+Nicholas+Schiefer%2C+Tom+Henighan%2C+Shauna+Kravec%2C+Zac+Hatfield%2DDodds%2C+Robert+Lasenby%2C+Dawn+Drain%2C+Carol+Chen%2C+Roger+Grosse%2C+Sam+McCandlish%2C+Jared+Kaplan%2C+D)
64. Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are linear. arXiv preprint arXiv:2405.14860, 2024. — [arXiv:2405.14860](https://arxiv.org/abs/2405.14860) [find](https://scholar.google.com/scholar?q=Joshua+Engels%2C+Eric+J+Michaud%2C+Isaac+Liao%2C+Wes+Gurnee%2C+and+Max+Tegmark%2E+Not+all+language+model+features+are+linear%2E+arXiv+preprint+arXiv%3A2405%2E14860%2C+2024%2E+%E2%80%94+%5BarXiv%3A2405%2E14860%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F24)
65. N Benjamin Erichson, Zhewei Yao, and Michael W Mahoney. Jumprelu: A retrofit defense strategy for adversarial attacks. arXiv preprint arXiv:1904.03750, 2019. — [arXiv:1904.03750](https://arxiv.org/abs/1904.03750) [find](https://scholar.google.com/scholar?q=N+Benjamin+Erichson%2C+Zhewei+Yao%2C+and+Michael+W+Mahoney%2E+Jumprelu%3A+A+retrofit+defense+strategy+for+adversarial+attacks%2E+arXiv+preprint+arXiv%3A1904%2E03750%2C+2019%2E+%E2%80%94+%5BarXiv%3A1904%2E03750%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
66. Jiahai Feng and Jacob Steinhardt. How do language models bind entities in context? arXiv:2310.17191, 2023. arXiv preprint Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-jussà. [A primer on the inner workings of transformer-based language models](https://md.research.elamur.ai/free/en/papers/ferrando-2024-primer-inner-workings-transformer-based/). arXiv preprint arXiv:2405.00208, 2…
67. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+Shawn+Presser%2C+and+Connor+Leahy%2E+The+Pile%3A+An+800gb+dataset+of+)
68. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Tom+Dupr%C3%A9+la+Tour%2C+Henk+Tillman%2C+Gabriel+Goh%2C+Rajan+Troll%2C+Alec+Radford%2C+Ilya+Sutskever%2C+Jan+Leike%2C+and+Jeffrey+Wu%2E+Scaling+and+evaluating+sparse+autoencoders%2E+arXiv+preprint+arXiv%3A2406%2E04093)
69. Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34:9574–9586, 2021. [find](https://scholar.google.com/scholar?q=Atticus+Geiger%2C+Hanson+Lu%2C+Thomas+Icard%2C+and+Christopher+Potts%2E+Causal+abstractions+of+neural+networks%2E+Advances+in+Neural+Information+Processing+Systems%2C+34%3A9574%E2%80%939586%2C+2021%2E)
70. Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. [Finding alignments between interpretable causal variables and distributed neural representations](https://md.research.elamur.ai/free/en/papers/geiger-2023-finding-alignments-between-interpretable-causal/). In Causal Learning and Reasoning, pp. 160–187. PMLR, 2024.
71. Atticus Geiger, Duligur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, et al. Causal abstraction: A theoretical foundation for mechanistic interpretability. Journal of Machine Learning Research, 26(83):1–64, 2025. [find](https://scholar.google.com/scholar?q=Atticus+Geiger%2C+Duligur+Ibeling%2C+Amir+Zur%2C+Maheep+Chaudhary%2C+Sonakshi+Chauhan%2C+Jing+Huang%2C+Aryaman+Arora%2C+Zhengxuan+Wu%2C+Noah+Goodman%2C+Christopher+Potts%2C+et+al%2E+Causal+abstraction%3A+A+theoretical+founda)
72. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484–5495, 2021. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Roei+Schuster%2C+Jonathan+Berant%2C+and+Omer+Levy%2E+Transformer+feed%2Dforward+layers+are+key%2Dvalue+memories%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processi)
73. Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. [Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space](https://md.research.elamur.ai/free/en/papers/geva-2022-transformer-feed-forward-layers-build/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 30–45, 2022.
74. Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12216–12235, 2023. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Jasmijn+Bastings%2C+Katja+Filippova%2C+and+Amir+Globerson%2E+Dissecting+recall+of+factual+associations+in+auto%2Dregressive+language+models%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Method)
75. Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. [Patchscopes: a unifying framework for inspecting hidden representations of language models](https://md.research.elamur.ai/free/en/papers/ghandeharioun-2024-patchscopes-unifying-framework-inspecting-hidden/). In Proceedings of the 41st International Conference on Machine Learning, pp. 15466–15490, 2024.
76. Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. [Localizing model behavior with path patching](https://md.research.elamur.ai/free/en/papers/goldowsky-dill-2023-localizing-model-behavior-path-patching/). arXiv preprint arXiv:2304.05969, 2023.
77. Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. [Successor heads: Recurring, interpretable attention heads in the wild](https://md.research.elamur.ai/free/en/papers/gould-2023-successor-heads-recurring-interpretable-attention/). In The Twelfth International Conference on Learning Representations, 2023.
78. Phillip Guo, Aaquib Syed, Abhay Sheshadri, Aidan Ewart, and Gintare Karolina Dziugaite. Mechanistic unlearning: Robust knowledge unlearning and editing via mechanistic localization. arXiv preprint arXiv:2410.12949, 2024. — [arXiv:2410.12949](https://arxiv.org/abs/2410.12949) [find](https://scholar.google.com/scholar?q=Phillip+Guo%2C+Aaquib+Syed%2C+Abhay+Sheshadri%2C+Aidan+Ewart%2C+and+Gintare+Karolina+Dziugaite%2E+Mechanistic+unlearning%3A+Robust+knowledge+unlearning+and+editing+via+mechanistic+localization%2E+arXiv+preprint+arX)
79. Anshita Gupta, Debanjan Mondal, Akshay Krishna Sheshadri, Wenlong Zhao, Xiang Lorraine Li, Sarah Wiegreffe, and Niket Tandon. Editing common sense in transformers. arXiv preprint arXiv:2305.14956, 2023. [find](https://scholar.google.com/scholar?q=Anshita+Gupta%2C+Debanjan+Mondal%2C+Akshay+Krishna+Sheshadri%2C+Wenlong+Zhao%2C+Xiang+Lorraine+Li%2C+Sarah+Wiegreffe%2C+and+Niket+Tandon%2E+Editing+common+sense+in+transformers%2E+arXiv+preprint+arXiv%3A2305%2E14956%2C+202)
80. Rohan Gupta, Iván Arcuschin, Thomas Kwa, and Adrià Garriga-Alonso. Interpbench: Semi-synthetic transformers for evaluating mechanistic interpretability techniques. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL [https://openreview.net/…](https://openreview.net/…) [↗](https://openreview.net/…)
81. Wes Gurnee. Sae reconstruction errors are (empirically) pathological. In AI Alignment Forum, pp. 16, 2024. [find](https://scholar.google.com/scholar?q=Wes+Gurnee%2E+Sae+reconstruction+errors+are+%28empirically%29+pathological%2E+In+AI+Alignment+Forum%2C+pp%2E+16%2C+2024%2E)
82. Wes Gurnee and Max Tegmark. Language models represent space and time. arXiv preprint arXiv:2310.02207, 2023. [find](https://scholar.google.com/scholar?q=Wes+Gurnee+and+Max+Tegmark%2E+Language+models+represent+space+and+time%2E+arXiv+preprint+arXiv%3A2310%2E02207%2C+2023%2E)
83. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. [Finding neurons in a haystack: Case studies with sparse probing](https://md.research.elamur.ai/free/en/papers/gurnee-2023-finding-neurons-haystack-case-studies/). Transactions on Machine Learning Research, 2023.
84. Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. Universal neurons in gpt2 language models. arXiv preprint arXiv:2401.12181, 2024. — [arXiv:2401.12181](https://arxiv.org/abs/2401.12181) [find](https://scholar.google.com/scholar?q=Wes+Gurnee%2C+Theo+Horsley%2C+Zifan+Carl+Guo%2C+Tara+Rezaei+Kheirkhah%2C+Qinyi+Sun%2C+Will+Hathaway%2C+Neel+Nanda%2C+and+Dimitris+Bertsimas%2E+Universal+neurons+in+gpt2+language+models%2E+arXiv+preprint+arXiv%3A2401%2E1218)
85. Michael Hanna, Ollie Liu, and Alexandre Variengien. [How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://md.research.elamur.ai/free/en/papers/hanna-2023-gpt-2-compute-greater-than/). Advances in Neural Information Processing Systems, 36, 2024a.
86. Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. [Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms](https://md.research.elamur.ai/free/en/papers/hanna-2024-have-faith-faithfulness-going-circuit/). arXiv preprint arXiv:2403.17806, 2024b.
87. Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, et al. [Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders](https://md.research.elamur.ai/free/en/papers/he-2024-llama-scope-extracting-millions-features/). arXiv preprint arXiv:2410.20526, 2024.
88. Stefan Heimersheim. You can remove gpt2’s layernorm by fine-tuning. arXiv preprint arXiv:2409.13710, 2024. — [arXiv:2409.13710](https://arxiv.org/abs/2409.13710) [find](https://scholar.google.com/scholar?q=Stefan+Heimersheim%2E+You+can+remove+gpt2%E2%80%99s+layernorm+by+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A2409%2E13710%2C+2024%2E+%E2%80%94+%5BarXiv%3A2409%2E13710%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2409%2E13710%29)
89. Stefan Heimersheim and Jett Janiak. A circuit for python docstrings in a 4-layer attention-only transformer. [https://www.alignmentforum.org/posts/u6KXXmKFbXfWzoAXn/](https://www.alignmentforum.org/posts/u6KXXmKFbXfWzoAXn/) acircuit-for-python-docstrings-in-a-4-layer-attention-only, 2023. [↗](https://www.alignmentforum.org/posts/u6KXXmKFbXfWzoAXn/)
90. Stefan Heimersheim and Neel Nanda. arXiv:2404.15255, 2024. — [arXiv:2404.15255](https://arxiv.org/abs/2404.15255) [find](https://scholar.google.com/scholar?q=Stefan+Heimersheim+and+Neel+Nanda%2E+arXiv%3A2404%2E15255%2C+2024%2E+%E2%80%94+%5BarXiv%3A2404%2E15255%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2404%2E15255%29)
91. How to use and interpret activation patching. arXiv preprint Roee Hendel, Mor Geva, and Amir Globerson. [In-context learning creates task vectors](https://md.research.elamur.ai/free/en/papers/hendel-2023-context-learning-creates-task-vectors/). arXiv preprint arXiv:2310.15916, 2023.
92. Tom Henighan, Shan Carter, Tristan Hume, Nelson Elhage, Robert Lasenby, Stanislav Fort, Nicholas Schiefer, and Christopher Olah. Superposition, memorization, and double descent. Transformer Circuits Thread, 6:24, 2023. [find](https://scholar.google.com/scholar?q=Tom+Henighan%2C+Shan+Carter%2C+Tristan+Hume%2C+Nelson+Elhage%2C+Robert+Lasenby%2C+Stanislav+Fort%2C+Nicholas+Schiefer%2C+and+Christopher+Olah%2E+Superposition%2C+memorization%2C+and+double+descent%2E+Transformer+Circuits+T)
93. Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. [Linearity of relation decoding in transformer language models](https://md.research.elamur.ai/free/en/papers/hernandez-2023-linearity-relation-decoding-transformer-language/). arXiv preprint arXiv:2308.09124, 2023.
94. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). arXiv preprint arXiv:2203.15556, 2022.
95. Yihuai Hong, Lei Yu, Haiqin Yang, Shauli Ravfogel, and Mor Geva. Intrinsic evaluation of unlearning using parametric knowledge traces. arXiv preprint arXiv:2406.11614, 2024. — [arXiv:2406.11614](https://arxiv.org/abs/2406.11614) [find](https://scholar.google.com/scholar?q=Yihuai+Hong%2C+Lei+Yu%2C+Haiqin+Yang%2C+Shauli+Ravfogel%2C+and+Mor+Geva%2E+Intrinsic+evaluation+of+unlearning+using+parametric+knowledge+traces%2E+arXiv+preprint+arXiv%3A2406%2E11614%2C+2024%2E+%E2%80%94+%5BarXiv%3A2406%2E11614%5D%28https)
96. Jie Huang and Kevin Chen-Chuan Chang. [Towards reasoning in large language models: A survey](https://md.research.elamur.ai/free/en/papers/huang-2023-reasoning-large-language-models-survey/). arXiv preprint arXiv:2212.10403, 2022.
97. Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. Ravel: Evaluating interpretability methods on disentangling language model representations. arXiv preprint arXiv:2402.17700, 2024a. [find](https://scholar.google.com/scholar?q=Jing+Huang%2C+Zhengxuan+Wu%2C+Christopher+Potts%2C+Mor+Geva%2C+and+Atticus+Geiger%2E+Ravel%3A+Evaluating+interpretability+methods+on+disentangling+language+model+representations%2E+arXiv+preprint+arXiv%3A2402%2E17700%2C+)
98. Yufei Huang, Shengding Hu, Xu Han, Zhiyuan Liu, and Maosong Sun. Unified view of grokking, double descent and emergent abilities: A perspective from circuits competition. arXiv preprint arXiv:2402.15175, 2024b. — [arXiv:2402.15175](https://arxiv.org/abs/2402.15175) [find](https://scholar.google.com/scholar?q=Yufei+Huang%2C+Shengding+Hu%2C+Xu+Han%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+Unified+view+of+grokking%2C+double+descent+and+emergent+abilities%3A+A+perspective+from+circuits+competition%2E+arXiv+preprint+arXiv%3A2402%2E151)
99. Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P Dick, Hidenori Tanaka, Edward Grefenstette, Tim Rocktäschel, and David Scott Krueger. Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks. arXiv preprint arXiv:2311.12786, 2023. — [arXiv:2311.12786](https://arxiv.org/abs/2311.12786) [find](https://scholar.google.com/scholar?q=Samyak+Jain%2C+Robert+Kirk%2C+Ekdeep+Singh+Lubana%2C+Robert+P+Dick%2C+Hidenori+Tanaka%2C+Edward+Grefenstette%2C+Tim+Rockt%C3%A4schel%2C+and+David+Scott+Krueger%2E+Mechanistically+analyzing+the+effects+of+fine%2Dtuning+on+pr)
100. Samyak Jain, Ekdeep Singh Lubana, Kemal Oksuz, Tom Joy, Philip HS Torr, Amartya Sanyal, and Puneet K Dokania. What makes and breaks safety fine-tuning? a mechanistic study. arXiv preprint arXiv:2407.10264, 2024. — [arXiv:2407.10264](https://arxiv.org/abs/2407.10264) [find](https://scholar.google.com/scholar?q=Samyak+Jain%2C+Ekdeep+Singh+Lubana%2C+Kemal+Oksuz%2C+Tom+Joy%2C+Philip+HS+Torr%2C+Amartya+Sanyal%2C+and+Puneet+K+Dokania%2E+What+makes+and+breaks+safety+fine%2Dtuning%3F+a+mechanistic+study%2E+arXiv+preprint+arXiv%3A2407%2E1)
101. Stanisław Jastrzębski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. arXiv preprint arXiv:1710.04773, 2017. — [arXiv:1710.04773](https://arxiv.org/abs/1710.04773) [find](https://scholar.google.com/scholar?q=Stanis%C5%82aw+Jastrz%C4%99bski%2C+Devansh+Arpit%2C+Nicolas+Ballas%2C+Vikas+Verma%2C+Tong+Che%2C+and+Yoshua+Bengio%2E+Residual+connections+encourage+iterative+inference%2E+arXiv+preprint+arXiv%3A1710%2E04773%2C+2017%2E+%E2%80%94+%5BarXiv%3A1710)
102. Gangwei Jiang, Caigao Jiang, Zhaoyi Li, Siqiao Xue, Jun Zhou, Linqi Song, Defu Lian, and Ying Wei. Interpretable catastrophic forgetting of large language model fine-tuning via instruction vector. arXiv preprint arXiv:2406.12227, 2024. — [arXiv:2406.12227](https://arxiv.org/abs/2406.12227) [find](https://scholar.google.com/scholar?q=Gangwei+Jiang%2C+Caigao+Jiang%2C+Zhaoyi+Li%2C+Siqiao+Xue%2C+Jun+Zhou%2C+Linqi+Song%2C+Defu+Lian%2C+and+Ying+Wei%2E+Interpretable+catastrophic+forgetting+of+large+language+model+fine%2Dtuning+via+instruction+vector%2E+arX)
103. Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Isaac Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Callum Stuart McDougall, Kola Ayonrinde, Demian Till, Matthew Wearden, Arthur Conmy, Samuel Marks, and Neel Nanda. SAEBench: A comprehensive benchmark for sparse autoencoders in language… [find](https://scholar.google.com/scholar?q=Adam+Karvonen%2C+Can+Rager%2C+Johnny+Lin%2C+Curt+Tigges%2C+Joseph+Isaac+Bloom%2C+David+Chanin%2C+Yeu%2DTong+Lau%2C+Eoin+Farrell%2C+Callum+Stuart+McDougall%2C+Kola+Ayonrinde%2C+Demian+Till%2C+Matthew+Wearden%2C+Arthur+Conmy%2C+Sa)
104. Shahar Katz, Yonatan Belinkov, Mor Geva, and Lior Wolf. Backward lens: Projecting language model gradients into the vocabulary space. arXiv preprint arXiv:2402.12865, 2024. [find](https://scholar.google.com/scholar?q=Shahar+Katz%2C+Yonatan+Belinkov%2C+Mor+Geva%2C+and+Lior+Wolf%2E+Backward+lens%3A+Projecting+language+model+gradients+into+the+vocabulary+space%2E+arXiv+preprint+arXiv%3A2402%2E12865%2C+2024%2E)
105. Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, and Neel Nanda. Interpreting attention layer outputs with sparse autoencoders. arXiv preprint arXiv:2406.17759, 2024. — [arXiv:2406.17759](https://arxiv.org/abs/2406.17759) [find](https://scholar.google.com/scholar?q=Connor+Kissane%2C+Robert+Krzyzanowski%2C+Joseph+Isaac+Bloom%2C+Arthur+Conmy%2C+and+Neel+Nanda%2E+Interpreting+attention+layer+outputs+with+sparse+autoencoders%2E+arXiv+preprint+arXiv%3A2406%2E17759%2C+2024%2E+%E2%80%94+%5BarXiv%3A24)
106. Ouail Kitouni, Niklas Nolte, Víctor Samuel Pérez-Díaz, Sokratis Trifinopoulos, and Mike Williams. From neurons to neutrons: A case study in interpretability. arXiv preprint arXiv:2405.17425, 2024. — [arXiv:2405.17425](https://arxiv.org/abs/2405.17425) [find](https://scholar.google.com/scholar?q=Ouail+Kitouni%2C+Niklas+Nolte%2C+V%C3%ADctor+Samuel+P%C3%A9rez%2DD%C3%ADaz%2C+Sokratis+Trifinopoulos%2C+and+Mike+Williams%2E+From+neurons+to+neutrons%3A+A+case+study+in+interpretability%2E+arXiv+preprint+arXiv%3A2405%2E17425%2C+2024%2E+%E2%80%94+%5B)
107. Robert Krzyzanowski, Connor Kissane, Arthur Conmy, and Neel Nanda. We inspected every head in gpt-2 small using saes so you don’t have to. In Alignment Forum, 2024. [find](https://scholar.google.com/scholar?q=Robert+Krzyzanowski%2C+Connor+Kissane%2C+Arthur+Conmy%2C+and+Neel+Nanda%2E+We+inspected+every+head+in+gpt%2D2+small+using+saes+so+you+don%E2%80%99t+have+to%2E+In+Alignment+Forum%2C+2024%2E)
108. Michael Lan, Philip Torr, Austin Meek, Ashkan Khakzar, David Krueger, and Fazl Barez. Sparse autoencoders reveal universal feature spaces across large language models. arXiv preprint arXiv:2410.06981, 2024. — [arXiv:2410.06981](https://arxiv.org/abs/2410.06981) [find](https://scholar.google.com/scholar?q=Michael+Lan%2C+Philip+Torr%2C+Austin+Meek%2C+Ashkan+Khakzar%2C+David+Krueger%2C+and+Fazl+Barez%2E+Sparse+autoencoders+reveal+universal+feature+spaces+across+large+language+models%2E+arXiv+preprint+arXiv%3A2410%2E06981%2C)
109. Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mihalcea. [A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity](https://md.research.elamur.ai/free/en/papers/lee-2024-mechanistic-understanding-alignment-algorithms-case/). In International Conference on Machine Learning, pp. 26361–26378. PMLR, 2024.
110. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. [Inference-time intervention: Eliciting truthful answers from a language model](https://md.research.elamur.ai/free/en/papers/li-2023-inference-time-intervention-eliciting-truthful/), 2024.
111. Tom Lieberum, Matthew Rahtz, János Kramár, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. [Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla](https://md.research.elamur.ai/free/en/papers/lieberum-2023-circuit-analysis-interpretability-scale-evidence/). arXiv preprint arXiv:2307.09458, 2023.
112. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. [Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2](https://md.research.elamur.ai/free/en/papers/lieberum-2024-gemma-scope-open-sparse-autoencoders/). arXiv preprint arXiv:2408.05147, 2024.
113. Johnny Lin. Neuronpedia: Interactive reference and tooling for analyzing neural networks, 2023. URL [https://www.neuronpedia.org](https://www.neuronpedia.org). Software available from neuronpedia.org. [↗](https://www.neuronpedia.org)
114. David Lindner, János Kramár, Sebastian Farquhar, Matthew Rahtz, Tom McGrath, and Vladimir Mikulik. Tracr: Compiled transformers as a laboratory for interpretability. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=David+Lindner%2C+J%C3%A1nos+Kram%C3%A1r%2C+Sebastian+Farquhar%2C+Matthew+Rahtz%2C+Tom+McGrath%2C+and+Vladimir+Mikulik%2E+Tracr%3A+Compiled+transformers+as+a+laboratory+for+interpretability%2E+Advances+in+Neural+Information+Pro)
115. Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. Rethinking machine unlearning for large language models. Nature Machine Intelligence, pp. 1–14, 2025. [find](https://scholar.google.com/scholar?q=Sijia+Liu%2C+Yuanshun+Yao%2C+Jinghan+Jia%2C+Stephen+Casper%2C+Nathalie+Baracaldo%2C+Peter+Hase%2C+Yuguang+Yao%2C+Chris+Yuhao+Liu%2C+Xiaojun+Xu%2C+Hang+Li%2C+et+al%2E+Rethinking+machine+unlearning+for+large+language+models%2E)
116. Scott Lundberg. [A unified approach to interpreting model predictions](https://md.research.elamur.ai/free/en/papers/lundberg-2017-unified-approach-interpreting-model-predictions/), 2017.
117. Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402.16835, 2024. — [arXiv:2402.16835](https://arxiv.org/abs/2402.16835) [find](https://scholar.google.com/scholar?q=Aengus+Lynch%2C+Phillip+Guo%2C+Aidan+Ewart%2C+Stephen+Casper%2C+and+Dylan+Hadfield%2DMenell%2E+Eight+methods+to+evaluate+robust+unlearning+in+llms%2E+arXiv+preprint+arXiv%3A2402%2E16835%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E16835%5D%28http)
118. Monte MacDiarmid, Timothy Maxwell, Nicholas Schiefer, Jesse Mu, Jared Kaplan, David Duvenaud, Sam Bowman, Alex Tamkin, Ethan Perez, Mrinank Sharma, Carson Denison, and Evan Hubinger. Simple probes can catch sleeper agents, 2024. URL [https://www.anthropic.com/news/](https://www.anthropic.com/news/) probes-catch-sleeper-agents. [↗](https://www.anthropic.com/news/)
119. Aleksandar Makelov. Sparse autoencoders match supervised features for model steering on the ioi task. In ICML 2024 Workshop on Mechanistic Interpretability, 2024. [find](https://scholar.google.com/scholar?q=Aleksandar+Makelov%2E+Sparse+autoencoders+match+supervised+features+for+model+steering+on+the+ioi+task%2E+In+ICML+2024+Workshop+on+Mechanistic+Interpretability%2C+2024%2E)
120. Aleksandar Makelov, George Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control. arXiv preprint arXiv:2405.08366, 2024. — [arXiv:2405.08366](https://arxiv.org/abs/2405.08366) [find](https://scholar.google.com/scholar?q=Aleksandar+Makelov%2C+George+Lange%2C+and+Neel+Nanda%2E+Towards+principled+evaluations+of+sparse+autoencoders+for+interpretability+and+control%2E+arXiv+preprint+arXiv%3A2405%2E08366%2C+2024%2E+%E2%80%94+%5BarXiv%3A2405%2E08366%5D%28ht)
121. Luke Marks, Alisdair Paren, David Krueger, and Fazl Barez. Enhancing neural network interpretability with feature-aligned sparse autoencoders. arXiv preprint arXiv:2411.01220, 2024a. — [arXiv:2411.01220](https://arxiv.org/abs/2411.01220) [find](https://scholar.google.com/scholar?q=Luke+Marks%2C+Alisdair+Paren%2C+David+Krueger%2C+and+Fazl+Barez%2E+Enhancing+neural+network+interpretability+with+feature%2Daligned+sparse+autoencoders%2E+arXiv+preprint+arXiv%3A2411%2E01220%2C+2024a%2E+%E2%80%94+%5BarXiv%3A2411%2E012)
122. Samuel Marks and Max Tegmark. [The geometry of truth: Emergent linear structure in large language model representations of true/false datasets](https://md.research.elamur.ai/free/en/papers/marks-2023-geometry-truth-emergent-linear-structure/). arXiv preprint arXiv:2310.06824, 2023.
123. Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403.19647, 2024b. [find](https://scholar.google.com/scholar?q=Samuel+Marks%2C+Can+Rager%2C+Eric+J+Michaud%2C+Yonatan+Belinkov%2C+David+Bau%2C+and+Aaron+Mueller%2E+Sparse+feature+circuits%3A+Discovering+and+editing+interpretable+causal+graphs+in+language+models%2E+arXiv+preprint)
124. Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. [Copy suppression: Comprehensively understanding an attention head](https://md.research.elamur.ai/free/en/papers/mcdougall-2023-copy-suppression-comprehensively-understanding-attention/). arXiv preprint arXiv:2310.04625, 2023.
125. Tianyi Men, Pengfei Cao, Zhuoran Jin, Yubo Chen, Kang Liu, and Jun Zhao. Unlocking the future: Exploring look-ahead planning mechanistic interpretability in large language models. arXiv preprint arXiv:2406.16033, 2024. [find](https://scholar.google.com/scholar?q=Tianyi+Men%2C+Pengfei+Cao%2C+Zhuoran+Jin%2C+Yubo+Chen%2C+Kang+Liu%2C+and+Jun+Zhao%2E+Unlocking+the+future%3A+Exploring+look%2Dahead+planning+mechanistic+interpretability+in+large+language+models%2E+arXiv+preprint+arXiv)
126. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372, 2022. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+Andonian%2C+and+Yonatan+Belinkov%2E+Locating+and+editing+factual+associations+in+gpt%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A17359%E2%80%9317372%2C+2022%2E)
127. Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+Arnab+Sen+Sharma%2C+Alex+Andonian%2C+Yonatan+Belinkov%2C+and+David+Bau%2E+Mass%2Dediting+memory+in+a+transformer%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2C+2023%2E)
128. Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Circuit component reuse across tasks in transformer language models. In The Twelfth International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=Jack+Merullo%2C+Carsten+Eickhoff%2C+and+Ellie+Pavlick%2E+Circuit+component+reuse+across+tasks+in+transformer+language+models%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+2023%2E)
129. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). Advances in neural information processing systems, 26, 2013.
130. Aaron Mueller, Atticus Geiger, Sarah Wiegreffe, Dana Arad, Iván Arcuschin, Adam Belfki, Yik Siu Chan, Jaden Fried Fiotto-Kaufman, Tal Haklay, Michael Hanna, Jing Huang, Rohan Gupta, Yaniv Nikankin, Hadas Orgad, Nikhil Prakash, Anja Reusch, Aruna Sankaranarayanan, Shun Shao, Alessandro Stolfo, Martin… [find](https://scholar.google.com/scholar?q=Aaron+Mueller%2C+Atticus+Geiger%2C+Sarah+Wiegreffe%2C+Dana+Arad%2C+Iv%C3%A1n+Arcuschin%2C+Adam+Belfki%2C+Yik+Siu+Chan%2C+Jaden+Fried+Fiotto%2DKaufman%2C+Tal+Haklay%2C+Michael+Hanna%2C+Jing+Huang%2C+Rohan+Gupta%2C+Yaniv+Nikankin%2C+Ha)
131. Aashiq Muhamed, Mona Diab, and Virginia Smith. Decoding dark matter: Specialized sparse autoencoders for interpreting rare concepts in foundation models. arXiv preprint arXiv:2411.00743, 2024. [find](https://scholar.google.com/scholar?q=Aashiq+Muhamed%2C+Mona+Diab%2C+and+Virginia+Smith%2E+Decoding+dark+matter%3A+Specialized+sparse+autoencoders+for+interpreting+rare+concepts+in+foundation+models%2E+arXiv+preprint+arXiv%3A2411%2E00743%2C+2024%2E)
132. Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003, 2021. [find](https://scholar.google.com/scholar?q=Preetum+Nakkiran%2C+Gal+Kaplun%2C+Yamini+Bansal%2C+Tristan+Yang%2C+Boaz+Barak%2C+and+Ilya+Sutskever%2E+Deep+double+descent%3A+Where+bigger+models+and+more+data+hurt%2E+Journal+of+Statistical+Mechanics%3A+Theory+and+Exp)
133. Neel Nanda, Chris Olah, Catherine Olsson, Nelson Elhage, and Hume Tristan. Attribution patching: Activation patching at industrial scale, 2023. [https://www.neelnanda.io/mechanistic-interpretability/](https://www.neelnanda.io/mechanistic-interpretability/) attribution-patching, 2022. [↗](https://www.neelnanda.io/mechanistic-interpretability/)
134. Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. [Progress measures for grokking via mechanistic interpretability](https://md.research.elamur.ai/free/en/papers/nanda-2023-progress-measures-grokking-mechanistic-interpretability/). arXiv preprint arXiv:2301.05217, 2023a.
135. Neel Nanda, Andrew Lee, and Martin Wattenberg. [Emergent linear representations in world models of self-supervised sequence models](https://md.research.elamur.ai/free/en/papers/nanda-2023-emergent-linear-representations-world-models/). arXiv preprint arXiv:2309.00941, 2023b.
136. Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. [Arithmetic without algorithms: Language models solve math with a bag of heuristics](https://md.research.elamur.ai/free/en/papers/nikankin-2024-arithmetic-algorithms-language-models-solve/). arXiv preprint arXiv:2410.21272, 2024. nostalgebraist. Interpreting gpt: the logit lens.
137. AI Alignment Forum, 2020. [https://www.lesswrong.com/](https://www.lesswrong.com/) posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens. [↗](https://www.lesswrong.com/)
138. Chris Olah and Adam Jermyn. What would be the most safety-relevant features in language models? (circuits updates - july 2023). [find](https://scholar.google.com/scholar?q=Chris+Olah+and+Adam+Jermyn%2E+What+would+be+the+most+safety%2Drelevant+features+in+language+models%3F+%28circuits+updates+%2D+july+2023%29%2E)
139. Transformer Circuits Thread, 2023. [find](https://scholar.google.com/scholar?q=Transformer+Circuits+Thread%2C+2023%2E)
140. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. [https://distill.pub/2020/circuits/zoom-in](https://distill.pub/2020/circuits/zoom-in). [↗](https://distill.pub/2020/circuits/zoom-in)
141. Christopher Olah. What is a linear representation? what is a multidimensional feature?, 2023. URL [https://transformer-circuits.pub/2024/july-update/\#linear-representations](https://transformer-circuits.pub/2024/july-update/\#linear-representations). [↗](https://transformer-circuits.pub/2024/july-update/\#linear-representations)
142. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amode… — [In-context Learning and Induction Heads](https://md.research.elamur.ai/free/en/papers/olsson-2022-context-learning-induction-heads/)
143. Koyena Pal, Jiuding Sun, Andrew Yuan, Byron C Wallace, and David Bau. [Future lens: Anticipating subsequent tokens from a single hidden state](https://md.research.elamur.ai/free/en/papers/pal-2023-future-lens-anticipating-subsequent-tokens/). arXiv preprint arXiv:2311.04897, 2023.
144. Judea Pearl. Causality. Cambridge university press, 2009. [find](https://scholar.google.com/scholar?q=Judea+Pearl%2E+Causality%2E+Cambridge+university+press%2C+2009%2E)
145. Jeffrey Pennington, Richard Socher, and Christopher D Manning. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532–1543, 2014.
146. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. [Language models as knowledge bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langua…
147. Nicholas Pochinkov and Nandi Schoots. Dissecting language models: Machine unlearning via selective pruning. arXiv preprint arXiv:2403.01267, 2024. — [arXiv:2403.01267](https://arxiv.org/abs/2403.01267) [find](https://scholar.google.com/scholar?q=Nicholas+Pochinkov+and+Nandi+Schoots%2E+Dissecting+language+models%3A+Machine+unlearning+via+selective+pruning%2E+arXiv+preprint+arXiv%3A2403%2E01267%2C+2024%2E+%E2%80%94+%5BarXiv%3A2403%2E01267%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2403%2E01267)
148. Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. [Grokking: Generalization beyond overfitting on small algorithmic datasets](https://md.research.elamur.ai/free/en/papers/power-2022-grokking-generalization-overfitting-small-algorithmic/). arXiv preprint arXiv:2201.02177, 2022.
149. Philip Quirke, Clement Neo, and Fazl Barez. Increasing trust in language models through the reuse of verified circuits. arXiv preprint arXiv:2402.02619, 2024. — [arXiv:2402.02619](https://arxiv.org/abs/2402.02619) [find](https://scholar.google.com/scholar?q=Philip+Quirke%2C+Clement+Neo%2C+and+Fazl+Barez%2E+Increasing+trust+in+language+models+through+the+reuse+of+verified+circuits%2E+arXiv+preprint+arXiv%3A2402%2E02619%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E02619%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
150. Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. [Learning to generate reviews and discovering sentiment](https://md.research.elamur.ai/free/en/papers/radford-2017-learning-generate-reviews-discovering-sentiment/). arXiv preprint arXiv:1704.01444, 2017.
151. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2C+2019%2E)
152. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). Advances in Neural Information Processing Systems, 36:53728–53741, 2023.
153. Daking Rai and Ziyu Yao. An investigation of neuron activation as a unified lens to explain chain-of-thought eliciting arithmetic reasoning of llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7174–7193, 2024. [find](https://scholar.google.com/scholar?q=Daking+Rai+and+Ziyu+Yao%2E+An+investigation+of+neuron+activation+as+a+unified+lens+to+explain+chain%2Dof%2Dthought+eliciting+arithmetic+reasoning+of+llms%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+As)
154. Daking Rai, Rydia R Weiland, Kayla Margaret Gabriella Herrera, Tyler H Shaw, and Ziyu Yao. Understanding the effect of algorithm transparency of model explanations in text-to-sql semantic parsing. arXiv preprint arXiv:2410.16283, 2024. — [arXiv:2410.16283](https://arxiv.org/abs/2410.16283) [find](https://scholar.google.com/scholar?q=Daking+Rai%2C+Rydia+R+Weiland%2C+Kayla+Margaret+Gabriella+Herrera%2C+Tyler+H+Shaw%2C+and+Ziyu+Yao%2E+Understanding+the+effect+of+algorithm+transparency+of+model+explanations+in+text%2Dto%2Dsql+semantic+parsing%2E+arX)
155. Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. [Improving dictionary learning with gated sparse autoencoders](https://md.research.elamur.ai/free/en/papers/rajamanoharan-2024-improving-dictionary-learning-gated-sparse/). arXiv preprint arXiv:2404.16014, 2024a.
156. Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407.14435, 2024b. — [arXiv:2407.14435](https://arxiv.org/abs/2407.14435) [find](https://scholar.google.com/scholar?q=Senthooran+Rajamanoharan%2C+Tom+Lieberum%2C+Nicolas+Sonnerat%2C+Arthur+Conmy%2C+Vikrant+Varma%2C+J%C3%A1nos+Kram%C3%A1r%2C+and+Neel+Nanda%2E+Jumping+ahead%3A+Improving+reconstruction+fidelity+with+jumprelu+sparse+autoencoders%2E)
157. Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp. 464–483. [find](https://scholar.google.com/scholar?q=Tilman+R%C3%A4uker%2C+Anson+Ho%2C+Stephen+Casper%2C+and+Dylan+Hadfield%2DMenell%2E+Toward+transparent+ai%3A+A+survey+on+interpreting+the+inner+structures+of+deep+neural+networks%2E+In+2023+IEEE+Conference+on+Secure+and+)
158. Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context classification task. In The Twelfth International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=Gautam+Reddy%2E+The+mechanistic+basis+of+data+dependence+and+abrupt+learning+in+an+in%2Dcontext+classification+task%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+2023%2E)
159. Jie Ren, Qipeng Guo, Hang Yan, Dongrui Liu, Xipeng Qiu, and Dahua Lin. Identifying semantic induction heads to understand in-context learning. arXiv preprint arXiv:2402.13055, 2024. [find](https://scholar.google.com/scholar?q=Jie+Ren%2C+Qipeng+Guo%2C+Hang+Yan%2C+Dongrui+Liu%2C+Xipeng+Qiu%2C+and+Dahua+Lin%2E+Identifying+semantic+induction+heads+to+understand+in%2Dcontext+learning%2E+arXiv+preprint+arXiv%3A2402%2E13055%2C+2024%2E)
160. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135[–1144, 2016](https://md.research.elamur.ai/free/en/papers/ribeiro-2016-why-should-i-trust-explaining/).
161. Logan Riggs. Found 600+ monosemantic features in a small lm using sparse autoencoders. [https://www.lesswrong.com/posts/wqRqb7h6ZC48iDgfK/](https://www.lesswrong.com/posts/wqRqb7h6ZC48iDgfK/) tentatively-found-600-monosemantic-features-in-a-small-lm, 2023. [↗](https://www.lesswrong.com/posts/wqRqb7h6ZC48iDgfK/)
162. Mansi Sakarvadia, Arham Khan, Aswathy Ajith, Daniel Grzenda, Nathaniel Hudson, André Bauer, Kyle Chard, and Ian Foster. Attention lens: A tool for mechanistically interpreting the attention head information retrieval mechanism. arXiv preprint arXiv:2310.16270, 2023. — [arXiv:2310.16270](https://arxiv.org/abs/2310.16270) [find](https://scholar.google.com/scholar?q=Mansi+Sakarvadia%2C+Arham+Khan%2C+Aswathy+Ajith%2C+Daniel+Grzenda%2C+Nathaniel+Hudson%2C+Andr%C3%A9+Bauer%2C+Kyle+Chard%2C+and+Ian+Foster%2E+Attention+lens%3A+A+tool+for+mechanistically+interpreting+the+attention+head+infor)
163. Abulhair Saparov, Srushti Pawar, Shreyas Pimpalgaonkar, Nitish Joshi, Richard Yuanzhe Pang, Vishakh Padmakumar, Seyed Mehran Kazemi, Najoung Kim, and He He. Transformers struggle to learn to search. arXiv preprint arXiv:2412.04703, 2024. [find](https://scholar.google.com/scholar?q=Abulhair+Saparov%2C+Srushti+Pawar%2C+Shreyas+Pimpalgaonkar%2C+Nitish+Joshi%2C+Richard+Yuanzhe+Pang%2C+Vishakh+Padmakumar%2C+Seyed+Mehran+Kazemi%2C+Najoung+Kim%2C+and+He+He%2E+Transformers+struggle+to+learn+to+search%2E+a)
164. Naomi Saphra and Sarah Wiegreffe. Mechanistic? arXiv preprint arXiv:2410.09087, 2024. — [arXiv:2410.09087](https://arxiv.org/abs/2410.09087) [find](https://scholar.google.com/scholar?q=Naomi+Saphra+and+Sarah+Wiegreffe%2E+Mechanistic%3F+arXiv+preprint+arXiv%3A2410%2E09087%2C+2024%2E+%E2%80%94+%5BarXiv%3A2410%2E09087%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2410%2E09087%29)
165. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ili%C4%87%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+et+al%2E+Bloom%3A+A+176b%2Dparameter+open%2Daccess+multilingu)
166. Adam Scherlis, Kshitij Sachan, Adam S Jermyn, Joe Benton, and Buck Shlegeris. Polysemanticity and capacity in neural networks. arXiv preprint arXiv:2210.01892, 2022. — [arXiv:2210.01892](https://arxiv.org/abs/2210.01892) [find](https://scholar.google.com/scholar?q=Adam+Scherlis%2C+Kshitij+Sachan%2C+Adam+S+Jermyn%2C+Joe+Benton%2C+and+Buck+Shlegeris%2E+Polysemanticity+and+capacity+in+neural+networks%2E+arXiv+preprint+arXiv%3A2210%2E01892%2C+2022%2E+%E2%80%94+%5BarXiv%3A2210%2E01892%5D%28https%3A%2F%2Farxiv)
167. Sarah Schwettmann, Tamar Rott Shaham, Joanna Materzynska, Neil Chowdhury, Shuang Li, Jacob Andreas, David Bau, and Antonio Torralba. FIND: A function description benchmark for evaluating interpretability methods. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benc… [find](https://scholar.google.com/scholar?q=Sarah+Schwettmann%2C+Tamar+Rott+Shaham%2C+Joanna+Materzynska%2C+Neil+Chowdhury%2C+Shuang+Li%2C+Jacob+Andreas%2C+David+Bau%2C+and+Antonio+Torralba%2E+FIND%3A+A+function+description+benchmark+for+evaluating+interpretabil)
168. Lee Sharkey, Dan Braun, and Beren Millidge. Taking features out of superposition with sparse autoencoders, 2022. [https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj/](https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj/) interim-research-report-taking-features-out-of-superposition, 2022. [↗](https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj/)
169. Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, et al. Open problems in mechanistic interpretability. arXiv preprint arXiv:2501.16496, 2025. — [arXiv:2501.16496](https://arxiv.org/abs/2501.16496) [find](https://scholar.google.com/scholar?q=Lee+Sharkey%2C+Bilal+Chughtai%2C+Joshua+Batson%2C+Jack+Lindsey%2C+Jeff+Wu%2C+Lucius+Bushnaq%2C+Nicholas+Goldowsky%2DDill%2C+Stefan+Heimersheim%2C+Alejandro+Ortega%2C+Joseph+Bloom%2C+et+al%2E+Open+problems+in+mechanistic+inte)
170. Arnab Sen Sharma, David Atkinson, and David Bau. [Locating and editing factual associations in mamba](https://md.research.elamur.ai/free/en/papers/sharma-2024-locating-editing-factual-associations-mamba/). arXiv preprint arXiv:2404.03646, 2024.
171. Dong Shu, Xuansheng Wu, Haiyan Zhao, Daking Rai, Ziyu Yao, Ninghao Liu, and Mengnan Du. A survey on sparse autoencoders: Interpreting the internal mechanisms of large language models. arXiv preprint arXiv:2503.05613, 2025. [find](https://scholar.google.com/scholar?q=Dong+Shu%2C+Xuansheng+Wu%2C+Haiyan+Zhao%2C+Daking+Rai%2C+Ziyu+Yao%2C+Ninghao+Liu%2C+and+Mengnan+Du%2E+A+survey+on+sparse+autoencoders%3A+Interpreting+the+internal+mechanisms+of+large+language+models%2E+arXiv+preprint+a)
172. Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7035–7052, 2023a. [find](https://scholar.google.com/scholar?q=Alessandro+Stolfo%2C+Yonatan+Belinkov%2C+and+Mrinmaya+Sachan%2E+A+mechanistic+interpretation+of+arithmetic+reasoning+in+language+models+using+causal+mediation+analysis%2E+In+Proceedings+of+the+2023+Conference)
173. Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. Understanding arithmetic reasoning in language models using causal mediation analysis. arXiv preprint arXiv:2305.15054, 2023b. [find](https://scholar.google.com/scholar?q=Alessandro+Stolfo%2C+Yonatan+Belinkov%2C+and+Mrinmaya+Sachan%2E+Understanding+arithmetic+reasoning+in+language+models+using+causal+mediation+analysis%2E+arXiv+preprint+arXiv%3A2305%2E15054%2C+2023b%2E)
174. Alessandro Stolfo, Ben Peng Wu, Wes Gurnee, Yonatan Belinkov, Xingyi Song, Mrinmaya Sachan, and Neel Nanda. Confidence regulation neurons in language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL [https://openreview.net/forum?id=0og7nmvDbe](https://openreview.net/forum?id=0og7nmvDbe). [↗](https://openreview.net/forum?id=0og7nmvDbe)
175. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319–3328. PMLR, 2017. [find](https://scholar.google.com/scholar?q=Mukund+Sundararajan%2C+Ankur+Taly%2C+and+Qiqi+Yan%2E+Axiomatic+attribution+for+deep+networks%2E+In+International+conference+on+machine+learning%2C+pp%2E+3319%E2%80%933328%2E+PMLR%2C+2017%2E)
176. Aaquib Syed, Can Rager, and Arthur Conmy. [Attribution patching outperforms automated circuit discovery](https://md.research.elamur.ai/free/en/papers/syed-2023-attribution-patching-outperforms-automated-circuit/). arXiv preprint arXiv:2310.10348, 2023.
177. Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. [Language-specific neurons: The key to multilingual capabilities in large language models](https://md.research.elamur.ai/free/en/papers/tang-2024-language-specific-neurons-key-multilingual/). In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the…
178. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Morgane+Riviere%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+L%C3%A9onard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Ram%C3%A9%2C+et+al%2E+Gemma+2%3A+Improving+open+la)
179. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Sh… [find](https://scholar.google.com/scholar?q=Adly+Templeton%2C+Tom+Conerly%2C+Jonathan+Marcus%2C+Jack+Lindsey%2C+Trenton+Bricken%2C+Brian+Chen%2C+Adam+Pearce%2C+Craig+Citro%2C+Emmanuel+Ameisen%2C+Andy+Jones%2C+Hoagy+Cunningham%2C+Nicholas+L+Turner%2C+Callum+McDougall%2C+)
180. Ian Tenney, Dipanjan Das, and Ellie Pavlick. [Bert rediscovers the classical nlp pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4593–4601, 2019a.
181. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. [What do you learn from context? probing for sentence structure in contextualized word representations](https://md.research.elamur.ai/free/en/papers/tenney-2019-learn-context-probing-sentence-structure/). arXiv preprint arXiv:1905.06316, 2019b.
182. Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. [Linear representations of sentiment in large language models](https://md.research.elamur.ai/free/en/papers/tigges-2023-linear-representations-sentiment-large-language/). arXiv preprint arXiv:2310.15154, 2023.
183. Curt Tigges, Michael Hanna, Qinan Yu, and Stella Biderman. Llm circuit analyses are consistent across training and scale. arXiv preprint arXiv:2407.10827, 2024. — [arXiv:2407.10827](https://arxiv.org/abs/2407.10827) [find](https://scholar.google.com/scholar?q=Curt+Tigges%2C+Michael+Hanna%2C+Qinan+Yu%2C+and+Stella+Biderman%2E+Llm+circuit+analyses+are+consistent+across+training+and+scale%2E+arXiv+preprint+arXiv%3A2407%2E10827%2C+2024%2E+%E2%80%94+%5BarXiv%3A2407%2E10827%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
184. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
185. Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36:74952–74965, 2023. — [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://md.research.elamur.ai/free/en/papers/turpin-2023-language-models-don-t-always/)
186. Niels uit de Bos and Adrià Garriga-Alonso. Adversarial circuit evaluation. In ICML 2024 Workshop on Mechanistic Interpretability, 2024. [find](https://scholar.google.com/scholar?q=Niels+uit+de+Bos+and+Adri%C3%A0+Garriga%2DAlonso%2E+Adversarial+circuit+evaluation%2E+In+ICML+2024+Workshop+on+Mechanistic+Interpretability%2C+2024%2E)
187. Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar. Explaining grokking through circuit efficiency. arXiv preprint arXiv:2309.02390, 2023. — [arXiv:2309.02390](https://arxiv.org/abs/2309.02390) [find](https://scholar.google.com/scholar?q=Vikrant+Varma%2C+Rohin+Shah%2C+Zachary+Kenton%2C+J%C3%A1nos+Kram%C3%A1r%2C+and+Ramana+Kumar%2E+Explaining+grokking+through+circuit+efficiency%2E+arXiv+preprint+arXiv%3A2309%2E02390%2C+2023%2E+%E2%80%94+%5BarXiv%3A2309%2E02390%5D%28https%3A%2F%2Farxiv%2Eorg)
188. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
189. Jesse Vig. Visualizing attention in transformer-based language representation models. arXiv:1904.02679, 2019. arXiv preprint Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. Neurons in large language models: Dead, ngram, positional. arXiv preprint arXiv:2309.04827, 2023. — [arXiv:1904.02679](https://arxiv.org/abs/1904.02679) [find](https://scholar.google.com/scholar?q=Jesse+Vig%2E+Visualizing+attention+in+transformer%2Dbased+language+representation+models%2E+arXiv%3A1904%2E02679%2C+2019%2E+arXiv+preprint+Elena+Voita%2C+Javier+Ferrando%2C+and+Christoforos+Nalmpantis%2E+Neurons+in+large)
190. Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. Neurons in large language models: Dead, ngram, positional. In Findings of the Association for Computational Linguistics ACL 2024, pp. 1288–1301, 2024. [find](https://scholar.google.com/scholar?q=Elena+Voita%2C+Javier+Ferrando%2C+and+Christoforos+Nalmpantis%2E+Neurons+in+large+language+models%3A+Dead%2C+ngram%2C+positional%2E+In+Findings+of+the+Association+for+Computational+Linguistics+ACL+2024%2C+pp%2E+1288%E2%80%9313)
191. Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. [Grokked transformers are implicit reasoners: A mechanistic journey to the edge of generalization](https://md.research.elamur.ai/free/en/papers/wang-2024-grokked-transformers-implicit-reasoners-mechanistic/). arXiv preprint arXiv:2405.15071, 2024a.
192. Junxuan Wang, Xuyang Ge, Wentao Shu, Qiong Tang, Yunhua Zhou, Zhengfu He, and Xipeng Qiu. Towards universality: Studying mechanistic similarity across language model architectures. arXiv preprint arXiv:2410.06672, 2024b. — [arXiv:2410.06672](https://arxiv.org/abs/2410.06672) [find](https://scholar.google.com/scholar?q=Junxuan+Wang%2C+Xuyang+Ge%2C+Wentao+Shu%2C+Qiong+Tang%2C+Yunhua+Zhou%2C+Zhengfu+He%2C+and+Xipeng+Qiu%2E+Towards+universality%3A+Studying+mechanistic+similarity+across+language+model+architectures%2E+arXiv+preprint+arXi)
193. Kevin Wang. Gears-level mental models of transformer interpretability. [https://www.lesswrong.com/](https://www.lesswrong.com/) posts/X26ksz4p3wSyycKNB/gears-level-mental-models-of-transformer-interpretability, 2022. [↗](https://www.lesswrong.com/)
194. Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. In The Eleventh International Conference on Learning Representations, 2022a. [find](https://scholar.google.com/scholar?q=Kevin+Ro+Wang%2C+Alexandre+Variengien%2C+Arthur+Conmy%2C+Buck+Shlegeris%2C+and+Jacob+Steinhardt%2E+Interpretability+in+the+wild%3A+a+circuit+for+indirect+object+identification+in+gpt%2D2+small%2E+In+The+Eleventh+Inte)
195. Mengru Wang, Yunzhi Yao, Ziwen Xu, Shuofei Qiao, Shumin Deng, Peng Wang, Xiang Chen, Jia-Chen Gu, Yong Jiang, Pengjun Xie, et al. Knowledge mechanisms in large language models: A survey and perspective. arXiv preprint arXiv:2407.15017, 2024c. [find](https://scholar.google.com/scholar?q=Mengru+Wang%2C+Yunzhi+Yao%2C+Ziwen+Xu%2C+Shuofei+Qiao%2C+Shumin+Deng%2C+Peng+Wang%2C+Xiang+Chen%2C+Jia%2DChen+Gu%2C+Yong+Jiang%2C+Pengjun+Xie%2C+et+al%2E+Knowledge+mechanisms+in+large+language+models%3A+A+survey+and+perspectiv)
196. Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. [Finding skill neurons in pre-trained transformer-based language models](https://md.research.elamur.ai/free/en/papers/wang-2022-finding-skill-neurons-pre-trained/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11132–11152, 2022b.
197. Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept algebra for (score-based) text-controlled generative models. Advances in Neural Information Processing Systems, 36, 2024d. [find](https://scholar.google.com/scholar?q=Zihao+Wang%2C+Lin+Gui%2C+Jeffrey+Negrea%2C+and+Victor+Veitch%2E+Concept+algebra+for+%28score%2Dbased%29+text%2Dcontrolled+generative+models%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2C+2024d%2E)
198. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). Transactions on Machine Learning Research, 2022a.
199. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824–24837, 2022b.
200. Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 214–22… [find](https://scholar.google.com/scholar?q=Laura+Weidinger%2C+Jonathan+Uesato%2C+Maribeth+Rauh%2C+Conor+Griffin%2C+Po%2DSen+Huang%2C+John+Mellor%2C+Amelia+Glaese%2C+Myra+Cheng%2C+Borja+Balle%2C+Atoosa+Kasirzadeh%2C+et+al%2E+Taxonomy+of+risks+posed+by+language+models%2E)
201. Gail Weiss, Yoav Goldberg, and Eran Yahav. Thinking like transformers. In International Conference on Machine Learning, pp. 11080–11090. PMLR, 2021. [find](https://scholar.google.com/scholar?q=Gail+Weiss%2C+Yoav+Goldberg%2C+and+Eran+Yahav%2E+Thinking+like+transformers%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+11080%E2%80%9311090%2E+PMLR%2C+2021%2E)
202. Eric Winsor. Re-examining layernorm. AI Alignment Forum, 2020. [https://www.alignmentforum.org/](https://www.alignmentforum.org/) posts/jfG6vdJZCwTQmG7kb/re-examining-layernorm. [↗](https://www.alignmentforum.org/)
203. Benjamin Wright and Lee Sharkey. Addressing feature suppression in saes. In AI Alignment Forum, pp. 16, 2024. [find](https://scholar.google.com/scholar?q=Benjamin+Wright+and+Lee+Sharkey%2E+Addressing+feature+suppression+in+saes%2E+In+AI+Alignment+Forum%2C+pp%2E+16%2C+2024%2E)
204. Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. [A survey on large language model (llm) security and privacy: The good, the bad, and the ugly](https://md.research.elamur.ai/free/en/papers/yao-2023-survey-large-language-model-llm/). High-Confidence Computing, pp. 100211, 2024.
205. Fred Zhang and Neel Nanda. [Towards best practices of activation patching in language models: Metrics and methods](https://md.research.elamur.ai/free/en/papers/zhang-2023-towards-best-practices-activation-patching/). In The Twelfth International Conference on Learning Representations, 2023.
206. Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. [Explainability for large language models: A survey](https://md.research.elamur.ai/free/en/papers/zhao-2023-explainability-large-language-models-survey/). ACM Transactions on Intelligent Systems and Technology, 15(2):1–38, 2024.
207. Yiran Zhao, Wenxuan Zhang, Yuxi Xie, Anirudh Goyal, Kenji Kawaguchi, and Michael Shieh. Understanding and enhancing safety mechanisms of llms via safety-specific neuron. In The Thirteenth International Conference on Learning Representations, 2025. [find](https://scholar.google.com/scholar?q=Yiran+Zhao%2C+Wenxuan+Zhang%2C+Yuxi+Xie%2C+Anirudh+Goyal%2C+Kenji+Kawaguchi%2C+and+Michael+Shieh%2E+Understanding+and+enhancing+safety+mechanisms+of+llms+via+safety%2Dspecific+neuron%2E+In+The+Thirteenth+Internationa)
208. Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Ziqian+Zhong%2C+Ziming+Liu%2C+Max+Tegmark%2C+and+Jacob+Andreas%2E+The+clock+and+the+pizza%3A+Two+stories+in+mechanistic+explanation+of+neural+networks%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2C+202)
209. Yilun Zhou, Serena Booth, Marco Tulio Ribeiro, and Julie Shah. Do feature attribution methods correctly attribute features? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 9623–9633, 2022. [find](https://scholar.google.com/scholar?q=Yilun+Zhou%2C+Serena+Booth%2C+Marco+Tulio+Ribeiro%2C+and+Julie+Shah%2E+Do+feature+attribution+methods+correctly+attribute+features%3F+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+36%2C)
210. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. [Representation engineering: A top-down approach to ai transparency](https://md.research.elamur.ai/free/en/papers/zou-2023-representation-engineering-top-down-approach/). arXiv preprint arXiv:2310.01405, 2023.
211. Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, J Zico Kolter, Matt Fredrikson, and Dan Hendrycks. [Improving alignment and robustness with circuit breakers](https://md.research.elamur.ai/free/en/papers/zou-2024-improving-alignment-robustness-circuit-breakers/). In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.

*Updated 3 November 2026*