---
title: "Finding Neurons in a Haystack: Case Studies with Sparse Probing"
description: "The authors introduce sparse probing: linear classifiers that may use at most k neurons from a… 108 references, cited by 111 papers here."
authors: ["Wes Gurnee", "Neel Nanda", "Matthew Pauly", "Katherine Harvey", "Dmitrii Troitskii", "Dimitris Bertsimas"]
year: 2023
venue: "arXiv:2305.01610"
source: "https://arxiv.org/abs/2305.01610"
doi: "10.48550/arxiv.2305.01610"
arxiv: "2305.01610"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gurnee-2023-finding-neurons-haystack-case-studies/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gurnee-2023-finding-neurons-haystack-case-studies/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Finding Neurons in a Haystack: Case Studies with Sparse Probing

# Finding Neurons in a Haystack: Case Studies with Sparse Probing

**Authors:** [Wes Gurnee](https://md.research.elamur.ai/free/en/authors/wes-gurnee/), [Neel Nanda](https://md.research.elamur.ai/free/en/authors/neel-nanda/), Matthew Pauly, Katherine Harvey, Dmitrii Troitskii and Dimitris Bertsimas · **Year:** 2023 · **Venue:** arXiv:2305.01610 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2305.01610) · [PDF](https://arxiv.org/pdf/2305.01610) · [DOI 10.48550/arxiv.2305.01610](https://doi.org/10.48550/arxiv.2305.01610) · [arXiv 2305.01610](https://arxiv.org/abs/2305.01610) · [OpenAlex](https://openalex.org/W4368304611)

**Citations:** 12 citations (OpenAlex, 2026-10-14) · [111](https://md.research.elamur.ai/free/en/papers/gurnee-2023-finding-neurons-haystack-case-studies/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models are hard to inspect. It is unclear how text properties that humans can recognize are stored in individual MLP neurons, especially when a network must encode more features than it has neurons (superposition). The authors introduce sparse probing: linear classifiers that may use at most k neurons from a layer's post-nonlinearity MLP activations. Varying k shows how concentrated a feature's representation is. Setting k = 1 isolates single candidate neurons, and for small k, optimal sparse prediction methods can prove that the selected neuron subset is optimal. The study covers more than 100 binary features in 10 categories across 7 models from 70M to 6.9B parameters, followed by detailed case studies. Early layers encode many features through sparse mixtures of polysemantic neurons that react to unrelated n-grams. Middle layers hold seemingly dedicated neurons for high-level context, such as whether text is code. Larger models are sparser on average, but individual features scale differently. The work matters because it provides a practical way to find interpretable neurons and tests superposition theory on real LLMs.

## Key points
- Scope: over 100 features in 10 categories, probed in 7 models from the Pythia suite (70M–6.9B parameters), using MLP activations after the GeLU.
- The small probe capacity limits memorization of spurious correlations. Exact sparse selection for small k separates neuron-ranking quality from classification quality.
- Evidence from weight statistics and toy models indicates that superposition is used far more heavily in roughly the earliest 25% of fully connected layers than in the layers that follow.
- Contextual features (e.g., `is_python_code`, Pile subset context neurons) appear to have dedicated mid-layer neurons, though the authors say monosemanticity cannot be proven conclusively.
- Scaling dynamics vary by feature: some dedicated neurons emerge only at larger sizes, some features split into finer-grained neurons, and others stay stable or behave erratically. Experiments used the TransformerLens library.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🔬 Interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/) · 149 works

## Cite

Cite as: Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii and Dimitris Bertsimas. 2023. Finding Neurons in a Haystack: Case Studies with Sparse Probing. arXiv:2305.01610.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{gurnee2023finding,
  title = {{Finding Neurons in a Haystack: Case Studies with Sparse Probing}},
  author = {Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii and Dimitris Bertsimas},
  year = {2023},
  doi = {10.48550/arxiv.2305.01610},
  eprint = {2305.01610},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.01610}
}
```

- [← previous Fast Distributed Inference Serving for Large Language Models](https://md.research.elamur.ai/free/en/papers/wu-2023-fast-distributed-inference-serving-large/)
- [next → INSTRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic Feedback](https://md.research.elamur.ai/free/en/papers/xu-2023-instructscore-explainable-text-generation-evaluation/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

108 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yoshua Bengio, Aaron Courville, and Pascal Vincent. [Representation learning: A review and new perspectives](https://md.research.elamur.ai/free/en/papers/bengio-2012-representation-learning-review-new-perspectives/). IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
2. Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. [Decaf: A deep convolutional activation feature for generic visual recognition](https://md.research.elamur.ai/free/en/papers/donahue-2013-decaf-deep-convolutional-activation-feature/). In International conference on machine learning, pages 647–655. PMLR, 2014.
3. Yonatan Belinkov. [Probing classifiers: Promises, shortcomings, and advances](https://md.research.elamur.ai/free/en/papers/belinkov-2022-probing-classifiers-promises-shortcomings-advances/). Computational Linguistics, 48(1):207–219, 2022.
4. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022. [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Tristan+Hume%2C+Catherine+Olsson%2C+Nicholas+Schiefer%2C+Tom+Henighan%2C+Shauna+Kravec%2C+Zac+Hatfield%2DDodds%2C+Robert+Lasenby%2C+Dawn+Drain%2C+Carol+Chen%2C+et+al%2E+Toy+models+of+superposition%2E+arXiv+pre)
5. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3):e00024–001, 2020. [find](https://scholar.google.com/scholar?q=Chris+Olah%2C+Nick+Cammarata%2C+Ludwig+Schubert%2C+Gabriel+Goh%2C+Michael+Petrov%2C+and+Shan+Carter%2E+Zoom+in%3A+An+introduction+to+circuits%2E+Distill%2C+5%283%29%3Ae00024%E2%80%93001%2C+2020%2E)
6. Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. [Learning to generate reviews and discovering sentiment](https://md.research.elamur.ai/free/en/papers/radford-2017-learning-generate-reviews-discovering-sentiment/). arXiv preprint arXiv:1704.01444, 2017.
7. David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences, 2020. [find](https://scholar.google.com/scholar?q=David+Bau%2C+Jun%2DYan+Zhu%2C+Hendrik+Strobelt%2C+Agata+Lapedriza%2C+Bolei+Zhou%2C+and+Antonio+Torralba%2E+Understanding+the+role+of+individual+units+in+a+deep+neural+network%2E+Proceedings+of+the+National+Academy+of)
8. Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 6(3):e30, 2021. [find](https://scholar.google.com/scholar?q=Gabriel+Goh%2C+Nick+Cammarata%2C+Chelsea+Voss%2C+Shan+Carter%2C+Michael+Petrov%2C+Ludwig+Schubert%2C+Alec+Radford%2C+and+Chris+Olah%2E+Multimodal+neurons+in+artificial+neural+networks%2E+Distill%2C+6%283%29%3Ae30%2C+2021%2E)
9. Nick Cammarata, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah. Curve circuits. Distill, 2021. [https://distill.pub/2020/circuits/curve-circuits](https://distill.pub/2020/circuits/curve-circuits). [↗](https://distill.pub/2020/circuits/curve-circuits)
10. Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer ElShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatfield-Dodds, Jackson Kernion, Tom Con… [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Tristan+Hume%2C+Catherine+Olsson%2C+Neel+Nanda%2C+Tom+Henighan%2C+Scott+Johnston%2C+Sheer+ElShowk%2C+Nicholas+Joseph%2C+Nova+DasSarma%2C+Ben+Mann%2C+Danny+Hernandez%2C+Amanda+Askell%2C+Kamal+Ndousse%2C+Dawn+Dr)
11. Adam Scherlis, Kshitij Sachan, Adam S Jermyn, Joe Benton, and Buck Shlegeris. Polysemanticity and capacity in neural networks. arXiv preprint arXiv:2210.01892, 2022. — [arXiv:2210.01892](https://arxiv.org/abs/2210.01892) [find](https://scholar.google.com/scholar?q=Adam+Scherlis%2C+Kshitij+Sachan%2C+Adam+S+Jermyn%2C+Joe+Benton%2C+and+Buck+Shlegeris%2E+Polysemanticity+and+capacity+in+neural+networks%2E+arXiv+preprint+arXiv%3A2210%2E01892%2C+2022%2E+%E2%80%94+%5BarXiv%3A2210%2E01892%5D%28https%3A%2F%2Farxiv)
12. Jesse Mu and Jacob Andreas. Compositional explanations of neurons. Advances in Neural Information Processing Systems, 33:17153–17163, 2020. [find](https://scholar.google.com/scholar?q=Jesse+Mu+and+Jacob+Andreas%2E+Compositional+explanations+of+neurons%2E+Advances+in+Neural+Information+Processing+Systems%2C+33%3A17153%E2%80%9317163%2C+2020%2E)
13. Omer Antverg and Yonatan Belinkov. On the pitfalls of analyzing individual neurons in language models. arXiv preprint arXiv:2110.07483, 2021. — [arXiv:2110.07483](https://arxiv.org/abs/2110.07483) [find](https://scholar.google.com/scholar?q=Omer+Antverg+and+Yonatan+Belinkov%2E+On+the+pitfalls+of+analyzing+individual+neurons+in+language+models%2E+arXiv+preprint+arXiv%3A2110%2E07483%2C+2021%2E+%E2%80%94+%5BarXiv%3A2110%2E07483%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2110%2E07483%29)
14. J Hewitt and P Liang. Designing and interpreting probes with control tasks. Proceedings of the 2019 Con, 2019. [find](https://scholar.google.com/scholar?q=J+Hewitt+and+P+Liang%2E+Designing+and+interpreting+probes+with+control+tasks%2E+Proceedings+of+the+2019+Con%2C+2019%2E)
15. Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. [Probing the probing paradigm: Does probing accuracy entail task relevance?](https://md.research.elamur.ai/free/en/papers/ravichander-2021-probing-probing-paradigm-does-probing/) arXiv preprint arXiv:2005.00719, 2020.
16. Dimitris Bertsimas, Jean Pauphilet, and Bart Van Parys. Sparse regression: Scalable algorithms and empirical performance. 2020. [find](https://scholar.google.com/scholar?q=Dimitris+Bertsimas%2C+Jean+Pauphilet%2C+and+Bart+Van+Parys%2E+Sparse+regression%3A+Scalable+algorithms+and+empirical+performance%2E+2020%2E)
17. Dimitris Bertsimas, Jean Pauphilet, and Bart Van Parys. Sparse classification: a scalable discrete optimization perspective. Machine Learning, 110:3177–3209, 2021. [find](https://scholar.google.com/scholar?q=Dimitris+Bertsimas%2C+Jean+Pauphilet%2C+and+Bart+Van+Parys%2E+Sparse+classification%3A+a+scalable+discrete+optimization+perspective%2E+Machine+Learning%2C+110%3A3177%E2%80%933209%2C+2021%2E)
18. Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016. [find](https://scholar.google.com/scholar?q=Guillaume+Alain+and+Yoshua+Bengio%2E+Understanding+intermediate+layers+using+linear+classifier+probes%2E+arXiv+preprint+arXiv%3A1610%2E01644%2C+2016%2E)
19. Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. What you can cram into a single vector: Probing sentence embeddings for linguistic properties, 2018. — [What you can cram into a single \$&!\#* vector: Probing sentence embeddings for linguistic properties](https://md.research.elamur.ai/free/en/papers/conneau-2018-what-you-can-cram-into/)
20. Ian Tenney, Dipanjan Das, and Ellie Pavlick. [Bert rediscovers the classical nlp pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/). arXiv preprint arXiv:1905.05950, 2019.
21. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. [What do you learn from context? probing for sentence structure in contextualized word representations](https://md.research.elamur.ai/free/en/papers/tenney-2019-learn-context-probing-sentence-structure/). arXiv preprint arXiv:1905.06316, 2019.
22. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. [A primer in bertology: What we know about how bert works](https://md.research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/), 2020.
23. Xavier Suau, Luca Zappella, and Nicholas Apostoloff. arXiv:2005.07647, 2020. Finding experts in transformer models. arXiv preprint — [arXiv:2005.07647](https://arxiv.org/abs/2005.07647) [find](https://scholar.google.com/scholar?q=Xavier+Suau%2C+Luca+Zappella%2C+and+Nicholas+Apostoloff%2E+arXiv%3A2005%2E07647%2C+2020%2E+Finding+experts+in+transformer+models%2E+arXiv+preprint+%E2%80%94+%5BarXiv%3A2005%2E07647%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2005%2E07647%29)
24. Hassan Sajjad, Nadir Durrani, and Fahim Dalvi. [Neuron-level interpretation of deep nlp models: A survey](https://md.research.elamur.ai/free/en/papers/sajjad-2022-neuron-level-interpretation-deep-nlp/). Transactions of the Association for Computational Linguistics, 10:1285–1303, 2022.
25. Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. [Finding skill neurons in pre-trained transformer-based language models](https://md.research.elamur.ai/free/en/papers/wang-2022-finding-skill-neurons-pre-trained/). arXiv preprint arXiv:2211.07349, 2022.
26. Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. Analyzing individual neurons in pre-trained language models. arXiv preprint arXiv:2010.02695, 2020. [find](https://scholar.google.com/scholar?q=Nadir+Durrani%2C+Hassan+Sajjad%2C+Fahim+Dalvi%2C+and+Yonatan+Belinkov%2E+Analyzing+individual+neurons+in+pre%2Dtrained+language+models%2E+arXiv+preprint+arXiv%3A2010%2E02695%2C+2020%2E)
27. Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6309–6317, 2019. [find](https://scholar.google.com/scholar?q=Fahim+Dalvi%2C+Nadir+Durrani%2C+Hassan+Sajjad%2C+Yonatan+Belinkov%2C+Anthony+Bau%2C+and+James+Glass%2E+What+is+one+grain+of+sand+in+the+desert%3F+analyzing+individual+neurons+in+deep+nlp+models%2E+In+Proceedings+of+t)
28. Jonathan Donnelly and Adam Roegiest. On interpretability and feature representations: an analysis of the sentiment neuron. In Advances in Information Retrieval: 41st European Conference on IR Research, ECIR 2019, Cologne, Germany, April 14–18, 2019, Proceedings, Part I 41, pages 795–802. Springer, 2… [find](https://scholar.google.com/scholar?q=Jonathan+Donnelly+and+Adam+Roegiest%2E+On+interpretability+and+feature+representations%3A+an+analysis+of+the+sentiment+neuron%2E+In+Advances+in+Information+Retrieval%3A+41st+European+Conference+on+IR+Research)
29. Rowan Hall Maudslay and Ryan Cotterell. Do syntactic probes probe syntax? experiments with jabberwocky probing. arXiv preprint arXiv:2106.02559, 2021. [find](https://scholar.google.com/scholar?q=Rowan+Hall+Maudslay+and+Ryan+Cotterell%2E+Do+syntactic+probes+probe+syntax%3F+experiments+with+jabberwocky+probing%2E+arXiv+preprint+arXiv%3A2106%2E02559%2C+2021%2E)
30. Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. [Amnesic probing: Behavioral explanation with amnesic counterfactuals](https://md.research.elamur.ai/free/en/papers/elazar-2021-amnesic-probing-behavioral-explanation-amnesic/). Transactions of the Association for Computational Linguistics, 9:160–175, 2021.
31. Lucas Torroba Hennigen, Adina Williams, and Ryan Cotterell. Intrinsic probing through dimension selection. arXiv preprint arXiv:2010.02812, 2020. [find](https://scholar.google.com/scholar?q=Lucas+Torroba+Hennigen%2C+Adina+Williams%2C+and+Ryan+Cotterell%2E+Intrinsic+probing+through+dimension+selection%2E+arXiv+preprint+arXiv%3A2010%2E02812%2C+2020%2E)
32. Elena Voita and Ivan Titov. [Information-theoretic probing with minimum description length](https://md.research.elamur.ai/free/en/papers/voita-2020-information-theoretic-probing-minimum-description/). arXiv preprint arXiv:2003.12298, 2020.
33. Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. [Information-theoretic probing for linguistic structure](https://md.research.elamur.ai/free/en/papers/pimentel-2020-information-theoretic-probing-linguistic-structure/). arXiv preprint arXiv:2004.03061, 2020.
34. Steven Cao, Victor Sanh, and Alexander M Rush. Low-complexity probing via finding subnetworks. arXiv preprint arXiv:2104.03514, 2021. [find](https://scholar.google.com/scholar?q=Steven+Cao%2C+Victor+Sanh%2C+and+Alexander+M+Rush%2E+Low%2Dcomplexity+probing+via+finding+subnetworks%2E+arXiv+preprint+arXiv%3A2104%2E03514%2C+2021%2E)
35. Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. [Identifying and controlling important neurons in neural machine translation](https://md.research.elamur.ai/free/en/papers/bau-2018-identifying-controlling-important-neurons-neural/). arXiv preprint arXiv:1811.01157, 2018.
36. Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. How important is a neuron? arXiv preprint arXiv:1805.12233, 2018. — [arXiv:1805.12233](https://arxiv.org/abs/1805.12233) [find](https://scholar.google.com/scholar?q=Kedar+Dhamdhere%2C+Mukund+Sundararajan%2C+and+Qiqi+Yan%2E+How+important+is+a+neuron%3F+arXiv+preprint+arXiv%3A1805%2E12233%2C+2018%2E+%E2%80%94+%5BarXiv%3A1805%2E12233%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1805%2E12233%29)
37. Rana Ali Amjad, Kairen Liu, and Bernhard C Geiger. Understanding neural networks and individual neuron importance via information-ordered cumulative ablation. IEEE Transactions on Neural Networks and Learning Systems, 2021. [find](https://scholar.google.com/scholar?q=Rana+Ali+Amjad%2C+Kairen+Liu%2C+and+Bernhard+C+Geiger%2E+Understanding+neural+networks+and+individual+neuron+importance+via+information%2Dordered+cumulative+ablation%2E+IEEE+Transactions+on+Neural+Networks+and+)
38. Nicola De Cao, Leon Schmid, Dieuwke Hupkes, and Ivan Titov. Sparse interventions in language models with differentiable masking. arXiv preprint arXiv:2112.06837, 2021. [find](https://scholar.google.com/scholar?q=Nicola+De+Cao%2C+Leon+Schmid%2C+Dieuwke+Hupkes%2C+and+Ivan+Titov%2E+Sparse+interventions+in+language+models+with+differentiable+masking%2E+arXiv+preprint+arXiv%3A2112%2E06837%2C+2021%2E)
39. Nadir Durrani, Fahim Dalvi, and Hassan Sajjad. Linguistic correlation analysis: Discovering salient neurons in deepnlp models. arXiv preprint arXiv:2206.13288, 2022. — [arXiv:2206.13288](https://arxiv.org/abs/2206.13288) [find](https://scholar.google.com/scholar?q=Nadir+Durrani%2C+Fahim+Dalvi%2C+and+Hassan+Sajjad%2E+Linguistic+correlation+analysis%3A+Discovering+salient+neurons+in+deepnlp+models%2E+arXiv+preprint+arXiv%3A2206%2E13288%2C+2022%2E+%E2%80%94+%5BarXiv%3A2206%2E13288%5D%28https%3A%2F%2Farxiv)
40. Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. [Localizing model behavior with path patching](https://md.research.elamur.ai/free/en/papers/goldowsky-dill-2023-localizing-model-behavior-path-patching/). arXiv preprint arXiv:2304.05969, 2023.
41. N Elhage, N Nanda, C Olsson, T Henighan, N Joseph, B Mann, A Askell, Y Bai, A Chen, T Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. [find](https://scholar.google.com/scholar?q=N+Elhage%2C+N+Nanda%2C+C+Olsson%2C+T+Henighan%2C+N+Joseph%2C+B+Mann%2C+A+Askell%2C+Y+Bai%2C+A+Chen%2C+T+Conerly%2C+et+al%2E+A+mathematical+framework+for+transformer+circuits%2E+Transformer+Circuits+Thread%2C+2021%2E)
42. Stephen Casper, Tilman Rauker, Anson Ho, and Dylan Hadfield-Menell. Sok: Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. In First IEEE Conference on Secure and Trustworthy Machine Learning. [find](https://scholar.google.com/scholar?q=Stephen+Casper%2C+Tilman+Rauker%2C+Anson+Ho%2C+and+Dylan+Hadfield%2DMenell%2E+Sok%3A+Toward+transparent+ai%3A+A+survey+on+interpreting+the+inner+structures+of+deep+neural+networks%2E+In+First+IEEE+Conference+on+Secur)
43. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amode… — [In-context Learning and Induction Heads](https://md.research.elamur.ai/free/en/papers/olsson-2022-context-learning-induction-heads/)
44. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022. [find](https://scholar.google.com/scholar?q=Kevin+Wang%2C+Alexandre+Variengien%2C+Arthur+Conmy%2C+Buck+Shlegeris%2C+and+Jacob+Steinhardt%2E+Interpretability+in+the+wild%3A+a+circuit+for+indirect+object+identification+in+gpt%2D2+small%2E+arXiv+preprint+arXiv%3A22)
45. Neel Nanda, Lawrence Chan, Tom Liberum, Jess Smith, and Jacob Steinhardt. [Progress measures for grokking via mechanistic interpretability](https://md.research.elamur.ai/free/en/papers/nanda-2023-progress-measures-grokking-mechanistic-interpretability/). arXiv preprint arXiv:2301.05217, 2023.
46. Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations. arXiv preprint arXiv:2302.03025, 2023. — [arXiv:2302.03025](https://arxiv.org/abs/2302.03025) [find](https://scholar.google.com/scholar?q=Bilal+Chughtai%2C+Lawrence+Chan%2C+and+Neel+Nanda%2E+A+toy+model+of+universality%3A+Reverse+engineering+how+networks+learn+group+operations%2E+arXiv+preprint+arXiv%3A2302%2E03025%2C+2023%2E+%E2%80%94+%5BarXiv%3A2302%2E03025%5D%28https%3A%2F)
47. Richard Ngo, Lawrence Chan, and Sören Mindermann. [The alignment problem from a deep learning perspective](https://md.research.elamur.ai/free/en/papers/ngo-2022-alignment-problem-deep-learning-perspective/), 2023.
48. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913, 2020. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Roei+Schuster%2C+Jonathan+Berant%2C+and+Omer+Levy%2E+Transformer+feed%2Dforward+layers+are+key%2Dvalue+memories%2E+arXiv+preprint+arXiv%3A2012%2E14913%2C+2020%2E)
49. Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. [Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space](https://md.research.elamur.ai/free/en/papers/geva-2022-transformer-feed-forward-layers-build/). arXiv preprint arXiv:2203.14680, 2022.
50. Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. [Analyzing transformers in embedding space](https://md.research.elamur.ai/free/en/papers/dar-2023-analyzing-transformers-embedding-space/). arXiv preprint arXiv:2209.02535, 2022.
51. Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. [Linear algebraic structure of word senses, with applications to polysemy](https://md.research.elamur.ai/free/en/papers/arora-2018-linear-algebraic-structure-word-senses/). Transactions of the Association for Computational Linguistics, 6:483–495, 2018.
52. Adam S Jermyn, Nicholas Schiefer, and Evan Hubinger. Engineering monosemanticity in toy models. arXiv preprint arXiv:2211.09169, 2022. — [arXiv:2211.09169](https://arxiv.org/abs/2211.09169) [find](https://scholar.google.com/scholar?q=Adam+S+Jermyn%2C+Nicholas+Schiefer%2C+and+Evan+Hubinger%2E+Engineering+monosemanticity+in+toy+models%2E+arXiv+preprint+arXiv%3A2211%2E09169%2C+2022%2E+%E2%80%94+%5BarXiv%3A2211%2E09169%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2211%2E09169%29)
53. Lee Sharkey, Dan Braun, and Beren Millidge. Taking features out of superposition with sparse autoencoders, 2022. [https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj](https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj). [↗](https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj)
54. David L Donoho. Compressed sensing. IEEE Transactions on information theory, 52(4):1289–1306, 2006. [find](https://scholar.google.com/scholar?q=David+L+Donoho%2E+Compressed+sensing%2E+IEEE+Transactions+on+information+theory%2C+52%284%29%3A1289%E2%80%931306%2C+2006%2E)
55. Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016. [find](https://scholar.google.com/scholar?q=Xi+Chen%2C+Yan+Duan%2C+Rein+Houthooft%2C+John+Schulman%2C+Ilya+Sutskever%2C+and+Pieter+Abbeel%2E+Infogan%3A+Interpretable+representation+learning+by+information+maximizing+generative+adversarial+nets%2E+Advances+in+n)
56. Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pages 2649–2658. PMLR, 2018. [find](https://scholar.google.com/scholar?q=Hyunjik+Kim+and+Andriy+Mnih%2E+Disentangling+by+factorising%2E+In+International+Conference+on+Machine+Learning%2C+pages+2649%E2%80%932658%2E+PMLR%2C+2018%2E)
57. Fred Rieke, David Warland, Rob de Ruyter Van Steveninck, and William Bialek. Spikes: exploring the neural code. MIT press, 1999. [find](https://scholar.google.com/scholar?q=Fred+Rieke%2C+David+Warland%2C+Rob+de+Ruyter+Van+Steveninck%2C+and+William+Bialek%2E+Spikes%3A+exploring+the+neural+code%2E+MIT+press%2C+1999%2E)
58. Bruno A Olshausen and David J Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23):3311–3325, 1997. [find](https://scholar.google.com/scholar?q=Bruno+A+Olshausen+and+David+J+Field%2E+Sparse+coding+with+an+overcomplete+basis+set%3A+A+strategy+employed+by+v1%3F+Vision+research%2C+37%2823%29%3A3311%E2%80%933325%2C+1997%2E)
59. Horace Barlow. Redundancy reduction revisited. Network: computation in neural systems, 12(3):241, 2001. [find](https://scholar.google.com/scholar?q=Horace+Barlow%2E+Redundancy+reduction+revisited%2E+Network%3A+computation+in+neural+systems%2C+12%283%29%3A241%2C+2001%2E)
60. R Quian Quiroga, Leila Reddy, Gabriel Kreiman, Christof Koch, and Itzhak Fried. Invariant visual representation by single neurons in the human brain. Nature, 435(7045):1102–1107, 2005. [find](https://scholar.google.com/scholar?q=R+Quian+Quiroga%2C+Leila+Reddy%2C+Gabriel+Kreiman%2C+Christof+Koch%2C+and+Itzhak+Fried%2E+Invariant+visual+representation+by+single+neurons+in+the+human+brain%2E+Nature%2C+435%287045%29%3A1102%E2%80%931107%2C+2005%2E)
61. William E Vinje and Jack L Gallant. Sparse coding and decorrelation in primary visual cortex during natural vision. Science, 287(5456):1273–1276, 2000. [find](https://scholar.google.com/scholar?q=William+E+Vinje+and+Jack+L+Gallant%2E+Sparse+coding+and+decorrelation+in+primary+visual+cortex+during+natural+vision%2E+Science%2C+287%285456%29%3A1273%E2%80%931276%2C+2000%2E)
62. Alexandre Pouget, Peter Dayan, and Richard Zemel. Information processing with population codes. Nature Reviews Neuroscience, 1(2):125–132, 2000. [find](https://scholar.google.com/scholar?q=Alexandre+Pouget%2C+Peter+Dayan%2C+and+Richard+Zemel%2E+Information+processing+with+population+codes%2E+Nature+Reviews+Neuroscience%2C+1%282%29%3A125%E2%80%93132%2C+2000%2E)
63. Stefano Panzeri, Jakob H Macke, Joachim Gross, and Christoph Kayser. Neural population coding: combining insights from microscopic and mass signals. Trends in cognitive sciences, 19(3):162–172, 2015. [find](https://scholar.google.com/scholar?q=Stefano+Panzeri%2C+Jakob+H+Macke%2C+Joachim+Gross%2C+and+Christoph+Kayser%2E+Neural+population+coding%3A+combining+insights+from+microscopic+and+mass+signals%2E+Trends+in+cognitive+sciences%2C+19%283%29%3A162%E2%80%93172%2C+2015%2E)
64. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+Ilya+Sutskever%2C+et+al%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+2018%2E)
65. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712, 2023.
66. Nelson Elhage, Robert Lasenby, and Christopher Olah. Privileged bases in the transformer residual stream. Transformer Circuits Thread, 2023. [https://transformer-circuits.pub/2023/privileged-basis/index.html](https://transformer-circuits.pub/2023/privileged-basis/index.html). [↗](https://transformer-circuits.pub/2023/privileged-basis/index.html)
67. Andreas M Tillmann, Daniel Bienstock, Andrea Lodi, and Alexandra Schwartz. Cardinality minimization, constraints, and regularization: a survey. arXiv preprint arXiv:2106.09606, 2021. — [arXiv:2106.09606](https://arxiv.org/abs/2106.09606) [find](https://scholar.google.com/scholar?q=Andreas+M+Tillmann%2C+Daniel+Bienstock%2C+Andrea+Lodi%2C+and+Alexandra+Schwartz%2E+Cardinality+minimization%2C+constraints%2C+and+regularization%3A+a+survey%2E+arXiv+preprint+arXiv%3A2106%2E09606%2C+2021%2E+%E2%80%94+%5BarXiv%3A2106%2E096)
68. Brian C Ross. Mutual information between discrete and continuous data sets. PloS one, 9(2):e87357, 2014. [find](https://scholar.google.com/scholar?q=Brian+C+Ross%2E+Mutual+information+between+discrete+and+continuous+data+sets%2E+PloS+one%2C+9%282%29%3Ae87357%2C+2014%2E)
69. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372, 2022. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+Andonian%2C+and+Yonatan+Belinkov%2E+Locating+and+editing+factual+associations+in+gpt%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A17359%E2%80%9317372%2C+2022%2E)
70. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+Aviya+Skowron%2C+Lintang+Sutaw)
71. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+et+al%2E+The+pile%3A+An+800gb+dataset+of+diverse+text+for+language+)
72. Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Viégas, and Martin Wattenberg. An interpretability illusion for bert. arXiv preprint arXiv:2104.07143, 2021. — [arXiv:2104.07143](https://arxiv.org/abs/2104.07143) [find](https://scholar.google.com/scholar?q=Tolga+Bolukbasi%2C+Adam+Pearce%2C+Ann+Yuan%2C+Andy+Coenen%2C+Emily+Reif%2C+Fernanda+Vi%C3%A9gas%2C+and+Martin+Wattenberg%2E+An+interpretability+illusion+for+bert%2E+arXiv+preprint+arXiv%3A2104%2E07143%2C+2021%2E+%E2%80%94+%5BarXiv%3A2104%2E071)
73. Neel Nanda. Neuroscope: A website for mechanistic interpretability of language models, 2022. [find](https://scholar.google.com/scholar?q=Neel+Nanda%2E+Neuroscope%3A+A+website+for+mechanistic+interpretability+of+language+models%2C+2022%2E)
74. Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607–609, 1996. [find](https://scholar.google.com/scholar?q=Bruno+A+Olshausen+and+David+J+Field%2E+Emergence+of+simple%2Dcell+receptive+field+properties+by+learning+a+sparse+code+for+natural+images%2E+Nature%2C+381%286583%29%3A607%E2%80%93609%2C+1996%2E)
75. Tomáš Hromádka, Michael R DeWeese, and Anthony M Zador. Sparse representation of sounds in the unanesthetized auditory cortex. PLoS biology, 6(1):e16, 2008. [find](https://scholar.google.com/scholar?q=Tom%C3%A1%C5%A1+Hrom%C3%A1dka%2C+Michael+R+DeWeese%2C+and+Anthony+M+Zador%2E+Sparse+representation+of+sounds+in+the+unanesthetized+auditory+cortex%2E+PLoS+biology%2C+6%281%29%3Ae16%2C+2008%2E)
76. Brett A Johnson and Michael Leon. Modular representations of odorants in the glomerular layer of the rat olfactory bulb and the effects of stimulus concentration. Journal of Comparative Neurology, 422(4):496–509, 2000. [find](https://scholar.google.com/scholar?q=Brett+A+Johnson+and+Michael+Leon%2E+Modular+representations+of+odorants+in+the+glomerular+layer+of+the+rat+olfactory+bulb+and+the+effects+of+stimulus+concentration%2E+Journal+of+Comparative+Neurology%2C+422)
77. Stefan Leutgeb, Jill K Leutgeb, Carol A Barnes, Edvard I Moser, Bruce L McNaughton, and May-Britt Moser. Independent codes for spatial and episodic memory in hippocampal neuronal ensembles. Science, 309(5734):619–623, 2005. [find](https://scholar.google.com/scholar?q=Stefan+Leutgeb%2C+Jill+K+Leutgeb%2C+Carol+A+Barnes%2C+Edvard+I+Moser%2C+Bruce+L+McNaughton%2C+and+May%2DBritt+Moser%2E+Independent+codes+for+spatial+and+episodic+memory+in+hippocampal+neuronal+ensembles%2E+Science%2C+3)
78. Philipp Koehn. Europarl: A parallel corpus for statistical machine translation. In Proceedings of machine translation summit x: papers, pages 79–86, 2005. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2E+Europarl%3A+A+parallel+corpus+for+statistical+machine+translation%2E+In+Proceedings+of+machine+translation+summit+x%3A+papers%2C+pages+79%E2%80%9386%2C+2005%2E)
79. Eric J Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. [The quantization model of neural scaling](https://md.research.elamur.ai/free/en/papers/michaud-2023-quantization-model-neural-scaling/). arXiv preprint arXiv:2303.13506, 2023.
80. Lawrence Chan, Leon Lang, and Erik Jenner. Natural abstractions: Key claims, theorems, and critiques, 2022. [https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj](https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj). [↗](https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj)
81. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). arXiv preprint arXiv:2206.07682, 2022.
82. Joseph Carlsmith. Is power-seeking ai an existential risk? arXiv preprint arXiv:2206.13353, 2022. — [arXiv:2206.13353](https://arxiv.org/abs/2206.13353) [find](https://scholar.google.com/scholar?q=Joseph+Carlsmith%2E+Is+power%2Dseeking+ai+an+existential+risk%3F+arXiv+preprint+arXiv%3A2206%2E13353%2C+2022%2E+%E2%80%94+%5BarXiv%3A2206%2E13353%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2206%2E13353%29)
83. Leo Z Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A Smith. [Probing across time: What does roberta know and when?](https://md.research.elamur.ai/free/en/papers/liu-2021-probing-across-time-what-does/) arXiv preprint arXiv:2104.07885, 2021.
84. Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. arXiv preprint arXiv:2212.03827, 2022. [find](https://scholar.google.com/scholar?q=Collin+Burns%2C+Haotian+Ye%2C+Dan+Klein%2C+and+Jacob+Steinhardt%2E+Discovering+latent+knowledge+in+language+models+without+supervision%2E+arXiv+preprint+arXiv%3A2212%2E03827%2C+2022%2E)
85. Neel Nanda. Transformerlens, 2022. [find](https://scholar.google.com/scholar?q=Neel+Nanda%2E+Transformerlens%2C+2022%2E)
86. Albert Reuther, Jeremy Kepner, Chansup Byun, Siddharth Samsi, William Arcand, David Bestor, Bill Bergeron, Vijay Gadepally, Michael Houle, Matthew Hubbell, Michael Jones, Anna Klein, Lauren Milechin, Julia Mullen, Andrew Prout, Antonio Rosa, Charles Yee, and Peter Michaleas. Interactive supercomputi… [find](https://scholar.google.com/scholar?q=Albert+Reuther%2C+Jeremy+Kepner%2C+Chansup+Byun%2C+Siddharth+Samsi%2C+William+Arcand%2C+David+Bestor%2C+Bill+Bergeron%2C+Vijay+Gadepally%2C+Michael+Houle%2C+Matthew+Hubbell%2C+Michael+Jones%2C+Anna+Klein%2C+Lauren+Milechin%2C+)
87. Uri Alon. Design principles of biological circuits. Biophysical Journal, 96(3):15a–15a, 2009. [find](https://scholar.google.com/scholar?q=Uri+Alon%2E+Design+principles+of+biological+circuits%2E+Biophysical+Journal%2C+96%283%29%3A15a%E2%80%9315a%2C+2009%2E)
88. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
89. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311, 2022.
90. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023.
91. Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018. [find](https://scholar.google.com/scholar?q=Julius+Adebayo%2C+Justin+Gilmer%2C+Michael+Muelly%2C+Ian+Goodfellow%2C+Moritz+Hardt%2C+and+Been+Kim%2E+Sanity+checks+for+saliency+maps%2E+Advances+in+neural+information+processing+systems%2C+31%2C+2018%2E)
92. Blair Bilodeau, Natasha Jaques, Pang Wei Koh, and Been Kim. Impossibility theorems for feature attribution. arXiv preprint arXiv:2212.11870, 2022. — [arXiv:2212.11870](https://arxiv.org/abs/2212.11870) [find](https://scholar.google.com/scholar?q=Blair+Bilodeau%2C+Natasha+Jaques%2C+Pang+Wei+Koh%2C+and+Been+Kim%2E+Impossibility+theorems+for+feature+attribution%2E+arXiv+preprint+arXiv%3A2212%2E11870%2C+2022%2E+%E2%80%94+%5BarXiv%3A2212%2E11870%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2212%2E11870)
93. Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. arXiv preprint arXiv:2301.04213, 2023. [find](https://scholar.google.com/scholar?q=Peter+Hase%2C+Mohit+Bansal%2C+Been+Kim%2C+and+Asma+Ghandeharioun%2E+Does+localization+inform+editing%3F+surprising+differences+in+causality%2Dbased+localization+vs%2E+knowledge+editing+in+language+models%2E+arXiv+pre)
94. Amir-Hossein Karimi, Krikamol Muandet, Simon Kornblith, Bernhard Schölkopf, and Been Kim. On the relationship between explanation and prediction: A causal view. arXiv preprint arXiv:2212.06925, 2022. — [arXiv:2212.06925](https://arxiv.org/abs/2212.06925) [find](https://scholar.google.com/scholar?q=Amir%2DHossein+Karimi%2C+Krikamol+Muandet%2C+Simon+Kornblith%2C+Bernhard+Sch%C3%B6lkopf%2C+and+Been+Kim%2E+On+the+relationship+between+explanation+and+prediction%3A+A+causal+view%2E+arXiv+preprint+arXiv%3A2212%2E06925%2C+2022%2E+)
95. Chris Olah. Interpretability vs neuroscience, 2021. [find](https://scholar.google.com/scholar?q=Chris+Olah%2E+Interpretability+vs+neuroscience%2C+2021%2E)
96. Chris Olah. Distributed representations: Composition & superposition, circuits.pub/2023/superposition-composition/index.html. 2023. https://transformer- [find](https://scholar.google.com/scholar?q=Chris+Olah%2E+Distributed+representations%3A+Composition+%26+superposition%2C+circuits%2Epub%2F2023%2Fsuperposition%2Dcomposition%2Findex%2Ehtml%2E+2023%2E+https%3A%2F%2Ftransformer%2D)
97. Sid Black, Lee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Merizian, Kip Parker, Carlos Ramón Guevara, Beren Millidge, Gabriel Alfour, and Connor Leahy. Interpreting neural networks through the polytope lens, 2022. [find](https://scholar.google.com/scholar?q=Sid+Black%2C+Lee+Sharkey%2C+Leo+Grinsztajn%2C+Eric+Winsor%2C+Dan+Braun%2C+Jacob+Merizian%2C+Kip+Parker%2C+Carlos+Ram%C3%B3n+Guevara%2C+Beren+Millidge%2C+Gabriel+Alfour%2C+and+Connor+Leahy%2E+Interpreting+neural+networks+through)
98. Anh Nguyen, Jason Yosinski, and Jeff Clune. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks, 2016. [find](https://scholar.google.com/scholar?q=Anh+Nguyen%2C+Jason+Yosinski%2C+and+Jeff+Clune%2E+Multifaceted+feature+visualization%3A+Uncovering+the+different+types+of+features+learned+by+each+neuron+in+deep+neural+networks%2C+2016%2E)
99. Neel Nanda. A comprehensive mechanistic interpretability explainer & glossary, 2023. [find](https://scholar.google.com/scholar?q=Neel+Nanda%2E+A+comprehensive+mechanistic+interpretability+explainer+%26+glossary%2C+2023%2E)
100. Tom Henighan, Shan Carter, Tristan Hume, Nelson Elhage, Robert Lasenby, Stanislav Fort, Nicholas Schiefer, and Christopher Olah. Superposition, memorization, and double descent. Transformer Circuits Thread, 2023. [https://transformer-circuits.pub/2023/toy-double-descent/index.html](https://transformer-circuits.pub/2023/toy-double-descent/index.html). [↗](https://transformer-circuits.pub/2023/toy-double-descent/index.html)
101. Adam Jermyn, Chris Olah, and Tom Henighan. Circuits updates — may 2023: Attention head superposition, 2023. [https://transformer-circuits.pub/2023/may-update/index.html\#attention-superposition](https://transformer-circuits.pub/2023/may-update/index.html\#attention-superposition). [↗](https://transformer-circuits.pub/2023/may-update/index.html\#attention-superposition)
102. Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. arXiv preprint arXiv:2210.13382, 2022. [find](https://scholar.google.com/scholar?q=Kenneth+Li%2C+Aspen+K+Hopkins%2C+David+Bau%2C+Fernanda+Vi%C3%A9gas%2C+Hanspeter+Pfister%2C+and+Martin+Wattenberg%2E+Emergent+world+representations%3A+Exploring+a+sequence+model+trained+on+a+synthetic+task%2E+arXiv+preprin)
103. Neel Nanda. Actually, othello-gpt has a linear emergent world model, Mar 2023. [find](https://scholar.google.com/scholar?q=Neel+Nanda%2E+Actually%2C+othello%2Dgpt+has+a+linear+emergent+world+model%2C+Mar+2023%2E)
104. Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Viégas, and Martin Wattenberg. [Visualizing and measuring the geometry of bert](https://md.research.elamur.ai/free/en/papers/coenen-2019-visualizing-measuring-geometry-bert/), 2019.
105. Rajiv Mehrotra, Kameswara Rao Namuduri, and Nagarajan Ranganathan. Gabor filter-based edge detection. Pattern recognition, 25(12):1479–1494, 1992. [find](https://scholar.google.com/scholar?q=Rajiv+Mehrotra%2C+Kameswara+Rao+Namuduri%2C+and+Nagarajan+Ranganathan%2E+Gabor+filter%2Dbased+edge+detection%2E+Pattern+recognition%2C+25%2812%29%3A1479%E2%80%931494%2C+1992%2E)
106. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. An overview of early vision in inceptionv1. Distill, 2020. [https://distill.pub/2020/circuits/early-vision](https://distill.pub/2020/circuits/early-vision). [↗](https://distill.pub/2020/circuits/early-vision)
107. Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks+and+Kevin+Gimpel%2E+Gaussian+error+linear+units+%28gelus%29%2E+arXiv+preprint+arXiv%3A1606%2E08415%2C+2016%2E)
108. Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. Analyzing redundancy in pretrained transformer models. arXiv preprint arXiv:2004.04010, 2020. [find](https://scholar.google.com/scholar?q=Fahim+Dalvi%2C+Hassan+Sajjad%2C+Nadir+Durrani%2C+and+Yonatan+Belinkov%2E+Analyzing+redundancy+in+pretrained+transformer+models%2E+arXiv+preprint+arXiv%3A2004%2E04010%2C+2020%2E)

*Updated 2 November 2026*