---
title: "Constructing Interpretable Features from Compositional Neuron"
description: "Mechanistic interpretability needs units of analysis that causally explain what an LLM outputs. 69 references."
authors: ["Or David Shafran", "Atticus Geiger", "Mor Geva"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1959/"
doi: "10.18653/v1/2026.acl-long.1959"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shafran-2026-constructing-interpretable-features-compositional-neuron/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shafran-2026-constructing-interpretable-features-compositional-neuron/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/11/) / Constructing Interpretable Features from Compositional Neuron Groups

# Constructing Interpretable Features from Compositional Neuron Groups

**Authors:** Or David Shafran, [Atticus Geiger](https://md.research.elamur.ai/free/en/authors/atticus-geiger/) and [Mor Geva](https://md.research.elamur.ai/free/en/authors/mor-geva/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1959/) · [PDF](https://aclanthology.org/2026.acl-long.1959.pdf) · [DOI 10.18653/v1/2026.acl-long.1959](https://doi.org/10.18653/v1/2026.acl-long.1959) · [OpenAlex](https://openalex.org/W4417357092)

**Citations:** 1 citations (OpenAlex, 2026-10-14)

## Summary
Mechanistic interpretability needs units of analysis that causally explain what an LLM outputs. Individual neurons tend to mix several concepts. Sparse autoencoders (SAEs), usually trained on the residual stream, learn directions from scratch that are not linked to the model's own computations, and they often underperform in causal tests.

The authors instead factorize MLP activations, collected over a large amount of text, with semi-nonnegative matrix factorization (SNMF). One factor holds features, each a sparse mix of neurons that fire together. The other is a nonnegative coefficient matrix that ties each feature to the tokens driving it, which makes labelling features easy.

Features were tested on Llama 3.1, Gemma 2 and GPT-2 for concept detection and concept steering. SNMF features match SAEs on detection. On steering, they beat Gemmascope, Llamascope and a supervised difference-in-means baseline. Further analysis shows that related features reuse shared neuron groups, which forms a hierarchy.

This gives a simple, mechanism-grounded tool for finding interpretable features. It also suggests that SAE feature splitting reflects how the model itself composes neurons.

## Key points
- **Factorization setup:** the MLP activation matrix is approximated as Z·Y. Z holds the neuron-combination features, and Y is a nonnegative token-to-feature coefficient matrix. Projecting a feature through the MLP output weights gives its residual-stream direction.
- **Evaluation:** features were compared against Gemmascope, Llamascope and DiffMeans on detection and steering. An LLM judge scored the outputs, and a human check on 50 examples showed strong agreement (Spearman ρ = 0.8).
- **Recursive SNMF:** applying SNMF again to the learned features merges fine-grained concepts into broader ones, for example JavaScript and Python into programming. This is the reverse of SAE feature splitting.
- **Weekday features in GPT-2:** these share a core set of neurons. With the prompt "I think that" in GPT-2 Large, amplifying the core raises the logits of all weekday tokens. Amplifying one day's exclusive neurons boosts that day and suppresses the others.
- **Limitations:** experiments used fewer than 500 features and optimization by multiplicative updates. Code is available at github.com/ordavid-s/snmf-mlp-decomposition.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tel Aviv University](https://md.research.elamur.ai/free/en/universities/tel-aviv-university/) — University, Tel Aviv, Israel · 82 works · 10,655 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

## Cite

Cite as: Or David Shafran, Atticus Geiger and Mor Geva. 2026. Constructing Interpretable Features from Compositional Neuron Groups. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shafran2026constructing,
  title = {{Constructing Interpretable Features from Compositional Neuron Groups}},
  author = {Or David Shafran, Atticus Geiger and Mor Geva},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1959},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1959}
}
```

- [← previous Constructing coherent spatial memory in LLM agents through graph rectification](https://md.research.elamur.ai/free/en/papers/zhang-2026-constructing-coherent-spatial-memory-llm/)
- [next → Context Volume Drives Performance: Tackling Domain Shift in Extremely Low-Resource Translation via RAG](https://md.research.elamur.ai/free/en/papers/setiawan-2026-context-volume-drives-performance-tackling/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/11/)

## References

69 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. 2023. Language models can explain neurons in language models. [https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html](https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html). [↗](https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html)
2. Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Viégas, and Martin Wattenberg. 2021. An interpretability illusion for bert. arXiv preprint arXiv:2104.07143. — [arXiv:2104.07143](https://arxiv.org/abs/2104.07143) [find](https://scholar.google.com/scholar?q=Tolga+Bolukbasi%2C+Adam+Pearce%2C+Ann+Yuan%2C+Andy+Coenen%2C+Emily+Reif%2C+Fernanda+Vi%C3%A9gas%2C+and+Martin+Wattenberg%2E+2021%2E+An+interpretability+illusion+for+bert%2E+arXiv+preprint+arXiv%3A2104%2E07143%2E+%E2%80%94+%5BarXiv%3A2104%2E071)
3. C. Boutsidis and E. Gallopoulos. 2008. Svd based initialization: A head start for nonnegative matrix factorization. Pattern Recognition, 41(4):1350–1362. [find](https://scholar.google.com/scholar?q=C%2E+Boutsidis+and+E%2E+Gallopoulos%2E+2008%2E+Svd+based+initialization%3A+A+head+start+for+nonnegative+matrix+factorization%2E+Pattern+Recognition%2C+41%284%29%3A1350%E2%80%931362%2E)
4. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah… [find](https://scholar.google.com/scholar?q=Trenton+Bricken%2C+Adly+Templeton%2C+Joshua+Batson%2C+Brian+Chen%2C+Adam+Jermyn%2C+Tom+Conerly%2C+Nick+Turner%2C+Cem+Anil%2C+Carson+Denison%2C+Amanda+Askell%2C+Robert+Lasenby%2C+Yifan+Wu%2C+Shauna+Kravec%2C+Nicholas+Schiefer%2C+)
5. Tue Minh Cao, Nhat Hoang-Xuan, Hieu Pham, Phi Le Nguyen, and My T. Thai. 2025. Neurflow: Interpreting neural networks through neuron groups and functional interactions. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tue+Minh+Cao%2C+Nhat+Hoang%2DXuan%2C+Hieu+Pham%2C+Phi+Le+Nguyen%2C+and+My+T%2E+Thai%2E+2025%2E+Neurflow%3A+Interpreting+neural+networks+through+neuron+groups+and+functional+interactions%2E+In+The+Thirteenth+International)
6. David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, and Joseph Isaac Bloom. 2024. A is for absorption: Studying feature splitting and absorption in sparse autoencoders. In Interpretable AI: Past, Present and Future. [find](https://scholar.google.com/scholar?q=David+Chanin%2C+James+Wilken%2DSmith%2C+Tom%C3%A1%C5%A1+Dulka%2C+Hardik+Bhatnagar%2C+and+Joseph+Isaac+Bloom%2E+2024%2E+A+is+for+absorption%3A+Studying+feature+splitting+and+absorption+in+sparse+autoencoders%2E+In+Interpretable+A)
7. Maheep Chaudhary and Atticus Geiger. 2024. Evaluating open-source sparse autoencoders on disentangling factual knowledge in gpt-2 small. arXiv preprint arXiv:2409.04478. — [arXiv:2409.04478](https://arxiv.org/abs/2409.04478) [find](https://scholar.google.com/scholar?q=Maheep+Chaudhary+and+Atticus+Geiger%2E+2024%2E+Evaluating+open%2Dsource+sparse+autoencoders+on+disentangling+factual+knowledge+in+gpt%2D2+small%2E+arXiv+preprint+arXiv%3A2409%2E04478%2E+%E2%80%94+%5BarXiv%3A2409%2E04478%5D%28https%3A%2F%2Fa)
8. Dami Choi, Vincent Huang, Kevin Meng, Daniel D Johnson, Jacob Steinhardt, and Sarah Schwettmann. 2024. Scaling automatic neuron description. [https://transluce.org/neuron-descriptions](https://transluce.org/neuron-descriptions). [↗](https://transluce.org/neuron-descriptions)
9. Róbert Csordás, Christopher Potts, Christopher D Manning, and Atticus Geiger. 2024. Recurrent neural networks learn to store and generate sequences using non-linear representations. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 248–262, Mia… [find](https://scholar.google.com/scholar?q=R%C3%B3bert+Csord%C3%A1s%2C+Christopher+Potts%2C+Christopher+D+Manning%2C+and+Atticus+Geiger%2E+2024%2E+Recurrent+neural+networks+learn+to+store+and+generate+sequences+using+non%2Dlinear+representations%2E+In+Proceedings+of+)
10. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable features in language models. Preprint, arXiv:2309.08600. [find](https://scholar.google.com/scholar?q=Hoagy+Cunningham%2C+Aidan+Ewart%2C+Logan+Riggs%2C+Robert+Huben%2C+and+Lee+Sharkey%2E+2023%2E+Sparse+autoencoders+find+highly+interpretable+features+in+language+models%2E+Preprint%2C+arXiv%3A2309%2E08600%2E)
11. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493– 8502, Dublin, Ireland. Association for Computatio… [find](https://scholar.google.com/scholar?q=Damai+Dai%2C+Li+Dong%2C+Yaru+Hao%2C+Zhifang+Sui%2C+Baobao+Chang%2C+and+Furu+Wei%2E+2022%2E+Knowledge+neurons+in+pretrained+transformers%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computationa)
12. Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James R. Glass. 2018. What is one grain of sand in the desert? analyzing individual neurons in deep NLP models. CoRR, abs/1812.09355. — [arXiv:1812.09355](https://arxiv.org/abs/1812.09355) [find](https://scholar.google.com/scholar?q=Fahim+Dalvi%2C+Nadir+Durrani%2C+Hassan+Sajjad%2C+Yonatan+Belinkov%2C+Anthony+Bau%2C+and+James+R%2E+Glass%2E+2018%2E+What+is+one+grain+of+sand+in+the+desert%3F+analyzing+individual+neurons+in+deep+NLP+models%2E+CoRR%2C+abs%2F)
13. Chris Ding, Tao Li, and Michael Jordan. 2010. Convex and semi-nonnegative matrix factorizations. ieee trans pattern anal mach intell. IEEE transactions on pattern analysis and machine intelligence, 32:45–55. [find](https://scholar.google.com/scholar?q=Chris+Ding%2C+Tao+Li%2C+and+Michael+Jordan%2E+2010%2E+Convex+and+semi%2Dnonnegative+matrix+factorizations%2E+ieee+trans+pattern+anal+mach+intell%2E+IEEE+transactions+on+pattern+analysis+and+machine+intelligence%2C+32)
14. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Ja… [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Neel+Nanda%2C+Catherine+Olsson%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Zac+Hatfield%2DDodds%2C+Da)
15. Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. 2025. Not all language model features are one-dimensionally linear. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Joshua+Engels%2C+Eric+J+Michaud%2C+Isaac+Liao%2C+Wes+Gurnee%2C+and+Max+Tegmark%2E+2025%2E+Not+all+language+model+features+are+one%2Ddimensionally+linear%2E+In+The+Thirteenth+International+Conference+on+Learning+Repre)
16. Thomas Fel, Agustin Picard, Louis Béthune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. 2023. Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 271… [find](https://scholar.google.com/scholar?q=Thomas+Fel%2C+Agustin+Picard%2C+Louis+B%C3%A9thune%2C+Thibaut+Boissin%2C+David+Vigouroux%2C+Julien+Colin%2C+R%C3%A9mi+Cad%C3%A8ne%2C+and+Thomas+Serre%2E+2023%2E+Craft%3A+Concept+recursive+activation+factorization+for+explainability%2E+In)
17. Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2025. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Tom+Dupre+la+Tour%2C+Henk+Tillman%2C+Gabriel+Goh%2C+Rajan+Troll%2C+Alec+Radford%2C+Ilya+Sutskever%2C+Jan+Leike%2C+and+Jeffrey+Wu%2E+2025%2E+Scaling+and+evaluating+sparse+autoencoders%2E+In+The+Thirteenth+Interna)
18. Atticus Geiger, Duligur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Icard. 2024. Causal abstraction: A theoretical foundation for mechanistic interpretability. Preprint, arXiv:2301.04709. [find](https://scholar.google.com/scholar?q=Atticus+Geiger%2C+Duligur+Ibeling%2C+Amir+Zur%2C+Maheep+Chaudhary%2C+Sonakshi+Chauhan%2C+Jing+Huang%2C+Aryaman+Arora%2C+Zhengxuan+Wu%2C+Noah+Goodman%2C+Christopher+Potts%2C+and+Thomas+Icard%2E+2024%2E+Causal+abstraction%3A+A+t)
19. Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D. Goodman. 2023. Finding alignments between interpretable causal variables and distributed neural representations. Ms., Stanford University. [find](https://scholar.google.com/scholar?q=Atticus+Geiger%2C+Zhengxuan+Wu%2C+Christopher+Potts%2C+Thomas+Icard%2C+and+Noah+D%2E+Goodman%2E+2023%2E+Finding+alignments+between+interpretable+causal+variables+and+distributed+neural+representations%2E+Ms%2E%2C+Stanfor)
20. Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. [Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space](https://md.research.elamur.ai/free/en/papers/geva-2022-transformer-feed-forward-layers-build/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30–45, Abu Dhabi, United Arab Emirates. Asso…
21. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational… [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Roei+Schuster%2C+Jonathan+Berant%2C+and+Omer+Levy%2E+2021%2E+Transformer+feed%2Dforward+layers+are+key%2Dvalue+memories%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Pr)
22. Google. 2025. Gemini 2.0 Flash. [https://cloud.google.com/vertex-ai/generative-ai/docs/](https://cloud.google.com/vertex-ai/generative-ai/docs/) models/gemini/2-0-flash. [↗](https://cloud.google.com/vertex-ai/generative-ai/docs/)
23. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao,… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
24. Yoav Gur-Arieh, Roy Mayan, Chen Agassy, Atticus Geiger, and Mor Geva. 2025. [Enhancing automated interpretability with output-centric feature descriptions](https://md.research.elamur.ai/free/en/papers/arieh-2025-enhancing-automated-interpretability-output-centric/). arXiv preprint arXiv:2501.08319.
25. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. 2023. Finding neurons in a haystack: Case studies with sparse probing. Transactions on Machine Learning Research. [find](https://scholar.google.com/scholar?q=Wes+Gurnee%2C+Neel+Nanda%2C+Matthew+Pauly%2C+Katherine+Harvey%2C+Dmitrii+Troitskii%2C+and+Dimitris+Bertsimas%2E+2023%2E+Finding+neurons+in+a+haystack%3A+Case+studies+with+sparse+probing%2E+Transactions+on+Machine+Learn)
26. Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, et al. 2024. Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders. arXiv preprint arXiv:2410.20526. [find](https://scholar.google.com/scholar?q=Zhengfu+He%2C+Wentao+Shu%2C+Xuyang+Ge%2C+Lingjie+Chen%2C+Junxuan+Wang%2C+Yunhua+Zhou%2C+Frances+Liu%2C+Qipeng+Guo%2C+Xuanjing+Huang%2C+Zuxuan+Wu%2C+et+al%2E+2024%2E+Llama+scope%3A+Extracting+millions+of+features+from+llama%2D3%2E1)
27. Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. 2021. Natural language descriptions of deep visual features. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Evan+Hernandez%2C+Sarah+Schwettmann%2C+David+Bau%2C+Teona+Bagashvili%2C+Antonio+Torralba%2C+and+Jacob+Andreas%2E+2021%2E+Natural+language+descriptions+of+deep+visual+features%2E+In+International+Conference+on+Learnin)
28. Jing Huang, Atticus Geiger, Karel D’Oosterlinck, Zhengxuan Wu, and Christopher Potts. 2023. [Rigorously assessing natural language explanations of neurons](https://md.research.elamur.ai/free/en/papers/huang-2023-rigorously-assessing-natural-language-explanations/). In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 317–331, Singapore. Association for Com…
29. Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. 2024. RAVEL: Evaluating interpretability methods on disentangling language model representations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8669–… [find](https://scholar.google.com/scholar?q=Jing+Huang%2C+Zhengxuan+Wu%2C+Christopher+Potts%2C+Mor+Geva%2C+and+Atticus+Geiger%2E+2024%2E+RAVEL%3A+Evaluating+interpretability+methods+on+disentangling+language+model+representations%2E+In+Proceedings+of+the+62nd+)
30. Fanny Jourdan, Agustin Picard, Thomas Fel, Laurent Risser, Jean-Michel Loubes, and Nicholas Asher. 2023. COCKATIEL: COntinuous concept ranKed ATtribution with interpretable ELements for explaining neural net classifiers on NLP. In Findings of the Association for Computational Linguistics: ACL 2023,… [find](https://scholar.google.com/scholar?q=Fanny+Jourdan%2C+Agustin+Picard%2C+Thomas+Fel%2C+Laurent+Risser%2C+Jean%2DMichel+Loubes%2C+and+Nicholas+Asher%2E+2023%2E+COCKATIEL%3A+COntinuous+concept+ranKed+ATtribution+with+interpretable+ELements+for+explaining+neu)
31. Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015. [Visualizing and understanding recurrent networks](https://md.research.elamur.ai/free/en/papers/karpathy-2015-visualizing-understanding-recurrent-networks/). arXiv preprint arXiv:1506.02078.
32. Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. 2019. [The emergence of number and syntax units in LSTM language models](https://md.research.elamur.ai/free/en/papers/lakretz-2019-emergence-number-syntax-units-lstm/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human La…
33. Daniel Lee and H. Seung. 1999. Learning the parts of objects by non-negative matrix factorization. Nature, 401:788–91. [find](https://scholar.google.com/scholar?q=Daniel+Lee+and+H%2E+Seung%2E+1999%2E+Learning+the+parts+of+objects+by+non%2Dnegative+matrix+factorization%2E+Nature%2C+401%3A788%E2%80%9391%2E)
34. Justin Lee, Tuomas Oikarinen, Arjun Chatha, Keng-Chi Chang, Yilan Chen, and Tsui-Wei Weng. 2023. The importance of prompt tuning for automated neuron explanations. Preprint, arXiv:2310.06200. — [arXiv:2310.06200](https://arxiv.org/abs/2310.06200) [find](https://scholar.google.com/scholar?q=Justin+Lee%2C+Tuomas+Oikarinen%2C+Arjun+Chatha%2C+Keng%2DChi+Chang%2C+Yilan+Chen%2C+and+Tsui%2DWei+Weng%2E+2023%2E+The+importance+of+prompt+tuning+for+automated+neuron+explanations%2E+Preprint%2C+arXiv%3A2310%2E06200%2E+%E2%80%94+%5BarXiv)
35. Amit Arnold Levy and Mor Geva. 2025. Language models encode numbers using digit representations in base 10. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 385–3… [find](https://scholar.google.com/scholar?q=Amit+Arnold+Levy+and+Mor+Geva%2E+2025%2E+Language+models+encode+numbers+using+digit+representations+in+base+10%2E+In+Proceedings+of+the+2025+Conference+of+the+Nations+of+the+Americas+Chapter+of+the+Associat)
36. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Janos Kramar, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Inte… [find](https://scholar.google.com/scholar?q=Tom+Lieberum%2C+Senthooran+Rajamanoharan%2C+Arthur+Conmy%2C+Lewis+Smith%2C+Nicolas+Sonnerat%2C+Vikrant+Varma%2C+Janos+Kramar%2C+Anca+Dragan%2C+Rohin+Shah%2C+and+Neel+Nanda%2E+2024%2E+Gemma+scope%3A+Open+sparse+autoencoders+e)
37. Johnny Lin. 2023. Neuronpedia: Interactive reference and tooling for analyzing neural networks. Software available from neuronpedia.org. [find](https://scholar.google.com/scholar?q=Johnny+Lin%2E+2023%2E+Neuronpedia%3A+Interactive+reference+and+tooling+for+analyzing+neural+networks%2E+Software+available+from+neuronpedia%2Eorg%2E)
38. Hanxiao Liu, Zihang Dai, David So, and Quoc V Le. 2021. Pay attention to MLPs. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Hanxiao+Liu%2C+Zihang+Dai%2C+David+So%2C+and+Quoc+V+Le%2E+2021%2E+Pay+attention+to+MLPs%2E+In+Advances+in+Neural+Information+Processing+Systems%2E)
39. Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. 2010. Online learning for matrix factorization and sparse coding. J. Mach. Learn. Res., 11:19–60. [find](https://scholar.google.com/scholar?q=Julien+Mairal%2C+Francis+Bach%2C+Jean+Ponce%2C+and+Guillermo+Sapiro%2E+2010%2E+Online+learning+for+matrix+factorization+and+sparse+coding%2E+J%2E+Mach%2E+Learn%2E+Res%2E%2C+11%3A19%E2%80%9360%2E)
40. Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. 2025. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Samuel+Marks%2C+Can+Rager%2C+Eric+J+Michaud%2C+Yonatan+Belinkov%2C+David+Bau%2C+and+Aaron+Mueller%2E+2025%2E+Sparse+feature+circuits%3A+Discovering+and+editing+interpretable+causal+graphs+in+language+models%2E+In+The+T)
41. Samuel Marks and Max Tegmark. 2024. [The geometry of truth: Emergent linear structure in large language model representations of true/false datasets](https://md.research.elamur.ai/free/en/papers/marks-2023-geometry-truth-emergent-linear-structure/). In First Conference on Language Modeling.
42. Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. 2022. Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8364–8375. [find](https://scholar.google.com/scholar?q=Luke+Melas%2DKyriazi%2C+Christian+Rupprecht%2C+Iro+Laina%2C+and+Andrea+Vedaldi%2E+2022%2E+Deep+spectral+methods%3A+A+surprisingly+strong+baseline+for+unsupervised+semantic+segmentation+and+localization%2E+In+Proceedi)
43. Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+J+Andonian%2C+and+Yonatan+Belinkov%2E+2022%2E+Locating+and+editing+factual+associations+in+GPT%2E+In+Advances+in+Neural+Information+Processing+Systems%2E)
44. Abhinav Menon, Manish Shrivastava, David Krueger, and Ekdeep Singh Lubana. 2025. Analyzing (in)abilities of SAEs via formal languages. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1:… [find](https://scholar.google.com/scholar?q=Abhinav+Menon%2C+Manish+Shrivastava%2C+David+Krueger%2C+and+Ekdeep+Singh+Lubana%2E+2025%2E+Analyzing+%28in%29abilities+of+SAEs+via+formal+languages%2E+In+Proceedings+of+the+2025+Conference+of+the+Nations+of+the+Ameri)
45. Jesse Mu and Jacob Andreas. 2020. Compositional explanations of neurons. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran Associates Inc. [find](https://scholar.google.com/scholar?q=Jesse+Mu+and+Jacob+Andreas%2E+2020%2E+Compositional+explanations+of+neurons%2E+In+Proceedings+of+the+34th+International+Conference+on+Neural+Information+Processing+Systems%2C+NIPS+%E2%80%9920%2C+Red+Hook%2C+NY%2C+USA%2E+Curr)
46. Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Arnab Sen Sharma, Jiuding Sun, Eric Todd, David Bau, and Yonatan Belinkov. 2024. The quest for the right mediator: A history, survey, and theoretical grounding of causal interp… — [arXiv:2408.01416](https://arxiv.org/abs/2408.01416) [find](https://scholar.google.com/scholar?q=Aaron+Mueller%2C+Jannik+Brinkmann%2C+Millicent+Li%2C+Samuel+Marks%2C+Koyena+Pal%2C+Nikhil+Prakash%2C+Can+Rager%2C+Aruna+Sankaranarayanan%2C+Arnab+Sen+Sharma%2C+Jiuding+Sun%2C+Eric+Todd%2C+David+Bau%2C+and+Yonatan+Belinkov%2E+2)
47. Aaron Mueller, Atticus Geiger, Sarah Wiegreffe, Dana Arad, Iván Arcuschin, Adam Belfki, Yik Siu Chan, Jaden Fiotto-Kaufman, Tal Haklay, Michael Hanna, Jing Huang, Rohan Gupta, Yaniv Nikankin, Hadas Orgad, Nikhil Prakash, Anja Reusch, Aruna Sankaranarayanan, Shun Shao, Alessandro Stolfo, Martin Tutek… — [arXiv:2504.13151](https://arxiv.org/abs/2504.13151) [find](https://scholar.google.com/scholar?q=Aaron+Mueller%2C+Atticus+Geiger%2C+Sarah+Wiegreffe%2C+Dana+Arad%2C+Iv%C3%A1n+Arcuschin%2C+Adam+Belfki%2C+Yik+Siu+Chan%2C+Jaden+Fiotto%2DKaufman%2C+Tal+Haklay%2C+Michael+Hanna%2C+Jing+Huang%2C+Rohan+Gupta%2C+Yaniv+Nikankin%2C+Hadas+Or)
48. Neel Nanda, Andrew Lee, and Martin Wattenberg. 2023. Emergent linear representations in world models of self-supervised sequence models. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2023, Singapore, December 7, 2023, pages 16–3… [find](https://scholar.google.com/scholar?q=Neel+Nanda%2C+Andrew+Lee%2C+and+Martin+Wattenberg%2E+2023%2E+Emergent+linear+representations+in+world+models+of+self%2Dsupervised+sequence+models%2E+In+Proceedings+of+the+6th+BlackboxNLP+Workshop%3A+Analyzing+and+I)
49. Tuomas P. Oikarinen and Tsui-Wei Weng. 2024. Linear explanations for individual neurons. In ICML. [find](https://scholar.google.com/scholar?q=Tuomas+P%2E+Oikarinen+and+Tsui%2DWei+Weng%2E+2024%2E+Linear+explanations+for+individual+neurons%2E+In+ICML%2E)
50. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. Zoom in: An introduction to circuits. Distill. Https://distill.pub/2020/circuits/zoom-in. [find](https://scholar.google.com/scholar?q=Chris+Olah%2C+Nick+Cammarata%2C+Ludwig+Schubert%2C+Gabriel+Goh%2C+Michael+Petrov%2C+and+Shan+Carter%2E+2020%2E+Zoom+in%3A+An+introduction+to+circuits%2E+Distill%2E+Https%3A%2F%2Fdistill%2Epub%2F2020%2Fcircuits%2Fzoom%2Din%2E)
51. James Oldfield, Christos Tzelepis, Yannis Panagakis, Mihalis Nicolaou, and Ioannis Patras. 2023. Panda: Unsupervised learning of parts and appearances in the feature maps of GANs. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=James+Oldfield%2C+Christos+Tzelepis%2C+Yannis+Panagakis%2C+Mihalis+Nicolaou%2C+and+Ioannis+Patras%2E+2023%2E+Panda%3A+Unsupervised+learning+of+parts+and+appearances+in+the+feature+maps+of+GANs%2E+In+The+Eleventh+Inte)
52. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex… — [GPT-4o System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/)
53. Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. 2025. The geometry of categorical and hierarchical concepts in large language models. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Kiho+Park%2C+Yo+Joong+Choe%2C+Yibo+Jiang%2C+and+Victor+Veitch%2E+2025%2E+The+geometry+of+categorical+and+hierarchical+concepts+in+large+language+models%2E+In+The+Thirteenth+International+Conference+on+Learning+Re)
54. Kiho Park, Yo Joong Choe, and Victor Veitch. 2024. [The linear representation hypothesis and the geometry of large language models](https://md.research.elamur.ai/free/en/papers/park-2023-linear-representation-hypothesis-geometry-large/). In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.
55. Gonçalo Paulo, Alex Mallen, Caden Juang, and Nora Belrose. 2024. Automatically interpreting millions of features in large language models. arXiv preprint arXiv:2410.13928. — [arXiv:2410.13928](https://arxiv.org/abs/2410.13928) [find](https://scholar.google.com/scholar?q=Gon%C3%A7alo+Paulo%2C+Alex+Mallen%2C+Caden+Juang%2C+and+Nora+Belrose%2E+2024%2E+Automatically+interpreting+millions+of+features+in+large+language+models%2E+arXiv+preprint+arXiv%3A2410%2E13928%2E+%E2%80%94+%5BarXiv%3A2410%2E13928%5D%28https%3A%2F)
56. Gonçalo Paulo and Nora Belrose. 2025. Sparse autoencoders trained on the same data learn different features. Preprint, arXiv:2501.16615. — [arXiv:2501.16615](https://arxiv.org/abs/2501.16615) [find](https://scholar.google.com/scholar?q=Gon%C3%A7alo+Paulo+and+Nora+Belrose%2E+2025%2E+Sparse+autoencoders+trained+on+the+same+data+learn+different+features%2E+Preprint%2C+arXiv%3A2501%2E16615%2E+%E2%80%94+%5BarXiv%3A2501%2E16615%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2501%2E16615%29)
57. Robert Peharz and Franz Pernkopf. 2012. Sparse nonnegative matrix factorization with ℓ 0 -constraints. Neurocomputing, 80:38–46. Special Issue on Machine Learning for Signal Processing 2010. [find](https://scholar.google.com/scholar?q=Robert+Peharz+and+Franz+Pernkopf%2E+2012%2E+Sparse+nonnegative+matrix+factorization+with+%E2%84%93+0+%2Dconstraints%2E+Neurocomputing%2C+80%3A38%E2%80%9346%2E+Special+Issue+on+Machine+Learning+for+Signal+Processing+2010%2E)
58. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI. Accessed: 2024-11-15. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI%2E+Accessed%3A+2024%2D11%2D15%2E)
59. Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. 2024. Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522, Bangkok, Thaila… [find](https://scholar.google.com/scholar?q=Nina+Rimsky%2C+Nick+Gabrieli%2C+Julian+Schulz%2C+Meg+Tong%2C+Evan+Hubinger%2C+and+Alexander+Turner%2E+2024%2E+Steering+llama+2+via+contrastive+activation+addition%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+A)
60. D. E. Rumelhart, J. L. McClelland, and PDP Research Group, editors. 1986. Parallel Distributed Processing. Volume 1: Foundations. MIT Press, Cambridge, MA. [find](https://scholar.google.com/scholar?q=D%2E+E%2E+Rumelhart%2C+J%2E+L%2E+McClelland%2C+and+PDP+Research+Group%2C+editors%2E+1986%2E+Parallel+Distributed+Processing%2E+Volume+1%3A+Foundations%2E+MIT+Press%2C+Cambridge%2C+MA%2E)
61. Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, Stella Biderman, Adria Garriga-Alonso, Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi Schoots, Joseph Miller, Eric J.… — [arXiv:2501.16496](https://arxiv.org/abs/2501.16496) [find](https://scholar.google.com/scholar?q=Lee+Sharkey%2C+Bilal+Chughtai%2C+Joshua+Batson%2C+Jack+Lindsey%2C+Jeff+Wu%2C+Lucius+Bushnaq%2C+Nicholas+Goldowsky%2DDill%2C+Stefan+Heimersheim%2C+Alejandro+Ortega%2C+Joseph+Bloom%2C+Stella+Biderman%2C+Adria+Garriga%2DAlonso%2C+A)
62. Paul Smolensky. 1986. Neural and conceptual interpretation of PDP models. In James L. McClelland, David E. Rumelhart, and the PDP Research Group, editors, Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Psychological and Biological Models, volume 2, pages 390– 431.… [find](https://scholar.google.com/scholar?q=Paul+Smolensky%2E+1986%2E+Neural+and+conceptual+interpretation+of+PDP+models%2E+In+James+L%2E+McClelland%2C+David+E%2E+Rumelhart%2C+and+the+PDP+Research+Group%2C+editors%2C+Parallel+Distributed+Processing%3A+Explorations)
63. Hyun Ah Song and Soo-Young Lee. 2013. Hierarchical representation using nmf. In Neural Information Processing, pages 466–473, Berlin, Heidelberg. Springer Berlin Heidelberg. [find](https://scholar.google.com/scholar?q=Hyun+Ah+Song+and+Soo%2DYoung+Lee%2E+2013%2E+Hierarchical+representation+using+nmf%2E+In+Neural+Information+Processing%2C+pages+466%E2%80%93473%2C+Berlin%2C+Heidelberg%2E+Springer+Berlin+Heidelberg%2E)
64. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton T… [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Morgane+Riviere%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+L%C3%A9onard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Ram%C3%A9%2C+Johan+Ferret%2C+Peter+Liu%2C+Pouya+Ta)
65. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. 2024. [Steering language models with activation engineering](https://md.research.elamur.ai/free/en/papers/turner-2023-steering-language-models-activation-engineering/). Preprint, arXiv:2308.10248.
66. Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. 2025. Axbench: Steering llms? even simple baselines outperform sparse autoencoders. Preprint, arXiv:2501.17148. [find](https://scholar.google.com/scholar?q=Zhengxuan+Wu%2C+Aryaman+Arora%2C+Atticus+Geiger%2C+Zheng+Wang%2C+Jing+Huang%2C+Dan+Jurafsky%2C+Christopher+D%2E+Manning%2C+and+Christopher+Potts%2E+2025%2E+Axbench%3A+Steering+llms%3F+even+simple+baselines+outperform+sparse+)
67. Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. 2024. ReFT: Representation fine-tuning for language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Zhengxuan+Wu%2C+Aryaman+Arora%2C+Zheng+Wang%2C+Atticus+Geiger%2C+Dan+Jurafsky%2C+Christopher+D+Manning%2C+and+Christopher+Potts%2E+2024%2E+ReFT%3A+Representation+fine%2Dtuning+for+language+models%2E+In+The+Thirty%2Deighth+An)
68. Zhengxuan Wu, Atticus Geiger, Thomas Icard, Christopher Potts, and Noah Goodman. 2023. [Interpretability at scale: Identifying causal mechanisms in alpaca](https://md.research.elamur.ai/free/en/papers/wu-2023-interpretability-scale-identifying-causal-mechanisms/). In Thirty-seventh Conference on Neural Information Processing Systems.
69. Zeyu Yun, Yubei Chen, Bruno Olshausen, and Yann Le-Cun. 2021. Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors. In Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration fo… [find](https://scholar.google.com/scholar?q=Zeyu+Yun%2C+Yubei+Chen%2C+Bruno+Olshausen%2C+and+Yann+Le%2DCun%2E+2021%2E+Transformer+visualization+via+dictionary+learning%3A+contextualized+embedding+as+a+linear+superposition+of+transformer+factors%2E+In+Proceedin)

*Updated 31 October 2026*