---
title: "Route Sparse Autoencoder to Interpret Large Language Models"
description: "The authors propose RouteSAE, which places a small router in front of one shared TopK SAE. 35 references, cited by 9 papers here."
authors: ["Wei Shi", "Sihang Li", "Tao Liang", "Mingyang Wan", "Guojun Ma", "Xiang Wang", "Xiangnan He"]
year: 2025
venue: "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2025.emnlp-main.346/"
doi: "10.18653/v1/2025.emnlp-main.346"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shi-2025-route-sparse-autoencoder-interpret-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shi-2025-route-sparse-autoencoder-interpret-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/) / Route Sparse Autoencoder to Interpret Large Language Models

# Route Sparse Autoencoder to Interpret Large Language Models

**Authors:** [Wei Shi](https://md.research.elamur.ai/free/en/authors/wei-shi/), [Sihang Li](https://md.research.elamur.ai/free/en/authors/sihang-li/), [Tao Liang](https://md.research.elamur.ai/free/en/authors/tao-liang/), Mingyang Wan, Guojun Ma, [Xiang Wang](https://md.research.elamur.ai/free/en/authors/xiang-wang/) and [Xiangnan He](https://md.research.elamur.ai/free/en/authors/xiangnan-he/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2025](https://md.research.elamur.ai/free/en/venues/emnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.emnlp-main.346/) · [PDF](https://aclanthology.org/2025.emnlp-main.346.pdf) · [DOI 10.18653/v1/2025.emnlp-main.346](https://doi.org/10.18653/v1/2025.emnlp-main.346) · [OpenAlex](https://openalex.org/W4416036715)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [9](https://md.research.elamur.ai/free/en/papers/shi-2025-route-sparse-autoencoder-interpret-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Sparse autoencoders (SAEs) are a popular way to break LLM activations into interpretable, mostly monosemantic features. They usually look at one layer only, which is a problem because features peak at different depths. Word-level features tend to be strongest early, while sentence-level or long-range patterns grow stronger in later layers. Crosscoders cover many layers but need separate encoders and decoders per layer, so they scale poorly and are hard to use for layer-specific interventions.

The authors propose RouteSAE, which places a small router in front of one shared TopK SAE. The router sum-pools residual streams from several layers and projects the result linearly to per-layer scores. It turns these into softmax probabilities and passes the most probable layer's activation, scaled by its probability, to the SAE. Router and SAE are trained jointly on reconstruction MSE.

On Llama-3.2-1B-Instruct, with the same sparsity, RouteSAE finds more interpretable features and gets higher interpretation scores than single-layer baselines. It adds very few parameters. This makes multi-layer interpretability and feature steering cheaper and easier to control.

## Key points
- At sparsity k = 64, RouteSAE yields 22.5% more interpretable features and a 22.3% higher interpretation score than baseline SAEs.
- Evaluation covered downstream KL divergence, interpretable feature count, and interpretation score.
- Compared with Crosscoder, which has roughly L times the parameters of an ordinary SAE for an L-layer model, RouteSAE adds only a linear router (W_router ∈ R^{L×d}) to one shared TopK SAE.
- Example features:
  - Shallow layers: "units of weight" and "Olympics".
  - Deeper layers: comparative phrasings that weigh one thing against another, and expressions about making the greatest possible effort.
- Steering works by setting a feature's activation to 20 at the layer the router chooses, which visibly shifts the model's outputs.
- Code: github.com/swei2001/RouteSAEs.
- Stated limitations:
  - The linear router is simple, and its weights depend on feature-space size M and sparsity k.
  - Discovering cross-layer features is not yet optimized.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Science and Technology of China](https://md.research.elamur.ai/free/en/universities/university-of-science-and-technology-of-china/) — University, Hefei, China · 108 works · 112,698 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Wei Shi, Sihang Li, Tao Liang, Mingyang Wan, Guojun Ma, Xiang Wang and Xiangnan He. 2025. Route Sparse Autoencoder to Interpret Large Language Models. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shi2025route,
  title = {{Route Sparse Autoencoder to Interpret Large Language Models}},
  author = {Wei Shi, Sihang Li, Tao Liang, Mingyang Wan, Guojun Ma, Xiang Wang and Xiangnan He},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.346},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.346}
}
```

- [← previous Retrofitting Large Language Models with Dynamic Tokenization](https://md.research.elamur.ai/free/en/papers/feher-2025-retrofitting-large-language-models-dynamic/)
- [next → SaRO: Enhancing LLM Safety through Reasoning-based Alignment](https://md.research.elamur.ai/free/en/papers/mou-2025-saro-enhancing-llm-safety-through/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/)

## References

35 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abien Fred Agarap. 2019. Deep learning using rectified linear units (relu). Preprint, arXiv:1803.08375. — [arXiv:1803.08375](https://arxiv.org/abs/1803.08375) [find](https://scholar.google.com/scholar?q=Abien+Fred+Agarap%2E+2019%2E+Deep+learning+using+rectified+linear+units+%28relu%29%2E+Preprint%2C+arXiv%3A1803%2E08375%2E+%E2%80%94+%5BarXiv%3A1803%2E08375%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1803%2E08375%29)
2. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, and 6 others. 2023. To… [find](https://scholar.google.com/scholar?q=Trenton+Bricken%2C+Adly+Templeton%2C+Joshua+Batson%2C+Brian+Chen%2C+Adam+Jermyn%2C+Tom+Conerly%2C+Nick+Turner%2C+Cem+Anil%2C+Carson+Denison%2C+Amanda+Askell%2C+Robert+Lasenby%2C+Yifan+Wu%2C+Shauna+Kravec%2C+Nicholas+Schiefer%2C+)
3. Ting-Yun Chang, Ta-Chung Chi, Shang-Chi Tsai, and Yun-Nung Chen. 2018. xsense: Learning sense-separated sparse representations and textual definitions for explainable word sense networks. CoRR, abs/1809.03348. — [arXiv:1809.03348](https://arxiv.org/abs/1809.03348) [find](https://scholar.google.com/scholar?q=Ting%2DYun+Chang%2C+Ta%2DChung+Chi%2C+Shang%2DChi+Tsai%2C+and+Yun%2DNung+Chen%2E+2018%2E+xsense%3A+Learning+sense%2Dseparated+sparse+representations+and+textual+definitions+for+explainable+word+sense+networks%2E+CoRR%2C+abs%2F18)
4. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 82 others. 2024… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
5. Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer ElShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Andy Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, and 14 others. 2022a. Softmax li… [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Tristan+Hume%2C+Catherine+Olsson%2C+Neel+Nanda%2C+Tom+Henighan%2C+Scott+Johnston%2C+Sheer+ElShowk%2C+Nicholas+Joseph%2C+Nova+DasSarma%2C+Ben+Mann%2C+Danny+Hernandez%2C+Amanda+Askell%2C+Kamal+Ndousse%2C+Andy+Jo)
6. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022b. Toy models of superposition. Transforme… [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Tristan+Hume%2C+Catherine+Olsson%2C+Nicholas+Schiefer%2C+Tom+Henighan%2C+Shauna+Kravec%2C+Zac+Hatfield%2DDodds%2C+Robert+Lasenby%2C+Dawn+Drain%2C+Carol+Chen%2C+Roger+Grosse%2C+Sam+McCandlish%2C+Jared+Kaplan%2C+D)
7. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, and 6 others. 2021. A mathe… [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Neel+Nanda%2C+Catherine+Olsson%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Zac+Hatfield%2DDodds%2C+Da)
8. N. Benjamin Erichson, Zhewei Yao, and Michael W. Mahoney. 2020. Jumprelu: A retrofit defense strategy for adversarial attacks. In ICPRAM, pages 103–114. SCITEPRESS. [find](https://scholar.google.com/scholar?q=N%2E+Benjamin+Erichson%2C+Zhewei+Yao%2C+and+Michael+W%2E+Mahoney%2E+2020%2E+Jumprelu%3A+A+retrofit+defense+strategy+for+adversarial+attacks%2E+In+ICPRAM%2C+pages+103%E2%80%93114%2E+SCITEPRESS%2E)
9. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+Shawn+Presser%2C+and+Connor+Leahy%2E+2020%2E+The+Pile%3A+An+800gb+datas)
10. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. Scaling and evaluating sparse autoencoders. CoRR, abs/2406.04093. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Tom+Dupr%C3%A9+la+Tour%2C+Henk+Tillman%2C+Gabriel+Goh%2C+Rajan+Troll%2C+Alec+Radford%2C+Ilya+Sutskever%2C+Jan+Leike%2C+and+Jeffrey+Wu%2E+2024%2E+Scaling+and+evaluating+sparse+autoencoders%2E+CoRR%2C+abs%2F2406%2E04093%2E)
11. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. 2023. Finding neurons in a haystack: Case studies with sparse probing. [find](https://scholar.google.com/scholar?q=Wes+Gurnee%2C+Neel+Nanda%2C+Matthew+Pauly%2C+Katherine+Harvey%2C+Dmitrii+Troitskii%2C+and+Dimitris+Bertsimas%2E+2023%2E+Finding+neurons+in+a+haystack%3A+Case+studies+with+sparse+probing%2E)
12. Trans. Mach. Learn. Res., 2023. [find](https://scholar.google.com/scholar?q=Trans%2E+Mach%2E+Learn%2E+Res%2E%2C+2023%2E)
13. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In CVPR, pages 770–778. IEEE Computer Society.
14. Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. 2024. Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders. CoRR, abs/2410.20526. [find](https://scholar.google.com/scholar?q=Zhengfu+He%2C+Wentao+Shu%2C+Xuyang+Ge%2C+Lingjie+Chen%2C+Junxuan+Wang%2C+Yunhua+Zhou%2C+Frances+Liu%2C+Qipeng+Guo%2C+Xuanjing+Huang%2C+Zuxuan+Wu%2C+Yu%2DGang+Jiang%2C+and+Xipeng+Qiu%2E+2024%2E+Llama+scope%3A+Extracting+millions+of)
15. G E Hinton and R R Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. Science, 313:504–507. [find](https://scholar.google.com/scholar?q=G+E+Hinton+and+R+R+Salakhutdinov%2E+2006%2E+Reducing+the+dimensionality+of+data+with+neural+networks%2E+Science%2C+313%3A504%E2%80%93507%2E)
16. Robert Huben, Hoagy Cunningham, Logan Riggs, Aidan Ewart, and Lee Sharkey. 2024. Sparse autoencoders find highly interpretable features in language models. In ICLR. OpenReview.net. [find](https://scholar.google.com/scholar?q=Robert+Huben%2C+Hoagy+Cunningham%2C+Logan+Riggs%2C+Aidan+Ewart%2C+and+Lee+Sharkey%2E+2024%2E+Sparse+autoencoders+find+highly+interpretable+features+in+language+models%2E+In+ICLR%2E+OpenReview%2Enet%2E)
17. Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, and 79 others. 2024. [Gpt-4o sy…](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/)
18. Diederik P. Kingma and Jimmy Ba. 2015. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). In ICLR (Poster).
19. Connor Kissane, Robert Krzyzanowski, Arthur Conmy, and Neel Nanda. 2024. Saes (usually) transfer between base and chat models. Alignment Forum. [find](https://scholar.google.com/scholar?q=Connor+Kissane%2C+Robert+Krzyzanowski%2C+Arthur+Conmy%2C+and+Neel+Nanda%2E+2024%2E+Saes+%28usually%29+transfer+between+base+and+chat+models%2E+Alignment+Forum%2E)
20. Vedang Lad, Wes Gurnee, and Max Tegmark. 2024. The remarkable robustness of llms: Stages of inference? CoRR, abs/2406.19384. — [arXiv:2406.19384](https://arxiv.org/abs/2406.19384) [find](https://scholar.google.com/scholar?q=Vedang+Lad%2C+Wes+Gurnee%2C+and+Max+Tegmark%2E+2024%2E+The+remarkable+robustness+of+llms%3A+Stages+of+inference%3F+CoRR%2C+abs%2F2406%2E19384%2E+%E2%80%94+%5BarXiv%3A2406%2E19384%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2406%2E19384%29)
21. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca D. Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. CoRR, abs/2408.05147. [find](https://scholar.google.com/scholar?q=Tom+Lieberum%2C+Senthooran+Rajamanoharan%2C+Arthur+Conmy%2C+Lewis+Smith%2C+Nicolas+Sonnerat%2C+Vikrant+Varma%2C+J%C3%A1nos+Kram%C3%A1r%2C+Anca+D%2E+Dragan%2C+Rohin+Shah%2C+and+Neel+Nanda%2E+2024%2E+Gemma+scope%3A+Open+sparse+autoencoder)
22. Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christopher Olah. 2024. Sparse crosscoders for cross-layer features and model diffing. Transformer Circuits Thread. [find](https://scholar.google.com/scholar?q=Jack+Lindsey%2C+Adly+Templeton%2C+Jonathan+Marcus%2C+Thomas+Conerly%2C+Joshua+Batson%2C+and+Christopher+Olah%2E+2024%2E+Sparse+crosscoders+for+cross%2Dlayer+features+and+model+diffing%2E+Transformer+Circuits+Thread%2E)
23. Julien Mairal, Francis R. Bach, Jean Ponce, and Guillermo Sapiro. 2009. Online dictionary learning for sparse coding. In ICML, volume 382, pages 689–696. [find](https://scholar.google.com/scholar?q=Julien+Mairal%2C+Francis+R%2E+Bach%2C+Jean+Ponce%2C+and+Guillermo+Sapiro%2E+2009%2E+Online+dictionary+learning+for+sparse+coding%2E+In+ICML%2C+volume+382%2C+pages+689%E2%80%93696%2E)
24. Alireza Makhzani and Brendan J. Frey. 2014. k-sparse autoencoders. In ICLR (Poster). [find](https://scholar.google.com/scholar?q=Alireza+Makhzani+and+Brendan+J%2E+Frey%2E+2014%2E+k%2Dsparse+autoencoders%2E+In+ICLR+%28Poster%29%2E)
25. Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. 2024. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. CoRR, abs/2403.19647. [find](https://scholar.google.com/scholar?q=Samuel+Marks%2C+Can+Rager%2C+Eric+J%2E+Michaud%2C+Yonatan+Belinkov%2C+David+Bau%2C+and+Aaron+Mueller%2E+2024%2E+Sparse+feature+circuits%3A+Discovering+and+editing+interpretable+causal+graphs+in+language+models%2E+CoRR%2C+a)
26. Roland Memisevic, Kishore Reddy Konda, and David Krueger. 2015. Zero-bias autoencoders and the benefits of co-adapting features. In ICLR (Poster). [find](https://scholar.google.com/scholar?q=Roland+Memisevic%2C+Kishore+Reddy+Konda%2C+and+David+Krueger%2E+2015%2E+Zero%2Dbias+autoencoders+and+the+benefits+of+co%2Dadapting+features%2E+In+ICLR+%28Poster%29%2E)
27. Anish Mudide, Joshua Engels, Eric J. Michaud, Max Tegmark, and Christian Schröder de Witt. 2024. Efficient dictionary learning with switch sparse autoencoders. CoRR, abs/2410.08201. — [arXiv:2410.08201](https://arxiv.org/abs/2410.08201) [find](https://scholar.google.com/scholar?q=Anish+Mudide%2C+Joshua+Engels%2C+Eric+J%2E+Michaud%2C+Max+Tegmark%2C+and+Christian+Schr%C3%B6der+de+Witt%2E+2024%2E+Efficient+dictionary+learning+with+switch+sparse+autoencoders%2E+CoRR%2C+abs%2F2410%2E08201%2E+%E2%80%94+%5BarXiv%3A2410%2E0820)
28. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+and+1+others%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
29. Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. 2024a. [Improving dictionary learning with gated sparse autoencoders](https://md.research.elamur.ai/free/en/papers/rajamanoharan-2024-improving-dictionary-learning-gated-sparse/). CoRR, abs/2404.16014.
30. Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. 2024b. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. CoRR, abs/2407.14435. — [arXiv:2407.14435](https://arxiv.org/abs/2407.14435) [find](https://scholar.google.com/scholar?q=Senthooran+Rajamanoharan%2C+Tom+Lieberum%2C+Nicolas+Sonnerat%2C+Arthur+Conmy%2C+Vikrant+Varma%2C+J%C3%A1nos+Kram%C3%A1r%2C+and+Neel+Nanda%2E+2024b%2E+Jumping+ahead%3A+Improving+reconstruction+fidelity+with+jumprelu+sparse+autoen)
31. Morgane Rivière, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, an… [find](https://scholar.google.com/scholar?q=Morgane+Rivi%C3%A8re%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+L%C3%A9onard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Ram%C3%A9%2C+Johan+Ferret%2C+Peter+Liu%2C+Pouya+Tafti%2C+Abe+Fri)
32. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, an… [find](https://scholar.google.com/scholar?q=Adly+Templeton%2C+Tom+Conerly%2C+Jonathan+Marcus%2C+Jack+Lindsey%2C+Trenton+Bricken%2C+Brian+Chen%2C+Adam+Pearce%2C+Craig+Citro%2C+Emmanuel+Ameisen%2C+Andy+Jones%2C+Hoagy+Cunningham%2C+Nicholas+L+Turner%2C+Callum+McDougall%2C+)
33. Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2023. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In ICLR. [find](https://scholar.google.com/scholar?q=Kevin+Ro+Wang%2C+Alexandre+Variengien%2C+Arthur+Conmy%2C+Buck+Shlegeris%2C+and+Jacob+Steinhardt%2E+2023%2E+Interpretability+in+the+wild%3A+a+circuit+for+indirect+object+identification+in+GPT%2D2+small%2E+In+ICLR%2E)
34. Zhaowen Wang, Ding Liu, Jianchao Yang, Wei Han, and Thomas S. Huang. 2015. Deeply improved sparse coding for image super-resolution. CoRR, abs/1507.08905. — [arXiv:1507.08905](https://arxiv.org/abs/1507.08905) [find](https://scholar.google.com/scholar?q=Zhaowen+Wang%2C+Ding+Liu%2C+Jianchao+Yang%2C+Wei+Han%2C+and+Thomas+S%2E+Huang%2E+2015%2E+Deeply+improved+sparse+coding+for+image+super%2Dresolution%2E+CoRR%2C+abs%2F1507%2E08905%2E+%E2%80%94+%5BarXiv%3A1507%2E08905%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F15)
35. Zeyu Yun, Yubei Chen, Bruno A. Olshausen, and Yann LeCun. 2021. Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors. In DeeLIO@NAACL-HLT, pages 1–10. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zeyu+Yun%2C+Yubei+Chen%2C+Bruno+A%2E+Olshausen%2C+and+Yann+LeCun%2E+2021%2E+Transformer+visualization+via+dictionary+learning%3A+contextualized+embedding+as+a+linear+superposition+of+transformer+factors%2E+In+DeeLIO%40)

*Updated 31 October 2026*