---
title: "Improving Dictionary Learning with Gated Sparse Autoencoders — Rajamanoharan 2024"
description: "The authors propose the Gated SAE, whose encoder splits two jobs. 13 references, cited by 37 papers here."
authors: ["Senthooran Rajamanoharan", "Arthur Conmy", "Lewis Smith", "Tom Lieberum", "Vikrant Varma", "Janos Kramar", "Rohin Shah", "Neel Nanda"]
year: 2024
venue: "arXiv:2404.16014"
source: "https://arxiv.org/abs/2404.16014"
doi: "10.48550/arxiv.2404.16014"
arxiv: "2404.16014"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rajamanoharan-2024-improving-dictionary-learning-gated-sparse/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rajamanoharan-2024-improving-dictionary-learning-gated-sparse/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Improving Dictionary Learning with Gated Sparse Autoencoders

# Improving Dictionary Learning with Gated Sparse Autoencoders

**Authors:** Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, [Tom Lieberum](https://md.research.elamur.ai/free/en/authors/tom-lieberum/), Vikrant Varma, [Janos Kramar](https://md.research.elamur.ai/free/en/authors/janos-kramar/), [Rohin Shah](https://md.research.elamur.ai/free/en/authors/rohin-shah/) and [Neel Nanda](https://md.research.elamur.ai/free/en/authors/neel-nanda/) · **Year:** 2024 · **Venue:** arXiv:2404.16014 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2404.16014) · [PDF](https://arxiv.org/pdf/2404.16014) · [DOI 10.48550/arxiv.2404.16014](https://doi.org/10.48550/arxiv.2404.16014) · [arXiv 2404.16014](https://arxiv.org/abs/2404.16014) · [OpenAlex](https://openalex.org/W4395484123)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · 37 in our library, of the 140,884 papers whose bibliographies we read

## Summary
Sparse autoencoders (SAEs) are used to break language model activations into sparse combinations of interpretable directions. The standard training objective uses an L1 sparsity penalty, which biases the encoder. Its most visible effect is shrinkage: feature activations come out systematically too small. The authors propose the Gated SAE, whose encoder splits two jobs. One path decides which dictionary features are switched on. A second path estimates how strongly each active feature fires. The L1 penalty applies only to the selection path, and the two paths share some weights so parameter count and inference cost stay close to a standard SAE. They trained SAEs on GELU-1L, Pythia-2.8B and Gemma-7B, covering MLP outputs, attention outputs and the residual stream. At equal training compute, Gated SAEs reach a given reconstruction quality with fewer active features, removing shrinkage without losing interpretability. This makes SAE-based tools for steering models and analysing circuits cheaper and more faithful.

## Key points
- Quality is measured with two metrics: L0 (average count of active features) and loss recovered (how much LM cross-entropy is preserved when SAE reconstructions are spliced in).
- At many sites, Gated SAEs need about half the L0 to match baseline loss recovered. The comparison uses equal compute, with baseline SAEs given 50% more features.
- Computing the Gated SAE training loss costs roughly 50% more than for a same-width baseline, because of an auxiliary loss term.
- A small double-blind human study rated Gated SAE features about as interpretable as baseline ones.
- An ablation study found every key component of the method necessary for strong performance. Gated SAEs also beat other approaches that target shrinkage.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cite

Cite as: Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, Janos Kramar, Rohin Shah and Neel Nanda. 2024. Improving Dictionary Learning with Gated Sparse Autoencoders. arXiv:2404.16014.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{rajamanoharan2024improving,
  title = {{Improving Dictionary Learning with Gated Sparse Autoencoders}},
  author = {Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, Janos Kramar, Rohin Shah and Neel Nanda},
  year = {2024},
  doi = {10.48550/arxiv.2404.16014},
  eprint = {2404.16014},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.16014}
}
```

- [← previous Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies](https://md.research.elamur.ai/free/en/papers/tao-2024-scaling-laws-vocabulary-larger-models/)
- [next → Maverick: Efficient and Accurate Coreference Resolution Defying Recent Trends](https://md.research.elamur.ai/free/en/papers/martinelli-2024-maverick-efficient-accurate-coreference-resolution/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

13 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. Aharon, M. Elad, and A. Bruckstein. K-svd: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on Signal Processing, 54(11):4311–4322, 2006. doi: 10.1109/TSP.2006.881199. — [doi:10.1109/tsp.2006.881199](https://doi.org/10.1109/tsp.2006.881199) [find](https://scholar.google.com/scholar?q=M%2E+Aharon%2C+M%2E+Elad%2C+and+A%2E+Bruckstein%2E+K%2Dsvd%3A+An+algorithm+for+designing+overcomplete+dictionaries+for+sparse+representation%2E+IEEE+Transactions+on+Signal+Processing%2C+54%2811%29%3A4311%E2%80%934322%2C+2006%2E+doi%3A+10%2E11)
2. Anthropic AI. Introducing the next generation of Claude. [https://www.anthropic.com/index/](https://www.anthropic.com/index/) introducing-the-next-generation-of-claude , 2024. Accessed: 2024-04-14. [↗](https://www.anthropic.com/index/)
3. J. Batson, B. Chen, A. Jones, A. Templeton, T. Conerly, J. Marcus, T. Henighan, N. L. Turner, and A. Pearce. Circuits Updates - March 2024. [find](https://scholar.google.com/scholar?q=J%2E+Batson%2C+B%2E+Chen%2C+A%2E+Jones%2C+A%2E+Templeton%2C+T%2E+Conerly%2C+J%2E+Marcus%2C+T%2E+Henighan%2C+N%2E+L%2E+Turner%2C+and+A%2E+Pearce%2E+Circuits+Updates+%2D+March+2024%2E)
4. Transformer Circuits Thread, 2024. URL [https://transformer-circuits.pub/2024/mar-update/index.html](https://transformer-circuits.pub/2024/mar-update/index.html) . [↗](https://transformer-circuits.pub/2024/mar-update/index.html)
5. Y. Bengio. Deep learning of representations: Looking forward, 2013. [find](https://scholar.google.com/scholar?q=Y%2E+Bengio%2E+Deep+learning+of+representations%3A+Looking+forward%2C+2013%2E)
6. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397–2430. PMLR, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Biderman%2C+H%2E+Schoelkopf%2C+Q%2E+G%2E+Anthony%2C+H%2E+Bradley%2C+K%2E+O%E2%80%99Brien%2C+E%2E+Hallahan%2C+M%2E+A%2E+Khan%2C+S%2E+Purohit%2C+U%2E+S%2E+Prashanth%2C+E%2E+Raff%2C+et+al%2E+Pythia%3A+A+suite+for+analyzing+large+language+models+across+trai)
7. S. Bills, N. Cammarata, D. Mossing, H. Tillman, L. Gao, G. Goh, I. Sutskever, J. Leike, J. Wu, and W. Saunders. Language models can explain neurons in language models. [https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html](https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html) , 2023. [↗](https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html)
8. J. Bloom. Open Source Sparse Autoencoders for all Residual Stream Layers of GPT-2 Small. [https://www.alignmentforum.org/posts/f9EgfLSurAiqRJySD/](https://www.alignmentforum.org/posts/f9EgfLSurAiqRJySD/) open-source-sparse-autoencoders-for-all-residual-stream , 2024. [↗](https://www.alignmentforum.org/posts/f9EgfLSurAiqRJySD/)
9. T. Blumensath and M. E. Davies. Gradient pursuits. IEEE Transactions on Signal Processing, 56(6): 2370–2382, 2008. [find](https://scholar.google.com/scholar?q=T%2E+Blumensath+and+M%2E+E%2E+Davies%2E+Gradient+pursuits%2E+IEEE+Transactions+on+Signal+Processing%2C+56%286%29%3A+2370%E2%80%932382%2C+2008%2E)
10. T. Bolukbasi, A. Pearce, A. Yuan, A. Coenen, E. Reif, F. Viégas, and M. Wattenberg. An interpretability illusion for bert. arXiv preprint arXiv:2104.07143, 2021. — [arXiv:2104.07143](https://arxiv.org/abs/2104.07143) [find](https://scholar.google.com/scholar?q=T%2E+Bolukbasi%2C+A%2E+Pearce%2C+A%2E+Yuan%2C+A%2E+Coenen%2C+E%2E+Reif%2C+F%2E+Vi%C3%A9gas%2C+and+M%2E+Wattenberg%2E+An+interpretability+illusion+for+bert%2E+arXiv+preprint+arXiv%3A2104%2E07143%2C+2021%2E+%E2%80%94+%5BarXiv%3A2104%2E07143%5D%28https%3A%2F%2Farxiv%2Eorg)
11. T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter, T. Henighan, and C. Olah. Towards mon… [find](https://scholar.google.com/scholar?q=T%2E+Bricken%2C+A%2E+Templeton%2C+J%2E+Batson%2C+B%2E+Chen%2C+A%2E+Jermyn%2C+T%2E+Conerly%2C+N%2E+Turner%2C+C%2E+Anil%2C+C%2E+Denison%2C+A%2E+Askell%2C+R%2E+Lasenby%2C+Y%2E+Wu%2C+S%2E+Kravec%2C+N%2E+Schiefer%2C+T%2E+Maxwell%2C+N%2E+Joseph%2C+Z%2E+Hatfield%2DDodds%2C+A%2E+)
12. R. T. Chen, X. Li, R. B. Grosse, and D. K. Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018. [find](https://scholar.google.com/scholar?q=R%2E+T%2E+Chen%2C+X%2E+Li%2C+R%2E+B%2E+Grosse%2C+and+D%2E+K%2E+Duvenaud%2E+Isolating+sources+of+disentanglement+in+variational+autoencoders%2E+Advances+in+neural+information+processing+systems%2C+31%2C+2018%2E)
13. X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016. [find](https://scholar.google.com/scholar?q=X%2E+Chen%2C+Y%2E+Duan%2C+R%2E+Houthooft%2C+J%2E+Schulman%2C+I%2E+Sutskever%2C+and+P%2E+Abbeel%2E+Infogan%3A+Interpretable+representation+learning+by+information+maximizing+generative+adversarial+nets%2E+Advances+in+neural+infor)