---
title: "AdaptiveK — paper by Yao et al. (2026) — summary and ideas"
description: "Sparse autoencoders (SAEs) break LLM activations into interpretable features. 49 references."
authors: ["Yifei Yao", "Hanrong Zhang", "Mengnan Du"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1187/"
doi: "10.18653/v1/2026.findings-acl.1187"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yao-2026-adaptivek-complexity-driven-sparse-autoencoders/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yao-2026-adaptivek-complexity-driven-sparse-autoencoders/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/21/) / AdaptiveK: Complexity-Driven Sparse Autoencoders for Interpretable Language Model Representations

# AdaptiveK: Complexity-Driven Sparse Autoencoders for Interpretable Language Model Representations

**Authors:** Yifei Yao, [Hanrong Zhang](https://md.research.elamur.ai/free/en/authors/hanrong-zhang/) and [Mengnan Du](https://md.research.elamur.ai/free/en/authors/mengnan-du/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1187/) · [PDF](https://aclanthology.org/2026.findings-acl.1187.pdf) · [DOI 10.18653/v1/2026.findings-acl.1187](https://doi.org/10.18653/v1/2026.findings-acl.1187) · [OpenAlex](https://openalex.org/W7166801322)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Sparse autoencoders (SAEs) break LLM activations into interpretable features. However, standard variants such as TopK, BatchTopK, Gated SAEs and P-anneal apply the same sparsity level to every input. As a result, easy texts get more features than they need, hard texts get too few, and finding a good sparsity setting takes costly hyperparameter searches.

The authors first show that text complexity is linearly recoverable from model activations. They rate contexts with GPT-4.1-mini on six semantic dimensions, combine the ratings into one score, and fit linear regression probes that predict this score with high correlation across eight LLMs. Building on this, AdaptiveK SAE uses the probe's complexity estimate to set the number of active latents per input during training, giving complex inputs more features and simple ones fewer.

Across ten language models, AdaptiveK outperforms fixed-sparsity SAEs in how faithfully it reconstructs activations, in cosine similarity and explained variance, and on interpretability metrics, all from a single training run. This makes the approach a practical, tuning-light route to cleaner feature decompositions.

## Key points
- Complexity labels come from GPT-4.1-mini scores on six semantic dimensions. Scores in the test examples range from 0 (simplest) to about 9.6 (most complex), and linear probes on activations predict them well.
- The SAE's active-feature count is computed as a function of the probe score, so it is not a fixed k. Figure 1 contrasts this with TopK baselines using k from 20 to 640 on Pythia-160M.
- Baselines were run with the dictionary_learning library. Training used 250,000 contexts, with the final token's activation taken from each context. This is far less data than the 500M tokens used in SAEBench, yet some results still exceed SAEBench baselines.
- Interpretability was checked with MaxAct and VocabProj on Llama-3.1-8B and Gemma-2-9B.
  - AdaptiveK concentrates on domain terms and fires less on function words.
  - Its features cover cross-lingual and morphological variants of a concept, such as "title"/"Titel" and "abstract"/"abstracto".
- Code is available at github.com/hiyukie/adaptiveK. Note that the paper is inconsistent about the model count: the abstract says ten LLMs, while the conclusion says eight.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Zhejiang University](https://md.research.elamur.ai/free/en/universities/zhejiang-university/) — University, Hangzhou, China · 224 works · 13,587 citations
- [University of Illinois Chicago](https://md.research.elamur.ai/free/en/universities/university-of-illinois-chicago/) — University, Chicago, United States · 70 works · 20,363 citations
- [Chinese University of Hong Kong, Shenzhen](https://md.research.elamur.ai/free/en/universities/chinese-university-of-hong-kong-shenzhen/) — University, Shenzhen, China · 56 works · 3,212 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Yifei Yao, Hanrong Zhang and Mengnan Du. 2026. AdaptiveK: Complexity-Driven Sparse Autoencoders for Interpretable Language Model Representations. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yao2026adaptivek,
  title = {{AdaptiveK: Complexity-Driven Sparse Autoencoders for Interpretable Language Model Representations}},
  author = {Yifei Yao, Hanrong Zhang and Mengnan Du},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1187},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1187}
}
```

- [← previous Adaptive Zooming via Relevance-Informed Positional Resource Allocation for Training-free LLM Context Extension](https://md.research.elamur.ai/free/en/papers/zhao-2026-adaptive-zooming-relevance-informed-positional/)
- [next → Addressing Domain Shift in Health Coaching Note Analysis through Factorized Synthetic Data Generation](https://md.research.elamur.ai/free/en/papers/tanzer-2026-addressing-domain-shift-health-coaching/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/21/)

## References

49 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and 8 oth… — [Phi-4 Technical Report](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-4-technical-report/)
2. Anthropic Interpretability Team. 2024. Circuits updates—april 2024. https://transformer-c ircuits.pub/2024/april-update/index.html. [find](https://scholar.google.com/scholar?q=Anthropic+Interpretability+Team%2E+2024%2E+Circuits+updates%E2%80%94april+2024%2E+https%3A%2F%2Ftransformer%2Dc+ircuits%2Epub%2F2024%2Fapril%2Dupdate%2Findex%2Ehtml%2E)
3. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. [Refusal in language models is mediated by a single direction](https://md.research.elamur.ai/free/en/papers/arditi-2024-refusal-language-models-mediated-single/). arXiv preprint arXiv:2406.11717.
4. Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2018. [Linear algebraic structure of word senses, with applications to polysemy](https://md.research.elamur.ai/free/en/papers/arora-2018-linear-algebraic-structure-word-senses/). Transactions of the Association for Computational Linguistics, 6:483–495.
5. Yonatan Belinkov. 2022. [Probing classifiers: Promises, shortcomings, and advances](https://md.research.elamur.ai/free/en/papers/belinkov-2022-probing-classifiers-promises-shortcomings-advances/). Computational Linguistics, 48(1):207–219.
6. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, and 1 others. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Co… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Gregory+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+and+1+others%2E+2023%2E+)
7. Dan Braun, Jordan Taylor, Nicholas Goldowsky-Dill, and Lee Sharkey. 2024. Identifying functionally important features with end-to-end sparse dictionary learning. Advances in Neural Information Processing Systems, 37:107286–107325. [find](https://scholar.google.com/scholar?q=Dan+Braun%2C+Jordan+Taylor%2C+Nicholas+Goldowsky%2DDill%2C+and+Lee+Sharkey%2E+2024%2E+Identifying+functionally+important+features+with+end%2Dto%2Dend+sparse+dictionary+learning%2E+Advances+in+Neural+Information+Process)
8. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, and 1 others. 2023. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2. [find](https://scholar.google.com/scholar?q=Trenton+Bricken%2C+Adly+Templeton%2C+Joshua+Batson%2C+Brian+Chen%2C+Adam+Jermyn%2C+Tom+Conerly%2C+Nick+Turner%2C+Cem+Anil%2C+Carson+Denison%2C+Amanda+Askell%2C+and+1+others%2E+2023%2E+Towards+monosemanticity%3A+Decomposing+lan)
9. Bart Bussmann, Patrick Leask, and Neel Nanda. 2024a. Batchtopk sparse autoencoders. arXiv preprint arXiv:2412.06410. — [arXiv:2412.06410](https://arxiv.org/abs/2412.06410) [find](https://scholar.google.com/scholar?q=Bart+Bussmann%2C+Patrick+Leask%2C+and+Neel+Nanda%2E+2024a%2E+Batchtopk+sparse+autoencoders%2E+arXiv+preprint+arXiv%3A2412%2E06410%2E+%E2%80%94+%5BarXiv%3A2412%2E06410%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2412%2E06410%29)
10. Bart Bussmann, Patrick Leask, and Neel Nanda. 2024b. Learning multi-level features with matryoshka saes, december 19 2024b. In URL [https://www.alignmentforum](https://www.alignmentforum). org/posts/rKM9b6B2LqwSB5ToN/learning-multi-level-features-with-matryoshka-saes. Alignment Forum. [↗](https://www.alignmentforum)
11. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600. [find](https://scholar.google.com/scholar?q=Hoagy+Cunningham%2C+Aidan+Ewart%2C+Logan+Riggs%2C+Robert+Huben%2C+and+Lee+Sharkey%2E+2023%2E+Sparse+autoencoders+find+highly+interpretable+features+in+language+models%2E+arXiv+preprint+arXiv%3A2309%2E08600%2E)
12. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, and 1 others. 2022. Toy models of superposition. arXiv preprint arXiv:2209.10652. [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Tristan+Hume%2C+Catherine+Olsson%2C+Nicholas+Schiefer%2C+Tom+Henighan%2C+Shauna+Kravec%2C+Zac+Hatfield%2DDodds%2C+Robert+Lasenby%2C+Dawn+Drain%2C+Carol+Chen%2C+and+1+others%2E+2022%2E+Toy+models+of+superpositi)
13. Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-Jussà. 2024. [A primer on the inner workings of transformer-based language models](https://md.research.elamur.ai/free/en/papers/ferrando-2024-primer-inner-workings-transformer-based/). arXiv preprint arXiv:2405.00208.
14. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, and 1 others. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+and+1+others%2E+2020%2E+The+pile%3A+An+800gb+dataset+of+diverse+text+)
15. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Tom+Dupr%C3%A9+la+Tour%2C+Henk+Tillman%2C+Gabriel+Goh%2C+Rajan+Troll%2C+Alec+Radford%2C+Ilya+Sutskever%2C+Jan+Leike%2C+and+Jeffrey+Wu%2E+2024%2E+Scaling+and+evaluating+sparse+autoencoders%2E+arXiv+preprint+arXiv%3A2406)
16. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
17. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. 2023. [Finding neurons in a haystack: Case studies with sparse probing](https://md.research.elamur.ai/free/en/papers/gurnee-2023-finding-neurons-haystack-case-studies/). arXiv preprint arXiv:2305.01610.
18. Wes Gurnee and Max Tegmark. 2023. Language models represent space and time. arXiv preprint arXiv:2310.02207. [find](https://scholar.google.com/scholar?q=Wes+Gurnee+and+Max+Tegmark%2E+2023%2E+Language+models+represent+space+and+time%2E+arXiv+preprint+arXiv%3A2310%2E02207%2E)
19. Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Robert+Huben%2C+Hoagy+Cunningham%2C+Logan+Riggs+Smith%2C+Aidan+Ewart%2C+and+Lee+Sharkey%2E+2023%2E+Sparse+autoencoders+find+highly+interpretable+features+in+language+models%2E+In+The+Twelfth+International+Conferenc)
20. Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Callum McDougall, Kola Ayonrinde, and 1 others. 2025. Saebench: A comprehensive benchmark for sparse autoencoders in language model interpretability. arXiv preprint arXiv:2503.09532. — [arXiv:2503.09532](https://arxiv.org/abs/2503.09532) [find](https://scholar.google.com/scholar?q=Adam+Karvonen%2C+Can+Rager%2C+Johnny+Lin%2C+Curt+Tigges%2C+Joseph+Bloom%2C+David+Chanin%2C+Yeu%2DTong+Lau%2C+Eoin+Farrell%2C+Callum+McDougall%2C+Kola+Ayonrinde%2C+and+1+others%2E+2025%2E+Saebench%3A+A+comprehensive+benchmark+for)
21. Adam Karvonen, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann, Logan Smith, Claudio Mayrink Verdun, David Bau, and Samuel Marks. 2024. Measuring progress in dictionary learning for language model interpretability with board game models. Advances in Neural Information Processing Systems, 3… [find](https://scholar.google.com/scholar?q=Adam+Karvonen%2C+Benjamin+Wright%2C+Can+Rager%2C+Rico+Angell%2C+Jannik+Brinkmann%2C+Logan+Smith%2C+Claudio+Mayrink+Verdun%2C+David+Bau%2C+and+Samuel+Marks%2E+2024%2E+Measuring+progress+in+dictionary+learning+for+language)
22. Junsol Kim, James Evans, and Aaron Schein. 2025. Linear representations of political perspective emerge in large language models. arXiv preprint arXiv:2503.02080. — [arXiv:2503.02080](https://arxiv.org/abs/2503.02080) [find](https://scholar.google.com/scholar?q=Junsol+Kim%2C+James+Evans%2C+and+Aaron+Schein%2E+2025%2E+Linear+representations+of+political+perspective+emerge+in+large+language+models%2E+arXiv+preprint+arXiv%3A2503%2E02080%2E+%E2%80%94+%5BarXiv%3A2503%2E02080%5D%28https%3A%2F%2Farxiv%2Eor)
23. Diederik P Kingma and Jimmy Ba. 2014. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980.
24. Sewoong Lee, Adam Davies, Marc E. Canby, and Julia Hockenmaier. 2025. Evaluating and designing sparse autoencoders by approximating quasi-orthogonality. Preprint, arXiv:2503.24277. — [arXiv:2503.24277](https://arxiv.org/abs/2503.24277) [find](https://scholar.google.com/scholar?q=Sewoong+Lee%2C+Adam+Davies%2C+Marc+E%2E+Canby%2C+and+Julia+Hockenmaier%2E+2025%2E+Evaluating+and+designing+sparse+autoencoders+by+approximating+quasi%2Dorthogonality%2E+Preprint%2C+arXiv%3A2503%2E24277%2E+%E2%80%94+%5BarXiv%3A2503%2E24277)
25. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023. [Inference-time intervention: Eliciting truthful answers from a language model](https://md.research.elamur.ai/free/en/papers/li-2023-inference-time-intervention-eliciting-truthful/). Advances in Neural Information Processing Systems, 36:41451–41530.
26. Sheng Liu, Haotian Ye, Lei Xing, and James Zou. 2024. [In-context vectors: Making in context learning more effective and controllable through latent space steering](https://md.research.elamur.ai/free/en/papers/liu-2023-context-vectors-making-context-learning/). URL https://arxiv.org/abs/2311.06668. [↗](https://arxiv.org/abs/2311.06668)
27. Luke Marks, Alasdair Paren, David Krueger, and Fazl Barez. 2024a. Enhancing neural network interpretability with feature-aligned sparse autoencoders. arXiv preprint arXiv:2411.01220. — [arXiv:2411.01220](https://arxiv.org/abs/2411.01220) [find](https://scholar.google.com/scholar?q=Luke+Marks%2C+Alasdair+Paren%2C+David+Krueger%2C+and+Fazl+Barez%2E+2024a%2E+Enhancing+neural+network+interpretability+with+feature%2Daligned+sparse+autoencoders%2E+arXiv+preprint+arXiv%3A2411%2E01220%2E+%E2%80%94+%5BarXiv%3A2411%2E012)
28. Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. [Linguistic regularities in continuous space word representations](https://md.research.elamur.ai/free/en/papers/mikolov-2013-linguistic-regularities-continuous-space-word/). In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 746–751.
29. Anish Mudide, Joshua Engels, Eric J Michaud, Max Tegmark, and Christian Schroeder de Witt. 2024. Efficient dictionary learning with switch sparse autoencoders. arXiv preprint arXiv:2410.08201. — [arXiv:2410.08201](https://arxiv.org/abs/2410.08201) [find](https://scholar.google.com/scholar?q=Anish+Mudide%2C+Joshua+Engels%2C+Eric+J+Michaud%2C+Max+Tegmark%2C+and+Christian+Schroeder+de+Witt%2E+2024%2E+Efficient+dictionary+learning+with+switch+sparse+autoencoders%2E+arXiv+preprint+arXiv%3A2410%2E08201%2E+%E2%80%94+%5BarXi)
30. Neel Nanda, Andrew Lee, and Martin Wattenberg. 2023. [Emergent linear representations in world models of self-supervised sequence models](https://md.research.elamur.ai/free/en/papers/nanda-2023-emergent-linear-representations-world-models/). arXiv preprint arXiv:2309.00941.
31. Chris Olah. 2023. Distributed representations: Composition & superposition. Transformer Circuits Thread, 27. [find](https://scholar.google.com/scholar?q=Chris+Olah%2E+2023%2E+Distributed+representations%3A+Composition+%26+superposition%2E+Transformer+Circuits+Thread%2C+27%2E)
32. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. Zoom in: An introduction to circuits. Distill, 5(3):e00024– 001. [find](https://scholar.google.com/scholar?q=Chris+Olah%2C+Nick+Cammarata%2C+Ludwig+Schubert%2C+Gabriel+Goh%2C+Michael+Petrov%2C+and+Shan+Carter%2E+2020%2E+Zoom+in%3A+An+introduction+to+circuits%2E+Distill%2C+5%283%29%3Ae00024%E2%80%93+001%2E)
33. OpenAI. 2024. GPT-4.1 Mini - OpenAI API. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2024%2E+GPT%2D4%2E1+Mini+%2D+OpenAI+API%2E)
34. Kiho Park, Yo Joong Choe, and Victor Veitch. 2023. [The linear representation hypothesis and the geometry of large language models](https://md.research.elamur.ai/free/en/papers/park-2023-linear-representation-hypothesis-geometry-large/). arXiv preprint arXiv:2311.03658.
35. Matthew E Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018. [Dissecting contextual word embeddings: Architecture and representation](https://md.research.elamur.ai/free/en/papers/peters-2018-dissecting-contextual-word-embeddings-architecture/). arXiv preprint arXiv:1808.08949.
36. Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. 2025. A practical review of mechanistic interpretability for transformer-based language models. Preprint, arXiv:2407.02646. [find](https://scholar.google.com/scholar?q=Daking+Rai%2C+Yilun+Zhou%2C+Shi+Feng%2C+Abulhair+Saparov%2C+and+Ziyu+Yao%2E+2025%2E+A+practical+review+of+mechanistic+interpretability+for+transformer%2Dbased+language+models%2E+Preprint%2C+arXiv%3A2407%2E02646%2E)
37. Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. 2024a. [Improving dictionary learning with gated sparse autoencoders](https://md.research.elamur.ai/free/en/papers/rajamanoharan-2024-improving-dictionary-learning-gated-sparse/). arXiv preprint arXiv:2404.16014.
38. Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. 2024b. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407.14435. — [arXiv:2407.14435](https://arxiv.org/abs/2407.14435) [find](https://scholar.google.com/scholar?q=Senthooran+Rajamanoharan%2C+Tom+Lieberum%2C+Nicolas+Sonnerat%2C+Arthur+Conmy%2C+Vikrant+Varma%2C+J%C3%A1nos+Kram%C3%A1r%2C+and+Neel+Nanda%2E+2024b%2E+Jumping+ahead%3A+Improving+reconstruction+fidelity+with+jumprelu+sparse+autoen)
39. Samuel Marks, Adam Karvonen, and Aaron Mueller. 2024b. dictionary_learning. [https://github.com/saprmarks/dictionary_learning](https://github.com/saprmarks/dictionary_learning). [↗](https://github.com/saprmarks/dictionary_learning)
40. Dong Shu, Xuansheng Wu, Haiyan Zhao, Daking Rai, Ziyu Yao, Ninghao Liu, and Mengnan Du. 2025. A survey on sparse autoencoders: Interpreting the internal mechanisms of large language models. Preprint, arXiv:2503.05613. [find](https://scholar.google.com/scholar?q=Dong+Shu%2C+Xuansheng+Wu%2C+Haiyan+Zhao%2C+Daking+Rai%2C+Ziyu+Yao%2C+Ninghao+Liu%2C+and+Mengnan+Du%2E+2025%2E+A+survey+on+sparse+autoencoders%3A+Interpreting+the+internal+mechanisms+of+large+language+models%2E+Preprint%2C+)
41. Samuel Marks and Max Tegmark. 2023. [The geometry of truth: Emergent linear structure in large language model representations of true/false datasets](https://md.research.elamur.ai/free/en/papers/marks-2023-geometry-truth-emergent-linear-structure/). arXiv preprint arXiv:2310.06824.
42. Glen M. Taggart. 2024. Prolu: A nonlinearity for sparse autoencoders. [https://www.alignmentforum.org/posts/HEpufTdakGTTKgoYF/prolu-a-nonli](https://www.alignmentforum.org/posts/HEpufTdakGTTKgoYF/prolu-a-nonli) nearity-for-sparse-autoencoders. [↗](https://www.alignmentforum.org/posts/HEpufTdakGTTKgoYF/prolu-a-nonli)
43. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Morgane+Riviere%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+L%C3%A9onard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Ram%C3%A9%2C+and+1+others%2E+2024%2E+Gemma+2%3A+Impr)
44. Adly Templeton, Tom Conerly, Joshua Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Jermyn, Nicholas L. Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, and 6 others. 2024. S… [find](https://scholar.google.com/scholar?q=Adly+Templeton%2C+Tom+Conerly%2C+Joshua+Marcus%2C+Jack+Lindsey%2C+Trenton+Bricken%2C+Brian+Chen%2C+Adam+Jermyn%2C+Nicholas+L%2E+Turner%2C+Cem+Anil%2C+Carson+Denison%2C+Amanda+Askell%2C+Robert+Lasenby%2C+Yifan+Wu%2C+Shauna+Kravec)
45. Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. 2023. [Linear representations of sentiment in large language models](https://md.research.elamur.ai/free/en/papers/tigges-2023-linear-representations-sentiment-large-language/). arXiv preprint arXiv:2310.15154.
46. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. [Steering language models with activation engineering](https://md.research.elamur.ai/free/en/papers/turner-2023-steering-language-models-activation-engineering/). arXiv preprint arXiv:2308.10248.
47. Dimitri Von Rütte, Sotiris Anagnostidis, Gregor Bachmann, and Thomas Hofmann. 2024. A language model’s guide through latent space. arXiv preprint arXiv:2402.14433. — [arXiv:2402.14433](https://arxiv.org/abs/2402.14433) [find](https://scholar.google.com/scholar?q=Dimitri+Von+R%C3%BCtte%2C+Sotiris+Anagnostidis%2C+Gregor+Bachmann%2C+and+Thomas+Hofmann%2E+2024%2E+A+language+model%E2%80%99s+guide+through+latent+space%2E+arXiv+preprint+arXiv%3A2402%2E14433%2E+%E2%80%94+%5BarXiv%3A2402%2E14433%5D%28https%3A%2F%2Farxiv%2Eo)
48. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). Preprint, arXiv:2505.09388.
49. Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. 2023. [Explainability for large language models: A survey](https://md.research.elamur.ai/free/en/papers/zhao-2023-explainability-large-language-models-survey/). Preprint, arXiv:2309.01029.

*Updated 2 November 2026*