---
title: "AtP* — paper by Kramar et al. (2024) — summary and ideas"
description: "This method replaces one component's activation with its value from a corrupted prompt and… 37 references, cited by 32 papers here."
authors: ["Janos Kramar", "Tom Lieberum", "Rohin Shah", "Neel Nanda"]
year: 2024
venue: "arXiv:2403.00745"
source: "https://arxiv.org/abs/2403.00745"
doi: "10.48550/arxiv.2403.00745"
arxiv: "2403.00745"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kramar-2024-atp-efficient-scalable-method-localizing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kramar-2024-atp-efficient-scalable-method-localizing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/) / AtP\*: An efficient and scalable method for localizing LLM behaviour to components

# AtP\*: An efficient and scalable method for localizing LLM behaviour to components

**Authors:** [Janos Kramar](https://md.research.elamur.ai/free/en/authors/janos-kramar/), [Tom Lieberum](https://md.research.elamur.ai/free/en/authors/tom-lieberum/), [Rohin Shah](https://md.research.elamur.ai/free/en/authors/rohin-shah/) and [Neel Nanda](https://md.research.elamur.ai/free/en/authors/neel-nanda/) · **Year:** 2024 · **Venue:** arXiv:2403.00745 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2403.00745) · [PDF](https://arxiv.org/pdf/2403.00745) · [DOI 10.48550/arxiv.2403.00745](https://doi.org/10.48550/arxiv.2403.00745) · [arXiv 2403.00745](https://arxiv.org/abs/2403.00745) · [OpenAlex](https://openalex.org/W4392428679)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [32](https://md.research.elamur.ai/free/en/papers/kramar-2024-atp-efficient-scalable-method-localizing/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Mechanistic interpretability often relies on Activation Patching. This method replaces one component's activation with its value from a corrupted prompt and measures how a behaviour metric changes. Run exhaustively, it needs a separate forward pass for every component, which is impractical for large LLMs with billions of neurons. The Google DeepMind authors study Attribution Patching (AtP). AtP is a first-order gradient approximation that scores all nodes at once. The highest-scoring nodes are then checked with real patching.

They identify two sources of false negatives in AtP:
- **Saturated attention softmax**, where the local gradient misrepresents the true effect.
- **Cancellation** between direct and indirect effects.

Their variant, AtP*, addresses these with two changes. It recomputes the softmax exactly when patching queries and keys, and it applies dropout in the backward pass. In the first systematic comparison against several faster patching baselines on Pythia models, AtP outperforms all of them, and AtP* improves further. A subsampling-based diagnostic statistically bounds any remaining missed effects. Together, these make fine-grained causal localization feasible at scale.

## Key points
- Exhaustive patching is costly: a 1024-token prompt in Chinchilla 70B has about 2.7·10⁹ neuron nodes. AtP scores every node with only two forward passes and one backward pass per prompt pair.
- AtP* combines two fixes, QK-fix (exact softmax recomputation) and GradDrop (backward-pass dropout). GradDrop has a noticeable upfront cost but reduces false negatives.
- The evaluation covers MLP neurons and attention nodes on prompt-pair tasks such as CITY-PP and IOI-PP, using Pythia models including Pythia-12B. Cost is measured in forward passes relative to an oracle verification order.
- The authors give practical recommendations depending on the setting:
  - AtP* for single prompt pairs.
  - AtP+QKFix for attention nodes over distributions.
  - Plain AtP for neuron nodes over distributions.
- The diagnostic first filters out high-impact nodes with AtP, then patches random subsets of the remaining nodes to bound undetected effects.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google DeepMind](https://md.research.elamur.ai/free/en/companies/google-deepmind/) — Company, London, United Kingdom · 82 works · 84,829 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [💻 Software](https://md.research.elamur.ai/free/en/shelves/computer-science/software/) · 150 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works

## Cite

Cite as: Janos Kramar, Tom Lieberum, Rohin Shah and Neel Nanda. 2024. AtP\*: An efficient and scalable method for localizing LLM behaviour to components. arXiv:2403.00745.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{kramar2024atp,
  title = {{AtP*: An efficient and scalable method for localizing LLM behaviour to components}},
  author = {Janos Kramar, Tom Lieberum, Rohin Shah and Neel Nanda},
  year = {2024},
  doi = {10.48550/arxiv.2403.00745},
  eprint = {2403.00745},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.00745}
}
```

- [← previous ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews](https://md.research.elamur.ai/free/en/papers/arcy-2024-aries-corpus-scientific-paper-edits/)
- [next → ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild](https://md.research.elamur.ai/free/en/papers/masry-2024-chartgemma-visual-instruction-tuning-chart/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/)

## References

37 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. N. Belrose, D. Schneider-Joseph, S. Ravfogel, R. Cotterell, E. Raff, and S. Biderman. Leace: Perfect linear concept erasure in closed form. arXiv preprint arXiv:2306.03819, 2023. — [arXiv:2306.03819](https://arxiv.org/abs/2306.03819) [find](https://scholar.google.com/scholar?q=N%2E+Belrose%2C+D%2E+Schneider%2DJoseph%2C+S%2E+Ravfogel%2C+R%2E+Cotterell%2C+E%2E+Raff%2C+and+S%2E+Biderman%2E+Leace%3A+Perfect+linear+concept+erasure+in+closed+form%2E+arXiv+preprint+arXiv%3A2306%2E03819%2C+2023%2E+%E2%80%94+%5BarXiv%3A2306%2E03819%5D%28)
2. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engel… [find](https://scholar.google.com/scholar?q=S%2E+Biderman%2C+H%2E+Schoelkopf%2C+Q%2E+G%2E+Anthony%2C+H%2E+Bradley%2C+K%2E+O%E2%80%99Brien%2C+E%2E+Hallahan%2C+M%2E+A%2E+Khan%2C+S%2E+Purohit%2C+U%2E+S%2E+Prashanth%2C+E%2E+Raff%2C+A%2E+Skowron%2C+L%2E+Sutawika%2C+and+O%2E+van+der+Wal%2E+Pythia%3A+A+suite+for+analy)
3. PMLR, 2023. URL [https://proceedings.mlr.press/v202/](https://proceedings.mlr.press/v202/) biderman23a.html . [↗](https://proceedings.mlr.press/v202/)
4. T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter, T. Henighan, and C. Olah. Towards mon… [find](https://scholar.google.com/scholar?q=T%2E+Bricken%2C+A%2E+Templeton%2C+J%2E+Batson%2C+B%2E+Chen%2C+A%2E+Jermyn%2C+T%2E+Conerly%2C+N%2E+Turner%2C+C%2E+Anil%2C+C%2E+Denison%2C+A%2E+Askell%2C+R%2E+Lasenby%2C+Y%2E+Wu%2C+S%2E+Kravec%2C+N%2E+Schiefer%2C+T%2E+Maxwell%2C+N%2E+Joseph%2C+Z%2E+Hatfield%2DDodds%2C+A%2E+)
5. N. Cammarata, S. Carter, G. Goh, C. Olah, M. Petrov, L. Schubert, C. Voss, B. Egan, and S. K. Lim. Thread: Circuits. Distill, 2020. doi: 10.23915/distill.00024. [https://distill.pub/2020/circuits](https://distill.pub/2020/circuits). — [doi:10.23915/distill.00024](https://doi.org/10.23915/distill.00024) [↗](https://distill.pub/2020/circuits)
6. N. D. Cao, L. Schmid, D. Hupkes, and I. Titov. Sparse interventions in language models with differentiable masking, 2021. [find](https://scholar.google.com/scholar?q=N%2E+D%2E+Cao%2C+L%2E+Schmid%2C+D%2E+Hupkes%2C+and+I%2E+Titov%2E+Sparse+interventions+in+language+models+with+differentiable+masking%2C+2021%2E)
7. L. Chan, A. Garriga-Alonso, N. Goldwosky-Dill, R. Greenblatt, J. Nitishinskaya, A. Radhakrishnan, B. Shlegeris, and N. Thomas. Causal scrubbing, a method for rigorously testing interpretability hypotheses. AI Alignment Forum, 2022. [https://www.alignmentforum.org/posts/](https://www.alignmentforum.org/posts/) JvZhhzycHu2Yd57RN/causal-scrub… [↗](https://www.alignmentforum.org/posts/)
8. A. Conmy, A. N. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Conmy%2C+A%2E+N%2E+Mavor%2DParker%2C+A%2E+Lynch%2C+S%2E+Heimersheim%2C+and+A%2E+Garriga%2DAlonso%2E+Towards+automated+circuit+discovery+for+mechanistic+interpretability%2C+2023%2E)
9. H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey. Sparse autoencoders find highly interpretable features in language models, 2023. [find](https://scholar.google.com/scholar?q=H%2E+Cunningham%2C+A%2E+Ewart%2C+L%2E+Riggs%2C+R%2E+Huben%2C+and+L%2E+Sharkey%2E+Sparse+autoencoders+find+highly+interpretable+features+in+language+models%2C+2023%2E)
10. J. Feng and J. Steinhardt. How do language models bind entities in context?, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Feng+and+J%2E+Steinhardt%2E+How+do+language+models+bind+entities+in+context%3F%2C+2023%2E)
11. M. Figurnov, A. Ibraimova, D. P. Vetrov, and P. Kohli. Perforatedcnns: Acceleration through elimination of redundant convolutions. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL h… [find](https://scholar.google.com/scholar?q=M%2E+Figurnov%2C+A%2E+Ibraimova%2C+D%2E+P%2E+Vetrov%2C+and+P%2E+Kohli%2E+Perforatedcnns%3A+Acceleration+through+elimination+of+redundant+convolutions%2E+In+D%2E+Lee%2C+M%2E+Sugiyama%2C+U%2E+Luxburg%2C+I%2E+Guyon%2C+and+R%2E+Garnett%2C+editors)
12. M. Finlayson, A. Mueller, S. Gehrmann, S. Shieber, T. Linzen, and Y. Belinkov. Causal analysis of syntactic agreement mechanisms in neural language models. In C. Zong, F. Xia, W. Li, and R. Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and… [find](https://scholar.google.com/scholar?q=M%2E+Finlayson%2C+A%2E+Mueller%2C+S%2E+Gehrmann%2C+S%2E+Shieber%2C+T%2E+Linzen%2C+and+Y%2E+Belinkov%2E+Causal+analysis+of+syntactic+agreement+mechanisms+in+neural+language+models%2E+In+C%2E+Zong%2C+F%2E+Xia%2C+W%2E+Li%2C+and+R%2E+Navigli%2C+e)
13. L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. [find](https://scholar.google.com/scholar?q=L%2E+Gao%2C+S%2E+Biderman%2C+S%2E+Black%2C+L%2E+Golding%2C+T%2E+Hoppe%2C+C%2E+Foster%2C+J%2E+Phang%2C+H%2E+He%2C+A%2E+Thite%2C+N%2E+Nabeshima%2C+S%2E+Presser%2C+and+C%2E+Leahy%2E+The+Pile%3A+An+800gb+dataset+of+diverse+text+for+language+modeling%2E+arX)
14. A. Geiger, K. Richardson, and C. Potts. [Neural natural language inference models partially embed theories of lexical entailment and negation](https://md.research.elamur.ai/free/en/papers/geiger-2020-neural-natural-language-inference-models/), 2020.
15. A. Geiger, H. Lu, T. Icard, and C. Potts. Causal abstractions of neural networks, 2021. [find](https://scholar.google.com/scholar?q=A%2E+Geiger%2C+H%2E+Lu%2C+T%2E+Icard%2C+and+C%2E+Potts%2E+Causal+abstractions+of+neural+networks%2C+2021%2E)
16. A. Geiger, Z. Wu, H. Lu, J. Rozner, E. Kreiss, T. Icard, N. D. Goodman, and C. Potts. Inducing causal structure for interpretable neural networks, 2022. [find](https://scholar.google.com/scholar?q=A%2E+Geiger%2C+Z%2E+Wu%2C+H%2E+Lu%2C+J%2E+Rozner%2C+E%2E+Kreiss%2C+T%2E+Icard%2C+N%2E+D%2E+Goodman%2C+and+C%2E+Potts%2E+Inducing+causal+structure+for+interpretable+neural+networks%2C+2022%2E)
17. A. Geiger, C. Potts, and T. Icard. Causal abstraction for faithful model interpretation, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Geiger%2C+C%2E+Potts%2C+and+T%2E+Icard%2E+Causal+abstraction+for+faithful+model+interpretation%2C+2023%2E)
18. M. Geva, J. Bastings, K. Filippova, and A. Globerson. Dissecting recall of factual associations in auto-regressive language models, 2023. [find](https://scholar.google.com/scholar?q=M%2E+Geva%2C+J%2E+Bastings%2C+K%2E+Filippova%2C+and+A%2E+Globerson%2E+Dissecting+recall+of+factual+associations+in+auto%2Dregressive+language+models%2C+2023%2E)
19. N. Goldowsky-Dill, C. MacLeod, L. Sato, and A. Arora. [Localizing model behavior with path patching](https://md.research.elamur.ai/free/en/papers/goldowsky-dill-2023-localizing-model-behavior-path-patching/), 2023.
20. W. Gurnee, N. Nanda, M. Pauly, K. Harvey, D. Troitskii, and D. Bertsimas. [Finding neurons in a haystack: Case studies with sparse probing](https://md.research.elamur.ai/free/en/papers/gurnee-2023-finding-neurons-haystack-case-studies/), 2023.
21. M. Hanna, O. Liu, and A. Variengien. [How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://md.research.elamur.ai/free/en/papers/hanna-2023-gpt-2-compute-greater-than/), 2023.
22. P. Hase, M. Bansal, B. Kim, and A. Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models, 2023. [find](https://scholar.google.com/scholar?q=P%2E+Hase%2C+M%2E+Bansal%2C+B%2E+Kim%2C+and+A%2E+Ghandeharioun%2E+Does+localization+inform+editing%3F+surprising+differences+in+causality%2Dbased+localization+vs%2E+knowledge+editing+in+language+models%2C+2023%2E)
23. R. Hendel, M. Geva, and A. Globerson. [In-context learning creates task vectors](https://md.research.elamur.ai/free/en/papers/hendel-2023-context-learning-creates-task-vectors/), 2023.
24. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, O. Vinyals, J. Rae, and L. Sifre. An empirical analysis o… [find](https://scholar.google.com/scholar?q=J%2E+Hoffmann%2C+S%2E+Borgeaud%2C+A%2E+Mensch%2C+E%2E+Buchatskaya%2C+T%2E+Cai%2C+E%2E+Rutherford%2C+D%2E+de+Las+Casas%2C+L%2E+A%2E+Hendricks%2C+J%2E+Welbl%2C+A%2E+Clark%2C+T%2E+Hennigan%2C+E%2E+Noland%2C+K%2E+Millican%2C+G%2E+van+den+Driessche%2C+B%2E+Damoc%2C+A)
25. J. Huang, A. Geiger, K. D’Oosterlinck, Z. Wu, and C. Potts. [Rigorously assessing natural language explanations of neurons](https://md.research.elamur.ai/free/en/papers/huang-2023-rigorously-assessing-natural-language-explanations/), 2023.
26. E. Jenner, A. Garriga-Alonso, and E. Zverev. A comparison of causal scrubbing, causal abstractions, and related methods. AI Alignment Forum, 2022. [https://www.alignmentforum.org/posts/uLMWMeBG3ruoBRhMW/](https://www.alignmentforum.org/posts/uLMWMeBG3ruoBRhMW/) a-comparison-of-causal-scrubbing-causal-abstractions-and . [↗](https://www.alignmentforum.org/posts/uLMWMeBG3ruoBRhMW/)
27. O. Jorgensen, D. Cope, N. Schoots, and M. Shanahan. Improving activation steering in language models with mean-centring, 2023. [find](https://scholar.google.com/scholar?q=O%2E+Jorgensen%2C+D%2E+Cope%2C+N%2E+Schoots%2C+and+M%2E+Shanahan%2E+Improving+activation+steering+in+language+models+with+mean%2Dcentring%2C+2023%2E)
28. K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg. [Inference-time intervention: Eliciting truthful answers from a language model](https://md.research.elamur.ai/free/en/papers/li-2023-inference-time-intervention-eliciting-truthful/), 2023.
29. T. Lieberum, M. Rahtz, J. Kramár, N. Nanda, G. Irving, R. Shah, and V. Mikulik. [Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla](https://md.research.elamur.ai/free/en/papers/lieberum-2023-circuit-analysis-interpretability-scale-evidence/), 2023.
30. C. Louizos, M. Welling, and D. P. Kingma. Learning sparse neural networks through 𝑙 0 regularization, 2018. [find](https://scholar.google.com/scholar?q=C%2E+Louizos%2C+M%2E+Welling%2C+and+D%2E+P%2E+Kingma%2E+Learning+sparse+neural+networks+through+%F0%9D%91%99+0+regularization%2C+2018%2E)
31. C. McDougall, A. Conmy, C. Rushing, T. McGrath, and N. Nanda. [Copy suppression: Comprehensively understanding an attention head](https://md.research.elamur.ai/free/en/papers/mcdougall-2023-copy-suppression-comprehensively-understanding-attention/), 2023.
32. T. McGrath, M. Rahtz, J. Kramár, V. Mikulik, and S. Legg. [The hydra effect: Emergent self-repair in language model computations](https://md.research.elamur.ai/free/en/papers/mcgrath-2023-hydra-effect-emergent-self-repair/), 2023.
33. K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in gpt, 2023. [find](https://scholar.google.com/scholar?q=K%2E+Meng%2C+D%2E+Bau%2C+A%2E+Andonian%2C+and+Y%2E+Belinkov%2E+Locating+and+editing+factual+associations+in+gpt%2C+2023%2E)
34. J. Merullo, C. Eickhoff, and E. Pavlick. Circuit component reuse across tasks in transformer language models, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Merullo%2C+C%2E+Eickhoff%2C+and+E%2E+Pavlick%2E+Circuit+component+reuse+across+tasks+in+transformer+language+models%2C+2023%2E)
35. P. Michel, O. Levy, and G. Neubig. [Are sixteen heads really better than one?](https://md.research.elamur.ai/free/en/papers/michel-2019-sixteen-heads-really-better-than/) In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL [https://proceedings.neurips.cc/…](https://proceedings.neurips.cc/…) [↗](https://proceedings.neurips.cc/…)
36. P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations, 2017. URL [https://openreview.net/forum?id=SJGCiw5gl](https://openreview.net/forum?id=SJGCiw5gl) . [↗](https://openreview.net/forum?id=SJGCiw5gl)
37. N. Nanda. Attribution patching: Activation patching at industrial scale. 2022. URL [https://www.neelnanda.io/mechanistic-interpretability/attribution-patching](https://www.neelnanda.io/mechanistic-interpretability/attribution-patching) . [↗](https://www.neelnanda.io/mechanistic-interpretability/attribution-patching)

*Updated 3 November 2026*