---
title: "A Mechanistic Account of Attention Sinks in GPT-2: One Circuit,"
description: "Transformers often send a large share of attention to the first token regardless of its content. 3 references."
authors: ["Yuval Ran-Milo", "Hila Ofek", "Shahar Mendel"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/2026.acl-short.9/"
doi: "10.18653/v1/2026.acl-short.9"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/milo-2026-mechanistic-account-attention-sinks-gpt/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/milo-2026-mechanistic-account-attention-sinks-gpt/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/20/) / A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation

# A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation

**Authors:** [Yuval Ran-Milo](https://md.research.elamur.ai/free/en/authors/yuval-ran-milo/), Hila Ofek and Shahar Mendel · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-short.9/) · [PDF](https://aclanthology.org/2026.acl-short.9.pdf) · [DOI 10.18653/v1/2026.acl-short.9](https://doi.org/10.18653/v1/2026.acl-short.9) · [OpenAlex](https://openalex.org/W7166890124)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Transformers often send a large share of attention to the first token regardless of its content. This "attention sink" hurts context use, quantization, streaming and interpretability. The authors trace how the sink is produced inside the 124M-parameter GPT-2, which uses learned query biases and absolute positional embeddings.

They combine structural analysis of attention scores with ten causal interventions, using inputs from natural language, math and code. The sink comes from three interacting parts:
- **The query bias b_Q.** It contributes a score term that favours a given target position for every source token alike.
- **The first-layer MLP.** It turns the first positional embedding into an "effective positional encoding" (EPE) with massive activations on a few coordinates.
- **The key projection W_k.** Its structure lines up those coordinates with the bias direction.

Breaking any of the three weakens, removes or relocates the sink. Other architectures still show sinks while lacking each of these parts. The authors therefore argue that sinks arise through different circuits in different models. As a result, mitigations may need to be tailored to each architecture, and pre-training fixes may not transfer.

## Key points
- **Data:** 300 inputs, 100 each from SST-2, GSM8K and HumanEval. All were cut to exactly 40 tokens, and per-head analyses cover layers 4–11 of 12.
- **EPE fidelity:** EPE (first-layer MLP output plus the positional vector) closely matches the actual positional signal. Its cosine similarity exceeds 0.82 at every position and 0.99 at position 1.
- **Causal tests:**
  - Nullifying b_Q sharply reduces the sink.
  - Zeroing the three W_k columns at EPE₁'s massive-activation coordinates also reduces it, while zeroing three random columns does almost nothing.
- **Ruled-out explanations:** The sink survives when the BOS embedding is zeroed and when only raw positional embeddings are swapped. This points to the MLP-processed signal rather than token identity.
- **Code:** Released at github.com/YuvMilo/MechanisticAccountofSinks.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tel Aviv University](https://md.research.elamur.ai/free/en/universities/tel-aviv-university/) — University, Tel Aviv, Israel · 82 works · 10,655 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🔬 Interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/) · 130 works

## Cite

Cite as: Yuval Ran-Milo, Hila Ofek and Shahar Mendel. 2026. A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ranmilo2026mechanistic,
  title = {{A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation}},
  author = {Yuval Ran-Milo, Hila Ofek and Shahar Mendel},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-short.9},
  url = {https://doi.org/10.18653/v1/2026.acl-short.9}
}
```

- [← previous A Longitudinal, Multinational, and Multilingual Corpus of News Coverage of the Russo-Ukrainian War](https://md.research.elamur.ai/free/en/papers/mohanty-2026-longitudinal-multinational-multilingual-corpus-news/)
- [next → A Method for Learning Large-Scale Computational Construction Grammars from Semantically Annotated Corpora](https://md.research.elamur.ai/free/en/papers/eecke-2026-method-learning-large-scale-computational/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/20/)

## References

3 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Federico Barbero, Álvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, and Razvan Pascanu. 2025. Why do llms attend to the first token? Preprint, arXiv:2504.02732. — [arXiv:2504.02732](https://arxiv.org/abs/2504.02732) [find](https://scholar.google.com/scholar?q=Federico+Barbero%2C+%C3%81lvaro+Arroyo%2C+Xiangming+Gu%2C+Christos+Perivolaropoulos%2C+Michael+Bronstein%2C+Petar+Velic%CC%8Ckovic%CC%81%2C+and+Razvan+Pascanu%2E+2025%2E+Why+do+llms+attend+to+the+first+token%3F+Preprint%2C+arXiv%3A2504%2E0)
2. Nicola Cancedda. 2024. Spectral filters, dark signals, and attention sinks. Preprint, arXiv:2402.09221. [find](https://scholar.google.com/scholar?q=Nicola+Cancedda%2E+2024%2E+Spectral+filters%2C+dark+signals%2C+and+attention+sinks%2E+Preprint%2C+arXiv%3A2402%2E09221%2E)
3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 202… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)

*Updated 1 November 2026*