---
title: "Exploring Attention Attractors in Large Language Models (2026)"
description: "This paper studies \"attention attractors,\" tokens in large language models that receive an unusually large share of attention. 33 references."
authors: ["Ziheng Wang", "Zihao Yue", "Wenxuan Wang", "Qin Jin"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.51/"
doi: "10.18653/v1/2026.acl-long.51"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2026-exploring-attention-attractors-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2026-exploring-attention-attractors-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/) / Exploring Attention Attractors in Large Language Models

# Exploring Attention Attractors in Large Language Models

**Authors:** [Ziheng Wang](https://md.research.elamur.ai/free/en/authors/ziheng-wang/), [Zihao Yue](https://md.research.elamur.ai/free/en/authors/zihao-yue/), [Wenxuan Wang](https://md.research.elamur.ai/free/en/authors/wenxuan-wang/) and [Qin Jin](https://md.research.elamur.ai/free/en/authors/qin-jin/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.51/) · [PDF](https://aclanthology.org/2026.acl-long.51.pdf) · [DOI 10.18653/v1/2026.acl-long.51](https://doi.org/10.18653/v1/2026.acl-long.51) · [OpenAlex](https://openalex.org/W7166880451)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
This paper studies "attention attractors," tokens in large language models that receive an unusually large share of attention. Earlier work has explained these tokens in conflicting ways: as summary tokens, as attention sinks, or as key tokens for KV cache compression. The authors examine them from three angles.

- **Function:** They analyse attention patterns and saliency-based information flow in Llama2-7b on long texts from the Quality test set. Attractors look further back into the context than ordinary tokens. Information moves from the context into attractors, and from attractors on to prediction positions, so they act as aggregation hubs.
- **Distribution:** Layer-wise and token-wise analyses of Llama2-7b, Llama3-8b and Mistral-7b show that attractors cluster in the early and middle layers. They are mostly low-semantic tokens.
- **Mechanism:** The attention a token receives correlates with the values of specific activation dimensions. This suggests attractors are already marked in hidden states before attention is computed.

In long-context and streaming pruning experiments, two KV cache strategies largely preserve task performance: keeping only low-semantic tokens, or ranking tokens by a single activation dimension. This matters for both interpretability and efficient inference.

## Key points
- Tokens are classified as attractors when later tokens direct no less than 0.03% of their attention to them. The functional analysis uses 232 Quality texts that fill Llama2-7b's 4,096-token window.
- Saliency scores, computed from the loss on the last 128 tokens, reveal a path: Context Tokens → Attention Attractors → Prediction Tokens.
- Typical attractors in shallow layers are "_the", ".", and "\n". In middle layers, "’" dominates for Llama2 and Mistral. Llama3 instead favours "didn"/"don", which usually come right before it.
- The activation-based pruning policy is compared with Full Cache and L2Norm on LongBench. Tested models are Llama2-7b-Chat, Llama2-13b-Chat, Llama3-8b-Instruct and Qwen2-7b-Instruct.
- A stated limitation is that only relatively small Llama, Mistral and Qwen models were studied. Code is promised at github.com/luyouqi233/AttentionAttractor.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Renmin University of China](https://md.research.elamur.ai/free/en/universities/renmin-university-of-china/) — University, Beijing, China · 114 works · 16,118 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Ziheng Wang, Zihao Yue, Wenxuan Wang and Qin Jin. 2026. Exploring Attention Attractors in Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2026exploring,
  title = {{Exploring Attention Attractors in Large Language Models}},
  author = {Ziheng Wang, Zihao Yue, Wenxuan Wang and Qin Jin},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.51},
  url = {https://doi.org/10.18653/v1/2026.acl-long.51}
}
```

- [← previous Explore Political Discourse with Transformers. Emergent Paradigmatic and Syntagmatic Representations.](https://md.research.elamur.ai/free/en/papers/vanni-2026-explore-political-discourse-transformers-emergent/)
- [next → Exploring Cognitively Informed Sentence Simplification with Gaze-Guided Text Generation](https://md.research.elamur.ai/free/en/papers/sauberli-2026-exploring-cognitively-informed-sentence-simplification/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/)

## References

33 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. BAAI. 2024. Infinity instruct. [find](https://scholar.google.com/scholar?q=BAAI%2E+2024%2E+Infinity+instruct%2E)
2. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A bilingual, multi-task benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Associat… [find](https://scholar.google.com/scholar?q=Yushi+Bai%2C+Xin+Lv%2C+Jiajie+Zhang%2C+Hongchang+Lyu%2C+Jiankai+Tang%2C+Zhidian+Huang%2C+Zhengxiao+Du%2C+Xiao+Liu%2C+Aohan+Zeng%2C+Lei+Hou%2C+Yuxiao+Dong%2C+Jie+Tang%2C+and+Juanzi+Li%2E+2024%2E+LongBench%3A+A+bilingual%2C+multi%2Dtask)
3. Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. 2024. [Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling](https://md.research.elamur.ai/free/en/papers/cai-2024-pyramidkv-dynamic-kv-cache-compression/).
4. Nicola Cancedda. 2024. Spectral filters, dark signals, and attention sinks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4792– 4808, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nicola+Cancedda%2E+2024%2E+Spectral+filters%2C+dark+signals%2C+and+attention+sinks%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29%2C+pages+47)
5. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+and+Eric+P%2E+Xing%2E+2023%2E+Vicuna%3A+An+open%2Dsource+cha)
6. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT‘s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy. Association for Computationa… [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Urvashi+Khandelwal%2C+Omer+Levy%2C+and+Christopher+D%2E+Manning%2E+2019%2E+What+does+BERT+look+at%3F+an+analysis+of+BERT%E2%80%98s+attention%2E+In+Proceedings+of+the+2019+ACL+Workshop+BlackboxNLP%3A+Analyzing+an)
7. Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. 2024. Vision transformers need registers. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Timoth%C3%A9e+Darcet%2C+Maxime+Oquab%2C+Julien+Mairal%2C+and+Piotr+Bojanowski%2E+2024%2E+Vision+transformers+need+registers%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2E)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
9. Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini. 2024. A simple and effective l_2 norm-based strategy for KV cache compression. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18476–18499, Miami, Florida, USA. Association for Comput… [find](https://scholar.google.com/scholar?q=Alessio+Devoto%2C+Yu+Zhao%2C+Simone+Scardapane%2C+and+Pasquale+Minervini%2E+2024%2E+A+simple+and+effective+l%5F2+norm%2Dbased+strategy+for+KV+cache+compression%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Me)
10. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). In Intern…
11. Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2024. [Model tells you what to discard: Adaptive KV cache compression for LLMs](https://md.research.elamur.ai/free/en/papers/ge-2023-model-tells-discard-adaptive-kv/). In The Twelfth International Conference on Learning Representations.
12. Aaron Grattafiori, Abhimanyu Dubey, and Abhinav Jauhri et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/).
13. Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vis… [find](https://scholar.google.com/scholar?q=Qidong+Huang%2C+Xiaoyi+Dong%2C+Pan+Zhang%2C+Bin+Wang%2C+Conghui+He%2C+Jiaqi+Wang%2C+Dahua+Lin%2C+Weiming+Zhang%2C+and+Nenghai+Yu%2E+2024%2E+Opera%3A+Alleviating+hallucination+in+multi%2Dmodal+large+language+models+via+over%2Dt)
14. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
15. Jianan Li, Quan Tu, Cunli Mao, Zhengtao Yu, Ji-Rong Wen, and Rui Yan. 2024. Streamingdialogue: Prolonged dialogue learning via long context compression with minimal losses. In Advances in Neural Information Processing Systems, volume 37, pages 86074–86101. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Jianan+Li%2C+Quan+Tu%2C+Cunli+Mao%2C+Zhengtao+Yu%2C+Ji%2DRong+Wen%2C+and+Rui+Yan%2E+2024%2E+Streamingdialogue%3A+Prolonged+dialogue+learning+via+long+context+compression+with+minimal+losses%2E+In+Advances+in+Neural+Infor)
16. Xin Li and Dan Roth. 2002. [Learning question classifiers](https://md.research.elamur.ai/free/en/papers/li-2002-learning-question-classifiers/). In COLING 2002: The 19th International Conference on Computational Linguistics.
17. QwenTeam. 2024. Qwen2 technical report. [find](https://scholar.google.com/scholar?q=QwenTeam%2E+2024%2E+Qwen2+technical+report%2E)
18. Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. [Compressive transformers for long-range sequence modelling](https://md.research.elamur.ai/free/en/papers/rae-2019-compressive-transformers-long-range-sequence/). In International Conference on Learning Representations.
19. Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. [Deep inside convolutional networks: Visualising image classification models and saliency maps](https://md.research.elamur.ai/free/en/papers/simonyan-2013-deep-inside-convolutional-networks-visualising/). In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings.
20. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063.
21. Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. 2024. Massive activations in large language models. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models. [find](https://scholar.google.com/scholar?q=Mingjie+Sun%2C+Xinlei+Chen%2C+J+Zico+Kolter%2C+and+Zhuang+Liu%2E+2024%2E+Massive+activations+in+large+language+models%2E+In+ICLR+2024+Workshop+on+Mathematical+and+Empirical+Understanding+of+Foundation+Models%2E)
22. Hugo Touvron, Louis Martin, and Kevin Stone et al. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/).
23. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). Transactions of the Association for Computational Linguistics, 12:157–173.
24. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
25. Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2023. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. In Advances in Neural Information Processing Systems, v… [find](https://scholar.google.com/scholar?q=Zichang+Liu%2C+Aditya+Desai%2C+Fangshuo+Liao%2C+Weitao+Wang%2C+Victor+Xie%2C+Zhaozhuo+Xu%2C+Anastasios+Kyrillidis%2C+and+Anshumali+Shrivastava%2E+2023%2E+Scissorhands%3A+Exploiting+the+persistence+of+importance+hypothesi)
26. Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023. [Label words are anchors: An information flow perspective for understanding in-context learning](https://md.research.elamur.ai/free/en/papers/wang-2023-label-words-anchors-information-flow/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9840–9855, Sin…
27. OpenAI. 2024. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
28. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+2024%2E+Efficient+streaming+language+models+with+attention+sinks%2E+In+The+Twelfth+International+Conference+on+Learning+Representations)
29. Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel Bowman. 2022. [QuALITY: Question answering with long input texts, yes!](https://md.research.elamur.ai/free/en/papers/pang-2022-quality-question-answering-long-input/) In Proceedings of the 2022 Conference of the North American Chapter of…
30. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. [The fineweb datasets: Decanting the web for the finest text data at scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/). In The Thirty-eight Conference on Neural Information Processing Systems Datasets a…
31. Yao Yao, Zuchao Li, and Hai Zhao. 2024. SirLLM: Streaming infinite retentive LLM. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2611–2624, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yao+Yao%2C+Zuchao+Li%2C+and+Hai+Zhao%2E+2024%2E+SirLLM%3A+Streaming+infinite+retentive+LLM%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29%2C+pa)
32. Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. 2024. Unveiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration. In Forty-first International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Zhongzhi+Yu%2C+Zheng+Wang%2C+Yonggan+Fu%2C+Huihong+Shi%2C+Khalid+Shaikh%2C+and+Yingyan+Celine+Lin%2E+2024%2E+Unveiling+and+harnessing+hidden+attention+sinks%3A+Enhancing+large+language+models+without+training+through)
33. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang "Atlas" Wang, and Beidi Chen. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Informati… [find](https://scholar.google.com/scholar?q=Zhenyu+Zhang%2C+Ying+Sheng%2C+Tianyi+Zhou%2C+Tianlong+Chen%2C+Lianmin+Zheng%2C+Ruisi+Cai%2C+Zhao+Song%2C+Yuandong+Tian%2C+Christopher+R%C3%A9%2C+Clark+Barrett%2C+Zhangyang+%22Atlas%22+Wang%2C+and+Beidi+Chen%2E+2023%2E+H2o%3A+Heavy%2Dhitter)

*Updated 1 November 2026*