---
title: "Attention Basin — paper by Yi et al. (2026) — summary and ideas"
description: "Building on this, the authors propose AttnRank. 1.58 points. 40 references, cited by 2 papers here."
authors: ["Zihao Yi", "Zhenqing Ling", "Delong Zeng", "Haohao Luo", "Zhe Xu", "Wei Liu", "Jian Luan", "Wanxia Cao", "Ying Shen"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1198/"
doi: "10.18653/v1/2026.acl-long.1198"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yi-2026-attention-basin-why-contextual-position/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yi-2026-attention-basin-why-contextual-position/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/) / Attention Basin: Why Contextual Position Matters in Large Language Models

# Attention Basin: Why Contextual Position Matters in Large Language Models

**Authors:** [Zihao Yi](https://md.research.elamur.ai/free/en/authors/zihao-yi/), Zhenqing Ling, Delong Zeng, [Haohao Luo](https://md.research.elamur.ai/free/en/authors/haohao-luo/), [Zhe Xu](https://md.research.elamur.ai/free/en/authors/zhe-xu/), [Wei Liu](https://md.research.elamur.ai/free/en/authors/wei-liu/), [Jian Luan](https://md.research.elamur.ai/free/en/authors/jian-luan/), Wanxia Cao and [Ying Shen](https://md.research.elamur.ai/free/en/authors/ying-shen/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1198/) · [PDF](https://aclanthology.org/2026.acl-long.1198.pdf) · [DOI 10.18653/v1/2026.acl-long.1198](https://doi.org/10.18653/v1/2026.acl-long.1198) · [OpenAlex](https://openalex.org/W7166861478)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 2 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models handle information unevenly depending on where it appears in the prompt. This paper asks why. Across 10 LLMs, the authors measured how much attention query tokens pay to each block in structured inputs, such as sets of retrieved documents or few-shot examples. Blocks at the start and end of the set receive the most attention and middle blocks the least, a U-shaped pattern they call the *attention basin*. When punctuation, capitalization and delimiters like "Document [1]" are stripped, the pattern disappears. This suggests the bias comes from the model recognizing segment boundaries, not from raw token position. A proposition (proof in Appendix C) argues that raising attention on the correct document is the most effective way to raise the correct answer's probability. Building on this, the authors propose AttnRank. It first profiles a model's positional attention preferences on a small calibration set. It then places the most relevant items in the high-attention slots. The method needs no training, works with any model, and adds little cost. It improves multi-hop QA and few-shot dialogue state tracking, offering a cheap fix for "lost-in-the-middle" failures in RAG pipelines.

## Key points
- **Attention basin:** a structure-level counterpart of token-level primacy/recency effects. It appeared consistently across models such as LLAMA3-8B-Instruct, Mistral-7B-Instruct, InternLM3-8B-Instruct and DeepSeek-R1-Distill variants, and vanished once structural delimiters were removed.
- **Multi-hop QA (2WikiMultiHopQA):** five retrieved documents per question, 10 models from 1.5B to 13B parameters (Qwen 2.5, LLAMA, DeepSeek, Mistral, InternLM3). AttnRank achieved the highest average accuracy of the five ordering strategies tested (34.72%), ahead of the LIM-based ordering of Liu et al., 2024 (32.10%).
- **Few-shot setting:** examples were retrieved with IC-DST and fed to Code Llama. AttnRank reached a Joint Goal Accuracy of 42.89% on MultiWOZ 2.1 and 51.51% on MultiWOZ 2.4 (average 47.20%), which is 1.58 points above random ordering.
- **Practical properties:** the method is plug-and-play and leaves model parameters untouched. It is compatible with existing acceleration frameworks.
- **Limitations:** the method requires access to attention scores, which most closed-source APIs do not expose. The theory rests on simplifying assumptions, such as roughly independent document representations. The authors suggest future work on reducing the bias itself, for example through attention modulation or bias-aware fine-tuning.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Sun Yat-sen University](https://md.research.elamur.ai/free/en/universities/sun-yat-sen-university/) — University, Guangzhou, China · 53 works · 9,185 citations
- [Xiaomi](https://md.research.elamur.ai/free/en/companies/xiaomi/) — Company, Beijing, China · 9 works · 334 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Zihao Yi, Zhenqing Ling, Delong Zeng, Haohao Luo, Zhe Xu, Wei Liu, Jian Luan, Wanxia Cao and Ying Shen. 2026. Attention Basin: Why Contextual Position Matters in Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yi2026attention,
  title = {{Attention Basin: Why Contextual Position Matters in Large Language Models}},
  author = {Zihao Yi, Zhenqing Ling, Delong Zeng, Haohao Luo, Zhe Xu, Wei Liu, Jian Luan, Wanxia Cao and Ying Shen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1198},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1198}
}
```

- [← previous Attacker's Noise Can Manipulate Your Audio-based LLM in the Real World](https://md.research.elamur.ai/free/en/papers/sadasivan-2026-attacker-s-noise-can-manipulate/)
- [next → Author-in-the-Loop Response Generation and Evaluation: Integrating Author Expertise and Intent in Responses to Peer Review](https://md.research.elamur.ai/free/en/papers/ruan-2026-author-loop-response-generation-evaluation/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/)

## References

40 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Samira Abnar and Willem Zuidema. 2020. [Quantifying attention flow in transformers](https://md.research.elamur.ai/free/en/papers/abnar-2020-quantifying-attention-flow-transformers/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190–4197.
2. Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. 2024. Make your llm fully utilize the context. Advances in Neural Information Processing Systems, 37:62160–62188. [find](https://scholar.google.com/scholar?q=Shengnan+An%2C+Zexiong+Ma%2C+Zeqi+Lin%2C+Nanning+Zheng%2C+Jian%2DGuang+Lou%2C+and+Weizhu+Chen%2E+2024%2E+Make+your+llm+fully+utilize+the+context%2E+Advances+in+Neural+Information+Processing+Systems%2C+37%3A62160%E2%80%9362188%2E)
3. Amit Ben Artzy and Roy Schwartz. 2024. Attend first, consolidate later: On the importance of attention in different LLM layers. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 177–184, Miami, Florida, US. Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Amit+Ben+Artzy+and+Roy+Schwartz%2E+2024%2E+Attend+first%2C+consolidate+later%3A+On+the+importance+of+attention+in+different+LLM+layers%2E+In+Proceedings+of+the+7th+BlackboxNLP+Workshop%3A+Analyzing+and+Interpreti)
4. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, and 1 others. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. [find](https://scholar.google.com/scholar?q=Xiao+Bi%2C+Deli+Chen%2C+Guanting+Chen%2C+Shanhuang+Chen%2C+Damai+Dai%2C+Chengqi+Deng%2C+Honghui+Ding%2C+Kai+Dong%2C+Qiushi+Du%2C+Zhe+Fu%2C+and+1+others%2E+2024%2E+Deepseek+llm%3A+Scaling+open%2Dsource+language+models+with+longte)
5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
6. Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, and 1 others. 2024. [Internlm2 technical report](https://md.research.elamur.ai/free/en/papers/cai-2024-internlm2-technical-report/). arXiv preprint arXiv:2403.17297.
7. Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Goyal, Peter Ku, and Dilek Hakkani-Tur. 2020. Multiwoz 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines. In Proceedings of the Twelfth Language… [find](https://scholar.google.com/scholar?q=Mihail+Eric%2C+Rahul+Goel%2C+Shachi+Paul%2C+Abhishek+Sethi%2C+Sanchit+Agarwal%2C+Shuyang+Gao%2C+Adarsh+Kumar%2C+Anuj+Goyal%2C+Peter+Ku%2C+and+Dilek+Hakkani%2DTur%2E+2020%2E+Multiwoz+2%2E1%3A+A+consolidated+multi%2Ddomain+dialogue+)
8. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
9. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
10. Minghao Guo, Qingcheng Zeng, Xujiang Zhao, Yanchi Liu, Wenchao Yu, Mengnan Du, Haifeng Chen, and Wei Cheng. 2025. DeepSieve: Information Sieving via LLM-as-a-Knowledge-Router. arXiv e-prints, arXiv:2507.22050. [find](https://scholar.google.com/scholar?q=Minghao+Guo%2C+Qingcheng+Zeng%2C+Xujiang+Zhao%2C+Yanchi+Liu%2C+Wenchao+Yu%2C+Mengnan+Du%2C+Haifeng+Chen%2C+and+Wei+Cheng%2E+2025%2E+DeepSieve%3A+Information+Sieving+via+LLM%2Das%2Da%2DKnowledge%2DRouter%2E+arXiv+e%2Dprints%2C+arXiv%3A25)
11. Matthew Henderson, Blaise Thomson, and Steve Young. 2014. [Word-based dialog state tracking with recurrent neural networks](https://md.research.elamur.ai/free/en/papers/henderson-2014-word-based-dialog-state-tracking/). In Proceedings of the 15th annual meeting of the special interest group on discourse and dialogue (SIGDIAL), pages 292–299.
12. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. [Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps](https://md.research.elamur.ai/free/en/papers/ho-2020-constructing-multi-hop-qa-dataset/). In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609– 6625.
13. Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A Smith, and Mari Ostendorf. 2022. [In-context learning for few-shot dialogue state tracking](https://md.research.elamur.ai/free/en/papers/hu-2022-context-learning-few-shot-dialogue/). In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2627–2643.
14. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
15. Qirui Jiao, Daoyuan Chen, Yilun Huang, Yaliang Li, and Ying Shen. 2024. From training-free to adaptive: Empirical insights into mllms’ understanding of detection information. arXiv preprint arXiv:2401.17981. — [arXiv:2401.17981](https://arxiv.org/abs/2401.17981) [find](https://scholar.google.com/scholar?q=Qirui+Jiao%2C+Daoyuan+Chen%2C+Yilun+Huang%2C+Yaliang+Li%2C+and+Ying+Shen%2E+2024%2E+From+training%2Dfree+to+adaptive%3A+Empirical+insights+into+mllms%E2%80%99+understanding+of+detection+information%2E+arXiv+preprint+arXiv%3A2401)
16. Qirui Jiao, Daoyuan Chen, Yilun Huang, Xika Lin, Ying Shen, and Yaliang Li. 2025. Detailmaster: Can your text-to-image model handle long prompts? arXiv preprint arXiv:2505.16915. — [arXiv:2505.16915](https://arxiv.org/abs/2505.16915) [find](https://scholar.google.com/scholar?q=Qirui+Jiao%2C+Daoyuan+Chen%2C+Yilun+Huang%2C+Xika+Lin%2C+Ying+Shen%2C+and+Yaliang+Li%2E+2025%2E+Detailmaster%3A+Can+your+text%2Dto%2Dimage+model+handle+long+prompts%3F+arXiv+preprint+arXiv%3A2505%2E16915%2E+%E2%80%94+%5BarXiv%3A2505%2E16915%5D%28)
17. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626.
18. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. [Retrieval-augmented generation for knowledge-intensive nlp tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). Advances in neural information processing systems, 33:9459–…
19. Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. 2023. How long can context length of open-source llms truly promise? In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following. [find](https://scholar.google.com/scholar?q=Dacheng+Li%2C+Rulin+Shao%2C+Anze+Xie%2C+Ying+Sheng%2C+Lianmin+Zheng%2C+Joseph+Gonzalez%2C+Ion+Stoica%2C+Xuezhe+Ma%2C+and+Hao+Zhang%2E+2023%2E+How+long+can+context+length+of+open%2Dsource+llms+truly+promise%3F+In+NeurIPS+2023)
20. Ruosen Li, Zimu Wang, Son Tran, Lei Xia, and Xinya Du. 2024. Meqa: A benchmark for multi-hop event-centric question answering with explanations. Advances in Neural Information Processing Systems, 37:126835–126862. [find](https://scholar.google.com/scholar?q=Ruosen+Li%2C+Zimu+Wang%2C+Son+Tran%2C+Lei+Xia%2C+and+Xinya+Du%2E+2024%2E+Meqa%3A+A+benchmark+for+multi%2Dhop+event%2Dcentric+question+answering+with+explanations%2E+Advances+in+Neural+Information+Processing+Systems%2C+37%3A1)
21. Yinghui Li, Qingyu Zhou, Yangning Li, Zhongli Li, Ruiyang Liu, Rongyi Sun, Zizhen Wang, Chao Li, Yunbo Cao, and Hai-Tao Zheng. 2022. The past mistake is the future wisdom: Error-driven contrastive probability optimization for chinese spell checking. In Findings of the Association for Computational L… [find](https://scholar.google.com/scholar?q=Yinghui+Li%2C+Qingyu+Zhou%2C+Yangning+Li%2C+Zhongli+Li%2C+Ruiyang+Liu%2C+Rongyi+Sun%2C+Zizhen+Wang%2C+Chao+Li%2C+Yunbo+Cao%2C+and+Hai%2DTao+Zheng%2E+2022%2E+The+past+mistake+is+the+future+wisdom%3A+Error%2Ddriven+contrastive+pro)
22. Zhenqing Ling, Yuexiang Xie, Chenhe Dong, and Ying Shen. 2025. Enhancing factual consistency in text summarization via counterfactual debiasing. In Proceedings of the 31st International Conference on Computational Linguistics, pages 7912–7924. [find](https://scholar.google.com/scholar?q=Zhenqing+Ling%2C+Yuexiang+Xie%2C+Chenhe+Dong%2C+and+Ying+Shen%2E+2025%2E+Enhancing+factual+consistency+in+text+summarization+via+counterfactual+debiasing%2E+In+Proceedings+of+the+31st+International+Conference+on+)
23. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). Transactions of the Association for Computational Linguistics, 12:157–173.
24. Da Ma, Lu Chen, Situo Zhang, Yuxun Miao, Su Zhu, Zhi Chen, Hongshen Xu, Hanqi Li, Shuai Fan, Lei Pan, and 1 others. 2024. Compressing kv cache for long-context llm inference with inter-layer attention similarity. arXiv preprint arXiv:2412.02252. — [arXiv:2412.02252](https://arxiv.org/abs/2412.02252) [find](https://scholar.google.com/scholar?q=Da+Ma%2C+Lu+Chen%2C+Situo+Zhang%2C+Yuxun+Miao%2C+Su+Zhu%2C+Zhi+Chen%2C+Hongshen+Xu%2C+Hanqi+Li%2C+Shuai+Fan%2C+Lei+Pan%2C+and+1+others%2E+2024%2E+Compressing+kv+cache+for+long%2Dcontext+llm+inference+with+inter%2Dlayer+attention)
25. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, and 1 others. 2023. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950.
26. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
27. Betty Van Aken, Benjamin Winter, Alexander Löser, and Felix A Gers. 2019. How does bert answer questions? a layer-wise analysis of transformer representations. In Proceedings of the 28th ACM international conference on information and knowledge management, pages 1823–1832. [find](https://scholar.google.com/scholar?q=Betty+Van+Aken%2C+Benjamin+Winter%2C+Alexander+L%C3%B6ser%2C+and+Felix+A+Gers%2E+2019%2E+How+does+bert+answer+questions%3F+a+layer%2Dwise+analysis+of+transformer+representations%2E+In+Proceedings+of+the+28th+ACM+internati)
28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in Neural Information Processing Systems, 30.
29. Qiuchen Wang, Ruixue Ding, Zehui Chen, Weiqi Wu, Shihang Wang, Pengjun Xie, and Feng Zhao. 2025. [Vidorag: Visual document retrieval-augmented generation via dynamic iterative reasoning agents](https://md.research.elamur.ai/free/en/papers/wang-2025-vidorag-visual-document-retrieval-augmented/). arXiv preprint arXiv:2502.18017.
30. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+2024%2E+Efficient+streaming+language+models+with+attention+sinks%2E+In+The+Twelfth+International+Conference+on+Learning+Representations)
31. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+and+1+others%2E+2024%2E+Qwen2%2E+5+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E15115)
32. An Yang, Baosong Yang, Beichen Zhang, Binyuan Wang, Bowen Yu, Chengpeng Li, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
33. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. [Hotpotqa: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–…
34. Fanghua Ye, Jarana Manotumruksa, and Emine Yilmaz. 2022. [Multiwoz 2.4: A multi-domain task-oriented dialogue dataset with essential annotation corrections to improve state tracking evaluation](https://md.research.elamur.ai/free/en/papers/ye-2022-multiwoz-2-4-multi-domain/). In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 35…
35. Zihao Yi, Jiarui Ouyang, Yuwen Liu, Tianhao Liao, Zhe Xu, and Ying Shen. 2024. [A survey on recent advances in llm-based multi-turn dialogue systems](https://md.research.elamur.ai/free/en/papers/yi-2024-survey-recent-advances-llm-based/). arXiv preprint arXiv:2402.18013.
36. Zihao Yi, Zhe Xu, and Ying Shen. 2025. Intent-driven in-context learning for few-shot dialogue state tracking. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. [find](https://scholar.google.com/scholar?q=Zihao+Yi%2C+Zhe+Xu%2C+and+Ying+Shen%2E+2025%2E+Intent%2Ddriven+in%2Dcontext+learning+for+few%2Dshot+dialogue+state+tracking%2E+In+ICASSP+2025%2D2025+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Process)
37. Jiahao Zhang, Haiyang Zhang, Dongmei Zhang, Liu Yong, and Shen Huang. 2024. End-to-end beam retrieval for multi-hop question answering. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pa… [find](https://scholar.google.com/scholar?q=Jiahao+Zhang%2C+Haiyang+Zhang%2C+Dongmei+Zhang%2C+Liu+Yong%2C+and+Shen+Huang%2E+2024%2E+End%2Dto%2Dend+beam+retrieval+for+multi%2Dhop+question+answering%2E+In+Proceedings+of+the+2024+Conference+of+the+North+American+Chap)
38. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, and 1 others. 2025a. [Qwen3 embedding: Advancing text embedding and reranking through foundation models](https://md.research.elamur.ai/free/en/papers/zhang-2025-qwen3-embedding-advancing-text/). arXiv preprint arXiv:2506.05176.
39. Zhiqiang Zhang, Liqiang Wen, and Wen Zhao. 2025b. Rule-kbqa: Rule-guided reasoning for complex knowledge base question answering with large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 8399–8417. [find](https://scholar.google.com/scholar?q=Zhiqiang+Zhang%2C+Liqiang+Wen%2C+and+Wen+Zhao%2E+2025b%2E+Rule%2Dkbqa%3A+Rule%2Dguided+reasoning+for+complex+knowledge+base+question+answering+with+large+language+models%2E+In+Proceedings+of+the+31st+International+Co)
40. Technical Appendix shown, requires a minimal number of inference runs. Once the attention profile is saved, it can be reused for all future inference tasks for that model at virtually no cost. The reranking itself is a simple array permutation, which is computationally negligible. The appendix is or… [find](https://scholar.google.com/scholar?q=Technical+Appendix+shown%2C+requires+a+minimal+number+of+inference+runs%2E+Once+the+attention+profile+is+saved%2C+it+can+be+reused+for+all+future+inference+tasks+for+that+model+at+virtually+no+cost%2E+The+rer)

*Updated 31 October 2026*