---
title: "Question Tells You Where the Answer Is: Intention-aware"
description: "The authors build on the pragmatic notion of intention , meaning the part of the prompt (usually the question) that states what… 52 references."
authors: ["Liang Zhao", "Xiaocheng Feng", "Weihong Zhong", "Lei Huang", "Kun Zhu", "Baoxin Wang", "Dayong Wu", "Guoping Hu", "Ting Liu", "Bing Qin"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1250/"
doi: "10.18653/v1/2026.acl-long.1250"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhao-2026-question-tells-you-where-answer/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhao-2026-question-tells-you-where-answer/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/46/) / Question Tells You Where the Answer Is: Intention-aware Long-Context KV Cache Compression

# Question Tells You Where the Answer Is: Intention-aware Long-Context KV Cache Compression

**Authors:** [Liang Zhao](https://md.research.elamur.ai/free/en/authors/liang-zhao/), [Xiaocheng Feng](https://md.research.elamur.ai/free/en/authors/xiaocheng-feng/), [Weihong Zhong](https://md.research.elamur.ai/free/en/authors/weihong-zhong/), [Lei Huang](https://md.research.elamur.ai/free/en/authors/lei-huang/), [Kun Zhu](https://md.research.elamur.ai/free/en/authors/kun-zhu/), [Baoxin Wang](https://md.research.elamur.ai/free/en/authors/baoxin-wang/), [Dayong Wu](https://md.research.elamur.ai/free/en/authors/dayong-wu/), [Guoping Hu](https://md.research.elamur.ai/free/en/authors/guoping-hu/), [Ting Liu](https://md.research.elamur.ai/free/en/authors/ting-liu/) and [Bing Qin](https://md.research.elamur.ai/free/en/authors/bing-qin/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1250/) · [PDF](https://aclanthology.org/2026.acl-long.1250.pdf) · [DOI 10.18653/v1/2026.acl-long.1250](https://doi.org/10.18653/v1/2026.acl-long.1250) · [OpenAlex](https://openalex.org/W7166820590)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Long context windows make the key-value (KV) cache of large language models very expensive in GPU memory and decoding time. Existing eviction methods often discard tokens that later turn out to matter. The authors build on the pragmatic notion of *intention*, meaning the part of the prompt (usually the question) that states what the user wants. Their analysis shows that intention tokens attend to the context in much the same way that generated tokens later do: both concentrate on the answer-bearing segment. Ordinary context tokens attend quite differently.

Based on this, they propose IntentKV, a training-free eviction policy:
1. It detects intention tokens by comparing attention distributions with Jensen-Shannon Divergence.
2. It sums their attention over fixed-size blocks.
3. It keeps the highest-scoring blocks in a single eviction step. Keeping whole blocks avoids fragmenting the context.

Across several long-context benchmarks and models, IntentKV shrinks the cache from 128K to 2K with little loss in quality. This matters for running long-document QA on limited hardware.

## Key points
- **Efficiency:** compressing the cache from 128K to 2K gives about 6.3x faster decoding and 7.8x better memory efficiency than the full-cache setting. Code is released at github.com/linkezh/IntentKV.
- **Ruler results with Llama3.1-8B:**
  - At a 2048-token cache, IntentKV averages 85.5%, compared with 69.3% for SnapKV and 27.7% for StreamingLLM.
  - At a 512-token cache, it averages 60.9%, compared with 52.5% for SnapKV and 16.4% for StreamingLLM.
- **Motivating analysis:**
  - In a 124K-token InfiniteBench example, the top 4K tokens capture about 96.6% of attention, so attention is sparse.
  - The attended tokens shift across generation steps and are scattered rather than contiguous, which motivates block-level retention.
- **Block-size ablation:** a block size of 1 or other small blocks hurt performance badly. Results stay stable once the block size exceeds 16.
- **Limitations:**
  - The method assumes the intention appears at the end of the prompt.
  - Speed gains are small for short contexts on GQA models.
  - Because eviction is irreversible, the method does not suit multi-turn chat.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Harbin Institute of Technology](https://md.research.elamur.ai/free/en/universities/harbin-institute-of-technology/) — University, Harbin, China · 185 works · 29,140 citations
- [Peng Cheng Laboratory](https://md.research.elamur.ai/free/en/labs/peng-cheng-laboratory/) — Research institute, Shenzhen, China · 47 works · 2,699 citations
- [IFlyTek](https://md.research.elamur.ai/free/en/companies/iflytek/) — Company, Hefei, China · 11 works · 2,810 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 297 works

## Cite

Cite as: Liang Zhao, Xiaocheng Feng, Weihong Zhong, Lei Huang, Kun Zhu, Baoxin Wang, Dayong Wu, Guoping Hu, Ting Liu and Bing Qin. 2026. Question Tells You Where the Answer Is: Intention-aware Long-Context KV Cache Compression. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhao2026question,
  title = {{Question Tells You Where the Answer Is: Intention-aware Long-Context KV Cache Compression}},
  author = {Liang Zhao, Xiaocheng Feng, Weihong Zhong, Lei Huang, Kun Zhu, Baoxin Wang, Dayong Wu, Guoping Hu, Ting Liu and Bing Qin},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1250},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1250}
}
```

- [← previous Question Difficulty Estimation for Large Language Models via Answer Plausibility Scoring](https://md.research.elamur.ai/free/en/papers/mozafari-2026-question-difficulty-estimation-large-language/)
- [next → Questionnaire Meets LLM: A Benchmark and Empirical Study of Structural Skills for Understanding Questions and Responses](https://md.research.elamur.ai/free/en/papers/nguyen-2026-questionnaire-meets-llm-benchmark-empirical/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/46/)

## References

52 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, and… — [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/)
2. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. 2023. [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–49…
3. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/). In Proceedings of the 62nd Annual Meeting of the Associati…
4. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. [Longformer: The Long-Document Transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). Preprint, arXiv:2004.05150.
5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretchen)
6. Zefan Cai., Yichi Zhang, Bofei Gao, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. 2024. [PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling](https://md.research.elamur.ai/free/en/papers/cai-2024-pyramidkv-dynamic-kv-cache-compression/). Preprint, arXiv:2406.02069.
7. Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, and 1 others. 2024. [Magicpig: Lsh sampling for efficient llm generation](https://md.research.elamur.ai/free/en/papers/chen-2024-magicpig-lsh-sampling-efficient-llm/). In Adaptive Foundation Models: Evolving AI for Personalized and Efficient Learning.
8. Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. 2023. [Adapting Language Models to Compress Contexts](https://md.research.elamur.ai/free/en/papers/chevalier-2023-adapting-language-models-compress-contexts/). Preprint, arXiv:2305.14788.
9. Ta-Chung Chi, Ting-Han Fan, and Alexander I. Rudnicky. 2023. Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation. Preprint, arXiv:2311.00684. [find](https://scholar.google.com/scholar?q=Ta%2DChung+Chi%2C+Ting%2DHan+Fan%2C+and+Alexander+I%2E+Rudnicky%2E+2023%2E+Attention+Alignment+and+Flexible+Positional+Embeddings+Improve+Transformer+Length+Extrapolation%2E+Preprint%2C+arXiv%3A2311%2E00684%2E)
10. Tri Dao. 2023. [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). Preprint, arXiv:2307.08691.
11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 514 others. 202… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
12. Benjamin L. Edelman, Surbhi Goel, Sham Kakade, and Cyril Zhang. 2022. Inductive Biases and Variable Creation in Self-Attention Mechanisms. In Proceedings of the 39th International Conference on Machine Learning, pages 5793–5831. PMLR. [find](https://scholar.google.com/scholar?q=Benjamin+L%2E+Edelman%2C+Surbhi+Goel%2C+Sham+Kakade%2C+and+Cyril+Zhang%2E+2022%2E+Inductive+Biases+and+Variable+Creation+in+Self%2DAttention+Mechanisms%2E+In+Proceedings+of+the+39th+International+Conference+on+Machin)
13. Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou. 2024. Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference. arXiv preprint arXiv:2407.11550. — [arXiv:2407.11550](https://arxiv.org/abs/2407.11550) [find](https://scholar.google.com/scholar?q=Yuan+Feng%2C+Junlin+Lv%2C+Yukun+Cao%2C+Xike+Xie%2C+and+S+Kevin+Zhou%2E+2024%2E+Ada%2Dkv%3A+Optimizing+kv+cache+eviction+by+adaptive+budget+allocation+for+efficient+llm+inference%2E+arXiv+preprint+arXiv%3A2407%2E11550%2E+%E2%80%94+%5Ba)
14. Yao Fu. 2024. Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysis. Preprint, arXiv:2405.08944. [find](https://scholar.google.com/scholar?q=Yao+Fu%2E+2024%2E+Challenges+in+Deploying+Long%2DContext+Transformers%3A+A+Theoretical+Peak+Performance+Analysis%2E+Preprint%2C+arXiv%3A2405%2E08944%2E)
15. Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2023. [Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs](https://md.research.elamur.ai/free/en/papers/ge-2023-model-tells-discard-adaptive-kv/). In The Twelfth International Conference on Learning Representations. gkamradt. 2024. radt/LLMTest_NeedleInAHaystack.
16. Gkam-Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, and 39 others. 2024. [ChatGLM: A Family of Large Language Models from GLM-1…](https://md.research.elamur.ai/free/en/papers/glm-2024-chatglm-family-large-language-models/)
17. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. [RULER: What’s the Real Context Size of Your Long-Context Language Models?](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/) Preprint, arXiv:2404.06654.
18. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and 1 others. 2025. [A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/). ACM Transactions on Information Systems,…
19. Rodnry Huddleston and Geqffrry Pullum. 2005. The cambridge grammar of the english language. Zeitschrift für Anglistik und Amerikanistik, 53(2):193–194. [find](https://scholar.google.com/scholar?q=Rodnry+Huddleston+and+Geqffrry+Pullum%2E+2005%2E+The+cambridge+grammar+of+the+english+language%2E+Zeitschrift+f%C3%BCr+Anglistik+und+Amerikanistik%2C+53%282%29%3A193%E2%80%93194%2E)
20. Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. [MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention](https://md.research.elamur.ai/free/en/papers/jiang-2024-minference-1-0-accelerating-pre/). Preprint, arXiv:2407.02490.
21. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2023. [SWE-bench: Can Language Models Resolve Real-world Github Issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) In The Twelfth International Conference on Learning Representations.
22. Philippe Laban, Alexander Fabbri, Caiming Xiong, and Chien-Sheng Wu. 2024. [Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems](https://md.research.elamur.ai/free/en/papers/laban-2024-summary-haystack-challenge-long-context/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9885–9903, Miami, Florida, USA. Association for Computa…
23. Kunxi Li, Zhonghua Jiang, Zhouzhou Shen, Zhaode Wang, Chengfei Lv, Shengyu Zhang, Fan Wu, Fei Wu, and Wanxiang Che. 2025. MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference. In Proceedings of the 63rd Annual Meeting of the Association for Computati… [find](https://scholar.google.com/scholar?q=Kunxi+Li%2C+Zhonghua+Jiang%2C+Zhouzhou+Shen%2C+Zhaode+Wang%2C+Chengfei+Lv%2C+Shengyu+Zhang%2C+Fan+Wu%2C+Fei+Wu%2C+and+Wanxiang+Che%2E+2025%2E+MadaKV%3A+Adaptive+Modality%2DPerception+KV+Cache+Eviction+for+Efficient+Multimoda)
24. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. [SnapKV: LLM Knows What You are Looking for Before Generation](https://md.research.elamur.ai/free/en/papers/li-2024-snapkv-llm-knows-looking-before/). In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
25. Valerii Likhosherstov, Krzysztof Choromanski, and Adrian Weller. 2023. On the Expressive Flexibility of Self-Attention Matrices. Proceedings of the AAAI Conference on Artificial Intelligence, 37(7):8773– 8781. [find](https://scholar.google.com/scholar?q=Valerii+Likhosherstov%2C+Krzysztof+Choromanski%2C+and+Adrian+Weller%2E+2023%2E+On+the+Expressive+Flexibility+of+Self%2DAttention+Matrices%2E+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+37%287%29%3A87)
26. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. [Lost in the Middle: How Language Models Use Long Contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). Transactions of the Association for Computational Linguistics, 12:157–173.
27. Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2023. Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time. Advances in Neural Information Processing Systems, 36:5… [find](https://scholar.google.com/scholar?q=Zichang+Liu%2C+Aditya+Desai%2C+Fangshuo+Liao%2C+Weitao+Wang%2C+Victor+Xie%2C+Zhaozhuo+Xu%2C+Anastasios+Kyrillidis%2C+and+Anshumali+Shrivastava%2E+2023%2E+Scissorhands%3A+Exploiting+the+Persistence+of+Importance+Hypothesi)
28. Weitao Ma, Xiyuan Du, Xiaocheng Feng, Lei Huang, Yichong Huang, Huiyi Zhang, Xiaoliang Yang, Baohang Li, Xiachong Feng, Ting Liu, and 1 others. 2025. One for all: Update parameterized knowledge across multiple models with once edit. In Proceedings of the 63rd Annual Meeting of the Association for Co… [find](https://scholar.google.com/scholar?q=Weitao+Ma%2C+Xiyuan+Du%2C+Xiaocheng+Feng%2C+Lei+Huang%2C+Yichong+Huang%2C+Huiyi+Zhang%2C+Xiaoliang+Yang%2C+Baohang+Li%2C+Xiachong+Feng%2C+Ting+Liu%2C+and+1+others%2E+2025%2E+One+for+all%3A+Update+parameterized+knowledge+across)
29. Jesse Mu, Xiang Li, and Noah Goodman. 2023. [Learning to Compress Prompts with Gist Tokens](https://md.research.elamur.ai/free/en/papers/mu-2023-learning-compress-prompts-gist-tokens/). Advances in Neural Information Processing Systems, 36:19327–19352.
30. OpenAI. 2023. [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). Preprint, arXiv:2303.08774.
31. Alexander Peysakhovich and Adam Lerer. 2023. Attention Sorting Combats Recency Bias In Long Context Language Models. Preprint, arXiv:2310.01427. — [arXiv:2310.01427](https://arxiv.org/abs/2310.01427) [find](https://scholar.google.com/scholar?q=Alexander+Peysakhovich+and+Adam+Lerer%2E+2023%2E+Attention+Sorting+Combats+Recency+Bias+In+Long+Context+Language+Models%2E+Preprint%2C+arXiv%3A2310%2E01427%2E+%E2%80%94+%5BarXiv%3A2310%2E01427%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E01427%29)
32. Ziran Qin, Yuchen Cao, Mingbao Lin, Wen Hu, Shixuan Fan, Ke Cheng, Weiyao Lin, and Jianguo Li. 2025. Cake: Cascading and adaptive kv cache eviction with layer preferences. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ziran+Qin%2C+Yuchen+Cao%2C+Mingbao+Lin%2C+Wen+Hu%2C+Shixuan+Fan%2C+Ke+Cheng%2C+Weiyao+Lin%2C+and+Jianguo+Li%2E+2025%2E+Cake%3A+Cascading+and+adaptive+kv+cache+eviction+with+layer+preferences%2E+In+The+Thirteenth+Internatio)
33. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. 2023. [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://md.research.elamur.ai/free/en/papers/shi-2023-large-language-models-easily-distracted/). In Proceedings of the 40th International Conference on Machine Learning, pages 31210–31227. PMLR.
34. Dan Sperber and Deirdre Wilson. 1986. Relevance: Communication and cognition, volume 142. Harvard University Press Cambridge, MA. [find](https://scholar.google.com/scholar?q=Dan+Sperber+and+Deirdre+Wilson%2E+1986%2E+Relevance%3A+Communication+and+cognition%2C+volume+142%2E+Harvard+University+Press+Cambridge%2C+MA%2E)
35. Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://md.research.elamur.ai/free/en/papers/tang-2024-quest-query-aware-sparsity-efficient/). Preprint, arXiv:2406.10774.
36. Gemini Team. 2024a. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). Preprint, arXiv:2403.05530.
37. Qwen Team. 2024b. Qwen2.5: A party of foundation models. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+2024b%2E+Qwen2%2E5%3A+A+party+of+foundation+models%2E)
38. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
39. Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, and Mi Zhang. 2023. [Efficient Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/wan-2023-efficient-large-language-models-survey/). Preprint, arXiv:2312.03863.
40. Cunxiang Wang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, Guangsheng Bao, Xiangkun Hu, Zheng Zhang, Qian Wang, and Yue Zhang. 2024. [NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens](https://md.research.elamur.ai/free/en/papers/wang-2024-novelqa-benchmarking-question-answering-documents/). Preprint, arXiv:2403.12766.
41. Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. 2024. [Retrieval Head Mechanistically Explains Long-Context Factuality](https://md.research.elamur.ai/free/en/papers/wu-2024-retrieval-head-mechanistically-explains-long/). Preprint, arXiv:2404.15574.
42. Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. 2024. [InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory](https://md.research.elamur.ai/free/en/papers/xiao-2024-infllm-training-free-long-context/). In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
43. Li Yang, and Amr Ahmed. 2020. [Big Bird: Transformers for Longer Sequences](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/). In Advances in Neural Information Processing Systems, volume 33, pages 17283–17297. Curran Associates, Inc.
44. Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. 2024. ınftyBench: Extending Long Context Evaluation Beyond 100K Tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V… [find](https://scholar.google.com/scholar?q=Xinrong+Zhang%2C+Yingfa+Chen%2C+Shengding+Hu%2C+Zihang+Xu%2C+Junhao+Chen%2C+Moo+Hao%2C+Xu+Han%2C+Zhen+Thai%2C+Shuo+Wang%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2024%2E+%C4%B1nftyBench%3A+Extending+Long+Context+Evaluation+Beyond+100K+T)
45. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. [H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2023-h-2-o-heavy-hitter/). Preprint, arXiv:2306.14048.
46. Liang Zhao, Xiachong Feng, Xiaocheng Feng, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, Bing Qin, and Ting Liu. 2024. Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 99… [find](https://scholar.google.com/scholar?q=Liang+Zhao%2C+Xiachong+Feng%2C+Xiaocheng+Feng%2C+Weihong+Zhong%2C+Dongliang+Xu%2C+Qing+Yang%2C+Hongtao+Liu%2C+Bing+Qin%2C+and+Ting+Liu%2E+2024%2E+Length+Extrapolation+of+Transformers%3A+A+Survey+from+the+Perspective+of+Pos)
47. Xiabin Zhou, Wenbin Wang, Minyan Zeng, Jiaxian Guo, Xuebo Liu, Li Shen, Min Zhang, and Liang Ding. 2024. Dynamickv: Task-aware adaptive kv cache compression for long context llms. arXiv preprint arXiv:2412.14838. [find](https://scholar.google.com/scholar?q=Xiabin+Zhou%2C+Wenbin+Wang%2C+Minyan+Zeng%2C+Jiaxian+Guo%2C+Xuebo+Liu%2C+Li+Shen%2C+Min+Zhang%2C+and+Liang+Ding%2E+2024%2E+Dynamickv%3A+Task%2Daware+adaptive+kv+cache+compression+for+long+context+llms%2E+arXiv+preprint+arXiv)
48. Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. 2025. [DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads](https://md.research.elamur.ai/free/en/papers/xiao-2024-duoattention-efficient-long-context-llm/). In The Thirteenth International Conference on Learning Representations.
49. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient Streaming Language Models with Attention Sinks. Preprint, arXiv:2309.17453. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+2023%2E+Efficient+Streaming+Language+Models+with+Attention+Sinks%2E+Preprint%2C+arXiv%3A2309%2E17453%2E)
50. Xiaohan Xu, Chongyang Tao, Tao Shen, Can Xu, Hongbo Xu, Guodong Long, Jian-Guang Lou, and Shuai Ma. 2024. Re-reading improves reasoning in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15549–15575. [find](https://scholar.google.com/scholar?q=Xiaohan+Xu%2C+Chongyang+Tao%2C+Tao+Shen%2C+Can+Xu%2C+Hongbo+Xu%2C+Guodong+Long%2C+Jian%2DGuang+Lou%2C+and+Shuai+Ma%2E+2024%2E+Re%2Dreading+improves+reasoning+in+large+language+models%2E+In+Proceedings+of+the+2024+Conference+)
51. Dongjie Yang, XiaoDong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. 2024. [PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference](https://md.research.elamur.ai/free/en/papers/yang-2024-pyramidinfer-pyramid-kv-cache-compression/). Preprint, arXiv:2405.12532.
52. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, [find](https://scholar.google.com/scholar?q=Manzil+Zaheer%2C+Guru+Guruganesh%2C+Kumar+Avinava+Dubey%2C+Joshua+Ainslie%2C+Chris+Alberti%2C+Santiago+Ontanon%2C+Philip+Pham%2C+Anirudh+Ravula%2C+Qifan+Wang%2C)

*Updated 2 November 2026*