---
title: "ZoomR — paper by Yang et al. (2026) — summary and ideas"
description: "Reasoning LLMs write long chains of thought, and the KV cache used during decoding grows with every… 3% accuracy. 30 references."
authors: ["David H. Yang", "Yuxuan Zhu", "Mohammad Mohammadi Amiri", "Keerthiram Murugesan", "Tejaswini Pedapati", "Subhajit Chaudhury", "Pin-Yu Chen"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.76/"
doi: "10.18653/v1/2026.acl-long.76"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yang-2026-zoomr-memory-efficient-reasoning-through/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2026-zoomr-memory-efficient-reasoning-through/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/57/) / ZoomR: Memory Efficient Reasoning through Multi-Granularity Key Value Retrieval

# ZoomR: Memory Efficient Reasoning through Multi-Granularity Key Value Retrieval

**Authors:** [David H. Yang](https://md.research.elamur.ai/free/en/authors/david-h-yang/), [Yuxuan Zhu](https://md.research.elamur.ai/free/en/authors/yuxuan-zhu/), [Mohammad Mohammadi Amiri](https://md.research.elamur.ai/free/en/authors/mohammad-mohammadi-amiri/), [Keerthiram Murugesan](https://md.research.elamur.ai/free/en/authors/keerthiram-murugesan/), [Tejaswini Pedapati](https://md.research.elamur.ai/free/en/authors/tejaswini-pedapati/), Subhajit Chaudhury and [Pin-Yu Chen](https://md.research.elamur.ai/free/en/authors/pin-yu-chen/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.76/) · [PDF](https://aclanthology.org/2026.acl-long.76.pdf) · [DOI 10.18653/v1/2026.acl-long.76](https://doi.org/10.18653/v1/2026.acl-long.76) · [OpenAlex](https://openalex.org/W7166808640)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Reasoning LLMs write long chains of thought, and the KV cache used during decoding grows with every generated token. This uses up GPU memory and slows each step. Most KV compression methods target long prompts at the prefill stage and still keep the full cache while decoding. Streaming and token-eviction schemes tend to drop distant but essential reasoning steps.

ZoomR addresses the output side of this problem. The authors add paragraph-level summaries, written by Llama3-70B and wrapped in special delimiter tokens, to the Bespoke-17K dataset. They then LoRA-fine-tune reasoning models so the models emit a summary after each paragraph. At inference, a training-free policy uses summary keys as a coarse index:

- each attention head scores the summaries against the current query;
- the heads reach a consensus;
- only the top-ranked segments are expanded back to their detailed KVs.

The full cache stays in CPU memory. On AIME2025 and MATH500, ZoomR's accuracy stays close to full-cache decoding and beats the baselines, while inference memory drops by more than 4×. This makes long reasoning more practical on memory-limited GPUs.

## Key points
- Motivating example: when Llama-3.1-8B generates a 16K-token output at batch size 8, the KV cache by itself takes up more than 16GB of GPU memory.
- Training data: Bespoke-17K, with each reasoning paragraph summarized by Llama3-70B. Summaries are marked with `<|begin_of_summary|>` and `<|end_of_summary|>` and taught to the model via LoRA fine-tuning. The selection policy itself requires no training.
- Evaluation covers Qwen and Llama reasoning models on AIME2025 and MATH500. Baselines include StreamingLLM- and H2O-style methods. ZoomR reduces memory by more than 4× while staying competitive with the full KV cache.
- Ablations on MATH500 with Llama:
  - Expanding only one summary (c = 1) costs about 3% accuracy compared with c = 2. Values of c above 2 add roughly 1% but select over 40% more KVs.
  - Using top-k = 1 also loses about 3%, and accuracy levels off from k = 2.
- Consensus analysis: agreement between heads fluctuates more for incorrect answers (std 14.82%) than for correct ones (7.87%). Correct answers also show about 5.7% higher agreement during the first 38 recompute steps.
- Stated limitations: summaries are fixed at the paragraph level, summary quality depends on Llama-70B, and evaluation is mostly limited to math.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [IBM](https://md.research.elamur.ai/free/en/companies/ibm/) — Company, Armonk, United States · 154 works · 72,606 citations
- [Rensselaer Polytechnic Institute](https://md.research.elamur.ai/free/en/universities/rensselaer-polytechnic-institute/) — University, Troy, United States · 23 works · 5,092 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 552 works

## Cite

Cite as: David H. Yang, Yuxuan Zhu, Mohammad Mohammadi Amiri, Keerthiram Murugesan, Tejaswini Pedapati, Subhajit Chaudhury and Pin-Yu Chen. 2026. ZoomR: Memory Efficient Reasoning through Multi-Granularity Key Value Retrieval. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yang2026zoomr,
  title = {{ZoomR: Memory Efficient Reasoning through Multi-Granularity Key Value Retrieval}},
  author = {David H. Yang, Yuxuan Zhu, Mohammad Mohammadi Amiri, Keerthiram Murugesan, Tejaswini Pedapati, Subhajit Chaudhury and Pin-Yu Chen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.76},
  url = {https://doi.org/10.18653/v1/2026.acl-long.76}
}
```

- [← previous ZIP: Quantifying Which Words Matter in Zero-Shot Instructional Prompts](https://md.research.elamur.ai/free/en/papers/sadr-2026-zip-quantifying-which-words-matter/)
- [next → “Decode the Law”: Towards Legal Text Simplification with Large Language Models](https://md.research.elamur.ai/free/en/papers/rabbani-2026-decode-law-legal-text-simplification/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/57/)

## References

30 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Simon A. Aytes, Jinheon Baek, and Sung Ju Hwang. 2025. [Sketch-of-thought: Efficient llm reasoning with adaptive cognitive-inspired sketching](https://md.research.elamur.ai/free/en/papers/aytes-2025-sketch-thought-efficient-llm-reasoning/). Preprint, arXiv:2503.05179.
2. Frederic Charles Bartlett. 1932. Remembering: A Study in Experimental and Social Psychology. Cambridge University Press, Cambridge. [find](https://scholar.google.com/scholar?q=Frederic+Charles+Bartlett%2E+1932%2E+Remembering%3A+A+Study+in+Experimental+and+Social+Psychology%2E+Cambridge+University+Press%2C+Cambridge%2E)
3. Guoxuan Chen, Han Shi, Jiawei Li, Yihang Gao, Xiaozhe Ren, Yimeng Chen, Xin Jiang, Zhenguo Li, Weiyang Liu, and Chao Huang. 2025. Sepllm: Accelerate large language models by compressing one segment into one separator. Preprint, arXiv:2412.12094. — [arXiv:2412.12094](https://arxiv.org/abs/2412.12094) [find](https://scholar.google.com/scholar?q=Guoxuan+Chen%2C+Han+Shi%2C+Jiawei+Li%2C+Yihang+Gao%2C+Xiaozhe+Ren%2C+Yimeng+Chen%2C+Xin+Jiang%2C+Zhenguo+Li%2C+Weiyang+Liu%2C+and+Chao+Huang%2E+2025%2E+Sepllm%3A+Accelerate+large+language+models+by+compressing+one+segment+in)
4. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. [Deepseek-r1: Incentivizing reasoning capability in ll…](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/)
5. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
6. Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. 2025. [Token-budget-aware llm reasoning](https://md.research.elamur.ai/free/en/papers/han-2024-token-budget-aware-llm-reasoning/). Preprint, arXiv:2412.18547.
7. Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. 2024a. Training large language models to reason in a continuous latent space. Preprint, arXiv:2412.06769. [find](https://scholar.google.com/scholar?q=Shibo+Hao%2C+Sainbayar+Sukhbaatar%2C+DiJia+Su%2C+Xian+Li%2C+Zhiting+Hu%2C+Jason+Weston%2C+and+Yuandong+Tian%2E+2024a%2E+Training+large+language+models+to+reason+in+a+continuous+latent+space%2E+Preprint%2C+arXiv%3A2412%2E0676)
8. Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. 2024b. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769. [find](https://scholar.google.com/scholar?q=Shibo+Hao%2C+Sainbayar+Sukhbaatar%2C+DiJia+Su%2C+Xian+Li%2C+Zhiting+Hu%2C+Jason+Weston%2C+and+Yuandong+Tian%2E+2024b%2E+Training+large+language+models+to+reason+in+a+continuous+latent+space%2E+arXiv+preprint+arXiv%3A2412)
9. Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2025. [Kvquant: Towards 10 million context length llm inference with kv cache quantization](https://md.research.elamur.ai/free/en/papers/hooper-2024-kvquant-towards-10-million-context/). Preprint, arXiv:2401.18079.
10. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). Preprint, arXiv:2106.09685.
11. Bespoke Labs. 2025. Bespoke-stratos: The unreasonable effectiveness of reasoning distillation. [https://www.bespokelabs.ai/blog/bespoke-stratos-the-unreasonable-effectiveness-of-reasoning-distillation](https://www.bespokelabs.ai/blog/bespoke-stratos-the-unreasonable-effectiveness-of-reasoning-distillation). Accessed: 2025-01-22. [↗](https://www.bespokelabs.ai/blog/bespoke-stratos-the-unreasonable-effectiveness-of-reasoning-distillation)
12. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. [Snapkv: Llm knows what you are looking for before generation](https://md.research.elamur.ai/free/en/papers/li-2024-snapkv-llm-knows-looking-before/). Advances in Neural Information Processing Systems, 37:22947–22970.
13. OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, and 244 others. 2024a. [Openai o1 s…](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/)
14. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others.… — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
15. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=Qwen%2C+%3A%2C+An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+)
16. Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, and Kaushik Roy. 2024. Eigen attention: Attention in low-rank space for kv cache compression. arXiv preprint arXiv:2408.05646. [find](https://scholar.google.com/scholar?q=Utkarsh+Saxena%2C+Gobinda+Saha%2C+Sakshi+Choudhary%2C+and+Kaushik+Roy%2E+2024%2E+Eigen+attention%3A+Attention+in+low%2Drank+space+for+kv+cache+compression%2E+arXiv+preprint+arXiv%3A2408%2E05646%2E)
17. Xuan Shen, Yizhou Wang, Xiangxi Shi, Yanzhi Wang, Pu Zhao, and Jiuxiang Gu. 2025a. Efficient reasoning with hidden thinking. Preprint, arXiv:2501.19201. — [arXiv:2501.19201](https://arxiv.org/abs/2501.19201) [find](https://scholar.google.com/scholar?q=Xuan+Shen%2C+Yizhou+Wang%2C+Xiangxi+Shi%2C+Yanzhi+Wang%2C+Pu+Zhao%2C+and+Jiuxiang+Gu%2E+2025a%2E+Efficient+reasoning+with+hidden+thinking%2E+Preprint%2C+arXiv%3A2501%2E19201%2E+%E2%80%94+%5BarXiv%3A2501%2E19201%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2501)
18. Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. 2025b. [Codi: Compressing chain-of-thought into continuous space via self-distillation](https://md.research.elamur.ai/free/en/papers/shen-2025-codi-compressing-chain-thought-into/). arXiv preprint arXiv:2502.21074.
19. Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Na Zou, Hanjie Chen, and Xia Hu. 2025. [Stop overthinking: A survey on efficient reasoning for large language models](https://md.research.elamur.ai/free/en/papers/sui-2025-stop-overthinking-survey-efficient-reasoning/). Preprint, arXiv:2503.16419.
20. Yuxuan Zhu, David H. Yang, Mohammad Mohammadi Amiri, Keerthiram Murugesan, Tejaswini Pedapati, and Pin-Yu Chen. 2025b. [Ojakv: Context-aware online low-rank kv cache compression](https://md.research.elamur.ai/free/en/papers/zhu-2026-ojakv-context-aware-online-low/) with oja’s rule. Preprint, arXiv:2509.21623.
21. Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2023. Kivi : Plug-and-play 2bit kv cache quantization with streaming asymmetric quantization. [find](https://scholar.google.com/scholar?q=Zirui+Liu%2C+Jiayi+Yuan%2C+Hongye+Jin%2C+Shaochen+Zhong%2C+Zhaozhuo+Xu%2C+Vladimir+Braverman%2C+Beidi+Chen%2C+and+Xia+Hu%2E+2023%2E+Kivi+%3A+Plug%2Dand%2Dplay+2bit+kv+cache+quantization+with+streaming+asymmetric+quantization)
22. Hanshi Sun, Li-Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. 2025. Shadowkv: Kv cache in shadows for high-throughput long-context llm inference. Preprint, arXiv:2410.21465. [find](https://scholar.google.com/scholar?q=Hanshi+Sun%2C+Li%2DWen+Chang%2C+Wenlei+Bao%2C+Size+Zheng%2C+Ningxin+Zheng%2C+Xin+Liu%2C+Harry+Dong%2C+Yuejie+Chi%2C+and+Beidi+Chen%2E+2025%2E+Shadowkv%3A+Kv+cache+in+shadows+for+high%2Dthroughput+long%2Dcontext+llm+inference%2E+Pr)
23. Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. [Quest: Query-aware sparsity for efficient long-context llm inference](https://md.research.elamur.ai/free/en/papers/tang-2024-quest-query-aware-sparsity-efficient/). Preprint, arXiv:2406.10774.
24. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, and 77 others. 2025. [Kimi k1.5: Scaling reinforcement l…](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k1-5-scaling-reinforcement/)
25. Heming Xia, Chak Tou Leong, Wenjie Wang, Yongqi Li, and Wenjie Li. 2025. [Tokenskip: Controllable chain-of-thought compression in llms](https://md.research.elamur.ai/free/en/papers/xia-2025-tokenskip-controllable-chain-thought-compression/). Preprint, arXiv:2502.12067.
26. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. Preprint, arXiv:2309.17453. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+2024%2E+Efficient+streaming+language+models+with+attention+sinks%2E+Preprint%2C+arXiv%3A2309%2E17453%2E)
27. Yuchen Yan, Yongliang Shen, Yang Liu, Jin Jiang, Mengdi Zhang, Jian Shao, and Yueting Zhuang. 2025. [Inftythink: Breaking the length limits of long-context reasoning in large language models](https://md.research.elamur.ai/free/en/papers/yan-2025-inftythink-breaking-length-limits-long/). Preprint, arXiv:2503.06692.
28. Jintian Zhang, Yuqi Zhu, Mengshu Sun, Yujie Luo, Shuofei Qiao, Lun Du, Da Zheng, Huajun Chen, and Ningyu Zhang. 2025. [Lightthinker: Thinking step-by-step compression](https://md.research.elamur.ai/free/en/papers/zhang-2025-lightthinker-thinking-step-step-compression/). Preprint, arXiv:2502.15589.
29. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. [H 2 o: Heavy-hitter oracle for efficient generative inference of large language models](https://md.research.elamur.ai/free/en/papers/zhang-2023-h-2-o-heavy-hitter/). Preprint, arXiv:2306.14048.
30. Yuxuan Zhu, Ali Falahati, David H. Yang, and Mohammad Mohammadi Amiri. 2025a. Sentencekv: Efficient llm inference via sentence-level semantic kv caching. Preprint, arXiv:2504.00970. [find](https://scholar.google.com/scholar?q=Yuxuan+Zhu%2C+Ali+Falahati%2C+David+H%2E+Yang%2C+and+Mohammad+Mohammadi+Amiri%2E+2025a%2E+Sentencekv%3A+Efficient+llm+inference+via+sentence%2Dlevel+semantic+kv+caching%2E+Preprint%2C+arXiv%3A2504%2E00970%2E)

*Updated 2 November 2026*