---
title: "PyramidInfer — paper by Yang et al. (2024) — summary and ideas"
description: "Serving LLMs at scale is limited by GPU memory, and the KV cache often takes far more memory… 29 references, cited by 63 papers here."
authors: ["Dongjie Yang", "Xiaodong Han", "Yan Gao", "Yao Hu", "Shilin Zhang", "Hai Zhao"]
year: 2024
venue: "Findings of the Association for Computational Linguistics: ACL 2024"
source: "https://aclanthology.org/2024.findings-acl.195/"
doi: "10.18653/v1/2024.findings-acl.195"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yang-2024-pyramidinfer-pyramid-kv-cache-compression/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2024-pyramidinfer-pyramid-kv-cache-compression/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference

# PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference

**Authors:** Dongjie Yang, Xiaodong Han, Yan Gao, Yao Hu, Shilin Zhang and [Hai Zhao](https://md.research.elamur.ai/free/en/authors/hai-zhao/) · **Year:** 2024 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2024](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2024](https://md.research.elamur.ai/free/en/venues/findings/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.findings-acl.195/) · [PDF](https://aclanthology.org/2024.findings-acl.195.pdf) · [DOI 10.18653/v1/2024.findings-acl.195](https://doi.org/10.18653/v1/2024.findings-acl.195) · [OpenAlex](https://openalex.org/W4402670433)

**Citations:** 28 citations (OpenAlex, 2026-10-14) · [63](https://md.research.elamur.ai/free/en/papers/yang-2024-pyramidinfer-pyramid-kv-cache-compression/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Serving LLMs at scale is limited by GPU memory, and the KV cache often takes far more memory than the model weights. For a 7B model, the weights need about 14 GB while the cache needs around 72 GB. Earlier compression methods such as H2O, Scissorhands and StreamingLLM only prune the cache after the whole prompt has been processed. They also treat every layer the same way. PyramidInfer instead shrinks the cache during the prefill phase as well as during generation.

It rests on two observations:
- **Inference Context Redundancy:** deeper layers need fewer keys and values.
- **Recent Attention Consistency:** the most recent tokens tend to attend to the same important context, called the Pivotal Context (PvC).

Using these, the method keeps a smaller set of PvC entries as depth grows, so the cache shrinks layer by layer into a pyramid shape. Tests on LLaMA 2 models used OpenCompass benchmarks, including LEval. Against Accelerate, PyramidInfer gives about 2.2x throughput with over 54% less KV memory and little loss in quality. This matters because it lets longer prompts and larger batches fit on a single GPU.

## Key points
- **Layer-wise probe:** on the 40-layer LLaMA 2-13B, cutting PvC hurts perplexity in shallow layers. From Layer 27 onward, perplexity barely changes even when 80% of keys and values are dropped. Sensitivity across layers follows a power-law shape.
- **Fixed-setting efficiency:** on LLaMA 2-13B (512+256 tokens, A100 80GB), KV memory falls from 24.2 GB to 11.0 GB (45.4%). Throughput is 1389 tok/s versus 621 for Accelerate, 934 for Deepspeed and 584 for H2O.
- **Maximum-batch efficiency:** batch size rises to 88, compared with 40–48 for the baselines, giving 2.8x the throughput of Accelerate. Combined with Deepspeed, the gain reaches 3.2x.
- **Larger model:** on LLaMA 2-70B, Accelerate, Deepspeed and H2O all run out of memory during prefill, while PyramidInfer still runs.
- **Ablations:**
  - Removing little in shallow layers and more in deep ones, following the power law, works best. It reaches PPL 4.20, GSM8K 29.56 and MMLU 55.7, slightly above the full cache.
  - A recent-sequence share of about 40% is the chosen trade-off between memory and perplexity.

## Interesting ideas
- **Budget compression per layer.** First measure how much each layer suffers when its context is cut, then allocate the budget accordingly. This probing approach could guide pruning or quantization budgets in other layered models.
- **Use a window of recent queries to decide what to keep.** Their shared attention predicts which context will matter later better than the last token alone. The same idea could select memory in retrieval or long-document systems.
- **Compress before the full computation, not after.** Saving memory while the prompt is being processed avoids the peak memory spike, which is often the real bottleneck for long inputs.
- **Test long-context memory explicitly.** Including a long-context benchmark like LEval shows that recency-only baselines quietly lose history. Short-context scores can hide this kind of failure.
- **Report when the method stops helping.** The paper states that the extra selection work limits speedup at small batch sizes. Being clear about where a method pays off is good practice for any efficiency paper.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 63 works · 15,047 citations
- [South China University of Technology](https://md.research.elamur.ai/free/en/universities/south-china-university-of-technology/) — University, Guangzhou, China · 8 works · 1,433 citations

Also on the byline: [Xiaohongshu (Little Red Book)](https://ror.org/02apw2076).

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 77 works

## Cite

Cite as: Dongjie Yang, Xiaodong Han, Yan Gao, Yao Hu, Shilin Zhang and Hai Zhao. 2024. PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference. Findings of the Association for Computational Linguistics: ACL 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yang2024pyramidinfer,
  title = {{PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference}},
  author = {Dongjie Yang, Xiaodong Han, Yan Gao, Yao Hu, Shilin Zhang and Hai Zhao},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.195},
  url = {https://doi.org/10.18653/v1/2024.findings-acl.195}
}
```

- [← previous LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://md.research.elamur.ai/free/en/papers/wu-2024-longmemeval-benchmarking-chat-assistants-long/)
- [next → Retrieval Head Mechanistically Explains Long-Context Factuality](https://md.research.elamur.ai/free/en/papers/wu-2024-retrieval-head-mechanistically-explains-long/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

29 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, and Yuxiong He. 2022. Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale. [find](https://scholar.google.com/scholar?q=Reza+Yazdani+Aminabadi%2C+Samyam+Rajbhandari%2C+Minjia+Zhang%2C+Ammar+Ahmad+Awan%2C+Cheng+Li%2C+Du+Li%2C+Elton+Zheng%2C+Jeff+Rasley%2C+Shaden+Smith%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+2022%2E+Deepspeed+inference%3A+Enablin)
2. Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. [L-eval: Instituting standardized evaluation for long context language models](https://md.research.elamur.ai/free/en/papers/an-2023-l-eval-instituting-standardized-evaluation/).
3. Anthropic. 2023. Introducing claude. [https://www.anthropic.com/index/introducing-claude](https://www.anthropic.com/index/introducing-claude). [↗](https://www.anthropic.com/index/introducing-claude)
4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/).
7. OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. [https://github.com/open-compass/](https://github.com/open-compass/) opencompass. [↗](https://github.com/open-compass/)
8. Tri Dao. 2023. FlashAttention-2: Faster attention with better parallelism and work partitioning. [find](https://scholar.google.com/scholar?q=Tri+Dao%2E+2023%2E+FlashAttention%2D2%3A+Faster+attention+with+better+parallelism+and+work+partitioning%2E)
9. Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2023. [Model tells you what to discard: Adaptive kv cache compression for llms](https://md.research.elamur.ai/free/en/papers/ge-2023-model-tells-discard-adaptive-kv/). arXiv preprint arXiv:2310.01801.
10. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/).
11. Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. 2019. Gpipe: Efficient training of giant neural networks using pipeline parallelism. [find](https://scholar.google.com/scholar?q=Yanping+Huang%2C+Youlong+Cheng%2C+Ankur+Bapna%2C+Orhan+Firat%2C+Mia+Xu+Chen%2C+Dehao+Chen%2C+HyoukJoong+Lee%2C+Jiquan+Ngiam%2C+Quoc+V%2E+Le%2C+Yonghui+Wu%2C+and+Zhifeng+Chen%2E+2019%2E+Gpipe%3A+Efficient+training+of+giant+neural)
12. HuggingFace. 2021. Hugging face accelerate. [https://huggingface.co/docs/accelerate/index](https://huggingface.co/docs/accelerate/index). [↗](https://huggingface.co/docs/accelerate/index)
13. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
14. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. [find](https://scholar.google.com/scholar?q=Woosuk+Kwon%2C+Zhuohan+Li%2C+Siyuan+Zhuang%2C+Ying+Sheng%2C+Lianmin+Zheng%2C+Cody+Hao+Yu%2C+Joseph+E%2E)
15. Gonzalez, Hao Zhang, and Ion Stoica. 2023. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/).
16. Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, A… [find](https://scholar.google.com/scholar?q=Pushkar+Mishra%2C+Igor+Molybog%2C+Yixin+Nie%2C+Andrew+Poulton%2C+Jeremy+Reizenstein%2C+Rashi+Rungta%2C+Kalyan+Saladi%2C+Alan+Schelten%2C+Ruan+Silva%2C+Eric+Michael+Smith%2C+Ranjan+Subramanian%2C+Xiaoqing+Ellen+Tan%2C+Binh+Ta)
17. Alex M Lamb, Anirudh Goyal ALIAS PARTH GOYAL, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. 2016. Professor forcing: A new algorithm for training recurrent networks. Advances in neural information processing systems, 29. [find](https://scholar.google.com/scholar?q=Alex+M+Lamb%2C+Anirudh+Goyal+ALIAS+PARTH+GOYAL%2C+Ying+Zhang%2C+Saizheng+Zhang%2C+Aaron+C+Courville%2C+and+Yoshua+Bengio%2E+2016%2E+Professor+forcing%3A+A+new+algorithm+for+training+recurrent+networks%2E+Advances+in+ne)
18. Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2023. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. [find](https://scholar.google.com/scholar?q=Zichang+Liu%2C+Aditya+Desai%2C+Fangshuo+Liao%2C+Weitao+Wang%2C+Victor+Xie%2C+Zhaozhuo+Xu%2C+Anastasios+Kyrillidis%2C+and+Anshumali+Shrivastava%2E+2023%2E+Scissorhands%3A+Exploiting+the+persistence+of+importance+hypothesi)
19. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/). arXiv preprint arXiv:1609.07843.
20. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
21. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
22. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal… — [Code Llama: Open Foundation Models for Code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
23. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+2023%2E+Efficient+streaming+language+models+with+attention+sinks%2E)
24. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068.
25. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. H 2 o: Heavy-hitter oracle for efficient generative inference of large language models. [find](https://scholar.google.com/scholar?q=Zhenyu+Zhang%2C+Ying+Sheng%2C+Tianyi+Zhou%2C+Tianlong+Chen%2C+Lianmin+Zheng%2C+Ruisi+Cai%2C+Zhao+Song%2C+Yuandong+Tian%2C+Christopher+R%C3%A9%2C+Clark+Barrett%2C+Zhangyang+Wang%2C+and+Beidi+Chen%2E+2023%2E+H+2+o%3A+Heavy%2Dhitter+oracl)
26. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+P%2E+Xing%2C+Hao+Zhang%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+2023%2E+Judging+llm%2Das)
27. Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. [Flexgen: High-throughput generative inference of large language models with a single gpu](https://md.research.elamur.ai/free/en/papers/sheng-2023-flexgen-high-throughput-generative-inference/).
28. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). arXiv preprint arXiv:2206.04615.
29. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+Dan+Bikel%2C+Lukas+Blecher%2C+Cristian+Canton+Fe)

*Updated 24 October 2026*