---
title: "DuoAttention — Xiao 2024"
description: "Serving LLMs over very long inputs is costly because the KV cache grows linearly with context length, and attention compute grows with it."
authors: ["Guangxuan Xiao", "Jiaming Tang", "Jingwei Zuo", "Junxian Guo", "Shang Yang", "Haotian Tang", "Yao Fu", "Song Han"]
year: 2024
venue: "arXiv:2410.10819"
source: "https://arxiv.org/abs/2410.10819"
doi: "10.48550/arxiv.2410.10819"
arxiv: "2410.10819"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xiao-2024-duoattention-efficient-long-context-llm/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xiao-2024-duoattention-efficient-long-context-llm/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/) / DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads

# DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads

**Authors:** [Guangxuan Xiao](https://md.research.elamur.ai/free/en/authors/guangxuan-xiao/), [Jiaming Tang](https://md.research.elamur.ai/free/en/authors/jiaming-tang/), [Jingwei Zuo](https://md.research.elamur.ai/free/en/authors/jingwei-zuo/), Junxian Guo, Shang Yang, [Haotian Tang](https://md.research.elamur.ai/free/en/authors/haotian-tang/), [Yao Fu](https://md.research.elamur.ai/free/en/authors/yao-fu/), [Song Han](https://md.research.elamur.ai/free/en/authors/song-han/) · **Year:** 2024 · **Venue:** arXiv:2410.10819 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2410.10819) · [PDF](https://arxiv.org/pdf/2410.10819) · [DOI 10.48550/arxiv.2410.10819](https://doi.org/10.48550/arxiv.2410.10819) · [arXiv 2410.10819](https://arxiv.org/abs/2410.10819) · [OpenAlex](https://openalex.org/W4403571350)

**Citations:** 2 citations (openalex-snapshot · 2026-10-14) · [107](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Serving LLMs over very long inputs is costly because the KV cache grows linearly with context length, and attention compute grows with it. Existing cache-pruning methods either hurt long-context ability or save little. The authors observe that attention heads split into two kinds. A small set of Retrieval Heads must see every token. Most heads are Streaming Heads, which mainly attend to attention sinks and the latest tokens. DuoAttention learns one gate per head on a synthetic passkey-recall dataset, with all model weights frozen. The training target is to stay close to the full-attention output, with a penalty that pushes gates toward streaming. At inference, heads above a threshold keep a full KV cache, and the rest keep a small fixed-size cache. On Llama-2/3 and Mistral, this reduces memory and speeds up both pre-filling and decoding, with accuracy close to full attention. Combined with quantization, it lets Llama-3-8B decode with millions of tokens of context on one GPU, which makes million-token workloads far more practical.

## Key points
- Reported gains: up to 2.55× memory reduction for MHA models and 1.67× for GQA models. Decoding is up to 2.18× (MHA) and 1.50× (GQA) faster, and pre-filling up to 1.73× and 1.63× faster.
- With 8-bit weights and a 4-bit KV cache, Llama-3-8B handles about 3.3 million tokens on a single A100. This is roughly 6.4× the capacity of a plain FP16 full-attention setup.
- Head identification needs only a few hours on one 8-GPU node, because the gates are the only trainable parameters. The training data has ten 32-word passkeys hidden in long text, and the loss is computed only on the passkey tokens.
- LongBench averages:
  - Llama-3-8B-Instruct-1048K at a 50% KV budget: DuoAttention 40.21 vs. full attention 40.08, H2O 35.76, TOVA 35.55, StreamingLLM 32.26.
  - Llama-2-7B-32K-Instruct at 25%: DuoAttention 34.49 vs. full attention 37.52, with the baselines between 26.84 and 29.78.
- DuoAttention also beats FastGen on the LongBench subsets FastGen could run: 40.01 vs. 32.82 and 32.81 vs. 19.01. FastGen's quadratic profiling cost limited it to about 24k–32k tokens.

## Interesting ideas
- **Score components by their effect on the output.** Each head is judged by how much the model's output shifts when its context is truncated, not by looking at attention maps. Ranking parts of a model by their impact on outputs could guide pruning or compression in many other architectures.
- **Freeze the model, train only cheap gates.** Learnable mixing weights between an expensive and a cheap path are trained with a sparsity penalty, then binarized with a threshold. This is a general, low-cost recipe for deciding where heavy computation is truly needed.
- **Targeted synthetic data.** Many passkeys are planted in long filler text, and the loss is applied only to the answer spans. This isolates exactly the skill to preserve, and it could be reused to probe long-range behavior in other languages or domains without labeled corpora.
- **Two caches per layer.** Each layer keeps a full cache for some heads and a constant-size cache for the others. The design stays simple and works alongside GQA and quantization, which suggests mixed-precision or mixed-retention designs as a practical deployment pattern.
- **Stress-testing decode-time eviction.** The last 50 tokens of each benchmark input are treated as if they were generated, so eviction-based baselines cannot escape their policy on short-answer tasks. This makes comparisons between cache-eviction methods fairer.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [Computing & GPUs](https://md.research.elamur.ai/free/en/shelves/computing/)

## Cited by

Cited by

107 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 106 not yet.

- [Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention](https://md.research.elamur.ai/free/en/papers/yuan-2025-native-sparse-attention-hardware/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{xiao2024duoattention,
  title = {{DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads}},
  author = {Xiao, Guangxuan and Tang, Jiaming and Zuo, Jingwei and Guo, Junxian and Yang, Shang and Tang, Haotian and Fu, Yao and Han, Song},
  year = {2024},
  doi = {10.48550/arxiv.2410.10819},
  eprint = {2410.10819},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.10819}
}
```

- [← previous Dreaming Out Loud: A Self-Synthesis Approach For Training Vision-Language Models With Developmentally Plausible Data](https://md.research.elamur.ai/free/en/papers/alkhamissi-2024-dreaming-out-loud-self-synthesis/)
- [next → dzStance at StanceEval2024: Arabic Stance Detection based on Sentence Transformers](https://md.research.elamur.ai/free/en/papers/lichouri-2024-dzstance-stanceeval2024-arabic-stance-detection/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/)

## References

57 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Griffin Adams, Faisal Ladhak, Hailey Schoelkopf, and Raja Biswas. Cold compress: A toolkit for benchmarking kv cache compression approaches, 8 2024. URL [https://www.answer.ai/posts/2024-08-01-cold-compress.html](https://www.answer.ai/posts/2024-08-01-cold-compress.html). [↗](https://www.answer.ai/posts/2024-08-01-cold-compress.html)
2. Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, and Ramachandran Ramjee. Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills, 2023. URL https://arxiv.org/abs/2308.16369. [↗](https://arxiv.org/abs/2308.16369)
3. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023. [find](https://scholar.google.com/scholar?q=Joshua+Ainslie%2C+James+Lee%2DThorp%2C+Michiel+de+Jong%2C+Yury+Zemlyanskiy%2C+Federico+Lebr%C3%B3n%2C+and+Sumit+Sanghai%2E+Gqa%3A+Training+generalized+multi%2Dquery+transformer+models+from+multi%2Dhead+checkpoints%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/)
4. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. [find](https://scholar.google.com/scholar?q=Jacob+Austin%2C+Augustus+Odena%2C+Maxwell+Nye%2C+Maarten+Bosma%2C+Henryk+Michalewski%2C+David+Dohan%2C+Ellen+Jiang%2C+Carrie+Cai%2C+Michael+Terry%2C+Quoc+Le%2C+et+al%2E+Program+synthesis+with+large+language+models%2E+arXiv+p) [our card](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/)
5. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023. [find](https://scholar.google.com/scholar?q=Yushi+Bai%2C+Xin+Lv%2C+Jiajie+Zhang%2C+Hongchang+Lyu%2C+Jiankai+Tang%2C+Zhidian+Huang%2C+Zhengxiao+Du%2C+Xiao+Liu%2C+Aohan+Zeng%2C+Lei+Hou%2C+Yuxiao+Dong%2C+Jie+Tang%2C+and+Juanzi+Li%2E+Longbench%3A+A+bilingual%2C+multitask+benchm) [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/)
6. Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer, 2020. arXiv:2004.05150. [find](https://scholar.google.com/scholar?q=Iz+Beltagy%2C+Matthew+E%2E+Peters%2C+and+Arman+Cohan%2E+Longformer%3A+The+long%2Ddocument+transformer%2C+2020%2E+arXiv%3A2004%2E05150%2E)
7. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20B: An open-source autoregressive… [find](https://scholar.google.com/scholar?q=Sid+Black%2C+Stella+Biderman%2C+Eric+Hallahan%2C+Quentin+Anthony%2C+Leo+Gao%2C+Laurence+Golding%2C+Horace+He%2C+Connor+Leahy%2C+Kyle+McDonell%2C+Jason+Phang%2C+Michael+Pieler%2C+USVSN+Sai+Prashanth%2C+Shivanshu+Purohit%2C+Lari) [our card](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/)
8. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL [https://lmsys.org/blog/2023-03-30-vicuna…](https://lmsys.org/blog/2023-03-30-vicuna…) [↗](https://lmsys.org/blog/2023-03-30-vicuna…)
9. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. 2019. [find](https://scholar.google.com/scholar?q=Rewon+Child%2C+Scott+Gray%2C+Alec+Radford%2C+and+Ilya+Sutskever%2E+Generating+long+sequences+with+sparse+transformers%2E+2019%2E)
10. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of BERT’s attention. In Tal Linzen, Grzegorz Chrupała, Yonatan Belinkov, and Dieuwke Hupkes (eds.), Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks… [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Urvashi+Khandelwal%2C+Omer+Levy%2C+and+Christopher+D%2E+Manning%2E+What+does+BERT+look+at%3F+an+analysis+of+BERT%E2%80%99s+attention%2E+In+Tal+Linzen%2C+Grzegorz+Chrupa%C5%82a%2C+Yonatan+Belinkov%2C+and+Dieuwke+Hupkes+)
11. Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning, 2023. [find](https://scholar.google.com/scholar?q=Tri+Dao%2E+FlashAttention%2D2%3A+Faster+attention+with+better+parallelism+and+work+partitioning%2C+2023%2E)
12. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness, 2022. arXiv:2205.14135. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Daniel+Y%2E+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+FlashAttention%3A+Fast+and+memory%2Defficient+exact+attention+with+IO%2Dawareness%2C+2022%2E+arXiv%3A2205%2E14135%2E)
13. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez,… [find](https://scholar.google.com/scholar?q=Abhimanyu+Dubey%2C+Abhinav+Jauhri%2C+Abhinav+Pandey%2C+Abhishek+Kadian%2C+Ahmad+Al%2DDahle%2C+Aiesha+Letman%2C+Akhil+Mathur%2C+Alan+Schelten%2C+Amy+Yang%2C+Angela+Fan%2C+Anirudh+Goyal%2C+Anthony+Hartshorn%2C+Aobo+Yang%2C+Archi+M)
14. Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for LLMs. In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=uNrFpDPMyo](https://openreview.net/forum?id=uNrFpDPMyo). [↗](https://openreview.net/forum?id=uNrFpDPMyo) [our card](https://md.research.elamur.ai/free/en/papers/ge-2023-model-tells-discard-adaptive-kv/)
15. Tanya Goyal and Greg Durrett. Evaluating factuality in generation with dependency-level entailment. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online, 2020. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tanya+Goyal+and+Greg+Durrett%2E+Evaluating+factuality+in+generation+with+dependency%2Dlevel+entailment%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+Online%2C+2020%2E+Association+) [our card](https://md.research.elamur.ai/free/en/papers/goyal-2020-evaluating-factuality-generation-dependency-level/)
16. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2023. [find](https://scholar.google.com/scholar?q=Albert+Gu+and+Tri+Dao%2E+Mamba%3A+Linear%2Dtime+sequence+modeling+with+selective+state+spaces%2C+2023%2E)
17. Junxian Guo, Haotian Tang, Shang Yang, Zhekai Zhang, Zhijian Liu, and Song Han. Block Sparse Attention. [https://github.com/mit-han-lab/Block-Sparse-Attention](https://github.com/mit-han-lab/Block-Sparse-Attention), 2024. [↗](https://github.com/mit-han-lab/Block-Sparse-Attention)
18. Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. LM-Infinite: Simple on-the-fly length generalization for large language models, 2023. [find](https://scholar.google.com/scholar?q=Chi+Han%2C+Qifan+Wang%2C+Wenhan+Xiong%2C+Yu+Chen%2C+Heng+Ji%2C+and+Sinong+Wang%2E+LM%2DInfinite%3A+Simple+on%2Dthe%2Dfly+length+generalization+for+large+language+models%2C+2023%2E)
19. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Collin+Burns%2C+Steven+Basart%2C+Andy+Zou%2C+Mantas+Mazeika%2C+Dawn+Song%2C+and+Jacob+Steinhardt%2E+Measuring+massive+multitask+language+understanding%2E+Proceedings+of+the+International+Conference+o) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
20. Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Yuhan Dong, and Yu Wang. Flashdecoding++: Faster large language model inference on gpus, 2024. [find](https://scholar.google.com/scholar?q=Ke+Hong%2C+Guohao+Dai%2C+Jiaming+Xu%2C+Qiuli+Mao%2C+Xiuhong+Li%2C+Jun+Liu%2C+Kangdi+Chen%2C+Yuhan+Dong%2C+and+Yu+Wang%2E+Flashdecoding%2B%2B%3A+Faster+large+language+model+inference+on+gpus%2C+2024%2E)
21. Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length llm inference with kv cache quantization, 2024. [find](https://scholar.google.com/scholar?q=Coleman+Hooper%2C+Sehoon+Kim%2C+Hiva+Mohammadzadeh%2C+Michael+W%2E+Mahoney%2C+Yakun+Sophia+Shao%2C+Kurt+Keutzer%2C+and+Amir+Gholami%2E+Kvquant%3A+Towards+10+million+context+length+llm+inference+with+kv+cache+quantizati) [our card](https://md.research.elamur.ai/free/en/papers/hooper-2024-kvquant-towards-10-million-context/)
22. Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models, 2023. URL https://arxiv.org/abs/2309.14509. [↗](https://arxiv.org/abs/2309.14509) [our card](https://md.research.elamur.ai/free/en/papers/jacobs-2023-deepspeed-ulysses-system-optimizations-enabling/)
23. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
24. Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. arXiv preprint arXiv:2407.02490, 2024. [find](https://scholar.google.com/scholar?q=Huiqiang+Jiang%2C+Yucheng+Li%2C+Chengruidong+Zhang%2C+Qianhui+Wu%2C+Xufang+Luo%2C+Surin+Ahn%2C+Zhenhua+Han%2C+Amir+H+Abdi%2C+Dongsheng+Li%2C+Chin%2DYew+Lin%2C+Yuqing+Yang%2C+and+Lili+Qiu%2E+Minference+1%2E0%3A+Accelerating+pre%2Dfil) [our card](https://md.research.elamur.ai/free/en/papers/jiang-2024-minference-1-0-accelerating-pre/)
25. Greg Kamradt. Llmtest_needleinahaystack: Doing simple retrieval from llm models at various context lengths to measure accuracy. [https://github.com/gkamradt/LLMTest_NeedleInAHaystack](https://github.com/gkamradt/LLMTest_NeedleInAHaystack), 2024. Accessed: 2024-05-23. [↗](https://github.com/gkamradt/LLMTest_NeedleInAHaystack)
26. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980. [↗](http://arxiv.org/abs/1412.6980)
27. Wojciech Kryściński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. Booksum: A collection of datasets for long-form narrative summarization. 2021. [find](https://scholar.google.com/scholar?q=Wojciech+Krys%CC%81cin%CC%81ski%2C+Nazneen+Rajani%2C+Divyansh+Agarwal%2C+Caiming+Xiong%2C+and+Dragomir+Radev%2E+Booksum%3A+A+collection+of+datasets+for+long%2Dform+narrative+summarization%2E+2021%2E)
28. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. [find](https://scholar.google.com/scholar?q=Woosuk+Kwon%2C+Zhuohan+Li%2C+Siyuan+Zhuang%2C+Ying+Sheng%2C+Lianmin+Zheng%2C+Cody+Hao+Yu%2C+Joseph+E%2E+Gonzalez%2C+Hao+Zhang%2C+and+Ion+Stoica%2E+Efficient+memory+management+for+large+language+model+serving+with+pagedat)
29. Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection, 2023. [find](https://scholar.google.com/scholar?q=Bin+Lin%2C+Yang+Ye%2C+Bin+Zhu%2C+Jiaxi+Cui%2C+Munan+Ning%2C+Peng+Jin%2C+and+Li+Yuan%2E+Video%2Dllava%3A+Learning+united+visual+representation+by+alignment+before+projection%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/lin-2023-video-llava-learning-united-visual/)
30. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration, 2024. [find](https://scholar.google.com/scholar?q=Ji+Lin%2C+Jiaming+Tang%2C+Haotian+Tang%2C+Shang+Yang%2C+Wei%2DMing+Chen%2C+Wei%2DChen+Wang%2C+Guangxuan+Xiao%2C+Xingyu+Dang%2C+Chuang+Gan%2C+and+Song+Han%2E+Awq%3A+Activation%2Daware+weight+quantization+for+llm+compression+and+a)
31. Yujun Lin*, Haotian Tang*, Shang Yang*, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. Qserve: W4a8kv4 quantization and system co-design for efficient llm serving. arXiv preprint arXiv:2405.04532, 2024. [find](https://scholar.google.com/scholar?q=Yujun+Lin%2A%2C+Haotian+Tang%2A%2C+Shang+Yang%2A%2C+Zhekai+Zhang%2C+Guangxuan+Xiao%2C+Chuang+Gan%2C+and+Song+Han%2E+Qserve%3A+W4a8kv4+quantization+and+system+co%2Ddesign+for+efficient+llm+serving%2E+arXiv+preprint+arXiv%3A2405%2E0)
32. Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context, 2023a. [find](https://scholar.google.com/scholar?q=Hao+Liu%2C+Matei+Zaharia%2C+and+Pieter+Abbeel%2E+Ring+attention+with+blockwise+transformers+for+near%2Dinfinite+context%2C+2023a%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-ring-attention-blockwise-transformers-near/)
33. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b. [find](https://scholar.google.com/scholar?q=Haotian+Liu%2C+Chunyuan+Li%2C+Qingyang+Wu%2C+and+Yong+Jae+Lee%2E+Visual+instruction+tuning%2C+2023b%2E)
34. Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In ICCV, 2017. [find](https://scholar.google.com/scholar?q=Zhuang+Liu%2C+Jianguo+Li%2C+Zhiqiang+Shen%2C+Gao+Huang%2C+Shoumeng+Yan%2C+and+Changshui+Zhang%2E+Learning+efficient+convolutional+networks+through+network+slimming%2E+In+ICCV%2C+2017%2E)
35. Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024. [find](https://scholar.google.com/scholar?q=Zirui+Liu%2C+Jiayi+Yuan%2C+Hongye+Jin%2C+Shaochen+Zhong%2C+Zhaozhuo+Xu%2C+Vladimir+Braverman%2C+Beidi+Chen%2C+and+Xia+Hu%2E+Kivi%3A+A+tuning%2Dfree+asymmetric+2bit+quantization+for+kv+cache%2E+arXiv+preprint+arXiv%3A2402%2E027) [our card](https://md.research.elamur.ai/free/en/papers/liu-2024-kivi-tuning-free-asymmetric-2bit/)
36. OpenAI. Gpt-4 technical report, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
37. Matanel Oren, Michael Hassid, Yossi Adi, and Roy Schwartz. Transformers are multi-state rnns, 2024. [find](https://scholar.google.com/scholar?q=Matanel+Oren%2C+Michael+Hassid%2C+Yossi+Adi%2C+and+Roy+Schwartz%2E+Transformers+are+multi%2Dstate+rnns%2C+2024%2E)
38. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and S… [find](https://scholar.google.com/scholar?q=Adam+Paszke%2C+Sam+Gross%2C+Francisco+Massa%2C+Adam+Lerer%2C+James+Bradbury%2C+Gregory+Chanan%2C+Trevor+Killeen%2C+Zeming+Lin%2C+Natalia+Gimelshein%2C+Luca+Antiga%2C+Alban+Desmaison%2C+Andreas+K%C3%B6pf%2C+Edward+Yang%2C+Zachary+De) [our card](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
39. Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models, 2023. URL https://arxiv.org/abs/2302.10866. [↗](https://arxiv.org/abs/2302.10866)
40. John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Felipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, et al. Chatgpt: Optimizing language models for dialogue. OpenAI blog, 2022. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Barret+Zoph%2C+Christina+Kim%2C+Jacob+Hilton%2C+Jacob+Menick%2C+Jiayi+Weng%2C+Juan+Felipe+Ceron+Uribe%2C+Liam+Fedus%2C+Luke+Metz%2C+Michael+Pokorny%2C+et+al%2E+Chatgpt%3A+Optimizing+language+models+for+dialo)
41. Noam Shazeer. Fast transformer decoding: One write-head is all you need, 2019. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+Fast+transformer+decoding%3A+One+write%2Dhead+is+all+you+need%2C+2019%2E)
42. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021. [find](https://scholar.google.com/scholar?q=Jianlin+Su%2C+Yu+Lu%2C+Shengfeng+Pan%2C+Ahmed+Murtadha%2C+Bo+Wen%2C+and+Yunfeng+Liu%2E+Roformer%3A+Enhanced+transformer+with+rotary+position+embedding%2E+arXiv+preprint+arXiv%3A2104%2E09864%2C+2021%2E)
43. Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Shikuan Hong, Yiwu Yao, and Gongyi Wang. Razorattention: Efficient kv cache compression through retrieval heads, 2024a. URL https://arxiv.org/abs/2407.15891. — [arXiv:2407.15891](https://arxiv.org/abs/2407.15891) [↗](https://arxiv.org/abs/2407.15891)
44. Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context llm inference, 2024b. [find](https://scholar.google.com/scholar?q=Jiaming+Tang%2C+Yilong+Zhao%2C+Kan+Zhu%2C+Guangxuan+Xiao%2C+Baris+Kasikci%2C+and+Song+Han%2E+Quest%3A+Query%2Daware+sparsity+for+efficient+long%2Dcontext+llm+inference%2C+2024b%2E) [our card](https://md.research.elamur.ai/free/en/papers/tang-2024-quest-query-aware-sparsity-efficient/)
45. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
46. R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society (Series B), 58:267–288, 1996. [find](https://scholar.google.com/scholar?q=R%2E+Tibshirani%2E+Regression+shrinkage+and+selection+via+the+lasso%2E+Journal+of+the+Royal+Statistical+Society+%28Series+B%29%2C+58%3A267%E2%80%93288%2C+1996%2E)
47. Together. Llama-2-7b-32k-instruct — and fine-tuning for llama-2 models with together api, June 2023. URL [https://together.ai/blog/llama-2-7b-32k-instruct](https://together.ai/blog/llama-2-7b-32k-instruct). [↗](https://together.ai/blog/llama-2-7b-32k-instruct)
48. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Thibaut+Lavril%2C+Gautier+Izacard%2C+Xavier+Martinet%2C+Marie%2DAnne+Lachaux%2C+Timoth%C3%A9e+Lacroix%2C+Baptiste+Rozi%C3%A8re%2C+Naman+Goyal%2C+Eric+Hambro%2C+Faisal+Azhar%2C+et+al%2E+Llama%3A+Open+and+efficient+foundat)
49. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtun)
50. Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanistically explains long-context factuality, 2024. [find](https://scholar.google.com/scholar?q=Wenhao+Wu%2C+Yizhong+Wang%2C+Guangxuan+Xiao%2C+Hao+Peng%2C+and+Yao+Fu%2E+Retrieval+head+mechanistically+explains+long%2Dcontext+factuality%2C+2024%2E)
51. Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, 2023a. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Ji+Lin%2C+Mickael+Seznec%2C+Hao+Wu%2C+Julien+Demouth%2C+and+Song+Han%2E+SmoothQuant%3A+Accurate+and+efficient+post%2Dtraining+quantization+for+large+language+models%2E+In+Proceedings+of+the+40th+Inter)
52. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv, 2023b. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+Efficient+streaming+language+models+with+attention+sinks%2E+arXiv%2C+2023b%2E)
53. Zihao Ye, Ruihang Lai, Roy Lu, Chien-Yu Lin, Size Zheng, Lequn Chen, Tianqi Chen, and Luis Ceze. Cascade inference: Memory bandwidth efficient shared prefix batch decoding. [https://flashinfer.ai/2024/01/08/cascade-inference.html](https://flashinfer.ai/2024/01/08/cascade-inference.html), Jan 2024. URL https://flashinfer.ai/2024/01/08/cascade-inference.html… [↗](https://flashinfer.ai/2024/01/08/cascade-inference.html)
54. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big Bird: Transformers for longer sequences. In Proc. of NeurIPS, volume 33, 2020. [find](https://scholar.google.com/scholar?q=Manzil+Zaheer%2C+Guru+Guruganesh%2C+Kumar+Avinava+Dubey%2C+Joshua+Ainslie%2C+Chris+Alberti%2C+Santiago+Ontanon%2C+Philip+Pham%2C+Anirudh+Ravula%2C+Qifan+Wang%2C+Li+Yang%2C+and+Amr+Ahmed%2E+Big+Bird%3A+Transformers+for+longer) [our card](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/)
55. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. Benchmarking large language models for news summarization, 2023a. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Faisal+Ladhak%2C+Esin+Durmus%2C+Percy+Liang%2C+Kathleen+McKeown%2C+and+Tatsunori+B%2E+Hashimoto%2E+Benchmarking+large+language+models+for+news+summarization%2C+2023a%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2024-benchmarking-large-language-models-news/)
56. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H 2 o: Heavy-hitter oracle for efficient generative inference of large language models, 2023b. [find](https://scholar.google.com/scholar?q=Zhenyu+Zhang%2C+Ying+Sheng%2C+Tianyi+Zhou%2C+Tianlong+Chen%2C+Lianmin+Zheng%2C+Ruisi+Cai%2C+Zhao+Song%2C+Yuandong+Tian%2C+Christopher+R%C3%A9%2C+Clark+Barrett%2C+Zhangyang+Wang%2C+and+Beidi+Chen%2E+H+2+o%3A+Heavy%2Dhitter+oracle+for+)
57. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+P%2E+Xing%2C+Hao+Zhang%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+Judging+llm%2Das%2Da%2Djud)