---
title: "KIVI: A Tuning-Free Asymmetric 2bit Quantization — Liu 2024"
description: "Serving LLMs cheaply means batching many requests. With large batches and long contexts, though, the KV cache becomes the main memory and speed…"
authors: ["Zirui Liu", "Jiayi Yuan", "Hongye Jin", "Shaochen Zhong", "Zhaozhuo Xu", "Vladimir Braverman", "Beidi Chen", "Xia Hu"]
year: 2024
venue: "arXiv:2402.02750"
source: "https://arxiv.org/abs/2402.02750"
doi: "10.13140/rg.2.2.28167.37282"
arxiv: "2402.02750"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2024-kivi-tuning-free-asymmetric-2bit/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2024-kivi-tuning-free-asymmetric-2bit/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

# KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

**Authors:** Zirui Liu, [Jiayi Yuan](https://md.research.elamur.ai/free/en/authors/jiayi-yuan/), [Hongye Jin](https://md.research.elamur.ai/free/en/authors/hongye-jin/), [Shaochen Zhong](https://md.research.elamur.ai/free/en/authors/shaochen-zhong/), Zhaozhuo Xu, Vladimir Braverman, [Beidi Chen](https://md.research.elamur.ai/free/en/authors/beidi-chen/), [Xia Hu](https://md.research.elamur.ai/free/en/authors/xia-hu/) · **Year:** 2024 · **Venue:** arXiv:2402.02750 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://doi.org/10.13140/rg.2.2.28167.37282) · [PDF](https://arxiv.org/pdf/2402.02750) · [DOI 10.13140/rg.2.2.28167.37282](https://doi.org/10.13140/rg.2.2.28167.37282) · [arXiv 2402.02750](https://arxiv.org/abs/2402.02750) · [OpenAlex](https://openalex.org/W4391591383)

**Citations:** 6 citations (openalex-snapshot · 2026-10-14) · [237](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Serving LLMs cheaply means batching many requests. With large batches and long contexts, though, the KV cache becomes the main memory and speed bottleneck: it can exceed the size of the model weights, and the GPU must read it in full for every generated token. The authors examine how values are distributed inside the KV caches of popular LLMs. They find that a few key channels carry consistently large outliers, while value tensors show no such pattern. Because values are mixed by attention weights, errors should stay confined within each token. From this they derive KIVI, a 2bit quantizer that needs no tuning. It quantizes keys per-channel and values per-token. To handle streaming generation, it stores grouped tokens in low precision and keeps a short residual window in full precision. Tested on Llama, Falcon and Mistral, KIVI keeps quality close to 16bit while cutting peak memory substantially. This permits much larger batches and higher throughput, making KV cache quantization a simple, training-free tool for efficient inference.

## Key points
- **Core design:** keys are grouped and quantized along the channel dimension, and values along the token dimension. Recent tokens that do not yet fill a group (the residual part) stay in full precision. The two parts are combined through tiled matrix multiplication.
- **Pilot study (Llama-2-13B, group size 32):**
  - On CoQA, 2bit per-token quantization of both caches drops the score from 66.37 to 52.93. The key-per-channel/value-per-token setup reaches 63.53.
  - Quantizing values per-channel collapses accuracy to roughly 3.
- **Efficiency on Llama-2-7B:** peak memory falls about 2.6× (model weights included). This allows up to 4× larger batches and 2.35×–3.47× throughput.
- **LongBench averages:**
  - Llama2-7B: 44.52 (16bit) vs. 44.27 (KIVI-2).
  - Mistral-7B: 46.58 vs. 45.85.
  - Falcon-7B shows a larger relative drop: 8.71 vs. 7.95.
- **Venue and code:** the paper appeared at ICML 2024, and the code is available at github.com/jy-yuan/KIVI. The approach is orthogonal to token eviction (H2O, StreamingLLM) and to system-level methods (vLLM).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/) · [Computing & GPUs](https://md.research.elamur.ai/free/en/shelves/computing/)

## Cited by

Cited by

237 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 231 not yet.

- [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://md.research.elamur.ai/free/en/papers/ashkboos-2024-quarot-outlier-free-4-bit/)
- [MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention](https://md.research.elamur.ai/free/en/papers/jiang-2024-minference-1-0-accelerating-pre/)
- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://md.research.elamur.ai/free/en/papers/sui-2025-stop-overthinking-survey-efficient-reasoning/)
- [DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads](https://md.research.elamur.ai/free/en/papers/xiao-2024-duoattention-efficient-long-context-llm/)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://md.research.elamur.ai/free/en/papers/zheng-2023-sglang-efficient-execution-structured-language/)
- [A Survey on Model Compression for Large Language Models](https://md.research.elamur.ai/free/en/papers/zhu-2024-survey-model-compression-large-language/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{liu2024kivi,
  title = {{KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache}},
  author = {Liu, Zirui and Yuan, Jiayi and Jin, Hongye and Zhong, Shaochen and Xu, Zhaozhuo and Braverman, Vladimir and Chen, Beidi and Hu, Xia},
  year = {2024},
  doi = {10.13140/rg.2.2.28167.37282},
  eprint = {2402.02750},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.13140/rg.2.2.28167.37282}
}
```

- [← previous Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge](https://md.research.elamur.ai/free/en/papers/ye-2024-justice-prejudice-quantifying-biases-llm/)
- [next → Knesset-DictaBERT: A Hebrew Language Model for Parliamentary Proceedings](https://md.research.elamur.ai/free/en/papers/langstream-2024-goldin-knesset-dictabert-hebrew-language-model/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

29 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. ShareGPT Team. [https://sharegpt.com/](https://sharegpt.com/), 2023. [↗](https://sharegpt.com/)
2. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023. [find](https://scholar.google.com/scholar?q=Joshua+Ainslie%2C+James+Lee%2DThorp%2C+Michiel+de+Jong%2C+Yury+Zemlyanskiy%2C+Federico+Lebr%C3%B3n%2C+and+Sumit+Sanghai%2E+Gqa%3A+Training+generalized+multi%2Dquery+transformer+models+from+multi%2Dhead+checkpoints%2E+arXiv+prep) [our card](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/)
3. Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629, 2023. [find](https://scholar.google.com/scholar?q=Sehoon+Kim%2C+Coleman+Hooper%2C+Amir+Gholami%2C+Zhen+Dong%2C+Xiuyu+Li%2C+Sheng+Shen%2C+Michael+W+Mahoney%2C+and+Kurt+Keutzer%2E+Squeezellm%3A+Dense%2Dand%2Dsparse+quantization%2E+arXiv+preprint+arXiv%3A2306%2E07629%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2023-squeezellm-dense-sparse-quantization/)
4. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023. [find](https://scholar.google.com/scholar?q=Yushi+Bai%2C+Xin+Lv%2C+Jiajie+Zhang%2C+Hongchang+Lyu%2C+Jiankai+Tang%2C+Zhidian+Huang%2C+Zhengxiao+Du%2C+Xiao+Liu%2C+Aohan+Zeng%2C+Lei+Hou%2C+Yuxiao+Dong%2C+Jie+Tang%2C+and+Juanzi+Li%2E+Longbench%3A+A+bilingual%2C+multitask+benchm) [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/)
5. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626, 2023. [find](https://scholar.google.com/scholar?q=Woosuk+Kwon%2C+Zhuohan+Li%2C+Siyuan+Zhuang%2C+Ying+Sheng%2C+Lianmin+Zheng%2C+Cody+Hao+Yu%2C+Joseph+Gonzalez%2C+Hao+Zhang%2C+and+Ion+Stoica%2E+Efficient+memory+management+for+large+language+model+serving+with+pagedatten)
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+)
7. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023. [find](https://scholar.google.com/scholar?q=Ji+Lin%2C+Jiaming+Tang%2C+Haotian+Tang%2C+Shang+Yang%2C+Xingyu+Dang%2C+and+Song+Han%2E+Awq%3A+Activation%2Daware+weight+quantization+for+llm+compression+and+acceleration%2E+arXiv+preprint+arXiv%3A2306%2E00978%2C+2023%2E)
8. Yu-Neng Chuang, Songchen Li, Jiayi Yuan, Guanchu Wang, Kwei-Herng Lai, Leisheng Yu, Sirui Ding, Chia-Yuan Chang, Qiaoyu Tan, Daochen Zha, et al. Understanding different design choices in training large time series models. arXiv preprint arXiv:2406.14045, 2024. — [arXiv:2406.14045](https://arxiv.org/abs/2406.14045) [find](https://scholar.google.com/scholar?q=Yu%2DNeng+Chuang%2C+Songchen+Li%2C+Jiayi+Yuan%2C+Guanchu+Wang%2C+Kwei%2DHerng+Lai%2C+Leisheng+Yu%2C+Sirui+Ding%2C+Chia%2DYuan+Chang%2C+Qiaoyu+Tan%2C+Daochen+Zha%2C+et+al%2E+Understanding+different+design+choices+in+training+larg)
9. Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Zichang+Liu%2C+Aditya+Desai%2C+Fangshuo+Liao%2C+Weitao+Wang%2C+Victor+Xie%2C+Zhaozhuo+Xu%2C+Anastasios+Kyrillidis%2C+and+Anshumali+Shrivastava%2E+Scissorhands%3A+Exploiting+the+persistence+of+importance+hypothesis+for+)
10. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339, 2022. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Mike+Lewis%2C+Younes+Belkada%2C+and+Luke+Zettlemoyer%2E+Llm%2E+int8+%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2E+arXiv+preprint+arXiv%3A2208%2E07339%2C+2022%2E)
11. Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv:2305.16300, 2023. [find](https://scholar.google.com/scholar?q=Amirkeivan+Mohtashami+and+Martin+Jaggi%2E+Landmark+attention%3A+Random%2Daccess+infinite+context+length+for+transformers%2E+arXiv+preprint+arXiv%3A2305%2E16300%2C+2023%2E)
12. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022. [find](https://scholar.google.com/scholar?q=Elias+Frantar%2C+Saleh+Ashkboos%2C+Torsten+Hoefler%2C+and+Dan+Alistarh%2E+Gptq%3A+Accurate+post%2Dtraining+quantization+for+generative+pre%2Dtrained+transformers%2E+arXiv+preprint+arXiv%3A2210%2E17323%2C+2022%2E)
13. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116… [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Alessandro+Cappelli%2C+Hamza+Alobeidli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+The+refinedweb+dataset+for+falcon+)
14. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedin… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Reiner+Pope%2C+Sholto+Douglas%2C+Aakanksha+Chowdhery%2C+Jacob+Devlin%2C+James+Bradbury%2C+Jonathan+He)
15. Zhaozhuo Xu, Zirui Liu, Beidi Chen, Yuxin Tang, Jue Wang, Kaixiong Zhou, Xia Hu, and Anshumali Shrivastava. Compress, then prompt: Improving accuracy-efficiency trade-off of llm inference with transferable prompt. arXiv preprint arXiv:2305.11186, 2023. [find](https://scholar.google.com/scholar?q=Zhaozhuo+Xu%2C+Zirui+Liu%2C+Beidi+Chen%2C+Yuxin+Tang%2C+Jue+Wang%2C+Kaixiong+Zhou%2C+Xia+Hu%2C+and+Anshumali+Shrivastava%2E+Compress%2C+then+prompt%3A+Improving+accuracy%2Defficiency+trade%2Doff+of+llm+inference+with+transfe)
16. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530,… [find](https://scholar.google.com/scholar?q=Machel+Reid%2C+Nikolay+Savinov%2C+Denis+Teplyashin%2C+Dmitry+Lepikhin%2C+Timothy+Lillicrap%2C+Jean%2Dbaptiste+Alayrac%2C+Radu+Soricut%2C+Angeliki+Lazaridou%2C+Orhan+Firat%2C+Julian+Schrittwieser%2C+et+al%2E+Gemini+1%2E5%3A+Unloc) [our card](https://md.research.elamur.ai/free/en/papers/gemini-2024-team-gemini-1-5-unlocking-multimodal/)
17. Jiayi Yuan, Ruixiang Tang, Xiaoqian Jiang, and Xia Hu. Large language models for healthcare data augmentation: An example on patient-trial matching. In AMIA Annual Symposium Proceedings, volume 2023, page 1324. [find](https://scholar.google.com/scholar?q=Jiayi+Yuan%2C+Ruixiang+Tang%2C+Xiaoqian+Jiang%2C+and+Xia+Hu%2E+Large+language+models+for+healthcare+data+augmentation%3A+An+example+on+patient%2Dtrial+matching%2E+In+AMIA+Annual+Symposium+Proceedings%2C+volume+2023%2C+)
18. American Medical Informatics Association, 2023. [find](https://scholar.google.com/scholar?q=American+Medical+Informatics+Association%2C+2023%2E)
19. Jiayi Yuan, Hongyi Liu, Shaochen Zhong, Yu-Neng Chuang, Songchen Li, Guanchu Wang, Duy Le, Hongye Jin, Vipin Chaudhary, Zhaozhuo Xu, Zirui Liu, and Xia Hu. Kv cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches. arXiv preprint arXiv:2407.0… [find](https://scholar.google.com/scholar?q=Jiayi+Yuan%2C+Hongyi+Liu%2C+Shaochen+Zhong%2C+Yu%2DNeng+Chuang%2C+Songchen+Li%2C+Guanchu+Wang%2C+Duy+Le%2C+Hongye+Jin%2C+Vipin+Chaudhary%2C+Zhaozhuo+Xu%2C+Zirui+Liu%2C+and+Xia+Hu%2E+Kv+cache+compression%2C+but+what+must+we+give+)
20. Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+Fast+transformer+decoding%3A+One+write%2Dhead+is+all+you+need%2E+arXiv+preprint+arXiv%3A1911%2E02150%2C+2019%2E)
21. Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, pages 31094–31116. PMLR, 20… [find](https://scholar.google.com/scholar?q=Ying+Sheng%2C+Lianmin+Zheng%2C+Binhang+Yuan%2C+Zhuohan+Li%2C+Max+Ryabinin%2C+Beidi+Chen%2C+Percy+Liang%2C+Christopher+R%C3%A9%2C+Ion+Stoica%2C+and+Ce+Zhang%2E+Flexgen%3A+High%2Dthroughput+generative+inference+of+large+language+mo) [our card](https://md.research.elamur.ai/free/en/papers/sheng-2023-flexgen-high-throughput-generative-inference/)
22. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H _2 o: Heavy-hitter oracle for efficient generative inference of large language models. arXiv preprint arXiv:2306.14048, 2023. [find](https://scholar.google.com/scholar?q=Zhenyu+Zhang%2C+Ying+Sheng%2C+Tianyi+Zhou%2C+Tianlong+Chen%2C+Lianmin+Zheng%2C+Ruisi+Cai%2C+Zhao+Song%2C+Yuandong+Tian%2C+Christopher+R%C3%A9%2C+Clark+Barrett%2C+et+al%2E+H+%5F2+o%3A+Heavy%2Dhitter+oracle+for+efficient+generative+inf)
23. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085, 2022. [find](https://scholar.google.com/scholar?q=Ross+Taylor%2C+Marcin+Kardas%2C+Guillem+Cucurull%2C+Thomas+Scialom%2C+Anthony+Hartshorn%2C+Elvis+Saravia%2C+Andrew+Poulton%2C+Viktor+Kerkez%2C+and+Robert+Stojnic%2E+Galactica%3A+A+large+language+model+for+science%2E+arXiv+) [our card](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/)
24. Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate llm serving. Proceedings of Machine Learning and Systems, 6:196–209, 2024. [find](https://scholar.google.com/scholar?q=Yilong+Zhao%2C+Chien%2DYu+Lin%2C+Kan+Zhu%2C+Zihao+Ye%2C+Lequn+Chen%2C+Size+Zheng%2C+Luis+Ceze%2C+Arvind+Krishnamurthy%2C+Tianqi+Chen%2C+and+Baris+Kasikci%2E+Atom%3A+Low%2Dbit+quantization+for+efficient+and+accurate+llm+serving)
25. Yuandong Tian, Yiping Wang, Beidi Chen, and Simon Du. Scan and snap: Understanding training dynamics and token composition in 1-layer transformer. arXiv preprint arXiv:2305.16380, 2023. — [arXiv:2305.16380](https://arxiv.org/abs/2305.16380) [find](https://scholar.google.com/scholar?q=Yuandong+Tian%2C+Yiping+Wang%2C+Beidi+Chen%2C+and+Simon+Du%2E+Scan+and+snap%3A+Understanding+training+dynamics+and+token+composition+in+1%2Dlayer+transformer%2E+arXiv+preprint+arXiv%3A2305%2E16380%2C+2023%2E+%E2%80%94+%5BarXiv%3A2305%2E)
26. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Thibaut+Lavril%2C+Gautier+Izacard%2C+Xavier+Martinet%2C+Marie%2DAnne+Lachaux%2C+Timoth%C3%A9e+Lacroix%2C+Baptiste+Rozi%C3%A8re%2C+Naman+Goyal%2C+Eric+Hambro%2C+Faisal+Azhar%2C+et+al%2E+Llama%3A+Open+and+efficient+foundat)
27. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtun)
28. Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087–38099. PMLR, 2023a. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Ji+Lin%2C+Mickael+Seznec%2C+Hao+Wu%2C+Julien+Demouth%2C+and+Song+Han%2E+Smoothquant%3A+Accurate+and+efficient+post%2Dtraining+quantization+for+large+language+models%2E+In+International+Conference+on+M)
29. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023b. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+Efficient+streaming+language+models+with+attention+sinks%2E+arXiv+preprint+arXiv%3A2309%2E17453%2C+2023b%2E)