---
title: "Reducing Transformer Key-Value Cache Size with Cross-Layer"
description: "When large language models generate text autoregressively, they store keys and values (the KV… 49 references, cited by 42 papers here."
authors: ["William Brandon", "Mayank Mishra", "Aniruddha Nrusimha", "Rameswar Panda", "Jonathan Ragan Kelly"]
year: 2024
venue: "arXiv:2405.12981"
source: "https://arxiv.org/abs/2405.12981"
doi: "10.48550/arxiv.2405.12981"
arxiv: "2405.12981"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/brandon-2024-reducing-transformer-key-value-cache/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/brandon-2024-reducing-transformer-key-value-cache/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / Reducing Transformer Key-Value Cache Size with Cross-Layer Attention

# Reducing Transformer Key-Value Cache Size with Cross-Layer Attention

**Authors:** [William Brandon](https://md.research.elamur.ai/free/en/authors/william-brandon/), [Mayank Mishra](https://md.research.elamur.ai/free/en/authors/mayank-mishra/), [Aniruddha Nrusimha](https://md.research.elamur.ai/free/en/authors/aniruddha-nrusimha/), [Rameswar Panda](https://md.research.elamur.ai/free/en/authors/rameswar-panda/) and Jonathan Ragan Kelly · **Year:** 2024 · **Venue:** [arXiv:2405.12981](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2024](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2024/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2405.12981) · [PDF](https://arxiv.org/pdf/2405.12981) · [DOI 10.48550/arxiv.2405.12981](https://doi.org/10.48550/arxiv.2405.12981) · [arXiv 2405.12981](https://arxiv.org/abs/2405.12981) · [OpenAlex](https://openalex.org/W4415796681)

**Citations:** 15 citations (OpenAlex, 2026-10-14) · [42](https://md.research.elamur.ai/free/en/papers/brandon-2024-reducing-transformer-key-value-cache/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
When large language models generate text autoregressively, they store keys and values (the KV cache) for every token and layer. This memory grows with context length and batch size and often limits serving. Earlier fixes such as Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) shrink the cache by letting several query heads share one key/value head within a layer. Brandon et al. add a separate dimension of sharing, across depth. In their Cross-Layer Attention (CLA), only some layers compute key/value projections, and neighbouring layers reuse those activations. The sharing factor sets how many layers use each projection: CLA2 groups pairs, CLA3 groups triples. The authors pretrained 1B- and 3B-parameter models from scratch on SlimPajama, compared many CLA configurations against tuned MQA/GQA/MHA baselines and mapped the accuracy-versus-memory frontier. Combining CLA2 with MQA halves the KV cache compared with plain MQA, and perplexity barely changes. This is a Pareto improvement. It matters because the change is simple and low-risk, and it lets models serve longer contexts or larger batches on the same hardware.

## Key points
- **Method:** CLA can be combined with MHA, GQA or MQA. It shrinks the KV cache by the sharing factor, or slightly less when that factor does not divide the layer count evenly. It also slightly reduces parameter count and FLOPs.
- **Setup:** The 1B and 3B models were trained from scratch on SlimPajama with the GPT-NeoX BPE tokenizer. Configurations were labelled by head size and attention type, for example H128-MQA, H64-MQA-CLA2 and H128-GQA4.
- **Main result:** MQA-CLA2 gives about a 2× smaller KV cache than an MQA baseline with nearly equal validation perplexity. The benefit held at both 1B and 3B scale.
- **Configuration guidance:**
  - A sharing factor of 2 worked better than larger ones.
  - CLA was most reliably useful together with MQA.
  - Preliminary evidence suggests CLA models benefit from higher learning rates.
- **Limitations:**
  - CLA does not lower per-step attention memory bandwidth or core attention latency.
  - Under pipeline parallelism, layers that share a cache must sit in the same stage, or KV activations must be sent between stages.
  - The paper does not evaluate end-to-end inference efficiency; the authors leave it for future work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,621 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 328 works

## Cite

Cite as: William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda and Jonathan Ragan Kelly. 2024. Reducing Transformer Key-Value Cache Size with Cross-Layer Attention. arXiv:2405.12981.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{brandon2024reducing,
  title = {{Reducing Transformer Key-Value Cache Size with Cross-Layer Attention}},
  author = {William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda and Jonathan Ragan Kelly},
  year = {2024},
  doi = {10.48550/arxiv.2405.12981},
  eprint = {2405.12981},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.12981}
}
```

- [← previous Mixture-of-Depths: Dynamically allocating compute in transformer-based language models](https://md.research.elamur.ai/free/en/papers/raposo-2024-mixture-depths-dynamically-allocating-compute/)
- [next → RegMix: Data Mixture as Regression for Language Model Pre-training](https://md.research.elamur.ai/free/en/papers/liu-2024-regmix-data-mixture-regression-language/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

49 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebron, and S. Sanghai. [GQA: Training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/). In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,…
2. J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai. [Gqa: Training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/), 2023b.
3. J. Ansel, E. Yang, H. He, N. Gimelshein, A. Jain, M. Voznesensky, B. Bao, P. Bell, D. Berard, E. Burovski, G. Chauhan, A. Chourdia, W. Constable, A. Desmaison, Z. DeVito, E. Ellison, W. Feng, J. Gong, M. Gschwind, B. Hirsh, S. Huang, K. Kalambarkar, L. Kirsch, M. Lazos, M. Lezcano, Y. Liang, J. Lian… — [PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation](https://md.research.elamur.ai/free/en/papers/ansel-2024-pytorch-2-faster-machine-learning/)
4. S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, M. Pieler, U. S. Prashanth, S. Purohit, L. Reynolds, J. Tow, B. Wang, and S. Weinbach. [Gpt-neox-20b: An open-source autoregressive language model](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/), 2022.
5. S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, D. De Las Casas, A. Guy, J. Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. Irving, O. Vinyals, S. Osindero, K. S… [find](https://scholar.google.com/scholar?q=S%2E+Borgeaud%2C+A%2E+Mensch%2C+J%2E+Hoffmann%2C+T%2E+Cai%2C+E%2E+Rutherford%2C+K%2E+Millican%2C+G%2E+B%2E+Van+Den+Driessche%2C+J%2E%2DB%2E+Lespiau%2C+B%2E+Damoc%2C+A%2E+Clark%2C+D%2E+De+Las+Casas%2C+A%2E+Guy%2C+J%2E+Menick%2C+R%2E+Ring%2C+T%2E+Hennigan%2C+S%2E+Huang%2C)
6. R. Child, S. Gray, A. Radford, and I. Sutskever. [Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/). CoRR, abs/1904.10509, 2019. URL http://arxiv.org/abs/1904.10509. [↗](http://arxiv.org/abs/1904.10509)
7. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
8. Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. Le, and R. Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. In A. Korhonen, D. Traum, and L. Màrquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, F… [find](https://scholar.google.com/scholar?q=Z%2E+Dai%2C+Z%2E+Yang%2C+Y%2E+Yang%2C+J%2E+Carbonell%2C+Q%2E+Le%2C+and+R%2E+Salakhutdinov%2E+Transformer%2DXL%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+In+A%2E+Korhonen%2C+D%2E+Traum%2C+and+L%2E+M%C3%A0rquez%2C+editors%2C+Proceed)
9. T. Dao. [Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/), 2023.
10. T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=T%2E+Dao%2C+D%2E+Y%2E+Fu%2C+S%2E+Ermon%2C+A%2E+Rudra%2C+and+C%2E+R%C3%A9%2E+FlashAttention%3A+Fast+and+memory%2Defficient+exact+attention+with+IO%2Dawareness%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2022%2E)
11. DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024. [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2E+Deepseek%2Dv2%3A+A+strong%2C+economical%2C+and+efficient+mixture%2Dof%2Dexperts+language+model%2C+2024%2E)
12. B. Gao, Z. He, P. Sharma, Q. Kang, D. Jevdjic, J. Deng, X. Yang, Z. Yu, and P. Zuo. Attentionstore: Cost-effective attention reuse across multi-turn conversations in large language model serving, 2024. [find](https://scholar.google.com/scholar?q=B%2E+Gao%2C+Z%2E+He%2C+P%2E+Sharma%2C+Q%2E+Kang%2C+D%2E+Jevdjic%2C+J%2E+Deng%2C+X%2E+Yang%2C+Z%2E+Yu%2C+and+P%2E+Zuo%2E+Attentionstore%3A+Cost%2Deffective+attention+reuse+across+multi%2Dturn+conversations+in+large+language+model+serving%2C+2024)
13. L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou. A framework for few-shot language m… [find](https://scholar.google.com/scholar?q=L%2E+Gao%2C+J%2E+Tow%2C+B%2E+Abbasi%2C+S%2E+Biderman%2C+S%2E+Black%2C+A%2E+DiPofi%2C+C%2E+Foster%2C+L%2E+Golding%2C+J%2E+Hsu%2C+A%2E+Le+Noac%E2%80%99h%2C+H%2E+Li%2C+K%2E+McDonell%2C+N%2E+Muennighoff%2C+C%2E+Ociepa%2C+J%2E+Phang%2C+L%2E+Reynolds%2C+H%2E+Schoelkopf%2C+A%2E+Skowro)
14. S. Ge, Y. Zhang, L. Liu, M. Zhang, J. Han, and J. Gao. [Model tells you what to discard: Adaptive kv cache compression for llms](https://md.research.elamur.ai/free/en/papers/ge-2023-model-tells-discard-adaptive-kv/), 2024.
15. Google. Context caching guide. [https://ai.google.dev/gemini-api/docs/caching](https://ai.google.dev/gemini-api/docs/caching), 2024. Accessed: 2024-05-20. [↗](https://ai.google.dev/gemini-api/docs/caching)
16. A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Gu+and+T%2E+Dao%2E+Mamba%3A+Linear%2Dtime+sequence+modeling+with+selective+state+spaces%2C+2023%2E)
17. K. Guu, K. Lee, Z. Tung, P. Pasupat, and M.-W. Chang. [Realm: retrieval-augmented language model pre-training](https://md.research.elamur.ai/free/en/papers/guu-2020-realm-retrieval-augmented-language-model/). In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020.
18. C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y. S. Shao, K. Keutzer, and A. Gholami. [Kvquant: Towards 10 million context length llm inference with kv cache quantization](https://md.research.elamur.ai/free/en/papers/hooper-2024-kvquant-towards-10-million-context/), 2024.
19. Y. Huang, Y. Cheng, A. Bapna, O. Firat, M. X. Chen, D. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu, and Z. Chen. GPipe: efficient training of giant neural networks using pipeline parallelism. Curran Associates Inc., Red Hook, NY, USA, 2019. [find](https://scholar.google.com/scholar?q=Y%2E+Huang%2C+Y%2E+Cheng%2C+A%2E+Bapna%2C+O%2E+Firat%2C+M%2E+X%2E+Chen%2C+D%2E+Chen%2C+H%2E+Lee%2C+J%2E+Ngiam%2C+Q%2E+V%2E+Le%2C+Y%2E+Wu%2C+and+Z%2E+Chen%2E+GPipe%3A+efficient+training+of+giant+neural+networks+using+pipeline+parallelism%2E+Curran+Assoc)
20. G. Izacard, P. Lewis, M. Lomeli, L. Hosseini, F. Petroni, T. Schick, J. Dwivedi-Yu, A. Joulin, S. Riedel, and E. Grave. [Atlas: few-shot learning with retrieval augmented language models](https://md.research.elamur.ai/free/en/papers/izacard-2022-atlas-few-shot-learning-retrieval/). J. Mach. Learn. Res., 24(1), mar 2024. ISSN 1532-4435.
21. D. Kalamkar, D. Mudigere, N. Mellempudi, D. Das, K. Banerjee, S. Avancha, D. T. Vooturi, N. Jammalamadaka, J. Huang, H. Yuen, J. Yang, J. Park, A. Heinecke, E. Georganas, S. Srinivasan, A. Kundu, M. Smelyanskiy, B. Kaul, and P. Dubey. A study of bfloat16 for deep learning training, 2019. [find](https://scholar.google.com/scholar?q=D%2E+Kalamkar%2C+D%2E+Mudigere%2C+N%2E+Mellempudi%2C+D%2E+Das%2C+K%2E+Banerjee%2C+S%2E+Avancha%2C+D%2E+T%2E+Vooturi%2C+N%2E+Jammalamadaka%2C+J%2E+Huang%2C+H%2E+Yuen%2C+J%2E+Yang%2C+J%2E+Park%2C+A%2E+Heinecke%2C+E%2E+Georganas%2C+S%2E+Srinivasan%2C+A%2E+Kundu%2C+M%2E+S)
22. A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are rnns: fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020. [find](https://scholar.google.com/scholar?q=A%2E+Katharopoulos%2C+A%2E+Vyas%2C+N%2E+Pappas%2C+and+F%2E+Fleuret%2E+Transformers+are+rnns%3A+fast+autoregressive+transformers+with+linear+attention%2E+In+Proceedings+of+the+37th+International+Conference+on+Machine+Lear)
23. Y. Liu, H. Li, Y. Cheng, S. Ray, Y. Huang, Q. Zhang, K. Du, J. Yao, S. Lu, G. Ananthanarayanan, M. Maire, H. Hoffmann, A. Holtzman, and J. Jiang. Cachegen: Kv cache compression and streaming for fast language model serving, 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Liu%2C+H%2E+Li%2C+Y%2E+Cheng%2C+S%2E+Ray%2C+Y%2E+Huang%2C+Q%2E+Zhang%2C+K%2E+Du%2C+J%2E+Yao%2C+S%2E+Lu%2C+G%2E+Ananthanarayanan%2C+M%2E+Maire%2C+H%2E+Hoffmann%2C+A%2E+Holtzman%2C+and+J%2E+Jiang%2E+Cachegen%3A+Kv+cache+compression+and+streaming+for+fast+)
24. Z. Liu, A. Desai, F. Liao, W. Wang, V. Xie, Z. Xu, A. Kyrillidis, and A. Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time, 2023. [find](https://scholar.google.com/scholar?q=Z%2E+Liu%2C+A%2E+Desai%2C+F%2E+Liao%2C+W%2E+Wang%2C+V%2E+Xie%2C+Z%2E+Xu%2C+A%2E+Kyrillidis%2C+and+A%2E+Shrivastava%2E+Scissorhands%3A+Exploiting+the+persistence+of+importance+hypothesis+for+llm+kv+cache+compression+at+test+time%2C+2023%2E)
25. I. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017. URL [https://openreview.net/forum?id=](https://openreview.net/forum?id=) Skq89Scxx. [↗](https://openreview.net/forum?id=)
26. I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL [https://openreview.net/forum?id=Bkg6RiCqY7](https://openreview.net/forum?id=Bkg6RiCqY7). [↗](https://openreview.net/forum?id=Bkg6RiCqY7)
27. S. Merity, C. Xiong, J. Bradbury, and R. Socher. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/), 2016.
28. P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu. Mixed precision training, 2018. [find](https://scholar.google.com/scholar?q=P%2E+Micikevicius%2C+S%2E+Narang%2C+J%2E+Alben%2C+G%2E+Diamos%2C+E%2E+Elsen%2C+D%2E+Garcia%2C+B%2E+Ginsburg%2C+M%2E+Houston%2C+O%2E+Kuchaiev%2C+G%2E+Venkatesh%2C+and+H%2E+Wu%2E+Mixed+precision+training%2C+2018%2E)
29. A. Mohtashami and M. Jaggi. Random-access infinite context length for transformers. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 54567–54585. Curran Associates, Inc., 2023. URL https://proceedings… [find](https://scholar.google.com/scholar?q=A%2E+Mohtashami+and+M%2E+Jaggi%2E+Random%2Daccess+infinite+context+length+for+transformers%2E+In+A%2E+Oh%2C+T%2E+Naumann%2C+A%2E+Globerson%2C+K%2E+Saenko%2C+M%2E+Hardt%2C+and+S%2E+Levine%2C+editors%2C+Advances+in+Neural+Information+Proc)
30. T. Munkhdalai, M. Faruqui, and S. Gopal. [Leave no context behind: Efficient infinite context transformers with infini-attention](https://md.research.elamur.ai/free/en/papers/munkhdalai-2024-leave-no-context-behind-efficient/), 2024.
31. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning… — [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
32. B. Peng, D. Goldstein, Q. Anthony, A. Albalak, E. Alcaide, S. Biderman, E. Cheah, X. Du, T. Ferdinan, H. Hou, P. Kazienko, K. K. GV, J. Kocoń, B. Koptyra, S. Krishna, R. M. J. au2, N. Muennighoff, F. Obeid, A. Saito, G. Song, H. Tu, S. Woźniak, R. Zhang, B. Zhao, Q. Zhao, P. Zhou, J. Zhu, and R.-J… [find](https://scholar.google.com/scholar?q=B%2E+Peng%2C+D%2E+Goldstein%2C+Q%2E+Anthony%2C+A%2E+Albalak%2C+E%2E+Alcaide%2C+S%2E+Biderman%2C+E%2E+Cheah%2C+X%2E+Du%2C+T%2E+Ferdinan%2C+H%2E+Hou%2C+P%2E+Kazienko%2C+K%2E+K%2E+GV%2C+J%2E+Kocon%CC%81%2C+B%2E+Koptyra%2C+S%2E+Krishna%2C+R%2E+M%2E+J%2E+au2%2C+N%2E+Muennighoff%2C+F%2E)
33. P. Ramachandran, B. Zoph, and Q. V. Le. Searching for activation functions, 2017. [find](https://scholar.google.com/scholar?q=P%2E+Ramachandran%2C+B%2E+Zoph%2C+and+Q%2E+V%2E+Le%2E+Searching+for+activation+functions%2C+2017%2E)
34. N. Shazeer. Fast transformer decoding: One write-head is all you need, 2019. [find](https://scholar.google.com/scholar?q=N%2E+Shazeer%2E+Fast+transformer+decoding%3A+One+write%2Dhead+is+all+you+need%2C+2019%2E)
35. N. Shazeer. Glu variants improve transformer, 2020. [find](https://scholar.google.com/scholar?q=N%2E+Shazeer%2E+Glu+variants+improve+transformer%2C+2020%2E)
36. Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Ré, I. Stoica, and C. Zhang. [Flexgen: high-throughput generative inference of large language models with a single gpu](https://md.research.elamur.ai/free/en/papers/sheng-2023-flexgen-high-throughput-generative-inference/). In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023.
37. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020. [find](https://scholar.google.com/scholar?q=M%2E+Shoeybi%2C+M%2E+Patwary%2C+R%2E+Puri%2C+P%2E+LeGresley%2C+J%2E+Casper%2C+and+B%2E+Catanzaro%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2C+2020%2E)
38. D. Soboleva, F. Al-Khateeb, R. Myers, J. R. Steeves, J. Hestness, and N. Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. [https://www.cerebras.net/blog/](https://www.cerebras.net/blog/) slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama, 2023. URL https://huggingface.co/datasets/cerebr… [↗](https://www.cerebras.net/blog/)
39. J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/), 2023.
40. Y. Sun, L. Dong, Y. Zhu, S. Huang, W. Wang, S. Ma, Q. Zhang, J. Wang, and F. Wei. You only cache once: Decoder-decoder architectures for language models, 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Sun%2C+L%2E+Dong%2C+Y%2E+Zhu%2C+S%2E+Huang%2C+W%2E+Wang%2C+S%2E+Ma%2C+Q%2E+Zhang%2C+J%2E+Wang%2C+and+F%2E+Wei%2E+You+only+cache+once%3A+Decoder%2Ddecoder+architectures+for+language+models%2C+2024%2E)
41. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
42. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL [https://proceedings.neurips.cc/paper_](https://proceedings.neurips.cc/paper_) files/paper/2017/file/3f5ee243… [↗](https://proceedings.neurips.cc/paper_)
43. C. Wang, K. Cho, and J. Gu. [Neural machine translation with byte-level subwords](https://md.research.elamur.ai/free/en/papers/wang-2019-neural-machine-translation-byte-level/), 2019.
44. S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma. Linformer: Self-attention with linear complexity. CoRR, abs/2006.04768, 2020. URL https://arxiv.org/abs/2006.04768. [↗](https://arxiv.org/abs/2006.04768)
45. Y. Wu, M. N. Rabe, D. Hutchins, and C. Szegedy. Memorizing transformers. In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?id=](https://openreview.net/forum?id=) TrjbxzRcnf-. [↗](https://openreview.net/forum?id=)
46. S. Yang, B. Wang, Y. Shen, R. Panda, and Y. Kim. Gated linear attention transformers with hardware-efficient training, 2024. [find](https://scholar.google.com/scholar?q=S%2E+Yang%2C+B%2E+Wang%2C+Y%2E+Shen%2C+R%2E+Panda%2C+and+Y%2E+Kim%2E+Gated+linear+attention+transformers+with+hardware%2Defficient+training%2C+2024%2E)
47. T. Zhang, J. Yi, Z. Xu, and A. Shrivastava. Kv cache is 1 bit per channel: Efficient large language model inference with coupled quantization, 2024. [find](https://scholar.google.com/scholar?q=T%2E+Zhang%2C+J%2E+Yi%2C+Z%2E+Xu%2C+and+A%2E+Shrivastava%2E+Kv+cache+is+1+bit+per+channel%3A+Efficient+large+language+model+inference+with+coupled+quantization%2C+2024%2E)
48. Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, Z. A. Wang, and B. Chen. H2o: Heavy-hitter oracle for efficient generative inference of large language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neu… [find](https://scholar.google.com/scholar?q=Z%2E+Zhang%2C+Y%2E+Sheng%2C+T%2E+Zhou%2C+T%2E+Chen%2C+L%2E+Zheng%2C+R%2E+Cai%2C+Z%2E+Song%2C+Y%2E+Tian%2C+C%2E+R%C3%A9%2C+C%2E+Barrett%2C+Z%2E+A%2E+Wang%2C+and+B%2E+Chen%2E+H2o%3A+Heavy%2Dhitter+oracle+for+efficient+generative+inference+of+large+language+mode)
49. X. Zhu, J. Li, Y. Liu, C. Ma, and W. Wang. [A survey on model compression for large language models](https://md.research.elamur.ai/free/en/papers/zhu-2024-survey-model-compression-large-language/), 2023.

*Updated 3 November 2026*