---
title: "A Survey on Efficient Inference for Large Language Models (2024)"
description: "The work offers practitioners a single map of the field and evidence for choosing between techniques. 313 references, cited by 88 papers here."
authors: ["Zixuan Zhou", "Xuefei Ning", "Ke Hong", "Tianyu Fu", "Jiaming Xu", "Shiyao Li", "Yuming Lou", "Luning Wang", "Zhihang Yuan", "Xiuhong Li", "Shengen Yan", "Guohao Dai", "Xiao-Ping Zhang", "Yuhan Dong", "Yu Wang"]
year: 2024
venue: "arXiv:2404.14294"
source: "https://arxiv.org/abs/2404.14294"
doi: "10.48550/arxiv.2404.14294"
arxiv: "2404.14294"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhou-2024-survey-efficient-inference-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhou-2024-survey-efficient-inference-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / A Survey on Efficient Inference for Large Language Models

# A Survey on Efficient Inference for Large Language Models

**Authors:** [Zixuan Zhou](https://md.research.elamur.ai/free/en/authors/zixuan-zhou/), [Xuefei Ning](https://md.research.elamur.ai/free/en/authors/xuefei-ning/), Ke Hong, Tianyu Fu, [Jiaming Xu](https://md.research.elamur.ai/free/en/authors/jiaming-xu/), [Shiyao Li](https://md.research.elamur.ai/free/en/authors/shiyao-li/), Yuming Lou, Luning Wang, [Zhihang Yuan](https://md.research.elamur.ai/free/en/authors/zhihang-yuan/), Xiuhong Li, [Shengen Yan](https://md.research.elamur.ai/free/en/authors/shengen-yan/), [Guohao Dai](https://md.research.elamur.ai/free/en/authors/guohao-dai/), Xiao-Ping Zhang, Yuhan Dong and [Yu Wang](https://md.research.elamur.ai/free/en/authors/yu-wang/) · **Year:** 2024 · **Venue:** arXiv:2404.14294 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2404.14294) · [PDF](https://arxiv.org/pdf/2404.14294) · [DOI 10.48550/arxiv.2404.14294](https://doi.org/10.48550/arxiv.2404.14294) · [arXiv 2404.14294](https://arxiv.org/abs/2404.14294) · [OpenAlex](https://openalex.org/W4395065783)

**Citations:** 25 citations (OpenAlex, 2026-10-14) · [88](https://md.research.elamur.ai/free/en/papers/zhou-2024-survey-efficient-inference-large-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Running large language models is expensive. They need a lot of computation, memory traffic and storage, which raises latency, lowers throughput and makes deployment difficult on edge devices and costly in the cloud. This survey, by authors from Tsinghua University, Shanghai Jiaotong University and Infinigence-AI, reviews research on making LLM inference cheaper. It first identifies three root causes of inefficiency: very large parameter counts, attention whose cost grows quadratically with sequence length, and token-by-token auto-regressive generation. It then sorts existing techniques into three levels of optimization: data, model and system. Other surveys typically cover only model compression or a systems view. This one spans all three levels and adds its own comparative experiments on representative methods in key sub-fields such as quantization and serving systems, from which it draws practical recommendations. It also discusses specific application scenarios and outlines open research directions. The work offers practitioners a single map of the field and evidence for choosing between techniques.

## Key points
- The taxonomy has three tiers: data-level, model-level and system-level optimization. It is contrasted in Table 1 with earlier surveys, including Zhu et al., Wan et al. and Xu et al.
- Inference is modeled as two phases. In the prefilling stage, the KV cache for the prompt is built and the first token is produced. In the decoding stage, tokens are generated one at a time while the KV cache is extended.
- The survey defines a set of efficiency metrics:
  - first token latency, per-output token latency and generation latency;
  - model size, KV cache size and peak memory;
  - token throughput and request throughput.
- A motivating example is LLaMA-2-70B. In FP16 its weights take about 140 GB of VRAM, which requires at least 6 RTX 3090Ti or 2 NVIDIA A100 (80 GB) GPUs. Each token takes roughly 100 ms on two A100s, so outputs of a few hundred tokens take more than 10 seconds.
- Comparative experiments cover representative methods in sub-fields such as model quantization and LLM serving systems.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations
- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,267 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 297 works

## Cite

Cite as: Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong and Yu Wang. 2024. A Survey on Efficient Inference for Large Language Models. arXiv:2404.14294.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhou2024survey,
  title = {{A Survey on Efficient Inference for Large Language Models}},
  author = {Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong and Yu Wang},
  year = {2024},
  doi = {10.48550/arxiv.2404.14294},
  eprint = {2404.14294},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.14294}
}
```

- [← previous V-STaR: Training Verifiers for Self-Taught Reasoners](https://md.research.elamur.ai/free/en/papers/hosseini-2024-v-star-training-verifiers-self/)
- [next → A Survey on Hallucination in Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/liu-2024-survey-hallucination-large-vision-language/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

313 references, 110 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Radford, K. Narasimhan, T. Salimans, I. Sutskever et al., “Improving language understanding by generative pre-training,” 2018. [find](https://scholar.google.com/scholar?q=A%2E+Radford%2C+K%2E+Narasimhan%2C+T%2E+Salimans%2C+I%2E+Sutskever+et+al%2E%2C+%E2%80%9CImproving+language+understanding+by+generative+pre%2Dtraining%2C%E2%80%9D+2018%2E)
2. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019. [find](https://scholar.google.com/scholar?q=A%2E+Radford%2C+J%2E+Wu%2C+R%2E+Child%2C+D%2E+Luan%2C+D%2E+Amodei%2C+I%2E+Sutskever+et+al%2E%2C+%E2%80%9CLanguage+models+are+unsupervised+multitask+learners%2C%E2%80%9D+OpenAI+blog%2C+vol%2E+1%2C+no%2E+8%2C+p%2E+9%2C+2019%2E)
3. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “[Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/),” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
4. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin et al., “[Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/),” arXiv preprint arXiv:2205.01068, 2022.
5. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., “[Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/),” arXiv preprint arXiv:2302.13971, 2023.
6. A. Yang, B. Xiao, B. Wang, B. Zhang, C. Bian, C. Yin, C. Lv, D. Pan, D. Wang, D. Yan et al., “[Baichuan 2: Open large-scale language models](https://md.research.elamur.ai/free/en/papers/yang-2023-baichuan-2-open-large-scale/),” arXiv preprint arXiv:2309.10305, 2023.
7. W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez et al., “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” See [https://vicuna.lmsys](https://vicuna.lmsys). org (accessed 14 April 2023), 2023. [↗](https://vicuna.lmsys)
8. D. Li, R. Shao, A. Xie, Y. Sheng, L. Zheng, J. Gonzalez, I. Stoica, X. Ma, and H. Zhang, “How long can context length of open-source llms truly promise?” in NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023. [find](https://scholar.google.com/scholar?q=D%2E+Li%2C+R%2E+Shao%2C+A%2E+Xie%2C+Y%2E+Sheng%2C+L%2E+Zheng%2C+J%2E+Gonzalez%2C+I%2E+Stoica%2C+X%2E+Ma%2C+and+H%2E+Zhang%2C+%E2%80%9CHow+long+can+context+length+of+open%2Dsource+llms+truly+promise%3F%E2%80%9D+in+NeurIPS+2023+Workshop+on+Instruction+Tuning)
9. B. Workshop, T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon et al., “Bloom: A 176b-parameter open-access multilingual language model,” arXiv preprint arXiv:2211.05100, 2022. [find](https://scholar.google.com/scholar?q=B%2E+Workshop%2C+T%2E+L%2E+Scao%2C+A%2E+Fan%2C+C%2E+Akiki%2C+E%2E+Pavlick%2C+S%2E+Ilic%CC%81%2C+D%2E+Hesslow%2C+R%2E+Castagne%CC%81%2C+A%2E+S%2E+Luccioni%2C+F%2E+Yvon+et+al%2E%2C+%E2%80%9CBloom%3A+A+176b%2Dparameter+open%2Daccess+multilingual+language+model%2C%E2%80%9D+arXiv+prep)
10. E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, É. Goffinet, D. Hesslow, J. Launay, Q. Malartic et al., “[The falcon series of open language models](https://md.research.elamur.ai/free/en/papers/almazrouei-2023-falcon-series-open-language-models/),” arXiv preprint arXiv:2311.16867, 2023.
11. Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang, “[Glm: General language model pretraining with autoregressive blank infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/),” arXiv preprint arXiv:2103.10360, 2021.
12. A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand et al., “Mixtral of experts,” arXiv preprint arXiv:2401.04088, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Q%2E+Jiang%2C+A%2E+Sablayrolles%2C+A%2E+Roux%2C+A%2E+Mensch%2C+B%2E+Savary%2C+C%2E+Bamford%2C+D%2E+S%2E+Chaplot%2C+D%2E+d%2E+l%2E+Casas%2C+E%2E+B%2E+Hanna%2C+F%2E+Bressand+et+al%2E%2C+%E2%80%9CMixtral+of+experts%2C%E2%80%9D+arXiv+preprint+arXiv%3A2401%2E04088%2C+2024%2E)
13. J. Yang, H. Jin, R. Tang, X. Han, Q. Feng, H. Jiang, S. Zhong, B. Yin, and X. Hu, “[Harnessing the power of llms in practice: A survey on chatgpt and beyond](https://md.research.elamur.ai/free/en/papers/yang-2023-harnessing-power-llms-practice-survey/),” ACM Transactions on Knowledge Discovery from Data, 2023.
14. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al., “[Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/),” Advances in Neural Information Processing Systems, vol. 35, pp. 24 824–24 837, 2022.
15. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., “[Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/),” arXiv preprint arXiv:2107.03374, 2021.
16. S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg et al., “[Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/),” arXiv preprint arXiv:2303.12712, 2023.
17. X. Zhu, J. Li, Y. Liu, C. Ma, and W. Wang, “[A survey on model compression for large language models](https://md.research.elamur.ai/free/en/papers/zhu-2024-survey-model-compression-large-language/),” arXiv preprint arXiv:2308.07633, 2023.
18. S. Park, J. Choi, S. Lee, and U. Kang, “A comprehensive survey of compression algorithms for language models,” arXiv preprint arXiv:2401.15347, 2024. — [arXiv:2401.15347](https://arxiv.org/abs/2401.15347) [find](https://scholar.google.com/scholar?q=S%2E+Park%2C+J%2E+Choi%2C+S%2E+Lee%2C+and+U%2E+Kang%2C+%E2%80%9CA+comprehensive+survey+of+compression+algorithms+for+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2401%2E15347%2C+2024%2E+%E2%80%94+%5BarXiv%3A2401%2E15347%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2401%2E15)
19. W. Wang, W. Chen, Y. Luo, Y. Long, Z. Lin, L. Zhang, B. Lin, D. Cai, and X. He, “Model compression and efficient inference for [large language models: A survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/),” arXiv preprint arXiv:2402.09748, 2024. — [arXiv:2402.09748](https://arxiv.org/abs/2402.09748)
20. Y. Tang, Y. Wang, J. Guo, Z. Tu, K. Han, H. Hu, and D. Tao, “A survey on transformer compression,” arXiv preprint arXiv:2402.05964, 2024. — [arXiv:2402.05964](https://arxiv.org/abs/2402.05964) [find](https://scholar.google.com/scholar?q=Y%2E+Tang%2C+Y%2E+Wang%2C+J%2E+Guo%2C+Z%2E+Tu%2C+K%2E+Han%2C+H%2E+Hu%2C+and+D%2E+Tao%2C+%E2%80%9CA+survey+on+transformer+compression%2C%E2%80%9D+arXiv+preprint+arXiv%3A2402%2E05964%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E05964%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E05964%29)
21. T. Ding, T. Chen, H. Zhu, J. Jiang, Y. Zhong, J. Zhou, G. Wang, Z. Zhu, I. Zharkov, and L. Liang, “The efficiency spectrum of large language models: An algorithmic survey,” arXiv preprint arXiv:2312.00678, 2023. — [arXiv:2312.00678](https://arxiv.org/abs/2312.00678) [find](https://scholar.google.com/scholar?q=T%2E+Ding%2C+T%2E+Chen%2C+H%2E+Zhu%2C+J%2E+Jiang%2C+Y%2E+Zhong%2C+J%2E+Zhou%2C+G%2E+Wang%2C+Z%2E+Zhu%2C+I%2E+Zharkov%2C+and+L%2E+Liang%2C+%E2%80%9CThe+efficiency+spectrum+of+large+language+models%3A+An+algorithmic+survey%2C%E2%80%9D+arXiv+preprint+arXiv%3A2312%2E0)
22. X. Miao, G. Oliaro, Z. Zhang, X. Cheng, H. Jin, T. Chen, and Z. Jia, “[Towards efficient generative large language model serving: A survey from algorithms to systems](https://md.research.elamur.ai/free/en/papers/miao-2023-towards-efficient-generative-large-language/),” arXiv preprint arXiv:2312.15234, 2023.
23. Z. Wan, X. Wang, C. Liu, S. Alam, Y. Zheng, Z. Qu, S. Yan, Y. Zhu, Q. Zhang, M. Chowdhury et al., “[Efficient large language models: A survey](https://md.research.elamur.ai/free/en/papers/wan-2023-efficient-large-language-models-survey/),” arXiv preprint arXiv:2312.03863, vol. 1, 2023.
24. M. Xu, W. Yin, D. Cai, R. Yi, D. Xu, Q. Wang, B. Wu, Y. Zhao, C. Yang, S. Wang et al., “A survey of resource-efficient llm and multimodal foundation models,” arXiv preprint arXiv:2401.08092, 2024. [find](https://scholar.google.com/scholar?q=M%2E+Xu%2C+W%2E+Yin%2C+D%2E+Cai%2C+R%2E+Yi%2C+D%2E+Xu%2C+Q%2E+Wang%2C+B%2E+Wu%2C+Y%2E+Zhao%2C+C%2E+Yang%2C+S%2E+Wang+et+al%2E%2C+%E2%80%9CA+survey+of+resource%2Defficient+llm+and+multimodal+foundation+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2401%2E08092%2C+2024%2E)
25. W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong et al., “[A survey of large language models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/),” arXiv preprint arXiv:2303.18223, 2023.
26. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “[Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/),” Advances in neural information processing systems, vol. 30, 2017.
27. Z. Yuan, Y. Shang, Y. Zhou, Z. Dong, C. Xue, B. Wu, Z. Li, Q. Gu, Y. J. Lee, Y. Yan et al., “[Llm inference unveiled: Survey and roofline model insights](https://md.research.elamur.ai/free/en/papers/yuan-2024-llm-inference-unveiled-survey-roofline/),” arXiv preprint arXiv:2402.16363, 2024.
28. A. Golden, S. Hsia, F. Sun, B. Acun, B. Hosmer, Y. Lee, Z. DeVito, J. Johnson, G.-Y. Wei, D. Brooks et al., “Is flash attention stable?” arXiv preprint arXiv:2405.02803, 2024. — [arXiv:2405.02803](https://arxiv.org/abs/2405.02803) [find](https://scholar.google.com/scholar?q=A%2E+Golden%2C+S%2E+Hsia%2C+F%2E+Sun%2C+B%2E+Acun%2C+B%2E+Hosmer%2C+Y%2E+Lee%2C+Z%2E+DeVito%2C+J%2E+Johnson%2C+G%2E%2DY%2E+Wei%2C+D%2E+Brooks+et+al%2E%2C+%E2%80%9CIs+flash+attention+stable%3F%E2%80%9D+arXiv+preprint+arXiv%3A2405%2E02803%2C+2024%2E+%E2%80%94+%5BarXiv%3A2405%2E02803%5D%28htt)
29. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel et al., “[Retrieval-augmented generation for knowledge-intensive nlp tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/),” Advances in Neural Information Processing Systems, vol. 33, pp. 9459– 9474, 2020.
30. A. Chevalier, A. Wettig, A. Ajith, and D. Chen, “[Adapting language models to compress contexts](https://md.research.elamur.ai/free/en/papers/chevalier-2023-adapting-language-models-compress-contexts/),” arXiv preprint arXiv:2305.14788, 2023.
31. W. Shi, S. Min, M. Yasunaga, M. Seo, R. James, M. Lewis, L. Zettlemoyer, and W. tau Yih, “[Replug: Retrieval-augmented black-box language models](https://md.research.elamur.ai/free/en/papers/shi-2023-replug-retrieval-augmented-black-box/),” 2023.
32. A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi, “[Self-rag: Learning to retrieve, generate, and critique through self-reflection](https://md.research.elamur.ai/free/en/papers/asai-2023-self-rag-learning-retrieve-generate/),” 2023.
33. D. Wingate, M. Shoeybi, and T. Sorensen, “Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models,” arXiv preprint arXiv:2210.03162, 2022. [find](https://scholar.google.com/scholar?q=D%2E+Wingate%2C+M%2E+Shoeybi%2C+and+T%2E+Sorensen%2C+%E2%80%9CPrompt+compression+and+contrastive+conditioning+for+controllability+and+toxicity+reduction+in+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2210%2E03162%2C+2022%2E)
34. J. Mu, X. L. Li, and N. Goodman, “[Learning to compress prompts with gist tokens](https://md.research.elamur.ai/free/en/papers/mu-2023-learning-compress-prompts-gist-tokens/),” arXiv preprint arXiv:2304.08467, 2023.
35. T. Ge, J. Hu, X. Wang, S.-Q. Chen, and F. Wei, “[In-context autoencoder for context compression in a large language model](https://md.research.elamur.ai/free/en/papers/ge-2023-context-autoencoder-context-compression-large/),” arXiv preprint arXiv:2307.06945, 2023.
36. F. Xu, W. Shi, and E. Choi, “[Recomp: Improving retrieval-augmented lms with compression and selective augmentation](https://md.research.elamur.ai/free/en/papers/xu-2023-recomp-improving-retrieval-augmented-lms/),” arXiv preprint arXiv:2310.04408, 2023.
37. W. Fei, X. Niu, P. Zhou, L. Hou, B. Bai, L. Deng, and W. Han, “Extending context window of large language models via semantic compression,” arXiv preprint arXiv:2312.09571, 2023. [find](https://scholar.google.com/scholar?q=W%2E+Fei%2C+X%2E+Niu%2C+P%2E+Zhou%2C+L%2E+Hou%2C+B%2E+Bai%2C+L%2E+Deng%2C+and+W%2E+Han%2C+%E2%80%9CExtending+context+window+of+large+language+models+via+semantic+compression%2C%E2%80%9D+arXiv+preprint+arXiv%3A2312%2E09571%2C+2023%2E)
38. W. Zhou, Y. E. Jiang, R. Cotterell, and M. Sachan, “Efficient prompting via dynamic in-context learning,” arXiv preprint arXiv:2305.11170, 2023. — [arXiv:2305.11170](https://arxiv.org/abs/2305.11170) [find](https://scholar.google.com/scholar?q=W%2E+Zhou%2C+Y%2E+E%2E+Jiang%2C+R%2E+Cotterell%2C+and+M%2E+Sachan%2C+%E2%80%9CEfficient+prompting+via+dynamic+in%2Dcontext+learning%2C%E2%80%9D+arXiv+preprint+arXiv%3A2305%2E11170%2C+2023%2E+%E2%80%94+%5BarXiv%3A2305%2E11170%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E11170%29)
39. Y. Li, B. Dong, F. Guerin, and C. Lin, “[Compressing context to enhance inference efficiency of large language models](https://md.research.elamur.ai/free/en/papers/li-2023-compressing-context-enhance-inference-efficiency/),” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 6342–6353.
40. F. Yin, J. Vig, P. Laban, S. Joty, C. Xiong, and C.-S. J. Wu, “Did you read the instructions? rethinking the effectiveness of task definitions in instruction learning,” arXiv preprint arXiv:2306.01150, 2023. [find](https://scholar.google.com/scholar?q=F%2E+Yin%2C+J%2E+Vig%2C+P%2E+Laban%2C+S%2E+Joty%2C+C%2E+Xiong%2C+and+C%2E%2DS%2E+J%2E+Wu%2C+%E2%80%9CDid+you+read+the+instructions%3F+rethinking+the+effectiveness+of+task+definitions+in+instruction+learning%2C%E2%80%9D+arXiv+preprint+arXiv%3A2306%2E01150)
41. H. Jung and K.-J. Kim, “Discrete prompt compression with reinforcement learning,” arXiv preprint arXiv:2308.08758, 2023. — [arXiv:2308.08758](https://arxiv.org/abs/2308.08758) [find](https://scholar.google.com/scholar?q=H%2E+Jung+and+K%2E%2DJ%2E+Kim%2C+%E2%80%9CDiscrete+prompt+compression+with+reinforcement+learning%2C%E2%80%9D+arXiv+preprint+arXiv%3A2308%2E08758%2C+2023%2E+%E2%80%94+%5BarXiv%3A2308%2E08758%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2308%2E08758%29)
42. H. Jiang, Q. Wu, C.-Y. Lin, Y. Yang, and L. Qiu, “[Llmlingua: Compressing prompts for accelerated inference of large language models](https://md.research.elamur.ai/free/en/papers/jiang-2023-llmlingua-compressing-prompts-accelerated-inference/),” in The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
43. H. Jiang, Q. Wu, X. Luo, D. Li, C.-Y. Lin, Y. Yang, and L. Qiu, “[Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression](https://md.research.elamur.ai/free/en/papers/jiang-2023-longllmlingua-accelerating-enhancing-llms-long/),” arXiv preprint arXiv:2310.06839, 2023.
44. X. Huang, L. L. Zhang, K.-T. Cheng, and M. Yang, “Boosting llm reasoning: Push the limits of few-shot learning with reinforced in-context pruning,” arXiv preprint arXiv:2312.08901, 2023. — [arXiv:2312.08901](https://arxiv.org/abs/2312.08901) [find](https://scholar.google.com/scholar?q=X%2E+Huang%2C+L%2E+L%2E+Zhang%2C+K%2E%2DT%2E+Cheng%2C+and+M%2E+Yang%2C+%E2%80%9CBoosting+llm+reasoning%3A+Push+the+limits+of+few%2Dshot+learning+with+reinforced+in%2Dcontext+pruning%2C%E2%80%9D+arXiv+preprint+arXiv%3A2312%2E08901%2C+2023%2E+%E2%80%94+%5BarXiv%3A2312)
45. Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui, “A survey for in-context learning,” arXiv preprint arXiv:2301.[00234, 2022](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/).
46. X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, pp. 4582–4597. [find](https://scholar.google.com/scholar?q=X%2E+L%2E+Li+and+P%2E+Liang%2C+%E2%80%9CPrefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2C%E2%80%9D+in+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11th+Internatio)
47. X. Ning, Z. Lin, Z. Zhou, H. Yang, and Y. Wang, “Skeleton-of-thought: Large language models can do parallel decoding,” arXiv preprint arXiv:2307.[15337, 2023](https://md.research.elamur.ai/free/en/papers/ning-2023-skeleton-thought-prompting-llms-efficient/).
48. S. Jin, Y. Wu, H. Zheng, Q. Zhang, M. Lentz, Z. M. Mao, A. Prakash, F. Qian, and D. Zhuo, “Adaptive skeleton graph decoding,” arXiv preprint arXiv:2402.12280, 2024. — [arXiv:2402.12280](https://arxiv.org/abs/2402.12280) [find](https://scholar.google.com/scholar?q=S%2E+Jin%2C+Y%2E+Wu%2C+H%2E+Zheng%2C+Q%2E+Zhang%2C+M%2E+Lentz%2C+Z%2E+M%2E+Mao%2C+A%2E+Prakash%2C+F%2E+Qian%2C+and+D%2E+Zhuo%2C+%E2%80%9CAdaptive+skeleton+graph+decoding%2C%E2%80%9D+arXiv+preprint+arXiv%3A2402%2E12280%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E12280%5D%28https%3A%2F%2Farxiv%2E)
49. M. Liu, A. Zeng, B. Wang, P. Zhang, J. Tang, and Y. Dong, “Apar: Llms can do auto-parallel auto-regressive decoding,” arXiv preprint arXiv:2401.06761, 2024. — [arXiv:2401.06761](https://arxiv.org/abs/2401.06761) [find](https://scholar.google.com/scholar?q=M%2E+Liu%2C+A%2E+Zeng%2C+B%2E+Wang%2C+P%2E+Zhang%2C+J%2E+Tang%2C+and+Y%2E+Dong%2C+%E2%80%9CApar%3A+Llms+can+do+auto%2Dparallel+auto%2Dregressive+decoding%2C%E2%80%9D+arXiv+preprint+arXiv%3A2401%2E06761%2C+2024%2E+%E2%80%94+%5BarXiv%3A2401%2E06761%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
50. T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao, “[Medusa: Simple llm inference acceleration framework with multiple decoding heads](https://md.research.elamur.ai/free/en/papers/cai-2024-medusa-simple-llm-inference-acceleration/),” 2024.
51. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “[Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/),” in Proceedings of the 29th Symposium on Operating Systems Principles, 2023, pp. 611–626.
52. L. Zheng, L. Yin, Z. Xie, J. Huang, C. Sun, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez et al., “Efficiently programming large language models using sglang,” arXiv preprint arXiv:2312.[07104, 2023](https://md.research.elamur.ai/free/en/papers/zheng-2023-sglang-efficient-execution-structured-language/).
53. S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,” Advances in Neural Information Processing Systems, vol. 36, 2024. [find](https://scholar.google.com/scholar?q=S%2E+Yao%2C+D%2E+Yu%2C+J%2E+Zhao%2C+I%2E+Shafran%2C+T%2E+Griffiths%2C+Y%2E+Cao%2C+and+K%2E+Narasimhan%2C+%E2%80%9CTree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2C%E2%80%9D+Advances+in+Neural+Information+Processing+System)
54. M. Besta, N. Blach, A. Kubicek, R. Gerstenberger, M. Podstawski, L. Gianinazzi, J. Gajda, T. Lehmann, H. Niewiadomski, P. Nyczyk et al., “Graph of thoughts: Solving elaborate problems with large language models,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 16, 2024… [find](https://scholar.google.com/scholar?q=M%2E+Besta%2C+N%2E+Blach%2C+A%2E+Kubicek%2C+R%2E+Gerstenberger%2C+M%2E+Podstawski%2C+L%2E+Gianinazzi%2C+J%2E+Gajda%2C+T%2E+Lehmann%2C+H%2E+Niewiadomski%2C+P%2E+Nyczyk+et+al%2E%2C+%E2%80%9CGraph+of+thoughts%3A+Solving+elaborate+problems+with+large+langu)
55. Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou et al., “[The rise and potential of large language model based agents: A survey](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/),” arXiv preprint arXiv:2309.07864, 2023.
56. Q. Sun, Z. Yin, X. Li, Z. Wu, X. Qiu, and L. Kong, “Corex: Pushing the boundaries of complex reasoning through multi-model collaboration,” arXiv preprint arXiv:2310.00280, 2023. [find](https://scholar.google.com/scholar?q=Q%2E+Sun%2C+Z%2E+Yin%2C+X%2E+Li%2C+Z%2E+Wu%2C+X%2E+Qiu%2C+and+L%2E+Kong%2C+%E2%80%9CCorex%3A+Pushing+the+boundaries+of+complex+reasoning+through+multi%2Dmodel+collaboration%2C%E2%80%9D+arXiv+preprint+arXiv%3A2310%2E00280%2C+2023%2E)
57. T. Guo, X. Chen, Y. Wang, R. Chang, S. Pei, N. V. Chawla, O. Wiest, and X. Zhang, “[Large language model based multi-agents: A survey of progress and challenges](https://md.research.elamur.ai/free/en/papers/guo-2024-large-language-model-based/),” arXiv preprint arXiv:2402.01680, 2024.
58. L. Chen, M. Zaharia, and J. Zou, “[Frugalgpt: How to use large language models while reducing cost and improving performance](https://md.research.elamur.ai/free/en/papers/chen-2023-frugalgpt-use-large-language-models/),” arXiv preprint arXiv:2305.05176, 2023.
59. Y. Li, T. Cai, Y. Zhang, D. Chen, and D. Dey, “What makes convolutional models great on long sequence modeling?” arXiv preprint arXiv:2210.09298, 2022. — [arXiv:2210.09298](https://arxiv.org/abs/2210.09298) [find](https://scholar.google.com/scholar?q=Y%2E+Li%2C+T%2E+Cai%2C+Y%2E+Zhang%2C+D%2E+Chen%2C+and+D%2E+Dey%2C+%E2%80%9CWhat+makes+convolutional+models+great+on+long+sequence+modeling%3F%E2%80%9D+arXiv+preprint+arXiv%3A2210%2E09298%2C+2022%2E+%E2%80%94+%5BarXiv%3A2210%2E09298%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2210%2E)
60. D. W. Romero, A. Kuzina, E. J. Bekkers, J. M. Tomczak, and M. Hoogendoorn, “Ckconv: Continuous kernel convolution for sequential data,” arXiv preprint arXiv:2102.02611, 2021. — [arXiv:2102.02611](https://arxiv.org/abs/2102.02611) [find](https://scholar.google.com/scholar?q=D%2E+W%2E+Romero%2C+A%2E+Kuzina%2C+E%2E+J%2E+Bekkers%2C+J%2E+M%2E+Tomczak%2C+and+M%2E+Hoogendoorn%2C+%E2%80%9CCkconv%3A+Continuous+kernel+convolution+for+sequential+data%2C%E2%80%9D+arXiv+preprint+arXiv%3A2102%2E02611%2C+2021%2E+%E2%80%94+%5BarXiv%3A2102%2E02611%5D%28http)
61. M. Poli, S. Massaroli, E. Nguyen, D. Y. Fu, T. Dao, S. Baccus, Y. Bengio, S. Ermon, and C. Ré, “Hyena hierarchy: Towards larger convolutional language models,” in International Conference on Machine Learning. PMLR, 2023, pp. 28 043–28 078. [find](https://scholar.google.com/scholar?q=M%2E+Poli%2C+S%2E+Massaroli%2C+E%2E+Nguyen%2C+D%2E+Y%2E+Fu%2C+T%2E+Dao%2C+S%2E+Baccus%2C+Y%2E+Bengio%2C+S%2E+Ermon%2C+and+C%2E+Re%CC%81%2C+%E2%80%9CHyena+hierarchy%3A+Towards+larger+convolutional+language+models%2C%E2%80%9D+in+International+Conference+on+Machine+)
62. B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, H. Cao, X. Cheng, M. Chung, M. Grella, K. K. GV et al., “Rwkv: Reinventing rnns for the transformer era,” arXiv preprint arXiv:2305.13048, 2023. [find](https://scholar.google.com/scholar?q=B%2E+Peng%2C+E%2E+Alcaide%2C+Q%2E+Anthony%2C+A%2E+Albalak%2C+S%2E+Arcadinho%2C+H%2E+Cao%2C+X%2E+Cheng%2C+M%2E+Chung%2C+M%2E+Grella%2C+K%2E+K%2E+GV+et+al%2E%2C+%E2%80%9CRwkv%3A+Reinventing+rnns+for+the+transformer+era%2C%E2%80%9D+arXiv+preprint+arXiv%3A2305%2E13048%2C+20)
63. Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, J. Xue, J. Wang, and F. Wei, “[Retentive network: A successor to transformer for large language models](https://md.research.elamur.ai/free/en/papers/sun-2023-retentive-network-successor-transformer-large/),” arXiv preprint arXiv:2307.08621, 2023.
64. A. Gu, T. Dao, S. Ermon, A. Rudra, and C. Ré, “Hippo: Recurrent memory with optimal polynomial projections,” Advances in neural information processing systems, vol. 33, pp. 1474–1487, 2020. [find](https://scholar.google.com/scholar?q=A%2E+Gu%2C+T%2E+Dao%2C+S%2E+Ermon%2C+A%2E+Rudra%2C+and+C%2E+Re%CC%81%2C+%E2%80%9CHippo%3A+Recurrent+memory+with+optimal+polynomial+projections%2C%E2%80%9D+Advances+in+neural+information+processing+systems%2C+vol%2E+33%2C+pp%2E+1474%E2%80%931487%2C+2020%2E)
65. A. Gu, I. Johnson, K. Goel, K. Saab, T. Dao, A. Rudra, and C. Ré, “Combining recurrent, convolutional, and continuous-time models with linear state space layers,” Advances in neural information processing systems, vol. 34, pp. 572–585, 2021. [find](https://scholar.google.com/scholar?q=A%2E+Gu%2C+I%2E+Johnson%2C+K%2E+Goel%2C+K%2E+Saab%2C+T%2E+Dao%2C+A%2E+Rudra%2C+and+C%2E+Re%CC%81%2C+%E2%80%9CCombining+recurrent%2C+convolutional%2C+and+continuous%2Dtime+models+with+linear+state+space+layers%2C%E2%80%9D+Advances+in+neural+information+proce)
66. A. Gu, K. Goel, and C. Ré, “Efficiently modeling long sequences with structured state spaces,” arXiv preprint arXiv:2111.00396, 2021. [find](https://scholar.google.com/scholar?q=A%2E+Gu%2C+K%2E+Goel%2C+and+C%2E+Re%CC%81%2C+%E2%80%9CEfficiently+modeling+long+sequences+with+structured+state+spaces%2C%E2%80%9D+arXiv+preprint+arXiv%3A2111%2E00396%2C+2021%2E)
67. A. Gupta, A. Gu, and J. Berant, “Diagonal state spaces are as effective as structured state spaces,” Advances in Neural Information Processing Systems, vol. 35, pp. 22 982–22 994, 2022. [find](https://scholar.google.com/scholar?q=A%2E+Gupta%2C+A%2E+Gu%2C+and+J%2E+Berant%2C+%E2%80%9CDiagonal+state+spaces+are+as+effective+as+structured+state+spaces%2C%E2%80%9D+Advances+in+Neural+Information+Processing+Systems%2C+vol%2E+35%2C+pp%2E+22+982%E2%80%9322+994%2C+2022%2E)
68. A. Gu, K. Goel, A. Gupta, and C. Ré, “On the parameterization and initialization of diagonal state space models,” Advances in Neural Information Processing Systems, vol. 35, pp. 35 971–35 983, 2022. [find](https://scholar.google.com/scholar?q=A%2E+Gu%2C+K%2E+Goel%2C+A%2E+Gupta%2C+and+C%2E+Re%CC%81%2C+%E2%80%9COn+the+parameterization+and+initialization+of+diagonal+state+space+models%2C%E2%80%9D+Advances+in+Neural+Information+Processing+Systems%2C+vol%2E+35%2C+pp%2E+35+971%E2%80%9335+983%2C+2022%2E)
69. H. Mehta, A. Gupta, A. Cutkosky, and B. Neyshabur, “Long range language modeling via gated state spaces,” in International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=H%2E+Mehta%2C+A%2E+Gupta%2C+A%2E+Cutkosky%2C+and+B%2E+Neyshabur%2C+%E2%80%9CLong+range+language+modeling+via+gated+state+spaces%2C%E2%80%9D+in+International+Conference+on+Learning+Representations%2C+2023%2E)
70. D. Y. Fu, T. Dao, K. K. Saab, A. W. Thomas, A. Rudra, and C. Ré, “Hungry hungry hippos: Towards language modeling with state space models,” arXiv preprint arXiv:2212.14052, 2022. [find](https://scholar.google.com/scholar?q=D%2E+Y%2E+Fu%2C+T%2E+Dao%2C+K%2E+K%2E+Saab%2C+A%2E+W%2E+Thomas%2C+A%2E+Rudra%2C+and+C%2E+Re%CC%81%2C+%E2%80%9CHungry+hungry+hippos%3A+Towards+language+modeling+with+state+space+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2212%2E14052%2C+2022%2E)
71. R. Hasani, M. Lechner, T.-H. Wang, M. Chahine, A. Amini, and D. Rus, “Liquid structural state-space models,” arXiv preprint arXiv:2209.12951, 2022. — [arXiv:2209.12951](https://arxiv.org/abs/2209.12951) [find](https://scholar.google.com/scholar?q=R%2E+Hasani%2C+M%2E+Lechner%2C+T%2E%2DH%2E+Wang%2C+M%2E+Chahine%2C+A%2E+Amini%2C+and+D%2E+Rus%2C+%E2%80%9CLiquid+structural+state%2Dspace+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2209%2E12951%2C+2022%2E+%E2%80%94+%5BarXiv%3A2209%2E12951%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2209%2E1295)
72. J. T. Smith, A. Warrington, and S. W. Linderman, “Simplified state space layers for sequence modeling,” arXiv preprint arXiv:2208.04933, 2022. — [arXiv:2208.04933](https://arxiv.org/abs/2208.04933) [find](https://scholar.google.com/scholar?q=J%2E+T%2E+Smith%2C+A%2E+Warrington%2C+and+S%2E+W%2E+Linderman%2C+%E2%80%9CSimplified+state+space+layers+for+sequence+modeling%2C%E2%80%9D+arXiv+preprint+arXiv%3A2208%2E04933%2C+2022%2E+%E2%80%94+%5BarXiv%3A2208%2E04933%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2208%2E04933%29)
73. J. Pilault, M. Fathi, O. Firat, C. Pal, P.-L. Bacon, and R. Goroshin, “[Block-state transformers](https://md.research.elamur.ai/free/en/papers/fathi-2023-block-state-transformers/),” Advances in Neural Information Processing Systems, vol. 36, 2024.
74. J. Wang, J. N. Yan, A. Gu, and A. M. Rush, “Pretraining without attention,” arXiv preprint arXiv:2212.10544, 2022. [find](https://scholar.google.com/scholar?q=J%2E+Wang%2C+J%2E+N%2E+Yan%2C+A%2E+Gu%2C+and+A%2E+M%2E+Rush%2C+%E2%80%9CPretraining+without+attention%2C%E2%80%9D+arXiv+preprint+arXiv%3A2212%2E10544%2C+2022%2E)
75. A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” arXiv preprint arXiv:2312.00752, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Gu+and+T%2E+Dao%2C+%E2%80%9CMamba%3A+Linear%2Dtime+sequence+modeling+with+selective+state+spaces%2C%E2%80%9D+arXiv+preprint+arXiv%3A2312%2E00752%2C+2023%2E)
76. J. Park, J. Park, Z. Xiong, N. Lee, J. Cho, S. Oymak, K. Lee, and D. Papailiopoulos, “Can mamba learn how to learn? a comparative study on in-context learning tasks,” arXiv preprint arXiv:2402.04248, 2024. [find](https://scholar.google.com/scholar?q=J%2E+Park%2C+J%2E+Park%2C+Z%2E+Xiong%2C+N%2E+Lee%2C+J%2E+Cho%2C+S%2E+Oymak%2C+K%2E+Lee%2C+and+D%2E+Papailiopoulos%2C+%E2%80%9CCan+mamba+learn+how+to+learn%3F+a+comparative+study+on+in%2Dcontext+learning+tasks%2C%E2%80%9D+arXiv+preprint+arXiv%3A2402%2E04248%2C+)
77. N. Shazeer, “Fast transformer decoding: One write-head is all you need,” arXiv preprint arXiv:1911.02150, 2019. [find](https://scholar.google.com/scholar?q=N%2E+Shazeer%2C+%E2%80%9CFast+transformer+decoding%3A+One+write%2Dhead+is+all+you+need%2C%E2%80%9D+arXiv+preprint+arXiv%3A1911%2E02150%2C+2019%2E)
78. J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai, “[Gqa: Training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/),” arXiv preprint arXiv:2305.13245, 2023.
79. S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma, “Linformer: Self-attention with linear complexity,” arXiv preprint arXiv:2006.04768, 2020. [find](https://scholar.google.com/scholar?q=S%2E+Wang%2C+B%2E+Z%2E+Li%2C+M%2E+Khabsa%2C+H%2E+Fang%2C+and+H%2E+Ma%2C+%E2%80%9CLinformer%3A+Self%2Dattention+with+linear+complexity%2C%E2%80%9D+arXiv+preprint+arXiv%3A2006%2E04768%2C+2020%2E)
80. G. I. Winata, S. Cahyawijaya, Z. Lin, Z. Liu, and P. Fung, “Lightweight and efficient end-to-end speech recognition using low-rank transformer,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6144–6148. [find](https://scholar.google.com/scholar?q=G%2E+I%2E+Winata%2C+S%2E+Cahyawijaya%2C+Z%2E+Lin%2C+Z%2E+Liu%2C+and+P%2E+Fung%2C+%E2%80%9CLightweight+and+efficient+end%2Dto%2Dend+speech+recognition+using+low%2Drank+transformer%2C%E2%80%9D+in+ICASSP+2020%2D2020+IEEE+International+Conference+on+Ac)
81. A. Gupta, Y. Yuan, Y. Zhou, and C. Mendis, “Flurka: Fast fused low-rank & kernel attention,” arXiv preprint arXiv:2306.15799, 2023. — [arXiv:2306.15799](https://arxiv.org/abs/2306.15799) [find](https://scholar.google.com/scholar?q=A%2E+Gupta%2C+Y%2E+Yuan%2C+Y%2E+Zhou%2C+and+C%2E+Mendis%2C+%E2%80%9CFlurka%3A+Fast+fused+low%2Drank+%26+kernel+attention%2C%E2%80%9D+arXiv+preprint+arXiv%3A2306%2E15799%2C+2023%2E+%E2%80%94+%5BarXiv%3A2306%2E15799%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2306%2E15799%29)
82. X. Ma, X. Kong, S. Wang, C. Zhou, J. May, H. Ma, and L. Zettlemoyer, “Luna: Linear unified nested attention,” Advances in Neural Information Processing Systems, vol. 34, pp. 2441–2453, 2021. [find](https://scholar.google.com/scholar?q=X%2E+Ma%2C+X%2E+Kong%2C+S%2E+Wang%2C+C%2E+Zhou%2C+J%2E+May%2C+H%2E+Ma%2C+and+L%2E+Zettlemoyer%2C+%E2%80%9CLuna%3A+Linear+unified+nested+attention%2C%E2%80%9D+Advances+in+Neural+Information+Processing+Systems%2C+vol%2E+34%2C+pp%2E+2441%E2%80%932453%2C+2021%2E)
83. J. Lee, Y. Lee, J. Kim, A. Kosiorek, S. Choi, and Y. W. Teh, “Set transformer: A framework for attention-based permutation-invariant neural networks,” in International conference on machine learning. PMLR, 2019, pp. 3744–3753. [find](https://scholar.google.com/scholar?q=J%2E+Lee%2C+Y%2E+Lee%2C+J%2E+Kim%2C+A%2E+Kosiorek%2C+S%2E+Choi%2C+and+Y%2E+W%2E+Teh%2C+%E2%80%9CSet+transformer%3A+A+framework+for+attention%2Dbased+permutation%2Dinvariant+neural+networks%2C%E2%80%9D+in+International+conference+on+machine+learning%2E+)
84. A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret, “Transformers are rnns: Fast autoregressive transformers with linear attention,” in International conference on machine learning. PMLR, 2020, pp. 5156–5165. [find](https://scholar.google.com/scholar?q=A%2E+Katharopoulos%2C+A%2E+Vyas%2C+N%2E+Pappas%2C+and+F%2E+Fleuret%2C+%E2%80%9CTransformers+are+rnns%3A+Fast+autoregressive+transformers+with+linear+attention%2C%E2%80%9D+in+International+conference+on+machine+learning%2E+PMLR%2C+2020%2C+pp%2E+)
85. K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Q. Davis, A. Mohiuddin, L. Kaiser et al., “[Rethinking attention with performers](https://md.research.elamur.ai/free/en/papers/choromanski-2020-rethinking-attention-performers/),” in International Conference on Learning Representations, 2020.
86. H. Peng, N. Pappas, D. Yogatama, R. Schwartz, N. Smith, and L. Kong, “Random feature attention,” in International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=H%2E+Peng%2C+N%2E+Pappas%2C+D%2E+Yogatama%2C+R%2E+Schwartz%2C+N%2E+Smith%2C+and+L%2E+Kong%2C+%E2%80%9CRandom+feature+attention%2C%E2%80%9D+in+International+Conference+on+Learning+Representations%2C+2022%2E)
87. P. Kacham, V. Mirrokni, and P. Zhong, “Polysketchformer: Fast transformers via sketches for polynomial kernels,” arXiv preprint arXiv:2310.01655, 2023. — [arXiv:2310.01655](https://arxiv.org/abs/2310.01655) [find](https://scholar.google.com/scholar?q=P%2E+Kacham%2C+V%2E+Mirrokni%2C+and+P%2E+Zhong%2C+%E2%80%9CPolysketchformer%3A+Fast+transformers+via+sketches+for+polynomial+kernels%2C%E2%80%9D+arXiv+preprint+arXiv%3A2310%2E01655%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E01655%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E)
88. W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” The Journal of Machine Learning Research, vol. 23, no. 1, pp. 5232– 5270, 2022. [find](https://scholar.google.com/scholar?q=W%2E+Fedus%2C+B%2E+Zoph%2C+and+N%2E+Shazeer%2C+%E2%80%9CSwitch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2C%E2%80%9D+The+Journal+of+Machine+Learning+Research%2C+vol%2E+23%2C+no%2E+1%2C+pp%2E+5232%E2%80%93+5)
89. Z. Zhang, Y. Lin, Z. Liu, P. Li, M. Sun, and J. Zhou, “[Moefication: Transformer feed-forward layers are mixtures of experts](https://md.research.elamur.ai/free/en/papers/zhang-2022-moefication-transformer-feed-forward-layers/),” in Findings of the Association for Computational Linguistics: ACL 2022, 2022, pp. 877–890.
90. Z.-F. Gao, P. Liu, W. X. Zhao, Z.-Y. Lu, and J.-R. Wen, “Parameter-efficient mixture-of-experts architecture for pre-trained language models,” in Proceedings of the 29th International Conference on Computational Linguistics, 2022, pp. 3263–3273. [find](https://scholar.google.com/scholar?q=Z%2E%2DF%2E+Gao%2C+P%2E+Liu%2C+W%2E+X%2E+Zhao%2C+Z%2E%2DY%2E+Lu%2C+and+J%2E%2DR%2E+Wen%2C+%E2%80%9CParameter%2Defficient+mixture%2Dof%2Dexperts+architecture+for+pre%2Dtrained+language+models%2C%E2%80%9D+in+Proceedings+of+the+29th+International+Conference+on+Co)
91. A. Komatsuzaki, J. Puigcerver, J. Lee-Thorp, C. R. Ruiz, B. Mustafa, J. Ainslie, Y. Tay, M. Dehghani, and N. Houlsby, “[Sparse upcycling: Training mixture-of-experts from dense checkpoints](https://md.research.elamur.ai/free/en/papers/komatsuzaki-2022-sparse-upcycling-training-mixture-experts/),” arXiv preprint arXiv:2212.05055, 2022.
92. M. Lewis, S. Bhosale, T. Dettmers, N. Goyal, and L. Zettlemoyer, “[Base layers: Simplifying training of large, sparse models](https://md.research.elamur.ai/free/en/papers/lewis-2021-base-layers-simplifying-training-large/),” in International Conference on Machine Learning. PMLR, 2021, pp. 6265–6274.
93. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Q. V. Le, J. Laudon et al., “Mixture-of-experts with expert choice routing,” Advances in Neural Information Processing Systems, vol. 35, pp. 7103–7114, 2022. [find](https://scholar.google.com/scholar?q=Y%2E+Zhou%2C+T%2E+Lei%2C+H%2E+Liu%2C+N%2E+Du%2C+Y%2E+Huang%2C+V%2E+Zhao%2C+A%2E+M%2E+Dai%2C+Q%2E+V%2E+Le%2C+J%2E+Laudon+et+al%2E%2C+%E2%80%9CMixture%2Dof%2Dexperts+with+expert+choice+routing%2C%E2%80%9D+Advances+in+Neural+Information+Processing+Systems%2C+vol%2E+35%2C+p)
94. B. Zoph, I. Bello, S. Kumar, N. Du, Y. Huang, J. Dean, N. Shazeer, and W. Fedus, “St-moe: Designing stable and transferable sparse expert models,” arXiv preprint arXiv:2202.08906, 2022. [find](https://scholar.google.com/scholar?q=B%2E+Zoph%2C+I%2E+Bello%2C+S%2E+Kumar%2C+N%2E+Du%2C+Y%2E+Huang%2C+J%2E+Dean%2C+N%2E+Shazeer%2C+and+W%2E+Fedus%2C+%E2%80%9CSt%2Dmoe%3A+Designing+stable+and+transferable+sparse+expert+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2202%2E08906%2C+2022%2E)
95. D. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei, “Stablemoe: Stable routing strategy for mixture of experts,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 7085–7095. [find](https://scholar.google.com/scholar?q=D%2E+Dai%2C+L%2E+Dong%2C+S%2E+Ma%2C+B%2E+Zheng%2C+Z%2E+Sui%2C+B%2E+Chang%2C+and+F%2E+Wei%2C+%E2%80%9CStablemoe%3A+Stable+routing+strategy+for+mixture+of+experts%2C%E2%80%9D+in+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computatio)
96. T. Chen, Z. Zhang, A. K. JAISWAL, S. Liu, and Z. Wang, “Sparse moe as the new dropout: Scaling dense and self-slimmable transformers,” in The Eleventh International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=T%2E+Chen%2C+Z%2E+Zhang%2C+A%2E+K%2E+JAISWAL%2C+S%2E+Liu%2C+and+Z%2E+Wang%2C+%E2%80%9CSparse+moe+as+the+new+dropout%3A+Scaling+dense+and+self%2Dslimmable+transformers%2C%E2%80%9D+in+The+Eleventh+International+Conference+on+Learning+Representati)
97. N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat et al., “[Glam: Efficient scaling of language models with mixture-of-experts](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/),” in International Conference on Machine Learning. PMLR, 2022, pp. 5547–5569.
98. N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” in International Conference on Learning Representations, 2016. [find](https://scholar.google.com/scholar?q=N%2E+Shazeer%2C+A%2E+Mirhoseini%2C+K%2E+Maziarz%2C+A%2E+Davis%2C+Q%2E+Le%2C+G%2E+Hinton%2C+and+J%2E+Dean%2C+%E2%80%9COutrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2C%E2%80%9D+in+International+Conference+on+Learni)
99. D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,” arXiv preprint arXiv:2006.16668, 2020. [find](https://scholar.google.com/scholar?q=D%2E+Lepikhin%2C+H%2E+Lee%2C+Y%2E+Xu%2C+D%2E+Chen%2C+O%2E+Firat%2C+Y%2E+Huang%2C+M%2E+Krikun%2C+N%2E+Shazeer%2C+and+Z%2E+Chen%2C+%E2%80%9CGshard%3A+Scaling+giant+models+with+conditional+computation+and+automatic+sharding%2C%E2%80%9D+arXiv+preprint+arXiv%3A20)
100. C. Hwang, W. Cui, Y. Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas, J. Jose, P. Ram et al., “[Tutel: Adaptive mixture-of-experts at scale](https://md.research.elamur.ai/free/en/papers/hwang-2022-tutel-adaptive-mixture-experts-scale/),” Proceedings of Machine Learning and Systems, vol. 5, 2023.
101. D. P. Bertsekas, “Auction algorithms for network flow problems: A tutorial introduction,” Computational optimization and applications, vol. 1, pp. 7–66, 1992. [find](https://scholar.google.com/scholar?q=D%2E+P%2E+Bertsekas%2C+%E2%80%9CAuction+algorithms+for+network+flow+problems%3A+A+tutorial+introduction%2C%E2%80%9D+Computational+optimization+and+applications%2C+vol%2E+1%2C+pp%2E+7%E2%80%9366%2C+1992%2E)
102. Z. Dai, G. Lai, Y. Yang, and Q. Le, “Funnel-transformer: Filtering out sequential redundancy for efficient language processing,” Advances in neural information processing systems, vol. 33, pp. 4271– 4282, 2020. [find](https://scholar.google.com/scholar?q=Z%2E+Dai%2C+G%2E+Lai%2C+Y%2E+Yang%2C+and+Q%2E+Le%2C+%E2%80%9CFunnel%2Dtransformer%3A+Filtering+out+sequential+redundancy+for+efficient+language+processing%2C%E2%80%9D+Advances+in+neural+information+processing+systems%2C+vol%2E+33%2C+pp%2E+4271%E2%80%93+4)
103. L. Zhu, B. Liao, Q. Zhang, X. Wang, W. Liu, and X. Wang, “[Vision mamba: Efficient visual representation learning with bidirectional state space model](https://md.research.elamur.ai/free/en/papers/zhu-2024-vision-mamba-efficient-visual-representation/),” arXiv preprint arXiv:2401.09417, 2024.
104. W. Hua, Z. Dai, H. Liu, and Q. Le, “[Transformer quality in linear time](https://md.research.elamur.ai/free/en/papers/hua-2022-transformer-quality-linear-time/),” in International Conference on Machine Learning. PMLR, 2022, pp. 9099–9117.
105. AI21, “Jamba: Ai21’s groundbreaking ssm-transformer model,” March 2024. [Online]. Available: [https://www.ai21.com/blog/announcing-jamba](https://www.ai21.com/blog/announcing-jamba) [↗](https://www.ai21.com/blog/announcing-jamba)
106. W. He, K. Han, Y. Tang, C. Wang, Y. Yang, T. Guo, and Y. Wang, “Densemamba: State space models with dense hidden connection for efficient large language models,” arXiv preprint arXiv:2403.00818, 2024. — [arXiv:2403.00818](https://arxiv.org/abs/2403.00818) [find](https://scholar.google.com/scholar?q=W%2E+He%2C+K%2E+Han%2C+Y%2E+Tang%2C+C%2E+Wang%2C+Y%2E+Yang%2C+T%2E+Guo%2C+and+Y%2E+Wang%2C+%E2%80%9CDensemamba%3A+State+space+models+with+dense+hidden+connection+for+efficient+large+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2403%2E00818%2C+2024%2E)
107. Q. Anthony, Y. Tokpanov, P. Glorioso, and B. Millidge, “Blackmamba: Mixture of experts for state-space models,” arXiv preprint arXiv:2402.01771, 2024. — [arXiv:2402.01771](https://arxiv.org/abs/2402.01771) [find](https://scholar.google.com/scholar?q=Q%2E+Anthony%2C+Y%2E+Tokpanov%2C+P%2E+Glorioso%2C+and+B%2E+Millidge%2C+%E2%80%9CBlackmamba%3A+Mixture+of+experts+for+state%2Dspace+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2402%2E01771%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E01771%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E0)
108. M. Pióro, K. Ciebiera, K. Król, J. Ludziejewski, and S. Jaszczur, “Moe-mamba: Efficient selective state space models with mixture of experts,” arXiv preprint arXiv:2401.04081, 2024. — [arXiv:2401.04081](https://arxiv.org/abs/2401.04081) [find](https://scholar.google.com/scholar?q=M%2E+Pio%CC%81ro%2C+K%2E+Ciebiera%2C+K%2E+Kro%CC%81l%2C+J%2E+Ludziejewski%2C+and+S%2E+Jaszczur%2C+%E2%80%9CMoe%2Dmamba%3A+Efficient+selective+state+space+models+with+mixture+of+experts%2C%E2%80%9D+arXiv+preprint+arXiv%3A2401%2E04081%2C+2024%2E+%E2%80%94+%5BarXiv%3A2401%2E04)
109. S. Zhai, W. Talbott, N. Srivastava, C. Huang, H. Goh, R. Zhang, and J. Susskind, “An attention free transformer,” arXiv preprint arXiv:2105.14103, 2021. — [arXiv:2105.14103](https://arxiv.org/abs/2105.14103) [find](https://scholar.google.com/scholar?q=S%2E+Zhai%2C+W%2E+Talbott%2C+N%2E+Srivastava%2C+C%2E+Huang%2C+H%2E+Goh%2C+R%2E+Zhang%2C+and+J%2E+Susskind%2C+%E2%80%9CAn+attention+free+transformer%2C%E2%80%9D+arXiv+preprint+arXiv%3A2105%2E14103%2C+2021%2E+%E2%80%94+%5BarXiv%3A2105%2E14103%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2105)
110. T. Schuster, A. Fisch, J. Gupta, M. Dehghani, D. Bahri, V. Tran, Y. Tay, and D. Metzler, “Confident adaptive language modeling,” Advances in Neural Information Processing Systems, vol. 35, pp. 17 456–17 472, 2022. [find](https://scholar.google.com/scholar?q=T%2E+Schuster%2C+A%2E+Fisch%2C+J%2E+Gupta%2C+M%2E+Dehghani%2C+D%2E+Bahri%2C+V%2E+Tran%2C+Y%2E+Tay%2C+and+D%2E+Metzler%2C+%E2%80%9CConfident+adaptive+language+modeling%2C%E2%80%9D+Advances+in+Neural+Information+Processing+Systems%2C+vol%2E+35%2C+pp%2E+17+456%E2%80%93)
111. L. Del Corro, A. Del Giorno, S. Agarwal, B. Yu, A. Awadallah, and S. Mukherjee, “Skipdecode: Autoregressive skip decoding with batching and caching for efficient llm inference,” arXiv preprint arXiv:2307.02628, 2023. — [arXiv:2307.02628](https://arxiv.org/abs/2307.02628) [find](https://scholar.google.com/scholar?q=L%2E+Del+Corro%2C+A%2E+Del+Giorno%2C+S%2E+Agarwal%2C+B%2E+Yu%2C+A%2E+Awadallah%2C+and+S%2E+Mukherjee%2C+%E2%80%9CSkipdecode%3A+Autoregressive+skip+decoding+with+batching+and+caching+for+efficient+llm+inference%2C%E2%80%9D+arXiv+preprint+arXiv%3A2)
112. W. Liu, P. Zhou, Z. Wang, Z. Zhao, H. Deng, and Q. Ju, “[Fastbert: a self-distilling bert with adaptive inference time](https://md.research.elamur.ai/free/en/papers/liu-2020-fastbert-self-distilling-bert-adaptive/),” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 6035–6044.
113. J. Kong, J. Wang, L.-C. Yu, and X. Zhang, “Accelerating inference for pretrained language models by unified multi-perspective early exiting,” in Proceedings of the 29th International Conference on Computational Linguistics, 2022, pp. 4677–4686. [find](https://scholar.google.com/scholar?q=J%2E+Kong%2C+J%2E+Wang%2C+L%2E%2DC%2E+Yu%2C+and+X%2E+Zhang%2C+%E2%80%9CAccelerating+inference+for+pretrained+language+models+by+unified+multi%2Dperspective+early+exiting%2C%E2%80%9D+in+Proceedings+of+the+29th+International+Conference+on+Com)
114. K. Liao, Y. Zhang, X. Ren, Q. Su, X. Sun, and B. He, “A global past-future early exit method for accelerating inference of pre-trained language models,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=K%2E+Liao%2C+Y%2E+Zhang%2C+X%2E+Ren%2C+Q%2E+Su%2C+X%2E+Sun%2C+and+B%2E+He%2C+%E2%80%9CA+global+past%2Dfuture+early+exit+method+for+accelerating+inference+of+pre%2Dtrained+language+models%2C%E2%80%9D+in+Proceedings+of+the+2021+Conference+of+the+No)
115. J. Xin, R. Tang, J. Lee, Y. Yu, and J. Lin, “[Deebert: Dynamic early exiting for accelerating bert inference](https://md.research.elamur.ai/free/en/papers/xin-2020-deebert-dynamic-early-exiting-accelerating/),” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 2246–2251.
116. W. Zhou, C. Xu, T. Ge, J. McAuley, K. Xu, and F. Wei, “Bert loses patience: Fast and robust inference with early exit,” Advances in Neural Information Processing Systems, vol. 33, pp. 18 330–18 341, 2020. [find](https://scholar.google.com/scholar?q=W%2E+Zhou%2C+C%2E+Xu%2C+T%2E+Ge%2C+J%2E+McAuley%2C+K%2E+Xu%2C+and+F%2E+Wei%2C+%E2%80%9CBert+loses+patience%3A+Fast+and+robust+inference+with+early+exit%2C%E2%80%9D+Advances+in+Neural+Information+Processing+Systems%2C+vol%2E+33%2C+pp%2E+18+330%E2%80%9318+341%2C+2)
117. T. Sun, X. Liu, W. Zhu, Z. Geng, L. Wu, Y. He, Y. Ni, G. Xie, X.-J. Huang, and X. Qiu, “A simple hash-based early exiting approach for language understanding and generation,” in Findings of the Association for Computational Linguistics: ACL 2022, 2022, pp. 2409– 2421. [find](https://scholar.google.com/scholar?q=T%2E+Sun%2C+X%2E+Liu%2C+W%2E+Zhu%2C+Z%2E+Geng%2C+L%2E+Wu%2C+Y%2E+He%2C+Y%2E+Ni%2C+G%2E+Xie%2C+X%2E%2DJ%2E+Huang%2C+and+X%2E+Qiu%2C+%E2%80%9CA+simple+hash%2Dbased+early+exiting+approach+for+language+understanding+and+generation%2C%E2%80%9D+in+Findings+of+the+Associ)
118. Y. Huang, Y. Chen, Z. Yu, and K. McKeown, “In-context learning distillation: Transferring few-shot learning ability of pre-trained language models,” arXiv preprint arXiv:2212.10670, 2022. — [arXiv:2212.10670](https://arxiv.org/abs/2212.10670) [find](https://scholar.google.com/scholar?q=Y%2E+Huang%2C+Y%2E+Chen%2C+Z%2E+Yu%2C+and+K%2E+McKeown%2C+%E2%80%9CIn%2Dcontext+learning+distillation%3A+Transferring+few%2Dshot+learning+ability+of+pre%2Dtrained+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2212%2E10670%2C+2022%2E+%E2%80%94+%5BarXiv%3A221)
119. J. Zhao, W. Zhao, A. Drozdov, B. Rozonoyer, M. A. Sultan, J.-Y. Lee, M. Iyyer, and A. McCallum, “Multistage collaborative knowledge distillation from large language models,” arXiv preprint arXiv:2311.08640, 2023. — [arXiv:2311.08640](https://arxiv.org/abs/2311.08640) [find](https://scholar.google.com/scholar?q=J%2E+Zhao%2C+W%2E+Zhao%2C+A%2E+Drozdov%2C+B%2E+Rozonoyer%2C+M%2E+A%2E+Sultan%2C+J%2E%2DY%2E+Lee%2C+M%2E+Iyyer%2C+and+A%2E+McCallum%2C+%E2%80%9CMultistage+collaborative+knowledge+distillation+from+large+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2311%2E)
120. C.-Y. Hsieh, C.-L. Li, C.-K. Yeh, H. Nakhost, Y. Fujii, A. Ratner, R. Krishna, C.-Y. Lee, and T. Pfister, “[Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes](https://md.research.elamur.ai/free/en/papers/hsieh-2023-distilling-step-step-outperforming-larger/),” arXiv preprint arXiv:2305.02301, 2023.
121. L. H. Li, J. Hessel, Y. Yu, X. Ren, K.-W. Chang, and Y. Choi, “Symbolic chain-of-thought distillation: Small models can also” think” step-by-step,” arXiv preprint arXiv:2306.[14050, 2023](https://md.research.elamur.ai/free/en/papers/li-2023-symbolic-chain-thought-distillation-small/).
122. L. C. Magister, J. Mallinson, J. Adamek, E. Malmi, and A. Severyn, “[Teaching small language models to reason](https://md.research.elamur.ai/free/en/papers/magister-2023-teaching-small-language-models-reason/),” arXiv preprint arXiv:2212.08410, 2022.
123. H. Chen, S. Wu, X. Quan, R. Wang, M. Yan, and J. Zhang, “Mcc-kd: Multi-cot consistent knowledge distillation,” arXiv preprint arXiv:2310.14747, 2023. [find](https://scholar.google.com/scholar?q=H%2E+Chen%2C+S%2E+Wu%2C+X%2E+Quan%2C+R%2E+Wang%2C+M%2E+Yan%2C+and+J%2E+Zhang%2C+%E2%80%9CMcc%2Dkd%3A+Multi%2Dcot+consistent+knowledge+distillation%2C%E2%80%9D+arXiv+preprint+arXiv%3A2310%2E14747%2C+2023%2E)
124. N. Ho, L. Schmid, and S.-Y. Yun, “[Large language models are reasoning teachers](https://md.research.elamur.ai/free/en/papers/ho-2023-large-language-models-are-reasoning/),” arXiv preprint arXiv:2212.10071, 2022.
125. K. Shridhar, A. Stolfo, and M. Sachan, “[Distilling reasoning capabilities into smaller language models](https://md.research.elamur.ai/free/en/papers/shridhar-2023-distilling-reasoning-capabilities-into-smaller/),” in Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 7059– 7073.
126. X. Zhu, B. Qi, K. Zhang, X. Long, and B. Zhou, “Pad: Program-aided distillation specializes large models in reasoning,” arXiv preprint arXiv:2305.13888, 2023. [find](https://scholar.google.com/scholar?q=X%2E+Zhu%2C+B%2E+Qi%2C+K%2E+Zhang%2C+X%2E+Long%2C+and+B%2E+Zhou%2C+%E2%80%9CPad%3A+Program%2Daided+distillation+specializes+large+models+in+reasoning%2C%E2%80%9D+arXiv+preprint+arXiv%3A2305%2E13888%2C+2023%2E)
127. P. Wang, Z. Wang, Z. Li, Y. Gao, B. Yin, and X. Ren, “[Scott: Self-consistent chain-of-thought distillation](https://md.research.elamur.ai/free/en/papers/wang-2023-scott-self-consistent-chain-thought/),” arXiv preprint arXiv:2305.01879, 2023.
128. Z. Chen, Q. Gao, A. Bosselut, A. Sabharwal, and K. Richardson, “Disco: distilling counterfactuals with large language models,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 5514– 5528. [find](https://scholar.google.com/scholar?q=Z%2E+Chen%2C+Q%2E+Gao%2C+A%2E+Bosselut%2C+A%2E+Sabharwal%2C+and+K%2E+Richardson%2C+%E2%80%9CDisco%3A+distilling+counterfactuals+with+large+language+models%2C%E2%80%9D+in+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computat)
129. M. Wu, A. Waheed, C. Zhang, M. Abdul-Mageed, and A. F. Aji, “[Lamini-lm: A diverse herd of distilled models from large-scale instructions](https://md.research.elamur.ai/free/en/papers/wu-2023-lamini-lm-diverse-herd-distilled/),” arXiv preprint arXiv:2304.14402, 2023.
130. Y. Jiang, C. Chan, M. Chen, and W. Wang, “[Lion: Adversarial distillation of proprietary large language models](https://md.research.elamur.ai/free/en/papers/jiang-2023-lion-adversarial-distillation-proprietary-large/),” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 3134–3154.
131. Y. Gu, L. Dong, F. Wei, and M. Huang, “Knowledge distillation of large language models,” arXiv preprint arXiv:2306.[08543, 2023](https://md.research.elamur.ai/free/en/papers/gu-2023-minillm-policy-distillation-large-language/).
132. R. Agarwal, N. Vieillard, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem, “Gkd: Generalized knowledge distillation for autoregressive sequence models,” arXiv preprint arXiv:2306.[13649, 2023](https://md.research.elamur.ai/free/en/papers/agarwal-2023-policy-distillation-language-models-learning/).
133. C. Liang, S. Zuo, Q. Zhang, P. He, W. Chen, and T. Zhao, “Less is more: Task-aware layer-wise distillation for language model compression,” in International Conference on Machine Learning. PMLR, 2023, pp. 20 852–20 867. [find](https://scholar.google.com/scholar?q=C%2E+Liang%2C+S%2E+Zuo%2C+Q%2E+Zhang%2C+P%2E+He%2C+W%2E+Chen%2C+and+T%2E+Zhao%2C+%E2%80%9CLess+is+more%3A+Task%2Daware+layer%2Dwise+distillation+for+language+model+compression%2C%E2%80%9D+in+International+Conference+on+Machine+Learning%2E+PMLR%2C+2023%2C)
134. I. Timiryasov and J.-L. Tastet, “Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty,” arXiv preprint arXiv:2308.02019, 2023. [find](https://scholar.google.com/scholar?q=I%2E+Timiryasov+and+J%2E%2DL%2E+Tastet%2C+%E2%80%9CBaby+llama%3A+knowledge+distillation+from+an+ensemble+of+teachers+trained+on+a+small+dataset+with+no+performance+penalty%2C%E2%80%9D+arXiv+preprint+arXiv%3A2308%2E02019%2C+2023%2E)
135. C. Zhang, Y. Yang, J. Liu, J. Wang, Y. Xian, B. Wang, and D. Song, “Lifting the curse of capacity gap in distilling language models,” arXiv preprint arXiv:2305.12129, 2023. [find](https://scholar.google.com/scholar?q=C%2E+Zhang%2C+Y%2E+Yang%2C+J%2E+Liu%2C+J%2E+Wang%2C+Y%2E+Xian%2C+B%2E+Wang%2C+and+D%2E+Song%2C+%E2%80%9CLifting+the+curse+of+capacity+gap+in+distilling+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2305%2E12129%2C+2023%2E)
136. L. Hou, Z. Huang, L. Shang, X. Jiang, X. Chen, and Q. Liu, “Dynabert: Dynamic bert with adaptive width and depth,” Advances in Neural Information Processing Systems, vol. 33, pp. 9782–9793, 2020. [find](https://scholar.google.com/scholar?q=L%2E+Hou%2C+Z%2E+Huang%2C+L%2E+Shang%2C+X%2E+Jiang%2C+X%2E+Chen%2C+and+Q%2E+Liu%2C+%E2%80%9CDynabert%3A+Dynamic+bert+with+adaptive+width+and+depth%2C%E2%80%9D+Advances+in+Neural+Information+Processing+Systems%2C+vol%2E+33%2C+pp%2E+9782%E2%80%939793%2C+2020%2E)
137. S. Padmanabhan, Y. Onoe, M. J. Zhang, G. Durrett, and E. Choi, “Propagating knowledge updates to lms through distillation,” arXiv preprint arXiv:2306.09306, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Padmanabhan%2C+Y%2E+Onoe%2C+M%2E+J%2E+Zhang%2C+G%2E+Durrett%2C+and+E%2E+Choi%2C+%E2%80%9CPropagating+knowledge+updates+to+lms+through+distillation%2C%E2%80%9D+arXiv+preprint+arXiv%3A2306%2E09306%2C+2023%2E)
138. Y. Yin, C. Chen, L. Shang, X. Jiang, X. Chen, and Q. Liu, “Autotinybert: Automatic hyper-parameter optimization for efficient pre-trained language models,” arXiv preprint arXiv:2107.13686, 2021. [find](https://scholar.google.com/scholar?q=Y%2E+Yin%2C+C%2E+Chen%2C+L%2E+Shang%2C+X%2E+Jiang%2C+X%2E+Chen%2C+and+Q%2E+Liu%2C+%E2%80%9CAutotinybert%3A+Automatic+hyper%2Dparameter+optimization+for+efficient+pre%2Dtrained+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2107%2E13686%2C+2021%2E)
139. J. Xu, X. Tan, R. Luo, K. Song, J. Li, T. Qin, and T.-Y. Liu, “Nas-bert: task-agnostic and adaptive-size bert compression with neural architecture search,” in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021, pp. 1933– 1943. [find](https://scholar.google.com/scholar?q=J%2E+Xu%2C+X%2E+Tan%2C+R%2E+Luo%2C+K%2E+Song%2C+J%2E+Li%2C+T%2E+Qin%2C+and+T%2E%2DY%2E+Liu%2C+%E2%80%9CNas%2Dbert%3A+task%2Dagnostic+and+adaptive%2Dsize+bert+compression+with+neural+architecture+search%2C%E2%80%9D+in+Proceedings+of+the+27th+ACM+SIGKDD+Confer)
140. A. Klein, J. Golebiowski, X. Ma, V. Perrone, and C. Archambeau, “Structural pruning of large language models via neural architecture search,” 2023. [find](https://scholar.google.com/scholar?q=A%2E+Klein%2C+J%2E+Golebiowski%2C+X%2E+Ma%2C+V%2E+Perrone%2C+and+C%2E+Archambeau%2C+%E2%80%9CStructural+pruning+of+large+language+models+via+neural+architecture+search%2C%E2%80%9D+2023%2E)
141. M. Javaheripi, G. de Rosa, S. Mukherjee, S. Shah, T. Religa, C. C. Teodoro Mendes, S. Bubeck, F. Koushanfar, and D. Dey, “Litetransformersearch: Training-free neural architecture search for efficient language models,” Advances in Neural Information Processing Systems, vol. 35, pp. 24 254–24 267, 202… [find](https://scholar.google.com/scholar?q=M%2E+Javaheripi%2C+G%2E+de+Rosa%2C+S%2E+Mukherjee%2C+S%2E+Shah%2C+T%2E+Religa%2C+C%2E+C%2E+Teodoro+Mendes%2C+S%2E+Bubeck%2C+F%2E+Koushanfar%2C+and+D%2E+Dey%2C+%E2%80%9CLitetransformersearch%3A+Training%2Dfree+neural+architecture+search+for+efficient+)
142. D. D. Xu, S. Mukherjee, X. Liu, D. Dey, W. Wang, X. Zhang, A. Awadallah, and J. Gao, “Few-shot task-agnostic neural architecture search for distilling large language models,” Advances in Neural Information Processing Systems, vol. 35, pp. 28 644–28 656, 2022. [find](https://scholar.google.com/scholar?q=D%2E+D%2E+Xu%2C+S%2E+Mukherjee%2C+X%2E+Liu%2C+D%2E+Dey%2C+W%2E+Wang%2C+X%2E+Zhang%2C+A%2E+Awadallah%2C+and+J%2E+Gao%2C+%E2%80%9CFew%2Dshot+task%2Dagnostic+neural+architecture+search+for+distilling+large+language+models%2C%E2%80%9D+Advances+in+Neural+Inform)
143. A. Kaushal, T. Vaidhya, and I. Rish, “Lord: Low rank decomposition of monolingual code llms for one-shot compression,” arXiv preprint arXiv:2309.14021, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Kaushal%2C+T%2E+Vaidhya%2C+and+I%2E+Rish%2C+%E2%80%9CLord%3A+Low+rank+decomposition+of+monolingual+code+llms+for+one%2Dshot+compression%2C%E2%80%9D+arXiv+preprint+arXiv%3A2309%2E14021%2C+2023%2E)
144. M. Xu, Y. L. Xu, and D. P. Mandic, “Tensorgpt: Efficient compression of the embedding layer in llms based on the tensor-train decomposition,” arXiv preprint arXiv:2307.00526, 2023. — [arXiv:2307.00526](https://arxiv.org/abs/2307.00526) [find](https://scholar.google.com/scholar?q=M%2E+Xu%2C+Y%2E+L%2E+Xu%2C+and+D%2E+P%2E+Mandic%2C+%E2%80%9CTensorgpt%3A+Efficient+compression+of+the+embedding+layer+in+llms+based+on+the+tensor%2Dtrain+decomposition%2C%E2%80%9D+arXiv+preprint+arXiv%3A2307%2E00526%2C+2023%2E+%E2%80%94+%5BarXiv%3A2307%2E00526)
145. Y. Li, Y. Yu, Q. Zhang, C. Liang, P. He, W. Chen, and T. Zhao, “Losparse: Structured compression of large language models based on low-rank and sparse approximation,” arXiv preprint arXiv:2306.11222, 2023. — [arXiv:2306.11222](https://arxiv.org/abs/2306.11222) [find](https://scholar.google.com/scholar?q=Y%2E+Li%2C+Y%2E+Yu%2C+Q%2E+Zhang%2C+C%2E+Liang%2C+P%2E+He%2C+W%2E+Chen%2C+and+T%2E+Zhao%2C+%E2%80%9CLosparse%3A+Structured+compression+of+large+language+models+based+on+low%2Drank+and+sparse+approximation%2C%E2%80%9D+arXiv+preprint+arXiv%3A2306%2E11222%2C+)
146. R. Saha, V. Srivastava, and M. Pilanci, “Matrix compression via randomized low rank and low precision factorization,” arXiv preprint arXiv:2310.11028, 2023. — [arXiv:2310.11028](https://arxiv.org/abs/2310.11028) [find](https://scholar.google.com/scholar?q=R%2E+Saha%2C+V%2E+Srivastava%2C+and+M%2E+Pilanci%2C+%E2%80%9CMatrix+compression+via+randomized+low+rank+and+low+precision+factorization%2C%E2%80%9D+arXiv+preprint+arXiv%3A2310%2E11028%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E11028%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
147. Z. Yao, X. Wu, C. Li, S. Youn, and Y. He, “Zeroquant-v2: Exploring post-training quantization in llms from comprehensive study to low rank compensation,” arXiv preprint arXiv:2303.08302, 2023. — [arXiv:2303.08302](https://arxiv.org/abs/2303.08302) [find](https://scholar.google.com/scholar?q=Z%2E+Yao%2C+X%2E+Wu%2C+C%2E+Li%2C+S%2E+Youn%2C+and+Y%2E+He%2C+%E2%80%9CZeroquant%2Dv2%3A+Exploring+post%2Dtraining+quantization+in+llms+from+comprehensive+study+to+low+rank+compensation%2C%E2%80%9D+arXiv+preprint+arXiv%3A2303%2E08302%2C+2023%2E+%E2%80%94+%5BarXi)
148. R. Chand, Y. Prabhu, and P. Kumar, “Dsformer: Effective compression of text-transformers by dense-sparse weight factorization,” arXiv preprint arXiv:2312.13211, 2023. — [arXiv:2312.13211](https://arxiv.org/abs/2312.13211) [find](https://scholar.google.com/scholar?q=R%2E+Chand%2C+Y%2E+Prabhu%2C+and+P%2E+Kumar%2C+%E2%80%9CDsformer%3A+Effective+compression+of+text%2Dtransformers+by+dense%2Dsparse+weight+factorization%2C%E2%80%9D+arXiv+preprint+arXiv%3A2312%2E13211%2C+2023%2E+%E2%80%94+%5BarXiv%3A2312%2E13211%5D%28https%3A%2F%2Farxi)
149. Z. Yuan, Y. Shang, Y. Song, Q. Wu, Y. Yan, and G. Sun, “Asvd: Activation-aware singular value decomposition for compressing large language models,” arXiv preprint arXiv:2312.05821, 2023. — [arXiv:2312.05821](https://arxiv.org/abs/2312.05821) [find](https://scholar.google.com/scholar?q=Z%2E+Yuan%2C+Y%2E+Shang%2C+Y%2E+Song%2C+Q%2E+Wu%2C+Y%2E+Yan%2C+and+G%2E+Sun%2C+%E2%80%9CAsvd%3A+Activation%2Daware+singular+value+decomposition+for+compressing+large+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2312%2E05821%2C+2023%2E+%E2%80%94+%5BarXiv%3A2312)
150. R. Child, S. Gray, A. Radford, and I. Sutskever, “[Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/),” arXiv preprint arXiv:1904.10509, 2019.
151. G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis, “Efficient streaming language models with attention sinks,” arXiv preprint arXiv:2309.17453, 2023. [find](https://scholar.google.com/scholar?q=G%2E+Xiao%2C+Y%2E+Tian%2C+B%2E+Chen%2C+S%2E+Han%2C+and+M%2E+Lewis%2C+%E2%80%9CEfficient+streaming+language+models+with+attention+sinks%2C%E2%80%9D+arXiv+preprint+arXiv%3A2309%2E17453%2C+2023%2E)
152. I. Beltagy, M. E. Peters, and A. Cohan, “[Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/),” arXiv preprint arXiv:2004.05150, 2020.
153. M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang et al., “[Big bird: Transformers for longer sequences](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/),” Advances in neural information processing systems, vol. 33, pp. 17 283–17 297, 2020.
154. S. Dai, H. Genc, R. Venkatesan, and B. Khailany, “Efficient transformer inference with statically structured sparse attention,” in 2023 60th ACM/IEEE Design Automation Conference (DAC). IEEE, 2023, pp. 1–6. [find](https://scholar.google.com/scholar?q=S%2E+Dai%2C+H%2E+Genc%2C+R%2E+Venkatesan%2C+and+B%2E+Khailany%2C+%E2%80%9CEfficient+transformer+inference+with+statically+structured+sparse+attention%2C%E2%80%9D+in+2023+60th+ACM%2FIEEE+Design+Automation+Conference+%28DAC%29%2E+IEEE%2C+2023%2C+pp)
155. Anonymous, “SemSA: Semantic sparse attention is hidden in large language models.” 2023. [Online]. Available: [https://openreview.net/forum?id=eG9AkHtYYH](https://openreview.net/forum?id=eG9AkHtYYH) [↗](https://openreview.net/forum?id=eG9AkHtYYH)
156. H. Wang, Z. Zhang, and S. Han, “[Spatten: Efficient sparse attention architecture with cascade token and head pruning](https://md.research.elamur.ai/free/en/papers/wang-2020-spatten-efficient-sparse-attention-architecture/),” in 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, pp. 97–110.
157. L. Ren, Y. Liu, S. Wang, Y. Xu, C. Zhu, and C. Zhai, “Sparse modular activation for efficient sequence modeling,” arXiv preprint arXiv:2306.11197, 2023. — [arXiv:2306.11197](https://arxiv.org/abs/2306.11197) [find](https://scholar.google.com/scholar?q=L%2E+Ren%2C+Y%2E+Liu%2C+S%2E+Wang%2C+Y%2E+Xu%2C+C%2E+Zhu%2C+and+C%2E+Zhai%2C+%E2%80%9CSparse+modular+activation+for+efficient+sequence+modeling%2C%E2%80%9D+arXiv+preprint+arXiv%3A2306%2E11197%2C+2023%2E+%E2%80%94+%5BarXiv%3A2306%2E11197%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2306)
158. S. Anagnostidis, D. Pavllo, L. Biggio, L. Noci, A. Lucchi, and T. Hoffmann, “Dynamic context pruning for efficient and interpretable autoregressive transformers,” arXiv preprint arXiv:2305.15805, 2023. — [arXiv:2305.15805](https://arxiv.org/abs/2305.15805) [find](https://scholar.google.com/scholar?q=S%2E+Anagnostidis%2C+D%2E+Pavllo%2C+L%2E+Biggio%2C+L%2E+Noci%2C+A%2E+Lucchi%2C+and+T%2E+Hoffmann%2C+%E2%80%9CDynamic+context+pruning+for+efficient+and+interpretable+autoregressive+transformers%2C%E2%80%9D+arXiv+preprint+arXiv%3A2305%2E15805%2C+2023)
159. N. Kitaev, Ł. Kaiser, and A. Levskaya, “[Reformer: The efficient transformer](https://md.research.elamur.ai/free/en/papers/kitaev-2020-reformer-efficient-transformer/),” arXiv preprint arXiv:2001.04451, 2020.
160. M. Pagliardini, D. Paliotta, M. Jaggi, and F. Fleuret, “Faster causal attention over large sequences through sparse flash attention,” arXiv preprint arXiv:2306.01160, 2023. — [arXiv:2306.01160](https://arxiv.org/abs/2306.01160) [find](https://scholar.google.com/scholar?q=M%2E+Pagliardini%2C+D%2E+Paliotta%2C+M%2E+Jaggi%2C+and+F%2E+Fleuret%2C+%E2%80%9CFaster+causal+attention+over+large+sequences+through+sparse+flash+attention%2C%E2%80%9D+arXiv+preprint+arXiv%3A2306%2E01160%2C+2023%2E+%E2%80%94+%5BarXiv%3A2306%2E01160%5D%28https%3A)
161. A. Roy, M. Saffar, A. Vaswani, and D. Grangier, “[Efficient content-based sparse attention with routing transformers](https://md.research.elamur.ai/free/en/papers/roy-2020-efficient-content-based-sparse-attention/),” Transactions of the Association for Computational Linguistics, vol. 9, pp. 53–68, 2021.
162. Y. Tay, D. Bahri, L. Yang, D. Metzler, and D.-C. Juan, “Sparse sinkhorn attention,” in International Conference on Machine Learning. PMLR, 2020, pp. 9438–9447. [find](https://scholar.google.com/scholar?q=Y%2E+Tay%2C+D%2E+Bahri%2C+L%2E+Yang%2C+D%2E+Metzler%2C+and+D%2E%2DC%2E+Juan%2C+%E2%80%9CSparse+sinkhorn+attention%2C%E2%80%9D+in+International+Conference+on+Machine+Learning%2E+PMLR%2C+2020%2C+pp%2E+9438%E2%80%939447%2E)
163. Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett et al., “H2o: Heavy-hitter oracle for efficient generative inference of large language models,” Advances in Neural Information Processing Systems, vol. 36, 2024. [find](https://scholar.google.com/scholar?q=Z%2E+Zhang%2C+Y%2E+Sheng%2C+T%2E+Zhou%2C+T%2E+Chen%2C+L%2E+Zheng%2C+R%2E+Cai%2C+Z%2E+Song%2C+Y%2E+Tian%2C+C%2E+Re%CC%81%2C+C%2E+Barrett+et+al%2E%2C+%E2%80%9CH2o%3A+Heavy%2Dhitter+oracle+for+efficient+generative+inference+of+large+language+models%2C%E2%80%9D+Advances+in)
164. A. Feng, I. Li, Y. Jiang, and R. Ying, “Diffuser: efficient transformers with multi-hop attention diffusion for long sequences,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 11, 2023, pp. 12 772–12 780. [find](https://scholar.google.com/scholar?q=A%2E+Feng%2C+I%2E+Li%2C+Y%2E+Jiang%2C+and+R%2E+Ying%2C+%E2%80%9CDiffuser%3A+efficient+transformers+with+multi%2Dhop+attention+diffusion+for+long+sequences%2C%E2%80%9D+in+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+vol%2E+)
165. E. Frantar and D. Alistarh, “Sparsegpt: Massive language models can be accurately pruned in one-shot,” 2023. [find](https://scholar.google.com/scholar?q=E%2E+Frantar+and+D%2E+Alistarh%2C+%E2%80%9CSparsegpt%3A+Massive+language+models+can+be+accurately+pruned+in+one%2Dshot%2C%E2%80%9D+2023%2E)
166. M. Sun, Z. Liu, A. Bair, and J. Z. Kolter, “A simple and effective pruning approach for large language models,” arXiv preprint arXiv:2306.11695, 2023. [find](https://scholar.google.com/scholar?q=M%2E+Sun%2C+Z%2E+Liu%2C+A%2E+Bair%2C+and+J%2E+Z%2E+Kolter%2C+%E2%80%9CA+simple+and+effective+pruning+approach+for+large+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2306%2E11695%2C+2023%2E)
167. H. Shao, B. Liu, and Y. Qian, “One-shot sensitivity-aware mixed sparsity pruning for large language models,” arXiv preprint arXiv:2310.09499, 2023. — [arXiv:2310.09499](https://arxiv.org/abs/2310.09499) [find](https://scholar.google.com/scholar?q=H%2E+Shao%2C+B%2E+Liu%2C+and+Y%2E+Qian%2C+%E2%80%9COne%2Dshot+sensitivity%2Daware+mixed+sparsity+pruning+for+large+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2310%2E09499%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E09499%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E0949)
168. A. Syed, P. H. Guo, and V. Sundarapandiyan, “Prune and tune: Improving efficient pruning techniques for massive language models,” 2023. [find](https://scholar.google.com/scholar?q=A%2E+Syed%2C+P%2E+H%2E+Guo%2C+and+V%2E+Sundarapandiyan%2C+%E2%80%9CPrune+and+tune%3A+Improving+efficient+pruning+techniques+for+massive+language+models%2C%E2%80%9D+2023%2E)
169. X. Wei, Y. Zhang, Y. Li, X. Zhang, R. Gong, J. Guo, and X. Liu, “Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling,” arXiv preprint arXiv:2304.09145, 2023. — [arXiv:2304.09145](https://arxiv.org/abs/2304.09145) [find](https://scholar.google.com/scholar?q=X%2E+Wei%2C+Y%2E+Zhang%2C+Y%2E+Li%2C+X%2E+Zhang%2C+R%2E+Gong%2C+J%2E+Guo%2C+and+X%2E+Liu%2C+%E2%80%9COutlier+suppression%2B%3A+Accurate+quantization+of+large+language+models+by+equivalent+and+optimal+shifting+and+scaling%2C%E2%80%9D+arXiv+preprint+ar)
170. P. Xu, W. Shao, M. Chen, S. Tang, K. Zhang, P. Gao, F. An, Y. Qiao, and P. Luo, “Besa: Pruning large language models with blockwise parameter-efficient sparsity allocation,” in The Twelfth International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=P%2E+Xu%2C+W%2E+Shao%2C+M%2E+Chen%2C+S%2E+Tang%2C+K%2E+Zhang%2C+P%2E+Gao%2C+F%2E+An%2C+Y%2E+Qiao%2C+and+P%2E+Luo%2C+%E2%80%9CBesa%3A+Pruning+large+language+models+with+blockwise+parameter%2Defficient+sparsity+allocation%2C%E2%80%9D+in+The+Twelfth+Internation)
171. E. Kurtic, D. Campos, T. Nguyen, E. Frantar, M. Kurtz, B. Fineran, M. Goin, and D. Alistarh, “The optimal bert surgeon: Scalable and accurate second-order pruning for large language models,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 4163–418… [find](https://scholar.google.com/scholar?q=E%2E+Kurtic%2C+D%2E+Campos%2C+T%2E+Nguyen%2C+E%2E+Frantar%2C+M%2E+Kurtz%2C+B%2E+Fineran%2C+M%2E+Goin%2C+and+D%2E+Alistarh%2C+%E2%80%9CThe+optimal+bert+surgeon%3A+Scalable+and+accurate+second%2Dorder+pruning+for+large+language+models%2C%E2%80%9D+in+Procee)
172. W. Kwon, S. Kim, M. W. Mahoney, J. Hassoun, K. Keutzer, and A. Gholami, “A fast post-training pruning framework for transformers,” Advances in Neural Information Processing Systems, vol. 35, pp. 24 101–24 116, 2022. [find](https://scholar.google.com/scholar?q=W%2E+Kwon%2C+S%2E+Kim%2C+M%2E+W%2E+Mahoney%2C+J%2E+Hassoun%2C+K%2E+Keutzer%2C+and+A%2E+Gholami%2C+%E2%80%9CA+fast+post%2Dtraining+pruning+framework+for+transformers%2C%E2%80%9D+Advances+in+Neural+Information+Processing+Systems%2C+vol%2E+35%2C+pp%2E+24+10)
173. Y. Zhang, H. Bai, H. Lin, J. Zhao, L. Hou, and C. V. Cannistraci, “An efficient plug-and-play post-training pruning strategy in large language models,” 2023. [find](https://scholar.google.com/scholar?q=Y%2E+Zhang%2C+H%2E+Bai%2C+H%2E+Lin%2C+J%2E+Zhao%2C+L%2E+Hou%2C+and+C%2E+V%2E+Cannistraci%2C+%E2%80%9CAn+efficient+plug%2Dand%2Dplay+post%2Dtraining+pruning+strategy+in+large+language+models%2C%E2%80%9D+2023%2E)
174. X. Ma, G. Fang, and X. Wang, “[Llm-pruner: On the structural pruning of large language models](https://md.research.elamur.ai/free/en/papers/ma-2023-llm-pruner-structural-pruning-large/),” Advances in neural information processing systems, vol. 36, 2024.
175. M. Xia, T. Gao, Z. Zeng, and D. Chen, “Sheared llama: Accelerating language model pre-training via structured pruning,” arXiv preprint arXiv:2310.06694, 2023. [find](https://scholar.google.com/scholar?q=M%2E+Xia%2C+T%2E+Gao%2C+Z%2E+Zeng%2C+and+D%2E+Chen%2C+%E2%80%9CSheared+llama%3A+Accelerating+language+model+pre%2Dtraining+via+structured+pruning%2C%E2%80%9D+arXiv+preprint+arXiv%3A2310%2E06694%2C+2023%2E)
176. E. Kurtić, E. Frantar, and D. Alistarh, “Ziplm: Inference-aware structured pruning of language models,” Advances in Neural Information Processing Systems, vol. 36, 2024. [find](https://scholar.google.com/scholar?q=E%2E+Kurtic%CC%81%2C+E%2E+Frantar%2C+and+D%2E+Alistarh%2C+%E2%80%9CZiplm%3A+Inference%2Daware+structured+pruning+of+language+models%2C%E2%80%9D+Advances+in+Neural+Information+Processing+Systems%2C+vol%2E+36%2C+2024%2E)
177. M. Zhang, H. Chen, C. Shen, Z. Yang, L. Ou, X. Yu, and B. Zhuang, “Loraprune: Pruning meets low-rank parameter-efficient fine-tuning,” 2023. [find](https://scholar.google.com/scholar?q=M%2E+Zhang%2C+H%2E+Chen%2C+C%2E+Shen%2C+Z%2E+Yang%2C+L%2E+Ou%2C+X%2E+Yu%2C+and+B%2E+Zhuang%2C+%E2%80%9CLoraprune%3A+Pruning+meets+low%2Drank+parameter%2Defficient+fine%2Dtuning%2C%E2%80%9D+2023%2E)
178. T. Chen, T. Ding, B. Yadav, I. Zharkov, and L. Liang, “Lorashear: Efficient large language model structured pruning and knowledge recovery,” arXiv preprint arXiv:2310.18356, 2023. [find](https://scholar.google.com/scholar?q=T%2E+Chen%2C+T%2E+Ding%2C+B%2E+Yadav%2C+I%2E+Zharkov%2C+and+L%2E+Liang%2C+%E2%80%9CLorashear%3A+Efficient+large+language+model+structured+pruning+and+knowledge+recovery%2C%E2%80%9D+arXiv+preprint+arXiv%3A2310%2E18356%2C+2023%2E)
179. S. Ashkboos, M. L. Croci, M. G. d. Nascimento, T. Hoefler, and J. Hensman, “Slicegpt: Compress large language models by deleting rows and columns,” arXiv preprint arXiv:2401.15024, 2024. [find](https://scholar.google.com/scholar?q=S%2E+Ashkboos%2C+M%2E+L%2E+Croci%2C+M%2E+G%2E+d%2E+Nascimento%2C+T%2E+Hoefler%2C+and+J%2E+Hensman%2C+%E2%80%9CSlicegpt%3A+Compress+large+language+models+by+deleting+rows+and+columns%2C%E2%80%9D+arXiv+preprint+arXiv%3A2401%2E15024%2C+2024%2E)
180. Q. Zhang, S. Zuo, C. Liang, A. Bukharin, P. He, W. Chen, and T. Zhao, “Platon: Pruning large transformer models with upper confidence bound of weight importance,” in International Conference on Machine Learning. PMLR, 2022, pp. 26 809–26 823. [find](https://scholar.google.com/scholar?q=Q%2E+Zhang%2C+S%2E+Zuo%2C+C%2E+Liang%2C+A%2E+Bukharin%2C+P%2E+He%2C+W%2E+Chen%2C+and+T%2E+Zhao%2C+%E2%80%9CPlaton%3A+Pruning+large+transformer+models+with+upper+confidence+bound+of+weight+importance%2C%E2%80%9D+in+International+Conference+on+Machin)
181. M. Xia, Z. Zhong, and D. Chen, “[Structured pruning learns compact and accurate models](https://md.research.elamur.ai/free/en/papers/xia-2022-structured-pruning-learns-compact-accurate/),” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 1513–1528.
182. C. Tao, L. Hou, H. Bai, J. Wei, X. Jiang, Q. Liu, P. Luo, and N. Wong, “Structured pruning for efficient generative pre-trained language models,” in Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 10 880–10 895. [find](https://scholar.google.com/scholar?q=C%2E+Tao%2C+L%2E+Hou%2C+H%2E+Bai%2C+J%2E+Wei%2C+X%2E+Jiang%2C+Q%2E+Liu%2C+P%2E+Luo%2C+and+N%2E+Wong%2C+%E2%80%9CStructured+pruning+for+efficient+generative+pre%2Dtrained+language+models%2C%E2%80%9D+in+Findings+of+the+Association+for+Computational+Lingu)
183. X. Lu, Q. Liu, Y. Xu, A. Zhou, S. Huang, B. Zhang, J. Yan, and H. Li, “[Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models](https://md.research.elamur.ai/free/en/papers/lu-2024-not-all-experts-are-equal/),” arXiv preprint arXiv:2402.14800, 2024.
184. A. Muzio, A. Sun, and C. He, “Seer-moe: Sparse expert efficiency through regularization for mixture-of-experts,” arXiv preprint arXiv:2404.05089, 2024. — [arXiv:2404.05089](https://arxiv.org/abs/2404.05089) [find](https://scholar.google.com/scholar?q=A%2E+Muzio%2C+A%2E+Sun%2C+and+C%2E+He%2C+%E2%80%9CSeer%2Dmoe%3A+Sparse+expert+efficiency+through+regularization+for+mixture%2Dof%2Dexperts%2C%E2%80%9D+arXiv+preprint+arXiv%3A2404%2E05089%2C+2024%2E+%E2%80%94+%5BarXiv%3A2404%2E05089%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2404%2E)
185. P. Dong, L. Li, Z. Tang, X. Liu, X. Pan, Q. Wang, and X. Chu, “Pruner-zero: Evolving symbolic pruning metric from scratch for large language models,” in International Conference on Machine Learning (ICML), 2024. [find](https://scholar.google.com/scholar?q=P%2E+Dong%2C+L%2E+Li%2C+Z%2E+Tang%2C+X%2E+Liu%2C+X%2E+Pan%2C+Q%2E+Wang%2C+and+X%2E+Chu%2C+%E2%80%9CPruner%2Dzero%3A+Evolving+symbolic+pruning+metric+from+scratch+for+large+language+models%2C%E2%80%9D+in+International+Conference+on+Machine+Learning+%28I)
186. Y. Zhang, L. Zhao, M. Lin, S. Yunyun, Y. Yao, X. Han, J. Tanner, S. Liu, and R. Ji, “Dynamic sparse no training: Training-free fine-tuning for sparse llms,” in International Conference on Learning Representations (ICLR), 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Zhang%2C+L%2E+Zhao%2C+M%2E+Lin%2C+S%2E+Yunyun%2C+Y%2E+Yao%2C+X%2E+Han%2C+J%2E+Tanner%2C+S%2E+Liu%2C+and+R%2E+Ji%2C+%E2%80%9CDynamic+sparse+no+training%3A+Training%2Dfree+fine%2Dtuning+for+sparse+llms%2C%E2%80%9D+in+International+Conference+on+Learning+Rep)
187. S.-y. Liu, Z. Liu, X. Huang, P. Dong, and K.-T. Cheng, “Llm-fp4: 4-bit floating-point quantized transformers,” in The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. [find](https://scholar.google.com/scholar?q=S%2E%2Dy%2E+Liu%2C+Z%2E+Liu%2C+X%2E+Huang%2C+P%2E+Dong%2C+and+K%2E%2DT%2E+Cheng%2C+%E2%80%9CLlm%2Dfp4%3A+4%2Dbit+floating%2Dpoint+quantized+transformers%2C%E2%80%9D+in+The+2023+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+2023%2E)
188. L. Li, Q. Li, B. Zhang, and X. Chu, “Norm tweaking: High-performance low-bit quantization of large language models,” arXiv preprint arXiv:2309.02784, 2023. — [arXiv:2309.02784](https://arxiv.org/abs/2309.02784) [find](https://scholar.google.com/scholar?q=L%2E+Li%2C+Q%2E+Li%2C+B%2E+Zhang%2C+and+X%2E+Chu%2C+%E2%80%9CNorm+tweaking%3A+High%2Dperformance+low%2Dbit+quantization+of+large+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2309%2E02784%2C+2023%2E+%E2%80%94+%5BarXiv%3A2309%2E02784%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
189. T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “[Qlora: Efficient finetuning of quantized llms](https://md.research.elamur.ai/free/en/papers/dettmers-2023-qlora-efficient-finetuning-quantized-llms/),” Advances in Neural Information Processing Systems, vol. 36, 2024.
190. Y. Xu, L. Xie, X. Gu, X. Chen, H. Chang, H. Zhang, Z. Chen, X. Zhang, and Q. Tian, “Qa-lora: Quantization-aware low-rank adaptation of large language models,” arXiv preprint arXiv:2309.14717, 2023. — [arXiv:2309.14717](https://arxiv.org/abs/2309.14717) [find](https://scholar.google.com/scholar?q=Y%2E+Xu%2C+L%2E+Xie%2C+X%2E+Gu%2C+X%2E+Chen%2C+H%2E+Chang%2C+H%2E+Zhang%2C+Z%2E+Chen%2C+X%2E+Zhang%2C+and+Q%2E+Tian%2C+%E2%80%9CQa%2Dlora%3A+Quantization%2Daware+low%2Drank+adaptation+of+large+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2309%2E14717%2C+2023%2E+%E2%80%94+)
191. Y. Li, Y. Yu, C. Liang, P. He, N. Karampatziakis, W. Chen, and T. Zhao, “Loftq: Lora-fine-tuning-aware quantization for large language models,” arXiv preprint arXiv:2310.08659, 2023. — [arXiv:2310.08659](https://arxiv.org/abs/2310.08659) [find](https://scholar.google.com/scholar?q=Y%2E+Li%2C+Y%2E+Yu%2C+C%2E+Liang%2C+P%2E+He%2C+N%2E+Karampatziakis%2C+W%2E+Chen%2C+and+T%2E+Zhao%2C+%E2%80%9CLoftq%3A+Lora%2Dfine%2Dtuning%2Daware+quantization+for+large+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2310%2E08659%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E086)
192. E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,” arXiv preprint arXiv:2210.17323, 2022. [find](https://scholar.google.com/scholar?q=E%2E+Frantar%2C+S%2E+Ashkboos%2C+T%2E+Hoefler%2C+and+D%2E+Alistarh%2C+%E2%80%9CGptq%3A+Accurate+post%2Dtraining+quantization+for+generative+pre%2Dtrained+transformers%2C%E2%80%9D+arXiv+preprint+arXiv%3A2210%2E17323%2C+2022%2E)
193. G. Park, M. Kim, S. Lee, J. Kim, B. Kwon, S. J. Kwon, B. Kim, Y. Lee, D. Lee et al., “[Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models](https://md.research.elamur.ai/free/en/papers/park-2022-lut-gemm-quantized-matrix-multiplication/),” in The Twelfth International Conference on Learning Representations, 2023.
194. J. Lin, J. Tang, H. Tang, S. Yang, X. Dang, and S. Han, “Awq: Activation-aware weight quantization for llm compression and acceleration,” arXiv preprint arXiv:2306.00978, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Lin%2C+J%2E+Tang%2C+H%2E+Tang%2C+S%2E+Yang%2C+X%2E+Dang%2C+and+S%2E+Han%2C+%E2%80%9CAwq%3A+Activation%2Daware+weight+quantization+for+llm+compression+and+acceleration%2C%E2%80%9D+arXiv+preprint+arXiv%3A2306%2E00978%2C+2023%2E)
195. C. Lee, J. Jin, T. Kim, H. Kim, and E. Park, “Owq: Lessons learned from activation outliers for weight quantization in large language models,” arXiv preprint arXiv:2306.[02272, 2023](https://md.research.elamur.ai/free/en/papers/lee-2023-owq-outlier-aware-weight-quantization/).
196. T. Dettmers, R. Svirschevski, V. Egiazarian, D. Kuznedelev, E. Frantar, S. Ashkboos, A. Borzunov, T. Hoefler, and D. Alistarh, “Spqr: A sparse-quantized representation for near-lossless llm weight compression,” arXiv preprint arXiv:2306.03078, 2023. [find](https://scholar.google.com/scholar?q=T%2E+Dettmers%2C+R%2E+Svirschevski%2C+V%2E+Egiazarian%2C+D%2E+Kuznedelev%2C+E%2E+Frantar%2C+S%2E+Ashkboos%2C+A%2E+Borzunov%2C+T%2E+Hoefler%2C+and+D%2E+Alistarh%2C+%E2%80%9CSpqr%3A+A+sparse%2Dquantized+representation+for+near%2Dlossless+llm+weight+com)
197. S. Kim, C. Hooper, A. Gholami, Z. Dong, X. Li, S. Shen, M. W. Mahoney, and K. Keutzer, “[Squeezellm: Dense-and-sparse quantization](https://md.research.elamur.ai/free/en/papers/kim-2023-squeezellm-dense-sparse-quantization/),” arXiv preprint arXiv:2306.07629, 2023.
198. J. Chee, Y. Cai, V. Kuleshov, and C. De Sa, “Quip: 2-bit quantization of large language models with guarantees,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Chee%2C+Y%2E+Cai%2C+V%2E+Kuleshov%2C+and+C%2E+De+Sa%2C+%E2%80%9CQuip%3A+2%2Dbit+quantization+of+large+language+models+with+guarantees%2C%E2%80%9D+in+Thirty%2Dseventh+Conference+on+Neural+Information+Processing+Systems%2C+2023%2E)
199. Y. J. Kim, R. Henry, R. Fahim, and H. H. Awadalla, “Finequant: Unlocking efficiency with fine-grained weight-only quantization for llms,” arXiv preprint arXiv:2308.09723, 2023. [find](https://scholar.google.com/scholar?q=Y%2E+J%2E+Kim%2C+R%2E+Henry%2C+R%2E+Fahim%2C+and+H%2E+H%2E+Awadalla%2C+%E2%80%9CFinequant%3A+Unlocking+efficiency+with+fine%2Dgrained+weight%2Donly+quantization+for+llms%2C%E2%80%9D+arXiv+preprint+arXiv%3A2308%2E09723%2C+2023%2E)
200. K. Behdin, A. Acharya, A. Gupta, S. Keerthi, and R. Mazumder, “Quantease: Optimization-based quantization for language models–an efficient and intuitive algorithm,” arXiv preprint arXiv:2309.01885, 2023. [find](https://scholar.google.com/scholar?q=K%2E+Behdin%2C+A%2E+Acharya%2C+A%2E+Gupta%2C+S%2E+Keerthi%2C+and+R%2E+Mazumder%2C+%E2%80%9CQuantease%3A+Optimization%2Dbased+quantization+for+language+models%E2%80%93an+efficient+and+intuitive+algorithm%2C%E2%80%9D+arXiv+preprint+arXiv%3A2309%2E01885%2C+20)
201. S. Li, X. Ning, K. Hong, T. Liu, L. Wang, X. Li, K. Zhong, G. Dai, H. Yang, and Y. Wang, “Llm-mq: Mixed-precision quantization for efficient llm deployment,” 2023. [find](https://scholar.google.com/scholar?q=S%2E+Li%2C+X%2E+Ning%2C+K%2E+Hong%2C+T%2E+Liu%2C+L%2E+Wang%2C+X%2E+Li%2C+K%2E+Zhong%2C+G%2E+Dai%2C+H%2E+Yang%2C+and+Y%2E+Wang%2C+%E2%80%9CLlm%2Dmq%3A+Mixed%2Dprecision+quantization+for+efficient+llm+deployment%2C%E2%80%9D+2023%2E)
202. Z. Yao, R. Y. Aminabadi, M. Zhang, X. Wu, C. Li, and Y. He, “[Zeroquant: Efficient and affordable post-training quantization for large-scale transformers](https://md.research.elamur.ai/free/en/papers/yao-2022-zeroquant-efficient-affordable-post-training/),” in Advances in Neural Information Processing Systems, 2022.
203. Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Re, I. Stoica, and C. Zhang, “[Flexgen: High-throughput generative inference of large language models with a single gpu](https://md.research.elamur.ai/free/en/papers/sheng-2023-flexgen-high-throughput-generative-inference/),” 2023.
204. T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, “Llm. int8 (): 8-bit matrix multiplication for transformers at scale,” arXiv preprint arXiv:2208.07339, 2022. [find](https://scholar.google.com/scholar?q=T%2E+Dettmers%2C+M%2E+Lewis%2C+Y%2E+Belkada%2C+and+L%2E+Zettlemoyer%2C+%E2%80%9CLlm%2E+int8+%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2C%E2%80%9D+arXiv+preprint+arXiv%3A2208%2E07339%2C+2022%2E)
205. G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “Smoothquant: Accurate and efficient post-training quantization for large language models,” in International Conference on Machine Learning. PMLR, 2023, pp. 38 087–38 099. [find](https://scholar.google.com/scholar?q=G%2E+Xiao%2C+J%2E+Lin%2C+M%2E+Seznec%2C+H%2E+Wu%2C+J%2E+Demouth%2C+and+S%2E+Han%2C+%E2%80%9CSmoothquant%3A+Accurate+and+efficient+post%2Dtraining+quantization+for+large+language+models%2C%E2%80%9D+in+International+Conference+on+Machine+Learning%2E+)
206. Z. Yao, X. Wu, C. Li, S. Youn, and Y. He, “Zeroquant-v2: Exploring post-training quantization in llms from comprehensive study to low rank compensation,” arXiv preprint arXiv:2303.08302, 2023. — [arXiv:2303.08302](https://arxiv.org/abs/2303.08302) [find](https://scholar.google.com/scholar?q=Z%2E+Yao%2C+X%2E+Wu%2C+C%2E+Li%2C+S%2E+Youn%2C+and+Y%2E+He%2C+%E2%80%9CZeroquant%2Dv2%3A+Exploring+post%2Dtraining+quantization+in+llms+from+comprehensive+study+to+low+rank+compensation%2C%E2%80%9D+arXiv+preprint+arXiv%3A2303%2E08302%2C+2023%2E+%E2%80%94+%5BarXi)
207. Z. Yuan, L. Niu, J. Liu, W. Liu, X. Wang, Y. Shang, G. Sun, Q. Wu, J. Wu, and B. Wu, “Rptq: Reorder-based post-training quantization for large language models,” arXiv preprint arXiv:2304.01089, 2023. — [arXiv:2304.01089](https://arxiv.org/abs/2304.01089) [find](https://scholar.google.com/scholar?q=Z%2E+Yuan%2C+L%2E+Niu%2C+J%2E+Liu%2C+W%2E+Liu%2C+X%2E+Wang%2C+Y%2E+Shang%2C+G%2E+Sun%2C+Q%2E+Wu%2C+J%2E+Wu%2C+and+B%2E+Wu%2C+%E2%80%9CRptq%3A+Reorder%2Dbased+post%2Dtraining+quantization+for+large+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2304%2E01089%2C+2023%2E+)
208. C. Guo, J. Tang, W. Hu, J. Leng, C. Zhang, F. Yang, Y. Liu, M. Guo, and Y. Zhu, “Olive: Accelerating large language models via hardware-friendly outlier-victim pair quantization,” in Proceedings of the 50th Annual International Symposium on Computer Architecture, 2023, pp. 1–15. [find](https://scholar.google.com/scholar?q=C%2E+Guo%2C+J%2E+Tang%2C+W%2E+Hu%2C+J%2E+Leng%2C+C%2E+Zhang%2C+F%2E+Yang%2C+Y%2E+Liu%2C+M%2E+Guo%2C+and+Y%2E+Zhu%2C+%E2%80%9COlive%3A+Accelerating+large+language+models+via+hardware%2Dfriendly+outlier%2Dvictim+pair+quantization%2C%E2%80%9D+in+Proceedings+of+th)
209. X. Wu, Z. Yao, and Y. He, “Zeroquant-fp: A leap forward in llms post-training w4a8 quantization using floating-point formats,” arXiv preprint arXiv:2307.09782, 2023. [find](https://scholar.google.com/scholar?q=X%2E+Wu%2C+Z%2E+Yao%2C+and+Y%2E+He%2C+%E2%80%9CZeroquant%2Dfp%3A+A+leap+forward+in+llms+post%2Dtraining+w4a8+quantization+using+floating%2Dpoint+formats%2C%E2%80%9D+arXiv+preprint+arXiv%3A2307%2E09782%2C+2023%2E)
210. W. Shao, M. Chen, Z. Zhang, P. Xu, L. Zhao, Z. Li, K. Zhang, P. Gao, Y. Qiao, and P. Luo, “[Omniquant: Omnidirectionally calibrated quantization for large language models](https://md.research.elamur.ai/free/en/papers/shao-2023-omniquant-omnidirectionally-calibrated-quantization-large/),” in The Twelfth International Conference on Learning Representations, 2023.
211. J. Liu, R. Gong, X. Wei, Z. Dong, J. Cai, and B. Zhuang, “Qllm: Accurate and efficient low-bitwidth quantization for large language models,” in The Twelfth International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Liu%2C+R%2E+Gong%2C+X%2E+Wei%2C+Z%2E+Dong%2C+J%2E+Cai%2C+and+B%2E+Zhuang%2C+%E2%80%9CQllm%3A+Accurate+and+efficient+low%2Dbitwidth+quantization+for+large+language+models%2C%E2%80%9D+in+The+Twelfth+International+Conference+on+Learning+Represe)
212. Y. Zhao, C.-Y. Lin, K. Zhu, Z. Ye, L. Chen, S. Zheng, L. Ceze, A. Krishnamurthy, T. Chen, and B. Kasikci, “[Atom: Low-bit quantization for efficient and accurate llm serving](https://md.research.elamur.ai/free/en/papers/zhao-2023-atom-low-bit-quantization-efficient/),” arXiv preprint arXiv:2310.19102, 2023.
213. W. Huang, Y. Liu, H. Qin, Y. Li, S. Zhang, X. Liu, M. Magno, and X. Qi, “Billm: Pushing the limit of post-training quantization for llms,” 2024. [find](https://scholar.google.com/scholar?q=W%2E+Huang%2C+Y%2E+Liu%2C+H%2E+Qin%2C+Y%2E+Li%2C+S%2E+Zhang%2C+X%2E+Liu%2C+M%2E+Magno%2C+and+X%2E+Qi%2C+%E2%80%9CBillm%3A+Pushing+the+limit+of+post%2Dtraining+quantization+for+llms%2C%E2%80%9D+2024%2E)
214. S. Li, X. Ning, L. Wang, T. Liu, X. Shi, S. Yan, G. Dai, H. Yang, and Y. Wang, “Evaluating quantized large language models,” arXiv preprint arXiv:2402.18158, 2024. — [arXiv:2402.18158](https://arxiv.org/abs/2402.18158) [find](https://scholar.google.com/scholar?q=S%2E+Li%2C+X%2E+Ning%2C+L%2E+Wang%2C+T%2E+Liu%2C+X%2E+Shi%2C+S%2E+Yan%2C+G%2E+Dai%2C+H%2E+Yang%2C+and+Y%2E+Wang%2C+%E2%80%9CEvaluating+quantized+large+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2402%2E18158%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E18158%5D%28https%3A%2F%2Farxiv%2Eo)
215. Y. Ma, H. Li, X. Zheng, F. Ling, X. Xiao, R. Wang, S. Wen, F. Chao, and R. Ji, “Affinequant: Affine transformation quantization for large language models,” in International Conference on Learning Representations (ICLR), 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Ma%2C+H%2E+Li%2C+X%2E+Zheng%2C+F%2E+Ling%2C+X%2E+Xiao%2C+R%2E+Wang%2C+S%2E+Wen%2C+F%2E+Chao%2C+and+R%2E+Ji%2C+%E2%80%9CAffinequant%3A+Affine+transformation+quantization+for+large+language+models%2C%E2%80%9D+in+International+Conference+on+Learning+Repr)
216. A. Tseng, J. Chee, Q. Sun, V. Kuleshov, and C. De Sa, “Quip\#: Even better llm quantization with hadamard incoherence and lattice codebooks,” arXiv preprint arXiv:2402.[04396, 2024](https://md.research.elamur.ai/free/en/papers/tseng-2024-quip-even-better-llm-quantization/).
217. S. Ashkboos, A. Mohtashami, M. L. Croci, B. Li, M. Jaggi, D. Alistarh, T. Hoefler, and J. Hensman, “[Quarot: Outlier-free 4-bit inference in rotated llms](https://md.research.elamur.ai/free/en/papers/ashkboos-2024-quarot-outlier-free-4-bit/),” arXiv preprint arXiv:2404.00456, 2024.
218. Z. Liu, C. Zhao, I. Fedorov, B. Soran, D. Choudhary, R. Krishnamoorthi, V. Chandra, Y. Tian, and T. Blankevoort, “Spinquant–llm quantization with learned rotations,” arXiv preprint arXiv:2405.16406, 2024. — [arXiv:2405.16406](https://arxiv.org/abs/2405.16406) [find](https://scholar.google.com/scholar?q=Z%2E+Liu%2C+C%2E+Zhao%2C+I%2E+Fedorov%2C+B%2E+Soran%2C+D%2E+Choudhary%2C+R%2E+Krishnamoorthi%2C+V%2E+Chandra%2C+Y%2E+Tian%2C+and+T%2E+Blankevoort%2C+%E2%80%9CSpinquant%E2%80%93llm+quantization+with+learned+rotations%2C%E2%80%9D+arXiv+preprint+arXiv%3A2405%2E16406%2C+2)
219. C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y. S. Shao, K. Keutzer, and A. Gholami, “[Kvquant: Towards 10 million context length llm inference with kv cache quantization](https://md.research.elamur.ai/free/en/papers/hooper-2024-kvquant-towards-10-million-context/),” arXiv preprint arXiv:2401.18079, 2024.
220. Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V. Braverman, B. Chen, and X. Hu, “[Kivi: A tuning-free asymmetric 2bit quantization for kv cache](https://md.research.elamur.ai/free/en/papers/liu-2024-kivi-tuning-free-asymmetric-2bit/),” arXiv preprint arXiv:2402.02750, 2024.
221. E. Frantar and D. Alistarh, “Optimal brain compression: A framework for accurate post-training quantization and pruning,” in Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=E%2E+Frantar+and+D%2E+Alistarh%2C+%E2%80%9COptimal+brain+compression%3A+A+framework+for+accurate+post%2Dtraining+quantization+and+pruning%2C%E2%80%9D+in+Advances+in+Neural+Information+Processing+Systems%2C+2022%2E)
222. N. Vaidya, F. Oh, and N. Comly, “Optimizing inference on large language models with nvidia tensorrt-llm, now publicly available,” [Online], 2023, [https://github.com/NVIDIA/TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM). [↗](https://github.com/NVIDIA/TensorRT-LLM)
223. InternLM, “Lmdeploy,” 2024. [Online]. Available: [https://github.com/InternLM/lmdeploy](https://github.com/InternLM/lmdeploy) [↗](https://github.com/InternLM/lmdeploy)
224. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “[Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/),” arXiv preprint arXiv:2106.09685, 2021.
225. B. Hassibi, D. G. Stork, and G. J. Wolff, “Optimal brain surgeon and general network pruning,” in IEEE international conference on neural networks. IEEE, 1993, pp. 293–299. [find](https://scholar.google.com/scholar?q=B%2E+Hassibi%2C+D%2E+G%2E+Stork%2C+and+G%2E+J%2E+Wolff%2C+%E2%80%9COptimal+brain+surgeon+and+general+network+pruning%2C%E2%80%9D+in+IEEE+international+conference+on+neural+networks%2E+IEEE%2C+1993%2C+pp%2E+293%E2%80%93299%2E)
226. Y. LeCun, J. Denker, and S. Solla, “Optimal brain damage,” Advances in neural information processing systems, vol. 2, 1989. [find](https://scholar.google.com/scholar?q=Y%2E+LeCun%2C+J%2E+Denker%2C+and+S%2E+Solla%2C+%E2%80%9COptimal+brain+damage%2C%E2%80%9D+Advances+in+neural+information+processing+systems%2C+vol%2E+2%2C+1989%2E)
227. D. C. Mocanu, E. Mocanu, P. Stone, P. H. Nguyen, M. Gibescu, and A. Liotta, “Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science,” Nature communications, vol. 9, no. 1, p. 2383, 2018. [find](https://scholar.google.com/scholar?q=D%2E+C%2E+Mocanu%2C+E%2E+Mocanu%2C+P%2E+Stone%2C+P%2E+H%2E+Nguyen%2C+M%2E+Gibescu%2C+and+A%2E+Liotta%2C+%E2%80%9CScalable+training+of+artificial+neural+networks+with+adaptive+sparse+connectivity+inspired+by+network+science%2C%E2%80%9D+Nature+comm)
228. B. Zoph and Q. Le, “[Neural architecture search with reinforcement learning](https://md.research.elamur.ai/free/en/papers/zoph-2016-neural-architecture-search-reinforcement-learning/),” in International Conference on Learning Representations, 2016.
229. X. Wang, Y. Zheng, Z. Wan, and M. Zhang, “Svd-llm: Truncation-aware singular value decomposition for large language model compression,” arXiv preprint arXiv:2403.07378, 2024. — [arXiv:2403.07378](https://arxiv.org/abs/2403.07378) [find](https://scholar.google.com/scholar?q=X%2E+Wang%2C+Y%2E+Zheng%2C+Z%2E+Wan%2C+and+M%2E+Zhang%2C+%E2%80%9CSvd%2Dllm%3A+Truncation%2Daware+singular+value+decomposition+for+large+language+model+compression%2C%E2%80%9D+arXiv+preprint+arXiv%3A2403%2E07378%2C+2024%2E+%E2%80%94+%5BarXiv%3A2403%2E07378%5D%28http)
230. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “[Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/), 2022,” URL https://arxiv.org/abs/2203.02155, vol. 13, 2022. [↗](https://arxiv.org/abs/2203.02155)
231. Y. Han, G. Huang, S. Song, L. Yang, H. Wang, and Y. Wang, “Dynamic neural networks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 11, pp. 7436–7456, 2021. [find](https://scholar.google.com/scholar?q=Y%2E+Han%2C+G%2E+Huang%2C+S%2E+Song%2C+L%2E+Yang%2C+H%2E+Wang%2C+and+Y%2E+Wang%2C+%E2%80%9CDynamic+neural+networks%3A+A+survey%2C%E2%80%9D+IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%2C+vol%2E+44%2C+no%2E+11%2C+pp%2E+7436%E2%80%937456%2C+2021%2E)
232. C. Xu and J. McAuley, “A survey on dynamic neural networks for natural language processing,” in Findings of the Association for Computational Linguistics: EACL 2023, 2023, pp. 2370–2381. [find](https://scholar.google.com/scholar?q=C%2E+Xu+and+J%2E+McAuley%2C+%E2%80%9CA+survey+on+dynamic+neural+networks+for+natural+language+processing%2C%E2%80%9D+in+Findings+of+the+Association+for+Computational+Linguistics%3A+EACL+2023%2C+2023%2C+pp%2E+2370%E2%80%932381%2E)
233. X. He, I. Keivanloo, Y. Xu, X. He, B. Zeng, S. Rajagopalan, and T. Chilimbi, “Magic pyramid: Accelerating inference with early exiting and token pruning,” Image, 2023. [find](https://scholar.google.com/scholar?q=X%2E+He%2C+I%2E+Keivanloo%2C+Y%2E+Xu%2C+X%2E+He%2C+B%2E+Zeng%2C+S%2E+Rajagopalan%2C+and+T%2E+Chilimbi%2C+%E2%80%9CMagic+pyramid%3A+Accelerating+inference+with+early+exiting+and+token+pruning%2C%E2%80%9D+Image%2C+2023%2E)
234. TogetherAI, “Paving the way to efficient architectures: Stripedhyena-7b, open source models offering a glimpse into a world beyond transformers,” December 2023. [Online]. Available: [https://www.together.ai/blog/stripedhyena-7b](https://www.together.ai/blog/stripedhyena-7b) [↗](https://www.together.ai/blog/stripedhyena-7b)
235. A. Jaiswal, Z. Gan, X. Du, B. Zhang, Z. Wang, and Y. Yang, “Compressing llms: The truth is rarely pure and never simple,” arXiv preprint arXiv:2310.01382, 2023. — [arXiv:2310.01382](https://arxiv.org/abs/2310.01382) [find](https://scholar.google.com/scholar?q=A%2E+Jaiswal%2C+Z%2E+Gan%2C+X%2E+Du%2C+B%2E+Zhang%2C+Z%2E+Wang%2C+and+Y%2E+Yang%2C+%E2%80%9CCompressing+llms%3A+The+truth+is+rarely+pure+and+never+simple%2C%E2%80%9D+arXiv+preprint+arXiv%3A2310%2E01382%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E01382%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
236. Y. Leviathan, M. Kalman, and Y. Matias, “Fast inference from transformers via speculative decoding,” in International Conference on Machine Learning. PMLR, 2023, pp. 19 274–19 286. [find](https://scholar.google.com/scholar?q=Y%2E+Leviathan%2C+M%2E+Kalman%2C+and+Y%2E+Matias%2C+%E2%80%9CFast+inference+from+transformers+via+speculative+decoding%2C%E2%80%9D+in+International+Conference+on+Machine+Learning%2E+PMLR%2C+2023%2C+pp%2E+19+274%E2%80%9319+286%2E)
237. C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper, “Accelerating large language model decoding with speculative sampling,” arXiv preprint arXiv:2302.01318, 2023. [find](https://scholar.google.com/scholar?q=C%2E+Chen%2C+S%2E+Borgeaud%2C+G%2E+Irving%2C+J%2E%2DB%2E+Lespiau%2C+L%2E+Sifre%2C+and+J%2E+Jumper%2C+%E2%80%9CAccelerating+large+language+model+decoding+with+speculative+sampling%2C%E2%80%9D+arXiv+preprint+arXiv%3A2302%2E01318%2C+2023%2E)
238. Y. Zhou, K. Lyu, A. S. Rawat, A. K. Menon, A. Rostamizadeh, S. Kumar, J.-F. Kagy, and R. Agarwal, “Distillspec: Improving speculative decoding via knowledge distillation,” arXiv preprint arXiv:2310.08461, 2023. — [arXiv:2310.08461](https://arxiv.org/abs/2310.08461) [find](https://scholar.google.com/scholar?q=Y%2E+Zhou%2C+K%2E+Lyu%2C+A%2E+S%2E+Rawat%2C+A%2E+K%2E+Menon%2C+A%2E+Rostamizadeh%2C+S%2E+Kumar%2C+J%2E%2DF%2E+Kagy%2C+and+R%2E+Agarwal%2C+%E2%80%9CDistillspec%3A+Improving+speculative+decoding+via+knowledge+distillation%2C%E2%80%9D+arXiv+preprint+arXiv%3A2310%2E08)
239. J. Zhang, J. Wang, H. Li, L. Shou, K. Chen, G. Chen, and S. Mehrotra, “[Draft & verify: Lossless large language model acceleration via self-speculative decoding](https://md.research.elamur.ai/free/en/papers/zhang-2024-draft-verify-lossless-large-language/),” arXiv preprint arXiv:2309.08168, 2023.
240. X. Liu, L. Hu, P. Bailis, I. Stoica, Z. Deng, A. Cheung, and H. Zhang, “[Online speculative decoding](https://md.research.elamur.ai/free/en/papers/liu-2023-online-speculative-decoding/),” arXiv preprint arXiv:2310.07177, 2023.
241. G. Monea, A. Joulin, and E. Grave, “Pass: Parallel speculative sampling,” arXiv preprint arXiv:2311.13581, 2023. — [arXiv:2311.13581](https://arxiv.org/abs/2311.13581) [find](https://scholar.google.com/scholar?q=G%2E+Monea%2C+A%2E+Joulin%2C+and+E%2E+Grave%2C+%E2%80%9CPass%3A+Parallel+speculative+sampling%2C%E2%80%9D+arXiv+preprint+arXiv%3A2311%2E13581%2C+2023%2E+%E2%80%94+%5BarXiv%3A2311%2E13581%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E13581%29)
242. Z. He, Z. Zhong, T. Cai, J. D. Lee, and D. He, “[Rest: Retrieval-based speculative decoding](https://md.research.elamur.ai/free/en/papers/he-2023-rest-retrieval-based-speculative-decoding/),” arXiv preprint arXiv:2311.08252, 2023.
243. X. Miao, G. Oliaro, Z. Zhang, X. Cheng, Z. Wang, R. Y. Y. Wong, Z. Chen, D. Arfeen, R. Abhyankar, and Z. Jia, “Specinfer: Accelerating generative llm serving with speculative inference and token tree verification,” arXiv preprint arXiv:2305.[09781, 2023](https://md.research.elamur.ai/free/en/papers/miao-2023-specinfer-accelerating-generative-large-language/).
244. B. Spector and C. Re, “Accelerating llm inference with staged speculative decoding,” arXiv preprint arXiv:2308.04623, 2023. — [arXiv:2308.04623](https://arxiv.org/abs/2308.04623) [find](https://scholar.google.com/scholar?q=B%2E+Spector+and+C%2E+Re%2C+%E2%80%9CAccelerating+llm+inference+with+staged+speculative+decoding%2C%E2%80%9D+arXiv+preprint+arXiv%3A2308%2E04623%2C+2023%2E+%E2%80%94+%5BarXiv%3A2308%2E04623%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2308%2E04623%29)
245. Z. Chen, X. Yang, J. Lin, C. Sun, J. Huang, and K. C.-C. Chang, “Cascade speculative drafting for even faster llm inference,” arXiv preprint arXiv:2312.11462, 2023. — [arXiv:2312.11462](https://arxiv.org/abs/2312.11462) [find](https://scholar.google.com/scholar?q=Z%2E+Chen%2C+X%2E+Yang%2C+J%2E+Lin%2C+C%2E+Sun%2C+J%2E+Huang%2C+and+K%2E+C%2E%2DC%2E+Chang%2C+%E2%80%9CCascade+speculative+drafting+for+even+faster+llm+inference%2C%E2%80%9D+arXiv+preprint+arXiv%3A2312%2E11462%2C+2023%2E+%E2%80%94+%5BarXiv%3A2312%2E11462%5D%28https%3A%2F%2Farxiv%2E)
246. Y. Fu, P. Bailis, I. Stoica, and H. Zhang, “Breaking the sequential dependency of llm inference using lookahead decoding,” November 2023. [Online]. Available: [https://lmsys.org/blog/2023-11-21-lookahead-decoding/](https://lmsys.org/blog/2023-11-21-lookahead-decoding/) [↗](https://lmsys.org/blog/2023-11-21-lookahead-decoding/)
247. Y. Li, C. Zhang, and H. Zhang, “Eagle: Lossless acceleration of llm decoding by feature extrapolation,” December 2023. [Online]. Available: [https://sites.google.com/view/eagle-llm](https://sites.google.com/view/eagle-llm) [↗](https://sites.google.com/view/eagle-llm)
248. Z. Sun, A. T. Suresh, J. H. Ro, A. Beirami, H. Jain, and F. Yu, “Spectr: Fast speculative decoding via optimal transport,” arXiv preprint arXiv:2310.15141, 2023. — [arXiv:2310.15141](https://arxiv.org/abs/2310.15141) [find](https://scholar.google.com/scholar?q=Z%2E+Sun%2C+A%2E+T%2E+Suresh%2C+J%2E+H%2E+Ro%2C+A%2E+Beirami%2C+H%2E+Jain%2C+and+F%2E+Yu%2C+%E2%80%9CSpectr%3A+Fast+speculative+decoding+via+optimal+transport%2C%E2%80%9D+arXiv+preprint+arXiv%3A2310%2E15141%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E15141%5D%28https%3A%2F%2Farxiv%2Eorg)
249. F. Liu, Y. Tang, Z. Liu, Y. Ni, K. Han, and Y. Wang, “Kangaroo: Lossless self-speculative decoding via double early exiting,” arXiv preprint arXiv:2404.18911, 2024. — [arXiv:2404.18911](https://arxiv.org/abs/2404.18911) [find](https://scholar.google.com/scholar?q=F%2E+Liu%2C+Y%2E+Tang%2C+Z%2E+Liu%2C+Y%2E+Ni%2C+K%2E+Han%2C+and+Y%2E+Wang%2C+%E2%80%9CKangaroo%3A+Lossless+self%2Dspeculative+decoding+via+double+early+exiting%2C%E2%80%9D+arXiv+preprint+arXiv%3A2404%2E18911%2C+2024%2E+%E2%80%94+%5BarXiv%3A2404%2E18911%5D%28https%3A%2F%2Farxiv%2E)
250. ggerganov, “Inference of meta’s llama model (and others) in pure c/c++,” 2024. [Online]. Available: [https://github.com/ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp) [↗](https://github.com/ggerganov/llama.cpp)
251. Y. Song, Z. Mi, H. Xie, and H. Chen, “[Powerinfer: Fast large language model serving with a consumer-grade gpu](https://md.research.elamur.ai/free/en/papers/song-2023-powerinfer-fast-large-language-model/),” arXiv preprint arXiv:2312.12456, 2023.
252. J. He and J. Zhai, “Fastdecode: High-throughput gpu-efficient llm serving using heterogeneous pipelines,” arXiv preprint arXiv:2403.11421, 2024. [find](https://scholar.google.com/scholar?q=J%2E+He+and+J%2E+Zhai%2C+%E2%80%9CFastdecode%3A+High%2Dthroughput+gpu%2Defficient+llm+serving+using+heterogeneous+pipelines%2C%E2%80%9D+arXiv+preprint+arXiv%3A2403%2E11421%2C+2024%2E)
253. K. Hong, G. Dai, J. Xu, Q. Mao, X. Li, J. Liu, K. Chen, Y. Dong, and Y. Wang, “Flashdecoding++: Faster large language model inference on gpus,” 2024. [find](https://scholar.google.com/scholar?q=K%2E+Hong%2C+G%2E+Dai%2C+J%2E+Xu%2C+Q%2E+Mao%2C+X%2E+Li%2C+J%2E+Liu%2C+K%2E+Chen%2C+Y%2E+Dong%2C+and+Y%2E+Wang%2C+%E2%80%9CFlashdecoding%2B%2B%3A+Faster+large+language+model+inference+on+gpus%2C%E2%80%9D+2024%2E)
254. T. Gale, D. Narayanan, C. Young, and M. Zaharia, “Megablocks: Efficient sparse training with mixture-of-experts,” in Proceedings of Machine Learning and Systems (MLSys), 2023. [find](https://scholar.google.com/scholar?q=T%2E+Gale%2C+D%2E+Narayanan%2C+C%2E+Young%2C+and+M%2E+Zaharia%2C+%E2%80%9CMegablocks%3A+Efficient+sparse+training+with+mixture%2Dof%2Dexperts%2C%E2%80%9D+in+Proceedings+of+Machine+Learning+and+Systems+%28MLSys%29%2C+2023%2E)
255. T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré, “Flashattention: Fast and memory-efficient exact attention with io-awareness,” Advances in Neural Information Processing Systems, vol. 35, pp. 16 344–16 359, 2022. [find](https://scholar.google.com/scholar?q=T%2E+Dao%2C+D%2E+Fu%2C+S%2E+Ermon%2C+A%2E+Rudra%2C+and+C%2E+Re%CC%81%2C+%E2%80%9CFlashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2C%E2%80%9D+Advances+in+Neural+Information+Processing+Systems%2C+vol%2E+35%2C+pp%2E+16+344%E2%80%9316+)
256. T. Dao, “[Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/),” arXiv preprint arXiv:2307.08691, 2023.
257. Y. Zhai, C. Jiang, L. Wang, X. Jia, S. Zhang, Z. Chen, X. Liu, and Y. Zhu, “Bytetransformer: A high-performance transformer boosted for variable-length inputs,” in 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 2023, pp. 344–355. [find](https://scholar.google.com/scholar?q=Y%2E+Zhai%2C+C%2E+Jiang%2C+L%2E+Wang%2C+X%2E+Jia%2C+S%2E+Zhang%2C+Z%2E+Chen%2C+X%2E+Liu%2C+and+Y%2E+Zhu%2C+%E2%80%9CBytetransformer%3A+A+high%2Dperformance+transformer+boosted+for+variable%2Dlength+inputs%2C%E2%80%9D+in+2023+IEEE+International+Parallel+and)
258. R. Y. Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Rasley et al., “Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale,” in SC22: International Conference for High Performance Computing, Networking, Storag… [find](https://scholar.google.com/scholar?q=R%2E+Y%2E+Aminabadi%2C+S%2E+Rajbhandari%2C+A%2E+A%2E+Awan%2C+C%2E+Li%2C+D%2E+Li%2C+E%2E+Zheng%2C+O%2E+Ruwase%2C+S%2E+Smith%2C+M%2E+Zhang%2C+J%2E+Rasley+et+al%2E%2C+%E2%80%9CDeepspeed%2Dinference%3A+enabling+efficient+inference+of+transformer+models+at+unprec)
259. T. Dao, D. Haziza, F. Massa, and G. Sizov, “Flash-decoding for long-context inference,” [Online], 2023, [https://crfm.stanford.edu/2023/10/12/flashdecoding.html](https://crfm.stanford.edu/2023/10/12/flashdecoding.html). [↗](https://crfm.stanford.edu/2023/10/12/flashdecoding.html)
260. HuggingFace, “Transformers: State-of-the-art machine learning for pytorch, tensorflow, and jax.” [Online], 2024, [https://github.com/huggingface/transformers](https://github.com/huggingface/transformers). [↗](https://github.com/huggingface/transformers)
261. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., “[Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/),” arXiv preprint arXiv:2302.13971, 2023.
262. Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang, “[Glm: General language model pretraining with autoregressive blank infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/),” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 320–335.
263. Sensetime, “Openppl: A high-performance deep learning inference platform,” [Online], 2023, [https://openppl.ai/home](https://openppl.ai/home). [↗](https://openppl.ai/home)
264. NVIDIA, “cublas: Basic linear algebra on nvidia gpus,” [Online], 2017, [https://developer.nvidia.com/cublas](https://developer.nvidia.com/cublas). [↗](https://developer.nvidia.com/cublas)
265. ——, “Cutlass: Cuda templates for linear algebra subroutines,” [Online], 2017, [https://github.com/NVIDIA/cutlass](https://github.com/NVIDIA/cutlass). [↗](https://github.com/NVIDIA/cutlass)
266. S. Wang, “Fastgemv: High-speed gemv kernels,” [Online], 2023, [https://github.com/wangsiping97/FastGEMV](https://github.com/wangsiping97/FastGEMV). [↗](https://github.com/wangsiping97/FastGEMV)
267. P. Tillet, H. T. Kung, and D. Cox, “[Triton: an intermediate language and compiler for tiled neural network computations](https://md.research.elamur.ai/free/en/papers/tillet-2019-triton-intermediate-language-compiler-tiled/),” in Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, 2019, pp. 10–19.
268. C. Zhang et al., “Beyond the speculative game: A survey of speculative execution in large language models,” arXiv preprint arXiv:2404.14897, 2024. — [arXiv:2404.14897](https://arxiv.org/abs/2404.14897) [find](https://scholar.google.com/scholar?q=C%2E+Zhang+et+al%2E%2C+%E2%80%9CBeyond+the+speculative+game%3A+A+survey+of+speculative+execution+in+large+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2404%2E14897%2C+2024%2E+%E2%80%94+%5BarXiv%3A2404%2E14897%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2404%2E14897)
269. H. Xia et al., “[Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding](https://md.research.elamur.ai/free/en/papers/xia-2024-unlocking-efficiency-large-language-model/),” arXiv preprint arXiv:2401.07851, 2024.
270. M. Stern, N. Shazeer, and J. Uszkoreit, “Blockwise parallel decoding for deep autoregressive models,” Advances in Neural Information Processing Systems, vol. 31, 2018. [find](https://scholar.google.com/scholar?q=M%2E+Stern%2C+N%2E+Shazeer%2C+and+J%2E+Uszkoreit%2C+%E2%80%9CBlockwise+parallel+decoding+for+deep+autoregressive+models%2C%E2%80%9D+Advances+in+Neural+Information+Processing+Systems%2C+vol%2E+31%2C+2018%2E)
271. V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proceedings of the 27th international conference on machine learning (ICML-10), 2010, pp. 807–814. [find](https://scholar.google.com/scholar?q=V%2E+Nair+and+G%2E+E%2E+Hinton%2C+%E2%80%9CRectified+linear+units+improve+restricted+boltzmann+machines%2C%E2%80%9D+in+Proceedings+of+the+27th+international+conference+on+machine+learning+%28ICML%2D10%29%2C+2010%2C+pp%2E+807%E2%80%93814%2E)
272. P. Patel, E. Choukse, C. Zhang, Íñigo Goiri, A. Shah, S. Maleki, and R. Bianchini, “[Splitwise: Efficient generative llm inference using phase splitting](https://md.research.elamur.ai/free/en/papers/patel-2023-splitwise-efficient-generative-llm-inference/),” arXiv preprint arXiv:2311.18677, 2023.
273. C. Hu, H. Huang, L. Xu, X. Chen, J. Xu, S. Chen, H. Feng, C. Wang, S. Wang, Y. Bao, N. Sun, and Y. Shan, “Inference without interference: Disaggregate llm inference for mixed downstream workloads,” arXiv preprint arXiv:2401.11181, 2024. — [arXiv:2401.11181](https://arxiv.org/abs/2401.11181) [find](https://scholar.google.com/scholar?q=C%2E+Hu%2C+H%2E+Huang%2C+L%2E+Xu%2C+X%2E+Chen%2C+J%2E+Xu%2C+S%2E+Chen%2C+H%2E+Feng%2C+C%2E+Wang%2C+S%2E+Wang%2C+Y%2E+Bao%2C+N%2E+Sun%2C+and+Y%2E+Shan%2C+%E2%80%9CInference+without+interference%3A+Disaggregate+llm+inference+for+mixed+downstream+workloads%2C%E2%80%9D+ar)
274. Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang, “[Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving](https://md.research.elamur.ai/free/en/papers/zhong-2024-distserve-disaggregating-prefill-decoding-goodput/),” arXiv preprint arXiv:2401.09670, 2024.
275. X. Miao, C. Shi, J. Duan, X. Xi, D. Lin, B. Cui, and Z. Jia, “[Spotserve: Serving generative large language models on preemptible instances](https://md.research.elamur.ai/free/en/papers/miao-2023-spotserve-serving-generative-large-language/),” arXiv preprint arXiv:2311.15566, 2023.
276. B. Lin, T. Peng, C. Zhang, M. Sun, L. Li, H. Zhao, W. Xiao, Q. Xu, X. Qiu, S. Li, Z. Ji, Y. Li, and W. Lin, “Infinite-llm: Efficient llm service for long context with distattention and distributed kvcache,” arXiv preprint arXiv:2401.02669, 2024. [find](https://scholar.google.com/scholar?q=B%2E+Lin%2C+T%2E+Peng%2C+C%2E+Zhang%2C+M%2E+Sun%2C+L%2E+Li%2C+H%2E+Zhao%2C+W%2E+Xiao%2C+Q%2E+Xu%2C+X%2E+Qiu%2C+S%2E+Li%2C+Z%2E+Ji%2C+Y%2E+Li%2C+and+W%2E+Lin%2C+%E2%80%9CInfinite%2Dllm%3A+Efficient+llm+service+for+long+context+with+distattention+and+distributed+kvc)
277. G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “[Orca: A distributed serving system for transformer-based generative models](https://md.research.elamur.ai/free/en/papers/yu-2022-orca-distributed-serving-system-transformer/),” in Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation, 2022, pp. 521–538.
278. ModelTC, “Lightllm,” February 2024. [Online]. Available: [https://github.com/ModelTC/lightllm/](https://github.com/ModelTC/lightllm/) [↗](https://github.com/ModelTC/lightllm/)
279. C. Holmes, M. Tanaka, M. Wyatt, A. A. Awan, J. Rasley, S. Rajbhandari, R. Y. Aminabadi, H. Qin, A. Bakhtiari, L. Kurilenko, and Y. He, “[Deepspeed-fastgen: High-throughput text generation for llms via mii and deepspeed-inference](https://md.research.elamur.ai/free/en/papers/holmes-2024-deepspeed-fastgen-high-throughput-text/),” arXiv preprint arXiv:2401.08671, 2024.
280. B. Wu, Y. Zhong, Z. Zhang, G. Huang, X. Liu, and X. Jin, “[Fast distributed inference serving for large language models](https://md.research.elamur.ai/free/en/papers/wu-2023-fast-distributed-inference-serving-large/),” arXiv preprint arXiv:2305.05920, 2023.
281. Y. Sheng, S. Cao, D. Li, B. Zhu, Z. Li, and D. Zhuo, “Fairness in serving large language models,” arXiv preprint arXiv:2401.00588, 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Sheng%2C+S%2E+Cao%2C+D%2E+Li%2C+B%2E+Zhu%2C+Z%2E+Li%2C+and+D%2E+Zhuo%2C+%E2%80%9CFairness+in+serving+large+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2401%2E00588%2C+2024%2E)
282. A. Agrawal, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, and R. Ramjee, “[Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills](https://md.research.elamur.ai/free/en/papers/agrawal-2023-sarathi-efficient-llm-inference-piggybacking/),” arXiv preprint arXiv:2308.16369, 2023.
283. A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, A. Tumanov, , and R. Ramjee, “[Taming throughput-latency tradeoff in llm inference with sarathi-serve](https://md.research.elamur.ai/free/en/papers/agrawal-2024-taming-throughput-latency-tradeoff-llm/),” arXiv preprint arXiv:2403.02310, 2024.
284. Y. Jin, C.-F. Wu, D. Brooks, and G.-Y. Wei, “S 3 : Increasing gpu utilization during generative inference for higher throughput,” arXiv preprint arXiv:2306.06000, 2023. [find](https://scholar.google.com/scholar?q=Y%2E+Jin%2C+C%2E%2DF%2E+Wu%2C+D%2E+Brooks%2C+and+G%2E%2DY%2E+Wei%2C+%E2%80%9CS+3+%3A+Increasing+gpu+utilization+during+generative+inference+for+higher+throughput%2C%E2%80%9D+arXiv+preprint+arXiv%3A2306%2E06000%2C+2023%2E)
285. Z. Ye, “flashinfer,” March 2024. [Online]. Available: [https://github.com/flashinfer-ai/flashinfer](https://github.com/flashinfer-ai/flashinfer) [↗](https://github.com/flashinfer-ai/flashinfer)
286. NVIDIA, “Fastertransformer: About transformer related optimization, including bert, gpt,” [Online], 2017, [https://github.com/NVIDIA/FasterTransformer](https://github.com/NVIDIA/FasterTransformer). [↗](https://github.com/NVIDIA/FasterTransformer)
287. H. Oh, K. Kim, J. Kim, S. Kim, J. Lee, D.-s. Chang, and J. Seo, “Exegpt: Constraint-aware resource scheduling for llm inference,” in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, 2024, pp. 369–384. [find](https://scholar.google.com/scholar?q=H%2E+Oh%2C+K%2E+Kim%2C+J%2E+Kim%2C+S%2E+Kim%2C+J%2E+Lee%2C+D%2E%2Ds%2E+Chang%2C+and+J%2E+Seo%2C+%E2%80%9CExegpt%3A+Constraint%2Daware+resource+scheduling+for+llm+inference%2C%E2%80%9D+in+Proceedings+of+the+29th+ACM+International+Conference+on+Architectur)
288. B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y. Li, and W. Lin, “[Llumnix: Dynamic scheduling for large language model serving](https://md.research.elamur.ai/free/en/papers/sun-2024-llumnix-dynamic-scheduling-large-language/),” arXiv preprint arXiv:2406.03243, 2024.
289. B. Wu, S. Liu, Y. Zhong, P. Sun, X. Liu, and X. Jin, “[Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism](https://md.research.elamur.ai/free/en/papers/wu-2024-loongserve-efficiently-serving-long-context/),” arXiv preprint arXiv:2404.09526, 2024.
290. B. Li, S. Pandey, H. Fang, Y. Lyv, J. Li, J. Chen, M. Xie, L. Wan, H. Liu, and C. Ding, “Ftrans: Energy-efficient acceleration of transformers using fpga,” arXiv preprint arXiv:2007.08563, 2020. [find](https://scholar.google.com/scholar?q=B%2E+Li%2C+S%2E+Pandey%2C+H%2E+Fang%2C+Y%2E+Lyv%2C+J%2E+Li%2C+J%2E+Chen%2C+M%2E+Xie%2C+L%2E+Wan%2C+H%2E+Liu%2C+and+C%2E+Ding%2C+%E2%80%9CFtrans%3A+Energy%2Defficient+acceleration+of+transformers+using+fpga%2C%E2%80%9D+arXiv+preprint+arXiv%3A2007%2E08563%2C+2020%2E)
291. T. J. Ham, Y. Lee, S. H. Seo, S. Kim, H. Choi, S. J. Jun, and J. W. Lee, “Elsa: Hardware-software co-design for efficient, lightweight self-attention mechanism in neural networks,” in ACM/IEEE 48th Annual International Symposium on Computer Architecture, 2021, pp. 692–705. [find](https://scholar.google.com/scholar?q=T%2E+J%2E+Ham%2C+Y%2E+Lee%2C+S%2E+H%2E+Seo%2C+S%2E+Kim%2C+H%2E+Choi%2C+S%2E+J%2E+Jun%2C+and+J%2E+W%2E+Lee%2C+%E2%80%9CElsa%3A+Hardware%2Dsoftware+co%2Ddesign+for+efficient%2C+lightweight+self%2Dattention+mechanism+in+neural+networks%2C%E2%80%9D+in+ACM%2FIEEE+48th+An)
292. H. Fan, T. Chau, S. I. Venieris, R. Lee, A. Kouris, W. Luk, N. D. Lane, and M. S. Abdelfattah, “Adaptable butterfly accelerator for attention-based nns via hardware and algorithm co-design,” in IEEE/ACM International Symposium on Microarchitecture, 2022, pp. 599–615. [find](https://scholar.google.com/scholar?q=H%2E+Fan%2C+T%2E+Chau%2C+S%2E+I%2E+Venieris%2C+R%2E+Lee%2C+A%2E+Kouris%2C+W%2E+Luk%2C+N%2E+D%2E+Lane%2C+and+M%2E+S%2E+Abdelfattah%2C+%E2%80%9CAdaptable+butterfly+accelerator+for+attention%2Dbased+nns+via+hardware+and+algorithm+co%2Ddesign%2C%E2%80%9D+in+IEEE%2FA)
293. Y. Qin, Y. Wang, D. Deng, Z. Zhao, X. Yang, L. Liu, S. Wei, Y. Hu, and S. Yin, “Fact: Ffn-attention co-optimized transformer architecture with eager correlation prediction,” in Proceedings of the 50th Annual International Symposium on Computer Architecture, 2023, pp. 1–14. [find](https://scholar.google.com/scholar?q=Y%2E+Qin%2C+Y%2E+Wang%2C+D%2E+Deng%2C+Z%2E+Zhao%2C+X%2E+Yang%2C+L%2E+Liu%2C+S%2E+Wei%2C+Y%2E+Hu%2C+and+S%2E+Yin%2C+%E2%80%9CFact%3A+Ffn%2Dattention+co%2Doptimized+transformer+architecture+with+eager+correlation+prediction%2C%E2%80%9D+in+Proceedings+of+the+50th)
294. H. Chen, J. Zhang, Y. Du, S. Xiang, Z. Yue, N. Zhang, Y. Cai, and Z. Zhang, “Understanding the potential of fpga-based spatial acceleration for large language model inference,” arXiv preprint arXiv:2312.15159, 2023. [find](https://scholar.google.com/scholar?q=H%2E+Chen%2C+J%2E+Zhang%2C+Y%2E+Du%2C+S%2E+Xiang%2C+Z%2E+Yue%2C+N%2E+Zhang%2C+Y%2E+Cai%2C+and+Z%2E+Zhang%2C+%E2%80%9CUnderstanding+the+potential+of+fpga%2Dbased+spatial+acceleration+for+large+language+model+inference%2C%E2%80%9D+arXiv+preprint+arXiv%3A23)
295. S. Hong, S. Moon, J. Kim, S. Lee, M. Kim, D. Lee, and J.-Y. Kim, “Dfx: A low-latency multi-fpga appliance for accelerating transformer-based text generation,” in IEEE Hot Chips 34 Symposium, 2022. [find](https://scholar.google.com/scholar?q=S%2E+Hong%2C+S%2E+Moon%2C+J%2E+Kim%2C+S%2E+Lee%2C+M%2E+Kim%2C+D%2E+Lee%2C+and+J%2E%2DY%2E+Kim%2C+%E2%80%9CDfx%3A+A+low%2Dlatency+multi%2Dfpga+appliance+for+accelerating+transformer%2Dbased+text+generation%2C%E2%80%9D+in+IEEE+Hot+Chips+34+Symposium%2C+2022%2E)
296. S. Zeng, J. Liu, G. Dai, X. Yang, T. Fu, H. Wang, W. Ma, H. Sun, S. Li, Z. Huang et al., “Flightllm: Efficient large language model inference with a complete mapping flow on fpga,” arXiv preprint arXiv:2401.03868, 2024. [find](https://scholar.google.com/scholar?q=S%2E+Zeng%2C+J%2E+Liu%2C+G%2E+Dai%2C+X%2E+Yang%2C+T%2E+Fu%2C+H%2E+Wang%2C+W%2E+Ma%2C+H%2E+Sun%2C+S%2E+Li%2C+Z%2E+Huang+et+al%2E%2C+%E2%80%9CFlightllm%3A+Efficient+large+language+model+inference+with+a+complete+mapping+flow+on+fpga%2C%E2%80%9D+arXiv+preprint+arXi)
297. S. teams, “Sharegpt,” 2023. [Online]. Available: [https://sharegpt.com/](https://sharegpt.com/) [↗](https://sharegpt.com/)
298. J. Xie, Z. Chen, R. Zhang, X. Wan, and G. Li, “Large multimodal agents: A survey,” arXiv preprint arXiv:2402.15116, 2024. — [arXiv:2402.15116](https://arxiv.org/abs/2402.15116) [find](https://scholar.google.com/scholar?q=J%2E+Xie%2C+Z%2E+Chen%2C+R%2E+Zhang%2C+X%2E+Wan%2C+and+G%2E+Li%2C+%E2%80%9CLarge+multimodal+agents%3A+A+survey%2C%E2%80%9D+arXiv+preprint+arXiv%3A2402%2E15116%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E15116%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E15116%29)
299. I. Lee, N. Jiang, and T. Berg-Kirkpatrick, “Exploring the relationship between model architecture and in-context learning ability,” arXiv preprint arXiv:2310.08049, 2023. — [arXiv:2310.08049](https://arxiv.org/abs/2310.08049) [find](https://scholar.google.com/scholar?q=I%2E+Lee%2C+N%2E+Jiang%2C+and+T%2E+Berg%2DKirkpatrick%2C+%E2%80%9CExploring+the+relationship+between+model+architecture+and+in%2Dcontext+learning+ability%2C%E2%80%9D+arXiv+preprint+arXiv%3A2310%2E08049%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E08049%5D%28https%3A%2F%2F)
300. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff et al., “Pythia: A suite for analyzing large language models across training and scaling,” in International Conference on Machine Learning. PMLR, 2023, pp. 2397–2430. [find](https://scholar.google.com/scholar?q=S%2E+Biderman%2C+H%2E+Schoelkopf%2C+Q%2E+G%2E+Anthony%2C+H%2E+Bradley%2C+K%2E+O%E2%80%99Brien%2C+E%2E+Hallahan%2C+M%2E+A%2E+Khan%2C+S%2E+Purohit%2C+U%2E+S%2E+Prashanth%2C+E%2E+Raff+et+al%2E%2C+%E2%80%9CPythia%3A+A+suite+for+analyzing+large+language+models+across+tra)
301. J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang et al., “[Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/),” arXiv preprint arXiv:2309.16609, 2023.
302. Y. Tang, F. Liu, Y. Ni, Y. Tian, Z. Bai, Y.-Q. Hu, S. Liu, S. Jui, K. Han, and Y. Wang, “Rethinking optimization and architecture for tiny language models,” arXiv preprint arXiv:2402.[02791, 2024](https://md.research.elamur.ai/free/en/papers/tang-2024-pangu-pi-pro-rethinking/).
303. Y. Li, S. Bubeck, R. Eldan, A. Del Giorno, S. Gunasekar, and Y. T. Lee, “[Textbooks are all you need ii: phi-1.5 technical report](https://md.research.elamur.ai/free/en/papers/li-2023-textbooks-all-need-ii-phi/),” arXiv preprint arXiv:2309.05463, 2023.
304. S. Gunasekar, Y. Zhang, J. Aneja, C. C. T. Mendes, A. Del Giorno, S. Gopi, M. Javaheripi, P. Kauffmann, G. de Rosa, O. Saarikivi et al., “Textbooks are all you need,” arXiv preprint arXiv:2306.11644, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Gunasekar%2C+Y%2E+Zhang%2C+J%2E+Aneja%2C+C%2E+C%2E+T%2E+Mendes%2C+A%2E+Del+Giorno%2C+S%2E+Gopi%2C+M%2E+Javaheripi%2C+P%2E+Kauffmann%2C+G%2E+de+Rosa%2C+O%2E+Saarikivi+et+al%2E%2C+%E2%80%9CTextbooks+are+all+you+need%2C%E2%80%9D+arXiv+preprint+arXiv%3A2306%2E11644%2C+)
305. P. Zhang, G. Zeng, T. Wang, and W. Lu, “[Tinyllama: An open-source small language model](https://md.research.elamur.ai/free/en/papers/zhang-2024-tinyllama-open-source-small-language/),” arXiv preprint arXiv:2401.02385, 2024.
306. C. Zhang, D. Song, Z. Ye, and Y. Gao, “[Towards the law of capacity gap in distilling language models](https://md.research.elamur.ai/free/en/papers/zhang-2025-law-capacity-gap-distilling-language/),” arXiv preprint arXiv:2311.07052, 2023.
307. X. Geng and H. Liu, “Openllama: An open reproduction of llama,” May 2023. [Online]. Available: [https://github.com/openlm-research/open](https://github.com/openlm-research/open) llama [↗](https://github.com/openlm-research/open)
308. M. Bellagente, J. Tow, D. Mahan, D. Phung, M. Zhuravinskyi, R. Adithyan, J. Baicoianu, B. Brooks, N. Cooper, A. Datta et al., “Stable lm 2 1.6 b technical report,” arXiv preprint arXiv:2402.[17834, 2024](https://md.research.elamur.ai/free/en/papers/bellagente-2024-stable-lm-2-1-6b/).
309. “Minicpm: Unveiling the potential of end-side large language models,” 2024. [find](https://scholar.google.com/scholar?q=%E2%80%9CMinicpm%3A+Unveiling+the+potential+of+end%2Dside+large+language+models%2C%E2%80%9D+2024%2E)
310. Z. Liu, C. Zhao, F. Iandola, C. Lai, Y. Tian, I. Fedorov, Y. Xiong, E. Chang, Y. Shi, R. Krishnamoorthi et al., “Mobilellm: Optimizing sub-billion parameter language models for on-device use cases,” arXiv preprint arXiv:2402.14905, 2024. [find](https://scholar.google.com/scholar?q=Z%2E+Liu%2C+C%2E+Zhao%2C+F%2E+Iandola%2C+C%2E+Lai%2C+Y%2E+Tian%2C+I%2E+Fedorov%2C+Y%2E+Xiong%2C+E%2E+Chang%2C+Y%2E+Shi%2C+R%2E+Krishnamoorthi+et+al%2E%2C+%E2%80%9CMobilellm%3A+Optimizing+sub%2Dbillion+parameter+language+models+for+on%2Ddevice+use+cases%2C%E2%80%9D+a)
311. M. team, “MLC-LLM,” 2023. [Online]. Available: [https://github.com/mlc-ai/mlc-llm](https://github.com/mlc-ai/mlc-llm) [↗](https://github.com/mlc-ai/mlc-llm)
312. Y. Yao, J. Duan, K. Xu, Y. Cai, Z. Sun, and Y. Zhang, “[A survey on large language model (llm) security and privacy: The good, the bad, and the ugly](https://md.research.elamur.ai/free/en/papers/yao-2023-survey-large-language-model-llm/),” High-Confidence Computing, p. 100211, 2024.
313. Y. Li, H. Wen, W. Wang, X. Li, Y. Yuan, G. Liu, J. Liu, W. Xu, X. Wang, Y. Sun et al., “[Personal llm agents: Insights and survey about the capability, efficiency and security](https://md.research.elamur.ai/free/en/papers/li-2024-personal-llm-agents-insights-survey/),” arXiv preprint arXiv:2401.05459, 2024.

*Updated 2 November 2026*