---
title: "Draft & Verify — Zhang 2024"
description: "Autoregressive generation in large language models is slow."
authors: ["Jun Zhang", "Jue Wang", "Huan Li", "Lidan Shou", "Ke Chen", "Gang Chen", "Sharad Mehrotra"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.607/"
doi: "10.18653/v1/2024.acl-long.607"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2024-draft-verify-lossless-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2024-draft-verify-lossless-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/) / Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding

# Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding

**Authors:** [Jun Zhang](https://md.research.elamur.ai/free/en/authors/jun-zhang/), [Jue Wang](https://md.research.elamur.ai/free/en/authors/jue-wang/), Huan Li, Lidan Shou, Ke Chen, [Gang Chen](https://md.research.elamur.ai/free/en/authors/gang-chen/), Sharad Mehrotra · **Year:** 2024 · **Venue:** Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.607/) · [PDF](https://aclanthology.org/2024.acl-long.607.pdf) · [DOI 10.18653/v1/2024.acl-long.607](https://doi.org/10.18653/v1/2024.acl-long.607) · [OpenAlex](https://openalex.org/W4402672007)

**Citations:** 40 citations (OpenAlex, 2026-10-14) · [106](https://md.research.elamur.ai/free/en/papers/zhang-2024-draft-verify-lossless-large-language/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Autoregressive generation in large language models is slow. Each new token needs a full, memory-bandwidth-bound pass through the network. Speculative decoding can help, but it usually depends on a separate draft model, which costs extra GPU memory and is hard to obtain for fine-tuned models such as LLaMA-2-Chat or CodeLLaMA. The authors propose self-speculative decoding, which uses one model for both roles. A cheap drafting pass skips selected intermediate attention and MLP layers to propose tokens quickly. The full model then checks all proposed tokens in a single forward pass, so the final output is exactly what ordinary decoding would produce.

Two components make this work:
- **Layer selection:** Bayesian optimization, run offline, picks which layers to skip by minimizing average per-token latency.
- **Adaptive draft-exiting:** drafting stops once the drafter's confidence drops below a threshold.

The method needs no training and no additional memory. Experiments on summarization and code generation with LLaMA-2 variants show end-to-end speedups of up to 1.99×, making it a practical drop-in accelerator.

## Key points
- Tested models include LLaMA-2-13B, LLaMA-2-13B-Chat, CodeLLaMA-13B and LLaMA-2-70B, with a best speedup of 1.99×. Code is released at github.com/dilab-zju/self-speculative-decoding.
- On LLaMA-2-13B with CNN/DM, average per-token latency fell from 56.3 ms (autoregressive) to 36.8 ms. Drafting takes most of the remaining time, while the threshold update costs under a microsecond.
- Varying the target acceptance rate α moved the speedup from 1.25× to a peak of 1.53×. ROUGE-2 stayed at roughly 0.108 throughout, and results were stable once α exceeded 0.8.
- The optimizer skipped attention layers more often than MLP layers, and the skipped layers clustered in the second half of the network.
- The layer search takes several hours but runs only once per model. Skipping too many layers without fine-tuning reduces the acceptance rate and therefore limits the achievable speedup.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2024draft,
  title = {{Draft \& Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding}},
  author = {Zhang, Jun and Wang, Jue and Li, Huan and Shou, Lidan and Chen, Ke and Chen, Gang and Mehrotra, Sharad},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.607},
  url = {https://doi.org/10.18653/v1/2024.acl-long.607}
}
```

- [← previous Measuring short-form factuality in large language models](https://md.research.elamur.ai/free/en/papers/wei-2024-measuring-short-form-factuality-large/)
- [next → How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition](https://md.research.elamur.ai/free/en/papers/dong-2024-how-abilities-large-language-models/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/)

## References

66 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. 2022. Deepspeed-inference: Enabling efficient inference of transformer models at unprecedented scale. In 2022 SC22: International Conference fo… [find](https://scholar.google.com/scholar?q=Reza+Yazdani+Aminabadi%2C+Samyam+Rajbhandari%2C+Ammar+Ahmad+Awan%2C+Cheng+Li%2C+Du+Li%2C+Elton+Zheng%2C+Olatunji+Ruwase%2C+Shaden+Smith%2C+Minjia+Zhang%2C+Jeff+Rasley%2C+et+al%2E+2022%2E+Deepspeed%2Dinference%3A+Enabling+efficie)
2. Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff, and Dan Roth. 2022. Rethinking the role of scale for in-context learning: An interpretability-based case study at 66 billion scale. arXiv preprint arXiv:2212.09095. [find](https://scholar.google.com/scholar?q=Hritik+Bansal%2C+Karthik+Gopalakrishnan%2C+Saket+Dingliwal%2C+Sravan+Bodapati%2C+Katrin+Kirchhoff%2C+and+Dan+Roth%2E+2022%2E+Rethinking+the+role+of+scale+for+in%2Dcontext+learning%3A+An+interpretability%2Dbased+case+stud)
3. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. [find](https://scholar.google.com/scholar?q=Rishi+Bommasani%2C+Drew+A+Hudson%2C+Ehsan+Adeli%2C+Russ+Altman%2C+Simran+Arora%2C+Sydney+von+Arx%2C+Michael+S+Bernstein%2C+Jeannette+Bohg%2C+Antoine+Bosselut%2C+Emma+Brunskill%2C+et+al%2E+2021%2E+On+the+opportunities+and+ris) [our card](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+2020%2E+Language+models+are+few%2Dshot+lear)
5. F Warren Burton. 1985. Speculative computation, parallelism, and functional programming. IEEE Transactions on Computers, 100(12):1190–1193. [find](https://scholar.google.com/scholar?q=F+Warren+Burton%2E+1985%2E+Speculative+computation%2C+parallelism%2C+and+functional+programming%2E+IEEE+Transactions+on+Computers%2C+100%2812%29%3A1190%E2%80%931193%2E)
6. Stephanie CY Chan, Adam Santoro, Andrew Kyle Lampinen, Jane X Wang, Aaditya K Singh, Pierre Harvey Richemond, James McClelland, and Felix Hill. 2022. Data distributional properties drive emergent in-context learning in transformers. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Stephanie+CY+Chan%2C+Adam+Santoro%2C+Andrew+Kyle+Lampinen%2C+Jane+X+Wang%2C+Aaditya+K+Singh%2C+Pierre+Harvey+Richemond%2C+James+McClelland%2C+and+Felix+Hill%2E+2022%2E+Data+distributional+properties+drive+emergent+in%2Dc)
7. Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. [find](https://scholar.google.com/scholar?q=Charlie+Chen%2C+Sebastian+Borgeaud%2C+Geoffrey+Irving%2C+Jean%2DBaptiste+Lespiau%2C+Laurent+Sifre%2C+and+John+Jumper%2E+2023%2E+Accelerating+large+language+model+decoding+with+speculative+sampling%2E+arXiv+preprint+arX)
8. Jang Hyun Cho and Bharath Hariharan. 2019. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4794–4802. [find](https://scholar.google.com/scholar?q=Jang+Hyun+Cho+and+Bharath+Hariharan%2E+2019%2E+On+the+efficacy+of+knowledge+distillation%2E+In+Proceedings+of+the+IEEE%2FCVF+international+conference+on+computer+vision%2C+pages+4794%E2%80%934802%2E)
9. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. [find](https://scholar.google.com/scholar?q=Karl+Cobbe%2C+Vineet+Kosaraju%2C+Mohammad+Bavarian%2C+Mark+Chen%2C+Heewoo+Jun%2C+Lukasz+Kaiser%2C+Matthias+Plappert%2C+Jerry+Tworek%2C+Jacob+Hilton%2C+Reiichiro+Nakano%2C+Christopher+Hesse%2C+and+John+Schulman%2E+2021%2E+Train) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
10. Tri Dao. 2023. FlashAttention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691. [find](https://scholar.google.com/scholar?q=Tri+Dao%2E+2023%2E+FlashAttention%2D2%3A+Faster+attention+with+better+parallelism+and+work+partitioning%2E+arXiv+preprint+arXiv%3A2307%2E08691%2E)
11. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Mike+Lewis%2C+Younes+Belkada%2C+and+Luke+Zettlemoyer%2E+2022%2E+Llm%2E+int8+%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2E+arXiv+preprint+arXiv%3A2208%2E07339%2E)
12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
13. Jiarui Fang, Yang Yu, Chengduo Zhao, and Jie Zhou. 2021. Turbotransformers: an efficient gpu serving system for transformer models. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 389–402. [find](https://scholar.google.com/scholar?q=Jiarui+Fang%2C+Yang+Yu%2C+Chengduo+Zhao%2C+and+Jie+Zhou%2E+2021%2E+Turbotransformers%3A+an+efficient+gpu+serving+system+for+transformer+models%2E+In+Proceedings+of+the+26th+ACM+SIGPLAN+Symposium+on+Principles+and+P)
14. Elias Frantar and Dan Alistarh. 2023. Massive language models can be accurately pruned in one-shot. arXiv preprint arXiv:2301.00774. [find](https://scholar.google.com/scholar?q=Elias+Frantar+and+Dan+Alistarh%2E+2023%2E+Massive+language+models+can+be+accurately+pruned+in+one%2Dshot%2E+arXiv+preprint+arXiv%3A2301%2E00774%2E)
15. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. [find](https://scholar.google.com/scholar?q=Elias+Frantar%2C+Saleh+Ashkboos%2C+Torsten+Hoefler%2C+and+Dan+Alistarh%2E+2022%2E+Gptq%3A+Accurate+post%2Dtraining+quantization+for+generative+pre%2Dtrained+transformers%2E+arXiv+preprint+arXiv%3A2210%2E17323%2E)
16. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+Shawn+Presser%2C+and+Connor+Leahy%2E+2020%2E+The+pile%3A+An+800gb+datas)
17. Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149. [find](https://scholar.google.com/scholar?q=Song+Han%2C+Huizi+Mao%2C+and+William+J+Dally%2E+2015%2E+Deep+compression%3A+Compressing+deep+neural+networks+with+pruning%2C+trained+quantization+and+huffman+coding%2E+arXiv+preprint+arXiv%3A1510%2E00149%2E)
18. Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. 2019. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4340–4349. [find](https://scholar.google.com/scholar?q=Yang+He%2C+Ping+Liu%2C+Ziwei+Wang%2C+Zhilan+Hu%2C+and+Yi+Yang%2E+2019%2E+Filter+pruning+via+geometric+median+for+deep+convolutional+neural+networks+acceleration%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+compu)
19. John L Hennessy and David A Patterson. 2011. Computer architecture: a quantitative approach. Elsevier. [find](https://scholar.google.com/scholar?q=John+L+Hennessy+and+David+A+Patterson%2E+2011%2E+Computer+architecture%3A+a+quantitative+approach%2E+Elsevier%2E)
20. Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2(7). [find](https://scholar.google.com/scholar?q=Geoffrey+Hinton%2C+Oriol+Vinyals%2C+Jeff+Dean%2C+et+al%2E+2015%2E+Distilling+the+knowledge+in+a+neural+network%2E+arXiv+preprint+arXiv%3A1503%2E02531%2C+2%287%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/)
21. Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. 2021. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. J. Mach. Learn. Res., 22(241):1–124. [find](https://scholar.google.com/scholar?q=Torsten+Hoefler%2C+Dan+Alistarh%2C+Tal+Ben%2DNun%2C+Nikoli+Dryden%2C+and+Alexandra+Peste%2E+2021%2E+Sparsity+in+deep+learning%3A+Pruning+and+growth+for+efficient+inference+and+training+in+neural+networks%2E+J%2E+Mach%2E+Le)
22. Xiaonan Li, Yunfan Shao, Tianxiang Sun, Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2021. Accelerating BERT inference for sequence labeling via early-exit. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xiaonan+Li%2C+Yunfan+Shao%2C+Tianxiang+Sun%2C+Hang+Yan%2C+Xipeng+Qiu%2C+and+Xuanjing+Huang%2E+2021%2E+Accelerating+BERT+inference+for+sequence+labeling+via+early%2Dexit%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+t)
23. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+et+al%2E+2022%2E+Holistic+evaluation+of+language+models%2E+)
24. Kaiyuan Liao, Yi Zhang, Xuancheng Ren, Qi Su, Xu Sun, and Bin He. 2021. A global past-future early exit method for accelerating inference of pre-trained language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua… [find](https://scholar.google.com/scholar?q=Kaiyuan+Liao%2C+Yi+Zhang%2C+Xuancheng+Ren%2C+Qi+Su%2C+Xu+Sun%2C+and+Bin+He%2E+2021%2E+A+global+past%2Dfuture+early+exit+method+for+accelerating+inference+of+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+2021+Con)
25. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+Rouge%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+summarization+branches+out%2C+pages+74%E2%80%9381%2E)
26. Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. 2020. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems, 33. [find](https://scholar.google.com/scholar?q=Lu+Hou%2C+Zhiqi+Huang%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Xiao+Chen%2C+and+Qun+Liu%2E+2020%2E+Dynabert%3A+Dynamic+bert+with+adaptive+width+and+depth%2E+Advances+in+Neural+Information+Processing+Systems%2C+33%2E)
27. Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and Qi Ju. 2020. Fastbert: a self-distilling BERT with adaptive inference time. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Weijie+Liu%2C+Peng+Zhou%2C+Zhiruo+Wang%2C+Zhe+Zhao%2C+Haotang+Deng%2C+and+Qi+Ju%2E+2020%2E+Fastbert%3A+a+self%2Ddistilling+BERT+with+adaptive+inference+time%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association) [our card](https://md.research.elamur.ai/free/en/papers/liu-2020-fastbert-self-distilling-bert-adaptive/)
28. HuggingFace. 2023. Text generation inference: Fast optimized inference for LLMs. [https://github.com/](https://github.com/) huggingface/text-generation-inference. [↗](https://github.com/)
29. Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. 2018. Rethinking the value of network pruning. arXiv preprint arXiv:1810.05270. — [arXiv:1810.05270](https://arxiv.org/abs/1810.05270) [find](https://scholar.google.com/scholar?q=Zhuang+Liu%2C+Mingjie+Sun%2C+Tinghui+Zhou%2C+Gao+Huang%2C+and+Trevor+Darrell%2E+2018%2E+Rethinking+the+value+of+network+pruning%2E+arXiv+preprint+arXiv%3A1810%2E05270%2E+%E2%80%94+%5BarXiv%3A1810%2E05270%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1810%2E05)
30. Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recogni… [find](https://scholar.google.com/scholar?q=Benoit+Jacob%2C+Skirmantas+Kligys%2C+Bo+Chen%2C+Menglong+Zhu%2C+Matthew+Tang%2C+Andrew+Howard%2C+Hartwig+Adam%2C+and+Dmitry+Kalenichenko%2E+2018%2E+Quantization+and+training+of+neural+networks+for+efficient+integer%2Dari)
31. Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. 2023. Deja vu: Contextual sparsity for efficient llms at inference time. In International Conference on Machine Learning, pages 22137–22176. PMLR. [find](https://scholar.google.com/scholar?q=Zichang+Liu%2C+Jue+Wang%2C+Tri+Dao%2C+Tianyi+Zhou%2C+Binhang+Yuan%2C+Zhao+Song%2C+Anshumali+Shrivastava%2C+Ce+Zhang%2C+Yuandong+Tian%2C+Christopher+Re%2C+et+al%2E+2023%2E+Deja+vu%3A+Contextual+sparsity+for+efficient+llms+at+in)
32. Donald R Jones, Matthias Schonlau, and William J Welch. 1998. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13:455–492. [find](https://scholar.google.com/scholar?q=Donald+R+Jones%2C+Matthias+Schonlau%2C+and+William+J+Welch%2E+1998%2E+Efficient+global+optimization+of+expensive+black%2Dbox+functions%2E+Journal+of+Global+optimization%2C+13%3A455%E2%80%93492%2E)
33. Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. 2019. Spoc: Search-based pseudocode to code. Advances in Neural Information Processing Systems, 32. [find](https://scholar.google.com/scholar?q=Sumith+Kulal%2C+Panupong+Pasupat%2C+Kartik+Chandra%2C+Mina+Lee%2C+Oded+Padon%2C+Alex+Aiken%2C+and+Percy+S+Liang%2E+2019%2E+Spoc%3A+Search%2Dbased+pseudocode+to+code%2E+Advances+in+Neural+Information+Processing+Systems%2C+32%2E)
34. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles. [find](https://scholar.google.com/scholar?q=Woosuk+Kwon%2C+Zhuohan+Li%2C+Siyuan+Zhuang%2C+Ying+Sheng%2C+Lianmin+Zheng%2C+Cody+Hao+Yu%2C+Joseph+E%2E+Gonzalez%2C+Hao+Zhang%2C+and+Ion+Stoica%2E+2023%2E+Efficient+memory+management+for+large+language+model+serving+with+p)
35. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR. [find](https://scholar.google.com/scholar?q=Yaniv+Leviathan%2C+Matan+Kalman%2C+and+Yossi+Matias%2E+2023%2E+Fast+inference+from+transformers+via+speculative+decoding%2E+In+International+Conference+on+Machine+Learning%2C+pages+19274%E2%80%9319286%2E+PMLR%2E)
36. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Xinxi+Lyu%2C+Ari+Holtzman%2C+Mikel+Artetxe%2C+Mike+Lewis%2C+Hannaneh+Hajishirzi%2C+and+Luke+Zettlemoyer%2E+2022%2E+Rethinking+the+role+of+demonstrations%3A+What+makes+in%2Dcontext+learning+work%3F+arXiv+prepri) [our card](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/)
37. Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. 2016. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440. — [arXiv:1611.06440](https://arxiv.org/abs/1611.06440) [find](https://scholar.google.com/scholar?q=Pavlo+Molchanov%2C+Stephen+Tyree%2C+Tero+Karras%2C+Timo+Aila%2C+and+Jan+Kautz%2E+2016%2E+Pruning+convolutional+neural+networks+for+resource+efficient+inference%2E+arXiv+preprint+arXiv%3A1611%2E06440%2E+%E2%80%94+%5BarXiv%3A1611%2E0644)
38. Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. 2019. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1325–1334. [find](https://scholar.google.com/scholar?q=Markus+Nagel%2C+Mart+van+Baalen%2C+Tijmen+Blankevoort%2C+and+Max+Welling%2E+2019%2E+Data%2Dfree+quantization+through+weight+equalization+and+bias+correction%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Conferenc)
39. NVIDIA. Fastertransformer. [https://github.com/](https://github.com/) NVIDIA/FasterTransformer. [↗](https://github.com/)
40. Gunho Park, Baeseong Park, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. 2022. nuqmm: Quantized matmul for efficient inference of large-scale generative language models. arXiv preprint arXiv:2206.09557. [find](https://scholar.google.com/scholar?q=Gunho+Park%2C+Baeseong+Park%2C+Se+Jung+Kwon%2C+Byeongwook+Kim%2C+Youngjoo+Lee%2C+and+Dongsoo+Lee%2E+2022%2E+nuqmm%3A+Quantized+matmul+for+efficient+inference+of+large%2Dscale+generative+language+models%2E+arXiv+preprint+)
41. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2022. Efficiently scaling transformer inference. arXiv preprint arXiv:2211.05102. [find](https://scholar.google.com/scholar?q=Reiner+Pope%2C+Sholto+Douglas%2C+Aakanksha+Chowdhery%2C+Jacob+Devlin%2C+James+Bradbury%2C+Anselm+Levskaya%2C+Jonathan+Heek%2C+Kefan+Xiao%2C+Shivani+Agrawal%2C+and+Jeff+Dean%2E+2022%2E+Efficiently+scaling+transformer+infere)
42. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+2020%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtex)
43. Carl Edward Rasmussen, Christopher K Williams, et al. 2006. Gaussian processes for machine learning, vol. 1. [find](https://scholar.google.com/scholar?q=Carl+Edward+Rasmussen%2C+Christopher+K+Williams%2C+et+al%2E+2006%2E+Gaussian+processes+for+machine+learning%2C+vol%2E+1%2E)
44. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, I. Evtimov, Joanna Bitton, Manish P Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre D’efossez, Jade Copet, Faisal… [find](https://scholar.google.com/scholar?q=Baptiste+Rozi%C3%A8re%2C+Jonas+Gehring%2C+Fabian+Gloeckle%2C+Sten+Sootla%2C+Itai+Gat%2C+Xiaoqing+Ellen+Tan%2C+Yossi+Adi%2C+Jingyu+Liu%2C+Tal+Remez%2C+J%C3%A9r%C3%A9my+Rapin%2C+Artyom+Kozhevnikov%2C+I%2E+Evtimov%2C+Joanna+Bitton%2C+Manish+P+Bha) [our card](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
45. Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. 2022. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35. [find](https://scholar.google.com/scholar?q=Tal+Schuster%2C+Adam+Fisch%2C+Jai+Gupta%2C+Mostafa+Dehghani%2C+Dara+Bahri%2C+Vinh+Tran%2C+Yi+Tay%2C+and+Donald+Metzler%2E+2022%2E+Confident+adaptive+language+modeling%2E+Advances+in+Neural+Information+Processing+Systems%2C)
46. Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, and Noah A. Smith. 2020. The right tool for the job: Matching model and instance complexities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Roy+Schwartz%2C+Gabriel+Stanovsky%2C+Swabha+Swayamdipta%2C+Jesse+Dodge%2C+and+Noah+A%2E+Smith%2E+2020%2E+The+right+tool+for+the+job%3A+Matching+model+and+instance+complexities%2E+In+Proceedings+of+the+58th+Annual+Meeti)
47. Noam Shazeer. 2019. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+2019%2E+Fast+transformer+decoding%3A+One+write%2Dhead+is+all+you+need%2E+arXiv+preprint+arXiv%3A1911%2E02150%2E)
48. Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E Gonzalez, et al. 2023. High-throughput generative inference of large language models with a single gpu. arXiv preprint arXiv:2303.06865. [find](https://scholar.google.com/scholar?q=Ying+Sheng%2C+Lianmin+Zheng%2C+Binhang+Yuan%2C+Zhuohan+Li%2C+Max+Ryabinin%2C+Daniel+Y+Fu%2C+Zhiqiang+Xie%2C+Beidi+Chen%2C+Clark+Barrett%2C+Joseph+E+Gonzalez%2C+et+al%2E+2023%2E+High%2Dthroughput+generative+inference+of+large+l) [our card](https://md.research.elamur.ai/free/en/papers/sheng-2023-flexgen-high-throughput-generative-inference/)
49. Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31. [find](https://scholar.google.com/scholar?q=Mitchell+Stern%2C+Noam+Shazeer%2C+and+Jakob+Uszkoreit%2E+2018%2E+Blockwise+parallel+decoding+for+deep+autoregressive+models%2E+Advances+in+Neural+Information+Processing+Systems%2C+31%2E)
50. Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019. Distilling task-specific knowledge from bert into simple neural networks. arXiv preprint arXiv:1903.12136. — [arXiv:1903.12136](https://arxiv.org/abs/1903.12136) [find](https://scholar.google.com/scholar?q=Raphael+Tang%2C+Yao+Lu%2C+Linqing+Liu%2C+Lili+Mou%2C+Olga+Vechtomova%2C+and+Jimmy+Lin%2E+2019%2E+Distilling+task%2Dspecific+knowledge+from+bert+into+simple+neural+networks%2E+arXiv+preprint+arXiv%3A1903%2E12136%2E+%E2%80%94+%5BarXiv%3A1)
51. Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2021. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347–10357. PMLR. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Matthieu+Cord%2C+Matthijs+Douze%2C+Francisco+Massa%2C+Alexandre+Sablayrolles%2C+and+Herv%C3%A9+J%C3%A9gou%2E+2021%2E+Training+data%2Defficient+image+transformers+%26+distillation+through+attention%2E+In+Internation) [our card](https://md.research.elamur.ai/free/en/papers/touvron-2020-training-data-efficient-image-transformers/)
52. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fi)
53. Xiaohui Wang, Ying Xiong, Yang Wei, Mingxuan Wang, and Lei Li. 2021. Lightseq: A high performance inference library for transformers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers, pa… [find](https://scholar.google.com/scholar?q=Xiaohui+Wang%2C+Ying+Xiong%2C+Yang+Wei%2C+Mingxuan+Wang%2C+and+Lei+Li%2E+2021%2E+Lightseq%3A+A+high+performance+inference+library+for+transformers%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapte)
54. Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2023. Sheared LLaMA: Accelerating language model pre-training via structured pruning. [find](https://scholar.google.com/scholar?q=Mengzhou+Xia%2C+Tianyu+Gao%2C+Zhiyuan+Zeng%2C+and+Danqi+Chen%2E+2023%2E+Sheared+LLaMA%3A+Accelerating+language+model+pre%2Dtraining+via+structured+pruning%2E)
55. Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. 2022. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211.10438. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Ji+Lin%2C+Mickael+Seznec%2C+Julien+Demouth%2C+and+Song+Han%2E+2022%2E+Smoothquant%3A+Accurate+and+efficient+post%2Dtraining+quantization+for+large+language+models%2E+arXiv+preprint+arXiv%3A2211%2E10438%2E)
56. Ji Xin, Rodrigo Nogueira, Yaoliang Yu, and Jimmy Lin. 2020a. Early exiting BERT for efficient document ranking. In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing. [find](https://scholar.google.com/scholar?q=Ji+Xin%2C+Rodrigo+Nogueira%2C+Yaoliang+Yu%2C+and+Jimmy+Lin%2E+2020a%2E+Early+exiting+BERT+for+efficient+document+ranking%2E+In+Proceedings+of+SustaiNLP%3A+Workshop+on+Simple+and+Efficient+Natural+Language+Processin)
57. Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. 2020b. DeeBERT: Dynamic early exiting for accelerating BERT inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ji+Xin%2C+Raphael+Tang%2C+Jaejun+Lee%2C+Yaoliang+Yu%2C+and+Jimmy+Lin%2E+2020b%2E+DeeBERT%3A+Dynamic+early+exiting+for+accelerating+BERT+inference%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Co) [our card](https://md.research.elamur.ai/free/en/papers/xin-2020-deebert-dynamic-early-exiting-accelerating/)
58. Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023. Inference with reference: Lossless acceleration of large language models. arXiv preprint arXiv:2304.04487. — [arXiv:2304.04487](https://arxiv.org/abs/2304.04487) [find](https://scholar.google.com/scholar?q=Nan+Yang%2C+Tao+Ge%2C+Liang+Wang%2C+Binxing+Jiao%2C+Daxin+Jiang%2C+Linjun+Yang%2C+Rangan+Majumder%2C+and+Furu+Wei%2E+2023%2E+Inference+with+reference%3A+Lossless+acceleration+of+large+language+models%2E+arXiv+preprint+arXi)
59. Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2023. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695. [find](https://scholar.google.com/scholar?q=Mingjie+Sun%2C+Zhuang+Liu%2C+Anna+Bair%2C+and+J%2E+Zico+Kolter%2E+2023%2E+A+simple+and+effective+pruning+approach+for+large+language+models%2E+arXiv+preprint+arXiv%3A2306%2E11695%2E)
60. Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. 2022. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. arXiv preprint arXiv:2206.01861. [find](https://scholar.google.com/scholar?q=Zhewei+Yao%2C+Reza+Yazdani+Aminabadi%2C+Minjia+Zhang%2C+Xiaoxia+Wu%2C+Conglong+Li%2C+and+Yuxiong+He%2E+2022%2E+Zeroquant%3A+Efficient+and+affordable+post%2Dtraining+quantization+for+large%2Dscale+transformers%2E+arXiv+prep) [our card](https://md.research.elamur.ai/free/en/papers/yao-2022-zeroquant-efficient-affordable-post-training/)
61. Tianxiang Sun, Xiangyang Liu, Wei Zhu, Zhichao Geng, Lingling Wu, Yilong He, Yuan Ni, Guotong Xie, Xuanjing Huang, and Xipeng Qiu. 2022. A simple hash-based early exiting approach for language understanding and generation. In Findings of the Association for Computational Linguistics: ACL 2022. [find](https://scholar.google.com/scholar?q=Tianxiang+Sun%2C+Xiangyang+Liu%2C+Wei+Zhu%2C+Zhichao+Geng%2C+Lingling+Wu%2C+Yilong+He%2C+Yuan+Ni%2C+Guotong+Xie%2C+Xuanjing+Huang%2C+and+Xipeng+Qiu%2E+2022%2E+A+simple+hash%2Dbased+early+exiting+approach+for+language+underst)
62. Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538. [find](https://scholar.google.com/scholar?q=Gyeong%2DIn+Yu%2C+Joo+Seong+Jeong%2C+Geon%2DWoo+Kim%2C+Soojeong+Kim%2C+and+Byung%2DGon+Chun%2E+2022%2E+Orca%3A+A+distributed+serving+system+for+%7BTransformer%2DBased%7D+generative+models%2E+In+16th+USENIX+Symposium+on+Operating)
63. Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. 2019. Improving neural network quantization without retraining using outlier channel splitting. In International conference on machine learning, pages 7543–7552. PMLR. using code. This benchmark provides a unique perspective on the… [find](https://scholar.google.com/scholar?q=Ritchie+Zhao%2C+Yuwei+Hu%2C+Jordan+Dotzel%2C+Chris+De+Sa%2C+and+Zhiru+Zhang%2E+2019%2E+Improving+neural+network+quantization+without+retraining+using+outlier+channel+splitting%2E+In+International+conference+on+mach)
64. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric%2E+P+Xing%2C+Hao+Zhang%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+2023%2E+Judging+llm%2Das)
65. B Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. 2020. Bert loses patience: Fast and robust inference with early exit. Advances in Neural Information Processing Systems, 33. [find](https://scholar.google.com/scholar?q=B+Wangchunshu+Zhou%2C+Canwen+Xu%2C+Tao+Ge%2C+Julian+McAuley%2C+Ke+Xu%2C+and+Furu+Wei%2E+2020%2E+Bert+loses+patience%3A+Fast+and+robust+inference+with+early+exit%2E+Advances+in+Neural+Information+Processing+Systems%2C+33%2E)
66. Wei Zhu. 2021. Leebert: Learned early exit for bert with cross-level optimization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wei+Zhu%2E+2021%2E+Leebert%3A+Learned+early+exit+for+bert+with+cross%2Dlevel+optimization%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2E)