---
title: "EAGLE-2: Faster Inference of Language Models with — Li 2024"
description: "Speculative sampling makes LLM inference faster: a cheap draft model proposes tokens, and the original model checks them in parallel."
authors: ["Yuhui Li", "Fangyun Wei", "Chao Zhang", "Hongyang Zhang"]
year: 2024
venue: "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2024.emnlp-main.422/"
doi: "10.18653/v1/2024.emnlp-main.422"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2024-eagle-2-faster-inference-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2024-eagle-2-faster-inference-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/) / EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees

# EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees

**Authors:** [Yuhui Li](https://md.research.elamur.ai/free/en/authors/yuhui-li/), [Fangyun Wei](https://md.research.elamur.ai/free/en/authors/fangyun-wei/), [Chao Zhang](https://md.research.elamur.ai/free/en/authors/chao-zhang/), [Hongyang Zhang](https://md.research.elamur.ai/free/en/authors/hongyang-zhang/) · **Year:** 2024 · **Venue:** Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.emnlp-main.422/) · [PDF](https://aclanthology.org/2024.emnlp-main.422.pdf) · [DOI 10.18653/v1/2024.emnlp-main.422](https://doi.org/10.18653/v1/2024.emnlp-main.422) · [OpenAlex](https://openalex.org/W4404782770)

**Citations:** 33 citations (openalex-snapshot · 2026-10-14) · 98 in our library, of the 140884 papers whose bibliographies we read

## Summary
Speculative sampling makes LLM inference faster: a cheap draft model proposes tokens, and the original model checks them in parallel. Earlier tree-based methods such as EAGLE, Medusa and Sequoia use a fixed draft tree shape. This implicitly treats a draft token's chance of acceptance as a function of its tree position alone. The authors tested this with Vicuna 7B on Alpaca and found that acceptance also varies strongly with context. They also showed that EAGLE's draft model is well-calibrated, meaning its confidence scores closely track real acceptance rates.

EAGLE-2 builds on these findings. It uses draft confidence to grow and rerank the tree differently for each context, adding more branches where prediction is uncertain and fewer where it is easy. The method keeps the original LLM untouched and does not loosen the acceptance rule, so the output distribution is provably unchanged. It also needs no training beyond EAGLE's existing draft model.

Across six tasks and three model families, EAGLE-2 was the fastest of all compared methods. This makes it a practical, lossless way to cut LLM serving latency.

## Key points
- **Speedups:** 3.05x–4.26x, which is roughly 20–40% faster than EAGLE-1. Across all experiments the range was about 2.5x–5x. On MT-bench, it outpaced Medusa by around 2x and Lookahead by roughly 2.3x.
- **Models and data:** Vicuna, LLaMA2-Chat and LLaMA3-Instruct were tested on MT-bench, HumanEval, GSM8K, Alpaca, CNN/Daily Mail and Natural Questions.
- **Baselines:** standard speculative sampling, PLD, Medusa, Lookahead, Hydra and EAGLE.
- **Ablation (Vicuna 7B, temperature 0, MT-bench):** removing both value scoring and reranking drops the speedup from 3.62x to 2.81x. Average acceptance length falls from 4.98 to 3.92.
- **Limitations:** EAGLE-2 relies on a well-calibrated draft model, which is trained on ShareGPT. Training one for a 70B model takes about 1–2 days on 4 A100 (40G) GPUs. Code is released at github.com/SafeAILab/EAGLE.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{li2024eagle,
  title = {{EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees}},
  author = {Li, Yuhui and Wei, Fangyun and Zhang, Chao and Zhang, Hongyang},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.422},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.422}
}
```

- [← previous dzStance at StanceEval2024: Arabic Stance Detection based on Sentence Transformers](https://md.research.elamur.ai/free/en/papers/lichouri-2024-dzstance-stanceeval2024-arabic-stance-detection/)
- [next → EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://md.research.elamur.ai/free/en/papers/li-2024-eagle-speculative-sampling-requires-rethinking/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/)

## References

56 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ankner, Z., Parthasarathy, R., Nrusimha, A., Rinard, C., Ragan-Kelley, J., and Brandon, W. Hydra: Sequentially-dependent draft heads for medusa decoding. arXiv preprint arXiv:2402.05109, 2024. — [arXiv:2402.05109](https://arxiv.org/abs/2402.05109) [find](https://scholar.google.com/scholar?q=Ankner%2C+Z%2E%2C+Parthasarathy%2C+R%2E%2C+Nrusimha%2C+A%2E%2C+Rinard%2C+C%2E%2C+Ragan%2DKelley%2C+J%2E%2C+and+Brandon%2C+W%2E+Hydra%3A+Sequentially%2Ddependent+draft+heads+for+medusa+decoding%2E+arXiv+preprint+arXiv%3A2402%2E05109%2C+2024%2E+%E2%80%94+%5BarXi)
2. Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, T. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv: 2401.10774, 2024. [find](https://scholar.google.com/scholar?q=Cai%2C+T%2E%2C+Li%2C+Y%2E%2C+Geng%2C+Z%2E%2C+Peng%2C+H%2E%2C+Lee%2C+J%2E+D%2E%2C+Chen%2C+D%2E%2C+and+Dao%2C+T%2E+Medusa%3A+Simple+llm+inference+acceleration+framework+with+multiple+decoding+heads%2E+arXiv+preprint+arXiv%3A+2401%2E10774%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/cai-2024-medusa-simple-llm-inference-acceleration/)
3. Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023a. [find](https://scholar.google.com/scholar?q=Chen%2C+C%2E%2C+Borgeaud%2C+S%2E%2C+Irving%2C+G%2E%2C+Lespiau%2C+J%2E%2DB%2E%2C+Sifre%2C+L%2E%2C+and+Jumper%2C+J%2E+Accelerating+large+language+model+decoding+with+speculative+sampling%2E+arXiv+preprint+arXiv%3A2302%2E01318%2C+2023a%2E)
4. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. [find](https://scholar.google.com/scholar?q=Chen%2C+M%2E%2C+Tworek%2C+J%2E%2C+Jun%2C+H%2E%2C+Yuan%2C+Q%2E%2C+Pinto%2C+H%2E+P%2E+d%2E+O%2E%2C+Kaplan%2C+J%2E%2C+Edwards%2C+H%2E%2C+Burda%2C+Y%2E%2C+Joseph%2C+N%2E%2C+Brockman%2C+G%2E%2C+et+al%2E+Evaluating+large+language+models+trained+on+code%2E+arXiv+preprint+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
5. Chen, Z., Yang, X., Lin, J., Sun, C., Huang, J., and Chang, K. C.-C. Cascade speculative drafting for even faster llm inference. arXiv preprint arXiv:2312.11462, 2023b. — [arXiv:2312.11462](https://arxiv.org/abs/2312.11462) [find](https://scholar.google.com/scholar?q=Chen%2C+Z%2E%2C+Yang%2C+X%2E%2C+Lin%2C+J%2E%2C+Sun%2C+C%2E%2C+Huang%2C+J%2E%2C+and+Chang%2C+K%2E+C%2E%2DC%2E+Cascade+speculative+drafting+for+even+faster+llm+inference%2E+arXiv+preprint+arXiv%3A2312%2E11462%2C+2023b%2E+%E2%80%94+%5BarXiv%3A2312%2E11462%5D%28https%3A%2F%2Far)
6. Chen, Z., May, A., Svirschevski, R., Huang, Y., Ryabinin, M., Jia, Z., and Chen, B. Sequoia: Scalable, robust, and hardware-aware speculative decoding. arXiv preprint arXiv:2402.12374, 2024. [find](https://scholar.google.com/scholar?q=Chen%2C+Z%2E%2C+May%2C+A%2E%2C+Svirschevski%2C+R%2E%2C+Huang%2C+Y%2E%2C+Ryabinin%2C+M%2E%2C+Jia%2C+Z%2E%2C+and+Chen%2C+B%2E+Sequoia%3A+Scalable%2C+robust%2C+and+hardware%2Daware+speculative+decoding%2E+arXiv+preprint+arXiv%3A2402%2E12374%2C+2024%2E)
7. Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL [https://lmsys.org/blog/2023-03-30-vicuna/](https://lmsys.org/blog/2023-03-30-vicuna/). [↗](https://lmsys.org/blog/2023-03-30-vicuna/)
8. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [find](https://scholar.google.com/scholar?q=Cobbe%2C+K%2E%2C+Kosaraju%2C+V%2E%2C+Bavarian%2C+M%2E%2C+Chen%2C+M%2E%2C+Jun%2C+H%2E%2C+Kaiser%2C+L%2E%2C+Plappert%2C+M%2E%2C+Tworek%2C+J%2E%2C+Hilton%2C+J%2E%2C+Nakano%2C+R%2E%2C+et+al%2E+Training+verifiers+to+solve+math+word+problems%2E+arXiv+preprint+arXiv%3A2110) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
9. Du, C., Jiang, J., Yuanchen, X., Wu, J., Yu, S., Li, Y., Li, S., Xu, K., Nie, L., Tu, Z., et al. Glide with a cape: A low-hassle method to accelerate speculative decoding. arXiv preprint arXiv:2402.02082, 2024. — [arXiv:2402.02082](https://arxiv.org/abs/2402.02082) [find](https://scholar.google.com/scholar?q=Du%2C+C%2E%2C+Jiang%2C+J%2E%2C+Yuanchen%2C+X%2E%2C+Wu%2C+J%2E%2C+Yu%2C+S%2E%2C+Li%2C+Y%2E%2C+Li%2C+S%2E%2C+Xu%2C+K%2E%2C+Nie%2C+L%2E%2C+Tu%2C+Z%2E%2C+et+al%2E+Glide+with+a+cape%3A+A+low%2Dhassle+method+to+accelerate+speculative+decoding%2E+arXiv+preprint+arXiv%3A2402%2E02)
10. Elhoushi, M., Shrivastava, A., Liskovich, D., Hosmer, B., Wasti, B., Lai, L., Mahmoud, A., Acun, B., Agarwal, S., Roman, A., et al. Layer skip: Enabling early exit inference and self-speculative decoding. arXiv preprint arXiv:2404.16710, 2024. [find](https://scholar.google.com/scholar?q=Elhoushi%2C+M%2E%2C+Shrivastava%2C+A%2E%2C+Liskovich%2C+D%2E%2C+Hosmer%2C+B%2E%2C+Wasti%2C+B%2E%2C+Lai%2C+L%2E%2C+Mahmoud%2C+A%2E%2C+Acun%2C+B%2E%2C+Agarwal%2C+S%2E%2C+Roman%2C+A%2E%2C+et+al%2E+Layer+skip%3A+Enabling+early+exit+inference+and+self%2Dspeculative+decod) [our card](https://md.research.elamur.ai/free/en/papers/elhoushi-2024-layerskip-enabling-early-exit-inference/)
11. Fu, Y., Bailis, P., Stoica, I., and Zhang, H. Breaking the sequential dependency of llm inference using lookahead decoding, November 2023. URL [https://lmsys.org/blog/2023-11-21-lookahead-decoding/](https://lmsys.org/blog/2023-11-21-lookahead-decoding/). [↗](https://lmsys.org/blog/2023-11-21-lookahead-decoding/)
12. Fu, Y., Bailis, P., Stoica, I., and Zhang, H. Break the sequential dependency of llm inference using lookahead decoding. arXiv preprint arXiv:2402.02057, 2024. [find](https://scholar.google.com/scholar?q=Fu%2C+Y%2E%2C+Bailis%2C+P%2E%2C+Stoica%2C+I%2E%2C+and+Zhang%2C+H%2E+Break+the+sequential+dependency+of+llm+inference+using+lookahead+decoding%2E+arXiv+preprint+arXiv%3A2402%2E02057%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/fu-2024-break-sequential-dependency-llm-inference/)
13. Gale, T., Elsen, E., and Hooker, S. The state of sparsity in deep neural networks.(2019). arXiv preprint cs.LG/1902.09574, 2019. [find](https://scholar.google.com/scholar?q=Gale%2C+T%2E%2C+Elsen%2C+E%2E%2C+and+Hooker%2C+S%2E+The+state+of+sparsity+in+deep+neural+networks%2E%282019%29%2E+arXiv+preprint+cs%2ELG%2F1902%2E09574%2C+2019%2E)
14. Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. [find](https://scholar.google.com/scholar?q=Hinton%2C+G%2E%2C+Vinyals%2C+O%2E%2C+and+Dean%2C+J%2E+Distilling+the+knowledge+in+a+neural+network%2E+arXiv+preprint+arXiv%3A1503%2E02531%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/)
15. Hooper, C., Kim, S., Mohammadzadeh, H., Genc, H., Keutzer, K., Gholami, A., and Shao, S. Speed: Speculative pipelined execution for efficient decoding. arXiv preprint arXiv:2310.12072, 2023. — [arXiv:2310.12072](https://arxiv.org/abs/2310.12072) [find](https://scholar.google.com/scholar?q=Hooper%2C+C%2E%2C+Kim%2C+S%2E%2C+Mohammadzadeh%2C+H%2E%2C+Genc%2C+H%2E%2C+Keutzer%2C+K%2E%2C+Gholami%2C+A%2E%2C+and+Shao%2C+S%2E+Speed%3A+Speculative+pipelined+execution+for+efficient+decoding%2E+arXiv+preprint+arXiv%3A2310%2E12072%2C+2023%2E+%E2%80%94+%5BarXiv%3A)
16. Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y. Quantized neural networks: Training neural networks with low precision weights and activations. journal of machine learning research, 18(187):1–30, 2018. [find](https://scholar.google.com/scholar?q=Hubara%2C+I%2E%2C+Courbariaux%2C+M%2E%2C+Soudry%2C+D%2E%2C+El%2DYaniv%2C+R%2E%2C+and+Bengio%2C+Y%2E+Quantized+neural+networks%3A+Training+neural+networks+with+low+precision+weights+and+activations%2E+journal+of+machine+learning+resear)
17. Joao Gante. Assisted generation: a new direction toward low-latency text generation, 2023. URL [https://huggingface.co/blog/assisted-generation](https://huggingface.co/blog/assisted-generation). [↗](https://huggingface.co/blog/assisted-generation)
18. Kim, S., Gholami, A., Yao, Z., Mahoney, M. W., and Keutzer, K. I-bert: Integer-only bert quantization. In International conference on machine learning, pp. 5506– 5518. PMLR, 2021. [find](https://scholar.google.com/scholar?q=Kim%2C+S%2E%2C+Gholami%2C+A%2E%2C+Yao%2C+Z%2E%2C+Mahoney%2C+M%2E+W%2E%2C+and+Keutzer%2C+K%2E+I%2Dbert%3A+Integer%2Donly+bert+quantization%2E+In+International+conference+on+machine+learning%2C+pp%2E+5506%E2%80%93+5518%2E+PMLR%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2021-i-bert-integer-only-bert/)
19. Kim, S., Mangalam, K., Moon, S., Malik, J., Mahoney, M. W., Gholami, A., and Keutzer, K. Speculative decoding with big little decoder. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Kim%2C+S%2E%2C+Mangalam%2C+K%2E%2C+Moon%2C+S%2E%2C+Malik%2C+J%2E%2C+Mahoney%2C+M%2E+W%2E%2C+Gholami%2C+A%2E%2C+and+Keutzer%2C+K%2E+Speculative+decoding+with+big+little+decoder%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2C+2024%2E)
20. Kou, S., Hu, L., He, Z., Deng, Z., and Zhang, H. Cllms: Consistency large language models. arXiv preprint arXiv:2403.00835, 2024. — [arXiv:2403.00835](https://arxiv.org/abs/2403.00835) [find](https://scholar.google.com/scholar?q=Kou%2C+S%2E%2C+Hu%2C+L%2E%2C+He%2C+Z%2E%2C+Deng%2C+Z%2E%2C+and+Zhang%2C+H%2E+Cllms%3A+Consistency+large+language+models%2E+arXiv+preprint+arXiv%3A2403%2E00835%2C+2024%2E+%E2%80%94+%5BarXiv%3A2403%2E00835%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2403%2E00835%29)
21. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019. [find](https://scholar.google.com/scholar?q=Kwiatkowski%2C+T%2E%2C+Palomaki%2C+J%2E%2C+Redfield%2C+O%2E%2C+Collins%2C+M%2E%2C+Parikh%2C+A%2E%2C+Alberti%2C+C%2E%2C+Epstein%2C+D%2E%2C+Polosukhin%2C+I%2E%2C+Devlin%2C+J%2E%2C+Lee%2C+K%2E%2C+et+al%2E+Natural+questions%3A+a+benchmark+for+question+answering+resear) [our card](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
22. Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274– 19286. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Leviathan%2C+Y%2E%2C+Kalman%2C+M%2E%2C+and+Matias%2C+Y%2E+Fast+inference+from+transformers+via+speculative+decoding%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+19274%E2%80%93+19286%2E+PMLR%2C+2023%2E)
23. Li, M., Chen, X., Holtzman, A., Chen, B., Lin, J., Yih, W.-t., and Lin, X. V. Nearest neighbor speculative decoding for llm generation and attribution. arXiv preprint arXiv:2405.19325, 2024a. — [arXiv:2405.19325](https://arxiv.org/abs/2405.19325) [find](https://scholar.google.com/scholar?q=Li%2C+M%2E%2C+Chen%2C+X%2E%2C+Holtzman%2C+A%2E%2C+Chen%2C+B%2E%2C+Lin%2C+J%2E%2C+Yih%2C+W%2E%2Dt%2E%2C+and+Lin%2C+X%2E+V%2E+Nearest+neighbor+speculative+decoding+for+llm+generation+and+attribution%2E+arXiv+preprint+arXiv%3A2405%2E19325%2C+2024a%2E+%E2%80%94+%5BarXiv)
24. Li, Y., Wei, F., Zhang, C., and Zhang, H. Eagle: Speculative sampling requires rethinking feature uncertainty. In International Conference on Machine Learning, 2024b. [find](https://scholar.google.com/scholar?q=Li%2C+Y%2E%2C+Wei%2C+F%2E%2C+Zhang%2C+C%2E%2C+and+Zhang%2C+H%2E+Eagle%3A+Speculative+sampling+requires+rethinking+feature+uncertainty%2E+In+International+Conference+on+Machine+Learning%2C+2024b%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2024-eagle-speculative-sampling-requires-rethinking/)
25. Liu, F., Tang, Y., Liu, Z., Ni, Y., Han, K., and Wang, Y. Kangaroo: Lossless self-speculative decoding via double early exiting. arXiv preprint arXiv:2404.18911, 2024. — [arXiv:2404.18911](https://arxiv.org/abs/2404.18911) [find](https://scholar.google.com/scholar?q=Liu%2C+F%2E%2C+Tang%2C+Y%2E%2C+Liu%2C+Z%2E%2C+Ni%2C+Y%2E%2C+Han%2C+K%2E%2C+and+Wang%2C+Y%2E+Kangaroo%3A+Lossless+self%2Dspeculative+decoding+via+double+early+exiting%2E+arXiv+preprint+arXiv%3A2404%2E18911%2C+2024%2E+%E2%80%94+%5BarXiv%3A2404%2E18911%5D%28https%3A%2F%2Farx)
26. Liu, X., Hu, L., Bailis, P., Stoica, I., Deng, Z., Cheung, A., and Zhang, H. Online speculative decoding. arXiv preprint arXiv:2310.07177, 2023a. [find](https://scholar.google.com/scholar?q=Liu%2C+X%2E%2C+Hu%2C+L%2E%2C+Bailis%2C+P%2E%2C+Stoica%2C+I%2E%2C+Deng%2C+Z%2E%2C+Cheung%2C+A%2E%2C+and+Zhang%2C+H%2E+Online+speculative+decoding%2E+arXiv+preprint+arXiv%3A2310%2E07177%2C+2023a%2E)
27. Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Re, C., et al. Deja vu: Contextual sparsity for efficient llms at inference time. In International Conference on Machine Learning, pp. 22137–22176. PMLR, 2023b. [find](https://scholar.google.com/scholar?q=Liu%2C+Z%2E%2C+Wang%2C+J%2E%2C+Dao%2C+T%2E%2C+Zhou%2C+T%2E%2C+Yuan%2C+B%2E%2C+Song%2C+Z%2E%2C+Shrivastava%2C+A%2E%2C+Zhang%2C+C%2E%2C+Tian%2C+Y%2E%2C+Re%2C+C%2E%2C+et+al%2E+Deja+vu%3A+Contextual+sparsity+for+efficient+llms+at+inference+time%2E+In+International+Confe)
28. Meta. LLaMA3. [https://github.com/pytorch-labs/gpt-fast/](https://github.com/pytorch-labs/gpt-fast/), 2024. [↗](https://github.com/pytorch-labs/gpt-fast/)
29. Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Wong, R. Y. Y., Chen, Z., Arfeen, D., Abhyankar, R., and Jia, Z. SpecInfer: Accelerating generative LLM serving with speculative inference and token tree verification. arXiv preprint arXiv:2305.09781, 2023. [find](https://scholar.google.com/scholar?q=Miao%2C+X%2E%2C+Oliaro%2C+G%2E%2C+Zhang%2C+Z%2E%2C+Cheng%2C+X%2E%2C+Wang%2C+Z%2E%2C+Wong%2C+R%2E+Y%2E+Y%2E%2C+Chen%2C+Z%2E%2C+Arfeen%2C+D%2E%2C+Abhyankar%2C+R%2E%2C+and+Jia%2C+Z%2E+SpecInfer%3A+Accelerating+generative+LLM+serving+with+speculative+inference+and+tok)
30. Monea, G., Joulin, A., and Grave, E. Pass: Parallel speculative sampling. arXiv preprint arXiv:2311.13581, 2023. — [arXiv:2311.13581](https://arxiv.org/abs/2311.13581) [find](https://scholar.google.com/scholar?q=Monea%2C+G%2E%2C+Joulin%2C+A%2E%2C+and+Grave%2C+E%2E+Pass%3A+Parallel+speculative+sampling%2E+arXiv+preprint+arXiv%3A2311%2E13581%2C+2023%2E+%E2%80%94+%5BarXiv%3A2311%2E13581%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E13581%29)
31. Nallapati, R., Zhou, B., Gulcehre, C., Xiang, B., et al. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023, 2016. [find](https://scholar.google.com/scholar?q=Nallapati%2C+R%2E%2C+Zhou%2C+B%2E%2C+Gulcehre%2C+C%2E%2C+Xiang%2C+B%2E%2C+et+al%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+rnns+and+beyond%2E+arXiv+preprint+arXiv%3A1602%2E06023%2C+2016%2E)
32. OpenAI, R. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2(5), 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2C+R%2E+Gpt%2D4+technical+report%2E+arxiv+2303%2E08774%2E+View+in+Article%2C+2%285%29%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
33. Sanh, V., Wolf, T., and Rush, A. Movement pruning: Adaptive sparsity by fine-tuning. Advances in Neural Information Processing Systems, 33:20378–20389, 2020. [find](https://scholar.google.com/scholar?q=Sanh%2C+V%2E%2C+Wolf%2C+T%2E%2C+and+Rush%2C+A%2E+Movement+pruning%3A+Adaptive+sparsity+by+fine%2Dtuning%2E+Advances+in+Neural+Information+Processing+Systems%2C+33%3A20378%E2%80%9320389%2C+2020%2E)
34. Santilli, A., Severino, S., Postolache, E., Maiorca, V., Mancusi, M., Marin, R., and Rodolà, E. Accelerating transformer inference for translation via parallel decoding. arXiv preprint arXiv:2305.10427, 2023. [find](https://scholar.google.com/scholar?q=Santilli%2C+A%2E%2C+Severino%2C+S%2E%2C+Postolache%2C+E%2E%2C+Maiorca%2C+V%2E%2C+Mancusi%2C+M%2E%2C+Marin%2C+R%2E%2C+and+Rodola%CC%80%2C+E%2E+Accelerating+transformer+inference+for+translation+via+parallel+decoding%2E+arXiv+preprint+arXiv%3A2305%2E104)
35. Saxena, A. Prompt lookup decoding, November 2023. URL [https://github.com/apoorvumang/prompt-lookup-decoding/](https://github.com/apoorvumang/prompt-lookup-decoding/). [↗](https://github.com/apoorvumang/prompt-lookup-decoding/)
36. Shen, S., Dong, Z., Ye, J., Ma, L., Yao, Z., Gholami, A., Mahoney, M. W., and Keutzer, K. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8815–8821, 2020. [find](https://scholar.google.com/scholar?q=Shen%2C+S%2E%2C+Dong%2C+Z%2E%2C+Ye%2C+J%2E%2C+Ma%2C+L%2E%2C+Yao%2C+Z%2E%2C+Gholami%2C+A%2E%2C+Mahoney%2C+M%2E+W%2E%2C+and+Keutzer%2C+K%2E+Q%2Dbert%3A+Hessian+based+ultra+low+precision+quantization+of+bert%2E+In+Proceedings+of+the+AAAI+Conference+on+Artif)
37. Spector, B. and Re, C. Accelerating llm inference with staged speculative decoding. arXiv preprint arXiv:2308.04623, 2023. — [arXiv:2308.04623](https://arxiv.org/abs/2308.04623) [find](https://scholar.google.com/scholar?q=Spector%2C+B%2E+and+Re%2C+C%2E+Accelerating+llm+inference+with+staged+speculative+decoding%2E+arXiv+preprint+arXiv%3A2308%2E04623%2C+2023%2E+%E2%80%94+%5BarXiv%3A2308%2E04623%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2308%2E04623%29)
38. Stern, M., Shazeer, N., and Uszkoreit, J. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31, 2018. [find](https://scholar.google.com/scholar?q=Stern%2C+M%2E%2C+Shazeer%2C+N%2E%2C+and+Uszkoreit%2C+J%2E+Blockwise+parallel+decoding+for+deep+autoregressive+models%2E+Advances+in+Neural+Information+Processing+Systems%2C+31%2C+2018%2E)
39. Sun, H., Chen, Z., Yang, X., Tian, Y., and Chen, B. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. arXiv preprint arXiv:2404.11912, 2024a. [find](https://scholar.google.com/scholar?q=Sun%2C+H%2E%2C+Chen%2C+Z%2E%2C+Yang%2C+X%2E%2C+Tian%2C+Y%2E%2C+and+Chen%2C+B%2E+Triforce%3A+Lossless+acceleration+of+long+sequence+generation+with+hierarchical+speculative+decoding%2E+arXiv+preprint+arXiv%3A2404%2E11912%2C+2024a%2E)
40. Sun, X., Ge, T., Wei, F., and Wang, H. Instantaneous grammatical error correction with shallow aggressive decoding. arXiv preprint arXiv:2106.04970, 2021. [find](https://scholar.google.com/scholar?q=Sun%2C+X%2E%2C+Ge%2C+T%2E%2C+Wei%2C+F%2E%2C+and+Wang%2C+H%2E+Instantaneous+grammatical+error+correction+with+shallow+aggressive+decoding%2E+arXiv+preprint+arXiv%3A2106%2E04970%2C+2021%2E)
41. Sun, Z., Ro, J. H., Beirami, A., and Suresh, A. T. Optimal block-level draft verification for accelerating speculative decoding. arXiv preprint arXiv:2403.10444, 2024b. — [arXiv:2403.10444](https://arxiv.org/abs/2403.10444) [find](https://scholar.google.com/scholar?q=Sun%2C+Z%2E%2C+Ro%2C+J%2E+H%2E%2C+Beirami%2C+A%2E%2C+and+Suresh%2C+A%2E+T%2E+Optimal+block%2Dlevel+draft+verification+for+accelerating+speculative+decoding%2E+arXiv+preprint+arXiv%3A2403%2E10444%2C+2024b%2E+%E2%80%94+%5BarXiv%3A2403%2E10444%5D%28https%3A%2F%2Far)
42. Sun, Z., Suresh, A. T., Ro, J. H., Beirami, A., Jain, H., and Yu, F. Spectr: Fast speculative decoding via optimal transport. Advances in Neural Information Processing Systems, 36, 2024c. [find](https://scholar.google.com/scholar?q=Sun%2C+Z%2E%2C+Suresh%2C+A%2E+T%2E%2C+Ro%2C+J%2E+H%2E%2C+Beirami%2C+A%2E%2C+Jain%2C+H%2E%2C+and+Yu%2C+F%2E+Spectr%3A+Fast+speculative+decoding+via+optimal+transport%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2C+2024c%2E)
43. Svirschevski, R., May, A., Chen, Z., Chen, B., Jia, Z., and Ryabinin, M. Specexec: Massively parallel speculative decoding for interactive llm inference on consumer devices. arXiv preprint arXiv:2406.02532, 2024. [find](https://scholar.google.com/scholar?q=Svirschevski%2C+R%2E%2C+May%2C+A%2E%2C+Chen%2C+Z%2E%2C+Chen%2C+B%2E%2C+Jia%2C+Z%2E%2C+and+Ryabinin%2C+M%2E+Specexec%3A+Massively+parallel+speculative+decoding+for+interactive+llm+inference+on+consumer+devices%2E+arXiv+preprint+arXiv%3A2406%2E)
44. Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
45. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models (2023). arXiv preprint arXiv:2302.13971, 2023. [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%2C+Lavril%2C+T%2E%2C+Izacard%2C+G%2E%2C+Martinet%2C+X%2E%2C+Lachaux%2C+M%2E%2DA%2E%2C+Lacroix%2C+T%2E%2C+Rozie%CC%80re%2C+B%2E%2C+Goyal%2C+N%2E%2C+Hambro%2C+E%2E%2C+Azhar%2C+F%2E%2C+et+al%2E+Llama%3A+Open+and+efficient+foundation+language+models+%282023%29%2E+arX)
46. Xia, H., Yang, Z., Dong, Q., Wang, P., Li, Y., Ge, T., Liu, T., Li, W., and Sui, Z. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding, 2024. [find](https://scholar.google.com/scholar?q=Xia%2C+H%2E%2C+Yang%2C+Z%2E%2C+Dong%2C+Q%2E%2C+Wang%2C+P%2E%2C+Li%2C+Y%2E%2C+Ge%2C+T%2E%2C+Liu%2C+T%2E%2C+Li%2C+W%2E%2C+and+Sui%2C+Z%2E+Unlocking+efficiency+in+large+language+model+inference%3A+A+comprehensive+survey+of+speculative+decoding%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/xia-2024-unlocking-efficiency-large-language-model/)
47. Yang, N., Ge, T., Wang, L., Jiao, B., Jiang, D., Yang, L., Majumder, R., and Wei, F. Inference with reference: Lossless acceleration of large language models. arXiv preprint arXiv:2304.04487, 2023a. — [arXiv:2304.04487](https://arxiv.org/abs/2304.04487) [find](https://scholar.google.com/scholar?q=Yang%2C+N%2E%2C+Ge%2C+T%2E%2C+Wang%2C+L%2E%2C+Jiao%2C+B%2E%2C+Jiang%2C+D%2E%2C+Yang%2C+L%2E%2C+Majumder%2C+R%2E%2C+and+Wei%2C+F%2E+Inference+with+reference%3A+Lossless+acceleration+of+large+language+models%2E+arXiv+preprint+arXiv%3A2304%2E04487%2C+2023a%2E+%E2%80%94)
48. Yang, S., Lee, G., Cho, J., Papailiopoulos, D., and Lee, K. Predictive pipelined decoding: A compute-latency trade-off for exact llm decoding. arXiv preprint arXiv:2307.05908, 2023b. — [arXiv:2307.05908](https://arxiv.org/abs/2307.05908) [find](https://scholar.google.com/scholar?q=Yang%2C+S%2E%2C+Lee%2C+G%2E%2C+Cho%2C+J%2E%2C+Papailiopoulos%2C+D%2E%2C+and+Lee%2C+K%2E+Predictive+pipelined+decoding%3A+A+compute%2Dlatency+trade%2Doff+for+exact+llm+decoding%2E+arXiv+preprint+arXiv%3A2307%2E05908%2C+2023b%2E+%E2%80%94+%5BarXiv%3A2307%2E059)
49. Yi, H., Lin, F., Li, H., Ning, P., Yu, X., and Xiao, R. Generation meets verification: Accelerating large language model inference with smart parallel auto-correct decoding. arXiv preprint arXiv:2402.11809, 2024. [find](https://scholar.google.com/scholar?q=Yi%2C+H%2E%2C+Lin%2C+F%2E%2C+Li%2C+H%2E%2C+Ning%2C+P%2E%2C+Yu%2C+X%2E%2C+and+Xiao%2C+R%2E+Generation+meets+verification%3A+Accelerating+large+language+model+inference+with+smart+parallel+auto%2Dcorrect+decoding%2E+arXiv+preprint+arXiv%3A2402%2E)
50. Zadeh, A. H., Edo, I., Awad, O. M., and Moshovos, A. Gobo: Quantizing attention-based nlp models for low latency and energy efficient inference. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pp. 811–824. [find](https://scholar.google.com/scholar?q=Zadeh%2C+A%2E+H%2E%2C+Edo%2C+I%2E%2C+Awad%2C+O%2E+M%2E%2C+and+Moshovos%2C+A%2E+Gobo%3A+Quantizing+attention%2Dbased+nlp+models+for+low+latency+and+energy+efficient+inference%2E+In+2020+53rd+Annual+IEEE%2FACM+International+Symposium+on)
51. Zafrir, O., Boudoukh, G., Izsak, P., and Wasserblat, M. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS), pp. 36–39. [find](https://scholar.google.com/scholar?q=Zafrir%2C+O%2E%2C+Boudoukh%2C+G%2E%2C+Izsak%2C+P%2E%2C+and+Wasserblat%2C+M%2E+Q8bert%3A+Quantized+8bit+bert%2E+In+2019+Fifth+Workshop+on+Energy+Efficient+Machine+Learning+and+Cognitive+Computing%2DNeurIPS+Edition+%28EMC2%2DNIPS%29%2C+pp)
52. Zhang, A., Wang, C., Wang, Y., Zhang, X., and Cheng, Y. Recurrent drafter for fast speculative decoding in large language models. arXiv preprint arXiv:2403.09919, 2024. [find](https://scholar.google.com/scholar?q=Zhang%2C+A%2E%2C+Wang%2C+C%2E%2C+Wang%2C+Y%2E%2C+Zhang%2C+X%2E%2C+and+Cheng%2C+Y%2E+Recurrent+drafter+for+fast+speculative+decoding+in+large+language+models%2E+arXiv+preprint+arXiv%3A2403%2E09919%2C+2024%2E)
53. Zhang, J., Wang, J., Li, H., Shou, L., Chen, K., Chen, G., and Mehrotra, S. Draft & verify: Lossless large language model acceleration via self-speculative decoding. arXiv preprint arXiv:2309.08168, 2023. [find](https://scholar.google.com/scholar?q=Zhang%2C+J%2E%2C+Wang%2C+J%2E%2C+Li%2C+H%2E%2C+Shou%2C+L%2E%2C+Chen%2C+K%2E%2C+Chen%2C+G%2E%2C+and+Mehrotra%2C+S%2E+Draft+%26+verify%3A+Lossless+large+language+model+acceleration+via+self%2Dspeculative+decoding%2E+arXiv+preprint+arXiv%3A2309%2E08168%2C+2) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2024-draft-verify-lossless-large-language/)
54. Zhao, W., Huang, Y., Han, X., Xiao, C., Liu, Z., and Sun, M. Ouroboros: Speculative decoding with large model enhanced drafting. arXiv preprint arXiv:2402.13720, 2024. — [arXiv:2402.13720](https://arxiv.org/abs/2402.13720) [find](https://scholar.google.com/scholar?q=Zhao%2C+W%2E%2C+Huang%2C+Y%2E%2C+Han%2C+X%2E%2C+Xiao%2C+C%2E%2C+Liu%2C+Z%2E%2C+and+Sun%2C+M%2E+Ouroboros%3A+Speculative+decoding+with+large+model+enhanced+drafting%2E+arXiv+preprint+arXiv%3A2402%2E13720%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E13720%5D%28https%3A%2F%2Farx)
55. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023. [find](https://scholar.google.com/scholar?q=Zheng%2C+L%2E%2C+Chiang%2C+W%2E%2DL%2E%2C+Sheng%2C+Y%2E%2C+Zhuang%2C+S%2E%2C+Wu%2C+Z%2E%2C+Zhuang%2C+Y%2E%2C+Lin%2C+Z%2E%2C+Li%2C+Z%2E%2C+Li%2C+D%2E%2C+Xing%2C+E%2E%2C+et+al%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2E+arXiv+preprint+arXiv%3A2306%2E05685%2C+)
56. Zhou, Y., Lyu, K., Rawat, A. S., Menon, A. K., Rostamizadeh, A., Kumar, S., Kagy, J.-F., and Agarwal, R. Distillspec: Improving speculative decoding via knowledge distillation. In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=rsY6J3ZaTF](https://openreview.net/forum?id=rsY6J3ZaTF). [↗](https://openreview.net/forum?id=rsY6J3ZaTF)