--- title: "SpecInfer — paper by Miao et al. (2023) · ASPLOS 2024" description: "Serving generative LLMs is slow and expensive because autoregressive decoding emits one token… 8 references, cited by 110 papers here." authors: ["Xupeng Miao", "Gabriele Oliaro", "Zhihao Zhang", "Xinhao Cheng", "Zeyu Wang", "Zhengxin Zhang", "Rae Ying Yee Wong", "Alan Zhu", "Lijie Yang", "Xiaoxiang Shi"] year: 2023 venue: "ASPLOS 2024" source: "https://arxiv.org/abs/2305.09781" doi: "10.1145/3620666.3651335" arxiv: "2305.09781" licence: "CC BY 4.0 (arXiv OAI)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/miao-2023-specinfer-accelerating-generative-large-language/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/miao-2023-specinfer-accelerating-generative-large-language/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) / SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification # SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification **Authors:** [Xupeng Miao](https://md.research.elamur.ai/free/en/authors/xupeng-miao/), [Gabriele Oliaro](https://md.research.elamur.ai/free/en/authors/gabriele-oliaro/), [Zhihao Zhang](https://md.research.elamur.ai/free/en/authors/zhihao-zhang/), [Xinhao Cheng](https://md.research.elamur.ai/free/en/authors/xinhao-cheng/), Zeyu Wang, [Zhengxin Zhang](https://md.research.elamur.ai/free/en/authors/zhengxin-zhang/), Rae Ying Yee Wong, Alan Zhu, Lijie Yang and Xiaoxiang Shi · **Year:** 2023 · **Venue:** [ASPLOS 2024](https://md.research.elamur.ai/free/en/venues/acm-international-conference-on-architectural-support-for-programming-languages/) ([ASPLOS 2023](https://md.research.elamur.ai/free/en/venues/acm-international-conference-on-architectural-support-for-programming-languages/2023/)) · **Licence:** [CC BY 4.0 (arXiv OAI)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://doi.org/10.1145/3620666.3651335) · [PDF](https://arxiv.org/pdf/2305.09781) · [DOI 10.1145/3620666.3651335](https://doi.org/10.1145/3620666.3651335) · [arXiv 2305.09781](https://arxiv.org/abs/2305.09781) · [OpenAlex](https://openalex.org/W4395112660) **Citations:** 110 citations (OpenAlex, 2026-10-14) · [110](https://md.research.elamur.ai/free/en/papers/miao-2023-specinfer-accelerating-generative-large-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Serving generative LLMs is slow and expensive because autoregressive decoding emits one token per step, which leaves GPUs underused and makes key-value caches costly in memory. Earlier speculative approaches let a single small model draft one token sequence for the LLM to check, but the large capacity gap between the two models limits how many drafted tokens survive. SpecInfer, built inside the FlexFlow framework, drafts many candidate continuations at once and arranges them as a token tree. The trees come from small speculative models (SSMs), which can be distilled, quantized, pruned or pre-trained smaller siblings of the LLM. The LLM is then used as a verifier rather than a step-by-step decoder: it checks every branch of the tree in a single decoding pass, using tree-based parallel decoding. A multi-step speculative sampling scheme keeps the output distribution identical to ordinary decoding, so quality is provably preserved. Reported speedups over existing serving systems are 1.5–2.8× for distributed multi-GPU inference and 2.6–3.5× for offloading-based single-GPU inference, with no change in generation quality. ## Key points - Token trees raise the per-token verification success rate under stochastic decoding from roughly 52–57% (sequence-based speculation) to 96–97% (Table 1). - Trees are built in two ways. Expansion draws several candidates from one SSM, and merging combines the outputs of multiple SSMs. - SSMs are typically 100–1000× smaller than the target LLM. For example, OPT-125M and OPT-350M were pre-trained on the same data as OPT-175B and can serve directly as drafters. - The search space is very large. OPT models have a 50,272-token vocabulary, and SpecInfer correctly guesses about 4 future tokens on average, which amounts to about 6×10¹⁸ possible combinations. - Code is released in the FlexFlow repository (github.com/flexflow/FlexFlow). ## Interesting ideas - **Drafting many branches instead of one.** A cheap model that is often wrong about the single best path can still usually contain the right answer among a few alternatives. This makes "generate many cheaply, verify once expensively" a useful pattern wherever a weak model assists a strong one. - **Attending only to a token's ancestors.** Each tree node is processed as if its ancestors were its preceding context, so all branches fit into one forward pass of the LLM. This style of masking could be reused to score many candidate continuations in parallel, for example in reranking or constrained generation. - **Merging several small models as an ensemble of drafters.** Combining predictions from different SSMs widens coverage of what the LLM might say. It is a practical way to use whatever small distilled or quantized variants are already available, for instance for a Ukrainian-tuned model family. - **Lossless acceleration as a design goal.** The sampling procedure is built so the final output distribution is provably identical to plain decoding. Speedups therefore need no re-evaluation of model quality, which makes such an optimization much easier to adopt in production pipelines. - **Turning a latency problem into a verification problem.** The LLM is treated as a checker rather than a generator. This reframing applies to other sequential bottlenecks where checking a proposal is cheaper or more parallel than producing it. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,053 citations - [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 598 works · 191,226 citations - [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations - [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,267 citations - [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 212 works · 19,544 citations - [University of California San Diego](https://md.research.elamur.ai/free/en/universities/university-of-california-san-diego/) — University, San Diego, United States · 89 works · 54,032 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works 🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works ## Cite Cite as: Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang and Xiaoxiang Shi. 2023. SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification. ASPLOS 2024. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{miao2023specinfer, title = {{SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification}}, author = {Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang and Xiaoxiang Shi}, booktitle = {ASPLOS 2024}, year = {2023}, doi = {10.1145/3620666.3651335}, eprint = {2305.09781}, archivePrefix = {arXiv}, url = {https://doi.org/10.1145/3620666.3651335} } ``` - [← previous AlpaGasus: Training A Better Alpaca with Fewer Data](https://md.research.elamur.ai/free/en/papers/chen-2023-alpagasus-training-better-alpaca-fewer/) - [next → Distilling Reasoning Capabilities into Smaller Language Models](https://md.research.elamur.ai/free/en/papers/shridhar-2023-distilling-reasoning-capabilities-into-smaller/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) ## References 8 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/), 2020. [4] F Warren Burton. Spec… 2. PMLR, 2022. [10] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot, 2023. [11] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate quantization for generative pre-trained transformers. In International Conference on L… [find](https://scholar.google.com/scholar?q=PMLR%2C+2022%2E+%5B10%5D+Elias+Frantar+and+Dan+Alistarh%2E+Sparsegpt%3A+Massive+language+models+can+be+accurately+pruned+in+one%2Dshot%2C+2023%2E+%5B11%5D+Elias+Frantar%2C+Saleh+Ashkboos%2C+Torsten+Hoefler%2C+and+Dan+Alistarh%2E+G) 3. Citeseer, 1996. [14] Mudasir A Ganaie, Minghui Hu, AK Malik, M Tanveer, and PN Suganthan. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence, 115:105151, 2022. [15] Aaron Gokaslan*, Vanya Cohen*, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. http://Skylio… [find](https://scholar.google.com/scholar?q=Citeseer%2C+1996%2E+%5B14%5D+Mudasir+A+Ganaie%2C+Minghui+Hu%2C+AK+Malik%2C+M+Tanveer%2C+and+PN+Suganthan%2E+Ensemble+deep+learning%3A+A+review%2E+Engineering+Applications+of+Artificial+Intelligence%2C+115%3A105151%2C+2022%2E+%5B15%5D+) 4. OpenWebTextCorpus, 2019. [16] John L Hennessy and David A Patterson. Computer architecture: a quantitative approach. [find](https://scholar.google.com/scholar?q=OpenWebTextCorpus%2C+2019%2E+%5B16%5D+John+L+Hennessy+and+David+A+Patterson%2E+Computer+architecture%3A+a+quantitative+approach%2E) 5. Elsevier, 2011. [17] Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry. Accelerated sparse neural training: A provable and efficient method to find n: m transposable masks. Advances in Neural Information Processing Systems, 34:21099–21111, 2021. [18] HuggingFace. La… [find](https://scholar.google.com/scholar?q=Elsevier%2C+2011%2E+%5B17%5D+Itay+Hubara%2C+Brian+Chmiel%2C+Moshe+Island%2C+Ron+Banner%2C+Joseph+Naor%2C+and+Daniel+Soudry%2E+Accelerated+sparse+neural+training%3A+A+provable+and+efficient+method+to+find+n%3A+m+transposable+) 6. USENIX Association, July 2021. [51] Heming Xia, Tao Ge, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decoding: Lossless speedup of autoregressive translation. [52] Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantiza… [find](https://scholar.google.com/scholar?q=USENIX+Association%2C+July+2021%2E+%5B51%5D+Heming+Xia%2C+Tao+Ge%2C+Si%2DQing+Chen%2C+Furu+Wei%2C+and+Zhifang+Sui%2E+Speculative+decoding%3A+Lossless+speedup+of+autoregressive+translation%2E+%5B52%5D+Guangxuan+Xiao%2C+Ji+Lin%2C+Mick) 7. George Zerveas, Vijay Korthikanti, et al. [Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model](https://md.research.elamur.ai/free/en/papers/smith-2022-using-deepspeed-megatron-train-megatron/). arXiv preprint arXiv:2201.11990, 2022. [44] Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models… 8. IEEE, 2021. [50] Haojie Wang, Jidong Zhai, Mingyu Gao, Zixuan Ma, Shizhi Tang, Liyan Zheng, Yuanzhi Li, Kaiyuan Rong, Yuanyong Chen, and Zhihao Jia. PET: Optimizing tensor programs with partially equivalent transformations and automated corrections. In 15th USENIX Symposium on [find](https://scholar.google.com/scholar?q=IEEE%2C+2021%2E+%5B50%5D+Haojie+Wang%2C+Jidong+Zhai%2C+Mingyu+Gao%2C+Zixuan+Ma%2C+Shizhi+Tang%2C+Liyan+Zheng%2C+Yuanzhi+Li%2C+Kaiyuan+Rong%2C+Yuanyong+Chen%2C+and+Zhihao+Jia%2E+PET%3A+Optimizing+tensor+programs+with+partially+equi) *Updated 2 November 2026*