--- title: "Online Speculative Decoding — paper by Liu et al. (2023)" description: "The authors propose online speculative decoding (OSD), which keeps the draft model adapting to real… 4 references, cited by 62 papers here." authors: ["Xiaoxuan Liu", "Lanxiang Hu", "Peter Bailis", "Alvin Cheung", "Zhijie Deng", "Ion Stoica", "Hao Zhang"] year: 2023 venue: "arXiv:2310.07177" source: "https://arxiv.org/abs/2310.07177" doi: "10.48550/arxiv.2310.07177" arxiv: "2310.07177" licence: "CC0 1.0 (arXiv)" licence_url: "http://creativecommons.org/publicdomain/zero/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/liu-2023-online-speculative-decoding/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2023-online-speculative-decoding/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/) / Online Speculative Decoding # Online Speculative Decoding **Authors:** Xiaoxuan Liu, Lanxiang Hu, [Peter Bailis](https://md.research.elamur.ai/free/en/authors/peter-d-bailis/), [Alvin Cheung](https://md.research.elamur.ai/free/en/authors/alvin-cheung/), [Zhijie Deng](https://md.research.elamur.ai/free/en/authors/zhijie-deng/), [Ion Stoica](https://md.research.elamur.ai/free/en/authors/ion-gabriel-stoica/) and [Hao Zhang](https://md.research.elamur.ai/free/en/authors/hao-zhang/) · **Year:** 2023 · **Venue:** arXiv:2310.07177 · **Licence:** [CC0 1.0 (arXiv)](http://creativecommons.org/publicdomain/zero/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2310.07177) · [PDF](https://arxiv.org/pdf/2310.07177) · [DOI 10.48550/arxiv.2310.07177](https://doi.org/10.48550/arxiv.2310.07177) · [arXiv 2310.07177](https://arxiv.org/abs/2310.07177) · [OpenAlex](https://openalex.org/W4387596095) **Citations:** 3 citations (OpenAlex, 2026-10-14) · [62](https://md.research.elamur.ai/free/en/papers/liu-2023-online-speculative-decoding/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Speculative decoding speeds up LLM inference by having a small draft model propose tokens that a large target model checks in parallel. The gain depends on how often those proposals are accepted, and that rate falls when the draft model is much weaker or when user inputs are varied. The authors propose online speculative decoding (OSD), which keeps the draft model adapting to real serving traffic. When the target model rejects a draft token, both probability distributions are saved to a buffer. Once the buffer is large or old enough, the draft model is retrained with knowledge distillation, using forward KL, reverse KL or JSD as the distance. OSD can also maintain several draft models, each specialised for a language or topic, and route every query to the best-matching one. Tested on synthetic and real query data, OSD raises token acceptance by 0.1–0.65 and cuts latency by 1.42×–2.17×. This matters because services can serve LLMs faster without enlarging the draft model. ## Key points - Tested model pairs were LLaMA-160M drafting for Vicuna-7B and TinyLLaMA-1.1B drafting for Vicuna-33B. Latency was measured with llamacpp on one A100-80G GPU. - For the latency experiments, the chosen distillation setup was forward KD on teacher-sampled data. Speedup peaks at 2.63× once acceptance exceeds 0.9. - On the Spider SQL dataset, distillation improved acceptance for domain-typical tokens such as SELECT and WHERE. The 100 most common target tokens make up 72.2% of all generated tokens. - On Chatbot Arena Conversations, separate draft models per language or topic achieved higher acceptance than a single shared draft model. - OSD also works with Medusa-v1: - On Spider, it raised Medusa's speedup from 1.34× to 2.01×, while standard speculative decoding with OSD reached 2.17×. - On Arena, OSD added 0.35× to Medusa's speedup. Code is available at github.com/LiuXiaoxuanPKU/OSD (ICML 2024). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works 🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 297 works ## Cite Cite as: Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica and Hao Zhang. 2023. Online Speculative Decoding. arXiv:2310.07177. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{liu2023online, title = {{Online Speculative Decoding}}, author = {Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica and Hao Zhang}, year = {2023}, doi = {10.48550/arxiv.2310.07177}, eprint = {2310.07177}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2310.07177} } ``` - [← previous MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection System](https://md.research.elamur.ai/free/en/papers/qin-2023-mmsd2-0-reliable-multi-modal/) - [next → Questions Are All You Need to Train a Dense Passage Retriever](https://md.research.elamur.ai/free/en/papers/sachan-2023-questions-are-all-you-need/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/) ## References 4 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Huggingface transformers, 2023. URL [https://huggingface.co/docs/transformers/](https://huggingface.co/docs/transformers/) en/index. *Table 4. OSD and Medusa* Dataset Spider Chatbot Arena Extra Parameters (B) [↗](https://huggingface.co/docs/transformers/) 2. Medusa-7B Medusa-7B + OSD Draft model + OSD 1.34× 2.01× 2.17× 2.03× 2.38× 1.51× 0.44 0.44 0.16 [find](https://scholar.google.com/scholar?q=Medusa%2D7B+Medusa%2D7B+%2B+OSD+Draft+model+%2B+OSD+1%2E34%C3%97+2%2E01%C3%97+2%2E17%C3%97+2%2E03%C3%97+2%2E38%C3%97+1%2E51%C3%97+0%2E44+0%2E44+0%2E16) 3. Sharegpt, 2023. URL [https://huggingface.co/datasets/Aeala/ShareGPT_Vicuna_](https://huggingface.co/datasets/Aeala/ShareGPT_Vicuna_) unfiltered. [↗](https://huggingface.co/datasets/Aeala/ShareGPT_Vicuna_) 4. Agarwal, R., Vieillard, N., Stanczyk, P., Ramos, S., Geist, M., and Bachem, O. Gkd: Generalized knowledge distillation for auto-regressive sequence models. arXiv preprint arXiv:2306.[13649, 2023](https://md.research.elamur.ai/free/en/papers/agarwal-2023-policy-distillation-language-models-learning/). \#\# 6. Conclusion *Updated 2 November 2026*