---
title: "Turning Trash into Treasure — paper by Luo et al. (2025)"
description: "A BFS-like procedure builds a draft tree from the matrix, tree attention verifies it, and fresh candidates… 43 references, cited by 20 papers here."
authors: ["Xianzhen Luo", "Yixuan Wang", "Qingfu Zhu", "Zhiming Zhang", "Xuanyu Zhang", "Qing Yang", "Dongliang Xu"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.338/"
doi: "10.18653/v1/2025.acl-long.338"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/luo-2025-turning-trash-into-treasure-accelerating/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/luo-2025-turning-trash-into-treasure-accelerating/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/) / Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling

# Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling

**Authors:** [Xianzhen Luo](https://md.research.elamur.ai/free/en/authors/xianzhen-luo/), [Yixuan Wang](https://md.research.elamur.ai/free/en/authors/yixuan-wang/), [Qingfu Zhu](https://md.research.elamur.ai/free/en/authors/qingfu-zhu/), Zhiming Zhang, [Xuanyu Zhang](https://md.research.elamur.ai/free/en/authors/xuanyu-zhang/), [Qing Yang](https://md.research.elamur.ai/free/en/authors/qing-yang/) and [Dongliang Xu](https://md.research.elamur.ai/free/en/authors/dongliang-xu/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.338/) · [PDF](https://aclanthology.org/2025.acl-long.338.pdf) · [DOI 10.18653/v1/2025.acl-long.338](https://doi.org/10.18653/v1/2025.acl-long.338) · [OpenAlex](https://openalex.org/W4412945522)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [20](https://md.research.elamur.ai/free/en/papers/luo-2025-turning-trash-into-treasure-accelerating/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLM inference is slow because autoregressive decoding emits one token per step, and each step is dominated by memory transfer of billions of weights. Speculative decoding fixes this losslessly by drafting tokens and verifying them in one pass. However, trained drafters need per-model training, and retrieval-based training-free approaches (REST, PLD, Lookahead) suffer from heavy storage, slow lookup, poor adaptability, or extra generation cost. The authors observe that the top-k candidate tokens a model produces at each step, normally thrown away, often reappear later. Token Recycling (TR) keeps these candidates in a compact token-to-candidates adjacency matrix with a hot start. A BFS-like procedure builds a draft tree from the matrix, tree attention verifies it, and fresh candidates continuously update the matrix. Under greedy decoding on SpecBench and MBPP with Vicuna and Code Llama, TR gives roughly 2x speedup at every model size, using under 2MB. This makes it a cheap plug-and-play accelerator that needs no training.

## Key points
- **Method:** TR is training-free and model-agnostic. It recycles top-k candidate tokens into an adjacency matrix (|V|×k), retrieves a static draft tree via BFS-like search, and verifies the tree with tree attention.
- **Experimental setup:** Experiments cover SpecBench (general) and MBPP (code), using Vicuna and Code Llama at 7b, 13b, and 33b/34b scales.
- **Main results:** TR achieves about 2x speedup with less than 2MB of extra storage. It beats prior training-free methods by more than 30% and exceeds the trained Medusa by about 25%.
- **Comparison with Eagle:**
  - TR surpasses Eagle1.
  - It reaches 91.23% of Eagle2's acceleration while using only 0.39% of Eagle2's memory.
  - Retrieval, preprocessing, and update steps add negligible time.
- **Limitation and code:** TR relies on a static draft tree, and the authors name dynamic trees as future work. Code is available at github.com/Luowaterbi/TokenRecycling.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Harbin Institute of Technology](https://md.research.elamur.ai/free/en/universities/harbin-institute-of-technology/) — University, Harbin, China · 185 works · 29,140 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Xianzhen Luo, Yixuan Wang, Qingfu Zhu, Zhiming Zhang, Xuanyu Zhang, Qing Yang and Dongliang Xu. 2025. Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{luo2025turning,
  title = {{Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling}},
  author = {Xianzhen Luo, Yixuan Wang, Qingfu Zhu, Zhiming Zhang, Xuanyu Zhang, Qing Yang and Dongliang Xu},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.338},
  url = {https://doi.org/10.18653/v1/2025.acl-long.338}
}
```

- [← previous TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configuration](https://md.research.elamur.ai/free/en/papers/li-2025-taco-enhancing-multimodal-context-learning/)
- [next → VISA: Retrieval Augmented Generation with Visual Source Attribution](https://md.research.elamur.ai/free/en/papers/ma-2025-visa-retrieval-augmented-generation-visual/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/)

## References

43 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732.
2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
3. Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://md.research.elamur.ai/free/en/papers/cai-2024-medusa-simple-llm-inference-acceleration/). Preprint, arXiv:2401.10774.
4. Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating Large Language Model Decoding with Speculative Sampling. Preprint, arXiv:2302.01318. [find](https://scholar.google.com/scholar?q=Charlie+Chen%2C+Sebastian+Borgeaud%2C+Geoffrey+Irving%2C+Jean%2DBaptiste+Lespiau%2C+Laurent+Sifre%2C+and+John+Jumper%2E+2023%2E+Accelerating+Large+Language+Model+Decoding+with+Speculative+Sampling%2E+Preprint%2C+arXiv%3A23)
5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
6. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+and+Eric+P%2E+Xing%2E+2023%2E+Vicuna%3A+An+open%2Dsource+cha)
7. Tri Dao. 2024. [FlashAttention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). In International Conference on Learning Representations (ICLR).
8. DeepSeek-AI. 2024. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. Preprint, arXiv:2405.04434. [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2E+2024%2E+Deepseek%2Dv2%3A+A+strong%2C+economical%2C+and+efficient+mixture%2Dof%2Dexperts+language+model%2E+Preprint%2C+arXiv%3A2405%2E04434%2E)
9. Elias Frantar and Dan Alistarh. 2023. SparseGPT: Massive language models can be accurately pruned in one-shot. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10323–10337. PMLR. [find](https://scholar.google.com/scholar?q=Elias+Frantar+and+Dan+Alistarh%2E+2023%2E+SparseGPT%3A+Massive+language+models+can+be+accurately+pruned+in+one%2Dshot%2E+In+Proceedings+of+the+40th+International+Conference+on+Machine+Learning%2C+volume+202+of+Pr)
10. Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024. [Break the Sequential Dependency of LLM Inference Using Lookahead Decoding](https://md.research.elamur.ai/free/en/papers/fu-2024-break-sequential-dependency-llm-inference/). Preprint, arXiv:2402.02057.
11. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805.
12. Zhuocheng Gong, Jiahao Liu, Ziyue Wang, Pengfei Wu, Jingang Wang, Xunliang Cai, Dongyan Zhao, and Rui Yan. 2024. Graph-structured speculative decoding. In Findings of the Association for Computational Linguistics ACL 2024, pages 11404–11415. [find](https://scholar.google.com/scholar?q=Zhuocheng+Gong%2C+Jiahao+Liu%2C+Ziyue+Wang%2C+Pengfei+Wu%2C+Jingang+Wang%2C+Xunliang+Cai%2C+Dongyan+Zhao%2C+and+Rui+Yan%2E+2024%2E+Graph%2Dstructured+speculative+decoding%2E+In+Findings+of+the+Association+for+Computational)
13. Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D Lee, and Di He. 2023. [Rest: Retrieval-based speculative decoding](https://md.research.elamur.ai/free/en/papers/he-2023-rest-retrieval-based-speculative-decoding/). arXiv preprint arXiv:2311.08252.
14. Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. 2024. [SqueezeLLM: Dense-and-Sparse Quantization](https://md.research.elamur.ai/free/en/papers/kim-2023-squeezellm-dense-sparse-quantization/). Preprint, arXiv:2306.07629.
15. Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. 2024. CLLMs: Consistency Large Language Models. Preprint, arXiv:2403.00835. — [arXiv:2403.00835](https://arxiv.org/abs/2403.00835) [find](https://scholar.google.com/scholar?q=Siqi+Kou%2C+Lanxiang+Hu%2C+Zhezhi+He%2C+Zhijie+Deng%2C+and+Hao+Zhang%2E+2024%2E+CLLMs%3A+Consistency+Large+Language+Models%2E+Preprint%2C+arXiv%3A2403%2E00835%2E+%E2%80%94+%5BarXiv%3A2403%2E00835%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2403%2E00835%29)
16. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast Inference from Transformers via Speculative Decoding. In Proceedings of the 40th International Conference on Machine Learning, pages 19274–19286. PMLR. [find](https://scholar.google.com/scholar?q=Yaniv+Leviathan%2C+Matan+Kalman%2C+and+Yossi+Matias%2E+2023%2E+Fast+Inference+from+Transformers+via+Speculative+Decoding%2E+In+Proceedings+of+the+40th+International+Conference+on+Machine+Learning%2C+pages+19274%E2%80%931)
17. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval). [↗](https://github.com/tatsu-lab/alpaca_eval)
18. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024a. [Eagle-2: Faster inference of language models with dynamic draft trees](https://md.research.elamur.ai/free/en/papers/li-2024-eagle-2-faster-inference-language/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7421–7432.
19. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024b. [Eagle: Speculative sampling requires rethinking feature uncertainty](https://md.research.elamur.ai/free/en/papers/li-2024-eagle-speculative-sampling-requires-rethinking/). In Forty-first International Conference on Machine Learning.
20. Feng Lin, Hanling Yi, Hongbin Li, Yifan Yang, Xiaotian Yu, Guangming Lu, and Rong Xiao. 2024. BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Models. Preprint, arXiv:2401.12522. — [arXiv:2401.12522](https://arxiv.org/abs/2401.12522) [find](https://scholar.google.com/scholar?q=Feng+Lin%2C+Hanling+Yi%2C+Hongbin+Li%2C+Yifan+Yang%2C+Xiaotian+Yu%2C+Guangming+Lu%2C+and+Rong+Xiao%2E+2024%2E+BiTA%3A+Bi%2DDirectional+Tuning+for+Lossless+Acceleration+in+Large+Language+Models%2E+Preprint%2C+arXiv%3A2401%2E12522)
21. Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Kai Han, and Yunhe Wang. 2024a. Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting. Preprint, arXiv:2404.18911. — [arXiv:2404.18911](https://arxiv.org/abs/2404.18911) [find](https://scholar.google.com/scholar?q=Fangcheng+Liu%2C+Yehui+Tang%2C+Zhenhua+Liu%2C+Yunsheng+Ni%2C+Kai+Han%2C+and+Yunhe+Wang%2E+2024a%2E+Kangaroo%3A+Lossless+Self%2DSpeculative+Decoding+via+Double+Early+Exiting%2E+Preprint%2C+arXiv%3A2404%2E18911%2E+%E2%80%94+%5BarXiv%3A2404%2E18)
22. Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. 2024b. [Online speculative decoding](https://md.research.elamur.ai/free/en/papers/liu-2023-online-speculative-decoding/). In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 31131–31146. PMLR.
23. Meta. 2024. Introducing meta llama 3: The most capable openly available llm to date. [find](https://scholar.google.com/scholar?q=Meta%2E+2024%2E+Introducing+meta+llama+3%3A+The+most+capable+openly+available+llm+to+date%2E)
24. Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. 2024. SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculat… [find](https://scholar.google.com/scholar?q=Xupeng+Miao%2C+Gabriele+Oliaro%2C+Zhihao+Zhang%2C+Xinhao+Cheng%2C+Zeyu+Wang%2C+Zhengxin+Zhang%2C+Rae+Ying+Yee+Wong%2C+Alan+Zhu%2C+Lijie+Yang%2C+Xiaoxiang+Shi%2C+Chunan+Shi%2C+Zhuoming+Chen%2C+Daiyaan+Arfeen%2C+Reyna+Abhyankar%2C)
25. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). abs/2303.08774.
26. CoRR, Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950.
27. Apoorv Saxena. 2023. Prompt lookup decoding. [find](https://scholar.google.com/scholar?q=Apoorv+Saxena%2E+2023%2E+Prompt+lookup+decoding%2E)
28. Noam Shazeer. 2019. Fast Transformer Decoding: One Write-Head is All You Need. Preprint, arXiv:1911.02150. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+2019%2E+Fast+Transformer+Decoding%3A+One+Write%2DHead+is+All+You+Need%2E+Preprint%2C+arXiv%3A1911%2E02150%2E)
29. Benjamin Frederick Spector and Christopher Re. Accelerating llm inference with staged speculative decoding. In Workshop on Efficient Systems for Foundation Models@ ICML2023. [find](https://scholar.google.com/scholar?q=Benjamin+Frederick+Spector+and+Christopher+Re%2E+Accelerating+llm+inference+with+staged+speculative+decoding%2E+In+Workshop+on+Efficient+Systems+for+Foundation+Models%40+ICML2023%2E)
30. Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. 2023. Spectr: Fast speculative decoding via optimal transport. In Thirty-seventh Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Ziteng+Sun%2C+Ananda+Theertha+Suresh%2C+Jae+Hun+Ro%2C+Ahmad+Beirami%2C+Himanshu+Jain%2C+and+Felix+Yu%2E+2023%2E+Spectr%3A+Fast+speculative+decoding+via+optimal+transport%2E+In+Thirty%2Dseventh+Conference+on+Neural+Inform)
31. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
32. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. 2024a. [A survey on large language model based autonomous agents](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-large-language-model-based/). Frontiers of Computer Science, 18:186345.
33. Yixuan Wang, Xianzhen Luo, Fuxuan Wei, Yijun Liu, Qingfu Zhu, Xuanyu Zhang, Qing Yang, Dongliang Xu, and Wanxiang Che. 2024b. Make some noise: Unlocking language model parallel inference capability through noisy training. arXiv preprint arXiv:2406.17404. [find](https://scholar.google.com/scholar?q=Yixuan+Wang%2C+Xianzhen+Luo%2C+Fuxuan+Wei%2C+Yijun+Liu%2C+Qingfu+Zhu%2C+Xuanyu+Zhang%2C+Qing+Yang%2C+Dongliang+Xu%2C+and+Wanxiang+Che%2E+2024b%2E+Make+some+noise%3A+Unlocking+language+model+parallel+inference+capability+th)
34. Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. 2023. [Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation](https://md.research.elamur.ai/free/en/papers/xia-2023-speculative-decoding-exploiting-speculative-execution/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3909–3925, Singapore. Association for Computation…
35. Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. 2024. [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://md.research.elamur.ai/free/en/papers/xia-2024-unlocking-efficiency-large-language-model/). Preprint, arXiv:2401.07851.
36. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+2024%2E+Efficient+streaming+language+models+with+attention+sinks%2E+In+The+Twelfth+International+Conference+on+Learning+Representations)
37. Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. 2024. OneBit: Towards Extremely Low-bit Large Language Models. Preprint, arXiv:2402.11295. — [arXiv:2402.11295](https://arxiv.org/abs/2402.11295) [find](https://scholar.google.com/scholar?q=Yuzhuang+Xu%2C+Xu+Han%2C+Zonghan+Yang%2C+Shuo+Wang%2C+Qingfu+Zhu%2C+Zhiyuan+Liu%2C+Weidong+Liu%2C+and+Wanxiang+Che%2E+2024%2E+OneBit%3A+Towards+Extremely+Low%2Dbit+Large+Language+Models%2E+Preprint%2C+arXiv%3A2402%2E11295%2E+%E2%80%94+%5BarXi)
38. Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023. Inference with reference: Lossless acceleration of large language models. Preprint, arXiv:2304.04487. — [arXiv:2304.04487](https://arxiv.org/abs/2304.04487) [find](https://scholar.google.com/scholar?q=Nan+Yang%2C+Tao+Ge%2C+Liang+Wang%2C+Binxing+Jiao%2C+Daxin+Jiang%2C+Linjun+Yang%2C+Rangan+Majumder%2C+and+Furu+Wei%2E+2023%2E+Inference+with+reference%3A+Lossless+acceleration+of+large+language+models%2E+Preprint%2C+arXiv%3A230)
39. Hongyi Yuan, Keming Lu, Fei Huang, Zheng Yuan, and Chang Zhou. 2024. Speculative contrastive decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hongyi+Yuan%2C+Keming+Lu%2C+Fei+Huang%2C+Zheng+Yuan%2C+and+Chang+Zhou%2E+2024%2E+Speculative+contrastive+decoding%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volum)
40. Weilin Zhao, Yuxiang Huang, Xu Han, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. 2024. Ouroboros: Speculative decoding with large model enhanced drafting. arXiv preprint arXiv:2402.13720. — [arXiv:2402.13720](https://arxiv.org/abs/2402.13720) [find](https://scholar.google.com/scholar?q=Weilin+Zhao%2C+Yuxiang+Huang%2C+Xu+Han%2C+Chaojun+Xiao%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2024%2E+Ouroboros%3A+Speculative+decoding+with+large+model+enhanced+drafting%2E+arXiv+preprint+arXiv%3A2402%2E13720%2E+%E2%80%94+%5BarXiv%3A2402)
41. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Preprint, arXiv:2306.05685.
42. Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-François Kagy, and Rishabh Agarwal. 2024a. Distillspec: Improving speculative decoding via knowledge distillation. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yongchao+Zhou%2C+Kaifeng+Lyu%2C+Ankit+Singh+Rawat%2C+Aditya+Krishna+Menon%2C+Afshin+Rostamizadeh%2C+Sanjiv+Kumar%2C+Jean%2DFran%C3%A7ois+Kagy%2C+and+Rishabh+Agarwal%2E+2024a%2E+Distillspec%3A+Improving+speculative+decoding+via+)
43. Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, et al. 2024b. [A survey on efficient inference for large language models](https://md.research.elamur.ai/free/en/papers/zhou-2024-survey-efficient-inference-large-language/). arXiv preprint arXiv:2404.14294.

*Updated 2 November 2026*