---
title: "Medusa: Simple LLM Inference Acceleration — Cai 2024"
description: "Generating text with an LLM is slow because tokens come out one at a time."
authors: ["Tianle Cai", "Yuhong Li", "Zhengyang Geng", "Hongwu Peng", "Jason D. Lee", "Deming Chen", "Tri Dao"]
year: 2024
venue: "arXiv:2401.10774"
source: "https://arxiv.org/abs/2401.10774"
doi: "10.48550/arxiv.2401.10774"
arxiv: "2401.10774"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cai-2024-medusa-simple-llm-inference-acceleration/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cai-2024-medusa-simple-llm-inference-acceleration/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

# Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

**Authors:** [Tianle Cai](https://md.research.elamur.ai/free/en/authors/tianle-cai/), [Yuhong Li](https://md.research.elamur.ai/free/en/authors/yuhong-li/), Zhengyang Geng, Hongwu Peng, Jason D. Lee, [Deming Chen](https://md.research.elamur.ai/free/en/authors/deming-chen/), Tri Dao · **Year:** 2024 · **Venue:** arXiv:2401.10774 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2401.10774) · [PDF](https://arxiv.org/pdf/2401.10774) · [DOI 10.48550/arxiv.2401.10774](https://doi.org/10.48550/arxiv.2401.10774) · [arXiv 2401.10774](https://arxiv.org/abs/2401.10774) · [OpenAlex](https://openalex.org/W4391124766)

**Citations:** 14 citations (openalex-snapshot · 2026-10-14) · [257](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Generating text with an LLM is slow because tokens come out one at a time. Each step has to read all of the model's weights from HBM, so the accelerator spends most of its time waiting on memory rather than computing. Speculative decoding helps, but it needs a separate draft model, which is costly to obtain and awkward to deploy in distributed systems. Medusa removes the draft model. It adds a few lightweight extra heads to the final hidden state of the existing LLM, and each head guesses a token further ahead. The top guesses from all heads are combined into a tree of candidate continuations. A special attention mask lets the model check every branch in a single forward pass and keep the longest accepted prefix. There are two training modes: Medusa-1 trains only the heads on a frozen backbone, and Medusa-2 trains the heads and the backbone together with a recipe meant to protect output quality. For models without usable training data, the authors generate it through self-distillation. They also propose a "typical acceptance" rule as a faster alternative to rejection sampling. The result is a 2.3–2.8× speedup on chat models without a noticeable loss in output quality.

## Key points
- Models tested: Vicuna-7B, Vicuna-13B, Vicuna-33B and Zephyr-7B. Experiments focus on batch size 1, which matches local, single-user serving.
- Medusa-1 (frozen backbone, quality unchanged) gives just over 2.2× speedup. Medusa-2 (joint training) reaches 2.3–2.8×. On Vicuna-7B the figures were 2.18× and 2.83×, while naive joint fine-tuning without the recipe made outputs worse.
- Ablation of cumulative speedups: heads alone ~1.5×, adding tree attention ~1.9×, an optimized tree layout ~2.2×, and Medusa-2 training ~2.8×.
- Each head is a single residual SiLU feed-forward layer. Its output projection starts as a copy of the original LM head and its inner weight starts at zero, so at first the heads predict exactly what the base model does. Heads can be trained on one GPU, and QLoRA-style quantization of the frozen backbone is also possible.
- Self-distillation produces training data for models whose fine-tuning data is private or that went through RLHF. Typical acceptance uses a temperature-dependent threshold to accept plausible candidates and increase accepted length. Published at ICML 2024.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/) · [Computing & GPUs](https://md.research.elamur.ai/free/en/shelves/computing/)

## Cited by

Cited by

257 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 250 not yet.

- [Babies Learn to Look Ahead: Multi-Token Prediction in Small LMs](https://md.research.elamur.ai/free/en/papers/aynetdinov-2025-babies-learn-look-ahead-multi/)
- [Break the Sequential Dependency of LLM Inference Using Lookahead Decoding](https://md.research.elamur.ai/free/en/papers/fu-2024-break-sequential-dependency-llm-inference/)
- [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://md.research.elamur.ai/free/en/papers/li-2024-eagle-2-faster-inference-language/)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://md.research.elamur.ai/free/en/papers/li-2024-snapkv-llm-knows-looking-before/)
- [Chronological Thinking in Full-Duplex Spoken Dialogue Language Models](https://md.research.elamur.ai/free/en/papers/wu-2026-chronological-thinking-full-duplex-spoken/)
- [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://md.research.elamur.ai/free/en/papers/xia-2024-unlocking-efficiency-large-language-model/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{cai2024medusa,
  title = {{Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads}},
  author = {Cai, Tianle and Li, Yuhong and Geng, Zhengyang and Peng, Hongwu and Lee, Jason D. and Chen, Deming and Dao, Tri},
  year = {2024},
  doi = {10.48550/arxiv.2401.10774},
  eprint = {2401.10774},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.10774}
}
```

- [← previous MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning](https://md.research.elamur.ai/free/en/papers/tang-2024-medagents-large-language-models-as/)
- [next → MenakBERT — Hebrew Diacriticizer](https://md.research.elamur.ai/free/en/papers/hbo-2024-cohen-menakbert-hebrew-diacriticizer/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

41 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023. [find](https://scholar.google.com/scholar?q=Ainslie%2C+J%2E%2C+Lee%2DThorp%2C+J%2E%2C+de+Jong%2C+M%2E%2C+Zemlyanskiy%2C+Y%2E%2C+Lebro%CC%81n%2C+F%2E%2C+and+Sanghai%2C+S%2E+Gqa%3A+Training+generalized+multi%2Dquery+transformer+models+from+multi%2Dhead+checkpoints%2E+arXiv+preprint+arXiv%3A2305%2E1) [our card](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/)
2. Axolotl. Axolotl. [https://github.com/](https://github.com/) OpenAccess-AI-Collective/axolotl, 2023. [↗](https://github.com/)
3. Dubois, Y., Li, X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023. [find](https://scholar.google.com/scholar?q=Dubois%2C+Y%2E%2C+Li%2C+X%2E%2C+Taori%2C+R%2E%2C+Zhang%2C+T%2E%2C+Gulrajani%2C+I%2E%2C+Ba%2C+J%2E%2C+Guestrin%2C+C%2E%2C+Liang%2C+P%2E%2C+and+Hashimoto%2C+T%2E+B%2E+Alpacafarm%3A+A+simulation+framework+for+methods+that+learn+from+human+feedback%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/)
4. Basu, S., Ramachandran, G. S., Keskar, N. S., and Varshney, L. R. {MIROSTAT}: A {neural} {text} {decoding} {algorithm} {that} {directly} {controls} {perplexity}. In International Conference on Learning Representations, Elfwing, S., Uchibe, E., and Doya, K. Sigmoid-weighted linear units for neural ne… — [doi:10.1016/j.neunet.2017.12.012](https://doi.org/10.1016/j.neunet.2017.12.012) [find](https://scholar.google.com/scholar?q=Basu%2C+S%2E%2C+Ramachandran%2C+G%2E+S%2E%2C+Keskar%2C+N%2E+S%2E%2C+and+Varshney%2C+L%2E+R%2E+%7BMIROSTAT%7D%3A+A+%7Bneural%7D+%7Btext%7D+%7Bdecoding%7D+%7Balgorithm%7D+%7Bthat%7D+%7Bdirectly%7D+%7Bcontrols%7D+%7Bperplexity%7D%2E+In+International+Conference+on+Learnin)
5. Fan, A., Lewis, M., and Dauphin, Y. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2018. doi: 10.18653/v1/p18-1082. [find](https://scholar.google.com/scholar?q=Fan%2C+A%2E%2C+Lewis%2C+M%2E%2C+and+Dauphin%2C+Y%2E+Hierarchical+neural+story+generation%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29%2E+Associatio)
6. Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. November 2022. doi: 10.48550/ARXIV.2211.17192. [find](https://scholar.google.com/scholar?q=Leviathan%2C+Y%2E%2C+Kalman%2C+M%2E%2C+and+Matias%2C+Y%2E+Fast+inference+from+transformers+via+speculative+decoding%2E+November+2022%2E+doi%3A+10%2E48550%2FARXIV%2E2211%2E17192%2E)
7. Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/](https://github.com/tatsu-lab/) alpaca_eval, 2023. [↗](https://github.com/tatsu-lab/)
8. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022. [find](https://scholar.google.com/scholar?q=Frantar%2C+E%2E%2C+Ashkboos%2C+S%2E%2C+Hoefler%2C+T%2E%2C+and+Alistarh%2C+D%2E+Gptq%3A+Accurate+post%2Dtraining+quantization+for+generative+pre%2Dtrained+transformers%2E+arXiv+preprint+arXiv%3A2210%2E17323%2C+2022%2E)
9. Google. Palm 2 technical report, 2023. URL [https://ai.google/static/documents/](https://ai.google/static/documents/) palm2techreport.pdf. [↗](https://ai.google/static/documents/)
10. Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., and Han, S. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023. [find](https://scholar.google.com/scholar?q=Lin%2C+J%2E%2C+Tang%2C+J%2E%2C+Tang%2C+H%2E%2C+Yang%2C+S%2E%2C+Dang%2C+X%2E%2C+and+Han%2C+S%2E+Awq%3A+Activation%2Daware+weight+quantization+for+llm+compression+and+acceleration%2E+arXiv+preprint+arXiv%3A2306%2E00978%2C+2023%2E)
11. Hewitt, J., Manning, C. D., and Liang, P. Truncation sampling as language model desmoothing. October 2022. doi: 10.48550/ARXIV.2210.15191. [find](https://scholar.google.com/scholar?q=Hewitt%2C+J%2E%2C+Manning%2C+C%2E+D%2E%2C+and+Liang%2C+P%2E+Truncation+sampling+as+language+model+desmoothing%2E+October+2022%2E+doi%3A+10%2E48550%2FARXIV%2E2210%2E15191%2E)
12. Meister, C., Wiher, G., Pimentel, T., and Cotterell, R. On the probability-quality paradox in language generation. March 2022. doi: 10.48550/ARXIV.2203.17217. [find](https://scholar.google.com/scholar?q=Meister%2C+C%2E%2C+Wiher%2C+G%2E%2C+Pimentel%2C+T%2E%2C+and+Cotterell%2C+R%2E+On+the+probability%2Dquality+paradox+in+language+generation%2E+March+2022%2E+doi%3A+10%2E48550%2FARXIV%2E2203%2E17217%2E)
13. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. [find](https://scholar.google.com/scholar?q=Hoffmann%2C+J%2E%2C+Borgeaud%2C+S%2E%2C+Mensch%2C+A%2E%2C+Buchatskaya%2C+E%2E%2C+Cai%2C+T%2E%2C+Rutherford%2C+E%2E%2C+Casas%2C+D%2E+d%2E+L%2E%2C+Hendricks%2C+L%2E+A%2E%2C+Welbl%2C+J%2E%2C+Clark%2C+A%2E%2C+et+al%2E+Training+compute%2Doptimal+large+language+models%2E+arXiv+)
14. Meister, C., Pimentel, T., Wiher, G., and Cotterell, R. Locally typical sampling. Transactions of the Association for Computational Linguistics, 11:102–121, 2023. [find](https://scholar.google.com/scholar?q=Meister%2C+C%2E%2C+Pimentel%2C+T%2E%2C+Wiher%2C+G%2E%2C+and+Cotterell%2C+R%2E+Locally+typical+sampling%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+11%3A102%E2%80%93121%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/meister-2023-locally-typical-sampling/)
15. Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Wong, R. Y. Y., Chen, Z., Arfeen, D., Abhyankar, R., and Jia, Z. Specinfer: Accelerating generative llm serving with speculative inference and token tree verification. arXiv preprint arXiv:2305.09781, 2023. [find](https://scholar.google.com/scholar?q=Miao%2C+X%2E%2C+Oliaro%2C+G%2E%2C+Zhang%2C+Z%2E%2C+Cheng%2C+X%2E%2C+Wang%2C+Z%2E%2C+Wong%2C+R%2E+Y%2E+Y%2E%2C+Chen%2C+Z%2E%2C+Arfeen%2C+D%2E%2C+Abhyankar%2C+R%2E%2C+and+Jia%2C+Z%2E+Specinfer%3A+Accelerating+generative+llm+serving+with+speculative+inference+and+tok)
16. Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=rygGQyrFvH. [↗](https://openreview.net/forum?)
17. NVIDIA. Nvidia a100 tensor core gpu. OpenAI. Gpt-4 technical report, 2023. [find](https://scholar.google.com/scholar?q=NVIDIA%2E+Nvidia+a100+tensor+core+gpu%2E+OpenAI%2E+Gpt%2D4+technical+report%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
18. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., and Chen, W. Lora: Low-rank adaptation of large language models. ICLR, 2021. [find](https://scholar.google.com/scholar?q=Hu%2C+E%2E+J%2E%2C+Shen%2C+Y%2E%2C+Wallis%2C+P%2E%2C+Allen%2DZhu%2C+Z%2E%2C+Li%2C+Y%2E%2C+Wang%2C+S%2E%2C+and+Chen%2C+W%2E+Lora%3A+Low%2Drank+adaptation+of+large+language+models%2E+ICLR%2C+2021%2E)
19. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. [find](https://scholar.google.com/scholar?q=Ouyang%2C+L%2E%2C+Wu%2C+J%2E%2C+Jiang%2C+X%2E%2C+Almeida%2C+D%2E%2C+Wainwright%2C+C%2E+L%2E%2C+Mishkin%2C+P%2E%2C+Zhang%2C+C%2E%2C+Agarwal%2C+S%2E%2C+Slama%2C+K%2E%2C+Ray%2C+A%2E%2C+et+al%2E+Training+language+models+to+follow+instructions+with+human+feedback%2E+arXi)
20. Joao Gante. Assisted generation: a new direction toward low-latency text generation, 2023. URL [https://huggingface.co/blog/](https://huggingface.co/blog/) assisted-generation. [↗](https://huggingface.co/blog/)
21. Pan, J. Tiny vicuna 1b. [https://huggingface.co/](https://huggingface.co/) Jiayi-Pan/Tiny-Vicuna-1B, 2023. [↗](https://huggingface.co/)
22. Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629, 2023. [find](https://scholar.google.com/scholar?q=Kim%2C+S%2E%2C+Hooper%2C+C%2E%2C+Gholami%2C+A%2E%2C+Dong%2C+Z%2E%2C+Li%2C+X%2E%2C+Shen%2C+S%2E%2C+Mahoney%2C+M%2E+W%2E%2C+and+Keutzer%2C+K%2E+Squeezellm%3A+Dense%2Dand%2Dsparse+quantization%2E+arXiv+preprint+arXiv%3A2306%2E07629%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2023-squeezellm-dense-sparse-quantization/)
23. Pillutla, K., Swayamdipta, S., Zellers, R., Thickstun, J., Welleck, S., Choi, Y., and Harchaoui, Z. MAUVE: Measuring the gap between neural text and human text using divergence frontiers. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing… [find](https://scholar.google.com/scholar?q=Pillutla%2C+K%2E%2C+Swayamdipta%2C+S%2E%2C+Zellers%2C+R%2E%2C+Thickstun%2C+J%2E%2C+Welleck%2C+S%2E%2C+Choi%2C+Y%2E%2C+and+Harchaoui%2C+Z%2E+MAUVE%3A+Measuring+the+gap+between+neural+text+and+human+text+using+divergence+frontiers%2E+In+Beygelzim) [our card](https://md.research.elamur.ai/free/en/papers/pillutla-2021-mauve-measuring-gap-between-neural/)
24. Kim, Y. and Rush, A. M. Sequence-level knowledge distillation. EMNLP, 2016. [find](https://scholar.google.com/scholar?q=Kim%2C+Y%2E+and+Rush%2C+A%2E+M%2E+Sequence%2Dlevel+knowledge+distillation%2E+EMNLP%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/)
25. Kumar, A., Raghunathan, A., Jones, R., Ma, T., and Liang, P. Fine-tuning can distort pretrained features and underperform out-of-distribution. International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=Kumar%2C+A%2E%2C+Raghunathan%2C+A%2E%2C+Jones%2C+R%2E%2C+Ma%2C+T%2E%2C+and+Liang%2C+P%2E+Fine%2Dtuning+can+distort+pretrained+features+and+underperform+out%2Dof%2Ddistribution%2E+International+Conference+on+Learning+Representations%2C+202)
26. Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Levskaya, A., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference. November 2022. doi: 10.48550/ARXIV.2211.05102. [find](https://scholar.google.com/scholar?q=Pope%2C+R%2E%2C+Douglas%2C+S%2E%2C+Chowdhery%2C+A%2E%2C+Devlin%2C+J%2E%2C+Bradbury%2C+J%2E%2C+Levskaya%2C+A%2E%2C+Heek%2C+J%2E%2C+Xiao%2C+K%2E%2C+Agrawal%2C+S%2E%2C+and+Dean%2C+J%2E+Efficiently+scaling+transformer+inference%2E+November+2022%2E+doi%3A+10%2E48550%2FARXI)
27. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. [find](https://scholar.google.com/scholar?q=Kwon%2C+W%2E%2C+Li%2C+Z%2E%2C+Zhuang%2C+S%2E%2C+Sheng%2C+Y%2E%2C+Zheng%2C+L%2E%2C+Yu%2C+C%2E+H%2E%2C+Gonzalez%2C+J%2E+E%2E%2C+Zhang%2C+H%2E%2C+and+Stoica%2C+I%2E+Efficient+memory+management+for+large+language+model+serving+with+pagedattention%2E+In+Proceedin)
28. ShareGPT. ShareGPT. [https://huggingface.co/datasets/Aeala/ShareGPT_Vicuna_](https://huggingface.co/datasets/Aeala/ShareGPT_Vicuna_) unfiltered, 2023. [↗](https://huggingface.co/datasets/Aeala/ShareGPT_Vicuna_)
29. Shazeer, N. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019. [find](https://scholar.google.com/scholar?q=Shazeer%2C+N%2E+Fast+transformer+decoding%3A+One+write%2Dhead+is+all+you+need%2E+arXiv+preprint+arXiv%3A1911%2E02150%2C+2019%2E)
30. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. [find](https://scholar.google.com/scholar?q=Zheng%2C+L%2E%2C+Chiang%2C+W%2E%2DL%2E%2C+Sheng%2C+Y%2E%2C+Zhuang%2C+S%2E%2C+Wu%2C+Z%2E%2C+Zhuang%2C+Y%2E%2C+Lin%2C+Z%2E%2C+Li%2C+Z%2E%2C+Li%2C+D%2E%2C+Xing%2C+E%2E+P%2E%2C+Zhang%2C+H%2E%2C+Gonzalez%2C+J%2E+E%2E%2C+and+Stoica%2C+I%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+a)
31. Spector, B. and Re, C. Accelerating llm inference with staged speculative decoding. arXiv preprint arXiv:2308.04623, 2023. — [arXiv:2308.04623](https://arxiv.org/abs/2308.04623) [find](https://scholar.google.com/scholar?q=Spector%2C+B%2E+and+Re%2C+C%2E+Accelerating+llm+inference+with+staged+speculative+decoding%2E+arXiv+preprint+arXiv%3A2308%2E04623%2C+2023%2E+%E2%80%94+%5BarXiv%3A2308%2E04623%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2308%2E04623%29)
32. Stern, M., Shazeer, N. M., and Uszkoreit, J. Blockwise parallel decoding for deep autoregressive models. Neural Information Processing Systems, 2018. [find](https://scholar.google.com/scholar?q=Stern%2C+M%2E%2C+Shazeer%2C+N%2E+M%2E%2C+and+Uszkoreit%2C+J%2E+Blockwise+parallel+decoding+for+deep+autoregressive+models%2E+Neural+Information+Processing+Systems%2C+2018%2E)
33. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%2C+Martin%2C+L%2E%2C+Stone%2C+K%2E%2C+Albert%2C+P%2E%2C+Almahairi%2C+A%2E%2C+Babaei%2C+Y%2E%2C+Bashlykov%2C+N%2E%2C+Batra%2C+S%2E%2C+Bhargava%2C+P%2E%2C+Bhosale%2C+S%2E%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtuned+chat+models%2E+arXiv+prepri)
34. Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Rasul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., Sarrazin, N., Sanseviero, O., Rush, A. M., and Wolf, T. Zephyr: Direct distillation of lm alignment, 2023. [find](https://scholar.google.com/scholar?q=Tunstall%2C+L%2E%2C+Beeching%2C+E%2E%2C+Lambert%2C+N%2E%2C+Rajani%2C+N%2E%2C+Rasul%2C+K%2E%2C+Belkada%2C+Y%2E%2C+Huang%2C+S%2E%2C+von+Werra%2C+L%2E%2C+Fourrier%2C+C%2E%2C+Habib%2C+N%2E%2C+Sarrazin%2C+N%2E%2C+Sanseviero%2C+O%2E%2C+Rush%2C+A%2E+M%2E%2C+and+Wolf%2C+T%2E+Zephyr%3A+Direct+d)
35. Xia, H., Ge, T., Chen, S.-Q., Wei, F., and Sui, Z. Speculative decoding: Lossless speedup of autoregressive translation, 2023. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=H-VlwsYvVi. [↗](https://openreview.net/forum?)
36. Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087–38099. PMLR, 2023a. [find](https://scholar.google.com/scholar?q=Xiao%2C+G%2E%2C+Lin%2C+J%2E%2C+Seznec%2C+M%2E%2C+Wu%2C+H%2E%2C+Demouth%2C+J%2E%2C+and+Han%2C+S%2E+Smoothquant%3A+Accurate+and+efficient+post%2Dtraining+quantization+for+large+language+models%2E+In+International+Conference+on+Machine+Learnin)
37. Xiao, Y., Wu, L., Guo, J., Li, J., Zhang, M., Qin, T., and Liu, T.-y. A survey on non-autoregressive generation for neural machine translation and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023b. [find](https://scholar.google.com/scholar?q=Xiao%2C+Y%2E%2C+Wu%2C+L%2E%2C+Guo%2C+J%2E%2C+Li%2C+J%2E%2C+Zhang%2C+M%2E%2C+Qin%2C+T%2E%2C+and+Liu%2C+T%2E%2Dy%2E+A+survey+on+non%2Dautoregressive+generation+for+neural+machine+translation+and+beyond%2E+IEEE+Transactions+on+Pattern+Analysis+and+Mac)
38. Ying, C., Cai, T., Luo, S., Zheng, S., Ke, G., He, D., Shen, Y., and Liu, T.-Y. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 34:28877–28888, 2021. [find](https://scholar.google.com/scholar?q=Ying%2C+C%2E%2C+Cai%2C+T%2E%2C+Luo%2C+S%2E%2C+Zheng%2C+S%2E%2C+Ke%2C+G%2E%2C+He%2C+D%2E%2C+Shen%2C+Y%2E%2C+and+Liu%2C+T%2E%2DY%2E+Do+transformers+really+perform+badly+for+graph+representation%3F+Advances+in+Neural+Information+Processing+Systems%2C+34%3A288)
39. Zhang, P., Zeng, G., Wang, T., and Lu, W. Tinyllama: An open-source small language model, 2024. [find](https://scholar.google.com/scholar?q=Zhang%2C+P%2E%2C+Zeng%2C+G%2E%2C+Wang%2C+T%2E%2C+and+Lu%2C+W%2E+Tinyllama%3A+An+open%2Dsource+small+language+model%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2024-tinyllama-open-source-small-language/)
40. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. [find](https://scholar.google.com/scholar?q=Zhang%2C+S%2E%2C+Roller%2C+S%2E%2C+Goyal%2C+N%2E%2C+Artetxe%2C+M%2E%2C+Chen%2C+M%2E%2C+Chen%2C+S%2E%2C+Dewan%2C+C%2E%2C+Diab%2C+M%2E%2C+Li%2C+X%2E%2C+Lin%2C+X%2E+V%2E%2C+et+al%2E+Opt%3A+Open+pre%2Dtrained+transformer+language+models%2E+arXiv+preprint+arXiv%3A2205%2E01068%2C+2)
41. Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., et al. H 2 o: Heavy-hitter oracle for efficient generative inference of large language models. arXiv preprint arXiv:2306.14048, 2023. [find](https://scholar.google.com/scholar?q=Zhang%2C+Z%2E%2C+Sheng%2C+Y%2E%2C+Zhou%2C+T%2E%2C+Chen%2C+T%2E%2C+Zheng%2C+L%2E%2C+Cai%2C+R%2E%2C+Song%2C+Z%2E%2C+Tian%2C+Y%2E%2C+Re%CC%81%2C+C%2E%2C+Barrett%2C+C%2E%2C+et+al%2E+H+2+o%3A+Heavy%2Dhitter+oracle+for+efficient+generative+inference+of+large+language+models%2E+a)