--- title: "Break the Sequential Dependency of LLM Inference — Fu 2024" description: "Generating text with LLMs one token at a time is slow." authors: ["Yichao Fu", "Peter Bailis", "Ion Stoica", "Hao Zhang"] year: 2024 venue: "arXiv:2402.02057" source: "https://arxiv.org/abs/2402.02057" doi: "10.48550/arxiv.2402.02057" arxiv: "2402.02057" licence: "arXiv — non-exclusive license to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/fu-2024-break-sequential-dependency-llm-inference/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/fu-2024-break-sequential-dependency-llm-inference/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/) / Break the Sequential Dependency of LLM Inference Using Lookahead Decoding # Break the Sequential Dependency of LLM Inference Using Lookahead Decoding **Authors:** Yichao Fu, Peter Bailis, [Ion Stoica](https://md.research.elamur.ai/free/en/authors/ion-stoica/), [Hao Zhang](https://md.research.elamur.ai/free/en/authors/hao-zhang/) · **Year:** 2024 · **Venue:** arXiv:2402.02057 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2402.02057) · [PDF](https://arxiv.org/pdf/2402.02057) · [DOI 10.48550/arxiv.2402.02057](https://doi.org/10.48550/arxiv.2402.02057) · [arXiv 2402.02057](https://arxiv.org/abs/2402.02057) · [OpenAlex](https://openalex.org/W4391590701) **Citations:** 3 citations (openalex-snapshot · 2026-10-14) · [109](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Generating text with LLMs one token at a time is slow. Each step is limited by memory bandwidth, so most of the GPU's compute sits idle. Speculative decoding helps, but it needs a draft model, which is hard to train and often fails to transfer across base models or datasets. The authors propose Lookahead Decoding, which needs no extra model, training or datastore. It treats decoding as a nonlinear system solved by Jacobi iteration. A lookahead branch uses a 2D window over recent iteration history to produce many disjoint n-grams at once. These n-grams are cached in a pool, and a verification branch checks candidates in the same forward pass, so the output distribution stays exactly the same. The method works with FlashAttention, supports several sampling schemes and scales across GPUs ("Lookahead Parallelism"). On LLaMA-2 it reaches about 1.8x speedup on MT-Bench and up to 4x on code completion with 8 GPUs. This matters because it turns idle compute into lower latency without any auxiliary model. ## Key points - **Speedups:** about 1.8x on MT-Bench, and up to 4x on code tasks (HumanEval, ClassEval) using Lookahead Parallelism on 8 GPUs. Results are for LLaMA-2 models, with Python and CUDA implementations available at github.com/hao-ai-lab/LookaheadDecoding. - **Scaling law:** the number of decoding steps falls linearly as per-step log(FLOPs) grows. In other words, each further step reduction requires exponentially more compute per step. - **Tuning knobs:** W is the lookahead window, N is the n-gram size and G is the verification size. Near-optimal settings on A100 (with G = W) are: - 7B: W=15, N=5 - 13B: W=10, N=5 - 34B: W=7, N=5 - **Main limitation:** the configurations above cost roughly 120x, 80x and 56x extra FLOPs per step. Gains therefore shrink in compute-bound settings, such as serving large batches. Gains are also smaller on weaker GPUs: about 30% on RTX 3090 versus more than 50% on A100 for MT-Bench. - **Ablation:** both branches are needed, and balanced lookahead and verification branches beat lopsided setups. The method was compared against the prompt lookup implementation in transformers v4.37. Adding prompt-based n-grams to the pool raises the speedup further. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Computing & GPUs](https://md.research.elamur.ai/free/en/shelves/computing/) ## Cited by Cited by 109 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 107 not yet. - [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://md.research.elamur.ai/free/en/papers/li-2024-eagle-2-faster-inference-language/) - [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://md.research.elamur.ai/free/en/papers/xia-2024-unlocking-efficiency-large-language-model/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{fu2024break, title = {{Break the Sequential Dependency of LLM Inference Using Lookahead Decoding}}, author = {Fu, Yichao and Bailis, Peter and Stoica, Ion and Zhang, Hao}, year = {2024}, doi = {10.48550/arxiv.2402.02057}, eprint = {2402.02057}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2402.02057} } ``` - [← previous BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages](https://md.research.elamur.ai/free/en/papers/myung-2024-blend-benchmark-llms-everyday-knowledge/) - [next → Bridging Cultural Nuances in Dialogue Agents through Cultural Value Surveys](https://md.research.elamur.ai/free/en/papers/cao-2024-bridging-cultural-nuances-dialogue-agents/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/) ## References 43 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Automatic tensor parallelism for huggingface models, 2023. URL [https://www.deepspeed.ai/tutoria](https://www.deepspeed.ai/tutoria) ls/automatic-tensor-parallelism. [↗](https://www.deepspeed.ai/tutoria) 2. Aminabadi, R. Y., Rajbhandari, S., Awan, A. A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., et al. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking… [find](https://scholar.google.com/scholar?q=Aminabadi%2C+R%2E+Y%2E%2C+Rajbhandari%2C+S%2E%2C+Awan%2C+A%2E+A%2E%2C+Li%2C+C%2E%2C+Li%2C+D%2E%2C+Zheng%2C+E%2E%2C+Ruwase%2C+O%2E%2C+Smith%2C+S%2E%2C+Zhang%2C+M%2E%2C+Rasley%2C+J%2E%2C+et+al%2E+Deepspeed%2Dinference%3A+enabling+efficient+inference+of+transformer+models+) 3. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., and Sutton, C. Program synthesis with large language models, 2021. [find](https://scholar.google.com/scholar?q=Austin%2C+J%2E%2C+Odena%2C+A%2E%2C+Nye%2C+M%2E%2C+Bosma%2C+M%2E%2C+Michalewski%2C+H%2E%2C+Dohan%2C+D%2E%2C+Jiang%2C+E%2E%2C+Cai%2C+C%2E%2C+Terry%2C+M%2E%2C+Le%2C+Q%2E%2C+and+Sutton%2C+C%2E+Program+synthesis+with+large+language+models%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/) 4. Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. [find](https://scholar.google.com/scholar?q=Ba%2C+J%2E+L%2E%2C+Kiros%2C+J%2E+R%2E%2C+and+Hinton%2C+G%2E+E%2E+Layer+normalization%2E+arXiv+preprint+arXiv%3A1607%2E06450%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/) 5. Ben Allal, L., Muennighoff, N., Kumar Umapathi, L., Lipkin, B., and von Werra, L. A framework for the evaluation of code generation models. [https://github.com/bigcode-project/bigcode-evaluation-h](https://github.com/bigcode-project/bigcode-evaluation-h) arness, 2022. [↗](https://github.com/bigcode-project/bigcode-evaluation-h) 6. Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, T. Medusa: Simple llm inference acceleration framework with multiple decoding heads, 2024. [find](https://scholar.google.com/scholar?q=Cai%2C+T%2E%2C+Li%2C+Y%2E%2C+Geng%2C+Z%2E%2C+Peng%2C+H%2E%2C+Lee%2C+J%2E+D%2E%2C+Chen%2C+D%2E%2C+and+Dao%2C+T%2E+Medusa%3A+Simple+llm+inference+acceleration+framework+with+multiple+decoding+heads%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/cai-2024-medusa-simple-llm-inference-acceleration/) 7. Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023. [find](https://scholar.google.com/scholar?q=Chen%2C+C%2E%2C+Borgeaud%2C+S%2E%2C+Irving%2C+G%2E%2C+Lespiau%2C+J%2E%2DB%2E%2C+Sifre%2C+L%2E%2C+and+Jumper%2C+J%2E+Accelerating+large+language+model+decoding+with+speculative+sampling%2E+arXiv+preprint+arXiv%3A2302%2E01318%2C+2023%2E) 8. Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C.… [find](https://scholar.google.com/scholar?q=Chen%2C+M%2E%2C+Tworek%2C+J%2E%2C+Jun%2C+H%2E%2C+Yuan%2C+Q%2E%2C+de+Oliveira+Pinto%2C+H%2E+P%2E%2C+Kaplan%2C+J%2E%2C+Edwards%2C+H%2E%2C+Burda%2C+Y%2E%2C+Joseph%2C+N%2E%2C+Brockman%2C+G%2E%2C+Ray%2C+A%2E%2C+Puri%2C+R%2E%2C+Krueger%2C+G%2E%2C+Petrov%2C+M%2E%2C+Khlaaf%2C+H%2E%2C+Sastry%2C+G%2E%2C+Mis) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/) 9. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [find](https://scholar.google.com/scholar?q=Cobbe%2C+K%2E%2C+Kosaraju%2C+V%2E%2C+Bavarian%2C+M%2E%2C+Chen%2C+M%2E%2C+Jun%2C+H%2E%2C+Kaiser%2C+L%2E%2C+Plappert%2C+M%2E%2C+Tworek%2C+J%2E%2C+Hilton%2C+J%2E%2C+Nakano%2C+R%2E%2C+Hesse%2C+C%2E%2C+and+Schulman%2C+J%2E+Training+verifiers+to+solve+math+word+problems%2E+arXi) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/) 10. Dao, T. FlashAttention-2: Faster attention with better parallelism and work partitioning. 2023. [find](https://scholar.google.com/scholar?q=Dao%2C+T%2E+FlashAttention%2D2%3A+Faster+attention+with+better+parallelism+and+work+partitioning%2E+2023%2E) 11. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Dao%2C+T%2E%2C+Fu%2C+D%2E+Y%2E%2C+Ermon%2C+S%2E%2C+Rudra%2C+A%2E%2C+and+Re%CC%81%2C+C%2E+FlashAttention%3A+Fast+and+memory%2Defficient+exact+attention+with+IO%2Dawareness%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2022%2E) 12. Du, X., Liu, M., Wang, K., Wang, H., Liu, J., Chen, Y., Feng, J., Sha, C., Peng, X., and Lou, Y. Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation, 2023. [find](https://scholar.google.com/scholar?q=Du%2C+X%2E%2C+Liu%2C+M%2E%2C+Wang%2C+K%2E%2C+Wang%2C+H%2E%2C+Liu%2C+J%2E%2C+Chen%2C+Y%2E%2C+Feng%2C+J%2E%2C+Sha%2C+C%2E%2C+Peng%2C+X%2E%2C+and+Lou%2C+Y%2E+Classeval%3A+A+manually%2Dcrafted+benchmark+for+evaluating+llms+on+class%2Dlevel+code+generation%2C+2023%2E) 13. Gugger, S., Debut, L., Wolf, T., Schmid, P., Mueller, Z., Mangrulkar, S., Sun, M., and Bossan, B. Accelerate: Training and inference at scale made simple, efficient and adaptable. [https://github.com/huggingface/accelerate](https://github.com/huggingface/accelerate), 2022. [↗](https://github.com/huggingface/accelerate) 14. He, Z., Zhong, Z., Cai, T., Lee, J. D., and He, D. Rest: Retrieval-based speculative decoding. arXiv preprint arXiv:2311.08252, 2023. [find](https://scholar.google.com/scholar?q=He%2C+Z%2E%2C+Zhong%2C+Z%2E%2C+Cai%2C+T%2E%2C+Lee%2C+J%2E+D%2E%2C+and+He%2C+D%2E+Rest%3A+Retrieval%2Dbased+speculative+decoding%2E+arXiv+preprint+arXiv%3A2311%2E08252%2C+2023%2E) 15. Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration, 2020. [find](https://scholar.google.com/scholar?q=Holtzman%2C+A%2E%2C+Buys%2C+J%2E%2C+Du%2C+L%2E%2C+Forbes%2C+M%2E%2C+and+Choi%2C+Y%2E+The+curious+case+of+neural+text+degeneration%2C+2020%2E) 16. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E%2C+Sablayrolles%2C+A%2E%2C+Mensch%2C+A%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2E+S%2E%2C+Casas%2C+D%2E+d%2E+l%2E%2C+Bressand%2C+F%2E%2C+Lengyel%2C+G%2E%2C+Lample%2C+G%2E%2C+Saulnier%2C+L%2E%2C+et+al%2E+Mistral+7b%2E+arXiv+preprint+arXiv%3A2310%2E06825%2C+2023%2E) 17. Kool, W., van Hoof, H., and Welling, M. Ancestral gumbel-top-k sampling for sampling without replacement. Journal of Machine Learning Research, 21(47):1–36, 2020. URL [http://jmlr.org/papers/v21/19-985](http://jmlr.org/papers/v21/19-985) .html. [↗](http://jmlr.org/papers/v21/19-985) 18. Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274– 19286. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Leviathan%2C+Y%2E%2C+Kalman%2C+M%2E%2C+and+Matias%2C+Y%2E+Fast+inference+from+transformers+via+speculative+decoding%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+19274%E2%80%93+19286%2E+PMLR%2C+2023%2E) 19. Li, Y., Zhang, C., and Zhang, H. Eagle: Lossless acceleration of llm decoding by feature extrapolation, December 2023. URL [https://sites.google.com/vie](https://sites.google.com/vie) w/eagle-llm. [↗](https://sites.google.com/vie) 20. Lin, C.-Y. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL [https://www.ac](https://www.ac) lweb.org/anthology/W04-1013. [↗](https://www.ac) 21. Liu, X., Hu, L., Bailis, P., Stoica, I., Deng, Z., Cheung, A., and Zhang, H. Online speculative decoding, 2023. [find](https://scholar.google.com/scholar?q=Liu%2C+X%2E%2C+Hu%2C+L%2E%2C+Bailis%2C+P%2E%2C+Stoica%2C+I%2E%2C+Deng%2C+Z%2E%2C+Cheung%2C+A%2E%2C+and+Zhang%2C+H%2E+Online+speculative+decoding%2C+2023%2E) 22. Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Wong, R. Y. Y., Zhu, A., Yang, L., Shi, X., Shi, C., Chen, Z., Arfeen, D., Abhyankar, R., and Jia, Z. Specinfer: Accelerating generative large language model serving with speculative inference and token tree verification, 2023. [find](https://scholar.google.com/scholar?q=Miao%2C+X%2E%2C+Oliaro%2C+G%2E%2C+Zhang%2C+Z%2E%2C+Cheng%2C+X%2E%2C+Wang%2C+Z%2E%2C+Wong%2C+R%2E+Y%2E+Y%2E%2C+Zhu%2C+A%2E%2C+Yang%2C+L%2E%2C+Shi%2C+X%2E%2C+Shi%2C+C%2E%2C+Chen%2C+Z%2E%2C+Arfeen%2C+D%2E%2C+Abhyankar%2C+R%2E%2C+and+Jia%2C+Z%2E+Specinfer%3A+Accelerating+generative+large+lan) 23. Narayan, S., Cohen, S. B., and Lapata, M. Don’t give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 2018. [find](https://scholar.google.com/scholar?q=Narayan%2C+S%2E%2C+Cohen%2C+S%2E+B%2E%2C+and+Lapata%2C+M%2E+Don%E2%80%99t+give+me+the+details%2C+just+the+summary%21+Topic%2Daware+convolutional+neural+networks+for+extreme+summarization%2E+In+Proceedings+of+the+2018+Conference+on+Emp) 24. Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V. A., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., and Zaharia, M. Efficient large-scale language model training on gpu clusters using megatron-lm, 2021. [find](https://scholar.google.com/scholar?q=Narayanan%2C+D%2E%2C+Shoeybi%2C+M%2E%2C+Casper%2C+J%2E%2C+LeGresley%2C+P%2E%2C+Patwary%2C+M%2E%2C+Korthikanti%2C+V%2E+A%2E%2C+Vainbrand%2C+D%2E%2C+Kashinkunti%2C+P%2E%2C+Bernauer%2C+J%2E%2C+Catanzaro%2C+B%2E%2C+Phanishayee%2C+A%2E%2C+and+Zaharia%2C+M%2E+Efficient+large%2Dsc) 25. OpenAI. Gpt-4 technical report, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/) 26. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022. [find](https://scholar.google.com/scholar?q=Ouyang%2C+L%2E%2C+Wu%2C+J%2E%2C+Jiang%2C+X%2E%2C+Almeida%2C+D%2E%2C+Wainwright%2C+C%2E%2C+Mishkin%2C+P%2E%2C+Zhang%2C+C%2E%2C+Agarwal%2C+S%2E%2C+Slama%2C+K%2E%2C+Ray%2C+A%2E%2C+et+al%2E+Training+language+models+to+follow+instructions+with+human+feedback%2E+Advance) 27. Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. [find](https://scholar.google.com/scholar?q=Roziere%2C+B%2E%2C+Gehring%2C+J%2E%2C+Gloeckle%2C+F%2E%2C+Sootla%2C+S%2E%2C+Gat%2C+I%2E%2C+Tan%2C+X%2E+E%2E%2C+Adi%2C+Y%2E%2C+Liu%2C+J%2E%2C+Remez%2C+T%2E%2C+Rapin%2C+J%2E%2C+et+al%2E+Code+llama%3A+Open+foundation+models+for+code%2E+arXiv+preprint+arXiv%3A2308%2E12950%2C+20) [our card](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/) 28. Ruan, J. T., Sabir, F., and Chopra, P. Best prompting practices for using the llama 2 chat llm through amazon sagemaker jumpstart, November 2023. URL [https://aws.amazon.com/cn/blogs/machine-lea](https://aws.amazon.com/cn/blogs/machine-lea) rning/best-prompting-practices-for-u sing-the-llama-2-chat-llm-through-a mazon-sagemaker-jumpstart/. [↗](https://aws.amazon.com/cn/blogs/machine-lea) 29. Santilli, A., Severino, S., Postolache, E., Maiorca, V., Mancusi, M., Marin, R., and Rodola, E. Accelerating transformer inference for translation via parallel decoding. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12336–1235… [find](https://scholar.google.com/scholar?q=Santilli%2C+A%2E%2C+Severino%2C+S%2E%2C+Postolache%2C+E%2E%2C+Maiorca%2C+V%2E%2C+Mancusi%2C+M%2E%2C+Marin%2C+R%2E%2C+and+Rodola%2C+E%2E+Accelerating+transformer+inference+for+translation+via+parallel+decoding%2E+In+Proceedings+of+the+61st+Ann) 30. Saxena, A. Prompt lookup decoding, November 2023. URL [https://github.com/apoorvumang/promp](https://github.com/apoorvumang/promp) t-lookup-decoding/. [↗](https://github.com/apoorvumang/promp) 31. See, A., Liu, P. J., and Manning, C. D. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1073–1083, 2017. [find](https://scholar.google.com/scholar?q=See%2C+A%2E%2C+Liu%2C+P%2E+J%2E%2C+and+Manning%2C+C%2E+D%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Vol) 32. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. [find](https://scholar.google.com/scholar?q=Shoeybi%2C+M%2E%2C+Patwary%2C+M%2E%2C+Puri%2C+R%2E%2C+LeGresley%2C+P%2E%2C+Casper%2C+J%2E%2C+and+Catanzaro%2C+B%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+arXiv+preprint+arXiv%3A1909%2E08053%2C) 33. Song, Y., Meng, C., Liao, R., and Ermon, S. Accelerating feedforward computation via parallel nonlinear equation solving, 2021. [find](https://scholar.google.com/scholar?q=Song%2C+Y%2E%2C+Meng%2C+C%2E%2C+Liao%2C+R%2E%2C+and+Ermon%2C+S%2E+Accelerating+feedforward+computation+via+parallel+nonlinear+equation+solving%2C+2021%2E) 34. Stern, M., Shazeer, N., and Uszkoreit, J. Blockwise parallel decoding for deep autoregressive models, 2018. [find](https://scholar.google.com/scholar?q=Stern%2C+M%2E%2C+Shazeer%2C+N%2E%2C+and+Uszkoreit%2C+J%2E+Blockwise+parallel+decoding+for+deep+autoregressive+models%2C+2018%2E) 35. Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. [find](https://scholar.google.com/scholar?q=Team%2C+G%2E%2C+Anil%2C+R%2E%2C+Borgeaud%2C+S%2E%2C+Wu%2C+Y%2E%2C+Alayrac%2C+J%2E%2DB%2E%2C+Yu%2C+J%2E%2C+Soricut%2C+R%2E%2C+Schalkwyk%2C+J%2E%2C+Dai%2C+A%2E+M%2E%2C+Hauth%2C+A%2E%2C+et+al%2E+Gemini%3A+a+family+of+highly+capable+multimodal+models%2E+arXiv+preprint+arXiv%3A2) [our card](https://md.research.elamur.ai/free/en/papers/gemini-2023-team-gemini-family-highly-capable-multimodal/) 36. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a. [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%2C+Lavril%2C+T%2E%2C+Izacard%2C+G%2E%2C+Martinet%2C+X%2E%2C+Lachaux%2C+M%2E%2DA%2E%2C+Lacroix%2C+T%2E%2C+Rozie%CC%80re%2C+B%2E%2C+Goyal%2C+N%2E%2C+Hambro%2C+E%2E%2C+Azhar%2C+F%2E%2C+et+al%2E+Llama%3A+Open+and+efficient+foundation+language+models%2E+arXiv+prep) 37. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b. [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%2C+Martin%2C+L%2E%2C+Stone%2C+K%2E%2C+Albert%2C+P%2E%2C+Almahairi%2C+A%2E%2C+Babaei%2C+Y%2E%2C+Bashlykov%2C+N%2E%2C+Batra%2C+S%2E%2C+Bhargava%2C+P%2E%2C+Bhosale%2C+S%2E%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtuned+chat+models%2E+arXiv+prepri) 38. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need, 2023. [find](https://scholar.google.com/scholar?q=Vaswani%2C+A%2E%2C+Shazeer%2C+N%2E%2C+Parmar%2C+N%2E%2C+Uszkoreit%2C+J%2E%2C+Jones%2C+L%2E%2C+Gomez%2C+A%2E+N%2E%2C+Kaiser%2C+L%2E%2C+and+Polosukhin%2C+I%2E+Attention+is+all+you+need%2C+2023%2E) 39. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Transformers: State-ofthe-art natural l… [find](https://scholar.google.com/scholar?q=Wolf%2C+T%2E%2C+Debut%2C+L%2E%2C+Sanh%2C+V%2E%2C+Chaumond%2C+J%2E%2C+Delangue%2C+C%2E%2C+Moi%2C+A%2E%2C+Cistac%2C+P%2E%2C+Rault%2C+T%2E%2C+Louf%2C+R%2E%2C+Funtowicz%2C+M%2E%2C+Davison%2C+J%2E%2C+Shleifer%2C+S%2E%2C+von+Platen%2C+P%2E%2C+Ma%2C+C%2E%2C+Jernite%2C+Y%2E%2C+Plu%2C+J%2E%2C+Xu%2C+C%2E%2C+Sca) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/) 40. Yang, N., Ge, T., Wang, L., Jiao, B., Jiang, D., Yang, L., Majumder, R., and Wei, F. Inference with reference: Lossless acceleration of large language models, 2023. [find](https://scholar.google.com/scholar?q=Yang%2C+N%2E%2C+Ge%2C+T%2E%2C+Wang%2C+L%2E%2C+Jiao%2C+B%2E%2C+Jiang%2C+D%2E%2C+Yang%2C+L%2E%2C+Majumder%2C+R%2E%2C+and+Wei%2C+F%2E+Inference+with+reference%3A+Lossless+acceleration+of+large+language+models%2C+2023%2E) 41. Zhang, B. and Sennrich, R. Root mean square layer normalization, 2019. [find](https://scholar.google.com/scholar?q=Zhang%2C+B%2E+and+Sennrich%2C+R%2E+Root+mean+square+layer+normalization%2C+2019%2E) 42. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. [find](https://scholar.google.com/scholar?q=Zheng%2C+L%2E%2C+Chiang%2C+W%2E%2DL%2E%2C+Sheng%2C+Y%2E%2C+Zhuang%2C+S%2E%2C+Wu%2C+Z%2E%2C+Zhuang%2C+Y%2E%2C+Lin%2C+Z%2E%2C+Li%2C+Z%2E%2C+Li%2C+D%2E%2C+Xing%2C+E%2E+P%2E%2C+Zhang%2C+H%2E%2C+Gonzalez%2C+J%2E+E%2E%2C+and+Stoica%2C+I%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+a) 43. A. Algorithms Algorithm 1 Jacobi decoding [find](https://scholar.google.com/scholar?q=A%2E+Algorithms+Algorithm+1+Jacobi+decoding)