---
title: "DeepSpeed-FastGen — paper by Holmes et al. (2024)"
description: "The design rests on three profiling observations: token count drives latency, throughput saturates… 14 references, cited by 72 papers here."
authors: ["Connor Holmes", "Masahiro Tanaka", "Michael Wyatt", "Ammar Ahmad Awan", "Jeff Rasley", "Samyam Rajbhandari", "Reza Yazdani Aminabadi", "Heyang Qin", "Arash Bakhtiari", "Lev Kurilenko"]
year: 2024
venue: "arXiv 2024 (Microsoft DeepSpeed)"
source: "https://arxiv.org/abs/2401.08671"
arxiv: "2401.08671"
licence: "CC BY 4.0 (arXiv OAI)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/holmes-2024-deepspeed-fastgen-high-throughput-text/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/holmes-2024-deepspeed-fastgen-high-throughput-text/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference

# DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference

**Authors:** Connor Holmes, [Masahiro Tanaka](https://md.research.elamur.ai/free/en/authors/masahiro-tanaka/), Michael Wyatt, [Ammar Ahmad Awan](https://md.research.elamur.ai/free/en/authors/ammar-ahmad-awan/), [Jeff Rasley](https://md.research.elamur.ai/free/en/authors/jeff-rasley/), [Samyam Rajbhandari](https://md.research.elamur.ai/free/en/authors/samyam-rajbhandari/), [Reza Yazdani Aminabadi](https://md.research.elamur.ai/free/en/authors/reza-yazdani-aminabadi/), Heyang Qin, [Arash Bakhtiari](https://md.research.elamur.ai/free/en/authors/arash-bakhtiari/) and Lev Kurilenko · **Year:** 2024 · **Venue:** arXiv 2024 (Microsoft DeepSpeed) · **Licence:** [CC BY 4.0 (arXiv OAI)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2401.08671) · [PDF](https://arxiv.org/pdf/2401.08671) · [arXiv 2401.08671](https://arxiv.org/abs/2401.08671) · [OpenAlex](https://openalex.org/W4391012746)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [72](https://md.research.elamur.ai/free/en/papers/holmes-2024-deepspeed-fastgen-high-throughput-text/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLM serving systems must deliver high throughput and low latency together, which is hard when prompts are long. In systems such as vLLM and TGI, processing a new prompt pauses ongoing generation, so streamed tokens arrive late. Orca instead runs whole prompts next to generation, which can still make some forward passes very large. DeepSpeed-FastGen combines DeepSpeed-MII (frontend and serving) with DeepSpeed-Inference (scheduling and kernels). It adds a batching strategy called Dynamic SplitFuse. The scheduler keeps every forward pass at a fixed token budget. It cuts long prompts into chunks spread over several iterations and packs short prompts and generation tokens together to fill the budget exactly. The design rests on three profiling observations: token count drives latency, throughput saturates once a pass has enough tokens, and the throughput curve is concave. Measured against vLLM, the system achieves effective throughput as much as 2.3x higher, average latency about 2x lower, and per-token tail latency reduced by as much as 3.7x. The alpha release is pip-installable and supports LLaMA, Mistral and OPT families.

## Key points
- Reported gains over vLLM: up to 2.3x effective throughput, 2x lower mean latency, and up to 3.7x lower P95 token-generation latency.
- Builds on two existing techniques: a blocked (paged) KV cache, as in PagedAttention, and continuous batching, as in Orca.
- Benchmarks use 1–32 concurrent clients sending 512 requests in total. One setup is Llama 2 7b on an A6000, with prompt and output lengths drawn from normal distributions (means 2600 and 60, 30% variance).
- The alpha release supports HuggingFace models from the LLaMA/LLaMA-2, Mistral and Facebook OPT families. It offers two deployment modes:
  - a non-persistent `pipeline` for interactive scripts;
  - a persistent GRPC server with a built-in load balancer.
- Prebuilt kernels ship as DeepSpeed-Kernels, targeting NVIDIA Ampere+ GPUs, CUDA 11.6+ and Ubuntu 20+. FlashAttention and FasterTransformer kernels are used with modifications.

## Interesting ideas
- **Schedule by a single signal.** Profiling showed that forward-pass latency depends mainly on the total token count, not on how many sequences are in the batch. A scheduler can therefore control just one variable, which is a useful simplification for any batched GPU workload.
- **Use concavity to justify equal splits.** Because throughput grows concavely with tokens per pass, dividing a pool of tokens evenly across passes is optimal. This short mathematical argument could guide chunk sizing in other pipelines, such as batch embedding or tagging of large corpora.
- **Chunk long inputs and top up short ones.** Splitting long prompts and fusing short ones into fixed-size passes keeps latency per step steady and avoids stalls from preemption. The same idea fits mixed-length document processing, where a few huge documents should not block streaming results for everyone else.
- **Measure user-facing metrics, not only averages.** Beyond throughput-latency curves, the paper evaluates an "effective throughput" tied to the chat experience: time to first token and the steady pace of streamed tokens. This is a good template for judging any interactive NLP service.
- **Offer two deployment modes with a tiny API.** A throwaway in-script pipeline and a persistent load-balanced server share the same model identifiers. This lowers the barrier from experimentation to production for small teams.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [✍️ Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) · 214 works

## Cite

Cite as: Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari and Lev Kurilenko. 2024. DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference. arXiv 2024 (Microsoft DeepSpeed).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{holmes2024deepspeed,
  title = {{DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference}},
  author = {Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari and Lev Kurilenko},
  year = {2024},
  eprint = {2401.08671},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.08671}
}
```

- [← previous Tuning Language Models by Proxy](https://md.research.elamur.ai/free/en/papers/liu-2024-tuning-language-models-proxy/)
- [next → Humans or LLMs as the Judge? A Study on Judgement Biases](https://md.research.elamur.ai/free/en/papers/chen-2024-humans-llms-judge-study-judgement/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

14 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. OpenAI. Gpt-4. [https://openai.com/gpt-4](https://openai.com/gpt-4), 2023. [↗](https://openai.com/gpt-4)
2. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
3. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. [Pytorch: An imperative style, high-performance deep learning library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/), 2019.
4. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E Gonzalez, Hao Zhang, and Ion Stoica. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). arXiv preprint arXiv:2309.06180, 2023.
5. GI Yu, Jeong JSeong, GW Kim, S Kim, and BG Chun. Orca: A distributed serving system for {Transformer-Based} generative models. 2022. — [Orca: A Distributed Serving System for Transformer-Based Generative Models](https://md.research.elamur.ai/free/en/papers/yu-2022-orca-distributed-serving-system-transformer/)
6. MosaicML. Introducing mpt-7b: A new standard for open-source, commercially usable llms. [https://www.mosaicml.com/blog/mpt-7b](https://www.mosaicml.com/blog/mpt-7b), 2023. [↗](https://www.mosaicml.com/blog/mpt-7b)
7. Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Leon Song, Samyam Rajbhandari, and Yuxiong He. [Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models](https://md.research.elamur.ai/free/en/papers/jacobs-2023-deepspeed-ulysses-system-optimizations-enabling/). arXiv preprint arXiv:2309.14509, 2023.
8. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E Gonzalez, Hao Zhang, and Ion Stoica. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). arXiv preprint arXiv:2309.06180, 2023.
9. HuggingFace. Text generation inference. [https://huggingface.co/text-generation-inference](https://huggingface.co/text-generation-inference). [↗](https://huggingface.co/text-generation-inference)
10. NVIDIA. Nvidia tensorrt-llm: A tensorrt toolbox for large language model. [https://github.com/NVIDIA/TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM). [↗](https://github.com/NVIDIA/TensorRT-LLM)
11. Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, and Ramachandran Ramjee. [Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills](https://md.research.elamur.ai/free/en/papers/agrawal-2023-sarathi-efficient-llm-inference-piggybacking/). arXiv preprint arXiv:2308.16369, 2023.
12. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Daniel+Y%2E+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+Re%CC%81%2E+FlashAttention%3A+Fast+and+memory%2Defficient+exact+attention+with+IO%2Dawareness%2E+In+Advances+in+Neural+Information+Processing+Systems)
13. Tri Dao. [FlashAttention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). 2023.
14. NVIDIA. Fastertransformer. [https://github.com/NVIDIA/FasterTransformer](https://github.com/NVIDIA/FasterTransformer). [↗](https://github.com/NVIDIA/FasterTransformer)

*Updated 2 November 2026*