---
title: "DistServe — paper by Zhong et al. (2024) · OSDI 2024"
description: "LLM inference has two phases with different profiles. 59 references, cited by 394 papers here."
authors: ["Yinmin Zhong", "Shengyu Liu", "Junda Chen", "Jianbo Hu", "Yibo Zhu", "Xuanzhe Liu", "Xin Jin", "Hao Zhang"]
year: 2024
venue: "OSDI 2024"
source: "https://arxiv.org/abs/2401.09670"
arxiv: "2401.09670"
licence: "CC BY-SA 4.0 (arXiv OAI)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhong-2024-distserve-disaggregating-prefill-decoding-goodput/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhong-2024-distserve-disaggregating-prefill-decoding-goodput/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/) / DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

# DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

**Authors:** [Yinmin Zhong](https://md.research.elamur.ai/free/en/authors/yinmin-zhong/), [Shengyu Liu](https://md.research.elamur.ai/free/en/authors/shengyu-liu/), Junda Chen, Jianbo Hu, [Yibo Zhu](https://md.research.elamur.ai/free/en/authors/yibo-zhu/), [Xuanzhe Liu](https://md.research.elamur.ai/free/en/authors/xuanzhe-liu/), [Xin Jin](https://md.research.elamur.ai/free/en/authors/xin-jin/) and [Hao Zhang](https://md.research.elamur.ai/free/en/authors/hao-zhang/) · **Year:** 2024 · **Venue:** OSDI 2024 · **Licence:** [CC BY-SA 4.0 (arXiv OAI)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2401.09670) · [PDF](https://arxiv.org/pdf/2401.09670) · [arXiv 2401.09670](https://arxiv.org/abs/2401.09670) · [OpenAlex](https://openalex.org/W4391045969)

**Citations:** 15 citations (OpenAlex, 2026-10-14) · [394](https://md.research.elamur.ai/free/en/papers/zhong-2024-distserve-disaggregating-prefill-decoding-goodput/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLM inference has two phases with different profiles. Prefill processes the whole prompt in one compute-heavy step. Decoding emits one token per step and is bound by memory bandwidth. Mainstream systems batch both phases on the same GPUs. As a result, prefill delays inflate TPOT, decoding work inflates TTFT, and both phases must share one resource and parallelism plan. DistServe runs prefill and decoding on separate GPUs. Given TTFT and TPOT targets, it jointly searches GPU counts and parallelism settings for each phase, then replicates that setup until it matches the traffic. Its placement step accounts for cluster bandwidth so that moving intermediate state between phases stays cheap. The objective is per-GPU goodput: the highest request rate per GPU at which a target share of requests (e.g., 90%) meets both SLOs. On chatbot, coding-assistant and summarization workloads, DistServe handles up to 7.4× more requests or 12.6× stricter SLOs than state-of-the-art systems. This translates directly into lower cost per query.

## Key points
- **Motivating experiment:** a 13B model on one 80GB A100 (input 512 tokens, output 64) reaches only about 1.6 rps of goodput when both phases are colocated. A GPU dedicated to prefill alone reaches about 5.6 rps, and one dedicated to decoding alone reaches about 10 rps.
- **Projected gain:** splitting into 2 prefill GPUs plus 1 decoding GPU would give about 10 rps overall, or roughly 3.3 rps per GPU. That is about 2.1× better than the colocated baseline.
- **Parallelism trade-offs:** intra-operator parallelism cuts execution time, especially TTFT, but needs fast links such as NVLINK. Inter-operator (pipeline) parallelism scales rate capacity roughly linearly. Both also shorten queuing delay.
- **Baselines and related systems:** DistServe is compared with colocated designs such as vLLM and SARATHI-style chunked prefill. Splitwise, TetriInfer and DéjàVu are concurrent works with a similar disaggregation idea.
- **Implementation and limits:** DistServe is built as an orchestration layer over an existing inference engine (code at github.com/LLMServe/DistServe). The authors note that it gives little benefit with only one or a few GPUs, or when throughput rather than latency is the priority.

## Interesting ideas
- **Goodput as the metric:** measuring the request rate per GPU that still satisfies latency SLOs, instead of raw throughput, ties evaluation directly to cost. The same framing would suit any latency-sensitive service, such as an NLP API with response-time guarantees.
- **Separate dissimilar stages:** when stages of one job have very different bottlenecks (compute vs. memory bandwidth), giving each its own hardware pool removes interference and lets each scale on its own. This applies to multi-stage data or NLP pipelines generally.
- **Measure each stage alone first:** profiling each phase on a dedicated device before combining them gives a quick estimate of the best resource ratio, as in the 2:1 prefill-to-decode example.
- **Search per unit, then replicate:** optimizing the configuration for one model replica and then copying it to meet demand keeps the search space small. This is a reusable recipe for capacity planning.
- **Bandwidth-aware placement:** placing communicating components with the cluster's network topology in mind keeps disaggregation overhead minor. This is worth remembering whenever a pipeline is split across machines.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,267 citations
- [University of California San Diego](https://md.research.elamur.ai/free/en/universities/university-of-california-san-diego/) — University, San Diego, United States · 89 works · 54,032 citations
- [StepFun AI](https://md.research.elamur.ai/free/en/companies/stepfun-ai/) — Company, Shanghai, China · 5 works · 540 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhong2024distserve,
  title = {{DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving}},
  author = {Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin and Hao Zhang},
  booktitle = {OSDI 2024},
  year = {2024},
  eprint = {2401.09670},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.09670}
}
```

- [← previous BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://md.research.elamur.ai/free/en/papers/labrak-2024-biomistral-collection-open-source-pretrained/)
- [next → Yi: Open Foundation Models by 01.AI](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/)

## References

59 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [1] Introducing chatgpt. chatgpt, 2022. [find](https://scholar.google.com/scholar?q=%5B1%5D+Introducing+chatgpt%2E+chatgpt%2C+2022%2E)
2. William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
3. [2] Bard, an experiment by google. google.com/, 2023. https://bard. [find](https://scholar.google.com/scholar?q=%5B2%5D+Bard%2C+an+experiment+by+google%2E+google%2Ecom%2F%2C+2023%2E+https%3A%2F%2Fbard%2E)
4. [3] Deepspeed model implementations for inference (mii), 2023. [find](https://scholar.google.com/scholar?q=%5B3%5D+Deepspeed+model+implementations+for+inference+%28mii%29%2C+2023%2E)
5. [15] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021.
6. [4] Inflection tech memo. [https://inflection.ai/](https://inflection.ai/) assets/Inflection-1.pdf, 2023. [↗](https://inflection.ai/)
7. [5] Lanchain usecase: Summarization, 2023. [find](https://scholar.google.com/scholar?q=%5B5%5D+Lanchain+usecase%3A+Summarization%2C+2023%2E)
8. [16] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. [find](https://scholar.google.com/scholar?q=%5B16%5D+Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+and+Eric+P%2E+Xing%2E+Vicuna%3A+An+open%2Dsource+chat)
9. [6] Nvidia collective communications library (nccl), 2023. [find](https://scholar.google.com/scholar?q=%5B6%5D+Nvidia+collective+communications+library+%28nccl%29%2C+2023%2E)
10. [7] Serve, optimize and scale pytorch models in production, 2023. [find](https://scholar.google.com/scholar?q=%5B7%5D+Serve%2C+optimize+and+scale+pytorch+models+in+production%2C+2023%2E)
11. [8] Sharegpt teams. [https://sharegpt.com/](https://sharegpt.com/), 2023. [↗](https://sharegpt.com/)
12. [17] Compute Express Link Consortium. Compute express link, 2023. Accessed: 2023-12-07. [find](https://scholar.google.com/scholar?q=%5B17%5D+Compute+Express+Link+Consortium%2E+Compute+express+link%2C+2023%2E+Accessed%3A+2023%2D12%2D07%2E)
13. [9] Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, and Ramachandran Ramjee. [Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills](https://md.research.elamur.ai/free/en/papers/agrawal-2023-sarathi-efficient-llm-inference-piggybacking/). arXiv preprint arXiv:2308.16369, 2023.
14. [18] NVIDIA Corporation. Fastertransformer, 2019. [find](https://scholar.google.com/scholar?q=%5B18%5D+NVIDIA+Corporation%2E+Fastertransformer%2C+2019%2E)
15. [19] NVIDIA Corporation. Triton inference server: An optimized cloud and edge inferencing solution., 2019. [find](https://scholar.google.com/scholar?q=%5B19%5D+NVIDIA+Corporation%2E+Triton+inference+server%3A+An+optimized+cloud+and+edge+inferencing+solution%2E%2C+2019%2E)
16. [10] Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. [Gqa: Training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/), 2023.
17. [20] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. [find](https://scholar.google.com/scholar?q=%5B20%5D+Tri+Dao%2C+Daniel+Y%2E+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2C+2022%2E)
18. [11] Anthropic. Introducing the next generation of claude. [https://www.anthropic.com/news/](https://www.anthropic.com/news/) claude-3-family, 2024. [↗](https://www.anthropic.com/news/)
19. [21] Jiarui Fang, Yang Yu, Chengduo Zhao, and Jie Zhou. Turbotransformers: an efficient gpu serving system for transformer models. In ACM PPoPP, 2021. [find](https://scholar.google.com/scholar?q=%5B21%5D+Jiarui+Fang%2C+Yang+Yu%2C+Chengduo+Zhao%2C+and+Jie+Zhou%2E+Turbotransformers%3A+an+efficient+gpu+serving+system+for+transformer+models%2E+In+ACM+PPoPP%2C+2021%2E)
20. [12] Andrew Audibert, Yang Chen, Dan Graur, Ana Klimovic, Jiri Simsa, and Chandramohan A. Thekkath. A case for disaggregation of ml data processing, 2022. [find](https://scholar.google.com/scholar?q=%5B12%5D+Andrew+Audibert%2C+Yang+Chen%2C+Dan+Graur%2C+Ana+Klimovic%2C+Jiri+Simsa%2C+and+Chandramohan+A%2E+Thekkath%2E+A+case+for+disaggregation+of+ml+data+processing%2C+2022%2E)
21. [22] Google. Our next-generation model: Gemini 1.5. [https://deepmind.google/technologies/](https://deepmind.google/technologies/) gemini/, 2024. [↗](https://deepmind.google/technologies/)
22. [13] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. [Longbench: A bilingual, multitask benchmark for long context understanding](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/), 2023.
23. [23] Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. Serving DNNs like clockwork: Performance predictability from the bottom up. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pages 443–462. USENIX Associat… [find](https://scholar.google.com/scholar?q=%5B23%5D+Arpan+Gujarati%2C+Reza+Karimi%2C+Safya+Alzayat%2C+Wei+Hao%2C+Antoine+Kaufmann%2C+Ymir+Vigfusson%2C+and+Jonathan+Mace%2E+Serving+DNNs+like+clockwork%3A+Performance+predictability+from+the+bottom+up%2E+In+14th+USENI)
24. [14] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, M… [find](https://scholar.google.com/scholar?q=%5B14%5D+Mark+Chen%2C+Jerry+Tworek%2C+Heewoo+Jun%2C+Qiming+Yuan%2C+Henrique+Ponde+de+Oliveira+Pinto%2C+Jared+Kaplan%2C+Harri+Edwards%2C+Yuri+Burda%2C+Nicholas+Joseph%2C+Greg+Brockman%2C+Alex+Ray%2C+Raul+Puri%2C+Gretchen+Krueger%2C)
25. [25] Zhiyuan Guo, Zijian He, and Yiying Zhang. Mira: A program-behavior-guided far memory system. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, page 692–708, New York, NY, USA, 2023. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=%5B25%5D+Zhiyuan+Guo%2C+Zijian+He%2C+and+Yiying+Zhang%2E+Mira%3A+A+program%2Dbehavior%2Dguided+far+memory+system%2E+In+Proceedings+of+the+29th+Symposium+on+Operating+Systems+Principles%2C+SOSP+%E2%80%9923%2C+page+692%E2%80%93708%2C+New+York)
26. [36] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. In ACM SOSP, 2019. [find](https://scholar.google.com/scholar?q=%5B36%5D+Deepak+Narayanan%2C+Aaron+Harlap%2C+Amar+Phanishayee%2C+Vivek+Seshadri%2C+Nikhil+R%2E+Devanur%2C+Gregory+R%2E+Ganger%2C+Phillip+B%2E+Gibbons%2C+and+Matei+Zaharia%2E+Pipedream%3A+Generalized+pipeline+parallelism+for+dnn+)
27. [26] Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen. Microsecond-scale preemption for concurrent GPU-accelerated DNN inferences. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 539–558, Carlsbad, CA, July 2022. USENIX Association. [find](https://scholar.google.com/scholar?q=%5B26%5D+Mingcong+Han%2C+Hanze+Zhang%2C+Rong+Chen%2C+and+Haibo+Chen%2E+Microsecond%2Dscale+preemption+for+concurrent+GPU%2Daccelerated+DNN+inferences%2E+In+16th+USENIX+Symposium+on+Operating+Systems+Design+and+Implemen)
28. [37] OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023.
29. [27] Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. Inference without interference: Disaggregate llm inference for mixed downstream workloads, 2024. [find](https://scholar.google.com/scholar?q=%5B27%5D+Cunchen+Hu%2C+Heyang+Huang%2C+Liangliang+Xu%2C+Xusheng+Chen%2C+Jiang+Xu%2C+Shuang+Chen%2C+Hao+Feng%2C+Chenxi+Wang%2C+Sa+Wang%2C+Yungang+Bao%2C+Ninghui+Sun%2C+and+Yizhou+Shan%2E+Inference+without+interference%3A+Disaggrega)
30. [38] Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, Aashaka Shah, Saeed Maleki, and Ricardo Bianchini. [Splitwise: Efficient generative llm inference using phase splitting](https://md.research.elamur.ai/free/en/papers/patel-2023-splitwise-efficient-generative-llm-inference/), 2023.
31. [39] Aurick Qiao, Sang Keun Choe, Suhas Jayaram Subramanya, Willie Neiswanger, Qirong Ho, Hao Zhang, Gregory R. Ganger, and Eric P. Xing. Pollux: Co-adaptive cluster scheduling for goodput-optimized deep learning. In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21), pag… [find](https://scholar.google.com/scholar?q=%5B39%5D+Aurick+Qiao%2C+Sang+Keun+Choe%2C+Suhas+Jayaram+Subramanya%2C+Willie+Neiswanger%2C+Qirong+Ho%2C+Hao+Zhang%2C+Gregory+R%2E+Ganger%2C+and+Eric+P%2E+Xing%2E+Pollux%3A+Co%2Dadaptive+cluster+scheduling+for+goodput%2Doptimized+d)
32. [28] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism, 2019. [find](https://scholar.google.com/scholar?q=%5B28%5D+Yanping+Huang%2C+Youlong+Cheng%2C+Ankur+Bapna%2C+Orhan+Firat%2C+Mia+Xu+Chen%2C+Dehao+Chen%2C+HyoukJoong+Lee%2C+Jiquan+Ngiam%2C+Quoc+V%2E+Le%2C+Yonghui+Wu%2C+and+Zhifeng+Chen%2E+Gpipe%3A+Efficient+training+of+giant+neural+)
33. [29] Suhas Jayaram Subramanya, Daiyaan Arfeen, Shouxu Lin, Aurick Qiao, Zhihao Jia, and Gregory R Ganger. Sia: Heterogeneity-aware, goodput-optimized ml-cluster scheduling. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 642–657, 2023. [find](https://scholar.google.com/scholar?q=%5B29%5D+Suhas+Jayaram+Subramanya%2C+Daiyaan+Arfeen%2C+Shouxu+Lin%2C+Aurick+Qiao%2C+Zhihao+Jia%2C+and+Gregory+R+Ganger%2E+Sia%3A+Heterogeneity%2Daware%2C+goodput%2Doptimized+ml%2Dcluster+scheduling%2E+In+Proceedings+of+the+29th+)
34. [40] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2020. [find](https://scholar.google.com/scholar?q=%5B40%5D+Samyam+Rajbhandari%2C+Jeff+Rasley%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+Zero%3A+Memory+optimizations+toward+training+trillion+parameter+models%2C+2020%2E)
35. [30] Xin Jin, Zhihao Bai, Zhen Zhang, Yibo Zhu, Yinmin Zhong, and Xuanzhe Liu. Distmind: Efficient resource disaggregation for deep learning workloads. IEEE/ACM Transactions on Networking, pages 1–16, 2024. [find](https://scholar.google.com/scholar?q=%5B30%5D+Xin+Jin%2C+Zhihao+Bai%2C+Zhen+Zhang%2C+Yibo+Zhu%2C+Yinmin+Zhong%2C+and+Xuanzhe+Liu%2E+Distmind%3A+Efficient+resource+disaggregation+for+deep+learning+workloads%2E+IEEE%2FACM+Transactions+on+Networking%2C+pages+1%E2%80%9316%2C)
36. [42] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950, 2023.
37. [31] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
38. [43] Yizhou Shan, Yutong Huang, Yilun Chen, and Yiying Zhang. {LegoOS}: A disseminated, distributed {OS} for hardware resource disaggregation. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 69–87, 2018. [find](https://scholar.google.com/scholar?q=%5B43%5D+Yizhou+Shan%2C+Yutong+Huang%2C+Yilun+Chen%2C+and+Yiying+Zhang%2E+%7BLegoOS%7D%3A+A+disseminated%2C+distributed+%7BOS%7D+for+hardware+resource+disaggregation%2E+In+13th+USENIX+Symposium+on+Operating+Systems+Design+and+)
39. [32] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/), 2023.
40. [44] Noam Shazeer. Fast transformer decoding: One write-head is all you need, 2019. [find](https://scholar.google.com/scholar?q=%5B44%5D+Noam+Shazeer%2E+Fast+transformer+decoding%3A+One+write%2Dhead+is+all+you+need%2C+2019%2E)
41. [45] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. [Flexgen: High-throughput generative inference of large language models with a single gpu](https://md.research.elamur.ai/free/en/papers/sheng-2023-flexgen-high-throughput-generative-inference/), 2023.
42. [33] Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E Gonzalez, et al. [Alpaserve: Statistical multiplexing with model parallelism for deep learning serving](https://md.research.elamur.ai/free/en/papers/li-2023-alpaserve-statistical-multiplexing-model-parallelism/). arXiv, 2023.
43. [46] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020. [find](https://scholar.google.com/scholar?q=%5B46%5D+Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2C+2020%2E)
44. [34] Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv, 2024. [find](https://scholar.google.com/scholar?q=%5B34%5D+Hao+Liu%2C+Wilson+Yan%2C+Matei+Zaharia%2C+and+Pieter+Abbeel%2E+World+model+on+million%2Dlength+video+and+language+with+ringattention%2E+arXiv%2C+2024%2E)
45. [35] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. Ray: A distributed framework for emerging AI applications. In USENIX OSDI, 2018. [find](https://scholar.google.com/scholar?q=%5B35%5D+Philipp+Moritz%2C+Robert+Nishihara%2C+Stephanie+Wang%2C+Alexey+Tumanov%2C+Richard+Liaw%2C+Eric+Liang%2C+Melih+Elibol%2C+Zongheng+Yang%2C+William+Paul%2C+Michael+I%2E+Jordan%2C+and+Ion+Stoica%2E+Ray%3A+A+distributed+framew)
46. [47] John F Shortle, James M Thompson, Donald Gross, and Carl M Harris. Fundamentals of queueing theory, volume 399. John Wiley & Sons, 2018. [find](https://scholar.google.com/scholar?q=%5B47%5D+John+F+Shortle%2C+James+M+Thompson%2C+Donald+Gross%2C+and+Carl+M+Harris%2E+Fundamentals+of+queueing+theory%2C+volume+399%2E+John+Wiley+%26+Sons%2C+2018%2E)
47. [55] Hong Zhang, Yupeng Tang, Anurag Khandelwal, and Ion Stoica. Shepherd: Serving dnns in the wild. 2023. [find](https://scholar.google.com/scholar?q=%5B55%5D+Hong+Zhang%2C+Yupeng+Tang%2C+Anurag+Khandelwal%2C+and+Ion+Stoica%2E+Shepherd%3A+Serving+dnns+in+the+wild%2E+2023%2E)
48. [48] Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. [Powerinfer: Fast large language model serving with a consumer-grade gpu](https://md.research.elamur.ai/free/en/papers/song-2023-powerinfer-fast-large-language-model/), 2023.
49. [56] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-train… — [OPT: Open Pre-trained Transformer Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/)
50. [49] Foteini Strati, Sara Mcallister, Amar Phanishayee, Jakub Tarnawski, and Ana Klimovic. Déjàvu: Kv-cache streaming for fast, fault-tolerant generative llm serving, 2024. [find](https://scholar.google.com/scholar?q=%5B49%5D+Foteini+Strati%2C+Sara+Mcallister%2C+Amar+Phanishayee%2C+Jakub+Tarnawski%2C+and+Ana+Klimovic%2E+D%C3%A9j%C3%A0vu%3A+Kv%2Dcache+streaming+for+fast%2C+fault%2Dtolerant+generative+llm+serving%2C+2024%2E)
51. [50] Yiming Su, Chengcheng Wan, Utsav Sethi, Shan Lu, Madan Musuvathi, and Suman Nath. Hotgpt: How to make software documentation more useful with a large language model? In Proceedings of the 19th Workshop on Hot Topics in Operating Systems, pages 87–93, 2023. [find](https://scholar.google.com/scholar?q=%5B50%5D+Yiming+Su%2C+Chengcheng+Wan%2C+Utsav+Sethi%2C+Shan+Lu%2C+Madan+Musuvathi%2C+and+Suman+Nath%2E+Hotgpt%3A+How+to+make+software+documentation+more+useful+with+a+large+language+model%3F+In+Proceedings+of+the+19th+Wo)
52. [57] Yiying Zhang. Make it real: An end-to-end implementation of a physically disaggregated data center. SIGOPS Oper. Syst. Rev., 57(1):1–9, jun 2023. [find](https://scholar.google.com/scholar?q=%5B57%5D+Yiying+Zhang%2E+Make+it+real%3A+An+end%2Dto%2Dend+implementation+of+a+physically+disaggregated+data+center%2E+SIGOPS+Oper%2E+Syst%2E+Rev%2E%2C+57%281%29%3A1%E2%80%939%2C+jun+2023%2E)
53. [51] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/), 2023.
54. [58] Kai Zhao, Sheng Di, Sihuan Li, Xin Liang, Yujia Zhai, Jieyang Chen, Kaiming Ouyang, Franck Cappello, and Zizhong Chen. Ft-cnn: Algorithm-based fault tolerance for convolutional neural networks. IEEE Transactions on Parallel and Distributed Systems, 32(7):1677–1689, 2021. [find](https://scholar.google.com/scholar?q=%5B58%5D+Kai+Zhao%2C+Sheng+Di%2C+Sihuan+Li%2C+Xin+Liang%2C+Yujia+Zhai%2C+Jieyang+Chen%2C+Kaiming+Ouyang%2C+Franck+Cappello%2C+and+Zizhong+Chen%2E+Ft%2Dcnn%3A+Algorithm%2Dbased+fault+tolerance+for+convolutional+neural+networks%2E+I)
55. [52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Neural Information Processing Systems, 2017.
56. [59] Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P. Xing, Joseph E. Gonzalez, and Ion Stoica. [Alpa: Automating inter- and Intra-Operator parallelism for distributed deep learning](https://md.research.elamur.ai/free/en/papers/zheng-2022-alpa-automating-inter-intra-operator/). In USENIX OSDI, 2022.
57. [53] Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. [Fast distributed inference serving for large language models](https://md.research.elamur.ai/free/en/papers/wu-2023-fast-distributed-inference-serving-large/). arXiv preprint arXiv:2305.05920, 2023.
58. [60] Zhe Zhou, Xuechao Wei, Jiejing Zhang, and Guangyu Sun. {PetS}: A unified framework for {Parameter-Efficient} transformers serving. In USENIX ATC, 2022. [find](https://scholar.google.com/scholar?q=%5B60%5D+Zhe+Zhou%2C+Xuechao+Wei%2C+Jiejing+Zhang%2C+and+Guangyu+Sun%2E+%7BPetS%7D%3A+A+unified+framework+for+%7BParameter%2DEfficient%7D+transformers+serving%2E+In+USENIX+ATC%2C+2022%2E)
59. [54] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for {Transformer-Based} generative models. In USENIX OSDI, 2022. — [Orca: A Distributed Serving System for Transformer-Based Generative Models](https://md.research.elamur.ai/free/en/papers/yu-2022-orca-distributed-serving-system-transformer/)

*Updated 2 November 2026*