---
title: "AlpaServe — paper by Li et al. (2023) · OSDI 2023"
description: "The authors show that model parallelism helps even when each model fits on one device. 57 references, cited by 151 papers here."
authors: ["Zhuohan Li", "Lianmin Zheng", "Yinmin Zhong", "Vincent Liu", "Ying Sheng", "Xin Jin", "Yanping Huang", "Zhifeng Chen", "Hao Zhang", "Joseph E. Gonzalez"]
year: 2023
venue: "OSDI 2023"
source: "https://arxiv.org/abs/2302.11665"
arxiv: "2302.11665"
licence: "arXiv — non-exclusive distribution license (arXiv OAI)"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2023-alpaserve-statistical-multiplexing-model-parallelism/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2023-alpaserve-statistical-multiplexing-model-parallelism/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving

# AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving

**Authors:** [Zhuohan Li](https://md.research.elamur.ai/free/en/authors/zhuohan-li/), [Lianmin Zheng](https://md.research.elamur.ai/free/en/authors/lianmin-zheng/), [Yinmin Zhong](https://md.research.elamur.ai/free/en/authors/yinmin-zhong/), Vincent Liu, [Ying Sheng](https://md.research.elamur.ai/free/en/authors/ying-sheng/), [Xin Jin](https://md.research.elamur.ai/free/en/authors/xin-jin/), [Yanping Huang](https://md.research.elamur.ai/free/en/authors/yanping-huang/), [Zhifeng Chen](https://md.research.elamur.ai/free/en/authors/zhifeng-chen/), [Hao Zhang](https://md.research.elamur.ai/free/en/authors/hao-zhang/) and [Joseph E. Gonzalez](https://md.research.elamur.ai/free/en/authors/joseph-e-gonzalez/) · **Year:** 2023 · **Venue:** OSDI 2023 · **Licence:** [arXiv — non-exclusive distribution license (arXiv OAI)](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2302.11665) · [PDF](https://arxiv.org/pdf/2302.11665) · [arXiv 2302.11665](https://arxiv.org/abs/2302.11665) · [OpenAlex](https://openalex.org/W4321853806)

**Citations:** 19 citations (OpenAlex, 2026-10-14) · [151](https://md.research.elamur.ai/free/en/papers/li-2023-alpaserve-statistical-multiplexing-model-parallelism/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
AlpaServe asks how to serve many large deep learning models on a shared GPU cluster when request traffic is highly bursty. The authors show that model parallelism helps even when each model fits on one device. Splitting models across several GPUs and colocating them lets a sudden burst for one model use hardware that would otherwise belong to idle models. The split adds communication overhead, but this statistical multiplexing can still lower latency, so the paper analyzes when the trade pays off. Taking as input a description of the cluster, the models to be served, and a workload profile that repeats over time, AlpaServe:
- splits the cluster into disjoint model-parallel groups,
- picks a parallelization for each group,
- places models into the groups using a simulator-guided greedy search,
- schedules incoming requests to maximize SLO attainment.

The authors evaluated it on a 64-GPU cluster with production traces. Compared with earlier serving systems, it sustains up to 10× higher request rates, 2.5× tighter deadlines, or 6× burstier traffic while keeping over 99% of requests within SLO. This matters because the usual alternative is costly over-provisioning for peak load.

## Key points
- **Motivation:** traffic spikes in common traces can reach 50× the mean load. Hugging Face alone hosts over 9,000 fine-tuned BERT variants, so serving many similar models is a real need.
- **Two-GPU experiment:** two 6.7B-parameter Transformers ran on two 16 GB V100s, with about 0.4 s per request.
  - Under Poisson arrivals at 1.5 req/s, a 2-stage pipeline placement cut mean latency from 0.70 s to 0.55 s.
  - With Gamma arrivals (CV 3), the gain grew to 1.9×.
  - With a 20/80 skew in demand between the two models, the gain reached 6.6×.
- **Toy example:** even with an assumed 10% parallelism overhead, colocation lowers mean burst latency from 2.5y to 1.85y, where y is single-model execution time.
- **Search space:** the system covers both inter-op (pipeline) and intra-op parallelism. It also adapts existing training-oriented auto-parallelization methods to inference.
- **Ablation:** adding group partitioning to greedy placement raised the sustainable rate at 99% SLO attainment by 1.5× and the tolerable burstiness by 1.3×. Round-robin placement never reached 99%.

## Interesting ideas
- **Trade per-request speed for pooling:** accepting slower individual requests can still reduce latency, because a pooled resource absorbs bursts better than dedicated slots. The same logic may apply to sharing workers or databases across several services.
- **Simulator-guided placement:** candidate placements are scored by replaying the workload profile in a simulator rather than with a closed-form cost model. This approach transfers well to scheduling problems that are hard to model analytically.
- **Two-level decomposition:** the search first partitions the cluster into groups and then assigns models within each group. This keeps a combinatorial problem tractable and could suit other cluster-allocation tasks.
- **Stress-testing with burstiness:** the evaluation varies the arrival process's coefficient of variance and the skew in demand across models, not just the average rate. This is a useful pattern for benchmarking any latency-sensitive service.
- **Optimizing SLO attainment:** the system targets the share of requests served within the deadline, instead of mean throughput. This metric better reflects what users of an online NLP API would actually experience.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works · [🌐 Computer Networks and Communications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-networks-and-communications/) · 100 works

## Cite

Cite as: Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang and Joseph E. Gonzalez. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. OSDI 2023.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{li2023alpaserve,
  title = {{AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving}},
  author = {Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang and Joseph E. Gonzalez},
  booktitle = {OSDI 2023},
  year = {2023},
  eprint = {2302.11665},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.11665}
}
```

- [← previous The Troubling Emergence of Hallucination in Large Language Models - An Extensive Definition, Quantification, and Prescriptive Remediations](https://md.research.elamur.ai/free/en/papers/rawte-2023-troubling-emergence-hallucination-large-language/)
- [next → SqueezeLLM: Dense-and-Sparse Quantization](https://md.research.elamur.ai/free/en/papers/kim-2023-squeezellm-dense-sparse-quantization/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

57 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, et al. Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale. arXiv preprint arXiv:2207.00032, 2022. [find](https://scholar.google.com/scholar?q=Reza+Yazdani+Aminabadi%2C+Samyam+Rajbhandari%2C+Minjia+Zhang%2C+Ammar+Ahmad+Awan%2C+Cheng+Li%2C+Du+Li%2C+Elton+Zheng%2C+Jeff+Rasley%2C+Shaden+Smith%2C+Olatunji+Ruwase%2C+et+al%2E+Deepspeed+inference%3A+Enabling+efficient+inf)
2. Zhihao Bai, Zhen Zhang, Yibo Zhu, and Xin Jin. {PipeSwitch}: Fast pipelined context switching for deep learning applications. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pages 499–514, 2020. [find](https://scholar.google.com/scholar?q=Zhihao+Bai%2C+Zhen+Zhang%2C+Yibo+Zhu%2C+and+Xin+Jin%2E+%7BPipeSwitch%7D%3A+Fast+pipelined+context+switching+for+deep+learning+applications%2E+In+14th+USENIX+Symposium+on+Operating+Systems+Design+and+Implementation+%28O)
3. Nirvik Baruah, Peter Kraft, Fiodar Kazhamiaka, Peter Bailis, and Matei Zaharia. Parallelism-optimizing data placement for faster data-parallel computations. Proceedings of the VLDB Endowment, 16(4):760–771, 2022. [find](https://scholar.google.com/scholar?q=Nirvik+Baruah%2C+Peter+Kraft%2C+Fiodar+Kazhamiaka%2C+Peter+Bailis%2C+and+Matei+Zaharia%2E+Parallelism%2Doptimizing+data+placement+for+faster+data%2Dparallel+computations%2E+Proceedings+of+the+VLDB+Endowment%2C+16%284%29%3A76)
4. Anirban Bhattacharjee, Ajay Dev Chhokra, Zhuangwei Kang, Hongyang Sun, Aniruddha Gokhale, and Gabor Karsai. Barista: Efficient and scalable serverless serving system for deep learning prediction services. In 2019 IEEE International Conference on Cloud Engineering (IC2E), pages 23–33. IEEE, 2019. [find](https://scholar.google.com/scholar?q=Anirban+Bhattacharjee%2C+Ajay+Dev+Chhokra%2C+Zhuangwei+Kang%2C+Hongyang+Sun%2C+Aniruddha+Gokhale%2C+and+Gabor+Karsai%2E+Barista%3A+Efficient+and+scalable+serverless+serving+system+for+deep+learning+prediction+servi)
5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
6. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311, 2022.
7. Copy.ai. Copy.ai: Write better marketing copy and content with ai. [https://www.copy.ai/](https://www.copy.ai/). [↗](https://www.copy.ai/)
8. Daniel Crankshaw, Gur-Eyal Sela, Xiangxi Mo, Corey Zumar, Ion Stoica, Joseph Gonzalez, and Alexey Tumanov. Inferline: latency-aware provisioning and scaling for prediction serving pipelines. In Proceedings of the 11th ACM Symposium on Cloud Computing, pages 477–491, 2020. [find](https://scholar.google.com/scholar?q=Daniel+Crankshaw%2C+Gur%2DEyal+Sela%2C+Xiangxi+Mo%2C+Corey+Zumar%2C+Ion+Stoica%2C+Joseph+Gonzalez%2C+and+Alexey+Tumanov%2E+Inferline%3A+latency%2Daware+provisioning+and+scaling+for+prediction+serving+pipelines%2E+In+Procee)
9. Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J Franklin, Joseph E Gonzalez, and Ion Stoica. Clipper: A low-latency online prediction serving system. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17), pages 613–627, 2017. [find](https://scholar.google.com/scholar?q=Daniel+Crankshaw%2C+Xin+Wang%2C+Guilio+Zhou%2C+Michael+J+Franklin%2C+Joseph+E+Gonzalez%2C+and+Ion+Stoica%2E+Clipper%3A+A+low%2Dlatency+online+prediction+serving+system%2E+In+14th+USENIX+Symposium+on+Networked+Systems+D)
10. Weihao Cui, Han Zhao, Quan Chen, Hao Wei, Zirui Li, Deze Zeng, Chao Li, and Minyi Guo. Dvabatch: Diversity-aware multi-entry multi-exit batching for efficient processing of dnn services on gpus. In 2022 USENIX Annual Technical Conference (USENIX ATC 22), pages 183–198, 2022. [find](https://scholar.google.com/scholar?q=Weihao+Cui%2C+Han+Zhao%2C+Quan+Chen%2C+Hao+Wei%2C+Zirui+Li%2C+Deze+Zeng%2C+Chao+Li%2C+and+Minyi+Guo%2E+Dvabatch%3A+Diversity%2Daware+multi%2Dentry+multi%2Dexit+batching+for+efficient+processing+of+dnn+services+on+gpus%2E+In+20)
11. Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Daniel+Y+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2E+Advances+in+Neural+Information+Processing+Systems%2C+202)
12. Robert I Davis, Ken W Tindell, and Alan Burns. Scheduling slack time in fixed priority pre-emptive systems. In 1993 Proceedings Real-Time Systems Symposium, pages 222–231. IEEE, 1993. [find](https://scholar.google.com/scholar?q=Robert+I+Davis%2C+Ken+W+Tindell%2C+and+Alan+Burns%2E+Scheduling+slack+time+in+fixed+priority+pre%2Demptive+systems%2E+In+1993+Proceedings+Real%2DTime+Systems+Symposium%2C+pages+222%E2%80%93231%2E+IEEE%2C+1993%2E)
13. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Mike+Lewis%2C+Younes+Belkada%2C+and+Luke+Zettlemoyer%2E+Llm%2E+int8+%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2E+Advances+in+Neural+Information+Processing+Systems%2C+2022%2E)
14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805, 2018.
15. Mengnan Du, Subhabrata Mukherjee, Yu Cheng, Milad Shokouhi, Xia Hu, and Ahmed Hassan Awadallah. What do compressed large language models forget? robustness challenges in model compression. arXiv preprint arXiv:2110.08419, 2021. — [arXiv:2110.08419](https://arxiv.org/abs/2110.08419) [find](https://scholar.google.com/scholar?q=Mengnan+Du%2C+Subhabrata+Mukherjee%2C+Yu+Cheng%2C+Milad+Shokouhi%2C+Xia+Hu%2C+and+Ahmed+Hassan+Awadallah%2E+What+do+compressed+large+language+models+forget%3F+robustness+challenges+in+model+compression%2E+arXiv+prepr)
16. Jiarui Fang, Yang Yu, Chengduo Zhao, and Jie Zhou. Turbotransformers: an efficient gpu serving system for transformer models. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 389–402, 2021. [find](https://scholar.google.com/scholar?q=Jiarui+Fang%2C+Yang+Yu%2C+Chengduo+Zhao%2C+and+Jie+Zhou%2E+Turbotransformers%3A+an+efficient+gpu+serving+system+for+transformer+models%2E+In+Proceedings+of+the+26th+ACM+SIGPLAN+Symposium+on+Principles+and+Practic)
17. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Barret+Zoph%2C+and+Noam+Shazeer%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+Journal+of+Machine+Learning+Research%2C+23%28120%29%3A1%E2%80%9339%2C+2022%2E)
18. Github. Github copilot: Your ai pair programmer. [https://github.com/features/copilot](https://github.com/features/copilot). [↗](https://github.com/features/copilot)
19. Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. Serving {DNNs} like clockwork: Performance predictability from the bottom up. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pages 443–462, 2020. [find](https://scholar.google.com/scholar?q=Arpan+Gujarati%2C+Reza+Karimi%2C+Safya+Alzayat%2C+Wei+Hao%2C+Antoine+Kaufmann%2C+Ymir+Vigfusson%2C+and+Jonathan+Mace%2E+Serving+%7BDNNs%7D+like+clockwork%3A+Performance+predictability+from+the+bottom+up%2E+In+14th+USENIX+S)
20. Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen. Microsecond-scale preemption for concurrent GPU-accelerated DNN inferences. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 539–558, Carlsbad, CA, July 2022. USENIX Association. [find](https://scholar.google.com/scholar?q=Mingcong+Han%2C+Hanze+Zhang%2C+Rong+Chen%2C+and+Haibo+Chen%2E+Microsecond%2Dscale+preemption+for+concurrent+GPU%2Daccelerated+DNN+inferences%2E+In+16th+USENIX+Symposium+on+Operating+Systems+Design+and+Implementatio)
21. Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen. Microsecond-scale preemption for concurrent {GPU-accelerated}{DNN} inferences. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 539–558, 2022. [find](https://scholar.google.com/scholar?q=Mingcong+Han%2C+Hanze+Zhang%2C+Rong+Chen%2C+and+Haibo+Chen%2E+Microsecond%2Dscale+preemption+for+concurrent+%7BGPU%2Daccelerated%7D%7BDNN%7D+inferences%2E+In+16th+USENIX+Symposium+on+Operating+Systems+Design+and+Implementa)
22. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/), 2021.
23. Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019. [find](https://scholar.google.com/scholar?q=Yanping+Huang%2C+Youlong+Cheng%2C+Ankur+Bapna%2C+Orhan+Firat%2C+Dehao+Chen%2C+Mia+Chen%2C+HyoukJoong+Lee%2C+Jiquan+Ngiam%2C+Quoc+V+Le%2C+Yonghui+Wu%2C+et+al%2E+Gpipe%3A+Efficient+training+of+giant+neural+networks+using+pipel)
24. Huggingface. Models - huggingface. huggingface.co/models. https:// [find](https://scholar.google.com/scholar?q=Huggingface%2E+Models+%2D+huggingface%2E+huggingface%2Eco%2Fmodels%2E+https%3A%2F%2F)
25. Vatche Ishakian, Vinod Muthusamy, and Aleksander Slominski. Serving deep learning models in a serverless platform. In 2018 IEEE International Conference on Cloud Engineering (IC2E), pages 257–262. IEEE, 2018. [find](https://scholar.google.com/scholar?q=Vatche+Ishakian%2C+Vinod+Muthusamy%2C+and+Aleksander+Slominski%2E+Serving+deep+learning+models+in+a+serverless+platform%2E+In+2018+IEEE+International+Conference+on+Cloud+Engineering+%28IC2E%29%2C+pages+257%E2%80%93262%2E+IEE)
26. Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun, Shigang Li, and Torsten Hoefler. Data movement is all you need: A case study on optimizing transformers. Proceedings of Machine Learning and Systems, 3:711–732, 2021. [find](https://scholar.google.com/scholar?q=Andrei+Ivanov%2C+Nikoli+Dryden%2C+Tal+Ben%2DNun%2C+Shigang+Li%2C+and+Torsten+Hoefler%2E+Data+movement+is+all+you+need%3A+A+case+study+on+optimizing+transformers%2E+Proceedings+of+Machine+Learning+and+Systems%2C+3%3A711%E2%80%937)
27. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2020. [find](https://scholar.google.com/scholar?q=Dmitry+Lepikhin%2C+HyoukJoong+Lee%2C+Yuanzhong+Xu%2C+Dehao+Chen%2C+Orhan+Firat%2C+Yanping+Huang%2C+Maxim+Krikun%2C+Noam+Shazeer%2C+and+Zhifeng+Chen%2E+Gshard%3A+Scaling+giant+models+with+conditional+computation+and+autom)
28. Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, and Ion Stoica. Terapipe: Token-level pipeline parallelism for training large-scale language models. In International Conference on Machine Learning, pages 6543–6552. PMLR, 2021. [find](https://scholar.google.com/scholar?q=Zhuohan+Li%2C+Siyuan+Zhuang%2C+Shiyuan+Guo%2C+Danyang+Zhuo%2C+Hao+Zhang%2C+Dawn+Song%2C+and+Ion+Stoica%2E+Terapipe%3A+Token%2Dlevel+pipeline+parallelism+for+training+large%2Dscale+language+models%2E+In+International+Confer)
29. Xiaoqiao Meng, Canturk Isci, Jeffrey Kephart, Li Zhang, Eric Bouillet, and Dimitrios Pendarakis. Efficient resource provisioning in compute clouds via vm multiplexing. In Proceedings of the 7th international conference on Autonomic computing, pages 11–20, 2010. [find](https://scholar.google.com/scholar?q=Xiaoqiao+Meng%2C+Canturk+Isci%2C+Jeffrey+Kephart%2C+Li+Zhang%2C+Eric+Bouillet%2C+and+Dimitrios+Pendarakis%2E+Efficient+resource+provisioning+in+compute+clouds+via+vm+multiplexing%2E+In+Proceedings+of+the+7th+intern)
30. Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pages 1–15, 2019. [find](https://scholar.google.com/scholar?q=Deepak+Narayanan%2C+Aaron+Harlap%2C+Amar+Phanishayee%2C+Vivek+Seshadri%2C+Nikhil+R+Devanur%2C+Gregory+R+Ganger%2C+Phillip+B+Gibbons%2C+and+Matei+Zaharia%2E+Pipedream%3A+generalized+pipeline+parallelism+for+dnn+training)
31. Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on gpu clusters using megatron-lm. In Pr… [find](https://scholar.google.com/scholar?q=Deepak+Narayanan%2C+Mohammad+Shoeybi%2C+Jared+Casper%2C+Patrick+LeGresley%2C+Mostofa+Patwary%2C+Vijay+Korthikanti%2C+Dmitri+Vainbrand%2C+Prethvi+Kashinkunti%2C+Julie+Bernauer%2C+Bryan+Catanzaro%2C+Amar+Phanishayee%2C+and+M)
32. NVIDIA. Fastertransformer. [https://github.com/NVIDIA/FasterTransformer](https://github.com/NVIDIA/FasterTransformer). [↗](https://github.com/NVIDIA/FasterTransformer)
33. NVIDIA. Triton inference [https://developer.nvidia.com/nvidia-triton-inference-server](https://developer.nvidia.com/nvidia-triton-inference-server). server. [↗](https://developer.nvidia.com/nvidia-triton-inference-server)
34. Christopher Olston, Noah Fiedel, Kiril Gorovoy, Jeremiah Harmsen, Li Lao, Fangwei Li, Vinu Rajashekhar, Sukriti Ramesh, and Jordan Soyke. Tensorflow-serving: Flexible, high-performance ml serving. arXiv preprint arXiv:1712.06139, 2017. — [arXiv:1712.06139](https://arxiv.org/abs/1712.06139) [find](https://scholar.google.com/scholar?q=Christopher+Olston%2C+Noah+Fiedel%2C+Kiril+Gorovoy%2C+Jeremiah+Harmsen%2C+Li+Lao%2C+Fangwei+Li%2C+Vinu+Rajashekhar%2C+Sukriti+Ramesh%2C+and+Jordan+Soyke%2E+Tensorflow%2Dserving%3A+Flexible%2C+high%2Dperformance+ml+serving%2E+arX)
35. OpenAI. Chatgpt. [https://chat.openai.com/chat](https://chat.openai.com/chat). [↗](https://chat.openai.com/chat)
36. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. arXiv preprint arXiv:2211.05102, 2022. [find](https://scholar.google.com/scholar?q=Reiner+Pope%2C+Sholto+Douglas%2C+Aakanksha+Chowdhery%2C+Jacob+Devlin%2C+James+Bradbury%2C+Anselm+Levskaya%2C+Jonathan+Heek%2C+Kefan+Xiao%2C+Shivani+Agrawal%2C+and+Jeff+Dean%2E+Efficiently+scaling+transformer+inference%2E+a)
37. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020. [find](https://scholar.google.com/scholar?q=Samyam+Rajbhandari%2C+Jeff+Rasley%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+Zero%3A+Memory+optimizations+toward+training+trillion+parameter+models%2E+In+SC20%3A+International+Conference+for+High+Performance+Computing)
38. KV Rashmi, Mosharaf Chowdhury, Jack Kosaian, Ion Stoica, and Kannan Ramchandran. Ec-cache: Load-balanced, low-latency cluster caching with online erasure coding. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pages 401–417, 2016. [find](https://scholar.google.com/scholar?q=KV+Rashmi%2C+Mosharaf+Chowdhury%2C+Jack+Kosaian%2C+Ion+Stoica%2C+and+Kannan+Ramchandran%2E+Ec%2Dcache%3A+Load%2Dbalanced%2C+low%2Dlatency+cluster+caching+with+online+erasure+coding%2E+In+12th+USENIX+Symposium+on+Operating+)
39. Stewart Robinson. Simulation: the practice of model development and use. Bloomsbury Publishing, 2014. [find](https://scholar.google.com/scholar?q=Stewart+Robinson%2E+Simulation%3A+the+practice+of+model+development+and+use%2E+Bloomsbury+Publishing%2C+2014%2E)
40. Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis. INFaaS: Automated model-less inference serving. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 397–411. USENIX Association, July 2021. [find](https://scholar.google.com/scholar?q=Francisco+Romero%2C+Qian+Li%2C+Neeraja+J%2E+Yadwadkar%2C+and+Christos+Kozyrakis%2E+INFaaS%3A+Automated+model%2Dless+inference+serving%2E+In+2021+USENIX+Annual+Technical+Conference+%28USENIX+ATC+21%29%2C+pages+397%E2%80%93411%2E+USEN)
41. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. [Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter](https://md.research.elamur.ai/free/en/papers/sanh-2019-distilbert-distilled-version-bert-smaller/). arXiv preprint arXiv:1910.01108, 2019.
42. Mohammad Shahrad, Rodrigo Fonseca, Íñigo Goiri, Gohar Chaudhry, Paul Batum, Jason Cooke, Eduardo Laureano, Colby Tresness, Mark Russinovich, and Ricardo Bianchini. Serverless in the wild: Characterizing and optimizing the serverless workload at a large cloud provider. In 2020 USENIX Annual Technical… [find](https://scholar.google.com/scholar?q=Mohammad+Shahrad%2C+Rodrigo+Fonseca%2C+%C3%8D%C3%B1igo+Goiri%2C+Gohar+Chaudhry%2C+Paul+Batum%2C+Jason+Cooke%2C+Eduardo+Laureano%2C+Colby+Tresness%2C+Mark+Russinovich%2C+and+Ricardo+Bianchini%2E+Serverless+in+the+wild%3A+Characterizi)
43. Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. [Mesh-TensorFlow: Deep learning for supercomputers](https://md.research.elamur.ai/free/en/papers/shazeer-2018-mesh-tensorflow-deep-learning-supercomputers/). In Neural Information Processing Systems, 2018.
44. Haichen Shen, Lequn Chen, Yuchen Jin, Liangyu Zhao, Bingyu Kong, Matthai Philipose, Arvind Krishnamurthy, and Ravi Sundaram. Nexus: A gpu cluster engine for accelerating dnn-based video analysis. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pages 322–337, 2019. [find](https://scholar.google.com/scholar?q=Haichen+Shen%2C+Lequn+Chen%2C+Yuchen+Jin%2C+Liangyu+Zhao%2C+Bingyu+Kong%2C+Matthai+Philipose%2C+Arvind+Krishnamurthy%2C+and+Ravi+Sundaram%2E+Nexus%3A+A+gpu+cluster+engine+for+accelerating+dnn%2Dbased+video+analysis%2E+In+P)
45. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+arXiv+preprin)
46. John F Shortle, James M Thompson, Donald Gross, and Carl M Harris. Fundamentals of queueing theory, volume 399. John Wiley & Sons, 2018. [find](https://scholar.google.com/scholar?q=John+F+Shortle%2C+James+M+Thompson%2C+Donald+Gross%2C+and+Carl+M+Harris%2E+Fundamentals+of+queueing+theory%2C+volume+399%2E+John+Wiley+%26+Sons%2C+2018%2E)
47. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
48. Qizhen Weng, Wencong Xiao, Yinghao Yu, Wei Wang, Cheng Wang, Jian He, Yong Li, Liping Zhang, Wei Lin, and Yu Ding. Mlaas in the wild: Workload analysis and scheduling in large-scale heterogeneous gpu clusters. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22), pages 9… [find](https://scholar.google.com/scholar?q=Qizhen+Weng%2C+Wencong+Xiao%2C+Yinghao+Yu%2C+Wei+Wang%2C+Cheng+Wang%2C+Jian+He%2C+Yong+Li%2C+Liping+Zhang%2C+Wei+Lin%2C+and+Yu+Ding%2E+Mlaas+in+the+wild%3A+Workload+analysis+and+scheduling+in+large%2Dscale+heterogeneous+gpu+)
49. Bingyang Wu, Zili Zhang, Zhihao Bai, Xuanzhe Liu, and Xin Jin. Transparent GPU sharing in container clouds for deep learning workloads. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 69–85, Boston, MA, April 2023. USENIX Association. [find](https://scholar.google.com/scholar?q=Bingyang+Wu%2C+Zili+Zhang%2C+Zhihao+Bai%2C+Xuanzhe+Liu%2C+and+Xin+Jin%2E+Transparent+GPU+sharing+in+container+clouds+for+deep+learning+workloads%2E+In+20th+USENIX+Symposium+on+Networked+Systems+Design+and+Impleme)
50. Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, et al. Gspmd: general and scalable parallelization for ml computation graphs. arXiv preprint arXiv:2105.04663, 2021. [find](https://scholar.google.com/scholar?q=Yuanzhong+Xu%2C+HyoukJoong+Lee%2C+Dehao+Chen%2C+Blake+Hechtman%2C+Yanping+Huang%2C+Rahul+Joshi%2C+Maxim+Krikun%2C+Dmitry+Lepikhin%2C+Andy+Ly%2C+Marcello+Maggioni%2C+et+al%2E+Gspmd%3A+general+and+scalable+parallelization+for+)
51. Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. [Orca: A distributed serving system for transformer-based generative models](https://md.research.elamur.ai/free/en/papers/yu-2022-orca-distributed-serving-system-transformer/). In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, 2022.
52. Hong Zhang, Yupeng Tang, Anurag Khandelwal, and Ion Stoica. {SHEPHERD}: Serving {DNNs} in the wild. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 787– 808, 2023. [find](https://scholar.google.com/scholar?q=Hong+Zhang%2C+Yupeng+Tang%2C+Anurag+Khandelwal%2C+and+Ion+Stoica%2E+%7BSHEPHERD%7D%3A+Serving+%7BDNNs%7D+in+the+wild%2E+In+20th+USENIX+Symposium+on+Networked+Systems+Design+and+Implementation+%28NSDI+23%29%2C+pages+787%E2%80%93+808%2C+2)
53. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068, 2022.
54. Yanqi Zhang, Íñigo Goiri, Gohar Irfan Chaudhry, Rodrigo Fonseca, Sameh Elnikety, Christina Delimitrou, and Ricardo Bianchini. Faster and cheaper serverless computing on harvested resources. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles, pages 724–739, 2021. [find](https://scholar.google.com/scholar?q=Yanqi+Zhang%2C+%C3%8D%C3%B1igo+Goiri%2C+Gohar+Irfan+Chaudhry%2C+Rodrigo+Fonseca%2C+Sameh+Elnikety%2C+Christina+Delimitrou%2C+and+Ricardo+Bianchini%2E+Faster+and+cheaper+serverless+computing+on+harvested+resources%2E+In+Proceed)
55. Yihao Zhao, Yuanqiang Liu, Yanghua Peng, Yibo Zhu, Xuanzhe Liu, and Xin Jin. Multi-resource interleaving for deep learning training. In Proceedings of the ACM SIGCOMM 2022 Conference, pages 428–440, 2022. [find](https://scholar.google.com/scholar?q=Yihao+Zhao%2C+Yuanqiang+Liu%2C+Yanghua+Peng%2C+Yibo+Zhu%2C+Xuanzhe+Liu%2C+and+Xin+Jin%2E+Multi%2Dresource+interleaving+for+deep+learning+training%2E+In+Proceedings+of+the+ACM+SIGCOMM+2022+Conference%2C+pages+428%E2%80%93440%2C+2)
56. Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Joseph E Gonzalez, et al. Alpa: Automating inter-and intra-operator parallelism for distributed deep learning. In 16th USENIX Symposium on Operating Systems Design and Implementa… — [Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning](https://md.research.elamur.ai/free/en/papers/zheng-2022-alpa-automating-inter-intra-operator/)
57. Zhe Zhou, Xuechao Wei, Jiejing Zhang, and Guangyu Sun. Pets: A unified framework for parameter-efficient transformers serving. In 2022 USENIX Annual Technical Conference (USENIX ATC 22), pages 489–504, 2022. [find](https://scholar.google.com/scholar?q=Zhe+Zhou%2C+Xuechao+Wei%2C+Jiejing+Zhang%2C+and+Guangyu+Sun%2E+Pets%3A+A+unified+framework+for+parameter%2Defficient+transformers+serving%2E+In+2022+USENIX+Annual+Technical+Conference+%28USENIX+ATC+22%29%2C+pages+489%E2%80%93504)

*Updated 2 November 2026*