---
title: "Splitwise: Efficient generative LLM inference using phase splitting"
description: "The authors use it to design homogeneous and heterogeneous clusters for throughput, cost, or power. 85 references, cited by 373 papers here."
authors: ["Pratyush Patel", "Esha Choukse", "Chaojie Zhang", "Aashaka Shah", "Íñigo Goiri", "Saeed Maleki", "Ricardo Bianchini"]
year: 2023
venue: "ISCA 2024"
source: "https://arxiv.org/abs/2311.18677"
arxiv: "2311.18677"
licence: "CC BY-NC-SA 4.0 (arXiv OAI)"
licence_url: "http://creativecommons.org/licenses/by-nc-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/patel-2023-splitwise-efficient-generative-llm-inference/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/patel-2023-splitwise-efficient-generative-llm-inference/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/) / Splitwise: Efficient generative LLM inference using phase splitting

# Splitwise: Efficient generative LLM inference using phase splitting

**Authors:** Pratyush Patel, [Esha Choukse](https://md.research.elamur.ai/free/en/authors/esha-choukse/), [Chaojie Zhang](https://md.research.elamur.ai/free/en/authors/chaojie-zhang/), Aashaka Shah, [Íñigo Goiri](https://md.research.elamur.ai/free/en/authors/inigo-goiri/), Saeed Maleki and Ricardo Bianchini · **Year:** 2023 · **Venue:** ISCA 2024 · **Licence:** [CC BY-NC-SA 4.0 (arXiv OAI)](http://creativecommons.org/licenses/by-nc-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2311.18677) · [PDF](https://arxiv.org/pdf/2311.18677) · [arXiv 2311.18677](https://arxiv.org/abs/2311.18677) · [OpenAlex](https://openalex.org/W4389261323)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [373](https://md.research.elamur.ai/free/en/papers/patel-2023-splitwise-efficient-generative-llm-inference/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Serving generative LLMs at scale is costly and power-hungry, and GPUs are often over-provisioned to meet latency SLOs. The authors profile LLM inference on NVIDIA A100 and H100 machines using production request traces from Microsoft Azure. Every request has two stages with different needs. The prompt stage processes all input tokens at once and is limited by compute. The token generation stage emits tokens one at a time and is limited by memory bandwidth and capacity, leaving compute idle even with modern batching. Splitwise runs these two stages on separate machine pools and moves the KV cache from the prompt machine to the token machine over the cluster's Infiniband back-plane, using optimized communication libraries. This allows each pool to be sized, power-capped, or built from a different GPU generation independently. The authors use it to design homogeneous and heterogeneous clusters for throughput, cost, or power. The best designs reach about 1.4× more throughput at 20% lower cost, or 2.35× more throughput under the same cost and power. The results suggest that cheaper or older GPUs are adequate for token generation.

## Key points
- Hardware trend motivating the work: H100 has 3.43× the compute of A100 and draws 1.75× the power. Its memory bandwidth grew only about 1.64×, and HBM capacity stayed at 80GB. Cost per machine is 2.16× higher.
- Evaluated models are Llama2-70B and BLOOM-176B, served with vLLM on 8-GPU H100 machines using tensor parallelism across all 8 GPUs. Mixed batching is the default.
- The workload consists of two Azure production traces, coding and conversation. Prompt text is unavailable for privacy reasons (e.g., GDPR), so synthetic prompts reproduce each request's input and output token counts.
- The latency metrics are E2E latency and TTFT, plus TBT, which the authors add to capture streaming speed. Throughput is measured in requests per second.
- Released artifacts are bundled on Zenodo (DOI 10.5281/zenodo.11003049):
  - the traces, under CC-BY;
  - a KV-cache transfer prototype for vLLM, built with MSCCL++;
  - SplitwiseSim, a discrete-event cluster simulator, under the MIT license.

## Interesting ideas
- **Split a workload by its bottleneck, not by request.** When one job has stages with opposite resource profiles, giving each stage its own hardware pool can beat co-locating them. The same reasoning could apply to other multi-stage NLP or data pipelines.
- **Use older or power-capped hardware for memory-bound stages.** Token generation gains little from the newest GPUs, so lower-cost or more available accelerators can handle it. This is useful for teams with limited budgets or no access to flagship GPUs.
- **Benchmark with size-only traces.** Real prompts could not be shared, so the authors replayed only the input and output token counts, since performance depends on those alone. This lets realistic load tests be shared without exposing private user data.
- **Measure time between tokens.** Tracking TBT separately from TTFT and E2E latency reveals harm to streaming users that averages would hide. It is worth adopting when evaluating any interactive text-generation service.
- **Explore cluster designs in a simulator first.** A discrete-event simulator fed with real traces makes it cheap to compare cluster layouts, power caps, and GPU mixes before buying or reserving hardware.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 378 works · 161,630 citations
- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 402 works · 212,763 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki and Ricardo Bianchini. 2023. Splitwise: Efficient generative LLM inference using phase splitting. ISCA 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{patel2023splitwise,
  title = {{Splitwise: Efficient generative LLM inference using phase splitting}},
  author = {Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki and Ricardo Bianchini},
  booktitle = {ISCA 2024},
  year = {2023},
  eprint = {2311.18677},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.18677}
}
```

- [← previous VILA: On Pre-training for Visual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2023-vila-pre-training-visual-language/)
- [next → Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning](https://md.research.elamur.ai/free/en/papers/pan-2023-logic-lm-empowering-large-language/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/)

## References

85 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Add Splitwise Implementation to vLLM. GitHub. [Online]. Available: [https://github.com/vllm-project/vllm/pull/2809](https://github.com/vllm-project/vllm/pull/2809) [↗](https://github.com/vllm-project/vllm/pull/2809)
2. AMD Instinct™ MI250 Accelerator. [Online]. Available: [https://www.amd.com/en/products/server-accelerators/instinct-mi250](https://www.amd.com/en/products/server-accelerators/instinct-mi250) [↗](https://www.amd.com/en/products/server-accelerators/instinct-mi250)
3. Azure InfiniBand HPC VMs. [Online]. Available: [https://learn.microsoft.com/en-us/azure/virtual-machines/overview-hb-hc](https://learn.microsoft.com/en-us/azure/virtual-machines/overview-hb-hc) [↗](https://learn.microsoft.com/en-us/azure/virtual-machines/overview-hb-hc)
4. Azure Public Dataset: Azure LLM Inference Trace 2023. GitHub. [Online]. Available: [https://github.com/Azure/AzurePublicDataset/blob/master/AzureLLMInferenceDataset2023.md](https://github.com/Azure/AzurePublicDataset/blob/master/AzureLLMInferenceDataset2023.md) [↗](https://github.com/Azure/AzurePublicDataset/blob/master/AzureLLMInferenceDataset2023.md)
5. CoreWeave - Specialized Cloud Provider. [Online]. Available: [https://www.coreweave.com](https://www.coreweave.com) [↗](https://www.coreweave.com)
6. Google Assistant with Bard. [Online]. Available: [https://blog.google/products/assistant/google-assistant-bard-generative-ai/](https://blog.google/products/assistant/google-assistant-bard-generative-ai/) [↗](https://blog.google/products/assistant/google-assistant-bard-generative-ai/)
7. HPC Interconnect on CoreWeave Cloud. [Online]. Available: [https://docs.coreweave.com/networking/hpc-interconnect](https://docs.coreweave.com/networking/hpc-interconnect) [↗](https://docs.coreweave.com/networking/hpc-interconnect)
8. Intel BigDL-LLM. [Online]. Available: [https://github.com/intel-analytics/BigDL](https://github.com/intel-analytics/BigDL) [↗](https://github.com/intel-analytics/BigDL)
9. Intel Sapphire Rapids with HBM. [Online]. Available: [https://www.anandtech.com/show/17422/intel-showcases-sapphire-rapids-plus-hbm-xeon-performance-isc-2022](https://www.anandtech.com/show/17422/intel-showcases-sapphire-rapids-plus-hbm-xeon-performance-isc-2022) [↗](https://www.anandtech.com/show/17422/intel-showcases-sapphire-rapids-plus-hbm-xeon-performance-isc-2022)
10. Microsoft Azure ND A100 v4-series . [Online]. Available: [https://learn.microsoft.com/en-us/azure/virtual-machines/nda100-v4-series](https://learn.microsoft.com/en-us/azure/virtual-machines/nda100-v4-series) [↗](https://learn.microsoft.com/en-us/azure/virtual-machines/nda100-v4-series)
11. MSCCL++: A GPU-driven communication stack for scalable AI applications. [Online]. Available: [https://github.com/microsoft/mscclpp](https://github.com/microsoft/mscclpp) [↗](https://github.com/microsoft/mscclpp)
12. Numenta Inference on CPUs. [Online]. Available: [https://www.servethehome.com/numenta-has-the-secret-to-ai-inference-on-cpus-like-the-intel-xeon-max/](https://www.servethehome.com/numenta-has-the-secret-to-ai-inference-on-cpus-like-the-intel-xeon-max/) [↗](https://www.servethehome.com/numenta-has-the-secret-to-ai-inference-on-cpus-like-the-intel-xeon-max/)
13. NVIDIA Accelerated InfiniBand Solutions. [Online]. Available: [https://www.nvidia.com/en-us/networking/products/infiniband/](https://www.nvidia.com/en-us/networking/products/infiniband/) [↗](https://www.nvidia.com/en-us/networking/products/infiniband/)
14. NVIDIA Chip Shortage. [Online]. Available: [https://www.wired.com/story/nvidia-chip-shortages-leave-ai-startups-scrambling-for-computing-power/](https://www.wired.com/story/nvidia-chip-shortages-leave-ai-startups-scrambling-for-computing-power/) [↗](https://www.wired.com/story/nvidia-chip-shortages-leave-ai-startups-scrambling-for-computing-power/)
15. NVIDIA DGX A100: Universal System for AI Infrastructure. [Online]. Available: [https://resources.nvidia.com/en-us-dgx-systems/dgx-ai](https://resources.nvidia.com/en-us-dgx-systems/dgx-ai) [↗](https://resources.nvidia.com/en-us-dgx-systems/dgx-ai)
16. NVIDIA DGX H100. [Online]. Available: [https://www.nvidia.com/en-us/data-center/dgx-h100/](https://www.nvidia.com/en-us/data-center/dgx-h100/) [↗](https://www.nvidia.com/en-us/data-center/dgx-h100/)
17. NVIDIA Hopper GPUs Expand Reach as Demand for AI Grows. [Online]. Available: [https://nvidianews.nvidia.com/news/nvidia-hopper-gpus-expand-reach-as-demand-for-ai-grows](https://nvidianews.nvidia.com/news/nvidia-hopper-gpus-expand-reach-as-demand-for-ai-grows) [↗](https://nvidianews.nvidia.com/news/nvidia-hopper-gpus-expand-reach-as-demand-for-ai-grows)
18. OpenAI ChatGPT APIs. [Online]. Available: [https://openai.com/blog/introducing-chatgpt-and-whisper-apis](https://openai.com/blog/introducing-chatgpt-and-whisper-apis) [↗](https://openai.com/blog/introducing-chatgpt-and-whisper-apis)
19. Power Availability Stymies Datacenter Growth. [Online]. Available: [https://www.networkworld.com/article/972483/power-availability-stymies-data-center-growth](https://www.networkworld.com/article/972483/power-availability-stymies-data-center-growth). [↗](https://www.networkworld.com/article/972483/power-availability-stymies-data-center-growth)
20. SplitwiseSim: LLM Serving Cluster Simulator. GitHub. [Online]. Available: [https://github.com/Mutinifni/splitwise-sim](https://github.com/Mutinifni/splitwise-sim) [↗](https://github.com/Mutinifni/splitwise-sim)
21. The New Bing. [Online]. Available: [https://www.microsoft.com/en-us/edge/features/the-new-bing?form=MT00D8](https://www.microsoft.com/en-us/edge/features/the-new-bing?form=MT00D8) [↗](https://www.microsoft.com/en-us/edge/features/the-new-bing?form=MT00D8)
22. TurboMind Inference Server. [Online]. Available: [https://github.com/InternLM/lmdeploy](https://github.com/InternLM/lmdeploy) [↗](https://github.com/InternLM/lmdeploy)
23. A. Agrawal, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, and R. Ramjee, “[SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills](https://md.research.elamur.ai/free/en/papers/agrawal-2023-sarathi-efficient-llm-inference-piggybacking/),” arXiv preprint arXiv:2308.16369, 2023.
24. H. Albahar, S. Dongare, Y. Du, N. Zhao, A. K. Paul, and A. R. Butt, “SchedTune: A heterogeneity-aware GPU Scheduler for Deep Learning,” in CCGrid, 2022. [find](https://scholar.google.com/scholar?q=H%2E+Albahar%2C+S%2E+Dongare%2C+Y%2E+Du%2C+N%2E+Zhao%2C+A%2E+K%2E+Paul%2C+and+A%2E+R%2E+Butt%2C+%E2%80%9CSchedTune%3A+A+heterogeneity%2Daware+GPU+Scheduler+for+Deep+Learning%2C%E2%80%9D+in+CCGrid%2C+2022%2E)
25. R. Y. Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Rasley, and Y. He, “DeepSpeed-Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale,” in SC, 2022. [find](https://scholar.google.com/scholar?q=R%2E+Y%2E+Aminabadi%2C+S%2E+Rajbhandari%2C+A%2E+A%2E+Awan%2C+C%2E+Li%2C+D%2E+Li%2C+E%2E+Zheng%2C+O%2E+Ruwase%2C+S%2E+Smith%2C+M%2E+Zhang%2C+J%2E+Rasley%2C+and+Y%2E+He%2C+%E2%80%9CDeepSpeed%2DInference%3A+Enabling+Efficient+Inference+of+Transformer+Models+at+Un)
26. L. A. Barroso, U. Hölzle, and P. Ranganathan, “The Datacenter as a Computer: Designing Warehouse-Scale Machines,” Synthesis Lectures on Computer Architecture, 2018. [find](https://scholar.google.com/scholar?q=L%2E+A%2E+Barroso%2C+U%2E+H%C3%B6lzle%2C+and+P%2E+Ranganathan%2C+%E2%80%9CThe+Datacenter+as+a+Computer%3A+Designing+Warehouse%2DScale+Machines%2C%E2%80%9D+Synthesis+Lectures+on+Computer+Architecture%2C+2018%2E)
27. E. Boutin, J. Ekanayake, W. Lin, B. Shi, J. Zhou, Z. Qian, M. Wu, and L. Zhou, “Apollo: Scalable and Coordinated Scheduling for Cloud-scale Computing,” in OSDI, 2014. [find](https://scholar.google.com/scholar?q=E%2E+Boutin%2C+J%2E+Ekanayake%2C+W%2E+Lin%2C+B%2E+Shi%2C+J%2E+Zhou%2C+Z%2E+Qian%2C+M%2E+Wu%2C+and+L%2E+Zhou%2C+%E2%80%9CApollo%3A+Scalable+and+Coordinated+Scheduling+for+Cloud%2Dscale+Computing%2C%E2%80%9D+in+OSDI%2C+2014%2E)
28. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Ber… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
29. J. Chen and L. K. John, “Efficient Program Scheduling for Heterogeneous Multi-core Processors,” in DAC, 2009. [find](https://scholar.google.com/scholar?q=J%2E+Chen+and+L%2E+K%2E+John%2C+%E2%80%9CEfficient+Program+Scheduling+for+Heterogeneous+Multi%2Dcore+Processors%2C%E2%80%9D+in+DAC%2C+2009%2E)
30. Y. Choi, J. Kim, and M. Rhu, “Hera: A Heterogeneity-Aware Multi-Tenant Inference Server for Personalized Recommendations,” arXiv preprint arXiv:2302.11750, 2023. — [arXiv:2302.11750](https://arxiv.org/abs/2302.11750) [find](https://scholar.google.com/scholar?q=Y%2E+Choi%2C+J%2E+Kim%2C+and+M%2E+Rhu%2C+%E2%80%9CHera%3A+A+Heterogeneity%2DAware+Multi%2DTenant+Inference+Server+for+Personalized+Recommendations%2C%E2%80%9D+arXiv+preprint+arXiv%3A2302%2E11750%2C+2023%2E+%E2%80%94+%5BarXiv%3A2302%2E11750%5D%28https%3A%2F%2Farxiv%2Eorg)
31. D. Crankshaw, G.-E. Sela, X. Mo, C. Zumar, I. Stoica, J. Gonzalez, and A. Tumanov, “InferLine: Latency-aware Provisioning and Scaling for Prediction Serving Pipelines,” in SoCC, 2020. [find](https://scholar.google.com/scholar?q=D%2E+Crankshaw%2C+G%2E%2DE%2E+Sela%2C+X%2E+Mo%2C+C%2E+Zumar%2C+I%2E+Stoica%2C+J%2E+Gonzalez%2C+and+A%2E+Tumanov%2C+%E2%80%9CInferLine%3A+Latency%2Daware+Provisioning+and+Scaling+for+Prediction+Serving+Pipelines%2C%E2%80%9D+in+SoCC%2C+2020%2E)
32. T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré, “FlashAttention: Fast and Memory-efficient Exact Attention with IO-Awareness,” in NeurIPS, 2022. [find](https://scholar.google.com/scholar?q=T%2E+Dao%2C+D%2E+Fu%2C+S%2E+Ermon%2C+A%2E+Rudra%2C+and+C%2E+R%C3%A9%2C+%E2%80%9CFlashAttention%3A+Fast+and+Memory%2Defficient+Exact+Attention+with+IO%2DAwareness%2C%E2%80%9D+in+NeurIPS%2C+2022%2E)
33. David Patterson. Domain Specific Architectures for Deep Neural Networks: Three Generations of Tensor Processing Units (TPUs). Allen School Distinguished Lecture. [Online]. Available: [https://www.youtube.com/watch?v=VCScWh966u4](https://www.youtube.com/watch?v=VCScWh966u4) [↗](https://www.youtube.com/watch?v=VCScWh966u4)
34. J. Dean and S. Ghemawat, “MapReduce: Simplified Data Processing on Large Clusters,” Communications of the ACM, 2008. [find](https://scholar.google.com/scholar?q=J%2E+Dean+and+S%2E+Ghemawat%2C+%E2%80%9CMapReduce%3A+Simplified+Data+Processing+on+Large+Clusters%2C%E2%80%9D+Communications+of+the+ACM%2C+2008%2E)
35. T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale,” arXiv preprint arXiv:2208.07339, 2022. [find](https://scholar.google.com/scholar?q=T%2E+Dettmers%2C+M%2E+Lewis%2C+Y%2E+Belkada%2C+and+L%2E+Zettlemoyer%2C+%E2%80%9CLLM%2Eint8%28%29%3A+8%2Dbit+Matrix+Multiplication+for+Transformers+at+Scale%2C%E2%80%9D+arXiv+preprint+arXiv%3A2208%2E07339%2C+2022%2E)
36. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “[BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/),” in NAACL, 2019.
37. A. Gujarati, R. Karimi, S. Alzayat, W. Hao, A. Kaufmann, Y. Vigfusson, and J. Mace, “Serving DNNs like Clockwork: Performance Predictability from the Bottom Up,” in OSDI, 2020. [find](https://scholar.google.com/scholar?q=A%2E+Gujarati%2C+R%2E+Karimi%2C+S%2E+Alzayat%2C+W%2E+Hao%2C+A%2E+Kaufmann%2C+Y%2E+Vigfusson%2C+and+J%2E+Mace%2C+%E2%80%9CServing+DNNs+like+Clockwork%3A+Performance+Predictability+from+the+Bottom+Up%2C%E2%80%9D+in+OSDI%2C+2020%2E)
38. U. Gupta, S. Hsia, V. Saraph, X. Wang, B. Reagen, G.-Y. Wei, H.-H. S. Lee, D. Brooks, and C.-J. Wu, “DeepRecSys: A System for Optimizing End-to-end At-scale Neural Recommendation Inference,” in ISCA, 2020. [find](https://scholar.google.com/scholar?q=U%2E+Gupta%2C+S%2E+Hsia%2C+V%2E+Saraph%2C+X%2E+Wang%2C+B%2E+Reagen%2C+G%2E%2DY%2E+Wei%2C+H%2E%2DH%2E+S%2E+Lee%2C+D%2E+Brooks%2C+and+C%2E%2DJ%2E+Wu%2C+%E2%80%9CDeepRecSys%3A+A+System+for+Optimizing+End%2Dto%2Dend+At%2Dscale+Neural+Recommendation+Inference%2C%E2%80%9D+in+ISCA%2C+)
39. V. Gupta, M. Harchol Balter, K. Sigman, and W. Whitt, “Analysis of Join-the-Shortest-Queue Routing for Web Server Farms,” Performance Evaluation, 2007. [find](https://scholar.google.com/scholar?q=V%2E+Gupta%2C+M%2E+Harchol+Balter%2C+K%2E+Sigman%2C+and+W%2E+Whitt%2C+%E2%80%9CAnalysis+of+Join%2Dthe%2DShortest%2DQueue+Routing+for+Web+Server+Farms%2C%E2%80%9D+Performance+Evaluation%2C+2007%2E)
40. M. E. Haque, Y. H. Eom, Y. He, S. Elnikety, R. Bianchini, and K. S. McKinley, “Few-to-Many: Incremental Parallelism for Reducing Tail Latency in Interactive Services,” ACM SIGPLAN Notices, 2015. [find](https://scholar.google.com/scholar?q=M%2E+E%2E+Haque%2C+Y%2E+H%2E+Eom%2C+Y%2E+He%2C+S%2E+Elnikety%2C+R%2E+Bianchini%2C+and+K%2E+S%2E+McKinley%2C+%E2%80%9CFew%2Dto%2DMany%3A+Incremental+Parallelism+for+Reducing+Tail+Latency+in+Interactive+Services%2C%E2%80%9D+ACM+SIGPLAN+Notices%2C+2015%2E)
41. M. E. Haque, Y. He, S. Elnikety, T. D. Nguyen, R. Bianchini, and K. S. McKinley, “Exploiting Heterogeneity for Tail Latency and Energy Efficiency,” in MICRO, 2017. [find](https://scholar.google.com/scholar?q=M%2E+E%2E+Haque%2C+Y%2E+He%2C+S%2E+Elnikety%2C+T%2E+D%2E+Nguyen%2C+R%2E+Bianchini%2C+and+K%2E+S%2E+McKinley%2C+%E2%80%9CExploiting+Heterogeneity+for+Tail+Latency+and+Energy+Efficiency%2C%E2%80%9D+in+MICRO%2C+2017%2E)
42. K. Hong, G. Dai, J. Xu, Q. Mao, X. Li, J. Liu, K. Chen, H. Dong, and Y. Wang, “FlashDecoding++: Faster Large Language Model Inference on GPUs,” arXiv preprint arXiv:2311.01282, 2023. [find](https://scholar.google.com/scholar?q=K%2E+Hong%2C+G%2E+Dai%2C+J%2E+Xu%2C+Q%2E+Mao%2C+X%2E+Li%2C+J%2E+Liu%2C+K%2E+Chen%2C+H%2E+Dong%2C+and+Y%2E+Wang%2C+%E2%80%9CFlashDecoding%2B%2B%3A+Faster+Large+Language+Model+Inference+on+GPUs%2C%E2%80%9D+arXiv+preprint+arXiv%3A2311%2E01282%2C+2023%2E)
43. Y. Hu, R. Ghosh, and R. Govindan, “Scrooge: A Cost-effective Deep Learning Inference System,” in SoCC, 2021. [find](https://scholar.google.com/scholar?q=Y%2E+Hu%2C+R%2E+Ghosh%2C+and+R%2E+Govindan%2C+%E2%80%9CScrooge%3A+A+Cost%2Deffective+Deep+Learning+Inference+System%2C%E2%80%9D+in+SoCC%2C+2021%2E)
44. Huggingface. Text Generation Inference (TGI). [Online]. Available: [https://github.com/huggingface/text-generation-inference](https://github.com/huggingface/text-generation-inference) [↗](https://github.com/huggingface/text-generation-inference)
45. R. Hwang, T. Kim, Y. Kwon, and M. Rhu, “Centaur: A Chiplet-based, Hybrid Sparse-Dense Accelerator for Personalized Recommendations,” in ISCA, 2020. [find](https://scholar.google.com/scholar?q=R%2E+Hwang%2C+T%2E+Kim%2C+Y%2E+Kwon%2C+and+M%2E+Rhu%2C+%E2%80%9CCentaur%3A+A+Chiplet%2Dbased%2C+Hybrid+Sparse%2DDense+Accelerator+for+Personalized+Recommendations%2C%E2%80%9D+in+ISCA%2C+2020%2E)
46. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, “Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks,” in EuroSys, 2007. [find](https://scholar.google.com/scholar?q=M%2E+Isard%2C+M%2E+Budiu%2C+Y%2E+Yu%2C+A%2E+Birrell%2C+and+D%2E+Fetterly%2C+%E2%80%9CDryad%3A+Distributed+Data%2DParallel+Programs+from+Sequential+Building+Blocks%2C%E2%80%9D+in+EuroSys%2C+2007%2E)
47. M. Javaheripi and S. Bubeck, “Phi-2: The Surprising Power of Small Language Models,” Microsoft Research Blog, 2023. [find](https://scholar.google.com/scholar?q=M%2E+Javaheripi+and+S%2E+Bubeck%2C+%E2%80%9CPhi%2D2%3A+The+Surprising+Power+of+Small+Language+Models%2C%E2%80%9D+Microsoft+Research+Blog%2C+2023%2E)
48. W. Jiang, Z. He, S. Zhang, K. Zeng, L. Feng, J. Zhang, T. Liu, Y. Li, J. Zhou, C. Zhang et al., “FleetRec: Large-scale Recommendation Inference on Hybrid GPU-FPGA Clusters,” in KDD, 2021. [find](https://scholar.google.com/scholar?q=W%2E+Jiang%2C+Z%2E+He%2C+S%2E+Zhang%2C+K%2E+Zeng%2C+L%2E+Feng%2C+J%2E+Zhang%2C+T%2E+Liu%2C+Y%2E+Li%2C+J%2E+Zhou%2C+C%2E+Zhang+et+al%2E%2C+%E2%80%9CFleetRec%3A+Large%2Dscale+Recommendation+Inference+on+Hybrid+GPU%2DFPGA+Clusters%2C%E2%80%9D+in+KDD%2C+2021%2E)
49. R. S. Kannan, L. Subramanian, A. Raju, J. Ahn, J. Mars, and L. Tang, “GrandSLAm: Guaranteeing SLAs for Jobs in Microservices Execution Frameworks,” in EuroSys, 2019. [find](https://scholar.google.com/scholar?q=R%2E+S%2E+Kannan%2C+L%2E+Subramanian%2C+A%2E+Raju%2C+J%2E+Ahn%2C+J%2E+Mars%2C+and+L%2E+Tang%2C+%E2%80%9CGrandSLAm%3A+Guaranteeing+SLAs+for+Jobs+in+Microservices+Execution+Frameworks%2C%E2%80%9D+in+EuroSys%2C+2019%2E)
50. R. Kumar, K. I. Farkas, N. P. Jouppi, P. Ranganathan, and D. M. Tullsen, “Single-ISA Heterogeneous Multi-core Architectures: The Potential for Processor Power Reduction,” in MICRO, 2003. [find](https://scholar.google.com/scholar?q=R%2E+Kumar%2C+K%2E+I%2E+Farkas%2C+N%2E+P%2E+Jouppi%2C+P%2E+Ranganathan%2C+and+D%2E+M%2E+Tullsen%2C+%E2%80%9CSingle%2DISA+Heterogeneous+Multi%2Dcore+Architectures%3A+The+Potential+for+Processor+Power+Reduction%2C%E2%80%9D+in+MICRO%2C+2003%2E)
51. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “[Efficient Memory Management for Large Language Model Serving with PagedAttention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/),” in SOSP, 2023.
52. Y. Kwon, Y. Lee, and M. Rhu, “TensorDIMM: A Practical Near-Memory Processing Architecture for Embeddings and Tensor Operations in Deep Learning,” in MICRO, 2019. [find](https://scholar.google.com/scholar?q=Y%2E+Kwon%2C+Y%2E+Lee%2C+and+M%2E+Rhu%2C+%E2%80%9CTensorDIMM%3A+A+Practical+Near%2DMemory+Processing+Architecture+for+Embeddings+and+Tensor+Operations+in+Deep+Learning%2C%E2%80%9D+in+MICRO%2C+2019%2E)
53. Z. Li, L. Zheng, Y. Zhong, V. Liu, Y. Sheng, X. Jin, Y. Huang, Z. Chen, H. Zhang, J. E. Gonzalez, and I. Stoica, “[AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving](https://md.research.elamur.ai/free/en/papers/li-2023-alpaserve-statistical-multiplexing-model-parallelism/),” in OSDI, 2023.
54. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “[RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/),” arXiv preprint arXiv:1907.11692, 2019.
55. Z. Liu, J. Wang, T. Dao, T. Zhou, B. Yuan, Z. Song, A. Shrivastava, C. Zhang, Y. Tian, C. Re, and B. Chen, “[Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time](https://md.research.elamur.ai/free/en/papers/liu-2023-deja-vu-contextual-sparsity-efficient/),” in ICML, 2023.
56. Meta. Introducing the AI Research SuperCluster — Meta’s Cutting-Edge AI Supercomputer for AI Research. [Online]. Available: [https://ai.facebook.com/blog/ai-rsc/](https://ai.facebook.com/blog/ai-rsc/) [↗](https://ai.facebook.com/blog/ai-rsc/)
57. “Azure OpenAI Service,” Microsoft Azure, 2022. [Online]. Available: [https://azure.microsoft.com/en-us/products/ai-services/openai-service](https://azure.microsoft.com/en-us/products/ai-services/openai-service) [↗](https://azure.microsoft.com/en-us/products/ai-services/openai-service)
58. R. Mittal, A. Shpiner, A. Panda, E. Zahavi, A. Krishnamurthy, S. Rat-nasamy, and S. Shenker, “Revisiting Network Support for RDMA,” arXiv preprint arXiv:1806.08159, 2018. — [arXiv:1806.08159](https://arxiv.org/abs/1806.08159) [find](https://scholar.google.com/scholar?q=R%2E+Mittal%2C+A%2E+Shpiner%2C+A%2E+Panda%2C+E%2E+Zahavi%2C+A%2E+Krishnamurthy%2C+S%2E+Rat%2Dnasamy%2C+and+S%2E+Shenker%2C+%E2%80%9CRevisiting+Network+Support+for+RDMA%2C%E2%80%9D+arXiv+preprint+arXiv%3A1806%2E08159%2C+2018%2E+%E2%80%94+%5BarXiv%3A1806%2E08159%5D%28https%3A%2F%2F)
59. P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, M. Elibol, Z. Yang, W. Paul, M. I. Jordan et al., “Ray: A Distributed Framework for Emerging AI Applications,” in OSDI, 2018. [find](https://scholar.google.com/scholar?q=P%2E+Moritz%2C+R%2E+Nishihara%2C+S%2E+Wang%2C+A%2E+Tumanov%2C+R%2E+Liaw%2C+E%2E+Liang%2C+M%2E+Elibol%2C+Z%2E+Yang%2C+W%2E+Paul%2C+M%2E+I%2E+Jordan+et+al%2E%2C+%E2%80%9CRay%3A+A+Distributed+Framework+for+Emerging+AI+Applications%2C%E2%80%9D+in+OSDI%2C+2018%2E)
60. OpenAI. Scaling Kubernetes to 7,500 Nodes. [Online]. Available: [https://openai.com/research/scaling-kubernetes-to-7500-nodes](https://openai.com/research/scaling-kubernetes-to-7500-nodes) [↗](https://openai.com/research/scaling-kubernetes-to-7500-nodes)
61. K. Ousterhout, P. Wendell, M. Zaharia, and I. Stoica, “Sparrow: Distributed, Low Latency Scheduling,” in SOSP, 2013. [find](https://scholar.google.com/scholar?q=K%2E+Ousterhout%2C+P%2E+Wendell%2C+M%2E+Zaharia%2C+and+I%2E+Stoica%2C+%E2%80%9CSparrow%3A+Distributed%2C+Low+Latency+Scheduling%2C%E2%80%9D+in+SOSP%2C+2013%2E)
62. P. Patel, E. Choukse, C. Zhang, Í. Goiri, B. Warrier, N. Mahalingam, and R. Bianchini, “POLCA: Power Oversubscription in LLM Cloud Providers,” arXiv preprint arXiv:2308.12908, 2023. [find](https://scholar.google.com/scholar?q=P%2E+Patel%2C+E%2E+Choukse%2C+C%2E+Zhang%2C+%C3%8D%2E+Goiri%2C+B%2E+Warrier%2C+N%2E+Mahalingam%2C+and+R%2E+Bianchini%2C+%E2%80%9CPOLCA%3A+Power+Oversubscription+in+LLM+Cloud+Providers%2C%E2%80%9D+arXiv+preprint+arXiv%3A2308%2E12908%2C+2023%2E)
63. P. Patel, E. Choukse, C. Zhang, Í. Goiri, B. Warrier, N. Mahalingam, and R. Bianchini, “Characterizing Power Management Opportunities for LLMs in the Cloud,” in ASPLOS, 2024. [find](https://scholar.google.com/scholar?q=P%2E+Patel%2C+E%2E+Choukse%2C+C%2E+Zhang%2C+%C3%8D%2E+Goiri%2C+B%2E+Warrier%2C+N%2E+Mahalingam%2C+and+R%2E+Bianchini%2C+%E2%80%9CCharacterizing+Power+Management+Opportunities+for+LLMs+in+the+Cloud%2C%E2%80%9D+in+ASPLOS%2C+2024%2E)
64. P. Patel, Z. Gong, S. Rizvi, E. Choukse, P. Misra, T. Anderson, and A. Sriraman, “Towards Improved Power Management in Cloud GPUs,” in IEEE CAL, 2023. [find](https://scholar.google.com/scholar?q=P%2E+Patel%2C+Z%2E+Gong%2C+S%2E+Rizvi%2C+E%2E+Choukse%2C+P%2E+Misra%2C+T%2E+Anderson%2C+and+A%2E+Sriraman%2C+%E2%80%9CTowards+Improved+Power+Management+in+Cloud+GPUs%2C%E2%80%9D+in+IEEE+CAL%2C+2023%2E)
65. P. Patel, K. Lim, K. Jhunjhunwalla, A. Martinez, M. Demoulin, J. Nelson, I. Zhang, and T. Anderson, “Hybrid Computing for Interactive Datacenter Applications,” arXiv preprint arXiv:2304.04488, 2023. — [arXiv:2304.04488](https://arxiv.org/abs/2304.04488) [find](https://scholar.google.com/scholar?q=P%2E+Patel%2C+K%2E+Lim%2C+K%2E+Jhunjhunwalla%2C+A%2E+Martinez%2C+M%2E+Demoulin%2C+J%2E+Nelson%2C+I%2E+Zhang%2C+and+T%2E+Anderson%2C+%E2%80%9CHybrid+Computing+for+Interactive+Datacenter+Applications%2C%E2%80%9D+arXiv+preprint+arXiv%3A2304%2E04488%2C+2023%2E+%E2%80%94)
66. R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean, “Efficiently Scaling Transformer Inference,” in MLSys, 2023. [find](https://scholar.google.com/scholar?q=R%2E+Pope%2C+S%2E+Douglas%2C+A%2E+Chowdhery%2C+J%2E+Devlin%2C+J%2E+Bradbury%2C+J%2E+Heek%2C+K%2E+Xiao%2C+S%2E+Agrawal%2C+and+J%2E+Dean%2C+%E2%80%9CEfficiently+Scaling+Transformer+Inference%2C%E2%80%9D+in+MLSys%2C+2023%2E)
67. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language Models are Unsupervised Multitask Learners,” OpenAI blog, 2019. [find](https://scholar.google.com/scholar?q=A%2E+Radford%2C+J%2E+Wu%2C+R%2E+Child%2C+D%2E+Luan%2C+D%2E+Amodei%2C+and+I%2E+Sutskever%2C+%E2%80%9CLanguage+Models+are+Unsupervised+Multitask+Learners%2C%E2%80%9D+OpenAI+blog%2C+2019%2E)
68. F. Romero, Q. Li, N. J. Yadwadkar, and C. Kozyrakis, “INFaaS: Automated Model-less Inference Serving,” in USENIX ATC, 2021. [find](https://scholar.google.com/scholar?q=F%2E+Romero%2C+Q%2E+Li%2C+N%2E+J%2E+Yadwadkar%2C+and+C%2E+Kozyrakis%2C+%E2%80%9CINFaaS%3A+Automated+Model%2Dless+Inference+Serving%2C%E2%80%9D+in+USENIX+ATC%2C+2021%2E)
69. T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguye… [find](https://scholar.google.com/scholar?q=T%2E+L%2E+Scao%2C+A%2E+Fan%2C+C%2E+Akiki%2C+E%2E+Pavlick%2C+S%2E+Ilic%CC%81%2C+D%2E+Hesslow%2C+R%2E+Castagn%C3%A9%2C+A%2E+S%2E+Luccioni%2C+F%2E+Yvon%2C+M%2E+Gall%C3%A9%2C+J%2E+Tow%2C+A%2E+M%2E+Rush%2C+S%2E+Biderman%2C+A%2E+Webson%2C+P%2E+S%2E+Ammanamanchi%2C+T%2E+Wang%2C+B%2E+Sagot%2C+N%2E+Mu)
70. P. Schmid. Fine-tune FLAN-T5 XL/XXL using DeepSpeed & Hugging Face Transformers. [Online]. Available: [https://www.philschmid.de/fine-tune-flan-t5-deepspeed](https://www.philschmid.de/fine-tune-flan-t5-deepspeed) [↗](https://www.philschmid.de/fine-tune-flan-t5-deepspeed)
71. P. Schmid, O. Sanseviero, P. Cuenca, and L. Tunstall. Llama 2 is here - Get it on Hugging Face. [Online]. Available: [https://huggingface.co/blog/llama2](https://huggingface.co/blog/llama2) [↗](https://huggingface.co/blog/llama2)
72. M. Schwarzkopf, A. Konwinski, M. Abd-El-Malek, and J. Wilkes, “Omega: Flexible, Scalable Schedulers for Large Compute Clusters,” in EuroSys, 2013. [find](https://scholar.google.com/scholar?q=M%2E+Schwarzkopf%2C+A%2E+Konwinski%2C+M%2E+Abd%2DEl%2DMalek%2C+and+J%2E+Wilkes%2C+%E2%80%9COmega%3A+Flexible%2C+Scalable+Schedulers+for+Large+Compute+Clusters%2C%E2%80%9D+in+EuroSys%2C+2013%2E)
73. Y. Sheng, S. Cao, D. Li, B. Zhu, Z. Li, D. Zhuo, J. E. Gonzalez, and I. Stoica, “Fairness in Serving Large Language Models,” arXiv preprint arXiv:2401.00588, 2023. [find](https://scholar.google.com/scholar?q=Y%2E+Sheng%2C+S%2E+Cao%2C+D%2E+Li%2C+B%2E+Zhu%2C+Z%2E+Li%2C+D%2E+Zhuo%2C+J%2E+E%2E+Gonzalez%2C+and+I%2E+Stoica%2C+%E2%80%9CFairness+in+Serving+Large+Language+Models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2401%2E00588%2C+2023%2E)
74. Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Ré, I. Stoica, and C. Zhang, “[FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://md.research.elamur.ai/free/en/papers/sheng-2023-flexgen-high-throughput-generative-inference/),” in ICML, 2023.
75. K. Shvachko, H. Kuang, S. Radia, and R. Chansler, “The Hadoop Distributed File System,” in MSST, 2010. [find](https://scholar.google.com/scholar?q=K%2E+Shvachko%2C+H%2E+Kuang%2C+S%2E+Radia%2C+and+R%2E+Chansler%2C+%E2%80%9CThe+Hadoop+Distributed+File+System%2C%E2%80%9D+in+MSST%2C+2010%2E)
76. K. Van Craeynest, A. Jaleel, L. Eeckhout, P. Narvaez, and J. Emer, “Scheduling Heterogeneous Multi-cores Through Performance Impact Estimation (PIE),” ACM SIGARCH Computer Architecture News, 2012. [find](https://scholar.google.com/scholar?q=K%2E+Van+Craeynest%2C+A%2E+Jaleel%2C+L%2E+Eeckhout%2C+P%2E+Narvaez%2C+and+J%2E+Emer%2C+%E2%80%9CScheduling+Heterogeneous+Multi%2Dcores+Through+Performance+Impact+Estimation+%28PIE%29%2C%E2%80%9D+ACM+SIGARCH+Computer+Architecture+News%2C+2012%2E)
77. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “[Attention is All You Need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/),” in NeurIPS, 2017.
78. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. v. Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush, “[Transformers: State-of-the-art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/),”…
79. B. Wu, Y. Zhong, Z. Zhang, G. Huang, X. Liu, and X. Jin, “[Fast Distributed Inference Serving for Large Language Models](https://md.research.elamur.ai/free/en/papers/wu-2023-fast-distributed-inference-serving-large/),” arXiv preprint arXiv:2305.05920, 2023.
80. H. Yang, Q. Chen, M. Riaz, Z. Luan, L. Tang, and J. Mars, “PowerChief: Intelligent Power Allocation for Multi-stage Applications to Improve Responsiveness on Power Constrained CMP,” in ISCA, 2017. [find](https://scholar.google.com/scholar?q=H%2E+Yang%2C+Q%2E+Chen%2C+M%2E+Riaz%2C+Z%2E+Luan%2C+L%2E+Tang%2C+and+J%2E+Mars%2C+%E2%80%9CPowerChief%3A+Intelligent+Power+Allocation+for+Multi%2Dstage+Applications+to+Improve+Responsiveness+on+Power+Constrained+CMP%2C%E2%80%9D+in+ISCA%2C+2017%2E)
81. G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “[Orca: A Distributed Serving System for Transformer-Based Generative Models](https://md.research.elamur.ai/free/en/papers/yu-2022-orca-distributed-serving-system-transformer/),” in OSDI, 2022.
82. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauly, M. J. Franklin, S. Shenker, and I. Stoica, “Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing,” in NSDI, 2012. [find](https://scholar.google.com/scholar?q=M%2E+Zaharia%2C+M%2E+Chowdhury%2C+T%2E+Das%2C+A%2E+Dave%2C+J%2E+Ma%2C+M%2E+McCauly%2C+M%2E+J%2E+Franklin%2C+S%2E+Shenker%2C+and+I%2E+Stoica%2C+%E2%80%9CResilient+Distributed+Datasets%3A+A+Fault%2DTolerant+Abstraction+for+In%2DMemory+Cluster+Computing%2C%E2%80%9D)
83. C. Zhang, M. Yu, W. Wang, and F. Yan, “MArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving,” in USENIX ATC, 2019. [find](https://scholar.google.com/scholar?q=C%2E+Zhang%2C+M%2E+Yu%2C+W%2E+Wang%2C+and+F%2E+Yan%2C+%E2%80%9CMArk%3A+Exploiting+Cloud+Services+for+Cost%2DEffective%2C+SLO%2DAware+Machine+Learning+Inference+Serving%2C%E2%80%9D+in+USENIX+ATC%2C+2019%2E)
84. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin et al., “[OPT: Open Pre-trained Transformer Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/),” arXiv preprint arXiv:2205.01068, 2022.
85. W. Zhu, “Analysis of JSQ Policy on Soft Real-time Scheduling in Cluster,” in HPCAsia, 2000. A PPENDIX A. Abstract We open source critical components needed to evaluate Splitwise; these could be repurposed to also evaluate future LLM inference serving systems. Our artifact includes: Production traces… [find](https://scholar.google.com/scholar?q=W%2E+Zhu%2C+%E2%80%9CAnalysis+of+JSQ+Policy+on+Soft+Real%2Dtime+Scheduling+in+Cluster%2C%E2%80%9D+in+HPCAsia%2C+2000%2E+A+PPENDIX+A%2E+Abstract+We+open+source+critical+components+needed+to+evaluate+Splitwise%3B+these+could+be+repurp)

*Updated 2 November 2026*