---
title: "SARATHI — paper by Agrawal et al. (2023) — summary and ideas"
description: "Decode-maximal batching builds each batch from one such piece and fills the remaining slots with… 46 references, cited by 128 papers here."
authors: ["Amey Agrawal", "Ashish Panwar", "Jayashree Mohan", "Nipun Kwatra", "Bhargav S. Gulavani", "Ramachandran Ramjee"]
year: 2023
venue: "arXiv 2023"
source: "https://arxiv.org/abs/2308.16369"
arxiv: "2308.16369"
licence: "arXiv — non-exclusive distribution license (arXiv OAI)"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/agrawal-2023-sarathi-efficient-llm-inference-piggybacking/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/agrawal-2023-sarathi-efficient-llm-inference-piggybacking/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills

# SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills

**Authors:** [Amey Agrawal](https://md.research.elamur.ai/free/en/authors/amey-agrawal/), [Ashish Panwar](https://md.research.elamur.ai/free/en/authors/ashish-panwar/), [Jayashree Mohan](https://md.research.elamur.ai/free/en/authors/jayashree-mohan/), [Nipun Kwatra](https://md.research.elamur.ai/free/en/authors/nipun-kwatra/), [Bhargav S. Gulavani](https://md.research.elamur.ai/free/en/authors/bhargav-s-gulavani/) and [Ramachandran Ramjee](https://md.research.elamur.ai/free/en/authors/ramachandran-ramjee/) · **Year:** 2023 · **Venue:** arXiv 2023 · **Licence:** [arXiv — non-exclusive distribution license (arXiv OAI)](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2308.16369) · [PDF](https://arxiv.org/pdf/2308.16369) · [arXiv 2308.16369](https://arxiv.org/abs/2308.16369) · [OpenAlex](https://openalex.org/W4386395487)

**Citations:** 16 citations (OpenAlex, 2026-10-14) · [128](https://md.research.elamur.ai/free/en/papers/agrawal-2023-sarathi-efficient-llm-inference-piggybacking/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Serving large language models is inefficient because generation has two very different stages. Processing the prompt (prefill) keeps a GPU fully busy even with a single request. Producing output one token at a time (decode) leaves most compute idle unless batches are large. Under pipeline parallelism, micro-batches that mix these stages take unequal time, which leaves stages idle (bubbles). SARATHI tackles both problems with two techniques. Chunked-prefills cuts each prompt into equal-sized pieces. Decode-maximal batching builds each batch from one such piece and fills the remaining slots with ongoing decodes. The prefill chunk saturates compute, so the decodes ride along at a small fraction of their usual cost. Because every hybrid batch needs roughly the same work, pipeline stages also stay balanced. The authors tested LLaMA-13B on A6000, LLaMA-33B on A100, and a simulated 64-GPU GPT-3 deployment. Decode throughput rose by up to 10×, and end-to-end throughput improved by up to 1.91× with pipeline parallelism. The approach is a scheduling change at the execution layer, so it can be combined with other serving optimizations.

## Key points
- At small batch sizes, the authors measured decode cost per token as high as roughly 200 times the prefill cost per token.
- LLaMA-13B on A6000 reached up to 10× higher decode throughput and up to 1.33× higher end-to-end throughput. LLaMA-33B on A100 reached up to 4.25× higher decode throughput and 1.25× higher end-to-end throughput.
- GPT-3 was run with 8-way pipeline and 8-way tensor parallelism on a simulated cluster of 64 A100 GPUs. Pipeline bubbles dropped by 6.29×, giving a 1.91× end-to-end speedup over Orca-style micro-batch scheduling.
- Chunk size is a trade-off:
  - Chunks of 64 tokens make attention about 3× slower and the whole prefill about 5× slower.
  - Chunks of 256 and 512 keep the prefill efficiency loss within about 20% and 10%, respectively.
  - Chunk sizes that are multiples of 128 perform better because of GPU tile quantization; for example, 256 beats 320.
- Decode savings can outweigh slower prefill. With 128-token chunks, prefill was more than 2× slower, yet overall throughput was up to 1.16× higher.

## Interesting ideas
- **Pair compute-bound and memory-bound work in one batch.** Instead of batching like with like, the scheduler combines a heavy operation with many light ones so the hardware stays saturated. This applies to any workload where small requests waste the hardware on their own.
- **Split large jobs to create more pairing opportunities.** Each request has only one prefill, so there would rarely be enough prompts to pair with all the decodes. Chunking one big task into many equal parts lets it host more small tasks over time. The same trick could help other batch schedulers with rare large items.
- **Use uniform work units to fix pipeline imbalance.** Pipeline bubbles are removed by making every micro-batch cost about the same, without changing the pipeline itself. This is a useful lesson for any staged system whose inputs vary widely in size.
- **Tune chunk size from the workload and the hardware.** The authors pick the chunk size from the application's average ratio of prompt to output tokens. They also choose sizes that align with the GPU's tile dimensions (multiples of 128). Matching batch shapes to hardware granularity is a cheap gain in many GPU pipelines.
- **Judge the trade-off end to end.** A setting that makes one stage markedly slower can still win overall, as with the slower prefill at chunk size 128. Evaluations should therefore report total throughput, not only the cost of individual components.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Georgia Institute of Technology](https://md.research.elamur.ai/free/en/universities/georgia-institute-of-technology/) — University, Atlanta, United States · 152 works · 27,763 citations
- [Microsoft Research](https://md.research.elamur.ai/free/en/companies/microsoft-research/) — Company, Bengaluru, India · 15 works · 4,115 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani and Ramachandran Ramjee. 2023. SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. arXiv 2023.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{agrawal2023sarathi,
  title = {{SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills}},
  author = {Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani and Ramachandran Ramjee},
  year = {2023},
  eprint = {2308.16369},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.16369}
}
```

- [← previous mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl2-revolutionizing-multi-modal/)
- [next → A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generation](https://md.research.elamur.ai/free/en/papers/varshney-2023-stitch-time-saves-nine-detecting/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

46 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amazon codewhisperer. [https://aws.amazon.com/codewhisperer/](https://aws.amazon.com/codewhisperer/). [↗](https://aws.amazon.com/codewhisperer/)
2. Anthropic claude. [https://claude.ai](https://claude.ai). [↗](https://claude.ai)
3. Bing ai. [https://www.bing.com/chat](https://www.bing.com/chat). [↗](https://www.bing.com/chat)
4. Character ai. [https://character.ai](https://character.ai). [↗](https://character.ai)
5. Chatgpt. [https://chat.openai.com](https://chat.openai.com). [↗](https://chat.openai.com)
6. Faster Transformer. [https://github.com/NVIDIA/FasterTransformer](https://github.com/NVIDIA/FasterTransformer). [↗](https://github.com/NVIDIA/FasterTransformer)
7. Github copilot. copilot. [https://github.com/features/](https://github.com/features/) [↗](https://github.com/features/)
8. Google bard. [https://bard.google.com](https://bard.google.com). [↗](https://bard.google.com)
9. Komo. [https://komo.ai/](https://komo.ai/). https://huggingface.co/ [↗](https://komo.ai/)
10. Matrix multiplication background user’s [https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html](https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html). guide. [↗](https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html)
11. nanogpt. [https://github.com/karpathy/nanoGPT](https://github.com/karpathy/nanoGPT). https: [↗](https://github.com/karpathy/nanoGPT)
12. Openai gpt-3: Understanding the architecture. [https://www.theaidream.com/post/openai-gpt-3-understanding-the-architecture](https://www.theaidream.com/post/openai-gpt-3-understanding-the-architecture). [↗](https://www.theaidream.com/post/openai-gpt-3-understanding-the-architecture)
13. Perplexity ai. [https://www.perplexity.ai/](https://www.perplexity.ai/). [↗](https://www.perplexity.ai/)
14. Replit ghostwriter. ghostwriter. [https://replit.com/site/](https://replit.com/site/) [↗](https://replit.com/site/)
15. Text generation inference. [https://huggingface.co/text-generation-inference](https://huggingface.co/text-generation-inference). [↗](https://huggingface.co/text-generation-inference)
16. The Secret Sauce behind 100K context window in LLMs: all tricks in one place. [https://blog.gopenai.com/how-to-speed-up-llms-and-use-100k-context-window-all-tricks-in-one-place-ffd40577b4c](https://blog.gopenai.com/how-to-speed-up-llms-and-use-100k-context-window-all-tricks-in-one-place-ffd40577b4c). [↗](https://blog.gopenai.com/how-to-speed-up-llms-and-use-100k-context-window-all-tricks-in-one-place-ffd40577b4c)
17. Using NVIDIA’s AI/ML Frameworks for Generative AI on VMware vSphere. [https://core.vmware.com/blog/using-nvidias-aiml-frameworks-generative-ai-vmware-vsphere](https://core.vmware.com/blog/using-nvidias-aiml-frameworks-generative-ai-vmware-vsphere). [↗](https://core.vmware.com/blog/using-nvidias-aiml-frameworks-generative-ai-vmware-vsphere)
18. vllm: Easy, fast, and cheap llm serving for everyone. [https://github.com/vllm-project/vllm](https://github.com/vllm-project/vllm). [↗](https://github.com/vllm-project/vllm)
19. XFORMERS OPTIMIZED OPERATORS. [https://facebookresearch.github.io/xformers/components/ops.html](https://facebookresearch.github.io/xformers/components/ops.html). [↗](https://facebookresearch.github.io/xformers/components/ops.html)
20. You.com. [https://you.com/](https://you.com/). [↗](https://you.com/)
21. Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giri Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O’Horo, Jeff Wang, Luke Zettle… — [Efficient Large Scale Language Modeling with Mixtures of Experts](https://md.research.elamur.ai/free/en/papers/artetxe-2021-efficient-large-scale-language-modeling/)
22. Sanjith Athlur, Nitika Saran, Muthian Sivathanu, Ramachandran Ramjee, and Nipun Kwatra. Varuna: scalable, low-cost training of massive deep learning models. In Proceedings of the Seventeenth European Conference on Computer Systems, pages 472–487, 2022. [find](https://scholar.google.com/scholar?q=Sanjith+Athlur%2C+Nitika+Saran%2C+Muthian+Sivathanu%2C+Ramachandran+Ramjee%2C+and+Nipun+Kwatra%2E+Varuna%3A+scalable%2C+low%2Dcost+training+of+massive+deep+learning+models%2E+In+Proceedings+of+the+Seventeenth+European+)
23. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
24. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
25. Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J Franklin, Joseph E Gonzalez, and Ion Stoica. Clipper: A {Low-Latency} online prediction serving system. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17), pages 613–627, 2017. [find](https://scholar.google.com/scholar?q=Daniel+Crankshaw%2C+Xin+Wang%2C+Guilio+Zhou%2C+Michael+J+Franklin%2C+Joseph+E+Gonzalez%2C+and+Ion+Stoica%2E+Clipper%3A+A+%7BLow%2DLatency%7D+online+prediction+serving+system%2E+In+14th+USENIX+Symposium+on+Networked+Systems)
26. Tri Dao. [Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/), 2023.
27. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Daniel+Y%2E+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2C+2022%2E)
28. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8(): 8-bit matrix multiplication for transformers at scale, 2022. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Mike+Lewis%2C+Younes+Belkada%2C+and+Luke+Zettlemoyer%2E+Llm%2Eint8%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2C+2022%2E)
29. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. [Qlora: Efficient finetuning of quantized llms](https://md.research.elamur.ai/free/en/papers/dettmers-2023-qlora-efficient-finetuning-quantized-llms/), 2023.
30. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023. [find](https://scholar.google.com/scholar?q=Elias+Frantar%2C+Saleh+Ashkboos%2C+Torsten+Hoefler%2C+and+Dan+Alistarh%2E+Gptq%3A+Accurate+post%2Dtraining+quantization+for+generative+pre%2Dtrained+transformers%2C+2023%2E)
31. Haiyang Huang, Newsha Ardalani, Anna Sun, Liu Ke, Hsien-Hsin S. Lee, Anjali Sridhar, Shruti Bhosale, Carole-Jean Wu, and Benjamin Lee. Towards moe deployment: Mitigating inefficiencies in mixture-of-expert (moe) inference, 2023. [find](https://scholar.google.com/scholar?q=Haiyang+Huang%2C+Newsha+Ardalani%2C+Anna+Sun%2C+Liu+Ke%2C+Hsien%2DHsin+S%2E+Lee%2C+Anjali+Sridhar%2C+Shruti+Bhosale%2C+Carole%2DJean+Wu%2C+and+Benjamin+Lee%2E+Towards+moe+deployment%3A+Mitigating+inefficiencies+in+mixture%2Dof%2De)
32. Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019. [find](https://scholar.google.com/scholar?q=Yanping+Huang%2C+Youlong+Cheng%2C+Ankur+Bapna%2C+Orhan+Firat%2C+Dehao+Chen%2C+Mia+Chen%2C+HyoukJoong+Lee%2C+Jiquan+Ngiam%2C+Quoc+V+Le%2C+Yonghui+Wu%2C+et+al%2E+Gpipe%3A+Efficient+training+of+giant+neural+networks+using+pipel)
33. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). CoRR, abs/2001.08361, 2020.
34. Jiamin Li, Yimin Jiang, Yibo Zhu, Cong Wang, and Hong Xu. [Accelerating distributed MoE training and inference with lina](https://md.research.elamur.ai/free/en/papers/li-2022-accelerating-distributed-moe-training-inference/). In 2023 USENIX Annual Technical Conference (USENIX ATC 23), pages 945–959, Boston, MA, July 2023. USENIX Association.
35. Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pages 1–15, 2019. [find](https://scholar.google.com/scholar?q=Deepak+Narayanan%2C+Aaron+Harlap%2C+Amar+Phanishayee%2C+Vivek+Seshadri%2C+Nikhil+R+Devanur%2C+Gregory+R+Ganger%2C+Phillip+B+Gibbons%2C+and+Matei+Zaharia%2E+Pipedream%3A+generalized+pipeline+parallelism+for+dnn+training)
36. OpenAI. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). abs/2303.08774, 2023. CoRR,
37. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference, 2022. [find](https://scholar.google.com/scholar?q=Reiner+Pope%2C+Sholto+Douglas%2C+Aakanksha+Chowdhery%2C+Jacob+Devlin%2C+James+Bradbury%2C+Anselm+Levskaya%2C+Jonathan+Heek%2C+Kefan+Xiao%2C+Shivani+Agrawal%2C+and+Jeff+Dean%2E+Efficiently+scaling+transformer+inference%2C+2)
38. Markus N. Rabe and Charles Staats. Self-attention does not need o(n 2 ) memory, 2022. [find](https://scholar.google.com/scholar?q=Markus+N%2E+Rabe+and+Charles+Staats%2E+Self%2Dattention+does+not+need+o%28n+2+%29+memory%2C+2022%2E)
39. Noam Shazeer. Fast transformer decoding: One write-head is all you need, 2019. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+Fast+transformer+decoding%3A+One+write%2Dhead+is+all+you+need%2C+2019%2E)
40. Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. [Flexgen: High-throughput generative inference of large language models with a single gpu](https://md.research.elamur.ai/free/en/papers/sheng-2023-flexgen-high-throughput-generative-inference/), 2023.
41. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+gpu+model+parallelism%2E+arXiv+pre)
42. Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. [Retentive network: A successor to transformer for large language models](https://md.research.elamur.ai/free/en/papers/sun-2023-retentive-network-successor-transformer-large/), 2023.
43. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). Trans. Mach. Learn. Res.…
44. Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. [Fast distributed inference serving for large language models](https://md.research.elamur.ai/free/en/papers/wu-2023-fast-distributed-inference-serving-large/), 2023.
45. Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models, 2023. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Ji+Lin%2C+Mickael+Seznec%2C+Hao+Wu%2C+Julien+Demouth%2C+and+Song+Han%2E+Smoothquant%3A+Accurate+and+efficient+post%2Dtraining+quantization+for+large+language+models%2C+2023%2E)
46. Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. [Orca: A distributed serving system for Transformer-Based generative models](https://md.research.elamur.ai/free/en/papers/yu-2022-orca-distributed-serving-system-transformer/). In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, Carlsbad, CA, July 2022. USENIX Association.

*Updated 2 November 2026*