--- title: "NanoFlow — paper by Zhu et al. (2024) · OSDI 2025" description: "This paper argues, both analytically and with measurements, that for typical workloads and models the… 4 references, cited by 79 papers here." authors: ["Kan Zhu", "Yufei Gao", "Yilong Zhao", "Liangyu Zhao", "Gefei Zuo", "Yile Gu", "Dedong Xie", "Tian Tang", "Qinyu Xu", "Zihao Ye"] year: 2024 venue: "OSDI 2025" source: "https://arxiv.org/abs/2408.12757" arxiv: "2408.12757" licence: "CC BY 4.0 (arXiv OAI)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/zhu-2024-nanoflow-optimal-large-language-model/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhu-2024-nanoflow-optimal-large-language-model/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / NanoFlow: Towards Optimal Large Language Model Serving Throughput # NanoFlow: Towards Optimal Large Language Model Serving Throughput **Authors:** [Kan Zhu](https://md.research.elamur.ai/free/en/authors/kan-zhu/), Yufei Gao, [Yilong Zhao](https://md.research.elamur.ai/free/en/authors/yilong-zhao/), Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Tian Tang, Qinyu Xu and [Zihao Ye](https://md.research.elamur.ai/free/en/authors/zihao-ye/) · **Year:** 2024 · **Venue:** OSDI 2025 · **Licence:** [CC BY 4.0 (arXiv OAI)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2408.12757) · [PDF](https://arxiv.org/pdf/2408.12757) · [arXiv 2408.12757](https://arxiv.org/abs/2408.12757) · [OpenAlex](https://openalex.org/W4402698921) **Citations:** 1 citations (OpenAlex, 2026-10-14) · [79](https://md.research.elamur.ai/free/en/papers/zhu-2024-nanoflow-optimal-large-language-model/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary LLM serving is usually assumed to be limited by memory bandwidth, because every generated token requires reading the model weights and the KV-cache. This paper argues, both analytically and with measurements, that for typical workloads and models the serving pipeline as a whole is compute-bound. Three factors drive this: batching prefill with decode spreads the cost of weight reads, GQA reduces KV-cache traffic, and GEMMs increasingly dominate as models grow. The real inefficiency is that each GPU runs compute-heavy, memory-heavy and network-heavy kernels one after another. NanoFlow addresses this with intra-device parallelism. It cuts each batch into nano-batches, gives every operation one copy per nano-batch, and overlaps these copies so different GPU resources are busy at the same time. Each kernel receives only a slice of the GPU. A two-stage auto-search chooses the number, sizes, order and resource split of nano-batches. The first stage ignores interference between kernels, and the second refines the plan using profiled interference. On LLaMA-2-70B with 8×A100, NanoFlow delivers on average 1.91× the throughput of the strongest existing engines. Since cost scales with tokens per GPU, this matters directly for large deployments. ## Key points - Evaluation setup: - Main model: LLaMA-2-70B on one 8×A100 80GB DGX node. - Workload traces: ShareGPT, LMSys (LMSYS-Chat-1M) and Splitwise. - Baselines: vLLM, DeepSpeed-FastGen and TensorRT-LLM. - Gap that motivates the work: in existing engines, each individual operation uses its bottleneck resource at roughly 80%, yet overall compute utilization is only about 40%. - Main results: - NanoFlow reaches 68.5% of the theoretical optimal throughput, with 1.91× average gain over the baselines. - At low load its latency is close to TensorRT-LLM's. - It sustains up to 1.64× higher request rates under a 200 ms/token SLO. - Generalization to other models: - Tested on LLaMA-3-70B, LLaMA-3-8B, QWen2-72B, Deepseek-67B and Mixtral 8×7B (MoE). - Generated pipelines reach 50–72% of optimal throughput, an average 2.66× over vLLM. - Ablation findings: - Nano-batching alone costs 13.2% throughput. - Overlapping network-bound kernels yields 1.07× over the non-overlapped baseline, and overlapping both network- and memory-bound kernels yields 1.17×. - KV-cache offloading slows the pipeline by 3.0%, but cuts compute by 3.02× on multi-round LMSYS-Chat. ## Interesting ideas - **Bottleneck analysis of the whole system, not each operator.** Judge a system by the bottleneck of the entire workload rather than by how well each operator saturates its own resource. Any pipeline can show high per-step utilization while the whole system stays far from its limit. - **Overlapping independent copies of work.** Splitting a batch into smaller independent pieces and duplicating operations over them creates overlap between steps that would otherwise depend on each other. The extra overhead is acceptable whenever it can hide under the dominant bottleneck. - **Two-stage planning.** First plan with an idealized, interference-free model, then correct the plan using measured interference. This makes a huge configuration space tractable while staying grounded in real hardware behavior. - **Partial resource allocation per kernel.** Give the heavy compute kernel most of the GPU and a memory-bound kernel only the remainder. A lower-priority task can still make good progress on a small resource share, which applies to scheduling mixed jobs in general. - **Reporting against a derived optimum.** Express results as a percentage of a theoretically derived optimal throughput, not only as speedups over baselines. This shows how much headroom is left and is a useful evaluation habit for any systems or NLP-infrastructure project. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 402 works · 212,763 citations - [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 209 works · 244,782 citations - [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations - [University of Michigan](https://md.research.elamur.ai/free/en/universities/university-of-michigan/) — University, Ann Arbor, United States · 86 works · 15,493 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works 🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works ## Cite Cite as: Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Tian Tang, Qinyu Xu and Zihao Ye. 2024. NanoFlow: Towards Optimal Large Language Model Serving Throughput. OSDI 2025. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{zhu2024nanoflow, title = {{NanoFlow: Towards Optimal Large Language Model Serving Throughput}}, author = {Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Tian Tang, Qinyu Xu and Zihao Ye}, booktitle = {OSDI 2025}, year = {2024}, eprint = {2408.12757}, archivePrefix = {arXiv}, url = {https://arxiv.org/abs/2408.12757} } ``` - [← previous Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention](https://md.research.elamur.ai/free/en/papers/munkhdalai-2024-leave-no-context-behind-efficient/) - [next → Paths-over-Graph: Knowledge Graph Empowered Large Language Model Reasoning](https://md.research.elamur.ai/free/en/papers/tan-2024-paths-over-graph-knowledge-graph/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) ## References 4 references, 2 of them with a card of ours. ↗ leads to the work. 1. Sharegpt. [https://huggingface.co/datasets/](https://huggingface.co/datasets/) anon8231489123/ShareGPT_Vicuna_unfiltered, 2023. [↗](https://huggingface.co/datasets/) 2. Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, Alexey Tumanov, and Ramachandran Ramjee. [Taming throughput-latency tradeoff in llm inference with sarathi-serve](https://md.research.elamur.ai/free/en/papers/agrawal-2024-taming-throughput-latency-tradeoff-llm/). arXiv preprint arXiv:2403.02310, 2024. *Figure 11: Performance of NanoFlow instances in terms… 3. Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, and Ramachandran Ramjee. [Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills](https://md.research.elamur.ai/free/en/papers/agrawal-2023-sarathi-efficient-llm-inference-piggybacking/). arXiv preprint arXiv:2308.16369, 2023. works require recompilation from scratch, which requires significant manual… 4. Mistral AI. Mistral nemo. [https://mistral.ai/](https://mistral.ai/) news/mistral-nemo/, 2024. [↗](https://mistral.ai/) *Updated 2 November 2026*