---
title: "S-LoRA: Serving Thousands of Concurrent LoRA Adapters (2023)"
description: "Organizations increasingly fine-tune one base LLM into many task-specific LoRA variants, but serving… 64 references, cited by 73 papers here."
authors: ["Ying Sheng", "Shiyi Cao", "Dacheng Li", "Coleman Hooper", "Nicholas Lee", "Shuo Yang", "Christopher Chou", "Banghua Zhu", "Lianmin Zheng", "Kurt Keutzer"]
year: 2023
venue: "MLSys 2024"
source: "https://arxiv.org/abs/2311.03285"
arxiv: "2311.03285"
licence: "arXiv — non-exclusive distribution license (arXiv OAI)"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sheng-2023-s-lora-serving-thousands-concurrent/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sheng-2023-s-lora-serving-thousands-concurrent/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/) / S-LoRA: Serving Thousands of Concurrent LoRA Adapters

# S-LoRA: Serving Thousands of Concurrent LoRA Adapters

**Authors:** [Ying Sheng](https://md.research.elamur.ai/free/en/authors/ying-sheng/), [Shiyi Cao](https://md.research.elamur.ai/free/en/authors/shiyi-cao/), [Dacheng Li](https://md.research.elamur.ai/free/en/authors/dacheng-li/), [Coleman Hooper](https://md.research.elamur.ai/free/en/authors/coleman-hooper/), Nicholas Lee, [Shuo Yang](https://md.research.elamur.ai/free/en/authors/shuo-yang/), Christopher Chou, [Banghua Zhu](https://md.research.elamur.ai/free/en/authors/banghua-zhu/), [Lianmin Zheng](https://md.research.elamur.ai/free/en/authors/lianmin-zheng/) and [Kurt Keutzer](https://md.research.elamur.ai/free/en/authors/kurt-keutzer/) · **Year:** 2023 · **Venue:** MLSys 2024 · **Licence:** [arXiv — non-exclusive distribution license (arXiv OAI)](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2311.03285) · [PDF](https://arxiv.org/pdf/2311.03285) · [arXiv 2311.03285](https://arxiv.org/abs/2311.03285) · [OpenAlex](https://openalex.org/W4388514131)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [73](https://md.research.elamur.ai/free/en/papers/sheng-2023-s-lora-serving-thousands-concurrent/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Organizations increasingly fine-tune one base LLM into many task-specific LoRA variants, but serving them all at once is poorly supported. Merging each adapter into the base weights or swapping adapters in and out works for one model at a time, but it hurts throughput when many adapters receive traffic simultaneously. S-LoRA is a serving system that treats the shared base model as one large batched computation and handles each request's low-rank adapter product separately. All adapters stay in host RAM, and only those needed by the current batch are copied to the GPU. A single paged memory pool, called Unified Paging, holds both adapter weights of different ranks and KV cache blocks, which limits fragmentation. Custom CUDA kernels batch LoRA math across mixed ranks and scattered memory, and a dedicated tensor-parallel scheme extends the system to multiple GPUs. Tested on Llama-7B/13B/30B/70B, S-LoRA serves thousands of adapters with small overhead. It beats HuggingFace PEFT and a naive LoRA setup in vLLM on throughput and on the number of adapters it can host. This makes large-scale personalized fine-tuning services practical.

## Key points
- Throughput gains reach up to 30× over HuggingFace PEFT and up to 4× over vLLM with naive LoRA support. The number of adapters that can be hosted grows by several orders of magnitude.
- Unified Paging extends vLLM's PagedAttention so that one pool stores both KV cache pages and dynamically loaded adapter weights.
- The heterogeneous batching kernels handle requests with different sequence lengths and adapter ranks. They read directly from non-contiguous memory, while the base model part runs as an ordinary GEMM.
- S-LoRA TP keeps LoRA communication cheap. It sends only small intermediate tensors separately and merges the larger transfers into the base model's existing collective operations.
- Scheduling adds two components. Adapter clustering limits how many distinct adapters share a batch, which had a modest effect in experiments. An SLO-aware admission controller drops requests that would miss the first-token deadline anyway, and it serves the newest requests first when the system is overloaded.

## Interesting ideas
- **Split shared work from per-tenant work.** Batch the common backbone for all requests and run the small customized parts with their own kernels. This pattern fits any multi-tenant setup built on one base model, such as many domain-specific Ukrainian NLP adapters over a single LLM.
- **One memory pool for different kinds of dynamic data.** Paging adapter weights and KV caches together lets the two compete flexibly for GPU memory and avoids fragmentation. The same idea could apply to other caches whose sizes change over time.
- **Treat host RAM as the adapter store.** Keeping the full adapter catalog in main memory and loading adapters into the GPU on demand removes the GPU as the limit on how many models can be offered.
- **Admission control based on user satisfaction.** The paper models user satisfaction as a decreasing reward over latency. It then proves that, under suitable conditions, serving the most recent requests in order is optimal. In practice, this means aborting hopeless requests and switching between newest-first and oldest-first depending on load, which is a cheap and principled policy for any latency-sensitive queue.
- **Hide new communication inside existing communication.** When extending a parallel system, the extra large transfers can be fused into collectives the system already performs, so the extension adds little overhead.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works

## Cite

Cite as: Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng and Kurt Keutzer. 2023. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. MLSys 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sheng2023s,
  title = {{S-LoRA: Serving Thousands of Concurrent LoRA Adapters}},
  author = {Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng and Kurt Keutzer},
  booktitle = {MLSys 2024},
  year = {2023},
  eprint = {2311.03285},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.03285}
}
```

- [← previous Mitigating Label Biases for In-context Learning](https://md.research.elamur.ai/free/en/papers/fei-2023-mitigating-label-biases-context-learning/)
- [next → Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts](https://md.research.elamur.ai/free/en/papers/tulchinskii-2023-intrinsic-dimension-estimation-robust-detection/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/)

## References

64 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. [Flamingo: a visual language model for few-shot learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/). Advances in Neural Information Processing Systems, 35:23716–23736, 2022.
2. Aminabadi, R. Y., Rajbhandari, S., Awan, A. A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., and He, Y. Deepspeed- inference: Enabling efficient inference of transformer models at unprecedented scale. In Wolf, F., Shende, S., Culhane, C., Alam, S. R., and Jagode, H. (eds.… [find](https://scholar.google.com/scholar?q=Aminabadi%2C+R%2E+Y%2E%2C+Rajbhandari%2C+S%2E%2C+Awan%2C+A%2E+A%2E%2C+Li%2C+C%2E%2C+Li%2C+D%2E%2C+Zheng%2C+E%2E%2C+Ruwase%2C+O%2E%2C+Smith%2C+S%2E%2C+Zhang%2C+M%2E%2C+Rasley%2C+J%2E%2C+and+He%2C+Y%2E+Deepspeed%2D+inference%3A+Enabling+efficient+inference+of+transformer+mo)
3. Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403, 2023.
4. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33: 1877–1901, 2020.
5. Chen, L. Potentials of multitenancy fine-tuned llm serving. [https://le.qun.ch/en/blog/2023/09/11/multi-lora-potentials/](https://le.qun.ch/en/blog/2023/09/11/multi-lora-potentials/), 2023. [↗](https://le.qun.ch/en/blog/2023/09/11/multi-lora-potentials/)
6. Chen, L., Ye, Z., Wu, Y., Zhuo, D., Ceze, L., and Krishnamurthy, A. [Punica: Multi-tenant lora serving](https://md.research.elamur.ai/free/en/papers/chen-2023-punica-multi-tenant-lora-serving/), 2023.
7. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311, 2022.
8. Crankshaw, D., Wang, X., Zhou, G., Franklin, M. J., Gonzalez, J. E., and Stoica, I. Clipper: A low-latency online prediction serving system. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17), pp. 613–627, 2017. [find](https://scholar.google.com/scholar?q=Crankshaw%2C+D%2E%2C+Wang%2C+X%2E%2C+Zhou%2C+G%2E%2C+Franklin%2C+M%2E+J%2E%2C+Gonzalez%2C+J%2E+E%2E%2C+and+Stoica%2C+I%2E+Clipper%3A+A+low%2Dlatency+online+prediction+serving+system%2E+In+14th+USENIX+Symposium+on+Networked+Systems+Design+and+Im)
9. Crankshaw, D., Sela, G.-E., Mo, X., Zumar, C., Stoica, I., Gonzalez, J., and Tumanov, A. Inferline: latency-aware provisioning and scaling for prediction serving pipelines. In Proceedings of the 11th ACM Symposium on Cloud Computing, pp. 477–491, 2020. [find](https://scholar.google.com/scholar?q=Crankshaw%2C+D%2E%2C+Sela%2C+G%2E%2DE%2E%2C+Mo%2C+X%2E%2C+Zumar%2C+C%2E%2C+Stoica%2C+I%2E%2C+Gonzalez%2C+J%2E%2C+and+Tumanov%2C+A%2E+Inferline%3A+latency%2Daware+provisioning+and+scaling+for+prediction+serving+pipelines%2E+In+Proceedings+of+the+11th+)
10. Cui, W., Zhao, H., Chen, Q., Wei, H., Li, Z., Zeng, D., Li, C., and Guo, M. Dvabatch: Diversity-aware multi-entry multi-exit batching for efficient processing of dnn services on gpus. In 2022 USENIX Annual Technical Conference (USENIX ATC 22), pp. 183–198, 2022. [find](https://scholar.google.com/scholar?q=Cui%2C+W%2E%2C+Zhao%2C+H%2E%2C+Chen%2C+Q%2E%2C+Wei%2C+H%2E%2C+Li%2C+Z%2E%2C+Zeng%2C+D%2E%2C+Li%2C+C%2E%2C+and+Guo%2C+M%2E+Dvabatch%3A+Diversity%2Daware+multi%2Dentry+multi%2Dexit+batching+for+efficient+processing+of+dnn+services+on+gpus%2E+In+2022+USENIX+A)
11. Dao, T. [Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). arXiv preprint arXiv:2307.08691, 2023.
12. Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Llm.int8(): 8-bit matrix multiplication for transformers at scale. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Dettmers%2C+T%2E%2C+Lewis%2C+M%2E%2C+Belkada%2C+Y%2E%2C+and+Zettlemoyer%2C+L%2E+Llm%2Eint8%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2E+In+Oh%2C+A%2E+H%2E%2C+Agarwal%2C+A%2E%2C+Belgrave%2C+D%2E%2C+and+Cho%2C+K%2E+%28eds%2E%29%2C+Advances+in+Ne)
13. Fang, J., Yu, Y., Zhao, C., and Zhou, J. Turbotransformers: an efficient gpu serving system for transformer models. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pp. 389–402, 2021. [find](https://scholar.google.com/scholar?q=Fang%2C+J%2E%2C+Yu%2C+Y%2E%2C+Zhao%2C+C%2E%2C+and+Zhou%2C+J%2E+Turbotransformers%3A+an+efficient+gpu+serving+system+for+transformer+models%2E+In+Proceedings+of+the+26th+ACM+SIGPLAN+Symposium+on+Principles+and+Practice+of+Paral)
14. Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1):5232–5270, 2022. [find](https://scholar.google.com/scholar?q=Fedus%2C+W%2E%2C+Zoph%2C+B%2E%2C+and+Shazeer%2C+N%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+The+Journal+of+Machine+Learning+Research%2C+23%281%29%3A5232%E2%80%935270%2C+2022%2E)
15. Frantar, E. and Alistarh, D. Massive language models can be accurately pruned in one-shot. arXiv preprint arXiv:2301.00774, 2023. [find](https://scholar.google.com/scholar?q=Frantar%2C+E%2E+and+Alistarh%2C+D%2E+Massive+language+models+can+be+accurately+pruned+in+one%2Dshot%2E+arXiv+preprint+arXiv%3A2301%2E00774%2C+2023%2E)
16. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022. [find](https://scholar.google.com/scholar?q=Frantar%2C+E%2E%2C+Ashkboos%2C+S%2E%2C+Hoefler%2C+T%2E%2C+and+Alistarh%2C+D%2E+Gptq%3A+Accurate+post%2Dtraining+quantization+for+generative+pre%2Dtrained+transformers%2E+arXiv+preprint+arXiv%3A2210%2E17323%2C+2022%2E)
17. Gujarati, A., Karimi, R., Alzayat, S., Hao, W., Kaufmann, A., Vigfusson, Y., and Mace, J. Serving {DNNs} like clockwork: Performance predictability from the bottom up. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pp. 443–462, 2020. [find](https://scholar.google.com/scholar?q=Gujarati%2C+A%2E%2C+Karimi%2C+R%2E%2C+Alzayat%2C+S%2E%2C+Hao%2C+W%2E%2C+Kaufmann%2C+A%2E%2C+Vigfusson%2C+Y%2E%2C+and+Mace%2C+J%2E+Serving+%7BDNNs%7D+like+clockwork%3A+Performance+predictability+from+the+bottom+up%2E+In+14th+USENIX+Symposium+on+Oper)
18. Han, M., Zhang, H., Chen, R., and Chen, H. Microsecond-scale preemption for concurrent {GPU-accelerated}{DNN} inferences. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 539–558, 2022. [find](https://scholar.google.com/scholar?q=Han%2C+M%2E%2C+Zhang%2C+H%2E%2C+Chen%2C+R%2E%2C+and+Chen%2C+H%2E+Microsecond%2Dscale+preemption+for+concurrent+%7BGPU%2Daccelerated%7D%7BDNN%7D+inferences%2E+In+16th+USENIX+Symposium+on+Operating+Systems+Design+and+Implementation+%28OSDI+)
19. Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790–2799. PMLR, 2019. [find](https://scholar.google.com/scholar?q=Houlsby%2C+N%2E%2C+Giurgiu%2C+A%2E%2C+Jastrzebski%2C+S%2E%2C+Morrone%2C+B%2E%2C+De+Laroussilhe%2C+Q%2E%2C+Gesmundo%2C+A%2E%2C+Attariyan%2C+M%2E%2C+and+Gelly%2C+S%2E+Parameter%2Defficient+transfer+learning+for+nlp%2E+In+International+Conference+on+Mac)
20. Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations, 2021.
21. Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019. [find](https://scholar.google.com/scholar?q=Huang%2C+Y%2E%2C+Cheng%2C+Y%2E%2C+Bapna%2C+A%2E%2C+Firat%2C+O%2E%2C+Chen%2C+D%2E%2C+Chen%2C+M%2E%2C+Lee%2C+H%2E%2C+Ngiam%2C+J%2E%2C+Le%2C+Q%2E+V%2E%2C+Wu%2C+Y%2E%2C+et+al%2E+Gpipe%3A+Efficient+training+of+giant+neural+networks+using+pipeline+parallelism%2E+Advances+in)
22. Jamin, S., Shenker, S., Zhang, L., and Clark, D. D. An admission control algorithm for predictive real-time service. In Network and Operating System Support for Digital Audio and Video: Third International Workshop La Jolla, California, USA, November 12–13, 1992 Proceedings 3, pp. 347–356. [find](https://scholar.google.com/scholar?q=Jamin%2C+S%2E%2C+Shenker%2C+S%2E%2C+Zhang%2C+L%2E%2C+and+Clark%2C+D%2E+D%2E+An+admission+control+algorithm+for+predictive+real%2Dtime+service%2E+In+Network+and+Operating+System+Support+for+Digital+Audio+and+Video%3A+Third+Internat)
23. Kenton, J. D. M.-W. C. and Toutanova, L. K. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of NAACL-HLT, pp. 4171– 4186, 2019.
24. Korthikanti, V. A., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B. [Reducing activation recomputation in large transformer models](https://md.research.elamur.ai/free/en/papers/korthikanti-2022-reducing-activation-recomputation-large-transformer/). Proceedings of Machine Learning and Systems, 5, 2023.
25. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Flinn, J., Seltzer, M. I., Druschel, P., Kaufmann, A., and Mace, J. (eds.), Proceedings of the 29th Symposium on…
26. Lester, B., Al-Rfou, R., and Constant, N. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045–3059, 2021.
27. Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582–4597, 2021. [find](https://scholar.google.com/scholar?q=Li%2C+X%2E+L%2E+and+Liang%2C+P%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11th+Internation)
28. Li, Z., Zheng, L., Zhong, Y., Liu, V., Sheng, Y., Jin, X., Huang, Y., Chen, Z., Zhang, H., Gonzalez, J. E., et al. {AlpaServe}: Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23), pp. 663–679, 2… — [AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving](https://md.research.elamur.ai/free/en/papers/li-2023-alpaserve-statistical-multiplexing-model-parallelism/)
29. Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., and Han, S. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023. [find](https://scholar.google.com/scholar?q=Lin%2C+J%2E%2C+Tang%2C+J%2E%2C+Tang%2C+H%2E%2C+Yang%2C+S%2E%2C+Dang%2C+X%2E%2C+and+Han%2C+S%2E+Awq%3A+Activation%2Daware+weight+quantization+for+llm+compression+and+acceleration%2E+arXiv+preprint+arXiv%3A2306%2E00978%2C+2023%2E)
30. Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. A. [Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning](https://md.research.elamur.ai/free/en/papers/liu-2022-few-shot-parameter-efficient-fine/). Advances in Neural Information Processing Systems, 35: 1950–1965, 2022.
31. Liu, X., Ji, K., Fu, Y., Tam, W. L., Du, Z., Yang, Z., and Tang, J. [P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks](https://md.research.elamur.ai/free/en/papers/liu-2021-p-tuning-v2-prompt-tuning/). arXiv preprint arXiv:2110.07602, 2021.
32. Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., and Tang, J. [Gpt understands, too](https://md.research.elamur.ai/free/en/papers/liu-2021-gpt-understands-too/). AI Open, 2023. ISSN 2666-6510. doi: https://doi.org/10.1016/j.aiopen.2023.08.012. URL https://www.sciencedirect.com/science/article/pii/S2666651023000141. [↗](https://doi.org/10.1016/j.aiopen.2023.08.012)
33. Mangrulkar, S., Gugger, S., Debut, L., Belkada, Y., Paul, S., and Bossan, B. Peft: State-of-the-art parameter-efficient fine-tuning methods. [https://github.com/huggingface/peft](https://github.com/huggingface/peft), 2022. [↗](https://github.com/huggingface/peft)
34. Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Wong, R. Y. Y., Chen, Z., Arfeen, D., Abhyankar, R., and Jia, Z. Specinfer: Accelerating generative llm serving with speculative inference and token tree verification. arXiv preprint arXiv:2305.[09781, 2023](https://md.research.elamur.ai/free/en/papers/miao-2023-specinfer-accelerating-generative-large-language/).
35. ModelTC. Lightllm: Python-based llm inference and serving framework. [https://github.com/ModelTC/lightllm](https://github.com/ModelTC/lightllm), 2023. GitHub repository. [↗](https://github.com/ModelTC/lightllm)
36. Naghshineh, M. and Schwartz, M. Distributed call admission control in mobile/wireless networks. IEEE Journal on Selected Areas in Communications, 14(4):711–717, 1996. [find](https://scholar.google.com/scholar?q=Naghshineh%2C+M%2E+and+Schwartz%2C+M%2E+Distributed+call+admission+control+in+mobile%2Fwireless+networks%2E+IEEE+Journal+on+Selected+Areas+in+Communications%2C+14%284%29%3A711%E2%80%93717%2C+1996%2E)
37. Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performanc… [find](https://scholar.google.com/scholar?q=Narayanan%2C+D%2E%2C+Shoeybi%2C+M%2E%2C+Casper%2C+J%2E%2C+LeGresley%2C+P%2E%2C+Patwary%2C+M%2E%2C+Korthikanti%2C+V%2E%2C+Vainbrand%2C+D%2E%2C+Kashinkunti%2C+P%2E%2C+Bernauer%2C+J%2E%2C+Catanzaro%2C+B%2E%2C+et+al%2E+Efficient+large%2Dscale+language+model+training+o)
38. NVIDIA. Cutlass gemm grouped. [https://github.com/NVIDIA/cutlass/blob/](https://github.com/NVIDIA/cutlass/blob/) main/examples/24_gemm_grouped/gemm_ grouped.cu. NVIDIA. Fastertransformer. https://github.com/NVIDIA/FasterTransformer, 2023. [↗](https://github.com/NVIDIA/cutlass/blob/)
39. Olston, C., Fiedel, N., Gorovoy, K., Harmsen, J., Lao, L., Li, F., Rajashekhar, V., Ramesh, S., and Soyke, J. Tensorflow-serving: Flexible, high-performance ml serving. arXiv preprint arXiv:1712.06139, 2017. — [arXiv:1712.06139](https://arxiv.org/abs/1712.06139) [find](https://scholar.google.com/scholar?q=Olston%2C+C%2E%2C+Fiedel%2C+N%2E%2C+Gorovoy%2C+K%2E%2C+Harmsen%2C+J%2E%2C+Lao%2C+L%2E%2C+Li%2C+F%2E%2C+Rajashekhar%2C+V%2E%2C+Ramesh%2C+S%2E%2C+and+Soyke%2C+J%2E+Tensorflow%2Dserving%3A+Flexible%2C+high%2Dperformance+ml+serving%2E+arXiv+preprint+arXiv%3A1712%2E06139)
40. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023.
41. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
42. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. [Pytorch: An imperative style, high-performance deep learning library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/). Advances in neural information processing systems, 32, 2019.
43. Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Levskaya, A., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference. arXiv preprint arXiv:2211.05102, 2022. [find](https://scholar.google.com/scholar?q=Pope%2C+R%2E%2C+Douglas%2C+S%2E%2C+Chowdhery%2C+A%2E%2C+Devlin%2C+J%2E%2C+Bradbury%2C+J%2E%2C+Levskaya%2C+A%2E%2C+Heek%2C+J%2E%2C+Xiao%2C+K%2E%2C+Agrawal%2C+S%2E%2C+and+Dean%2C+J%2E+Efficiently+scaling+transformer+inference%2E+arXiv+preprint+arXiv%3A2211%2E05102%2C+)
44. Shazeer, N. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019. [find](https://scholar.google.com/scholar?q=Shazeer%2C+N%2E+Fast+transformer+decoding%3A+One+write%2Dhead+is+all+you+need%2E+arXiv+preprint+arXiv%3A1911%2E02150%2C+2019%2E)
45. Shen, H., Chen, L., Jin, Y., Zhao, L., Kong, B., Philipose, M., Krishnamurthy, A., and Sundaram, R. Nexus: A gpu cluster engine for accelerating dnn-based video analysis. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pp. 322–337, 2019. [find](https://scholar.google.com/scholar?q=Shen%2C+H%2E%2C+Chen%2C+L%2E%2C+Jin%2C+Y%2E%2C+Zhao%2C+L%2E%2C+Kong%2C+B%2E%2C+Philipose%2C+M%2E%2C+Krishnamurthy%2C+A%2E%2C+and+Sundaram%2C+R%2E+Nexus%3A+A+gpu+cluster+engine+for+accelerating+dnn%2Dbased+video+analysis%2E+In+Proceedings+of+the+27th+AC)
46. Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C. [Flexgen: High-throughput generative inference of large language models with a single GPU](https://md.research.elamur.ai/free/en/papers/sheng-2023-flexgen-high-throughput-generative-inference/). In International Conference on Machine Learning, ICML 2023, volume 202 of Proceedings of Machine Lea…
47. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. [find](https://scholar.google.com/scholar?q=Shoeybi%2C+M%2E%2C+Patwary%2C+M%2E%2C+Puri%2C+R%2E%2C+LeGresley%2C+P%2E%2C+Casper%2C+J%2E%2C+and+Catanzaro%2C+B%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+arXiv+preprint+arXiv%3A1909%2E08053%2C)
48. Stern, M., Shazeer, N., and Uszkoreit, J. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31, 2018. [find](https://scholar.google.com/scholar?q=Stern%2C+M%2E%2C+Shazeer%2C+N%2E%2C+and+Uszkoreit%2C+J%2E+Blockwise+parallel+decoding+for+deep+autoregressive+models%2E+Advances+in+Neural+Information+Processing+Systems%2C+31%2C+2018%2E)
49. Tillet, P., Kung, H.-T., and Cox, D. [Triton: an intermediate language and compiler for tiled neural network computations](https://md.research.elamur.ai/free/en/papers/tillet-2019-triton-intermediate-language-compiler-tiled/). In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pp. 10–19, 2019.
50. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023a.
51. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023b.
52. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
53. Vin, H., Goyal, P., and Goyal, A. A statistical admission control algorithm for multimedia servers. In Proceedings of the second ACM international conference on Multimedia, pp. 33–40, 1994. [find](https://scholar.google.com/scholar?q=Vin%2C+H%2E%2C+Goyal%2C+P%2E%2C+and+Goyal%2C+A%2E+A+statistical+admission+control+algorithm+for+multimedia+servers%2E+In+Proceedings+of+the+second+ACM+international+conference+on+Multimedia%2C+pp%2E+33%E2%80%9340%2C+1994%2E)
54. Wang, X., Xiong, Y., Wei, Y., Wang, M., and Li, L. Lightseq: A high performance inference library for transformers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers, pp. 113–120, 2021. [find](https://scholar.google.com/scholar?q=Wang%2C+X%2E%2C+Xiong%2C+Y%2E%2C+Wei%2C+Y%2E%2C+Wang%2C+M%2E%2C+and+Li%2C+L%2E+Lightseq%3A+A+high+performance+inference+library+for+transformers%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Associati)
55. Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, ICML 2023, volume 202 of Proceedings of Machine Learning Research, pp. 38087–38099. [find](https://scholar.google.com/scholar?q=Xiao%2C+G%2E%2C+Lin%2C+J%2E%2C+Seznec%2C+M%2E%2C+Wu%2C+H%2E%2C+Demouth%2C+J%2E%2C+and+Han%2C+S%2E+Smoothquant%3A+Accurate+and+efficient+post%2Dtraining+quantization+for+large+language+models%2E+In+International+Conference+on+Machine+Learnin)
56. Yao, Z., Aminabadi, R. Y., Zhang, M., Wu, X., Li, C., and He, Y. [Zeroquant: Efficient and affordable post-training quantization for large-scale transformers](https://md.research.elamur.ai/free/en/papers/yao-2022-zeroquant-efficient-affordable-post-training/). In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022.
57. Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.-G. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 521–538, 2022. — [Orca: A Distributed Serving System for Transformer-Based Generative Models](https://md.research.elamur.ai/free/en/papers/yu-2022-orca-distributed-serving-system-transformer/)
58. Zhang, H., Tang, Y., Khandelwal, A., and Stoica, I. SHEP-HERD: Serving DNNs in the wild. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pp. 787–808, Boston, MA, April 2023a. USENIX Association. ISBN 978-1-939133-33-5. URL [https://www.usenix.org/conference/nsdi23/p…](https://www.usenix.org/conference/nsdi23/p…) [↗](https://www.usenix.org/conference/nsdi23/p…)
59. Zhang, Q., Chen, M., Bukharin, A., He, P., Cheng, Y., Chen, W., and Zhao, T. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=Zhang%2C+Q%2E%2C+Chen%2C+M%2E%2C+Bukharin%2C+A%2E%2C+He%2C+P%2E%2C+Cheng%2C+Y%2E%2C+Chen%2C+W%2E%2C+and+Zhao%2C+T%2E+Adaptive+budget+allocation+for+parameter%2Defficient+fine%2Dtuning%2E+In+The+Eleventh+International+Conference+on+Learning+Repres)
60. Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., et al. H 2 o: Heavy-hitter oracle for efficient generative inference of large language models. arXiv preprint arXiv:2306.[14048, 2023b](https://md.research.elamur.ai/free/en/papers/zhang-2023-h-2-o-heavy-hitter/).
61. Zheng, L., Li, Z., Zhang, H., Zhuang, Y., Chen, Z., Huang, Y., Wang, Y., Xu, Y., Zhuo, D., Xing, E. P., et al. Alpa: Automating inter-and intra-operator parallelism for distributed deep learning. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 559–578, 2022. — [Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning](https://md.research.elamur.ai/free/en/papers/zheng-2022-alpa-automating-inter-intra-operator/)
62. Zheng, L., Chiang, W.-L., Sheng, Y., Li, T., Zhuang, S., Wu, Z., Zhuang, Y., Li, Z., Lin, Z., Xing, E., et al. [Lmsys-chat-1m: A large-scale real-world llm conversation dataset](https://md.research.elamur.ai/free/en/papers/zheng-2023-lmsys-chat-1m-large-scale/). arXiv preprint arXiv:2309.11998, 2023a.
63. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Conference on Neural Information Processing Systems, Datasets and Benchmarks Track, 2023b.
64. Zhou, Z., Wei, X., Zhang, J., and Sun, G. {PetS}: A unified framework for {Parameter-Efficient} transformers serving. In 2022 USENIX Annual Technical Conference (USENIX ATC 22), pp. 489–504, 2022. [find](https://scholar.google.com/scholar?q=Zhou%2C+Z%2E%2C+Wei%2C+X%2E%2C+Zhang%2C+J%2E%2C+and+Sun%2C+G%2E+%7BPetS%7D%3A+A+unified+framework+for+%7BParameter%2DEfficient%7D+transformers+serving%2E+In+2022+USENIX+Annual+Technical+Conference+%28USENIX+ATC+22%29%2C+pp%2E+489%E2%80%93504%2C+2022%2E)

*Updated 2 November 2026*