---
title: "QServe — paper by Lin et al. (2024) · MLSys 2025"
description: "Low-bit integer quantization promises faster LLM inference, but INT4 approaches have mostly… 44 references, cited by 114 papers here."
authors: ["Yujun Lin", "Haotian Tang", "Shang Yang", "Zhekai Zhang", "Guangxuan Xiao", "Chuang Gan", "Song Han"]
year: 2024
venue: "MLSys 2025"
source: "https://arxiv.org/abs/2405.04532"
arxiv: "2405.04532"
licence: "arXiv — non-exclusive distribution license (arXiv OAI)"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lin-2024-qserve-w4a8kv4-quantization-system-co/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lin-2024-qserve-w4a8kv4-quantization-system-co/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving

# QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving

**Authors:** Yujun Lin, [Haotian Tang](https://md.research.elamur.ai/free/en/authors/haotian-tang/), [Shang Yang](https://md.research.elamur.ai/free/en/authors/shang-yang/), [Zhekai Zhang](https://md.research.elamur.ai/free/en/authors/zhekai-zhang/), [Guangxuan Xiao](https://md.research.elamur.ai/free/en/authors/guangxuan-xiao/), [Chuang Gan](https://md.research.elamur.ai/free/en/authors/chuang-gan/) and [Song Han](https://md.research.elamur.ai/free/en/authors/song-han/) · **Year:** 2024 · **Venue:** MLSys 2025 · **Licence:** [arXiv — non-exclusive distribution license (arXiv OAI)](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2405.04532) · [PDF](https://arxiv.org/pdf/2405.04532) · [arXiv 2405.04532](https://arxiv.org/abs/2405.04532) · [OpenAlex](https://openalex.org/W4396815596)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [114](https://md.research.elamur.ai/free/en/papers/lin-2024-qserve-w4a8kv4-quantization-system-co/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Low-bit integer quantization promises faster LLM inference, but INT4 approaches have mostly helped small-batch edge scenarios. In large-batch cloud serving they often lose to W8A8 or W4A16. The authors trace this to dequantization work running on slow CUDA cores, which costs 20–90% extra runtime on GPUs. Their answer is a co-designed algorithm and system. QoQ is a W4A8KV4 scheme: 4-bit weights, 8-bit activations and a 4-bit KV cache. QServe is the inference library built to run it.

QoQ uses two-stage ("progressive") weight quantization so that every GEMM runs on INT8 tensor cores. It also adds SmoothAttention, which limits accuracy loss when keys are stored in 4 bits. On the system side, QServe reorders weights and exploits register-level parallelism to make dequantization cheap, and it keeps KV4 attention memory-bound.

Across seven LLMs on A100 and L40S, QServe beats TensorRT-LLM, Atom and QuaRot in maximum throughput. A cheaper L40S running QServe often outperforms an A100 running TensorRT-LLM, which cuts serving cost by about 3×.

## Key points
- **Measured throughput gains over TensorRT-LLM:**
  - Llama-3-8B: 1.2× on A100 and 1.4× on L40S.
  - Qwen1.5-72B: 2.4× on A100 and 3.5× on L40S.
- **Gains over W4A4 systems:** on A100, QServe delivers 2.5–2.9× the throughput of Atom and QuaRot. The paper notes that Atom itself runs 20–25% slower than TensorRT-LLM's W4A16/W8A8 on Llama-2-7B.
- **Progressive group quantization:**
  - Weights are first mapped to INT8 with per-channel FP16 scales, using a protective range of [−119, 119].
  - These INT8 values are then group-quantized to INT4.
  - At runtime, INT4→INT8 dequantization multiplies first and subtracts afterward.
- **Roofline analysis on A100:**
  - W4A16 is theoretically faster when the batch size is below about 78; W8A8 wins above that.
  - W4A8 dominates both across batch sizes.
  - One CUDA-core operation costs roughly as much as 50 INT4 tensor-core operations.
- **Release and reproducibility:**
  - Code is in the mit-han-lab/omniserve repository under Apache 2.0, with a Docker artifact.
  - Accuracy results can be reproduced with the deepcompressor library.

## Interesting ideas
- **Profile the slow units, not the peak numbers.** The paper shows that lower nominal precision can lose to higher precision once you count overhead on slower hardware units. Judging any accelerated pipeline by where its bottleneck sits, rather than by headline TOPS, is a habit that transfers well beyond LLM serving.
- **Quantize in stages with headroom.** Weights pass through an 8-bit intermediate whose range is deliberately clipped so the later 4-bit step cannot overflow. That spare range is what makes cheap, parallel unpacking possible, so the same trick could help any custom compact encoding.
- **Move the difficulty to the side you keep precise.** SmoothAttention rescales keys and queries so that outliers end up in the queries, which are never quantized. This rebalancing applies to other paired computations where only one side must be compressed.
- **Use roofline reasoning to choose formats.** Plotting compute intensity against hardware limits makes it clear which precision suits which batch-size regime. The same approach could guide deployment choices for Ukrainian NLP models on modest GPUs.
- **Optimize the cost-per-token metric.** The authors frame success as getting more throughput from a cheaper GPU than from a premium one. That makes a practical evaluation target for budget-limited projects.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works · [⚙️ Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/) · 188 works · [⚙️ Electrical and Electronic Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/electrical-and-electronic-engineering/) · 33 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 297 works

## Cite

Cite as: Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan and Song Han. 2024. QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving. MLSys 2025.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{lin2024qserve,
  title = {{QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving}},
  author = {Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan and Song Han},
  booktitle = {MLSys 2025},
  year = {2024},
  eprint = {2405.04532},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.04532}
}
```

- [← previous Hallucination of Multimodal Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/bai-2024-hallucination-multimodal-large-language-models/)
- [next → Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text](https://md.research.elamur.ai/free/en/papers/hans-2024-spotting-llms-binoculars-zero-shot/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

44 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. [Gqa: Training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/). arXiv preprint arXiv:2305.13245, 2023.
2. Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. [Quarot: Outlier-free 4-bit inference in rotated llms](https://md.research.elamur.ai/free/en/papers/ashkboos-2024-quarot-outlier-free-4-bit/). arXiv preprint arXiv:2404.00456, 2024.
3. Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
4. Chee, J., Cai, Y., Kuleshov, V., and Sa, C. D. Quip: 2-bit quantization of large language models with guarantees, 2024. [find](https://scholar.google.com/scholar?q=Chee%2C+J%2E%2C+Cai%2C+Y%2E%2C+Kuleshov%2C+V%2E%2C+and+Sa%2C+C%2E+D%2E+Quip%3A+2%2Dbit+quantization+of+large+language+models+with+guarantees%2C+2024%2E)
5. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/), 2018.
6. Contributors, L. Lightllm: A light and fast inference service for llm. [https://github.com/ModelTC/lightllm](https://github.com/ModelTC/lightllm), 2023a. [↗](https://github.com/ModelTC/lightllm)
7. Contributors, L. Lmdeploy: A toolkit for compressing, deploying, and serving llm. [https://github.com/InternLM/lmdeploy](https://github.com/InternLM/lmdeploy), 2023b. [↗](https://github.com/InternLM/lmdeploy)
8. Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. GPT3.int8(): 8-bit matrix multiplication for transformers at scale. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Dettmers%2C+T%2E%2C+Lewis%2C+M%2E%2C+Belkada%2C+Y%2E%2C+and+Zettlemoyer%2C+L%2E+GPT3%2Eint8%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2E+In+Oh%2C+A%2E+H%2E%2C+Agarwal%2C+A%2E%2C+Belgrave%2C+D%2E%2C+and+Cho%2C+K%2E+%28eds%2E%29%2C+Advances+in+N)
9. Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. [Qlora: Efficient finetuning of quantized llms](https://md.research.elamur.ai/free/en/papers/dettmers-2023-qlora-efficient-finetuning-quantized-llms/). arXiv preprint arXiv:2305.14314, 2023a.
10. Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D. Spqr: A sparse-quantized representation for near-lossless llm weight compression, 2023b. [find](https://scholar.google.com/scholar?q=Dettmers%2C+T%2E%2C+Svirschevski%2C+R%2E%2C+Egiazarian%2C+V%2E%2C+Kuznedelev%2C+D%2E%2C+Frantar%2C+E%2E%2C+Ashkboos%2C+S%2E%2C+Borzunov%2C+A%2E%2C+Hoefler%2C+T%2E%2C+and+Alistarh%2C+D%2E+Spqr%3A+A+sparse%2Dquantized+representation+for+near%2Dlossless+llm+wei)
11. Fang, C., Zhou, A., and Wang, Z. An algorithm–hardware co-optimized framework for accelerating n: M sparse transformers. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 30(11):1573–1586, 2022. [find](https://scholar.google.com/scholar?q=Fang%2C+C%2E%2C+Zhou%2C+A%2E%2C+and+Wang%2C+Z%2E+An+algorithm%E2%80%93hardware+co%2Doptimized+framework+for+accelerating+n%3A+M+sparse+transformers%2E+IEEE+Transactions+on+Very+Large+Scale+Integration+%28VLSI%29+Systems%2C+30%2811%29%3A1573%E2%80%931)
12. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: Accurate post-training compression for generative pretrained transformers. arXiv preprint arXiv:2210.17323, 2022. [find](https://scholar.google.com/scholar?q=Frantar%2C+E%2E%2C+Ashkboos%2C+S%2E%2C+Hoefler%2C+T%2E%2C+and+Alistarh%2C+D%2E+GPTQ%3A+Accurate+post%2Dtraining+compression+for+generative+pretrained+transformers%2E+arXiv+preprint+arXiv%3A2210%2E17323%2C+2022%2E)
13. Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework… [find](https://scholar.google.com/scholar?q=Gao%2C+L%2E%2C+Tow%2C+J%2E%2C+Abbasi%2C+B%2E%2C+Biderman%2C+S%2E%2C+Black%2C+S%2E%2C+DiPofi%2C+A%2E%2C+Foster%2C+C%2E%2C+Golding%2C+L%2E%2C+Hsu%2C+J%2E%2C+Le+Noac%E2%80%99h%2C+A%2E%2C+Li%2C+H%2E%2C+McDonell%2C+K%2E%2C+Muennighoff%2C+N%2E%2C+Ociepa%2C+C%2E%2C+Phang%2C+J%2E%2C+Reynolds%2C+L%2E%2C+Schoelko)
14. Ham, T. J., Jung, S. J., Kim, S., Oh, Y. H., Park, Y., Song, Y., Park, J.-H., Lee, S., Park, K., Lee, J. W., et al. Aˆ 3: Accelerating attention mechanisms in neural networks with approximation. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 328–341. [find](https://scholar.google.com/scholar?q=Ham%2C+T%2E+J%2E%2C+Jung%2C+S%2E+J%2E%2C+Kim%2C+S%2E%2C+Oh%2C+Y%2E+H%2E%2C+Park%2C+Y%2E%2C+Song%2C+Y%2E%2C+Park%2C+J%2E%2DH%2E%2C+Lee%2C+S%2E%2C+Park%2C+K%2E%2C+Lee%2C+J%2E+W%2E%2C+et+al%2E+A%CB%86+3%3A+Accelerating+attention+mechanisms+in+neural+networks+with+approximation%2E+In+20)
15. Ham, T. J., Lee, Y., Seo, S. H., Kim, S., Choi, H., Jung, S. J., and Lee, J. W. Elsa: Hardware-software co-design for efficient, lightweight self-attention mechanism in neural networks. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), pp. 692– 705. [find](https://scholar.google.com/scholar?q=Ham%2C+T%2E+J%2E%2C+Lee%2C+Y%2E%2C+Seo%2C+S%2E+H%2E%2C+Kim%2C+S%2E%2C+Choi%2C+H%2E%2C+Jung%2C+S%2E+J%2E%2C+and+Lee%2C+J%2E+W%2E+Elsa%3A+Hardware%2Dsoftware+co%2Ddesign+for+efficient%2C+lightweight+self%2Dattention+mechanism+in+neural+networks%2E+In+2021+ACM%2FIE)
16. Hong, S., Moon, S., Kim, J., Lee, S., Kim, M., Lee, D., and Kim, J.-Y. Dfx: A low-latency multi-fpga appliance for accelerating transformer-based text generation. In 2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO), pp. 616–630. [find](https://scholar.google.com/scholar?q=Hong%2C+S%2E%2C+Moon%2C+S%2E%2C+Kim%2C+J%2E%2C+Lee%2C+S%2E%2C+Kim%2C+M%2E%2C+Lee%2C+D%2E%2C+and+Kim%2C+J%2E%2DY%2E+Dfx%3A+A+low%2Dlatency+multi%2Dfpga+appliance+for+accelerating+transformer%2Dbased+text+generation%2E+In+2022+55th+IEEE%2FACM+International+S)
17. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E%2C+Sablayrolles%2C+A%2E%2C+Mensch%2C+A%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2E+S%2E%2C+Casas%2C+D%2E+d%2E+l%2E%2C+Bressand%2C+F%2E%2C+Lengyel%2C+G%2E%2C+Lample%2C+G%2E%2C+Saulnier%2C+L%2E%2C+et+al%2E+Mistral+7b%2E+arXiv+preprint+arXiv%3A2310%2E06825%2C+2023%2E)
18. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E%2C+Sablayrolles%2C+A%2E%2C+Roux%2C+A%2E%2C+Mensch%2C+A%2E%2C+Savary%2C+B%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2E+S%2E%2C+Casas%2C+D%2E+d%2E+l%2E%2C+Hanna%2C+E%2E+B%2E%2C+Bressand%2C+F%2E%2C+et+al%2E+Mixtral+of+experts%2E+arXiv+preprint+arXiv%3A2401%2E04088%2C+)
19. Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. [Squeezellm: Dense-and-sparse quantization](https://md.research.elamur.ai/free/en/papers/kim-2023-squeezellm-dense-sparse-quantization/), 2024.
20. Kim, Y. J., Henry, R., Fahim, R., and Awadalla, H. H. Who says elephants can’t run: Bringing large scale moe models into cloud scale production. arXiv preprint arXiv:2211.10017, 2022. [find](https://scholar.google.com/scholar?q=Kim%2C+Y%2E+J%2E%2C+Henry%2C+R%2E%2C+Fahim%2C+R%2E%2C+and+Awadalla%2C+H%2E+H%2E+Who+says+elephants+can%E2%80%99t+run%3A+Bringing+large+scale+moe+models+into+cloud+scale+production%2E+arXiv+preprint+arXiv%3A2211%2E10017%2C+2022%2E)
21. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611–626, 2023a.
22. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023b.
23. Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for llm compression and acceleration. In MLSys, 2024. [find](https://scholar.google.com/scholar?q=Lin%2C+J%2E%2C+Tang%2C+J%2E%2C+Tang%2C+H%2E%2C+Yang%2C+S%2E%2C+Chen%2C+W%2E%2DM%2E%2C+Wang%2C+W%2E%2DC%2E%2C+Xiao%2C+G%2E%2C+Dang%2C+X%2E%2C+Gan%2C+C%2E%2C+and+Han%2C+S%2E+Awq%3A+Activation%2Daware+weight+quantization+for+llm+compression+and+acceleration%2E+In+MLSys%2C+2024)
24. Merity, S., Xiong, C., Bradbury, J., and Socher, R. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/), 2016.
25. NVIDIA. TensorRT-LLM: A TensorRT Toolbox for Optimized Large Language Model Inference, 2023. URL [https://github.com/NVIDIA/TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM). [↗](https://github.com/NVIDIA/TensorRT-LLM)
26. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, high-perfo… — [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
27. Qu, Z., Liu, L., Tu, F., Chen, Z., Ding, Y., and Xie, Y. Dota: detect and omit weak attentions for scalable transformer acceleration. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 14– 26, 2022. [find](https://scholar.google.com/scholar?q=Qu%2C+Z%2E%2C+Liu%2C+L%2E%2C+Tu%2C+F%2E%2C+Chen%2C+Z%2E%2C+Ding%2C+Y%2E%2C+and+Xie%2C+Y%2E+Dota%3A+detect+and+omit+weak+attentions+for+scalable+transformer+acceleration%2E+In+Proceedings+of+the+27th+ACM+International+Conference+on+Archite)
28. Rouhani, B. D., Zhao, R., More, A., Hall, M., Khodamoradi, A., Deng, S., Choudhary, D., Cornea, M., Dellinger, E., Denolf, K., et al. [Microscaling data formats for deep learning](https://md.research.elamur.ai/free/en/papers/rouhani-2023-microscaling-data-formats-deep-learning/). arXiv preprint arXiv:2310.10537, 2023.
29. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641, 2019. [find](https://scholar.google.com/scholar?q=Sakaguchi%2C+K%2E%2C+Bras%2C+R%2E+L%2E%2C+Bhagavatula%2C+C%2E%2C+and+Choi%2C+Y%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+arXiv+preprint+arXiv%3A1907%2E10641%2C+2019%2E)
30. Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L., Li, Z., Zhang, K. Z., Gao, P., Qiao, Y., and Luo, P. [Omniquant: Omnidirectionally calibrated quantization for large language models](https://md.research.elamur.ai/free/en/papers/shao-2023-omniquant-omnidirectionally-calibrated-quantization-large/). arXiv preprint arXiv:2308.13137, 2023.
31. Tambe, T., Hooper, C., Pentecost, L., Jia, T., Yang, E.-Y., Donato, M., Sanh, V., Whatmough, P., Rush, A. M., Brooks, D., et al. Edgebert: Sentence-level energy optimizations for latency-aware multi-task nlp inference. In MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture, p… [find](https://scholar.google.com/scholar?q=Tambe%2C+T%2E%2C+Hooper%2C+C%2E%2C+Pentecost%2C+L%2E%2C+Jia%2C+T%2E%2C+Yang%2C+E%2E%2DY%2E%2C+Donato%2C+M%2E%2C+Sanh%2C+V%2E%2C+Whatmough%2C+P%2E%2C+Rush%2C+A%2E+M%2E%2C+Brooks%2C+D%2E%2C+et+al%2E+Edgebert%3A+Sentence%2Dlevel+energy+optimizations+for+latency%2Daware+multi%2Dt)
32. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/), 2023a.
33. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023b.
34. Wang, H., Zhang, Z., and Han, S. [Spatten: Efficient sparse attention architecture with cascade token and head pruning](https://md.research.elamur.ai/free/en/papers/wang-2020-spatten-efficient-sparse-attention-architecture/). In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pp. 97–110.
35. Wei, X., Zhang, Y., Zhang, X., Gong, R., Zhang, S., Zhang, Q., Yu, F., and Liu, X. Outlier suppression: Pushing the limit of low-bit transformer language models. arXiv preprint arXiv:2209.13325, 2022. — [arXiv:2209.13325](https://arxiv.org/abs/2209.13325) [find](https://scholar.google.com/scholar?q=Wei%2C+X%2E%2C+Zhang%2C+Y%2E%2C+Zhang%2C+X%2E%2C+Gong%2C+R%2E%2C+Zhang%2C+S%2E%2C+Zhang%2C+Q%2E%2C+Yu%2C+F%2E%2C+and+Liu%2C+X%2E+Outlier+suppression%3A+Pushing+the+limit+of+low%2Dbit+transformer+language+models%2E+arXiv+preprint+arXiv%3A2209%2E13325%2C+2022%2E)
36. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Huggingface’s transformers: State-of-th… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
37. Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, 2023. [find](https://scholar.google.com/scholar?q=Xiao%2C+G%2E%2C+Lin%2C+J%2E%2C+Seznec%2C+M%2E%2C+Wu%2C+H%2E%2C+Demouth%2C+J%2E%2C+and+Han%2C+S%2E+SmoothQuant%3A+Accurate+and+efficient+post%2Dtraining+quantization+for+large+language+models%2E+In+Proceedings+of+the+40th+International+Confe)
38. Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., Yu, K., Liu, P., Liu, Q., Yue, S., Yang, S., Yang, S., Yu, T., Xie, W., Huang, W., Hu, X., Ren, X., Niu, X., Nie, P., Xu, Y., Liu, Y., Wang, Y., Cai, Y., Gu, Z., Liu, Z., and Dai, Z. Yi: Open foundati… [find](https://scholar.google.com/scholar?q=Young%2C+A%2E%2C+Chen%2C+B%2E%2C+Li%2C+C%2E%2C+Huang%2C+C%2E%2C+Zhang%2C+G%2E%2C+Zhang%2C+G%2E%2C+Li%2C+H%2E%2C+Zhu%2C+J%2E%2C+Chen%2C+J%2E%2C+Chang%2C+J%2E%2C+Yu%2C+K%2E%2C+Liu%2C+P%2E%2C+Liu%2C+Q%2E%2C+Yue%2C+S%2E%2C+Yang%2C+S%2E%2C+Yang%2C+S%2E%2C+Yu%2C+T%2E%2C+Xie%2C+W%2E%2C+Huang%2C+W%2E%2C+Hu%2C+X%2E%2C+Ren%2C+X%2E%2C+)
39. Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.-G. [Orca: A distributed serving system for Transformer-Based generative models](https://md.research.elamur.ai/free/en/papers/yu-2022-orca-distributed-serving-system-transformer/). In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 521–538, Carlsbad, CA, July 2022. USENIX Association. ISBN 978-1-939133-28-…
40. Zadeh, A. H., Edo, I., Awad, O. M., and Moshovos, A. Gobo: Quantizing attention-based nlp models for low latency and energy efficient inference. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pp. 811–824. [find](https://scholar.google.com/scholar?q=Zadeh%2C+A%2E+H%2E%2C+Edo%2C+I%2E%2C+Awad%2C+O%2E+M%2E%2C+and+Moshovos%2C+A%2E+Gobo%3A+Quantizing+attention%2Dbased+nlp+models+for+low+latency+and+energy+efficient+inference%2E+In+2020+53rd+Annual+IEEE%2FACM+International+Symposium+on)
41. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? CoRR, abs/1905.07830, 2019. URL http://arxiv.org/abs/1905.07830. [↗](http://arxiv.org/abs/1905.07830)
42. Zhang, L., Fei, W., Wu, W., He, Y., Lou, Z., and Zhou, H. Dual grained quantization: Efficient fine-grained quantization for llm. arXiv preprint arXiv:2310.04836, 2023. — [arXiv:2310.04836](https://arxiv.org/abs/2310.04836) [find](https://scholar.google.com/scholar?q=Zhang%2C+L%2E%2C+Fei%2C+W%2E%2C+Wu%2C+W%2E%2C+He%2C+Y%2E%2C+Lou%2C+Z%2E%2C+and+Zhou%2C+H%2E+Dual+grained+quantization%3A+Efficient+fine%2Dgrained+quantization+for+llm%2E+arXiv+preprint+arXiv%3A2310%2E04836%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E04836%5D%28https%3A%2F%2Far)
43. Zhao, Y., Lin, C.-Y., Zhu, K., Ye, Z., Chen, L., Zheng, S., Ceze, L., Krishnamurthy, A., Chen, T., and Kasikci, B. [Atom: Low-bit quantization for efficient and accurate llm serving](https://md.research.elamur.ai/free/en/papers/zhao-2023-atom-low-bit-quantization-efficient/). In MLSys, 2023.
44. Zheng, L., Yin, L., Xie, Z., Huang, J., Sun, C., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. Efficiently programming large language models using sglang, 2023. [find](https://scholar.google.com/scholar?q=Zheng%2C+L%2E%2C+Yin%2C+L%2E%2C+Xie%2C+Z%2E%2C+Huang%2C+J%2E%2C+Sun%2C+C%2E%2C+Yu%2C+C%2E+H%2E%2C+Cao%2C+S%2E%2C+Kozyrakis%2C+C%2E%2C+Stoica%2C+I%2E%2C+Gonzalez%2C+J%2E+E%2E%2C+Barrett%2C+C%2E%2C+and+Sheng%2C+Y%2E+Efficiently+programming+large+language+models+using+sglang%2C+)

*Updated 2 November 2026*