---
title: "Atom: Low-bit Quantization for Efficient and Accurate LLM Serving"
description: "LLM serving relies on batching to keep GPUs busy, and quantization can further reduce memory… 58 references, cited by 132 papers here."
authors: ["Yilong Zhao", "Chien-Yu Lin", "Kan Zhu", "Zihao Ye", "Lequn Chen", "Size Zheng", "Luis Ceze", "Arvind Krishnamurthy", "Tianqi Chen", "Baris Kasikci"]
year: 2023
venue: "MLSys 2024"
source: "https://arxiv.org/abs/2310.19102"
arxiv: "2310.19102"
licence: "arXiv — non-exclusive distribution license (arXiv OAI)"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhao-2023-atom-low-bit-quantization-efficient/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhao-2023-atom-low-bit-quantization-efficient/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / Atom: Low-bit Quantization for Efficient and Accurate LLM Serving

# Atom: Low-bit Quantization for Efficient and Accurate LLM Serving

**Authors:** [Yilong Zhao](https://md.research.elamur.ai/free/en/authors/yilong-zhao/), Chien-Yu Lin, [Kan Zhu](https://md.research.elamur.ai/free/en/authors/kan-zhu/), [Zihao Ye](https://md.research.elamur.ai/free/en/authors/zihao-ye/), [Lequn Chen](https://md.research.elamur.ai/free/en/authors/lequn-chen/), Size Zheng, [Luis Ceze](https://md.research.elamur.ai/free/en/authors/luis-ceze/), [Arvind Krishnamurthy](https://md.research.elamur.ai/free/en/authors/arvind-krishnamurthy/), [Tianqi Chen](https://md.research.elamur.ai/free/en/authors/tianqi-chen/) and [Baris Kasikci](https://md.research.elamur.ai/free/en/authors/baris-kasikci/) · **Year:** 2023 · **Venue:** MLSys 2024 · **Licence:** [arXiv — non-exclusive distribution license (arXiv OAI)](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.19102) · [PDF](https://arxiv.org/pdf/2310.19102) · [arXiv 2310.19102](https://arxiv.org/abs/2310.19102) · [OpenAlex](https://openalex.org/W4388093177)

**Citations:** 24 citations (OpenAlex, 2026-10-14) · [132](https://md.research.elamur.ai/free/en/papers/zhao-2023-atom-low-bit-quantization-efficient/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLM serving relies on batching to keep GPUs busy, and quantization can further reduce memory use and compute cost. Current methods fall short. Weight-only schemes still multiply in FP16. INT8 weight-activation schemes leave INT4 Tensor Cores unused. Earlier 4-bit attempts such as OmniQuant and QLLM lose noticeable perplexity. Atom targets 4-bit quantization of both weights and activations (W4A4) without that loss.

It keeps a small set of outlier channels at higher precision (8-bit), after reordering channels so memory access stays regular. It also uses fine-grained group quantization, computes activation scales dynamically for each input, and stores the KV-cache in low bit-width. Quantization and reordering are fused into existing operators. Custom CUDA kernels for low-bit Tensor Cores are integrated, together with FlashInfer attention, into an end-to-end serving framework.

On Llama models from 7B to 65B, Atom stays close to FP16 quality. At the same latency target, it raises serving throughput by up to 7.73× over FP16 and 2.53× over INT8. This shows that accurate 4-bit serving is practical.

## Key points
- On Llama-65B with 4-bit weights and activations, the average zero-shot accuracy drop is about 1.4%, and WikiText2 perplexity rises by only about 0.3.
- At similar latency, end-to-end throughput improves by up to 7.7× vs FP16, 5.5× vs W4A16 (weight-only), and 2.5× vs W8A8.
- The motivation is hardware peak throughput on A100: INT4 reaches 1248 TOPS, INT8 reaches 624 TOPS, and FP16 Tensor Cores reach only 312 TFLOPS.
- A runtime profile of Llama-7b shows that dense layers plus self-attention take over 90% of execution time. These are therefore the parts worth quantizing.
- Symmetric quantization is used to avoid the extra cross-term computations that asymmetric zero points would add to integer matrix multiplication.

## Interesting ideas
- **Reorder channels before mixed precision:** Outlier channels are moved to the end of the matrix so the high-precision part forms one contiguous block. This keeps mixed-precision kernels regular and fast, a trick that applies to any format mixing two precisions.
- **Fuse cheap preprocessing into neighbouring kernels:** Dynamic quantization and reordering are folded into operators that already run, which hides their cost. This is a general way to make per-input adaptive steps affordable.
- **Use a middle tier for outliers:** Outliers are stored in INT8 rather than FP16, trading a little accuracy for much better efficiency. A graded precision scheme like this may beat a strict "low vs full" split elsewhere too.
- **Profile before optimizing:** The paper first breaks down where serving time goes across batch sizes, then targets those layers. This is a useful evaluation habit for any efficiency project.
- **Measure throughput under a fixed latency target:** Results are reported at matched latency rather than as raw speed alone. This gives a fairer, deployment-relevant comparison between methods.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 598 works · 191,226 citations
- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 402 works · 212,763 citations
- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,267 citations
- [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 212 works · 19,544 citations

Also on the byline: [Seattle University](https://ror.org/02jqc0m91).

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 321 works · [⚙️ Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/) · 188 works · [⚙️ Electrical and Electronic Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/electrical-and-electronic-engineering/) · 33 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 297 works

## Cite

Cite as: Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen and Baris Kasikci. 2023. Atom: Low-bit Quantization for Efficient and Accurate LLM Serving. MLSys 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhao2023atom,
  title = {{Atom: Low-bit Quantization for Efficient and Accurate LLM Serving}},
  author = {Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen and Baris Kasikci},
  booktitle = {MLSys 2024},
  year = {2023},
  eprint = {2310.19102},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.19102}
}
```

- [← previous Locally Typical Sampling](https://md.research.elamur.ai/free/en/papers/meister-2023-locally-typical-sampling/)
- [next → Large Language Models Effectively Leverage Document-level Context for Literary Translation, but Critical Errors Persist](https://md.research.elamur.ai/free/en/papers/karpinska-2023-large-language-models-effectively-leverage/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

58 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abdelkhalik, H., Arafa, Y., Santhi, N., and Badawy, A.-H. Demystifying the nvidia ampere architecture through microbenchmarking and instruction-level analysis, 2022. [find](https://scholar.google.com/scholar?q=Abdelkhalik%2C+H%2E%2C+Arafa%2C+Y%2E%2C+Santhi%2C+N%2E%2C+and+Badawy%2C+A%2E%2DH%2E+Demystifying+the+nvidia+ampere+architecture+through+microbenchmarking+and+instruction%2Dlevel+analysis%2C+2022%2E)
2. Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., Tumanov, A., and Ramjee, R. [Taming throughput-latency tradeoff in llm inference with sarathi-serve](https://md.research.elamur.ai/free/en/papers/agrawal-2024-taming-throughput-latency-tradeoff-llm/), 2024.
3. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. [Gqa: Training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/), 2023.
4. Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/), 2019.
5. Chee, J., Cai, Y., Kuleshov, V., and Sa, C. D. Quip: 2-bit quantization of large language models with guarantees, 2023. [find](https://scholar.google.com/scholar?q=Chee%2C+J%2E%2C+Cai%2C+Y%2E%2C+Kuleshov%2C+V%2E%2C+and+Sa%2C+C%2E+D%2E+Quip%3A+2%2Dbit+quantization+of+large+language+models+with+guarantees%2C+2023%2E)
6. Chen, L. Dissecting batching effects in gpt inference, May 2023. URL [https://le.qun.ch/en/blog/2023/05/13/transformer-batching/](https://le.qun.ch/en/blog/2023/05/13/transformer-batching/). [↗](https://le.qun.ch/en/blog/2023/05/13/transformer-batching/)
7. Chen, L., Ye, Z., Wu, Y., Zhuo, D., Ceze, L., and Krishnamurthy, A. [Punica: Multi-tenant lora serving](https://md.research.elamur.ai/free/en/papers/chen-2023-punica-multi-tenant-lora-serving/), 2023.
8. Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/), 2019.
9. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/), 2018.
10. Dai, D., Deng, C., Zhao, C., Xu, R. X., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., Xie, Z., Li, Y. K., Huang, P., Luo, F., Ruan, C., Sui, Z., and Liang, W. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models, 2024. [find](https://scholar.google.com/scholar?q=Dai%2C+D%2E%2C+Deng%2C+C%2E%2C+Zhao%2C+C%2E%2C+Xu%2C+R%2E+X%2E%2C+Gao%2C+H%2E%2C+Chen%2C+D%2E%2C+Li%2C+J%2E%2C+Zeng%2C+W%2E%2C+Yu%2C+X%2E%2C+Wu%2C+Y%2E%2C+Xie%2C+Z%2E%2C+Li%2C+Y%2E+K%2E%2C+Huang%2C+P%2E%2C+Luo%2C+F%2E%2C+Ruan%2C+C%2E%2C+Sui%2C+Z%2E%2C+and+Liang%2C+W%2E+Deepseekmoe%3A+Towards+ultimate+expe)
11. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. [find](https://scholar.google.com/scholar?q=Dao%2C+T%2E%2C+Fu%2C+D%2E+Y%2E%2C+Ermon%2C+S%2E%2C+Rudra%2C+A%2E%2C+and+Re%CC%81%2C+C%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2C+2022%2E)
12. Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Llm.int8(): 8-bit matrix multiplication for transformers at scale, 2022. [find](https://scholar.google.com/scholar?q=Dettmers%2C+T%2E%2C+Lewis%2C+M%2E%2C+Belkada%2C+Y%2E%2C+and+Zettlemoyer%2C+L%2E+Llm%2Eint8%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2C+2022%2E)
13. Duarte, F. Number of chatgpt users, Jul 2023. URL [https://explodingtopics.com/blog/chatgpt-users](https://explodingtopics.com/blog/chatgpt-users). [↗](https://explodingtopics.com/blog/chatgpt-users)
14. Elimian, G. Chatgpt costs 700,000 to run daily, openai may go bankrupt in 2024, Aug 2023. URL [https://technext24.com/2023/08/14/chatgpt-costs-700000-daily-openai](https://technext24.com/2023/08/14/chatgpt-costs-700000-daily-openai). [↗](https://technext24.com/2023/08/14/chatgpt-costs-700000-daily-openai)
15. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023. [find](https://scholar.google.com/scholar?q=Frantar%2C+E%2E%2C+Ashkboos%2C+S%2E%2C+Hoefler%2C+T%2E%2C+and+Alistarh%2C+D%2E+Gptq%3A+Accurate+post%2Dtraining+quantization+for+generative+pre%2Dtrained+transformers%2C+2023%2E)
16. Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., McDonell, K., Muennighoff, N., Phang, J., Reynolds, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, September 2021. URL https://doi.org/10.5281/zenod… — [doi:10.5281/zenodo.5371628](https://doi.org/10.5281/zenodo.5371628) [↗](https://doi.org/10.5281/zenod…)
17. Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces, 2023. [find](https://scholar.google.com/scholar?q=Gu%2C+A%2E+and+Dao%2C+T%2E+Mamba%3A+Linear%2Dtime+sequence+modeling+with+selective+state+spaces%2C+2023%2E)
18. Gu, A., Goel, K., and Ré, C. Efficiently modeling long sequences with structured state spaces, 2022. [find](https://scholar.google.com/scholar?q=Gu%2C+A%2E%2C+Goel%2C+K%2E%2C+and+Re%CC%81%2C+C%2E+Efficiently+modeling+long+sequences+with+structured+state+spaces%2C+2022%2E)
19. Guo, C., Tang, J., Hu, W., Leng, J., Zhang, C., Yang, F., Liu, Y., Guo, M., and Zhu, Y. OliVe: Accelerating large language models via hardware-friendly outlier-victim pair quantization. In Proceedings of the 50th Annual International Symposium on Computer Architecture. ACM, jun 2023. doi: 10.1145/35… [find](https://scholar.google.com/scholar?q=Guo%2C+C%2E%2C+Tang%2C+J%2E%2C+Hu%2C+W%2E%2C+Leng%2C+J%2E%2C+Zhang%2C+C%2E%2C+Yang%2C+F%2E%2C+Liu%2C+Y%2E%2C+Guo%2C+M%2E%2C+and+Zhu%2C+Y%2E+OliVe%3A+Accelerating+large+language+models+via+hardware%2Dfriendly+outlier%2Dvictim+pair+quantization%2E+In+Proceedings)
20. Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, 2016. [find](https://scholar.google.com/scholar?q=Han%2C+S%2E%2C+Mao%2C+H%2E%2C+and+Dally%2C+W%2E+J%2E+Deep+compression%3A+Compressing+deep+neural+networks+with+pruning%2C+trained+quantization+and+huffman+coding%2C+2016%2E)
21. HuggingFace. Sharegpt vicuna unfiltered, May 2023. URL [https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered). [↗](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered)
22. Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integer-arithmetic-only inference, 2017. [find](https://scholar.google.com/scholar?q=Jacob%2C+B%2E%2C+Kligys%2C+S%2E%2C+Chen%2C+B%2E%2C+Zhu%2C+M%2E%2C+Tang%2C+M%2E%2C+Howard%2C+A%2E%2C+Adam%2C+H%2E%2C+and+Kalenichenko%2C+D%2E+Quantization+and+training+of+neural+networks+for+efficient+integer%2Darithmetic%2Donly+inference%2C+2017%2E)
23. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., La… [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E%2C+Sablayrolles%2C+A%2E%2C+Roux%2C+A%2E%2C+Mensch%2C+A%2E%2C+Savary%2C+B%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2E+S%2E%2C+de+las+Casas%2C+D%2E%2C+Hanna%2C+E%2E+B%2E%2C+Bressand%2C+F%2E%2C+Lengyel%2C+G%2E%2C+Bour%2C+G%2E%2C+Lample%2C+G%2E%2C+Lavaud%2C+L%2E+R%2E%2C+Saulnier%2C)
24. Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. [Squeezellm: Dense-and-sparse quantization](https://md.research.elamur.ai/free/en/papers/kim-2023-squeezellm-dense-sparse-quantization/), 2023.
25. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/), 2023.
26. Lee, C., Jin, J., Kim, T., Kim, H., and Park, E. Owq: Lessons learned from activation outliers for weight quantization in large language models. ArXiv, abs/2306.[02272, 2023](https://md.research.elamur.ai/free/en/papers/lee-2023-owq-outlier-aware-weight-quantization/). URL [https://api.semanticscholar.org/CorpusID:259076427](https://api.semanticscholar.org/CorpusID:259076427). [↗](https://api.semanticscholar.org/CorpusID:259076427)
27. Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for llm compression and acceleration, 2023. [find](https://scholar.google.com/scholar?q=Lin%2C+J%2E%2C+Tang%2C+J%2E%2C+Tang%2C+H%2E%2C+Yang%2C+S%2E%2C+Dang%2C+X%2E%2C+Gan%2C+C%2E%2C+and+Han%2C+S%2E+Awq%3A+Activation%2Daware+weight+quantization+for+llm+compression+and+acceleration%2C+2023%2E)
28. Liu, J., Gong, R., Wei, X., Dong, Z., Cai, J., and Zhuang, B. Qllm: Accurate and efficient low-bitwidth quantization for large language models, 2023a. [find](https://scholar.google.com/scholar?q=Liu%2C+J%2E%2C+Gong%2C+R%2E%2C+Wei%2C+X%2E%2C+Dong%2C+Z%2E%2C+Cai%2C+J%2E%2C+and+Zhuang%2C+B%2E+Qllm%3A+Accurate+and+efficient+low%2Dbitwidth+quantization+for+large+language+models%2C+2023a%2E)
29. Liu, S., Liu, Z., Huang, X., Dong, P., and Cheng, K. Llm-fp4: 4-bit floating-point quantized transformers, 2023b. [find](https://scholar.google.com/scholar?q=Liu%2C+S%2E%2C+Liu%2C+Z%2E%2C+Huang%2C+X%2E%2C+Dong%2C+P%2E%2C+and+Cheng%2C+K%2E+Llm%2Dfp4%3A+4%2Dbit+floating%2Dpoint+quantized+transformers%2C+2023b%2E)
30. Marcus, M., Kim, G., Marcinkiewicz, M. A., MacIntyre, R., Bies, A., Ferguson, M., Katz, K., and Schasberger, B. [The penn treebank: Annotating predicate argument structure](https://md.research.elamur.ai/free/en/papers/marcus-1994-penn-treebank-annotating-predicate-argument/). In Proceedings of the Workshop on Human Language Technology, HLT ’94, pp. 114–119, USA, 1994. Association for Computational Ling…
31. Merity, S., Xiong, C., Bradbury, J., and Socher, R. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/), 2016.
32. Micikevicius, P., Stosic, D., Burgess, N., Cornea, M., Dubey, P., Grisenthwaite, R., Ha, S., Heinecke, A., Judd, P., Kamalu, J., Mellempudi, N., Oberman, S., Shoeybi, M., Siu, M., and Wu, H. Fp8 formats for deep learning, 2022. [find](https://scholar.google.com/scholar?q=Micikevicius%2C+P%2E%2C+Stosic%2C+D%2E%2C+Burgess%2C+N%2E%2C+Cornea%2C+M%2E%2C+Dubey%2C+P%2E%2C+Grisenthwaite%2C+R%2E%2C+Ha%2C+S%2E%2C+Heinecke%2C+A%2E%2C+Judd%2C+P%2E%2C+Kamalu%2C+J%2E%2C+Mellempudi%2C+N%2E%2C+Oberman%2C+S%2E%2C+Shoeybi%2C+M%2E%2C+Siu%2C+M%2E%2C+and+Wu%2C+H%2E+Fp8+forma)
33. Nagel, M., Fournarakis, M., Amjad, R. A., Bondarenko, Y., van Baalen, M., and Blankevoort, T. A white paper on neural network quantization, 2021. [find](https://scholar.google.com/scholar?q=Nagel%2C+M%2E%2C+Fournarakis%2C+M%2E%2C+Amjad%2C+R%2E+A%2E%2C+Bondarenko%2C+Y%2E%2C+van+Baalen%2C+M%2E%2C+and+Blankevoort%2C+T%2E+A+white+paper+on+neural+network+quantization%2C+2021%2E)
34. NVIDIA. Nvidia a100 specifications, a. URL [https://www.nvidia.com/en-us/data-center/a100/](https://www.nvidia.com/en-us/data-center/a100/). NVIDIA. Nvidia tensor core, b. URL https://www.nvidia.com/en-us/data-center/tensor-cores/. NVIDIA. Nvidia blackwell platform arrives to power a new era of computing, March 2024a. URL https://nvidianews.nvidi… [↗](https://www.nvidia.com/en-us/data-center/a100/)
35. Patel, P., Choukse, E., Zhang, C., Íñigo Goiri, Shah, A., Maleki, S., and Bianchini, R. [Splitwise: Efficient generative llm inference using phase splitting](https://md.research.elamur.ai/free/en/papers/patel-2023-splitwise-efficient-generative-llm-inference/), 2023.
36. Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Levskaya, A., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference. ArXiv, abs/2211.05102, 2022. URL [https://api.semanticscholar.org/CorpusID:253420623](https://api.semanticscholar.org/CorpusID:253420623). [↗](https://api.semanticscholar.org/CorpusID:253420623)
37. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). J. Mach. Learn. Res., 21(1), jan 2020. ISSN 1532-4435.
38. Rouhani, B. D., Zhao, R., More, A., Hall, M., Khodamoradi, A., Deng, S., Choudhary, D., Cornea, M., Dellinger, E., Denolf, K., Dusan, S., Elango, V., Golub, M., Heinecke, A., James-Roxby, P., Jani, D., Kolhe, G., Langhammer, M., Li, A., Melnick, L., Mesmakhosroshahi, M., Rodriguez, A., Schulte, M.,… — [Microscaling Data Formats for Deep Learning](https://md.research.elamur.ai/free/en/papers/rouhani-2023-microscaling-data-formats-deep-learning/)
39. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale, 2019. [find](https://scholar.google.com/scholar?q=Sakaguchi%2C+K%2E%2C+Bras%2C+R%2E+L%2E%2C+Bhagavatula%2C+C%2E%2C+and+Choi%2C+Y%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2C+2019%2E)
40. Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L., Li, Z., Zhang, K., Gao, P., Qiao, Y., and Luo, P. [Omniquant: Omnidirectionally calibrated quantization for large language models](https://md.research.elamur.ai/free/en/papers/shao-2023-omniquant-omnidirectionally-calibrated-quantization-large/), 2023.
41. Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Fu, D. Y., Xie, Z., Chen, B., Barrett, C. W., Gonzalez, J., Liang, P., Ré, C., Stoica, I., and Zhang, C. High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, 2023. URL… [find](https://scholar.google.com/scholar?q=Sheng%2C+Y%2E%2C+Zheng%2C+L%2E%2C+Yuan%2C+B%2E%2C+Li%2C+Z%2E%2C+Ryabinin%2C+M%2E%2C+Fu%2C+D%2E+Y%2E%2C+Xie%2C+Z%2E%2C+Chen%2C+B%2E%2C+Barrett%2C+C%2E+W%2E%2C+Gonzalez%2C+J%2E%2C+Liang%2C+P%2E%2C+Re%CC%81%2C+C%2E%2C+Stoica%2C+I%2E%2C+and+Zhang%2C+C%2E+High%2Dthroughput+generative+inference+of+)
42. Thakkar, V., Ramani, P., Cecka, C., Shivam, A., Lu, H., Yan, E., Kosaian, J., Hoemmen, M., Wu, H., Kerr, A., Nicely, M., Merrill, D., Blasig, D., Qiao, F., Majcher, P., Springer, P., Hohnerbach, M., Wang, J., and Gupta, M. CUTLASS, January 2023. URL [https://github.com/NVIDIA/cutlass](https://github.com/NVIDIA/cutlass). [↗](https://github.com/NVIDIA/cutlass)
43. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hossei… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
44. Trauzettel-Klosinski, S., Dietz, K., and the IReST Study Group. Standardized Assessment of Reading Performance: The New International Reading Speed Texts IReST. Investigative Ophthalmology & Visual Science, 53(9):5452–5461, 08 2012. ISSN 1552-5783. doi: 10.1167/iovs.11-8284. URL https://doi.org/10.1… — [doi:10.1167/iovs.11-8284](https://doi.org/10.1167/iovs.11-8284) [find](https://scholar.google.com/scholar?q=Trauzettel%2DKlosinski%2C+S%2E%2C+Dietz%2C+K%2E%2C+and+the+IReST+Study+Group%2E+Standardized+Assessment+of+Reading+Performance%3A+The+New+International+Reading+Speed+Texts+IReST%2E+Investigative+Ophthalmology+%26+Visual+Sc)
45. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need, 2023. [find](https://scholar.google.com/scholar?q=Vaswani%2C+A%2E%2C+Shazeer%2C+N%2E%2C+Parmar%2C+N%2E%2C+Uszkoreit%2C+J%2E%2C+Jones%2C+L%2E%2C+Gomez%2C+A%2E+N%2E%2C+Kaiser%2C+L%2E%2C+and+Polosukhin%2C+I%2E+Attention+is+all+you+need%2C+2023%2E)
46. Wei, X., Zhang, Y., Li, Y., Zhang, X., Gong, R., Guo, J., and Liu, X. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling, 2023. [find](https://scholar.google.com/scholar?q=Wei%2C+X%2E%2C+Zhang%2C+Y%2E%2C+Li%2C+Y%2E%2C+Zhang%2C+X%2E%2C+Gong%2C+R%2E%2C+Guo%2C+J%2E%2C+and+Liu%2C+X%2E+Outlier+suppression%2B%3A+Accurate+quantization+of+large+language+models+by+equivalent+and+optimal+shifting+and+scaling%2C+2023%2E)
47. Wikipedia contributors. List of qualcomm snapdragon systems on chips — Wikipedia, the free encyclopedia, 2023. URL [https://en.wikipedia.org/w/index.php?title=List_of_Qualcomm_Snapdragon_systems_on_chips&oldid=1182026635](https://en.wikipedia.org/w/index.php?title=List_of_Qualcomm_Snapdragon_systems_on_chips&oldid=1182026635). [Online; accessed 26-October-2023]. [↗](https://en.wikipedia.org/w/index.php?title=List_of_Qualcomm_Snapdragon_systems_on_chips&oldid=1182026635)
48. Williams, S., Waterman, A., and Patterson, D. [Roofline: an insightful visual performance model for multicore architectures](https://md.research.elamur.ai/free/en/papers/williams-2009-roofline-insightful-visual-performance-model/). Communications of the ACM, 52(4):65–76, 2009.
49. Wu, X., Yao, Z., and He, Y. Zeroquant-fp: A leap forward in llms post-training w4a8 quantization using floating-point formats, 2023. [find](https://scholar.google.com/scholar?q=Wu%2C+X%2E%2C+Yao%2C+Z%2E%2C+and+He%2C+Y%2E+Zeroquant%2Dfp%3A+A+leap+forward+in+llms+post%2Dtraining+w4a8+quantization+using+floating%2Dpoint+formats%2C+2023%2E)
50. Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models, 2023. [find](https://scholar.google.com/scholar?q=Xiao%2C+G%2E%2C+Lin%2C+J%2E%2C+Seznec%2C+M%2E%2C+Wu%2C+H%2E%2C+Demouth%2C+J%2E%2C+and+Han%2C+S%2E+Smoothquant%3A+Accurate+and+efficient+post%2Dtraining+quantization+for+large+language+models%2C+2023%2E)
51. Yao, Z., Aminabadi, R. Y., Zhang, M., Wu, X., Li, C., and He, Y. [Zeroquant: Efficient and affordable post-training quantization for large-scale transformers](https://md.research.elamur.ai/free/en/papers/yao-2022-zeroquant-efficient-affordable-post-training/), 2022.
52. Ye, Z., Chen, L., Lai, R., Zhao, Y., Zheng, S., Shao, J., Hou, B., Jin, H., Zuo, Y., Yin, L., Chen, T., and Ceze, L. Accelerating self-attentions for llm serving with flashinfer, February 2024. URL [https://flashinfer.ai/2024/02/02/introduce-flashinfer.html](https://flashinfer.ai/2024/02/02/introduce-flashinfer.html). [↗](https://flashinfer.ai/2024/02/02/introduce-flashinfer.html)
53. Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.-G. [Orca: A distributed serving system for Transformer-Based generative models](https://md.research.elamur.ai/free/en/papers/yu-2022-orca-distributed-serving-system-transformer/). In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 521–538, Carlsbad, CA, July 2022. USENIX Association. ISBN 978-1-939133-28-…
54. Yuan, Z., Niu, L., Liu, J., Liu, W., Wang, X., Shang, Y., Sun, G., Wu, Q., Wu, J., and Wu, B. Rptq: Reorder-based post-training quantization for large language models, 2023. [find](https://scholar.google.com/scholar?q=Yuan%2C+Z%2E%2C+Niu%2C+L%2E%2C+Liu%2C+J%2E%2C+Liu%2C+W%2E%2C+Wang%2C+X%2E%2C+Shang%2C+Y%2E%2C+Sun%2C+G%2E%2C+Wu%2C+Q%2E%2C+Wu%2C+J%2E%2C+and+Wu%2C+B%2E+Rptq%3A+Reorder%2Dbased+post%2Dtraining+quantization+for+large+language+models%2C+2023%2E)
55. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence?, 2019. [find](https://scholar.google.com/scholar?q=Zellers%2C+R%2E%2C+Holtzman%2C+A%2E%2C+Bisk%2C+Y%2E%2C+Farhadi%2C+A%2E%2C+and+Choi%2C+Y%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F%2C+2019%2E)
56. Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., Wang, Z., and Chen, B. H 2 o: Heavy-hitter oracle for efficient generative inference of large language models, 2023. — [H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2023-h-2-o-heavy-hitter/)
57. Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., and Zhang, H. [Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving](https://md.research.elamur.ai/free/en/papers/zhong-2024-distserve-disaggregating-prefill-decoding-goodput/), 2024.
58. Zhou, C., Richard, V., Savarese, P., Hassman, Z., Maire, M., DiBrino, M., and Li, Y. Sysmol: A hardware-software co-design framework for ultra-low and fine-grained mixed-precision neural networks, 2023. [find](https://scholar.google.com/scholar?q=Zhou%2C+C%2E%2C+Richard%2C+V%2E%2C+Savarese%2C+P%2E%2C+Hassman%2C+Z%2E%2C+Maire%2C+M%2E%2C+DiBrino%2C+M%2E%2C+and+Li%2C+Y%2E+Sysmol%3A+A+hardware%2Dsoftware+co%2Ddesign+framework+for+ultra%2Dlow+and+fine%2Dgrained+mixed%2Dprecision+neural+networks%2C+202)

*Updated 2 November 2026*