---
title: "LUT-GEMM — paper by Park et al. (2022) · ICLR 2024"
description: "When large generative language models produce text token by token, inference is limited mainly by… 55 references, cited by 85 papers here."
authors: ["Gunho Park", "Baeseong Park", "Minsub Kim", "Sungjae Lee", "Jeonghoon Kim", "Beomseok Kwon", "Se Jung Kwon", "Byeongwook Kim", "Youngjoo Lee", "Dongsoo Lee"]
year: 2022
venue: "ICLR 2024"
source: "https://arxiv.org/abs/2206.09557"
arxiv: "2206.09557"
licence: "arXiv — non-exclusive distribution license (arXiv OAI)"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/park-2022-lut-gemm-quantized-matrix-multiplication/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/park-2022-lut-gemm-quantized-matrix-multiplication/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models

# LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models

**Authors:** Gunho Park, Baeseong Park, Minsub Kim, [Sungjae Lee](https://md.research.elamur.ai/free/en/authors/sungjae-lee/), Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee and Dongsoo Lee · **Year:** 2022 · **Venue:** [ICLR 2024](https://md.research.elamur.ai/free/en/venues/international-conference-on-learning-representations-iclr/) ([ICLR 2022](https://md.research.elamur.ai/free/en/venues/international-conference-on-learning-representations-iclr/2022/)) · **Licence:** [arXiv — non-exclusive distribution license (arXiv OAI)](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2206.09557) · [PDF](https://arxiv.org/pdf/2206.09557) · [arXiv 2206.09557](https://arxiv.org/abs/2206.09557) · [OpenAlex](https://openalex.org/W4283313765)

**Citations:** 18 citations (OpenAlex, 2026-10-14) · [85](https://md.research.elamur.ai/free/en/papers/park-2022-lut-gemm-quantized-matrix-multiplication/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
When large generative language models produce text token by token, inference is limited mainly by memory traffic rather than arithmetic. Earlier weight-only quantization kernels, such as those used with OPTQ and AWQ, keep activations in FP16 but expand low-bit weights back to FP16 before multiplying. That saves memory movement but adds a dequantization step that costs real resources. LUT-GEMM is a GPU kernel that multiplies FP16 activations directly with weights stored in the binary-coding quantization (BCQ) format. Activation partial results are precomputed into lookup tables and then retrieved using the binary weight patterns, so no dequantization is needed and fewer operations are performed. The authors show that ordinary uniform quantization can be rewritten as BCQ, which lets one kernel handle both uniform and non-uniform schemes. They also add group-wise scaling factors, which give a tunable balance between compression and accuracy. With 3-bit weights, OPT-175B runs on a single GPU and generates tokens 2.1× faster than OPTQ. This matters because huge models can be served on fewer GPUs with lower power use.

## Key points
- The authors report that matrix multiplications take at least 75% of processing time across a range of LM sizes and input lengths, which is why the kernel targets GEMM.
- For OPT-175B with 3-bit LUT-GEMM:
  - Model size drops from 347.9 GB to 65.3 GB (5.33× compression).
  - Latency is 51.6 ms per token on 1 GPU, compared with 42.4 ms for FP16 cuBLAS on 8 GPUs.
  - Wiki2 perplexity rises only from 8.34 to 8.68.
- For LLaMA models with AWQ-quantized weights and g=128:
  - LLaMA-7B per-token latency on 1 GPU falls from 10.0 ms to 5.5 ms at 3 bits.
  - LLaMA-65B, which does not fit on one GPU in FP16, runs in 31.3 ms at 3 bits.
- In a group-size study on OPT-6.7B/13B/30B evaluated on LAMBADA, group-wise BCQ found better compression/accuracy points than row-wise quantization. Small groups were essential for the larger models.
- Code is released at github.com/naver-aics/lut-gemm.

## Interesting ideas
- **Lookup tables instead of multiplications.** Weights are binary (±1) vectors with scales, so the possible sums of activation sub-vectors can be computed once and then simply looked up. This trick could carry over to other low-bit or binary inference settings, including CPUs and edge devices.
- **One format for many schemes.** Rewriting uniform quantization as a special case of BCQ means a single kernel can serve both uniform and non-uniform formats. Choosing a general representation reduces the engineering cost of supporting many quantization methods.
- **Group size as a separate hyper-parameter.** Bit width (q) and group size (g) are treated as independent knobs and searched jointly for a target compression ratio. The paper's footprint example shows that fewer bits with tiny groups can use more memory than more bits with larger groups (864 vs 640 bits). This reminds us to measure real storage, scaling factors included, rather than just counting bits.
- **Optimize for the phase that actually bottlenecks.** The paper separates compute-bound prompt summarization from memory-bound autoregressive generation and targets the latter. This is a useful lens when deciding which efficiency method fits a given serving workload.
- **Check that compression does not cause collapse.** Results with naive RTN quantization at 3 bits (e.g., OPT-66B Wiki2 perplexity jumping to 51.15) show that speed tables must be paired with quality checks across several datasets.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 321 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 297 works

## Cite

Cite as: Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee and Dongsoo Lee. 2022. LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models. ICLR 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{park2022lut,
  title = {{LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models}},
  author = {Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee and Dongsoo Lee},
  booktitle = {ICLR 2024},
  year = {2022},
  eprint = {2206.09557},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.09557}
}
```

- [← previous Text-Free Prosody-Aware Generative Spoken Language Modeling](https://md.research.elamur.ai/free/en/papers/kharitonov-2022-text-free-prosody-aware-generative/)
- [next → Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction](https://md.research.elamur.ai/free/en/papers/jie-2022-learning-reason-deductively-math-word/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

55 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ghazanfar Ali, Sridutt Bhalachandra, Nicholas Wright, Alan Sill, and Yong Chen. Evaluation of power controls and counters on general-purpose graphics processing units (gpus). 2020. [find](https://scholar.google.com/scholar?q=Ghazanfar+Ali%2C+Sridutt+Bhalachandra%2C+Nicholas+Wright%2C+Alan+Sill%2C+and+Yong+Chen%2E+Evaluation+of+power+controls+and+counters+on+general%2Dpurpose+graphics+processing+units+%28gpus%29%2E+2020%2E)
2. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. [wav2vec 2.0: A framework for self-supervised learning of speech representations](https://md.research.elamur.ai/free/en/papers/baevski-2020-wav2vec-2-0-framework-self/). Advances in neural information processing systems, 33:12449–12460, 2020.
3. Aishwarya Bhandare, Vamsi Sripathi, Deepthi Karkada, Vivek Menon, Sun Choi, Kushal Datta, and Vikram Saletore. Efficient 8-bit quantization of transformer neural machine language translation model. arXiv:1906.00532, 2019. — [arXiv:1906.00532](https://arxiv.org/abs/1906.00532) [find](https://scholar.google.com/scholar?q=Aishwarya+Bhandare%2C+Vamsi+Sripathi%2C+Deepthi+Karkada%2C+Vivek+Menon%2C+Sun+Choi%2C+Kushal+Datta%2C+and+Vikram+Saletore%2E+Efficient+8%2Dbit+quantization+of+transformer+neural+machine+language+translation+model%2E+ar)
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
5. Patrick Chen, Si Si, Yang Li, Ciprian Chelba, and Cho-Jui Hsieh. GroupReduce: Block-wise low-rank approximation for neural language model shrinking. In Advances in Neural Information Processing Systems, 2018. [find](https://scholar.google.com/scholar?q=Patrick+Chen%2C+Si+Si%2C+Yang+Li%2C+Ciprian+Chelba%2C+and+Cho%2DJui+Hsieh%2E+GroupReduce%3A+Block%2Dwise+low%2Drank+approximation+for+neural+language+model+shrinking%2E+In+Advances+in+Neural+Information+Processing+System)
6. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. [A simple framework for contrastive learning of visual representations](https://md.research.elamur.ai/free/en/papers/chen-2020-simple-framework-contrastive-learning-visual/). In International conference on machine learning, pp. 1597–1607. PMLR, 2020.
7. Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Towards the limit of network quantization. In International Conference on Learning Representations (ICLR), 2017. [find](https://scholar.google.com/scholar?q=Yoojin+Choi%2C+Mostafa+El%2DKhamy%2C+and+Jungwon+Lee%2E+Towards+the+limit+of+network+quantization%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2C+2017%2E)
8. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
9. Insoo Chung, Byeongwook Kim, Yoonjung Choi, Se Jung Kwon, Yongkweon Jeon, Baeseong Park, Sangha Kim, and Dongsoo Lee. Extremely low bit transformer quantization for on-device neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4812–4826, 2020. [find](https://scholar.google.com/scholar?q=Insoo+Chung%2C+Byeongwook+Kim%2C+Yoonjung+Choi%2C+Se+Jung+Kwon%2C+Yongkweon+Jeon%2C+Baeseong+Park%2C+Sangha+Kim%2C+and+Dongsoo+Lee%2E+Extremely+low+bit+transformer+quantization+for+on%2Ddevice+neural+machine+translatio)
10. R.L. de Queiroz and P.A. Stein. LUT filters for quantized processing of signals. IEEE Transactions on Signal Processing, 52(3):687–693, 2004. [find](https://scholar.google.com/scholar?q=R%2EL%2E+de+Queiroz+and+P%2EA%2E+Stein%2E+LUT+filters+for+quantized+processing+of+signals%2E+IEEE+Transactions+on+Signal+Processing%2C+52%283%29%3A687%E2%80%93693%2C+2004%2E)
11. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339, 2022. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Mike+Lewis%2C+Younes+Belkada%2C+and+Luke+Zettlemoyer%2E+Llm%2E+int8+%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2E+arXiv+preprint+arXiv%3A2208%2E07339%2C+2022%2E)
12. Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078, 2023. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Ruslan+Svirschevski%2C+Vage+Egiazarian%2C+Denis+Kuznedelev%2C+Elias+Frantar%2C+Saleh+Ashkboos%2C+Alexander+Borzunov%2C+Torsten+Hoefler%2C+and+Dan+Alistarh%2E+Spqr%3A+A+sparse%2Dquantized+representation+for+)
13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum…
14. Ali Edalati, Marzieh Tahaei, Ahmad Rashid, Vahid Partovi Nia, James J. Clark, and Mehdi Rezagholizadeh. Kronecker decomposition for gpt compression, 2021. [find](https://scholar.google.com/scholar?q=Ali+Edalati%2C+Marzieh+Tahaei%2C+Ahmad+Rashid%2C+Vahid+Partovi+Nia%2C+James+J%2E+Clark%2C+and+Mehdi+Rezagholizadeh%2E+Kronecker+decomposition+for+gpt+compression%2C+2021%2E)
15. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Optq: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=Elias+Frantar%2C+Saleh+Ashkboos%2C+Torsten+Hoefler%2C+and+Dan+Alistarh%2E+Optq%3A+Accurate+quantization+for+generative+pre%2Dtrained+transformers%2E+In+The+Eleventh+International+Conference+on+Learning+Representati)
16. Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574, 2019. — [arXiv:1902.09574](https://arxiv.org/abs/1902.09574) [find](https://scholar.google.com/scholar?q=Trevor+Gale%2C+Erich+Elsen%2C+and+Sara+Hooker%2E+The+state+of+sparsity+in+deep+neural+networks%2E+arXiv+preprint+arXiv%3A1902%2E09574%2C+2019%2E+%E2%80%94+%5BarXiv%3A1902%2E09574%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1902%2E09574%29)
17. Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. In International Conference on Learning Representations (ICLR), 2016. [find](https://scholar.google.com/scholar?q=Song+Han%2C+Huizi+Mao%2C+and+William+J%2E+Dally%2E+Deep+compression%3A+Compressing+deep+neural+networks+with+pruning%2C+trained+quantization+and+Huffman+coding%2E+In+International+Conference+on+Learning+Representat)
18. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). In NIPS Deep Learning and Representation Learning Workshop, 2015.
19. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). arXiv preprint arXiv:2203.15556, 2022.
20. Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,… [find](https://scholar.google.com/scholar?q=Benoit+Jacob%2C+Skirmantas+Kligys%2C+Bo+Chen%2C+Menglong+Zhu%2C+Matthew+Tang%2C+Andrew+Howard%2C+Hartwig+Adam%2C+and+Dmitry+Kalenichenko%2E+Quantization+and+training+of+neural+networks+for+efficient+integer%2Darithmeti)
21. Yongkweon Jeon, Baeseong Park, Se Jung Kwon, Byeongwook Kim, Jeongin Yun, and Dongsoo Lee. Biqgemm: matrix multiplication with lookup table for binary-coding-based quantized dnns. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–14. IEEE, 2020… [find](https://scholar.google.com/scholar?q=Yongkweon+Jeon%2C+Baeseong+Park%2C+Se+Jung+Kwon%2C+Byeongwook+Kim%2C+Jeongin+Yun%2C+and+Dongsoo+Lee%2E+Biqgemm%3A+matrix+multiplication+with+lookup+table+for+binary%2Dcoding%2Dbased+quantized+dnns%2E+In+SC20%3A+Internation)
22. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv:2001.08361, 2020.
23. Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Jeon Dong Hyeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dongpil Seo, et al. What changes can large-scale language models bring? intensive study on hyperclova: Billions-scale korean generative pre-trained transformers. In Proceed… [find](https://scholar.google.com/scholar?q=Boseop+Kim%2C+HyoungSeok+Kim%2C+Sang%2DWoo+Lee%2C+Gichang+Lee%2C+Donghyun+Kwak%2C+Jeon+Dong+Hyeon%2C+Sunghyun+Park%2C+Sungju+Kim%2C+Seonhoon+Kim%2C+Dongpil+Seo%2C+et+al%2E+What+changes+can+large%2Dscale+language+models+bring%3F+)
24. Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. [I-bert: Integer-only bert quantization](https://md.research.elamur.ai/free/en/papers/kim-2021-i-bert-integer-only-bert/). In International conference on machine learning, pp. 5506–5518. PMLR, 2021b.
25. Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. [Squeezellm: Dense-and-sparse quantization](https://md.research.elamur.ai/free/en/papers/kim-2023-squeezellm-dense-sparse-quantization/). arXiv preprint arXiv:2306.07629, 2023.
26. Se Jung Kwon, Jeonghoon Kim, Jeongin Bae, Kang Min Yoo, Jin-Hwa Kim, Baeseong Park, Byeongwook Kim, Jung-Woo Ha, Nako Sung, and Dongsoo Lee. Alphatuning: Quantization-aware parameter-efficient adaptation of large-scale pre-trained language models. In Findings of EMNLP 2022, 2022. [find](https://scholar.google.com/scholar?q=Se+Jung+Kwon%2C+Jeonghoon+Kim%2C+Jeongin+Bae%2C+Kang+Min+Yoo%2C+Jin%2DHwa+Kim%2C+Baeseong+Park%2C+Byeongwook+Kim%2C+Jung%2DWoo+Ha%2C+Nako+Sung%2C+and+Dongsoo+Lee%2E+Alphatuning%3A+Quantization%2Daware+parameter%2Defficient+adaptat)
27. Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lee. Flexround: Learnable rounding based on element-wise division for post-training quantization. arXiv preprint arXiv:2306.00317, 2023. — [arXiv:2306.00317](https://arxiv.org/abs/2306.00317) [find](https://scholar.google.com/scholar?q=Jung+Hyun+Lee%2C+Jeonghoon+Kim%2C+Se+Jung+Kwon%2C+and+Dongsoo+Lee%2E+Flexround%3A+Learnable+rounding+based+on+element%2Dwise+division+for+post%2Dtraining+quantization%2E+arXiv+preprint+arXiv%3A2306%2E00317%2C+2023%2E+%E2%80%94+%5BarXi)
28. Darryl Lin, Sachin Talathi, and Sreekanth Annapureddy. Fixed point quantization of deep convolutional networks. In International Conference on Machine Learning, pp. 2849–2858, 2016. [find](https://scholar.google.com/scholar?q=Darryl+Lin%2C+Sachin+Talathi%2C+and+Sreekanth+Annapureddy%2E+Fixed+point+quantization+of+deep+convolutional+networks%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+2849%E2%80%932858%2C+2016%2E)
29. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023. [find](https://scholar.google.com/scholar?q=Ji+Lin%2C+Jiaming+Tang%2C+Haotian+Tang%2C+Shang+Yang%2C+Xingyu+Dang%2C+and+Song+Han%2E+Awq%3A+Activation%2Daware+weight+quantization+for+llm+compression+and+acceleration%2E+arXiv+preprint+arXiv%3A2306%2E00978%2C+2023%2E)
30. Stefano Markidis, Steven Wei Der Chien, Erwin Laure, Ivy Bo Peng, and Jeffrey S Vetter. [Nvidia tensor core programmability, performance & precision](https://md.research.elamur.ai/free/en/papers/markidis-2018-nvidia-tensor-core-programmability-performance/). In 2018 IEEE international parallel and distributed processing symposium workshops (IPDPSW), pp. 522–531.
31. Mark D. McDonnell. Training wide residual networks for deployment using a single bit for each weight. In International Conference on Learning Representations (ICLR), 2018. [find](https://scholar.google.com/scholar?q=Mark+D%2E+McDonnell%2E+Training+wide+residual+networks+for+deployment+using+a+single+bit+for+each+weight%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2C+2018%2E)
32. Pramod Kumar Meher. LUT optimization for memory-based computation. IEEE Transactions on Circuits and Systems II: Express Briefs, 57(4):285–289, 2010. [find](https://scholar.google.com/scholar?q=Pramod+Kumar+Meher%2E+LUT+optimization+for+memory%2Dbased+computation%2E+IEEE+Transactions+on+Circuits+and+Systems+II%3A+Express+Briefs%2C+57%284%29%3A285%E2%80%93289%2C+2010%2E)
33. Szymon Migacz. 8-bit inference with TensorRT. In NVIDIA GPU Technology conference, 2017. [find](https://scholar.google.com/scholar?q=Szymon+Migacz%2E+8%2Dbit+inference+with+TensorRT%2E+In+NVIDIA+GPU+Technology+conference%2C+2017%2E)
34. Tara N. Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low-rank matrix factorization for deep neural network training with high-dimensional output targets. In ICASSP, pp. 6655–6659, 2013. [find](https://scholar.google.com/scholar?q=Tara+N%2E+Sainath%2C+Brian+Kingsbury%2C+Vikas+Sindhwani%2C+Ebru+Arisoy%2C+and+Bhuvana+Ramabhadran%2E+Low%2Drank+matrix+factorization+for+deep+neural+network+training+with+high%2Ddimensional+output+targets%2E+In+ICASSP%2C)
35. Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning (ICML), pp. 7696–7705, 2017. [find](https://scholar.google.com/scholar?q=Markus+Nagel%2C+Rana+Ali+Amjad%2C+Mart+van+Baalen%2C+Christos+Louizos%2C+and+Tijmen+Blankevoort%2E+Up+or+down%3F+adaptive+rounding+for+post%2Dtraining+quantization%2E+In+International+Conference+on+Machine+Learning+%28)
36. Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International… [find](https://scholar.google.com/scholar?q=Deepak+Narayanan%2C+Mohammad+Shoeybi%2C+Jared+Casper%2C+Patrick+LeGresley%2C+Mostofa+Patwary%2C+Vijay+Korthikanti%2C+Dmitri+Vainbrand%2C+Prethvi+Kashinkunti%2C+Julie+Bernauer%2C+Bryan+Catanzaro%2C+et+al%2E+Efficient+large%2D)
37. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset, August 2016. URL https://doi.org/10.5281/zenodo.2630551. — [doi:10.5281/zenodo.2630551](https://doi.org/10.5281/zenodo.2630551) [↗](https://doi.org/10.5281/zenodo.2630551)
38. Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantization. In International Conference on Learning Representations (ICLR), 2018. [find](https://scholar.google.com/scholar?q=Antonio+Polino%2C+Razvan+Pascanu%2C+and+Dan+Alistarh%2E+Model+compression+via+distillation+and+quantization%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2C+2018%2E)
39. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen… [find](https://scholar.google.com/scholar?q=Jack+W%2E+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+Eliza+Rutherford%2C+Tom+Hennigan%2C+Jacob+Menick%2C+A)
40. Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. [XNOR-Net: Imagenet classification using binary convolutional neural networks](https://md.research.elamur.ai/free/en/papers/rastegari-2016-xnor-net-imagenet-classification-using/). In ECCV, 2016.
41. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+arXiv+preprin)
42. Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. [Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model](https://md.research.elamur.ai/free/en/papers/smith-2022-using-deepspeed-megatron-train-megatron/). arXiv preprint arXiv:22…
43. Devesh Tiwari, Saurabh Gupta, George Gallarno, Jim Rogers, and Don Maxwell. Reliability lessons learned from gpu experience with the titan supercomputer at oak ridge leadership computing facility. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and… [find](https://scholar.google.com/scholar?q=Devesh+Tiwari%2C+Saurabh+Gupta%2C+George+Gallarno%2C+Jim+Rogers%2C+and+Don+Maxwell%2E+Reliability+lessons+learned+from+gpu+experience+with+the+titan+supercomputer+at+oak+ridge+leadership+computing+facility%2E+In+)
44. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
45. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
46. Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with integers in deep neural networks. In International Conference on Learning Representations (ICLR), 2018. [find](https://scholar.google.com/scholar?q=Shuang+Wu%2C+Guoqi+Li%2C+Feng+Chen%2C+and+Luping+Shi%2E+Training+and+inference+with+integers+in+deep+neural+networks%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2C+2018%2E)
47. Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models, 2022. URL https://arxiv.org/abs/2211.10438. [↗](https://arxiv.org/abs/2211.10438)
48. Chen Xu, Jianqiang Yao, Zhouchen Lin, Wenwu Ou, Yuanbin Cao, Zhirong Wang, and Hongbin Zha. Alternating multi-bit quantization for recurrent neural networks. In International Conference on Learning Representations (ICLR), 2018. [find](https://scholar.google.com/scholar?q=Chen+Xu%2C+Jianqiang+Yao%2C+Zhouchen+Lin%2C+Wenwu+Ou%2C+Yuanbin+Cao%2C+Zhirong+Wang%2C+and+Hongbin+Zha%2E+Alternating+multi%2Dbit+quantization+for+recurrent+neural+networks%2E+In+International+Conference+on+Learning+Re)
49. Shiyu Xu, Qi Wang, Xingbo Wang, Shihang Wang, and Terry Tao Ye. Multiplication through a single look-up-table (LUT) in CNN inference computation. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, pp. 1–1, 2021. [find](https://scholar.google.com/scholar?q=Shiyu+Xu%2C+Qi+Wang%2C+Xingbo+Wang%2C+Shihang+Wang%2C+and+Terry+Tao+Ye%2E+Multiplication+through+a+single+look%2Dup%2Dtable+%28LUT%29+in+CNN+inference+computation%2E+IEEE+Transactions+on+Computer%2DAided+Design+of+Integrat)
50. Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. [Zeroquant: Efficient and affordable post-training quantization for large-scale transformers](https://md.research.elamur.ai/free/en/papers/yao-2022-zeroquant-efficient-affordable-post-training/). arXiv preprint arXiv:2206.01861, 2022.
51. Jiecao Yu, Andrew Lukefahr, David Palframan, Ganesh Dasika, Reetuparna Das, and Scott Mahlke. Scalpel: Customizing DNN pruning to the underlying hardware parallelism. In Proceedings of the 44th Annual International Symposium on Computer Architecture, pp. 548–560, 2017. [find](https://scholar.google.com/scholar?q=Jiecao+Yu%2C+Andrew+Lukefahr%2C+David+Palframan%2C+Ganesh+Dasika%2C+Reetuparna+Das%2C+and+Scott+Mahlke%2E+Scalpel%3A+Customizing+DNN+pruning+to+the+underlying+hardware+parallelism%2E+In+Proceedings+of+the+44th+Annual)
52. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, and Jie Tang. [Glm-130b: An open bilingual pre-trained model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/), 2022. URL https://arxiv.org/abs/22… [↗](https://arxiv.org/abs/22…)
53. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: Open pre-trained tr… — [OPT: Open Pre-trained Transformer Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/)
54. Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. CoRR, abs/1710.01878, 2017. — [arXiv:1710.01878](https://arxiv.org/abs/1710.01878) [find](https://scholar.google.com/scholar?q=Michael+Zhu+and+Suyog+Gupta%2E+To+prune%2C+or+not+to+prune%3A+exploring+the+efficacy+of+pruning+for+model+compression%2E+CoRR%2C+abs%2F1710%2E01878%2C+2017%2E+%E2%80%94+%5BarXiv%3A1710%2E01878%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1710%2E01878%29)
55. MatMul Communication OPT-6.7B (1 GPU) OPT-13B (1 GPU) Occupancy (%) Number of input tokens Figure 5: Latency breakdown of various OPT models. The overall performance is dominated by MatMul and communication operations. Experiments are conducted with A100 80GB and FasterTransformer (v4.0) framework. [find](https://scholar.google.com/scholar?q=MatMul+Communication+OPT%2D6%2E7B+%281+GPU%29+OPT%2D13B+%281+GPU%29+Occupancy+%28%25%29+Number+of+input+tokens+Figure+5%3A+Latency+breakdown+of+various+OPT+models%2E+The+overall+performance+is+dominated+by+MatMul+and+communi)

*Updated 2 November 2026*