---
title: "OWQ — paper by Lee et al. (2023) — summary and ideas"
description: "The paper also proposes Weak Column Tuning (WCT), a parameter-efficient fine-tuning method. 36 references, cited by 69 papers here."
authors: ["Changhun Lee", "Jungyu Jin", "Taesu Kim", "Hyungjun Kim", "Eunhyeok Park"]
year: 2023
venue: "arXiv:2306.02272"
source: "https://arxiv.org/abs/2306.02272"
doi: "10.48550/arxiv.2306.02272"
arxiv: "2306.02272"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lee-2023-owq-outlier-aware-weight-quantization/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lee-2023-owq-outlier-aware-weight-quantization/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models

# OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models

**Authors:** Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim and Eunhyeok Park · **Year:** 2023 · **Venue:** [arXiv:2306.02272](https://md.research.elamur.ai/free/en/venues/proceedings-of-the-aaai-conference-on-artificial-intelligence/) ([Proceedings of the AAAI Conference on Artificial Intelligence 2023](https://md.research.elamur.ai/free/en/venues/proceedings-of-the-aaai-conference-on-artificial-intelligence/2023/)) · **Publisher:** [Association for the Advancement of Artificial Intelligence](https://md.research.elamur.ai/free/en/publishers/association-for-the-advancement-of-artificial-intelligence/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2306.02272) · [PDF](https://arxiv.org/pdf/2306.02272) · [DOI 10.48550/arxiv.2306.02272](https://doi.org/10.48550/arxiv.2306.02272) · [arXiv 2306.02272](https://arxiv.org/abs/2306.02272) · [OpenAlex](https://openalex.org/W4393147284)

**Citations:** 80 citations (OpenAlex, 2026-10-14) · [69](https://md.research.elamur.ai/free/en/papers/lee-2023-owq-outlier-aware-weight-quantization/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
LLMs are expensive to serve. Weight-only quantization below 4 bits reduces this cost, but OPTQ (also called GPTQ) still loses noticeable quality at 3 bits, and smaller models suffer most. The authors observe that the few activation outliers known from int8 work also hurt weight-only quantization. The input channels that carry those outliers make the matching weight columns highly sensitive.

OWQ uses this observation. It ranks weight columns with a Hessian-based score, keeps a handful of these "weak columns" in fp16, and quantizes the remaining dense weights with a tuned OPTQ procedure. With about 3.1 bits per weight, OWQ reaches roughly the quality of 4-bit OPTQ, and the extra latency and memory are small.

The paper also proposes Weak Column Tuning (WCT), a parameter-efficient fine-tuning method. WCT updates only the fp16 weak columns and leaves the quantized part frozen. It matches full-precision LoRA while using a fraction of the memory, and it beats QLoRA in GPT-4-judged comparisons. This lets one compressed model serve both inference and task adaptation. Code is available at github.com/xvyaward/owq.

## Key points
- **Perplexity gains over OPTQ:** at about 3.01 effective bits, WikiText-2 perplexity improves over 3-bit OPTQ on both tested models.
  - OPT-6.7B: 12.76 → 11.22.
  - LLaMA-7B: 8.08 → 6.65.
- **Bit-width sweep on OPT-6.7B:**
  - Even 3.005 bits already beats 3-bit OPTQ, with about 2.9% latency overhead and 0.21% memory overhead.
  - Gains level off as more columns are kept in fp16, while the overhead keeps growing.
- **Comparison with group-wise quantization:** adding OPTQ-style grouping on top of OWQ helps little. Plain 3.01-bit OWQ needs only about 10% of the storage overhead of grouped OPTQ (group size 128) and gives similar or better perplexity and zero-shot accuracy.
- **WCT efficiency:** WCT matches full-precision LoRA while using 24.4% of the memory at inference.
- **Fine-tuning evaluation:** fine-tuned LLaMA-7B variants answered Vicuna Benchmark questions and were judged by GPT-4. WCT on OWQ won against QLoRA and against WCT built on simpler quantizers (minmax RTN and minmax OPTQ).
- **Unquantized parts:** embedding and head weights stay in full precision.

## Interesting ideas
- **Use activation statistics to protect weights.** In weight-only compression, decide which parameters to protect by looking at how strongly their inputs fire, not just at the weights themselves. The same reasoning could guide pruning or low-rank compression of other models.
- **Score the error that matters downstream.** The selection metric multiplies Hessian-based sensitivity by the weight perturbation. In the paper's ablation, scoring by weight error alone was clearly worse, which suggests that compression criteria should target the layer's output error.
- **Mix precision at column granularity.** Keeping whole columns in fp16 stays hardware-friendly and costs very little. A tiny budget (around 0.01 extra bits) captures most of the gain, which makes this a cheap add-on to almost any uniform quantizer.
- **Fine-tune only the parameters you already kept in high precision.** Reusing the protected columns as the trainable set means no separate adapter weights are needed. Because these columns are the most sensitive, a small rank is enough.
- **Judge the frozen base model after adaptation.** The paper compares quantizers after fine-tuning rather than before. This shows that the quality of the frozen base still matters once adapters are trained, which is a useful check for any QLoRA-style pipeline.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Pohang University of Science and Technology](https://md.research.elamur.ai/free/en/universities/pohang-university-of-science-and-technology/) — University, Pohang, South Korea · 2 works · 354 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 77 works

## Cite

Cite as: Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim and Eunhyeok Park. 2023. OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models. arXiv:2306.02272.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{lee2023owq,
  title = {{OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models}},
  author = {Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim and Eunhyeok Park},
  year = {2023},
  doi = {10.48550/arxiv.2306.02272},
  eprint = {2306.02272},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.02272}
}
```

- [← previous SemEval-2023 Task 3: Detecting the Category, the Framing, and the Persuasion Techniques in Online News in a Multi-lingual Setup](https://md.research.elamur.ai/free/en/papers/piskorski-2023-semeval-2023-task-3-detecting/)
- [next → SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2023-scibench-evaluating-college-level-scientific/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

36 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Beeching, E.; Fourrier, C.; Habib, N.; Han, S.; Lambert, N.; Rajani, N.; Sanseviero, O.; Tunstall, L.; and Wolf, T. 2023. Open LLM Leaderboard. [https://huggingface.co/](https://huggingface.co/) spaces/HuggingFaceH4/open llm leaderboard. [↗](https://huggingface.co/)
2. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). NeurIPS, 33: 1877–1901.
3. Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; Stoica, I.; and Xing, E. P. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. [find](https://scholar.google.com/scholar?q=Chiang%2C+W%2E%2DL%2E%3B+Li%2C+Z%2E%3B+Lin%2C+Z%2E%3B+Sheng%2C+Y%2E%3B+Wu%2C+Z%2E%3B+Zhang%2C+H%2E%3B+Zheng%2C+L%2E%3B+Zhuang%2C+S%2E%3B+Zhuang%2C+Y%2E%3B+Gonzalez%2C+J%2E+E%2E%3B+Stoica%2C+I%2E%3B+and+Xing%2C+E%2E+P%2E+2023%2E+Vicuna%3A+An+Open%2DSource+Chatbot+Impressing+GPT%2D4+with)
4. Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457. [find](https://scholar.google.com/scholar?q=Clark%2C+P%2E%3B+Cowhey%2C+I%2E%3B+Etzioni%2C+O%2E%3B+Khot%2C+T%2E%3B+Sabharwal%2C+A%2E%3B+Schoenick%2C+C%2E%3B+and+Tafjord%2C+O%2E+2018%2E+Think+you+have+solved+question+answering%3F+try+arc%2C+the+ai2+reasoning+challenge%2E+arXiv%3A1803%2E05457%2E)
5. Dettmers, T.; Lewis, M.; Belkada, Y.; and Zettlemoyer, L. 2022. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv:2208.07339. [find](https://scholar.google.com/scholar?q=Dettmers%2C+T%2E%3B+Lewis%2C+M%2E%3B+Belkada%2C+Y%2E%3B+and+Zettlemoyer%2C+L%2E+2022%2E+Llm%2E+int8+%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2E+arXiv%3A2208%2E07339%2E)
6. Dettmers, T.; Pagnoni, A.; Holtzman, A.; and Zettlemoyer, L. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314. [find](https://scholar.google.com/scholar?q=Dettmers%2C+T%2E%3B+Pagnoni%2C+A%2E%3B+Holtzman%2C+A%2E%3B+and+Zettlemoyer%2C+L%2E+2023%2E+QLoRA%3A+Efficient+Finetuning+of+Quantized+LLMs%2E+arXiv%3A2305%2E14314%2E)
7. Dong, Z.; Yao, Z.; Arfeen, D.; Gholami, A.; Mahoney, M. W.; and Keutzer, K. 2020. Hawq-v2: Hessian aware trace-weighted quantization of neural networks. NeurIPS, 33: 18518–18529. [find](https://scholar.google.com/scholar?q=Dong%2C+Z%2E%3B+Yao%2C+Z%2E%3B+Arfeen%2C+D%2E%3B+Gholami%2C+A%2E%3B+Mahoney%2C+M%2E+W%2E%3B+and+Keutzer%2C+K%2E+2020%2E+Hawq%2Dv2%3A+Hessian+aware+trace%2Dweighted+quantization+of+neural+networks%2E+NeurIPS%2C+33%3A+18518%E2%80%9318529%2E)
8. Dong, Z.; Yao, Z.; Gholami, A.; Mahoney, M. W.; and Keutzer, K. 2019. Hawq: Hessian aware quantization of neural networks with mixed-precision. In CVPR, 293–302. [find](https://scholar.google.com/scholar?q=Dong%2C+Z%2E%3B+Yao%2C+Z%2E%3B+Gholami%2C+A%2E%3B+Mahoney%2C+M%2E+W%2E%3B+and+Keutzer%2C+K%2E+2019%2E+Hawq%3A+Hessian+aware+quantization+of+neural+networks+with+mixed%2Dprecision%2E+In+CVPR%2C+293%E2%80%93302%2E)
9. Esser, S. K.; McKinstry, J. L.; Bablani, D.; Appuswamy, R.; and Modha, D. S. 2019. Learned step size quantization. arXiv:1902.08153. — [arXiv:1902.08153](https://arxiv.org/abs/1902.08153) [find](https://scholar.google.com/scholar?q=Esser%2C+S%2E+K%2E%3B+McKinstry%2C+J%2E+L%2E%3B+Bablani%2C+D%2E%3B+Appuswamy%2C+R%2E%3B+and+Modha%2C+D%2E+S%2E+2019%2E+Learned+step+size+quantization%2E+arXiv%3A1902%2E08153%2E+%E2%80%94+%5BarXiv%3A1902%2E08153%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1902%2E08153%29)
10. Frantar, E.; and Alistarh, D. 2022. Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning. In NeurIPS. [find](https://scholar.google.com/scholar?q=Frantar%2C+E%2E%3B+and+Alistarh%2C+D%2E+2022%2E+Optimal+Brain+Compression%3A+A+Framework+for+Accurate+Post%2DTraining+Quantization+and+Pruning%2E+In+NeurIPS%2E)
11. Frantar, E.; Ashkboos, S.; Hoefler, T.; and Alistarh, D. 2022. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323. [find](https://scholar.google.com/scholar?q=Frantar%2C+E%2E%3B+Ashkboos%2C+S%2E%3B+Hoefler%2C+T%2E%3B+and+Alistarh%2C+D%2E+2022%2E+GPTQ%3A+Accurate+Post%2DTraining+Quantization+for+Generative+Pre%2Dtrained+Transformers%2E+arXiv%3A2210%2E17323%2E)
12. Frantar, E.; Ashkboos, S.; Hoefler, T.; and Alistarh, D. 2023. OPTQ: Accurate Quantization for Generative Pre-trained Transformers. In ICLR. [find](https://scholar.google.com/scholar?q=Frantar%2C+E%2E%3B+Ashkboos%2C+S%2E%3B+Hoefler%2C+T%2E%3B+and+Alistarh%2C+D%2E+2023%2E+OPTQ%3A+Accurate+Quantization+for+Generative+Pre%2Dtrained+Transformers%2E+In+ICLR%2E)
13. Gao, L.; Tow, J.; Abbasi, B.; Biderman, S.; Black, S.; DiPofi, A.; Foster, C.; Golding, L.; Hsu, J.; Le Noac’h, A.; Li, H.; McDonell, K.; Muennighoff, N.; Ociepa, C.; Phang, J.; Reynolds, L.; Schoelkopf, H.; Skowron, A.; Sutawika, L.; Tang, E.; Thite, A.; Wang, B.; Wang, K.; and Zou, A. 2023. A fram… [find](https://scholar.google.com/scholar?q=Gao%2C+L%2E%3B+Tow%2C+J%2E%3B+Abbasi%2C+B%2E%3B+Biderman%2C+S%2E%3B+Black%2C+S%2E%3B+DiPofi%2C+A%2E%3B+Foster%2C+C%2E%3B+Golding%2C+L%2E%3B+Hsu%2C+J%2E%3B+Le+Noac%E2%80%99h%2C+A%2E%3B+Li%2C+H%2E%3B+McDonell%2C+K%2E%3B+Muennighoff%2C+N%2E%3B+Ociepa%2C+C%2E%3B+Phang%2C+J%2E%3B+Reynolds%2C+L%2E%3B+Schoelko)
14. Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2021. [Measuring Massive Multitask Language Understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv:2009.03300.
15. Horowitz, M. 2014. 1.1 computing’s energy problem (and what we can do about it). In ISSCC, 10–14. [find](https://scholar.google.com/scholar?q=Horowitz%2C+M%2E+2014%2E+1%2E1+computing%E2%80%99s+energy+problem+%28and+what+we+can+do+about+it%29%2E+In+ISSCC%2C+10%E2%80%9314%2E)
16. IEEE. Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022. [LoRA: Low-Rank Adaptation of Large Language Models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In ICLR.
17. OpenReview.net. IST-DASLab. 2022. gptq. — [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://md.research.elamur.ai/free/en/papers/kopf-2023-openassistant-conversations-democratizing-large-language/) [https://github.com/IST-DASLab/](https://github.com/IST-DASLab/) gptq. Köpf, A.; Kilcher, Y.; von Rütte, D.; Anagnostidis, S.; Tam, Z.-R.; Stevens, K.; Barhoum, A.; Duc, N. M.; Stanley, O.; Nagyfi, R.; et al. 2023. OpenAssistant Conversations– Democratizing Large Language Model Alignment. ar… [↗](https://github.com/IST-DASLab/)
18. Li, Y.; Gong, R.; Tan, X.; Yang, Y.; Hu, P.; Zhang, Q.; Yu, F.; Wang, W.; and Gu, S. 2021. BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction. In ICLR. [find](https://scholar.google.com/scholar?q=Li%2C+Y%2E%3B+Gong%2C+R%2E%3B+Tan%2C+X%2E%3B+Yang%2C+Y%2E%3B+Hu%2C+P%2E%3B+Zhang%2C+Q%2E%3B+Yu%2C+F%2E%3B+Wang%2C+W%2E%3B+and+Gu%2C+S%2E+2021%2E+BRECQ%3A+Pushing+the+Limit+of+Post%2DTraining+Quantization+by+Block+Reconstruction%2E+In+ICLR%2E)
19. Marcus, M.; Kim, G.; Marcinkiewicz, M. A.; MacIntyre, R.; Bies, A.; Ferguson, M.; Katz, K.; and Schasberger, B. 1994. [The Penn treebank: Annotating predicate argument structure](https://md.research.elamur.ai/free/en/papers/marcus-1994-penn-treebank-annotating-predicate-argument/). In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.
20. Merity, S.; Xiong, C.; Bradbury, J.; and Socher, R. 2016. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/). arXiv:1609.07843.
21. Nagel, M.; Amjad, R. A.; Van Baalen, M.; Louizos, C.; and Blankevoort, T. 2020. Up or down? adaptive rounding for post-training quantization. In ICML, 7197–7206. PMLR. Nagel, M.; Baalen, M. v.; Blankevoort, T.; and Welling, M. 2019. Data-free quantization through weight equalization and bias correct… [find](https://scholar.google.com/scholar?q=Nagel%2C+M%2E%3B+Amjad%2C+R%2E+A%2E%3B+Van+Baalen%2C+M%2E%3B+Louizos%2C+C%2E%3B+and+Blankevoort%2C+T%2E+2020%2E+Up+or+down%3F+adaptive+rounding+for+post%2Dtraining+quantization%2E+In+ICML%2C+7197%E2%80%937206%2E+PMLR%2E+Nagel%2C+M%2E%3B+Baalen%2C+M%2E+v%2E%3B+Blanke)
22. Nahshan, Y.; Chmiel, B.; Baskin, C.; Zheltonozhskii, E.; Banner, R.; Bronstein, A. M.; and Mendelson, A. 2021. Loss aware post-training quantization. Machine Learning, 110(11-12): 3245–3262. [find](https://scholar.google.com/scholar?q=Nahshan%2C+Y%2E%3B+Chmiel%2C+B%2E%3B+Baskin%2C+C%2E%3B+Zheltonozhskii%2C+E%2E%3B+Banner%2C+R%2E%3B+Bronstein%2C+A%2E+M%2E%3B+and+Mendelson%2C+A%2E+2021%2E+Loss+aware+post%2Dtraining+quantization%2E+Machine+Learning%2C+110%2811%2D12%29%3A+3245%E2%80%933262%2E)
23. OpenAI. 2023. [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv:2303.08774.
24. Park, E.; Kim, D.; and Yoo, S. 2018. Energy-efficient neural network accelerator based on outlier-aware low-precision computation. In ISCA, 688–698. [find](https://scholar.google.com/scholar?q=Park%2C+E%2E%3B+Kim%2C+D%2E%3B+and+Yoo%2C+S%2E+2018%2E+Energy%2Defficient+neural+network+accelerator+based+on+outlier%2Daware+low%2Dprecision+computation%2E+In+ISCA%2C+688%E2%80%93698%2E)
25. IEEE. Park, E.; Yoo, S.; and Vajda, P. 2018. Value-aware quantization for training and inference of neural networks. In ECCV, 580–595. [find](https://scholar.google.com/scholar?q=IEEE%2E+Park%2C+E%2E%3B+Yoo%2C+S%2E%3B+and+Vajda%2C+P%2E+2018%2E+Value%2Daware+quantization+for+training+and+inference+of+neural+networks%2E+In+ECCV%2C+580%E2%80%93595%2E)
26. Park, G.; Park, B.; Kim, M.; Lee, S.; Kim, J.; Kwon, B.; Kwon, S. J.; Kim, B.; Lee, Y.; and Lee, D. 2023. LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models. arXiv:2206.09557. [find](https://scholar.google.com/scholar?q=Park%2C+G%2E%3B+Park%2C+B%2E%3B+Kim%2C+M%2E%3B+Lee%2C+S%2E%3B+Kim%2C+J%2E%3B+Kwon%2C+B%2E%3B+Kwon%2C+S%2E+J%2E%3B+Kim%2C+B%2E%3B+Lee%2C+Y%2E%3B+and+Lee%2C+D%2E+2023%2E+LUT%2DGEMM%3A+Quantized+Matrix+Multiplication+based+on+LUTs+for+Efficient+Inference+in+Large%2DScale)
27. Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019. [Pytorch: An imperative style, high-performance deep learning library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/). NeurIPS, 32.
28. Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8): 9. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%3B+Wu%2C+J%2E%3B+Child%2C+R%2E%3B+Luan%2C+D%2E%3B+Amodei%2C+D%2E%3B+Sutskever%2C+I%2E%3B+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A+9%2E)
29. Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). The Journal of Machine Learning Research, 21(1): 5485–5551. Scao, T. L.; Fan, A.; Akiki, C.; Pavlick, E.; I…
30. Wei, X.; Gong, R.; Li, Y.; Liu, X.; and Yu, F. 2022a. QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization. In ICLR. [find](https://scholar.google.com/scholar?q=Wei%2C+X%2E%3B+Gong%2C+R%2E%3B+Li%2C+Y%2E%3B+Liu%2C+X%2E%3B+and+Yu%2C+F%2E+2022a%2E+QDrop%3A+Randomly+Dropping+Quantization+for+Extremely+Low%2Dbit+Post%2DTraining+Quantization%2E+In+ICLR%2E)
31. Wei, X.; Zhang, Y.; Zhang, X.; Gong, R.; Zhang, S.; Zhang, Q.; Yu, F.; and Liu, X. 2022b. Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Wei%2C+X%2E%3B+Zhang%2C+Y%2E%3B+Zhang%2C+X%2E%3B+Gong%2C+R%2E%3B+Zhang%2C+S%2E%3B+Zhang%2C+Q%2E%3B+Yu%2C+F%2E%3B+and+Liu%2C+X%2E+2022b%2E+Outlier+Suppression%3A+Pushing+the+Limit+of+Low%2Dbit+Transformer+Language+Models%2E+In+Advances+in+Neural+Informati)
32. Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; et al. 2019. Huggingface’s [transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). arXiv:1910.03771.
33. Xiao, G.; Lin, J.; Seznec, M.; Demouth, J.; and Han, S. 2022. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv:2211.10438. [find](https://scholar.google.com/scholar?q=Xiao%2C+G%2E%3B+Lin%2C+J%2E%3B+Seznec%2C+M%2E%3B+Demouth%2C+J%2E%3B+and+Han%2C+S%2E+2022%2E+Smoothquant%3A+Accurate+and+efficient+post%2Dtraining+quantization+for+large+language+models%2E+arXiv%3A2211%2E10438%2E)
34. Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? arXiv:1905.07830. [find](https://scholar.google.com/scholar?q=Zellers%2C+R%2E%3B+Holtzman%2C+A%2E%3B+Bisk%2C+Y%2E%3B+Farhadi%2C+A%2E%3B+and+Choi%2C+Y%2E+2019%2E+HellaSwag%3A+Can+a+Machine+Really+Finish+Your+Sentence%3F+arXiv%3A1905%2E07830%2E)
35. Zhang, S.; Roller, S.; Goyal, N.; Artetxe, M.; Chen, M.; Chen, S.; Dewan, C.; Diab, M.; Li, X.; Lin, X. V.; et al. 2022. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv:2205.01068.
36. Zhou, S.; Wu, Y.; Ni, Z.; Zhou, X.; Wen, H.; and Zou, Y. 2016. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv:1606.06160. [find](https://scholar.google.com/scholar?q=Zhou%2C+S%2E%3B+Wu%2C+Y%2E%3B+Ni%2C+Z%2E%3B+Zhou%2C+X%2E%3B+Wen%2C+H%2E%3B+and+Zou%2C+Y%2E+2016%2E+Dorefa%2Dnet%3A+Training+low+bitwidth+convolutional+neural+networks+with+low+bitwidth+gradients%2E+arXiv%3A1606%2E06160%2E)

*Updated 24 October 2026*