---
title: "BinaryBERT: Pushing the Limit of BERT Quantization (2021)"
description: "The authors show that binarizing weights directly causes a sudden accuracy loss. 54 references, cited by 74 papers here."
authors: ["Haoli Bai", "Wei Zhang", "Lu Hou", "Lifeng Shang", "Jin Jin", "Xin Jiang", "Qun Liu", "Michael Lyu", "Irwin King"]
year: 2021
venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)"
source: "https://aclanthology.org/2021.acl-long.334/"
doi: "10.18653/v1/2021.acl-long.334"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bai-2021-binarybert-pushing-limit-bert-quantization/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bai-2021-binarybert-pushing-limit-bert-quantization/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / BinaryBERT: Pushing the Limit of BERT Quantization

# BinaryBERT: Pushing the Limit of BERT Quantization

**Authors:** Haoli Bai, [Wei Zhang](https://md.research.elamur.ai/free/en/authors/wei-zhang/), [Lu Hou](https://md.research.elamur.ai/free/en/authors/lu-hou/), [Lifeng Shang](https://md.research.elamur.ai/free/en/authors/lifeng-shang/), Jin Jin, [Xin Jiang](https://md.research.elamur.ai/free/en/authors/xin-jiang/), [Qun Liu](https://md.research.elamur.ai/free/en/authors/qun-liu/), [Michael Lyu](https://md.research.elamur.ai/free/en/authors/michael-r-lyu/) and Irwin King · **Year:** 2021 · **Venue:** [Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2021](https://md.research.elamur.ai/free/en/venues/acl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.acl-long.334/) · [PDF](https://aclanthology.org/2021.acl-long.334.pdf) · [DOI 10.18653/v1/2021.acl-long.334](https://doi.org/10.18653/v1/2021.acl-long.334) · [OpenAlex](https://openalex.org/W3173374050)

**Citations:** 91 citations (OpenAlex, 2026-10-14) · [74](https://md.research.elamur.ai/free/en/papers/bai-2021-binarybert-pushing-limit-bert-quantization/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Large pre-trained language models are hard to run on edge devices, and quantization shrinks them without redesigning the architecture. Earlier BERT quantization work reached ternary (2-bit) weights but never full 1-bit binarization. The authors show that binarizing weights directly causes a sudden accuracy loss. They trace this to the loss landscape, which becomes much steeper and more irregular for binary weights. They show this through surface visualizations and Hessian top-eigenvalue estimates. Their fix, ternary weight splitting (TWS), first trains a half-width ternary BERT. Its latent full-precision weights and quantized weights are then split, preserving equivalence, into a full-width binary model, which is fine-tuned further. An adaptive variant keeps the most important modules ternary under a size or FLOPs budget. On GLUE and SQuAD, BinaryBERT stays close to full-precision BERT-base while being 24× smaller, setting state-of-the-art BERT compression results. This makes very compact BERT models practical for constrained hardware.

## Key points
- Going from 32-bit to 2-bit weights (with 8-bit activations) costs about 0.6% on MRPC and 0.2% on MNLI-m. Moving to 1-bit costs roughly 3.8% and 0.9%, respectively.
- For the MHA-O layers, the largest Hessian eigenvalue of the binary model is about 15× that of the full-precision one, indicating a much sharper loss surface.
- The baselines are TWN (ternary), BWN (binary) and LAQ (8/4/3-bit), with uniformly quantized activations. TWS also beat LAB, BiReal and gradual-quantization BWN variants in most settings.
- Fine-tuning after splitting from TWN 0.5× brought consistent gains on SQuAD v1.1, MNLI-m, QNLI and MRPC, under both 8-bit and 4-bit activations.
- Adaptive splitting produces mixed ternary/binary models whose arithmetic precision remains easy to deploy. Its importance-based criterion outperformed other splitting criteria across model sizes.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Chinese University of Hong Kong](https://md.research.elamur.ai/free/en/universities/chinese-university-of-hong-kong/) — University, Hong Kong, China · 53 works · 12,921 citations
- [Huawei Technologies](https://md.research.elamur.ai/free/en/companies/huawei-technologies/) — Company, Shenzhen, China · 29 works · 9,991 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 77 works

## Cite

Cite as: Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jin Jin, Xin Jiang, Qun Liu, Michael Lyu and Irwin King. 2021. BinaryBERT: Pushing the Limit of BERT Quantization. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{bai2021binarybert,
  title = {{BinaryBERT: Pushing the Limit of BERT Quantization}},
  author = {Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jin Jin, Xin Jiang, Qun Liu, Michael Lyu and Irwin King},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.334},
  url = {https://doi.org/10.18653/v1/2021.acl-long.334}
}
```

- [← previous Arabic Offensive Language on Twitter: Analysis and Experiments](https://md.research.elamur.ai/free/en/papers/mubarak-2021-arabic-offensive-language-twitter-analysis/)
- [next → Sustainable Modular Debiasing of Language Models](https://md.research.elamur.ai/free/en/papers/lauscher-2021-sustainable-modular-debiasing-language-models/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

54 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. H. Bai, J. Wu, I. King, and M. Lyu. 2020. Few shot network compression via cross distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3203–3210. [find](https://scholar.google.com/scholar?q=H%2E+Bai%2C+J%2E+Wu%2C+I%2E+King%2C+and+M%2E+Lyu%2E+2020%2E+Few+shot+network+compression+via+cross+distillation%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+pages+3203%E2%80%933210%2E)
2. Y. Bai, Y. Wang, and E. Liberty. 2018. Proxquant: Quantized neural networks via proximal operators. In International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Y%2E+Bai%2C+Y%2E+Wang%2C+and+E%2E+Liberty%2E+2018%2E+Proxquant%3A+Quantized+neural+networks+via+proximal+operators%2E+In+International+Conference+on+Machine+Learning%2E)
3. A. Bulat, B. Martinez, and G. Tzimiropoulos. 2020. Bats: Binary architecture search. In European Conference on Computer Vision, pages 309–325. [find](https://scholar.google.com/scholar?q=A%2E+Bulat%2C+B%2E+Martinez%2C+and+G%2E+Tzimiropoulos%2E+2020%2E+Bats%3A+Binary+architecture+search%2E+In+European+Conference+on+Computer+Vision%2C+pages+309%E2%80%93325%2E)
4. T. Chen, I. Goodfellow, and J. Shlens. 2016. Net2net: Accelerating learning via knowledge transfer. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=T%2E+Chen%2C+I%2E+Goodfellow%2C+and+J%2E+Shlens%2E+2016%2E+Net2net%3A+Accelerating+learning+via+knowledge+transfer%2E+In+International+Conference+on+Learning+Representations%2E)
5. M. Courbariaux, Y. Bengio, and J. David. 2015. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems. [find](https://scholar.google.com/scholar?q=M%2E+Courbariaux%2C+Y%2E+Bengio%2C+and+J%2E+David%2E+2015%2E+Binaryconnect%3A+Training+deep+neural+networks+with+binary+weights+during+propagations%2E+In+Advances+in+neural+information+processing+systems%2E)
6. M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and L. Kaiser. 2019. Universal transformers. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=M%2E+Dehghani%2C+S%2E+Gouws%2C+O%2E+Vinyals%2C+J%2E+Uszkoreit%2C+and+L%2E+Kaiser%2E+2019%2E+Universal+transformers%2E+In+International+Conference+on+Learning+Representations%2E)
7. J. Devlin, M. Chang, K. Lee, and K. Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In North American Chapter of the Association for Computational Linguistics.
8. Z. Dong, Z. Yao, A. Gholami, M. Mahoney, and K. Keutzer. 2019. Hawq: Hessian aware quantization of neural networks with mixed-precision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 293–302. [find](https://scholar.google.com/scholar?q=Z%2E+Dong%2C+Z%2E+Yao%2C+A%2E+Gholami%2C+M%2E+Mahoney%2C+and+K%2E+Keutzer%2E+2019%2E+Hawq%3A+Hessian+aware+quantization+of+neural+networks+with+mixed%2Dprecision%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Conference+on+Comp)
9. S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha. 2019. Learned step size quantization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=S%2E+K%2E+Esser%2C+J%2E+L%2E+McKinstry%2C+D%2E+Bablani%2C+R%2E+Appuswamy%2C+and+D%2E+S%2E+Modha%2E+2019%2E+Learned+step+size+quantization%2E+In+International+Conference+on+Learning+Representations%2E)
10. A. Fan, E. Grave, and A. Joulin. 2019. [Reducing transformer depth on demand with structured dropout](https://md.research.elamur.ai/free/en/papers/fan-2019-reducing-transformer-depth-demand-structured/). In International Conference on Learning Representations.
11. A. Fan, P. Stock, B. Graham, E. Grave, R. Gribonval, H. Jegou, and A. Joulin. 2020. Training with quantization noise for extreme model compression. Preprint arXiv:2004.07320. — [arXiv:2004.07320](https://arxiv.org/abs/2004.07320) [find](https://scholar.google.com/scholar?q=A%2E+Fan%2C+P%2E+Stock%2C+B%2E+Graham%2C+E%2E+Grave%2C+R%2E+Gribonval%2C+H%2E+Jegou%2C+and+A%2E+Joulin%2E+2020%2E+Training+with+quantization+noise+for+extreme+model+compression%2E+Preprint+arXiv%3A2004%2E07320%2E+%E2%80%94+%5BarXiv%3A2004%2E07320%5D%28http)
12. H. Habi, R. Jennings, and A. Netzer. 2020. Hmq: Hardware friendly mixed precision quantization block for cnns. In European Conference on Computer Vision, pages 448–463. [find](https://scholar.google.com/scholar?q=H%2E+Habi%2C+R%2E+Jennings%2C+and+A%2E+Netzer%2E+2020%2E+Hmq%3A+Hardware+friendly+mixed+precision+quantization+block+for+cnns%2E+In+European+Conference+on+Computer+Vision%2C+pages+448%E2%80%93463%2E)
13. Y. Hao, L. Dong, F. Wei, and K. Xu. 2019. [Visualizing and understanding the effectiveness of BERT](https://md.research.elamur.ai/free/en/papers/hao-2019-visualizing-understanding-effectiveness-bert/). In Conference on Empirical Methods in Natural Language Processing.
14. L. Hou, Z. Huang, L. Shang, X. Jiang, X. Chen, and Q. Liu. 2020. Dynabert: Dynamic bert with adaptive width and depth. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=L%2E+Hou%2C+Z%2E+Huang%2C+L%2E+Shang%2C+X%2E+Jiang%2C+X%2E+Chen%2C+and+Q%2E+Liu%2E+2020%2E+Dynabert%3A+Dynamic+bert+with+adaptive+width+and+depth%2E+In+Advances+in+Neural+Information+Processing+Systems%2E)
15. L. Hou and J. T. Kwok. 2018. Loss-aware weight quantization of deep networks. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=L%2E+Hou+and+J%2E+T%2E+Kwok%2E+2018%2E+Loss%2Daware+weight+quantization+of+deep+networks%2E+In+International+Conference+on+Learning+Representations%2E)
16. L. Hou, Yao Q., and J. T. Kwok. 2017. Loss-aware binarization of deep networks. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=L%2E+Hou%2C+Yao+Q%2E%2C+and+J%2E+T%2E+Kwok%2E+2017%2E+Loss%2Daware+binarization+of+deep+networks%2E+In+International+Conference+on+Learning+Representations%2E)
17. Z. Huang, L Hou, L. Shang, X. Jiang, X. Chen, and Q. Liu. 2021. Ghostbert: Generate more features with cheap operations for bert. In Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Z%2E+Huang%2C+L+Hou%2C+L%2E+Shang%2C+X%2E+Jiang%2C+X%2E+Chen%2C+and+Q%2E+Liu%2E+2021%2E+Ghostbert%3A+Generate+more+features+with+cheap+operations+for+bert%2E+In+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2E)
18. I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio. 2016. Binarized neural networks. In Advances in neural information processing systems. [find](https://scholar.google.com/scholar?q=I%2E+Hubara%2C+M%2E+Courbariaux%2C+D%2E+Soudry%2C+R%2E+El%2DYaniv%2C+and+Y%2E+Bengio%2E+2016%2E+Binarized+neural+networks%2E+In+Advances+in+neural+information+processing+systems%2E)
19. X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu. 2020. Tinybert: Distilling bert for natural language understanding. In Findings of Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=X%2E+Jiao%2C+Y%2E+Yin%2C+L%2E+Shang%2C+X%2E+Jiang%2C+X%2E+Chen%2C+L%2E+Li%2C+F%2E+Wang%2C+and+Q%2E+Liu%2E+2020%2E+Tinybert%3A+Distilling+bert+for+natural+language+understanding%2E+In+Findings+of+Empirical+Methods+in+Natural+Language+Proce)
20. D. Kim, K Singh, and J. Choi. 2020. Learning architectures for binary networks. In European Conference on Computer Vision, pages 575–591. [find](https://scholar.google.com/scholar?q=D%2E+Kim%2C+K+Singh%2C+and+J%2E+Choi%2E+2020%2E+Learning+architectures+for+binary+networks%2E+In+European+Conference+on+Computer+Vision%2C+pages+575%E2%80%93591%2E)
21. H. Kim, K. Kim, J. Kim, and J. Kim. 2019. Bina-ryduo: Reducing gradient mismatch in binary activation network by coupling binary activations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=H%2E+Kim%2C+K%2E+Kim%2C+J%2E+Kim%2C+and+J%2E+Kim%2E+2019%2E+Bina%2Dryduo%3A+Reducing+gradient+mismatch+in+binary+activation+network+by+coupling+binary+activations%2E+In+International+Conference+on+Learning+Representations%2E)
22. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Z%2E+Lan%2C+M%2E+Chen%2C+S%2E+Goodman%2C+K%2E+Gimpel%2C+P%2E+Sharma%2C+and+R%2E+Soricut%2E+2020%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+In+International+Conference+on+Learning+Represent)
23. F. Li, B. Zhang, and B. Liu. 2016. Ternary weight networks. Preprint arXiv:1605.04711. — [arXiv:1605.04711](https://arxiv.org/abs/1605.04711) [find](https://scholar.google.com/scholar?q=F%2E+Li%2C+B%2E+Zhang%2C+and+B%2E+Liu%2E+2016%2E+Ternary+weight+networks%2E+Preprint+arXiv%3A1605%2E04711%2E+%E2%80%94+%5BarXiv%3A1605%2E04711%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1605%2E04711%29)
24. H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein. 2018. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=H%2E+Li%2C+Z%2E+Xu%2C+G%2E+Taylor%2C+C%2E+Studer%2C+and+T%2E+Goldstein%2E+2018%2E+Visualizing+the+loss+landscape+of+neural+nets%2E+In+Advances+in+Neural+Information+Processing+Systems%2E)
25. Y. Li, X. Dong, and W. Wang. 2020a. Additive powersof-two quantization: a non-uniform discretization for neural networks. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Y%2E+Li%2C+X%2E+Dong%2C+and+W%2E+Wang%2E+2020a%2E+Additive+powersof%2Dtwo+quantization%3A+a+non%2Duniform+discretization+for+neural+networks%2E+In+International+Conference+on+Learning+Representations%2E)
26. Y. Li, W. Wang, H. Bai, R. Gong, X. Dong, and F. Yu. 2020b. Efficient bitwidth search for practical mixed precision neural network. Preprint arXiv:2003.07577. — [arXiv:2003.07577](https://arxiv.org/abs/2003.07577) [find](https://scholar.google.com/scholar?q=Y%2E+Li%2C+W%2E+Wang%2C+H%2E+Bai%2C+R%2E+Gong%2C+X%2E+Dong%2C+and+F%2E+Yu%2E+2020b%2E+Efficient+bitwidth+search+for+practical+mixed+precision+neural+network%2E+Preprint+arXiv%3A2003%2E07577%2E+%E2%80%94+%5BarXiv%3A2003%2E07577%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
27. H. Liu, K. Simonyan, and Y. Yang. 2019. Darts: Differentiable architecture search. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=H%2E+Liu%2C+K%2E+Simonyan%2C+and+Y%2E+Yang%2E+2019%2E+Darts%3A+Differentiable+architecture+search%2E+In+International+Conference+on+Learning+Representations%2E)
28. Z. Liu, Z. Shen, M. Savvides, and K. Cheng. 2020. Reactnet: Towards precise binary neural network with generalized activation functions. In European Conference on Computer Vision, pages 143–159. [find](https://scholar.google.com/scholar?q=Z%2E+Liu%2C+Z%2E+Shen%2C+M%2E+Savvides%2C+and+K%2E+Cheng%2E+2020%2E+Reactnet%3A+Towards+precise+binary+neural+network+with+generalized+activation+functions%2E+In+European+Conference+on+Computer+Vision%2C+pages+143%E2%80%93159%2E)
29. Z. Liu, B. Wu, W. Luo, X. Yang, W. Liu, and K. Cheng. 2018. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In European Conference on Computer Vision. [find](https://scholar.google.com/scholar?q=Z%2E+Liu%2C+B%2E+Wu%2C+W%2E+Luo%2C+X%2E+Yang%2C+W%2E+Liu%2C+and+K%2E+Cheng%2E+2018%2E+Bi%2Dreal+net%3A+Enhancing+the+performance+of+1%2Dbit+cnns+with+improved+representational+capability+and+advanced+training+algorithm%2E+In+European+)
30. X. Ma, P. Zhang, S. Zhang, N. Duan, Y. Hou, D. Song, and M. Zhou. 2019. A tensorized transformer for language modeling. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=X%2E+Ma%2C+P%2E+Zhang%2C+S%2E+Zhang%2C+N%2E+Duan%2C+Y%2E+Hou%2C+D%2E+Song%2C+and+M%2E+Zhou%2E+2019%2E+A+tensorized+transformer+for+language+modeling%2E+In+Advances+in+Neural+Information+Processing+Systems%2E)
31. B. Martinez, J. Yang, A. Bulat, and G. Tzimiropoulos. 2020. Training binary neural networks with real-to-binary convolutions. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=B%2E+Martinez%2C+J%2E+Yang%2C+A%2E+Bulat%2C+and+G%2E+Tzimiropoulos%2E+2020%2E+Training+binary+neural+networks+with+real%2Dto%2Dbinary+convolutions%2E+In+International+Conference+on+Learning+Representations%2E)
32. P. Michel, O. Levy, and G. Neubig. 2019. [Are sixteen heads really better than one?](https://md.research.elamur.ai/free/en/papers/michel-2019-sixteen-heads-really-better-than/) In Advances in Neural Information Processing Systems.
33. Y. Nahshan, B. Chmiel, C. Baskin, E. Zheltonozhskii, R. Banner, A. M. Bronstein, and A. Mendelson. 2019. Loss aware post-training quantization. Preprint arXiv:1911.07190. — [arXiv:1911.07190](https://arxiv.org/abs/1911.07190) [find](https://scholar.google.com/scholar?q=Y%2E+Nahshan%2C+B%2E+Chmiel%2C+C%2E+Baskin%2C+E%2E+Zheltonozhskii%2C+R%2E+Banner%2C+A%2E+M%2E+Bronstein%2C+and+A%2E+Mendelson%2E+2019%2E+Loss+aware+post%2Dtraining+quantization%2E+Preprint+arXiv%3A1911%2E07190%2E+%E2%80%94+%5BarXiv%3A1911%2E07190%5D%28https%3A%2F%2F)
34. P. Rajpurkar, R. Jia, and P. Liang. 2018. [Know what you don’t know: Unanswerable questions for squad](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/). Preprint arXiv:1806.03822.
35. J. Wang, H. Bai, J. Wu, X. Shi, J. Huang, I. King, M. Lyu, and J. Cheng. 2020. Revisiting parameter sharing for automatic neural channel number search. In Advances in Neural Information Processing Systems, volume 33. [find](https://scholar.google.com/scholar?q=J%2E+Wang%2C+H%2E+Bai%2C+J%2E+Wu%2C+X%2E+Shi%2C+J%2E+Huang%2C+I%2E+King%2C+M%2E+Lyu%2C+and+J%2E+Cheng%2E+2020%2E+Revisiting+parameter+sharing+for+automatic+neural+channel+number+search%2E+In+Advances+in+Neural+Information+Processing+Sys)
36. K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han. 2019. Haq: Hardware-aware automated quantization with mixed precision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8612–8620. [find](https://scholar.google.com/scholar?q=K%2E+Wang%2C+Z%2E+Liu%2C+Y%2E+Lin%2C+J%2E+Lin%2C+and+S%2E+Han%2E+2019%2E+Haq%3A+Hardware%2Daware+automated+quantization+with+mixed+precision%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition)
37. B. Wu, Y. Wang, P. Zhang, Y. Tian, P. Vajda, and K. Keutzer. 2018a. Mixed precision quantization of convnets via differentiable neural architecture search. Preprint arXiv:1812.00090. — [arXiv:1812.00090](https://arxiv.org/abs/1812.00090) [find](https://scholar.google.com/scholar?q=B%2E+Wu%2C+Y%2E+Wang%2C+P%2E+Zhang%2C+Y%2E+Tian%2C+P%2E+Vajda%2C+and+K%2E+Keutzer%2E+2018a%2E+Mixed+precision+quantization+of+convnets+via+differentiable+neural+architecture+search%2E+Preprint+arXiv%3A1812%2E00090%2E+%E2%80%94+%5BarXiv%3A1812%2E000)
38. J. Wu, Y. Zhang, H. Bai, H. Zhong, J. Hou, W. Liu, and J Huang. 2018b. Pocketflow: An automated framework for compressing and accelerating deep neural networks. In Advances in Neural Information Processing Systems, Workshop on Compact Deep Neural Networks with Industrial Applications. [find](https://scholar.google.com/scholar?q=J%2E+Wu%2C+Y%2E+Zhang%2C+H%2E+Bai%2C+H%2E+Zhong%2C+J%2E+Hou%2C+W%2E+Liu%2C+and+J+Huang%2E+2018b%2E+Pocketflow%3A+An+automated+framework+for+compressing+and+accelerating+deep+neural+networks%2E+In+Advances+in+Neural+Information+Proce)
39. L. Wu, D. Wang, and Q. Liu. 2019. Splitting steepest descent for growing neural architectures. In Advances in Neural Information Processing Systems, volume 32. [find](https://scholar.google.com/scholar?q=L%2E+Wu%2C+D%2E+Wang%2C+and+Q%2E+Liu%2E+2019%2E+Splitting+steepest+descent+for+growing+neural+architectures%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+32%2E)
40. J. Xin, R. Tang, J. Lee, Y. Yu, and J. Lin. 2020. [Deebert: Dynamic early exiting for accelerating bert inference](https://md.research.elamur.ai/free/en/papers/xin-2020-deebert-dynamic-early-exiting-accelerating/). In Annual Meeting of the Association for Computational Linguistics.
41. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). Preprint arXiv:1606.05250.
42. A. H. Zadeh and A. Moshovos. 2020. Gobo: Quantizing attention-based nlp models for low latency and energy efficient inference. Preprint arXiv:2005.03842. [find](https://scholar.google.com/scholar?q=A%2E+H%2E+Zadeh+and+A%2E+Moshovos%2E+2020%2E+Gobo%3A+Quantizing+attention%2Dbased+nlp+models+for+low+latency+and+energy+efficient+inference%2E+Preprint+arXiv%3A2005%2E03842%2E)
43. M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. 2016. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision. [find](https://scholar.google.com/scholar?q=M%2E+Rastegari%2C+V%2E+Ordonez%2C+J%2E+Redmon%2C+and+A%2E+Farhadi%2E+2016%2E+Xnor%2Dnet%3A+Imagenet+classification+using+binary+convolutional+neural+networks%2E+In+European+Conference+on+Computer+Vision%2E)
44. O. Zafrir, G. Boudoukh, P. Izsak, and M. Wasserblat. 2019. Q8bert: Quantized 8bit bert. Preprint arXiv:1910.06188. — [arXiv:1910.06188](https://arxiv.org/abs/1910.06188) [find](https://scholar.google.com/scholar?q=O%2E+Zafrir%2C+G%2E+Boudoukh%2C+P%2E+Izsak%2C+and+M%2E+Wasserblat%2E+2019%2E+Q8bert%3A+Quantized+8bit+bert%2E+Preprint+arXiv%3A1910%2E06188%2E+%E2%80%94+%5BarXiv%3A1910%2E06188%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1910%2E06188%29)
45. V. Sanh, L. Debut, J. Chaumond, and T. Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. Preprint arXiv:1910.01108. [find](https://scholar.google.com/scholar?q=V%2E+Sanh%2C+L%2E+Debut%2C+J%2E+Chaumond%2C+and+T%2E+Wolf%2E+2019%2E+Distilbert%2C+a+distilled+version+of+bert%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E+Preprint+arXiv%3A1910%2E01108%2E)
46. D. Zhang, J. Yang, D. Ye, and G. Hua. 2018. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In European conference on computer vision, pages 365–382. [find](https://scholar.google.com/scholar?q=D%2E+Zhang%2C+J%2E+Yang%2C+D%2E+Ye%2C+and+G%2E+Hua%2E+2018%2E+Lq%2Dnets%3A+Learned+quantization+for+highly+accurate+and+compact+deep+neural+networks%2E+In+European+conference+on+computer+vision%2C+pages+365%E2%80%93382%2E)
47. S. Shen, Z. Dong, J. Ye, L. Ma, Z. Yao, A. Gholami, M. W. Mahoney, and K. Keutzer. 2020. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=S%2E+Shen%2C+Z%2E+Dong%2C+J%2E+Ye%2C+L%2E+Ma%2C+Z%2E+Yao%2C+A%2E+Gholami%2C+M%2E+W%2E+Mahoney%2C+and+K%2E+Keutzer%2E+2020%2E+Q%2Dbert%3A+Hessian+based+ultra+low+precision+quantization+of+bert%2E+In+Proceedings+of+the+AAAI+Conference+on+Artifi)
48. W. Zhang, L. Hou, Y. Yin, L. Shang, X. Chen, X. Jiang, and Q. Liu. 2020. [Ternarybert: Distillation-aware ultra-low bit bert](https://md.research.elamur.ai/free/en/papers/zhang-2020-ternarybert-distillation-aware-ultra-low/). In Conference on Empirical Methods in Natural Language Processing.
49. S. Sun, Y. Cheng, Z. Gan, and J. Liu. 2019. [Patient knowledge distillation for bert model compression](https://md.research.elamur.ai/free/en/papers/sun-2019-patient-knowledge-distillation-bert-model/). In Conference on Empirical Methods in Natural Language Processing.
50. R. Zhao, Y. Hu, J. Dotzel, C. De Sa, and Z. Zhang. 2019. Improving neural network quantization without retraining using outlier channel splitting. In International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=R%2E+Zhao%2C+Y%2E+Hu%2C+J%2E+Dotzel%2C+C%2E+De+Sa%2C+and+Z%2E+Zhang%2E+2019%2E+Improving+neural+network+quantization+without+retraining+using+outlier+channel+splitting%2E+In+International+Conference+on+Machine+Learning%2E)
51. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in neural information processing systems.
52. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. 2018. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). Preprint arXiv:1804.07461.
53. S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen, and Y. Zou. 2016. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. Preprint arXiv:1606.06160. [find](https://scholar.google.com/scholar?q=S%2E+Zhou%2C+Y%2E+Wu%2C+Z%2E+Ni%2C+X%2E+Zhou%2C+H%2E+Wen%2C+and+Y%2E+Zou%2E+2016%2E+Dorefa%2Dnet%3A+Training+low+bitwidth+convolutional+neural+networks+with+low+bitwidth+gradients%2E+Preprint+arXiv%3A1606%2E06160%2E)
54. W. Zhou, C. Xu, T. Ge, J. McAuley, K. Xu, and F. Wei. 2020. Bert loses patience: Fast and robust inference with early exit. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=W%2E+Zhou%2C+C%2E+Xu%2C+T%2E+Ge%2C+J%2E+McAuley%2C+K%2E+Xu%2C+and+F%2E+Wei%2E+2020%2E+Bert+loses+patience%3A+Fast+and+robust+inference+with+early+exit%2E+In+Advances+in+Neural+Information+Processing+Systems%2E)

*Updated 24 October 2026*