---
title: "TernaryBERT: Distillation-aware Ultra-low Bit BERT (2020)"
description: "Fine-tuned BERT models are large: BERT-base has 109M parameters, over 400 MB in 32-bit floats. 40 references, cited by 65 papers here."
authors: ["Wei Zhang", "Lu Hou", "Yichun Yin", "Lifeng Shang", "Xiao Chen", "Xin Jiang", "Qun Liu"]
year: 2020
venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/2020.emnlp-main.37/"
doi: "10.18653/v1/2020.emnlp-main.37"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2020-ternarybert-distillation-aware-ultra-low/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2020-ternarybert-distillation-aware-ultra-low/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) / TernaryBERT: Distillation-aware Ultra-low Bit BERT

# TernaryBERT: Distillation-aware Ultra-low Bit BERT

**Authors:** [Wei Zhang](https://md.research.elamur.ai/free/en/authors/wei-zhang/), [Lu Hou](https://md.research.elamur.ai/free/en/authors/lu-hou/), Yichun Yin, [Lifeng Shang](https://md.research.elamur.ai/free/en/authors/lifeng-shang/), Xiao Chen, [Xin Jiang](https://md.research.elamur.ai/free/en/authors/xin-jiang/) and [Qun Liu](https://md.research.elamur.ai/free/en/authors/qun-liu/) · **Year:** 2020 · **Venue:** [Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2020](https://md.research.elamur.ai/free/en/venues/emnlp/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-main.37/) · [PDF](https://aclanthology.org/2020.emnlp-main.37.pdf) · [DOI 10.18653/v1/2020.emnlp-main.37](https://doi.org/10.18653/v1/2020.emnlp-main.37) · [OpenAlex](https://openalex.org/W3098576111)

**Citations:** 148 citations (OpenAlex, 2026-10-14) · [65](https://md.research.elamur.ai/free/en/papers/zhang-2020-ternarybert-distillation-aware-ultra-low/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Fine-tuned BERT models are large: BERT-base has 109M parameters, over 400 MB in 32-bit floats. This makes them hard to run on phones and other constrained hardware. Earlier quantization work kept accuracy at 8 bits. Going lower usually needed mixed precision, which some hardware handles poorly, or product quantization, which adds a clustering step.

TernaryBERT limits the weights of a fine-tuned BERT to three values (−1, 0, +1) plus a scaling factor, so each weight needs only 2 bits. The authors adapt two existing ternarization schemes, the approximation-based TWN and the loss-aware LAT. They also test how finely to apply scaling in different parts of the model. Activations are quantized to 8 bits. To make up for the lost capacity, a full-precision teacher of the same size guides the ternary student. Distillation targets both the Transformer layer outputs and the final logits, while full-precision weight copies are kept for gradient updates.

On GLUE and SQuAD, TernaryBERT beats other BERT quantization methods and nearly matches full-precision accuracy while being 14.9x smaller. This shows that extremely low-bit BERT is practical for deployment.

## Key points
- **Bit settings and granularity:** weights and word embedding use 2 bits, activations use 8 bits. The word embedding is ternarized row-wise, and Transformer weights layer-wise. Min-max activation quantization did better than symmetric quantization on SQuAD v1.1 (EM/F1 79.9/87.4 vs 79.0/86.9).
- **What is left unquantized:** segment and position embeddings, biases, softmax, layer normalization and the task-specific output layer stay in full precision.
- **MNLI-m results:** TernaryBERT reaches 83.5% at 28 MB, and TernaryTinyBERT reaches 83.4% at 18 MB. For comparison, Q-BERT (2/3-bit mixed precision, 46 MB) gets 81.8% and GOBO with 2-bit weights gets 71.0%.
- **Effect of distillation:** removing distillation on both Transformer layers and logits causes large drops, e.g. CoLA falls from 55.7 to 45.4 and RTE from 72.9 to 56.3.
- **Other helpful factors:** starting from a fine-tuned rather than a pre-trained BERT, and using data augmentation, both improve results, most visibly on CoLA and RTE. Replacing LAT with 3-bit LAQ gives a 3-bit BERT that reaches 84.2% on MNLI-m at 41 MB.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Huawei Technologies](https://md.research.elamur.ai/free/en/companies/huawei-technologies/) — Company, Shenzhen, China · 29 works · 9,991 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 77 works

## Cite

Cite as: Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang and Qun Liu. 2020. TernaryBERT: Distillation-aware Ultra-low Bit BERT. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2020ternarybert,
  title = {{TernaryBERT: Distillation-aware Ultra-low Bit BERT}},
  author = {Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang and Qun Liu},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.37},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.37}
}
```

- [← previous TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task](https://md.research.elamur.ai/free/en/papers/alt-2020-tacred-revisited-thorough-evaluation-tacred/)
- [next → HopeEDI: A Multilingual Hope Speech Detection Dataset for Equality, Diversity, and Inclusion](https://md.research.elamur.ai/free/en/papers/chakravarthi-2020-hopeedi-multilingual-hope-speech-detection/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/)

## References

40 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio. 2016. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=M%2E+Courbariaux%2C+I%2E+Hubara%2C+D%2E+Soudry%2C+R%2E+El%2DYaniv%2C+and+Y%2E+Bengio%2E+2016%2E+Binarized+neural+networks%3A+Training+deep+neural+networks+with+weights+and+activations+constrained+to%2B+1+or%2D1%2E+In+Advances+in+Neu)
2. M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and L. Kaiser. 2019. Universal transformers. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=M%2E+Dehghani%2C+S%2E+Gouws%2C+O%2E+Vinyals%2C+J%2E+Uszkoreit%2C+and+L%2E+Kaiser%2E+2019%2E+Universal+transformers%2E+In+International+Conference+on+Learning+Representations%2E)
3. J. Devlin, M. Chang, K. Lee, and K. Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171–4186.
4. A. Fan, E. Grave, and A. Joulin. 2019. [Reducing transformer depth on demand with structured dropout](https://md.research.elamur.ai/free/en/papers/fan-2019-reducing-transformer-depth-demand-structured/). In International Conference on Learning Representations.
5. J. Kim, Y. Bhalgat, J. Lee, C. Patel, and N. Kwak. 2019. Qkd: Quantization-aware knowledge distillation. Preprint arXiv:1911.12491. — [arXiv:1911.12491](https://arxiv.org/abs/1911.12491) [find](https://scholar.google.com/scholar?q=J%2E+Kim%2C+Y%2E+Bhalgat%2C+J%2E+Lee%2C+C%2E+Patel%2C+and+N%2E+Kwak%2E+2019%2E+Qkd%3A+Quantization%2Daware+knowledge+distillation%2E+Preprint+arXiv%3A1911%2E12491%2E+%E2%80%94+%5BarXiv%3A1911%2E12491%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1911%2E12491%29)
6. Y. Kim and A. M. Rush. 2016. [Sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/). In Conference on Empirical Methods in Natural Language Processing.
7. D. P. Kingma and J. Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=D%2E+P%2E+Kingma+and+J%2E+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+International+Conference+on+Learning+Representations%2E)
8. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Z%2E+Lan%2C+M%2E+Chen%2C+S%2E+Goodman%2C+K%2E+Gimpel%2C+P%2E+Sharma%2C+and+R%2E+Soricut%2E+2019%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+In+International+Conference+on+Learning+Represent)
9. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Z%2E+Lan%2C+M%2E+Chen%2C+S%2E+Goodman%2C+K%2E+Gimpel%2C+P%2E+Sharma%2C+and+R%2E+Soricut%2E+2020%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+In+International+Conference+on+Learning+Represent)
10. C. Leng, Z. Dou, H. Li, S. Zhu, and R. Jin. 2018. Extremely low bit neural network: Squeeze the last bit out with admm. In AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=C%2E+Leng%2C+Z%2E+Dou%2C+H%2E+Li%2C+S%2E+Zhu%2C+and+R%2E+Jin%2E+2018%2E+Extremely+low+bit+neural+network%3A+Squeeze+the+last+bit+out+with+admm%2E+In+AAAI+Conference+on+Artificial+Intelligence%2E)
11. F. Li, B. Zhang, and B. Liu. 2016. Ternary weight networks. Preprint arXiv:1605.04711. — [arXiv:1605.04711](https://arxiv.org/abs/1605.04711) [find](https://scholar.google.com/scholar?q=F%2E+Li%2C+B%2E+Zhang%2C+and+B%2E+Liu%2E+2016%2E+Ternary+weight+networks%2E+Preprint+arXiv%3A1605%2E04711%2E+%E2%80%94+%5BarXiv%3A1605%2E04711%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1605%2E04711%29)
12. Z. Lin, M. Courbariaux, R. Memisevic, and Y. Bengio. 2016. Neural networks with few multiplications. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Z%2E+Lin%2C+M%2E+Courbariaux%2C+R%2E+Memisevic%2C+and+Y%2E+Bengio%2E+2016%2E+Neural+networks+with+few+multiplications%2E+In+International+Conference+on+Learning+Representations%2E)
13. W. Liu, P. Zhou, Z. Zhao, Z. Wang, H. Deng, and Q. Ju. 2020. [Fastbert: a self-distilling bert with adaptive inference time](https://md.research.elamur.ai/free/en/papers/liu-2020-fastbert-self-distilling-bert-adaptive/). In Annual Conference of the Association for Computational Linguistics.
14. A. Fan, P. Stock, B. Graham, E. Grave, R. Gribonval, H. Jegou, and A. Joulin. 2020. Training with quantization noise for extreme model compression. Preprint arXiv:2004.07320. — [arXiv:2004.07320](https://arxiv.org/abs/2004.07320) [find](https://scholar.google.com/scholar?q=A%2E+Fan%2C+P%2E+Stock%2C+B%2E+Graham%2C+E%2E+Grave%2C+R%2E+Gribonval%2C+H%2E+Jegou%2C+and+A%2E+Joulin%2E+2020%2E+Training+with+quantization+noise+for+extreme+model+compression%2E+Preprint+arXiv%3A2004%2E07320%2E+%E2%80%94+%5BarXiv%3A2004%2E07320%5D%28http)
15. I. Loshchilov and F. Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=I%2E+Loshchilov+and+F%2E+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+International+Conference+on+Learning+Representations%2E)
16. G. Hinton, O. Vinyals, and J. Dean. 2015. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). Preprint arXiv:1503.02531.
17. X. Ma, P. Zhang, S. Zhang, N. Duan, Y. Hou, D. Song, and M. Zhou. 2019. A tensorized transformer for language modeling. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=X%2E+Ma%2C+P%2E+Zhang%2C+S%2E+Zhang%2C+N%2E+Duan%2C+Y%2E+Hou%2C+D%2E+Song%2C+and+M%2E+Zhou%2E+2019%2E+A+tensorized+transformer+for+language+modeling%2E+In+Advances+in+Neural+Information+Processing+Systems%2E)
18. L. Hou and J. T. Kwok. 2018. Loss-aware weight quantization of deep networks. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=L%2E+Hou+and+J%2E+T%2E+Kwok%2E+2018%2E+Loss%2Daware+weight+quantization+of+deep+networks%2E+In+International+Conference+on+Learning+Representations%2E)
19. L. Hou, Yao Q., and J. T. Kwok. 2017. Loss-aware binarization of deep networks. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=L%2E+Hou%2C+Yao+Q%2E%2C+and+J%2E+T%2E+Kwok%2E+2017%2E+Loss%2Daware+binarization+of+deep+networks%2E+In+International+Conference+on+Learning+Representations%2E)
20. Y. Mao, Y. Wang, C. Wu, C. Zhang, Y. Wang, Y. Yang, Q. Zhang, Y. Tong, and J. Bai. 2020. Ladabert: Lightweight adaptation of bert through hybrid model compression. Preprint arXiv:2004.04124. [find](https://scholar.google.com/scholar?q=Y%2E+Mao%2C+Y%2E+Wang%2C+C%2E+Wu%2C+C%2E+Zhang%2C+Y%2E+Wang%2C+Y%2E+Yang%2C+Q%2E+Zhang%2C+Y%2E+Tong%2C+and+J%2E+Bai%2E+2020%2E+Ladabert%3A+Lightweight+adaptation+of+bert+through+hybrid+model+compression%2E+Preprint+arXiv%3A2004%2E04124%2E)
21. J. S. McCarley. 2019. Pruning a bert-based question answering model. Preprint arXiv:1910.06360. — [arXiv:1910.06360](https://arxiv.org/abs/1910.06360) [find](https://scholar.google.com/scholar?q=J%2E+S%2E+McCarley%2E+2019%2E+Pruning+a+bert%2Dbased+question+answering+model%2E+Preprint+arXiv%3A1910%2E06360%2E+%E2%80%94+%5BarXiv%3A1910%2E06360%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1910%2E06360%29)
22. L. Hou, L. Shang, X. Jiang, and Q. Liu. 2020. Dynabert: Dynamic bert with adaptive width and depth. Preprint arXiv:2004.04037. [find](https://scholar.google.com/scholar?q=L%2E+Hou%2C+L%2E+Shang%2C+X%2E+Jiang%2C+and+Q%2E+Liu%2E+2020%2E+Dynabert%3A+Dynamic+bert+with+adaptive+width+and+depth%2E+Preprint+arXiv%3A2004%2E04037%2E)
23. P. Michel, O. Levy, and G. Neubig. 2019. [Are sixteen heads really better than one?](https://md.research.elamur.ai/free/en/papers/michel-2019-sixteen-heads-really-better-than/) Preprint arXiv:1905.10650.
24. X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu. 2019. Tinybert: Distilling bert for natural language understanding. Preprint arXiv:1909.10351. [find](https://scholar.google.com/scholar?q=X%2E+Jiao%2C+Y%2E+Yin%2C+L%2E+Shang%2C+X%2E+Jiang%2C+X%2E+Chen%2C+L%2E+Li%2C+F%2E+Wang%2C+and+Q%2E+Liu%2E+2019%2E+Tinybert%3A+Distilling+bert+for+natural+language+understanding%2E+Preprint+arXiv%3A1909%2E10351%2E)
25. A. Polino, R. Pascanu, and D. Alistarh. 2018. Model compression via distillation and quantization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=A%2E+Polino%2C+R%2E+Pascanu%2C+and+D%2E+Alistarh%2E+2018%2E+Model+compression+via+distillation+and+quantization%2E+In+International+Conference+on+Learning+Representations%2E)
26. G. Prato, E. Charlaix, and M. Rezagholizadeh. 2019. Fully quantized transformer for improved translation. Preprint arXiv:1910.10485. — [arXiv:1910.10485](https://arxiv.org/abs/1910.10485) [find](https://scholar.google.com/scholar?q=G%2E+Prato%2C+E%2E+Charlaix%2C+and+M%2E+Rezagholizadeh%2E+2019%2E+Fully+quantized+transformer+for+improved+translation%2E+Preprint+arXiv%3A1910%2E10485%2E+%E2%80%94+%5BarXiv%3A1910%2E10485%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1910%2E10485%29)
27. O. Zafrir, G. Boudoukh, P. Izsak, and M. Wasserblat. 2019. Q8bert: Quantized 8bit bert. Preprint arXiv:1910.06188. — [arXiv:1910.06188](https://arxiv.org/abs/1910.06188) [find](https://scholar.google.com/scholar?q=O%2E+Zafrir%2C+G%2E+Boudoukh%2C+P%2E+Izsak%2C+and+M%2E+Wasserblat%2E+2019%2E+Q8bert%3A+Quantized+8bit+bert%2E+Preprint+arXiv%3A1910%2E06188%2E+%E2%80%94+%5BarXiv%3A1910%2E06188%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1910%2E06188%29)
28. P. Rajpurkar, R. Jia, and P. Liang. 2018. [Know what you don’t know: Unanswerable questions for squad](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/). Preprint arXiv:1806.03822.
29. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). Preprint arXiv:1606.05250.
30. M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. 2016. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision, pages 525–542. [find](https://scholar.google.com/scholar?q=M%2E+Rastegari%2C+V%2E+Ordonez%2C+J%2E+Redmon%2C+and+A%2E+Farhadi%2E+2016%2E+Xnor%2Dnet%3A+Imagenet+classification+using+binary+convolutional+neural+networks%2E+In+European+Conference+on+Computer+Vision%2C+pages+525%E2%80%93542%2E)
31. A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio. 2014. Fitnets: Hints for thin deep nets. Preprint arXiv:1412.6550. [find](https://scholar.google.com/scholar?q=A%2E+Romero%2C+N%2E+Ballas%2C+S%2E+E%2E+Kahou%2C+A%2E+Chassang%2C+C%2E+Gatta%2C+and+Y%2E+Bengio%2E+2014%2E+Fitnets%3A+Hints+for+thin+deep+nets%2E+Preprint+arXiv%3A1412%2E6550%2E)
32. V. Sanh, L. Debut, J. Chaumond, and T. Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. Preprint arXiv:1910.01108. [find](https://scholar.google.com/scholar?q=V%2E+Sanh%2C+L%2E+Debut%2C+J%2E+Chaumond%2C+and+T%2E+Wolf%2E+2019%2E+Distilbert%2C+a+distilled+version+of+bert%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E+Preprint+arXiv%3A1910%2E01108%2E)
33. S. Shen, Z. Dong, J. Ye, L. Ma, Z. Yao, A. Gholami, M. W. Mahoney, and K. Keutzer. 2020. Q-bert: Hessian based ultra low precision quantization of bert. In AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=S%2E+Shen%2C+Z%2E+Dong%2C+J%2E+Ye%2C+L%2E+Ma%2C+Z%2E+Yao%2C+A%2E+Gholami%2C+M%2E+W%2E+Mahoney%2C+and+K%2E+Keutzer%2E+2020%2E+Q%2Dbert%3A+Hessian+based+ultra+low+precision+quantization+of+bert%2E+In+AAAI+Conference+on+Artificial+Intelligence%2E)
34. P. Stock, A. Joulin, R. Gribonval, B. Graham, and H. Jégou. 2020. And the bit goes down: Revisiting the quantization of neural networks. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=P%2E+Stock%2C+A%2E+Joulin%2C+R%2E+Gribonval%2C+B%2E+Graham%2C+and+H%2E+Je%CC%81gou%2E+2020%2E+And+the+bit+goes+down%3A+Revisiting+the+quantization+of+neural+networks%2E+In+International+Conference+on+Learning+Representations%2E)
35. S. Sun, Y. Cheng, Z. Gan, and J. Liu. 2019. [Patient knowledge distillation for bert model compression](https://md.research.elamur.ai/free/en/papers/sun-2019-patient-knowledge-distillation-bert-model/). In Conference on Empirical Methods in Natural Language Processing, pages 4314–4323.
36. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, pages 5998– 6008.
37. E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov. 2019. [Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned](https://md.research.elamur.ai/free/en/papers/voita-2019-analyzing-multi-head-self-attention/). In Annual Conference of the Association for Computational Linguistics.
38. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. 2018. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). Preprint arXiv:1804.07461.
39. W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Preprint arXiv:2002.10957. [find](https://scholar.google.com/scholar?q=W%2E+Wang%2C+F%2E+Wei%2C+L%2E+Dong%2C+H%2E+Bao%2C+N%2E+Yang%2C+and+M%2E+Zhou%2E+2020%2E+Minilm%3A+Deep+self%2Dattention+distillation+for+task%2Dagnostic+compression+of+pre%2Dtrained+transformers%2E+Preprint+arXiv%3A2002%2E10957%2E)
40. A. H. Zadeh and A. Moshovos. 2020. Gobo: Quantizing attention-based nlp models for low latency and energy efficient inference. Preprint arXiv:2005.03842. [find](https://scholar.google.com/scholar?q=A%2E+H%2E+Zadeh+and+A%2E+Moshovos%2E+2020%2E+Gobo%3A+Quantizing+attention%2Dbased+nlp+models+for+low+latency+and+energy+efficient+inference%2E+Preprint+arXiv%3A2005%2E03842%2E)

*Updated 24 October 2026*