---
title: "TinyBERT: Distilling BERT for Natural Language Understanding"
description: "The work offers a practical way to obtain compact NLP models for systems with limited… 42 references, cited by 649 papers here."
authors: ["Xiaoqi Jiao", "Yichun Yin", "Lifeng Shang", "Xin Jiang", "Xiao Chen", "Linlin Li", "Fang Wang", "Qun Liu"]
year: 2020
venue: "Findings of the Association for Computational Linguistics: EMNLP 2020"
source: "https://aclanthology.org/2020.findings-emnlp.372/"
doi: "10.18653/v1/2020.findings-emnlp.372"
arxiv: "1909.10351"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jiao-2020-tinybert-distilling-bert-natural-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jiao-2020-tinybert-distilling-bert-natural-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/) / TinyBERT: Distilling BERT for Natural Language Understanding

# TinyBERT: Distilling BERT for Natural Language Understanding

**Authors:** [Xiaoqi Jiao](https://md.research.elamur.ai/free/en/authors/xiaoqi-jiao/), [Yichun Yin](https://md.research.elamur.ai/free/en/authors/yichun-yin/), [Lifeng Shang](https://md.research.elamur.ai/free/en/authors/lifeng-shang/), [Xin Jiang](https://md.research.elamur.ai/free/en/authors/xin-jiang/), [Xiao Chen](https://md.research.elamur.ai/free/en/authors/xiao-chen/), [Linlin Li](https://md.research.elamur.ai/free/en/authors/linlin-li/), Fang Wang and [Qun Liu](https://md.research.elamur.ai/free/en/authors/qun-liu/) · **Year:** 2020 · **Venue:** [Findings of the Association for Computational Linguistics: EMNLP 2020](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2020](https://md.research.elamur.ai/free/en/venues/findings/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.findings-emnlp.372/) · [PDF](https://aclanthology.org/2020.findings-emnlp.372.pdf) · [DOI 10.18653/v1/2020.findings-emnlp.372](https://doi.org/10.18653/v1/2020.findings-emnlp.372) · [arXiv 1909.10351](https://arxiv.org/abs/1909.10351) · [OpenAlex](https://openalex.org/W3105966348)

**Citations:** 1,729 citations (OpenAlex, 2026-10-14) · [649](https://md.research.elamur.ai/free/en/papers/jiao-2020-tinybert-distilling-bert-natural-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large pre-trained language models such as BERT achieve very high performance. However, they have many parameters and slow inference, which makes them difficult to run on phones and other low-resource devices. The authors compress BERT with a knowledge distillation method designed specifically for the Transformer architecture. The small "student" model TinyBERT learns to reproduce not only the final predictions of the "teacher". It also mimics the teacher's embedding layer and the hidden states and attention matrices of its intermediate layers. Training has two stages. First comes general distillation on a large unlabeled corpus, where the teacher is the original BERT without fine-tuning. Then comes task-specific distillation: the teacher is a fine-tuned BERT, and the training set is expanded through data augmentation. The four-layer TinyBERT4 retains more than 96.8% of the performance of BERT BASE on GLUE. At the same time, it is 7.5 times smaller and 9.4 times faster. The six-layer TinyBERT6 performs practically on par with its teacher. The work offers a practical way to obtain compact NLP models for systems with limited computational resources.

## Key points
- **Loss functions:** distillation operates at three levels. The student is fitted to the output of the embedding layer, to the hidden states and attention matrices of the Transformer layers (via MSE), and to the logits of the prediction layer. Attention is included because it carries syntactic and coreference information.
- **Two-stage framework:** general distillation yields a general-purpose General TinyBERT. It serves as the starting point for task-specific distillation on the augmented dataset. According to the authors, both stages are necessary for performance and generalization ability.
- **Comparison with other models:** TinyBERT4 has only about 13.3% of the parameters of BERT BASE and about 10.6% of its inference time. It clearly outperforms other four-layer distilled models (BERT-PKD, DistilBERT), despite having roughly 28% of their parameters and 31% of their inference time.
- **Choice of teacher layers:** the uniform strategy, in which the student draws on layers from the full depth of BERT, achieved an average score of 75.6 on the dev sets of MNLI, MRPC and CoLA. For strategies that use only the top or only the bottom layers, the scores are 70.9 and 71.3, respectively. Hence, different tasks require knowledge from different layers.
- **Resources:** the code and models are open-sourced in the `huawei-noah/Pretrained-Language-Model` repository on GitHub (TinyBERT folder). Among directions for future work, the authors name distillation from BERT LARGE and combining distillation with quantization or pruning.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Huawei Technologies](https://md.research.elamur.ai/free/en/companies/huawei-technologies/) — Company, Shenzhen, China · 96 works · 15,432 citations
- [Huazhong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/huazhong-university-of-science-and-technology/) — University, Wuhan, China · 59 works · 8,310 citations

Also on the byline: [Wuhan National Laboratory for Optoelectronics](https://ror.org/03c9ncn37).

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

💡 Concepts (1): [💡 Natural language understanding](https://md.research.elamur.ai/free/en/concepts/natural-language-understanding/) · 52 works

## Cite

Cite as: Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang and Qun Liu. 2020. TinyBERT: Distilling BERT for Natural Language Understanding. Findings of the Association for Computational Linguistics: EMNLP 2020.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{jiao2020tinybert,
  title = {{TinyBERT: Distilling BERT for Natural Language Understanding}},
  author = {Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang and Qun Liu},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.372},
  eprint = {1909.10351},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.372}
}
```

- [← previous Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://md.research.elamur.ai/free/en/papers/gu-2020-domain-specific-language-model-pretraining/)
- [next → Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://md.research.elamur.ai/free/en/papers/qi-2020-stanza-python-natural-language-processing/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/)

## References

42 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. L. Bentivogli, P. Clark, I. Dagan, and D. Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. [find](https://scholar.google.com/scholar?q=L%2E+Bentivogli%2C+P%2E+Clark%2C+I%2E+Dagan%2C+and+D%2E+Giampiccolo%2E+2009%2E+The+fifth+pascal+recognizing+textual+entailment+challenge%2E)
2. D. Cer, M. Diab, E. Agirre, I. Lopez-Gazpio, and L. Specia. 2017. [Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation](https://md.research.elamur.ai/free/en/papers/cer-2017-semeval-2017-task-1-semantic/). In Proceedings of the 11th International Workshop on Semantic Evaluation.
3. Z. Chen, H. Zhang, X. Zhang, and L. Zhao. 2018. Quora question pairs. [find](https://scholar.google.com/scholar?q=Z%2E+Chen%2C+H%2E+Zhang%2C+X%2E+Zhang%2C+and+L%2E+Zhao%2E+2018%2E+Quora+question+pairs%2E)
4. K. Clark, U. Khandelwal, O. Levy, and C. D. Manning. 2019. What does bert look at? an analysis of bert’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. [find](https://scholar.google.com/scholar?q=K%2E+Clark%2C+U%2E+Khandelwal%2C+O%2E+Levy%2C+and+C%2E+D%2E+Manning%2E+2019%2E+What+does+bert+look+at%3F+an+analysis+of+bert%E2%80%99s+attention%2E+In+Proceedings+of+the+2019+ACL+Workshop+BlackboxNLP%3A+Analyzing+and+Interpreting+Neur)
5. K. Clark, M. Luong, Q. V. Le, and C. D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In ICLR. [find](https://scholar.google.com/scholar?q=K%2E+Clark%2C+M%2E+Luong%2C+Q%2E+V%2E+Le%2C+and+C%2E+D%2E+Manning%2E+2020%2E+Electra%3A+Pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+In+ICLR%2E)
6. B. Cui, Y. Li, M. Chen, and Z. Zhang. 2019. Fine-tune bert with sparse self-attention mechanism. In EMNLP. [find](https://scholar.google.com/scholar?q=B%2E+Cui%2C+Y%2E+Li%2C+M%2E+Chen%2C+and+Z%2E+Zhang%2E+2019%2E+Fine%2Dtune+bert+with+sparse+self%2Dattention+mechanism%2E+In+EMNLP%2E)
7. M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and L. Kaiser. 2019. Universal transformers. In ICLR. [find](https://scholar.google.com/scholar?q=M%2E+Dehghani%2C+S%2E+Gouws%2C+O%2E+Vinyals%2C+J%2E+Uszkoreit%2C+and+L%2E+Kaiser%2E+2019%2E+Universal+transformers%2E+In+ICLR%2E)
8. G. Hinton, O. Vinyals, and J. Dean. 2015. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). arXiv preprint arXiv:1503.02531.
9. L. Hou, L. Shang, X. Jiang, and Q. Liu. 2020. Dyn-abert: Dynamic bert with adaptive width and depth. arXiv preprint arXiv:2004.04037. [find](https://scholar.google.com/scholar?q=L%2E+Hou%2C+L%2E+Shang%2C+X%2E+Jiang%2C+and+Q%2E+Liu%2E+2020%2E+Dyn%2Dabert%3A+Dynamic+bert+with+adaptive+width+and+depth%2E+arXiv+preprint+arXiv%3A2004%2E04037%2E)
10. M. Hu, Y. Peng, F. Wei, Z. Huang, D. Li, N. Yang, and M. Zhou. 2018. Attention-guided answer distillation for machine reading comprehension. In EMNLP. [find](https://scholar.google.com/scholar?q=M%2E+Hu%2C+Y%2E+Peng%2C+F%2E+Wei%2C+Z%2E+Huang%2C+D%2E+Li%2C+N%2E+Yang%2C+and+M%2E+Zhou%2E+2018%2E+Attention%2Dguided+answer+distillation+for+machine+reading+comprehension%2E+In+EMNLP%2E)
11. Y. Kim and A. M. Rush. 2016. [Sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/). In EMNLP.
12. O. Kovaleva, A. Romanov, A. Rogers, and A. Rumshisky. 2019. [Revealing the dark secrets of bert](https://md.research.elamur.ai/free/en/papers/kovaleva-2019-revealing-dark-secrets-bert/). In EMNLP.
13. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut. 2020. [Albert: A lite bert for self-supervised learning of language representations](https://md.research.elamur.ai/free/en/papers/lan-2019-albert-lite-bert-self-supervised/). In ICLR.
14. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+winograd+schema+challenge%2E+In+Thirteenth+International+Conference+on+the+Principles+of+Knowledge+Representation+and+Reasoning%2E)
15. W. Liu, P. Zhou, Z. Zhao, Z. Wang, H. Deng, and Q. Ju. 2020. [Fastbert: a self-distilling bert with adaptive inference time](https://md.research.elamur.ai/free/en/papers/liu-2020-fastbert-self-distilling-bert-adaptive/). arXiv preprint arXiv:2004.02178.
16. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692.
17. X. Ma, P. Zhang, S. Zhang, N. Duan, Y. Hou, M. Zhou, and D. Song. 2019. A tensorized transformer for language modeling. In NIPS. [find](https://scholar.google.com/scholar?q=X%2E+Ma%2C+P%2E+Zhang%2C+S%2E+Zhang%2C+N%2E+Duan%2C+Y%2E+Hou%2C+M%2E+Zhou%2C+and+D%2E+Song%2E+2019%2E+A+tensorized+transformer+for+language+modeling%2E+In+NIPS%2E)
18. J. S. McCarley. 2019. Pruning a bert-based question answering model. arXiv preprint arXiv:1910.06360. — [arXiv:1910.06360](https://arxiv.org/abs/1910.06360) [find](https://scholar.google.com/scholar?q=J%2E+S%2E+McCarley%2E+2019%2E+Pruning+a+bert%2Dbased+question+answering+model%2E+arXiv+preprint+arXiv%3A1910%2E06360%2E+%E2%80%94+%5BarXiv%3A1910%2E06360%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1910%2E06360%29)
19. P. Michel, O. Levy, and G. Neubig. 2019. [Are sixteen heads really better than one?](https://md.research.elamur.ai/free/en/papers/michel-2019-sixteen-heads-really-better-than/) In NIPS.
20. J. Pennington, R. Socher, and C. D. Manning. 2014. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In EMNLP.
21. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. 2019. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). arXiv preprint arXiv:1910.10683.
22. P. Rajpurkar, R. Jia, and P. Liang. 2018. [Know what you don’t know: Unanswerable questions for squad](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/). In ACL.
23. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In EMNLP.
24. A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio. 2014. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550. [find](https://scholar.google.com/scholar?q=A%2E+Romero%2C+N%2E+Ballas%2C+S%2E+E%2E+Kahou%2C+A%2E+Chassang%2C+C%2E+Gatta%2C+and+Y%2E+Bengio%2E+2014%2E+Fitnets%3A+Hints+for+thin+deep+nets%2E+arXiv+preprint+arXiv%3A1412%2E6550%2E)
25. H. Sajjad, F. Dalvi, N. Durrani, and P. Nakov. 2020. Poor man’s bert: Smaller and faster transformer models. arXiv preprint arXiv:2004.03844. — [arXiv:2004.03844](https://arxiv.org/abs/2004.03844) [find](https://scholar.google.com/scholar?q=H%2E+Sajjad%2C+F%2E+Dalvi%2C+N%2E+Durrani%2C+and+P%2E+Nakov%2E+2020%2E+Poor+man%E2%80%99s+bert%3A+Smaller+and+faster+transformer+models%2E+arXiv+preprint+arXiv%3A2004%2E03844%2E+%E2%80%94+%5BarXiv%3A2004%2E03844%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2004%2E03844%29)
26. V. Sanh, L. Debut, J. Chaumond, and T. Wolf. 2019. [Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter](https://md.research.elamur.ai/free/en/papers/sanh-2019-distilbert-distilled-version-bert-smaller/). arXiv preprint arXiv:1910.01108.
27. S. Shen, Z. Dong, J. Ye, L. Ma, Z. Yao, A. Gholami, M. W. Mahoney, and K. Keutzer. 2019. Q-bert: Hessian based ultra low precision quantization of bert. arXiv preprint arXiv:1909.05840. [find](https://scholar.google.com/scholar?q=S%2E+Shen%2C+Z%2E+Dong%2C+J%2E+Ye%2C+L%2E+Ma%2C+Z%2E+Yao%2C+A%2E+Gholami%2C+M%2E+W%2E+Mahoney%2C+and+K%2E+Keutzer%2E+2019%2E+Q%2Dbert%3A+Hessian+based+ultra+low+precision+quantization+of+bert%2E+arXiv+preprint+arXiv%3A1909%2E05840%2E)
28. R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts. 2013. [Recursive deep models for semantic compositionality over a sentiment treebank](https://md.research.elamur.ai/free/en/papers/socher-2013-recursive-deep-models-semantic-compositionality/). In EMNLP.
29. S. Sun, Y. Cheng, Z. Gan, and J. Liu. 2019. [Patient knowledge distillation for bert model compression](https://md.research.elamur.ai/free/en/papers/sun-2019-patient-knowledge-distillation-bert-model/). In EMNLP.
30. Z. Sun, H. Yu, X. Song, R. Liu, Y. Yang, and D. Zhou. 2020. [Mobilebert: a compact task-agnostic bert for resource-limited devices](https://md.research.elamur.ai/free/en/papers/sun-2020-mobilebert-compact-task-agnostic-bert/). arXiv preprint arXiv:2004.02984.
31. R. Tang, Y. Lu, L. Liu, L. Mou, O. Vechtomova, and J. Lin. 2019. Distilling task-specific knowledge from bert into simple neural networks. arXiv preprint arXiv:1903.12136. [find](https://scholar.google.com/scholar?q=R%2E+Tang%2C+Y%2E+Lu%2C+L%2E+Liu%2C+L%2E+Mou%2C+O%2E+Vechtomova%2C+and+J%2E+Lin%2E+2019%2E+Distilling+task%2Dspecific+knowledge+from+bert+into+simple+neural+networks%2E+arXiv+preprint+arXiv%3A1903%2E12136%2E)
32. I. Turc, M. Chang, K. Lee, and K. Toutanova. 2019. [Well-read students learn better: The impact of student initialization on knowledge distillation](https://md.research.elamur.ai/free/en/papers/turc-2019-well-read-students-learn-better/). arXiv preprint arXiv:1908.08962.
33. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NIPS.
34. E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov. 2019. [Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned](https://md.research.elamur.ai/free/en/papers/voita-2019-analyzing-multi-head-self-attention/). In ACL.
35. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman. 2018. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.
36. W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. arXiv preprint arXiv:2002.10957. [find](https://scholar.google.com/scholar?q=W%2E+Wang%2C+F%2E+Wei%2C+L%2E+Dong%2C+H%2E+Bao%2C+N%2E+Yang%2C+and+M%2E+Zhou%2E+2020%2E+Minilm%3A+Deep+self%2Dattention+distillation+for+task%2Dagnostic+compression+of+pre%2Dtrained+transformers%2E+arXiv+preprint+arXiv%3A2002%2E10957%2E)
37. A. Warstadt, A. Singh, and S. R. Bowman. 2019. Neural network acceptability judgments. TACL. [find](https://scholar.google.com/scholar?q=A%2E+Warstadt%2C+A%2E+Singh%2C+and+S%2E+R%2E+Bowman%2E+2019%2E+Neural+network+acceptability+judgments%2E+TACL%2E)
38. A. Williams, N. Nangia, and S. Bowman. 2018. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). In NAACL.
39. X. Wu, S. Lv, L. Zang, J. Han, and S. Hu. 2019. Conditional bert contextual augmentation. In International Conference on Computational Science. [find](https://scholar.google.com/scholar?q=X%2E+Wu%2C+S%2E+Lv%2C+L%2E+Zang%2C+J%2E+Han%2C+and+S%2E+Hu%2E+2019%2E+Conditional+bert+contextual+augmentation%2E+In+International+Conference+on+Computational+Science%2E)
40. C. Xu, W. Zhou, T. Ge, F. Wei, and M. Zhou. 2020. [Bert-of-theseus: Compressing bert by progressive module replacing](https://md.research.elamur.ai/free/en/papers/xu-2020-bert-theseus-compressing-bert-progressive/). arXiv preprint arXiv:2002.02925.
41. Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le. 2019. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). In NIPS.
42. O. Zafrir, G. Boudoukh, P. Izsak, and M. Wasserblat. 2019. Q8bert: Quantized 8bit bert. arXiv preprint arXiv:1910.06188. — [arXiv:1910.06188](https://arxiv.org/abs/1910.06188) [find](https://scholar.google.com/scholar?q=O%2E+Zafrir%2C+G%2E+Boudoukh%2C+P%2E+Izsak%2C+and+M%2E+Wasserblat%2E+2019%2E+Q8bert%3A+Quantized+8bit+bert%2E+arXiv+preprint+arXiv%3A1910%2E06188%2E+%E2%80%94+%5BarXiv%3A1910%2E06188%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1910%2E06188%29)

*Updated 2 November 2026*