---
title: "Patient Knowledge Distillation for BERT Model — Sun 2019"
description: "Pre-trained encoders like BERT are expensive to fine-tune and deploy."
authors: ["Siqi Sun", "Yu Cheng", "Zhe Gan", "Jingjing Liu"]
year: 2019
venue: "arXiv:1908.09355"
source: "https://arxiv.org/abs/1908.09355"
doi: "10.48550/arxiv.1908.09355"
arxiv: "1908.09355"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sun-2019-patient-knowledge-distillation-bert-model/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sun-2019-patient-knowledge-distillation-bert-model/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Patient Knowledge Distillation for BERT Model Compression

# Patient Knowledge Distillation for BERT Model Compression

**Authors:** [Siqi Sun](https://md.research.elamur.ai/free/en/authors/siqi-sun/), [Yu Cheng](https://md.research.elamur.ai/free/en/authors/yu-cheng/), [Zhe Gan](https://md.research.elamur.ai/free/en/authors/zhe-gan/), [Jingjing Liu](https://md.research.elamur.ai/free/en/authors/jingjing-liu/) · **Year:** 2019 · **Venue:** arXiv:1908.09355 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1908.09355) · [PDF](https://arxiv.org/pdf/1908.09355) · [DOI 10.48550/arxiv.1908.09355](https://doi.org/10.48550/arxiv.1908.09355) · [arXiv 1908.09355](https://arxiv.org/abs/1908.09355) · [OpenAlex](https://openalex.org/W2969515962)

**Citations:** 631 citations (openalex-snapshot · 2026-10-14) · [308](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Pre-trained encoders like BERT are expensive to fine-tune and deploy. BERT-Base alone stacks 12 Transformer layers with about 110M parameters. This paper asks how to shrink such a model into a shallower network while preserving task accuracy. The authors start from standard knowledge distillation, where a small student fits both gold labels and a fine-tuned teacher's temperature-softened predictions. They note that this setup plateaus early on test data, possibly from overfitting. Their remedy, Patient Knowledge Distillation (Patient-KD), also makes the student imitate the teacher's intermediate-layer representations. There are two layer-matching schemes. PKD-Skip aligns the student with every k-th teacher layer, and PKD-Last uses the teacher's final k layers. Students are initialized from the lower layers of BERT and tested on seven datasets spanning sentiment classification, paraphrase matching, NLI and machine reading comprehension. Patient-KD beats vanilla KD. It produces 6- and 3-layer models that are much smaller and faster, with accuracy the authors describe as comparable to the teacher. This matters for running pre-trained models on limited hardware.

## Key points
- Footprint: BERT 6 has 67.0M parameters (1.64× smaller than BERT-Base) and BERT 3 has 45.7M (2.40× smaller). The embedding layer stays fixed at 23.8M parameters.
- Inference speed is nearly linear in depth: 1.94× faster for BERT 6 and 3.73× for BERT 3. This was measured on 105k QNLI samples on one Titan RTX GPU (batch size 128, sequence length 128, FP16).
- Teacher size matters little. Distilling BERT 6 [Base] with vanilla KD from BERT 12 versus BERT 24 gives similar GLUE scores (e.g., SST-2 91.5 vs 91.2, MNLI-m 79.7 vs 80.2).
- A 6-layer student built from BERT-Large layers performs clearly worse. The authors attribute this to a harsher 4:1 compression ratio and an initialization mismatch.
- Replacing vanilla KD with PKD-Skip improves the BERT 6 [Large] student on almost all tasks. This suggests the method works regardless of which teacher is chosen.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

308 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 292 not yet.

- [MiniLLM: On-Policy Distillation of Large Language Models](https://md.research.elamur.ai/free/en/papers/gu-2023-minillm-policy-distillation-large-language/)
- [Parameter-Efficient Transfer Learning with Diff Pruning](https://md.research.elamur.ai/free/en/papers/guo-2021-parameter-efficient-transfer-learning-diff/)
- [I-BERT: Integer-only BERT Quantization](https://md.research.elamur.ai/free/en/papers/kim-2021-i-bert-integer-only-bert/)
- [DistiLLM: Towards Streamlined Distillation for Large Language Models](https://md.research.elamur.ai/free/en/papers/ko-2024-distillm-towards-streamlined-distillation-large/)
- [Improving Stance Detection with Multi-Dataset Learning and Knowledge Distillation](https://md.research.elamur.ai/free/en/papers/li-2021-improving-stance-detection-multi-dataset/)
- [Distilling Calibrated Knowledge for Stance Detection](https://md.research.elamur.ai/free/en/papers/li-2023-distilling-calibrated-knowledge-stance-detection/)
- [FastBERT: a Self-distilling BERT with Adaptive Inference Time](https://md.research.elamur.ai/free/en/papers/liu-2020-fastbert-self-distilling-bert-adaptive/)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://md.research.elamur.ai/free/en/papers/ma-2023-llm-pruner-structural-pruning-large/)
- [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/)
- [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://md.research.elamur.ai/free/en/papers/rajbhandari-2022-deepspeed-moe-advancing-mixture-experts/)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://md.research.elamur.ai/free/en/papers/sun-2020-mobilebert-compact-task-agnostic-bert/)
- [Well-Read Students Learn Better: On the Importance of Pre-training Compact Models](https://md.research.elamur.ai/free/en/papers/turc-2019-well-read-students-learn-better/)
- [Structured Pruning of Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2020-structured-pruning-large-language-models/)
- [Structured Pruning Learns Compact and Accurate Models](https://md.research.elamur.ai/free/en/papers/xia-2022-structured-pruning-learns-compact-accurate/)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://md.research.elamur.ai/free/en/papers/yao-2022-zeroquant-efficient-affordable-post-training/)
- [Terminology-Aware Retrieval-Augmented Knowledge Distillation for Biomedical Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/zafar-2026-terminology-aware-retrieval-augmented-knowledge/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{sun2019patient,
  title = {{Patient Knowledge Distillation for BERT Model Compression}},
  author = {Sun, Siqi and Cheng, Yu and Gan, Zhe and Liu, Jingjing},
  year = {2019},
  doi = {10.48550/arxiv.1908.09355},
  eprint = {1908.09355},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1908.09355}
}
```

- [← previous Passage Re-ranking with BERT](https://md.research.elamur.ai/free/en/papers/nogueira-2019-passage-re-ranking-bert/)
- [next → PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification](https://md.research.elamur.ai/free/en/papers/yang-2019-paws-x-cross-lingual-adversarial/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

40 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. TACL. [find](https://scholar.google.com/scholar?q=Piotr+Bojanowski%2C+Edouard+Grave%2C+Armand+Joulin%2C+and+Tomas+Mikolov%2E+2017%2E+Enriching+word+vectors+with+subword+information%2E+TACL%2E)
2. Soravit Changpinyo, Mark Sandler, and Andrey Zhmoginov. 2017. The power of sparsity in convolutional neural networks. arXiv preprint arXiv:1702.06257. — [arXiv:1702.06257](https://arxiv.org/abs/1702.06257) [find](https://scholar.google.com/scholar?q=Soravit+Changpinyo%2C+Mark+Sandler%2C+and+Andrey+Zhmoginov%2E+2017%2E+The+power+of+sparsity+in+convolutional+neural+networks%2E+arXiv+preprint+arXiv%3A1702%2E06257%2E+%E2%80%94+%5BarXiv%3A1702%2E06257%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1702%2E0)
3. Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. 2015. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv:1511.05641. — [arXiv:1511.05641](https://arxiv.org/abs/1511.05641) [find](https://scholar.google.com/scholar?q=Tianqi+Chen%2C+Ian+J%2E+Goodfellow%2C+and+Jonathon+Shlens%2E+2015%2E+Net2net%3A+Accelerating+learning+via+knowledge+transfer%2E+arXiv+preprint+arXiv%3A1511%2E05641%2E+%E2%80%94+%5BarXiv%3A1511%2E05641%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1511%2E05641)
4. Yu Cheng, Felix X. Yu, Rogerio S. Feris, Sanjiv Kumar, Alok Choudhary, and Shi-Fu Chang. 2015. An exploration of parameter redundancy in deep networks with circulant projections. In ICCV. [find](https://scholar.google.com/scholar?q=Yu+Cheng%2C+Felix+X%2E+Yu%2C+Rogerio+S%2E+Feris%2C+Sanjiv+Kumar%2C+Alok+Choudhary%2C+and+Shi%2DFu+Chang%2E+2015%2E+An+exploration+of+parameter+redundancy+in+deep+networks+with+circulant+projections%2E+In+ICCV%2E)
5. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In EMNLP. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Douwe+Kiela%2C+Holger+Schwenk%2C+Loic+Barrault%2C+and+Antoine+Bordes%2E+2017%2E+Supervised+learning+of+universal+sentence+representations+from+natural+language+inference+data%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/)
6. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In NIPS. [find](https://scholar.google.com/scholar?q=Andrew+M+Dai+and+Quoc+V+Le%2E+2015%2E+Semi%2Dsupervised+sequence+learning%2E+In+NIPS%2E)
7. Misha Denil, Babak Shakibi, Laurent Dinh, Marc’Aurelio Ranzato, and Nando de Freitas. 2013. Predicting parameters in deep learning. In NIPS. [find](https://scholar.google.com/scholar?q=Misha+Denil%2C+Babak+Shakibi%2C+Laurent+Dinh%2C+Marc%E2%80%99Aurelio+Ranzato%2C+and+Nando+de+Freitas%2E+2013%2E+Predicting+parameters+in+deep+learning%2E+In+NIPS%2E)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2E)
9. William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing. [find](https://scholar.google.com/scholar?q=William+B+Dolan+and+Chris+Brockett%2E+2005%2E+Automatically+constructing+a+corpus+of+sentential+paraphrases%2E+In+Proceedings+of+the+Third+International+Workshop+on+Paraphrasing%2E) [our card](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/)
10. Yunchao Gong, Liu Liu, Ming Yang, and Lubomir D. Bourdev. 2014. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115. — [arXiv:1412.6115](https://arxiv.org/abs/1412.6115) [find](https://scholar.google.com/scholar?q=Yunchao+Gong%2C+Liu+Liu%2C+Ming+Yang%2C+and+Lubomir+D%2E+Bourdev%2E+2014%2E+Compressing+deep+convolutional+networks+using+vector+quantization%2E+arXiv+preprint+arXiv%3A1412%2E6115%2E+%E2%80%94+%5BarXiv%3A1412%2E6115%5D%28https%3A%2F%2Farxiv%2Eorg)
11. Song Han, Huizi Mao, and William J Dally. 2016. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR. [find](https://scholar.google.com/scholar?q=Song+Han%2C+Huizi+Mao%2C+and+William+J+Dally%2E+2016%2E+Deep+compression%3A+Compressing+deep+neural+networks+with+pruning%2C+trained+quantization+and+huffman+coding%2E+In+ICLR%2E)
12. Song Han, Jeff Pool, John Tran, and William J. Dally. 2015. Learning both weights and connections for efficient neural networks. In NIPS. [find](https://scholar.google.com/scholar?q=Song+Han%2C+Jeff+Pool%2C+John+Tran%2C+and+William+J%2E+Dally%2E+2015%2E+Learning+both+weights+and+connections+for+efficient+neural+networks%2E+In+NIPS%2E)
13. Yihui He, Xiangyu Zhang, and Jian Sun. 2017. Channel pruning for accelerating very deep neural networks. In ICCV. [find](https://scholar.google.com/scholar?q=Yihui+He%2C+Xiangyu+Zhang%2C+and+Jian+Sun%2E+2017%2E+Channel+pruning+for+accelerating+very+deep+neural+networks%2E+In+ICCV%2E)
14. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. [find](https://scholar.google.com/scholar?q=Geoffrey+Hinton%2C+Oriol+Vinyals%2C+and+Jeff+Dean%2E+2015%2E+Distilling+the+knowledge+in+a+neural+network%2E+arXiv+preprint+arXiv%3A1503%2E02531%2E) [our card](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/)
15. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+ACL%2E)
16. Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. 2016. Deep networks with stochastic depth. In ECCV. [find](https://scholar.google.com/scholar?q=Gao+Huang%2C+Yu+Sun%2C+Zhuang+Liu%2C+Daniel+Sedra%2C+and+Kilian+Q%2E+Weinberger%2E+2016%2E+Deep+networks+with+stochastic+depth%2E+In+ECCV%2E)
17. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In NIPS. [find](https://scholar.google.com/scholar?q=Ryan+Kiros%2C+Yukun+Zhu%2C+Ruslan+R+Salakhutdinov%2C+Richard+Zemel%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+2015%2E+Skip%2Dthought+vectors%2E+In+NIPS%2E)
18. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683. [find](https://scholar.google.com/scholar?q=Guokun+Lai%2C+Qizhe+Xie%2C+Hanxiao+Liu%2C+Yiming+Yang%2C+and+Eduard+Hovy%2E+2017%2E+Race%3A+Large%2Dscale+reading+comprehension+dataset+from+examinations%2E+arXiv+preprint+arXiv%3A1704%2E04683%2E) [our card](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/)
19. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. Improving multi-task deep neural networks via knowledge distillation for natural language understanding. arXiv preprint arXiv:1904.09482. — [arXiv:1904.09482](https://arxiv.org/abs/1904.09482) [find](https://scholar.google.com/scholar?q=Xiaodong+Liu%2C+Pengcheng+He%2C+Weizhu+Chen%2C+and+Jianfeng+Gao%2E+2019a%2E+Improving+multi%2Dtask+deep+neural+networks+via+knowledge+distillation+for+natural+language+understanding%2E+arXiv+preprint+arXiv%3A1904%2E094)
20. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019b%2E+Roberta%3A+A+robustly+optimized+BERT+pretraining+approach%2E+)
21. Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In ICLR. [find](https://scholar.google.com/scholar?q=Lajanugen+Logeswaran+and+Honglak+Lee%2E+2018%2E+An+efficient+framework+for+learning+sentence+representations%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/logeswaran-2018-efficient-framework-learning-sentence-representations/)
22. Liang Lu, Michelle Guo, and Steve Renals. 2017. Knowledge distillation for small-footprint highway networks. In ICASSP. [find](https://scholar.google.com/scholar?q=Liang+Lu%2C+Michelle+Guo%2C+and+Steve+Renals%2E+2017%2E+Knowledge+distillation+for+small%2Dfootprint+highway+networks%2E+In+ICASSP%2E)
23. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In NIPS. [find](https://scholar.google.com/scholar?q=Bryan+McCann%2C+James+Bradbury%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2017%2E+Learned+in+translation%3A+Contextualized+word+vectors%2E+In+NIPS%2E) [our card](https://md.research.elamur.ai/free/en/papers/mccann-2017-learned-translation-contextualized-word-vectors/)
24. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Greg+S+Corrado%2C+and+Jeff+Dean%2E+2013%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+NIPS%2E)
25. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+EMNLP%2E)
26. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL. [find](https://scholar.google.com/scholar?q=Matthew+E+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+NAACL%2E)
27. Antonio Polino, Razvan Pascanu, and Dan Alistarh. 2018. Model compression via distillation and quantization. arXiv preprint arXiv:1802.05668. — [arXiv:1802.05668](https://arxiv.org/abs/1802.05668) [find](https://scholar.google.com/scholar?q=Antonio+Polino%2C+Razvan+Pascanu%2C+and+Dan+Alistarh%2E+2018%2E+Model+compression+via+distillation+and+quantization%2E+arXiv+preprint+arXiv%3A1802%2E05668%2E+%E2%80%94+%5BarXiv%3A1802%2E05668%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1802%2E05668%29)
28. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. arXiv. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+arXiv%2E)
29. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In EMNLP. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+Squad%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
30. Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2015. Fitnets: Hints for thin deep nets. In ICLR. [find](https://scholar.google.com/scholar?q=Adriana+Romero%2C+Nicolas+Ballas%2C+Samira+Ebrahimi+Kahou%2C+Antoine+Chassang%2C+Carlo+Gatta%2C+and+Yoshua+Bengio%2E+2015%2E+Fitnets%3A+Hints+for+thin+deep+nets%2E+In+ICLR%2E)
31. Bharat Bhusan Sau and Vineeth N Balasubramanian. 2016. Deep model compression: Distilling knowledge from noisy teachers. arXiv preprint arXiv:1610.09650. — [arXiv:1610.09650](https://arxiv.org/abs/1610.09650) [find](https://scholar.google.com/scholar?q=Bharat+Bhusan+Sau+and+Vineeth+N+Balasubramanian%2E+2016%2E+Deep+model+compression%3A+Distilling+knowledge+from+noisy+teachers%2E+arXiv+preprint+arXiv%3A1610%2E09650%2E+%E2%80%94+%5BarXiv%3A1610%2E09650%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F161)
32. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP. [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In+)
33. Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Tie-Yan Liu. 2019. Multilingual neural machine translation with knowledge distillation. In ICLR. [find](https://scholar.google.com/scholar?q=Xu+Tan%2C+Yi+Ren%2C+Di+He%2C+Tao+Qin%2C+Zhou+Zhao%2C+and+Tie%2DYan+Liu%2E+2019%2E+Multilingual+neural+machine+translation+with+knowledge+distillation%2E+In+ICLR%2E)
34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+NIPS%2E)
35. Dmitry P. Vetrov, Jonathan Huang, Li Zhang, Maxwell Collins, Michael Figurnov, Ruslan Salakhutdinov, and Yukun Zhu. 2017. Spatially adaptive computation time for residual networks. In CVPR. [find](https://scholar.google.com/scholar?q=Dmitry+P%2E+Vetrov%2C+Jonathan+Huang%2C+Li+Zhang%2C+Maxwell+Collins%2C+Michael+Figurnov%2C+Ruslan+Salakhutdinov%2C+and+Yukun+Zhu%2E+2017%2E+Spatially+adaptive+computation+time+for+residual+networks%2E+In+CVPR%2E)
36. Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amapreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R+Bowman%2E+2018%2E+Glue%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+arXiv+preprint+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
37. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+R+Bowman%2E+2017%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+arXiv+preprint+arXiv%3A1704%2E05426%2E)
38. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Ruslan+Salakhutdinov%2C+and+Quoc+V+Le%2E+2019%2E+Xlnet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+arXiv+preprint+arXiv%3A1906%2E08) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
39. Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Mark Zhang. 2016. Doubly convolutional neural networks. In NIPS. [find](https://scholar.google.com/scholar?q=Shuangfei+Zhai%2C+Yu+Cheng%2C+Weining+Lu%2C+and+Zhongfei+Mark+Zhang%2E+2016%2E+Doubly+convolutional+neural+networks%2E+In+NIPS%2E)
40. Chenguang Zhu, Michael Zeng, and Xuedong Huang. 2018. Sdnet: Contextualized attention-based deep network for conversational question answering. arXiv preprint arXiv:1812.03593. — [arXiv:1812.03593](https://arxiv.org/abs/1812.03593) [find](https://scholar.google.com/scholar?q=Chenguang+Zhu%2C+Michael+Zeng%2C+and+Xuedong+Huang%2E+2018%2E+Sdnet%3A+Contextualized+attention%2Dbased+deep+network+for+conversational+question+answering%2E+arXiv+preprint+arXiv%3A1812%2E03593%2E+%E2%80%94+%5BarXiv%3A1812%2E03593%5D%28ht)