---
title: "Well-Read Students Learn Better — Turc 2019"
description: "Large pre-trained Transformers such as BERT perform well but are too slow and memory-hungry for many deployments."
authors: ["Iulia Turc", "Ming-Wei Chang", "Kenton Lee", "Kristina Toutanova"]
year: 2019
venue: "arXiv:1908.08962"
source: "https://arxiv.org/abs/1908.08962"
doi: "10.48550/arxiv.1908.08962"
arxiv: "1908.08962"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/turc-2019-well-read-students-learn-better/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/turc-2019-well-read-students-learn-better/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Well-Read Students Learn Better: On the Importance of Pre-training Compact Models

# Well-Read Students Learn Better: On the Importance of Pre-training Compact Models

**Authors:** [Iulia Turc](https://md.research.elamur.ai/free/en/authors/iulia-turc/), [Ming-Wei Chang](https://md.research.elamur.ai/free/en/authors/ming-wei-chang/), [Kenton Lee](https://md.research.elamur.ai/free/en/authors/kenton-lee/), [Kristina Toutanova](https://md.research.elamur.ai/free/en/authors/kristina-toutanova/) · **Year:** 2019 · **Venue:** arXiv:1908.08962 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1908.08962) · [PDF](https://arxiv.org/pdf/1908.08962) · [DOI 10.48550/arxiv.1908.08962](https://doi.org/10.48550/arxiv.1908.08962) · [arXiv 1908.08962](https://arxiv.org/abs/1908.08962) · [OpenAlex](https://openalex.org/W2975429091)

**Citations:** 398 citations (openalex-snapshot · 2026-10-14) · [273](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Large pre-trained Transformers such as BERT perform well but are too slow and memory-hungry for many deployments. Earlier compression work used elaborate schemes, while the obvious baseline was largely ignored: running ordinary masked-LM pre-training and fine-tuning directly on a small network. The authors test this baseline and find it competitive. They then extend it into Pre-trained Distillation (PD), a three-stage recipe:

1. MLM pre-training of a compact student on unlabeled general text.
2. Standard knowledge distillation from a large fine-tuned teacher (BERT BASE or BERT LARGE), using the teacher's soft labels on unlabeled task data.
3. An optional fine-tuning pass on labeled data.

On GLUE tasks, PD matches or beats more complex concurrent methods such as Patient-KD and DistilBERT, and it works with any architecture. Systematic experiments vary student size and the amount and domain similarity of the transfer data. Pre-training and distillation help each other even when both use the same data. The released compact models give practitioners strong starting points across a range of compute budgets.

## Key points
- **Released models:** 24 miniature BERT models, from about 4M parameters (L=2, H=128, "Tiny") to 110M (L=12, H=768, "Base"). The smallest runs roughly 65× faster than BERT LARGE on TPU v2.
- **Comparison with Patient-KD:** With 6-layer/768-hidden students, PD reaches a GLUE test meta score of 82.1, versus 81.7 for Patient-KD (Sun et al., 2019a) and 81.6 for plain pre-training+fine-tuning.
- **Comparison with DistilBERT:** On dev, PD averages 84.4, versus 82.3 for DistilBERT (Sanh, 2019).
- **Depth vs. width:** Pre-trained students gain more from added depth than from added width. Randomly initialized students show no such pattern.
- **Robustness and compounding:** PD tolerates smaller or less in-domain transfer sets better than distillation from scratch. Applying pre-training and then distillation to the same dataset outperforms either step alone.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

273 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 264 not yet.

- [Rethinking embedding coupling in pre-trained language models](https://md.research.elamur.ai/free/en/papers/chung-2020-rethinking-embedding-coupling-pre-trained/)
- [Linguistic Features in Automatic Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/ducret-2020-linguistic-features-automatic-sarcasm-detection/)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://md.research.elamur.ai/free/en/papers/fan-2019-reducing-transformer-depth-demand-structured/)
- [Parameter-Efficient Transfer Learning with Diff Pruning](https://md.research.elamur.ai/free/en/papers/guo-2021-parameter-efficient-transfer-learning-diff/)
- [Weakly-Supervised Modeling of Contextualized Event Embedding for Discourse Relations](https://md.research.elamur.ai/free/en/papers/lee-2020-weakly-supervised-modeling-contextualized-event/)
- [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://md.research.elamur.ai/free/en/papers/sun-2020-mobilebert-compact-task-agnostic-bert/)
- [Measuring and Reducing Gendered Correlations in Pre-trained Models](https://md.research.elamur.ai/free/en/papers/webster-2020-measuring-reducing-gendered-correlations-pre/)
- [LinkBERT: Pretraining Language Models with Document Links](https://md.research.elamur.ai/free/en/papers/yasunaga-2022-linkbert-pretraining-language-models-document/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{turc2019well,
  title = {{Well-Read Students Learn Better: On the Importance of Pre-training Compact Models}},
  author = {Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  year = {2019},
  doi = {10.48550/arxiv.1908.08962},
  eprint = {1908.08962},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1908.08962}
}
```

- [← previous VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://md.research.elamur.ai/free/en/papers/su-2019-vl-bert-pre-training-generic/)
- [next → What do you learn from context? Probing for sentence structure in contextualized word representations](https://md.research.elamur.ai/free/en/papers/tenney-2019-learn-context-probing-sentence-structure/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

41 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth pascal recognizing textual entailment challenge. In Proc Text Analysis Conference (TAC09, 2009. [find](https://scholar.google.com/scholar?q=Luisa+Bentivogli%2C+Ido+Dagan%2C+Hoa+Trang+Dang%2C+Danilo+Giampiccolo%2C+and+Bernardo+Magnini%2E+The+fifth+pascal+recognizing+textual+entailment+challenge%2E+In+Proc+Text+Analysis+Conference+%28TAC09%2C+2009%2E)
2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2015. [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Meth)
3. C Bucilă, R Caruana, and A Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 535– 541, 2006. [find](https://scholar.google.com/scholar?q=C+Bucila%CC%86%2C+R+Caruana%2C+and+A+Niculescu%2DMizil%2E+Model+compression%2E+In+Proceedings+of+the+12th+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining%2C+pp%2E+535%E2%80%93+541%2C+2006%2E)
4. Z. Chen, H. Zhang, X. Zhang, and L. Zhao. Quora question pairs. 2018. [find](https://scholar.google.com/scholar?q=Z%2E+Chen%2C+H%2E+Zhang%2C+X%2E+Zhang%2C+and+L%2E+Zhao%2E+Quora+question+pairs%2E+2018%2E)
5. Yew Ken Chia, Sam Witteveen, and Martin Andrews. Transformer to cnn: Label-scarce distillation for efficient text classification. 2018. [find](https://scholar.google.com/scholar?q=Yew+Ken+Chia%2C+Sam+Witteveen%2C+and+Martin+Andrews%2E+Transformer+to+cnn%3A+Label%2Dscarce+distillation+for+efficient+text+classification%2E+2018%2E)
6. Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoc V. Le. BAM! born-again multi-task networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for C… [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Urvashi+Khandelwal%2C+Christopher+D%2E+Manning%2C+and+Quoc+V%2E+Le%2E+BAM%21+born%2Dagain+multi%2Dtask+networks+for+natural+language+understanding%2E+In+Proceedings+of+the+57th+Annual+Mee)
7. Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In Advances in neural information processing systems, pp. 3079–3087, 2015. [find](https://scholar.google.com/scholar?q=Andrew+M+Dai+and+Quoc+V+Le%2E+Semi%2Dsupervised+sequence+learning%2E+In+Advances+in+neural+information+processing+systems%2C+pp%2E+3079%E2%80%933087%2C+2015%2E)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2C+2018%2E)
9. Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In ICML, 2015. [find](https://scholar.google.com/scholar?q=Suyog+Gupta%2C+Ankur+Agrawal%2C+Kailash+Gopalakrishnan%2C+and+Pritish+Narayanan%2E+Deep+learning+with+limited+numerical+precision%2E+In+ICML%2C+2015%2E)
10. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In NAACL, 2018. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Swabha+Swayamdipta%2C+Omer+Levy%2C+Roy+Schwartz%2C+Samuel+Bowman%2C+and+Noah+A%2E+Smith%2E+Annotation+artifacts+in+natural+language+inference+data%2E+In+NAACL%2C+2018%2E)
11. Suchin Gururangan, Tam Dang, Dallas Card, and Noah A. Smith. Variational pretraining for semi-supervised text classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Tam+Dang%2C+Dallas+Card%2C+and+Noah+A%2E+Smith%2E+Variational+pretraining+for+semi%2Dsupervised+text+classification%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computati)
12. Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In NIPS, 2016. [find](https://scholar.google.com/scholar?q=Song+Han%2C+Huizi+Mao%2C+and+William+J+Dally%2E+Deep+compression%3A+Compressing+deep+neural+networks+with+pruning%2C+trained+quantization+and+huffman+coding%2E+In+NIPS%2C+2016%2E)
13. Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pp. 507–517. International World Wide Web Conferences Steering Committee, 2016. [find](https://scholar.google.com/scholar?q=Ruining+He+and+Julian+McAuley%2E+Ups+and+downs%3A+Modeling+the+visual+evolution+of+fashion+trends+with+one%2Dclass+collaborative+filtering%2E+In+proceedings+of+the+25th+international+conference+on+world+wide+)
14. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. [find](https://scholar.google.com/scholar?q=Geoffrey+Hinton%2C+Oriol+Vinyals%2C+and+Jeff+Dean%2E+Distilling+the+knowledge+in+a+neural+network%2E+arXiv+preprint+arXiv%3A1503%2E02531%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/)
15. Minghao Hu, Yuxing Peng, Furu Wei, Zhen Huang, Dongsheng Li, Nan Yang, and Ming Zhou. Attention-guided answer distillation for machine reading comprehension. arXiv preprint arXiv:1808.07644, 2018. [find](https://scholar.google.com/scholar?q=Minghao+Hu%2C+Yuxing+Peng%2C+Furu+Wei%2C+Zhen+Huang%2C+Dongsheng+Li%2C+Nan+Yang%2C+and+Ming+Zhou%2E+Attention%2Dguided+answer+distillation+for+machine+reading+comprehension%2E+arXiv+preprint+arXiv%3A1808%2E07644%2C+2018%2E)
16. Johansson, Stig, and Knut Hofland. Frequency Analysis of English vocabulary and grammar, based on the LOB corpus. Claredon, Oxford, 1989. [find](https://scholar.google.com/scholar?q=Johansson%2C+Stig%2C+and+Knut+Hofland%2E+Frequency+Analysis+of+English+vocabulary+and+grammar%2C+based+on+the+LOB+corpus%2E+Claredon%2C+Oxford%2C+1989%2E)
17. Rie Johnson and Tong Zhang. Semi-supervised convolutional neural networks for text categorization via region embedding. In Advances in neural information processing systems, pp. 919–927, 2015. [find](https://scholar.google.com/scholar?q=Rie+Johnson+and+Tong+Zhang%2E+Semi%2Dsupervised+convolutional+neural+networks+for+text+categorization+via+region+embedding%2E+In+Advances+in+neural+information+processing+systems%2C+pp%2E+919%E2%80%93927%2C+2015%2E)
18. Adam Kilgarriff and Tony Rose. Measures for corpus similarity and homogeneity. In Proceedings of the Third Conference on Empirical Methods for Natural Language Processing, pp. 46–52, 1998. [find](https://scholar.google.com/scholar?q=Adam+Kilgarriff+and+Tony+Rose%2E+Measures+for+corpus+similarity+and+homogeneity%2E+In+Proceedings+of+the+Third+Conference+on+Empirical+Methods+for+Natural+Language+Processing%2C+pp%2E+46%E2%80%9352%2C+1998%2E)
19. Young-Bum Kim, Karl Stratos, and Dongchan Kim. Adversarial adaptation of synthetic or stale data. In ACL, 2017. [find](https://scholar.google.com/scholar?q=Young%2DBum+Kim%2C+Karl+Stratos%2C+and+Dongchan+Kim%2E+Adversarial+adaptation+of+synthetic+or+stale+data%2E+In+ACL%2C+2017%2E)
20. Akisato Kimura, Zoubin Ghahramani, Koh Takeuchi, Tomoharu Iwata, and Naonori Ueda. Few-shot learning of neural networks from scratch by pseudo example optimization. arXiv preprint arXiv:1802.03039, 2018. — [arXiv:1802.03039](https://arxiv.org/abs/1802.03039) [find](https://scholar.google.com/scholar?q=Akisato+Kimura%2C+Zoubin+Ghahramani%2C+Koh+Takeuchi%2C+Tomoharu+Iwata%2C+and+Naonori+Ueda%2E+Few%2Dshot+learning+of+neural+networks+from+scratch+by+pseudo+example+optimization%2E+arXiv+preprint+arXiv%3A1802%2E03039%2C+20)
21. Jinyu Li, Rui Zhao, Jui-Ting Huang, and Yifan Gong. Learning small-size dnn with output-distribution-based criteria. In Fifteenth annual conference of the international speech communication association, 2014. [find](https://scholar.google.com/scholar?q=Jinyu+Li%2C+Rui+Zhao%2C+Jui%2DTing+Huang%2C+and+Yifan+Gong%2E+Learning+small%2Dsize+dnn+with+output%2Ddistribution%2Dbased+criteria%2E+In+Fifteenth+annual+conference+of+the+international+speech+communication+associatio)
22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2C+2019%2E)
23. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111–3119, 2013. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Greg+S+Corrado%2C+and+Jeff+Dean%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+Advances+in+neural+information+processing+system)
24. Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532–1543, 2014. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+Manning%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+2014+conference+on+empirical+methods+in+natural+language+processing+%28E)
25. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018. [find](https://scholar.google.com/scholar?q=Matthew+E+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+Deep+contextualized+word+representations%2E+arXiv+preprint+arXiv%3A1802%2E05365%2C+2018%2E)
26. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL [https://s3-us-west-2.amazonaws](https://s3-us-west-2.amazonaws). com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf, 2018. [↗](https://s3-us-west-2.amazonaws)
27. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. URL [https://openai.com/blog/better-language-models](https://openai.com/blog/better-language-models), 2019. [↗](https://openai.com/blog/better-language-models)
28. Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014. [find](https://scholar.google.com/scholar?q=Adriana+Romero%2C+Nicolas+Ballas%2C+Samira+Ebrahimi+Kahou%2C+Antoine+Chassang%2C+Carlo+Gatta%2C+and+Yoshua+Bengio%2E+Fitnets%3A+Hints+for+thin+deep+nets%2E+arXiv+preprint+arXiv%3A1412%2E6550%2C+2014%2E)
29. Victor Sanh. Smaller, faster, cheaper, lighter: Introducing DistilBERT, a distilled version of BERT. [https://medium.com/huggingface/distilbert-8cf3380435b5](https://medium.com/huggingface/distilbert-8cf3380435b5), 2019. [↗](https://medium.com/huggingface/distilbert-8cf3380435b5)
30. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631–1642, 201… [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In+Procee)
31. Spearman. The proof and measurement of association between two things. In American Journal of Psychology., pp. 72–101, 1904. [find](https://scholar.google.com/scholar?q=Spearman%2E+The+proof+and+measurement+of+association+between+two+things%2E+In+American+Journal+of+Psychology%2E%2C+pp%2E+72%E2%80%93101%2C+1904%2E)
32. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint arXiv:1908.09355, 2019a. [find](https://scholar.google.com/scholar?q=Siqi+Sun%2C+Yu+Cheng%2C+Zhe+Gan%2C+and+Jingjing+Liu%2E+Patient+knowledge+distillation+for+bert+model+compression%2E+arXiv+preprint+arXiv%3A1908%2E09355%2C+2019a%2E) [our card](https://md.research.elamur.ai/free/en/papers/sun-2019-patient-knowledge-distillation-bert-model/)
33. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. Ernie 2.0: A continual pre-training framework for language understanding, 2019b. [find](https://scholar.google.com/scholar?q=Yu+Sun%2C+Shuohuan+Wang%2C+Yukun+Li%2C+Shikun+Feng%2C+Hao+Tian%2C+Hua+Wu%2C+and+Haifeng+Wang%2E+Ernie+2%2E0%3A+A+continual+pre%2Dtraining+framework+for+language+understanding%2C+2019b%2E) [our card](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-2-0-continual/)
34. Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task-specific knowledge from bert into simple neural networks, 2019. [find](https://scholar.google.com/scholar?q=Raphael+Tang%2C+Yao+Lu%2C+Linqing+Liu%2C+Lili+Mou%2C+Olga+Vechtomova%2C+and+Jimmy+Lin%2E+Distilling+task%2Dspecific+knowledge+from+bert+into+simple+neural+networks%2C+2019%2E)
35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998–6008, 2017. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+Attention+is+all+you+need%2E+In+Advances+in+neural+information+processing+sys)
36. Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amapreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R+Bowman%2E+Glue%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+arXiv+preprint+arXiv%3A1804%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
37. Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers),… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of)
38. Ze Yang, Linjun Shou, Ming Gong, Wutao Lin, and Daxin Jiang. Model compression with multi-task knowledge distillation for web-scale question answering system. arXiv preprint arXiv:1904.09636, 2019a. — [arXiv:1904.09636](https://arxiv.org/abs/1904.09636) [find](https://scholar.google.com/scholar?q=Ze+Yang%2C+Linjun+Shou%2C+Ming+Gong%2C+Wutao+Lin%2C+and+Daxin+Jiang%2E+Model+compression+with+multi%2Dtask+knowledge+distillation+for+web%2Dscale+question+answering+system%2E+arXiv+preprint+arXiv%3A1904%2E09636%2C+2019a%2E+%E2%80%94)
39. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237, 2019b. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Ruslan+Salakhutdinov%2C+and+Quoc+V+Le%2E+Xlnet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+arXiv+preprint+arXiv%3A1906%2E08237%2C+2) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
40. Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. [find](https://scholar.google.com/scholar?q=Junho+Yim%2C+Donggyu+Joo%2C+Jihoon+Bae%2C+and+Junmo+Kim%2E+A+gift+from+knowledge+distillation%3A+Fast+optimization%2C+network+minimization+and+transfer+learning%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+)
41. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pp. 19–27, 2015… [find](https://scholar.google.com/scholar?q=Yukun+Zhu%2C+Ryan+Kiros%2C+Rich+Zemel%2C+Ruslan+Salakhutdinov%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+Aligning+books+and+movies%3A+Towards+story%2Dlike+visual+explanations+by+watching+movies+and+re) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/)