---
title: "BERT-of-Theseus — paper by Xu et al. (2020)"
description: "Xu et al. propose Theseus Compression, named after the Ship of Theseus paradox. 47 references, cited by 71 papers here."
authors: ["Canwen Xu", "Wangchunshu Zhou", "Tao Ge", "Furu Wei", "Ming Zhou"]
year: 2020
venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/2020.emnlp-main.633/"
doi: "10.18653/v1/2020.emnlp-main.633"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2020-bert-theseus-compressing-bert-progressive/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2020-bert-theseus-compressing-bert-progressive/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) / BERT-of-Theseus: Compressing BERT by Progressive Module Replacing

# BERT-of-Theseus: Compressing BERT by Progressive Module Replacing

**Authors:** [Canwen Xu](https://md.research.elamur.ai/free/en/authors/canwen-xu/), [Wangchunshu Zhou](https://md.research.elamur.ai/free/en/authors/wangchunshu-zhou/), [Tao Ge](https://md.research.elamur.ai/free/en/authors/tao-ge/), [Furu Wei](https://md.research.elamur.ai/free/en/authors/furu-wei/) and [Ming Zhou](https://md.research.elamur.ai/free/en/authors/ming-zhou/) · **Year:** 2020 · **Venue:** [Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2020](https://md.research.elamur.ai/free/en/venues/emnlp/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-main.633/) · [PDF](https://aclanthology.org/2020.emnlp-main.633.pdf) · [DOI 10.18653/v1/2020.emnlp-main.633](https://doi.org/10.18653/v1/2020.emnlp-main.633) · [OpenAlex](https://openalex.org/W3101248447)

**Citations:** 168 citations (OpenAlex, 2026-10-14) · [71](https://md.research.elamur.ai/free/en/papers/xu-2020-bert-theseus-compressing-bert-progressive/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Large pretrained Transformers such as BERT are expensive to deploy because they use a lot of memory and run slowly. The usual way to shrink them, knowledge distillation, depends on carefully designed and balanced extra loss terms. Xu et al. propose Theseus Compression, named after the Ship of Theseus paradox. The original network (the "predecessor") is split into modules, and each module gets a smaller substitute (a "successor" module). For example, a group of several Transformer layers is mapped to a single layer.

During task fine-tuning, each predecessor module is randomly swapped for its successor. The resulting mixed networks are trained with only the normal task loss. A curriculum-style scheduler gradually raises the replacement probability. After training, the successor modules alone form the compact model.

On GLUE, a 6-layer BERT-of-Theseus runs almost twice as fast as BERT-base, keeps more than 98% of its quality, and beats distillation-based baselines. The approach is appealing because it needs no extra losses and no Transformer-specific machinery, so it could carry over to other architectures.

## Key points
- The 6-layer model is 1.94× faster than BERT-base. Its GLUE-dev macro score is 81.2, compared with 82.5 for BERT-base and 77.2 for simply fine-tuning a truncated 6-layer BERT.
- Stronger compression still beats truncated fine-tuning:
  - 4 layers (2.82× speedup): 77.2 vs 73.9.
  - 3 layers (3.66× speedup): 74.1 vs 71.9.
- The curriculum replacement scheduler outperforms both a constant rate and an anti-curriculum schedule. For example, on CoLA it scores 51.1, against 44.4 for a constant rate and 42.8 for anti-curriculum.
- With a fixed replacing rate, values of about 0.5–0.7 perform well on all GLUE tasks, while a very low rate (p = 0.1) clearly hurts.
- Code and the pretrained model are released at github.com/JetRunner/BERT-of-Theseus.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft Research Asia](https://md.research.elamur.ai/free/en/companies/microsoft-research-asia/) — Company, Beijing, China · 54 works · 59,921 citations
- [University of California San Diego](https://md.research.elamur.ai/free/en/universities/university-of-california-san-diego/) — University, San Diego, United States · 34 works · 51,167 citations
- [Beihang University](https://md.research.elamur.ai/free/en/universities/beihang-university/) — University, Beijing, China · 27 works · 8,089 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 77 works

## Cite

Cite as: Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei and Ming Zhou. 2020. BERT-of-Theseus: Compressing BERT by Progressive Module Replacing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xu2020bert,
  title = {{BERT-of-Theseus: Compressing BERT by Progressive Module Replacing}},
  author = {Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei and Ming Zhou},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.633},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.633}
}
```

- [← previous Analysing Lexical Semantic Change with Contextualised Word Representations](https://md.research.elamur.ai/free/en/papers/giulianelli-2020-analysing-lexical-semantic-change-contextualised/)
- [next → Curriculum Learning for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/xu-2020-curriculum-learning-natural-language-understanding/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/)

## References

47 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. [Curriculum learning](https://md.research.elamur.ai/free/en/papers/bengio-2009-curriculum-learning/). In ICML.
2. Alexis Conneau and Douwe Kiela. 2018. [Senteval: An evaluation toolkit for universal sentence representations](https://md.research.elamur.ai/free/en/papers/conneau-2018-senteval-evaluation-toolkit-universal-sentence/). In LREC.
3. Misha Denil, Babak Shakibi, Laurent Dinh, Marc’Aurelio Ranzato, and Nando de Freitas. 2013. Predicting parameters in deep learning. In NeurIPS. [find](https://scholar.google.com/scholar?q=Misha+Denil%2C+Babak+Shakibi%2C+Laurent+Dinh%2C+Marc%E2%80%99Aurelio+Ranzato%2C+and+Nando+de+Freitas%2E+2013%2E+Predicting+parameters+in+deep+learning%2E+In+NeurIPS%2E)
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In NAACL-HLT.
5. William B. Dolan and Chris Brockett. 2005. [Automatically constructing a corpus of sentential paraphrases](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/). In IWP@IJCNLP.
6. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. [Unified language model pre-training for natural language understanding and generation](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/). In NeurIPS.
7. Angela Fan, Edouard Grave, and Armand Joulin. 2020. [Reducing transformer depth on demand with structured dropout](https://md.research.elamur.ai/free/en/papers/fan-2019-reducing-transformer-depth-demand-structured/). In ICLR.
8. Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR. [find](https://scholar.google.com/scholar?q=Jonathan+Frankle+and+Michael+Carbin%2E+2019%2E+The+lottery+ticket+hypothesis%3A+Finding+sparse%2C+trainable+neural+networks%2E+In+ICLR%2E)
9. Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. 2018. Born-again neural networks. In ICML. [find](https://scholar.google.com/scholar?q=Tommaso+Furlanello%2C+Zachary+Chase+Lipton%2C+Michael+Tschannen%2C+Laurent+Itti%2C+and+Anima+Anandkumar%2E+2018%2E+Born%2Dagain+neural+networks%2E+In+ICML%2E)
10. Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. 2014. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115. — [arXiv:1412.6115](https://arxiv.org/abs/1412.6115) [find](https://scholar.google.com/scholar?q=Yunchao+Gong%2C+Liu+Liu%2C+Ming+Yang%2C+and+Lubomir+Bourdev%2E+2014%2E+Compressing+deep+convolutional+networks+using+vector+quantization%2E+arXiv+preprint+arXiv%3A1412%2E6115%2E+%E2%80%94+%5BarXiv%3A1412%2E6115%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
11. Song Han, Huizi Mao, and William J. Dally. 2016. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In ICLR. [find](https://scholar.google.com/scholar?q=Song+Han%2C+Huizi+Mao%2C+and+William+J%2E+Dally%2E+2016%2E+Deep+compression%3A+Compressing+deep+neural+network+with+pruning%2C+trained+quantization+and+huffman+coding%2E+In+ICLR%2E)
12. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+2016%2E+Deep+residual+learning+for+image+recognition%2E+In+CVPR%2E)
13. Yihui He, Xiangyu Zhang, and Jian Sun. 2017. Channel pruning for accelerating very deep neural networks. In ICCV. [find](https://scholar.google.com/scholar?q=Yihui+He%2C+Xiangyu+Zhang%2C+and+Jian+Sun%2E+2017%2E+Channel+pruning+for+accelerating+very+deep+neural+networks%2E+In+ICCV%2E)
14. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). arXiv preprint arXiv:1503.02531.
15. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351. [find](https://scholar.google.com/scholar?q=Xiaoqi+Jiao%2C+Yichun+Yin%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Xiao+Chen%2C+Linlin+Li%2C+Fang+Wang%2C+and+Qun+Liu%2E+2019%2E+Tinybert%3A+Distilling+bert+for+natural+language+understanding%2E+arXiv+preprint+arXiv%3A1909%2E10351%2E)
16. Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. [Reformer: The efficient transformer](https://md.research.elamur.ai/free/en/papers/kitaev-2020-reformer-efficient-transformer/). In ICLR.
17. Animesh Koratana, Daniel Kang, Peter Bailis, and Matei Zaharia. 2019. LIT: Learned intermediate representation training for model compression. In ICML. [find](https://scholar.google.com/scholar?q=Animesh+Koratana%2C+Daniel+Kang%2C+Peter+Bailis%2C+and+Matei+Zaharia%2E+2019%2E+LIT%3A+Learned+intermediate+representation+training+for+model+compression%2E+In+ICML%2E)
18. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In ICLR. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2020%2E+ALBERT%3A+A+Lite+BERT+for+Self%2Dsupervised+Learning+of+Language+Representations%2E+In+ICLR%2E)
19. Hector J. Levesque. 2011. The winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning. [find](https://scholar.google.com/scholar?q=Hector+J%2E+Levesque%2E+2011%2E+The+winograd+schema+challenge%2E+In+AAAI+Spring+Symposium%3A+Logical+Formalizations+of+Commonsense+Reasoning%2E)
20. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. Improving multi-task deep neural networks via knowledge distillation for natural language understanding. arXiv preprint arXiv:1904.09482. [find](https://scholar.google.com/scholar?q=Xiaodong+Liu%2C+Pengcheng+He%2C+Weizhu+Chen%2C+and+Jianfeng+Gao%2E+2019a%2E+Improving+multi%2Dtask+deep+neural+networks+via+knowledge+distillation+for+natural+language+understanding%2E+arXiv+preprint+arXiv%3A1904%2E094)
21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019b%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+)
22. Paul Michel, Omer Levy, and Graham Neubig. 2019. [Are sixteen heads really better than one?](https://md.research.elamur.ai/free/en/papers/michel-2019-sixteen-heads-really-better-than/) In NeurIPS.
23. Pietro Morerio, Jacopo Cavazza, Riccardo Volpi, René Vidal, and Vittorio Murino. 2017. Curriculum dropout. In ICCV. [find](https://scholar.google.com/scholar?q=Pietro+Morerio%2C+Jacopo+Cavazza%2C+Riccardo+Volpi%2C+Rene%CC%81+Vidal%2C+and+Vittorio+Murino%2E+2017%2E+Curriculum+dropout%2E+In+ICCV%2E)
24. Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. 2020. Deep double descent: Where bigger models and more data hurt. In ICLR. [find](https://scholar.google.com/scholar?q=Preetum+Nakkiran%2C+Gal+Kaplun%2C+Yamini+Bansal%2C+Tristan+Yang%2C+Boaz+Barak%2C+and+Ilya+Sutskever%2E+2020%2E+Deep+double+descent%3A+Where+bigger+models+and+more+data+hurt%2E+In+ICLR%2E)
25. Martin Popel and Ondřej Bojar. 2018. [Training tips for the transformer model](https://md.research.elamur.ai/free/en/papers/popel-2018-training-tips-transformer-model/). The Prague Bulletin of Mathematical Linguistics, 110(1):43–70.
26. Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020. [Intermediate-task transfer learning with pretrained language models: When and why does it work?](https://md.research.elamur.ai/free/en/papers/pruksachatkun-2020-intermediate-task-transfer-learning-pretrained/) In ACL.
27. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [Squad: 100, 000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In EMNLP.
28. Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2015. Fitnets: Hints for thin deep nets. In ICLR. [find](https://scholar.google.com/scholar?q=Adriana+Romero%2C+Nicolas+Ballas%2C+Samira+Ebrahimi+Kahou%2C+Antoine+Chassang%2C+Carlo+Gatta%2C+and+Yoshua+Bengio%2E+2015%2E+Fitnets%3A+Hints+for+thin+deep+nets%2E+In+ICLR%2E)
29. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+2019%2E+Distilbert%2C+a+distilled+version+of+bert%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E+arXiv+preprint+arXiv%3A1910%2E01108%2E)
30. Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. 2020. Q-BERT: hessian based ultra low precision quantization of BERT. In AAAI. [find](https://scholar.google.com/scholar?q=Sheng+Shen%2C+Zhen+Dong%2C+Jiayu+Ye%2C+Linjian+Ma%2C+Zhewei+Yao%2C+Amir+Gholami%2C+Michael+W%2E+Mahoney%2C+and+Kurt+Keutzer%2E+2020%2E+Q%2DBERT%3A+hessian+based+ultra+low+precision+quantization+of+BERT%2E+In+AAAI%2E)
31. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP. [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D%2E+Manning%2C+Andrew+Y%2E+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E)
32. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. [MASS: masked sequence to sequence pre-training for language generation](https://md.research.elamur.ai/free/en/papers/song-2019-mass-masked-sequence-sequence-pre/). In ICML.
33. Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, and Ming Zhou. 2020a. Scheduled drophead: A regularization method for transformer models. arXiv preprint arXiv:2004.13342. [find](https://scholar.google.com/scholar?q=Wangchunshu+Zhou%2C+Tao+Ge%2C+Ke+Xu%2C+Furu+Wei%2C+and+Ming+Zhou%2E+2020a%2E+Scheduled+drophead%3A+A+regularization+method+for+transformer+models%2E+arXiv+preprint+arXiv%3A2004%2E13342%2E)
34. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1929–1958. [find](https://scholar.google.com/scholar?q=Nitish+Srivastava%2C+Geoffrey+E%2E+Hinton%2C+Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Ruslan+Salakhutdinov%2E+2014%2E+Dropout%3A+a+simple+way+to+prevent+neural+networks+from+overfitting%2E+J%2E+Mach%2E+Learn%2E+Res%2E%2C+15%281%29%3A1)
35. Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. 2020b. Bert loses patience: Fast and robust inference with early exit. In NeurIPS. [find](https://scholar.google.com/scholar?q=Wangchunshu+Zhou%2C+Canwen+Xu%2C+Tao+Ge%2C+Julian+McAuley%2C+Ke+Xu%2C+and+Furu+Wei%2E+2020b%2E+Bert+loses+patience%3A+Fast+and+robust+inference+with+early+exit%2E+In+NeurIPS%2E)
36. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. [Patient knowledge distillation for BERT model compression](https://md.research.elamur.ai/free/en/papers/sun-2019-patient-knowledge-distillation-bert-model/). In EMNLP-IJCNLP.
37. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. [Mobilebert: a compact task-agnostic BERT for resource-limited devices](https://md.research.elamur.ai/free/en/papers/sun-2020-mobilebert-compact-task-agnostic-bert/). In ACL.
38. Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Tie-Yan Liu. 2019. Multilingual neural machine translation with knowledge distillation. In ICLR. [find](https://scholar.google.com/scholar?q=Xu+Tan%2C+Yi+Ren%2C+Di+He%2C+Tao+Qin%2C+Zhou+Zhao%2C+and+Tie%2DYan+Liu%2E+2019%2E+Multilingual+neural+machine+translation+with+knowledge+distillation%2E+In+ICLR%2E)
39. Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019. Distilling task-specific knowledge from bert into simple neural networks. arXiv preprint arXiv:1903.12136. [find](https://scholar.google.com/scholar?q=Raphael+Tang%2C+Yao+Lu%2C+Linqing+Liu%2C+Lili+Mou%2C+Olga+Vechtomova%2C+and+Jimmy+Lin%2E+2019%2E+Distilling+task%2Dspecific+knowledge+from+bert+into+simple+neural+networks%2E+arXiv+preprint+arXiv%3A1903%2E12136%2E)
40. Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Well-read students learn better: On the importance of pre-training compact models](https://md.research.elamur.ai/free/en/papers/turc-2019-well-read-students-learn-better/). arXiv preprint arXiv:1908.08962.
41. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NeurIPS.
42. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In ICLR.
43. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. TACL, 7:625–641. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R%2E+Bowman%2E+2019%2E+Neural+network+acceptability+judgments%2E+TACL%2C+7%3A625%E2%80%93641%2E)
44. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL-HLT. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+R%2E+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+NAACL%2DHLT%2E)
45. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). In NeurIPS.
46. Shuangfei Zhai, Yu Cheng, Zhongfei (Mark) Zhang, and Weining Lu. 2016. Doubly convolutional neural networks. In NeurIPS. [find](https://scholar.google.com/scholar?q=Shuangfei+Zhai%2C+Yu+Cheng%2C+Zhongfei+%28Mark%29+Zhang%2C+and+Weining+Lu%2E+2016%2E+Doubly+convolutional+neural+networks%2E+In+NeurIPS%2E)
47. Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. 2018. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR. [find](https://scholar.google.com/scholar?q=Xiangyu+Zhang%2C+Xinyu+Zhou%2C+Mengxiao+Lin%2C+and+Jian+Sun%2E+2018%2E+Shufflenet%3A+An+extremely+efficient+convolutional+neural+network+for+mobile+devices%2E+In+CVPR%2E)

*Updated 24 October 2026*