---
title: "Compressing BERT — Gordon 2020"
description: "Pre-trained encoders like BERT transfer well to new tasks, but they are too large for many deployment settings, such as phones or consumer GPUs."
authors: ["Mitchell Gordon", "Kevin Duh", "Nicholas Andrews"]
year: 2020
venue: "Proceedings of the 5th Workshop on Representation Learning for NLP"
source: "https://aclanthology.org/2020.repl4nlp-1.18/"
doi: "10.18653/v1/2020.repl4nlp-1.18"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gordon-2020-compressing-bert-studying-effects-weight/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gordon-2020-compressing-bert-studying-effects-weight/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/) / Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning

# Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning

**Authors:** [Mitchell Gordon](https://md.research.elamur.ai/free/en/authors/mitchell-gordon/), [Kevin Duh](https://md.research.elamur.ai/free/en/authors/kevin-duh/), [Nicholas Andrews](https://md.research.elamur.ai/free/en/authors/nicholas-andrews/) · **Year:** 2020 · **Venue:** Proceedings of the 5th Workshop on Representation Learning for NLP · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.repl4nlp-1.18/) · [PDF](https://aclanthology.org/2020.repl4nlp-1.18.pdf) · [DOI 10.18653/v1/2020.repl4nlp-1.18](https://doi.org/10.18653/v1/2020.repl4nlp-1.18) · [OpenAlex](https://openalex.org/W2997710335)

**Citations:** 297 citations (openalex-snapshot · 2026-10-14) · [103](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Pre-trained encoders like BERT transfer well to new tasks, but they are too large for many deployment settings, such as phones or consumer GPUs. The authors ask whether compressing BERT before it is distributed hurts its ability to transfer, and whether compressing it after task-specific fine-tuning works better. They applied magnitude weight pruning to English BERT-Base at sparsity levels from 0% to 90%. After pruning, they continued pre-training on English Wikipedia and BookCorpus. They then fine-tuned each pruned model on several GLUE tasks.

The results fall into three regimes:
- **Light pruning (30–40%)** leaves both pre-training loss and downstream accuracy unchanged.
- **Moderate pruning** raises pre-training loss and blocks useful pre-trained knowledge from reaching the tasks. Tasks suffer to different degrees.
- **Heavy pruning** also limits the model's capacity to fit the downstream data.

Pruning after fine-tuning gave no advantage over pruning during pre-training. The practical conclusion is that one task-agnostic pruned BERT can be shared across many tasks. This avoids repeated per-task compression and retraining.

## Key points
- **Pruning setup:**
  - Target sparsity ranged from 0–90% in steps of 10%.
  - Pruning was applied gradually over the first 10k steps, followed by 90k more pre-training steps on English Wikipedia + BookCorpus.
  - Thresholds were computed per weight matrix (matrix-local pruning), using the TensorFlow model_pruning package.
- **What was pruned:** the attention projections (key, query, value, output), the feed-forward matrices and the sub-word embeddings.
  - Embeddings alone take about 21% of BERT-Base memory.
  - Feed-forward layers and embeddings together take about 72%.
- **Evaluation:**
  - Five GLUE tasks were used. WNLI was excluded, as were tasks with fewer than 5k training examples (RTE, MRPC, STS-B).
  - Each task was fine-tuned for 3 epochs, with 4 learning rates tried (2–5 × 10⁻⁵).
- **Accuracy vs. loss:** downstream accuracy drops roughly linearly as pre-training loss rises, with a different slope for each task.
- **Effect of fine-tuning on pruning:**
  - Fine-tuning alters the set of pruned weights by under 6%.
  - On average, weights shift only 0–4% in the magnitude ranking.
  - At 60% sparsity, every attention head individually ends up 55–65% pruned.
- **Code:** released as `bert-prune` on GitHub.

## Interesting ideas
- **Separating two effects of pruning.** The authors run a control in which weights are set to zero but are allowed to train again downstream. This separates losing pre-trained information from losing model capacity. The same control could test any compression method on other pre-trained models.
- **Measuring stability of the weight ranking.** They track how far each weight moves in the magnitude ordering during fine-tuning. This is a cheap way to check whether task-specific training could change what gets pruned, before running costly prune-and-retrain loops.
- **Pruning embeddings along with other layers.** Sub-word embeddings are sparsified and stored as sparse matrices, just like the other layers. This is relevant for vocabulary-heavy models, where embeddings take a large share of memory.
- **Compressing once, before distribution.** A single pruned checkpoint is produced during pre-training rather than one per task. This approach is attractive wherever one base model serves many downstream applications.
- **Using pre-training loss as a predictor.** Because accuracy falls roughly linearly with pre-training loss, that loss can serve as a proxy for how much transfer quality a compression level will cost. This helps choose an accuracy/memory trade-off without fine-tuning on every task.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

103 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 96 not yet.

- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://md.research.elamur.ai/free/en/papers/ben-2021-zaken-bitfit-simple-parameter-efficient-fine/)
- [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
- [Parameter-Efficient Transfer Learning with Diff Pruning](https://md.research.elamur.ai/free/en/papers/guo-2021-parameter-efficient-transfer-learning-diff/)
- [I-BERT: Integer-only BERT Quantization](https://md.research.elamur.ai/free/en/papers/kim-2021-i-bert-integer-only-bert/)
- [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://md.research.elamur.ai/free/en/papers/yao-2022-zeroquant-efficient-affordable-post-training/)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://md.research.elamur.ai/free/en/papers/ye-2022-zerogen-efficient-zero-shot-learning/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{gordon2020compressing,
  title = {{Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning}},
  author = {Gordon, Mitchell and Duh, Kevin and Andrews, Nicholas},
  booktitle = {Proceedings of the 5th Workshop on Representation Learning for NLP},
  year = {2020},
  doi = {10.18653/v1/2020.repl4nlp-1.18},
  url = {https://doi.org/10.18653/v1/2020.repl4nlp-1.18}
}
```

- [← previous Comparison by Conversion: Reverse-Engineering UCCA from Syntax and Lexical Semantics](https://md.research.elamur.ai/free/en/papers/hershcovich-2020-comparison-conversion-reverse-engineering-ucca/)
- [next → Conformer: Convolution-augmented Transformer for Speech Recognition](https://md.research.elamur.ai/free/en/papers/gulati-2020-conformer-convolution-augmented-transformer-speech/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/)

## References

36 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Martı́n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh… [find](https://scholar.google.com/scholar?q=Mart%C4%B1%CC%81n+Abadi%2C+Ashish+Agarwal%2C+Paul+Barham%2C+Eugene+Brevdo%2C+Zhifeng+Chen%2C+Craig+Citro%2C+Gregory+S%2E+Corrado%2C+Andy+Davis%2C+Jeffrey+Dean%2C+Matthieu+Devin%2C+Sanjay+Ghemawat%2C+Ian+J%2E+Goodfellow%2C+Andrew+Harp%2C+Geo) [our card](https://md.research.elamur.ai/free/en/papers/abadi-2016-tensorflow-large-scale-machine-learning/)
2. Jay Alammar. 2018. The illustrated transformer. [find](https://scholar.google.com/scholar?q=Jay+Alammar%2E+2018%2E+The+illustrated+transformer%2E)
3. Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. 2018. Stronger generalization bounds for deep nets via a compression approach. CoRR, abs/1802.05296. — [arXiv:1802.05296](https://arxiv.org/abs/1802.05296) [find](https://scholar.google.com/scholar?q=Sanjeev+Arora%2C+Rong+Ge%2C+Behnam+Neyshabur%2C+and+Yi+Zhang%2E+2018%2E+Stronger+generalization+bounds+for+deep+nets+via+a+compression+approach%2E+CoRR%2C+abs%2F1802%2E05296%2E+%E2%80%94+%5BarXiv%3A1802%2E05296%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
4. Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mand al. 2018. Reconciling modern machine learning practice and the bias-variance trade-off. arXiv e-prints, page arXiv:1812.11118. — [arXiv:1812.11118](https://arxiv.org/abs/1812.11118) [find](https://scholar.google.com/scholar?q=Mikhail+Belkin%2C+Daniel+Hsu%2C+Siyuan+Ma%2C+and+Soumik+Mand+al%2E+2018%2E+Reconciling+modern+machine+learning+practice+and+the+bias%2Dvariance+trade%2Doff%2E+arXiv+e%2Dprints%2C+page+arXiv%3A1812%2E11118%2E+%E2%80%94+%5BarXiv%3A1812%2E1111)
5. Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model compression. In Proceedings of the Twelfth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Philadelphia, PA, USA, August 20-23, 2006, pages 535–541. [find](https://scholar.google.com/scholar?q=Cristian+Bucila%2C+Rich+Caruana%2C+and+Alexandru+Niculescu%2DMizil%2E+2006%2E+Model+compression%2E+In+Proceedings+of+the+Twelfth+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining%2C+Philade)
6. William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, and Jakob Uszkoreit. 2019. KERMIT: generative insertion-based modeling for sequences. CoRR, abs/1906.01604. — [arXiv:1906.01604](https://arxiv.org/abs/1906.01604) [find](https://scholar.google.com/scholar?q=William+Chan%2C+Nikita+Kitaev%2C+Kelvin+Guu%2C+Mitchell+Stern%2C+and+Jakob+Uszkoreit%2E+2019%2E+KERMIT%3A+generative+insertion%2Dbased+modeling+for+sequences%2E+CoRR%2C+abs%2F1906%2E01604%2E+%E2%80%94+%5BarXiv%3A1906%2E01604%5D%28https%3A%2F%2Farxiv%2E)
7. Tim Dettmers and Luke S. Zettlemoyer. 2019. Sparse networks from scratch: Faster training without losing performance. ArXiv, abs/1907.04840. — [arXiv:1907.04840](https://arxiv.org/abs/1907.04840) [find](https://scholar.google.com/scholar?q=Tim+Dettmers+and+Luke+S%2E+Zettlemoyer%2E+2019%2E+Sparse+networks+from+scratch%3A+Faster+training+without+losing+performance%2E+ArXiv%2C+abs%2F1907%2E04840%2E+%E2%80%94+%5BarXiv%3A1907%2E04840%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1907%2E04840%29)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+CoRR%2C+abs%2F1810%2E04805%2E)
9. Utku Evci, Fabian Pedregosa, Aidan N. Gomez, and Erich Elsen. 2019. The difficulty of training sparse neural networks. CoRR, abs/1906.10732. — [arXiv:1906.10732](https://arxiv.org/abs/1906.10732) [find](https://scholar.google.com/scholar?q=Utku+Evci%2C+Fabian+Pedregosa%2C+Aidan+N%2E+Gomez%2C+and+Erich+Elsen%2E+2019%2E+The+difficulty+of+training+sparse+neural+networks%2E+CoRR%2C+abs%2F1906%2E10732%2E+%E2%80%94+%5BarXiv%3A1906%2E10732%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1906%2E10732%29)
10. Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jonathan+Frankle+and+Michael+Carbin%2E+2019%2E+The+lottery+ticket+hypothesis%3A+Finding+sparse%2C+trainable+neural+networks%2E+In+International+Conference+on+Learning+Representations%2E)
11. Trevor Gale, Erich Elsen, and Sara Hooker. 2019. The state of sparsity in deep neural networks. CoRR, abs/1902.09574. — [arXiv:1902.09574](https://arxiv.org/abs/1902.09574) [find](https://scholar.google.com/scholar?q=Trevor+Gale%2C+Erich+Elsen%2C+and+Sara+Hooker%2E+2019%2E+The+state+of+sparsity+in+deep+neural+networks%2E+CoRR%2C+abs%2F1902%2E09574%2E+%E2%80%94+%5BarXiv%3A1902%2E09574%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1902%2E09574%29)
12. Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A. Horowitz, and William J. Dally. 2016. Eie: Efficient inference engine on compressed deep neural network. In Proceedings of the 43rd International Symposium on Computer Architecture, ISCA ’16, pages 243–254, Piscataway, NJ, USA. IEEE P… [find](https://scholar.google.com/scholar?q=Song+Han%2C+Xingyu+Liu%2C+Huizi+Mao%2C+Jing+Pu%2C+Ardavan+Pedram%2C+Mark+A%2E+Horowitz%2C+and+William+J%2E+Dally%2E+2016%2E+Eie%3A+Efficient+inference+engine+on+compressed+deep+neural+network%2E+In+Proceedings+of+the+43rd+In)
13. Song Han, Jeff Pool, John Tran, and William Dally. 2015. Learning both weights and connections for efficient neural network. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 1135–1143. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Song+Han%2C+Jeff+Pool%2C+John+Tran%2C+and+William+Dally%2E+2015%2E+Learning+both+weights+and+connections+for+efficient+neural+network%2E+In+C%2E+Cortes%2C+N%2E+D%2E+Lawrence%2C+D%2E+D%2E+Lee%2C+M%2E+Sugiyama%2C+and+R%2E+Garnett%2C+edito)
14. Dan Hendrycks, Kimin Lee, and Mantas Mazeika. 2019. Using pre-training can improve model robustness and uncertainty. In ICML, pages 2712–2721. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Kimin+Lee%2C+and+Mantas+Mazeika%2E+2019%2E+Using+pre%2Dtraining+can+improve+model+robustness+and+uncertainty%2E+In+ICML%2C+pages+2712%E2%80%932721%2E)
15. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. In Proc. ACL. [find](https://scholar.google.com/scholar?q=Guillaume+Klein%2C+Yoon+Kim%2C+Yuntian+Deng%2C+Jean+Senellart%2C+and+Alexander+M%2E+Rush%2E+2017%2E+Opennmt%3A+Open%2Dsource+toolkit+for+neural+machine+translation%2E+In+Proc%2E+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/klein-2017-opennmt-open-source-toolkit-neural/)
16. Nimit Sharad Sohoni, Christopher Richard Aberger, Megan Leszczynski, Jian Zhang, and Christopher Ré. 2019. Low-memory neural network training: A technical report. CoRR, abs/1904.10631. — [arXiv:1904.10631](https://arxiv.org/abs/1904.10631) [find](https://scholar.google.com/scholar?q=Nimit+Sharad+Sohoni%2C+Christopher+Richard+Aberger%2C+Megan+Leszczynski%2C+Jian+Zhang%2C+and+Christopher+Re%CC%81%2E+2019%2E+Low%2Dmemory+neural+network+training%3A+A+technical+report%2E+CoRR%2C+abs%2F1904%2E10631%2E+%E2%80%94+%5BarXiv%3A1904%2E)
17. Adhiguna Kuncoro, Chris Dyer, Laura Rimell, Stephen Clark, and Phil Blunsom. 2019. Scalable syntax-aware language models using knowledge distillation. CoRR, abs/1906.06438. [find](https://scholar.google.com/scholar?q=Adhiguna+Kuncoro%2C+Chris+Dyer%2C+Laura+Rimell%2C+Stephen+Clark%2C+and+Phil+Blunsom%2E+2019%2E+Scalable+syntax%2Daware+language+models+using+knowledge+distillation%2E+CoRR%2C+abs%2F1906%2E06438%2E)
18. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. CoRR, abs/1906.02243. [find](https://scholar.google.com/scholar?q=Emma+Strubell%2C+Ananya+Ganesh%2C+and+Andrew+McCallum%2E+2019%2E+Energy+and+policy+considerations+for+deep+learning+in+NLP%2E+CoRR%2C+abs%2F1906%2E02243%2E) [our card](https://md.research.elamur.ai/free/en/papers/strubell-2019-energy-policy-considerations-deep-learning/)
19. Xiang Lisa Li and Jason Eisner. 2019. Specializing word embeddings (for parsing) by information bottleneck. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Hong Kong. [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Jason+Eisner%2E+2019%2E+Specializing+word+embeddings+%28for+parsing%29+by+information+bottleneck%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+an)
20. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019a%2E+Roberta%3A+A+robustly+optimized+BERT+pretraining+approach%2E+)
21. Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. 2019b. Rethinking the value of network pruning. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhuang+Liu%2C+Mingjie+Sun%2C+Tinghui+Zhou%2C+Gao+Huang%2C+and+Trevor+Darrell%2E+2019b%2E+Rethinking+the+value+of+network+pruning%2E+In+International+Conference+on+Learning+Representations%2E)
22. Christos Louizos, Max Welling, and Diederik P. Kingma. 2018. Learning sparse neural networks through l-0 regularization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Christos+Louizos%2C+Max+Welling%2C+and+Diederik+P%2E+Kingma%2E+2018%2E+Learning+sparse+neural+networks+through+l%2D0+regularization%2E+In+International+Conference+on+Learning+Representations%2E)
23. Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? ArXiv, abs/1905.10650. [find](https://scholar.google.com/scholar?q=Paul+Michel%2C+Omer+Levy%2C+and+Graham+Neubig%2E+2019%2E+Are+sixteen+heads+really+better+than+one%3F+ArXiv%2C+abs%2F1905%2E10650%2E) [our card](https://md.research.elamur.ai/free/en/papers/michel-2019-sixteen-heads-really-better-than/)
24. Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. 2017. Variational dropout sparsifies deep neural networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, pages 2498–2507. JMLR.org. [find](https://scholar.google.com/scholar?q=Dmitry+Molchanov%2C+Arsenii+Ashukha%2C+and+Dmitry+Vetrov%2E+2017%2E+Variational+dropout+sparsifies+deep+neural+networks%2E+In+Proceedings+of+the+34th+International+Conference+on+Machine+Learning+%2D+Volume+70%2C+IC)
25. Ari S. Morcos, Haonan Yu, Michela Paganini, and Yuand ong Tian. 2019. One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. arXiv e-prints, page arXiv:1906.02773. — [arXiv:1906.02773](https://arxiv.org/abs/1906.02773) [find](https://scholar.google.com/scholar?q=Ari+S%2E+Morcos%2C+Haonan+Yu%2C+Michela+Paganini%2C+and+Yuand+ong+Tian%2E+2019%2E+One+ticket+to+win+them+all%3A+generalizing+lottery+ticket+initializations+across+datasets+and+optimizers%2E+arXiv+e%2Dprints%2C+page+arXiv)
26. Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019. Distilling task-specific knowledge from BERT into simple neural networks. CoRR, abs/1903.12136. — [arXiv:1903.12136](https://arxiv.org/abs/1903.12136) [find](https://scholar.google.com/scholar?q=Raphael+Tang%2C+Yao+Lu%2C+Linqing+Liu%2C+Lili+Mou%2C+Olga+Vechtomova%2C+and+Jimmy+Lin%2E+2019%2E+Distilling+task%2Dspecific+knowledge+from+BERT+into+simple+neural+networks%2E+CoRR%2C+abs%2F1903%2E12136%2E+%E2%80%94+%5BarXiv%3A1903%2E12136%5D%28)
27. Yuandong Tian, Tina Jiang, Qucheng Gong, and Ari S. Morcos. 2019. Luck matters: Understanding training dynamics of deep relu networks. CoRR, abs/1905.13405. — [arXiv:1905.13405](https://arxiv.org/abs/1905.13405) [find](https://scholar.google.com/scholar?q=Yuandong+Tian%2C+Tina+Jiang%2C+Qucheng+Gong%2C+and+Ari+S%2E+Morcos%2E+2019%2E+Luck+matters%3A+Understanding+training+dynamics+of+deep+relu+networks%2E+CoRR%2C+abs%2F1905%2E13405%2E+%E2%80%94+%5BarXiv%3A1905%2E13405%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+CoRR%2C+abs%2F1706%2E03762%2E)
29. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R%2E+Bowman%2E+2019%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+International+C) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
30. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR, abs/1906.08237. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Yiming+Yang%2C+Jaime+G%2E+Carbonell%2C+Ruslan+Salakhutdinov%2C+and+Quoc+V%2E+Le%2E+2019%2E+Xlnet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+CoRR%2C+abs%2F1906%2E08237%2E) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
31. Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S. Morcos. 2019. Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP. arXiv e-prints, page arXiv:1906.02768. — [arXiv:1906.02768](https://arxiv.org/abs/1906.02768) [find](https://scholar.google.com/scholar?q=Haonan+Yu%2C+Sergey+Edunov%2C+Yuandong+Tian%2C+and+Ari+S%2E+Morcos%2E+2019%2E+Playing+the+lottery+with+rewards+and+multiple+languages%3A+lottery+tickets+in+RL+and+NLP%2E+arXiv+e%2Dprints%2C+page+arXiv%3A1906%2E02768%2E+%E2%80%94+%5BarXi)
32. Michael Zhu and Suyog Gupta. 2017. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv e-prints, page arXiv:1710.01878. — [arXiv:1710.01878](https://arxiv.org/abs/1710.01878) [find](https://scholar.google.com/scholar?q=Michael+Zhu+and+Suyog+Gupta%2E+2017%2E+To+prune%2C+or+not+to+prune%3A+exploring+the+efficacy+of+pruning+for+model+compression%2E+arXiv+e%2Dprints%2C+page+arXiv%3A1710%2E01878%2E+%E2%80%94+%5BarXiv%3A1710%2E01878%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
33. A Hesham Mostafa and Xin Wang. 2019. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. [find](https://scholar.google.com/scholar?q=A+Hesham+Mostafa+and+Xin+Wang%2E+2019%2E+Parameter+efficient+training+of+deep+convolutional+neural+networks+by+dynamic+sparse+reparameterization%2E)
34. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. CoRR, abs/1802.05365. [find](https://scholar.google.com/scholar?q=Matthew+E%2E+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+CoRR%2C+abs%2F1802%2E05365%2E)
35. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
36. Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv e-prints, page arXiv:1409.1556. [find](https://scholar.google.com/scholar?q=Karen+Simonyan+and+Andrew+Zisserman%2E+2014%2E+Very+Deep+Convolutional+Networks+for+Large%2DScale+Image+Recognition%2E+arXiv+e%2Dprints%2C+page+arXiv%3A1409%2E1556%2E) [our card](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/)