---
title: "Structured Pruning of Large Language Models — Wang 2020"
description: "The authors propose FLOP (Factorized Low-rank Pruning). 0.8 perplexity. 59 references, cited by 102 papers here."
authors: ["Ziheng Wang", "Jeremy Wohlwend", "Tao Lei"]
year: 2020
venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/2020.emnlp-main.496/"
doi: "10.18653/v1/2020.emnlp-main.496"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2020-structured-pruning-large-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2020-structured-pruning-large-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / Structured Pruning of Large Language Models

# Structured Pruning of Large Language Models

**Authors:** Ziheng Wang, Jeremy Wohlwend and [Tao Lei](https://md.research.elamur.ai/free/en/authors/tao-lei/) · **Year:** 2020 · **Venue:** [Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2020](https://md.research.elamur.ai/free/en/venues/emnlp/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-main.496/) · [PDF](https://aclanthology.org/2020.emnlp-main.496.pdf) · [DOI 10.18653/v1/2020.emnlp-main.496](https://doi.org/10.18653/v1/2020.emnlp-main.496) · [OpenAlex](https://openalex.org/W2979691890)

**Citations:** 95 citations (OpenAlex, 2026-10-14) · [102](https://md.research.elamur.ai/free/en/papers/wang-2020-structured-pruning-large-language-models/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary

Large pretrained language models are expensive to run, and the standard ways of shrinking them each fall short. Pruning individual weights leaves irregular sparsity that ordinary hardware can't exploit. Removing whole rows, columns or blocks gives speedups but usually hurts accuracy more. The authors propose FLOP (Factorized Low-rank Pruning). It rewrites every weight matrix as a product of two smaller factors and learns, during training, which rank-1 components can be dropped. The pruning gates follow the Hard Concrete l0 relaxation of Louizos et al. (2018). An augmented Lagrangian extension lets the user set the final model size directly. Because the pruned matrices stay dense, no special kernels or hardware are needed.

FLOP was tested on:
- SRU and Transformer-XL language models on Wiki-103 and Enwiki8,
- adaptive word embeddings, where it learns embedding sizes per word cluster,
- BERT fine-tuning on GLUE.

It beat block-structured pruning and matched or exceeded unstructured pruning, while making both training and inference about twice as fast. This shows that large models can be made much cheaper without exotic infrastructure.

## Key points

- On a word-level model with adaptive embeddings (Wiki-103), FLOP halves the parameter count at a cost of only 0.8 perplexity.
- On Enwiki8 with 6-layer SRU models, FLOP-l0 reaches 1.25 BPC at 70% compression, against 1.24 for the unpruned model. Its relative losses were smaller than input-feature pruning (NP-l0) at 70%, 80%, 85% and 90% compression.
- Measured CPU inference speedups range from 1.5x to 2.2x, and GPU training is up to 2.4x faster. Unstructured AGP pruning gave no speedup, even with PyTorch sparse routines or a sparse linear-algebra compiler.
- When applied to BERT on downstream classification tasks, FLOP removes about 35% of parameters while keeping roughly 99% of performance. Further gains are limited by the large embedding layers.
- The code is released at github.com/asappresearch/flop as a PyTorch library.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 77 works

## Cite

Cite as: Ziheng Wang, Jeremy Wohlwend and Tao Lei. 2020. Structured Pruning of Large Language Models. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2020structured,
  title = {{Structured Pruning of Large Language Models}},
  author = {Ziheng Wang, Jeremy Wohlwend and Tao Lei},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.496},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.496}
}
```

- [← previous Multimodal Transformer for Multimodal Machine Translation](https://md.research.elamur.ai/free/en/papers/yao-2020-multimodal-transformer-multimodal-machine-translation/)
- [next → Authorship Attribution for Neural Text Generation](https://md.research.elamur.ai/free/en/papers/uchendu-2020-authorship-attribution-neural-text-generation/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/)

## References

59 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jimmy Ba and Rich Caruana. 2014. Do deep nets really need to be deep? In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 2654–2662. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Jimmy+Ba+and+Rich+Caruana%2E+2014%2E+Do+deep+nets+really+need+to+be+deep%3F+In+Z%2E+Ghahramani%2C+M%2E+Welling%2C+C%2E+Cortes%2C+N%2E+D%2E+Lawrence%2C+and+K%2E+Q%2E+Weinberger%2C+editors%2C+Advances+in+Neural+Information+Processing+)
2. Alexei Baevski and Michael Auli. 2019. Adaptive input representations for neural language modeling. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Alexei+Baevski+and+Michael+Auli%2E+2019%2E+Adaptive+input+representations+for+neural+language+modeling%2E+In+International+Conference+on+Learning+Representations%2E)
3. Jasmijn Bastings, Wilker Aziz, and Ivan Titov. 2019. [Interpretable neural predictions with differentiable binary variables](https://md.research.elamur.ai/free/en/papers/bastings-2019-interpretable-neural-predictions-differentiable-binary/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2963–2977, Florence, Italy. Association for Computational Linguistics.
4. Shijie Cao, Chen Zhang, Zhuliang Yao, Wencong Xiao, Lanshun Nie, Dechen Zhan, Yunxin Liu, Ming Wu, and Lintao Zhang. 2019. Efficient and effective sparse lstm on fpga with bank-balanced sparsity. In Proceedings of the 2019 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, FPGA ’19… [find](https://scholar.google.com/scholar?q=Shijie+Cao%2C+Chen+Zhang%2C+Zhuliang+Yao%2C+Wencong+Xiao%2C+Lanshun+Nie%2C+Dechen+Zhan%2C+Yunxin+Liu%2C+Ming+Wu%2C+and+Lintao+Zhang%2E+2019%2E+Efficient+and+effective+sparse+lstm+on+fpga+with+bank%2Dbalanced+sparsity%2E+In+P)
5. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. [SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation](https://md.research.elamur.ai/free/en/papers/cer-2017-semeval-2017-task-1-semantic/). In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada.…
6. Yew Ken Chia, Sam Witteveen, and Martin Andrews. 2019. Transformer to cnn: Label-scarce distillation for efficient text classification. arXiv preprint arXiv:1909.03508. — [arXiv:1909.03508](https://arxiv.org/abs/1909.03508) [find](https://scholar.google.com/scholar?q=Yew+Ken+Chia%2C+Sam+Witteveen%2C+and+Martin+Andrews%2E+2019%2E+Transformer+to+cnn%3A+Label%2Dscarce+distillation+for+efficient+text+classification%2E+arXiv+preprint+arXiv%3A1909%2E03508%2E+%E2%80%94+%5BarXiv%3A1909%2E03508%5D%28https%3A%2F%2Far)
7. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 3079–3087. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Andrew+M+Dai+and+Quoc+V+Le%2E+2015%2E+Semi%2Dsupervised+sequence+learning%2E+In+C%2E+Cortes%2C+N%2E+D%2E+Lawrence%2C+D%2E+D%2E+Lee%2C+M%2E+Sugiyama%2C+and+R%2E+Garnett%2C+editors%2C+Advances+in+Neural+Information+Processing+Systems+28)
8. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, Florence, Italy. Assoc… [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Quoc+Le%2C+and+Ruslan+Salakhutdinov%2E+2019%2E+Transformer%2DXL%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+In+Proceedings+of+the+57th+Annu)
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
10. William B. Dolan and Chris Brockett. 2005. [Automatically constructing a corpus of sentential paraphrases](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/). In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
11. Angela Fan, Edouard Grave, and Armand Joulin. 2020. [Reducing transformer depth on demand with structured dropout](https://md.research.elamur.ai/free/en/papers/fan-2019-reducing-transformer-depth-demand-structured/). In International Conference on Learning Representations.
12. Jeffrey Flanigan, Sam Thomson, Jaime Carbonell, Chris Dyer, and Noah A. Smith. 2014. [A discriminative graph-based parser for the Abstract Meaning Representation](https://md.research.elamur.ai/free/en/papers/flanigan-2014-discriminative-graph-based-parser-abstract/). In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1426– 1436, Balt…
13. Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jonathan+Frankle+and+Michael+Carbin%2E+2019%2E+The+lottery+ticket+hypothesis%3A+Finding+sparse%2C+trainable+neural+networks%2E+In+International+Conference+on+Learning+Representations%2E)
14. Trevor Gale, Erich Elsen, and Sara Hooker. 2019. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574. — [arXiv:1902.09574](https://arxiv.org/abs/1902.09574) [find](https://scholar.google.com/scholar?q=Trevor+Gale%2C+Erich+Elsen%2C+and+Sara+Hooker%2E+2019%2E+The+state+of+sparsity+in+deep+neural+networks%2E+arXiv+preprint+arXiv%3A1902%2E09574%2E+%E2%80%94+%5BarXiv%3A1902%2E09574%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1902%2E09574%29)
15. Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. 2014. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115. — [arXiv:1412.6115](https://arxiv.org/abs/1412.6115) [find](https://scholar.google.com/scholar?q=Yunchao+Gong%2C+Liu+Liu%2C+Ming+Yang%2C+and+Lubomir+Bourdev%2E+2014%2E+Compressing+deep+convolutional+networks+using+vector+quantization%2E+arXiv+preprint+arXiv%3A1412%2E6115%2E+%E2%80%94+%5BarXiv%3A1412%2E6115%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
16. Edouard Grave, Armand Joulin, Moustapha Cissé, Hervé Jégou, et al. 2017. [Efficient softmax approximation for gpus](https://md.research.elamur.ai/free/en/papers/grave-2016-efficient-softmax-approximation-gpus/). In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1302–1310. JMLR. org.
17. Scott Gray, Alec Radford, and Diederik P Kingma. 2017. Gpu kernels for block-sparse weights. arXiv preprint arXiv:1711.09224. [find](https://scholar.google.com/scholar?q=Scott+Gray%2C+Alec+Radford%2C+and+Diederik+P+Kingma%2E+2017%2E+Gpu+kernels+for+block%2Dsparse+weights%2E+arXiv+preprint+arXiv%3A1711%2E09224%2E)
18. Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William J Dally. 2016. Eie: efficient inference engine on compressed deep neural network. In 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), pages 243–254. IEEE. [find](https://scholar.google.com/scholar?q=Song+Han%2C+Xingyu+Liu%2C+Huizi+Mao%2C+Jing+Pu%2C+Ardavan+Pedram%2C+Mark+A+Horowitz%2C+and+William+J+Dally%2E+2016%2E+Eie%3A+efficient+inference+engine+on+compressed+deep+neural+network%2E+In+2016+ACM%2FIEEE+43rd+Annual+In)
19. Song Han, Huizi Mao, and William J Dally. 2015a. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149. [find](https://scholar.google.com/scholar?q=Song+Han%2C+Huizi+Mao%2C+and+William+J+Dally%2E+2015a%2E+Deep+compression%3A+Compressing+deep+neural+networks+with+pruning%2C+trained+quantization+and+huffman+coding%2E+arXiv+preprint+arXiv%3A1510%2E00149%2E)
20. Song Han, Jeff Pool, John Tran, and William Dally. 2015b. Learning both weights and connections for efficient neural network. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 1135–1143. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Song+Han%2C+Jeff+Pool%2C+John+Tran%2C+and+William+Dally%2E+2015b%2E+Learning+both+weights+and+connections+for+efficient+neural+network%2E+In+C%2E+Cortes%2C+N%2E+D%2E+Lawrence%2C+D%2E+D%2E+Lee%2C+M%2E+Sugiyama%2C+and+R%2E+Garnett%2C+edit)
21. Thirty-Second AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Thirty%2DSecond+AAAI+Conference+on+Artificial+Intelligence%2E)
22. Liyuan Liu, Xiang Ren, Jingbo Shang, Xiaotao Gu, Jian Peng, and Jiawei Han. 2018. Efficient contextualized representation: Language model pruning for sequence labeling. In EMNLP, pages 1215–1225. [find](https://scholar.google.com/scholar?q=Liyuan+Liu%2C+Xiang+Ren%2C+Jingbo+Shang%2C+Xiaotao+Gu%2C+Jian+Peng%2C+and+Jiawei+Han%2E+2018%2E+Efficient+contextualized+representation%3A+Language+model+pruning+for+sequence+labeling%2E+In+EMNLP%2C+pages+1215%E2%80%931225%2E)
23. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
24. Christos Louizos, Karen Ullrich, and Max Welling. 2017. Bayesian compression for deep learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 3288–3298. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Christos+Louizos%2C+Karen+Ullrich%2C+and+Max+Welling%2E+2017%2E+Bayesian+compression+for+deep+learning%2E+In+I%2E+Guyon%2C+U%2E+V%2E+Luxburg%2C+S%2E+Bengio%2C+H%2E+Wallach%2C+R%2E+Fergus%2C+S%2E+Vishwanathan%2C+and+R%2E+Garnett%2C+editors%2C+)
25. Christos Louizos, Max Welling, and Diederik P. Kingma. 2018. Learning sparse neural networks through l 0 regularization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Christos+Louizos%2C+Max+Welling%2C+and+Diederik+P%2E+Kingma%2E+2018%2E+Learning+sparse+neural+networks+through+l+0+regularization%2E+In+International+Conference+on+Learning+Representations%2E)
26. André F. T. Martins, Mario A. T. Figueiredo, Pedro M. Q. Aguiar, Noah A. Smith, and Eric P. Xing. 2011. An augmented lagrangian approach to constrained map inference. In ICML, pages 169–176. [find](https://scholar.google.com/scholar?q=Andre%CC%81+F%2E+T%2E+Martins%2C+Mario+A%2E+T%2E+Figueiredo%2C+Pedro+M%2E+Q%2E+Aguiar%2C+Noah+A%2E+Smith%2C+and+Eric+P%2E+Xing%2E+2011%2E+An+augmented+lagrangian+approach+to+constrained+map+inference%2E+In+ICML%2C+pages+169%E2%80%93176%2E)
27. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). arXiv preprint arXiv:1503.02531.
28. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. An analysis of neural language modeling at multiple scales. CoRR, abs/1803.08240. [find](https://scholar.google.com/scholar?q=Stephen+Merity%2C+Nitish+Shirish+Keskar%2C+and+Richard+Socher%2E+2018%2E+An+analysis+of+neural+language+modeling+at+multiple+scales%2E+CoRR%2C+abs%2F1803%2E08240%2E)
29. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351. [find](https://scholar.google.com/scholar?q=Xiaoqi+Jiao%2C+Yichun+Yin%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Xiao+Chen%2C+Linlin+Li%2C+Fang+Wang%2C+and+Qun+Liu%2E+2019%2E+Tinybert%3A+Distilling+bert+for+natural+language+understanding%2E+arXiv+preprint+arXiv%3A1909%2E10351%2E)
30. Dmitry Molchanov, Arsenii Ashukha, and Dmitry P. Vetrov. 2017. Variational dropout sparsifies deep neural networks. In ICML, pages 2498–2507. [find](https://scholar.google.com/scholar?q=Dmitry+Molchanov%2C+Arsenii+Ashukha%2C+and+Dmitry+P%2E+Vetrov%2E+2017%2E+Variational+dropout+sparsifies+deep+neural+networks%2E+In+ICML%2C+pages+2498%E2%80%932507%2E)
31. Yoon Kim and Alexander M. Rush. 2016. [Sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, Austin, Texas. Association for Computational Linguistics.
32. Fredrik Kjolstad, Shoaib Kamil, Stephen Chou, David Lugato, and Saman Amarasinghe. 2017. The tensor algebra compiler. Proc. ACM Program. Lang., 1(OOPSLA). [find](https://scholar.google.com/scholar?q=Fredrik+Kjolstad%2C+Shoaib+Kamil%2C+Stephen+Chou%2C+David+Lugato%2C+and+Saman+Amarasinghe%2E+2017%2E+The+tensor+algebra+compiler%2E+Proc%2E+ACM+Program%2E+Lang%2E%2C+1%28OOPSLA%29%2E)
33. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2020%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+In+International+Confe)
34. Sharan Narang, Erich Elsen, Gregory Diamos, and Shubho Sengupta. 2017a. Exploring sparsity in recurrent neural networks. arXiv preprint arXiv:1704.05119. [find](https://scholar.google.com/scholar?q=Sharan+Narang%2C+Erich+Elsen%2C+Gregory+Diamos%2C+and+Shubho+Sengupta%2E+2017a%2E+Exploring+sparsity+in+recurrent+neural+networks%2E+arXiv+preprint+arXiv%3A1704%2E05119%2E)
35. Sharan Narang, Eric Undersander, and Gregory Diamos. 2017b. Block-sparse recurrent neural networks. arXiv preprint arXiv:1711.02782. — [arXiv:1711.02782](https://arxiv.org/abs/1711.02782) [find](https://scholar.google.com/scholar?q=Sharan+Narang%2C+Eric+Undersander%2C+and+Gregory+Diamos%2E+2017b%2E+Block%2Dsparse+recurrent+neural+networks%2E+arXiv+preprint+arXiv%3A1711%2E02782%2E+%E2%80%94+%5BarXiv%3A1711%2E02782%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1711%2E02782%29)
36. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop. [find](https://scholar.google.com/scholar?q=Adam+Paszke%2C+Sam+Gross%2C+Soumith+Chintala%2C+Gregory+Chanan%2C+Edward+Yang%2C+Zachary+DeVito%2C+Zeming+Lin%2C+Alban+Desmaison%2C+Luca+Antiga%2C+and+Adam+Lerer%2E+2017%2E+Automatic+differentiation+in+PyTorch%2E+In+NIPS+Aut)
37. Tao Lei, Yu Zhang, Sida I. Wang, Hui Dai, and Yoav Artzi. 2018. Simple recurrent units for highly parallelizable recurrence. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4470–4481, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tao+Lei%2C+Yu+Zhang%2C+Sida+I%2E+Wang%2C+Hui+Dai%2C+and+Yoav+Artzi%2E+2018%2E+Simple+recurrent+units+for+highly+parallelizable+recurrence%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Langu)
38. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog…
39. Zhongliang Li, Raymond Kulhanek, Shaojun Wang, Yunxin Zhao, and Shuang Wu. 2018. Slim embedding layers for recurrent neural language models. In Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Zhongliang+Li%2C+Raymond+Kulhanek%2C+Shaojun+Wang%2C+Yunxin+Zhao%2C+and+Shuang+Wu%2E+2018%2E+Slim+embedding+layers+for+recurrent+neural+language+models%2E+In+Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya)
40. Alexander M. Rush, David Sontag, Michael Collins, and Tommi Jaakkola. 2010. [On dual decomposition and linear programming relaxations for natural language processing](https://md.research.elamur.ai/free/en/papers/rush-2010-dual-decomposition-linear-programming-relaxations/). In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1–11, Cambridge, MA. Association for…
41. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+2019%2E+Distilbert%2C+a+distilled+version+of+bert%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E+arXiv+preprint+arXiv%3A1910%2E01108%2E)
42. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Proces…
43. Abigail See, Minh-Thang Luong, and Christopher D. Manning. 2016. Compression of neural machine translation models via pruning. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 291–301, Berlin, Germany. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Minh%2DThang+Luong%2C+and+Christopher+D%2E+Manning%2E+2016%2E+Compression+of+neural+machine+translation+models+via+pruning%2E+In+Proceedings+of+The+20th+SIGNLL+Conference+on+Computational+Natural+Lan)
44. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. [Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned](https://md.research.elamur.ai/free/en/papers/voita-2019-analyzing-multi-head-self-attention/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797–5808, Florence, I…
45. Yuan Shangguan, Jian Li, Liang Qiao, Raziel Alvarez, and Ian McGraw. 2019. Optimizing speech recognition for the edge. arXiv preprint arXiv:1909.12408. — [arXiv:1909.12408](https://arxiv.org/abs/1909.12408) [find](https://scholar.google.com/scholar?q=Yuan+Shangguan%2C+Jian+Li%2C+Liang+Qiao%2C+Raziel+Alvarez%2C+and+Ian+McGraw%2E+2019%2E+Optimizing+speech+recognition+for+the+edge%2E+arXiv+preprint+arXiv%3A1909%2E12408%2E+%E2%80%94+%5BarXiv%3A1909%2E12408%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1909%2E)
46. Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. 2019. Q-bert: Hessian based ultra low precision quantization of bert. arXiv preprint arXiv:1909.05840. — [arXiv:1909.05840](https://arxiv.org/abs/1909.05840) [find](https://scholar.google.com/scholar?q=Sheng+Shen%2C+Zhen+Dong%2C+Jiayu+Ye%2C+Linjian+Ma%2C+Zhewei+Yao%2C+Amir+Gholami%2C+Michael+W+Mahoney%2C+and+Kurt+Keutzer%2E+2019%2E+Q%2Dbert%3A+Hessian+based+ultra+low+precision+quantization+of+bert%2E+arXiv+preprint+arXiv%3A1)
47. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–… [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D%2E+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In)
48. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. [Patient knowledge distillation for BERT model compression](https://md.research.elamur.ai/free/en/papers/sun-2019-patient-knowledge-distillation-bert-model/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Associat…
49. Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li, and Amelia Archer. 2019. Small and practical BERT models for sequence labeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Languag… [find](https://scholar.google.com/scholar?q=Henry+Tsai%2C+Jason+Riesa%2C+Melvin+Johnson%2C+Naveen+Arivazhagan%2C+Xin+Li%2C+and+Amelia+Archer%2E+2019%2E+Small+and+practical+BERT+models+for+sequence+labeling%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+)
50. Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Well-read students learn better: The impact of student initialization on knowledge distillation](https://md.research.elamur.ai/free/en/papers/turc-2019-well-read-students-learn-better/). arXiv preprint arXiv:1908.08962.
51. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In International Conference on Learning Representations.
52. Wei Wen, Yuxiong He, Samyam Rajbhandari, Minjia Zhang, Wenhan Wang, Fang Liu, Bin Hu, Yiran Chen, and Hai Li. 2018. Learning intrinsic sparse structures within long short-term memory. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Wei+Wen%2C+Yuxiong+He%2C+Samyam+Rajbhandari%2C+Minjia+Zhang%2C+Wenhan+Wang%2C+Fang+Liu%2C+Bin+Hu%2C+Yiran+Chen%2C+and+Hai+Li%2E+2018%2E+Learning+intrinsic+sparse+structures+within+long+short%2Dterm+memory%2E+In+International)
53. Yuhuai Wu, Saizheng Zhang, Ying Zhang, Yoshua Bengio, and Ruslan R Salakhutdinov. 2016. On multiplicative integration with recurrent neural networks. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Yuhuai+Wu%2C+Saizheng+Zhang%2C+Ying+Zhang%2C+Yoshua+Bengio%2C+and+Ruslan+R+Salakhutdinov%2E+2016%2E+On+multiplicative+integration+with+recurrent+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Syste)
54. Zhuliang Yao, Shijie Cao, Wencong Xiao, Chen Zhang, and Lanshun Nie. 2019. Balanced sparsity for efficient dnn inference on gpu. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5676–5683. [find](https://scholar.google.com/scholar?q=Zhuliang+Yao%2C+Shijie+Cao%2C+Wencong+Xiao%2C+Chen+Zhang%2C+and+Lanshun+Nie%2E+2019%2E+Balanced+sparsity+for+efficient+dnn+inference+on+gpu%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volum)
55. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. Ernie: Enhanced language representation with informative entities. arXiv preprint arXiv:1905.07129. [find](https://scholar.google.com/scholar?q=Zhengyan+Zhang%2C+Xu+Han%2C+Zhiyuan+Liu%2C+Xin+Jiang%2C+Maosong+Sun%2C+and+Qun+Liu%2E+2019%2E+Ernie%3A+Enhanced+language+representation+with+informative+entities%2E+arXiv+preprint+arXiv%3A1905%2E07129%2E)
56. Chenzhuo Zhu, Song Han, Huizi Mao, and William J. Dally. 2017. Trained ternary quantization. In 5th International Conference on Learning Representations ICLR. [find](https://scholar.google.com/scholar?q=Chenzhuo+Zhu%2C+Song+Han%2C+Huizi+Mao%2C+and+William+J%2E+Dally%2E+2017%2E+Trained+ternary+quantization%2E+In+5th+International+Conference+on+Learning+Representations+ICLR%2E)
57. Michael Zhu and Suyog Gupta. 2017. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878. — [arXiv:1710.01878](https://arxiv.org/abs/1710.01878) [find](https://scholar.google.com/scholar?q=Michael+Zhu+and+Suyog+Gupta%2E+2017%2E+To+prune%2C+or+not+to+prune%3A+exploring+the+efficacy+of+pruning+for+model+compression%2E+arXiv+preprint+arXiv%3A1710%2E01878%2E+%E2%80%94+%5BarXiv%3A1710%2E01878%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1710%2E)
58. Neta Zmora, Guy Jacob, Lev Zlotnik, Bar Elharar, and Gal Novik. 2019. Neural network distiller: A python package for dnn compression research. [find](https://scholar.google.com/scholar?q=Neta+Zmora%2C+Guy+Jacob%2C+Lev+Zlotnik%2C+Bar+Elharar%2C+and+Gal+Novik%2E+2019%2E+Neural+network+distiller%3A+A+python+package+for+dnn+compression+research%2E)
59. Ehsan Variani, Ananda Theertha Suresh, and Mitchel Weintraub. 2019. West: Word encoded sequence [find](https://scholar.google.com/scholar?q=Ehsan+Variani%2C+Ananda+Theertha+Suresh%2C+and+Mitchel+Weintraub%2E+2019%2E+West%3A+Word+encoded+sequence)

*Updated 24 October 2026*