---
title: "Structured Pruning Learns Compact and Accurate — Xia 2022"
description: "Compressing large pre-trained language models usually involves a trade-off."
authors: ["Mengzhou Xia", "Zexuan Zhong", "Danqi Chen"]
year: 2022
venue: "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2022.acl-long.107/"
doi: "10.18653/v1/2022.acl-long.107"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xia-2022-structured-pruning-learns-compact-accurate/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xia-2022-structured-pruning-learns-compact-accurate/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / Structured Pruning Learns Compact and Accurate Models

# Structured Pruning Learns Compact and Accurate Models

**Authors:** [Mengzhou Xia](https://md.research.elamur.ai/free/en/authors/mengzhou-xia/), [Zexuan Zhong](https://md.research.elamur.ai/free/en/authors/zexuan-zhong/), [Danqi Chen](https://md.research.elamur.ai/free/en/authors/danqi-chen/) · **Year:** 2022 · **Venue:** Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.acl-long.107/) · [PDF](https://aclanthology.org/2022.acl-long.107.pdf) · [DOI 10.18653/v1/2022.acl-long.107](https://doi.org/10.18653/v1/2022.acl-long.107) · [OpenAlex](https://openalex.org/W4226126941)

**Citations:** 128 citations (openalex-snapshot · 2026-10-14) · [88](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Compressing large pre-trained language models usually involves a trade-off. Distillation produces fast compact models but needs costly general training on large unlabeled corpora. Structured pruning is cheaper but has seldom delivered more than 2–3× speedups. The authors propose CoFi (Coarse- and Fine-grained Pruning), a task-specific structured pruning method for Transformers. It starts from a fine-tuned BERT model and learns several mask types at once. Coarse masks remove whole MHA or FFN layers, while fine masks remove individual heads, FFN intermediate dimensions and hidden dimensions. Each parameter is therefore governed by masks at multiple granularities. CoFi also adds a layerwise distillation objective that learns, during training, how layers of the shrinking student map onto the unpruned teacher. On GLUE and SQuAD v1.1, CoFi matches or beats strong distillation and pruning baselines in both accuracy and latency. It exceeds 10× speedup at roughly 95% sparsity and uses no extra unlabeled data. This makes structured pruning a practical, low-compute alternative to distillation for extreme compression.

## Key points
- CoFi learns five mask variables jointly: z_FFN, z_int, z_MHA, z_head and z_hidn. In ablations, dropping the layer-level masks greatly reduces speedup at high sparsity, and dropping the hidden-dimension masks noticeably hurts accuracy.
- On MNLI, measured on an NVIDIA V100 against BERT base (84.8, 85M params), CoFi reaches 84.9 at 2.7× speedup with 26M params. It reaches 80.6 at 12.1× speedup with 4.4M params, which beats TinyBERT 4 at 11.4× (78.8).
- Training cost is low. CoFi needs at most about 20 hours on one GPU per GLUE task, and under 3 hours for smaller datasets. For comparison, TinyBERT's general distillation alone takes about 3.5 days on 4 GPUs over 2,500M tokens.
- The pruned architectures differ by dataset. They tend to keep more MHA than FFN layers and usually preserve the first two MHA layers, while middle layers are often removed.
- Code and models are released at github.com/princeton-nlp/CoFiPruning.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

88 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 86 not yet.

- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://md.research.elamur.ai/free/en/papers/ma-2023-llm-pruner-structural-pruning-large/)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://md.research.elamur.ai/free/en/papers/yu-2023-language-models-super-mario-absorbing/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xia2022structured,
  title = {{Structured Pruning Learns Compact and Accurate Models}},
  author = {Xia, Mengzhou and Zhong, Zexuan and Chen, Danqi},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.107},
  url = {https://doi.org/10.18653/v1/2022.acl-long.107}
}
```

- [← previous Structured Dialogue Discourse Parsing](https://md.research.elamur.ai/free/en/papers/chi-2022-structured-dialogue-discourse-parsing/)
- [next → Style Classification of Rabbinic Literature for Detection of Lost Midrash Tanhuma Material](https://md.research.elamur.ai/free/en/papers/aclbulk-2022-tannor-style-classification-rabbinic-literature-detection/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

55 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada.… [find](https://scholar.google.com/scholar?q=Daniel+Cer%2C+Mona+Diab%2C+Eneko+Agirre%2C+I%C3%B1igo+Lopez%2DGazpio%2C+and+Lucia+Specia%2E+2017%2E+SemEval%2D2017+task+1%3A+Semantic+textual+similarity+multilingual+and+crosslingual+focused+evaluation%2E+In+Proceedings+of+th)
2. Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. 2020a. The lottery ticket hypothesis for pre-trained BERT networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 15834– 15846. [find](https://scholar.google.com/scholar?q=Tianlong+Chen%2C+Jonathan+Frankle%2C+Shiyu+Chang%2C+Sijia+Liu%2C+Yang+Zhang%2C+Zhangyang+Wang%2C+and+Michael+Carbin%2E+2020a%2E+The+lottery+ticket+hypothesis+for+pre%2Dtrained+BERT+networks%2E+In+Advances+in+Neural+Infor)
3. Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Zhangyang Wang, and Jingjing Liu. 2020b. Earlybert: Efficient bert training via early-bird lottery tickets. arXiv preprint arXiv:2101.00063. [find](https://scholar.google.com/scholar?q=Xiaohan+Chen%2C+Yu+Cheng%2C+Shuohang+Wang%2C+Zhe+Gan%2C+Zhangyang+Wang%2C+and+Jingjing+Liu%2E+2020b%2E+Earlybert%3A+Efficient+bert+training+via+early%2Dbird+lottery+tickets%2E+arXiv+preprint+arXiv%3A2101%2E00063%2E)
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics (NAACL), pages 4171–4186. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+North+American+Chapter+of+the+Association+f)
5. William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). [find](https://scholar.google.com/scholar?q=William+B+Dolan+and+Chris+Brockett%2E+2005%2E+Automatically+constructing+a+corpus+of+sentential+paraphrases%2E+In+Proceedings+of+the+Third+International+Workshop+on+Paraphrasing+%28IWP2005%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/)
6. Angela Fan, Edouard Grave, and Armand Joulin. 2020. Reducing Transformer depth on demand with structured dropout. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Edouard+Grave%2C+and+Armand+Joulin%2E+2020%2E+Reducing+Transformer+depth+on+demand+with+structured+dropout%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/fan-2019-reducing-transformer-depth-demand-structured/)
7. Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Rémi Gribonval, Herve Jegou, and Armand Joulin. 2021. Training with quantization noise for extreme model compression. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Pierre+Stock%2C+Benjamin+Graham%2C+Edouard+Grave%2C+R%C3%A9mi+Gribonval%2C+Herve+Jegou%2C+and+Armand+Joulin%2E+2021%2E+Training+with+quantization+noise+for+extreme+model+compression%2E+In+International+Confere)
8. Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Jonathan+Frankle+and+Michael+Carbin%2E+2019%2E+The+lottery+ticket+hypothesis%3A+Finding+sparse%2C+trainable+neural+networks%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
9. Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. 2020. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning (ICML), pages 3259–3269. [find](https://scholar.google.com/scholar?q=Jonathan+Frankle%2C+Gintare+Karolina+Dziugaite%2C+Daniel+Roy%2C+and+Michael+Carbin%2E+2020%2E+Linear+mode+connectivity+and+the+lottery+ticket+hypothesis%2E+In+International+Conference+on+Machine+Learning+%28ICML%29%2C+)
10. Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Deming Chen, and Marianne Winslett. 2021. Compressing large-scale Transformer-based models: A case study on BERT. Transactions of the Association of Computational Linguistics (TACL), 9:1061–1080. [find](https://scholar.google.com/scholar?q=Prakhar+Ganesh%2C+Yao+Chen%2C+Xin+Lou%2C+Mohammad+Ali+Khan%2C+Yin+Yang%2C+Hassan+Sajjad%2C+Preslav+Nakov%2C+Deming+Chen%2C+and+Marianne+Winslett%2E+2021%2E+Compressing+large%2Dscale+Transformer%2Dbased+models%3A+A+case+study+o)
11. Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. 2020. DynaBERT: Dynamic bert with adaptive width and depth. In Advances in Neural Information Processing Systems (NeurIPS), volume 33. [find](https://scholar.google.com/scholar?q=Lu+Hou%2C+Zhiqi+Huang%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Xiao+Chen%2C+and+Qun+Liu%2E+2020%2E+DynaBERT%3A+Dynamic+bert+with+adaptive+width+and+depth%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2C+volume)
12. Shaoyi Huang, Dongkuan Xu, Ian EH Yen, Sung-en Chang, Bingbing Li, Shiyang Chen, Mimi Xie, Hang Liu, and Caiwen Ding. 2021. Sparse progressive distillation: Resolving overfitting under pretrain-and-finetune paradigm. arXiv e-prints, pages arXiv– 2110. [find](https://scholar.google.com/scholar?q=Shaoyi+Huang%2C+Dongkuan+Xu%2C+Ian+EH+Yen%2C+Sung%2Den+Chang%2C+Bingbing+Li%2C+Shiyang+Chen%2C+Mimi+Xie%2C+Hang+Liu%2C+and+Caiwen+Ding%2E+2021%2E+Sparse+progressive+distillation%3A+Resolving+overfitting+under+pretrain%2Dand%2Dfi)
13. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Findings of Empirical Methods in Natural Language Processing (EMNLP), pages 4163– 4174. [find](https://scholar.google.com/scholar?q=Xiaoqi+Jiao%2C+Yichun+Yin%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Xiao+Chen%2C+Linlin+Li%2C+Fang+Wang%2C+and+Qun+Liu%2E+2020%2E+TinyBERT%3A+Distilling+BERT+for+natural+language+understanding%2E+In+Findings+of+Empirical+Methods+in+)
14. François Lagunas, Ella Charlaix, Victor Sanh, and Alexander M Rush. 2021. Block pruning for faster transformers. arXiv preprint arXiv:2109.04838. [find](https://scholar.google.com/scholar?q=Fran%C3%A7ois+Lagunas%2C+Ella+Charlaix%2C+Victor+Sanh%2C+and+Alexander+M+Rush%2E+2021%2E+Block+pruning+for+faster+transformers%2E+arXiv+preprint+arXiv%3A2109%2E04838%2E)
15. Jiaoda Li, Ryan Cotterell, and Mrinmaya Sachan. 2021. Differentiable subset pruning of Transformer heads. Transactions of the Association of Computational Linguistics (TACL). [find](https://scholar.google.com/scholar?q=Jiaoda+Li%2C+Ryan+Cotterell%2C+and+Mrinmaya+Sachan%2E+2021%2E+Differentiable+subset+pruning+of+Transformer+heads%2E+Transactions+of+the+Association+of+Computational+Linguistics+%28TACL%29%2E)
16. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019a%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+)
17. Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. 2019b. Metapruning: Meta learning for automatic neural network channel pruning. In International Conference on Computer Vision (ICCV), pages 3296–3305. [find](https://scholar.google.com/scholar?q=Zechun+Liu%2C+Haoyuan+Mu%2C+Xiangyu+Zhang%2C+Zichao+Guo%2C+Xin+Yang%2C+Kwang%2DTing+Cheng%2C+and+Jian+Sun%2E+2019b%2E+Metapruning%3A+Meta+learning+for+automatic+neural+network+channel+pruning%2E+In+International+Conference)
18. Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. 2017. Learning efficient convolutional networks through network slimming. In International Conference on Computer Vision (ICCV), pages 2736–2744. [find](https://scholar.google.com/scholar?q=Zhuang+Liu%2C+Jianguo+Li%2C+Zhiqiang+Shen%2C+Gao+Huang%2C+Shoumeng+Yan%2C+and+Changshui+Zhang%2E+2017%2E+Learning+efficient+convolutional+networks+through+network+slimming%2E+In+International+Conference+on+Computer+V)
19. Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. 2019c. Rethinking the value of network pruning. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Zhuang+Liu%2C+Mingjie+Sun%2C+Tinghui+Zhou%2C+Gao+Huang%2C+and+Trevor+Darrell%2E+2019c%2E+Rethinking+the+value+of+network+pruning%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
20. Shaopeng Guo, Yujie Wang, Quanquan Li, and Junjie Yan. 2020. Dmcp: Differentiable markov channel pruning for neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [find](https://scholar.google.com/scholar?q=Shaopeng+Guo%2C+Yujie+Wang%2C+Quanquan+Li%2C+and+Junjie+Yan%2E+2020%2E+Dmcp%3A+Differentiable+markov+channel+pruning+for+neural+networks%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+R)
21. C Louizos, M Welling, and DP Kingma. 2018. Learning sparse neural networks through l0 regularization. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=C+Louizos%2C+M+Welling%2C+and+DP+Kingma%2E+2018%2E+Learning+sparse+neural+networks+through+l0+regularization%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
22. Yihui He, Xiangyu Zhang, and Jian Sun. 2017. Channel pruning for accelerating very deep neural networks. In International Conference on Computer Vision (ICCV), pages 1389–1397. [find](https://scholar.google.com/scholar?q=Yihui+He%2C+Xiangyu+Zhang%2C+and+Jian+Sun%2E+2017%2E+Channel+pruning+for+accelerating+very+deep+neural+networks%2E+In+International+Conference+on+Computer+Vision+%28ICCV%29%2C+pages+1389%E2%80%931397%2E)
23. Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. 2017. Thinet: A filter level pruning method for deep neural network compression. In International Conference on Computer Vision (ICCV), pages 5058–5066. [find](https://scholar.google.com/scholar?q=Jian%2DHao+Luo%2C+Jianxin+Wu%2C+and+Weiyao+Lin%2E+2017%2E+Thinet%3A+A+filter+level+pruning+method+for+deep+neural+network+compression%2E+In+International+Conference+on+Computer+Vision+%28ICCV%29%2C+pages+5058%E2%80%935066%2E)
24. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. [find](https://scholar.google.com/scholar?q=Geoffrey+Hinton%2C+Oriol+Vinyals%2C+and+Jeff+Dean%2E+2015%2E+Distilling+the+knowledge+in+a+neural+network%2E+arXiv+preprint+arXiv%3A1503%2E02531%2E) [our card](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/)
25. JS McCarley, Rishav Chakravarti, and Avirup Sil. 2019. Structured pruning of a BERT-based question answering model. arXiv preprint arXiv:1910.06360. — [arXiv:1910.06360](https://arxiv.org/abs/1910.06360) [find](https://scholar.google.com/scholar?q=JS+McCarley%2C+Rishav+Chakravarti%2C+and+Avirup+Sil%2E+2019%2E+Structured+pruning+of+a+BERT%2Dbased+question+answering+model%2E+arXiv+preprint+arXiv%3A1910%2E06360%2E+%E2%80%94+%5BarXiv%3A1910%2E06360%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1910%2E063)
26. Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems (NeurIPS). [find](https://scholar.google.com/scholar?q=Paul+Michel%2C+Omer+Levy%2C+and+Graham+Neubig%2E+2019%2E+Are+sixteen+heads+really+better+than+one%3F+In+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/michel-2019-sixteen-heads-really-better-than/)
27. Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. 2019. Importance estimation for neural network pruning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11264–11272. [find](https://scholar.google.com/scholar?q=Pavlo+Molchanov%2C+Arun+Mallya%2C+Stephen+Tyree%2C+Iuri+Frosio%2C+and+Jan+Kautz%2E+2019%2E+Importance+estimation+for+neural+network+pruning%2E+In+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR)
28. Matan Ben Noach and Yoav Goldberg. 2020. Compressing pre-trained language models by matrix decomposition. In Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 884–889. [find](https://scholar.google.com/scholar?q=Matan+Ben+Noach+and+Yoav+Goldberg%2E+2020%2E+Compressing+pre%2Dtrained+language+models+by+matrix+decomposition%2E+In+Asia%2DPacific+Chapter+of+the+Association+for+Computational+Linguistics+and+the+10th+Internat)
29. Sai Prasanna, Anna Rogers, and Anna Rumshisky. 2020. When BERT plays the lottery, all tickets are winning. In Empirical Methods in Natural Language Processing (EMNLP), pages 3208–3229. [find](https://scholar.google.com/scholar?q=Sai+Prasanna%2C+Anna+Rogers%2C+and+Anna+Rumshisky%2E+2020%2E+When+BERT+plays+the+lottery%2C+all+tickets+are+winning%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+3208%E2%80%933229%2E)
30. Ofir Press, Noah A Smith, and Omer Levy. 2020. Improving transformer models by reordering their sub-layers. In Association for Computational Linguistics (ACL), pages 2996–3005. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Noah+A+Smith%2C+and+Omer+Levy%2E+2020%2E+Improving+transformer+models+by+reordering+their+sub%2Dlayers%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2C+pages+2996%E2%80%933005%2E)
31. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text Transformer. The Journal of Machine Learning Research (JMLR), 21(140). [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J+Liu%2E+2020%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext)
32. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP), pages 2383–2392. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
33. Alex Renda, Jonathan Frankle, and Michael Carbin. 2020. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Alex+Renda%2C+Jonathan+Frankle%2C+and+Michael+Carbin%2E+2020%2E+Comparing+rewinding+and+fine%2Dtuning+in+neural+network+pruning%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
34. Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. 2020. Poor man’s BERT: Smaller and faster transformer models. arXiv preprint arXiv:2004.03844. — [arXiv:2004.03844](https://arxiv.org/abs/2004.03844) [find](https://scholar.google.com/scholar?q=Hassan+Sajjad%2C+Fahim+Dalvi%2C+Nadir+Durrani%2C+and+Preslav+Nakov%2E+2020%2E+Poor+man%E2%80%99s+BERT%3A+Smaller+and+faster+transformer+models%2E+arXiv+preprint+arXiv%3A2004%2E03844%2E+%E2%80%94+%5BarXiv%3A2004%2E03844%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
35. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+2019%2E+DistilBERT%2C+a+distilled+version+of+bert%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E+arXiv+preprint+arXiv%3A1910%2E01108%2E)
36. Victor Sanh, Thomas Wolf, and Alexander Rush. 2020. Movement pruning: Adaptive sparsity by fine-tuning. Advances in Neural Information Processing Systems (NeurIPS), 33. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Thomas+Wolf%2C+and+Alexander+Rush%2E+2020%2E+Movement+pruning%3A+Adaptive+sparsity+by+fine%2Dtuning%2E+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2C+33%2E)
37. Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. 2020. Q-BERT: Hessian based ultra low precision quantization of BERT. In Conference on Artificial Intelligence (AAAI), pages 8815–8821. [find](https://scholar.google.com/scholar?q=Sheng+Shen%2C+Zhen+Dong%2C+Jiayu+Ye%2C+Linjian+Ma%2C+Zhewei+Yao%2C+Amir+Gholami%2C+Michael+W+Mahoney%2C+and+Kurt+Keutzer%2E+2020%2E+Q%2DBERT%3A+Hessian+based+ultra+low+precision+quantization+of+BERT%2E+In+Conference+on+Artif)
38. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D%2E+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In)
39. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model compression. In Empirical Methods in Natural Language Processing (EMNLP), pages 4314–4323. [find](https://scholar.google.com/scholar?q=Siqi+Sun%2C+Yu+Cheng%2C+Zhe+Gan%2C+and+Jingjing+Liu%2E+2019%2E+Patient+knowledge+distillation+for+BERT+model+compression%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+4314%E2%80%934323%2E) [our card](https://md.research.elamur.ai/free/en/papers/sun-2019-patient-knowledge-distillation-bert-model/)
40. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. MobileBERT: a compact task-agnostic bert for resource-limited devices. In Association for Computational Linguistics (ACL), pages 2158–2170. [find](https://scholar.google.com/scholar?q=Zhiqing+Sun%2C+Hongkun+Yu%2C+Xiaodan+Song%2C+Renjie+Liu%2C+Yiming+Yang%2C+and+Denny+Zhou%2E+2020%2E+MobileBERT%3A+a+compact+task%2Dagnostic+bert+for+resource%2Dlimited+devices%2E+In+Association+for+Computational+Linguistic) [our card](https://md.research.elamur.ai/free/en/papers/sun-2020-mobilebert-compact-task-agnostic-bert/)
41. Philippe Tillet, Hsiang-Tsung Kung, and David Cox. 2019. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pages 10–19. [find](https://scholar.google.com/scholar?q=Philippe+Tillet%2C+Hsiang%2DTsung+Kung%2C+and+David+Cox%2E+2019%2E+Triton%3A+an+intermediate+language+and+compiler+for+tiled+neural+network+computations%2E+In+Proceedings+of+the+3rd+ACM+SIGPLAN+International+Worksh) [our card](https://md.research.elamur.ai/free/en/papers/tillet-2019-triton-intermediate-language-compiler-tiled/)
42. Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv:1908.08962. [find](https://scholar.google.com/scholar?q=Iulia+Turc%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+Well%2Dread+students+learn+better%3A+On+the+importance+of+pre%2Dtraining+compact+models%2E+arXiv+preprint+arXiv%3A1908%2E08962%2E) [our card](https://md.research.elamur.ai/free/en/papers/turc-2019-well-read-students-learn-better/)
43. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems (NIPS), 30:5998–6008. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+Advances+in+Neural+Information+Processing+)
44. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Association for Computational Linguistics (ACL), pages 5797–5808. [find](https://scholar.google.com/scholar?q=Elena+Voita%2C+David+Talbot%2C+Fedor+Moiseev%2C+Rico+Sennrich%2C+and+Ivan+Titov%2E+2019%2E+Analyzing+multi%2Dhead+self%2Dattention%3A+Specialized+heads+do+the+heavy+lifting%2C+the+rest+can+be+pruned%2E+In+Association+for+C) [our card](https://md.research.elamur.ai/free/en/papers/voita-2019-analyzing-multi-head-self-attention/)
45. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R+Bowman%2E+2019%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+International+Co) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
46. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020a. MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained Transformers. In Advances in Neural Information Processing Systems (NeurIPS). [find](https://scholar.google.com/scholar?q=Wenhui+Wang%2C+Furu+Wei%2C+Li+Dong%2C+Hangbo+Bao%2C+Nan+Yang%2C+and+Ming+Zhou%2E+2020a%2E+MiniLM%3A+Deep+self%2Dattention+distillation+for+task%2Dagnostic+compression+of+pre%2Dtrained+Transformers%2E+In+Advances+in+Neural+In)
47. Ziheng Wang, Jeremy Wohlwend, and Tao Lei. 2020b. Structured pruning of large language models. In Empirical Methods in Natural Language Processing (EMNLP), pages 6151–6162. [find](https://scholar.google.com/scholar?q=Ziheng+Wang%2C+Jeremy+Wohlwend%2C+and+Tao+Lei%2E+2020b%2E+Structured+pruning+of+large+language+models%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+6151%E2%80%936162%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2020-structured-pruning-large-language-models/)
48. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Neural network acceptability judgments. In Transactions of the Association of Computational Linguistics (TACL), volume 7, pages 625–641. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R+Bowman%2E+2019%2E+Neural+network+acceptability+judgments%2E+In+Transactions+of+the+Association+of+Computational+Linguistics+%28TACL%29%2C+volume+7%2C+pages+625%E2%80%93641%2E)
49. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+North+American+Chapter+of+the+Association+for+Computational+L)
50. Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. 2020. DeeBERT: Dynamic early exiting for accelerating BERT inference. In Association for Computational Linguistics (ACL), pages 2246–2251. [find](https://scholar.google.com/scholar?q=Ji+Xin%2C+Raphael+Tang%2C+Jaejun+Lee%2C+Yaoliang+Yu%2C+and+Jimmy+Lin%2E+2020%2E+DeeBERT%3A+Dynamic+early+exiting+for+accelerating+BERT+inference%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2C+pages+2246%E2%80%932251%2E) [our card](https://md.research.elamur.ai/free/en/papers/xin-2020-deebert-dynamic-early-exiting-accelerating/)
51. Dongkuan Xu, Ian En-Hsu Yen, Jinxi Zhao, and Zhibin Xiao. 2021. Rethinking network pruning–under the pre-train and fine-tune paradigm. In North American Chapter of the Association for Computational Linguistics (NAACL), pages 2376–2382. [find](https://scholar.google.com/scholar?q=Dongkuan+Xu%2C+Ian+En%2DHsu+Yen%2C+Jinxi+Zhao%2C+and+Zhibin+Xiao%2E+2021%2E+Rethinking+network+pruning%E2%80%93under+the+pre%2Dtrain+and+fine%2Dtune+paradigm%2E+In+North+American+Chapter+of+the+Association+for+Computational+Li)
52. Zhewei Yao, Linjian Ma, Sheng Shen, Kurt Keutzer, and Michael W Mahoney. 2021. MLPruning: A multilevel structured pruning framework for transformer-based models. arXiv preprint arXiv:2105.14636. — [arXiv:2105.14636](https://arxiv.org/abs/2105.14636) [find](https://scholar.google.com/scholar?q=Zhewei+Yao%2C+Linjian+Ma%2C+Sheng+Shen%2C+Kurt+Keutzer%2C+and+Michael+W+Mahoney%2E+2021%2E+MLPruning%3A+A+multilevel+structured+pruning+framework+for+transformer%2Dbased+models%2E+arXiv+preprint+arXiv%3A2105%2E14636%2E+%E2%80%94+%5Bar)
53. Yichun Yin, Cheng Chen, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. 2021. AutoTinyBERT: Automatic hyper-parameter optimization for efficient pre-trained language models. In Association for Computational Linguistics (ACL), pages 5146–5157. [find](https://scholar.google.com/scholar?q=Yichun+Yin%2C+Cheng+Chen%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Xiao+Chen%2C+and+Qun+Liu%2E+2021%2E+AutoTinyBERT%3A+Automatic+hyper%2Dparameter+optimization+for+efficient+pre%2Dtrained+language+models%2E+In+Association+for+Comput)
54. Ofir Zafrir, Ariel Larey, Guy Boudoukh, Haihao Shen, and Moshe Wasserblat. 2021. Prune once for all: Sparse pre-trained language models. arXiv preprint arXiv:2111.05754. — [arXiv:2111.05754](https://arxiv.org/abs/2111.05754) [find](https://scholar.google.com/scholar?q=Ofir+Zafrir%2C+Ariel+Larey%2C+Guy+Boudoukh%2C+Haihao+Shen%2C+and+Moshe+Wasserblat%2E+2021%2E+Prune+once+for+all%3A+Sparse+pre%2Dtrained+language+models%2E+arXiv+preprint+arXiv%3A2111%2E05754%2E+%E2%80%94+%5BarXiv%3A2111%2E05754%5D%28https%3A%2F%2Fa)
55. Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. 2019. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Advances in Neural Information Processing Systems (NeurIPS), volume 32. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Hattie+Zhou%2C+Janice+Lan%2C+Rosanne+Liu%2C+and+Jason+Yosinski%2E+2019%2E+Deconstructing+lottery+tickets%3A+Zeros%2C+signs%2C+and+the+supermask%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2C+volume+)