---
title: "Raise a Child in Large Language Model: Towards Effective and"
description: "The authors propose CHILD-TUNING. 38 references, cited by 67 papers here. Among them: 133."
authors: ["Runxin Xu", "Fuli Luo", "Zhiyuan Zhang", "Chuanqi Tan", "Baobao Chang", "Songfang Huang", "Fei Huang"]
year: 2021
venue: "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2021.emnlp-main.749/"
doi: "10.18653/v1/2021.emnlp-main.749"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2021-raise-child-large-language-model/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2021-raise-child-large-language-model/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/) / Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning

# Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning

**Authors:** [Runxin Xu](https://md.research.elamur.ai/free/en/authors/runxin-xu/), [Fuli Luo](https://md.research.elamur.ai/free/en/authors/fuli-luo/), [Zhiyuan Zhang](https://md.research.elamur.ai/free/en/authors/zhiyuan-zhang/), [Chuanqi Tan](https://md.research.elamur.ai/free/en/authors/chuanqi-tan/), [Baobao Chang](https://md.research.elamur.ai/free/en/authors/baobao-chang/), [Songfang Huang](https://md.research.elamur.ai/free/en/authors/songfang-huang/) and [Fei Huang](https://md.research.elamur.ai/free/en/authors/fei-huang/) · **Year:** 2021 · **Venue:** [Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2021](https://md.research.elamur.ai/free/en/venues/emnlp/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.emnlp-main.749/) · [PDF](https://aclanthology.org/2021.emnlp-main.749.pdf) · [DOI 10.18653/v1/2021.emnlp-main.749](https://doi.org/10.18653/v1/2021.emnlp-main.749) · [OpenAlex](https://openalex.org/W3199761064)

**Citations:** 133 citations (OpenAlex, 2026-10-14) · [67](https://md.research.elamur.ai/free/en/papers/xu-2021-raise-child-large-language-model/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
When very large pretrained language models are fine-tuned on small labeled datasets, they tend to overfit and transfer poorly to new domains or related tasks. The authors propose CHILD-TUNING. The forward pass still runs through the whole network. During backpropagation, however, gradients are set to zero for all parameters outside a selected subset, the "child network", so only that subset is updated. This is not pruning, because no weights are removed.

The paper offers two ways to choose the subset:
- **CHILD-TUNING_F** draws a new Bernoulli mask at every step and needs no task data. It acts as gradient noise, which the authors connect theoretically to flatter minima and a tighter generalization bound.
- **CHILD-TUNING_D** ranks parameters by Fisher information estimated on the task's training data and keeps all others frozen at their pretrained values.

On GLUE tasks with four pretrained models, the method outperforms vanilla fine-tuning and earlier regularization approaches. It also generalizes better in domain- and task-transfer settings. Because it only adds a mask step inside the optimizer, it is cheap and can be combined with other fine-tuning methods.

## Key points
- Compared with vanilla fine-tuning, average scores rise by 1.5–8.6 points across four PLMs, with the largest gains (up to 8.60) on CoLA/RTE/MRPC/STS-B. The method also beats prior techniques such as RecAdam and Mixout by 0.6–1.3 points.
- The method is implemented by inserting a gradient-masking step into Adam. Setting p_F = 1 in CHILD-TUNING_F recovers ordinary fine-tuning, and kept gradients are rescaled by 1/p_F so their expected value is unchanged.
- In ablations of CHILD-TUNING_D, a randomly chosen fixed child network still beats vanilla fine-tuning by 0.18 on average. Choosing the lowest-Fisher parameters instead of the highest lowers the average score by 6.65.
- Measured by Jaccard similarity, child networks overlap more for similar tasks, such as the NLI tasks RTE, QNLI and MNLI. Single-sentence tasks SST2 and CoLA overlap mainly with each other.
- CHILD-TUNING adds no extra parameters. The task-driven variant only needs one Fisher Information Matrix estimate before training, and the approach complements other fine-tuning techniques.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 100 works · 35,618 citations
- [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) — Company, Hangzhou, China · 64 works · 14,113 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works

## Cite

Cite as: Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang and Fei Huang. 2021. Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xu2021raise,
  title = {{Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning}},
  author = {Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang and Fei Huang},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.749},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.749}
}
```

- [← previous HateCheck: Functional Tests for Hate Speech Detection Models](https://md.research.elamur.ai/free/en/papers/rottger-2021-hatecheck-functional-tests-hate-speech/)
- [next → Block Pruning For Faster Transformers](https://md.research.elamur.ai/free/en/papers/lagunas-2021-block-pruning-faster-transformers/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/)

## References

38 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. 2021. Better fine-tuning by reducing representational collapse. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Armen+Aghajanyan%2C+Akshat+Shrivastava%2C+Anchit+Gupta%2C+Naman+Goyal%2C+Luke+Zettlemoyer%2C+and+Sonal+Gupta%2E+2021%2E+Better+fine%2Dtuning+by+reducing+representational+collapse%2E+In+International+Conference+on+Learn)
2. Parameter-efficient Fine-tuning. There are also studies focusing on parameter-efficient fine-tuning, for example, the adapter-based methods (Houlsby et al., 2019; Pfeiffer et al., Samuel R. Bowman, Gabor Angeli, Christopher Potts, 2020; Karimi Mahabadi et al., 2021), and the and Christopher D. Manni… [find](https://scholar.google.com/scholar?q=Parameter%2Defficient+Fine%2Dtuning%2E+There+are+also+studies+focusing+on+parameter%2Defficient+fine%2Dtuning%2C+for+example%2C+the+adapter%2Dbased+methods+%28Houlsby+et+al%2E%2C+2019%3B+Pfeiffer+et+al%2E%2C+Samuel+R%2E+Bowman%2C+Ga)
3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
4. Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. 2020. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Sanyuan+Chen%2C+Yutai+Hou%2C+Yiming+Cui%2C+Wanxiang+Che%2C+Ting+Liu%2C+and+Xiangzhan+Yu%2E+2020%2E+Recall+and+learn%3A+Fine%2Dtuning+deep+pretrained+language+models+with+less+forgetting%2E+In+Proceedings+of+the+2020+Conf)
5. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V%2E+Le%2C+and+Christopher+D%2E+Manning%2E+2020%2E+Electra%3A+Pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+In+International+Conference+on+Learning+Repre)
6. Hal Daumé III. 2007. [Frustratingly easy domain adaptation](https://md.research.elamur.ai/free/en/papers/iii-2007-frustratingly-easy-domain-adaptation/). In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics (ACL).
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
8. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah A. Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint, arXiv:2002.06305. [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Gabriel+Ilharco%2C+Roy+Schwartz%2C+Ali+Farhadi%2C+Hannaneh+Hajishirzi%2C+and+Noah+A%2E+Smith%2E+2020%2E+Fine%2Dtuning+pretrained+language+models%3A+Weight+initializations%2C+data+orders%2C+and+early+stopping%2E+)
9. Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. 2021. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Pierre+Foret%2C+Ariel+Kleiner%2C+Hossein+Mobahi%2C+and+Behnam+Neyshabur%2E+2021%2E+Sharpness%2Daware+minimization+for+efficiently+improving+generalization%2E+In+International+Conference+on+Learning+Representations+)
10. Yichen Gong, Heng Luo, and Jian Zhang. 2018. Natural language inference over interaction space. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Yichen+Gong%2C+Heng+Luo%2C+and+Jian+Zhang%2E+2018%2E+Natural+language+inference+over+interaction+space%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
11. Henry Gouk, Timothy M. Hospedales, and Massimiliano Pontil. 2020. Distance-based regularisation of deep networks for fine-tuning. arXiv preprint, arXiv:2002.08253. — [arXiv:2002.08253](https://arxiv.org/abs/2002.08253) [find](https://scholar.google.com/scholar?q=Henry+Gouk%2C+Timothy+M%2E+Hospedales%2C+and+Massimiliano+Pontil%2E+2020%2E+Distance%2Dbased+regularisation+of+deep+networks+for+fine%2Dtuning%2E+arXiv+preprint%2C+arXiv%3A2002%2E08253%2E+%E2%80%94+%5BarXiv%3A2002%2E08253%5D%28https%3A%2F%2Farxiv%2Eo)
12. Demi Guo, Alexander Rush, and Yoon Kim. 2021. [Parameter-efficient transfer learning with diff pruning](https://md.research.elamur.ai/free/en/papers/guo-2021-parameter-efficient-transfer-learning-diff/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL).
13. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning (ICML). [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+De+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+NLP%2E+In+)
14. Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. [SMART: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization](https://md.research.elamur.ai/free/en/papers/jiang-2020-smart-robust-efficient-fine-tuning/). In Proceedings of the 58th Annual Meeting of the Association for Computational Lingui…
15. Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. 2021. [Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks](https://md.research.elamur.ai/free/en/papers/mahabadi-2021-parameter-efficient-multi-task-fine/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL).
16. Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. 2017. On large-batch training for deep learning: Generalization gap and sharp minima. In 5th International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Nitish+Shirish+Keskar%2C+Dheevatsa+Mudigere%2C+Jorge+Nocedal%2C+Mikhail+Smelyanskiy%2C+and+Ping+Tak+Peter+Tang%2E+2017%2E+On+large%2Dbatch+training+for+deep+learning%3A+Generalization+gap+and+sharp+minima%2E+In+5th+Int)
17. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In Proceedings of the Thirty-Second Conference on Artificial Intelligence (AAAI). [find](https://scholar.google.com/scholar?q=Tushar+Khot%2C+Ashish+Sabharwal%2C+and+Peter+Clark%2E+2018%2E+Scitail%3A+A+textual+entailment+dataset+from+science+question+answering%2E+In+Proceedings+of+the+Thirty%2DSecond+Conference+on+Artificial+Intelligence+%28)
18. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
19. James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2016. Overcoming catastrophic forgetting in neural netwo… [find](https://scholar.google.com/scholar?q=James+Kirkpatrick%2C+Razvan+Pascanu%2C+Neil+C%2E+Rabinowitz%2C+Joel+Veness%2C+Guillaume+Desjardins%2C+Andrei+A%2E+Rusu%2C+Kieran+Milan%2C+John+Quan%2C+Tiago+Ramalho%2C+Agnieszka+Grabska%2DBarwinska%2C+Demis+Hassabis%2C+Claudia+C)
20. Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. 2020. Mixout: Effective regularization to finetune large-scale pretrained language models. In 8th International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Cheolhyoung+Lee%2C+Kyunghyun+Cho%2C+and+Wanmo+Kang%2E+2020%2E+Mixout%3A+Effective+regularization+to+finetune+large%2Dscale+pretrained+language+models%2E+In+8th+International+Conference+on+Learning+Representations+%28)
21. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the…
22. Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. 2017. Pruning filters for efficient convnets. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Hao+Li%2C+Asim+Kadav%2C+Igor+Durdanovic%2C+Hanan+Samet%2C+and+Hans+Peter+Graf%2E+2017%2E+Pruning+filters+for+efficient+convnets%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
23. Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. 2020. Dynamic model pruning with feedback. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Tao+Lin%2C+Sebastian+U%2E+Stich%2C+Luis+Barba%2C+Daniil+Dmitriev%2C+and+Martin+Jaggi%2E+2020%2E+Dynamic+model+pruning+with+feedback%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
24. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint, arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+BERT+pretraining+approach%2E+a)
25. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
26. Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. 2021. Variational information bottleneck for effective low-resource fine-tuning. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Rabeeh+Karimi+Mahabadi%2C+Yonatan+Belinkov%2C+and+James+Henderson%2E+2021%2E+Variational+information+bottleneck+for+effective+low%2Dresource+fine%2Dtuning%2E+In+International+Conference+on+Learning+Representations+)
27. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. [A SICK cure for the evaluation of compositional distributional semantic models](https://md.research.elamur.ai/free/en/papers/marelli-2014-sick-cure-evaluation-compositional-distributional/). In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC).
28. Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021. On the stability of fine-tuning BERT: Misconceptions, explanations, and strong baselines. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Marius+Mosbach%2C+Maksym+Andriushchenko%2C+and+Dietrich+Klakow%2E+2021%2E+On+the+stability+of+fine%2Dtuning+BERT%3A+Misconceptions%2C+explanations%2C+and+strong+baselines%2E+In+International+Conference+on+Learning+Repr)
29. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-mad-x-adapter-based-framework/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
30. Jason Phang, Thibault Févry, and Samuel R. Bowman. 2018. [Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks](https://md.research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/). arXiv preprint, arXiv:1811.01088.
31. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2018%2E+Language+models+are+unsupervised+multitask+learners%2E)
32. Xu Sun, Zhiyuan Zhang, Xuancheng Ren, Ruixuan Luo, and Liangyou Li. 2020. Exploring the vulnerability of deep neural networks: A study of parameter corruption. arXiv preprint, arXiv:2006.05620. — [arXiv:2006.05620](https://arxiv.org/abs/2006.05620) [find](https://scholar.google.com/scholar?q=Xu+Sun%2C+Zhiyuan+Zhang%2C+Xuancheng+Ren%2C+Ruixuan+Luo%2C+and+Liangyou+Li%2E+2020%2E+Exploring+the+vulnerability+of+deep+neural+networks%3A+A+study+of+parameter+corruption%2E+arXiv+preprint%2C+arXiv%3A2006%2E05620%2E+%E2%80%94+%5BarX)
33. M. Tu, V. Berisha, Y. Cao, and J. Seo. 2016a. Reducing the model order of deep neural networks using information theory. In 2016 IEEE Computer Society Annual Symposium on VLSI (ISVLSI). [find](https://scholar.google.com/scholar?q=M%2E+Tu%2C+V%2E+Berisha%2C+Y%2E+Cao%2C+and+J%2E+Seo%2E+2016a%2E+Reducing+the+model+order+of+deep+neural+networks+using+information+theory%2E+In+2016+IEEE+Computer+Society+Annual+Symposium+on+VLSI+%28ISVLSI%29%2E)
34. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In International Conference on Learning Representations (ICLR).
35. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
36. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). In Advances in Neural Information Processing Systems (NeurIPS).
37. Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. 2021. [Revisiting few-sample BERT fine-tuning](https://md.research.elamur.ai/free/en/papers/zhang-2020-revisiting-few-sample-bert-fine/). In International Conference on Learning Representations (ICLR).
38. Michael Zhu and Suyog Gupta. 2018. To prune, or not to prune: Exploring the efficacy of pruning for model compression. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Michael+Zhu+and+Suyog+Gupta%2E+2018%2E+To+prune%2C+or+not+to+prune%3A+Exploring+the+efficacy+of+pruning+for+model+compression%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)

*Updated 24 October 2026*