---
title: "On the Effectiveness of Adapter-based Tuning for — He 2021"
description: "Adapter-based tuning inserts small trainable modules into a frozen pretrained language model (PrLM)."
authors: ["Ruidan He", "Linlin Liu", "Hai Ye", "Qingyu Tan", "Bosheng Ding", "Liying Cheng", "Jiawei Low", "Lidong Bing", "Luo Si"]
year: 2021
venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)"
source: "https://aclanthology.org/2021.acl-long.172/"
doi: "10.18653/v1/2021.acl-long.172"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/he-2021-effectiveness-adapter-based-tuning-pretrained/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/he-2021-effectiveness-adapter-based-tuning-pretrained/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation

# On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation

**Authors:** [Ruidan He](https://md.research.elamur.ai/free/en/authors/ruidan-he/), [Linlin Liu](https://md.research.elamur.ai/free/en/authors/linlin-liu/), Hai Ye, Qingyu Tan, [Bosheng Ding](https://md.research.elamur.ai/free/en/authors/bosheng-ding/), Liying Cheng, Jiawei Low, [Lidong Bing](https://md.research.elamur.ai/free/en/authors/lidong-bing/), Luo Si · **Year:** 2021 · **Venue:** Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.acl-long.172/) · [PDF](https://aclanthology.org/2021.acl-long.172.pdf) · [DOI 10.18653/v1/2021.acl-long.172](https://doi.org/10.18653/v1/2021.acl-long.172) · [OpenAlex](https://openalex.org/W3176693010)

**Citations:** 128 citations (openalex-snapshot · 2026-10-14) · [76](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Adapter-based tuning inserts small trainable modules into a frozen pretrained language model (PrLM). Earlier work valued it mainly for saving parameters. This paper asks a different question: when does it actually beat full fine-tuning? The authors use the Houlsby et al. adapter design. Each transformer layer gets two bottleneck adapters with skip connections, and only the adapters, layer norms and classifier are trained. Representational Similarity Analysis (RSA) on BERT-base shows that adapter-tuned representations stay closer to those of the original model than fine-tuned ones, which suggests less catastrophic forgetting. The two methods are then compared on several settings: the eight TAE classification datasets (four domains), GLUE, zero-shot cross-lingual tasks, and stability analyses. Adapters perform better in low-resource and cross-lingual scenarios, match fine-tuning when data is plentiful, and are less sensitive to learning rate. They also show flatter loss minima, a sign of better generalization. The practical lesson is that adapters can serve as an effective regularizer, not merely a way to save storage.

## Key points
- **RSA forgetting analysis:** BERT-base was tuned on GLUE subsets. Layer-wise RSA used 5000 randomly sampled dev-set tokens, cosine similarity matrices and Pearson correlation. Changes from tuning concentrate in the top layers, and adapters deviate less from the base model at every layer.
- **TAE results with RoBERTa:** The setup used adapter hidden size 256, five random seeds, and a low-resource threshold of fewer than 5k training examples. On the four low-resource tasks (CHEMPROT, ACL-ARC, SCIERC, HYPERPARTISAN), adapters raised the average score by 1.9%. On RCT, AGNEWS, HELPFULNESS and IMDB, results were roughly on par with fine-tuning.
- **Stability:** Dev scores tracked over training showed better mean results for adapters and smaller gaps between mean and best. Loss-landscape interpolation (α in [−2, 2], 20 points) on CoLA and SST revealed wider, flatter minima for adapters.
- **Comparison with Mixout:** Mixout (probability 0.9) was used as a regularizer. Plain adapter tuning was usually best. Mixout helped fine-tuning only on CoLA and MRPC and tended to hurt adapters.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

76 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 74 not yet.

- [Is GPT-3 a Good Data Annotator?](https://md.research.elamur.ai/free/en/papers/ding-2022-gpt-3-good-data-annotator/)
- [Bridging Argument Quality and Deliberative Quality Annotations with Adapters](https://md.research.elamur.ai/free/en/papers/falk-2023-bridging-argument-quality-deliberative-quality/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{he2021effectiveness,
  title = {{On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation}},
  author = {He, Ruidan and Liu, Linlin and Ye, Hai and Tan, Qingyu and Ding, Bosheng and Cheng, Liying and Low, Jiawei and Bing, Lidong and Si, Luo},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.172},
  url = {https://doi.org/10.18653/v1/2021.acl-long.172}
}
```

- [← previous On Releasing Annotator-Level Labels and Information in Datasets](https://md.research.elamur.ai/free/en/papers/prabhakaran-2021-releasing-annotator-level-labels-information/)
- [next → On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

46 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Samira Abnar, Lisa Beinborn, Rochelle Choenni, and Jelle Zuidema. 2019. Blackbox meets blackbox: Representational similarity and stability analysis of neural language models and brains. In Proceedings of the ACL-Workshop on Analyzing and Interpreting Neural Networks for NLP. [find](https://scholar.google.com/scholar?q=Samira+Abnar%2C+Lisa+Beinborn%2C+Rochelle+Choenni%2C+and+Jelle+Zuidema%2E+2019%2E+Blackbox+meets+blackbox%3A+Representational+similarity+and+stability+analysis+of+neural+language+models+and+brains%2E+In+Proceedings)
2. Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. 2021. Better fine-tuning by reducing representational collapse. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Armen+Aghajanyan%2C+Akshat+Shrivastava%2C+Anchit+Gupta%2C+Naman+Goyal%2C+Luke+Zettlemoyer%2C+and+Sonal+Gupta%2E+2021%2E+Better+fine%2Dtuning+by+reducing+representational+collapse%2E+In+Proceedings+of+ICLR%2E)
3. Gaurav Arora, Afshin Rahimi, and Timothy Baldwin. 2019. Does an LSTM forget more than a cnn? an empirical study of catastrophic forgetting in NLP. In Proceedings of ALTA. [find](https://scholar.google.com/scholar?q=Gaurav+Arora%2C+Afshin+Rahimi%2C+and+Timothy+Baldwin%2E+2019%2E+Does+an+LSTM+forget+more+than+a+cnn%3F+an+empirical+study+of+catastrophic+forgetting+in+NLP%2E+In+Proceedings+of+ALTA%2E)
4. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Sebastian+Ruder%2C+and+Dani+Yogatama%2E+2020%2E+On+the+cross%2Dlingual+transferability+of+monolingual+representations%2E+In+Proceedings+of+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/)
5. Ankur Bapna and Orhan Firat. 2019. Simple, scalable adaptation for neural machine translation. In Proceedings of EMNLP-IJCNLP. [find](https://scholar.google.com/scholar?q=Ankur+Bapna+and+Orhan+Firat%2E+2019%2E+Simple%2C+scalable+adaptation+for+neural+machine+translation%2E+In+Proceedings+of+EMNLP%2DIJCNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/bapna-2019-simple-scalable-adaptation-neural-machine/)
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretchen)
7. Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. 2020. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Sanyuan+Chen%2C+Yutai+Hou%2C+Yiming+Cui%2C+Wanxiang+Che%2C+Ting+Liu%2C+and+Xiangzhan+Yu%2E+2020%2E+Recall+and+learn%3A+Fine%2Dtuning+deep+pretrained+language+models+with+less+forgetting%2E+In+Proceedings+of+EMNLP%2E)
8. Grzegorz Chrupała and Afra Alishahi. 2019. Correlating neural and symbolic representations of language. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Grzegorz+Chrupa%C5%82a+and+Afra+Alishahi%2E+2019%2E+Correlating+neural+and+symbolic+representations+of+language%2E+In+Proceedings+of+ACL%2E)
9. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020a. Unsupervised cross-lingual representation learning at scale. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzma%CC%81n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020a%2E+Unsupervised+cross%2Dlin)
10. Alexis Conneau, Ruty Rinott, Guillaume Lample, Holger Schwenk, Ves Stoyanov, Adina Williams, and Samuel R. Bowman. 2020b. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Holger+Schwenk%2C+Ves+Stoyanov%2C+Adina+Williams%2C+and+Samuel+R%2E+Bowman%2E+2020b%2E+Xnli%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+EMN)
11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+NAACL%2E)
12. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah A. Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. CoRR. [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Gabriel+Ilharco%2C+Roy+Schwartz%2C+Ali+Farhadi%2C+Hannaneh+Hajishirzi%2C+and+Noah+A%2E+Smith%2E+2020%2E+Fine%2Dtuning+pretrained+language+models%3A+Weight+initializations%2C+data+orders%2C+and+early+stopping%2E+)
13. Robert M French. 1999. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128–135. [find](https://scholar.google.com/scholar?q=Robert+M+French%2E+1999%2E+Catastrophic+forgetting+in+connectionist+networks%2E+Trends+in+cognitive+sciences%2C+3%284%29%3A128%E2%80%93135%2E)
14. Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. 2013. An empirical investigation of catastrophic forgetting in gradient-based neural networks. CoRR. [find](https://scholar.google.com/scholar?q=Ian+J+Goodfellow%2C+Mehdi+Mirza%2C+Da+Xiao%2C+Aaron+Courville%2C+and+Yoshua+Bengio%2E+2013%2E+An+empirical+investigation+of+catastrophic+forgetting+in+gradient%2Dbased+neural+networks%2E+CoRR%2E)
15. Junliang Guo, Zhirui Zhang, Linli Xu, Hao-Ran Wei, Boxing Chen, and Enhong Chen. 2020. Incorporating BERT into parallel sequence decoding with adapters. In Proceedings of NeurIPS. [find](https://scholar.google.com/scholar?q=Junliang+Guo%2C+Zhirui+Zhang%2C+Linli+Xu%2C+Hao%2DRan+Wei%2C+Boxing+Chen%2C+and+Enhong+Chen%2E+2020%2E+Incorporating+BERT+into+parallel+sequence+decoding+with+adapters%2E+In+Proceedings+of+NeurIPS%2E)
16. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Ana+Marasovic%CC%81%2C+Swabha+Swayamdipta%2C+Kyle+Lo%2C+Iz+Beltagy%2C+Doug+Downey%2C+and+Noah+A%2E+Smith%2E+2020%2E+Don%E2%80%99t+stop+pretraining%3A+Adapt+language+models+to+domains+and+tasks%2E+In+Proceedings+of+)
17. Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2019. Visualizing and understanding the effectiveness of BERT. In Proceedings of the EMNLP-IJCNLP. [find](https://scholar.google.com/scholar?q=Yaru+Hao%2C+Li+Dong%2C+Furu+Wei%2C+and+Ke+Xu%2E+2019%2E+Visualizing+and+understanding+the+effectiveness+of+BERT%2E+In+Proceedings+of+the+EMNLP%2DIJCNLP%2E)
18. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Flat minima. Neural Comput., 9(1):1–42. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Flat+minima%2E+Neural+Comput%2E%2C+9%281%29%3A1%E2%80%9342%2E)
19. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of ICML. [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+De+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+NLP%2E+In+)
20. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+Proceedings+of+ACL%2E)
21. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Proceedings of ICML. [find](https://scholar.google.com/scholar?q=Junjie+Hu%2C+Sebastian+Ruder%2C+Aditya+Siddhant%2C+Graham+Neubig%2C+Orhan+Firat%2C+and+Melvin+Johnson%2E+2020%2E+XTREME%3A+A+massively+multilingual+multi%2Dtask+benchmark+for+evaluating+cross%2Dlingual+generalisation%2E+In)
22. Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. SMART: robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization. In Proceedings ACL. [find](https://scholar.google.com/scholar?q=Haoming+Jiang%2C+Pengcheng+He%2C+Weizhu+Chen%2C+Xiaodong+Liu%2C+Jianfeng+Gao%2C+and+Tuo+Zhao%2E+2020%2E+SMART%3A+robust+and+efficient+fine%2Dtuning+for+pre%2Dtrained+natural+language+models+through+principled+regularized) [our card](https://md.research.elamur.ai/free/en/papers/jiang-2020-smart-robust-efficient-fine-tuning/)
23. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+Proceedings+of+ICLR%2E)
24. Aarre Laakso and Garrison Cottrell. 2000. Content and cluster analysis: assessing representational similarity in neural systems. Philosophical psychology, 13(1):47–76. [find](https://scholar.google.com/scholar?q=Aarre+Laakso+and+Garrison+Cottrell%2E+2000%2E+Content+and+cluster+analysis%3A+assessing+representational+similarity+in+neural+systems%2E+Philosophical+psychology%2C+13%281%29%3A47%E2%80%9376%2E)
25. Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. 2020. Mixout: Effective regularization to finetune large-scale pretrained language models. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Cheolhyoung+Lee%2C+Kyunghyun+Cho%2C+and+Wanmo+Kang%2E+2020%2E+Mixout%3A+Effective+regularization+to+finetune+large%2Dscale+pretrained+language+models%2E+In+Proceedings+of+ICLR%2E)
26. Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. 2018. Visualizing the loss landscape of neural nets. In Proceedings of NeurIPS. [find](https://scholar.google.com/scholar?q=Hao+Li%2C+Zheng+Xu%2C+Gavin+Taylor%2C+Christoph+Studer%2C+and+Tom+Goldstein%2E+2018%2E+Visualizing+the+loss+landscape+of+neural+nets%2E+In+Proceedings+of+NeurIPS%2E)
27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. CoRR. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+C)
28. Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier. [find](https://scholar.google.com/scholar?q=Michael+McCloskey+and+Neal+J+Cohen%2E+1989%2E+Catastrophic+interference+in+connectionist+networks%3A+The+sequential+learning+problem%2E+In+Psychology+of+learning+and+motivation%2C+volume+24%2C+pages+109%E2%80%93165%2E+Else)
29. Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. 2020. What happens to BERT embeddings during fine-tuning? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. [find](https://scholar.google.com/scholar?q=Amil+Merchant%2C+Elahe+Rahimtoroghi%2C+Ellie+Pavlick%2C+and+Ian+Tenney%2E+2020%2E+What+happens+to+BERT+embeddings+during+fine%2Dtuning%3F+In+Proceedings+of+the+Third+BlackboxNLP+Workshop+on+Analyzing+and+Interpreti) [our card](https://md.research.elamur.ai/free/en/papers/merchant-2020-what-happens-bert-embeddings-during/)
30. Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021. On the stability of fine-tuning BERT: misconceptions, explanations, and strong baselines. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Marius+Mosbach%2C+Maksym+Andriushchenko%2C+and+Dietrich+Klakow%2E+2021%2E+On+the+stability+of+fine%2Dtuning+BERT%3A+misconceptions%2C+explanations%2C+and+strong+baselines%2E+In+Proceedings+of+ICLR%2E)
31. Lili Mou, Zhao Meng, Rui Yan, Ge Li, Yan Xu, Lu Zhang, and Zhi Jin. 2016. How transferable are neural networks in NLP applications? In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Lili+Mou%2C+Zhao+Meng%2C+Rui+Yan%2C+Ge+Li%2C+Yan+Xu%2C+Lu+Zhang%2C+and+Zhi+Jin%2E+2016%2E+How+transferable+are+neural+networks+in+NLP+applications%3F+In+Proceedings+of+EMNLP%2E)
32. Joakim Nivre, Rogier Blokland, Niko Partanen, and Michael Rießler. 2018. Universal dependencies 2.2. [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Rogier+Blokland%2C+Niko+Partanen%2C+and+Michael+Rie%C3%9Fler%2E+2018%2E+Universal+dependencies+2%2E2%2E)
33. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Xiaoman+Pan%2C+Boliang+Zhang%2C+Jonathan+May%2C+Joel+Nothman%2C+Kevin+Knight%2C+and+Heng+Ji%2E+2017%2E+Cross%2Dlingual+name+tagging+and+linking+for+282+languages%2E+In+Proceedings+of+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/pan-2017-cross-lingual-name-tagging-linking/)
34. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2020a. Adapterfusion: Non-destructive task composition for transfer learning. CoRR. [find](https://scholar.google.com/scholar?q=Jonas+Pfeiffer%2C+Aishwarya+Kamath%2C+Andreas+Ru%CC%88ckle%CC%81%2C+Kyunghyun+Cho%2C+and+Iryna+Gurevych%2E+2020a%2E+Adapterfusion%3A+Non%2Ddestructive+task+composition+for+transfer+learning%2E+CoRR%2E) [our card](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterfusion-non-destructive-task-composition/)
35. Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020b. Adapterhub: A framework for adapting transformers. In Proceedings of EMNLP: System Demonstrations. [find](https://scholar.google.com/scholar?q=Jonas+Pfeiffer%2C+Andreas+Ru%CC%88ckle%CC%81%2C+Clifton+Poth%2C+Aishwarya+Kamath%2C+Ivan+Vulic%2C+Sebastian+Ruder%2C+Kyunghyun+Cho%2C+and+Iryna+Gurevych%2E+2020b%2E+Adapterhub%3A+A+framework+for+adapting+transformers%2E+In+Proceedin) [our card](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterhub-framework-adapting-transformers/)
36. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020c. Mad-x: An adapter-based framework for multi-task cross-lingual transfer. CoRR. [find](https://scholar.google.com/scholar?q=Jonas+Pfeiffer%2C+Ivan+Vulic%CC%81%2C+Iryna+Gurevych%2C+and+Sebastian+Ruder%2E+2020c%2E+Mad%2Dx%3A+An+adapter%2Dbased+framework+for+multi%2Dtask+cross%2Dlingual+transfer%2E+CoRR%2E) [our card](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-mad-x-adapter-based-framework/)
37. Jason Phang, Thibault Févry, and Samuel R. Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. CoRR. [find](https://scholar.google.com/scholar?q=Jason+Phang%2C+Thibault+Fe%CC%81vry%2C+and+Samuel+R%2E+Bowman%2E+2018%2E+Sentence+encoders+on+stilts%3A+Supplementary+training+on+intermediate+labeled%2Ddata+tasks%2E+CoRR%2E) [our card](https://md.research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/)
38. Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2020. Adapterdrop: On the efficiency of adapters in transformers. CoRR. [find](https://scholar.google.com/scholar?q=Andreas+Ru%CC%88ckle%CC%81%2C+Gregor+Geigle%2C+Max+Glockner%2C+Tilman+Beck%2C+Jonas+Pfeiffer%2C+Nils+Reimers%2C+and+Iryna+Gurevych%2E+2020%2E+Adapterdrop%3A+On+the+efficiency+of+adapters+in+transformers%2E+CoRR%2E) [our card](https://md.research.elamur.ai/free/en/papers/ruckle-2021-adapterdrop-efficiency-adapters-transformers/)
39. Asa Cooper Stickland and Iain Murray. 2019. BERT and PALs: Projected attention layers for efficient adaptation in multi-task learning. In Proceedings of ICML. [find](https://scholar.google.com/scholar?q=Asa+Cooper+Stickland+and+Iain+Murray%2E+2019%2E+BERT+and+PALs%3A+Projected+attention+layers+for+efficient+adaptation+in+multi%2Dtask+learning%2E+In+Proceedings+of+ICML%2E)
40. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune BERT for text classification? In Proceedings of CCL. [find](https://scholar.google.com/scholar?q=Chi+Sun%2C+Xipeng+Qiu%2C+Yige+Xu%2C+and+Xuanjing+Huang%2E+2019%2E+How+to+fine%2Dtune+BERT+for+text+classification%3F+In+Proceedings+of+CCL%2E) [our card](https://md.research.elamur.ai/free/en/papers/sun-2019-fine-tune-bert-text-classification/)
41. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NeurIPS. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81+ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Proceedings+of+NeurIPS%2E)
42. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+Bowman%2E+2018%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+Proceedings+of+the) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
43. Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2020. K-adapter: Infusing knowledge into pre-trained models with adapters. CoRR. [find](https://scholar.google.com/scholar?q=Ruize+Wang%2C+Duyu+Tang%2C+Nan+Duan%2C+Zhongyu+Wei%2C+Xuanjing+Huang%2C+Jianshu+ji%2C+Guihong+Cao%2C+Daxin+Jiang%2C+and+Ming+Zhou%2E+2020%2E+K%2Dadapter%3A+Infusing+knowledge+into+pre%2Dtrained+models+with+adapters%2E+CoRR%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2020-k-adapter-infusing-knowledge-pre/)
44. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. CoRR. [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pier%2Dric+Cistac%2C+Tim+Rault%2C+Re%CC%81mi+Louf%2C+Morgan+Funtowicz%2C+and+Jamie+Brew%2E+2019%2E+Huggingface%E2%80%99s+transformers%3A+St) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
45. Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Weinberger, and Yoav Artzi. 2021. Revisiting few-sample BERT fine-tuning. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Felix+Wu%2C+Arzoo+Katiyar%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2021%2E+Revisiting+few%2Dsample+BERT+fine%2Dtuning%2E+In+Proceedings+of+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2020-revisiting-few-sample-bert-fine/)
46. Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2020. Freelb: Enhanced adversarial training for natural language understanding. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Chen+Zhu%2C+Yu+Cheng%2C+Zhe+Gan%2C+Siqi+Sun%2C+Tom+Goldstein%2C+and+Jingjing+Liu%2E+2020%2E+Freelb%3A+Enhanced+adversarial+training+for+natural+language+understanding%2E+In+Proceedings+of+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2019-freelb-enhanced-adversarial-training-natural/)