---
title: "AdapterHub: A Framework for Adapting — Pfeiffer 2020"
description: "Fine-tuning large pre-trained transformers separately for every task produces a full model copy each time."
authors: ["Jonas Pfeiffer", "Andreas Rücklé", "Clifton Poth", "Aishwarya Kamath", "Ivan Vulić", "Sebastian Ruder", "Kyunghyun Cho", "Iryna Gurevych"]
year: 2020
venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations"
source: "https://aclanthology.org/2020.emnlp-demos.7/"
doi: "10.18653/v1/2020.emnlp-demos.7"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pfeiffer-2020-adapterhub-framework-adapting-transformers/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pfeiffer-2020-adapterhub-framework-adapting-transformers/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/) / AdapterHub: A Framework for Adapting Transformers

# AdapterHub: A Framework for Adapting Transformers

**Authors:** [Jonas Pfeiffer](https://md.research.elamur.ai/free/en/authors/jonas-pfeiffer/), [Andreas Rücklé](https://md.research.elamur.ai/free/en/authors/andreas-ruckle/), Clifton Poth, [Aishwarya Kamath](https://md.research.elamur.ai/free/en/authors/aishwarya-kamath/), [Ivan Vulić](https://md.research.elamur.ai/free/en/authors/ivan-vulic/), [Sebastian Ruder](https://md.research.elamur.ai/free/en/authors/sebastian-ruder/), [Kyunghyun Cho](https://md.research.elamur.ai/free/en/authors/kyunghyun-cho/), [Iryna Gurevych](https://md.research.elamur.ai/free/en/authors/iryna-gurevych/) · **Year:** 2020 · **Venue:** Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-demos.7/) · [PDF](https://aclanthology.org/2020.emnlp-demos.7.pdf) · [DOI 10.18653/v1/2020.emnlp-demos.7](https://doi.org/10.18653/v1/2020.emnlp-demos.7) · [OpenAlex](https://openalex.org/W3042667808)

**Citations:** 461 citations (openalex-snapshot · 2026-10-14) · [286](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Fine-tuning large pre-trained transformers separately for every task produces a full model copy each time. These copies are costly to store, slow to share and hard to combine. Adapters address this. They are small bottleneck modules added to each layer and trained while the base model stays frozen. In practice, though, they were seldom released on their own, and their designs differed in subtle ways. AdapterHub, built on HuggingFace Transformers, standardizes how adapters are trained, saved, uploaded and loaded for models such as BERT, RoBERTa and XLM-R. Its configuration files express the known adapter architecture variants. Its public website organizes shared adapters by task, language and dataset. Training an adapter takes about two extra lines in an existing script, and loading a published one takes a single line. On GLUE with BERT-Base, two adapter designs score close to full fine-tuning while needing only megabytes per task. This makes multi-task, cross-lingual and low-resource transfer research cheaper, more reproducible and easier to build on.

## Key points
- **GLUE comparison (BERT-Base, mean of 3 runs, bottleneck 48):**
  - Full fine-tuning was compared with the Pfeiffer et al. (2020a) adapter and the Houlsby et al. (2019) adapter.
  - The scores are broadly comparable. For example, on RTE the scores are 66.2 (full), 70.8 (Pfeiffer) and 69.8 (Houlsby). On MNLI all three reach 84.1.
- **Adapter size (Pfeiffer design):**
  - At a compression rate of 64, an adapter adds about 0.2M parameters (0.9Mb) on a Base model and 0.8M (3.2Mb) on a Large model.
  - At a rate of 2, it grows to 7.1M parameters (28Mb) on Base and 25.2M (97Mb) on Large.
  - The Houlsby variant needs roughly twice the space.
- **Storage comparison:**
  - BERT-Base is about 440Mb, and XLM-R Large is about 2.2Gb compressed.
  - The disk space of two fully fine-tuned BERT-Base copies can hold about 125 task adapters.
  - Over 99% of the parameters stay shared and frozen.
- **Framework features:**
  - Adapter weights are extracted and saved separately from the base model.
  - Prediction heads can optionally be uploaded alongside the adapter.
  - Stacking and composition of adapters are supported out of the box.
- **Website and compatibility:**
  - The AdapterHub.ml site browses adapters through three levels: task or language, then task category or training language, then dataset or domain (e.g., SST, Swahili Wikipedia).
  - Adapters only work with the base model they were trained on.

## Interesting ideas
- **Describe architecture variants in a declarative config.** Placement choices and extra LayerNorms are set in configuration rather than in code. This lets one codebase reproduce many published variants and lets users fairly compare their own variants, which suits any family of small design tweaks.
- **Ship the delta, not the whole model.** Saving only the trained adapter keeps per-task artifacts tiny. This makes it practical to maintain many task- or language-specific models, for example for Ukrainian tasks, on limited storage or mobile devices.
- **Exploit frozen-backbone compatibility.** Each adapter must produce output that fits the frozen next layer. Because of this, independently trained task and language adapters can be stacked or swapped. One could train a language adapter once and reuse it across several task adapters.
- **Organize shared models hierarchically.** Indexing models by task or language, then by subcategory, then by dataset or domain, and filtering by base model, makes a shared library easy to search. The same structure could organize the Elamur library's own models and datasets.
- **Keep integration cost to a line or two.** Requiring minimal changes to existing training scripts lowers the barrier to adoption. This is worth copying when introducing any internal tool to collaborators.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

286 papers of our library cite it (of the 140884 whose bibliographies we read); 17 of them have a page here, 269 not yet.

- [The eBible Corpus: Data and Model Benchmarks for Bible Translation for Low-Resource Languages](https://md.research.elamur.ai/free/en/papers/akerman-2023-ebible-corpus-data-model-benchmarks/)
- [Bridging Argument Quality and Deliberative Quality Annotations with Adapters](https://md.research.elamur.ai/free/en/papers/falk-2023-bridging-argument-quality-deliberative-quality/)
- [Applying Occam's Razor to Transformer-Based Dependency Parsing: What Works, What Doesn't, and What is Really Necessary](https://md.research.elamur.ai/free/en/papers/grunewald-2021-applying-occam-s-razor-transformer/)
- [On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation](https://md.research.elamur.ai/free/en/papers/he-2021-effectiveness-adapter-based-tuning-pretrained/)
- [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/)
- [Sustainable Modular Debiasing of Language Models](https://md.research.elamur.ai/free/en/papers/lauscher-2021-sustainable-modular-debiasing-language-models/)
- [DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rules](https://md.research.elamur.ai/free/en/papers/liu-2023-dada-dialect-adaptation-dynamic-aggregation/)
- [Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks](https://md.research.elamur.ai/free/en/papers/mahabadi-2021-parameter-efficient-multi-task-fine/)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterfusion-non-destructive-task-composition/)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-mad-x-adapter-based-framework/)
- [UNKs Everywhere: Adapting Multilingual Language Models to New Scripts](https://md.research.elamur.ai/free/en/papers/pfeiffer-2021-unks-everywhere-adapting-multilingual-language/)
- [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](https://md.research.elamur.ai/free/en/papers/pfeiffer-2022-lifting-curse-multilinguality-pre-training/)
- [AdapterDrop: On the Efficiency of Adapters in Transformers](https://md.research.elamur.ai/free/en/papers/ruckle-2021-adapterdrop-efficiency-adapters-transformers/)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/)
- [Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing](https://md.research.elamur.ai/free/en/papers/ud-2021-nguyen-trankit-light-weight-transformer-based/)
- [NRD: A Hybrid Disentanglement Framework for Mitigating Interference in Multilingual Machine Translation](https://md.research.elamur.ai/free/en/papers/zhang-2026-nrd-hybrid-disentanglement-framework-mitigating/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{pfeiffer2020adapterhub,
  title = {{AdapterHub: A Framework for Adapting Transformers}},
  author = {Pfeiffer, Jonas and Rücklé, Andreas and Poth, Clifton and Kamath, Aishwarya and Vulić, Ivan and Ruder, Sebastian and Cho, Kyunghyun and Gurevych, Iryna},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-demos.7},
  url = {https://doi.org/10.18653/v1/2020.emnlp-demos.7}
}
```

- [← previous AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterfusion-non-destructive-task-composition/)
- [next → Adapting BERT to Implicit Discourse Relation Classification with a Focus on Discourse Connectives](https://md.research.elamur.ai/free/en/papers/kishimoto-2020-adapting-bert-implicit-discourse-relation/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/)

## References

39 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Learning to compose neural networks for question answering. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, U… [find](https://scholar.google.com/scholar?q=Jacob+Andreas%2C+Marcus+Rohrbach%2C+Trevor+Darrell%2C+and+Dan+Klein%2E+2016%2E+Learning+to+compose+neural+networks+for+question+answering%2E+In+NAACL+HLT+2016%2C+The+2016+Conference+of+the+North+American+Chapter+of) [our card](https://md.research.elamur.ai/free/en/papers/andreas-2016-learning-compose-neural-networks-question/)
2. Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. arXiv preprint. [find](https://scholar.google.com/scholar?q=Lei+Jimmy+Ba%2C+Jamie+Ryan+Kiros%2C+and+Geoffrey+E%2E+Hinton%2E+2016%2E+Layer+normalization%2E+arXiv+preprint%2E) [our card](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/)
3. William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing, IWP@IJCNLP 2005, Jeju Island, Korea, October 2005, 2005. [find](https://scholar.google.com/scholar?q=William+B%2E+Dolan+and+Chris+Brockett%2E+2005%2E+Automatically+constructing+a+corpus+of+sentential+paraphrases%2E+In+Proceedings+of+the+Third+International+Workshop+on+Paraphrasing%2C+IWP%40IJCNLP+2005%2C+Jeju+Isla) [our card](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/)
4. Ankur Bapna and Orhan Firat. 2019. Simple, scalable adaptation for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, Nov… [find](https://scholar.google.com/scholar?q=Ankur+Bapna+and+Orhan+Firat%2E+2019%2E+Simple%2C+scalable+adaptation+for+neural+machine+translation%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+Int) [our card](https://md.research.elamur.ai/free/en/papers/bapna-2019-simple-scalable-adaptation-neural-machine/)
5. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the PASCAL@ACL 2007. [find](https://scholar.google.com/scholar?q=Danilo+Giampiccolo%2C+Bernardo+Magnini%2C+Ido+Dagan%2C+and+Bill+Dolan%2E+2007%2E+The+third+pascal+recognizing+textual+entailment+challenge%2E+In+Proceedings+of+the+PASCAL%40ACL+2007%2E)
6. Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10… [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Ana+Marasovic%2C+Swabha+Swayamdipta%2C+Kyle+Lo%2C+Iz+Beltagy%2C+Doug+Downey%2C+and+Noah+A%2E+Smith%2E+2020%2E+Don%E2%80%99t+stop+pretraining%3A+Adapt+language+models+to+domains+and+tasks%2E+In+Proceedings+of+t)
7. Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second pascal recognising textual entailment challenge. In Proceedings of the PASCAL@ACL 2006. [find](https://scholar.google.com/scholar?q=Roy+Bar%2DHaim%2C+Ido+Dagan%2C+Bill+Dolan%2C+Lisa+Ferro%2C+Danilo+Giampiccolo%2C+Bernardo+Magnini%2C+and+Idan+Szpektor%2E+2006%2E+The+second+pascal+recognising+textual+entailment+challenge%2E+In+Proceedings+of+the+PASCAL)
8. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche) [our card](https://md.research.elamur.ai/free/en/papers/hashimoto-2016-joint-many-task-model-growing/)
9. Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Ass… [find](https://scholar.google.com/scholar?q=Yada+Pruksachatkun%2C+Jason+Phang%2C+Haokun+Liu%2C+Phu+Mon+Htut%2C+Xiaoyi+Zhang%2C+Richard+Yuanzhe+Pang%2C+Clara+Vania%2C+Katharina+Kann%2C+and+Samuel+R%2E+Bowman%2E+2020%2E+Intermediate%2Dtask+transfer+learning+with+pretrai) [our card](https://md.research.elamur.ai/free/en/papers/pruksachatkun-2020-intermediate-task-transfer-learning-pretrained/)
10. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzkebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Lo… [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzkebski%2C+Bruna+Morrone%2C+Quentin+de+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+NLP%2E+In)
11. Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 328–339. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+Language+Model+Fine%2Dtuning+for+Text+Classification%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+ACL+2)
12. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv preprint. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+2019%2E+Exploring+the+Limits+of+Transfer+Learning+with+a+Unified+Text%2Dto%2DTex)
13. Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. First quora dataset release: Question pairs [online]. 2017. [find](https://scholar.google.com/scholar?q=Shankar+Iyer%2C+Nikhil+Dandekar%2C+and+Kornel+Csernai%2E+First+quora+dataset+release%3A+Question+pairs+%5Bonline%5D%2E+2017%2E)
14. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383–2392. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+Questions+for+Machine+Comprehension+of+Text%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natu) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
15. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. arXiv preprint. [find](https://scholar.google.com/scholar?q=Jared+Kaplan%2C+Sam+McCandlish%2C+Tom+Henighan%2C+Tom+B%2E+Brown%2C+Benjamin+Chess%2C+Rewon+Child%2C+Scott+Gray%2C+Alec+Radford%2C+Jeffrey+Wu%2C+and+Dario+Amodei%2E+2020%2E+Scaling+Laws+for+Neural+Language+Models%2E+arXiv+prep)
16. Anne Lauscher, Ivan Vulić, Edoardo Maria Ponti, Anna Korhonen, and Goran Glavaš. 2019. Specializing unsupervised pretraining models for word-level semantic similarity. arXiv preprint. [find](https://scholar.google.com/scholar?q=Anne+Lauscher%2C+Ivan+Vulic%CC%81%2C+Edoardo+Maria+Ponti%2C+Anna+Korhonen%2C+and+Goran+Glavas%CC%8C%2E+2019%2E+Specializing+unsupervised+pretraining+models+for+word%2Dlevel+semantic+similarity%2E+arXiv+preprint%2E)
17. Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 506–516. [find](https://scholar.google.com/scholar?q=Sylvestre%2DAlvise+Rebuffi%2C+Hakan+Bilen%2C+and+Andrea+Vedaldi%2E+2017%2E+Learning+multiple+visual+domains+with+residual+adapters%2E+In+Advances+in+Neural+Information+Processing+Systems+30%3A+Annual+Conference+on+) [our card](https://md.research.elamur.ai/free/en/papers/rebuffi-2017-learning-multiple-visual-domains-residual/)
18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
19. Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2020. AdapterDrop: On the Efficiency of Adapters in Transformers. arXiv preprint. [find](https://scholar.google.com/scholar?q=Andreas+Ru%CC%88ckle%CC%81%2C+Gregor+Geigle%2C+Max+Glockner%2C+Tilman+Beck%2C+Jonas+Pfeiffer%2C+Nils+Reimers%2C+and+Iryna+Gurevych%2E+2020%2E+AdapterDrop%3A+On+the+Efficiency+of+Adapters+in+Transformers%2E+arXiv+preprint%2E) [our card](https://md.research.elamur.ai/free/en/papers/ruckle-2021-adapterdrop-efficiency-adapters-transformers/)
20. Cyprien de Masson d’Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019. Episodic memory in lifelong language learning. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, B… [find](https://scholar.google.com/scholar?q=Cyprien+de+Masson+d%E2%80%99Autume%2C+Sebastian+Ruder%2C+Lingpeng+Kong%2C+and+Dani+Yogatama%2E+2019%2E+Episodic+memory+in+lifelong+language+learning%2E+In+Advances+in+Neural+Information+Processing+Systems+32%3A+Annual+Conf)
21. Sebastian Ruder. 2017. An Overview of Multi-Task Learning in Deep Neural Networks. arXiv preprint. [find](https://scholar.google.com/scholar?q=Sebastian+Ruder%2E+2017%2E+An+Overview+of+Multi%2DTask+Learning+in+Deep+Neural+Networks%2E+arXiv+preprint%2E) [our card](https://md.research.elamur.ai/free/en/papers/ruder-2017-overview-multi-task-learning-deep/)
22. Sebastian Ruder, Matthew E Peters, Swabha Swayamdipta, and Thomas Wolf. 2019. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapol… [find](https://scholar.google.com/scholar?q=Sebastian+Ruder%2C+Matthew+E+Peters%2C+Swabha+Swayamdipta%2C+and+Thomas+Wolf%2E+2019%2E+Transfer+learning+in+natural+language+processing%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+t) [our card](https://md.research.elamur.ai/free/en/papers/ruder-2019-transfer-learning-natural-language-processing/)
23. Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. 2019. To tune or not to tune? adapting pre-trained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP, RepL4NLP@ACL 2019, Florence, Italy, August 2, 2019, pages 7–14. [find](https://scholar.google.com/scholar?q=Matthew+E%2E+Peters%2C+Sebastian+Ruder%2C+and+Noah+A%2E+Smith%2E+2019%2E+To+tune+or+not+to+tune%3F+adapting+pre%2Dtrained+representations+to+diverse+tasks%2E+In+Proceedings+of+the+4th+Workshop+on+Representation+Learnin)
24. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2020a. AdapterFusion: Non-destructive task composition for transfer learning. arXiv preprint. [find](https://scholar.google.com/scholar?q=Jonas+Pfeiffer%2C+Aishwarya+Kamath%2C+Andreas+Ru%CC%88ckle%CC%81%2C+Kyunghyun+Cho%2C+and+Iryna+Gurevych%2E+2020a%2E+AdapterFusion%3A+Non%2Ddestructive+task+composition+for+transfer+learning%2E+arXiv+preprint%2E) [our card](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterfusion-non-destructive-task-composition/)
25. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020b. MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Virtual Conference. [find](https://scholar.google.com/scholar?q=Jonas+Pfeiffer%2C+Ivan+Vulic%CC%81%2C+Iryna+Gurevych%2C+and+Sebastian+Ruder%2E+2020b%2E+MAD%2DX%3A+An+Adapter%2Dbased+Framework+for+Multi%2Dtask+Cross%2Dlingual+Transfer%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Met) [our card](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-mad-x-adapter-based-framework/)
26. Victor Sanh, Thomas Wolf, and Sebastian Ruder. 2019. A hierarchical multi-task approach for learning embeddings from semantic tasks. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, T… [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Thomas+Wolf%2C+and+Sebastian+Ruder%2E+2019%2E+A+hierarchical+multi%2Dtask+approach+for+learning+embeddings+from+semantic+tasks%2E+In+The+Thirty%2DThird+AAAI+Conference+on+Artificial+Intelligence%2C+AAA)
27. Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint. [find](https://scholar.google.com/scholar?q=Jason+Phang%2C+Thibault+Fe%CC%81vry%2C+and+Samuel+R+Bowman%2E+2018%2E+Sentence+encoders+on+stilts%3A+Supplementary+training+on+intermediate+labeled%2Ddata+tasks%2E+arXiv+preprint%2E) [our card](https://md.research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/)
28. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2C+Azalia+Mirhoseini%2C+Krzysztof+Maziarz%2C+Andy+Davis%2C+Quoc+V%2E+Le%2C+Geoffrey+E%2E+Hinton%2C+and+Jeff+Dean%2E+2017%2E+Outrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2E)
29. June 1-6, 2018, Volume 1 (Long Papers), pages 1112–1122. [find](https://scholar.google.com/scholar?q=June+1%2D6%2C+2018%2C+Volume+1+%28Long+Papers%29%2C+pages+1112%E2%80%931122%2E)
30. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. [find](https://scholar.google.com/scholar?q=In+5th+International+Conference+on+Learning+Representations%2C+ICLR+2017%2C+Toulon%2C+France%2C+April+24%2D26%2C+2017%2C+Conference+Track+Proceedings%2E)
31. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi an-dArt Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2020. Hugging-Face’s Transformers: State-of-the-art Natural Language Processing. In Proceedings of the 2020 Conference on Empirica… [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi+an%2DdArt+Pierric+Cistac%2C+Tim+Rault%2C+Re%CC%81mi+Louf%2C+Morgan+Funtowicz%2C+and+Jamie+Brew%2E+2020%2E+Hugging%2DFace%E2%80%99s+Transform) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
32. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 20… [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D%2E+Manning%2C+Andrew+Y%2E+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E)
33. Asa Cooper Stickland and Iain Murray. 2019. BERT and pals: Projected attention layers for efficient adaptation in multi-task learning. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pages 5986–5995. [find](https://scholar.google.com/scholar?q=Asa+Cooper+Stickland+and+Iain+Murray%2E+2019%2E+BERT+and+pals%3A+Projected+attention+layers+for+efficient+adaptation+in+multi%2Dtask+learning%2E+In+Proceedings+of+the+36th+International+Conference+on+Machine+Le)
34. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 3645– 3650. [find](https://scholar.google.com/scholar?q=Emma+Strubell%2C+Ananya+Ganesh%2C+and+Andrew+McCallum%2E+2019%2E+Energy+and+policy+considerations+for+deep+learning+in+NLP%2E+In+Proceedings+of+the+57th+Conference+of+the+Association+for+Computational+Linguisti) [our card](https://md.research.elamur.ai/free/en/papers/strubell-2019-energy-policy-considerations-deep-learning/)
35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017,… [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+Is+All+You+Need%2E+In+Advances+in+Neural+Information+Process)
36. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the Workshop: Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2018, Brussels,… [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R%2E+Bowman%2E+2018%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+Proceedings+of+) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
37. Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2020. K-adapter: Infusing knowledge into pre-trained models with adapters. arXiv preprint. [find](https://scholar.google.com/scholar?q=Ruize+Wang%2C+Duyu+Tang%2C+Nan+Duan%2C+Zhongyu+Wei%2C+Xuanjing+Huang%2C+Jianshu+Ji%2C+Guihong+Cao%2C+Daxin+Jiang%2C+and+Ming+Zhou%2E+2020%2E+K%2Dadapter%3A+Infusing+knowledge+into+pre%2Dtrained+models+with+adapters%2E+arXiv+prep) [our card](https://md.research.elamur.ai/free/en/papers/wang-2020-k-adapter-infusing-knowledge-pre/)
38. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R%2E+Bowman%2E+2019%2E+Neural+network+acceptability+judgments%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%3A625%E2%80%93641%2E)
39. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+R%2E+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+C)