---
title: "MerA: Merging Pretrained Adapters For Few-Shot Learning (2023)"
description: "The authors therefore propose MerA, which combines several pretrained adapters from AdapterHub… 28 references, cited by 9 papers here."
authors: ["Shwai He", "Run-Ze Fan", "Liang Ding", "Li Shen", "Tianyi Zhou", "Dacheng Tao"]
year: 2023
venue: "arXiv:2308.15982"
source: "https://arxiv.org/abs/2308.15982"
doi: "10.48550/arxiv.2308.15982"
arxiv: "2308.15982"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/he-2023-mera-merging-pretrained-adapters-few/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/he-2023-mera-merging-pretrained-adapters-few/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/13/) / MerA: Merging Pretrained Adapters For Few-Shot Learning

# MerA: Merging Pretrained Adapters For Few-Shot Learning

**Authors:** [Shwai He](https://md.research.elamur.ai/free/en/authors/shwai-he/), [Run-Ze Fan](https://md.research.elamur.ai/free/en/authors/run-ze-fan/), [Liang Ding](https://md.research.elamur.ai/free/en/authors/liang-ding/), [Li Shen](https://md.research.elamur.ai/free/en/authors/li-shen/), [Tianyi Zhou](https://md.research.elamur.ai/free/en/authors/tianyi-zhou/) and [Dacheng Tao](https://md.research.elamur.ai/free/en/authors/dacheng-tao/) · **Year:** 2023 · **Venue:** arXiv:2308.15982 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2308.15982) · [PDF](https://arxiv.org/pdf/2308.15982) · [DOI 10.48550/arxiv.2308.15982](https://doi.org/10.48550/arxiv.2308.15982) · [arXiv 2308.15982](https://arxiv.org/abs/2308.15982) · [OpenAlex](https://openalex.org/W4386355389)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [9](https://md.research.elamur.ai/free/en/papers/he-2023-mera-merging-pretrained-adapters-few/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Adapter tuning trains only small bottleneck modules inside a frozen pretrained language model. These modules start from random weights, so they tend to underperform when only a handful of labeled examples are available. AdapterFusion reuses previously trained adapters, but its attention-style composition layers add many trainable parameters and require keeping several adapters at inference. A pilot study in this paper even shows a single adapter beating AdapterFusion on 10-shot GLUE tasks. The authors therefore propose MerA, which combines several pretrained adapters from AdapterHub into one adapter before few-shot tuning. Besides plain summation and averaging, MerA can first align neurons across adapters with optimal transport, based on either weights or activations. On GLUE with BERT and RoBERTa, the activation-based variant consistently beats standard adapters and AdapterFusion at the same small parameter cost. Merging adapters from the same NLP track as the target task gives the largest gains. The approach offers a cheap way to transfer adapter knowledge in low-data settings.

## Key points
- Source adapters come from AdapterHub and were trained on imdb, boolq, scitail and winogrande, covering sentiment, QA, NLI and commonsense reasoning. Evaluation uses GLUE with 10, 20 and 30 training shots.
- Four merging methods are compared: "Sum", "Avg", "Wts" and "Acts". The two optimal-transport alignments beat the naive ones. "Acts" is the default, with up to 2.7% average improvement over a standard adapter.
- Trainable parameters: MerA and a plain adapter use about 0.8% of the model, AdapterFusion about 18%. In 30-shot runs averaged over head, prompt and prompt-with-demonstration tuning, MerA scores 71.1 vs 69.2 for a standard adapter on BERT-Base, and 78.9 vs 76.4 on RoBERTa-Base. Both MerA averages also exceed full fine-tuning.
- Under the "same-track" setting, MerA beats the standard adapter by 3.5% on MRPC (using STS adapters) and 5.0% on MNLI (using NLI adapters). Adapters from mismatched tracks can hurt accuracy.
- In a zero-shot check, plugging MerA into BERT gives a better starting point than random adapters or no adapters, for example +2.4% on MNLI and +0.98% on SST-2.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Maryland, College Park](https://md.research.elamur.ai/free/en/universities/university-of-maryland-college-park/) — University, College Park, United States · 141 works · 29,174 citations
- [The University of Sydney](https://md.research.elamur.ai/free/en/universities/the-university-of-sydney/) — University, Sydney, Australia · 49 works · 68,350 citations
- [University of Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/universities/university-of-chinese-academy-of-sciences/) — University, Beijing, China · 188 works · 8,744 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou and Dacheng Tao. 2023. MerA: Merging Pretrained Adapters For Few-Shot Learning. arXiv:2308.15982.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{he2023mera,
  title = {{MerA: Merging Pretrained Adapters For Few-Shot Learning}},
  author = {Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou and Dacheng Tao},
  year = {2023},
  doi = {10.48550/arxiv.2308.15982},
  eprint = {2308.15982},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.15982}
}
```

- [← previous Less is More: Mitigate Spurious Correlations for Open-Domain Dialogue Response Generation Models by Causal Discovery](https://md.research.elamur.ai/free/en/papers/feng-2023-less-is-more-mitigate-spurious/)
- [next → Minanto at SemEval-2023 Task 2: Fine-tuning XLM-RoBERTa for Named Entity Recognition on English Data](https://md.research.elamur.ai/free/en/papers/hofer-2023-minanto-semeval-2023-task-2/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/13/)

## References

28 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Trapit Bansal, Salaheddin Alzubi, Tong Wang, Jay-Yoon Lee, and Andrew McCallum. 2022. Meta-adapters: Parameter efficient few-shot fine-tuning through meta-learning. In AutoML. [find](https://scholar.google.com/scholar?q=Trapit+Bansal%2C+Salaheddin+Alzubi%2C+Tong+Wang%2C+Jay%2DYoon+Lee%2C+and+Andrew+McCallum%2E+2022%2E+Meta%2Dadapters%3A+Parameter+efficient+few%2Dshot+fine%2Dtuning+through+meta%2Dlearning%2E+In+AutoML%2E)
2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/)
3. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In ICLR. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V+Le%2C+and+Christopher+D+Manning%2E+2020%2E+Electra%3A+Pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+In+ICLR%2E)
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In NAACL.
5. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint. [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Gabriel+Ilharco%2C+Roy+Schwartz%2C+Ali+Farhadi%2C+Hannaneh+Hajishirzi%2C+and+Noah+Smith%2E+2020%2E+Fine%2Dtuning+pretrained+language+models%3A+Weight+initializations%2C+data+orders%2C+and+early+stopping%2E+arX)
6. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. [Making pre-trained language models better few-shot learners](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/). In ACL.
7. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022a. [Towards a unified view of parameter-efficient transfer learning](https://md.research.elamur.ai/free/en/papers/he-2021-towards-unified-view-parameter-efficient/). In ICLR.
8. Shwai He, Liang Ding, Daize Dong, Boan Liu, Fuqiang Yu, and Dacheng Tao. 2023. Pad-net: An efficient framework for dynamic networks. In ACL. [find](https://scholar.google.com/scholar?q=Shwai+He%2C+Liang+Ding%2C+Daize+Dong%2C+Boan+Liu%2C+Fuqiang+Yu%2C+and+Dacheng+Tao%2E+2023%2E+Pad%2Dnet%3A+An+efficient+framework+for+dynamic+networks%2E+In+ACL%2E)
9. Shwai He, Liang Ding, Daize Dong, Jeremy Zhang, and Dacheng Tao. 2022b. SparseAdapter: An easy approach for improving the parameter-efficiency of adapters. In EMNLP. [find](https://scholar.google.com/scholar?q=Shwai+He%2C+Liang+Ding%2C+Daize+Dong%2C+Jeremy+Zhang%2C+and+Dacheng+Tao%2E+2022b%2E+SparseAdapter%3A+An+easy+approach+for+improving+the+parameter%2Defficiency+of+adapters%2E+In+EMNLP%2E)
10. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In ICML. [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+De+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+nlp%2E+In+)
11. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In AAAI. [find](https://scholar.google.com/scholar?q=Tushar+Khot%2C+Ashish+Sabharwal%2C+and+Peter+Clark%2E+2018%2E+Scitail%3A+A+textual+entailment+dataset+from+science+question+answering%2E+In+AAAI%2E)
12. Diederik P Kingma and Jimmy Ba. 2015. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). In ICLR.
13. Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. 2020. Mixout: Effective regularization to finetune large-scale pretrained language models. In ICLR. [find](https://scholar.google.com/scholar?q=Cheolhyoung+Lee%2C+Kyunghyun+Cho%2C+and+Wanmo+Kang%2E+2020%2E+Mixout%3A+Effective+regularization+to+finetune+large%2Dscale+pretrained+language+models%2E+In+ICLR%2E)
14. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Others. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint.
15. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. [Learning word vectors for sentiment analysis](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/). In ACL.
16. Nafise Sadat Moosavi, Quentin Delfosse, Kristian Kersting, and Iryna Gurevych. 2022. Adaptable adapters. In NAACL. [find](https://scholar.google.com/scholar?q=Nafise+Sadat+Moosavi%2C+Quentin+Delfosse%2C+Kristian+Kersting%2C+and+Iryna+Gurevych%2E+2022%2E+Adaptable+adapters%2E+In+NAACL%2E)
17. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. [Adapterfusion: Non-destructive task composition for transfer learning](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterfusion-non-destructive-task-composition/). In EACL.
18. Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. [Adapterhub: A framework for adapting transformers](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterhub-framework-adapting-transformers/). In EMNLP Demonstrations.
19. Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. [Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks](https://md.research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/). arXiv preprint.
20. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Commun. ACM. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2021%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Commun%2E+ACM%2E)
21. Sidak Pal Singh and Martin Jaggi. 2020. Model fusion via optimal transport. In NeurIPS. [find](https://scholar.google.com/scholar?q=Sidak+Pal+Singh+and+Martin+Jaggi%2E+2020%2E+Model+fusion+via+optimal+transport%2E+In+NeurIPS%2E)
22. Justin Solomon, Fernando de Goes, Gabriel Peyré, Marco Cuturi, Adrian Butscher, Andy Nguyen, Tao Du, and Leonidas J. Guibas. 2015. Convolutional wasserstein distances: efficient optimal transportation on geometric domains. ACM Trans. Graph. [find](https://scholar.google.com/scholar?q=Justin+Solomon%2C+Fernando+de+Goes%2C+Gabriel+Peyr%C3%A9%2C+Marco+Cuturi%2C+Adrian+Butscher%2C+Andy+Nguyen%2C+Tao+Du%2C+and+Leonidas+J%2E+Guibas%2E+2015%2E+Convolutional+wasserstein+distances%3A+efficient+optimal+transportation)
23. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In ICLR.
24. Zhenyi Wang, Xiaoyang Wang, Li Shen, Qiuling Suo, Kaiqiang Song, Dong Yu, Yan Shen, and Mingchen Gao. 2022. Meta-learning without data via wasserstein distributionally-robust model fusion. In UAI. [find](https://scholar.google.com/scholar?q=Zhenyi+Wang%2C+Xiaoyang+Wang%2C+Li+Shen%2C+Qiuling+Suo%2C+Kaiqiang+Song%2C+Dong+Yu%2C+Yan+Shen%2C+and+Mingchen+Gao%2E+2022%2E+Meta%2Dlearning+without+data+via+wasserstein+distributionally%2Drobust+model+fusion%2E+In+UAI%2E)
25. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). In NeurIPS.
26. Changtong Zan, Liang Ding, Li Shen, Yu Cao, Weifeng Liu, and Dacheng Tao. 2022. On the complementarity between pre-training and random-initialization for resource-rich machine translation. In COLING. [find](https://scholar.google.com/scholar?q=Changtong+Zan%2C+Liang+Ding%2C+Li+Shen%2C+Yu+Cao%2C+Weifeng+Liu%2C+and+Dacheng+Tao%2E+2022%2E+On+the+complementarity+between+pre%2Dtraining+and+random%2Dinitialization+for+resource%2Drich+machine+translation%2E+In+COLING%2E)
27. Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2022. Panda: Prompt transfer meets knowledge distillation for efficient model adaptation. arXiv preprint. [find](https://scholar.google.com/scholar?q=Qihuang+Zhong%2C+Liang+Ding%2C+Juhua+Liu%2C+Bo+Du%2C+and+Dacheng+Tao%2E+2022%2E+Panda%3A+Prompt+transfer+meets+knowledge+distillation+for+efficient+model+adaptation%2E+arXiv+preprint%2E)
28. Qihuang Zhong, Liang Ding, Juhua Liu, Xuebo Liu, Min Zhang, Bo Du, and Dacheng Tao. 2023. Revisiting token dropping strategy in efficient BERT pretraining. In ACL. [find](https://scholar.google.com/scholar?q=Qihuang+Zhong%2C+Liang+Ding%2C+Juhua+Liu%2C+Xuebo+Liu%2C+Min+Zhang%2C+Bo+Du%2C+and+Dacheng+Tao%2E+2023%2E+Revisiting+token+dropping+strategy+in+efficient+BERT+pretraining%2E+In+ACL%2E)

*Updated 30 October 2026*