---
title: "AdaMix — paper by Wang et al. (2022) — summary and ideas"
description: "The design draws on sparse mixture-of-experts models. 47 references, cited by 58 papers here."
authors: ["Yaqing Wang", "Sahaj Agarwal", "Subhabrata Mukherjee", "Xiaodong Liu", "Jing Gao", "Ahmed Hassan Awadallah", "Jianfeng Gao"]
year: 2022
venue: "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2022.emnlp-main.388/"
doi: "10.18653/v1/2022.emnlp-main.388"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2022-adamix-mixture-adaptations-parameter-efficient/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2022-adamix-mixture-adaptations-parameter-efficient/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) / AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning

# AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning

**Authors:** Yaqing Wang, [Sahaj Agarwal](https://md.research.elamur.ai/free/en/authors/sahaj-agarwal/), [Subhabrata Mukherjee](https://md.research.elamur.ai/free/en/authors/subhabrata-mukherjee/), [Xiaodong Liu](https://md.research.elamur.ai/free/en/authors/xiaodong-liu/), Jing Gao, [Ahmed Hassan Awadallah](https://md.research.elamur.ai/free/en/authors/ahmed-awadallah/) and [Jianfeng Gao](https://md.research.elamur.ai/free/en/authors/jianfeng-gao/) · **Year:** 2022 · **Venue:** [Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2022](https://md.research.elamur.ai/free/en/venues/emnlp/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.emnlp-main.388/) · [PDF](https://aclanthology.org/2022.emnlp-main.388.pdf) · [DOI 10.18653/v1/2022.emnlp-main.388](https://doi.org/10.18653/v1/2022.emnlp-main.388) · [OpenAlex](https://openalex.org/W4385572883)

**Citations:** 74 citations (OpenAlex, 2026-10-14) · [58](https://md.research.elamur.ai/free/en/papers/wang-2022-adamix-mixture-adaptations-parameter-efficient/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Full fine-tuning of large pre-trained language models requires a separate multi-hundred-megabyte copy of the weights for every task. Parameter-efficient fine-tuning (PEFT) methods reduce this cost but usually perform worse than full tuning. AdaMix closes that gap by placing several adaptation modules in each Transformer layer instead of a single one. These can be Houlsby-style adapters or LoRA low-rank matrices.

The design draws on sparse mixture-of-experts models. During training, each step randomly picks which up- and down-projection modules a batch passes through, so no extra router parameters are needed. A consistency loss between two stochastic forward passes keeps the modules from diverging, and the modules share parameters. Before inference, the modules are averaged into one, in the spirit of model soups. As a result, FLOPs and stored parameters match the underlying PEFT method.

Tuning only 0.1–0.2% of parameters, AdaMix beats strong PEFT baselines and even full fine-tuning on GLUE. It also outperforms competing methods on NLG and few-shot NLU tasks. This makes cheap per-task adaptation of a frozen PLM more attractive.

## Key points
- On the GLUE dev set with RoBERTa-large, AdaMix reaches an average of about 89.9. Full fine-tuning gets about 88.9, LoRA 88.6, Pfeiffer 88.4 and Houlsby 87.8.
- The authors call it the first PEFT method to surpass full fine-tuning on all GLUE NLU tasks. Per-task storage drops roughly 444x, from 355MB to 0.8MB.
- Ablations vary the number of modules (2, 4, 8; 4 is the default) and the adapter bottleneck size (8, 16, 32; 16 is the default). Both show diminishing returns, and low-resource tasks such as RTE suffer with more modules.
- Training needs roughly 1–2x the iterations of standard PEFT. Prompt-tuning and prefix-tuning were not tested as base methods.
- Code and models are available at https://aka.ms/AdaMix.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 240 works · 123,537 citations
- [Purdue University West Lafayette](https://md.research.elamur.ai/free/en/universities/purdue-university-west-lafayette/) — University, West Lafayette, United States · 9 works · 1,254 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah and Jianfeng Gao. 2022. AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2022adamix,
  title = {{AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning}},
  author = {Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah and Jianfeng Gao},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.388},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.388}
}
```

- [← previous STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation](https://md.research.elamur.ai/free/en/papers/fang-2022-stemm-self-learning-speech-text/)
- [next → FLUTE: Figurative Language Understanding through Textual Explanations](https://md.research.elamur.ai/free/en/papers/chakrabarty-2022-flute-figurative-language-understanding-through/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/)

## References

47 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. 2021. [Intrinsic dimensionality explains the effectiveness of language model fine-tuning](https://md.research.elamur.ai/free/en/papers/aghajanyan-2021-intrinsic-dimensionality-explains-effectiveness-language/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pro…
2. Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second PASCAL recognising textual entailment challenge. [find](https://scholar.google.com/scholar?q=Roy+Bar+Haim%2C+Ido+Dagan%2C+Bill+Dolan%2C+Lisa+Ferro%2C+Danilo+Giampiccolo%2C+Bernardo+Magnini%2C+and+Idan+Szpektor%2E+2006%2E+The+second+PASCAL+recognising+textual+entailment+challenge%2E)
3. Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth PASCAL recognizing textual entailment challenge. In TAC. [find](https://scholar.google.com/scholar?q=Luisa+Bentivogli%2C+Peter+Clark%2C+Ido+Dagan%2C+and+Danilo+Giampiccolo%2E+2009%2E+The+fifth+PASCAL+recognizing+textual+entailment+challenge%2E+In+TAC%2E)
4. [Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
5. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2005%2E+The+PASCAL+recognising+textual+entailment+challenge%2E+In+the+First+International+Conference+on+Machine+Learning+Challenges%3A+Evaluating+Predictive+U)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
7. William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961. [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Barret+Zoph%2C+and+Noam+Shazeer%2E+2021%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+arXiv+preprint+arXiv%3A2101%2E03961%2E)
8. Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. 2020. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259–3269. PMLR. [find](https://scholar.google.com/scholar?q=Jonathan+Frankle%2C+Gintare+Karolina+Dziugaite%2C+Daniel+Roy%2C+and+Michael+Carbin%2E+2020%2E+Linear+mode+connectivity+and+the+lottery+ticket+hypothesis%2E+In+International+Conference+on+Machine+Learning%2C+pages+3)
9. Yarin Gal and Zoubin Ghahramani. 2015. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. CoRR, abs/1506.02142. [find](https://scholar.google.com/scholar?q=Yarin+Gal+and+Zoubin+Ghahramani%2E+2015%2E+Dropout+as+a+bayesian+approximation%3A+Representing+model+uncertainty+in+deep+learning%2E+CoRR%2C+abs%2F1506%2E02142%2E)
10. Yarin Gal, Riashat Islam, and Zoubin Ghahramani. 2017. Deep Bayesian active learning with image data. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1183–1192. PMLR. [find](https://scholar.google.com/scholar?q=Yarin+Gal%2C+Riashat+Islam%2C+and+Zoubin+Ghahramani%2E+2017%2E+Deep+Bayesian+active+learning+with+image+data%2E+In+Proceedings+of+the+34th+International+Conference+on+Machine+Learning%2C+volume+70+of+Proceedings+)
11. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. [Making pre-trained language models better few-shot learners](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/). In Association for Computational Linguistics (ACL).
12. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. [The webnlg challenge: Generating text from rdf data](https://md.research.elamur.ai/free/en/papers/gardent-2017-webnlg-challenge-generating-text-rdf/). In Proceedings of the 10th International Conference on Natural Language Generation, pages 124–133.
13. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. [find](https://scholar.google.com/scholar?q=Danilo+Giampiccolo%2C+Bernardo+Magnini%2C+Ido+Dagan%2C+and+Bill+Dolan%2E+2007%2E+The+third+PASCAL+recognizing+textual+entailment+challenge%2E+In+the+ACL%2DPASCAL+Workshop+on+Textual+Entailment+and+Paraphrasing%2E)
14. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790–2799. PMLR. [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+De+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+nlp%2E+In+)
15. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). arXiv preprint arXiv:2106.09685.
16. Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018. [Averaging weights leads to wider optima and better generalization](https://md.research.elamur.ai/free/en/papers/izmailov-2018-averaging-weights-leads-wider-optima/). arXiv preprint arXiv:1803.05407.
17. Jaejun Lee, Raphael Tang, and Jimmy Lin. 2019. What would elsa do? freezing layers during transformer fine-tuning. arXiv preprint arXiv:1911.03090. — [arXiv:1911.03090](https://arxiv.org/abs/1911.03090) [find](https://scholar.google.com/scholar?q=Jaejun+Lee%2C+Raphael+Tang%2C+and+Jimmy+Lin%2E+2019%2E+What+would+elsa+do%3F+freezing+layers+during+transformer+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A1911%2E03090%2E+%E2%80%94+%5BarXiv%3A1911%2E03090%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1911%2E0309)
18. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668. [find](https://scholar.google.com/scholar?q=Dmitry+Lepikhin%2C+HyoukJoong+Lee%2C+Yuanzhong+Xu%2C+Dehao+Chen%2C+Orhan+Firat%2C+Yanping+Huang%2C+Maxim+Krikun%2C+Noam+Shazeer%2C+and+Zhifeng+Chen%2E+2020%2E+Gshard%3A+Scaling+giant+models+with+conditional+computation+and)
19. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. CoRR, abs/2104.08691. [find](https://scholar.google.com/scholar?q=Brian+Lester%2C+Rami+Al%2DRfou%2C+and+Noah+Constant%2E+2021%2E+The+power+of+scale+for+parameter%2Defficient+prompt+tuning%2E+CoRR%2C+abs%2F2104%2E08691%2E)
20. Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. 2021. Base layers: Simplifying training of large, sparse models. In ICML. [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Shruti+Bhosale%2C+Tim+Dettmers%2C+Naman+Goyal%2C+and+Luke+Zettlemoyer%2E+2021%2E+Base+layers%3A+Simplifying+training+of+large%2C+sparse+models%2E+In+ICML%2E)
21. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. CoRR, abs/2101.00190. [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+CoRR%2C+abs%2F2101%2E00190%2E)
22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+BERT+pretraining+approach%2E+C)
23. Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Wen-tau Yih, and Madian Khabsa. 2021. [Unipelt: A unified framework for parameter-efficient language model tuning](https://md.research.elamur.ai/free/en/papers/mao-2022-unipelt-unified-framework-parameter-efficient/). arXiv preprint arXiv:2110.07577.
24. Michael Matena and Colin Raffel. 2021. [Merging models with fisher-weighted averaging](https://md.research.elamur.ai/free/en/papers/matena-2021-merging-models-fisher-weighted-averaging/). arXiv preprint arXiv:2111.09832.
25. Thibault Sellam, Steve Yadlowsky, Ian Tenney, Jason Wei, Naomi Saphra, Alexander D’Amour, Tal Linzen, Jasmijn Bastings, Iulia Raluca Turc, Jacob Eisenstein, Dipanjan Das, and Ellie Pavlick. 2022. The multiB-ERTs: BERT reproductions for robustness analysis. In International Conference on Learning Rep… [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Steve+Yadlowsky%2C+Ian+Tenney%2C+Jason+Wei%2C+Naomi+Saphra%2C+Alexander+D%E2%80%99Amour%2C+Tal+Linzen%2C+Jasmijn+Bastings%2C+Iulia+Raluca+Turc%2C+Jacob+Eisenstein%2C+Dipanjan+Das%2C+and+Ellie+Pavlick%2E+2022%2E+The+)
26. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2C+Azalia+Mirhoseini%2C+Krzysztof+Maziarz%2C+Andy+Davis%2C+Quoc+Le%2C+Geoffrey+Hinton%2C+and+Jeff+Dean%2E+2017%2E+Outrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2E+arXiv+pr)
27. Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, et al. 2020. [Dart: Open-domain structured data record to text generation](https://md.research.elamur.ai/free/en/papers/nan-2020-dart-open-domain-structured-data/). arXiv preprint arXiv:2007.02871.
28. Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022. [Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model](https://md.research.elamur.ai/free/en/papers/smith-2022-using-deepspeed-megatron-train-megatron/). arXiv preprint ar…
29. Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. 2020. What is being transferred in transfer learning? Advances in neural information processing systems, 33:512–523. [find](https://scholar.google.com/scholar?q=Behnam+Neyshabur%2C+Hanie+Sedghi%2C+and+Chiyuan+Zhang%2E+2020%2E+What+is+being+transferred+in+transfer+learning%3F+Advances+in+neural+information+processing+systems%2C+33%3A512%E2%80%93523%2E)
30. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017. [The e2e dataset: New challenges for end-to-end…](https://md.research.elamur.ai/free/en/papers/novikova-2017-e2e-dataset-new-challenges-end/)
31. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826. [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Vincent+Vanhoucke%2C+Sergey+Ioffe%2C+Jon+Shlens%2C+and+Zbigniew+Wojna%2E+2016%2E+Rethinking+the+inception+architecture+for+computer+vision%2E+In+Proceedings+of+the+IEEE+conference+on+computer+v)
32. Bo Pang and Lillian Lee. 2004. [A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts](https://md.research.elamur.ai/free/en/papers/pang-2004-sentimental-education-sentiment-analysis-subjectivity/).
33. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. [True few-shot learning with language models](https://md.research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/). arXiv preprint arXiv:2105.11447.
34. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. [Adapterfusion: Non-destructive task composition for transfer learning](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterfusion-non-destructive-task-composition/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487–503…
35. Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. [Adapterhub: A framework for adapting transformers](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterhub-framework-adapting-transformers/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020): Systems De…
36. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). arXiv preprint arXiv:1910.10683.
37. Stephen Roller, Sainbayar Sukhbaatar, Arthur D. Szlam, and Jason Weston. 2021. Hash layers for large sparse models. ArXiv, abs/2106.04426. — [arXiv:2106.04426](https://arxiv.org/abs/2106.04426) [find](https://scholar.google.com/scholar?q=Stephen+Roller%2C+Sainbayar+Sukhbaatar%2C+Arthur+D%2E+Szlam%2C+and+Jason+Weston%2E+2021%2E+Hash+layers+for+large+sparse+models%2E+ArXiv%2C+abs%2F2106%2E04426%2E+%E2%80%94+%5BarXiv%3A2106%2E04426%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2106%2E04426%29)
38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30.
39. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/).
40. Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. 2021. List: Lite self-training makes efficient few-shot learners. arXiv preprint arXiv:2110.06274. — [arXiv:2110.06274](https://arxiv.org/abs/2110.06274) [find](https://scholar.google.com/scholar?q=Yaqing+Wang%2C+Subhabrata+Mukherjee%2C+Xiaodong+Liu%2C+Jing+Gao%2C+Ahmed+Hassan+Awadallah%2C+and+Jianfeng+Gao%2E+2021%2E+List%3A+Lite+self%2Dtraining+makes+efficient+few%2Dshot+learners%2E+arXiv+preprint+arXiv%3A2110%2E06274%2E+)
41. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2):165–210. [find](https://scholar.google.com/scholar?q=Janyce+Wiebe%2C+Theresa+Wilson%2C+and+Claire+Cardie%2E+2005%2E+Annotating+expressions+of+opinions+and+emotions+in+language%2E+Language+resources+and+evaluation%2C+39%282%29%3A165%E2%80%93210%2E)
42. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E)
43. Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. 2022. [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://md.research.elamur.ai/free/en/papers/wortsman-2022-model-soups-averaging-weights-multiple/).…
44. An Yang, Junyang Lin, Rui Men, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, Jie Zhang, Jiamang Wang, Yong Li, et al. 2021. M6-t: Exploring sparse expert models and beyond. arXiv preprint arXiv:2105.15082. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Junyang+Lin%2C+Rui+Men%2C+Chang+Zhou%2C+Le+Jiang%2C+Xianyan+Jia%2C+Ang+Wang%2C+Jie+Zhang%2C+Jiamang+Wang%2C+Yong+Li%2C+et+al%2E+2021%2E+M6%2Dt%3A+Exploring+sparse+expert+models+and+beyond%2E+arXiv+preprint+arXiv%3A2105%2E15)
45. Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. [Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models](https://md.research.elamur.ai/free/en/papers/ben-zaken-2021-bitfit-simple-parameter-efficient-fine/). arXiv preprint arXiv:2106.10199.
46. Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. 2021. [Revisiting few-sample BERT fine-tuning](https://md.research.elamur.ai/free/en/papers/zhang-2020-revisiting-few-sample-bert-fine/).
47. Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo Zhao, and Jianfeng Gao. 2021. Taming sparsely activated transformer with stochastic experts. arXiv preprint arXiv:2110.04260. — [arXiv:2110.04260](https://arxiv.org/abs/2110.04260) [find](https://scholar.google.com/scholar?q=Simiao+Zuo%2C+Xiaodong+Liu%2C+Jian+Jiao%2C+Young+Jin+Kim%2C+Hany+Hassan%2C+Ruofei+Zhang%2C+Tuo+Zhao%2C+and+Jianfeng+Gao%2E+2021%2E+Taming+sparsely+activated+transformer+with+stochastic+experts%2E+arXiv+preprint+arXiv%3A211)

*Updated 24 October 2026*