---
title: "Parameter-efficient Multi-task Fine-tuning — Mahabadi 2021"
description: "Adapters let a frozen pretrained model be tuned cheaply, but each task normally gets its own isolated adapter, so related tasks cannot benefit from one…"
authors: ["Rabeeh Karimi Mahabadi", "Sebastian Ruder", "Mostafa Dehghani", "James Henderson"]
year: 2021
venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)"
source: "https://aclanthology.org/2021.acl-long.47/"
doi: "10.18653/v1/2021.acl-long.47"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mahabadi-2021-parameter-efficient-multi-task-fine/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mahabadi-2021-parameter-efficient-multi-task-fine/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks

# Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks

**Authors:** [Rabeeh Karimi Mahabadi](https://md.research.elamur.ai/free/en/authors/rabeeh-karimi-mahabadi/), [Sebastian Ruder](https://md.research.elamur.ai/free/en/authors/sebastian-ruder/), [Mostafa Dehghani](https://md.research.elamur.ai/free/en/authors/mostafa-dehghani/), [James Henderson](https://md.research.elamur.ai/free/en/authors/james-henderson/) · **Year:** 2021 · **Venue:** Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.acl-long.47/) · [PDF](https://aclanthology.org/2021.acl-long.47.pdf) · [DOI 10.18653/v1/2021.acl-long.47](https://doi.org/10.18653/v1/2021.acl-long.47) · [OpenAlex](https://openalex.org/W3167473228)

**Citations:** 148 citations (openalex-snapshot · 2026-10-14) · [100](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Adapters let a frozen pretrained model be tuned cheaply, but each task normally gets its own isolated adapter, so related tasks cannot benefit from one another. Plain multi-task fine-tuning shares knowledge across tasks but risks negative interference and capacity limits on high-resource tasks. The authors propose HYPERFORMER and its more compact variant HYPERFORMER++, built on the encoder-decoder T5 model. Small hypernetworks produce the weights of the adapter modules and of conditional layer normalization. In HYPERFORMER++, a single hypernetwork serves all layers and tasks. It is conditioned on an embedding that combines the task, the layer id, and the adapter's position. Only the hypernetworks, the task embeddings, and T5's layer norms are trained, and T5's task prefixes are replaced by learned task embeddings. On GLUE, the method improves multi-task results over T5 BASE and standard adapters while adding very few parameters per task. It also gives clear gains in few-shot transfer to unseen in-domain tasks. The work shows that knowledge can be shared across tasks without giving up task-specific adaptation or parameter efficiency.

## Key points
- HYPERFORMER++ adds only about 0.29% extra parameters per task. A new task needs just one additional learned task embedding.
- Adapters are placed after the attention block and after the feed-forward block of each T5 layer (following Houlsby et al., 2019). Each adapter has a down-projection, a GeLU nonlinearity, and an up-projection, and their weights are generated by linear hypernetworks.
- The method is evaluated on the GLUE benchmark. For small datasets (RTE, MRPC, STS-B, CoLA), the original validation set is split in half, giving one part for validation and one for testing.
- Few-shot domain transfer experiments use tasks such as CB, SciTail, BoolQ, PAWS, IMDB, and Yelp polarity. The learned task embeddings cluster by task type: sentiment, NLI, QA, and paraphrase tasks group together, while CoLA stays isolated.
- The approach is far more compact than related methods: Tay et al. (2021) need about 43× more parameters, and Üstün et al. (2020) train 2.88× the base model's size. Code is released at github.com/rabeehk/hyperformer.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

100 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 96 not yet.

- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/han-2024-parameter-efficient-fine-tuning-large/)
- [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](https://md.research.elamur.ai/free/en/papers/pfeiffer-2022-lifting-curse-multilinguality-pre-training/)
- [大模型参数高效微调方法综述: 技术、趋势与挑战](https://md.research.elamur.ai/free/en/papers/tang-2026-parameter-efficient-fine/)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://md.research.elamur.ai/free/en/papers/vu-2022-spot-better-frozen-model-adaptation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mahabadi2021parameter,
  title = {{Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks}},
  author = {Mahabadi, Rabeeh Karimi and Ruder, Sebastian and Dehghani, Mostafa and Henderson, James},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.47},
  url = {https://doi.org/10.18653/v1/2021.acl-long.47}
}
```

- [← previous Paragraph-level Simplification of Medical Texts](https://md.research.elamur.ai/free/en/papers/devaraj-2021-paragraph-level-simplification-medical-texts/)
- [next → Parameter-Efficient Transfer Learning with Diff Pruning](https://md.research.elamur.ai/free/en/papers/guo-2021-parameter-efficient-transfer-learning-diff/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

42 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint arXiv:1907.05019. [find](https://scholar.google.com/scholar?q=Naveen+Arivazhagan%2C+Ankur+Bapna%2C+Orhan+Firat%2C+Dmitry+Lepikhin%2C+Melvin+Johnson%2C+Maxim+Krikun%2C+Mia+Xu+Chen%2C+Yuan+Cao%2C+George+Foster%2C+Colin+Cherry%2C+et+al%2E+2019%2E+Massively+multilingual+neural+machine+tran) [our card](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/)
2. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450. [find](https://scholar.google.com/scholar?q=Jimmy+Lei+Ba%2C+Jamie+Ryan+Kiros%2C+and+Geoffrey+E+Hinton%2E+2016%2E+Layer+normalization%2E+arXiv+preprint+arXiv%3A1607%2E06450%2E) [our card](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/)
3. Jason Baldridge, Luheng He, and Yuan Zhang. 2019. Paws: Paraphrase adversaries from word scrambling. In NAACL. [find](https://scholar.google.com/scholar?q=Jason+Baldridge%2C+Luheng+He%2C+and+Yuan+Zhang%2E+2019%2E+Paws%3A+Paraphrase+adversaries+from+word+scrambling%2E+In+NAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2019-paws-paraphrase-adversaries-word-scrambling/)
4. Trapit Bansal, Rishikesh Jha, and Andrew McCallum. 2020. Learning to Few-Shot Learn Across Diverse Natural Language Classification Tasks. In COLING. [find](https://scholar.google.com/scholar?q=Trapit+Bansal%2C+Rishikesh+Jha%2C+and+Andrew+McCallum%2E+2020%2E+Learning+to+Few%2DShot+Learn+Across+Diverse+Natural+Language+Classification+Tasks%2E+In+COLING%2E)
5. Ankur Bapna and Orhan Firat. 2019. Simple, scalable adaptation for neural machine translation. In EMNLP. [find](https://scholar.google.com/scholar?q=Ankur+Bapna+and+Orhan+Firat%2E+2019%2E+Simple%2C+scalable+adaptation+for+neural+machine+translation%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/bapna-2019-simple-scalable-adaptation-neural-machine/)
6. Tian Jin, Zhun Liu, Shengjia Yan, Alexandre Eichenberger, and Louis-Philippe Morency. 2020. Language to network: Conditional parameter adaptation with natural language descriptions. In ACL. [find](https://scholar.google.com/scholar?q=Tian+Jin%2C+Zhun+Liu%2C+Shengjia+Yan%2C+Alexandre+Eichenberger%2C+and+Louis%2DPhilippe+Morency%2E+2020%2E+Language+to+network%3A+Conditional+parameter+adaptation+with+natural+language+descriptions%2E+In+ACL%2E)
7. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In AAAI. [find](https://scholar.google.com/scholar?q=Tushar+Khot%2C+Ashish+Sabharwal%2C+and+Peter+Clark%2E+2018%2E+Scitail%3A+A+textual+entailment+dataset+from+science+question+answering%2E+In+AAAI%2E)
8. Xin Li and Dan Roth. 2002. Learning question classifiers. In COLING. [find](https://scholar.google.com/scholar?q=Xin+Li+and+Dan+Roth%2E+2002%2E+Learning+question+classifiers%2E+In+COLING%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2002-learning-question-classifiers/)
9. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. Multi-task deep neural networks for natural language understanding. In ACL. [find](https://scholar.google.com/scholar?q=Xiaodong+Liu%2C+Pengcheng+He%2C+Weizhu+Chen%2C+and+Jianfeng+Gao%2E+2019a%2E+Multi%2Dtask+deep+neural+networks+for+natural+language+understanding%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/)
10. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019b%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+)
11. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019a. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL. [find](https://scholar.google.com/scholar?q=Christopher+Clark%2C+Kenton+Lee%2C+Ming%2DWei+Chang%2C+Tom+Kwiatkowski%2C+Michael+Collins%2C+and+Kristina+Toutanova%2E+2019a%2E+Boolq%3A+Exploring+the+surprising+difficulty+of+natural+yes%2Fno+questions%2E+In+NAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/)
12. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In ACL. [find](https://scholar.google.com/scholar?q=Andrew+L%2E+Maas%2C+Raymond+E%2E+Daly%2C+Peter+T%2E+Pham%2C+Dan+Huang%2C+Andrew+Y%2E+Ng%2C+and+Christopher+Potts%2E+2011%2E+Learning+word+vectors+for+sentiment+analysis%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/)
13. Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D Manning, and Quoc Le. 2019b. Bam! born-again multi-task networks for natural language understanding. In ACL. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Urvashi+Khandelwal%2C+Christopher+D+Manning%2C+and+Quoc+Le%2E+2019b%2E+Bam%21+born%2Dagain+multi%2Dtask+networks+for+natural+language+understanding%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/clark-2019-bam-born-again-multi-task/)
14. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730. [find](https://scholar.google.com/scholar?q=Bryan+McCann%2C+Nitish+Shirish+Keskar%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2018%2E+The+natural+language+decathlon%3A+Multitask+learning+as+question+answering%2E+arXiv+preprint+arXiv%3A1806%2E08730%2E) [our card](https://md.research.elamur.ai/free/en/papers/mccann-2018-natural-language-decathlon-multitask-learning/)
15. Alexis Conneau and Douwe Kiela. 2018. Senteval: An evaluation toolkit for universal sentence representations. In LREC. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Douwe+Kiela%2E+2018%2E+Senteval%3A+An+evaluation+toolkit+for+universal+sentence+representations%2E+In+LREC%2E) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2018-senteval-evaluation-toolkit-universal-sentence/)
16. Johannes Von Oswald, Christian Henning, João Sacramento, and Benjamin F Grewe. 2020. Continual learning with hypernetworks. In ICLR. [find](https://scholar.google.com/scholar?q=Johannes+Von+Oswald%2C+Christian+Henning%2C+Joa%CC%83o+Sacramento%2C+and+Benjamin+F+Grewe%2E+2020%2E+Continual+learning+with+hypernetworks%2E+In+ICLR%2E)
17. Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. 2018. Film: Visual reasoning with a general conditioning layer. In AAAI. [find](https://scholar.google.com/scholar?q=Ethan+Perez%2C+Florian+Strub%2C+Harm+De+Vries%2C+Vincent+Dumoulin%2C+and+Aaron+Courville%2E+2018%2E+Film%3A+Visual+reasoning+with+a+general+conditioning+layer%2E+In+AAAI%2E) [our card](https://md.research.elamur.ai/free/en/papers/perez-2017-film-visual-reasoning-general-conditioning/)
18. Matthew E Peters, Sebastian Ruder, and Noah A Smith. 2019. To tune or not to tune? adapting pretrained representations to diverse tasks. In RepL4NLP. [find](https://scholar.google.com/scholar?q=Matthew+E+Peters%2C+Sebastian+Ruder%2C+and+Noah+A+Smith%2E+2019%2E+To+tune+or+not+to+tune%3F+adapting+pretrained+representations+to+diverse+tasks%2E+In+RepL4NLP%2E)
19. Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. AdapterHub: A framework for adapting transformers. In EMNLP: System Demonstrations. [find](https://scholar.google.com/scholar?q=Jonas+Pfeiffer%2C+Andreas+Ru%CC%88ckle%CC%81%2C+Clifton+Poth%2C+Aishwarya+Kamath%2C+Ivan+Vulic%CC%81%2C+Sebastian+Ruder%2C+Kyunghyun+Cho%2C+and+Iryna+Gurevych%2E+2020%2E+AdapterHub%3A+A+framework+for+adapting+transformers%2E+In+EMNLP%3A+Sy) [our card](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterhub-framework-adapting-transformers/)
20. Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088. [find](https://scholar.google.com/scholar?q=Jason+Phang%2C+Thibault+Fe%CC%81vry%2C+and+Samuel+R+Bowman%2E+2018%2E+Sentence+encoders+on+stilts%3A+Supplementary+training+on+intermediate+labeled%2Ddata+tasks%2E+arXiv+preprint+arXiv%3A1811%2E01088%2E) [our card](https://md.research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/)
21. Emmanouil Antonios Platanios, Mrinmaya Sachan, Graham Neubig, and Tom Mitchell. 2018. Contextual parameter generation for universal neural machine translation. In EMNLP. [find](https://scholar.google.com/scholar?q=Emmanouil+Antonios+Platanios%2C+Mrinmaya+Sachan%2C+Graham+Neubig%2C+and+Tom+Mitchell%2E+2018%2E+Contextual+parameter+generation+for+universal+neural+machine+translation%2E+In+EMNLP%2E)
22. Edoardo M Ponti, Ivan Vulić, Ryan Cotterell, Marinela Parovic, Roi Reichart, and Anna Korhonen. 2020. Parameter space factorization for zero-shot learning across tasks and languages. arXiv preprint arXiv:2001.11453. [find](https://scholar.google.com/scholar?q=Edoardo+M+Ponti%2C+Ivan+Vulic%CC%81%2C+Ryan+Cotterell%2C+Marinela+Parovic%2C+Roi+Reichart%2C+and+Anna+Korhonen%2E+2020%2E+Parameter+space+factorization+for+zero%2Dshot+learning+across+tasks+and+languages%2E+arXiv+preprint+a)
23. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
24. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J+Liu%2E+2020%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext)
25. Alex Ratner, Braden Hancock, Jared Dunnmon, Roger Goldman, and Christopher Ré. 2018. Snorkel metal: Weak supervision for multi-task learning. In DEEM workshop. [find](https://scholar.google.com/scholar?q=Alex+Ratner%2C+Braden+Hancock%2C+Jared+Dunnmon%2C+Roger+Goldman%2C+and+Christopher+Re%CC%81%2E+2018%2E+Snorkel+metal%3A+Weak+supervision+for+multi%2Dtask+learning%2E+In+DEEM+workshop%2E)
26. Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2018. Efficient parametrization of multi-domain deep neural networks. In CVPR. [find](https://scholar.google.com/scholar?q=Sylvestre%2DAlvise+Rebuffi%2C+Hakan+Bilen%2C+and+Andrea+Vedaldi%2E+2018%2E+Efficient+parametrization+of+multi%2Ddomain+deep+neural+networks%2E+In+CVPR%2E)
27. Sebastian Ruder. 2017. An Overview of Multi-Task Learning in Deep Neural Networks. In arXiv preprint arXiv:1706.05098. [find](https://scholar.google.com/scholar?q=Sebastian+Ruder%2E+2017%2E+An+Overview+of+Multi%2DTask+Learning+in+Deep+Neural+Networks%2E+In+arXiv+preprint+arXiv%3A1706%2E05098%2E) [our card](https://md.research.elamur.ai/free/en/papers/ruder-2017-overview-multi-task-learning-deep/)
28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+NeurIPS%2E)
29. Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, and Mohit Iyyer. 2020. Exploring and predicting transferability across NLP tasks. In EMNLP. [find](https://scholar.google.com/scholar?q=Tu+Vu%2C+Tong+Wang%2C+Tsendsuren+Munkhdalai%2C+Alessandro+Sordoni%2C+Adam+Trischler%2C+Andrew+Mattarella%2DMicke%2C+Subhransu+Maji%2C+and+Mohit+Iyyer%2E+2020%2E+Exploring+and+predicting+transferability+across+NLP+tasks%2E+) [our card](https://md.research.elamur.ai/free/en/papers/vu-2020-exploring-predicting-transferability-across-nlp/)
30. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurIPS. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Yada+Pruksachatkun%2C+Nikita+Nangia%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R+Bowman%2E+2019a%2E+Superglue%3A+A+stickier+benchmark+for+general%2Dpurpose+language+understan) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
31. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R%2E+Bowman%2E+2019b%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
32. Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. 2019c. Characterizing and avoiding negative transfer. In CVPR. [find](https://scholar.google.com/scholar?q=Zirui+Wang%2C+Zihang+Dai%2C+Barnaba%CC%81s+Po%CC%81czos%2C+and+Jaime+Carbonell%2E+2019c%2E+Characterizing+and+avoiding+negative+transfer%2E+In+CVPR%2E)
33. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. TACL. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R%2E+Bowman%2E+2019%2E+Neural+network+acceptability+judgments%2E+TACL%2E)
34. Svante Wold, Kim Esbensen, and Paul Geladi. 1987. Principal component analysis. Chemometrics and intelligent laboratory systems. [find](https://scholar.google.com/scholar?q=Svante+Wold%2C+Kim+Esbensen%2C+and+Paul+Geladi%2E+1987%2E+Principal+component+analysis%2E+Chemometrics+and+intelligent+laboratory+systems%2E)
35. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pierric+Cistac%2C+Tim+Rault%2C+Re%CC%81mi+Louf%2C+Morgan+Funtowicz%2C+Joe+Davison%2C+Sam+Shleifer%2C+Patrick+von+Platen%2C+Clara+) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
36. Thomas Wolf, Quentin Lhoest, Patrick von Platen, Yacine Jernite, Mariama Drame, Julien Plu, Julien Chaumond, Clement Delangue, Clara Ma, Abhishek Thakur, Suraj Patil, Joe Davison, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angie McMillan-Major, Simon Brandeis, Sylvain Gugger, François La… [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Quentin+Lhoest%2C+Patrick+von+Platen%2C+Yacine+Jernite%2C+Mariama+Drame%2C+Julien+Plu%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Clara+Ma%2C+Abhishek+Thakur%2C+Suraj+Patil%2C+Joe+Davison%2C+Teven+Le+Scao%2C+Victo)
37. Asa Cooper Stickland and Iain Murray. 2019. Bert and pals: Projected attention layers for efficient adaptation in multi-task learning. In ICML. [find](https://scholar.google.com/scholar?q=Asa+Cooper+Stickland+and+Iain+Murray%2E+2019%2E+Bert+and+pals%3A+Projected+attention+layers+for+efficient+adaptation+in+multi%2Dtask+learning%2E+In+ICML%2E)
38. Wenpeng Yin, Nazneen Fatema Rajani, Dragomir Radev, Richard Socher, and Caiming Xiong. 2020. Universal natural language processing with limited annotations: Try few-shot textual entailment as a start. In EMNLP. [find](https://scholar.google.com/scholar?q=Wenpeng+Yin%2C+Nazneen+Fatema+Rajani%2C+Dragomir+Radev%2C+Richard+Socher%2C+and+Caiming+Xiong%2E+2020%2E+Universal+natural+language+processing+with+limited+annotations%3A+Try+few%2Dshot+textual+entailment+as+a+start%2E)
39. Yi Tay, Zhe Zhao, Dara Bahri, Don Metzler, and Da-Cheng Juan. 2021. Hypergrid transformers: Towards a single model for multiple tasks. In ICLR. [find](https://scholar.google.com/scholar?q=Yi+Tay%2C+Zhe+Zhao%2C+Dara+Bahri%2C+Don+Metzler%2C+and+Da%2DCheng+Juan%2E+2021%2E+Hypergrid+transformers%3A+Towards+a+single+model+for+multiple+tasks%2E+In+ICLR%2E)
40. Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. 2021. Revisiting few-sample bert fine-tuning. In ICLR. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Felix+Wu%2C+Arzoo+Katiyar%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2021%2E+Revisiting+few%2Dsample+bert+fine%2Dtuning%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2020-revisiting-few-sample-bert-fine/)
41. Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gertjan van Noord. 2020. Udapter: Language adaptation for truly universal dependency parsing. In EMNLP. [find](https://scholar.google.com/scholar?q=Ahmet+U%CC%88stu%CC%88n%2C+Arianna+Bisazza%2C+Gosse+Bouma%2C+and+Gertjan+van+Noord%2E+2020%2E+Udapter%3A+Language+adaptation+for+truly+universal+dependency+parsing%2E+In+EMNLP%2E)
42. Xiang Zhang, Junbo Zhao, and Yann Lecun. 2015. Character-level convolutional networks for text classification. In NeurIPS. [find](https://scholar.google.com/scholar?q=Xiang+Zhang%2C+Junbo+Zhao%2C+and+Yann+Lecun%2E+2015%2E+Character%2Dlevel+convolutional+networks+for+text+classification%2E+In+NeurIPS%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/)