--- title: "The Power of Scale for Parameter-Efficient Prompt Tuning (2021)" description: "The authors propose prompt tuning: several \"soft\" tokens are prepended to the input text. 51 references, cited by 1,372 papers here." authors: ["Brian Lester", "Rami Al-Rfou", "Noah Constant"] year: 2021 venue: "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing" source: "https://aclanthology.org/2021.emnlp-main.243/" doi: "10.18653/v1/2021.emnlp-main.243" arxiv: "2104.08691" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/) / The Power of Scale for Parameter-Efficient Prompt Tuning # The Power of Scale for Parameter-Efficient Prompt Tuning **Authors:** [Brian Lester](https://md.research.elamur.ai/free/en/authors/brian-lester/), [Rami Al-Rfou](https://md.research.elamur.ai/free/en/authors/rami-al-rfou/) and [Noah Constant](https://md.research.elamur.ai/free/en/authors/noah-constant/) · **Year:** 2021 · **Venue:** [Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2021](https://md.research.elamur.ai/free/en/venues/emnlp/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2021.emnlp-main.243/) · [PDF](https://aclanthology.org/2021.emnlp-main.243.pdf) · [DOI 10.18653/v1/2021.emnlp-main.243](https://doi.org/10.18653/v1/2021.emnlp-main.243) · [arXiv 2104.08691](https://arxiv.org/abs/2104.08691) · [OpenAlex](https://openalex.org/W3152956381) **Citations:** 2,915 citations (OpenAlex, 2026-10-14) · [1372](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper examines how to adapt a large pretrained language model to specific tasks without changing its weights. The authors propose prompt tuning: several "soft" tokens are prepended to the input text. Their embedding vectors are learned through backpropagation, while the T5 model itself remains frozen. The method was evaluated on SuperGLUE with T5 models ranging in size from Small to XXL. The authors also separately investigated how to initialize the prompt, how long it should be, and which pretraining is better suited: span corruption or additional LM adaptation. The main conclusion is that the larger the model, the smaller the gap between prompt tuning and full fine-tuning. At the scale of several billion parameters, it effectively catches up with fine-tuning while substantially outperforming GPT-3 few-shot prompts. The practical benefit is that a single frozen model serves many tasks, only a tiny set of parameters is stored per task, and requests for different tasks can be processed in a single batch. In addition, soft prompts are more robust to domain shift and allow ensembles to be built cheaply. The authors released their code and checkpoints. ## Key points - **The gap being closed:** GPT-3 175B in the few-shot setting scores 71.8 on SuperGLUE, whereas T5-XXL after fine-tuning reaches 89.3. Prompt tuning on T5-XXL reaches the level of fine-tuning. - **Parameter savings:** a single task-tuned copy of T5-XXL has 11 billion parameters. A 5-token prompt for the same model amounts to only 20,480 parameters, i.e., roughly five orders of magnitude fewer. For XXL, the fraction of trainable parameters is typically below 0.01%. - **Pretraining matters:** the T5 1.1 model, trained only on span corruption, is poorly controllable via prompts. The authors therefore additionally train it with an LM objective (LM adaptation, up to 100K steps) so that it generates natural text. - **Comparison with related methods:** the method is simpler than prefix tuning (Li & Liang, 2021), since it does not add prefixes to intermediate layers and has no separate task-specific output layers. Nevertheless, on large models it performs on par with fine-tuning. - **Additional advantages:** better transfer to other domains (MRQA out-of-domain, the QQP/MRPC pairs) and "prompt ensembling", i.e., multiple prompts for a single task on a shared model. Code: github.com/google-research/prompt-tuning. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 430 works · 424,363 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 💡 Concepts (1): [💡 Fine-tuning](https://md.research.elamur.ai/free/en/concepts/fine-tuning/) · 27 works ## Cite Cite as: Brian Lester, Rami Al-Rfou and Noah Constant. 2021. The Power of Scale for Parameter-Efficient Prompt Tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{lester2021power, title = {{The Power of Scale for Parameter-Efficient Prompt Tuning}}, author = {Brian Lester, Rami Al-Rfou and Noah Constant}, booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing}, year = {2021}, doi = {10.18653/v1/2021.emnlp-main.243}, eprint = {2104.08691}, archivePrefix = {arXiv}, url = {https://doi.org/10.18653/v1/2021.emnlp-main.243} } ``` - [← previous Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/) - [next → SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/) - [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/) ## References 51 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. 2018. JAX: composable transformations of Python+NumPy programs. [find](https://scholar.google.com/scholar?q=Maclaurin%2C+George+Necula%2C+Adam+Paszke%2C+Jake+VanderPlas%2C+Skye+Wanderman%2DMilne%2C+and+Qiao+Zhang%2E+2018%2E+JAX%3A+composable+transformations+of+Python%2BNumPy+programs%2E) 2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/) 3. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [BoolQ: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:… 4. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177–190. Springer. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2005%2E+The+PASCAL+recognising+textual+entailment+challenge%2E+In+Machine+Learning+Challenges+Workshop%2C+pages+177%E2%80%93190%2E+Springer%2E) 5. Marie-Catherine De Marneff, Mandy Simons, and Judith Tonhauser. 2019. [The CommitmentBank: Investigating projection in naturally occurring discourse](https://md.research.elamur.ai/free/en/papers/marneffe-2019-commitmentbank-investigating-projection-naturally/). Proceedings of Sinn und Bedeutung 23. 6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… 7. William B Dolan and Chris Brockett. 2005. [Automatically constructing a corpus of sentential paraphrases](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/). In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). 8. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. [DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistic… 9. Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. [find](https://scholar.google.com/scholar?q=Adam+Fisch%2C+Alon+Talmor%2C+Robin+Jia%2C+Minjoon+Seo%2C+Eunsol+Choi%2C+and+Danqi+Chen%2E+2019%2E) 10. [MRQA 2019 shared task: Evaluating generalization in reading comprehension](https://md.research.elamur.ai/free/en/papers/fisch-2019-mrqa-2019-shared-task-evaluating/). In Proceedings of 2nd Machine Reading for Reading Comprehension (MRQA) Workshop at EMNLP. 11. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. [The third PASCAL recognizing textual entailment challenge](https://md.research.elamur.ai/free/en/papers/giampiccolo-2007-third-pascal-recognizing-textual-entailment/). In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1–9. Association for Computational Linguistics. 12. Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: Word-level Adversarial ReProgramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), p… [find](https://scholar.google.com/scholar?q=Karen+Hambardzumyan%2C+Hrant+Khachatrian%2C+and+Jonathan+May%2E+2021%2E+WARP%3A+Word%2Dlevel+Adversarial+ReProgramming%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+a) 13. L. K. Hansen and P. Salamon. 1990. Neural network ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(10):993–1001. [find](https://scholar.google.com/scholar?q=L%2E+K%2E+Hansen+and+P%2E+Salamon%2E+1990%2E+Neural+network+ensembles%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%2C+12%2810%29%3A993%E2%80%931001%2E) 14. Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. 2020. Flax: A neural network library and ecosystem for JAX. [find](https://scholar.google.com/scholar?q=Jonathan+Heek%2C+Anselm+Levskaya%2C+Avital+Oliver%2C+Marvin+Ritter%2C+Bertrand+Rondepierre%2C+Andreas+Steiner%2C+and+Marc+van+Zee%2E+2020%2E+Flax%3A+A+neural+network+library+and+ecosystem+for+JAX%2E) 15. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Ma… [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+De+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+NLP%2E+In+) 16. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volum) 17. Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. 2017. First Quora dataset release: Question pairs. [find](https://scholar.google.com/scholar?q=Shankar+Iyer%2C+Nikhil+Dandekar%2C+and+Kornel+Csernai%2E+2017%2E+First+Quora+dataset+release%3A+Question+pairs%2E) 18. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. [How can we know what language models know?](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/) Transactions of the Association for Computational Linguistics, 8:423–438. 19. A. Kembhavi, M. Seo, D. Schwenk, J. Choi, A. Farhadi, and H. Hajishirzi. 2017. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5376–5384. [find](https://scholar.google.com/scholar?q=A%2E+Kembhavi%2C+M%2E+Seo%2C+D%2E+Schwenk%2C+J%2E+Choi%2C+A%2E+Farhadi%2C+and+H%2E+Hajishirzi%2E+2017%2E+Are+you+smarter+than+a+sixth+grader%3F+textbook+question+answering+for+multimodal+machine+comprehension%2E+In+2017+IEEE+Confe) 20. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. [Looking beyond the surface: A challenge set for reading comprehension over multiple sentences](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/). In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL). 21. Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. 2019. [A surprisingly robust trick for the Winograd schema challenge](https://md.research.elamur.ai/free/en/papers/kocijan-2019-surprisingly-robust-trick-winograd-schema/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4837–4842, Florence, Italy. Association fo… 22. Taku Kudo and John Richardson. 2018. [SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ… 23. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. [RACE: Large-scale ReAding comprehension dataset from examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794, Copenhagen, Denmark. Association for Computational Ling… 24. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. [GPT understands, too](https://md.research.elamur.ai/free/en/papers/liu-2021-gpt-understands-too/). CoRR, abs/2103.10385. 25. Lajanugen Logeswaran, Ann Lee, Myle Ott, Honglak Lee, Marc’Aurelio Ranzato, and Arthur Szlam. 2020. Few-shot sequence learning with transformers. CoRR, abs/2012.09543. — [arXiv:2012.09543](https://arxiv.org/abs/2012.09543) [find](https://scholar.google.com/scholar?q=Lajanugen+Logeswaran%2C+Ann+Lee%2C+Myle+Ott%2C+Honglak+Lee%2C+Marc%E2%80%99Aurelio+Ranzato%2C+and+Arthur+Szlam%2E+2020%2E+Few%2Dshot+sequence+learning+with+transformers%2E+CoRR%2C+abs%2F2012%2E09543%2E+%E2%80%94+%5BarXiv%3A2012%2E09543%5D%28https%3A%2F%2Farx) 26. Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML’10, page 807–814, Madison, WI, USA. Omnipress. [find](https://scholar.google.com/scholar?q=Vinod+Nair+and+Geoffrey+E%2E+Hinton%2E+2010%2E+Rectified+linear+units+improve+restricted+Boltzmann+machines%2E+In+Proceedings+of+the+27th+International+Conference+on+International+Conference+on+Machine+Learni) 27. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… 28. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Balaji+Lakshminarayanan%2C+Alexander+Pritzel%2C+and+Charles+Blundell%2E+2017%2E+Simple+and+scalable+predictive+uncertainty+estimation+using+deep+ensembles%2E+In+Advances+in+Neural+Information+Processing+Systems) 29. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-mad-x-adapter-based-framework/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673, Online. Association for Computational L… 30. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+Winograd+schema+challenge%2E+In+Thirteenth+International+Conference+on+the+Principles+of+Knowledge+Representation+and+Reasoning%2E) 31. Mohammad Taher Pilehvar and Jose Camacho-Collados. 2018. [WiC: 10,000 example pairs for evaluating context-sensitive representations](https://md.research.elamur.ai/free/en/papers/pilehvar-2019-wic-word-context-dataset-evaluating/). CoRR, abs/1808.09121. 32. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. [Zero-shot relation extraction via reading comprehension](https://md.research.elamur.ai/free/en/papers/levy-2017-zero-shot-relation-extraction-reading/). In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 333–342, Vancouver, Canada. Association for Computational Linguistics. 33. Guanghui Qin and Jason Eisner. 2021. [Learning how to ask: Querying LMs with mixtures of soft prompts](https://md.research.elamur.ai/free/en/papers/qin-2021-learning-ask-querying-lms-mixtures/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5203–5212, Online. Association for Computatio… 34. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the… 35. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E) 36. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–… [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11t) 37. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2E) 38. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67. 39. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. 40. Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. [Learning multiple visual domains with residual adapters](https://md.research.elamur.ai/free/en/papers/rebuffi-2017-learning-multiple-visual-domains-residual/). In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. 41. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series. [find](https://scholar.google.com/scholar?q=Melissa+Roemmele%2C+Cosmin+Adrian+Bejan%2C+and+Andrew+S+Gordon%2E+2011%2E+Choice+of+plausible+alternatives%3A+An+evaluation+of+commonsense+causal+reasoning%2E+In+2011+AAAI+Spring+Symposium+Series%2E) 42. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In the Proceedings of ICLR. 43. Michael L. Waskom. 2021. seaborn: statistical data visualization. Journal of Open Source Software, 6(60):3021. [find](https://scholar.google.com/scholar?q=Michael+L%2E+Waskom%2E+2021%2E+seaborn%3A+statistical+data+visualization%2E+Journal+of+Open+Source+Software%2C+6%2860%29%3A3021%2E) 44. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. [ReCoRD: Bridging the gap between human and machine commonsense reading comprehension](https://md.research.elamur.ai/free/en/papers/zhang-2018-record-bridging-gap-between-human/). CoRR, abs/1810.12885. 45. Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, and Karthik Sankaranarayanan. 2018. [DuoRC: Towards complex language understanding with paraphrased reading comprehension](https://md.research.elamur.ai/free/en/papers/saha-2018-duorc-complex-language-understanding-paraphrased/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1683–16… 46. Timo Schick and Hinrich Schütze. 2021. [Exploiting cloze-questions for few-shot text classification and natural language inference](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, Online. Association for Compu… 47. Noam Shazeer. 2020. GLU variants improve transformer. CoRR, abs/2002.05202. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+2020%2E+GLU+variants+improve+transformer%2E+CoRR%2C+abs%2F2002%2E05202%2E) 48. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR. [find](https://scholar.google.com/scholar?q=Noam+Shazeer+and+Mitchell+Stern%2E+2018%2E+Adafactor%3A+Adaptive+learning+rates+with+sublinear+memory+cost%2E+In+Proceedings+of+the+35th+International+Conference+on+Machine+Learning%2C+volume+80+of+Proceedings+) 49. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235, Onlin… 50. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30, pages 5998–6008. 51. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019a. [SuperGLUE: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. *Updated 29 October 2026*