--- title: "Cutting Down on Prompts and Parameters — IV 2022" description: "Few-shot learning with masked language models usually depends on carefully crafted prompts." authors: ["Robert Logan IV", "Ivana Balazevic", "Eric Wallace", "Fabio Petroni", "Sameer Singh", "Sebastian Riedel"] year: 2022 venue: "Findings of the Association for Computational Linguistics: ACL 2022" source: "https://aclanthology.org/2022.findings-acl.222/" doi: "10.18653/v1/2022.findings-acl.222" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/iv-2022-cutting-down-prompts-parameters-simple/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/iv-2022-cutting-down-prompts-parameters-simple/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/) / Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models # Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models **Authors:** Robert Logan IV, [Ivana Balazevic](https://md.research.elamur.ai/free/en/authors/ivana-balazevic/), [Eric Wallace](https://md.research.elamur.ai/free/en/authors/eric-wallace/), [Fabio Petroni](https://md.research.elamur.ai/free/en/authors/fabio-petroni/), [Sameer Singh](https://md.research.elamur.ai/free/en/authors/sameer-singh/), [Sebastian Riedel](https://md.research.elamur.ai/free/en/authors/sebastian-riedel/) · **Year:** 2022 · **Venue:** Findings of the Association for Computational Linguistics: ACL 2022 · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2022.findings-acl.222/) · [PDF](https://aclanthology.org/2022.findings-acl.222.pdf) · [DOI 10.18653/v1/2022.findings-acl.222](https://doi.org/10.18653/v1/2022.findings-acl.222) · [OpenAlex](https://openalex.org/W3173617765) **Citations:** 128 citations (openalex-snapshot · 2026-10-14) · [87](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Few-shot learning with masked language models usually depends on carefully crafted prompts. Prompt-based finetuning is accurate but needs a full model copy for every task, while in-context learning reuses one frozen model but is very sensitive to prompt wording. The authors test whether simple prompt design and memory efficiency can be achieved together. They use RoBERTa and ALBERT on eight GLUE/SuperGLUE classification tasks, with 16 examples per label and cross-validated model selection. They introduce "null prompts," which simply concatenate the input fields with a [MASK] token and use no template text or demonstrations. Under finetuning, these prompts perform about as well as manually written ones. Automated prompt tuning for frozen models generally fell short of manual prompts. Among lightweight finetuning options, updating only the bias terms (BitFit) gave the best accuracy-to-cost balance, often matching or beating full finetuning. The recommended recipe is null prompts plus BitFit. It reduces the effort of prompt engineering and per-task storage, and it suggests that few-shot gains come largely from the [MASK]-prediction setup itself rather than from clever patterns. ## Key points - The eight evaluation tasks were SST-2, RTE, QQP, QNLI, MRPC, MNLI, CB and BoolQ, each trained with K = 16 examples per label. Hyperparameters were chosen by 4-fold cross-validation, and results were averaged over 10 random seeds. WiC and WNLI were dropped because every model performed near chance. - With RoBERTa, null prompts plus BitFit reached 90.6 F1 on CB and 65.1 F1 on QQP. In-context learning scored 51.2 and 55.6, and full prompt-based finetuning scored 90.6 and 62.9. - BitFit trains roughly 10^5 parameters per task, about 0.1% of the model, which the authors describe as around 1000x more memory efficient than full finetuning. The other lightweight methods compared were Adapters (~10^7 parameters), LM head tuning (~10^3) and calibration (~10^1). - Accuracy showed diminishing returns as more parameters were trained. Going beyond BitFit to Adapters or all parameters added little or even hurt results on 16-example training sets. - Code is released at github.com/ucinlp/null-prompts. Extending the findings to very large and left-to-right LMs is left for future work. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 87 papers of our library cite it (of the 140884 whose bibliographies we read); 15 of them have a page here, 72 not yet. - [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/) - [Promptagator: Few-shot Dense Retrieval From 8 Examples](https://md.research.elamur.ai/free/en/papers/dai-2022-promptagator-few-shot-dense-retrieval/) - [OpenPrompt: An Open-source Framework for Prompt-learning](https://md.research.elamur.ai/free/en/papers/ding-2022-openprompt-open-source-framework-prompt/) - [PPT: Pre-trained Prompt Tuning for Few-shot Learning](https://md.research.elamur.ai/free/en/papers/gu-2022-ppt-pre-trained-prompt-tuning/) - [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/) - [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://md.research.elamur.ai/free/en/papers/meng-2022-generating-training-data-language-models/) - [Noisy Channel Language Model Prompting for Few-Shot Text Classification](https://md.research.elamur.ai/free/en/papers/min-2021-noisy-channel-language-model-prompting/) - [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/) - [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) - [Reframing Instructional Prompts to GPTk's Language](https://md.research.elamur.ai/free/en/papers/mishra-2022-reframing-instructional-prompts-gptk-s/) - [Crosslingual Generalization through Multitask Finetuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/) - [SGPT: GPT Sentence Embeddings for Semantic Search](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-sgpt-gpt-sentence-embeddings-semantic/) - [GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models](https://md.research.elamur.ai/free/en/papers/prasad-2023-grips-gradient-free-edit-based/) - [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/) - [Do Prompt-Based Models Really Understand the Meaning of Their Prompts?](https://md.research.elamur.ai/free/en/papers/webson-2022-do-prompt-based-models-really/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{iv2022cutting, title = {{Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models}}, author = {IV, Robert Logan and Balazevic, Ivana and Wallace, Eric and Petroni, Fabio and Singh, Sameer and Riedel, Sebastian}, booktitle = {Findings of the Association for Computational Linguistics: ACL 2022}, year = {2022}, doi = {10.18653/v1/2022.findings-acl.222}, url = {https://doi.org/10.18653/v1/2022.findings-acl.222} } ``` - [← previous Current Trends and Approaches in Synonyms Extraction: Potential Adaptation to Arabic](https://md.research.elamur.ai/free/en/papers/naser-2022-karajah-current-trends-approaches-synonyms-extraction/) - [next → Dartmouth at SemEval-2022 Task 6: Detection of Sarcasm](https://md.research.elamur.ai/free/en/papers/lad-2022-dartmouth-semeval-2022-task-6/) - [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/) ## References 27 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Elad Ben-Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. [find](https://scholar.google.com/scholar?q=Elad+Ben%2DZaken%2C+Shauli+Ravfogel%2C+and+Yoav+Goldberg%2E+2021%2E+BitFit%3A+Simple+parameter%2Defficient+fine%2Dtuning+for+transformer%2Dbased+masked+language%2Dmodels%2E) [our card](https://md.research.elamur.ai/free/en/papers/ben-2021-zaken-bitfit-simple-parameter-efficient-fine/) 2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche) 3. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. ArXiv preprint, abs/2002.06305. [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Gabriel+Ilharco%2C+Roy+Schwartz%2C+Ali+Farhadi%2C+Hannaneh+Hajishirzi%2C+and+Noah+Smith%2E+2020%2E+Fine%2Dtuning+pretrained+language+models%3A+Weight+initializations%2C+data+orders%2C+and+early+stopping%2E+ArX) 4. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pa… [find](https://scholar.google.com/scholar?q=Tianyu+Gao%2C+Adam+Fisch%2C+and+Danqi+Chen%2E+2021%2E+Making+pre%2Dtrained+language+models+better+few%2Dshot+learners%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+an) [our card](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/) 5. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Lon… [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+de+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+NLP%2E+In+) 6. Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. 2015. Human-level concept learning through probabilistic program induction. In Science. [find](https://scholar.google.com/scholar?q=Brenden+M+Lake%2C+Ruslan+Salakhutdinov%2C+and+Joshua+B+Tenenbaum%2E+2015%2E+Human%2Dlevel+concept+learning+through+probabilistic+program+induction%2E+In+Science%2E) 7. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRevi… [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2020%2E+ALBERT%3A+A+lite+BERT+for+self%2Dsupervised+learning+of+language+representations%2E+In+8th+International+C) 8. Teven Le Scao and Alexander Rush. 2021. How many data points is a prompt worth? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2627–2636, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao+and+Alexander+Rush%2E+2021%2E+How+many+data+points+is+a+prompt+worth%3F+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Hu) [our card](https://md.research.elamur.ai/free/en/papers/scao-2021-how-many-data-points-is/) 9. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Brian+Lester%2C+Rami+Al%2DRfou%2C+and+Noah+Constant%2E+2021%2E+The+power+of+scale+for+parameter%2Defficient+prompt+tuning%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processing) 10. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–… [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11t) 11. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT understands, too. ArXiv preprint, abs/2103.10385. [find](https://scholar.google.com/scholar?q=Xiao+Liu%2C+Yanan+Zheng%2C+Zhengxiao+Du%2C+Ming+Ding%2C+Yujie+Qian%2C+Zhilin+Yang%2C+and+Jie+Tang%2E+2021%2E+GPT+understands%2C+too%2E+ArXiv+preprint%2C+abs%2F2103%2E10385%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2021-gpt-understands-too/) 12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. ArXiv preprint, abs/1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+A) 13. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. ArXiv preprint, abs/2104.08786. [find](https://scholar.google.com/scholar?q=Yao+Lu%2C+Max+Bartolo%2C+Alastair+Moore%2C+Sebastian+Riedel%2C+and+Pontus+Stenetorp%2E+2021%2E+Fantastically+ordered+prompts+and+where+to+find+them%3A+Overcoming+few%2Dshot+prompt+order+sensitivity%2E+ArXiv+preprint%2C+a) [our card](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/) 14. Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021. On the stability of fine-tuning BERT: misconceptions, explanations, and strong baselines. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. [find](https://scholar.google.com/scholar?q=Marius+Mosbach%2C+Maksym+Andriushchenko%2C+and+Dietrich+Klakow%2E+2021%2E+On+the+stability+of+fine%2Dtuning+BERT%3A+misconceptions%2C+explanations%2C+and+strong+baselines%2E+In+9th+International+Conference+on+Learning+) 15. Juri Opitz. 2019. Argumentative relation classification as plausibility ranking. In Proceedings of the 15th Conference on Natural Language Processing (KON-VENS 2019): Long Papers, pages 193–202, Erlangen, Germany. German Society for Computational Linguistics & Language Technology. [find](https://scholar.google.com/scholar?q=Juri+Opitz%2E+2019%2E+Argumentative+relation+classification+as+plausibility+ranking%2E+In+Proceedings+of+the+15th+Conference+on+Natural+Language+Processing+%28KON%2DVENS+2019%29%3A+Long+Papers%2C+pages+193%E2%80%93202%2C+Erlan) 16. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 12697–12706. PMLR. [find](https://scholar.google.com/scholar?q=In+Proceedings+of+the+38th+International+Conference+on+Machine+Learning%2C+ICML+2021%2C+18%2D24+July+2021%2C+Virtual+Event%2C+volume+139+of+Proceedings+of+Machine+Learning+Research%2C+pages+12697%E2%80%9312706%2E+PMLR%2E) 17. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. ArXiv preprint, abs/2105.11447. [find](https://scholar.google.com/scholar?q=Ethan+Perez%2C+Douwe+Kiela%2C+and+Kyunghyun+Cho%2E+2021%2E+True+few%2Dshot+learning+with+language+models%2E+ArXiv+preprint%2C+abs%2F2105%2E11447%2E) [our card](https://md.research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/) 18. Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5203–5212, Online. Association for Computatio… [find](https://scholar.google.com/scholar?q=Guanghui+Qin+and+Jason+Eisner%2E+2021%2E+Learning+how+to+ask%3A+Querying+LMs+with+mixtures+of+soft+prompts%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association+for+Computa) [our card](https://md.research.elamur.ai/free/en/papers/qin-2021-learning-ask-querying-lms-mixtures/) 19. Timo Schick and Hinrich Schütze. 2021a. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, Online. Association for Comp… [find](https://scholar.google.com/scholar?q=Timo+Schick+and+Hinrich+Sch%C3%BCtze%2E+2021a%2E+Exploiting+cloze%2Dquestions+for+few%2Dshot+text+classification+and+natural+language+inference%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Chapter+of+the) [our card](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/) 20. Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [MASK]: Learning vs. learning to recall. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5017–5033, Online. Association for Co… [find](https://scholar.google.com/scholar?q=Zexuan+Zhong%2C+Dan+Friedman%2C+and+Danqi+Chen%2E+2021%2E+Factual+probing+is+%5BMASK%5D%3A+Learning+vs%2E+learning+to+recall%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association+for) [our card](https://md.research.elamur.ai/free/en/papers/zhong-2021-factual-probing-is-mask-learning/) 21. Timo Schick and Hinrich Schütze. 2021b. It’s not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339–2352, Online. Assoc… [find](https://scholar.google.com/scholar?q=Timo+Schick+and+Hinrich+Sch%C3%BCtze%2E+2021b%2E+It%E2%80%99s+not+just+size+that+matters%3A+Small+language+models+are+also+few%2Dshot+learners%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+As) [our card](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/) 22. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235, Onlin… [find](https://scholar.google.com/scholar?q=Taylor+Shin%2C+Yasaman+Razeghi%2C+Robert+L%2E+Logan+IV%2C+Eric+Wallace%2C+and+Sameer+Singh%2E+2020%2E+AutoPrompt%3A+Eliciting+Knowledge+from+Language+Models+with+Automatically+Generated+Prompts%2E+In+Proceedings+of+the) [our card](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/) 23. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural… [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Yada+Pruksachatkun%2C+Nikita+Nangia%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R%2E+Bowman%2E+2019a%2E+Superglue%3A+A+stickier+benchmark+for+general%2Dpurpose+language+understa) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/) 24. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRevi… [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R%2E+Bowman%2E+2019b%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+7th+Internatio) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/) 25. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2020. HuggingFace’s Transformers: State-of-the-art natural language processing. In EMNLP Demo Track. [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pier%2Dric+Cistac%2C+Tim+Rault%2C+R%E2%80%99emi+Louf%2C+Morgan+Funtowicz%2C+and+Jamie+Brew%2E+2020%2E+HuggingFace%E2%80%99s+Transformers%3A+St) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/) 26. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. [find](https://scholar.google.com/scholar?q=Zihao+Zhao%2C+Eric+Wallace%2C+Shi+Feng%2C+Dan+Klein%2C+and+Sameer+Singh%2E+2021%2E+Calibrate+before+use%3A+Improving+few%2Dshot+performance+of+language+models%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/) 27. Dataset Pattern Verbalizer BoolQ {passage}. Question: {question}? Answer: [MASK]. True: "Yes" False: "No" CB {premise}? [SEP] [MASK], {hypothesis} entailment: "Yes" contradiction: "No" neutral: "Maybe" MNLI {sentence1}? [SEP] [MASK], {sentence2} entailment: "Yes" contradiction: "No" neutral: "Maybe"… [find](https://scholar.google.com/scholar?q=Dataset+Pattern+Verbalizer+BoolQ+%7Bpassage%7D%2E+Question%3A+%7Bquestion%7D%3F+Answer%3A+%5BMASK%5D%2E+True%3A+%22Yes%22+False%3A+%22No%22+CB+%7Bpremise%7D%3F+%5BSEP%5D+%5BMASK%5D%2C+%7Bhypothesis%7D+entailment%3A+%22Yes%22+contradiction%3A+%22No%22+neutral%3A+%22Maybe)