---
title: "Making Pre-trained Language Models Better — Gao 2021"
description: "GPT-3 showed that a very large model can handle new tasks from a prompt and a few examples alone."
authors: ["Tianyu Gao", "Adam Fisch", "Danqi Chen"]
year: 2021
venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)"
source: "https://aclanthology.org/2021.acl-long.295/"
doi: "10.18653/v1/2021.acl-long.295"
arxiv: "2012.15723"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / Making Pre-trained Language Models Better Few-shot Learners

# Making Pre-trained Language Models Better Few-shot Learners

**Authors:** [Tianyu Gao](https://md.research.elamur.ai/free/en/authors/tianyu-gao/), [Adam Fisch](https://md.research.elamur.ai/free/en/authors/adam-fisch/), [Danqi Chen](https://md.research.elamur.ai/free/en/authors/danqi-chen/) · **Year:** 2021 · **Venue:** Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.acl-long.295/) · [PDF](https://aclanthology.org/2021.acl-long.295.pdf) · [DOI 10.18653/v1/2021.acl-long.295](https://doi.org/10.18653/v1/2021.acl-long.295) · [arXiv 2012.15723](https://arxiv.org/abs/2012.15723) · [OpenAlex](https://openalex.org/W3173777717)

**Citations:** 1254 citations (openalex-snapshot · 2026-10-14) · [698](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
GPT-3 showed that a very large model can handle new tasks from a prompt and a few examples alone. At 175B parameters, however, it is impractical for most users. This paper asks whether mid-sized masked language models such as BERT or RoBERTa, which can be fine-tuned on ordinary research hardware, can also learn well from only a handful of labelled examples.

The authors propose LM-BFF, a set of techniques built on prompt-based fine-tuning. Each task is recast as predicting a label word for a [MASK] slot inside a template. Templates are generated automatically with T5, and label words are found through a pruned brute-force search, using only the few-shot data. Each input is also extended with one demonstration per class, preferably examples semantically close to the input.

Tested with RoBERTa-large on 15 English tasks under a strict evaluation protocol, LM-BFF clearly outperforms standard fine-tuning. The work matters because it offers a task-agnostic recipe that needs no unlabeled data, auxiliary tasks or expert prompt engineering.

## Key points
- **Evaluation setup:**
  - The benchmark covers 8 single-sentence and 7 sentence-pair tasks: 8 GLUE tasks, SNLI, SST-5, MR, CR, MPQA, Subj and TREC.
  - Default setting is K = 16 examples per class.
  - Results are averaged over 5 random train/dev splits, with the dev set the same size as the training set.
- **Main result:** Gains over vanilla fine-tuning reach as much as 30 points absolute and average 11 points. RoBERTa-large reaches roughly 90% accuracy on most binary classification tasks with just 32 training examples.
- **Prompt search:** Automatically searched templates (Auto T) match or exceed manual prompts. Ensembles of automatic templates perform on par with or better than PET's manual prompts on MNLI and RTE.
- **Demonstration sampling:** Choosing similar demonstrations with SBERT, or with mean-pooled RoBERTa embeddings, beats random sampling. On SNLI, the advantage over standard fine-tuning disappears around K = 256.
- **Limitations and code:**
  - Performance still trails full-data fine-tuning and shows high variance.
  - The method suits short, fill-in-the-blank tasks with few classes.
  - Code is available at github.com/princeton-nlp/LM-BFF.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

698 papers of our library cite it (of the 140884 whose bibliographies we read); 65 of them have a page here, 633 not yet.

- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://md.research.elamur.ai/free/en/papers/bach-2022-promptsource-integrated-development-environment-repository/)
- [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
- [DiscoPrompt: Path Prediction Prompt Tuning for Implicit Discourse Relation Recognition](https://md.research.elamur.ai/free/en/papers/chan-2023-discoprompt-path-prediction-prompt-tuning/)
- [A Survey on Evaluation of Large Language Models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://md.research.elamur.ai/free/en/papers/chao-2023-jailbreaking-black-box-large-language/)
- [Template-Based Named Entity Recognition Using BART](https://md.research.elamur.ai/free/en/papers/cui-2021-template-based-named-entity-recognition/)
- [Promptagator: Few-shot Dense Retrieval From 8 Examples](https://md.research.elamur.ai/free/en/papers/dai-2022-promptagator-few-shot-dense-retrieval/)
- [AugGPT: Leveraging ChatGPT for Text Data Augmentation](https://md.research.elamur.ai/free/en/papers/dai-2023-auggpt-leveraging-chatgpt-text-data/)
- [Security and Privacy Challenges of Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/das-2024-security-privacy-challenges-large-language/)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deng-2022-rlprompt-optimizing-discrete-text-prompts/)
- [Is GPT-3 a Good Data Annotator?](https://md.research.elamur.ai/free/en/papers/ding-2022-gpt-3-good-data-annotator/)
- [OpenPrompt: An Open-source Framework for Prompt-learning](https://md.research.elamur.ai/free/en/papers/ding-2022-openprompt-open-source-framework-prompt/)
- [Improving alignment of dialogue agents via targeted human judgements](https://md.research.elamur.ai/free/en/papers/glaese-2022-improving-alignment-dialogue-agents-targeted/)
- [PPT: Pre-trained Prompt Tuning for Few-shot Learning](https://md.research.elamur.ai/free/en/papers/gu-2022-ppt-pre-trained-prompt-tuning/)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://md.research.elamur.ai/free/en/papers/hartvigsen-2022-toxigen-large-scale-machine-generated/)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://md.research.elamur.ai/free/en/papers/holtzman-2021-surface-form-competition-why-highest/)
- [Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification](https://md.research.elamur.ai/free/en/papers/hu-2022-knowledgeable-prompt-tuning-incorporating-knowledge/)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://md.research.elamur.ai/free/en/papers/huang-2022-language-models-zero-shot-planners/)
- [Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models](https://md.research.elamur.ai/free/en/papers/iv-2022-cutting-down-prompts-parameters-simple/)
- [ChatGPT: Jack of all trades, master of none](https://md.research.elamur.ai/free/en/papers/kocon-2023-chatgpt-jack-all-trades-master/)
- [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)
- [Pseudo Zero Pronoun Resolution Improves Zero Anaphora Resolution](https://md.research.elamur.ai/free/en/papers/konno-2021-pseudo-zero-pronoun-resolution-improves/)
- [Few-Shot Anaphora Resolution in Scientific Protocols via Mixtures of In-Context Experts](https://md.research.elamur.ai/free/en/papers/le-2022-few-shot-anaphora-resolution-scientific/)
- [Few-shot Learning with Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
- [GPT Understands, Too](https://md.research.elamur.ai/free/en/papers/liu-2021-gpt-understands-too/)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://md.research.elamur.ai/free/en/papers/liu-2021-p-tuning-v2-prompt-tuning/)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://md.research.elamur.ai/free/en/papers/liu-2022-few-shot-parameter-efficient-fine/)
- [P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks](https://md.research.elamur.ai/free/en/papers/liu-2022-p-tuning-prompt-tuning-can/)
- [What Makes Good In-Context Examples for GPT-3?](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/)
- [Prompt Tuning with Contradictory Intentions for Sarcasm Recognition](https://md.research.elamur.ai/free/en/papers/liu-2023-prompt-tuning-contradictory-intentions-sarcasm/)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/)
- [Few-Shot Self-Rationalization with Natural Language Prompts](https://md.research.elamur.ai/free/en/papers/marasovic-2022-few-shot-self-rationalization-natural/)
- [COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining](https://md.research.elamur.ai/free/en/papers/meng-2021-coco-lm-correcting-contrasting-text/)
- [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://md.research.elamur.ai/free/en/papers/meng-2022-generating-training-data-language-models/)
- [Noisy Channel Language Model Prompting for Few-Shot Text Classification](https://md.research.elamur.ai/free/en/papers/min-2021-noisy-channel-language-model-prompting/)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/)
- [KCIF: Knowledge-Conditioned Instruction Following](https://md.research.elamur.ai/free/en/papers/murthy-2026-kcif-knowledge-conditioned-instruction-following/)
- [True Few-Shot Learning with Language Models](https://md.research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/)
- [GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models](https://md.research.elamur.ai/free/en/papers/prasad-2023-grips-gradient-free-edit-based/)
- [Automatic Prompt Optimization with “Gradient Descent” and Beam Search](https://md.research.elamur.ai/free/en/papers/pryzant-2023-automatic-prompt-optimization-gradient-descent/)
- [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/)
- [Learning Transferable Visual Models From Natural Language Supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://md.research.elamur.ai/free/en/papers/reynolds-2021-prompt-programming-large-language-models/)
- [Generating Datasets with Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/schick-2021-generating-datasets-pretrained-language-models/)
- [The Prompt Report: A Systematic Survey of Prompt Engineering Techniques](https://md.research.elamur.ai/free/en/papers/schulhoff-2024-prompt-report-systematic-survey-prompt/)
- [On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning](https://md.research.elamur.ai/free/en/papers/shaikh-2023-second-thought-let-s-not/)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/)
- [TyDiP: A Dataset for Politeness Classification in Nine Typologically Diverse Languages](https://md.research.elamur.ai/free/en/papers/srinivasan-2022-tydip-dataset-politeness-classification-nine/)
- [Large Language Models for Data Annotation and Synthesis: A Survey](https://md.research.elamur.ai/free/en/papers/tan-2024-large-language-models-data-annotation/)
- [MaskedVerbalizer: Automatic Verbalizer Construction for Few-Shot Text Classification in Low-Resource Right-to-Left Languages](https://md.research.elamur.ai/free/en/papers/ullah-2026-maskedverbalizer-automatic-verbalizer-construction-few/)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://md.research.elamur.ai/free/en/papers/vu-2022-spot-better-frozen-model-adaptation/)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/)
- [Emergent Abilities of Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://md.research.elamur.ai/free/en/papers/xie-2021-explanation-context-learning-implicit-bayesian/)
- [UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models](https://md.research.elamur.ai/free/en/papers/xie-2022-unifiedskg-unifying-multi-tasking-structured/)
- [Large Language Models as Optimizers](https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-optimizers/)
- [LinkBERT: Pretraining Language Models with Document Links](https://md.research.elamur.ai/free/en/papers/yasunaga-2022-linkbert-pretraining-language-models-document/)
- [Active Example Selection for In-Context Learning](https://md.research.elamur.ai/free/en/papers/zhang-2022-active-example-selection-context-learning/)
- [Prompting Large Language Model for Machine Translation: A Case Study](https://md.research.elamur.ai/free/en/papers/zhang-2023-prompting-large-language-model-machine/)
- [Supervised Contrastive Fine-Tuning for Active Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/zhang-2026-supervised-contrastive-fine-tuning-active/)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)
- [Factual Probing Is MASK: Learning vs. Learning to Recall](https://md.research.elamur.ai/free/en/papers/zhong-2021-factual-probing-is-mask-learning/)
- [Large Language Models Are Human-Level Prompt Engineers](https://md.research.elamur.ai/free/en/papers/zhou-2022-large-language-models-human-level/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{gao2021making,
  title = {{Making Pre-trained Language Models Better Few-shot Learners}},
  author = {Gao, Tianyu and Fisch, Adam and Chen, Danqi},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.295},
  eprint = {2012.15723},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.acl-long.295}
}
```

- [← previous Maintaining Common Ground in Dynamic Environments](https://md.research.elamur.ai/free/en/papers/udagawa-2021-maintaining-common-ground-dynamic-environments/)
- [next → Manually Annotated Spelling Error Corpus for Amharic](https://md.research.elamur.ai/free/en/papers/gezmu-2021-manually-annotated-spelling-error-corpus/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

50 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Trapit Bansal, Rishikesh Jha, and Andrew McCallum. 2020a. Learning to few-shot learn across diverse natural language classification tasks. In International Conference on Computational Linguistics (COLING). [find](https://scholar.google.com/scholar?q=Trapit+Bansal%2C+Rishikesh+Jha%2C+and+Andrew+McCallum%2E+2020a%2E+Learning+to+few%2Dshot+learn+across+diverse+natural+language+classification+tasks%2E+In+International+Conference+on+Computational+Linguistics+%28COL)
2. Trapit Bansal, Rishikesh Jha, Tsendsuren Munkhdalai, and Andrew McCallum. 2020b. Self-supervised meta-learning for few-shot natural language classification tasks. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Trapit+Bansal%2C+Rishikesh+Jha%2C+Tsendsuren+Munkhdalai%2C+and+Andrew+McCallum%2E+2020b%2E+Self%2Dsupervised+meta%2Dlearning+for+few%2Dshot+natural+language+classification+tasks%2E+In+Empirical+Methods+in+Natural+Langu)
3. Yujia Bao, Menghua Wu, Shiyu Chang, and Regina Barzilay. 2020. Few-shot text classification with distributional signatures. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Yujia+Bao%2C+Menghua+Wu%2C+Shiyu+Chang%2C+and+Regina+Barzilay%2E+2020%2E+Few%2Dshot+text+classification+with+distributional+signatures%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
4. Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second PASCAL recognising textual entailment challenge. [find](https://scholar.google.com/scholar?q=Roy+Bar+Haim%2C+Ido+Dagan%2C+Bill+Dolan%2C+Lisa+Ferro%2C+Danilo+Giampiccolo%2C+Bernardo+Magnini%2C+and+Idan+Szpektor%2E+2006%2E+The+second+PASCAL+recognising+textual+entailment+challenge%2E)
5. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document Transformer. arXiv:2004.05150. [find](https://scholar.google.com/scholar?q=Iz+Beltagy%2C+Matthew+E%2E+Peters%2C+and+Arman+Cohan%2E+2020%2E+Longformer%3A+The+long%2Ddocument+Transformer%2E+arXiv%3A2004%2E05150%2E)
6. Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth PASCAL recognizing textual entailment challenge. In TAC. [find](https://scholar.google.com/scholar?q=Luisa+Bentivogli%2C+Peter+Clark%2C+Ido+Dagan%2C+and+Danilo+Giampiccolo%2E+2009%2E+The+fifth+PASCAL+recognizing+textual+entailment+challenge%2E+In+TAC%2E)
7. Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Samuel+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28)
8. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS). [find](https://scholar.google.com/scholar?q=Tom+B+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+2020%2E+Language+models+are+few%2Dshot+lear)
9. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In the 11th International Workshop on Semantic Evaluation (SemEval-2017). [find](https://scholar.google.com/scholar?q=Daniel+Cer%2C+Mona+Diab%2C+Eneko+Agirre%2C+In%CC%83igo+Lopez%2DGazpio%2C+and+Lucia+Specia%2E+2017%2E+SemEval%2D2017+task+1%3A+Semantic+textual+similarity+multilingual+and+crosslingual+focused+evaluation%2E+In+the+11th+Interna)
10. Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. Mix-Text: Linguistically-informed interpolation of hidden space for semi-supervised text classification. In Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Jiaao+Chen%2C+Zichao+Yang%2C+and+Diyi+Yang%2E+2020%2E+Mix%2DText%3A+Linguistically%2Dinformed+interpolation+of+hidden+space+for+semi%2Dsupervised+text+classification%2E+In+Association+for+Computational+Linguistics+%28ACL)
11. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2005%2E+The+PASCAL+recognising+textual+entailment+challenge%2E+In+the+First+International+Conference+on+Machine+Learning+Challenges%3A+Evaluating+Predictive+U)
12. Joe Davison, Joshua Feldman, and Alexander M Rush. 2019. Commonsense knowledge mining from pre-trained models. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Joe+Davison%2C+Joshua+Feldman%2C+and+Alexander+M+Rush%2E+2019%2E+Commonsense+knowledge+mining+from+pre%2Dtrained+models%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E)
13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional Transformers for language understanding. In North American Chapter of the Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+Transformers+for+language+understanding%2E+In+North+American+Chapter+of+the+Association+f)
14. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305. [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Gabriel+Ilharco%2C+Roy+Schwartz%2C+Ali+Farhadi%2C+Hannaneh+Hajishirzi%2C+and+Noah+Smith%2E+2020%2E+Fine%2Dtuning+pretrained+language+models%3A+Weight+initializations%2C+data+orders%2C+and+early+stopping%2E+arX)
15. William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In the Third International Workshop on Paraphrasing (IWP2005). [find](https://scholar.google.com/scholar?q=William+B%2E+Dolan+and+Chris+Brockett%2E+2005%2E+Automatically+constructing+a+corpus+of+sentential+paraphrases%2E+In+the+Third+International+Workshop+on+Paraphrasing+%28IWP2005%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/)
16. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. [find](https://scholar.google.com/scholar?q=Danilo+Giampiccolo%2C+Bernardo+Magnini%2C+Ido+Dagan%2C+and+Bill+Dolan%2E+2007%2E+The+third+PASCAL+recognizing+textual+entailment+challenge%2E+In+the+ACL%2DPASCAL+Workshop+on+Textual+Entailment+and+Paraphrasing%2E)
17. Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Xu+Han%2C+Hao+Zhu%2C+Pengfei+Yu%2C+Ziyun+Wang%2C+Yuan+Yao%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2018%2E+Fewrel%3A+A+large%2Dscale+supervised+few%2Dshot+relation+classification+dataset+with+state%2Dof%2Dthe%2Dart+evaluation%2E+In+Em)
18. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2E)
19. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In ACM SIGKDD international conference on Knowledge discovery and data mining. [find](https://scholar.google.com/scholar?q=Minqing+Hu+and+Bing+Liu%2E+2004%2E+Mining+and+summarizing+customer+reviews%2E+In+ACM+SIGKDD+international+conference+on+Knowledge+discovery+and+data+mining%2E)
20. Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association of Computational Linguistics (TACL). [find](https://scholar.google.com/scholar?q=Zhengbao+Jiang%2C+Frank+F+Xu%2C+Jun+Araki%2C+and+Graham+Neubig%2E+2020%2E+How+can+we+know+what+language+models+know%3F+Transactions+of+the+Association+of+Computational+Linguistics+%28TACL%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/)
21. Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. 2020. Mixout: Effective regularization to finetune large-scale pretrained language models. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Cheolhyoung+Lee%2C+Kyunghyun+Cho%2C+and+Wanmo+Kang%2E+2020%2E+Mixout%3A+Effective+regularization+to+finetune+large%2Dscale+pretrained+language+models%2E+In+International+Conference+on+Learning+Representations+%28ICLR)
22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+a)
23. Pascal Mettes, Elise van der Pol, and Cees Snoek. 2019. Hyperspherical prototype networks. In Advances in Neural Information Processing Systems (NeurIPS). [find](https://scholar.google.com/scholar?q=Pascal+Mettes%2C+Elise+van+der+Pol%2C+and+Cees+Snoek%2E+2019%2E+Hyperspherical+prototype+networks%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2E)
24. Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2017. Adversarial training methods for semi-supervised text classification. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Takeru+Miyato%2C+Andrew+M+Dai%2C+and+Ian+Goodfellow%2E+2017%2E+Adversarial+training+methods+for+semi%2Dsupervised+text+classification%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/miyato-2016-adversarial-training-methods-semi-supervised/)
25. Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Bo+Pang+and+Lillian+Lee%2E+2004%2E+A+sentimental+education%3A+Sentiment+analysis+using+subjectivity+summarization+based+on+minimum+cuts%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/pang-2004-sentimental-education-sentiment-analysis-subjectivity/)
26. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Bo+Pang+and+Lillian+Lee%2E+2005%2E+Seeing+stars%3A+Exploiting+class+relationships+for+sentiment+categorization+with+respect+to+rating+scales%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/pang-2005-seeing-stars-exploiting-class-relationships/)
27. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Fabio+Petroni%2C+Tim+Rockta%CC%88schel%2C+Sebastian+Riedel%2C+Patrick+Lewis%2C+Anton+Bakhtin%2C+Yuxiang+Wu%2C+and+Alexander+Miller%2E+2019%2E+Language+models+as+knowledge+bases%3F+In+Empirical+Methods+in+Natural+Language+Pr) [our card](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/)
28. Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on STILTs: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088. [find](https://scholar.google.com/scholar?q=Jason+Phang%2C+Thibault+Fe%CC%81vry%2C+and+Samuel+R+Bowman%2E+2018%2E+Sentence+encoders+on+STILTs%3A+Supplementary+training+on+intermediate+labeled%2Ddata+tasks%2E+arXiv+preprint+arXiv%3A1811%2E01088%2E) [our card](https://md.research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/)
29. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Technical report, OpenAI. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+Technical+report%2C+OpenAI%2E)
30. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, OpenAI. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+Technical+report%2C+OpenAI%2E)
31. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text Transformer. The Journal of Machine Learning Research (JMLR), 21(140). [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J+Liu%2E+2020%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext)
32. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
33. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). [find](https://scholar.google.com/scholar?q=Nils+Reimers+and+Iryna+Gurevych%2E+2019%2E+Sentence%2DBERT%3A+Sentence+embeddings+using+Siamese+BERT%2Dnetworks%2E+In+Empirical+Methods+in+Natural+Language+Processing+and+International+Joint+Conference+on+Natural)
34. Timo Schick, Helmut Schmid, and Hinrich Schütze. 2020. Automatically identifying words that can serve as labels for few-shot text classification. In International Conference on Computational Linguistics (COLING). [find](https://scholar.google.com/scholar?q=Timo+Schick%2C+Helmut+Schmid%2C+and+Hinrich+Schu%CC%88tze%2E+2020%2E+Automatically+identifying+words+that+can+serve+as+labels+for+few%2Dshot+text+classification%2E+In+International+Conference+on+Computational+Linguist)
35. Timo Schick and Hinrich Schütze. 2021a. Exploiting cloze questions for few-shot text classification and natural language inference. In European Chapter of the Association for Computational Linguistics (EACL). [find](https://scholar.google.com/scholar?q=Timo+Schick+and+Hinrich+Schu%CC%88tze%2E+2021a%2E+Exploiting+cloze+questions+for+few%2Dshot+text+classification+and+natural+language+inference%2E+In+European+Chapter+of+the+Association+for+Computational+Linguistic) [our card](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/)
36. Timo Schick and Hinrich Schütze. 2021b. It’s not just size that matters: Small language models are also few-shot learners. In North American Chapter of the Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=Timo+Schick+and+Hinrich+Schu%CC%88tze%2E+2021b%2E+It%E2%80%99s+not+just+size+that+matters%3A+Small+language+models+are+also+few%2Dshot+learners%2E+In+North+American+Chapter+of+the+Association+for+Computational+Linguistics+%28) [our card](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/)
37. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Automatic prompt construction for masked language models. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Taylor+Shin%2C+Yasaman+Razeghi%2C+Robert+L%2E+Logan+IV%2C+Eric+Wallace%2C+and+Sameer+Singh%2E+2020%2E+AutoPrompt%3A+Automatic+prompt+construction+for+masked+language+models%2E+In+Empirical+Methods+in+Natural+Language+P)
38. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D%2E+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In)
39. Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020. oLMpics-on what language model pre-training captures. Transactions of the Association of Computational Linguistics (TACL), 8. [find](https://scholar.google.com/scholar?q=Alon+Talmor%2C+Yanai+Elazar%2C+Yoav+Goldberg%2C+and+Jonathan+Berant%2E+2020%2E+oLMpics%2Don+what+language+model+pre%2Dtraining+captures%2E+Transactions+of+the+Association+of+Computational+Linguistics+%28TACL%29%2C+8%2E) [our card](https://md.research.elamur.ai/free/en/papers/talmor-2019-olmpics-language-model-pre-training/)
40. Trieu H Trinh and Quoc V Le. 2018. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847. [find](https://scholar.google.com/scholar?q=Trieu+H+Trinh+and+Quoc+V+Le%2E+2018%2E+A+simple+method+for+commonsense+reasoning%2E+arXiv+preprint+arXiv%3A1806%2E02847%2E)
41. Ellen M Voorhees and Dawn M Tice. 2000. Building a question answering test collection. In the 23rd annual international ACM SIGIR conference on Research and development in information retrieval. [find](https://scholar.google.com/scholar?q=Ellen+M+Voorhees+and+Dawn+M+Tice%2E+2000%2E+Building+a+question+answering+test+collection%2E+In+the+23rd+annual+international+ACM+SIGIR+conference+on+Research+and+development+in+information+retrieval%2E)
42. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R+Bowman%2E+2019%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+International+Co) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
43. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. Transactions of the Association of Computational Linguistics (TACL), 7. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R%2E+Bowman%2E+2019%2E+Neural+network+acceptability+judgments%2E+Transactions+of+the+Association+of+Computational+Linguistics+%28TACL%29%2C+7%2E)
44. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2-3). [find](https://scholar.google.com/scholar?q=Janyce+Wiebe%2C+Theresa+Wilson%2C+and+Claire+Cardie%2E+2005%2E+Annotating+expressions+of+opinions+and+emotions+in+language%2E+Language+resources+and+evaluation%2C+39%282%2D3%29%2E)
45. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+North+American+Chapter+of+the+Association+for+Computational+L)
46. Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. 2020. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems (NeurIPS), 33. [find](https://scholar.google.com/scholar?q=Qizhe+Xie%2C+Zihang+Dai%2C+Eduard+Hovy%2C+Thang+Luong%2C+and+Quoc+Le%2E+2020%2E+Unsupervised+data+augmentation+for+consistency+training%2E+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2C+33%2E) [our card](https://md.research.elamur.ai/free/en/papers/xie-2019-unsupervised-data-augmentation-consistency-training/)
47. Wenpeng Yin, Nazneen Fatema Rajani, Dragomir Radev, Richard Socher, and Caiming Xiong. 2020. Universal natural language processing with limited annotations: Try few-shot textual entailment as a start. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Wenpeng+Yin%2C+Nazneen+Fatema+Rajani%2C+Dragomir+Radev%2C+Richard+Socher%2C+and+Caiming+Xiong%2E+2020%2E+Universal+natural+language+processing+with+limited+annotations%3A+Try+few%2Dshot+textual+entailment+as+a+start%2E)
48. Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. 2018. Diverse few-shot text classification with multiple metrics. In North American Chapter of the Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=Mo+Yu%2C+Xiaoxiao+Guo%2C+Jinfeng+Yi%2C+Shiyu+Chang%2C+Saloni+Potdar%2C+Yu+Cheng%2C+Gerald+Tesauro%2C+Haoyu+Wang%2C+and+Bowen+Zhou%2E+2018%2E+Diverse+few%2Dshot+text+classification+with+multiple+metrics%2E+In+North+American+C)
49. Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. 2021. Revisiting few-sample BERT fine-tuning. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Felix+Wu%2C+Arzoo+Katiyar%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2021%2E+Revisiting+few%2Dsample+BERT+fine%2Dtuning%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2020-revisiting-few-sample-bert-fine/)
50. Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [MASK]: Learning vs. learning to recall. In North American Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=Zexuan+Zhong%2C+Dan+Friedman%2C+and+Danqi+Chen%2E+2021%2E+Factual+probing+is+%5BMASK%5D%3A+Learning+vs%2E+learning+to+recall%2E+In+North+American+Association+for+Computational+Linguistics+%28NAACL%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhong-2021-factual-probing-is-mask-learning/)