---
title: "Calibrate Before Use — Zhao 2021"
description: "Few-shot in-context learning lets GPT-3 handle a task from a handful of examples in the prompt, with no weight updates."
authors: ["Tony Z. Zhao", "Eric Wallace", "Shi Feng", "Dan Klein", "Sameer Singh"]
year: 2021
venue: "arXiv:2102.09690"
source: "https://arxiv.org/abs/2102.09690"
doi: "10.48550/arxiv.2102.09690"
arxiv: "2102.09690"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/) / Calibrate Before Use: Improving Few-Shot Performance of Language Models

# Calibrate Before Use: Improving Few-Shot Performance of Language Models

**Authors:** Tony Z. Zhao, [Eric Wallace](https://md.research.elamur.ai/free/en/authors/eric-wallace/), [Shi Feng](https://md.research.elamur.ai/free/en/authors/shi-feng/), [Dan Klein](https://md.research.elamur.ai/free/en/authors/dan-klein/), [Sameer Singh](https://md.research.elamur.ai/free/en/authors/sameer-singh/) · **Year:** 2021 · **Venue:** arXiv:2102.09690 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2102.09690) · [PDF](https://arxiv.org/pdf/2102.09690) · [DOI 10.48550/arxiv.2102.09690](https://doi.org/10.48550/arxiv.2102.09690) · [arXiv 2102.09690](https://arxiv.org/abs/2102.09690) · [OpenAlex](https://openalex.org/W3132736064)

**Citations:** 69 citations (openalex-snapshot · 2026-10-14) · [721](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Few-shot in-context learning lets GPT-3 handle a task from a handful of examples in the prompt, with no weight updates. The authors show this is fragile: accuracy swings widely depending on the prompt format, which examples are chosen, and even their order. They trace the problem to systematic answer preferences: recency bias, majority label bias and common token bias. These shift the output distribution toward certain labels. Their fix, contextual calibration, pairs the prompt with a dummy, content-free input like "N/A". The model's skewed prediction on that input serves as an estimate of its bias, and calibration parameters are then fitted so the dummy receives uniform scores over all answers. No additional labeled data is required.

Evaluation covers text classification (SST-2, TREC, CB, RTE, AGNews, DBPedia), fact retrieval (LAMA) and slot filling (ATIS, MIT Movies trivia10k13). The models are GPT-3 2.7B, 13B and 175B, plus GPT-2 1.5B. Calibration raises average accuracy by as much as 30.0 points absolute and makes results far less sensitive to prompt choices, which reduces the need for prompt engineering.

## Key points
- On 4-shot SST-2 with GPT-3 2.7B, merely reordering the same training examples moved accuracy from 54.3% (near chance) to 93.4% (near state of the art).
- More shots or bigger models do not remove the instability. Variance stays high at 16 examples and for 175B. On DBPedia, mean accuracy even dropped from 36.0% (0-shot) to 25.9% (1-shot).
- Bias example: putting "N/A" in place of the test review in a two-shot sentiment prompt still yields 62% Positive.
- Contextual calibration needs no training data, yet on AGNews with GPT-3 175B it performs about as well as an "oracle" calibration whose matrix W is tuned on the validation set.
- Metrics are accuracy for classification and LAMA, and Exact Match for ATIS/MIT Movies slots. Code is released at github.com/tonyzhaozh/few-shot-learning.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

721 papers of our library cite it (of the 140673 whose bibliographies we read); 73 of them have a page here, 648 not yet.

- [MEGA: Multilingual Evaluation of Generative AI](https://md.research.elamur.ai/free/en/papers/ahuja-2023-mega-multilingual-evaluation-generative-ai/)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
- [MGKM at StanceEval2024 Fine-Tuning Large Language Models for Arabic Stance Detection](https://md.research.elamur.ai/free/en/papers/alghaslan-2024-mgkm-stanceeval2024-fine-tuning-large/)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://md.research.elamur.ai/free/en/papers/bach-2022-promptsource-integrated-development-environment-repository/)
- [Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering](https://md.research.elamur.ai/free/en/papers/baek-2023-knowledge-augmented-language-model-prompting/)
- [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
- [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
- [Adapting Language Models to Compress Contexts](https://md.research.elamur.ai/free/en/papers/chevalier-2023-adapting-language-models-compress-contexts/)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deng-2022-rlprompt-optimizing-discrete-text-prompts/)
- [OpenPrompt: An Open-source Framework for Prompt-learning](https://md.research.elamur.ai/free/en/papers/ding-2022-openprompt-open-source-framework-prompt/)
- [A Survey on In-context Learning](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/)
- [Complexity-Based Prompting for Multi-Step Reasoning](https://md.research.elamur.ai/free/en/papers/fu-2022-complexity-based-prompting-multi-step/)
- [Bias and Fairness in Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gallegos-2023-bias-fairness-large-language-models/)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://md.research.elamur.ai/free/en/papers/holtzman-2021-surface-form-competition-why-highest/)
- [Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting](https://md.research.elamur.ai/free/en/papers/huang-2023-all-languages-created-equal-llms/)
- [Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models](https://md.research.elamur.ai/free/en/papers/iv-2022-cutting-down-prompts-parameters-simple/)
- [Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations](https://md.research.elamur.ai/free/en/papers/jung-2022-maieutic-prompting-logically-consistent-reasoning/)
- [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/)
- [Few-Shot Anaphora Resolution in Scientific Protocols via Mixtures of In-Context Experts](https://md.research.elamur.ai/free/en/papers/le-2022-few-shot-anaphora-resolution-scientific/)
- [Making Language Models Better Reasoners with Step-Aware Verifier](https://md.research.elamur.ai/free/en/papers/li-2023-making-language-models-better-reasoners/)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://md.research.elamur.ai/free/en/papers/li-2024-llms-judges-comprehensive-survey-llm/)
- [Few-shot Learning with Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
- [GPT Understands, Too](https://md.research.elamur.ai/free/en/papers/liu-2021-gpt-understands-too/)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://md.research.elamur.ai/free/en/papers/liu-2022-few-shot-parameter-efficient-fine/)
- [What Makes Good In-Context Examples for GPT-3?](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/)
- [Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning](https://md.research.elamur.ai/free/en/papers/lu-2022-dynamic-prompt-learning-policy-gradient/)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/)
- [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://md.research.elamur.ai/free/en/papers/meng-2022-generating-training-data-language-models/)
- [Discursive Socratic Questioning: Evaluating the Faithfulness of Language Models' Understanding of Discourse Relations](https://md.research.elamur.ai/free/en/papers/miao-2024-discursive-socratic-questioning-evaluating-faithfulness/)
- [Noisy Channel Language Model Prompting for Few-Shot Text Classification](https://md.research.elamur.ai/free/en/papers/min-2021-noisy-channel-language-model-prompting/)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/)
- [Reframing Instructional Prompts to GPTk's Language](https://md.research.elamur.ai/free/en/papers/mishra-2022-reframing-instructional-prompts-gptk-s/)
- [Are the LLMs Capable of Maintaining at Least the Language Genus?](https://md.research.elamur.ai/free/en/papers/mitrovic-2026-are-llms-capable-maintaining-least/)
- [True Few-Shot Learning with Language Models](https://md.research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/)
- [Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions](https://md.research.elamur.ai/free/en/papers/pezeshkpour-2023-large-language-models-sensitivity-order/)
- [GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models](https://md.research.elamur.ai/free/en/papers/prasad-2023-grips-gradient-free-edit-based/)
- [Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning](https://md.research.elamur.ai/free/en/papers/razeghi-2022-impact-pretraining-term-frequencies-few/)
- [Leveraging Large Language Models for Multiple Choice Question Answering](https://md.research.elamur.ai/free/en/papers/robinson-2022-leveraging-large-language-models-multiple/)
- [Learning To Retrieve Prompts for In-Context Learning](https://md.research.elamur.ai/free/en/papers/rubin-2021-learning-retrieve-prompts-context-learning/)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
- [The Prompt Report: A Systematic Survey of Prompt Engineering Techniques](https://md.research.elamur.ai/free/en/papers/schulhoff-2024-prompt-report-systematic-survey-prompt/)
- [On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning](https://md.research.elamur.ai/free/en/papers/shaikh-2023-second-thought-let-s-not/)
- [mGPT: Few-Shot Learners Go Multilingual](https://md.research.elamur.ai/free/en/papers/shliazhko-2022-mgpt-few-shot-learners-go/)
- [Prompting GPT-3 To Be Reliable](https://md.research.elamur.ai/free/en/papers/si-2022-prompting-gpt-3-reliable/)
- [Prompting Instruction-tuned LLMs for Semantic Similarity Values](https://md.research.elamur.ai/free/en/papers/snelder-2026-prompting-instruction-tuned-llms-semantic/)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://md.research.elamur.ai/free/en/papers/vu-2022-spot-better-frozen-model-adaptation/)
- [From Latents to Labels: Zero-Shot Named Entity Recognition using Sparse Autoencoder Features](https://md.research.elamur.ai/free/en/papers/vuth-2026-latents-labels-zero-shot-named/)
- [GPT-RE: In-context Learning for Relation Extraction using Large Language Models](https://md.research.elamur.ai/free/en/papers/wan-2023-gpt-re-context-learning-relation/)
- [Want To Reduce Labeling Cost? GPT-3 Can Help](https://md.research.elamur.ai/free/en/papers/wang-2021-want-reduce-labeling-cost-gpt/)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/)
- [Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning](https://md.research.elamur.ai/free/en/papers/wang-2023-label-words-anchors-information-flow/)
- [GPT-NER: Named Entity Recognition via Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2025-gpt-ner-named-entity-recognition/)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
- [Emergent Abilities of Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/)
- [Larger language models do in-context learning differently](https://md.research.elamur.ai/free/en/papers/wei-2023-larger-language-models-context-learning/)
- [Ethical and social risks of harm from Language Models](https://md.research.elamur.ai/free/en/papers/weidinger-2021-ethical-social-risks-harm-language/)
- [A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions](https://md.research.elamur.ai/free/en/papers/wu-2025-survey-llm-generated-text-detection/)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://md.research.elamur.ai/free/en/papers/xie-2021-explanation-context-learning-implicit-bayesian/)
- [基于大语言模型的中文实体链接实证研究](https://md.research.elamur.ai/free/en/papers/xu-2025-empirical-study-chinese/)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://md.research.elamur.ai/free/en/papers/yang-2023-harnessing-power-llms-practice-survey/)
- [Large Language Models as Optimizers](https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-optimizers/)
- [The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning](https://md.research.elamur.ai/free/en/papers/ye-2022-unreliability-explanations-few-shot-prompting/)
- [GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation](https://md.research.elamur.ai/free/en/papers/yoo-2021-gpt3mix-leveraging-large-scale-language/)
- [Active Example Selection for In-Context Learning](https://md.research.elamur.ai/free/en/papers/zhang-2022-active-example-selection-context-learning/)
- [Instruction Tuning for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/zhang-2023-instruction-tuning-large-language-models/)
- [Prompting Large Language Model for Machine Translation: A Case Study](https://md.research.elamur.ai/free/en/papers/zhang-2023-prompting-large-language-model-machine/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)
- [LLMs as Annotators: Evaluating Model–Human Alignment in Detecting Contentious Language in Historical Corpora](https://md.research.elamur.ai/free/en/papers/zhao-2026-llms-as-annotators-evaluating-model/)
- [Can Large Language Models Transform Computational Social Science?](https://md.research.elamur.ai/free/en/papers/ziems-2023-large-language-models-transform-computational/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhao2021calibrate,
  title = {{Calibrate Before Use: Improving Few-Shot Performance of Language Models}},
  author = {Zhao, Tony Z. and Wallace, Eric and Feng, Shi and Klein, Dan and Singh, Sameer},
  year = {2021},
  doi = {10.48550/arxiv.2102.09690},
  eprint = {2102.09690},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2102.09690}
}
```

- [← previous Bridging Resolution: Making Sense of the State of the Art](https://md.research.elamur.ai/free/en/papers/kobayashi-2021-bridging-resolution-making-sense-state/)
- [next → Calliar: An Online Handwritten Dataset for Arabic Calligraphy](https://md.research.elamur.ai/free/en/papers/alyafeai-2021-calliar-online-handwritten-dataset-arabic/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/)

## References

34 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bosselut, A., Rashkin, H., Sap, M., Malaviya, C., Celikyilmaz, A., and Choi, Y. COMET: Commonsense transformers for automatic knowledge graph construction. In ACL, 2019. [find](https://scholar.google.com/scholar?q=Bosselut%2C+A%2E%2C+Rashkin%2C+H%2E%2C+Sap%2C+M%2E%2C+Malaviya%2C+C%2E%2C+Celikyilmaz%2C+A%2E%2C+and+Choi%2C+Y%2E+COMET%3A+Commonsense+transformers+for+automatic+knowledge+graph+construction%2E+In+ACL%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/bosselut-2019-comet-commonsense-transformers-automatic-knowledge/)
2. Braverman, M., Chen, X., Kakade, S., Narasimhan, K., Zhang, C., and Zhang, Y. Calibration, entropy rates, and memory in language models. In ICML, 2020. [find](https://scholar.google.com/scholar?q=Braverman%2C+M%2E%2C+Chen%2C+X%2E%2C+Kakade%2C+S%2E%2C+Narasimhan%2C+K%2E%2C+Zhang%2C+C%2E%2C+and+Zhang%2C+Y%2E+Calibration%2C+entropy+rates%2C+and+memory+in+language+models%2E+In+ICML%2C+2020%2E)
3. Brier, G. W. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 1950. [find](https://scholar.google.com/scholar?q=Brier%2C+G%2E+W%2E+Verification+of+forecasts+expressed+in+terms+of+probability%2E+Monthly+Weather+Review%2C+1950%2E)
4. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Ch… [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E+B%2E%2C+Mann%2C+B%2E%2C+Ryder%2C+N%2E%2C+Subbiah%2C+M%2E%2C+Kaplan%2C+J%2E%2C+Dhariwal%2C+P%2E%2C+Neelakantan%2C+A%2E%2C+Shyam%2C+P%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+Agarwal%2C+S%2E%2C+Herbert%2DVoss%2C+A%2E%2C+Krueger%2C+G%2E%2C+Henighan%2C+T%2E%2C+Child%2C+R%2E%2C+Rames)
5. Dagan, I., Glickman, O., and Magnini, B. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges Workshop, 2005. de Marneffe, M.-C., Simons, M., and Tonhauser, J. The CommitmentBank: Investigating projection in naturally occurring discourse. In Sinn und Bedeutung, 2019. [find](https://scholar.google.com/scholar?q=Dagan%2C+I%2E%2C+Glickman%2C+O%2E%2C+and+Magnini%2C+B%2E+The+PASCAL+recognising+textual+entailment+challenge%2E+In+Machine+Learning+Challenges+Workshop%2C+2005%2E+de+Marneffe%2C+M%2E%2DC%2E%2C+Simons%2C+M%2E%2C+and+Tonhauser%2C+J%2E+The+Commi) [our card](https://md.research.elamur.ai/free/en/papers/enprag-2019-marneffe-commitmentbank-investigating-projection-naturally/)
6. Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019. [find](https://scholar.google.com/scholar?q=Devlin%2C+J%2E%2C+Chang%2C+M%2E%2C+Lee%2C+K%2E%2C+and+Toutanova%2C+K%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+NAACL%2C+2019%2E)
7. Gao, T., Fisch, A., and Chen, D. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723, 2020. [find](https://scholar.google.com/scholar?q=Gao%2C+T%2E%2C+Fisch%2C+A%2E%2C+and+Chen%2C+D%2E+Making+pre%2Dtrained+language+models+better+few%2Dshot+learners%2E+arXiv+preprint+arXiv%3A2012%2E15723%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/)
8. Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In ICML, 2017. [find](https://scholar.google.com/scholar?q=Guo%2C+C%2E%2C+Pleiss%2C+G%2E%2C+Sun%2C+Y%2E%2C+and+Weinberger%2C+K%2E+Q%2E+On+calibration+of+modern+neural+networks%2E+In+ICML%2C+2017%2E)
9. Hemphill, C. T., Godfrey, J. J., and Doddington, G. R. The ATIS spoken language systems pilot corpus. In Speech and Natural Language Workshop, 1990. [find](https://scholar.google.com/scholar?q=Hemphill%2C+C%2E+T%2E%2C+Godfrey%2C+J%2E+J%2E%2C+and+Doddington%2C+G%2E+R%2E+The+ATIS+spoken+language+systems+pilot+corpus%2E+In+Speech+and+Natural+Language+Workshop%2C+1990%2E)
10. Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. In ICLR, 2020. [find](https://scholar.google.com/scholar?q=Holtzman%2C+A%2E%2C+Buys%2C+J%2E%2C+Du%2C+L%2E%2C+Forbes%2C+M%2E%2C+and+Choi%2C+Y%2E+The+curious+case+of+neural+text+degeneration%2E+In+ICLR%2C+2020%2E)
11. Jiang, Z., Araki, J., Ding, H., and Neubig, G. How can we know when language models know? arXiv preprint arXiv:2012.00955, 2020a. [find](https://scholar.google.com/scholar?q=Jiang%2C+Z%2E%2C+Araki%2C+J%2E%2C+Ding%2C+H%2E%2C+and+Neubig%2C+G%2E+How+can+we+know+when+language+models+know%3F+arXiv+preprint+arXiv%3A2012%2E00955%2C+2020a%2E)
12. Jiang, Z., Xu, F. F., Araki, J., and Neubig, G. How can we know what language models know? In TACL, 2020b. [find](https://scholar.google.com/scholar?q=Jiang%2C+Z%2E%2C+Xu%2C+F%2E+F%2E%2C+Araki%2C+J%2E%2C+and+Neubig%2C+G%2E+How+can+we+know+what+language+models+know%3F+In+TACL%2C+2020b%2E) [our card](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/)
13. Khandelwal, U., He, H., Qi, P., and Jurafsky, D. Sharp nearby, fuzzy far away: How neural language models use context. In ACL, 2018. [find](https://scholar.google.com/scholar?q=Khandelwal%2C+U%2E%2C+He%2C+H%2E%2C+Qi%2C+P%2E%2C+and+Jurafsky%2C+D%2E+Sharp+nearby%2C+fuzzy+far+away%3A+How+neural+language+models+use+context%2E+In+ACL%2C+2018%2E)
14. Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B. Human-level concept learning through probabilistic program induction. In Science, 2015. [find](https://scholar.google.com/scholar?q=Lake%2C+B%2E+M%2E%2C+Salakhutdinov%2C+R%2E%2C+and+Tenenbaum%2C+J%2E+B%2E+Human%2Dlevel+concept+learning+through+probabilistic+program+induction%2E+In+Science%2C+2015%2E)
15. Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, B. A diversity-promoting objective function for neural conversation models. In NAACL, 2016. [find](https://scholar.google.com/scholar?q=Li%2C+J%2E%2C+Galley%2C+M%2E%2C+Brockett%2C+C%2E%2C+Gao%2C+J%2E%2C+and+Dolan%2C+B%2E+A+diversity%2Dpromoting+objective+function+for+neural+conversation+models%2E+In+NAACL%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2016-diversity-promoting-objective-function-neural/)
16. Liu, J., Cyphers, S., Pasupat, P., McGraw, I., and Glass, J. A conversational movie search system based on conditional random fields. In INTERSPEECH, 2012. [find](https://scholar.google.com/scholar?q=Liu%2C+J%2E%2C+Cyphers%2C+S%2E%2C+Pasupat%2C+P%2E%2C+McGraw%2C+I%2E%2C+and+Glass%2C+J%2E+A+conversational+movie+search+system+based+on+conditional+random+fields%2E+In+INTERSPEECH%2C+2012%2E)
17. Logan, R. L., Liu, N. F., Peters, M. E., Gardner, M., and Singh, S. Barack’s wife Hillary: Using knowledge-graphs for fact-aware language modeling. In ACL, 2019. [find](https://scholar.google.com/scholar?q=Logan%2C+R%2E+L%2E%2C+Liu%2C+N%2E+F%2E%2C+Peters%2C+M%2E+E%2E%2C+Gardner%2C+M%2E%2C+and+Singh%2C+S%2E+Barack%E2%80%99s+wife+Hillary%3A+Using+knowledge%2Dgraphs+for+fact%2Daware+language+modeling%2E+In+ACL%2C+2019%2E)
18. Paulus, R., Xiong, C., and Socher, R. A deep reinforced model for abstractive summarization. In ICLR, 2018. [find](https://scholar.google.com/scholar?q=Paulus%2C+R%2E%2C+Xiong%2C+C%2E%2C+and+Socher%2C+R%2E+A+deep+reinforced+model+for+abstractive+summarization%2E+In+ICLR%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/paulus-2017-deep-reinforced-model-abstractive-summarization/)
19. Petroni, F., Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A. H., and Riedel, S. Language models as knowledge bases? In EMNLP, 2019. [find](https://scholar.google.com/scholar?q=Petroni%2C+F%2E%2C+Rockta%CC%88schel%2C+T%2E%2C+Lewis%2C+P%2E%2C+Bakhtin%2C+A%2E%2C+Wu%2C+Y%2E%2C+Miller%2C+A%2E+H%2E%2C+and+Riedel%2C+S%2E+Language+models+as+knowledge+bases%3F+In+EMNLP%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/)
20. Platt, J. C. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers, 1999. [find](https://scholar.google.com/scholar?q=Platt%2C+J%2E+C%2E+Probabilistic+outputs+for+support+vector+machines+and+comparisons+to+regularized+likelihood+methods%2E+In+Advances+in+Large+Margin+Classifiers%2C+1999%2E)
21. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training. Technical Report, 2018. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Narasimhan%2C+K%2E%2C+Salimans%2C+T%2E%2C+and+Sutskever%2C+I%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+Technical+Report%2C+2018%2E)
22. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. Technical Report, 2019. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+Child%2C+R%2E%2C+Luan%2C+D%2E%2C+Amodei%2C+D%2E%2C+and+Sutskever%2C+I%2E+Language+models+are+unsupervised+multitask+learners%2E+Technical+Report%2C+2019%2E)
23. Ravfogel, S., Goldberg, Y., and Linzen, T. Studying the inductive biases of RNNs with synthetic variations of natural languages. In NAACL, 2019. [find](https://scholar.google.com/scholar?q=Ravfogel%2C+S%2E%2C+Goldberg%2C+Y%2E%2C+and+Linzen%2C+T%2E+Studying+the+inductive+biases+of+RNNs+with+synthetic+variations+of+natural+languages%2E+In+NAACL%2C+2019%2E)
24. Schick, T. and Schütze, H. It’s not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.07118, 2020. [find](https://scholar.google.com/scholar?q=Schick%2C+T%2E+and+Schu%CC%88tze%2C+H%2E+It%E2%80%99s+not+just+size+that+matters%3A+Small+language+models+are+also+few%2Dshot+learners%2E+arXiv+preprint+arXiv%3A2009%2E07118%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/)
25. Schick, T. and Schütze, H. Exploiting cloze questions for few-shot text classification and natural language inference. In EACL, 2021. [find](https://scholar.google.com/scholar?q=Schick%2C+T%2E+and+Schu%CC%88tze%2C+H%2E+Exploiting+cloze+questions+for+few%2Dshot+text+classification+and+natural+language+inference%2E+In+EACL%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/)
26. Schwartz, R., Sap, M., Konstas, I., Zilles, L., Choi, Y., and Smith, N. A. The effect of different writing tasks on linguistic style: A case study of the ROC story cloze task. In ACL, 2017. [find](https://scholar.google.com/scholar?q=Schwartz%2C+R%2E%2C+Sap%2C+M%2E%2C+Konstas%2C+I%2E%2C+Zilles%2C+L%2E%2C+Choi%2C+Y%2E%2C+and+Smith%2C+N%2E+A%2E+The+effect+of+different+writing+tasks+on+linguistic+style%3A+A+case+study+of+the+ROC+story+cloze+task%2E+In+ACL%2C+2017%2E)
27. Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., and Singh, S. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. In EMNLP, 2020. [find](https://scholar.google.com/scholar?q=Shin%2C+T%2E%2C+Razeghi%2C+Y%2E%2C+Logan+IV%2C+R%2E+L%2E%2C+Wallace%2C+E%2E%2C+and+Singh%2C+S%2E+AutoPrompt%3A+Eliciting+knowledge+from+language+models+with+automatically+generated+prompts%2E+In+EMNLP%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/)
28. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013. [find](https://scholar.google.com/scholar?q=Socher%2C+R%2E%2C+Perelygin%2C+A%2E%2C+Wu%2C+J%2E%2C+Chuang%2C+J%2E%2C+Manning%2C+C%2E+D%2E%2C+Ng%2C+A%2E%2C+and+Potts%2C+C%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In+EMNLP%2C+2013%2E)
29. Trinh, T. H. and Le, Q. V. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847, 2018. [find](https://scholar.google.com/scholar?q=Trinh%2C+T%2E+H%2E+and+Le%2C+Q%2E+V%2E+A+simple+method+for+commonsense+reasoning%2E+arXiv+preprint+arXiv%3A1806%2E02847%2C+2018%2E)
30. Voorhees, E. M. and Tice, D. M. Building a question answering test collection. In SIGIR, 2000. [find](https://scholar.google.com/scholar?q=Voorhees%2C+E%2E+M%2E+and+Tice%2C+D%2E+M%2E+Building+a+question+answering+test+collection%2E+In+SIGIR%2C+2000%2E)
31. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In NeurIPS, 2019. [find](https://scholar.google.com/scholar?q=Wang%2C+A%2E%2C+Pruksachatkun%2C+Y%2E%2C+Nangia%2C+N%2E%2C+Singh%2C+A%2E%2C+Michael%2C+J%2E%2C+Hill%2C+F%2E%2C+Levy%2C+O%2E%2C+and+Bowman%2C+S%2E+SuperGLUE%3A+A+stickier+benchmark+for+general%2Dpurpose+language+understanding+systems%2E+In+NeurIPS%2C+2019) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
32. Yogatama, D., d’Autume, C. d. M., Connor, J., Kocisky, T., Chrzanowski, M., Kong, L., Lazaridou, A., Ling, W., Yu, L., Dyer, C., et al. Learning and evaluating general linguistic intelligence. arXiv preprint arXiv:1901.11373, 2019. [find](https://scholar.google.com/scholar?q=Yogatama%2C+D%2E%2C+d%E2%80%99Autume%2C+C%2E+d%2E+M%2E%2C+Connor%2C+J%2E%2C+Kocisky%2C+T%2E%2C+Chrzanowski%2C+M%2E%2C+Kong%2C+L%2E%2C+Lazaridou%2C+A%2E%2C+Ling%2C+W%2E%2C+Yu%2C+L%2E%2C+Dyer%2C+C%2E%2C+et+al%2E+Learning+and+evaluating+general+linguistic+intelligence%2E+arXiv+p)
33. Zhang, X., Zhao, J., and LeCun, Y. Character-level convolutional networks for text classification. In NeurIPS, 2015. [find](https://scholar.google.com/scholar?q=Zhang%2C+X%2E%2C+Zhao%2C+J%2E%2C+and+LeCun%2C+Y%2E+Character%2Dlevel+convolutional+networks+for+text+classification%2E+In+NeurIPS%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/)
34. A. Additional Results on Variance and Calibration Table 2 shows an example of the sensitivity to ordering. Prompt (test input not shown) Acc. Review: the whole thing ’s fairly lame , making it par for the course for disney sequels . Answer: Negative 88.5% Review: this quiet , introspective and enter… [find](https://scholar.google.com/scholar?q=A%2E+Additional+Results+on+Variance+and+Calibration+Table+2+shows+an+example+of+the+sensitivity+to+ordering%2E+Prompt+%28test+input+not+shown%29+Acc%2E+Review%3A+the+whole+thing+%E2%80%99s+fairly+lame+%2C+making+it+par+for)