---
title: "An Explanation of In-context Learning as Implicit — Xie 2021"
description: "The paper asks why large LMs such as GPT-3 can pick up a task from input-output examples in a prompt, even though they were never trained to learn from…"
authors: ["Sang Michael Xie", "Aditi Raghunathan", "Percy Liang", "Tengyu Ma"]
year: 2021
venue: "arXiv:2111.02080"
source: "https://arxiv.org/abs/2111.02080"
doi: "10.48550/arxiv.2111.02080"
arxiv: "2111.02080"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xie-2021-explanation-context-learning-implicit-bayesian/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xie-2021-explanation-context-learning-implicit-bayesian/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/) / An Explanation of In-context Learning as Implicit Bayesian Inference

# An Explanation of In-context Learning as Implicit Bayesian Inference

**Authors:** Sang Michael Xie, [Aditi Raghunathan](https://md.research.elamur.ai/free/en/authors/aditi-raghunathan/), [Percy Liang](https://md.research.elamur.ai/free/en/authors/percy-liang/), [Tengyu Ma](https://md.research.elamur.ai/free/en/authors/tengyu-ma/) · **Year:** 2021 · **Venue:** arXiv:2111.02080 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2111.02080) · [PDF](https://arxiv.org/pdf/2111.02080) · [DOI 10.48550/arxiv.2111.02080](https://doi.org/10.48550/arxiv.2111.02080) · [arXiv 2111.02080](https://arxiv.org/abs/2111.02080) · [OpenAlex](https://openalex.org/W3208828709)

**Citations:** 20 citations (openalex-snapshot · 2026-10-14) · [248](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The paper asks why large LMs such as GPT-3 can pick up a task from input-output examples in a prompt, even though they were never trained to learn from examples. The authors suggest that pretraining text has long-range coherence: to predict the next token well, an LM must implicitly infer a latent document-level "concept." At test time, in-context learning happens when the model infers the concept that the prompt examples share. In this view, prompting amounts to implicit Bayesian inference over concepts.

The authors formalize this with pretraining data drawn from a mixture of HMMs. They prove that in-context prediction becomes asymptotically optimal when each example carries more signal about the concept than the error introduced by the prompt/pretraining mismatch. To test the theory, they build GINC, a small synthetic dataset. Transformers and LSTMs trained on it learn in context and reproduce effects seen in large models. The work offers a tractable, small-scale testbed and a theoretical account that could guide better pretraining and prompting.

## Key points
- **Theory:** in-context error is shown to fall as more examples are added and as each example gets longer. This means the input tokens themselves, not only the input-to-label mapping, help identify the concept. The standard Bernstein-von Mises result does not apply here, because prompt examples are neither independent nor drawn from the pretraining distribution.
- **GINC experiments:** they compare Transformers (4, 12 and 16 layers; 29M, 85M and 115M parameters) and a 28M LSTM. With vocabulary size 50, in-context accuracy climbs from about 60% (4-layer) to about 85% (16-layer), even though the 12- and 16-layer models reach nearly identical pretraining loss (~1.31). The LSTM reaches about 96%.
- **Ablations:** removing the latent concept structure from the pretraining data prevents in-context learning from emerging.
- **Mirrored real-world quirks:**
  - Accuracy varies widely depending on the order of the examples.
  - Zero-shot can beat few-shot in some settings, such as low-entropy transitions (temperature 0.01, 12 concepts, vocab 100). This echoes GPT-3's behavior on LAMBADA.
- **Availability:** code and data are released on GitHub and CodaLab.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

248 papers of our library cite it (of the 140673 whose bibliographies we read); 23 of them have a page here, 225 not yet.

- [Many-Shot In-Context Learning](https://md.research.elamur.ai/free/en/papers/agarwal-2024-many-shot-context-learning/)
- [What learning algorithm is in-context learning? Investigations with linear models](https://md.research.elamur.ai/free/en/papers/akyurek-2022-learning-algorithm-context-learning-investigations/)
- [Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models](https://md.research.elamur.ai/free/en/papers/bohnet-2022-attributed-question-answering-evaluation-modeling/)
- [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
- [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2023-meditron-70b-scaling-medical-pretraining/)
- [A Survey on In-context Learning](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/)
- [Complexity-Based Prompting for Multi-Step Reasoning](https://md.research.elamur.ai/free/en/papers/fu-2022-complexity-based-prompting-multi-step/)
- [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/)
- [Can language models learn from explanations in context?](https://md.research.elamur.ai/free/en/papers/lampinen-2022-can-language-models-learn-explanations/)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/)
- [Reasoning with Language Model Prompting: A Survey](https://md.research.elamur.ai/free/en/papers/qiao-2022-reasoning-language-model-prompting-survey/)
- [Learning To Retrieve Prompts for In-Context Learning](https://md.research.elamur.ai/free/en/papers/rubin-2021-learning-retrieve-prompts-context-learning/)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://md.research.elamur.ai/free/en/papers/shen-2023-hugginggpt-solving-ai-tasks-chatgpt/)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/)
- [Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning](https://md.research.elamur.ai/free/en/papers/wang-2023-label-words-anchors-information-flow/)
- [Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters](https://md.research.elamur.ai/free/en/papers/wang-2023-understanding-chain-thought-prompting-empirical/)
- [Emergent Abilities of Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/)
- [Larger language models do in-context learning differently](https://md.research.elamur.ai/free/en/papers/wei-2023-larger-language-models-context-learning/)
- [AI Safety Lost in Translation: Evaluating the Effectiveness of English-Italian Cross-Lingual LLM Safety Alignment](https://md.research.elamur.ai/free/en/papers/wu-2026-ai-safety-lost-translation-evaluating/)
- [MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/)
- [STaR: Bootstrapping Reasoning With Reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/)
- [Active Example Selection for In-Context Learning](https://md.research.elamur.ai/free/en/papers/zhang-2022-active-example-selection-context-learning/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{xie2021explanation,
  title = {{An Explanation of In-context Learning as Implicit Bayesian Inference}},
  author = {Xie, Sang Michael and Raghunathan, Aditi and Liang, Percy and Ma, Tengyu},
  year = {2021},
  doi = {10.48550/arxiv.2111.02080},
  eprint = {2111.02080},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2111.02080}
}
```

- [← previous An Error Analysis Framework for Shallow Surface Realization](https://md.research.elamur.ai/free/en/papers/shimorina-2021-error-analysis-framework-shallow-surface/)
- [next → Anaphora Resolution in Dialogue: Description of the DFKI-TalkingRobots System for the CODI-CRAC 2021 Shared-Task](https://md.research.elamur.ai/free/en/papers/anikina-2021-anaphora-resolution-dialogue-description-dfki/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/)

## References

43 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Leonard E Baum and Ted Petrie. Statistical inference for probabilistic functions of finite state markov chains. The annals of mathematical statistics, 37(6):1554–1563, 1966. [find](https://scholar.google.com/scholar?q=Leonard+E+Baum+and+Ted+Petrie%2E+Statistical+inference+for+probabilistic+functions+of+finite+state+markov+chains%2E+The+annals+of+mathematical+statistics%2C+37%286%29%3A1554%E2%80%931563%2C+1966%2E)
2. D. Blei, Andrew Ng, and M. I. Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research (JMLR), 3:993–1022, 2003. [find](https://scholar.google.com/scholar?q=D%2E+Blei%2C+Andrew+Ng%2C+and+M%2E+I%2E+Jordan%2E+Latent+Dirichlet+allocation%2E+Journal+of+Machine+Learning+Research+%28JMLR%29%2C+3%3A993%E2%80%931022%2C+2003%2E) [our card](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/)
3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
4. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations (ICLR), 2020. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V%2E+Le%2C+and+Christopher+D%2E+Manning%2E+Electra%3A+Pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+In+International+Conference+on+Learning+Representat)
5. A. P. Dempster, Laird N. M., and Rubin D. B. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B, 39(1):1–38, 1977. [find](https://scholar.google.com/scholar?q=A%2E+P%2E+Dempster%2C+Laird+N%2E+M%2E%2C+and+Rubin+D%2E+B%2E+Maximum+likelihood+from+incomplete+data+via+the+EM+algorithm%2E+Journal+of+the+Royal+Statistical+Society%3A+Series+B%2C+39%281%29%3A1%E2%80%9338%2C+1977%2E)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Association for Computational Linguistics (ACL), pages 4171–4186, 2019. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2C+)
7. Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv, 2021. [find](https://scholar.google.com/scholar?q=Tianyu+Gao%2C+Adam+Fisch%2C+and+Danqi+Chen%2E+Making+pre%2Dtrained+language+models+better+few%2Dshot+learners%2E+arXiv%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/)
8. Zoubin Ghahramani and Michael Jordan. Factorial hidden Markov models. Machine Learning, 29: 245–273, 1997. [find](https://scholar.google.com/scholar?q=Zoubin+Ghahramani+and+Michael+Jordan%2E+Factorial+hidden+Markov+models%2E+Machine+Learning%2C+29%3A+245%E2%80%93273%2C+1997%2E)
9. Amit Gruber, Yair Weiss, and Michal Rosen-Zvi. Hidden topic Markov models. In Artificial Intelligence and Statistics (AISTATS), 2007. [find](https://scholar.google.com/scholar?q=Amit+Gruber%2C+Yair+Weiss%2C+and+Michal+Rosen%2DZvi%2E+Hidden+topic+Markov+models%2E+In+Artificial+Intelligence+and+Statistics+%28AISTATS%29%2C+2007%2E)
10. M. Gunst and O. Shcherbakova. Asymptotic behavior of Bayes estimators for hidden Markov models with application to ion channels. Mathematical Methods of Statistics, 17, 2008. [find](https://scholar.google.com/scholar?q=M%2E+Gunst+and+O%2E+Shcherbakova%2E+Asymptotic+behavior+of+Bayes+estimators+for+hidden+Markov+models+with+application+to+ion+channels%2E+Mathematical+Methods+of+Statistics%2C+17%2C+2008%2E)
11. Keith W. Hastings. Monte Carlo sampling methods using Markov chains and their applications. Biometrika, 57(1):97–109, 1970. [find](https://scholar.google.com/scholar?q=Keith+W%2E+Hastings%2E+Monte+Carlo+sampling+methods+using+Markov+chains+and+their+applications%2E+Biometrika%2C+57%281%29%3A97%E2%80%93109%2C+1970%2E)
12. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735–1780, 1997. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+J%C3%BCrgen+Schmidhuber%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2C+9%288%29%3A+1735%E2%80%931780%2C+1997%2E)
13. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations (ICLR), 2020. [find](https://scholar.google.com/scholar?q=Ari+Holtzman%2C+Jan+Buys%2C+Li+Du%2C+Maxwell+Forbes%2C+and+Yejin+Choi%2E+The+curious+case+of+neural+text+degeneration%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2C+2020%2E)
14. Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form competition: Why the highest probability answer isn’t always right, 2021. [find](https://scholar.google.com/scholar?q=Ari+Holtzman%2C+Peter+West%2C+Vered+Shwartz%2C+Yejin+Choi%2C+and+Luke+Zettlemoyer%2E+Surface+form+competition%3A+Why+the+highest+probability+answer+isn%E2%80%99t+always+right%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/holtzman-2021-surface-form-competition-why-highest/)
15. Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? In Association for Computational Linguistics (ACL), 2020. [find](https://scholar.google.com/scholar?q=Zhengbao+Jiang%2C+Frank+F+Xu%2C+Jun+Araki%2C+and+Graham+Neubig%2E+How+can+we+know+what+language+models+know%3F+In+Association+for+Computational+Linguistics+%28ACL%29%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/)
16. Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. An introduction to variational methods for graphical models. Machine Learning, 37:183–233, 1999. [find](https://scholar.google.com/scholar?q=Michael+I%2E+Jordan%2C+Zoubin+Ghahramani%2C+Tommi+S%2E+Jaakkola%2C+and+Lawrence+K%2E+Saul%2E+An+introduction+to+variational+methods+for+graphical+models%2E+Machine+Learning%2C+37%3A183%E2%80%93233%2C+1999%2E)
17. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Association for Computational Linguistics (ACL), 2017. [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Eunsol+Choi%2C+Daniel+Weld%2C+and+Luke+Zettlemoyer%2E+TriviaQA%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+In+Association+for+Computational+Linguistics+%28AC) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
18. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015. [find](https://scholar.google.com/scholar?q=Diederik+Kingma+and+Jimmy+Ba%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2C+2015%2E)
19. B.J.K. Kleijn and A.W. van der Vaart. The Bernstein-von mises theorem under misspecification. Electronic Journal of Statistics, 6, 2012. [find](https://scholar.google.com/scholar?q=B%2EJ%2EK%2E+Kleijn+and+A%2EW%2E+van+der+Vaart%2E+The+Bernstein%2Dvon+mises+theorem+under+misspecification%2E+Electronic+Journal+of+Statistics%2C+6%2C+2012%2E)
20. Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021. [find](https://scholar.google.com/scholar?q=Brian+Lester%2C+Rami+Al%2DRfou%2C+and+Noah+Constant%2E+The+power+of+scale+for+parameter%2Defficient+prompt+tuning%2E+arXiv+preprint+arXiv%3A2104%2E08691%2C+2021%2E)
21. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Association for Computational Linguistics (ACL), 2020. [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Yinhan+Liu%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Abdelrahman+Mohamed%2C+Omer+Levy%2C+Ves+Stoyanov%2C+and+Luke+Zettlemoyer%2E+Bart%3A+Denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+language+ge) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
22. Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Association for Computational Linguistics (ACL), 2021. [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2C+2021%2E)
23. Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. Jurassic-1: Technical details and evaluation. Technical report, AI21 Labs, August 2021. [find](https://scholar.google.com/scholar?q=Opher+Lieber%2C+Or+Sharir%2C+Barak+Lenz%2C+and+Yoav+Shoham%2E+Jurassic%2D1%3A+Technical+details+and+evaluation%2E+Technical+report%2C+AI21+Labs%2C+August+2021%2E)
24. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692, 2019. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+arXiv+p)
25. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+Decoupled+weight+decay+regularization%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2C+2019%2E)
26. Nicholas Metropolis, Arianna W. Rosenbluth, Marshall N. Rosenbluth, Augusta H. Teller, and Edward Teller. Equation of state calculations by fast computing machines. The journal of chemical physics, 21(6):1087–1092, 1953. [find](https://scholar.google.com/scholar?q=Nicholas+Metropolis%2C+Arianna+W%2E+Rosenbluth%2C+Marshall+N%2E+Rosenbluth%2C+Augusta+H%2E+Teller%2C+and+Edward+Teller%2E+Equation+of+state+calculations+by+fast+computing+machines%2E+The+journal+of+chemical+physics%2C+21)
27. Denis Paperno, German Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Association for Computational Linguistics (ACL), 2016. [find](https://scholar.google.com/scholar?q=Denis+Paperno%2C+German+Kruszewski%2C+Angeliki+Lazaridou%2C+Quan+Ngoc+Pham%2C+Raffaella+Bernardi%2C+Sandro+Pezzelle%2C+Marco+Baroni%2C+Gemma+Boleda%2C+and+Raquel+Fernandez%2E+The+LAMBADA+dataset%3A+Word+prediction+requir) [our card](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/)
28. Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. In ICLR MATH AI Workshop, 2021. [find](https://scholar.google.com/scholar?q=Alethea+Power%2C+Yuri+Burda%2C+Harri+Edwards%2C+Igor+Babuschkin%2C+and+Vedant+Misra%2E+Grokking%3A+Generalization+beyond+overfitting+on+small+algorithmic+datasets%2E+In+ICLR+MATH+AI+Workshop%2C+2021%2E)
29. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%2C+2019%2E)
30. Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International Conference on Learning Representations (ICLR), 2017. [find](https://scholar.google.com/scholar?q=Sachin+Ravi+and+Hugo+Larochelle%2E+Optimization+as+a+model+for+few%2Dshot+learning%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2C+2017%2E)
31. Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Da… [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Albert+Webson%2C+Colin+Raffel%2C+Stephen+H%2E+Bach%2C+Lintang+Sutawika%2C+Zaid+Alyafeai%2C+Antoine+Chaffin%2C+Arnaud+Stiegler%2C+Teven+Le+Scao%2C+Arun+Raja%2C+Manan+Dey%2C+M+Saiful+Bari%2C+Canwen+Xu%2C+Urmish+Thak) [our card](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
32. Timo Schick and Hinrich Schütze. Exploiting cloze questions for few shot text classification and natural language inference. In European Association for Computational Linguistics (EACL), 2021. [find](https://scholar.google.com/scholar?q=Timo+Schick+and+Hinrich+Sch%C3%BCtze%2E+Exploiting+cloze+questions+for+few+shot+text+classification+and+natural+language+inference%2E+In+European+Association+for+Computational+Linguistics+%28EACL%29%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/)
33. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Eliciting knowledge from language models using automatically generated prompts. In Empirical Methods in Natural Language Processing (EMNLP), 2020. [find](https://scholar.google.com/scholar?q=Taylor+Shin%2C+Yasaman+Razeghi%2C+Robert+L+Logan+IV%2C+Eric+Wallace%2C+and+Sameer+Singh%2E+Eliciting+knowledge+from+language+models+using+automatically+generated+prompts%2E+In+Empirical+Methods+in+Natural+Languag)
34. Ingo Steinwart. How to compare different loss functions and their risks. Constructive Approximation, 26, 2007. [find](https://scholar.google.com/scholar?q=Ingo+Steinwart%2E+How+to+compare+different+loss+functions+and+their+risks%2E+Constructive+Approximation%2C+26%2C+2007%2E)
35. A. W. van der Vaart. Asymptotic statistics. Cambridge University Press, 1998. [find](https://scholar.google.com/scholar?q=A%2E+W%2E+van+der+Vaart%2E+Asymptotic+statistics%2E+Cambridge+University+Press%2C+1998%2E)
36. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.03762, 2017. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+Attention+is+all+you+need%2E+arXiv+preprint+arXiv%3A1706%2E03762%2C+2017%2E)
37. Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [https://github.com/kingoflolz/mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax), May 2021. [↗](https://github.com/kingoflolz/mesh-transformer-jax)
38. Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning. arXiv, 2021a. [find](https://scholar.google.com/scholar?q=Colin+Wei%2C+Sang+Michael+Xie%2C+and+Tengyu+Ma%2E+Why+do+pretrained+language+models+help+in+downstream+tasks%3F+an+analysis+of+head+and+prompt+tuning%2E+arXiv%2C+2021a%2E)
39. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. arXiv, 2021b. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Bosma%2C+Vincent+Y%2E+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+M%2E+Dai%2C+and+Quoc+V%2E+Le%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+arXiv%2C+2021b%2E)
40. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. HuggingFace’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019. [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pierric+Cistac%2C+Tim+Rault%2C+R%E2%80%99emi+Louf%2C+Morgan+Funtowicz%2C+and+Jamie+Brew%2E+HuggingFace%E2%80%99s+transformers%3A+State%2Dof%2D) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
41. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations (ICLR), 2017. [find](https://scholar.google.com/scholar?q=Chiyuan+Zhang%2C+Samy+Bengio%2C+Moritz+Hardt%2C+Benjamin+Recht%2C+and+Oriol+Vinyals%2E+Understanding+deep+learning+requires+rethinking+generalization%2E+In+International+Conference+on+Learning+Representations+%28IC)
42. Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning (ICML), 2021. [find](https://scholar.google.com/scholar?q=Tony+Z%2E+Zhao%2C+Eric+Wallace%2C+Shi+Feng%2C+Dan+Klein%2C+and+Sameer+Singh%2E+Calibrate+before+use%3A+Improving+few%2Dshot+performance+of+language+models%2E+In+International+Conference+on+Machine+Learning+%28ICML%29%2C+2021) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/)
43. Bernardo Ávila Pires and Csaba Szepesvári. Multiclass classification calibration functions. arXiv, 2016. [find](https://scholar.google.com/scholar?q=Bernardo+%C3%81vila+Pires+and+Csaba+Szepesv%C3%A1ri%2E+Multiclass+classification+calibration+functions%2E+arXiv%2C+2016%2E)