--- title: "Prompt Programming for Large Language Models — Reynolds 2021" description: "This exploratory paper challenges the common view that GPT-3 learns new tasks at inference time from a handful of in-context examples." authors: ["Laria Reynolds", "Kyle McDonell"] year: 2021 venue: "arXiv:2102.07350" source: "https://arxiv.org/abs/2102.07350" doi: "10.48550/arxiv.2102.07350" arxiv: "2102.07350" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/reynolds-2021-prompt-programming-large-language-models/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/reynolds-2021-prompt-programming-large-language-models/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm # Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm **Authors:** [Laria Reynolds](https://md.research.elamur.ai/free/en/authors/laria-reynolds/), [Kyle McDonell](https://md.research.elamur.ai/free/en/authors/kyle-mcdonell/) · **Year:** 2021 · **Venue:** arXiv:2102.07350 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2102.07350) · [PDF](https://arxiv.org/pdf/2102.07350) · [DOI 10.48550/arxiv.2102.07350](https://doi.org/10.48550/arxiv.2102.07350) · [arXiv 2102.07350](https://arxiv.org/abs/2102.07350) · [OpenAlex](https://openalex.org/W3130319171) **Citations:** 882 citations (openalex-snapshot · 2026-10-14) · [229](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary This exploratory paper challenges the common view that GPT-3 learns new tasks at inference time from a handful of in-context examples. The authors argue that few-shot demonstrations mostly help the model identify which task it already knows, a process they call task location, rather than serving as meta-learning. They test this with a single case study, WMT'14 French-to-English translation, scored with BLEU via SacreBLEU on the Babbage and Curie models from the OpenAI API. Carefully worded 0-shot prompts matched or beat few-shot prompts, while adding a single example sometimes lowered scores. From this the authors develop a natural-language, author-modelling approach to writing prompts. It includes narrative and cultural framing, prompts that make the model reason step by step before giving an answer, and metaprompts that have the model write its own task prompts. The work matters because it suggests that standard benchmarks underestimated the 0-shot abilities of large language models. It also proposes better ways to evaluate them. ## Key points - On Curie, the "Simple Colon" 0-shot prompt (language names plus a colon) scored 33.3 BLEU, compared with 27.9 for the reproduced 10-shot OpenAI-style prompt and 18.7 for the reproduced OpenAI 0-shot prompt. - On Babbage, the "Master Translator" 0-shot prompt reached 26.5 BLEU, above the reproduced 10-shot result of 25.1. The 64-shot setting could not be replicated because of API limits. - With the Simple Colon format, moving from 0-shot to 1-shot reduced BLEU (Babbage 23.5 to 18.0; Curie 33.3 to 27.6). The authors attribute this to semantic contamination, where the model treats the example as part of the content rather than as a format guide. - Many failures of the original 0-shot prompt were catastrophic: the model did not attempt translation at all and instead continued in French or produced blanks. - For benchmarking, the authors propose: - reporting scores both with and without catastrophic failures, - evaluating with serializing or fill-in-the-blank metaprompts (illustrated on SAT analogies), - using text-based games to probe world-modelling. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) ## Cited by Cited by 229 papers of our library cite it (of the 140884 whose bibliographies we read); 26 of them have a page here, 203 not yet. - [Flamingo: a Visual Language Model for Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/) - [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/) - [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://md.research.elamur.ai/free/en/papers/du-2023-improving-factuality-reasoning-language-models/) - [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://md.research.elamur.ai/free/en/papers/holtzman-2021-surface-form-competition-why-highest/) - [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/) - [Can language models learn from explanations in context?](https://md.research.elamur.ai/free/en/papers/lampinen-2022-can-language-models-learn-explanations/) - [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://md.research.elamur.ai/free/en/papers/li-2023-camel-communicative-agents-mind/) - [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/) - [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/) - [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) - [Reframing Instructional Prompts to GPTk's Language](https://md.research.elamur.ai/free/en/papers/mishra-2022-reframing-instructional-prompts-gptk-s/) - [Automatic Prompt Optimization with “Gradient Descent” and Beam Search](https://md.research.elamur.ai/free/en/papers/pryzant-2023-automatic-prompt-optimization-gradient-descent/) - [A Recipe For Arbitrary Text Style Transfer with Large Language Models](https://md.research.elamur.ai/free/en/papers/reif-2021-recipe-arbitrary-text-style-transfer/) - [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/) - [The Prompt Report: A Systematic Survey of Prompt Engineering Techniques](https://md.research.elamur.ai/free/en/papers/schulhoff-2024-prompt-report-systematic-survey-prompt/) - [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://md.research.elamur.ai/free/en/papers/wang-2023-decodingtrust-comprehensive-assessment-trustworthiness-gpt/) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/) - [Emergent Abilities of Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/) - [Larger language models do in-context learning differently](https://md.research.elamur.ai/free/en/papers/wei-2023-larger-language-models-context-learning/) - [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://md.research.elamur.ai/free/en/papers/white-2023-prompt-pattern-catalog-enhance-prompt/) - [Large Language Models as Optimizers](https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-optimizers/) - [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://md.research.elamur.ai/free/en/papers/ye-2022-zerogen-efficient-zero-shot-learning/) - [GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation](https://md.research.elamur.ai/free/en/papers/yoo-2021-gpt3mix-leveraging-large-scale-language/) - [Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias](https://md.research.elamur.ai/free/en/papers/yu-2023-large-language-model-attributed-training/) - [BARTScore: Evaluating Generated Text as Text Generation](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/) - [Large Language Models Are Human-Level Prompt Engineers](https://md.research.elamur.ai/free/en/papers/zhou-2022-large-language-models-human-level/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{reynolds2021prompt, title = {{Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm}}, author = {Reynolds, Laria and McDonell, Kyle}, year = {2021}, doi = {10.48550/arxiv.2102.07350}, eprint = {2102.07350}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2102.07350} } ``` - [← previous Program Synthesis with Large Language Models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/) - [next → ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language](https://md.research.elamur.ai/free/en/papers/tafjord-2021-proofwriter-generating-implications-proofs-abductive/) - [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) ## References 15 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Ondrej Bojar et al. “Findings of the 2014 Workshop on Statistical Machine Translation”. In: Proceedings of the Ninth Workshop on Statistical Machine Translation. Baltimore, Maryland, USA: Association for Computational Linguistics, June 2014, pp. 12–58. Language models for evaluations [find](https://scholar.google.com/scholar?q=Ondrej+Bojar+et+al%2E+%E2%80%9CFindings+of+the+2014+Workshop+on+Statistical+Machine+Translation%E2%80%9D%2E+In%3A+Proceedings+of+the+Ninth+Workshop+on+Statistical+Machine+Translation%2E+Baltimore%2C+Maryland%2C+USA%3A+Association+) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2014-findings-2014-workshop-statistical-machine/) 2. Gwern Branwen. “GPT-3 Creative Fiction”. In: (2020). As language models become more powerful, it becomes conceivable to use other language models to evaluate the quality of responses to open-ended [find](https://scholar.google.com/scholar?q=Gwern+Branwen%2E+%E2%80%9CGPT%2D3+Creative+Fiction%E2%80%9D%2E+In%3A+%282020%29%2E+As+language+models+become+more+powerful%2C+it+becomes+conceivable+to+use+other+language+models+to+evaluate+the+quality+of+responses+to+open%2Dended) 3. Tom B Brown et al. “Language models are few-shot learners”. In: arXiv preprint arXiv:2005.14165 (2020). 5.2.3 [find](https://scholar.google.com/scholar?q=Tom+B+Brown+et+al%2E+%E2%80%9CLanguage+models+are+few%2Dshot+learners%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2005%2E14165+%282020%29%2E+5%2E2%2E3) 4. Marc-Alexandre Côté et al. “TextWorld: A Learning Environment for Text-based Games”. In: (2019). arXiv: 1806.11532 [cs.LG]. [16] Ben Krause et al. “GeDi: Generative Discriminator Guided Sequence Generation”. In: arXiv preprint arXiv:2009.06367 (2020). [find](https://scholar.google.com/scholar?q=Marc%2DAlexandre+Co%CC%82te%CC%81+et+al%2E+%E2%80%9CTextWorld%3A+A+Learning+Environment+for+Text%2Dbased+Games%E2%80%9D%2E+In%3A+%282019%29%2E+arXiv%3A+1806%2E11532+%5Bcs%2ELG%5D%2E+%5B16%5D+Ben+Krause+et+al%2E+%E2%80%9CGeDi%3A+Generative+Discriminator+Guided+Sequence+Gen) [our card](https://md.research.elamur.ai/free/en/papers/cote-2018-textworld-learning-environment-text-based/) 5. Jacob Devlin et al. “Bert: Pre-training of deep bidirectional transformers for language understanding”. In: arXiv preprint arXiv:1810.04805 (2018). [17] Xiang Lisa Li and Percy Liang. “Prefix-Tuning: Optimizing Continuous Prompts for Generation”. In: arXiv preprint arXiv:2101.00190 (2021). [find](https://scholar.google.com/scholar?q=Jacob+Devlin+et+al%2E+%E2%80%9CBert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A1810%2E04805+%282018%29%2E+%5B17%5D+Xiang+Lisa+Li+and+Percy+Liang%2E+%E2%80%9CPrefix%2DTuning%3A+) 6. Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical Neural Story Generation. 2018. arXiv: 1805.04833 [cs.CL]. [18] Jiangming Liu and Matt Gardner. “Multi-Step Inference for Reasoning Over Paragraphs”. In: arXiv preprint arXiv:2004.02995 (2020). [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Mike+Lewis%2C+and+Yann+Dauphin%2E+Hierarchical+Neural+Story+Generation%2E+2018%2E+arXiv%3A+1805%2E04833+%5Bcs%2ECL%5D%2E+%5B18%5D+Jiangming+Liu+and+Matt+Gardner%2E+%E2%80%9CMulti%2DStep+Inference+for+Reasoning+Over+Paragraph) 7. Zhe Gan et al. Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog. 2019. arXiv: 1902.00579 [cs.CV]. url: https://arxiv.org/abs/1902.00579. [19] Matt Post. “A Call for Clarity in Reporting BLEU Scores”. In: Proceedings of the Third Conference on Machine Translation: Research Papers.… [↗](https://arxiv.org/abs/1902.00579) [our card](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/) 8. Leo Gao. “Building AGI Using Language Models”. In: leogao.dev (2020). url: [https://bit.ly/3rViLGk](https://bit.ly/3rViLGk). [↗](https://bit.ly/3rViLGk) 9. Tianyu Gao, Adam Fisch, and Danqi Chen. Making Pre-trained Language Models Better Few-shot Learners. 2020. arXiv: 2012.15723 [cs.CL]. [20] Zachary Robertson. You Can Probably Amplify GPT3 Directly. 2020. url: [https://bit.ly/3tXT7Cw](https://bit.ly/3tXT7Cw). [21] Arram Sabeti. GPT-3: Using Fiction to Demonstrate How Prompts… [↗](https://bit.ly/3tXT7Cw) [our card](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/) 10. Dan Hendrycks et al. “Measuring massive multitask language understanding”. In: arXiv preprint arXiv:2009.03300 (2020). url: https://arxiv.org/abs/2009.03300. [22] Taylor Shin et al. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. 2020. arXiv: 2010.15980 [cs… [↗](https://arxiv.org/abs/2009.03300) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/) 11. Ari Holtzman et al. The Curious Case of Neural Text Degeneration. 2020. arXiv: 1904.09751 [cs.CL]. [23] Latitude Team. World Creation by Analogy. 2020. url: [https://bit.ly/2N4vXK0](https://bit.ly/2N4vXK0). [↗](https://bit.ly/2N4vXK0) 12. Jeremy Howard and Sebastian Ruder. “Universal language model fine-tuning for text classification”. In: arXiv preprint arXiv:1801.06146 (2018). url: https://arxiv.org/abs/1801.06146. [24] Lilian Wang. “Controllable Neural Text Generation”. In: (2021). url: https://bit.ly/3pl2eKa. [25] Qinyuan Ye and… [↗](https://arxiv.org/abs/1801.06146) 13. KaryoKleptid. Seems to work. 2020. url: [https://bit.ly/37dA1hY](https://bit.ly/37dA1hY). [↗](https://bit.ly/37dA1hY) 14. KaryoKleptid. Teaching GPT-3 to do a brute force ’for loop’ checking answers. 2020. url: [https://bit.ly/2N7khX1](https://bit.ly/2N7khX1). [26] Jianxing Yu et al. “Low-resource generation of multi-hop reasoning questions”. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020, pp.… [↗](https://bit.ly/2N7khX1) 15. Nitish Shirish Keskar et al. “CTRL: A Conditional Transformer Language Model for Controllable Generation”. In: CoRR abs/1909.05858 (2019). arXiv: 1909.05858. url: http://arxiv.org/abs/1909.05858. [27] Eliezer Yudkowsky. “Rationalization”. In: lesswrong.com (2007). url: https://bit.ly/3pmYt6I. [↗](http://arxiv.org/abs/1909.05858) [our card](https://md.research.elamur.ai/free/en/papers/keskar-2019-ctrl-conditional-transformer-language-model/)