---
title: "Demystifying Prompts in Language Models via Perplexity"
description: "The authors propose that a prompt works better when the model finds its wording… 71.2 accuracy. 32 references, cited by 109 papers here."
authors: ["Hila Gonen", "Srini Iyer", "Terra Blevins", "Noah A. Smith", "Luke Zettlemoyer"]
year: 2022
venue: "arXiv:2212.04037"
source: "https://arxiv.org/abs/2212.04037"
doi: "10.48550/arxiv.2212.04037"
arxiv: "2212.04037"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gonen-2022-demystifying-prompts-language-models-perplexity/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gonen-2022-demystifying-prompts-language-models-perplexity/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/) / Demystifying Prompts in Language Models via Perplexity Estimation

# Demystifying Prompts in Language Models via Perplexity Estimation

**Authors:** [Hila Gonen](https://md.research.elamur.ai/free/en/authors/hila-gonen/), [Srini Iyer](https://md.research.elamur.ai/free/en/authors/srini-iyer/), [Terra Blevins](https://md.research.elamur.ai/free/en/authors/terra-blevins/), [Noah A. Smith](https://md.research.elamur.ai/free/en/authors/noah-a-smith/) and [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) · **Year:** 2022 · **Venue:** arXiv:2212.04037 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2212.04037) · [PDF](https://arxiv.org/pdf/2212.04037) · [DOI 10.48550/arxiv.2212.04037](https://doi.org/10.48550/arxiv.2212.04037) · [arXiv 2212.04037](https://arxiv.org/abs/2212.04037) · [OpenAlex](https://openalex.org/W4389520255)

**Citations:** 104 citations (OpenAlex, 2026-10-14) · [109](https://md.research.elamur.ai/free/en/papers/gonen-2022-demystifying-prompts-language-models-perplexity/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Prompts that look equally reasonable to a human can produce very different results from a large language model. This paper asks why. The authors propose that a prompt works better when the model finds its wording more familiar. Pretraining corpora are huge and often unavailable, so they use the prompt's perplexity as a stand-in for how often similar phrasing appears in that data. To get enough prompts per task for a meaningful analysis, they expand a few handwritten seed prompts automatically. First, GPT3 paraphrases them. Then each paraphrase is translated into other languages with NLLB and back into English. They tested OPT models of several sizes and Bloom on word-prediction and classification tasks. Across these, prompts with lower perplexity tended to perform better. Building on this, the authors propose SPELL, which ranks candidate prompts by perplexity and keeps the lowest few. SPELL needs no labels, only sample inputs, so non-experts can use it to choose stable, effective prompts for new tasks.

## Key points
- On AG News with OPT 175B, four manual prompts ranged from 40.9 to 71.2 accuracy, a gap of more than 30 points. Only 9 of 108 AG News prompts appeared verbatim in the OPT training data, which is why the authors used perplexity instead of exact counts.
- The expansion pipeline has three stages:
  - seed prompts written by hand;
  - paraphrasing with GPT3 text-davinci-002 using 7 meta-prompts;
  - backtranslation through 8 high-resource languages via NLLB, with 8 more added (including Ukrainian) when needed, aiming for about 100 prompts per task.
- The models tested were OPT (1.3B, 30B, 175B) and Bloom (176B). Tasks included:
  - word-level translation, using NorthEuraLex (1016 words, 107 languages);
  - predicting a word's opposite (antonym prediction);
  - classification on Emotion, Tweet Offensive, DBpedia, IMDB, AG News, Newspop and the Cola task from GLUE.
- Prompt perplexity is computed over the prompt plus the task input, excluding the label, and averaged over 1,000 examples. Performance is measured mainly by the log-likelihood of the correct label, with accuracy also reported for classification tasks.
- SPELL keeps the 3 lowest-perplexity prompts. Compared with manual prompts, it improves accuracy by 1.8 points on average with OPT and 2.3 with Bloom, and results vary less across prompts.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 402 works · 212,763 citations
- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 304 works · 106,137 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith and Luke Zettlemoyer. 2022. Demystifying Prompts in Language Models via Perplexity Estimation. arXiv:2212.04037.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{gonen2022demystifying,
  title = {{Demystifying Prompts in Language Models via Perplexity Estimation}},
  author = {Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith and Luke Zettlemoyer},
  year = {2022},
  doi = {10.48550/arxiv.2212.04037},
  eprint = {2212.04037},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2212.04037}
}
```

- [← previous Systematic Inequalities in Language Technology Performance across the World's Languages](https://md.research.elamur.ai/free/en/papers/blasi-2022-systematic-inequalities-language-technology-performance/)
- [next → Language Models of Code are Few-Shot Commonsense Learners](https://md.research.elamur.ai/free/en/papers/madaan-2022-language-models-code-few-shot/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/)

## References

32 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Stephen H. Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged S. Al-shaib… — [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://md.research.elamur.ai/free/en/papers/bach-2022-promptsource-integrated-development-environment-repository/)
2. Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa-Anke, and Leonardo Neves. 2020. [TweetEval:Unified Benchmark and Comparative Evaluation for Tweet Classification](https://md.research.elamur.ai/free/en/papers/barbieri-2020-tweeteval-unified-benchmark-comparative-evaluation/). In Proceedings of Findings of EMNLP.
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
4. Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. 2022. On the relation between sensitivity and accuracy in in-context learning. arXiv e-prints, pages arXiv–2209. [find](https://scholar.google.com/scholar?q=Yanda+Chen%2C+Chen+Zhao%2C+Zhou+Yu%2C+Kathleen+McKeown%2C+and+He+He%2E+2022%2E+On+the+relation+between+sensitivity+and+accuracy+in+in%2Dcontext+learning%2E+arXiv+e%2Dprints%2C+pages+arXiv%E2%80%932209%2E)
5. Singh, and Yejin Choi. 2022b. Prompt waywardness: The curious case of discretized interpretation of continuous prompts. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. [find](https://scholar.google.com/scholar?q=Singh%2C+and+Yejin+Choi%2E+2022b%2E+Prompt+waywardness%3A+The+curious+case+of+discretized+interpretation+of+continuous+prompts%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Chapter+of+the+Assoc)
6. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. [find](https://scholar.google.com/scholar?q=Marta+R+Costa%2Djuss%C3%A0%2C+James+Cross%2C+Onur+%C3%87elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+Jean+Maillard%2C+et+al%2E+2022%2E+No+language+left+behind%3A+Scaling+h)
7. Teven Le Scao and Alexander Rush. 2021. [How many data points is a prompt worth?](https://md.research.elamur.ai/free/en/papers/scao-2021-how-many-data-points-is/) In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics.
8. Johannes Dellert, Thora Daneyko, Alla Münch, Alina Ladygina, Armin Buch, Natalie Clarius, Ilja Grigor-jew, Mohamed Balabel, Hizniye Isabella Boga, Zalina Baysarova, et al. 2019. Northeuralex: a wide-coverage lexical database of northern eurasia. Language Resources and Evaluation, pages 1–29. [find](https://scholar.google.com/scholar?q=Johannes+Dellert%2C+Thora+Daneyko%2C+Alla+M%C3%BCnch%2C+Alina+Ladygina%2C+Armin+Buch%2C+Natalie+Clarius%2C+Ilja+Grigor%2Djew%2C+Mohamed+Balabel%2C+Hizniye+Isabella+Boga%2C+Zalina+Baysarova%2C+et+al%2E+2019%2E+Northeuralex%3A+a+wide%2Dc)
9. Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. 2015. [Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia](https://md.research.elamur.ai/free/en/papers/lehmann-2015-dbpedia-large-scale-multilingual-knowledge/). Semantic web, 6(2):167–195.
10. Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P Xing, and Zhiting Hu. 2022. [Rlprompt: Optimizing discrete text prompts with reinforcement learning](https://md.research.elamur.ai/free/en/papers/deng-2022-rlprompt-optimizing-discrete-text-prompts/). arXiv preprint arXiv:2205.12548.
11. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059.
12. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schütze, and Yoav Goldberg. 2022. Measuring causal effects of data statistics on language model’sfactual’predictions. arXiv preprint arXiv:2207.14251. — [arXiv:2207.14251](https://arxiv.org/abs/2207.14251) [find](https://scholar.google.com/scholar?q=Yanai+Elazar%2C+Nora+Kassner%2C+Shauli+Ravfogel%2C+Amir+Feder%2C+Abhilasha+Ravichander%2C+Marius+Mosbach%2C+Yonatan+Belinkov%2C+Hinrich+Sch%C3%BCtze%2C+and+Yoav+Goldberg%2E+2022%2E+Measuring+causal+effects+of+data+statistics+)
13. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–… [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11t)
14. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. [Making pre-trained language models better few-shot learners](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pa…
15. Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. 2022. Estimating the carbon footprint of bloom, a 176b parameter language model. arXiv preprint arXiv:2211.02001. — [arXiv:2211.02001](https://arxiv.org/abs/2211.02001) [find](https://scholar.google.com/scholar?q=Alexandra+Sasha+Luccioni%2C+Sylvain+Viguier%2C+and+Anne%2DLaure+Ligozat%2E+2022%2E+Estimating+the+carbon+footprint+of+bloom%2C+a+176b+parameter+language+model%2E+arXiv+preprint+arXiv%3A2211%2E02001%2E+%E2%80%94+%5BarXiv%3A2211%2E02001)
16. Xiaochuang Han and Yulia Tsvetkov. 2022. Orca: Interpreting prompted language models via locating supporting data evidence in the ocean of pretraining data. arXiv preprint arXiv:2205.12600. — [arXiv:2205.12600](https://arxiv.org/abs/2205.12600) [find](https://scholar.google.com/scholar?q=Xiaochuang+Han+and+Yulia+Tsvetkov%2E+2022%2E+Orca%3A+Interpreting+prompted+language+models+via+locating+supporting+data+evidence+in+the+ocean+of+pretraining+data%2E+arXiv+preprint+arXiv%3A2205%2E12600%2E+%E2%80%94+%5BarXiv%3A2)
17. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. [Learning word vectors for sentiment analysis](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/). In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon…
18. Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. [Surface form competition: Why the highest probability answer isn’t always right](https://md.research.elamur.ai/free/en/papers/holtzman-2021-surface-form-competition-why-highest/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038–7051.
19. George A Miller. 1995. [Wordnet: a lexical database for english](https://md.research.elamur.ai/free/en/papers/miller-1994-wordnet/). Communications of the ACM, 38(11):39–41.
20. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2022. [Large language models struggle to learn long-tail knowledge](https://md.research.elamur.ai/free/en/papers/kandpal-2022-large-language-models-struggle-learn/). arXiv preprint arXiv:2211.08411.
21. N. Moniz and L. Torgo. 2018. Multi-source social feedback of online news feeds. ArXiv, abs/1801.07055. — [arXiv:1801.07055](https://arxiv.org/abs/1801.07055) [find](https://scholar.google.com/scholar?q=N%2E+Moniz+and+L%2E+Torgo%2E+2018%2E+Multi%2Dsource+social+feedback+of+online+news+feeds%2E+ArXiv%2C+abs%2F1801%2E07055%2E+%E2%80%94+%5BarXiv%3A1801%2E07055%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1801%2E07055%29)
22. Guanghui Qin and Jason Eisner. 2021. [Learning how to ask: Querying lms with mixtures of soft prompts](https://md.research.elamur.ai/free/en/papers/qin-2021-learning-ask-querying-lms-mixtures/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5203–5212.
23. Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. 2022a. [Reframing instructional prompts to gptk’s language](https://md.research.elamur.ai/free/en/papers/mishra-2022-reframing-instructional-prompts-gptk-s/). In Findings of the Association for Computational Linguistics: ACL 2022, pages 589–612.
24. Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot reasoning. arXiv preprint arXiv:2202.07206. — [arXiv:2202.07206](https://arxiv.org/abs/2202.07206) [find](https://scholar.google.com/scholar?q=Yasaman+Razeghi%2C+Robert+L+Logan+IV%2C+Matt+Gardner%2C+and+Sameer+Singh%2E+2022%2E+Impact+of+pretraining+term+frequencies+on+few%2Dshot+reasoning%2E+arXiv+preprint+arXiv%3A2202%2E07206%2E+%E2%80%94+%5BarXiv%3A2202%2E07206%5D%28https%3A%2F%2Far)
25. Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Hannaneh Hajishirzi, Tushar Khot, Ashish Sabharwal, Sameer Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. 2018. [CARER: Contextualized affect representations for emotion recognition](https://md.research.elamur.ai/free/en/papers/saravia-2018-carer-contextualized-affect-representations-emotion/). In…
26. Timo Schick and Hinrich Schütze. 2020. [It’s not just size that matters: Small language models are also few-shot learners](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/). arXiv preprint arXiv:2009.07118.
27. Weijia Shi, Xiaochuang Han, Hila Gonen, Ari Holtzman, Yulia Tsvetkov, and Luke Zettlemoyer. 2022. Toward human readable prompt tuning: Kubrick’s the shining is a good movie, and a good prompt too? arXiv preprint arXiv:2212.10539. [find](https://scholar.google.com/scholar?q=Weijia+Shi%2C+Xiaochuang+Han%2C+Hila+Gonen%2C+Ari+Holtzman%2C+Yulia+Tsvetkov%2C+and+Luke+Zettlemoyer%2E+2022%2E+Toward+human+readable+prompt+tuning%3A+Kubrick%E2%80%99s+the+shining+is+a+good+movie%2C+and+a+good+prompt+too%3F+arX)
28. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. [Autoprompt: Eliciting knowledge from language models with automatically generated prompts](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/). arXiv preprint arXiv:2010.15980.
29. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R+Bowman%2E+2018%2E+Neural+network+acceptability+judgments%2E+arXiv+preprint+arXiv%3A1805%2E12471%2E)
30. John Wieting, Jonathan Mallinson, and Kevin Gimpel. 2017. Learning paraphrastic sentence embeddings from back-translated bitext. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 274–285. [find](https://scholar.google.com/scholar?q=John+Wieting%2C+Jonathan+Mallinson%2C+and+Kevin+Gimpel%2E+2017%2E+Learning+paraphrastic+sentence+embeddings+from+back%2Dtranslated+bitext%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+L)
31. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068.
32. Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. [Character-level convolutional networks for text classification](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/). In NIPS.

*Updated 2 November 2026*