---
title: "Prompting is not a substitute for probability — Hu 2023"
description: "Many recent claims about what large language models (LLMs) know about language rely on prompting, where the model is asked in natural language to…"
authors: ["Jennifer Hu", "Roger Levy"]
year: 2023
venue: "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2023.emnlp-main.306/"
doi: "10.18653/v1/2023.emnlp-main.306"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hu-2023-prompting-is-not-substitute-probability/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hu-2023-prompting-is-not-substitute-probability/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / Prompting is not a substitute for probability measurements in large language models

# Prompting is not a substitute for probability measurements in large language models

**Authors:** [Jennifer Hu](https://md.research.elamur.ai/free/en/authors/jennifer-hu/), [Roger Levy](https://md.research.elamur.ai/free/en/authors/roger-levy/) · **Institutions:** [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) · [Harvard University](https://md.research.elamur.ai/free/en/universities/harvard-university/) · **Year:** 2023 · **Venue:** Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.emnlp-main.306/) · [PDF](https://aclanthology.org/2023.emnlp-main.306.pdf) · [DOI 10.18653/v1/2023.emnlp-main.306](https://doi.org/10.18653/v1/2023.emnlp-main.306) · [OpenAlex](https://openalex.org/W4389518756)

**Citations:** 42 citations (OpenAlex, 2026-10-14) · [91](https://md.research.elamur.ai/free/en/papers/hu-2023-prompting-is-not-substitute-probability/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Many recent claims about what large language models (LLMs) know about language rely on prompting, where the model is asked in natural language to evaluate a sentence. Hu and Levy argue that this adds a second skill to the test: metalinguistic judgment, meaning the ability to report what the model's own probability distribution encodes. They ask whether such prompted answers agree with probabilities read directly from the output logits.

The study has four experiments: word prediction, word comparison for semantic plausibility, and isolated versus paired sentence judgments on syntax. Each contrasts a direct probability method with three zero-shot metalinguistic prompts. The materials draw on SyntaxGym, BLiMP, Vassallo et al. (2018), Pereira et al. (2018), and a set of news stories published in March 2023.

Direct measurements generally matched or outperformed prompting. Agreement between the two approaches weakened as the prompt moved further from plain next-word prediction.

The upshot is that failure under a metalinguistic prompt does not prove a model lacks a generalization. Closed APIs that hide log-probabilities also take away an important research tool.

## Key points
- Every experiment compares four methods: one direct probability readout and three zero-shot metalinguistic prompts.
- Word prediction is the baseline, where direct next-token probabilities serve as ground truth. The other tasks (word comparison, sentence judgment, sentence comparison) are ordered by how closely they resemble it.
- Minimal-pair comparisons revealed more of a model's syntactic knowledge than judging single sentences in isolation.
- The authors suggest a competence–performance split for LLMs. Competence is the string probability distribution, and performance is behavior under prompts. This position contrasts with Katzir (2023).
- Shortly before the paper's EMNLP acceptance, OpenAI removed arbitrary token log-probabilities for gpt-3.5-turbo-instruct. The OpenAI results may not be reproducible, and code and data are at github.com/jennhu/metalinguistic-prompting.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{hu2023prompting,
  title = {{Prompting is not a substitute for probability measurements in large language models}},
  author = {Hu, Jennifer and Levy, Roger},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.306},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.306}
}
```

- [← previous Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models](https://md.research.elamur.ai/free/en/papers/singh-2023-human-data-scaling-self-training/)
- [next → Self-Alignment with Instruction Backtranslation](https://md.research.elamur.ai/free/en/papers/li-2023-self-alignment-instruction-backtranslation/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

51 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marco Baroni. 2022. On the proper role of linguistically-oriented deep net analysis in linguistic theorizing. In Shalom Lappin and Jean-Philippe Bernardy, editors, Algebraic Structures in Natural Language. Taylor & Francis. [find](https://scholar.google.com/scholar?q=Marco+Baroni%2E+2022%2E+On+the+proper+role+of+linguistically%2Doriented+deep+net+analysis+in+linguistic+theorizing%2E+In+Shalom+Lappin+and+Jean%2DPhilippe+Bernardy%2C+editors%2C+Algebraic+Structures+in+Natural+Lang)
2. Gašper Beguš, Maksymilian D abkowski, and Ryan Rhodes. 2023. Large Linguistic Models: Analyzing theoretical linguistic abilities of LLMs. [find](https://scholar.google.com/scholar?q=Ga%C5%A1per+Begu%C5%A1%2C+Maksymilian+D+abkowski%2C+and+Ryan+Rhodes%2E+2023%2E+Large+Linguistic+Models%3A+Analyzing+theoretical+linguistic+abilities+of+LLMs%2E)
3. Anne Beyer, Sharid Loáiciga, and David Schlangen. 2021. Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, page… [find](https://scholar.google.com/scholar?q=Anne+Beyer%2C+Sharid+Lo%C3%A1iciga%2C+and+David+Schlangen%2E+2021%2E+Is+Incoherence+Surprising%3F+Targeted+Evaluation+of+Coherence+Prediction+from+Language+Models%2E+In+Proceedings+of+the+2021+Conference+of+the+North+)
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretchen)
5. Kristy Choi, Chris Cundy, Sanjari Srivastava, and Stefano Ermon. 2022. LMPriors: Pre-Trained Language Models as Task-Specific Priors. [find](https://scholar.google.com/scholar?q=Kristy+Choi%2C+Chris+Cundy%2C+Sanjari+Srivastava%2C+and+Stefano+Ermon%2E+2022%2E+LMPriors%3A+Pre%2DTrained+Language+Models+as+Task%2DSpecific+Priors%2E)
6. Noam Chomsky. 1965. Aspects of the Theory of Syntax. MIT Press. [find](https://scholar.google.com/scholar?q=Noam+Chomsky%2E+1965%2E+Aspects+of+the+Theory+of+Syntax%2E+MIT+Press%2E)
7. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, A… [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+Albert+Webson%2C+Shixiang+Shane+Gu%2C+Zhuyun+Dai%2C+Mirac+Suzgun%2C+Xiny)
8. Pablo Contreras Kallens, Ross Deans Kristensen-McLachlan, and Morten H. Christiansen. 2023. Large Language Models Demonstrate the Potential of Statistical Learning in Language. Cognitive Science, 47(3):e13256. Publisher: John Wiley & Sons, Ltd. [find](https://scholar.google.com/scholar?q=Pablo+Contreras+Kallens%2C+Ross+Deans+Kristensen%2DMcLachlan%2C+and+Morten+H%2E+Christiansen%2E+2023%2E+Large+Language+Models+Demonstrate+the+Potential+of+Statistical+Learning+in+Language%2E+Cognitive+Science%2C+47%283)
9. Vittoria Dentella, Elliot Murphy, Gary Marcus, and Evelina Leivada. 2023. Testing AI performance on less frequent aspects of language reveals insensitivity to underlying meaning. [find](https://scholar.google.com/scholar?q=Vittoria+Dentella%2C+Elliot+Murphy%2C+Gary+Marcus%2C+and+Evelina+Leivada%2E+2023%2E+Testing+AI+performance+on+less+frequent+aspects+of+language+reveals+insensitivity+to+underlying+meaning%2E)
10. Gabe Dupre. 2021. (What) Can Deep Learning Contribute to Theoretical Linguistics? Minds and Machines, 31(4):617–635. [find](https://scholar.google.com/scholar?q=Gabe+Dupre%2E+2021%2E+%28What%29+Can+Deep+Learning+Contribute+to+Theoretical+Linguistics%3F+Minds+and+Machines%2C+31%284%29%3A617%E2%80%93635%2E)
11. Chaz Firestone. 2020. Performance vs. competence in human–machine comparisons. Proceedings of the National Academy of Sciences, 117(43):26562– 26571. Publisher: Proceedings of the National Academy of Sciences. [find](https://scholar.google.com/scholar?q=Chaz+Firestone%2E+2020%2E+Performance+vs%2E+competence+in+human%E2%80%93machine+comparisons%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+117%2843%29%3A26562%E2%80%93+26571%2E+Publisher%3A+Proceedings+of+the+National+Academy+of+)
12. Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Richard+Futrell%2C+Ethan+Wilcox%2C+Takashi+Morita%2C+Peng+Qian%2C+Miguel+Ballesteros%2C+and+Roger+Levy%2E+2019%2E+Neural+language+models+as+psycholinguistic+subjects%3A+Representations+of+syntactic+state%2E+In+Proceedi) [our card](https://md.research.elamur.ai/free/en/papers/futrell-2019-neural-language-models-as-psycholinguistic/)
13. Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. SyntaxGym: An Online Platform for Targeted Evaluation of Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70–76, Online. Association for… [find](https://scholar.google.com/scholar?q=Jon+Gauthier%2C+Jennifer+Hu%2C+Ethan+Wilcox%2C+Peng+Qian%2C+and+Roger+Levy%2E+2020%2E+SyntaxGym%3A+An+Online+Platform+for+Targeted+Evaluation+of+Language+Models%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Ass) [our card](https://md.research.elamur.ai/free/en/papers/gauthier-2020-syntaxgym-online-platform-targeted-evaluation/)
14. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 1321–1330. [find](https://scholar.google.com/scholar?q=Chuan+Guo%2C+Geoff+Pleiss%2C+Yu+Sun%2C+and+Kilian+Q%2E+Weinberger%2E+2017%2E+On+Calibration+of+Modern+Neural+Networks%2E+In+Proceedings+of+the+34th+International+Conference+on+Machine+Learning%2C+volume+70%2C+pages+132)
15. Michael Hahn, Richard Futrell, Roger P. Levy, and Edward Gibson. 2022. A resource-rational model of human processing of recursive linguistic structure. 119(43). [find](https://scholar.google.com/scholar?q=Michael+Hahn%2C+Richard+Futrell%2C+Roger+P%2E+Levy%2C+and+Edward+Gibson%2E+2022%2E+A+resource%2Drational+model+of+human+processing+of+recursive+linguistic+structure%2E+119%2843%29%2E)
16. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A Systematic Assessment of Syntactic Generalization in Neural Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online. Association for Computational… [find](https://scholar.google.com/scholar?q=Jennifer+Hu%2C+Jon+Gauthier%2C+Peng+Qian%2C+Ethan+Wilcox%2C+and+Roger+Levy%2E+2020%2E+A+Systematic+Assessment+of+Syntactic+Generalization+in+Neural+Language+Models%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+th)
17. Janus. 2022. Mysteries of mode collapse. [find](https://scholar.google.com/scholar?q=Janus%2E+2022%2E+Mysteries+of+mode+collapse%2E)
18. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny… [find](https://scholar.google.com/scholar?q=Saurav+Kadavath%2C+Tom+Conerly%2C+Amanda+Askell%2C+Tom+Henighan%2C+Dawn+Drain%2C+Ethan+Perez%2C+Nicholas+Schiefer%2C+Zac+Hatfield%2DDodds%2C+Nova+DasSarma%2C+Eli+Tran%2DJohnson%2C+Scott+Johnston%2C+Sheer+El%2DShowk%2C+Andy+Jones%2C+)
19. Roni Katzir. 2023. Why large language models are poor theories of human linguistic cognition: A reply to Piantadosi (2023). [find](https://scholar.google.com/scholar?q=Roni+Katzir%2E+2023%2E+Why+large+language+models+are+poor+theories+of+human+linguistic+cognition%3A+A+reply+to+Piantadosi+%282023%29%2E)
20. Carina Kauf, Anna A. Ivanova, Giulia Rambelli, Emmanuele Chersoni, Jingyuan S. She, Zawad Chowdhury, Evelina Fedorenko, and Alessandro Lenci. 2022. Event knowledge in large language models: the gap between the impossible and the unlikely. [find](https://scholar.google.com/scholar?q=Carina+Kauf%2C+Anna+A%2E+Ivanova%2C+Giulia+Rambelli%2C+Emmanuele+Chersoni%2C+Jingyuan+S%2E+She%2C+Zawad+Chowdhury%2C+Evelina+Fedorenko%2C+and+Alessandro+Lenci%2E+2022%2E+Event+knowledge+in+large+language+models%3A+the+gap+be)
21. Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Hannaneh Hajishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, and Yejin Choi. 2022. Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts. In Proceedings of the 2022 Conference o… [find](https://scholar.google.com/scholar?q=Daniel+Khashabi%2C+Xinxi+Lyu%2C+Sewon+Min%2C+Lianhui+Qin%2C+Kyle+Richardson%2C+Sean+Welleck%2C+Hannaneh+Hajishirzi%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+Sameer+Singh%2C+and+Yejin+Choi%2E+2022%2E+Prompt+Waywardness%3A+The+Curio)
22. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-Shot Reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199–22213. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Takeshi+Kojima%2C+Shixiang+%28Shane%29+Gu%2C+Machel+Reid%2C+Yutaka+Matsuo%2C+and+Yusuke+Iwasawa%2E+2022%2E+Large+Language+Models+are+Zero%2DShot+Reasoners%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+3) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)
23. Andrew Kyle Lampinen. 2023. Can language models handle recursively nested grammatical structures? A case study on comparing models and humans. [find](https://scholar.google.com/scholar?q=Andrew+Kyle+Lampinen%2E+2023%2E+Can+language+models+handle+recursively+nested+grammatical+structures%3F+A+case+study+on+comparing+models+and+humans%2E)
24. Nur Lan, Emmanuel Chemla, and Roni Katzir. 2022. Large Language Models and the Argument From the Poverty of the Stimulus. [find](https://scholar.google.com/scholar?q=Nur+Lan%2C+Emmanuel+Chemla%2C+and+Roni+Katzir%2E+2022%2E+Large+Language+Models+and+the+Argument+From+the+Poverty+of+the+Stimulus%2E)
25. Belinda Z. Li, William Chen, Pratyusha Sharma, and Jacob Andreas. 2023. LaMPP: Language Models as Probabilistic Priors for Perception and Action. [find](https://scholar.google.com/scholar?q=Belinda+Z%2E+Li%2C+William+Chen%2C+Pratyusha+Sharma%2C+and+Jacob+Andreas%2E+2023%2E+LaMPP%3A+Language+Models+as+Probabilistic+Priors+for+Perception+and+Action%2E)
26. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies. Transactions of the Association for Computational Linguistics, 4:521– 535. [find](https://scholar.google.com/scholar?q=Tal+Linzen%2C+Emmanuel+Dupoux%2C+and+Yoav+Goldberg%2E+2016%2E+Assessing+the+Ability+of+LSTMs+to+Learn+Syntax%2DSensitive+Dependencies%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+4%3A521%E2%80%93+535%2E) [our card](https://md.research.elamur.ai/free/en/papers/linzen-2016-assessing-ability-lstms-learn-syntax/)
27. Benjamin Lipkin, Lionel Wong, Gabriel Grand, and Joshua B. Tenenbaum. 2023. Evaluating statistical language models as pragmatic reasoners. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 45. [find](https://scholar.google.com/scholar?q=Benjamin+Lipkin%2C+Lionel+Wong%2C+Gabriel+Grand%2C+and+Joshua+B%2E+Tenenbaum%2E+2023%2E+Evaluating+statistical+language+models+as+pragmatic+reasoners%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Cognitive+Science) [our card](https://md.research.elamur.ai/free/en/papers/lipkin-2023-evaluating-statistical-language-models-pragmatic/)
28. R. Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L. Griffiths. 2023. Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve. [find](https://scholar.google.com/scholar?q=R%2E+Thomas+McCoy%2C+Shunyu+Yao%2C+Dan+Friedman%2C+Matthew+Hardy%2C+and+Thomas+L%2E+Griffiths%2E+2023%2E+Embers+of+Autoregression%3A+Understanding+Large+Language+Models+Through+the+Problem+They+are+Trained+to+Solve%2E)
29. Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022. Reducing Conversational Agents’ Overconfidence Through Linguistic Calibration. Transactions of the Association for Computational Linguistics, 10:857–872. [find](https://scholar.google.com/scholar?q=Sabrina+J%2E+Mielke%2C+Arthur+Szlam%2C+Emily+Dinan%2C+and+Y%2DLan+Boureau%2E+2022%2E+Reducing+Conversational+Agents%E2%80%99+Overconfidence+Through+Linguistic+Calibration%2E+Transactions+of+the+Association+for+Computational+)
30. Daniel Milway. 2023. A Response to Piantadosi (2023). [find](https://scholar.google.com/scholar?q=Daniel+Milway%2E+2023%2E+A+Response+to+Piantadosi+%282023%29%2E)
31. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Ab… [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Xinxi+Lyu%2C+Ari+Holtzman%2C+Mikel+Artetxe%2C+Mike+Lewis%2C+Hannaneh+Hajishirzi%2C+and+Luke+Zettlemoyer%2E+2022%2E+Rethinking+the+Role+of+Demonstrations%3A+What+Makes+In%2DContext+Learning+Work%3F+In+Proceedin) [our card](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/)
32. Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. 2021. Revisiting the Calibration of Modern Neural Networks. In Advances in Neural Information Processing Systems, volume 34, pages 15682–15694. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Matthias+Minderer%2C+Josip+Djolonga%2C+Rob+Romijnders%2C+Frances+Hubis%2C+Xiaohua+Zhai%2C+Neil+Houlsby%2C+Dustin+Tran%2C+and+Mario+Lucic%2E+2021%2E+Revisiting+the+Calibration+of+Modern+Neural+Networks%2E+In+Advances+in+N)
33. Arseny Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. 2023. The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain. [find](https://scholar.google.com/scholar?q=Arseny+Moskvichev%2C+Victor+Vikram+Odouard%2C+and+Melanie+Mitchell%2E+2023%2E+The+ConceptARC+Benchmark%3A+Evaluating+Understanding+and+Generalization+in+the+ARC+Domain%2E)
34. Elliot Murphy. 2023. Notes on Large Language Models and Linguistic Theory. [find](https://scholar.google.com/scholar?q=Elliot+Murphy%2E+2023%2E+Notes+on+Large+Language+Models+and+Linguistic+Theory%2E)
35. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show Your Work: Scratchpads for Intermediate Computation with Language Models. [find](https://scholar.google.com/scholar?q=Maxwell+Nye%2C+Anders+Johan+Andreassen%2C+Guy+Gur%2DAri%2C+Henryk+Michalewski%2C+Jacob+Austin%2C+David+Bieber%2C+David+Dohan%2C+Aitor+Lewkowycz%2C+Maarten+Bosma%2C+David+Luan%2C+Charles+Sutton%2C+and+Augustus+Odena%2E+2021%2E+Sh) [our card](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/)
36. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train… [find](https://scholar.google.com/scholar?q=Long+Ouyang%2C+Jeff+Wu%2C+Xu+Jiang%2C+Diogo+Almeida%2C+Carroll+L%2E+Wainwright%2C+Pamela+Mishkin%2C+Chong+Zhang%2C+Sandhini+Agarwal%2C+Katarina+Slama%2C+Alex+Ray%2C+John+Schulman%2C+Jacob+Hilton%2C+Fraser+Kelton%2C+Luke+Miller%2C+)
37. Roma Patel and Ellie Pavlick. 2022. Mapping Language Models to Grounded Conceptual Spaces. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Roma+Patel+and+Ellie+Pavlick%2E+2022%2E+Mapping+Language+Models+to+Grounded+Conceptual+Spaces%2E+In+International+Conference+on+Learning+Representations%2E)
38. Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J. Gershman, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. 2018. Toward a universal decoder of linguistic meaning from brain activation. Nature Communications, 9(1):963. [find](https://scholar.google.com/scholar?q=Francisco+Pereira%2C+Bin+Lou%2C+Brianna+Pritchett%2C+Samuel+Ritter%2C+Samuel+J%2E+Gershman%2C+Nancy+Kanwisher%2C+Matthew+Botvinick%2C+and+Evelina+Fedorenko%2E+2018%2E+Toward+a+universal+decoder+of+linguistic+meaning+from)
39. Steven T. Piantadosi. 2023. Modern language models refute Chomsky’s approach to language. [find](https://scholar.google.com/scholar?q=Steven+T%2E+Piantadosi%2E+2023%2E+Modern+language+models+refute+Chomsky%E2%80%99s+approach+to+language%2E)
40. Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. 2023. GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3845–3864, Dubrovnik, Croatia… [find](https://scholar.google.com/scholar?q=Archiki+Prasad%2C+Peter+Hase%2C+Xiang+Zhou%2C+and+Mohit+Bansal%2E+2023%2E+GrIPS%3A+Gradient%2Dfree%2C+Edit%2Dbased+Instruction+Search+for+Prompting+Large+Language+Models%2E+In+Proceedings+of+the+17th+Conference+of+the+Eu) [our card](https://md.research.elamur.ai/free/en/papers/prasad-2023-grips-gradient-free-edit-based/)
41. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked Language Model Scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Julian+Salazar%2C+Davis+Liang%2C+Toan+Q%2E+Nguyen%2C+and+Katrin+Kirchhoff%2E+2020%2E+Masked+Language+Model+Scoring%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+page) [our card](https://md.research.elamur.ai/free/en/papers/salazar-2020-masked-language-model-scoring/)
42. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. [find](https://scholar.google.com/scholar?q=Miles+Turpin%2C+Julian+Michael%2C+Ethan+Perez%2C+and+Samuel+R%2E+Bowman%2E+2023%2E+Language+Models+Don%E2%80%99t+Always+Say+What+They+Think%3A+Unfaithful+Explanations+in+Chain%2Dof%2DThought+Prompting%2E) [our card](https://md.research.elamur.ai/free/en/papers/turpin-2023-language-models-don-t-always/)
43. Tomer Ullman. 2023. Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks. [find](https://scholar.google.com/scholar?q=Tomer+Ullman%2E+2023%2E+Large+Language+Models+Fail+on+Trivial+Alterations+to+Theory%2Dof%2DMind+Tasks%2E)
44. Paolo Vassallo, Emmanuele Chersoni, Enrico Santus, Alessandro Lenci, and Philippe Blache. 2018. Event Knowledge in Sentence Processing: A New Dataset for the Evaluation of Argument Typicality. In LREC 2018 Workshop on Linguistic and Neurocognitive Resources (LiNCR), Miyazaki, Japan. [find](https://scholar.google.com/scholar?q=Paolo+Vassallo%2C+Emmanuele+Chersoni%2C+Enrico+Santus%2C+Alessandro+Lenci%2C+and+Philippe+Blache%2E+2018%2E+Event+Knowledge+in+Sentence+Processing%3A+A+New+Dataset+for+the+Evaluation+of+Argument+Typicality%2E+In+LREC)
45. Yiwen Wang, Jennifer Hu, Roger Levy, and Peng Qian. 2021. Controlled Evaluation of Grammatical Knowledge in Mandarin Chinese Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5604– 5620, Online and Punta Cana, Dominican Republic. Associ… [find](https://scholar.google.com/scholar?q=Yiwen+Wang%2C+Jennifer+Hu%2C+Roger+Levy%2C+and+Peng+Qian%2E+2021%2E+Controlled+Evaluation+of+Grammatical+Knowledge+in+Mandarin+Chinese+Language+Models%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods)
46. Alex Warstadt and Samuel R. Bowman. 2022. What artificial neural networks can tell us about human language acquisition. In Shalom Lappin and Jean-Philippe Bernardy, editors, Algebraic Structures in Natural Language. Taylor & Francis. [find](https://scholar.google.com/scholar?q=Alex+Warstadt+and+Samuel+R%2E+Bowman%2E+2022%2E+What+artificial+neural+networks+can+tell+us+about+human+language+acquisition%2E+In+Shalom+Lappin+and+Jean%2DPhilippe+Bernardy%2C+editors%2C+Algebraic+Structures+in+Na)
47. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the Association for Computational Linguistics, 8. Publisher: MIT Press. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Alicia+Parrish%2C+Haokun+Liu%2C+Anhad+Mohananey%2C+Wei+Peng%2C+Sheng%2DFu+Wang%2C+and+Samuel+R%2E+Bowman%2E+2020%2E+BLiMP%3A+The+Benchmark+of+Linguistic+Minimal+Pairs+for+English%2E+Transactions+of+the+Assoc)
48. Taylor Webb, Keith J. Holyoak, and Hongjing Lu. 2023. Emergent Analogical Reasoning in Large Language Models. [find](https://scholar.google.com/scholar?q=Taylor+Webb%2C+Keith+J%2E+Holyoak%2C+and+Hongjing+Lu%2E+2023%2E+Emergent+Analogical+Reasoning+in+Large+Language+Models%2E)
49. Albert Webson, Alyssa Marie Loo, Qinan Yu, and Ellie Pavlick. 2023. Are Language Models Worse than Humans at Following Prompts? It’s Complicated. [find](https://scholar.google.com/scholar?q=Albert+Webson%2C+Alyssa+Marie+Loo%2C+Qinan+Yu%2C+and+Ellie+Pavlick%2E+2023%2E+Are+Language+Models+Worse+than+Humans+at+Following+Prompts%3F+It%E2%80%99s+Complicated%2E)
50. Albert Webson and Ellie Pavlick. 2022. Do Prompt-Based Models Really Understand the Meaning of Their Prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2300–2344, Seattle, United States. As… [find](https://scholar.google.com/scholar?q=Albert+Webson+and+Ellie+Pavlick%2E+2022%2E+Do+Prompt%2DBased+Models+Really+Understand+the+Meaning+of+Their+Prompts%3F+In+Proceedings+of+the+2022+Conference+of+the+North+American+Chapter+of+the+Association+for) [our card](https://md.research.elamur.ai/free/en/papers/webson-2022-do-prompt-based-models-really/)
51. Figures 7 to 10 show the relationship between responses measured by the direct and metalinguistic prompting methods, for Experiments 1-3b, respectively. [find](https://scholar.google.com/scholar?q=Figures+7+to+10+show+the+relationship+between+responses+measured+by+the+direct+and+metalinguistic+prompting+methods%2C+for+Experiments+1%2D3b%2C+respectively%2E)