---
title: "On the Self-Verification Limitations of Large Language Models on"
description: "The authors argue for LLM-Modulo designs, where the LLM generates candidates and trusted… 55 references, cited by 39 papers here."
authors: ["Kaya Stechly", "Karthik Valmeekam", "Subbarao Kambhampati"]
year: 2024
venue: "arXiv:2402.08115"
source: "https://arxiv.org/abs/2402.08115"
doi: "10.48550/arxiv.2402.08115"
arxiv: "2402.08115"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/stechly-2024-self-verification-limitations-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/stechly-2024-self-verification-limitations-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/) / On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks

# On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks

**Authors:** [Kaya Stechly](https://md.research.elamur.ai/free/en/authors/kaya-stechly/), [Karthik Valmeekam](https://md.research.elamur.ai/free/en/authors/karthik-valmeekam/) and [Subbarao Kambhampati](https://md.research.elamur.ai/free/en/authors/subbarao-kambhampati/) · **Year:** 2024 · **Venue:** arXiv:2402.08115 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.08115) · [PDF](https://arxiv.org/pdf/2402.08115) · [DOI 10.48550/arxiv.2402.08115](https://doi.org/10.48550/arxiv.2402.08115) · [arXiv 2402.08115](https://arxiv.org/abs/2402.08115) · [OpenAlex](https://openalex.org/W4391833261)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [39](https://md.research.elamur.ai/free/en/papers/stechly-2024-self-verification-limitations-large-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Many researchers hold that LLMs, though weak reasoners, can still fix their own mistakes by repeatedly checking and revising their answers, because checking is supposedly easier than solving. Stechly, Valmeekam and Kambhampati test this with GPT-4 on three tasks whose answers can be formally checked: STRIPS planning, Graph Coloring and the Game of 24. They compare an LLM+LLM loop, where the model both proposes and critiques solutions, with an LLM+Sound Verifier setup. They then strip the second setup down in stages, first varying how detailed the feedback is and finally removing feedback entirely, so the same prompt is resent until a correct answer appears. Self-critique proves harmful. The model often rejects valid solutions and writes misleading critiques, so accuracy can drop below its first attempt. External verification helps substantially, but feedback detail barely matters. Plain resampling with a sound checker keeps most of the gains at much lower token cost. The authors argue for LLM-Modulo designs, where the LLM generates candidates and trusted external tools verify them.

## Key points
- Domains: STRIPS planning (including the obfuscated Mystery Blocksworld variant), Graph Coloring and the Game of 24. All have machine-checkable ground truth for both answers and critiques.
- In the LLM+LLM setting, GPT-4 has a high false-negative rate as a verifier. It also hallucinates constraint violations and action preconditions, so performance degrades as backprompting continues.
- With a sound verifier, the authors tested binary, first-error and all-errors feedback (only the first two for Game of 24). The three levels gave similar results, and in two domains richer feedback slightly lowered accuracy.
- Resending the identical prompt for up to 15 rounds with a sound verifier matched the feedback-based gains. Taking a self-consistency majority vote over 15 samples gave no improvement over standard prompting.
- Recommended verifiers include VAL for STRIPS plans, expression evaluation for Game of 24, and constraint checkers for CSPs. The work fits within the LLM-Modulo framework.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Arizona State University](https://md.research.elamur.ai/free/en/universities/arizona-state-university/) — University, Tempe, United States · 58 works · 5,334 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,948 works

## Cite

Cite as: Kaya Stechly, Karthik Valmeekam and Subbarao Kambhampati. 2024. On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks. arXiv:2402.08115.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{stechly2024self,
  title = {{On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks}},
  author = {Kaya Stechly, Karthik Valmeekam and Subbarao Kambhampati},
  year = {2024},
  doi = {10.48550/arxiv.2402.08115},
  eprint = {2402.08115},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.08115}
}
```

- [← previous LUQ: Long-text Uncertainty Quantification for LLMs](https://md.research.elamur.ai/free/en/papers/zhang-2024-luq-long-text-uncertainty-quantification/)
- [next → Programming Refusal with Conditional Activation Steering](https://md.research.elamur.ai/free/en/papers/lee-2024-programming-refusal-conditional-activation-steering/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/)

## References

55 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah I Abdin, Suriya Gunasekar, Varun Chandrasekaran, Jerry Li, Mert Yuksekgonul, Rahee Ghosh Peshawaria, Ranjita Naik, and Besmira Nushi. Kitab: Evaluating llms on constraint satisfaction for information retrieval. arXiv preprint arXiv:2310.15511, 2023. — [arXiv:2310.15511](https://arxiv.org/abs/2310.15511) [find](https://scholar.google.com/scholar?q=Marah+I+Abdin%2C+Suriya+Gunasekar%2C+Varun+Chandrasekaran%2C+Jerry+Li%2C+Mert+Yuksekgonul%2C+Rahee+Ghosh+Peshawaria%2C+Ranjita+Naik%2C+and+Besmira+Nushi%2E+Kitab%3A+Evaluating+llms+on+constraint+satisfaction+for+inform)
2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
3. Shushan Arakelyan, Rocktim Jyoti Das, Yi Mao, and Xiang Ren. Exploring distributional shifts in large language models for code analysis. arXiv preprint arXiv:2303.09128, 2023. [find](https://scholar.google.com/scholar?q=Shushan+Arakelyan%2C+Rocktim+Jyoti+Das%2C+Yi+Mao%2C+and+Xiang+Ren%2E+Exploring+distributional+shifts+in+large+language+models+for+code+analysis%2E+arXiv+preprint+arXiv%3A2303%2E09128%2C+2023%2E)
4. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712, 2023.
5. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023. [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Maxwell+Lin%2C+Nathanael+Sch%C3%A4rli%2C+and+Denny+Zhou%2E+Teaching+large+language+models+to+self%2Ddebug%2E+arXiv+preprint+arXiv%3A2304%2E05128%2C+2023%2E)
6. Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. [Lm vs lm: Detecting factual errors via cross examination](https://md.research.elamur.ai/free/en/papers/cohen-2023-lm-vs-lm-detecting-factual/). arXiv preprint arXiv:2305.13281, 2023.
7. Joost CF de Winter. Can chatgpt pass high school exams on english language comprehension? International Journal of Artificial Intelligence in Education, pages 1–16, 2023. [find](https://scholar.google.com/scholar?q=Joost+CF+de+Winter%2E+Can+chatgpt+pass+high+school+exams+on+english+language+comprehension%3F+International+Journal+of+Artificial+Intelligence+in+Education%2C+pages+1%E2%80%9316%2C+2023%2E)
8. DIMACS. DIMACS Implementation Challenges. Archive available at [http://archive.dimacs.rutgers.edu/Challenges/](http://archive.dimacs.rutgers.edu/Challenges/). [↗](http://archive.dimacs.rutgers.edu/Challenges/)
9. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. [A survey on in-context learning](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/). arXiv preprint arXiv:2301.00234, 2022.
10. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. [Improving factuality and reasoning in language models through multiagent debate](https://md.research.elamur.ai/free/en/papers/du-2023-improving-factuality-reasoning-language-models/). arXiv preprint arXiv:2305.14325, 2023.
11. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. [Faith and fate: Limits of transformers on compositionality](https://md.research.elamur.ai/free/en/papers/dziri-2023-faith-fate-limits-transformers-compositionality/).…
12. Gaël Gendron, Qiming Bao, Michael Witbrock, and Gillian Dobbie. Large language models are not abstract reasoners. arXiv preprint arXiv:2305.[19555, 2023](https://md.research.elamur.ai/free/en/papers/gendron-2023-large-language-models-strong-abstract/).
13. Aidan Gilson, Conrad W Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, David Chartash, et al. How does chatgpt perform on the united states medical licensing examination (usmle)? the implications of large language models for medical education and knowledge assessment. JMIR m… [find](https://scholar.google.com/scholar?q=Aidan+Gilson%2C+Conrad+W+Safranek%2C+Thomas+Huang%2C+Vimig+Socrates%2C+Ling+Chi%2C+Richard+Andrew+Taylor%2C+David+Chartash%2C+et+al%2E+How+does+chatgpt+perform+on+the+united+states+medical+licensing+examination+%28usml)
14. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. [Critic: Large language models can self-correct with tool-interactive critiquing](https://md.research.elamur.ai/free/en/papers/gou-2023-critic-large-language-models-self/). arXiv preprint arXiv:2305.11738, 2023.
15. Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning with Language Model is Planning with World Model. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8154–8173,… [find](https://scholar.google.com/scholar?q=Shibo+Hao%2C+Yi+Gu%2C+Haodi+Ma%2C+Joshua+Hong%2C+Zhen+Wang%2C+Daisy+Wang%2C+and+Zhiting+Hu%2E+Reasoning+with+Language+Model+is+Planning+with+World+Model%2E+In+Houda+Bouamor%2C+Juan+Pino%2C+and+Kalika+Bali%2C+editors%2C+Proce)
16. Ruixin Hong, Hongming Zhang, Xinyu Pang, Dong Yu, and Changshui Zhang. A closer look at the self-verification abilities of large language models in logical reasoning. arXiv preprint arXiv:2311.07954, 2023. [find](https://scholar.google.com/scholar?q=Ruixin+Hong%2C+Hongming+Zhang%2C+Xinyu+Pang%2C+Dong+Yu%2C+and+Changshui+Zhang%2E+A+closer+look+at+the+self%2Dverification+abilities+of+large+language+models+in+logical+reasoning%2E+arXiv+preprint+arXiv%3A2311%2E07954%2C+)
17. Richard Howey, Derek Long, and Maria Fox. VAL: Automatic plan validation, continuous effects and mixed initiative planning using PDDL. In 16th IEEE International Conference on Tools with Artificial Intelligence, pages 294–301. IEEE, 2004. [find](https://scholar.google.com/scholar?q=Richard+Howey%2C+Derek+Long%2C+and+Maria+Fox%2E+VAL%3A+Automatic+plan+validation%2C+continuous+effects+and+mixed+initiative+planning+using+PDDL%2E+In+16th+IEEE+International+Conference+on+Tools+with+Artificial+In)
18. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. [Large language models cannot self-correct reasoning yet](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-cannot-self/). arXiv preprint arXiv:2310.01798, 2023.
19. Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022. [find](https://scholar.google.com/scholar?q=Wenlong+Huang%2C+Fei+Xia%2C+Ted+Xiao%2C+Harris+Chan%2C+Jacky+Liang%2C+Pete+Florence%2C+Andy+Zeng%2C+Jonathan+Tompson%2C+Igor+Mordatch%2C+Yevgen+Chebotar%2C+et+al%2E+Inner+monologue%3A+Embodied+reasoning+through+planning+with)
20. IPC. International planning competition, 1998. [find](https://scholar.google.com/scholar?q=IPC%2E+International+planning+competition%2C+1998%2E)
21. Weisen Jiang, Han Shi, Longhui Yu, Zhengying Liu, Yu Zhang, Zhenguo Li, and James T Kwok. Backward reasoning in large language models for verification. arXiv preprint arXiv:2308.[07758, 2023](https://md.research.elamur.ai/free/en/papers/jiang-2024-forward-backward-reasoning-large-language/).
22. Subbarao Kambhampati. Can large language models reason and plan? Annals of the New York Academy of Sciences, 2024. [find](https://scholar.google.com/scholar?q=Subbarao+Kambhampati%2E+Can+large+language+models+reason+and+plan%3F+Annals+of+the+New+York+Academy+of+Sciences%2C+2024%2E)
23. Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Kaya Stechly, Mudit Verma, Siddhant Bhambri, Lucas Saldyt, and Anil Murthy. Llms can’t plan, but can help planning in llm-modulo frameworks. arXiv preprint arXiv:2402.[01817, 2024](https://md.research.elamur.ai/free/en/papers/kambhampati-2024-llms-t-plan-but-help/).
24. Subbarao Kambhampati, Karthik Valmeekam, Matthew Marquez, and Lin Guan. On the role of large language models in planning. tutorial presented at the international conference on automated planning and scheduling (icaps), prague, July 2023. [find](https://scholar.google.com/scholar?q=Subbarao+Kambhampati%2C+Karthik+Valmeekam%2C+Matthew+Marquez%2C+and+Lin+Guan%2E+On+the+role+of+large+language+models+in+planning%2E+tutorial+presented+at+the+international+conference+on+automated+planning+and+s)
25. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). Advances in neural information processing systems, 35:22199–22213, 2022.
26. Gerd Kortemeyer. Could an artificial-intelligence agent pass an introductory physics course? Physical Review Physics Education Research, 19(1):010132, 2023. [find](https://scholar.google.com/scholar?q=Gerd+Kortemeyer%2E+Could+an+artificial%2Dintelligence+agent+pass+an+introductory+physics+course%3F+Physical+Review+Physics+Education+Research%2C+19%281%29%3A010132%2C+2023%2E)
27. Jacqueline P Leighton and Robert J Sternberg. The nature of reasoning. Cambridge University Press, 2004. [find](https://scholar.google.com/scholar?q=Jacqueline+P+Leighton+and+Robert+J+Sternberg%2E+The+nature+of+reasoning%2E+Cambridge+University+Press%2C+2004%2E)
28. Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Commongen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1823–1840, 2020. [find](https://scholar.google.com/scholar?q=Bill+Yuchen+Lin%2C+Wangchunshu+Zhou%2C+Ming+Shen%2C+Pei+Zhou%2C+Chandra+Bhagavatula%2C+Yejin+Choi%2C+and+Xiang+Ren%2E+Commongen%3A+A+constrained+text+generation+challenge+for+generative+commonsense+reasoning%2E+In+Find)
29. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+et+al%2E+Self%2Drefine%3A+Iterative+refinement+with+self%2Dfeedbac)
30. R Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L Griffiths. Embers of autoregression: Understanding large language models through the problem they are trained to solve. arXiv preprint arXiv:2309.13638, 2023. [find](https://scholar.google.com/scholar?q=R+Thomas+McCoy%2C+Shunyu+Yao%2C+Dan+Friedman%2C+Matthew+Hardy%2C+and+Thomas+L+Griffiths%2E+Embers+of+autoregression%3A+Understanding+large+language+models+through+the+problem+they+are+trained+to+solve%2E+arXiv+prep)
31. Drew McDermott, Malik Ghallab, Adele E. Howe, Craig A. Knoblock, Ashwin Ram, Manuela M. Veloso, Daniel S. Weld, and David E. Wilkins. Pddl-the planning domain definition language. 1998. [find](https://scholar.google.com/scholar?q=Drew+McDermott%2C+Malik+Ghallab%2C+Adele+E%2E+Howe%2C+Craig+A%2E+Knoblock%2C+Ashwin+Ram%2C+Manuela+M%2E+Veloso%2C+Daniel+S%2E+Weld%2C+and+David+E%2E+Wilkins%2E+Pddl%2Dthe+planning+domain+definition+language%2E+1998%2E)
32. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. [find](https://scholar.google.com/scholar?q=Jack+W+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+et+al%2E+Scaling+language+models%3A+Methods%2C+analysi)
33. Raffaele Raimondi, Nikolaos Tzoumas, Thomas Salisbury, Sandro Di Simplicio, and Mario R Romano. Comparative analysis of large language models in the royal college of ophthalmologists fellowship exams. Eye, 37(17):3530–3533, 2023. [find](https://scholar.google.com/scholar?q=Raffaele+Raimondi%2C+Nikolaos+Tzoumas%2C+Thomas+Salisbury%2C+Sandro+Di+Simplicio%2C+and+Mario+R+Romano%2E+Comparative+analysis+of+large+language+models+in+the+royal+college+of+ophthalmologists+fellowship+exams%2E)
34. Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. Data contamination through the lens of time. arXiv preprint arXiv:2310.10628, 2023. [find](https://scholar.google.com/scholar?q=Manley+Roberts%2C+Himanshu+Thakur%2C+Christine+Herlihy%2C+Colin+White%2C+and+Samuel+Dooley%2E+Data+contamination+through+the+lens+of+time%2E+arXiv+preprint+arXiv%3A2310%2E10628%2C+2023%2E)
35. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. [Mathematical discoveries from program search with large language models](https://md.research.elamur.ai/free/en/papers/paredes-2023-mathematical-discoveries-program-search-large/). Nature, pages 1–3, 2023.
36. Michael Saxon. Benchmarks as Microscopes: A Call for Model Metrology, 2024. [find](https://scholar.google.com/scholar?q=Michael+Saxon%2E+Benchmarks+as+Microscopes%3A+A+Call+for+Model+Metrology%2C+2024%2E)
37. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. [Reflexion: Language agents with verbal reinforcement learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/). In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
38. Tom Silver, Varun Hariprasad, Reece S Shuttleworth, Nishanth Kumar, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. Pddl planning with pretrained large language models. In NeurIPS 2022 foundation models for decision making workshop, 2022. [find](https://scholar.google.com/scholar?q=Tom+Silver%2C+Varun+Hariprasad%2C+Reece+S+Shuttleworth%2C+Nishanth+Kumar%2C+Tom%C3%A1s+Lozano%2DP%C3%A9rez%2C+and+Leslie+Pack+Kaelbling%2E+Pddl+planning+with+pretrained+large+language+models%2E+In+NeurIPS+2022+foundation+model)
39. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). Transactions on Machine Learning Resear…
40. Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. Chain of thoughtlessness: An analysis of cot in planning. arXiv preprint arXiv:2405.04776, 2024. — [arXiv:2405.04776](https://arxiv.org/abs/2405.04776) [find](https://scholar.google.com/scholar?q=Kaya+Stechly%2C+Karthik+Valmeekam%2C+and+Subbarao+Kambhampati%2E+Chain+of+thoughtlessness%3A+An+analysis+of+cot+in+planning%2E+arXiv+preprint+arXiv%3A2405%2E04776%2C+2024%2E+%E2%80%94+%5BarXiv%3A2405%2E04776%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
41. Nikhil G Thaker, Navid Redjal, Arturo Loaiza-Bonilla, David Penberthy, Tim Showalter, Ajay Choudhri, Shirnett Williamson, Gautam Thaker, Chirag Shah, Matthew C Ward, et al. Large language models encode radiation oncology domain knowledge: Performance on the american college of radiology standardized… [find](https://scholar.google.com/scholar?q=Nikhil+G+Thaker%2C+Navid+Redjal%2C+Arturo+Loaiza%2DBonilla%2C+David+Penberthy%2C+Tim+Showalter%2C+Ajay+Choudhri%2C+Shirnett+Williamson%2C+Gautam+Thaker%2C+Chirag+Shah%2C+Matthew+C+Ward%2C+et+al%2E+Large+language+models+encod)
42. Tomer Ullman. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399, 2023. [find](https://scholar.google.com/scholar?q=Tomer+Ullman%2E+Large+language+models+fail+on+trivial+alterations+to+theory%2Dof%2Dmind+tasks%2E+arXiv+preprint+arXiv%3A2302%2E08399%2C+2023%2E)
43. Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. [Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change](https://md.research.elamur.ai/free/en/papers/valmeekam-2022-planbench-extensible-benchmark-evaluating-large/). In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks…
44. Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models - a critical investigation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. — [On the Planning Abilities of Large Language Models : A Critical Investigation](https://md.research.elamur.ai/free/en/papers/valmeekam-2023-planning-abilities-large-language-models/)
45. Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. [Scibench: Evaluating college-level scientific problem-solving abilities of large language models](https://md.research.elamur.ai/free/en/papers/wang-2023-scibench-evaluating-college-level-scientific/). arXiv preprint arXiv:2307.10635, 2023.
46. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171, 2022.
47. Taylor Webb, Keith J Holyoak, and Hongjing Lu. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7(9):1526–1541, 2023. [find](https://scholar.google.com/scholar?q=Taylor+Webb%2C+Keith+J+Holyoak%2C+and+Hongjing+Lu%2E+Emergent+analogical+reasoning+in+large+language+models%2E+Nature+Human+Behaviour%2C+7%289%29%3A1526%E2%80%931541%2C+2023%2E)
48. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824–24837, 2022.
49. Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. [Generating sequences by learning to self-correct](https://md.research.elamur.ai/free/en/papers/welleck-2022-generating-sequences-learning-self-correct/). In The Eleventh International Conference on Learning Representations, 2022.
50. Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. [Large language models are better reasoners with self-verification](https://md.research.elamur.ai/free/en/papers/weng-2023-large-language-models-are-better/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2550–2575, 2023.
51. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. [Hotpotqa: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380,…
52. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L%2E+Griffiths%2C+Yuan+Cao%2C+and+Karthik+R+Narasimhan%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+In+Thirty%2Dseventh+Co)
53. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. [React: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). arXiv preprint arXiv:2210.03629, 2022.
54. Will Yeadon and Douglas P Halliday. Exploring durham university physics exams with large language models. arXiv preprint arXiv:2306.15609, 2023. — [arXiv:2306.15609](https://arxiv.org/abs/2306.15609) [find](https://scholar.google.com/scholar?q=Will+Yeadon+and+Douglas+P+Halliday%2E+Exploring+durham+university+physics+exams+with+large+language+models%2E+arXiv+preprint+arXiv%3A2306%2E15609%2C+2023%2E+%E2%80%94+%5BarXiv%3A2306%2E15609%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2306%2E15609%29)
55. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/). arXiv preprint arXiv:2205.10625, 2022.

*Updated 3 November 2026*