---
title: "On the Planning Abilities of Large Language Models : A Critical"
description: "This NeurIPS 2023 study asks whether large language models can plan, given claims that… 40 references, cited by 105 papers here."
authors: ["Karthik Valmeekam", "Matthew Marquez", "Sarath Sreedharan", "Subbarao Kambhampati"]
year: 2023
venue: "arXiv:2305.15771"
source: "https://arxiv.org/abs/2305.15771"
doi: "10.48550/arxiv.2305.15771"
arxiv: "2305.15771"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/valmeekam-2023-planning-abilities-large-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/valmeekam-2023-planning-abilities-large-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) / On the Planning Abilities of Large Language Models : A Critical Investigation

# On the Planning Abilities of Large Language Models : A Critical Investigation

**Authors:** [Karthik Valmeekam](https://md.research.elamur.ai/free/en/authors/karthik-valmeekam/), [Matthew Marquez](https://md.research.elamur.ai/free/en/authors/matthew-marquez/), [Sarath Sreedharan](https://md.research.elamur.ai/free/en/authors/sarath-sreedharan/) and [Subbarao Kambhampati](https://md.research.elamur.ai/free/en/authors/subbarao-kambhampati/) · **Year:** 2023 · **Venue:** [arXiv:2305.15771](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2023](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2023/)) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2305.15771) · [PDF](https://arxiv.org/pdf/2305.15771) · [DOI 10.48550/arxiv.2305.15771](https://doi.org/10.48550/arxiv.2305.15771) · [arXiv 2305.15771](https://arxiv.org/abs/2305.15771) · [OpenAlex](https://openalex.org/W7133237860)

**Citations:** 86 citations (OpenAlex, 2026-10-14) · [105](https://md.research.elamur.ai/free/en/papers/valmeekam-2023-planning-abilities-large-language-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This NeurIPS 2023 study asks whether large language models can plan, given claims that web-trained LLMs show emergent reasoning. The authors built a benchmark of classical planning problems modeled on International Planning Competition domains such as Blocksworld and Logistics, with domains specified in PDDL-style form. Plans are checked automatically with standard planning tools rather than by human judgment. The GPT family (GPT-4, GPT-3.5, InstructGPT-3.5, InstructGPT-3, GPT-3) was tested in two modes:

* **Autonomous mode:** the model must produce a valid plan on its own. Results were poor. Even GPT-4 solved only about 12% of instances on average, and accuracy dropped further when action and object names were obfuscated. Human participants did clearly better on the simplest domain.
* **LLM-Modulo mode:** model output serves as heuristic input for sound external tools. Here the picture improves. The LPG planner repairs LLM plans more easily than empty or random seed plans, and feedback from the VAL verifier helps GPT-4 fix its plans.

The work argues that LLMs should be paired with verifiers and planners rather than trusted as standalone planners.

## Key points
- With GPT-4, autonomous plan generation succeeded on only about 12% of instances on average across domains. Neither switching models nor fine-tuning changed this picture much.
- Seeding the local-search planner LPG with LLM plans required fewer search steps than seeding it with an empty plan or a random plan.
- Back-prompting GPT-4 with VAL feedback (at most 15 rounds, 50 previously failed instances per domain) gave these results:

  | Domain | Instances fixed | Avg. feedback rounds |
  |---|---|---|
  | Blocksworld | 41/50 (82%) | 3.68 |
  | Logistics | 35/50 (70%) | 3.31 |
  | Mystery BW (disguised names) | 5/50 (10%) | 7.0 |

- In a between-subjects user study (49 unassisted vs. 48 GPT-4-assisted participants), there was no significant difference in accuracy, time or NASA-TLX cognitive load (p = 0.22, 0.92, 0.52). Three assisted participants accepted wrong suggestions, which points to a risk of automation bias.
- The benchmark and code are released at github.com/karthikv792/LLMs-Planning.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan and Subbarao Kambhampati. 2023. On the Planning Abilities of Large Language Models : A Critical Investigation. arXiv:2305.15771.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{valmeekam2023planning,
  title = {{On the Planning Abilities of Large Language Models : A Critical Investigation}},
  author = {Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan and Subbarao Kambhampati},
  year = {2023},
  doi = {10.48550/arxiv.2305.15771},
  eprint = {2305.15771},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.15771}
}
```

- [← previous On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning](https://md.research.elamur.ai/free/en/papers/shaikh-2023-second-thought-let-s-not/)
- [next → Towards Making the Most of ChatGPT for Machine Translation](https://md.research.elamur.ai/free/en/papers/peng-2023-making-most-chatgpt-machine-translation/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/)

## References

40 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Clever Hans. [https://en.wikipedia.org/wiki/Clever_Hans](https://en.wikipedia.org/wiki/Clever_Hans). [↗](https://en.wikipedia.org/wiki/Clever_Hans)
2. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. [Do as i can, not as i say: Grounding language in robotic affordances](https://md.research.elamur.ai/free/en/papers/ahn-2022-i-i-say-grounding-language/). arXiv preprint arXiv:2204.01691, 2022.
3. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
4. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311, 2022.
5. Mary L Cummings. Automation bias in intelligent time critical decision support systems. In Decision making in aviation, pages 289–294. Routledge, 2017. [find](https://scholar.google.com/scholar?q=Mary+L+Cummings%2E+Automation+bias+in+intelligent+time+critical+decision+support+systems%2E+In+Decision+making+in+aviation%2C+pages+289%E2%80%93294%2E+Routledge%2C+2017%2E)
6. Alfonso Gerevini and Ivan Serina. Lpg: A planner based on local search for planning graphs with action costs. In AIPS, volume 2, pages 281–290, 2002. [find](https://scholar.google.com/scholar?q=Alfonso+Gerevini+and+Ivan+Serina%2E+Lpg%3A+A+planner+based+on+local+search+for+planning+graphs+with+action+costs%2E+In+AIPS%2C+volume+2%2C+pages+281%E2%80%93290%2C+2002%2E)
7. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346–361, 2021. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Daniel+Khashabi%2C+Elad+Segal%2C+Tushar+Khot%2C+Dan+Roth%2C+and+Jonathan+Berant%2E+Did+aristotle+use+a+laptop%3F+a+question+answering+benchmark+with+implicit+reasoning+strategies%2E+Transactions+of+the+As)
8. Malik Ghallab, Dana S. Nau, and Paolo Traverso. Automated Planning and Acting. Cambridge University Press, 2016. [find](https://scholar.google.com/scholar?q=Malik+Ghallab%2C+Dana+S%2E+Nau%2C+and+Paolo+Traverso%2E+Automated+Planning+and+Acting%2E+Cambridge+University+Press%2C+2016%2E)
9. Kristian J. Hammond. CHEF: A model of case-based planning. In Tom Kehler, editor, Proceedings of the 5th National Conference on Artificial Intelligence. Philadelphia, PA, USA, August 11-15, 1986. Volume 1: Science, pages 267–271. Morgan Kaufmann, 1986. [find](https://scholar.google.com/scholar?q=Kristian+J%2E+Hammond%2E+CHEF%3A+A+model+of+case%2Dbased+planning%2E+In+Tom+Kehler%2C+editor%2C+Proceedings+of+the+5th+National+Conference+on+Artificial+Intelligence%2E+Philadelphia%2C+PA%2C+USA%2C+August+11%2D15%2C+1986%2E+Volu)
10. Sandra G Hart and Lowell E Staveland. Development of nasa-tlx (task load index): Results of empirical and theoretical research. Advances in psychology, 52:139–183, 1988. [find](https://scholar.google.com/scholar?q=Sandra+G+Hart+and+Lowell+E+Staveland%2E+Development+of+nasa%2Dtlx+%28task+load+index%29%3A+Results+of+empirical+and+theoretical+research%2E+Advances+in+psychology%2C+52%3A139%E2%80%93183%2C+1988%2E)
11. Richard Howey, Derek Long, and Maria Fox. VAL: Automatic plan validation, continuous effects and mixed initiative planning using PDDL. In 16th IEEE International Conference on Tools with Artificial Intelligence, pages 294–301. IEEE, 2004. [find](https://scholar.google.com/scholar?q=Richard+Howey%2C+Derek+Long%2C+and+Maria+Fox%2E+VAL%3A+Automatic+plan+validation%2C+continuous+effects+and+mixed+initiative+planning+using+PDDL%2E+In+16th+IEEE+International+Conference+on+Tools+with+Artificial+In)
12. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. [Language models as zero-shot planners: Extracting actionable knowledge for embodied agents](https://md.research.elamur.ai/free/en/papers/huang-2022-language-models-zero-shot-planners/). In International Conference on Machine Learning, pages 9118–9147. PMLR, 2022.
13. Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022. [find](https://scholar.google.com/scholar?q=Wenlong+Huang%2C+Fei+Xia%2C+Ted+Xiao%2C+Harris+Chan%2C+Jacky+Liang%2C+Pete+Florence%2C+Andy+Zeng%2C+Jonathan+Tompson%2C+Igor+Mordatch%2C+Yevgen+Chebotar%2C+et+al%2E+Inner+monologue%3A+Embodied+reasoning+through+planning+with)
14. IPC. International planning competition, 1998. [find](https://scholar.google.com/scholar?q=IPC%2E+International+planning+competition%2C+1998%2E)
15. Liwei Jiang, Jena D. Hwang, Chandrasekhar Bhagavatula, Ronan Le Bras, Maxwell Forbes, Jon Borchardt, Jenny Liang, Oren Etzioni, Maarten Sap, and Yejin Choi. Delphi: Towards Machine Ethics and Norms. ArXiv, abs/2110.[07574, 2021](https://md.research.elamur.ai/free/en/papers/jiang-2021-machines-learn-morality-delphi-experiment/).
16. Subbarao Kambhampati. AI as (an Ersatz) Natural Science? [https://cacm.acm.org/blogs/blog-cacm/261732-ai-as-an-ersatz-natural-science/fulltext](https://cacm.acm.org/blogs/blog-cacm/261732-ai-as-an-ersatz-natural-science/fulltext), Jun 2022. [↗](https://cacm.acm.org/blogs/blog-cacm/261732-ai-as-an-ersatz-natural-science/fulltext)
17. Subbarao Kambhampati and James A. Hendler. A validation-structure-based theory of plan modification and reuse. Artif. Intell., 55(2):193–258, 1992. [find](https://scholar.google.com/scholar?q=Subbarao+Kambhampati+and+James+A%2E+Hendler%2E+A+validation%2Dstructure%2Dbased+theory+of+plan+modification+and+reuse%2E+Artif%2E+Intell%2E%2C+55%282%29%3A193%E2%80%93258%2C+1992%2E)
18. Subbarao Kambhampati, Karthik Valmeekam, Matthew Marquez, and Lin Guan. On the role of large language models in planning, July 2023. Tutorial presented at the International Conference on Automated Planning and Scheduling (ICAPS), Prague. [https://yochan-lab.github.io/tutorial/ICAPS-2023/](https://yochan-lab.github.io/tutorial/ICAPS-2023/). [↗](https://yochan-lab.github.io/tutorial/ICAPS-2023/)
19. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. [Large Language Models are Zero-Shot Reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). arXiv preprint arXiv:2205.11916, 2022.
20. Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. [Towards understanding and mitigating social biases in language models](https://md.research.elamur.ai/free/en/papers/liang-2021-towards-understanding-mitigating-social-biases/). In International Conference on Machine Learning, pages 6565–6576. PMLR, 2021.
21. Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.[11477, 2023](https://md.research.elamur.ai/free/en/papers/liu-2023-llm-p-empowering-large-language/).
22. Drew McDermott, Malik Ghallab, Adele E. Howe, Craig A. Knoblock, Ashwin Ram, Manuela M. Veloso, Daniel S. Weld, and David E. Wilkins. Pddl-the planning domain definition language. 1998. [find](https://scholar.google.com/scholar?q=Drew+McDermott%2C+Malik+Ghallab%2C+Adele+E%2E+Howe%2C+Craig+A%2E+Knoblock%2C+Ashwin+Ram%2C+Manuela+M%2E+Veloso%2C+Daniel+S%2E+Weld%2C+and+David+E%2E+Wilkins%2E+Pddl%2Dthe+planning+domain+definition+language%2E+1998%2E)
23. Tuan Anh Nguyen, Minh Do, Alfonso Emilio Gerevini, Ivan Serina, Biplav Srivastava, and Subbarao Kambhampati. Generating diverse plans to handle unknown and partially known user preferences. Artificial Intelligence, 190:1–31, 2012. [find](https://scholar.google.com/scholar?q=Tuan+Anh+Nguyen%2C+Minh+Do%2C+Alfonso+Emilio+Gerevini%2C+Ivan+Serina%2C+Biplav+Srivastava%2C+and+Subbarao+Kambhampati%2E+Generating+diverse+plans+to+handle+unknown+and+partially+known+user+preferences%2E+Artificial)
24. OpenAI. Introducing chatgpt by openai, 2022. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Introducing+chatgpt+by+openai%2C+2022%2E)
25. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023.
26. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). arXiv preprint arXiv:2203.02155, 2022.
27. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+2018%2E)
28. Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex. Planning with large language models via corrective re-prompting. arXiv preprint arXiv:2211.09935, 2022. — [arXiv:2211.09935](https://arxiv.org/abs/2211.09935) [find](https://scholar.google.com/scholar?q=Shreyas+Sundara+Raman%2C+Vanya+Cohen%2C+Eric+Rosen%2C+Ifrah+Idrees%2C+David+Paulius%2C+and+Stefanie+Tellex%2E+Planning+with+large+language+models+via+corrective+re%2Dprompting%2E+arXiv+preprint+arXiv%3A2211%2E09935%2C+2022)
29. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+)
30. Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A Rothkopf, and Kristian Kersting. Large pre-trained language models contain human-like biases of what is right and wrong to do. Nature Machine Intelligence, 4(3):258–268, 2022. [find](https://scholar.google.com/scholar?q=Patrick+Schramowski%2C+Cigdem+Turan%2C+Nico+Andersen%2C+Constantin+A+Rothkopf%2C+and+Kristian+Kersting%2E+Large+pre%2Dtrained+language+models+contain+human%2Dlike+biases+of+what+is+right+and+wrong+to+do%2E+Nature+Mac)
31. Big Science. BigScience Large Open-science Open-access Multilingual Language Model. [https://huggingface.co/bigscience/bloom](https://huggingface.co/bigscience/bloom), 2022. [↗](https://huggingface.co/bigscience/bloom)
32. Tom Silver, Varun Hariprasad, Reece S Shuttleworth, Nishanth Kumar, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. PDDL planning with pretrained large language models. In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022. [find](https://scholar.google.com/scholar?q=Tom+Silver%2C+Varun+Hariprasad%2C+Reece+S+Shuttleworth%2C+Nishanth+Kumar%2C+Tom%C3%A1s+Lozano%2DP%C3%A9rez%2C+and+Leslie+Pack+Kaelbling%2E+PDDL+planning+with+pretrained+large+language+models%2E+In+NeurIPS+2022+Foundation+Model)
33. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). arXiv preprint arXiv:2206.04615, 2022.
34. Kaya Stechly, Matthew Marquez, and Subbarao Kambhampati. Gpt-4 doesn’t know it’s wrong: An analysis of iterative prompting for reasoning problems. arXiv preprint arXiv:2310.12397, 2023. — [arXiv:2310.12397](https://arxiv.org/abs/2310.12397) [find](https://scholar.google.com/scholar?q=Kaya+Stechly%2C+Matthew+Marquez%2C+and+Subbarao+Kambhampati%2E+Gpt%2D4+doesn%E2%80%99t+know+it%E2%80%99s+wrong%3A+An+analysis+of+iterative+prompting+for+reasoning+problems%2E+arXiv+preprint+arXiv%3A2310%2E12397%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E)
35. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. [Commonsenseqa: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). arXiv preprint arXiv:1811.00937, 2018.
36. Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati. Can large language models really improve by self-critiquing their own plans? arXiv preprint arXiv:2310.08118, 2023. — [arXiv:2310.08118](https://arxiv.org/abs/2310.08118) [find](https://scholar.google.com/scholar?q=Karthik+Valmeekam%2C+Matthew+Marquez%2C+and+Subbarao+Kambhampati%2E+Can+large+language+models+really+improve+by+self%2Dcritiquing+their+own+plans%3F+arXiv+preprint+arXiv%3A2310%2E08118%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E08118%5D%28h)
37. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.[11903, 2022](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/).
38. Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. Translating natural language to planning goals with large-language models. arXiv preprint arXiv:2302.05128, 2023. — [arXiv:2302.05128](https://arxiv.org/abs/2302.05128) [find](https://scholar.google.com/scholar?q=Yaqi+Xie%2C+Chen+Yu%2C+Tongyao+Zhu%2C+Jinbin+Bai%2C+Ze+Gong%2C+and+Harold+Soh%2E+Translating+natural+language+to+planning+goals+with+large%2Dlanguage+models%2E+arXiv+preprint+arXiv%3A2302%2E05128%2C+2023%2E+%E2%80%94+%5BarXiv%3A2302%2E051)
39. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. [React: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). In The Eleventh International Conference on Learning Representations, 2023.
40. Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, and Guy Van den Broeck. On the paradox of learning to reason from data. arXiv preprint arXiv:2205.11502, 2022. — [arXiv:2205.11502](https://arxiv.org/abs/2205.11502) [find](https://scholar.google.com/scholar?q=Honghua+Zhang%2C+Liunian+Harold+Li%2C+Tao+Meng%2C+Kai%2DWei+Chang%2C+and+Guy+Van+den+Broeck%2E+On+the+paradox+of+learning+to+reason+from+data%2E+arXiv+preprint+arXiv%3A2205%2E11502%2C+2022%2E+%E2%80%94+%5BarXiv%3A2205%2E11502%5D%28https%3A%2F%2Fa)

*Updated 2 November 2026*