---
title: "Q*: Improving Multi-step Reasoning for LLMs with Deliberative"
description: "The authors propose Q*. It treats reasoning as a Markov Decision Process: a state is the question… 52 references, cited by 44 papers here."
authors: ["Chaojie Wang", "Yanchen Deng", "Zhiyi Lyu", "Liang Zeng", "Jujie He", "Shuicheng Yan", "Bo An"]
year: 2024
venue: "arXiv:2406.14283"
source: "https://arxiv.org/abs/2406.14283"
doi: "10.48550/arxiv.2406.14283"
arxiv: "2406.14283"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2024-q-improving-multi-step-reasoning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2024-q-improving-multi-step-reasoning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/) / Q\*: Improving Multi-step Reasoning for LLMs with Deliberative Planning

# Q\*: Improving Multi-step Reasoning for LLMs with Deliberative Planning

**Authors:** [Chaojie Wang](https://md.research.elamur.ai/free/en/authors/chaojie-wang/), Yanchen Deng, Zhiyi Lyu, [Liang Zeng](https://md.research.elamur.ai/free/en/authors/liang-zeng/), [Jujie He](https://md.research.elamur.ai/free/en/authors/jujie-he/), [Shuicheng Yan](https://md.research.elamur.ai/free/en/authors/shuicheng-yan/) and [Bo An](https://md.research.elamur.ai/free/en/authors/bo-an/) · **Year:** 2024 · **Venue:** arXiv:2406.14283 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2406.14283) · [PDF](https://arxiv.org/pdf/2406.14283) · [DOI 10.48550/arxiv.2406.14283](https://doi.org/10.48550/arxiv.2406.14283) · [arXiv 2406.14283](https://arxiv.org/abs/2406.14283) · [OpenAlex](https://openalex.org/W4399911690)

**Citations:** 9 citations (OpenAlex, 2026-10-14) · [44](https://md.research.elamur.ai/free/en/papers/wang-2024-q-improving-multi-step-reasoning/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Because LLMs generate text token by token, an early mistake in a multi-step solution tends to carry through and spoil the final answer. Existing tree-search remedies like MCTS or A* variants often need hand-crafted, task-specific utility functions or many expensive rollouts. The authors propose Q*. It treats reasoning as a Markov Decision Process: a state is the question plus the steps written so far, and an action is the next step. Decoding then becomes a best-first, A*-style search. Each partial trace is scored by combining an accumulated utility with a heuristic. That heuristic comes from a separately trained, plug-and-play Q-value model (QVM), which predicts the chance of reaching a correct answer. Training labels for the QVM require only ground-truth answers. The base LLM is never fine-tuned, and each search step expands just one reasoning step rather than a full simulation. On GSM8K, MATH and MBPP, Q* beat Best-of-N reranking and PPO alignment that used the same reward or value models. This suggests a cheap, general way to strengthen open-source 7B models.

## Key points
- The search score is f = g + λh. Here g aggregates process-level utility, h is the QVM estimate, and λ balances the two terms. Q-value labels come from one of three sources: fitted-Q-iteration (offline RL), the best of several rollouts, or completions by a stronger LLM.
- **GSM8K:** Llama-2-7b fine-tuned on MetaMath starts at 65.2%. Best-of-N with the QVM reaches 74.5%, and Q* with the QVM alone reaches 78.8%. Q* combining a PRM trained on PRM800K with the QVM reaches 80.8%, which exceeds ChatGPT-turbo.
- **MATH:** Llama-2-7b trained on Synthetic Data improves from 41.9% to 49.1%, against 46.8% for Best-of-N. DeepSeek-Math-7b improves from 50.8% to 55.4%, against 54.3% for Best-of-N. That result is above Gemini Ultra (4-shot) at 53.2%.
- **MBPP:** CodeQwen1.5-7b-Chat improves from 74.6% to 77.0%, against 75.0% for Best-of-N. Its utility function penalizes code that triggers a TokenError from `tokenize.generate_tokens` by -0.5.
- On GSM8K, using a given PRM or QVM to rerank candidates worked noticeably better than using it as the reward for PPO alignment.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Nanyang Technological University](https://md.research.elamur.ai/free/en/universities/nanyang-technological-university/) — University, Singapore, Singapore · 184 works · 27,484 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Chaojie Wang, Yanchen Deng, Zhiyi Lyu, Liang Zeng, Jujie He, Shuicheng Yan and Bo An. 2024. Q\*: Improving Multi-step Reasoning for LLMs with Deliberative Planning. arXiv:2406.14283.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wang2024q,
  title = {{Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planning}},
  author = {Chaojie Wang, Yanchen Deng, Zhiyi Lyu, Liang Zeng, Jujie He, Shuicheng Yan and Bo An},
  year = {2024},
  doi = {10.48550/arxiv.2406.14283},
  eprint = {2406.14283},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.14283}
}
```

- [← previous IndoCulture: Exploring Geographically Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provinces](https://md.research.elamur.ai/free/en/papers/koto-2024-indoculture-exploring-geographically-influenced-cultural/)
- [next → RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style](https://md.research.elamur.ai/free/en/papers/liu-2024-rm-bench-benchmarking-reward-models/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/)

## References

52 references, 33 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. [Large language models for mathematical reasoning: Progresses and challenges](https://md.research.elamur.ai/free/en/papers/ahn-2024-large-language-models-mathematical-reasoning/). arXiv preprint arXiv:2402.00157, 2024.
2. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
3. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874, 2021.
4. Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. Math-shepherd: A label-free step-by-step verifier for LLMs in mathematical reasoning. arXiv preprint arXiv:2312.[08935, 2023](https://md.research.elamur.ai/free/en/papers/wang-2024-math-shepherd-verify-reinforce-llms/).
5. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). arXiv preprint arXiv:2309.12284, 2023.
6. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. [Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/). arXiv preprint arXiv:2308.09583, 2023.
7. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568, 2023. [find](https://scholar.google.com/scholar?q=Ziyang+Luo%2C+Can+Xu%2C+Pu+Zhao%2C+Qingfeng+Sun%2C+Xiubo+Geng%2C+Wenxiang+Hu%2C+Chongyang+Tao%2C+Jing+Ma%2C+Qingwei+Lin%2C+and+Daxin+Jiang%2E+Wizardcoder%3A+Empowering+code+large+language+models+with+evol%2Dinstruct%2E+arXiv+p)
8. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950, 2023.
9. CodeGemma Team, Ale Jakse Hartman, Andrea Hu, Christopher A. Choquette-Choo, Heri Zhao, Jane Fine, Jeffrey Hui, Jingyue Shen, et al. [Codegemma: Open code models based on gemma](https://md.research.elamur.ai/free/en/papers/zhao-2024-codegemma-open-code-models-based/). 2024. URL [https://goo.gle/codegemma](https://goo.gle/codegemma). [↗](https://goo.gle/codegemma)
10. Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. [TravelPlanner: A benchmark for real-world planning with language agents](https://md.research.elamur.ai/free/en/papers/xie-2024-travelplanner-benchmark-real-world-planning/). arXiv preprint arXiv:2402.01622, 2024.
11. Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. [LLM+P: Empowering large language models with optimal planning proficiency](https://md.research.elamur.ai/free/en/papers/liu-2023-llm-p-empowering-large-language/). arXiv preprint arXiv:2304.11477, 2023.
12. Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. [Leveraging pre-trained large language models to construct and utilize world models for model-based task planning](https://md.research.elamur.ai/free/en/papers/guan-2023-leveraging-pre-trained-large-language/). In NeurIPS, pages 79081–79094, 2023.
13. Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models - A critical investigation. In NeurIPS, pages 75993–76005, 2023. — [On the Planning Abilities of Large Language Models : A Critical Investigation](https://md.research.elamur.ai/free/en/papers/valmeekam-2023-planning-abilities-large-language-models/)
14. Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. Chain of thoughtlessness: An analysis of CoT in planning. arXiv preprint arXiv:2405.04776, 2024. — [arXiv:2405.04776](https://arxiv.org/abs/2405.04776) [find](https://scholar.google.com/scholar?q=Kaya+Stechly%2C+Karthik+Valmeekam%2C+and+Subbarao+Kambhampati%2E+Chain+of+thoughtlessness%3A+An+analysis+of+CoT+in+planning%2E+arXiv+preprint+arXiv%3A2405%2E04776%2C+2024%2E+%E2%80%94+%5BarXiv%3A2405%2E04776%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
15. Kahneman Daniel. Thinking, Fast and Slow. Macmillan, 2011. [find](https://scholar.google.com/scholar?q=Kahneman+Daniel%2E+Thinking%2C+Fast+and+Slow%2E+Macmillan%2C+2011%2E)
16. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In NeurIPS, pages 24824–24837, 2022.
17. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171, 2022.
18. Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. [Complexity-based prompting for multi-step reasoning](https://md.research.elamur.ai/free/en/papers/fu-2022-complexity-based-prompting-multi-step/). In ICLR, 2022.
19. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/). arXiv preprint arXiv:2205.10625, 2022.
20. Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. [Llemma: An open language model for mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/). arXiv preprint arXiv:2310.10631, 2023.
21. Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. [Mammoth: Building math generalist models through hybrid instruction tuning](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/). arXiv preprint arXiv:2309.05653, 2023.
22. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.[20050, 2023](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/).
23. Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.[14275, 2022](https://md.research.elamur.ai/free/en/papers/uesato-2022-solving-math-word-problems-process/).
24. Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. Discriminator-guided multi-step reasoning with language models. arXiv preprint arXiv:2305.14934, 2023. — [arXiv:2305.14934](https://arxiv.org/abs/2305.14934) [find](https://scholar.google.com/scholar?q=Muhammad+Khalifa%2C+Lajanugen+Logeswaran%2C+Moontae+Lee%2C+Honglak+Lee%2C+and+Lu+Wang%2E+Discriminator%2Dguided+multi%2Dstep+reasoning+with+language+models%2E+arXiv+preprint+arXiv%3A2305%2E14934%2C+2023%2E+%E2%80%94+%5BarXiv%3A2305%2E1493)
25. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In NeurIPS, 2023. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Tom+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+In+NeurIPS%2C+2023%2E)
26. Xidong Feng, Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang, and Jun Wang. AlphaZero-like tree-search can guide large language model decoding and training. arXiv preprint arXiv:2309.17179, 2023. [find](https://scholar.google.com/scholar?q=Xidong+Feng%2C+Ziyu+Wan%2C+Muning+Wen%2C+Ying+Wen%2C+Weinan+Zhang%2C+and+Jun+Wang%2E+AlphaZero%2Dlike+tree%2Dsearch+can+guide+large+language+model+decoding+and+training%2E+arXiv+preprint+arXiv%3A2309%2E17179%2C+2023%2E)
27. Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992, 2023. [find](https://scholar.google.com/scholar?q=Shibo+Hao%2C+Yi+Gu%2C+Haodi+Ma%2C+Joshua+Jiahua+Hong%2C+Zhen+Wang%2C+Daisy+Zhe+Wang%2C+and+Zhiting+Hu%2E+Reasoning+with+language+model+is+planning+with+world+model%2E+arXiv+preprint+arXiv%3A2305%2E14992%2C+2023%2E)
28. Yuchen Zhuang, Xiang Chen, Tong Yu, Saayan Mitra, Victor Bursztyn, Ryan A Rossi, Somdeb Sarkhel, and Chao Zhang. Toolchain*: Efficient action space navigation in large language models with A* search. arXiv preprint arXiv:2310.13227, 2023. — [arXiv:2310.13227](https://arxiv.org/abs/2310.13227) [find](https://scholar.google.com/scholar?q=Yuchen+Zhuang%2C+Xiang+Chen%2C+Tong+Yu%2C+Saayan+Mitra%2C+Victor+Bursztyn%2C+Ryan+A+Rossi%2C+Somdeb+Sarkhel%2C+and+Chao+Zhang%2E+Toolchain%2A%3A+Efficient+action+space+navigation+in+large+language+models+with+A%2A+search%2E+)
29. Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in Games, 4(1):1–43, 2012… [find](https://scholar.google.com/scholar?q=Cameron+B+Browne%2C+Edward+Powley%2C+Daniel+Whitehouse%2C+Simon+M+Lucas%2C+Peter+I+Cowling%2C+Philipp+Rohlfshagen%2C+Stephen+Tavener%2C+Diego+Perez%2C+Spyridon+Samothrakis%2C+and+Simon+Colton%2E+A+survey+of+monte+carlo+t)
30. Peter E Hart, Nils J Nilsson, and Bertram Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics, 4(2):100–107, 1968. [find](https://scholar.google.com/scholar?q=Peter+E+Hart%2C+Nils+J+Nilsson%2C+and+Bertram+Raphael%2E+A+formal+basis+for+the+heuristic+determination+of+minimum+cost+paths%2E+IEEE+Transactions+on+Systems+Science+and+Cybernetics%2C+4%282%29%3A100%E2%80%93107%2C+1968%2E)
31. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). In NeurIPS, pages 27730–27744, 2022.
32. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). arXiv preprint arXiv:2305.18290, 2023.
33. Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang. Aligner: Achieving efficient alignment through weak-to-strong correction. arXiv preprint arXiv:2402.[02416, 2024](https://md.research.elamur.ai/free/en/papers/ji-2024-aligner-efficient-alignment-learning-correct/).
34. Rishi Hazra, Pedro Zuidberg Dos Martires, and Luc De Raedt. SayCanPay: Heuristic planning with large language models using learnable domain knowledge. In AAAI, pages 20123–20133, 2024. [find](https://scholar.google.com/scholar?q=Rishi+Hazra%2C+Pedro+Zuidberg+Dos+Martires%2C+and+Luc+De+Raedt%2E+SayCanPay%3A+Heuristic+planning+with+large+language+models+using+learnable+domain+knowledge%2E+In+AAAI%2C+pages+20123%E2%80%9320133%2C+2024%2E)
35. Dong Huang, Qingwen Bu, Jie M Zhang, Michael Luck, and Heming Cui. [Agentcoder: Multi-agent-based code generation with iterative testing and optimisation](https://md.research.elamur.ai/free/en/papers/huang-2023-agentcoder-multi-agent-based/). arXiv preprint arXiv:2312.13010, 2023.
36. Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. [Reflexion: Language agents with verbal reinforcement learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/). arXiv preprint cs.AI/2303.11366, 2023.
37. Qwen Team. Code with codeqwen1.5, April 2024. URL [https://qwenlm.github.io/](https://qwenlm.github.io/) blog/codeqwen1.5/. [↗](https://qwenlm.github.io/)
38. Blai Bonet and Héctor Geffner. Planning as heuristic search. Artificial Intelligence, 129(1-2): 5–33, 2001. [find](https://scholar.google.com/scholar?q=Blai+Bonet+and+H%C3%A9ctor+Geffner%2E+Planning+as+heuristic+search%2E+Artificial+Intelligence%2C+129%281%2D2%29%3A+5%E2%80%9333%2C+2001%2E)
39. David Silver. Cooperative pathfinding. In AAAI-AIIDE, pages 117–122, 2005. [find](https://scholar.google.com/scholar?q=David+Silver%2E+Cooperative+pathfinding%2E+In+AAAI%2DAIIDE%2C+pages+117%E2%80%93122%2C+2005%2E)
40. Bobak Pezeshki, Radu Marinescu, Alexander Ihler, and Rina Dechter. AND/OR branch-and-bound for computational protein design optimizing K. In UAI, pages 1602–1612, 2022. [find](https://scholar.google.com/scholar?q=Bobak+Pezeshki%2C+Radu+Marinescu%2C+Alexander+Ihler%2C+and+Rina+Dechter%2E+AND%2FOR+branch%2Dand%2Dbound+for+computational+protein+design+optimizing+K%2E+In+UAI%2C+pages+1602%E2%80%931612%2C+2022%2E)
41. Stuart J Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. Pearson, 2016. [find](https://scholar.google.com/scholar?q=Stuart+J+Russell+and+Peter+Norvig%2E+Artificial+Intelligence%3A+A+Modern+Approach%2E+Pearson%2C+2016%2E)
42. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. [Fine-grained human feedback gives better rewards for language model training](https://md.research.elamur.ai/free/en/papers/wu-2023-fine-grained-human-feedback-gives/). arXiv preprint arXiv:2306.01693, 2023.
43. Martin Riedmiller. Neural fitted Q iteration–first experiences with a data efficient neural reinforcement learning method. In ECML, pages 317–328, 2005. [find](https://scholar.google.com/scholar?q=Martin+Riedmiller%2E+Neural+fitted+Q+iteration%E2%80%93first+experiences+with+a+data+efficient+neural+reinforcement+learning+method%2E+In+ECML%2C+pages+317%E2%80%93328%2C+2005%2E)
44. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732, 2021.
45. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
46. Kumar Shridhar, Koustuv Sinha, Andrew Cohen, Tianlu Wang, Ping Yu, Ram Pasunuru, Mrinmaya Sachan, Jason Weston, and Asli Celikyilmaz. The art of LLM refinement: Ask, refine, and trust. arXiv preprint arXiv:2311.07961, 2023. [find](https://scholar.google.com/scholar?q=Kumar+Shridhar%2C+Koustuv+Sinha%2C+Andrew+Cohen%2C+Tianlu+Wang%2C+Ping+Yu%2C+Ram+Pasunuru%2C+Mrinmaya+Sachan%2C+Jason+Weston%2C+and+Asli+Celikyilmaz%2E+The+art+of+LLM+refinement%3A+Ask%2C+refine%2C+and+trust%2E+arXiv+preprint+)
47. Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. [Common 7b language models already possess strong math capabilities](https://md.research.elamur.ai/free/en/papers/li-2024-common-7b-language-models-already/). arXiv preprint arXiv:2403.04706, 2024.
48. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. [Sparks of artificial general intelligence: Early experiments with GPT-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712, 2023.
49. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Y Wu, and Daya Guo. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300, 2024.
50. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
51. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. [Gemini: A family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
52. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023. [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Maxwell+Lin%2C+Nathanael+Sch%C3%A4rli%2C+and+Denny+Zhou%2E+Teaching+large+language+models+to+self%2Ddebug%2E+arXiv+preprint+arXiv%3A2304%2E05128%2C+2023%2E)

*Updated 2 November 2026*