---
title: "Recursive Introspection — paper by Qu et al. (2024)"
description: "The authors introduce RISE (Recursive IntroSpEction), a fine-tuning method that teaches a single… 66 references, cited by 56 papers here."
authors: ["Yuxiao Qu", "Tianjun Zhang", "Naman Garg", "Aviral Kumar"]
year: 2024
venue: "arXiv:2407.18219"
source: "https://arxiv.org/abs/2407.18219"
doi: "10.48550/arxiv.2407.18219"
arxiv: "2407.18219"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/qu-2024-recursive-introspection-teaching-language-model/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/qu-2024-recursive-introspection-teaching-language-model/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / Recursive Introspection: Teaching Language Model Agents How to Self-Improve

# Recursive Introspection: Teaching Language Model Agents How to Self-Improve

**Authors:** Yuxiao Qu, [Tianjun Zhang](https://md.research.elamur.ai/free/en/authors/tianjun-zhang/), [Naman Garg](https://md.research.elamur.ai/free/en/authors/naman-garg/) and [Aviral Kumar](https://md.research.elamur.ai/free/en/authors/aviral-kumar/) · **Year:** 2024 · **Venue:** [arXiv:2407.18219](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2024](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2024/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2407.18219) · [PDF](https://arxiv.org/pdf/2407.18219) · [DOI 10.48550/arxiv.2407.18219](https://doi.org/10.48550/arxiv.2407.18219) · [arXiv 2407.18219](https://arxiv.org/abs/2407.18219) · [OpenAlex](https://openalex.org/W4415797621)

**Citations:** 19 citations (OpenAlex, 2026-10-14) · [56](https://md.research.elamur.ai/free/en/papers/qu-2024-recursive-introspection-teaching-language-model/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models, including strong proprietary ones, rarely fix their own mistakes when given more attempts at a problem, even when told an answer is wrong. The authors introduce RISE (Recursive IntroSpEction), a fine-tuning method that teaches a single model to revise its answers over several turns. A single-turn prompt is recast as a multi-turn Markov decision process, with the prompt as the initial state. RISE draws on online imitation learning and RL. It collects on-policy rollouts from the learner and builds improved next-turn responses with best-of-N selection, using a task success signal. The candidates come either from the learner itself or from a stronger model. Training uses reward-weighted regression, so the model learns from both good and bad parts of each rollout, and the procedure is repeated over iterations. At test time, majority voting across turns produces the final answer. Llama2, Llama3 and Mistral models trained this way improve steadily over turns on GSM8K and MATH, and their first-turn accuracy is not hurt. This shows that fine-tuning can instill intrinsic self-correction.

## Key points
- On GSM8K, RISE trained only on each model's own data improved LLaMa3-8B by 8.2% and Mistral-7B by 6.6%.
- Over five turns on GSM8K, LLaMa2-7B gained 17.7% and Mistral-7B gained 23.9%, while GPT-3.5 improved by only 4.6%. On MATH, the gains were 4.6% for LLaMa2-7B and 11.1% for Mistral-7B.
- Sequential self-improvement with RISE beat parallel sampling at equal inference compute. Simply imitating multi-turn data from other models was not enough to produce the same ability.
- Error analysis, following Huang et al., found that correct→incorrect flips decreased over turns and that more problems stayed solved. The model sometimes rewrote a response completely and sometimes fixed only the faulty steps.
- Limitations: compute allowed at most two training iterations, and only one when the model supervised itself. Training was done on one task at a time.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Yuxiao Qu, Tianjun Zhang, Naman Garg and Aviral Kumar. 2024. Recursive Introspection: Teaching Language Model Agents How to Self-Improve. arXiv:2407.18219.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{qu2024recursive,
  title = {{Recursive Introspection: Teaching Language Model Agents How to Self-Improve}},
  author = {Yuxiao Qu, Tianjun Zhang, Naman Garg and Aviral Kumar},
  year = {2024},
  doi = {10.48550/arxiv.2407.18219},
  eprint = {2407.18219},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.18219}
}
```

- [← previous PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities](https://md.research.elamur.ai/free/en/papers/sravanthi-2024-pub-pragmatics-understanding-benchmark-assessing/)
- [next → SciCode: A Research Coding Benchmark Curated by Scientists](https://md.research.elamur.ai/free/en/papers/tian-2024-scicode-research-coding-benchmark/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

66 references, 42 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rishabh Agarwal, Nino Vieillard, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. Gkd: Generalized knowledge distillation for auto-regressive sequence models. arXiv preprint arXiv:2306.[13649, 2023](https://md.research.elamur.ai/free/en/papers/agarwal-2023-policy-distillation-language-models-learning/).
2. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073, 2022.
3. Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeff Wu. [Weak-to-strong generalization: Eliciting strong capabilities with weak supervision](https://md.research.elamur.ai/free/en/papers/burns-2023-weak-strong-generalization-eliciting-strong/), 2023. URL https://arxiv.org/abs/2… [↗](https://arxiv.org/abs/2…)
4. Jonathan D Chang, Wenhao Shan, Owen Oertell, Kianté Brantley, Dipendra Misra, Jason D Lee, and Wen Sun. Dataset reset policy optimization for rlhf. arXiv preprint arXiv:2404.08495, 2024. — [arXiv:2404.08495](https://arxiv.org/abs/2404.08495) [find](https://scholar.google.com/scholar?q=Jonathan+D+Chang%2C+Wenhao+Shan%2C+Owen+Oertell%2C+Kiant%C3%A9+Brantley%2C+Dipendra+Misra%2C+Jason+D+Lee%2C+and+Wen+Sun%2E+Dataset+reset+policy+optimization+for+rlhf%2E+arXiv+preprint+arXiv%3A2404%2E08495%2C+2024%2E+%E2%80%94+%5BarXiv%3A2404)
5. Yiannis Charalambous, Norbert Tihanyi, Ridhi Jain, Youcheng Sun, Mohamed Amine Ferrag, and Lucas C Cordeiro. A new era in software security: Towards self-healing software via large language models and formal verification. arXiv preprint arXiv:2305.14752, 2023. — [arXiv:2305.14752](https://arxiv.org/abs/2305.14752) [find](https://scholar.google.com/scholar?q=Yiannis+Charalambous%2C+Norbert+Tihanyi%2C+Ridhi+Jain%2C+Youcheng+Sun%2C+Mohamed+Amine+Ferrag%2C+and+Lucas+C+Cordeiro%2E+A+new+era+in+software+security%3A+Towards+self%2Dhealing+software+via+large+language+models+and)
6. Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. [Fireact: Toward language agent fine-tuning](https://md.research.elamur.ai/free/en/papers/chen-2023-fireact-toward-language-agent-fine/), 2023.
7. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023. [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Maxwell+Lin%2C+Nathanael+Sch%C3%A4rli%2C+and+Denny+Zhou%2E+Teaching+large+language+models+to+self%2Ddebug%2E+arXiv+preprint+arXiv%3A2304%2E05128%2C+2023%2E)
8. Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024. [find](https://scholar.google.com/scholar?q=Zixiang+Chen%2C+Yihe+Deng%2C+Huizhuo+Yuan%2C+Kaixuan+Ji%2C+and+Quanquan+Gu%2E+Self%2Dplay+fine%2Dtuning+converts+weak+language+models+to+strong+language+models%2E+arXiv+preprint+arXiv%3A2401%2E01335%2C+2024%2E)
9. Yew Ken Chia, Guizhen Chen, Luu Anh Tuan, Soujanya Poria, and Lidong Bing. [Contrastive chain-of-thought prompting](https://md.research.elamur.ai/free/en/papers/chia-2023-contrastive-chain-thought-prompting/). arXiv preprint arXiv:2311.09277, 2023.
10. Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. arXiv preprint arXiv:1912.01588, 2019. [find](https://scholar.google.com/scholar?q=Karl+Cobbe%2C+Christopher+Hesse%2C+Jacob+Hilton%2C+and+John+Schulman%2E+Leveraging+procedural+generation+to+benchmark+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1912%2E01588%2C+2019%2E)
11. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
12. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. [Improving factuality and reasoning in language models through multiagent debate](https://md.research.elamur.ai/free/en/papers/du-2023-improving-factuality-reasoning-language-models/). arXiv preprint arXiv:2305.14325, 2023.
13. Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah D Goodman. Stream of search (sos): Learning to search in language. arXiv preprint arXiv:2404.03683, 2024. [find](https://scholar.google.com/scholar?q=Kanishk+Gandhi%2C+Denise+Lee%2C+Gabriel+Grand%2C+Muxin+Liu%2C+Winson+Cheng%2C+Archit+Sharma%2C+and+Noah+D+Goodman%2E+Stream+of+search+%28sos%29%3A+Learning+to+search+in+language%2E+arXiv+preprint+arXiv%3A2404%2E03683%2C+2024%2E)
14. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764–10799. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Pengfei+Liu%2C+Yiming+Yang%2C+Jamie+Callan%2C+and+Graham+Neubig%2E+Pal%3A+Program%2Daided+language+models%2E+In+International+Conference+on+Machine+Learning%2C+pages+1076)
15. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. [Critic: Large Language Models can Self-Correct with Tool-Interactive Critiquing](https://md.research.elamur.ai/free/en/papers/gou-2023-critic-large-language-models-self/). arXiv preprint arXiv:2305.11738, 2023.
16. David Y. Hancock, Jeremy Fischer, John Michael Lowe, Winona Snapp-Childs, Marlon Pierce, Suresh Marru, J. Eric Coulter, Matthew Vaughn, Brian Beck, Nirav Merchant, Edwin Skidmore, and Gwen Jacobs. Jetstream2: Accelerating cloud computing via jetstream. In Practice and Experience in Advanced Research… — [doi:10.1145/3437359.3465565](https://doi.org/10.1145/3437359.3465565) [find](https://scholar.google.com/scholar?q=David+Y%2E+Hancock%2C+Jeremy+Fischer%2C+John+Michael+Lowe%2C+Winona+Snapp%2DChilds%2C+Marlon+Pierce%2C+Suresh+Marru%2C+J%2E+Eric+Coulter%2C+Matthew+Vaughn%2C+Brian+Beck%2C+Nirav+Merchant%2C+Edwin+Skidmore%2C+and+Gwen+Jacobs%2E+Jet)
17. Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, and Roberta Railneau. [Glore: When, where, and how to improve llm reasoning via global and local refinements](https://md.research.elamur.ai/free/en/papers/havrilla-2024-glore-where-improve-llm-reasoning/). arXiv preprint arXiv:2402.10963, 2024.
18. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). NeurIPS, 2021.
19. Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. [V-star: Training verifiers for self-taught reasoners](https://md.research.elamur.ai/free/en/papers/hosseini-2024-v-star-training-verifiers-self/). arXiv preprint arXiv:2402.06457, 2024.
20. Dong Huang, Qingwen Bu, Jie M Zhang, Michael Luck, and Heming Cui. [Agentcoder: Multi-agent-based code generation with iterative testing and optimisation](https://md.research.elamur.ai/free/en/papers/huang-2023-agentcoder-multi-agent-based/). arXiv preprint arXiv:2312.13010, 2023.
21. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. [Large language models cannot self-correct reasoning yet](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-cannot-self/). arXiv preprint arXiv:2310.01798, 2023.
22. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. [Language models as zero-shot planners: Extracting actionable knowledge for embodied agents](https://md.research.elamur.ai/free/en/papers/huang-2022-language-models-zero-shot-planners/). In International Conference on Machine Learning, pages 9118–9147. PMLR, 2022.
23. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+Mistral+7b)
24. Katie Kang, Eric Wallace, Claire Tomlin, Aviral Kumar, and Sergey Levine. [Unfamiliar finetuning examples control how language models hallucinate](https://md.research.elamur.ai/free/en/papers/kang-2025-unfamiliar-finetuning-examples-control-how/), 2024.
25. Diederik P Kingma and Max Welling. [Auto-encoding variational bayes](https://md.research.elamur.ai/free/en/papers/kingma-2013-auto-encoding-variational-bayes/), 2022. URL https://arxiv.org/abs/1312.6114 . [↗](https://arxiv.org/abs/1312.6114)
26. Lucas Lehnert, Sainbayar Sukhbaatar, Paul Mcvay, Michael Rabbat, and Yuandong Tian. Beyond a*: Better planning with transformers via search dynamics bootstrapping. arXiv preprint arXiv:2402.14083, 2024. [find](https://scholar.google.com/scholar?q=Lucas+Lehnert%2C+Sainbayar+Sukhbaatar%2C+Paul+Mcvay%2C+Michael+Rabbat%2C+and+Yuandong+Tian%2E+Beyond+a%2A%3A+Better+planning+with+transformers+via+search+dynamics+bootstrapping%2E+arXiv+preprint+arXiv%3A2402%2E14083%2C+202)
27. Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. [Common 7b language models already possess strong math capabilities](https://md.research.elamur.ai/free/en/papers/li-2024-common-7b-language-models-already/). arXiv preprint arXiv:2403.04706, 2024.
28. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.[20050, 2023](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/).
29. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. [Agentbench: Evaluating llms as agents](https://md.research.elamur.ai/free/en/papers/liu-2023-agentbench-evaluating-llms-agents/). arXiv preprint arXiv:2308.03688, 2023.
30. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. [Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/). arXiv preprint arXiv:2308.09583, 2023.
31. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+et+al%2E+Self%2Drefine%3A+Iterative+refinement+with+self%2Dfeedbac)
32. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. ICLR, 2023. [find](https://scholar.google.com/scholar?q=Erik+Nijkamp%2C+Bo+Pang%2C+Hiroaki+Hayashi%2C+Lifu+Tu%2C+Huan+Wang%2C+Yingbo+Zhou%2C+Silvio+Savarese%2C+and+Caiming+Xiong%2E+CodeGen%3A+An+Open+Large+Language+Model+for+Code+with+Multi%2DTurn+Program+Synthesis%2E+ICLR%2C+202)
33. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. [Show your work: Scratchpads for intermediate computation with language models](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/). arXiv preprint arXiv:2112.00114, 2021.
34. Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. [Advantage-weighted regression: Simple and scalable off-policy reinforcement learning](https://md.research.elamur.ai/free/en/papers/peng-2019-advantage-weighted-regression-simple-scalable/). arXiv preprint arXiv:1910.00177, 2019.
35. Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pages 745–750. ACM, 2007. [find](https://scholar.google.com/scholar?q=Jan+Peters+and+Stefan+Schaal%2E+Reinforcement+learning+by+reward%2Dweighted+regression+for+operational+space+control%2E+In+Proceedings+of+the+24th+international+conference+on+Machine+learning%2C+pages+745%E2%80%93750)
36. Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík, editors, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics,… [find](https://scholar.google.com/scholar?q=Stephane+Ross%2C+Geoffrey+Gordon%2C+and+Drew+Bagnell%2E+A+reduction+of+imitation+learning+and+structured+prediction+to+no%2Dregret+online+learning%2E+In+Geoffrey+Gordon%2C+David+Dunson%2C+and+Miroslav+Dud%C3%ADk%2C+editor)
37. Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715, 2024. [find](https://scholar.google.com/scholar?q=Corby+Rosset%2C+Ching%2DAn+Cheng%2C+Arindam+Mitra%2C+Michael+Santacroce%2C+Ahmed+Awadallah%2C+and+Tengyang+Xie%2E+Direct+nash+optimization%3A+Teaching+language+models+to+self%2Dimprove+with+general+preferences%2E+arXiv+p)
38. Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. [Branch-solve-merge improves large language model evaluation and generation](https://md.research.elamur.ai/free/en/papers/saha-2023-branch-solve-merge-improves-large/). arXiv preprint arXiv:2310.15123, 2023.
39. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. [Toolformer: Language models can teach themselves to use tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/). arXiv preprint arXiv:2302.04761, 2023.
40. Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.[11366, 2023](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/).
41. Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206.11871, 2022. — [arXiv:2206.11871](https://arxiv.org/abs/2206.11871) [find](https://scholar.google.com/scholar?q=Charlie+Snell%2C+Ilya+Kostrikov%2C+Yi+Su%2C+Mengjiao+Yang%2C+and+Sergey+Levine%2E+Offline+rl+for+natural+language+generation+with+implicit+language+q+learning%2E+arXiv+preprint+arXiv%3A2206%2E11871%2C+2022%2E+%E2%80%94+%5BarXiv%3A22)
42. Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. [Deep unsupervised learning using nonequilibrium thermodynamics](https://md.research.elamur.ai/free/en/papers/sohl-dickstein-2015-deep-unsupervised-learning-using-nonequilibrium/), 2015. URL https://arxiv.org/abs/1503. 03585 . [↗](https://arxiv.org/abs/1503)
43. Yang Song and Diederik P. Kingma. How to train your energy-based models, 2021. URL https://arxiv.org/abs/2101.03288 . — [arXiv:2101.03288](https://arxiv.org/abs/2101.03288) [↗](https://arxiv.org/abs/2101.03288)
44. Liting Sun, Cheng Peng, Wei Zhan, and Masayoshi Tomizuka. A fast integrated planning and control framework for autonomous driving via imitation learning. In Dynamic Systems and Control Conference, volume 51913, page V003T37A012. American Society of Mechanical Engineers, 2018. [find](https://scholar.google.com/scholar?q=Liting+Sun%2C+Cheng+Peng%2C+Wei+Zhan%2C+and+Masayoshi+Tomizuka%2E+A+fast+integrated+planning+and+control+framework+for+autonomous+driving+via+imitation+learning%2E+In+Dynamic+Systems+and+Control+Conference%2C+vol)
45. Gokul Swamy, Sanjiban Choudhury, J. Andrew Bagnell, and Zhiwei Steven Wu. Inverse reinforcement learning without reinforcement learning, 2024. URL https://arxiv.org/abs/2303. 14623 . [↗](https://arxiv.org/abs/2303)
46. Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. [Openmathinstruct-1: A 1.8 million math instruction tuning dataset](https://md.research.elamur.ai/free/en/papers/toshniwal-2024-openmathinstruct-1-1-8-million/). arXiv preprint arXiv:2402.10176, 2024.
47. Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.[14275, 2022](https://md.research.elamur.ai/free/en/papers/uesato-2022-solving-math-word-problems-process/).
48. Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. [Towards understanding chain-of-thought prompting: An empirical study of what matters](https://md.research.elamur.ai/free/en/papers/wang-2023-understanding-chain-thought-prompting-empirical/). arXiv preprint arXiv:2212.10001, 2022.
49. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv: Arxiv-2305.16291, 2023. [find](https://scholar.google.com/scholar?q=Guanzhi+Wang%2C+Yuqi+Xie%2C+Yunfan+Jiang%2C+Ajay+Mandlekar%2C+Chaowei+Xiao%2C+Yuke+Zhu%2C+Linxi+Fan%2C+and+Anima+Anandkumar%2E+Voyager%3A+An+open%2Dended+embodied+agent+with+large+language+models%2E+arXiv+preprint+arXiv%3A+A)
50. Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. [Math-shepherd: Verify and reinforce llms step-by-step without human annotations](https://md.research.elamur.ai/free/en/papers/wang-2024-math-shepherd-verify-reinforce-llms/). CoRR, abs/2312.08935, 2023.
51. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171, 2022.
52. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). NeurIPS, 2022.
53. Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. [Generating sequences by learning to self-correct](https://md.research.elamur.ai/free/en/papers/welleck-2022-generating-sequences-learning-self-correct/). In The Eleventh International Conference on Learning Representations, 2023. URL [https://openreview.net/forum?id=hH36JeQZDaO](https://openreview.net/forum?id=hH36JeQZDaO) . [↗](https://openreview.net/forum?id=hH36JeQZDaO)
54. Hui Yang, Sifu Yue, and Yunzhong He. [Auto-gpt for online decision making: Benchmarks and additional opinions](https://md.research.elamur.ai/free/en/papers/yang-2023-auto-gpt-online-decision-making/). arXiv preprint arXiv:2306.02224, 2023.
55. Kaiyu Yang, Aidan M Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar. [LeanDojo: Theorem Proving with Retrieval-Augmented Language Models](https://md.research.elamur.ai/free/en/papers/yang-2023-leandojo-theorem-proving-retrieval-augmented/). arXiv preprint arXiv:2306.15626, 2023.
56. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. [React: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). arXiv preprint arXiv:2210.03629, 2022.
57. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). arXiv preprint arXiv:2309.12284, 2023.
58. Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. [Advancing llm reasoning generalists with preference trees](https://md.research.elamur.ai/free/en/papers/yuan-2024-advancing-llm-reasoning-generalists-preference/). arXiv preprint arXiv:2404.02078, 2024.
59. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024. [find](https://scholar.google.com/scholar?q=Weizhe+Yuan%2C+Richard+Yuanzhe+Pang%2C+Kyunghyun+Cho%2C+Sainbayar+Sukhbaatar%2C+Jing+Xu%2C+and+Jason+Weston%2E+Self%2Drewarding+language+models%2E+arXiv+preprint+arXiv%3A2401%2E10020%2C+2024%2E)
60. Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. [Mammoth: Building math generalist models through hybrid instruction tuning](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/). arXiv preprint arXiv:2309.05653, 2023.
61. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. [Star: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). Advances in Neural Information Processing Systems, 35:15476–15488, 2022.
62. Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. [Agenttuning: Enabling generalized agent abilities for llms](https://md.research.elamur.ai/free/en/papers/zeng-2023-agenttuning-enabling-generalized-agent/). arXiv preprint arXiv:2310.12823, 2023.
63. Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E Gonzalez. Tempera: Test-time prompting via reinforcement learning. arXiv preprint arXiv:2211.11890, 2022. — [arXiv:2211.11890](https://arxiv.org/abs/2211.11890) [find](https://scholar.google.com/scholar?q=Tianjun+Zhang%2C+Xuezhi+Wang%2C+Denny+Zhou%2C+Dale+Schuurmans%2C+and+Joseph+E+Gonzalez%2E+Tempera%3A+Test%2Dtime+prompting+via+reinforcement+learning%2E+arXiv+preprint+arXiv%3A2211%2E11890%2C+2022%2E+%E2%80%94+%5BarXiv%3A2211%2E11890%5D%28htt)
64. Tianjun Zhang, Aman Madaan, Luyu Gao, Steven Zheng, Swaroop Mishra, Yiming Yang, Niket Tandon, and Uri Alon. [In-context principle learning from mistakes](https://md.research.elamur.ai/free/en/papers/zhang-2024-context-principle-learning-mistakes/). arXiv preprint arXiv:2402.05403, 2024.
65. Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406, 2023. [find](https://scholar.google.com/scholar?q=Andy+Zhou%2C+Kai+Yan%2C+Michal+Shlapentokh%2DRothman%2C+Haohan+Wang%2C+and+Yu%2DXiong+Wang%2E+Language+agent+tree+search+unifies+reasoning+acting+and+planning+in+language+models%2E+arXiv+preprint+arXiv%3A2310%2E04406%2C+20)
66. Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl. arXiv preprint arXiv:2402.19446, 2024. [find](https://scholar.google.com/scholar?q=Yifei+Zhou%2C+Andrea+Zanette%2C+Jiayi+Pan%2C+Sergey+Levine%2C+and+Aviral+Kumar%2E+Archer%3A+Training+language+model+agents+via+hierarchical+multi%2Dturn+rl%2E+arXiv+preprint+arXiv%3A2402%2E19446%2C+2024%2E)

*Updated 2 November 2026*