--- title: "Training Language Models to Self-Correct via — Kumar 2024" description: "Large language models rarely fix their own mistakes when they get no outside feedback, a setting called intrinsic self-correction." authors: ["Aviral Kumar", "Vincent Zhuang", "Rishabh Agarwal", "Yi Su", "John D Co-Reyes", "Avi Singh", "Kate Baumli", "Shariq Iqbal", "Colton Bishop", "Rebecca Roelofs", "Lei M Zhang", "Kay McKinney", "Disha Shrivastava", "Cosmin Paduraru", "George Tucker", "Doina Precup", "Feryal Behbahani", "Aleksandra Faust"] year: 2024 venue: "arXiv:2409.12917" source: "https://arxiv.org/abs/2409.12917" doi: "10.48550/arxiv.2409.12917" arxiv: "2409.12917" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/kumar-2024-training-language-models-self-correct/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kumar-2024-training-language-models-self-correct/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / Training Language Models to Self-Correct via Reinforcement Learning # Training Language Models to Self-Correct via Reinforcement Learning **Authors:** [Aviral Kumar](https://md.research.elamur.ai/free/en/authors/aviral-kumar/), Vincent Zhuang, [Rishabh Agarwal](https://md.research.elamur.ai/free/en/authors/rishabh-agarwal/), Yi Su, [John D Co-Reyes](https://md.research.elamur.ai/free/en/authors/john-d-co-reyes/), [Avi Singh](https://md.research.elamur.ai/free/en/authors/avi-singh/), Kate Baumli, Shariq Iqbal, Colton Bishop, [Rebecca Roelofs](https://md.research.elamur.ai/free/en/authors/rebecca-roelofs/), [Lei M Zhang](https://md.research.elamur.ai/free/en/authors/lei-m-zhang/), Kay McKinney et al. (18 authors) · **Year:** 2024 · **Venue:** arXiv:2409.12917 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2409.12917) · [PDF](https://arxiv.org/pdf/2409.12917) · [DOI 10.48550/arxiv.2409.12917](https://doi.org/10.48550/arxiv.2409.12917) · [arXiv 2409.12917](https://arxiv.org/abs/2409.12917) · [OpenAlex](https://openalex.org/W4403885103) **Citations:** 8 citations (openalex-snapshot · 2026-10-14) · [143](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Large language models rarely fix their own mistakes when they get no outside feedback, a setting called intrinsic self-correction. Earlier training recipes for this needed extra models, a stronger teacher, or added supervision. Google DeepMind researchers propose SCoRe, a multi-turn online reinforcement learning method. It trains a single model to answer and then revise its answer, using only data the model generates itself. The authors first show why supervised fine-tuning on offline correction traces, such as STaR-style training, falls short. In some cases the errors in the training data differ from the ones the model actually makes (distribution shift). In others the model learns to give its best first answer and then barely change it (behavior collapse). SCoRe avoids both problems with two stages. Stage I improves the second attempt while keeping the first attempt close to the base model. Stage II trains both attempts with a shaped reward that pays for improvement between turns. On Gemini models, SCoRe is the first method to show clearly positive intrinsic self-correction, so models can spend extra inference compute refining their own reasoning. ## Key points - SCoRe raised the base models' self-correction by 15.6 points (absolute) on MATH and 9.1 points on HumanEval, using Gemini 1.0 Pro and Gemini 1.5 Flash. It needs no oracle feedback, teacher model or separate refinement model. - In the MATH ablation, full SCoRe went from 60.0% accuracy on the first attempt to 64.4% on the second, a gain of +4.4. Training for a single turn only made the second attempt worse (−2.4). Removing Stage I (+2.2) or the reward shaping (+2.6) shrank the gain. - Replacing on-policy REINFORCE with STaR in Stage II lowered accuracy (58.4% on the second attempt). This suggests that training on the model's own fresh samples matters especially for multi-turn correction. - With a budget of 32 samples per problem, spending part of it on one self-correction round beat pure parallel sampling with majority voting: a 10.5% gain versus 7.4%. - Limitations: the model was trained for only one correction round because of infrastructure constraints. Stages I and II are still separate training runs. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/) ## Cited by Cited by 143 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 136 not yet. - [Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs](https://md.research.elamur.ai/free/en/papers/chen-2024-think-that-much-2-3/) - [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2025-towards-reasoning-era-survey-long/) - [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deepseek-2025-ai-deepseek-r1-incentivizing-reasoning-capability/) - [Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs](https://md.research.elamur.ai/free/en/papers/gandhi-2025-cognitive-behaviors-that-enable-self/) - [rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking](https://md.research.elamur.ai/free/en/papers/guan-2025-rstar-math-small-llms-can/) - [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/jin-2025-search-r1-training-llms-reason/) - [How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark](https://md.research.elamur.ai/free/en/papers/yang-2025-llm-reasoning-distracted-irrelevant-context/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{kumar2024training, title = {{Training Language Models to Self-Correct via Reinforcement Learning}}, author = {Kumar, Aviral and Zhuang, Vincent and Agarwal, Rishabh and Su, Yi and Co-Reyes, John D and Singh, Avi and Baumli, Kate and Iqbal, Shariq and Bishop, Colton and Roelofs, Rebecca and Zhang, Lei M and McKinney, Kay and Shrivastava, Disha and Paduraru, Cosmin and Tucker, George and Precup, Doina and Behbahani, Feryal and Faust, Aleksandra}, year = {2024}, doi = {10.48550/arxiv.2409.12917}, eprint = {2409.12917}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2409.12917} } ``` - [← previous Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://md.research.elamur.ai/free/en/papers/alves-2024-tower-open-multilingual-large-language/) - [next → Training LLMs to Recognize Hedges in Dialogues about Roadrunner Cartoons](https://md.research.elamur.ai/free/en/papers/paige-2024-training-llms-recognize-hedges-dialogues/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) ## References 45 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, A. Üstün, and S. Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Ahmadian%2C+C%2E+Cremer%2C+M%2E+Gall%C3%A9%2C+M%2E+Fadaee%2C+J%2E+Kreutzer%2C+A%2E+%C3%9Cst%C3%BCn%2C+and+S%2E+Hooker%2E+Back+to+basics%3A+Revisiting+reinforce+style+optimization+for+learning+from+human+feedback+in+llms%2E+arXiv+preprint+arXi) 2. A. F. Akyürek, E. Akyürek, A. Madaan, A. Kalyan, P. Clark, D. Wijaya, and N. Tandon. Rl4f: Generating natural language feedback with reinforcement learning for repairing model outputs. arXiv preprint arXiv:2305.08844, 2023. [find](https://scholar.google.com/scholar?q=A%2E+F%2E+Aky%C3%BCrek%2C+E%2E+Aky%C3%BCrek%2C+A%2E+Madaan%2C+A%2E+Kalyan%2C+P%2E+Clark%2C+D%2E+Wijaya%2C+and+N%2E+Tandon%2E+Rl4f%3A+Generating+natural+language+feedback+with+reinforcement+learning+for+repairing+model+outputs%2E+arXiv+preprint+) 3. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. [find](https://scholar.google.com/scholar?q=J%2E+Austin%2C+A%2E+Odena%2C+M%2E+Nye%2C+M%2E+Bosma%2C+H%2E+Michalewski%2C+D%2E+Dohan%2C+E%2E+Jiang%2C+C%2E+Cai%2C+M%2E+Terry%2C+Q%2E+Le%2C+et+al%2E+Program+synthesis+with+large+language+models%2E+arXiv+preprint+arXiv%3A2108%2E07732%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/) 4. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. [find](https://scholar.google.com/scholar?q=M%2E+Chen%2C+J%2E+Tworek%2C+H%2E+Jun%2C+Q%2E+Yuan%2C+H%2E+P%2E+D%2E+O%2E+Pinto%2C+J%2E+Kaplan%2C+H%2E+Edwards%2C+Y%2E+Burda%2C+N%2E+Joseph%2C+G%2E+Brockman%2C+et+al%2E+Evaluating+large+language+models+trained+on+code%2E+arXiv+preprint+arXiv%3A2107%2E0337) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/) 5. X. Chen, M. Lin, N. Schärli, and D. Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023. [find](https://scholar.google.com/scholar?q=X%2E+Chen%2C+M%2E+Lin%2C+N%2E+Sch%C3%A4rli%2C+and+D%2E+Zhou%2E+Teaching+large+language+models+to+self%2Ddebug%2E+arXiv+preprint+arXiv%3A2304%2E05128%2C+2023%2E) 6. J. Farebrother, J. Orbay, Q. Vuong, A. A. Taïga, Y. Chebotar, T. Xiao, A. Irpan, S. Levine, P. S. Castro, A. Faust, et al. Stop regressing: Training value functions via classification for scalable deep rl. arXiv preprint arXiv:2403.03950, 2024. — [arXiv:2403.03950](https://arxiv.org/abs/2403.03950) [find](https://scholar.google.com/scholar?q=J%2E+Farebrother%2C+J%2E+Orbay%2C+Q%2E+Vuong%2C+A%2E+A%2E+Ta%C3%AFga%2C+Y%2E+Chebotar%2C+T%2E+Xiao%2C+A%2E+Irpan%2C+S%2E+Levine%2C+P%2E+S%2E+Castro%2C+A%2E+Faust%2C+et+al%2E+Stop+regressing%3A+Training+value+functions+via+classification+for+scalable+dee) 7. A. Havrilla, Y. Du, S. C. Raparthy, C. Nalmpantis, J. Dwivedi-Yu, M. Zhuravinskyi, E. Hambro, S. Sukhbaatar, and R. Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.04642, 2024a. [find](https://scholar.google.com/scholar?q=A%2E+Havrilla%2C+Y%2E+Du%2C+S%2E+C%2E+Raparthy%2C+C%2E+Nalmpantis%2C+J%2E+Dwivedi%2DYu%2C+M%2E+Zhuravinskyi%2C+E%2E+Hambro%2C+S%2E+Sukhbaatar%2C+and+R%2E+Raileanu%2E+Teaching+large+language+models+to+reason+with+reinforcement+learning%2E+arXi) 8. A. Havrilla, S. Raparthy, C. Nalmpantis, J. Dwivedi-Yu, M. Zhuravinskyi, E. Hambro, and R. Railneau. Glore: When, where, and how to improve llm reasoning via global and local refinements. arXiv preprint arXiv:2402.10963, 2024b. [find](https://scholar.google.com/scholar?q=A%2E+Havrilla%2C+S%2E+Raparthy%2C+C%2E+Nalmpantis%2C+J%2E+Dwivedi%2DYu%2C+M%2E+Zhuravinskyi%2C+E%2E+Hambro%2C+and+R%2E+Railneau%2E+Glore%3A+When%2C+where%2C+and+how+to+improve+llm+reasoning+via+global+and+local+refinements%2E+arXiv+prepri) 9. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021. [find](https://scholar.google.com/scholar?q=D%2E+Hendrycks%2C+C%2E+Burns%2C+S%2E+Kadavath%2C+A%2E+Arora%2C+S%2E+Basart%2C+E%2E+Tang%2C+D%2E+Song%2C+and+J%2E+Steinhardt%2E+Measuring+mathematical+problem+solving+with+the+math+dataset%2E+NeurIPS%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/) 10. J. Hong, N. Lee, and J. Thorne. Reference-free monolithic preference optimization with odds ratio. arXiv preprint arXiv:2403.07691, 2024. [find](https://scholar.google.com/scholar?q=J%2E+Hong%2C+N%2E+Lee%2C+and+J%2E+Thorne%2E+Reference%2Dfree+monolithic+preference+optimization+with+odds+ratio%2E+arXiv+preprint+arXiv%3A2403%2E07691%2C+2024%2E) 11. J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Huang%2C+X%2E+Chen%2C+S%2E+Mishra%2C+H%2E+S%2E+Zheng%2C+A%2E+W%2E+Yu%2C+X%2E+Song%2C+and+D%2E+Zhou%2E+Large+language+models+cannot+self%2Dcorrect+reasoning+yet%2E+arXiv+preprint+arXiv%3A2310%2E01798%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-cannot-self/) 12. N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024. [find](https://scholar.google.com/scholar?q=N%2E+Jain%2C+K%2E+Han%2C+A%2E+Gu%2C+W%2E%2DD%2E+Li%2C+F%2E+Yan%2C+T%2E+Zhang%2C+S%2E+Wang%2C+A%2E+Solar%2DLezama%2C+K%2E+Sen%2C+and+I%2E+Stoica%2E+Livecodebench%3A+Holistic+and+contamination+free+evaluation+of+large+language+models+for+code%2E+arXiv+) [our card](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/) 13. R. Kamoi, Y. Zhang, N. Zhang, J. Han, and R. Zhang. When can llms actually correct their own mistakes? a critical survey of self-correction of llms. arXiv preprint arXiv:2406.01297, 2024. [find](https://scholar.google.com/scholar?q=R%2E+Kamoi%2C+Y%2E+Zhang%2C+N%2E+Zhang%2C+J%2E+Han%2C+and+R%2E+Zhang%2E+When+can+llms+actually+correct+their+own+mistakes%3F+a+critical+survey+of+self%2Dcorrection+of+llms%2E+arXiv+preprint+arXiv%3A2406%2E01297%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/kamoi-2024-when-can-llms-actually-correct/) 14. G. Kim, P. Baldi, and S. McAleer. arXiv:2303.17491, 2023. [find](https://scholar.google.com/scholar?q=G%2E+Kim%2C+P%2E+Baldi%2C+and+S%2E+McAleer%2E+arXiv%3A2303%2E17491%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2023-language-models-solve-computer-tasks/) 15. Language models can solve computer tasks. arXiv preprint H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023. [find](https://scholar.google.com/scholar?q=Language+models+can+solve+computer+tasks%2E+arXiv+preprint+H%2E+Lightman%2C+V%2E+Kosaraju%2C+Y%2E+Burda%2C+H%2E+Edwards%2C+B%2E+Baker%2C+T%2E+Lee%2C+J%2E+Leike%2C+J%2E+Schulman%2C+I%2E+Sutskever%2C+and+K%2E+Cobbe%2E+Let%E2%80%99s+verify+step+by+step%2E) [our card](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/) 16. X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, et al. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688, 2023. [find](https://scholar.google.com/scholar?q=X%2E+Liu%2C+H%2E+Yu%2C+H%2E+Zhang%2C+Y%2E+Xu%2C+X%2E+Lei%2C+H%2E+Lai%2C+Y%2E+Gu%2C+H%2E+Ding%2C+K%2E+Men%2C+K%2E+Yang%2C+et+al%2E+Agentbench%3A+Evaluating+llms+as+agents%2E+arXiv+preprint+arXiv%3A2308%2E03688%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-agentbench-evaluating-llms-agents/) 17. A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y. Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Lozhkov%2C+R%2E+Li%2C+L%2E+B%2E+Allal%2C+F%2E+Cassano%2C+J%2E+Lamy%2DPoirier%2C+N%2E+Tazi%2C+A%2E+Tang%2C+D%2E+Pykhtar%2C+J%2E+Liu%2C+Y%2E+Wei%2C+et+al%2E+Starcoder+2+and+the+stack+v2%3A+The+next+generation%2E+arXiv+preprint+arXiv%3A2402%2E19173%2C+20) 18. A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Madaan%2C+N%2E+Tandon%2C+P%2E+Gupta%2C+S%2E+Hallinan%2C+L%2E+Gao%2C+S%2E+Wiegreffe%2C+U%2E+Alon%2C+N%2E+Dziri%2C+S%2E+Prabhumoye%2C+Y%2E+Yang%2C+et+al%2E+Self%2Drefine%3A+Iterative+refinement+with+self%2Dfeedback%2E+arXiv+preprint+arXiv%3A2303%2E176) 19. A. Ni, M. Allamanis, A. Cohan, Y. Deng, K. Shi, C. Sutton, and P. Yin. Next: Teaching large language models to reason about code execution. arXiv preprint arXiv:2404.14662, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Ni%2C+M%2E+Allamanis%2C+A%2E+Cohan%2C+Y%2E+Deng%2C+K%2E+Shi%2C+C%2E+Sutton%2C+and+P%2E+Yin%2E+Next%3A+Teaching+large+language+models+to+reason+about+code+execution%2E+arXiv+preprint+arXiv%3A2404%2E14662%2C+2024%2E) 20. T. X. Olausson, J. P. Inala, C. Wang, J. Gao, and A. Solar-Lezama. Is self-repair a silver bullet for code generation? In The Twelfth International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=T%2E+X%2E+Olausson%2C+J%2E+P%2E+Inala%2C+C%2E+Wang%2C+J%2E+Gao%2C+and+A%2E+Solar%2DLezama%2E+Is+self%2Drepair+a+silver+bullet+for+code+generation%3F+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+2023%2E) 21. L. Pan, M. Saxon, W. Xu, D. Nathani, X. Wang, and W. Y. Wang. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. arXiv preprint arXiv:2308.03188, 2023. [find](https://scholar.google.com/scholar?q=L%2E+Pan%2C+M%2E+Saxon%2C+W%2E+Xu%2C+D%2E+Nathani%2C+X%2E+Wang%2C+and+W%2E+Y%2E+Wang%2E+Automatically+correcting+large+language+models%3A+Surveying+the+landscape+of+diverse+self%2Dcorrection+strategies%2E+arXiv+preprint+arXiv%3A2308%2E0) 22. D. Paul, M. Ismayilzada, M. Peyrard, B. Borges, A. Bosselut, R. West, and B. Faltings. Refiner: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304.01904, 2023. [find](https://scholar.google.com/scholar?q=D%2E+Paul%2C+M%2E+Ismayilzada%2C+M%2E+Peyrard%2C+B%2E+Borges%2C+A%2E+Bosselut%2C+R%2E+West%2C+and+B%2E+Faltings%2E+Refiner%3A+Reasoning+feedback+on+intermediate+representations%2E+arXiv+preprint+arXiv%3A2304%2E01904%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/paul-2023-refiner-reasoning-feedback-intermediate-representations/) 23. Y. Qu, T. Zhang, N. Garg, and A. Kumar. Recursive introspection: Teaching language model agents how to self-improve. arXiv preprint arXiv:2407.18219, 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Qu%2C+T%2E+Zhang%2C+N%2E+Garg%2C+and+A%2E+Kumar%2E+Recursive+introspection%3A+Teaching+language+model+agents+how+to+self%2Dimprove%2E+arXiv+preprint+arXiv%3A2407%2E18219%2C+2024%2E) 24. W. Saunders, C. Yeh, J. Wu, S. Bills, L. Ouyang, J. Ward, and J. Leike. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802, 2022. [find](https://scholar.google.com/scholar?q=W%2E+Saunders%2C+C%2E+Yeh%2C+J%2E+Wu%2C+S%2E+Bills%2C+L%2E+Ouyang%2C+J%2E+Ward%2C+and+J%2E+Leike%2E+Self%2Dcritiquing+models+for+assisting+human+evaluators%2E+arXiv+preprint+arXiv%3A2206%2E05802%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/saunders-2022-self-critiquing-models-assisting-human/) 25. L. Shani, A. Rosenberg, A. Cassel, O. Lang, D. Calandriello, A. Zipori, H. Noga, O. Keller, B. Piot, I. Szpektor, et al. Multi-turn reinforcement learning from preference human feedback. arXiv preprint arXiv:2405.14655, 2024. — [arXiv:2405.14655](https://arxiv.org/abs/2405.14655) [find](https://scholar.google.com/scholar?q=L%2E+Shani%2C+A%2E+Rosenberg%2C+A%2E+Cassel%2C+O%2E+Lang%2C+D%2E+Calandriello%2C+A%2E+Zipori%2C+H%2E+Noga%2C+O%2E+Keller%2C+B%2E+Piot%2C+I%2E+Szpektor%2C+et+al%2E+Multi%2Dturn+reinforcement+learning+from+preference+human+feedback%2E+arXiv+preprin) 26. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. Li, Y. Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. [find](https://scholar.google.com/scholar?q=Z%2E+Shao%2C+P%2E+Wang%2C+Q%2E+Zhu%2C+R%2E+Xu%2C+J%2E+Song%2C+M%2E+Zhang%2C+Y%2E+Li%2C+Y%2E+Wu%2C+and+D%2E+Guo%2E+Deepseekmath%3A+Pushing+the+limits+of+mathematical+reasoning+in+open+language+models%2E+arXiv+preprint+arXiv%3A2402%2E03300%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/) 27. N. Shinn, B. Labash, and A. Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023. [find](https://scholar.google.com/scholar?q=N%2E+Shinn%2C+B%2E+Labash%2C+and+A%2E+Gopinath%2E+Reflexion%3A+an+autonomous+agent+with+dynamic+memory+and+self%2Dreflection%2E+arXiv+preprint+arXiv%3A2303%2E11366%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/) 28. A. Singh, J. D. Co-Reyes, R. Agarwal, A. Anand, P. Patil, P. J. Liu, J. Harrison, J. Lee, K. Xu, A. Parisi, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Singh%2C+J%2E+D%2E+Co%2DReyes%2C+R%2E+Agarwal%2C+A%2E+Anand%2C+P%2E+Patil%2C+P%2E+J%2E+Liu%2C+J%2E+Harrison%2C+J%2E+Lee%2C+K%2E+Xu%2C+A%2E+Parisi%2C+et+al%2E+Beyond+human+data%3A+Scaling+self%2Dtraining+for+problem%2Dsolving+with+language+models%2E+ar) [our card](https://md.research.elamur.ai/free/en/papers/singh-2023-human-data-scaling-self-training/) 29. A. Singh, J. D. Co-Reyes, R. Agarwal, A. Anand, P. Patil, X. Garcia, P. J. Liu, J. Harrison, J. Lee, K. Xu, A. Parisi, A. Kumar, A. Alemi, A. Rizkowsky, A. Nova, B. Adlam, B. Bohnet, G. Elsayed, H. Sedghi, I. Mordatch, I. Simpson, I. Gur, J. Snoek, J. Pennington, J. Hron, K. Kenealy, K. Swersky, K.… [find](https://scholar.google.com/scholar?q=A%2E+Singh%2C+J%2E+D%2E+Co%2DReyes%2C+R%2E+Agarwal%2C+A%2E+Anand%2C+P%2E+Patil%2C+X%2E+Garcia%2C+P%2E+J%2E+Liu%2C+J%2E+Harrison%2C+J%2E+Lee%2C+K%2E+Xu%2C+A%2E+Parisi%2C+A%2E+Kumar%2C+A%2E+Alemi%2C+A%2E+Rizkowsky%2C+A%2E+Nova%2C+B%2E+Adlam%2C+B%2E+Bohnet%2C+G%2E+Elsayed%2C+H%2E+Se) [our card](https://md.research.elamur.ai/free/en/papers/singh-2023-human-data-scaling-self-training/) 30. C. Snell, I. Kostrikov, Y. Su, M. Yang, and S. Levine. Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206.11871, 2022. — [arXiv:2206.11871](https://arxiv.org/abs/2206.11871) [find](https://scholar.google.com/scholar?q=C%2E+Snell%2C+I%2E+Kostrikov%2C+Y%2E+Su%2C+M%2E+Yang%2C+and+S%2E+Levine%2E+Offline+rl+for+natural+language+generation+with+implicit+language+q+learning%2E+arXiv+preprint+arXiv%3A2206%2E11871%2C+2022%2E+%E2%80%94+%5BarXiv%3A2206%2E11871%5D%28https%3A%2F) 31. C. Snell, J. Lee, K. Xu, and A. Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024. [find](https://scholar.google.com/scholar?q=C%2E+Snell%2C+J%2E+Lee%2C+K%2E+Xu%2C+and+A%2E+Kumar%2E+Scaling+llm+test%2Dtime+compute+optimally+can+be+more+effective+than+scaling+model+parameters%2E+arXiv+preprint+arXiv%3A2408%2E03314%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/snell-2024-scaling-llm-test-time-compute/) 32. C. Team. Codegemma: Open code models based on gemma. arXiv preprint arXiv:2406.11409, 2024. [find](https://scholar.google.com/scholar?q=C%2E+Team%2E+Codegemma%3A+Open+code+models+based+on+gemma%2E+arXiv+preprint+arXiv%3A2406%2E11409%2C+2024%2E) 33. G. Tyen, H. Mansoor, V. Cărbune, Y. P. Chen, and T. Mak. Llms cannot find reasoning errors, but can correct them given the error location. In Findings of the Association for Computational Linguistics ACL 2024, pages 13894–13908, 2024. [find](https://scholar.google.com/scholar?q=G%2E+Tyen%2C+H%2E+Mansoor%2C+V%2E+C%C4%83rbune%2C+Y%2E+P%2E+Chen%2C+and+T%2E+Mak%2E+Llms+cannot+find+reasoning+errors%2C+but+can+correct+them+given+the+error+location%2E+In+Findings+of+the+Association+for+Computational+Linguistics+) 34. J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022. [find](https://scholar.google.com/scholar?q=J%2E+Uesato%2C+N%2E+Kushman%2C+R%2E+Kumar%2C+F%2E+Song%2C+N%2E+Siegel%2C+L%2E+Wang%2C+A%2E+Creswell%2C+G%2E+Irving%2C+and+I%2E+Higgins%2E+Solving+math+word+problems+with+process%2Dand+outcome%2Dbased+feedback%2E+arXiv+preprint+arXiv%3A2211%2E1427) [our card](https://md.research.elamur.ai/free/en/papers/uesato-2022-solving-math-word-problems-process/) 35. X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. [find](https://scholar.google.com/scholar?q=X%2E+Wang%2C+J%2E+Wei%2C+D%2E+Schuurmans%2C+Q%2E+Le%2C+E%2E+Chi%2C+S%2E+Narang%2C+A%2E+Chowdhery%2C+and+D%2E+Zhou%2E+Self%2Dconsistency+improves+chain+of+thought+reasoning+in+language+models%2E+arXiv+preprint+arXiv%3A2203%2E11171%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/) 36. S. Welleck, X. Lu, P. West, F. Brahman, T. Shen, D. Khashabi, and Y. Choi. Generating sequences by learning to self-correct. In The Eleventh International Conference on Learning Representations, 2023. URL [https://openreview.net/forum?id=hH36JeQZDaO](https://openreview.net/forum?id=hH36JeQZDaO) . [↗](https://openreview.net/forum?id=hH36JeQZDaO) [our card](https://md.research.elamur.ai/free/en/papers/welleck-2022-generating-sequences-learning-self-correct/) 37. W. Xiong, C. Shi, J. Shen, A. Rosenberg, Z. Qin, D. Calandriello, M. Khalman, R. Joshi, B. Piot, M. Saleh, et al. Building math agents with multi-turn iterative preference learning. arXiv preprint arXiv:2409.02392, 2024. — [arXiv:2409.02392](https://arxiv.org/abs/2409.02392) [find](https://scholar.google.com/scholar?q=W%2E+Xiong%2C+C%2E+Shi%2C+J%2E+Shen%2C+A%2E+Rosenberg%2C+Z%2E+Qin%2C+D%2E+Calandriello%2C+M%2E+Khalman%2C+R%2E+Joshi%2C+B%2E+Piot%2C+M%2E+Saleh%2C+et+al%2E+Building+math+agents+with+multi%2Dturn+iterative+preference+learning%2E+arXiv+preprint+arX) 38. S. Yang, E. Gribovskaya, N. Kassner, M. Geva, and S. Riedel. Do large language models latently perform multi-hop reasoning? arXiv preprint arXiv:2402.16837, 2024. [find](https://scholar.google.com/scholar?q=S%2E+Yang%2C+E%2E+Gribovskaya%2C+N%2E+Kassner%2C+M%2E+Geva%2C+and+S%2E+Riedel%2E+Do+large+language+models+latently+perform+multi%2Dhop+reasoning%3F+arXiv+preprint+arXiv%3A2402%2E16837%2C+2024%2E) 39. S. Ye, Y. Jo, D. Kim, S. Kim, H. Hwang, and M. Seo. Selfee: Iterative self-revising llm empowered by self-feedback generation. Blog post, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Ye%2C+Y%2E+Jo%2C+D%2E+Kim%2C+S%2E+Kim%2C+H%2E+Hwang%2C+and+M%2E+Seo%2E+Selfee%3A+Iterative+self%2Drevising+llm+empowered+by+self%2Dfeedback+generation%2E+Blog+post%2C+2023%2E) 40. T. Ye, Z. Xu, Y. Li, and Z. Allen-Zhu. Physics of language models: Part 2.2, how to learn from mistakes on grade-school math problems, 2024. URL https://arxiv.org/abs/2408.16293 . [↗](https://arxiv.org/abs/2408.16293) 41. M. Yin, G. Tucker, M. Zhou, S. Levine, and C. Finn. Meta-learning without memorization. arXiv preprint arXiv:1912.03820, 2019. — [arXiv:1912.03820](https://arxiv.org/abs/1912.03820) [find](https://scholar.google.com/scholar?q=M%2E+Yin%2C+G%2E+Tucker%2C+M%2E+Zhou%2C+S%2E+Levine%2C+and+C%2E+Finn%2E+Meta%2Dlearning+without+memorization%2E+arXiv+preprint+arXiv%3A1912%2E03820%2C+2019%2E+%E2%80%94+%5BarXiv%3A1912%2E03820%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1912%2E03820%29) 42. E. Zelikman, Y. Wu, J. Mu, and N. Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488, 2022. [find](https://scholar.google.com/scholar?q=E%2E+Zelikman%2C+Y%2E+Wu%2C+J%2E+Mu%2C+and+N%2E+Goodman%2E+Star%3A+Bootstrapping+reasoning+with+reasoning%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A15476%E2%80%9315488%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/) 43. Y. Zhang, M. Khalifa, L. Logeswaran, J. Kim, M. Lee, H. Lee, and L. Wang. Small language models need strong verifiers to self-correct reasoning. arXiv preprint arXiv:2404.17140, 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Zhang%2C+M%2E+Khalifa%2C+L%2E+Logeswaran%2C+J%2E+Kim%2C+M%2E+Lee%2C+H%2E+Lee%2C+and+L%2E+Wang%2E+Small+language+models+need+strong+verifiers+to+self%2Dcorrect+reasoning%2E+arXiv+preprint+arXiv%3A2404%2E17140%2C+2024%2E) 44. H. S. Zheng, S. Mishra, H. Zhang, X. Chen, M. Chen, A. Nova, L. Hou, H.-T. Cheng, Q. V. Le, E. H. Chi, et al. Natural plan: Benchmarking llms on natural language planning. arXiv preprint arXiv:2406.04520, 2024. [find](https://scholar.google.com/scholar?q=H%2E+S%2E+Zheng%2C+S%2E+Mishra%2C+H%2E+Zhang%2C+X%2E+Chen%2C+M%2E+Chen%2C+A%2E+Nova%2C+L%2E+Hou%2C+H%2E%2DT%2E+Cheng%2C+Q%2E+V%2E+Le%2C+E%2E+H%2E+Chi%2C+et+al%2E+Natural+plan%3A+Benchmarking+llms+on+natural+language+planning%2E+arXiv+preprint+arXiv%3A2406%2E04) 45. Y. Zhou, A. Zanette, J. Pan, S. Levine, and A. Kumar. Archer: Training language model agents via hierarchical multi-turn rl. arXiv preprint arXiv:2402.19446, 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Zhou%2C+A%2E+Zanette%2C+J%2E+Pan%2C+S%2E+Levine%2C+and+A%2E+Kumar%2E+Archer%3A+Training+language+model+agents+via+hierarchical+multi%2Dturn+rl%2E+arXiv+preprint+arXiv%3A2402%2E19446%2C+2024%2E)