---
title: "VinePPO: Refining Credit Assignment in RL Training of LLMs"
description: "They propose VinePPO, which keeps the PPO framework but removes the critic. 65% accuracy. 45 references, cited by 62 papers here."
authors: ["Amirhossein Kazemnejad", "Milad Aghajohari", "Eva Portelance", "Alessandro Sordoni", "Siva Reddy", "Aaron Courville", "Nicolas Le Roux"]
year: 2024
venue: "arXiv:2410.01679"
source: "https://arxiv.org/abs/2410.01679"
doi: "10.48550/arxiv.2410.01679"
arxiv: "2410.01679"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kazemnejad-2024-vineppo-refining-credit-assignment-rl/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kazemnejad-2024-vineppo-refining-credit-assignment-rl/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / VinePPO: Refining Credit Assignment in RL Training of LLMs

# VinePPO: Refining Credit Assignment in RL Training of LLMs

**Authors:** [Amirhossein Kazemnejad](https://md.research.elamur.ai/free/en/authors/amirhossein-kazemnejad/), Milad Aghajohari, Eva Portelance, [Alessandro Sordoni](https://md.research.elamur.ai/free/en/authors/alessandro-sordoni/), [Siva Reddy](https://md.research.elamur.ai/free/en/authors/siva-reddy/), [Aaron Courville](https://md.research.elamur.ai/free/en/authors/aaron-courville/) and [Nicolas Le Roux](https://md.research.elamur.ai/free/en/authors/nicolas-le-roux/) · **Year:** 2024 · **Venue:** arXiv:2410.01679 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2410.01679) · [PDF](https://arxiv.org/pdf/2410.01679) · [DOI 10.48550/arxiv.2410.01679](https://doi.org/10.48550/arxiv.2410.01679) · [arXiv 2410.01679](https://arxiv.org/abs/2410.01679) · [OpenAlex](https://openalex.org/W4403852025)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [62](https://md.research.elamur.ai/free/en/papers/kazemnejad-2024-vineppo-refining-credit-assignment-rl/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
When large language models are trained with reinforcement learning on multi-step reasoning, the reward arrives only at the end. That makes it hard to tell which intermediate steps actually helped, a challenge known as credit assignment. PPO addresses this with a learned value network (critic), yet critic-free methods such as GRPO, RLOO and DPO perform comparably. The authors examine PPO's critic on math reasoning and find that its value estimates are biased and that it ranks candidate next steps barely better than chance.

They propose VinePPO, which keeps the PPO framework but removes the critic. A text environment can return to any partial solution by re-feeding the prefix, so VinePPO samples extra completions from each intermediate step and averages their rewards to get unbiased Monte Carlo value estimates. On MATH and GSM8K, VinePPO outperforms PPO and critic-free baselines and reaches their peak accuracy in less wall-clock time. It also achieves higher test accuracy at the same training accuracy, which matters because hard, verifiable reasoning tasks are scarce.

## Key points
- **Experimental setup:** experiments use the MATH and GSM8K datasets with models including DeepSeekMath 7B (the largest tested) and RhoMath 1.1B. Code is released at github.com/McGill-NLP/VinePPO, and the paper appeared at ICML 2025.
- **Speed:** VinePPO is slower per iteration because of the extra sampling. It still matches the baselines' best results with fewer gradient steps and up to 3.0x less wall-clock time.
- **Diagnostic method:** to build a "ground truth" value for each reasoning step, the authors averaged returns over 256 Monte Carlo samples.
- **Estimation accuracy:** a prediction counted as correct if it fell within 0.05 of the ground truth. By this measure, PPO's critic gradually rises to about 65% accuracy, while VinePPO stays at 70–90% throughout training. On DeepSeekMath 7B / MATH, the reported MAE is roughly 0.03 for VinePPO versus 0.11 for PPO.
- **Error patterns:** PPO's value error grows in later reasoning steps, which the authors attribute to the critic relying on memorization of early, familiar steps. VinePPO's error shrinks as the solution context lengthens. VinePPO also identifies the best next step reliably, whereas PPO's critic stays near chance for most of training.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Université de Montréal](https://md.research.elamur.ai/free/en/universities/universite-de-montreal/) — University, Montreal, Canada · 87 works · 91,597 citations
- [McGill University](https://md.research.elamur.ai/free/en/universities/mcgill-university/) — University, Montreal, Canada · 91 works · 11,178 citations
- [Canadian Institute for Advanced Research](https://md.research.elamur.ai/free/en/labs/canadian-institute-for-advanced-research/) — Research institute, Toronto, Canada · 35 works · 19,316 citations
- [HEC Montréal](https://md.research.elamur.ai/free/en/universities/hec-montreal/) — University, Montreal, Canada · 7 works · 9,970 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville and Nicolas Le Roux. 2024. VinePPO: Refining Credit Assignment in RL Training of LLMs. arXiv:2410.01679.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{kazemnejad2024vineppo,
  title = {{VinePPO: Refining Credit Assignment in RL Training of LLMs}},
  author = {Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville and Nicolas {Le Roux}},
  year = {2024},
  doi = {10.48550/arxiv.2410.01679},
  eprint = {2410.01679},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.01679}
}
```

- [← previous Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning](https://md.research.elamur.ai/free/en/papers/yang-2024-self-distillation-bridges-distribution-gap/)
- [next → ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models](https://md.research.elamur.ai/free/en/papers/chen-2024-allava-harnessing-gpt4v-synthesized-data/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

45 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., and Hooker, S. Back to Basics: Revisiting REINFORCE-style Optimization for Learning from Human Feedback in LLMs. In Ku, L., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the A… [find](https://scholar.google.com/scholar?q=Ahmadian%2C+A%2E%2C+Cremer%2C+C%2E%2C+Galle%CC%81%2C+M%2E%2C+Fadaee%2C+M%2E%2C+Kreutzer%2C+J%2E%2C+Pietquin%2C+O%2E%2C+U%CC%88stu%CC%88n%2C+A%2E%2C+and+Hooker%2C+S%2E+Back+to+Basics%3A+Revisiting+REINFORCE%2Dstyle+Optimization+for+Learning+from+Human+Feedback+in+LL)
2. DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., et al. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/), 2025. URL https://arxiv.org/abs/2501.12948. [↗](https://arxiv.org/abs/2501.12948)
3. Greensmith, E., Bartlett, P. L., and Baxter, J. Variance Reduction Techniques for Gradient Estimates in Reinforcement Learning. In Dietterich, T. G., Becker, S., and Ghahramani, Z. (eds.), Advances in Neural Information Processing Systems 14 [Neural Information Processing Systems: Natural and Synthe… [find](https://scholar.google.com/scholar?q=Greensmith%2C+E%2E%2C+Bartlett%2C+P%2E+L%2E%2C+and+Baxter%2C+J%2E+Variance+Reduction+Techniques+for+Gradient+Estimates+in+Reinforcement+Learning%2E+In+Dietterich%2C+T%2E+G%2E%2C+Becker%2C+S%2E%2C+and+Ghahramani%2C+Z%2E+%28eds%2E%29%2C+Advances+in)
4. Anthony, T., Tian, Z., and Barber, D. Thinking Fast and Slow with Deep Learning and Tree Search. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Ha… [find](https://scholar.google.com/scholar?q=Anthony%2C+T%2E%2C+Tian%2C+Z%2E%2C+and+Barber%2C+D%2E+Thinking+Fast+and+Slow+with+Deep+Learning+and+Tree+Search%2E+In+Guyon%2C+I%2E%2C+von+Luxburg%2C+U%2E%2C+Bengio%2C+S%2E%2C+Wallach%2C+H%2E+M%2E%2C+Fergus%2C+R%2E%2C+Vishwanathan%2C+S%2E+V%2E+N%2E%2C+and+Garn)
5. Problems with Language Models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, 2022. URL [http://papers.nips.cc…](http://papers.nips.cc…) [↗](http://papers.nips.cc…)
6. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. [Measuring Mathematical Problem Solving With the MATH Dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Vanschoren, J. and Yeung, S. (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, Neu…
7. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s Verify Step by Step. In The Twelfth International Conference on Learning Representations, ICLR 2024, [Vienna, Austria, 2024](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). OpenReview.net. URL [https://openreview.net/…](https://openreview.net/…) [↗](https://openreview.net/…)
8. Lin, Z., Gou, Z., Gong, Y., Liu, X., Shen, Y., Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., and Chen, W. [Rho-1: Not All Tokens Are What You Need](https://md.research.elamur.ai/free/en/papers/lin-2024-rho-1-all-tokens-need/). CoRR, abs/2404.07965, 2024. doi: 10.48550/ARXIV.2404.07965. URL https://doi.org/10.48550/arXiv.2404.07965. [↗](https://doi.org/10.48550/arXiv.2404.07965)
9. Huang, S., Noukhovitch, M., Hosseini, A., Rasul, K., Wang, W., and Tunstall, L. The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization. CoRR, abs/2403.17031, 2024. doi: 10.48550/ARXIV.2403.17031. URL https://doi.org/10.48550/arXiv.2403.17031. [↗](https://doi.org/10.48550/arXiv.2403.17031)
10. Luo, L., Liu, Y., Liu, R., Phatale, S., Lara, H., Li, Y., Shu, L., Zhu, Y., Meng, L., Sun, J., and Rastogi, A. [Improve Mathematical Reasoning in Language Models by Automated Process Supervision](https://md.research.elamur.ai/free/en/papers/luo-2024-improve-mathematical-reasoning-language-models/). CoRR, abs/2406.06592, 2024. doi: 10.48550/ARXIV.2406.06592. URL https://doi.org/10.48550/arXiv.2406.0659… [↗](https://doi.org/10.48550/arXiv.2406.0659…)
11. Hwang, H., Kim, D., Kim, S., Ye, S., and Seo, M. Self-explore to Avoid the Pit: Improving the Reasoning Capabilities of Language Models with Fine-grained Rewards. CoRR, abs/2404.10346, 2024. doi: 10.48550/ARXIV.2404.10346. URL https://doi.org/10.48550/arXiv.2404.10346. — [doi:10.48550/arxiv.2404.10346](https://doi.org/10.48550/arxiv.2404.10346) [↗](https://doi.org/10.48550/arXiv.2404.10346)
12. Ma, Q., Zhou, H., Liu, T., Yuan, J., Liu, P., You, Y., and Yang, H. Let’s reward step by step: Step-level reward model as the Navigators for Reasoning. CoRR, abs/2310.10080, 2023. doi: 10.48550/ ARXIV.2310.10080. URL https://doi.org/ 10.48550/arXiv.2310.10080. [find](https://scholar.google.com/scholar?q=Ma%2C+Q%2E%2C+Zhou%2C+H%2E%2C+Liu%2C+T%2E%2C+Yuan%2C+J%2E%2C+Liu%2C+P%2E%2C+You%2C+Y%2E%2C+and+Yang%2C+H%2E+Let%E2%80%99s+reward+step+by+step%3A+Step%2Dlevel+reward+model+as+the+Navigators+for+Reasoning%2E+CoRR%2C+abs%2F2310%2E10080%2C+2023%2E+doi%3A+10%2E48550%2F+ARXIV)
13. Ivison, H., Wang, Y., Liu, J., Wu, Z., Pyatkin, V., Lambert, N., Smith, N. A., Choi, Y., and Hajishirzi, H. [Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback](https://md.research.elamur.ai/free/en/papers/ivison-2024-unpacking-dpo-ppo-disentangling-best/). CoRR, abs/2406.09279, 2024. doi: 10.48550/ARXIV.2406.09279. URL https://doi.org/10.48550/arXiv.2406.… [↗](https://doi.org/10.48550/arXiv.2406.…)
14. OpenAI. [OpenAI o1 System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/), 2024. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 272684752. [↗](https://api.semanticscholar.org/CorpusID:)
15. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions w… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
16. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Flinn, J., Seltzer, M. I., Druschel, P., Kaufmann, A., and Mace, J. (eds.), Proceedings of the 29th Symposium on…
17. Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V. V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., Wu, Y., Neyshabur, B., Gur-Ari, G., and Misra, V. Solving Quantitative Reasoning Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., and White, C. Smaug: Fi… — [Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive](https://md.research.elamur.ai/free/en/papers/pal-2024-smaug-fixing-failure-modes-preference/)
18. Setlur, A., Garg, S., Geng, X., Garg, N., Smith, V., and Kumar, A. RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-fold. CoRR, abs/2406.14532, 2024. doi: 10.48550/ ARXIV.2406.14532. URL https://doi.org/ 10.48550/arXiv.2406.14532. [find](https://scholar.google.com/scholar?q=Setlur%2C+A%2E%2C+Garg%2C+S%2E%2C+Geng%2C+X%2E%2C+Garg%2C+N%2E%2C+Smith%2C+V%2E%2C+and+Kumar%2C+A%2E+RL+on+Incorrect+Synthetic+Data+Scales+the+Efficiency+of+LLM+Math+Reasoning+by+Eight%2Dfold%2E+CoRR%2C+abs%2F2406%2E14532%2C+2024%2E+doi%3A+10%2E48550%2F+)
19. Putta, P., Mills, E., Garg, N., Motwani, S., Finn, C., Garg, D., and Rafailov, R. [Agent q: Advanced reasoning and learning for autonomous ai agents](https://md.research.elamur.ai/free/en/papers/putta-2024-agent-q-advanced-reasoning-learning/), 2024. URL https://arxiv.org/abs/2408.07199. [↗](https://arxiv.org/abs/2408.07199)
20. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. DeepSeek-Math: Pushing the Limits of Mathematical Reasoning in Open Language Models. CoRR, abs/2402.[03300, 2024](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). doi: 10.48550/ARXIV.2402.03300. URL https://doi.org/10.48550/arXiv.2402.03300. [↗](https://doi.org/10.48550/arXiv.2402.03300)
21. Qwen. Qwen2.5-Math: The world’s leading open-sourced mathematical LLMs. [https://qwenlm.github.io/blog/qwen2.5-math/](https://qwenlm.github.io/blog/qwen2.5-math/), 2024. Accessed: 2024-09-23. [↗](https://qwenlm.github.io/blog/qwen2.5-math/)
22. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T. P., Leach, M., Kavukcuoglu, K., Graepel, T., and Hassabis, D. Mast… [find](https://scholar.google.com/scholar?q=Silver%2C+D%2E%2C+Huang%2C+A%2E%2C+Maddison%2C+C%2E+J%2E%2C+Guez%2C+A%2E%2C+Sifre%2C+L%2E%2C+van+den+Driessche%2C+G%2E%2C+Schrittwieser%2C+J%2E%2C+Antonoglou%2C+I%2E%2C+Panneershelvam%2C+V%2E%2C+Lanctot%2C+M%2E%2C+Dieleman%2C+S%2E%2C+Grewe%2C+D%2E%2C+Nham%2C+J%2E%2C+Kalchbrenner%2C)
23. Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: A…
24. Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., Lillicrap, T. P., Simonyan, K., and Hassabis, D. [Mastering Chess and Shogi by Self-play with a General Reinforcement Learning Algorithm](https://md.research.elamur.ai/free/en/papers/al-2017-mastering-chess-shogi-self-play/). CoRR, abs/1712.01815, 2017. URL htt…
25. Schulman, J. Notes on the KL-divergence Approximation. [http://joschu.net/blog/kl-approx.html](http://joschu.net/blog/kl-approx.html), 2020. Accessed: 2024-09-23. [↗](http://joschu.net/blog/kl-approx.html)
26. Singh, A., Co-Reyes, J. D., Agarwal, R., Anand, A., Patil, P., Garcia, X., Liu, P. J., Harrison, J., Lee, J., Xu, K., Parisi, A. T., Kumar, A., Alemi, A. A., Rizkowsky, A., Nova, A., Adlam, B., Bohnet, B., Elsayed, G. F., Sedghi, H., Mordatch, I., Simpson, I., Gur, I., Snoek, J., Pennington, J., Hro… — [Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models](https://md.research.elamur.ai/free/en/papers/singh-2023-human-data-scaling-self-training/)
27. Schulman, J., Levine, S., Abbeel, P., Jordan, M. I., and Moritz, P. Trust Region Policy Optimization. In Bach, F. R. and Blei, D. M. (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 1889–1897, Lille, F… [find](https://scholar.google.com/scholar?q=Schulman%2C+J%2E%2C+Levine%2C+S%2E%2C+Abbeel%2C+P%2E%2C+Jordan%2C+M%2E+I%2E%2C+and+Moritz%2C+P%2E+Trust+Region+Policy+Optimization%2E+In+Bach%2C+F%2E+R%2E+and+Blei%2C+D%2E+M%2E+%28eds%2E%29%2C+Proceedings+of+the+32nd+International+Conference+on+Machine)
28. Schulman, J., Moritz, P., Levine, S., Jordan, M. I., and Abbeel, P. High-dimensional Continuous Control Using Generalized Advantage Estimation. In Bengio, Y. and LeCun, Y. (eds.), 4th International Conference on Learning Representations, ICLR 2016Proceedings, San Juan, Puerto Rico, 2016. URL http://… [find](https://scholar.google.com/scholar?q=Schulman%2C+J%2E%2C+Moritz%2C+P%2E%2C+Levine%2C+S%2E%2C+Jordan%2C+M%2E+I%2E%2C+and+Abbeel%2C+P%2E+High%2Ddimensional+Continuous+Control+Using+Generalized+Advantage+Estimation%2E+In+Bengio%2C+Y%2E+and+LeCun%2C+Y%2E+%28eds%2E%29%2C+4th+International+Co)
29. Sutton, R. S. and Barto, A. G. Introduction to Reinforcement Learning. In Introduction to Reinforcement Learning, 1998. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 261579713. [↗](https://api.semanticscholar.org/CorpusID:)
30. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal Policy Optimization Algorithms. CoRR, abs/1707.06347, 2017. URL http://arxiv.org/abs/1707.06347. [↗](http://arxiv.org/abs/1707.06347)
31. Sutton, R. S., McAllester, D. A., Singh, S., and Mansour, Y. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In Solla, S. A., Leen, T. K., and Müller, K. (eds.), Search Boosts Reasoning via Iterative Preference Learning. CoRR, abs/2405.00451, 2024. doi: 10.48550/ ARX… [find](https://scholar.google.com/scholar?q=Sutton%2C+R%2E+S%2E%2C+McAllester%2C+D%2E+A%2E%2C+Singh%2C+S%2E%2C+and+Mansour%2C+Y%2E+Policy+Gradient+Methods+for+Reinforcement+Learning+with+Function+Approximation%2E+In+Solla%2C+S%2E+A%2E%2C+Leen%2C+T%2E+K%2E%2C+and+Mu%CC%88ller%2C+K%2E+%28eds%2E%29%2C+Searc)
32. Advances in Neural Information Processing Systems 12, [NIPS Conference, pp. 1057–1063, Denver, Colorado, USA, 1999. The MIT Press. URL [http://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation](http://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation). [↗](http://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation)
33. Xu, S., Fu, W., Gao, J., Ye, W., Liu, W., Mei, Z., Wang, G., Yu, C., and Wu, Y. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, 2024. OpenReview.net. URL [https://openreview.net/forum?id=6XH8R7Y…](https://openreview.net/forum?id=6XH8R7Y…) [↗](https://openreview.net/forum?id=6XH8R7Y…)
34. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Canton-Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., et… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
35. Yuan, Z., Yuan, H., Li, C., Dong, G., Tan, C., and Zhou, C. [Scaling Relationship on Learning Mathematical Reasoning with Large Language Models](https://md.research.elamur.ai/free/en/papers/yuan-2023-scaling-relationship-learning-mathematical-reasoning/). CoRR, abs/2308.01825, 2023. doi: 10.48550/ARXIV.2308.01825. URL https://doi.org/10.48550/arXiv.2308.01825. [↗](https://doi.org/10.48550/arXiv.2308.01825)
36. Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., De Cola, G., Deleu, T., Goulão, M., Kallinteris, A., Krimmel, M., KG, A., et al. [Gymnasium: A standard interface for reinforcement learning environments](https://md.research.elamur.ai/free/en/papers/towers-2024-gymnasium-standard-interface-reinforcement-learning/). arXiv preprint arXiv:2407.17032, 2024.
37. Zhang, D., Zhoubian, S., Yue, Y., Dong, Y., and Tang, J. [ReST-MCTS*: LLM Self-training via Process Reward Guided Tree Search](https://md.research.elamur.ai/free/en/papers/zhang-2024-rest-mcts-llm-self-training/). CoRR, abs/2406.03816, 2024. doi: 10.48550/ARXIV.2406.03816. URL https://doi.org/10.48550/arXiv.2406.03816. [↗](https://doi.org/10.48550/arXiv.2406.03816)
38. Trung, L. Q., Zhang, X., Jie, Z., Sun, P., Jin, X., and Li, H. [ReFT: Reasoning with Reinforced Fine-tuning](https://md.research.elamur.ai/free/en/papers/trung-2024-reft-reasoning-reinforced-fine-tuning/). In Ku, L., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, pp. 7601–7614, Bangk…
39. Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. [Sglang: Efficient execution of structured language model programs](https://md.research.elamur.ai/free/en/papers/zheng-2023-sglang-efficient-execution-structured-language/). CoRR, abs/2312.07104, 2024. URL [https://arxiv.org/](https://arxiv.org/) abs/2312.07104. [↗](https://arxiv.org/)
40. Zheng, R., Dou, S., Gao, S., Hua, Y., Shen, W., Wang, B., Liu, Y., Jin, S., Liu, Q., Zhou, Y., Xiong, L., Chen, L., Xi, Z., Xu, N., Lai, W., Zhu, M., Chang, C., Yin, Z., Weng, R., Cheng, W., Huang, H., Sun, T., Yan, H., Gui, T., Zhang, Q., Qiu, X., and Huang, X. Secrets of RLHF in Large Language Mod… — [Secrets of RLHF in Large Language Models Part I: PPO](https://md.research.elamur.ai/free/en/papers/zheng-2023-secrets-rlhf-large-language-models/)
41. Uesato, J., Kushman, N., Kumar, R., Song, H. F., Siegel, N. Y., Wang, L., Creswell, A., Irving, G., and Higgins, I. Solving math word problems with process-and outcome-based feedback. CoRR, abs/2211.[14275, 2022](https://md.research.elamur.ai/free/en/papers/uesato-2022-solving-math-word-problems-process/). doi: 10.48550/ARXIV.2211.14275. URL https://doi.org/10.48550/arXiv.2211.14275. [↗](https://doi.org/10.48550/arXiv.2211.14275)
42. Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P. F., and Irving, G. Fine-tuning Language Models from Human Preferences. CoRR, abs/1909.08593, 2019. URL http://arxiv.org/abs/1909.08593. [↗](http://arxiv.org/abs/1909.08593)
43. Wan, Z., Feng, X., Wen, M., McAleer, S. M., Wen, Y., Zhang, W., and Wang, J. AlphaZero-like Tree-search can Guide Large Language Model Decoding and Training. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, 2024. OpenReview.net. URL [https://openreview.net/](https://openreview.net/) for… [↗](https://openreview.net/)
44. Wang, P., Li, L., Shao, Z., Xu, R. X., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. CoRR, abs/2406.[06592, 2024](https://md.research.elamur.ai/free/en/papers/luo-2024-improve-mathematical-reasoning-language-models/). URL https://arxiv.org/abs/2312.08935. [↗](https://arxiv.org/abs/2312.08935)
45. Xie, Y., Goyal, A., Zheng, W., Kan, M., Lillicrap, T. P., Kawaguchi, K., and Shieh, M. Monte Carlo Tree [find](https://scholar.google.com/scholar?q=Xie%2C+Y%2E%2C+Goyal%2C+A%2E%2C+Zheng%2C+W%2E%2C+Kan%2C+M%2E%2C+Lillicrap%2C+T%2E+P%2E%2C+Kawaguchi%2C+K%2E%2C+and+Shieh%2C+M%2E+Monte+Carlo+Tree)

*Updated 2 November 2026*