---
title: "Entropy-Regularized Process Reward Model — Zhang 2024"
description: "Process reward models (PRMs) score each step of a math solution. 41 references, cited by 16 papers here."
authors: ["Hanning Zhang", "Pengcheng Wang", "Shizhe Diao", "Yong Lin", "Rui Pan", "Hanze Dong", "Dylan Zhang", "Pavlo Molchanov", "Tong Zhang"]
year: 2024
venue: "arXiv:2412.11006"
source: "https://arxiv.org/abs/2412.11006"
doi: "10.48550/arxiv.2412.11006"
arxiv: "2412.11006"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2024-entropy-regularized-process-reward-model/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2024-entropy-regularized-process-reward-model/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/12/) / Entropy-Regularized Process Reward Model

# Entropy-Regularized Process Reward Model

**Authors:** [Hanning Zhang](https://md.research.elamur.ai/free/en/authors/hanning-zhang/), Pengcheng Wang, [Shizhe Diao](https://md.research.elamur.ai/free/en/authors/shizhe-diao/), [Yong Lin](https://md.research.elamur.ai/free/en/authors/yong-lin/), [Rui Pan](https://md.research.elamur.ai/free/en/authors/rui-pan/), [Hanze Dong](https://md.research.elamur.ai/free/en/authors/hanze-dong/), Dylan Zhang, [Pavlo Molchanov](https://md.research.elamur.ai/free/en/authors/pavlo-molchanov/) and [Tong Zhang](https://md.research.elamur.ai/free/en/authors/tong-zhang/) · **Year:** 2024 · **Venue:** arXiv:2412.11006 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2412.11006) · [PDF](https://arxiv.org/pdf/2412.11006) · [DOI 10.48550/arxiv.2412.11006](https://doi.org/10.48550/arxiv.2412.11006) · [arXiv 2412.11006](https://arxiv.org/abs/2412.11006) · [OpenAlex](https://openalex.org/W4405469673)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [16](https://md.research.elamur.ai/free/en/papers/zhang-2024-entropy-regularized-process-reward-model/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Process reward models (PRMs) score each step of a math solution. Automatic labelling schemes such as Math-Shepherd estimate a step's value from rollouts, but they rely on a standard MDP. Modern RLHF instead optimizes reward under a KL penalty that keeps the policy near its reference model. This paper closes that gap. It treats multi-step reasoning as a KL-regularized MDP and derives a step-level reward from the initial policy's samples: a log-mean-exponential of final-answer correctness over completions, scaled by a temperature η.

This yields the ER-PRM labelling scheme:
- As η approaches zero, it reduces to soft labels (the average success rate).
- As η grows large, it approaches hard labels (whether any rollout succeeds).

PRMs trained on these labels were tested on MATH and GSM8K, both for best-of-N reranking and for guiding rejection-sampling RLHF. ER-PRM beat the soft-label and hard-label PRM baselines and outcome reward models. This suggests that matching reward construction to the KL-regularized objective used in practice gives cleaner supervision for math reasoning.

## Key points
- Under best-of-N, ER-PRM improves on prior PRMs by about 1% on GSM8K and 2–3% on MATH, and gives over 1% gain when used in RLHF via rejection sampling.
- The best η depends on the dataset: η = 1–2 for the harder MATH and η = 15 for the easier GSM8K. Very small η (e.g., 0.1) performs poorly.
- On MATH500, a Llama-3.1-8B PRM trained on DeepSeek data reranked outputs from OpenMath2-Llama3.1-8B. It reached about 70.5% at N=16, versus 69.3% (soft-label) and 69.7% (hard-label).
- Autoregressive PRM training outperformed a regression variant that predicts a score directly, especially on MATH. Mistral-MetaMath-7b served as the generator in the ablations.
- The authors state that data, code, and checkpoints will be released.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Illinois Urbana-Champaign](https://md.research.elamur.ai/free/en/universities/university-of-illinois-urbana-champaign/) — University, Urbana, United States · 228 works · 74,591 citations
- [Princeton University](https://md.research.elamur.ai/free/en/universities/princeton-university/) — University, Princeton, United States · 137 works · 53,849 citations
- [University of Toronto](https://md.research.elamur.ai/free/en/universities/university-of-toronto/) — University, Toronto, Canada · 121 works · 179,943 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

## Cite

Cite as: Hanning Zhang, Pengcheng Wang, Shizhe Diao, Yong Lin, Rui Pan, Hanze Dong, Dylan Zhang, Pavlo Molchanov and Tong Zhang. 2024. Entropy-Regularized Process Reward Model. arXiv:2412.11006.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhang2024entropy,
  title = {{Entropy-Regularized Process Reward Model}},
  author = {Hanning Zhang, Pengcheng Wang, Shizhe Diao, Yong Lin, Rui Pan, Hanze Dong, Dylan Zhang, Pavlo Molchanov and Tong Zhang},
  year = {2024},
  doi = {10.48550/arxiv.2412.11006},
  eprint = {2412.11006},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.11006}
}
```

- [← previous DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph](https://md.research.elamur.ai/free/en/papers/zhang-2024-darg-dynamic-evaluation-large-language/)
- [next → Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2024-let-expert-stick-his/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/12/)

## References

41 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Saurav+Kadavath%2C+Jackson+Kernion%2C+Tom+Conerly%2C+)
2. Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. [Alphamath almost zero: process supervision without process](https://md.research.elamur.ai/free/en/papers/chen-2024-alphamath-almost-zero-process-supervision/), 2024a. URL https://arxiv.org/abs/2405.03553. [↗](https://arxiv.org/abs/2405.03553)
3. Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Step-level value preference optimization for mathematical reasoning, 2024b. URL https://arxiv.org/abs/2406.10858. [↗](https://arxiv.org/abs/2406.10858)
4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/), 2021. URL https://arxiv.org/abs/2110.14168. [↗](https://arxiv.org/abs/2110.14168)
5. Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. [Raft: Reward ranked finetuning for generative foundation model alignment](https://md.research.elamur.ai/free/en/papers/dong-2023-raft-reward-ranked-finetuning-generative/). arXiv preprint arXiv:2304.06767, 2023.
6. Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. [Rlhf workflow: From reward modeling to online rlhf](https://md.research.elamur.ai/free/en/papers/dong-2024-rlhf-workflow-reward-modeling-online/). arXiv preprint arXiv:2405.07863, 2024.
7. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez,… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
8. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. [Tora: A tool-integrated reasoning agent for mathematical problem solving](https://md.research.elamur.ai/free/en/papers/gou-2023-tora-tool-integrated-reasoning-agent/). arXiv preprint arXiv:2309.17452, 2023.
9. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/), 2021. URL https://arxiv.org/abs/2103.03874. [↗](https://arxiv.org/abs/2103.03874)
10. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
11. Fangkai Jiao, Chengwei Qin, Zhengyuan Liu, Nancy F Chen, and Shafiq Joty. [Learning planning-based reasoning by trajectories collection and process reward synthesizing](https://md.research.elamur.ai/free/en/papers/jiao-2024-learning-planning-based-reasoning-trajectories/). arXiv preprint arXiv:2402.00658, 2024.
12. Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. [Step-dpo: Step-wise preference optimization for long-chain reasoning of llms](https://md.research.elamur.ai/free/en/papers/lai-2024-step-dpo-step-wise-preference/). arXiv preprint arXiv:2406.18629, 2024.
13. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step, 2023. URL — [Let's Verify Step by Step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/) [https://arxiv.org/](https://arxiv.org/) abs/2305.20050. [↗](https://arxiv.org/)
14. Haoxiong Liu and Andrew Chi-Chih Yao. [Augmenting math word problems via iterative question composing](https://md.research.elamur.ai/free/en/papers/liu-2024-augmenting-math-word-problems-iterative/). arXiv preprint arXiv:2401.09003, 2024.
15. Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. [Improve mathematical reasoning in language models by automated process supervision](https://md.research.elamur.ai/free/en/papers/luo-2024-improve-mathematical-reasoning-language-models/), 2024. URL https://arxiv.org/abs/2406.06592. [↗](https://arxiv.org/abs/2406.06592)
16. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
17. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training la… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
18. Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. [Iterative reasoning preference optimization](https://md.research.elamur.ai/free/en/papers/pang-2024-iterative-reasoning-preference-optimization/). arXiv preprint arXiv:2404.19733, 2024.
19. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)
20. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/), 2024. URL https://arxiv.org/abs/2402.03300. [↗](https://arxiv.org/abs/2402.03300)
21. Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. [Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving](https://md.research.elamur.ai/free/en/papers/tong-2024-dart-math-difficulty-aware-rejection/). 2024.
22. Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. [Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data](https://md.research.elamur.ai/free/en/papers/toshniwal-2024-openmathinstruct-2-accelerating-ai-math/). arXiv preprint arXiv:2410.01560, 2024a.
23. Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. [Openmathinstruct-1: A 1.8 million math instruction tuning dataset](https://md.research.elamur.ai/free/en/papers/toshniwal-2024-openmathinstruct-1-1-8-million/). arXiv preprint arXiv:2402.10176, 2024b.
24. Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. [Math-shepherd: Verify and reinforce llms step-by-step without human annotations](https://md.research.elamur.ai/free/en/papers/wang-2024-math-shepherd-verify-reinforce-llms/), 2024. URL https://arxiv.org/abs/2312.08935. [↗](https://arxiv.org/abs/2312.08935)
25. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/), 2023. URL https://arxiv.org/abs/2201.11903. [↗](https://arxiv.org/abs/2201.11903)
26. Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.00451, 2024. [find](https://scholar.google.com/scholar?q=Yuxi+Xie%2C+Anirudh+Goyal%2C+Wenyue+Zheng%2C+Min%2DYen+Kan%2C+Timothy+P+Lillicrap%2C+Kenji+Kawaguchi%2C+and+Michael+Shieh%2E+Monte+carlo+tree+search+boosts+reasoning+via+iterative+preference+learning%2E+arXiv+preprint+)
27. Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. [Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint](https://md.research.elamur.ai/free/en/papers/xiong-2023-iterative-preference-learning-human-feedback/), 2024a. URL https://arxiv.org/abs/2312.11456. [↗](https://arxiv.org/abs/2312.11456)
28. Wei Xiong, Chengshuai Shi, Jiaming Shen, Aviv Rosenberg, Zhen Qin, Daniele Calandriello, Misha Khalman, Rishabh Joshi, Bilal Piot, Mohammad Saleh, Chi Jin, Tong Zhang, and Tianqi Liu. Building math agents with multi-turn iterative preference learning, 2024b. URL https://arxiv.org/abs/2409.02392. — [arXiv:2409.02392](https://arxiv.org/abs/2409.02392) [↗](https://arxiv.org/abs/2409.02392)
29. Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682, 2023. — [arXiv:2312.16682](https://arxiv.org/abs/2312.16682) [find](https://scholar.google.com/scholar?q=Jing+Xu%2C+Andrew+Lee%2C+Sainbayar+Sukhbaatar%2C+and+Jason+Weston%2E+Some+things+are+more+cringe+than+others%3A+Preference+optimization+with+the+pairwise+cringe+loss%2E+arXiv+preprint+arXiv%3A2312%2E16682%2C+2023%2E+%E2%80%94+%5Ba)
30. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. [Qwen2.5-math technical report: Toward mathematical expert model via self-improvement](https://md.research.elamur.ai/free/en/papers/yang-2024-qwen2-5-math-technical-report/), 2024. U…
31. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). arXiv preprint arXiv:2309.12284, 2023.
32. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/), 2024. URL https://arxiv.org/abs/2309.12284. [↗](https://arxiv.org/abs/2309.12284)
33. Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. [Advancing llm reasoning generalists with preference trees](https://md.research.elamur.ai/free/en/papers/yuan-2024-advancing-llm-reasoning-generalists-preference/). arXiv preprint arXiv:2404.02078, 2024.
34. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. [Scaling relationship on learning mathematical reasoning with large language models](https://md.research.elamur.ai/free/en/papers/yuan-2023-scaling-relationship-learning-mathematical-reasoning/). arXiv preprint arXiv:2308.01825, 2023.
35. Xiang Yue, Ge Zhang Xingwei Qu, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. [Mammoth: Building math generalist models through hybrid instruction tuning](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/). arXiv preprint arXiv:2309.05653, 2023.
36. Di Zhang, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang. Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with llama-3 8b, 2024. URL https://arxiv.org/abs/2406.07394. [↗](https://arxiv.org/abs/2406.07394)
37. Tong Zhang. Mathematical analysis of machine learning algorithms. Cambridge University Press, 2023. [find](https://scholar.google.com/scholar?q=Tong+Zhang%2E+Mathematical+analysis+of+machine+learning+algorithms%2E+Cambridge+University+Press%2C+2023%2E)
38. Han Zhong, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf, 2024a. URL https://arxiv.org/abs/2404.18922. — [arXiv:2404.18922](https://arxiv.org/abs/2404.18922) [↗](https://arxiv.org/abs/2404.18922)
39. Han Zhong, Guhao Feng, Wei Xiong, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv:2404.18922, 2024b. — [arXiv:2404.18922](https://arxiv.org/abs/2404.18922) [find](https://scholar.google.com/scholar?q=Han+Zhong%2C+Guhao+Feng%2C+Wei+Xiong%2C+Li+Zhao%2C+Di+He%2C+Jiang+Bian%2C+and+Liwei+Wang%2E+Dpo+meets+ppo%3A+Reinforced+token+optimization+for+rlhf%2E+arXiv+preprint+arXiv%3A2404%2E18922%2C+2024b%2E+%E2%80%94+%5BarXiv%3A2404%2E18922%5D%28https%3A)
40. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/). arXiv preprint arXiv:2205.10625, 2022.
41. Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang. [Solving math word problems via cooperative reasoning induced language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-solving-math-word-problems-cooperative/). arXiv preprint arXiv:2210.16257, 2022.

*Updated 3 November 2026*