---
title: "R-PRM: Reasoning-Driven Process Reward Modeling (2025)"
description: "Process Reward Models (PRMs) score each step of an LLM's mathematical… 8.5 F1 on PRMBench. 26 references, cited by 16 papers here."
authors: ["Shuaijie She", "Junxiao Liu", "Yifeng Liu", "Jiajun Chen", "Xin Huang", "Shujian Huang"]
year: 2025
venue: "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2025.emnlp-main.679/"
doi: "10.18653/v1/2025.emnlp-main.679"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/she-2025-r-prm-reasoning-driven-process/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/she-2025-r-prm-reasoning-driven-process/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/) / R-PRM: Reasoning-Driven Process Reward Modeling

# R-PRM: Reasoning-Driven Process Reward Modeling

**Authors:** [Shuaijie She](https://md.research.elamur.ai/free/en/authors/shuaijie-she/), Junxiao Liu, [Yifeng Liu](https://md.research.elamur.ai/free/en/authors/yifeng-liu/), [Jiajun Chen](https://md.research.elamur.ai/free/en/authors/jiajun-chen/), [Xin Huang](https://md.research.elamur.ai/free/en/authors/xin-huang/) and [Shujian Huang](https://md.research.elamur.ai/free/en/authors/shujian-huang/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2025](https://md.research.elamur.ai/free/en/venues/emnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.emnlp-main.679/) · [PDF](https://aclanthology.org/2025.emnlp-main.679.pdf) · [DOI 10.18653/v1/2025.emnlp-main.679](https://doi.org/10.18653/v1/2025.emnlp-main.679) · [OpenAlex](https://openalex.org/W4416036437)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [16](https://md.research.elamur.ai/free/en/papers/she-2025-r-prm-reasoning-driven-process/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Process Reward Models (PRMs) score each step of an LLM's mathematical reasoning. Most existing PRMs output a score directly, which limits how well they learn and gives no explanation for a judgment. Step-level labeled data is also scarce. R-PRM addresses this by having the reward model write an analysis of each step before giving a Yes/No verdict. The analysis covers earlier steps, the current step's goal and data sources, its consistency with what came before, and its calculations.

Training has three stages:
- **Cold start:** a stronger LLM generates analyses for PRM800K samples, and only those whose verdicts match the human labels are kept. These are used to fine-tune Qwen2.5-Math-7B-Instruct.
- **Self-improvement:** DPO-based preference optimization refines the model without any new annotations.
- **Inference-time scaling:** several evaluation trajectories are sampled and their verdicts aggregated.

R-PRM clearly outperforms baselines trained on the same data and works well for guiding solution search. This shows that reasoning-based judges can make up for limited step-level labels.

## Key points
- Over the strongest same-data baseline, R-PRM gains 13.9 F1 on ProcessBench and 8.5 F1 on PRMBench.
- With Best-of-N and Guided Search, accuracy rises by about 8.6 and 8.4 points over Pass@1, beating majority voting and prior PRMs. The test sets were MATH500, Minerva Math, OlympiadBench, College Math, AIME24 and AMC23, with Qwen2.5-7B-Instruct as the policy model.
- Accuracy keeps improving as more evaluation trajectories are sampled. The authors suggest K = 4 as a good balance between cost and accuracy.
- R-PRM is less sensitive to the classification threshold than Qwen2.5-Math-7B-PRM800K, including on out-of-domain OlympiadBench.
- Experiments cover only 7B models. Larger models and search methods such as MCTS or Beam Search remain untested.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Nanjing University](https://md.research.elamur.ai/free/en/universities/nanjing-university/) — University, Nanjing, China · 74 works · 7,814 citations
- [China Mobile](https://md.research.elamur.ai/free/en/companies/china-mobile/) — Company, Beijing, China · 9 works · 21 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Shuaijie She, Junxiao Liu, Yifeng Liu, Jiajun Chen, Xin Huang and Shujian Huang. 2025. R-PRM: Reasoning-Driven Process Reward Modeling. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{she2025r,
  title = {{R-PRM: Reasoning-Driven Process Reward Modeling}},
  author = {Shuaijie She, Junxiao Liu, Yifeng Liu, Jiajun Chen, Xin Huang and Shujian Huang},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.679},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.679}
}
```

- [← previous OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization](https://md.research.elamur.ai/free/en/papers/he-2025-openwebvoyager-building-multimodal-web-agents/)
- [next → Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoning](https://md.research.elamur.ai/free/en/papers/shi-2025-search-refine-during-think-facilitating/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/)

## References

26 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). Preprint, arXiv:2110.14168.
2. DeepSeek-AI. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). Preprint, arXiv:2501.12948.
3. Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. 2024. Rlhf workflow: From reward modeling to online rlhf. Preprint, arXiv:2405.07863. [find](https://scholar.google.com/scholar?q=Hanze+Dong%2C+Wei+Xiong%2C+Bo+Pang%2C+Haoxiang+Wang%2C+Han+Zhao%2C+Yingbo+Zhou%2C+Nan+Jiang%2C+Doyen+Sahoo%2C+Caiming+Xiong%2C+and+Tong+Zhang%2E+2024%2E+Rlhf+workflow%3A+From+reward+modeling+to+online+rlhf%2E+Preprint%2C+arXiv%3A2)
4. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. 2024. [Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems](https://md.research.elamur.ai/free/en/papers/he-2024-olympiadbench-challenging-benchmark-promoting-agi/). arXiv preprint arXiv:2402.14008.
5. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. [Scaling llm test-time compute optimally can be more effective than scaling model parameters](https://md.research.elamur.ai/free/en/papers/snell-2024-scaling-llm-test-time-compute/). arXiv preprint arXiv:2408.03314.
6. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). Advances in Neural Information Processing Systems, 35:3843– 3857.
7. Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng. 2025. Prmbench: A fine-grained and challenging benchmark for process-level reward models. Preprint, arXiv:2501.03124. [find](https://scholar.google.com/scholar?q=Mingyang+Song%2C+Zhaochen+Su%2C+Xiaoye+Qu%2C+Jiawei+Zhou%2C+and+Yu+Cheng%2E+2025%2E+Prmbench%3A+A+fine%2Dgrained+and+challenging+benchmark+for+process%2Dlevel+reward+models%2E+Preprint%2C+arXiv%3A2501%2E03124%2E)
8. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). arXiv preprint arXiv:2305.20050.
9. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Yansong Tang, and Dongmei Zhang. 2025. [Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/). Preprint, arXiv:2308.09583.
10. Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. 2024. Easy-to-hard generalization: Scalable alignment beyond human supervision. Preprint, arXiv:2403.09472. [find](https://scholar.google.com/scholar?q=Zhiqing+Sun%2C+Longhui+Yu%2C+Yikang+Shen%2C+Weiyang+Liu%2C+Yiming+Yang%2C+Sean+Welleck%2C+and+Chuang+Gan%2E+2024%2E+Easy%2Dto%2Dhard+generalization%3A+Scalable+alignment+beyond+human+supervision%2E+Preprint%2C+arXiv%3A2403%2E09472)
11. Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. 2024. Mathscale: Scaling instruction tuning for mathematical reasoning. arXiv preprint arXiv:2403.02884. [find](https://scholar.google.com/scholar?q=Zhengyang+Tang%2C+Xingxing+Zhang%2C+Benyou+Wang%2C+and+Furu+Wei%2E+2024%2E+Mathscale%3A+Scaling+instruction+tuning+for+mathematical+reasoning%2E+arXiv+preprint+arXiv%3A2403%2E02884%2E)
12. Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. [Solving math word problems with process- and outcome-based feedback](https://md.research.elamur.ai/free/en/papers/uesato-2022-solving-math-word-problems-process/). Preprint, arXiv:2211.14275.
13. Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. 2024a. [Improve mathematical reasoning in language models by automated process supervision](https://md.research.elamur.ai/free/en/papers/luo-2024-improve-mathematical-reasoning-language-models/). Preprint, arXiv:2406.06592.
14. Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M. Ni, Linyi Yang, Ying Wen, and Weinan Zhang. 2024a. Openr: An open source framework for advanced reasoning with large language models. Preprint, arXiv:2410.09671. [find](https://scholar.google.com/scholar?q=Jun+Wang%2C+Meng+Fang%2C+Ziyu+Wan%2C+Muning+Wen%2C+Jiachen+Zhu%2C+Anjie+Liu%2C+Ziqin+Gong%2C+Yan+Song%2C+Lei+Chen%2C+Lionel+M%2E+Ni%2C+Linyi+Yang%2C+Ying+Wen%2C+and+Weinan+Zhang%2E+2024a%2E+Openr%3A+An+open+source+framework+for+adva)
15. Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. 2024b. [Improve mathematical reasoning in language models by automated process supervision](https://md.research.elamur.ai/free/en/papers/luo-2024-improve-mathematical-reasoning-language-models/). arXiv preprint arXiv:2406.06592.
16. Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024b. [Math-shepherd: Verify and reinforce llms step-by-step without human annotations](https://md.research.elamur.ai/free/en/papers/wang-2024-math-shepherd-verify-reinforce-llms/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape…
17. OpenAI O1. 2023. Learning to reason with llms. [https://openai.com/index/](https://openai.com/index/) learning-to-reason-with-llms/. Accessed: 2025-02-08. [↗](https://openai.com/index/)
18. Skywork o1 Team. 2024. Skywork-o1 open series. [https://huggingface.co/Skywork](https://huggingface.co/Skywork). [↗](https://huggingface.co/Skywork)
19. QwQ. 2023. Qwq: Reflect deeply on the boundaries of the unknown. [https://qwenlm.github.io/blog/](https://qwenlm.github.io/blog/) qwq-32b-preview//. Accessed: 2025-02-08. [↗](https://qwenlm.github.io/blog/)
20. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). Preprint, arXiv:2305.18290.
21. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). Preprint, arXiv:2402.03300.
22. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. Preprint, arXiv:2212.10560. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A%2E+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2023%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+Preprin)
23. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Preprint, arXiv:2201.11903.
24. Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, and Pengfei Liu. 2025. [Evaluating mathematical reasoning beyond accuracy](https://md.research.elamur.ai/free/en/papers/xia-2024-evaluating-mathematical-reasoning-accuracy/). Preprint, arXiv:2404.05692.
25. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024. [Qwen2.5-math technical report: Toward mathematical expert model via self-improvement](https://md.research.elamur.ai/free/en/papers/yang-2024-qwen2-5-math-technical-report/). P…
26. Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2024. [Processbench: Identifying process errors in mathematical reasoning](https://md.research.elamur.ai/free/en/papers/zheng-2025-processbench-identifying-process-errors-mathematical/). Preprint, arXiv:2412.06559.

*Updated 31 October 2026*