---
title: "RM-R1: Reward Modeling as Reasoning (2025)"
description: "The authors treat preference judgment as a reasoning problem and introduce Reasoning Reward… 47 references, cited by 47 papers here."
authors: ["Xiusi Chen", "Gaotang Li", "Ziqi Wang", "Bowen Jin", "Cheng Qian", "Yu Wang", "Hongru Wang", "Yu Zhang", "Denghui Zhang", "Tong Zhang", "Hanghang Tong", "Heng Ji"]
year: 2025
venue: "arXiv:2505.02387"
source: "https://arxiv.org/abs/2505.02387"
doi: "10.48550/arxiv.2505.02387"
arxiv: "2505.02387"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2025-rm-r1-reward-modeling-reasoning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2025-rm-r1-reward-modeling-reasoning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/) / RM-R1: Reward Modeling as Reasoning

# RM-R1: Reward Modeling as Reasoning

**Authors:** [Xiusi Chen](https://md.research.elamur.ai/free/en/authors/xiusi-chen/), Gaotang Li, [Ziqi Wang](https://md.research.elamur.ai/free/en/authors/ziqi-wang/), [Bowen Jin](https://md.research.elamur.ai/free/en/authors/bowen-jin/), [Cheng Qian](https://md.research.elamur.ai/free/en/authors/cheng-qian/), [Yu Wang](https://md.research.elamur.ai/free/en/authors/yu-wang/), [Hongru Wang](https://md.research.elamur.ai/free/en/authors/hongru-wang/), [Yu Zhang](https://md.research.elamur.ai/free/en/authors/yu-zhang/), [Denghui Zhang](https://md.research.elamur.ai/free/en/authors/denghui-zhang/), [Tong Zhang](https://md.research.elamur.ai/free/en/authors/tong-zhang/), [Hanghang Tong](https://md.research.elamur.ai/free/en/authors/hanghang-tong/) and [Heng Ji](https://md.research.elamur.ai/free/en/authors/heng-ji/) · **Year:** 2025 · **Venue:** arXiv:2505.02387 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2505.02387) · [PDF](https://arxiv.org/pdf/2505.02387) · [DOI 10.48550/arxiv.2505.02387](https://doi.org/10.48550/arxiv.2505.02387) · [arXiv 2505.02387](https://arxiv.org/abs/2505.02387) · [OpenAlex](https://openalex.org/W4417299315)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [47](https://md.research.elamur.ai/free/en/papers/chen-2025-rm-r1-reward-modeling-reasoning/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Reward models guide the alignment of LLMs, but each existing family has a weakness. Scalar RMs give no justification for their scores. Generative RMs explain their verdicts, yet their reasoning is often shallow. The authors treat preference judgment as a reasoning problem and introduce Reasoning Reward Models (ReasRMs), trained as the RM-R1 family (7B–32B).

Training has two stages:
- **Distillation:** an instruct model such as Qwen-2.5-14B-Instruct learns from structured judgment traces written by a strong "oracle" model.
- **RLVR:** reinforcement learning with verifiable rewards uses correctness of the preferred-response label as the reward.

At inference, a Chain-of-Rubrics procedure first classifies the input as chat or reasoning. For chat, the model writes tailored rubrics and grades both responses against them. For reasoning tasks, it solves the problem itself before comparing.

Averaged over RewardBench, RM-Bench and RMB, RM-R1 beats far larger open and proprietary judges by up to 4.9%. This shows that explicit, inspectable reasoning makes reward signals both more accurate and more transparent.

## Key points
- Benchmarks are RewardBench, RM-Bench and RMB. Baselines that RM-R1 surpasses include INF-ORM-Llama3.1-70B, GPT-4o and Claude models, with a margin of up to 4.9% on average.
- Distillation traces come from oracle models such as o3 or Claude-3-7-sonnet. Training uses an NLL loss on the trace concatenated with the gold label.
- Ablations show that RLVR alone underuses the model's reasoning ability, and plain CoT misses distinctions between question types. This motivates distillation before RL, plus the Chain-of-Rubrics procedure.
- Existing reasoning models can also be converted into reward models using RLVR alone, with no extra distillation stage.
- A theoretical appendix argues that SFT barely penalizes shortcut features that happen to match the labels in the training data. RL penalizes such shortcuts much more strongly, which explains why the RL stage is needed.
- Code, data and models are public at github.com/RM-R1-UIUC/RM-R1.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Illinois Urbana-Champaign](https://md.research.elamur.ai/free/en/universities/university-of-illinois-urbana-champaign/) — University, Urbana, United States · 209 works · 71,394 citations
- [University of California San Diego](https://md.research.elamur.ai/free/en/universities/university-of-california-san-diego/) — University, San Diego, United States · 89 works · 54,032 citations
- [Texas A&M University](https://md.research.elamur.ai/free/en/universities/texas-a-m-university/) — University, College Station, United States · 29 works · 1,871 citations
- [Stevens Institute of Technology](https://md.research.elamur.ai/free/en/universities/stevens-institute-of-technology/) — University, Hoboken, United States · 13 works · 95 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, Hanghang Tong and Heng Ji. 2025. RM-R1: Reward Modeling as Reasoning. arXiv:2505.02387.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chen2025rm,
  title = {{RM-R1: Reward Modeling as Reasoning}},
  author = {Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, Hanghang Tong and Heng Ji},
  year = {2025},
  doi = {10.48550/arxiv.2505.02387},
  eprint = {2505.02387},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.02387}
}
```

- [← previous Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges](https://md.research.elamur.ai/free/en/papers/ma-2025-pragmatics-era-large-language-models/)
- [next → OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis](https://md.research.elamur.ai/free/en/papers/sun-2025-os-genesis-automating-gui-agent/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/)

## References

47 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
2. Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, et al. [Nemotron-4 340b technical report](https://md.research.elamur.ai/free/en/papers/adler-2024-nemotron-4-340b-technical-report/). arXiv preprint arXiv:2406.11704, 2024.
3. AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 1:1, 2024. [find](https://scholar.google.com/scholar?q=AI+Anthropic%2E+The+claude+3+model+family%3A+Opus%2C+sonnet%2C+haiku%2E+Claude%2D3+Model+Card%2C+1%3A1%2C+2024%2E)
4. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073, 2022.
5. Chris L Baker, Rebecca Saxe, and Joshua B Tenenbaum. Action understanding as inverse planning. Cognition, 113(3):329–349, 2009. [find](https://scholar.google.com/scholar?q=Chris+L+Baker%2C+Rebecca+Saxe%2C+and+Joshua+B+Tenenbaum%2E+Action+understanding+as+inverse+planning%2E+Cognition%2C+113%283%29%3A329%E2%80%93349%2C+2009%2E)
6. Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. [Internlm2 technical report](https://md.research.elamur.ai/free/en/papers/cai-2024-internlm2-technical-report/). arXiv preprint arXiv:2403.17297, 2024.
7. Nuo Chen, Zhiyuan Hu, Qingyun Zou, Jiaying Wu, Qian Wang, Bryan Hooi, and Bingsheng He. Judgelrm: Large reasoning models as a judge. arXiv preprint arXiv:2504.00050, 2025. — [arXiv:2504.00050](https://arxiv.org/abs/2504.00050) [find](https://scholar.google.com/scholar?q=Nuo+Chen%2C+Zhiyuan+Hu%2C+Qingyun+Zou%2C+Jiaying+Wu%2C+Qian+Wang%2C+Bryan+Hooi%2C+and+Bingsheng+He%2E+Judgelrm%3A+Large+reasoning+models+as+a+judge%2E+arXiv+preprint+arXiv%3A2504%2E00050%2C+2025%2E+%E2%80%94+%5BarXiv%3A2504%2E00050%5D%28https%3A%2F)
8. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. [find](https://scholar.google.com/scholar?q=Paul+F+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+neural+information+processing+systems%2C+30%2C+2017%2E)
9. Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. [SFT memorizes, RL generalizes: A comparative study of foundation model post-training](https://md.research.elamur.ai/free/en/papers/chu-2025-sft-memorizes-rl-generalizes-comparative/). In Forty-second International Conference on Machine Learning, 2025. URL [https://openreview.…](https://openreview.…) [↗](https://openreview.…)
10. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. [Process reinforcement through implicit rewards](https://md.research.elamur.ai/free/en/papers/cui-2025-process-reinforcement-through-implicit-rewards/). arXiv preprint arXiv:2502.01456, 2025.
11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783, 2024.
12. Andrea Fernández-Sánchez, Juan José Lorenzo-Castiñeiras, and Ana Sánchez-Bello. Navigating the future of pedagogy: The integration of ai tools in developing educational assessment rubrics. European Journal of Education, 60(1):e12826, 2025. [find](https://scholar.google.com/scholar?q=Andrea+Fern%C3%A1ndez%2DS%C3%A1nchez%2C+Juan+Jos%C3%A9+Lorenzo%2DCasti%C3%B1eiras%2C+and+Ana+S%C3%A1nchez%2DBello%2E+Navigating+the+future+of+pedagogy%3A+The+integration+of+ai+tools+in+developing+educational+assessment+rubrics%2E+European+Jo)
13. Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Bing Liu, and Sean Hendryx. [Rubrics as rewards: Reinforcement learning beyond verifiable domains](https://md.research.elamur.ai/free/en/papers/gunjal-2025-rubrics-rewards-reinforcement-learning-verifiable/). arXiv preprint arXiv:2507.17746, 2025.
14. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948, 2025.
15. Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. [Openrlhf: An easy-to-use, scalable and high-performance rlhf framework](https://md.research.elamur.ai/free/en/papers/hu-2024-openrlhf-easy-use-scalable/). arXiv preprint arXiv:2405.11143, 2024.
16. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). arXiv preprint arXiv:2410.21276, 2024.
17. Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, and Hanna Hajishirzi. [Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback](https://md.research.elamur.ai/free/en/papers/ivison-2024-unpacking-dpo-ppo-disentangling-best/). Advances in neural information processing systems, 37: 36602–36633, 2024.
18. Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Stepwise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.[18629, 2024](https://md.research.elamur.ai/free/en/papers/lai-2024-step-dpo-step-wise-preference/).
19. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. RewardBench: Evaluating reward models for language modeling. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Fin… [find](https://scholar.google.com/scholar?q=Nathan+Lambert%2C+Valentina+Pyatkin%2C+Jacob+Morrison%2C+LJ+Miranda%2C+Bill+Yuchen+Lin%2C+Khyathi+Chandu%2C+Nouha+Dziri%2C+Sachin+Kumar%2C+Tom+Zick%2C+Yejin+Choi%2C+Noah+A%2E+Smith%2C+and+Hannaneh+Hajishirzi%2E+RewardBench%3A+Ev)
20. Xuefeng Li, Haoyang Zou, and Pengfei Liu. [Torl: Scaling tool-integrated rl](https://md.research.elamur.ai/free/en/papers/li-2025-torl-scaling-tool-integrated-rl/). arXiv preprint arXiv:2503.23383, 2025.
21. Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers transformers to solve inherently serial problems. In The Twelfth International Conference on Learning Representations. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Sc… [find](https://scholar.google.com/scholar?q=Zhiyuan+Li%2C+Hong+Liu%2C+Denny+Zhou%2C+and+Tengyu+Ma%2E+Chain+of+thought+empowers+transformers+to+solve+inherently+serial+problems%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2E+Hunter)
22. Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. [Skywork-reward: Bag of tricks for reward modeling in llms](https://md.research.elamur.ai/free/en/papers/liu-2024-skywork-reward-bag-tricks/). arXiv preprint arXiv:2410.18451, 2024.
23. Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. [RM-bench: Benchmarking reward models of language models with subtlety and style](https://md.research.elamur.ai/free/en/papers/liu-2024-rm-bench-benchmarking-reward-models/). In The Thirteenth International Conference on Learning Representations, 2025a. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=QEHrmQPBdd. [↗](https://openreview.net/forum?)
24. Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. [Inference-time scaling for generalist reward modeling](https://md.research.elamur.ai/free/en/papers/liu-2025-inference-time-scaling-generalist/). arXiv preprint arXiv:2504.02495, 2025b.
25. William Merrill and Ashish Sabharwal. The expressive power of transformers with chain of thought. In The Twelfth International Conference on Learning Representations. Gilberto Montibeller and Alberto Franco. Multi-criteria decision analysis for strategic decision making. In Handbook of multicriteria… [find](https://scholar.google.com/scholar?q=William+Merrill+and+Ashish+Sabharwal%2E+The+expressive+power+of+transformers+with+chain+of+thought%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2E+Gilberto+Montibeller+and+Alberto+)
26. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 35:27730– 27744, 2022.
27. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.05530,…
28. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)
29. Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. [Rewarding progress: Scaling automated process verifiers for LLM reasoning](https://md.research.elamur.ai/free/en/papers/setlur-2024-rewarding-progress-scaling-automated-process/). In The Thirteenth International Conference on Learning Representations, 2025. URL ht…
30. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300, 2024.
31. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. [Hybridflow: A flexible and efficient rlhf framework](https://md.research.elamur.ai/free/en/papers/sheng-2024-hybridflow-flexible-efficient-rlhf-framework/). In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, pp. 1279–1297, New York, NY, USA, 2025. Associ…
32. Tu Shiwen, Zhao Liang, Chris Yuhao Liu, Liang Zeng, and Yang Liu. Skywork critic model series. [https://huggingface.co/Skywork](https://huggingface.co/Skywork), September 2024. URL https://huggingface.co/Skywork. [↗](https://huggingface.co/Skywork)
33. Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances in neural information processing systems, 34: 6906–6919, 2021. [find](https://scholar.google.com/scholar?q=Samuel+Stanton%2C+Pavel+Izmailov%2C+Polina+Kirichenko%2C+Alexander+A+Alemi%2C+and+Andrew+G+Wilson%2E+Does+knowledge+distillation+really+work%3F+Advances+in+neural+information+processing+systems%2C+34%3A+6906%E2%80%936919%2C+20)
34. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F+Christiano%2E+Learning+to+summarize+with+human+feedback%2E+Advances+in+Neural+Infor)
35. Nicole Van Hoeck, Patrick D Watson, and Aron K Barbey. Cognitive neuroscience of human counterfactual reasoning. Frontiers in human neuroscience, 9:420, 2015. [find](https://scholar.google.com/scholar?q=Nicole+Van+Hoeck%2C+Patrick+D+Watson%2C+and+Aron+K+Barbey%2E+Cognitive+neuroscience+of+human+counterfactual+reasoning%2E+Frontiers+in+human+neuroscience%2C+9%3A420%2C+2015%2E)
36. Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. [Interpretable preferences via multi-objective reward modeling and mixture-of-experts](https://md.research.elamur.ai/free/en/papers/wang-2024-interpretable-preferences-multi-objective-reward/). In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 10582–10592,…
37. Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. [Self-taught evaluators](https://md.research.elamur.ai/free/en/papers/wang-2024-self-taught-evaluators/). arXiv preprint arXiv:2408.02666, 2024b.
38. Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. [HelpSteer: Multi-attribute helpfulness dataset for SteerLM](https://md.research.elamur.ai/free/en/papers/wang-2024-helpsteer-multi-attribute-helpfulness-dataset/). In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings o…
39. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+et+al%2E+Qwen2%2E+5+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E15115%2C+2024%2E)
40. Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, Melanie Kambadur, Dhruv Mahajan, and Rui Hou. Self-generated critiques boost reward modeling for language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang (e… [find](https://scholar.google.com/scholar?q=Yue+Yu%2C+Zhengxing+Chen%2C+Aston+Zhang%2C+Liang+Tan%2C+Chenguang+Zhu%2C+Richard+Yuanzhe+Pang%2C+Yundi+Qian%2C+Xuewei+Wang%2C+Suchin+Gururangan%2C+Chao+Zhang%2C+Melanie+Kambadur%2C+Dhruv+Mahajan%2C+and+Rui+Hou%2E+Self%2Dgenerate)
41. Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Boji Shan, Zeyuan Liu, Jia Deng, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. [Advancing LLM reasoning generalists with preference trees](https://md.research.elamur.ai/free/en/papers/yuan-2024-advancing-llm-reasoning-generalists-preference/). In The Thirteenth International Conference…
42. Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. [Generative verifiers: Reward modeling as next-token prediction](https://md.research.elamur.ai/free/en/papers/zhang-2024-generative-verifiers-reward-modeling-as/). In The Thirteenth International Conference on Learning Representations, 2025. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=Ccwp4tFEtE. [↗](https://openreview.net/forum?)
43. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in Neural Information Processing Systems, 36:46595–46623, 2023.
44. Jialun Zhong, Wei Shen, Yanzeng Li, Songyang Gao, Hua Lu, Yicheng Chen, Yang Zhang, Wei Zhou, Jinjie Gu, and Lei Zou. A comprehensive survey of reward models: Taxonomy, applications, challenges, and future. arXiv preprint arXiv:2504.12328, 2025. — [arXiv:2504.12328](https://arxiv.org/abs/2504.12328) [find](https://scholar.google.com/scholar?q=Jialun+Zhong%2C+Wei+Shen%2C+Yanzeng+Li%2C+Songyang+Gao%2C+Hua+Lu%2C+Yicheng+Chen%2C+Yang+Zhang%2C+Wei+Zhou%2C+Jinjie+Gu%2C+and+Lei+Zou%2E+A+comprehensive+survey+of+reward+models%3A+Taxonomy%2C+applications%2C+challenges%2C+and+f)
45. Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. [RMB: Comprehensively benchmarking reward models in LLM alignment](https://md.research.elamur.ai/free/en/papers/zhou-2024-rmb-comprehensively-benchmarking-reward-models/). In The Thirteenth International Conference on Learning Represe…
46. Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, Karthik Ganesan, Wei-Lin Chiang, Jian Zhang, and Jiantao Jiao. Starling-7b: Improving helpfulness and harmlessness with RLAIF. In First Conference on Language Modeling, 2024. URL [https://openreview.net/forum?id=](https://openreview.net/forum?id=) GqDntYTTbk. [↗](https://openreview.net/forum?id=)
47. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. [find](https://scholar.google.com/scholar?q=Daniel+M+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint+arXiv%3A19)

*Updated 2 November 2026*