---
title: "Rethinking Reward Model Evaluation — paper by Wen et al. (2024)"
description: "The authors test this in a controlled synthetic setup. 60 references, cited by 9 papers here."
authors: ["Xueru Wen", "Jie Lou", "Yaojie Lu", "Hongyu Lin", "Xing Yu", "Xinyu Lu", "Ben He", "Xianpei Han", "Debing Zhang", "Le Sun"]
year: 2024
venue: "arXiv:2410.05584"
source: "https://arxiv.org/abs/2410.05584"
doi: "10.48550/arxiv.2410.05584"
arxiv: "2410.05584"
licence: "CC BY-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wen-2024-rethinking-reward-model-evaluation-barking/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wen-2024-rethinking-reward-model-evaluation-barking/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/14/) / Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?

# Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?

**Authors:** Xueru Wen, [Jie Lou](https://md.research.elamur.ai/free/en/authors/jie-lou/), [Yaojie Lu](https://md.research.elamur.ai/free/en/authors/yaojie-lu/), [Hongyu Lin](https://md.research.elamur.ai/free/en/authors/hongyu-lin/), Xing Yu, [Xinyu Lu](https://md.research.elamur.ai/free/en/authors/xinyu-lu/), [Ben He](https://md.research.elamur.ai/free/en/authors/ben-he/), [Xianpei Han](https://md.research.elamur.ai/free/en/authors/xianpei-han/), [Debing Zhang](https://md.research.elamur.ai/free/en/authors/debing-zhang/) and [Le Sun](https://md.research.elamur.ai/free/en/authors/le-sun/) · **Year:** 2024 · **Venue:** arXiv:2410.05584 · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2410.05584) · [PDF](https://arxiv.org/pdf/2410.05584) · [DOI 10.48550/arxiv.2410.05584](https://doi.org/10.48550/arxiv.2410.05584) · [arXiv 2410.05584](https://arxiv.org/abs/2410.05584) · [OpenAlex](https://openalex.org/W4403346561)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [9](https://md.research.elamur.ai/free/en/papers/wen-2024-rethinking-reward-model-evaluation-barking/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Reward models (RMs) are usually judged by pairwise accuracy on human-labelled preference data, but it is unclear whether higher accuracy actually produces a better policy after RLHF. The authors test this in a controlled synthetic setup. Ten RMs are fine-tuned from Llama-3-instruct-8B on preference data with different shares of flipped labels. Each RM takes a turn as the "golden" reward while the rest act as proxies, which yields 90 golden–proxy pairs. Policies are optimized with best-of-n sampling and PPO. The cost of optimizing a proxy instead of the golden reward is measured with a new metric, the Normalised Drop Ratio (NDR).

On RewardBench-based test data, accuracy correlates only weakly and positively with policy regret, and RMs with near-identical accuracy can lead to very different outcomes. How accuracy is computed also matters: mismatched prompt distributions weaken it, while evaluating more responses per prompt improves its predictive value. A Regressional Goodhart analysis shows that accuracy cannot fully capture the risk of overoptimization. For anyone building or selecting RMs, benchmark accuracy alone is an unreliable selection criterion.

## Key points
- **Framework:** N = 10 RMs are trained with varied label-flip rates, all initialized from Llama-3-instruct-8B with an NLL loss plus a regularization term. Every RM serves in turn as gold, giving N×(N−1) evaluation pairs.
- **Optimization:** Policies are optimized with best-of-n (n = 1280, about 35M sampled and scored responses) and with PPO via the OpenRLHF framework. Test data for both RMs and policies are built from RewardBench.
- **Main finding:** Accuracy shows only a weak positive correlation with policy regret (estimated by NDR), and policies trained on RMs with similar accuracy can show quite different regret.
- **Better measurement:** Policy regret is predicted better when RM test prompts match downstream prompts and when more responses are evaluated per prompt. This holds even under equal sample-size or annotation budgets. The golden-reward rank of the compared responses matters more than which model generated them.
- **Theory:** Under Regressional Goodhart, with normally distributed rewards and noise, the authors derive an expected link between accuracy and the degree of overoptimization. Empirically, however, RMs with matching accuracy diverge in how much they overoptimize.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) — Government body, Beijing, China · 256 works · 25,136 citations
- [University of Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/universities/university-of-chinese-academy-of-sciences/) — University, Beijing, China · 196 works · 8,907 citations
- [Institute of Software](https://md.research.elamur.ai/free/en/labs/institute-of-software/) — Research institute, Beijing, China · 30 works · 5,038 citations
- [Xiaohongshu (Little Red Book)](https://md.research.elamur.ai/free/en/companies/xiaohongshu-little-red-book/) — Company, Shanghai, China · 16 works · 210 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,394 works

## Cite

Cite as: Xueru Wen, Jie Lou, Yaojie Lu, Hongyu Lin, Xing Yu, Xinyu Lu, Ben He, Xianpei Han, Debing Zhang and Le Sun. 2024. Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?. arXiv:2410.05584.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wen2024rethinking,
  title = {{Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?}},
  author = {Xueru Wen, Jie Lou, Yaojie Lu, Hongyu Lin, Xing Yu, Xinyu Lu, Ben He, Xianpei Han, Debing Zhang and Le Sun},
  year = {2024},
  doi = {10.48550/arxiv.2410.05584},
  eprint = {2410.05584},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.05584}
}
```

- [← previous Perceptions of Language Technology Failures from South Asian English Speakers](https://md.research.elamur.ai/free/en/papers/holt-2024-perceptions-language-technology-failures-south/)
- [next → Role-Play Paradox in Large Language Models: Reasoning Performance Gains and Ethical Dilemmas](https://md.research.elamur.ai/free/en/papers/zhao-2024-role-play-paradox-large-language/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/14/)

## References

60 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. AI@Meta. Llama 3 model card. 2024. URL [https://github.com/meta-llama/](https://github.com/meta-llama/) llama3/blob/main/MODEL_CARD.md. [↗](https://github.com/meta-llama/)
2. Argilla. Capybara-dpo 7k binarized dataset. [https://huggingface.co/datasets/](https://huggingface.co/datasets/) argilla/distilabel-capybara-dpo-7k-binarized, 2024. [↗](https://huggingface.co/datasets/)
3. Stuart Armstrong and Sören Mindermann. Occam’s razor is insufficient to infer the preferences of irrational agents, 2019. URL https://arxiv.org/abs/1712.05812. — [arXiv:1712.05812](https://arxiv.org/abs/1712.05812) [↗](https://arxiv.org/abs/1712.05812)
4. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Saurav+Kadavath%2C+Jackson+Kernion%2C+Tom+Conerly%2C+)
5. Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324, 1952. doi: 10.2307/2334029. URL https://doi.org/10.2307/2334029. — [doi:10.2307/2334029](https://doi.org/10.2307/2334029) [↗](https://doi.org/10.2307/2334029)
6. Daniel S. Brown and Scott Niekum. Deep bayesian reward learning from preferences, 2019. URL https://arxiv.org/abs/1912.04472. — [arXiv:1912.04472](https://arxiv.org/abs/1912.04472) [↗](https://arxiv.org/abs/1912.04472)
7. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman An… [find](https://scholar.google.com/scholar?q=Stephen+Casper%2C+Xander+Davies%2C+Claudia+Shi%2C+Thomas+Krendl+Gilbert%2C+Je%CC%81re%CC%81my+Scheurer%2C+Javier+Rando%2C+Rachel+Freedman%2C+Tomasz+Korbak%2C+David+Lindner%2C+Pedro+Freire%2C+Tony+Wang%2C+Samuel+Marks%2C+Charbel%2DRaphae)
8. Sourav Chatterjee. A new coefficient of correlation, 2020. URL https://arxiv.org/abs/ 1909.10140. [↗](https://arxiv.org/abs/)
9. Douglas E Critchlow, Michael A Fligner, and Joseph S Verducci. Probability models on rankings. Journal of Mathematical Psychology, 35(3):294–318, September 1991. ISSN 00222496. doi: 10.1016/0022-2496(91)90050-4. — [doi:10.1016/0022-2496(91)90050-4](https://doi.org/10.1016/0022-2496%2891%2990050-4) [find](https://scholar.google.com/scholar?q=Douglas+E+Critchlow%2C+Michael+A+Fligner%2C+and+Joseph+S+Verducci%2E+Probability+models+on+rankings%2E+Journal+of+Mathematical+Psychology%2C+35%283%29%3A294%E2%80%93318%2C+September+1991%2E+ISSN+00222496%2E+doi%3A+10%2E1016%2F0022%2D2496%28)
10. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023. [find](https://scholar.google.com/scholar?q=Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Guanming+Yao%2C+Wei+Zhu%2C+Yuan+Ni%2C+Guotong+Xie%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+Ultrafeedback%3A+Boosting+language+models+with+high%2Dquality+feedback%2C+2023%2E)
11. Luigi Daniele and Suphavadeeprasit. Amplify-instruct: Synthetically generated diverse multi-turn conversations for efficient llm training. arXiv preprint arXiv:(coming soon), 2023. URL [https://huggingface.co/datasets/LDJnr/Capybara](https://huggingface.co/datasets/LDJnr/Capybara). [↗](https://huggingface.co/datasets/LDJnr/Capybara)
12. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, and other. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/), 2024. URL https://arxiv.org/abs/240… [↗](https://arxiv.org/abs/240…)
13. Lukas Fluri, Leon Lang, Alessandro Abate, Patrick Forré, David Krueger, and Joar Skalse. The perils of optimizing learned reward functions: Low training error does not guarantee low regret, 2024. URL https://arxiv.org/abs/2406.15753. — [arXiv:2406.15753](https://arxiv.org/abs/2406.15753) [↗](https://arxiv.org/abs/2406.15753)
14. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan… — [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://md.research.elamur.ai/free/en/papers/ganguli-2022-red-teaming-language-models-reduce/)
15. Leo Gao, John Schulman, and Jacob Hilton. [Scaling laws for reward model overoptimization](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/), 2022. URL https://arxiv.org/abs/2210.10760. [↗](https://arxiv.org/abs/2210.10760)
16. Yang Gao, Dana Alon, and Donald Metzler. Impact of preference noise on the alignment performance of generative language models, 2024. URL https://arxiv.org/abs/2404. 09824. [↗](https://arxiv.org/abs/2404)
17. Adam Gleave, Michael Dennis, Shane Legg, Stuart Russell, and Jan Leike. Quantifying differences in reward functions, 2021. URL https://arxiv.org/abs/2006.13900. — [arXiv:2006.13900](https://arxiv.org/abs/2006.13900) [↗](https://arxiv.org/abs/2006.13900)
18. C. A. E. Goodhart. Problems of Monetary Management: The UK Experience, pp. 91–121. Macmillan Education UK, London, 1984. ISBN 978-1-349-17295-5. doi: 10.1007/978-1-349-17295-5 4. URL https://doi.org/10.1007/978-1-349-17295-5_4. — [doi:10.1007/978-1-349-17295-5](https://doi.org/10.1007/978-1-349-17295-5) [↗](https://doi.org/10.1007/978-1-349-17295-5_4)
19. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/), 2021. URL https://arxiv.org/abs/2009.03300. [↗](https://arxiv.org/abs/2009.03300)
20. Jacob Hilton. KL divergence of max-of-n. December 2023. [find](https://scholar.google.com/scholar?q=Jacob+Hilton%2E+KL+divergence+of+max%2Dof%2Dn%2E+December+2023%2E)
21. Zhenyu Hou, Yilin Niu, Zhengxiao Du, Xiaohan Zhang, Xiao Liu, Aohan Zeng, Qinkai Zheng, Minlie Huang, Hongning Wang, Jie Tang, and Yuxiao Dong. [Chatglm-rlhf: Practices of aligning large language models with human feedback](https://md.research.elamur.ai/free/en/papers/hou-2024-chatglm-rlhf-practices-aligning/), 2024. URL https://arxiv.org/abs/ 2404.00934. [↗](https://arxiv.org/abs/)
22. Jian Hu, Xibin Wu, Weixun Wang, Xianyu, Dehao Zhang, and Yu Cao. [Openrlhf: An easy-to-use, scalable and high-performance rlhf framework](https://md.research.elamur.ai/free/en/papers/hu-2024-openrlhf-easy-use-scalable/). arXiv preprint arXiv:2405.11143, 2024.
23. Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari, 2018. URL https://arxiv.org/abs/1811.06521. — [arXiv:1811.06521](https://arxiv.org/abs/1811.06521) [↗](https://arxiv.org/abs/1811.06521)
24. Intel. Orca dpo pairs dataset. [https://huggingface.co/datasets/Intel/orca_dpo_](https://huggingface.co/datasets/Intel/orca_dpo_) pairs, 2023. Available on Hugging Face Datasets. [↗](https://huggingface.co/datasets/Intel/orca_dpo_)
25. Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Qiu, Boxun Li, and Yaodong Yang. Pku-saferlhf: A safety alignment preference dataset for llama family models, 2024. URL https://arxiv.org/abs/2406.15513. — [arXiv:2406.15513](https://arxiv.org/abs/2406.15513) [↗](https://arxiv.org/abs/2406.15513)
26. Jacek Karwowski, Oliver Hayman, Xingjian Bai, Klaus Kiendlhofer, Charlie Griffin, and Joar Skalse. Goodhart’s law in reinforcement learning, 2023. URL https://arxiv.org/abs/ 2310.09144. [↗](https://arxiv.org/abs/)
27. Sunghwan Kim, Dongjin Kang, Taeyoon Kwon, Hyungjoo Chae, Jungsoo Won, Dongha Lee, and Jinyoung Yeo. Evaluating robustness of reward models for mathematical reasoning, 2024. URL https://arxiv.org/abs/2410.01729. — [arXiv:2410.01729](https://arxiv.org/abs/2410.01729) [↗](https://arxiv.org/abs/2410.01729)
28. Victoria Krakovna. Specification gaming: The flip side of ai ingenuity, April 2020. URL [https://deepmind.google/discover/blog/](https://deepmind.google/discover/blog/) specification-gaming-the-flip-side-of-ai-ingenuity/. [↗](https://deepmind.google/discover/blog/)
29. Nathan Lambert, Thomas Krendl Gilbert, and Tom Zick. The history and risks of reinforcement learning and human feedback, 2023. URL https://arxiv.org/abs/2310.13595. — [arXiv:2310.13595](https://arxiv.org/abs/2310.13595) [↗](https://arxiv.org/abs/2310.13595)
30. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024. URL https://arxiv.org/abs/2403.13787. [↗](https://arxiv.org/abs/2403.13787)
31. Leon Lang, Davis Foote, Stuart Russell, Anca Dragan, Erik Jenner, and Scott Emmons. When your ais deceive you: Challenges of partial observability in reinforcement learning from human feedback, 2024. URL https://arxiv.org/abs/2402.17747. — [arXiv:2402.17747](https://arxiv.org/abs/2402.17747) [↗](https://arxiv.org/abs/2402.17747)
32. Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J. Bentley, Samuel Bernard, Guillaume Beslon, David M. Bryson, Patryk Chrabaszcz, Nick Cheney, Antoine Cully, Stephane Doncieux, Fred C. Dyer, Kai Olav Ellefsen, Robert Feldt, Stephan Fischer, Stephanie For… — [arXiv:1803.03453](https://arxiv.org/abs/1803.03453) [find](https://scholar.google.com/scholar?q=Joel+Lehman%2C+Jeff+Clune%2C+Dusan+Misevic%2C+Christoph+Adami%2C+Lee+Altenberg%2C+Julie+Beaulieu%2C+Peter+J%2E+Bentley%2C+Samuel+Bernard%2C+Guillaume+Beslon%2C+David+M%2E+Bryson%2C+Patryk+Chrabaszcz%2C+Nick+Cheney%2C+Antoine+Cul)
33. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval), 5 2023. [↗](https://github.com/tatsu-lab/alpaca_eval)
34. Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and ”Teknium”. Openorca: An open dataset of gpt augmented flan reasoning traces. https://https://huggingface.co/Open-Orca/OpenOrca, 2023. [find](https://scholar.google.com/scholar?q=Wing+Lian%2C+Bleys+Goodson%2C+Eugene+Pentland%2C+Austin+Cook%2C+Chanvichet+Vong%2C+and+%E2%80%9DTeknium%E2%80%9D%2E+Openorca%3A+An+open+dataset+of+gpt+augmented+flan+reasoning+traces%2E+https%3A%2F%2Fhttps%3A%2F%2Fhuggingface%2Eco%2FOpen%2DOrca%2FOpenO)
35. David Manheim and Scott Garrabrant. Categorizing variants of goodhart’s law, 2019. URL https://arxiv.org/abs/1803.04585. — [arXiv:1803.04585](https://arxiv.org/abs/1803.04585) [↗](https://arxiv.org/abs/1803.04585)
36. Eric J. Michaud, Adam Gleave, and Stuart Russell. Understanding learned reward functions, 2020. URL https://arxiv.org/abs/2012.05862. — [arXiv:2012.05862](https://arxiv.org/abs/2012.05862) [↗](https://arxiv.org/abs/2012.05862)
37. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted qu… — [WebGPT: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)
38. Andrew Y. Ng and Stuart J. Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, ICML ’00, pp. 663–670, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1558607072. [find](https://scholar.google.com/scholar?q=Andrew+Y%2E+Ng+and+Stuart+J%2E+Russell%2E+Algorithms+for+inverse+reinforcement+learning%2E+In+Proceedings+of+the+Seventeenth+International+Conference+on+Machine+Learning%2C+ICML+%E2%80%9900%2C+pp%2E+663%E2%80%93670%2C+San+Francisco%2C)
39. Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning, ICML ’99, pp. 278–287, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers… [find](https://scholar.google.com/scholar?q=Andrew+Y%2E+Ng%2C+Daishi+Harada%2C+and+Stuart+J%2E+Russell%2E+Policy+invariance+under+reward+transformations%3A+Theory+and+application+to+reward+shaping%2E+In+Proceedings+of+the+Sixteenth+International+Conference+o)
40. OpenAI. [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). gpt-4o-system-card.pdf, 2024. [https://openai.com/research/](https://openai.com/research/) [↗](https://openai.com/research/)
41. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training la… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
42. Malayandi Palan, Nicholas C. Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward functions by integrating human demonstrations and preferences, 2019. URL https://arxiv.org/abs/1906.08928. — [arXiv:1906.08928](https://arxiv.org/abs/1906.08928) [↗](https://arxiv.org/abs/1906.08928)
43. Jacob Russell and Eugene Santos. Explaining reward functions in markov decision processes. In The Thirty-Second International FLAIRS Conference, 2019. [find](https://scholar.google.com/scholar?q=Jacob+Russell+and+Eugene+Santos%2E+Explaining+reward+functions+in+markov+decision+processes%2E+In+The+Thirty%2DSecond+International+FLAIRS+Conference%2C+2019%2E)
44. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347. [↗](https://arxiv.org/abs/1707.06347)
45. Joar Skalse and Alessandro Abate. Misspecification in inverse reinforcement learning, 2023. URL https://arxiv.org/abs/2212.03201. — [arXiv:2212.03201](https://arxiv.org/abs/2212.03201) [↗](https://arxiv.org/abs/2212.03201)
46. Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking, 2022. URL https://arxiv.org/abs/2209.13085. — [arXiv:2209.13085](https://arxiv.org/abs/2209.13085) [↗](https://arxiv.org/abs/2209.13085)
47. Joar Skalse, Matthew Farrugia-Roberts, Stuart Russell, Alessandro Abate, and Adam Gleave. Invariance in policy optimisation and partial identifiability in reward learning, 2023. URL https://arxiv.org/abs/2203.07475. — [arXiv:2203.07475](https://arxiv.org/abs/2203.07475) [↗](https://arxiv.org/abs/2203.07475)
48. Joar Skalse, Lucy Farnik, Sumeet Ramesh Motwani, Erik Jenner, Adam Gleave, and Alessandro Abate. Starc: A general framework for quantifying differences between reward functions, 2024. URL https://arxiv.org/abs/2309.15257. — [arXiv:2309.15257](https://arxiv.org/abs/2309.15257) [↗](https://arxiv.org/abs/2309.15257)
49. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022. URL https://arxiv.org/abs/2009.01325. [↗](https://arxiv.org/abs/2009.01325)
50. Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL [https://qwenlm.github.io/blog/qwen2.5/](https://qwenlm.github.io/blog/qwen2.5/). [↗](https://qwenlm.github.io/blog/qwen2.5/)
51. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Juny… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Zhou%2C+Chengpeng+Li%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Guanting+Dong%2C+Haoran+Wei%2C+Huan+Lin%2C+Jialong+Tang%2C+Jialin+Wang%2C+Jian+Yang%2C+Jian)
52. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/), 2023.
53. Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. [Rmb: Comprehensively benchmarking reward models in llm alignment](https://md.research.elamur.ai/free/en/papers/zhou-2024-rmb-comprehensively-benchmarking-reward-models/), 2024. URL https://arxiv.org/abs/2410.09893. [↗](https://arxiv.org/abs/2410.09893)
54. Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving llm helpfulness & harmlessness with rlaif, November 2023. [find](https://scholar.google.com/scholar?q=Banghua+Zhu%2C+Evan+Frick%2C+Tianhao+Wu%2C+Hanlin+Zhu%2C+and+Jiantao+Jiao%2E+Starling%2D7b%3A+Improving+llm+helpfulness+%26+harmlessness+with+rlaif%2C+November+2023%2E)
55. Simon Zhuang and Dylan Hadfield-Menell. Consequences of misaligned ai, 2021. URL https://arxiv.org/abs/2102.03896. — [arXiv:2102.03896](https://arxiv.org/abs/2102.03896) [↗](https://arxiv.org/abs/2102.03896)
56. A PPENDIX 8.1 D ATASET C ONSTRUCTION We build up a RM training dataset by mixing the following open-sourced datasets: • Nectar (Zhu et al., 2023), a high-quality 7-wise comparison dataset generated by GPT-4 ranking. [find](https://scholar.google.com/scholar?q=A+PPENDIX+8%2E1+D+ATASET+C+ONSTRUCTION+We+build+up+a+RM+training+dataset+by+mixing+the+following+open%2Dsourced+datasets%3A+%E2%80%A2+Nectar+%28Zhu+et+al%2E%2C+2023%29%2C+a+high%2Dquality+7%2Dwise+comparison+dataset+generated+by)
57. Capybara-7K-binarized (Argilla, 2024), a binarized prefernce dataset built with distilabel atop the Capybara (Daniele & Suphavadeeprasit, 2023). [find](https://scholar.google.com/scholar?q=Capybara%2D7K%2Dbinarized+%28Argilla%2C+2024%29%2C+a+binarized+prefernce+dataset+built+with+distilabel+atop+the+Capybara+%28Daniele+%26+Suphavadeeprasit%2C+2023%29%2E)
58. Orca-pairs (Intel, 2023), a dataset contains 12k examples from OpenOrca dataset (Lian et al., 2023). [find](https://scholar.google.com/scholar?q=Orca%2Dpairs+%28Intel%2C+2023%29%2C+a+dataset+contains+12k+examples+from+OpenOrca+dataset+%28Lian+et+al%2E%2C+2023%29%2E)
59. UltraFeedback (Cui et al., 2023), a large-scale, fine-grained, diverse preference dataset. [find](https://scholar.google.com/scholar?q=UltraFeedback+%28Cui+et+al%2E%2C+2023%29%2C+a+large%2Dscale%2C+fine%2Dgrained%2C+diverse+preference+dataset%2E)
60. PKU-SafeRLHF (Ji et al., 2024), a high-quality dataset consisting of 83.4K preference entries, which is annotated across two dimensions: harmlessness and helpfulness. [find](https://scholar.google.com/scholar?q=PKU%2DSafeRLHF+%28Ji+et+al%2E%2C+2024%29%2C+a+high%2Dquality+dataset+consisting+of+83%2E4K+preference+entries%2C+which+is+annotated+across+two+dimensions%3A+harmlessness+and+helpfulness%2E)

*Updated 3 November 2026*