---
title: "Generative Reward Models — paper by Mahan et al. (2024)"
description: "Reward models trained with the Bradley-Terry objective on human preference pairs score well on… 58 references, cited by 58 papers here."
authors: ["Dakota Mahan", "Duy Van Phung", "Rafael Rafailov", "Chase Blagden", "Nathan Lile", "Louis Castricato", "Jan-Philipp Franken", "Chelsea Finn", "Alon Albalak"]
year: 2024
venue: "arXiv:2410.12832"
source: "https://arxiv.org/abs/2410.12832"
doi: "10.48550/arxiv.2410.12832"
arxiv: "2410.12832"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mahan-2024-generative-reward-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mahan-2024-generative-reward-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / Generative Reward Models

# Generative Reward Models

**Authors:** [Dakota Mahan](https://md.research.elamur.ai/free/en/authors/dakota-mahan/), [Duy Van Phung](https://md.research.elamur.ai/free/en/authors/duy-phung/), [Rafael Rafailov](https://md.research.elamur.ai/free/en/authors/rafael-rafailov/), [Chase Blagden](https://md.research.elamur.ai/free/en/authors/chase-blagden/), [Nathan Lile](https://md.research.elamur.ai/free/en/authors/nathan-lile/), [Louis Castricato](https://md.research.elamur.ai/free/en/authors/louis-castricato/), [Jan-Philipp Franken](https://md.research.elamur.ai/free/en/authors/jan-philipp-franken/), [Chelsea Finn](https://md.research.elamur.ai/free/en/authors/chelsea-finn/) and [Alon Albalak](https://md.research.elamur.ai/free/en/authors/alon-albalak/) · **Year:** 2024 · **Venue:** arXiv:2410.12832 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2410.12832) · [PDF](https://arxiv.org/pdf/2410.12832) · [DOI 10.48550/arxiv.2410.12832](https://doi.org/10.48550/arxiv.2410.12832) · [arXiv 2410.12832](https://arxiv.org/abs/2410.12832) · [OpenAlex](https://openalex.org/W4403578740)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [58](https://md.research.elamur.ai/free/en/papers/mahan-2024-generative-reward-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Reward models trained with the Bradley-Terry objective on human preference pairs score well on familiar data but generalize poorly to unseen tasks. Off-the-shelf LLM judges (the RLAIF route) are more robust but often disagree with real user choices. The authors combine the two ideas. They treat preference modelling as next-token prediction: the LLM outputs an indicator token for response A or B. In the variant with reasoning (CoT-GenRM), the model first writes a rationale and then gives its verdict. Training follows a STaR-style loop. The model samples its own judgments, keeps those that match the human label, and is fine-tuned on them with SFT or DPO. A "Rationalizer" variant instead generates explanations conditioned on the correct answer.

Untuned LLM judges fall 9–36% behind Bradley-Terry models on in-distribution data. GenRM matches Bradley-Terry in-distribution and beats it by 10–45% out-of-distribution. It also outperforms zero-shot judges in both settings. This suggests that human-feedback training plus self-generated reasoning yields more reliable synthetic preference labels.

## Key points
- Training data comes from UltraFeedback and UltraInteract, with RewardBench serving as the out-of-distribution test. Prompts use Llama 3.1 instruct chat templates.
- The compared methods are Bradley-Terry RM, PairRM, LLM-as-a-judge, GenRM, STaR-SFT, STaR-DPO, and the Rationalizer variants.
- GenRM outperforms LLM-as-a-judge by 9–31% in-distribution and by 2–6% out-of-distribution.
- STaR-DPO improves across 3 iterations, each run on a disjoint third of the data. After iteration 3, Maj@32 reaches about 90.2 on UltraInteract and about 82.6 on RewardBench when trained on UltraFeedback.
- The Rationalizer trained on UltraInteract gains in-distribution but drops on RewardBench (Maj@32 falls from about 68.6 to 58.2). Generation used SGLang 0.3.0 with temperature 1.0 and top-p 0.95.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 567 works · 335,190 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

## Cite

Cite as: Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Franken, Chelsea Finn and Alon Albalak. 2024. Generative Reward Models. arXiv:2410.12832.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{mahan2024generative,
  title = {{Generative Reward Models}},
  author = {Dakota Mahan, Duy {Van Phung}, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Franken, Chelsea Finn and Alon Albalak},
  year = {2024},
  doi = {10.48550/arxiv.2410.12832},
  eprint = {2410.12832},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.12832}
}
```

- [← previous Free Process Rewards without Process Labels](https://md.research.elamur.ai/free/en/papers/yuan-2024-free-process-rewards-process-labels/)
- [next → MaLA-500: Massive Language Adaptation of Large Language Models](https://md.research.elamur.ai/free/en/papers/lin-2024-mala-500-massive-language-adaptation/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

58 references, 34 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms, 2024. URL https://arxiv.org/abs/2402.14740. [↗](https://arxiv.org/abs/2402.14740)
2. Alon Albalak, Colin A Raffel, and William Yang Wang. [Improving few-shot generalization by exploring and exploiting auxiliary data](https://md.research.elamur.ai/free/en/papers/albalak-2023-improving-few-shot-generalization-exploring/). In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 24754–24780. Curran Asso…
3. Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. [A survey on data selection for language models](https://md.research.elamur.ai/free/en/papers/albalak-2024-survey-data-selection-language-models/), 2024. URL [https://arxiv.org/abs…](https://arxiv.org/abs…) [↗](https://arxiv.org/abs…)
4. Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D. Goodman. [Star-gate: Teaching language models to ask clarifying questions](https://md.research.elamur.ai/free/en/papers/andukuri-2024-star-gate-teaching-language-models/), 2024.
5. Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D. Chang, and Prithviraj Ammanabrolu. [Critique-out-loud reward models](https://md.research.elamur.ai/free/en/papers/ankner-2024-critique-out-loud-reward-models/), 2024. URL https://arxiv.org/abs/2408.11791. [↗](https://arxiv.org/abs/2408.11791)
6. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. [find](https://scholar.google.com/scholar?q=Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Andy+Jones%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Nova+DasSarma%2C+et+al%2E+A+general+language+assistant+as+a+laboratory+for+alignment%2E+a)
7. Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. [A general theoretical paradigm to understand learning from human preferences](https://md.research.elamur.ai/free/en/papers/azar-2023-general-theoretical-paradigm-understand-learning/), 2023.
8. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Saurav+Kadavath%2C+Jackson+Kernion%2C+Tom+Conerly%2C+)
9. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared… — [Constitutional AI: Harmlessness from AI Feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/)
10. Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. doi: https://doi.org/10.2307/ 2334029. [find](https://scholar.google.com/scholar?q=Ralph+Allan+Bradley+and+Milton+E%2E+Terry%2E+Rank+analysis+of+incomplete+block+designs%3A+I%2E+the+method+of+paired+comparisons%2E+Biometrika%2C+39%283%2F4%29%3A324%E2%80%93345%2C+1952%2E+doi%3A+https%3A%2F%2Fdoi%2Eorg%2F10%2E2307%2F+2334029%2E)
11. Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, Rishabh Joshi, Zeyu Zheng, and Bilal Piot. Human alignment of large language models through online preference optimisation, 2024. URL htt… — [arXiv:2403.08635](https://arxiv.org/abs/2403.08635) [find](https://scholar.google.com/scholar?q=Daniele+Calandriello%2C+Daniel+Guo%2C+Remi+Munos%2C+Mark+Rowland%2C+Yunhao+Tang%2C+Bernardo+Avila+Pires%2C+Pierre+Harvey+Richemond%2C+Charline+Le+Lan%2C+Michal+Valko%2C+Tianqi+Liu%2C+Rishabh+Joshi%2C+Zeyu+Zheng%2C+and+Bilal+)
12. Louis Castricato, Nathan Lile, Rafael Rafailov, Jan-Philipp Fränken, and Chelsea Finn. Persona: A reproducible testbed for pluralistic alignment, 2024. URL https://arxiv.org/abs/ 2407.17387. [↗](https://arxiv.org/abs/)
13. Zhaorun Chen, Yichao Du, Zichen Wen, Yiyang Zhou, Chenhang Cui, Zhenzhen Weng, Haoqin Tu, Chaoqi Wang, Zhengwei Tong, Qinglan Huang, Canyu Chen, Qinghao Ye, Zhihong Zhu, Yuqing Zhang, Jiawei Zhou, Zhuokai Zhao, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Mj-bench: Is your multimodal reward model… — [arXiv:2407.04842](https://arxiv.org/abs/2407.04842) [find](https://scholar.google.com/scholar?q=Zhaorun+Chen%2C+Yichao+Du%2C+Zichen+Wen%2C+Yiyang+Zhou%2C+Chenhang+Cui%2C+Zhenzhen+Weng%2C+Haoqin+Tu%2C+Chaoqi+Wang%2C+Zhengwei+Tong%2C+Qinglan+Huang%2C+Canyu+Chen%2C+Qinghao+Ye%2C+Zhihong+Zhu%2C+Yuqing+Zhang%2C+Jiawei+Zhou%2C+Zhu)
14. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023. [find](https://scholar.google.com/scholar?q=Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Guanming+Yao%2C+Wei+Zhu%2C+Yuan+Ni%2C+Guotong+Xie%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+Ultrafeedback%3A+Boosting+language+models+with+high%2Dquality+feedback%2C+2023%2E)
15. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with scaled ai feedback, 2024. URL https://arxiv.org/abs/2310.01377. [↗](https://arxiv.org/abs/2310.01377)
16. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez,… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
17. Jan-Philipp Fränken, Eric Zelikman, Rafael Rafailov, Kanishk Gandhi, Tobias Gerstenberg, and Noah D Goodman. [Self-supervised alignment with mutual information: Learning to follow principles without preference labels](https://md.research.elamur.ai/free/en/papers/franken-2024-self-supervised-alignment-mutual-information/). arXiv preprint arXiv:2404.14313, 2024.
18. Leo Gao, John Schulman, and Jacob Hilton. [Scaling laws for reward model overoptimization](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/). International Conference on machine Learning, 2023.
19. Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, and Mathieu Blondel. Direct language model alignment from online ai feedback, 2024. URL https://arxiv.org/abs/ 2402.04792. [↗](https://arxiv.org/abs/)
20. Alexander Havrilla, Maksym Zhuravinskyi, Duy Phung, Aman Tiwari, Jonathan Tow, Stella Biderman, Quentin Anthony, and Louis Castricato. trlX: A framework for large scale reinforcement learning from human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processi… [find](https://scholar.google.com/scholar?q=Alexander+Havrilla%2C+Maksym+Zhuravinskyi%2C+Duy+Phung%2C+Aman+Tiwari%2C+Jonathan+Tow%2C+Stella+Biderman%2C+Quentin+Anthony%2C+and+Louis+Castricato%2E+trlX%3A+A+framework+for+large+scale+reinforcement+learning+from+hum)
21. Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. [V-star: Training verifiers for self-taught reasoners](https://md.research.elamur.ai/free/en/papers/hosseini-2024-v-star-training-verifiers-self/), 2024.
22. Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. [Llm-blender: Ensembling large language models with pairwise ranking and generative fusion](https://md.research.elamur.ai/free/en/papers/jiang-2023-llm-blender-ensembling-large-language/), 2023. URL https://arxiv.org/abs/2306. 02561. [↗](https://arxiv.org/abs/2306)
23. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. [Prometheus: Inducing fine-grained evaluation capability in language models](https://md.research.elamur.ai/free/en/papers/kim-2023-prometheus-inducing-fine-grained-evaluation/), 2024. URL https://arxiv.org/abs/ 2310.08491. [↗](https://arxiv.org/abs/)
24. Sandipan Kundu, Yuntao Bai, Saurav Kadavath, Amanda Askell, Andrew Callahan, Anna Chen, Anna Goldie, Avital Balwit, Azalia Mirhoseini, Brayden McLean, et al. Specific versus general principles for constitutional ai. arXiv preprint arXiv:2310.13798, 2023. [find](https://scholar.google.com/scholar?q=Sandipan+Kundu%2C+Yuntao+Bai%2C+Saurav+Kadavath%2C+Amanda+Askell%2C+Andrew+Callahan%2C+Anna+Chen%2C+Anna+Goldie%2C+Avital+Balwit%2C+Azalia+Mirhoseini%2C+Brayden+McLean%2C+et+al%2E+Specific+versus+general+principles+for+con)
25. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024. [find](https://scholar.google.com/scholar?q=Nathan+Lambert%2C+Valentina+Pyatkin%2C+Jacob+Morrison%2C+LJ+Miranda%2C+Bill+Yuchen+Lin%2C+Khyathi+Chandu%2C+Nouha+Dziri%2C+Sachin+Kumar%2C+Tom+Zick%2C+Yejin+Choi%2C+Noah+A%2E+Smith%2C+and+Hannaneh+Hajishirzi%2E+Rewardbench%3A+Ev)
26. Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhur… — [DataComp-LM: In search of the next generation of training sets for language models](https://md.research.elamur.ai/free/en/papers/li-2024-datacomp-lm-search-next-generation/)
27. Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, and Bilal Piot. Nash learning from hu… — [Nash Learning from Human Feedback](https://md.research.elamur.ai/free/en/papers/munos-2023-nash-learning-human-feedback/)
28. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint, 2023. https://arxiv.org/abs/2303. 08774. [↗](https://arxiv.org/abs/2303)
29. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
30. Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. [Iterative reasoning preference optimization](https://md.research.elamur.ai/free/en/papers/pang-2024-iterative-reasoning-preference-optimization/), 2024. URL [https://arxiv.org/](https://arxiv.org/) abs/2404.19733. [↗](https://arxiv.org/)
31. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2305.18290. [↗](https://arxiv.org/abs/2305.18290)
32. Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms, 2024. URL https://arxiv.org/abs/2406.02900. — [arXiv:2406.02900](https://arxiv.org/abs/2406.02900) [↗](https://arxiv.org/abs/2406.02900)
33. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.05530,…
34. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2C+2017%2E)
35. Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. [A roadmap to pluralistic alignment](https://md.research.elamur.ai/free/en/papers/sorensen-2024-roadmap-pluralistic-alignment/), 2024. URL https://arxiv.org/abs/2402. 05070. [↗](https://arxiv.org/abs/2402)
36. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeff+Wu%2C+Daniel+M%2E+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+Christiano%2E+Learning+to+summarize+from+human+feedback%2C+2022%2E)
37. Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. [https://github.com/huggingface/trl](https://github.com/huggingface/trl), 2020. [↗](https://github.com/huggingface/trl)
38. Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and… — [Secrets of RLHF in Large Language Models Part II: Reward Modeling](https://md.research.elamur.ai/free/en/papers/wang-2024-secrets-rlhf-large-language-models/)
39. Peifeng Wang, Austin Xu, Yilun Zhou, Caiming Xiong, and Shafiq Joty. [Direct judgement preference optimization](https://md.research.elamur.ai/free/en/papers/wang-2025-direct-judgement-preference-optimization/), 2024b. URL https://arxiv.org/abs/2409.14664. [↗](https://arxiv.org/abs/2409.14664)
40. Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O’Brien, Ramakanth Pasunuru, Jane Dwivedi-Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. [Shepherd: A critic for language model generation](https://md.research.elamur.ai/free/en/papers/wang-2023-shepherd-critic-language-model-generation/), 2023. URL https://arxiv.org/abs/2308. 04592. [↗](https://arxiv.org/abs/2308)
41. Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. [Self-taught evaluators](https://md.research.elamur.ai/free/en/papers/wang-2024-self-taught-evaluators/), 2024c. URL https://arxiv.org/abs/2408.02666. [↗](https://arxiv.org/abs/2408.02666)
42. Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, and Jiantao Jiao. Pairwise proximal policy optimization: Harnessing relative feedback for llm alignment, 2023. URL https://arxiv.org/abs/2310.00212. — [arXiv:2310.00212](https://arxiv.org/abs/2310.00212) [↗](https://arxiv.org/abs/2310.00212)
43. Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. [Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge](https://md.research.elamur.ai/free/en/papers/wu-2025-meta-rewarding-language-models-self/), 2024. URL https://arxiv.org/abs/2407.19594. [↗](https://arxiv.org/abs/2407.19594)
44. Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling, 2023. URL https://arxiv.org/abs/2302.03169. [↗](https://arxiv.org/abs/2302.03169)
45. Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Iterative preference optimization with the pairwise cringe loss, 2024a. URL https://arxiv.org/abs/2312.16682. — [arXiv:2312.16682](https://arxiv.org/abs/2312.16682) [↗](https://arxiv.org/abs/2312.16682)
46. Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study, 2024b. URL https://arxiv.org/abs/2404.10719. — [arXiv:2404.10719](https://arxiv.org/abs/2404.10719) [↗](https://arxiv.org/abs/2404.10719)
47. Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. [Advancing llm reasoning generalists with preference trees](https://md.research.elamur.ai/free/en/papers/yuan-2024-advancing-llm-reasoning-generalists-preference/), 2024a. URL https://arxiv.org/abs/2404.02078. [↗](https://arxiv.org/abs/2404.02078)
48. Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. [Advancing llm reasoning generalists with preference trees](https://md.research.elamur.ai/free/en/papers/yuan-2024-advancing-llm-reasoning-generalists-preference/). arXiv preprint arXiv:2404.02078, 2024b.
49. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. [Star: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). Advances in Neural Information Processing Systems, 35:15476–15488, 2022.
50. Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. [Quiet-star: Language models can teach themselves to think before speaking](https://md.research.elamur.ai/free/en/papers/al-2024-quiet-star-language-models-teach/). arXiv preprint arXiv:2403.09629, 2024.
51. Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. arXiv preprint arXiv:2310.07641, 2023. [find](https://scholar.google.com/scholar?q=Zhiyuan+Zeng%2C+Jiatong+Yu%2C+Tianyu+Gao%2C+Yu+Meng%2C+Tanya+Goyal%2C+and+Danqi+Chen%2E+Evaluating+large+language+models+at+evaluating+instruction+following%2E+arXiv+preprint+arXiv%3A2310%2E07641%2C+2023%2E)
52. Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. [Generative verifiers: Reward modeling as next-token prediction](https://md.research.elamur.ai/free/en/papers/zhang-2024-generative-verifiers-reward-modeling-as/), 2024. URL https://arxiv.org/abs/2408.15240. [↗](https://arxiv.org/abs/2408.15240)
53. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Conference on Neural Information Processing Systems Track on Dat…
54. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/), 2023b. URL [https://arxiv.org/](https://arxiv.org/) abs/2306.05685. [↗](https://arxiv.org/)
55. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. [Sglang: Efficient execution of structured language model programs](https://md.research.elamur.ai/free/en/papers/zheng-2023-sglang-efficient-execution-structured-language/), 2024. URL https://arxiv.org/abs/2312.07104. [↗](https://arxiv.org/abs/2312.07104)
56. Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang, Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang, Tianxiang Sun, Hang Yan, Tao Gui, Qi Zhang, Xipen… — [Secrets of RLHF in Large Language Models Part I: PPO](https://md.research.elamur.ai/free/en/papers/zheng-2023-secrets-rlhf-large-language-models/)
57. Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, Karthik Ganesan, Wei-Lin Chiang, Jian Zhang, and Jiantao Jiao. Starling-7b: Improving helpfulness and harmlessness with RLAIF. In First Conference on Language Modeling, 2024. URL [https://openreview.net/forum?id=](https://openreview.net/forum?id=) GqDntYTTbk. [↗](https://openreview.net/forum?id=)
58. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020. [find](https://scholar.google.com/scholar?q=Daniel+M%2E+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B%2E+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+Fine%2Dtuning+language+models+from+human+preferences%2C+2020%2E)

*Updated 3 November 2026*