---
title: "RLHF Workflow: From Reward Modeling to Online RLHF (2024)"
description: "This report offers a reproducible recipe for the online iterative approach. 95 references, cited by 107 papers here."
authors: ["Hanze Dong", "Wei Xiong", "Bo Pang", "Haoxiang Wang", "Han Zhao", "Yingbo Zhou", "Nan Jiang", "Doyen Sahoo", "Caiming Xiong", "Tong Zhang"]
year: 2024
venue: "arXiv:2405.07863"
source: "https://arxiv.org/abs/2405.07863"
doi: "10.48550/arxiv.2405.07863"
arxiv: "2405.07863"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/dong-2024-rlhf-workflow-reward-modeling-online/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dong-2024-rlhf-workflow-reward-modeling-online/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / RLHF Workflow: From Reward Modeling to Online RLHF

# RLHF Workflow: From Reward Modeling to Online RLHF

**Authors:** [Hanze Dong](https://md.research.elamur.ai/free/en/authors/hanze-dong/), [Wei Xiong](https://md.research.elamur.ai/free/en/authors/wei-xiong/), [Bo Pang](https://md.research.elamur.ai/free/en/authors/bo-pang/), [Haoxiang Wang](https://md.research.elamur.ai/free/en/authors/haoxiang-wang/), [Han Zhao](https://md.research.elamur.ai/free/en/authors/han-zhao/), [Yingbo Zhou](https://md.research.elamur.ai/free/en/authors/yingbo-zhou/), [Nan Jiang](https://md.research.elamur.ai/free/en/authors/nan-jiang/), [Doyen Sahoo](https://md.research.elamur.ai/free/en/authors/doyen-sahoo/), [Caiming Xiong](https://md.research.elamur.ai/free/en/authors/caiming-xiong/) and [Tong Zhang](https://md.research.elamur.ai/free/en/authors/tong-zhang/) · **Year:** 2024 · **Venue:** arXiv:2405.07863 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2405.07863) · [PDF](https://arxiv.org/pdf/2405.07863) · [DOI 10.48550/arxiv.2405.07863](https://doi.org/10.48550/arxiv.2405.07863) · [arXiv 2405.07863](https://arxiv.org/abs/2405.07863) · [OpenAlex](https://openalex.org/W4396914049)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [107](https://md.research.elamur.ai/free/en/papers/dong-2024-rlhf-workflow-reward-modeling-online/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Online iterative RLHF has been reported to clearly beat offline alignment, yet open-source alignment work mostly stays offline. This report offers a reproducible recipe for the online iterative approach. Real-time human judgments are too costly for most open teams, so the authors first train a proxy preference (reward) model on a broad mix of open-source preference datasets and use it in place of human raters. They then explain the theory behind online exploration and describe a practical pipeline. In each round, the current policy produces fresh responses, the proxy model labels them, and the new pairs are added to the data for direct preference learning in the DPO family. The resulting model, trained only on open data after SFT, performs strongly on AlpacaEval-2, Arena-Hard and MT-Bench, as well as academic tests such as HumanEval and TruthfulQA. The work matters because the authors release models, curated datasets and step-by-step code. This makes online RLHF practical for groups with limited resources.

## Key points
- The method avoids PPO, which needs many models in GPU memory at once and heavy hyper-parameter tuning. It relies on direct preference learning with online, on-policy data collection to reduce out-of-distribution problems.
- Main model results from Table 4: LC AlpacaEval-2 31.3, Arena-Hard 29.1, GSM-8K 80.7, MMLU 65.3, HumanEval 64.6. With the "alpaca eval gpt4 turbo fn" configuration, LC AlpacaEval-2 is 37.2.
- Adding a character-length penalty to the reward produced a "concise" variant:
  - LC AlpacaEval-2 rose to 38.1 and some academic scores improved.
  - Average response length fell from 656 to 382.
  - Arena-Hard dropped to 22.1, which suggests that benchmark favours verbose answers.
- Replacing the authors' reward model with UltraRM-13B made results worse (LC AlpacaEval-2 20.7, longer outputs, weaker reasoning scores). This shows how much the outcome depends on reward model quality.
- The authors warn about benchmark hacking: higher benchmark scores were reachable (44.84 LC AlpacaEval-2), but at a large cost on academic benchmarks.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Illinois Urbana-Champaign](https://md.research.elamur.ai/free/en/universities/university-of-illinois-urbana-champaign/) — University, Urbana, United States · 209 works · 71,394 citations
- [Salesforce](https://md.research.elamur.ai/free/en/companies/salesforce/) — Company, San Francisco, United States · 59 works · 18,172 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong and Tong Zhang. 2024. RLHF Workflow: From Reward Modeling to Online RLHF. arXiv:2405.07863.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{dong2024rlhf,
  title = {{RLHF Workflow: From Reward Modeling to Online RLHF}},
  author = {Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong and Tong Zhang},
  year = {2024},
  doi = {10.48550/arxiv.2405.07863},
  eprint = {2405.07863},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.07863}
}
```

- [← previous ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models](https://md.research.elamur.ai/free/en/papers/baek-2024-researchagent-iterative-research-idea/)
- [next → RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors](https://md.research.elamur.ai/free/en/papers/dugan-2024-raid-shared-benchmark-robust-evaluation/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

95 references, 38 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. Introducing claude. 2023. URL [https://www.anthropic.com/index/introducing-claude](https://www.anthropic.com/index/introducing-claude). [↗](https://www.anthropic.com/index/introducing-claude)
2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732, 2021.
3. Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. [A general theoretical paradigm to understand learning from human preferences](https://md.research.elamur.ai/free/en/papers/azar-2023-general-theoretical-paradigm-understand-learning/). arXiv preprint arXiv:2310.12036, 2023.
4. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. 5 [https://huggingface.co/d…](https://huggingface.co/d…) [↗](https://huggingface.co/d…)
5. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073, 2022b.
6. Hritik Bansal, John Dang, and Aditya Grover. Peering through preferences: Unraveling feedback acquisition for aligning large language models. arXiv preprint arXiv:2308.15812, 2023. — [arXiv:2308.15812](https://arxiv.org/abs/2308.15812) [find](https://scholar.google.com/scholar?q=Hritik+Bansal%2C+John+Dang%2C+and+Aditya+Grover%2E+Peering+through+preferences%3A+Unraveling+feedback+acquisition+for+aligning+large+language+models%2E+arXiv+preprint+arXiv%3A2308%2E15812%2C+2023%2E+%E2%80%94+%5BarXiv%3A2308%2E15812)
7. Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D’Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees on the best-of-n alignment policy. arXiv preprint arXiv:2401.01879, 2024. — [arXiv:2401.01879](https://arxiv.org/abs/2401.01879) [find](https://scholar.google.com/scholar?q=Ahmad+Beirami%2C+Alekh+Agarwal%2C+Jonathan+Berant%2C+Alexander+D%E2%80%99Amour%2C+Jacob+Eisenstein%2C+Chirag+Nagpal%2C+and+Ananda+Theertha+Suresh%2E+Theoretical+guarantees+on+the+best%2Dof%2Dn+alignment+policy%2E+arXiv+preprint+)
8. Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. [find](https://scholar.google.com/scholar?q=Ralph+Allan+Bradley+and+Milton+E+Terry%2E+Rank+analysis+of+incomplete+block+designs%3A+I%2E+the+method+of+paired+comparisons%2E+Biometrika%2C+39%283%2F4%29%3A324%E2%80%93345%2C+1952%2E)
9. Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. [Weak-to-strong generalization: Eliciting strong capabilities with weak supervision](https://md.research.elamur.ai/free/en/papers/burns-2023-weak-strong-generalization-eliciting-strong/). arXiv preprint arXiv:2312.09390, 2023.
10. Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, et al. Human alignment of large language models through online preference optimisation. arXiv preprint arXiv:2403.08635, 2024. — [arXiv:2403.08635](https://arxiv.org/abs/2403.08635) [find](https://scholar.google.com/scholar?q=Daniele+Calandriello%2C+Daniel+Guo%2C+Remi+Munos%2C+Mark+Rowland%2C+Yunhao+Tang%2C+Bernardo+Avila+Pires%2C+Pierre+Harvey+Richemond%2C+Charline+Le+Lan%2C+Michal+Valko%2C+Tianqi+Liu%2C+et+al%2E+Human+alignment+of+large+langu)
11. Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang, Dale Schuurmans, Yuejie Chi, and Bo Dai. Value-incentivized preference optimization: A unified approach to online and offline rlhf. arXiv preprint arXiv:2405.19320, 2024. — [arXiv:2405.19320](https://arxiv.org/abs/2405.19320) [find](https://scholar.google.com/scholar?q=Shicong+Cen%2C+Jincheng+Mei%2C+Katayoon+Goshvadi%2C+Hanjun+Dai%2C+Tong+Yang%2C+Sherry+Yang%2C+Dale+Schuurmans%2C+Yuejie+Chi%2C+and+Bo+Dai%2E+Value%2Dincentivized+preference+optimization%3A+A+unified+approach+to+online+and+)
12. Alex J Chan, Hao Sun, Samuel Holt, and Mihaela van der Schaar. Dense reward for free in reinforcement learning from human feedback. arXiv preprint arXiv:2402.00782, 2024. — [arXiv:2402.00782](https://arxiv.org/abs/2402.00782) [find](https://scholar.google.com/scholar?q=Alex+J+Chan%2C+Hao+Sun%2C+Samuel+Holt%2C+and+Mihaela+van+der+Schaar%2E+Dense+reward+for+free+in+reinforcement+learning+from+human+feedback%2E+arXiv+preprint+arXiv%3A2402%2E00782%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E00782%5D%28https%3A%2F%2F)
13. Jonathan D Chang, Wenhao Shan, Owen Oertell, Kianté Brantley, Dipendra Misra, Jason D Lee, and Wen Sun. Dataset reset policy optimization for rlhf. arXiv preprint arXiv:2404.08495, 2024. — [arXiv:2404.08495](https://arxiv.org/abs/2404.08495) [find](https://scholar.google.com/scholar?q=Jonathan+D+Chang%2C+Wenhao+Shan%2C+Owen+Oertell%2C+Kiante%CC%81+Brantley%2C+Dipendra+Misra%2C+Jason+D+Lee%2C+and+Wen+Sun%2E+Dataset+reset+policy+optimization+for+rlhf%2E+arXiv+preprint+arXiv%3A2404%2E08495%2C+2024%2E+%E2%80%94+%5BarXiv%3A240)
14. Huayu Chen, Guande He, Hang Su, and Jun Zhu. Noise contrastive alignment of language models with explicit rewards. arXiv preprint arXiv:2402.05369, 2024a. — [arXiv:2402.05369](https://arxiv.org/abs/2402.05369) [find](https://scholar.google.com/scholar?q=Huayu+Chen%2C+Guande+He%2C+Hang+Su%2C+and+Jun+Zhu%2E+Noise+contrastive+alignment+of+language+models+with+explicit+rewards%2E+arXiv+preprint+arXiv%3A2402%2E05369%2C+2024a%2E+%E2%80%94+%5BarXiv%3A2402%2E05369%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F24)
15. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
16. Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024b. [find](https://scholar.google.com/scholar?q=Zixiang+Chen%2C+Yihe+Deng%2C+Huizhuo+Yuan%2C+Kaixuan+Ji%2C+and+Quanquan+Gu%2E+Self%2Dplay+fine%2Dtuning+converts+weak+language+models+to+strong+language+models%2E+arXiv+preprint+arXiv%3A2401%2E01335%2C+2024b%2E)
17. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL [https://lmsys.org/blog/2023-03-30-vicuna/…](https://lmsys.org/blog/2023-03-30-vicuna/…) [↗](https://lmsys.org/blog/2023-03-30-vicuna/…)
18. Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. On the weaknesses of reinforcement learning for neural machine translation. arXiv preprint arXiv:1907.01752, 2019. — [arXiv:1907.01752](https://arxiv.org/abs/1907.01752) [find](https://scholar.google.com/scholar?q=Leshem+Choshen%2C+Lior+Fox%2C+Zohar+Aizenbud%2C+and+Omri+Abend%2E+On+the+weaknesses+of+reinforcement+learning+for+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1907%2E01752%2C+2019%2E+%E2%80%94+%5BarXiv%3A1907%2E01752%5D%28https%3A)
19. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. [find](https://scholar.google.com/scholar?q=Paul+F+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+neural+information+processing+systems%2C+30%2C+2017%2E)
20. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv:1803.05457v1, 2018.
21. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
22. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023. [find](https://scholar.google.com/scholar?q=Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Guanming+Yao%2C+Wei+Zhu%2C+Yuan+Ni%2C+Guotong+Xie%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+Ultrafeedback%3A+Boosting+language+models+with+high%2Dquality+feedback%2C+2023%2E)
23. Luigi Daniele and Suphavadeeprasit. Amplify-instruct: Synthetically generated diverse multi-turn conversations for effecient llm training. arXiv preprint arXiv:(coming soon), 2023. URL [https://huggingface.co/datasets/LDJnr/Capybara](https://huggingface.co/datasets/LDJnr/Capybara). [↗](https://huggingface.co/datasets/LDJnr/Capybara)
24. Shizhe Diao, Rui Pan, Hanze Dong, Ka Shun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang. Lmflow: An extensible toolkit for finetuning and inference of large foundation models. arXiv preprint arXiv:2306.12420, 2023. [find](https://scholar.google.com/scholar?q=Shizhe+Diao%2C+Rui+Pan%2C+Hanze+Dong%2C+Ka+Shun+Shum%2C+Jipeng+Zhang%2C+Wei+Xiong%2C+and+Tong+Zhang%2E+Lmflow%3A+An+extensible+toolkit+for+finetuning+and+inference+of+large+foundation+models%2E+arXiv+preprint+arXiv%3A230)
25. Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang. [RAFT: Reward ranked finetuning for generative foundation model alignment](https://md.research.elamur.ai/free/en/papers/dong-2023-raft-reward-ranked-finetuning-generative/). Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL [https://openreview.net/fo…](https://openreview.net/fo…) [↗](https://openreview.net/fo…)
26. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. [Alpacafarm: A simulation framework for methods that learn from human feedback](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/). arXiv preprint arXiv:2305.14387, 2023.
27. Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo. arXiv preprint arXiv:2005.12729, 2020. [find](https://scholar.google.com/scholar?q=Logan+Engstrom%2C+Andrew+Ilyas%2C+Shibani+Santurkar%2C+Dimitris+Tsipras%2C+Firdaus+Janoos%2C+Larry+Rudolph%2C+and+Aleksander+Madry%2E+Implementation+matters+in+deep+policy+gradients%3A+A+case+study+on+ppo+and+trpo%2E+a)
28. Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of… [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2C+Yejin+Choi%2C+and+Swabha+Swayamdipta%2E+Understanding+dataset+difficulty+with+V%2Dusable+information%2E+In+Kamalika+Chaudhuri%2C+Stefanie+Jegelka%2C+Le+Song%2C+Csaba+Szepesvari%2C+Gang+Niu%2C+and+Siva)
29. Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024. [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2C+Winnie+Xu%2C+Niklas+Muennighoff%2C+Dan+Jurafsky%2C+and+Douwe+Kiela%2E+Kto%3A+Model+alignment+as+prospect+theoretic+optimization%2E+arXiv+preprint+arXiv%3A2402%2E01306%2C+2024%2E)
30. Leo Gao, John Schulman, and Jacob Hilton. [Scaling laws for reward model overoptimization](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/). In International Conference on Machine Learning, pp. 10835–10866. PMLR, 2023.
31. Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. [Reinforced self-training (rest) for language modeling](https://md.research.elamur.ai/free/en/papers/gulcehre-2023-reinforced-self-training-rest-language/). arXiv preprint arXiv:2308.08998, 2023.
32. Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792, 2024. — [arXiv:2402.04792](https://arxiv.org/abs/2402.04792) [find](https://scholar.google.com/scholar?q=Shangmin+Guo%2C+Biao+Zhang%2C+Tianlin+Liu%2C+Tianqi+Liu%2C+Misha+Khalman%2C+Felipe+Llinares%2C+Alexandre+Rame%2C+Thomas+Mesnard%2C+Yao+Zhao%2C+Bilal+Piot%2C+et+al%2E+Direct+language+model+alignment+from+online+ai+feedback%2E)
33. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300, 2020.
34. Braden Hancock Hoang Tran, Chris Glaze. Snorkel-mistral-pairrm-dpo. [https://huggingface.co/snorkelai/Snorkel-Mistral-PairRM-DPO](https://huggingface.co/snorkelai/Snorkel-Mistral-PairRM-DPO), 2024. URL https://huggingface.co/snorkelai/Snorkel-Mistral-PairRM-DPO. [↗](https://huggingface.co/snorkelai/Snorkel-Mistral-PairRM-DPO)
35. Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. [Beavertails: Towards improved safety alignment of llm via a human-preference dataset](https://md.research.elamur.ai/free/en/papers/ji-2023-beavertails-towards-improved-safety-alignment/). Advances in Neural Information Processing Systems, 36, 2024.
36. Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise comparison and generative fusion. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (ACL 2023), 2023. [find](https://scholar.google.com/scholar?q=Dongfu+Jiang%2C+Xiang+Ren%2C+and+Bill+Yuchen+Lin%2E+Llm%2Dblender%3A+Ensembling+large+language+models+with+pairwise+comparison+and+generative+fusion%2E+In+Proceedings+of+the+61th+Annual+Meeting+of+the+Association)
37. Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, Filip Radlinski, and Geri Gay. Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Transactions on Information Systems (TOIS), 25(2):7–es, 2007. [find](https://scholar.google.com/scholar?q=Thorsten+Joachims%2C+Laura+Granka%2C+Bing+Pan%2C+Helene+Hembrooke%2C+Filip+Radlinski%2C+and+Geri+Gay%2E+Evaluating+the+accuracy+of+implicit+feedback+from+clicks+and+query+reformulations+in+web+search%2E+ACM+Transac)
38. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
39. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787, 2024. [find](https://scholar.google.com/scholar?q=Nathan+Lambert%2C+Valentina+Pyatkin%2C+Jacob+Morrison%2C+LJ+Miranda%2C+Bill+Yuchen+Lin%2C+Khyathi+Chandu%2C+Nouha+Dziri%2C+Sachin+Kumar%2C+Tom+Zick%2C+Yejin+Choi%2C+et+al%2E+Rewardbench%3A+Evaluating+reward+models+for+langua)
40. Ziniu Li, Tian Xu, Yushun Zhang, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. arXiv e-prints, pp. arXiv–2310, 2023. [find](https://scholar.google.com/scholar?q=Ziniu+Li%2C+Tian+Xu%2C+Yushun+Zhang%2C+Yang+Yu%2C+Ruoyu+Sun%2C+and+Zhi%2DQuan+Luo%2E+Remax%3A+A+simple%2C+effective%2C+and+efficient+reinforcement+learning+method+for+aligning+large+language+models%2E+arXiv+e%2Dprints%2C+pp%2E+a)
41. Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and ”Teknium”. Openorca: An open dataset of gpt augmented flan reasoning traces. https://https://huggingface.co/Open-Orca/OpenOrca, 2023a. [find](https://scholar.google.com/scholar?q=Wing+Lian%2C+Bleys+Goodson%2C+Eugene+Pentland%2C+Austin+Cook%2C+Chanvichet+Vong%2C+and+%E2%80%9DTeknium%E2%80%9D%2E+Openorca%3A+An+open+dataset+of+gpt+augmented+flan+reasoning+traces%2E+https%3A%2F%2Fhttps%3A%2F%2Fhuggingface%2Eco%2FOpen%2DOrca%2FOpenO)
42. Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and ”Teknium”. Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification, 2023b. URL https://https://huggingface.co/Open-Orca/SlimOrca. [find](https://scholar.google.com/scholar?q=Wing+Lian%2C+Guan+Wang%2C+Bleys+Goodson%2C+Eugene+Pentland%2C+Austin+Cook%2C+Chanvichet+Vong%2C+and+%E2%80%9DTeknium%E2%80%9D%2E+Slimorca%3A+An+open+dataset+of+gpt%2D4+augmented+flan+reasoning+traces%2C+with+verification%2C+2023b%2E+URL+htt)
43. Stephanie Lin, Jacob Hilton, and Owain Evans. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). arXiv preprint arXiv:2109.07958, 2021.
44. Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. [Statistical rejection sampling improves preference optimization](https://md.research.elamur.ai/free/en/papers/liu-2023-statistical-rejection-sampling-improves-preference/). arXiv preprint arXiv:2309.06657, 2023a.
45. Zhihan Liu, Miao Lu, Wei Xiong, Han Zhong, Hao Hu, Shenao Zhang, Sirui Zheng, Zhuoran Yang, and Zhaoran Wang. Maximize to explore: One objective function fusing estimation, planning, and exploration. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. [find](https://scholar.google.com/scholar?q=Zhihan+Liu%2C+Miao+Lu%2C+Wei+Xiong%2C+Han+Zhong%2C+Hao+Hu%2C+Shenao+Zhang%2C+Sirui+Zheng%2C+Zhuoran+Yang%2C+and+Zhaoran+Wang%2E+Maximize+to+explore%3A+One+objective+function+fusing+estimation%2C+planning%2C+and+exploration%2E+)
46. Meta. Introducing meta llama 3: The most capable openly available llm to date. Meta AI Blog, 2024. [https://ai.meta.com/blog/meta-llama-3/](https://ai.meta.com/blog/meta-llama-3/). [↗](https://ai.meta.com/blog/meta-llama-3/)
47. Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. [Orca-math: Unlocking the potential of slms in grade school math](https://md.research.elamur.ai/free/en/papers/mitra-2024-orca-math-unlocking-potential-slms/), 2024.
48. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. [Webgpt: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/). arXiv preprint arXiv:2112.09332, 2021.
49. Zach Nussbaum, John X. Morris, Brandon Duderstadt, and Andriy Mulyar. [Nomic embed: Training a reproducible long context text embedder](https://md.research.elamur.ai/free/en/papers/nussbaum-2024-nomic-embed-training-reproducible-long/), 2024.
50. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). ArXiv, abs/2303.08774, 2023.
51. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
52. Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. West-of-n: Synthetic preference generation for improved reward modeling. arXiv preprint arXiv:2401.[12086, 2024](https://md.research.elamur.ai/free/en/papers/pace-2024-west-n-synthetic-preferences-self/).
53. Bo Pang, Caiming Xiong, and Yingbo Zhou. Arm: Alignment with residual energy-based model. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, June 2024. [find](https://scholar.google.com/scholar?q=Bo+Pang%2C+Caiming+Xiong%2C+and+Yingbo+Zhou%2E+Arm%3A+Alignment+with+residual+energy%2Dbased+model%2E+In+Proceedings+of+the+2024+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Lingu)
54. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. [Instruction tuning with gpt-4](https://md.research.elamur.ai/free/en/papers/peng-2023-instruction-tuning-gpt-4/). arXiv preprint arXiv:2304.03277, 2023.
55. Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, et al. [Large language models are effective text rankers with pairwise ranking prompting](https://md.research.elamur.ai/free/en/papers/qin-2024-large-language-models-are-effective/). arXiv preprint arXiv:2306.17563, 2023.
56. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). arXiv preprint arXiv:2305.18290, 2023.
57. Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715, 2024. [find](https://scholar.google.com/scholar?q=Corby+Rosset%2C+Ching%2DAn+Cheng%2C+Arindam+Mitra%2C+Michael+Santacroce%2C+Ahmed+Awadallah%2C+and+Tengyang+Xie%2E+Direct+nash+optimization%3A+Teaching+language+models+to+self%2Dimprove+with+general+preferences%2E+arXiv+p)
58. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)
59. Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make rl efficient. arXiv preprint arXiv:2210.06718, 2022. — [arXiv:2210.06718](https://arxiv.org/abs/2210.06718) [find](https://scholar.google.com/scholar?q=Yuda+Song%2C+Yifei+Zhou%2C+Ayush+Sekhari%2C+J+Andrew+Bagnell%2C+Akshay+Krishnamurthy%2C+and+Wen+Sun%2E+Hybrid+rl%3A+Using+both+offline+and+online+data+can+make+rl+efficient%2E+arXiv+preprint+arXiv%3A2210%2E06718%2C+2022%2E+%E2%80%94)
60. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In NeurIPS, 2020. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeff+Wu%2C+Daniel+M%2E+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+Christiano%2E+Learning+to+summarize+from+human+feedback%2E+In+NeurIPS%2C+2020%2E)
61. Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint arXiv:2401.04056, 2024. — [arXiv:2401.04056](https://arxiv.org/abs/2401.04056) [find](https://scholar.google.com/scholar?q=Gokul+Swamy%2C+Christoph+Dann%2C+Rahul+Kidambi%2C+Zhiwei+Steven+Wu%2C+and+Alekh+Agarwal%2E+A+minimaximalist+approach+to+reinforcement+learning+from+human+feedback%2E+arXiv+preprint+arXiv%3A2401%2E04056%2C+2024%2E+%E2%80%94+%5BarXi)
62. Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of llms should leverage suboptimal, on-policy data. arXiv preprint arXiv:2404.14367, 2024. [find](https://scholar.google.com/scholar?q=Fahim+Tajwar%2C+Anikait+Singh%2C+Archit+Sharma%2C+Rafael+Rafailov%2C+Jeff+Schneider%2C+Tengyang+Xie%2C+Stefano+Ermon%2C+Chelsea+Finn%2C+and+Aviral+Kumar%2E+Preference+fine%2Dtuning+of+llms+should+leverage+suboptimal%2C+on%2D)
63. Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749, 2024. — [arXiv:2402.05749](https://arxiv.org/abs/2402.05749) [find](https://scholar.google.com/scholar?q=Yunhao+Tang%2C+Zhaohan+Daniel+Guo%2C+Zeyu+Zheng%2C+Daniele+Calandriello%2C+Re%CC%81mi+Munos%2C+Mark+Rowland%2C+Pierre+Harvey+Richemond%2C+Michal+Valko%2C+Bernardo+A%CC%81vila+Pires%2C+and+Bilal+Piot%2E+Generalized+preference+optim)
64. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
65. Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023. URL [https://huggingface.co/datasets/teknium/OpenHermes-2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5). [↗](https://huggingface.co/datasets/teknium/OpenHermes-2.5)
66. Teknium1. Gpteacher, 2023. URL [https://github.com/teknium1/GPTeacher](https://github.com/teknium1/GPTeacher). GitHub repository. [↗](https://github.com/teknium1/GPTeacher)
67. Li* Tianle, Chiang Wei-Lin, Dunlap Evan, Frick nad Lisa, Zhu Banghua, Gonzalez Joseph E., and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024. URL [https://lmsys.org/blog/2024-04-19-arena-hard/](https://lmsys.org/blog/2024-04-19-arena-hard/). [↗](https://lmsys.org/blog/2024-04-19-arena-hard/)
68. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
69. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944, 2023. [find](https://scholar.google.com/scholar?q=Lewis+Tunstall%2C+Edward+Beeching%2C+Nathan+Lambert%2C+Nazneen+Rajani%2C+Kashif+Rasul%2C+Younes+Belkada%2C+Shengyi+Huang%2C+Leandro+von+Werra%2C+Cle%CC%81mentine+Fourrier%2C+Nathan+Habib%2C+et+al%2E+Zephyr%3A+Direct+distillation+)
70. Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. [Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards](https://md.research.elamur.ai/free/en/papers/wang-2024-arithmetic-control-llms-diverse-user/). arXiv preprint arXiv:2402.18571, 2024.
71. Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. [Helpsteer: Multi-attribute helpfulness dataset for steerlm](https://md.research.elamur.ai/free/en/papers/wang-2024-helpsteer-multi-attribute-helpfulness-dataset/), 2023.
72. Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is all you need. arXiv preprint arXiv:2312.02120, 2023. [find](https://scholar.google.com/scholar?q=Yuxiang+Wei%2C+Zhe+Wang%2C+Jiawei+Liu%2C+Yifeng+Ding%2C+and+Lingming+Zhang%2E+Magicoder%3A+Source+code+is+all+you+need%2E+arXiv+preprint+arXiv%3A2312%2E02120%2C+2023%2E)
73. Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. [Recursively summarizing books with human feedback](https://md.research.elamur.ai/free/en/papers/wu-2021-recursively-summarizing-books-human-feedback/). arXiv preprint arXiv:2109.10862, 2021.
74. Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675, 2024. — [arXiv:2405.00675](https://arxiv.org/abs/2405.00675) [find](https://scholar.google.com/scholar?q=Yue+Wu%2C+Zhiqing+Sun%2C+Huizhuo+Yuan%2C+Kaixuan+Ji%2C+Yiming+Yang%2C+and+Quanquan+Gu%2E+Self%2Dplay+preference+optimization+for+language+model+alignment%2E+arXiv+preprint+arXiv%3A2405%2E00675%2C+2024%2E+%E2%80%94+%5BarXiv%3A2405%2E00675%5D)
75. Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 34: 27395–27407, 2021. [find](https://scholar.google.com/scholar?q=Tengyang+Xie%2C+Nan+Jiang%2C+Huan+Wang%2C+Caiming+Xiong%2C+and+Yu+Bai%2E+Policy+finetuning%3A+Bridging+sample%2Defficient+offline+and+online+reinforcement+learning%2E+Advances+in+neural+information+processing+systems)
76. Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf. arXiv preprint arXiv:2405.21046, 2024. — [arXiv:2405.21046](https://arxiv.org/abs/2405.21046) [find](https://scholar.google.com/scholar?q=Tengyang+Xie%2C+Dylan+J+Foster%2C+Akshay+Krishnamurthy%2C+Corby+Rosset%2C+Ahmed+Awadallah%2C+and+Alexander+Rakhlin%2E+Exploratory+preference+optimization%3A+Harnessing+implicit+q%2A%2Dapproximation+for+sample%2Defficient)
77. Wei Xiong. A sufficient condition of sample-efficient reinforcement learning with general function approximation. The Hong Kong University of Science and Technology, 2023. [find](https://scholar.google.com/scholar?q=Wei+Xiong%2E+A+sufficient+condition+of+sample%2Defficient+reinforcement+learning+with+general+function+approximation%2E+The+Hong+Kong+University+of+Science+and+Technology%2C+2023%2E)
78. Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. [Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint](https://md.research.elamur.ai/free/en/papers/xiong-2023-iterative-preference-learning-human-feedback/). 2023.
79. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.[12244, 2023a](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/).
80. Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682, 2023b. — [arXiv:2312.16682](https://arxiv.org/abs/2312.16682) [find](https://scholar.google.com/scholar?q=Jing+Xu%2C+Andrew+Lee%2C+Sainbayar+Sukhbaatar%2C+and+Jason+Weston%2E+Some+things+are+more+cringe+than+others%3A+Preference+optimization+with+the+pairwise+cringe+loss%2E+arXiv+preprint+arXiv%3A2312%2E16682%2C+2023b%2E+%E2%80%94+%5B)
81. Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du, Wenyi Zhao, et al. [Chatglm-math: Improving math problem-solving in large language models with a self-critique pipeline](https://md.research.elamur.ai/free/en/papers/xu-2024-chatglm-math-improving-math/). arXiv preprint arXiv:2404.02893, 2024.
82. Chenlu Ye, Wei Xiong, Yuheng Zhang, Nan Jiang, and Tong Zhang. A theoretical analysis of nash learning from human feedback under general kl-regularized preference. arXiv preprint arXiv:2402.07314, 2024. — [arXiv:2402.07314](https://arxiv.org/abs/2402.07314) [find](https://scholar.google.com/scholar?q=Chenlu+Ye%2C+Wei+Xiong%2C+Yuheng+Zhang%2C+Nan+Jiang%2C+and+Tong+Zhang%2E+A+theoretical+analysis+of+nash+learning+from+human+feedback+under+general+kl%2Dregularized+preference%2E+arXiv+preprint+arXiv%3A2402%2E07314%2C+202)
83. Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. [Advancing llm reasoning generalists with preference trees](https://md.research.elamur.ai/free/en/papers/yuan-2024-advancing-llm-reasoning-generalists-preference/), 2024a.
84. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024b. [find](https://scholar.google.com/scholar?q=Weizhe+Yuan%2C+Richard+Yuanzhe+Pang%2C+Kyunghyun+Cho%2C+Sainbayar+Sukhbaatar%2C+Jing+Xu%2C+and+Jason+Weston%2E+Self%2Drewarding+language+models%2E+arXiv+preprint+arXiv%3A2401%2E10020%2C+2024b%2E)
85. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. [Rrhf: Rank responses to align language models with human feedback without tears](https://md.research.elamur.ai/free/en/papers/yuan-2023-rrhf-rank-responses-align-language/). arXiv preprint arXiv:2304.05302, 2023.
86. Xiang Yue, Ge Zhang Xingwei Qu, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. [Mammoth: Building math generalist models through hybrid instruction tuning](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/). arXiv preprint arXiv:2309.05653, 2023.
87. Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The k-armed dueling bandits problem. Journal of Computer and System Sciences, 78(5):1538–1556, 2012. [find](https://scholar.google.com/scholar?q=Yisong+Yue%2C+Josef+Broder%2C+Robert+Kleinberg%2C+and+Thorsten+Joachims%2E+The+k%2Darmed+dueling+bandits+problem%2E+Journal+of+Computer+and+System+Sciences%2C+78%285%29%3A1538%E2%80%931556%2C+2012%2E)
88. Shenao Zhang, Donghan Yu, Hiteshi Sharma, Ziyi Yang, Shuohang Wang, Hany Hassan, and Zhaoran Wang. Self-exploring language models: Active preference elicitation for online alignment. arXiv preprint arXiv:2405.19332, 2024. — [arXiv:2405.19332](https://arxiv.org/abs/2405.19332) [find](https://scholar.google.com/scholar?q=Shenao+Zhang%2C+Donghan+Yu%2C+Hiteshi+Sharma%2C+Ziyi+Yang%2C+Shuohang+Wang%2C+Hany+Hassan%2C+and+Zhaoran+Wang%2E+Self%2Dexploring+language+models%3A+Active+preference+elicitation+for+online+alignment%2E+arXiv+preprint+ar)
89. Tong Zhang. Mathematical analysis of machine learning algorithms. Cambridge University Press, 2023. [find](https://scholar.google.com/scholar?q=Tong+Zhang%2E+Mathematical+analysis+of+machine+learning+algorithms%2E+Cambridge+University+Press%2C+2023%2E)
90. Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. [Slic-hf: Sequence likelihood calibration with human feedback](https://md.research.elamur.ai/free/en/papers/zhao-2023-slic-hf-sequence-likelihood-calibration/). arXiv preprint arXiv:2305.10425, 2023.
91. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). arXiv preprint arXiv:2306.05685, 2023.
92. Han Zhong, Wei Xiong, Sirui Zheng, Liwei Wang, Zhaoran Wang, Zhuoran Yang, and Tong Zhang. Gec: A unified framework for interactive decision making in mdp, pomdp, and beyond. arXiv preprint arXiv:2211.01962, 2022. — [arXiv:2211.01962](https://arxiv.org/abs/2211.01962) [find](https://scholar.google.com/scholar?q=Han+Zhong%2C+Wei+Xiong%2C+Sirui+Zheng%2C+Liwei+Wang%2C+Zhaoran+Wang%2C+Zhuoran+Yang%2C+and+Tong+Zhang%2E+Gec%3A+A+unified+framework+for+interactive+decision+making+in+mdp%2C+pomdp%2C+and+beyond%2E+arXiv+preprint+arXiv%3A2211)
93. Han Zhong, Guhao Feng, Wei Xiong, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv:2404.18922, 2024. — [arXiv:2404.18922](https://arxiv.org/abs/2404.18922) [find](https://scholar.google.com/scholar?q=Han+Zhong%2C+Guhao+Feng%2C+Wei+Xiong%2C+Li+Zhao%2C+Di+He%2C+Jiang+Bian%2C+and+Liwei+Wang%2E+Dpo+meets+ppo%3A+Reinforced+token+optimization+for+rlhf%2E+arXiv+preprint+arXiv%3A2404%2E18922%2C+2024%2E+%E2%80%94+%5BarXiv%3A2404%2E18922%5D%28https%3A%2F)
94. Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving llm helpfulness & harmlessness with rlaif, 2023. [find](https://scholar.google.com/scholar?q=Banghua+Zhu%2C+Evan+Frick%2C+Tianhao+Wu%2C+Hanlin+Zhu%2C+and+Jiantao+Jiao%2E+Starling%2D7b%3A+Improving+llm+helpfulness+%26+harmlessness+with+rlaif%2C+2023%2E)
95. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. [find](https://scholar.google.com/scholar?q=Daniel+M+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint+arXiv%3A19)

*Updated 2 November 2026*