---
title: "Unpacking DPO and PPO — paper by Ivison et al. (2024)"
description: "Preference-based training is now a standard final stage for language models. 67 references, cited by 43 papers here."
authors: ["Hamish Ivison", "Yizhong Wang", "Jiacheng Liu", "Zeqiu Wu", "Valentina Pyatkin", "Nathan Lambert", "Noah A. Smith", "Yejin Choi", "Hannaneh Hajishirzi"]
year: 2024
venue: "arXiv:2406.09279"
source: "https://arxiv.org/abs/2406.09279"
doi: "10.48550/arxiv.2406.09279"
arxiv: "2406.09279"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ivison-2024-unpacking-dpo-ppo-disentangling-best/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ivison-2024-unpacking-dpo-ppo-disentangling-best/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback

# Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback

**Authors:** [Hamish Ivison](https://md.research.elamur.ai/free/en/authors/hamish-ivison/), [Yizhong Wang](https://md.research.elamur.ai/free/en/authors/yizhong-wang/), [Jiacheng Liu](https://md.research.elamur.ai/free/en/authors/jiacheng-liu/), [Zeqiu Wu](https://md.research.elamur.ai/free/en/authors/zeqiu-wu/), [Valentina Pyatkin](https://md.research.elamur.ai/free/en/authors/valentina-pyatkin/), [Nathan Lambert](https://md.research.elamur.ai/free/en/authors/nathan-lambert/), [Noah A. Smith](https://md.research.elamur.ai/free/en/authors/noah-a-smith/), [Yejin Choi](https://md.research.elamur.ai/free/en/authors/yejin-choi/) and [Hannaneh Hajishirzi](https://md.research.elamur.ai/free/en/authors/hannaneh-hajishirzi/) · **Year:** 2024 · **Venue:** [arXiv:2406.09279](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2024](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2024/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2406.09279) · [PDF](https://arxiv.org/pdf/2406.09279) · [DOI 10.48550/arxiv.2406.09279](https://doi.org/10.48550/arxiv.2406.09279) · [arXiv 2406.09279](https://arxiv.org/abs/2406.09279) · [OpenAlex](https://openalex.org/W4415798152)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [43](https://md.research.elamur.ai/free/en/papers/ivison-2024-unpacking-dpo-ppo-disentangling-best/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Preference-based training is now a standard final stage for language models. Published setups, however, combine different datasets, algorithms and evaluation suites, so it is unclear which ingredient actually drives the gains. The authors divide the pipeline into four components: preference data, the learning algorithm (PPO vs. DPO), the reward model, and the unlabelled prompts used during policy training. They vary each one in isolation, starting from the TÜLU 2 13B SFT model, a Llama 2 finetune.

They run DPO on 14 existing preference datasets of human, web-scraped and synthetic origin. They then compare DPO and PPO under matched conditions, scale reward models in both size and training data, and test domain-targeted prompts. Every variant is scored on 11 benchmarks.

The resulting order of impact is data quality first, then algorithm, then reward model, then prompts. The recommended recipe is synthetic UltraFeedback data combined with PPO and a large reward model. The study gives practitioners an evidence-based list of priorities and an openly released model and data suite to build on.

## Key points
- **Preference data had the largest effect.** Synthetic sets with fine-grained, per-aspect annotations worked best. Getting the chosen/rejected pairs right mattered more than how good the responses themselves were. Better data improved instruction following and truthfulness by up to 8%.
- **PPO consistently beat DPO** when both started from the same model and data. The margin was up to 2.5% on math and 1.2% on general benchmarks.
- **Larger reward models had a narrow payoff.** Scaling up to 70B, or adding reward-model training data, improved direct reward-model benchmarks and lifted GSM accuracy by up to about 5%. Other categories barely changed, and a 13B reward model came close when compute is limited.
- **Targeted prompts helped only in specific domains.** Policy-training prompts that match the target domain (e.g., math) gave further gains there, but did little for overall performance.
- **The best model outperformed strong baselines.** TÜLU 2+PPO 13B with the 70B UltraFeedback reward model beat the best DPO model and Llama 2 Chat 13B. Experiments on Llama 3.0 8B showed broadly similar trends. Models and datasets are released as the Tülu v2.5 suite, with code in EasyLM and open-instruct.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works

## Cite

Cite as: Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi and Hannaneh Hajishirzi. 2024. Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. arXiv:2406.09279.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ivison2024unpacking,
  title = {{Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback}},
  author = {Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi and Hannaneh Hajishirzi},
  year = {2024},
  doi = {10.48550/arxiv.2406.09279},
  eprint = {2406.09279},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.09279}
}
```

- [← previous mDPO: Conditional Preference Optimization for Multimodal Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2024-mdpo-conditional-preference-optimization-multimodal/)
- [next → WebLINX: Real-World Website Navigation with Multi-Turn Dialogue](https://md.research.elamur.ai/free/en/papers/lu-2024-weblinx-real-world-website-navigation/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

67 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Ahmadian%2C+C%2E+Cremer%2C+M%2E+Gall%C3%A9%2C+M%2E+Fadaee%2C+J%2E+Kreutzer%2C+O%2E+Pietquin%2C+A%2E+%C3%9Cst%C3%BCn%2C+and+S%2E+Hooker%2E+Back+to+Basics%3A+Revisiting+REINFORCE+Style+Optimization+for+Learning+from+Human+Feedback+in+LLMs%2C+2024%2E)
2. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton. [Program Synthesis with Large Language Models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/), 2021.
3. M. G. Azar, M. Rowland, B. Piot, D. Guo, D. Calandriello, M. Valko, and R. Munos. [A general theoretical paradigm to understand learning from human preferences](https://md.research.elamur.ai/free/en/papers/azar-2023-general-theoretical-paradigm-understand-learning/). arXiv preprint arXiv:2310.12036, 2023.
4. Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. [find](https://scholar.google.com/scholar?q=Y%2E+Bai%2C+A%2E+Jones%2C+K%2E+Ndousse%2C+A%2E+Askell%2C+A%2E+Chen%2C+N%2E+DasSarma%2C+D%2E+Drain%2C+S%2E+Fort%2C+D%2E+Ganguli%2C+T%2E+Henighan%2C+et+al%2E+Training+a+helpful+and+harmless+assistant+with+reinforcement+learning+from+human+feedb)
5. B. bench authors. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL [https://openreview.net/forum?id=uyTL5Bvosj](https://openreview.net/forum?id=uyTL5Bvosj). [↗](https://openreview.net/forum?id=uyTL5Bvosj)
6. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021.
7. W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, and I. Stoica. [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://md.research.elamur.ai/free/en/papers/chiang-2024-chatbot-arena-open-platform-evaluating/), 2024.
8. P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017. [find](https://scholar.google.com/scholar?q=P%2E+F%2E+Christiano%2C+J%2E+Leike%2C+T%2E+Brown%2C+M%2E+Martic%2C+S%2E+Legg%2C+and+D%2E+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+Neural+Information+Processing+Systems%2C+30%2C+2017%2E)
9. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. [Training Verifiers to Solve Math Word Problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
10. T. Coste, U. Anwar, R. Kirk, and D. Krueger. Reward model ensembles help mitigate overoptimization. In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=dcjtMYkpXx](https://openreview.net/forum?id=dcjtMYkpXx). [↗](https://openreview.net/forum?id=dcjtMYkpXx)
11. G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Y. Ni, G. Xie, Z. Liu, and M. Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023. [find](https://scholar.google.com/scholar?q=G%2E+Cui%2C+L%2E+Yuan%2C+N%2E+Ding%2C+G%2E+Yao%2C+W%2E+Zhu%2C+Y%2E+Ni%2C+G%2E+Xie%2C+Z%2E+Liu%2C+and+M%2E+Sun%2E+Ultrafeedback%3A+Boosting+language+models+with+high%2Dquality+feedback%2E+arXiv+preprint+arXiv%3A2310%2E01377%2C+2023%2E)
12. L. Daniele and Suphavadeeprasit. Amplify-Instruct: Synthetically Generated Diverse Multi-turn Conversations for Effecient LLM Training. arXiv preprint arXiv:(coming soon), 2023. URL [https://huggingface.co/datasets/LDJnr/Capybara](https://huggingface.co/datasets/LDJnr/Capybara). [↗](https://huggingface.co/datasets/LDJnr/Capybara)
13. Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arXiv preprint arXiv:2404.04475, 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Dubois%2C+B%2E+Galambosi%2C+P%2E+Liang%2C+and+T%2E+B%2E+Hashimoto%2E+Length%2DControlled+AlpacaEval%3A+A+Simple+Way+to+Debias+Automatic+Evaluators%2E+arXiv+preprint+arXiv%3A2404%2E04475%2C+2024%2E)
14. Y. Dubois, C. X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. S. Liang, and T. B. Hashimoto. [Alpacafarm: A simulation framework for methods that learn from human feedback](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/). Advances in Neural Information Processing Systems, 36, 2024.
15. K. Ethayarajh, Y. Choi, and S. Swayamdipta. Understanding Dataset Difficulty with V-Usable Information. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Lear… [find](https://scholar.google.com/scholar?q=K%2E+Ethayarajh%2C+Y%2E+Choi%2C+and+S%2E+Swayamdipta%2E+Understanding+Dataset+Difficulty+with+V%2DUsable+Information%2E+In+K%2E+Chaudhuri%2C+S%2E+Jegelka%2C+L%2E+Song%2C+C%2E+Szepesvari%2C+G%2E+Niu%2C+and+S%2E+Sabato%2C+editors%2C+Proceedings)
16. K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024. [find](https://scholar.google.com/scholar?q=K%2E+Ethayarajh%2C+W%2E+Xu%2C+N%2E+Muennighoff%2C+D%2E+Jurafsky%2C+and+D%2E+Kiela%2E+Kto%3A+Model+alignment+as+prospect+theoretic+optimization%2E+arXiv+preprint+arXiv%3A2402%2E01306%2C+2024%2E)
17. L. Gao, J. Schulman, and J. Hilton. [Scaling laws for reward model overoptimization](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/). In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023.
18. X. Geng. EasyLM: A Simple And Scalable Training Framework for Large Language Models, 2023. URL [https://github.com/young-geng/EasyLM](https://github.com/young-geng/EasyLM). [↗](https://github.com/young-geng/EasyLM)
19. T. Hartvigsen, S. Gabriel, H. Palangi, M. Sap, D. Ray, and E. Kamar. TOXIGEN: Controlling Language Models to Generate Implied and Adversarial Toxicity. In ACL, 2022. [URL https://arxiv.org/abs/2203.09509](https://md.research.elamur.ai/free/en/papers/hartvigsen-2022-toxigen-large-scale-machine-generated/). [↗](https://arxiv.org/abs/2203.09509)
20. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. [Measuring Massive Multitask Language Understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations (ICLR), 2020.
21. J. Hong, N. Lee, and J. Thorne. ORPO: Monolithic Preference Optimization without Reference Model, 2024. [find](https://scholar.google.com/scholar?q=J%2E+Hong%2C+N%2E+Lee%2C+and+J%2E+Thorne%2E+ORPO%3A+Monolithic+Preference+Optimization+without+Reference+Model%2C+2024%2E)
22. H. Ivison, Y. Wang, V. Pyatkin, N. Lambert, M. Peters, P. Dasigi, J. Jang, D. Wadden, N. A. Smith, I. Beltagy, and H. Hajishirzi. [Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2](https://md.research.elamur.ai/free/en/papers/ivison-2023-camels-changing-climate-enhancing-lm/), 2023.
23. H. Ivison, Y. Wang, V. Pyatkin, J. Liu, J. Lu, and Z. Wu. EasyLM-Fork: A Simple And Scalable Training Framework for Large Language Models, 2023. URL [https://github.com/hamishivi/EasyLM](https://github.com/hamishivi/EasyLM). [↗](https://github.com/hamishivi/EasyLM)
24. A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M.-A. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao, T. Gervet, T. Lavril, T. Wang, T. L… [find](https://scholar.google.com/scholar?q=A%2E+Q%2E+Jiang%2C+A%2E+Sablayrolles%2C+A%2E+Roux%2C+A%2E+Mensch%2C+B%2E+Savary%2C+C%2E+Bamford%2C+D%2E+S%2E+Chaplot%2C+D%2E+de+las+Casas%2C+E%2E+B%2E+Hanna%2C+F%2E+Bressand%2C+G%2E+Lengyel%2C+G%2E+Bour%2C+G%2E+Lample%2C+L%2E+R%2E+Lavaud%2C+L%2E+Saulnier%2C+M%2E%2DA%2E+Lach)
25. N. Lambert, L. Tunstall, N. Rajani, and T. Thrush. HuggingFace H4 Stack Exchange Preference Dataset, 2023. URL [https://huggingface.co/datasets/HuggingFaceH4/](https://huggingface.co/datasets/HuggingFaceH4/) stack-exchange-preferences. [↗](https://huggingface.co/datasets/HuggingFaceH4/)
26. N. Lambert, V. Pyatkin, J. Morrison, L. Miranda, B. Y. Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y. Choi, N. A. Smith, and H. Hajishirzi. RewardBench: Evaluating Reward Models for Language Modeling, 2024. [find](https://scholar.google.com/scholar?q=N%2E+Lambert%2C+V%2E+Pyatkin%2C+J%2E+Morrison%2C+L%2E+Miranda%2C+B%2E+Y%2E+Lin%2C+K%2E+Chandu%2C+N%2E+Dziri%2C+S%2E+Kumar%2C+T%2E+Zick%2C+Y%2E+Choi%2C+N%2E+A%2E+Smith%2C+and+H%2E+Hajishirzi%2E+RewardBench%3A+Evaluating+Reward+Models+for+Language+Modeling)
27. X. Li, T. Zhang, Y. Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto. AlpacaEval: An Automatic Evaluator of Instruction-following Models. Github repository, 2023. URL [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval). [↗](https://github.com/tatsu-lab/alpaca_eval)
28. H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s Verify Step by Step, 2023. — [Let's Verify Step by Step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/)
29. S. Lin, J. Hilton, and O. Evans. [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, 2022.
30. J. Liu, S. Hallinan, X. Lu, P. He, S. Welleck, H. Hajishirzi, and Y. Choi. [Rainier: Reinforced knowledge introspector for commonsense question answering](https://md.research.elamur.ai/free/en/papers/liu-2022-rainier-reinforced-knowledge-introspector-commonsense/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8938–8958, Abu Dhabi, United Arab Emirates, Dec.…
31. J. Liu, R. Pasunuru, H. Hajishirzi, Y. Choi, and A. Celikyilmaz. Crystal: Introspective Reasoners Reinforced with Self-Feedback. In Conference on Empirical Methods in Natural Language Processing, 2023. URL [https://api.semanticscholar.org/CorpusID:263830316](https://api.semanticscholar.org/CorpusID:263830316). [↗](https://api.semanticscholar.org/CorpusID:263830316)
32. J. Liu, C. S. Xia, Y. Wang, and L. Zhang. [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/). In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL [https://openreview.net/forum?id=1qvx610Cu7](https://openreview.net/forum?id=1qvx610Cu7). [↗](https://openreview.net/forum?id=1qvx610Cu7)
33. X. Lu, S. Welleck, L. Jiang, J. Hessel, L. Qin, P. West, P. Ammanabrolu, and Y. Choi. [Quark: Controllable Text Generation with Reinforced Unlearning](https://md.research.elamur.ai/free/en/papers/lu-2022-quark-controllable-text-generation-reinforced/). ArXiv, abs/2205.13636, 2022. URL [https://api.semanticscholar.org/CorpusID:249152301](https://api.semanticscholar.org/CorpusID:249152301). [↗](https://api.semanticscholar.org/CorpusID:249152301)
34. K. Lv, W. Zhang, and H. Shen. Supervised Fine-Tuning and Direct Preference Optimization on Intel Gaudi2, 2023. URL [https://medium.com/intel-analytics-software/](https://medium.com/intel-analytics-software/) a1197d8a3cd3. [↗](https://medium.com/intel-analytics-software/)
35. Meta. Introducing Meta Llama 3: The most capable openly available LLM to date, May 2024. URL [https://ai.meta.com/blog/meta-llama-3/](https://ai.meta.com/blog/meta-llama-3/). [↗](https://ai.meta.com/blog/meta-llama-3/)
36. N. Muennighoff, Q. Liu, A. Zebaze, Q. Zheng, B. Hui, T. Y. Zhuo, S. Singh, X. Tang, L. von Werra, and S. Longpre. OctoPack: Instruction Tuning Code Large Language Models. arXiv preprint arXiv:2308.07124, 2023. [find](https://scholar.google.com/scholar?q=N%2E+Muennighoff%2C+Q%2E+Liu%2C+A%2E+Zebaze%2C+Q%2E+Zheng%2C+B%2E+Hui%2C+T%2E+Y%2E+Zhuo%2C+S%2E+Singh%2C+X%2E+Tang%2C+L%2E+von+Werra%2C+and+S%2E+Longpre%2E+OctoPack%3A+Instruction+Tuning+Code+Large+Language+Models%2E+arXiv+preprint+arXiv%3A2308%2E071)
37. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. E. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. J. Lowe. Training language models to follow instructions with human fee… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
38. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. [Training Language Models to Follow Instructions with Human Feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). In Advances in Neural Information Processing Systems (NeurIPS), 2022.
39. R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). arXiv preprint arXiv:2305.18290, 2023.
40. A. Ramé, N. Vieillard, L. Hussenot, R. Dadashi, G. Cideron, O. Bachem, and J. Ferret. WARM: On the Benefits of Weight Averaged Reward Models, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Ram%C3%A9%2C+N%2E+Vieillard%2C+L%2E+Hussenot%2C+R%2E+Dadashi%2C+G%2E+Cideron%2C+O%2E+Bachem%2C+and+J%2E+Ferret%2E+WARM%3A+On+the+Benefits+of+Weight+Averaged+Reward+Models%2C+2024%2E)
41. K. Rasul, E. Beeching, L. Tunstall, L. von Werra, and O. Sanseviero. Preference Tuning LLMs with Direct Preference Optimization Methods. [https://huggingface.co/blog/](https://huggingface.co/blog/) pref-tuning, 2024. Accessed: 2024-05-17. [↗](https://huggingface.co/blog/)
42. P. Röttger, H. R. Kirk, B. Vidgen, G. Attanasio, F. Bianchi, and D. Hovy. [XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models](https://md.research.elamur.ai/free/en/papers/rottger-2023-xstest-test-suite-identifying-exaggerated/), 2024.
43. A. Saeidi, S. Verma, and C. Baral. Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks. 2024. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 269303161. [↗](https://api.semanticscholar.org/CorpusID:)
44. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=J%2E+Schulman%2C+F%2E+Wolski%2C+P%2E+Dhariwal%2C+A%2E+Radford%2C+and+O%2E+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)
45. J. Schulman, B. Zoph, C. Kim, and more. ChatGPT: Optimizing Language Models for Dialogue. [https://openai.com/blog/chatgpt/](https://openai.com/blog/chatgpt/), 2022. Accessed: 2023-02-12. [↗](https://openai.com/blog/chatgpt/)
46. R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In S. Solla, T. Leen, and K. Müller, editors, Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999. URL [https://proceedings.neurips.cc/paper…](https://proceedings.neurips.cc/paper…) [↗](https://proceedings.neurips.cc/paper…)
47. M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. [Challenging BIG-Bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). arXiv preprint arXiv:2210.09261, 2022.
48. F. Tajwar, A. Singh, A. Sharma, R. Rafailov, J. Schneider, T. Xie, S. Ermon, C. Finn, and A. Kumar. Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data, 2024. [find](https://scholar.google.com/scholar?q=F%2E+Tajwar%2C+A%2E+Singh%2C+A%2E+Sharma%2C+R%2E+Rafailov%2C+J%2E+Schneider%2C+T%2E+Xie%2C+S%2E+Ermon%2C+C%2E+Finn%2C+and+A%2E+Kumar%2E+Preference+Fine%2DTuning+of+LLMs+Should+Leverage+Suboptimal%2C+On%2DPolicy+Data%2C+2024%2E)
49. Y. Tang, D. Z. Guo, Z. Zheng, D. Calandriello, Y. Cao, E. Tarassov, R. Munos, B. Ávila Pires, M. Valko, Y. Cheng, and W. Dabney. Understanding the performance gap between online and offline alignment algorithms, 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Tang%2C+D%2E+Z%2E+Guo%2C+Z%2E+Zheng%2C+D%2E+Calandriello%2C+Y%2E+Cao%2C+E%2E+Tarassov%2C+R%2E+Munos%2C+B%2E+%C3%81vila+Pires%2C+M%2E+Valko%2C+Y%2E+Cheng%2C+and+W%2E+Dabney%2E+Understanding+the+performance+gap+between+online+and+offline+alignment+)
50. G. Team. Gemma: Open Models Based on Gemini Research and Technology, 2024. [find](https://scholar.google.com/scholar?q=G%2E+Team%2E+Gemma%3A+Open+Models+Based+on+Gemini+Research+and+Technology%2C+2024%2E)
51. "Teknium". Nous-hermes-llama2-13b. URL [https://huggingface.co/NousResearch/ Nous-Hermes-Llama2-13b][NousResearch/Nous-Hermes-Llama2-13b](https://huggingface.co/NousResearch/Nous-Hermes-Llama2-13b)). [find](https://scholar.google.com/scholar?q=%22Teknium%22%2E+Nous%2Dhermes%2Dllama2%2D13b%2E+URL+%5Bhttps%3A%2F%2Fhuggingface%2Eco%2FNousResearch%2F+Nous%2DHermes%2DLlama2%2D13b%5D%5BNousResearch%2FNous%2DHermes%2DLlama2%2D13b%5D%28https%3A%2F%2Fhuggingface%2Eco%2FNousResearch%2FNous%2DHermes%2DLlama2%2D13b%29%29%2E)
52. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
53. L. Tunstall, E. Beeching, N. Lambert, N. Rajani, K. Rasul, Y. Belkada, S. Huang, L. von Werra, C. Fourrier, N. Habib, et al. Zephyr: Direct Distillation of LM Alignment. arXiv preprint arXiv:2310.16944, 2023. [find](https://scholar.google.com/scholar?q=L%2E+Tunstall%2C+E%2E+Beeching%2C+N%2E+Lambert%2C+N%2E+Rajani%2C+K%2E+Rasul%2C+Y%2E+Belkada%2C+S%2E+Huang%2C+L%2E+von+Werra%2C+C%2E+Fourrier%2C+N%2E+Habib%2C+et+al%2E+Zephyr%3A+Direct+Distillation+of+LM+Alignment%2E+arXiv+preprint+arXiv%3A2310%2E1694)
54. P. Wang, L. Li, Z. Shao, R. X. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui. [Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations](https://md.research.elamur.ai/free/en/papers/wang-2024-math-shepherd-verify-reinforce-llms/), 2024.
55. Y. Wang, H. Ivison, P. Dasigi, J. Hessel, T. Khot, K. R. Chandu, D. Wadden, K. MacMillan, N. A. Smith, I. Beltagy, et al. [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://md.research.elamur.ai/free/en/papers/wang-2023-far-camels-go-exploring-state/). arXiv preprint arXiv:2306.04751, 2023.
56. Z. Wang, Y. Dong, J. Zeng, V. Adams, M. N. Sreedhar, D. Egert, O. Delalleau, J. P. Scowcroft, N. Kant, A. Swope, and O. Kuchaiev. [HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM](https://md.research.elamur.ai/free/en/papers/wang-2024-helpsteer-multi-attribute-helpfulness-dataset/), 2023.
57. J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.[11903, 2022](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/).
58. Z. Wu, Y. Hu, W. Shi, N. Dziri, A. Suhr, P. Ammanabrolu, N. A. Smith, M. Ostendorf, and H. Hajishirzi. [Fine-Grained Human Feedback Gives Better Rewards for Language Model Training](https://md.research.elamur.ai/free/en/papers/wu-2023-fine-grained-human-feedback-gives/). ArXiv, abs/2306.01693, 2023. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 259064099. [↗](https://api.semanticscholar.org/CorpusID:)
59. H. Xu, A. Sharaf, Y. Chen, W. Tan, L. Shen, B. Van Durme, K. Murray, and Y. J. Kim. [Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation](https://md.research.elamur.ai/free/en/papers/xu-2024-contrastive-preference-optimization-pushing-boundaries/). arXiv preprint arXiv:2401.08417, 2024.
60. S. Xu, W. Fu, J. Gao, W. Ye, W. Liu, Z. Mei, G. Wang, C. Yu, and Y. Wu. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study, 2024. [find](https://scholar.google.com/scholar?q=S%2E+Xu%2C+W%2E+Fu%2C+J%2E+Gao%2C+W%2E+Ye%2C+W%2E+Liu%2C+Z%2E+Mei%2C+G%2E+Wang%2C+C%2E+Yu%2C+and+Y%2E+Wu%2E+Is+DPO+Superior+to+PPO+for+LLM+Alignment%3F+A+Comprehensive+Study%2C+2024%2E)
61. L. Yuan, G. Cui, H. Wang, N. Ding, X. Wang, J. Deng, B. Shan, H. Chen, R. Xie, Y. Lin, Z. Liu, B. Zhou, H. Peng, Z. Liu, and M. Sun. Advancing llm reasoning generalists with preference trees, 2024. [find](https://scholar.google.com/scholar?q=L%2E+Yuan%2C+G%2E+Cui%2C+H%2E+Wang%2C+N%2E+Ding%2C+X%2E+Wang%2C+J%2E+Deng%2C+B%2E+Shan%2C+H%2E+Chen%2C+R%2E+Xie%2C+Y%2E+Lin%2C+Z%2E+Liu%2C+B%2E+Zhou%2C+H%2E+Peng%2C+Z%2E+Liu%2C+and+M%2E+Sun%2E+Advancing+llm+reasoning+generalists+with+preference+trees%2C+2024%2E)
62. W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng. (InThe)WildChat: 570K ChatGPT Interaction Logs In The Wild. In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=Bl8u7ZRlbM](https://openreview.net/forum?id=Bl8u7ZRlbM). [↗](https://openreview.net/forum?id=Bl8u7ZRlbM)
63. L. Zheng, W.-L. Chiang, Y. Sheng, T. Li, S. Zhuang, Z. Wu, Y. Zhuang, Z. Li, Z. Lin, E. P. Xing, J. E. Gonzalez, I. Stoica, and H. Zhang. [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://md.research.elamur.ai/free/en/papers/zheng-2023-lmsys-chat-1m-large-scale/), 2023.
64. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. [Judging LLM-as-a-judge with MT-Bench and Chatbot Arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In NeurIPS Datasets and Benchmarks Track, 2023.
65. J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou. [Instruction-Following Evaluation for Large Language Models](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/), 2023.
66. B. Zhu, E. Frick, T. Wu, H. Zhu, and J. Jiao. Starling-7B: Improving LLM Helpfulness & Harmlessness with RLAIF, November 2023. [find](https://scholar.google.com/scholar?q=B%2E+Zhu%2C+E%2E+Frick%2C+T%2E+Wu%2C+H%2E+Zhu%2C+and+J%2E+Jiao%2E+Starling%2D7B%3A+Improving+LLM+Helpfulness+%26+Harmlessness+with+RLAIF%2C+November+2023%2E)
67. D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. [find](https://scholar.google.com/scholar?q=D%2E+M%2E+Ziegler%2C+N%2E+Stiennon%2C+J%2E+Wu%2C+T%2E+B%2E+Brown%2C+A%2E+Radford%2C+D%2E+Amodei%2C+P%2E+Christiano%2C+and+G%2E+Irving%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint+arXiv%3A1909%2E08593%2C+2019%2E)

*Updated 29 October 2026*