---
title: "A General Theoretical Paradigm to Understand — Azar 2023"
description: "This paper by researchers at Google DeepMind provides a theoretical foundation for methods that train language models from human preferences."
authors: ["Mohammad Gheshlaghi Azar", "Mark Rowland", "Bilal Piot", "Daniel Guo", "Daniele Calandriello", "Michal Valko", "Rémi Munos"]
year: 2023
venue: "arXiv:2310.12036"
source: "https://arxiv.org/abs/2310.12036"
doi: "10.48550/arxiv.2310.12036"
arxiv: "2310.12036"
licence: "CC BY-NC-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/azar-2023-general-theoretical-paradigm-understand-learning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/azar-2023-general-theoretical-paradigm-understand-learning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/) / A General Theoretical Paradigm to Understand Learning from Human Preferences

# A General Theoretical Paradigm to Understand Learning from Human Preferences

**Authors:** Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, Rémi Munos · **Year:** 2023 · **Venue:** arXiv:2310.12036 · **Licence:** [CC BY-NC-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.12036) · [PDF](https://arxiv.org/pdf/2310.12036) · [DOI 10.48550/arxiv.2310.12036](https://doi.org/10.48550/arxiv.2310.12036) · [arXiv 2310.12036](https://arxiv.org/abs/2310.12036) · [OpenAlex](https://openalex.org/W4387801352)

**Citations:** 14 citations (openalex-snapshot · 2026-10-14) · [236](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
This paper by researchers at Google DeepMind provides a theoretical foundation for methods that train language models from human preferences. The authors show that RLHF and DPO rest on a shared assumption: a pairwise comparison of two responses can be replaced by a pointwise reward under the Bradley-Terry model. To get rid of this assumption, they propose a general objective, ΨPO. It depends only on pairwise preference probabilities and includes KL regularization that keeps the policy close to the reference policy. RLHF and DPO turn out to be special cases of ΨPO. The analysis shows that both methods can overfit: when the preferences in the data are deterministic or nearly deterministic, the regularization effectively stops working. As a solution, the authors take Ψ to be the identity mapping and obtain the IPO method. For it, they derive a simple empirical loss function that is optimized on samples of comparisons, and they prove guarantees. On small synthetic examples, IPO behaves stably where DPO produces degenerate policies. The paper explains why DPO can ignore the reference model and offers a simple alternative. However, IPO has not yet been tested on real language models.

## Key points
- **ΨPO** is a general objective with a non-decreasing function Ψ. If Ψ(q) = log(q/(1−q)) and the Bradley-Terry model holds, the optimal policies of ΨPO, RLHF and DPO coincide (Proposition 1).
- **IPO (Identity-PO)**: Ψ is the identity mapping. The problem is reduced to finding the root of an equation, and an empirical loss function is derived from this. It requires neither a reward model nor the Bradley-Terry assumption.
- **Experiment 1** has 3 actions and a dataset D₁ with a total order. DPO converges to a deterministic policy for any τ. IPO stays close to π_ref under strong regularization.
- **Experiment 2** uses the dataset D₃ = {(y_a, y_b), (y_b, y_a)}, where action y_c is never compared. DPO drives its probability to 0 regardless of τ. IPO lowers it gradually, depending on the regularization strength.
- **Experimental setup**: softmax parameterization, Adam (lr 0.01), 18000 steps, mini-batch 9, 10 seeds, 95% confidence intervals, implementation in flax and optax. The authors leave evaluation on language models for future work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cited by

Cited by

236 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 229 not yet.

- [Fanar: An Arabic-Centric Multimodal Generative AI Platform](https://md.research.elamur.ai/free/en/papers/abbas-2025-fanar-arabic-centric-multimodal-generative/)
- [ALLaM: Large Language Models for Arabic and English](https://md.research.elamur.ai/free/en/papers/bari-2024-allam-large-language-models-arabic/)
- [Process Reinforcement through Implicit Rewards](https://md.research.elamur.ai/free/en/papers/cui-2025-process-reinforcement-through-implicit-rewards/)
- [Orca-Math: Unlocking the potential of SLMs in Grade School Math](https://md.research.elamur.ai/free/en/papers/mitra-2024-orca-math-unlocking-potential-slms/)
- [ReFT: Reasoning with Reinforced Fine-Tuning](https://md.research.elamur.ai/free/en/papers/trung-2024-reft-reasoning-reinforced-fine-tuning/)
- [Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts](https://md.research.elamur.ai/free/en/papers/wang-2024-interpretable-preferences-multi-objective-reward/)
- [Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing](https://md.research.elamur.ai/free/en/papers/xu-2024-magpie-alignment-data-synthesis-scratch/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{azar2023general,
  title = {{A General Theoretical Paradigm to Understand Learning from Human Preferences}},
  author = {Azar, Mohammad Gheshlaghi and Rowland, Mark and Piot, Bilal and Guo, Daniel and Calandriello, Daniele and Valko, Michal and Munos, Rémi},
  year = {2023},
  doi = {10.48550/arxiv.2310.12036},
  eprint = {2310.12036},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.12036}
}
```

- [← previous A fine-grained comparison of pragmatic language understanding in humans and language models](https://md.research.elamur.ai/free/en/papers/hu-2023-fine-grained-comparison-pragmatic-language/)
- [next → A Massive Scale Semantic Similarity Dataset of Historical English](https://md.research.elamur.ai/free/en/papers/silcock-2023-massive-scale-semantic-similarity-dataset/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/)

## References

27 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, et al. The DeepMind JAX ecosystem, 2020, 2020. URL [http://github.com/deepmind](http://github.com/deepmind). [↗](http://github.com/deepmind)
2. Quentin Bertrand, Wojciech Marian Czarnecki, and Gauthier Gidel. On the limitations of the Elo: Real-world games are transitive, not additive. In Proceedings of the International Conference on Artificial Intelligence and Statistics, 2023. [find](https://scholar.google.com/scholar?q=Quentin+Bertrand%2C+Wojciech+Marian+Czarnecki%2C+and+Gauthier+Gidel%2E+On+the+limitations+of+the+Elo%3A+Real%2Dworld+games+are+transitive%2C+not+additive%2E+In+Proceedings+of+the+International+Conference+on+Artific)
3. Stephen P. Boyd and Lieven Vandenberghe. Convex optimization. Cambridge University Press, 2004. [find](https://scholar.google.com/scholar?q=Stephen+P%2E+Boyd+and+Lieven+Vandenberghe%2E+Convex+optimization%2E+Cambridge+University+Press%2C+2004%2E)
4. James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL [http://github.com/google/jax](http://github.com/google/jax). [↗](http://github.com/google/jax)
5. Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. [find](https://scholar.google.com/scholar?q=Ralph+Allan+Bradley+and+Milton+E+Terry%2E+Rank+analysis+of+incomplete+block+designs%3A+I%2E+The+method+of+paired+comparisons%2E+Biometrika%2C+39%283%2F4%29%3A324%E2%80%93345%2C+1952%2E)
6. Róbert Busa-Fekete, Balázs Szörényi, Paul Weng, Weiwei Cheng, and Eyke Hüllermeier. Preference-based reinforcement learning: Evolutionary direct policy search using a preference-based racing algorithm. Machine Learning, (3):327–351, 2014. [find](https://scholar.google.com/scholar?q=Ro%CC%81bert+Busa%2DFekete%2C+Bala%CC%81zs+Szo%CC%88re%CC%81nyi%2C+Paul+Weng%2C+Weiwei+Cheng%2C+and+Eyke+Hu%CC%88llermeier%2E+Preference%2Dbased+reinforcement+learning%3A+Evolutionary+direct+policy+search+using+a+preference%2Dbased+racing+algo)
7. Róbert Busa-Fekete, Balázs Szörenyi, Paul Weng, Weiwei Cheng, and Eyke Hüllermeier. Preference-based evolutionary direct policy search. In Autonomous Learning Workshop @ ICRA, 2013. [find](https://scholar.google.com/scholar?q=Ro%CC%81bert+Busa%2DFekete%2C+Bala%CC%81zs+Szo%CC%88renyi%2C+Paul+Weng%2C+Weiwei+Cheng%2C+and+Eyke+Hu%CC%88llermeier%2E+Preference%2Dbased+evolutionary+direct+policy+search%2E+In+Autonomous+Learning+Workshop+%40+ICRA%2C+2013%2E)
8. Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang, and Liwei Wang. Human-in-the-loop: Provably efficient preference-based reinforcement learning with general function approximation. In Proceedings of the International Conference on Machine Learning, 2022. [find](https://scholar.google.com/scholar?q=Xiaoyu+Chen%2C+Han+Zhong%2C+Zhuoran+Yang%2C+Zhaoran+Wang%2C+and+Liwei+Wang%2E+Human%2Din%2Dthe%2Dloop%3A+Provably+efficient+preference%2Dbased+reinforcement+learning+with+general+function+approximation%2E+In+Proceedings+of)
9. Paul F. Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, 2017. [find](https://scholar.google.com/scholar?q=Paul+F%2E+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2017%2E)
10. Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized Markov decision processes. In Proceedings of the International Conference on Machine Learning, 2019. [find](https://scholar.google.com/scholar?q=Matthieu+Geist%2C+Bruno+Scherrer%2C+and+Olivier+Pietquin%2E+A+theory+of+regularized+Markov+decision+processes%2E+In+Proceedings+of+the+International+Conference+on+Machine+Learning%2C+2019%2E)
11. Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for JAX, 2023. URL [http://github.com/google/flax](http://github.com/google/flax). [↗](http://github.com/google/flax)
12. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations, 2014. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+Proceedings+of+the+International+Conference+on+Learning+Representations%2C+2014%2E)
13. Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agent communication meets natural language: Synergies between functional and structural language learning. In Proceedings of the Annual Meeting of Association for Computational Linguistics, 2020. [find](https://scholar.google.com/scholar?q=Angeliki+Lazaridou%2C+Anna+Potapenko%2C+and+Olivier+Tieleman%2E+Multi%2Dagent+communication+meets+natural+language%3A+Synergies+between+functional+and+structural+language+learning%2E+In+Proceedings+of+the+Annual+)
14. Tyler Lu, Dávid Pál, and Martin Pál. Contextual multi-armed bandits. In Proceedings of the International Conference on Artificial Intelligence and Statistics, 2010. [find](https://scholar.google.com/scholar?q=Tyler+Lu%2C+Da%CC%81vid+Pa%CC%81l%2C+and+Martin+Pa%CC%81l%2E+Contextual+multi%2Darmed+bandits%2E+In+Proceedings+of+the+International+Conference+on+Artificial+Intelligence+and+Statistics%2C+2010%2E)
15. Yuchen Lu, Soumye Singhal, Florian Strub, Aaron Courville, and Olivier Pietquin. Countering language drift with seeded iterated learning. In Proceedings of the International Conference on Machine Learning, 2020. [find](https://scholar.google.com/scholar?q=Yuchen+Lu%2C+Soumye+Singhal%2C+Florian+Strub%2C+Aaron+Courville%2C+and+Olivier+Pietquin%2E+Countering+language+drift+with+seeded+iterated+learning%2E+In+Proceedings+of+the+International+Conference+on+Machine+Lear)
16. Ellen Novoseller, Yibing Wei, Yanan Sui, Yisong Yue, and Joel Burdick. Dueling posterior sampling for preference-based reinforcement learning. In Proceedings of the Conference on Uncertainty in Artificial Intelligence, 2020. [find](https://scholar.google.com/scholar?q=Ellen+Novoseller%2C+Yibing+Wei%2C+Yanan+Sui%2C+Yisong+Yue%2C+and+Joel+Burdick%2E+Dueling+posterior+sampling+for+preference%2Dbased+reinforcement+learning%2E+In+Proceedings+of+the+Conference+on+Uncertainty+in+Artifi)
17. OpenAI. Gpt-4 technical report, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
18. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller amd Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training… [find](https://scholar.google.com/scholar?q=Long+Ouyang%2C+Jeffrey+Wu%2C+Xu+Jiang%2C+Diogo+Almeida%2C+Carroll+Wainwright%2C+Pamela+Mishkin%2C+Chong+Zhang%2C+Sandhini+Agarwal%2C+Katarina+Slama%2C+Alex+Ray%2C+John+Schulman%2C+Jacob+Hilton%2C+Fraser+Kelton%2C+Luke+Miller+a)
19. Aldo Pacchiano, Aadirupa Saha, and Jonathan Lee. Dueling RL: Reinforcement learning with trajectory preferences. arXiv, 2023. [find](https://scholar.google.com/scholar?q=Aldo+Pacchiano%2C+Aadirupa+Saha%2C+and+Jonathan+Lee%2E+Dueling+RL%3A+Reinforcement+learning+with+trajectory+preferences%2E+arXiv%2C+2023%2E)
20. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+2018%2E)
21. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv, 2023. [find](https://scholar.google.com/scholar?q=Rafael+Rafailov%2C+Archit+Sharma%2C+Eric+Mitchell%2C+Stefano+Ermon%2C+Christopher+D%2E+Manning%2C+and+Chelsea+Finn%2E+Direct+preference+optimization%3A+Your+language+model+is+secretly+a+reward+model%2E+arXiv%2C+2023%2E)
22. Prajit Ramachandran, Peter J. Liu, and Quoc V. Le. Unsupervised pretraining for sequence to sequence learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processings, 2016. [find](https://scholar.google.com/scholar?q=Prajit+Ramachandran%2C+Peter+J%2E+Liu%2C+and+Quoc+V%2E+Le%2E+Unsupervised+pretraining+for+sequence+to+sequence+learning%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Processings%2C+20)
23. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv, 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv%2C+2017%2E)
24. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 2020. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F%2E+Christiano%2E+Learning+to+summarize+with+human+feedback%2E+Advances+in+Neural+Info)
25. Yuanhao Wang, Qinghua Liu, and Chi Jin. Is RLHF more difficult than standard RL? arXiv, 2023. [find](https://scholar.google.com/scholar?q=Yuanhao+Wang%2C+Qinghua+Liu%2C+and+Chi+Jin%2E+Is+RLHF+more+difficult+than+standard+RL%3F+arXiv%2C+2023%2E)
26. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In Proceedings of the International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Bosma%2C+Vincent+Y%2E+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+M%2E+Dai%2C+and+Quoc+V%2E+Le%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+In+Proceedings+of+the+Intern)
27. Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. SLiC-HF: Sequence likelihood calibration with human feedback. arXiv, 2023. [find](https://scholar.google.com/scholar?q=Yao+Zhao%2C+Rishabh+Joshi%2C+Tianqi+Liu%2C+Misha+Khalman%2C+Mohammad+Saleh%2C+and+Peter+J+Liu%2E+SLiC%2DHF%3A+Sequence+likelihood+calibration+with+human+feedback%2E+arXiv%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2023-slic-hf-sequence-likelihood-calibration/)