---
title: "From r to Q*: Your Language Model is Secretly a Q-Function"
description: "Direct Preference Optimization (DPO) is normally derived as a contextual bandit, where a whole… 69 references, cited by 71 papers here."
authors: ["Rafael Rafailov", "Joey Hejna", "Ryan Park", "Chelsea Finn"]
year: 2024
venue: "arXiv:2404.12358"
source: "https://arxiv.org/abs/2404.12358"
doi: "10.48550/arxiv.2404.12358"
arxiv: "2404.12358"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rafailov-2024-r-q-language-model-secretly/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rafailov-2024-r-q-language-model-secretly/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / From r to Q\*: Your Language Model is Secretly a Q-Function

# From r to Q\*: Your Language Model is Secretly a Q-Function

**Authors:** [Rafael Rafailov](https://md.research.elamur.ai/free/en/authors/rafael-rafailov/), Joey Hejna, [Ryan Park](https://md.research.elamur.ai/free/en/authors/ryan-park/) and [Chelsea Finn](https://md.research.elamur.ai/free/en/authors/chelsea-finn/) · **Year:** 2024 · **Venue:** arXiv:2404.12358 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2404.12358) · [PDF](https://arxiv.org/pdf/2404.12358) · [DOI 10.48550/arxiv.2404.12358](https://doi.org/10.48550/arxiv.2404.12358) · [arXiv 2404.12358](https://arxiv.org/abs/2404.12358) · [OpenAlex](https://openalex.org/W4394972204)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [71](https://md.research.elamur.ai/free/en/papers/rafailov-2024-r-q-language-model-secretly/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Direct Preference Optimization (DPO) is normally derived as a contextual bandit, where a whole response counts as one action. Classical RLHF with PPO, by contrast, optimizes value functions token by token. The authors close this gap by re-deriving DPO inside a token-level MDP. In this MDP, states are the prompt plus the tokens generated so far, actions are vocabulary tokens, and transitions simply append the chosen token. Preferences follow a multi-step Bradley-Terry model. Drawing on maximum-entropy and inverse RL, they show that DPO behaves as an inverse Q-learning method. The language model's logits can be read as an optimal Q-function that satisfies the Bellman equation, and DPO can express any dense per-token reward in this setting.

The theory yields three practical lessons:
- DPO's implicit rewards perform a kind of token-level credit assignment.
- Reward-guided search methods such as MCTS reduce to likelihood search over a DPO policy, so ordinary beam search already improves DPO outputs.
- The choice of reference policy explains why implicit rewards fall during training.

This matters because it supports applying direct alignment to sequential tasks: multi-turn dialogue, agents and multi-model pipelines.

## Key points
- **Theory:** DPO is cast as learning an optimal Q-function in the token-level MDP. Its implied per-token rewards belong to the family of optimal advantage functions.
- **Search:** Likelihood-based decoding over a DPO model is shown to be equivalent to search on a reward function, as in Liu et al. (2023b) and Feng et al. (2024). Switching to beam search takes about a one-line code change and gives measurable gains over the base DPO policy.
- **Training dynamics:** Under the reference model, the expected implicit reward equals −β·KL(π_ref‖π). It therefore must decline when training starts from an SFT model, especially one fine-tuned only on chosen responses.
- **Experiments:** DPO was tested on TLDR and CPL (Hejna et al., 2024) on a bin-picking dataset. Without prior SFT, DPO's implicit rewards for chosen responses stay positive and CPL's rewards rise.
- **Future directions:** The authors (Rafailov, Hejna, Park, Finn; Stanford) propose learning reasoning from outcome feedback, multi-turn conversation, agentic LLMs such as WebGPT, and end-to-end tuning of LLM-plus-diffusion systems like Dalle 3.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,053 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Rafael Rafailov, Joey Hejna, Ryan Park and Chelsea Finn. 2024. From r to Q\*: Your Language Model is Secretly a Q-Function. arXiv:2404.12358.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{rafailov2024r,
  title = {{From r to Q*: Your Language Model is Secretly a Q-Function}},
  author = {Rafael Rafailov, Joey Hejna, Ryan Park and Chelsea Finn},
  year = {2024},
  doi = {10.48550/arxiv.2404.12358},
  eprint = {2404.12358},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.12358}
}
```

- [← previous Black-Box Prompt Optimization: Aligning Large Language Models without Model Training](https://md.research.elamur.ai/free/en/papers/cheng-2024-black-box-prompt-optimization-aligning/)
- [next → HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM](https://md.research.elamur.ai/free/en/papers/wang-2024-helpsteer-multi-attribute-helpfulness-dataset/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

69 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Riad Akrour, Marc Schoenauer, and Michele Sebag. Preference-based policy learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2011. [find](https://scholar.google.com/scholar?q=Riad+Akrour%2C+Marc+Schoenauer%2C+and+Michele+Sebag%2E+Preference%2Dbased+policy+learning%2E+In+Joint+European+Conference+on+Machine+Learning+and+Knowledge+Discovery+in+Databases%2C+2011%2E)
2. Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D. Goodman. Star-gate: Teaching language models to ask clarifying questions, 2024. [find](https://scholar.google.com/scholar?q=Chinmaya+Andukuri%2C+Jan%2DPhilipp+Fr%C3%A4nken%2C+Tobias+Gerstenberg%2C+and+Noah+D%2E+Goodman%2E+Star%2Dgate%3A+Teaching+language+models+to+ask+clarifying+questions%2C+2024%2E)
3. Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. [A general theoretical paradigm to understand learning from human preferences](https://md.research.elamur.ai/free/en/papers/azar-2023-general-theoretical-paradigm-understand-learning/), 2023.
4. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Saurav+Kadavath%2C+Jackson+Kernion%2C+Tom+Conerly%2C+)
5. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared… — [Constitutional AI: Harmlessness from AI Feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/)
6. James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and Aditya Ramesh. Improving image generation with better captions, 2023. URL [https://cdn.openai.com/papers/dall-e-3…](https://cdn.openai.com/papers/dall-e-3…) [↗](https://cdn.openai.com/papers/dall-e-3…)
7. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+Aviya+Skowron%2C+Lintang+Sutaw)
8. Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301, 2023a. — [arXiv:2305.13301](https://arxiv.org/abs/2305.13301) [find](https://scholar.google.com/scholar?q=Kevin+Black%2C+Michael+Janner%2C+Yilun+Du%2C+Ilya+Kostrikov%2C+and+Sergey+Levine%2E+Training+diffusion+models+with+reinforcement+learning%2E+arXiv+preprint+arXiv%3A2305%2E13301%2C+2023a%2E+%E2%80%94+%5BarXiv%3A2305%2E13301%5D%28https%3A%2F%2Far)
9. Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301, 2023b. — [arXiv:2305.13301](https://arxiv.org/abs/2305.13301) [find](https://scholar.google.com/scholar?q=Kevin+Black%2C+Michael+Janner%2C+Yilun+Du%2C+Ilya+Kostrikov%2C+and+Sergey+Levine%2E+Training+diffusion+models+with+reinforcement+learning%2E+arXiv+preprint+arXiv%3A2305%2E13301%2C+2023b%2E+%E2%80%94+%5BarXiv%3A2305%2E13301%5D%28https%3A%2F%2Far)
10. Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. doi: https://doi.org/10.2307/2334029. — [doi:10.2307/2334029](https://doi.org/10.2307/2334029) [↗](https://doi.org/10.2307/2334029)
11. Haoyang Cao, Samuel Cohen, and Lukasz Szpruch. Identifiability in inverse reinforcement learning. Advances in Neural Information Processing Systems, 34:12362–12373, 2021. [find](https://scholar.google.com/scholar?q=Haoyang+Cao%2C+Samuel+Cohen%2C+and+Lukasz+Szpruch%2E+Identifiability+in+inverse+reinforcement+learning%2E+Advances+in+Neural+Information+Processing+Systems%2C+34%3A12362%E2%80%9312373%2C+2021%2E)
12. Alex J Chan, Hao Sun, Samuel Holt, and Mihaela van der Schaar. Dense reward for free in reinforcement learning from human feedback. arXiv preprint arXiv:2402.00782, 2024. — [arXiv:2402.00782](https://arxiv.org/abs/2402.00782) [find](https://scholar.google.com/scholar?q=Alex+J+Chan%2C+Hao+Sun%2C+Samuel+Holt%2C+and+Mihaela+van+der+Schaar%2E+Dense+reward+for+free+in+reinforcement+learning+from+human+feedback%2E+arXiv+preprint+arXiv%3A2402%2E00782%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E00782%5D%28https%3A%2F%2F)
13. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 3… [find](https://scholar.google.com/scholar?q=Paul+F+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2E+In+I%2E+Guyon%2C+U%2E+Von+Luxburg%2C+S%2E+Bengio%2C+H%2E+Wallach%2C+R%2E+Fergus%2C)
14. Chris Cundy and Stefano Ermon. Sequencematch: Imitation learning for autoregressive sequence modelling with backtracking. arXiv preprint arXiv:2306.05426, 2023. [find](https://scholar.google.com/scholar?q=Chris+Cundy+and+Stefano+Ermon%2E+Sequencematch%3A+Imitation+learning+for+autoregressive+sequence+modelling+with+backtracking%2E+arXiv+preprint+arXiv%3A2306%2E05426%2C+2023%2E)
15. Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo. arXiv preprint arXiv:2005.12729, 2020. [find](https://scholar.google.com/scholar?q=Logan+Engstrom%2C+Andrew+Ilyas%2C+Shibani+Santurkar%2C+Dimitris+Tsipras%2C+Firdaus+Janoos%2C+Larry+Rudolph%2C+and+Aleksander+Madry%2E+Implementation+matters+in+deep+policy+gradients%3A+A+case+study+on+ppo+and+trpo%2E+a)
16. Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolut… [find](https://scholar.google.com/scholar?q=Patrick+Esser%2C+Sumith+Kulal%2C+Andreas+Blattmann%2C+Rahim+Entezari%2C+Jonas+M%C3%BCller%2C+Harry+Saini%2C+Yam+Levi%2C+Dominik+Lorenz%2C+Axel+Sauer%2C+Frederic+Boesel%2C+Dustin+Podell%2C+Tim+Dockhorn%2C+Zion+English%2C+Kyle+Lacey%2C)
17. Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. arXiv preprint arXiv:2305.16381, 2023. — [arXiv:2305.16381](https://arxiv.org/abs/2305.16381) [find](https://scholar.google.com/scholar?q=Ying+Fan%2C+Olivia+Watkins%2C+Yuqing+Du%2C+Hao+Liu%2C+Moonkyung+Ryu%2C+Craig+Boutilier%2C+Pieter+Abbeel%2C+Mohammad+Ghavamzadeh%2C+Kangwook+Lee%2C+and+Kimin+Lee%2E+Dpok%3A+Reinforcement+learning+for+fine%2Dtuning+text%2Dto%2Dima)
18. Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training, 2024. [find](https://scholar.google.com/scholar?q=Xidong+Feng%2C+Ziyu+Wan%2C+Muning+Wen%2C+Stephen+Marcus+McAleer%2C+Ying+Wen%2C+Weinan+Zhang%2C+and+Jun+Wang%2E+Alphazero%2Dlike+tree%2Dsearch+can+guide+large+language+model+decoding+and+training%2C+2024%2E)
19. Leo Gao, John Schulman, and Jacob Hilton. [Scaling laws for reward model overoptimization](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/). International Conference on machine Learning, 2023.
20. Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, Matthieu Geist, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation, 2022. [find](https://scholar.google.com/scholar?q=Divyansh+Garg%2C+Shuvam+Chakraborty%2C+Chris+Cundy%2C+Jiaming+Song%2C+Matthieu+Geist%2C+and+Stefano+Ermon%2E+Iq%2Dlearn%3A+Inverse+soft%2Dq+learning+for+imitation%2C+2022%2E)
21. Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models, 2023. [find](https://scholar.google.com/scholar?q=Agrim+Gupta%2C+Lijun+Yu%2C+Kihyuk+Sohn%2C+Xiuye+Gu%2C+Meera+Hahn%2C+Li+Fei%2DFei%2C+Irfan+Essa%2C+Lu+Jiang%2C+and+Jos%C3%A9+Lezama%2E+Photorealistic+video+generation+with+diffusion+models%2C+2023%2E)
22. Joey Hejna and Dorsa Sadigh. Inverse preference learning: Preference-based rl without a reward function. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Joey+Hejna+and+Dorsa+Sadigh%2E+Inverse+preference+learning%3A+Preference%2Dbased+rl+without+a+reward+function%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2C+2024%2E)
23. Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W. Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: Learning from human feedback without reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.…](https://openreview.…) [↗](https://openreview.…)
24. Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. [V-star: Training verifiers for self-taught reasoners](https://md.research.elamur.ai/free/en/papers/hosseini-2024-v-star-training-verifiers-self/), 2024.
25. Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving, 2023. [find](https://scholar.google.com/scholar?q=Anthony+Hu%2C+Lloyd+Russell%2C+Hudson+Yeo%2C+Zak+Murez%2C+George+Fedoseev%2C+Alex+Kendall%2C+Jamie+Shotton%2C+and+Gianluca+Corrado%2E+Gaia%2D1%3A+A+generative+world+model+for+autonomous+driving%2C+2023%2E)
26. James Y. Huang, Sailik Sengupta, Daniele Bonadiman, Yi an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchhoff, and Dan Roth. [Deal: Decoding-time alignment for large language models](https://md.research.elamur.ai/free/en/papers/huang-2025-deal-decoding-time-alignment-large/), 2024.
27. Hyeonbin Hwang, Doyoung Kim, Seungone Kim, Seonghyeon Ye, and Minjoon Seo. Self-explore to avoid the pit: Improving the reasoning capabilities of language models with fine-grained rewards, 2024. URL https://arxiv.org/abs/2404.10346. — [arXiv:2404.10346](https://arxiv.org/abs/2404.10346) [↗](https://arxiv.org/abs/2404.10346)
28. Minbeom Kim, Hwanhee Lee, Kang Min Yoo, Joonsuk Park, Hwaran Lee, and Kyomin Jung. Critic-guided decoding for controlled text generation. arXiv preprint arXiv:2212.10938, 2022. [find](https://scholar.google.com/scholar?q=Minbeom+Kim%2C+Hwanhee+Lee%2C+Kang+Min+Yoo%2C+Joonsuk+Park%2C+Hwaran+Lee%2C+and+Kyomin+Jung%2E+Critic%2Dguided+decoding+for+controlled+text+generation%2E+arXiv+preprint+arXiv%3A2212%2E10938%2C+2022%2E)
29. W Bradley Knox, Stephane Hatgis-Kessell, Serena Booth, Scott Niekum, Peter Stone, and Alessandro G Allievi. Models of human preference for learning reward functions. Transactions on Machine Learning Research, 2023. [find](https://scholar.google.com/scholar?q=W+Bradley+Knox%2C+Stephane+Hatgis%2DKessell%2C+Serena+Booth%2C+Scott+Niekum%2C+Peter+Stone%2C+and+Alessandro+G+Allievi%2E+Models+of+human+preference+for+learning+reward+functions%2E+Transactions+on+Machine+Learning+R)
30. W Bradley Knox, Stephane Hatgis-Kessell, Sigurdur Orn Adalgeirsson, Serena Booth, Anca Dragan, Peter Stone, and Scott Niekum. Learning optimal advantage from preferences and mistaking it for reward. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 10066–10073, 2024. [find](https://scholar.google.com/scholar?q=W+Bradley+Knox%2C+Stephane+Hatgis%2DKessell%2C+Sigurdur+Orn+Adalgeirsson%2C+Serena+Booth%2C+Anca+Dragan%2C+Peter+Stone%2C+and+Scott+Niekum%2E+Learning+optimal+advantage+from+preferences+and+mistaking+it+for+reward%2E+I)
31. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024. [find](https://scholar.google.com/scholar?q=Nathan+Lambert%2C+Valentina+Pyatkin%2C+Jacob+Morrison%2C+LJ+Miranda%2C+Bill+Yuchen+Lin%2C+Khyathi+Chandu%2C+Nouha+Dziri%2C+Sachin+Kumar%2C+Tom+Zick%2C+Yejin+Choi%2C+Noah+A%2E+Smith%2C+and+Hannaneh+Hajishirzi%2E+Rewardbench%3A+Ev)
32. Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv e-prints, pp. arXiv–2302, 2023. [find](https://scholar.google.com/scholar?q=Kimin+Lee%2C+Hao+Liu%2C+Moonkyung+Ryu%2C+Olivia+Watkins%2C+Yuqing+Du%2C+Craig+Boutilier%2C+Pieter+Abbeel%2C+Mohammad+Ghavamzadeh%2C+and+Shixiang+Shane+Gu%2E+Aligning+text%2Dto%2Dimage+models+using+human+feedback%2E+arXiv+e%2Dp)
33. Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018. [find](https://scholar.google.com/scholar?q=Sergey+Levine%2E+Reinforcement+learning+and+control+as+probabilistic+inference%3A+Tutorial+and+review%2C+2018%2E)
34. Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020. [find](https://scholar.google.com/scholar?q=Sergey+Levine%2C+Aviral+Kumar%2C+George+Tucker%2C+and+Justin+Fu%2E+Offline+reinforcement+learning%3A+Tutorial%2C+review%2C+and+perspectives+on+open+problems%2C+2020%2E)
35. Jiwei Li, Will Monroe, and Dan Jurafsky. Learning to decode for future success. arXiv preprint arXiv:1701.06549, 2017. [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Will+Monroe%2C+and+Dan+Jurafsky%2E+Learning+to+decode+for+future+success%2E+arXiv+preprint+arXiv%3A1701%2E06549%2C+2017%2E)
36. Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A. Smith. [Tuning language models by proxy](https://md.research.elamur.ai/free/en/papers/liu-2024-tuning-language-models-proxy/), 2024a. URL https://arxiv.org/abs/2401.08565. [↗](https://arxiv.org/abs/2401.08565)
37. Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, and Asli Celikyilmaz. Don’t throw away your value model! making ppo even better via value-guided monte-carlo tree search decoding, 2023a. [find](https://scholar.google.com/scholar?q=Jiacheng+Liu%2C+Andrew+Cohen%2C+Ramakanth+Pasunuru%2C+Yejin+Choi%2C+Hannaneh+Hajishirzi%2C+and+Asli+Celikyilmaz%2E+Don%E2%80%99t+throw+away+your+value+model%21+making+ppo+even+better+via+value%2Dguided+monte%2Dcarlo+tree+searc)
38. Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, and Asli Celikyilmaz. Making ppo even better: Value-guided monte-carlo tree search decoding. arXiv preprint arXiv:2309.[15028, 2023b](https://md.research.elamur.ai/free/en/papers/liu-2023-don-t-throw-away-value/).
39. Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding-time realignment of language models, 2024b. URL https://arxiv.org/abs/2402.02992. — [arXiv:2402.02992](https://arxiv.org/abs/2402.02992) [↗](https://arxiv.org/abs/2402.02992)
40. Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D. Manning. [An emulator for fine-tuning large language models using small language models](https://md.research.elamur.ai/free/en/papers/mitchell-2023-emulator-fine-tuning-large-language/), 2023.
41. Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decoding from language models. arXiv preprint arXiv:2310.17022, 2023. [find](https://scholar.google.com/scholar?q=Sidharth+Mudgal%2C+Jong+Lee%2C+Harish+Ganapathy%2C+YaGuang+Li%2C+Tao+Wang%2C+Yanping+Huang%2C+Zhifeng+Chen%2C+Heng%2DTze+Cheng%2C+Michael+Collins%2C+Trevor+Strohman%2C+et+al%2E+Controlled+decoding+from+language+models%2E+arXiv)
42. Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami. Controlled decoding from language models, 2024. [find](https://scholar.google.com/scholar?q=Sidharth+Mudgal%2C+Jong+Lee%2C+Harish+Ganapathy%2C+YaGuang+Li%2C+Tao+Wang%2C+Yanping+Huang%2C+Zhifeng+Chen%2C+Heng%2DTze+Cheng%2C+Michael+Collins%2C+Trevor+Strohman%2C+Jilin+Chen%2C+Alex+Beutel%2C+and+Ahmad+Beirami%2E+Controlled)
43. Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning, 2017. [find](https://scholar.google.com/scholar?q=Ofir+Nachum%2C+Mohammad+Norouzi%2C+Kelvin+Xu%2C+and+Dale+Schuurmans%2E+Bridging+the+gap+between+value+and+policy+based+reinforcement+learning%2C+2017%2E)
44. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. [Webgpt: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/). arXiv preprint arXiv:2112.09332, 2021.
45. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted qu… — [WebGPT: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)
46. Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, pp. 278–287, 1999. [find](https://scholar.google.com/scholar?q=Andrew+Y+Ng%2C+Daishi+Harada%2C+and+Stuart+Russell%2E+Policy+invariance+under+reward+transformations%3A+Theory+and+application+to+reward+shaping%2E+In+Icml%2C+volume+99%2C+pp%2E+278%E2%80%93287%2C+1999%2E)
47. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
48. Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. [Smaug: Fixing failure modes of preference optimisation with dpo-positive](https://md.research.elamur.ai/free/en/papers/pal-2024-smaug-fixing-failure-modes-preference/). arXiv preprint arXiv:2402.13228, 2024.
49. Sarah Pan, Vladislav Lialin, Sherin Muckatira, and Anna Rumshisky. Let’s reinforce step by step. arXiv preprint arXiv:2311.05821, 2023. — [arXiv:2311.05821](https://arxiv.org/abs/2311.05821) [find](https://scholar.google.com/scholar?q=Sarah+Pan%2C+Vladislav+Lialin%2C+Sherin+Muckatira%2C+and+Anna+Rumshisky%2E+Let%E2%80%99s+reinforce+step+by+step%2E+arXiv+preprint+arXiv%3A2311%2E05821%2C+2023%2E+%E2%80%94+%5BarXiv%3A2311%2E05821%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E05821%29)
50. Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. [Iterative reasoning preference optimization](https://md.research.elamur.ai/free/en/papers/pang-2024-iterative-reasoning-preference-optimization/), 2024. URL https://arxiv.org/abs/2404.19733. [↗](https://arxiv.org/abs/2404.19733)
51. Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. [Disentangling length from quality in direct preference optimization](https://md.research.elamur.ai/free/en/papers/park-2024-disentangling-length-quality-direct-preference/), 2024.
52. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2305.18290. [↗](https://arxiv.org/abs/2305.18290)
53. Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms, 2024. URL https://arxiv.org/abs/2406.02900. — [arXiv:2406.02900](https://arxiv.org/abs/2406.02900) [↗](https://arxiv.org/abs/2406.02900)
54. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2C+2017%2E)
55. Charlie Victor Snell, Ilya Kostrikov, Yi Su, Sherry Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning. In The Eleventh International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=Charlie+Victor+Snell%2C+Ilya+Kostrikov%2C+Yi+Su%2C+Sherry+Yang%2C+and+Sergey+Levine%2E+Offline+rl+for+natural+language+generation+with+implicit+language+q+learning%2E+In+The+Eleventh+International+Conference+on+L)
56. Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-based trajectory optimization for llm agents, 2024. [find](https://scholar.google.com/scholar?q=Yifan+Song%2C+Da+Yin%2C+Xiang+Yue%2C+Jie+Huang%2C+Sujian+Li%2C+and+Bill+Yuchen+Lin%2E+Trial+and+error%3A+Exploration%2Dbased+trajectory+optimization+for+llm+agents%2C+2024%2E)
57. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeff+Wu%2C+Daniel+M%2E+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+Christiano%2E+Learning+to+summarize+from+human+feedback%2C+2022%2E)
58. Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization, 2023. [find](https://scholar.google.com/scholar?q=Bram+Wallace%2C+Meihua+Dang%2C+Rafael+Rafailov%2C+Linqi+Zhou%2C+Aaron+Lou%2C+Senthil+Purushwalkam%2C+Stefano+Ermon%2C+Caiming+Xiong%2C+Shafiq+Joty%2C+and+Nikhil+Naik%2E+Diffusion+model+alignment+using+direct+preference+o)
59. Joe Watson, Sandy Huang, and Nicolas Heess. Coherent soft imitation learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL [https://openreview.net/forum?id=kCCD8d2aEu](https://openreview.net/forum?id=kCCD8d2aEu). [↗](https://openreview.net/forum?id=kCCD8d2aEu)
60. Aaron Wilson, Alan Fern, and Prasad Tadepalli. A bayesian approach for policy learning from trajectory preference queries. In Advances in Neural Information Processing Systems, 2012. [find](https://scholar.google.com/scholar?q=Aaron+Wilson%2C+Alan+Fern%2C+and+Prasad+Tadepalli%2E+A+bayesian+approach+for+policy+learning+from+trajectory+preference+queries%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2012%2E)
61. Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. Agentgym: Evolving large language model-based agents… [find](https://scholar.google.com/scholar?q=Zhiheng+Xi%2C+Yiwen+Ding%2C+Wenxiang+Chen%2C+Boyang+Hong%2C+Honglin+Guo%2C+Junzhe+Wang%2C+Dingwen+Yang%2C+Chenyang+Liao%2C+Xin+Guo%2C+Wei+He%2C+Songyang+Gao%2C+Lu+Chen%2C+Rui+Zheng%2C+Yicheng+Zou%2C+Tao+Gui%2C+Qi+Zhang%2C+Xipeng+Qiu)
62. Kevin Yang and Dan Klein. [Fudge: Controlled text generation with future discriminators](https://md.research.elamur.ai/free/en/papers/yang-2021-fudge-controlled-text-generation-future/). arXiv preprint arXiv:2104.05218, 2021.
63. Zishun Yu, Yunzhe Tao, Liyu Chen, Tao Sun, and Hongxia Yang. B-coder: Value-based deep reinforcement learning for program synthesis, 2024. [find](https://scholar.google.com/scholar?q=Zishun+Yu%2C+Yunzhe+Tao%2C+Liyu+Chen%2C+Tao+Sun%2C+and+Hongxia+Yang%2E+B%2Dcoder%3A+Value%2Dbased+deep+reinforcement+learning+for+program+synthesis%2C+2024%2E)
64. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. [Star: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). Advances in Neural Information Processing Systems, 35:15476–15488, 2022.
65. Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. [Slic-hf: Sequence likelihood calibration with human feedback](https://md.research.elamur.ai/free/en/papers/zhao-2023-slic-hf-sequence-likelihood-calibration/). arXiv preprint arXiv:2305.10425, 2023a.
66. Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization. arXiv preprint arXiv:2311.16839, 2023b. — [arXiv:2311.16839](https://arxiv.org/abs/2311.16839) [find](https://scholar.google.com/scholar?q=Zhiyuan+Zhao%2C+Bin+Wang%2C+Linke+Ouyang%2C+Xiaoyi+Dong%2C+Jiaqi+Wang%2C+and+Conghui+He%2E+Beyond+hallucinations%3A+Enhancing+lvlms+through+hallucination%2Daware+direct+preference+optimization%2E+arXiv+preprint+arXiv%3A2)
67. Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010. [find](https://scholar.google.com/scholar?q=Brian+D+Ziebart%2E+Modeling+purposeful+adaptive+behavior+with+the+principle+of+maximum+causal+entropy%2E+Carnegie+Mellon+University%2C+2010%2E)
68. Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433–1438. Chicago, IL, USA, 2008. [find](https://scholar.google.com/scholar?q=Brian+D+Ziebart%2C+Andrew+L+Maas%2C+J+Andrew+Bagnell%2C+Anind+K+Dey%2C+et+al%2E+Maximum+entropy+inverse+reinforcement+learning%2E+In+Aaai%2C+volume+8%2C+pp%2E+1433%E2%80%931438%2E+Chicago%2C+IL%2C+USA%2C+2008%2E)
69. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020. [find](https://scholar.google.com/scholar?q=Daniel+M%2E+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B%2E+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+Fine%2Dtuning+language+models+from+human+preferences%2C+2020%2E)

*Updated 2 November 2026*