---
title: "ODIN: Disentangled Reward Mitigates Hacking in RLHF (2024)"
description: "RLHF-tuned language models often exploit the reward model by producing longer, neatly formatted… 65 references, cited by 37 papers here."
authors: ["Lichang Chen", "Chen Zhu", "Davit Soselia", "Jiuhai Chen", "Tianyi Zhou", "Tom Goldstein", "Heng Huang", "Mohammad Shoeybi", "Bryan Catanzaro"]
year: 2024
venue: "arXiv:2402.07319"
source: "https://arxiv.org/abs/2402.07319"
doi: "10.48550/arxiv.2402.07319"
arxiv: "2402.07319"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2024-odin-disentangled-reward-mitigates-hacking/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2024-odin-disentangled-reward-mitigates-hacking/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/) / ODIN: Disentangled Reward Mitigates Hacking in RLHF

# ODIN: Disentangled Reward Mitigates Hacking in RLHF

**Authors:** [Lichang Chen](https://md.research.elamur.ai/free/en/authors/lichang-chen/), [Chen Zhu](https://md.research.elamur.ai/free/en/authors/chen-zhu/), Davit Soselia, [Jiuhai Chen](https://md.research.elamur.ai/free/en/authors/jiuhai-chen/), [Tianyi Zhou](https://md.research.elamur.ai/free/en/authors/tianyi-zhou/), [Tom Goldstein](https://md.research.elamur.ai/free/en/authors/tom-goldstein/), [Heng Huang](https://md.research.elamur.ai/free/en/authors/heng-huang/), [Mohammad Shoeybi](https://md.research.elamur.ai/free/en/authors/mohammad-shoeybi/) and [Bryan Catanzaro](https://md.research.elamur.ai/free/en/authors/bryan-catanzaro/) · **Year:** 2024 · **Venue:** arXiv:2402.07319 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.07319) · [PDF](https://arxiv.org/pdf/2402.07319) · [DOI 10.48550/arxiv.2402.07319](https://doi.org/10.48550/arxiv.2402.07319) · [arXiv 2402.07319](https://arxiv.org/abs/2402.07319) · [OpenAlex](https://openalex.org/W4391800970)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [37](https://md.research.elamur.ai/free/en/papers/chen-2024-odin-disentangled-reward-mitigates-hacking/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
RLHF-tuned language models often exploit the reward model by producing longer, neatly formatted answers that are not actually more useful. Both learned reward models and LLM judges such as GPT-4 tend to favor verbosity. The authors address this length hacking in evaluation and in training. For evaluation, they run large hyperparameter grid searches and compare training setups by the Pareto front of judge score against response length, which reduces the effect of the judge's own length preference. With this protocol, they test common RL controls: KL strength, PPO clipping, the size of the off-policy experience pool, reward clipping and length penalties. Tuning these improves results but does not produce a simple recipe. Their main contribution, ODIN, is a reward model with two linear heads on shared features. One head is trained to track length, and the other is trained to be uncorrelated with it. Only the quality head is used during RL. ODIN almost removes the link between reward and length, and it improves the Pareto front for both PPO and ReMax. This makes the alignment of helpful assistants more robust.

## Key points
- **ODIN reward model.** It is trained on the same human preference data as a vanilla Bradley–Terry reward model. Its loss sums both heads' rewards and adds extra terms that push one head toward length and the other away from it. The length head is discarded before RL fine-tuning.
- **Evaluation protocol.** Methods are ranked by their score-versus-length Pareto front. The score is the "Win Score," defined as win rate minus loss rate as judged by an LLM. This counters length bias seen on leaderboards such as Alpaca-Eval.
- **PPO hyperparameter findings.**
  - A more conservative PPO clipping value reduced hacking.
  - With batch size 32, experience pools of N = 32, 64 and 256 (0%, 50% and 87.5% off-policy) showed that larger N did not help.
  - Reward clipping (c = 2 or 4) helped only when carefully tuned.
  - The KL coefficient (β = 0.1, 0.2, 0.4) had little effect near SFT-level lengths.
- **Comparison with tuned baselines.** ODIN outperforms PPO and ReMax baselines that used reward clipping and length penalties with a larger compute budget. It is also compared with DPO and with models such as GPT-3.5-turbo, tulu2-dpo-7b and vicuna-7b-v1.5.
- **Capability checks.** Base capabilities are checked on BBH, MMLU, DROP and TruthfulQA. The authors expect these scores to stay stable after RLHF rather than improve.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Maryland, College Park](https://md.research.elamur.ai/free/en/universities/university-of-maryland-college-park/) — University, College Park, United States · 151 works · 29,733 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

## Cite

Cite as: Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi and Bryan Catanzaro. 2024. ODIN: Disentangled Reward Mitigates Hacking in RLHF. arXiv:2402.07319.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chen2024odin,
  title = {{ODIN: Disentangled Reward Mitigates Hacking in RLHF}},
  author = {Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi and Bryan Catanzaro},
  year = {2024},
  doi = {10.48550/arxiv.2402.07319},
  eprint = {2402.07319},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.07319}
}
```

- [← previous Introducing v0.5 of the AI Safety Benchmark from MLCommons](https://md.research.elamur.ai/free/en/papers/vidgen-2024-introducing-v0-5-ai-safety/)
- [next → Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus](https://md.research.elamur.ai/free/en/papers/lee-2024-reasoning-abilities-large-language-models/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/)

## References

65 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. Anthropic introducing claude., March 2023. URL [https://www.anthropic.com/](https://www.anthropic.com/) index/introducing-claude. [↗](https://www.anthropic.com/)
2. A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. [find](https://scholar.google.com/scholar?q=A%2E+Askell%2C+Y%2E+Bai%2C+A%2E+Chen%2C+D%2E+Drain%2C+D%2E+Ganguli%2C+T%2E+Henighan%2C+A%2E+Jones%2C+N%2E+Joseph%2C+B%2E+Mann%2C+N%2E+DasSarma%2C+et+al%2E+A+general+language+assistant+as+a+laboratory+for+alignment%2E+arXiv+preprint+arXiv%3A2112%2E0)
3. M. G. Azar, M. Rowland, B. Piot, D. Guo, D. Calandriello, M. Valko, and R. Munos. [A general theoretical paradigm to understand learning from human preferences](https://md.research.elamur.ai/free/en/papers/azar-2023-general-theoretical-paradigm-understand-learning/), 2023.
4. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073, 2022.
5. R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. [find](https://scholar.google.com/scholar?q=R%2E+A%2E+Bradley+and+M%2E+E%2E+Terry%2E+Rank+analysis+of+incomplete+block+designs%3A+I%2E+the+method+of+paired+comparisons%2E+Biometrika%2C+39%283%2F4%29%3A324%E2%80%93345%2C+1952%2E)
6. C. Burns, P. Izmailov, J. H. Kirchner, B. Baker, L. Gao, L. Aschenbrenner, Y. Chen, A. Ecoffet, M. Joglekar, J. Leike, et al. [Weak-to-strong generalization: Eliciting strong capabilities with weak supervision](https://md.research.elamur.ai/free/en/papers/burns-2023-weak-strong-generalization-eliciting-strong/). arXiv preprint arXiv:2312.09390, 2023.
7. L. Chen, S. Li, J. Yan, H. Wang, K. Gunaratna, V. Yadav, Z. Tang, V. Srinivasan, T. Zhou, H. Huang, and H. Jin. [Alpagasus: Training a better alpaca with fewer data](https://md.research.elamur.ai/free/en/papers/chen-2023-alpagasus-training-better-alpaca-fewer/), 2023.
8. Y. K. Chia, P. Hong, L. Bing, and S. Poria. [Instructeval: Towards holistic evaluation of instruction-tuned large language models](https://md.research.elamur.ai/free/en/papers/chia-2023-instructeval-towards-holistic-evaluation-instruction/), 2023.
9. W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL [https://lmsys.org/blog/2023-03-30-vicuna/](https://lmsys.org/blog/2023-03-30-vicuna/). [↗](https://lmsys.org/blog/2023-03-30-vicuna/)
10. D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. [DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/). In Proc. of NAACL, 2019.
11. Y. Dubois, X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. Liang, and T. B. Hashimoto. [Alpacafarm: A simulation framework for methods that learn from human feedback](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/). arXiv preprint arXiv:2305.14387, 2023.
12. J. Eisenstein, C. Nagpal, A. Agarwal, A. Beirami, A. D’Amour, D. Dvijotham, A. Fisch, K. Heller, S. Pfohl, D. Ramachandran, et al. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09244, 2023. — [arXiv:2312.09244](https://arxiv.org/abs/2312.09244) [find](https://scholar.google.com/scholar?q=J%2E+Eisenstein%2C+C%2E+Nagpal%2C+A%2E+Agarwal%2C+A%2E+Beirami%2C+A%2E+D%E2%80%99Amour%2C+D%2E+Dvijotham%2C+A%2E+Fisch%2C+K%2E+Heller%2C+S%2E+Pfohl%2C+D%2E+Ramachandran%2C+et+al%2E+Helping+or+herding%3F+reward+model+ensembles+mitigate+but+do+not+elimin)
13. L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph, and A. Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo. arXiv preprint arXiv:2005.12729, 2020. [find](https://scholar.google.com/scholar?q=L%2E+Engstrom%2C+A%2E+Ilyas%2C+S%2E+Santurkar%2C+D%2E+Tsipras%2C+F%2E+Janoos%2C+L%2E+Rudolph%2C+and+A%2E+Madry%2E+Implementation+matters+in+deep+policy+gradients%3A+A+case+study+on+ppo+and+trpo%2E+arXiv+preprint+arXiv%3A2005%2E12729%2C+20)
14. K. Ethayarajh, W. Xu, D. Jurafsky, and D. Kiela. Human-centered loss functions (halos). Technical report, Contextual AI, 2023. [find](https://scholar.google.com/scholar?q=K%2E+Ethayarajh%2C+W%2E+Xu%2C+D%2E+Jurafsky%2C+and+D%2E+Kiela%2E+Human%2Dcentered+loss+functions+%28halos%29%2E+Technical+report%2C+Contextual+AI%2C+2023%2E)
15. L. Gao, J. Schulman, and J. Hilton. [Scaling laws for reward model overoptimization](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/). In International Conference on Machine Learning, pages 10835–10866. PMLR, 2023.
16. X. Geng, A. Gudibande, H. Liu, E. Wallace, P. Abbeel, S. Levine, and D. Song. Koala: A dialogue model for academic research. Blog post, April 2023. URL [https://bair.berkeley.edu/](https://bair.berkeley.edu/) blog/2023/04/03/koala/. [↗](https://bair.berkeley.edu/)
17. Google. [Gemini: A family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/), 2023.
18. C. Gulcehre, T. L. Paine, S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma, A. Siddhant, A. Ahern, M. Wang, C. Gu, et al. [Reinforced self-training (rest) for language modeling](https://md.research.elamur.ai/free/en/papers/gulcehre-2023-reinforced-self-training-rest-language/). arXiv preprint arXiv:2308.08998, 2023.
19. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300, 2020.
20. D. Hendrycks, N. Carlini, J. Schulman, and J. Steinhardt. [Unsolved problems in ml safety](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-unsolved-problems-ml-safety/). arXiv preprint arXiv:2109.13916, 2021.
21. H. Ivison, Y. Wang, V. Pyatkin, N. Lambert, M. Peters, P. Dasigi, J. Jang, D. Wadden, N. A. Smith, I. Beltagy, and H. Hajishirzi. [Camels in a changing climate: Enhancing lm adaptation with tulu 2](https://md.research.elamur.ai/free/en/papers/ivison-2023-camels-changing-climate-enhancing-lm/), 2023.
22. A. Köpf, Y. Kilcher, D. von Rütte, S. Anagnostidis, Z.-R. Tam, K. Stevens, A. Barhoum, N. M. Duc, O. Stanley, R. Nagyfi, S. ES, S. Suri, D. Glushkov, A. Dantuluri, A. Maguire, C. Schuhmann, H. Nguyen, and A. Mattick. Openassistant conversations – democratizing large language model alignment, 2023. — [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://md.research.elamur.ai/free/en/papers/kopf-2023-openassistant-conversations-democratizing-large-language/)
23. H. Lee, S. Phatale, H. Mansoor, K. Lu, T. Mesnard, C. Bishop, V. Carbune, and A. Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023. [find](https://scholar.google.com/scholar?q=H%2E+Lee%2C+S%2E+Phatale%2C+H%2E+Mansoor%2C+K%2E+Lu%2C+T%2E+Mesnard%2C+C%2E+Bishop%2C+V%2E+Carbune%2C+and+A%2E+Rastogi%2E+Rlaif%3A+Scaling+reinforcement+learning+from+human+feedback+with+ai+feedback%2E+arXiv+preprint+arXiv%3A2309%2E00267%2C+2)
24. X. Li, T. Zhang, Y. Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/](https://github.com/) tatsu-lab/alpaca_eval, 2023a. [↗](https://github.com/)
25. Z. Li, T. Xu, Y. Zhang, Z. Lin, Y. Yu, R. Sun, and Z.-Q. Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models, 2023b. [find](https://scholar.google.com/scholar?q=Z%2E+Li%2C+T%2E+Xu%2C+Y%2E+Zhang%2C+Z%2E+Lin%2C+Y%2E+Yu%2C+R%2E+Sun%2C+and+Z%2E%2DQ%2E+Luo%2E+Remax%3A+A+simple%2C+effective%2C+and+efficient+reinforcement+learning+method+for+aligning+large+language+models%2C+2023b%2E)
26. S. Lin, J. Hilton, and O. Evans. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/), 2021.
27. J. Liu, Y. Zhu, K. Xiao, Q. Fu, X. Han, W. Yang, and D. Ye. Rltf: Reinforcement learning from unit test feedback. arXiv preprint arXiv:2307.04349, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Liu%2C+Y%2E+Zhu%2C+K%2E+Xiao%2C+Q%2E+Fu%2C+X%2E+Han%2C+W%2E+Yang%2C+and+D%2E+Ye%2E+Rltf%3A+Reinforcement+learning+from+unit+test+feedback%2E+arXiv+preprint+arXiv%3A2307%2E04349%2C+2023%2E)
28. P. J. Liu, 2024. URL [https://twitter.com/i/web/status/1750318955808583997](https://twitter.com/i/web/status/1750318955808583997). [↗](https://twitter.com/i/web/status/1750318955808583997)
29. T. Liu, Y. Zhao, R. Joshi, M. Khalman, M. Saleh, P. J. Liu, and J. Liu. [Statistical rejection sampling improves preference optimization](https://md.research.elamur.ai/free/en/papers/liu-2023-statistical-rejection-sampling-improves-preference/). ICLR, 2024.
30. F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 4114–4124. PMLR, 2019. [find](https://scholar.google.com/scholar?q=F%2E+Locatello%2C+S%2E+Bauer%2C+M%2E+Lucic%2C+G%2E+Raetsch%2C+S%2E+Gelly%2C+B%2E+Sch%C3%B6lkopf%2C+and+O%2E+Bachem%2E+Challenging+common+assumptions+in+the+unsupervised+learning+of+disentangled+representations%2E+In+international+confe)
31. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015. [find](https://scholar.google.com/scholar?q=V%2E+Mnih%2C+K%2E+Kavukcuoglu%2C+D%2E+Silver%2C+A%2E+A%2E+Rusu%2C+J%2E+Veness%2C+M%2E+G%2E+Bellemare%2C+A%2E+Graves%2C+M%2E+Riedmiller%2C+A%2E+K%2E+Fidjeland%2C+G%2E+Ostrovski%2C+et+al%2E+Human%2Dlevel+control+through+deep+reinforcement+learning%2E+nat)
32. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al. [Webgpt: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/). arXiv preprint arXiv:2112.09332, 2021.
33. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023.
34. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
35. S. Pichai. An important next step on our ai journey, February 2023. URL [https://blog.google/](https://blog.google/) technology/ai/bard-google-ai-search-updates/. [↗](https://blog.google/)
36. R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/), 2023.
37. A. Rame, G. Couairon, M. Shukor, C. Dancette, J.-B. Gaya, L. Soulier, and M. Cord. [Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards](https://md.research.elamur.ai/free/en/papers/rame-2023-rewarded-soups-towards-pareto-optimal/). arXiv preprint arXiv:2306.04488, 2023.
38. A. Ramé, N. Vieillard, L. Hussenot, R. Dadashi, G. Cideron, O. Bachem, and J. Ferret. Warm: On the benefits of weight averaged reward models. arXiv preprint arXiv:2401.12187, 2024. — [arXiv:2401.12187](https://arxiv.org/abs/2401.12187) [find](https://scholar.google.com/scholar?q=A%2E+Ram%C3%A9%2C+N%2E+Vieillard%2C+L%2E+Hussenot%2C+R%2E+Dadashi%2C+G%2E+Cideron%2C+O%2E+Bachem%2C+and+J%2E+Ferret%2E+Warm%3A+On+the+benefits+of+weight+averaged+reward+models%2E+arXiv+preprint+arXiv%3A2401%2E12187%2C+2024%2E+%E2%80%94+%5BarXiv%3A2401%2E12187)
39. T. Salimans and D. P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems, 29, 2016. [find](https://scholar.google.com/scholar?q=T%2E+Salimans+and+D%2E+P%2E+Kingma%2E+Weight+normalization%3A+A+simple+reparameterization+to+accelerate+training+of+deep+neural+networks%2E+Advances+in+neural+information+processing+systems%2C+29%2C+2016%2E)
40. J. Schulman. Approximating kl divergence, 2020. URL [http://joschu.net/blog/kl-approx](http://joschu.net/blog/kl-approx). html. [↗](http://joschu.net/blog/kl-approx)
41. J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015. [find](https://scholar.google.com/scholar?q=J%2E+Schulman%2C+P%2E+Moritz%2C+S%2E+Levine%2C+M%2E+Jordan%2C+and+P%2E+Abbeel%2E+High%2Ddimensional+continuous+control+using+generalized+advantage+estimation%2E+arXiv+preprint+arXiv%3A1506%2E02438%2C+2015%2E)
42. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=J%2E+Schulman%2C+F%2E+Wolski%2C+P%2E+Dhariwal%2C+A%2E+Radford%2C+and+O%2E+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)
43. J. Schulman, B. Zoph, C. Kim, J. Hilton, J. Menick, J. Weng, J. F. C. Uribe, L. Fedus, L. Metz, M. Pokorny, et al. Chatgpt: Optimizing language models for dialogue. OpenAI blog, 2022. [find](https://scholar.google.com/scholar?q=J%2E+Schulman%2C+B%2E+Zoph%2C+C%2E+Kim%2C+J%2E+Hilton%2C+J%2E+Menick%2C+J%2E+Weng%2C+J%2E+F%2E+C%2E+Uribe%2C+L%2E+Fedus%2C+L%2E+Metz%2C+M%2E+Pokorny%2C+et+al%2E+Chatgpt%3A+Optimizing+language+models+for+dialogue%2E+OpenAI+blog%2C+2022%2E)
44. W. Shen, R. Zheng, W. Zhan, J. Zhao, S. Dou, T. Gui, Q. Zhang, and X.-J. Huang. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2859–2873, 2023. [find](https://scholar.google.com/scholar?q=W%2E+Shen%2C+R%2E+Zheng%2C+W%2E+Zhan%2C+J%2E+Zhao%2C+S%2E+Dou%2C+T%2E+Gui%2C+Q%2E+Zhang%2C+and+X%2E%2DJ%2E+Huang%2E+Loose+lips+sink+ships%3A+Mitigating+length+bias+in+reinforcement+learning+from+human+feedback%2E+In+Findings+of+the+Associat)
45. P. Singhal, T. Goyal, J. Xu, and G. Durrett. [A long way to go: Investigating length correlations in rlhf](https://md.research.elamur.ai/free/en/papers/singhal-2023-long-way-go-investigating-length/), 2023.
46. P. Sountsov and S. Sarawagi. Length bias in encoder decoder models and a case for global conditioning. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1516–1525, 2016. [find](https://scholar.google.com/scholar?q=P%2E+Sountsov+and+S%2E+Sarawagi%2E+Length+bias+in+encoder+decoder+models+and+a+case+for+global+conditioning%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+)
47. N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. [find](https://scholar.google.com/scholar?q=N%2E+Stiennon%2C+L%2E+Ouyang%2C+J%2E+Wu%2C+D%2E+Ziegler%2C+R%2E+Lowe%2C+C%2E+Voss%2C+A%2E+Radford%2C+D%2E+Amodei%2C+and+P%2E+F%2E+Christiano%2E+Learning+to+summarize+with+human+feedback%2E+Advances+in+Neural+Information+Processing+Systems%2C+)
48. M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, , and J. Wei. [Challenging big-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). arXiv preprint arXiv:2210.09261, 2022.
49. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
50. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
51. B. Wang, R. Zheng, L. Chen, Y. Liu, S. Dou, C. Huang, W. Shen, S. Jin, E. Zhou, C. Shi, et al. [Secrets of rlhf in large language models part ii: Reward modeling](https://md.research.elamur.ai/free/en/papers/wang-2024-secrets-rlhf-large-language-models/). arXiv preprint arXiv:2401.06080, 2024.
52. P. Wang, L. Li, L. Chen, D. Zhu, B. Lin, Y. Cao, Q. Liu, T. Liu, and Z. Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023a. [find](https://scholar.google.com/scholar?q=P%2E+Wang%2C+L%2E+Li%2C+L%2E+Chen%2C+D%2E+Zhu%2C+B%2E+Lin%2C+Y%2E+Cao%2C+Q%2E+Liu%2C+T%2E+Liu%2C+and+Z%2E+Sui%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv+preprint+arXiv%3A2305%2E17926%2C+2023a%2E)
53. Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022. [find](https://scholar.google.com/scholar?q=Y%2E+Wang%2C+Y%2E+Kordi%2C+S%2E+Mishra%2C+A%2E+Liu%2C+N%2E+A%2E+Smith%2C+D%2E+Khashabi%2C+and+H%2E+Hajishirzi%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+arXiv+preprint+arXiv%3A2212%2E10560%2C+2022%2E)
54. Y. Wang, H. Ivison, P. Dasigi, J. Hessel, T. Khot, K. R. Chandu, D. Wadden, K. MacMillan, N. A. Smith, I. Beltagy, et al. [How far can camels go? exploring the state of instruction tuning on open resources](https://md.research.elamur.ai/free/en/papers/wang-2023-far-camels-go-exploring-state/). arXiv preprint arXiv:2306.04751, 2023b.
55. R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229–256, 1992. [find](https://scholar.google.com/scholar?q=R%2E+J%2E+Williams%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Machine+learning%2C+8%3A229%E2%80%93256%2C+1992%2E)
56. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush. [Transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/).…
57. Z. Wu, Y. Hu, W. Shi, N. Dziri, A. Suhr, P. Ammanabrolu, N. A. Smith, M. Ostendorf, and H. Hajishirzi. [Fine-grained human feedback gives better rewards for language model training](https://md.research.elamur.ai/free/en/papers/wu-2023-fine-grained-human-feedback-gives/). arXiv preprint arXiv:2306.01693, 2023.
58. C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, and D. Jiang. Wizardlm: Empowering large language models to follow complex instructions, 2023. [find](https://scholar.google.com/scholar?q=C%2E+Xu%2C+Q%2E+Sun%2C+K%2E+Zheng%2C+X%2E+Geng%2C+P%2E+Zhao%2C+J%2E+Feng%2C+C%2E+Tao%2C+and+D%2E+Jiang%2E+Wizardlm%3A+Empowering+large+language+models+to+follow+complex+instructions%2C+2023%2E)
59. Z. Yao, R. Y. Aminabadi, O. Ruwase, S. Rajbhandari, X. Wu, A. A. Awan, J. Rasley, M. Zhang, C. Li, C. Holmes, et al. Deepspeed-chat: Easy, fast and affordable rlhf training of chatgpt-like models at all scales. arXiv preprint arXiv:2308.01320, 2023. — [arXiv:2308.01320](https://arxiv.org/abs/2308.01320) [find](https://scholar.google.com/scholar?q=Z%2E+Yao%2C+R%2E+Y%2E+Aminabadi%2C+O%2E+Ruwase%2C+S%2E+Rajbhandari%2C+X%2E+Wu%2C+A%2E+A%2E+Awan%2C+J%2E+Rasley%2C+M%2E+Zhang%2C+C%2E+Li%2C+C%2E+Holmes%2C+et+al%2E+Deepspeed%2Dchat%3A+Easy%2C+fast+and+affordable+rlhf+training+of+chatgpt%2Dlike+models+at+a)
60. Z. Zeng, J. Yu, T. Gao, Y. Meng, T. Goyal, and D. Chen. Evaluating large language models at evaluating instruction following. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023. [find](https://scholar.google.com/scholar?q=Z%2E+Zeng%2C+J%2E+Yu%2C+T%2E+Gao%2C+Y%2E+Meng%2C+T%2E+Goyal%2C+and+D%2E+Chen%2E+Evaluating+large+language+models+at+evaluating+instruction+following%2E+In+NeurIPS+2023+Workshop+on+Instruction+Tuning+and+Instruction+Following%2C+)
61. Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu. [Slic-hf: Sequence likelihood calibration with human feedback](https://md.research.elamur.ai/free/en/papers/zhao-2023-slic-hf-sequence-likelihood-calibration/), 2023.
62. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). arXiv preprint arXiv:2306.05685, 2023a.
63. R. Zheng, S. Dou, S. Gao, Y. Hua, W. Shen, B. Wang, Y. Liu, S. Jin, Q. Liu, Y. Zhou, et al. [Secrets of rlhf in large language models part i: Ppo](https://md.research.elamur.ai/free/en/papers/zheng-2023-secrets-rlhf-large-language-models/). arXiv preprint arXiv:2307.04964, 2023b.
64. C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, S. Zhang, G. Ghosh, M. Lewis, L. Zettlemoyer, and O. Levy. Lima: Less is more for alignment, 2023. [find](https://scholar.google.com/scholar?q=C%2E+Zhou%2C+P%2E+Liu%2C+P%2E+Xu%2C+S%2E+Iyer%2C+J%2E+Sun%2C+Y%2E+Mao%2C+X%2E+Ma%2C+A%2E+Efrat%2C+P%2E+Yu%2C+L%2E+Yu%2C+S%2E+Zhang%2C+G%2E+Ghosh%2C+M%2E+Lewis%2C+L%2E+Zettlemoyer%2C+and+O%2E+Levy%2E+Lima%3A+Less+is+more+for+alignment%2C+2023%2E)
65. D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences, 2019. [find](https://scholar.google.com/scholar?q=D%2E+M%2E+Ziegler%2C+N%2E+Stiennon%2C+J%2E+Wu%2C+T%2E+B%2E+Brown%2C+A%2E+Radford%2C+D%2E+Amodei%2C+P%2E+Christiano%2C+and+G%2E+Irving%2E+Fine%2Dtuning+language+models+from+human+preferences%2C+2019%2E)

*Updated 3 November 2026*