---
title: "Secrets of RLHF in Large Language Models Part II: Reward"
description: "In RLHF, reward models stand in for human judgment, but they have two weaknesses. 43 references, cited by 65 papers here."
authors: ["Binghai Wang", "Rui Zheng", "Lu Chen", "Yan Liu", "Shihan Dou", "Caishuang Huang", "Wei Shen", "Senjie Jin", "Enyu Zhou", "Chenyu Shi", "Songyang Gao", "Nuo Xu", "Yuhao Zhou", "Xiaoran Fan", "Zhiheng Xi", "Jun Zhao", "Xiao Wang", "Tao Ji", "Hang Yan", "Lixing Shen", "Zhan Chen", "Tao Gui", "Qi Zhang", "Xipeng Qiu", "Xuanjing Huang", "Zuxuan Wu", "Yu-Gang Jiang"]
year: 2024
venue: "arXiv:2401.06080"
source: "https://arxiv.org/abs/2401.06080"
doi: "10.48550/arxiv.2401.06080"
arxiv: "2401.06080"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2024-secrets-rlhf-large-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2024-secrets-rlhf-large-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / Secrets of RLHF in Large Language Models Part II: Reward Modeling

# Secrets of RLHF in Large Language Models Part II: Reward Modeling

**Authors:** [Binghai Wang](https://md.research.elamur.ai/free/en/authors/binghai-wang/), [Rui Zheng](https://md.research.elamur.ai/free/en/authors/rui-zheng/), [Lu Chen](https://md.research.elamur.ai/free/en/authors/lu-chen/), [Yan Liu](https://md.research.elamur.ai/free/en/authors/yan-liu/), [Shihan Dou](https://md.research.elamur.ai/free/en/authors/shihan-dou/), [Caishuang Huang](https://md.research.elamur.ai/free/en/authors/caishuang-huang/), [Wei Shen](https://md.research.elamur.ai/free/en/authors/wei-shen/), [Senjie Jin](https://md.research.elamur.ai/free/en/authors/senjie-jin/), [Enyu Zhou](https://md.research.elamur.ai/free/en/authors/enyu-zhou/), Chenyu Shi, [Songyang Gao](https://md.research.elamur.ai/free/en/authors/songyang-gao/), [Nuo Xu](https://md.research.elamur.ai/free/en/authors/nuo-xu/), [Yuhao Zhou](https://md.research.elamur.ai/free/en/authors/yuhao-zhou/), [Xiaoran Fan](https://md.research.elamur.ai/free/en/authors/xiaoran-fan/), [Zhiheng Xi](https://md.research.elamur.ai/free/en/authors/zhiheng-xi/), [Jun Zhao](https://md.research.elamur.ai/free/en/authors/jun-zhao/), [Xiao Wang](https://md.research.elamur.ai/free/en/authors/xiao-wang/), [Tao Ji](https://md.research.elamur.ai/free/en/authors/tao-ji/), [Hang Yan](https://md.research.elamur.ai/free/en/authors/hang-yan/), Lixing Shen, Zhan Chen, [Tao Gui](https://md.research.elamur.ai/free/en/authors/tao-gui/), [Qi Zhang](https://md.research.elamur.ai/free/en/authors/qi-zhang/), [Xipeng Qiu](https://md.research.elamur.ai/free/en/authors/xipeng-qiu/), [Xuanjing Huang](https://md.research.elamur.ai/free/en/authors/xuanjing-huang/), Zuxuan Wu and [Yu-Gang Jiang](https://md.research.elamur.ai/free/en/authors/yu-gang-jiang/) · **Year:** 2024 · **Venue:** arXiv:2401.06080 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2401.06080) · [PDF](https://arxiv.org/pdf/2401.06080) · [DOI 10.48550/arxiv.2401.06080](https://doi.org/10.48550/arxiv.2401.06080) · [arXiv 2401.06080](https://arxiv.org/abs/2401.06080) · [OpenAlex](https://openalex.org/W4390833718)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [65](https://md.research.elamur.ai/free/en/papers/wang-2024-secrets-rlhf-large-language-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
In RLHF, reward models stand in for human judgment, but they have two weaknesses. Preference labels are often wrong or unclear because annotators frequently disagree. Reward models also transfer poorly to out-of-distribution outputs, which makes iterative RLHF unreliable. This report from Fudan NLP Lab and Hikvision addresses both problems.

On the data side, the authors train an ensemble of reward models on the same comparisons. The average and spread of the models' score gaps serve as a measure of preference strength. This lets them separate likely mislabeled, ambiguous and normal pairs. They then flip incorrect labels, soften ambiguous ones and add an adaptive margin, scaled by strength, to the ranking loss.

On the algorithm side, a contrastive objective helps the model tell chosen and rejected answers apart. MetaRM, a meta-learning procedure, keeps the reward model sensitive to the policy's shifting outputs.

The authors report more stable PPO and steady gains over 3–4 iterative rounds on two datasets. This matters because a sturdier reward model underpins safer and more helpful alignment.

## Key points
- Prior annotation studies show weak human agreement: about 63% for Anthropic's crowd workers and 72.6 ± 1.5% among OpenAI labelers.
- An ensemble of M = 10 reward models scored the Anthropic HH-RLHF training set:
  - About 25% of pairs had a negative mean preference difference, suggesting flipped labels.
  - Score variance traced a U-shape across strength groups.
  - Agreement with GPT-4 labels rose as preference strength grew.
- Data-side fixes: label flipping, label smoothing and an adaptive margin in the Bradley-Terry loss.
- Algorithm-side methods: contrastive learning and MetaRM. They enabled iterative RLHF with consistent gains over 3–4 rounds on HH-RLHF and OpenAI's summarization data.
- Released resources (github.com/OpenLMLab/MOSS-RLHF): training code, HH-RLHF annotated with preference strength, and a GPT-4-cleaned validation set.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 202 works · 22,509 citations
- [Hikvision](https://md.research.elamur.ai/free/en/companies/hikvision/) — Company, Hangzhou, China · 2 works · 165 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu and Yu-Gang Jiang. 2024. Secrets of RLHF in Large Language Models Part II: Reward Modeling. arXiv:2401.06080.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wang2024secrets,
  title = {{Secrets of RLHF in Large Language Models Part II: Reward Modeling}},
  author = {Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu and Yu-Gang Jiang},
  year = {2024},
  doi = {10.48550/arxiv.2401.06080},
  eprint = {2401.06080},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.06080}
}
```

- [← previous Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthiness](https://md.research.elamur.ai/free/en/papers/chen-2024-quantifying-uncertainty-answers-any-language/)
- [next → Towards Safer Large Language Models through Machine Unlearning](https://md.research.elamur.ai/free/en/papers/liu-2024-safer-large-language-models-through/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

43 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Leike, J., D. Krueger, T. Everitt, et al. [Scalable agent alignment via reward modeling: a research direction](https://md.research.elamur.ai/free/en/papers/leike-2018-scalable-agent-alignment-reward-modeling/), 2018.
2. Kenton, Z., T. Everitt, L. Weidinger, et al. [Alignment of language agents](https://md.research.elamur.ai/free/en/papers/kenton-2021-alignment-language-agents/). arXiv preprint arXiv:2103.14659, 2021.
3. Xi, Z., W. Chen, X. Guo, et al. [The rise and potential of large language model based agents: A survey](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/). arXiv preprint arXiv:2309.07864, 2023.
4. Ouyang, L., J. Wu, X. Jiang, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744, 2022. Challenge 1: Noise Incorrect Low training acc Ambiguous ... Poor PPO stability ... Reward Model Intention modeling , ) Enh…
5. Bai, Y., A. Jones, K. Ndousse, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. [find](https://scholar.google.com/scholar?q=Bai%2C+Y%2E%2C+A%2E+Jones%2C+K%2E+Ndousse%2C+et+al%2E+Training+a+helpful+and+harmless+assistant+with+reinforcement+learning+from+human+feedback%2E+arXiv+preprint+arXiv%3A2204%2E05862%2C+2022%2E)
6. Bai, Y., S. Kadavath, S. Kundu, et al. [Constitutional AI: Harmlessness from AI feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/), 2022.
7. Kundu, S., Y. Bai, S. Kadavath, et al. Specific versus general principles for constitutional ai. arXiv preprint arXiv:2310.13798, 2023. [find](https://scholar.google.com/scholar?q=Kundu%2C+S%2E%2C+Y%2E+Bai%2C+S%2E+Kadavath%2C+et+al%2E+Specific+versus+general+principles+for+constitutional+ai%2E+arXiv+preprint+arXiv%3A2310%2E13798%2C+2023%2E)
8. Lambert, N., T. Krendl Gilbert, T. Zick. The history and risks of reinforcement learning and human feedback. arXiv e-prints, pages arXiv–2310, 2023. [find](https://scholar.google.com/scholar?q=Lambert%2C+N%2E%2C+T%2E+Krendl+Gilbert%2C+T%2E+Zick%2E+The+history+and+risks+of+reinforcement+learning+and+human+feedback%2E+arXiv+e%2Dprints%2C+pages+arXiv%E2%80%932310%2C+2023%2E)
9. Pitis, S. Failure modes of learning reward models for llms and other sequence models. In ICML 2023 Workshop The Many Facets of Preference-Based Learning. 2023. [find](https://scholar.google.com/scholar?q=Pitis%2C+S%2E+Failure+modes+of+learning+reward+models+for+llms+and+other+sequence+models%2E+In+ICML+2023+Workshop+The+Many+Facets+of+Preference%2DBased+Learning%2E+2023%2E)
10. McKinney, L., Y. Duan, D. Krueger, et al. On the fragility of learned reward functions. arXiv preprint arXiv:2301.03652, 2023. — [arXiv:2301.03652](https://arxiv.org/abs/2301.03652) [find](https://scholar.google.com/scholar?q=McKinney%2C+L%2E%2C+Y%2E+Duan%2C+D%2E+Krueger%2C+et+al%2E+On+the+fragility+of+learned+reward+functions%2E+arXiv+preprint+arXiv%3A2301%2E03652%2C+2023%2E+%E2%80%94+%5BarXiv%3A2301%2E03652%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2301%2E03652%29)
11. Zheng, R., Z. Xi, Q. Liu, et al. Characterizing the impacts of instances on robustness. In Findings of the Association for Computational Linguistics: ACL 2023, pages 2314–2332. 2023. [find](https://scholar.google.com/scholar?q=Zheng%2C+R%2E%2C+Z%2E+Xi%2C+Q%2E+Liu%2C+et+al%2E+Characterizing+the+impacts+of+instances+on+robustness%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2023%2C+pages+2314%E2%80%932332%2E+2023%2E)
12. Stiennon, N., L. Ouyang, J. Wu, et al. Learning to summarize from human feedback. CoRR, abs/2009.01325, 2020. [find](https://scholar.google.com/scholar?q=Stiennon%2C+N%2E%2C+L%2E+Ouyang%2C+J%2E+Wu%2C+et+al%2E+Learning+to+summarize+from+human+feedback%2E+CoRR%2C+abs%2F2009%2E01325%2C+2020%2E)
13. Ziegler, D. M., N. Stiennon, J. Wu, et al. Fine-tuning language models from human preferences. CoRR, abs/1909.08593, 2019. [find](https://scholar.google.com/scholar?q=Ziegler%2C+D%2E+M%2E%2C+N%2E+Stiennon%2C+J%2E+Wu%2C+et+al%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+CoRR%2C+abs%2F1909%2E08593%2C+2019%2E)
14. Glaese, A., N. McAleese, M. Trebacz, et al. [Improving alignment of dialogue agents via targeted human judgements](https://md.research.elamur.ai/free/en/papers/glaese-2022-improving-alignment-dialogue-agents-targeted/). CoRR, abs/2209.14375, 2022.
15. Schulman, J., F. Wolski, P. Dhariwal, et al. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. [find](https://scholar.google.com/scholar?q=Schulman%2C+J%2E%2C+F%2E+Wolski%2C+P%2E+Dhariwal%2C+et+al%2E+Proximal+policy+optimization+algorithms%2E+CoRR%2C+abs%2F1707%2E06347%2C+2017%2E)
16. Bradley, R. A., M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. [find](https://scholar.google.com/scholar?q=Bradley%2C+R%2E+A%2E%2C+M%2E+E%2E+Terry%2E+Rank+analysis+of+incomplete+block+designs%3A+I%2E+the+method+of+paired+comparisons%2E+Biometrika%2C+39%283%2F4%29%3A324%E2%80%93345%2C+1952%2E)
17. Jaques, N., A. Ghandeharioun, J. H. Shen, et al. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. CoRR, abs/1907.00456, 2019. — [arXiv:1907.00456](https://arxiv.org/abs/1907.00456) [find](https://scholar.google.com/scholar?q=Jaques%2C+N%2E%2C+A%2E+Ghandeharioun%2C+J%2E+H%2E+Shen%2C+et+al%2E+Way+off%2Dpolicy+batch+deep+reinforcement+learning+of+implicit+human+preferences+in+dialog%2E+CoRR%2C+abs%2F1907%2E00456%2C+2019%2E+%E2%80%94+%5BarXiv%3A1907%2E00456%5D%28https%3A%2F%2Farxi)
18. Laidlaw, C., S. Singhal, A. Dragan. Preventing reward hacking with occupancy measure regularization. In ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems. 2023. [find](https://scholar.google.com/scholar?q=Laidlaw%2C+C%2E%2C+S%2E+Singhal%2C+A%2E+Dragan%2E+Preventing+reward+hacking+with+occupancy+measure+regularization%2E+In+ICML+Workshop+on+New+Frontiers+in+Learning%2C+Control%2C+and+Dynamical+Systems%2E+2023%2E)
19. Reed, S., H. Lee, D. Anguelov, et al. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596, 2014. [find](https://scholar.google.com/scholar?q=Reed%2C+S%2E%2C+H%2E+Lee%2C+D%2E+Anguelov%2C+et+al%2E+Training+deep+neural+networks+on+noisy+labels+with+bootstrapping%2E+arXiv+preprint+arXiv%3A1412%2E6596%2C+2014%2E)
20. Müller, R., S. Kornblith, G. E. Hinton. When does label smoothing help? In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, R. Garnett, eds., Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, D… [find](https://scholar.google.com/scholar?q=M%C3%BCller%2C+R%2E%2C+S%2E+Kornblith%2C+G%2E+E%2E+Hinton%2E+When+does+label+smoothing+help%3F+In+H%2E+M%2E+Wallach%2C+H%2E+Larochelle%2C+A%2E+Beygelzimer%2C+F%2E+d%E2%80%99Alch%C3%A9%2DBuc%2C+E%2E+B%2E+Fox%2C+R%2E+Garnett%2C+eds%2E%2C+Advances+in+Neural+Information+Pro)
21. Touvron, H., L. Martin, K. Stone, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). CoRR, abs/2307.09288, 2023.
22. Zheng, R., S. Dou, S. Gao, et al. [Secrets of rlhf in large language models part i: Ppo](https://md.research.elamur.ai/free/en/papers/zheng-2023-secrets-rlhf-large-language-models/). arXiv preprint arXiv:2307.04964, 2023.
23. Caron, M., I. Misra, J. Mairal, et al. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912–9924, 2020. [find](https://scholar.google.com/scholar?q=Caron%2C+M%2E%2C+I%2E+Misra%2C+J%2E+Mairal%2C+et+al%2E+Unsupervised+learning+of+visual+features+by+contrasting+cluster+assignments%2E+Advances+in+neural+information+processing+systems%2C+33%3A9912%E2%80%939924%2C+2020%2E)
24. Gao, T., X. Yao, D. Chen. [Simcse: Simple contrastive learning of sentence embeddings](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/). arXiv preprint arXiv:2104.08821, 2021.
25. Bai, Y., A. Jones, K. Ndousse, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. CoRR, abs/2204.05862, 2022. [find](https://scholar.google.com/scholar?q=Bai%2C+Y%2E%2C+A%2E+Jones%2C+K%2E+Ndousse%2C+et+al%2E+Training+a+helpful+and+harmless+assistant+with+reinforcement+learning+from+human+feedback%2E+CoRR%2C+abs%2F2204%2E05862%2C+2022%2E)
26. Christiano, P. F., J. Leike, T. Brown, et al. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. [find](https://scholar.google.com/scholar?q=Christiano%2C+P%2E+F%2E%2C+J%2E+Leike%2C+T%2E+Brown%2C+et+al%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+neural+information+processing+systems%2C+30%2C+2017%2E)
27. Kaufmann, T., P. Weng, V. Bengs, et al. [A survey of reinforcement learning from human feedback](https://md.research.elamur.ai/free/en/papers/kaufmann-2023-survey-reinforcement-learning-human-feedback/). arXiv preprint arXiv:2312.14925, 2023.
28. Hong, J., K. Bhatia, A. Dragan. On the sensitivity of reward inference to misspecified human models. arXiv preprint arXiv:2212.04717, 2022. — [arXiv:2212.04717](https://arxiv.org/abs/2212.04717) [find](https://scholar.google.com/scholar?q=Hong%2C+J%2E%2C+K%2E+Bhatia%2C+A%2E+Dragan%2E+On+the+sensitivity+of+reward+inference+to+misspecified+human+models%2E+arXiv+preprint+arXiv%3A2212%2E04717%2C+2022%2E+%E2%80%94+%5BarXiv%3A2212%2E04717%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2212%2E04717%29)
29. Knox, W. B., S. Hatgis-Kessell, S. Booth, et al. Models of human preference for learning reward functions. arXiv preprint arXiv:2206.02231, 2022. — [arXiv:2206.02231](https://arxiv.org/abs/2206.02231) [find](https://scholar.google.com/scholar?q=Knox%2C+W%2E+B%2E%2C+S%2E+Hatgis%2DKessell%2C+S%2E+Booth%2C+et+al%2E+Models+of+human+preference+for+learning+reward+functions%2E+arXiv+preprint+arXiv%3A2206%2E02231%2C+2022%2E+%E2%80%94+%5BarXiv%3A2206%2E02231%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2206%2E02231%29)
30. Casper, S., X. Davies, C. Shi, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023. [find](https://scholar.google.com/scholar?q=Casper%2C+S%2E%2C+X%2E+Davies%2C+C%2E+Shi%2C+et+al%2E+Open+problems+and+fundamental+limitations+of+reinforcement+learning+from+human+feedback%2E+arXiv+preprint+arXiv%3A2307%2E15217%2C+2023%2E)
31. Sharma, M., M. Tong, T. Korbak, et al. [Towards understanding sycophancy in language models](https://md.research.elamur.ai/free/en/papers/sharma-2023-towards-understanding-sycophancy-language-models/). arXiv preprint arXiv:2310.13548, 2023.
32. Tamkin, A., A. Askell, L. Lovitt, et al. Evaluating and mitigating discrimination in language model decisions. arXiv preprint arXiv:2312.03689, 2023. [find](https://scholar.google.com/scholar?q=Tamkin%2C+A%2E%2C+A%2E+Askell%2C+L%2E+Lovitt%2C+et+al%2E+Evaluating+and+mitigating+discrimination+in+language+model+decisions%2E+arXiv+preprint+arXiv%3A2312%2E03689%2C+2023%2E)
33. Ziegler, D. M., N. Stiennon, J. Wu, et al. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. [find](https://scholar.google.com/scholar?q=Ziegler%2C+D%2E+M%2E%2C+N%2E+Stiennon%2C+J%2E+Wu%2C+et+al%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint+arXiv%3A1909%2E08593%2C+2019%2E)
34. Zheng, R., W. Shen, Y. Hua, et al. Improving generalization of alignment with human preferences through group invariant learning. arXiv preprint arXiv:2310.11971, 2023. — [arXiv:2310.11971](https://arxiv.org/abs/2310.11971) [find](https://scholar.google.com/scholar?q=Zheng%2C+R%2E%2C+W%2E+Shen%2C+Y%2E+Hua%2C+et+al%2E+Improving+generalization+of+alignment+with+human+preferences+through+group+invariant+learning%2E+arXiv+preprint+arXiv%3A2310%2E11971%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E11971%5D%28https%3A%2F%2Far)
35. Touvron, H., L. Martin, K. Stone, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
36. Chiang, W.-L., Z. Li, Z. Lin, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. [find](https://scholar.google.com/scholar?q=Chiang%2C+W%2E%2DL%2E%2C+Z%2E+Li%2C+Z%2E+Lin%2C+et+al%2E+Vicuna%3A+An+open%2Dsource+chatbot+impressing+gpt%2D4+with+90%25%2A+chatgpt+quality%2C+2023%2E)
37. Völske, M., M. Potthast, S. Syed, et al. Tl; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59–63. 2017. — [TL;DR: Mining Reddit to Learn Automatic Summarization](https://md.research.elamur.ai/free/en/papers/volske-2017-tl-dr-mining-reddit-learn/)
38. Köpf, A., Y. Kilcher, D. von Rütte, et al. Openassistant conversations–democratizing large language model alignment. arXiv preprint arXiv:2304.[07327, 2023](https://md.research.elamur.ai/free/en/papers/kopf-2023-openassistant-conversations-democratizing-large-language/).
39. Holtzman, A., J. Buys, L. Du, et al. [The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/), 2020.
40. Schulman, J., P. Moritz, S. Levine, et al. High-dimensional continuous control using generalized advantage estimation, 2018. [find](https://scholar.google.com/scholar?q=Schulman%2C+J%2E%2C+P%2E+Moritz%2C+S%2E+Levine%2C+et+al%2E+High%2Ddimensional+continuous+control+using+generalized+advantage+estimation%2C+2018%2E)
41. Chang, Y., X. Wang, J. Wang, et al. [A survey on evaluation of large language models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/). arXiv preprint arXiv:2307.03109, 2023.
42. Zheng, L., W.-L. Chiang, Y. Sheng, et al. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). arXiv preprint arXiv:2306.05685, 2023.
43. Xi, Z., S. Jin, Y. Zhou, et al. Self-polish: Enhance reasoning in large language models via problem refinement. arXiv preprint arXiv:2305.14497, 2023. train eval 0.9 0.8 Accuracy 0.7 0.6 0.5 4000 6000 Steps Figure 19: Left: training trajectory of the reward model. Right: reward scores of chosen and… — [arXiv:2305.14497](https://arxiv.org/abs/2305.14497) [find](https://scholar.google.com/scholar?q=Xi%2C+Z%2E%2C+S%2E+Jin%2C+Y%2E+Zhou%2C+et+al%2E+Self%2Dpolish%3A+Enhance+reasoning+in+large+language+models+via+problem+refinement%2E+arXiv+preprint+arXiv%3A2305%2E14497%2C+2023%2E+train+eval+0%2E9+0%2E8+Accuracy+0%2E7+0%2E6+0%2E5+4000+6000)

*Updated 2 November 2026*