---
title: "Preference Ranking Optimization for Human — Song 2023"
description: "Large language models are usually aligned with human values through RLHF, which is complex, unstable and sensitive to hyperparameters."
authors: ["Feifan Song", "Bowen Yu", "Minghao Li", "Haiyang Yu", "Fei Huang", "Yongbin Li", "Houfeng Wang"]
year: 2023
venue: "arXiv:2306.17492"
source: "https://arxiv.org/abs/2306.17492"
doi: "10.48550/arxiv.2306.17492"
arxiv: "2306.17492"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/song-2023-preference-ranking-optimization-human-alignment/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/song-2023-preference-ranking-optimization-human-alignment/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / Preference Ranking Optimization for Human Alignment

# Preference Ranking Optimization for Human Alignment

**Authors:** [Feifan Song](https://md.research.elamur.ai/free/en/authors/feifan-song/), [Bowen Yu](https://md.research.elamur.ai/free/en/authors/bowen-yu/), [Minghao Li](https://md.research.elamur.ai/free/en/authors/minghao-li/), [Haiyang Yu](https://md.research.elamur.ai/free/en/authors/haiyang-yu/), [Fei Huang](https://md.research.elamur.ai/free/en/authors/fei-huang/), [Yongbin Li](https://md.research.elamur.ai/free/en/authors/yongbin-li/), [Houfeng Wang](https://md.research.elamur.ai/free/en/authors/houfeng-wang/) · **Year:** 2023 · **Venue:** arXiv:2306.17492 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2306.17492) · [PDF](https://arxiv.org/pdf/2306.17492) · [DOI 10.48550/arxiv.2306.17492](https://doi.org/10.48550/arxiv.2306.17492) · [arXiv 2306.17492](https://arxiv.org/abs/2306.17492) · [OpenAlex](https://openalex.org/W4393160410)

**Citations:** 71 citations (openalex-snapshot · 2026-10-14) · [112](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Large language models are usually aligned with human values through RLHF, which is complex, unstable and sensitive to hyperparameters. Most supervised alternatives contrast responses only in pairs, even when longer human rankings are available. The authors propose Preference Ranking Optimization (PRO), a purely supervised fine-tuning objective in which the LLM serves as its own reward model, scoring each candidate by its length-normalised log-likelihood. Given n ranked responses to a prompt, PRO marks the top candidate as positive and every lower-ranked one as negative. It then drops that candidate and repeats down the list. This recursive one-to-N contrast resembles the Plackett-Luce ranking model, and an extra NLL term on the best response keeps generations fluent. Rankings can be lengthened cheaply by adding responses from other LLMs (Alpaca, ChatGPT) ordered by a reward model, or through self-bootstrapping. On HH-RLHF, PRO outperforms baselines, and longer, more varied rankings bring the largest gains, so a 7B model approaches ChatGPT's reward scores. The method offers a simpler and more stable route to alignment from ranked preference data.

## Key points
- **Objective:** L_PRO combines the ranking loss with β·L_SFT, where β balances preference fit against text quality. With n = 2, PRO reduces to a pairwise Bradley-Terry contrast.
- **Data:** Experiments use HH-RLHF. Some ranking variants are extended to length 3 with candidates from Alpaca or ChatGPT, which are ranked by a reward model.
- **Evaluation:** The authors evaluate with automatic metrics, reward-model scoring, GPT-4 judgments and human evaluation. They report results comparable to ChatGPT and to human-written responses using only 7B parameters.
- **Findings:** Gains grow as rankings get longer. Heterogeneous candidates help more than homogeneous ones, and candidate quality matters.
- **Ablation:** The study covers the ranking loss, L_SFT and a dynamic temperature T, which penalises near-positive negatives less. Removing both L_SFT and T causes the sharpest performance collapse. The paper appeared at AAAI 2024.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

112 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 104 not yet.

- [Qwen Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://md.research.elamur.ai/free/en/papers/dai-2023-safe-rlhf-safe-reinforcement-learning/)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://md.research.elamur.ai/free/en/papers/li-2023-api-bank-comprehensive-benchmark-tool/)
- [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/)
- [A Comprehensive Overview of Large Language Models](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/)
- [Aligning Large Language Models with Human: A Survey](https://md.research.elamur.ai/free/en/papers/wang-2023-aligning-large-language-models-human/)
- [Scaling Relationship on Learning Mathematical Reasoning with Large Language Models](https://md.research.elamur.ai/free/en/papers/yuan-2023-scaling-relationship-learning-mathematical-reasoning/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{song2023preference,
  title = {{Preference Ranking Optimization for Human Alignment}},
  author = {Song, Feifan and Yu, Bowen and Li, Minghao and Yu, Haiyang and Huang, Fei and Li, Yongbin and Wang, Houfeng},
  year = {2023},
  doi = {10.48550/arxiv.2306.17492},
  eprint = {2306.17492},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.17492}
}
```

- [← previous Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches](https://md.research.elamur.ai/free/en/papers/fried-2023-pragmatics-language-grounding-phenomena-tasks/)
- [next → PreWoMe: Exploiting Presuppositions as Working Memory for Long Form Question Answering](https://md.research.elamur.ai/free/en/papers/han-2023-prewome-exploiting-presuppositions-as-working/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

44 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bai, Y.; Jones, A.; Ndousse, K.; Askell, A.; Chen, A.; Das-Sarma, N.; Drain, D.; Fort, S.; Ganguli, D.; Henighan, T.; et al. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv:2204.05862. [find](https://scholar.google.com/scholar?q=Bai%2C+Y%2E%3B+Jones%2C+A%2E%3B+Ndousse%2C+K%2E%3B+Askell%2C+A%2E%3B+Chen%2C+A%2E%3B+Das%2DSarma%2C+N%2E%3B+Drain%2C+D%2E%3B+Fort%2C+S%2E%3B+Ganguli%2C+D%2E%3B+Henighan%2C+T%2E%3B+et+al%2E+2022a%2E+Training+a+helpful+and+harmless+assistant+with+reinforcement+learnin)
2. Bai, Y.; Kadavath, S.; Kundu, S.; Askell, A.; Kernion, J.; Jones, A.; Chen, A.; Goldie, A.; Mirhoseini, A.; McKinnon, C.; et al. 2022b. Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073. [find](https://scholar.google.com/scholar?q=Bai%2C+Y%2E%3B+Kadavath%2C+S%2E%3B+Kundu%2C+S%2E%3B+Askell%2C+A%2E%3B+Kernion%2C+J%2E%3B+Jones%2C+A%2E%3B+Chen%2C+A%2E%3B+Goldie%2C+A%2E%3B+Mirhoseini%2C+A%2E%3B+McKinnon%2C+C%2E%3B+et+al%2E+2022b%2E+Constitutional+AI%3A+Harmlessness+from+AI+Feedback%2E+arXiv%3A2212%2E080) [our card](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/)
3. Bradley, R. A.; and Terry, M. E. 1952. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4): 324–345. [find](https://scholar.google.com/scholar?q=Bradley%2C+R%2E+A%2E%3B+and+Terry%2C+M%2E+E%2E+1952%2E+Rank+analysis+of+incomplete+block+designs%3A+I%2E+The+method+of+paired+comparisons%2E+Biometrika%2C+39%283%2F4%29%3A+324%E2%80%93345%2E)
4. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877– 1901. [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E%3B+Mann%2C+B%2E%3B+Ryder%2C+N%2E%3B+Subbiah%2C+M%2E%3B+Kaplan%2C+J%2E+D%2E%3B+Dhariwal%2C+P%2E%3B+Neelakantan%2C+A%2E%3B+Shyam%2C+P%2E%3B+Sastry%2C+G%2E%3B+Askell%2C+A%2E%3B+et+al%2E+2020%2E+Language+models+are+few%2Dshot+learners%2E+Advances+in+neural+inf)
5. Bubeck, S.; Chandrasekaran, V.; Eldan, R.; Gehrke, J.; Horvitz, E.; Kamar, E.; Lee, P.; Lee, Y. T.; Li, Y.; Lundberg, S.; et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv:2303.12712. [find](https://scholar.google.com/scholar?q=Bubeck%2C+S%2E%3B+Chandrasekaran%2C+V%2E%3B+Eldan%2C+R%2E%3B+Gehrke%2C+J%2E%3B+Horvitz%2C+E%2E%3B+Kamar%2C+E%2E%3B+Lee%2C+P%2E%3B+Lee%2C+Y%2E+T%2E%3B+Li%2C+Y%2E%3B+Lundberg%2C+S%2E%3B+et+al%2E+2023%2E+Sparks+of+artificial+general+intelligence%3A+Early+experiments+with) [our card](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/)
6. Chowdhery, A.; Narang, S.; Devlin, J.; Bosma, M.; Mishra, G.; Roberts, A.; Barham, P.; Chung, H. W.; Sutton, C.; Gehrmann, S.; et al. 2022. Palm: Scaling language modeling with pathways. arXiv:2204.02311. [find](https://scholar.google.com/scholar?q=Chowdhery%2C+A%2E%3B+Narang%2C+S%2E%3B+Devlin%2C+J%2E%3B+Bosma%2C+M%2E%3B+Mishra%2C+G%2E%3B+Roberts%2C+A%2E%3B+Barham%2C+P%2E%3B+Chung%2C+H%2E+W%2E%3B+Sutton%2C+C%2E%3B+Gehrmann%2C+S%2E%3B+et+al%2E+2022%2E+Palm%3A+Scaling+language+modeling+with+pathways%2E+arXiv%3A2204%2E02)
7. Christiano, P. F.; Leike, J.; Brown, T.; Martic, M.; Legg, S.; and Amodei, D. 2017. Deep Reinforcement Learning from Human Preferences. In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., Advances in Neural Information Processing Systems, volum… [find](https://scholar.google.com/scholar?q=Christiano%2C+P%2E+F%2E%3B+Leike%2C+J%2E%3B+Brown%2C+T%2E%3B+Martic%2C+M%2E%3B+Legg%2C+S%2E%3B+and+Amodei%2C+D%2E+2017%2E+Deep+Reinforcement+Learning+from+Human+Preferences%2E+In+Guyon%2C+I%2E%3B+Luxburg%2C+U%2E+V%2E%3B+Bengio%2C+S%2E%3B+Wallach%2C+H%2E%3B+Fergus%2C+R)
8. Curran Associates, Inc. Dong, H.; Xiong, W.; Goyal, D.; Pan, R.; Diao, S.; Zhang, J.; Shum, K.; and Zhang, T. 2023. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv:2304.06767. [find](https://scholar.google.com/scholar?q=Curran+Associates%2C+Inc%2E+Dong%2C+H%2E%3B+Xiong%2C+W%2E%3B+Goyal%2C+D%2E%3B+Pan%2C+R%2E%3B+Diao%2C+S%2E%3B+Zhang%2C+J%2E%3B+Shum%2C+K%2E%3B+and+Zhang%2C+T%2E+2023%2E+Raft%3A+Reward+ranked+finetuning+for+generative+foundation+model+alignment%2E+arXiv%3A2304) [our card](https://md.research.elamur.ai/free/en/papers/dong-2023-raft-reward-ranked-finetuning-generative/)
9. Du, Z.; Qian, Y.; Liu, X.; Ding, M.; Qiu, J.; Yang, Z.; and Tang, J. 2022. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 320–335. Dublin, Ireland: Associ… [find](https://scholar.google.com/scholar?q=Du%2C+Z%2E%3B+Qian%2C+Y%2E%3B+Liu%2C+X%2E%3B+Ding%2C+M%2E%3B+Qiu%2C+J%2E%3B+Yang%2C+Z%2E%3B+and+Tang%2C+J%2E+2022%2E+GLM%3A+General+Language+Model+Pretraining+with+Autoregressive+Blank+Infilling%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the) [our card](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/)
10. Gao, L.; Schulman, J.; and Hilton, J. 2022. Scaling Laws for Reward Model Overoptimization. arXiv:2210.10760. [find](https://scholar.google.com/scholar?q=Gao%2C+L%2E%3B+Schulman%2C+J%2E%3B+and+Hilton%2C+J%2E+2022%2E+Scaling+Laws+for+Reward+Model+Overoptimization%2E+arXiv%3A2210%2E10760%2E)
11. Gugger, S.; Debut, L.; Wolf, T.; Schmid, P.; Mueller, Z.; and Mangrulkar, S. 2022. Accelerate: Training and inference at scale made simple, efficient and adaptable. [find](https://scholar.google.com/scholar?q=Gugger%2C+S%2E%3B+Debut%2C+L%2E%3B+Wolf%2C+T%2E%3B+Schmid%2C+P%2E%3B+Mueller%2C+Z%2E%3B+and+Mangrulkar%2C+S%2E+2022%2E+Accelerate%3A+Training+and+inference+at+scale+made+simple%2C+efficient+and+adaptable%2E)
12. He, K.; Fan, H.; Wu, Y.; Xie, S.; and Girshick, R. 2020. Momentum Contrast for Unsupervised Visual Representation Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [find](https://scholar.google.com/scholar?q=He%2C+K%2E%3B+Fan%2C+H%2E%3B+Wu%2C+Y%2E%3B+Xie%2C+S%2E%3B+and+Girshick%2C+R%2E+2020%2E+Momentum+Contrast+for+Unsupervised+Visual+Representation+Learning%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Rec) [our card](https://md.research.elamur.ai/free/en/papers/he-2019-momentum-contrast-unsupervised-visual-representation/)
13. Lee, K.; Smith, L.; and Abbeel, P. 2021. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv:2106.05091. — [arXiv:2106.05091](https://arxiv.org/abs/2106.05091) [find](https://scholar.google.com/scholar?q=Lee%2C+K%2E%3B+Smith%2C+L%2E%3B+and+Abbeel%2C+P%2E+2021%2E+Pebble%3A+Feedback%2Defficient+interactive+reinforcement+learning+via+relabeling+experience+and+unsupervised+pre%2Dtraining%2E+arXiv%3A2106%2E05091%2E+%E2%80%94+%5BarXiv%3A2106%2E05091%5D%28h)
14. Lei, W.; Zhang, Y.; Song, F.; Liang, H.; Mao, J.; Lv, J.; Yang, Z.; and Chua, T.-S. 2022. Interacting with Non-Cooperative User: A New Paradigm for Proactive Dialogue Policy. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22… [find](https://scholar.google.com/scholar?q=Lei%2C+W%2E%3B+Zhang%2C+Y%2E%3B+Song%2C+F%2E%3B+Liang%2C+H%2E%3B+Mao%2C+J%2E%3B+Lv%2C+J%2E%3B+Yang%2C+Z%2E%3B+and+Chua%2C+T%2E%2DS%2E+2022%2E+Interacting+with+Non%2DCooperative+User%3A+A+New+Paradigm+for+Proactive+Dialogue+Policy%2E+In+Proceedings+of+the+45t)
15. Li, M.; Song, F.; Yu, B.; Yu, H.; Li, Z.; Huang, F.; and Li, Y. 2023. Api-bank: A benchmark for tool-augmented llms. arXiv:2304.08244. — [arXiv:2304.08244](https://arxiv.org/abs/2304.08244) [find](https://scholar.google.com/scholar?q=Li%2C+M%2E%3B+Song%2C+F%2E%3B+Yu%2C+B%2E%3B+Yu%2C+H%2E%3B+Li%2C+Z%2E%3B+Huang%2C+F%2E%3B+and+Li%2C+Y%2E+2023%2E+Api%2Dbank%3A+A+benchmark+for+tool%2Daugmented+llms%2E+arXiv%3A2304%2E08244%2E+%E2%80%94+%5BarXiv%3A2304%2E08244%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2304%2E08244%29)
16. Liu, H.; Sferrazza, C.; and Abbeel, P. 2023. Chain of Hindsight aligns Language Models with Feedback. arXiv:2302.02676. [find](https://scholar.google.com/scholar?q=Liu%2C+H%2E%3B+Sferrazza%2C+C%2E%3B+and+Abbeel%2C+P%2E+2023%2E+Chain+of+Hindsight+aligns+Language+Models+with+Feedback%2E+arXiv%3A2302%2E02676%2E)
17. Luce, R. D. 2012. Individual choice behavior: A theoretical analysis. Courier Corporation. [find](https://scholar.google.com/scholar?q=Luce%2C+R%2E+D%2E+2012%2E+Individual+choice+behavior%3A+A+theoretical+analysis%2E+Courier+Corporation%2E)
18. MacGlashan, J.; Ho, M. K.; Loftin, R.; Peng, B.; Wang, G.; Roberts, D. L.; Taylor, M. E.; and Littman, M. L. 2017. Interactive Learning from Policy-Dependent Human Feedback. In Precup, D.; and Teh, Y. W., eds., Proceedings of the 34th International Conference on Machine Learning, volume 70 of Procee… [find](https://scholar.google.com/scholar?q=MacGlashan%2C+J%2E%3B+Ho%2C+M%2E+K%2E%3B+Loftin%2C+R%2E%3B+Peng%2C+B%2E%3B+Wang%2C+G%2E%3B+Roberts%2C+D%2E+L%2E%3B+Taylor%2C+M%2E+E%2E%3B+and+Littman%2C+M%2E+L%2E+2017%2E+Interactive+Learning+from+Policy%2DDependent+Human+Feedback%2E+In+Precup%2C+D%2E%3B+and+Teh%2C+Y%2E)
19. Nakano, R.; Hilton, J.; Balaji, S.; Wu, J.; Ouyang, L.; Kim, C.; Hesse, C.; Jain, S.; Kosaraju, V.; Saunders, W.; et al. 2021. Webgpt: Browser-assisted question-answering with human feedback. arXiv:2112.09332. [find](https://scholar.google.com/scholar?q=Nakano%2C+R%2E%3B+Hilton%2C+J%2E%3B+Balaji%2C+S%2E%3B+Wu%2C+J%2E%3B+Ouyang%2C+L%2E%3B+Kim%2C+C%2E%3B+Hesse%2C+C%2E%3B+Jain%2C+S%2E%3B+Kosaraju%2C+V%2E%3B+Saunders%2C+W%2E%3B+et+al%2E+2021%2E+Webgpt%3A+Browser%2Dassisted+question%2Danswering+with+human+feedback%2E+arXiv%3A21) [our card](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)
20. OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023%2E+GPT%2D4+Technical+Report%2E+arXiv%3A2303%2E08774%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
21. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730–27744. [find](https://scholar.google.com/scholar?q=Ouyang%2C+L%2E%3B+Wu%2C+J%2E%3B+Jiang%2C+X%2E%3B+Almeida%2C+D%2E%3B+Wainwright%2C+C%2E%3B+Mishkin%2C+P%2E%3B+Zhang%2C+C%2E%3B+Agarwal%2C+S%2E%3B+Slama%2C+K%2E%3B+Ray%2C+A%2E%3B+et+al%2E+2022%2E+Training+language+models+to+follow+instructions+with+human+feedback%2E+A)
22. Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 311–318. Philadelphia, Pennsylvania, USA: Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Papineni%2C+K%2E%3B+Roukos%2C+S%2E%3B+Ward%2C+T%2E%3B+and+Zhu%2C+W%2E%2DJ%2E+2002%2E+Bleu%3A+a+Method+for+Automatic+Evaluation+of+Machine+Translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+Association+for+Computational+) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
23. Peng, B.; Li, C.; He, P.; Galley, M.; and Gao, J. 2023. Instruction tuning with gpt-4. arXiv:2304.03277. [find](https://scholar.google.com/scholar?q=Peng%2C+B%2E%3B+Li%2C+C%2E%3B+He%2C+P%2E%3B+Galley%2C+M%2E%3B+and+Gao%2C+J%2E+2023%2E+Instruction+tuning+with+gpt%2D4%2E+arXiv%3A2304%2E03277%2E) [our card](https://md.research.elamur.ai/free/en/papers/peng-2023-instruction-tuning-gpt-4/)
24. Plackett, R. L. 1975. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2): 193–202. [find](https://scholar.google.com/scholar?q=Plackett%2C+R%2E+L%2E+1975%2E+The+analysis+of+permutations%2E+Journal+of+the+Royal+Statistical+Society+Series+C%3A+Applied+Statistics%2C+24%282%29%3A+193%E2%80%93202%2E)
25. Rafailov, R.; Sharma, A.; Mitchell, E.; Ermon, S.; Manning, C. D.; and Finn, C. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290. [find](https://scholar.google.com/scholar?q=Rafailov%2C+R%2E%3B+Sharma%2C+A%2E%3B+Mitchell%2C+E%2E%3B+Ermon%2C+S%2E%3B+Manning%2C+C%2E+D%2E%3B+and+Finn%2C+C%2E+2023%2E+Direct+Preference+Optimization%3A+Your+Language+Model+is+Secretly+a+Reward+Model%2E+arXiv%3A2305%2E18290%2E)
26. Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal policy optimization algorithms. arXiv:1707.06347. [find](https://scholar.google.com/scholar?q=Schulman%2C+J%2E%3B+Wolski%2C+F%2E%3B+Dhariwal%2C+P%2E%3B+Radford%2C+A%2E%3B+and+Klimov%2C+O%2E+2017%2E+Proximal+policy+optimization+algorithms%2E+arXiv%3A1707%2E06347%2E)
27. Snell, C.; Kostrikov, I.; Su, Y.; Yang, M.; and Levine, S. 2022. Offline rl for natural language generation with implicit language q learning. arXiv:2206.11871. — [arXiv:2206.11871](https://arxiv.org/abs/2206.11871) [find](https://scholar.google.com/scholar?q=Snell%2C+C%2E%3B+Kostrikov%2C+I%2E%3B+Su%2C+Y%2E%3B+Yang%2C+M%2E%3B+and+Levine%2C+S%2E+2022%2E+Offline+rl+for+natural+language+generation+with+implicit+language+q+learning%2E+arXiv%3A2206%2E11871%2E+%E2%80%94+%5BarXiv%3A2206%2E11871%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
28. Stiennon, N.; Ouyang, L.; Wu, J.; Ziegler, D.; Lowe, R.; Voss, C.; Radford, A.; Amodei, D.; and Christiano, P. F. 2020. Learning to summarize with human feedback. [find](https://scholar.google.com/scholar?q=Stiennon%2C+N%2E%3B+Ouyang%2C+L%2E%3B+Wu%2C+J%2E%3B+Ziegler%2C+D%2E%3B+Lowe%2C+R%2E%3B+Voss%2C+C%2E%3B+Radford%2C+A%2E%3B+Amodei%2C+D%2E%3B+and+Christiano%2C+P%2E+F%2E+2020%2E+Learning+to+summarize+with+human+feedback%2E)
29. Advances in Neural Information Processing Systems, 33: 3008–3021. Taori, R.; Gulrajani, I.; Zhang, T.; Dubois, Y.; Li, X.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. Stanford Alpaca: An Instruction-following LLaMA model. [find](https://scholar.google.com/scholar?q=Advances+in+Neural+Information+Processing+Systems%2C+33%3A+3008%E2%80%933021%2E+Taori%2C+R%2E%3B+Gulrajani%2C+I%2E%3B+Zhang%2C+T%2E%3B+Dubois%2C+Y%2E%3B+Li%2C+X%2E%3B+Guestrin%2C+C%2E%3B+Liang%2C+P%2E%3B+and+Hashimoto%2C+T%2E+B%2E+2023%2E+Stanford+Alpaca%3A+An+Instr)
30. Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023. Llama: Open and efficient foundation language models. arXiv:2302.13971. [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%3B+Lavril%2C+T%2E%3B+Izacard%2C+G%2E%3B+Martinet%2C+X%2E%3B+Lachaux%2C+M%2E%2DA%2E%3B+Lacroix%2C+T%2E%3B+Rozie%CC%80re%2C+B%2E%3B+Goyal%2C+N%2E%3B+Hambro%2C+E%2E%3B+Azhar%2C+F%2E%3B+et+al%2E+2023%2E+Llama%3A+Open+and+efficient+foundation+language+models%2E+arXi)
31. Wang, P.; Li, L.; Chen, L.; Zhu, D.; Lin, B.; Cao, Y.; Liu, Q.; Liu, T.; and Sui, Z. 2023. Large language models are not fair evaluators. arXiv:2305.17926. [find](https://scholar.google.com/scholar?q=Wang%2C+P%2E%3B+Li%2C+L%2E%3B+Chen%2C+L%2E%3B+Zhu%2C+D%2E%3B+Lin%2C+B%2E%3B+Cao%2C+Y%2E%3B+Liu%2C+Q%2E%3B+Liu%2C+T%2E%3B+and+Sui%2C+Z%2E+2023%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv%3A2305%2E17926%2E)
32. Wang, Y.; Kordi, Y.; Mishra, S.; Liu, A.; Smith, N. A.; Khashabi, D.; and Hajishirzi, H. 2022. Self-Instruct: Aligning Language Model with Self Generated Instructions. arXiv:2212.10560. [find](https://scholar.google.com/scholar?q=Wang%2C+Y%2E%3B+Kordi%2C+Y%2E%3B+Mishra%2C+S%2E%3B+Liu%2C+A%2E%3B+Smith%2C+N%2E+A%2E%3B+Khashabi%2C+D%2E%3B+and+Hajishirzi%2C+H%2E+2022%2E+Self%2DInstruct%3A+Aligning+Language+Model+with+Self+Generated+Instructions%2E+arXiv%3A2212%2E10560%2E)
33. Warnell, G.; Waytowich, N.; Lawhern, V.; and Stone, P. 2018. Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32. [find](https://scholar.google.com/scholar?q=Warnell%2C+G%2E%3B+Waytowich%2C+N%2E%3B+Lawhern%2C+V%2E%3B+and+Stone%2C+P%2E+2018%2E+Deep+TAMER%3A+Interactive+Agent+Shaping+in+High%2DDimensional+State+Spaces%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+v)
34. Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; Davison, J.; Shleifer, S.; von Platen, P.; Ma, C.; Jernite, Y.; Plu, J.; Xu, C.; Le Scao, T.; Gugger, S.; Drame, M.; Lhoest, Q.; and Rush, A. 2020. Transformers: State-of-the-Art Natur… [find](https://scholar.google.com/scholar?q=Wolf%2C+T%2E%3B+Debut%2C+L%2E%3B+Sanh%2C+V%2E%3B+Chaumond%2C+J%2E%3B+Delangue%2C+C%2E%3B+Moi%2C+A%2E%3B+Cistac%2C+P%2E%3B+Rault%2C+T%2E%3B+Louf%2C+R%2E%3B+Funtowicz%2C+M%2E%3B+Davison%2C+J%2E%3B+Shleifer%2C+S%2E%3B+von+Platen%2C+P%2E%3B+Ma%2C+C%2E%3B+Jernite%2C+Y%2E%3B+Plu%2C+J%2E%3B+Xu%2C+C%2E%3B+Le+) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
35. Wu, Z.; Hu, Y.; Shi, W.; Dziri, N.; Suhr, A.; Ammanabrolu, P.; Smith, N. A.; Ostendorf, M.; and Hajishirzi, H. 2023. [find](https://scholar.google.com/scholar?q=Wu%2C+Z%2E%3B+Hu%2C+Y%2E%3B+Shi%2C+W%2E%3B+Dziri%2C+N%2E%3B+Suhr%2C+A%2E%3B+Ammanabrolu%2C+P%2E%3B+Smith%2C+N%2E+A%2E%3B+Ostendorf%2C+M%2E%3B+and+Hajishirzi%2C+H%2E+2023%2E)
36. Fine-Grained Human Feedback Gives Better Rewards for Language Model Training. arXiv:2306.01693. [find](https://scholar.google.com/scholar?q=Fine%2DGrained+Human+Feedback+Gives+Better+Rewards+for+Language+Model+Training%2E+arXiv%3A2306%2E01693%2E) [our card](https://md.research.elamur.ai/free/en/papers/wu-2023-fine-grained-human-feedback-gives/)
37. Xue, W.; An, B.; Yan, S.; and Xu, Z. 2023. Reinforcement Learning from Diverse Human Preferences. arXiv:2301.11774. — [arXiv:2301.11774](https://arxiv.org/abs/2301.11774) [find](https://scholar.google.com/scholar?q=Xue%2C+W%2E%3B+An%2C+B%2E%3B+Yan%2C+S%2E%3B+and+Xu%2C+Z%2E+2023%2E+Reinforcement+Learning+from+Diverse+Human+Preferences%2E+arXiv%3A2301%2E11774%2E+%E2%80%94+%5BarXiv%3A2301%2E11774%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2301%2E11774%29)
38. Yuan, Z.; Yuan, H.; Tan, C.; Wang, W.; Huang, S.; and Huang, F. 2023. Rrhf: Rank responses to align language models with human feedback without tears. arXiv:2304.05302. [find](https://scholar.google.com/scholar?q=Yuan%2C+Z%2E%3B+Yuan%2C+H%2E%3B+Tan%2C+C%2E%3B+Wang%2C+W%2E%3B+Huang%2C+S%2E%3B+and+Huang%2C+F%2E+2023%2E+Rrhf%3A+Rank+responses+to+align+language+models+with+human+feedback+without+tears%2E+arXiv%3A2304%2E05302%2E) [our card](https://md.research.elamur.ai/free/en/papers/yuan-2023-rrhf-rank-responses-align-language/)
39. Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; et al. 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv:2306.05685. [find](https://scholar.google.com/scholar?q=Zheng%2C+L%2E%3B+Chiang%2C+W%2E%2DL%2E%3B+Sheng%2C+Y%2E%3B+Zhuang%2C+S%2E%3B+Wu%2C+Z%2E%3B+Zhuang%2C+Y%2E%3B+Lin%2C+Z%2E%3B+Li%2C+Z%2E%3B+Li%2C+D%2E%3B+Xing%2C+E%2E%3B+et+al%2E+2023%2E+Judging+LLM%2Das%2Da%2Djudge+with+MT%2DBench+and+Chatbot+Arena%2E+arXiv%3A2306%2E05685%2E)
40. Zhou, C.; Liu, P.; Xu, P.; Iyer, S.; Sun, J.; Mao, Y.; Ma, X.; Efrat, A.; Yu, P.; Yu, L.; et al. 2023. Lima: Less is more for alignment. arXiv:2305.11206. [find](https://scholar.google.com/scholar?q=Zhou%2C+C%2E%3B+Liu%2C+P%2E%3B+Xu%2C+P%2E%3B+Iyer%2C+S%2E%3B+Sun%2C+J%2E%3B+Mao%2C+Y%2E%3B+Ma%2C+X%2E%3B+Efrat%2C+A%2E%3B+Yu%2C+P%2E%3B+Yu%2C+L%2E%3B+et+al%2E+2023%2E+Lima%3A+Less+is+more+for+alignment%2E+arXiv%3A2305%2E11206%2E)
41. Zhu, B.; Jiao, J.; and Jordan, M. I. 2023. Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons. arXiv:2301.11270. — [arXiv:2301.11270](https://arxiv.org/abs/2301.11270) [find](https://scholar.google.com/scholar?q=Zhu%2C+B%2E%3B+Jiao%2C+J%2E%3B+and+Jordan%2C+M%2E+I%2E+2023%2E+Principled+Reinforcement+Learning+with+Human+Feedback+from+Pairwise+or+K%2Dwise+Comparisons%2E+arXiv%3A2301%2E11270%2E+%E2%80%94+%5BarXiv%3A2301%2E11270%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2301%2E)
42. Zhu, B.; Sharma, H.; Frujeri, F. V.; Dong, S.; Zhu, C.; Jordan, M. I.; and Jiao, J. 2023. Fine-Tuning Language Models with Advantage-Induced Policy Alignment. arXiv:2306.02231. — [arXiv:2306.02231](https://arxiv.org/abs/2306.02231) [find](https://scholar.google.com/scholar?q=Zhu%2C+B%2E%3B+Sharma%2C+H%2E%3B+Frujeri%2C+F%2E+V%2E%3B+Dong%2C+S%2E%3B+Zhu%2C+C%2E%3B+Jordan%2C+M%2E+I%2E%3B+and+Jiao%2C+J%2E+2023%2E+Fine%2DTuning+Language+Models+with+Advantage%2DInduced+Policy+Alignment%2E+arXiv%3A2306%2E02231%2E+%E2%80%94+%5BarXiv%3A2306%2E02231%5D%28ht)
43. Ziegler, D. M.; Stiennon, N.; Wu, J.; Brown, T. B.; Radford, A.; Amodei, D.; Christiano, P.; and Irving, G. 2019. [find](https://scholar.google.com/scholar?q=Ziegler%2C+D%2E+M%2E%3B+Stiennon%2C+N%2E%3B+Wu%2C+J%2E%3B+Brown%2C+T%2E+B%2E%3B+Radford%2C+A%2E%3B+Amodei%2C+D%2E%3B+Christiano%2C+P%2E%3B+and+Irving%2C+G%2E+2019%2E)
44. Fine-tuning language models from human preferences. arXiv:1909.08593. Qualitative Analysis of PRO [find](https://scholar.google.com/scholar?q=Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv%3A1909%2E08593%2E+Qualitative+Analysis+of+PRO)