---
title: "Statistical Rejection Sampling Improves Preference Optimization"
description: "The authors propose Statistical Rejection Sampling Optimization (RSO). 28 references, cited by 94 papers here."
authors: ["Tianqi Liu", "Yao Zhao", "Rishabh Joshi", "Misha Khalman", "Mohammad Saleh", "Peter J. Liu", "Jialu Liu"]
year: 2023
venue: "arXiv:2309.06657"
source: "https://arxiv.org/abs/2309.06657"
doi: "10.48550/arxiv.2309.06657"
arxiv: "2309.06657"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2023-statistical-rejection-sampling-improves-preference/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2023-statistical-rejection-sampling-improves-preference/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Statistical Rejection Sampling Improves Preference Optimization

# Statistical Rejection Sampling Improves Preference Optimization

**Authors:** [Tianqi Liu](https://md.research.elamur.ai/free/en/authors/tianqi-liu/), [Yao Zhao](https://md.research.elamur.ai/free/en/authors/yao-zhao/), [Rishabh Joshi](https://md.research.elamur.ai/free/en/authors/rishabh-joshi/), [Misha Khalman](https://md.research.elamur.ai/free/en/authors/misha-khalman/), [Mohammad Saleh](https://md.research.elamur.ai/free/en/authors/mohammad-saleh/), [Peter J. Liu](https://md.research.elamur.ai/free/en/authors/peter-j-liu/) and [Jialu Liu](https://md.research.elamur.ai/free/en/authors/jialu-liu/) · **Year:** 2023 · **Venue:** arXiv:2309.06657 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2309.06657) · [PDF](https://arxiv.org/pdf/2309.06657) · [DOI 10.48550/arxiv.2309.06657](https://doi.org/10.48550/arxiv.2309.06657) · [arXiv 2309.06657](https://arxiv.org/abs/2309.06657) · [OpenAlex](https://openalex.org/W4386793209)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [94](https://md.research.elamur.ai/free/en/papers/liu-2023-statistical-rejection-sampling-improves-preference/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper addresses a statistical gap in offline alignment methods. DPO and SLiC learn from preference pairs, but those pairs come from human-annotated data or the SFT policy, not from the optimal policy being estimated. As a result, their fitting is not a proper maximum-likelihood estimate. The authors propose Statistical Rejection Sampling Optimization (RSO). First, a pairwise T5-XXL reward-ranking model is trained on human preferences. Next, classical rejection sampling turns SFT candidates into samples that approximate the optimal policy. These samples are paired and labeled by the reward model, and the policy is then fitted with a classification loss. The paper also presents a unified view of existing methods: DPO amounts to logistic regression on preferences, while SLiC closely resembles a hinge-loss SVM. This yields a normalized hinge variant. RSO is evaluated on Reddit TL;DR summarization and AnthropicHH dialogue helpfulness. According to gold reward, LLM judges and human raters, it consistently beats DPO and SLiC. The work matters because it shows that an explicit reward model with careful sampling helps offline alignment, at little added cost.

## Key points
- The losses compared are sigmoid-norm (DPO-style), hinge (SLiC-style) and the new hinge-norm. The temperature γ is decoupled from the RLHF parameter β.
- The pairwise T5-XXL reward-ranking model reaches 73.23% validation accuracy on summarization and 69.75% on the AI assistant task. Pointwise rewards are derived by comparing each candidate against a random SFT baseline sequence.
- Best-of-N / top-k-over-N sampling, common in RLHF work, is shown to be a special case of RSO's rejection sampler. RSO additionally balances reward exploitation against staying close to the SFT policy, which limits reward hacking.
- In practice, 64 responses are sampled per prompt from the SFT policy. The sample-and-rank stage (rso-sample-rank) takes under 10% of total training time.
- Unlike PPO, which needs four networks, RSO trains only a policy network, and its sampling step is parallelizable across the whole dataset.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 353 works · 364,806 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu and Jialu Liu. 2023. Statistical Rejection Sampling Improves Preference Optimization. arXiv:2309.06657.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{liu2023statistical,
  title = {{Statistical Rejection Sampling Improves Preference Optimization}},
  author = {Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu and Jialu Liu},
  year = {2023},
  doi = {10.48550/arxiv.2309.06657},
  eprint = {2309.06657},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.06657}
}
```

- [← previous Copyright Violations and Large Language Models](https://md.research.elamur.ai/free/en/papers/karamolegkou-2023-copyright-violations-large-language-models/)
- [next → Rethinking Benchmark and Contamination for Language Models with Rephrased Samples](https://md.research.elamur.ai/free/en/papers/yang-2023-rethinking-benchmark-contamination-language-models/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

28 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403, 2023.
2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+Training+a+helpful+and+harmless+assistant+with+reinforcem)
3. Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. [find](https://scholar.google.com/scholar?q=Ralph+Allan+Bradley+and+Milton+E+Terry%2E+Rank+analysis+of+incomplete+block+designs%3A+I%2E+the+method+of+paired+comparisons%2E+Biometrika%2C+39%283%2F4%29%3A324%E2%80%93345%2C+1952%2E)
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
5. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. [Scaling instruction-finetuned language models](https://md.research.elamur.ai/free/en/papers/chung-2022-scaling-instruction-finetuned-language-models/). arXiv preprint arXiv:2210.11416, 2022.
6. Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. [Raft: Reward ranked finetuning for generative foundation model alignment](https://md.research.elamur.ai/free/en/papers/dong-2023-raft-reward-ranked-finetuning-generative/). arXiv preprint arXiv:2304.06767, 2023.
7. Leo Gao, John Schulman, and Jacob Hilton. [Scaling laws for reward model overoptimization](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/). In International Conference on Machine Learning, pp. 10835–10866. PMLR, 2023.
8. Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. [Reinforced self-training (rest) for language modeling](https://md.research.elamur.ai/free/en/papers/gulcehre-2023-reinforced-self-training-rest-language/). arXiv preprint arXiv:2308.08998, 2023.
9. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). Advances in neural information processing systems, 28, 2015.
10. Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023. [find](https://scholar.google.com/scholar?q=Harrison+Lee%2C+Samrat+Phatale%2C+Hassan+Mansoor%2C+Kellie+Lu%2C+Thomas+Mesnard%2C+Colton+Bishop%2C+Victor+Carbune%2C+and+Abhinav+Rastogi%2E+Rlaif%3A+Scaling+reinforcement+learning+from+human+feedback+with+ai+feedback%2E)
11. Radford M Neal. Slice sampling. The annals of statistics, 31(3):705–767, 2003. [find](https://scholar.google.com/scholar?q=Radford+M+Neal%2E+Slice+sampling%2E+The+annals+of+statistics%2C+31%283%29%3A705%E2%80%93767%2C+2003%2E)
12. R OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv, pp. 2303–08774, 2023.
13. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5, 2023. [find](https://scholar.google.com/scholar?q=Reiner+Pope%2C+Sholto+Douglas%2C+Aakanksha+Chowdhery%2C+Jacob+Devlin%2C+James+Bradbury%2C+Jonathan+Heek%2C+Kefan+Xiao%2C+Shivani+Agrawal%2C+and+Jeff+Dean%2E+Efficiently+scaling+transformer+inference%2E+Proceedings+of+Mac)
14. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). arXiv preprint arXiv:2305.18290, 2023.
15. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
16. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)
17. Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596–4604. PMLR, 2018. [find](https://scholar.google.com/scholar?q=Noam+Shazeer+and+Mitchell+Stern%2E+Adafactor%3A+Adaptive+learning+rates+with+sublinear+memory+cost%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+4596%E2%80%934604%2E+PMLR%2C+2018%2E)
18. Lei Shu, Liangchen Luo, Jayakumar Hoskere, Yun Zhu, Canoee Liu, Simon Tong, Jindong Chen, and Lei Meng. Rewritelm: An instruction-tuned large language model for text rewriting. arXiv preprint arXiv:2305.15685, 2023. [find](https://scholar.google.com/scholar?q=Lei+Shu%2C+Liangchen+Luo%2C+Jayakumar+Hoskere%2C+Yun+Zhu%2C+Canoee+Liu%2C+Simon+Tong%2C+Jindong+Chen%2C+and+Lei+Meng%2E+Rewritelm%3A+An+instruction%2Dtuned+large+language+model+for+text+rewriting%2E+arXiv+preprint+arXiv%3A23)
19. Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. [Preference ranking optimization for human alignment](https://md.research.elamur.ai/free/en/papers/song-2023-preference-ranking-optimization-human-alignment/). arXiv preprint arXiv:2306.17492, 2023.
20. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F+Christiano%2E+Learning+to+summarize+with+human+feedback%2E+Advances+in+Neural+Infor)
21. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+Dan+Bikel%2C+Lukas+Blecher%2C+Cristian+Canton+Fe)
22. Peiyi Wang, Lei Li, Liang Chen, Feifan Song, Binghuai Lin, Yunbo Cao, Tianyu Liu, and Zhifang Sui. Making large language models better reasoners with alignment. arXiv preprint arXiv:2309.02144, 2023. — [arXiv:2309.02144](https://arxiv.org/abs/2309.02144) [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Feifan+Song%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+Making+large+language+models+better+reasoners+with+alignment%2E+arXiv+preprint+arXiv%3A2309%2E02144%2C+2023%2E+)
23. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Bosma%2C+Vincent+Y+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+M+Dai%2C+and+Quoc+V+Le%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+arXiv+preprint+arXiv%3A2109%2E01652)
24. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. [Rrhf: Rank responses to align language models with human feedback without tears](https://md.research.elamur.ai/free/en/papers/yuan-2023-rrhf-rank-responses-align-language/). arXiv preprint arXiv:2304.05302, 2023.
25. Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. Calibrating sequence likelihood improves conditional language generation. arXiv preprint arXiv:2210.00045, 2022. — [arXiv:2210.00045](https://arxiv.org/abs/2210.00045) [find](https://scholar.google.com/scholar?q=Yao+Zhao%2C+Misha+Khalman%2C+Rishabh+Joshi%2C+Shashi+Narayan%2C+Mohammad+Saleh%2C+and+Peter+J+Liu%2E+Calibrating+sequence+likelihood+improves+conditional+language+generation%2E+arXiv+preprint+arXiv%3A2210%2E00045%2C+2022)
26. Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. [Slic-hf: Sequence likelihood calibration with human feedback](https://md.research.elamur.ai/free/en/papers/zhao-2023-slic-hf-sequence-likelihood-calibration/). arXiv preprint arXiv:2305.10425, 2023.
27. Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond one-preference-for-all: Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03708, 2023. — [arXiv:2310.03708](https://arxiv.org/abs/2310.03708) [find](https://scholar.google.com/scholar?q=Zhanhui+Zhou%2C+Jie+Liu%2C+Chao+Yang%2C+Jing+Shao%2C+Yu+Liu%2C+Xiangyu+Yue%2C+Wanli+Ouyang%2C+and+Yu+Qiao%2E+Beyond+one%2Dpreference%2Dfor%2Dall%3A+Multi%2Dobjective+direct+preference+optimization%2E+arXiv+preprint+arXiv%3A2310%2E03)
28. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. [find](https://scholar.google.com/scholar?q=Daniel+M+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint+arXiv%3A19)

*Updated 24 October 2026*