---
title: "Arithmetic Control of LLMs for Diverse User Preferences:"
description: "The authors propose Directional Preference Alignment (DPA) to address this. 84 references, cited by 59 papers here."
authors: ["Haoxiang Wang", "Yong Lin", "Wei Xiong", "Rui Yang", "Shizhe Diao", "Shuang Qiu", "Han Zhao", "Tong Zhang"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.468/"
doi: "10.18653/v1/2024.acl-long.468"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2024-arithmetic-control-llms-diverse-user/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2024-arithmetic-control-llms-diverse-user/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards

# Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards

**Authors:** [Haoxiang Wang](https://md.research.elamur.ai/free/en/authors/haoxiang-wang/), [Yong Lin](https://md.research.elamur.ai/free/en/authors/yong-lin/), [Wei Xiong](https://md.research.elamur.ai/free/en/authors/wei-xiong/), [Rui Yang](https://md.research.elamur.ai/free/en/authors/rui-yang/), [Shizhe Diao](https://md.research.elamur.ai/free/en/authors/shizhe-diao/), Shuang Qiu, [Han Zhao](https://md.research.elamur.ai/free/en/authors/han-zhao/) and [Tong Zhang](https://md.research.elamur.ai/free/en/authors/tong-zhang/) · **Year:** 2024 · **Venue:** [Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.468/) · [PDF](https://aclanthology.org/2024.acl-long.468.pdf) · [DOI 10.18653/v1/2024.acl-long.468](https://doi.org/10.18653/v1/2024.acl-long.468) · [OpenAlex](https://openalex.org/W4402671571)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [59](https://md.research.elamur.ai/free/en/papers/wang-2024-arithmetic-control-llms-diverse-user/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Standard RLHF compresses human judgments into one scalar reward. This pushes a model toward an "average user" and ignores people whose tastes conflict, such as those who want useful but short answers. The authors propose Directional Preference Alignment (DPA) to address this.

First, a multi-objective reward model is trained with a regression loss. It predicts a separate rating for each attribute, such as helpfulness and verbosity. A user's preference is then expressed as a unit vector in this reward space, and the target reward is the dot product of that vector with the attribute ratings.

The policy is trained with a preference-conditioned, iterative form of Rejection Sampling Finetuning (RSF), the RLHF method used for Llama 2. Training repeatedly samples directions, places them in the system prompt, picks the best-of-n response under the weighted reward and fine-tunes on it.

Applied to Mistral-7B, a single model lets users adjust the balance between helpfulness and verbosity with simple arithmetic, and it remains competitive with DPO. This removes the need to keep and switch between separate models for different user preferences.

## Key points
- User preferences are unit vectors v on the sphere (‖v‖₂ = 1). The reward to maximize is v₁·helpfulness + v₂·verbosity, and v is given numerically in the system prompt.
- In the paper's example, setting v = ⟨0.8, −0.6⟩ shortens a "Romeo and Juliet" summary from 869 characters (DPO) to 140 characters.
- Alignment uses iterative RSF with best-of-n selection rather than PPO, a choice the authors justify by RSF's greater simplicity and stability.
- Experiments on Mistral-7B show a better helpfulness–verbosity trade-off (Pareto front) than strong baselines such as DPO.
- Code and the trained model are released at github.com/RLHFlow/directional-preference-alignment.
- The main stated limitation is dependence on the reward model: its biases, or failure to detect harmful content, can be amplified during fine-tuning.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Illinois Urbana-Champaign](https://md.research.elamur.ai/free/en/universities/university-of-illinois-urbana-champaign/) — University, Urbana, United States · 108 works · 67,432 citations
- [Hong Kong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/hong-kong-university-of-science-and-technology/) — University, Hong Kong, China · 58 works · 23,058 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works

## Cite

Cite as: Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao and Tong Zhang. 2024. Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2024arithmetic,
  title = {{Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards}},
  author = {Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao and Tong Zhang},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.468},
  url = {https://doi.org/10.18653/v1/2024.acl-long.468}
}
```

- [← previous Language Models as Inductive Reasoners](https://md.research.elamur.ai/free/en/papers/yang-2024-language-models-as-inductive-reasoners/)
- [next → Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Models](https://md.research.elamur.ai/free/en/papers/amayuelas-2024-knowledge-knowledge-exploring-known-unknowns/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

84 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rich Caruana. 1997. Multitask learning. Machine learning, 28:41–75. [find](https://scholar.google.com/scholar?q=Rich+Caruana%2E+1997%2E+Multitask+learning%2E+Machine+learning%2C+28%3A41%E2%80%9375%2E)
2. Anthropic. 2023. Introducing claude. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2023%2E+Introducing+claude%2E)
3. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861. [find](https://scholar.google.com/scholar?q=Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Andy+Jones%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Nova+DasSarma%2C+et+al%2E+2021%2E+A+general+language+assistant+as+a+laboratory+for+alignm)
4. Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. 2023. [A general theoretical paradigm to understand learning from human preferences](https://md.research.elamur.ai/free/en/papers/azar-2023-general-theoretical-paradigm-understand-learning/). arXiv preprint arXiv:2310.12036.
5. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. 2023. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217. [find](https://scholar.google.com/scholar?q=Stephen+Casper%2C+Xander+Davies%2C+Claudia+Shi%2C+Thomas+Krendl+Gilbert%2C+J%C3%A9r%C3%A9my+Scheurer%2C+Javier+Rando%2C+Rachel+Freedman%2C+Tomasz+Korbak%2C+David+Lindner%2C+Pedro+Freire%2C+et+al%2E+2023%2E+Open+problems+and+fundamenta)
6. Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Odin: Disentangled reward mitigates hacking in rlhf. [find](https://scholar.google.com/scholar?q=Lichang+Chen%2C+Chen+Zhu%2C+Davit+Soselia%2C+Jiuhai+Chen%2C+Tianyi+Zhou%2C+Tom+Goldstein%2C+Heng+Huang%2C+Mohammad+Shoeybi%2C+and+Bryan+Catanzaro%2E+2024%2E+Odin%3A+Disentangled+reward+mitigates+hacking+in+rlhf%2E)
7. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+2022a%2E+Training+a+helpful+and+harmless+assistant+with+rei)
8. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+and+Eric+P%2E+Xing%2E+2023%2E+Vicuna%3A+An+open%2Dsource+cha)
9. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022b. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073.
10. Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. 2019. On the weaknesses of reinforcement learning for neural machine translation. arXiv preprint arXiv:1907.01752. — [arXiv:1907.01752](https://arxiv.org/abs/1907.01752) [find](https://scholar.google.com/scholar?q=Leshem+Choshen%2C+Lior+Fox%2C+Zohar+Aizenbud%2C+and+Omri+Abend%2E+2019%2E+On+the+weaknesses+of+reinforcement+learning+for+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1907%2E01752%2E+%E2%80%94+%5BarXiv%3A1907%2E01752%5D%28https%3A)
11. Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et al. 2022. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Proc… [find](https://scholar.google.com/scholar?q=Michiel+Bakker%2C+Martin+Chadwick%2C+Hannah+Sheahan%2C+Michael+Tessler%2C+Lucy+Campbell%2DGillingham%2C+Jan+Balaguer%2C+Nat+McAleese%2C+Amelia+Glaese%2C+John+Aslanides%2C+Matt+Botvinick%2C+et+al%2E+2022%2E+Fine%2Dtuning+language)
12. Hritik Bansal, John Dang, and Aditya Grover. 2023. Peering through preferences: Unraveling feedback acquisition for aligning large language models. arXiv preprint arXiv:2308.15812. — [arXiv:2308.15812](https://arxiv.org/abs/2308.15812) [find](https://scholar.google.com/scholar?q=Hritik+Bansal%2C+John+Dang%2C+and+Aditya+Grover%2E+2023%2E+Peering+through+preferences%3A+Unraveling+feedback+acquisition+for+aligning+large+language+models%2E+arXiv+preprint+arXiv%3A2308%2E15812%2E+%E2%80%94+%5BarXiv%3A2308%2E15812)
13. Erdem Biyik and Dorsa Sadigh. 2018. Batch active preference-based learning of reward functions. In Conference on robot learning, pages 519–528. PMLR. [find](https://scholar.google.com/scholar?q=Erdem+Biyik+and+Dorsa+Sadigh%2E+2018%2E+Batch+active+preference%2Dbased+learning+of+reward+functions%2E+In+Conference+on+robot+learning%2C+pages+519%E2%80%93528%2E+PMLR%2E)
14. Andreea Bobu, Andi Peng, Pulkit Agrawal, Julie Shah, and Anca D Dragan. 2023. Aligning robot and human representations. arXiv preprint arXiv:2302.01928. — [arXiv:2302.01928](https://arxiv.org/abs/2302.01928) [find](https://scholar.google.com/scholar?q=Andreea+Bobu%2C+Andi+Peng%2C+Pulkit+Agrawal%2C+Julie+Shah%2C+and+Anca+D+Dragan%2E+2023%2E+Aligning+robot+and+human+representations%2E+arXiv+preprint+arXiv%3A2302%2E01928%2E+%E2%80%94+%5BarXiv%3A2302%2E01928%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2302)
15. Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. 2019. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning, pages 783–792. PMLR. [find](https://scholar.google.com/scholar?q=Daniel+Brown%2C+Wonjoon+Goo%2C+Prabhat+Nagarajan%2C+and+Scott+Niekum%2E+2019%2E+Extrapolating+beyond+suboptimal+demonstrations+via+inverse+reinforcement+learning+from+observations%2E+In+International+conference+o)
16. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
17. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research, 24(240):1–113.
18. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30. [find](https://scholar.google.com/scholar?q=Paul+F+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+2017%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+neural+information+processing+systems%2C+30%2E)
19. CA Coello Coello. 2000. Handling preferences in evolutionary multiobjective optimization: A survey. In Proceedings of the 2000 congress on evolutionary computation. CEC00 (Cat. No. 00TH8512), volume 1, pages 30–37. IEEE. [find](https://scholar.google.com/scholar?q=CA+Coello+Coello%2E+2000%2E+Handling+preferences+in+evolutionary+multiobjective+optimization%3A+A+survey%2E+In+Proceedings+of+the+2000+congress+on+evolutionary+computation%2E+CEC00+%28Cat%2E+No%2E+00TH8512%29%2C+volume+1)
20. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. [find](https://scholar.google.com/scholar?q=Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Guanming+Yao%2C+Wei+Zhu%2C+Yuan+Ni%2C+Guotong+Xie%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2023%2E+Ultrafeedback%3A+Boosting+language+models+with+high%2Dquality+feedback%2E)
21. Shizhe Diao, Rui Pan, Hanze Dong, Ka Shun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang. 2023. Lmflow: An extensible toolkit for finetuning and inference of large foundation models. arXiv preprint arXiv:2306.12420. [find](https://scholar.google.com/scholar?q=Shizhe+Diao%2C+Rui+Pan%2C+Hanze+Dong%2C+Ka+Shun+Shum%2C+Jipeng+Zhang%2C+Wei+Xiong%2C+and+Tong+Zhang%2E+2023%2E+Lmflow%3A+An+extensible+toolkit+for+finetuning+and+inference+of+large+foundation+models%2E+arXiv+preprint+arX)
22. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. [Enhancing chat language models by scaling high-quality instructional conversations](https://md.research.elamur.ai/free/en/papers/ding-2023-enhancing-chat-language-models-scaling/). arXiv preprint arXiv:2305.14233.
23. Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang. 2023a. 8651 10 [find](https://scholar.google.com/scholar?q=Hanze+Dong%2C+Wei+Xiong%2C+Deepanshu+Goyal%2C+Yihan+Zhang%2C+Winnie+Chow%2C+Rui+Pan%2C+Shizhe+Diao%2C+Jipeng+Zhang%2C+KaShun+SHUM%2C+and+Tong+Zhang%2E+2023a%2E+8651+10)
24. [RAFT: Reward ranked finetuning for generative foundation model alignment](https://md.research.elamur.ai/free/en/papers/dong-2023-raft-reward-ranked-finetuning-generative/). Transactions on Machine Learning Research.
25. Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xianchao Wu, and Oleksii Kuchaiev. 2023b. Steerlm: Attribute conditioned sft as an (user-steerable) alternative to rlhf. arXiv preprint arXiv:2310.05344. [find](https://scholar.google.com/scholar?q=Yi+Dong%2C+Zhilin+Wang%2C+Makesh+Narsimhan+Sreedhar%2C+Xianchao+Wu%2C+and+Oleksii+Kuchaiev%2E+2023b%2E+Steerlm%3A+Attribute+conditioned+sft+as+an+%28user%2Dsteerable%29+alternative+to+rlhf%2E+arXiv+preprint+arXiv%3A2310%2E0534)
26. Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. 2020. Implementation matters in deep policy gradients: A case study on ppo and trpo. arXiv preprint arXiv:2005.12729. [find](https://scholar.google.com/scholar?q=Logan+Engstrom%2C+Andrew+Ilyas%2C+Shibani+Santurkar%2C+Dimitris+Tsipras%2C+Firdaus+Janoos%2C+Larry+Rudolph%2C+and+Aleksander+Madry%2E+2020%2E+Implementation+matters+in+deep+policy+gradients%3A+A+case+study+on+ppo+and+t)
27. Michael Feffer, Hoda Heidari, and Zachary C Lipton. 2023. Moral machine or tyranny of the majority? arXiv preprint arXiv:2305.17319. — [arXiv:2305.17319](https://arxiv.org/abs/2305.17319) [find](https://scholar.google.com/scholar?q=Michael+Feffer%2C+Hoda+Heidari%2C+and+Zachary+C+Lipton%2E+2023%2E+Moral+machine+or+tyranny+of+the+majority%3F+arXiv+preprint+arXiv%3A2305%2E17319%2E+%E2%80%94+%5BarXiv%3A2305%2E17319%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E17319%29)
28. Peter C Fishburn. 1984. Probabilistic social choice based on simple voting comparisons. The Review of Economic Studies, 51(4):683–692. [find](https://scholar.google.com/scholar?q=Peter+C+Fishburn%2E+1984%2E+Probabilistic+social+choice+based+on+simple+voting+comparisons%2E+The+Review+of+Economic+Studies%2C+51%284%29%3A683%E2%80%93692%2E)
29. William V Gehrlein. 2002. Condorcet’s paradox and the likelihood of its occurrence: different perspectives on balanced preferences. Theory and decision, 52:171– 199. [find](https://scholar.google.com/scholar?q=William+V+Gehrlein%2E+2002%2E+Condorcet%E2%80%99s+paradox+and+the+likelihood+of+its+occurrence%3A+different+perspectives+on+balanced+preferences%2E+Theory+and+decision%2C+52%3A171%E2%80%93+199%2E)
30. A Ghane-Kanafi and E Khorram. 2015. A new scalarization method for finding the efficient frontier in non-convex multi-objective problems. Applied Mathematical Modelling, 39(23-24):7483–7498. [find](https://scholar.google.com/scholar?q=A+Ghane%2DKanafi+and+E+Khorram%2E+2015%2E+A+new+scalarization+method+for+finding+the+efficient+frontier+in+non%2Dconvex+multi%2Dobjective+problems%2E+Applied+Mathematical+Modelling%2C+39%2823%2D24%29%3A7483%E2%80%937498%2E)
31. Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. 2023. [Reinforced self-training (rest) for language modeling](https://md.research.elamur.ai/free/en/papers/gulcehre-2023-reinforced-self-training-rest-language/). arXiv preprint arXiv:2308.08998.
32. Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. 2022. Optimizing prompts for text-to-image generation. arXiv preprint arXiv:2212.09611. — [arXiv:2212.09611](https://arxiv.org/abs/2212.09611) [find](https://scholar.google.com/scholar?q=Yaru+Hao%2C+Zewen+Chi%2C+Li+Dong%2C+and+Furu+Wei%2E+2022%2E+Optimizing+prompts+for+text%2Dto%2Dimage+generation%2E+arXiv+preprint+arXiv%3A2212%2E09611%2E+%E2%80%94+%5BarXiv%3A2212%2E09611%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2212%2E09611%29)
33. Yuzheng Hu, Ruicheng Xian, Qilong Wu, Qiuling Fan, Lang Yin, and Han Zhao. 2023. Revisiting scalarization in multi-task learning: A theoretical perspective. In Thirty-seventh Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Yuzheng+Hu%2C+Ruicheng+Xian%2C+Qilong+Wu%2C+Qiuling+Fan%2C+Lang+Yin%2C+and+Han+Zhao%2E+2023%2E+Revisiting+scalarization+in+multi%2Dtask+learning%3A+A+theoretical+perspective%2E+In+Thirty%2Dseventh+Conference+on+Neural+Info)
34. Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. 2023. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv:2310.11564. [find](https://scholar.google.com/scholar?q=Joel+Jang%2C+Seungone+Kim%2C+Bill+Yuchen+Lin%2C+Yizhong+Wang%2C+Jack+Hessel%2C+Luke+Zettlemoyer%2C+Hannaneh+Hajishirzi%2C+Yejin+Choi%2C+and+Prithviraj+Ammanabrolu%2E+2023%2E+Personalized+soups%3A+Personalized+large+languag)
35. Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. [Beavertails: Towards improved safety alignment of LLM via a human-preference dataset](https://md.research.elamur.ai/free/en/papers/ji-2023-beavertails-towards-improved-safety-alignment/). In Thirty-seventh Conference on Neural Information Processing Systems Datasets and…
36. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023%2E+Mist)
37. Samia Kabir, David N Udo-Imeh, Bonan Kou, and Tianyi Zhang. 2023. Who answers it better? an in-depth analysis of chatgpt and stack overflow answers to software engineering questions. arXiv preprint arXiv:2308.02312. — [arXiv:2308.02312](https://arxiv.org/abs/2308.02312) [find](https://scholar.google.com/scholar?q=Samia+Kabir%2C+David+N+Udo%2DImeh%2C+Bonan+Kou%2C+and+Tianyi+Zhang%2E+2023%2E+Who+answers+it+better%3F+an+in%2Ddepth+analysis+of+chatgpt+and+stack+overflow+answers+to+software+engineering+questions%2E+arXiv+preprint+ar)
38. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Minh Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Alexandrovich Glushkov, Arnav Varma Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alex… — [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://md.research.elamur.ai/free/en/papers/kopf-2023-openassistant-conversations-democratizing-large-language/)
39. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
40. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2024. Rewardbench: Evaluating reward models for language modeling. [find](https://scholar.google.com/scholar?q=Nathan+Lambert%2C+Valentina+Pyatkin%2C+Jacob+Morrison%2C+LJ+Miranda%2C+Bill+Yuchen+Lin%2C+Khyathi+Chandu%2C+Nouha+Dziri%2C+Sachin+Kumar%2C+Tom+Zick%2C+Yejin+Choi%2C+Noah+A%2E+Smith%2C+and+Hannaneh+Hajishirzi%2E+2024%2E+Rewardben)
41. Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. 2023. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267. [find](https://scholar.google.com/scholar?q=Harrison+Lee%2C+Samrat+Phatale%2C+Hassan+Mansoor%2C+Kellie+Lu%2C+Thomas+Mesnard%2C+Colton+Bishop%2C+Victor+Carbune%2C+and+Abhinav+Rastogi%2E+2023%2E+Rlaif%3A+Scaling+reinforcement+learning+from+human+feedback+with+ai+fee)
42. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval). [↗](https://github.com/tatsu-lab/alpaca_eval)
43. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+et+al%2E+2022%2E+Holistic+evaluation+of+language+models%2E+)
44. Yong Lin, Lu Tan, Hangyu Lin, Zeming Zheng, Renjie Pi, Jipeng Zhang, Shizhe Diao, Haoxiang Wang, Han Zhao, Yuan Yao, et al. 2023. Speciality vs generality: An empirical study on catastrophic forgetting in fine-tuning foundation models. arXiv preprint arXiv:2309.06256. — [arXiv:2309.06256](https://arxiv.org/abs/2309.06256) [find](https://scholar.google.com/scholar?q=Yong+Lin%2C+Lu+Tan%2C+Hangyu+Lin%2C+Zeming+Zheng%2C+Renjie+Pi%2C+Jipeng+Zhang%2C+Shizhe+Diao%2C+Haoxiang+Wang%2C+Han+Zhao%2C+Yuan+Yao%2C+et+al%2E+2023%2E+Speciality+vs+generality%3A+An+empirical+study+on+catastrophic+forgettin)
45. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. 8652 11 [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+International+Conference+on+Learning+Representations%2E+8652+11)
46. Kenneth O May. 1954. Intransitivity, utility, and the aggregation of preference patterns. Econometrica: Journal of the Econometric Society, pages 1–13. [find](https://scholar.google.com/scholar?q=Kenneth+O+May%2E+1954%2E+Intransitivity%2C+utility%2C+and+the+aggregation+of+preference+patterns%2E+Econometrica%3A+Journal+of+the+Econometric+Society%2C+pages+1%E2%80%9313%2E)
47. Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. 2023. Nash learning from human feedback. arXiv preprint arXiv:2312.00886. [find](https://scholar.google.com/scholar?q=R%C3%A9mi+Munos%2C+Michal+Valko%2C+Daniele+Calandriello%2C+Mohammad+Gheshlaghi+Azar%2C+Mark+Rowland%2C+Zhaohan+Daniel+Guo%2C+Yunhao+Tang%2C+Matthieu+Geist%2C+Thomas+Mesnard%2C+Andrea+Michi%2C+et+al%2E+2023%2E+Nash+learning+from+h)
48. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). abs/2303.08774.
49. ArXiv, Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744.
50. Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Hanlin Zhang, Scott Emmons, and Dan Hendrycks. 2023. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark. In International Conference on Machine… [find](https://scholar.google.com/scholar?q=Alexander+Pan%2C+Jun+Shern+Chan%2C+Andy+Zou%2C+Nathaniel+Li%2C+Steven+Basart%2C+Thomas+Woodside%2C+Hanlin+Zhang%2C+Scott+Emmons%2C+and+Dan+Hendrycks%2E+2023%2E+Do+the+rewards+justify+the+means%3F+measuring+trade%2Doffs+betwe)
51. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. [Pytorch: An imperative style, high-performance deep learning library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/). Advances in neural information processing systems, 32.
52. Bruno L Pereira, Alberto Ueda, Gustavo Penha, Rodrygo LT Santos, and Nivio Ziviani. 2019. Online learning to rank for sequential music recommendation. In Proceedings of the 13th ACM Conference on Recommender Systems, pages 237–245. [find](https://scholar.google.com/scholar?q=Bruno+L+Pereira%2C+Alberto+Ueda%2C+Gustavo+Penha%2C+Rodrygo+LT+Santos%2C+and+Nivio+Ziviani%2E+2019%2E+Online+learning+to+rank+for+sequential+music+recommendation%2E+In+Proceedings+of+the+13th+ACM+Conference+on+Reco)
53. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). arXiv preprint arXiv:2305.18290.
54. Alexandre Rame, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier, and Matthieu Cord. 2023. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. arXiv preprint arXiv:2306.04488. [find](https://scholar.google.com/scholar?q=Alexandre+Rame%2C+Guillaume+Couairon%2C+Mustafa+Shukor%2C+Corentin+Dancette%2C+Jean%2DBaptiste+Gaya%2C+Laure+Soulier%2C+and+Matthieu+Cord%2E+2023%2E+Rewarded+soups%3A+towards+pareto%2Doptimal+alignment+by+interpolating+wei)
55. Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. 2023. Verbosity bias in preference labeling by large language models. arXiv preprint arXiv:2310.10076. — [arXiv:2310.10076](https://arxiv.org/abs/2310.10076) [find](https://scholar.google.com/scholar?q=Keita+Saito%2C+Akifumi+Wachi%2C+Koki+Wataoka%2C+and+Youhei+Akimoto%2E+2023%2E+Verbosity+bias+in+preference+labeling+by+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E10076%2E+%E2%80%94+%5BarXiv%3A2310%2E10076%5D%28https%3A%2F%2Farxiv%2E)
56. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+2017%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2E)
57. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. 2023a. [Towards expert-level medical question answering with large language models](https://md.research.elamur.ai/free/en/papers/singhal-2023-towards-expert-level-medical-question/). arXiv preprint arXiv:2305.09617.
58. Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2023b. [A long way to go: Investigating length correlations in rlhf](https://md.research.elamur.ai/free/en/papers/singhal-2023-long-way-go-investigating-length/). arXiv preprint arXiv:2310.03716.
59. Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022. [Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model](https://md.research.elamur.ai/free/en/papers/smith-2022-using-deepspeed-megatron-train-megatron/). arXiv preprint ar…
60. Saul H Sternberg. 1965. Mathematics and Social Sciences: Proceedings of the Seminars of Menthon-Saint-Bernard, France (1-27 July, 1960) and of Gösing, Austria (3-27 July, 1961), volume 1. Mouton. [find](https://scholar.google.com/scholar?q=Saul+H+Sternberg%2E+1965%2E+Mathematics+and+Social+Sciences%3A+Proceedings+of+the+Seminars+of+Menthon%2DSaint%2DBernard%2C+France+%281%2D27+July%2C+1960%29+and+of+G%C3%B6sing%2C+Austria+%283%2D27+July%2C+1961%29%2C+volume+1%2E+Mouton%2E)
61. Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. 2024. A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint arXiv:2401.04056. — [arXiv:2401.04056](https://arxiv.org/abs/2401.04056) [find](https://scholar.google.com/scholar?q=Gokul+Swamy%2C+Christoph+Dann%2C+Rahul+Kidambi%2C+Zhiwei+Steven+Wu%2C+and+Alekh+Agarwal%2E+2024%2E+A+minimaximalist+approach+to+reinforcement+learning+from+human+feedback%2E+arXiv+preprint+arXiv%3A2401%2E04056%2E+%E2%80%94+%5BarXi)
62. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. [https://crfm.stanford](https://crfm.stanford). edu/2023/03/13/alpaca. html, 3(6):… [↗](https://crfm.stanford)
63. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805.
64. Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine. Nature medicine, 29(8):1930– 1940. [find](https://scholar.google.com/scholar?q=Arun+James+Thirunavukarasu%2C+Darren+Shu+Jeng+Ting%2C+Kabilan+Elangovan%2C+Laura+Gutierrez%2C+Ting+Fang+Tan%2C+and+Daniel+Shu+Wei+Ting%2E+2023%2E+Large+language+models+in+medicine%2E+Nature+medicine%2C+29%288%29%3A1930%E2%80%93+1940)
65. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fi)
66. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. 2023. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944. 8653 12 [find](https://scholar.google.com/scholar?q=Lewis+Tunstall%2C+Edward+Beeching%2C+Nathan+Lambert%2C+Nazneen+Rajani%2C+Kashif+Rasul%2C+Younes+Belkada%2C+Shengyi+Huang%2C+Leandro+von+Werra%2C+Cl%C3%A9mentine+Fourrier%2C+Nathan+Habib%2C+et+al%2E+2023%2E+Zephyr%3A+Direct+distilla)
67. Amos Tversky. 1969. Intransitivity of preferences. Psychological review, 76(1):31. general kl-regularized preference. arXiv preprint arXiv:2402.07314. — [arXiv:2402.07314](https://arxiv.org/abs/2402.07314) [find](https://scholar.google.com/scholar?q=Amos+Tversky%2E+1969%2E+Intransitivity+of+preferences%2E+Psychological+review%2C+76%281%29%3A31%2E+general+kl%2Dregularized+preference%2E+arXiv+preprint+arXiv%3A2402%2E07314%2E+%E2%80%94+%5BarXiv%3A2402%2E07314%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E0)
68. Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. 2020. Trl: Transformer reinforcement learning. [https://github.com/huggingface/trl](https://github.com/huggingface/trl). [↗](https://github.com/huggingface/trl)
69. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. arXiv preprint arXiv:2401.10020. [find](https://scholar.google.com/scholar?q=Weizhe+Yuan%2C+Richard+Yuanzhe+Pang%2C+Kyunghyun+Cho%2C+Sainbayar+Sukhbaatar%2C+Jing+Xu%2C+and+Jason+Weston%2E+2024%2E+Self%2Drewarding+language+models%2E+arXiv+preprint+arXiv%3A2401%2E10020%2E)
70. Benyou Wang, Qianqian Xie, Jiahuan Pei, Zhihong Chen, Prayag Tiwari, Zhao Li, and Jie Fu. 2023a. Pre-trained language models in biomedical domain: A systematic survey. ACM Computing Surveys, 56(3):1– 52. [find](https://scholar.google.com/scholar?q=Benyou+Wang%2C+Qianqian+Xie%2C+Jiahuan+Pei%2C+Zhihong+Chen%2C+Prayag+Tiwari%2C+Zhao+Li%2C+and+Jie+Fu%2E+2023a%2E+Pre%2Dtrained+language+models+in+biomedical+domain%3A+A+systematic+survey%2E+ACM+Computing+Surveys%2C+56%283%29%3A1%E2%80%93+)
71. Yuanhao Wang, Qinghua Liu, and Chi Jin. 2023b. Is rlhf more difficult than standard rl? arXiv preprint arXiv:2306.14111. — [arXiv:2306.14111](https://arxiv.org/abs/2306.14111) [find](https://scholar.google.com/scholar?q=Yuanhao+Wang%2C+Qinghua+Liu%2C+and+Chi+Jin%2E+2023b%2E+Is+rlhf+more+difficult+than+standard+rl%3F+arXiv+preprint+arXiv%3A2306%2E14111%2E+%E2%80%94+%5BarXiv%3A2306%2E14111%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2306%2E14111%29)
72. Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, et al. 2023c. [Helpsteer: Multi-attribute helpfulness dataset for steerlm](https://md.research.elamur.ai/free/en/papers/wang-2024-helpsteer-multi-attribute-helpfulness-dataset/). arXiv preprint arXiv:2311.09528.
73. Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, et al. 2023d. [Helpsteer: Multi-attribute helpfulness dataset for steerlm](https://md.research.elamur.ai/free/en/papers/wang-2024-helpsteer-multi-attribute-helpfulness-dataset/). arXiv preprint arXiv:2311.09528.
74. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837.
75. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. [Rrhf: Rank responses to align language models with human feedback without tears](https://md.research.elamur.ai/free/en/papers/yuan-2023-rrhf-rank-responses-align-language/). arXiv preprint arXiv:2304.05302.
76. Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. 2023. [Slic-hf: Sequence likelihood calibration with human feedback](https://md.research.elamur.ai/free/en/papers/zhao-2023-slic-hf-sequence-likelihood-calibration/). arXiv preprint arXiv:2305.10425.
77. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+et+al%2E+2023%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2E+a)
78. Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. 2023. Beyond one-preference-for-all: Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03708. — [arXiv:2310.03708](https://arxiv.org/abs/2310.03708) [find](https://scholar.google.com/scholar?q=Zhanhui+Zhou%2C+Jie+Liu%2C+Chao+Yang%2C+Jing+Shao%2C+Yu+Liu%2C+Xiangyu+Yue%2C+Wanli+Ouyang%2C+and+Yu+Qiao%2E+2023%2E+Beyond+one%2Dpreference%2Dfor%2Dall%3A+Multi%2Dobjective+direct+preference+optimization%2E+arXiv+preprint+arXiv%3A2)
79. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. [find](https://scholar.google.com/scholar?q=Daniel+M+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+2019%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint+ar)
80. BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=BigScience+Workshop%2C+Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+et+al%2E+2022%2E+Bloom%3A+A+176b%2Dpara)
81. Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. 2023a. Better aligning text-to-image models with human preference. arXiv preprint arXiv:2303.14420. — [arXiv:2303.14420](https://arxiv.org/abs/2303.14420) [find](https://scholar.google.com/scholar?q=Xiaoshi+Wu%2C+Keqiang+Sun%2C+Feng+Zhu%2C+Rui+Zhao%2C+and+Hongsheng+Li%2E+2023a%2E+Better+aligning+text%2Dto%2Dimage+models+with+human+preference%2E+arXiv+preprint+arXiv%3A2303%2E14420%2E+%E2%80%94+%5BarXiv%3A2303%2E14420%5D%28https%3A%2F%2Farxiv%2Eor)
82. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2023b. [Fine-grained human feedback gives better rewards for language model training](https://md.research.elamur.ai/free/en/papers/wu-2023-fine-grained-human-feedback-gives/). arXiv preprint arXiv:2306.01693.
83. Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. 2023. Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf. arXiv preprint arXiv:2312.11456. [find](https://scholar.google.com/scholar?q=Wei+Xiong%2C+Hanze+Dong%2C+Chenlu+Ye%2C+Han+Zhong%2C+Nan+Jiang%2C+and+Tong+Zhang%2E+2023%2E+Gibbs+sampling+from+human+feedback%3A+A+provable+kl%2Dconstrained+framework+for+rlhf%2E+arXiv+preprint+arXiv%3A2312%2E11456%2E)
84. Chenlu Ye, Wei Xiong, Yuheng Zhang, Nan Jiang, and Tong Zhang. 2024. A theoretical analysis of nash learning from human feedback under 8654 13 [find](https://scholar.google.com/scholar?q=Chenlu+Ye%2C+Wei+Xiong%2C+Yuheng+Zhang%2C+Nan+Jiang%2C+and+Tong+Zhang%2E+2024%2E+A+theoretical+analysis+of+nash+learning+from+human+feedback+under+8654+13)

*Updated 24 October 2026*