---
title: "Personalized Soups — paper by Jang et al. (2023)"
description: "Standard RLHF trains a policy against a single reward model that reflects averaged annotator taste. 57 references, cited by 96 papers here."
authors: ["Joel Jang", "Seungone Kim", "Bill Yuchen Lin", "Yizhong Wang", "Jack Hessel", "Luke Zettlemoyer", "Hannaneh Hajishirzi", "Yejin Choi", "Prithviraj Ammanabrolu"]
year: 2023
venue: "arXiv:2310.11564"
source: "https://arxiv.org/abs/2310.11564"
doi: "10.48550/arxiv.2310.11564"
arxiv: "2310.11564"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jang-2023-personalized-soups-personalized-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jang-2023-personalized-soups-personalized-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/) / Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging

# Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging

**Authors:** [Joel Jang](https://md.research.elamur.ai/free/en/authors/joel-jang/), [Seungone Kim](https://md.research.elamur.ai/free/en/authors/seungone-kim/), [Bill Yuchen Lin](https://md.research.elamur.ai/free/en/authors/bill-yuchen-lin/), [Yizhong Wang](https://md.research.elamur.ai/free/en/authors/yizhong-wang/), [Jack Hessel](https://md.research.elamur.ai/free/en/authors/jack-hessel/), [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/), [Hannaneh Hajishirzi](https://md.research.elamur.ai/free/en/authors/hannaneh-hajishirzi/), [Yejin Choi](https://md.research.elamur.ai/free/en/authors/yejin-choi/) and [Prithviraj Ammanabrolu](https://md.research.elamur.ai/free/en/authors/prithviraj-ammanabrolu/) · **Year:** 2023 · **Venue:** arXiv:2310.11564 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.11564) · [PDF](https://arxiv.org/pdf/2310.11564) · [DOI 10.48550/arxiv.2310.11564](https://doi.org/10.48550/arxiv.2310.11564) · [arXiv 2310.11564](https://arxiv.org/abs/2310.11564) · [OpenAlex](https://openalex.org/W4387800281)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [96](https://md.research.elamur.ai/free/en/papers/jang-2023-personalized-soups-personalized-large-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Standard RLHF trains a policy against a single reward model that reflects averaged annotator taste. This ignores that individual users may want different, even opposing, qualities in a response. The authors frame this as Reinforcement Learning from Personalized Human Feedback (RLPHF) and treat it as a Multi-Objective Reinforcement Learning (MORL) problem. User preferences are split into explicit dimensions: expertise level, informativeness and style. Using Tulu-7B as the base model, they gather pairwise feedback from GPT-4 acting as a simulated annotator for each preference, then train a separate reward model per preference. They compare two approaches. PROMPTED-MORL is a baseline that optimizes several rewards at once with preference prompts during RL. PERSONALIZED SOUPS trains one PPO policy per preference and merges the chosen policies' parameters at inference time. Merging matches the multi-objective baseline while cutting training cost from exponential to linear in the number of preferences. New preferences can also be added without retraining existing policies. This makes user-controlled personalization of LLMs practical.

## Key points
- Base model is Tulu-7B (instruction-tuned LLaMA-7B). Rollouts and PPO training use 10k prompts drawn from GPT4-Alpaca.
- Six conflicting preferences span 3 dimensions (P1A/P1B, P2A/P2B, P3A/P3B), giving 8 combinations from AAA to BBB:
  - Elementary vs. PhD-level expertise.
  - Concise vs. informative.
  - Friendly vs. unfriendly.
- GPT-4 serves as a scalable stand-in for human annotators, and the evaluations are cross-checked with smaller-scale crowdworker judgments.
- PERSONALIZED SOUPS combines independently trained PPO policies through post-hoc weighted parameter merging. It stays competitive with PROMPTED-MORL, including when users add new preferences such as sassy or sarcastic styles, which extends the space to 16 combinations.
- Code is released at github.com/joeljang/RLPHF.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 402 works · 212,763 citations
- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 304 works · 106,137 citations
- [University of California San Diego](https://md.research.elamur.ai/free/en/universities/university-of-california-san-diego/) — University, San Diego, United States · 89 works · 54,032 citations
- [Korea Advanced Institute of Science and Technology](https://md.research.elamur.ai/free/en/universities/korea-advanced-institute-of-science-and-technology/) — University, Daejeon, South Korea · 56 works · 3,269 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi and Prithviraj Ammanabrolu. 2023. Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging. arXiv:2310.11564.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{jang2023personalized,
  title = {{Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging}},
  author = {Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi and Prithviraj Ammanabrolu},
  year = {2023},
  doi = {10.48550/arxiv.2310.11564},
  eprint = {2310.11564},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.11564}
}
```

- [← previous Parallel Context Windows for Large Language Models](https://md.research.elamur.ai/free/en/papers/ratner-2023-parallel-context-windows-large-language/)
- [next → PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization](https://md.research.elamur.ai/free/en/papers/wang-2023-promptagent-strategic-planning-language-models/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/)

## References

57 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Xiang Ao, Xiting Wang, Ling Luo, Ying Qiao, Qing He, and Xing Xie. PENS: A dataset and generic framework for personalized news headline generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Langua… [find](https://scholar.google.com/scholar?q=Xiang+Ao%2C+Xiting+Wang%2C+Ling+Luo%2C+Ying+Qiao%2C+Qing+He%2C+and+Xing+Xie%2E+PENS%3A+A+dataset+and+generic+framework+for+personalized+news+headline+generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Ass)
2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sha… [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+T%2E+J%2E+Henighan%2C+Nicholas+Joseph%2C+Saurav+Kadavath%2C+John+Kernion%2C+Tom+Conerly%2C+S)
3. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073, 2022b.
4. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023. [find](https://scholar.google.com/scholar?q=Stephen+Casper%2C+Xander+Davies%2C+Claudia+Shi%2C+Thomas+Krendl+Gilbert%2C+Je%CC%81re%CC%81my+Scheurer%2C+Javier+Rando%2C+Rachel+Freedman%2C+Tomasz+Korbak%2C+David+Lindner%2C+Pedro+Freire%2C+et+al%2E+Open+problems+and+fundamental+li)
5. Deborah Cohen, Moonkyung Ryu, Yinlam Chow, Orgad Keller, Ido Greenberg, Avinatan Hassidim, Michael Fink, Yossi Matias, Idan Szpektor, Craig Boutilier, et al. Dynamic planning in open-ended dialogue using reinforcement learning. arXiv preprint arXiv:2208.02294, 2022. — [arXiv:2208.02294](https://arxiv.org/abs/2208.02294) [find](https://scholar.google.com/scholar?q=Deborah+Cohen%2C+Moonkyung+Ryu%2C+Yinlam+Chow%2C+Orgad+Keller%2C+Ido+Greenberg%2C+Avinatan+Hassidim%2C+Michael+Fink%2C+Yossi+Matias%2C+Idan+Szpektor%2C+Craig+Boutilier%2C+et+al%2E+Dynamic+planning+in+open%2Dended+dialogue+us)
6. Shachar Don-Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Cold fusion: Collaborative descent for distributed multitask finetuning. arXiv preprint arXiv:2212.01378, 2022. [find](https://scholar.google.com/scholar?q=Shachar+Don%2DYehiya%2C+Elad+Venezian%2C+Colin+Raffel%2C+Noam+Slonim%2C+Yoav+Katz%2C+and+Leshem+Choshen%2E+Cold+fusion%3A+Collaborative+descent+for+distributed+multitask+finetuning%2E+arXiv+preprint+arXiv%3A2212%2E01378%2C+2)
7. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. [Alpacafarm: A simulation framework for methods that learn from human feedback](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/). arXiv preprint arXiv:2305.14387, 2023.
8. Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. [From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair nlp models](https://md.research.elamur.ai/free/en/papers/feng-2023-pretraining-data-language-models-downstream/). arXiv preprint arXiv:2305.08283, 2023.
9. Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April, 1, 2023. [find](https://scholar.google.com/scholar?q=Xinyang+Geng%2C+Arnav+Gudibande%2C+Hao+Liu%2C+Eric+Wallace%2C+Pieter+Abbeel%2C+Sergey+Levine%2C+and+Dawn+Song%2E+Koala%3A+A+dialogue+model+for+academic+research%2E+Blog+post%2C+April%2C+1%2C+2023%2E)
10. Conor F Hayes, Roxana Rădulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Sys… [find](https://scholar.google.com/scholar?q=Conor+F+Hayes%2C+Roxana+Ra%CC%86dulescu%2C+Eugenio+Bargiacchi%2C+Johan+Ka%CC%88llstro%CC%88m%2C+Matthew+Macfarlane%2C+Mathieu+Reymond%2C+Timothy+Verstraeten%2C+Luisa+M+Zintgraf%2C+Richard+Dazeley%2C+Fredrik+Heintz%2C+et+al%2E+A+practical)
11. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=nZeVKeeFYf9. [↗](https://openreview.net/forum?)
12. Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. [Lorahub: Efficient cross-task generalization via dynamic lora composition](https://md.research.elamur.ai/free/en/papers/huang-2023-lorahub-efficient-cross-task-generalization/). ArXiv, abs/2307.13269, 2023. URL [https://api.semanticscholar.org/CorpusID:260155012](https://api.semanticscholar.org/CorpusID:260155012). [↗](https://api.semanticscholar.org/CorpusID:260155012)
13. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. [Editing models with task arithmetic](https://md.research.elamur.ai/free/en/papers/ilharco-2022-editing-models-task-arithmetic/). ICLR, 2022.
14. Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. [Exploring the benefits of training expert language models over instruction tuning](https://md.research.elamur.ai/free/en/papers/jang-2023-exploring-benefits-training-expert-language/). In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan S…
15. Sungdong Kim, Sanghwan Bae, Jamin Shin, Soyoung Kang, Donghyun Kwak, Kang Min Yoo, and Minjoon Seo. Aligning large language models through synthetic feedback. arXiv preprint arXiv:2305.13735, 2023. [find](https://scholar.google.com/scholar?q=Sungdong+Kim%2C+Sanghwan+Bae%2C+Jamin+Shin%2C+Soyoung+Kang%2C+Donghyun+Kwak%2C+Kang+Min+Yoo%2C+and+Minjoon+Seo%2E+Aligning+large+language+models+through+synthetic+feedback%2E+arXiv+preprint+arXiv%3A2305%2E13735%2C+2023%2E)
16. Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. [Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback](https://md.research.elamur.ai/free/en/papers/kirk-2023-personalisation-within-bounds-risk-taxonomy/). arXiv preprint arXiv:2303.05453, 2023.
17. Noriyuki Kojima, Alane Suhr, and Yoav Artzi. Continual learning for grounded instruction generation by observing human following behavior. Transactions of the Association for Computational Linguistics, 9:1303–1319, 2021. [find](https://scholar.google.com/scholar?q=Noriyuki+Kojima%2C+Alane+Suhr%2C+and+Yoav+Artzi%2E+Continual+learning+for+grounded+instruction+generation+by+observing+human+following+behavior%2E+Transactions+of+the+Association+for+Computational+Linguistics)
18. Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. [Branch-train-merge: Embarrassingly parallel training of expert language models](https://md.research.elamur.ai/free/en/papers/li-2022-branch-train-merge-embarrassingly-parallel/). arXiv preprint arXiv:2208.03306, 2022.
19. Pan Li and Alexander Tuzhilin. Towards controllable and personalized review generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3237–3245, Hong Kong, China,… [find](https://scholar.google.com/scholar?q=Pan+Li+and+Alexander+Tuzhilin%2E+Towards+controllable+and+personalized+review+generation%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+Internatio)
20. Bodhisattwa Prasad Majumder, Shuyang Li, Jianmo Ni, and Julian McAuley. Generating personalized recipes from historical user preferences. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processin… [find](https://scholar.google.com/scholar?q=Bodhisattwa+Prasad+Majumder%2C+Shuyang+Li%2C+Jianmo+Ni%2C+and+Julian+McAuley%2E+Generating+personalized+recipes+from+historical+user+preferences%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+)
21. Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes. [Training millions of personalized dialogue agents](https://md.research.elamur.ai/free/en/papers/mazare-2018-training-millions-personalized-dialogue-agents/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2775–2779, Brussels, Belgium, October-November 2018. Association for Computatio…
22. Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. [Teaching language models to support answers with verified quotes](https://md.research.elamur.ai/free/en/papers/menick-2022-teaching-language-models-support-answers/). arXiv preprint arXiv:2203.11147, 2022.
23. Reiichiro Nakano, Jacob Hilton, S. Arun Balaji, Jeff Wu, Ouyang Long, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted q… — [WebGPT: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)
24. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. [Webgpt: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/). arXiv preprint arXiv:2112.09332, 2021b.
25. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Low… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
26. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35: 27730–27744, 2022b.
27. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. [Instruction tuning with gpt-4](https://md.research.elamur.ai/free/en/papers/peng-2023-instruction-tuning-gpt-4/). arXiv preprint arXiv:2304.03277, 2023.
28. Jonas Pfeiffer, Naman Goyal, Xi Victoria Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. [Lifting the curse of multilinguality by pre-training modular transformers](https://md.research.elamur.ai/free/en/papers/pfeiffer-2022-lifting-curse-multilinguality-pre-training/). arXiv preprint arXiv:2205.06266, 2022.
29. Alexandre Rame, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier, and Matthieu Cord. [Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards](https://md.research.elamur.ai/free/en/papers/rame-2023-rewarded-soups-towards-pareto-optimal/). arXiv preprint arXiv:2306.04488, 2023.
30. Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. [Lamp: When large language models meet personalization](https://md.research.elamur.ai/free/en/papers/salemi-2024-lamp-when-large-language-models/). arXiv preprint arXiv:2304.11406, 2023.
31. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. [Whose opinions do language models reflect?](https://md.research.elamur.ai/free/en/papers/santurkar-2023-whose-opinions-language-models-reflect/) arXiv preprint arXiv:2303.17548, 2023.
32. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)
33. Umer Siddique, Paul Weng, and Matthieu Zimmer. Learning fair policies in multi-objective (deep) reinforcement learning with average and discounted rewards. In International Conference on Machine Learning, pp. 8905–8915. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Umer+Siddique%2C+Paul+Weng%2C+and+Matthieu+Zimmer%2E+Learning+fair+policies+in+multi%2Dobjective+%28deep%29+reinforcement+learning+with+average+and+discounted+rewards%2E+In+International+Conference+on+Machine+Learn)
34. David Silver, Satinder Singh, Doina Precup, and Richard S Sutton. Reward is enough. Artificial Intelligence, 299:103535, 2021. [find](https://scholar.google.com/scholar?q=David+Silver%2C+Satinder+Singh%2C+Doina+Precup%2C+and+Richard+S+Sutton%2E+Reward+is+enough%2E+Artificial+Intelligence%2C+299%3A103535%2C+2021%2E)
35. Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. [A long way to go: Investigating length correlations in rlhf](https://md.research.elamur.ai/free/en/papers/singhal-2023-long-way-go-investigating-length/). arXiv preprint arXiv:2310.03716, 2023.
36. Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. [Preference ranking optimization for human alignment](https://md.research.elamur.ai/free/en/papers/song-2023-preference-ranking-optimization-human-alignment/). arXiv preprint arXiv:2306.17492, 2023.
37. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F+Christiano%2E+Learning+to+summarize+with+human+feedback%2E+Advances+in+Neural+Infor)
38. Alane Suhr and Yoav Artzi. Continual learning for instruction following from realtime feedback. arXiv preprint arXiv:2212.09710, 2022. [find](https://scholar.google.com/scholar?q=Alane+Suhr+and+Yoav+Artzi%2E+Continual+learning+for+instruction+following+from+realtime+feedback%2E+arXiv+preprint+arXiv%3A2212%2E09710%2C+2022%2E)
39. Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton+and+Andrew+G+Barto%2E+Reinforcement+learning%3A+An+introduction%2E+MIT+press%2C+2018%2E)
40. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. [Lamda: Language models for dialog applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/). arXiv preprint arXiv:2201.08239, 2022.
41. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
42. Peter Vamplew, Richard Dazeley, Cameron Foale, Sally Firmin, and Jane Mummery. Human-aligned artificial intelligence is a multiobjective problem. Ethics and Information Technology, 20:27–40, 2018. [find](https://scholar.google.com/scholar?q=Peter+Vamplew%2C+Richard+Dazeley%2C+Cameron+Foale%2C+Sally+Firmin%2C+and+Jane+Mummery%2E+Human%2Daligned+artificial+intelligence+is+a+multiobjective+problem%2E+Ethics+and+Information+Technology%2C+20%3A27%E2%80%9340%2C+2018%2E)
43. Kristof Van Moffaert and Ann Nowé. Multi-objective reinforcement learning using sets of pareto dominating policies. The Journal of Machine Learning Research, 15(1):3483–3512, 2014. [find](https://scholar.google.com/scholar?q=Kristof+Van+Moffaert+and+Ann+Nowe%CC%81%2E+Multi%2Dobjective+reinforcement+learning+using+sets+of+pareto+dominating+policies%2E+The+Journal+of+Machine+Learning+Research%2C+15%281%29%3A3483%E2%80%933512%2C+2014%2E)
44. Kristof Van Moffaert, Madalina M Drugan, and Ann Nowé. Scalarized multi-objective reinforcement learning: Novel design techniques. In 2013 IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL), pp. 191–199. [find](https://scholar.google.com/scholar?q=Kristof+Van+Moffaert%2C+Madalina+M+Drugan%2C+and+Ann+Nowe%CC%81%2E+Scalarized+multi%2Dobjective+reinforcement+learning%3A+Novel+design+techniques%2E+In+2013+IEEE+symposium+on+adaptive+dynamic+programming+and+reinforce)
45. Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. [How far can camels go? exploring the state of instruction tuning on open resources](https://md.research.elamur.ai/free/en/papers/wang-2023-far-camels-go-exploring-state/). arXiv preprint arXiv:2306.04751, 2023.
46. Mitchell Wortsman, Suchin Gururangan, Shen Li, Ali Farhadi, Ludwig Schmidt, Michael Rabbat, and Ari S Morcos. lo-fi: distributed fine-tuning without communication. arXiv preprint arXiv:2210.11948, 2022a. — [arXiv:2210.11948](https://arxiv.org/abs/2210.11948) [find](https://scholar.google.com/scholar?q=Mitchell+Wortsman%2C+Suchin+Gururangan%2C+Shen+Li%2C+Ali+Farhadi%2C+Ludwig+Schmidt%2C+Michael+Rabbat%2C+and+Ari+S+Morcos%2E+lo%2Dfi%3A+distributed+fine%2Dtuning+without+communication%2E+arXiv+preprint+arXiv%3A2210%2E11948%2C+202)
47. Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://md.research.elamur.ai/free/en/papers/wortsman-2022-model-soups-averaging-weights-multiple/). In Internat…
48. Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. [Recursively summarizing books with human feedback](https://md.research.elamur.ai/free/en/papers/wu-2021-recursively-summarizing-books-human-feedback/). arXiv preprint arXiv:2109.10862, 2021a.
49. Yuwei Wu, Xuezhe Ma, and Diyi Yang. Personalized response generation via generative split memory network. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1956–1970, 2021b. [find](https://scholar.google.com/scholar?q=Yuwei+Wu%2C+Xuezhe+Ma%2C+and+Diyi+Yang%2E+Personalized+response+generation+via+generative+split+memory+network%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association+for+Com)
50. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. [Fine-grained human feedback gives better rewards for language model training](https://md.research.elamur.ai/free/en/papers/wu-2023-fine-grained-human-feedback-gives/). arXiv preprint arXiv:2306.01693, 2023.
51. Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojciech Matusik. Prediction-guided multi-objective reinforcement learning for continuous robot control. In International conference on machine learning, pp. 10607–10616. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Jie+Xu%2C+Yunsheng+Tian%2C+Pingchuan+Ma%2C+Daniela+Rus%2C+Shinjiro+Sueda%2C+and+Wojciech+Matusik%2E+Prediction%2Dguided+multi%2Dobjective+reinforcement+learning+for+continuous+robot+control%2E+In+International+conferen)
52. Jing Xu, Arthur Szlam, and Jason Weston. Beyond goldfish memory: Long-term open-domain conversation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5180–5197, Dublin, Ireland, May 2022. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Jing+Xu%2C+Arthur+Szlam%2C+and+Jason+Weston%2E+Beyond+goldfish+memory%3A+Long%2Dterm+open%2Ddomain+conversation%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+)
53. Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. Advances in neural information processing systems, 32, 2019. [find](https://scholar.google.com/scholar?q=Runzhe+Yang%2C+Xingyuan+Sun%2C+and+Karthik+Narasimhan%2E+A+generalized+algorithm+for+multi%2Dobjective+reinforcement+learning+and+policy+adaptation%2E+Advances+in+neural+information+processing+systems%2C+32%2C+2019)
54. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. [Personalizing dialogue agents: I have a dog, do you have pets too?](https://md.research.elamur.ai/free/en/papers/zhang-2018-personalizing-dialogue-agents-i-have/) arXiv preprint arXiv:1801.07243, 2018.
55. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). arXiv preprint arXiv:2306.05685, 2023.
56. Yinhe Zheng, Guanyi Chen, Minlie Huang, Song Liu, and Xuan Zhu. Personalized dialogue generation with diversified traits. arXiv preprint arXiv:1901.09672, 2019. — [arXiv:1901.09672](https://arxiv.org/abs/1901.09672) [find](https://scholar.google.com/scholar?q=Yinhe+Zheng%2C+Guanyi+Chen%2C+Minlie+Huang%2C+Song+Liu%2C+and+Xuan+Zhu%2E+Personalized+dialogue+generation+with+diversified+traits%2E+arXiv+preprint+arXiv%3A1901%2E09672%2C+2019%2E+%E2%80%94+%5BarXiv%3A1901%2E09672%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
57. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. [find](https://scholar.google.com/scholar?q=Daniel+M+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint+arXiv%3A19)

*Updated 2 November 2026*