---
title: "Learning to Optimize Multi-Objective Alignment Through Dynamic"
description: "The authors propose dynamic reward weighting, where objective weights change during training. 59 references, cited by 3 papers here."
authors: ["Yining Lu", "Zilong Wang", "Shiyang Li", "Xin Liu", "Changlong Yu", "Qingyu Yin", "Zhan Shi", "Zixuan Zhang", "Meng Jiang"]
year: 2026
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2026.tacl-1.47/"
doi: "10.1162/tacl.a.696"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lu-2026-learning-optimize-multi-objective-alignment/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lu-2026-learning-optimize-multi-objective-alignment/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/) / Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weighting

# Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weighting

**Authors:** [Yining Lu](https://md.research.elamur.ai/free/en/authors/yining-lu/), [Zilong Wang](https://md.research.elamur.ai/free/en/authors/zilong-wang/), [Shiyang Li](https://md.research.elamur.ai/free/en/authors/shiyang-li/), [Xin Liu](https://md.research.elamur.ai/free/en/authors/xin-liu/), [Changlong Yu](https://md.research.elamur.ai/free/en/authors/changlong-yu/), [Qingyu Yin](https://md.research.elamur.ai/free/en/authors/qingyu-yin/), [Zhan Shi](https://md.research.elamur.ai/free/en/authors/zhan-shi/), [Zixuan Zhang](https://md.research.elamur.ai/free/en/authors/zixuan-zhang/) and [Meng Jiang](https://md.research.elamur.ai/free/en/authors/meng-jiang/) · **Year:** 2026 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2026](https://md.research.elamur.ai/free/en/venues/tacl/2026/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.tacl-1.47/) · [PDF](https://aclanthology.org/2026.tacl-1.47.pdf) · [DOI 10.1162/tacl.a.696](https://doi.org/10.1162/tacl.a.696) · [OpenAlex](https://openalex.org/W4415088394)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/lu-2026-learning-optimize-multi-objective-alignment/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

The paper addresses a weakness in multi-objective online RL for aligning LLMs. Combining rewards with fixed linear weights cannot reach solutions on non-convex parts of the Pareto front. It also keeps spending gradient effort on objectives that have already saturated. The authors propose dynamic reward weighting, where objective weights change during training. They offer two variants. Hypervolume-guided weight adaptation is for cases where user preferences are known: it rewards checkpoints that add new non-dominated solutions. Gradient-based weight optimization is for cases without preferences: it estimates through gradients how much each objective helps overall progress and redistributes weight accordingly.

The main setup trains Qwen3-8B on Math500 with GRPO, balancing accuracy, conciseness and clarity. Both methods produce Pareto fronts that dominate fixed-weight baselines and the concurrent method PAMA, and they need fewer training steps. The gradient-based variant performs best. This matters because it offers a general way to tune several objectives at once, instead of relying on hand-picked weights or task-specific rules.

## Key points

- The methods were tested with three RL algorithms (GRPO, REINFORCE, RLOO) and three datasets: Math500, MATH, and SafeSQL. SafeSQL is an SQL-injection benchmark with two objectives, accuracy and security.
- Generalization was checked on several model families: Qwen3-8B, Deepseek-7B, Mistral-7B and Llama3-8B.
- The gradient-based method reaches its Pareto front about 6.1 fewer training steps sooner on average across RL algorithms. The hypervolume-guided method gives only small efficiency gains.
- Gains were limited for Ministral-8B-Instruct and Llama-3.1-8B-Instruct, where making responses shorter consistently lowered accuracy. The authors attribute this to pretraining saturation or limited model capacity.
- Published in TACL vol. 14 (2026), pp. 1051–1073. Code is available at github.com/yining610/dynamic-reward-weighting.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Amazon](https://md.research.elamur.ai/free/en/companies/amazon/) — Company, Seattle, United States · 112 works · 40,565 citations
- [University of Notre Dame](https://md.research.elamur.ai/free/en/universities/university-of-notre-dame/) — University, Notre Dame, United States · 51 works · 36,717 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 252 works

## Cite

Cite as: Yining Lu, Zilong Wang, Shiyang Li, Xin Liu, Changlong Yu, Qingyu Yin, Zhan Shi, Zixuan Zhang and Meng Jiang. 2026. Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weighting. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{lu2026learning,
  title = {{Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weighting}},
  author = {Yining Lu, Zilong Wang, Shiyang Li, Xin Liu, Changlong Yu, Qingyu Yin, Zhan Shi, Zixuan Zhang and Meng Jiang},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.696},
  url = {https://doi.org/10.1162/tacl.a.696}
}
```

- [← previous Large Language Models for Mental Health: A Multilingual Evaluation](https://md.research.elamur.ai/free/en/papers/raihan-2026-large-language-models-mental-health/)
- [next → LLM Agents in Law: Taxonomy, Applications, and Challenges](https://md.research.elamur.ai/free/en/papers/liu-2026-llm-agents-law-taxonomy-applications/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/)

## References

59 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Pranjal Aggarwal and Sean Welleck. 2025. [L1: Controlling how long a reasoning model thinks with reinforcement learning](https://md.research.elamur.ai/free/en/papers/aggarwal-2025-l1-controlling-long-reasoning-model/).
2. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. 2024. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. [find](https://scholar.google.com/scholar?q=Arash+Ahmadian%2C+Chris+Cremer%2C+Matthias+Gall%C3%A9%2C+Marzieh+Fadaee%2C+Julia+Kreutzer%2C+Olivier+Pietquin%2C+Ahmet+%C3%9Cst%C3%BCn%2C+and+Sara+Hooker%2E+2024%2E+Back+to+basics%3A+Revisiting+reinforce+style+optimization+for+learning)
3. Amir Beck and Marc Teboulle. 2003. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):167–175. [find](https://scholar.google.com/scholar?q=Amir+Beck+and+Marc+Teboulle%2E+2003%2E+Mirror+descent+and+nonlinear+projected+subgradient+methods+for+convex+optimization%2E+Operations+Research+Letters%2C+31%283%29%3A167%E2%80%93175%2E)
4. S.P. Boyd and L. Vandenberghe. 2004. Convex Optimization. Number pt. 1 in Berichte über verteilte messysteme. Cambridge University Press. [find](https://scholar.google.com/scholar?q=S%2EP%2E+Boyd+and+L%2E+Vandenberghe%2E+2004%2E+Convex+Optimization%2E+Number+pt%2E+1+in+Berichte+%C3%BCber+verteilte+messysteme%2E+Cambridge+University+Press%2E)
5. Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2025. Acereason-nemotron: Advancing math and code reasoning through reinforcement learning. [find](https://scholar.google.com/scholar?q=Yang+Chen%2C+Zhuolin+Yang%2C+Zihan+Liu%2C+Chankyu+Lee%2C+Peng+Xu%2C+Mohammad+Shoeybi%2C+Bryan+Catanzaro%2C+and+Wei+Ping%2E+2025%2E+Acereason%2Dnemotron%3A+Advancing+math+and+code+reasoning+through+reinforcement+learning%2E)
6. DeepSeek-AI, :, Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jia… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+%3A%2C+Xiao+Bi%2C+Deli+Chen%2C+Guanting+Chen%2C+Shanhuang+Chen%2C+Damai+Dai%2C+Chengqi+Deng%2C+Honghui+Ding%2C+Kai+Dong%2C+Qiushi+Du%2C+Zhe+Fu%2C+Huazuo+Gao%2C+Kaige+Gao%2C+Wenjun+Gao%2C+Ruiqi+Ge%2C+Kang+Guan%2C+Daya+Guo%2C)
7. Jean-Antoine Désidéri. 2012. Multiple-gradient descent algorithm (mgda) for multiobjective optimization. Comptes Rendus Mathematique, 350(5):313–318. [find](https://scholar.google.com/scholar?q=Jean%2DAntoine+D%C3%A9sid%C3%A9ri%2E+2012%2E+Multiple%2Dgradient+descent+algorithm+%28mgda%29+for+multiobjective+optimization%2E+Comptes+Rendus+Mathematique%2C+350%285%29%3A313%E2%80%93318%2E)
8. Michael TM Emmerich and André H Deutz. 2018. A tutorial on multiobjective optimization: fundamentals and evolutionary methods. Natural computing, 17:585–609. [find](https://scholar.google.com/scholar?q=Michael+TM+Emmerich+and+Andr%C3%A9+H+Deutz%2E+2018%2E+A+tutorial+on+multiobjective+optimization%3A+fundamentals+and+evolutionary+methods%2E+Natural+computing%2C+17%3A585%E2%80%93609%2E)
9. Simin Fan, Matteo Pagliardini, and Martin Jaggi. 2024. Doge: domain reweighting with generalization estimation. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. [find](https://scholar.google.com/scholar?q=Simin+Fan%2C+Matteo+Pagliardini%2C+and+Martin+Jaggi%2E+2024%2E+Doge%3A+domain+reweighting+with+generalization+estimation%2E+In+Proceedings+of+the+41st+International+Conference+on+Machine+Learning%2C+ICML%E2%80%9924%2E+JMLR%2Eo)
10. Peter J. Fleming, Robin C. Purshouse, and Robert J. Lygoe. 2005. Many-objective optimization: An engineering design perspective. In Evolutionary Multi-Criterion Optimization, pages 14–32, Berlin, Heidelberg. Springer Berlin Heidelberg. [find](https://scholar.google.com/scholar?q=Peter+J%2E+Fleming%2C+Robin+C%2E+Purshouse%2C+and+Robert+J%2E+Lygoe%2E+2005%2E+Many%2Dobjective+optimization%3A+An+engineering+design+perspective%2E+In+Evolutionary+Multi%2DCriterion+Optimization%2C+pages+14%E2%80%9332%2C+Berlin%2C+Heid)
11. Carlos M. Fonseca and Peter J. Fleming. 1996. On the performance assessment and comparison of stochastic multiobjective optimizers. In Parallel Problem Solving from Nature — PPSN IV, pages 584–593, Berlin, Heidelberg. Springer Berlin Heidelberg. [find](https://scholar.google.com/scholar?q=Carlos+M%2E+Fonseca+and+Peter+J%2E+Fleming%2E+1996%2E+On+the+performance+assessment+and+comparison+of+stochastic+multiobjective+optimizers%2E+In+Parallel+Problem+Solving+from+Nature+%E2%80%94+PPSN+IV%2C+pages+584%E2%80%93593%2C+Be)
12. C.M. Fonseca, L. Paquete, and M. Lopez-Ibanez. 2006. An improved dimension-sweep algorithm for the hypervolume indicator. In 2006 IEEE International Conference on Evolutionary Computation, pages 1157–1163. [find](https://scholar.google.com/scholar?q=C%2EM%2E+Fonseca%2C+L%2E+Paquete%2C+and+M%2E+Lopez%2DIbanez%2E+2006%2E+An+improved+dimension%2Dsweep+algorithm+for+the+hypervolume+indicator%2E+In+2006+IEEE+International+Conference+on+Evolutionary+Computation%2C+pages+1157%E2%80%93)
13. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao,… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
14. Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilė Lukošiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. 2023. Studying large language model general… [find](https://scholar.google.com/scholar?q=Roger+Grosse%2C+Juhan+Bae%2C+Cem+Anil%2C+Nelson+Elhage%2C+Alex+Tamkin%2C+Amirhossein+Tajdini%2C+Benoit+Steiner%2C+Dustin+Li%2C+Esin+Durmus%2C+Ethan+Perez%2C+Evan+Hubinger%2C+Kamile%CC%87+Luko%C5%A1iu%CC%84te%CC%87%2C+Karina+Nguyen%2C+Nicholas+Jos)
15. Shangding Gu, Bilgehan Sel, Yuhao Ding, Lu Wang, Qingwei Lin, Alois Knoll, and Ming Jin. 2024. Safe and balanced: A framework for constrained multi-objective reinforcement learning. [find](https://scholar.google.com/scholar?q=Shangding+Gu%2C+Bilgehan+Sel%2C+Yuhao+Ding%2C+Lu+Wang%2C+Qingwei+Lin%2C+Alois+Knoll%2C+and+Ming+Jin%2E+2024%2E+Safe+and+balanced%3A+A+framework+for+constrained+multi%2Dobjective+reinforcement+learning%2E)
16. Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Zexu Sun, Bowen Sun, Huimin Chen, Ruobing Xie, Jie Zhou, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. Controllable preference optimization: Toward controllable multi-objective alignment. In Proceedings of the 2024 Conference on Empirical Methods in Natu… [find](https://scholar.google.com/scholar?q=Yiju+Guo%2C+Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Zexu+Sun%2C+Bowen+Sun%2C+Huimin+Chen%2C+Ruobing+Xie%2C+Jie+Zhou%2C+Yankai+Lin%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2024%2E+Controllable+preference+optimization%3A+Toward+contro)
17. Conor F Hayes, Roxana Rădulescu, Eugenio Bar-giacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. 2022. A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agen… [find](https://scholar.google.com/scholar?q=Conor+F+Hayes%2C+Roxana+Ra%CC%86dulescu%2C+Eugenio+Bar%2Dgiacchi%2C+Johan+K%C3%A4llstr%C3%B6m%2C+Matthew+Macfarlane%2C+Mathieu+Reymond%2C+Timothy+Verstraeten%2C+Luisa+M+Zintgraf%2C+Richard+Dazeley%2C+Fredrik+Heintz%2C+et+al%2E+2022%2E+A+prac)
18. Qiang He and Setareh Maghsudi. 2025. Pareto multi-objective alignment for language models. [find](https://scholar.google.com/scholar?q=Qiang+He+and+Setareh+Maghsudi%2E+2025%2E+Pareto+multi%2Dobjective+alignment+for+language+models%2E)
19. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1.
20. Jian Hu, Xibin Wu, Wei Shen, Jason Klein Liu, Zilin Zhu, Weixun Wang, Songlin Jiang, Haoran Wang, Hao Chen, Bin Chen, Weikai Fang, Xianyu, Yu Cao, and Haotian Xu. 2025. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models. [find](https://scholar.google.com/scholar?q=Jian+Hu%2C+Xibin+Wu%2C+Wei+Shen%2C+Jason+Klein+Liu%2C+Zilin+Zhu%2C+Weixun+Wang%2C+Songlin+Jiang%2C+Haoran+Wang%2C+Hao+Chen%2C+Bin+Chen%2C+Weikai+Fang%2C+Xianyu%2C+Yu+Cao%2C+and+Haotian+Xu%2E+2025%2E+Reinforce%2B%2B%3A+An+efficient+rlhf+)
21. Zenan Huang, Yihong Zhuang, Guoshan Lu, Zeyu Qin, Haokai Xu, Tianyu Zhao, Ru Peng, Jiaqi Hu, Zhanming Shen, Xiaomeng Hu, Xijun Gu, Peiyi Tu, Jiaxin Liu, Wenyu Chen, Yuzhuo Fu, Zhiting Fan, Yanmei Gu, Yuanyuan Wang, Zhengkai Yang, Jianguo Li, and Junbo Zhao. 2025. Reinforcement learning with rubric a… [find](https://scholar.google.com/scholar?q=Zenan+Huang%2C+Yihong+Zhuang%2C+Guoshan+Lu%2C+Zeyu+Qin%2C+Haokai+Xu%2C+Tianyu+Zhao%2C+Ru+Peng%2C+Jiaqi+Hu%2C+Zhanming+Shen%2C+Xiaomeng+Hu%2C+Xijun+Gu%2C+Peiyi+Tu%2C+Jiaxin+Liu%2C+Wenyu+Chen%2C+Yuzhuo+Fu%2C+Zhiting+Fan%2C+Yanmei+Gu%2C+)
22. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
23. Pang Wei Koh and Percy Liang. 2017. [Understanding black-box predictions via influence functions](https://md.research.elamur.ai/free/en/papers/koh-2017-understanding-black-box-predictions-influence/). In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1885–1894. PMLR.
24. Harshat Kumar, Alec Koppel, and Alejandro Ribeiro. 2023. On the sample complexity of actor-critic method for reinforcement learning with function approximation. Mach. Learn., 112(7):2433–2467. [find](https://scholar.google.com/scholar?q=Harshat+Kumar%2C+Alec+Koppel%2C+and+Alejandro+Ribeiro%2E+2023%2E+On+the+sample+complexity+of+actor%2Dcritic+method+for+reinforcement+learning+with+function+approximation%2E+Mach%2E+Learn%2E%2C+112%287%29%3A2433%E2%80%932467%2E)
25. Chengao Li, Hanyu Zhang, Yunkun Xu, Hongyan Xue, Xiang Ao, and Qing He. 2025a. Gradient-adaptive policy optimization: Towards multi-objective alignment of large language models. [find](https://scholar.google.com/scholar?q=Chengao+Li%2C+Hanyu+Zhang%2C+Yunkun+Xu%2C+Hongyan+Xue%2C+Xiang+Ao%2C+and+Qing+He%2E+2025a%2E+Gradient%2Dadaptive+policy+optimization%3A+Towards+multi%2Dobjective+alignment+of+large+language+models%2E)
26. Zhuo Li, Guodong Du, Weiyang Guo, Yigeng Zhou, Xiucheng Li, Wenya Wang, Fangming Liu, Yequan Wang, Deheng Ye, Min Zhang, and Jing Li. 2025b. Multi-objective large language model alignment with hierarchical experts. [find](https://scholar.google.com/scholar?q=Zhuo+Li%2C+Guodong+Du%2C+Weiyang+Guo%2C+Yigeng+Zhou%2C+Xiucheng+Li%2C+Wenya+Wang%2C+Fangming+Liu%2C+Yequan+Wang%2C+Deheng+Ye%2C+Min+Zhang%2C+and+Jing+Li%2E+2025b%2E+Multi%2Dobjective+large+language+model+alignment+with+hierarc)
27. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/).
28. Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. 2025. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. [find](https://scholar.google.com/scholar?q=Mingjie+Liu%2C+Shizhe+Diao%2C+Ximing+Lu%2C+Jian+Hu%2C+Xin+Dong%2C+Yejin+Choi%2C+Jan+Kautz%2C+and+Yi+Dong%2E+2025%2E+Prorl%3A+Prolonged+reinforcement+learning+expands+reasoning+boundaries+in+large+language+models%2E)
29. Suyun Liu and Luis Nunes Vicente. 2024. The stochastic multi-gradient algorithm for multi-objective optimization and its application to supervised machine learning. Annals of Operations Research, 339(3):1119–1148. [find](https://scholar.google.com/scholar?q=Suyun+Liu+and+Luis+Nunes+Vicente%2E+2024%2E+The+stochastic+multi%2Dgradient+algorithm+for+multi%2Dobjective+optimization+and+its+application+to+supervised+machine+learning%2E+Annals+of+Operations+Research%2C+339%28)
30. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. [Learning word vectors for sentiment analysis](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/). In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon…
31. Mistral AI Team. Ministral-8b-instruct-2410. 2024. [find](https://scholar.google.com/scholar?q=Mistral+AI+Team%2E+Ministral%2D8b%2Dinstruct%2D2410%2E+2024%2E)
32. Subhojyoti Mukherjee, Anusha Lalitha, Sailik Sengupta, Aniket Deshmukh, and Branislav Kveton. 2024. Multi-objective alignment of large language models through hypervolume maximization. [find](https://scholar.google.com/scholar?q=Subhojyoti+Mukherjee%2C+Anusha+Lalitha%2C+Sailik+Sengupta%2C+Aniket+Deshmukh%2C+and+Branislav+Kveton%2E+2024%2E+Multi%2Dobjective+alignment+of+large+language+models+through+hypervolume+maximization%2E)
33. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. [Train…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
34. Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. 2023. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. In Advances in Neural Information Processing Systems, volume 36,… [find](https://scholar.google.com/scholar?q=Alexandre+Rame%2C+Guillaume+Couairon%2C+Corentin+Dancette%2C+Jean%2DBaptiste+Gaya%2C+Mustafa+Shukor%2C+Laure+Soulier%2C+and+Matthieu+Cord%2E+2023%2E+Rewarded+soups%3A+towards+pareto%2Doptimal+alignment+by+interpolating+wei)
35. Diederik M. Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. 2013. A survey of multi-objective sequential decision-making. J. Artif. Int. Res., 48(1):67–113. [find](https://scholar.google.com/scholar?q=Diederik+M%2E+Roijers%2C+Peter+Vamplew%2C+Shimon+Whiteson%2C+and+Richard+Dazeley%2E+2013%2E+A+survey+of+multi%2Dobjective+sequential+decision%2Dmaking%2E+J%2E+Artif%2E+Int%2E+Res%2E%2C+48%281%29%3A67%E2%80%93113%2E)
36. Roxana Rădulescu, Patrick Mannion, Yijie Zhang, Diederik M. Roijers, and Ann Nowé. 2020. A utility-based analysis of equilibria in multi-objective normal-form games. The Knowledge Engineering Review, 35:e32. [find](https://scholar.google.com/scholar?q=Roxana+Ra%CC%86dulescu%2C+Patrick+Mannion%2C+Yijie+Zhang%2C+Diederik+M%2E+Roijers%2C+and+Ann+Now%C3%A9%2E+2020%2E+A+utility%2Dbased+analysis+of+equilibria+in+multi%2Dobjective+normal%2Dform+games%2E+The+Knowledge+Engineering+Review%2C)
37. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+2017%2E+Proximal+policy+optimization+algorithms%2E)
38. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/).
39. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. [Hybridflow: A flexible and efficient rlhf framework](https://md.research.elamur.ai/free/en/papers/sheng-2024-hybridflow-flexible-efficient-rlhf-framework/). In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, page 1279–1297. ACM.
40. Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, volume 12. MIT Press. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton%2C+David+McAllester%2C+Satinder+Singh%2C+and+Yishay+Mansour%2E+1999%2E+Policy+gradient+methods+for+reinforcement+learning+with+function+approximation%2E+In+Advances+in+Neural+Information+Processi)
41. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Z… — [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k1-5-scaling-reinforcement/)
42. Kaiwen Wang, Rahul Kidambi, Ryan Sullivan, Alekh Agarwal, Christoph Dann, Andrea Michi, Marco Gelmi, Yunxuan Li, Raghav Gupta, Kumar Avinava Dubey, Alexandre Rame, Johan Ferret, Geoffrey Cideron, Le Hou, Hongkun Yu, Amr Ahmed, Aranyak Mehta, Leonard Hussenot, Olivier Bachem, and Edouard Leurent. 202… [find](https://scholar.google.com/scholar?q=Kaiwen+Wang%2C+Rahul+Kidambi%2C+Ryan+Sullivan%2C+Alekh+Agarwal%2C+Christoph+Dann%2C+Andrea+Michi%2C+Marco+Gelmi%2C+Yunxuan+Li%2C+Raghav+Gupta%2C+Kumar+Avinava+Dubey%2C+Alexandre+Rame%2C+Johan+Ferret%2C+Geoffrey+Cideron%2C+Le+H)
43. Tianze Wang, Dongnan Gui, Yifan Hu, Shuhang Lin, and Linjun Zhang. 2025. Mpo: An efficient post-processing framework for mixing diverse preference alignment. [find](https://scholar.google.com/scholar?q=Tianze+Wang%2C+Dongnan+Gui%2C+Yifan+Hu%2C+Shuhang+Lin%2C+and+Linjun+Zhang%2E+2025%2E+Mpo%3A+An+efficient+post%2Dprocessing+framework+for+mixing+diverse+preference+alignment%2E)
44. Yue Wang, Wei Chen, Yuting Liu, Zhi-Ming Ma, and Tie-Yan Liu. 2017. Finite sample analysis of the gtd policy evaluation algorithms in markov setting. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Yue+Wang%2C+Wei+Chen%2C+Yuting+Liu%2C+Zhi%2DMing+Ma%2C+and+Tie%2DYan+Liu%2E+2017%2E+Finite+sample+analysis+of+the+gtd+policy+evaluation+algorithms+in+markov+setting%2E+In+Advances+in+Neural+Information+Processing+Syste)
45. Susan Wei and Marc Niethammer. 2022. The fairness-accuracy pareto front. Statistical Analysis and Data Mining: The ASA Data Science Journal, 15(3):287–302. [find](https://scholar.google.com/scholar?q=Susan+Wei+and+Marc+Niethammer%2E+2022%2E+The+fairness%2Daccuracy+pareto+front%2E+Statistical+Analysis+and+Data+Mining%3A+The+ASA+Data+Science+Journal%2C+15%283%29%3A287%E2%80%93302%2E)
46. Yue Frank Wu, Weitong ZHANG, Pan Xu, and Quanquan Gu. 2020. A finite-time analysis of two time-scale actor-critic methods. In Advances in Neural Information Processing Systems, volume 33, pages 17617–17628. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Yue+Frank+Wu%2C+Weitong+ZHANG%2C+Pan+Xu%2C+and+Quanquan+Gu%2E+2020%2E+A+finite%2Dtime+analysis+of+two+time%2Dscale+actor%2Dcritic+methods%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+33%2C+pages+17617%E2%80%93)
47. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2023. [Fine-grained human feedback gives better rewards for language model training](https://md.research.elamur.ai/free/en/papers/wu-2023-fine-grained-human-feedback-gives/). In Advances in Neural Information Processing Systems, volume 36, pages 59008– 59…
48. Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. 2025. [Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning](https://md.research.elamur.ai/free/en/papers/xie-2025-logic-rl-unleashing-llm/).
49. Nuoya Xiong and Aarti Singh. 2025. Projection optimization: A general framework for multi-objective and multi-group rlhf. [find](https://scholar.google.com/scholar?q=Nuoya+Xiong+and+Aarti+Singh%2E+2025%2E+Projection+optimization%3A+A+general+framework+for+multi%2Dobjective+and+multi%2Dgroup+rlhf%2E)
50. Mingjing Xu, Peizhong Ju, Jia Liu, and Haibo Yang. 2025. Psmgd: Periodic stochastic multi-gradient descent for fast multi-objective optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 39(20):21770–21778. [find](https://scholar.google.com/scholar?q=Mingjing+Xu%2C+Peizhong+Ju%2C+Jia+Liu%2C+and+Haibo+Yang%2E+2025%2E+Psmgd%3A+Periodic+stochastic+multi%2Dgradient+descent+for+fast+multi%2Dobjective+optimization%2E+Proceedings+of+the+AAAI+Conference+on+Artificial+Intel)
51. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jin… — [Qwen3 Technical Report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/)
52. Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Tianlin Zhang, and Sophia Ananiadou. 2024a. Metaaligner: Towards generalizable multi-objective alignment of language models. In Advances in Neural Information Processing Systems, volume 37, pages 34453–34486. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Kailai+Yang%2C+Zhiwei+Liu%2C+Qianqian+Xie%2C+Jimin+Huang%2C+Tianlin+Zhang%2C+and+Sophia+Ananiadou%2E+2024a%2E+Metaaligner%3A+Towards+generalizable+multi%2Dobjective+alignment+of+language+models%2E+In+Advances+in+Neural+I)
53. Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. 2024b. Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. In International Conference on Machine Learning, pages 56276– 56297. PMLR. [find](https://scholar.google.com/scholar?q=Rui+Yang%2C+Xiaoman+Pan%2C+Feng+Luo%2C+Shuang+Qiu%2C+Han+Zhong%2C+Dong+Yu%2C+and+Jianshu+Chen%2E+2024b%2E+Rewards%2Din%2Dcontext%3A+Multi%2Dobjective+alignment+of+foundation+models+with+dynamic+preference+adjustment%2E+In+Inte)
54. Feng Yao, Zilong Wang, Liyuan Liu, Junxia Cui, Li Zhong, Xiaohan Fu, Haohui Mai, Vish Krishnan, Jianfeng Gao, and Jingbo Shang. 2025. Training language models to generate quality code with program analysis feedback. [find](https://scholar.google.com/scholar?q=Feng+Yao%2C+Zilong+Wang%2C+Liyuan+Liu%2C+Junxia+Cui%2C+Li+Zhong%2C+Xiaohan+Fu%2C+Haohui+Mai%2C+Vish+Krishnan%2C+Jianfeng+Gao%2C+and+Jingbo+Shang%2E+2025%2E+Training+language+models+to+generate+quality+code+with+program+ana)
55. Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, Qiaobo Chen, Yinyuting Yin, Hao Zhang, Tengfei Shi, Liang Wang, Qiang Fu, Wei Yang, and Lanx-iao Huang. 2020. Mastering complex control in moba games with deep reinforcement learning.… [find](https://scholar.google.com/scholar?q=Deheng+Ye%2C+Zhao+Liu%2C+Mingfei+Sun%2C+Bei+Shi%2C+Peilin+Zhao%2C+Hao+Wu%2C+Hongsheng+Yu%2C+Shaojie+Yang%2C+Xipeng+Wu%2C+Qingwei+Guo%2C+Qiaobo+Chen%2C+Yinyuting+Yin%2C+Hao+Zhang%2C+Tengfei+Shi%2C+Liang+Wang%2C+Qiang+Fu%2C+Wei+Yang%2C+)
56. Jixiao Zhang and Chunsheng Zuo. 2025. [Grpo-lead: A difficulty-aware reinforcement learning approach for concise mathematical reasoning in language models](https://md.research.elamur.ai/free/en/papers/zhang-2025-grpo-lead-difficulty-aware-reinforcement/).
57. Yifan Zhong, Chengdong Ma, Xiaoyuan Zhang, Ziran Yang, Haojun Chen, Qingfu Zhang, Siyuan Qi, and Yaodong Yang. 2024. Panacea: Pareto alignment via preference adaptation for llms. In Advances in Neural Information Processing Systems, volume 37, pages 75522– 75558. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Yifan+Zhong%2C+Chengdong+Ma%2C+Xiaoyuan+Zhang%2C+Ziran+Yang%2C+Haojun+Chen%2C+Qingfu+Zhang%2C+Siyuan+Qi%2C+and+Yaodong+Yang%2E+2024%2E+Panacea%3A+Pareto+alignment+via+preference+adaptation+for+llms%2E+In+Advances+in+Neural)
58. Shiji Zhou, Wenpeng Zhang, Jiyan Jiang, Wenliang Zhong, Jinjie GU, and Wenwu Zhu. 2022. On the convergence of stochastic multi-objective gradient manipulation and beyond. In Advances in Neural Information Processing Systems, volume 35, pages 38103–38115. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Shiji+Zhou%2C+Wenpeng+Zhang%2C+Jiyan+Jiang%2C+Wenliang+Zhong%2C+Jinjie+GU%2C+and+Wenwu+Zhu%2E+2022%2E+On+the+convergence+of+stochastic+multi%2Dobjective+gradient+manipulation+and+beyond%2E+In+Advances+in+Neural+Informa)
59. Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. 2024. Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization. In Findings of the Association for Computational Linguistics: ACL 2024, pages 10586–10613, Bangkok, Thailand. Associati… [find](https://scholar.google.com/scholar?q=Zhanhui+Zhou%2C+Jie+Liu%2C+Jing+Shao%2C+Xiangyu+Yue%2C+Chao+Yang%2C+Wanli+Ouyang%2C+and+Yu+Qiao%2E+2024%2E+Beyond+one%2Dpreference%2Dfits%2Dall+alignment%3A+Multi%2Dobjective+direct+preference+optimization%2E+In+Findings+of+the+)

*Updated 1 November 2026*