---
title: "Understanding Conflicts in Multi-Objective Alignment through"
description: "The authors define REWARD CONSISTENCY (RC): a preference pair qualifies only if reward models score the… 4.63 points. 38 references."
authors: ["Zhihao Xu", "Yongqi Tong", "Xin Zhang", "Jun Zhou", "Xiting Wang"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.269/"
doi: "10.18653/v1/2026.findings-acl.269"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2026-understanding-conflicts-multi-objective-alignment/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2026-understanding-conflicts-multi-objective-alignment/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/55/) / Understanding Conflicts in Multi-Objective Alignment through Reward Consistency

# Understanding Conflicts in Multi-Objective Alignment through Reward Consistency

**Authors:** [Zhihao Xu](https://md.research.elamur.ai/free/en/authors/zhihao-xu/), [Yongqi Tong](https://md.research.elamur.ai/free/en/authors/yongqi-tong/), [Xin Zhang](https://md.research.elamur.ai/free/en/authors/xin-zhang/), [Jun Zhou](https://md.research.elamur.ai/free/en/authors/jun-zhou/) and [Xiting Wang](https://md.research.elamur.ai/free/en/authors/xiting-wang/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.269/) · [PDF](https://aclanthology.org/2026.findings-acl.269.pdf) · [DOI 10.18653/v1/2026.findings-acl.269](https://doi.org/10.18653/v1/2026.findings-acl.269) · [OpenAlex](https://openalex.org/W7166800331)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary

When a language model is aligned to several preferences at once, such as helpfulness and harmlessness, gains on one objective often reduce performance on another. Most earlier work treats this as an algorithmic problem. This paper instead examines the training data. The authors define REWARD CONSISTENCY (RC): a preference pair qualifies only if reward models score the chosen response above the rejected one on every objective. They prove that, for multi-objective DPO variants such as MODPO and SPO, the objectives' gradients do not oppose each other exactly when a sample is RC.

Building on this, they propose REWARD CONSISTENCY SAMPLING (RCS). It samples candidate responses, annotates them with rewards, and builds RC-compliant pairs. A relaxation strategy admits a controlled number of inconsistent pairs to keep the data diverse. Using plain DPO on the curated data, RCS beats both data-level and algorithm-level baselines in simultaneous and sequential alignment. The abstract reports a 23.07% average gain over the vanilla dataset. Because the method only changes the data, it can be combined with existing alignment algorithms.

## Key points

- **Pilot study:** About 60% of HelpSteer2 pairs violate RC for helpfulness plus harmlessness.
  - A harmlessness-tuned model was further tuned for helpfulness on UltraFeedback.
  - With the original data, its harmless rate fell from 90.38 to 56.53. With only the inconsistent pairs, it fell to 43.12.
  - RC-only pairs kept harmlessness at 90.96, but helpfulness reached only 43.35 (vs. 72.29 with the original data), likely because the subset is smaller.
- **Setup:** LLaMA-3-SFT, the ArmoRM-Llama3-8B-v0.1 reward model, and a training mix of SafeRLHF-10K with 10K UltraFeedback samples. Pairs are selected by the averaged reward gap across objectives.
- **Two objectives:** RCS reaches a 96.34% harmless rate, a 77.01% helpful win rate and an 86.67% average. This is 4.63 points above the next-best method (RSDPO-W). Baselines include DPO-Soups, MOD, MODPO and MO-ODPO.
- **Three objectives (adding truthfulness via TruthfulQA MC2):** RCS leads on all four metrics, scoring MC2 68.89 and a 78.69 average (+2.90). An appendix experiment adds conciseness as a fourth objective.
- **Limitations:** The method is tested mainly on LLaMA-3 and only on text generation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Renmin University of China](https://md.research.elamur.ai/free/en/universities/renmin-university-of-china/) — University, Beijing, China · 139 works · 17,752 citations
- [Ant Group](https://md.research.elamur.ai/free/en/companies/ant-group/) — Company, Hangzhou, China · 51 works · 1,211 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 626 works

## Cite

Cite as: Zhihao Xu, Yongqi Tong, Xin Zhang, Jun Zhou and Xiting Wang. 2026. Understanding Conflicts in Multi-Objective Alignment through Reward Consistency. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xu2026understanding,
  title = {{Understanding Conflicts in Multi-Objective Alignment through Reward Consistency}},
  author = {Zhihao Xu, Yongqi Tong, Xin Zhang, Jun Zhou and Xiting Wang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.269},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.269}
}
```

- [← previous Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization](https://md.research.elamur.ai/free/en/papers/xu-2026-understanding-preventing-entropy-collapse-rlvr/)
- [next → Understanding Emergent Misalignment via Feature Superposition Geometry](https://md.research.elamur.ai/free/en/papers/minegishi-2026-understanding-emergent-misalignment-feature-superposition/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/55/)

## References

38 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073.
2. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. 2023. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217. [find](https://scholar.google.com/scholar?q=Stephen+Casper%2C+Xander+Davies%2C+Claudia+Shi%2C+Thomas+Krendl+Gilbert%2C+J%C3%A9r%C3%A9my+Scheurer%2C+Javier+Rando%2C+Rachel+Freedman%2C+Tomasz+Korbak%2C+David+Lindner%2C+Pedro+Freire%2C+et+al%2E+2023%2E+Open+problems+and+fundamenta)
3. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. [find](https://scholar.google.com/scholar?q=Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Guanming+Yao%2C+Wei+Zhu%2C+Yuan+Ni%2C+Guotong+Xie%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2023%2E+Ultrafeedback%3A+Boosting+language+models+with+high%2Dquality+feedback%2E)
4. Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. 2023. [Safe rlhf: Safe reinforcement learning from human feedback](https://md.research.elamur.ai/free/en/papers/dai-2023-safe-rlhf-safe-reinforcement-learning/). arXiv preprint arXiv:2310.12773.
5. Xun Deng, Han Zhong, Rui Ai, Fuli Feng, Zheng Wang, and Xiangnan He. 2025. Less is more: Improving llm alignment via preference data selection. arXiv preprint arXiv:2502.14560. — [arXiv:2502.14560](https://arxiv.org/abs/2502.14560) [find](https://scholar.google.com/scholar?q=Xun+Deng%2C+Han+Zhong%2C+Rui+Ai%2C+Fuli+Feng%2C+Zheng+Wang%2C+and+Xiangnan+He%2E+2025%2E+Less+is+more%3A+Improving+llm+alignment+via+preference+data+selection%2E+arXiv+preprint+arXiv%3A2502%2E14560%2E+%E2%80%94+%5BarXiv%3A2502%2E14560%5D%28ht)
6. Hang Ding, Qiming Feng, Dongqi Liu, Qi Zhao, Tao Yao, Shuo Wang, Dongsheng Chen, Jian Li, Zhenye Gan, Jiangning Zhang, et al. 2025. Rolermbench & rolerm: Towards reward modeling for profile-based role play in dialogue systems. arXiv preprint arXiv:2512.10575. — [arXiv:2512.10575](https://arxiv.org/abs/2512.10575) [find](https://scholar.google.com/scholar?q=Hang+Ding%2C+Qiming+Feng%2C+Dongqi+Liu%2C+Qi+Zhao%2C+Tao+Yao%2C+Shuo+Wang%2C+Dongsheng+Chen%2C+Jian+Li%2C+Zhenye+Gan%2C+Jiangning+Zhang%2C+et+al%2E+2025%2E+Rolermbench+%26+rolerm%3A+Towards+reward+modeling+for+profile%2Dbased+role)
7. Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xianchao Wu, and Oleksii Kuchaiev. 2023. Steerlm: Attribute conditioned sft as an (user-steerable) alternative to rlhf. arXiv preprint arXiv:2310.05344. [find](https://scholar.google.com/scholar?q=Yi+Dong%2C+Zhilin+Wang%2C+Makesh+Narsimhan+Sreedhar%2C+Xianchao+Wu%2C+and+Oleksii+Kuchaiev%2E+2023%2E+Steerlm%3A+Attribute+conditioned+sft+as+an+%28user%2Dsteerable%29+alternative+to+rlhf%2E+arXiv+preprint+arXiv%3A2310%2E05344)
8. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
9. Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Zexu Sun, Bowen Sun, Huimin Chen, Ruobing Xie, Jie Zhou, Yankai Lin, et al. 2024. Controllable preference optimization: Toward controllable multi-objective alignment. arXiv preprint arXiv:2402.19085. [find](https://scholar.google.com/scholar?q=Yiju+Guo%2C+Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Zexu+Sun%2C+Bowen+Sun%2C+Huimin+Chen%2C+Ruobing+Xie%2C+Jie+Zhou%2C+Yankai+Lin%2C+et+al%2E+2024%2E+Controllable+preference+optimization%3A+Toward+controllable+multi%2Dobjective+)
10. Raghav Gupta, Ryan Sullivan, Yunxuan Li, Samrat Phatale, and Abhinav Rastogi. 2025. Robust multi-objective preference alignment with online dpo. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 27321–27329. [find](https://scholar.google.com/scholar?q=Raghav+Gupta%2C+Ryan+Sullivan%2C+Yunxuan+Li%2C+Samrat+Phatale%2C+and+Abhinav+Rastogi%2E+2025%2E+Robust+multi%2Dobjective+preference+alignment+with+online+dpo%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Int)
11. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). arXiv preprint arXiv:2106.09685.
12. Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Qiu, Boxun Li, and Yaodong Yang. 2024. Pku-saferlhf: A safety alignment preference dataset for llama family models. arXiv e-prints, pages arXiv–2406. [find](https://scholar.google.com/scholar?q=Jiaming+Ji%2C+Donghai+Hong%2C+Borong+Zhang%2C+Boyuan+Chen%2C+Josef+Dai%2C+Boren+Zheng%2C+Tianyi+Qiu%2C+Boxun+Li%2C+and+Yaodong+Yang%2E+2024%2E+Pku%2Dsaferlhf%3A+A+safety+alignment+preference+dataset+for+llama+family+models%2E+)
13. Saeed Khaki, JinJin Li, Lan Ma, Liu Yang, and Prathap Ramachandra. 2024. Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large language models. arXiv preprint arXiv:2402.10038. [find](https://scholar.google.com/scholar?q=Saeed+Khaki%2C+JinJin+Li%2C+Lan+Ma%2C+Liu+Yang%2C+and+Prathap+Ramachandra%2E+2024%2E+Rs%2Ddpo%3A+A+hybrid+rejection+sampling+and+direct+preference+optimization+method+for+alignment+of+large+language+models%2E+arXiv+pre)
14. Diederik P Kingma. 2014. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980.
15. Jongwoo Ko, Saket Dingliwal, Bhavana Ganesh, Sailik Sengupta, Sravan Bodapati, and Aram Galstyan. 2024. Sera: Self-reviewing and alignment of large language models using implicit reward margins. arXiv preprint arXiv:2410.09362. — [arXiv:2410.09362](https://arxiv.org/abs/2410.09362) [find](https://scholar.google.com/scholar?q=Jongwoo+Ko%2C+Saket+Dingliwal%2C+Bhavana+Ganesh%2C+Sailik+Sengupta%2C+Sravan+Bodapati%2C+and+Aram+Galstyan%2E+2024%2E+Sera%3A+Self%2Dreviewing+and+alignment+of+large+language+models+using+implicit+reward+margins%2E+arXiv)
16. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). Advances in Neural Information Processing Systems, 36.
17. Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. 2024. [Step-dpo: Step-wise preference optimization for long-chain reasoning of llms](https://md.research.elamur.ai/free/en/papers/lai-2024-step-dpo-step-wise-preference/). arXiv preprint arXiv:2406.18629.
18. Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. 2024. [Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards](https://md.research.elamur.ai/free/en/papers/rame-2023-rewarded-soups-towards-pareto-optimal/). Advances in Neural Information Processing Systems, 36.
19. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. 2024. [T\\" ulu 3: Pushing frontiers in open language model post-training](https://md.research.elamur.ai/free/en/papers/lambert-2024-tulu-3-pushing-frontiers-open/). arXiv preprint arXiv:2411.15124.
20. Meng Li, Guangda Huzhang, Haibo Zhang, Xiting Wang, and Anxiang Zeng. 2025a. Optimal transport-based token weighting scheme for enhanced preference optimization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 21311– 21334. [find](https://scholar.google.com/scholar?q=Meng+Li%2C+Guangda+Huzhang%2C+Haibo+Zhang%2C+Xiting+Wang%2C+and+Anxiang+Zeng%2E+2025a%2E+Optimal+transport%2Dbased+token+weighting+scheme+for+enhanced+preference+optimization%2E+In+Proceedings+of+the+63rd+Annual+Meet)
21. Moxin Li, Yuantao Zhang, Wenjie Wang, Wentao Shi, Zhuo Liu, Fuli Feng, and Tat-Seng Chua. 2025b. Self-improvement towards pareto optimality: Mitigating preference conflicts in multi-objective alignment. arXiv preprint arXiv:2502.14354. [find](https://scholar.google.com/scholar?q=Moxin+Li%2C+Yuantao+Zhang%2C+Wenjie+Wang%2C+Wentao+Shi%2C+Zhuo+Liu%2C+Fuli+Feng%2C+and+Tat%2DSeng+Chua%2E+2025b%2E+Self%2Dimprovement+towards+pareto+optimality%3A+Mitigating+preference+conflicts+in+multi%2Dobjective+alignmen)
22. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. [find](https://scholar.google.com/scholar?q=Xuechen+Li%2C+Tianyi+Zhang%2C+Yann+Dubois%2C+Rohan+Taori%2C+Ishaan+Gulrajani%2C+Carlos+Guestrin%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+2023%2E+Alpacaeval%3A+An+automatic+evaluator+of+instruction%2Dfollowing+models%2E)
23. Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). arXiv preprint arXiv:2109.07958.
24. Yong Lin, Skyler Seto, Maartje Ter Hoeve, Katherine Metcalf, Barry-John Theobald, Xuan Wang, Yizhe Zhang, Chen Huang, and Tong Zhang. 2024. On the limited generalization capability of the implicit reward model induced by direct preference optimization. arXiv preprint arXiv:2409.03650. [find](https://scholar.google.com/scholar?q=Yong+Lin%2C+Skyler+Seto%2C+Maartje+Ter+Hoeve%2C+Katherine+Metcalf%2C+Barry%2DJohn+Theobald%2C+Xuan+Wang%2C+Yizhe+Zhang%2C+Chen+Huang%2C+and+Tong+Zhang%2E+2024%2E+On+the+limited+generalization+capability+of+the+implicit+rew)
25. Ruizhe Shi, Yifang Chen, Yushi Hu, Alisa Liu, Hanna Hajishirzi, Noah A Smith, and Simon S Du. 2024. Decoding-time language model alignment with multiple objectives. Advances in Neural Information Processing Systems, 37:48875–48920. [find](https://scholar.google.com/scholar?q=Ruizhe+Shi%2C+Yifang+Chen%2C+Yushi+Hu%2C+Alisa+Liu%2C+Hanna+Hajishirzi%2C+Noah+A+Smith%2C+and+Simon+S+Du%2E+2024%2E+Decoding%2Dtime+language+model+alignment+with+multiple+objectives%2E+Advances+in+Neural+Information+Proc)
26. Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, et al. 2024. Understanding the performance gap between online and offline alignment algorithms. arXiv preprint arXiv:2405.08448. — [arXiv:2405.08448](https://arxiv.org/abs/2405.08448) [find](https://scholar.google.com/scholar?q=Yunhao+Tang%2C+Daniel+Zhaohan+Guo%2C+Zeyu+Zheng%2C+Daniele+Calandriello%2C+Yuan+Cao%2C+Eugene+Tarassov%2C+R%C3%A9mi+Munos%2C+Bernardo+%C3%81vila+Pires%2C+Michal+Valko%2C+Yong+Cheng%2C+et+al%2E+2024%2E+Understanding+the+performance+gap)
27. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. 2024. [Helpsteer2: Open-source dataset for training top-performing reward models](https://md.research.elamur.ai/free/en/papers/wang-2024-helpsteer2-open-source-dataset-training/). arXiv preprint arXiv:2406.08673.
28. Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024. [Less: Selecting influential data for targeted instruction tuning](https://md.research.elamur.ai/free/en/papers/xia-2024-less-selecting-influential-data-targeted/). arXiv preprint arXiv:2402.04333.
29. Wenyi Xiao, Zechuan Wang, Leilei Gan, Shuai Zhao, Wanggui He, Luu Anh Tuan, Long Chen, Hao Jiang, Zhou Zhao, and Fei Wu. 2024. A comprehensive survey of datasets, theories, variants, and applications in direct preference optimization. arXiv e-prints, pages arXiv–2410. [find](https://scholar.google.com/scholar?q=Wenyi+Xiao%2C+Zechuan+Wang%2C+Leilei+Gan%2C+Shuai+Zhao%2C+Wanggui+He%2C+Luu+Anh+Tuan%2C+Long+Chen%2C+Hao+Jiang%2C+Zhou+Zhao%2C+and+Fei+Wu%2E+2024%2E+A+comprehensive+survey+of+datasets%2C+theories%2C+variants%2C+and+applications+)
30. Xingzhou Lou, Junge Zhang, Jian Xie, Lifeng Liu, Dong Yan, and Kaiqi Huang. 2024. Spo: Multi-dimensional preference sequential alignment with implicit reward modeling. arXiv preprint arXiv:2405.12739. — [arXiv:2405.12739](https://arxiv.org/abs/2405.12739) [find](https://scholar.google.com/scholar?q=Xingzhou+Lou%2C+Junge+Zhang%2C+Jian+Xie%2C+Lifeng+Liu%2C+Dong+Yan%2C+and+Kaiqi+Huang%2E+2024%2E+Spo%3A+Multi%2Ddimensional+preference+sequential+alignment+with+implicit+reward+modeling%2E+arXiv+preprint+arXiv%3A2405%2E12739%2E)
31. Zhihao Xu, Ruixuan Huang, Changyu Chen, and Xiting Wang. 2024. Uncovering safety risks of large language models through concept activation vector. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Zhihao+Xu%2C+Ruixuan+Huang%2C+Changyu+Chen%2C+and+Xiting+Wang%2E+2024%2E+Uncovering+safety+risks+of+large+language+models+through+concept+activation+vector%2E+In+The+Thirty%2Deighth+Annual+Conference+on+Neural+Info)
32. Rui Pan, Dylan Zhang, Hanning Zhang, Xingyuan Pan, Minrui Xu, Jipeng Zhang, Renjie Pi, Xiaoyu Wang, and Tong Zhang. 2024. Scalebio: Scalable bilevel optimization for llm data reweighting. arXiv preprint arXiv:2406.19976. [find](https://scholar.google.com/scholar?q=Rui+Pan%2C+Dylan+Zhang%2C+Hanning+Zhang%2C+Xingyuan+Pan%2C+Minrui+Xu%2C+Jipeng+Zhang%2C+Renjie+Pi%2C+Xiaoyu+Wang%2C+and+Tong+Zhang%2E+2024%2E+Scalebio%3A+Scalable+bilevel+optimization+for+llm+data+reweighting%2E+arXiv+prepri)
33. Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. 2024. Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. arXiv preprint arXiv:2402.10207. [find](https://scholar.google.com/scholar?q=Rui+Yang%2C+Xiaoman+Pan%2C+Feng+Luo%2C+Shuang+Qiu%2C+Han+Zhong%2C+Dong+Yu%2C+and+Jianshu+Chen%2E+2024%2E+Rewards%2Din%2Dcontext%3A+Multi%2Dobjective+alignment+of+foundation+models+with+dynamic+preference+adjustment%2E+arXiv+pr)
34. Pulkit Pattnaik, Rishabh Maheshwary, Kelechi Ogueji, Vikas Yadav, and Sathwik Tejaswi Madhusudhan. 2024. Curry-dpo: Enhancing alignment using curriculum learning & ranked preferences. arXiv preprint arXiv:2403.07230. [find](https://scholar.google.com/scholar?q=Pulkit+Pattnaik%2C+Rishabh+Maheshwary%2C+Kelechi+Ogueji%2C+Vikas+Yadav%2C+and+Sathwik+Tejaswi+Madhusudhan%2E+2024%2E+Curry%2Ddpo%3A+Enhancing+alignment+using+curriculum+learning+%26+ranked+preferences%2E+arXiv+preprint+a)
35. Yifan Zhong, Chengdong Ma, Xiaoyuan Zhang, Ziran Yang, Haojun Chen, Qingfu Zhang, Siyuan Qi, and Yaodong Yang. 2024. Panacea: Pareto alignment via preference adaptation for llms. arXiv preprint arXiv:2402.02030. — [arXiv:2402.02030](https://arxiv.org/abs/2402.02030) [find](https://scholar.google.com/scholar?q=Yifan+Zhong%2C+Chengdong+Ma%2C+Xiaoyuan+Zhang%2C+Ziran+Yang%2C+Haojun+Chen%2C+Qingfu+Zhang%2C+Siyuan+Qi%2C+and+Yaodong+Yang%2E+2024%2E+Panacea%3A+Pareto+alignment+via+preference+adaptation+for+llms%2E+arXiv+preprint+arXiv%3A)
36. Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2024a. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Pengfei+Liu%2C+Puxin+Xu%2C+Srinivasan+Iyer%2C+Jiao+Sun%2C+Yuning+Mao%2C+Xuezhe+Ma%2C+Avia+Efrat%2C+Ping+Yu%2C+Lili+Yu%2C+et+al%2E+2024a%2E+Lima%3A+Less+is+more+for+alignment%2E+Advances+in+Neural+Information+Pro)
37. Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. 2024b. Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization. In Findings of the Association for Computational Linguistics ACL 2024, pages 10586–10613. [find](https://scholar.google.com/scholar?q=Zhanhui+Zhou%2C+Jie+Liu%2C+Jing+Shao%2C+Xiangyu+Yue%2C+Chao+Yang%2C+Wanli+Ouyang%2C+and+Yu+Qiao%2E+2024b%2E+Beyond+one%2Dpreference%2Dfits%2Dall+alignment%3A+Multi%2Dobjective+direct+preference+optimization%2E+In+Findings+of+the)
38. Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023. [Universal and transferable adversarial attacks on aligned language models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/). arXiv preprint arXiv:2307.15043.

*Updated 2 November 2026*