---
title: "Sem-DPO — paper by Mohamed et al. (2026)"
description: "Automatic prompt engineering for text-to-image generation often relies on preference tuning. 50 references."
authors: ["Anas Mohamed", "Azal Ahmad Khan", "Xinran Wang", "Ahmad Faraz Khan", "Shuwen Ge", "Saman Bahzad Khan", "Ayaan Ahmad", "Ali Anwar"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1184/"
doi: "10.18653/v1/2026.findings-acl.1184"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mohamed-2026-sem-dpo-mitigating-semantic-inconsistency/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mohamed-2026-sem-dpo-mitigating-semantic-inconsistency/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/42/) / Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering

# Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering

**Authors:** Anas Mohamed, Azal Ahmad Khan, Xinran Wang, Ahmad Faraz Khan, Shuwen Ge, Saman Bahzad Khan, Ayaan Ahmad and Ali Anwar · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1184/) · [PDF](https://aclanthology.org/2026.findings-acl.1184.pdf) · [DOI 10.18653/v1/2026.findings-acl.1184](https://doi.org/10.18653/v1/2026.findings-acl.1184) · [OpenAlex](https://openalex.org/W7166841412)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary

Automatic prompt engineering for text-to-image generation often relies on preference tuning. Standard DPO tends to produce rewritten prompts that score well with human-preference models but lose the meaning of the user's request, a failure the authors call semantic drift. They attribute it to DPO's token-level objective, which has no term for meaning. Their method, Sem-DPO, multiplies each preference triplet's loss by a weight that falls exponentially with the cosine distance between embeddings of the original prompt and the preferred rewrite. Pairs whose "winner" strays far from the input therefore contribute less to training. The weights are precomputed offline, so DPO's cost and simplicity are kept. The paper also gives what it claims is the first theoretical bound on semantic drift in preference-tuned prompt generators. Across three text-to-image prompt-optimization benchmarks and several language models, Sem-DPO improves both semantic fidelity (CLIP) and preference metrics over DPO, DPO variants and other prompt optimizers. This matters for multi-stage AI pipelines, where drift in one component can spread downstream.

## Key points

- **Weighting scheme:** W(x, y_w) = exp(−α · d(e_φ(x), e_φ(y_w))), where e_φ is a frozen pre-trained embedding model and d is cosine distance. After manual tuning, α ≈ 4 worked best. At α ≥ 10, too many useful gradients are suppressed (e.g., CLIP ≈ 0.266 at α = 15).
- **Gains over DPO:** CLIP similarity is 8–12% higher, and the human-preference scores HPSv2.1 and PickScore are 5–9% higher.
- **Human evaluation:** With 9 participants and 20 prompts, Sem-DPO won 148 of 180 pairwise comparisons (82.2%). Every participant favoured it, and it won on 18 of 20 prompts. An earlier round with 57 participants found 95.8% preference.
- **Ablation:** Using Qwen 1.5-1.8B on the Lexica dataset, CLIP and HPSv2.1 correlate strongly (r = 0.84). The correlation between CLIP and PickScore is weaker (r = 0.35).
- **Limitations:** Results depend on the choice of embedding model and on α, and cosine distance is only a proxy for meaning. Bias in CLIP embeddings and in the preference data can also carry over into the weights.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of California, Santa Cruz](https://md.research.elamur.ai/free/en/universities/university-of-california-santa-cruz/) — University, Santa Cruz, United States · 36 works · 3,515 citations
- [Virginia Tech](https://md.research.elamur.ai/free/en/universities/virginia-tech/) — University, Blacksburg, United States · 36 works · 43,036 citations
- [University of Minnesota](https://md.research.elamur.ai/free/en/universities/university-of-minnesota/) — University, Minneapolis, United States · 28 works · 2,184 citations
- [Lahore University of Management Sciences](https://md.research.elamur.ai/free/en/universities/lahore-university-of-management-sciences/) — University, Lahore, Pakistan · 3 works · 61 citations

Also on the byline: [Xi'an University of Technology](https://ror.org/038avdt50).

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

## Cite

Cite as: Anas Mohamed, Azal Ahmad Khan, Xinran Wang, Ahmad Faraz Khan, Shuwen Ge, Saman Bahzad Khan, Ayaan Ahmad and Ali Anwar. 2026. Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mohamed2026sem,
  title = {{Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering}},
  author = {Anas Mohamed, Azal Ahmad Khan, Xinran Wang, Ahmad Faraz Khan, Shuwen Ge, Saman Bahzad Khan, Ayaan Ahmad and Ali Anwar},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1184},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1184}
}
```

- [← previous Self-Sum: Teaching an Agent to Decide Itself When and What to Summarize](https://md.research.elamur.ai/free/en/papers/wang-2026-self-sum-teaching-agent-decide/)
- [next → Semantic Clustering of Obfuscated Command-Line Detections for Alert Reduction](https://md.research.elamur.ai/free/en/papers/stepankova-2026-semantic-clustering-obfuscated-command-line/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/42/)

## References

50 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. 2024. [On-policy distillation of language models: Learning from self-generated mistakes](https://md.research.elamur.ai/free/en/papers/agarwal-2023-policy-distillation-language-models-learning/). In The Twelfth International Conference on Learning Representations.
2. Afra Amini, Tim Vieira, and Ryan Cotterell. 2024. Direct preference optimization with an offset. arXiv preprint arXiv:2402.10571. [find](https://scholar.google.com/scholar?q=Afra+Amini%2C+Tim+Vieira%2C+and+Ryan+Cotterell%2E+2024%2E+Direct+preference+optimization+with+an+offset%2E+arXiv+preprint+arXiv%3A2402%2E10571%2E)
3. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+2022%2E+Training+a+helpful+and+harmless+assistant+with+rein)
4. Miaomiao Cai, Simiao Li, Wei Li, Xudong Huang, Hanting Chen, Jie Hu, and Yunhe Wang. 2025. Dspo: Direct semantic preference optimization for real-world image super-resolution. arXiv preprint arXiv:2504.15176. — [arXiv:2504.15176](https://arxiv.org/abs/2504.15176) [find](https://scholar.google.com/scholar?q=Miaomiao+Cai%2C+Simiao+Li%2C+Wei+Li%2C+Xudong+Huang%2C+Hanting+Chen%2C+Jie+Hu%2C+and+Yunhe+Wang%2E+2025%2E+Dspo%3A+Direct+semantic+preference+optimization+for+real%2Dworld+image+super%2Dresolution%2E+arXiv+preprint+arXiv%3A250)
5. Tingfeng Cao, Chengyu Wang, Bingyan Liu, Ziheng Wu, Jinhui Zhu, and Jun Huang. 2023. Beautifulprompt: Towards automatic prompt engineering for text-to-image synthesis. arXiv preprint arXiv:2311.06752. [find](https://scholar.google.com/scholar?q=Tingfeng+Cao%2C+Chengyu+Wang%2C+Bingyan+Liu%2C+Ziheng+Wu%2C+Jinhui+Zhu%2C+and+Jun+Huang%2E+2023%2E+Beautifulprompt%3A+Towards+automatic+prompt+engineering+for+text%2Dto%2Dimage+synthesis%2E+arXiv+preprint+arXiv%3A2311%2E06752%2E)
6. Amitava Das, Suranjana Trivedy, Danush Khanna, Rajarshi Roy, Gurpreet Singh, Basab Ghosh, Yaswanth Narsupalli, Vinija Jain, Vasu Sharma, Aishwarya Naresh Reganti, et al. 2025. Dpo kernels: A semantically-aware, kernel-enhanced, and divergence-rich paradigm for direct preference optimization. arXiv p… [find](https://scholar.google.com/scholar?q=Amitava+Das%2C+Suranjana+Trivedy%2C+Danush+Khanna%2C+Rajarshi+Roy%2C+Gurpreet+Singh%2C+Basab+Ghosh%2C+Yaswanth+Narsupalli%2C+Vinija+Jain%2C+Vasu+Sharma%2C+Aishwarya+Naresh+Reganti%2C+et+al%2E+2025%2E+Dpo+kernels%3A+A+semantica)
7. Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou. 2024. Self-play with execution feedback: Improving instruction-following capabilities of large language models. arXiv preprint arXiv:2406.13542. [find](https://scholar.google.com/scholar?q=Guanting+Dong%2C+Keming+Lu%2C+Chengpeng+Li%2C+Tingyu+Xia%2C+Bowen+Yu%2C+Chang+Zhou%2C+and+Jingren+Zhou%2E+2024%2E+Self%2Dplay+with+execution+feedback%3A+Improving+instruction%2Dfollowing+capabilities+of+large+language+mode)
8. Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306. [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2C+Winnie+Xu%2C+Niklas+Muennighoff%2C+Dan+Jurafsky%2C+and+Douwe+Kiela%2E+2024%2E+Kto%3A+Model+alignment+as+prospect+theoretic+optimization%2E+arXiv+preprint+arXiv%3A2402%2E01306%2E)
9. Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. 2022. Optimizing prompts for text-to-image generation. arXiv preprint arXiv:2212.09611. — [arXiv:2212.09611](https://arxiv.org/abs/2212.09611) [find](https://scholar.google.com/scholar?q=Yaru+Hao%2C+Zewen+Chi%2C+Li+Dong%2C+and+Furu+Wei%2E+2022%2E+Optimizing+prompts+for+text%2Dto%2Dimage+generation%2E+arXiv+preprint+arXiv%3A2212%2E09611%2E+%E2%80%94+%5BarXiv%3A2212%2E09611%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2212%2E09611%29)
10. Jiwoo Hong, Noah Lee, and James Thorne. 2024. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691. [find](https://scholar.google.com/scholar?q=Jiwoo+Hong%2C+Noah+Lee%2C+and+James+Thorne%2E+2024%2E+Orpo%3A+Monolithic+preference+optimization+without+reference+model%2E+arXiv+preprint+arXiv%3A2403%2E07691%2E)
11. Yuu Jinnai and Ukyo Honda. 2024. Annotation-efficient preference optimization for language model alignment. arXiv preprint arXiv:2405.13541. — [arXiv:2405.13541](https://arxiv.org/abs/2405.13541) [find](https://scholar.google.com/scholar?q=Yuu+Jinnai+and+Ukyo+Honda%2E+2024%2E+Annotation%2Defficient+preference+optimization+for+language+model+alignment%2E+arXiv+preprint+arXiv%3A2405%2E13541%2E+%E2%80%94+%5BarXiv%3A2405%2E13541%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2405%2E13541%29)
12. Sunghee Jung, Donghun Lee, Shinbok Lee, Gaeun Seo, Daniel Lee, Byeongil Ko, Junrae Cho, Kihyun Kim, Eunggyun Kim, and Myeongcheol Shin. 2025. Diatool-dpo: Multi-turn direct preference optimization for tool-augmented large language models. arXiv preprint arXiv:2504.02882. [find](https://scholar.google.com/scholar?q=Sunghee+Jung%2C+Donghun+Lee%2C+Shinbok+Lee%2C+Gaeun+Seo%2C+Daniel+Lee%2C+Byeongil+Ko%2C+Junrae+Cho%2C+Kihyun+Kim%2C+Eunggyun+Kim%2C+and+Myeongcheol+Shin%2E+2025%2E+Diatool%2Ddpo%3A+Multi%2Dturn+direct+preference+optimization+for)
13. Saeed Khaki, JinJin Li, Lan Ma, Liu Yang, and Prathap Ramachandra. 2024. Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large language models. arXiv preprint arXiv:2402.10038. [find](https://scholar.google.com/scholar?q=Saeed+Khaki%2C+JinJin+Li%2C+Lan+Ma%2C+Liu+Yang%2C+and+Prathap+Ramachandra%2E+2024%2E+Rs%2Ddpo%3A+A+hybrid+rejection+sampling+and+direct+preference+optimization+method+for+alignment+of+large+language+models%2E+arXiv+pre)
14. Azal Ahmad Khan, Sayan Alam, Xinran Wang, Ahmad Faraz Khan, Debanga Raj Neog, and Ali Anwar. 2024. Mitigating sycophancy in large language models via direct preference optimization. In 2024 IEEE International Conference on Big Data (Big-Data), pages 1664–1671. IEEE. [find](https://scholar.google.com/scholar?q=Azal+Ahmad+Khan%2C+Sayan+Alam%2C+Xinran+Wang%2C+Ahmad+Faraz+Khan%2C+Debanga+Raj+Neog%2C+and+Ali+Anwar%2E+2024%2E+Mitigating+sycophancy+in+large+language+models+via+direct+preference+optimization%2E+In+2024+IEEE+Inter)
15. Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. 2024. Pick-a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Yuval+Kirstain%2C+Adam+Polyak%2C+Uriel+Singer%2C+Shahbuland+Matiana%2C+Joe+Penna%2C+and+Omer+Levy%2E+2024%2E+Pick%2Da%2Dpic%3A+An+open+dataset+of+user+preferences+for+text%2Dto%2Dimage+generation%2E+Advances+in+Neural+Informat)
16. Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. 2023. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. [find](https://scholar.google.com/scholar?q=Harrison+Lee%2C+Samrat+Phatale%2C+Hassan+Mansoor%2C+Kellie+Ren+Lu%2C+Thomas+Mesnard%2C+Johan+Ferret%2C+Colton+Bishop%2C+Ethan+Hall%2C+Victor+Carbune%2C+and+Abhinav+Rastogi%2E+2023%2E+Rlaif%3A+Scaling+reinforcement+learning+f)
17. Adam Dahlgren Lindström, Leila Methnani, Lea Krause, Petter Ericson, Íñigo Martínez de Rituerto de Troya, Dimitri Coelho Mollo, and Roel Dobbe. 2024. Ai alignment through reinforcement learning from human feedback? contradictions and limitations. arXiv preprint arXiv:2406.18346. — [arXiv:2406.18346](https://arxiv.org/abs/2406.18346) [find](https://scholar.google.com/scholar?q=Adam+Dahlgren+Lindstr%C3%B6m%2C+Leila+Methnani%2C+Lea+Krause%2C+Petter+Ericson%2C+%C3%8D%C3%B1igo+Mart%C3%ADnez+de+Rituerto+de+Troya%2C+Dimitri+Coelho+Mollo%2C+and+Roel+Dobbe%2E+2024%2E+Ai+alignment+through+reinforcement+learning+from+h)
18. Junru Lu, Siyu An, Min Zhang, Yulan He, Di Yin, and Xing Sun. 2024. Fipo: Free-form instruction-oriented prompt optimization with preference dataset and modular fine-tuning schema. arXiv preprint arXiv:2402.11811. [find](https://scholar.google.com/scholar?q=Junru+Lu%2C+Siyu+An%2C+Min+Zhang%2C+Yulan+He%2C+Di+Yin%2C+and+Xing+Sun%2E+2024%2E+Fipo%3A+Free%2Dform+instruction%2Doriented+prompt+optimization+with+preference+dataset+and+modular+fine%2Dtuning+schema%2E+arXiv+preprint+arXi)
19. Motoki Omura, Yasuhiro Fujita, and Toshiki Kataoka. 2024. Entropy controllable direct preference optimization. arXiv preprint arXiv:2411.07595. — [arXiv:2411.07595](https://arxiv.org/abs/2411.07595) [find](https://scholar.google.com/scholar?q=Motoki+Omura%2C+Yasuhiro+Fujita%2C+and+Toshiki+Kataoka%2E+2024%2E+Entropy+controllable+direct+preference+optimization%2E+arXiv+preprint+arXiv%3A2411%2E07595%2E+%E2%80%94+%5BarXiv%3A2411%2E07595%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2411%2E07595%29)
20. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744.
21. Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. 2024. [Disentangling length from quality in direct preference optimization](https://md.research.elamur.ai/free/en/papers/park-2024-disentangling-length-quality-direct-preference/). arXiv preprint arXiv:2403.19159.
22. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). arXiv preprint arXiv:2305.18290.
23. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. [Hierarchical text-conditional image generation with clip latents](https://md.research.elamur.ai/free/en/papers/ramesh-2022-hierarchical-text-conditional-image-generation/). arXiv preprint arXiv:2204.06125, 1(2):3.
24. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. [Zero-shot text-to-image generation](https://md.research.elamur.ai/free/en/papers/ramesh-2021-zero-shot-text-image-generation/). In International Conference on Machine Learning, pages 8821–8831. PMLR.
25. Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. 2024. Unintentional unalignment: Likelihood displacement in direct preference optimization. arXiv preprint arXiv:2410.08847. [find](https://scholar.google.com/scholar?q=Noam+Razin%2C+Sadhika+Malladi%2C+Adithya+Bhaskar%2C+Danqi+Chen%2C+Sanjeev+Arora%2C+and+Boris+Hanin%2E+2024%2E+Unintentional+unalignment%3A+Likelihood+displacement+in+direct+preference+optimization%2E+arXiv+preprint+arX)
26. Laria Reynolds and Kyle McDonell. 2021. [Prompt programming for large language models: Beyond the few-shot paradigm](https://md.research.elamur.ai/free/en/papers/reynolds-2021-prompt-programming-large-language-models/). In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–7.
27. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695. [find](https://scholar.google.com/scholar?q=Robin+Rombach%2C+Andreas+Blattmann%2C+Dominik+Lorenz%2C+Patrick+Esser%2C+and+Bj%C3%B6rn+Ommer%2E+2022%2E+High%2Dresolution+image+synthesis+with+latent+diffusion+models%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+compu)
28. Amir Saeidi, Shivanshu Verma, Md Nayem Uddin, and Chitta Baral. 2024. Insights into alignment: Evaluating dpo and its variants across multiple tasks. arXiv preprint arXiv:2404.14723. [find](https://scholar.google.com/scholar?q=Amir+Saeidi%2C+Shivanshu+Verma%2C+Md+Nayem+Uddin%2C+and+Chitta+Baral%2E+2024%2E+Insights+into+alignment%3A+Evaluating+dpo+and+its+variants+across+multiple+tasks%2E+arXiv+preprint+arXiv%3A2404%2E14723%2E)
29. Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. 2022. [Photorealistic text-to-image diffusion models with deep language understanding](https://md.research.elamur.ai/free/en/papers/saharia-2022-photorealistic-text-image-diffusion-models/). Advances in Neural Information Processing Syste…
30. Hendrik Schuff, Lindsey Vanderlyn, Heike Adel, and Ngoc Thang Vu. 2023. How to do human evaluation: A brief introduction to user studies in nlp. Natural Language Engineering, 29(5):1199–1222. [find](https://scholar.google.com/scholar?q=Hendrik+Schuff%2C+Lindsey+Vanderlyn%2C+Heike+Adel%2C+and+Ngoc+Thang+Vu%2E+2023%2E+How+to+do+human+evaluation%3A+A+brief+introduction+to+user+studies+in+nlp%2E+Natural+Language+Engineering%2C+29%285%29%3A1199%E2%80%931222%2E)
31. Zhao Shan, Chenyou Fan, Shuang Qiu, Jiyuan Shi, and Chenjia Bai. 2025. Forward kl regularized preference optimization for aligning diffusion policies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 14386–14395. [find](https://scholar.google.com/scholar?q=Zhao+Shan%2C+Chenyou+Fan%2C+Shuang+Qiu%2C+Jiyuan+Shi%2C+and+Chenjia+Bai%2E+2025%2E+Forward+kl+regularized+preference+optimization+for+aligning+diffusion+policies%2E+In+Proceedings+of+the+AAAI+Conference+on+Artifici)
32. Shivanshu Shekhar, Shreyas Singh, and Tong Zhang. 2024. See-dpo: Self entropy enhanced direct preference optimization. arXiv preprint arXiv:2411.04712. — [arXiv:2411.04712](https://arxiv.org/abs/2411.04712) [find](https://scholar.google.com/scholar?q=Shivanshu+Shekhar%2C+Shreyas+Singh%2C+and+Tong+Zhang%2E+2024%2E+See%2Ddpo%3A+Self+entropy+enhanced+direct+preference+optimization%2E+arXiv+preprint+arXiv%3A2411%2E04712%2E+%E2%80%94+%5BarXiv%3A2411%2E04712%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2411%2E)
33. Xiaohang Tang, Sangwoong Yoon, Seongho Son, Huizhuo Yuan, Quanquan Gu, and Ilija Bogunovic. 2025. Game-theoretic regularized self-play alignment of large language models. arXiv preprint arXiv:2503.00030. — [arXiv:2503.00030](https://arxiv.org/abs/2503.00030) [find](https://scholar.google.com/scholar?q=Xiaohang+Tang%2C+Sangwoong+Yoon%2C+Seongho+Son%2C+Huizhuo+Yuan%2C+Quanquan+Gu%2C+and+Ilija+Bogunovic%2E+2025%2E+Game%2Dtheoretic+regularized+self%2Dplay+alignment+of+large+language+models%2E+arXiv+preprint+arXiv%3A2503%2E000)
34. Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. 2024. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit… [find](https://scholar.google.com/scholar?q=Bram+Wallace%2C+Meihua+Dang%2C+Rafael+Rafailov%2C+Linqi+Zhou%2C+Aaron+Lou%2C+Senthil+Purushwalkam%2C+Stefano+Ermon%2C+Caiming+Xiong%2C+Shafiq+Joty%2C+and+Nikhil+Naik%2E+2024%2E+Diffusion+model+alignment+using+direct+prefer)
35. Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. 2023. Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints. arXiv preprint arXiv:2309.16240. — [arXiv:2309.16240](https://arxiv.org/abs/2309.16240) [find](https://scholar.google.com/scholar?q=Chaoqi+Wang%2C+Yibo+Jiang%2C+Chenghao+Yang%2C+Han+Liu%2C+and+Yuxin+Chen%2E+2023%2E+Beyond+reverse+kl%3A+Generalizing+direct+preference+optimization+with+diverse+divergence+constraints%2E+arXiv+preprint+arXiv%3A2309%2E162)
36. Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. 2024. Beta-dpo: Direct preference optimization with dynamic beta. Advances in Neural Information Processing Systems, 37:129944–129966. [find](https://scholar.google.com/scholar?q=Junkang+Wu%2C+Yuexiang+Xie%2C+Zhengyi+Yang%2C+Jiancan+Wu%2C+Jinyang+Gao%2C+Bolin+Ding%2C+Xiang+Wang%2C+and+Xiangnan+He%2E+2024%2E+Beta%2Ddpo%3A+Direct+preference+optimization+with+dynamic+beta%2E+Advances+in+Neural+Informati)
37. Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. 2023. Human preference score: Better aligning text-to-image models with human preference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2096–2105. [find](https://scholar.google.com/scholar?q=Xiaoshi+Wu%2C+Keqiang+Sun%2C+Feng+Zhu%2C+Rui+Zhao%2C+and+Hongsheng+Li%2E+2023%2E+Human+preference+score%3A+Better+aligning+text%2Dto%2Dimage+models+with+human+preference%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Co)
38. Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, and Vasant G Honavar. 2024. Cal-dpo: Calibrated direct preference optimization for language model alignment. arXiv preprint arXiv:2412.14516. — [arXiv:2412.14516](https://arxiv.org/abs/2412.14516) [find](https://scholar.google.com/scholar?q=Teng+Xiao%2C+Yige+Yuan%2C+Huaisheng+Zhu%2C+Mingxiao+Li%2C+and+Vasant+G+Honavar%2E+2024%2E+Cal%2Ddpo%3A+Calibrated+direct+preference+optimization+for+language+model+alignment%2E+arXiv+preprint+arXiv%3A2412%2E14516%2E+%E2%80%94+%5BarXiv)
39. Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, et al. 2024a. Visionreward: Fine-grained multi-dimensional human preference learning for image and video generation. arXiv preprint arXiv:2412.21059. — [arXiv:2412.21059](https://arxiv.org/abs/2412.21059) [find](https://scholar.google.com/scholar?q=Jiazheng+Xu%2C+Yu+Huang%2C+Jiale+Cheng%2C+Yuanming+Yang%2C+Jiajun+Xu%2C+Yuan+Wang%2C+Wenbo+Duan%2C+Shen+Yang%2C+Qunlin+Jin%2C+Shurun+Li%2C+et+al%2E+2024a%2E+Visionreward%3A+Fine%2Dgrained+multi%2Ddimensional+human+preference+learn)
40. Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. 2024b. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Jiazheng+Xu%2C+Xiao+Liu%2C+Yuchen+Wu%2C+Yuxuan+Tong%2C+Qinkai+Li%2C+Ming+Ding%2C+Jie+Tang%2C+and+Yuxiao+Dong%2E+2024b%2E+Imagereward%3A+Learning+and+evaluating+human+preferences+for+text%2Dto%2Dimage+generation%2E+Advances+in+)
41. Chenxu Yang, Ruipeng Jia, Naibin Gu, Zheng Lin, Siyuan Chen, Chao Pang, Weichong Yin, Yu Sun, Hua Wu, and Weiping Wang. 2024a. Orthogonal finetuning for direct preference optimization. arXiv preprint arXiv:2409.14836. — [arXiv:2409.14836](https://arxiv.org/abs/2409.14836) [find](https://scholar.google.com/scholar?q=Chenxu+Yang%2C+Ruipeng+Jia%2C+Naibin+Gu%2C+Zheng+Lin%2C+Siyuan+Chen%2C+Chao+Pang%2C+Weichong+Yin%2C+Yu+Sun%2C+Hua+Wu%2C+and+Weiping+Wang%2E+2024a%2E+Orthogonal+finetuning+for+direct+preference+optimization%2E+arXiv+preprint+)
42. Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020. [Denoising diffusion implicit models](https://md.research.elamur.ai/free/en/papers/song-2020-denoising-diffusion-implicit-models/). arXiv preprint arXiv:2010.02502.
43. Jiuding Yang, Weidong Guo, Kaitong Yang, Xiangyang Li, Yu Xu, and Di Niu. 2024b. Enhancing and assessing instruction-following with fine-grained instruction variants. arXiv preprint arXiv:2406.11301. — [arXiv:2406.11301](https://arxiv.org/abs/2406.11301) [find](https://scholar.google.com/scholar?q=Jiuding+Yang%2C+Weidong+Guo%2C+Kaitong+Yang%2C+Xiangyang+Li%2C+Yu+Xu%2C+and+Di+Niu%2E+2024b%2E+Enhancing+and+assessing+instruction%2Dfollowing+with+fine%2Dgrained+instruction+variants%2E+arXiv+preprint+arXiv%3A2406%2E11301%2E+)
44. Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang, Baohua Dong, Ran Lin, and Ruohui Huang. 2024. [Conifer: Improving complex constrained instruction-following ability of large language models](https://md.research.elamur.ai/free/en/papers/sun-2024-conifer-improving-complex-constrained-instruction/). arXiv preprint arXiv:2404.02823.
45. Qingyu Yin, Chak Tou Leong, Hongbo Zhang, Minjun Zhu, Hanqi Yan, Qiang Zhang, Yulan He, Wenjie Li, Jun Wang, Yue Zhang, et al. 2024. Direct preference optimization using sparse feature-level constraints. arXiv preprint arXiv:2411.07618. — [arXiv:2411.07618](https://arxiv.org/abs/2411.07618) [find](https://scholar.google.com/scholar?q=Qingyu+Yin%2C+Chak+Tou+Leong%2C+Hongbo+Zhang%2C+Minjun+Zhu%2C+Hanqi+Yan%2C+Qiang+Zhang%2C+Yulan+He%2C+Wenjie+Li%2C+Jun+Wang%2C+Yue+Zhang%2C+et+al%2E+2024%2E+Direct+preference+optimization+using+sparse+feature%2Dlevel+constrain)
46. Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. 2022. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2(3):5. — [arXiv:2206.10789](https://arxiv.org/abs/2206.10789) [find](https://scholar.google.com/scholar?q=Jiahui+Yu%2C+Yuanzhong+Xu%2C+Jing+Yu+Koh%2C+Thang+Luong%2C+Gunjan+Baid%2C+Zirui+Wang%2C+Vijay+Vasudevan%2C+Alexander+Ku%2C+Yinfei+Yang%2C+Burcu+Karagol+Ayan%2C+et+al%2E+2022%2E+Scaling+autoregressive+models+for+content%2Drich+)
47. Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. 2023. Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization. arXiv preprint arXiv:2311.16839. — [arXiv:2311.16839](https://arxiv.org/abs/2311.16839) [find](https://scholar.google.com/scholar?q=Zhiyuan+Zhao%2C+Bin+Wang%2C+Linke+Ouyang%2C+Xiaoyi+Dong%2C+Jiaqi+Wang%2C+and+Conghui+He%2E+2023%2E+Beyond+hallucinations%3A+Enhancing+lvlms+through+hallucination%2Daware+direct+preference+optimization%2E+arXiv+preprint+a)
48. Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, and Chenguang Zhu. 2024a. Wpo: Enhancing rlhf with weighted preference optimization. arXiv preprint arXiv:2406.11827. [find](https://scholar.google.com/scholar?q=Wenxuan+Zhou%2C+Ravi+Agrawal%2C+Shujian+Zhang%2C+Sathish+Reddy+Indurthi%2C+Sanqiang+Zhao%2C+Kaiqiang+Song%2C+Silei+Xu%2C+and+Chenguang+Zhu%2E+2024a%2E+Wpo%3A+Enhancing+rlhf+with+weighted+preference+optimization%2E+arXiv+pr)
49. Wenxuan Zhou, Shujian Zhang, Lingxiao Zhao, and Tao Meng. 2024b. T-reg: Preference optimization with token-level reward regularization. arXiv preprint arXiv:2412.02685. [find](https://scholar.google.com/scholar?q=Wenxuan+Zhou%2C+Shujian+Zhang%2C+Lingxiao+Zhao%2C+and+Tao+Meng%2E+2024b%2E+T%2Dreg%3A+Preference+optimization+with+token%2Dlevel+reward+regularization%2E+arXiv+preprint+arXiv%3A2412%2E02685%2E)
50. Zixiao Zhu, Hanzhang Zhou, Zijian Feng, Tianjiao Li, Chua Jia Jim Deryl, Mak Lee Onn, Gee Wah Ng, and Kezhi Mao. 2025. [Rethinking prompt optimizers: From prompt merits to optimization](https://md.research.elamur.ai/free/en/papers/zhu-2026-rethinking-prompt-optimizers-prompt-merits/). arXiv preprint arXiv:2505.09930.

*Updated 1 November 2026*