---
title: "S2H-DPO — paper by Shukla et al. (2026) — summary and ideas"
description: "The authors argue that existing multi-image preference alignment, notably MIA-DPO, trains only on questions that… 6.94 points. 46 references."
authors: ["Nitish Shukla", "Surgan Jandial", "Arun Ross"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1825/"
doi: "10.18653/v1/2026.findings-acl.1825"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shukla-2026-s2h-dpo-hardness-aware-preference/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shukla-2026-s2h-dpo-hardness-aware-preference/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/42/) / S2H-DPO: Hardness-Aware Preference Optimization for Vision–Language Models

# S2H-DPO: Hardness-Aware Preference Optimization for Vision–Language Models

**Authors:** Nitish Shukla, Surgan Jandial and Arun Ross · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1825/) · [PDF](https://aclanthology.org/2026.findings-acl.1825.pdf) · [DOI 10.18653/v1/2026.findings-acl.1825](https://doi.org/10.18653/v1/2026.findings-acl.1825) · [OpenAlex](https://openalex.org/W7166833701)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Open-source vision–language models still struggle to reason over several images at once. The authors argue that existing multi-image preference alignment, notably MIA-DPO, trains only on questions that name the image to inspect. That setup never teaches a model to compare images on its own or to search across all of them.

S2H-DPO addresses this with a Simple-to-Hard hierarchy of synthetic preference data built from single-image sources such as LLaVA 665k and ImageNet. The three levels are:
- **L1:** questions about one indexed image.
- **L2:** comparisons across several indexed images.
- **L3:** open-ended global search, where the model must find which image meets a condition.

Chosen/rejected pairs come from prompt construction rather than from a particular model's hallucinations, so the data can be reused across architectures. LLaVA and Qwen-VL models are then trained jointly on all levels with DPO.

The method beats MIA-DPO, with the clearest gains on harder L3-style tasks, and keeps single-image performance at the original level. This suggests that covering more kinds of reasoning, not just adding more data, drives multi-image alignment.

## Key points
- Evaluation uses BLINK, MANTIS and NLVR2. For LLaVA-v1.5, mean accuracy rises from 43.70% to 46.92% with SFT on the same data and to 48.97% with DPO.
- Mixing levels in flat training beats curriculum ordering. For example, L2 flat training reaches 48.13% versus 44.83% for L1→L2, and the full L1→L2→L3 curriculum reaches only 45.55%.
- L2 data includes Kinship Recognition and Visual Arithmetic tasks. For L3, using 3 distractor images gave the best results, compared with 2, 4 or 5.
- On 500 L3-style ImageNet validation questions, the tuned model scores 31.05%, which is 6.94 points above baseline LLaVA-v1.5.
- On Qwen3-VL-2B, GRPO with simple rewards improves on DPO on MANTIS and NLVR2 but drops on BLINK.
- Stated limitations:
  - The pipeline depends on existing datasets and a captioning model, whose biases can carry over.
  - Experiments use at most 6 images.
  - Temporal and causal reasoning across images are not covered.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 539 works · 179,533 citations
- [Michigan State University](https://md.research.elamur.ai/free/en/universities/michigan-state-university/) — University, East Lansing, United States · 23 works · 1,077 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works

## Cite

Cite as: Nitish Shukla, Surgan Jandial and Arun Ross. 2026. S2H-DPO: Hardness-Aware Preference Optimization for Vision–Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shukla2026s2h,
  title = {{S2H-DPO: Hardness-Aware Preference Optimization for Vision–Language Models}},
  author = {Nitish Shukla, Surgan Jandial and Arun Ross},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1825},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1825}
}
```

- [← previous S-VoCAL: A Dataset and Evaluation Framework for Inferring Speaking Voice Character Attributes in Literature](https://md.research.elamur.ai/free/en/papers/pardo-2026-s-vocal-dataset-evaluation-framework/)
- [next → S2ST-Omni: Hierarchical Language-Aware SpeechLLM Adaptation for Multilingual Speech-to-Speech Translation](https://md.research.elamur.ai/free/en/papers/pan-2026-s2st-omni-hierarchical-language-aware/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/42/)

## References

46 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, and 1 others. 2023. [Openflamingo: An open-source framework for training large autoregressive vision-language models](https://md.research.elamur.ai/free/en/papers/awadalla-2023-openflamingo-open-source-framework-training/). arXiv preprint arXiv:2308.01390.
3. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. [Qwen-VL: A frontier large vision-language model with versatile abilities](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/). arXiv preprint arXiv:2308.12966.
4. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, and 1 others. 2025a. [Qwen3-vl technical report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen3-vl-technical-report/). arXiv preprint arXiv:2511.21631.
5. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025b. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. — [Qwen2.5-VL Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen2-5-vl-technical/)
6. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+and+1+others%2E+2022%2E+Training+a+helpful+and+harmless+assistant+wi)
7. Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Tasırlar. 2023. Introducing our multimodal models. Adept Blog. [find](https://scholar.google.com/scholar?q=Rohan+Bavishi%2C+Erich+Elsen%2C+Curtis+Hawthorne%2C+Maxwell+Nye%2C+Augustus+Odena%2C+Arushi+Somani%2C+and+Sagnak+Tas%C4%B1rlar%2E+2023%2E+Introducing+our+multimodal+models%2E+Adept+Blog%2E)
8. Jan Betley, Daniel Chee Hian Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, and Owain Evans. 2025. Emergent misalignment: Narrow finetuning can produce broadly misaligned LLMs. In Proceedings of the 42nd International Conference on Machine Learning, pages 4043–4068. [find](https://scholar.google.com/scholar?q=Jan+Betley%2C+Daniel+Chee+Hian+Tan%2C+Niels+Warncke%2C+Anna+Sztyber%2DBetley%2C+Xuchan+Bao%2C+Mart%C3%ADn+Soto%2C+Nathan+Labenz%2C+and+Owain+Evans%2E+2025%2E+Emergent+misalignment%3A+Narrow+finetuning+can+produce+broadly+misali)
9. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. 2024a. [Are we on the right way for evaluating large vision-language models?](https://md.research.elamur.ai/free/en/papers/chen-2024-right-way-evaluating-large-vision/) In Proceedings of the 38th International Conference on Neural Information Processing Systems…
10. Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. 2024b. ShareGPT4Video: improving video understanding and generation with better captions. In Proceedings of the 38th Internati… [find](https://scholar.google.com/scholar?q=Lin+Chen%2C+Xilin+Wei%2C+Jinsong+Li%2C+Xiaoyi+Dong%2C+Pan+Zhang%2C+Yuhang+Zang%2C+Zehui+Chen%2C+Haodong+Duan%2C+Bin+Lin%2C+Zhenyu+Tang%2C+Li+Yuan%2C+Yu+Qiao%2C+Dahua+Lin%2C+Feng+Zhao%2C+and+Jiaqi+Wang%2E+2024b%2E+ShareGPT4Video%3A+imp)
11. Zhipeng Chen, Kun Zhou, Zheng Zhang, Beichen Gong, Xin Zhao, and Ji-Rong Wen. 2023. ChatCoT: Tool-augmented chain-of-thought reasoning on chat-based large language models. In Proceedings of the Findings of the Association for Computational Linguistics: EMNLP. [find](https://scholar.google.com/scholar?q=Zhipeng+Chen%2C+Kun+Zhou%2C+Zheng+Zhang%2C+Beichen+Gong%2C+Xin+Zhao%2C+and+Ji%2DRong+Wen%2E+2023%2E+ChatCoT%3A+Tool%2Daugmented+chain%2Dof%2Dthought+reasoning+on+chat%2Dbased+large+language+models%2E+In+Proceedings+of+the+Findin)
12. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. [Instructblip: towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/). In Proceedings of the 37th International Conference on Neural Information Proce…
13. Anurag Das, Adrian Bulat, Alberto Baldrati, Ioannis Maniadis Metaxas, Bernt Schiele, Georgios Tzimiropoulos, and Brais Martinez. 2026. More Images, More Problems? A Controlled Analysis of VLM Failure Modes. arXiv preprint arXiv:2601.07812. [find](https://scholar.google.com/scholar?q=Anurag+Das%2C+Adrian+Bulat%2C+Alberto+Baldrati%2C+Ioannis+Maniadis+Metaxas%2C+Bernt+Schiele%2C+Georgios+Tzimiropoulos%2C+and+Brais+Martinez%2E+2026%2E+More+Images%2C+More+Problems%3F+A+Controlled+Analysis+of+VLM+Failure+)
14. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255. [find](https://scholar.google.com/scholar?q=Jia+Deng%2C+Wei+Dong%2C+Richard+Socher%2C+Li%2DJia+Li%2C+Kai+Li%2C+and+Li+Fei%2DFei%2E+2009%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+Proceedings+of+Conference+on+Computer+Vision+and+Pattern+Recognitio)
15. Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024. Blink: Multimodal large language models can see but not perceive. In Proceedings of European Conference on Computer Vision (ECCV). [find](https://scholar.google.com/scholar?q=Xingyu+Fu%2C+Yushi+Hu%2C+Bangzheng+Li%2C+Yu+Feng%2C+Haoyu+Wang%2C+Xudong+Lin%2C+Dan+Roth%2C+Noah+A%2E+Smith%2C+Wei%2DChiu+Ma%2C+and+Ranjay+Krishna%2E+2024%2E+Blink%3A+Multimodal+large+language+models+can+see+but+not+perceive%2E+In)
16. Shaz Furniturewala, Surgan Jandial, Abhinav Java, Pragyan Banerjee, Simra Shahid, Sumit Bhatia, and Kokil Jaidka. 2024. “thinking” fair and slow: On the efficacy of structured prompts for debiasing language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Shaz+Furniturewala%2C+Surgan+Jandial%2C+Abhinav+Java%2C+Pragyan+Banerjee%2C+Simra+Shahid%2C+Sumit+Bhatia%2C+and+Kokil+Jaidka%2E+2024%2E+%E2%80%9Cthinking%E2%80%9D+fair+and+slow%3A+On+the+efficacy+of+structured+prompts+for+debiasing+la)
17. Bryan Guan, Tanya Roosta, Peyman Passban, and Mehdi Rezagholizadeh. 2025. The order effect: Investigating prompt sensitivity to input order in llms. arXiv preprint arXiv:2502.04134. — [arXiv:2502.04134](https://arxiv.org/abs/2502.04134) [find](https://scholar.google.com/scholar?q=Bryan+Guan%2C+Tanya+Roosta%2C+Peyman+Passban%2C+and+Mehdi+Rezagholizadeh%2E+2025%2E+The+order+effect%3A+Investigating+prompt+sensitivity+to+input+order+in+llms%2E+arXiv+preprint+arXiv%3A2502%2E04134%2E+%E2%80%94+%5BarXiv%3A2502%2E0413)
18. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. [GPT-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). arXiv preprint arXiv:2410.21276.
19. Surgan Jandial, Yinheng Li, Justin Wagle, and Kazuhito Koishida. 2026. Do GUI grounders truly understand UI elements? In Proceedings of Findings of the Association for Computational Linguistics: EACL. [find](https://scholar.google.com/scholar?q=Surgan+Jandial%2C+Yinheng+Li%2C+Justin+Wagle%2C+and+Kazuhito+Koishida%2E+2026%2E+Do+GUI+grounders+truly+understand+UI+elements%3F+In+Proceedings+of+Findings+of+the+Association+for+Computational+Linguistics%3A+EACL%2E)
20. Surgan Jandial, Yinong Oliver Wang, Andrea Bajcsy, and Fernando De la Torre. 2025. On the fine-grained planning abilities of VLM web agents. In Proceedings of Findings of the Association for Computational Linguistics: EMNLP. [find](https://scholar.google.com/scholar?q=Surgan+Jandial%2C+Yinong+Oliver+Wang%2C+Andrea+Bajcsy%2C+and+Fernando+De+la+Torre%2E+2025%2E+On+the+fine%2Dgrained+planning+abilities+of+VLM+web+agents%2E+In+Proceedings+of+Findings+of+the+Association+for+Computati)
21. Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max W.F. Ku, Qian Liu, and Wenhu Chen. 2024. Mantis: Interleaved multi-image instruction tuning. Transactions on Machine Learning Research. [find](https://scholar.google.com/scholar?q=Dongfu+Jiang%2C+Xuan+He%2C+Huaye+Zeng%2C+Cong+Wei%2C+Max+W%2EF%2E+Ku%2C+Qian+Liu%2C+and+Wenhu+Chen%2E+2024%2E+Mantis%3A+Interleaved+multi%2Dimage+instruction+tuning%2E+Transactions+on+Machine+Learning+Research%2E)
22. Hugo Laurençon, Leo Tronchon, Matthieu Cord, and Victor Sanh. 2024. [What matters when building vision-language models?](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2024-matters-building-vision-language-models/) In Proceedings of The Thirty-eighth Annual Conference on Neural Information Processing Systems.
23. Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun MA, and Chunyuan Li. 2025. LLaVA-neXT-interleave: Tackling multi-image, video, and 3D in large multimodal models. In Proceedings of The Thirteenth International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Feng+Li%2C+Renrui+Zhang%2C+Hao+Zhang%2C+Yuanhan+Zhang%2C+Bo+Li%2C+Wei+Li%2C+Zejun+MA%2C+and+Chunyuan+Li%2E+2025%2E+LLaVA%2DneXT%2Dinterleave%3A+Tackling+multi%2Dimage%2C+video%2C+and+3D+in+large+multimodal+models%2E+In+Proceedings+o)
24. Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, Lingpeng Kong, and Qi Liu. 2024. VLFeedback: A large-scale AI feedback dataset for large vision-language models alignment. In Proceedings of the Conference on Empirical Methods in Natural Language Processi… [find](https://scholar.google.com/scholar?q=Lei+Li%2C+Zhihui+Xie%2C+Mukai+Li%2C+Shunian+Chen%2C+Peiyi+Wang%2C+Liang+Chen%2C+Yazheng+Yang%2C+Benyou+Wang%2C+Lingpeng+Kong%2C+and+Qi+Liu%2E+2024%2E+VLFeedback%3A+A+large%2Dscale+AI+feedback+dataset+for+large+vision%2Dlanguage+)
25. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. [Evaluating object hallucination in large vision-language models](https://md.research.elamur.ai/free/en/papers/li-2023-evaluating-object-hallucination-large-vision/). In Proceedings of The Conference on Empirical Methods in Natural Language Processing (EMNLP).
26. Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2024. [Video-LLaVA: Learning united visual representation by alignment before projection](https://md.research.elamur.ai/free/en/papers/lin-2023-video-llava-learning-united-visual/). In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
27. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024a. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/). In Proceedings of Computer Vision and Pattern Recognition (CVPR).
28. Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi Dong, Pan Zhang, Zijian Liang, Yuanjun Xiong, Yu Qiao, Dahua Lin, and 1 others. 2024b. MMDU: A multi-turn multi-image dialog understanding benchmark and instruction-tuning dataset for LVLMs. arXiv preprint arXiv:2406.11833. — [arXiv:2406.11833](https://arxiv.org/abs/2406.11833) [find](https://scholar.google.com/scholar?q=Ziyu+Liu%2C+Tao+Chu%2C+Yuhang+Zang%2C+Xilin+Wei%2C+Xiaoyi+Dong%2C+Pan+Zhang%2C+Zijian+Liang%2C+Yuanjun+Xiong%2C+Yu+Qiao%2C+Dahua+Lin%2C+and+1+others%2E+2024b%2E+MMDU%3A+A+multi%2Dturn+multi%2Dimage+dialog+understanding+benchmark+a)
29. Ziyu Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Conghui He, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. 2025. MIA-DPO: Multi-image augmented direct preference optimization for large vision-language models. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ziyu+Liu%2C+Yuhang+Zang%2C+Xiaoyi+Dong%2C+Pan+Zhang%2C+Yuhang+Cao%2C+Haodong+Duan%2C+Conghui+He%2C+Yuanjun+Xiong%2C+Dahua+Lin%2C+and+Jiaqi+Wang%2E+2025%2E+MIA%2DDPO%3A+Multi%2Dimage+augmented+direct+preference+optimization+for+l)
30. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In Proceedings of the International Conference on Neural Information Processing Systems.
31. Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. 2025. Vision language models are blind: Failing to translate detailed visual features into words. arXiv preprint arXiv:2407.06581. [find](https://scholar.google.com/scholar?q=Pooyan+Rahmanzadehgervi%2C+Logan+Bolton%2C+Mohammad+Reza+Taesiri%2C+and+Anh+Totti+Nguyen%2E+2025%2E+Vision+language+models+are+blind%3A+Failing+to+translate+detailed+visual+features+into+words%2E+arXiv+preprint+arX)
32. Joseph P. Robinson, Ming Shao, Yue Wu, and Yun Fu. 2016. Families in the wild (fiw): Large-scale kinship image database and benchmarks. In Proceedings of the ACM on Multimedia Conference, pages 242–246. [find](https://scholar.google.com/scholar?q=Joseph+P%2E+Robinson%2C+Ming+Shao%2C+Yue+Wu%2C+and+Yun+Fu%2E+2016%2E+Families+in+the+wild+%28fiw%29%3A+Large%2Dscale+kinship+image+database+and+benchmarks%2E+In+Proceedings+of+the+ACM+on+Multimedia+Conference%2C+pages+242%E2%80%9324)
33. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+2017%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2E)
34. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300.
35. Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019. [A corpus for reasoning about natural language grounded in photographs](https://md.research.elamur.ai/free/en/papers/suhr-2019-corpus-reasoning-natural-language-grounded/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
36. Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. 2024. Generative multimodal models are in-context learners. In Proceedings of Computer Vision and Pattern Recognition (CVPR). [find](https://scholar.google.com/scholar?q=Quan+Sun%2C+Yufeng+Cui%2C+Xiaosong+Zhang%2C+Fan+Zhang%2C+Qiying+Yu%2C+Yueze+Wang%2C+Yongming+Rao%2C+Jingjing+Liu%2C+Tiejun+Huang%2C+and+Xinlong+Wang%2E+2024%2E+Generative+multimodal+models+are+in%2Dcontext+learners%2E+In+Proce)
37. Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, and 1 others. 2023. [Aligning large multimodal models with factually augmented RLHF](https://md.research.elamur.ai/free/en/papers/sun-2023-aligning-large-multimodal-models-factually/). arXiv preprint arXiv:2309.14525.
38. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. [Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting](https://md.research.elamur.ai/free/en/papers/turpin-2023-language-models-don-t-always/). In Thirty-seventh Conference on Neural Information Processing Systems.
39. Bertie Vidgen, Adarsh Agrawal, Ahmed M. Ahmed, Victor Akinwande, Namir Al-Nuaimi, Najla Alfaraj, Elie Alhajjar, Lora Aroyo, Trupti Bavalatti, Max Bartolo, Borhane Blili-Hamelin, Kurt Bollacker, Rishi Bomassani, Marisa Ferrara Boston, Siméon Campos, Kal Chakra, Canyu Chen, Cody Coleman, Zacharie Delp… — [Introducing v0.5 of the AI Safety Benchmark from MLCommons](https://md.research.elamur.ai/free/en/papers/vidgen-2024-introducing-v0-5-ai-safety/)
40. Shuyang Wang, Joseph P Robinson, and Yun Fu. 2017. Kinship verification on families in the wild with marginalized denoising metric learning. In Proceedings of Automatic Face and Gesture Recognition (FG). [find](https://scholar.google.com/scholar?q=Shuyang+Wang%2C+Joseph+P+Robinson%2C+and+Yun+Fu%2E+2017%2E+Kinship+verification+on+families+in+the+wild+with+marginalized+denoising+metric+learning%2E+In+Proceedings+of+Automatic+Face+and+Gesture+Recognition+%28F)
41. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, Jiazheng Xu, Keqin Chen, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. 2024. CogVLM: Visual expert for pre-trained language models. In Proceedings of The Thirty-eighth Annual… [find](https://scholar.google.com/scholar?q=Weihan+Wang%2C+Qingsong+Lv%2C+Wenmeng+Yu%2C+Wenyi+Hong%2C+Ji+Qi%2C+Yan+Wang%2C+Junhui+Ji%2C+Zhuoyi+Yang%2C+Lei+Zhao%2C+Song+XiXuan%2C+Jiazheng+Xu%2C+Keqin+Chen%2C+Bin+Xu%2C+Juanzi+Li%2C+Yuxiao+Dong%2C+Ming+Ding%2C+and+Jie+Tang%2E+2024)
42. Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, and Tat-Seng Chua. 2024a. RLHF-V: Towards trustworthy MLLMs via behavior alignment from fine-grained correctional human feedback. In Proceedings of Computer Vision and Pattern Recog… [find](https://scholar.google.com/scholar?q=Tianyu+Yu%2C+Yuan+Yao%2C+Haoye+Zhang%2C+Taiwen+He%2C+Yifeng+Han%2C+Ganqu+Cui%2C+Jinyi+Hu%2C+Zhiyuan+Liu%2C+Hai%2DTao+Zheng%2C+Maosong+Sun%2C+and+Tat%2DSeng+Chua%2E+2024a%2E+RLHF%2DV%3A+Towards+trustworthy+MLLMs+via+behavior+alignmen)
43. Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, and 1 others. 2024b. RlHF-V: Towards trustworthy MLLMs via behavior alignment from fine-grained correctional human feedback. arXiv preprint arXiv:2405.17220. [find](https://scholar.google.com/scholar?q=Tianyu+Yu%2C+Haoye+Zhang%2C+Yuan+Yao%2C+Yunkai+Dang%2C+Da+Chen%2C+Xiaoman+Lu%2C+Ganqu+Cui%2C+Taiwen+He%2C+Zhiyuan+Liu%2C+Tat%2DSeng+Chua%2C+and+1+others%2E+2024b%2E+RlHF%2DV%3A+Towards+trustworthy+MLLMs+via+behavior+alignment+from)
44. Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, and 1 others. 2024. [Internlm-Xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output](https://md.research.elamur.ai/free/en/papers/zhang-2024-internlm-xcomposer-2-5/). arXiv preprint arXiv:2407.03320.
45. Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. 2023. Beyond hallucinations: Enhancing LVLMs through hallucination-aware direct preference optimization. arXiv preprint arXiv:2311.16839. — [arXiv:2311.16839](https://arxiv.org/abs/2311.16839) [find](https://scholar.google.com/scholar?q=Zhiyuan+Zhao%2C+Bin+Wang%2C+Linke+Ouyang%2C+Xiaoyi+Dong%2C+Jiaqi+Wang%2C+and+Conghui+He%2E+2023%2E+Beyond+hallucinations%3A+Enhancing+LVLMs+through+hallucination%2Daware+direct+preference+optimization%2E+arXiv+preprint+a)
46. Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. 2024. Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.11411. — [arXiv:2402.11411](https://arxiv.org/abs/2402.11411) [find](https://scholar.google.com/scholar?q=Yiyang+Zhou%2C+Chenhang+Cui%2C+Rafael+Rafailov%2C+Chelsea+Finn%2C+and+Huaxiu+Yao%2E+2024%2E+Aligning+modalities+in+vision+large+language+models+via+preference+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A2402%2E11411%2E+%E2%80%94+%5BarXi)

*Updated 1 November 2026*