---
title: "RLHF-V — paper by Yu et al. (2023) — summary and ideas"
description: "Multimodal LLMs often describe things that are not in the image, such as invented objects, wrong… 63 references, cited by 104 papers here."
authors: ["Tianyu Yu", "Yuan Yao", "Haoye Zhang", "Taiwen He", "Yifeng Han", "Ganqu Cui", "Jinyi Hu", "Zhiyuan Liu", "Hai-Tao Zheng", "Maosong Sun", "Tat-Seng Chua"]
year: 2023
venue: "arXiv:2312.00849"
source: "https://arxiv.org/abs/2312.00849"
doi: "10.48550/arxiv.2312.00849"
arxiv: "2312.00849"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yu-2023-rlhf-v-towards-trustworthy-mllms/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yu-2023-rlhf-v-towards-trustworthy-mllms/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) / RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback

# RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback

**Authors:** [Tianyu Yu](https://md.research.elamur.ai/free/en/authors/tianyu-yu/), [Yuan Yao](https://md.research.elamur.ai/free/en/authors/yuan-yao/), [Haoye Zhang](https://md.research.elamur.ai/free/en/authors/haoye-zhang/), Taiwen He, Yifeng Han, [Ganqu Cui](https://md.research.elamur.ai/free/en/authors/ganqu-cui/), [Jinyi Hu](https://md.research.elamur.ai/free/en/authors/jinyi-hu/), [Zhiyuan Liu](https://md.research.elamur.ai/free/en/authors/zhiyuan-liu/), [Hai-Tao Zheng](https://md.research.elamur.ai/free/en/authors/hai-tao-zheng/), [Maosong Sun](https://md.research.elamur.ai/free/en/authors/maosong-sun/) and [Tat-Seng Chua](https://md.research.elamur.ai/free/en/authors/tat-seng-chua/) · **Year:** 2023 · **Venue:** [arXiv:2312.00849](https://md.research.elamur.ai/free/en/venues/ieee-computer-society-conference-on-computer-vision-and-pattern-recognition/) ([IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2023](https://md.research.elamur.ai/free/en/venues/ieee-computer-society-conference-on-computer-vision-and-pattern-recognition/2023/)) · **Publisher:** [Institute of Electrical and Electronics Engineers](https://md.research.elamur.ai/free/en/publishers/institute-of-electrical-and-electronics-engineers/) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2312.00849) · [PDF](https://arxiv.org/pdf/2312.00849) · [DOI 10.48550/arxiv.2312.00849](https://doi.org/10.48550/arxiv.2312.00849) · [arXiv 2312.00849](https://arxiv.org/abs/2312.00849) · [OpenAlex](https://openalex.org/W4402727405)

**Citations:** 113 citations (OpenAlex, 2026-10-14) · [104](https://md.research.elamur.ai/free/en/papers/yu-2023-rlhf-v-towards-trustworthy-mllms/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Multimodal LLMs often describe things that are not in the image, such as invented objects, wrong counts, or misplaced items. This makes them unreliable for sensitive uses. The authors argue that standard RLHF ranking labels are a poor fix. Annotators struggle to choose between long answers that each have different errors, a ranking does not say what the correct answer should be, and coarse signals waste data and invite reward hacking. RLHF-V takes a different approach: annotators edit only the wrong fragments of a model's answer. Each edit produces a pair of answers, the original and the corrected version, that differ only where the error was. Training uses DDPO, a variant of DPO that gives extra weight to the corrected segments. The authors also address a vision-language mismatch that causes hallucinations during training. With roughly 1.4k annotated examples, RLHF-V cuts hallucinations of the base model by about a third. It beats LLaVA-RLHF, which used around seven times more data, showing that targeted feedback can be far more data-efficient than rankings.

## Key points
- **Dataset:** 1.4k prompts, taken from an existing instruction-tuning dataset plus GPT-4-generated image description prompts. The responses come from Muffin, the base MLLM. Corrected answers average 64.4 words with 2.65 edited segments each.
- **Error types corrected:** objects 41.2%, positions 20.3%, numbers 16.5%, attributes 10.0%, actions 5.3%, other 6.8%.
- **Main result:** the base model's hallucination rate drops by 34.8% across five benchmarks, using automatic and human evaluation. RLHF-V outperforms LLaVA-RLHF, which was trained on 10k preference samples.
- **Comparison with GPT-4V:** the authors' human evaluation found obvious hallucinations in 45.9% of GPT-4V responses. RLHF-V is reported to resist over-generalization hallucinations better than GPT-4V and achieves state-of-the-art trustworthiness among open-source MLLMs.
- **Availability:** code, data and model weights are released at github.com/RLHF-V/RLHF-V.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations
- [National University of Singapore](https://md.research.elamur.ai/free/en/universities/national-university-of-singapore/) — University, Singapore, Singapore · 207 works · 39,576 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun and Tat-Seng Chua. 2023. RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback. arXiv:2312.00849.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{yu2023rlhf,
  title = {{RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback}},
  author = {Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun and Tat-Seng Chua},
  year = {2023},
  doi = {10.48550/arxiv.2312.00849},
  eprint = {2312.00849},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.00849}
}
```

- [← previous Making Language Models Better Reasoners with Step-Aware Verifier](https://md.research.elamur.ai/free/en/papers/li-2023-making-language-models-better-reasoners/)
- [next → Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://md.research.elamur.ai/free/en/papers/dai-2023-safe-rlhf-safe-reinforcement-learning/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/)

## References

63 references, 42 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. GPT-4 vision document. [https://platform.openai](https://platform.openai) . com / docs / guides / vision / calculating-costs. Accessed: 2023-11-20. 12 [↗](https://platform.openai)
2. John schulman - reinforcement learning from human feedback: Progress and challenges. [https://www.youtube.com/watch?v=hhiLw5Q_UFg](https://www.youtube.com/watch?v=hhiLw5Q_UFg). Accessed: 2023-11-20. 12 [↗](https://www.youtube.com/watch?v=hhiLw5Q_UFg)
3. Large multi-modal models for strong performance and efficient deployment. [https://github.com/OpenBMB/OmniLMM](https://github.com/OpenBMB/OmniLMM). Accessed: 2024-03-05. 7 [↗](https://github.com/OpenBMB/OmniLMM)
4. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. [Flamingo: a visual language model for few-shot learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/). NeurIPS, 35: 23716–23736, 2022. 1, 7
5. Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. [OpenFlamingo: An open-source framework for training large autoregressive vision-language models](https://md.research.elamur.ai/free/en/papers/awadalla-2023-openflamingo-open-source-framework-training/). arXiv preprint arXiv:2308.01390, 2023. 7
6. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.[12966, 2023](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/). 1, 5, 6, 7, 13
7. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. 2, 3 [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+Training+a+helpful+and+harmless+assistant+with+reinforcem)
8. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. [Constitutional AI: Harmlessness from AI feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073, 2022. 8
9. Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. COYO-700M: Image-text pair dataset, 2022. 4 [find](https://scholar.google.com/scholar?q=Minwoo+Byeon%2C+Beomhee+Park%2C+Haecheon+Kim%2C+Sungjun+Lee%2C+Woonhyuk+Baek%2C+and+Saehoon+Kim%2E+COYO%2D700M%3A+Image%2Dtext+pair+dataset%2C+2022%2E+4)
10. Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, pages 3558–3568, 2021. 4 [find](https://scholar.google.com/scholar?q=Soravit+Changpinyo%2C+Piyush+Sharma%2C+Nan+Ding%2C+and+Radu+Soricut%2E+Conceptual+12M%3A+Pushing+web%2Dscale+image%2Dtext+pre%2Dtraining+to+recognize+long%2Dtail+visual+concepts%2E+In+CVPR%2C+pages+3558%E2%80%933568%2C+2021%2E+4)
11. Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. [PaLI: A jointly-scaled multilingual language-image model](https://md.research.elamur.ai/free/en/papers/chen-2022-pali-jointly-scaled-multilingual-language/). arXiv preprint arXiv:2209.06794, 2022. 7
12. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality, 2023. 5 [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+and+Eric+P%2E+Xing%2E+Vicuna%3A+An+open%2Dsource+chatbot+i)
13. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023. 2, 3, 8 [find](https://scholar.google.com/scholar?q=Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Guanming+Yao%2C+Wei+Zhu%2C+Yuan+Ni%2C+Guotong+Xie%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+Ultrafeedback%3A+Boosting+language+models+with+high%2Dquality+feedback%2E+arXiv+preprint+arXiv%3A23)
14. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. [InstructBLIP: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/), 2023. 1, 4, 5, 7, 8, 13
15. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. [Enhancing chat language models by scaling high-quality instructional conversations](https://md.research.elamur.ai/free/en/papers/ding-2023-enhancing-chat-language-models-scaling/). arXiv preprint arXiv:2305.14233, 2023. 8
16. Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. PaLM-E: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023. 1 [find](https://scholar.google.com/scholar?q=Danny+Driess%2C+Fei+Xia%2C+Mehdi+SM+Sajjadi%2C+Corey+Lynch%2C+Aakanksha+Chowdhery%2C+Brian+Ichter%2C+Ayzaan+Wahid%2C+Jonathan+Tompson%2C+Quan+Vuong%2C+Tianhe+Yu%2C+et+al%2E+PaLM%2DE%3A+An+embodied+multimodal+language+model%2E+ar)
17. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. [MME: A comprehensive evaluation benchmark for multimodal large language models](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/). arXiv preprint arXiv:2306.13394, 2023. 4, 7
18. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. [Making the v in vqa matter: Elevating the role of image understanding in visual question answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). In CVPR, pages 6904–6913, 2017. 4, 5, 13, 15
19. Anisha Gunjal, Jihan Yin, and Erhan Bas. [Detecting and preventing hallucinations in large vision language models](https://md.research.elamur.ai/free/en/papers/gunjal-2023-detecting-preventing-hallucinations-large-vision/). arXiv preprint arXiv:2308.06394, 2023. 8, 13
20. Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. [Language is not all you need: Aligning perception with language models](https://md.research.elamur.ai/free/en/papers/huang-2023-language-all-need-aligning-perception/). arXiv preprint arXiv:2302.14045, 2023. 7
21. Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021. If you use this software, please cite it as below. 13 [find](https://scholar.google.com/scholar?q=Gabriel+Ilharco%2C+Mitchell+Wortsman%2C+Ross+Wightman%2C+Cade+Gordon%2C+Nicholas+Carlini%2C+Rohan+Taori%2C+Achal+Dave%2C+Vaishaal+Shankar%2C+Hongseok+Namkoong%2C+John+Miller%2C+Hannaneh+Hajishirzi%2C+Ali+Farhadi%2C+and+Ludwi)
22. Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. [Alignment of language agents](https://md.research.elamur.ai/free/en/papers/kenton-2021-alignment-language-agents/). arXiv preprint arXiv:2103.14659, 2021. 8
23. Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International… [find](https://scholar.google.com/scholar?q=Alina+Kuznetsova%2C+Hassan+Rom%2C+Neil+Alldrin%2C+Jasper+Uijlings%2C+Ivan+Krasin%2C+Jordi+Pont%2DTuset%2C+Shahab+Kamali%2C+Stefan+Popov%2C+Matteo+Malloci%2C+Alexander+Kolesnikov%2C+et+al%2E+The+open+images+dataset+v4%3A+Unifie)
24. Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. [Scalable agent alignment via reward modeling: a research direction](https://md.research.elamur.ai/free/en/papers/leike-2018-scalable-agent-alignment-reward-modeling/). arXiv preprint arXiv:1811.07871, 2018. 8
25. Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. [Seed-bench: Benchmarking multimodal llms with generative comprehension](https://md.research.elamur.ai/free/en/papers/li-2023-seed-bench-benchmarking-multimodal-llms/). arXiv preprint arXiv:2307.16125, 2023. 7
26. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. [Otter: A multi-modal model with in-context instruction tuning](https://md.research.elamur.ai/free/en/papers/li-2023-otter-multi-modal-model-context/). arXiv preprint arXiv:2305.03726, 2023. 7
27. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. [BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). arXiv preprint arXiv:2301.12597, 2023. 1, 4, 7
28. Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al. M3IT: A large-scale dataset towards multimodal multilingual instruction tuning. arXiv preprint arXiv:2306.04387, 2023. 8 — [arXiv:2306.04387](https://arxiv.org/abs/2306.04387) [find](https://scholar.google.com/scholar?q=Lei+Li%2C+Yuwei+Yin%2C+Shicheng+Li%2C+Liang+Chen%2C+Peiyi+Wang%2C+Shuhuai+Ren%2C+Mukai+Li%2C+Yazheng+Yang%2C+Jingjing+Xu%2C+Xu+Sun%2C+et+al%2E+M3IT%3A+A+large%2Dscale+dataset+towards+multimodal+multilingual+instruction+tuning%2E)
29. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. [Evaluating object hallucination in large vision-language models](https://md.research.elamur.ai/free/en/papers/li-2023-evaluating-object-hallucination-large-vision/). arXiv preprint arXiv:2305.10355, 2023. 1, 4, 8, 13
30. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.[20050, 2023](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). 8
31. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. [Microsoft COCO: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In ECCV, pages 740–755. Springer, 2014. 5, 13
32. Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusion-Bench: You see what you think? Or you think what you see? An image-context reasoning benchmark challenging for GPT-4V(ision), LLaVA-1.5, and other multi-modality models. arXiv preprint arXiv:… — [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/guan-2023-hallusionbench-advanced-diagnostic-suite-entangled/)
33. Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.[14565, 2023](https://md.research.elamur.ai/free/en/papers/liu-2023-mitigating-hallucination-large-multi-modal/). 1, 4, 5, 8, 13
34. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/). arXiv preprint arXiv:2310.03744, 2023. 5, 6, 8, 13
35. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). arXiv preprint arXiv:2304.08485, 2023. 1, 4, 5, 7, 8, 13, 15
36. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. [MathVista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). arXiv preprint arXiv:2310.02255, 2023. 1
37. OpenAI. GPT-4V(ision) system card. 2023. 1, 2, 5, 6, 8, 12 [find](https://scholar.google.com/scholar?q=OpenAI%2E+GPT%2D4V%28ision%29+system+card%2E+2023%2E+1%2C+2%2C+5%2C+6%2C+8%2C+12)
38. OpenAI. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023. 1, 3, 8
39. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). NeurIPS, 35:27730–27744, 2022. 2, 3, 8, 9
40. Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of ICCV, 2015. 13 [find](https://scholar.google.com/scholar?q=Bryan+A+Plummer%2C+Liwei+Wang%2C+Chris+M+Cervantes%2C+Juan+C+Caicedo%2C+Julia+Hockenmaier%2C+and+Svetlana+Lazebnik%2E+Flickr30k+entities%3A+Collecting+region%2Dto%2Dphrase+correspondences+for+richer+image%2Dto%2Dsentence+m)
41. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In ICML, pages 8748–8763. PMLR, 2021. 4
42. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). arXiv preprint arXiv:2305.18290, 2023. 2, 3, 8
43. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. [Object hallucination in image captioning](https://md.research.elamur.ai/free/en/papers/rohrbach-2018-object-hallucination-image-captioning/). In EMNLP, pages 4035–4045, 2018. 4, 13
44. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. [LAION-5B: An open large-scale dataset for training next generation image-text models](https://md.research.elamur.ai/free/en/papers/schuhmann-2022-laion-5b-open-large-scale/). NeurIPS, 35:25278– 25294, 2022. 4
45. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 5, 8, 13 [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E+5%2C+8%2C+13)
46. Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. [A-okvqa: A benchmark for visual question answering using world knowledge](https://md.research.elamur.ai/free/en/papers/schwenk-2022-okvqa-benchmark-visual-question-answering/). In Proceedings of ECCV, 2022. 13
47. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. NeurIPS, 33:3008–3021, 2020. 8 [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F+Christiano%2E+Learning+to+summarize+with+human+feedback%2E+NeurIPS%2C+33%3A3008%E2%80%933021%2C+2)
48. Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. [Aligning large multimodal models with factually augmented RLHF](https://md.research.elamur.ai/free/en/papers/sun-2023-aligning-large-multimodal-models-factually/). arXiv preprint arXiv:2309.14525, 2023. 1, 2, 4, 5, 6, 8, 9, 13
49. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford Alpaca: An instruction-following LLaMA model, 2023. 8 [find](https://scholar.google.com/scholar?q=Rohan+Taori%2C+Ishaan+Gulrajani%2C+Tianyi+Zhang%2C+Yann+Dubois%2C+Xuechen+Li%2C+Carlos+Guestrin%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+Stanford+Alpaca%3A+An+instruction%2Dfollowing+LLaMA+model%2C+2023%2E+8)
50. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [LLaMA: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023. 5
51. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [LLaMA 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023. 1, 2, 3, 5
52. Bin Wang, Fan Wu, Xiao Han, Jiahui Peng, Huaping Zhong, Pan Zhang, Xiaoyi Dong, Weijia Li, Wei Li, Jiaqi Wang, et al. VIGC: Visual instruction generation and correction. arXiv preprint arXiv:2308.12714, 2023. 8 — [arXiv:2308.12714](https://arxiv.org/abs/2308.12714) [find](https://scholar.google.com/scholar?q=Bin+Wang%2C+Fan+Wu%2C+Xiao+Han%2C+Jiahui+Peng%2C+Huaping+Zhong%2C+Pan+Zhang%2C+Xiaoyi+Dong%2C+Weijia+Li%2C+Wei+Li%2C+Jiaqi+Wang%2C+et+al%2E+VIGC%3A+Visual+instruction+generation+and+correction%2E+arXiv+preprint+arXiv%3A2308%2E1271)
53. Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: BEiT pretraining for vision and vision-language tasks. In CVPR, pages 19175–19186, 2023. 4, 5 [find](https://scholar.google.com/scholar?q=Wenhui+Wang%2C+Hangbo+Bao%2C+Li+Dong%2C+Johan+Bjorck%2C+Zhiliang+Peng%2C+Qiang+Liu%2C+Kriti+Aggarwal%2C+Owais+Khan+Mohammed%2C+Saksham+Singhal%2C+Subhojit+Som%2C+et+al%2E+Image+as+a+foreign+language%3A+BEiT+pretraining+for+v)
54. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. [CogVLM: Visual expert for pretrained language models](https://md.research.elamur.ai/free/en/papers/wang-2023-cogvlm-visual-expert-pretrained/). arXiv preprint arXiv:2311.03079, 2023. 1, 7
55. Licheng Wen, Xuemeng Yang, Daocheng Fu, Xiaofeng Wang, Pinlong Cai, Xin Li, Tao Ma, Yingxuan Li, Linran Xu, Dengke Shang, et al. On the road with GPT-4V (ision): Early explorations of visual-language model on autonomous driving. arXiv preprint arXiv:2311.05332, 2023. 1 — [arXiv:2311.05332](https://arxiv.org/abs/2311.05332) [find](https://scholar.google.com/scholar?q=Licheng+Wen%2C+Xuemeng+Yang%2C+Daocheng+Fu%2C+Xiaofeng+Wang%2C+Pinlong+Cai%2C+Xin+Li%2C+Tao+Ma%2C+Yingxuan+Li%2C+Linran+Xu%2C+Dengke+Shang%2C+et+al%2E+On+the+road+with+GPT%2D4V+%28ision%29%3A+Early+explorations+of+visual%2Dlanguage+)
56. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. [Fine-grained human feedback gives better rewards for language model training](https://md.research.elamur.ai/free/en/papers/wu-2023-fine-grained-human-feedback-gives/). arXiv preprint arXiv:2306.01693, 2023. 8
57. Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. [The dawn of LMMs: Preliminary explorations with GPT-4V(ision)](https://md.research.elamur.ai/free/en/papers/yang-2023-dawn-lmms-preliminary-explorations-gpt/). arXiv preprint arXiv:2309.17421, 9, 2023. 1
58. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. [mPLUG-Owl: Modularization empowers large language models with multimodality](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/). arXiv preprint arXiv:2304.14178, 2023. 7
59. Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. [Woodpecker: Hallucination correction for multimodal large language models](https://md.research.elamur.ai/free/en/papers/yin-2023-woodpecker-hallucination-correction-multimodal-large/). arXiv preprint arXiv:2310.16045, 2023. 8
60. Tianyu Yu, Jinyi Hu, Yuan Yao, Haoye Zhang, Yue Zhao, Chongyi Wang, Shan Wang, Yinxv Pan, Jiao Xue, Dahai Li, et al. Reformulating vision-language foundation models and datasets towards universal multimodal assistants. arXiv preprint arXiv:2310.00653, 2023. 1, 3, 4, 5, 7, 8, 13 — [arXiv:2310.00653](https://arxiv.org/abs/2310.00653) [find](https://scholar.google.com/scholar?q=Tianyu+Yu%2C+Jinyi+Hu%2C+Yuan+Yao%2C+Haoye+Zhang%2C+Yue+Zhao%2C+Chongyi+Wang%2C+Shan+Wang%2C+Yinxv+Pan%2C+Jiao+Xue%2C+Dahai+Li%2C+et+al%2E+Reformulating+vision%2Dlanguage+foundation+models+and+datasets+towards+universal+mult)
61. Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. [LLaMA-Adapter: Efficient fine-tuning of language models with zero-init attention](https://md.research.elamur.ai/free/en/papers/zhang-2023-llama-adapter-efficient-fine-tuning/). arXiv preprint arXiv:2303.16199, 2023. 7
62. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. [Judging LLM-as-a-judge with MT-Bench and Chatbot Arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/), 2023. 5, 13
63. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. [MiniGPT-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/). arXiv preprint arXiv:2304.10592, 2023. 1

*Updated 2 November 2026*