---
title: "VLM2-Bench — paper by Zhang et al. (2025)"
description: "The authors argue that existing multi-image and video benchmarks miss this skill. 64 references, cited by 11 papers here."
authors: ["Jianshu Zhang", "Dongyu Yao", "Renjie Pi", "Paul Pu Liang", "Yi R. Fung"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.372/"
doi: "10.18653/v1/2025.acl-long.372"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2025-vlm2-bench-closer-look-how/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2025-vlm2-bench-closer-look-how/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/) / VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cues

# VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cues

**Authors:** [Jianshu Zhang](https://md.research.elamur.ai/free/en/authors/jianshu-zhang/), Dongyu Yao, [Renjie Pi](https://md.research.elamur.ai/free/en/authors/renjie-pi/), [Paul Pu Liang](https://md.research.elamur.ai/free/en/authors/paul-pu-liang/) and [Yi R. Fung](https://md.research.elamur.ai/free/en/authors/yi-fung/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.372/) · [PDF](https://aclanthology.org/2025.acl-long.372.pdf) · [DOI 10.18653/v1/2025.acl-long.372](https://doi.org/10.18653/v1/2025.acl-long.372) · [OpenAlex](https://openalex.org/W4412945593)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [11](https://md.research.elamur.ai/free/en/papers/zhang-2025-vlm2-bench-closer-look-how/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether vision-language models can do something people do routinely: recognise that the same object or person appears in several images or video frames by comparing visual traits, without knowing who or what it is. The authors argue that existing multi-image and video benchmarks miss this skill. They tend to rely on world knowledge, broad comparisons or retrieval rather than direct cue matching. To fill the gap they built VLM2-Bench, which has over 3,000 QA pairs spanning 9 subtasks in three cue categories. The data come from repurposed image-editing datasets plus manually collected photos of objects and people, processed through a semi-automated pipeline with human checks. Evaluating twelve VLMs (the count given in the abstract) and testing language-side and vision-side prompting produced eight findings. Some models score near or below chance, and even GPT-4o trails humans by 34.70%. The authors call for stronger core visual skills and less dependence on prior knowledge. They also want clearer rules for when language reasoning helps vision tasks, and training that teaches models to relate visual cues on their own.

## Key points
- **Three cue categories and their subtasks:**
  - General Cue (GC): Matching, Tracking.
  - Object-centric Cue (OC): Comparison, Counting, Grouping.
  - Person-centric Cue (PC): Comparison, Counting, Grouping, plus Video Identity Describing (VID).
- **GC construction:** GC items are built from image-editing pairs (original, edited image, instruction). Easy cases are removed with a salient score, which is the average log-probability that Llama3-8B assigns to the edit instruction given captions of both images. Only samples scoring below −2.0 are kept.
- **Bias control and distractors:**
  - True/false comparisons use consistency-pair validation, so a model gets credit only if it answers both a statement and its negation correctly.
  - Distractor images share visual cues with the target. For PC, they are chosen by CLIP similarity.
- **Question formats:** true/false, multiple-choice, numerical and open-ended. Open-ended items are limited because they are hard to score automatically.
- **Prompting findings:**
  - Chain-of-thought can hurt object grouping, because free-form language describes fine visual details incompletely and inconsistently.
  - The VP-zoom-o visual prompt helps strong models such as GPT-4o but has little effect on weaker ones.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

## Cite

Cite as: Jianshu Zhang, Dongyu Yao, Renjie Pi, Paul Pu Liang and Yi R. Fung. 2025. VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cues. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2025vlm2,
  title = {{VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cues}},
  author = {Jianshu Zhang, Dongyu Yao, Renjie Pi, Paul Pu Liang and Yi R. Fung},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.372},
  url = {https://doi.org/10.18653/v1/2025.acl-long.372}
}
```

- [← previous TripleFact: Defending Data Contamination in the Evaluation of LLM-driven Fake News Detection](https://md.research.elamur.ai/free/en/papers/xu-2025-triplefact-defending-data-contamination-evaluation/)
- [next → What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Length](https://md.research.elamur.ai/free/en/papers/tjuatja-2025-what-goes-into-lm-acceptability/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/)

## References

64 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Vicki Bruce and Andrew W Young. 1986. Understanding face recognition. British journal of psychology, 77 ( Pt 3):305–27. [find](https://scholar.google.com/scholar?q=Vicki+Bruce+and+Andrew+W+Young%2E+1986%2E+Understanding+face+recognition%2E+British+journal+of+psychology%2C+77+%28+Pt+3%29%3A305%E2%80%9327%2E)
2. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2023. [Sharegpt4v: Improving large multimodal models with better captions](https://md.research.elamur.ai/free/en/papers/chen-2023-sharegpt4v-improving-large-multi-modal/). Preprint, arXiv:2311.12793.
3. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. 2024a. [Are we on the right way for evaluating large vision-language models?](https://md.research.elamur.ai/free/en/papers/chen-2024-right-way-evaluating-large-vision/) arXiv preprint arXiv:2403.20330.
4. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024b. [Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks](https://md.research.elamur.ai/free/en/papers/chen-2023-internvl-scaling-up-vision/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern…
5. Dawei Dai, Xu Long, Li Yutang, Zhang Yuanhui, and Shuyin Xia. 2024. Humanvlm: Foundation for human-scene vision-language model. Preprint, arXiv:2411.03034. — [arXiv:2411.03034](https://arxiv.org/abs/2411.03034) [find](https://scholar.google.com/scholar?q=Dawei+Dai%2C+Xu+Long%2C+Li+Yutang%2C+Zhang+Yuanhui%2C+and+Shuyin+Xia%2E+2024%2E+Humanvlm%3A+Foundation+for+human%2Dscene+vision%2Dlanguage+model%2E+Preprint%2C+arXiv%3A2411%2E03034%2E+%E2%80%94+%5BarXiv%3A2411%2E03034%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
6. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
7. Zhiyuan Fan, Yumeng Wang, Sandeep Polisetty, and Yi R. Fung. 2025. Unveiling the lack of lvlm robustness to fundamental visual variations: Why and path forward. Preprint, arXiv:2504.16727. [find](https://scholar.google.com/scholar?q=Zhiyuan+Fan%2C+Yumeng+Wang%2C+Sandeep+Polisetty%2C+and+Yi+R%2E+Fung%2E+2025%2E+Unveiling+the+lack+of+lvlm+robustness+to+fundamental+visual+variations%3A+Why+and+path+forward%2E+Preprint%2C+arXiv%3A2504%2E16727%2E)
8. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. [find](https://scholar.google.com/scholar?q=Joseph+L+Fleiss%2E+1971%2E+Measuring+nominal+scale+agreement+among+many+raters%2E+Psychological+bulletin%2C+76%285%29%3A378%2E)
9. Roopal Garg, Andrea Burns, Burcu Karagol Ayan, Yonatan Bitton, Ceslee Montgomery, Yasumasa Onoe, Andrew Bunner, Ranjay Krishna, Jason Baldridge, and Radu Soricut. 2024. Imageinwords: Unlocking hyper-detailed image descriptions. Preprint, arXiv:2405.02793. [find](https://scholar.google.com/scholar?q=Roopal+Garg%2C+Andrea+Burns%2C+Burcu+Karagol+Ayan%2C+Yonatan+Bitton%2C+Ceslee+Montgomery%2C+Yasumasa+Onoe%2C+Andrew+Bunner%2C+Ranjay+Krishna%2C+Jason+Baldridge%2C+and+Radu+Soricut%2E+2024%2E+Imageinwords%3A+Unlocking+hyper%2Dd)
10. Adam Geitgey. 2016. Machine learning is fun! part 4: Modern face recognition with deep learning. Medium. Medium Corporation, 24:2016. [find](https://scholar.google.com/scholar?q=Adam+Geitgey%2E+2016%2E+Machine+learning+is+fun%21+part+4%3A+Modern+face+recognition+with+deep+learning%2E+Medium%2E+Medium+Corporation%2C+24%3A2016%2E)
11. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. [Making the v in vqa matter: Elevating the role of image understanding in visual question answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913.
12. Jiayi He, Hehai Lin, Qingyun Wang, Yi Fung, and Heng Ji. 2024. Self-correction is more than refinement: A learning framework for visual and language reasoning tasks. Preprint, arXiv:2410.04055. [find](https://scholar.google.com/scholar?q=Jiayi+He%2C+Hehai+Lin%2C+Qingyun+Wang%2C+Yi+Fung%2C+and+Heng+Ji%2E+2024%2E+Self%2Dcorrection+is+more+than+refinement%3A+A+learning+framework+for+visual+and+language+reasoning+tasks%2E+Preprint%2C+arXiv%3A2410%2E04055%2E)
13. Zhitao He, Sandeep Polisetty, Zhiyuan Fan, Yuchen Huang, Shujin Wu, and Yi R. Fung. 2025. Mmbound-ary: Advancing mllm knowledge boundary awareness through reasoning step confidence calibration. Preprint, arXiv:2505.23224. — [arXiv:2505.23224](https://arxiv.org/abs/2505.23224) [find](https://scholar.google.com/scholar?q=Zhitao+He%2C+Sandeep+Polisetty%2C+Zhiyuan+Fan%2C+Yuchen+Huang%2C+Shujin+Wu%2C+and+Yi+R%2E+Fung%2E+2025%2E+Mmbound%2Dary%3A+Advancing+mllm+knowledge+boundary+awareness+through+reasoning+step+confidence+calibration%2E+Prepri)
14. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. [Clipscore: A reference-free evaluation metric for image captioning](https://md.research.elamur.ai/free/en/papers/hessel-2021-clipscore-reference-free-evaluation-metric/). arXiv preprint arXiv:2104.08718.
15. Kung-Hsiang Huang, Hou Pong Chan, May Fung, Haoyi Qiu, Mingyang Zhou, Shafiq Joty, Shih-Fu Chang, and Heng Ji. 2025. From pixels to insights: A survey on automatic chart understanding in the era of large foundation models. IEEE Transactions on Knowledge and Data Engineering, 37(5):2550–2568. [find](https://scholar.google.com/scholar?q=Kung%2DHsiang+Huang%2C+Hou+Pong+Chan%2C+May+Fung%2C+Haoyi+Qiu%2C+Mingyang+Zhou%2C+Shafiq+Joty%2C+Shih%2DFu+Chang%2C+and+Heng+Ji%2E+2025%2E+From+pixels+to+insights%3A+A+survey+on+automatic+chart+understanding+in+the+era+of+la)
16. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). arXiv preprint arXiv:2410.21276.
17. Bo Jiang, Shaoyu Chen, Bencheng Liao, Xingyu Zhang, Wei Yin, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. 2024. Senna: Bridging large vision-language models and end-to-end autonomous driving. Preprint, arXiv:2410.22313. — [arXiv:2410.22313](https://arxiv.org/abs/2410.22313) [find](https://scholar.google.com/scholar?q=Bo+Jiang%2C+Shaoyu+Chen%2C+Bencheng+Liao%2C+Xingyu+Zhang%2C+Wei+Yin%2C+Qian+Zhang%2C+Chang+Huang%2C+Wenyu+Liu%2C+and+Xinggang+Wang%2E+2024%2E+Senna%3A+Bridging+large+vision%2Dlanguage+models+and+end%2Dto%2Dend+autonomous+driving)
18. Qirui Jiao, Daoyuan Chen, Yilun Huang, Yaliang Li, and Ying Shen. 2024. Img-diff: Contrastive data synthesis for multimodal large language models. arXiv preprint arXiv:2408.04594. — [arXiv:2408.04594](https://arxiv.org/abs/2408.04594) [find](https://scholar.google.com/scholar?q=Qirui+Jiao%2C+Daoyuan+Chen%2C+Yilun+Huang%2C+Yaliang+Li%2C+and+Ying+Shen%2E+2024%2E+Img%2Ddiff%3A+Contrastive+data+synthesis+for+multimodal+large+language+models%2E+arXiv+preprint+arXiv%3A2408%2E04594%2E+%E2%80%94+%5BarXiv%3A2408%2E04594%5D)
19. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). Advances in neural information processing systems, 35:22199– 22213.
20. Max Ku, Tianle Li, Kai Zhang, Yujie Lu, Xingyu Fu, Wenwen Zhuang, and Wenhu Chen. 2023. Im-agenhub: Standardizing the evaluation of conditional image generation models. arXiv preprint arXiv:2310.01596. — [arXiv:2310.01596](https://arxiv.org/abs/2310.01596) [find](https://scholar.google.com/scholar?q=Max+Ku%2C+Tianle+Li%2C+Kai+Zhang%2C+Yujie+Lu%2C+Xingyu+Fu%2C+Wenwen+Zhuang%2C+and+Wenhu+Chen%2E+2023%2E+Im%2Dagenhub%3A+Standardizing+the+evaluation+of+conditional+image+generation+models%2E+arXiv+preprint+arXiv%3A2310%2E01596)
21. Xuanyu Lei, Zonghan Yang, Xinrui Chen, Peng Li, and Yang Liu. 2024. [Scaffolding coordinates to promote vision-language coordination in large multi-modal models](https://md.research.elamur.ai/free/en/papers/lei-2025-scaffolding-coordinates-promote-vision-language/). Preprint, arXiv:2402.12058.
22. Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. 2024a. Naturalbench: Evaluating vision-language models on natural adversarial samples. arXiv preprint arXiv:2410.14669. [find](https://scholar.google.com/scholar?q=Baiqi+Li%2C+Zhiqiu+Lin%2C+Wenxuan+Peng%2C+Jean+de+Dieu+Nyandwi%2C+Daniel+Jiang%2C+Zixian+Ma%2C+Simran+Khanuja%2C+Ranjay+Krishna%2C+Graham+Neubig%2C+and+Deva+Ramanan%2E+2024a%2E+Naturalbench%3A+Evaluating+vision%2Dlanguage+mode)
23. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. 2024b. [Llava-onevision: Easy visual task transfer](https://md.research.elamur.ai/free/en/papers/li-2024-llava-onevision-easy-visual-task/). arXiv preprint arXiv:2408.03326.
24. Cheng Li, May Fung, Qingyun Wang, Chi Han, Manling Li, Jindong Wang, and Heng Ji. 2025a. Mentalarena: Self-play training of language models for diagnosis and treatment of mental health disorders. Preprint, arXiv:2410.06845. — [arXiv:2410.06845](https://arxiv.org/abs/2410.06845) [find](https://scholar.google.com/scholar?q=Cheng+Li%2C+May+Fung%2C+Qingyun+Wang%2C+Chi+Han%2C+Manling+Li%2C+Jindong+Wang%2C+and+Heng+Ji%2E+2025a%2E+Mentalarena%3A+Self%2Dplay+training+of+language+models+for+diagnosis+and+treatment+of+mental+health+disorders%2E+Prep)
25. Hengzhi Li, Megan Tjandrasuwita, Yi R. Fung, Armando Solar-Lezama, and Paul Pu Liang. 2025b. Mimeqa: Towards socially-intelligent nonverbal foundation models. Preprint, arXiv:2502.16671. — [arXiv:2502.16671](https://arxiv.org/abs/2502.16671) [find](https://scholar.google.com/scholar?q=Hengzhi+Li%2C+Megan+Tjandrasuwita%2C+Yi+R%2E+Fung%2C+Armando+Solar%2DLezama%2C+and+Paul+Pu+Liang%2E+2025b%2E+Mimeqa%3A+Towards+socially%2Dintelligent+nonverbal+foundation+models%2E+Preprint%2C+arXiv%3A2502%2E16671%2E+%E2%80%94+%5BarXiv%3A2502)
26. Li Li, Peilin Cai, Ryan A Rossi, Franck Dernoncourt, Branislav Kveton, Junda Wu, Tong Yu, Linxin Song, Tiankai Yang, Yuehan Qin, et al. 2025c. A personalized conversational benchmark: Towards simulating personalized conversations. arXiv preprint arXiv:2505.14106. — [arXiv:2505.14106](https://arxiv.org/abs/2505.14106) [find](https://scholar.google.com/scholar?q=Li+Li%2C+Peilin+Cai%2C+Ryan+A+Rossi%2C+Franck+Dernoncourt%2C+Branislav+Kveton%2C+Junda+Wu%2C+Tong+Yu%2C+Linxin+Song%2C+Tiankai+Yang%2C+Yuehan+Qin%2C+et+al%2E+2025c%2E+A+personalized+conversational+benchmark%3A+Towards+simulati)
27. You Li, Heyu Huang, Chi Chen, Kaiyu Huang, Chao Huang, Zonghao Guo, Zhiyuan Liu, Jinan Xu, Yuhua Li, Ruixuan Li, et al. 2025d. Migician: Revealing the magic of free-form multi-image grounding in multimodal large language models. arXiv preprint arXiv:2501.05767. [find](https://scholar.google.com/scholar?q=You+Li%2C+Heyu+Huang%2C+Chi+Chen%2C+Kaiyu+Huang%2C+Chao+Huang%2C+Zonghao+Guo%2C+Zhiyuan+Liu%2C+Jinan+Xu%2C+Yuhua+Li%2C+Ruixuan+Li%2C+et+al%2E+2025d%2E+Migician%3A+Revealing+the+magic+of+free%2Dform+multi%2Dimage+grounding+in+multi)
28. Paul Pu Liang, Akshay Goindani, Talha Chafekar, Leena Mathur, Haofei Yu, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2024a. Hemm: Holistic evaluation of multimodal foundation models. arXiv preprint arXiv:2407.03418. [find](https://scholar.google.com/scholar?q=Paul+Pu+Liang%2C+Akshay+Goindani%2C+Talha+Chafekar%2C+Leena+Mathur%2C+Haofei+Yu%2C+Ruslan+Salakhutdinov%2C+and+Louis%2DPhilippe+Morency%2E+2024a%2E+Hemm%3A+Holistic+evaluation+of+multimodal+foundation+models%2E+arXiv+prepr)
29. Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. 2024b. Foundations & trends in multimodal machine learning: Principles, challenges, and open questions. ACM Computing Surveys, 56(10):1– 42. [find](https://scholar.google.com/scholar?q=Paul+Pu+Liang%2C+Amir+Zadeh%2C+and+Louis%2DPhilippe+Morency%2E+2024b%2E+Foundations+%26+trends+in+multimodal+machine+learning%3A+Principles%2C+challenges%2C+and+open+questions%2E+ACM+Computing+Surveys%2C+56%2810%29%3A1%E2%80%93+42%2E)
30. Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, et al. 2024a. Mibench: Evaluating multimodal large language models over multiple images. arXiv preprint arXiv:2407.15272. [find](https://scholar.google.com/scholar?q=Haowei+Liu%2C+Xi+Zhang%2C+Haiyang+Xu%2C+Yaya+Shi%2C+Chaoya+Jiang%2C+Ming+Yan%2C+Ji+Zhang%2C+Fei+Huang%2C+Chunfeng+Yuan%2C+Bing+Li%2C+et+al%2E+2024a%2E+Mibench%3A+Evaluating+multimodal+large+language+models+over+multiple+images)
31. Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. 2024b. Tempcompass: Do video llms really understand videos? arXiv preprint arXiv:2403.00476. [find](https://scholar.google.com/scholar?q=Yuanxin+Liu%2C+Shicheng+Li%2C+Yi+Liu%2C+Yuxiang+Wang%2C+Shuhuai+Ren%2C+Lei+Li%2C+Sishuo+Chen%2C+Xu+Sun%2C+and+Lu+Hou%2E+2024b%2E+Tempcompass%3A+Do+video+llms+really+understand+videos%3F+arXiv+preprint+arXiv%3A2403%2E00476%2E)
32. Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi Dong, Pan Zhang, Zijian Liang, Yuanjun Xiong, Yu Qiao, Dahua Lin, et al. 2024c. Mmdu: A multi-turn multi-image dialog understanding benchmark and instruction-tuning dataset for lvlms. arXiv preprint arXiv:2406.11833. — [arXiv:2406.11833](https://arxiv.org/abs/2406.11833) [find](https://scholar.google.com/scholar?q=Ziyu+Liu%2C+Tao+Chu%2C+Yuhang+Zang%2C+Xilin+Wei%2C+Xiaoyi+Dong%2C+Pan+Zhang%2C+Zijian+Liang%2C+Yuanjun+Xiong%2C+Yu+Qiao%2C+Dahua+Lin%2C+et+al%2E+2024c%2E+Mmdu%3A+A+multi%2Dturn+multi%2Dimage+dialog+understanding+benchmark+and+inst)
33. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521. [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Swaroop+Mishra%2C+Tanglin+Xia%2C+Liang+Qiu%2C+Kai%2DWei+Chang%2C+Song%2DChun+Zhu%2C+Oyvind+Tafjord%2C+Peter+Clark%2C+and+Ashwin+Kalyan%2E+2022%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+scienc)
34. Romina Palermo and Gillian Rhodes. 2007. Are you always on my mind? a review of how face perception and attention interact. Neuropsychologia, 45:75–92. [find](https://scholar.google.com/scholar?q=Romina+Palermo+and+Gillian+Rhodes%2E+2007%2E+Are+you+always+on+my+mind%3F+a+review+of+how+face+perception+and+attention+interact%2E+Neuropsychologia%2C+45%3A75%E2%80%9392%2E)
35. Renjie Pi, Jianshu Zhang, Tianyang Han, Jipeng Zhang, Rui Pan, and Tong Zhang. 2024a. Personalized visual instruction tuning. arXiv preprint arXiv:2410.07113. — [arXiv:2410.07113](https://arxiv.org/abs/2410.07113) [find](https://scholar.google.com/scholar?q=Renjie+Pi%2C+Jianshu+Zhang%2C+Tianyang+Han%2C+Jipeng+Zhang%2C+Rui+Pan%2C+and+Tong+Zhang%2E+2024a%2E+Personalized+visual+instruction+tuning%2E+arXiv+preprint+arXiv%3A2410%2E07113%2E+%E2%80%94+%5BarXiv%3A2410%2E07113%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
36. Renjie Pi, Jianshu Zhang, Jipeng Zhang, Rui Pan, Zhekai Chen, and Tong Zhang. 2024b. Image textualization: An automatic framework for creating accurate and detailed image descriptions. arXiv preprint arXiv:2406.07502. — [arXiv:2406.07502](https://arxiv.org/abs/2406.07502) [find](https://scholar.google.com/scholar?q=Renjie+Pi%2C+Jianshu+Zhang%2C+Jipeng+Zhang%2C+Rui+Pan%2C+Zhekai+Chen%2C+and+Tong+Zhang%2E+2024b%2E+Image+textualization%3A+An+automatic+framework+for+creating+accurate+and+detailed+image+descriptions%2E+arXiv+preprint+)
37. Zeyu Qin, Qingxiu Dong, Xingxing Zhang, Li Dong, Xiaolong Huang, Ziyi Yang, Mahmoud Khademi, Dongdong Zhang, Hany Hassan Awadalla, Yi R. Fung, Weizhu Chen, Minhao Cheng, and Furu Wei. 2025. Scaling laws of synthetic data for language models. Preprint, arXiv:2503.19551. — [arXiv:2503.19551](https://arxiv.org/abs/2503.19551) [find](https://scholar.google.com/scholar?q=Zeyu+Qin%2C+Qingxiu+Dong%2C+Xingxing+Zhang%2C+Li+Dong%2C+Xiaolong+Huang%2C+Ziyi+Yang%2C+Mahmoud+Khademi%2C+Dongdong+Zhang%2C+Hany+Hassan+Awadalla%2C+Yi+R%2E+Fung%2C+Weizhu+Chen%2C+Minhao+Cheng%2C+and+Furu+Wei%2E+2025%2E+Scaling+la)
38. Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. 2024. Grounded sam: Assembling open-world models for diverse visual tasks. Preprint, arXiv:2401.14… — [arXiv:2401.14159](https://arxiv.org/abs/2401.14159) [find](https://scholar.google.com/scholar?q=Tianhe+Ren%2C+Shilong+Liu%2C+Ailing+Zeng%2C+Jing+Lin%2C+Kunchang+Li%2C+He+Cao%2C+Jiayu+Chen%2C+Xinyu+Huang%2C+Yukang+Chen%2C+Feng+Yan%2C+Zhaoyang+Zeng%2C+Hao+Zhang%2C+Feng+Li%2C+Jie+Yang%2C+Hongyang+Li%2C+Qing+Jiang%2C+and+Lei+Zhang)
39. Jongyoon Song, Sangwon Yu, and Sungroh Yoon. 2024. Large language models are skeptics: False negative problem of input-conflicting hallucination. arXiv preprint arXiv:2406.13929. — [arXiv:2406.13929](https://arxiv.org/abs/2406.13929) [find](https://scholar.google.com/scholar?q=Jongyoon+Song%2C+Sangwon+Yu%2C+and+Sungroh+Yoon%2E+2024%2E+Large+language+models+are+skeptics%3A+False+negative+problem+of+input%2Dconflicting+hallucination%2E+arXiv+preprint+arXiv%3A2406%2E13929%2E+%E2%80%94+%5BarXiv%3A2406%2E13929%5D%28)
40. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. [Fever: a large-scale dataset for fact extraction and verification](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/). arXiv preprint arXiv:1803.05355.
41. Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. [Eyes wide shut? exploring the visual shortcomings of multimodal llms](https://md.research.elamur.ai/free/en/papers/tong-2024-eyes-wide-shut-exploring-visual/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578.
42. Anne Treisman and Garry A. Gelade. 1980. A feature-integration theory of attention. Cognitive Psychology, 12:97–136. [find](https://scholar.google.com/scholar?q=Anne+Treisman+and+Garry+A%2E+Gelade%2E+1980%2E+A+feature%2Dintegration+theory+of+attention%2E+Cognitive+Psychology%2C+12%3A97%E2%80%93136%2E)
43. Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. 2024a. Muirbench: A comprehensive benchmark for robust multi-image understanding. arXiv preprint arXiv:2406.09411. [find](https://scholar.google.com/scholar?q=Fei+Wang%2C+Xingyu+Fu%2C+James+Y+Huang%2C+Zekun+Li%2C+Qin+Liu%2C+Xiaogeng+Liu%2C+Mingyu+Derek+Ma%2C+Nan+Xu%2C+Wenxuan+Zhou%2C+Kai+Zhang%2C+et+al%2E+2024a%2E+Muirbench%3A+A+comprehensive+benchmark+for+robust+multi%2Dimage+underst)
44. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024b. [Qwen2-vl: Enhancing vision-language model’s perception of the…](https://md.research.elamur.ai/free/en/papers/wang-2024-qwen2-vl-enhancing-vision/)
45. Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, Conghui He, Ping Luo, Ziwei Liu, Yali Wang, Limin Wang, and Yu Qiao. 2024c. Internvid: A large-scale video-text dataset for multimodal understanding and generation. Preprint, arXiv:2307.069… — [arXiv:2307.06942](https://arxiv.org/abs/2307.06942) [find](https://scholar.google.com/scholar?q=Yi+Wang%2C+Yinan+He%2C+Yizhuo+Li%2C+Kunchang+Li%2C+Jiashuo+Yu%2C+Xin+Ma%2C+Xinhao+Li%2C+Guo+Chen%2C+Xinyuan+Chen%2C+Yaohui+Wang%2C+Conghui+He%2C+Ping+Luo%2C+Ziwei+Liu%2C+Yali+Wang%2C+Limin+Wang%2C+and+Yu+Qiao%2E+2024c%2E+Internvid%3A+A+)
46. Kasun Weerakoon, Mohamed Elnoor, Gershom Seneviratne, Vignesh Rajagopal, Senthil Hariharan Arul, Jing Liang, Mohamed Khalid M Jaffar, and Dinesh Manocha. 2024. Behav: Behavioral rule guided autonomy using vlms for robot navigation in outdoor scenes. Preprint, arXiv:2409.16484. — [arXiv:2409.16484](https://arxiv.org/abs/2409.16484) [find](https://scholar.google.com/scholar?q=Kasun+Weerakoon%2C+Mohamed+Elnoor%2C+Gershom+Seneviratne%2C+Vignesh+Rajagopal%2C+Senthil+Hariharan+Arul%2C+Jing+Liang%2C+Mohamed+Khalid+M+Jaffar%2C+and+Dinesh+Manocha%2E+2024%2E+Behav%3A+Behavioral+rule+guided+autonomy+u)
47. Cong Wei, Zheyang Xiong, Weiming Ren, Xinrun Du, Ge Zhang, and Wenhu Chen. 2024. Omniedit: Building image editing generalist models through specialist supervision. arXiv preprint arXiv:2411.07199. — [arXiv:2411.07199](https://arxiv.org/abs/2411.07199) [find](https://scholar.google.com/scholar?q=Cong+Wei%2C+Zheyang+Xiong%2C+Weiming+Ren%2C+Xinrun+Du%2C+Ge+Zhang%2C+and+Wenhu+Chen%2E+2024%2E+Omniedit%3A+Building+image+editing+generalist+models+through+specialist+supervision%2E+arXiv+preprint+arXiv%3A2411%2E07199%2E+%E2%80%94+%5B)
48. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Preprint, arXiv:2201.11903.
49. Haoning Wu, Hanwei Zhu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Annan Wang, Wenxiu Sun, Qiong Yan, et al. 2025. Towards open-ended visual quality comparison. In European Conference on Computer Vision, pages 360–377. Springer. [find](https://scholar.google.com/scholar?q=Haoning+Wu%2C+Hanwei+Zhu%2C+Zicheng+Zhang%2C+Erli+Zhang%2C+Chaofeng+Chen%2C+Liang+Liao%2C+Chunyi+Li%2C+Annan+Wang%2C+Wenxiu+Sun%2C+Qiong+Yan%2C+et+al%2E+2025%2E+Towards+open%2Dended+visual+quality+comparison%2E+In+European+Confe)
50. Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. 2022. Grit: A generative region-to-text transformer for object understanding. arXiv preprint arXiv:2212.00280. — [arXiv:2212.00280](https://arxiv.org/abs/2212.00280) [find](https://scholar.google.com/scholar?q=Jialian+Wu%2C+Jianfeng+Wang%2C+Zhengyuan+Yang%2C+Zhe+Gan%2C+Zicheng+Liu%2C+Junsong+Yuan%2C+and+Lijuan+Wang%2E+2022%2E+Grit%3A+A+generative+region%2Dto%2Dtext+transformer+for+object+understanding%2E+arXiv+preprint+arXiv%3A2212%2E)
51. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. [A survey on multimodal large language models](https://md.research.elamur.ai/free/en/papers/yin-2023-survey-multimodal-large-language-models/). arXiv preprint arXiv:2306.13549.
52. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. [Mm-vet: Evaluating large multimodal models for integrated capabilities](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/). arXiv preprint arXiv:2308.02490.
53. Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. 2019. [Activitynet-qa: A dataset for understanding complex web videos via question answering](https://md.research.elamur.ai/free/en/papers/yu-2019-activitynet-qa-dataset-understanding-complex/). In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127–9134.
54. Shujin Wu, Yi Fung, Sha Li, Yixin Wan, Kai-Wei Chang, and Heng Ji. 2024. MACAROON: Training vision-language models to be your engaged partners. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7715–7731, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shujin+Wu%2C+Yi+Fung%2C+Sha+Li%2C+Yixin+Wan%2C+Kai%2DWei+Chang%2C+and+Heng+Ji%2E+2024%2E+MACAROON%3A+Training+vision%2Dlanguage+models+to+be+your+engaged+partners%2E+In+Findings+of+the+Association+for+Computational+Linguis)
55. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. [Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/). In Proceedings of the IEEE/CVF Conference on Computer Vision an…
56. Lingfeng Yang, Yueze Wang, Xiang Li, Xinlong Wang, and Jian Yang. 2023. Fine-grained visual prompting. Preprint, arXiv:2306.04356. — [arXiv:2306.04356](https://arxiv.org/abs/2306.04356) [find](https://scholar.google.com/scholar?q=Lingfeng+Yang%2C+Yueze+Wang%2C+Xiang+Li%2C+Xinlong+Wang%2C+and+Jian+Yang%2E+2023%2E+Fine%2Dgrained+visual+prompting%2E+Preprint%2C+arXiv%3A2306%2E04356%2E+%E2%80%94+%5BarXiv%3A2306%2E04356%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2306%2E04356%29)
57. Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. 2024a. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852. [find](https://scholar.google.com/scholar?q=Peiyuan+Zhang%2C+Kaichen+Zhang%2C+Bo+Li%2C+Guangtao+Zeng%2C+Jingkang+Yang%2C+Yuanhan+Zhang%2C+Ziyue+Wang%2C+Haoran+Tan%2C+Chunyuan+Li%2C+and+Ziwei+Liu%2E+2024a%2E+Long+context+transfer+from+language+to+vision%2E+arXiv+prepri)
58. Zhutian Yang, Caelan Garrett, Dieter Fox, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. 2024. Guiding long-horizon task and motion planning with vision language models. Preprint, arXiv:2410.02193. — [arXiv:2410.02193](https://arxiv.org/abs/2410.02193) [find](https://scholar.google.com/scholar?q=Zhutian+Yang%2C+Caelan+Garrett%2C+Dieter+Fox%2C+Tom%C3%A1s+Lozano%2DP%C3%A9rez%2C+and+Leslie+Pack+Kaelbling%2E+2024%2E+Guiding+long%2Dhorizon+task+and+motion+planning+with+vision+language+models%2E+Preprint%2C+arXiv%3A2410%2E02193%2E+%E2%80%94+)
59. Dongyu Yao and Boheng Li. 2023. Dual-level interaction for domain adaptive semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 4527–4536. [find](https://scholar.google.com/scholar?q=Dongyu+Yao+and+Boheng+Li%2E+2023%2E+Dual%2Dlevel+interaction+for+domain+adaptive+semantic+segmentation%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Conference+on+Computer+Vision+%28ICCV%29+Workshops%2C+pages+452)
60. Dongyu Yao, Jianshu Zhang, Ian G. Harris, and Marcel Carlsson. 2024. Fuzzllm: A novel and universal fuzzing framework for proactively discovering jailbreak vulnerabilities in large language models. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP… [find](https://scholar.google.com/scholar?q=Dongyu+Yao%2C+Jianshu+Zhang%2C+Ian+G%2E+Harris%2C+and+Marcel+Carlsson%2E+2024%2E+Fuzzllm%3A+A+novel+and+universal+fuzzing+framework+for+proactively+discovering+jailbreak+vulnerabilities+in+large+language+models%2E+In)
61. Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2024a. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. Preprint, arXiv:2408.04840. [find](https://scholar.google.com/scholar?q=Jiabo+Ye%2C+Haiyang+Xu%2C+Haowei+Liu%2C+Anwen+Hu%2C+Ming+Yan%2C+Qi+Qian%2C+Ji+Zhang%2C+Fei+Huang%2C+and+Jingren+Zhou%2E+2024a%2E+mplug%2Dowl3%3A+Towards+long+image%2Dsequence+understanding+in+multi%2Dmodal+large+language+models%2E)
62. Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. 2024b. Video instruction tuning with synthetic data. Preprint, arXiv:2410.02713. [find](https://scholar.google.com/scholar?q=Yuanhan+Zhang%2C+Jinming+Wu%2C+Wei+Li%2C+Bo+Li%2C+Zejun+Ma%2C+Ziwei+Liu%2C+and+Chunyuan+Li%2E+2024b%2E+Video+instruction+tuning+with+synthetic+data%2E+Preprint%2C+arXiv%3A2410%2E02713%2E)
63. Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. 2024c. Video instruction tuning with synthetic data. Preprint, arXiv:2410.02713. [find](https://scholar.google.com/scholar?q=Yuanhan+Zhang%2C+Jinming+Wu%2C+Wei+Li%2C+Bo+Li%2C+Zejun+Ma%2C+Ziwei+Liu%2C+and+Chunyuan+Li%2E+2024c%2E+Video+instruction+tuning+with+synthetic+data%2E+Preprint%2C+arXiv%3A2410%2E02713%2E)
64. Bingchen Zhao, Yongshuo Zong, Letian Zhang, and Timothy Hospedales. 2024. Benchmarking multi-image understanding in vision and language models: Perception, knowledge, reasoning, and multi-hop reasoning. arXiv preprint arXiv:2406.12742. — [arXiv:2406.12742](https://arxiv.org/abs/2406.12742) [find](https://scholar.google.com/scholar?q=Bingchen+Zhao%2C+Yongshuo+Zong%2C+Letian+Zhang%2C+and+Timothy+Hospedales%2E+2024%2E+Benchmarking+multi%2Dimage+understanding+in+vision+and+language+models%3A+Perception%2C+knowledge%2C+reasoning%2C+and+multi%2Dhop+reasonin)

*Updated 31 October 2026*