---
title: "MMMU-Pro — paper by Yue et al. (2024) — summary and ideas"
description: "MMMU is a widely used college-level benchmark of image–text questions, but some of its items can… 72 references, cited by 120 papers here."
authors: ["Xiang Yue", "Tianyu Zheng", "Yuansheng Ni", "Yubo Wang", "Kai Zhang", "Shengbang Tong", "Yuxuan Sun", "Botao Yu", "Ge Zhang", "Huan Sun", "Yu Su", "Wenhu Chen", "Graham Neubig"]
year: 2024
venue: "arXiv:2409.02813"
source: "https://arxiv.org/abs/2409.02813"
doi: "10.48550/arxiv.2409.02813"
arxiv: "2409.02813"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yue-2024-mmmu-pro-more-robust-multi/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yue-2024-mmmu-pro-more-robust-multi/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark

# MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark

**Authors:** [Xiang Yue](https://md.research.elamur.ai/free/en/authors/xiang-yue/), [Tianyu Zheng](https://md.research.elamur.ai/free/en/authors/tianyu-zheng/), [Yuansheng Ni](https://md.research.elamur.ai/free/en/authors/yuansheng-ni/), [Yubo Wang](https://md.research.elamur.ai/free/en/authors/yubo-wang/), [Kai Zhang](https://md.research.elamur.ai/free/en/authors/kai-zhang/), [Shengbang Tong](https://md.research.elamur.ai/free/en/authors/shengbang-tong/), [Yuxuan Sun](https://md.research.elamur.ai/free/en/authors/yuxuan-sun/), [Botao Yu](https://md.research.elamur.ai/free/en/authors/botao-yu/), [Ge Zhang](https://md.research.elamur.ai/free/en/authors/ge-zhang/), [Huan Sun](https://md.research.elamur.ai/free/en/authors/huan-sun/), [Yu Su](https://md.research.elamur.ai/free/en/authors/yu-su/), [Wenhu Chen](https://md.research.elamur.ai/free/en/authors/wenhu-chen/) and [Graham Neubig](https://md.research.elamur.ai/free/en/authors/graham-neubig/) · **Year:** 2024 · **Venue:** arXiv:2409.02813 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2409.02813) · [PDF](https://arxiv.org/pdf/2409.02813) · [DOI 10.48550/arxiv.2409.02813](https://doi.org/10.48550/arxiv.2409.02813) · [arXiv 2409.02813](https://arxiv.org/abs/2409.02813) · [OpenAlex](https://openalex.org/W4403589924)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [120](https://md.research.elamur.ai/free/en/papers/yue-2024-mmmu-pro-more-robust-multi/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
MMMU is a widely used college-level benchmark of image–text questions, but some of its items can be solved without the image. Text-only LLMs succeed either because the picture is irrelevant or because they exploit cues in the answer choices. MMMU-Pro rebuilds the benchmark in three stages:
- **Filtering:** questions that strong open-source LLMs answer correctly without images are discarded.
- **Option expansion:** the remaining items get ten answer choices instead of four, which makes guessing harder.
- **Vision-only variant:** annotators photograph or screenshot each question so that text and image arrive as a single picture.

Across proprietary and open-source multimodal models, accuracy falls substantially compared with MMMU. Explicit OCR prompting barely changes results. Chain-of-Thought prompting usually helps, but its benefit depends on the model and domain. The authors conclude that reading embedded text is not the bottleneck. The harder problem is reasoning over mixed visual-textual content. This makes MMMU-Pro a more realistic test of how people actually share problems with AI systems, for example as screenshots.

## Key points
- **Filtering:** Llama3-70B-Instruct, Qwen2-72B-Instruct, Yi-1.5-34B-Chat and Mixtral-8×22B-Instruct each attempted every question ten times without images. A question was dropped if at least three of the four models solved it in most runs. Then 1,800 items were sampled, 60 for each of 30 subjects.
- **Option augmentation:** GPT-4o drafted the extra options, Claude 3.5 screened them, and humans reviewed them in two rounds. Experts removed 70 incoherent items, leaving 1,730 questions.
- **Final size:** the release contains 3,460 items, 1,730 in standard format plus 1,730 vision-only versions. The vision-only versions vary in background, font style and font size.
- **Model performance:** scores drop by 16.8%–26.9% relative to MMMU, where GPT-4o had reached 69.1%. Chain-of-Thought helps most in Tech & Engineering and Business, but is weak or even harmful in Art & Design.
- **Additional resources:**
  - A tool that converts a JSON file of questions and images into screenshots for generating text-rich training data.
  - A vision-encoder comparison: DINOv2 scores 37.1 on MMMU and 17.4 on MMMU-Pro Vision, versus 37.9 and 16.7 for SigLIP.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works

## Cite

Cite as: Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen and Graham Neubig. 2024. MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark. arXiv:2409.02813.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{yue2024mmmu,
  title = {{MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark}},
  author = {Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen and Graham Neubig},
  year = {2024},
  doi = {10.48550/arxiv.2409.02813},
  eprint = {2409.02813},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2409.02813}
}
```

- [← previous Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models](https://md.research.elamur.ai/free/en/papers/li-2024-mini-gemini-mining-potential-multi/)
- [next → Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://md.research.elamur.ai/free/en/papers/hubinger-2024-sleeper-agents-training-deceptive-llms/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

72 references, 49 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. [Phi-3 technical report: A highly capable language model locally on your phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/). ArXiv preprint, abs/2404.14219.
2. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. [Flamingo: a visual language model for few-shot learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/). In Advances in Neural Information Processing Systems.
3. Anthropic. 2024. Claude 3.5 sonnet. [https://www.anthropic.com/news/claude-3-5-sonnet](https://www.anthropic.com/news/claude-3-5-sonnet). [↗](https://www.anthropic.com/news/claude-3-5-sonnet)
4. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: visual question answering. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 2425–2433. IEEE Computer Society… [find](https://scholar.google.com/scholar?q=Stanislaw+Antol%2C+Aishwarya+Agrawal%2C+Jiasen+Lu%2C+Margaret+Mitchell%2C+Dhruv+Batra%2C+C%2E+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+2015%2E+VQA%3A+visual+question+answering%2E+In+2015+IEEE+International+Conference+on+Comp)
5. Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. 2023. [Openflamingo: An open-source framework for training large autoregressive vision-language models](https://md.research.elamur.ai/free/en/papers/awadalla-2023-openflamingo-open-source-framework-training/). ArXiv preprint, abs/2308.01390.
6. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. [Uniter: Universal image-text representation learning](https://md.research.elamur.ai/free/en/papers/chen-2019-uniter-universal-image-text-representation/). In European Conference on Computer Vision, pages 104–120.
7. Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024. [How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites](https://md.research.elamur.ai/free/en/papers/chen-2024-far-gpt-4v-closing/). ArXiv preprint, abs/2404.16821.
8. Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. 2023. Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges. ArXiv preprint, abs/2311.03287. — [arXiv:2311.03287](https://arxiv.org/abs/2311.03287) [find](https://scholar.google.com/scholar?q=Chenhang+Cui%2C+Yiyang+Zhou%2C+Xinyu+Yang%2C+Shirley+Wu%2C+Linjun+Zhang%2C+James+Zou%2C+and+Huaxiu+Yao%2E+2023%2E+Holistic+analysis+of+hallucination+in+gpt%2D4v+%28ision%29%3A+Bias+and+interference+challenges%2E+ArXiv+preprint)
9. Wenliang Dai, Junnan Li, DONGXU LI, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2023. [Instructblip: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/). In Advances in Neural Information Processing Systems, volume 36, pages 49250–49267. Cur…
10. Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu. 2023. Shortcut learning of large language models in natural language understanding. Communications of the ACM, 67(1):110–120. [find](https://scholar.google.com/scholar?q=Mengnan+Du%2C+Fengxiang+He%2C+Na+Zou%2C+Dacheng+Tao%2C+and+Xia+Hu%2E+2023%2E+Shortcut+learning+of+large+language+models+in+natural+language+understanding%2E+Communications+of+the+ACM%2C+67%281%29%3A110%E2%80%93120%2E)
11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). ArXiv preprint, abs/2407.21783.
12. Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024. [Blink: Multimodal large language models can see but not perceive](https://md.research.elamur.ai/free/en/papers/fu-2024-blink-multimodal-large-language-models/). ArXiv preprint, abs/2404.12390.
13. Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. 2023. [Llama-adapter v2: Parameter-efficient visual instruction model](https://md.research.elamur.ai/free/en/papers/gao-2023-llama-adapter-v2-parameter-efficient/). ArXiv preprint, abs/2304.15010.
14. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. [Making the V in VQA matter: Elevating the role of image understanding in visual question answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pa…
15. Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. 2024. [Mantis: Interleaved multi-image instruction tuning](https://md.research.elamur.ai/free/en/papers/jiang-2024-mantis-interleaved-multi-image-instruction/). ArXiv preprint, abs/2405.01483.
16. Yizhang Jin, Jian Li, Yexin Liu, Tianjun Gu, Kai Wu, Zhengkai Jiang, Muyang He, Bo Zhao, Xin Tan, Zhenye Gan, et al. 2024. Efficient multimodal [large language models: A survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/). ArXiv preprint, abs/2405.10739. — [arXiv:2405.10739](https://arxiv.org/abs/2405.10739)
17. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. 2024. [VisualWebArena: Evaluating multimodal agents on realistic visual web tasks](https://md.research.elamur.ai/free/en/papers/koh-2024-visualwebarena-evaluating-multimodal-agents-realistic/). In Proceedings of the 62nd Annual Meeting of the Association for Computatio…
18. Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. 2024. Building and better understanding vision-language models: insights and future directions. ArXiv preprint, abs/2408.12637. — [arXiv:2408.12637](https://arxiv.org/abs/2408.12637) [find](https://scholar.google.com/scholar?q=Hugo+Lauren%C3%A7on%2C+Andr%C3%A9s+Marafioti%2C+Victor+Sanh%2C+and+L%C3%A9o+Tronchon%2E+2024%2E+Building+and+better+understanding+vision%2Dlanguage+models%3A+insights+and+future+directions%2E+ArXiv+preprint%2C+abs%2F2408%2E12637%2E+%E2%80%94+%5BarXi)
19. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. 2023. [Otter: A multi-modal model with in-context instruction tuning](https://md.research.elamur.ai/free/en/papers/li-2023-otter-multi-modal-model-context/). ArXiv preprint, abs/2305.03726.
20. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024a. [Llava-onevision: Easy visual task transfer](https://md.research.elamur.ai/free/en/papers/li-2024-llava-onevision-easy-visual-task/). ArXiv preprint, abs/2408.03326.
21. Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2024b. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13299–13308. [find](https://scholar.google.com/scholar?q=Bohao+Li%2C+Yuying+Ge%2C+Yixiao+Ge%2C+Guangzhi+Wang%2C+Rui+Wang%2C+Ruimao+Zhang%2C+and+Ying+Shan%2E+2024b%2E+Seed%2Dbench%3A+Benchmarking+multimodal+large+language+models%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Com)
22. Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024c. [Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models](https://md.research.elamur.ai/free/en/papers/li-2024-llava-next-interleave-tackling-multi/). ArXiv preprint, abs/2407.07895.
23. Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020. [Oscar: Object-semantics aligned pre-training for vision-language tasks](https://md.research.elamur.ai/free/en/papers/li-2020-oscar-object-semantics-aligned-pre/). In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceeding…
24. Zekun Li, Xianjun Yang, Kyuri Choi, Wanrong Zhu, Ryan Hsieh, HyeonJung Kim, Jin Hyuk Lim, Sungyoung Ji, Byungju Lee, Xifeng Yan, et al. 2024d. [Mmsci: A multimodal multi-discipline dataset for phd-level scientific comprehension](https://md.research.elamur.ai/free/en/papers/li-2024-mmsci-dataset-graduate-level-multi/). ArXiv preprint, abs/2407.04903.
25. Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. 2024. [Vila: On pre-training for visual language models](https://md.research.elamur.ai/free/en/papers/lin-2023-vila-pre-training-visual-language/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26689–26699.
26. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pag…
27. Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2023a. [Hallusionbench: You see what you think?](https://md.research.elamur.ai/free/en/papers/guan-2023-hallusionbench-advanced-diagnostic-suite-entangled/) or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models. ArXiv preprint…
28. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023b. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/). In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following.
29. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024a. Llava-next: Improved reasoning, ocr, and world knowledge. [find](https://scholar.google.com/scholar?q=Haotian+Liu%2C+Chunyuan+Li%2C+Yuheng+Li%2C+Bo+Li%2C+Yuanhan+Zhang%2C+Sheng+Shen%2C+and+Yong+Jae+Lee%2E+2024a%2E+Llava%2Dnext%3A+Improved+reasoning%2C+ocr%2C+and+world+knowledge%2E)
30. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023c. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). In Advances in Neural Information Processing Systems, volume 36, pages 34892–34916. Curran Associates, Inc.
31. Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. 2024b. Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding? Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Junpeng+Liu%2C+Yifan+Song%2C+Bill+Yuchen+Lin%2C+Wai+Lam%2C+Graham+Neubig%2C+Yuanzhi+Li%2C+and+Xiang+Yue%2E+2024b%2E+Visualwebbench%3A+How+far+have+multimodal+llms+evolved+in+web+page+understanding+and+grounding%3F+Confer)
32. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2023d. [Mmbench: Is your multi-modal model an all-around player?](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/) ArXiv preprint, abs/2307.06281.
33. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. [Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks](https://md.research.elamur.ai/free/en/papers/lu-2019-vilbert-pretraining-task-agnostic-visiolinguistic/). In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, Decembe…
34. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023a. [Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). ArXiv preprint, abs/2310.02255.
35. Yujie Lu, Xiujun Li, William Yang Wang, and Yejin Choi. 2023b. Vim: Probing multimodal large language models for visual embedded instruction following. ArXiv preprint, abs/2311.17647. — [arXiv:2311.17647](https://arxiv.org/abs/2311.17647) [find](https://scholar.google.com/scholar?q=Yujie+Lu%2C+Xiujun+Li%2C+William+Yang+Wang%2C+and+Yejin+Choi%2E+2023b%2E+Vim%3A+Probing+multimodal+large+language+models+for+visual+embedded+instruction+following%2E+ArXiv+preprint%2C+abs%2F2311%2E17647%2E+%E2%80%94+%5BarXiv%3A2311%2E17)
36. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. OK-VQA: A visual question answering benchmark requiring external knowledge. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 3195–3204. Computer Vision F… [find](https://scholar.google.com/scholar?q=Kenneth+Marino%2C+Mohammad+Rastegari%2C+Ali+Farhadi%2C+and+Roozbeh+Mottaghi%2E+2019%2E+OK%2DVQA%3A+A+visual+question+answering+benchmark+requiring+external+knowledge%2E+In+IEEE+Conference+on+Computer+Vision+and+Patte)
37. Mistral. 2024. [https://mistral.ai/news/pixtral-12b](https://mistral.ai/news/pixtral-12b). [↗](https://mistral.ai/news/pixtral-12b)
38. Masoud Monajatipoor, Liunian Harold Li, Mozhdeh Rouhsedaghat, Lin Yang, and Kai-Wei Chang. 2023. MetaVL: Transferring in-context learning ability from language models to vision-language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Shor… [find](https://scholar.google.com/scholar?q=Masoud+Monajatipoor%2C+Liunian+Harold+Li%2C+Mozhdeh+Rouhsedaghat%2C+Lin+Yang%2C+and+Kai%2DWei+Chang%2E+2023%2E+MetaVL%3A+Transferring+in%2Dcontext+learning+ability+from+language+models+to+vision%2Dlanguage+models%2E+In+Pro)
39. OpenAI. 2023. Gpt-4v(ision) system card. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023%2E+Gpt%2D4v%28ision%29+system+card%2E)
40. OpenAI. 2024a. Gpt-4o mini: advancing cost-efficient intelligence. [https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/). [↗](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/)
41. OpenAI. 2024b. Hello [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/). gpt4-o. [↗](https://openai.com/index/hello-gpt-4o/)
42. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. 2023. [Dinov2: Learning robust visual features without supervision](https://md.research.elamur.ai/free/en/papers/oquab-2023-dinov2-learning-robust-visual-features/). arXiv preprint arXiv:2304.07193.
43. Qwen. 2024. Qwen2-vl: To see the world more clearly. [https://qwenlm.github.io/blog/qwen2-vl/](https://qwenlm.github.io/blog/qwen2-vl/) . [↗](https://qwenlm.github.io/blog/qwen2-vl/)
44. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). ArXiv preprint, abs/2403.0…
45. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). ArXiv preprint, abs/2312.11805.
46. Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. 2024a. [Cambrian-1: A fully open, vision-centric exploration of multimodal llms](https://md.research.elamur.ai/free/en/papers/tong-2024-cambrian-1-fully-open-vision/). ArXiv preprint, abs/2406.16860.
47. Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024b. [Eyes wide shut? exploring the visual shortcomings of multimodal llms](https://md.research.elamur.ai/free/en/papers/tong-2024-eyes-wide-shut-exploring-visual/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578.
48. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. 2024. [Mmlu-pro: A more robust and challenging multi-task language understanding benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/). ArXiv preprint, abs/2406.01574.
49. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824–24837.
50. Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui. 2024. [From decoding to meta-generation: Inference-time algorithms for large language models](https://md.research.elamur.ai/free/en/papers/welleck-2024-decoding-meta-generation-inference-time/). arXiv preprint arXiv:2406.16838.
51. Penghao Wu and Saining Xie. 2024. [V*: Guided visual search as a core mechanism in multimodal llms](https://md.research.elamur.ai/free/en/papers/wu-2023-v-guided-visual-search-core/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13084– 13094.
52. Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. 2023. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. ArXiv preprint, abs/2306.09265. — [arXiv:2306.09265](https://arxiv.org/abs/2306.09265) [find](https://scholar.google.com/scholar?q=Peng+Xu%2C+Wenqi+Shao%2C+Kaipeng+Zhang%2C+Peng+Gao%2C+Shuo+Liu%2C+Meng+Lei%2C+Fanqing+Meng%2C+Siyuan+Huang%2C+Yu+Qiao%2C+and+Ping+Luo%2E+2023%2E+Lvlm%2Dehub%3A+A+comprehensive+evaluation+benchmark+for+large+vision%2Dlanguage+mod)
53. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. ArXiv preprint, abs/2407.10671. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Zhou%2C+Chengpeng+Li%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+et+al%2E+2024%2E+Qwen2+technical+report%2E+ArXiv+preprint%2C+abs%2F2407%2E10671%2E)
54. Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. 2024. [Minicpm-v: A gpt-4v level mllm on your phone](https://md.research.elamur.ai/free/en/papers/yao-2024-minicpm-v-gpt-4v/). ArXiv preprint, abs/2408.01800.
55. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. 2023a. [mplug-owl: Modularization empowers large language models with multimodality](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/). ArXiv preprint, abs/2304.14178.
56. Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2023b. [mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl2-revolutionizing-multi-modal/). ArXiv preprint, abs/2311.04257.
57. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023a. [A survey on multimodal large language models](https://md.research.elamur.ai/free/en/papers/yin-2023-survey-multimodal-large-language-models/). ArXiv preprint, abs/2306.13549.
58. Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Xiaoshui Huang, Zhiyong Wang, Lu Sheng, LEI BAI, Jing Shao, and Wanli Ouyang. 2023b. Lamm: Language-assisted multimodal instruction-tuning dataset, framework, and benchmark. In Advances in Neural Information Processing System… [find](https://scholar.google.com/scholar?q=Zhenfei+Yin%2C+Jiong+Wang%2C+Jianjian+Cao%2C+Zhelun+Shi%2C+Dingning+Liu%2C+Mukai+Li%2C+Xiaoshui+Huang%2C+Zhiyong+Wang%2C+Lu+Sheng%2C+LEI+BAI%2C+Jing+Shao%2C+and+Wanli+Ouyang%2E+2023b%2E+Lamm%3A+Language%2Dassisted+multimodal+instr)
59. Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024. [Yi: Open foundation models by 01](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/). ai. ArXiv preprint, abs/2403.04652.
60. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2024. [MM-vet: Evaluating large multimodal models for integrated capabilities](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/). In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Lear…
61. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. [Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/). In Proceedings of the IEEE/CVF Conference on Computer Vision an…
62. Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2023. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Mert+Yuksekgonul%2C+Federico+Bianchi%2C+Pratyusha+Kalluri%2C+Dan+Jurafsky%2C+and+James+Zou%2E+2023%2E+When+and+why+vision%2Dlanguage+models+behave+like+bags%2Dof%2Dwords%2C+and+what+to+do+about+it%3F+In+The+Eleventh+Intern)
63. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. [Sigmoid loss for language image pre-training](https://md.research.elamur.ai/free/en/papers/zhai-2023-sigmoid-loss-language-image-pre/). In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986.
64. Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen, Jifeng Dai, Yu Qiao, Dah… — [InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output](https://md.research.elamur.ai/free/en/papers/zhang-2024-internlm-xcomposer-2-5/)
65. Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021. Vinvl: Revisiting visual representations in vision-language models. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 5579–5588.… [find](https://scholar.google.com/scholar?q=Pengchuan+Zhang%2C+Xiujun+Li%2C+Xiaowei+Hu%2C+Jianwei+Yang%2C+Lei+Zhang%2C+Lijuan+Wang%2C+Yejin+Choi%2C+and+Jianfeng+Gao%2E+2021%2E+Vinvl%3A+Revisiting+visual+representations+in+vision%2Dlanguage+models%2E+In+IEEE+Conference)
66. Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. 2023. [Llama-adapter: Efficient fine-tuning of language models with zero-init attention](https://md.research.elamur.ai/free/en/papers/zhang-2023-llama-adapter-efficient-fine-tuning/). ArXiv preprint, abs/2303.16199.
67. Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. 2024b. [Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?](https://md.research.elamur.ai/free/en/papers/zhang-2024-mathverse-multi-modal-llm-truly/) arXiv preprint arXiv:2403.14624.
68. Bo Zhao, Boya Wu, and Tiejun Huang. 2023. Svit: Scaling up visual instruction tuning. ArXiv preprint, abs/2307.04087. — [arXiv:2307.04087](https://arxiv.org/abs/2307.04087) [find](https://scholar.google.com/scholar?q=Bo+Zhao%2C+Boya+Wu%2C+and+Tiejun+Huang%2E+2023%2E+Svit%3A+Scaling+up+visual+instruction+tuning%2E+ArXiv+preprint%2C+abs%2F2307%2E04087%2E+%E2%80%94+%5BarXiv%3A2307%2E04087%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2307%2E04087%29)
69. Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. 2024. [Mmicl: Empowering vision-language model with multi-modal in-context learning](https://md.research.elamur.ai/free/en/papers/zhao-2023-mmicl-empowering-vision-language-model/). The Twelfth International Conference on Learning Representations.
70. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. [Gpt-4v(ision) is a generalist web agent, if grounded](https://md.research.elamur.ai/free/en/papers/zheng-2024-gpt-4v-ision-generalist-web/). In Forty-first International Conference on Machine Learning.
71. Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao. 2020. Unified vision-language pre-training for image captioning and VQA. In Proceedings of the AAAI Conference on Artificial Intelligence, 34, pages 13041–13049. AAAI Press. [find](https://scholar.google.com/scholar?q=Luowei+Zhou%2C+Hamid+Palangi%2C+Lei+Zhang%2C+Houdong+Hu%2C+Jason+J%2E+Corso%2C+and+Jianfeng+Gao%2E+2020%2E+Unified+vision%2Dlanguage+pre%2Dtraining+for+image+captioning+and+VQA%2E+In+Proceedings+of+the+AAAI+Conference+on+A)
72. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. [Minigpt-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/). ArXiv preprint, abs/2304.10592.

*Updated 2 November 2026*