---
title: "Are We on the Right Way for Evaluating Large — Chen 2024"
description: "Multi-modal benchmarks are supposed to measure how well large vision-language models (LVLMs) use images."
authors: ["Lin Chen", "Jinsong Li", "Xiaoyi Dong", "Pan Zhang", "Yuhang Zang", "Zehui Chen", "Haodong Duan", "Jiaqi Wang", "Yu Qiao", "Dahua Lin", "Feng Zhao"]
year: 2024
venue: "arXiv:2403.20330"
source: "https://arxiv.org/abs/2403.20330"
doi: "10.48550/arxiv.2403.20330"
arxiv: "2403.20330"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2024-right-way-evaluating-large-vision/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2024-right-way-evaluating-large-vision/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/) / Are We on the Right Way for Evaluating Large Vision-Language Models?

# Are We on the Right Way for Evaluating Large Vision-Language Models?

**Authors:** [Lin Chen](https://md.research.elamur.ai/free/en/authors/lin-chen/), [Jinsong Li](https://md.research.elamur.ai/free/en/authors/jinsong-li/), [Xiaoyi Dong](https://md.research.elamur.ai/free/en/authors/xiaoyi-dong/), [Pan Zhang](https://md.research.elamur.ai/free/en/authors/pan-zhang/), Yuhang Zang, [Zehui Chen](https://md.research.elamur.ai/free/en/authors/zehui-chen/), [Haodong Duan](https://md.research.elamur.ai/free/en/authors/haodong-duan/), [Jiaqi Wang](https://md.research.elamur.ai/free/en/authors/jiaqi-wang/), [Yu Qiao](https://md.research.elamur.ai/free/en/authors/yu-qiao/), [Dahua Lin](https://md.research.elamur.ai/free/en/authors/dahua-lin/), [Feng Zhao](https://md.research.elamur.ai/free/en/authors/feng-zhao/) · **Year:** 2024 · **Venue:** arXiv:2403.20330 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2403.20330) · [PDF](https://arxiv.org/pdf/2403.20330) · [DOI 10.48550/arxiv.2403.20330](https://doi.org/10.48550/arxiv.2403.20330) · [arXiv 2403.20330](https://arxiv.org/abs/2403.20330) · [OpenAlex](https://openalex.org/W4415798312)

**Citations:** 103 citations (openalex-snapshot · 2026-10-14) · [160](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Multi-modal benchmarks are supposed to measure how well large vision-language models (LVLMs) use images. The authors show that many existing test items can be solved without the picture. Some questions give away the answer in their wording, and others can be answered from general knowledge stored in the language backbone. A third group seems to have been memorised during LLM or LVLM training. The authors tested over 20 text-only LLMs, plus LVLMs with images removed, on six popular benchmarks. They found these shortcuts are widespread and make models look better at multi-modal tasks than they are. In response, they built MMStar, a benchmark of 1,500 items drawn from existing datasets. An automatic filter removes items that text-only LLMs can answer. LVLMs then sort the remaining items by difficulty, and human reviewers make the final choice so that every item needs the image. The paper also introduces two metrics: multi-modal gain (MG) and multi-modal leakage (ML). On MMStar, high-resolution GPT-4V ranks first with 57.1% accuracy. The benchmark helps separate real visual reasoning from text shortcuts and training-data contamination.

## Key points
- Without seeing any images, GeminiPro scores 42.9% on MMMU. Across six benchmarks (MMMU, MMBench, ScienceQA, AI2D, SEED, MathVista), it beats random guessing by more than 24 points on average. Most LLMs can answer over half of ScienceQA and about a fifth of MMMU from text alone.
- The paper finds signs of leakage in multi-modal training. With images removed, Sphinx-X-MoE reaches 43.6% on MMMU, 17.9 points above its Mixtral-8x7B backbone. Yi-VL-34B beats its own LLM backbone on ScienceQA by 15.0 points.
- How MMStar was filtered: eight LLMs checked each item (GPT4-Turbo, GeminiPro, and six open models such as LLaMA-70B, Qwen-1.5-72B and Mixtral-8x7B). Any item answered correctly by more than 2 of them was dropped. Next, 16 LVLMs sorted the remaining items into four difficulty levels, and human reviewers made the final selection.
- MMStar covers 6 core capabilities split into 18 detailed axes. The authors evaluated 16 LVLMs on MMStar and also applied the MG and ML metrics to 7 benchmarks.
- GPT-4V (high-resolution) has the best accuracy (57.1%) and the best MG on MMStar, and its ML is low.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

160 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 149 not yet.

- [Qwen2.5-VL Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen2-5-vl-technical/)
- [Qwen3-VL Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen3-vl-technical-report/)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://md.research.elamur.ai/free/en/papers/chen-2024-expanding-performance-boundaries-open/)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://md.research.elamur.ai/free/en/papers/li-2024-llava-onevision-easy-visual-task/)
- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://md.research.elamur.ai/free/en/papers/tong-2024-cambrian-1-fully-open-vision/)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://md.research.elamur.ai/free/en/papers/wang-2024-qwen2-vl-enhancing-vision/)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://md.research.elamur.ai/free/en/papers/wang-2025-internvl3-5-advancing-open/)
- [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://md.research.elamur.ai/free/en/papers/wu-2024-deepseek-vl2-mixture-experts/)
- [Qwen2.5-Omni Technical Report](https://md.research.elamur.ai/free/en/papers/xu-2025-qwen2-5-omni-technical/)
- [Qwen3-Omni Technical Report](https://md.research.elamur.ai/free/en/papers/xu-2025-qwen3-omni-technical-report/)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://md.research.elamur.ai/free/en/papers/zhu-2025-internvl3-exploring-advanced-training/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chen2024are,
  title = {{Are We on the Right Way for Evaluating Large Vision-Language Models?}},
  author = {Chen, Lin and Li, Jinsong and Dong, Xiaoyi and Zhang, Pan and Zang, Yuhang and Chen, Zehui and Duan, Haodong and Wang, Jiaqi and Qiao, Yu and Lin, Dahua and Zhao, Feng},
  year = {2024},
  doi = {10.48550/arxiv.2403.20330},
  eprint = {2403.20330},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.20330}
}
```

- [← previous Are LLMs Breaking MT Metrics? Results of the WMT24 Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/freitag-2024-are-llms-breaking-mt-metrics/)
- [next → Assessing the Capabilities of Large Language Models in Coreference: An Evaluation](https://md.research.elamur.ai/free/en/papers/gan-2024-assessing-capabilities-large-language-models/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/)

## References

54 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Bai%2C+S%2E+Bai%2C+Y%2E+Chu%2C+Z%2E+Cui%2C+K%2E+Dang%2C+X%2E+Deng%2C+Y%2E+Fan%2C+W%2E+Ge%2C+Y%2E+Han%2C+F%2E+Huang%2C+et+al%2E+Qwen+technical+report%2E+arXiv+preprint+arXiv%3A2309%2E16609%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
2. J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Bai%2C+S%2E+Bai%2C+S%2E+Yang%2C+S%2E+Wang%2C+S%2E+Tan%2C+P%2E+Wang%2C+J%2E+Lin%2C+C%2E+Zhou%2C+and+J%2E+Zhou%2E+Qwen%2Dvl%3A+A+frontier+large+vision%2Dlanguage+model+with+versatile+abilities%2E+arXiv+preprint+arXiv%3A2308%2E12966%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/)
3. X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fu, et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. [find](https://scholar.google.com/scholar?q=X%2E+Bi%2C+D%2E+Chen%2C+G%2E+Chen%2C+S%2E+Chen%2C+D%2E+Dai%2C+C%2E+Deng%2C+H%2E+Ding%2C+K%2E+Dong%2C+Q%2E+Du%2C+Z%2E+Fu%2C+et+al%2E+Deepseek+llm%3A+Scaling+open%2Dsource+language+models+with+longtermism%2E+arXiv+preprint+arXiv%3A2401%2E02954%2C+2024%2E)
4. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. [find](https://scholar.google.com/scholar?q=T%2E+Brown%2C+B%2E+Mann%2C+N%2E+Ryder%2C+M%2E+Subbiah%2C+J%2E+D%2E+Kaplan%2C+P%2E+Dhariwal%2C+A%2E+Neelakantan%2C+P%2E+Shyam%2C+G%2E+Sastry%2C+A%2E+Askell%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+Advances+in+neural+information+process)
5. L. Chen, J. Li, X. Dong, P. Zhang, C. He, J. Wang, F. Zhao, and D. Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. [find](https://scholar.google.com/scholar?q=L%2E+Chen%2C+J%2E+Li%2C+X%2E+Dong%2C+P%2E+Zhang%2C+C%2E+He%2C+J%2E+Wang%2C+F%2E+Zhao%2C+and+D%2E+Lin%2E+Sharegpt4v%3A+Improving+large+multi%2Dmodal+models+with+better+captions%2E+arXiv+preprint+arXiv%3A2311%2E12793%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2023-sharegpt4v-improving-large-multi-modal/)
6. Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, Z. Muyan, Q. Zhang, X. Zhu, L. Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238, 2023. [find](https://scholar.google.com/scholar?q=Z%2E+Chen%2C+J%2E+Wu%2C+W%2E+Wang%2C+W%2E+Su%2C+G%2E+Chen%2C+S%2E+Xing%2C+Z%2E+Muyan%2C+Q%2E+Zhang%2C+X%2E+Zhu%2C+L%2E+Lu%2C+et+al%2E+Internvl%3A+Scaling+up+vision+foundation+models+and+aligning+for+generic+visual%2Dlinguistic+tasks%2E+arXiv+prepri) [our card](https://md.research.elamur.ai/free/en/papers/chen-2023-internvl-scaling-up-vision/)
7. S. Cheng, Z. Guo, J. Wu, K. Fang, P. Li, H. Liu, and Y. Liu. Can vision-language models think from a first-person perspective? arXiv preprint arXiv:2311.15596, 2023. — [arXiv:2311.15596](https://arxiv.org/abs/2311.15596) [find](https://scholar.google.com/scholar?q=S%2E+Cheng%2C+Z%2E+Guo%2C+J%2E+Wu%2C+K%2E+Fang%2C+P%2E+Li%2C+H%2E+Liu%2C+and+Y%2E+Liu%2E+Can+vision%2Dlanguage+models+think+from+a+first%2Dperson+perspective%3F+arXiv+preprint+arXiv%3A2311%2E15596%2C+2023%2E+%E2%80%94+%5BarXiv%3A2311%2E15596%5D%28https%3A%2F%2Farxiv)
8. W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See [https://vicuna.lmsys](https://vicuna.lmsys). org (accessed 14 April 2023), 2023. [↗](https://vicuna.lmsys)
9. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. [find](https://scholar.google.com/scholar?q=A%2E+Chowdhery%2C+S%2E+Narang%2C+J%2E+Devlin%2C+M%2E+Bosma%2C+G%2E+Mishra%2C+A%2E+Roberts%2C+P%2E+Barham%2C+H%2E+W%2E+Chung%2C+C%2E+Sutton%2C+S%2E+Gehrmann%2C+et+al%2E+Palm%3A+Scaling+language+modeling+with+pathways%2E+arXiv+preprint+arXiv%3A2204%2E023)
10. O. Contributors. Opencompass: A universal evaluation platform for foundation models. [https://github.com/open-compass/opencompass](https://github.com/open-compass/opencompass), 2023. [↗](https://github.com/open-compass/opencompass)
11. W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023. [find](https://scholar.google.com/scholar?q=W%2E+Dai%2C+J%2E+Li%2C+D%2E+Li%2C+A%2E+M%2E+H%2E+Tiong%2C+J%2E+Zhao%2C+W%2E+Wang%2C+B%2E+Li%2C+P%2E+Fung%2C+and+S%2E+Hoi%2E+Instructblip%3A+Towards+general%2Dpurpose+vision%2Dlanguage+models+with+instruction+tuning%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/)
12. X. Dong, P. Zhang, Y. Zang, Y. Cao, B. Wang, L. Ouyang, X. Wei, S. Zhang, H. Duan, M. Cao, et al. Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401.16420, 2024. [find](https://scholar.google.com/scholar?q=X%2E+Dong%2C+P%2E+Zhang%2C+Y%2E+Zang%2C+Y%2E+Cao%2C+B%2E+Wang%2C+L%2E+Ouyang%2C+X%2E+Wei%2C+S%2E+Zhang%2C+H%2E+Duan%2C+M%2E+Cao%2C+et+al%2E+Internlm%2Dxcomposer2%3A+Mastering+free%2Dform+text%2Dimage+composition+and+comprehension+in+vision%2Dlanguage+l)
13. Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang. Glm: General language model pretraining with autoregressive blank infilling. arXiv preprint arXiv:2103.10360, 2021. [find](https://scholar.google.com/scholar?q=Z%2E+Du%2C+Y%2E+Qian%2C+X%2E+Liu%2C+M%2E+Ding%2C+J%2E+Qiu%2C+Z%2E+Yang%2C+and+J%2E+Tang%2E+Glm%3A+General+language+model+pretraining+with+autoregressive+blank+infilling%2E+arXiv+preprint+arXiv%3A2103%2E10360%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/)
14. C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, Z. Qiu, W. Lin, J. Yang, X. Zheng, K. Li, X. Sun, and R. Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. [find](https://scholar.google.com/scholar?q=C%2E+Fu%2C+P%2E+Chen%2C+Y%2E+Shen%2C+Y%2E+Qin%2C+M%2E+Zhang%2C+X%2E+Lin%2C+Z%2E+Qiu%2C+W%2E+Lin%2C+J%2E+Yang%2C+X%2E+Zheng%2C+K%2E+Li%2C+X%2E+Sun%2C+and+R%2E+Ji%2E+Mme%3A+A+comprehensive+evaluation+benchmark+for+multimodal+large+language+models%2E+arXiv+pr) [our card](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/)
15. P. Gao, R. Zhang, C. Liu, L. Qiu, S. Huang, W. Lin, S. Zhao, S. Geng, Z. Lin, P. Jin, et al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. arXiv preprint arXiv:2402.05935, 2024. — [arXiv:2402.05935](https://arxiv.org/abs/2402.05935) [find](https://scholar.google.com/scholar?q=P%2E+Gao%2C+R%2E+Zhang%2C+C%2E+Liu%2C+L%2E+Qiu%2C+S%2E+Huang%2C+W%2E+Lin%2C+S%2E+Zhao%2C+S%2E+Geng%2C+Z%2E+Lin%2C+P%2E+Jin%2C+et+al%2E+Sphinx%2Dx%3A+Scaling+data+and+parameters+for+a+family+of+multi%2Dmodal+large+language+models%2E+arXiv+preprint+arX)
16. Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017. [find](https://scholar.google.com/scholar?q=Y%2E+Goyal%2C+T%2E+Khot%2C+D%2E+Summers%2DStay%2C+D%2E+Batra%2C+and+D%2E+Parikh%2E+Making+the+v+in+vqa+matter%3A+Elevating+the+role+of+image+understanding+in+visual+question+answering%2E+In+Proceedings+of+the+IEEE+conference+o) [our card](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/)
17. C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR, 2021. [find](https://scholar.google.com/scholar?q=C%2E+Jia%2C+Y%2E+Yang%2C+Y%2E+Xia%2C+Y%2E%2DT%2E+Chen%2C+Z%2E+Parekh%2C+H%2E+Pham%2C+Q%2E+Le%2C+Y%2E%2DH%2E+Sung%2C+Z%2E+Li%2C+and+T%2E+Duerig%2E+Scaling+up+visual+and+vision%2Dlanguage+representation+learning+with+noisy+text+supervision%2E+In+Internat) [our card](https://md.research.elamur.ai/free/en/papers/jia-2021-scaling-up-visual-vision-language/)
18. A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Q%2E+Jiang%2C+A%2E+Sablayrolles%2C+A%2E+Mensch%2C+C%2E+Bamford%2C+D%2E+S%2E+Chaplot%2C+D%2E+d%2E+l%2E+Casas%2C+F%2E+Bressand%2C+G%2E+Lengyel%2C+G%2E+Lample%2C+L%2E+Saulnier%2C+et+al%2E+Mistral+7b%2E+arXiv+preprint+arXiv%3A2310%2E06825%2C+2023%2E)
19. A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Q%2E+Jiang%2C+A%2E+Sablayrolles%2C+A%2E+Roux%2C+A%2E+Mensch%2C+B%2E+Savary%2C+C%2E+Bamford%2C+D%2E+S%2E+Chaplot%2C+D%2E+d%2E+l%2E+Casas%2C+E%2E+B%2E+Hanna%2C+F%2E+Bressand%2C+et+al%2E+Mixtral+of+experts%2E+arXiv+preprint+arXiv%3A2401%2E04088%2C+2024%2E)
20. A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi. A diagram is worth a dozen images. ArXiv, abs/1603.07396, 2016. [find](https://scholar.google.com/scholar?q=A%2E+Kembhavi%2C+M%2E+Salvato%2C+E%2E+Kolve%2C+M%2E+Seo%2C+H%2E+Hajishirzi%2C+and+A%2E+Farhadi%2E+A+diagram+is+worth+a+dozen+images%2E+ArXiv%2C+abs%2F1603%2E07396%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/kembhavi-2016-diagram-worth-dozen-images/)
21. B. Li, R. Wang, G. Wang, Y. Ge, Y. Ge, and Y. Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. [find](https://scholar.google.com/scholar?q=B%2E+Li%2C+R%2E+Wang%2C+G%2E+Wang%2C+Y%2E+Ge%2C+Y%2E+Ge%2C+and+Y%2E+Shan%2E+Seed%2Dbench%3A+Benchmarking+multimodal+llms+with+generative+comprehension%2E+arXiv+preprint+arXiv%3A2307%2E16125%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-seed-bench-benchmarking-multimodal-llms/)
22. J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Li%2C+D%2E+Li%2C+S%2E+Savarese%2C+and+S%2E+Hoi%2E+Blip%2D2%3A+Bootstrapping+language%2Dimage+pre%2Dtraining+with+frozen+image+encoders+and+large+language+models%2E+arXiv+preprint+arXiv%3A2301%2E12597%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/)
23. Z. Li, B. Yang, Q. Liu, Z. Ma, S. Zhang, J. Yang, Y. Sun, Y. Liu, and X. Bai. Monkey: Image resolution and text label are important things for large multi-modal models. arXiv preprint arXiv:2311.06607, 2023. [find](https://scholar.google.com/scholar?q=Z%2E+Li%2C+B%2E+Yang%2C+Q%2E+Liu%2C+Z%2E+Ma%2C+S%2E+Zhang%2C+J%2E+Yang%2C+Y%2E+Sun%2C+Y%2E+Liu%2C+and+X%2E+Bai%2E+Monkey%3A+Image+resolution+and+text+label+are+important+things+for+large+multi%2Dmodal+models%2E+arXiv+preprint+arXiv%3A2311%2E06607)
24. H. Liu, C. Li, Y. Li, and Y. J. Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. [find](https://scholar.google.com/scholar?q=H%2E+Liu%2C+C%2E+Li%2C+Y%2E+Li%2C+and+Y%2E+J%2E+Lee%2E+Improved+baselines+with+visual+instruction+tuning%2E+arXiv+preprint+arXiv%3A2310%2E03744%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/)
25. H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. [find](https://scholar.google.com/scholar?q=H%2E+Liu%2C+C%2E+Li%2C+Y%2E+Li%2C+B%2E+Li%2C+Y%2E+Zhang%2C+S%2E+Shen%2C+and+Y%2E+J%2E+Lee%2E+Llava%2Dnext%3A+Improved+reasoning%2C+ocr%2C+and+world+knowledge%2C+January+2024%2E)
26. H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023. [find](https://scholar.google.com/scholar?q=H%2E+Liu%2C+C%2E+Li%2C+Q%2E+Wu%2C+and+Y%2E+J%2E+Lee%2E+Visual+instruction+tuning%2E+arXiv+preprint+arXiv%3A2304%2E08485%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/)
27. Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023. [find](https://scholar.google.com/scholar?q=Y%2E+Liu%2C+H%2E+Duan%2C+Y%2E+Zhang%2C+B%2E+Li%2C+S%2E+Zhang%2C+W%2E+Zhao%2C+Y%2E+Yuan%2C+J%2E+Wang%2C+C%2E+He%2C+Z%2E+Liu%2C+et+al%2E+Mmbench%3A+Is+your+multi%2Dmodal+model+an+all%2Daround+player%3F+arXiv+preprint+arXiv%3A2307%2E06281%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/)
28. H. Lu, W. Liu, B. Zhang, B. Wang, K. Dong, B. Liu, J. Sun, T. Ren, Z. Li, Y. Sun, et al. Deepseek-vl: Towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525, 2024. [find](https://scholar.google.com/scholar?q=H%2E+Lu%2C+W%2E+Liu%2C+B%2E+Zhang%2C+B%2E+Wang%2C+K%2E+Dong%2C+B%2E+Liu%2C+J%2E+Sun%2C+T%2E+Ren%2C+Z%2E+Li%2C+Y%2E+Sun%2C+et+al%2E+Deepseek%2Dvl%3A+Towards+real%2Dworld+vision%2Dlanguage+understanding%2E+arXiv+preprint+arXiv%3A2403%2E05525%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/lu-2024-deepseek-vl-real-world/)
29. P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023. [find](https://scholar.google.com/scholar?q=P%2E+Lu%2C+H%2E+Bansal%2C+T%2E+Xia%2C+J%2E+Liu%2C+C%2E+Li%2C+H%2E+Hajishirzi%2C+H%2E+Cheng%2C+K%2E%2DW%2E+Chang%2C+M%2E+Galley%2C+and+J%2E+Gao%2E+Mathvista%3A+Evaluating+mathematical+reasoning+of+foundation+models+in+visual+contexts%2E+arXiv+prepri) [our card](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/)
30. P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022. [find](https://scholar.google.com/scholar?q=P%2E+Lu%2C+S%2E+Mishra%2C+T%2E+Xia%2C+L%2E+Qiu%2C+K%2E%2DW%2E+Chang%2C+S%2E%2DC%2E+Zhu%2C+O%2E+Tafjord%2C+P%2E+Clark%2C+and+A%2E+Kalyan%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+science+question+answering%2E+Advances+in+Neu)
31. G. Luo, Y. Zhou, T. Ren, S. Chen, X. Sun, and R. Ji. Cheap and quick: Efficient vision-language instruction tuning for large language models. arXiv preprint arXiv:2305.15023, 2023. — [arXiv:2305.15023](https://arxiv.org/abs/2305.15023) [find](https://scholar.google.com/scholar?q=G%2E+Luo%2C+Y%2E+Zhou%2C+T%2E+Ren%2C+S%2E+Chen%2C+X%2E+Sun%2C+and+R%2E+Ji%2E+Cheap+and+quick%3A+Efficient+vision%2Dlanguage+instruction+tuning+for+large+language+models%2E+arXiv+preprint+arXiv%3A2305%2E15023%2C+2023%2E+%E2%80%94+%5BarXiv%3A2305%2E15023)
32. Microsoft. Phi2: The surprising power of small language models. [https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/](https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/), 2023. [↗](https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/)
33. NousResearch. Nous-hermes-2-yi-34b. Nous-Hermes-2-Yi-34B, 2023. [find](https://scholar.google.com/scholar?q=NousResearch%2E+Nous%2Dhermes%2D2%2Dyi%2D34b%2E+Nous%2DHermes%2D2%2DYi%2D34B%2C+2023%2E)
34. OpenAI. Chatgpt. [https://chat.openai.com/](https://chat.openai.com/), 2023. [↗](https://chat.openai.com/)
35. OpenAI. Gpt-4v(ision) system card. [https://cdn.openai.com/papers/GPTV_System_Card.pdf](https://cdn.openai.com/papers/GPTV_System_Card.pdf), 2023. [↗](https://cdn.openai.com/papers/GPTV_System_Card.pdf)
36. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022. [find](https://scholar.google.com/scholar?q=L%2E+Ouyang%2C+J%2E+Wu%2C+X%2E+Jiang%2C+D%2E+Almeida%2C+C%2E+Wainwright%2C+P%2E+Mishkin%2C+C%2E+Zhang%2C+S%2E+Agarwal%2C+K%2E+Slama%2C+A%2E+Ray%2C+et+al%2E+Training+language+models+to+follow+instructions+with+human+feedback%2E+Advances+in+Neura)
37. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021. [find](https://scholar.google.com/scholar?q=A%2E+Radford%2C+J%2E+W%2E+Kim%2C+C%2E+Hallacy%2C+A%2E+Ramesh%2C+G%2E+Goh%2C+S%2E+Agarwal%2C+G%2E+Sastry%2C+A%2E+Askell%2C+P%2E+Mishkin%2C+J%2E+Clark%2C+et+al%2E+Learning+transferable+visual+models+from+natural+language+supervision%2E+In+Internati) [our card](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/)
38. D. Schwenk, A. Khandelwal, C. Clark, K. Marino, and R. Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, pages 146– 162. Springer, 2022. [find](https://scholar.google.com/scholar?q=D%2E+Schwenk%2C+A%2E+Khandelwal%2C+C%2E+Clark%2C+K%2E+Marino%2C+and+R%2E+Mottaghi%2E+A%2Dokvqa%3A+A+benchmark+for+visual+question+answering+using+world+knowledge%2E+In+European+Conference+on+Computer+Vision%2C+pages+146%E2%80%93+162%2E+Sp) [our card](https://md.research.elamur.ai/free/en/papers/schwenk-2022-okvqa-benchmark-visual-question-answering/)
39. P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018. [find](https://scholar.google.com/scholar?q=P%2E+Sharma%2C+N%2E+Ding%2C+S%2E+Goodman%2C+and+R%2E+Soricut%2E+Conceptual+captions%3A+A+cleaned%2C+hypernymed%2C+image+alt%2Dtext+dataset+for+automatic+image+captioning%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Asso) [our card](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/)
40. H. Taud and J.-F. Mas. Multilayer perceptron (mlp). Geomatic approaches for modeling land change scenarios, pages 451–455, 2018. [find](https://scholar.google.com/scholar?q=H%2E+Taud+and+J%2E%2DF%2E+Mas%2E+Multilayer+perceptron+%28mlp%29%2E+Geomatic+approaches+for+modeling+land+change+scenarios%2C+pages+451%E2%80%93455%2C+2018%2E)
41. G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. [find](https://scholar.google.com/scholar?q=G%2E+Team%2C+R%2E+Anil%2C+S%2E+Borgeaud%2C+Y%2E+Wu%2C+J%2E%2DB%2E+Alayrac%2C+J%2E+Yu%2C+R%2E+Soricut%2C+J%2E+Schalkwyk%2C+A%2E+M%2E+Dai%2C+A%2E+Hauth%2C+et+al%2E+Gemini%3A+a+family+of+highly+capable+multimodal+models%2E+arXiv+preprint+arXiv%3A2312%2E11805%2C) [our card](https://md.research.elamur.ai/free/en/papers/gemini-2023-team-gemini-family-highly-capable-multimodal/)
42. I. Team. Internlm: A multilingual language model with progressively enhanced capabilities, 2023. [https://huggingface.co/NousResearch/](https://huggingface.co/NousResearch/) [↗](https://huggingface.co/NousResearch/)
43. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [find](https://scholar.google.com/scholar?q=H%2E+Touvron%2C+L%2E+Martin%2C+K%2E+Stone%2C+P%2E+Albert%2C+A%2E+Almahairi%2C+Y%2E+Babaei%2C+N%2E+Bashlykov%2C+S%2E+Batra%2C+P%2E+Bhargava%2C+S%2E+Bhosale%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtuned+chat+models%2E+arXiv+preprint+arXiv%3A2)
44. J. Wang, L. Meng, Z. Weng, B. He, Z. Wu, and Y.-G. Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574, 2023. — [arXiv:2311.07574](https://arxiv.org/abs/2311.07574) [find](https://scholar.google.com/scholar?q=J%2E+Wang%2C+L%2E+Meng%2C+Z%2E+Weng%2C+B%2E+He%2C+Z%2E+Wu%2C+and+Y%2E%2DG%2E+Jiang%2E+To+see+is+to+believe%3A+Prompting+gpt%2D4v+for+better+visual+instruction+tuning%2E+arXiv+preprint+arXiv%3A2311%2E07574%2C+2023%2E+%E2%80%94+%5BarXiv%3A2311%2E07574%5D%28https)
45. W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y. Wang, J. Ji, Z. Yang, L. Zhao, X. Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023. [find](https://scholar.google.com/scholar?q=W%2E+Wang%2C+Q%2E+Lv%2C+W%2E+Yu%2C+W%2E+Hong%2C+J%2E+Qi%2C+Y%2E+Wang%2C+J%2E+Ji%2C+Z%2E+Yang%2C+L%2E+Zhao%2C+X%2E+Song%2C+et+al%2E+Cogvlm%3A+Visual+expert+for+pretrained+language+models%2E+arXiv+preprint+arXiv%3A2311%2E03079%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2023-cogvlm-visual-expert-pretrained/)
46. H. Wu, Z. Zhang, E. Zhang, C. Chen, L. Liao, A. Wang, C. Li, W. Sun, Q. Yan, G. Zhai, et al. Q-bench: A benchmark for general-purpose foundation models on low-level vision. arXiv preprint arXiv:2309.14181, 2023. — [arXiv:2309.14181](https://arxiv.org/abs/2309.14181) [find](https://scholar.google.com/scholar?q=H%2E+Wu%2C+Z%2E+Zhang%2C+E%2E+Zhang%2C+C%2E+Chen%2C+L%2E+Liao%2C+A%2E+Wang%2C+C%2E+Li%2C+W%2E+Sun%2C+Q%2E+Yan%2C+G%2E+Zhai%2C+et+al%2E+Q%2Dbench%3A+A+benchmark+for+general%2Dpurpose+foundation+models+on+low%2Dlevel+vision%2E+arXiv+preprint+arXiv%3A2309%2E1)
47. A. Yang, B. Xiao, B. Wang, B. Zhang, C. Yin, C. Lv, D. Pan, D. Wang, D. Yan, F. Yang, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Yang%2C+B%2E+Xiao%2C+B%2E+Wang%2C+B%2E+Zhang%2C+C%2E+Yin%2C+C%2E+Lv%2C+D%2E+Pan%2C+D%2E+Wang%2C+D%2E+Yan%2C+F%2E+Yang%2C+et+al%2E+Baichuan+2%3A+Open+large%2Dscale+language+models%2E+arXiv+preprint+arXiv%3A2309%2E10305%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/yang-2023-baichuan-2-open-large-scale/)
48. Q. Ye, H. Xu, G. Xu, J. Ye, M. Yan, Y. Zhou, J. Wang, A. Hu, P. Shi, Y. Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023. [find](https://scholar.google.com/scholar?q=Q%2E+Ye%2C+H%2E+Xu%2C+G%2E+Xu%2C+J%2E+Ye%2C+M%2E+Yan%2C+Y%2E+Zhou%2C+J%2E+Wang%2C+A%2E+Hu%2C+P%2E+Shi%2C+Y%2E+Shi%2C+et+al%2E+mplug%2Dowl%3A+Modularization+empowers+large+language+models+with+multimodality%2E+arXiv+preprint+arXiv%3A2304%2E14178%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/)
49. A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Young%2C+B%2E+Chen%2C+C%2E+Li%2C+C%2E+Huang%2C+G%2E+Zhang%2C+G%2E+Zhang%2C+H%2E+Li%2C+J%2E+Zhu%2C+J%2E+Chen%2C+J%2E+Chang%2C+et+al%2E+Yi%3A+Open+foundation+models+by+01%2E+ai%2E+arXiv+preprint+arXiv%3A2403%2E04652%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/)
50. W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. [find](https://scholar.google.com/scholar?q=W%2E+Yu%2C+Z%2E+Yang%2C+L%2E+Li%2C+J%2E+Wang%2C+K%2E+Lin%2C+Z%2E+Liu%2C+X%2E+Wang%2C+and+L%2E+Wang%2E+Mm%2Dvet%3A+Evaluating+large+multimodal+models+for+integrated+capabilities%2E+arXiv+preprint+arXiv%3A2308%2E02490%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/)
51. X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502, 2023. [find](https://scholar.google.com/scholar?q=X%2E+Yue%2C+Y%2E+Ni%2C+K%2E+Zhang%2C+T%2E+Zheng%2C+R%2E+Liu%2C+G%2E+Zhang%2C+S%2E+Stevens%2C+D%2E+Jiang%2C+W%2E+Ren%2C+Y%2E+Sun%2C+et+al%2E+Mmmu%3A+A+massive+multi%2Ddiscipline+multimodal+understanding+and+reasoning+benchmark+for+expert+agi%2E+arXi) [our card](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/)
52. P. Zhang, X. D. B. Wang, Y. Cao, C. Xu, L. Ouyang, Z. Zhao, S. Ding, S. Zhang, H. Duan, H. Yan, et al. Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.15112, 2023. [find](https://scholar.google.com/scholar?q=P%2E+Zhang%2C+X%2E+D%2E+B%2E+Wang%2C+Y%2E+Cao%2C+C%2E+Xu%2C+L%2E+Ouyang%2C+Z%2E+Zhao%2C+S%2E+Ding%2C+S%2E+Zhang%2C+H%2E+Duan%2C+H%2E+Yan%2C+et+al%2E+Internlm%2Dxcomposer%3A+A+vision%2Dlanguage+large+model+for+advanced+text%2Dimage+comprehension+and+compo)
53. B. Zhou, Y. Hu, X. Weng, J. Jia, J. Luo, X. Liu, J. Wu, and L. Huang. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402.14289, 2024. — [arXiv:2402.14289](https://arxiv.org/abs/2402.14289) [find](https://scholar.google.com/scholar?q=B%2E+Zhou%2C+Y%2E+Hu%2C+X%2E+Weng%2C+J%2E+Jia%2C+J%2E+Luo%2C+X%2E+Liu%2C+J%2E+Wu%2C+and+L%2E+Huang%2E+Tinyllava%3A+A+framework+of+small%2Dscale+large+multimodal+models%2E+arXiv+preprint+arXiv%3A2402%2E14289%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E14289%5D%28https%3A%2F)
54. D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. [find](https://scholar.google.com/scholar?q=D%2E+Zhu%2C+J%2E+Chen%2C+X%2E+Shen%2C+X%2E+Li%2C+and+M%2E+Elhoseiny%2E+Minigpt%2D4%3A+Enhancing+vision%2Dlanguage+understanding+with+advanced+large+language+models%2E+arXiv+preprint+arXiv%3A2304%2E10592%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/)