---
title: "Evaluating Discriminability of — Muraoka 2026"
description: "The paper asks whether current vision-language models (VLMs) can separate the details that matter from distracting or redundant content in text and…"
authors: ["Masayasu Muraoka", "Naoaki Okazaki"]
year: 2026
venue: "Proceedings of the Fifteenth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2026.lrec-1.736/"
doi: "10.63317/2iiwqjvxmcca"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/muraoka-2026-evaluating-discriminability-vision-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/muraoka-2026-evaluating-discriminability-vision-language-models/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2026](https://md.research.elamur.ai/free/papers/2026/page/2/) / Evaluating Discriminability of Vision-Language Models

# Evaluating Discriminability of Vision-Language Models

**Authors:** Masayasu Muraoka, [Naoaki Okazaki](https://md.research.elamur.ai/free/authors/naoaki-okazaki/) · **Year:** 2026 · **Venue:** Proceedings of the Fifteenth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.lrec-1.736/) · [PDF](https://aclanthology.org/2026.lrec-1.736.pdf) · [DOI 10.63317/2iiwqjvxmcca](https://doi.org/10.63317/2iiwqjvxmcca) · [OpenAlex](https://openalex.org/W7160018029)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14)

## Summary
The paper asks whether current vision-language models (VLMs) can separate the details that matter from distracting or redundant content in text and images, as people do in everyday tasks like shopping or trip planning. The authors call this skill discriminability. Existing benchmarks either use short, simple inputs or require extra skills such as multi-hop reasoning or domain knowledge. To isolate this ability, they built DiscriBench, a bilingual (English and Japanese) multiple-choice VQA benchmark. Each item has a situation, a short two-person dialogue, a question and four candidate images.

The benchmark draws on two sources. Illustrated items come from the English listening section of Japan's Common Test for University Admissions. Natural-image items come from COCO and were built to match the exam's style and difficulty.

Open and proprietary VLMs trail humans by 14.0 to 69.3 points, while humans reach at least 90% accuracy. Ablations point mainly to vision encoders that fail to tell apart near-identical images. Models also reason inconsistently across modalities. The authors will release the dataset to support work on more practical multimodal assistants.

## Key points
- **DiscriBench** has 1,200 samples in English and Japanese. Each answer is an option number from 1 to 4, and the dataset is hosted at github.com/muraoka7/discribench.
- **Exam subset:** it adapts listening questions from the Common Test, which target roughly CEFR A1–B1. Japanese situation texts were machine-translated and then manually checked, and images were resized to 1280×1280.
- **COCO subset:** the four-image sets were selected with a method from Tong et al. (2024) that looks for images one encoder wrongly treats as alike. Images had to look similar to a CLIP-style encoder but differ under DINOv2. Faces were blurred, and Claude drafted the dialogues, which were then manually validated and edited.
- **Models tested:** LLaVA-onevision, Qwen2-VL, InternVL-2.5, Claude 3.5 Sonnet and GPT-4o. All fall well short of humans.
- **Ablation findings:** easing visual discrimination helps performance more than simplifying the text. In case studies, models often fail to identify or count objects even when their stated reasoning is plausible.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{muraoka2026evaluating,
  title = {{Evaluating Discriminability of Vision-Language Models}},
  author = {Muraoka, Masayasu and Okazaki, Naoaki},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2iiwqjvxmcca},
  url = {https://doi.org/10.63317/2iiwqjvxmcca}
}
```

- [← previous Estonian WinoGrande Dataset: Comparative Analysis of LLM Performance on Human and Machine Translation](https://md.research.elamur.ai/free/papers/ojastu-2026-estonian-winogrande-dataset-comparative-analysis/)
- [next → Evaluating Frontier LLM Translation Capability for Lakota](https://md.research.elamur.ai/free/papers/robertson-2026-evaluating-frontier-llm-translation-capability/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/papers/2026/page/2/)

## References

76 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ibrahim M Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. 2023. Getting vit in shape: Scaling laws for compute-optimal model design. In NeurIPS, volume 36, pages 16406– 16425. [find](https://scholar.google.com/scholar?q=Ibrahim+M+Alabdulmohsin%2C+Xiaohua+Zhai%2C+Alexander+Kolesnikov%2C+and+Lucas+Beyer%2E+2023%2E+Getting+vit+in+shape%3A+Scaling+laws+for+compute%2Doptimal+model+design%2E+In+NeurIPS%2C+volume+36%2C+pages+16406%E2%80%93+16425%2E)
2. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy B… [find](https://scholar.google.com/scholar?q=Jean%2DBaptiste+Alayrac%2C+Jeff+Donahue%2C+Pauline+Luc%2C+Antoine+Miech%2C+Iain+Barr%2C+Yana+Hasson%2C+Karel+Lenc%2C+Arthur+Mensch%2C+Katherine+Millican%2C+Malcolm+Reynolds%2C+Roman+Ring%2C+Eliza+Rutherford%2C+Serkan+Cabi%2C+Ten) [our card](https://md.research.elamur.ai/free/papers/alayrac-2022-flamingo-visual-language-model-few/)
3. Anthropic. 2023. Anthropic python api library. Accessed on September 15, 2024. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2023%2E+Anthropic+python+api+library%2E+Accessed+on+September+15%2C+2024%2E)
4. Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. Technical report, Anthropic. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2024%2E+The+claude+3+model+family%3A+Opus%2C+sonnet%2C+haiku%2E+Technical+report%2C+Anthropic%2E)
5. Anthropic. 2025. System card:claude sonnet 4.5. Technical report, Anthropic. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2025%2E+System+card%3Aclaude+sonnet+4%2E5%2E+Technical+report%2C+Anthropic%2E)
6. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023a. Qwen technical report. arXiv preprint arXiv:2309.16609. [find](https://scholar.google.com/scholar?q=Jinze+Bai%2C+Shuai+Bai%2C+Yunfei+Chu%2C+Zeyu+Cui%2C+Kai+Dang%2C+Xiaodong+Deng%2C+Yang+Fan%2C+Wenbin+Ge%2C+Yu+Han%2C+Fei+Huang%2C+et+al%2E+2023a%2E+Qwen+technical+report%2E+arXiv+preprint+arXiv%3A2309%2E16609%2E) [our card](https://md.research.elamur.ai/free/papers/bai-2023-qwen-technical-report/)
7. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023b. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966. [find](https://scholar.google.com/scholar?q=Jinze+Bai%2C+Shuai+Bai%2C+Shusheng+Yang%2C+Shijie+Wang%2C+Sinan+Tan%2C+Peng+Wang%2C+Junyang+Lin%2C+Chang+Zhou%2C+and+Jingren+Zhou%2E+2023b%2E+Qwen%2Dvl%3A+A+frontier+large+vision%2Dlanguage+model+with+versatile+abilities%2E+arXi) [our card](https://md.research.elamur.ai/free/papers/bai-2023-qwen-vl-versatile-vision-language/)
8. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretchen)
9. Harry Bunt, Jan Alexandersson, Jae-Woong Choe, Alex Chengyu Fang, Koiti Hasida, Volha Petukhova, Andrei Popescu-Belis, and David Traum. 2012. ISO 24617-2: A semantically-based standard for dialogue annotation. In Proc. 2012 LREC, pages 430–437. [find](https://scholar.google.com/scholar?q=Harry+Bunt%2C+Jan+Alexandersson%2C+Jae%2DWoong+Choe%2C+Alex+Chengyu+Fang%2C+Koiti+Hasida%2C+Volha+Petukhova%2C+Andrei+Popescu%2DBelis%2C+and+David+Traum%2E+2012%2E+ISO+24617%2D2%3A+A+semantically%2Dbased+standard+for+dialogue+an) [our card](https://md.research.elamur.ai/free/papers/bunt-2012-iso-24617-2-semantically-based/)
10. Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. 2024a. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271. [find](https://scholar.google.com/scholar?q=Zhe+Chen%2C+Weiyun+Wang%2C+Yue+Cao%2C+Yangzhou+Liu%2C+Zhangwei+Gao%2C+Erfei+Cui%2C+Jinguo+Zhu%2C+Shenglong+Ye%2C+Hao+Tian%2C+Zhaoyang+Liu%2C+et+al%2E+2024a%2E+Expanding+performance+boundaries+of+open%2Dsource+multimodal+models) [our card](https://md.research.elamur.ai/free/papers/chen-2024-expanding-performance-boundaries-open/)
11. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2024b. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proc. IEEE/CVF CVPR, page… [find](https://scholar.google.com/scholar?q=Zhe+Chen%2C+Jiannan+Wu%2C+Wenhai+Wang%2C+Weijie+Su%2C+Guo+Chen%2C+Sen+Xing%2C+Muyan+Zhong%2C+Qinglong+Zhang%2C+Xizhou+Zhu%2C+Lewei+Lu%2C+Bin+Li%2C+Ping+Luo%2C+Tong+Lu%2C+Yu+Qiao%2C+and+Jifeng+Dai%2E+2024b%2E+Internvl%3A+Scaling+up+vis) [our card](https://md.research.elamur.ai/free/papers/chen-2023-internvl-scaling-up-vision/)
12. Council of Europe. 2020. Common European Framework of Reference for Languages: Learning, teaching, assessment – Companion volume. Council of Europe Publishing, Strasbourg. [find](https://scholar.google.com/scholar?q=Council+of+Europe%2E+2020%2E+Common+European+Framework+of+Reference+for+Languages%3A+Learning%2C+teaching%2C+assessment+%E2%80%93+Companion+volume%2E+Council+of+Europe+Publishing%2C+Strasbourg%2E)
13. Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. 2024. Vision transformers need registers. In 20th ICLR. [find](https://scholar.google.com/scholar?q=Timoth%C3%A9e+Darcet%2C+Maxime+Oquab%2C+Julien+Mairal%2C+and+Piotr+Bojanowski%2E+2024%2E+Vision+transformers+need+registers%2E+In+20th+ICLR%2E)
14. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR. [find](https://scholar.google.com/scholar?q=Alexey+Dosovitskiy%2C+Lucas+Beyer%2C+Alexander+Kolesnikov%2C+Dirk+Weissenborn%2C+Xiaohua+Zhai%2C+Thomas+Unterthiner%2C+Mostafa+Dehghani%2C+Matthias+Minderer%2C+Georg+Heigold%2C+Sylvain+Gelly%2C+Jakob+Uszkoreit%2C+and+Neil+)
15. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. [find](https://scholar.google.com/scholar?q=Abhimanyu+Dubey%2C+Abhinav+Jauhri%2C+Abhinav+Pandey%2C+Abhishek+Kadian%2C+Ahmad+Al%2DDahle%2C+Aiesha+Letman%2C+Akhil+Mathur%2C+Alan+Schelten%2C+Amy+Yang%2C+Angela+Fan%2C+et+al%2E+2024%2E+The+llama+3+herd+of+models%2E+arXiv+prepr)
16. Bradley Efron and R. J. Tibshirani. 1993. An introduction to the bootstrap. 57. Chapman & Hall. [find](https://scholar.google.com/scholar?q=Bradley+Efron+and+R%2E+J%2E+Tibshirani%2E+1993%2E+An+introduction+to+the+bootstrap%2E+57%2E+Chapman+%26+Hall%2E)
17. Flickr. 2004. Flickr. Accessed on December 14, 2024. [find](https://scholar.google.com/scholar?q=Flickr%2E+2004%2E+Flickr%2E+Accessed+on+December+14%2C+2024%2E)
18. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. CLIPScore: A reference-free evaluation metric for image captioning. In Proc. 2021 EMNLP, pages 7514– 7528, Online and Punta Cana, Dominican Republic. [find](https://scholar.google.com/scholar?q=Jack+Hessel%2C+Ari+Holtzman%2C+Maxwell+Forbes%2C+Ronan+Le+Bras%2C+and+Yejin+Choi%2E+2021%2E+CLIPScore%3A+A+reference%2Dfree+evaluation+metric+for+image+captioning%2E+In+Proc%2E+2021+EMNLP%2C+pages+7514%E2%80%93+7528%2C+Online+and+Pu) [our card](https://md.research.elamur.ai/free/papers/hessel-2021-clipscore-reference-free-evaluation-metric/)
19. Raisa Islam and Owana Marzia Moushi. 2024. Gpt-4o: The cutting-edge advancement in multimodal llm. TechRxiv. [find](https://scholar.google.com/scholar?q=Raisa+Islam+and+Owana+Marzia+Moushi%2E+2024%2E+Gpt%2D4o%3A+The+cutting%2Dedge+advancement+in+multimodal+llm%2E+TechRxiv%2E)
20. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. [find](https://scholar.google.com/scholar?q=Saurav+Kadavath%2C+Tom+Conerly%2C+Amanda+Askell%2C+Tom+Henighan%2C+Dawn+Drain%2C+Ethan+Perez%2C+Nicholas+Schiefer%2C+Zac+Hatfield%2DDodds%2C+Nova+DasSarma%2C+Eli+Tran%2DJohnson%2C+et+al%2E+2022%2E+Language+models+%28mostly%29+know+w)
21. Mutsumi Kondo and Mari Miyao. 2023. Does the common test measure the thinking skills necessary for the 21st century? ARELE, 34:81–96. [find](https://scholar.google.com/scholar?q=Mutsumi+Kondo+and+Mari+Miyao%2E+2023%2E+Does+the+common+test+measure+the+thinking+skills+necessary+for+the+21st+century%3F+ARELE%2C+34%3A81%E2%80%9396%2E)
22. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proc. 29th SOSP, page 611–626, New York, NY, USA. [find](https://scholar.google.com/scholar?q=Woosuk+Kwon%2C+Zhuohan+Li%2C+Siyuan+Zhuang%2C+Ying+Sheng%2C+Lianmin+Zheng%2C+Cody+Hao+Yu%2C+Joseph+Gonzalez%2C+Hao+Zhang%2C+and+Ion+Stoica%2E+2023%2E+Efficient+memory+management+for+large+language+model+serving+with+page)
23. Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. 2024. Building and better understanding vision-language models: insights and future directions. arXiv preprint arXiv:2408.12637. — [arXiv:2408.12637](https://arxiv.org/abs/2408.12637) [find](https://scholar.google.com/scholar?q=Hugo+Lauren%C3%A7on%2C+Andr%C3%A9s+Marafioti%2C+Victor+Sanh%2C+and+L%C3%A9o+Tronchon%2E+2024%2E+Building+and+better+understanding+vision%2Dlanguage+models%3A+insights+and+future+directions%2E+arXiv+preprint+arXiv%3A2408%2E12637%2E+%E2%80%94+%5BarX)
24. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326. [find](https://scholar.google.com/scholar?q=Bo+Li%2C+Yuanhan+Zhang%2C+Dong+Guo%2C+Renrui+Zhang%2C+Feng+Li%2C+Hao+Zhang%2C+Kaichen+Zhang%2C+Yanwei+Li%2C+Ziwei+Liu%2C+and+Chunyuan+Li%2E+2024%2E+Llava%2Donevision%3A+Easy+visual+task+transfer%2E+arXiv+preprint+arXiv%3A2408%2E0332) [our card](https://md.research.elamur.ai/free/papers/li-2024-llava-onevision-easy-visual-task/)
25. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proc. 40th ICML. [find](https://scholar.google.com/scholar?q=Junnan+Li%2C+Dongxu+Li%2C+Silvio+Savarese%2C+and+Steven+Hoi%2E+2023%2E+Blip%2D2%3A+bootstrapping+language%2Dimage+pre%2Dtraining+with+frozen+image+encoders+and+large+language+models%2E+In+Proc%2E+40th+ICML%2E) [our card](https://md.research.elamur.ai/free/papers/li-2023-blip-2-bootstrapping-language-image/)
26. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proc. 39th ICML, volume 162, pages 12888–12900. [find](https://scholar.google.com/scholar?q=Junnan+Li%2C+Dongxu+Li%2C+Caiming+Xiong%2C+and+Steven+Hoi%2E+2022%2E+BLIP%3A+Bootstrapping+language%2Dimage+pre%2Dtraining+for+unified+vision%2Dlanguage+understanding+and+generation%2E+In+Proc%2E+39th+ICML%2C+volume+162%2C+pag) [our card](https://md.research.elamur.ai/free/papers/li-2022-blip-bootstrapping-language-image-pre/)
27. Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. 2024. Vila: On pre-training for visual language models. In Proc. IEEE/CVF CVPR, pages 26689–26699. [find](https://scholar.google.com/scholar?q=Ji+Lin%2C+Hongxu+Yin%2C+Wei+Ping%2C+Pavlo+Molchanov%2C+Mohammad+Shoeybi%2C+and+Song+Han%2E+2024%2E+Vila%3A+On+pre%2Dtraining+for+visual+language+models%2E+In+Proc%2E+IEEE%2FCVF+CVPR%2C+pages+26689%E2%80%9326699%2E) [our card](https://md.research.elamur.ai/free/papers/lin-2023-vila-pre-training-visual-language/)
28. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In Proc. IEEE/CVF CVPR, pages 26296–26306. [find](https://scholar.google.com/scholar?q=Haotian+Liu%2C+Chunyuan+Li%2C+Yuheng+Li%2C+and+Yong+Jae+Lee%2E+2024%2E+Improved+baselines+with+visual+instruction+tuning%2E+In+Proc%2E+IEEE%2FCVF+CVPR%2C+pages+26296%E2%80%9326306%2E) [our card](https://md.research.elamur.ai/free/papers/liu-2023-improved-baselines-visual-instruction-tuning/)
29. Meta AI. 2025. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. Accessed: 2026-03-04. [find](https://scholar.google.com/scholar?q=Meta+AI%2E+2025%2E+The+llama+4+herd%3A+The+beginning+of+a+new+era+of+natively+multimodal+ai+innovation%2E+Accessed%3A+2026%2D03%2D04%2E)
30. MIC. 2024. WHITE PAPER Information and Communications in Japan, pages 75–88. Ministry of Internal Affairs and Communications, Japan. [find](https://scholar.google.com/scholar?q=MIC%2E+2024%2E+WHITE+PAPER+Information+and+Communications+in+Japan%2C+pages+75%E2%80%9388%2E+Ministry+of+Internal+Affairs+and+Communications%2C+Japan%2E)
31. Microsoft, :, Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi ling Chen, Qi Dai, Xiyang Dai, Ruchao Fan, Mei Gao, Min Gao,… [find](https://scholar.google.com/scholar?q=Microsoft%2C+%3A%2C+Abdelrahman+Abouelenin%2C+Atabak+Ashfaq%2C+Adam+Atkinson%2C+Hany+Awadalla%2C+Nguyen+Bach%2C+Jianmin+Bao%2C+Alon+Benhaim%2C+Martin+Cai%2C+Vishrav+Chaudhary%2C+Congcong+Chen%2C+Dong+Chen%2C+Dongdong+Chen%2C+Junku) [our card](https://md.research.elamur.ai/free/papers/microsoft-2025-phi-4-mini-technical-report/)
32. Microsoft. 2024. Phi-3 technical report: A highly capable language model locally on your phone. Technical report, Microsoft. [find](https://scholar.google.com/scholar?q=Microsoft%2E+2024%2E+Phi%2D3+technical+report%3A+A+highly+capable+language+model+locally+on+your+phone%2E+Technical+report%2C+Microsoft%2E) [our card](https://md.research.elamur.ai/free/papers/abdin-2024-phi-3-technical-report-highly/)
33. Mixtral AI. 2024. Pixtral 12b. Technical report, Mixtral AI. [find](https://scholar.google.com/scholar?q=Mixtral+AI%2E+2024%2E+Pixtral+12b%2E+Technical+report%2C+Mixtral+AI%2E)
34. NCUEE. 2024. About the national center for university entrance examinations. Accessed on December 14, 2024. [find](https://scholar.google.com/scholar?q=NCUEE%2E+2024%2E+About+the+national+center+for+university+entrance+examinations%2E+Accessed+on+December+14%2C+2024%2E)
35. NVIDIA Research Projects. 2024. Vila: Optimized vision language models. Accessed on August 23, 2024. [find](https://scholar.google.com/scholar?q=NVIDIA+Research+Projects%2E+2024%2E+Vila%3A+Optimized+vision+language+models%2E+Accessed+on+August+23%2C+2024%2E)
36. OpenAI. 2023a. Gpt-4 technical report. Technical report, OpenAI. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023a%2E+Gpt%2D4+technical+report%2E+Technical+report%2C+OpenAI%2E) [our card](https://md.research.elamur.ai/free/papers/openai-2023-gpt-4-technical-report/)
37. OpenAI. 2023b. Gpt-4v(ision) system card. Technical report, OpenAI. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023b%2E+Gpt%2D4v%28ision%29+system+card%2E+Technical+report%2C+OpenAI%2E)
38. OpenAI. 2023. Openai python api library. Accessed on October 11, 2024. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023%2E+Openai+python+api+library%2E+Accessed+on+October+11%2C+2024%2E)
39. OpenAI. 2025. Gpt-5 system card. Technical report, OpenAI. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025%2E+Gpt%2D5+system+card%2E+Technical+report%2C+OpenAI%2E)
40. OpenAI. 2025. Introducing gpt-4.1 in the api. Accessed on March 4, 2026. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025%2E+Introducing+gpt%2D4%2E1+in+the+api%2E+Accessed+on+March+4%2C+2026%2E)
41. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZ-IZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabrie… [find](https://scholar.google.com/scholar?q=Maxime+Oquab%2C+Timoth%C3%A9e+Darcet%2C+Th%C3%A9o+Moutakanni%2C+Huy+V%2E+Vo%2C+Marc+Szafraniec%2C+Vasil+Khalidov%2C+Pierre+Fernandez%2C+Daniel+HAZ%2DIZA%2C+Francisco+Massa%2C+Alaaeldin+El%2DNouby%2C+Mido+Assran%2C+Nicolas+Ballas%2C+Wojciech) [our card](https://md.research.elamur.ai/free/papers/oquab-2023-dinov2-learning-robust-visual-features/)
42. Mansi Phute, Alec Helbling, Matthew Daniel Hull, ShengYun Peng, Sebastian Szyller, Cory Cornelius, and Duen Horng Chau. 2024. LLM self defense: By self examination, LLMs know they are being tricked. In 2nd Tiny Papers Track at ICLR 2024. [find](https://scholar.google.com/scholar?q=Mansi+Phute%2C+Alec+Helbling%2C+Matthew+Daniel+Hull%2C+ShengYun+Peng%2C+Sebastian+Szyller%2C+Cory+Cornelius%2C+and+Duen+Horng+Chau%2E+2024%2E+LLM+self+defense%3A+By+self+examination%2C+LLMs+know+they+are+being+tricked%2E+I)
43. QwenTeam. 2025. Qwen3-vl: Sharper vision, deeper thought, broader action. Technical report, QwenTeam. [find](https://scholar.google.com/scholar?q=QwenTeam%2E+2025%2E+Qwen3%2Dvl%3A+Sharper+vision%2C+deeper+thought%2C+broader+action%2E+Technical+report%2C+QwenTeam%2E)
44. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proc. 38th ICML, volume 139, pages… [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jong+Wook+Kim%2C+Chris+Hallacy%2C+Aditya+Ramesh%2C+Gabriel+Goh%2C+Sandhini+Agarwal%2C+Girish+Sastry%2C+Amanda+Askell%2C+Pamela+Mishkin%2C+Jack+Clark%2C+Gretchen+Krueger%2C+and+Ilya+Sutskever%2E+2021%2E+Learning) [our card](https://md.research.elamur.ai/free/papers/radford-2021-learning-transferable-visual-models-natural/)
45. Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proc. IEEE/CVF CVPR, pages 9568–9578. [find](https://scholar.google.com/scholar?q=Shengbang+Tong%2C+Zhuang+Liu%2C+Yuexiang+Zhai%2C+Yi+Ma%2C+Yann+LeCun%2C+and+Saining+Xie%2E+2024%2E+Eyes+wide+shut%3F+exploring+the+visual+shortcomings+of+multimodal+llms%2E+In+Proc%2E+IEEE%2FCVF+CVPR%2C+pages+9568%E2%80%939578%2E) [our card](https://md.research.elamur.ai/free/papers/tong-2024-eyes-wide-shut-exploring-visual/)
46. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Thibaut+Lavril%2C+Gautier+Izacard%2C+Xavier+Martinet%2C+Marie%2DAnne+Lachaux%2C+Timoth%C3%A9e+Lacroix%2C+Baptiste+Rozi%C3%A8re%2C+Naman+Goyal%2C+Eric+Hambro%2C+Faisal+Azhar%2C+et+al%2E+2023a%2E+Llama%3A+Open+and+efficient+)
47. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023b%2E+Llama+2%3A+Open+foundation+and+f)
48. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the w… [find](https://scholar.google.com/scholar?q=Peng+Wang%2C+Shuai+Bai%2C+Sinan+Tan%2C+Shijie+Wang%2C+Zhihao+Fan%2C+Jinze+Bai%2C+Keqin+Chen%2C+Xuejing+Liu%2C+Jialin+Wang%2C+Wenbin+Ge%2C+Yang+Fan%2C+Kai+Dang%2C+Mengfei+Du%2C+Xuancheng+Ren%2C+Rui+Men%2C+Dayiheng+Liu%2C+Chang+Zhou%2C+) [our card](https://md.research.elamur.ai/free/papers/wang-2024-qwen2-vl-enhancing-vision/)
49. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, Zhaokai Wang, Zhe Chen, Hongjie Zhang, Ganlin Yang, Haomin Wang, Qi Wei, Jinhui Yin, Wenhao Li, Erfei Cui, Guanzhou Chen, Zichen Ding, Changyao Tian, Zhenyu Wu, Jingjing Xie,… [find](https://scholar.google.com/scholar?q=Weiyun+Wang%2C+Zhangwei+Gao%2C+Lixin+Gu%2C+Hengjun+Pu%2C+Long+Cui%2C+Xingguang+Wei%2C+Zhaoyang+Liu%2C+Linglin+Jing%2C+Shenglong+Ye%2C+Jie+Shao%2C+Zhaokai+Wang%2C+Zhe+Chen%2C+Hongjie+Zhang%2C+Ganlin+Yang%2C+Haomin+Wang%2C+Qi+Wei%2C+J) [our card](https://md.research.elamur.ai/free/papers/wang-2025-internvl3-5-advancing-open/)
50. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pierric+Cistac%2C+Tim+Rault%2C+Remi+Louf%2C+Morgan+Funtowicz%2C+Joe+Davison%2C+Sam+Shleifer%2C+Patrick+von+Platen%2C+Clara+M) [our card](https://md.research.elamur.ai/free/papers/wolf-2020-transformers-state-art-natural-language/)
51. Junda Wu, Zhehao Zhang, Yu Xia, Xintong Li, Zhaoyang Xia, Aaron Chang, Tong Yu, Sungchul Kim, Ryan A Rossi, Ruiyi Zhang, et al. 2024. Visual prompting in multimodal large language models: A survey. arXiv preprint arXiv:2409.15310. — [arXiv:2409.15310](https://arxiv.org/abs/2409.15310) [find](https://scholar.google.com/scholar?q=Junda+Wu%2C+Zhehao+Zhang%2C+Yu+Xia%2C+Xintong+Li%2C+Zhaoyang+Xia%2C+Aaron+Chang%2C+Tong+Yu%2C+Sungchul+Kim%2C+Ryan+A+Rossi%2C+Ruiyi+Zhang%2C+et+al%2E+2024%2E+Visual+prompting+in+multimodal+large+language+models%3A+A+survey%2E+ar)
52. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Zhou%2C+Chengpeng+Li%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+et+al%2E+2024%2E+Qwen2+technical+report%2E+arXiv+preprint+arXiv%3A2407%2E10671%2E)
53. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proc. IEEE/CVF ICCV, pages 11975–11986. 9. [find](https://scholar.google.com/scholar?q=Xiaohua+Zhai%2C+Basil+Mustafa%2C+Alexander+Kolesnikov%2C+and+Lucas+Beyer%2E+2023%2E+Sigmoid+loss+for+language+image+pre%2Dtraining%2E+In+Proc%2E+IEEE%2FCVF+ICCV%2C+pages+11975%E2%80%9311986%2E+9%2E) [our card](https://md.research.elamur.ai/free/papers/zhai-2023-sigmoid-loss-language-image-pre/)
54. Language Resource References Antol, Stanislaw and Agrawal, Aishwarya and Lu, Jiasen and Mitchell, Margaret and Batra, Dhruv and Lawrence Zitnick, C and Parikh, Devi. 2015. Vqa: Visual question answering. [find](https://scholar.google.com/scholar?q=Language+Resource+References+Antol%2C+Stanislaw+and+Agrawal%2C+Aishwarya+and+Lu%2C+Jiasen+and+Mitchell%2C+Margaret+and+Batra%2C+Dhruv+and+Lawrence+Zitnick%2C+C+and+Parikh%2C+Devi%2E+2015%2E+Vqa%3A+Visual+question+answeri)
55. Byeon, Minwoo and Park, Beomhee and Kim, Haecheon and Lee, Sungjun and Baek, Woonhyuk and Kim, Saehoon. 2022. COYO-700M: Image-Text Pair Dataset. [find](https://scholar.google.com/scholar?q=Byeon%2C+Minwoo+and+Park%2C+Beomhee+and+Kim%2C+Haecheon+and+Lee%2C+Sungjun+and+Baek%2C+Woonhyuk+and+Kim%2C+Saehoon%2E+2022%2E+COYO%2D700M%3A+Image%2DText+Pair+Dataset%2E)
56. Chen, Lin and Li, Jisong and Dong, Xiaoyi and Zhang, Pan and He, Conghui and Wang, Jiaqi and Zhao, Feng and Lin, Dahua. 2023. ShareGPT4V: Improving Large Multi-Modal Models with Better Captions. [find](https://scholar.google.com/scholar?q=Chen%2C+Lin+and+Li%2C+Jisong+and+Dong%2C+Xiaoyi+and+Zhang%2C+Pan+and+He%2C+Conghui+and+Wang%2C+Jiaqi+and+Zhao%2C+Feng+and+Lin%2C+Dahua%2E+2023%2E+ShareGPT4V%3A+Improving+Large+Multi%2DModal+Models+with+Better+Captions%2E) [our card](https://md.research.elamur.ai/free/papers/chen-2023-sharegpt4v-improving-large-multi-modal/)
57. Fu, Chaoyou and Chen, Peixian and Shen, Yunhang and Qin, Yulei and Zhang, Mengdan and Lin, Xu and Yang, Jinrui and Zheng, Xiawu and Li, Ke and Sun, Xing and others. 2023. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. [find](https://scholar.google.com/scholar?q=Fu%2C+Chaoyou+and+Chen%2C+Peixian+and+Shen%2C+Yunhang+and+Qin%2C+Yulei+and+Zhang%2C+Mengdan+and+Lin%2C+Xu+and+Yang%2C+Jinrui+and+Zheng%2C+Xiawu+and+Li%2C+Ke+and+Sun%2C+Xing+and+others%2E+2023%2E+MME%3A+A+Comprehensive+Evaluati) [our card](https://md.research.elamur.ai/free/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/)
58. Fu, Xingyu and Hu, Yushi and Li, Bangzheng and Feng, Yu and Wang, Haoyu and Lin, Xudong and Roth, Dan and Smith, Noah A. and Ma, Wei-Chiu and Krishna, Ranjay. 2024. BLINK: Multimodal Large Language Models Can See but Not Perceive. [find](https://scholar.google.com/scholar?q=Fu%2C+Xingyu+and+Hu%2C+Yushi+and+Li%2C+Bangzheng+and+Feng%2C+Yu+and+Wang%2C+Haoyu+and+Lin%2C+Xudong+and+Roth%2C+Dan+and+Smith%2C+Noah+A%2E+and+Ma%2C+Wei%2DChiu+and+Krishna%2C+Ranjay%2E+2024%2E+BLINK%3A+Multimodal+Large+Language+Mo)
59. Gurari, Danna and Li, Qing and Stangl, Abigale J and Guo, Anhong and Lin, Chi and Grauman, Kristen and Luo, Jiebo and Bigham, Jeffrey P. 2018. Vizwiz grand challenge: Answering visual questions from blind people. [find](https://scholar.google.com/scholar?q=Gurari%2C+Danna+and+Li%2C+Qing+and+Stangl%2C+Abigale+J+and+Guo%2C+Anhong+and+Lin%2C+Chi+and+Grauman%2C+Kristen+and+Luo%2C+Jiebo+and+Bigham%2C+Jeffrey+P%2E+2018%2E+Vizwiz+grand+challenge%3A+Answering+visual+questions+from+b)
60. Kil, Jihyung and Mai, Zheda and Lee, Justin and Chowdhury, Arpita and Wang, Zihe and Cheng, Kerrie and Wang, Lemeng and Liu, Ye and Chao, Wei-Lun. 2024. MLLM-CompBench: A Comparative Reasoning Benchmark for Multimodal LLMs. [find](https://scholar.google.com/scholar?q=Kil%2C+Jihyung+and+Mai%2C+Zheda+and+Lee%2C+Justin+and+Chowdhury%2C+Arpita+and+Wang%2C+Zihe+and+Cheng%2C+Kerrie+and+Wang%2C+Lemeng+and+Liu%2C+Ye+and+Chao%2C+Wei%2DLun%2E+2024%2E+MLLM%2DCompBench%3A+A+Comparative+Reasoning+Benchma)
61. Li, Bo and Zhang, Kaichen and Zhang, Hao and Guo, Dong and Zhang, Renrui and Li, Feng and Zhang, Yuanhan and Liu, Ziwei and Li, Chunyuan. 2024a. LLaVA-NeXT: Stronger LLMs Supercharge Multimodal Capabilities in the Wild. [find](https://scholar.google.com/scholar?q=Li%2C+Bo+and+Zhang%2C+Kaichen+and+Zhang%2C+Hao+and+Guo%2C+Dong+and+Zhang%2C+Renrui+and+Li%2C+Feng+and+Zhang%2C+Yuanhan+and+Liu%2C+Ziwei+and+Li%2C+Chunyuan%2E+2024a%2E+LLaVA%2DNeXT%3A+Stronger+LLMs+Supercharge+Multimodal+Capabi)
62. Li, Bohao and Ge, Yuying and Ge, Yixiao and Wang, Guangzhi and Wang, Rui and Zhang, Ruimao and Shan, Ying. 2024b. SEED-Bench: Benchmarking Multimodal Large Language Models. [find](https://scholar.google.com/scholar?q=Li%2C+Bohao+and+Ge%2C+Yuying+and+Ge%2C+Yixiao+and+Wang%2C+Guangzhi+and+Wang%2C+Rui+and+Zhang%2C+Ruimao+and+Shan%2C+Ying%2E+2024b%2E+SEED%2DBench%3A+Benchmarking+Multimodal+Large+Language+Models%2E)
63. Li, Feng and Zhang, Renrui and Zhang, Hao and Zhang, Yuanhan and Li, Bo and Li, Wei and Ma, Zejun and Li, Chunyuan. 2024c. LLaVA-NeXT: Tackling Multi-image, Video, and 3D in Large Multimodal Models. [find](https://scholar.google.com/scholar?q=Li%2C+Feng+and+Zhang%2C+Renrui+and+Zhang%2C+Hao+and+Zhang%2C+Yuanhan+and+Li%2C+Bo+and+Li%2C+Wei+and+Ma%2C+Zejun+and+Li%2C+Chunyuan%2E+2024c%2E+LLaVA%2DNeXT%3A+Tackling+Multi%2Dimage%2C+Video%2C+and+3D+in+Large+Multimodal+Models%2E)
64. Li, Yifan and Du, Yifan and Zhou, Kun and Wang, Jinpeng and Zhao, Xin and Wen, Ji-Rong. 2023. Evaluating Object Hallucination in Large Vision-Language Models. [find](https://scholar.google.com/scholar?q=Li%2C+Yifan+and+Du%2C+Yifan+and+Zhou%2C+Kun+and+Wang%2C+Jinpeng+and+Zhao%2C+Xin+and+Wen%2C+Ji%2DRong%2E+2023%2E+Evaluating+Object+Hallucination+in+Large+Vision%2DLanguage+Models%2E) [our card](https://md.research.elamur.ai/free/papers/li-2023-evaluating-object-hallucination-large-vision/)
65. Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Dollár, Piotr and Zitnick, C Lawrence. 2014. Microsoft coco: Common objects in context. [find](https://scholar.google.com/scholar?q=Lin%2C+Tsung%2DYi+and+Maire%2C+Michael+and+Belongie%2C+Serge+and+Hays%2C+James+and+Perona%2C+Pietro+and+Ramanan%2C+Deva+and+Doll%C3%A1r%2C+Piotr+and+Zitnick%2C+C+Lawrence%2E+2014%2E+Microsoft+coco%3A+Common+objects+in+context%2E) [our card](https://md.research.elamur.ai/free/papers/lin-2014-microsoft-coco-common-objects-context/)
66. Liu, Haotian and Li, Chunyuan and Wu, Qingyang and Lee, Yong Jae. 2023. Visual Instruction Tuning. [find](https://scholar.google.com/scholar?q=Liu%2C+Haotian+and+Li%2C+Chunyuan+and+Wu%2C+Qingyang+and+Lee%2C+Yong+Jae%2E+2023%2E+Visual+Instruction+Tuning%2E) [our card](https://md.research.elamur.ai/free/papers/liu-2023-visual-instruction-tuning/)
67. Liu, Yuan and Duan, Haodong and Zhang, Yuanhan and Li, Bo and Zhang, Songyang and Zhao, Wangbo and Yuan, Yike and Wang, Jiaqi and He, Conghui and Liu, Ziwei and others. 2024. Mmbench: Is your multi-modal model an all-around player? [find](https://scholar.google.com/scholar?q=Liu%2C+Yuan+and+Duan%2C+Haodong+and+Zhang%2C+Yuanhan+and+Li%2C+Bo+and+Zhang%2C+Songyang+and+Zhao%2C+Wangbo+and+Yuan%2C+Yike+and+Wang%2C+Jiaqi+and+He%2C+Conghui+and+Liu%2C+Ziwei+and+others%2E+2024%2E+Mmbench%3A+Is+your+multi%2Dmo) [our card](https://md.research.elamur.ai/free/papers/liu-2023-mmbench-multi-modal-model-all/)
68. Pan Lu and Hritik Bansal and Tony Xia and Jiacheng Liu and Chunyuan Li and Hannaneh Hajishirzi and Hao Cheng and Kai-Wei Chang and Michel Galley and Jianfeng Gao. 2024. Math-Vista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts. [find](https://scholar.google.com/scholar?q=Pan+Lu+and+Hritik+Bansal+and+Tony+Xia+and+Jiacheng+Liu+and+Chunyuan+Li+and+Hannaneh+Hajishirzi+and+Hao+Cheng+and+Kai%2DWei+Chang+and+Michel+Galley+and+Jianfeng+Gao%2E+2024%2E+Math%2DVista%3A+Evaluating+Mathemat)
69. Miyai, Atsuyuki and Yang, Jingkang and Zhang, Jingyang and Ming, Yifei and Yu, Qing and Irie, Go and Li, Yixuan and Li, Hai Helen and Liu, Ziwei and Aizawa, Kiyoharu. 2025. Unsolvable Problem Detection: Robust Understanding Evaluation for Large Multimodal Models. [find](https://scholar.google.com/scholar?q=Miyai%2C+Atsuyuki+and+Yang%2C+Jingkang+and+Zhang%2C+Jingyang+and+Ming%2C+Yifei+and+Yu%2C+Qing+and+Irie%2C+Go+and+Li%2C+Yixuan+and+Li%2C+Hai+Helen+and+Liu%2C+Ziwei+and+Aizawa%2C+Kiyoharu%2E+2025%2E+Unsolvable+Problem+Detectio)
70. Suhr, Alane and Zhou, Stephanie and Zhang, Ally and Zhang, Iris and Bai, Huajun and Artzi, Yoav. 2019. A Corpus for Reasoning about Natural Language Grounded in Photographs. [find](https://scholar.google.com/scholar?q=Suhr%2C+Alane+and+Zhou%2C+Stephanie+and+Zhang%2C+Ally+and+Zhang%2C+Iris+and+Bai%2C+Huajun+and+Artzi%2C+Yoav%2E+2019%2E+A+Corpus+for+Reasoning+about+Natural+Language+Grounded+in+Photographs%2E) [our card](https://md.research.elamur.ai/free/papers/suhr-2019-corpus-reasoning-natural-language-grounded/)
71. Fei Wang and Xingyu Fu and James Y. Huang and Zekun Li and Qin Liu and Xiaogeng Liu and Mingyu Derek Ma and Nan Xu and Wenxuan Zhou and Kai Zhang and Tianyi Lorena Yan and Wenjie Jacky Mo and Hsiang-Hui Liu and Pan Lu and Chunyuan Li and Chaowei Xiao and Kai-Wei Chang and Dan Roth and Sheng Zhang an… [find](https://scholar.google.com/scholar?q=Fei+Wang+and+Xingyu+Fu+and+James+Y%2E+Huang+and+Zekun+Li+and+Qin+Liu+and+Xiaogeng+Liu+and+Mingyu+Derek+Ma+and+Nan+Xu+and+Wenxuan+Zhou+and+Kai+Zhang+and+Tianyi+Lorena+Yan+and+Wenjie+Jacky+Mo+and+Hsiang%2DH)
72. Weihao Yu and Zhengyuan Yang and Linjie Li and Jianfeng Wang and Kevin Lin and Zicheng Liu and Xinchao Wang and Lijuan Wang. 2024. MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities. [find](https://scholar.google.com/scholar?q=Weihao+Yu+and+Zhengyuan+Yang+and+Linjie+Li+and+Jianfeng+Wang+and+Kevin+Lin+and+Zicheng+Liu+and+Xinchao+Wang+and+Lijuan+Wang%2E+2024%2E+MM%2DVet%3A+Evaluating+Large+Multimodal+Models+for+Integrated+Capabilitie) [our card](https://md.research.elamur.ai/free/papers/yu-2023-mm-vet-evaluating-large-multimodal/)
73. Yue, Xiang and Ni, Yuansheng and Zhang, Kai and Zheng, Tianyu and Liu, Ruoqi and Zhang, Ge and Stevens, Samuel and Jiang, Dongfu and Ren, Weiming and Sun, Yuxuan and others. 2024a. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. [find](https://scholar.google.com/scholar?q=Yue%2C+Xiang+and+Ni%2C+Yuansheng+and+Zhang%2C+Kai+and+Zheng%2C+Tianyu+and+Liu%2C+Ruoqi+and+Zhang%2C+Ge+and+Stevens%2C+Samuel+and+Jiang%2C+Dongfu+and+Ren%2C+Weiming+and+Sun%2C+Yuxuan+and+others%2E+2024a%2E+Mmmu%3A+A+massive+mul) [our card](https://md.research.elamur.ai/free/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/)
74. Yue, Xiang and Zheng, Tianyu and Ni, Yuansheng and Wang, Yubo and Zhang, Kai and Tong, Shengbang and Sun, Yuxuan and Yu, Botao and Zhang, Ge and Sun, Huan and others. 2024b. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. [find](https://scholar.google.com/scholar?q=Yue%2C+Xiang+and+Zheng%2C+Tianyu+and+Ni%2C+Yuansheng+and+Wang%2C+Yubo+and+Zhang%2C+Kai+and+Tong%2C+Shengbang+and+Sun%2C+Yuxuan+and+Yu%2C+Botao+and+Zhang%2C+Ge+and+Sun%2C+Huan+and+others%2E+2024b%2E+Mmmu%2Dpro%3A+A+more+robust+mu)
75. Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and Zhang, Hao and Gonzalez, Joseph E. and Stoica, Ion. 2024. Judging LLM-as-a-judge with MT-bench and Chatbot Arena. [find](https://scholar.google.com/scholar?q=Zheng%2C+Lianmin+and+Chiang%2C+Wei%2DLin+and+Sheng%2C+Ying+and+Zhuang%2C+Siyuan+and+Wu%2C+Zhanghao+and+Zhuang%2C+Yonghao+and+Lin%2C+Zi+and+Li%2C+Zhuohan+and+Li%2C+Dacheng+and+Xing%2C+Eric+P%2E+and+Zhang%2C+Hao+and+Gonzalez%2C+Jo)
76. Vision encoder A OpenAI’s CLIP-L/14-336 SigLIP-SO400M-14/384 InternViT-6B-448-V1-2 LLaVA-1.5’s CLIP-L/14-336 VILA-13B’s SigLIP-SO400M-14/384 VILA-40B’s InternViT-6B-448-V1-2 0.95 0.95 0.988 0.91 0.94 0.99 Table 1: Threshold α used for each vision encoder in four-image set collection. [find](https://scholar.google.com/scholar?q=Vision+encoder+A+OpenAI%E2%80%99s+CLIP%2DL%2F14%2D336+SigLIP%2DSO400M%2D14%2F384+InternViT%2D6B%2D448%2DV1%2D2+LLaVA%2D1%2E5%E2%80%99s+CLIP%2DL%2F14%2D336+VILA%2D13B%E2%80%99s+SigLIP%2DSO400M%2D14%2F384+VILA%2D40B%E2%80%99s+InternViT%2D6B%2D448%2DV1%2D2+0%2E95+0%2E95+0%2E988+0%2E91+0%2E94+)