---
title: "MLLM-as-a-Judge — paper by Chen et al. (2024)"
description: "The authors build the MLLM-as-a-Judge benchmark. 82 references, cited by 55 papers here."
authors: ["Dongping Chen", "Ruoxi Chen", "Shilin Zhang", "Yinuo Liu", "Yaochen Wang", "Huichi Zhou", "Qihui Zhang", "Yao Wan", "Pan Zhou", "Lichao Sun"]
year: 2024
venue: "arXiv:2402.04788"
source: "https://arxiv.org/abs/2402.04788"
doi: "10.48550/arxiv.2402.04788"
arxiv: "2402.04788"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2024-mllm-judge-assessing-multimodal-llm/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2024-mllm-judge-assessing-multimodal-llm/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark

# MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark

**Authors:** [Dongping Chen](https://md.research.elamur.ai/free/en/authors/dongping-chen/), Ruoxi Chen, [Shilin Zhang](https://md.research.elamur.ai/free/en/authors/shilin-zhang/), Yinuo Liu, Yaochen Wang, Huichi Zhou, [Qihui Zhang](https://md.research.elamur.ai/free/en/authors/qihui-zhang/), [Yao Wan](https://md.research.elamur.ai/free/en/authors/yao-wan/), [Pan Zhou](https://md.research.elamur.ai/free/en/authors/pan-zhou/) and [Lichao Sun](https://md.research.elamur.ai/free/en/authors/lichao-sun/) · **Year:** 2024 · **Venue:** arXiv:2402.04788 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.04788) · [PDF](https://arxiv.org/pdf/2402.04788) · [DOI 10.48550/arxiv.2402.04788](https://doi.org/10.48550/arxiv.2402.04788) · [arXiv 2402.04788](https://arxiv.org/abs/2402.04788) · [OpenAlex](https://openalex.org/W4391673253)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [55](https://md.research.elamur.ai/free/en/papers/chen-2024-mllm-judge-assessing-multimodal-llm/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Automatic metrics for multimodal model outputs, such as exact matching or embedding similarity, miss fine-grained quality criteria. This paper asks whether multimodal LLMs can take over the evaluator role that text-only LLMs already play, and how well their verdicts agree with people. The authors build the MLLM-as-a-Judge benchmark. They collected 4,414 image-instruction pairs from 14 source datasets covering captioning, math, charts, text reading, diffusion and other tasks. Six MLLMs answered these pairs, producing about 17,000 responses. Eleven MLLMs then judged the responses in three formats: 1–5 scoring, pairwise choice with a tie option, and full ranking of a batch. Their verdicts were compared with independent human annotations. Judges matched human preferences fairly well in pairwise comparison but diverged clearly in scoring and batch ranking, especially on reasoning-heavy tasks. GPT-4V was the strongest judge overall. Every judge showed biases, hallucinations and inconsistent decisions. The work shows that MLLMs are not yet dependable stand-ins for human evaluators, and it supplies annotated preference data usable for reward modelling and DPO.

## Key points
- The benchmark has three steps: collecting image-instruction pairs, gathering responses from GPT-4V, Gemini, LLaVA-1.5-13b, LLaVA-1.6-34b, CogVLM and Qwen-VL-Max, and comparing judge outputs with annotations by six of the authors.
- Judging was tested with and without vision input on 11 MLLMs, including the GPT-4V, Gemini-Pro-1.0/1.5, LLaVA-1.5/1.6, Qwen-VL and CogVLM families. CogVLM could not perform batch ranking.
- The paper releases two datasets: MLLM-AS-A-JUDGE-HQ, where responses agree strongly with humans, and MLLM-AS-A-JUDGE-HARD, which holds human-inconsistent and hallucinated cases.
- The authors identify egocentric, position and length biases plus hallucination. Adding extra CoT reasoning steps and a vision expert system partly reduces these problems.
- The paper appeared at ICML 2024. Code and data are available at https://mllm-judge.github.io/.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Huazhong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/huazhong-university-of-science-and-technology/) — University, Wuhan, China · 45 works · 6,345 citations
- [Lehigh University](https://md.research.elamur.ai/free/en/universities/lehigh-university/) — University, Bethlehem, United States · 7 works · 1,280 citations
- [Zhejiang University of Technology](https://md.research.elamur.ai/free/en/universities/zhejiang-university-of-technology/) — University, Hangzhou, China · 6 works · 98 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works

## Cite

Cite as: Dongping Chen, Ruoxi Chen, Shilin Zhang, Yinuo Liu, Yaochen Wang, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou and Lichao Sun. 2024. MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark. arXiv:2402.04788.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chen2024mllm,
  title = {{MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark}},
  author = {Dongping Chen, Ruoxi Chen, Shilin Zhang, Yinuo Liu, Yaochen Wang, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou and Lichao Sun},
  year = {2024},
  doi = {10.48550/arxiv.2402.04788},
  eprint = {2402.04788},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.04788}
}
```

- [← previous Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2024-limits-survey-techniques-extend-context/)
- [next → Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning](https://md.research.elamur.ai/free/en/papers/li-2024-superfiltering-weak-strong-data-filtering/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

82 references, 39 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Chu, Z., Chen, J., Chen, Q., Yu, W., He, T., Wang, H., Peng, W., Liu, M., Qin, B., and Liu, T. A survey of chain of thought reasoning: Advances, frontiers and future. arXiv preprint arXiv:2309.15402, 2023. [find](https://scholar.google.com/scholar?q=Chu%2C+Z%2E%2C+Chen%2C+J%2E%2C+Chen%2C+Q%2E%2C+Yu%2C+W%2E%2C+He%2C+T%2E%2C+Wang%2C+H%2E%2C+Peng%2C+W%2E%2C+Liu%2C+M%2E%2C+Qin%2C+B%2E%2C+and+Liu%2C+T%2E+A+survey+of+chain+of+thought+reasoning%3A+Advances%2C+frontiers+and+future%2E+arXiv+preprint+arXiv%3A2309%2E15402%2C+)
2. Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425–2433, 2015. [find](https://scholar.google.com/scholar?q=Antol%2C+S%2E%2C+Agrawal%2C+A%2E%2C+Lu%2C+J%2E%2C+Mitchell%2C+M%2E%2C+Batra%2C+D%2E%2C+Zitnick%2C+C%2E+L%2E%2C+and+Parikh%2C+D%2E+Vqa%3A+Visual+question+answering%2E+In+Proceedings+of+the+IEEE+international+conference+on+computer+vision%2C+pp%2E+2425)
3. Cui, C., Zhou, Y., Yang, X., Wu, S., Zhang, L., Zou, J., and Yao, H. Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges. arXiv preprint arXiv:2311.03287, 2023. — [arXiv:2311.03287](https://arxiv.org/abs/2311.03287) [find](https://scholar.google.com/scholar?q=Cui%2C+C%2E%2C+Zhou%2C+Y%2E%2C+Yang%2C+X%2E%2C+Wu%2C+S%2E%2C+Zhang%2C+L%2E%2C+Zou%2C+J%2E%2C+and+Yao%2C+H%2E+Holistic+analysis+of+hallucination+in+gpt%2D4v+%28ision%29%3A+Bias+and+interference+challenges%2E+arXiv+preprint+arXiv%3A2311%2E03287%2C+2023%2E+%E2%80%94+%5Ba)
4. Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. [Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/). arXiv preprint arXiv:2308.12966, 2023a.
5. Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. [Mind2web: Towards a generalist agent for the web](https://md.research.elamur.ai/free/en/papers/deng-2023-mind2web-towards-generalist-agent-web/). Advances in Neural Information Processing Systems, 36, 2024.
6. Bai, S., Yang, S., Bai, J., Wang, P., Zhang, X., Lin, J., Wang, X., Zhou, C., and Zhou, J. Touchstone: Evaluating vision-language models by language models. arXiv preprint arXiv:2308.16890, 2023b. — [arXiv:2308.16890](https://arxiv.org/abs/2308.16890) [find](https://scholar.google.com/scholar?q=Bai%2C+S%2E%2C+Yang%2C+S%2E%2C+Bai%2C+J%2E%2C+Wang%2C+P%2E%2C+Zhang%2C+X%2E%2C+Lin%2C+J%2E%2C+Wang%2C+X%2E%2C+Zhou%2C+C%2E%2C+and+Zhou%2C+J%2E+Touchstone%3A+Evaluating+vision%2Dlanguage+models+by+language+models%2E+arXiv+preprint+arXiv%3A2308%2E16890%2C+2023b%2E+%E2%80%94+%5B)
7. Deutsch, D., Foster, G., and Freitag, M. [Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration](https://md.research.elamur.ai/free/en/papers/deutsch-2023-ties-matter-meta-evaluating-modern/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12914–12929, 2023.
8. Banerjee, S. and Lavie, A. [Meteor: An automatic metric for mt evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65–72, 2005.
9. GeminiTeam. [Gemini: A family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/), 2023.
10. Bitton, Y., Bansal, H., Hessel, J., Shao, R., Zhu, W., Awadalla, A., Gardner, J., Taori, R., and Schimdt, L. Visit-bench: A benchmark for vision-language instruction following inspired by real-world use. ArXiv, abs/2308.06595, 2023. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 260887670. — [arXiv:2308.06595](https://arxiv.org/abs/2308.06595) [↗](https://api.semanticscholar.org/CorpusID:)
11. Goutte, C. and Gaussier, E. A probabilistic interpretation of precision, recall and f-score, with implication for evaluation. In European conference on information retrieval, pp. 345–359. Springer, 2005. [find](https://scholar.google.com/scholar?q=Goutte%2C+C%2E+and+Gaussier%2C+E%2E+A+probabilistic+interpretation+of+precision%2C+recall+and+f%2Dscore%2C+with+implication+for+evaluation%2E+In+European+conference+on+information+retrieval%2C+pp%2E+345%E2%80%93359%2E+Springer%2C+20)
12. Gunjal, A., Yin, J., and Bas, E. Detecting and preventing hallucinations in large vision language models. arXiv preprint arXiv:2308.06394, 2023. [find](https://scholar.google.com/scholar?q=Gunjal%2C+A%2E%2C+Yin%2C+J%2E%2C+and+Bas%2C+E%2E+Detecting+and+preventing+hallucinations+in+large+vision+language+models%2E+arXiv+preprint+arXiv%3A2308%2E06394%2C+2023%2E)
13. Blunch, N. J. Position bias in multiple-choice questions. Journal of Marketing Research, 21(2):216–220, 1984. [find](https://scholar.google.com/scholar?q=Blunch%2C+N%2E+J%2E+Position+bias+in+multiple%2Dchoice+questions%2E+Journal+of+Marketing+Research%2C+21%282%29%3A216%E2%80%93220%2C+1984%2E)
14. Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. [A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/). arXiv preprint arXiv:2311.05232, 2023a.
15. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33: 1877–1901, 2020.
16. Huang, S., Hu, J., Yang, Z., Yang, L., Luo, T., Chen, H., Sun, L., and Yang, B. Decision mamba: Reinforcement learning via hybrid selective sequence modeling, 2024a. [find](https://scholar.google.com/scholar?q=Huang%2C+S%2E%2C+Hu%2C+J%2E%2C+Yang%2C+Z%2E%2C+Yang%2C+L%2E%2C+Luo%2C+T%2E%2C+Chen%2C+H%2E%2C+Sun%2C+L%2E%2C+and+Yang%2C+B%2E+Decision+mamba%3A+Reinforcement+learning+via+hybrid+selective+sequence+modeling%2C+2024a%2E)
17. Cai, Y., Mao, S., Wu, W., Wang, Z., Liang, Y., Ge, T., Wu, C., You, W., Song, T., Xia, Y., et al. Low-code Huang, Y., Zhang, Q., Sun, L., et al. Trustgpt: A benchmark for trustworthy and responsible large language models. arXiv preprint arXiv:2306.11507, 2023b. — [arXiv:2306.11507](https://arxiv.org/abs/2306.11507) [find](https://scholar.google.com/scholar?q=Cai%2C+Y%2E%2C+Mao%2C+S%2E%2C+Wu%2C+W%2E%2C+Wang%2C+Z%2E%2C+Liang%2C+Y%2E%2C+Ge%2C+T%2E%2C+Wu%2C+C%2E%2C+You%2C+W%2E%2C+Song%2C+T%2E%2C+Xia%2C+Y%2E%2C+et+al%2E+Low%2Dcode+Huang%2C+Y%2E%2C+Zhang%2C+Q%2E%2C+Sun%2C+L%2E%2C+et+al%2E+Trustgpt%3A+A+benchmark+for+trustworthy+and+responsible+l)
18. Li, L., Xie, Z., Li, M., Chen, S., Wang, P., Chen, L., Yang, Y., Wang, B., and Kong, L. Silkie: Preference distillation for large visual language models. arXiv preprint arXiv:2312.10665, 2023c. — [arXiv:2312.10665](https://arxiv.org/abs/2312.10665) [find](https://scholar.google.com/scholar?q=Li%2C+L%2E%2C+Xie%2C+Z%2E%2C+Li%2C+M%2E%2C+Chen%2C+S%2E%2C+Wang%2C+P%2E%2C+Chen%2C+L%2E%2C+Yang%2C+Y%2E%2C+Wang%2C+B%2E%2C+and+Kong%2C+L%2E+Silkie%3A+Preference+distillation+for+large+visual+language+models%2E+arXiv+preprint+arXiv%3A2312%2E10665%2C+2023c%2E+%E2%80%94+%5BarX)
19. Huang, Y., Yuan, Q., Sheng, X., Yang, Z., Wu, H., Chen, P., Yang, Y., Li, L., and Lin, W. Aesbench: An expert benchmark for multimodal large language models on image aesthetics perception. arXiv preprint arXiv:2401.08276, 2024b. — [arXiv:2401.08276](https://arxiv.org/abs/2401.08276) [find](https://scholar.google.com/scholar?q=Huang%2C+Y%2E%2C+Yuan%2C+Q%2E%2C+Sheng%2C+X%2E%2C+Yang%2C+Z%2E%2C+Wu%2C+H%2E%2C+Chen%2C+P%2E%2C+Yang%2C+Y%2E%2C+Li%2C+L%2E%2C+and+Lin%2C+W%2E+Aesbench%3A+An+expert+benchmark+for+multimodal+large+language+models+on+image+aesthetics+perception%2E+arXiv+prepr)
20. Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacaeval: An automatic evaluator of instruction-following models. GitHub repository, 2023d. [find](https://scholar.google.com/scholar?q=Li%2C+X%2E%2C+Zhang%2C+T%2E%2C+Dubois%2C+Y%2E%2C+Taori%2C+R%2E%2C+Gulrajani%2C+I%2E%2C+Guestrin%2C+C%2E%2C+Liang%2C+P%2E%2C+and+Hashimoto%2C+T%2E+B%2E+Alpacaeval%3A+An+automatic+evaluator+of+instruction%2Dfollowing+models%2E+GitHub+repository%2C+2023d%2E)
21. Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., and Fung, P. [Survey of hallucination in natural language generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/). ACM Computing Surveys, 55(12):1–38, 2023.
22. Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.-R. [Evaluating object hallucination in large vision-language models](https://md.research.elamur.ai/free/en/papers/li-2023-evaluating-object-hallucination-large-vision/). arXiv preprint arXiv:2305.10355, 2023e.
23. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., La… [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E%2C+Sablayrolles%2C+A%2E%2C+Roux%2C+A%2E%2C+Mensch%2C+A%2E%2C+Savary%2C+B%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2E+S%2E%2C+de+las+Casas%2C+D%2E%2C+Hanna%2C+E%2E+B%2E%2C+Bressand%2C+F%2E%2C+Lengyel%2C+G%2E%2C+Bour%2C+G%2E%2C+Lample%2C+G%2E%2C+Lavaud%2C+L%2E+R%2E%2C+Saulnier%2C)
24. Li, Z., Xu, X., Shen, T., Xu, C., Gu, J.-C., and Tao, C. Leveraging large language models for nlg evaluation: A survey. arXiv preprint arXiv:2401.07103, 2024. — [arXiv:2401.07103](https://arxiv.org/abs/2401.07103) [find](https://scholar.google.com/scholar?q=Li%2C+Z%2E%2C+Xu%2C+X%2E%2C+Shen%2C+T%2E%2C+Xu%2C+C%2E%2C+Gu%2C+J%2E%2DC%2E%2C+and+Tao%2C+C%2E+Leveraging+large+language+models+for+nlg+evaluation%3A+A+survey%2E+arXiv+preprint+arXiv%3A2401%2E07103%2C+2024%2E+%E2%80%94+%5BarXiv%3A2401%2E07103%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
25. Lin, C.-Y. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pp. 74–81, 2004.
26. Jin, P., Takanobu, R., Zhang, C., Cao, X., and Yuan, L. Chat-univi: Unified visual representation empowers large language models with image and video understanding. arXiv preprint arXiv:2311.08046, 2023. — [arXiv:2311.08046](https://arxiv.org/abs/2311.08046) [find](https://scholar.google.com/scholar?q=Jin%2C+P%2E%2C+Takanobu%2C+R%2E%2C+Zhang%2C+C%2E%2C+Cao%2C+X%2E%2C+and+Yuan%2C+L%2E+Chat%2Dunivi%3A+Unified+visual+representation+empowers+large+language+models+with+image+and+video+understanding%2E+arXiv+preprint+arXiv%3A2311%2E08046%2C+20)
27. Lin, T.-Y., Maire, M., Belongie, S. J., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In European Conference on Computer Vision, 2014. URL [https://api.semanticscholar.org/CorpusID:14113767](https://api.semanticscholar.org/CorpusID:14113767). [↗](https://api.semanticscholar.org/CorpusID:14113767)
28. Kim, S., Shin, J., Cho, Y., Jang, J., Longpre, S., Lee, H., Yun, S., Shin, S., Kim, S., Thorne, J., et al. [Prometheus: Inducing fine-grained evaluation capability in language models](https://md.research.elamur.ai/free/en/papers/kim-2023-prometheus-inducing-fine-grained-evaluation/). arXiv preprint arXiv:2310.08491, 2023.
29. Liu, F., Guan, T., Li, Z., Chen, L., Yacoob, Y., Manocha, D., and Zhou, T. Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models. arXiv preprint arXiv:2310.[14566, 2023a](https://md.research.elamur.ai/free/en/papers/guan-2023-hallusionbench-advanced-diagnostic-suite-entangled/).
30. Kocmi, T. and Federmann, C. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520, 2023. [find](https://scholar.google.com/scholar?q=Kocmi%2C+T%2E+and+Federmann%2C+C%2E+Large+language+models+are+state%2Dof%2Dthe%2Dart+evaluators+of+translation+quality%2E+arXiv+preprint+arXiv%3A2302%2E14520%2C+2023%2E)
31. Lee, S., Kim, S., Park, S. H., Kim, G., and Seo, M. Prometheus-vision: Vision-language model as a judge for fine-grained evaluation. arXiv preprint arXiv:2401.06591, 2024. [find](https://scholar.google.com/scholar?q=Lee%2C+S%2E%2C+Kim%2C+S%2E%2C+Park%2C+S%2E+H%2E%2C+Kim%2C+G%2E%2C+and+Seo%2C+M%2E+Prometheus%2Dvision%3A+Vision%2Dlanguage+model+as+a+judge+for+fine%2Dgrained+evaluation%2E+arXiv+preprint+arXiv%3A2401%2E06591%2C+2024%2E)
32. Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., and Wang, L. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.[14565, 2023b](https://md.research.elamur.ai/free/en/papers/liu-2023-mitigating-hallucination-large-multi-modal/).
33. Liu, H., Li, C., Li, Y., and Lee, Y. J. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/), 2023c.
34. Lee Rodgers, J. and Nicewander, W. A. Thirteen ways to look at the correlation coefficient. The American Statistician, 42(1):59–66, 1988. [find](https://scholar.google.com/scholar?q=Lee+Rodgers%2C+J%2E+and+Nicewander%2C+W%2E+A%2E+Thirteen+ways+to+look+at+the+correlation+coefficient%2E+The+American+Statistician%2C+42%281%29%3A59%E2%80%9366%2C+1988%2E)
35. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning, 2023d. [find](https://scholar.google.com/scholar?q=Liu%2C+H%2E%2C+Li%2C+C%2E%2C+Wu%2C+Q%2E%2C+and+Lee%2C+Y%2E+J%2E+Visual+instruction+tuning%2C+2023d%2E)
36. Levenshtein, V. I. et al. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pp. 707–710. Soviet Union, 1966. [find](https://scholar.google.com/scholar?q=Levenshtein%2C+V%2E+I%2E+et+al%2E+Binary+codes+capable+of+correcting+deletions%2C+insertions%2C+and+reversals%2E+In+Soviet+physics+doklady%2C+volume+10%2C+pp%2E+707%E2%80%93710%2E+Soviet+Union%2C+1966%2E)
37. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). arXiv preprint arXiv:2307.03172, 2023e.
38. Li, J., Sun, S., Yuan, W., Fan, R.-Z., Zhao, H., and Liu, P. [Generative judge for evaluating alignment](https://md.research.elamur.ai/free/en/papers/li-2023-generative-judge-evaluating-alignment/). arXiv preprint arXiv:2310.05470, 2023a.
39. Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022. [find](https://scholar.google.com/scholar?q=Lu%2C+P%2E%2C+Mishra%2C+S%2E%2C+Xia%2C+T%2E%2C+Qiu%2C+L%2E%2C+Chang%2C+K%2E%2DW%2E%2C+Zhu%2C+S%2E%2DC%2E%2C+Tafjord%2C+O%2E%2C+Clark%2C+P%2E%2C+and+Kalyan%2C+A%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+science+question+answering%2E+Advance)
40. Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Luo, P., et al. [Mvbench: A comprehensive multi-modal video understanding benchmark](https://md.research.elamur.ai/free/en/papers/li-2023-mvbench-comprehensive-multi-modal-video/). arXiv preprint arXiv:2311.17005, 2023b.
41. Lu, P., Bansal, H., Xia, T., Liu, J., yue Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating math reasoning in visual contexts with gpt-4v, bard, and other large multimodal models. ArXiv, abs/2310.[02255, 2023](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). URL [https://api.semanticscholar](https://api.semanticscholar). org/CorpusID:… [↗](https://api.semanticscholar)
42. Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950, 2023.
43. Masry, A., Long, D., Tan, J. Q., Joty, S., and Hoque, E. [ChartQA: A benchmark for question answering about charts with visual and logical reasoning](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/). In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263–2279, Dublin, Ireland, May 2022. Association for Computational Linguis…
44. Sharma, P., Ding, N., Goodman, S., and Soricut, R. [Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/). In Annual Meeting of the Association for Computational Linguistics, 2018. URL [https://api.semanticscholar.org/CorpusID:51876975](https://api.semanticscholar.org/CorpusID:51876975). [↗](https://api.semanticscholar.org/CorpusID:51876975)
45. Mathew, M., Bagal, V., Tito, R. P., Karatzas, D., Valveny, E., and Jawahar, C. Infographicvqa. 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2582–2591, 2021. URL [https://api.semanticscholar.org/CorpusID:233394125](https://api.semanticscholar.org/CorpusID:233394125). [↗](https://api.semanticscholar.org/CorpusID:233394125)
46. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). 2023.
47. Saito, K., Wachi, A., Wataoka, K., and Akimoto, Y. Verbosity bias in preference labeling by large language models. arXiv preprint arXiv:2310.10076, 2023. — [arXiv:2310.10076](https://arxiv.org/abs/2310.10076) [find](https://scholar.google.com/scholar?q=Saito%2C+K%2E%2C+Wachi%2C+A%2E%2C+Wataoka%2C+K%2E%2C+and+Akimoto%2C+Y%2E+Verbosity+bias+in+preference+labeling+by+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E10076%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E10076%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F231)
48. Shi, Y., Peng, D., Liao, W., Lin, Z., Chen, X., Liu, C., Zhang, Y., and Jin, L. Exploring ocr capabilities of gpt-4v (ision): A quantitative and in-depth evaluation. arXiv preprint arXiv:2310.16809, 2023. — [arXiv:2310.16809](https://arxiv.org/abs/2310.16809) [find](https://scholar.google.com/scholar?q=Shi%2C+Y%2E%2C+Peng%2C+D%2E%2C+Liao%2C+W%2E%2C+Lin%2C+Z%2E%2C+Chen%2C+X%2E%2C+Liu%2C+C%2E%2C+Zhang%2C+Y%2E%2C+and+Jin%2C+L%2E+Exploring+ocr+capabilities+of+gpt%2D4v+%28ision%29%3A+A+quantitative+and+in%2Ddepth+evaluation%2E+arXiv+preprint+arXiv%3A2310%2E16809%2C+2)
49. Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8309–8318, 2019. URL [https://api.semanticscholar.org/CorpusID:85553602](https://api.semanticscholar.org/CorpusID:85553602). [↗](https://api.semanticscholar.org/CorpusID:85553602)
50. Srinivasan, K., Raman, K., Chen, J., Bendersky, M., and Najork, M. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021. URL [https://api.semanticscho…](https://api.semanticscho…) [↗](https://api.semanticscho…)
51. OpenAI. Openai models - gpt-4-vision. [https://openai.com/research/](https://openai.com/research/) gpt-4v-system-card, 2023. [↗](https://openai.com/research/)
52. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
53. Sun, L., Huang, Y., Wang, H., Wu, S., Zhang, Q., Gao, C., Huang, Y., Lyu, W., Zhang, Y., Li, X., et al. [Trustllm: Trustworthiness in large language models](https://md.research.elamur.ai/free/en/papers/huang-2024-trustllm-trustworthiness-large-language-models/). arXiv preprint arXiv:2401.05561, 2024.
54. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311–318, 2002.
55. Sun, W., Nasraoui, O., and Shafto, P. Evolution and impact of bias in human and machine learning algorithm interaction. Plos one, 15(8):e0235502, 2020. [find](https://scholar.google.com/scholar?q=Sun%2C+W%2E%2C+Nasraoui%2C+O%2E%2C+and+Shafto%2C+P%2E+Evolution+and+impact+of+bias+in+human+and+machine+learning+algorithm+interaction%2E+Plos+one%2C+15%288%29%3Ae0235502%2C+2020%2E)
56. Prendki, J. Are you spending too much money labeling data?, 2023. [find](https://scholar.google.com/scholar?q=Prendki%2C+J%2E+Are+you+spending+too+much+money+labeling+data%3F%2C+2023%2E)
57. Sun, Z., Shen, S., Cao, S., Liu, H., Li, C., Shen, Y., Gan, C., Gui, L.-Y., Wang, Y.-X., Yang, Y., et al. [Aligning large multimodal models with factually augmented rlhf](https://md.research.elamur.ai/free/en/papers/sun-2023-aligning-large-multimodal-models-factually/). arXiv preprint arXiv:2309.14525, 2023.
58. Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). Advances in Neural Information Processing Systems, 36, 2024.
59. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
60. Vedantam, R., Lawrence Zitnick, C., and Parikh, D. [Cider: Consensus-based image description evaluation](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566–4575, 2015.
61. Raghubir, P. and Valenzuela, A. Center-of-inattention: Position biases in decision-making. Organizational Behavior and Human Decision Processes, 99(1):66–80, 2006. [find](https://scholar.google.com/scholar?q=Raghubir%2C+P%2E+and+Valenzuela%2C+A%2E+Center%2Dof%2Dinattention%3A+Position+biases+in+decision%2Dmaking%2E+Organizational+Behavior+and+Human+Decision+Processes%2C+99%281%29%3A66%E2%80%9380%2C+2006%2E)
62. Wang, J., Zhou, Y., Xu, G., Shi, P., Zhao, C., Xu, H., Ye, Q., Yan, M., Zhang, J., Zhu, J., et al. Evaluation and analysis of hallucination in large vision-language models. arXiv preprint arXiv:2308.15126, 2023a. — [arXiv:2308.15126](https://arxiv.org/abs/2308.15126) [find](https://scholar.google.com/scholar?q=Wang%2C+J%2E%2C+Zhou%2C+Y%2E%2C+Xu%2C+G%2E%2C+Shi%2C+P%2E%2C+Zhao%2C+C%2E%2C+Xu%2C+H%2E%2C+Ye%2C+Q%2E%2C+Yan%2C+M%2E%2C+Zhang%2C+J%2E%2C+Zhu%2C+J%2E%2C+et+al%2E+Evaluation+and+analysis+of+hallucination+in+large+vision%2Dlanguage+models%2E+arXiv+preprint+arXiv%3A2308%2E1)
63. Yang, Z., Li, L., Lin, K., Wang, J., Lin, C.-C., Liu, Z., and Wang, L. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.[17421, 9 (1):1, 2023](https://md.research.elamur.ai/free/en/papers/yang-2023-dawn-lmms-preliminary-explorations-gpt/).
64. Wang, P., Li, L., Chen, L., Zhu, D., Lin, B., Cao, Y., Liu, Q., Liu, T., and Sui, Z. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023b. [find](https://scholar.google.com/scholar?q=Wang%2C+P%2E%2C+Li%2C+L%2E%2C+Chen%2C+L%2E%2C+Zhu%2C+D%2E%2C+Lin%2C+B%2E%2C+Cao%2C+Y%2E%2C+Liu%2C+Q%2E%2C+Liu%2C+T%2E%2C+and+Sui%2C+Z%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv+preprint+arXiv%3A2305%2E17926%2C+2023b%2E)
65. Yin, S., Fu, C., Zhao, S., Xu, T., Wang, H., Sui, D., Shen, Y., Li, K., Sun, X., and Chen, E. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint arXiv:2310.16045, 2023. [find](https://scholar.google.com/scholar?q=Yin%2C+S%2E%2C+Fu%2C+C%2E%2C+Zhao%2C+S%2E%2C+Xu%2C+T%2E%2C+Wang%2C+H%2E%2C+Sui%2C+D%2E%2C+Shen%2C+Y%2E%2C+Li%2C+K%2E%2C+Sun%2C+X%2E%2C+and+Chen%2C+E%2E+Woodpecker%3A+Hallucination+correction+for+multimodal+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E16045)
66. Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., Song, X., Xu, J., Xu, B., Li, J., Dong, Y., Ding, M., and Tang, J. [Cogvlm: Visual expert for pretrained language models](https://md.research.elamur.ai/free/en/papers/wang-2023-cogvlm-visual-expert-pretrained/), 2023c.
67. Yu, T., Yao, Y., Zhang, H., He, T., Han, Y., Cui, G., Hu, J., Liu, Z., Zheng, H.-T., Sun, M., et al. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. arXiv preprint arXiv:2312.00849, 2023a. [find](https://scholar.google.com/scholar?q=Yu%2C+T%2E%2C+Yao%2C+Y%2E%2C+Zhang%2C+H%2E%2C+He%2C+T%2E%2C+Han%2C+Y%2E%2C+Cui%2C+G%2E%2C+Hu%2C+J%2E%2C+Liu%2C+Z%2E%2C+Zheng%2C+H%2E%2DT%2E%2C+Sun%2C+M%2E%2C+et+al%2E+Rlhf%2Dv%3A+Towards+trustworthy+mllms+via+behavior+alignment+from+fine%2Dgrained+correctional+human+feedb)
68. Wang, X., Golbandi, N., Bendersky, M., Metzler, D., and Najork, M. Position bias estimation for unbiased learning to rank in personal search. In Proceedings of the eleventh ACM international conference on web search and data mining, pp. 610–618, 2018. [find](https://scholar.google.com/scholar?q=Wang%2C+X%2E%2C+Golbandi%2C+N%2E%2C+Bendersky%2C+M%2E%2C+Metzler%2C+D%2E%2C+and+Najork%2C+M%2E+Position+bias+estimation+for+unbiased+learning+to+rank+in+personal+search%2E+In+Proceedings+of+the+eleventh+ACM+international+conferenc)
69. Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., and Wang, L. [Mm-vet: Evaluating large multimodal models for integrated capabilities](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/). ArXiv, abs/2308.02490, 2023b. URL [https://api.semanticscholar.org/CorpusID:260611572](https://api.semanticscholar.org/CorpusID:260611572). [↗](https://api.semanticscholar.org/CorpusID:260611572)
70. Wang, X., Ma, B., Hu, C., Weber-Genzel, L., Röttger, P., Kreuter, F., Hovy, D., and Plank, B. ” my answer is c”: First-token probabilities do not match text answers in instruction-tuned language models. arXiv preprint arXiv:2402.[14499, 2024a](https://md.research.elamur.ai/free/en/papers/wang-2024-my-answer-is-c-first/).
71. Zhang, R., Gui, L., Sun, Z., Feng, Y., Xu, K., Zhang, Y., Fu, D., Li, C., Hauptmann, A., Bisk, Y., et al. [Direct preference optimization of video large multimodal models from language model reward](https://md.research.elamur.ai/free/en/papers/zhang-2025-direct-preference-optimization-video-large/). arXiv preprint arXiv:2404.01258, 2024.
72. Wang, X., Zhou, Y., Liu, X., Lu, H., Xu, Y., He, F., Yoon, J., Lu, T., Bertasius, G., Bansal, M., et al. Mementos: A comprehensive benchmark for multimodal large language model reasoning over image sequences. arXiv preprint arXiv:2401.10529, 2024b. [find](https://scholar.google.com/scholar?q=Wang%2C+X%2E%2C+Zhou%2C+Y%2E%2C+Liu%2C+X%2E%2C+Lu%2C+H%2E%2C+Xu%2C+Y%2E%2C+He%2C+F%2E%2C+Yoon%2C+J%2E%2C+Lu%2C+T%2E%2C+Bertasius%2C+G%2E%2C+Bansal%2C+M%2E%2C+et+al%2E+Mementos%3A+A+comprehensive+benchmark+for+multimodal+large+language+model+reasoning+over+image+se)
73. Zheng, C., Zhou, H., Meng, F., Zhou, J., and Huang, M. On large language models’ selection bias in multi-choice questions. arXiv preprint arXiv:2309.03882, 2023a. [find](https://scholar.google.com/scholar?q=Zheng%2C+C%2E%2C+Zhou%2C+H%2E%2C+Meng%2C+F%2E%2C+Zhou%2C+J%2E%2C+and+Huang%2C+M%2E+On+large+language+models%E2%80%99+selection+bias+in+multi%2Dchoice+questions%2E+arXiv+preprint+arXiv%3A2309%2E03882%2C+2023a%2E)
74. Wang, Z. J., Montoya, E., Munechika, D., Yang, H., Hoover, B., and Chau, D. H. Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models. ArXiv, abs/2210.14896, 2022. URL [https://api.semanticscholar.org/CorpusID:253116574](https://api.semanticscholar.org/CorpusID:253116574). [↗](https://api.semanticscholar.org/CorpusID:253116574)
75. Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). arXiv preprint arXiv:2109.01652, 2021.
76. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). arXiv preprint arXiv:2306.05685, 2023b.
77. Zhou, Y., Cui, C., Yoon, J., Zhang, L., Deng, Z., Finn, C., Bansal, M., and Yao, H. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00754, 2023. [find](https://scholar.google.com/scholar?q=Zhou%2C+Y%2E%2C+Cui%2C+C%2E%2C+Yoon%2C+J%2E%2C+Zhang%2C+L%2E%2C+Deng%2C+Z%2E%2C+Finn%2C+C%2E%2C+Bansal%2C+M%2E%2C+and+Yao%2C+H%2E+Analyzing+and+mitigating+object+hallucination+in+large+vision%2Dlanguage+models%2E+arXiv+preprint+arXiv%3A2310%2E00754%2C+2023)
78. Zhu, L., Wang, X., and Wang, X. [Judgelm: Fine-tuned large language models are scalable judges](https://md.research.elamur.ai/free/en/papers/zhu-2023-judgelm-fine-tuned-large-language/). arXiv preprint arXiv:2310.17631, 2023.
79. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35: 24824–24837, 2022.
80. Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T.-S. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519, 2023a. [find](https://scholar.google.com/scholar?q=Wu%2C+S%2E%2C+Fei%2C+H%2E%2C+Qu%2C+L%2E%2C+Ji%2C+W%2E%2C+and+Chua%2C+T%2E%2DS%2E+Next%2Dgpt%3A+Any%2Dto%2Dany+multimodal+llm%2E+arXiv+preprint+arXiv%3A2309%2E05519%2C+2023a%2E)
81. Wu, Y., Wang, S., Yang, H., Zheng, T., Zhang, H., Zhao, Y., and Qin, B. An early evaluation of gpt-4v (ision). arXiv preprint arXiv:2310.16534, 2023b. — [arXiv:2310.16534](https://arxiv.org/abs/2310.16534) [find](https://scholar.google.com/scholar?q=Wu%2C+Y%2E%2C+Wang%2C+S%2E%2C+Yang%2C+H%2E%2C+Zheng%2C+T%2E%2C+Zhang%2C+H%2E%2C+Zhao%2C+Y%2E%2C+and+Qin%2C+B%2E+An+early+evaluation+of+gpt%2D4v+%28ision%29%2E+arXiv+preprint+arXiv%3A2310%2E16534%2C+2023b%2E+%E2%80%94+%5BarXiv%3A2310%2E16534%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E1)
82. A. Comprehensive Related Works A.1. Large Model as Judge [find](https://scholar.google.com/scholar?q=A%2E+Comprehensive+Related+Works+A%2E1%2E+Large+Model+as+Judge)

*Updated 31 October 2026*