---
title: "Visual CoT — paper by Shao et al. (2024) — summary and ideas"
description: "Multi-modal large language models (MLLMs) often fail when the answer depends on a small detail in… 96 references, cited by 60 papers here."
authors: ["Hao Shao", "Shengju Qian", "Han Xiao", "Guanglu Song", "Zhuofan Zong", "Letian Wang", "Yu Liu", "Hongsheng Li"]
year: 2024
venue: "arXiv:2403.16999"
source: "https://arxiv.org/abs/2403.16999"
doi: "10.48550/arxiv.2403.16999"
arxiv: "2403.16999"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shao-2024-visual-cot-advancing-multi-modal/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shao-2024-visual-cot-advancing-multi-modal/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning

# Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning

**Authors:** Hao Shao, [Shengju Qian](https://md.research.elamur.ai/free/en/authors/shengju-qian/), [Han Xiao](https://md.research.elamur.ai/free/en/authors/han-xiao/), Guanglu Song, Zhuofan Zong, Letian Wang, [Yu Liu](https://md.research.elamur.ai/free/en/authors/yu-liu/) and [Hongsheng Li](https://md.research.elamur.ai/free/en/authors/hongsheng-li/) · **Year:** 2024 · **Venue:** [arXiv:2403.16999](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2024](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2024/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2403.16999) · [PDF](https://arxiv.org/pdf/2403.16999) · [DOI 10.48550/arxiv.2403.16999](https://doi.org/10.48550/arxiv.2403.16999) · [arXiv 2403.16999](https://arxiv.org/abs/2403.16999) · [OpenAlex](https://openalex.org/W4415799075)

**Citations:** 57 citations (OpenAlex, 2026-10-14) · [60](https://md.research.elamur.ai/free/en/papers/shao-2024-visual-cot-advancing-multi-modal/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Multi-modal large language models (MLLMs) often fail when the answer depends on a small detail in a large or high-resolution image, and their single-pass, black-box answering offers little insight into how a response was produced. The authors address both problems by making the model first locate the relevant evidence and then answer. They built the Visual CoT dataset, which contains 438k visual question-answer pairs. Each pair has a bounding box marking the image region needed to answer, and about 98k pairs also include step-by-step reasoning. They also train a multi-turn model, VisCoT, that predicts this key region, crops and zooms into it, and combines the full image with the enlarged region to produce the answer. A matching benchmark tests how well models handle questions that require local focus. Experiments show clear gains over answering without the CoT step, especially on text-heavy and document images. The predicted box also exposes an intermediate reasoning stage that people can inspect.

## Key points
- **Dataset scale and coverage:** The dataset's five domains cover text and document images, charts, general visual question answering, fine-grained visual understanding, and reasoning about relations. It draws on sources such as TextVQA, TextCaps, DocVQA, DUDE, SROIE, Birds-200-2011, Flickr30k, Visual7W, InfographicsVQA, VSR, GQA, and Open Images. Some annotations, including those for GQA, TextCaps, and Flickr30k, were produced with GPT-4.
- **Receipt extraction:** On SROIE, the visual CoT pipeline reportedly scores about 8× higher than the standard pipeline without the CoT step.
- **Token efficiency:** VisCoT-7B with CoT at 224 resolution outperforms the standard pipeline at 448 resolution while using only half as many visual tokens.
- **Box-selection ablations:**
  - Ground-truth boxes give the upper bound on performance.
  - Random boxes perform about the same as using no CoT at all.
  - A fixed center crop beats random boxes.
  - Together, these results show that accurate localization drives the gains.
- **Release and venue:** The dataset, benchmark, and pre-trained models are publicly released. The paper appeared at NeurIPS 2024 in the Datasets and Benchmarks Track.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (4): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works

## Cite

Cite as: Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu and Hongsheng Li. 2024. Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning. arXiv:2403.16999.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{shao2024visual,
  title = {{Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning}},
  author = {Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu and Hongsheng Li},
  year = {2024},
  doi = {10.48550/arxiv.2403.16999},
  eprint = {2403.16999},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.16999}
}
```

- [← previous QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks](https://md.research.elamur.ai/free/en/papers/tseng-2024-quip-even-better-llm-quantization/)
- [next → DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search](https://md.research.elamur.ai/free/en/papers/xin-2024-deepseek-prover-v1-5-harnessing/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

96 references, 48 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
2. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. [Flamingo: a visual language model for few-shot learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/). Advances in Neural Information Processing Systems, 35:23716–23736, 2022.
3. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.[12966, 2023](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/).
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
5. Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. [Minigpt-v2: large language model as a unified interface for vision-language multi-task learning](https://md.research.elamur.ai/free/en/papers/chen-2023-minigpt-v2-large-language-model/). arXiv preprint arXiv:2310.09478, 2023.
6. Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.[15195, 2023](https://md.research.elamur.ai/free/en/papers/chen-2023-shikra-unleashing-multimodal-llm-s/).
7. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. [Sharegpt4v: Improving large multi-modal models with better captions](https://md.research.elamur.ai/free/en/papers/chen-2023-sharegpt4v-improving-large-multi-modal/). arXiv preprint arXiv:2311.12793, 2023.
8. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. [Are we on the right way for evaluating large vision-language models?](https://md.research.elamur.ai/free/en/papers/chen-2024-right-way-evaluating-large-vision/) arXiv preprint arXiv:2403.20330, 2024.
9. Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprint arXiv:2406.04325, 2024. — [arXiv:2406.04325](https://arxiv.org/abs/2406.04325) [find](https://scholar.google.com/scholar?q=Lin+Chen%2C+Xilin+Wei%2C+Jinsong+Li%2C+Xiaoyi+Dong%2C+Pan+Zhang%2C+Yuhang+Zang%2C+Zehui+Chen%2C+Haodong+Duan%2C+Bin+Lin%2C+Zhenyu+Tang%2C+et+al%2E+Sharegpt4video%3A+Improving+video+understanding+and+generation+with+better+ca)
10. Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. [Pali: A jointly-scaled multilingual language-image model](https://md.research.elamur.ai/free/en/papers/chen-2022-pali-jointly-scaled-multilingual-language/). arXiv preprint arXiv:2209.06794, 2022.
11. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. [Longlora: Efficient fine-tuning of long-context large language models](https://md.research.elamur.ai/free/en/papers/chen-2023-longlora-efficient-fine-tuning-long/). arXiv preprint arXiv:2309.12307, 2023.
12. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. [Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks](https://md.research.elamur.ai/free/en/papers/chen-2023-internvl-scaling-up-vision/). arXiv preprint arXiv:2312.14238, 20…
13. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See [https://vicuna.lmsys](https://vicuna.lmsys). org (accessed 14 April 2023), 2023. [↗](https://vicuna.lmsys)
14. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. [Instructblip: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/), 2023.
15. Yuning Du, Chenxia Li, Ruoyu Guo, Xiaoting Yin, Weiwei Liu, Jun Zhou, Yifan Bai, Zilin Yu, Yehua Yang, Qingqing Dang, et al. Pp-ocr: A practical ultra lightweight ocr system. arXiv preprint arXiv:2009.09941, 2020. — [arXiv:2009.09941](https://arxiv.org/abs/2009.09941) [find](https://scholar.google.com/scholar?q=Yuning+Du%2C+Chenxia+Li%2C+Ruoyu+Guo%2C+Xiaoting+Yin%2C+Weiwei+Liu%2C+Jun+Zhou%2C+Yifan+Bai%2C+Zilin+Yu%2C+Yehua+Yang%2C+Qingqing+Dang%2C+et+al%2E+Pp%2Docr%3A+A+practical+ultra+lightweight+ocr+system%2E+arXiv+preprint+arXiv%3A2009)
16. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. [Mme: A comprehensive evaluation benchmark for multimodal large language models](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/). arXiv preprint arXiv:2306.13394, 2023.
17. Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, et al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. arXiv preprint arXiv:2402.05935, 2024. — [arXiv:2402.05935](https://arxiv.org/abs/2402.05935) [find](https://scholar.google.com/scholar?q=Peng+Gao%2C+Renrui+Zhang%2C+Chris+Liu%2C+Longtian+Qiu%2C+Siyuan+Huang%2C+Weifeng+Lin%2C+Shitian+Zhao%2C+Shijie+Geng%2C+Ziyi+Lin%2C+Peng+Jin%2C+et+al%2E+Sphinx%2Dx%3A+Scaling+data+and+parameters+for+a+family+of+multi%2Dmodal+larg)
18. Anisha Gunjal, Jihan Yin, and Erhan Bas. [Detecting and preventing hallucinations in large vision language models](https://md.research.elamur.ai/free/en/papers/gunjal-2023-detecting-preventing-hallucinations-large-vision/). arXiv preprint arXiv:2308.06394, 2023.
19. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. [Cogagent: A visual language model for gui agents](https://md.research.elamur.ai/free/en/papers/hong-2023-cogagent-visual-language-model/). arXiv preprint arXiv:2312.08914, 2023.
20. Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and CV Jawahar. Icdar2019 competition on scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1516–1520. IEEE, 2019. [find](https://scholar.google.com/scholar?q=Zheng+Huang%2C+Kai+Chen%2C+Jianhua+He%2C+Xiang+Bai%2C+Dimosthenis+Karatzas%2C+Shijian+Lu%2C+and+CV+Jawahar%2E+Icdar2019+competition+on+scanned+receipt+ocr+and+information+extraction%2E+In+2019+International+Conferenc)
21. Drew A Hudson and Christopher D Manning. [Gqa: A new dataset for real-world visual reasoning and compositional question answering](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/). In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019.
22. Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu, Jiayi Lei, Pengshuo Qiu, Pan Lu, Zehui Chen, Guanglu Song, Peng Gao, et al. Mmsearch: Benchmarking the potential of large models as multi-modal search engines. arXiv preprint arXiv:2409.12959, 2024. — [arXiv:2409.12959](https://arxiv.org/abs/2409.12959) [find](https://scholar.google.com/scholar?q=Dongzhi+Jiang%2C+Renrui+Zhang%2C+Ziyu+Guo%2C+Yanmin+Wu%2C+Jiayi+Lei%2C+Pengshuo+Qiu%2C+Pan+Lu%2C+Zehui+Chen%2C+Guanglu+Song%2C+Peng+Gao%2C+et+al%2E+Mmsearch%3A+Benchmarking+the+potential+of+large+models+as+multi%2Dmodal+search)
23. Yang Jiao, Shaoxiang Chen, Zequn Jie, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. Lumen: Unleashing versatile vision-centric capabilities of large multimodal models. arXiv preprint arXiv:2403.07304, 2024. — [arXiv:2403.07304](https://arxiv.org/abs/2403.07304) [find](https://scholar.google.com/scholar?q=Yang+Jiao%2C+Shaoxiang+Chen%2C+Zequn+Jie%2C+Jingjing+Chen%2C+Lin+Ma%2C+and+Yu%2DGang+Jiang%2E+Lumen%3A+Unleashing+versatile+vision%2Dcentric+capabilities+of+large+multimodal+models%2E+arXiv+preprint+arXiv%3A2403%2E07304%2C+202)
24. Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. [Referitgame: Referring to objects in photographs of natural scenes](https://md.research.elamur.ai/free/en/papers/kazemzadeh-2014-referitgame-referring-objects-photographs-natural/). In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014.
25. Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Generating images with multimodal language models. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Jing+Yu+Koh%2C+Daniel+Fried%2C+and+Russ+R+Salakhutdinov%2E+Generating+images+with+multimodal+language+models%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2C+2024%2E)
26. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). Advances in neural information processing systems, 35:22199–22213, 2022.
27. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. [Visual genome: Connecting language and vision using crowdsourced dense image annotations](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/). International journal of computer vision, 123:32–73, 2017…
28. Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International… [find](https://scholar.google.com/scholar?q=Alina+Kuznetsova%2C+Hassan+Rom%2C+Neil+Alldrin%2C+Jasper+Uijlings%2C+Ivan+Krasin%2C+Jordi+Pont%2DTuset%2C+Shahab+Kamali%2C+Stefan+Popov%2C+Matteo+Malloci%2C+Alexander+Kolesnikov%2C+et+al%2E+The+open+images+dataset+v4%3A+Unifie)
29. Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308.00692, 2023. [find](https://scholar.google.com/scholar?q=Xin+Lai%2C+Zhuotao+Tian%2C+Yukang+Chen%2C+Yanwei+Li%2C+Yuhui+Yuan%2C+Shu+Liu%2C+and+Jiaya+Jia%2E+Lisa%3A+Reasoning+segmentation+via+large+language+model%2E+arXiv+preprint+arXiv%3A2308%2E00692%2C+2023%2E)
30. Hugo Laurençon, Daniel van Strien, Stas Bekman, Leo Tronchon, Lucile Saulnier, Thomas Wang, Siddharth Karamcheti, Amanpreet Singh, Giada Pistilli, Yacine Jernite, et al. Introducing idefics: An open reproduction of state-of-the-art visual language model, 2023. URL https://huggingface. co/blog/idefic… [find](https://scholar.google.com/scholar?q=Hugo+Lauren%C3%A7on%2C+Daniel+van+Strien%2C+Stas+Bekman%2C+Leo+Tronchon%2C+Lucile+Saulnier%2C+Thomas+Wang%2C+Siddharth+Karamcheti%2C+Amanpreet+Singh%2C+Giada+Pistilli%2C+Yacine+Jernite%2C+et+al%2E+Introducing+idefics%3A+An+open+r)
31. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. [Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). arXiv preprint arXiv:2301.12597, 2023.
32. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. [Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/). In International Conference on Machine Learning, pages 12888–12900. PMLR, 2022.
33. KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. [Videochat: Chat-centric video understanding](https://md.research.elamur.ai/free/en/papers/li-2023-videochat-chat-centric-video-understanding/). arXiv preprint arXiv:2305.06355, 2023.
34. Yanwei Li, Chengyao Wang, and Jiaya Jia. [Llama-vid: An image is worth 2 tokens in large language models](https://md.research.elamur.ai/free/en/papers/li-2023-llama-vid-image-worth-2/). arXiv preprint arXiv:2311.17043, 2023.
35. Yian Li, Wentao Tian, Yang Jiao, Jingjing Chen, and Yu-Gang Jiang. Eyes can deceive: Benchmarking counterfactual reasoning abilities of multi-modal large language models. arXiv preprint arXiv:2404.12966, 2024. — [arXiv:2404.12966](https://arxiv.org/abs/2404.12966) [find](https://scholar.google.com/scholar?q=Yian+Li%2C+Wentao+Tian%2C+Yang+Jiao%2C+Jingjing+Chen%2C+and+Yu%2DGang+Jiang%2E+Eyes+can+deceive%3A+Benchmarking+counterfactual+reasoning+abilities+of+multi%2Dmodal+large+language+models%2E+arXiv+preprint+arXiv%3A2404%2E129)
36. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. [Evaluating object hallucination in large vision-language models](https://md.research.elamur.ai/free/en/papers/li-2023-evaluating-object-hallucination-large-vision/). arXiv preprint arXiv:2305.10355, 2023.
37. Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575, 2023. — [arXiv:2311.07575](https://arxiv.org/abs/2311.07575) [find](https://scholar.google.com/scholar?q=Ziyi+Lin%2C+Chris+Liu%2C+Renrui+Zhang%2C+Peng+Gao%2C+Longtian+Qiu%2C+Han+Xiao%2C+Han+Qiu%2C+Chen+Lin%2C+Wenqi+Shao%2C+Keqin+Chen%2C+et+al%2E+Sphinx%3A+The+joint+mixing+of+weights%2C+tasks%2C+and+visual+embeddings+for+multi%2Dmodal)
38. Fangyu Liu, Guy Edward Toh Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 2023. [find](https://scholar.google.com/scholar?q=Fangyu+Liu%2C+Guy+Edward+Toh+Emerson%2C+and+Nigel+Collier%2E+Visual+spatial+reasoning%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+2023%2E)
39. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/), 2023.
40. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). In NeurIPS, 2023.
41. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. [Grounding dino: Marrying dino with grounded pre-training for open-set object detection](https://md.research.elamur.ai/free/en/papers/liu-2023-grounding-dino-marrying-dino-grounded/). arXiv preprint arXiv:2303.05499, 2023.
42. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. [Mmbench: Is your multi-modal model an all-around player?](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/) arXiv preprint arXiv:2307.06281, 2023.
43. Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022. [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Swaroop+Mishra%2C+Tony+Xia%2C+Liang+Qiu%2C+Kai%2DWei+Chang%2C+Song%2DChun+Zhu%2C+Oyvind+Tafjord%2C+Peter+Clark%2C+and+Ashwin+Kalyan%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+science+questio)
44. Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient vision-language instruction tuning for large language models. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Gen+Luo%2C+Yiyi+Zhou%2C+Tianhe+Ren%2C+Shengxin+Chen%2C+Xiaoshuai+Sun%2C+and+Rongrong+Ji%2E+Cheap+and+quick%3A+Efficient+vision%2Dlanguage+instruction+tuning+for+large+language+models%2E+Advances+in+Neural+Information+P)
45. Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. Multi-task collaborative network for joint referring expression comprehension and segmentation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 10034–10043, 2020. [find](https://scholar.google.com/scholar?q=Gen+Luo%2C+Yiyi+Zhou%2C+Xiaoshuai+Sun%2C+Liujuan+Cao%2C+Chenglin+Wu%2C+Cheng+Deng%2C+and+Rongrong+Ji%2E+Multi%2Dtask+collaborative+network+for+joint+referring+expression+comprehension+and+segmentation%2E+In+Proceedings)
46. Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07207, 2023. — [arXiv:2306.07207](https://arxiv.org/abs/2306.07207) [find](https://scholar.google.com/scholar?q=Ruipu+Luo%2C+Ziwang+Zhao%2C+Min+Yang%2C+Junwei+Dong%2C+Minghui+Qiu%2C+Pengcheng+Lu%2C+Tao+Wang%2C+and+Zhongyu+Wei%2E+Valley%3A+Video+assistant+with+large+language+model+enhanced+ability%2E+arXiv+preprint+arXiv%3A2306%2E07207)
47. Bingqi Ma, Zhuofan Zong, Guanglu Song, Hongsheng Li, and Yu Liu. Exploring the role of large language models in prompt encoding for diffusion models. arXiv preprint arXiv:2406.11831, 2024. — [arXiv:2406.11831](https://arxiv.org/abs/2406.11831) [find](https://scholar.google.com/scholar?q=Bingqi+Ma%2C+Zhuofan+Zong%2C+Guanglu+Song%2C+Hongsheng+Li%2C+and+Yu+Liu%2E+Exploring+the+role+of+large+language+models+in+prompt+encoding+for+diffusion+models%2E+arXiv+preprint+arXiv%3A2406%2E11831%2C+2024%2E+%E2%80%94+%5BarXiv%3A24)
48. Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. [Generation and comprehension of unambiguous object descriptions](https://md.research.elamur.ai/free/en/papers/mao-2015-generation-comprehension-unambiguous-object-descriptions/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016.
49. Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697–1706, 2022. [find](https://scholar.google.com/scholar?q=Minesh+Mathew%2C+Viraj+Bagal%2C+Rub%C3%A8n+Tito%2C+Dimosthenis+Karatzas%2C+Ernest+Valveny%2C+and+CV+Jawahar%2E+Infographicvqa%2E+In+Proceedings+of+the+IEEE%2FCVF+Winter+Conference+on+Applications+of+Computer+Vision%2C+pages)
50. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. [Docvqa: A dataset for vqa on document images](https://md.research.elamur.ai/free/en/papers/mathew-2020-docvqa-dataset-vqa-document-images/). In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021.
51. Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of-thought prompting for large multimodal models. arXiv preprint arXiv:2311.17076, 2023. [find](https://scholar.google.com/scholar?q=Chancharik+Mitra%2C+Brandon+Huang%2C+Trevor+Darrell%2C+and+Roei+Herzig%2E+Compositional+chain%2Dof%2Dthought+prompting+for+large+multimodal+models%2E+arXiv+preprint+arXiv%3A2311%2E17076%2C+2023%2E)
52. OpenAI. Chatgpt. [https://openai.com/blog/chatgpt/](https://openai.com/blog/chatgpt/), 2023. [↗](https://openai.com/blog/chatgpt/)
53. Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. [Kosmos-2: Grounding multimodal large language models to the world](https://md.research.elamur.ai/free/en/papers/peng-2023-kosmos-2-grounding-multimodal-large/). arXiv preprint arXiv:2306.14824, 2023.
54. Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641–2649, 2015… [find](https://scholar.google.com/scholar?q=Bryan+A+Plummer%2C+Liwei+Wang%2C+Chris+M+Cervantes%2C+Juan+C+Caicedo%2C+Julia+Hockenmaier%2C+and+Svetlana+Lazebnik%2E+Flickr30k+entities%3A+Collecting+region%2Dto%2Dphrase+correspondences+for+richer+image%2Dto%2Dsentence+m)
55. Ji Qi, Ming Ding, Weihan Wang, Yushi Bai, Qingsong Lv, Wenyi Hong, Bin Xu, Lei Hou, Juanzi Li, Yuxiao Dong, et al. Cogcom: Train large vision-language models diving into details through chain of manipulations. arXiv preprint arXiv:2402.04236, 2024. — [arXiv:2402.04236](https://arxiv.org/abs/2402.04236) [find](https://scholar.google.com/scholar?q=Ji+Qi%2C+Ming+Ding%2C+Weihan+Wang%2C+Yushi+Bai%2C+Qingsong+Lv%2C+Wenyi+Hong%2C+Bin+Xu%2C+Lei+Hou%2C+Juanzi+Li%2C+Yuxiao+Dong%2C+et+al%2E+Cogcom%3A+Train+large+vision%2Dlanguage+models+diving+into+details+through+chain+of+manip)
56. Shengju Qian, Huiwen Chang, Yuanzhen Li, Zizhao Zhang, Jiaya Jia, and Han Zhang. Strait: Non-autoregressive generation with stratified image transformer. arXiv preprint arXiv:2303.00750, 2023. — [arXiv:2303.00750](https://arxiv.org/abs/2303.00750) [find](https://scholar.google.com/scholar?q=Shengju+Qian%2C+Huiwen+Chang%2C+Yuanzhen+Li%2C+Zizhao+Zhang%2C+Jiaya+Jia%2C+and+Han+Zhang%2E+Strait%3A+Non%2Dautoregressive+generation+with+stratified+image+transformer%2E+arXiv+preprint+arXiv%3A2303%2E00750%2C+2023%2E+%E2%80%94+%5BarXi)
57. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In International conference on machine learning, pages 8748–8763. PMLR, 20…
58. Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15120–15130, 2024. [find](https://scholar.google.com/scholar?q=Hao+Shao%2C+Yuxuan+Hu%2C+Letian+Wang%2C+Guanglu+Song%2C+Steven+L+Waslander%2C+Yu+Liu%2C+and+Hongsheng+Li%2E+Lmdrive%3A+Closed%2Dloop+end%2Dto%2Dend+driving+with+large+language+models%2E+In+Proceedings+of+the+IEEE%2FCVF+Confere)
59. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. [Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face](https://md.research.elamur.ai/free/en/papers/shen-2023-hugginggpt-solving-ai-tasks-chatgpt/). Advances in Neural Information Processing Systems, 36, 2024.
60. Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. [Textcaps: a dataset for image captioning with reading comprehension](https://md.research.elamur.ai/free/en/papers/sidorov-2020-textcaps-dataset-image-captioning-reading/). In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 742–758. Springer, 2020.
61. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. [find](https://scholar.google.com/scholar?q=Amanpreet+Singh%2C+Vivek+Natarajan%2C+Meet+Shah%2C+Yu+Jiang%2C+Xinlei+Chen%2C+Dhruv+Batra%2C+Devi+Parikh%2C+and+Marcus+Rohrbach%2E+Towards+vqa+models+that+can+read%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+comput)
62. Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. [Eva-clip: Improved training techniques for clip at scale](https://md.research.elamur.ai/free/en/papers/sun-2023-eva-clip-improved-training-techniques/). arXiv preprint arXiv:2303.15389, 2023.
63. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
64. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
65. Jordy Van Landeghem, Rubèn Tito, Łukasz Borchmann, Michał Pietruszka, Pawel Joziak, Rafal Powalski, Dawid Jurkiewicz, Mickaël Coustaty, Bertrand Anckaert, Ernest Valveny, et al. Document understanding dataset and evaluation (dude). In Proceedings of the IEEE/CVF International Conference on Computer… [find](https://scholar.google.com/scholar?q=Jordy+Van+Landeghem%2C+Rub%C3%A8n+Tito%2C+%C5%81ukasz+Borchmann%2C+Micha%C5%82+Pietruszka%2C+Pawel+Joziak%2C+Rafal+Powalski%2C+Dawid+Jurkiewicz%2C+Micka%C3%ABl+Coustaty%2C+Bertrand+Anckaert%2C+Ernest+Valveny%2C+et+al%2E+Document+understanding)
66. C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. [find](https://scholar.google.com/scholar?q=C%2E+Wah%2C+S%2E+Branson%2C+P%2E+Welinder%2C+P%2E+Perona%2C+and+S%2E+Belongie%2E+The+caltech%2Ducsd+birds%2D200%2D2011+dataset%2E+Technical+Report+CNS%2DTR%2D2011%2D001%2C+California+Institute+of+Technology%2C+2011%2E)
67. Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. [Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework](https://md.research.elamur.ai/free/en/papers/wang-2022-ofa-unifying-architectures-tasks-modalities/). In International Conference on Machine Learning, pages 23318–23340. PM…
68. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. [Cogvlm: Visual expert for pretrained language models](https://md.research.elamur.ai/free/en/papers/wang-2023-cogvlm-visual-expert-pretrained/). arXiv preprint arXiv:2311.03079, 2023.
69. Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Wenhai+Wang%2C+Zhe+Chen%2C+Xiaokang+Chen%2C+Jiannan+Wu%2C+Xizhou+Zhu%2C+Gang+Zeng%2C+Ping+Luo%2C+Tong+Lu%2C+Jie+Zhou%2C+Yu+Qiao%2C+et+al%2E+Visionllm%3A+Large+language+model+is+also+an+open%2Dended+decoder+for+vision%2Dcentric+t)
70. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
71. Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. [Visual chatgpt: Talking, drawing and editing with visual foundation models](https://md.research.elamur.ai/free/en/papers/wu-2023-visual-chatgpt-talking-drawing-editing/). arXiv preprint arXiv:2303.04671, 2023.
72. Penghao Wu and Saining Xie. [V*: Guided visual search as a core mechanism in multimodal llms](https://md.research.elamur.ai/free/en/papers/wu-2023-v-guided-visual-search-core/). arXiv preprint arXiv:2312.14135, 17, 2023.
73. Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, et al. Cares: A comprehensive benchmark of trustworthiness in medical vision language models. arXiv preprint arXiv:2406.06007, 2024. — [arXiv:2406.06007](https://arxiv.org/abs/2406.06007) [find](https://scholar.google.com/scholar?q=Peng+Xia%2C+Ze+Chen%2C+Juanxi+Tian%2C+Yangrui+Gong%2C+Ruibo+Hou%2C+Yue+Xu%2C+Zhenbang+Wu%2C+Zhiyuan+Fan%2C+Yiyang+Zhou%2C+Kangyu+Zhu%2C+et+al%2E+Cares%3A+A+comprehensive+benchmark+of+trustworthiness+in+medical+vision+languag)
74. Peng Xia, Siwei Han, Shi Qiu, Yiyang Zhou, Zhaoyang Wang, Wenhao Zheng, Zhaorun Chen, Chenhang Cui, Mingyu Ding, Linjie Li, et al. Mmie: Massive multimodal interleaved comprehension benchmark for large vision-language models. arXiv preprint arXiv:2410.10139, 2024. — [arXiv:2410.10139](https://arxiv.org/abs/2410.10139) [find](https://scholar.google.com/scholar?q=Peng+Xia%2C+Siwei+Han%2C+Shi+Qiu%2C+Yiyang+Zhou%2C+Zhaoyang+Wang%2C+Wenhao+Zheng%2C+Zhaorun+Chen%2C+Chenhang+Cui%2C+Mingyu+Ding%2C+Linjie+Li%2C+et+al%2E+Mmie%3A+Massive+multimodal+interleaved+comprehension+benchmark+for+larg)
75. Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang, Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, and Huaxiu Yao. Mmed-rag: Versatile multimodal rag system for medical vision language models. arXiv preprint arXiv:2410.13085, 2024. — [arXiv:2410.13085](https://arxiv.org/abs/2410.13085) [find](https://scholar.google.com/scholar?q=Peng+Xia%2C+Kangyu+Zhu%2C+Haoran+Li%2C+Tianze+Wang%2C+Weijia+Shi%2C+Sheng+Wang%2C+Linjun+Zhang%2C+James+Zou%2C+and+Huaxiu+Yao%2E+Mmed%2Drag%3A+Versatile+multimodal+rag+system+for+medical+vision+language+models%2E+arXiv+prepr)
76. Peng Xia, Kangyu Zhu, Haoran Li, Hongtu Zhu, Yun Li, Gang Li, Linjun Zhang, and Huaxiu Yao. Rule: Reliable multimodal rag for factuality in medical vision language models. arXiv preprint arXiv:2407.05131, 2024. [find](https://scholar.google.com/scholar?q=Peng+Xia%2C+Kangyu+Zhu%2C+Haoran+Li%2C+Hongtu+Zhu%2C+Yun+Li%2C+Gang+Li%2C+Linjun+Zhang%2C+and+Huaxiu+Yao%2E+Rule%3A+Reliable+multimodal+rag+for+factuality+in+medical+vision+language+models%2E+arXiv+preprint+arXiv%3A2407%2E05)
77. Jinjin Xu, Liwu Xu, Yuzhe Yang, Xiang Li, Yanchun Xie, Yi-Jie Huang, and Yaqian Li. u-llava: Unifying multi-modal tasks via large language model. arXiv preprint arXiv:2311.05348, 2023. — [arXiv:2311.05348](https://arxiv.org/abs/2311.05348) [find](https://scholar.google.com/scholar?q=Jinjin+Xu%2C+Liwu+Xu%2C+Yuzhe+Yang%2C+Xiang+Li%2C+Yanchun+Xie%2C+Yi%2DJie+Huang%2C+and+Yaqian+Li%2E+u%2Dllava%3A+Unifying+multi%2Dmodal+tasks+via+large+language+model%2E+arXiv+preprint+arXiv%3A2311%2E05348%2C+2023%2E+%E2%80%94+%5BarXiv%3A2311%2E0)
78. Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15325–15336, 2023. [find](https://scholar.google.com/scholar?q=Bin+Yan%2C+Yi+Jiang%2C+Jiannan+Wu%2C+Dong+Wang%2C+Ping+Luo%2C+Zehuan+Yuan%2C+and+Huchuan+Lu%2E+Universal+instance+perception+as+object+discovery+and+retrieval%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+)
79. Xu Yang, Yongliang Wu, Mingzhuo Yang, Haokun Chen, and Xin Geng. Exploring diverse in-context configurations for image captioning. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Xu+Yang%2C+Yongliang+Wu%2C+Mingzhuo+Yang%2C+Haokun+Chen%2C+and+Xin+Geng%2E+Exploring+diverse+in%2Dcontext+configurations+for+image+captioning%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2C+2024%2E)
80. Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. [Mm-react: Prompting chatgpt for multimodal reasoning and action](https://md.research.elamur.ai/free/en/papers/yang-2023-mm-react-prompting-chatgpt-multimodal/). arXiv preprint arXiv:2303.11381, 2023.
81. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Tom+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+Advances+in+Neural+Informati)
82. Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023. — [arXiv:2310.07704](https://arxiv.org/abs/2310.07704) [find](https://scholar.google.com/scholar?q=Haoxuan+You%2C+Haotian+Zhang%2C+Zhe+Gan%2C+Xianzhi+Du%2C+Bowen+Zhang%2C+Zirui+Wang%2C+Liangliang+Cao%2C+Shih%2DFu+Chang%2C+and+Yinfei+Yang%2E+Ferret%3A+Refer+and+ground+anything+anywhere+at+any+granularity%2E+arXiv+preprint+)
83. Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, et al. Map-neo: Highly capable and transparent bilingual large language model series. arXiv preprint arXiv:2405.19327, 2024. [find](https://scholar.google.com/scholar?q=Ge+Zhang%2C+Scott+Qu%2C+Jiaheng+Liu%2C+Chenchen+Zhang%2C+Chenghua+Lin%2C+Chou+Leuang+Yu%2C+Danny+Pan%2C+Esther+Cheng%2C+Jie+Liu%2C+Qunshu+Lin%2C+et+al%2E+Map%2Dneo%3A+Highly+capable+and+transparent+bilingual+large+language+mod)
84. Hang Zhang, Xin Li, and Lidong Bing. [Video-llama: An instruction-tuned audio-visual language model for video understanding](https://md.research.elamur.ai/free/en/papers/zhang-2023-video-llama-instruction-tuned-audio/). arXiv preprint arXiv:2306.02858, 2023.
85. Jiacheng Zhang, Yang Jiao, Shaoxiang Chen, Jingjing Chen, and Yu-Gang Jiang. Eventhallusion: Diagnosing event hallucinations in video llms. arXiv preprint arXiv:2409.16597, 2024. — [arXiv:2409.16597](https://arxiv.org/abs/2409.16597) [find](https://scholar.google.com/scholar?q=Jiacheng+Zhang%2C+Yang+Jiao%2C+Shaoxiang+Chen%2C+Jingjing+Chen%2C+and+Yu%2DGang+Jiang%2E+Eventhallusion%3A+Diagnosing+event+hallucinations+in+video+llms%2E+arXiv+preprint+arXiv%3A2409%2E16597%2C+2024%2E+%E2%80%94+%5BarXiv%3A2409%2E16597%5D%28)
86. Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv preprint arXiv:2307.03601, 2023. — [arXiv:2307.03601](https://arxiv.org/abs/2307.03601) [find](https://scholar.google.com/scholar?q=Shilong+Zhang%2C+Peize+Sun%2C+Shoufa+Chen%2C+Min+Xiao%2C+Wenqi+Shao%2C+Wenwei+Zhang%2C+Kai+Chen%2C+and+Ping+Luo%2E+Gpt4roi%3A+Instruction+tuning+large+language+model+on+region%2Dof%2Dinterest%2E+arXiv+preprint+arXiv%3A2307%2E036)
87. Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context learning? Advances in Neural Information Processing Systems, 2023. [find](https://scholar.google.com/scholar?q=Yuanhan+Zhang%2C+Kaiyang+Zhou%2C+and+Ziwei+Liu%2E+What+makes+good+examples+for+visual+in%2Dcontext+learning%3F+Advances+in+Neural+Information+Processing+Systems%2C+2023%2E)
88. Yuechen Zhang, Shengju Qian, Bohao Peng, Shu Liu, and Jiaya Jia. Prompt highlighter: Interactive control for multi-modal llms. arXiv preprint arXiv:2312.04302, 2023. — [arXiv:2312.04302](https://arxiv.org/abs/2312.04302) [find](https://scholar.google.com/scholar?q=Yuechen+Zhang%2C+Shengju+Qian%2C+Bohao+Peng%2C+Shu+Liu%2C+and+Jiaya+Jia%2E+Prompt+highlighter%3A+Interactive+control+for+multi%2Dmodal+llms%2E+arXiv+preprint+arXiv%3A2312%2E04302%2C+2023%2E+%E2%80%94+%5BarXiv%3A2312%2E04302%5D%28https%3A%2F%2Farxiv)
89. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. [Automatic chain of thought prompting in large language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-automatic-chain-thought-prompting-large/). arXiv preprint arXiv:2210.03493, 2022.
90. Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. [Multimodal chain-of-thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/zhang-2023-multimodal-chain-thought-reasoning-language/). arXiv preprint arXiv:2302.00923, 2023.
91. Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models. Advances in Neural Information Processing Systems, 36:5168–5191, 2023. [find](https://scholar.google.com/scholar?q=Ge+Zheng%2C+Bin+Yang%2C+Jiajin+Tang%2C+Hong%2DYu+Zhou%2C+and+Sibei+Yang%2E+Ddcot%3A+Duty%2Ddistinct+chain%2Dof%2Dthought+prompting+for+multimodal+reasoning+in+language+models%2E+Advances+in+Neural+Information+Processing+Sy)
92. Yiyi Zhou, Tianhe Ren, Chaoyang Zhu, Xiaoshuai Sun, Jianzhuang Liu, Xinghao Ding, Mingliang Xu, and Rongrong Ji. Trar: Routing the attention spans in transformer for visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2074–2084, 2021. [find](https://scholar.google.com/scholar?q=Yiyi+Zhou%2C+Tianhe+Ren%2C+Chaoyang+Zhu%2C+Xiaoshuai+Sun%2C+Jianzhuang+Liu%2C+Xinghao+Ding%2C+Mingliang+Xu%2C+and+Rongrong+Ji%2E+Trar%3A+Routing+the+attention+spans+in+transformer+for+visual+question+answering%2E+In+Proc)
93. Chaoyang Zhu, Yiyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao, Xiaoshuai Sun, and Rongrong Ji. Seqtr: A simple yet universal network for visual grounding. In European Conference on Computer Vision, pages 598–615. Springer, 2022. [find](https://scholar.google.com/scholar?q=Chaoyang+Zhu%2C+Yiyi+Zhou%2C+Yunhang+Shen%2C+Gen+Luo%2C+Xingjia+Pan%2C+Mingbao+Lin%2C+Chao+Chen%2C+Liujuan+Cao%2C+Xiaoshuai+Sun%2C+and+Rongrong+Ji%2E+Seqtr%3A+A+simple+yet+universal+network+for+visual+grounding%2E+In+Europea)
94. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. [Minigpt-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/). arXiv preprint arXiv:2304.10592, 2023.
95. Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. [Visual7w: Grounded question answering in images](https://md.research.elamur.ai/free/en/papers/zhu-2015-visual7w-grounded-question-answering-images/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4995–5004, 2016.
96. Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li, and Yu Liu. Mova: Adapting mixture of vision experts to multimodal context. arXiv preprint arXiv:2404.13046, 2024. — [arXiv:2404.13046](https://arxiv.org/abs/2404.13046) [find](https://scholar.google.com/scholar?q=Zhuofan+Zong%2C+Bingqi+Ma%2C+Dazhong+Shen%2C+Guanglu+Song%2C+Hao+Shao%2C+Dongzhi+Jiang%2C+Hongsheng+Li%2C+and+Yu+Liu%2E+Mova%3A+Adapting+mixture+of+vision+experts+to+multimodal+context%2E+arXiv+preprint+arXiv%3A2404%2E13046%2C)

*Updated 2 November 2026*