---
title: "Improved Baselines with Visual Instruction Tuning — Liu 2023"
description: "Large multimodal models (LMMs) differ widely in architecture and training data, so it has been hard to tell which design choices actually matter."
authors: ["Haotian Liu", "Chunyuan Li", "Yuheng Li", "Yong Jae Lee"]
year: 2023
venue: "arXiv:2310.03744"
source: "https://arxiv.org/abs/2310.03744"
doi: "10.48550/arxiv.2310.03744"
arxiv: "2310.03744"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) / Improved Baselines with Visual Instruction Tuning

# Improved Baselines with Visual Instruction Tuning

**Authors:** [Haotian Liu](https://md.research.elamur.ai/free/en/authors/haotian-liu/), [Chunyuan Li](https://md.research.elamur.ai/free/en/authors/chunyuan-li/), Yuheng Li, [Yong Jae Lee](https://md.research.elamur.ai/free/en/authors/yong-jae-lee/) · **Year:** 2023 · **Venue:** arXiv:2310.03744 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.03744) · [PDF](https://arxiv.org/pdf/2310.03744) · [DOI 10.48550/arxiv.2310.03744](https://doi.org/10.48550/arxiv.2310.03744) · [arXiv 2310.03744](https://arxiv.org/abs/2310.03744) · [OpenAlex](https://openalex.org/W4402727764)

**Citations:** 1637 citations (openalex-snapshot · 2026-10-14) · [920](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Large multimodal models (LMMs) differ widely in architecture and training data, so it has been hard to tell which design choices actually matter. For example, LLaVA does well in open-ended visual chat, while InstructBLIP is stronger on short-answer VQA. The authors run a controlled study within the LLaVA framework and find that its plain fully-connected vision-language connector is surprisingly strong and data-efficient. Their improved baseline, LLaVA-1.5, makes three simple changes:
- it uses a CLIP-ViT-L-336px encoder,
- it replaces the linear projection with an MLP,
- it adds academic VQA data, with a brief instruction appended to each question to request short answers.

The format instruction lets one model switch between short and detailed replies. LLaVA-1.5 reaches state-of-the-art results on 11 benchmarks using only public data. The paper also explores higher-resolution inputs, compositional generalization, data efficiency and hallucination. Because training is cheap and reproducible, the work lowers the barrier for open LMM research.

## Key points
- The final 13B checkpoint is trained on about 1.2M publicly available samples and takes roughly one day on a single 8-A100 node. InstructBLIP and Qwen-VL, by contrast, pretrain resamplers on 129M and 1.4B image-text pairs, respectively.
- Adding VQAv2 together with the response formatting prompt raises the MME score from 809.6 to 1323.8, which is 111 points above InstructBLIP.
- Randomly cutting the training mixture by up to 75% causes no significant performance drop.
- Splitting images into grids allows higher-resolution input, which improves fine-grained perception and reduces hallucination.
- Training on English visual data plus multilingual text-only ShareGPT yields multilingual visual conversation, though errors remain (e.g., in Korean). Single-image-only input and remaining hallucinations are acknowledged as limitations.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

920 papers of our library cite it (of the 140673 whose bibliographies we read); 48 of them have a page here, 872 not yet.

- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/)
- [Yi: Open Foundation Models by 01.AI](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/)
- [Amharic LLaMA and LLaVA: Multimodal LLMs for Low Resource Languages](https://md.research.elamur.ai/free/en/papers/andersland-2024-amharic-llama-llava-multimodal-llms/)
- [Qwen2.5-VL Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen2-5-vl-technical/)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://md.research.elamur.ai/free/en/papers/chen-2023-internvl-scaling-up-vision/)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://md.research.elamur.ai/free/en/papers/chen-2023-sharegpt4v-improving-large-multi-modal/)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://md.research.elamur.ai/free/en/papers/chen-2024-expanding-performance-boundaries-open/)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://md.research.elamur.ai/free/en/papers/chen-2024-far-gpt-4v-closing/)
- [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/chen-2024-image-worth-1-2-tokens/)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://md.research.elamur.ai/free/en/papers/chen-2024-right-way-evaluating-large-vision/)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://md.research.elamur.ai/free/en/papers/chen-2025-janus-pro-unified-multimodal/)
- [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training](https://md.research.elamur.ai/free/en/papers/chu-2025-sft-memorizes-rl-generalizes-comparative/)
- [Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models](https://md.research.elamur.ai/free/en/papers/deitke-2024-molmo-pixmo-open-weights-open/)
- [InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model](https://md.research.elamur.ai/free/en/papers/dong-2024-internlm-xcomposer2-mastering-free/)
- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/guan-2023-hallusionbench-advanced-diagnostic-suite-entangled/)
- [Baseer: A Vision-Language Model for Arabic Document-to-Markdown OCR](https://md.research.elamur.ai/free/en/papers/hennara-2025-baseer-vision-language-model-arabic/)
- [CogAgent: A Visual Language Model for GUI Agents](https://md.research.elamur.ai/free/en/papers/hong-2023-cogagent-visual-language-model/)
- [E-CONAN (Entailment, CONtradition And Neutral) Benchmarks: Arabic Textual Entailment and Natural Inference Datasets](https://md.research.elamur.ai/free/en/papers/jallad-2026-e-conan-entailment-contradition-neutral-benchmarks/)
- [VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks](https://md.research.elamur.ai/free/en/papers/koh-2024-visualwebarena-evaluating-multimodal-agents-realistic/)
- [What matters when building vision-language models?](https://md.research.elamur.ai/free/en/papers/lauren-2024-ccon-matters-building-vision-language-models/)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://md.research.elamur.ai/free/en/papers/li-2024-llava-onevision-easy-visual-task/)
- [Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/li-2024-multimodal-arxiv-dataset-improving-scientific/)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://md.research.elamur.ai/free/en/papers/lin-2023-video-llava-learning-united-visual/)
- [VILA: On Pre-training for Visual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2023-vila-pre-training-visual-language/)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/)
- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://md.research.elamur.ai/free/en/papers/liu-2023-ocrbench-hidden-mystery-ocr-large/)
- [Visual Instruction Tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/)
- [On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey](https://md.research.elamur.ai/free/en/papers/long-2024-llms-driven-synthetic-data-generation/)
- [A Large-Scale Instruction-Tuning Dataset and Models for Slovenian Vision-Language Tasks](https://md.research.elamur.ai/free/en/papers/martinc-2026-large-scale-instruction-tuning-dataset/)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://md.research.elamur.ai/free/en/papers/mazeika-2024-harmbench-standardized-evaluation-framework-automated/)
- [Evaluating Discriminability of Vision-Language Models](https://md.research.elamur.ai/free/en/papers/muraoka-2026-evaluating-discriminability-vision-language-models/)
- [ViX-Ray: A Vietnamese Chest X-Ray Dataset for Vision-Language Models](https://md.research.elamur.ai/free/en/papers/nguyen-2026-vix-ray-vietnamese-chest-x/)
- [Multimodal LLMs Do Not Compose Skills Optimally across Modalities](https://md.research.elamur.ai/free/en/papers/ontalvilla-2026-multimodal-llms-do-not-compose/)
- [Evaluating Multimodal Large Language Models on Vertically Written Japanese Text](https://md.research.elamur.ai/free/en/papers/sasagawa-2026-evaluating-multimodal-large-language-models/)
- [Seeing the Other Side: Diagnostic Tasks for Viewpoint Reasoning in Vision–Language Models](https://md.research.elamur.ai/free/en/papers/takenaka-2026-seeing-other-side-diagnostic-tasks/)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://md.research.elamur.ai/free/en/papers/team-2024-chameleon-mixed-modal-early-fusion/)
- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://md.research.elamur.ai/free/en/papers/tong-2024-cambrian-1-fully-open-vision/)
- [CogVLM: Visual Expert for Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/wang-2023-cogvlm-visual-expert-pretrained/)
- [Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset](https://md.research.elamur.ai/free/en/papers/wang-2024-measuring-multimodal-mathematical-reasoning-math/)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://md.research.elamur.ai/free/en/papers/wang-2024-qwen2-vl-enhancing-vision/)
- [Qwen2.5-Omni Technical Report](https://md.research.elamur.ai/free/en/papers/xu-2025-qwen2-5-omni-technical/)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://md.research.elamur.ai/free/en/papers/zhang-2023-llama-adapter-efficient-fine-tuning/)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://md.research.elamur.ai/free/en/papers/zhang-2024-mathverse-multi-modal-llm-truly/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://md.research.elamur.ai/free/en/papers/zheng-2023-sglang-efficient-execution-structured-language/)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://md.research.elamur.ai/free/en/papers/zheng-2024-gpt-4v-ision-generalist-web/)
- [SPARTA: Evaluating Reasoning Segmentation Robustness through Black-Box Adversarial Paraphrasing in Text Autoencoder Latent Space](https://md.research.elamur.ai/free/en/papers/zinkovich-2026-sparta-evaluating-reasoning-segmentation-robustness/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{liu2023improved,
  title = {{Improved Baselines with Visual Instruction Tuning}},
  author = {Liu, Haotian and Li, Chunyuan and Li, Yuheng and Lee, Yong Jae},
  year = {2023},
  doi = {10.48550/arxiv.2310.03744},
  eprint = {2310.03744},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.03744}
}
```

- [← previous Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasing](https://md.research.elamur.ai/free/en/papers/jung-2023-impossible-distillation-low-quality-model/)
- [next → Improving Cross-lingual Transfer through Subtree-aware Word Reordering](https://md.research.elamur.ai/free/en/papers/arviv-2023-improving-cross-lingual-transfer-through/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/)

## References

62 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Adept AI. Fuyu-8b: A multimodal architecture for ai agents. [https://www.adept.ai/blog/fuyu-8b](https://www.adept.ai/blog/fuyu-8b), 2024. 2 [↗](https://www.adept.ai/blog/fuyu-8b)
2. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022. 1 [find](https://scholar.google.com/scholar?q=Jean%2DBaptiste+Alayrac%2C+Jeff+Donahue%2C+Pauline+Luc%2C+Antoine+Miech%2C+Iain+Barr%2C+Yana+Hasson%2C+Karel+Lenc%2C+Arthur+Mensch%2C+Katie+Millican%2C+Malcolm+Reynolds%2C+et+al%2E+Flamingo%3A+a+visual+language+model+for+few%2Ds) [our card](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
3. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 1, 2, 4, 5, 6, 13 [find](https://scholar.google.com/scholar?q=Jinze+Bai%2C+Shuai+Bai%2C+Shusheng+Yang%2C+Shijie+Wang%2C+Sinan+Tan%2C+Peng+Wang%2C+Junyang+Lin%2C+Chang+Zhou%2C+and+Jingren+Zhou%2E+Qwen%2Dvl%3A+A+frontier+large+vision%2Dlanguage+model+with+versatile+abilities%2E+arXiv+prepr) [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/)
4. Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schimdt. Visit-bench: A benchmark for vision-language instruction following inspired by real-world use, 2023. 3 [find](https://scholar.google.com/scholar?q=Yonatan+Bitton%2C+Hritik+Bansal%2C+Jack+Hessel%2C+Rulin+Shao%2C+Wanrong+Zhu%2C+Anas+Awadalla%2C+Josh+Gardner%2C+Rohan+Taori%2C+and+Ludwig+Schimdt%2E+Visit%2Dbench%3A+A+benchmark+for+vision%2Dlanguage+instruction+following+in)
5. Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301, 2023. 1 — [arXiv:2305.13301](https://arxiv.org/abs/2305.13301) [find](https://scholar.google.com/scholar?q=Kevin+Black%2C+Michael+Janner%2C+Yilun+Du%2C+Ilya+Kostrikov%2C+and+Sergey+Levine%2E+Training+diffusion+models+with+reinforcement+learning%2E+arXiv+preprint+arXiv%3A2305%2E13301%2C+2023%2E+1+%E2%80%94+%5BarXiv%3A2305%2E13301%5D%28https%3A%2F%2Fa)
6. Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, et al. Are aligned neural networks adversarially aligned? arXiv preprint arXiv:2306.15447, 2023. 1 [find](https://scholar.google.com/scholar?q=Nicholas+Carlini%2C+Milad+Nasr%2C+Christopher+A+Choquette%2DChoo%2C+Matthew+Jagielski%2C+Irena+Gao%2C+Anas+Awadalla%2C+Pang+Wei+Koh%2C+Daphne+Ippolito%2C+Katherine+Lee%2C+Florian+Tramer%2C+et+al%2E+Are+aligned+neural+network)
7. Delong Chen, Jianfeng Liu, Wenliang Dai, and Baoyuan Wang. Visual instruction tuning with polite flamingo. arXiv preprint arXiv:2307.01003, 2023. 2, 3 — [arXiv:2307.01003](https://arxiv.org/abs/2307.01003) [find](https://scholar.google.com/scholar?q=Delong+Chen%2C+Jianfeng+Liu%2C+Wenliang+Dai%2C+and+Baoyuan+Wang%2E+Visual+instruction+tuning+with+polite+flamingo%2E+arXiv+preprint+arXiv%3A2307%2E01003%2C+2023%2E+2%2C+3+%E2%80%94+%5BarXiv%3A2307%2E01003%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2307%2E0)
8. Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. 1, 2, 4, 5 [find](https://scholar.google.com/scholar?q=Keqin+Chen%2C+Zhao+Zhang%2C+Weili+Zeng%2C+Richong+Zhang%2C+Feng+Zhu%2C+and+Rui+Zhao%2E+Shikra%3A+Unleashing+multimodal+llm%E2%80%99s+referential+dialogue+magic%2E+arXiv+preprint+arXiv%3A2306%2E15195%2C+2023%2E+1%2C+2%2C+4%2C+5) [our card](https://md.research.elamur.ai/free/en/papers/chen-2023-shikra-unleashing-multimodal-llm-s/)
9. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 3 [find](https://scholar.google.com/scholar?q=Ting+Chen%2C+Simon+Kornblith%2C+Mohammad+Norouzi%2C+and+Geoffrey+Hinton%2E+A+simple+framework+for+contrastive+learning+of+visual+representations%2E+In+ICML%2C+2020%2E+3) [our card](https://md.research.elamur.ai/free/en/papers/chen-2020-simple-framework-contrastive-learning-visual/)
10. Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020. 3 — [arXiv:2003.04297](https://arxiv.org/abs/2003.04297) [find](https://scholar.google.com/scholar?q=Xinlei+Chen%2C+Haoqi+Fan%2C+Ross+Girshick%2C+and+Kaiming+He%2E+Improved+baselines+with+momentum+contrastive+learning%2E+arXiv+preprint+arXiv%3A2003%2E04297%2C+2020%2E+3+%E2%80%94+%5BarXiv%3A2003%2E04297%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2003%2E0)
11. Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.18565, 2023. 5 — [arXiv:2305.18565](https://arxiv.org/abs/2305.18565) [find](https://scholar.google.com/scholar?q=Xi+Chen%2C+Josip+Djolonga%2C+Piotr+Padlewski%2C+Basil+Mustafa%2C+Soravit+Changpinyo%2C+Jialin+Wu%2C+Carlos+Riquelme+Ruiz%2C+Sebastian+Goodman%2C+Xiao+Wang%2C+Yi+Tay%2C+et+al%2E+Pali%2Dx%3A+On+scaling+up+a+multilingual+vision+a)
12. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 9 [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+and+Eric+P%2E+Xing%2E+Vicuna%3A+An+open%2Dsource+chatbot+i)
13. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. 2 [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Yunxuan+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXiv+p)
14. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500, 2023. 1, 2, 3, 5, 6, 8, 13 [find](https://scholar.google.com/scholar?q=Wenliang+Dai%2C+Junnan+Li%2C+Dongxu+Li%2C+Anthony+Meng+Huat+Tiong%2C+Junqi+Zhao%2C+Weisheng+Wang%2C+Boyang+Li%2C+Pascale+Fung%2C+and+Steven+Hoi%2E+Instructblip%3A+Towards+general%2Dpurpose+vision%2Dlanguage+models+with+instr) [our card](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/)
15. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 4 [find](https://scholar.google.com/scholar?q=Alexey+Dosovitskiy%2C+Lucas+Beyer%2C+Alexander+Kolesnikov%2C+Dirk+Weissenborn%2C+Xiaohua+Zhai%2C+Thomas+Unterthiner%2C+Mostafa+Dehghani%2C+Matthias+Minderer%2C+Georg+Heigold%2C+Sylvain+Gelly%2C+et+al%2E+An+image+is+worth+1)
16. Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19358– 19369, 2023.… [find](https://scholar.google.com/scholar?q=Yuxin+Fang%2C+Wen+Wang%2C+Binhui+Xie%2C+Quan+Sun%2C+Ledell+Wu%2C+Xinggang+Wang%2C+Tiejun+Huang%2C+Xinlong+Wang%2C+and+Yue+Cao%2E+Eva%3A+Exploring+the+limits+of+masked+visual+representation+learning+at+scale%2E+In+Proceedin)
17. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 1, 4, 5 [find](https://scholar.google.com/scholar?q=Chaoyou+Fu%2C+Peixian+Chen%2C+Yunhang+Shen%2C+Yulei+Qin%2C+Mengdan+Zhang%2C+Xu+Lin%2C+Zhenyu+Qiu%2C+Wei+Lin%2C+Jinrui+Yang%2C+Xiawu+Zheng%2C+et+al%2E+Mme%3A+A+comprehensive+evaluation+benchmark+for+multimodal+large+language+) [our card](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/)
18. Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans. arXiv preprint arXiv:2305.04790, 2023. 1 — [arXiv:2305.04790](https://arxiv.org/abs/2305.04790) [find](https://scholar.google.com/scholar?q=Tao+Gong%2C+Chengqi+Lyu%2C+Shilong+Zhang%2C+Yudong+Wang%2C+Miao+Zheng%2C+Qian+Zhao%2C+Kuikun+Liu%2C+Wenwei+Zhang%2C+Ping+Luo%2C+and+Kai+Chen%2E+Multimodal%2Dgpt%3A+A+vision+and+language+model+for+dialogue+with+humans%2E+arXiv+)
19. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017. 3, 5, 6, 9 [find](https://scholar.google.com/scholar?q=Yash+Goyal%2C+Tejas+Khot%2C+Douglas+Summers%2DStay%2C+Dhruv+Batra%2C+and+Devi+Parikh%2E+Making+the+v+in+vqa+matter%3A+Elevating+the+role+of+image+understanding+in+visual+question+answering%2E+In+Proceedings+of+the+IE) [our card](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/)
20. Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617, 2018. 5, 6 [find](https://scholar.google.com/scholar?q=Danna+Gurari%2C+Qing+Li%2C+Abigale+J+Stangl%2C+Anhong+Guo%2C+Chi+Lin%2C+Kristen+Grauman%2C+Jiebo+Luo%2C+and+Jeffrey+P+Bigham%2E+Vizwiz+grand+challenge%3A+Answering+visual+questions+from+blind+people%2E+In+Proceedings+of+)
21. Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 4, 5, 9 [find](https://scholar.google.com/scholar?q=Drew+A+Hudson+and+Christopher+D+Manning%2E+Gqa%3A+A+new+dataset+for+real%2Dworld+visual+reasoning+and+compositional+question+answering%2E+In+CVPR%2C+2019%2E+4%2C+5%2C+9) [our card](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/)
22. IDEFICS. Introducing idefics: An open reproduction of state-of-the-art visual language model. [https://huggingface.co/blog/idefics](https://huggingface.co/blog/idefics), 2023. 5, 6 [↗](https://huggingface.co/blog/idefics)
23. Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. 2021. If you use this software, please cite it as below. 6 [find](https://scholar.google.com/scholar?q=Gabriel+Ilharco%2C+Mitchell+Wortsman%2C+Ross+Wightman%2C+Cade+Gordon%2C+Nicholas+Carlini%2C+Rohan+Taori%2C+Achal+Dave%2C+Vaishaal+Shankar%2C+Hongseok+Namkoong%2C+John+Miller%2C+Hannaneh+Hajishirzi%2C+Ali+Farhadi%2C+and+Ludwi)
24. Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014. 4, 9 [find](https://scholar.google.com/scholar?q=Sahar+Kazemzadeh%2C+Vicente+Ordonez%2C+Mark+Matten%2C+and+Tamara+Berg%2E+Referitgame%3A+Referring+to+objects+in+photographs+of+natural+scenes%2E+In+Proceedings+of+the+2014+conference+on+empirical+methods+in+natur) [our card](https://md.research.elamur.ai/free/en/papers/kazemzadeh-2014-referitgame-referring-objects-photographs-natural/)
25. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32–73, 2017… [find](https://scholar.google.com/scholar?q=Ranjay+Krishna%2C+Yuke+Zhu%2C+Oliver+Groth%2C+Justin+Johnson%2C+Kenji+Hata%2C+Joshua+Kravitz%2C+Stephanie+Chen%2C+Yannis+Kalantidis%2C+Li%2DJia+Li%2C+David+A+Shamma%2C+et+al%2E+Visual+genome%3A+Connecting+language+and+vision+u) [our card](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/)
26. Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308.00692, 2023. 1 — [arXiv:2308.00692](https://arxiv.org/abs/2308.00692) [find](https://scholar.google.com/scholar?q=Xin+Lai%2C+Zhuotao+Tian%2C+Yukang+Chen%2C+Yanwei+Li%2C+Yuhui+Yuan%2C+Shu+Liu%2C+and+Jiaya+Jia%2E+Lisa%3A+Reasoning+segmentation+via+large+language+model%2E+arXiv+preprint+arXiv%3A2308%2E00692%2C+2023%2E+1+%E2%80%94+%5BarXiv%3A2308%2E00692%5D%28)
27. Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. 1, 5 [find](https://scholar.google.com/scholar?q=Bohao+Li%2C+Rui+Wang%2C+Guangzhi+Wang%2C+Yuying+Ge%2C+Yixiao+Ge%2C+and+Ying+Shan%2E+Seed%2Dbench%3A+Benchmarking+multimodal+llms+with+generative+comprehension%2E+arXiv+preprint+arXiv%3A2307%2E16125%2C+2023%2E+1%2C+5) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-seed-bench-benchmarking-multimodal-llms/)
28. Bo Li, Peiyuan Zhang, Jingkang Yang, Yuanhan Zhang, Fanyi Pu, and Ziwei Liu. Otterhd: A high-resolution multi-modality model, 2023. 2 [find](https://scholar.google.com/scholar?q=Bo+Li%2C+Peiyuan+Zhang%2C+Jingkang+Yang%2C+Yuanhan+Zhang%2C+Fanyi+Pu%2C+and+Ziwei+Liu%2E+Otterhd%3A+A+high%2Dresolution+multi%2Dmodality+model%2C+2023%2E+2)
29. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023. 1 — [arXiv:2306.05425](https://arxiv.org/abs/2306.05425) [find](https://scholar.google.com/scholar?q=Bo+Li%2C+Yuanhan+Zhang%2C+Liangyu+Chen%2C+Jinghao+Wang%2C+Fanyi+Pu%2C+Jingkang+Yang%2C+Chunyuan+Li%2C+and+Ziwei+Liu%2E+Mimic%2Dit%3A+Multi%2Dmodal+in%2Dcontext+instruction+tuning%2E+arXiv+preprint+arXiv%3A2306%2E05425%2C+2023%2E+1+%E2%80%94+%5B)
30. Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, and Jianfeng Gao. Multimodal foundation models: From specialists to general-purpose assistants. arXiv preprint arXiv:2309.10020, 2023. 1 — [arXiv:2309.10020](https://arxiv.org/abs/2309.10020) [find](https://scholar.google.com/scholar?q=Chunyuan+Li%2C+Zhe+Gan%2C+Zhengyuan+Yang%2C+Jianwei+Yang%2C+Linjie+Li%2C+Lijuan+Wang%2C+and+Jianfeng+Gao%2E+Multimodal+foundation+models%3A+From+specialists+to+general%2Dpurpose+assistants%2E+arXiv+preprint+arXiv%3A2309%2E10)
31. Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. arXiv preprint arXiv:2306.00890, 2023. 1 [find](https://scholar.google.com/scholar?q=Chunyuan+Li%2C+Cliff+Wong%2C+Sheng+Zhang%2C+Naoto+Usuyama%2C+Haotian+Liu%2C+Jianwei+Yang%2C+Tristan+Naumann%2C+Hoifung+Poon%2C+and+Jianfeng+Gao%2E+Llava%2Dmed%3A+Training+a+large+language%2Dand%2Dvision+assistant+for+biomedici) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-llava-med-training-large-language/)
32. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. 1, 2, 3, 4, 5, 6, 13 [find](https://scholar.google.com/scholar?q=Junnan+Li%2C+Dongxu+Li%2C+Silvio+Savarese%2C+and+Steven+Hoi%2E+Blip%2D2%3A+Bootstrapping+language%2Dimage+pre%2Dtraining+with+frozen+image+encoders+and+large+language+models%2E+arXiv+preprint+arXiv%3A2301%2E12597%2C+2023%2E+1%2C) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/)
33. Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021. 3 [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+arXiv+preprint+arXiv%3A2101%2E00190%2C+2021%2E+3)
34. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 1, 5 [find](https://scholar.google.com/scholar?q=Yifan+Li%2C+Yifan+Du%2C+Kun+Zhou%2C+Jinpeng+Wang%2C+Wayne+Xin+Zhao%2C+and+Ji%2DRong+Wen%2E+Evaluating+object+hallucination+in+large+vision%2Dlanguage+models%2E+arXiv+preprint+arXiv%3A2305%2E10355%2C+2023%2E+1%2C+5) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-evaluating-object-hallucination-large-vision/)
35. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014. 2, 5 [find](https://scholar.google.com/scholar?q=Tsung%2DYi+Lin%2C+Michael+Maire%2C+Serge+Belongie%2C+James+Hays%2C+Pietro+Perona%2C+Deva+Ramanan%2C+Piotr+Doll%C3%A1r%2C+and+C+Lawrence+Zitnick%2E+Microsoft+COCO%3A+Common+objects+in+context%2E+In+ECCV%2C+2014%2E+2%2C+5) [our card](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/)
36. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 1, 2, 3, 4, 5, 6, 8, 9 [find](https://scholar.google.com/scholar?q=Haotian+Liu%2C+Chunyuan+Li%2C+Qingyang+Wu%2C+and+Yong+Jae+Lee%2E+Visual+instruction+tuning%2E+In+NeurIPS%2C+2023%2E+1%2C+2%2C+3%2C+4%2C+5%2C+6%2C+8%2C+9) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/)
37. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023. 1, 5, 7 [find](https://scholar.google.com/scholar?q=Yuan+Liu%2C+Haodong+Duan%2C+Yuanhan+Zhang%2C+Bo+Li%2C+Songyang+Zhang%2C+Wangbo+Zhao%2C+Yike+Yuan%2C+Jiaqi+Wang%2C+Conghui+He%2C+Ziwei+Liu%2C+et+al%2E+Mmbench%3A+Is+your+multi%2Dmodal+model+an+all%2Daround+player%3F+arXiv+preprint+) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/)
38. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 2022. 5 [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Swaroop+Mishra%2C+Tanglin+Xia%2C+Liang+Qiu%2C+Kai%2DWei+Chang%2C+Song%2DChun+Zhu%2C+Oyvind+Tafjord%2C+Peter+Clark%2C+and+Ashwin+Kalyan%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+science+ques)
39. Yadong Lu, Chunyuan Li, Haotian Liu, Jianwei Yang, Jianfeng Gao, and Yelong Shen. An empirical study of scaling instruct-tuned large multimodal models. arXiv preprint arXiv:2309.09958, 2023. 1, 4 — [arXiv:2309.09958](https://arxiv.org/abs/2309.09958) [find](https://scholar.google.com/scholar?q=Yadong+Lu%2C+Chunyuan+Li%2C+Haotian+Liu%2C+Jianwei+Yang%2C+Jianfeng+Gao%2C+and+Yelong+Shen%2E+An+empirical+study+of+scaling+instruct%2Dtuned+large+multimodal+models%2E+arXiv+preprint+arXiv%3A2309%2E09958%2C+2023%2E+1%2C+4+%E2%80%94+%5Ba)
40. Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016. 4, 9 [find](https://scholar.google.com/scholar?q=Junhua+Mao%2C+Jonathan+Huang%2C+Alexander+Toshev%2C+Oana+Camburu%2C+Alan+L+Yuille%2C+and+Kevin+Murphy%2E+Generation+and+comprehension+of+unambiguous+object+descriptions%2E+In+Proceedings+of+the+IEEE+conference+on+c) [our card](https://md.research.elamur.ai/free/en/papers/mao-2015-generation-comprehension-unambiguous-object-descriptions/)
41. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 4, 9 [find](https://scholar.google.com/scholar?q=Kenneth+Marino%2C+Mohammad+Rastegari%2C+Ali+Farhadi%2C+and+Roozbeh+Mottaghi%2E+Ok%2Dvqa%3A+A+visual+question+answering+benchmark+requiring+external+knowledge%2E+In+Conference+on+Computer+Vision+and+Pattern+Recognit)
42. Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947–952. IEEE, 2019. 4, 9 [find](https://scholar.google.com/scholar?q=Anand+Mishra%2C+Shashank+Shekhar%2C+Ajeet+Kumar+Singh%2C+and+Anirban+Chakraborty%2E+Ocr%2Dvqa%3A+Visual+question+answering+by+reading+text+in+images%2E+In+2019+international+conference+on+document+analysis+and+reco)
43. OpenAI. Gpt-4v(ision) system card. [https://cdn.openai](https://cdn.openai) .com /papers/GPTV_System_Card.pdf, 2023. 1 [↗](https://cdn.openai)
44. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021. 6 [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jong+Wook+Kim%2C+Chris+Hallacy%2C+Aditya+Ramesh%2C+Gabriel+Goh%2C+Sandhini+Agarwal%2C+Girish+Sastry%2C+Amanda+Askell%2C+Pamela+Mishkin%2C+Jack+Clark%2C+et+al%2E+Learning+transferable+visual+models+from+natu) [our card](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/)
45. Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, pages 146–162. Springer, 2022. 4, 9 [find](https://scholar.google.com/scholar?q=Dustin+Schwenk%2C+Apoorv+Khandelwal%2C+Christopher+Clark%2C+Kenneth+Marino%2C+and+Roozbeh+Mottaghi%2E+A%2Dokvqa%3A+A+benchmark+for+visual+question+answering+using+world+knowledge%2E+In+European+Conference+on+Computer) [our card](https://md.research.elamur.ai/free/en/papers/schwenk-2022-okvqa-benchmark-visual-question-answering/)
46. ShareGPT. [https://sharegpt.com/](https://sharegpt.com/), 2023. 4, 7, 8, 9 [↗](https://sharegpt.com/)
47. Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 742–758. Springer, 2020. 4, 9 [find](https://scholar.google.com/scholar?q=Oleksii+Sidorov%2C+Ronghang+Hu%2C+Marcus+Rohrbach%2C+and+Amanpreet+Singh%2E+Textcaps%3A+a+dataset+for+image+captioning+with+reading+comprehension%2E+In+Computer+Vision%E2%80%93ECCV+2020%3A+16th+European+Conference%2C+Glasgow)
48. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 5 [find](https://scholar.google.com/scholar?q=Amanpreet+Singh%2C+Vivek+Natarajan%2C+Meet+Shah%2C+Yu+Jiang%2C+Xinlei+Chen%2C+Dhruv+Batra%2C+Devi+Parikh%2C+and+Marcus+Rohrbach%2E+Towards+vqa+models+that+can+read%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+comput)
49. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 7 [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtun)
50. Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. arXiv preprint arXiv:2305.11175, 2023. 1 — [arXiv:2305.11175](https://arxiv.org/abs/2305.11175) [find](https://scholar.google.com/scholar?q=Wenhai+Wang%2C+Zhe+Chen%2C+Xiaokang+Chen%2C+Jiannan+Wu%2C+Xizhou+Zhu%2C+Gang+Zeng%2C+Ping+Luo%2C+Tong+Lu%2C+Jie+Zhou%2C+Yu+Qiao%2C+et+al%2E+Visionllm%3A+Large+language+model+is+also+an+open%2Dended+decoder+for+vision%2Dcentric+t)
51. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021. 2 [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Bosma%2C+Vincent+Y+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+M+Dai%2C+and+Quoc+V+Le%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+arXiv+preprint+arXiv%3A2109%2E01652)
52. Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421, 2023. 6 [find](https://scholar.google.com/scholar?q=Zhengyuan+Yang%2C+Linjie+Li%2C+Kevin+Lin%2C+Jianfeng+Wang%2C+Chung%2DChing+Lin%2C+Zicheng+Liu%2C+and+Lijuan+Wang%2E+The+dawn+of+lmms%3A+Preliminary+explorations+with+gpt%2D4v+%28ision%29%2E+arXiv+preprint+arXiv%3A2309%2E17421%2C+202) [our card](https://md.research.elamur.ai/free/en/papers/yang-2023-dawn-lmms-preliminary-explorations-gpt/)
53. Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, Qin Jin, Liang He, Xin Alex Lin, and Fei Huang. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model, 2023. 2 [find](https://scholar.google.com/scholar?q=Jiabo+Ye%2C+Anwen+Hu%2C+Haiyang+Xu%2C+Qinghao+Ye%2C+Ming+Yan%2C+Guohai+Xu%2C+Chenliang+Li%2C+Junfeng+Tian%2C+Qi+Qian%2C+Ji+Zhang%2C+Qin+Jin%2C+Liang+He%2C+Xin+Alex+Lin%2C+and+Fei+Huang%2E+Ureader%3A+Universal+ocr%2Dfree+visually%2Dsit)
54. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023. 1 [find](https://scholar.google.com/scholar?q=Qinghao+Ye%2C+Haiyang+Xu%2C+Guohai+Xu%2C+Jiabo+Ye%2C+Ming+Yan%2C+Yiyang+Zhou%2C+Junyang+Wang%2C+Anwen+Hu%2C+Pengcheng+Shi%2C+Yaya+Shi%2C+et+al%2E+mplug%2Dowl%3A+Modularization+empowers+large+language+models+with+multimodality%2E) [our card](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/)
55. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. 1, 3, 4, 5, 8 [find](https://scholar.google.com/scholar?q=Weihao+Yu%2C+Zhengyuan+Yang%2C+Linjie+Li%2C+Jianfeng+Wang%2C+Kevin+Lin%2C+Zicheng+Liu%2C+Xinchao+Wang%2C+and+Lijuan+Wang%2E+Mm%2Dvet%3A+Evaluating+large+multimodal+models+for+integrated+capabilities%2E+arXiv+preprint+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/)
56. Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv preprint arXiv:2307.03601, 2023. 1 — [arXiv:2307.03601](https://arxiv.org/abs/2307.03601) [find](https://scholar.google.com/scholar?q=Shilong+Zhang%2C+Peize+Sun%2C+Shoufa+Chen%2C+Min+Xiao%2C+Wenqi+Shao%2C+Wenwei+Zhang%2C+Kai+Chen%2C+and+Ping+Luo%2E+Gpt4roi%3A+Instruction+tuning+large+language+model+on+region%2Dof%2Dinterest%2E+arXiv+preprint+arXiv%3A2307%2E036)
57. Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107, 2023. 2 [find](https://scholar.google.com/scholar?q=Yanzhe+Zhang%2C+Ruiyi+Zhang%2C+Jiuxiang+Gu%2C+Yufan+Zhou%2C+Nedim+Lipka%2C+Diyi+Yang%2C+and+Tong+Sun%2E+Llavar%3A+Enhanced+visual+instruction+tuning+for+text%2Drich+image+understanding%2E+arXiv+preprint+arXiv%3A2306%2E17107%2C)
58. Bo Zhao, Boya Wu, and Tiejun Huang. Svit: Scaling up visual instruction tuning. arXiv preprint arXiv:2307.04087, 2023. 1, 2 — [arXiv:2307.04087](https://arxiv.org/abs/2307.04087) [find](https://scholar.google.com/scholar?q=Bo+Zhao%2C+Boya+Wu%2C+and+Tiejun+Huang%2E+Svit%3A+Scaling+up+visual+instruction+tuning%2E+arXiv+preprint+arXiv%3A2307%2E04087%2C+2023%2E+1%2C+2+%E2%80%94+%5BarXiv%3A2307%2E04087%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2307%2E04087%29)
59. Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models. arXiv preprint arXiv:2305.16934, 2023. 1 — [arXiv:2305.16934](https://arxiv.org/abs/2305.16934) [find](https://scholar.google.com/scholar?q=Yunqing+Zhao%2C+Tianyu+Pang%2C+Chao+Du%2C+Xiao+Yang%2C+Chongxuan+Li%2C+Ngai%2DMan+Cheung%2C+and+Min+Lin%2E+On+evaluating+adversarial+robustness+of+large+vision%2Dlanguage+models%2E+arXiv+preprint+arXiv%3A2305%2E16934%2C+2023%2E+)
60. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. 10 [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric%2E+P+Xing%2C+Hao+Zhang%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+Judging+llm%2Das%2Da%2Djud)
61. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023. 2, 8 [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Pengfei+Liu%2C+Puxin+Xu%2C+Srini+Iyer%2C+Jiao+Sun%2C+Yuning+Mao%2C+Xuezhe+Ma%2C+Avia+Efrat%2C+Ping+Yu%2C+Lili+Yu%2C+et+al%2E+Lima%3A+Less+is+more+for+alignment%2E+arXiv+preprint+arXiv%3A2305%2E11206%2C+2023%2E+2%2C+8)
62. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 2 [find](https://scholar.google.com/scholar?q=Deyao+Zhu%2C+Jun+Chen%2C+Xiaoqian+Shen%2C+Xiang+Li%2C+and+Mohamed+Elhoseiny%2E+Minigpt%2D4%3A+Enhancing+vision%2Dlanguage+understanding+with+advanced+large+language+models%2E+arXiv+preprint+arXiv%3A2304%2E10592%2C+2023%2E+1%2C+2) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/)