--- title: "MM-Vet: Evaluating Large Multimodal Models for — Yu 2023" description: "Classic vision-language benchmarks such as VQA v2, COCO captioning, and TextVQA each probe only one or two skills." authors: ["Weihao Yu", "Zhengyuan Yang", "Linjie Li", "Jianfeng Wang", "Kevin Lin", "Zicheng Liu", "Xinchao Wang", "Lijuan Wang"] year: 2023 venue: "arXiv:2308.02490" source: "https://arxiv.org/abs/2308.02490" doi: "10.48550/arxiv.2308.02490" arxiv: "2308.02490" licence: "arXiv — non-exclusive license to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/) / MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities # MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities **Authors:** [Weihao Yu](https://md.research.elamur.ai/free/en/authors/weihao-yu/), [Zhengyuan Yang](https://md.research.elamur.ai/free/en/authors/zhengyuan-yang/), [Linjie Li](https://md.research.elamur.ai/free/en/authors/linjie-li/), [Jianfeng Wang](https://md.research.elamur.ai/free/en/authors/jianfeng-wang/), [Kevin Lin](https://md.research.elamur.ai/free/en/authors/kevin-lin/), [Zicheng Liu](https://md.research.elamur.ai/free/en/authors/zicheng-liu/), [Xinchao Wang](https://md.research.elamur.ai/free/en/authors/xinchao-wang/), [Lijuan Wang](https://md.research.elamur.ai/free/en/authors/lijuan-wang/) · **Year:** 2023 · **Venue:** arXiv:2308.02490 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2308.02490) · [PDF](https://arxiv.org/pdf/2308.02490) · [DOI 10.48550/arxiv.2308.02490](https://doi.org/10.48550/arxiv.2308.02490) · [arXiv 2308.02490](https://arxiv.org/abs/2308.02490) · [OpenAlex](https://openalex.org/W4385645323) **Citations:** 58 citations (openalex-snapshot · 2026-10-14) · [270](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Classic vision-language benchmarks such as VQA v2, COCO captioning, and TextVQA each probe only one or two skills. That leaves a gap for testing large multimodal models (LMMs) on demanding tasks like explaining memes or solving arithmetic written on a board. MM-Vet fills this gap by framing complex tasks as mixtures of six core abilities: math, spatial awareness, text generation, world knowledge, reading text in images (OCR), and visual recognition. The benchmark covers 16 combinations of these abilities. Because answers can range from a single word to a long essay, the authors score them with GPT-4 rather than exact matching. A few-shot prompt guides GPT-4 to give each response a score between 0 and 1, based on the question, the ground truth, and the model output. The authors benchmark end-to-end LMMs (LLaVA, MiniGPT-4, InstructBLIP, Otter, OpenFlamingo), tool-using agents (MM-ReAct, Transformers Agent), and commercial systems (GPT-4V, Bard). Even GPT-4V scores only about 68%, so integrated multimodal reasoning remains far from solved. The per-capability breakdown also shows more than a single leaderboard number would. ## Key points - **Capability coverage:** Recognition appears in 68.8% of samples, OCR in 44.0%, knowledge in 38.5%, generation in 36.7%, spatial awareness in 34.4%, and math in 11.9%. Most items require several capabilities at once. - **Evaluator reliability:** The authors compared evaluator scores with human scores on 138 objective questions answered by MM-ReAct-GPT-4. GPT-4 (0613) alone deviated from human scores by an average of 0.0423. Pairing it with Llama-2, Mistral, Gemini Pro, or Claude 3 Opus increased this error. - **Prompt ablation:** Using all seven few-shot grading examples gave the closest agreement with human judgments. - **Model rankings:** GPT-4V ranked first, MM-ReAct-GPT-4 second, and LLaVA-13B led the end-to-end open models. A stronger underlying LLM clearly helped, while the effect of the choice of vision encoder was unclear. - **Availability:** Code and data are at github.com/yuweihao/MM-Vet, with an online evaluator on Hugging Face Spaces. The paper was published at ICML 2024. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) Topics: [Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) ## Cited by Cited by 270 papers of our library cite it (of the 140884 whose bibliographies we read); 33 of them have a page here, 237 not yet. - [A Survey on Evaluation of Large Language Models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/) - [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/) - [Improved Baselines with Visual Instruction Tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/) - [Qwen2.5-VL Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen2-5-vl-technical/) - [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/) - [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://md.research.elamur.ai/free/en/papers/wang-2024-qwen2-vl-enhancing-vision/) - [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://md.research.elamur.ai/free/en/papers/lin-2023-video-llava-learning-united-visual/) - [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://md.research.elamur.ai/free/en/papers/chen-2024-far-gpt-4v-closing/) - [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://md.research.elamur.ai/free/en/papers/chen-2023-sharegpt4v-improving-large-multi-modal/) - [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://md.research.elamur.ai/free/en/papers/li-2023-mvbench-comprehensive-multi-modal-video/) - [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/) - [LLaVA-OneVision: Easy Visual Task Transfer](https://md.research.elamur.ai/free/en/papers/li-2024-llava-onevision-easy-visual-task/) - [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://md.research.elamur.ai/free/en/papers/fu-2024-video-mme-first-ever-comprehensive/) - [CogVLM: Visual Expert for Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/wang-2023-cogvlm-visual-expert-pretrained/) - [VILA: On Pre-training for Visual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2023-vila-pre-training-visual-language/) - [CogAgent: A Visual Language Model for GUI Agents](https://md.research.elamur.ai/free/en/papers/hong-2023-cogagent-visual-language-model/) - [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/guan-2023-hallusionbench-advanced-diagnostic-suite-entangled/) - [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://md.research.elamur.ai/free/en/papers/chen-2024-expanding-performance-boundaries-open/) - [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/chen-2024-image-worth-1-2-tokens/) - [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://md.research.elamur.ai/free/en/papers/chen-2024-right-way-evaluating-large-vision/) - [Qwen2.5-Omni Technical Report](https://md.research.elamur.ai/free/en/papers/xu-2025-qwen2-5-omni-technical/) - [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://md.research.elamur.ai/free/en/papers/lu-2024-deepseek-vl-real-world/) - [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://md.research.elamur.ai/free/en/papers/zhu-2025-internvl3-exploring-advanced-training/) - [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://md.research.elamur.ai/free/en/papers/wang-2025-internvl3-5-advancing-open/) - [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://md.research.elamur.ai/free/en/papers/zhu-2023-judgelm-fine-tuned-large-language/) - [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl2-revolutionizing-multi-modal/) - [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://md.research.elamur.ai/free/en/papers/cheng-2024-seeclick-harnessing-gui-grounding-advanced/) - [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://md.research.elamur.ai/free/en/papers/chen-2025-janus-pro-unified-multimodal/) - [InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model](https://md.research.elamur.ai/free/en/papers/dong-2024-internlm-xcomposer2-mastering-free/) - [OASIS: A Multilingual and Multimodal Dataset for Culturally Grounded Spoken Visual QA](https://md.research.elamur.ai/free/en/papers/alam-2025-oasis-multilingual-multimodal-dataset-culturally/) - [Multi-modal, Multi-task, Multi-criteria Automatic Evaluation with Vision Language Models](https://md.research.elamur.ai/free/en/papers/oi-2026-multi-modal-multi-task-multi/) - [Evaluating Discriminability of Vision-Language Models](https://md.research.elamur.ai/free/en/papers/muraoka-2026-evaluating-discriminability-vision-language-models/) - [STRUDEL: Unrolling a Benchmark for Evaluating Vision-Language Models on Structured Diagram Understanding across Domains](https://md.research.elamur.ai/free/en/papers/steinigen-2026-strudel-unrolling-benchmark-evaluating-vision/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{yu2023mm, title = {{MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities}}, author = {Yu, Weihao and Yang, Zhengyuan and Li, Linjie and Wang, Jianfeng and Lin, Kevin and Liu, Zicheng and Wang, Xinchao and Wang, Lijuan}, year = {2023}, doi = {10.48550/arxiv.2308.02490}, eprint = {2308.02490}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2308.02490} } ``` - [← previous Scaling Speech Technology to 1,000+ Languages](https://md.research.elamur.ai/free/en/papers/pratap-2023-scaling-speech-technology-1-000/) - [next → Textbooks Are All You Need II: phi-1.5 technical report](https://md.research.elamur.ai/free/en/papers/li-2023-textbooks-all-need-ii-phi/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/) ## References 94 references, 38 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Mpt. [https://github.com/mosaicml/llm-foundry\#mpt](https://github.com/mosaicml/llm-foundry\#mpt), 2023. [↗](https://github.com/mosaicml/llm-foundry\#mpt) 2. Chatgpt can now see, hear, and speak. [https://openai.com/blog/chatgpt-can-now-see-hear-and-speak](https://openai.com/blog/chatgpt-can-now-see-hear-and-speak), 2023. [↗](https://openai.com/blog/chatgpt-can-now-see-hear-and-speak) 3. Agrawal, H., Desai, K., Wang, Y., Chen, X., Jain, R., Johnson, M., Batra, D., Parikh, D., Lee, S., and Anderson, P. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8948–8957, 2019. [find](https://scholar.google.com/scholar?q=Agrawal%2C+H%2E%2C+Desai%2C+K%2E%2C+Wang%2C+Y%2E%2C+Chen%2C+X%2E%2C+Jain%2C+R%2E%2C+Johnson%2C+M%2E%2C+Batra%2C+D%2E%2C+Parikh%2C+D%2E%2C+Lee%2C+S%2E%2C+and+Anderson%2C+P%2E+Nocaps%3A+Novel+object+captioning+at+scale%2E+In+Proceedings+of+the+IEEE%2FCVF+internation) 4. Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–23736, 2022. [find](https://scholar.google.com/scholar?q=Alayrac%2C+J%2E%2DB%2E%2C+Donahue%2C+J%2E%2C+Luc%2C+P%2E%2C+Miech%2C+A%2E%2C+Barr%2C+I%2E%2C+Hasson%2C+Y%2E%2C+Lenc%2C+K%2E%2C+Mensch%2C+A%2E%2C+Millican%2C+K%2E%2C+Reynolds%2C+M%2E%2C+et+al%2E+Flamingo%3A+a+visual+language+model+for+few%2Dshot+learning%2E+Advances+in+Neu) [our card](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/) 5. Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. [find](https://scholar.google.com/scholar?q=Anil%2C+R%2E%2C+Dai%2C+A%2E+M%2E%2C+Firat%2C+O%2E%2C+Johnson%2C+M%2E%2C+Lepikhin%2C+D%2E%2C+Passos%2C+A%2E%2C+Shakeri%2C+S%2E%2C+Taropa%2C+E%2E%2C+Bailey%2C+P%2E%2C+Chen%2C+Z%2E%2C+et+al%2E+Palm+2+technical+report%2E+arXiv+preprint+arXiv%3A2305%2E10403%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/) 6. Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D. VQA: Visual Question Answering. In ICCV, 2015. [find](https://scholar.google.com/scholar?q=Antol%2C+S%2E%2C+Agrawal%2C+A%2E%2C+Lu%2C+J%2E%2C+Mitchell%2C+M%2E%2C+Batra%2C+D%2E%2C+Zitnick%2C+C%2E+L%2E%2C+and+Parikh%2C+D%2E+VQA%3A+Visual+Question+Answering%2E+In+ICCV%2C+2015%2E) 7. Ao, J., Wang, R., Zhou, L., Wang, C., Ren, S., Wu, Y., Liu, S., Ko, T., Li, Q., Zhang, Y., et al. Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing. arXiv preprint arXiv:2110.07205, 2021. [find](https://scholar.google.com/scholar?q=Ao%2C+J%2E%2C+Wang%2C+R%2E%2C+Zhou%2C+L%2E%2C+Wang%2C+C%2E%2C+Ren%2C+S%2E%2C+Wu%2C+Y%2E%2C+Liu%2C+S%2E%2C+Ko%2C+T%2E%2C+Li%2C+Q%2E%2C+Zhang%2C+Y%2E%2C+et+al%2E+Speecht5%3A+Unified%2Dmodal+encoder%2Ddecoder+pre%2Dtraining+for+spoken+language+processing%2E+arXiv+preprint+ar) 8. Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S., Jitsev, J., Kornblith, S., Koh, P. W., Ilharco, G., Wortsman, M., and Schmidt, L. Openflamingo, March 2023a. URL https://doi.org/10.5281/zenodo.7733589. — [doi:10.5281/zenodo.7733589](https://doi.org/10.5281/zenodo.7733589) [↗](https://doi.org/10.5281/zenodo.7733589) 9. Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S., Sagawa, S., Jitsev, J., Kornblith, S., Koh, P. W., Ilharco, G., Wortsman, M., and Schmidt, L. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXi… [find](https://scholar.google.com/scholar?q=Awadalla%2C+A%2E%2C+Gao%2C+I%2E%2C+Gardner%2C+J%2E%2C+Hessel%2C+J%2E%2C+Hanafy%2C+Y%2E%2C+Zhu%2C+W%2E%2C+Marathe%2C+K%2E%2C+Bitton%2C+Y%2E%2C+Gadre%2C+S%2E%2C+Sagawa%2C+S%2E%2C+Jitsev%2C+J%2E%2C+Kornblith%2C+S%2E%2C+Koh%2C+P%2E+W%2E%2C+Ilharco%2C+G%2E%2C+Wortsman%2C+M%2E%2C+and+Schmidt%2C+L%2E+O) [our card](https://md.research.elamur.ai/free/en/papers/awadalla-2023-openflamingo-open-source-framework-training/) 10. Azure, M. Azure cognitive services apis. [https://azure.microsoft.com/en-us/products/](https://azure.microsoft.com/en-us/products/) ai-services/ai-vision, 2023. Accessed: 2023-06-20. [↗](https://azure.microsoft.com/en-us/products/) 11. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901, 2020a. [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E%2C+Mann%2C+B%2E%2C+Ryder%2C+N%2E%2C+Subbiah%2C+M%2E%2C+Kaplan%2C+J%2E+D%2E%2C+Dhariwal%2C+P%2E%2C+Neelakantan%2C+A%2E%2C+Shyam%2C+P%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+Advances+in+neural+informati) 12. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In NeurIPS, 2020b. [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E+B%2E%2C+Mann%2C+B%2E%2C+Ryder%2C+N%2E%2C+Subbiah%2C+M%2E%2C+Kaplan%2C+J%2E%2C+Dhariwal%2C+P%2E%2C+Neelakantan%2C+A%2E%2C+Shyam%2C+P%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+In+NeurIPS%2C+2020b%2E) 13. Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. [find](https://scholar.google.com/scholar?q=Bubeck%2C+S%2E%2C+Chandrasekaran%2C+V%2E%2C+Eldan%2C+R%2E%2C+Gehrke%2C+J%2E%2C+Horvitz%2C+E%2E%2C+Kamar%2C+E%2E%2C+Lee%2C+P%2E%2C+Lee%2C+Y%2E+T%2E%2C+Li%2C+Y%2E%2C+Lundberg%2C+S%2E%2C+et+al%2E+Sparks+of+artificial+general+intelligence%3A+Early+experiments+with+gpt%2D4) [our card](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/) 14. Byeon, M., Park, B., Kim, H., Lee, S., Baek, W., and Kim, S. Coyo-700m: Image-text pair dataset. [https://github.com/kakaobrain/coyo-dataset](https://github.com/kakaobrain/coyo-dataset), 2022. [↗](https://github.com/kakaobrain/coyo-dataset) 15. Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3558–3568, 2021. [find](https://scholar.google.com/scholar?q=Changpinyo%2C+S%2E%2C+Sharma%2C+P%2E%2C+Ding%2C+N%2E%2C+and+Soricut%2C+R%2E+Conceptual+12m%3A+Pushing+web%2Dscale+image%2Dtext+pre%2Dtraining+to+recognize+long%2Dtail+visual+concepts%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Com) 16. Chen, T., Saxena, S., Li, L., Fleet, D. J., and Hinton, G. Pix2seq: A language modeling framework for object detection. In ICLR, 2022. [find](https://scholar.google.com/scholar?q=Chen%2C+T%2E%2C+Saxena%2C+S%2E%2C+Li%2C+L%2E%2C+Fleet%2C+D%2E+J%2E%2C+and+Hinton%2C+G%2E+Pix2seq%3A+A+language+modeling+framework+for+object+detection%2E+In+ICLR%2C+2022%2E) 17. Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollár, P., and Zitnick, C. L. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. [find](https://scholar.google.com/scholar?q=Chen%2C+X%2E%2C+Fang%2C+H%2E%2C+Lin%2C+T%2E%2DY%2E%2C+Vedantam%2C+R%2E%2C+Gupta%2C+S%2E%2C+Doll%C3%A1r%2C+P%2E%2C+and+Zitnick%2C+C%2E+L%2E+Microsoft+coco+captions%3A+Data+collection+and+evaluation+server%2E+arXiv+preprint+arXiv%3A1504%2E00325%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2015-microsoft-coco-captions-data-collection/) 18. Chen, Y.-C., Li, L., Yu, L., Kholy, A. E., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J. Uniter: Learning universal image-text representations. In ECCV, 2020. [find](https://scholar.google.com/scholar?q=Chen%2C+Y%2E%2DC%2E%2C+Li%2C+L%2E%2C+Yu%2C+L%2E%2C+Kholy%2C+A%2E+E%2E%2C+Ahmed%2C+F%2E%2C+Gan%2C+Z%2E%2C+Cheng%2C+Y%2E%2C+and+Liu%2C+J%2E+Uniter%3A+Learning+universal+image%2Dtext+representations%2E+In+ECCV%2C+2020%2E) 19. Chiang, C.-H. and Lee, H.-y. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937, 2023. [find](https://scholar.google.com/scholar?q=Chiang%2C+C%2E%2DH%2E+and+Lee%2C+H%2E%2Dy%2E+Can+large+language+models+be+an+alternative+to+human+evaluations%3F+arXiv+preprint+arXiv%3A2305%2E01937%2C+2023%2E) 20. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. [find](https://scholar.google.com/scholar?q=Chowdhery%2C+A%2E%2C+Narang%2C+S%2E%2C+Devlin%2C+J%2E%2C+Bosma%2C+M%2E%2C+Mishra%2C+G%2E%2C+Roberts%2C+A%2E%2C+Barham%2C+P%2E%2C+Chung%2C+H%2E+W%2E%2C+Sutton%2C+C%2E%2C+Gehrmann%2C+S%2E%2C+et+al%2E+Palm%3A+Scaling+language+modeling+with+pathways%2E+arXiv+preprint+arXi) 21. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. [find](https://scholar.google.com/scholar?q=Chung%2C+H%2E+W%2E%2C+Hou%2C+L%2E%2C+Longpre%2C+S%2E%2C+Zoph%2C+B%2E%2C+Tay%2C+Y%2E%2C+Fedus%2C+W%2E%2C+Li%2C+E%2E%2C+Wang%2C+X%2E%2C+Dehghani%2C+M%2E%2C+Brahma%2C+S%2E%2C+et+al%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXiv+preprint+arXiv%3A2210%2E11416%2C+202) 22. Costa-jussà, M. R., Cross, J., Çelebi, O., Elbayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., Licht, D., Maillard, J., et al. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672, 2022. [find](https://scholar.google.com/scholar?q=Costa%2Djuss%C3%A0%2C+M%2E+R%2E%2C+Cross%2C+J%2E%2C+%C3%87elebi%2C+O%2E%2C+Elbayad%2C+M%2E%2C+Heafield%2C+K%2E%2C+Heffernan%2C+K%2E%2C+Kalbassi%2C+E%2E%2C+Lam%2C+J%2E%2C+Licht%2C+D%2E%2C+Maillard%2C+J%2E%2C+et+al%2E+No+language+left+behind%3A+Scaling+human%2Dcentered+machine+tran) 23. Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500, 2023. [find](https://scholar.google.com/scholar?q=Dai%2C+W%2E%2C+Li%2C+J%2E%2C+Li%2C+D%2E%2C+Tiong%2C+A%2E+M%2E+H%2E%2C+Zhao%2C+J%2E%2C+Wang%2C+W%2E%2C+Li%2C+B%2E%2C+Fung%2C+P%2E%2C+and+Hoi%2C+S%2E+Instructblip%3A+Towards+general%2Dpurpose+vision%2Dlanguage+models+with+instruction+tuning%2E+arXiv+preprint+arXiv%3A2) [our card](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/) 24. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. [find](https://scholar.google.com/scholar?q=Dosovitskiy%2C+A%2E%2C+Beyer%2C+L%2E%2C+Kolesnikov%2C+A%2E%2C+Weissenborn%2C+D%2E%2C+Zhai%2C+X%2E%2C+Unterthiner%2C+T%2E%2C+Dehghani%2C+M%2E%2C+Minderer%2C+M%2E%2C+Heigold%2C+G%2E%2C+Gelly%2C+S%2E%2C+et+al%2E+An+image+is+worth+16x16+words%3A+Transformers+for+image) 25. Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., Huang, W., Chebotar, Y., Sermanet, P., Duckworth, D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., Zeng, A., Mordatch, I., and Florence, P. Palm-e: An embodie… [find](https://scholar.google.com/scholar?q=Driess%2C+D%2E%2C+Xia%2C+F%2E%2C+Sajjadi%2C+M%2E+S%2E+M%2E%2C+Lynch%2C+C%2E%2C+Chowdhery%2C+A%2E%2C+Ichter%2C+B%2E%2C+Wahid%2C+A%2E%2C+Tompson%2C+J%2E%2C+Vuong%2C+Q%2E%2C+Yu%2C+T%2E%2C+Huang%2C+W%2E%2C+Chebotar%2C+Y%2E%2C+Sermanet%2C+P%2E%2C+Duckworth%2C+D%2E%2C+Levine%2C+S%2E%2C+Vanhoucke%2C+V%2E) 26. Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., and Cao, Y. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19358–19369, 2023. [find](https://scholar.google.com/scholar?q=Fang%2C+Y%2E%2C+Wang%2C+W%2E%2C+Xie%2C+B%2E%2C+Sun%2C+Q%2E%2C+Wu%2C+L%2E%2C+Wang%2C+X%2E%2C+Huang%2C+T%2E%2C+Wang%2C+X%2E%2C+and+Cao%2C+Y%2E+Eva%3A+Exploring+the+limits+of+masked+visual+representation+learning+at+scale%2E+In+Proceedings+of+the+IEEE%2FCVF+Con) 27. Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Qiu, Z., Lin, W., Yang, J., Zheng, X., et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023a. [find](https://scholar.google.com/scholar?q=Fu%2C+C%2E%2C+Chen%2C+P%2E%2C+Shen%2C+Y%2E%2C+Qin%2C+Y%2E%2C+Zhang%2C+M%2E%2C+Lin%2C+X%2E%2C+Qiu%2C+Z%2E%2C+Lin%2C+W%2E%2C+Yang%2C+J%2E%2C+Zheng%2C+X%2E%2C+et+al%2E+Mme%3A+A+comprehensive+evaluation+benchmark+for+multimodal+large+language+models%2E+arXiv+preprint+ar) [our card](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/) 28. Fu, J., Ng, S.-K., Jiang, Z., and Liu, P. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166, 2023b. [find](https://scholar.google.com/scholar?q=Fu%2C+J%2E%2C+Ng%2C+S%2E%2DK%2E%2C+Jiang%2C+Z%2E%2C+and+Liu%2C+P%2E+Gptscore%3A+Evaluate+as+you+desire%2E+arXiv+preprint+arXiv%3A2302%2E04166%2C+2023b%2E) 29. Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., and Gao, J. Vision-language pre-training: Basics, recent advances, and future trends. arXiv preprint arXiv:2210.09263, 2022. — [arXiv:2210.09263](https://arxiv.org/abs/2210.09263) [find](https://scholar.google.com/scholar?q=Gan%2C+Z%2E%2C+Li%2C+L%2E%2C+Li%2C+C%2E%2C+Wang%2C+L%2E%2C+Liu%2C+Z%2E%2C+and+Gao%2C+J%2E+Vision%2Dlanguage+pre%2Dtraining%3A+Basics%2C+recent+advances%2C+and+future+trends%2E+arXiv+preprint+arXiv%3A2210%2E09263%2C+2022%2E+%E2%80%94+%5BarXiv%3A2210%2E09263%5D%28https%3A%2F%2Far) 30. Gao, D., Ji, L., Zhou, L., Lin, K. Q., Chen, J., Fan, Z., and Shou, M. Z. Assistgpt: A general multi-modal assistant that can plan, execute, inspect, and learn. arXiv preprint arXiv:2306.08640, 2023a. — [arXiv:2306.08640](https://arxiv.org/abs/2306.08640) [find](https://scholar.google.com/scholar?q=Gao%2C+D%2E%2C+Ji%2C+L%2E%2C+Zhou%2C+L%2E%2C+Lin%2C+K%2E+Q%2E%2C+Chen%2C+J%2E%2C+Fan%2C+Z%2E%2C+and+Shou%2C+M%2E+Z%2E+Assistgpt%3A+A+general+multi%2Dmodal+assistant+that+can+plan%2C+execute%2C+inspect%2C+and+learn%2E+arXiv+preprint+arXiv%3A2306%2E08640%2C+2023a%2E) 31. Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. Pal: Program-aided language models. arXiv preprint arXiv:2211.10435, 2022. [find](https://scholar.google.com/scholar?q=Gao%2C+L%2E%2C+Madaan%2C+A%2E%2C+Zhou%2C+S%2E%2C+Alon%2C+U%2E%2C+Liu%2C+P%2E%2C+Yang%2C+Y%2E%2C+Callan%2C+J%2E%2C+and+Neubig%2C+G%2E+Pal%3A+Program%2Daided+language+models%2E+arXiv+preprint+arXiv%3A2211%2E10435%2C+2022%2E) 32. Gao, P., Han, J., Zhang, R., Lin, Z., Geng, S., Zhou, A., Zhang, W., Lu, P., He, C., Yue, X., et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010, 2023b. [find](https://scholar.google.com/scholar?q=Gao%2C+P%2E%2C+Han%2C+J%2E%2C+Zhang%2C+R%2E%2C+Lin%2C+Z%2E%2C+Geng%2C+S%2E%2C+Zhou%2C+A%2E%2C+Zhang%2C+W%2E%2C+Lu%2C+P%2E%2C+He%2C+C%2E%2C+Yue%2C+X%2E%2C+et+al%2E+Llama%2Dadapter+v2%3A+Parameter%2Defficient+visual+instruction+model%2E+arXiv+preprint+arXiv%3A2304%2E15010%2C+20) 33. Gong, T., Lyu, C., Zhang, S., Wang, Y., Zheng, M., Zhao, Q., Liu, K., Zhang, W., Luo, P., and Chen, K. Multimodal-gpt: A vision and language model for dialogue with humans, 2023. [find](https://scholar.google.com/scholar?q=Gong%2C+T%2E%2C+Lyu%2C+C%2E%2C+Zhang%2C+S%2E%2C+Wang%2C+Y%2E%2C+Zheng%2C+M%2E%2C+Zhao%2C+Q%2E%2C+Liu%2C+K%2E%2C+Zhang%2C+W%2E%2C+Luo%2C+P%2E%2C+and+Chen%2C+K%2E+Multimodal%2Dgpt%3A+A+vision+and+language+model+for+dialogue+with+humans%2C+2023%2E) 34. Google. Bard. [https://bard.google.com](https://bard.google.com), 2023. Accessed: 2023-07-17. [↗](https://bard.google.com) 35. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904–6913, 2017. [find](https://scholar.google.com/scholar?q=Goyal%2C+Y%2E%2C+Khot%2C+T%2E%2C+Summers%2DStay%2C+D%2E%2C+Batra%2C+D%2E%2C+and+Parikh%2C+D%2E+Making+the+v+in+vqa+matter%3A+Elevating+the+role+of+image+understanding+in+visual+question+answering%2E+In+Proceedings+of+the+IEEE+conferen) [our card](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/) 36. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. [find](https://scholar.google.com/scholar?q=Hoffmann%2C+J%2E%2C+Borgeaud%2C+S%2E%2C+Mensch%2C+A%2E%2C+Buchatskaya%2C+E%2E%2C+Cai%2C+T%2E%2C+Rutherford%2C+E%2E%2C+Casas%2C+D%2E+d%2E+L%2E%2C+Hendricks%2C+L%2E+A%2E%2C+Welbl%2C+J%2E%2C+Clark%2C+A%2E%2C+et+al%2E+Training+compute%2Doptimal+large+language+models%2E+arXiv+) 37. Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. [find](https://scholar.google.com/scholar?q=Hudson%2C+D%2E+A%2E+and+Manning%2C+C%2E+D%2E+Gqa%3A+A+new+dataset+for+real%2Dworld+visual+reasoning+and+compositional+question+answering%2E+In+CVPR%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/) 38. Huggingface. Transformers agent. [https://huggingface.co/docs/transformers/](https://huggingface.co/docs/transformers/) transformers_agents, 2023. Accessed: 2023-07-20. [↗](https://huggingface.co/docs/transformers/) 39. Kim, G., Hong, T., Yim, M., Nam, J., Park, J., Yim, J., Hwang, W., Yun, S., Han, D., and Park, S. Ocr-free document understanding transformer. In European Conference on Computer Vision, pp. 498–517. Springer, 2022. [find](https://scholar.google.com/scholar?q=Kim%2C+G%2E%2C+Hong%2C+T%2E%2C+Yim%2C+M%2E%2C+Nam%2C+J%2E%2C+Park%2C+J%2E%2C+Yim%2C+J%2E%2C+Hwang%2C+W%2E%2C+Yun%2C+S%2E%2C+Han%2C+D%2E%2C+and+Park%2C+S%2E+Ocr%2Dfree+document+understanding+transformer%2E+In+European+Conference+on+Computer+Vision%2C+pp%2E+498%E2%80%93517%2E+S) [our card](https://md.research.elamur.ai/free/en/papers/kim-2021-ocr-free-document-understanding-transformer/) 40. Kim, W., Son, B., and Kim, I. Vilt: Vision-and-language transformer without convolution or region supervision. In ICML, 2021. [find](https://scholar.google.com/scholar?q=Kim%2C+W%2E%2C+Son%2C+B%2E%2C+and+Kim%2C+I%2E+Vilt%3A+Vision%2Dand%2Dlanguage+transformer+without+convolution+or+region+supervision%2E+In+ICML%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2021-vilt-vision-language-transformer-convolution/) 41. Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017. [find](https://scholar.google.com/scholar?q=Krishna%2C+R%2E%2C+Zhu%2C+Y%2E%2C+Groth%2C+O%2E%2C+Johnson%2C+J%2E%2C+Hata%2C+K%2E%2C+Kravitz%2C+J%2E%2C+Chen%2C+S%2E%2C+Kalantidis%2C+Y%2E%2C+Li%2C+L%2E%2DJ%2E%2C+Shamma%2C+D%2E+A%2E%2C+et+al%2E+Visual+genome%3A+Connecting+language+and+vision+using+crowdsourced+dense+i) [our card](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/) 42. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019. [find](https://scholar.google.com/scholar?q=Lewis%2C+M%2E%2C+Liu%2C+Y%2E%2C+Goyal%2C+N%2E%2C+Ghazvininejad%2C+M%2E%2C+Mohamed%2C+A%2E%2C+Levy%2C+O%2E%2C+Stoyanov%2C+V%2E%2C+and+Zettlemoyer%2C+L%2E+Bart%3A+Denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+language+generation%2C+translatio) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/) 43. Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y., and Shan, Y. Seed-bench: Benchmarking multimodal llms with generative comprehension, 2023a. [find](https://scholar.google.com/scholar?q=Li%2C+B%2E%2C+Wang%2C+R%2E%2C+Wang%2C+G%2E%2C+Ge%2C+Y%2E%2C+Ge%2C+Y%2E%2C+and+Shan%2C+Y%2E+Seed%2Dbench%3A+Benchmarking+multimodal+llms+with+generative+comprehension%2C+2023a%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-seed-bench-benchmarking-multimodal-llms/) 44. Li, B., Zhang, Y., Chen, L., Wang, J., Pu, F., Yang, J., Li, C., and Liu, Z. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023b. — [arXiv:2306.05425](https://arxiv.org/abs/2306.05425) [find](https://scholar.google.com/scholar?q=Li%2C+B%2E%2C+Zhang%2C+Y%2E%2C+Chen%2C+L%2E%2C+Wang%2C+J%2E%2C+Pu%2C+F%2E%2C+Yang%2C+J%2E%2C+Li%2C+C%2E%2C+and+Liu%2C+Z%2E+Mimic%2Dit%3A+Multi%2Dmodal+in%2Dcontext+instruction+tuning%2E+arXiv+preprint+arXiv%3A2306%2E05425%2C+2023b%2E+%E2%80%94+%5BarXiv%3A2306%2E05425%5D%28https%3A%2F%2Fa) 45. Li, B., Zhang, Y., Chen, L., Wang, J., Yang, J., and Liu, Z. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023c. [find](https://scholar.google.com/scholar?q=Li%2C+B%2E%2C+Zhang%2C+Y%2E%2C+Chen%2C+L%2E%2C+Wang%2C+J%2E%2C+Yang%2C+J%2E%2C+and+Liu%2C+Z%2E+Otter%3A+A+multi%2Dmodal+model+with+in%2Dcontext+instruction+tuning%2E+arXiv+preprint+arXiv%3A2305%2E03726%2C+2023c%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-otter-multi-modal-model-context/) 46. Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp. 12888–12900. PMLR, 2022. [find](https://scholar.google.com/scholar?q=Li%2C+J%2E%2C+Li%2C+D%2E%2C+Xiong%2C+C%2E%2C+and+Hoi%2C+S%2E+Blip%3A+Bootstrapping+language%2Dimage+pre%2Dtraining+for+unified+vision%2Dlanguage+understanding+and+generation%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+12) [our card](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/) 47. Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023d. [find](https://scholar.google.com/scholar?q=Li%2C+J%2E%2C+Li%2C+D%2E%2C+Savarese%2C+S%2E%2C+and+Hoi%2C+S%2E+Blip%2D2%3A+Bootstrapping+language%2Dimage+pre%2Dtraining+with+frozen+image+encoders+and+large+language+models%2E+arXiv+preprint+arXiv%3A2301%2E12597%2C+2023d%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/) 48. Li, X., Yin, X., Li, C., Hu, X., Zhang, P., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV, 2020. [find](https://scholar.google.com/scholar?q=Li%2C+X%2E%2C+Yin%2C+X%2E%2C+Li%2C+C%2E%2C+Hu%2C+X%2E%2C+Zhang%2C+P%2E%2C+Zhang%2C+L%2E%2C+Wang%2C+L%2E%2C+Hu%2C+H%2E%2C+Dong%2C+L%2E%2C+Wei%2C+F%2E%2C+et+al%2E+Oscar%3A+Object%2Dsemantics+aligned+pre%2Dtraining+for+vision%2Dlanguage+tasks%2E+In+ECCV%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2020-oscar-object-semantics-aligned-pre/) 49. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In ECCV, 2014. [find](https://scholar.google.com/scholar?q=Lin%2C+T%2E%2DY%2E%2C+Maire%2C+M%2E%2C+Belongie%2C+S%2E%2C+Hays%2C+J%2E%2C+Perona%2C+P%2E%2C+Ramanan%2C+D%2E%2C+Doll%C3%A1r%2C+P%2E%2C+and+Zitnick%2C+C%2E+L%2E+Microsoft+coco%3A+Common+objects+in+context%2E+In+ECCV%2C+2014%2E) [our card](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/) 50. Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., and Wang, L. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.14565, 2023a. [find](https://scholar.google.com/scholar?q=Liu%2C+F%2E%2C+Lin%2C+K%2E%2C+Li%2C+L%2E%2C+Wang%2C+J%2E%2C+Yacoob%2C+Y%2E%2C+and+Wang%2C+L%2E+Aligning+large+multi%2Dmodal+model+with+robust+instruction+tuning%2E+arXiv+preprint+arXiv%3A2306%2E14565%2C+2023a%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-mitigating-hallucination-large-multi-modal/) 51. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023b. [find](https://scholar.google.com/scholar?q=Liu%2C+H%2E%2C+Li%2C+C%2E%2C+Wu%2C+Q%2E%2C+and+Lee%2C+Y%2E+J%2E+Visual+instruction+tuning%2E+arXiv+preprint+arXiv%3A2304%2E08485%2C+2023b%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/) 52. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023c. [find](https://scholar.google.com/scholar?q=Liu%2C+H%2E%2C+Li%2C+C%2E%2C+Wu%2C+Q%2E%2C+and+Lee%2C+Y%2E+J%2E+Visual+instruction+tuning%2E+arXiv+preprint+arXiv%3A2304%2E08485%2C+2023c%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/) 53. Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023d. [find](https://scholar.google.com/scholar?q=Liu%2C+Y%2E%2C+Duan%2C+H%2E%2C+Zhang%2C+Y%2E%2C+Li%2C+B%2E%2C+Zhang%2C+S%2E%2C+Zhao%2C+W%2E%2C+Yuan%2C+Y%2E%2C+Wang%2C+J%2E%2C+He%2C+C%2E%2C+Liu%2C+Z%2E%2C+et+al%2E+Mmbench%3A+Is+your+multi%2Dmodal+model+an+all%2Daround+player%3F+arXiv+preprint+arXiv%3A2307%2E06281%2C+2023d%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/) 54. Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C. Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634, 2023e. [find](https://scholar.google.com/scholar?q=Liu%2C+Y%2E%2C+Iter%2C+D%2E%2C+Xu%2C+Y%2E%2C+Wang%2C+S%2E%2C+Xu%2C+R%2E%2C+and+Zhu%2C+C%2E+Gpteval%3A+Nlg+evaluation+using+gpt%2D4+with+better+human+alignment%2E+arXiv+preprint+arXiv%3A2303%2E16634%2C+2023e%2E) 55. Liu, Y., Li, Z., Li, H., Yu, W., Huang, M., Peng, D., Liu, M., Chen, M., Li, C., Jin, L., et al. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07895, 2023f. [find](https://scholar.google.com/scholar?q=Liu%2C+Y%2E%2C+Li%2C+Z%2E%2C+Li%2C+H%2E%2C+Yu%2C+W%2E%2C+Huang%2C+M%2E%2C+Peng%2C+D%2E%2C+Liu%2C+M%2E%2C+Chen%2C+M%2E%2C+Li%2C+C%2E%2C+Jin%2C+L%2E%2C+et+al%2E+On+the+hidden+mystery+of+ocr+in+large+multimodal+models%2E+arXiv+preprint+arXiv%3A2305%2E07895%2C+2023f%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-ocrbench-hidden-mystery-ocr-large/) 56. Lu, J., Batra, D., Parikh, D., and Lee, S. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS, 2019. [find](https://scholar.google.com/scholar?q=Lu%2C+J%2E%2C+Batra%2C+D%2E%2C+Parikh%2C+D%2E%2C+and+Lee%2C+S%2E+Vilbert%3A+Pretraining+task%2Dagnostic+visiolinguistic+representations+for+vision%2Dand%2Dlanguage+tasks%2E+In+NeurIPS%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/lu-2019-vilbert-pretraining-task-agnostic-visiolinguistic/) 57. Lüddecke, T. and Ecker, A. Image segmentation using text and image prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7086–7096, 2022. [find](https://scholar.google.com/scholar?q=L%C3%BCddecke%2C+T%2E+and+Ecker%2C+A%2E+Image+segmentation+using+text+and+image+prompts%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+pp%2E+7086%E2%80%937096%2C+2022%2E) 58. Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR, pp. 3195–3204, 2019. [find](https://scholar.google.com/scholar?q=Marino%2C+K%2E%2C+Rastegari%2C+M%2E%2C+Farhadi%2C+A%2E%2C+and+Mottaghi%2C+R%2E+Ok%2Dvqa%3A+A+visual+question+answering+benchmark+requiring+external+knowledge%2E+In+CVPR%2C+pp%2E+3195%E2%80%933204%2C+2019%2E) 59. OpenAI. Gpt-4v(ision) system card. 2023a. URL [https://cdn.openai.com/papers/GPTV_System_Card.pdf](https://cdn.openai.com/papers/GPTV_System_Card.pdf). [↗](https://cdn.openai.com/papers/GPTV_System_Card.pdf) 60. OpenAI. Gpt-4v(ision) technical work and authors. 2023b. URL [https://cdn.openai.com/contributions/gpt-4v.pdf](https://cdn.openai.com/contributions/gpt-4v.pdf). [↗](https://cdn.openai.com/contributions/gpt-4v.pdf) 61. OpenAI. Gpt-4 technical report, 2023c. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2C+2023c%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/) 62. Ordonez, V., Kulkarni, G., and Berg, T. L. Im2text: Describing images using 1 million captioned photographs. In NeurIPS, 2011. [find](https://scholar.google.com/scholar?q=Ordonez%2C+V%2E%2C+Kulkarni%2C+G%2E%2C+and+Berg%2C+T%2E+L%2E+Im2text%3A+Describing+images+using+1+million+captioned+photographs%2E+In+NeurIPS%2C+2011%2E) 63. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022. [find](https://scholar.google.com/scholar?q=Ouyang%2C+L%2E%2C+Wu%2C+J%2E%2C+Jiang%2C+X%2E%2C+Almeida%2C+D%2E%2C+Wainwright%2C+C%2E%2C+Mishkin%2C+P%2E%2C+Zhang%2C+C%2E%2C+Agarwal%2C+S%2E%2C+Slama%2C+K%2E%2C+Ray%2C+A%2E%2C+et+al%2E+Training+language+models+to+follow+instructions+with+human+feedback%2E+Advance) 64. Peng, B., Li, C., He, P., Galley, M., and Gao, J. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023. [find](https://scholar.google.com/scholar?q=Peng%2C+B%2E%2C+Li%2C+C%2E%2C+He%2C+P%2E%2C+Galley%2C+M%2E%2C+and+Gao%2C+J%2E+Instruction+tuning+with+gpt%2D4%2E+arXiv+preprint+arXiv%3A2304%2E03277%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/peng-2023-instruction-tuning-gpt-4/) 65. Politzer, T. Vision is our dominant sense. [https://www.brainline.org/article/vision-our-dominant-sense](https://www.brainline.org/article/vision-our-dominant-sense). Accessed: 2023-05-20. [↗](https://www.brainline.org/article/vision-our-dominant-sense) 66. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Kim%2C+J%2E+W%2E%2C+Hallacy%2C+C%2E%2C+Ramesh%2C+A%2E%2C+Goh%2C+G%2E%2C+Agarwal%2C+S%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+Mishkin%2C+P%2E%2C+Clark%2C+J%2E%2C+et+al%2E+Learning+transferable+visual+models+from+natural+language+supervision%2E+ar) [our card](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/) 67. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pp. 28492–28518. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Kim%2C+J%2E+W%2E%2C+Xu%2C+T%2E%2C+Brockman%2C+G%2E%2C+McLeavey%2C+C%2E%2C+and+Sutskever%2C+I%2E+Robust+speech+recognition+via+large%2Dscale+weak+supervision%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+28492%E2%80%9328) [our card](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/) 68. Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. [find](https://scholar.google.com/scholar?q=Schuhmann%2C+C%2E%2C+Vencu%2C+R%2E%2C+Beaumont%2C+R%2E%2C+Kaczmarczyk%2C+R%2E%2C+Mullis%2C+C%2E%2C+Katta%2C+A%2E%2C+Coombes%2C+T%2E%2C+Jitsev%2C+J%2E%2C+and+Komatsuzaki%2C+A%2E+Laion%2D400m%3A+Open+dataset+of+clip%2Dfiltered+400+million+image%2Dtext+pairs%2E+arX) [our card](https://md.research.elamur.ai/free/en/papers/schuhmann-2021-laion-400m-open-dataset-clip/) 69. Sharma, P., Ding, N., Goodman, S., and Soricut, R. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018. [find](https://scholar.google.com/scholar?q=Sharma%2C+P%2E%2C+Ding%2C+N%2E%2C+Goodman%2C+S%2E%2C+and+Soricut%2C+R%2E+Conceptual+captions%3A+A+cleaned%2C+hypernymed%2C+image+alt%2Dtext+dataset+for+automatic+image+captioning%2E+In+ACL%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/) 70. Shen, Y., Song, K., Tan, X., Li, D., Lu, W., and Zhuang, Y. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580, 2023. [find](https://scholar.google.com/scholar?q=Shen%2C+Y%2E%2C+Song%2C+K%2E%2C+Tan%2C+X%2E%2C+Li%2C+D%2E%2C+Lu%2C+W%2E%2C+and+Zhuang%2C+Y%2E+Hugginggpt%3A+Solving+ai+tasks+with+chatgpt+and+its+friends+in+huggingface%2E+arXiv+preprint+arXiv%3A2303%2E17580%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/shen-2023-hugginggpt-solving-ai-tasks-chatgpt/) 71. Sidorov, O., Hu, R., Rohrbach, M., and Singh, A. Textcaps: a dataset for image captioning with reading comprehension. In ECCV, pp. 742–758, 2020. [find](https://scholar.google.com/scholar?q=Sidorov%2C+O%2E%2C+Hu%2C+R%2E%2C+Rohrbach%2C+M%2E%2C+and+Singh%2C+A%2E+Textcaps%3A+a+dataset+for+image+captioning+with+reading+comprehension%2E+In+ECCV%2C+pp%2E+742%E2%80%93758%2C+2020%2E) 72. Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317–8326, 2019. [find](https://scholar.google.com/scholar?q=Singh%2C+A%2E%2C+Natarajan%2C+V%2E%2C+Shah%2C+M%2E%2C+Jiang%2C+Y%2E%2C+Chen%2C+X%2E%2C+Batra%2C+D%2E%2C+Parikh%2C+D%2E%2C+and+Rohrbach%2C+M%2E+Towards+vqa+models+that+can+read%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+vision+and+patt) 73. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a. [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%2C+Lavril%2C+T%2E%2C+Izacard%2C+G%2E%2C+Martinet%2C+X%2E%2C+Lachaux%2C+M%2E%2DA%2E%2C+Lacroix%2C+T%2E%2C+Rozi%C3%A8re%2C+B%2E%2C+Goyal%2C+N%2E%2C+Hambro%2C+E%2E%2C+Azhar%2C+F%2E%2C+et+al%2E+Llama%3A+Open+and+efficient+foundation+language+models%2E+arXiv+prepr) 74. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b. [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%2C+Martin%2C+L%2E%2C+Stone%2C+K%2E%2C+Albert%2C+P%2E%2C+Almahairi%2C+A%2E%2C+Babaei%2C+Y%2E%2C+Bashlykov%2C+N%2E%2C+Batra%2C+S%2E%2C+Bhargava%2C+P%2E%2C+Bhosale%2C+S%2E%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtuned+chat+models%2E+arXiv+prepri) 75. Tsimpoukelli, M., Menick, J., Cabi, S., Eslami, S., Vinyals, O., and Hill, F. Multimodal few-shot learning with frozen language models. arXiv preprint arXiv:2106.13884, 2021. [find](https://scholar.google.com/scholar?q=Tsimpoukelli%2C+M%2E%2C+Menick%2C+J%2E%2C+Cabi%2C+S%2E%2C+Eslami%2C+S%2E%2C+Vinyals%2C+O%2E%2C+and+Hill%2C+F%2E+Multimodal+few%2Dshot+learning+with+frozen+language+models%2E+arXiv+preprint+arXiv%3A2106%2E13884%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/tsimpoukelli-2021-multimodal-few-shot-learning-frozen/) 76. Wang, J., Yang, Z., Hu, X., Li, L., Lin, K., Gan, Z., Liu, Z., Liu, C., and Wang, L. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100, 2022a. [find](https://scholar.google.com/scholar?q=Wang%2C+J%2E%2C+Yang%2C+Z%2E%2C+Hu%2C+X%2E%2C+Li%2C+L%2E%2C+Lin%2C+K%2E%2C+Gan%2C+Z%2E%2C+Liu%2C+Z%2E%2C+Liu%2C+C%2E%2C+and+Wang%2C+L%2E+Git%3A+A+generative+image%2Dto%2Dtext+transformer+for+vision+and+language%2E+arXiv+preprint+arXiv%3A2205%2E14100%2C+2022a%2E) 77. Wang, W., Chen, Z., Chen, X., Wu, J., Zhu, X., Zeng, G., Luo, P., Lu, T., Zhou, J., Qiao, Y., et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. arXiv preprint arXiv:2305.11175, 2023. — [arXiv:2305.11175](https://arxiv.org/abs/2305.11175) [find](https://scholar.google.com/scholar?q=Wang%2C+W%2E%2C+Chen%2C+Z%2E%2C+Chen%2C+X%2E%2C+Wu%2C+J%2E%2C+Zhu%2C+X%2E%2C+Zeng%2C+G%2E%2C+Luo%2C+P%2E%2C+Lu%2C+T%2E%2C+Zhou%2C+J%2E%2C+Qiao%2C+Y%2E%2C+et+al%2E+Visionllm%3A+Large+language+model+is+also+an+open%2Dended+decoder+for+vision%2Dcentric+tasks%2E+arXiv+prepr) 78. Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., and Summers, R. M. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 20… [find](https://scholar.google.com/scholar?q=Wang%2C+X%2E%2C+Peng%2C+Y%2E%2C+Lu%2C+L%2E%2C+Lu%2C+Z%2E%2C+Bagheri%2C+M%2E%2C+and+Summers%2C+R%2E+M%2E+Chestx%2Dray8%3A+Hospital%2Dscale+chest+x%2Dray+database+and+benchmarks+on+weakly%2Dsupervised+classification+and+localization+of+common+thora) 79. Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., and Cao, Y. Simvlm: Simple visual language model pretraining with weak supervision. In ICLR, 2022b. [find](https://scholar.google.com/scholar?q=Wang%2C+Z%2E%2C+Yu%2C+J%2E%2C+Yu%2C+A%2E+W%2E%2C+Dai%2C+Z%2E%2C+Tsvetkov%2C+Y%2E%2C+and+Cao%2C+Y%2E+Simvlm%3A+Simple+visual+language+model+pretraining+with+weak+supervision%2E+In+ICLR%2C+2022b%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2021-simvlm-simple-visual-language-model/) 80. Xu, P., Shao, W., Zhang, K., Gao, P., Liu, S., Lei, M., Meng, F., Huang, S., Qiao, Y., and Luo, P. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09265, 2023. — [arXiv:2306.09265](https://arxiv.org/abs/2306.09265) [find](https://scholar.google.com/scholar?q=Xu%2C+P%2E%2C+Shao%2C+W%2E%2C+Zhang%2C+K%2E%2C+Gao%2C+P%2E%2C+Liu%2C+S%2E%2C+Lei%2C+M%2E%2C+Meng%2C+F%2E%2C+Huang%2C+S%2E%2C+Qiao%2C+Y%2E%2C+and+Luo%2C+P%2E+Lvlm%2Dehub%3A+A+comprehensive+evaluation+benchmark+for+large+vision%2Dlanguage+models%2E+arXiv+preprint+arXi) 81. Yang, J., Zhang, H., Li, F., Zou, X., Li, C., and Gao, J. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441, 2023a. [find](https://scholar.google.com/scholar?q=Yang%2C+J%2E%2C+Zhang%2C+H%2E%2C+Li%2C+F%2E%2C+Zou%2C+X%2E%2C+Li%2C+C%2E%2C+and+Gao%2C+J%2E+Set%2Dof%2Dmark+prompting+unleashes+extraordinary+visual+grounding+in+gpt%2D4v%2E+arXiv+preprint+arXiv%3A2310%2E11441%2C+2023a%2E) 82. Yang, Z., Lu, Y., Wang, J., Yin, X., Florencio, D., Wang, L., Zhang, C., Zhang, L., and Luo, J. Tap: Text-aware pre-training for text-vqa and text-caption. In CVPR, pp. 8751–8761, 2021. [find](https://scholar.google.com/scholar?q=Yang%2C+Z%2E%2C+Lu%2C+Y%2E%2C+Wang%2C+J%2E%2C+Yin%2C+X%2E%2C+Florencio%2C+D%2E%2C+Wang%2C+L%2E%2C+Zhang%2C+C%2E%2C+Zhang%2C+L%2E%2C+and+Luo%2C+J%2E+Tap%3A+Text%2Daware+pre%2Dtraining+for+text%2Dvqa+and+text%2Dcaption%2E+In+CVPR%2C+pp%2E+8751%E2%80%938761%2C+2021%2E) 83. Yang, Z., Gan, Z., Wang, J., Hu, X., Ahmed, F., Liu, Z., Lu, Y., and Wang, L. Unitab: Unifying text and box outputs for grounded vision-language modeling. In European Conference on Computer Vision, pp. 521–539. Springer, 2022a. [find](https://scholar.google.com/scholar?q=Yang%2C+Z%2E%2C+Gan%2C+Z%2E%2C+Wang%2C+J%2E%2C+Hu%2C+X%2E%2C+Ahmed%2C+F%2E%2C+Liu%2C+Z%2E%2C+Lu%2C+Y%2E%2C+and+Wang%2C+L%2E+Unitab%3A+Unifying+text+and+box+outputs+for+grounded+vision%2Dlanguage+modeling%2E+In+European+Conference+on+Computer+Vision%2C+pp) 84. Yang, Z., Gan, Z., Wang, J., Hu, X., Lu, Y., Liu, Z., and Wang, L. An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 3081–3089, 2022b. [find](https://scholar.google.com/scholar?q=Yang%2C+Z%2E%2C+Gan%2C+Z%2E%2C+Wang%2C+J%2E%2C+Hu%2C+X%2E%2C+Lu%2C+Y%2E%2C+Liu%2C+Z%2E%2C+and+Wang%2C+L%2E+An+empirical+study+of+gpt%2D3+for+few%2Dshot+knowledge%2Dbased+vqa%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volum) 85. Yang, Z., Li, L., Lin, K., Wang, J., Lin, C.-C., Liu, Z., and Wang, L. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421, 2023b. [find](https://scholar.google.com/scholar?q=Yang%2C+Z%2E%2C+Li%2C+L%2E%2C+Lin%2C+K%2E%2C+Wang%2C+J%2E%2C+Lin%2C+C%2E%2DC%2E%2C+Liu%2C+Z%2E%2C+and+Wang%2C+L%2E+The+dawn+of+lmms%3A+Preliminary+explorations+with+gpt%2D4v+%28ision%29%2E+arXiv+preprint+arXiv%3A2309%2E17421%2C+2023b%2E) [our card](https://md.research.elamur.ai/free/en/papers/yang-2023-dawn-lmms-preliminary-explorations-gpt/) 86. Yang, Z., Li, L., Wang, J., Lin, K., Azarnasab, E., Ahmed, F., Liu, Z., Liu, C., Zeng, M., and Wang, L. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381, 2023c. [find](https://scholar.google.com/scholar?q=Yang%2C+Z%2E%2C+Li%2C+L%2E%2C+Wang%2C+J%2E%2C+Lin%2C+K%2E%2C+Azarnasab%2C+E%2E%2C+Ahmed%2C+F%2E%2C+Liu%2C+Z%2E%2C+Liu%2C+C%2E%2C+Zeng%2C+M%2E%2C+and+Wang%2C+L%2E+Mm%2Dreact%3A+Prompting+chatgpt+for+multimodal+reasoning+and+action%2E+arXiv+preprint+arXiv%3A2303%2E11381) 87. Ye, Q., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang, J., Hu, A., Shi, P., Shi, Y., et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023. [find](https://scholar.google.com/scholar?q=Ye%2C+Q%2E%2C+Xu%2C+H%2E%2C+Xu%2C+G%2E%2C+Ye%2C+J%2E%2C+Yan%2C+M%2E%2C+Zhou%2C+Y%2E%2C+Wang%2C+J%2E%2C+Hu%2C+A%2E%2C+Shi%2C+P%2E%2C+Shi%2C+Y%2E%2C+et+al%2E+mplug%2Dowl%3A+Modularization+empowers+large+language+models+with+multimodality%2E+arXiv+preprint+arXiv%3A2304%2E141) [our card](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/) 88. Zellers, R., Bisk, Y., Farhadi, A., and Choi, Y. From recognition to cognition: Visual commonsense reasoning. In CVPR, pp. 6720–6731, 2019. [find](https://scholar.google.com/scholar?q=Zellers%2C+R%2E%2C+Bisk%2C+Y%2E%2C+Farhadi%2C+A%2E%2C+and+Choi%2C+Y%2E+From+recognition+to+cognition%3A+Visual+commonsense+reasoning%2E+In+CVPR%2C+pp%2E+6720%E2%80%936731%2C+2019%2E) 89. Zeng, A., Wong, A., Welker, S., Choromanski, K., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., Lee, J., Vanhoucke, V., et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022. [find](https://scholar.google.com/scholar?q=Zeng%2C+A%2E%2C+Wong%2C+A%2E%2C+Welker%2C+S%2E%2C+Choromanski%2C+K%2E%2C+Tombari%2C+F%2E%2C+Purohit%2C+A%2E%2C+Ryoo%2C+M%2E%2C+Sindhwani%2C+V%2E%2C+Lee%2C+J%2E%2C+Vanhoucke%2C+V%2E%2C+et+al%2E+Socratic+models%3A+Composing+zero%2Dshot+multimodal+reasoning+with+langua) [our card](https://md.research.elamur.ai/free/en/papers/zeng-2022-socratic-models-composing-zero-shot/) 90. Zeng, Y., Zhang, H., Zheng, J., Xia, J., Wei, G., Wei, Y., Zhang, Y., and Kong, T. What matters in training a gpt4-style language model with multimodal inputs? arXiv preprint arXiv:2307.02469, 2023. [find](https://scholar.google.com/scholar?q=Zeng%2C+Y%2E%2C+Zhang%2C+H%2E%2C+Zheng%2C+J%2E%2C+Xia%2C+J%2E%2C+Wei%2C+G%2E%2C+Wei%2C+Y%2E%2C+Zhang%2C+Y%2E%2C+and+Kong%2C+T%2E+What+matters+in+training+a+gpt4%2Dstyle+language+model+with+multimodal+inputs%3F+arXiv+preprint+arXiv%3A2307%2E02469%2C+2023%2E) 91. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. [find](https://scholar.google.com/scholar?q=Zhang%2C+S%2E%2C+Roller%2C+S%2E%2C+Goyal%2C+N%2E%2C+Artetxe%2C+M%2E%2C+Chen%2C+M%2E%2C+Chen%2C+S%2E%2C+Dewan%2C+C%2E%2C+Diab%2C+M%2E%2C+Li%2C+X%2E%2C+Lin%2C+X%2E+V%2E%2C+et+al%2E+Opt%3A+Open+pre%2Dtrained+transformer+language+models%2E+arXiv+preprint+arXiv%3A2205%2E01068%2C+2) 92. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. [find](https://scholar.google.com/scholar?q=Zheng%2C+L%2E%2C+Chiang%2C+W%2E%2DL%2E%2C+Sheng%2C+Y%2E%2C+Zhuang%2C+S%2E%2C+Wu%2C+Z%2E%2C+Zhuang%2C+Y%2E%2C+Lin%2C+Z%2E%2C+Li%2C+Z%2E%2C+Li%2C+D%2E%2C+Xing%2C+E%2E+P%2E%2C+Zhang%2C+H%2E%2C+Gonzalez%2C+J%2E+E%2E%2C+and+Stoica%2C+I%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+a) 93. Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023a. [find](https://scholar.google.com/scholar?q=Zhu%2C+D%2E%2C+Chen%2C+J%2E%2C+Shen%2C+X%2E%2C+Li%2C+X%2E%2C+and+Elhoseiny%2C+M%2E+Minigpt%2D4%3A+Enhancing+vision%2Dlanguage+understanding+with+advanced+large+language+models%2E+arXiv+preprint+arXiv%3A2304%2E10592%2C+2023a%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/) 94. Zhu, W., Hessel, J., Awadalla, A., Gadre, S. Y., Dodge, J., Fang, A., Yu, Y., Schmidt, L., Wang, W. Y., and Choi, Y. Multimodal c4: An open, billion-scale corpus of images interleaved with text. arXiv preprint arXiv:2304.06939, 2023b. — [arXiv:2304.06939](https://arxiv.org/abs/2304.06939) [find](https://scholar.google.com/scholar?q=Zhu%2C+W%2E%2C+Hessel%2C+J%2E%2C+Awadalla%2C+A%2E%2C+Gadre%2C+S%2E+Y%2E%2C+Dodge%2C+J%2E%2C+Fang%2C+A%2E%2C+Yu%2C+Y%2E%2C+Schmidt%2C+L%2E%2C+Wang%2C+W%2E+Y%2E%2C+and+Choi%2C+Y%2E+Multimodal+c4%3A+An+open%2C+billion%2Dscale+corpus+of+images+interleaved+with+text%2E+arXiv)