---
title: "MMC — paper by Liu et al. (2024) — summary and ideas"
description: "General-purpose open-source large multimodal models (LMMs) handle natural photos well but… 46 references, cited by 57 papers here."
authors: ["Fuxiao Liu", "Xiaoyang Wang", "Wenlin Yao", "Jianshu Chen", "Kaiqiang Song", "Sangwoo Cho", "Yaser Yacoob", "Dong Yu"]
year: 2024
venue: "Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.naacl-long.70/"
doi: "10.18653/v1/2024.naacl-long.70"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2024-mmc-advancing-multimodal-chart-understanding/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2024-mmc-advancing-multimodal-chart-understanding/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning

# MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning

**Authors:** [Fuxiao Liu](https://md.research.elamur.ai/free/en/authors/fuxiao-liu/), [Xiaoyang Wang](https://md.research.elamur.ai/free/en/authors/xiaoyang-wang/), [Wenlin Yao](https://md.research.elamur.ai/free/en/authors/wenlin-yao/), [Jianshu Chen](https://md.research.elamur.ai/free/en/authors/jianshu-chen/), [Kaiqiang Song](https://md.research.elamur.ai/free/en/authors/kaiqiang-song/), [Sangwoo Cho](https://md.research.elamur.ai/free/en/authors/sangwoo-cho/), [Yaser Yacoob](https://md.research.elamur.ai/free/en/authors/yaser-yacoob/) and [Dong Yu](https://md.research.elamur.ai/free/en/authors/dong-yu/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2024](https://md.research.elamur.ai/free/en/venues/naacl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.naacl-long.70/) · [PDF](https://aclanthology.org/2024.naacl-long.70.pdf) · [DOI 10.18653/v1/2024.naacl-long.70](https://doi.org/10.18653/v1/2024.naacl-long.70) · [OpenAlex](https://openalex.org/W4401042875)

**Citations:** 26 citations (OpenAlex, 2026-10-14) · [57](https://md.research.elamur.ai/free/en/papers/liu-2024-mmc-advancing-multimodal-chart-understanding/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
General-purpose open-source large multimodal models (LMMs) handle natural photos well but struggle with charts. Charts encode data through abstract elements such as axes, legends and trend lines, and these models get little training on them. The authors address this with three contributions.

- **MMC-Instruction** is a 600k-instance corpus. It mixes chart–caption alignment pairs, drawn from arXiv LaTeX sources and existing chart datasets, with instruction data generated by GPT-4 in varied styles and task types.
- **MMCA** is a 7B LMM tuned on this corpus. Unlike typical setups that freeze the CLIP encoder, it also updates the vision component during training. MMCA reaches state-of-the-art results among open-source models on existing chart QA benchmarks.
- **MMC-Benchmark** is a human-annotated test suite with nine tasks. It can be scored either through GPT-4-judged free-form answers or through multiple-choice questions that need no GPT-4.

Experiments show that current LMMs, GPT-4V included, still misread charts. This makes the resources useful for measuring and improving chart reasoning in data-heavy applications.

## Key points
- MMC-Instruction contains three parts:
  - 210k Scientific Chart-Caption pairs from arXiv papers (2010–2020), keeping only captions of at least 25 tokens.
  - About 190k pairs sampled from Statista, PlotQA, VisText, ChartInfo and Unichart.
  - 200k GPT-4-generated instructions.
- MMC-Benchmark covers 1,063 unique images, with 1,275 multiple-choice and 851 free-form questions. Its tasks include chart reasoning, stock chart analysis, Chart To DataTable and Chart To Json.
- GPT-4V outperforms open-source LMMs on 7 of the 9 tasks but does poorly on Chart To DataTable and Chart To Json.
- A manual review of 100 GPT-4V errors attributed them to perception (39%), language bias (35%), reasoning (15%) and missing knowledge (11%).
- Code and data are available at github.com/FuxiaoLiu/MMC.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Maryland, College Park](https://md.research.elamur.ai/free/en/universities/university-of-maryland-college-park/) — University, College Park, United States · 83 works · 23,784 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 269 works

## Cite

Cite as: Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob and Dong Yu. 2024. MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liu2024mmc,
  title = {{MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning}},
  author = {Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob and Dong Yu},
  booktitle = {Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.70},
  url = {https://doi.org/10.18653/v1/2024.naacl-long.70}
}
```

- [← previous Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation](https://md.research.elamur.ai/free/en/papers/kocmi-2024-error-span-annotation-balanced-approach/)
- [next → CodeGemma: Open Code Models Based on Gemma](https://md.research.elamur.ai/free/en/papers/zhao-2024-codegemma-open-code-models-based/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

46 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
2. Yunkang Cao, Xiaohao Xu, Chen Sun, Xiaonan Huang, and Weiming Shen. 2023. Towards generic anomaly detection and understanding: Large-scale visual-linguistic model (gpt-4v) takes the lead. arXiv preprint arXiv:2311.02782. — [arXiv:2311.02782](https://arxiv.org/abs/2311.02782) [find](https://scholar.google.com/scholar?q=Yunkang+Cao%2C+Xiaohao+Xu%2C+Chen+Sun%2C+Xiaonan+Huang%2C+and+Weiming+Shen%2E+2023%2E+Towards+generic+anomaly+detection+and+understanding%3A+Large%2Dscale+visual%2Dlinguistic+model+%28gpt%2D4v%29+takes+the+lead%2E+arXiv+prepri)
3. Ritwick Chaudhry, Sumit Shekhar, Utkarsh Gupta, Pranav Maneriker, Prann Bansal, and Ajay Joshi. 2020. Leaf-qa: Locate, encode & attend for figure question answering. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3512–3521. [find](https://scholar.google.com/scholar?q=Ritwick+Chaudhry%2C+Sumit+Shekhar%2C+Utkarsh+Gupta%2C+Pranav+Maneriker%2C+Prann+Bansal%2C+and+Ajay+Joshi%2E+2020%2E+Leaf%2Dqa%3A+Locate%2C+encode+%26+attend+for+figure+question+answering%2E+In+Proceedings+of+the+IEEE%2FCVF+Win)
4. Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. 2023a. [Minigpt-v2: large language model as a unified interface for vision-language multi-task learning](https://md.research.elamur.ai/free/en/papers/chen-2023-minigpt-v2-large-language-model/). arXiv preprint arXiv:2310.09478.
5. Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. 2023b. [Shikra: Unleashing multimodal llm’s referential dialogue magic](https://md.research.elamur.ai/free/en/papers/chen-2023-shikra-unleashing-multimodal-llm-s/). arXiv preprint arXiv:2306.15195.
6. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311.
7. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. [Instructblip: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/). arXiv preprint arXiv:2305.06500.
8. Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. 2023. Multimodal-gpt: A vision and language model for dialogue with humans. arXiv preprint arXiv:2305.04790. — [arXiv:2305.04790](https://arxiv.org/abs/2305.04790) [find](https://scholar.google.com/scholar?q=Tao+Gong%2C+Chengqi+Lyu%2C+Shilong+Zhang%2C+Yudong+Wang%2C+Miao+Zheng%2C+Qian+Zhao%2C+Kuikun+Liu%2C+Wenwei+Zhang%2C+Ping+Luo%2C+and+Kai+Chen%2E+2023%2E+Multimodal%2Dgpt%3A+A+vision+and+language+model+for+dialogue+with+humans%2E+)
9. Ting-Yao Hsu, C Lee Giles, and Ting-Hao’Kenneth’ Huang. 2021. Scicap: Generating captions for scientific figures. arXiv preprint arXiv:2110.11624. [find](https://scholar.google.com/scholar?q=Ting%2DYao+Hsu%2C+C+Lee+Giles%2C+and+Ting%2DHao%E2%80%99Kenneth%E2%80%99+Huang%2E+2021%2E+Scicap%3A+Generating+captions+for+scientific+figures%2E+arXiv+preprint+arXiv%3A2110%2E11624%2E)
10. Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. 2018. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648–5656. [find](https://scholar.google.com/scholar?q=Kushal+Kafle%2C+Brian+Price%2C+Scott+Cohen%2C+and+Christopher+Kanan%2E+2018%2E+Dvqa%3A+Understanding+data+visualizations+via+question+answering%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+patter)
11. Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. 2017. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300. [find](https://scholar.google.com/scholar?q=Samira+Ebrahimi+Kahou%2C+Vincent+Michalski%2C+Adam+Atkinson%2C+%C3%81kos+K%C3%A1d%C3%A1r%2C+Adam+Trischler%2C+and+Yoshua+Bengio%2E+2017%2E+Figureqa%3A+An+annotated+figure+dataset+for+visual+reasoning%2E+arXiv+preprint+arXiv%3A1710%2E0730)
12. Shankar Kantharaj, Rixie Tiffany Ko Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. 2022. [Chart-to-text: A large-scale benchmark for chart summarization](https://md.research.elamur.ai/free/en/papers/kantharaj-2022-chart-text-large-scale-benchmark/). arXiv preprint arXiv:2203.06486.
13. Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. 2022. [Ocr-free document understanding transformer](https://md.research.elamur.ai/free/en/papers/kim-2021-ocr-free-document-understanding-transformer/). In European Conference on Computer Vision, pages 498–517. Springer.
14. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023. [Segment anything](https://md.research.elamur.ai/free/en/papers/kirillov-2023-segment-anything/). arXiv preprint arXiv:2304.02643.
15. Jay Lal, Aditya Mitkari, Mahesh Bhosale, and David Doermann. 2023. Lineformer: Line chart data extraction using instance segmentation. In International Conference on Document Analysis and Recognition, pages 387–400. Springer. [find](https://scholar.google.com/scholar?q=Jay+Lal%2C+Aditya+Mitkari%2C+Mahesh+Bhosale%2C+and+David+Doermann%2E+2023%2E+Lineformer%3A+Line+chart+data+extraction+using+instance+segmentation%2E+In+International+Conference+on+Document+Analysis+and+Recognition%2C)
16. Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2023. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In International Conference on Machine Learni… [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Mandar+Joshi%2C+Iulia+Raluca+Turc%2C+Hexiang+Hu%2C+Fangyu+Liu%2C+Julian+Martin+Eisenschlos%2C+Urvashi+Khandelwal%2C+Peter+Shaw%2C+Ming%2DWei+Chang%2C+and+Kristina+Toutanova%2E+2023%2E+Pix2struct%3A+Screenshot+par)
17. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. [Chartqa: A benchmark for question answering about charts with visual and logical reasoning](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/). arXiv preprint arXiv:2203.10244.
18. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. [Docvqa: A dataset for vqa on document images](https://md.research.elamur.ai/free/en/papers/mathew-2020-docvqa-dataset-vqa-document-images/). In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209.
19. Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. 2020. Plotqa: Reasoning over scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1527–1536. [find](https://scholar.google.com/scholar?q=Nitesh+Methani%2C+Pritha+Ganguly%2C+Mitesh+M+Khapra%2C+and+Pratyush+Kumar%2E+2020%2E+Plotqa%3A+Reasoning+over+scientific+plots%2E+In+Proceedings+of+the+IEEE%2FCVF+Winter+Conference+on+Applications+of+Computer+Vision%2C)
20. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. [Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). arXiv preprint arXiv:2301.12597.
21. OpenAI. 2022. Introducing chatgpt. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2022%2E+Introducing+chatgpt%2E)
22. Shengzhi Li and Nima Tajbakhsh. 2023. Scigraphqa: A large-scale synthetic multi-turn question-answering dataset for scientific graphs. arXiv preprint arXiv:2308.03349. [find](https://scholar.google.com/scholar?q=Shengzhi+Li+and+Nima+Tajbakhsh%2E+2023%2E+Scigraphqa%3A+A+large%2Dscale+synthetic+multi%2Dturn+question%2Danswering+dataset+for+scientific+graphs%2E+arXiv+preprint+arXiv%3A2308%2E03349%2E)
23. OpenAI. 2023b. Gpt-4v(ision) system card. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023b%2E+Gpt%2D4v%28ision%29+system+card%2E)
24. Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2023a. [Hallusionbench: You see what you think?](https://md.research.elamur.ai/free/en/papers/guan-2023-hallusionbench-advanced-diagnostic-suite-entangled/) or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models. arXiv preprint…
25. Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2023b. [Aligning large multi-modal model with robust instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-mitigating-hallucination-large-multi-modal/). arXiv preprint arXiv:2306.14565.
26. Fuxiao Liu, Hao Tan, and Chris Tensmeyer. 2023c. Documentclip: Linking figures and main body text in reflowed documents. arXiv preprint arXiv:2306.06306. — [arXiv:2306.06306](https://arxiv.org/abs/2306.06306) [find](https://scholar.google.com/scholar?q=Fuxiao+Liu%2C+Hao+Tan%2C+and+Chris+Tensmeyer%2E+2023c%2E+Documentclip%3A+Linking+figures+and+main+body+text+in+reflowed+documents%2E+arXiv+preprint+arXiv%3A2306%2E06306%2E+%E2%80%94+%5BarXiv%3A2306%2E06306%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F230)
27. Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez. 2020. Visualnews : Benchmark and challenges in entity-aware image captioning. [find](https://scholar.google.com/scholar?q=Fuxiao+Liu%2C+Yinghan+Wang%2C+Tianlu+Wang%2C+and+Vicente+Ordonez%2E+2020%2E+Visualnews+%3A+Benchmark+and+challenges+in+entity%2Daware+image+captioning%2E)
28. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023d. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/). arXiv preprint arXiv:2310.03744.
29. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023e. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). arXiv preprint arXiv:2304.08485.
30. James Manyika. 2023. An overview of bard: an early experiment with generative ai. AI. Google Static Documents. [find](https://scholar.google.com/scholar?q=James+Manyika%2E+2023%2E+An+overview+of+bard%3A+an+early+experiment+with+generative+ai%2E+AI%2E+Google+Static+Documents%2E)
31. Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, and Shafiq Joty. 2023. [Unichart: A universal vision-language pretrained model for chart comprehension and reasoning](https://md.research.elamur.ai/free/en/papers/masry-2023-unichart-universal-vision-language-pretrained/). arXiv preprint arXiv:2305.14761.
32. OpenAI. 2023a. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
33. Benny J Tang, Angie Boggust, and Arvind Satyanarayan. 2023. Vistext: A benchmark for semantically rich chart captioning. arXiv preprint arXiv:2307.05356. [find](https://scholar.google.com/scholar?q=Benny+J+Tang%2C+Angie+Boggust%2C+and+Arvind+Satyanarayan%2E+2023%2E+Vistext%3A+A+benchmark+for+semantically+rich+chart+captioning%2E+arXiv+preprint+arXiv%3A2307%2E05356%2E)
34. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
35. Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. 2023a. [Visual chatgpt: Talking, drawing and editing with visual foundation models](https://md.research.elamur.ai/free/en/papers/wu-2023-visual-chatgpt-talking-drawing-editing/). arXiv preprint arXiv:2303.04671.
36. Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. 2023b. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519. [find](https://scholar.google.com/scholar?q=Shengqiong+Wu%2C+Hao+Fei%2C+Leigang+Qu%2C+Wei+Ji%2C+and+Tat%2DSeng+Chua%2E+2023b%2E+Next%2Dgpt%3A+Any%2Dto%2Dany+multimodal+llm%2E+arXiv+preprint+arXiv%3A2309%2E05519%2E)
37. Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. 2023a. Gpt4tools: Teaching large language model to use tools via self-instruction. arXiv preprint arXiv:2305.18752. [find](https://scholar.google.com/scholar?q=Rui+Yang%2C+Lin+Song%2C+Yanwei+Li%2C+Sijie+Zhao%2C+Yixiao+Ge%2C+Xiu+Li%2C+and+Ying+Shan%2E+2023a%2E+Gpt4tools%3A+Teaching+large+language+model+to+use+tools+via+self%2Dinstruction%2E+arXiv+preprint+arXiv%3A2305%2E18752%2E)
38. Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023b. [The dawn of lmms: Preliminary explorations with gpt-4v (ision)](https://md.research.elamur.ai/free/en/papers/yang-2023-dawn-lmms-preliminary-explorations-gpt/). arXiv preprint arXiv:2309.17421, 9.
39. Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023c. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381. [find](https://scholar.google.com/scholar?q=Zhengyuan+Yang%2C+Linjie+Li%2C+Jianfeng+Wang%2C+Kevin+Lin%2C+Ehsan+Azarnasab%2C+Faisal+Ahmed%2C+Zicheng+Liu%2C+Ce+Liu%2C+Michael+Zeng%2C+and+Lijuan+Wang%2E+2023c%2E+Mm%2Dreact%3A+Prompting+chatgpt+for+multimodal+reasoning+and+)
40. Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Florencio, Lijuan Wang, Cha Zhang, Lei Zhang, and Jiebo Luo. 2021. Tap: Text-aware pre-training for text-vqa and text-caption. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8751–8761. [find](https://scholar.google.com/scholar?q=Zhengyuan+Yang%2C+Yijuan+Lu%2C+Jianfeng+Wang%2C+Xi+Yin%2C+Dinei+Florencio%2C+Lijuan+Wang%2C+Cha+Zhang%2C+Lei+Zhang%2C+and+Jiebo+Luo%2E+2021%2E+Tap%3A+Text%2Daware+pre%2Dtraining+for+text%2Dvqa+and+text%2Dcaption%2E+In+Proceedings+of)
41. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. 2023. [mplug-owl: Modularization empowers large language models with multimodality](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/). arXiv preprint arXiv:2304.14178.
42. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. [A survey on multimodal large language models](https://md.research.elamur.ai/free/en/papers/yin-2023-survey-multimodal-large-language-models/). arXiv preprint arXiv:2306.13549.
43. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. [Mm-vet: Evaluating large multimodal models for integrated capabilities](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/). arXiv preprint arXiv:2308.02490.
44. Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. 2023. Investigating the catastrophic forgetting in multimodal large language models. arXiv preprint arXiv:2309.10313. [find](https://scholar.google.com/scholar?q=Yuexiang+Zhai%2C+Shengbang+Tong%2C+Xiao+Li%2C+Mu+Cai%2C+Qing+Qu%2C+Yong+Jae+Lee%2C+and+Yi+Ma%2E+2023%2E+Investigating+the+catastrophic+forgetting+in+multimodal+large+language+models%2E+arXiv+preprint+arXiv%3A2309%2E10313%2E)
45. Ao Zhang, Hao Fei, Yuan Yao, Wei Ji, Li Li, Zhiyuan Liu, and Tat-Seng Chua. 2023. Transfer visual prompt generator across llms. arXiv preprint arXiv:2305.01278. [find](https://scholar.google.com/scholar?q=Ao+Zhang%2C+Hao+Fei%2C+Yuan+Yao%2C+Wei+Ji%2C+Li+Li%2C+Zhiyuan+Liu%2C+and+Tat%2DSeng+Chua%2E+2023%2E+Transfer+visual+prompt+generator+across+llms%2E+arXiv+preprint+arXiv%3A2305%2E01278%2E)
46. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. [Minigpt-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/). arXiv preprint arXiv:2304.10592.

*Updated 24 October 2026*