---
title: "CharXiv — paper by Wang et al. (2024) — summary and ideas"
description: "The authors show this with a stress test. 34.5 points. 71 references, cited by 94 papers here."
authors: ["Zirui Wang", "Mengzhou Xia", "Luxi He", "Howard Chen", "Yitao Liu", "Richard Zhu", "Kaiqu Liang", "Xindi Wu", "Haotian Liu", "Sadhika Malladi", "Alexis Chevalier", "Sanjeev Arora", "Danqi Chen"]
year: 2024
venue: "arXiv:2406.18521"
source: "https://arxiv.org/abs/2406.18521"
doi: "10.48550/arxiv.2406.18521"
arxiv: "2406.18521"
licence: "CC BY-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2024-charxiv-charting-gaps-realistic-chart/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2024-charxiv-charting-gaps-realistic-chart/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs

# CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs

**Authors:** [Zirui Wang](https://md.research.elamur.ai/free/en/authors/zirui-wang/), [Mengzhou Xia](https://md.research.elamur.ai/free/en/authors/mengzhou-xia/), [Luxi He](https://md.research.elamur.ai/free/en/authors/luxi-he/), [Howard Chen](https://md.research.elamur.ai/free/en/authors/howard-chen/), [Yitao Liu](https://md.research.elamur.ai/free/en/authors/yitao-liu/), Richard Zhu, Kaiqu Liang, Xindi Wu, [Haotian Liu](https://md.research.elamur.ai/free/en/authors/haotian-liu/), [Sadhika Malladi](https://md.research.elamur.ai/free/en/authors/sadhika-malladi/), [Alexis Chevalier](https://md.research.elamur.ai/free/en/authors/alexis-chevalier/), [Sanjeev Arora](https://md.research.elamur.ai/free/en/authors/sanjeev-arora/) and [Danqi Chen](https://md.research.elamur.ai/free/en/authors/danqi-chen/) · **Year:** 2024 · **Venue:** [arXiv:2406.18521](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2024](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2024/)) · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2406.18521) · [PDF](https://arxiv.org/pdf/2406.18521) · [DOI 10.48550/arxiv.2406.18521](https://doi.org/10.48550/arxiv.2406.18521) · [arXiv 2406.18521](https://arxiv.org/abs/2406.18521) · [OpenAlex](https://openalex.org/W4415795850)

**Citations:** 25 citations (OpenAlex, 2026-10-14) · [94](https://md.research.elamur.ai/free/en/papers/wang-2024-charxiv-charting-gaps-realistic-chart/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Benchmarks commonly used to evaluate chart understanding in multimodal LLMs, such as FigureQA, DVQA and ChartQA, rely on synthetic or visually uniform charts and on template-generated questions. As a result, they make models look more capable than they are. The authors show this with a stress test. When either the questions or the charts are changed slightly, open-source models that had seemed to beat proprietary ones lose up to 34.5 points of accuracy. To measure the skill more honestly, they built CharXiv, a benchmark of 2,323 real charts taken from arXiv papers in eight subject areas. Human experts selected every chart and wrote and checked every question and answer. Questions come in two kinds: descriptive ones about basic chart elements and reasoning ones that require combining visual information. Answers are short, so an LLM can grade them automatically. Across 24 tested models, GPT-4o leads on reasoning with 47.1%, compared with 29.2% for the best open model, InternVL Chat V1.5. Humans score 80.5%. CharXiv therefore shows a much larger gap than earlier benchmarks did.

## Key points
- **Stress test:** The authors took 174 MathVista samples from DVQA, FigureQA and ChartQA and made two variants, one with rewritten questions and one with substitute charts from arXiv. SPHINX V2's accuracy fell from 63.2% to 28.6% on the rewritten questions.
- **Benchmark design:** CharXiv contains 2,323 arXiv charts covering 8 subjects. A quarter of the descriptive questions cannot be answered from the chart, which the authors say no earlier chart benchmark had done. No domain expertise is needed to answer the questions.
- **Results:** The evaluation covered 13 open-source and 11 proprietary MLLMs. On descriptive questions, GPT-4o scored 84.5%, InternVL Chat V1.5 scored 58.5% and humans scored 92.1%. On reasoning questions the gap between the best proprietary and best open model is 17.9 points, compared with about 0.6 on the older benchmarks.
- **Weak spots:** Counting labeled axis ticks is easy for humans, yet 20 of the 24 models scored below 10% on it. On charts with 6 or more subplots, descriptive accuracy fell by 30–50% for open models and by 10–30% for proprietary ones.
- **Subjects:** Descriptive accuracy is lowest on physics charts and higher on charts from economics, finance and electrical engineering. Reasoning accuracy shows no consistent pattern across subjects.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works

## Cite

Cite as: Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora and Danqi Chen. 2024. CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs. arXiv:2406.18521.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wang2024charxiv,
  title = {{CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs}},
  author = {Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora and Danqi Chen},
  year = {2024},
  doi = {10.48550/arxiv.2406.18521},
  eprint = {2406.18521},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.18521}
}
```

- [← previous Best Practices and Lessons Learned on Synthetic Data](https://md.research.elamur.ai/free/en/papers/liu-2024-best-practices-lessons-learned-synthetic/)
- [next → LLaVA-Video: Video Instruction Tuning With Synthetic Data](https://md.research.elamur.ai/free/en/papers/zhang-2024-llava-video-video-instruction-tuning/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

71 references, 47 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
2. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew B… — [Flamingo: a Visual Language Model for Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
3. Anthropic. The claude 3 model family: Opus, Sonnet, Haiku, March 2024. [find](https://scholar.google.com/scholar?q=Anthropic%2E+The+claude+3+model+family%3A+Opus%2C+Sonnet%2C+Haiku%2C+March+2024%2E)
4. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015. [find](https://scholar.google.com/scholar?q=Stanislaw+Antol%2C+Aishwarya+Agrawal%2C+Jiasen+Lu%2C+Margaret+Mitchell%2C+Dhruv+Batra%2C+C+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+VQA%3A+Visual+question+answering%2E+In+Proceedings+of+the+IEEE+international+conference+)
5. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. [Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/). arXiv preprint arXiv:2308.12966, 2023.
6. Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sağnak Taşırlar. Fuyu-8B: A multimodal architecture for ai agents, 2023. [find](https://scholar.google.com/scholar?q=Rohan+Bavishi%2C+Erich+Elsen%2C+Curtis+Hawthorne%2C+Maxwell+Nye%2C+Augustus+Odena%2C+Arushi+Somani%2C+and+Sag%CC%86nak+Tas%CC%A7%C4%B1rlar%2E+Fuyu%2D8B%3A+A+multimodal+architecture+for+ai+agents%2C+2023%2E)
7. Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. PaLI-X: On scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.18565, 2023. [find](https://scholar.google.com/scholar?q=Xi+Chen%2C+Josip+Djolonga%2C+Piotr+Padlewski%2C+Basil+Mustafa%2C+Soravit+Changpinyo%2C+Jialin+Wu%2C+Carlos+Riquelme+Ruiz%2C+Sebastian+Goodman%2C+Xiao+Wang%2C+Yi+Tay%2C+et+al%2E+PaLI%2DX%3A+On+scaling+up+a+multilingual+vision+a)
8. Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V Thapliyal, James Bradbury, Weicheng Kuo… — [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://md.research.elamur.ai/free/en/papers/chen-2022-pali-jointly-scaled-multilingual-language/)
9. Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. [How far are we to GPT-4V? closing the gap to commercial multimodal models with open-source suites](https://md.research.elamur.ai/free/en/papers/chen-2024-far-gpt-4v-closing/). arXiv preprint arXiv:2404.16821, 2024.
10. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+and+Eric+P%2E+Xing%2E+Vicuna%3A+An+open%2Dsource+chatbot+i)
11. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. [InstructBLIP: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/). In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
12. Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. InternLM-XComposer2… — [InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model](https://md.research.elamur.ai/free/en/papers/dong-2024-internlm-xcomposer2-mastering-free/)
13. Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingcheng Zhang, Jifeng Dai, Yu Qiao, Dahua Lin, and Jiaqi Wang. InternLM… — [InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD](https://md.research.elamur.ai/free/en/papers/dong-2024-internlm-xcomposer2-4khd-pioneering/)
14. Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. [Alpacafarm: A simulation framework for methods that learn from human feedback](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/). Advances in Neural Information Processing Systems, 36, 2024.
15. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. [MME: A comprehensive evaluation benchmark for multimodal large language models](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/). arXiv preprint arXiv:2306.13394, 2023.
16. Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, et al. SPHINX-X: Scaling data and parameters for a family of multi-modal large language models. arXiv preprint arXiv:2402.05935, 2024. — [arXiv:2402.05935](https://arxiv.org/abs/2402.05935) [find](https://scholar.google.com/scholar?q=Peng+Gao%2C+Renrui+Zhang%2C+Chris+Liu%2C+Longtian+Qiu%2C+Siyuan+Huang%2C+Weifeng+Lin%2C+Shitian+Zhao%2C+Shijie+Geng%2C+Ziyi+Lin%2C+Peng+Jin%2C+et+al%2E+SPHINX%2DX%3A+Scaling+data+and+parameters+for+a+family+of+multi%2Dmodal+larg)
17. Tianyu Gao, Zirui Wang, Adithya Bhaskar, and Danqi Chen. Improving language understanding from screenshots. arXiv preprint arXiv:2402.14073, 2024. [find](https://scholar.google.com/scholar?q=Tianyu+Gao%2C+Zirui+Wang%2C+Adithya+Bhaskar%2C+and+Danqi+Chen%2E+Improving+language+understanding+from+screenshots%2E+arXiv+preprint+arXiv%3A2402%2E14073%2C+2024%2E)
18. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets. Commun. ACM, 64(12):86–92, nov 2021. [find](https://scholar.google.com/scholar?q=Timnit+Gebru%2C+Jamie+Morgenstern%2C+Briana+Vecchione%2C+Jennifer+Wortman+Vaughan%2C+Hanna+Wallach%2C+Hal+Daum%C3%A9+III%2C+and+Kate+Crawford%2E+Datasheets+for+datasets%2E+Commun%2E+ACM%2C+64%2812%29%3A86%E2%80%9392%2C+nov+2021%2E)
19. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. [Making the V in VQA Matter: Elevating the role of image understanding in visual question answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017.
20. Ting-Yao Hsu, C Lee Giles, and Ting-Hao Huang. SciCap: Generating captions for scientific figures. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3258–3264, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ting%2DYao+Hsu%2C+C+Lee+Giles%2C+and+Ting%2DHao+Huang%2E+SciCap%3A+Generating+captions+for+scientific+figures%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2021%2C+pages+3258%E2%80%933264%2C+Punta+Cana)
21. Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, Xu Han, Yankai Lin, Jiao Xue, Dahai Li, Zhiyuan Liu, and Maosong Sun. [Large multilingual models pivot zero-shot multimodal learning across languages](https://md.research.elamur.ai/free/en/papers/hu-2023-large-multilingual-models-pivot-zero/). arXiv preprint arXiv:2308.12038, 20…
22. Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. Language Is Not All You Need: Aligning perception with lang… — [Language Is Not All You Need: Aligning Perception with Language Models](https://md.research.elamur.ai/free/en/papers/huang-2023-language-all-need-aligning-perception/)
23. Drew A Hudson and Christopher D Manning. [GQA: A new dataset for real-world visual reasoning and compositional question answering](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/). In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019.
24. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+Mistral+7b)
25. Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. DVQA: Understanding data visualizations via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648–5656, 2018. [find](https://scholar.google.com/scholar?q=Kushal+Kafle%2C+Brian+Price%2C+Scott+Cohen%2C+and+Christopher+Kanan%2E+DVQA%3A+Understanding+data+visualizations+via+question+answering%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+reco)
26. Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. [FigureQA: An annotated figure dataset for visual reasoning](https://md.research.elamur.ai/free/en/papers/kahou-2017-figureqa-annotated-figure-dataset-visual/). arXiv preprint arXiv:1710.07300, 2017.
27. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. [A diagram is worth a dozen images](https://md.research.elamur.ai/free/en/papers/kembhavi-2016-diagram-worth-dozen-images/). In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–251. Springer, 2016.
28. Hugo Laurençon, Lucile Saulnier, Leo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. [OBELICS: An open web-scale filtered dataset of interleaved image-text documents](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2023-obelics-open-web-scale-filtered/). In Thirty-seventh Conference…
29. Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. [What matters when building vision-language models?](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2024-matters-building-vision-language-models/) arXiv preprint arXiv:2405.02246, 2024.
30. Byung-Kwan Lee, Beomchan Park, Chae Won Kim, and Yong Man Ro. MoAI: Mixture of all intelligence for large language and vision models. arXiv preprint arXiv:2403.07508, 2024. — [arXiv:2403.07508](https://arxiv.org/abs/2403.07508) [find](https://scholar.google.com/scholar?q=Byung%2DKwan+Lee%2C+Beomchan+Park%2C+Chae+Won+Kim%2C+and+Yong+Man+Ro%2E+MoAI%3A+Mixture+of+all+intelligence+for+large+language+and+vision+models%2E+arXiv+preprint+arXiv%3A2403%2E07508%2C+2024%2E+%E2%80%94+%5BarXiv%3A2403%2E07508%5D%28https%3A)
31. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023. — [arXiv:2306.05425](https://arxiv.org/abs/2306.05425) [find](https://scholar.google.com/scholar?q=Bo+Li%2C+Yuanhan+Zhang%2C+Liangyu+Chen%2C+Jinghao+Wang%2C+Fanyi+Pu%2C+Jingkang+Yang%2C+Chunyuan+Li%2C+and+Ziwei+Liu%2E+Mimic%2Dit%3A+Multi%2Dmodal+in%2Dcontext+instruction+tuning%2E+arXiv+preprint+arXiv%3A2306%2E05425%2C+2023%2E+%E2%80%94+%5Bar)
32. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. [BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). In International conference on machine learning, pages 19730–19742. PMLR, 2023.
33. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. [BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/). In International conference on machine learning, pages 12888–12900. PMLR, 2022.
34. Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. [Multimodal ArXiv: A dataset for improving scientific comprehension of large vision-language models](https://md.research.elamur.ai/free/en/papers/li-2024-multimodal-arxiv-dataset-improving-scientific/). arXiv preprint arXiv:2403.00231, 2024.
35. Shengzhi Li and Nima Tajbakhsh. [SciGraphQA: A large-scale synthetic multi-turn question-answering dataset for scientific graphs](https://md.research.elamur.ai/free/en/papers/li-2023-scigraphqa-large-scale-synthetic-multi/). arXiv preprint arXiv:2308.03349, 2023.
36. Wangyue Li, Liangzhi Li, Tong Xiang, Xiao Liu, Wei Deng, and Noa Garcia. [Can multiple-choice questions really be useful in detecting the abilities of LLMs?](https://md.research.elamur.ai/free/en/papers/li-2024-can-multiple-choice-questions-really/) In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pa…
37. Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. [Mini-Gemini: Mining the potential of multi-modality vision language models](https://md.research.elamur.ai/free/en/papers/li-2024-mini-gemini-mining-potential-multi/). arXiv:2403.18814, 2024.
38. Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. SPHINX: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575, 2023. — [arXiv:2311.07575](https://arxiv.org/abs/2311.07575) [find](https://scholar.google.com/scholar?q=Ziyi+Lin%2C+Chris+Liu%2C+Renrui+Zhang%2C+Peng+Gao%2C+Longtian+Qiu%2C+Han+Xiao%2C+Han+Qiu%2C+Chen+Lin%2C+Wenqi+Shao%2C+Keqin+Chen%2C+et+al%2E+SPHINX%3A+The+joint+mixing+of+weights%2C+tasks%2C+and+visual+embeddings+for+multi%2Dmodal)
39. Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu. [MMC: Advancing multimodal chart understanding with large-scale instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2024-mmc-advancing-multimodal-chart-understanding/). In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Ling…
40. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296–26306, June 2024.
41. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaVA-NeXT: Improved reasoning, ocr, and world knowledge, January 2024. [find](https://scholar.google.com/scholar?q=Haotian+Liu%2C+Chunyuan+Li%2C+Yuheng+Li%2C+Bo+Li%2C+Yuanhan+Zhang%2C+Sheng+Shen%2C+and+Yong+Jae+Lee%2E+LLaVA%2DNeXT%3A+Improved+reasoning%2C+ocr%2C+and+world+knowledge%2C+January+2024%2E)
42. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). In NeurIPS, 2023.
43. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. [MMBench: Is your multi-modal model an all-around player?](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/) arXiv preprint arXiv:2307.06281, 2023.
44. Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. [DeepSeek-VL: Towards real-world vision-language understanding](https://md.research.elamur.ai/free/en/papers/lu-2024-deepseek-vl-real-world/). arXiv preprint arXiv:2403.05525, 2024.
45. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. [MathVista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). In International Conference on Learning Representations (ICLR), 2024.
46. Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to Explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Swaroop+Mishra%2C+Tony+Xia%2C+Liang+Qiu%2C+Kai%2DWei+Chang%2C+Song%2DChun+Zhu%2C+Oyvind+Tafjord%2C+Peter+Clark%2C+and+Ashwin+Kalyan%2E+Learn+to+Explain%3A+Multimodal+reasoning+via+thought+chains+for+science+questio)
47. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019. [find](https://scholar.google.com/scholar?q=Kenneth+Marino%2C+Mohammad+Rastegari%2C+Ali+Farhadi%2C+and+Roozbeh+Mottaghi%2E+OK%2DVQA%3A+A+visual+question+answering+benchmark+requiring+external+knowledge%2E+In+Proceedings+of+the+IEEE%2Fcvf+conference+on+computer)
48. Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. [ChartQA: A benchmark for question answering about charts with visual and logical reasoning](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/). In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, 2022.
49. Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. InfographicVQA. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697–1706, 2022. [find](https://scholar.google.com/scholar?q=Minesh+Mathew%2C+Viraj+Bagal%2C+Rub%C3%A8n+Tito%2C+Dimosthenis+Karatzas%2C+Ernest+Valveny%2C+and+CV+Jawahar%2E+InfographicVQA%2E+In+Proceedings+of+the+IEEE%2FCVF+Winter+Conference+on+Applications+of+Computer+Vision%2C+pages)
50. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. [DocVQA: A dataset for vqa on document images](https://md.research.elamur.ai/free/en/papers/mathew-2020-docvqa-dataset-vqa-document-images/). In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021.
51. Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra, and Pratyush Kumar. PlotQA: Reasoning over scientific plots. In The IEEE Winter Conference on Applications of Computer Vision (WACV), March 2020. [find](https://scholar.google.com/scholar?q=Nitesh+Methani%2C+Pritha+Ganguly%2C+Mitesh+M%2E+Khapra%2C+and+Pratyush+Kumar%2E+PlotQA%3A+Reasoning+over+scientific+plots%2E+In+The+IEEE+Winter+Conference+on+Applications+of+Computer+Vision+%28WACV%29%2C+March+2020%2E)
52. Aitor Ormazabal, Che Zheng, Cyprien de Masson d’Autume, Dani Yogatama, Deyu Fu, Donovan Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, et al. [Reka Core, Flash, and Edge: A series of powerful multimodal language models](https://md.research.elamur.ai/free/en/papers/ormazabal-2024-reka-core-flash-edge-series/). arXiv preprint arXiv:2404.12387, 2024.
53. Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia, July 2018. [Association for Computational…](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)
54. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. [find](https://scholar.google.com/scholar?q=Amanpreet+Singh%2C+Vivek+Natarajan%2C+Meet+Shah%2C+Yu+Jiang%2C+Xinlei+Chen%2C+Dhruv+Batra%2C+Devi+Parikh%2C+and+Marcus+Rohrbach%2E+Towards+vqa+models+that+can+read%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+comput)
55. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
56. Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. Label Studio: Data labeling software, 2020-2022. Open source software available from [https://github.com/heartexlabs/label-studio](https://github.com/heartexlabs/label-studio). [↗](https://github.com/heartexlabs/label-studio)
57. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
58. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. [CogVLM: Visual expert for pretrained language models](https://md.research.elamur.ai/free/en/papers/wang-2023-cogvlm-visual-expert-pretrained/). arXiv preprint arXiv:2311.03079, 2023.
59. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). In International Conference on Learning Representations, 2022.
60. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022. — [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
61. Renqiu Xia, Bo Zhang, Hancheng Ye, Xiangchao Yan, Qi Liu, Hongbin Zhou, Zijun Chen, Min Dou, Botian Shi, Junchi Yan, et al. ChartX & ChartVLM: A versatile benchmark and foundation model for complicated chart reasoning. arXiv preprint arXiv:2402.12185, 2024. — [arXiv:2402.12185](https://arxiv.org/abs/2402.12185) [find](https://scholar.google.com/scholar?q=Renqiu+Xia%2C+Bo+Zhang%2C+Hancheng+Ye%2C+Xiangchao+Yan%2C+Qi+Liu%2C+Hongbin+Zhou%2C+Zijun+Chen%2C+Min+Dou%2C+Botian+Shi%2C+Junchi+Yan%2C+et+al%2E+ChartX+%26+ChartVLM%3A+A+versatile+benchmark+and+foundation+model+for+complicate)
62. Zhengzhuo Xu, Sinan Du, Yiyan Qi, Chengjin Xu, Chun Yuan, and Jian Guo. ChartBench: A benchmark for complex visual reasoning in charts. arXiv preprint arXiv:2312.15915, 2023. — [arXiv:2312.15915](https://arxiv.org/abs/2312.15915) [find](https://scholar.google.com/scholar?q=Zhengzhuo+Xu%2C+Sinan+Du%2C+Yiyan+Qi%2C+Chengjin+Xu%2C+Chun+Yuan%2C+and+Jian+Guo%2E+ChartBench%3A+A+benchmark+for+complex+visual+reasoning+in+charts%2E+arXiv+preprint+arXiv%3A2312%2E15915%2C+2023%2E+%E2%80%94+%5BarXiv%3A2312%2E15915%5D%28http)
63. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. [mPLUG-Owl: Modularization empowers large language models with multimodality](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/). arXiv preprint arXiv:2304.14178, 2023.
64. Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.[04652, 2024](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/).
65. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. [MM-Vet: Evaluating large multimodal models for integrated capabilities](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/). arXiv preprint arXiv:2308.02490, 2023.
66. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: A massive multi-discipline… — [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/)
67. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. [Sigmoid loss for language image pre-training](https://md.research.elamur.ai/free/en/papers/zhai-2023-sigmoid-loss-language-image-pre/). In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023.
68. Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. LLaMA-adapter: Efficient fine-tuning of large language models with zero-initialized attention. In The Twelfth International Conference on Learning Representations, 2024. [find](https://scholar.google.com/scholar?q=Renrui+Zhang%2C+Jiaming+Han%2C+Chris+Liu%2C+Aojun+Zhou%2C+Pan+Lu%2C+Yu+Qiao%2C+Hongsheng+Li%2C+and+Peng+Gao%2E+LLaMA%2Dadapter%3A+Efficient+fine%2Dtuning+of+large+language+models+with+zero%2Dinitialized+attention%2E+In+The+Twe)
69. Zhuosheng Zhang, Aston Zhang, Mu Li, hai zhao, George Karypis, and Alex Smola. [Multimodal chain-of-thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/zhang-2023-multimodal-chain-thought-reasoning-language/). Transactions on Machine Learning Research, 2024.
70. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. [WebArena: A realistic web environment for building autonomous agents](https://md.research.elamur.ai/free/en/papers/zhou-2023-webarena-realistic-web-environment-building/). In The Twelfth International Conference on Learning Representations, 202…
71. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. [MiniGPT-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/). arXiv preprint arXiv:2304.10592, 2023. Contents \#\# Introduction Existing Benchmarks Overestimate Chart Understanding Capabilities 2.1 Related W…

*Updated 2 November 2026*