---
title: "We-Math — paper by Qiao et al. (2024) — summary and ideas"
description: "We-Math tries to fix this by grounding evaluation in textbook knowledge concepts. 109 references, cited by 65 papers here."
authors: ["Runqi Qiao", "Qiuna Tan", "Guanting Dong", "Minhui Wu", "Chong Sun", "Xiaoshuai Song", "Zhuoma GongQue", "Shanglin Lei", "Zhe Wei", "Miaoxuan Zhang", "Runfeng Qiao", "Yifan Zhang", "Xiao Zong", "Yida Xu", "Muxi Diao", "Zhimin Bao", "Chen Li", "Honggang Zhang"]
year: 2024
venue: "arXiv:2407.01284"
source: "https://arxiv.org/abs/2407.01284"
doi: "10.48550/arxiv.2407.01284"
arxiv: "2407.01284"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/qiao-2024-math-large-multimodal-model-achieve/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/qiao-2024-math-large-multimodal-model-achieve/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?

# We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?

**Authors:** Runqi Qiao, Qiuna Tan, [Guanting Dong](https://md.research.elamur.ai/free/en/authors/guanting-dong/), Minhui Wu, Chong Sun, [Xiaoshuai Song](https://md.research.elamur.ai/free/en/authors/xiaoshuai-song/), Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, Runfeng Qiao, [Yifan Zhang](https://md.research.elamur.ai/free/en/authors/yifan-zhang/), Xiao Zong, Yida Xu, Muxi Diao, Zhimin Bao, [Chen Li](https://md.research.elamur.ai/free/en/authors/chen-li/) and [Honggang Zhang](https://md.research.elamur.ai/free/en/authors/honggang-zhang/) · **Year:** 2024 · **Venue:** arXiv:2407.01284 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2407.01284) · [PDF](https://arxiv.org/pdf/2407.01284) · [DOI 10.48550/arxiv.2407.01284](https://doi.org/10.48550/arxiv.2407.01284) · [arXiv 2407.01284](https://arxiv.org/abs/2407.01284) · [OpenAlex](https://openalex.org/W4400337724)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [65](https://md.research.elamur.ai/free/en/papers/qiao-2024-math-large-multimodal-model-achieve/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most benchmarks for visual math in large multimodal models (LMMs) only check whether the final answer is right. That hides how a model actually reasons. We-Math tries to fix this by grounding evaluation in textbook knowledge concepts. The authors built about 6.5K visual math problems, mostly elementary-level geometry, measurement and spatial tasks, organised into a 5-layer hierarchy covering 67 concepts. Expert annotators split roughly 1.5K multi-concept problems into one-step sub-problems. This lets the authors compare a model's success on the full problem with its success on each component. They then place each result in one of four diagnostic categories: IK, where the model lacks the needed knowledge; IG, where it handles the parts but fails to generalize to the whole; CM, where it fully masters both; and RM, where it solves the whole by rote while missing the parts. They also test a knowledge concept augmentation (KCA) strategy that adds concept descriptions to the prompt. The main findings are that accuracy drops as the number of required steps grows, and that IK is the dominant weakness of most models. GPT-4o is the exception: its main gap has moved from IK to IG. The benchmark matters because it shows when correct answers do not reflect genuine understanding.

## Key points
- **Dataset scale:** 6,524 newly collected questions and 6,307 images, arranged in a knowledge system of 99 nodes over 5 layers that includes 67 leaf concepts. Data and code are public at github.com/We-Math/We-Math.
- **Model comparison:** GPT-4o performs best overall. Closed models and large open ones such as LLaVA-NeXT-110B also do well. Fine-grained visual measurement ("Angles and Length") is the hardest category for most models.
- **IK and RM findings:** Small models such as LLaVA-1.6-7B and DeepSeek-VL-1.3B each show more than 350 IK cases. G-LLaVA-13B reaches about 36% RM under the loose metric, meaning it solves composite problems while failing their sub-problems.
- **KCA results:** KCA uses concept descriptions drawn from Wikipedia and textbooks. It reliably reduces IK across model sizes but barely changes IG.
- **GPT-4o error analysis:** Knowledge errors appear in more than 45 concepts. Visual errors cluster in a few concepts, such as protractor-based angle reading, and are absent from over 38 concepts. This points to the need for better fine-grained visual encoders.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [➗ Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/) · 189 works · [➗ Applied Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/applied-mathematics/) · 52 works

🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works

## Cite

Cite as: Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, Runfeng Qiao, Yifan Zhang, Xiao Zong, Yida Xu, Muxi Diao, Zhimin Bao, Chen Li and Honggang Zhang. 2024. We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?. arXiv:2407.01284.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{qiao2024we,
  title = {{We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?}},
  author = {Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, Runfeng Qiao, Yifan Zhang, Xiao Zong, Yida Xu, Muxi Diao, Zhimin Bao, Chen Li and Honggang Zhang},
  year = {2024},
  doi = {10.48550/arxiv.2407.01284},
  eprint = {2407.01284},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.01284}
}
```

- [← previous Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models](https://md.research.elamur.ai/free/en/papers/li-2024-synthetic-data-almost-scratch-generalized/)
- [next → A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity](https://md.research.elamur.ai/free/en/papers/lee-2024-mechanistic-understanding-alignment-algorithms-case/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

109 references, 58 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436–444, 2015. [find](https://scholar.google.com/scholar?q=Yann+LeCun%2C+Yoshua+Bengio%2C+and+Geoffrey+Hinton%2E+Deep+learning%2E+nature%2C+521%287553%29%3A436%E2%80%93444%2C+2015%2E)
2. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. [find](https://scholar.google.com/scholar?q=Yann+LeCun%2C+L%C3%A9on+Bottou%2C+Yoshua+Bengio%2C+and+Patrick+Haffner%2E+Gradient%2Dbased+learning+applied+to+document+recognition%2E+Proceedings+of+the+IEEE%2C+86%2811%29%3A2278%E2%80%932324%2C+1998%2E)
3. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
4. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training la… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
5. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
6. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
7. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403, 2023.
8. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). Advances in neural information processing systems, 36, 2024.
9. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. [Instructblip: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/). Advances in Neural Information Processing Systems, 36, 2024.
10. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. [Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). In International conference on machine learning, pages 19730–19742. PMLR, 2023.
11. Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. [Llama-adapter: Efficient fine-tuning of language models with zero-init attention](https://md.research.elamur.ai/free/en/papers/zhang-2023-llama-adapter-efficient-fine-tuning/). arXiv preprint arXiv:2303.16199, 2023.
12. Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. [Llama-adapter v2: Parameter-efficient visual instruction model](https://md.research.elamur.ai/free/en/papers/gao-2023-llama-adapter-v2-parameter-efficient/). arXiv preprint arXiv:2304.15010, 2023.
13. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. [Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/), 2023.
14. Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. [Pandagpt: One model to instruction-follow them all](https://md.research.elamur.ai/free/en/papers/su-2023-pandagpt-one-model-instruction-follow/). arXiv preprint arXiv:2305.16355, 2023.
15. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. [mplug-owl: Modularization empowers large language models with multimodality](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/). arXiv preprint arXiv:2304.14178, 2023.
16. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. [Minigpt-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/). arXiv preprint arXiv:2304.10592, 2023.
17. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/), 2023.
18. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. CoRR, 2022. [find](https://scholar.google.com/scholar?q=Wenhu+Chen%2C+Xueguang+Ma%2C+Xinyi+Wang%2C+and+William+W%2E+Cohen%2E+Program+of+thoughts+prompting%3A+Disentangling+computation+from+reasoning+for+numerical+reasoning+tasks%2E+CoRR%2C+2022%2E)
19. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: program-aided language models. In Proceedings of the 40th International Conference on Machine Learning, pages 10764–10799, 2023. [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Pengfei+Liu%2C+Yiming+Yang%2C+Jamie+Callan%2C+and+Graham+Neubig%2E+Pal%3A+program%2Daided+language+models%2E+In+Proceedings+of+the+40th+International+Conference+on+Mach)
20. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. [Tora: A tool-integrated reasoning agent for mathematical problem solving](https://md.research.elamur.ai/free/en/papers/gou-2023-tora-tool-integrated-reasoning-agent/), 2024.
21. Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. [Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning](https://md.research.elamur.ai/free/en/papers/wang-2023-mathcoder-seamless-code-integration-llms/). CoRR, 2023.
22. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. [Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/), 2023.
23. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. [Scaling relationship on learning mathematical reasoning with large language models](https://md.research.elamur.ai/free/en/papers/yuan-2023-scaling-relationship-learning-mathematical-reasoning/), 2023.
24. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/), 2024.
25. Chengpeng Li, Zheng Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang, and Chang Zhou. Query and response augmentation cannot help out-of-domain math reasoning generalization. arXiv preprint arXiv:2310.05506, 2023. [find](https://scholar.google.com/scholar?q=Chengpeng+Li%2C+Zheng+Yuan%2C+Guanting+Dong%2C+Keming+Lu%2C+Jiancan+Wu%2C+Chuanqi+Tan%2C+Xiang+Wang%2C+and+Chang+Zhou%2E+Query+and+response+augmentation+cannot+help+out%2Dof%2Ddomain+math+reasoning+generalization%2E+arXiv+)
26. R OpenAI. Gpt-4v (ision) system card. Citekey: gptvision, 2023. [find](https://scholar.google.com/scholar?q=R+OpenAI%2E+Gpt%2D4v+%28ision%29+system+card%2E+Citekey%3A+gptvision%2C+2023%2E)
27. R. Gemini Team. [Gemini: A family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/), 2024.
28. Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.[17421, 9(1):1, 2023](https://md.research.elamur.ai/free/en/papers/yang-2023-dawn-lmms-preliminary-explorations-gpt/).
29. Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, et al. [G-llava: Solving geometric problem with multi-modal large language model](https://md.research.elamur.ai/free/en/papers/gao-2023-g-llava-solving-geometric-problem/). arXiv preprint arXiv:2312.11370, 2023.
30. Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. [Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning](https://md.research.elamur.ai/free/en/papers/lu-2021-inter-gps-interpretable-geometry-problem/). arXiv preprint arXiv:2105.04165, 2021.
31. Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. [Solving geometry problems: Combining text and diagram interpretation](https://md.research.elamur.ai/free/en/papers/seo-2015-solving-geometry-problems-combining-text/). In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 1466–1476, 2015.
32. Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P Xing, and Liang Lin. [Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning](https://md.research.elamur.ai/free/en/papers/chen-2021-geoqa-geometric-question-answering-benchmark/). arXiv preprint arXiv:2105.14517, 2021.
33. Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. [Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression](https://md.research.elamur.ai/free/en/papers/chen-2022-unigeo-unifying-geometry-logical-reasoning/). arXiv preprint arXiv:2212.02746, 2022.
34. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. [Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). arXiv preprint arXiv:2310.02255, 2023.
35. Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. [Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?](https://md.research.elamur.ai/free/en/papers/zhang-2024-mathverse-multi-modal-llm-truly/) arXiv preprint arXiv:2403.14624, 2024.
36. Richard Fitzpatrick. Euclid’s elements of geometry, 2008. [find](https://scholar.google.com/scholar?q=Richard+Fitzpatrick%2E+Euclid%E2%80%99s+elements+of+geometry%2C+2008%2E)
37. Wikipedia contributors. Wikipedia, 2004. [find](https://scholar.google.com/scholar?q=Wikipedia+contributors%2E+Wikipedia%2C+2004%2E)
38. OpenAI. Hello gpt-4o, 2024. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Hello+gpt%2D4o%2C+2024%2E)
39. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. [find](https://scholar.google.com/scholar?q=Haotian+Liu%2C+Chunyuan+Li%2C+Yuheng+Li%2C+Bo+Li%2C+Yuanhan+Zhang%2C+Sheng+Shen%2C+and+Yong+Jae+Lee%2E+Llava%2Dnext%3A+Improved+reasoning%2C+ocr%2C+and+world+knowledge%2C+January+2024%2E)
40. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
41. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/), 2023.
42. Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. [Deepseek-vl: Towards real-world vision-language understanding](https://md.research.elamur.ai/free/en/papers/lu-2024-deepseek-vl-real-world/), 2024.
43. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. [Phi-3 technical report: A highly capable language model locally on your phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/). arXiv preprint arXiv:2404.14219, 2024.
44. Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, et al. [Large multilingual models pivot zero-shot multimodal learning across languages](https://md.research.elamur.ai/free/en/papers/hu-2023-large-multilingual-models-pivot-zero/). arXiv preprint arXiv:2308.12038, 2023.
45. Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. Internlm-xcomposer2… — [InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model](https://md.research.elamur.ai/free/en/papers/dong-2024-internlm-xcomposer2-mastering-free/)
46. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. [Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks](https://md.research.elamur.ai/free/en/papers/chen-2023-internvl-scaling-up-vision/). arXiv preprint arXiv:2312.14238, 2023.
47. Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinka… — [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://md.research.elamur.ai/free/en/papers/glm-2024-chatglm-family-large-language-models/)
48. Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. [Long context transfer from language to vision](https://md.research.elamur.ai/free/en/papers/zhang-2024-long-context-transfer-language-vision/), 2024.
49. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. [MathQA: Towards interpretable math word problem solving with operation-based formalisms](https://md.research.elamur.ai/free/en/papers/amini-2019-mathqa-interpretable-math-word-problem/). In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the Nort…
50. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
51. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874, 2021.
52. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/), 2021.
53. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/), 2023.
54. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. [Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/). arXiv preprint arXiv:2311.16502, 2023.
55. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. [C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/), 2023.
56. Guanting Dong, Jinxu Zhao, Tingfeng Hui, Daichi Guo, Wenlong Wan, Boqi Feng, Yueyan Qiu, Zhuoma Gongque, Keqing He, Zechen Wang, and Weiran Xu. Revisit input perturbation problems for llms: A unified robustness evaluation framework for noisy slot filling task, 2023. [find](https://scholar.google.com/scholar?q=Guanting+Dong%2C+Jinxu+Zhao%2C+Tingfeng+Hui%2C+Daichi+Guo%2C+Wenlong+Wan%2C+Boqi+Feng%2C+Yueyan+Qiu%2C+Zhuoma+Gongque%2C+Keqing+He%2C+Zechen+Wang%2C+and+Weiran+Xu%2E+Revisit+input+perturbation+problems+for+llms%3A+A+unified+)
57. Xiaoshuai Song, Muxi Diao, Guanting Dong, Zhengyang Wang, Yujia Fu, Runqi Qiao, Zhexu Wang, Dayuan Fu, Huangxuan Wu, Bin Liang, Weihao Zeng, Yejie Wang, Zhuoma GongQue, Jianing Yu, Qiuna Tan, and Weiran Xu. Cs-bench: A comprehensive benchmark for large language models towards computer science master… [find](https://scholar.google.com/scholar?q=Xiaoshuai+Song%2C+Muxi+Diao%2C+Guanting+Dong%2C+Zhengyang+Wang%2C+Yujia+Fu%2C+Runqi+Qiao%2C+Zhexu+Wang%2C+Dayuan+Fu%2C+Huangxuan+Wu%2C+Bin+Liang%2C+Weihao+Zeng%2C+Yejie+Wang%2C+Zhuoma+GongQue%2C+Jianing+Yu%2C+Qiuna+Tan%2C+and+Weir)
58. Xiaoshuai Song, Keqing He, Pei Wang, Guanting Dong, Yutao Mou, Jingang Wang, Yunsen Xian, Xunliang Cai, and Weiran Xu. Large language models meet open-world intent discovery and recognition: An evaluation of chatgpt, 2023. [find](https://scholar.google.com/scholar?q=Xiaoshuai+Song%2C+Keqing+He%2C+Pei+Wang%2C+Guanting+Dong%2C+Yutao+Mou%2C+Jingang+Wang%2C+Yunsen+Xian%2C+Xunliang+Cai%2C+and+Weiran+Xu%2E+Large+language+models+meet+open%2Dworld+intent+discovery+and+recognition%3A+An+evalua)
59. Guanting Dong, Yutao Zhu, Chenghao Zhang, Zechen Wang, Zhicheng Dou, and Ji-Rong Wen. Understand what llm needs: Dual preference alignment for retrieval-augmented generation, 2024. [find](https://scholar.google.com/scholar?q=Guanting+Dong%2C+Yutao+Zhu%2C+Chenghao+Zhang%2C+Zechen+Wang%2C+Zhicheng+Dou%2C+and+Ji%2DRong+Wen%2E+Understand+what+llm+needs%3A+Dual+preference+alignment+for+retrieval%2Daugmented+generation%2C+2024%2E)
60. Mingfeng Xue, Dayiheng Liu, Kexin Yang, Guanting Dong, Wenqiang Lei, Zheng Yuan, Chang Zhou, and Jingren Zhou. Occuquest: Mitigating occupational bias for inclusive large language models, 2023. [find](https://scholar.google.com/scholar?q=Mingfeng+Xue%2C+Dayiheng+Liu%2C+Kexin+Yang%2C+Guanting+Dong%2C+Wenqiang+Lei%2C+Zheng+Yuan%2C+Chang+Zhou%2C+and+Jingren+Zhou%2E+Occuquest%3A+Mitigating+occupational+bias+for+inclusive+large+language+models%2C+2023%2E)
61. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F%2C+2019%2E)
62. Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. [Superglue: Learning feature matching with graph neural networks](https://md.research.elamur.ai/free/en/papers/sarlin-2019-superglue-learning-feature-matching-graph/), 2020.
63. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015. [find](https://scholar.google.com/scholar?q=Stanislaw+Antol%2C+Aishwarya+Agrawal%2C+Jiasen+Lu%2C+Margaret+Mitchell%2C+Dhruv+Batra%2C+C+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+Vqa%3A+Visual+question+answering%2E+In+Proceedings+of+the+IEEE+international+conference+)
64. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. [Making the v in vqa matter: Elevating the role of image understanding in visual question answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017.
65. Kushal Kafle and Christopher Kanan. An analysis of visual question answering algorithms. In Proceedings of the IEEE international conference on computer vision, pages 1965–1973, 2017. [find](https://scholar.google.com/scholar?q=Kushal+Kafle+and+Christopher+Kanan%2E+An+analysis+of+visual+question+answering+algorithms%2E+In+Proceedings+of+the+IEEE+international+conference+on+computer+vision%2C+pages+1965%E2%80%931973%2C+2017%2E)
66. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. [find](https://scholar.google.com/scholar?q=Amanpreet+Singh%2C+Vivek+Natarajan%2C+Meet+Shah%2C+Yu+Jiang%2C+Xinlei+Chen%2C+Dhruv+Batra%2C+Devi+Parikh%2C+and+Marcus+Rohrbach%2E+Towards+vqa+models+that+can+read%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+comput)
67. Drew A Hudson and Christopher D Manning. [Gqa: A new dataset for real-world visual reasoning and compositional question answering](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/). In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019.
68. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. [Docvqa: A dataset for vqa on document images](https://md.research.elamur.ai/free/en/papers/mathew-2020-docvqa-dataset-vqa-document-images/). In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021.
69. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019. [find](https://scholar.google.com/scholar?q=Kenneth+Marino%2C+Mohammad+Rastegari%2C+Ali+Farhadi%2C+and+Roozbeh+Mottaghi%2E+Ok%2Dvqa%3A+A+visual+question+answering+benchmark+requiring+external+knowledge%2E+In+Proceedings+of+the+IEEE%2Fcvf+conference+on+computer)
70. Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4291–4301, 2019. [find](https://scholar.google.com/scholar?q=Ali+Furkan+Biten%2C+Ruben+Tito%2C+Andres+Mafla%2C+Lluis+Gomez%2C+Mar%C3%A7al+Rusinol%2C+Ernest+Valveny%2C+CV+Jawahar%2C+and+Dimosthenis+Karatzas%2E+Scene+text+visual+question+answering%2E+In+Proceedings+of+the+IEEE%2FCVF+inte)
71. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. [Mm-vet: Evaluating large multimodal models for integrated capabilities](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/). arXiv preprint arXiv:2308.02490, 2023.
72. Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947–952. IEEE, 2019. [find](https://scholar.google.com/scholar?q=Anand+Mishra%2C+Shashank+Shekhar%2C+Ajeet+Kumar+Singh%2C+and+Anirban+Chakraborty%2E+Ocr%2Dvqa%3A+Visual+question+answering+by+reading+text+in+images%2E+In+2019+international+conference+on+document+analysis+and+reco)
73. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. [Mme: A comprehensive evaluation benchmark for multimodal large language models](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/). arXiv preprint arXiv:2306.13394, 2024.
74. Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and techno… [find](https://scholar.google.com/scholar?q=Jeffrey+P+Bigham%2C+Chandrika+Jayant%2C+Hanjie+Ji%2C+Greg+Little%2C+Andrew+Miller%2C+Robert+C+Miller%2C+Robin+Miller%2C+Aubrey+Tatarowicz%2C+Brandyn+White%2C+Samual+White%2C+et+al%2E+Vizwiz%3A+nearly+real%2Dtime+answers+to+vis)
75. Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022. [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Swaroop+Mishra%2C+Tony+Xia%2C+Liang+Qiu%2C+Kai%2DWei+Chang%2C+Song%2DChun+Zhu%2C+Oyvind+Tafjord%2C+Peter+Clark%2C+and+Ashwin+Kalyan%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+science+questio)
76. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–…
77. Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8948–8957, 2019. [find](https://scholar.google.com/scholar?q=Harsh+Agrawal%2C+Karan+Desai%2C+Yufei+Wang%2C+Xinlei+Chen%2C+Rishabh+Jain%2C+Mark+Johnson%2C+Dhruv+Batra%2C+Devi+Parikh%2C+Stefan+Lee%2C+and+Peter+Anderson%2E+Nocaps%3A+Novel+object+captioning+at+scale%2E+In+Proceedings+of+t)
78. Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641–2649, 2015… [find](https://scholar.google.com/scholar?q=Bryan+A+Plummer%2C+Liwei+Wang%2C+Chris+M+Cervantes%2C+Juan+C+Caicedo%2C+Julia+Hockenmaier%2C+and+Svetlana+Lazebnik%2E+Flickr30k+entities%3A+Collecting+region%2Dto%2Dphrase+correspondences+for+richer+image%2Dto%2Dsentence+m)
79. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. [Mmbench: Is your multi-modal model an all-around player?](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/) arXiv preprint arXiv:2307.06281, 2023.
80. Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. [Seed-bench: Benchmarking multimodal llms with generative comprehension](https://md.research.elamur.ai/free/en/papers/li-2023-seed-bench-benchmarking-multimodal-llms/). arXiv preprint arXiv:2307.16125, 2023.
81. Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299–13308, 2024. [find](https://scholar.google.com/scholar?q=Bohao+Li%2C+Yuying+Ge%2C+Yixiao+Ge%2C+Guangzhi+Wang%2C+Rui+Wang%2C+Ruimao+Zhang%2C+and+Ying+Shan%2E+Seed%2Dbench%3A+Benchmarking+multimodal+large+language+models%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+V)
82. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. [Are we on the right way for evaluating large vision-language models?](https://md.research.elamur.ai/free/en/papers/chen-2024-right-way-evaluating-large-vision/) arXiv preprint arXiv:2403.20330, 2024.
83. Poorav Desai, Tanmoy Chakraborty, and Md Shad Akhtar. Nice perfume. how long did you marinate in it? multimodal sarcasm explanation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10563–10571, 2022. [find](https://scholar.google.com/scholar?q=Poorav+Desai%2C+Tanmoy+Chakraborty%2C+and+Md+Shad+Akhtar%2E+Nice+perfume%2E+how+long+did+you+marinate+in+it%3F+multimodal+sarcasm+explanation%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+v)
84. Jack Hessel, Ana Marasović, Jena D Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi. Do androids laugh at electric sheep? humor" understanding" benchmarks from the new yorker caption contest. arXiv preprint arXiv:2209.[06293, 2022](https://md.research.elamur.ai/free/en/papers/hessel-2023-do-androids-laugh-electric-sheep/).
85. Winnie Street, John Oliver Siy, Geoff Keeling, Adrien Baranes, Benjamin Barnett, Michael McKibben, Tatenda Kanyere, Alison Lentz, Robin IM Dunbar, et al. Llms achieve adult human performance on higher-order theory of mind tasks. arXiv preprint arXiv:2405.18870, 2024. — [arXiv:2405.18870](https://arxiv.org/abs/2405.18870) [find](https://scholar.google.com/scholar?q=Winnie+Street%2C+John+Oliver+Siy%2C+Geoff+Keeling%2C+Adrien+Baranes%2C+Benjamin+Barnett%2C+Michael+McKibben%2C+Tatenda+Kanyere%2C+Alison+Lentz%2C+Robin+IM+Dunbar%2C+et+al%2E+Llms+achieve+adult+human+performance+on+higher)
86. Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou. Self-play with execution feedback: Improving instruction-following capabilities of large language models, 2024. [find](https://scholar.google.com/scholar?q=Guanting+Dong%2C+Keming+Lu%2C+Chengpeng+Li%2C+Tingyu+Xia%2C+Bowen+Yu%2C+Chang+Zhou%2C+and+Jingren+Zhou%2E+Self%2Dplay+with+execution+feedback%3A+Improving+instruction%2Dfollowing+capabilities+of+large+language+models%2C+20)
87. Ziqiang Liu, Feiteng Fang, Xi Feng, Xinrun Du, Chenhao Zhang, Zekun Wang, Yuelin Bai, Qixuan Zhao, Liyang Fan, Chengguang Gan, et al. Ii-bench: An image implication understanding benchmark for multimodal large language models. arXiv preprint arXiv:2406.05862, 2024. — [arXiv:2406.05862](https://arxiv.org/abs/2406.05862) [find](https://scholar.google.com/scholar?q=Ziqiang+Liu%2C+Feiteng+Fang%2C+Xi+Feng%2C+Xinrun+Du%2C+Chenhao+Zhang%2C+Zekun+Wang%2C+Yuelin+Bai%2C+Qixuan+Zhao%2C+Liyang+Fan%2C+Chengguang+Gan%2C+et+al%2E+Ii%2Dbench%3A+An+image+implication+understanding+benchmark+for+multimo)
88. Chenshuang Zhang, Fei Pan, Junmo Kim, In So Kweon, and Chengzhi Mao. Imagenet-d: Benchmarking neural network robustness on diffusion synthetic object, 2024. [find](https://scholar.google.com/scholar?q=Chenshuang+Zhang%2C+Fei+Pan%2C+Junmo+Kim%2C+In+So+Kweon%2C+and+Chengzhi+Mao%2E+Imagenet%2Dd%3A+Benchmarking+neural+network+robustness+on+diffusion+synthetic+object%2C+2024%2E)
89. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248–255, 2009. [find](https://scholar.google.com/scholar?q=Jia+Deng%2C+Wei+Dong%2C+Richard+Socher%2C+Li%2DJia+Li%2C+Kai+Li%2C+and+Li+Fei%2DFei%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+2009+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pages+248%E2%80%932)
90. Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303–338, 2010. [find](https://scholar.google.com/scholar?q=Mark+Everingham%2C+Luc+Van+Gool%2C+Christopher+KI+Williams%2C+John+Winn%2C+and+Andrew+Zisserman%2E+The+pascal+visual+object+classes+%28voc%29+challenge%2E+International+journal+of+computer+vision%2C+88%282%29%3A303%E2%80%93338%2C+2010)
91. Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912–1920, 2015. [find](https://scholar.google.com/scholar?q=Zhirong+Wu%2C+Shuran+Song%2C+Aditya+Khosla%2C+Fisher+Yu%2C+Linguang+Zhang%2C+Xiaoou+Tang%2C+and+Jianxiong+Xiao%2E+3d+shapenets%3A+A+deep+representation+for+volumetric+shapes%2E+In+Proceedings+of+the+IEEE+conference+on+)
92. Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository, 2015. [find](https://scholar.google.com/scholar?q=Angel+X%2E+Chang%2C+Thomas+Funkhouser%2C+Leonidas+Guibas%2C+Pat+Hanrahan%2C+Qixing+Huang%2C+Zimo+Li%2C+Silvio+Savarese%2C+Manolis+Savva%2C+Shuran+Song%2C+Hao+Su%2C+Jianxiong+Xiao%2C+Li+Yi%2C+and+Fisher+Yu%2E+Shapenet%3A+An+informa)
93. Andreas Geiger, Philipp Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. International Journal of Robotics Research (IJRR), 32(11):1231–1237, 2013. [find](https://scholar.google.com/scholar?q=Andreas+Geiger%2C+Philipp+Lenz%2C+Christoph+Stiller%2C+and+Raquel+Urtasun%2E+Vision+meets+robotics%3A+The+kitti+dataset%2E+International+Journal+of+Robotics+Research+%28IJRR%29%2C+32%2811%29%3A1231%E2%80%931237%2C+2013%2E)
94. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210. IEEE, 2015. [find](https://scholar.google.com/scholar?q=Vassil+Panayotov%2C+Guoguo+Chen%2C+Daniel+Povey%2C+and+Sanjeev+Khudanpur%2E+Librispeech%3A+an+asr+corpus+based+on+public+domain+audio+books%2E+In+2015+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal)
95. Christophe Veaux, Junichi Yamagishi, and Kirsten MacDonald. The voice bank corpus: Design, collection and data analysis of a large regional accent speech database. In 12th Annual Conference of the International Speech Communication Association (Interspeech), pages 121–125, 2017. [find](https://scholar.google.com/scholar?q=Christophe+Veaux%2C+Junichi+Yamagishi%2C+and+Kirsten+MacDonald%2E+The+voice+bank+corpus%3A+Design%2C+collection+and+data+analysis+of+a+large+regional+accent+speech+database%2E+In+12th+Annual+Conference+of+the+Int)
96. Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. [Air-bench: Benchmarking large audio-language models via generative comprehension](https://md.research.elamur.ai/free/en/papers/yang-2024-air-bench-benchmarking-large-audio/), 2024.
97. Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy F. Chen. [Audiobench: A universal benchmark for audio large language models](https://md.research.elamur.ai/free/en/papers/wang-2025-audiobench-universal-benchmark-audio-large/), 2024.
98. Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models, 2023. [find](https://scholar.google.com/scholar?q=Munan+Ning%2C+Bin+Zhu%2C+Yujia+Xie%2C+Bin+Lin%2C+Jiaxi+Cui%2C+Lu+Yuan%2C+Dongdong+Chen%2C+and+Li+Yuan%2E+Video%2Dbench%3A+A+comprehensive+benchmark+and+toolkit+for+evaluating+video%2Dbased+large+language+models%2C+2023%2E)
99. Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. Youtube-8m: A large-scale video classification benchmark. In arXiv preprint arXiv:1609.08675, 2016. — [arXiv:1609.08675](https://arxiv.org/abs/1609.08675) [find](https://scholar.google.com/scholar?q=Sami+Abu%2DEl%2DHaija%2C+Nisarg+Kothari%2C+Joonseok+Lee%2C+Paul+Natsev%2C+George+Toderici%2C+Balakrishnan+Varadarajan%2C+and+Sudheendra+Vijayanarasimhan%2E+Youtube%2D8m%3A+A+large%2Dscale+video+classification+benchmark%2E+In+a)
100. Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Vbench: Comprehensive benchmark suite for video generative models, 2023. [find](https://scholar.google.com/scholar?q=Ziqi+Huang%2C+Yinan+He%2C+Jiashuo+Yu%2C+Fan+Zhang%2C+Chenyang+Si%2C+Yuming+Jiang%2C+Yuanhan+Zhang%2C+Tianxing+Wu%2C+Qingyang+Jin%2C+Nattapol+Chanpaisit%2C+Yaohui+Wang%2C+Xinyuan+Chen%2C+Limin+Wang%2C+Dahua+Lin%2C+Yu+Qiao%2C+and+Zi)
101. Junlin Xie, Zhihong Chen, Ruifei Zhang, Xiang Wan, and Guanbin Li. Large multimodal agents: A survey. arXiv preprint arXiv:2402.15116, 2024. — [arXiv:2402.15116](https://arxiv.org/abs/2402.15116) [find](https://scholar.google.com/scholar?q=Junlin+Xie%2C+Zhihong+Chen%2C+Ruifei+Zhang%2C+Xiang+Wan%2C+and+Guanbin+Li%2E+Large+multimodal+agents%3A+A+survey%2E+arXiv+preprint+arXiv%3A2402%2E15116%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E15116%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E15116%29)
102. Chenyu Wang, Weixin Luo, Qianyu Chen, Haonan Mai, Jindi Guo, Sixun Dong, Xiaohua, Xuan, Zhengxin Li, Lin Ma, and Shenghua Gao. Mllm-tool: A multimodal large language model for tool agent learning, 2024. [find](https://scholar.google.com/scholar?q=Chenyu+Wang%2C+Weixin+Luo%2C+Qianyu+Chen%2C+Haonan+Mai%2C+Jindi+Guo%2C+Sixun+Dong%2C+Xiaohua%2C+Xuan%2C+Zhengxin+Li%2C+Lin+Ma%2C+and+Shenghua+Gao%2E+Mllm%2Dtool%3A+A+multimodal+large+language+model+for+tool+agent+learning%2C+202)
103. Xiao Liu, Jianfeng Lin, and Jiawei Zhang. Beyond text: Unveiling multimodal proficiency of large language models with multiapi benchmark, 2023. [find](https://scholar.google.com/scholar?q=Xiao+Liu%2C+Jianfeng+Lin%2C+and+Jiawei+Zhang%2E+Beyond+text%3A+Unveiling+multimodal+proficiency+of+large+language+models+with+multiapi+benchmark%2C+2023%2E)
104. Xinwei Long, Jiali Zeng, Fandong Meng, Zhiyuan Ma, Kaiyan Zhang, Bowen Zhou, and Jie Zhou. Generative multi-modal knowledge retrieval with large language models, 2024. [find](https://scholar.google.com/scholar?q=Xinwei+Long%2C+Jiali+Zeng%2C+Fandong+Meng%2C+Zhiyuan+Ma%2C+Kaiyan+Zhang%2C+Bowen+Zhou%2C+and+Jie+Zhou%2E+Generative+multi%2Dmodal+knowledge+retrieval+with+large+language+models%2C+2024%2E)
105. Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, and Shafiq Joty. Retrieving multimodal information for augmented generation: A survey. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Associati… [find](https://scholar.google.com/scholar?q=Ruochen+Zhao%2C+Hailin+Chen%2C+Weishi+Wang%2C+Fangkai+Jiao%2C+Xuan+Long+Do%2C+Chengwei+Qin%2C+Bosheng+Ding%2C+Xiaobao+Guo%2C+Minzhi+Li%2C+Xingxuan+Li%2C+and+Shafiq+Joty%2E+Retrieving+multimodal+information+for+augmented+ge)
106. Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A. Lee, Yuke Zhu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Multibench: Multiscale benchmarks for multimodal representation learning, 2021. [find](https://scholar.google.com/scholar?q=Paul+Pu+Liang%2C+Yiwei+Lyu%2C+Xiang+Fan%2C+Zetian+Wu%2C+Yun+Cheng%2C+Jason+Wu%2C+Leslie+Chen%2C+Peter+Wu%2C+Michelle+A%2E+Lee%2C+Yuke+Zhu%2C+Ruslan+Salakhutdinov%2C+and+Louis%2DPhilippe+Morency%2E+Multibench%3A+Multiscale+benchmar)
107. Wentao Ge, Shunian Chen, Guiming Hardy Chen, Zhihong Chen, Junying Chen, Shuo Yan, Chenghao Zhu, Ziyue Lin, Wenya Xie, Xinyi Zhang, Yichen Chai, Xiaoyu Liu, Dingjie Song, Xidong Wang, Anningzhe Gao, Zhiyi Zhang, Jianquan Li, Xiang Wan, and Benyou Wang. Mllm-bench: Evaluating multimodal llms with per… [find](https://scholar.google.com/scholar?q=Wentao+Ge%2C+Shunian+Chen%2C+Guiming+Hardy+Chen%2C+Zhihong+Chen%2C+Junying+Chen%2C+Shuo+Yan%2C+Chenghao+Zhu%2C+Ziyue+Lin%2C+Wenya+Xie%2C+Xinyi+Zhang%2C+Yichen+Chai%2C+Xiaoyu+Liu%2C+Dingjie+Song%2C+Xidong+Wang%2C+Anningzhe+Gao%2C+Z)
108. Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, Jiayi Lei, Quanfeng Lu, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang, Peng Gao, Yali Wang, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. Mmt-bench: A comprehensive multimodal ben… [find](https://scholar.google.com/scholar?q=Kaining+Ying%2C+Fanqing+Meng%2C+Jin+Wang%2C+Zhiqian+Li%2C+Han+Lin%2C+Yue+Yang%2C+Hao+Zhang%2C+Wenbo+Zhang%2C+Yuqi+Lin%2C+Shuo+Liu%2C+Jiayi+Lei%2C+Quanfeng+Lu%2C+Runjian+Chen%2C+Peng+Xu%2C+Renrui+Zhang%2C+Haozhe+Zhang%2C+Peng+Gao%2C+Ya)
109. Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers, 2023. Contents Introduction 2 W E -M ATH 2.1 Hierachial Structured Dataset Composition . . . . . . . . . . . . . . . . . . . . .… [find](https://scholar.google.com/scholar?q=Cong+Wei%2C+Yang+Chen%2C+Haonan+Chen%2C+Hexiang+Hu%2C+Ge+Zhang%2C+Jie+Fu%2C+Alan+Ritter%2C+and+Wenhu+Chen%2E+Uniir%3A+Training+and+benchmarking+universal+multimodal+information+retrievers%2C+2023%2E+Contents+Introduction+2)

*Updated 2 November 2026*