---
title: "LLaVA-CoT: Let Vision Language Models Reason Step-by-Step"
description: "Vision-language models (VLMs) often answer complex visual questions without first organizing the task. 68 references, cited by 86 papers here."
authors: ["Guowei Xu", "Peng Jin", "Ziang Wu", "Hao Li", "Yibing Song", "Lichao Sun", "Li Yuan"]
year: 2024
venue: "arXiv:2411.10440"
source: "https://arxiv.org/abs/2411.10440"
doi: "10.48550/arxiv.2411.10440"
arxiv: "2411.10440"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2024-llava-cot-let-vision-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2024-llava-cot-let-vision-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / LLaVA-CoT: Let Vision Language Models Reason Step-by-Step

# LLaVA-CoT: Let Vision Language Models Reason Step-by-Step

**Authors:** Guowei Xu, [Peng Jin](https://md.research.elamur.ai/free/en/authors/peng-jin/), Ziang Wu, [Hao Li](https://md.research.elamur.ai/free/en/authors/hao-li/), Yibing Song, [Lichao Sun](https://md.research.elamur.ai/free/en/authors/lichao-sun/) and [Li Yuan](https://md.research.elamur.ai/free/en/authors/li-yuan/) · **Year:** 2024 · **Venue:** [arXiv:2411.10440](https://md.research.elamur.ai/free/en/venues/ieee-cvf-international-conference-on-computer-vision-iccv/) ([ICCV 2024](https://md.research.elamur.ai/free/en/venues/ieee-cvf-international-conference-on-computer-vision-iccv/2024/)) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2411.10440) · [PDF](https://arxiv.org/pdf/2411.10440) · [DOI 10.48550/arxiv.2411.10440](https://doi.org/10.48550/arxiv.2411.10440) · [arXiv 2411.10440](https://arxiv.org/abs/2411.10440) · [OpenAlex](https://openalex.org/W7160207210)

**Citations:** 52 citations (OpenAlex, 2026-10-14) · [86](https://md.research.elamur.ai/free/en/papers/xu-2024-llava-cot-let-vision-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Vision-language models (VLMs) often answer complex visual questions without first organizing the task. Even with chain-of-thought prompting, they tend to commit to an answer early and then rationalize it, which produces errors and hallucinations. LLaVA-CoT addresses this by training a VLM to reason through a fixed sequence of tagged stages: outlining the task, describing the relevant image content, working through the logic and stating a conclusion. The authors built LLaVA-CoT-100k, a set of about 100k samples drawn from several VQA sources, with structured answers generated by GPT-4o. They used it to fine-tune Llama-3.2-11B-Vision-Instruct. At inference, SWIRES (stage-wise retracing search) samples several candidates per stage and keeps the best one according to a reward model. If every candidate looks weak, it goes back and regenerates the previous stage. With SWIRES, the 11B model gains 9.4 points on average over its base across six reasoning benchmarks. It also beats Gemini-1.5-Pro, GPT-4o-mini and Llama-3.2-90B-Vision-Instruct. This suggests that structured data and smarter search can partly replace scale. Code, data and weights are public.

## Key points
- Each response follows four tagged stages (`<SUMMARY>`, `<CAPTION>`, `<REASONING>`, `<CONCLUSION>`). After fine-tuning, the model moves through them without extra prompting.
- Evaluation covers MMStar, MMBench, MMVet, MathVista, AI2D and HallusionBench. MMStar-R, MMBench-R and MMVet-R are filtered versions that drop pure perception and OCR items.
- Table 5 averages: the base model scores 56.9, LLaVA-CoT 63.1 and LLaVA-CoT with SWIRES 66.3. For comparison, GPT-4o-mini scores 63.8, Gemini-1.5-Pro 63.6 and Llama-3.2-90B 62.3.
- On MMStar, SWIRES raises accuracy from 57.6 to 62.5. InternLM-XComposer2.5-Reward serves as the judge for SWIRES, Best-of-N and stage-wise beam search alike.
- Best-of-N and stage-wise beam search plateau around 10,000 s of test-time compute, while SWIRES keeps improving beyond that point.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations
- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,267 citations
- [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) — Company, Hangzhou, China · 201 works · 17,173 citations
- [Alibaba DAMO Academy](https://md.research.elamur.ai/free/en/companies/alibaba-damo-academy/) — Company, Hangzhou, China · 45 works · 6,874 citations
- [Alibaba Group (Cayman Islands)](https://md.research.elamur.ai/free/en/companies/alibaba-group-cayman-islands/) — Company, George Town, Cayman Islands · 36 works · 3,645 citations
- [Lehigh University](https://md.research.elamur.ai/free/en/universities/lehigh-university/) — University, Bethlehem, United States · 10 works · 1,573 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [⚙️ Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/) · 188 works · [⚙️ Control and Systems Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/control-and-systems-engineering/) · 62 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works

## Cite

Cite as: Guowei Xu, Peng Jin, Ziang Wu, Hao Li, Yibing Song, Lichao Sun and Li Yuan. 2024. LLaVA-CoT: Let Vision Language Models Reason Step-by-Step. arXiv:2411.10440.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{xu2024llava,
  title = {{LLaVA-CoT: Let Vision Language Models Reason Step-by-Step}},
  author = {Guowei Xu, Peng Jin, Ziang Wu, Hao Li, Yibing Song, Lichao Sun and Li Yuan},
  year = {2024},
  doi = {10.48550/arxiv.2411.10440},
  eprint = {2411.10440},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2411.10440}
}
```

- [← previous OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework](https://md.research.elamur.ai/free/en/papers/hu-2024-openrlhf-easy-use-scalable/)
- [next → Long-context LLMs Struggle with Long In-context Learning](https://md.research.elamur.ai/free/en/papers/li-2024-long-context-llms-struggle-long/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

68 references, 37 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Detailed results on openvlm leaderboard. [https://opencompass.openxlab](https://opencompass.openxlab) . space / assets / OpenVLM.json. 6 [↗](https://opencompass.openxlab)
2. Claude 3.5 sonnet, 2024. Available at: [https://www.anthropic.com/news/claude-3-5-sonnet](https://www.anthropic.com/news/claude-3-5-sonnet). 8 [↗](https://www.anthropic.com/news/claude-3-5-sonnet)
3. OpenAI (2024). [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/), 2024. 2, 4, 8, 3
4. Afra Amini, Tim Vieira, and Ryan Cotterell. Variational best-of-n alignment, 2024. 3 [find](https://scholar.google.com/scholar?q=Afra+Amini%2C+Tim+Vieira%2C+and+Ryan+Cotterell%2E+Variational+best%2Dof%2Dn+alignment%2C+2024%2E+3)
5. Saeed Amizadeh, Hamid Palangi, Alex Polozov, Yichen Huang, and Kazuhito Koishida. Neuro-symbolic visual reasoning: Disentangling. In International Conference on Machine Learning, pages 279–290. Pmlr, 2020. 3 [find](https://scholar.google.com/scholar?q=Saeed+Amizadeh%2C+Hamid+Palangi%2C+Alex+Polozov%2C+Yichen+Huang%2C+and+Kazuhito+Koishida%2E+Neuro%2Dsymbolic+visual+reasoning%3A+Disentangling%2E+In+International+Conference+on+Machine+Learning%2C+pages+279%E2%80%93290%2E+Pmlr%2C+)
6. Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Foundational models defining a new era in vision: A survey and outlook. arXiv preprint arXiv:2307.13721, 2023. 1 — [arXiv:2307.13721](https://arxiv.org/abs/2307.13721) [find](https://scholar.google.com/scholar?q=Muhammad+Awais%2C+Muzammal+Naseer%2C+Salman+Khan%2C+Rao+Muhammad+Anwer%2C+Hisham+Cholakkal%2C+Mubarak+Shah%2C+Ming%2DHsuan+Yang%2C+and+Fahad+Shahbaz+Khan%2E+Foundational+models+defining+a+new+era+in+vision%3A+A+survey+an)
7. Jie Cao and Jing Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1511–1520, Gyeongju, Republic of Korea, 2022. International Committee on Computational L… [find](https://scholar.google.com/scholar?q=Jie+Cao+and+Jing+Xiao%2E+An+augmented+benchmark+dataset+for+geometric+question+answering+through+dual+parallel+text+encoding%2E+In+Proceedings+of+the+29th+International+Conference+on+Computational+Linguis)
8. Franz Louis Cesista. Multimodal structured generation: Cvpr’s 2nd mmfm challenge technical report, 2024. 3 [find](https://scholar.google.com/scholar?q=Franz+Louis+Cesista%2E+Multimodal+structured+generation%3A+Cvpr%E2%80%99s+2nd+mmfm+challenge+technical+report%2C+2024%2E+3)
9. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. [Sharegpt4v: Improving large multi-modal models with better captions](https://md.research.elamur.ai/free/en/papers/chen-2023-sharegpt4v-improving-large-multi-modal/). In European Conference on Computer Vision (ECCV), 2024. 4
10. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. [Are we on the right way for evaluating large vision-language models?](https://md.research.elamur.ai/free/en/papers/chen-2024-right-way-evaluating-large-vision/), 2024. 3, 6
11. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. [Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks](https://md.research.elamur.ai/free/en/papers/chen-2023-internvl-scaling-up-vision/). In Proceedings of the IEEE/CVF Conf…
12. Minkyu Choi, Harsh Goel, Mohammad Omama, Y Yang, S Shah, and S Chinchali. Towards neuro-symbolic video understanding. In Proceedings of the European Conference on Computer Vision (ECCV), Milan, Italy, pages 9–13. Springer, 2024. 3 [find](https://scholar.google.com/scholar?q=Minkyu+Choi%2C+Harsh+Goel%2C+Mohammad+Omama%2C+Y+Yang%2C+S+Shah%2C+and+S+Chinchali%2E+Towards+neuro%2Dsymbolic+video+understanding%2E+In+Proceedings+of+the+European+Conference+on+Computer+Vision+%28ECCV%29%2C+Milan%2C+Italy%2C)
13. Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. [Navigate through enigmatic labyrinth a survey of chain of thought reasoning: Advances, frontiers and future](https://md.research.elamur.ai/free/en/papers/chu-2023-navigate-through-enigmatic-labyrinth-survey/). In Proceedings of the 62nd Annual Meeting of the Association for C…
14. Adam Dahlgren Lindström and Savitha Sam Abraham. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. In International Joint Conference on Learning and Reasoning, 16th International Workshop on Neural-Symbolic Learning and Reasoning (NeSy 2022), Windsor, UK, Septembe… [find](https://scholar.google.com/scholar?q=Adam+Dahlgren+Lindstro%CC%88m+and+Savitha+Sam+Abraham%2E+Clevr%2Dmath%3A+A+dataset+for+compositional+language%2C+visual+and+mathematical+reasoning%2E+In+International+Joint+Conference+on+Learning+and+Reasoning%2C+16th)
15. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chengg… — [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/)
16. Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models, 2024. 6 [find](https://scholar.google.com/scholar?q=Haodong+Duan%2C+Junming+Yang%2C+Yuxuan+Qiao%2C+Xinyu+Fang%2C+Lin+Chen%2C+Yuan+Liu%2C+Xiaoyi+Dong%2C+Yuhang+Zang%2C+Pan+Zhang%2C+Jiaqi+Wang%2C+Dahua+Lin%2C+and+Kai+Chen%2E+Vlmevalkit%3A+An+open%2Dsource+toolkit+for+evaluating+lar)
17. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346–361, 2021. 3 [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Daniel+Khashabi%2C+Elad+Segal%2C+Tushar+Khot%2C+Dan+Roth%2C+and+Jonathan+Berant%2E+Did+aristotle+use+a+laptop%3F+a+question+answering+benchmark+with+implicit+reasoning+strategies%2E+Transactions+of+the+As)
18. Team GLM, :, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Jingyu Sun, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minli… — [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://md.research.elamur.ai/free/en/papers/glm-2024-chatglm-family-large-language-models/)
19. Alex Graves. [Sequence transduction with recurrent neural networks](https://md.research.elamur.ai/free/en/papers/graves-2012-sequence-transduction-recurrent-neural-networks/). arXiv preprint arXiv:1211.3711, 2012. 3
20. Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, and et al. [Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models](https://md.research.elamur.ai/free/en/papers/guan-2023-hallusionbench-advanced-diagnostic-suite-entangled/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),…
21. Tanmay Gupta and Aniruddha Kembhavi. [Visual programming: Compositional visual reasoning without training](https://md.research.elamur.ai/free/en/papers/gupta-2022-visual-programming-compositional-visual-reasoning/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14953–14962, 2023. 3
22. Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. [Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes](https://md.research.elamur.ai/free/en/papers/hsieh-2023-distilling-step-step-outperforming-larger/), 2023. 3
23. Hanxu Hu, Simon Yu, Pinzhen Chen, and Edoardo M. Ponti. Fine-tuning large language models with sequential instructions, 2024. 3 [find](https://scholar.google.com/scholar?q=Hanxu+Hu%2C+Simon+Yu%2C+Pinzhen+Chen%2C+and+Edoardo+M%2E+Ponti%2E+Fine%2Dtuning+large+language+models+with+sequential+instructions%2C+2024%2E+3)
24. Yushi Hu, Otilia Stretcu, Chun-Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman. Visual program distillation: Distilling tools and programmatic reasoning into vision-language models, 2024. 3 [find](https://scholar.google.com/scholar?q=Yushi+Hu%2C+Otilia+Stretcu%2C+Chun%2DTa+Lu%2C+Krishnamurthy+Viswanathan%2C+Kenji+Hata%2C+Enming+Luo%2C+Ranjay+Krishna%2C+and+Ariel+Fuxman%2E+Visual+program+distillation%3A+Distilling+tools+and+programmatic+reasoning+into)
25. Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. [Large language models can self-improve](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/). arXiv preprint arXiv:2210.11610, 2022. 3
26. Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13700– 13710, 2024. 1, 3 [find](https://scholar.google.com/scholar?q=Peng+Jin%2C+Ryuichi+Takanobu%2C+Wancai+Zhang%2C+Xiaochun+Cao%2C+and+Li+Yuan%2E+Chat%2Dunivi%3A+Unified+visual+representation+empowers+large+language+models+with+image+and+video+understanding%2E+In+Proceedings+of+the+)
27. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. [Clevr: A diagnostic dataset for compositional language and elementary visual reasoning](https://md.research.elamur.ai/free/en/papers/johnson-2016-clevr-diagnostic-dataset-compositional-language/). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 3, 4
28. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. [A diagram is worth a dozen images](https://md.research.elamur.ai/free/en/papers/kembhavi-2016-diagram-worth-dozen-images/). pages 235–251, 2016. 3, 4, 5, 6
29. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, S… — [Measuring Faithfulness in Chain-of-Thought Reasoning](https://md.research.elamur.ai/free/en/papers/lanham-2023-measuring-faithfulness-chain-thought-reasoning/)
30. Hao Li, Jinfa Huang, Peng Jin, Guoli Song, Qi Wu, and Jie Chen. Weakly-supervised 3d spatial reasoning for text-based visual question answering. IEEE Transactions on Image Processing, 32:3367–3382, 2023. 3 [find](https://scholar.google.com/scholar?q=Hao+Li%2C+Jinfa+Huang%2C+Peng+Jin%2C+Guoli+Song%2C+Qi+Wu%2C+and+Jie+Chen%2E+Weakly%2Dsupervised+3d+spatial+reasoning+for+text%2Dbased+visual+question+answering%2E+IEEE+Transactions+on+Image+Processing%2C+32%3A3367%E2%80%933382%2C+20)
31. Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S. Kankanhalli. People in social context (pisc) dataset, 2017. Data set. 4 [find](https://scholar.google.com/scholar?q=Junnan+Li%2C+Yongkang+Wong%2C+Qi+Zhao%2C+and+Mohan+S%2E+Kankanhalli%2E+People+in+social+context+%28pisc%29+dataset%2C+2017%2E+Data+set%2E+4)
32. Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jianke Zhu, and Lei Zhang. Tokenpacker: Efficient visual projector for multimodal llm. arXiv preprint arXiv:2407.02392, 2024. 3 — [arXiv:2407.02392](https://arxiv.org/abs/2407.02392) [find](https://scholar.google.com/scholar?q=Wentong+Li%2C+Yuqian+Yuan%2C+Jian+Liu%2C+Dongqi+Tang%2C+Song+Wang%2C+Jianke+Zhu%2C+and+Lei+Zhang%2E+Tokenpacker%3A+Efficient+visual+projector+for+multimodal+llm%2E+arXiv+preprint+arXiv%3A2407%2E02392%2C+2024%2E+3+%E2%80%94+%5BarXiv%3A2407)
33. Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. [Vila: On pre-training for visual language models](https://md.research.elamur.ai/free/en/papers/lin-2023-vila-pre-training-visual-language/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26689–26699, 2024. 1, 8
34. Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, and Hao Su. [Deductive verification of chain-of-thought reasoning](https://md.research.elamur.ai/free/en/papers/ling-2023-deductive-verification-chain-thought-reasoning/), 2023. 1
35. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). Advances in neural information processing systems, 36, 2023. 1, 3
36. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. [Mmbench: Is your multi-modal model an all-around player?](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/) arXiv:2307.06281, 2023. 3, 6
37. Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022. 4, 5 [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Swaroop+Mishra%2C+Tony+Xia%2C+Liang+Qiu%2C+Kai%2DWei+Chang%2C+Song%2DChun+Zhu%2C+Oyvind+Tafjord%2C+Peter+Clark%2C+and+Ashwin+Kalyan%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+science+questio)
38. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. [Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). In International Conference on Learning Representations (ICLR), 2024. 3, 6
39. Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embedding alignment for multimodal large language model. arXiv:2405.20797, 2024. 8 — [arXiv:2405.20797](https://arxiv.org/abs/2405.20797) [find](https://scholar.google.com/scholar?q=Shiyin+Lu%2C+Yang+Li%2C+Qing%2DGuo+Chen%2C+Zhao+Xu%2C+Weihua+Luo%2C+Kaifu+Zhang%2C+and+Han%2DJia+Ye%2E+Ovis%3A+Structural+embedding+alignment+for+multimodal+large+language+model%2E+arXiv%3A2405%2E20797%2C+2024%2E+8+%E2%80%94+%5BarXiv%3A2405%2E2)
40. Mikołaj Małkiński and Jacek Mańdziuk. A review of emerging research directions in abstract visual reasoning. Information Fusion, 91:713–736, 2023. 3 [find](https://scholar.google.com/scholar?q=Miko%C5%82aj+Ma%C5%82kin%CC%81ski+and+Jacek+Man%CC%81dziuk%2E+A+review+of+emerging+research+directions+in+abstract+visual+reasoning%2E+Information+Fusion%2C+91%3A713%E2%80%93736%2C+2023%2E+3)
41. Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. [ChartQA: A benchmark for question answering about charts with visual and logical reasoning](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/). In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, Dublin, Ireland, 2022. Association for Computational…
42. Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. [Docvqa: A dataset for vqa on document images](https://md.research.elamur.ai/free/en/papers/mathew-2020-docvqa-dataset-vqa-document-images/). In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 2199–2208, 2021. 4
43. Meta AI. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. [https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/), 2024. 1, 5, 8, 3 [↗](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/)
44. OpenAI. Gpt-4o mini: advancing cost-efficient intelligence. [https://openai.com/index/gpt](https://openai.com/index/gpt) - 4o - mini - advancing - cost - efficient - intelligence/, 2024. 8 [↗](https://openai.com/index/gpt)
45. Yuxuan Qiao, Haodong Duan, Xinyu Fang, Junming Yang, Lin Chen, Songyang Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. Prism: A framework for decoupling and assessing the capabilities of vlms, 2024. 3, 8 [find](https://scholar.google.com/scholar?q=Yuxuan+Qiao%2C+Haodong+Duan%2C+Xinyu+Fang%2C+Junming+Yang%2C+Lin+Chen%2C+Songyang+Zhang%2C+Jiaqi+Wang%2C+Dahua+Lin%2C+and+Kai+Chen%2E+Prism%3A+A+framework+for+decoupling+and+assessing+the+capabilities+of+vlms%2C+2024%2E+3%2C+8)
46. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, and et al. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/), 2024. 8
47. Maarten Sap, Vered Shwartz, Antoine Bosselut, Yejin Choi, and Dan Roth. Commonsense reasoning for natural language processing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 27–33, 2020. 3 [find](https://scholar.google.com/scholar?q=Maarten+Sap%2C+Vered+Shwartz%2C+Antoine+Bosselut%2C+Yejin+Choi%2C+and+Dan+Roth%2E+Commonsense+reasoning+for+natural+language+processing%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computat)
48. Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. [A-okvqa: A benchmark for visual question answering using world knowledge](https://md.research.elamur.ai/free/en/papers/schwenk-2022-okvqa-benchmark-visual-question-answering/), 2022. 4
49. Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. [Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning](https://md.research.elamur.ai/free/en/papers/shao-2024-visual-cot-advancing-multi-modal/), 2024. 3, 8
50. Ming Shen. Rethinking data selection for supervised fine-tuning. arXiv preprint arXiv:2402.06094, 2024. 3 — [arXiv:2402.06094](https://arxiv.org/abs/2402.06094) [find](https://scholar.google.com/scholar?q=Ming+Shen%2E+Rethinking+data+selection+for+supervised+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A2402%2E06094%2C+2024%2E+3+%E2%80%94+%5BarXiv%3A2402%2E06094%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E06094%29)
51. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. [Scaling llm test-time compute optimally can be more effective than scaling model parameters](https://md.research.elamur.ai/free/en/papers/snell-2024-scaling-llm-test-time-compute/), 2024. 1, 3
52. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. [Sequence to sequence learning with neural networks](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/). In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2014. 3
53. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting, 2023. 1 — [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://md.research.elamur.ai/free/en/papers/turpin-2023-language-models-don-t-always/)
54. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s perception of the world a… — [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://md.research.elamur.ai/free/en/papers/wang-2024-qwen2-vl-enhancing-vision/)
55. Xiaofei Wang, Jinhua Li, and Yifan Zhang. Improved value alignment in large language models using variational best-of-n techniques, 2024. 3 [find](https://scholar.google.com/scholar?q=Xiaofei+Wang%2C+Jinhua+Li%2C+and+Yifan+Zhang%2E+Improved+value+alignment+in+large+language+models+using+variational+best%2Dof%2Dn+techniques%2C+2024%2E+3)
56. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824–24837, 2022. 1, 3
57. Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wan… — [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://md.research.elamur.ai/free/en/papers/wu-2024-deepseek-vl2-mixture-experts/)
58. Siheng Xiong, Yuan Yang, Ali Payani, James C Kerce, and Faramarz Fekri. Teilp: Time prediction over knowledge graphs via logical reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 16112–16119, 2024. 3 [find](https://scholar.google.com/scholar?q=Siheng+Xiong%2C+Yuan+Yang%2C+Ali+Payani%2C+James+C+Kerce%2C+and+Faramarz+Fekri%2E+Teilp%3A+Time+prediction+over+knowledge+graphs+via+logical+reasoning%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intellig)
59. Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. [The dawn of lmms: Preliminary explorations with gpt-4v(ision)](https://md.research.elamur.ai/free/en/papers/yang-2023-dawn-lmms-preliminary-explorations-gpt/), 2023. 4
60. Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. [Minicpm-v: A gpt-4v level mllm on your phone](https://md.research.elamur.ai/free/en/papers/yao-2024-minicpm-v-gpt-4v/). arXiv preprint arXiv:2408.01800, 2024. 8
61. Wangbo Yu, Chaoran Feng, Jiye Tang, Xu Jia, Li Yuan, and Yonghong Tian. Evagaussians: Event stream assisted gaussian splatting from blurry images. arXiv preprint arXiv:2405.20224, 2024. 3 — [arXiv:2405.20224](https://arxiv.org/abs/2405.20224) [find](https://scholar.google.com/scholar?q=Wangbo+Yu%2C+Chaoran+Feng%2C+Jiye+Tang%2C+Xu+Jia%2C+Li+Yuan%2C+and+Yonghong+Tian%2E+Evagaussians%3A+Event+stream+assisted+gaussian+splatting+from+blurry+images%2E+arXiv+preprint+arXiv%3A2405%2E20224%2C+2024%2E+3+%E2%80%94+%5BarXiv%3A240)
62. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. [Mm-vet: Evaluating large multimodal models for integrated capabilities](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/). In International conference on machine learning. PMLR, 2024. 3, 6
63. JD Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang. Why johnny can’t prompt: how non-ai experts try (and fail) to design llm prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–21, 2023. 3 [find](https://scholar.google.com/scholar?q=JD+Zamfirescu%2DPereira%2C+Richmond+Y+Wong%2C+Bjoern+Hartmann%2C+and+Qian+Yang%2E+Why+johnny+can%E2%80%99t+prompt%3A+how+non%2Dai+experts+try+%28and+fail%29+to+design+llm+prompts%2E+In+Proceedings+of+the+2023+CHI+Conference+on+H)
64. Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding, Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, et al. Internlm-xcomposer2. 5-reward: A simple yet effective multi-modal reward model. arXiv preprint arXiv:2501.12368, 2025. 7, 2 [find](https://scholar.google.com/scholar?q=Yuhang+Zang%2C+Xiaoyi+Dong%2C+Pan+Zhang%2C+Yuhang+Cao%2C+Ziyu+Liu%2C+Shengyuan+Ding%2C+Shenxi+Wu%2C+Yubo+Ma%2C+Haodong+Duan%2C+Wenwei+Zhang%2C+et+al%2E+Internlm%2Dxcomposer2%2E+5%2Dreward%3A+A+simple+yet+effective+multi%2Dmodal+rewa)
65. Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning, 2019. 1 [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+From+recognition+to+cognition%3A+Visual+commonsense+reasoning%2C+2019%2E+1)
66. Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang, Ruoming Pang, and Yiming Yang. [Improve vision language model chain-of-thought reasoning](https://md.research.elamur.ai/free/en/papers/zhang-2025-improve-vision-language-model-chain/), 2024. 1
67. Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. [Marco-o1: Towards open reasoning models for open-ended solutions](https://md.research.elamur.ai/free/en/papers/zhao-2024-marco-o1-towards-open-reasoning/), 2024. 3
68. Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, and et al. [Evaluation of openai o1: Opportunities and challenges of agi](https://md.research.elamur.ai/free/en/papers/zhong-2024-evaluation-openai-o1-opportunities-challenges/), 2024. 1 LLaVA-CoT: Let Vision Language Models Reason Step-by-Step

*Updated 2 November 2026*