---
title: "Cambrian-1: A Fully Open, Vision-Centric — Tong 2024"
description: "Multimodal LLMs (MLLMs) typically improve by scaling the language backbone, while choices about the vision side get little systematic study and stay…"
authors: ["Shengbang Tong", "Ellis Brown", "Penghao Wu", "Sanghyun Woo", "Manoj Middepogu", "Sai Charitha Akula", "Jihan Yang", "Shusheng Yang", "Adithya Iyer", "Xichen Pan", "Ziteng Wang", "Rob Fergus", "Yann LeCun", "Saining Xie"]
year: 2024
venue: "arXiv:2406.16860"
source: "https://arxiv.org/abs/2406.16860"
doi: "10.48550/arxiv.2406.16860"
arxiv: "2406.16860"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tong-2024-cambrian-1-fully-open-vision/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tong-2024-cambrian-1-fully-open-vision/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/) / Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

# Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

**Authors:** [Shengbang Tong](https://md.research.elamur.ai/free/en/authors/shengbang-tong/), Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, [Jihan Yang](https://md.research.elamur.ai/free/en/authors/jihan-yang/), [Shusheng Yang](https://md.research.elamur.ai/free/en/authors/shusheng-yang/), Adithya Iyer, Xichen Pan, Ziteng Wang, [Rob Fergus](https://md.research.elamur.ai/free/en/authors/rob-fergus/) et al. (14 authors) · **Year:** 2024 · **Venue:** arXiv:2406.16860 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2406.16860) · [PDF](https://arxiv.org/pdf/2406.16860) · [DOI 10.48550/arxiv.2406.16860](https://doi.org/10.48550/arxiv.2406.16860) · [arXiv 2406.16860](https://arxiv.org/abs/2406.16860) · [OpenAlex](https://openalex.org/W4415796658)

**Citations:** 63 citations (openalex-snapshot · 2026-10-14) · [110](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Multimodal LLMs (MLLMs) typically improve by scaling the language backbone, while choices about the vision side get little systematic study and stay disconnected from visual representation learning research. The authors argue this weakens how well models ground answers in what they actually see. Cambrian-1, from New York University, is a family of MLLMs built around the vision component. The team uses an LLM with visual instruction tuning as a testbed to compare more than 20 vision encoders, including self-supervised, strongly supervised and hybrid ones. They also audit existing MLLM benchmarks, introduce a vision-focused benchmark called CV-Bench, and design a new connector, the Spatial Vision Aggregator (SVA). In addition, they assemble instruction-tuning data from public sources, giving close attention to how sources are balanced and mixed. The resulting models reach state-of-the-art results. Everything is released openly, including weights, code, data and evaluation recipes, so the work also serves as a practical guide for building instruction-tuned MLLMs.

## Key points
- More than 20 vision encoders, spanning self-supervised, strongly supervised and combined approaches, are compared by plugging each into an instruction-tuned LLM.
- CV-Bench is a new benchmark targeting vision-centric abilities. It was motivated by an analysis showing that existing MLLM benchmarks are hard to consolidate and interpret.
- The Spatial Vision Aggregator (SVA) is a dynamic, spatially aware connector. It feeds high-resolution visual features into the LLM while cutting the number of visual tokens.
- The instruction-tuning dataset Cambrian-10M was curated from public sources, with emphasis on source balancing and mixture ratios. The paper also uses system prompts to counter the "answer machine phenomenon".
- The release is fully open: models (huggingface.co/nyu-visionx), data, CV-Bench, code and evaluation tools (github.com/cambrian-mllm/cambrian).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

110 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 100 not yet.

- [Qwen2.5-VL Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen2-5-vl-technical/)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://md.research.elamur.ai/free/en/papers/chen-2024-expanding-performance-boundaries-open/)
- [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training](https://md.research.elamur.ai/free/en/papers/chu-2025-sft-memorizes-rl-generalizes-comparative/)
- [Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models](https://md.research.elamur.ai/free/en/papers/deitke-2024-molmo-pixmo-open-weights-open/)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://md.research.elamur.ai/free/en/papers/li-2024-llava-onevision-easy-visual-task/)
- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://md.research.elamur.ai/free/en/papers/liu-2023-ocrbench-hidden-mystery-ocr-large/)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://md.research.elamur.ai/free/en/papers/wang-2025-internvl3-5-advancing-open/)
- [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://md.research.elamur.ai/free/en/papers/wu-2024-deepseek-vl2-mixture-experts/)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://md.research.elamur.ai/free/en/papers/yao-2024-minicpm-v-gpt-4v/)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://md.research.elamur.ai/free/en/papers/zhu-2025-internvl3-exploring-advanced-training/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{tong2024cambrian,
  title = {{Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs}},
  author = {Tong, Shengbang and Brown, Ellis and Wu, Penghao and Woo, Sanghyun and Middepogu, Manoj and Akula, Sai Charitha and Yang, Jihan and Yang, Shusheng and Iyer, Adithya and Pan, Xichen and Wang, Ziteng and Fergus, Rob and LeCun, Yann and Xie, Saining},
  year = {2024},
  doi = {10.48550/arxiv.2406.16860},
  eprint = {2406.16860},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.16860}
}
```

- [← previous Bridging the Preference Gap between Retrievers and LLMs](https://md.research.elamur.ai/free/en/papers/ke-2024-bridging-preference-gap-between-retrievers/)
- [next → Camel Morph MSA: A Large-Scale Open-Source Morphological Analyzer for Modern Standard Arabic](https://md.research.elamur.ai/free/en/papers/coling-2024-khairallah-camel-morph-msa-large-scale/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/)

## References

156 references, 50 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. Acharya, K. Kafle, and C. Kanan. “TallyQA: Answering complex counting questions”. In: AAAI. 2019. [find](https://scholar.google.com/scholar?q=M%2E+Acharya%2C+K%2E+Kafle%2C+and+C%2E+Kanan%2E+%E2%80%9CTallyQA%3A+Answering+complex+counting+questions%E2%80%9D%2E+In%3A+AAAI%2E+2019%2E)
2. A. Agrawal et al. “Don’t just assume; look and answer: Overcoming priors for visual question answering”. In: CVPR. 2018. [find](https://scholar.google.com/scholar?q=A%2E+Agrawal+et+al%2E+%E2%80%9CDon%E2%80%99t+just+assume%3B+look+and+answer%3A+Overcoming+priors+for+visual+question+answering%E2%80%9D%2E+In%3A+CVPR%2E+2018%2E)
3. A. Ahmadyan et al. “Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotations”. In: CVPR (2021). [find](https://scholar.google.com/scholar?q=A%2E+Ahmadyan+et+al%2E+%E2%80%9CObjectron%3A+A+Large+Scale+Dataset+of+Object%2DCentric+Videos+in+the+Wild+with+Pose+Annotations%E2%80%9D%2E+In%3A+CVPR+%282021%29%2E)
4. AI@Meta. “Llama 3 Model Card”. In: (2024). [find](https://scholar.google.com/scholar?q=AI%40Meta%2E+%E2%80%9CLlama+3+Model+Card%E2%80%9D%2E+In%3A+%282024%29%2E)
5. H. A. Alawwad et al. “Enhancing Textbook Question Answering Task with Large Language Models and Retrieval Augmented Generation”. In: arXiv preprint arXiv:2402.05128 (2024). — [arXiv:2402.05128](https://arxiv.org/abs/2402.05128) [find](https://scholar.google.com/scholar?q=H%2E+A%2E+Alawwad+et+al%2E+%E2%80%9CEnhancing+Textbook+Question+Answering+Task+with+Large+Language+Models+and+Retrieval+Augmented+Generation%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2402%2E05128+%282024%29%2E+%E2%80%94+%5BarXiv%3A2402%2E05128%5D%28https%3A)
6. J.-B. Alayrac et al. “Flamingo: a visual language model for few-shot learning”. In: NeurIPS. 2022. [find](https://scholar.google.com/scholar?q=J%2E%2DB%2E+Alayrac+et+al%2E+%E2%80%9CFlamingo%3A+a+visual+language+model+for+few%2Dshot+learning%E2%80%9D%2E+In%3A+NeurIPS%2E+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
7. T. Aquinas. Quaestiones Disputatae de Veritate. q.2 a.3 arg.19, 1259. [find](https://scholar.google.com/scholar?q=T%2E+Aquinas%2E+Quaestiones+Disputatae+de+Veritate%2E+q%2E2+a%2E3+arg%2E19%2C+1259%2E)
8. Aristotle. Metaphysics. Ed. by T. by W. D. Ross. The Internet Classics Archive, 350BCE. [find](https://scholar.google.com/scholar?q=Aristotle%2E+Metaphysics%2E+Ed%2E+by+T%2E+by+W%2E+D%2E+Ross%2E+The+Internet+Classics+Archive%2C+350BCE%2E)
9. M. Assran et al. “Self-supervised learning from images with a joint-embedding predictive architecture”. In: CVPR. 2023. [find](https://scholar.google.com/scholar?q=M%2E+Assran+et+al%2E+%E2%80%9CSelf%2Dsupervised+learning+from+images+with+a+joint%2Dembedding+predictive+architecture%E2%80%9D%2E+In%3A+CVPR%2E+2023%2E)
10. J. Bai et al. “Qwen Technical Report”. In: arXiv preprint arXiv:2309.16609 (2023). [find](https://scholar.google.com/scholar?q=J%2E+Bai+et+al%2E+%E2%80%9CQwen+Technical+Report%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2309%2E16609+%282023%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
11. J. Bai et al. “Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond”. In: (2023). [find](https://scholar.google.com/scholar?q=J%2E+Bai+et+al%2E+%E2%80%9CQwen%2Dvl%3A+A+versatile+vision%2Dlanguage+model+for+understanding%2C+localization%2C+text+reading%2C+and+beyond%E2%80%9D%2E+In%3A+%282023%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/)
12. M. E. Banani et al. “Probing the 3D Awareness of Visual Foundation Models”. In: arXiv preprint arXiv:2404.08636 (2024). — [arXiv:2404.08636](https://arxiv.org/abs/2404.08636) [find](https://scholar.google.com/scholar?q=M%2E+E%2E+Banani+et+al%2E+%E2%80%9CProbing+the+3D+Awareness+of+Visual+Foundation+Models%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2404%2E08636+%282024%29%2E+%E2%80%94+%5BarXiv%3A2404%2E08636%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2404%2E08636%29)
13. G. Baruch et al. “ARKitScenes - A Diverse Real-World Dataset for 3D Indoor Scene Understanding Using Mobile RGB-D Data”. In: NeurIPS Datasets and Benchmarks Track (Round 1). 2021. [find](https://scholar.google.com/scholar?q=G%2E+Baruch+et+al%2E+%E2%80%9CARKitScenes+%2D+A+Diverse+Real%2DWorld+Dataset+for+3D+Indoor+Scene+Understanding+Using+Mobile+RGB%2DD+Data%E2%80%9D%2E+In%3A+NeurIPS+Datasets+and+Benchmarks+Track+%28Round+1%29%2E+2021%2E)
14. J. Belouadi, A. Lauscher, and S. Eger. “Automatikz: Text-guided synthesis of scientific vector graphics with tikz”. In: ICLR. 2024. [find](https://scholar.google.com/scholar?q=J%2E+Belouadi%2C+A%2E+Lauscher%2C+and+S%2E+Eger%2E+%E2%80%9CAutomatikz%3A+Text%2Dguided+synthesis+of+scientific+vector+graphics+with+tikz%E2%80%9D%2E+In%3A+ICLR%2E+2024%2E)
15. R. Birkl, D. Wofk, and M. Müller. “Midas v3. 1–a model zoo for robust monocular relative depth estimation”. In: arXiv preprint arXiv:2307.14460 (2023). — [arXiv:2307.14460](https://arxiv.org/abs/2307.14460) [find](https://scholar.google.com/scholar?q=R%2E+Birkl%2C+D%2E+Wofk%2C+and+M%2E+M%C3%BCller%2E+%E2%80%9CMidas+v3%2E+1%E2%80%93a+model+zoo+for+robust+monocular+relative+depth+estimation%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2307%2E14460+%282023%29%2E+%E2%80%94+%5BarXiv%3A2307%2E14460%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2307%2E)
16. A. F. Biten et al. “Latr: Layout-aware transformer for scene-text vqa”. In: CVPR. 2022. [find](https://scholar.google.com/scholar?q=A%2E+F%2E+Biten+et+al%2E+%E2%80%9CLatr%3A+Layout%2Daware+transformer+for+scene%2Dtext+vqa%E2%80%9D%2E+In%3A+CVPR%2E+2022%2E)
17. A. F. Biten et al. “Scene text visual question answering”. In: ICCV. 2019. [find](https://scholar.google.com/scholar?q=A%2E+F%2E+Biten+et+al%2E+%E2%80%9CScene+text+visual+question+answering%E2%80%9D%2E+In%3A+ICCV%2E+2019%2E)
18. G. Brazil et al. “Omni3d: A large benchmark and model for 3d object detection in the wild”. In: CVPR. 2023. [find](https://scholar.google.com/scholar?q=G%2E+Brazil+et+al%2E+%E2%80%9COmni3d%3A+A+large+benchmark+and+model+for+3d+object+detection+in+the+wild%E2%80%9D%2E+In%3A+CVPR%2E+2023%2E)
19. J. Buchner. imagehash (fork). [https://github.com/JohannesBuchner/imagehash](https://github.com/JohannesBuchner/imagehash) . 2021. [↗](https://github.com/JohannesBuchner/imagehash)
20. H. Caesar et al. “nuscenes: A multimodal dataset for autonomous driving”. In: CVPR. 2020. [find](https://scholar.google.com/scholar?q=H%2E+Caesar+et+al%2E+%E2%80%9Cnuscenes%3A+A+multimodal+dataset+for+autonomous+driving%E2%80%9D%2E+In%3A+CVPR%2E+2020%2E)
21. J. Cha et al. “Honeybee: Locality-enhanced projector for multimodal llm”. In: CVPR. 2024. [find](https://scholar.google.com/scholar?q=J%2E+Cha+et+al%2E+%E2%80%9CHoneybee%3A+Locality%2Denhanced+projector+for+multimodal+llm%E2%80%9D%2E+In%3A+CVPR%2E+2024%2E)
22. S. Cha et al. “Visually Dehallucinative Instruction Generation: Know What You Don’t Know”. In: arXiv preprint arXiv:2402.09717 (2024). — [arXiv:2402.09717](https://arxiv.org/abs/2402.09717) [find](https://scholar.google.com/scholar?q=S%2E+Cha+et+al%2E+%E2%80%9CVisually+Dehallucinative+Instruction+Generation%3A+Know+What+You+Don%E2%80%99t+Know%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2402%2E09717+%282024%29%2E+%E2%80%94+%5BarXiv%3A2402%2E09717%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E09717%29)
23. D. J. Chalmers. “Does Thought Require Sensory Grounding? From Pure Thinkers to Large Language Models”. In: Proceedings and Addresses of the American Philosophical Association 97 (2023), pp. 22–45. [find](https://scholar.google.com/scholar?q=D%2E+J%2E+Chalmers%2E+%E2%80%9CDoes+Thought+Require+Sensory+Grounding%3F+From+Pure+Thinkers+to+Large+Language+Models%E2%80%9D%2E+In%3A+Proceedings+and+Addresses+of+the+American+Philosophical+Association+97+%282023%29%2C+pp%2E+22%E2%80%9345%2E)
24. Y. Chang et al. “A survey on evaluation of large language models”. In: ACM Transactions on Intelligent Systems and Technology 15.3 (2024), pp. 1–45. [find](https://scholar.google.com/scholar?q=Y%2E+Chang+et+al%2E+%E2%80%9CA+survey+on+evaluation+of+large+language+models%E2%80%9D%2E+In%3A+ACM+Transactions+on+Intelligent+Systems+and+Technology+15%2E3+%282024%29%2C+pp%2E+1%E2%80%9345%2E) [our card](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/)
25. G. H. Chen et al. “ALLaVA: Harnessing GPT4V-synthesized Data for A Lite Vision-Language Model”. In: arXiv preprint arXiv:2402.11684 (2024). [find](https://scholar.google.com/scholar?q=G%2E+H%2E+Chen+et+al%2E+%E2%80%9CALLaVA%3A+Harnessing+GPT4V%2Dsynthesized+Data+for+A+Lite+Vision%2DLanguage+Model%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2402%2E11684+%282024%29%2E)
26. L. Chen et al. “Are We on the Right Way for Evaluating Large Vision-Language Models?” In: arXiv preprint arXiv:2403.20330 (2024). [find](https://scholar.google.com/scholar?q=L%2E+Chen+et+al%2E+%E2%80%9CAre+We+on+the+Right+Way+for+Evaluating+Large+Vision%2DLanguage+Models%3F%E2%80%9D+In%3A+arXiv+preprint+arXiv%3A2403%2E20330+%282024%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2024-right-way-evaluating-large-vision/)
27. L. Chen et al. “Sharegpt4v: Improving large multi-modal models with better captions”. In: arXiv preprint arXiv:2311.12793 (2023). [find](https://scholar.google.com/scholar?q=L%2E+Chen+et+al%2E+%E2%80%9CSharegpt4v%3A+Improving+large+multi%2Dmodal+models+with+better+captions%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2311%2E12793+%282023%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2023-sharegpt4v-improving-large-multi-modal/)
28. X. Chen et al. “Pali: A jointly-scaled multilingual language-image model”. In: ICLR. 2023. [find](https://scholar.google.com/scholar?q=X%2E+Chen+et+al%2E+%E2%80%9CPali%3A+A+jointly%2Dscaled+multilingual+language%2Dimage+model%E2%80%9D%2E+In%3A+ICLR%2E+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2022-pali-jointly-scaled-multilingual-language/)
29. X. Chen, S. Xie, and K. He. “An empirical study of training self-supervised vision transformers”. In: ICCV. 2021. [find](https://scholar.google.com/scholar?q=X%2E+Chen%2C+S%2E+Xie%2C+and+K%2E+He%2E+%E2%80%9CAn+empirical+study+of+training+self%2Dsupervised+vision+transformers%E2%80%9D%2E+In%3A+ICCV%2E+2021%2E)
30. Z. Chen et al. “How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites”. In: arXiv preprint arXiv:2404.16821 (2024). [find](https://scholar.google.com/scholar?q=Z%2E+Chen+et+al%2E+%E2%80%9CHow+far+are+we+to+gpt%2D4v%3F+closing+the+gap+to+commercial+multimodal+models+with+open%2Dsource+suites%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2404%2E16821+%282024%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2024-far-gpt-4v-closing/)
31. Z. Chen et al. “Finqa: A dataset of numerical reasoning over financial data”. In: EMNLP. 2021. [find](https://scholar.google.com/scholar?q=Z%2E+Chen+et+al%2E+%E2%80%9CFinqa%3A+A+dataset+of+numerical+reasoning+over+financial+data%E2%80%9D%2E+In%3A+EMNLP%2E+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-finqa-dataset-numerical-reasoning-over/)
32. Z. Cheng et al. “HiTab: A hierarchical table dataset for question answering and natural language generation”. In: ACL. 2022. [find](https://scholar.google.com/scholar?q=Z%2E+Cheng+et+al%2E+%E2%80%9CHiTab%3A+A+hierarchical+table+dataset+for+question+answering+and+natural+language+generation%E2%80%9D%2E+In%3A+ACL%2E+2022%2E)
33. M. Cherti et al. “Reproducible scaling laws for contrastive language-image learning”. In: CVPR. 2023. [find](https://scholar.google.com/scholar?q=M%2E+Cherti+et+al%2E+%E2%80%9CReproducible+scaling+laws+for+contrastive+language%2Dimage+learning%E2%80%9D%2E+In%3A+CVPR%2E+2023%2E)
34. W.-L. Chiang et al. “Chatbot arena: An open platform for evaluating llms by human preference”. In: arXiv preprint arXiv:2403.04132 (2024). [find](https://scholar.google.com/scholar?q=W%2E%2DL%2E+Chiang+et+al%2E+%E2%80%9CChatbot+arena%3A+An+open+platform+for+evaluating+llms+by+human+preference%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2403%2E04132+%282024%29%2E)
35. X. Chu et al. “Mobilevlm v2: Faster and stronger baseline for vision language model”. In: arXiv preprint arXiv:2402.03766 (2024). — [arXiv:2402.03766](https://arxiv.org/abs/2402.03766) [find](https://scholar.google.com/scholar?q=X%2E+Chu+et+al%2E+%E2%80%9CMobilevlm+v2%3A+Faster+and+stronger+baseline+for+vision+language+model%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2402%2E03766+%282024%29%2E+%E2%80%94+%5BarXiv%3A2402%2E03766%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E03766%29)
36. M. Conover et al. Free Dolly: Introducing the World’s First Truly Open Instruction-Tuned LLM. 2023. U R L : [https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm) (visited on 06/30/2023). [↗](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm)
37. W. Dai et al. “Instructblip: Towards general-purpose vision-language models with instruction tuning”. In: NeurIPS. 2024. [find](https://scholar.google.com/scholar?q=W%2E+Dai+et+al%2E+%E2%80%9CInstructblip%3A+Towards+general%2Dpurpose+vision%2Dlanguage+models+with+instruction+tuning%E2%80%9D%2E+In%3A+NeurIPS%2E+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/)
38. H. Dong et al. “Rlhf workflow: From reward modeling to online rlhf”. In: arXiv preprint arXiv:2405.07863 (2024). [find](https://scholar.google.com/scholar?q=H%2E+Dong+et+al%2E+%E2%80%9CRlhf+workflow%3A+From+reward+modeling+to+online+rlhf%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2405%2E07863+%282024%29%2E)
39. A. Dosovitskiy et al. “An image is worth 16x16 words: Transformers for image recognition at scale”. In: ICLR. 2021. [find](https://scholar.google.com/scholar?q=A%2E+Dosovitskiy+et+al%2E+%E2%80%9CAn+image+is+worth+16x16+words%3A+Transformers+for+image+recognition+at+scale%E2%80%9D%2E+In%3A+ICLR%2E+2021%2E)
40. A. Fang et al. “Data filtering networks”. In: ICLR. 2024. [find](https://scholar.google.com/scholar?q=A%2E+Fang+et+al%2E+%E2%80%9CData+filtering+networks%E2%80%9D%2E+In%3A+ICLR%2E+2024%2E)
41. X. Fu et al. “BLINK: Multimodal Large Language Models Can See but Not Perceive”. In: arXiv preprint arXiv:2404.12390 (2024). [find](https://scholar.google.com/scholar?q=X%2E+Fu+et+al%2E+%E2%80%9CBLINK%3A+Multimodal+Large+Language+Models+Can+See+but+Not+Perceive%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2404%2E12390+%282024%29%2E)
42. S. Y. Gadre et al. “Datacomp: In search of the next generation of multimodal datasets”. In: vol. 36. 2024. [find](https://scholar.google.com/scholar?q=S%2E+Y%2E+Gadre+et+al%2E+%E2%80%9CDatacomp%3A+In+search+of+the+next+generation+of+multimodal+datasets%E2%80%9D%2E+In%3A+vol%2E+36%2E+2024%2E)
43. J. Gao et al. “G-llava: Solving geometric problem with multi-modal large language model”. In: arXiv preprint arXiv:2312.11370 (2023). [find](https://scholar.google.com/scholar?q=J%2E+Gao+et+al%2E+%E2%80%9CG%2Dllava%3A+Solving+geometric+problem+with+multi%2Dmodal+large+language+model%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2312%2E11370+%282023%29%2E)
44. P. Gao et al. “Llama-adapter v2: Parameter-efficient visual instruction model”. In: arXiv preprint arXiv:2304.15010 (2023). [find](https://scholar.google.com/scholar?q=P%2E+Gao+et+al%2E+%E2%80%9CLlama%2Dadapter+v2%3A+Parameter%2Defficient+visual+instruction+model%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2304%2E15010+%282023%29%2E)
45. P. Gao et al. “SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models”. In: arXiv preprint arXiv:2402.05935 (2024). — [arXiv:2402.05935](https://arxiv.org/abs/2402.05935) [find](https://scholar.google.com/scholar?q=P%2E+Gao+et+al%2E+%E2%80%9CSPHINX%2DX%3A+Scaling+Data+and+Parameters+for+a+Family+of+Multi%2Dmodal+Large+Language+Models%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2402%2E05935+%282024%29%2E+%E2%80%94+%5BarXiv%3A2402%2E05935%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E059)
46. Y. Ge et al. “Planting a seed of vision in large language model”. In: arXiv preprint arXiv:2307.08041 (2023). [find](https://scholar.google.com/scholar?q=Y%2E+Ge+et+al%2E+%E2%80%9CPlanting+a+seed+of+vision+in+large+language+model%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2307%2E08041+%282023%29%2E)
47. A. Geiger, P. Lenz, and R. Urtasun. “Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite”. In: CVPR. 2012. [find](https://scholar.google.com/scholar?q=A%2E+Geiger%2C+P%2E+Lenz%2C+and+R%2E+Urtasun%2E+%E2%80%9CAre+we+ready+for+Autonomous+Driving%3F+The+KITTI+Vision+Benchmark+Suite%E2%80%9D%2E+In%3A+CVPR%2E+2012%2E)
48. R. Geirhos et al. “Shortcut learning in deep neural networks”. In: Nature Machine Intelligence (2020). [find](https://scholar.google.com/scholar?q=R%2E+Geirhos+et+al%2E+%E2%80%9CShortcut+learning+in+deep+neural+networks%E2%80%9D%2E+In%3A+Nature+Machine+Intelligence+%282020%29%2E)
49. R. Girshick et al. “Rich feature hierarchies for accurate object detection and semantic segmentation”. In: CVPR. 2014. [find](https://scholar.google.com/scholar?q=R%2E+Girshick+et+al%2E+%E2%80%9CRich+feature+hierarchies+for+accurate+object+detection+and+semantic+segmentation%E2%80%9D%2E+In%3A+CVPR%2E+2014%2E) [our card](https://md.research.elamur.ai/free/en/papers/girshick-2013-rich-feature-hierarchies-accurate-object/)
50. Y. Goyal et al. “Making the v in vqa matter: Elevating the role of image understanding in visual question answering”. In: CVPR. 2017. [find](https://scholar.google.com/scholar?q=Y%2E+Goyal+et+al%2E+%E2%80%9CMaking+the+v+in+vqa+matter%3A+Elevating+the+role+of+image+understanding+in+visual+question+answering%E2%80%9D%2E+In%3A+CVPR%2E+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/)
51. D. Gurari et al. “Vizwiz grand challenge: Answering visual questions from blind people”. In: CVPR. 2018. [find](https://scholar.google.com/scholar?q=D%2E+Gurari+et+al%2E+%E2%80%9CVizwiz+grand+challenge%3A+Answering+visual+questions+from+blind+people%E2%80%9D%2E+In%3A+CVPR%2E+2018%2E)
52. K. He et al. “Masked autoencoders are scalable vision learners”. In: CVPR. 2022. [find](https://scholar.google.com/scholar?q=K%2E+He+et+al%2E+%E2%80%9CMasked+autoencoders+are+scalable+vision+learners%E2%80%9D%2E+In%3A+CVPR%2E+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/he-2021-masked-autoencoders-scalable-vision-learners/)
53. X. He et al. “PathVQA: 30000+ Questions for Medical Visual Question Answering”. In: CoRR abs/2003.10286 (2020). — [arXiv:2003.10286](https://arxiv.org/abs/2003.10286) [find](https://scholar.google.com/scholar?q=X%2E+He+et+al%2E+%E2%80%9CPathVQA%3A+30000%2B+Questions+for+Medical+Visual+Question+Answering%E2%80%9D%2E+In%3A+CoRR+abs%2F2003%2E10286+%282020%29%2E+%E2%80%94+%5BarXiv%3A2003%2E10286%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2003%2E10286%29)
54. T. Hiippala et al. “AI2D-RST: A multimodal corpus of 1000 primary school science diagrams”. In: Language Resources and Evaluation 55 (2021), pp. 661–688. [find](https://scholar.google.com/scholar?q=T%2E+Hiippala+et+al%2E+%E2%80%9CAI2D%2DRST%3A+A+multimodal+corpus+of+1000+primary+school+science+diagrams%E2%80%9D%2E+In%3A+Language+Resources+and+Evaluation+55+%282021%29%2C+pp%2E+661%E2%80%93688%2E)
55. J. Hoffmann et al. “Training compute-optimal large language models”. In: NeurIPS (2023). [find](https://scholar.google.com/scholar?q=J%2E+Hoffmann+et+al%2E+%E2%80%9CTraining+compute%2Doptimal+large+language+models%E2%80%9D%2E+In%3A+NeurIPS+%282023%29%2E)
56. Y.-C. Hsiao, F. Zubach, M. Wang, et al. “Screenqa: Large-scale question-answer pairs over mobile app screenshots”. In: arXiv preprint arXiv:2209.08199 (2022). — [arXiv:2209.08199](https://arxiv.org/abs/2209.08199) [find](https://scholar.google.com/scholar?q=Y%2E%2DC%2E+Hsiao%2C+F%2E+Zubach%2C+M%2E+Wang%2C+et+al%2E+%E2%80%9CScreenqa%3A+Large%2Dscale+question%2Danswer+pairs+over+mobile+app+screenshots%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2209%2E08199+%282022%29%2E+%E2%80%94+%5BarXiv%3A2209%2E08199%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
57. D. A. Hudson and C. D. Manning. “GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering”. In: CVPR. 2019. [find](https://scholar.google.com/scholar?q=D%2E+A%2E+Hudson+and+C%2E+D%2E+Manning%2E+%E2%80%9CGQA%3A+A+New+Dataset+for+Real%2DWorld+Visual+Reasoning+and+Compositional+Question+Answering%E2%80%9D%2E+In%3A+CVPR%2E+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/)
58. A. Jaegle et al. “Perceiver: General perception with iterative attention”. In: ICML. 2021. [find](https://scholar.google.com/scholar?q=A%2E+Jaegle+et+al%2E+%E2%80%9CPerceiver%3A+General+perception+with+iterative+attention%E2%80%9D%2E+In%3A+ICML%2E+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/jaegle-2021-perceiver-general-perception-iterative-attention/)
59. J. Johnson et al. “Clevr: A diagnostic dataset for compositional language and elementary visual reasoning”. In: CVPR. 2017. [find](https://scholar.google.com/scholar?q=J%2E+Johnson+et+al%2E+%E2%80%9CClevr%3A+A+diagnostic+dataset+for+compositional+language+and+elementary+visual+reasoning%E2%80%9D%2E+In%3A+CVPR%2E+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/johnson-2016-clevr-diagnostic-dataset-compositional-language/)
60. N. Jouppi et al. “Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings”. In: Proceedings of the 50th Annual International Symposium on Computer Architecture. 2023. [find](https://scholar.google.com/scholar?q=N%2E+Jouppi+et+al%2E+%E2%80%9CTpu+v4%3A+An+optically+reconfigurable+supercomputer+for+machine+learning+with+hardware+support+for+embeddings%E2%80%9D%2E+In%3A+Proceedings+of+the+50th+Annual+International+Symposium+on+Computer+A)
61. K. Kafle et al. “Dvqa: Understanding data visualizations via question answering”. In: CVPR. 2018. [find](https://scholar.google.com/scholar?q=K%2E+Kafle+et+al%2E+%E2%80%9CDvqa%3A+Understanding+data+visualizations+via+question+answering%E2%80%9D%2E+In%3A+CVPR%2E+2018%2E)
62. S. Kantharaj et al. “Chart-to-text: A large-scale benchmark for chart summarization”. In: ACL. 2022. [find](https://scholar.google.com/scholar?q=S%2E+Kantharaj+et+al%2E+%E2%80%9CChart%2Dto%2Dtext%3A+A+large%2Dscale+benchmark+for+chart+summarization%E2%80%9D%2E+In%3A+ACL%2E+2022%2E)
63. S. Karamcheti et al. “Prismatic vlms: Investigating the design space of visually-conditioned language models”. In: arXiv preprint arXiv:2402.07865 (2024). — [arXiv:2402.07865](https://arxiv.org/abs/2402.07865) [find](https://scholar.google.com/scholar?q=S%2E+Karamcheti+et+al%2E+%E2%80%9CPrismatic+vlms%3A+Investigating+the+design+space+of+visually%2Dconditioned+language+models%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2402%2E07865+%282024%29%2E+%E2%80%94+%5BarXiv%3A2402%2E07865%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F24)
64. M. Kazemi et al. “Geomverse: A systematic evaluation of large models for geometric reasoning”. In: 2023. [find](https://scholar.google.com/scholar?q=M%2E+Kazemi+et+al%2E+%E2%80%9CGeomverse%3A+A+systematic+evaluation+of+large+models+for+geometric+reasoning%E2%80%9D%2E+In%3A+2023%2E)
65. A. Kembhavi et al. “A diagram is worth a dozen images”. In: ECCV. 2016. [find](https://scholar.google.com/scholar?q=A%2E+Kembhavi+et+al%2E+%E2%80%9CA+diagram+is+worth+a+dozen+images%E2%80%9D%2E+In%3A+ECCV%2E+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/kembhavi-2016-diagram-worth-dozen-images/)
66. D. Kiela et al. “The hateful memes challenge: Detecting hate speech in multimodal memes”. In: NeurIPS. 2020. [find](https://scholar.google.com/scholar?q=D%2E+Kiela+et+al%2E+%E2%80%9CThe+hateful+memes+challenge%3A+Detecting+hate+speech+in+multimodal+memes%E2%80%9D%2E+In%3A+NeurIPS%2E+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/kiela-2020-hateful-memes-challenge-detecting-hate/)
67. G. Kim et al. “Donut: Document understanding transformer without ocr”. In: ECCV. 2022. [find](https://scholar.google.com/scholar?q=G%2E+Kim+et+al%2E+%E2%80%9CDonut%3A+Document+understanding+transformer+without+ocr%E2%80%9D%2E+In%3A+ECCV%2E+2022%2E)
68. A. Kirillov et al. “Segment anything”. In: ICCV. 2023. [find](https://scholar.google.com/scholar?q=A%2E+Kirillov+et+al%2E+%E2%80%9CSegment+anything%E2%80%9D%2E+In%3A+ICCV%2E+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/kirillov-2023-segment-anything/)
69. R. Krishna et al. “Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations”. In: IJCV (2016). [find](https://scholar.google.com/scholar?q=R%2E+Krishna+et+al%2E+%E2%80%9CVisual+Genome%3A+Connecting+Language+and+Vision+Using+Crowdsourced+Dense+Image+Annotations%E2%80%9D%2E+In%3A+IJCV+%282016%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/)
70. LAION. laion/gpt4v-dataset. 2023. [find](https://scholar.google.com/scholar?q=LAION%2E+laion%2Fgpt4v%2Ddataset%2E+2023%2E)
71. H. Laurençon, L. Tronchon, and V. Sanh. “Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset”. In: arXiv preprint arXiv:2403.09029 (2024). — [arXiv:2403.09029](https://arxiv.org/abs/2403.09029) [find](https://scholar.google.com/scholar?q=H%2E+Lauren%C3%A7on%2C+L%2E+Tronchon%2C+and+V%2E+Sanh%2E+%E2%80%9CUnlocking+the+conversion+of+Web+Screenshots+into+HTML+Code+with+the+WebSight+Dataset%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2403%2E09029+%282024%29%2E+%E2%80%94+%5BarXiv%3A2403%2E09029%5D%28https%3A%2F)
72. H. Laurençon et al. “What matters when building vision-language models?” In: arXiv preprint arXiv:2405.02246 (2024). [find](https://scholar.google.com/scholar?q=H%2E+Lauren%C3%A7on+et+al%2E+%E2%80%9CWhat+matters+when+building+vision%2Dlanguage+models%3F%E2%80%9D+In%3A+arXiv+preprint+arXiv%3A2405%2E02246+%282024%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/lauren-2024-ccon-matters-building-vision-language-models/)
73. A. C. Li et al. “Internet Explorer: Targeted Representation Learning on the Open Web”. In: ICML. 2023. [find](https://scholar.google.com/scholar?q=A%2E+C%2E+Li+et+al%2E+%E2%80%9CInternet+Explorer%3A+Targeted+Representation+Learning+on+the+Open+Web%E2%80%9D%2E+In%3A+ICML%2E+2023%2E)
74. A. C. Li et al. “Your diffusion model is secretly a zero-shot classifier”. In: ICCV. 2023. [find](https://scholar.google.com/scholar?q=A%2E+C%2E+Li+et+al%2E+%E2%80%9CYour+diffusion+model+is+secretly+a+zero%2Dshot+classifier%E2%80%9D%2E+In%3A+ICCV%2E+2023%2E)
75. B. Li et al. LLaVA-NeXT: Stronger LLMs Supercharge Multimodal Capabilities in the Wild. 2024. [find](https://scholar.google.com/scholar?q=B%2E+Li+et+al%2E+LLaVA%2DNeXT%3A+Stronger+LLMs+Supercharge+Multimodal+Capabilities+in+the+Wild%2E+2024%2E)
76. L. Li et al. “Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models”. In: arXiv preprint arXiv:2403.00231 (2024). [find](https://scholar.google.com/scholar?q=L%2E+Li+et+al%2E+%E2%80%9CMultimodal+ArXiv%3A+A+Dataset+for+Improving+Scientific+Comprehension+of+Large+Vision%2DLanguage+Models%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2403%2E00231+%282024%29%2E)
77. Y. Li et al. “Mini-gemini: Mining the potential of multi-modality vision language models”. In: arXiv preprint arXiv:2403.18814 (2024). [find](https://scholar.google.com/scholar?q=Y%2E+Li+et+al%2E+%E2%80%9CMini%2Dgemini%3A+Mining+the+potential+of+multi%2Dmodality+vision+language+models%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2403%2E18814+%282024%29%2E)
78. W. Lian et al. OpenOrca: An Open Dataset of GPT Augmented FLAN Reasoning Traces. https://https://huggingface.co/Open-Orca/OpenOrca . 2023. [find](https://scholar.google.com/scholar?q=W%2E+Lian+et+al%2E+OpenOrca%3A+An+Open+Dataset+of+GPT+Augmented+FLAN+Reasoning+Traces%2E+https%3A%2F%2Fhttps%3A%2F%2Fhuggingface%2Eco%2FOpen%2DOrca%2FOpenOrca+%2E+2023%2E)
79. T.-Y. Lin et al. “Microsoft coco: Common objects in context”. In: ECCV. 2014. [find](https://scholar.google.com/scholar?q=T%2E%2DY%2E+Lin+et+al%2E+%E2%80%9CMicrosoft+coco%3A+Common+objects+in+context%E2%80%9D%2E+In%3A+ECCV%2E+2014%2E) [our card](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/)
80. H. Liu et al. “Improved baselines with visual instruction tuning”. In: arXiv preprint arXiv:2310.03744 (2023). [find](https://scholar.google.com/scholar?q=H%2E+Liu+et+al%2E+%E2%80%9CImproved+baselines+with+visual+instruction+tuning%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2310%2E03744+%282023%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/)
81. H. Liu et al. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge. 2024. [find](https://scholar.google.com/scholar?q=H%2E+Liu+et+al%2E+LLaVA%2DNeXT%3A+Improved+reasoning%2C+OCR%2C+and+world+knowledge%2E+2024%2E)
82. H. Liu et al. “Visual Instruction Tuning”. In: NeurIPS. 2023. [find](https://scholar.google.com/scholar?q=H%2E+Liu+et+al%2E+%E2%80%9CVisual+Instruction+Tuning%E2%80%9D%2E+In%3A+NeurIPS%2E+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/)
83. Y. Liu et al. “Mmbench: Is your multi-modal model an all-around player?” In: arXiv preprint arXiv:2307.06281 (2023). [find](https://scholar.google.com/scholar?q=Y%2E+Liu+et+al%2E+%E2%80%9CMmbench%3A+Is+your+multi%2Dmodal+model+an+all%2Daround+player%3F%E2%80%9D+In%3A+arXiv+preprint+arXiv%3A2307%2E06281+%282023%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/)
84. Y. Liu et al. “On the hidden mystery of ocr in large multimodal models”. In: arXiv preprint arXiv:2305.07895 (2023). [find](https://scholar.google.com/scholar?q=Y%2E+Liu+et+al%2E+%E2%80%9COn+the+hidden+mystery+of+ocr+in+large+multimodal+models%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2305%2E07895+%282023%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-ocrbench-hidden-mystery-ocr-large/)
85. Z. Liu and K. He. “A Decade’s Battle on Dataset Bias: Are We There Yet?” In: arXiv preprint arXiv:2403.08632 (2024). — [arXiv:2403.08632](https://arxiv.org/abs/2403.08632) [find](https://scholar.google.com/scholar?q=Z%2E+Liu+and+K%2E+He%2E+%E2%80%9CA+Decade%E2%80%99s+Battle+on+Dataset+Bias%3A+Are+We+There+Yet%3F%E2%80%9D+In%3A+arXiv+preprint+arXiv%3A2403%2E08632+%282024%29%2E+%E2%80%94+%5BarXiv%3A2403%2E08632%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2403%2E08632%29)
86. Z. Liu et al. “A convnet for the 2020s”. In: CVPR. 2022. [find](https://scholar.google.com/scholar?q=Z%2E+Liu+et+al%2E+%E2%80%9CA+convnet+for+the+2020s%E2%80%9D%2E+In%3A+CVPR%2E+2022%2E)
87. H. Lu et al. “DeepSeek-VL: towards real-world vision-language understanding”. In: arXiv preprint arXiv:2403.05525 (2024). [find](https://scholar.google.com/scholar?q=H%2E+Lu+et+al%2E+%E2%80%9CDeepSeek%2DVL%3A+towards+real%2Dworld+vision%2Dlanguage+understanding%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2403%2E05525+%282024%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/lu-2024-deepseek-vl-real-world/)
88. P. Lu et al. “Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning”. In: ICLR. 2023. [find](https://scholar.google.com/scholar?q=P%2E+Lu+et+al%2E+%E2%80%9CDynamic+prompt+learning+via+policy+gradient+for+semi%2Dstructured+mathematical+reasoning%E2%80%9D%2E+In%3A+ICLR%2E+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/lu-2022-dynamic-prompt-learning-policy-gradient/)
89. P. Lu et al. “Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning”. In: NeurIPS. 2021. [find](https://scholar.google.com/scholar?q=P%2E+Lu+et+al%2E+%E2%80%9CIconqa%3A+A+new+benchmark+for+abstract+diagram+understanding+and+visual+language+reasoning%E2%80%9D%2E+In%3A+NeurIPS%2E+2021%2E)
90. P. Lu et al. “Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning”. In: ACL. 2021. [find](https://scholar.google.com/scholar?q=P%2E+Lu+et+al%2E+%E2%80%9CInter%2DGPS%3A+Interpretable+geometry+problem+solving+with+formal+language+and+symbolic+reasoning%E2%80%9D%2E+In%3A+ACL%2E+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/lu-2021-inter-gps-interpretable-geometry-problem/)
91. P. Lu et al. “Learn to explain: Multimodal reasoning via thought chains for science question answering”. In: NeurIPS. 2022. [find](https://scholar.google.com/scholar?q=P%2E+Lu+et+al%2E+%E2%80%9CLearn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+science+question+answering%E2%80%9D%2E+In%3A+NeurIPS%2E+2022%2E)
92. P. Lu et al. “Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts”. In: ICLR (2023). [find](https://scholar.google.com/scholar?q=P%2E+Lu+et+al%2E+%E2%80%9CMathvista%3A+Evaluating+mathematical+reasoning+of+foundation+models+in+visual+contexts%E2%80%9D%2E+In%3A+ICLR+%282023%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/)
93. Z. Luo et al. “Wizardcoder: Empowering code large language models with evol-instruct”. In: ICLR. 2024. [find](https://scholar.google.com/scholar?q=Z%2E+Luo+et+al%2E+%E2%80%9CWizardcoder%3A+Empowering+code+large+language+models+with+evol%2Dinstruct%E2%80%9D%2E+In%3A+ICLR%2E+2024%2E)
94. A. Majumdar et al. “OpenEQA: Embodied Question Answering in the Era of Foundation Models”. In: 2nd Workshop on Mobile Manipulation and Embodied Intelligence at ICRA 2024. 2024. [find](https://scholar.google.com/scholar?q=A%2E+Majumdar+et+al%2E+%E2%80%9COpenEQA%3A+Embodied+Question+Answering+in+the+Era+of+Foundation+Models%E2%80%9D%2E+In%3A+2nd+Workshop+on+Mobile+Manipulation+and+Embodied+Intelligence+at+ICRA+2024%2E+2024%2E)
95. K. Marino et al. “OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge”. In: CVPR. 2019. [find](https://scholar.google.com/scholar?q=K%2E+Marino+et+al%2E+%E2%80%9COK%2DVQA%3A+A+Visual+Question+Answering+Benchmark+Requiring+External+Knowledge%E2%80%9D%2E+In%3A+CVPR%2E+2019%2E)
96. A. Masry et al. “Chartqa: A benchmark for question answering about charts with visual and logical reasoning”. In: ACL. 2022. [find](https://scholar.google.com/scholar?q=A%2E+Masry+et+al%2E+%E2%80%9CChartqa%3A+A+benchmark+for+question+answering+about+charts+with+visual+and+logical+reasoning%E2%80%9D%2E+In%3A+ACL%2E+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/)
97. M. Mathew, D. Karatzas, and C. Jawahar. “Docvqa: A dataset for vqa on document images”. In: WACV. 2021. [find](https://scholar.google.com/scholar?q=M%2E+Mathew%2C+D%2E+Karatzas%2C+and+C%2E+Jawahar%2E+%E2%80%9CDocvqa%3A+A+dataset+for+vqa+on+document+images%E2%80%9D%2E+In%3A+WACV%2E+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/mathew-2020-docvqa-dataset-vqa-document-images/)
98. B. McKinzie et al. “Mm1: Methods, analysis & insights from multimodal llm pre-training”. In: arXiv preprint arXiv:2403.09611 (2024). — [arXiv:2403.09611](https://arxiv.org/abs/2403.09611) [find](https://scholar.google.com/scholar?q=B%2E+McKinzie+et+al%2E+%E2%80%9CMm1%3A+Methods%2C+analysis+%26+insights+from+multimodal+llm+pre%2Dtraining%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2403%2E09611+%282024%29%2E+%E2%80%94+%5BarXiv%3A2403%2E09611%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2403%2E09611%29)
99. A. Mitra et al. Orca-Math: Unlocking the potential of SLMs in Grade School Math. 2024. arXiv: 2402.14830 [cs.CL] . [find](https://scholar.google.com/scholar?q=A%2E+Mitra+et+al%2E+Orca%2DMath%3A+Unlocking+the+potential+of+SLMs+in+Grade+School+Math%2E+2024%2E+arXiv%3A+2402%2E14830+%5Bcs%2ECL%5D+%2E) [our card](https://md.research.elamur.ai/free/en/papers/mitra-2024-orca-math-unlocking-potential-slms/)
100. “OCR-VQA: Visual Question Answering by Reading Text in Images”. In: 2019. [find](https://scholar.google.com/scholar?q=%E2%80%9COCR%2DVQA%3A+Visual+Question+Answering+by+Reading+Text+in+Images%E2%80%9D%2E+In%3A+2019%2E)
101. M. Oquab et al. “Dinov2: Learning robust visual features without supervision”. In: TMLR (2023). [find](https://scholar.google.com/scholar?q=M%2E+Oquab+et+al%2E+%E2%80%9CDinov2%3A+Learning+robust+visual+features+without+supervision%E2%80%9D%2E+In%3A+TMLR+%282023%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/oquab-2023-dinov2-learning-robust-visual-features/)
102. L. Ouyang et al. “Training language models to follow instructions with human feedback”. In: NeurIPS. 2022. [find](https://scholar.google.com/scholar?q=L%2E+Ouyang+et+al%2E+%E2%80%9CTraining+language+models+to+follow+instructions+with+human+feedback%E2%80%9D%2E+In%3A+NeurIPS%2E+2022%2E)
103. A. Parker. In the blink of an eye: how vision sparked the big bang of evolution. 2003. [find](https://scholar.google.com/scholar?q=A%2E+Parker%2E+In+the+blink+of+an+eye%3A+how+vision+sparked+the+big+bang+of+evolution%2E+2003%2E)
104. P. Pasupat and P. Liang. “Compositional semantic parsing on semi-structured tables”. In: ACL. 2015. [find](https://scholar.google.com/scholar?q=P%2E+Pasupat+and+P%2E+Liang%2E+%E2%80%9CCompositional+semantic+parsing+on+semi%2Dstructured+tables%E2%80%9D%2E+In%3A+ACL%2E+2015%2E)
105. J. Piaget, M. Cook, et al. The origins of intelligence in children. Vol. 8. 5. International Universities Press New York, 1952. [find](https://scholar.google.com/scholar?q=J%2E+Piaget%2C+M%2E+Cook%2C+et+al%2E+The+origins+of+intelligence+in+children%2E+Vol%2E+8%2E+5%2E+International+Universities+Press+New+York%2C+1952%2E)
106. J. Pont-Tuset et al. “Connecting Vision and Language with Localized Narratives”. In: ECCV. 2020. [find](https://scholar.google.com/scholar?q=J%2E+Pont%2DTuset+et+al%2E+%E2%80%9CConnecting+Vision+and+Language+with+Localized+Narratives%E2%80%9D%2E+In%3A+ECCV%2E+2020%2E)
107. A. Radford et al. “Learning transferable visual models from natural language supervision”. In: ICML. 2021. [find](https://scholar.google.com/scholar?q=A%2E+Radford+et+al%2E+%E2%80%9CLearning+transferable+visual+models+from+natural+language+supervision%E2%80%9D%2E+In%3A+ICML%2E+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/)
108. R. Rafailov et al. “Direct preference optimization: Your language model is secretly a reward model”. In: NeurIPS. 2024. [find](https://scholar.google.com/scholar?q=R%2E+Rafailov+et+al%2E+%E2%80%9CDirect+preference+optimization%3A+Your+language+model+is+secretly+a+reward+model%E2%80%9D%2E+In%3A+NeurIPS%2E+2024%2E)
109. M. Roberts et al. “Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding”. In: ICCV. 2021. [find](https://scholar.google.com/scholar?q=M%2E+Roberts+et+al%2E+%E2%80%9CHypersim%3A+A+Photorealistic+Synthetic+Dataset+for+Holistic+Indoor+Scene+Understanding%E2%80%9D%2E+In%3A+ICCV%2E+2021%2E)
110. R. Rombach et al. “High-Resolution Image Synthesis With Latent Diffusion Models”. In: CVPR. 2022. [find](https://scholar.google.com/scholar?q=R%2E+Rombach+et+al%2E+%E2%80%9CHigh%2DResolution+Image+Synthesis+With+Latent+Diffusion+Models%E2%80%9D%2E+In%3A+CVPR%2E+2022%2E)
111. O. Russakovsky et al. “Imagenet large scale visual recognition challenge”. In: IJCV (2015). [find](https://scholar.google.com/scholar?q=O%2E+Russakovsky+et+al%2E+%E2%80%9CImagenet+large+scale+visual+recognition+challenge%E2%80%9D%2E+In%3A+IJCV+%282015%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/russakovsky-2014-imagenet-large-scale-visual-recognition/)
112. O. Sanseviero. LLM Evals and Benchmarking. 2022. [find](https://scholar.google.com/scholar?q=O%2E+Sanseviero%2E+LLM+Evals+and+Benchmarking%2E+2022%2E)
113. C. Schuhmann et al. “Laion-5b: An open large-scale dataset for training next generation image-text models”. In: NeurIPS. 2022. [find](https://scholar.google.com/scholar?q=C%2E+Schuhmann+et+al%2E+%E2%80%9CLaion%2D5b%3A+An+open+large%2Dscale+dataset+for+training+next+generation+image%2Dtext+models%E2%80%9D%2E+In%3A+NeurIPS%2E+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/schuhmann-2022-laion-5b-open-large-scale/)
114. D. Schwenk et al. “A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge”. In: ECCV. 2022. [find](https://scholar.google.com/scholar?q=D%2E+Schwenk+et+al%2E+%E2%80%9CA%2DOKVQA%3A+A+Benchmark+for+Visual+Question+Answering+using+World+Knowledge%E2%80%9D%2E+In%3A+ECCV%2E+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/schwenk-2022-okvqa-benchmark-visual-question-answering/)
115. M. Shridhar et al. “ALFWorld: Aligning Text and Embodied Environments for Interactive Learning”. In: ICLR. 2021. [find](https://scholar.google.com/scholar?q=M%2E+Shridhar+et+al%2E+%E2%80%9CALFWorld%3A+Aligning+Text+and+Embodied+Environments+for+Interactive+Learning%E2%80%9D%2E+In%3A+ICLR%2E+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/shridhar-2020-alfworld-aligning-text-embodied-environments/)
116. C. Si et al. “Design2Code: How Far Are We From Automating Front-End Engineering?” In: arXiv preprint arXiv:2403.03163 (2024). — [arXiv:2403.03163](https://arxiv.org/abs/2403.03163) [find](https://scholar.google.com/scholar?q=C%2E+Si+et+al%2E+%E2%80%9CDesign2Code%3A+How+Far+Are+We+From+Automating+Front%2DEnd+Engineering%3F%E2%80%9D+In%3A+arXiv+preprint+arXiv%3A2403%2E03163+%282024%29%2E+%E2%80%94+%5BarXiv%3A2403%2E03163%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2403%2E03163%29)
117. O. Sidorov et al. TextCaps: a Dataset for Image Captioning with Reading Comprehension. 2020. arXiv: 2003.12462 [cs.CV] . [find](https://scholar.google.com/scholar?q=O%2E+Sidorov+et+al%2E+TextCaps%3A+a+Dataset+for+Image+Captioning+with+Reading+Comprehension%2E+2020%2E+arXiv%3A+2003%2E12462+%5Bcs%2ECV%5D+%2E)
118. A. Singh et al. “Towards vqa models that can read”. In: CVPR. 2019. [find](https://scholar.google.com/scholar?q=A%2E+Singh+et+al%2E+%E2%80%9CTowards+vqa+models+that+can+read%E2%80%9D%2E+In%3A+CVPR%2E+2019%2E)
119. S. Song, S. P. Lichtenberg, and J. Xiao. “Sun rgb-d: A rgb-d scene understanding benchmark suite”. In: CVPR. 2015. [find](https://scholar.google.com/scholar?q=S%2E+Song%2C+S%2E+P%2E+Lichtenberg%2C+and+J%2E+Xiao%2E+%E2%80%9CSun+rgb%2Dd%3A+A+rgb%2Dd+scene+understanding+benchmark+suite%E2%80%9D%2E+In%3A+CVPR%2E+2015%2E)
120. Q. Sun et al. “Eva-clip: Improved training techniques for clip at scale”. In: arXiv preprint arXiv:2303.15389 (2023). [find](https://scholar.google.com/scholar?q=Q%2E+Sun+et+al%2E+%E2%80%9CEva%2Dclip%3A+Improved+training+techniques+for+clip+at+scale%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2303%2E15389+%282023%29%2E)
121. R. Tanaka, K. Nishida, and S. Yoshida. “VisualMRC: Machine Reading Comprehension on Document Images”. In: AAAI. 2021. [find](https://scholar.google.com/scholar?q=R%2E+Tanaka%2C+K%2E+Nishida%2C+and+S%2E+Yoshida%2E+%E2%80%9CVisualMRC%3A+Machine+Reading+Comprehension+on+Document+Images%E2%80%9D%2E+In%3A+AAAI%2E+2021%2E)
122. B. J. Tang, A. Boggust, and A. Satyanarayan. “Vistext: A benchmark for semantically rich chart captioning”. In: arXiv preprint arXiv:2307.05356 (2023). [find](https://scholar.google.com/scholar?q=B%2E+J%2E+Tang%2C+A%2E+Boggust%2C+and+A%2E+Satyanarayan%2E+%E2%80%9CVistext%3A+A+benchmark+for+semantically+rich+chart+captioning%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2307%2E05356+%282023%29%2E)
123. S. Tong, E. Jones, and J. Steinhardt. “Mass-producing failures of multimodal systems with language models”. In: NeurIPS. 2024. [find](https://scholar.google.com/scholar?q=S%2E+Tong%2C+E%2E+Jones%2C+and+J%2E+Steinhardt%2E+%E2%80%9CMass%2Dproducing+failures+of+multimodal+systems+with+language+models%E2%80%9D%2E+In%3A+NeurIPS%2E+2024%2E)
124. S. Tong et al. “Eyes wide shut? exploring the visual shortcomings of multimodal llms”. In: CVPR. 2024. [find](https://scholar.google.com/scholar?q=S%2E+Tong+et+al%2E+%E2%80%9CEyes+wide+shut%3F+exploring+the+visual+shortcomings+of+multimodal+llms%E2%80%9D%2E+In%3A+CVPR%2E+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/tong-2024-eyes-wide-shut-exploring-visual/)
125. H. Touvron et al. “LLaMA 2: Open foundation and fine-tuned chat models”. In: (2023). [find](https://scholar.google.com/scholar?q=H%2E+Touvron+et+al%2E+%E2%80%9CLLaMA+2%3A+Open+foundation+and+fine%2Dtuned+chat+models%E2%80%9D%2E+In%3A+%282023%29%2E)
126. H. Touvron et al. “LLaMA: Open and efficient foundation language models”. In: arXiv preprint arXiv:2302.13971 (2023). [find](https://scholar.google.com/scholar?q=H%2E+Touvron+et+al%2E+%E2%80%9CLLaMA%3A+Open+and+efficient+foundation+language+models%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2302%2E13971+%282023%29%2E)
127. H. Tu et al. “How many unicorns are in this image? a safety evaluation benchmark for vision llms”. In: arXiv preprint arXiv:2311.16101 (2023). — [arXiv:2311.16101](https://arxiv.org/abs/2311.16101) [find](https://scholar.google.com/scholar?q=H%2E+Tu+et+al%2E+%E2%80%9CHow+many+unicorns+are+in+this+image%3F+a+safety+evaluation+benchmark+for+vision+llms%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2311%2E16101+%282023%29%2E+%E2%80%94+%5BarXiv%3A2311%2E16101%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E16101%29)
128. K. Vishniakov, Z. Shen, and Z. Liu. “ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracy”. In: ICML. 2024. [find](https://scholar.google.com/scholar?q=K%2E+Vishniakov%2C+Z%2E+Shen%2C+and+Z%2E+Liu%2E+%E2%80%9CConvNet+vs+Transformer%2C+Supervised+vs+CLIP%3A+Beyond+ImageNet+Accuracy%E2%80%9D%2E+In%3A+ICML%2E+2024%2E)
129. J. Wang et al. “To see is to believe: Prompting gpt-4v for better visual instruction tuning”. In: arXiv preprint arXiv:2311.07574 (2023). — [arXiv:2311.07574](https://arxiv.org/abs/2311.07574) [find](https://scholar.google.com/scholar?q=J%2E+Wang+et+al%2E+%E2%80%9CTo+see+is+to+believe%3A+Prompting+gpt%2D4v+for+better+visual+instruction+tuning%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2311%2E07574+%282023%29%2E+%E2%80%94+%5BarXiv%3A2311%2E07574%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E07574%29)
130. K. Wang et al. “Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset”. In: arXiv preprint arXiv:2402.14804 (2024). [find](https://scholar.google.com/scholar?q=K%2E+Wang+et+al%2E+%E2%80%9CMeasuring+Multimodal+Mathematical+Reasoning+with+MATH%2DVision+Dataset%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2402%2E14804+%282024%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2024-measuring-multimodal-mathematical-reasoning-math/)
131. J. Wei et al. “Chain-of-thought prompting elicits reasoning in large language models”. In: NeurIPS. 2022. [find](https://scholar.google.com/scholar?q=J%2E+Wei+et+al%2E+%E2%80%9CChain%2Dof%2Dthought+prompting+elicits+reasoning+in+large+language+models%E2%80%9D%2E+In%3A+NeurIPS%2E+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
132. C. Wendler. wendlerc/RenderedText. 2023. [find](https://scholar.google.com/scholar?q=C%2E+Wendler%2E+wendlerc%2FRenderedText%2E+2023%2E)
133. H. Wu et al. “Q-instruct: Improving low-level visual abilities for multi-modality foundation models”. In: arXiv preprint arXiv:2311.06783 (2023). — [arXiv:2311.06783](https://arxiv.org/abs/2311.06783) [find](https://scholar.google.com/scholar?q=H%2E+Wu+et+al%2E+%E2%80%9CQ%2Dinstruct%3A+Improving+low%2Dlevel+visual+abilities+for+multi%2Dmodality+foundation+models%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2311%2E06783+%282023%29%2E+%E2%80%94+%5BarXiv%3A2311%2E06783%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E06783%29)
134. P. Wu and S. Xie. “V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs”. In: CVPR. 2024. [find](https://scholar.google.com/scholar?q=P%2E+Wu+and+S%2E+Xie%2E+%E2%80%9CV%2A%3A+Guided+Visual+Search+as+a+Core+Mechanism+in+Multimodal+LLMs%E2%80%9D%2E+In%3A+CVPR%2E+2024%2E)
135. H. Xu et al. “Demystifying clip data”. In: ICLR. 2024. [find](https://scholar.google.com/scholar?q=H%2E+Xu+et+al%2E+%E2%80%9CDemystifying+clip+data%E2%80%9D%2E+In%3A+ICLR%2E+2024%2E)
136. A. Young et al. “Yi: Open foundation models by 01. ai”. In: arXiv preprint arXiv:2403.04652 (2024). [find](https://scholar.google.com/scholar?q=A%2E+Young+et+al%2E+%E2%80%9CYi%3A+Open+foundation+models+by+01%2E+ai%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2403%2E04652+%282024%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/)
137. L. Yu et al. Modeling Context in Referring Expressions. 2016. arXiv: 1608.00272 [cs.CV] . [find](https://scholar.google.com/scholar?q=L%2E+Yu+et+al%2E+Modeling+Context+in+Referring+Expressions%2E+2016%2E+arXiv%3A+1608%2E00272+%5Bcs%2ECV%5D+%2E) [our card](https://md.research.elamur.ai/free/en/papers/yu-2016-modeling-context-referring-expressions/)
138. T. Yu et al. “Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback”. In: arXiv preprint arXiv:2312.00849 (2023). [find](https://scholar.google.com/scholar?q=T%2E+Yu+et+al%2E+%E2%80%9CRlhf%2Dv%3A+Towards+trustworthy+mllms+via+behavior+alignment+from+fine%2Dgrained+correctional+human+feedback%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2312%2E00849+%282023%29%2E)
139. X. Yue et al. “Mammoth: Building math generalist models through hybrid instruction tuning”. In: ICLR. 2024. [find](https://scholar.google.com/scholar?q=X%2E+Yue+et+al%2E+%E2%80%9CMammoth%3A+Building+math+generalist+models+through+hybrid+instruction+tuning%E2%80%9D%2E+In%3A+ICLR%2E+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/)
140. X. Yue et al. “Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi”. In: CVPR. 2024. [find](https://scholar.google.com/scholar?q=X%2E+Yue+et+al%2E+%E2%80%9CMmmu%3A+A+massive+multi%2Ddiscipline+multimodal+understanding+and+reasoning+benchmark+for+expert+agi%E2%80%9D%2E+In%3A+CVPR%2E+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/)
141. M. Yuksekgonul et al. “When and why vision-language models behave like bags-of-words, and what to do about it?” In: ICLR. 2022. [find](https://scholar.google.com/scholar?q=M%2E+Yuksekgonul+et+al%2E+%E2%80%9CWhen+and+why+vision%2Dlanguage+models+behave+like+bags%2Dof%2Dwords%2C+and+what+to+do+about+it%3F%E2%80%9D+In%3A+ICLR%2E+2022%2E)
142. X. Zhai et al. “Sigmoid loss for language image pre-training”. In: ICCV. 2023. [find](https://scholar.google.com/scholar?q=X%2E+Zhai+et+al%2E+%E2%80%9CSigmoid+loss+for+language+image+pre%2Dtraining%E2%80%9D%2E+In%3A+ICCV%2E+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhai-2023-sigmoid-loss-language-image-pre/)
143. Y. Zhai et al. “Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning”. In: arXiv preprint arXiv:2405.10292 (2024). — [arXiv:2405.10292](https://arxiv.org/abs/2405.10292) [find](https://scholar.google.com/scholar?q=Y%2E+Zhai+et+al%2E+%E2%80%9CFine%2DTuning+Large+Vision%2DLanguage+Models+as+Decision%2DMaking+Agents+via+Reinforcement+Learning%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2405%2E10292+%282024%29%2E+%E2%80%94+%5BarXiv%3A2405%2E10292%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
144. Y. Zhai et al. “Investigating the catastrophic forgetting in multimodal large language models”. In: CPAL. 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Zhai+et+al%2E+%E2%80%9CInvestigating+the+catastrophic+forgetting+in+multimodal+large+language+models%E2%80%9D%2E+In%3A+CPAL%2E+2024%2E)
145. C. Zhang et al. “Raven: A dataset for relational and analogical visual reasoning”. In: CVPR. 2019. [find](https://scholar.google.com/scholar?q=C%2E+Zhang+et+al%2E+%E2%80%9CRaven%3A+A+dataset+for+relational+and+analogical+visual+reasoning%E2%80%9D%2E+In%3A+CVPR%2E+2019%2E)
146. Y. Zhang et al. “Llavar: Enhanced visual instruction tuning for text-rich image understanding”. In: arXiv preprint arXiv:2306.17107 (2023). [find](https://scholar.google.com/scholar?q=Y%2E+Zhang+et+al%2E+%E2%80%9CLlavar%3A+Enhanced+visual+instruction+tuning+for+text%2Drich+image+understanding%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2306%2E17107+%282023%29%2E)
147. Y. Zhao et al. “Pytorch fsdp: experiences on scaling fully sharded data parallel”. In: arXiv preprint arXiv:2304.11277 (2023). [find](https://scholar.google.com/scholar?q=Y%2E+Zhao+et+al%2E+%E2%80%9CPytorch+fsdp%3A+experiences+on+scaling+fully+sharded+data+parallel%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2304%2E11277+%282023%29%2E)
148. L. Zheng et al. “Judging llm-as-a-judge with mt-bench and chatbot arena”. In: NeurIPS. 2024. [find](https://scholar.google.com/scholar?q=L%2E+Zheng+et+al%2E+%E2%80%9CJudging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%E2%80%9D%2E+In%3A+NeurIPS%2E+2024%2E)
149. T. Zheng et al. “OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement”. In: arXiv preprint arXiv:2402.14658 (2024). [find](https://scholar.google.com/scholar?q=T%2E+Zheng+et+al%2E+%E2%80%9COpenCodeInterpreter%3A+Integrating+Code+Generation+with+Execution+and+Refinement%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2402%2E14658+%282024%29%2E)
150. V. Zhong, C. Xiong, and R. Socher. “Seq2sql: Generating structured queries from natural language using reinforcement learning”. In: 2017. [find](https://scholar.google.com/scholar?q=V%2E+Zhong%2C+C%2E+Xiong%2C+and+R%2E+Socher%2E+%E2%80%9CSeq2sql%3A+Generating+structured+queries+from+natural+language+using+reinforcement+learning%E2%80%9D%2E+In%3A+2017%2E)
151. B. Zhou et al. “Semantic understanding of scenes through the ade20k dataset”. In: IJCV (2019). [find](https://scholar.google.com/scholar?q=B%2E+Zhou+et+al%2E+%E2%80%9CSemantic+understanding+of+scenes+through+the+ade20k+dataset%E2%80%9D%2E+In%3A+IJCV+%282019%29%2E)
152. K. Zhou et al. “Don’t Make Your LLM an Evaluation Benchmark Cheater”. In: arXiv preprint arXiv:2311.01964 (2023). [find](https://scholar.google.com/scholar?q=K%2E+Zhou+et+al%2E+%E2%80%9CDon%E2%80%99t+Make+Your+LLM+an+Evaluation+Benchmark+Cheater%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2311%2E01964+%282023%29%2E)
153. B. Zhu et al. Starling-7b: Improving llm helpfulness & harmlessness with rlaif. 2023. [find](https://scholar.google.com/scholar?q=B%2E+Zhu+et+al%2E+Starling%2D7b%3A+Improving+llm+helpfulness+%26+harmlessness+with+rlaif%2E+2023%2E)
154. D. Zhu et al. “Minigpt-4: Enhancing vision-language understanding with advanced large language models”. In: arXiv preprint arXiv:2304.10592 (2023). [find](https://scholar.google.com/scholar?q=D%2E+Zhu+et+al%2E+%E2%80%9CMinigpt%2D4%3A+Enhancing+vision%2Dlanguage+understanding+with+advanced+large+language+models%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2304%2E10592+%282023%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/)
155. F. Zhu et al. “TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance”. In: ACL. 2021. [find](https://scholar.google.com/scholar?q=F%2E+Zhu+et+al%2E+%E2%80%9CTAT%2DQA%3A+A+question+answering+benchmark+on+a+hybrid+of+tabular+and+textual+content+in+finance%E2%80%9D%2E+In%3A+ACL%2E+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2021-tat-qa-question-answering-benchmark/)
156. Y. Zhu et al. “Visual7w: Grounded question answering in images”. In: CVPR. 2016. A. Training, Infrastructure, and Implementation [find](https://scholar.google.com/scholar?q=Y%2E+Zhu+et+al%2E+%E2%80%9CVisual7w%3A+Grounded+question+answering+in+images%E2%80%9D%2E+In%3A+CVPR%2E+2016%2E+A%2E+Training%2C+Infrastructure%2C+and+Implementation) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2015-visual7w-grounded-question-answering-images/)