---
title: "NVLM: Open Frontier-Class Multimodal LLMs (2024)"
description: "Open multimodal LLMs usually trail proprietary systems such as GPT-4o. 176 references, cited by 42 papers here."
authors: ["Wenliang Dai", "Nayeon Lee", "Boxin Wang", "Zhuolin Yang", "Zihan Liu", "Jon Barker", "Tuomas Rintamaki", "Mohammad Shoeybi", "Bryan Catanzaro", "Wei Ping"]
year: 2024
venue: "arXiv:2409.11402"
source: "https://arxiv.org/abs/2409.11402"
doi: "10.48550/arxiv.2409.11402"
arxiv: "2409.11402"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/dai-2024-nvlm-open-frontier-class-multimodal/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dai-2024-nvlm-open-frontier-class-multimodal/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / NVLM: Open Frontier-Class Multimodal LLMs

# NVLM: Open Frontier-Class Multimodal LLMs

**Authors:** [Wenliang Dai](https://md.research.elamur.ai/free/en/authors/wenliang-dai/), [Nayeon Lee](https://md.research.elamur.ai/free/en/authors/nayeon-lee/), [Boxin Wang](https://md.research.elamur.ai/free/en/authors/boxin-wang/), [Zhuolin Yang](https://md.research.elamur.ai/free/en/authors/zhuolin-yang/), [Zihan Liu](https://md.research.elamur.ai/free/en/authors/zihan-liu/), Jon Barker, Tuomas Rintamaki, [Mohammad Shoeybi](https://md.research.elamur.ai/free/en/authors/mohammad-shoeybi/), [Bryan Catanzaro](https://md.research.elamur.ai/free/en/authors/bryan-catanzaro/) and [Wei Ping](https://md.research.elamur.ai/free/en/authors/wei-ping/) · **Year:** 2024 · **Venue:** arXiv:2409.11402 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2409.11402) · [PDF](https://arxiv.org/pdf/2409.11402) · [DOI 10.48550/arxiv.2409.11402](https://doi.org/10.48550/arxiv.2409.11402) · [arXiv 2409.11402](https://arxiv.org/abs/2409.11402) · [OpenAlex](https://openalex.org/W4403707794)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [42](https://md.research.elamur.ai/free/en/papers/dai-2024-nvlm-open-frontier-class-multimodal/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Open multimodal LLMs usually trail proprietary systems such as GPT-4o. Their architectures are rarely compared under equal conditions, and they typically lose text-only skill after vision-language training. NVIDIA's NVLM 1.0 addresses all three issues with three variants trained on one shared, carefully curated data mix:

- **NVLM-D:** decoder-only, in the LLaVA style.
- **NVLM-X:** cross-attention, in the Flamingo style.
- **NVLM-H:** a new hybrid meant to pair training efficiency with stronger multimodal reasoning.

High-resolution images are split into dynamic tiles, and a 1-D tile-tagging scheme marks them, which notably improves OCR and reasoning results. The authors find that data quality and task diversity matter more than raw volume, even during pretraining. They add a high-quality text-only SFT set and much multimodal math data to training. As a result, the 72B models compete with leading proprietary and open systems on vision-language benchmarks and even beat their text backbone on math and coding. The work shows that a single openly released model can handle both modalities without sacrificing text ability.

## Key points

- All three 72B variants are built on Qwen2-72B-Instruct. Weights for NVLM-D 72B are published on Hugging Face (nvidia/NVLM-D-72B), and the training code is promised.
- On MMLU, GSM8K, MATH and HumanEval, NVLM-D 1.0 72B raises the average from 79.8 (backbone) to 84.1, a gain of 4.3. NVLM-X gains 2.5 points and NVLM-H gains 2.7.
- Open competitors lose about 6–7 points of text accuracy versus their backbones, e.g. VILA-1.5 40B drops from 54.8 to 47.9. LLaVA-OneVision 72B and InternVL-2-Llama3-76B show similar declines.
- Llama 3-V avoids text degradation by freezing its LLM. For NVLM-X 72B, freezing hurts vision-language scores, especially OCR-heavy ones: OCRBench falls to 722 versus 828 unfrozen, and AI2D to 76.2 versus 84.2.
- Evaluation covers MMMU, MathVista, OCRBench, AI2D, ChartQA, DocVQA, TextVQA, RealWorldQA and VQAv2.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works

## Cite

Cite as: Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro and Wei Ping. 2024. NVLM: Open Frontier-Class Multimodal LLMs. arXiv:2409.11402.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{dai2024nvlm,
  title = {{NVLM: Open Frontier-Class Multimodal LLMs}},
  author = {Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro and Wei Ping},
  year = {2024},
  doi = {10.48550/arxiv.2409.11402},
  eprint = {2409.11402},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2409.11402}
}
```

- [← previous MMToM-QA: Multimodal Theory of Mind Question Answering](https://md.research.elamur.ai/free/en/papers/jin-2024-mmtom-qa-multimodal-theory-mind/)
- [next → Polysemy–-Evidence from Linguistics, Behavioral Science, and Contextualized Language Models](https://md.research.elamur.ai/free/en/papers/haber-2024-polysemy-evidence-linguistics-behavioral-science/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

176 references, 75 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. VQAonBD 2023 Dataset, 2023. URL [https://ilocr.iiit.ac.in/vqabd/dataset.html](https://ilocr.iiit.ac.in/vqabd/dataset.html). [↗](https://ilocr.iiit.ac.in/vqabd/dataset.html)
2. Acharya, M., Kafle, K., and Kanan, C. Tallyqa: Answering complex counting questions. In Proceedings of the AAAI conference on artificial intelligence, 2019. [find](https://scholar.google.com/scholar?q=Acharya%2C+M%2E%2C+Kafle%2C+K%2E%2C+and+Kanan%2C+C%2E+Tallyqa%3A+Answering+complex+counting+questions%2E+In+Proceedings+of+the+AAAI+conference+on+artificial+intelligence%2C+2019%2E)
3. Adept. Adept Fuyu-Heavy: A new multimodal model, 2024. [find](https://scholar.google.com/scholar?q=Adept%2E+Adept+Fuyu%2DHeavy%3A+A+new+multimodal+model%2C+2024%2E)
4. Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. [Flamingo: A visual language model for few-shot learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/). In NeurIPS, 2022.
5. Anthropic. Introducing the next generation of Claude, 2024. [find](https://scholar.google.com/scholar?q=Anthropic%2E+Introducing+the+next+generation+of+Claude%2C+2024%2E)
6. Anthropic. Claude 3.5 Sonnet, 2024. [find](https://scholar.google.com/scholar?q=Anthropic%2E+Claude+3%2E5+Sonnet%2C+2024%2E)
7. Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S., Sagawa, S., Jitsev, J., Kornblith, S., Koh, P. W., Ilharco, G., Wortsman, M., and Schmidt, L. [Openflamingo: An open-source framework for training large autoregressive vision-language models](https://md.research.elamur.ai/free/en/papers/awadalla-2023-openflamingo-open-source-framework-training/). arXi…
8. Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-VL: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.[12966, 2023](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/).
9. Bao, H., Dong, L., Piao, S., and Wei, F. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021. — [arXiv:2106.08254](https://arxiv.org/abs/2106.08254) [find](https://scholar.google.com/scholar?q=Bao%2C+H%2E%2C+Dong%2C+L%2E%2C+Piao%2C+S%2E%2C+and+Wei%2C+F%2E+Beit%3A+Bert+pre%2Dtraining+of+image+transformers%2E+arXiv+preprint+arXiv%3A2106%2E08254%2C+2021%2E+%E2%80%94+%5BarXiv%3A2106%2E08254%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2106%2E08254%29)
10. Biten, A. F., Tito, R., Mafla, A., Gomez, L., Rusinol, M., Valveny, E., Jawahar, C., and Karatzas, D. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4291–4301, 2019. [find](https://scholar.google.com/scholar?q=Biten%2C+A%2E+F%2E%2C+Tito%2C+R%2E%2C+Mafla%2C+A%2E%2C+Gomez%2C+L%2E%2C+Rusinol%2C+M%2E%2C+Valveny%2C+E%2E%2C+Jawahar%2C+C%2E%2C+and+Karatzas%2C+D%2E+Scene+text+visual+question+answering%2E+In+Proceedings+of+the+IEEE%2FCVF+international+conference+on+c)
11. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). NeurIPS, 2020.
12. Byeon, M., Park, B., Kim, H., Lee, S., Baek, W., and Kim, S. Coyo-700m: Image-text pair dataset. [https://github.com/kakaobrain/coyo-dataset](https://github.com/kakaobrain/coyo-dataset), 2022. [↗](https://github.com/kakaobrain/coyo-dataset)
13. Cao, J. and Xiao, J. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings of the 29th International Conference on Computational Linguistics, pp. 1511–1520, 2022. [find](https://scholar.google.com/scholar?q=Cao%2C+J%2E+and+Xiao%2C+J%2E+An+augmented+benchmark+dataset+for+geometric+question+answering+through+dual+parallel+text+encoding%2E+In+Proceedings+of+the+29th+International+Conference+on+Computational+Linguisti)
14. Chen, J., Li, T., Qin, J., Lu, P., Lin, L., Chen, C., and Liang, X. [Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression](https://md.research.elamur.ai/free/en/papers/chen-2022-unigeo-unifying-geometry-logical-reasoning/). arXiv preprint arXiv:2212.02746, 2022.
15. Chen, J., Zhu, D., Shen, X., Li, X., Liu, Z., Zhang, P., Krishnamoorthi, R., Chandra, V., Xiong, Y., and Elhoseiny, M. [MiniGPT-v2: Large language model as a unified interface for vision-language multi-task learning](https://md.research.elamur.ai/free/en/papers/chen-2023-minigpt-v2-large-language-model/). arXiv preprint arXiv:2310.09478, 2023.
16. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021.
17. Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., et al. [Pali: A jointly-scaled multilingual language-image model](https://md.research.elamur.ai/free/en/papers/chen-2022-pali-jointly-scaled-multilingual-language/). arXiv preprint arXiv:2209.06794, 2022.
18. Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al. How far are we to GTP-4V? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.[16821, 2024](https://md.research.elamur.ai/free/en/papers/chen-2024-far-gpt-4v-closing/).
19. Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. [InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks](https://md.research.elamur.ai/free/en/papers/chen-2023-internvl-scaling-up-vision/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185–24198,…
20. Cheng, Z., Dong, H., Wang, Z., Jia, R., Guo, J., Gao, Y., Han, S., Lou, J.-G., and Zhang, D. Hitab: A hierarchical table dataset for question answering and natural language generation. arXiv preprint arXiv:2108.06712, 2021. [find](https://scholar.google.com/scholar?q=Cheng%2C+Z%2E%2C+Dong%2C+H%2E%2C+Wang%2C+Z%2E%2C+Jia%2C+R%2E%2C+Guo%2C+J%2E%2C+Gao%2C+Y%2E%2C+Han%2C+S%2E%2C+Lou%2C+J%2E%2DG%2E%2C+and+Zhang%2C+D%2E+Hitab%3A+A+hierarchical+table+dataset+for+question+answering+and+natural+language+generation%2E+arXiv+preprint+)
21. Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. URL [https://lmsys.org/blog/2023-03-30-vicuna/](https://lmsys.org/blog/2023-03-30-vicuna/). [↗](https://lmsys.org/blog/2023-03-30-vicuna/)
22. Chng, C. K., Liu, Y., Sun, Y., Ng, C. C., Luo, C., Ni, Z., Fang, C., Zhang, S., Han, J., Ding, E., et al. Icdar2019 robust reading challenge on arbitrary-shaped text-rrc-art. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1571–1576. IEEE, 2019. [find](https://scholar.google.com/scholar?q=Chng%2C+C%2E+K%2E%2C+Liu%2C+Y%2E%2C+Sun%2C+Y%2E%2C+Ng%2C+C%2E+C%2E%2C+Luo%2C+C%2E%2C+Ni%2C+Z%2E%2C+Fang%2C+C%2E%2C+Zhang%2C+S%2E%2C+Han%2C+J%2E%2C+Ding%2C+E%2E%2C+et+al%2E+Icdar2019+robust+reading+challenge+on+arbitrary%2Dshaped+text%2Drrc%2Dart%2E+In+2019+International+Con)
23. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 2024. [find](https://scholar.google.com/scholar?q=Chung%2C+H%2E+W%2E%2C+Hou%2C+L%2E%2C+Longpre%2C+S%2E%2C+Zoph%2C+B%2E%2C+Tay%2C+Y%2E%2C+Fedus%2C+W%2E%2C+Li%2C+Y%2E%2C+Wang%2C+X%2E%2C+Dehghani%2C+M%2E%2C+Brahma%2C+S%2E%2C+et+al%2E+Scaling+instruction%2Dfinetuned+language+models%2E+Journal+of+Machine+Learning+Research)
24. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
25. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/), 2021. URL https://arxiv.org/abs/2110.14168, 2021. [↗](https://arxiv.org/abs/2110.14168)
26. Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/), 2023.
27. Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pp. 7480–7512. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Dehghani%2C+M%2E%2C+Djolonga%2C+J%2E%2C+Mustafa%2C+B%2E%2C+Padlewski%2C+P%2E%2C+Heek%2C+J%2E%2C+Gilmer%2C+J%2E%2C+Steiner%2C+A%2E+P%2E%2C+Caron%2C+M%2E%2C+Geirhos%2C+R%2E%2C+Alabdulmohsin%2C+I%2E%2C+et+al%2E+Scaling+vision+transformers+to+22+billion+parameters%2E+In)
28. Diem, M., Fiel, S., Kleber, F., Sablatnig, R., Saavedra, J. M., Contreras, D., Barrios, J. M., and Oliveira, L. S. Icfhr 2014 competition on handwritten digit string recognition in challenging datasets (hdsrc 2014). In 2014 14th International Conference on Frontiers in Handwriting Recognition, pp. 7… [find](https://scholar.google.com/scholar?q=Diem%2C+M%2E%2C+Fiel%2C+S%2E%2C+Kleber%2C+F%2E%2C+Sablatnig%2C+R%2E%2C+Saavedra%2C+J%2E+M%2E%2C+Contreras%2C+D%2E%2C+Barrios%2C+J%2E+M%2E%2C+and+Oliveira%2C+L%2E+S%2E+Icfhr+2014+competition+on+handwritten+digit+string+recognition+in+challenging+dataset)
29. Ding, Y., Luo, S., Chung, H., and Han, S. C. Pdf-vqa: A new dataset for real-world vqa on pdf documents. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 585–601. Springer, 2023. [find](https://scholar.google.com/scholar?q=Ding%2C+Y%2E%2C+Luo%2C+S%2E%2C+Chung%2C+H%2E%2C+and+Han%2C+S%2E+C%2E+Pdf%2Dvqa%3A+A+new+dataset+for+real%2Dworld+vqa+on+pdf+documents%2E+In+Joint+European+Conference+on+Machine+Learning+and+Knowledge+Discovery+in+Databases%2C+pp%2E+585%E2%80%93)
30. Dong, X., Zhang, P., Zang, Y., Cao, Y., Wang, B., Ouyang, L., Zhang, S., Duan, H., Zhang, W., Li, Y., et al. [InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD](https://md.research.elamur.ai/free/en/papers/dong-2024-internlm-xcomposer2-4khd-pioneering/). arXiv preprint arXiv:2404.06512, 2024.
31. Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, L., Liu, Y., Dong, X., Zang, Y., Zhang, P., Wang, J., Lin, D., and Chen, K. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models, 2024. URL https://arxiv.org/abs/2407.11691. — [arXiv:2407.11691](https://arxiv.org/abs/2407.11691) [↗](https://arxiv.org/abs/2407.11691)
32. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783, 2024.
33. Gadre, S. Y., Ilharco, G., Fang, A., Hayase, J., Smyrnis, G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang, J., et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Gadre%2C+S%2E+Y%2E%2C+Ilharco%2C+G%2E%2C+Fang%2C+A%2E%2C+Hayase%2C+J%2E%2C+Smyrnis%2C+G%2E%2C+Nguyen%2C+T%2E%2C+Marten%2C+R%2E%2C+Wortsman%2C+M%2E%2C+Ghosh%2C+D%2E%2C+Zhang%2C+J%2E%2C+et+al%2E+Datacomp%3A+In+search+of+the+next+generation+of+multimodal+datasets%2E+Adva)
34. Gao, J., Pi, R., Zhang, J., Ye, J., Zhong, W., Wang, Y., Hong, L., Han, J., Xu, H., Li, Z., et al. G-llava: Solving geometric problem with multi-modal large language model. arXiv preprint arXiv:2312.11370, 2023. [find](https://scholar.google.com/scholar?q=Gao%2C+J%2E%2C+Pi%2C+R%2E%2C+Zhang%2C+J%2E%2C+Ye%2C+J%2E%2C+Zhong%2C+W%2E%2C+Wang%2C+Y%2E%2C+Hong%2C+L%2E%2C+Han%2C+J%2E%2C+Xu%2C+H%2E%2C+Li%2C+Z%2E%2C+et+al%2E+G%2Dllava%3A+Solving+geometric+problem+with+multi%2Dmodal+large+language+model%2E+arXiv+preprint+arXiv%3A2312%2E1)
35. Gemini-Team. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
36. Gemini-Team. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.05530, 2024.
37. Glaive-AI. GlaiveCodeAssistant, 2023. URL [https://huggingface.co/datasets/glaiveai/](https://huggingface.co/datasets/glaiveai/) glaive-code-assistant-v2. [↗](https://huggingface.co/datasets/glaiveai/)
38. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. [Making the v in VQA matter: Elevating the role of image understanding in visual question answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904–6913, 2017.
39. Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., and Bigham, J. P. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3608–3617, 2018. [find](https://scholar.google.com/scholar?q=Gurari%2C+D%2E%2C+Li%2C+Q%2E%2C+Stangl%2C+A%2E+J%2E%2C+Guo%2C+A%2E%2C+Lin%2C+C%2E%2C+Grauman%2C+K%2E%2C+Luo%2C+J%2E%2C+and+Bigham%2C+J%2E+P%2E+Vizwiz+grand+challenge%3A+Answering+visual+questions+from+blind+people%2E+In+Proceedings+of+the+IEEE+conference)
40. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300, 2020.
41. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874, 2021.
42. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). arXiv preprint arXiv:2203.15556, 2022.
43. Hsiao, Y.-C., Zubach, F., Wang, M., and Chen, J. ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots, 2024. [find](https://scholar.google.com/scholar?q=Hsiao%2C+Y%2E%2DC%2E%2C+Zubach%2C+F%2E%2C+Wang%2C+M%2E%2C+and+Chen%2C+J%2E+ScreenQA%3A+Large%2DScale+Question%2DAnswer+Pairs+over+Mobile+App+Screenshots%2C+2024%2E)
44. Huang, Z., Chen, K., He, J., Bai, X., Karatzas, D., Lu, S., and Jawahar, C. Icdar2019 competition on scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1516–1520. IEEE, 2019. [find](https://scholar.google.com/scholar?q=Huang%2C+Z%2E%2C+Chen%2C+K%2E%2C+He%2C+J%2E%2C+Bai%2C+X%2E%2C+Karatzas%2C+D%2E%2C+Lu%2C+S%2E%2C+and+Jawahar%2C+C%2E+Icdar2019+competition+on+scanned+receipt+ocr+and+information+extraction%2E+In+2019+International+Conference+on+Document+Analys)
45. Hudson, D. A. and Manning, C. D. [GQA: A new dataset for real-world visual reasoning and compositional question answering](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/). In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700–6709, 2019.
46. IIIT. URL [http://cvit.iiit.ac.in/projects/SceneTextUnderstanding/IIIT5K.html](http://cvit.iiit.ac.in/projects/SceneTextUnderstanding/IIIT5K.html). [↗](http://cvit.iiit.ac.in/projects/SceneTextUnderstanding/IIIT5K.html)
47. Ilharco, G., Wortsman, M., Wightman, R., Gordon, C., Carlini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., and Schmidt, L. OpenCLIP, 2021. [find](https://scholar.google.com/scholar?q=Ilharco%2C+G%2E%2C+Wortsman%2C+M%2E%2C+Wightman%2C+R%2E%2C+Gordon%2C+C%2E%2C+Carlini%2C+N%2E%2C+Taori%2C+R%2E%2C+Dave%2C+A%2E%2C+Shankar%2C+V%2E%2C+Namkoong%2C+H%2E%2C+Miller%2C+J%2E%2C+Hajishirzi%2C+H%2E%2C+Farhadi%2C+A%2E%2C+and+Schmidt%2C+L%2E+OpenCLIP%2C+2021%2E)
48. Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., and Carreira, J. [Perceiver: General perception with iterative attention](https://md.research.elamur.ai/free/en/papers/jaegle-2021-perceiver-general-perception-iterative-attention/). In ICML, 2021.
49. Jaume, G., Ekenel, H. K., and Thiran, J.-P. Funsd: A dataset for form understanding in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2, pp. 1–6. IEEE, 2019. [find](https://scholar.google.com/scholar?q=Jaume%2C+G%2E%2C+Ekenel%2C+H%2E+K%2E%2C+and+Thiran%2C+J%2E%2DP%2E+Funsd%3A+A+dataset+for+form+understanding+in+noisy+scanned+documents%2E+In+2019+International+Conference+on+Document+Analysis+and+Recognition+Workshops+%28ICDARW%29)
50. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E%2C+Sablayrolles%2C+A%2E%2C+Mensch%2C+A%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2E+S%2E%2C+Casas%2C+D%2E+d%2E+l%2E%2C+Bressand%2C+F%2E%2C+Lengyel%2C+G%2E%2C+Lample%2C+G%2E%2C+Saulnier%2C+L%2E%2C+et+al%2E+Mistral+7b%2E+arXiv+preprint+arXiv%3A2310%2E06825%2C+2023%2E)
51. Kafle, K., Price, B., Cohen, S., and Kanan, C. DVQA: Understanding data visualizations via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5648–5656, 2018. [find](https://scholar.google.com/scholar?q=Kafle%2C+K%2E%2C+Price%2C+B%2E%2C+Cohen%2C+S%2E%2C+and+Kanan%2C+C%2E+DVQA%3A+Understanding+data+visualizations+via+question+answering%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pp%2E+564)
52. Kahou, S. E., Michalski, V., Atkinson, A., Kádár, Á., Trischler, A., and Bengio, Y. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300, 2017. [find](https://scholar.google.com/scholar?q=Kahou%2C+S%2E+E%2E%2C+Michalski%2C+V%2E%2C+Atkinson%2C+A%2E%2C+K%C3%A1d%C3%A1r%2C+%C3%81%2E%2C+Trischler%2C+A%2E%2C+and+Bengio%2C+Y%2E+Figureqa%3A+An+annotated+figure+dataset+for+visual+reasoning%2E+arXiv+preprint+arXiv%3A1710%2E07300%2C+2017%2E)
53. Kamizuru, Y., 2023. URL [https://huggingface.co/datasets/Kamizuru00/diagram_image_](https://huggingface.co/datasets/Kamizuru00/diagram_image_) to_text. [↗](https://huggingface.co/datasets/Kamizuru00/diagram_image_)
54. Kazemi, M., Alvari, H., Anand, A., Wu, J., Chen, X., and Soricut, R. Geomverse: A systematic evaluation of large models for geometric reasoning. arXiv preprint arXiv:2312.12241, 2023. — [arXiv:2312.12241](https://arxiv.org/abs/2312.12241) [find](https://scholar.google.com/scholar?q=Kazemi%2C+M%2E%2C+Alvari%2C+H%2E%2C+Anand%2C+A%2E%2C+Wu%2C+J%2E%2C+Chen%2C+X%2E%2C+and+Soricut%2C+R%2E+Geomverse%3A+A+systematic+evaluation+of+large+models+for+geometric+reasoning%2E+arXiv+preprint+arXiv%3A2312%2E12241%2C+2023%2E+%E2%80%94+%5BarXiv%3A2312%2E12)
55. Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. [A diagram is worth a dozen images](https://md.research.elamur.ai/free/en/papers/kembhavi-2016-diagram-worth-dozen-images/). In ECCV, 2016.
56. Kembhavi, A., Seo, M., Schwenk, D., Choi, J., Farhadi, A., and Hajishirzi, H. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition, 2017. [find](https://scholar.google.com/scholar?q=Kembhavi%2C+A%2E%2C+Seo%2C+M%2E%2C+Schwenk%2C+D%2E%2C+Choi%2C+J%2E%2C+Farhadi%2C+A%2E%2C+and+Hajishirzi%2C+H%2E+Are+you+smarter+than+a+sixth+grader%3F+textbook+question+answering+for+multimodal+machine+comprehension%2E+In+Proceedings+of+t)
57. Kim, G., Hong, T., Yim, M., Nam, J., Park, J., Yim, J., Hwang, W., Yun, S., Han, D., and Park, S. [Ocr-free document understanding transformer](https://md.research.elamur.ai/free/en/papers/kim-2021-ocr-free-document-understanding-transformer/). In European Conference on Computer Vision (ECCV), 2022.
58. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. [Segment anything](https://md.research.elamur.ai/free/en/papers/kirillov-2023-segment-anything/). In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015–4026, 2023.
59. Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. [Visual genome: Connecting language and vision using crowdsourced dense image annotations](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/). International journal of computer vision, 123:32–73, 2017.
60. Kuang, J., Hua, W., Liang, D., Yang, M., Jiang, D., Ren, B., and Bai, X. Visual information extraction in the wild: practical dataset and end-to-end solution. In International Conference on Document Analysis and Recognition, 2023. [find](https://scholar.google.com/scholar?q=Kuang%2C+J%2E%2C+Hua%2C+W%2E%2C+Liang%2C+D%2E%2C+Yang%2C+M%2E%2C+Jiang%2C+D%2E%2C+Ren%2C+B%2E%2C+and+Bai%2C+X%2E+Visual+information+extraction+in+the+wild%3A+practical+dataset+and+end%2Dto%2Dend+solution%2E+In+International+Conference+on+Document+A)
61. Laboratory, S. A. Sharegpt-4o, 2024. URL [https://sharegpt4o.github.io/](https://sharegpt4o.github.io/). [↗](https://sharegpt4o.github.io/)
62. Laurencon, H., van Strien, D., Bekman, S., Tronchon, L., Saulnier, L., Wang, T., Karamcheti, S., Singh, A., Pistilli, G., Jernite, Y., et al. Introducing idefics: An open reproduction of state-of-the-art visual language model, 2023. [find](https://scholar.google.com/scholar?q=Laurencon%2C+H%2E%2C+van+Strien%2C+D%2E%2C+Bekman%2C+S%2E%2C+Tronchon%2C+L%2E%2C+Saulnier%2C+L%2E%2C+Wang%2C+T%2E%2C+Karamcheti%2C+S%2E%2C+Singh%2C+A%2E%2C+Pistilli%2C+G%2E%2C+Jernite%2C+Y%2E%2C+et+al%2E+Introducing+idefics%3A+An+open+reproduction+of+state%2Dof%2Dthe%2D)
63. Laurençon, H., Saulnier, L., Tronchon, L., Bekman, S., Singh, A., Lozhkov, A., Wang, T., Karamcheti, S., Rush, A., Kiela, D., et al. [Obelics: An open web-scale filtered dataset of interleaved image-text documents](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2023-obelics-open-web-scale-filtered/). Advances in Neural Information Processing Systems, 2024.
64. Laurençon, H., Tronchon, L., and Sanh, V. Unlocking the conversion of web screenshots into html code with the websight dataset. arXiv preprint arXiv:2403.09029, 2024. — [arXiv:2403.09029](https://arxiv.org/abs/2403.09029) [find](https://scholar.google.com/scholar?q=Lauren%C3%A7on%2C+H%2E%2C+Tronchon%2C+L%2E%2C+and+Sanh%2C+V%2E+Unlocking+the+conversion+of+web+screenshots+into+html+code+with+the+websight+dataset%2E+arXiv+preprint+arXiv%3A2403%2E09029%2C+2024%2E+%E2%80%94+%5BarXiv%3A2403%2E09029%5D%28https%3A%2F%2Farxi)
65. Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Li, Y., Liu, Z., and Li, C. [Llava-onevision: Easy visual task transfer](https://md.research.elamur.ai/free/en/papers/li-2024-llava-onevision-easy-visual-task/). arXiv preprint arXiv:2408.03326, 2024.
66. Li, J., Li, D., Xiong, C., and Hoi, S. [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/). In ICML, 2022.
67. Li, J., Li, D., Savarese, S., and Hoi, S. [BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). In ICML, 2023.
68. Li, L., Wang, Y., Xu, R., Wang, P., Feng, X., Kong, L., and Liu, Q. [Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models](https://md.research.elamur.ai/free/en/papers/li-2024-multimodal-arxiv-dataset-improving-scientific/). arXiv preprint arXiv:2403.00231, 2024.
69. Li, Z., Wang, X., Stengel-Eskin, E., Kortylewski, A., Ma, W., Van Durme, B., and Yuille, A. L. Super-CLEVR: A virtual benchmark to diagnose domain robustness in visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14963–14973, 2023. [find](https://scholar.google.com/scholar?q=Li%2C+Z%2E%2C+Wang%2C+X%2E%2C+Stengel%2DEskin%2C+E%2E%2C+Kortylewski%2C+A%2E%2C+Ma%2C+W%2E%2C+Van+Durme%2C+B%2E%2C+and+Yuille%2C+A%2E+L%2E+Super%2DCLEVR%3A+A+virtual+benchmark+to+diagnose+domain+robustness+in+visual+reasoning%2E+In+Proceedings+of+the)
70. Lian, W., Wang, G., Goodson, B., Pentland, E., Cook, A., Vong, C., and "Teknium". Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification, 2023. URL https://https://huggingface.co/Open-Orca/SlimOrca. [find](https://scholar.google.com/scholar?q=Lian%2C+W%2E%2C+Wang%2C+G%2E%2C+Goodson%2C+B%2E%2C+Pentland%2C+E%2E%2C+Cook%2C+A%2E%2C+Vong%2C+C%2E%2C+and+%22Teknium%22%2E+Slimorca%3A+An+open+dataset+of+gpt%2D4+augmented+flan+reasoning+traces%2C+with+verification%2C+2023%2E+URL+https%3A%2F%2Fhttps%3A%2F%2Fhuggi)
71. Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., and Han, S. [Vila: On pre-training for visual language models](https://md.research.elamur.ai/free/en/papers/lin-2023-vila-pre-training-visual-language/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26689–26699, 2024.
72. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In ECCV 2014, 2014.
73. Lindström, A. D. and Abraham, S. S. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. arXiv preprint arXiv:2208.05358, 2022. — [arXiv:2208.05358](https://arxiv.org/abs/2208.05358) [find](https://scholar.google.com/scholar?q=Lindstr%C3%B6m%2C+A%2E+D%2E+and+Abraham%2C+S%2E+S%2E+Clevr%2Dmath%3A+A+dataset+for+compositional+language%2C+visual+and+mathematical+reasoning%2E+arXiv+preprint+arXiv%3A2208%2E05358%2C+2022%2E+%E2%80%94+%5BarXiv%3A2208%2E05358%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
74. Liu, F., Emerson, G., and Collier, N. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635–651, 2023. [find](https://scholar.google.com/scholar?q=Liu%2C+F%2E%2C+Emerson%2C+G%2E%2C+and+Collier%2C+N%2E+Visual+spatial+reasoning%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+11%3A635%E2%80%93651%2C+2023%2E)
75. Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., and Wang, L. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.[14565, 2023](https://md.research.elamur.ai/free/en/papers/liu-2023-mitigating-hallucination-large-multi-modal/).
76. Liu, F., Wang, X., Yao, W., Chen, J., Song, K., Cho, S., Yacoob, Y., and Yu, D. [Mmc: Advancing multimodal chart understanding with large-scale instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2024-mmc-advancing-multimodal-chart-understanding/). arXiv preprint arXiv:2311.10774, 2023.
77. Liu, H. 2023. Llava-pretrain. [https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain](https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain), [↗](https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain)
78. Liu, H., Li, C., Li, Y., and Lee, Y. J. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/), 2023.
79. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning, 2023. [find](https://scholar.google.com/scholar?q=Liu%2C+H%2E%2C+Li%2C+C%2E%2C+Wu%2C+Q%2E%2C+and+Lee%2C+Y%2E+J%2E+Visual+instruction+tuning%2C+2023%2E)
80. Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL [https://llava-vl.github.io/blog/](https://llava-vl.github.io/blog/) 2024-01-30-llava-next/. [↗](https://llava-vl.github.io/blog/)
81. Liu, Y., Li, Z., Huang, M., Yang, B., Yu, W., Li, C., Yin, X., lin Liu, C., Jin, L., and Bai, X. [Ocrbench: On the hidden mystery of ocr in large multimodal models](https://md.research.elamur.ai/free/en/papers/liu-2023-ocrbench-hidden-mystery-ocr-large/), 2024. URL https://arxiv.org/abs/2305. 07895. [↗](https://arxiv.org/abs/2305)
82. Llama-Team. [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2302.04858, 2023.
83. Lu, P., Gong, R., Jiang, S., Qiu, L., Huang, S., Liang, X., and Zhu, S.-C. [Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning](https://md.research.elamur.ai/free/en/papers/lu-2021-inter-gps-interpretable-geometry-problem/). arXiv preprint arXiv:2105.04165, 2021.
84. Lu, P., Qiu, L., Chen, J., Xia, T., Zhao, Y., Zhang, W., Yu, Z., Liang, X., and Zhu, S.-C. IconQA: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2110.13214, 2021. [find](https://scholar.google.com/scholar?q=Lu%2C+P%2E%2C+Qiu%2C+L%2E%2C+Chen%2C+J%2E%2C+Xia%2C+T%2E%2C+Zhao%2C+Y%2E%2C+Zhang%2C+W%2E%2C+Yu%2C+Z%2E%2C+Liang%2C+X%2E%2C+and+Zhu%2C+S%2E%2DC%2E+IconQA%3A+A+new+benchmark+for+abstract+diagram+understanding+and+visual+language+reasoning%2E+arXiv+preprint+arXi)
85. Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022. [find](https://scholar.google.com/scholar?q=Lu%2C+P%2E%2C+Mishra%2C+S%2E%2C+Xia%2C+T%2E%2C+Qiu%2C+L%2E%2C+Chang%2C+K%2E%2DW%2E%2C+Zhu%2C+S%2E%2DC%2E%2C+Tafjord%2C+O%2E%2C+Clark%2C+P%2E%2C+and+Kalyan%2C+A%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+science+question+answering%2E+Advance)
86. Lu, P., Qiu, L., Chang, K.-W., Wu, Y. N., Zhu, S.-C., Rajpurohit, T., Clark, P., and Kalyan, A. [Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning](https://md.research.elamur.ai/free/en/papers/lu-2022-dynamic-prompt-learning-policy-gradient/). arXiv preprint arXiv:2209.14610, 2022.
87. Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. [Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). In International Conference on Learning Representations (ICLR), 2024.
88. Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang, D. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568, 2023. [find](https://scholar.google.com/scholar?q=Luo%2C+Z%2E%2C+Xu%2C+C%2E%2C+Zhao%2C+P%2E%2C+Sun%2C+Q%2E%2C+Geng%2C+X%2E%2C+Hu%2C+W%2E%2C+Tao%2C+C%2E%2C+Ma%2C+J%2E%2C+Lin%2C+Q%2E%2C+and+Jiang%2C+D%2E+Wizardcoder%3A+Empowering+code+large+language+models+with+evol%2Dinstruct%2E+arXiv+preprint+arXiv%3A2306%2E08568%2C+20)
89. Mahamoud, I. S., Coustaty, M., Joseph, A., d’Andecy, V. P., and Ogier, J.-M. Qalayout: Question answering layout based on multimodal attention for visual question answering on corporate document. In International Workshop on Document Analysis Systems, pp. 659–673. Springer, 2022. [find](https://scholar.google.com/scholar?q=Mahamoud%2C+I%2E+S%2E%2C+Coustaty%2C+M%2E%2C+Joseph%2C+A%2E%2C+d%E2%80%99Andecy%2C+V%2E+P%2E%2C+and+Ogier%2C+J%2E%2DM%2E+Qalayout%3A+Question+answering+layout+based+on+multimodal+attention+for+visual+question+answering+on+corporate+document%2E+In+I)
90. Marafioti, A. and Laurencon, H. Docmatix - a huge dataset for document visual question answering, 2024. [find](https://scholar.google.com/scholar?q=Marafioti%2C+A%2E+and+Laurencon%2C+H%2E+Docmatix+%2D+a+huge+dataset+for+document+visual+question+answering%2C+2024%2E)
91. Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. OK-VQA: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pp. 3195–3204, 2019. [find](https://scholar.google.com/scholar?q=Marino%2C+K%2E%2C+Rastegari%2C+M%2E%2C+Farhadi%2C+A%2E%2C+and+Mottaghi%2C+R%2E+OK%2DVQA%3A+A+visual+question+answering+benchmark+requiring+external+knowledge%2E+In+Proceedings+of+the+IEEE%2Fcvf+conference+on+computer+vision+and+pa)
92. Marti, U.-V. and Bunke, H. The iam-database: an english sentence database for offline handwriting recognition. International journal on document analysis and recognition, 5:39–46, 2002. [find](https://scholar.google.com/scholar?q=Marti%2C+U%2E%2DV%2E+and+Bunke%2C+H%2E+The+iam%2Ddatabase%3A+an+english+sentence+database+for+offline+handwriting+recognition%2E+International+journal+on+document+analysis+and+recognition%2C+5%3A39%E2%80%9346%2C+2002%2E)
93. Masry, A., Long, D. X., Tan, J. Q., Joty, S., and Hoque, E. [ChartQA: A benchmark for question answering about charts with visual and logical reasoning](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/). arXiv preprint arXiv:2203.10244, 2022.
94. Mathew, M., Gomez, L., Karatzas, D., and Jawahar, C. Asking questions on handwritten document collections. International Journal on Document Analysis and Recognition (IJDAR), 24(3):235–249, 2021. [find](https://scholar.google.com/scholar?q=Mathew%2C+M%2E%2C+Gomez%2C+L%2E%2C+Karatzas%2C+D%2E%2C+and+Jawahar%2C+C%2E+Asking+questions+on+handwritten+document+collections%2E+International+Journal+on+Document+Analysis+and+Recognition+%28IJDAR%29%2C+24%283%29%3A235%E2%80%93249%2C+2021%2E)
95. Mathew, M., Karatzas, D., and Jawahar, C. [DocVQA: A dataset for VQA on document images](https://md.research.elamur.ai/free/en/papers/mathew-2020-docvqa-dataset-vqa-document-images/). In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200–2209, 2021.
96. Mathew, M., Bagal, V., Tito, R., Karatzas, D., Valveny, E., and Jawahar, C. InfographicVQA. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1697–1706, 2022. [find](https://scholar.google.com/scholar?q=Mathew%2C+M%2E%2C+Bagal%2C+V%2E%2C+Tito%2C+R%2E%2C+Karatzas%2C+D%2E%2C+Valveny%2C+E%2E%2C+and+Jawahar%2C+C%2E+InfographicVQA%2E+In+Proceedings+of+the+IEEE%2FCVF+Winter+Conference+on+Applications+of+Computer+Vision%2C+pp%2E+1697%E2%80%931706%2C+2022%2E)
97. Methani, N., Ganguly, P., Khapra, M. M., and Kumar, P. Plotqa: Reasoning over scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1527–1536, 2020. [find](https://scholar.google.com/scholar?q=Methani%2C+N%2E%2C+Ganguly%2C+P%2E%2C+Khapra%2C+M%2E+M%2E%2C+and+Kumar%2C+P%2E+Plotqa%3A+Reasoning+over+scientific+plots%2E+In+Proceedings+of+the+IEEE%2FCVF+Winter+Conference+on+Applications+of+Computer+Vision%2C+pp%2E+1527%E2%80%931536%2C+2020)
98. Mishra, A., Shekhar, S., Singh, A. K., and Chakraborty, A. OCR-VQA: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pp. 947–952. IEEE, 2019. [find](https://scholar.google.com/scholar?q=Mishra%2C+A%2E%2C+Shekhar%2C+S%2E%2C+Singh%2C+A%2E+K%2E%2C+and+Chakraborty%2C+A%2E+OCR%2DVQA%3A+Visual+question+answering+by+reading+text+in+images%2E+In+2019+international+conference+on+document+analysis+and+recognition+%28ICDAR%29%2C+)
99. Mitra, A., Khanpour, H., Rosset, C., and Awadallah, A. [Orca-math: Unlocking the potential of slms in grade school math](https://md.research.elamur.ai/free/en/papers/mitra-2024-orca-math-unlocking-potential-slms/). arXiv preprint arXiv:2402.14830, 2024.
100. mlfoundations. An open source implementation of CLIP, 2021. mlfoundations/open_clip. URL [https://github.com/](https://github.com/) [↗](https://github.com/)
101. Mukherjee, S., Mitra, A., Jawahar, G., Agarwal, S., Palangi, H., and Awadallah, A. [Orca: Progressive learning from complex explanation traces of gpt-4](https://md.research.elamur.ai/free/en/papers/mukherjee-2023-orca-progressive-learning-complex-explanation/). arXiv preprint arXiv:2306.02707, 2023.
102. Nous. Nous-Hermes-2-Yi-34B, 2024. [find](https://scholar.google.com/scholar?q=Nous%2E+Nous%2DHermes%2D2%2DYi%2D34B%2C+2024%2E)
103. Obeid, J. and Hoque, E. Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model. arXiv preprint arXiv:2010.09142, 2020. [find](https://scholar.google.com/scholar?q=Obeid%2C+J%2E+and+Hoque%2C+E%2E+Chart%2Dto%2Dtext%3A+Generating+natural+language+descriptions+for+charts+by+adapting+the+transformer+model%2E+arXiv+preprint+arXiv%3A2010%2E09142%2C+2020%2E)
104. OpenAI. Introducing ChatGPT, 2022. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Introducing+ChatGPT%2C+2022%2E)
105. OpenAI. New models and developer products announced at DevDay, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+New+models+and+developer+products+announced+at+DevDay%2C+2023%2E)
106. OpenAI. GPT-4V(ision) System Card, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+GPT%2D4V%28ision%29+System+Card%2C+2023%2E)
107. OpenAI. Hello GPT-4o, 2024. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Hello+GPT%2D4o%2C+2024%2E)
108. OpenAI. GPT-4o mini: advancing cost-efficient intelligence, 2024. [find](https://scholar.google.com/scholar?q=OpenAI%2E+GPT%2D4o+mini%3A+advancing+cost%2Defficient+intelligence%2C+2024%2E)
109. OpenAI. [GPT-4o System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/), 2024.
110. OpenGVLab. InternVL2: Better than the Best – Expanding Performance Boundaries of Open-Source Multimodal Models with the Progressive Scaling Strategy, 2024. [find](https://scholar.google.com/scholar?q=OpenGVLab%2E+InternVL2%3A+Better+than+the+Best+%E2%80%93+Expanding+Performance+Boundaries+of+Open%2DSource+Multimodal+Models+with+the+Progressive+Scaling+Strategy%2C+2024%2E)
111. OpenGVLab. InternVL2-Llama3-76B, 2024. InternVL2-Llama3-76B. URL [https://huggingface.co/OpenGVLab/](https://huggingface.co/OpenGVLab/) [↗](https://huggingface.co/OpenGVLab/)
112. OpenGVLab. InternViT-6B-448px-V1-5, 2024. InternViT-6B-448px-V1-5. URL [https://huggingface.co/OpenGVLab/](https://huggingface.co/OpenGVLab/) [↗](https://huggingface.co/OpenGVLab/)
113. Ordonez, V., Kulkarni, G., and Berg, T. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems, 24, 2011. [find](https://scholar.google.com/scholar?q=Ordonez%2C+V%2E%2C+Kulkarni%2C+G%2E%2C+and+Berg%2C+T%2E+Im2text%3A+Describing+images+using+1+million+captioned+photographs%2E+Advances+in+neural+information+processing+systems%2C+24%2C+2011%2E)
114. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 2022.
115. Pasupat, P. and Liang, P. [Compositional semantic parsing on semi-structured tables](https://md.research.elamur.ai/free/en/papers/pasupat-2015-compositional-semantic-parsing-semi-structured/). arXiv preprint arXiv:1508.00305, 2015.
116. PDF-Association, 2024. URL [https://huggingface.co/datasets/pixparse/pdfa-eng-wds](https://huggingface.co/datasets/pixparse/pdfa-eng-wds). [↗](https://huggingface.co/datasets/pixparse/pdfa-eng-wds)
117. Peng, B., Li, C., He, P., Galley, M., and Gao, J. [Instruction tuning with gpt-4](https://md.research.elamur.ai/free/en/papers/peng-2023-instruction-tuning-gpt-4/). arXiv preprint arXiv:2304.03277, 2023.
118. Qwen-Team. Qwen2 technical report, 2024. [find](https://scholar.google.com/scholar?q=Qwen%2DTeam%2E+Qwen2+technical+report%2C+2024%2E)
119. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In ICML, 2021.
120. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). arXiv preprint arXiv:1606.05250, 2016.
121. Reka. Reka Core: Our Frontier Class Multimodal Language Model, 2024. [find](https://scholar.google.com/scholar?q=Reka%2E+Reka+Core%3A+Our+Frontier+Class+Multimodal+Language+Model%2C+2024%2E)
122. Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. [Laion-400m: Open dataset of clip-filtered 400 million image-text pairs](https://md.research.elamur.ai/free/en/papers/schuhmann-2021-laion-400m-open-dataset-clip/). arXiv preprint arXiv:2111.02114, 2021.
123. Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. [Laion-5B: An open large-scale dataset for training next generation image-text models](https://md.research.elamur.ai/free/en/papers/schuhmann-2022-laion-5b-open-large-scale/). Advances in Neural Information Processing Systems, 35:25278–25294, 2022.
124. Schwenk, D., Khandelwal, A., Clark, C., Marino, K., and Mottaghi, R. [A-OKVQA: A benchmark for visual question answering using world knowledge](https://md.research.elamur.ai/free/en/papers/schwenk-2022-okvqa-benchmark-visual-question-answering/). In European conference on computer vision, pp. 146–162. Springer, 2022.
125. Seo, M., Hajishirzi, H., Farhadi, A., Etzioni, O., and Malcolm, C. [Solving geometry problems: Combining text and diagram interpretation](https://md.research.elamur.ai/free/en/papers/seo-2015-solving-geometry-problems-combining-text/). In Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 1466–1476, 2015.
126. Sharma, P., Ding, N., Goodman, S., and Soricut, R. [Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/). In ACL, 2018.
127. Shi, B., Yao, C., Liao, M., Yang, M., Xu, P., Cui, L., Belongie, S., Lu, S., and Bai, X. Icdar2017 competition on reading chinese text in the wild (rctw-17). In 2017 14th iapr international conference on document analysis and recognition (ICDAR), volume 1, pp. 1429–1434. IEEE, 2017. [find](https://scholar.google.com/scholar?q=Shi%2C+B%2E%2C+Yao%2C+C%2E%2C+Liao%2C+M%2E%2C+Yang%2C+M%2E%2C+Xu%2C+P%2E%2C+Cui%2C+L%2E%2C+Belongie%2C+S%2E%2C+Lu%2C+S%2E%2C+and+Bai%2C+X%2E+Icdar2017+competition+on+reading+chinese+text+in+the+wild+%28rctw%2D17%29%2E+In+2017+14th+iapr+international+conference)
128. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. URL [https://github.com/NVIDIA/Megatron-LM](https://github.com/NVIDIA/Megatron-LM). [↗](https://github.com/NVIDIA/Megatron-LM)
129. Sidorov, O., Hu, R., Rohrbach, M., and Singh, A. Textcaps: a dataset for image captioning with reading comprehension. In ECCV, 2020. [find](https://scholar.google.com/scholar?q=Sidorov%2C+O%2E%2C+Hu%2C+R%2E%2C+Rohrbach%2C+M%2E%2C+and+Singh%2C+A%2E+Textcaps%3A+a+dataset+for+image+captioning+with+reading+comprehension%2E+In+ECCV%2C+2020%2E)
130. Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards VQA models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317–8326, 2019. [find](https://scholar.google.com/scholar?q=Singh%2C+A%2E%2C+Natarajan%2C+V%2E%2C+Shah%2C+M%2E%2C+Jiang%2C+Y%2E%2C+Chen%2C+X%2E%2C+Batra%2C+D%2E%2C+Parikh%2C+D%2E%2C+and+Rohrbach%2C+M%2E+Towards+VQA+models+that+can+read%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+vision+and+patt)
131. Singh, A., Pang, G., Toh, M., Huang, J., Galuba, W., and Hassner, T. TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. In The Conference on Computer Vision and Pattern Recognition, 2021. [find](https://scholar.google.com/scholar?q=Singh%2C+A%2E%2C+Pang%2C+G%2E%2C+Toh%2C+M%2E%2C+Huang%2C+J%2E%2C+Galuba%2C+W%2E%2C+and+Hassner%2C+T%2E+TextOCR%3A+Towards+large%2Dscale+end%2Dto%2Dend+reasoning+for+arbitrary%2Dshaped+scene+text%2E+In+The+Conference+on+Computer+Vision+and+Pattern)
132. Smith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., et al. [Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model](https://md.research.elamur.ai/free/en/papers/smith-2022-using-deepspeed-megatron-train-megatron/). arXiv preprint arXiv:2201.11990, 2022.
133. Sun, Y., Ni, Z., Chng, C.-K., Liu, Y., Luo, C., Ng, C. C., Han, J., Ding, E., Liu, J., Karatzas, D., et al. Icdar 2019 competition on large-scale street view text with partial labeling-rrc-lsvt. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1557–1562. IEEE, 2019. [find](https://scholar.google.com/scholar?q=Sun%2C+Y%2E%2C+Ni%2C+Z%2E%2C+Chng%2C+C%2E%2DK%2E%2C+Liu%2C+Y%2E%2C+Luo%2C+C%2E%2C+Ng%2C+C%2E+C%2E%2C+Han%2C+J%2E%2C+Ding%2C+E%2E%2C+Liu%2C+J%2E%2C+Karatzas%2C+D%2E%2C+et+al%2E+Icdar+2019+competition+on+large%2Dscale+street+view+text+with+partial+labeling%2Drrc%2Dlsvt%2E+In+20)
134. Tanaka, R., Nishida, K., and Yoshida, S. Visualmrc: Machine reading comprehension on document images. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021. [find](https://scholar.google.com/scholar?q=Tanaka%2C+R%2E%2C+Nishida%2C+K%2E%2C+and+Yoshida%2C+S%2E+Visualmrc%3A+Machine+reading+comprehension+on+document+images%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+2021%2E)
135. Tanaka, R., Nishida, K., Nishida, K., Hasegawa, T., Saito, I., and Saito, K. SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images. In AAAI, 2023. [find](https://scholar.google.com/scholar?q=Tanaka%2C+R%2E%2C+Nishida%2C+K%2E%2C+Nishida%2C+K%2E%2C+Hasegawa%2C+T%2E%2C+Saito%2C+I%2E%2C+and+Saito%2C+K%2E+SlideVQA%3A+A+Dataset+for+Document+Visual+Question+Answering+on+Multiple+Images%2E+In+AAAI%2C+2023%2E)
136. Teknium. GPTeacher-General-Instruct, 2023. URL [https://huggingface.co/datasets/teknium/](https://huggingface.co/datasets/teknium/) GPTeacher-General-Instruct. [↗](https://huggingface.co/datasets/teknium/)
137. The-Vicuna-Team. ShareGPT-Vicuna, 2023. URL [https://huggingface.co/datasets/](https://huggingface.co/datasets/) anon8231489123/ShareGPT_Vicuna_unfiltered. [↗](https://huggingface.co/datasets/)
138. Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., Iyer, A., Pan, X., et al. [Cambrian-1: A fully open, vision-centric exploration of multimodal llms](https://md.research.elamur.ai/free/en/papers/tong-2024-cambrian-1-fully-open-vision/). arXiv preprint arXiv:2406.16860, 2024.
139. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
140. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
141. Van Landeghem, J., Tito, R., Borchmann, Ł., Pietruszka, M., Joziak, P., Powalski, R., Jurkiewicz, D., Coustaty, M., Anckaert, B., Valveny, E., et al. Document understanding dataset and evaluation (dude). In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19528–19540, 202… [find](https://scholar.google.com/scholar?q=Van+Landeghem%2C+J%2E%2C+Tito%2C+R%2E%2C+Borchmann%2C+%C5%81%2E%2C+Pietruszka%2C+M%2E%2C+Joziak%2C+P%2E%2C+Powalski%2C+R%2E%2C+Jurkiewicz%2C+D%2E%2C+Coustaty%2C+M%2E%2C+Anckaert%2C+B%2E%2C+Valveny%2C+E%2E%2C+et+al%2E+Document+understanding+dataset+and+evaluation+%28dud)
142. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in Neural Information Processing Systems, 2017.
143. Veit, A., Matera, T., Neumann, L., Matas, J., and Belongie, S. Coco-text: Dataset and benchmark for text detection and recognition in natural images. arXiv preprint arXiv:1601.07140, 2016. — [arXiv:1601.07140](https://arxiv.org/abs/1601.07140) [find](https://scholar.google.com/scholar?q=Veit%2C+A%2E%2C+Matera%2C+T%2E%2C+Neumann%2C+L%2E%2C+Matas%2C+J%2E%2C+and+Belongie%2C+S%2E+Coco%2Dtext%3A+Dataset+and+benchmark+for+text+detection+and+recognition+in+natural+images%2E+arXiv+preprint+arXiv%3A1601%2E07140%2C+2016%2E+%E2%80%94+%5BarXiv%3A16)
144. Wang, J., Yang, Z., Hu, X., Li, L., Lin, K., Gan, Z., Liu, Z., Liu, C., and Wang, L. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100, 2022. [find](https://scholar.google.com/scholar?q=Wang%2C+J%2E%2C+Yang%2C+Z%2E%2C+Hu%2C+X%2E%2C+Li%2C+L%2E%2C+Lin%2C+K%2E%2C+Gan%2C+Z%2E%2C+Liu%2C+Z%2E%2C+Liu%2C+C%2E%2C+and+Wang%2C+L%2E+Git%3A+A+generative+image%2Dto%2Dtext+transformer+for+vision+and+language%2E+arXiv+preprint+arXiv%3A2205%2E14100%2C+2022%2E)
145. Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442, 2022. [find](https://scholar.google.com/scholar?q=Wang%2C+W%2E%2C+Bao%2C+H%2E%2C+Dong%2C+L%2E%2C+Bjorck%2C+J%2E%2C+Peng%2C+Z%2E%2C+Liu%2C+Q%2E%2C+Aggarwal%2C+K%2E%2C+Mohammed%2C+O%2E+K%2E%2C+Singhal%2C+S%2E%2C+Som%2C+S%2E%2C+et+al%2E+Image+as+a+foreign+language%3A+Beit+pretraining+for+all+vision+and+vision%2Dlanguage)
146. Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., Song, X., et al. [Cogvlm: Visual expert for pretrained language models](https://md.research.elamur.ai/free/en/papers/wang-2023-cogvlm-visual-expert-pretrained/). arXiv preprint arXiv:2311.03079, 2023.
147. Wang, X., Liu, Y., Shen, C., Ng, C. C., Luo, C., Jin, L., Chan, C. S., Hengel, A. v. d., and Wang, L. On the general value of evidence, and bilingual scene-text visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10126–10135, 2020. [find](https://scholar.google.com/scholar?q=Wang%2C+X%2E%2C+Liu%2C+Y%2E%2C+Shen%2C+C%2E%2C+Ng%2C+C%2E+C%2E%2C+Luo%2C+C%2E%2C+Jin%2C+L%2E%2C+Chan%2C+C%2E+S%2E%2C+Hengel%2C+A%2E+v%2E+d%2E%2C+and+Wang%2C+L%2E+On+the+general+value+of+evidence%2C+and+bilingual+scene%2Dtext+visual+question+answering%2E+In+Proceedin)
148. Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., and Cao, Y. [SimVLM: Simple visual language model pretraining with weak supervision](https://md.research.elamur.ai/free/en/papers/wang-2021-simvlm-simple-visual-language-model/). In ICLR, 2022.
149. Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). arXiv preprint arXiv:2109.01652, 2021.
150. Wei, Y., Wang, Z., Liu, J., Ding, Y., and Zhang, L. Magicoder: Empowering code generation with oss-instruct. In Forty-first International Conference on Machine Learning, 2024. [find](https://scholar.google.com/scholar?q=Wei%2C+Y%2E%2C+Wang%2C+Z%2E%2C+Liu%2C+J%2E%2C+Ding%2C+Y%2E%2C+and+Zhang%2C+L%2E+Magicoder%3A+Empowering+code+generation+with+oss%2Dinstruct%2E+In+Forty%2Dfirst+International+Conference+on+Machine+Learning%2C+2024%2E)
151. Wendler, C., 2023. URL [https://huggingface.co/datasets/wendlerc/RenderedText](https://huggingface.co/datasets/wendlerc/RenderedText). [↗](https://huggingface.co/datasets/wendlerc/RenderedText)
152. xAI. Grok-1.5 vision preview: Connecting the digital and physical worlds with our first multimodal model, 2024. [find](https://scholar.google.com/scholar?q=xAI%2E+Grok%2D1%2E5+vision+preview%3A+Connecting+the+digital+and+physical+worlds+with+our+first+multimodal+model%2C+2024%2E)
153. xAI. Grok-2 beta release, 2024. [find](https://scholar.google.com/scholar?q=xAI%2E+Grok%2D2+beta+release%2C+2024%2E)
154. Xie, X., Fu, L., Zhang, Z., Wang, Z., and Bai, X. Toward understanding wordart: Corner-guided transformer for scene text recognition. In European conference on computer vision, pp. 303–321. Springer, 2022. [find](https://scholar.google.com/scholar?q=Xie%2C+X%2E%2C+Fu%2C+L%2E%2C+Zhang%2C+Z%2E%2C+Wang%2C+Z%2E%2C+and+Bai%2C+X%2E+Toward+understanding+wordart%3A+Corner%2Dguided+transformer+for+scene+text+recognition%2E+In+European+conference+on+computer+vision%2C+pp%2E+303%E2%80%93321%2E+Springer%2C+)
155. Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., Lin, Q., and Jiang, D. [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/). In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=](https://openreview.net/forum?id=) CfXh93NDgH. [↗](https://openreview.net/forum?id=)
156. Xu, J., Mei, T., Yao, T., and Rui, Y. MSR-VTT: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288–5296, 2016. [find](https://scholar.google.com/scholar?q=Xu%2C+J%2E%2C+Mei%2C+T%2E%2C+Yao%2C+T%2E%2C+and+Rui%2C+Y%2E+MSR%2DVTT%3A+A+large+video+description+dataset+for+bridging+video+and+language%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pp%2E+)
157. Yang, Z., Ping, W., Liu, Z., Korthikanti, V., Nie, W., Huang, D.-A., Fan, L., Yu, Z., Lan, S., Li, B., et al. Re-ViLM: Retrieval-augmented visual language model for zero and few-shot image captioning. arXiv preprint arXiv:2302.04858, 2023. [find](https://scholar.google.com/scholar?q=Yang%2C+Z%2E%2C+Ping%2C+W%2E%2C+Liu%2C+Z%2E%2C+Korthikanti%2C+V%2E%2C+Nie%2C+W%2E%2C+Huang%2C+D%2E%2DA%2E%2C+Fan%2C+L%2E%2C+Yu%2C+Z%2E%2C+Lan%2C+S%2E%2C+Li%2C+B%2E%2C+et+al%2E+Re%2DViLM%3A+Retrieval%2Daugmented+visual+language+model+for+zero+and+few%2Dshot+image+captioning%2E)
158. Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., et al. [Minicpm-v: A gpt-4v level mllm on your phone](https://md.research.elamur.ai/free/en/papers/yao-2024-minicpm-v-gpt-4v/). arXiv preprint arXiv:2408.01800, 2024.
159. Ye, J., Hu, A., Xu, H., Ye, Q., Yan, M., Xu, G., Li, C., Tian, J., Qian, Q., Zhang, J., et al. [Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model](https://md.research.elamur.ai/free/en/papers/ye-2023-ureader-universal-ocr-free-visually/). arXiv preprint arXiv:2310.05126, 2023.
160. Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.[04652, 2024](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/).
161. Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). arXiv preprint arXiv:2309.12284, 2023.
162. Yuan, L., Cui, G., Wang, H., Ding, N., Wang, X., Deng, J., Shan, B., Chen, H., Xie, R., Lin, Y., et al. Advancing llm reasoning generalists with preference trees. arXiv preprint arXiv:2404.02078, 2024. [find](https://scholar.google.com/scholar?q=Yuan%2C+L%2E%2C+Cui%2C+G%2E%2C+Wang%2C+H%2E%2C+Ding%2C+N%2E%2C+Wang%2C+X%2E%2C+Deng%2C+J%2E%2C+Shan%2C+B%2E%2C+Chen%2C+H%2E%2C+Xie%2C+R%2E%2C+Lin%2C+Y%2E%2C+et+al%2E+Advancing+llm+reasoning+generalists+with+preference+trees%2E+arXiv+preprint+arXiv%3A2404%2E02078%2C+2024)
163. Yuan, Y., Liu, X., Dikubab, W., Liu, H., Ji, Z., Wu, Z., and Bai, X. Syntax-aware network for handwritten mathematical expression recognition. arXiv preprint arXiv:2203.01601, 2022. — [arXiv:2203.01601](https://arxiv.org/abs/2203.01601) [find](https://scholar.google.com/scholar?q=Yuan%2C+Y%2E%2C+Liu%2C+X%2E%2C+Dikubab%2C+W%2E%2C+Liu%2C+H%2E%2C+Ji%2C+Z%2E%2C+Wu%2C+Z%2E%2C+and+Bai%2C+X%2E+Syntax%2Daware+network+for+handwritten+mathematical+expression+recognition%2E+arXiv+preprint+arXiv%3A2203%2E01601%2C+2022%2E+%E2%80%94+%5BarXiv%3A2203%2E0160)
164. Yue, X., Qu, X., Zhang, G., Fu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. [Mammoth: Building math generalist models through hybrid instruction tuning](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/). arXiv preprint arXiv:2309.05653, 2023.
165. Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark f… — [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/)
166. Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. [Sigmoid loss for language image pre-training](https://md.research.elamur.ai/free/en/papers/zhai-2023-sigmoid-loss-language-image-pre/). In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11975–11986, 2023.
167. Zhang, C., Gao, F., Jia, B., Zhu, Y., and Zhu, S.-C. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5317–5327, 2019. [find](https://scholar.google.com/scholar?q=Zhang%2C+C%2E%2C+Gao%2C+F%2E%2C+Jia%2C+B%2E%2C+Zhu%2C+Y%2E%2C+and+Zhu%2C+S%2E%2DC%2E+Raven%3A+A+dataset+for+relational+and+analogical+visual+reasoning%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+vision+and+pattern+recogniti)
168. Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., and Gao, J. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5579–5588, 2021. [find](https://scholar.google.com/scholar?q=Zhang%2C+P%2E%2C+Li%2C+X%2E%2C+Hu%2C+X%2E%2C+Yang%2C+J%2E%2C+Zhang%2C+L%2E%2C+Wang%2C+L%2E%2C+Choi%2C+Y%2E%2C+and+Gao%2C+J%2E+Vinvl%3A+Revisiting+visual+representations+in+vision%2Dlanguage+models%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Compute)
169. Zhang, R., Zhou, Y., Jiang, Q., Song, Q., Li, N., Zhou, K., Wang, L., Wang, D., Liao, M., Yang, M., et al. Icdar 2019 robust reading challenge on reading chinese text on signboard. In 2019 international conference on document analysis and recognition (ICDAR), pp. 1577–1581. IEEE, 2019. [find](https://scholar.google.com/scholar?q=Zhang%2C+R%2E%2C+Zhou%2C+Y%2E%2C+Jiang%2C+Q%2E%2C+Song%2C+Q%2E%2C+Li%2C+N%2E%2C+Zhou%2C+K%2E%2C+Wang%2C+L%2E%2C+Wang%2C+D%2E%2C+Liao%2C+M%2E%2C+Yang%2C+M%2E%2C+et+al%2E+Icdar+2019+robust+reading+challenge+on+reading+chinese+text+on+signboard%2E+In+2019+internation)
170. Zhang, R., Wei, X., Jiang, D., Zhang, Y., Guo, Z., Tong, C., Liu, J., Zhou, A., Wei, B., Zhang, S., et al. Mavis: Mathematical visual instruction tuning. arXiv preprint arXiv:2407.08739, 2024. — [arXiv:2407.08739](https://arxiv.org/abs/2407.08739) [find](https://scholar.google.com/scholar?q=Zhang%2C+R%2E%2C+Wei%2C+X%2E%2C+Jiang%2C+D%2E%2C+Zhang%2C+Y%2E%2C+Guo%2C+Z%2E%2C+Tong%2C+C%2E%2C+Liu%2C+J%2E%2C+Zhou%2C+A%2E%2C+Wei%2C+B%2E%2C+Zhang%2C+S%2E%2C+et+al%2E+Mavis%3A+Mathematical+visual+instruction+tuning%2E+arXiv+preprint+arXiv%3A2407%2E08739%2C+2024%2E+%E2%80%94+%5BarXi)
171. Zhang, Y., Zhang, R., Gu, J., Zhou, Y., Lipka, N., Yang, D., and Sun, T. LLaVAR: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107, 2023. [find](https://scholar.google.com/scholar?q=Zhang%2C+Y%2E%2C+Zhang%2C+R%2E%2C+Gu%2C+J%2E%2C+Zhou%2C+Y%2E%2C+Lipka%2C+N%2E%2C+Yang%2C+D%2E%2C+and+Sun%2C+T%2E+LLaVAR%3A+Enhanced+visual+instruction+tuning+for+text%2Drich+image+understanding%2E+arXiv+preprint+arXiv%3A2306%2E17107%2C+2023%2E)
172. Zhao, Y., Zhao, C., Nan, L., Qi, Z., Zhang, W., Tang, X., Mi, B., and Radev, D. RobuT: A systematic study of table QA robustness against human-annotated adversarial perturbations. arXiv preprint arXiv:2306.14321, 2023. [find](https://scholar.google.com/scholar?q=Zhao%2C+Y%2E%2C+Zhao%2C+C%2E%2C+Nan%2C+L%2E%2C+Qi%2C+Z%2E%2C+Zhang%2C+W%2E%2C+Tang%2C+X%2E%2C+Mi%2C+B%2E%2C+and+Radev%2C+D%2E+RobuT%3A+A+systematic+study+of+table+QA+robustness+against+human%2Dannotated+adversarial+perturbations%2E+arXiv+preprint+arXiv)
173. Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. [MiniGPT-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/). arXiv preprint arXiv:2304.10592, 2023.
174. Zhu, F., Lei, W., Feng, F., Wang, C., Zhang, H., and Chua, T.-S. Towards complex document understanding by discrete reasoning. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 4857–4866, 2022. [find](https://scholar.google.com/scholar?q=Zhu%2C+F%2E%2C+Lei%2C+W%2E%2C+Feng%2C+F%2E%2C+Wang%2C+C%2E%2C+Zhang%2C+H%2E%2C+and+Chua%2C+T%2E%2DS%2E+Towards+complex+document+understanding+by+discrete+reasoning%2E+In+Proceedings+of+the+30th+ACM+International+Conference+on+Multimedia%2C+pp)
175. Zhu, W., Hessel, J., Awadalla, A., Gadre, S. Y., Dodge, J., Fang, A., Yu, Y., Schmidt, L., Wang, W. Y., and Choi, Y. Multimodal C4: An open, billion-scale corpus of images interleaved with text. arXiv preprint arXiv:2304.06939, 2023. — [arXiv:2304.06939](https://arxiv.org/abs/2304.06939) [find](https://scholar.google.com/scholar?q=Zhu%2C+W%2E%2C+Hessel%2C+J%2E%2C+Awadalla%2C+A%2E%2C+Gadre%2C+S%2E+Y%2E%2C+Dodge%2C+J%2E%2C+Fang%2C+A%2E%2C+Yu%2C+Y%2E%2C+Schmidt%2C+L%2E%2C+Wang%2C+W%2E+Y%2E%2C+and+Choi%2C+Y%2E+Multimodal+C4%3A+An+open%2C+billion%2Dscale+corpus+of+images+interleaved+with+text%2E+arXiv)
176. Zhu, Y., Groth, O., Bernstein, M., and Fei-Fei, L. [Visual7W: Grounded question answering in images](https://md.research.elamur.ai/free/en/papers/zhu-2015-visual7w-grounded-question-answering-images/). In CVPR, 2016. *Figure 6: Demonstration of NVLM-1.0-D 72B model’s strong scene understanding capability. It has common-* sense knowledge to recognize potential dangers or accidents and correctly sugge…

*Updated 30 October 2026*