---
title: "DeepSeek-VL: Towards Real-World Vision-Language — Lu 2024"
description: "DeepSeek-VL is an open-source vision-language model from DeepSeek-AI."
authors: ["Haoyu Lu", "Wen Liu", "Bo Zhang", "Bingxuan Wang", "Kai Dong", "Bo Liu", "Jingxiang Sun", "Tongzheng Ren", "Zhuoshu Li", "Hao Yang", "Yaofeng Sun", "Chengqi Deng", "Hanwei Xu", "Zhenda Xie", "Chong Ruan"]
year: 2024
venue: "arXiv:2403.05525"
source: "https://arxiv.org/abs/2403.05525"
doi: "10.48550/arxiv.2403.05525"
arxiv: "2403.05525"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lu-2024-deepseek-vl-real-world/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lu-2024-deepseek-vl-real-world/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/) / DeepSeek-VL: Towards Real-World Vision-Language Understanding

# DeepSeek-VL: Towards Real-World Vision-Language Understanding

**Authors:** Haoyu Lu, [Wen Liu](https://md.research.elamur.ai/free/en/authors/wen-liu/), [Bo Zhang](https://md.research.elamur.ai/free/en/authors/bo-zhang/), [Bingxuan Wang](https://md.research.elamur.ai/free/en/authors/bingxuan-wang/), Kai Dong, [Bo Liu](https://md.research.elamur.ai/free/en/authors/bo-liu/), Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, [Hao Yang](https://md.research.elamur.ai/free/en/authors/hao-yang/), Yaofeng Sun, [Chengqi Deng](https://md.research.elamur.ai/free/en/authors/chengqi-deng/) et al. (15 authors) · **Year:** 2024 · **Venue:** arXiv:2403.05525 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2403.05525) · [PDF](https://arxiv.org/pdf/2403.05525) · [DOI 10.48550/arxiv.2403.05525](https://doi.org/10.48550/arxiv.2403.05525) · [arXiv 2403.05525](https://arxiv.org/abs/2403.05525) · [OpenAlex](https://openalex.org/W4392678661)

**Citations:** 50 citations (openalex-snapshot · 2026-10-14) · [151](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
DeepSeek-VL is an open-source vision-language model from DeepSeek-AI. It targets a known problem: open models score well on academic benchmarks but often fall short of proprietary systems like GPT-4V in practical use. The authors attribute this to three causes: too little multimodal pretraining, instruction data built mostly from academic datasets, and low input resolution. To address them, they collected broad pretraining data (web screenshots, PDFs, OCR, charts, textbooks and expert knowledge). They also built supervised fine-tuning data around a taxonomy of real user requests. The model pairs a DeepSeek LLM with a hybrid vision encoder that reads 1024×1024 images while keeping the number of visual tokens fixed. Training has three stages: an adaptor-only stage, joint vision-language pretraining, and fine-tuning. Text data stays dominant throughout so the model keeps its language skills. Released at 1.3B and 7B parameters, DeepSeek-VL matches or beats similar-sized models on many multimodal benchmarks and still performs well on text-only ones. This makes it a useful open base for multimodal research.

## Key points
- **Hybrid vision encoder:** SigLIP handles low-resolution semantics and SAM handles high-resolution detail (1024×1024 input). In ablations, adding a vision-only self-supervised encoder (DINO or SAM) to SigLIP lowered training loss compared with CLIP or SigLIP alone.
- **Modality warm-up:** pretraining starts with language data only, then gradually shifts toward a final mix of about 70% text and 30% multimodal data. This avoids an early drop in language ability.
- **Adaptor design:** merging the two encoders' features along the embedding dimension worked better than token pooling along the sequence. A hybrid of shared and separate MLPs gave the most stable results (Table 10).
- **Evaluation:** benchmarks include MMBench, SEED, POPE, ScienceQA, MMMU and OCRBench, plus language benchmarks and a human evaluation.
- **Release and plans:** the 1.3B and 7B (6.7B) models are public at github.com/deepseek-ai/DeepSeek-VL. The authors plan larger Mixture of Experts (MoE) versions.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

151 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 139 not yet.

- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://md.research.elamur.ai/free/en/papers/chen-2024-expanding-performance-boundaries-open/)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://md.research.elamur.ai/free/en/papers/chen-2024-far-gpt-4v-closing/)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://md.research.elamur.ai/free/en/papers/chen-2024-right-way-evaluating-large-vision/)
- [Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models](https://md.research.elamur.ai/free/en/papers/deitke-2024-molmo-pixmo-open-weights-open/)
- [What matters when building vision-language models?](https://md.research.elamur.ai/free/en/papers/lauren-2024-ccon-matters-building-vision-language-models/)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://md.research.elamur.ai/free/en/papers/li-2024-llava-onevision-easy-visual-task/)
- [METAL: A Multi-Agent Framework for Chart Generation with Test-Time Scaling](https://md.research.elamur.ai/free/en/papers/li-2025-metal-multi-agent-framework-chart/)
- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://md.research.elamur.ai/free/en/papers/liu-2023-ocrbench-hidden-mystery-ocr-large/)
- [ViX-Ray: A Vietnamese Chest X-Ray Dataset for Vision-Language Models](https://md.research.elamur.ai/free/en/papers/nguyen-2026-vix-ray-vietnamese-chest-x/)
- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://md.research.elamur.ai/free/en/papers/tong-2024-cambrian-1-fully-open-vision/)
- [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://md.research.elamur.ai/free/en/papers/wu-2024-deepseek-vl2-mixture-experts/)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://md.research.elamur.ai/free/en/papers/yao-2024-minicpm-v-gpt-4v/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{lu2024deepseek,
  title = {{DeepSeek-VL: Towards Real-World Vision-Language Understanding}},
  author = {Lu, Haoyu and Liu, Wen and Zhang, Bo and Wang, Bingxuan and Dong, Kai and Liu, Bo and Sun, Jingxiang and Ren, Tongzheng and Li, Zhuoshu and Yang, Hao and Sun, Yaofeng and Deng, Chengqi and Xu, Hanwei and Xie, Zhenda and Ruan, Chong},
  year = {2024},
  doi = {10.48550/arxiv.2403.05525},
  eprint = {2403.05525},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.05525}
}
```

- [← previous DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://md.research.elamur.ai/free/en/papers/wu-2024-deepseek-vl2-mixture-experts/)
- [next → DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/)

## References

88 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 01-ai. Yi-34B vision language model. [https://huggingface.co/01-ai/Yi-VL-34B](https://huggingface.co/01-ai/Yi-VL-34B) , 2024. [↗](https://huggingface.co/01-ai/Yi-VL-34B)
2. Abi. Screenshot to code. [https://github.com/abi/screenshot-to-code](https://github.com/abi/screenshot-to-code) , 2024. [↗](https://github.com/abi/screenshot-to-code)
3. Anna’s Archive. Anna’s archive. [https://annas-archive.org/](https://annas-archive.org/) , 2024. [↗](https://annas-archive.org/)
4. Anthropic. Introducing Claude, 2023. URL [https://www.anthropic.com/index/introd](https://www.anthropic.com/index/introd) ucing-claude . [↗](https://www.anthropic.com/index/introd)
5. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. [find](https://scholar.google.com/scholar?q=J%2E+Austin%2C+A%2E+Odena%2C+M%2E+Nye%2C+M%2E+Bosma%2C+H%2E+Michalewski%2C+D%2E+Dohan%2C+E%2E+Jiang%2C+C%2E+Cai%2C+M%2E+Terry%2C+Q%2E+Le%2C+et+al%2E+Program+synthesis+with+large+language+models%2E+arXiv+preprint+arXiv%3A2108%2E07732%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/)
6. J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Bai%2C+S%2E+Bai%2C+S%2E+Yang%2C+S%2E+Wang%2C+S%2E+Tan%2C+P%2E+Wang%2C+J%2E+Lin%2C+C%2E+Zhou%2C+and+J%2E+Zhou%2E+Qwen%2Dvl%3A+A+versatile+vision%2Dlanguage+model+for+understanding%2C+localization%2C+text+reading%2C+and+beyond%2E+arXiv+preprint+ar) [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/)
7. R. Bavishi, E. Elsen, C. Hawthorne, M. Nye, A. Odena, A. Somani, and S. Taşırlar. Introducing our multimodal models, 2023. URL [https://www.adept.ai/blog/fuyu-8b](https://www.adept.ai/blog/fuyu-8b) . [↗](https://www.adept.ai/blog/fuyu-8b)
8. L. Blecher. Latex-ocr. GitHub repository, 2024. URL [https://github.com/lukas-blecher/LaTeX-OCR](https://github.com/lukas-blecher/LaTeX-OCR) . [↗](https://github.com/lukas-blecher/LaTeX-OCR)
9. L. Blecher, G. Cucurull, T. Scialom, and R. Stojnic. Nougat: Neural optical understanding for academic documents. arXiv preprint arXiv:2308.13418, 2023. [find](https://scholar.google.com/scholar?q=L%2E+Blecher%2C+G%2E+Cucurull%2C+T%2E+Scialom%2C+and+R%2E+Stojnic%2E+Nougat%3A+Neural+optical+understanding+for+academic+documents%2E+arXiv+preprint+arXiv%3A2308%2E13418%2C+2023%2E)
10. A. Burns, K. Srinivasan, J. Ainslie, G. Brown, B. A. Plummer, K. Saenko, J. Ni, and M. Guo. A suite of generative tasks for multi-level multimodal webpage understanding. In The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. URL [https://openreview.net/forum?id=rwcL…](https://openreview.net/forum?id=rwcL…) [↗](https://openreview.net/forum?id=rwcL…)
11. J. Carter. Textocr-gpt4v. [https://huggingface.co/datasets/jimmycarter/textocr-g](https://huggingface.co/datasets/jimmycarter/textocr-g) pt4v , 2024. [↗](https://huggingface.co/datasets/jimmycarter/textocr-g)
12. L. Chen, J. Li, X. Dong, P. Zhang, C. He, J. Wang, F. Zhao, and D. Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. [find](https://scholar.google.com/scholar?q=L%2E+Chen%2C+J%2E+Li%2C+X%2E+Dong%2C+P%2E+Zhang%2C+C%2E+He%2C+J%2E+Wang%2C+F%2E+Zhao%2C+and+D%2E+Lin%2E+Sharegpt4v%3A+Improving+large+multi%2Dmodal+models+with+better+captions%2E+arXiv+preprint+arXiv%3A2311%2E12793%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2023-sharegpt4v-improving-large-multi-modal/)
13. C. K. Chng, Y. Liu, Y. Sun, C. C. Ng, C. Luo, Z. Ni, C. Fang, S. Zhang, J. Han, E. Ding, et al. Icdar2019 robust reading challenge on arbitrary-shaped text-rrc-art. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1571–1576. [find](https://scholar.google.com/scholar?q=C%2E+K%2E+Chng%2C+Y%2E+Liu%2C+Y%2E+Sun%2C+C%2E+C%2E+Ng%2C+C%2E+Luo%2C+Z%2E+Ni%2C+C%2E+Fang%2C+S%2E+Zhang%2C+J%2E+Han%2C+E%2E+Ding%2C+et+al%2E+Icdar2019+robust+reading+challenge+on+arbitrary%2Dshaped+text%2Drrc%2Dart%2E+In+2019+International+Conference+on)
14. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [find](https://scholar.google.com/scholar?q=K%2E+Cobbe%2C+V%2E+Kosaraju%2C+M%2E+Bavarian%2C+M%2E+Chen%2C+H%2E+Jun%2C+L%2E+Kaiser%2C+M%2E+Plappert%2C+J%2E+Tworek%2C+J%2E+Hilton%2C+R%2E+Nakano%2C+et+al%2E+Training+verifiers+to+solve+math+word+problems%2E+arXiv+preprint+arXiv%3A2110%2E14168%2C+20) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
15. W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023. [find](https://scholar.google.com/scholar?q=W%2E+Dai%2C+J%2E+Li%2C+D%2E+Li%2C+A%2E+M%2E+H%2E+Tiong%2C+J%2E+Zhao%2C+W%2E+Wang%2C+B%2E+Li%2C+P%2E+Fung%2C+and+S%2E+Hoi%2E+Instructblip%3A+Towards+general%2Dpurpose+vision%2Dlanguage+models+with+instruction+tuning%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/)
16. DeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. URL [https://github.com/deepseek-ai/DeepSeek-L](https://github.com/deepseek-ai/DeepSeek-L) LM . [↗](https://github.com/deepseek-ai/DeepSeek-L)
17. X. Dong, P. Zhang, Y. Zang, Y. Cao, B. Wang, L. Ouyang, X. Wei, S. Zhang, H. Duan, M. Cao, et al. Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401.16420, 2024. echo840. Detailed caption dataset. https://huggin… [find](https://scholar.google.com/scholar?q=X%2E+Dong%2C+P%2E+Zhang%2C+Y%2E+Zang%2C+Y%2E+Cao%2C+B%2E+Wang%2C+L%2E+Ouyang%2C+X%2E+Wei%2C+S%2E+Zhang%2C+H%2E+Duan%2C+M%2E+Cao%2C+et+al%2E+Internlm%2Dxcomposer2%3A+Mastering+free%2Dform+text%2Dimage+composition+and+comprehension+in+vision%2Dlanguage+l) [our card](https://md.research.elamur.ai/free/en/papers/dong-2024-internlm-xcomposer2-mastering-free/)
18. W. Foundation. Wikimedia downloads. URL [https://dumps.wikimedia.org](https://dumps.wikimedia.org) . J. Gao, R. Pi, J. Zhang, J. Ye, W. Zhong, Y. Wang, L. Hong, J. Han, H. Xu, Z. Li, et al. Gllava: Solving geometric problem with multi-modal large language model. arXiv preprint arXiv:2312.11370, 2023. [↗](https://dumps.wikimedia.org)
19. L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. [find](https://scholar.google.com/scholar?q=L%2E+Gao%2C+S%2E+Biderman%2C+S%2E+Black%2C+L%2E+Golding%2C+T%2E+Hoppe%2C+C%2E+Foster%2C+J%2E+Phang%2C+H%2E+He%2C+A%2E+Thite%2C+N%2E+Nabeshima%2C+et+al%2E+The+Pile%3A+An+800GB+dataset+of+diverse+text+for+language+modeling%2E+arXiv+preprint+arXiv%3A2)
20. Google. An important next step on our AI journey, 2023. URL [https://blog.google/tech](https://blog.google/tech) nology/ai/bard-google-ai-search-updates/ . [↗](https://blog.google/tech)
21. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. [find](https://scholar.google.com/scholar?q=D%2E+Hendrycks%2C+C%2E+Burns%2C+S%2E+Basart%2C+A%2E+Zou%2C+M%2E+Mazeika%2C+D%2E+Song%2C+and+J%2E+Steinhardt%2E+Measuring+massive+multitask+language+understanding%2E+arXiv+preprint+arXiv%3A2009%2E03300%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
22. High-flyer. Hai-llm: 高效且轻量的大模型训练工具, 2023. URL [https://www.high-flyer.c](https://www.high-flyer.c) n/en/blog/hai-llm . [↗](https://www.high-flyer.c)
23. Y.-C. Hsiao, F. Zubach, M. Wang, et al. Screenqa: Large-scale question-answer pairs over mobile app screenshots. arXiv preprint arXiv:2209.08199, 2022. — [arXiv:2209.08199](https://arxiv.org/abs/2209.08199) [find](https://scholar.google.com/scholar?q=Y%2E%2DC%2E+Hsiao%2C+F%2E+Zubach%2C+M%2E+Wang%2C+et+al%2E+Screenqa%3A+Large%2Dscale+question%2Danswer+pairs+over+mobile+app+screenshots%2E+arXiv+preprint+arXiv%3A2209%2E08199%2C+2022%2E+%E2%80%94+%5BarXiv%3A2209%2E08199%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2209%2E)
24. A. Hu, Y. Shi, H. Xu, J. Ye, Q. Ye, M. Yan, C. Li, Q. Qian, J. Zhang, and F. Huang. mplug-paperowl: Scientific diagram analysis with the multimodal large language model. arXiv preprint arXiv:2311.18248, 2023. — [arXiv:2311.18248](https://arxiv.org/abs/2311.18248) [find](https://scholar.google.com/scholar?q=A%2E+Hu%2C+Y%2E+Shi%2C+H%2E+Xu%2C+J%2E+Ye%2C+Q%2E+Ye%2C+M%2E+Yan%2C+C%2E+Li%2C+Q%2E+Qian%2C+J%2E+Zhang%2C+and+F%2E+Huang%2E+mplug%2Dpaperowl%3A+Scientific+diagram+analysis+with+the+multimodal+large+language+model%2E+arXiv+preprint+arXiv%3A2311%2E1824)
25. HuggingFaceM4. Websight dataset. [https://huggingface.co/datasets/HuggingFaceM](https://huggingface.co/datasets/HuggingFaceM) 4/WebSight , 2024. [↗](https://huggingface.co/datasets/HuggingFaceM)
26. S. Kantharaj, R. T. Leong, X. Lin, A. Masry, M. Thakkar, E. Hoque, and S. Joty. Chart-to-text: A large-scale benchmark for chart summarization. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: L… [find](https://scholar.google.com/scholar?q=S%2E+Kantharaj%2C+R%2E+T%2E+Leong%2C+X%2E+Lin%2C+A%2E+Masry%2C+M%2E+Thakkar%2C+E%2E+Hoque%2C+and+S%2E+Joty%2E+Chart%2Dto%2Dtext%3A+A+large%2Dscale+benchmark+for+chart+summarization%2E+In+S%2E+Muresan%2C+P%2E+Nakov%2C+and+A%2E+Villavicencio%2C+editors%2C+)
27. A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Kirillov%2C+E%2E+Mintun%2C+N%2E+Ravi%2C+H%2E+Mao%2C+C%2E+Rolland%2C+L%2E+Gustafson%2C+T%2E+Xiao%2C+S%2E+Whitehead%2C+A%2E+C%2E+Berg%2C+W%2E%2DY%2E+Lo%2C+et+al%2E+Segment+anything%2E+arXiv+preprint+arXiv%3A2304%2E02643%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/kirillov-2023-segment-anything/)
28. D. Kocetkov, R. Li, L. B. Allal, J. Li, C. Mou, C. M. Ferrandis, Y. Jernite, M. Mitchell, S. Hughes, T. Wolf, D. Bahdanau, L. von Werra, and H. de Vries. The stack: 3 tb of permissively licensed source code. In Transactions on Machine Learning Research, 2023. [find](https://scholar.google.com/scholar?q=D%2E+Kocetkov%2C+R%2E+Li%2C+L%2E+B%2E+Allal%2C+J%2E+Li%2C+C%2E+Mou%2C+C%2E+M%2E+Ferrandis%2C+Y%2E+Jernite%2C+M%2E+Mitchell%2C+S%2E+Hughes%2C+T%2E+Wolf%2C+D%2E+Bahdanau%2C+L%2E+von+Werra%2C+and+H%2E+de+Vries%2E+The+stack%3A+3+tb+of+permissively+licensed+sourc)
29. V. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023. [find](https://scholar.google.com/scholar?q=V%2E+A%2E+Korthikanti%2C+J%2E+Casper%2C+S%2E+Lym%2C+L%2E+McAfee%2C+M%2E+Andersch%2C+M%2E+Shoeybi%2C+and+B%2E+Catanzaro%2E+Reducing+activation+recomputation+in+large+transformer+models%2E+Proceedings+of+Machine+Learning+and+Systems%2C+) [our card](https://md.research.elamur.ai/free/en/papers/korthikanti-2022-reducing-activation-recomputation-large-transformer/)
30. I. Krylov, S. Nosov, and V. Sovrasov. Open images v5 text annotation and yet another mask text spotter. In Asian Conference on Machine Learning, pages 379–389. PMLR, 2021. [find](https://scholar.google.com/scholar?q=I%2E+Krylov%2C+S%2E+Nosov%2C+and+V%2E+Sovrasov%2E+Open+images+v5+text+annotation+and+yet+another+mask+text+spotter%2E+In+Asian+Conference+on+Machine+Learning%2C+pages+379%E2%80%93389%2E+PMLR%2C+2021%2E)
31. A. Kulkarni and J. Truelsen. wkhtmltopdf. [https://wkhtmltopdf.org/](https://wkhtmltopdf.org/) . Project maintained by Ashish Kulkarni, originally created by Jakob Truelsen. Accessed: 2024-02-22. [↗](https://wkhtmltopdf.org/)
32. LAION. Gpt-4v dataset. [https://huggingface.co/datasets/laion/gpt4v-dataset](https://huggingface.co/datasets/laion/gpt4v-dataset) , 2023. [↗](https://huggingface.co/datasets/laion/gpt4v-dataset)
33. B. Li, R. Wang, G. Wang, Y. Ge, Y. Ge, and Y. Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023a. [find](https://scholar.google.com/scholar?q=B%2E+Li%2C+R%2E+Wang%2C+G%2E+Wang%2C+Y%2E+Ge%2C+Y%2E+Ge%2C+and+Y%2E+Shan%2E+Seed%2Dbench%3A+Benchmarking+multimodal+llms+with+generative+comprehension%2E+arXiv+preprint+arXiv%3A2307%2E16125%2C+2023a%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-seed-bench-benchmarking-multimodal-llms/)
34. S. Li and N. Tajbakhsh. Scigraphqa: A large-scale synthetic multi-turn question-answering dataset for scientific graphs, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Li+and+N%2E+Tajbakhsh%2E+Scigraphqa%3A+A+large%2Dscale+synthetic+multi%2Dturn+question%2Danswering+dataset+for+scientific+graphs%2C+2023%2E)
35. Y. Li, G. Li, L. He, J. Zheng, H. Li, and Z. Guan. Widget captioning: Generating natural language description for mobile user interface elements. arXiv preprint arXiv:2010.04295, 2020. [find](https://scholar.google.com/scholar?q=Y%2E+Li%2C+G%2E+Li%2C+L%2E+He%2C+J%2E+Zheng%2C+H%2E+Li%2C+and+Z%2E+Guan%2E+Widget+captioning%3A+Generating+natural+language+description+for+mobile+user+interface+elements%2E+arXiv+preprint+arXiv%3A2010%2E04295%2C+2020%2E)
36. Y. Li, H. Mao, R. Girshick, and K. He. Exploring plain vision transformer backbones for object detection. In European Conference on Computer Vision, pages 280–296. Springer, 2022. [find](https://scholar.google.com/scholar?q=Y%2E+Li%2C+H%2E+Mao%2C+R%2E+Girshick%2C+and+K%2E+He%2E+Exploring+plain+vision+transformer+backbones+for+object+detection%2E+In+European+Conference+on+Computer+Vision%2C+pages+280%E2%80%93296%2E+Springer%2C+2022%2E)
37. Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023b. [find](https://scholar.google.com/scholar?q=Y%2E+Li%2C+Y%2E+Du%2C+K%2E+Zhou%2C+J%2E+Wang%2C+W%2E+X%2E+Zhao%2C+and+J%2E%2DR%2E+Wen%2E+Evaluating+object+hallucination+in+large+vision%2Dlanguage+models%2E+arXiv+preprint+arXiv%3A2305%2E10355%2C+2023b%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-evaluating-object-hallucination-large-vision/)
38. J. Lin, H. Yin, W. Ping, Y. Lu, P. Molchanov, A. Tao, H. Mao, J. Kautz, M. Shoeybi, and S. Han. Vila: On pre-training for visual language models. arXiv preprint arXiv:2312.07533, 2023a. [find](https://scholar.google.com/scholar?q=J%2E+Lin%2C+H%2E+Yin%2C+W%2E+Ping%2C+Y%2E+Lu%2C+P%2E+Molchanov%2C+A%2E+Tao%2C+H%2E+Mao%2C+J%2E+Kautz%2C+M%2E+Shoeybi%2C+and+S%2E+Han%2E+Vila%3A+On+pre%2Dtraining+for+visual+language+models%2E+arXiv+preprint+arXiv%3A2312%2E07533%2C+2023a%2E) [our card](https://md.research.elamur.ai/free/en/papers/lin-2023-vila-pre-training-visual-language/)
39. Z. Lin, C. Liu, R. Zhang, P. Gao, L. Qiu, H. Xiao, H. Qiu, C. Lin, W. Shao, K. Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575, 2023b. — [arXiv:2311.07575](https://arxiv.org/abs/2311.07575) [find](https://scholar.google.com/scholar?q=Z%2E+Lin%2C+C%2E+Liu%2C+R%2E+Zhang%2C+P%2E+Gao%2C+L%2E+Qiu%2C+H%2E+Xiao%2C+H%2E+Qiu%2C+C%2E+Lin%2C+W%2E+Shao%2C+K%2E+Chen%2C+et+al%2E+Sphinx%3A+The+joint+mixing+of+weights%2C+tasks%2C+and+visual+embeddings+for+multi%2Dmodal+large+language+models%2E+arX)
40. F. Liu, F. Piccinno, S. Krichene, C. Pang, K. Lee, M. Joshi, Y. Altun, N. Collier, and J. M. Eisenschlos. Matcha: Enhancing visual language pretraining with math reasoning and chart derendering. arXiv preprint arXiv:2212.09662, 2022a. [find](https://scholar.google.com/scholar?q=F%2E+Liu%2C+F%2E+Piccinno%2C+S%2E+Krichene%2C+C%2E+Pang%2C+K%2E+Lee%2C+M%2E+Joshi%2C+Y%2E+Altun%2C+N%2E+Collier%2C+and+J%2E+M%2E+Eisenschlos%2E+Matcha%3A+Enhancing+visual+language+pretraining+with+math+reasoning+and+chart+derendering%2E+arXiv)
41. H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL [https://llava-vl.github.io/blog/202](https://llava-vl.github.io/blog/202) 4-01-30-llava-next/ . [↗](https://llava-vl.github.io/blog/202)
42. H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b. [find](https://scholar.google.com/scholar?q=H%2E+Liu%2C+C%2E+Li%2C+Q%2E+Wu%2C+and+Y%2E+J%2E+Lee%2E+Visual+instruction+tuning%2E+Advances+in+neural+information+processing+systems%2C+36%2C+2024b%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/)
43. Y. Liu, G. Zhu, B. Zhu, Q. Song, G. Ge, H. Chen, G. Qiao, R. Peng, L. Wu, and J. Wang. Taisu: A 166m large-scale high-quality dataset for chinese vision-language pre-training. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Sys… [find](https://scholar.google.com/scholar?q=Y%2E+Liu%2C+G%2E+Zhu%2C+B%2E+Zhu%2C+Q%2E+Song%2C+G%2E+Ge%2C+H%2E+Chen%2C+G%2E+Qiao%2C+R%2E+Peng%2C+L%2E+Wu%2C+and+J%2E+Wang%2E+Taisu%3A+A+166m+large%2Dscale+high%2Dquality+dataset+for+chinese+vision%2Dlanguage+pre%2Dtraining%2E+In+S%2E+Koyejo%2C+S%2E+Mohamed)
44. Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023a. [find](https://scholar.google.com/scholar?q=Y%2E+Liu%2C+H%2E+Duan%2C+Y%2E+Zhang%2C+B%2E+Li%2C+S%2E+Zhang%2C+W%2E+Zhao%2C+Y%2E+Yuan%2C+J%2E+Wang%2C+C%2E+He%2C+Z%2E+Liu%2C+et+al%2E+Mmbench%3A+Is+your+multi%2Dmodal+model+an+all%2Daround+player%3F+arXiv+preprint+arXiv%3A2307%2E06281%2C+2023a%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/)
45. Y. Liu, Z. Li, H. Li, W. Yu, M. Huang, D. Peng, M. Liu, M. Chen, C. Li, L. Jin, et al. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07895, 2023b. [find](https://scholar.google.com/scholar?q=Y%2E+Liu%2C+Z%2E+Li%2C+H%2E+Li%2C+W%2E+Yu%2C+M%2E+Huang%2C+D%2E+Peng%2C+M%2E+Liu%2C+M%2E+Chen%2C+C%2E+Li%2C+L%2E+Jin%2C+et+al%2E+On+the+hidden+mystery+of+ocr+in+large+multimodal+models%2E+arXiv+preprint+arXiv%3A2305%2E07895%2C+2023b%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-ocrbench-hidden-mystery-ocr-large/)
46. S. Long, S. Qin, D. Panteleev, A. Bissacco, Y. Fujii, and M. Raptis. Towards end-to-end unified scene text detection and layout analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. [find](https://scholar.google.com/scholar?q=S%2E+Long%2C+S%2E+Qin%2C+D%2E+Panteleev%2C+A%2E+Bissacco%2C+Y%2E+Fujii%2C+and+M%2E+Raptis%2E+Towards+end%2Dto%2Dend+unified+scene+text+detection+and+layout+analysis%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+a)
47. P. Lu, L. Qiu, J. Chen, T. Xia, Y. Zhao, W. Zhang, Z. Yu, X. Liang, and S.-C. Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2110.13214, 2021. [find](https://scholar.google.com/scholar?q=P%2E+Lu%2C+L%2E+Qiu%2C+J%2E+Chen%2C+T%2E+Xia%2C+Y%2E+Zhao%2C+W%2E+Zhang%2C+Z%2E+Yu%2C+X%2E+Liang%2C+and+S%2E%2DC%2E+Zhu%2E+Iconqa%3A+A+new+benchmark+for+abstract+diagram+understanding+and+visual+language+reasoning%2E+arXiv+preprint+arXiv%3A2110%2E1)
48. P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022a. [find](https://scholar.google.com/scholar?q=P%2E+Lu%2C+S%2E+Mishra%2C+T%2E+Xia%2C+L%2E+Qiu%2C+K%2E%2DW%2E+Chang%2C+S%2E%2DC%2E+Zhu%2C+O%2E+Tafjord%2C+P%2E+Clark%2C+and+A%2E+Kalyan%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+science+question+answering%2E+In+The+36th+Con)
49. P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022b. [find](https://scholar.google.com/scholar?q=P%2E+Lu%2C+S%2E+Mishra%2C+T%2E+Xia%2C+L%2E+Qiu%2C+K%2E%2DW%2E+Chang%2C+S%2E%2DC%2E+Zhu%2C+O%2E+Tafjord%2C+P%2E+Clark%2C+and+A%2E+Kalyan%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+science+question+answering%2E+Advances+in+Neu)
50. P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023. [find](https://scholar.google.com/scholar?q=P%2E+Lu%2C+H%2E+Bansal%2C+T%2E+Xia%2C+J%2E+Liu%2C+C%2E+Li%2C+H%2E+Hajishirzi%2C+H%2E+Cheng%2C+K%2E%2DW%2E+Chang%2C+M%2E+Galley%2C+and+J%2E+Gao%2E+Mathvista%3A+Evaluating+mathematical+reasoning+of+foundation+models+in+visual+contexts%2E+arXiv+prepri) [our card](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/)
51. J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016. [find](https://scholar.google.com/scholar?q=J%2E+Mao%2C+J%2E+Huang%2C+A%2E+Toshev%2C+O%2E+Camburu%2C+A%2E+L%2E+Yuille%2C+and+K%2E+Murphy%2E+Generation+and+comprehension+of+unambiguous+object+descriptions%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+patt) [our card](https://md.research.elamur.ai/free/en/papers/mao-2015-generation-comprehension-unambiguous-object-descriptions/)
52. A. Masry, P. Kavehzadeh, X. L. Do, E. Hoque, and S. Joty. Unichart: A universal vision-language pretrained model for chart comprehension and reasoning. arXiv preprint arXiv:2305.14761, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Masry%2C+P%2E+Kavehzadeh%2C+X%2E+L%2E+Do%2C+E%2E+Hoque%2C+and+S%2E+Joty%2E+Unichart%3A+A+universal+vision%2Dlanguage+pretrained+model+for+chart+comprehension+and+reasoning%2E+arXiv+preprint+arXiv%3A2305%2E14761%2C+2023%2E)
53. D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computin… [find](https://scholar.google.com/scholar?q=D%2E+Narayanan%2C+M%2E+Shoeybi%2C+J%2E+Casper%2C+P%2E+LeGresley%2C+M%2E+Patwary%2C+V%2E+Korthikanti%2C+D%2E+Vainbrand%2C+P%2E+Kashinkunti%2C+J%2E+Bernauer%2C+B%2E+Catanzaro%2C+et+al%2E+Efficient+large%2Dscale+language+model+training+on+gpu+clus)
54. N. Nayef, F. Yin, I. Bizid, H. Choi, Y. Feng, D. Karatzas, Z. Luo, U. Pal, C. Rigaud, J. Chazalon, et al. Icdar2017 robust reading challenge on multi-lingual scene text detection and script identification-rrc-mlt. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR… [find](https://scholar.google.com/scholar?q=N%2E+Nayef%2C+F%2E+Yin%2C+I%2E+Bizid%2C+H%2E+Choi%2C+Y%2E+Feng%2C+D%2E+Karatzas%2C+Z%2E+Luo%2C+U%2E+Pal%2C+C%2E+Rigaud%2C+J%2E+Chazalon%2C+et+al%2E+Icdar2017+robust+reading+challenge+on+multi%2Dlingual+scene+text+detection+and+script+identifica)
55. OpenAI. Chatgpt: Optimizing language models for dialogue. 2022. URL [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt) . [↗](https://openai.com/blog/chatgpt)
56. OpenAI. GPT-4 technical report. arXiv, 2023a. [find](https://scholar.google.com/scholar?q=OpenAI%2E+GPT%2D4+technical+report%2E+arXiv%2C+2023a%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
57. R. OpenAI. Gpt-4v(ision) system card. 2023b. [find](https://scholar.google.com/scholar?q=R%2E+OpenAI%2E+Gpt%2D4v%28ision%29+system+card%2E+2023b%2E)
58. J. A. Rodriguez, D. Vazquez, I. Laradji, M. Pedersoli, and P. Rodriguez. Ocr-vqgan: Taming text-within-image generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3689–3698, 2023. [find](https://scholar.google.com/scholar?q=J%2E+A%2E+Rodriguez%2C+D%2E+Vazquez%2C+I%2E+Laradji%2C+M%2E+Pedersoli%2C+and+P%2E+Rodriguez%2E+Ocr%2Dvqgan%3A+Taming+text%2Dwithin%2Dimage+generation%2E+In+Proceedings+of+the+IEEE%2FCVF+Winter+Conference+on+Applications+of+Computer+Vi)
59. R. Schaeffer, B. Miranda, and S. Koyejo. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=R%2E+Schaeffer%2C+B%2E+Miranda%2C+and+S%2E+Koyejo%2E+Are+emergent+abilities+of+large+language+models+a+mirage%3F+Advances+in+Neural+Information+Processing+Systems%2C+36%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/schaeffer-2023-emergent-abilities-large-language-models/)
60. N. Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. [find](https://scholar.google.com/scholar?q=N%2E+Shazeer%2E+Glu+variants+improve+transformer%2E+arXiv+preprint+arXiv%3A2002%2E05202%2C+2020%2E)
61. B. Shi, C. Yao, M. Liao, M. Yang, P. Xu, L. Cui, S. Belongie, S. Lu, and X. Bai. Icdar2017 competition on reading chinese text in the wild (rctw-17). In 2017 14th iapr international conference on document analysis and recognition (ICDAR), volume 1, pages 1429–1434. [find](https://scholar.google.com/scholar?q=B%2E+Shi%2C+C%2E+Yao%2C+M%2E+Liao%2C+M%2E+Yang%2C+P%2E+Xu%2C+L%2E+Cui%2C+S%2E+Belongie%2C+S%2E+Lu%2C+and+X%2E+Bai%2E+Icdar2017+competition+on+reading+chinese+text+in+the+wild+%28rctw%2D17%29%2E+In+2017+14th+iapr+international+conference+on+docu)
62. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. [find](https://scholar.google.com/scholar?q=M%2E+Shoeybi%2C+M%2E+Patwary%2C+R%2E+Puri%2C+P%2E+LeGresley%2C+J%2E+Casper%2C+and+B%2E+Catanzaro%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+arXiv+preprint+arXiv%3A1909%2E08053%2C+2019)
63. A. Singh, G. Pang, M. Toh, J. Huang, W. Galuba, and T. Hassner. Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8802–8812, 2021. [find](https://scholar.google.com/scholar?q=A%2E+Singh%2C+G%2E+Pang%2C+M%2E+Toh%2C+J%2E+Huang%2C+W%2E+Galuba%2C+and+T%2E+Hassner%2E+Textocr%3A+Towards+large%2Dscale+end%2Dto%2Dend+reasoning+for+arbitrary%2Dshaped+scene+text%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer)
64. J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. [find](https://scholar.google.com/scholar?q=J%2E+Su%2C+M%2E+Ahmed%2C+Y%2E+Lu%2C+S%2E+Pan%2C+W%2E+Bo%2C+and+Y%2E+Liu%2E+Roformer%3A+Enhanced+transformer+with+rotary+position+embedding%2E+Neurocomputing%2C+568%3A127063%2C+2024%2E)
65. Q. Sun, Q. Yu, Y. Cui, F. Zhang, X. Zhang, Y. Wang, H. Gao, J. Liu, T. Huang, and X. Wang. Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222, 2023. — [arXiv:2307.05222](https://arxiv.org/abs/2307.05222) [find](https://scholar.google.com/scholar?q=Q%2E+Sun%2C+Q%2E+Yu%2C+Y%2E+Cui%2C+F%2E+Zhang%2C+X%2E+Zhang%2C+Y%2E+Wang%2C+H%2E+Gao%2C+J%2E+Liu%2C+T%2E+Huang%2C+and+X%2E+Wang%2E+Generative+pretraining+in+multimodality%2E+arXiv+preprint+arXiv%3A2307%2E05222%2C+2023%2E+%E2%80%94+%5BarXiv%3A2307%2E05222%5D%28https%3A%2F%2F)
66. Y. Sun, Z. Ni, C.-K. Chng, Y. Liu, C. Luo, C. C. Ng, J. Han, E. Ding, J. Liu, D. Karatzas, et al. Icdar 2019 competition on large-scale street view text with partial labeling-rrc-lsvt. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1557–1562. [find](https://scholar.google.com/scholar?q=Y%2E+Sun%2C+Z%2E+Ni%2C+C%2E%2DK%2E+Chng%2C+Y%2E+Liu%2C+C%2E+Luo%2C+C%2E+C%2E+Ng%2C+J%2E+Han%2C+E%2E+Ding%2C+J%2E+Liu%2C+D%2E+Karatzas%2C+et+al%2E+Icdar+2019+competition+on+large%2Dscale+street+view+text+with+partial+labeling%2Drrc%2Dlsvt%2E+In+2019+Interna)
67. G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. [find](https://scholar.google.com/scholar?q=G%2E+Team%2C+R%2E+Anil%2C+S%2E+Borgeaud%2C+Y%2E+Wu%2C+J%2E%2DB%2E+Alayrac%2C+J%2E+Yu%2C+R%2E+Soricut%2C+J%2E+Schalkwyk%2C+A%2E+M%2E+Dai%2C+A%2E+Hauth%2C+et+al%2E+Gemini%3A+a+family+of+highly+capable+multimodal+models%2E+arXiv+preprint+arXiv%3A2312%2E11805%2C) [our card](https://md.research.elamur.ai/free/en/papers/gemini-2023-team-gemini-family-highly-capable-multimodal/)
68. S. Tong, Z. Liu, Y. Zhai, Y. Ma, Y. LeCun, and S. Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. arXiv preprint arXiv:2401.06209, 2024. [find](https://scholar.google.com/scholar?q=S%2E+Tong%2C+Z%2E+Liu%2C+Y%2E+Zhai%2C+Y%2E+Ma%2C+Y%2E+LeCun%2C+and+S%2E+Xie%2E+Eyes+wide+shut%3F+exploring+the+visual+shortcomings+of+multimodal+llms%2E+arXiv+preprint+arXiv%3A2401%2E06209%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/tong-2024-eyes-wide-shut-exploring-visual/)
69. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a. [find](https://scholar.google.com/scholar?q=H%2E+Touvron%2C+T%2E+Lavril%2C+G%2E+Izacard%2C+X%2E+Martinet%2C+M%2E%2DA%2E+Lachaux%2C+T%2E+Lacroix%2C+B%2E+Rozi%C3%A8re%2C+N%2E+Goyal%2C+E%2E+Hambro%2C+F%2E+Azhar%2C+et+al%2E+LLaMA%3A+Open+and+efficient+foundation+language+models%2E+arXiv+preprint+arXiv%3A)
70. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. In… [find](https://scholar.google.com/scholar?q=H%2E+Touvron%2C+L%2E+Martin%2C+K%2E+Stone%2C+P%2E+Albert%2C+A%2E+Almahairi%2C+Y%2E+Babaei%2C+N%2E+Bashlykov%2C+S%2E+Batra%2C+P%2E+Bhargava%2C+S%2E+Bhosale%2C+D%2E+Bikel%2C+L%2E+Blecher%2C+C%2E+Canton%2DFerrer%2C+M%2E+Chen%2C+G%2E+Cucurull%2C+D%2E+Esiobu%2C+J%2E+Fernan)
71. A. Veit, T. Matera, L. Neumann, J. Matas, and S. Belongie. Coco-text: Dataset and benchmark for text detection and recognition in natural images. arXiv preprint arXiv:1601.07140, 2016. — [arXiv:1601.07140](https://arxiv.org/abs/1601.07140) [find](https://scholar.google.com/scholar?q=A%2E+Veit%2C+T%2E+Matera%2C+L%2E+Neumann%2C+J%2E+Matas%2C+and+S%2E+Belongie%2E+Coco%2Dtext%3A+Dataset+and+benchmark+for+text+detection+and+recognition+in+natural+images%2E+arXiv+preprint+arXiv%3A1601%2E07140%2C+2016%2E+%E2%80%94+%5BarXiv%3A1601%2E0)
72. B. Wang, G. Li, X. Zhou, Z. Chen, T. Grossman, and Y. Li. Screen2words: Automatic mobile ui summarization with multimodal learning. In The 34th Annual ACM Symposium on User Interface Software and Technology, pages 498–510, 2021. [find](https://scholar.google.com/scholar?q=B%2E+Wang%2C+G%2E+Li%2C+X%2E+Zhou%2C+Z%2E+Chen%2C+T%2E+Grossman%2C+and+Y%2E+Li%2E+Screen2words%3A+Automatic+mobile+ui+summarization+with+multimodal+learning%2E+In+The+34th+Annual+ACM+Symposium+on+User+Interface+Software+and+Tech)
73. J. Wang, L. Meng, Z. Weng, B. He, Z. Wu, and Y.-G. Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574, 2023a. — [arXiv:2311.07574](https://arxiv.org/abs/2311.07574) [find](https://scholar.google.com/scholar?q=J%2E+Wang%2C+L%2E+Meng%2C+Z%2E+Weng%2C+B%2E+He%2C+Z%2E+Wu%2C+and+Y%2E%2DG%2E+Jiang%2E+To+see+is+to+believe%3A+Prompting+gpt%2D4v+for+better+visual+instruction+tuning%2E+arXiv+preprint+arXiv%3A2311%2E07574%2C+2023a%2E+%E2%80%94+%5BarXiv%3A2311%2E07574%5D%28http)
74. W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y. Wang, J. Ji, Z. Yang, L. Zhao, X. Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023b. [find](https://scholar.google.com/scholar?q=W%2E+Wang%2C+Q%2E+Lv%2C+W%2E+Yu%2C+W%2E+Hong%2C+J%2E+Qi%2C+Y%2E+Wang%2C+J%2E+Ji%2C+Z%2E+Yang%2C+L%2E+Zhao%2C+X%2E+Song%2C+et+al%2E+Cogvlm%3A+Visual+expert+for+pretrained+language+models%2E+arXiv+preprint+arXiv%3A2311%2E03079%2C+2023b%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2023-cogvlm-visual-expert-pretrained/)
75. H. Wei, L. Kong, J. Chen, L. Zhao, Z. Ge, J. Yang, J. Sun, C. Han, and X. Zhang. Vary: Scaling up the vision vocabulary for large vision-language models. arXiv preprint arXiv:2312.06109, 2023. — [arXiv:2312.06109](https://arxiv.org/abs/2312.06109) [find](https://scholar.google.com/scholar?q=H%2E+Wei%2C+L%2E+Kong%2C+J%2E+Chen%2C+L%2E+Zhao%2C+Z%2E+Ge%2C+J%2E+Yang%2C+J%2E+Sun%2C+C%2E+Han%2C+and+X%2E+Zhang%2E+Vary%3A+Scaling+up+the+vision+vocabulary+for+large+vision%2Dlanguage+models%2E+arXiv+preprint+arXiv%3A2312%2E06109%2C+2023%2E+%E2%80%94+%5BarXi)
76. Y. Yang, A. Panagopoulou, Q. Lyu, L. Zhang, M. Yatskar, and C. Callison-Burch. Visual goal-step inference using wikihow. arXiv preprint arXiv:2104.05845, 2021. [find](https://scholar.google.com/scholar?q=Y%2E+Yang%2C+A%2E+Panagopoulou%2C+Q%2E+Lyu%2C+L%2E+Zhang%2C+M%2E+Yatskar%2C+and+C%2E+Callison%2DBurch%2E+Visual+goal%2Dstep+inference+using+wikihow%2E+arXiv+preprint+arXiv%3A2104%2E05845%2C+2021%2E)
77. J. Ye, A. Hu, H. Xu, Q. Ye, M. Yan, G. Xu, C. Li, J. Tian, Q. Qian, J. Zhang, et al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. arXiv preprint arXiv:2310.05126, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Ye%2C+A%2E+Hu%2C+H%2E+Xu%2C+Q%2E+Ye%2C+M%2E+Yan%2C+G%2E+Xu%2C+C%2E+Li%2C+J%2E+Tian%2C+Q%2E+Qian%2C+J%2E+Zhang%2C+et+al%2E+Ureader%3A+Universal+ocr%2Dfree+visually%2Dsituated+language+understanding+with+multimodal+large+language+model%2E+arXiv+pr)
78. Q. Yu, Q. Sun, X. Zhang, Y. Cui, F. Zhang, Y. Cao, X. Wang, and J. Liu. Capsfusion: Rethinking image-text data at scale. arXiv preprint arXiv:2310.20550, 2023a. — [arXiv:2310.20550](https://arxiv.org/abs/2310.20550) [find](https://scholar.google.com/scholar?q=Q%2E+Yu%2C+Q%2E+Sun%2C+X%2E+Zhang%2C+Y%2E+Cui%2C+F%2E+Zhang%2C+Y%2E+Cao%2C+X%2E+Wang%2C+and+J%2E+Liu%2E+Capsfusion%3A+Rethinking+image%2Dtext+data+at+scale%2E+arXiv+preprint+arXiv%3A2310%2E20550%2C+2023a%2E+%E2%80%94+%5BarXiv%3A2310%2E20550%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
79. W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023b. [find](https://scholar.google.com/scholar?q=W%2E+Yu%2C+Z%2E+Yang%2C+L%2E+Li%2C+J%2E+Wang%2C+K%2E+Lin%2C+Z%2E+Liu%2C+X%2E+Wang%2C+and+L%2E+Wang%2E+Mm%2Dvet%3A+Evaluating+large+multimodal+models+for+integrated+capabilities%2E+arXiv+preprint+arXiv%3A2308%2E02490%2C+2023b%2E) [our card](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/)
80. X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502, 2023. [find](https://scholar.google.com/scholar?q=X%2E+Yue%2C+Y%2E+Ni%2C+K%2E+Zhang%2C+T%2E+Zheng%2C+R%2E+Liu%2C+G%2E+Zhang%2C+S%2E+Stevens%2C+D%2E+Jiang%2C+W%2E+Ren%2C+Y%2E+Sun%2C+et+al%2E+Mmmu%3A+A+massive+multi%2Ddiscipline+multimodal+understanding+and+reasoning+benchmark+for+expert+agi%2E+arXi) [our card](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/)
81. R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 20… [find](https://scholar.google.com/scholar?q=R%2E+Zellers%2C+A%2E+Holtzman%2C+Y%2E+Bisk%2C+A%2E+Farhadi%2C+and+Y%2E+Choi%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+A%2E+Korhonen%2C+D%2E+R%2E+Traum%2C+and+L%2E+M%C3%A0rquez%2C+editors%2C+Proceedings+of+the+57th+Conferenc)
82. B. Zhang and R. Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019. [find](https://scholar.google.com/scholar?q=B%2E+Zhang+and+R%2E+Sennrich%2E+Root+mean+square+layer+normalization%2E+Advances+in+Neural+Information+Processing+Systems%2C+32%2C+2019%2E)
83. G. Zhang, X. Du, B. Chen, Y. Liang, T. Luo, T. Zheng, K. Zhu, Y. Cheng, C. Xu, S. Guo, et al. Cmmmu: A chinese massive multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2401.11944, 2024. — [arXiv:2401.11944](https://arxiv.org/abs/2401.11944) [find](https://scholar.google.com/scholar?q=G%2E+Zhang%2C+X%2E+Du%2C+B%2E+Chen%2C+Y%2E+Liang%2C+T%2E+Luo%2C+T%2E+Zheng%2C+K%2E+Zhu%2C+Y%2E+Cheng%2C+C%2E+Xu%2C+S%2E+Guo%2C+et+al%2E+Cmmmu%3A+A+chinese+massive+multi%2Ddiscipline+multimodal+understanding+benchmark%2E+arXiv+preprint+arXiv%3A2401%2E11)
84. R. Zhang, Y. Zhou, Q. Jiang, Q. Song, N. Li, K. Zhou, L. Wang, D. Wang, M. Liao, M. Yang, et al. Icdar 2019 robust reading challenge on reading chinese text on signboard. In 2019 international conference on document analysis and recognition (ICDAR), pages 1577–1581. [find](https://scholar.google.com/scholar?q=R%2E+Zhang%2C+Y%2E+Zhou%2C+Q%2E+Jiang%2C+Q%2E+Song%2C+N%2E+Li%2C+K%2E+Zhou%2C+L%2E+Wang%2C+D%2E+Wang%2C+M%2E+Liao%2C+M%2E+Yang%2C+et+al%2E+Icdar+2019+robust+reading+challenge+on+reading+chinese+text+on+signboard%2E+In+2019+international+confere)
85. Y. Zhang, L. Gueguen, I. Zharkov, P. Zhang, K. Seifert, and B. Kadlec. Uber-text: A large-scale dataset for optical character recognition from street-level imagery. In SUNw: Scene Understanding Workshop - CVPR 2017, Hawaii, U.S.A., 2017. URL [http://sunw.csail.m](http://sunw.csail.m) it.edu/abstract/uberText.pdf . [↗](http://sunw.csail.m)
86. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=L%2E+Zheng%2C+W%2E%2DL%2E+Chiang%2C+Y%2E+Sheng%2C+S%2E+Zhuang%2C+Z%2E+Wu%2C+Y%2E+Zhuang%2C+Z%2E+Lin%2C+Z%2E+Li%2C+D%2E+Li%2C+E%2E+Xing%2C+et+al%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2E+Advances+in+Neural+Information+Processing+S)
87. W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan. AGIEval: A human-centric benchmark for evaluating foundation models. CoRR, abs/2304.06364, 2023. doi: 10.48550/arXiv.2304.06364. URL https://doi.org/10.48550/arXiv.2304.06364 . [↗](https://doi.org/10.48550/arXiv.2304.06364) [our card](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/)
88. W. Zhu, J. Hessel, A. Awadalla, S. Y. Gadre, J. Dodge, A. Fang, Y. Yu, L. Schmidt, W. Y. Wang, and Y. Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=W%2E+Zhu%2C+J%2E+Hessel%2C+A%2E+Awadalla%2C+S%2E+Y%2E+Gadre%2C+J%2E+Dodge%2C+A%2E+Fang%2C+Y%2E+Yu%2C+L%2E+Schmidt%2C+W%2E+Y%2E+Wang%2C+and+Y%2E+Choi%2E+Multimodal+c4%3A+An+open%2C+billion%2Dscale+corpus+of+images+interleaved+with+text%2E+Advances+in+Ne)