---
title: "Qwen-VL: A Versatile Vision-Language Model for — Bai 2023"
description: "Open-source large vision-language models (LVLMs) have trailed proprietary systems."
authors: ["Jinze Bai", "Shuai Bai", "Shusheng Yang", "Shijie Wang", "Sinan Tan", "Peng Wang", "Junyang Lin", "Chang Zhou", "Jingren Zhou"]
year: 2023
venue: "arXiv:2308.12966"
source: "https://arxiv.org/abs/2308.12966"
doi: "10.48550/arxiv.2308.12966"
arxiv: "2308.12966"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/)
[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/) / Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
# Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
**Authors:** [Jinze Bai](https://md.research.elamur.ai/free/en/authors/jinze-bai/), [Shuai Bai](https://md.research.elamur.ai/free/en/authors/shuai-bai/), [Shusheng Yang](https://md.research.elamur.ai/free/en/authors/shusheng-yang/), [Shijie Wang](https://md.research.elamur.ai/free/en/authors/shijie-wang/), [Sinan Tan](https://md.research.elamur.ai/free/en/authors/sinan-tan/), [Peng Wang](https://md.research.elamur.ai/free/en/authors/peng-wang/), [Junyang Lin](https://md.research.elamur.ai/free/en/authors/junyang-lin/), [Chang Zhou](https://md.research.elamur.ai/free/en/authors/chang-zhou/), [Jingren Zhou](https://md.research.elamur.ai/free/en/authors/jingren-zhou/) · **Institutions:** [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) · **Year:** 2023 · **Venue:** arXiv:2308.12966 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)
**Links:** [Publisher's page](https://arxiv.org/abs/2308.12966) · [PDF](https://arxiv.org/pdf/2308.12966) · [DOI 10.48550/arxiv.2308.12966](https://doi.org/10.48550/arxiv.2308.12966) · [arXiv 2308.12966](https://arxiv.org/abs/2308.12966) · [OpenAlex](https://openalex.org/W4386185600)
**Citations:** 149 citations (OpenAlex, 2026-10-14) · [764](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/cited-by/) in our library, of the 140884 papers whose bibliographies we read
## Summary
Open-source large vision-language models (LVLMs) have trailed proprietary systems. Most also view images only coarsely, so they handle tasks like object localization or reading text in pictures poorly. This paper introduces the Qwen-VL series, built on the Qwen-7B language model. A ViT visual encoder feeds the LLM through a compact adapter. The adapter uses cross-attention with learnable queries and 2D positional encodings to shrink every image to a fixed sequence of 256 tokens. Bounding boxes are expressed as plain-text coordinates wrapped in special tokens, so the model can both interpret and generate region references. Training has three phases. First comes large-scale pre-training on cleaned English and Chinese web image-text pairs. Second is higher-resolution multi-task pre-training covering captioning, VQA, grounding, OCR and interleaved data. Third, supervised instruction tuning produces Qwen-VL-Chat. Among generalist models of similar size, Qwen-VL leads on captioning, VQA, grounding and text-oriented benchmarks. The chat variant outperforms competing assistants on instruction-following evaluations. All weights are openly released, offering a strong bilingual baseline for multimodal research.
## Key points
- Architecture totals about 9.6B parameters: a 1.9B ViT-bigG encoder initialized from OpenCLIP, a 0.08B single-layer cross-attention adapter, and a 7.7B LLM initialized from Qwen-7B.
- Grounding uses box coordinates normalized to [0, 1000) and marked with ``/``. Referred phrases are tagged with `[`/`]`, and images are delimited by `![]()
`/``.
- Stage-one data was filtered from roughly 5B image-text pairs down to 1.4B, with 77.3% English and 22.7% Chinese text. Sources include LAION-en, LAION-zh, LAION-COCO, DataComp, Coyo, CC12M, CC3M, SBU, COCO Caption and in-house data.
- Stage one keeps the LLM frozen and uses 224×224 inputs. It runs for 50,000 steps at batch size 30720, about 1.5B samples. Stage two trains on 7 tasks jointly, packing interleaved data into 2048-length sequences.
- Qwen-VL-Chat was assessed on TouchStone (English and Chinese), SEED-Bench (19K multiple-choice questions across 12 dimensions) and MME (14 subtasks). It showed clear gains on all three and handled video by sampling four frames.
*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*
Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)
Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/)
## Cite
The paper itself, not this page. BibTeX from the library's record of the work.
```bibtex
@misc{bai2023qwen,
title = {{Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond}},
author = {Bai, Jinze and Bai, Shuai and Yang, Shusheng and Wang, Shijie and Tan, Sinan and Wang, Peng and Lin, Junyang and Zhou, Chang and Zhou, Jingren},
year = {2023},
doi = {10.48550/arxiv.2308.12966},
eprint = {2308.12966},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2308.12966}
}
```
- [← previous C-Pack: Packed Resources For General Chinese Embeddings](https://md.research.elamur.ai/free/en/papers/xiao-2023-c-pack-packed-resources/)
- [next → MMBench: Is Your Multi-modal Model an All-around Player?](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/)
## References
83 references, 42 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.
1. Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. In ICCV, 2019. [find](https://scholar.google.com/scholar?q=Harsh+Agrawal%2C+Karan+Desai%2C+Yufei+Wang%2C+Xinlei+Chen%2C+Rishabh+Jain%2C+Mark+Johnson%2C+Dhruv+Batra%2C+Devi+Parikh%2C+Stefan+Lee%2C+and+Peter+Anderson%2E+nocaps%3A+novel+object+captioning+at+scale%2E+In+ICCV%2C+2019%2E)
2. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022. [find](https://scholar.google.com/scholar?q=Jean%2DBaptiste+Alayrac%2C+Jeff+Donahue%2C+Pauline+Luc%2C+Antoine+Miech%2C+Iain+Barr%2C+Yana+Hasson%2C+Karel+Lenc%2C+Arthur+Mensch%2C+Katherine+Millican%2C+Malcolm+Reynolds%2C+et+al%2E+Flamingo%3A+a+visual+language+model+for+f) [our card](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
3. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv:2305.10403, 2023. [find](https://scholar.google.com/scholar?q=Rohan+Anil%2C+Andrew+M+Dai%2C+Orhan+Firat%2C+Melvin+Johnson%2C+Dmitry+Lepikhin%2C+Alexandre+Passos%2C+Siamak+Shakeri%2C+Emanuel+Taropa%2C+Paige+Bailey%2C+Zhifeng+Chen%2C+et+al%2E+Palm+2+technical+report%2E+arXiv%3A2305%2E10403%2C+) [our card](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/)
4. Jinze Bai, Rui Men, Hao Yang, Xuancheng Ren, Kai Dang, Yichang Zhang, Xiaohuan Zhou, Peng Wang, Sinan Tan, An Yang, et al. Ofasys: A multi-modal multi-task learning system for building generalist models. arXiv:2212.04408, 2022. — [arXiv:2212.04408](https://arxiv.org/abs/2212.04408) [find](https://scholar.google.com/scholar?q=Jinze+Bai%2C+Rui+Men%2C+Hao+Yang%2C+Xuancheng+Ren%2C+Kai+Dang%2C+Yichang+Zhang%2C+Xiaohuan+Zhou%2C+Peng+Wang%2C+Sinan+Tan%2C+An+Yang%2C+et+al%2E+Ofasys%3A+A+multi%2Dmodal+multi%2Dtask+learning+system+for+building+generalist+mode)
5. Shuai Bai, Shusheng Yang, Jinze Bai, Peng Wang, Xingxuan Zhang, Junyang Lin, Xinggang Wang, Chang Zhou, and Jingren Zhou. Touchstone: Evaluating vision-language models by language models. arXiv:2308.16890, 2023. — [arXiv:2308.16890](https://arxiv.org/abs/2308.16890) [find](https://scholar.google.com/scholar?q=Shuai+Bai%2C+Shusheng+Yang%2C+Jinze+Bai%2C+Peng+Wang%2C+Xingxuan+Zhang%2C+Junyang+Lin%2C+Xinggang+Wang%2C+Chang+Zhou%2C+and+Jingren+Zhou%2E+Touchstone%3A+Evaluating+vision%2Dlanguage+models+by+language+models%2E+arXiv%3A2308%2E1)
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020. [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+) [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
7. Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset, 2022. URL [https://github.com/kakaobrain/coyo-dataset](https://github.com/kakaobrain/coyo-dataset). [↗](https://github.com/kakaobrain/coyo-dataset)
8. Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021. [find](https://scholar.google.com/scholar?q=Soravit+Changpinyo%2C+Piyush+Sharma%2C+Nan+Ding%2C+and+Radu+Soricut%2E+Conceptual+12m%3A+Pushing+web%2Dscale+image%2Dtext+pre%2Dtraining+to+recognize+long%2Dtail+visual+concepts%2E+In+CVPR%2C+2021%2E)
9. Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv:2306.15195, 2023a. [find](https://scholar.google.com/scholar?q=Keqin+Chen%2C+Zhao+Zhang%2C+Weili+Zeng%2C+Richong+Zhang%2C+Feng+Zhu%2C+and+Rui+Zhao%2E+Shikra%3A+Unleashing+multimodal+llm%E2%80%99s+referential+dialogue+magic%2E+arXiv%3A2306%2E15195%2C+2023a%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2023-shikra-unleashing-multimodal-llm-s/)
10. Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv:2209.06794, 2022. [find](https://scholar.google.com/scholar?q=Xi+Chen%2C+Xiao+Wang%2C+Soravit+Changpinyo%2C+AJ+Piergiovanni%2C+Piotr+Padlewski%2C+Daniel+Salz%2C+Sebastian+Goodman%2C+Adam+Grycner%2C+Basil+Mustafa%2C+Lucas+Beyer%2C+et+al%2E+Pali%3A+A+jointly%2Dscaled+multilingual+language%2D) [our card](https://md.research.elamur.ai/free/en/papers/chen-2022-pali-jointly-scaled-multilingual-language/)
11. Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.18565, 2023b. — [arXiv:2305.18565](https://arxiv.org/abs/2305.18565) [find](https://scholar.google.com/scholar?q=Xi+Chen%2C+Josip+Djolonga%2C+Piotr+Padlewski%2C+Basil+Mustafa%2C+Soravit+Changpinyo%2C+Jialin+Wu%2C+Carlos+Riquelme+Ruiz%2C+Sebastian+Goodman%2C+Xiao+Wang%2C+Yi+Tay%2C+et+al%2E+Pali%2Dx%3A+On+scaling+up+a+multilingual+vision+a)
12. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv:1504.00325, 2015. [find](https://scholar.google.com/scholar?q=Xinlei+Chen%2C+Hao+Fang%2C+Tsung%2DYi+Lin%2C+Ramakrishna+Vedantam%2C+Saurabh+Gupta%2C+Piotr+Doll%C3%A1r%2C+and+C+Lawrence+Zitnick%2E+Microsoft+coco+captions%3A+Data+collection+and+evaluation+server%2E+arXiv%3A1504%2E00325%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2015-microsoft-coco-captions-data-collection/)
13. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, 2020. [find](https://scholar.google.com/scholar?q=Yen%2DChun+Chen%2C+Linjie+Li%2C+Licheng+Yu%2C+Ahmed+El+Kholy%2C+Faisal+Ahmed%2C+Zhe+Gan%2C+Yu+Cheng%2C+and+Jingjing+Liu%2E+Uniter%3A+Universal+image%2Dtext+representation+learning%2E+In+ECCV%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2019-uniter-universal-image-text-representation/)
14. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv:2305.06500, 2023. [find](https://scholar.google.com/scholar?q=Wenliang+Dai%2C+Junnan+Li%2C+Dongxu+Li%2C+Anthony+Meng+Huat+Tiong%2C+Junqi+Zhao%2C+Weisheng+Wang%2C+Boyang+Li%2C+Pascale+Fung%2C+and+Steven+Hoi%2E+Instructblip%3A+Towards+general%2Dpurpose+vision%2Dlanguage+models+with+instr) [our card](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/)
15. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. [find](https://scholar.google.com/scholar?q=Alexey+Dosovitskiy%2C+Lucas+Beyer%2C+Alexander+Kolesnikov%2C+Dirk+Weissenborn%2C+Xiaohua+Zhai%2C+Thomas+Unterthiner%2C+Mostafa+Dehghani%2C+Matthias+Minderer%2C+Georg+Heigold%2C+Sylvain+Gelly%2C+Jakob+Uszkoreit%2C+and+Neil+) [our card](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/)
16. Zi-Yi* Dou, Aishwarya* Kamath, Zhe* Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, Jianfeng Gao, and Lijuan Wang. Coarse-to-fine vision-language pre-training with fusion in the backbone. In NeurIPS, 2022. [find](https://scholar.google.com/scholar?q=Zi%2DYi%2A+Dou%2C+Aishwarya%2A+Kamath%2C+Zhe%2A+Gan%2C+Pengchuan+Zhang%2C+Jianfeng+Wang%2C+Linjie+Li%2C+Zicheng+Liu%2C+Ce+Liu%2C+Yann+LeCun%2C+Nanyun+Peng%2C+Jianfeng+Gao%2C+and+Lijuan+Wang%2E+Coarse%2Dto%2Dfine+vision%2Dlanguage+pre%2Dtrai)
17. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv:2306.13394, 2023. [find](https://scholar.google.com/scholar?q=Chaoyou+Fu%2C+Peixian+Chen%2C+Yunhang+Shen%2C+Yulei+Qin%2C+Mengdan+Zhang%2C+Xu+Lin%2C+Zhenyu+Qiu%2C+Wei+Lin%2C+Jinrui+Yang%2C+Xiawu+Zheng%2C+et+al%2E+Mme%3A+A+comprehensive+evaluation+benchmark+for+multimodal+large+language+) [our card](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/)
18. Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. arXiv:2304.14108, 2023. [find](https://scholar.google.com/scholar?q=Samir+Yitzhak+Gadre%2C+Gabriel+Ilharco%2C+Alex+Fang%2C+Jonathan+Hayase%2C+Georgios+Smyrnis%2C+Thao+Nguyen%2C+Ryan+Marten%2C+Mitchell+Wortsman%2C+Dhruba+Ghosh%2C+Jieyu+Zhang%2C+et+al%2E+Datacomp%3A+In+search+of+the+next+gener)
19. Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv:2304.15010, 2023. [find](https://scholar.google.com/scholar?q=Peng+Gao%2C+Jiaming+Han%2C+Renrui+Zhang%2C+Ziyi+Lin%2C+Shijie+Geng%2C+Aojun+Zhou%2C+Wei+Zhang%2C+Pan+Lu%2C+Conghui+He%2C+Xiangyu+Yue%2C+et+al%2E+Llama%2Dadapter+v2%3A+Parameter%2Defficient+visual+instruction+model%2E+arXiv%3A2304%2E15)
20. Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In CVPR, 2023. [find](https://scholar.google.com/scholar?q=Rohit+Girdhar%2C+Alaaeldin+El%2DNouby%2C+Zhuang+Liu%2C+Mannat+Singh%2C+Kalyan+Vasudev+Alwala%2C+Armand+Joulin%2C+and+Ishan+Misra%2E+Imagebind%3A+One+embedding+space+to+bind+them+all%2E+In+CVPR%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/girdhar-2023-imagebind-one-embedding-space-bind/)
21. Google. Puppeteer, 2023. URL [https://github.com/puppeteer/puppeteer](https://github.com/puppeteer/puppeteer). [↗](https://github.com/puppeteer/puppeteer)
22. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017. [find](https://scholar.google.com/scholar?q=Yash+Goyal%2C+Tejas+Khot%2C+Douglas+Summers%2DStay%2C+Dhruv+Batra%2C+and+Devi+Parikh%2E+Making+the+v+in+vqa+matter%3A+Elevating+the+role+of+image+understanding+in+visual+question+answering%2E+In+CVPR%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/)
23. Tanmay Gupta, Ryan Marten, Aniruddha Kembhavi, and Derek Hoiem. Grit: General robust image task benchmark. arXiv:2204.13653, 2022. — [arXiv:2204.13653](https://arxiv.org/abs/2204.13653) [find](https://scholar.google.com/scholar?q=Tanmay+Gupta%2C+Ryan+Marten%2C+Aniruddha+Kembhavi%2C+and+Derek+Hoiem%2E+Grit%3A+General+robust+image+task+benchmark%2E+arXiv%3A2204%2E13653%2C+2022%2E+%E2%80%94+%5BarXiv%3A2204%2E13653%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2204%2E13653%29)
24. Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, 2018. [find](https://scholar.google.com/scholar?q=Danna+Gurari%2C+Qing+Li%2C+Abigale+J+Stangl%2C+Anhong+Guo%2C+Chi+Lin%2C+Kristen+Grauman%2C+Jiebo+Luo%2C+and+Jeffrey+P+Bigham%2E+Vizwiz+grand+challenge%3A+Answering+visual+questions+from+blind+people%2E+In+CVPR%2C+2018%2E)
25. Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with a unified transformer. In ICCV, 2021. [find](https://scholar.google.com/scholar?q=Ronghang+Hu+and+Amanpreet+Singh%2E+Unit%3A+Multimodal+multitask+learning+with+a+unified+transformer%2E+In+ICCV%2C+2021%2E)
26. Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv:2302.14045, 2023. [find](https://scholar.google.com/scholar?q=Shaohan+Huang%2C+Li+Dong%2C+Wenhui+Wang%2C+Yaru+Hao%2C+Saksham+Singhal%2C+Shuming+Ma%2C+Tengchao+Lv%2C+Lei+Cui%2C+Owais+Khan+Mohammed%2C+Qiang+Liu%2C+et+al%2E+Language+is+not+all+you+need%3A+Aligning+perception+with+language) [our card](https://md.research.elamur.ai/free/en/papers/huang-2023-language-all-need-aligning-perception/)
27. Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. [find](https://scholar.google.com/scholar?q=Drew+A+Hudson+and+Christopher+D+Manning%2E+Gqa%3A+A+new+dataset+for+real%2Dworld+visual+reasoning+and+compositional+question+answering%2E+In+CVPR%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/)
28. Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021. URL https://doi.org/10.5281/zenodo.5143773. — [doi:10.5281/zenodo.5143773](https://doi.org/10.5281/zenodo.5143773) [↗](https://doi.org/10.5281/zenodo.5143773)
29. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. arXiv:2102.05918, 2021. [find](https://scholar.google.com/scholar?q=Chao+Jia%2C+Yinfei+Yang%2C+Ye+Xia%2C+Yi%2DTing+Chen%2C+Zarana+Parekh%2C+Hieu+Pham%2C+Quoc+V+Le%2C+Yunhsuan+Sung%2C+Zhen+Li%2C+and+Tom+Duerig%2E+Scaling+up+visual+and+vision%2Dlanguage+representation+learning+with+noisy+text+) [our card](https://md.research.elamur.ai/free/en/papers/jia-2021-scaling-up-visual-vision-language/)
30. Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In CVPR, 2018. [find](https://scholar.google.com/scholar?q=Kushal+Kafle%2C+Brian+Price%2C+Scott+Cohen%2C+and+Christopher+Kanan%2E+Dvqa%3A+Understanding+data+visualizations+via+question+answering%2E+In+CVPR%2C+2018%2E)
31. Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, 2014. [find](https://scholar.google.com/scholar?q=Sahar+Kazemzadeh%2C+Vicente+Ordonez%2C+Mark+Matten%2C+and+Tamara+Berg%2E+Referitgame%3A+Referring+to+objects+in+photographs+of+natural+scenes%2E+In+EMNLP%2C+2014%2E) [our card](https://md.research.elamur.ai/free/en/papers/kazemzadeh-2014-referitgame-referring-objects-photographs-natural/)
32. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, 2016. [find](https://scholar.google.com/scholar?q=Aniruddha+Kembhavi%2C+Mike+Salvato%2C+Eric+Kolve%2C+Minjoon+Seo%2C+Hannaneh+Hajishirzi%2C+and+Ali+Farhadi%2E+A+diagram+is+worth+a+dozen+images%2E+In+ECCV%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/kembhavi-2016-diagram-worth-dozen-images/)
33. Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In ECCV, 2022. [find](https://scholar.google.com/scholar?q=Geewook+Kim%2C+Teakgyu+Hong%2C+Moonbin+Yim%2C+JeongYeon+Nam%2C+Jinyoung+Park%2C+Jinyeong+Yim%2C+Wonseok+Hwang%2C+Sangdoo+Yun%2C+Dongyoon+Han%2C+and+Seunghyun+Park%2E+Ocr%2Dfree+document+understanding+transformer%2E+In+ECCV%2C+) [our card](https://md.research.elamur.ai/free/en/papers/kim-2021-ocr-free-document-understanding-transformer/)
34. Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In ICML, 2021. [find](https://scholar.google.com/scholar?q=Wonjae+Kim%2C+Bokyung+Son%2C+and+Ildoo+Kim%2E+Vilt%3A+Vision%2Dand%2Dlanguage+transformer+without+convolution+or+region+supervision%2E+In+ICML%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2021-vilt-vision-language-transformer-convolution/)
35. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. In IJCV, 2017. [find](https://scholar.google.com/scholar?q=Ranjay+Krishna%2C+Yuke+Zhu%2C+Oliver+Groth%2C+Justin+Johnson%2C+Kenji+Hata%2C+Joshua+Kravitz%2C+Stephanie+Chen%2C+Yannis+Kalantidis%2C+Li%2DJia+Li%2C+David+A+Shamma%2C+et+al%2E+Visual+genome%3A+Connecting+language+and+vision+u) [our card](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/)
36. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv:2305.03726, 2023a. [find](https://scholar.google.com/scholar?q=Bo+Li%2C+Yuanhan+Zhang%2C+Liangyu+Chen%2C+Jinghao+Wang%2C+Jingkang+Yang%2C+and+Ziwei+Liu%2E+Otter%3A+A+multi%2Dmodal+model+with+in%2Dcontext+instruction+tuning%2E+arXiv%3A2305%2E03726%2C+2023a%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-otter-multi-modal-model-context/)
37. Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv:2307.16125, 2023b. [find](https://scholar.google.com/scholar?q=Bohao+Li%2C+Rui+Wang%2C+Guangzhi+Wang%2C+Yuying+Ge%2C+Yixiao+Ge%2C+and+Ying+Shan%2E+Seed%2Dbench%3A+Benchmarking+multimodal+llms+with+generative+comprehension%2E+arXiv%3A2307%2E16125%2C+2023b%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-seed-bench-benchmarking-multimodal-llms/)
38. Junnan Li, Ramprasaath R Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurIPS, 2021a. [find](https://scholar.google.com/scholar?q=Junnan+Li%2C+Ramprasaath+R+Selvaraju%2C+Akhilesh+Deepak+Gotmare%2C+Shafiq+Joty%2C+Caiming+Xiong%2C+and+Steven+Hoi%2E+Align+before+fuse%3A+Vision+and+language+representation+learning+with+momentum+distillation%2E+In+N)
39. Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022. [find](https://scholar.google.com/scholar?q=Junnan+Li%2C+Dongxu+Li%2C+Caiming+Xiong%2C+and+Steven+C%2E+H%2E+Hoi%2E+Blip%3A+Bootstrapping+language%2Dimage+pre%2Dtraining+for+unified+vision%2Dlanguage+understanding+and+generation%2E+In+ICML%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/)
40. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv:2301.12597, 2023c. [find](https://scholar.google.com/scholar?q=Junnan+Li%2C+Dongxu+Li%2C+Silvio+Savarese%2C+and+Steven+Hoi%2E+Blip%2D2%3A+Bootstrapping+language%2Dimage+pre%2Dtraining+with+frozen+image+encoders+and+large+language+models%2E+arXiv%3A2301%2E12597%2C+2023c%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/)
41. Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. UNIMO: towards unified-modal understanding and generation via cross-modal contrastive learning. In ACL, 2021b. [find](https://scholar.google.com/scholar?q=Wei+Li%2C+Can+Gao%2C+Guocheng+Niu%2C+Xinyan+Xiao%2C+Hao+Liu%2C+Jiachen+Liu%2C+Hua+Wu%2C+and+Haifeng+Wang%2E+UNIMO%3A+towards+unified%2Dmodal+understanding+and+generation+via+cross%2Dmodal+contrastive+learning%2E+In+ACL%2C+2021)
42. Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV, 2020. [find](https://scholar.google.com/scholar?q=Xiujun+Li%2C+Xi+Yin%2C+Chunyuan+Li%2C+Xiaowei+Hu%2C+Pengchuan+Zhang%2C+Lei+Zhang%2C+Lijuan+Wang%2C+Houdong+Hu%2C+Li+Dong%2C+Furu+Wei%2C+Yejin+Choi%2C+and+Jianfeng+Gao%2E+Oscar%3A+Object%2Dsemantics+aligned+pre%2Dtraining+for+visio) [our card](https://md.research.elamur.ai/free/en/papers/li-2020-oscar-object-semantics-aligned-pre/)
43. Junyang Lin, Rui Men, An Yang, Chang Zhou, Ming Ding, Yichang Zhang, Peng Wang, Ang Wang, Le Jiang, Xianyan Jia, et al. M6: A chinese multimodal pretrainer. In KDD, 2021. [find](https://scholar.google.com/scholar?q=Junyang+Lin%2C+Rui+Men%2C+An+Yang%2C+Chang+Zhou%2C+Ming+Ding%2C+Yichang+Zhang%2C+Peng+Wang%2C+Ang+Wang%2C+Le+Jiang%2C+Xianyan+Jia%2C+et+al%2E+M6%3A+A+chinese+multimodal+pretrainer%2E+In+KDD%2C+2021%2E)
44. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. [find](https://scholar.google.com/scholar?q=Tsung%2DYi+Lin%2C+Michael+Maire%2C+Serge+Belongie%2C+James+Hays%2C+Pietro+Perona%2C+Deva+Ramanan%2C+Piotr+Doll%C3%A1r%2C+and+C+Lawrence+Zitnick%2E+Microsoft+coco%3A+Common+objects+in+context%2E+In+ECCV%2C+2014%2E) [our card](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/)
45. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv:2304.08485, 2023. [find](https://scholar.google.com/scholar?q=Haotian+Liu%2C+Chunyuan+Li%2C+Qingyang+Wu%2C+and+Yong+Jae+Lee%2E+Visual+instruction+tuning%2E+arXiv%3A2304%2E08485%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/)
46. Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv:2206.08916, 2022a. — [arXiv:2206.08916](https://arxiv.org/abs/2206.08916) [find](https://scholar.google.com/scholar?q=Jiasen+Lu%2C+Christopher+Clark%2C+Rowan+Zellers%2C+Roozbeh+Mottaghi%2C+and+Aniruddha+Kembhavi%2E+Unified%2Dio%3A+A+unified+model+for+vision%2C+language%2C+and+multi%2Dmodal+tasks%2E+arXiv%3A2206%2E08916%2C+2022a%2E+%E2%80%94+%5BarXiv%3A2206%2E0)
47. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS, 2022b. [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Swaroop+Mishra%2C+Tanglin+Xia%2C+Liang+Qiu%2C+Kai%2DWei+Chang%2C+Song%2DChun+Zhu%2C+Oyvind+Tafjord%2C+Peter+Clark%2C+and+Ashwin+Kalyan%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+science+ques)
48. Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In CVPR, 2016. [find](https://scholar.google.com/scholar?q=Junhua+Mao%2C+Jonathan+Huang%2C+Alexander+Toshev%2C+Oana+Camburu%2C+Alan+L+Yuille%2C+and+Kevin+Murphy%2E+Generation+and+comprehension+of+unambiguous+object+descriptions%2E+In+CVPR%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/mao-2015-generation-comprehension-unambiguous-object-descriptions/)
49. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR, 2019. [find](https://scholar.google.com/scholar?q=Kenneth+Marino%2C+Mohammad+Rastegari%2C+Ali+Farhadi%2C+and+Roozbeh+Mottaghi%2E+Ok%2Dvqa%3A+A+visual+question+answering+benchmark+requiring+external+knowledge%2E+In+CVPR%2C+2019%2E)
50. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv:2203.10244, 2022. [find](https://scholar.google.com/scholar?q=Ahmed+Masry%2C+Do+Xuan+Long%2C+Jia+Qing+Tan%2C+Shafiq+Joty%2C+and+Enamul+Hoque%2E+Chartqa%3A+A+benchmark+for+question+answering+about+charts+with+visual+and+logical+reasoning%2E+arXiv%3A2203%2E10244%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/)
51. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In WACV, 2021. [find](https://scholar.google.com/scholar?q=Minesh+Mathew%2C+Dimosthenis+Karatzas%2C+and+CV+Jawahar%2E+Docvqa%3A+A+dataset+for+vqa+on+document+images%2E+In+WACV%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/mathew-2020-docvqa-dataset-vqa-document-images/)
52. Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019. [find](https://scholar.google.com/scholar?q=Anand+Mishra%2C+Shashank+Shekhar%2C+Ajeet+Kumar+Singh%2C+and+Anirban+Chakraborty%2E+Ocr%2Dvqa%3A+Visual+question+answering+by+reading+text+in+images%2E+In+ICDAR%2C+2019%2E)
53. Openai. Chatml documents. URL [https://github.com/openai/openai-python/blob/main/chatml.md](https://github.com/openai/openai-python/blob/main/chatml.md). OpenAI. Gpt-4 technical report, 2023. [↗](https://github.com/openai/openai-python/blob/main/chatml.md) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
54. Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned photographs. In NeurIPS, 2011. [find](https://scholar.google.com/scholar?q=Vicente+Ordonez%2C+Girish+Kulkarni%2C+and+Tamara+Berg%2E+Im2text%3A+Describing+images+using+1+million+captioned+photographs%2E+In+NeurIPS%2C+2011%2E)
55. Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv:2306.14824, 2023. [find](https://scholar.google.com/scholar?q=Zhiliang+Peng%2C+Wenhui+Wang%2C+Li+Dong%2C+Yaru+Hao%2C+Shaohan+Huang%2C+Shuming+Ma%2C+and+Furu+Wei%2E+Kosmos%2D2%3A+Grounding+multimodal+large+language+models+to+the+world%2E+arXiv%3A2306%2E14824%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/peng-2023-kosmos-2-grounding-multimodal-large/)
56. Qwen. Introducing qwen-7b: Open foundation and human-aligned models (of the state-of-the-arts), 2023. URL [https://github.com/QwenLM/Qwen-7B](https://github.com/QwenLM/Qwen-7B). [↗](https://github.com/QwenLM/Qwen-7B)
57. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jong+Wook+Kim%2C+Chris+Hallacy%2C+Aditya+Ramesh%2C+Gabriel+Goh%2C+Sandhini+Agarwal%2C+Girish+Sastry%2C+Amanda+Askell%2C+Pamela+Mishkin%2C+Jack+Clark%2C+et+al%2E+Learning+transferable+visual+models+from+natu) [our card](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/)
58. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv:2210.08402, 2022a. [find](https://scholar.google.com/scholar?q=Christoph+Schuhmann%2C+Romain+Beaumont%2C+Richard+Vencu%2C+Cade+Gordon%2C+Ross+Wightman%2C+Mehdi+Cherti%2C+Theo+Coombes%2C+Aarush+Katta%2C+Clayton+Mullis%2C+Mitchell+Wortsman%2C+et+al%2E+Laion%2D5b%3A+An+open+large%2Dscale+datas) [our card](https://md.research.elamur.ai/free/en/papers/schuhmann-2022-laion-5b-open-large-scale/)
59. Christoph Schuhmann, Andreas Köpf, Richard Vencu, Theo Coombes, and Romain Beaumont. Laion coco: 600m synthetic captions from laion2b-en. [https://laion.ai/blog/laion-coco/](https://laion.ai/blog/laion-coco/), 2022b. [↗](https://laion.ai/blog/laion-coco/)
60. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018. [find](https://scholar.google.com/scholar?q=Piyush+Sharma%2C+Nan+Ding%2C+Sebastian+Goodman%2C+and+Radu+Soricut%2E+Conceptual+captions%3A+A+cleaned%2C+hypernymed%2C+image+alt%2Dtext+dataset+for+automatic+image+captioning%2E+In+ACL%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/)
61. Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In ECCV, 2020. [find](https://scholar.google.com/scholar?q=Oleksii+Sidorov%2C+Ronghang+Hu%2C+Marcus+Rohrbach%2C+and+Amanpreet+Singh%2E+Textcaps%3A+a+dataset+for+image+captioning+with+reading+comprehension%2E+In+ECCV%2C+2020%2E)
62. Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In CVPR, 2022. [find](https://scholar.google.com/scholar?q=Amanpreet+Singh%2C+Ronghang+Hu%2C+Vedanuj+Goswami%2C+Guillaume+Couairon%2C+Wojciech+Galuba%2C+Marcus+Rohrbach%2C+and+Douwe+Kiela%2E+Flava%3A+A+foundational+language+and+vision+alignment+model%2E+In+CVPR%2C+2022%2E)
63. Artifex Software. Pymupdf, 2015. URL [https://github.com/pymupdf/PyMuPDF](https://github.com/pymupdf/PyMuPDF). [↗](https://github.com/pymupdf/PyMuPDF)
64. Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. In ICLR, 2019. [find](https://scholar.google.com/scholar?q=Weijie+Su%2C+Xizhou+Zhu%2C+Yue+Cao%2C+Bin+Li%2C+Lewei+Lu%2C+Furu+Wei%2C+and+Jifeng+Dai%2E+Vl%2Dbert%3A+Pre%2Dtraining+of+generic+visual%2Dlinguistic+representations%2E+In+ICLR%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/su-2019-vl-bert-pre-training-generic/)
65. Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv:2307.05222, 2023. — [arXiv:2307.05222](https://arxiv.org/abs/2307.05222) [find](https://scholar.google.com/scholar?q=Quan+Sun%2C+Qiying+Yu%2C+Yufeng+Cui%2C+Fan+Zhang%2C+Xiaosong+Zhang%2C+Yueze+Wang%2C+Hongcheng+Gao%2C+Jingjing+Liu%2C+Tiejun+Huang%2C+and+Xinlong+Wang%2E+Generative+pretraining+in+multimodality%2E+arXiv%3A2307%2E05222%2C+2023%2E+%E2%80%94+)
66. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In CVPR, 2015. [find](https://scholar.google.com/scholar?q=Ramakrishna+Vedantam%2C+C+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+Cider%3A+Consensus%2Dbased+image+description+evaluation%2E+In+CVPR%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/)
67. Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In ICML, 2022a. [find](https://scholar.google.com/scholar?q=Peng+Wang%2C+An+Yang%2C+Rui+Men%2C+Junyang+Lin%2C+Shuai+Bai%2C+Zhikang+Li%2C+Jianxin+Ma%2C+Chang+Zhou%2C+Jingren+Zhou%2C+and+Hongxia+Yang%2E+Ofa%3A+Unifying+architectures%2C+tasks%2C+and+modalities+through+a+simple+sequence%2Dto) [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-ofa-unifying-architectures-tasks-modalities/)
68. Peng Wang, Shijie Wang, Junyang Lin, Shuai Bai, Xiaohuan Zhou, Jingren Zhou, Xinggang Wang, and Chang Zhou. One-peace: Exploring one general representation model toward unlimited modalities. arXiv:2305.11172, 2023. — [arXiv:2305.11172](https://arxiv.org/abs/2305.11172) [find](https://scholar.google.com/scholar?q=Peng+Wang%2C+Shijie+Wang%2C+Junyang+Lin%2C+Shuai+Bai%2C+Xiaohuan+Zhou%2C+Jingren+Zhou%2C+Xinggang+Wang%2C+and+Chang+Zhou%2E+One%2Dpeace%3A+Exploring+one+general+representation+model+toward+unlimited+modalities%2E+arXiv%3A230)
69. Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv:2208.10442, 2022b. — [arXiv:2208.10442](https://arxiv.org/abs/2208.10442) [find](https://scholar.google.com/scholar?q=Wenhui+Wang%2C+Hangbo+Bao%2C+Li+Dong%2C+Johan+Bjorck%2C+Zhiliang+Peng%2C+Qiang+Liu%2C+Kriti+Aggarwal%2C+Owais+Khan+Mohammed%2C+Saksham+Singhal%2C+Subhojit+Som%2C+et+al%2E+Image+as+a+foreign+language%3A+Beit+pretraining+for+a)
70. An Yang, Junshu Pan, Junyang Lin, Rui Men, Yichang Zhang, Jingren Zhou, and Chang Zhou. Chinese clip: Contrastive vision-language pretraining in chinese. arXiv:2211.01335, 2022a. — [arXiv:2211.01335](https://arxiv.org/abs/2211.01335) [find](https://scholar.google.com/scholar?q=An+Yang%2C+Junshu+Pan%2C+Junyang+Lin%2C+Rui+Men%2C+Yichang+Zhang%2C+Jingren+Zhou%2C+and+Chang+Zhou%2E+Chinese+clip%3A+Contrastive+vision%2Dlanguage+pretraining+in+chinese%2E+arXiv%3A2211%2E01335%2C+2022a%2E+%E2%80%94+%5BarXiv%3A2211%2E01335%5D%28)
71. Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In AAAI, 2022b. [find](https://scholar.google.com/scholar?q=Zhengyuan+Yang%2C+Zhe+Gan%2C+Jianfeng+Wang%2C+Xiaowei+Hu%2C+Yumao+Lu%2C+Zicheng+Liu%2C+and+Lijuan+Wang%2E+An+empirical+study+of+gpt%2D3+for+few%2Dshot+knowledge%2Dbased+vqa%2E+In+AAAI%2C+2022b%2E)
72. Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, et al. mplug-docowl: Modularized multimodal large language model for document understanding. arXiv:2307.02499, 2023a. — [arXiv:2307.02499](https://arxiv.org/abs/2307.02499) [find](https://scholar.google.com/scholar?q=Jiabo+Ye%2C+Anwen+Hu%2C+Haiyang+Xu%2C+Qinghao+Ye%2C+Ming+Yan%2C+Yuhao+Dan%2C+Chenlin+Zhao%2C+Guohai+Xu%2C+Chenliang+Li%2C+Junfeng+Tian%2C+et+al%2E+mplug%2Ddocowl%3A+Modularized+multimodal+large+language+model+for+document+unde)
73. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv:2304.14178, 2023b. [find](https://scholar.google.com/scholar?q=Qinghao+Ye%2C+Haiyang+Xu%2C+Guohai+Xu%2C+Jiabo+Ye%2C+Ming+Yan%2C+Yiyang+Zhou%2C+Junyang+Wang%2C+Anwen+Hu%2C+Pengcheng+Shi%2C+Yaya+Shi%2C+et+al%2E+mplug%2Dowl%3A+Modularization+empowers+large+language+models+with+multimodality%2E) [our card](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/)
74. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. In ACL, 2014. [find](https://scholar.google.com/scholar?q=Peter+Young%2C+Alice+Lai%2C+Micah+Hodosh%2C+and+Julia+Hockenmaier%2E+From+image+descriptions+to+visual+denotations%3A+New+similarity+metrics+for+semantic+inference+over+event+descriptions%2E+In+ACL%2C+2014%2E)
75. Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv:2205.01917, 2022. [find](https://scholar.google.com/scholar?q=Jiahui+Yu%2C+Zirui+Wang%2C+Vijay+Vasudevan%2C+Legg+Yeung%2C+Mojtaba+Seyedhosseini%2C+and+Yonghui+Wu%2E+Coca%3A+Contrastive+captioners+are+image%2Dtext+foundation+models%2E+arXiv%3A2205%2E01917%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/yu-2022-coca-contrastive-captioners-image-text/)
76. Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel C. F. Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang. Fl… — [arXiv:2111.11432](https://arxiv.org/abs/2111.11432) [find](https://scholar.google.com/scholar?q=Lu+Yuan%2C+Dongdong+Chen%2C+Yi%2DLing+Chen%2C+Noel+C%2E+F%2E+Codella%2C+Xiyang+Dai%2C+Jianfeng+Gao%2C+Houdong+Hu%2C+Xuedong+Huang%2C+Boxin+Li%2C+Chunyuan+Li%2C+Ce+Liu%2C+Mengchen+Liu%2C+Zicheng+Liu%2C+Yumao+Lu%2C+Yu+Shi%2C+Lijuan+Wang%2C+)
77. Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. arXiv:2111.08276, 2021. — [arXiv:2111.08276](https://arxiv.org/abs/2111.08276) [find](https://scholar.google.com/scholar?q=Yan+Zeng%2C+Xinsong+Zhang%2C+and+Hang+Li%2E+Multi%2Dgrained+vision+language+pre%2Dtraining%3A+Aligning+texts+with+visual+concepts%2E+arXiv%3A2111%2E08276%2C+2021%2E+%E2%80%94+%5BarXiv%3A2111%2E08276%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2111%2E08276%29)
78. Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In CVPR, 2022. [find](https://scholar.google.com/scholar?q=Xiaohua+Zhai%2C+Xiao+Wang%2C+Basil+Mustafa%2C+Andreas+Steiner%2C+Daniel+Keysers%2C+Alexander+Kolesnikov%2C+and+Lucas+Beyer%2E+Lit%3A+Zero%2Dshot+transfer+with+locked%2Dimage+text+tuning%2E+In+CVPR%2C+2022%2E)
79. Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv:2306.02858, 2023. [find](https://scholar.google.com/scholar?q=Hang+Zhang%2C+Xin+Li%2C+and+Lidong+Bing%2E+Video%2Dllama%3A+An+instruction%2Dtuned+audio%2Dvisual+language+model+for+video+understanding%2E+arXiv%3A2306%2E02858%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2023-video-llama-instruction-tuned-audio/)
80. Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In CVPR, 2021. [find](https://scholar.google.com/scholar?q=Pengchuan+Zhang%2C+Xiujun+Li%2C+Xiaowei+Hu%2C+Jianwei+Yang%2C+Lei+Zhang%2C+Lijuan+Wang%2C+Yejin+Choi%2C+and+Jianfeng+Gao%2E+Vinvl%3A+Revisiting+visual+representations+in+vision%2Dlanguage+models%2E+In+CVPR%2C+2021%2E)
81. Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, and Bingyi Kang. Bubogpt: Enabling visual grounding in multi-modal llms. arXiv:2307.08581, 2023. — [arXiv:2307.08581](https://arxiv.org/abs/2307.08581) [find](https://scholar.google.com/scholar?q=Yang+Zhao%2C+Zhijie+Lin%2C+Daquan+Zhou%2C+Zilong+Huang%2C+Jiashi+Feng%2C+and+Bingyi+Kang%2E+Bubogpt%3A+Enabling+visual+grounding+in+multi%2Dmodal+llms%2E+arXiv%3A2307%2E08581%2C+2023%2E+%E2%80%94+%5BarXiv%3A2307%2E08581%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
82. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv:2304.10592, 2023. [find](https://scholar.google.com/scholar?q=Deyao+Zhu%2C+Jun+Chen%2C+Xiaoqian+Shen%2C+Xiang+Li%2C+and+Mohamed+Elhoseiny%2E+Minigpt%2D4%3A+Enhancing+vision%2Dlanguage+understanding+with+advanced+large+language+models%2E+arXiv%3A2304%2E10592%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/)
83. Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. In CVPR, 2022. [find](https://scholar.google.com/scholar?q=Xizhou+Zhu%2C+Jinguo+Zhu%2C+Hao+Li%2C+Xiaoshi+Wu%2C+Hongsheng+Li%2C+Xiaohua+Wang%2C+and+Jifeng+Dai%2E+Uni%2Dperceiver%3A+Pre%2Dtraining+unified+architecture+for+generic+perception+for+zero%2Dshot+and+few%2Dshot+tasks%2E+In+CVP)