---
title: "InternLM-XComposer — paper by Zhang et al. (2023)"
description: "Most open vision-language large models (VLLMs) produce only text. 105 references, cited by 54 papers here."
authors: ["Pan Zhang", "Xiaoyi Dong", "Bin Wang", "Yuhang Cao", "Chao Xu", "Linke Ouyang", "Zhiyuan Zhao", "Haodong Duan", "Songyang Zhang", "Shuangrui Ding", "Wenwei Zhang", "Hang Yan", "Xinyue Zhang", "Wei Li", "Jingwen Li", "Kai Chen", "Conghui He", "Xingcheng Zhang", "Yu Qiao", "Dahua Lin", "Jiaqi Wang"]
year: 2023
venue: "arXiv:2309.15112"
source: "https://arxiv.org/abs/2309.15112"
doi: "10.48550/arxiv.2309.15112"
arxiv: "2309.15112"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2023-internlm-xcomposer-vision-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2023-internlm-xcomposer-vision-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition

# InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition

**Authors:** [Pan Zhang](https://md.research.elamur.ai/free/en/authors/pan-zhang/), [Xiaoyi Dong](https://md.research.elamur.ai/free/en/authors/xiaoyi-dong/), [Bin Wang](https://md.research.elamur.ai/free/en/authors/bin-wang/), [Yuhang Cao](https://md.research.elamur.ai/free/en/authors/yuhang-cao/), Chao Xu, [Linke Ouyang](https://md.research.elamur.ai/free/en/authors/linke-ouyang/), Zhiyuan Zhao, [Haodong Duan](https://md.research.elamur.ai/free/en/authors/haodong-duan/), [Songyang Zhang](https://md.research.elamur.ai/free/en/authors/songyang-zhang/), Shuangrui Ding, [Wenwei Zhang](https://md.research.elamur.ai/free/en/authors/wenwei-zhang/), [Hang Yan](https://md.research.elamur.ai/free/en/authors/hang-yan/), Xinyue Zhang, [Wei Li](https://md.research.elamur.ai/free/en/authors/wei-li/), Jingwen Li, [Kai Chen](https://md.research.elamur.ai/free/en/authors/kai-chen/), [Conghui He](https://md.research.elamur.ai/free/en/authors/conghui-he/), Xingcheng Zhang, [Yu Qiao](https://md.research.elamur.ai/free/en/authors/yu-qiao/), [Dahua Lin](https://md.research.elamur.ai/free/en/authors/dahua-lin/) and [Jiaqi Wang](https://md.research.elamur.ai/free/en/authors/jiaqi-wang/) · **Year:** 2023 · **Venue:** arXiv:2309.15112 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2309.15112) · [PDF](https://arxiv.org/pdf/2309.15112) · [DOI 10.48550/arxiv.2309.15112](https://doi.org/10.48550/arxiv.2309.15112) · [arXiv 2309.15112](https://arxiv.org/abs/2309.15112) · [OpenAlex](https://openalex.org/W4387156669)

**Citations:** 31 citations (OpenAlex, 2026-10-14) · [54](https://md.research.elamur.ai/free/en/papers/zhang-2023-internlm-xcomposer-vision-language/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Most open vision-language large models (VLLMs) produce only text. InternLM-XComposer, from Shanghai Artificial Intelligence Laboratory, extends this to writing long articles with illustrations placed inside the text, while also improving image understanding. The model pairs a visual encoder and a Perceive Sampler with the InternLM language model. It is first pretrained to align vision and language, then fine-tuned on multiple tasks using LoRA.

To widen concept coverage, the authors gathered a multilingual web dataset covering more than 11 million semantic concepts and combined it with open image-text corpora. Training mixes text-only and image-text data, mostly in English and Chinese, so the base LLM keeps its original skills.

When composing, the model drafts the article and decides where pictures belong. It writes a description for each slot, collects candidates from a web-crawled image database using CLIP, and then picks the best match itself. The model reports state-of-the-art results on major English and Chinese benchmarks. No standard metric exists for illustrated articles, so the authors built a human plus GPT4-V evaluation, where the model scored competitively against GPT4-V and GPT3.5.

## Key points
- **Benchmarks with reported state-of-the-art results:**
  - English: MME, MMBench, Seed-Bench, QBench and Tiny LVLM.
  - Chinese: MMBench-CN and CCBench (Chinese Cultural Benchmark).
- **Composition evaluation protocol:**
  - Eight dimensions: four for text, three for images and one for overall preference.
  - 20 generated articles, rated by 10 human experts and by GPT4-V.
  - Baselines: GPT4-V, GPT3.5, InternLM-20B and Qwen-14B.
  - The text-only baselines used top-1 CLIP retrieval for their images.
  - GPT4-V led on all text-related dimensions.
- **Image retrieval:** CLIP shortlists the candidate images, and the model chooses among them. Users can plug in their own image repository.
- **Ablations:**
  - The Perceive Sampler and the LoRA adapters on attention and FFN layers are important for fine-tuning.
  - Swapping InternLM-7B for LLaMA2-7B gives similar English scores but noticeably weaker Chinese results.
- **Availability:** the models are released at github.com/InternLM/InternLM-XComposer.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 44 works · 10,162 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 269 works

## Cite

Cite as: Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Haodong Duan, Songyang Zhang, Shuangrui Ding, Wenwei Zhang, Hang Yan, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin and Jiaqi Wang. 2023. InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition. arXiv:2309.15112.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhang2023internlm,
  title = {{InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition}},
  author = {Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Haodong Duan, Songyang Zhang, Shuangrui Ding, Wenwei Zhang, Hang Yan, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin and Jiaqi Wang},
  year = {2023},
  doi = {10.48550/arxiv.2309.15112},
  eprint = {2309.15112},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.15112}
}
```

- [← previous Universal Self-Consistency for Large Language Model Generation](https://md.research.elamur.ai/free/en/papers/chen-2023-universal-self-consistency-large-language/)
- [next → Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM](https://md.research.elamur.ai/free/en/papers/bawden-2023-investigating-translation-performance-large-multilingual/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

105 references, 47 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock…](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
2. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In International Conference on Computer Vision (ICCV), 2015. 4 [find](https://scholar.google.com/scholar?q=Stanislaw+Antol%2C+Aishwarya+Agrawal%2C+Jiasen+Lu%2C+Margaret+Mitchell%2C+Dhruv+Batra%2C+C%2E+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+Vqa%3A+Visual+question+answering%2E+In+International+Conference+on+Computer+Vision+%28ICC)
3. [Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source framework for training large…](https://md.research.elamur.ai/free/en/papers/awadalla-2023-openflamingo-open-source-framework-training/)
4. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv.org, 2023. 3, 6, 7, 17 [find](https://scholar.google.com/scholar?q=Jinze+Bai%2C+Shuai+Bai%2C+Shusheng+Yang%2C+Shijie+Wang%2C+Sinan+Tan%2C+Peng+Wang%2C+Junyang+Lin%2C+Chang+Zhou%2C+and+Jingren+Zhou%2E+Qwen%2Dvl%3A+A+frontier+large+vision%2Dlanguage+model+with+versatile+abilities%2E+arXiv%2Eorg%2C+)
5. Baichuan. [Baichuan 2: Open large-scale language models](https://md.research.elamur.ai/free/en/papers/yang-2023-baichuan-2-open-large-scale/). arXiv.org, 2023. 2, 3
6. James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and Aditya Ramesh. Improving image generation with better captions. 2 [find](https://scholar.google.com/scholar?q=James+Betker%2C+Gabriel+Goh%2C+Li+Jing%2C+Tim+Brooks%2C+Jianfeng+Wang%2C+Linjie+Li%2C+Long+Ouyang%2C+Juntang+Zhuang%2C+Joyce+Lee%2C+Yufei+Guo%2C+Wesam+Manassra%2C+Prafulla+Dhariwal%2C+Casey+Chu%2C+Yunxin+Jiao%2C+and+Aditya+Rames)
7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in Neural Information Processing Systems (NeurIPS), 33:1877– 1901, 2020. 2, 3
8. Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558–3568, 2021. 2, 4 [find](https://scholar.google.com/scholar?q=Soravit+Changpinyo%2C+Piyush+Sharma%2C+Nan+Ding%2C+and+Radu+Soricut%2E+Conceptual+12m%3A+Pushing+web%2Dscale+image%2Dtext+pre%2Dtraining+to+recognize+long%2Dtail+visual+concepts%2E+In+Proceedings+of+the+IEEE%2FCVF+Conferen)
9. Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechu Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. [Minigpt-v2: large language model as a unified interface for vision-language multi-task learning](https://md.research.elamur.ai/free/en/papers/chen-2023-minigpt-v2-large-language-model/). arXiv preprint arXiv:2310.09478, 2023. 3
10. [Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv.org, 2023. 3, 6, 7](https://md.research.elamur.ai/free/en/papers/chen-2023-shikra-unleashing-multimodal-llm-s/)
11. Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, Siamak Shakeri, Mostafa Dehghani, Daniel Salz, Mario Lucic, Michael Tschannen, Arsha Nagrani, Hexiang Hu, Mandar Joshi, Bo Pang, Ceslee Montgomery, Paul… [find](https://scholar.google.com/scholar?q=Xi+Chen%2C+Josip+Djolonga%2C+Piotr+Padlewski%2C+Basil+Mustafa%2C+Soravit+Changpinyo%2C+Jialin+Wu%2C+Carlos+Riquelme+Ruiz%2C+Sebastian+Goodman%2C+Xiao+Wang%2C+Yi+Tay%2C+Siamak+Shakeri%2C+Mostafa+Dehghani%2C+Daniel+Salz%2C+Mario)
12. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. [Microsoft coco captions: Data collection and evaluation server](https://md.research.elamur.ai/free/en/papers/chen-2015-microsoft-coco-captions-data-collection/), 2015. 4
13. Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, Daniel Salz, Xi Xiong, Daniel Vlasic, Filip Pavetic, Keran Rong, Tianli Yu, Daniel Keysers, Xiaohua Zhai, and Radu Soricut. Pali-3 vision lan… [find](https://scholar.google.com/scholar?q=Xi+Chen%2C+Xiao+Wang%2C+Lucas+Beyer%2C+Alexander+Kolesnikov%2C+Jialin+Wu%2C+Paul+Voigtlaender%2C+Basil+Mustafa%2C+Sebastian+Goodman%2C+Ibrahim+Alabdulmohsin%2C+Piotr+Padlewski%2C+Daniel+Salz%2C+Xi+Xiong%2C+Daniel+Vlasic%2C+Fil)
14. [Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo,…](https://md.research.elamur.ai/free/en/papers/chen-2022-pali-jointly-scaled-multilingual-language/)
15. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. 2, 3, 4 [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+and+Eric+P%2E+Xing%2E+Vicuna%3A+An+open%2Dsource+chatbot+i)
16. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv.org, 2022. 2, 3
17. Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In CVPR, Jun 2019. 2 [find](https://scholar.google.com/scholar?q=Yin+Cui%2C+Menglin+Jia%2C+Tsung%2DYi+Lin%2C+Yang+Song%2C+and+Serge+Belongie%2E+Class%2Dbalanced+loss+based+on+effective+number+of+samples%2E+In+CVPR%2C+Jun+2019%2E+2)
18. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. [Instructblip: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/), 2023. 2, 3, 6, 7, 17
19. Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 326–335, 2017. 4 [find](https://scholar.google.com/scholar?q=Abhishek+Das%2C+Satwik+Kottur%2C+Khushi+Gupta%2C+Avi+Singh%2C+Deshraj+Yadav%2C+Jose%CC%81+MF+Moura%2C+Devi+Parikh%2C+and+Dhruv+Batra%2E+Visual+dialog%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+r)
20. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255, 2009. 3 [find](https://scholar.google.com/scholar?q=Jia+Deng%2C+Wei+Dong%2C+Richard+Socher%2C+Li%2DJia+Li%2C+Kai+Li%2C+and+Li+Fei%2DFei%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Rec)
21. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv.org, 2018. 3
22. Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.11499, 2023. 3 — [arXiv:2309.11499](https://arxiv.org/abs/2309.11499) [find](https://scholar.google.com/scholar?q=Runpei+Dong%2C+Chunrui+Han%2C+Yuang+Peng%2C+Zekun+Qi%2C+Zheng+Ge%2C+Jinrong+Yang%2C+Liang+Zhao%2C+Jianjian+Sun%2C+Hongyu+Zhou%2C+Haoran+Wei%2C+Xiangwen+Kong%2C+Xiangyu+Zhang%2C+Kaisheng+Ma%2C+and+Li+Yi%2E+Dreamllm%3A+Synergistic+m)
23. Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zen… [find](https://scholar.google.com/scholar?q=Danny+Driess%2C+Fei+Xia%2C+Mehdi+S%2E+M%2E+Sajjadi%2C+Corey+Lynch%2C+Aakanksha+Chowdhery%2C+Brian+Ichter%2C+Ayzaan+Wahid%2C+Jonathan+Tompson%2C+Quan+Vuong%2C+Tianhe+Yu%2C+Wenlong+Huang%2C+Yevgen+Chebotar%2C+Pierre+Sermanet%2C+Dani)
24. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. [Glm: General language model pretraining with autoregressive blank infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335, 2022…
25. Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19358–19369,… [find](https://scholar.google.com/scholar?q=Yuxin+Fang%2C+Wen+Wang%2C+Binhui+Xie%2C+Quan+Sun%2C+Ledell+Wu%2C+Xinggang+Wang%2C+Tiejun+Huang%2C+Xinlong+Wang%2C+and+Yue+Cao%2E+Eva%3A+Exploring+the+limits+of+masked+visual+representation+learning+at+scale%2E+In+Proceedin)
26. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. [Mme: A comprehensive evaluation benchmark for multimodal large language models](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/). arXiv preprint arXiv:2306.13394, 2023. 2, 6
27. Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, W. Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Jiao Qiao. Llama-adapter v2: Parameter-efficient visual instruction model. ArXiv, abs/2304.15010, 2023. 6, 7, 17 [find](https://scholar.google.com/scholar?q=Peng+Gao%2C+Jiaming+Han%2C+Renrui+Zhang%2C+Ziyi+Lin%2C+Shijie+Geng%2C+Aojun+Zhou%2C+W%2E+Zhang%2C+Pan+Lu%2C+Conghui+He%2C+Xiangyu+Yue%2C+Hongsheng+Li%2C+and+Yu+Jiao+Qiao%2E+Llama%2Dadapter+v2%3A+Parameter%2Defficient+visual+instruct)
28. Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model. 3 [find](https://scholar.google.com/scholar?q=Yuying+Ge%2C+Yixiao+Ge%2C+Ziyun+Zeng%2C+Xintao+Wang%2C+and+Ying+Shan%2E+Planting+a+seed+of+vision+in+large+language+model%2E+3)
29. Jiaxi Gu, Xiaojun Meng, Guansong Lu, Lu Hou, Niu Minzhe, Xiaodan Liang, Lewei Yao, Runhui Huang, Wei Zhang, Xin Jiang, et al. Wukong: A 100 million large-scale chinese cross-modal pre-training benchmark. Advances in Neural Information Processing Systems, 35:26418–26431, 2022. 4 [find](https://scholar.google.com/scholar?q=Jiaxi+Gu%2C+Xiaojun+Meng%2C+Guansong+Lu%2C+Lu+Hou%2C+Niu+Minzhe%2C+Xiaodan+Liang%2C+Lewei+Yao%2C+Runhui+Huang%2C+Wei+Zhang%2C+Xin+Jiang%2C+et+al%2E+Wukong%3A+A+100+million+large%2Dscale+chinese+cross%2Dmodal+pre%2Dtraining+benchma)
30. Conghui He, Zhenjiang Jin, Chaoxi Xu, Jiantao Qiu, Bin Wang, Wei Li, Hang Yan, Jiaqi Wang, and Da Lin. Wanjuan: A comprehensive multimodal dataset for advancing english and chinese large models. ArXiv, abs/2308.10755, 2023. 4, 14 — [arXiv:2308.10755](https://arxiv.org/abs/2308.10755) [find](https://scholar.google.com/scholar?q=Conghui+He%2C+Zhenjiang+Jin%2C+Chaoxi+Xu%2C+Jiantao+Qiu%2C+Bin+Wang%2C+Wei+Li%2C+Hang+Yan%2C+Jiaqi+Wang%2C+and+Da+Lin%2E+Wanjuan%3A+A+comprehensive+multimodal+dataset+for+advancing+english+and+chinese+large+models%2E+ArXiv)
31. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations, 2022. 5, 14
32. W. Hu, Y. Xu, Y. Li, W. Li, Z. Chen, and Z. Tu. Bliva: A simple multimodal llm for better handling of text-rich visual questions. ArXiv, abs/2308.09936, 2023. 6 — [arXiv:2308.09936](https://arxiv.org/abs/2308.09936) [find](https://scholar.google.com/scholar?q=W%2E+Hu%2C+Y%2E+Xu%2C+Y%2E+Li%2C+W%2E+Li%2C+Z%2E+Chen%2C+and+Z%2E+Tu%2E+Bliva%3A+A+simple+multimodal+llm+for+better+handling+of+text%2Drich+visual+questions%2E+ArXiv%2C+abs%2F2308%2E09936%2C+2023%2E+6+%E2%80%94+%5BarXiv%3A2308%2E09936%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
33. Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A Ross, and Alireza Fathi. Reveal: Retrieval-augmented visual-language pre-training with multi-source multimodal knowledge memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern R… [find](https://scholar.google.com/scholar?q=Ziniu+Hu%2C+Ahmet+Iscen%2C+Chen+Sun%2C+Zirui+Wang%2C+Kai%2DWei+Chang%2C+Yizhou+Sun%2C+Cordelia+Schmid%2C+David+A+Ross%2C+and+Alireza+Fathi%2E+Reveal%3A+Retrieval%2Daugmented+visual%2Dlanguage+pre%2Dtraining+with+multi%2Dsource+mul)
34. Drew A Hudson and Christopher D Manning. [Gqa: A new dataset for real-world visual reasoning and compositional question answering](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/). Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 4
35. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. [Scaling up visual and vision-language representation learning with noisy text supervision](https://md.research.elamur.ai/free/en/papers/jia-2021-scaling-up-visual-vision-language/). In Proceedings of the International Conference on Machine learning (ICML), pages 4904–49…
36. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroi… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+Le%CC%81lio+Renard+La)
37. Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. 2023. 3 [find](https://scholar.google.com/scholar?q=Jing+Yu+Koh%2C+Ruslan+Salakhutdinov%2C+and+Daniel+Fried%2E+Grounding+language+models+to+images+for+multimodal+inputs+and+outputs%2E+2023%2E+3)
38. [Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Op…](https://md.research.elamur.ai/free/en/papers/kopf-2023-openassistant-conversations-democratizing-large-language/)
39. Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. [Obelics: An open web-scale filtered dataset of interleaved image-text documents](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2023-obelics-open-web-scale-filtered/), 2023. 7
40. Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. [Seed-bench: Benchmarking multimodal llms with generative comprehension](https://md.research.elamur.ai/free/en/papers/li-2023-seed-bench-benchmarking-multimodal-llms/), 2023. 2, 6
41. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. [Otter: A multi-modal model with in-context instruction tuning](https://md.research.elamur.ai/free/en/papers/li-2023-otter-multi-modal-model-context/). arXiv.org, 2023. 3, 6, 7, 17
42. Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. [Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). ArXiv, abs/2301.12597, 2023. 2, 3, 4, 17
43. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. [Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/). In Proceedings of the International Conference on Machine learning (ICML), pages 12888–12900. PMLR, 2022. 3, 7
44. Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Hanwang Zhang, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, and Yueting Zhuang. Empowering vision-language models to follow interleaved vision-language instructions. ArXiv, abs/2308.04152, 2023. 6 — [arXiv:2308.04152](https://arxiv.org/abs/2308.04152) [find](https://scholar.google.com/scholar?q=Juncheng+Li%2C+Kaihang+Pan%2C+Zhiqi+Ge%2C+Minghe+Gao%2C+Hanwang+Zhang%2C+Wei+Ji%2C+Wenqiao+Zhang%2C+Tat%2DSeng+Chua%2C+Siliang+Tang%2C+and+Yueting+Zhuang%2E+Empowering+vision%2Dlanguage+models+to+follow+interleaved+vision%2Dla)
45. Liunian Harold Li*, Pengchuan Zhang*, Haotian Zhang*, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (… [find](https://scholar.google.com/scholar?q=Liunian+Harold+Li%2A%2C+Pengchuan+Zhang%2A%2C+Haotian+Zhang%2A%2C+Jianwei+Yang%2C+Chunyuan+Li%2C+Yiwu+Zhong%2C+Lijuan+Wang%2C+Lu+Yuan%2C+Lei+Zhang%2C+Jenq%2DNeng+Hwang%2C+Kai%2DWei+Chang%2C+and+Jianfeng+Gao%2E+Grounded+language%2Dimage+)
46. Yunxin Li, Baotian Hu, Xinyu Chen, Lin Ma, Yong Xu, and Min Zhang. Lmeye: An interactive perception network for large language models, 2023. 6 [find](https://scholar.google.com/scholar?q=Yunxin+Li%2C+Baotian+Hu%2C+Xinyu+Chen%2C+Lin+Ma%2C+Yong+Xu%2C+and+Min+Zhang%2E+Lmeye%3A+An+interactive+perception+network+for+large+language+models%2C+2023%2E+6)
47. Fangyu Liu, Guy Edward Toh Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 2023. 4 [find](https://scholar.google.com/scholar?q=Fangyu+Liu%2C+Guy+Edward+Toh+Emerson%2C+and+Nigel+Collier%2E+Visual+spatial+reasoning%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+2023%2E+4)
48. Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.[14565, 2023](https://md.research.elamur.ai/free/en/papers/liu-2023-mitigating-hallucination-large-multi-modal/). 4
49. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/). arXiv preprint arXiv:2310.03744, 2023. 6, 7, 17
50. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). arXiv.org, 2023. 2, 3, 4, 6, 7, 17
51. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. [Grounding dino: Marrying dino with grounded pre-training for open-set object detection](https://md.research.elamur.ai/free/en/papers/liu-2023-grounding-dino-marrying-dino-grounded/). arXiv.org, 2023. 3
52. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhnag, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. [Mmbench: Is your multi-modal model an all-around player?](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/) arXiv:2307.06281, 2023. 2, 6, 7
53. Yulong Liu, Guibo Zhu, Bin Zhu, Qi Song, Guojing Ge, Haoran Chen, GuanHui Qiao, Ru Peng, Lingxiang Wu, and Jinqiao Wang. Taisu: A 166m large-scale high-quality dataset for chinese vision-language pre-training. Advances in Neural Information Processing Systems, 35:16705– 16717, 2022. 4 [find](https://scholar.google.com/scholar?q=Yulong+Liu%2C+Guibo+Zhu%2C+Bin+Zhu%2C+Qi+Song%2C+Guojing+Ge%2C+Haoran+Chen%2C+GuanHui+Qiao%2C+Ru+Peng%2C+Lingxiang+Wu%2C+and+Jinqiao+Wang%2E+Taisu%3A+A+166m+large%2Dscale+high%2Dquality+dataset+for+chinese+vision%2Dlanguage+pre%2D)
54. Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large-scale long-tailed recognition in an open world. In CVPR, Jun 2019. 2 [find](https://scholar.google.com/scholar?q=Ziwei+Liu%2C+Zhongqi+Miao%2C+Xiaohang+Zhan%2C+Jiayun+Wang%2C+Boqing+Gong%2C+and+Stella+X%2E+Yu%2E+Large%2Dscale+long%2Dtailed+recognition+in+an+open+world%2E+In+CVPR%2C+Jun+2019%2E+2)
55. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022. 4 [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Swaroop+Mishra%2C+Tanglin+Xia%2C+Liang+Qiu%2C+Kai%2DWei+Chang%2C+Song%2DChun+Zhu%2C+Oyvind+Tafjord%2C+Peter+Clark%2C+and+Ashwin+Kalyan%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+science+ques)
56. Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2110.13214, 2021. 4 [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Liang+Qiu%2C+Jiaqi+Chen%2C+Tony+Xia%2C+Yizhou+Zhao%2C+Wei+Zhang%2C+Zhou+Yu%2C+Xiaodan+Liang%2C+and+Song%2DChun+Zhu%2E+Iconqa%3A+A+new+benchmark+for+abstract+diagram+understanding+and+visual+language+reasoning%2E+ar)
57. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019. 4 [find](https://scholar.google.com/scholar?q=Kenneth+Marino%2C+Mohammad+Rastegari%2C+Ali+Farhadi%2C+and+Roozbeh+Mottaghi%2E+Ok%2Dvqa%3A+A+visual+question+answering+benchmark+requiring+external+knowledge%2E+In+Proceedings+of+the+IEEE%2Fcvf+conference+on+computer)
58. Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019. 4 [find](https://scholar.google.com/scholar?q=Anand+Mishra%2C+Shashank+Shekhar%2C+Ajeet+Kumar+Singh%2C+and+Anirban+Chakraborty%2E+Ocr%2Dvqa%3A+Visual+question+answering+by+reading+text+in+images%2E+In+ICDAR%2C+2019%2E+4)
59. MEJ Newman. Power laws, pareto distributions and zipf’s law. Contemporary Physics, page 323–351, Sep 2005. 2 [find](https://scholar.google.com/scholar?q=MEJ+Newman%2E+Power+laws%2C+pareto+distributions+and+zipf%E2%80%99s+law%2E+Contemporary+Physics%2C+page+323%E2%80%93351%2C+Sep+2005%2E+2)
60. OpenAI. Chatgpt. [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt), 2022. 2, 3, 8 [↗](https://openai.com/blog/chatgpt)
61. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023. 2, 8
62. Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. Im2text: Describing images using 1 million captioned photographs. In Neural Information Processing Systems (NIPS), 2011. 2, 4 [find](https://scholar.google.com/scholar?q=Vicente+Ordonez%2C+Girish+Kulkarni%2C+and+Tamara+L%2E+Berg%2E+Im2text%3A+Describing+images+using+1+million+captioned+photographs%2E+In+Neural+Information+Processing+Systems+%28NIPS%29%2C+2011%2E+2%2C+4)
63. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems (NeurIPS), 35:27730–27744, 2022…
64. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023. 3 [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Alessandro+Cappelli%2C+Hamza+Alobeidli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+The+refinedweb+dataset+for+falcon+)
65. Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. [Kosmos-2: Grounding multimodal large language models to the world](https://md.research.elamur.ai/free/en/papers/peng-2023-kosmos-2-grounding-multimodal-large/). arXiv.org, 2023. 3, 7
66. Qwen. Introducing qwen-7b: Open foundation and human-aligned models (of the state-of-the-arts), 2023. 2, 3, 8 [find](https://scholar.google.com/scholar?q=Qwen%2E+Introducing+qwen%2D7b%3A+Open+foundation+and+human%2Daligned+models+%28of+the+state%2Dof%2Dthe%2Darts%29%2C+2023%2E+2%2C+3%2C+8)
67. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In Proceedings of the International Conference on Machine learning (ICML),…
68. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018. 3 [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+Ilya+Sutskever%2C+et+al%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+2018%2E+3)
69. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research (JMLR), 21(1):5485–5551, 2020. 2, 3
70. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. [Hierarchical text-conditional image generation with clip latents](https://md.research.elamur.ai/free/en/papers/ramesh-2022-hierarchical-text-conditional-image-generation/). 2
71. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. [Zero-shot text-to-image generation](https://md.research.elamur.ai/free/en/papers/ramesh-2021-zero-shot-text-image-generation/). ICML, Jul 2021.
72. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684–10695, June 2022. [find](https://scholar.google.com/scholar?q=Robin+Rombach%2C+Andreas+Blattmann%2C+Dominik+Lorenz%2C+Patrick+Esser%2C+and+Bjo%CC%88rn+Ommer%2E+High%2Dresolution+image+synthesis+with+latent+diffusion+models%2E+In+CVPR%2C+pages+10684%E2%80%9310695%2C+June+2022%2E)
73. [Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar, Seyed Ghasemipour, Burcu Karagol, SSara Mahdavi, RaphaGon-tijo Lopes, Tim Salimans, Jonathan Ho, DavidJ Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language underst…](https://md.research.elamur.ai/free/en/papers/saharia-2022-photorealistic-text-image-diffusion-models/)
74. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. [Laion-5b: An open large-scale dataset for training next generation image-text models](https://md.research.elamur.ai/free/en/papers/schuhmann-2022-laion-5b-open-large-scale/). Advances in Neural Information Processing Systems…
75. Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. [Laion-400m: Open dataset of clip-filtered 400 million image-text pairs](https://md.research.elamur.ai/free/en/papers/schuhmann-2021-laion-400m-open-dataset-clip/). arXiv preprint arXiv:2111.02114, 2021. 2, 4
76. Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. [A-okvqa: A benchmark for visual question answering using world knowledge](https://md.research.elamur.ai/free/en/papers/schwenk-2022-okvqa-benchmark-visual-question-answering/). In European Conference on Computer Vision, pages 146–162. Springer, 2022. 4
77. Wenqi Shao, Yutao Hu, Peng Gao, Meng Lei, Kaipeng Zhang, Fanqing Meng, Peng Xu, Siyuan Huang, Hongsheng Li, Yu Qiao, et al. Tiny lvlm-ehub: Early multimodal experiments with bard. arXiv preprint arXiv:2308.03729, 2023. 2 — [arXiv:2308.03729](https://arxiv.org/abs/2308.03729) [find](https://scholar.google.com/scholar?q=Wenqi+Shao%2C+Yutao+Hu%2C+Peng+Gao%2C+Meng+Lei%2C+Kaipeng+Zhang%2C+Fanqing+Meng%2C+Peng+Xu%2C+Siyuan+Huang%2C+Hongsheng+Li%2C+Yu+Qiao%2C+et+al%2E+Tiny+lvlm%2Dehub%3A+Early+multimodal+experiments+with+bard%2E+arXiv+preprint+arXiv)
78. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. [Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018.…
79. Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 742–758. Springer, 2020. 4 [find](https://scholar.google.com/scholar?q=Oleksii+Sidorov%2C+Ronghang+Hu%2C+Marcus+Rohrbach%2C+and+Amanpreet+Singh%2E+Textcaps%3A+a+dataset+for+image+captioning+with+reading+comprehension%2E+In+Computer+Vision%E2%80%93+ECCV+2020%3A+16th+European+Conference%2C+Glasgo)
80. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 4 [find](https://scholar.google.com/scholar?q=Amanpreet+Singh%2C+Vivek+Natarajan%2C+Meet+Shah%2C+Yu+Jiang%2C+Xinlei+Chen%2C+Dhruv+Batra%2C+Devi+Parikh%2C+and+Marcus+Rohrbach%2E+Towards+vqa+models+that+can+read%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+comput)
81. Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2443–244… [find](https://scholar.google.com/scholar?q=Krishna+Srinivasan%2C+Karthik+Raman%2C+Jiecao+Chen%2C+Michael+Bendersky%2C+and+Marc+Najork%2E+Wit%3A+Wikipedia%2Dbased+image+text+dataset+for+multimodal+multilingual+machine+learning%2E+In+Proceedings+of+the+44th+Int)
82. Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. Jul 2023. 3 [find](https://scholar.google.com/scholar?q=Quan+Sun%2C+Qiying+Yu%2C+Yufeng+Cui%2C+Fan+Zhang%2C+Xiaosong+Zhang%2C+Yueze+Wang%2C+Hongcheng+Gao%2C+Jingjing+Liu%2C+Tiejun+Huang%2C+and+Xinlong+Wang%2E+Generative+pretraining+in+multimodality%2E+Jul+2023%2E+3)
83. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. 3, 4 [↗](https://github.com/tatsu-lab/stanford_alpaca)
84. InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities. [https://github.com/InternLM/InternLM](https://github.com/InternLM/InternLM), 2023. 2, 3, 4, 8 [↗](https://github.com/InternLM/InternLM)
85. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv.org, 2023. 2, 3
86. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models, 2023. 2, 3, 8 [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtun)
87. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in Neural Information Processing Systems (NeurIPS), 30, 2017. 3
88. Bin Wang, Fan Wu, Xiao Han, Jiahui Peng, Huaping Zhong, Pan Zhang, Xiaoyi Dong, Weijia Li, Wei Li, Jiaqi Wang, et al. Vigc: Visual instruction generation and correction. arXiv.org, 2023. 3, 4 [find](https://scholar.google.com/scholar?q=Bin+Wang%2C+Fan+Wu%2C+Xiao+Han%2C+Jiahui+Peng%2C+Huaping+Zhong%2C+Pan+Zhang%2C+Xiaoyi+Dong%2C+Weijia+Li%2C+Wei+Li%2C+Jiaqi+Wang%2C+et+al%2E+Vigc%3A+Visual+instruction+generation+and+correction%2E+arXiv%2Eorg%2C+2023%2E+3%2C+4)
89. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. [Cogvlm: Visual expert for pretrained language models](https://md.research.elamur.ai/free/en/papers/wang-2023-cogvlm-visual-expert-pretrained/), 2023. 3
90. Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, et al. Q-bench: A benchmark for general-purpose foundation models on low-level vision. arXiv preprint arXiv:2309.14181, 2023. 2, 7 — [arXiv:2309.14181](https://arxiv.org/abs/2309.14181) [find](https://scholar.google.com/scholar?q=Haoning+Wu%2C+Zicheng+Zhang%2C+Erli+Zhang%2C+Chaofeng+Chen%2C+Liang+Liao%2C+Annan+Wang%2C+Chunyi+Li%2C+Wenxiu+Sun%2C+Qiong+Yan%2C+Guangtao+Zhai%2C+et+al%2E+Q%2Dbench%3A+A+benchmark+for+general%2Dpurpose+foundation+models+on+low%2D)
91. Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models, 2023. 16 [find](https://scholar.google.com/scholar?q=Peng+Xu%2C+Wenqi+Shao%2C+Kaipeng+Zhang%2C+Peng+Gao%2C+Shuo+Liu%2C+Meng+Lei%2C+Fanqing+Meng%2C+Siyuan+Huang%2C+Yu+Qiao%2C+and+Ping+Luo%2E+Lvlm%2Dehub%3A+A+comprehensive+evaluation+benchmark+for+large+vision%2Dlanguage+models%2C+2)
92. An Yang, Junshu Pan, Junyang Lin, Rui Men, Yichang Zhang, Jingren Zhou, and Chang Zhou. Chinese clip: Contrastive vision-language pretraining in chinese. arXiv.org, 2022. 2, 3, 4 [find](https://scholar.google.com/scholar?q=An+Yang%2C+Junshu+Pan%2C+Junyang+Lin%2C+Rui+Men%2C+Yichang+Zhang%2C+Jingren+Zhou%2C+and+Chang+Zhou%2E+Chinese+clip%3A+Contrastive+vision%2Dlanguage+pretraining+in+chinese%2E+arXiv%2Eorg%2C+2022%2E+2%2C+3%2C+4)
93. Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Richard James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Retrieval-augmented multimodal language modeling. 2023. 3 [find](https://scholar.google.com/scholar?q=Michihiro+Yasunaga%2C+Armen+Aghajanyan%2C+Weijia+Shi%2C+Richard+James%2C+Jure+Leskovec%2C+Percy+Liang%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Wen%2Dtau+Yih%2E+Retrieval%2Daugmented+multimodal+language+modeling%2E+2023%2E+3)
94. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. [mplug-owl: Modularization empowers large language models with multimodality](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/). arXiv.org, 2023. 3, 6, 7
95. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. [A survey on multimodal large language models](https://md.research.elamur.ai/free/en/papers/yin-2023-survey-multimodal-large-language-models/). arXiv preprint arXiv:2306.13549, 2023. 2
96. Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, Candace Ross, Adam Polyak, Russell Howes, Vasu Sharma, Puxin Xu, Hovhannes Tamoyan, Oron Ashual, Uriel Singer, Shang-Wen Li, Susan Zhang, Gargi Ghosh, Yaniv Taigm… [find](https://scholar.google.com/scholar?q=Lili+Yu%2C+Bowen+Shi%2C+Ramakanth+Pasunuru%2C+Benjamin+Muller%2C+Olga+Golovneva%2C+Tianlu+Wang%2C+Arun+Babu%2C+Binh+Tang%2C+Brian+Karrer%2C+Shelly+Sheynin%2C+Candace+Ross%2C+Adam+Polyak%2C+Russell+Howes%2C+Vasu+Sharma%2C+Puxin+X)
97. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. [GLM-130b: An open bilingual pre-trained model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/). In The Eleventh Inter…
98. Yan Zeng, Hanbo Zhang, Jiani Zheng, Jiangnan Xia, Guoqiang Wei, Yang Wei, Yuchen Zhang, and Tao Kong. What matters in training a gpt4-style language model with multimodal inputs? ArXiv, abs/2307.02469, 2023. 6, 17 [find](https://scholar.google.com/scholar?q=Yan+Zeng%2C+Hanbo+Zhang%2C+Jiani+Zheng%2C+Jiangnan+Xia%2C+Guoqiang+Wei%2C+Yang+Wei%2C+Yuchen+Zhang%2C+and+Tao+Kong%2E+What+matters+in+training+a+gpt4%2Dstyle+language+model+with+multimodal+inputs%3F+ArXiv%2C+abs%2F2307%2E02469)
99. Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding. Advances in Neural Information Processing Systems (NeurIPS), 35:36067–36080, 2022. 3 [find](https://scholar.google.com/scholar?q=Haotian+Zhang%2C+Pengchuan+Zhang%2C+Xiaowei+Hu%2C+Yen%2DChun+Chen%2C+Liunian+Li%2C+Xiyang+Dai%2C+Lijuan+Wang%2C+Lu+Yuan%2C+Jenq%2DNeng+Hwang%2C+and+Jianfeng+Gao%2E+Glipv2%3A+Unifying+localization+and+vision%2Dlanguage+understand)
100. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/), 2022. arXiv.org. 3
101. Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. Mmicl: Empowering vision-language model with multi-modal in-context learning. arXiv.org, 2023. 3, 6 [find](https://scholar.google.com/scholar?q=Haozhe+Zhao%2C+Zefan+Cai%2C+Shuzheng+Si%2C+Xiaojian+Ma%2C+Kaikai+An%2C+Liang+Chen%2C+Zixuan+Liu%2C+Sheng+Wang%2C+Wenjuan+Han%2C+and+Baobao+Chang%2E+Mmicl%3A+Empowering+vision%2Dlanguage+model+with+multi%2Dmodal+in%2Dcontext+lear)
102. Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. Mmicl: Empowering vision-language model with multi-modal in-context learning. ArXiv, abs/2309.07915, 2023. 6 [find](https://scholar.google.com/scholar?q=Haozhe+Zhao%2C+Zefan+Cai%2C+Shuzheng+Si%2C+Xiaojian+Ma%2C+Kaikai+An%2C+Liang+Chen%2C+Zixuan+Liu%2C+Sheng+Wang%2C+Wenjuan+Han%2C+and+Baobao+Chang%2E+Mmicl%3A+Empowering+vision%2Dlanguage+model+with+multi%2Dmodal+in%2Dcontext+lear)
103. Zhiyuan Zhao, Linke Ouyang, Bin Wang, Siyuan Huang, Pan Zhang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Mllm-dataengine: An iterative refinement approach for mllm. arXiv.org, 2023. 3 [find](https://scholar.google.com/scholar?q=Zhiyuan+Zhao%2C+Linke+Ouyang%2C+Bin+Wang%2C+Siyuan+Huang%2C+Pan+Zhang%2C+Xiaoyi+Dong%2C+Jiaqi+Wang%2C+and+Conghui+He%2E+Mllm%2Ddataengine%3A+An+iterative+refinement+approach+for+mllm%2E+arXiv%2Eorg%2C+2023%2E+3)
104. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. [Minigpt-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/). arXiv.org, 2023. 2, 3, 6, 7, 17
105. Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. arXiv preprint arXiv:2304.06939, 2023. 4 — [arXiv:2304.06939](https://arxiv.org/abs/2304.06939) [find](https://scholar.google.com/scholar?q=Wanrong+Zhu%2C+Jack+Hessel%2C+Anas+Awadalla%2C+Samir+Yitzhak+Gadre%2C+Jesse+Dodge%2C+Alex+Fang%2C+Youngjae+Yu%2C+Ludwig+Schmidt%2C+William+Yang+Wang%2C+and+Yejin+Choi%2E+Multimodal+c4%3A+An+open%2C+billion%2Dscale+corpus+of+im)

*Updated 24 October 2026*