---
title: "Large Multilingual Models Pivot Zero-Shot Multimodal Learning"
description: "The authors propose MPM, a training paradigm that uses a multilingual LLM as a bridge. 80 references, cited by 30 papers here."
authors: ["Jinyi Hu", "Yuan Yao", "Chongyi Wang", "Shan Wang", "Yinxu Pan", "Qianyu Chen", "Tianyu Yu", "Hanghao Wu", "Yue Zhao", "Haoye Zhang", "Xu Han", "Yankai Lin", "Jiao Xue", "Dahai Li", "Zhiyuan Liu", "Maosong Sun"]
year: 2023
venue: "arXiv:2308.12038"
source: "https://arxiv.org/abs/2308.12038"
doi: "10.48550/arxiv.2308.12038"
arxiv: "2308.12038"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hu-2023-large-multilingual-models-pivot-zero/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hu-2023-large-multilingual-models-pivot-zero/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/) / Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages

# Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages

**Authors:** [Jinyi Hu](https://md.research.elamur.ai/free/en/authors/jinyi-hu/), [Yuan Yao](https://md.research.elamur.ai/free/en/authors/yuan-yao/), [Chongyi Wang](https://md.research.elamur.ai/free/en/authors/chongyi-wang/), [Shan Wang](https://md.research.elamur.ai/free/en/authors/shan-wang/), [Yinxu Pan](https://md.research.elamur.ai/free/en/authors/yinxu-pan/), [Qianyu Chen](https://md.research.elamur.ai/free/en/authors/qianyu-chen/), [Tianyu Yu](https://md.research.elamur.ai/free/en/authors/tianyu-yu/), Hanghao Wu, [Yue Zhao](https://md.research.elamur.ai/free/en/authors/yue-zhao/), Haoye Zhang, [Xu Han](https://md.research.elamur.ai/free/en/authors/xu-han/), [Yankai Lin](https://md.research.elamur.ai/free/en/authors/yankai-lin/), Jiao Xue, Dahai Li, [Zhiyuan Liu](https://md.research.elamur.ai/free/en/authors/zhiyuan-liu/) and [Maosong Sun](https://md.research.elamur.ai/free/en/authors/maosong-sun/) · **Year:** 2023 · **Venue:** arXiv:2308.12038 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2308.12038) · [PDF](https://arxiv.org/pdf/2308.12038) · [DOI 10.48550/arxiv.2308.12038](https://doi.org/10.48550/arxiv.2308.12038) · [arXiv 2308.12038](https://arxiv.org/abs/2308.12038) · [OpenAlex](https://openalex.org/W4386148428)

**Citations:** 13 citations (OpenAlex, 2026-10-14) · [30](https://md.research.elamur.ai/free/en/papers/hu-2023-large-multilingual-models-pivot-zero/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most large multimodal systems for image understanding and image generation work well only in English. Other languages lack the huge image-text collections such training needs: BLIP-2 used over 100M pairs, and Stable Diffusion used over 2B. The authors propose MPM, a training paradigm that uses a multilingual LLM as a bridge. English serves as the pivot language. The visual modules are aligned with the language model using English image-text data only, and the model's shared cross-lingual representations then carry this skill over to other languages with little or no native multimodal data.

For Chinese, built on the bilingual CPM-Bee model, they create VisCPM-Chat (image-to-text dialogue) and VisCPM-Paint (text-to-image generation). Although pretrained only on English pairs, VisCPM achieves state-of-the-art results among open-source Chinese models and outperforms systems trained on native Chinese pairs. The same recipe also yields a six-language conversational model based on LLaMA.

The approach offers a low-cost path to multimodal models for languages with scarce image-text data, which is relevant for Ukrainian. Code and weights are public.

## Key points
- **Two-stage design.** Training consists of multilingual alignment, which comes from a pretrained multilingual LLM, followed by multimodal alignment on English data. For image-to-text, the LLM stays frozen while the visual module is pretrained, and instruction tuning follows.
- **English training data.** VisCPM-Chat uses COCO, Visual Genome, CC3M, CC12M and Laion-COCO. The text-to-image side relies on Laion-2B.
- **Ablation comparisons.** English-only pretraining is compared against 20M native Chinese pairs filtered from Wukong and Zero, and against 390M machine-translated Chinese captions from Laion-COCO. Instruction tuning in the ablations uses LLaVA and its translated Chinese version. The authors found many Wukong captions to be of poor quality, with broken images, unrelated text and truncated sentences.
- **Evaluation.** Generated images are scored with Chinese-CLIP (CLIP score). Human preference is measured on Chinese Drawbench against Taiyi and Alt, rated for alignment and fidelity.
- **Multilingual extension.** The six-language model covers English, German, French, Spanish, Italian and Portuguese. Code and weights are released at github.com/OpenBMB/VisCPM.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 333 works · 87,019 citations
- [Renmin University of China](https://md.research.elamur.ai/free/en/universities/renmin-university-of-china/) — University, Beijing, China · 114 works · 16,118 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 117 works · 11,281 citations
- [Beijing University of Posts and Telecommunications](https://md.research.elamur.ai/free/en/universities/beijing-university-of-posts-and-telecommunications/) — University, Beijing, China · 57 works · 32,843 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works

## Cite

Cite as: Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, Xu Han, Yankai Lin, Jiao Xue, Dahai Li, Zhiyuan Liu and Maosong Sun. 2023. Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages. arXiv:2308.12038.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hu2023large,
  title = {{Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages}},
  author = {Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, Xu Han, Yankai Lin, Jiao Xue, Dahai Li, Zhiyuan Liu and Maosong Sun},
  year = {2023},
  doi = {10.48550/arxiv.2308.12038},
  eprint = {2308.12038},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.12038}
}
```

- [← previous Halo: Estimation and Reduction of Hallucinations in Open-Source Weak Large Language Models](https://md.research.elamur.ai/free/en/papers/elaraby-2023-halo-estimation-reduction-hallucinations-open/)
- [next → Goal Driven Discovery of Distributional Differences via Language Descriptions](https://md.research.elamur.ai/free/en/papers/zhong-2023-goal-driven-discovery-distributional-differences/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/)

## References

80 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In Proceedings of ICCV, 2015. [find](https://scholar.google.com/scholar?q=Stanislaw+Antol%2C+Aishwarya+Agrawal%2C+Jiasen+Lu%2C+Margaret+Mitchell%2C+Dhruv+Batra%2C+C%2E+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+VQA%3A+Visual+Question+Answering%2E+In+Proceedings+of+ICCV%2C+2015%2E)
2. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.[12966, 2023](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/).
3. Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Procee… [find](https://scholar.google.com/scholar?q=Fan+Bao%2C+Shen+Nie%2C+Kaiwen+Xue%2C+Chongxuan+Li%2C+Shi+Pu%2C+Yaole+Wang%2C+Gang+Yue%2C+Yue+Cao%2C+Hang+Su%2C+and+Jun+Zhu%2E+One+transformer+fits+all+distributions+in+multi%2Dmodal+diffusion+at+scale%2E+In+Andreas+Krause%2C+E)
4. Marco Bellagente, Manuel Brack, Hannah Teufel, Felix Friedrich, Björn Deiseroth, Constantin Eichenberg, Andrew Dai, Robert Baldock, Souradeep Nanda, Koen Oostermeijer, et al. Multifusion: Fusing pre-trained models for multi-lingual, multi-modal image generation. arXiv preprint arXiv:2305.15296, 2023… — [arXiv:2305.15296](https://arxiv.org/abs/2305.15296) [find](https://scholar.google.com/scholar?q=Marco+Bellagente%2C+Manuel+Brack%2C+Hannah+Teufel%2C+Felix+Friedrich%2C+Bj%C3%B6rn+Deiseroth%2C+Constantin+Eichenberg%2C+Andrew+Dai%2C+Robert+Baldock%2C+Souradeep+Nanda%2C+Koen+Oostermeijer%2C+et+al%2E+Multifusion%3A+Fusing+pre%2Dt)
5. Federico Bianchi, Giuseppe Attanasio, Raphael Pisoni, Silvia Terragni, Gabriele Sarti, and Sri Lakshmi. Contrastive language-image pre-training for the italian language. arXiv preprint arXiv:2108.08688, 2021. [find](https://scholar.google.com/scholar?q=Federico+Bianchi%2C+Giuseppe+Attanasio%2C+Raphael+Pisoni%2C+Silvia+Terragni%2C+Gabriele+Sarti%2C+and+Sri+Lakshmi%2E+Contrastive+language%2Dimage+pre%2Dtraining+for+the+italian+language%2E+arXiv+preprint+arXiv%3A2108%2E0868)
6. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, et al. [Findings of the 2017 conference on machine translation (wmt17)](https://md.research.elamur.ai/free/en/papers/bojar-2017-findings-2017-conference-machine-translation/). Association for Computational Linguistics, 2017.
7. Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, and Ivan Vulić. IGLUE: A benchmark for transfer learning across modalities, tasks, and languages. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.)… [find](https://scholar.google.com/scholar?q=Emanuele+Bugliarello%2C+Fangyu+Liu%2C+Jonas+Pfeiffer%2C+Siva+Reddy%2C+Desmond+Elliott%2C+Edoardo+Maria+Ponti%2C+and+Ivan+Vulic%CC%81%2E+IGLUE%3A+A+benchmark+for+transfer+learning+across+modalities%2C+tasks%2C+and+languages%2E+I)
8. Fredrik Carlsson, Philipp Eisen, Faton Rekathati, and Magnus Sahlgren. Cross-lingual and multilingual clip. In Proceedings of LREC, pp. 6848–6854, 2022. [find](https://scholar.google.com/scholar?q=Fredrik+Carlsson%2C+Philipp+Eisen%2C+Faton+Rekathati%2C+and+Magnus+Sahlgren%2E+Cross%2Dlingual+and+multilingual+clip%2E+In+Proceedings+of+LREC%2C+pp%2E+6848%E2%80%936854%2C+2022%2E)
9. Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704, 2023. — [arXiv:2301.00704](https://arxiv.org/abs/2301.00704) [find](https://scholar.google.com/scholar?q=Huiwen+Chang%2C+Han+Zhang%2C+Jarred+Barber%2C+AJ+Maschinot%2C+Jose+Lezama%2C+Lu+Jiang%2C+Ming%2DHsuan+Yang%2C+Kevin+Murphy%2C+William+T+Freeman%2C+Michael+Rubinstein%2C+et+al%2E+Muse%3A+Text%2Dto%2Dimage+generation+via+masked+gene)
10. Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of CVPR, pp. 3558–3568, June 2021. [find](https://scholar.google.com/scholar?q=Soravit+Changpinyo%2C+Piyush+Sharma%2C+Nan+Ding%2C+and+Radu+Soricut%2E+Conceptual+12M%3A+Pushing+web%2Dscale+image%2Dtext+pre%2Dtraining+to+recognize+long%2Dtail+visual+concepts%2E+In+Proceedings+of+CVPR%2C+pp%2E+3558%E2%80%933568%2C+)
11. Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. [Pali: A jointly-scaled multilingual language-image model](https://md.research.elamur.ai/free/en/papers/chen-2022-pali-jointly-scaled-multilingual-language/). In Proceedings of ICLR, 2022.
12. Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Qinghong Yang, and Ledell Wu. AltCLIP: Altering the language encoder in CLIP for extended language capabilities. In Findings of ACL, pp. 8666–8682, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. findings-acl.552. [find](https://scholar.google.com/scholar?q=Zhongzhi+Chen%2C+Guang+Liu%2C+Bo%2DWen+Zhang%2C+Qinghong+Yang%2C+and+Ledell+Wu%2E+AltCLIP%3A+Altering+the+language+encoder+in+CLIP+for+extended+language+capabilities%2E+In+Findings+of+ACL%2C+pp%2E+8666%E2%80%938682%2C+Toronto%2C+Can)
13. Schuhmann Christoph, Köpf Andreas, Vencu Richard, Coombes Theo, and Beaumont Romain. Laion COCO: 600M synthetic captions from Laion2B-En, 2022. URL [https://laion.ai/blog/](https://laion.ai/blog/) laion-coco/. [↗](https://laion.ai/blog/)
14. James M Clark and Allan Paivio. Dual coding theory and education. Educational psychology review, 3:149–210, 1991. [find](https://scholar.google.com/scholar?q=James+M+Clark+and+Allan+Paivio%2E+Dual+coding+theory+and+education%2E+Educational+psychology+review%2C+3%3A149%E2%80%93210%2C+1991%2E)
15. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. [InstructBLIP: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/), 2023.
16. Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M.F. Moura, Devi Parikh, and Dhruv Batra. Visual Dialog. In Proceedings of CVPR, 2017. [find](https://scholar.google.com/scholar?q=Abhishek+Das%2C+Satwik+Kottur%2C+Khushi+Gupta%2C+Avi+Singh%2C+Deshraj+Yadav%2C+Jos%C3%A9+M%2EF%2E+Moura%2C+Devi+Parikh%2C+and+Dhruv+Batra%2E+Visual+Dialog%2E+In+Proceedings+of+CVPR%2C+2017%2E)
17. Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. NeurIPS, 35:16890–16902, 2022. [find](https://scholar.google.com/scholar?q=Ming+Ding%2C+Wendi+Zheng%2C+Wenyi+Hong%2C+and+Jie+Tang%2E+Cogview2%3A+Faster+and+better+text%2Dto%2Dimage+generation+via+hierarchical+transformers%2E+NeurIPS%2C+35%3A16890%E2%80%9316902%2C+2022%2E)
18. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). In Proceedings of ICLR, 2020.
19. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. [Glm: General language model pretraining with autoregressive blank infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/). In Proceedings of ACL, pp. 320–335, 2022.
20. Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of CVPR, pp. 12873–12883, 2021. [find](https://scholar.google.com/scholar?q=Patrick+Esser%2C+Robin+Rombach%2C+and+Bjorn+Ommer%2E+Taming+transformers+for+high%2Dresolution+image+synthesis%2E+In+Proceedings+of+CVPR%2C+pp%2E+12873%E2%80%9312883%2C+2021%2E)
21. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. JMLR, 22(1):4839–4886, 2021. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Shruti+Bhosale%2C+Holger+Schwenk%2C+Zhiyi+Ma%2C+Ahmed+El%2DKishky%2C+Siddharth+Goyal%2C+Mandeep+Baines%2C+Onur+Celebi%2C+Guillaume+Wenzek%2C+Vishrav+Chaudhary%2C+et+al%2E+Beyond+english%2Dcentric+multilingual+mac)
22. Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-Scene: Scene-based text-to-image generation with human priors. In Proceedings of ECCV, pp. 89–106. Springer, 2022. [find](https://scholar.google.com/scholar?q=Oran+Gafni%2C+Adam+Polyak%2C+Oron+Ashual%2C+Shelly+Sheynin%2C+Devi+Parikh%2C+and+Yaniv+Taigman%2E+Make%2Da%2DScene%3A+Scene%2Dbased+text%2Dto%2Dimage+generation+with+human+priors%2E+In+Proceedings+of+ECCV%2C+pp%2E+89%E2%80%93106%2E+Springer)
23. Jiaxi Gu, Xiaojun Meng, Guansong Lu, Lu Hou, Minzhe Niu, Hang Xu, Xiaodan Liang, Wei Zhang, Xin Jiang, and Chunjing Xu. Wukong: 100 million large-scale chinese cross-modal pre-training dataset and a foundation framework, 2022. [find](https://scholar.google.com/scholar?q=Jiaxi+Gu%2C+Xiaojun+Meng%2C+Guansong+Lu%2C+Lu+Hou%2C+Minzhe+Niu%2C+Hang+Xu%2C+Xiaodan+Liang%2C+Wei+Zhang%2C+Xin+Jiang%2C+and+Chunjing+Xu%2E+Wukong%3A+100+million+large%2Dscale+chinese+cross%2Dmodal+pre%2Dtraining+dataset+and+a+f)
24. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS, 30, 2017. [find](https://scholar.google.com/scholar?q=Martin+Heusel%2C+Hubert+Ramsauer%2C+Thomas+Unterthiner%2C+Bernhard+Nessler%2C+and+Sepp+Hochreiter%2E+GANs+trained+by+a+two+time%2Dscale+update+rule+converge+to+a+local+nash+equilibrium%2E+NeurIPS%2C+30%2C+2017%2E)
25. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33: 6840–6851, 2020. [find](https://scholar.google.com/scholar?q=Jonathan+Ho%2C+Ajay+Jain%2C+and+Pieter+Abbeel%2E+Denoising+diffusion+probabilistic+models%2E+NeurIPS%2C+33%3A+6840%E2%80%936851%2C+2020%2E)
26. Jinyi Hu, Xu Han, Xiaoyuan Yi, Yutong Chen, Wenhao Li, Zhiyuan Liu, and Maosong Sun. Efficient cross-lingual transfer for chinese stable diffusion with images as pivots. arXiv preprint arXiv:2305.11540, 2023. — [arXiv:2305.11540](https://arxiv.org/abs/2305.11540) [find](https://scholar.google.com/scholar?q=Jinyi+Hu%2C+Xu+Han%2C+Xiaoyuan+Yi%2C+Yutong+Chen%2C+Wenhao+Li%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+Efficient+cross%2Dlingual+transfer+for+chinese+stable+diffusion+with+images+as+pivots%2E+arXiv+preprint+arXiv%3A2305%2E1154)
27. Drew A Hudson and Christopher D Manning. [Gqa: A new dataset for real-world visual reasoning and compositional question answering](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/). In Proceedings of CVPR, pp. 6700–6709, 2019.
28. Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. If you use this software, please cite it as below. [find](https://scholar.google.com/scholar?q=Gabriel+Ilharco%2C+Mitchell+Wortsman%2C+Ross+Wightman%2C+Cade+Gordon%2C+Nicholas+Carlini%2C+Rohan+Taori%2C+Achal+Dave%2C+Vaishaal+Shankar%2C+Hongseok+Namkoong%2C+John+Miller%2C+Hannaneh+Hajishirzi%2C+Ali+Farhadi%2C+and+Ludwi)
29. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. [Segment anything](https://md.research.elamur.ai/free/en/papers/kirillov-2023-segment-anything/). arXiv:2304.02643, 2023.
30. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. [Visual Genome: Connecting language and vision using crowdsourced dense image annotations](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/). IJCV, 123:32–73, 2017.
31. Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr. Controllable text-to-image generation. NeurIPS, 32, 2019. [find](https://scholar.google.com/scholar?q=Bowen+Li%2C+Xiaojuan+Qi%2C+Thomas+Lukasiewicz%2C+and+Philip+Torr%2E+Controllable+text%2Dto%2Dimage+generation%2E+NeurIPS%2C+32%2C+2019%2E)
32. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. [BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/). In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of ICML, volume 162, pp. 1…
33. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. [BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of ICML, volume…
34. Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al. M 3 IT: A large-scale dataset towards multi-modal multilingual instruction tuning. arXiv preprint arXiv:2306.04387, 2023b. — [arXiv:2306.04387](https://arxiv.org/abs/2306.04387) [find](https://scholar.google.com/scholar?q=Lei+Li%2C+Yuwei+Yin%2C+Shicheng+Li%2C+Liang+Chen%2C+Peiyi+Wang%2C+Shuhuai+Ren%2C+Mukai+Li%2C+Yazheng+Yang%2C+Jingjing+Xu%2C+Xu+Sun%2C+et+al%2E+M+3+IT%3A+A+large%2Dscale+dataset+towards+multi%2Dmodal+multilingual+instruction+tuni)
35. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. [Microsoft COCO: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In Proceedings of ECCV, pp. 740–755. Springer, 2014.
36. Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. Visually grounded reasoning across languages and cultures. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Metho… [find](https://scholar.google.com/scholar?q=Fangyu+Liu%2C+Emanuele+Bugliarello%2C+Edoardo+Maria+Ponti%2C+Siva+Reddy%2C+Nigel+Collier%2C+and+Desmond+Elliott%2E+Visually+grounded+reasoning+across+languages+and+cultures%2E+In+Marie%2DFrancine+Moens%2C+Xuanjing+Huan)
37. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). arXiv preprint arXiv:2304.08485, 2023a.
38. Shanyuan Liu, Dawei Leng, and Yuhui Yin. Bridge diffusion model: bridge non-english language-native text-to-image diffusion model with english communities. arXiv preprint arXiv:2309.00952, 2023b. — [arXiv:2309.00952](https://arxiv.org/abs/2309.00952) [find](https://scholar.google.com/scholar?q=Shanyuan+Liu%2C+Dawei+Leng%2C+and+Yuhui+Yin%2E+Bridge+diffusion+model%3A+bridge+non%2Denglish+language%2Dnative+text%2Dto%2Dimage+diffusion+model+with+english+communities%2E+arXiv+preprint+arXiv%3A2309%2E00952%2C+2023b%2E+%E2%80%94+%5Ba)
39. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. [Grounding DINO: Marrying dino with grounded pre-training for open-set object detection](https://md.research.elamur.ai/free/en/papers/liu-2023-grounding-dino-marrying-dino-grounded/). arXiv preprint arXiv:2303.05499, 2023c.
40. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: A visual question answering benchmark requiring external knowledge. In Proceedings of CVPR, 2019. [find](https://scholar.google.com/scholar?q=Kenneth+Marino%2C+Mohammad+Rastegari%2C+Ali+Farhadi%2C+and+Roozbeh+Mottaghi%2E+OK%2DVQA%3A+A+visual+question+answering+benchmark+requiring+external+knowledge%2E+In+Proceedings+of+CVPR%2C+2019%2E)
41. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel.… — [Crosslingual Generalization through Multitask Finetuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/)
42. Minheng Ni, Haoyang Huang, Lin Su, Edward Cui, Taroon Bharti, Lijuan Wang, Dongdong Zhang, and Nan Duan. M3P: Learning universal representations via multitask multilingual multimodal pre-training. In Proceedings of CVPR, pp. 3977–3986, 2021. [find](https://scholar.google.com/scholar?q=Minheng+Ni%2C+Haoyang+Huang%2C+Lin+Su%2C+Edward+Cui%2C+Taroon+Bharti%2C+Lijuan+Wang%2C+Dongdong+Zhang%2C+and+Nan+Duan%2E+M3P%3A+Learning+universal+representations+via+multitask+multilingual+multimodal+pre%2Dtraining%2E+In+)
43. Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Ni… [find](https://scholar.google.com/scholar?q=Alexander+Quinn+Nichol%2C+Prafulla+Dhariwal%2C+Aditya+Ramesh%2C+Pranav+Shyam%2C+Pamela+Mishkin%2C+Bob+Mcgrew%2C+Ilya+Sutskever%2C+and+Mark+Chen%2E+GLIDE%3A+Towards+photorealistic+image+generation+and+editing+with+text%2D)
44. OpenAI. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
45. Allan Paivio. Bilingual Dual Coding Theory and Memory, pp. 41–62. Springer New York, New York, NY, 2014. ISBN 978-1-4614-9218-4. doi: P10.1007/978-1-4614-9218-4_3. [find](https://scholar.google.com/scholar?q=Allan+Paivio%2E+Bilingual+Dual+Coding+Theory+and+Memory%2C+pp%2E+41%E2%80%9362%2E+Springer+New+York%2C+New+York%2C+NY%2C+2014%2E+ISBN+978%2D1%2D4614%2D9218%2D4%2E+doi%3A+P10%2E1007%2F978%2D1%2D4614%2D9218%2D4%5F3%2E)
46. Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin Steitz, Stefan Roth, Ivan Vulić, and Iryna Gurevych. xGQA: Cross-lingual visual question answering. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp… [find](https://scholar.google.com/scholar?q=Jonas+Pfeiffer%2C+Gregor+Geigle%2C+Aishwarya+Kamath%2C+Jan%2DMartin+Steitz%2C+Stefan+Roth%2C+Ivan+Vulic%CC%81%2C+and+Iryna+Gurevych%2E+xGQA%3A+Cross%2Dlingual+visual+question+answering%2E+In+Smaranda+Muresan%2C+Preslav+Nakov%2C+and)
47. Chen Qiu, Dan Oneat , ă, Emanuele Bugliarello, Stella Frank, and Desmond Elliott. Multilingual multimodal learning with machine translated text. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 4178–4193, Abu Dhabi… [find](https://scholar.google.com/scholar?q=Chen+Qiu%2C+Dan+Oneat+%2C+a%CC%86%2C+Emanuele+Bugliarello%2C+Stella+Frank%2C+and+Desmond+Elliott%2E+Multilingual+multimodal+learning+with+machine+translated+text%2E+In+Yoav+Goldberg%2C+Zornitsa+Kozareva%2C+and+Yue+Zhang+%28ed)
48. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In ICML, pp. 8748–8763. PMLR, 2021.
49. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. [Zero-shot text-to-image generation](https://md.research.elamur.ai/free/en/papers/ramesh-2021-zero-shot-text-image-generation/). In Marina Meila and Tong Zhang (eds.), Proceedings of ICML, volume 139, pp. 8821–8831. PMLR, 18–24 Jul 2021.
50. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. [Hierarchical text-conditional image generation with clip latents](https://md.research.elamur.ai/free/en/papers/ramesh-2022-hierarchical-text-conditional-image-generation/). arXiv preprint arXiv:2204.06125, 2022.
51. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of CVPR, pp. 10684–10695, 2022. [find](https://scholar.google.com/scholar?q=Robin+Rombach%2C+Andreas+Blattmann%2C+Dominik+Lorenz%2C+Patrick+Esser%2C+and+Bj%C3%B6rn+Ommer%2E+High%2Dresolution+image+synthesis+with+latent+diffusion+models%2E+In+Proceedings+of+CVPR%2C+pp%2E+10684%E2%80%9310695%2C+2022%2E)
52. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. [U-Net: Convolutional networks for biomedical image segmentation](https://md.research.elamur.ai/free/en/papers/ronneberger-2015-u-net-convolutional-networks-biomedical/). In MICCAI, pp. 234–241. Springer, 2015.
53. Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. [Photorealistic text-to-image diffusion models with deep language understanding](https://md.research.elamur.ai/free/en/papers/saharia-2022-photorealistic-text-image-diffusion-models/). NeurIPS, 35:36479–36494, 2022.
54. Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. [Laion-400M: Open dataset of clip-filtered 400 million image-text pairs](https://md.research.elamur.ai/free/en/papers/schuhmann-2021-laion-400m-open-dataset-clip/). arXiv preprint arXiv:2111.02114, 2021.
55. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. [Laion-5B: An open large-scale dataset for training next generation image-text models](https://md.research.elamur.ai/free/en/papers/schuhmann-2022-laion-5b-open-large-scale/). NeurIPS, 35:25278–25294, 2022.
56. Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. [A-OKVQA: A benchmark for visual question answering using world knowledge](https://md.research.elamur.ai/free/en/papers/schwenk-2022-okvqa-benchmark-visual-question-answering/). arXiv, 2022.
57. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. [Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/). In Proceedings of ACL, pp. 2556–2565, 2018.
58. Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. [Score-based generative modeling through stochastic differential equations](https://md.research.elamur.ai/free/en/papers/song-2020-score-based-generative-modeling-through/). In Proceedings of ICLR, 2020.
59. Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. [PandaGPT: One model to instruction-follow them all](https://md.research.elamur.ai/free/en/papers/su-2023-pandagpt-one-model-instruction-follow/). arXiv preprint arXiv:2305.16355, 2023.
60. Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. [Aligning large multimodal models with factually augmented rlhf](https://md.research.elamur.ai/free/en/papers/sun-2023-aligning-large-multimodal-models-factually/). arXiv preprint arXiv:2309.14525, 2023.
61. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [LLaMA: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
62. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). NeurIPS, 30, 2017.
63. Junjie Wang, Yuxiang Zhang, Lin Zhang, Ping Yang, Xinyu Gao, Ziwei Wu, Xiaoqun Dong, Junqing He, Jianheng Zhuo, Qi Yang, Yongfeng Huang, Xiayu Li, Yanghan Wu, Junyu Lu, Xinyu Zhu, Weifeng Chen, Ting Han, Kunhao Pan, Rui Wang, Hao Wang, Xiaojun Wu, Zhongshen Zeng, Chongpei Chen, Ruyi Gan, and Jiaxing… — [arXiv:2209.02970](https://arxiv.org/abs/2209.02970) [find](https://scholar.google.com/scholar?q=Junjie+Wang%2C+Yuxiang+Zhang%2C+Lin+Zhang%2C+Ping+Yang%2C+Xinyu+Gao%2C+Ziwei+Wu%2C+Xiaoqun+Dong%2C+Junqing+He%2C+Jianheng+Zhuo%2C+Qi+Yang%2C+Yongfeng+Huang%2C+Xiayu+Li%2C+Yanghan+Wu%2C+Junyu+Lu%2C+Xinyu+Zhu%2C+Weifeng+Chen%2C+Ting+H)
64. Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. [OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework](https://md.research.elamur.ai/free/en/papers/wang-2022-ofa-unifying-architectures-tasks-modalities/). In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Ga…
65. Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for vision and vision-language tasks. In Proceedings of CVPR, pp. 19175–19186, 2023. [find](https://scholar.google.com/scholar?q=Wenhui+Wang%2C+Hangbo+Bao%2C+Li+Dong%2C+Johan+Bjorck%2C+Zhiliang+Peng%2C+Qiang+Liu%2C+Kriti+Aggarwal%2C+Owais+Khan+Mohammed%2C+Saksham+Singhal%2C+Subhojit+Som%2C+et+al%2E+Image+as+a+foreign+language%3A+Beit+pretraining+for+v)
66. Chunyu Xie, Heng Cai, Jianfei Song, Jincheng Li, Fanjing Kong, Xiaoyu Wu, Henrique Morimitsu, Lin Yao, Dexin Wang, Xiangzheng Zhang, Dawei Leng, Xiangyang Ji, and Yafeng Deng. Zero and R2D2: A large-scale chinese cross-modal benchmark and a vision-language framework, 2022. [find](https://scholar.google.com/scholar?q=Chunyu+Xie%2C+Heng+Cai%2C+Jianfei+Song%2C+Jincheng+Li%2C+Fanjing+Kong%2C+Xiaoyu+Wu%2C+Henrique+Morimitsu%2C+Lin+Yao%2C+Dexin+Wang%2C+Xiangzheng+Zhang%2C+Dawei+Leng%2C+Xiangyang+Ji%2C+and+Yafeng+Deng%2E+Zero+and+R2D2%3A+A+large%2Ds)
67. An Yang, Junshu Pan, Junyang Lin, Rui Men, Yichang Zhang, Jingren Zhou, and Chang Zhou. Chinese CLIP: Contrastive vision-language pretraining in chinese. arXiv preprint arXiv:2211.01335, 2022. — [arXiv:2211.01335](https://arxiv.org/abs/2211.01335) [find](https://scholar.google.com/scholar?q=An+Yang%2C+Junshu+Pan%2C+Junyang+Lin%2C+Rui+Men%2C+Yichang+Zhang%2C+Jingren+Zhou%2C+and+Chang+Zhou%2E+Chinese+CLIP%3A+Contrastive+vision%2Dlanguage+pretraining+in+chinese%2E+arXiv+preprint+arXiv%3A2211%2E01335%2C+2022%2E+%E2%80%94+%5BarXi)
68. Yukang Yang, Dongnan Gui, Yuhui Yuan, Haisong Ding, Han Hu, and Kai Chen. Glyphcontrol: Glyph conditional control for visual text generation. 2023. [find](https://scholar.google.com/scholar?q=Yukang+Yang%2C+Dongnan+Gui%2C+Yuhui+Yuan%2C+Haisong+Ding%2C+Han+Hu%2C+and+Kai+Chen%2E+Glyphcontrol%3A+Glyph+conditional+control+for+visual+text+generation%2E+2023%2E)
69. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. [mPLUG-Owl: Modularization empowers large language models with multimodality](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/), 2023…
70. Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. [CoCa: Contrastive captioners are image-text foundation models](https://md.research.elamur.ai/free/en/papers/yu-2022-coca-contrastive-captioners-image-text/). arXiv preprint arXiv:2205.01917, 2022a.
71. Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generat… [find](https://scholar.google.com/scholar?q=Jiahui+Yu%2C+Yuanzhong+Xu%2C+Jing+Yu+Koh%2C+Thang+Luong%2C+Gunjan+Baid%2C+Zirui+Wang%2C+Vijay+Vasudevan%2C+Alexander+Ku%2C+Yinfei+Yang%2C+Burcu+Karagol+Ayan%2C+Ben+Hutchinson%2C+Wei+Han%2C+Zarana+Parekh%2C+Xin+Li%2C+Han+Zhang%2C+J)
72. Tianyu Yu, Jinyi Hu, Yuan Yao, Haoye Zhang, Yue Zhao, Chongyi Wang, Shan Wang, Yinxv Pan, Jiao Xue, Dahai Li, et al. Reformulating vision-language foundation models and datasets towards universal multimodal assistants. arXiv preprint arXiv:2310.00653, 2023a. — [arXiv:2310.00653](https://arxiv.org/abs/2310.00653) [find](https://scholar.google.com/scholar?q=Tianyu+Yu%2C+Jinyi+Hu%2C+Yuan+Yao%2C+Haoye+Zhang%2C+Yue+Zhao%2C+Chongyi+Wang%2C+Shan+Wang%2C+Yinxv+Pan%2C+Jiao+Xue%2C+Dahai+Li%2C+et+al%2E+Reformulating+vision%2Dlanguage+foundation+models+and+datasets+towards+universal+mult)
73. Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. arXiv preprint arXiv:2312.00849, 2023b. [find](https://scholar.google.com/scholar?q=Tianyu+Yu%2C+Yuan+Yao%2C+Haoye+Zhang%2C+Taiwen+He%2C+Yifeng+Han%2C+Ganqu+Cui%2C+Jinyi+Hu%2C+Zhiyuan+Liu%2C+Hai%2DTao+Zheng%2C+Maosong+Sun%2C+et+al%2E+Rlhf%2Dv%3A+Towards+trustworthy+mllms+via+behavior+alignment+from+fine%2Dgrained)
74. Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021. — [arXiv:2111.11432](https://arxiv.org/abs/2111.11432) [find](https://scholar.google.com/scholar?q=Lu+Yuan%2C+Dongdong+Chen%2C+Yi%2DLing+Chen%2C+Noel+Codella%2C+Xiyang+Dai%2C+Jianfeng+Gao%2C+Houdong+Hu%2C+Xuedong+Huang%2C+Boxin+Li%2C+Chunyuan+Li%2C+et+al%2E+Florence%3A+A+new+foundation+model+for+computer+vision%2E+arXiv+prepr)
75. Yan Zeng, Wangchunshu Zhou, Ao Luo, Ziming Cheng, and Xinsong Zhang. Cross-view language modeling: Towards unified cross-lingual cross-modal pre-training. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Yan+Zeng%2C+Wangchunshu+Zhou%2C+Ao+Luo%2C+Ziming+Cheng%2C+and+Xinsong+Zhang%2E+Cross%2Dview+language+modeling%3A+Towards+unified+cross%2Dlingual+cross%2Dmodal+pre%2Dtraining%2E+In+Anna+Rogers%2C+Jordan+Boyd%2DGraber%2C+and+Naoak)
76. Ao Zhang, Hao Fei, Yuan Yao, Wei Ji, Li Li, Zhiyuan Liu, and Tat-Seng Chua. Transfer visual prompt generator across llms. arXiv preprint arXiv:2305.01278, 2023. [find](https://scholar.google.com/scholar?q=Ao+Zhang%2C+Hao+Fei%2C+Yuan+Yao%2C+Wei+Ji%2C+Li+Li%2C+Zhiyuan+Liu%2C+and+Tat%2DSeng+Chua%2E+Transfer+visual+prompt+generator+across+llms%2E+arXiv+preprint+arXiv%3A2305%2E01278%2C+2023%2E)
77. Zhengyan Zhang, Xu Han, Hao Zhou, Pei Ke, Yuxian Gu, Deming Ye, Yujia Qin, Yusheng Su, Haozhe Ji, Jian Guan, et al. [CPM: A large-scale generative chinese pre-trained language model](https://md.research.elamur.ai/free/en/papers/zhang-2020-cpm-large-scale-generative/). AI Open, 2:93–99, 2021.
78. Mingyang Zhou, Luowei Zhou, Shuohang Wang, Yu Cheng, Linjie Li, Zhou Yu, and Jingjing Liu. UC2: Universal cross-lingual cross-modal vision-and-language pre-training. In Proceedings of CVPR, pp. 4155–4165, 2021. [find](https://scholar.google.com/scholar?q=Mingyang+Zhou%2C+Luowei+Zhou%2C+Shuohang+Wang%2C+Yu+Cheng%2C+Linjie+Li%2C+Zhou+Yu%2C+and+Jingjing+Liu%2E+UC2%3A+Universal+cross%2Dlingual+cross%2Dmodal+vision%2Dand%2Dlanguage+pre%2Dtraining%2E+In+Proceedings+of+CVPR%2C+pp%2E+4155%E2%80%934)
79. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. [Minigpt-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/). arXiv preprint arXiv:2304.10592, 2023.
80. Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. DM-GAN: Dynamic memory generative adversarial networks for text-to-image synthesis. In Proceedings of CVPR, pp. 5802–5810, 2019. [find](https://scholar.google.com/scholar?q=Minfeng+Zhu%2C+Pingbo+Pan%2C+Wei+Chen%2C+and+Yi+Yang%2E+DM%2DGAN%3A+Dynamic+memory+generative+adversarial+networks+for+text%2Dto%2Dimage+synthesis%2E+In+Proceedings+of+CVPR%2C+pp%2E+5802%E2%80%935810%2C+2019%2E)

*Updated 31 October 2026*