---
title: "MM-LLMs — paper by Zhang et al. (2024) — summary and ideas"
description: "The authors describe a generic architecture with five parts: a modality encoder, an input projector… 310 references, cited by 68 papers here."
authors: ["Duzhen Zhang", "Yahan Yu", "Jiahua Dong", "Chenxing Li", "Dan Su", "Chenhui Chu", "Dong Yu"]
year: 2024
venue: "Findings of the Association for Computational Linguistics: ACL 2024"
source: "https://aclanthology.org/2024.findings-acl.738/"
doi: "10.18653/v1/2024.findings-acl.738"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2024-mm-llms-recent-advances-multimodal/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2024-mm-llms-recent-advances-multimodal/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/) / MM-LLMs: Recent Advances in MultiModal Large Language Models

# MM-LLMs: Recent Advances in MultiModal Large Language Models

**Authors:** [Duzhen Zhang](https://md.research.elamur.ai/free/en/authors/duzhen-zhang/), Yahan Yu, Jiahua Dong, Chenxing Li, [Dan Su](https://md.research.elamur.ai/free/en/authors/dan-su/), [Chenhui Chu](https://md.research.elamur.ai/free/en/authors/chenhui-chu/) and [Dong Yu](https://md.research.elamur.ai/free/en/authors/dong-yu/) · **Year:** 2024 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2024](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2024](https://md.research.elamur.ai/free/en/venues/findings/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.findings-acl.738/) · [PDF](https://aclanthology.org/2024.findings-acl.738.pdf) · [DOI 10.18653/v1/2024.findings-acl.738](https://doi.org/10.18653/v1/2024.findings-acl.738) · [OpenAlex](https://openalex.org/W4402670135)

**Citations:** 250 citations (OpenAlex, 2026-10-14) · [68](https://md.research.elamur.ai/free/en/papers/zhang-2024-mm-llms-recent-advances-multimodal/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
This survey maps the fast-growing field of multimodal large language models (MM-LLMs). These systems extend existing pretrained LLMs to handle inputs or outputs beyond text, such as images, video, audio and 3D data. Training such systems from scratch is costly. The field therefore reuses frozen unimodal foundation models, and its main problem is connecting them to an LLM so that they reason together. The authors describe a generic architecture with five parts: a modality encoder, an input projector, the LLM backbone, an output projector and a modality generator. They also describe a standard two-stage training recipe of MM pre-training followed by MM instruction tuning, and they list the main datasets for each stage. Using this framework, they classify 126 recent models, trace how the field has developed, compare leading systems on common benchmarks and extract practical training advice. They close with open research problems. The survey is useful as a structured entry point to the area and comes with a crowdsourced website that is updated over time.

## Key points
- The taxonomy covers 126 MM-LLMs, from Flamingo (2022) and BLIP-2 through LLaVA and MiniGPT-4 to any-to-any systems such as NExT-GPT, CoDi-2 and ModaVerse.
- The encoder, LLM and generator usually stay frozen, and only the lightweight projectors are trained. As a result, trainable parameters are typically about 2% of the total.
- Input projector designs range from simple linear or MLP layers to Cross-attention (Perceiver Resampler), Q-Former, P-Former and MQ-Former. The last three need an extra pre-training step for initialization.
- Common encoders include CLIP ViT and Eva-CLIP ViT for images, HuBERT, BEATs and Whisper for audio, ULIP-2 with Point-BERT for 3D, and ImageBind, which unifies six modalities.
- The paper names these future directions:
  - embodied agents (e.g., PaLM-E, EmbodiedGPT)
  - continual learning, which faces catastrophic forgetting and negative forward transfer
  - reducing hallucination
  - addressing bias and ethics
- Updates are tracked at https://mm-llms.github.io.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tencent](https://md.research.elamur.ai/free/en/companies/tencent/) — Company, Shenzhen, China · 66 works · 17,945 citations
- [Kyoto University](https://md.research.elamur.ai/free/en/universities/kyoto-university/) — University, Kyoto, Japan · 27 works · 8,049 citations
- [Mohamed bin Zayed University of Artificial Intelligence](https://md.research.elamur.ai/free/en/universities/mohamed-bin-zayed-university-of-artificial-intelligence/) — University, Abu Dhabi, United Arab Emirates · 47 works · 4,456 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 269 works

## Cite

Cite as: Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu and Dong Yu. 2024. MM-LLMs: Recent Advances in MultiModal Large Language Models. Findings of the Association for Computational Linguistics: ACL 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2024mm,
  title = {{MM-LLMs: Recent Advances in MultiModal Large Language Models}},
  author = {Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu and Dong Yu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.738},
  url = {https://doi.org/10.18653/v1/2024.findings-acl.738}
}
```

- [← previous xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://md.research.elamur.ai/free/en/papers/guerreiro-2024-xcomet-transparent-machine-translation-evaluation/)
- [next → LightRAG: Simple and Fast Retrieval-Augmented Generation](https://md.research.elamur.ai/free/en/papers/guo-2024-lightrag-simple-fast-retrieval-augmented/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/)

## References

310 references, 111 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 2023. Bliva: A simple multimodal llm for better handling of text-rich visual questions. arXiv preprint arXiv:2308.09936. — [arXiv:2308.09936](https://arxiv.org/abs/2308.09936) [find](https://scholar.google.com/scholar?q=2023%2E+Bliva%3A+A+simple+multimodal+llm+for+better+handling+of+text%2Drich+visual+questions%2E+arXiv+preprint+arXiv%3A2308%2E09936%2E+%E2%80%94+%5BarXiv%3A2308%2E09936%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2308%2E09936%29)
2. Emanuele Aiello, Lili Yu, Yixin Nie, Armen Aghajanyan, and Barlas Oguz. 2023. Jointly Training Large Autoregressive Multimodal Models. arXiv preprint arXiv:2309.15564. — [arXiv:2309.15564](https://arxiv.org/abs/2309.15564) [find](https://scholar.google.com/scholar?q=Emanuele+Aiello%2C+Lili+Yu%2C+Yixin+Nie%2C+Armen+Aghajanyan%2C+and+Barlas+Oguz%2E+2023%2E+Jointly+Training+Large+Autoregressive+Multimodal+Models%2E+arXiv+preprint+arXiv%3A2309%2E15564%2E+%E2%80%94+%5BarXiv%3A2309%2E15564%5D%28https%3A%2F%2Farx)
3. Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. 2021. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems, 34:24206–24221. [find](https://scholar.google.com/scholar?q=Hassan+Akbari%2C+Liangzhe+Yuan%2C+Rui+Qian%2C+Wei%2DHong+Chuang%2C+Shih%2DFu+Chang%2C+Yin+Cui%2C+and+Boqing+Gong%2E+2021%2E+Vatt%3A+Transformers+for+multimodal+self%2Dsupervised+learning+from+raw+video%2C+audio+and+text%2E+Advan)
4. Reynolds, et al. 2022. [Flamingo: a visual language model for few-shot learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/). Advances in Neural Information Processing Systems, 35:23716–23736.
5. Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. 2023. [Retrieval-based language models and applications](https://md.research.elamur.ai/free/en/papers/asai-2023-retrieval-based-language-models-applications/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts), pages 41–46.
6. Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. 2023. [Openflamingo: An open-source framework for training large autoregressive vision-language models](https://md.research.elamur.ai/free/en/papers/awadalla-2023-openflamingo-open-source-framework-training/). arXiv preprint arXiv:2308.01390.
7. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023a. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609.
8. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023b. [Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/). CoRR, abs/2308.12966.
9. Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. 2021. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728–1738. [find](https://scholar.google.com/scholar?q=Max+Bain%2C+Arsha+Nagrani%2C+G%C3%BCl+Varol%2C+and+Andrew+Zisserman%2E+2021%2E+Frozen+in+time%3A+A+joint+video+and+image+encoder+for+end%2Dto%2Dend+retrieval%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Conference+on+Com)
10. Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. 2022. Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Fan+Bao%2C+Chongxuan+Li%2C+Jun+Zhu%2C+and+Bo+Zhang%2E+2022%2E+Analytic%2DDPM%3A+an+Analytic+Estimate+of+the+Optimal+Reverse+Variance+in+Diffusion+Probabilistic+Models%2E+In+International+Conference+on+Learning+Repres)
11. Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sağnak Taşırlar. 2023. Introducing our Multimodal Models. [find](https://scholar.google.com/scholar?q=Rohan+Bavishi%2C+Erich+Elsen%2C+Curtis+Hawthorne%2C+Maxwell+Nye%2C+Augustus+Odena%2C+Arushi+Somani%2C+and+Sag%CC%86nak+Tas%CC%A7%C4%B1rlar%2E+2023%2E+Introducing+our+Multimodal+Models%2E)
12. Ali Furkan Biten, Ron Litman, Yusheng Xie, Srikar Appalaraju, and R Manmatha. 2022. Latr: Layout-aware transformer for scene-text vqa. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16548–16558. [find](https://scholar.google.com/scholar?q=Ali+Furkan+Biten%2C+Ron+Litman%2C+Yusheng+Xie%2C+Srikar+Appalaraju%2C+and+R+Manmatha%2E+2022%2E+Latr%3A+Layout%2Daware+transformer+for+scene%2Dtext+vqa%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+vision+and+)
13. Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. 2021. High-performance large-scale image recognition without normalization. In International Conference on Machine Learning, pages 1059–1071. PMLR. [find](https://scholar.google.com/scholar?q=Andy+Brock%2C+Soham+De%2C+Samuel+L+Smith%2C+and+Karen+Simonyan%2E+2021%2E+High%2Dperformance+large%2Dscale+image+recognition+without+normalization%2E+In+International+Conference+on+Machine+Learning%2C+pages+1059%E2%80%931071%2E+)
14. Afra Feyza Akyürek, Ekin Akyürek, Aman Madaan, Ashwin Kalyan, Peter Clark, Derry Wijaya, and Niket Tandon. 2023. [RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs](https://md.research.elamur.ai/free/en/papers/akyurek-2023-rl4f-generating-natural-language-feedback/). arXiv preprint arXiv:2305.08844.
15. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
16. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. 2022. Coyo-700m: Image-text pair dataset. [find](https://scholar.google.com/scholar?q=Jean%2DBaptiste+Alayrac%2C+Jeff+Donahue%2C+Pauline+Luc%2C+Antoine+Miech%2C+Iain+Barr%2C+Yana+Hasson%2C+Karel+Lenc%2C+Arthur+Mensch%2C+Katherine+Millican%2C+Malcolm+Minwoo+Byeon%2C+Beomhee+Park%2C+Haecheon+Kim%2C+Sungjun+Lee%2C+W)
17. Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961–970. [find](https://scholar.google.com/scholar?q=Fabian+Caba+Heilbron%2C+Victor+Escorcia%2C+Bernard+Ghanem%2C+and+Juan+Carlos+Niebles%2E+2015%2E+Activitynet%3A+A+large%2Dscale+video+benchmark+for+human+activity+understanding%2E+In+Proceedings+of+the+ieee+conference)
18. Rizhao Cai, Zirui Song, Dayan Guan, Zhenhao Chen, Xing Luo, Chenyu Yi, and Alex Kot. 2023. BenchLMM: Benchmarking cross-style visual capability of large multimodal models. arXiv preprint arXiv:2312.02896. — [arXiv:2312.02896](https://arxiv.org/abs/2312.02896) [find](https://scholar.google.com/scholar?q=Rizhao+Cai%2C+Zirui+Song%2C+Dayan+Guan%2C+Zhenhao+Chen%2C+Xing+Luo%2C+Chenyu+Yi%2C+and+Alex+Kot%2E+2023%2E+BenchLMM%3A+Benchmarking+cross%2Dstyle+visual+capability+of+large+multimodal+models%2E+arXiv+preprint+arXiv%3A2312%2E02)
19. Cerspense. 2023. Zeroscope: Diffusion-based text-to-video synthesis. [find](https://scholar.google.com/scholar?q=Cerspense%2E+2023%2E+Zeroscope%3A+Diffusion%2Dbased+text%2Dto%2Dvideo+synthesis%2E)
20. Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558–3568. [find](https://scholar.google.com/scholar?q=Soravit+Changpinyo%2C+Piyush+Sharma%2C+Nan+Ding%2C+and+Radu+Soricut%2E+2021%2E+Conceptual+12m%3A+Pushing+web%2Dscale+image%2Dtext+pre%2Dtraining+to+recognize+long%2Dtail+visual+concepts%2E+In+Proceedings+of+the+IEEE%2FCVF+Co)
21. Fei-Long Chen, Du-Zhen Zhang, Ming-Lun Han, Xiu-Yi Chen, Jing Shi, Shuang Xu, and Bo Xu. 2023a. Vlp: A survey on vision-language pre-training. Machine Intelligence Research, 20(1):38–56. [find](https://scholar.google.com/scholar?q=Fei%2DLong+Chen%2C+Du%2DZhen+Zhang%2C+Ming%2DLun+Han%2C+Xiu%2DYi+Chen%2C+Jing+Shi%2C+Shuang+Xu%2C+and+Bo+Xu%2E+2023a%2E+Vlp%3A+A+survey+on+vision%2Dlanguage+pre%2Dtraining%2E+Machine+Intelligence+Research%2C+20%281%29%3A38%E2%80%9356%2E)
22. Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. 2023b. Xllm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages. arXiv preprint arXiv:2305.04160. — [arXiv:2305.04160](https://arxiv.org/abs/2305.04160) [find](https://scholar.google.com/scholar?q=Feilong+Chen%2C+Minglun+Han%2C+Haozhi+Zhao%2C+Qingyang+Zhang%2C+Jing+Shi%2C+Shuang+Xu%2C+and+Bo+Xu%2E+2023b%2E+Xllm%3A+Bootstrapping+advanced+large+language+models+by+treating+multi%2Dmodalities+as+foreign+languages%2E+arX)
23. Gongwei Chen, Leyang Shen, Rui Shao, Xiang Deng, and Liqiang Nie. 2023c. LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge. arXiv preprint arXiv:2311.11860. — [arXiv:2311.11860](https://arxiv.org/abs/2311.11860) [find](https://scholar.google.com/scholar?q=Gongwei+Chen%2C+Leyang+Shen%2C+Rui+Shao%2C+Xiang+Deng%2C+and+Liqiang+Nie%2E+2023c%2E+LION%3A+Empowering+Multimodal+Large+Language+Model+with+Dual%2DLevel+Visual+Knowledge%2E+arXiv+preprint+arXiv%3A2311%2E11860%2E+%E2%80%94+%5BarXiv%3A23)
24. Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. 2023d. [Minigpt-v2: large language model as a unified interface for vision-language multi-task learning](https://md.research.elamur.ai/free/en/papers/chen-2023-minigpt-v2-large-language-model/). arXiv preprint arXiv:2310.09478.
25. Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. 2023e. [Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic](https://md.research.elamur.ai/free/en/papers/chen-2023-shikra-unleashing-multimodal-llm-s/). arXiv preprint arXiv:2306.15195.
26. Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2023f. [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://md.research.elamur.ai/free/en/papers/chen-2023-sharegpt4v-improving-large-multi-modal/). arXiv preprint arXiv:2311.12793.
27. Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. 2023g. BEATs: Audio Pre-Training with Acoustic Tokenizers. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, pages 5178–5193. [find](https://scholar.google.com/scholar?q=Sanyuan+Chen%2C+Yu+Wu%2C+Chengyi+Wang%2C+Shujie+Liu%2C+Daniel+Tompkins%2C+Zhuo+Chen%2C+Wanxiang+Che%2C+Xiangzhan+Yu%2C+and+Furu+Wei%2E+2023g%2E+BEATs%3A+Audio+Pre%2DTraining+with+Acoustic+Tokenizers%2E+In+International+Confere)
28. Shaoxiang Chen, Zequn Jie, and Lin Ma. 2024. LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs. arXiv preprint arXiv:2401.16160. — [arXiv:2401.16160](https://arxiv.org/abs/2401.16160) [find](https://scholar.google.com/scholar?q=Shaoxiang+Chen%2C+Zequn+Jie%2C+and+Lin+Ma%2E+2024%2E+LLaVA%2DMoLE%3A+Sparse+Mixture+of+LoRA+Experts+for+Mitigating+Data+Conflicts+in+Instruction+Finetuning+MLLMs%2E+arXiv+preprint+arXiv%3A2401%2E16160%2E+%E2%80%94+%5BarXiv%3A2401%2E16)
29. Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. 2022a. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664–16678. [find](https://scholar.google.com/scholar?q=Shoufa+Chen%2C+Chongjian+Ge%2C+Zhan+Tong%2C+Jiangliu+Wang%2C+Yibing+Song%2C+Jue+Wang%2C+and+Ping+Luo%2E+2022a%2E+Adaptformer%3A+Adapting+vision+transformers+for+scalable+visual+recognition%2E+Advances+in+Neural+Informati)
30. Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. 2023h. PaLI-X: On Scaling up a Multilingual Vision and Language Model. arXiv preprint arXiv:2305.18565. — [arXiv:2305.18565](https://arxiv.org/abs/2305.18565) [find](https://scholar.google.com/scholar?q=Xi+Chen%2C+Josip+Djolonga%2C+Piotr+Padlewski%2C+Basil+Mustafa%2C+Soravit+Changpinyo%2C+Jialin+Wu%2C+Carlos+Riquelme+Ruiz%2C+Sebastian+Goodman%2C+Xiao+Wang%2C+Yi+Tay%2C+et+al%2E+2023h%2E+PaLI%2DX%3A+On+Scaling+up+a+Multilingual+V)
31. Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. 2022b. [Pali: A jointly-scaled multilingual language-image model](https://md.research.elamur.ai/free/en/papers/chen-2022-pali-jointly-scaled-multilingual-language/). arXiv preprint arXiv:2209.06794.
32. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015. [Microsoft coco captions: Data collection and evaluation server](https://md.research.elamur.ai/free/en/papers/chen-2015-microsoft-coco-captions-data-collection/). arXiv preprint arXiv:1504.00325.
33. Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. 2023i. Dress: Instructing large vision-language models to align and interact with humans via natural language feedback. arXiv preprint arXiv:2311.10081. — [arXiv:2311.10081](https://arxiv.org/abs/2311.10081) [find](https://scholar.google.com/scholar?q=Yangyi+Chen%2C+Karan+Sikka%2C+Michael+Cogswell%2C+Heng+Ji%2C+and+Ajay+Divakaran%2E+2023i%2E+Dress%3A+Instructing+large+vision%2Dlanguage+models+to+align+and+interact+with+humans+via+natural+language+feedback%2E+arXiv+p)
34. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2023j. [Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks](https://md.research.elamur.ai/free/en/papers/chen-2023-internvl-scaling-up-vision/). arXiv preprint arXiv:2312.14238.
35. Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. 2023. [Reproducible scaling laws for contrastive language-image learning](https://md.research.elamur.ai/free/en/papers/cherti-2022-reproducible-scaling-laws-contrastive-language/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco…
36. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* Chat-GPT Quality. [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+and+Eric+P%2E+Xing%2E+2023%2E+Vicuna%3A+An+Open%2DSource+Cha)
37. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research, 24(240):1–113.
38. Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. 2023a. Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886. [find](https://scholar.google.com/scholar?q=Xiangxiang+Chu%2C+Limeng+Qiao%2C+Xinyang+Lin%2C+Shuang+Xu%2C+Yang+Yang%2C+Yiming+Hu%2C+Fei+Wei%2C+Xinyu+Zhang%2C+Bo+Zhang%2C+Xiaolin+Wei%2C+et+al%2E+2023a%2E+Mobilevlm%3A+A+fast%2C+reproducible+and+strong+vision+language+assista)
39. Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. 2024. MobileVLM V2: Faster and Stronger Baseline for Vision Language Model. arXiv preprint arXiv:2402.03766. — [arXiv:2402.03766](https://arxiv.org/abs/2402.03766) [find](https://scholar.google.com/scholar?q=Xiangxiang+Chu%2C+Limeng+Qiao%2C+Xinyu+Zhang%2C+Shuang+Xu%2C+Fei+Wei%2C+Yang+Yang%2C+Xiaofei+Sun%2C+Yiming+Hu%2C+Xinyang+Lin%2C+Bo+Zhang%2C+et+al%2E+2024%2E+MobileVLM+V2%3A+Faster+and+Stronger+Baseline+for+Vision+Language+Mode)
40. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023b. [Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models](https://md.research.elamur.ai/free/en/papers/chu-2023-qwen-audio-advancing-universal/). arXiv preprint arXiv:2311.07919.
41. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. [Scaling instruction-finetuned language models](https://md.research.elamur.ai/free/en/papers/chung-2022-scaling-instruction-finetuned-language-models/). arXiv preprint arXiv:2210.11416.
42. XTuner Contributors. 2023. XTuner: A Toolkit for Efficiently Fine-tuning LLM. [https://github.com/](https://github.com/) InternLM/xtuner. [↗](https://github.com/)
43. Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al. 2024. A survey on multimodal large language models for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 958–979… [find](https://scholar.google.com/scholar?q=Can+Cui%2C+Yunsheng+Ma%2C+Xu+Cao%2C+Wenqian+Ye%2C+Yang+Zhou%2C+Kaizhao+Liang%2C+Jintai+Chen%2C+Juanwu+Lu%2C+Zichong+Yang%2C+Kuei%2DDa+Liao%2C+et+al%2E+2024%2E+A+survey+on+multimodal+large+language+models+for+autonomous+driving)
44. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. 2023. [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/). In Thirty-seventh Conference on Neural Information Processing Systems.
45. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Artidoro+Pagnoni%2C+Ari+Holtzman%2C+and+Luke+Zettlemoyer%2E+2023%2E+Qlora%3A+Efficient+finetuning+of+quantized+llms%2E+arXiv+preprint+arXiv%3A2305%2E14314%2E)
46. Jiahua Dong, Hongliu Li, Yang Cong, Gan Sun, Yulun Zhang, and Luc Van Gool. 2024a. No One Left Behind: Real-World Federated Class-Incremental Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(4):2054–2070. [find](https://scholar.google.com/scholar?q=Jiahua+Dong%2C+Hongliu+Li%2C+Yang+Cong%2C+Gan+Sun%2C+Yulun+Zhang%2C+and+Luc+Van+Gool%2E+2024a%2E+No+One+Left+Behind%3A+Real%2DWorld+Federated+Class%2DIncremental+Learning%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machin)
47. Jiahua Dong, Wenqi Liang, Yang Cong, and Gan Sun. 2023a. Heterogeneous forgetting compensation for class-incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11742–11751. [find](https://scholar.google.com/scholar?q=Jiahua+Dong%2C+Wenqi+Liang%2C+Yang+Cong%2C+and+Gan+Sun%2E+2023a%2E+Heterogeneous+forgetting+compensation+for+class%2Dincremental+learning%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Conference+on+Computer+Visio)
48. Jiahua Dong, Lixu Wang, Zhen Fang, Gan Sun, Shichao Xu, Xiao Wang, and Qi Zhu. 2022. Federated Class-Incremental Learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [find](https://scholar.google.com/scholar?q=Jiahua+Dong%2C+Lixu+Wang%2C+Zhen+Fang%2C+Gan+Sun%2C+Shichao+Xu%2C+Xiao+Wang%2C+and+Qi+Zhu%2E+2022%2E+Federated+Class%2DIncremental+Learning%2E+In+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2E)
49. Jiahua Dong, Duzhen Zhang, Yang Cong, Wei Cong, Henghui Ding, and Dengxin Dai. 2023b. Federated Incremental Semantic Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3934–3943. [find](https://scholar.google.com/scholar?q=Jiahua+Dong%2C+Duzhen+Zhang%2C+Yang+Cong%2C+Wei+Cong%2C+Henghui+Ding%2C+and+Dengxin+Dai%2E+2023b%2E+Federated+Incremental+Semantic+Segmentation%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Patt)
50. Linhao Dong and Bo Xu. 2020. Cif: Continuous integrate-and-fire for end-to-end speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6079–6083. IEEE. [find](https://scholar.google.com/scholar?q=Linhao+Dong+and+Bo+Xu%2E+2020%2E+Cif%3A+Continuous+integrate%2Dand%2Dfire+for+end%2Dto%2Dend+speech+recognition%2E+In+ICASSP+2020%2D2020+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29)
51. Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. 2024b. Dreamllm: Synergistic multimodal comprehension and creation. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Runpei+Dong%2C+Chunrui+Han%2C+Yuang+Peng%2C+Zekun+Qi%2C+Zheng+Ge%2C+Jinrong+Yang%2C+Liang+Zhao%2C+Jianjian+Sun%2C+Hongyu+Zhou%2C+Haoran+Wei%2C+et+al%2E+2024b%2E+Dreamllm%3A+Synergistic+multimodal+comprehension+and+creation%2E+In)
52. Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. 2024c. [InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model](https://md.research.elamur.ai/free/en/papers/dong-2024-internlm-xcomposer2-mastering-free/). arXiv preprint arXiv:2401.16420.
53. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). In International Conference on Learni…
54. Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. 2023. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378. [find](https://scholar.google.com/scholar?q=Danny+Driess%2C+Fei+Xia%2C+Mehdi+SM+Sajjadi%2C+Corey+Lynch%2C+Aakanksha+Chowdhery%2C+Brian+Ichter%2C+Ayzaan+Wahid%2C+Jonathan+Tompson%2C+Quan+Vuong%2C+Tianhe+Yu%2C+et+al%2E+2023%2E+Palm%2De%3A+An+embodied+multimodal+language+mod)
55. Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. 2022a. A Survey of Vision-Language Pre-Trained Models. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 5436–5443. [find](https://scholar.google.com/scholar?q=Yifan+Du%2C+Zikang+Liu%2C+Junyi+Li%2C+and+Wayne+Xin+Zhao%2E+2022a%2E+A+Survey+of+Vision%2DLanguage+Pre%2DTrained+Models%2E+In+Proceedings+of+the+Thirty%2DFirst+International+Joint+Conference+on+Artificial+Intelligence%2C)
56. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022b. [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–33…
57. Erich Elsen, Augustus Odena, Maxwell Nye, Sağ-nak Taşırlar, Tri Dao, Curtis Hawthorne, Deepak Moparthi, and Arushi Somani. 2023. Releasing Persimmon-8B. [find](https://scholar.google.com/scholar?q=Erich+Elsen%2C+Augustus+Odena%2C+Maxwell+Nye%2C+Sag%CC%86%2Dnak+Tas%CC%A7%C4%B1rlar%2C+Tri+Dao%2C+Curtis+Hawthorne%2C+Deepak+Moparthi%2C+and+Arushi+Somani%2E+2023%2E+Releasing+Persimmon%2D8B%2E)
58. Yue Fan, Jing Gu, Kaiwen Zhou, Qianqi Yan, Shan Jiang, Ching-Chen Kuo, Xinze Guan, and Xin Eric Wang. 2024. Muffin or Chihuahua? Challenging Large Vision-Language Models with Multipanel VQA. arXiv preprint arXiv:2401.15847. — [arXiv:2401.15847](https://arxiv.org/abs/2401.15847) [find](https://scholar.google.com/scholar?q=Yue+Fan%2C+Jing+Gu%2C+Kaiwen+Zhou%2C+Qianqi+Yan%2C+Shan+Jiang%2C+Ching%2DChen+Kuo%2C+Xinze+Guan%2C+and+Xin+Eric+Wang%2E+2024%2E+Muffin+or+Chihuahua%3F+Challenging+Large+Vision%2DLanguage+Models+with+Multipanel+VQA%2E+arXiv+pre)
59. Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. 2021. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106.11097. 2023b. CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Update. arXiv preprint arXiv:2312.10908. — [arXiv:2106.11097](https://arxiv.org/abs/2106.11097) [find](https://scholar.google.com/scholar?q=Han+Fang%2C+Pengfei+Xiong%2C+Luhui+Xu%2C+and+Yu+Chen%2E+2021%2E+Clip2video%3A+Mastering+video%2Dtext+retrieval+via+image+clip%2E+arXiv+preprint+arXiv%3A2106%2E11097%2E+2023b%2E+CLOVA%3A+A+Closed%2DLoop+Visual+Assistant+with+Tool)
60. Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. 2023. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19358– 19369. [find](https://scholar.google.com/scholar?q=Yuxin+Fang%2C+Wen+Wang%2C+Binhui+Xie%2C+Quan+Sun%2C+Ledell+Wu%2C+Xinggang+Wang%2C+Tiejun+Huang%2C+Xinlong+Wang%2C+and+Yue+Cao%2E+2023%2E+Eva%3A+Exploring+the+limits+of+masked+visual+representation+learning+at+scale%2E+In+Pro)
61. Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. 2023. Planting a seed of vision in large language model. arXiv preprint arXiv:2307.08041. [find](https://scholar.google.com/scholar?q=Yuying+Ge%2C+Yixiao+Ge%2C+Ziyun+Zeng%2C+Xintao+Wang%2C+and+Ying+Shan%2E+2023%2E+Planting+a+seed+of+vision+in+large+language+model%2E+arXiv+preprint+arXiv%3A2307%2E08041%2E)
62. Hao Feng, Qi Liu, Hao Liu, Wengang Zhou, Houqiang Li, and Can Huang. 2023. DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding. arXiv preprint arXiv:2311.11810. — [arXiv:2311.11810](https://arxiv.org/abs/2311.11810) [find](https://scholar.google.com/scholar?q=Hao+Feng%2C+Qi+Liu%2C+Hao+Liu%2C+Wengang+Zhou%2C+Houqiang+Li%2C+and+Can+Huang%2E+2023%2E+DocPedia%3A+Unleashing+the+Power+of+Large+Multimodal+Model+in+the+Frequency+Domain+for+Versatile+Document+Understanding%2E+arXiv+)
63. Roya Firoozi, Johnathan Tucker, Stephen Tian, Anirudha Majumdar, Jiankai Sun, Weiyu Liu, Yuke Zhu, Shuran Song, Ashish Kapoor, Karol Hausman, et al. 2023. Foundation Models in Robotics: Applications, Challenges, and the Future. arXiv preprint arXiv:2312.07843. — [arXiv:2312.07843](https://arxiv.org/abs/2312.07843) [find](https://scholar.google.com/scholar?q=Roya+Firoozi%2C+Johnathan+Tucker%2C+Stephen+Tian%2C+Anirudha+Majumdar%2C+Jiankai+Sun%2C+Weiyu+Liu%2C+Yuke+Zhu%2C+Shuran+Song%2C+Ashish+Kapoor%2C+Karol+Hausman%2C+et+al%2E+2023%2E+Foundation+Models+in+Robotics%3A+Applications%2C+)
64. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. 2023. [Mme: A comprehensive evaluation benchmark for multimodal large language models](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/). arXiv preprint arXiv:2306.13394.
65. Chin-Lun Fu, Zih-Ching Chen, Yun-Ru Lee, and Hung-Yi Lee. 2022. AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2608–2621. [find](https://scholar.google.com/scholar?q=Chin%2DLun+Fu%2C+Zih%2DChing+Chen%2C+Yun%2DRu+Lee%2C+and+Hung%2DYi+Lee%2E+2022%2E+AdapterBias%3A+Parameter%2Defficient+Token%2Ddependent+Representation+Shift+for+Adapters+in+NLP+Tasks%2E+In+Findings+of+the+Association+for+Comp)
66. Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. 2023. Datacomp: In search of the next generation of multimodal datasets. arXiv preprint arXiv:2304.14108. [find](https://scholar.google.com/scholar?q=Samir+Yitzhak+Gadre%2C+Gabriel+Ilharco%2C+Alex+Fang%2C+Jonathan+Hayase%2C+Georgios+Smyrnis%2C+Thao+Nguyen%2C+Ryan+Marten%2C+Mitchell+Wortsman%2C+Dhruba+Ghosh%2C+Jieyu+Zhang%2C+et+al%2E+2023%2E+Datacomp%3A+In+search+of+the+next)
67. Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2023. [Bias and fairness in large language models: A survey](https://md.research.elamur.ai/free/en/papers/gallegos-2023-bias-fairness-large-language-models/). arXiv preprint arXiv:2309.00770.
68. Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, et al. 2024. SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models. arXiv preprint arXiv:2402.05935. — [arXiv:2402.05935](https://arxiv.org/abs/2402.05935) [find](https://scholar.google.com/scholar?q=Peng+Gao%2C+Renrui+Zhang%2C+Chris+Liu%2C+Longtian+Qiu%2C+Siyuan+Huang%2C+Weifeng+Lin%2C+Shitian+Zhao%2C+Shijie+Geng%2C+Ziyi+Lin%2C+Peng+Jin%2C+et+al%2E+2024%2E+SPHINX%2DX%3A+Scaling+Data+and+Parameters+for+a+Family+of+Multi%2Dmoda)
69. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023a. [Retrieval-augmented generation for large language models: A survey](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/). arXiv preprint arXiv:2312.10997.
70. Zhi Gao, Yuntao Du, Xintong Zhang, Xiaojian Ma, Wenjuan Han, Song-Chun Zhu, and Qing Li. [find](https://scholar.google.com/scholar?q=Zhi+Gao%2C+Yuntao+Du%2C+Xintong+Zhang%2C+Xiaojian+Ma%2C+Wenjuan+Han%2C+Song%2DChun+Zhu%2C+and+Qing+Li%2E)
71. Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. [Imagebind: One embedding space to bind them all](https://md.research.elamur.ai/free/en/papers/girdhar-2023-imagebind-one-embedding-space-bind/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190.
72. Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. 2023. Multimodal-gpt: A vision and language model for dialogue with humans. arXiv preprint arXiv:2305.04790. — [arXiv:2305.04790](https://arxiv.org/abs/2305.04790) [find](https://scholar.google.com/scholar?q=Tao+Gong%2C+Chengqi+Lyu%2C+Shilong+Zhang%2C+Yudong+Wang%2C+Miao+Zheng%2C+Qian+Zhao%2C+Kuikun+Liu%2C+Wenwei+Zhang%2C+Ping+Luo%2C+and+Kai+Chen%2E+2023%2E+Multimodal%2Dgpt%3A+A+vision+and+language+model+for+dialogue+with+humans%2E+)
73. Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. 2013. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211. [find](https://scholar.google.com/scholar?q=Ian+J+Goodfellow%2C+Mehdi+Mirza%2C+Da+Xiao%2C+Aaron+Courville%2C+and+Yoshua+Bengio%2E+2013%2E+An+empirical+investigation+of+catastrophic+forgetting+in+gradient%2Dbased+neural+networks%2E+arXiv+preprint+arXiv%3A1312%2E621)
74. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. [Making the v in vqa matter: Elevating the role of image understanding in visual question answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913.
75. Jiaxi Gu, Xiaojun Meng, Guansong Lu, Lu Hou, Niu Minzhe, Xiaodan Liang, Lewei Yao, Runhui Huang, Wei Zhang, Xin Jiang, et al. 2022. Wukong: A 100 million large-scale chinese cross-modal pre-training benchmark. Advances in Neural Information Processing Systems, 35:26418–26431. [find](https://scholar.google.com/scholar?q=Jiaxi+Gu%2C+Xiaojun+Meng%2C+Guansong+Lu%2C+Lu+Hou%2C+Niu+Minzhe%2C+Xiaodan+Liang%2C+Lewei+Yao%2C+Runhui+Huang%2C+Wei+Zhang%2C+Xin+Jiang%2C+et+al%2E+2022%2E+Wukong%3A+A+100+million+large%2Dscale+chinese+cross%2Dmodal+pre%2Dtraining+b)
76. Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617. [find](https://scholar.google.com/scholar?q=Danna+Gurari%2C+Qing+Li%2C+Abigale+J+Stangl%2C+Anhong+Guo%2C+Chi+Lin%2C+Kristen+Grauman%2C+Jiebo+Luo%2C+and+Jeffrey+P+Bigham%2E+2018%2E+Vizwiz+grand+challenge%3A+Answering+visual+questions+from+blind+people%2E+In+Proceedin)
77. Minglun Han, Feilong Chen, Jing Shi, Shuang Xu, and Bo Xu. 2023. Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation. arXiv preprint arXiv:2301.13003. — [arXiv:2301.13003](https://arxiv.org/abs/2301.13003) [find](https://scholar.google.com/scholar?q=Minglun+Han%2C+Feilong+Chen%2C+Jing+Shi%2C+Shuang+Xu%2C+and+Bo+Xu%2E+2023%2E+Knowledge+Transfer+from+Pre%2Dtrained+Language+Models+to+Cif%2Dbased+Speech+Recognizers+via+Hierarchical+Distillation%2E+arXiv+preprint+arXiv)
78. Minglun Han, Linhao Dong, Zhenlin Liang, Meng Cai, Shiyu Zhou, Zejun Ma, and Bo Xu. 2022. Improving end-to-end contextual speech recognition with fine-grained contextual knowledge selection. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages… [find](https://scholar.google.com/scholar?q=Minglun+Han%2C+Linhao+Dong%2C+Zhenlin+Liang%2C+Meng+Cai%2C+Shiyu+Zhou%2C+Zejun+Ma%2C+and+Bo+Xu%2E+2022%2E+Improving+end%2Dto%2Dend+contextual+speech+recognition+with+fine%2Dgrained+contextual+knowledge+selection%2E+In+ICASSP)
79. Zongbo Han, Zechen Bai, Haiyang Mei, Qianli Xu, Changqing Zhang, and Mike Zheng Shou. 2024. Skip \\n: A simple method to reduce hallucination in large vision-language models. arXiv preprint arXiv:2402.01345. — [arXiv:2402.01345](https://arxiv.org/abs/2402.01345) [find](https://scholar.google.com/scholar?q=Zongbo+Han%2C+Zechen+Bai%2C+Haiyang+Mei%2C+Qianli+Xu%2C+Changqing+Zhang%2C+and+Mike+Zheng+Shou%2E+2024%2E+Skip+%5C%5Cn%3A+A+simple+method+to+reduce+hallucination+in+large+vision%2Dlanguage+models%2E+arXiv+preprint+arXiv%3A2402)
80. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://md.research.elamur.ai/free/en/papers/he-2024-webvoyager-building-end-end-web/). arXiv preprint arXiv:2401.13919.
81. Anwen Hu, Yaya Shi, Haiyang Xu, Jiabo Ye, Qinghao Ye, Ming Yan, Chenliang Li, Qi Qian, Ji Zhang, and Fei Huang. 2023a. mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Model. arXiv preprint arXiv:2311.18248. — [arXiv:2311.18248](https://arxiv.org/abs/2311.18248) [find](https://scholar.google.com/scholar?q=Anwen+Hu%2C+Yaya+Shi%2C+Haiyang+Xu%2C+Jiabo+Ye%2C+Qinghao+Ye%2C+Ming+Yan%2C+Chenliang+Li%2C+Qi+Qian%2C+Ji+Zhang%2C+and+Fei+Huang%2E+2023a%2E+mPLUG%2DPaperOwl%3A+Scientific+Diagram+Analysis+with+the+Multimodal+Large+Language+Mo)
82. Jinghan He, Haiyun Guo, Ming Tang, and Jinqiao Wang. 2023. Continual instruction tuning for large multimodal models. arXiv preprint arXiv:2311.16206. — [arXiv:2311.16206](https://arxiv.org/abs/2311.16206) [find](https://scholar.google.com/scholar?q=Jinghan+He%2C+Haiyun+Guo%2C+Ming+Tang%2C+and+Jinqiao+Wang%2E+2023%2E+Continual+instruction+tuning+for+large+multimodal+models%2E+arXiv+preprint+arXiv%3A2311%2E16206%2E+%E2%80%94+%5BarXiv%3A2311%2E16206%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E16)
83. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021. [LoRA: Low-Rank Adaptation of Large Language Models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations.
84. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2021. [Towards a Unified View of Parameter-Efficient Transfer Learning](https://md.research.elamur.ai/free/en/papers/he-2021-towards-unified-view-parameter-efficient/). In International Conference on Learning Representations.
85. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. [Masked autoencoders are scalable vision learners](https://md.research.elamur.ai/free/en/papers/he-2021-masked-autoencoders-scalable-vision-learners/). In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009.
86. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+2016%2E+Deep+residual+learning+for+image+recognition%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pages+770%E2%80%93+)
87. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). arXiv preprint arXiv:2203.15556.
88. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. 2023a. [Cogagent: A visual language model for gui agents](https://md.research.elamur.ai/free/en/papers/hong-2023-cogagent-visual-language-model/). arXiv preprint arXiv:2312.08914.
89. Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 2023b. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482–20494. [find](https://scholar.google.com/scholar?q=Yining+Hong%2C+Haoyu+Zhen%2C+Peihao+Chen%2C+Shuhong+Zheng%2C+Yilun+Du%2C+Zhenfang+Chen%2C+and+Chuang+Gan%2E+2023b%2E+3d%2Dllm%3A+Injecting+the+3d+world+into+large+language+models%2E+Advances+in+Neural+Information+Processin)
90. Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022. [Unnatural instructions: Tuning language models with (almost) no human labor](https://md.research.elamur.ai/free/en/papers/honovich-2023-unnatural-instructions-tuning-language-models/). arXiv preprint arXiv:2212.09689.
91. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning, pages 2790–2799. PMLR. [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+De+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+NLP%2E+In+)
92. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. [Hubert: Self-supervised speech representation learning by masked prediction of hidden units](https://md.research.elamur.ai/free/en/papers/hsu-2021-hubert-self-supervised-speech-representation/). IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460.
93. Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, et al. 2023b. Large multilingual models pivot zero-shot multimodal learning across languages. arXiv preprint arXiv:2308.12038. [find](https://scholar.google.com/scholar?q=Jinyi+Hu%2C+Yuan+Yao%2C+Chongyi+Wang%2C+Shan+Wang%2C+Yinxu+Pan%2C+Qianyu+Chen%2C+Tianyu+Yu%2C+Hanghao+Wu%2C+Yue+Zhao%2C+Haoye+Zhang%2C+et+al%2E+2023b%2E+Large+multilingual+models+pivot+zero%2Dshot+multimodal+learning+across+la)
94. Jiaxing Huang, Jingyi Zhang, Kai Jiang, Han Qiu, and Shijian Lu. 2023a. Visual Instruction Tuning towards General-Purpose Multimodal Model: A Survey. arXiv preprint arXiv:2312.16602. — [arXiv:2312.16602](https://arxiv.org/abs/2312.16602) [find](https://scholar.google.com/scholar?q=Jiaxing+Huang%2C+Jingyi+Zhang%2C+Kai+Jiang%2C+Han+Qiu%2C+and+Shijian+Lu%2E+2023a%2E+Visual+Instruction+Tuning+towards+General%2DPurpose+Multimodal+Model%3A+A+Survey%2E+arXiv+preprint+arXiv%3A2312%2E16602%2E+%E2%80%94+%5BarXiv%3A2312%2E166)
95. Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al. 2023b. Audiogpt: Understanding and generating speech, music, sound, and talking head. arXiv preprint arXiv:2304.12995. — [arXiv:2304.12995](https://arxiv.org/abs/2304.12995) [find](https://scholar.google.com/scholar?q=Rongjie+Huang%2C+Mingze+Li%2C+Dongchao+Yang%2C+Jiatong+Shi%2C+Xuankai+Chang%2C+Zhenhui+Ye%2C+Yuning+Wu%2C+Zhiqing+Hong%2C+Jiawei+Huang%2C+Jinglin+Liu%2C+et+al%2E+2023b%2E+Audiogpt%3A+Understanding+and+generating+speech%2C+music%2C)
96. Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. 2023c. [Language is not all you need: Aligning perception with language models](https://md.research.elamur.ai/free/en/papers/huang-2023-language-all-need-aligning-perception/). arXiv preprint arXiv:2302.14045.
97. Drew A Hudson and Christopher D Manning. 2019. [Gqa: A new dataset for real-world visual reasoning and compositional question answering](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/). In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709.
98. IDEFICS. 2023. Introducing IDEFICS: An Open Reproduction of State-of-the-Art Visual Language Model. [find](https://scholar.google.com/scholar?q=IDEFICS%2E+2023%2E+Introducing+IDEFICS%3A+An+Open+Reproduction+of+State%2Dof%2Dthe%2DArt+Visual+Language+Model%2E)
99. Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. 2022. [Opt-iml: Scaling language model instruction meta learning through the lens of generalization](https://md.research.elamur.ai/free/en/papers/iyer-2022-opt-iml-scaling-language-model/). arXiv preprint arXiv:2212.12017.
100. Jitesh Jain, Jianwei Yang, and Humphrey Shi. 2023. Vcoder: Versatile vision encoders for multimodal large language models. arXiv preprint arXiv:2312.14233. — [arXiv:2312.14233](https://arxiv.org/abs/2312.14233) [find](https://scholar.google.com/scholar?q=Jitesh+Jain%2C+Jianwei+Yang%2C+and+Humphrey+Shi%2E+2023%2E+Vcoder%3A+Versatile+vision+encoders+for+multimodal+large+language+models%2E+arXiv+preprint+arXiv%3A2312%2E14233%2E+%E2%80%94+%5BarXiv%3A2312%2E14233%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
101. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. [Scaling up visual and vision-language representation learning with noisy text supervision](https://md.research.elamur.ai/free/en/papers/jia-2021-scaling-up-visual-vision-language/). In International conference on machine learning, pages 4904–4916. PMLR.
102. Yiren Jian, Chongyang Gao, and Soroush Vosoughi. 2023. Bootstrapping Vision-Language Learning with Decoupled Language Pre-training. In Thirty-seventh Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Yiren+Jian%2C+Chongyang+Gao%2C+and+Soroush+Vosoughi%2E+2023%2E+Bootstrapping+Vision%2DLanguage+Learning+with+Decoupled+Language+Pre%2Dtraining%2E+In+Thirty%2Dseventh+Conference+on+Neural+Information+Processing+System)
103. Yang Jin, Zhicheng Sun, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, et al. 2024a. Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization. arXiv preprint arXiv:2402.03161. — [arXiv:2402.03161](https://arxiv.org/abs/2402.03161) [find](https://scholar.google.com/scholar?q=Yang+Jin%2C+Zhicheng+Sun%2C+Kun+Xu%2C+Liwei+Chen%2C+Hao+Jiang%2C+Quzhe+Huang%2C+Chengru+Song%2C+Yuliang+Liu%2C+Di+Zhang%2C+Yang+Song%2C+et+al%2E+2024a%2E+Video%2DLaVIT%3A+Unified+Video%2DLanguage+Pre%2Dtraining+with+Decoupled+Visual)
104. Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Bin Chen, Chenyi Lei, An Liu, Chengru Song, Xiaoqiang Lei, et al. 2024b. Unified language-vision pretraining with dynamic discrete visual tokenization. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yang+Jin%2C+Kun+Xu%2C+Liwei+Chen%2C+Chao+Liao%2C+Jianchao+Tan%2C+Bin+Chen%2C+Chenyi+Lei%2C+An+Liu%2C+Chengru+Song%2C+Xiaoqiang+Lei%2C+et+al%2E+2024b%2E+Unified+language%2Dvision+pretraining+with+dynamic+discrete+visual+tokeniz)
105. Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. 2018. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648–5656. [find](https://scholar.google.com/scholar?q=Kushal+Kafle%2C+Brian+Price%2C+Scott+Cohen%2C+and+Christopher+Kanan%2E+2018%2E+Dvqa%3A+Understanding+data+visualizations+via+question+answering%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+patter)
106. Mintong Kang, Nezihe Merve Gürel, Ning Yu, Dawn Song, and Bo Li. 2024. C-RAG: Certified Generation Risks for Retrieval-Augmented Language Models. arXiv preprint arXiv:2402.03181. — [arXiv:2402.03181](https://arxiv.org/abs/2402.03181) [find](https://scholar.google.com/scholar?q=Mintong+Kang%2C+Nezihe+Merve+G%C3%BCrel%2C+Ning+Yu%2C+Dawn+Song%2C+and+Bo+Li%2E+2024%2E+C%2DRAG%3A+Certified+Generation+Risks+for+Retrieval%2DAugmented+Language+Models%2E+arXiv+preprint+arXiv%3A2402%2E03181%2E+%E2%80%94+%5BarXiv%3A2402%2E03181%5D%28)
107. Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34:1022–1035. [find](https://scholar.google.com/scholar?q=Rabeeh+Karimi+Mahabadi%2C+James+Henderson%2C+and+Sebastian+Ruder%2E+2021%2E+Compacter%3A+Efficient+low%2Drank+hypercomplex+adapter+layers%2E+Advances+in+Neural+Information+Processing+Systems%2C+34%3A1022%E2%80%931035%2E)
108. Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. [Referitgame: Referring to objects in photographs of natural scenes](https://md.research.elamur.ai/free/en/papers/kazemzadeh-2014-referitgame-referring-objects-photographs-natural/). In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787– 798.
109. Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. 2023a. Generating images with multimodal language models. In Thirty-seventh Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Jing+Yu+Koh%2C+Daniel+Fried%2C+and+Ruslan+Salakhutdinov%2E+2023a%2E+Generating+images+with+multimodal+language+models%2E+In+Thirty%2Dseventh+Conference+on+Neural+Information+Processing+Systems%2E)
110. Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. 2023b. Grounding language models to images for multimodal inputs and outputs. In International Conference on Machine Learning, pages 17283–17300. PMLR. [find](https://scholar.google.com/scholar?q=Jing+Yu+Koh%2C+Ruslan+Salakhutdinov%2C+and+Daniel+Fried%2E+2023b%2E+Grounding+language+models+to+images+for+multimodal+inputs+and+outputs%2E+In+International+Conference+on+Machine+Learning%2C+pages+17283%E2%80%9317300%2E+P)
111. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. [Visual genome: Connecting language and vision using crowdsourced dense image annotations](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/). International journal of computer vision, 123:32–73…
112. Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. 2023. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308.00692. — [arXiv:2308.00692](https://arxiv.org/abs/2308.00692) [find](https://scholar.google.com/scholar?q=Xin+Lai%2C+Zhuotao+Tian%2C+Yukang+Chen%2C+Yanwei+Li%2C+Yuhui+Yuan%2C+Shu+Liu%2C+and+Jiaya+Jia%2E+2023%2E+Lisa%3A+Reasoning+segmentation+via+large+language+model%2E+arXiv+preprint+arXiv%3A2308%2E00692%2E+%E2%80%94+%5BarXiv%3A2308%2E00692%5D%28ht)
113. Hugo Laurençon, Lucile Saulnier, Leo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M Rush, Douwe Kiela, et al. 2023. [OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2023-obelics-open-web-scale-filtered/). In Thirty-seventh Conference on Neural Informati…
114. Seongyun Lee, Sue Hyun Park, Yongrae Jo, and Minjoon Seo. 2023. Volcano: mitigating multimodal hallucination through self-feedback guided revision. arXiv preprint arXiv:2311.07362. [find](https://scholar.google.com/scholar?q=Seongyun+Lee%2C+Sue+Hyun+Park%2C+Yongrae+Jo%2C+and+Minjoon+Seo%2E+2023%2E+Volcano%3A+mitigating+multimodal+hallucination+through+self%2Dfeedback+guided+revision%2E+arXiv+preprint+arXiv%3A2311%2E07362%2E)
115. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The Power of Scale for Parameter-Efficient Prompt Tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059. [find](https://scholar.google.com/scholar?q=Brian+Lester%2C+Rami+Al%2DRfou%2C+and+Noah+Constant%2E+2021%2E+The+Power+of+Scale+for+Parameter%2DEfficient+Prompt+Tuning%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processing)
116. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of NAACL-HLT, pages 4171–4186.
117. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. 2023a. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425. — [arXiv:2306.05425](https://arxiv.org/abs/2306.05425) [find](https://scholar.google.com/scholar?q=Bo+Li%2C+Yuanhan+Zhang%2C+Liangyu+Chen%2C+Jinghao+Wang%2C+Fanyi+Pu%2C+Jingkang+Yang%2C+Chunyuan+Li%2C+and+Ziwei+Liu%2E+2023a%2E+Mimic%2Dit%3A+Multi%2Dmodal+in%2Dcontext+instruction+tuning%2E+arXiv+preprint+arXiv%3A2306%2E05425%2E+%E2%80%94+%5Ba)
118. Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020. [The hateful memes challenge: Detecting hate speech in multimodal memes](https://md.research.elamur.ai/free/en/papers/kiela-2020-hateful-memes-challenge-detecting-hate/). Advances in neural information processing systems, 33:2611–2624.
119. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. 2023b. [Otter: A multi-modal model with in-context instruction tuning](https://md.research.elamur.ai/free/en/papers/li-2023-otter-multi-modal-model-context/). arXiv preprint arXiv:2305.03726.
120. Diederik P Kingma and Max Welling. 2013. [Autoencoding variational bayes](https://md.research.elamur.ai/free/en/papers/kingma-2013-auto-encoding-variational-bayes/). arXiv preprint arXiv:1312.6114.
121. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023. [Segment anything](https://md.research.elamur.ai/free/en/papers/kirillov-2023-segment-anything/). arXiv preprint arXiv:2304.02643.
122. Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023c. [Seed-bench: Benchmarking multimodal llms with generative comprehension](https://md.research.elamur.ai/free/en/papers/li-2023-seed-bench-benchmarking-multimodal-llms/). arXiv preprint arXiv:2307.16125.
123. Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2023d. [Llava-med: Training a large language-and-vision assistant for biomedicine in one day](https://md.research.elamur.ai/free/en/papers/li-2023-llava-med-training-large-language/). arXiv preprint arXiv:2306.00890.
124. Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023e. [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, pages 19730–19742.
125. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. [Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/). In International Conference on Machine Learning, pages 12888–12900. PMLR.
126. Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694–9705. [find](https://scholar.google.com/scholar?q=Junnan+Li%2C+Ramprasaath+Selvaraju%2C+Akhilesh+Gotmare%2C+Shafiq+Joty%2C+Caiming+Xiong%2C+and+Steven+Chu+Hong+Hoi%2E+2021%2E+Align+before+fuse%3A+Vision+and+language+representation+learning+with+momentum+distillation)
127. Kaixin Li, Yuchen Tian, Qisheng Hu, Ziyang Luo, and Jing Ma. 2024a. MMCode: Evaluating Multi-Modal Code Large Language Models with Visually Rich Programming Problems. ArXiv, abs/2404.09486. — [arXiv:2404.09486](https://arxiv.org/abs/2404.09486) [find](https://scholar.google.com/scholar?q=Kaixin+Li%2C+Yuchen+Tian%2C+Qisheng+Hu%2C+Ziyang+Luo%2C+and+Jing+Ma%2E+2024a%2E+MMCode%3A+Evaluating+Multi%2DModal+Code+Large+Language+Models+with+Visually+Rich+Programming+Problems%2E+ArXiv%2C+abs%2F2404%2E09486%2E+%E2%80%94+%5BarXiv%3A2)
128. KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023f. [Videochat: Chat-centric video understanding](https://md.research.elamur.ai/free/en/papers/li-2023-videochat-chat-centric-video-understanding/). arXiv preprint arXiv:2305.06355.
129. Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. 2024b. [Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/li-2024-multimodal-arxiv-dataset-improving-scientific/). arXiv preprint arXiv:2403.00231.
130. Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. 2023g. Silkie: Preference Distillation for Large Visual Language Models. arXiv preprint arXiv:2312.10665. — [arXiv:2312.10665](https://arxiv.org/abs/2312.10665) [find](https://scholar.google.com/scholar?q=Lei+Li%2C+Zhihui+Xie%2C+Mukai+Li%2C+Shunian+Chen%2C+Peiyi+Wang%2C+Liang+Chen%2C+Yazheng+Yang%2C+Benyou+Wang%2C+and+Lingpeng+Kong%2E+2023g%2E+Silkie%3A+Preference+Distillation+for+Large+Visual+Language+Models%2E+arXiv+preprin)
131. Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al. 2023h. M 3 IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning. arXiv preprint arXiv:2306.04387. — [arXiv:2306.04387](https://arxiv.org/abs/2306.04387) [find](https://scholar.google.com/scholar?q=Lei+Li%2C+Yuwei+Yin%2C+Shicheng+Li%2C+Liang+Chen%2C+Peiyi+Wang%2C+Shuhuai+Ren%2C+Mukai+Li%2C+Yazheng+Yang%2C+Jingjing+Xu%2C+Xu+Sun%2C+et+al%2E+2023h%2E+M+3+IT%3A+A+Large%2DScale+Dataset+towards+Multi%2DModal+Multilingual+Instructi)
132. Mukai Li, Lei Li, Yuwei Yin, Masood Ahmed, Zhenguang Liu, and Qi Liu. 2024c. Red teaming visual language models. arXiv preprint arXiv:2401.12915. [find](https://scholar.google.com/scholar?q=Mukai+Li%2C+Lei+Li%2C+Yuwei+Yin%2C+Masood+Ahmed%2C+Zhenguang+Liu%2C+and+Qi+Liu%2E+2024c%2E+Red+teaming+visual+language+models%2E+arXiv+preprint+arXiv%3A2401%2E12915%2E)
133. Xiang Lisa Li and Percy Liang. 2021. Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–… [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2DTuning%3A+Optimizing+Continuous+Prompts+for+Generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11t)
134. Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020. [Oscar: Object-semantics aligned pre-training for vision-language tasks](https://md.research.elamur.ai/free/en/papers/li-2020-oscar-object-semantics-aligned-pre/). In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceeding…
135. Yanda Li, Chi Zhang, Gang Yu, Zhibin Wang, Bin Fu, Guosheng Lin, Chunhua Shen, Ling Chen, and Yunchao Wei. 2023i. Stablellava: Enhanced visual instruction tuning with synthesized image-dialogue data. arXiv preprint arXiv:2308.10253. [find](https://scholar.google.com/scholar?q=Yanda+Li%2C+Chi+Zhang%2C+Gang+Yu%2C+Zhibin+Wang%2C+Bin+Fu%2C+Guosheng+Lin%2C+Chunhua+Shen%2C+Ling+Chen%2C+and+Yunchao+Wei%2E+2023i%2E+Stablellava%3A+Enhanced+visual+instruction+tuning+with+synthesized+image%2Ddialogue+data%2E+)
136. Yanwei Li, Chengyao Wang, and Jiaya Jia. 2023j. LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models. arXiv preprint arXiv:2311.17043. [find](https://scholar.google.com/scholar?q=Yanwei+Li%2C+Chengyao+Wang%2C+and+Jiaya+Jia%2E+2023j%2E+LLaMA%2DVID%3A+An+Image+is+Worth+2+Tokens+in+Large+Language+Models%2E+arXiv+preprint+arXiv%3A2311%2E17043%2E)
137. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023k. [Evaluating object hallucination in large vision-language models](https://md.research.elamur.ai/free/en/papers/li-2023-evaluating-object-hallucination-large-vision/). arXiv preprint arXiv:2305.10355.
138. Zeju Li, Chao Zhang, Xiaoyan Wang, Ruilong Ren, Yifan Xu, Ruifei Ma, and Xiangde Liu. 2024d. 3DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding. arXiv preprint arXiv:2401.03201. — [arXiv:2401.03201](https://arxiv.org/abs/2401.03201) [find](https://scholar.google.com/scholar?q=Zeju+Li%2C+Chao+Zhang%2C+Xiaoyan+Wang%2C+Ruilong+Ren%2C+Yifan+Xu%2C+Ruifei+Ma%2C+and+Xiangde+Liu%2E+2024d%2E+3DMIT%3A+3D+Multi%2Dmodal+Instruction+Tuning+for+Scene+Understanding%2E+arXiv+preprint+arXiv%3A2401%2E03201%2E+%E2%80%94+%5BarXiv)
139. Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. 2023l. Monkey: Image Resolution and Text Label Are Important Things for Large Multimodal Models. arXiv preprint arXiv:2311.06607. [find](https://scholar.google.com/scholar?q=Zhang+Li%2C+Biao+Yang%2C+Qiang+Liu%2C+Zhiyin+Ma%2C+Shuo+Zhang%2C+Jingxu+Yang%2C+Yabo+Sun%2C+Yuliang+Liu%2C+and+Xiang+Bai%2E+2023l%2E+Monkey%3A+Image+Resolution+and+Text+Label+Are+Important+Things+for+Large+Multimodal+Model)
140. Zhaowei Li, Qi Xu, Dong Zhang, Hang Song, Yiqing Cai, Qi Qi, Ran Zhou, Junting Pan, Zefeng Li, Van Tu Vu, et al. 2024e. LEGO: Language Enhanced Multi-modal Grounding Model. arXiv preprint arXiv:2401.06071. — [arXiv:2401.06071](https://arxiv.org/abs/2401.06071) [find](https://scholar.google.com/scholar?q=Zhaowei+Li%2C+Qi+Xu%2C+Dong+Zhang%2C+Hang+Song%2C+Yiqing+Cai%2C+Qi+Qi%2C+Ran+Zhou%2C+Junting+Pan%2C+Zefeng+Li%2C+Van+Tu+Vu%2C+et+al%2E+2024e%2E+LEGO%3A+Language+Enhanced+Multi%2Dmodal+Grounding+Model%2E+arXiv+preprint+arXiv%3A2401%2E0)
141. Zhong-Zhi Li, Ming-Liang Zhang, Fei Yin, and Cheng-Lin Liu. 2024f. LANS: A Layout-Aware Neural Solver for Plane Geometry Problem. [find](https://scholar.google.com/scholar?q=Zhong%2DZhi+Li%2C+Ming%2DLiang+Zhang%2C+Fei+Yin%2C+and+Cheng%2DLin+Liu%2E+2024f%2E+LANS%3A+A+Layout%2DAware+Neural+Solver+for+Plane+Geometry+Problem%2E)
142. Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. 2024a. MoE-LLaVA: Mixture of Experts for Large Vision-Language Models. arXiv preprint arXiv:2401.15947. — [arXiv:2401.15947](https://arxiv.org/abs/2401.15947) [find](https://scholar.google.com/scholar?q=Bin+Lin%2C+Zhenyu+Tang%2C+Yang+Ye%2C+Jiaxi+Cui%2C+Bin+Zhu%2C+Peng+Jin%2C+Junwu+Zhang%2C+Munan+Ning%2C+and+Li+Yuan%2E+2024a%2E+MoE%2DLLaVA%3A+Mixture+of+Experts+for+Large+Vision%2DLanguage+Models%2E+arXiv+preprint+arXiv%3A2401%2E1594)
143. Hongzhan Lin, Ziyang Luo, Bo Wang, Ruichao Yang, and Jing Ma. 2024b. GOAT-Bench: Safety Insights to Large Multimodal Models through Meme-Based Social Abuse. arXiv preprint arXiv:2401.01523. — [arXiv:2401.01523](https://arxiv.org/abs/2401.01523) [find](https://scholar.google.com/scholar?q=Hongzhan+Lin%2C+Ziyang+Luo%2C+Bo+Wang%2C+Ruichao+Yang%2C+and+Jing+Ma%2E+2024b%2E+GOAT%2DBench%3A+Safety+Insights+to+Large+Multimodal+Models+through+Meme%2DBased+Social+Abuse%2E+arXiv+preprint+arXiv%3A2401%2E01523%2E+%E2%80%94+%5BarXiv%3A2)
144. Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. 2023. [VILA: On Pre-training for Visual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2023-vila-pre-training-visual-language/). arXiv preprint arXiv:2312.07533.
145. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pag…
146. LinkSoul-AI. 2023. Chinese-LLaVA. [find](https://scholar.google.com/scholar?q=LinkSoul%2DAI%2E+2023%2E+Chinese%2DLLaVA%2E)
147. Fangyu Liu, Guy Emerson, and Nigel Collier. 2023a. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635–651. [find](https://scholar.google.com/scholar?q=Fangyu+Liu%2C+Guy+Emerson%2C+and+Nigel+Collier%2E+2023a%2E+Visual+spatial+reasoning%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+11%3A635%E2%80%93651%2E)
148. Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024a. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253. [find](https://scholar.google.com/scholar?q=Hanchao+Liu%2C+Wenyuan+Xue%2C+Yifei+Chen%2C+Dapeng+Chen%2C+Xiutian+Zhao%2C+Ke+Wang%2C+Liping+Hou%2C+Rongjun+Li%2C+and+Wei+Peng%2E+2024a%2E+A+survey+on+hallucination+in+large+vision%2Dlanguage+models%2E+arXiv+preprint+arXiv%3A2)
149. Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo P. Mandic, Wenwu Wang, and Mark D. Plumbley. 2023b. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, pages 21450– 21474. [find](https://scholar.google.com/scholar?q=Haohe+Liu%2C+Zehua+Chen%2C+Yi+Yuan%2C+Xinhao+Mei%2C+Xubo+Liu%2C+Danilo+P%2E+Mandic%2C+Wenwu+Wang%2C+and+Mark+D%2E+Plumbley%2E+2023b%2E+AudioLDM%3A+Text%2Dto%2DAudio+Generation+with+Latent+Diffusion+Models%2E+In+International+Confe)
150. Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D. Plumbley. 2023c. AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining. CoRR, abs/2308.05734. — [arXiv:2308.05734](https://arxiv.org/abs/2308.05734) [find](https://scholar.google.com/scholar?q=Haohe+Liu%2C+Qiao+Tian%2C+Yi+Yuan%2C+Xubo+Liu%2C+Xinhao+Mei%2C+Qiuqiang+Kong%2C+Yuping+Wang%2C+Wenwu+Wang%2C+Yuxuan+Wang%2C+and+Mark+D%2E+Plumbley%2E+2023c%2E+AudioLDM+2%3A+Learning+Holistic+Audio+Generation+with+Self%2Dsupervis)
151. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023d. [Improved Baselines with Visual Instruction Tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/). In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following.
152. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024b. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge. [find](https://scholar.google.com/scholar?q=Haotian+Liu%2C+Chunyuan+Li%2C+Yuheng+Li%2C+Bo+Li%2C+Yuanhan+Zhang%2C+Sheng+Shen%2C+and+Yong+Jae+Lee%2E+2024b%2E+LLaVA%2DNeXT%3A+Improved+reasoning%2C+OCR%2C+and+world+knowledge%2E)
153. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023e. [Visual Instruction Tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). In Thirty-seventh Conference on Neural Information Processing Systems.
154. Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. 2023f. Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint arXiv:2311.05437. — [arXiv:2311.05437](https://arxiv.org/abs/2311.05437) [find](https://scholar.google.com/scholar?q=Shilong+Liu%2C+Hao+Cheng%2C+Haotian+Liu%2C+Hao+Zhang%2C+Feng+Li%2C+Tianhe+Ren%2C+Xueyan+Zou%2C+Jianwei+Yang%2C+Hang+Su%2C+Jun+Zhu%2C+et+al%2E+2023f%2E+Llava%2Dplus%3A+Learning+to+use+tools+for+creating+multimodal+agents%2E+arXiv+p)
155. Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021b. [Swin transformer: Hierarchical vision transformer using shifted windows](https://md.research.elamur.ai/free/en/papers/liu-2021-swin-transformer-hierarchical-vision-transformer/). In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022.
156. Zhaoyang Liu, Zeqiang Lai, Zhangwei Gao, Erfei Cui, Zhiheng Li, Xizhou Zhu, Lewei Lu, Qifeng Chen, Yu Qiao, Jifeng Dai, et al. 2023i. Controlllm: Augment language models with tools by searching on graphs. arXiv preprint arXiv:2310.17796. — [arXiv:2310.17796](https://arxiv.org/abs/2310.17796) [find](https://scholar.google.com/scholar?q=Zhaoyang+Liu%2C+Zeqiang+Lai%2C+Zhangwei+Gao%2C+Erfei+Cui%2C+Zhiheng+Li%2C+Xizhou+Zhu%2C+Lewei+Lu%2C+Qifeng+Chen%2C+Yu+Qiao%2C+Jifeng+Dai%2C+et+al%2E+2023i%2E+Controlllm%3A+Augment+language+models+with+tools+by+searching+on+gra)
157. Siqu Long, Feiqi Cao, Soyeon Caren Han, and Haiqin Yang. 2022. Vision-and-Language Pretrained Models: A Survey. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 5530–5537. [find](https://scholar.google.com/scholar?q=Siqu+Long%2C+Feiqi+Cao%2C+Soyeon+Caren+Han%2C+and+Haiqin+Yang%2E+2022%2E+Vision%2Dand%2DLanguage+Pretrained+Models%3A+A+Survey%2E+In+Proceedings+of+the+Thirty%2DFirst+International+Joint+Conference+on+Artificial+Intellig)
158. Junyu Lu, Ruyi Gan, Dixiang Zhang, Xiaojun Wu, Ziwei Wu, Renliang Sun, Jiaxing Zhang, Pingjian Zhang, and Yan Song. 2023a. Lyrics: Boosting Fine-grained Language-Vision Alignment and Comprehension via Semantic-aware Visual Objects. arXiv preprint arXiv:2312.05278. — [arXiv:2312.05278](https://arxiv.org/abs/2312.05278) [find](https://scholar.google.com/scholar?q=Junyu+Lu%2C+Ruyi+Gan%2C+Dixiang+Zhang%2C+Xiaojun+Wu%2C+Ziwei+Wu%2C+Renliang+Sun%2C+Jiaxing+Zhang%2C+Pingjian+Zhang%2C+and+Yan+Song%2E+2023a%2E+Lyrics%3A+Boosting+Fine%2Dgrained+Language%2DVision+Alignment+and+Comprehension+via)
159. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. [Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). In The Twelfth International Conference on Learning Representations.
160. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521. [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Swaroop+Mishra%2C+Tanglin+Xia%2C+Liang+Qiu%2C+Kai%2DWei+Chang%2C+Song%2DChun+Zhu%2C+Oyvind+Tafjord%2C+Peter+Clark%2C+and+Ashwin+Kalyan%2E+2022%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+scienc)
161. Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022. [P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks](https://md.research.elamur.ai/free/en/papers/liu-2022-p-tuning-prompt-tuning-can/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61–6…
162. Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. 2021. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Trac… [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Liang+Qiu%2C+Jiaqi+Chen%2C+Tony+Xia%2C+Yizhou+Zhao%2C+Wei+Zhang%2C+Zhou+Yu%2C+Xiaodan+Liang%2C+and+Song%2DChun+Zhu%2E+2021%2E+Iconqa%3A+A+new+benchmark+for+abstract+diagram+understanding+and+visual+language+reasoni)
163. Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021a. [P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks](https://md.research.elamur.ai/free/en/papers/liu-2021-p-tuning-v2-prompt-tuning/). arXiv preprint arXiv:2110.07602.
164. Yujie Lu, Xiujun Li, William Yang Wang, and Yejin Choi. 2023b. Vim: Probing multimodal large language models for visual embedded instruction following. arXiv preprint arXiv:2311.17647. — [arXiv:2311.17647](https://arxiv.org/abs/2311.17647) [find](https://scholar.google.com/scholar?q=Yujie+Lu%2C+Xiujun+Li%2C+William+Yang+Wang%2C+and+Yejin+Choi%2E+2023b%2E+Vim%3A+Probing+multimodal+large+language+models+for+visual+embedded+instruction+following%2E+arXiv+preprint+arXiv%3A2311%2E17647%2E+%E2%80%94+%5BarXiv%3A2311%2E1)
165. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2023g. [Mmbench: Is your multi-modal model an all-around player?](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/) arXiv preprint arXiv:2307.06281.
166. Yan Luo, Min Shi, Muhammad Osama Khan, Muhammad Muneeb Afzal, Hao Huang, Shuaihang Yuan, Yu Tian, Luo Song, Ava Kouhana, Tobias Elze, et al. 2024. FairCLIP: Harnessing Fairness in Vision-Language Learning. arXiv preprint arXiv:2403.19949. — [arXiv:2403.19949](https://arxiv.org/abs/2403.19949) [find](https://scholar.google.com/scholar?q=Yan+Luo%2C+Min+Shi%2C+Muhammad+Osama+Khan%2C+Muhammad+Muneeb+Afzal%2C+Hao+Huang%2C+Shuaihang+Yuan%2C+Yu+Tian%2C+Luo+Song%2C+Ava+Kouhana%2C+Tobias+Elze%2C+et+al%2E+2024%2E+FairCLIP%3A+Harnessing+Fairness+in+Vision%2DLanguage+Lear)
167. Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, et al. 2023h. [On the hidden mystery of ocr in large multimodal models](https://md.research.elamur.ai/free/en/papers/liu-2023-ocrbench-hidden-mystery-ocr-large/). arXiv preprint arXiv:2305.07895.
168. Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, et al. 2023. Kosmos-2.5: A multimodal literate model. arXiv preprint arXiv:2309.11419. — [arXiv:2309.11419](https://arxiv.org/abs/2309.11419) [find](https://scholar.google.com/scholar?q=Tengchao+Lv%2C+Yupan+Huang%2C+Jingye+Chen%2C+Lei+Cui%2C+Shuming+Ma%2C+Yaoyao+Chang%2C+Shaohan+Huang%2C+Wenhui+Wang%2C+Li+Dong%2C+Weiyao+Luo%2C+et+al%2E+2023%2E+Kosmos%2D2%2E5%3A+A+multimodal+literate+model%2E+arXiv+preprint+arXiv%3A23)
169. Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. 2023. Dolphins: Multimodal language model for driving. arXiv preprint arXiv:2312.00438. — [arXiv:2312.00438](https://arxiv.org/abs/2312.00438) [find](https://scholar.google.com/scholar?q=Yingzi+Ma%2C+Yulong+Cao%2C+Jiachen+Sun%2C+Marco+Pavone%2C+and+Chaowei+Xiao%2E+2023%2E+Dolphins%3A+Multimodal+language+model+for+driving%2E+arXiv+preprint+arXiv%3A2312%2E00438%2E+%E2%80%94+%5BarXiv%3A2312%2E00438%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
170. Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2023. [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://md.research.elamur.ai/free/en/papers/maaz-2023-video-chatgpt-towards-detailed-video/). arXiv preprint arXiv:2306.05424.
171. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. [Docvqa: A dataset for vqa on document images](https://md.research.elamur.ai/free/en/papers/mathew-2020-docvqa-dataset-vqa-document-images/). In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209.
172. Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier. [find](https://scholar.google.com/scholar?q=Michael+McCloskey+and+Neal+J+Cohen%2E+1989%2E+Catastrophic+interference+in+connectionist+networks%3A+The+sequential+learning+problem%2E+In+Psychology+of+learning+and+motivation%2C+volume+24%2C+pages+109%E2%80%93165%2E+Else)
173. Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. 2023. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. arXiv preprint arXiv:2303.17395. [find](https://scholar.google.com/scholar?q=Xinhao+Mei%2C+Chutong+Meng%2C+Haohe+Liu%2C+Qiuqiang+Kong%2C+Tom+Ko%2C+Chengqi+Zhao%2C+Mark+D+Plumbley%2C+Yuexian+Zou%2C+and+Wenwu+Wang%2E+2023%2E+Wavcaps%3A+A+chatgpt%2Dassisted+weakly%2Dlabelled+audio+captioning+dataset+for+a)
174. Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. 2019. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947–952. IEEE. [find](https://scholar.google.com/scholar?q=Anand+Mishra%2C+Shashank+Shekhar%2C+Ajeet+Kumar+Singh%2C+and+Anirban+Chakraborty%2E+2019%2E+Ocr%2Dvqa%3A+Visual+question+answering+by+reading+text+in+images%2E+In+2019+international+conference+on+document+analysis+an)
175. Yuhong Mo, Hao Qin, Yushan Dong, Ziyi Zhu, and Zhenglin Li. 2024. Large Language Model (LLM) AI Text Generation Detection based on Transformer Deep Learning Algorithm. International Journal of Engineering and Management Research, 14(2):154– 159. [find](https://scholar.google.com/scholar?q=Yuhong+Mo%2C+Hao+Qin%2C+Yushan+Dong%2C+Ziyi+Zhu%2C+and+Zhenglin+Li%2E+2024%2E+Large+Language+Model+%28LLM%29+AI+Text+Generation+Detection+based+on+Transformer+Deep+Learning+Algorithm%2E+International+Journal+of+Enginee)
176. Debjyoti Mondal, Suraj Modi, Subhadarshi Panda, Rituraj Singh, and Godawari Sudhakar Rao. 2024. KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning. arXiv preprint arXiv:2401.12863. — [arXiv:2401.12863](https://arxiv.org/abs/2401.12863) [find](https://scholar.google.com/scholar?q=Debjyoti+Mondal%2C+Suraj+Modi%2C+Subhadarshi+Panda%2C+Rituraj+Singh%2C+and+Godawari+Sudhakar+Rao%2E+2024%2E+KAM%2DCoT%3A+Knowledge+Augmented+Multimodal+Chain%2Dof%2DThoughts+Reasoning%2E+arXiv+preprint+arXiv%3A2401%2E12863%2E+%E2%80%94+)
177. Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Matt Smith, Shashank Jain, Chun-Fu Yeh, Prakash Murugesan, Peyman Heidari, Yue Liu, et al. 2023. Anymal: An efficient and scalable any-modality augmented language model. arXiv preprint arXiv:2309.16058. [find](https://scholar.google.com/scholar?q=Seungwhan+Moon%2C+Andrea+Madotto%2C+Zhaojiang+Lin%2C+Tushar+Nagarajan%2C+Matt+Smith%2C+Shashank+Jain%2C+Chun%2DFu+Yeh%2C+Prakash+Murugesan%2C+Peyman+Heidari%2C+Yue+Liu%2C+et+al%2E+2023%2E+Anymal%3A+An+efficient+and+scalable+any%2D)
178. Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. 2023. Embodiedgpt: Vision-language pre-training via embodied chain of thought. In Thirty-seventh Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Yao+Mu%2C+Qinglong+Zhang%2C+Mengkang+Hu%2C+Wenhai+Wang%2C+Mingyu+Ding%2C+Jun+Jin%2C+Bin+Wang%2C+Jifeng+Dai%2C+Yu+Qiao%2C+and+Ping+Luo%2E+2023%2E+Embodiedgpt%3A+Vision%2Dlanguage+pre%2Dtraining+via+embodied+chain+of+thought%2E+In+T)
179. Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Nick Barnes, and Ajmal Mian. 2023. [A comprehensive overview of large language models](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/). arXiv preprint arXiv:2307.06435.
180. Ziyi Ni, Minglun Han, Feilong Chen, Linghui Meng, Jing Shi, Pin Lv, and Bo Xu. 2024. VILAS: Exploring the Effects of Vision and Language Context in Automatic Speech Recognition. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. [find](https://scholar.google.com/scholar?q=Ziyi+Ni%2C+Minglun+Han%2C+Feilong+Chen%2C+Linghui+Meng%2C+Jing+Shi%2C+Pin+Lv%2C+and+Bo+Xu%2E+2024%2E+VILAS%3A+Exploring+the+Effects+of+Vision+and+Language+Context+in+Automatic+Speech+Recognition%2E+In+ICASSP+2024%2D2024+IE)
181. OpenAI. 2022. OpenAI: Introducing ChatGPT. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2022%2E+OpenAI%3A+Introducing+ChatGPT%2E)
182. OpenAI. 2023. [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
183. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. 2023. [Dinov2: Learning robust visual features without supervision](https://md.research.elamur.ai/free/en/papers/oquab-2023-dinov2-learning-robust-visual-features/). arXiv preprint arXiv:2304.07193.
184. Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 2011. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems, 24. [find](https://scholar.google.com/scholar?q=Vicente+Ordonez%2C+Girish+Kulkarni%2C+and+Tamara+Berg%2E+2011%2E+Im2text%3A+Describing+images+using+1+million+captioned+photographs%2E+Advances+in+neural+information+processing+systems%2C+24%2E)
185. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744.
186. Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. 2023. Kosmos-g: Generating images in context with multimodal large language models. arXiv preprint arXiv:2310.02992. — [arXiv:2310.02992](https://arxiv.org/abs/2310.02992) [find](https://scholar.google.com/scholar?q=Xichen+Pan%2C+Li+Dong%2C+Shaohan+Huang%2C+Zhiliang+Peng%2C+Wenhu+Chen%2C+and+Furu+Wei%2E+2023%2E+Kosmos%2Dg%3A+Generating+images+in+context+with+multimodal+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E02992%2E+%E2%80%94+%5BarX)
187. Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. 2023. X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning. arXiv preprint arXiv:2311.18… — [arXiv:2311.18799](https://arxiv.org/abs/2311.18799) [find](https://scholar.google.com/scholar?q=Artemis+Panagopoulou%2C+Le+Xue%2C+Ning+Yu%2C+Junnan+Li%2C+Dongxu+Li%2C+Shafiq+Joty%2C+Ran+Xu%2C+Silvio+Savarese%2C+Caiming+Xiong%2C+and+Juan+Carlos+Niebles%2E+2023%2E+X%2DInstructBLIP%3A+A+Framework+for+aligning+X%2DModal+instru)
188. Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. 2023. [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://md.research.elamur.ai/free/en/papers/peng-2023-kosmos-2-grounding-multimodal-large/). arXiv preprint arXiv:2306.14824.
189. Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, and Lingpeng Kong Tong Zhang. 2023. DetGPT: Detect What You Need via Reasoning. arXiv preprint arXiv:2305.14167. [find](https://scholar.google.com/scholar?q=Renjie+Pi%2C+Jiahui+Gao%2C+Shizhe+Diao%2C+Rui+Pan%2C+Hanze+Dong%2C+Jipeng+Zhang%2C+Lewei+Yao%2C+Jianhua+Han%2C+Hang+Xu%2C+and+Lingpeng+Kong+Tong+Zhang%2E+2023%2E+DetGPT%3A+Detect+What+You+Need+via+Reasoning%2E+arXiv+preprint+a)
190. Ji Qi, Ming Ding, Weihan Wang, Yushi Bai, Qingsong Lv, Wenyi Hong, Bin Xu, Lei Hou, Juanzi Li, Yuxiao Dong, et al. 2024. CogCoM: Train Large Vision-Language Models Diving into Details through Chain of Manipulations. arXiv preprint arXiv:2402.04236. — [arXiv:2402.04236](https://arxiv.org/abs/2402.04236) [find](https://scholar.google.com/scholar?q=Ji+Qi%2C+Ming+Ding%2C+Weihan+Wang%2C+Yushi+Bai%2C+Qingsong+Lv%2C+Wenyi+Hong%2C+Bin+Xu%2C+Lei+Hou%2C+Juanzi+Li%2C+Yuxiao+Dong%2C+et+al%2E+2024%2E+CogCoM%3A+Train+Large+Vision%2DLanguage+Models+Diving+into+Details+through+Chain+of)
191. Jie Qin, Jie Wu, Weifeng Chen, Yuxi Ren, Huixia Li, Hefeng Wu, Xuefeng Xiao, Rui Wang, and Shilei Wen. 2024. DiffusionGPT: LLM-Driven Text-to-Image Generation System. arXiv preprint arXiv:2401.10061. — [arXiv:2401.10061](https://arxiv.org/abs/2401.10061) [find](https://scholar.google.com/scholar?q=Jie+Qin%2C+Jie+Wu%2C+Weifeng+Chen%2C+Yuxi+Ren%2C+Huixia+Li%2C+Hefeng+Wu%2C+Xuefeng+Xiao%2C+Rui+Wang%2C+and+Shilei+Wen%2E+2024%2E+DiffusionGPT%3A+LLM%2DDriven+Text%2Dto%2DImage+Generation+System%2E+arXiv+preprint+arXiv%3A2401%2E10061%2E+)
192. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In International conference on machine learning, pages 8748–8763. PM…
193. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. [Robust Speech Recognition via Large-Scale Weak Supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/). In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, pages 28492–28518.
194. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). The Journal of Machine Learning Research, 21(1):5485–5551.
195. Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Erix Xing, Ming-Hsuan Yang, and Fahad S Khan. 2023. Glamm: Pixel grounding large multimodal model. arXiv preprint arXiv:2311.03356. — [arXiv:2311.03356](https://arxiv.org/abs/2311.03356) [find](https://scholar.google.com/scholar?q=Hanoona+Rasheed%2C+Muhammad+Maaz%2C+Sahal+Shaji%2C+Abdelrahman+Shaker%2C+Salman+Khan%2C+Hisham+Cholakkal%2C+Rao+M+Anwer%2C+Erix+Xing%2C+Ming%2DHsuan+Yang%2C+and+Fahad+S+Khan%2E+2023%2E+Glamm%3A+Pixel+grounding+large+multimodal)
196. Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. [Learning multiple visual domains with residual adapters](https://md.research.elamur.ai/free/en/papers/rebuffi-2017-learning-multiple-visual-domains-residual/). Advances in neural information processing systems, 30.
197. Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. 2023. PixelLM: Pixel Reasoning with Large Multimodal Model. arXiv preprint arXiv:2312.02228. — [arXiv:2312.02228](https://arxiv.org/abs/2312.02228) [find](https://scholar.google.com/scholar?q=Zhongwei+Ren%2C+Zhicheng+Huang%2C+Yunchao+Wei%2C+Yao+Zhao%2C+Dongmei+Fu%2C+Jiashi+Feng%2C+and+Xiaojie+Jin%2E+2023%2E+PixelLM%3A+Pixel+Reasoning+with+Large+Multimodal+Model%2E+arXiv+preprint+arXiv%3A2312%2E02228%2E+%E2%80%94+%5BarXiv%3A231)
198. Anthony Robins. 1995. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2):123–146. [find](https://scholar.google.com/scholar?q=Anthony+Robins%2E+1995%2E+Catastrophic+forgetting%2C+rehearsal+and+pseudorehearsal%2E+Connection+Science%2C+7%282%29%3A123%E2%80%93146%2E)
199. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695. [find](https://scholar.google.com/scholar?q=Robin+Rombach%2C+Andreas+Blattmann%2C+Dominik+Lorenz%2C+Patrick+Esser%2C+and+Bj%C3%B6rn+Ommer%2E+2022%2E+High%2Dresolution+image+synthesis+with+latent+diffusion+models%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+compu)
200. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. [U-net: Convolutional networks for biomedical image segmentation](https://md.research.elamur.ai/free/en/papers/ronneberger-2015-u-net-convolutional-networks-biomedical/). In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234…
201. Ludan Ruan and Qin Jin. 2022. Survey: Transformer based video-language pre-training. AI Open, 3:1–13. [find](https://scholar.google.com/scholar?q=Ludan+Ruan+and+Qin+Jin%2E+2022%2E+Survey%3A+Transformer+based+video%2Dlanguage+pre%2Dtraining%2E+AI+Open%2C+3%3A1%E2%80%9313%2E)
202. Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. 2023. [AudioPaLM: A Large Language Model That Can Speak and Listen](https://md.research.elamur.ai/free/en/papers/rubenstein-2023-audiopalm-large-language-model-that/). arXiv preprint arXiv:2306.12925.
203. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022. [Laion-5b: An open large-scale dataset for training next generation image-text models](https://md.research.elamur.ai/free/en/papers/schuhmann-2022-laion-5b-open-large-scale/). Advances in Neural Information Processing S…
204. Christoph Schuhmann, Andreas Köpf, Richard Vencu, Theo Coombes, and Romain Beaumont. 2022b. Laion coco: 600m synthetic captions from laion2ben. [find](https://scholar.google.com/scholar?q=Christoph+Schuhmann%2C+Andreas+K%C3%B6pf%2C+Richard+Vencu%2C+Theo+Coombes%2C+and+Romain+Beaumont%2E+2022b%2E+Laion+coco%3A+600m+synthetic+captions+from+laion2ben%2E)
205. Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021. [Laion-400m: Open dataset of clip-filtered 400 million image-text pairs](https://md.research.elamur.ai/free/en/papers/schuhmann-2021-laion-400m-open-dataset-clip/). arXiv preprint arXiv:2111.02114.
206. Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. [A-okvqa: A benchmark for visual question answering using world knowledge](https://md.research.elamur.ai/free/en/papers/schwenk-2022-okvqa-benchmark-visual-question-answering/). In European Conference on Computer Vision, pages 146–162. Springer.
207. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. [Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565.
208. Weizhou Shen, Chenliang Li, Hongzhan Chen, Ming Yan, Xiaojun Quan, Hehong Chen, Ji Zhang, and Fei Huang. 2024. Small llms are weak tool learners: A multi-llm agent. arXiv preprint arXiv:2401.07324. [find](https://scholar.google.com/scholar?q=Weizhou+Shen%2C+Chenliang+Li%2C+Hongzhan+Chen%2C+Ming+Yan%2C+Xiaojun+Quan%2C+Hehong+Chen%2C+Ji+Zhang%2C+and+Fei+Huang%2E+2024%2E+Small+llms+are+weak+tool+learners%3A+A+multi%2Dllm+agent%2E+arXiv+preprint+arXiv%3A2401%2E07324%2E)
209. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. [Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface](https://md.research.elamur.ai/free/en/papers/shen-2023-hugginggpt-solving-ai-tasks-chatgpt/). arXiv preprint arXiv:2303.17580.
210. Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. 2020. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 742–758. Springer. [find](https://scholar.google.com/scholar?q=Oleksii+Sidorov%2C+Ronghang+Hu%2C+Marcus+Rohrbach%2C+and+Amanpreet+Singh%2E+2020%2E+Textcaps%3A+a+dataset+for+image+captioning+with+reading+comprehension%2E+In+Computer+Vision%E2%80%93ECCV+2020%3A+16th+European+Conference%2C+G)
211. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326. [find](https://scholar.google.com/scholar?q=Amanpreet+Singh%2C+Vivek+Natarajan%2C+Meet+Shah%2C+Yu+Jiang%2C+Xinlei+Chen%2C+Dhruv+Batra%2C+Devi+Parikh%2C+and+Marcus+Rohrbach%2E+2019%2E+Towards+vqa+models+that+can+read%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+)
212. Shezheng Song, Xiaopeng Li, and Shasha Li. 2023. How to Bridge the Gap between Modalities: A Comprehensive Survey on Multimodal Large Language Model. arXiv preprint arXiv:2311.07594. — [arXiv:2311.07594](https://arxiv.org/abs/2311.07594) [find](https://scholar.google.com/scholar?q=Shezheng+Song%2C+Xiaopeng+Li%2C+and+Shasha+Li%2E+2023%2E+How+to+Bridge+the+Gap+between+Modalities%3A+A+Comprehensive+Survey+on+Multimodal+Large+Language+Model%2E+arXiv+preprint+arXiv%3A2311%2E07594%2E+%E2%80%94+%5BarXiv%3A2311%2E075)
213. Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2021. [Score-Based Generative Modeling through Stochastic Differential Equations](https://md.research.elamur.ai/free/en/papers/song-2020-score-based-generative-modeling-through/). In International Conference on Learning Representations.
214. Zezheng Song, Jiaxin Yuan, and Haizhao Yang. 2024. FMint: Bridging Human Designed and Data Pre-trained Models for Differential Equation Foundation Model. arXiv preprint arXiv:2404.14688. — [arXiv:2404.14688](https://arxiv.org/abs/2404.14688) [find](https://scholar.google.com/scholar?q=Zezheng+Song%2C+Jiaxin+Yuan%2C+and+Haizhao+Yang%2E+2024%2E+FMint%3A+Bridging+Human+Designed+and+Data+Pre%2Dtrained+Models+for+Differential+Equation+Foundation+Model%2E+arXiv+preprint+arXiv%3A2404%2E14688%2E+%E2%80%94+%5BarXiv%3A2404)
215. Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. 2023. [Pandagpt: One model to instruction-follow them all](https://md.research.elamur.ai/free/en/papers/su-2023-pandagpt-one-model-instruction-follow/). arXiv preprint arXiv:2305.16355.
216. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805.
217. InternLM Team. 2023. Internlm: A multilingual language model with progressively enhanced capabilities. [find](https://scholar.google.com/scholar?q=InternLM+Team%2E+2023%2E+Internlm%3A+A+multilingual+language+model+with+progressively+enhanced+capabilities%2E)
218. Changyao Tian, Xizhou Zhu, Yuwen Xiong, Weiyun Wang, Zhe Chen, Wenhai Wang, Yuntao Chen, Lewei Lu, Tong Lu, Jie Zhou, et al. 2024. MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer. arXiv preprint arXiv:2401.10208. — [arXiv:2401.10208](https://arxiv.org/abs/2401.10208) [find](https://scholar.google.com/scholar?q=Changyao+Tian%2C+Xizhou+Zhu%2C+Yuwen+Xiong%2C+Weiyun+Wang%2C+Zhe+Chen%2C+Wenhai+Wang%2C+Yuntao+Chen%2C+Lewei+Lu%2C+Tong+Lu%2C+Jie+Zhou%2C+et+al%2E+2024%2E+MM%2DInterleaved%3A+Interleaved+Image%2DText+Generative+Modeling+via+Multi%2D)
219. Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, et al. 2023a. Generative multimodal models are in-context learners. arXiv preprint arXiv:2312.13286. — [arXiv:2312.13286](https://arxiv.org/abs/2312.13286) [find](https://scholar.google.com/scholar?q=Quan+Sun%2C+Yufeng+Cui%2C+Xiaosong+Zhang%2C+Fan+Zhang%2C+Qiying+Yu%2C+Zhengxiong+Luo%2C+Yueze+Wang%2C+Yongming+Rao%2C+Jingjing+Liu%2C+Tiejun+Huang%2C+et+al%2E+2023a%2E+Generative+multimodal+models+are+in%2Dcontext+learners%2E+ar)
220. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
221. Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. 2024. Generative pretraining in multimodality. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Quan+Sun%2C+Qiying+Yu%2C+Yufeng+Cui%2C+Fan+Zhang%2C+Xiaosong+Zhang%2C+Yueze+Wang%2C+Hongcheng+Gao%2C+Jingjing+Liu%2C+Tiejun+Huang%2C+and+Xinlong+Wang%2E+2024%2E+Generative+pretraining+in+multimodality%2E+In+The+Twelfth+Inter)
222. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023b%2E+Llama+2%3A+Open+foundation+and+f)
223. Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. 2023b. [Aligning large multimodal models with factually augmented rlhf](https://md.research.elamur.ai/free/en/papers/sun-2023-aligning-large-multimodal-models-factually/). arXiv preprint arXiv:2309.14525.
224. Dídac Surís, Sachit Menon, and Carl Vondrick. 2023. [Vipergpt: Visual inference via python execution for reasoning](https://md.research.elamur.ai/free/en/papers/suris-2023-vipergpt-visual-inference-python-execution/). arXiv preprint arXiv:2303.08128.
225. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2023a. [Salmonn: Towards generic hearing abilities for large language models](https://md.research.elamur.ai/free/en/papers/tang-2023-salmonn-towards-generic-hearing-abilities/). arXiv preprint arXiv:2310.13289.
226. Zineng Tang, Ziyi Yang, Mahmoud Khademi, Yang Liu, Chenguang Zhu, and Mohit Bansal. 2023b. CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation. arXiv preprint arXiv:2311.18775. — [arXiv:2311.18775](https://arxiv.org/abs/2311.18775) [find](https://scholar.google.com/scholar?q=Zineng+Tang%2C+Ziyi+Yang%2C+Mahmoud+Khademi%2C+Yang+Liu%2C+Chenguang+Zhu%2C+and+Mohit+Bansal%2E+2023b%2E+CoDi%2D2%3A+In%2DContext%2C+Interleaved%2C+and+Interactive+Any%2Dto%2DAny+Generation%2E+arXiv+preprint+arXiv%3A2311%2E18775%2E+%E2%80%94+%5Ba)
227. Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. 2023c. Any-to-Any Generation via Composable Diffusion. In Thirty-seventh Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Zineng+Tang%2C+Ziyi+Yang%2C+Chenguang+Zhu%2C+Michael+Zeng%2C+and+Mohit+Bansal%2E+2023c%2E+Any%2Dto%2DAny+Generation+via+Composable+Diffusion%2E+In+Thirty%2Dseventh+Conference+on+Neural+Information+Processing+Systems%2E)
228. Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, et al. 2022. [Ul2: Unifying language learning paradigms](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/). In The Eleventh International Conference on Learning Representations.
229. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30.
230. Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. 2024a. [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://md.research.elamur.ai/free/en/papers/wang-2023-decodingtrust-comprehensive-assessment-trustworthiness-gpt/). Advances in Neural Information Processing Systems, 36.
231. Chenyu Wang, Weixin Luo, Qianyu Chen, Haonan Mai, Jindi Guo, Sixun Dong, Zhengxin Li, Lin Ma, Shenghua Gao, et al. 2024b. Tool-LMM: A Large Multi-Modal Model for Tool Agent Learning. arXiv preprint arXiv:2401.10727. — [arXiv:2401.10727](https://arxiv.org/abs/2401.10727) [find](https://scholar.google.com/scholar?q=Chenyu+Wang%2C+Weixin+Luo%2C+Qianyu+Chen%2C+Haonan+Mai%2C+Jindi+Guo%2C+Sixun+Dong%2C+Zhengxin+Li%2C+Lin+Ma%2C+Shenghua+Gao%2C+et+al%2E+2024b%2E+Tool%2DLMM%3A+A+Large+Multi%2DModal+Model+for+Tool+Agent+Learning%2E+arXiv+preprint+ar)
232. Dongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Armineh Nourbakhsh, and Xiaomo Liu. 2023a. DocLLM: A layout-aware generative language model for multimodal document understanding. arXiv preprint arXiv:2401.00908. [find](https://scholar.google.com/scholar?q=Dongsheng+Wang%2C+Natraj+Raman%2C+Mathieu+Sibue%2C+Zhiqiang+Ma%2C+Petr+Babkin%2C+Simerjot+Kaur%2C+Yulong+Pei%2C+Armineh+Nourbakhsh%2C+and+Xiaomo+Liu%2E+2023a%2E+DocLLM%3A+A+layout%2Daware+generative+language+model+for+multim)
233. Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024c. Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception. arXiv preprint arXiv:2401.16158. — [arXiv:2401.16158](https://arxiv.org/abs/2401.16158) [find](https://scholar.google.com/scholar?q=Junyang+Wang%2C+Haiyang+Xu%2C+Jiabo+Ye%2C+Ming+Yan%2C+Weizhou+Shen%2C+Ji+Zhang%2C+Fei+Huang%2C+and+Jitao+Sang%2E+2024c%2E+Mobile%2DAgent%3A+Autonomous+Multi%2DModal+Mobile+Device+Agent+with+Visual+Perception%2E+arXiv+preprint+)
234. Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022a. [Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework](https://md.research.elamur.ai/free/en/papers/wang-2022-ofa-unifying-architectures-tasks-modalities/). In International Conference on Machine Learning, pages 23318–23…
235. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. 2023b. [Cogvlm: Visual expert for pretrained language models](https://md.research.elamur.ai/free/en/papers/wang-2023-cogvlm-visual-expert-pretrained/). arXiv preprint arXiv:2311.03079.
236. Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhenhang Huang, Linjie Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, et al. 2023c. The all-seeing project: Towards panoptic visual recognition and understanding of the open world. arXiv preprint arXiv:2308.01907. — [arXiv:2308.01907](https://arxiv.org/abs/2308.01907) [find](https://scholar.google.com/scholar?q=Weiyun+Wang%2C+Min+Shi%2C+Qingyun+Li%2C+Wenhai+Wang%2C+Zhenhang+Huang%2C+Linjie+Xing%2C+Zhe+Chen%2C+Hao+Li%2C+Xizhou+Zhu%2C+Zhiguo+Cao%2C+et+al%2E+2023c%2E+The+all%2Dseeing+project%3A+Towards+panoptic+visual+recognition+and+unde)
237. Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. 2022b. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442. [find](https://scholar.google.com/scholar?q=Wenhui+Wang%2C+Hangbo+Bao%2C+Li+Dong%2C+Johan+Bjorck%2C+Zhiliang+Peng%2C+Qiang+Liu%2C+Kriti+Aggarwal%2C+Owais+Khan+Mohammed%2C+Saksham+Singhal%2C+Subhojit+Som%2C+et+al%2E+2022b%2E+Image+as+a+foreign+language%3A+Beit+pretrainin)
238. Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. 2023d. Image as a Foreign Language: BEiT Pretraining for Vision and Vision-Language Tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision… [find](https://scholar.google.com/scholar?q=Wenhui+Wang%2C+Hangbo+Bao%2C+Li+Dong%2C+Johan+Bjorck%2C+Zhiliang+Peng%2C+Qiang+Liu%2C+Kriti+Aggarwal%2C+Owais+Khan+Mohammed%2C+Saksham+Singhal%2C+Subhojit+Som%2C+et+al%2E+2023d%2E+Image+as+a+Foreign+Language%3A+BEiT+Pretrainin)
239. Xinyu Wang, Bohan Zhuang, and Qi Wu. 2024d. ModaVerse: Efficiently Transforming Modalities with LLMs. arXiv preprint arXiv:2401.06395. — [arXiv:2401.06395](https://arxiv.org/abs/2401.06395) [find](https://scholar.google.com/scholar?q=Xinyu+Wang%2C+Bohan+Zhuang%2C+and+Qi+Wu%2E+2024d%2E+ModaVerse%3A+Efficiently+Transforming+Modalities+with+LLMs%2E+arXiv+preprint+arXiv%3A2401%2E06395%2E+%E2%80%94+%5BarXiv%3A2401%2E06395%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2401%2E06395%29)
240. Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. 2023e. Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprint arXiv:2308.08769. — [arXiv:2308.08769](https://arxiv.org/abs/2308.08769) [find](https://scholar.google.com/scholar?q=Zehan+Wang%2C+Haifeng+Huang%2C+Yang+Zhao%2C+Ziang+Zhang%2C+and+Zhou+Zhao%2E+2023e%2E+Chat%2D3d%3A+Data%2Defficiently+tuning+large+language+model+for+universal+dialogue+of+3d+scenes%2E+arXiv+preprint+arXiv%3A2308%2E08769%2E+%E2%80%94+%5B)
241. Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, En Yu, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. 2024. Small Language Model Meets with Reinforced Vision Vocabulary. arXiv preprint arXiv:2401.12503. [find](https://scholar.google.com/scholar?q=Haoran+Wei%2C+Lingyu+Kong%2C+Jinyue+Chen%2C+Liang+Zhao%2C+Zheng+Ge%2C+En+Yu%2C+Jianjian+Sun%2C+Chunrui+Han%2C+and+Xiangyu+Zhang%2E+2024%2E+Small+Language+Model+Meets+with+Reinforced+Vision+Vocabulary%2E+arXiv+preprint+arXi)
242. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned Language Models are Zero-Shot Learners. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Bosma%2C+Vincent+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+M+Dai%2C+and+Quoc+V+Le%2E+2021%2E+Finetuned+Language+Models+are+Zero%2DShot+Learners%2E+In+International+Conference)
243. Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. 2023a. [Visual chatgpt: Talking, drawing and editing with visual foundation models](https://md.research.elamur.ai/free/en/papers/wu-2023-visual-chatgpt-talking-drawing-editing/). arXiv preprint arXiv:2303.04671.
244. Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, et al. 2023b. Qbench: A benchmark for general-purpose foundation models on low-level vision. arXiv preprint arXiv:2309.14181. — [arXiv:2309.14181](https://arxiv.org/abs/2309.14181) [find](https://scholar.google.com/scholar?q=Haoning+Wu%2C+Zicheng+Zhang%2C+Erli+Zhang%2C+Chaofeng+Chen%2C+Liang+Liao%2C+Annan+Wang%2C+Chunyi+Li%2C+Wenxiu+Sun%2C+Qiong+Yan%2C+Guangtao+Zhai%2C+et+al%2E+2023b%2E+Qbench%3A+A+benchmark+for+general%2Dpurpose+foundation+models+o)
245. Jiahong Wu, He Zheng, Bo Zhao, Yixin Li, Baoming Yan, Rui Liang, Wenjia Wang, Shipei Zhou, Guosen Lin, Yanwei Fu, et al. 2017. Ai challenger: A large-scale dataset for going deeper in image understanding. arXiv preprint arXiv:1711.06475. — [arXiv:1711.06475](https://arxiv.org/abs/1711.06475) [find](https://scholar.google.com/scholar?q=Jiahong+Wu%2C+He+Zheng%2C+Bo+Zhao%2C+Yixin+Li%2C+Baoming+Yan%2C+Rui+Liang%2C+Wenjia+Wang%2C+Shipei+Zhou%2C+Guosen+Lin%2C+Yanwei+Fu%2C+et+al%2E+2017%2E+Ai+challenger%3A+A+large%2Dscale+dataset+for+going+deeper+in+image+understand)
246. Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Philip S Yu. 2023c. Multimodal [large language models: A survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/). arXiv preprint arXiv:2311.13165. — [arXiv:2311.13165](https://arxiv.org/abs/2311.13165)
247. Penghao Wu and Saining Xie. 2023. V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs. arXiv preprint arXiv:2312.14135, 17. [find](https://scholar.google.com/scholar?q=Penghao+Wu+and+Saining+Xie%2E+2023%2E+V%2A%3A+Guided+Visual+Search+as+a+Core+Mechanism+in+Multimodal+LLMs%2E+arXiv+preprint+arXiv%3A2312%2E14135%2C+17%2E)
248. Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. 2023d. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519. [find](https://scholar.google.com/scholar?q=Shengqiong+Wu%2C+Hao+Fei%2C+Leigang+Qu%2C+Wei+Ji%2C+and+Tat%2DSeng+Chua%2E+2023d%2E+Next%2Dgpt%3A+Any%2Dto%2Dany+multimodal+llm%2E+arXiv+preprint+arXiv%3A2309%2E05519%2E)
249. Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. 2024. Continual Learning for [Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/). arXiv preprint arXiv:2402.01364.
250. Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2023e. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (I… [find](https://scholar.google.com/scholar?q=Yusong+Wu%2C+Ke+Chen%2C+Tianyu+Zhang%2C+Yuchen+Hui%2C+Taylor+Berg%2DKirkpatrick%2C+and+Shlomo+Dubnov%2E+2023e%2E+Large%2Dscale+contrastive+language%2Daudio+pretraining+with+feature+fusion+and+keyword%2Dto%2Dcaption+augmentat)
251. Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296. [find](https://scholar.google.com/scholar?q=Jun+Xu%2C+Tao+Mei%2C+Ting+Yao%2C+and+Yong+Rui%2E+2016%2E+Msr%2Dvtt%3A+A+large+video+description+dataset+for+bridging+video+and+language%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognit)
252. Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. 2023. Pointllm: Empowering large language models to understand point clouds. arXiv preprint arXiv:2308.16911. — [arXiv:2308.16911](https://arxiv.org/abs/2308.16911) [find](https://scholar.google.com/scholar?q=Runsen+Xu%2C+Xiaolong+Wang%2C+Tai+Wang%2C+Yilun+Chen%2C+Jiangmiao+Pang%2C+and+Dahua+Lin%2E+2023%2E+Pointllm%3A+Empowering+large+language+models+to+understand+point+clouds%2E+arXiv+preprint+arXiv%3A2308%2E16911%2E+%E2%80%94+%5BarXiv%3A23)
253. An Yan, Zhengyuan Yang, Junda Wu, Wanrong Zhu, Jianwei Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Julian McAuley, Jianfeng Gao, et al. 2024a. List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs. arXiv preprint arXiv:2404.16375. — [arXiv:2404.16375](https://arxiv.org/abs/2404.16375) [find](https://scholar.google.com/scholar?q=An+Yan%2C+Zhengyuan+Yang%2C+Junda+Wu%2C+Wanrong+Zhu%2C+Jianwei+Yang%2C+Linjie+Li%2C+Kevin+Lin%2C+Jianfeng+Wang%2C+Julian+McAuley%2C+Jianfeng+Gao%2C+et+al%2E+2024a%2E+List+Items+One+by+One%3A+A+New+Data+Source+and+Learning+Para)
254. Rui Yan, Mike Zheng Shou, Yixiao Ge, Alex Jinpeng Wang, Xudong Lin, Guanyu Cai, and Jinhui Tang. 2021. Video-text pre-training with learned regions. arXiv preprint arXiv:2112.01194. — [arXiv:2112.01194](https://arxiv.org/abs/2112.01194) [find](https://scholar.google.com/scholar?q=Rui+Yan%2C+Mike+Zheng+Shou%2C+Yixiao+Ge%2C+Alex+Jinpeng+Wang%2C+Xudong+Lin%2C+Guanyu+Cai%2C+and+Jinhui+Tang%2E+2021%2E+Video%2Dtext+pre%2Dtraining+with+learned+regions%2E+arXiv+preprint+arXiv%3A2112%2E01194%2E+%E2%80%94+%5BarXiv%3A2112%2E0119)
255. Siming Yan, Min Bai, Weifeng Chen, Xiong Zhou, Qixing Huang, and Li Erran Li. 2024b. ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grained Reward Modeling. arXiv preprint arXiv:2402.06118. — [arXiv:2402.06118](https://arxiv.org/abs/2402.06118) [find](https://scholar.google.com/scholar?q=Siming+Yan%2C+Min+Bai%2C+Weifeng+Chen%2C+Xiong+Zhou%2C+Qixing+Huang%2C+and+Li+Erran+Li%2E+2024b%2E+ViGoR%3A+Improving+Visual+Grounding+of+Large+Vision+Language+Models+with+Fine%2DGrained+Reward+Modeling%2E+arXiv+preprint)
256. Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. 2022. Vision-language pre-training with triple contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15671–15680. [find](https://scholar.google.com/scholar?q=Jinyu+Yang%2C+Jiali+Duan%2C+Son+Tran%2C+Yi+Xu%2C+Sampath+Chanda%2C+Liqun+Chen%2C+Belinda+Zeng%2C+Trishul+Chilimbi%2C+and+Junzhou+Huang%2E+2022%2E+Vision%2Dlanguage+pre%2Dtraining+with+triple+contrastive+learning%2E+In+Proceedi)
257. Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui. 2024. Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs. arXiv preprint arXiv:2401.11708. — [arXiv:2401.11708](https://arxiv.org/abs/2401.11708) [find](https://scholar.google.com/scholar?q=Ling+Yang%2C+Zhaochen+Yu%2C+Chenlin+Meng%2C+Minkai+Xu%2C+Stefano+Ermon%2C+and+Bin+Cui%2E+2024%2E+Mastering+Text%2Dto%2DImage+Diffusion%3A+Recaptioning%2C+Planning%2C+and+Generating+with+Multimodal+LLMs%2E+arXiv+preprint+arXiv%3A)
258. Zhen Yang, Yingxue Zhang, Fandong Meng, and Jie Zhou. 2023a. TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models. arXiv preprint arXiv:2311.04589. — [arXiv:2311.04589](https://arxiv.org/abs/2311.04589) [find](https://scholar.google.com/scholar?q=Zhen+Yang%2C+Yingxue+Zhang%2C+Fandong+Meng%2C+and+Jie+Zhou%2E+2023a%2E+TEAL%3A+Tokenize+and+Embed+ALL+for+Multi%2Dmodal+Large+Language+Models%2E+arXiv+preprint+arXiv%3A2311%2E04589%2E+%E2%80%94+%5BarXiv%3A2311%2E04589%5D%28https%3A%2F%2Farxiv%2Eorg)
259. Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023b. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381. [find](https://scholar.google.com/scholar?q=Zhengyuan+Yang%2C+Linjie+Li%2C+Jianfeng+Wang%2C+Kevin+Lin%2C+Ehsan+Azarnasab%2C+Faisal+Ahmed%2C+Zicheng+Liu%2C+Ce+Liu%2C+Michael+Zeng%2C+and+Lijuan+Wang%2E+2023b%2E+Mm%2Dreact%3A+Prompting+chatgpt+for+multimodal+reasoning+and+)
260. Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, et al. 2023a. mplug-docowl: Modularized multimodal large language model for document understanding. arXiv preprint arXiv:2307.02499. — [arXiv:2307.02499](https://arxiv.org/abs/2307.02499) [find](https://scholar.google.com/scholar?q=Jiabo+Ye%2C+Anwen+Hu%2C+Haiyang+Xu%2C+Qinghao+Ye%2C+Ming+Yan%2C+Yuhao+Dan%2C+Chenlin+Zhao%2C+Guohai+Xu%2C+Chenliang+Li%2C+Junfeng+Tian%2C+et+al%2E+2023a%2E+mplug%2Ddocowl%3A+Modularized+multimodal+large+language+model+for+docume)
261. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. 2023b. [mplug-owl: Modularization empowers large language models with multimodality](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/). arXiv preprint arXiv:2304.14178.
262. Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2023c. [mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl2-revolutionizing-multi-modal/). arXiv preprint arXiv:2311.04257.
263. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023a. [A Survey on Multimodal Large Language Models](https://md.research.elamur.ai/free/en/papers/yin-2023-survey-multimodal-large-language-models/). arXiv preprint arXiv:2306.13549.
264. Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Lu Sheng, Lei Bai, Xiaoshui Huang, Zhiyong Wang, et al. 2023b. Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. arXiv preprint arXiv:2306.06687. [find](https://scholar.google.com/scholar?q=Zhenfei+Yin%2C+Jiong+Wang%2C+Jianjian+Cao%2C+Zhelun+Shi%2C+Dingning+Liu%2C+Mukai+Li%2C+Lu+Sheng%2C+Lei+Bai%2C+Xiaoshui+Huang%2C+Zhiyong+Wang%2C+et+al%2E+2023b%2E+Lamm%3A+Language%2Dassisted+multi%2Dmodal+instruction%2Dtuning+dataset)
265. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78. [find](https://scholar.google.com/scholar?q=Peter+Young%2C+Alice+Lai%2C+Micah+Hodosh%2C+and+Julia+Hockenmaier%2E+2014%2E+From+image+descriptions+to+visual+denotations%3A+New+similarity+metrics+for+semantic+inference+over+event+descriptions%2E+Transactions+of)
266. Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. 2016. [Modeling context in referring expressions](https://md.research.elamur.ai/free/en/papers/yu-2016-modeling-context-referring-expressions/). In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69–85. Springer.
267. Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. 2023a. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint arXiv:2309.02591. — [arXiv:2309.02591](https://arxiv.org/abs/2309.02591) [find](https://scholar.google.com/scholar?q=Lili+Yu%2C+Bowen+Shi%2C+Ramakanth+Pasunuru%2C+Benjamin+Muller%2C+Olga+Golovneva%2C+Tianlu+Wang%2C+Arun+Babu%2C+Binh+Tang%2C+Brian+Karrer%2C+Shelly+Sheynin%2C+et+al%2E+2023a%2E+Scaling+autoregressive+multi%2Dmodal+models%3A+Pretr)
268. Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. 2023b. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. arXiv preprint arXiv:2312.00849. [find](https://scholar.google.com/scholar?q=Tianyu+Yu%2C+Yuan+Yao%2C+Haoye+Zhang%2C+Taiwen+He%2C+Yifeng+Han%2C+Ganqu+Cui%2C+Jinyi+Hu%2C+Zhiyuan+Liu%2C+Hai%2DTao+Zheng%2C+Maosong+Sun%2C+et+al%2E+2023b%2E+Rlhf%2Dv%3A+Towards+trustworthy+mllms+via+behavior+alignment+from+fine%2D)
269. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023c. [Mm-vet: Evaluating large multimodal models for integrated capabilities](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/). arXiv preprint arXiv:2308.02490.
270. Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. 2022. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19313–19322. [find](https://scholar.google.com/scholar?q=Xumin+Yu%2C+Lulu+Tang%2C+Yongming+Rao%2C+Tiejun+Huang%2C+Jie+Zhou%2C+and+Jiwen+Lu%2E+2022%2E+Point%2Dbert%3A+Pre%2Dtraining+3d+point+cloud+transformers+with+masked+point+modeling%2E+In+Proceedings+of+the+IEEE%2FCVF+Conferenc)
271. Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. 2023a. Osprey: Pixel Understanding with Visual Instruction Tuning. arXiv preprint arXiv:2312.10032. — [arXiv:2312.10032](https://arxiv.org/abs/2312.10032) [find](https://scholar.google.com/scholar?q=Yuqian+Yuan%2C+Wentong+Li%2C+Jian+Liu%2C+Dongqi+Tang%2C+Xinjie+Luo%2C+Chi+Qin%2C+Lei+Zhang%2C+and+Jianke+Zhu%2E+2023a%2E+Osprey%3A+Pixel+Understanding+with+Visual+Instruction+Tuning%2E+arXiv+preprint+arXiv%3A2312%2E10032%2E+%E2%80%94+%5Ba)
272. Zhengqing Yuan, Zhaoxu Li, and Lichao Sun. 2023b. TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones. arXiv preprint arXiv:2312.16862. — [arXiv:2312.16862](https://arxiv.org/abs/2312.16862) [find](https://scholar.google.com/scholar?q=Zhengqing+Yuan%2C+Zhaoxu+Li%2C+and+Lichao+Sun%2E+2023b%2E+TinyGPT%2DV%3A+Efficient+Multimodal+Large+Language+Model+via+Small+Backbones%2E+arXiv+preprint+arXiv%3A2312%2E16862%2E+%E2%80%94+%5BarXiv%3A2312%2E16862%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
273. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2023. [Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/). arXiv preprint arXiv:2311.16502.
274. Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. 2022. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter… [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Jiasen+Lu%2C+Ximing+Lu%2C+Youngjae+Yu%2C+Yanpeng+Zhao%2C+Mohammadreza+Salehi%2C+Aditya+Kusupati%2C+Jack+Hessel%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2022%2E+Merlot+reserve%3A+Neural+script+knowledge+through+vi)
275. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022a. [GLM-130B: An Open Bilingual Pre-trained Model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/). In The Eleventh International Conference on Learning Representations.
276. Yan Zeng, Hanbo Zhang, Jiani Zheng, Jiangnan Xia, Guoqiang Wei, Yang Wei, Yuchen Zhang, and Tao Kong. 2023. What Matters in Training a GPT4-Style Language Model with Multimodal Inputs? arXiv preprint arXiv:2307.02469. [find](https://scholar.google.com/scholar?q=Yan+Zeng%2C+Hanbo+Zhang%2C+Jiani+Zheng%2C+Jiangnan+Xia%2C+Guoqiang+Wei%2C+Yang+Wei%2C+Yuchen+Zhang%2C+and+Tao+Kong%2E+2023%2E+What+Matters+in+Training+a+GPT4%2DStyle+Language+Model+with+Multimodal+Inputs%3F+arXiv+preprint+)
277. Yan Zeng, Xinsong Zhang, and Hang Li. 2022b. Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts. In International Conference on Machine Learning, pages 25994–26009. PMLR. [find](https://scholar.google.com/scholar?q=Yan+Zeng%2C+Xinsong+Zhang%2C+and+Hang+Li%2E+2022b%2E+Multi%2DGrained+Vision+Language+Pre%2DTraining%3A+Aligning+Texts+with+Visual+Concepts%2E+In+International+Conference+on+Machine+Learning%2C+pages+25994%E2%80%9326009%2E+PMLR%2E)
278. Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023a. [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://md.research.elamur.ai/free/en/papers/zhang-2023-speechgpt-empowering-large-language-models/). In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, page…
279. Duzhen Zhang, Wei Cong, Jiahua Dong, Yahan Yu, Xiuyi Chen, Yonggang Zhang, and Zhen Fang. 2023b. Continual Named Entity Recognition without Catastrophic Forgetting. In The 2023 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Duzhen+Zhang%2C+Wei+Cong%2C+Jiahua+Dong%2C+Yahan+Yu%2C+Xiuyi+Chen%2C+Yonggang+Zhang%2C+and+Zhen+Fang%2E+2023b%2E+Continual+Named+Entity+Recognition+without+Catastrophic+Forgetting%2E+In+The+2023+Conference+on+Empirical)
280. Duzhen Zhang, Hongliu Li, Wei Cong, Rongtao Xu, Jiahua Dong, and Xiuyi Chen. 2023c. Task relation distillation and prototypical pseudo label for incremental named entity recognition. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 3319–3329. [find](https://scholar.google.com/scholar?q=Duzhen+Zhang%2C+Hongliu+Li%2C+Wei+Cong%2C+Rongtao+Xu%2C+Jiahua+Dong%2C+and+Xiuyi+Chen%2E+2023c%2E+Task+relation+distillation+and+prototypical+pseudo+label+for+incremental+named+entity+recognition%2E+In+Proceedings+of)
281. Duzhen Zhang, Yahan Yu, Feilong Chen, and Xiuyi Chen. 2023d. Decomposing Logits Distillation for Incremental Named Entity Recognition. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1919–1923. [find](https://scholar.google.com/scholar?q=Duzhen+Zhang%2C+Yahan+Yu%2C+Feilong+Chen%2C+and+Xiuyi+Chen%2E+2023d%2E+Decomposing+Logits+Distillation+for+Incremental+Named+Entity+Recognition%2E+In+Proceedings+of+the+46th+International+ACM+SIGIR+Conference+on+)
282. Duzhen Zhang, Tielin Zhang, Shuncheng Jia, Qingyu Wang, and Bo Xu. 2022a. Recent Advances and New Frontiers in Spiking Neural Networks. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 5670–5677. [find](https://scholar.google.com/scholar?q=Duzhen+Zhang%2C+Tielin+Zhang%2C+Shuncheng+Jia%2C+Qingyu+Wang%2C+and+Bo+Xu%2E+2022a%2E+Recent+Advances+and+New+Frontiers+in+Spiking+Neural+Networks%2E+In+Proceedings+of+the+Thirty%2DFirst+International+Joint+Conferenc)
283. Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang Cheng, Chunpu Xu, Shuyue Guo, et al. 2024a. CM-MMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark. arXiv preprint arXiv:2401.11944. — [arXiv:2401.11944](https://arxiv.org/abs/2401.11944) [find](https://scholar.google.com/scholar?q=Ge+Zhang%2C+Xinrun+Du%2C+Bei+Chen%2C+Yiming+Liang%2C+Tongxu+Luo%2C+Tianyu+Zheng%2C+Kang+Zhu%2C+Yuyang+Cheng%2C+Chunpu+Xu%2C+Shuyue+Guo%2C+et+al%2E+2024a%2E+CM%2DMMU%3A+A+Chinese+Massive+Multi%2Ddiscipline+Multimodal+Understanding+)
284. Hang Zhang, Xin Li, and Lidong Bing. 2023e. [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://md.research.elamur.ai/free/en/papers/zhang-2023-video-llama-instruction-tuned-audio/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 -System Demonstrations, Singapore, December 6-10, 2023, pages 543–5…
285. Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung-Yeung Shum. 2022b. DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Hao+Zhang%2C+Feng+Li%2C+Shilong+Liu%2C+Lei+Zhang%2C+Hang+Su%2C+Jun+Zhu%2C+Lionel+Ni%2C+and+Heung%2DYeung+Shum%2E+2022b%2E+DINO%3A+DETR+with+Improved+DeNoising+Anchor+Boxes+for+End%2Dto%2DEnd+Object+Detection%2E+In+The+Eleventh+I)
286. Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. 2020. Side-tuning: a baseline for network adaptation via additive side networks. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, pages 698–714. Springer… [find](https://scholar.google.com/scholar?q=Jeffrey+O+Zhang%2C+Alexander+Sax%2C+Amir+Zamir%2C+Leonidas+Guibas%2C+and+Jitendra+Malik%2E+2020%2E+Side%2Dtuning%3A+a+baseline+for+network+adaptation+via+additive+side+networks%2E+In+Computer+Vision%E2%80%93ECCV+2020%3A+16th+Eur)
287. Jiaxin Zhang, Zhongzhi Li, Mingliang Zhang, Fei Yin, Chenglin Liu, and Yashar Moshfeghi. 2024b. Geo-Eval: Benchmark for Evaluating LLMs and Multi-Modal Models on Geometry Problem-Solving. [find](https://scholar.google.com/scholar?q=Jiaxin+Zhang%2C+Zhongzhi+Li%2C+Mingliang+Zhang%2C+Fei+Yin%2C+Chenglin+Liu%2C+and+Yashar+Moshfeghi%2E+2024b%2E+Geo%2DEval%3A+Benchmark+for+Evaluating+LLMs+and+Multi%2DModal+Models+on+Geometry+Problem%2DSolving%2E)
288. Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. 2023f. [Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition](https://md.research.elamur.ai/free/en/papers/zhang-2023-internlm-xcomposer-vision-language/). arXiv preprint arXiv:2309.15112.
289. Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. 2023g. Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv preprint arXiv:2307.03601. — [arXiv:2307.03601](https://arxiv.org/abs/2307.03601) [find](https://scholar.google.com/scholar?q=Shilong+Zhang%2C+Peize+Sun%2C+Shoufa+Chen%2C+Min+Xiao%2C+Wenqi+Shao%2C+Wenwei+Zhang%2C+Kai+Chen%2C+and+Ping+Luo%2E+2023g%2E+Gpt4roi%3A+Instruction+tuning+large+language+model+on+region%2Dof%2Dinterest%2E+arXiv+preprint+arXiv%3A2)
290. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022c. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068.
291. Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. 2023h. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107. [find](https://scholar.google.com/scholar?q=Yanzhe+Zhang%2C+Ruiyi+Zhang%2C+Jiuxiang+Gu%2C+Yufan+Zhou%2C+Nedim+Lipka%2C+Diyi+Yang%2C+and+Tong+Sun%2E+2023h%2E+Llavar%3A+Enhanced+visual+instruction+tuning+for+text%2Drich+image+understanding%2E+arXiv+preprint+arXiv%3A2306)
292. Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. 2023i. Recognize Anything: A Strong Image Tagging Model. arXiv preprint arXiv:2306.03514. — [arXiv:2306.03514](https://arxiv.org/abs/2306.03514) [find](https://scholar.google.com/scholar?q=Youcai+Zhang%2C+Xinyu+Huang%2C+Jinyu+Ma%2C+Zhaoyang+Li%2C+Zhaochuan+Luo%2C+Yanchun+Xie%2C+Yuzhuo+Qin%2C+Tong+Luo%2C+Yaqian+Li%2C+Shilong+Liu%2C+et+al%2E+2023i%2E+Recognize+Anything%3A+A+Strong+Image+Tagging+Model%2E+arXiv+prepri)
293. Bingchen Zhao, Haoqin Tu, Chen Wei, and Cihang Xie. 2024. Tuning LayerNorm in Attention: Towards Efficient Multimodal LLM Finetuning. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Bingchen+Zhao%2C+Haoqin+Tu%2C+Chen+Wei%2C+and+Cihang+Xie%2E+2024%2E+Tuning+LayerNorm+in+Attention%3A+Towards+Efficient+Multimodal+LLM+Finetuning%2E+In+The+Twelfth+International+Conference+on+Learning+Representation)
294. Bo Zhao, Boya Wu, and Tiejun Huang. 2023a. Svit: Scaling up visual instruction tuning. arXiv preprint arXiv:2307.04087. — [arXiv:2307.04087](https://arxiv.org/abs/2307.04087) [find](https://scholar.google.com/scholar?q=Bo+Zhao%2C+Boya+Wu%2C+and+Tiejun+Huang%2E+2023a%2E+Svit%3A+Scaling+up+visual+instruction+tuning%2E+arXiv+preprint+arXiv%3A2307%2E04087%2E+%E2%80%94+%5BarXiv%3A2307%2E04087%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2307%2E04087%29)
295. Liang Zhao, En Yu, Zheng Ge, Jinrong Yang, Haoran Wei, Hongyu Zhou, Jianjian Sun, Yuang Peng, Runpei Dong, Chunrui Han, et al. 2023b. Chatspot: Bootstrapping multimodal llms via precise referring instruction tuning. arXiv preprint arXiv:2307.09474. — [arXiv:2307.09474](https://arxiv.org/abs/2307.09474) [find](https://scholar.google.com/scholar?q=Liang+Zhao%2C+En+Yu%2C+Zheng+Ge%2C+Jinrong+Yang%2C+Haoran+Wei%2C+Hongyu+Zhou%2C+Jianjian+Sun%2C+Yuang+Peng%2C+Runpei+Dong%2C+Chunrui+Han%2C+et+al%2E+2023b%2E+Chatspot%3A+Bootstrapping+multimodal+llms+via+precise+referring+inst)
296. Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. 2022. EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equations. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA,… [find](https://scholar.google.com/scholar?q=Min+Zhao%2C+Fan+Bao%2C+Chongxuan+Li%2C+and+Jun+Zhu%2E+2022%2E+EGSDE%3A+Unpaired+Image%2Dto%2DImage+Translation+via+Energy%2DGuided+Stochastic+Differential+Equations%2E+In+Advances+in+Neural+Information+Processing+Systems)
297. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023c. [A survey of large language models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/). arXiv preprint arXiv:2303.18223.
298. Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2016. [Visual7w: Grounded question answering in images](https://md.research.elamur.ai/free/en/papers/zhu-2015-visual7w-grounded-question-answering-images/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4995–5004.
299. Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, and Bingyi Kang. 2023d. Bubogpt: Enabling visual grounding in multi-modal llms. arXiv preprint arXiv:2307.08581. — [arXiv:2307.08581](https://arxiv.org/abs/2307.08581) [find](https://scholar.google.com/scholar?q=Yang+Zhao%2C+Zhijie+Lin%2C+Daquan+Zhou%2C+Zilong+Huang%2C+Jiashi+Feng%2C+and+Bingyi+Kang%2E+2023d%2E+Bubogpt%3A+Enabling+visual+grounding+in+multi%2Dmodal+llms%2E+arXiv+preprint+arXiv%3A2307%2E08581%2E+%E2%80%94+%5BarXiv%3A2307%2E08581%5D%28htt)
300. Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, and Lei Zhang. 2023. Delta-lora: Fine-tuning high-rank parameters with the delta of low-rank matrices. arXiv preprint arXiv:2309.02411. — [arXiv:2309.02411](https://arxiv.org/abs/2309.02411) [find](https://scholar.google.com/scholar?q=Bojia+Zi%2C+Xianbiao+Qi%2C+Lingzhi+Wang%2C+Jianan+Wang%2C+Kam%2DFai+Wong%2C+and+Lei+Zhang%2E+2023%2E+Delta%2Dlora%3A+Fine%2Dtuning+high%2Drank+parameters+with+the+delta+of+low%2Drank+matrices%2E+arXiv+preprint+arXiv%3A2309%2E02411%2E+)
301. Junhao Zheng, Qianli Ma, Zhen Liu, Binquan Wu, and Huawen Feng. 2024. Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfer. arXiv preprint arXiv:2401.09181. — [arXiv:2401.09181](https://arxiv.org/abs/2401.09181) [find](https://scholar.google.com/scholar?q=Junhao+Zheng%2C+Qianli+Ma%2C+Zhen+Liu%2C+Binquan+Wu%2C+and+Huawen+Feng%2E+2024%2E+Beyond+Anti%2DForgetting%3A+Multimodal+Continual+Instruction+Tuning+with+Positive+Forward+Transfer%2E+arXiv+preprint+arXiv%3A2401%2E09181%2E+%E2%80%94)
302. Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Timothy Hospedales. 2024. Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models. arXiv preprint arXiv:2402.02207. [find](https://scholar.google.com/scholar?q=Yongshuo+Zong%2C+Ondrej+Bohdal%2C+Tingyang+Yu%2C+Yongxin+Yang%2C+and+Timothy+Hospedales%2E+2024%2E+Safety+Fine%2DTuning+at+%28Almost%29+No+Cost%3A+A+Baseline+for+Vision+Large+Language+Models%2E+arXiv+preprint+arXiv%3A2402%2E02)
303. Junhao Zheng, Shengjie Qiu, and Qianli Ma. 2023a. Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Models. arXiv preprint arXiv:2312.07887. [find](https://scholar.google.com/scholar?q=Junhao+Zheng%2C+Shengjie+Qiu%2C+and+Qianli+Ma%2E+2023a%2E+Learn+or+Recall%3F+Revisiting+Incremental+Learning+with+Pre%2Dtrained+Language+Models%2E+arXiv+preprint+arXiv%3A2312%2E07887%2E)
304. Kaizhi Zheng, Xuehai He, and Xin Eric Wang. 2023b. Minigpt-5: Interleaved vision-and-language generation via generative vokens. arXiv preprint arXiv:2310.02239. — [arXiv:2310.02239](https://arxiv.org/abs/2310.02239) [find](https://scholar.google.com/scholar?q=Kaizhi+Zheng%2C+Xuehai+He%2C+and+Xin+Eric+Wang%2E+2023b%2E+Minigpt%2D5%3A+Interleaved+vision%2Dand%2Dlanguage+generation+via+generative+vokens%2E+arXiv+preprint+arXiv%3A2310%2E02239%2E+%E2%80%94+%5BarXiv%3A2310%2E02239%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
305. Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. 2024a. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. In The Twelfth International Conference on Learning Representations… [find](https://scholar.google.com/scholar?q=Bin+Zhu%2C+Bin+Lin%2C+Munan+Ning%2C+Yang+Yan%2C+Jiaxi+Cui%2C+HongFa+Wang%2C+Yatian+Pang%2C+Wenhao+Jiang%2C+Junwu+Zhang%2C+Zongwei+Li%2C+et+al%2E+2024a%2E+Languagebind%3A+Extending+video%2Dlanguage+pretraining+to+n%2Dmodality+by+la)
306. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023a. [Minigpt-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/). arXiv preprint arXiv:2304.10592.
307. Dongsheng Zhu, Xunzhu Tang, Weidong Han, Jinghui Lu, Yukun Zhao, Guoliang Xing, Junfeng Wang, and Dawei Yin. 2024b. VisLingInstruct: Elevating Zero-Shot Learning in Multi-Modal Language Models with Autonomous Instruction Optimization. arXiv preprint arXiv:2402.07398. [find](https://scholar.google.com/scholar?q=Dongsheng+Zhu%2C+Xunzhu+Tang%2C+Weidong+Han%2C+Jinghui+Lu%2C+Yukun+Zhao%2C+Guoliang+Xing%2C+Junfeng+Wang%2C+and+Dawei+Yin%2E+2024b%2E+VisLingInstruct%3A+Elevating+Zero%2DShot+Learning+in+Multi%2DModal+Language+Models+with+Au)
308. Jinguo Zhu, Xiaohan Ding, Yixiao Ge, Yuying Ge, Sijie Zhao, Hengshuang Zhao, Xiaohua Wang, and Ying Shan. 2023b. Vl-gpt: A generative pre-trained transformer for vision and language understanding and generation. arXiv preprint arXiv:2312.09251. — [arXiv:2312.09251](https://arxiv.org/abs/2312.09251) [find](https://scholar.google.com/scholar?q=Jinguo+Zhu%2C+Xiaohan+Ding%2C+Yixiao+Ge%2C+Yuying+Ge%2C+Sijie+Zhao%2C+Hengshuang+Zhao%2C+Xiaohua+Wang%2C+and+Ying+Shan%2E+2023b%2E+Vl%2Dgpt%3A+A+generative+pre%2Dtrained+transformer+for+vision+and+language+understanding+and+)
309. Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. 2023c. Multimodal c4: An open, billion-scale corpus of images interleaved with text. arXiv preprint arXiv:2304.06939. — [arXiv:2304.06939](https://arxiv.org/abs/2304.06939) [find](https://scholar.google.com/scholar?q=Wanrong+Zhu%2C+Jack+Hessel%2C+Anas+Awadalla%2C+Samir+Yitzhak+Gadre%2C+Jesse+Dodge%2C+Alex+Fang%2C+Youngjae+Yu%2C+Ludwig+Schmidt%2C+William+Yang+Wang%2C+and+Yejin+Choi%2E+2023c%2E+Multimodal+c4%3A+An+open%2C+billion%2Dscale+corpu)
310. Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. 2024c. LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Model. arXiv preprint arXiv:2401.02330. — [arXiv:2401.02330](https://arxiv.org/abs/2401.02330) [find](https://scholar.google.com/scholar?q=Yichen+Zhu%2C+Minjie+Zhu%2C+Ning+Liu%2C+Zhicai+Ou%2C+Xiaofeng+Mou%2C+and+Jian+Tang%2E+2024c%2E+LLaVA%2DPhi%3A+Efficient+Multi%2DModal+Assistant+with+Small+Language+Model%2E+arXiv+preprint+arXiv%3A2401%2E02330%2E+%E2%80%94+%5BarXiv%3A2401%2E02)

*Updated 24 October 2026*