--- title: "A Survey on Multimodal Large Language Models — Yin 2023" description: "This survey by Yin and colleagues (USTC and Tencent YouTu Lab) maps the fast-moving field of Multimodal Large Language Models (MLLMs)." authors: ["Shukang Yin", "Chaoyou Fu", "Sirui Zhao", "Ke Li", "Xing Sun", "Tong Xu", "Enhong Chen"] year: 2023 venue: "arXiv:2306.13549" source: "https://arxiv.org/abs/2306.13549" doi: "10.1093/nsr/nwae403" arxiv: "2306.13549" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/yin-2023-survey-multimodal-large-language-models/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yin-2023-survey-multimodal-large-language-models/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/) / A Survey on Multimodal Large Language Models # A Survey on Multimodal Large Language Models **Authors:** Shukang Yin, [Chaoyou Fu](https://md.research.elamur.ai/free/en/authors/chaoyou-fu/), Sirui Zhao, [Ke Li](https://md.research.elamur.ai/free/en/authors/ke-li/), [Xing Sun](https://md.research.elamur.ai/free/en/authors/xing-sun/), Tong Xu, [Enhong Chen](https://md.research.elamur.ai/free/en/authors/enhong-chen/) · **Year:** 2023 · **Venue:** arXiv:2306.13549 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://doi.org/10.1093/nsr/nwae403) · [PDF](https://arxiv.org/pdf/2306.13549) · [DOI 10.1093/nsr/nwae403](https://doi.org/10.1093/nsr/nwae403) · [arXiv 2306.13549](https://arxiv.org/abs/2306.13549) · [OpenAlex](https://openalex.org/W4404356490) **Citations:** 853 citations (openalex-snapshot · 2026-10-14) · [207](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary This survey by Yin and colleagues (USTC and Tencent YouTu Lab) maps the fast-moving field of Multimodal Large Language Models (MLLMs). These systems use a pre-trained LLM as the reasoning core and attach perception components so the model can accept, and sometimes generate, non-text signals. The motivation is that two kinds of models have complementary gaps: text-only LLMs cannot perceive images, while large vision models see well but reason weakly. The authors describe theirs as the first survey on the topic. They organise the literature around three areas: - architecture: modality encoder, connector, LLM and an optional output generator - training strategy and data - evaluation practice They then cover extensions toward finer region-level control, extra modalities and languages beyond English (e.g. Chinese with limited corpora), and new domains such as medicine, documents and agents. Further sections address multimodal hallucination and three techniques: M-ICL, M-CoT and LLM-Aided Visual Reasoning (LAVR). The paper introduces no new model or dataset. Its value is a structured reference plus a continuously updated paper list, which is useful when planning vision-language work for under-resourced languages like Ukrainian. ## Key points - Connectors fall into three families: projection-based, query-based (e.g. Q-Former) and fusion-based. The first two merge features at the token level, while fusion-based connectors combine them at the feature level inside the LLM. - The image encoders surveyed range from CLIP-ViT-L/14 (OpenAI WIT, 224/336 px, ~304M parameters) to OpenCLIP-ViT-bigG/14 (LAION-2B, ~1.84B parameters). Fuyu-8B skips the encoder entirely, and ImageBind-LLM also handles audio, depth, thermal and IMU input. - Empirical studies cited in the survey find that input resolution matters more than encoder size or pretraining data mix. Resolution is raised in two ways: - direct scaling, e.g. CogAgent's dual encoder - patch division, e.g. Monkey and SPHINX - Common LLM backbones include Flan-T5 (used in BLIP-2 and InstructBLIP) and LLaMA/Vicuna, which are mostly English-centric, and Qwen, which is bilingual in Chinese and English. Liu et al. report broad gains from scaling the LLM from 7B to 13B. - The survey is maintained over time (v4, Nov 2024), with a companion paper list on GitHub: BradyFU/Awesome-Multimodal-Large-Language-Models. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 207 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 197 not yet. - [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/) - [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://md.research.elamur.ai/free/en/papers/fu-2024-video-mme-first-ever-comprehensive/) - [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/guan-2023-hallusionbench-advanced-diagnostic-suite-entangled/) - [LLaVA-OneVision: Easy Visual Task Transfer](https://md.research.elamur.ai/free/en/papers/li-2024-llava-onevision-easy-visual-task/) - [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://md.research.elamur.ai/free/en/papers/lin-2023-video-llava-learning-united-visual/) - [A Comprehensive Overview of Large Language Models](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/) - [Seeing the Other Side: Diagnostic Tasks for Viewpoint Reasoning in Vision–Language Models](https://md.research.elamur.ai/free/en/papers/takenaka-2026-seeing-other-side-diagnostic-tasks/) - [The Rise and Potential of Large Language Model Based Agents: A Survey](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/) - [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://md.research.elamur.ai/free/en/papers/zhang-2023-video-llama-instruction-tuned-audio/) - [大模型驱动的文本属性图学习研究进展](https://md.research.elamur.ai/free/en/papers/zheng-2026-recent-advances-text/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{yin2023survey, title = {{A Survey on Multimodal Large Language Models}}, author = {Yin, Shukang and Fu, Chaoyou and Zhao, Sirui and Li, Ke and Sun, Xing and Xu, Tong and Chen, Enhong}, year = {2023}, doi = {10.1093/nsr/nwae403}, eprint = {2306.13549}, archivePrefix = {arXiv}, url = {https://doi.org/10.1093/nsr/nwae403} } ``` - [← previous A Survey on Large Language Model based Autonomous Agents](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-large-language-model-based/) - [next → A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://md.research.elamur.ai/free/en/papers/laskar-2023-systematic-study-comprehensive-evaluation-chatgpt/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong et al., “A survey of large language models,” arXiv:2303.18223, 2023. 1 OpenAI, “Chatgpt: A language model for conversational ai,” OpenAI, Tech. Rep., 2023. [Online]. Available: [https://www.openai.com/research/…](https://www.openai.com/research/…) [↗](https://www.openai.com/research/…) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)