---
title: "SEED-Bench: Benchmarking Multimodal LLMs with — Li 2023"
description: "Multimodal LLMs are usually tested on a handful of hand-picked examples, or on older VQA datasets that expect a one- or two-word exact match."
authors: ["Bohao Li", "Rui Wang", "Guangzhi Wang", "Yuying Ge", "Yixiao Ge", "Ying Shan"]
year: 2023
venue: "arXiv:2307.16125"
source: "https://arxiv.org/abs/2307.16125"
doi: "10.48550/arxiv.2307.16125"
arxiv: "2307.16125"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2023-seed-bench-benchmarking-multimodal-llms/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2023-seed-bench-benchmarking-multimodal-llms/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/) / SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension

# SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension

**Authors:** Bohao Li, [Rui Wang](https://md.research.elamur.ai/free/en/authors/rui-wang/), Guangzhi Wang, Yuying Ge, Yixiao Ge, Ying Shan · **Year:** 2023 · **Venue:** arXiv:2307.16125 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2307.16125) · [PDF](https://arxiv.org/pdf/2307.16125) · [DOI 10.48550/arxiv.2307.16125](https://doi.org/10.48550/arxiv.2307.16125) · [arXiv 2307.16125](https://arxiv.org/abs/2307.16125) · [OpenAlex](https://openalex.org/W4385473806)

**Citations:** 53 citations (OpenAlex, 2026-10-14) · [194](https://md.research.elamur.ai/free/en/papers/li-2023-seed-bench-benchmarking-multimodal-llms/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Multimodal LLMs are usually tested on a handful of hand-picked examples, or on older VQA datasets that expect a one- or two-word exact match. Newer MLLM benchmarks have their own problems. Some depend on human or GPT judges, which makes scoring slow and subjective. Others, like MME and MMBench, are small, so their scores are noisy. SEED-Bench is a set of 19,242 multiple-choice questions with human-checked answers, covering 12 skills across images and video. To build it, the authors had off-the-shelf vision models describe each image (captions, object-level descriptions, OCR text); for videos they reused the existing human annotations. ChatGPT/GPT-4 turned these descriptions into four-option questions. Several text-only LLMs then removed any question they could answer without seeing the visual, and human annotators confirmed the correct option and the skill category. Models are scored by the log-likelihood of each option, so they don't need to follow instructions well enough to output a letter. Across 18 models, most scored below 50%. That gives the community a large, objective, reproducible test of visual understanding.

## Key points
- **Scale:** 19K questions, roughly 6× MMBench and 9× MME. There are 9 image skills (e.g., scene understanding, instance counting, spatial relation, text recognition) and 3 video skills (action recognition, action prediction, procedure understanding).
- **Leaderboard:** InstructBLIP Vicuna has the best average at 53.37%, ahead of InstructBLIP (52.73%) and BLIP2 (46.35%). Text-only baselines Vicuna, Flan-T5 and LLaMA score about 27–28%.
- **InstructBLIP** leads on 8 of the 12 skills. The authors credit its 16M-sample instruction-tuning set and its frozen LLM weights.
- **Weak spots:**
  - Spatial relations: the best model reaches only about 40%.
  - Text recognition: every model except InstructBLIP is below 40%.
  - Procedure understanding: the best model is only about 5 points above LLaMA.
- **VideoLLMs** (VideoChat, Video-ChatGPT, Valley) are no better than image-only models at temporal understanding. Code and the leaderboard are at github.com/AILab-CVC/SEED-Bench.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

Topics: [Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/)

## Institutions

- [Tencent](https://md.research.elamur.ai/free/en/companies/tencent/) — Company, Shenzhen, China · 40 works · 13 547 citations

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{li2023seed,
  title = {{SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension}},
  author = {Li, Bohao and Wang, Rui and Wang, Guangzhi and Ge, Yuying and Ge, Yixiao and Shan, Ying},
  year = {2023},
  doi = {10.48550/arxiv.2307.16125},
  eprint = {2307.16125},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.16125}
}
```

- [← previous LeanDojo: Theorem Proving with Retrieval-Augmented Language Models](https://md.research.elamur.ai/free/en/papers/yang-2023-leandojo-theorem-proving-retrieval-augmented/)
- [next → Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecture](https://md.research.elamur.ai/free/en/papers/cui-2023-chatlaw-multi-agent-legal-assistant/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/)

## References

42 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXiv+prep) [our card](https://md.research.elamur.ai/free/en/papers/chung-2022-scaling-instruction-finetuned-language-models/)
2. OpenAI. Gpt-4 technical report, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
3. OpenAI. Introducing chatgpt. [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt), 2022. [↗](https://openai.com/blog/chatgpt)
4. FastChat. Vicuna. [https://github.com/lm-sys/FastChat](https://github.com/lm-sys/FastChat), 2023. [↗](https://github.com/lm-sys/FastChat)
5. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Thibaut+Lavril%2C+Gautier+Izacard%2C+Xavier+Martinet%2C+Marie%2DAnne+Lachaux%2C+Timoth%C3%A9e+Lacroix%2C+Baptiste+Rozi%C3%A8re%2C+Naman+Goyal%2C+Eric+Hambro%2C+Faisal+Azhar%2C+et+al%2E+Llama%3A+Open+and+efficient+foundat) [our card](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)
6. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ICML, 2023. [find](https://scholar.google.com/scholar?q=Junnan+Li%2C+Dongxu+Li%2C+Silvio+Savarese%2C+and+Steven+Hoi%2E+Blip%2D2%3A+Bootstrapping+language%2Dimage+pre%2Dtraining+with+frozen+image+encoders+and+large+language+models%2E+ICML%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/)
7. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. [find](https://scholar.google.com/scholar?q=Deyao+Zhu%2C+Jun+Chen%2C+Xiaoqian+Shen%2C+Xiang+Li%2C+and+Mohamed+Elhoseiny%2E+Minigpt%2D4%3A+Enhancing+vision%2Dlanguage+understanding+with+advanced+large+language+models%2E+arXiv+preprint+arXiv%3A2304%2E10592%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/)
8. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023. [find](https://scholar.google.com/scholar?q=Haotian+Liu%2C+Chunyuan+Li%2C+Qingyang+Wu%2C+and+Yong+Jae+Lee%2E+Visual+instruction+tuning%2E+arXiv+preprint+arXiv%3A2304%2E08485%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/)
9. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023. [find](https://scholar.google.com/scholar?q=Qinghao+Ye%2C+Haiyang+Xu%2C+Guohai+Xu%2C+Jiabo+Ye%2C+Ming+Yan%2C+Yiyang+Zhou%2C+Junyang+Wang%2C+Anwen+Hu%2C+Pengcheng+Shi%2C+Yaya+Shi%2C+et+al%2E+mplug%2Dowl%3A+Modularization+empowers+large+language+models+with+multimodality%2E) [our card](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/)
10. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500, 2023. [find](https://scholar.google.com/scholar?q=Wenliang+Dai%2C+Junnan+Li%2C+Dongxu+Li%2C+Anthony+Meng+Huat+Tiong%2C+Junqi+Zhao%2C+Weisheng+Wang%2C+Boyang+Li%2C+Pascale+Fung%2C+and+Steven+Hoi%2E+Instructblip%3A+Towards+general%2Dpurpose+vision%2Dlanguage+models+with+instr) [our card](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/)
11. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023. [find](https://scholar.google.com/scholar?q=Bo+Li%2C+Yuanhan+Zhang%2C+Liangyu+Chen%2C+Jinghao+Wang%2C+Jingkang+Yang%2C+and+Ziwei+Liu%2E+Otter%3A+A+multi%2Dmodal+model+with+in%2Dcontext+instruction+tuning%2E+arXiv+preprint+arXiv%3A2305%2E03726%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-otter-multi-modal-model-context/)
12. Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans, 2023. [find](https://scholar.google.com/scholar?q=Tao+Gong%2C+Chengqi+Lyu%2C+Shilong+Zhang%2C+Yudong+Wang%2C+Miao+Zheng%2C+Qian+Zhao%2C+Kuikun+Liu%2C+Wenwei+Zhang%2C+Ping+Luo%2C+and+Kai+Chen%2E+Multimodal%2Dgpt%3A+A+vision+and+language+model+for+dialogue+with+humans%2C+2023%2E)
13. Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305.16355, 2023. [find](https://scholar.google.com/scholar?q=Yixuan+Su%2C+Tian+Lan%2C+Huayang+Li%2C+Jialu+Xu%2C+Yan+Wang%2C+and+Deng+Cai%2E+Pandagpt%3A+One+model+to+instruction%2Dfollow+them+all%2E+arXiv+preprint+arXiv%3A2305%2E16355%2C+2023%2E)
14. Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023. [find](https://scholar.google.com/scholar?q=Zhiliang+Peng%2C+Wenhui+Wang%2C+Li+Dong%2C+Yaru+Hao%2C+Shaohan+Huang%2C+Shuming+Ma%2C+and+Furu+Wei%2E+Kosmos%2D2%3A+Grounding+multimodal+large+language+models+to+the+world%2E+arXiv+preprint+arXiv%3A2306%2E14824%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/peng-2023-kosmos-2-grounding-multimodal-large/)
15. KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. [find](https://scholar.google.com/scholar?q=KunChang+Li%2C+Yinan+He%2C+Yi+Wang%2C+Yizhuo+Li%2C+Wenhai+Wang%2C+Ping+Luo%2C+Yali+Wang%2C+Limin+Wang%2C+and+Yu+Qiao%2E+Videochat%3A+Chat%2Dcentric+video+understanding%2E+arXiv+preprint+arXiv%3A2305%2E06355%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-videochat-chat-centric-video-understanding/)
16. Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. [find](https://scholar.google.com/scholar?q=Muhammad+Maaz%2C+Hanoona+Rasheed%2C+Salman+Khan%2C+and+Fahad+Shahbaz+Khan%2E+Video%2Dchatgpt%3A+Towards+detailed+video+understanding+via+large+vision+and+language+models%2E+arXiv+preprint+arXiv%3A2306%2E05424%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/maaz-2023-video-chatgpt-towards-detailed-video/)
17. Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07207, 2023. — [arXiv:2306.07207](https://arxiv.org/abs/2306.07207) [find](https://scholar.google.com/scholar?q=Ruipu+Luo%2C+Ziwang+Zhao%2C+Min+Yang%2C+Junwei+Dong%2C+Minghui+Qiu%2C+Pengcheng+Lu%2C+Tao+Wang%2C+and+Zhongyu+Wei%2E+Valley%3A+Video+assistant+with+large+language+model+enhanced+ability%2E+arXiv+preprint+arXiv%3A2306%2E07207)
18. Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model. arXiv preprint arXiv:2307.08041, 2023. [find](https://scholar.google.com/scholar?q=Yuying+Ge%2C+Yixiao+Ge%2C+Ziyun+Zeng%2C+Xintao+Wang%2C+and+Ying+Shan%2E+Planting+a+seed+of+vision+in+large+language+model%2E+arXiv+preprint+arXiv%3A2307%2E08041%2C+2023%2E)
19. Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222, 2023. — [arXiv:2307.05222](https://arxiv.org/abs/2307.05222) [find](https://scholar.google.com/scholar?q=Quan+Sun%2C+Qiying+Yu%2C+Yufeng+Cui%2C+Fan+Zhang%2C+Xiaosong+Zhang%2C+Yueze+Wang%2C+Hongcheng+Gao%2C+Jingjing+Liu%2C+Tiejun+Huang%2C+and+Xinlong+Wang%2E+Generative+pretraining+in+multimodality%2E+arXiv+preprint+arXiv%3A2307%2E)
20. Yu Lili, Shi Bowen, Pasunuru Ram, Miller Benjamin, Golovneva Olga, Wang Tianlu, Babu Arun, Tang Binh, Karrer Brian, Sheynin Shelly, Ross Candace, Polyak Adam, Howes Russ, Sharma Vasu, Xu Jacob, Singer Uriel, Li (AI) Daniel, Ghosh Gargi, Taigman Yaniv, Fazel-Zarandi Maryam, Celikyilmaz Asli, Zettlemo… [find](https://scholar.google.com/scholar?q=Yu+Lili%2C+Shi+Bowen%2C+Pasunuru+Ram%2C+Miller+Benjamin%2C+Golovneva+Olga%2C+Wang+Tianlu%2C+Babu+Arun%2C+Tang+Binh%2C+Karrer+Brian%2C+Sheynin+Shelly%2C+Ross+Candace%2C+Polyak+Adam%2C+Howes+Russ%2C+Sharma+Vasu%2C+Xu+Jacob%2C+Singer)
21. Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Generating images with multimodal language models. arXiv preprint arXiv:2305.17216, 2023. — [arXiv:2305.17216](https://arxiv.org/abs/2305.17216) [find](https://scholar.google.com/scholar?q=Jing+Yu+Koh%2C+Daniel+Fried%2C+and+Ruslan+Salakhutdinov%2E+Generating+images+with+multimodal+language+models%2E+arXiv+preprint+arXiv%3A2305%2E17216%2C+2023%2E+%E2%80%94+%5BarXiv%3A2305%2E17216%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E17216%29)
22. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017. [find](https://scholar.google.com/scholar?q=Yash+Goyal%2C+Tejas+Khot%2C+Douglas+Summers%2DStay%2C+Dhruv+Batra%2C+and+Devi+Parikh%2E+Making+the+v+in+vqa+matter%3A+Elevating+the+role+of+image+understanding+in+visual+question+answering%2E+In+Proceedings+of+the+IE) [our card](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/)
23. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. [find](https://scholar.google.com/scholar?q=Chaoyou+Fu%2C+Peixian+Chen%2C+Yunhang+Shen%2C+Yulei+Qin%2C+Mengdan+Zhang%2C+Xu+Lin%2C+Zhenyu+Qiu%2C+Wei+Lin%2C+Jinrui+Yang%2C+Xiawu+Zheng%2C+Ke+Li%2C+Xing+Sun%2C+and+Rongrong+Ji%2E+Mme%3A+A+comprehensive+evaluation+benchmark+for) [our card](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/)
24. Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Lu Sheng, Lei Bai, Xiaoshui Huang, Zhiyong Wang, et al. Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. arXiv preprint arXiv:2306.06687, 2023. [find](https://scholar.google.com/scholar?q=Zhenfei+Yin%2C+Jiong+Wang%2C+Jianjian+Cao%2C+Zhelun+Shi%2C+Dingning+Liu%2C+Mukai+Li%2C+Lu+Sheng%2C+Lei+Bai%2C+Xiaoshui+Huang%2C+Zhiyong+Wang%2C+et+al%2E+Lamm%3A+Language%2Dassisted+multi%2Dmodal+instruction%2Dtuning+dataset%2C+frame)
25. Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09265, 2023. — [arXiv:2306.09265](https://arxiv.org/abs/2306.09265) [find](https://scholar.google.com/scholar?q=Peng+Xu%2C+Wenqi+Shao%2C+Kaipeng+Zhang%2C+Peng+Gao%2C+Shuo+Liu%2C+Meng+Lei%2C+Fanqing+Meng%2C+Siyuan+Huang%2C+Yu+Qiao%2C+and+Ping+Luo%2E+Lvlm%2Dehub%3A+A+comprehensive+evaluation+benchmark+for+large+vision%2Dlanguage+models%2E+a)
26. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023. [find](https://scholar.google.com/scholar?q=Yuan+Liu%2C+Haodong+Duan%2C+Yuanhan+Zhang%2C+Bo+Li%2C+Songyang+Zhang%2C+Wangbo+Zhao%2C+Yike+Yuan%2C+Jiaqi+Wang%2C+Conghui+He%2C+Ziwei+Liu%2C+et+al%2E+Mmbench%3A+Is+your+multi%2Dmodal+model+an+all%2Daround+player%3F+arXiv+preprint+) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/)
27. Xinyu Huang, Youcai Zhang, Jinyu Ma, Weiwei Tian, Rui Feng, Yuejie Zhang, Yaqian Li, Yandong Guo, and Lei Zhang. Tag2text: Guiding vision-language model via image tagging. arXiv preprint arXiv:2303.05657, 2023. — [arXiv:2303.05657](https://arxiv.org/abs/2303.05657) [find](https://scholar.google.com/scholar?q=Xinyu+Huang%2C+Youcai+Zhang%2C+Jinyu+Ma%2C+Weiwei+Tian%2C+Rui+Feng%2C+Yuejie+Zhang%2C+Yaqian+Li%2C+Yandong+Guo%2C+and+Lei+Zhang%2E+Tag2text%3A+Guiding+vision%2Dlanguage+model+via+image+tagging%2E+arXiv+preprint+arXiv%3A2303%2E05)
28. Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A generative region-to-text transformer for object understanding. arXiv preprint arXiv:2212.00280, 2022. — [arXiv:2212.00280](https://arxiv.org/abs/2212.00280) [find](https://scholar.google.com/scholar?q=Jialian+Wu%2C+Jianfeng+Wang%2C+Zhengyuan+Yang%2C+Zhe+Gan%2C+Zicheng+Liu%2C+Junsong+Yuan%2C+and+Lijuan+Wang%2E+Grit%3A+A+generative+region%2Dto%2Dtext+transformer+for+object+understanding%2E+arXiv+preprint+arXiv%3A2212%2E00280%2C)
29. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.02643, 2023. [find](https://scholar.google.com/scholar?q=Alexander+Kirillov%2C+Eric+Mintun%2C+Nikhila+Ravi%2C+Hanzi+Mao%2C+Chloe+Rolland%2C+Laura+Gustafson%2C+Tete+Xiao%2C+Spencer+Whitehead%2C+Alexander+C%2E+Berg%2C+Wan%2DYen+Lo%2C+Piotr+Doll%C3%A1r%2C+and+Ross+Girshick%2E+Segment+anything) [our card](https://md.research.elamur.ai/free/en/papers/kirillov-2023-segment-anything/)
30. Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In CVPR, 2021. [find](https://scholar.google.com/scholar?q=Pengchuan+Zhang%2C+Xiujun+Li%2C+Xiaowei+Hu%2C+Jianwei+Yang%2C+Lei+Zhang%2C+Lijuan+Wang%2C+Yejin+Choi%2C+and+Jianfeng+Gao%2E+Vinvl%3A+Revisiting+visual+representations+in+vision%2Dlanguage+models%2E+In+CVPR%2C+2021%2E)
31. [https://github.com/PaddlePaddle/PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR). Paddleocr. [↗](https://github.com/PaddlePaddle/PaddleOCR)
32. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+) [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
33. Guangzhi Wang, Yixiao Ge, Xiaohan Ding, Mohan Kankanhalli, and Ying Shan. What makes for good visual tokenizers for large language models? arXiv preprint arXiv:2305.12223, 2023. — [arXiv:2305.12223](https://arxiv.org/abs/2305.12223) [find](https://scholar.google.com/scholar?q=Guangzhi+Wang%2C+Yixiao+Ge%2C+Xiaohan+Ding%2C+Mohan+Kankanhalli%2C+and+Ying+Shan%2E+What+makes+for+good+visual+tokenizers+for+large+language+models%3F+arXiv+preprint+arXiv%3A2305%2E12223%2C+2023%2E+%E2%80%94+%5BarXiv%3A2305%2E12223%5D%28h)
34. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018. [find](https://scholar.google.com/scholar?q=Piyush+Sharma%2C+Nan+Ding%2C+Sebastian+Goodman%2C+and+Radu+Soricut%2E+Conceptual+captions%3A+A+cleaned%2C+hypernymed%2C+image+alt%2Dtext+dataset+for+automatic+image+captioning%2E+In+ACL%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/)
35. Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, 2017. [find](https://scholar.google.com/scholar?q=Raghav+Goyal%2C+Samira+Ebrahimi+Kahou%2C+Vincent+Michalski%2C+Joanna+Materzynska%2C+Susanne+Westphal%2C+Heuna+Kim%2C+Valentin+Haenel%2C+Ingo+Fruend%2C+Peter+Yianilos%2C+Moritz+Mueller%2DFreitag%2C+et+al%2E+The%22+something+som)
36. Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision. arXiv preprint arXiv:2006.13256, 2020. — [arXiv:2006.13256](https://arxiv.org/abs/2006.13256) [find](https://scholar.google.com/scholar?q=Dima+Damen%2C+Hazel+Doughty%2C+Giovanni+Maria+Farinella%2C+Antonino+Furnari%2C+Evangelos+Kazakos%2C+Jian+Ma%2C+Davide+Moltisanti%2C+Jonathan+Munro%2C+Toby+Perrett%2C+Will+Price%2C+et+al%2E+Rescaling+egocentric+vision%2E+arXi)
37. Hilde Kuehne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In CVPR, 2014. [find](https://scholar.google.com/scholar?q=Hilde+Kuehne%2C+Ali+Arslan%2C+and+Thomas+Serre%2E+The+language+of+actions%3A+Recovering+the+syntax+and+semantics+of+goal%2Ddirected+human+activities%2E+In+CVPR%2C+2014%2E)
38. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022. [find](https://scholar.google.com/scholar?q=Junnan+Li%2C+Dongxu+Li%2C+Caiming+Xiong%2C+and+Steven+Hoi%2E+Blip%3A+Bootstrapping+language%2Dimage+pre%2Dtraining+for+unified+vision%2Dlanguage+understanding+and+generation%2E+In+ICML%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/)
39. Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021. [find](https://scholar.google.com/scholar?q=Stephanie+Lin%2C+Jacob+Hilton%2C+and+Owain+Evans%2E+Truthfulqa%3A+Measuring+how+models+mimic+human+falsehoods%2E+arXiv+preprint+arXiv%3A2109%2E07958%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/)
40. Ao Zhang, Hao Fei, Yuan Yao, Wei Ji, Li Li, Zhiyuan Liu, and Tat-Seng Chua. Transfer visual prompt generator across llms. abs/23045.01278, 2023. [find](https://scholar.google.com/scholar?q=Ao+Zhang%2C+Hao+Fei%2C+Yuan+Yao%2C+Wei+Ji%2C+Li+Li%2C+Zhiyuan+Liu%2C+and+Tat%2DSeng+Chua%2E+Transfer+visual+prompt+generator+across+llms%2E+abs%2F23045%2E01278%2C+2023%2E)
41. ml_foundations. Openflamingo. [https://github.com/mlfoundations/open_flamingo](https://github.com/mlfoundations/open_flamingo), 2023. [↗](https://github.com/mlfoundations/open_flamingo)
42. Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010, 2023. [find](https://scholar.google.com/scholar?q=Peng+Gao%2C+Jiaming+Han%2C+Renrui+Zhang%2C+Ziyi+Lin%2C+Shijie+Geng%2C+Aojun+Zhou%2C+Wei+Zhang%2C+Pan+Lu%2C+Conghui+He%2C+Xiangyu+Yue%2C+Hongsheng+Li%2C+and+Yu+Qiao%2E+Llama%2Dadapter+v2%3A+Parameter%2Defficient+visual+instruction+)