---
title: "InternLM-XComposer-2.5 — paper by Zhang et al. (2024)"
description: "They present InternLM-XComposer-2.5 (IXC-2.5), built on InternLM2-7B and trained on 24K-token… 181 references, cited by 36 papers here."
authors: ["Pan Zhang", "Xiaoyi Dong", "Yuhang Zang", "Yuhang Cao", "Rui Qian", "Lin Chen", "Qipeng Guo", "Haodong Duan", "Bin Wang", "Linke Ouyang", "Songyang Zhang", "Wenwei Zhang", "Yining Li", "Yang Gao", "Peng Sun", "Xinyue Zhang", "Wei Li", "Jingwen Li", "Wenhai Wang", "Hang Yan", "Conghui He", "Xingcheng Zhang", "Kai Chen", "Jifeng Dai", "Yu Qiao", "Dahua Lin", "Jiaqi Wang"]
year: 2024
venue: "arXiv:2407.03320"
source: "https://arxiv.org/abs/2407.03320"
doi: "10.48550/arxiv.2407.03320"
arxiv: "2407.03320"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2024-internlm-xcomposer-2-5/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2024-internlm-xcomposer-2-5/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output

# InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output

**Authors:** [Pan Zhang](https://md.research.elamur.ai/free/en/authors/pan-zhang/), [Xiaoyi Dong](https://md.research.elamur.ai/free/en/authors/xiaoyi-dong/), [Yuhang Zang](https://md.research.elamur.ai/free/en/authors/yuhang-zang/), [Yuhang Cao](https://md.research.elamur.ai/free/en/authors/yuhang-cao/), [Rui Qian](https://md.research.elamur.ai/free/en/authors/rui-qian/), [Lin Chen](https://md.research.elamur.ai/free/en/authors/lin-chen/), [Qipeng Guo](https://md.research.elamur.ai/free/en/authors/qipeng-guo/), [Haodong Duan](https://md.research.elamur.ai/free/en/authors/haodong-duan/), [Bin Wang](https://md.research.elamur.ai/free/en/authors/bin-wang/), [Linke Ouyang](https://md.research.elamur.ai/free/en/authors/linke-ouyang/), [Songyang Zhang](https://md.research.elamur.ai/free/en/authors/songyang-zhang/), [Wenwei Zhang](https://md.research.elamur.ai/free/en/authors/wenwei-zhang/), [Yining Li](https://md.research.elamur.ai/free/en/authors/yining-li/), [Yang Gao](https://md.research.elamur.ai/free/en/authors/yang-gao/), Peng Sun, Xinyue Zhang, [Wei Li](https://md.research.elamur.ai/free/en/authors/wei-li/), Jingwen Li, [Wenhai Wang](https://md.research.elamur.ai/free/en/authors/wenhai-wang/), [Hang Yan](https://md.research.elamur.ai/free/en/authors/hang-yan/), [Conghui He](https://md.research.elamur.ai/free/en/authors/conghui-he/), Xingcheng Zhang, [Kai Chen](https://md.research.elamur.ai/free/en/authors/kai-chen/), [Jifeng Dai](https://md.research.elamur.ai/free/en/authors/jifeng-dai/), [Yu Qiao](https://md.research.elamur.ai/free/en/authors/yu-qiao/), [Dahua Lin](https://md.research.elamur.ai/free/en/authors/dahua-lin/) and [Jiaqi Wang](https://md.research.elamur.ai/free/en/authors/jiaqi-wang/) · **Year:** 2024 · **Venue:** arXiv:2407.03320 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2407.03320) · [PDF](https://arxiv.org/pdf/2407.03320) · [DOI 10.48550/arxiv.2407.03320](https://doi.org/10.48550/arxiv.2407.03320) · [arXiv 2407.03320](https://arxiv.org/abs/2407.03320) · [OpenAlex](https://openalex.org/W4400377049)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [36](https://md.research.elamur.ai/free/en/papers/zhang-2024-internlm-xcomposer-2-5/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Open-source vision-language models still lag behind proprietary systems like GPT-4V in versatility. The authors attribute this mainly to narrow training data and weak handling of long inputs and outputs. They present InternLM-XComposer-2.5 (IXC-2.5), built on InternLM2-7B and trained on 24K-token interleaved image-text sequences, which can be extended to 96K via RoPE extrapolation. Compared with IXC-2, it improves three comprehension skills:
- ultra-high-resolution images, using a 560×560 ViT plus the dynamic partitioning from IXC2-4KHD;
- fine-grained video, by tiling frames into one large composite image;
- multi-turn dialogue over several images.

Additional LoRA modules enable two generation tasks: producing webpage code from instructions and writing illustrated articles, refined with Chain-of-Thought and DPO. Across 28 benchmarks, IXC-2.5 achieves the best open-source result on 16 and matches or beats GPT-4V and Gemini Pro on many others. The work shows that a compact 7B model can deliver broad long-context multimodal abilities, and the model is publicly released.

## Key points
- Backbone is InternLM2-7B. The native training context is 24K tokens, extendable to 96K at inference through RoPE extrapolation.
- Evaluation covers 28 benchmarks (5 video, 9 structural high-resolution, 12 general VQA, MMDU, Design2Code), mostly run with OpenCompass VLMEvalKit.
- Video results include 69.1 on MVBench, 8.7% above VideoChat2-7B. On documents, it scores 90.9% on DocVQA, equal to InternVL-1.5, which is roughly four times larger.
- On MMDU (multi-image, multi-turn dialogue), it beats the previous open-source best by 13.8%. On MMStar, it reaches 59.9%, ahead of GPT-4V and Gemini-Pro.
- On Design2Code (screenshot-to-code), it averages 85.1 versus 84.8 for GPT-4V. The article-writing DPO stage uses LoRA with rank 256.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 333 works · 87,019 citations
- [Chinese University of Hong Kong](https://md.research.elamur.ai/free/en/universities/chinese-university-of-hong-kong/) — University, Hong Kong, China · 126 works · 14,073 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 117 works · 11,281 citations
- [Sensetime](https://md.research.elamur.ai/free/en/companies/sensetime/) — Company, Beijing, China · 12 works · 2,065 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works

## Cite

Cite as: Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen, Jifeng Dai, Yu Qiao, Dahua Lin and Jiaqi Wang. 2024. InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output. arXiv:2407.03320.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhang2024internlm,
  title = {{InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output}},
  author = {Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen, Jifeng Dai, Yu Qiao, Dahua Lin and Jiaqi Wang},
  year = {2024},
  doi = {10.48550/arxiv.2407.03320},
  eprint = {2407.03320},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.03320}
}
```

- [← previous Dual-Space Knowledge Distillation for Large Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2024-dual-space-knowledge-distillation-large/)
- [next → Learning to Edit: Aligning LLMs with Knowledge Editing](https://md.research.elamur.ai/free/en/papers/jiang-2024-learning-edit-aligning-llms-knowledge/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

181 references, 75 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019. 8 [find](https://scholar.google.com/scholar?q=Harsh+Agrawal%2C+Karan+Desai%2C+Yufei+Wang%2C+Xinlei+Chen%2C+Rishabh+Jain%2C+Mark+Johnson%2C+Dhruv+Batra%2C+Devi+Parikh%2C+Stefan+Lee%2C+and+Peter+Anderson%2E+Nocaps%3A+Novel+object+captioning+at+scale%2E+In+Proceedings+of+t)
2. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock… — [Flamingo: a Visual Language Model for Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
3. Anthropic. Claude 3 haiku: our fastest model yet, 2024. Available at: [https://www.anthropic.com/news/claude-3-haiku](https://www.anthropic.com/news/claude-3-haiku). 1, 8 [↗](https://www.anthropic.com/news/claude-3-haiku)
4. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2015. 8 [find](https://scholar.google.com/scholar?q=Stanislaw+Antol%2C+Aishwarya+Agrawal%2C+Jiasen+Lu%2C+Margaret+Mitchell%2C+Dhruv+Batra%2C+C%2E+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+VQA%3A+Visual+question+answering%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Confer)
5. Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source framework for training large… — [OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models](https://md.research.elamur.ai/free/en/papers/awadalla-2023-openflamingo-open-source-framework-training/)
6. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A frontier large vision-language model with versatile abilities. arXiv.org, 2023. 2, 9 [find](https://scholar.google.com/scholar?q=Jinze+Bai%2C+Shuai+Bai%2C+Shusheng+Yang%2C+Shijie+Wang%2C+Sinan+Tan%2C+Peng+Wang%2C+Junyang+Lin%2C+Chang+Zhou%2C+and+Jingren+Zhou%2E+Qwen%2DVL%3A+A+frontier+large+vision%2Dlanguage+model+with+versatile+abilities%2E+arXiv%2Eorg%2C+)
7. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. [Constitutional AI: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073, 2022. 6
8. Baichuan. [Baichuan 2: Open large-scale language models](https://md.research.elamur.ai/free/en/papers/yang-2023-baichuan-2-open-large-scale/). arXiv.org, 2023. 2
9. Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sağnak Taşırlar. Introducing our multimodal models, 2023. 2 [find](https://scholar.google.com/scholar?q=Rohan+Bavishi%2C+Erich+Elsen%2C+Curtis+Hawthorne%2C+Maxwell+Nye%2C+Augustus+Odena%2C+Arushi+Somani%2C+and+Sag%CC%86nak+Tas%CC%A7%C4%B1rlar%2E+Introducing+our+multimodal+models%2C+2023%2E+2)
10. Tony Beltramelli. pix2code: Generating code from a graphical user interface screenshot. In Proceedings of the ACM SIGCHI symposium on engineering interactive computing systems, 2018. 6 [find](https://scholar.google.com/scholar?q=Tony+Beltramelli%2E+pix2code%3A+Generating+code+from+a+graphical+user+interface+screenshot%2E+In+Proceedings+of+the+ACM+SIGCHI+symposium+on+engineering+interactive+computing+systems%2C+2018%2E+6)
11. Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In ICCV, 2019. 8 [find](https://scholar.google.com/scholar?q=Ali+Furkan+Biten%2C+Ruben+Tito%2C+Andres+Mafla%2C+Lluis+Gomez%2C+Marc%CC%A7al+Rusinol%2C+Ernest+Valveny%2C+CV+Jawahar%2C+and+Dimosthenis+Karatzas%2E+Scene+text+visual+question+answering%2E+In+ICCV%2C+2019%2E+8)
12. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in Neural Information Processing Systems (NeurIPS), 33:1877– 1901, 2020. 2
13. Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjia… — [InternLM2 Technical Report](https://md.research.elamur.ai/free/en/papers/cai-2024-internlm2-technical-report/)
14. Yuhang Cao, Pan Zhang, Xiaoyi Dong, Dahua Lin, and Jiaqi Wang. DualFocus: Integrating macro and micro perspectives in multi-modal large language models. arXiv preprint arXiv:2402.14767, 2024. 2 — [arXiv:2402.14767](https://arxiv.org/abs/2402.14767) [find](https://scholar.google.com/scholar?q=Yuhang+Cao%2C+Pan+Zhang%2C+Xiaoyi+Dong%2C+Dahua+Lin%2C+and+Jiaqi+Wang%2E+DualFocus%3A+Integrating+macro+and+micro+perspectives+in+multi%2Dmodal+large+language+models%2E+arXiv+preprint+arXiv%3A2402%2E14767%2C+2024%2E+2+%E2%80%94+%5BarX)
15. Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. ALLaVA harnessing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.[11684, 2024](https://md.research.elamur.ai/free/en/papers/chen-2024-allava-harnessing-gpt4v-synthesized-data/). 8
16. Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv.org, 2023. 8 — [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://md.research.elamur.ai/free/en/papers/chen-2023-shikra-unleashing-multimodal-llm-s/)
17. Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. [Sharegpt4v: Improving large multi-modal models with better captions](https://md.research.elamur.ai/free/en/papers/chen-2023-sharegpt4v-improving-large-multi-modal/). arXiv preprint arXiv:2311.12793, 2023. 2, 8
18. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. [Are we on the right way for evaluating large vision-language models?](https://md.research.elamur.ai/free/en/papers/chen-2024-right-way-evaluating-large-vision/) arXiv preprint arXiv:2403.20330, 2024. 2, 9
19. Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. ShareGPT4Video: Improving video understanding and generation with better captions. arXiv preprint arXiv:2406.04325, 2024. 8 — [arXiv:2406.04325](https://arxiv.org/abs/2406.04325) [find](https://scholar.google.com/scholar?q=Lin+Chen%2C+Xilin+Wei%2C+Jinsong+Li%2C+Xiaoyi+Dong%2C+Pan+Zhang%2C+Yuhang+Zang%2C+Zehui+Chen%2C+Haodong+Duan%2C+Bin+Lin%2C+Zhenyu+Tang%2C+et+al%2E+ShareGPT4Video%3A+Improving+video+understanding+and+generation+with+better+ca)
20. Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. [TabFact: A large-scale dataset for table-based fact verification](https://md.research.elamur.ai/free/en/papers/chen-2019-tabfact-large-scale-dataset-table/). In Proceedings of the International Conference on Learning Representations (ICLR), 2020. 2, 8, 9
21. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. [Microsoft coco captions: Data collection and evaluation server](https://md.research.elamur.ai/free/en/papers/chen-2015-microsoft-coco-captions-data-collection/), 2015. 8
22. Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, Siamak Shakeri, Mostafa Dehghani, Daniel Salz, Mario Lucic, Michael Tschannen, Arsha Nagrani, Hexiang Hu, Mandar Joshi, Bo Pang, Ceslee Montgomery, Paul… [find](https://scholar.google.com/scholar?q=Xi+Chen%2C+Josip+Djolonga%2C+Piotr+Padlewski%2C+Basil+Mustafa%2C+Soravit+Changpinyo%2C+Jialin+Wu%2C+Carlos+Riquelme+Ruiz%2C+Sebastian+Goodman%2C+Xiao+Wang%2C+Yi+Tay%2C+Siamak+Shakeri%2C+Mostafa+Dehghani%2C+Daniel+Salz%2C+Mario)
23. Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, Daniel Salz, Xi Xiong, Daniel Vlasic, Filip Pavetic, Keran Rong, Tianli Yu, Daniel Keysers, Xiaohua Zhai, and Radu Soricut. Pali-3 vision lan… [find](https://scholar.google.com/scholar?q=Xi+Chen%2C+Xiao+Wang%2C+Lucas+Beyer%2C+Alexander+Kolesnikov%2C+Jialin+Wu%2C+Paul+Voigtlaender%2C+Basil+Mustafa%2C+Sebastian+Goodman%2C+Ibrahim+Alabdulmohsin%2C+Piotr+Padlewski%2C+Daniel+Salz%2C+Xi+Xiong%2C+Daniel+Vlasic%2C+Fil)
24. Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo,… — [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://md.research.elamur.ai/free/en/papers/chen-2022-pali-jointly-scaled-multilingual-language/)
25. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. [Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks](https://md.research.elamur.ai/free/en/papers/chen-2023-internvl-scaling-up-vision/). arXiv preprint arXiv:2312.14238, 20…
26. Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng W… — [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://md.research.elamur.ai/free/en/papers/chen-2024-far-gpt-4v-closing/)
27. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 8 [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+and+Eric+P%2E+Xing%2E+Vicuna%3A+An+open%2Dsource+chatbot+i)
28. Chee Kheng Chng, Yuliang Liu, Yipeng Sun, Chun Chet Ng, Canjie Luo, Zihan Ni, ChuanMing Fang, Shuaitao Zhang, Junyu Han, Errui Ding, et al. Icdar2019 robust reading challenge on arbitrary-shaped text-rrc-art. In International Conference on Document Analysis and Recognition (ICDAR), 2019. 8 [find](https://scholar.google.com/scholar?q=Chee+Kheng+Chng%2C+Yuliang+Liu%2C+Yipeng+Sun%2C+Chun+Chet+Ng%2C+Canjie+Luo%2C+Zihan+Ni%2C+ChuanMing+Fang%2C+Shuaitao+Zhang%2C+Junyu+Han%2C+Errui+Ding%2C+et+al%2E+Icdar2019+robust+reading+challenge+on+arbitrary%2Dshaped+text%2D)
29. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv.org, 2022. 1, 2
30. OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. [https://github.com/open](https://github.com/open) - compass / opencompass, 2023. 9 [↗](https://github.com/open)
31. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. [Instructblip: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/), 2023. 1, 2
32. Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M.F. Moura, Devi Parikh, and Dhruv Batra. Visual Dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 8 [find](https://scholar.google.com/scholar?q=Abhishek+Das%2C+Satwik+Kottur%2C+Khushi+Gupta%2C+Avi+Singh%2C+Deshraj+Yadav%2C+Jose%CC%81+M%2EF%2E+Moura%2C+Devi+Parikh%2C+and+Dhruv+Batra%2E+Visual+Dialog%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern)
33. Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. Internlm-xcomposer2… — [InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model](https://md.research.elamur.ai/free/en/papers/dong-2024-internlm-xcomposer2-mastering-free/)
34. Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingcheng Zhang, Jifeng Dai, Yu Qiao, Dahua Lin, and Jiaqi Wang. Internlm… — [InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD](https://md.research.elamur.ai/free/en/papers/dong-2024-internlm-xcomposer2-4khd-pioneering/)
35. Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zen… [find](https://scholar.google.com/scholar?q=Danny+Driess%2C+Fei+Xia%2C+Mehdi+S%2E+M%2E+Sajjadi%2C+Corey+Lynch%2C+Aakanksha+Chowdhery%2C+Brian+Ichter%2C+Ayzaan+Wahid%2C+Jonathan+Tompson%2C+Quan+Vuong%2C+Tianhe+Yu%2C+Wenlong+Huang%2C+Yevgen+Chebotar%2C+Pierre+Sermanet%2C+Dani)
36. Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024. 6 [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2C+Winnie+Xu%2C+Niklas+Muennighoff%2C+Dan+Jurafsky%2C+and+Douwe+Kiela%2E+KTO%3A+Model+alignment+as+prospect+theoretic+optimization%2E+arXiv+preprint+arXiv%3A2402%2E01306%2C+2024%2E+6)
37. Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. ActivityNet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015. 2, 8 [find](https://scholar.google.com/scholar?q=Bernard+Ghanem+Fabian+Caba+Heilbron%2C+Victor+Escorcia+and+Juan+Carlos+Niebles%2E+ActivityNet%3A+A+large%2Dscale+video+benchmark+for+human+activity+understanding%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+)
38. Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. MMBench-Video: A long-form multi-shot benchmark for holistic video understanding. arXiv preprint arXiv:2406.14515, 2024. 2, 9 — [arXiv:2406.14515](https://arxiv.org/abs/2406.14515) [find](https://scholar.google.com/scholar?q=Xinyu+Fang%2C+Kangrui+Mao%2C+Haodong+Duan%2C+Xiangyu+Zhao%2C+Yining+Li%2C+Dahua+Lin%2C+and+Kai+Chen%2E+MMBench%2DVideo%3A+A+long%2Dform+multi%2Dshot+benchmark+for+holistic+video+understanding%2E+arXiv+preprint+arXiv%3A2406%2E145)
39. Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding, 2024. 2 [find](https://scholar.google.com/scholar?q=Xinyu+Fang%2C+Kangrui+Mao%2C+Haodong+Duan%2C+Xiangyu+Zhao%2C+Yining+Li%2C+Dahua+Lin%2C+and+Kai+Chen%2E+Mmbench%2Dvideo%3A+A+long%2Dform+multi%2Dshot+benchmark+for+holistic+video+understanding%2C+2024%2E+2)
40. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. [Mme: A comprehensive evaluation benchmark for multimodal large language models](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/). arXiv preprint arXiv:2306.13394, 2023. 2, 9
41. Chaoyou Fu, Renrui Zhang, Zihan Wang, Yubo Huang, Zhengye Zhang, Longtian Qiu, Gaoxiang Ye, Yunhang Shen, Mengdan Zhang, Peixian Chen, Sirui Zhao, Shaohui Lin, Deqiang Jiang, Di Yin, Peng Gao, Ke Li, Hongsheng Li, and Xing Sun. A challenger to gpt-4v? early explorations of gemini in visual expertise… — [arXiv:2312.12436](https://arxiv.org/abs/2312.12436) [find](https://scholar.google.com/scholar?q=Chaoyou+Fu%2C+Renrui+Zhang%2C+Zihan+Wang%2C+Yubo+Huang%2C+Zhengye+Zhang%2C+Longtian+Qiu%2C+Gaoxiang+Ye%2C+Yunhang+Shen%2C+Mengdan+Zhang%2C+Peixian+Chen%2C+Sirui+Zhao%2C+Shaohui+Lin%2C+Deqiang+Jiang%2C+Di+Yin%2C+Peng+Gao%2C+Ke+Li%2C+)
42. Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. [Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis](https://md.research.elamur.ai/free/en/papers/fu-2024-video-mme-first-ever-comprehensive/). arXiv preprint arXiv:2405.21075, 2024. 2, 9
43. Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. [ChatGLM: A family of large language models from glm-130b to glm-4 all tools](https://md.research.elamur.ai/free/en/papers/glm-2024-chatglm-family-large-language-models/). arXiv preprint arXiv:2406.12793, 2024. 9
44. Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: An advanced diagnostic suite for entangled language hallucination & visual illusion in large vision-language models, 2023. 2… [find](https://scholar.google.com/scholar?q=Tianrui+Guan%2C+Fuxiao+Liu%2C+Xiyang+Wu%2C+Ruiqi+Xian%2C+Zongxia+Li%2C+Xiaoyu+Liu%2C+Xijun+Wang%2C+Lichang+Chen%2C+Furong+Huang%2C+Yaser+Yacoob%2C+Dinesh+Manocha%2C+and+Tianyi+Zhou%2E+Hallusionbench%3A+An+advanced+diagnostic+s)
45. Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. arXiv preprint arXiv:2404.05726, 2024. 2 [find](https://scholar.google.com/scholar?q=Bo+He%2C+Hengduo+Li%2C+Young+Kyun+Jang%2C+Menglin+Jia%2C+Xuefei+Cao%2C+Ashish+Shah%2C+Abhinav+Shrivastava%2C+and+Ser%2DNam+Lim%2E+Ma%2Dlmm%3A+Memory%2Daugmented+large+multimodal+model+for+long%2Dterm+video+understanding%2E+arXiv)
46. Conghui He, Zhenjiang Jin, Chaoxi Xu, Jiantao Qiu, Bin Wang, Wei Li, Hang Yan, Jiaqi Wang, and Da Lin. Wanjuan: A comprehensive multimodal dataset for advancing english and chinese large models. ArXiv, abs/2308.10755, 2023. 8 — [arXiv:2308.10755](https://arxiv.org/abs/2308.10755) [find](https://scholar.google.com/scholar?q=Conghui+He%2C+Zhenjiang+Jin%2C+Chaoxi+Xu%2C+Jiantao+Qiu%2C+Bin+Wang%2C+Wei+Li%2C+Hang+Yan%2C+Jiaqi+Wang%2C+and+Da+Lin%2E+Wanjuan%3A+A+comprehensive+multimodal+dataset+for+advancing+english+and+chinese+large+models%2E+ArXiv)
47. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. [Cogagent: A visual language model for gui agents](https://md.research.elamur.ai/free/en/papers/hong-2023-cogagent-visual-language-model/). arXiv preprint arXiv:2312.08914, 2023. 2
48. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. [CogAgent: A visual language model for gui agents](https://md.research.elamur.ai/free/en/papers/hong-2023-cogagent-visual-language-model/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 10
49. Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. [mplug-docowl 1.5: Unified structure learning for ocr-free document understanding](https://md.research.elamur.ai/free/en/papers/hu-2024-mplug-docowl-1-5-unified/). arXiv preprint arXiv:2403.12895, 2024. 2
50. Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. [mPLUG-DocOwl 1.5: Unified structure learning for ocr-free document understanding](https://md.research.elamur.ai/free/en/papers/hu-2024-mplug-docowl-1-5-unified/). arXiv preprint arXiv:2403.12895, 2024. 9
51. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In Proceedings of the International Conference on Learning Representations (ICLR), 2022. 8
52. Suyuan Huang, Haoxin Zhang, Yan Gao, Yao Hu, and Zengchang Qin. From image to video, what do we need in multimodal llms? arXiv preprint arXiv:2404.11865, 2024. 2 — [arXiv:2404.11865](https://arxiv.org/abs/2404.11865) [find](https://scholar.google.com/scholar?q=Suyuan+Huang%2C+Haoxin+Zhang%2C+Yan+Gao%2C+Yao+Hu%2C+and+Zengchang+Qin%2E+From+image+to+video%2C+what+do+we+need+in+multimodal+llms%3F+arXiv+preprint+arXiv%3A2404%2E11865%2C+2024%2E+2+%E2%80%94+%5BarXiv%3A2404%2E11865%5D%28https%3A%2F%2Farxiv%2Eorg)
53. Drew A Hudson and Christopher D Manning. [Gqa: A new dataset for real-world visual reasoning and compositional question answering](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/). Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 8
54. Md Mohaiminul Islam, Ngan Ho, Xitong Yang, Tushar Nagarajan, Lorenzo Torresani, and Gedas Bertasius. Video recap: Recursive captioning of hour-long videos. arXiv preprint arXiv:2402.13250, 2024. 2 — [arXiv:2402.13250](https://arxiv.org/abs/2402.13250) [find](https://scholar.google.com/scholar?q=Md+Mohaiminul+Islam%2C+Ngan+Ho%2C+Xitong+Yang%2C+Tushar+Nagarajan%2C+Lorenzo+Torresani%2C+and+Gedas+Bertasius%2E+Video+recap%3A+Recursive+captioning+of+hour%2Dlong+videos%2E+arXiv+preprint+arXiv%3A2402%2E13250%2C+2024%2E+2+%E2%80%94+%5B)
55. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroi… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+Le%CC%81lio+Renard+La)
56. Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning, 2024. 2 [find](https://scholar.google.com/scholar?q=Dongfu+Jiang%2C+Xuan+He%2C+Huaye+Zeng%2C+Cong+Wei%2C+Max+Ku%2C+Qian+Liu%2C+and+Wenhu+Chen%2E+Mantis%3A+Interleaved+multi%2Dimage+instruction+tuning%2C+2024%2E+2)
57. Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. arXiv preprint arXiv:2311.08046, 2023. 2 — [arXiv:2311.08046](https://arxiv.org/abs/2311.08046) [find](https://scholar.google.com/scholar?q=Peng+Jin%2C+Ryuichi+Takanobu%2C+Caiwan+Zhang%2C+Xiaochun+Cao%2C+and+Li+Yuan%2E+Chat%2Dunivi%3A+Unified+visual+representation+empowers+large+language+models+with+image+and+video+understanding%2E+arXiv+preprint+arXiv%3A2)
58. Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. DVQA: Understanding data visualizations via question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 8 [find](https://scholar.google.com/scholar?q=Kushal+Kafle%2C+Brian+Price%2C+Scott+Cohen%2C+and+Christopher+Kanan%2E+DVQA%3A+Understanding+data+visualizations+via+question+answering%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+)
59. Kumara Kahatapitiya, Kanchana Ranasinghe, Jongwoo Park, and Michael S Ryoo. Language repository for long video understanding. arXiv preprint arXiv:2403.14622, 2024. 2 [find](https://scholar.google.com/scholar?q=Kumara+Kahatapitiya%2C+Kanchana+Ranasinghe%2C+Jongwoo+Park%2C+and+Michael+S+Ryoo%2E+Language+repository+for+long+video+understanding%2E+arXiv+preprint+arXiv%3A2403%2E14622%2C+2024%2E+2)
60. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361, 2020. 2
61. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. [A diagram is worth a dozen images](https://md.research.elamur.ai/free/en/papers/kembhavi-2016-diagram-worth-dozen-images/). In Proceedings of the European Conference on Computer Vision (ECCV), 2016. 2, 8, 9
62. Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 20… [find](https://scholar.google.com/scholar?q=Aniruddha+Kembhavi%2C+Minjoon+Seo%2C+Dustin+Schwenk%2C+Jonghyun+Choi%2C+Ali+Farhadi%2C+and+Hannaneh+Hajishirzi%2E+Are+you+smarter+than+a+sixth+grader%3F+textbook+question+answering+for+multimodal+machine+comprehens)
63. Wonkyun Kim, Changin Choi, Wonseok Lee, and Wonjong Rhee. An image grid can be worth a video: Zero-shot video question answering using a vlm, 2024. 6 [find](https://scholar.google.com/scholar?q=Wonkyun+Kim%2C+Changin+Choi%2C+Wonseok+Lee%2C+and+Wonjong+Rhee%2E+An+image+grid+can+be+worth+a+video%3A+Zero%2Dshot+video+question+answering+using+a+vlm%2C+2024%2E+6)
64. Hugo Laurençon, Léo Tronchon, and Victor Sanh. Unlocking the conversion of web screenshots into html code with the websight dataset. arXiv preprint arXiv:2403.09029, 2024. 6, 7, 10 — [arXiv:2403.09029](https://arxiv.org/abs/2403.09029) [find](https://scholar.google.com/scholar?q=Hugo+Laurenc%CC%A7on%2C+Le%CC%81o+Tronchon%2C+and+Victor+Sanh%2E+Unlocking+the+conversion+of+web+screenshots+into+html+code+with+the+websight+dataset%2E+arXiv+preprint+arXiv%3A2403%2E09029%2C+2024%2E+6%2C+7%2C+10+%E2%80%94+%5BarXiv%3A2403%2E090)
65. Paul Lerner, Olivier Ferret, Camille Guinaudeau, Hervé Le Borgne, Romaric Besançon, José G Moreno, and Jesús Lovón Melgarejo. Viquae, a dataset for knowledge-based visual question answering about named entities. In Proceedings of the 45th International ACM SIGIR Conference on Research and Devel… [find](https://scholar.google.com/scholar?q=Paul+Lerner%2C+Olivier+Ferret%2C+Camille+Guinaudeau%2C+Herve%CC%81+Le+Borgne%2C+Romaric+Besanc%CC%A7on%2C+Jose%CC%81+G+Moreno%2C+and+Jesu%CC%81s+Lovo%CC%81n+Melgarejo%2E+Viquae%2C+a+dataset+for+knowledge%2Dbased+visual+question+answering+about)
66. Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. [Seed-bench: Benchmarking multimodal llms with generative comprehension](https://md.research.elamur.ai/free/en/papers/li-2023-seed-bench-benchmarking-multimodal-llms/), 2023. 2, 9
67. Bo Li, Peiyuan Zhang, Jingkang Yang, Yuanhan Zhang, Fanyi Pu, and Ziwei Liu. Otterhd: A high-resolution multimodality model, 2023. 2 [find](https://scholar.google.com/scholar?q=Bo+Li%2C+Peiyuan+Zhang%2C+Jingkang+Yang%2C+Yuanhan+Zhang%2C+Fanyi+Pu%2C+and+Ziwei+Liu%2E+Otterhd%3A+A+high%2Dresolution+multimodality+model%2C+2023%2E+2)
68. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. [Otter: A multi-modal model with in-context instruction tuning](https://md.research.elamur.ai/free/en/papers/li-2023-otter-multi-modal-model-context/). arXiv.org, 2023. 2
69. KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. [Videochat: Chat-centric video understanding](https://md.research.elamur.ai/free/en/papers/li-2023-videochat-chat-centric-video-understanding/). arXiv preprint arXiv:2305.06355, 2023. 2
70. Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. [Mvbench: A comprehensive multi-modal video understanding benchmark](https://md.research.elamur.ai/free/en/papers/li-2023-mvbench-comprehensive-multi-modal-video/). arXiv preprint arXiv:2311.17005, 2023. 2
71. Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. [Mvbench: A comprehensive multi-modal video understanding benchmark](https://md.research.elamur.ai/free/en/papers/li-2023-mvbench-comprehensive-multi-modal-video/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2, 9
72. Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. arXiv preprint arXiv:2312.10665, 2023. 6 — [arXiv:2312.10665](https://arxiv.org/abs/2312.10665) [find](https://scholar.google.com/scholar?q=Lei+Li%2C+Zhihui+Xie%2C+Mukai+Li%2C+Shunian+Chen%2C+Peiyi+Wang%2C+Liang+Chen%2C+Yazheng+Yang%2C+Benyou+Wang%2C+and+Lingpeng+Kong%2E+Silkie%3A+Preference+distillation+for+large+visual+language+models%2E+arXiv+preprint+arXiv)
73. Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.17043, 2023. 2 [find](https://scholar.google.com/scholar?q=Yanwei+Li%2C+Chengyao+Wang%2C+and+Jiaya+Jia%2E+Llama%2Dvid%3A+An+image+is+worth+2+tokens+in+large+language+models%2E+arXiv+preprint+arXiv%3A2311%2E17043%2C+2023%2E+2)
74. Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-Gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814, 2024. 2 [find](https://scholar.google.com/scholar?q=Yanwei+Li%2C+Yuechen+Zhang%2C+Chengyao+Wang%2C+Zhisheng+Zhong%2C+Yixin+Chen%2C+Ruihang+Chu%2C+Shaoteng+Liu%2C+and+Jiaya+Jia%2E+Mini%2DGemini%3A+Mining+the+potential+of+multi%2Dmodality+vision+language+models%2E+arXiv+preprin)
75. Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan L Yuille. Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 8 [find](https://scholar.google.com/scholar?q=Zhuowan+Li%2C+Xingrui+Wang%2C+Elias+Stengel%2DEskin%2C+Adam+Kortylewski%2C+Wufei+Ma%2C+Benjamin+Van+Durme%2C+and+Alan+L+Yuille%2E+Super%2Dclevr%3A+A+virtual+benchmark+to+diagnose+domain+robustness+in+visual+reasoning%2E+In)
76. Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. arXiv preprint arXiv:2311.06607, 2023. 2 [find](https://scholar.google.com/scholar?q=Zhang+Li%2C+Biao+Yang%2C+Qiang+Liu%2C+Zhiyin+Ma%2C+Shuo+Zhang%2C+Jingxu+Yang%2C+Yabo+Sun%2C+Yuliang+Liu%2C+and+Xiang+Bai%2E+Monkey%3A+Image+resolution+and+text+label+are+important+things+for+large+multi%2Dmodal+models%2E+arX)
77. Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. [Video-llava: Learning united visual representation by alignment before projection](https://md.research.elamur.ai/free/en/papers/lin-2023-video-llava-learning-united-visual/). arXiv preprint arXiv:2311.10122, 2023. 2
78. Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. [Vila: On pre-training for visual language models](https://md.research.elamur.ai/free/en/papers/lin-2023-vila-pre-training-visual-language/), 2024. 2, 9
79. Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575, 2023. 2 — [arXiv:2311.07575](https://arxiv.org/abs/2311.07575) [find](https://scholar.google.com/scholar?q=Ziyi+Lin%2C+Chris+Liu%2C+Renrui+Zhang%2C+Peng+Gao%2C+Longtian+Qiu%2C+Han+Xiao%2C+Han+Qiu%2C+Chen+Lin%2C+Wenqi+Shao%2C+Keqin+Chen%2C+et+al%2E+Sphinx%3A+The+joint+mixing+of+weights%2C+tasks%2C+and+visual+embeddings+for+multi%2Dmodal)
80. Adam Dahlgren Lindström and Savitha Sam Abraham. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. arXiv preprint arXiv:2208.05358, 2022. 8 — [arXiv:2208.05358](https://arxiv.org/abs/2208.05358) [find](https://scholar.google.com/scholar?q=Adam+Dahlgren+Lindstro%CC%88m+and+Savitha+Sam+Abraham%2E+Clevr%2Dmath%3A+A+dataset+for+compositional+language%2C+visual+and+mathematical+reasoning%2E+arXiv+preprint+arXiv%3A2208%2E05358%2C+2022%2E+8+%E2%80%94+%5BarXiv%3A2208%2E05358%5D%28htt)
81. Fangyu Liu, Guy Edward Toh Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics (TACL), 2023. 8 [find](https://scholar.google.com/scholar?q=Fangyu+Liu%2C+Guy+Edward+Toh+Emerson%2C+and+Nigel+Collier%2E+Visual+spatial+reasoning%2E+Transactions+of+the+Association+for+Computational+Linguistics+%28TACL%29%2C+2023%2E+8)
82. Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu. [Mmc: Advancing multimodal chart understanding with large-scale instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2024-mmc-advancing-multimodal-chart-understanding/). arXiv preprint arXiv:2311.10774, 2023. 8
83. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/). arXiv preprint arXiv:2310.03744, 2023. 9
84. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). arXiv.org, 2023. 1, 8
85. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024. 2 [find](https://scholar.google.com/scholar?q=Haotian+Liu%2C+Chunyuan+Li%2C+Yuheng+Li%2C+Bo+Li%2C+Yuanhan+Zhang%2C+Sheng+Shen%2C+and+Yong+Jae+Lee%2E+Llava%2Dnext%3A+Improved+reasoning%2C+ocr%2C+and+world+knowledge%2C+2024%2E+2)
86. Shuo Liu, Kaining Ying, Hao Zhang, Yue Yang, Yuqi Lin, Tianle Zhang, Chuanhao Li, Yu Qiao, Ping Luo, Wenqi Shao, et al. Convbench: A multi-turn conversation evaluation benchmark with hierarchical capability for large vision-language models. arXiv preprint arXiv:2403.20194, 2024. 2 — [arXiv:2403.20194](https://arxiv.org/abs/2403.20194) [find](https://scholar.google.com/scholar?q=Shuo+Liu%2C+Kaining+Ying%2C+Hao+Zhang%2C+Yue+Yang%2C+Yuqi+Lin%2C+Tianle+Zhang%2C+Chuanhao+Li%2C+Yu+Qiao%2C+Ping+Luo%2C+Wenqi+Shao%2C+et+al%2E+Convbench%3A+A+multi%2Dturn+conversation+evaluation+benchmark+with+hierarchical+capa)
87. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhnag, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. [Mmbench: Is your multi-modal model an all-around player?](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/) arXiv:2307.06281, 2023. 2, 9
88. Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Temp-Compass: Do video llms really understand videos? arXiv preprint arXiv:2403.00476, 2024. 2, 9 — [arXiv:2403.00476](https://arxiv.org/abs/2403.00476) [find](https://scholar.google.com/scholar?q=Yuanxin+Liu%2C+Shicheng+Li%2C+Yi+Liu%2C+Yuxiang+Wang%2C+Shuhuai+Ren%2C+Lei+Li%2C+Sishuo+Chen%2C+Xu+Sun%2C+and+Lu+Hou%2E+Temp%2DCompass%3A+Do+video+llms+really+understand+videos%3F+arXiv+preprint+arXiv%3A2403%2E00476%2C+2024%2E+2%2C+9+)
89. Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models, 2024. 2, 9 [find](https://scholar.google.com/scholar?q=Yuliang+Liu%2C+Zhang+Li%2C+Biao+Yang%2C+Chunyuan+Li%2C+Xucheng+Yin%2C+Cheng+lin+Liu%2C+Lianwen+Jin%2C+and+Xiang+Bai%2E+On+the+hidden+mystery+of+ocr+in+large+multimodal+models%2C+2024%2E+2%2C+9)
90. Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473, 2024. 2 — [arXiv:2403.04473](https://arxiv.org/abs/2403.04473) [find](https://scholar.google.com/scholar?q=Yuliang+Liu%2C+Biao+Yang%2C+Qiang+Liu%2C+Zhang+Li%2C+Zhiyin+Ma%2C+Shuo+Zhang%2C+and+Xiang+Bai%2E+Textmonkey%3A+An+ocr%2Dfree+large+multimodal+model+for+understanding+document%2E+arXiv+preprint+arXiv%3A2403%2E04473%2C+2024%2E+2+%E2%80%94)
91. Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s, 2022. 2 [find](https://scholar.google.com/scholar?q=Zhuang+Liu%2C+Hanzi+Mao%2C+Chao%2DYuan+Wu%2C+Christoph+Feichtenhofer%2C+Trevor+Darrell%2C+and+Saining+Xie%2E+A+convnet+for+the+2020s%2C+2022%2E+2)
92. Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi Dong, Pan Zhang, Zijian Liang, Yuanjun Xiong, Yu Qiao, Dahua Lin, et al. MMDU: A multi-turn multi-image dialog understanding benchmark and instruction-tuning dataset for lvlms. arXiv preprint arXiv:2406.11833, 2024. 2, 7, 8, 9, 10 — [arXiv:2406.11833](https://arxiv.org/abs/2406.11833) [find](https://scholar.google.com/scholar?q=Ziyu+Liu%2C+Tao+Chu%2C+Yuhang+Zang%2C+Xilin+Wei%2C+Xiaoyi+Dong%2C+Pan+Zhang%2C+Zijian+Liang%2C+Yuanjun+Xiong%2C+Yu+Qiao%2C+Dahua+Lin%2C+et+al%2E+MMDU%3A+A+multi%2Dturn+multi%2Dimage+dialog+understanding+benchmark+and+instruction)
93. Ziyu Liu, Zeyi Sun, Yuhang Zang, Wei Li, Pan Zhang, Xiaoyi Dong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. RAR: Retrieving and ranking augmented mllms for visual recognition. arXiv preprint arXiv:2403.13805, 2024. 2 — [arXiv:2403.13805](https://arxiv.org/abs/2403.13805) [find](https://scholar.google.com/scholar?q=Ziyu+Liu%2C+Zeyi+Sun%2C+Yuhang+Zang%2C+Wei+Li%2C+Pan+Zhang%2C+Xiaoyi+Dong%2C+Yuanjun+Xiong%2C+Dahua+Lin%2C+and+Jiaqi+Wang%2E+RAR%3A+Retrieving+and+ranking+augmented+mllms+for+visual+recognition%2E+arXiv+preprint+arXiv%3A2403)
94. LocalLLaMA. Dynamically scaled rope further increases performance of long context llama with zero fine-tuning, 2023. 2 [find](https://scholar.google.com/scholar?q=LocalLLaMA%2E+Dynamically+scaled+rope+further+increases+performance+of+long+context+llama+with+zero+fine%2Dtuning%2C+2023%2E+2)
95. Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173, 2024. 7 [find](https://scholar.google.com/scholar?q=Anton+Lozhkov%2C+Raymond+Li%2C+Loubna+Ben+Allal%2C+Federico+Cassano%2C+Joel+Lamy%2DPoirier%2C+Nouamane+Tazi%2C+Ao+Tang%2C+Dmytro+Pykhtar%2C+Jiawei+Liu%2C+Yuxiang+Wei%2C+et+al%2E+Starcoder+2+and+the+stack+v2%3A+The+next+generat)
96. Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. [Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning](https://md.research.elamur.ai/free/en/papers/lu-2021-inter-gps-interpretable-geometry-problem/). In The 59th Annual Meeting of the Association for Computational Linguistics (ACL), 2021. 8
97. Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2110.13214, 2021. 8 [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Liang+Qiu%2C+Jiaqi+Chen%2C+Tony+Xia%2C+Yizhou+Zhao%2C+Wei+Zhang%2C+Zhou+Yu%2C+Xiaodan+Liang%2C+and+Song%2DChun+Zhu%2E+Iconqa%3A+A+new+benchmark+for+abstract+diagram+understanding+and+visual+language+reasoning%2E+ar)
98. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 8 [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Swaroop+Mishra%2C+Tanglin+Xia%2C+Liang+Qiu%2C+Kai%2DWei+Chang%2C+Song%2DChun+Zhu%2C+Oyvind+Tafjord%2C+Peter+Clark%2C+and+Ashwin+Kalyan%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+science+ques)
99. Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. [Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning](https://md.research.elamur.ai/free/en/papers/lu-2022-dynamic-prompt-learning-policy-gradient/). arXiv preprint arXiv:2209.14610, 2022. 8
100. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. [Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). In International Conference on Learning Representations (ICLR), 2024. 2, 9
101. Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07207, 2023. 2 — [arXiv:2306.07207](https://arxiv.org/abs/2306.07207) [find](https://scholar.google.com/scholar?q=Ruipu+Luo%2C+Ziwang+Zhao%2C+Min+Yang%2C+Junwei+Dong%2C+Minghui+Qiu%2C+Pengcheng+Lu%2C+Tao+Wang%2C+and+Zhongyu+Wei%2E+Valley%3A+Video+assistant+with+large+language+model+enhanced+ability%2E+arXiv+preprint+arXiv%3A2306%2E07207)
102. Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, and Furu Wei. Kosmos-2.5: A multimodal literate model, 2023. 2 [find](https://scholar.google.com/scholar?q=Tengchao+Lv%2C+Yupan+Huang%2C+Jingye+Chen%2C+Lei+Cui%2C+Shuming+Ma%2C+Yaoyao+Chang%2C+Shaohan+Huang%2C+Wenhui+Wang%2C+Li+Dong%2C+Weiyao+Luo%2C+Shaoxiang+Wu%2C+Guoxin+Wang%2C+Cha+Zhang%2C+and+Furu+Wei%2E+Kosmos%2D2%2E5%3A+A+multimodal+)
103. Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Jiang Yu-Gang Pan, Liangming, Jiaqi Wang, Yixin Cao, and Aixin Sun. MMLongBench-Doc: Benchmarking long-context document understanding with visualizations. arXiv preprint arXiv:… [find](https://scholar.google.com/scholar?q=Yubo+Ma%2C+Yuhang+Zang%2C+Liangyu+Chen%2C+Meiqi+Chen%2C+Yizhu+Jiao%2C+Xinze+Li%2C+Xinyuan+Lu%2C+Ziyu+Liu%2C+Yan+Ma%2C+Xiaoyi+Dong%2C+Pan+Zhang%2C+Jiang+Yu%2DGang+Pan%2C+Liangming%2C+Jiaqi+Wang%2C+Yixin+Cao%2C+and+Aixin+Sun%2E+MMLongBe)
104. Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. [Video-chatgpt: Towards detailed video understanding via large vision and language models](https://md.research.elamur.ai/free/en/papers/maaz-2023-video-chatgpt-towards-detailed-video/). arXiv preprint arXiv:2306.05424, 2023. 2
105. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3195–3204, 2019. 8 [find](https://scholar.google.com/scholar?q=Kenneth+Marino%2C+Mohammad+Rastegari%2C+Ali+Farhadi%2C+and+Roozbeh+Mottaghi%2E+Ok%2Dvqa%3A+A+visual+question+answering+benchmark+requiring+external+knowledge%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer)
106. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. [Chartqa: A benchmark for question answering about charts with visual and logical reasoning](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/). arXiv preprint arXiv:2203.10244, 2022. 2, 8, 9
107. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. [Docvqa: A dataset for vqa on document images](https://md.research.elamur.ai/free/en/papers/mathew-2020-docvqa-dataset-vqa-document-images/). In Proc. of the IEEE Winter Conference on Applications of Computer Vision (WACV), 2021. 8, 9
108. Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proc. of the IEEE Winter Conference on Applications of Computer Vision (WACV), 2022. 2, 8, 9 [find](https://scholar.google.com/scholar?q=Minesh+Mathew%2C+Viraj+Bagal%2C+Rube%CC%80n+Tito%2C+Dimosthenis+Karatzas%2C+Ernest+Valveny%2C+and+CV+Jawahar%2E+Infographicvqa%2E+In+Proc%2E+of+the+IEEE+Winter+Conference+on+Applications+of+Computer+Vision+%28WACV%29%2C+2022%2E+2)
109. Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. OCR-VQA: Visual question answering by reading text in images. In International Conference on Document Analysis and Recognition (ICDAR), 2019. 8 [find](https://scholar.google.com/scholar?q=Anand+Mishra%2C+Shashank+Shekhar%2C+Ajeet+Kumar+Singh%2C+and+Anirban+Chakraborty%2E+OCR%2DVQA%3A+Visual+question+answering+by+reading+text+in+images%2E+In+International+Conference+on+Document+Analysis+and+Recogniti)
110. Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models. arXiv preprint arXiv:2311.16103, 2023. 2 — [arXiv:2311.16103](https://arxiv.org/abs/2311.16103) [find](https://scholar.google.com/scholar?q=Munan+Ning%2C+Bin+Zhu%2C+Yujia+Xie%2C+Bin+Lin%2C+Jiaxi+Cui%2C+Lu+Yuan%2C+Dongdong+Chen%2C+and+Li+Yuan%2E+Video%2Dbench%3A+A+comprehensive+benchmark+and+toolkit+for+evaluating+video%2Dbased+large+language+models%2E+arXiv+prep)
111. OpenAI. Chatgpt. [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt), 2022. 1, 2 [↗](https://openai.com/blog/chatgpt)
112. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023. 1, 2, 9, 10
113. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabri… — [DINOv2: Learning Robust Visual Features without Supervision](https://md.research.elamur.ai/free/en/papers/oquab-2023-dinov2-learning-robust-visual-features/)
114. Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. Im2text: Describing images using 1 million captioned photographs. In Advances in Neural Information Processing Systems (NeurIPS), 2011. 8 [find](https://scholar.google.com/scholar?q=Vicente+Ordonez%2C+Girish+Kulkarni%2C+and+Tamara+L%2E+Berg%2E+Im2text%3A+Describing+images+using+1+million+captioned+photographs%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2C+2011%2E+8)
115. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems (NeurIPS), 2022. 2, 6
116. Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228, 2024. 6 [find](https://scholar.google.com/scholar?q=Arka+Pal%2C+Deep+Karkhanis%2C+Samuel+Dooley%2C+Manley+Roberts%2C+Siddartha+Naidu%2C+and+Colin+White%2E+Smaug%3A+Fixing+failure+modes+of+preference+optimisation+with+dpo%2Dpositive%2E+arXiv+preprint+arXiv%3A2402%2E13228%2C+20)
117. Panupong Pasupat and Percy Liang. [Compositional semantic parsing on semi-structured tables](https://md.research.elamur.ai/free/en/papers/pasupat-2015-compositional-semantic-parsing-semi-structured/). In The Annual Meeting of the Association for Computational Linguistics (ACL), 2015. 2, 8, 9
118. Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. [Kosmos-2: Grounding multimodal large language models to the world](https://md.research.elamur.ai/free/en/papers/peng-2023-kosmos-2-grounding-multimodal-large/). arXiv.org, 2023. 2
119. Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, and Tong Zhang. Strengthening multimodal large language model with bootstrapped preference optimization. arXiv preprint arXiv:2403.08730, 2024. 6 — [arXiv:2403.08730](https://arxiv.org/abs/2403.08730) [find](https://scholar.google.com/scholar?q=Renjie+Pi%2C+Tianyang+Han%2C+Wei+Xiong%2C+Jipeng+Zhang%2C+Runtao+Liu%2C+Rui+Pan%2C+and+Tong+Zhang%2E+Strengthening+multimodal+large+language+model+with+bootstrapped+preference+optimization%2E+arXiv+preprint+arXiv%3A240)
120. Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models, 2024. 2 [find](https://scholar.google.com/scholar?q=Rui+Qian%2C+Xiaoyi+Dong%2C+Pan+Zhang%2C+Yuhang+Zang%2C+Shuangrui+Ding%2C+Dahua+Lin%2C+and+Jiaqi+Wang%2E+Streaming+long+video+understanding+with+large+language+models%2C+2024%2E+2)
121. Qwen. Introducing Qwen-7B: Open foundation and human-aligned models (of the state-of-the-arts), 2023. 1, 2 [find](https://scholar.google.com/scholar?q=Qwen%2E+Introducing+Qwen%2D7B%3A+Open+foundation+and+human%2Daligned+models+%28of+the+state%2Dof%2Dthe%2Darts%29%2C+2023%2E+1%2C+2)
122. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In Proceedings of the International Conference on Machine learning (ICML),…
123. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. [Robust speech recognition via large-scale weak supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/). arxiv 2022. arXiv preprint arXiv:2212.04356, 10, 2022. 2, 7
124. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In Advances in Neural Information Processing Systems (NeurIPS), 2024. 2, 6, 9
125. Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. [Laion-400m: Open dataset of clip-filtered 400 million image-text pairs](https://md.research.elamur.ai/free/en/papers/schuhmann-2021-laion-400m-open-dataset-clip/). arXiv preprint arXiv:2111.02114, 2021. 8
126. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 6 [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E+6)
127. Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. [A-okvqa: A benchmark for visual question answering using world knowledge](https://md.research.elamur.ai/free/en/papers/schwenk-2022-okvqa-benchmark-visual-question-answering/). In Proceedings of the European Conference on Computer Vision (ECCV), 2022. 8
128. Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. Kvqa: Knowledge-aware visual question answering. In Proceedings of the Conference on Artificial Intelligence (AAAI), 2019. 8 [find](https://scholar.google.com/scholar?q=Sanket+Shah%2C+Anand+Mishra%2C+Naganand+Yadati%2C+and+Partha+Pratim+Talukdar%2E+Kvqa%3A+Knowledge%2Daware+visual+question+answering%2E+In+Proceedings+of+the+Conference+on+Artificial+Intelligence+%28AAAI%29%2C+2019%2E+8)
129. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. [Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/). In The Annual Meeting of the Association for Computational Linguistics (ACL), 2018. 8
130. Baoguang Shi, Cong Yao, Minghui Liao, Mingkun Yang, Pei Xu, Linyan Cui, Serge Belongie, Shijian Lu, and Xiang Bai. Icdar2017 competition on reading chinese text in the wild (rctw-17). In International Conference on Document Analysis and Recognition (ICDAR), 2017. 8 [find](https://scholar.google.com/scholar?q=Baoguang+Shi%2C+Cong+Yao%2C+Minghui+Liao%2C+Mingkun+Yang%2C+Pei+Xu%2C+Linyan+Cui%2C+Serge+Belongie%2C+Shijian+Lu%2C+and+Xiang+Bai%2E+Icdar2017+competition+on+reading+chinese+text+in+the+wild+%28rctw%2D17%29%2E+In+International)
131. Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. Design2code: How far are we from automating front-end engineering? arXiv preprint arXiv:2403.03163, 2024. 2, 6, 9, 10 — [arXiv:2403.03163](https://arxiv.org/abs/2403.03163) [find](https://scholar.google.com/scholar?q=Chenglei+Si%2C+Yanzhe+Zhang%2C+Zhengyuan+Yang%2C+Ruibo+Liu%2C+and+Diyi+Yang%2E+Design2code%3A+How+far+are+we+from+automating+front%2Dend+engineering%3F+arXiv+preprint+arXiv%3A2403%2E03163%2C+2024%2E+2%2C+6%2C+9%2C+10+%E2%80%94+%5BarXiv%3A2403)
132. Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In Proceedings of the European Conference on Computer Vision (ECCV), 2020. 8 [find](https://scholar.google.com/scholar?q=Oleksii+Sidorov%2C+Ronghang+Hu%2C+Marcus+Rohrbach%2C+and+Amanpreet+Singh%2E+Textcaps%3A+a+dataset+for+image+captioning+with+reading+comprehension%2E+In+Proceedings+of+the+European+Conference+on+Computer+Vision+%28E)
133. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2, 8, 9 [find](https://scholar.google.com/scholar?q=Amanpreet+Singh%2C+Vivek+Natarajan%2C+Meet+Shah%2C+Yu+Jiang%2C+Xinlei+Chen%2C+Dhruv+Batra%2C+Devi+Parikh%2C+and+Marcus+Rohrbach%2E+Towards+vqa+models+that+can+read%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Comput)
134. Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token to sparse memory for long video understanding. arXiv preprint arXiv:2307.16449, 2023. 2 [find](https://scholar.google.com/scholar?q=Enxin+Song%2C+Wenhao+Chai%2C+Guanhong+Wang%2C+Yucheng+Zhang%2C+Haoyang+Zhou%2C+Feiyang+Wu%2C+Xun+Guo%2C+Tian+Ye%2C+Yan+Lu%2C+Jenq%2DNeng+Hwang%2C+et+al%2E+Moviechat%3A+From+dense+token+to+sparse+memory+for+long+video+understan)
135. Enxin Song, Wenhao Chai, Tian Ye, Jenq-Neng Hwang, Xi Li, and Gaoang Wang. Moviechat+: Question-aware sparse memory for long video question answering. arXiv preprint arXiv:2404.17176, 2024. 2 — [arXiv:2404.17176](https://arxiv.org/abs/2404.17176) [find](https://scholar.google.com/scholar?q=Enxin+Song%2C+Wenhao+Chai%2C+Tian+Ye%2C+Jenq%2DNeng+Hwang%2C+Xi+Li%2C+and+Gaoang+Wang%2E+Moviechat%2B%3A+Question%2Daware+sparse+memory+for+long+video+question+answering%2E+arXiv+preprint+arXiv%3A2404%2E17176%2C+2024%2E+2+%E2%80%94+%5BarXiv)
136. Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners, 2024. 2 [find](https://scholar.google.com/scholar?q=Quan+Sun%2C+Yufeng+Cui%2C+Xiaosong+Zhang%2C+Fan+Zhang%2C+Qiying+Yu%2C+Zhengxiong+Luo%2C+Yueze+Wang%2C+Yongming+Rao%2C+Jingjing+Liu%2C+Tiejun+Huang%2C+and+Xinlong+Wang%2E+Generative+multimodal+models+are+in%2Dcontext+learners)
137. Yipeng Sun, Zihan Ni, Chee-Kheng Chng, Yuliang Liu, Canjie Luo, Chun Chet Ng, Junyu Han, Errui Ding, Jingtuo Liu, Dimosthenis Karatzas, et al. Icdar 2019 competition on large-scale street view text with partial labeling-rrc-lsvt. In International Conference on Document Analysis and Recognition (ICDA… [find](https://scholar.google.com/scholar?q=Yipeng+Sun%2C+Zihan+Ni%2C+Chee%2DKheng+Chng%2C+Yuliang+Liu%2C+Canjie+Luo%2C+Chun+Chet+Ng%2C+Junyu+Han%2C+Errui+Ding%2C+Jingtuo+Liu%2C+Dimosthenis+Karatzas%2C+et+al%2E+Icdar+2019+competition+on+large%2Dscale+street+view+text+wi)
138. Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Alpha-CLIP: A clip model focusing on wherever you want. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2 [find](https://scholar.google.com/scholar?q=Zeyi+Sun%2C+Ye+Fang%2C+Tong+Wu%2C+Pan+Zhang%2C+Yuhang+Zang%2C+Shu+Kong%2C+Yuanjun+Xiong%2C+Dahua+Lin%2C+and+Jiaqi+Wang%2E+Alpha%2DCLIP%3A+A+clip+model+focusing+on+wherever+you+want%2E+In+Proceedings+of+the+IEEE%2FCVF+Conferenc)
139. S. Svetlichnaya. DeepForm: Understand structured documents at scale., 2020. 2, 8, 9 [find](https://scholar.google.com/scholar?q=S%2E+Svetlichnaya%2E+DeepForm%3A+Understand+structured+documents+at+scale%2E%2C+2020%2E+2%2C+8%2C+9)
140. Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. VisualMRC: Machine reading comprehension on document images. In Proceedings of the Conference on Artificial Intelligence (AAAI), 2021. 2, 8, 9 [find](https://scholar.google.com/scholar?q=Ryota+Tanaka%2C+Kyosuke+Nishida%2C+and+Sen+Yoshida%2E+VisualMRC%3A+Machine+reading+comprehension+on+document+images%2E+In+Proceedings+of+the+Conference+on+Artificial+Intelligence+%28AAAI%29%2C+2021%2E+2%2C+8%2C+9)
141. Jingqun Tang, Chunhui Lin, Zhen Zhao, Shu Wei, Binghong Wu, Qi Liu, Hao Feng, Yang Li, Siqi Wang, Lei Liao, Wei Shi, Yuliang Liu, Hao Liu, Yuan Xie, Xiang Bai, and Can Huang. Textsquare: Scaling up text-centric visual instruction tuning, 2024. 2 [find](https://scholar.google.com/scholar?q=Jingqun+Tang%2C+Chunhui+Lin%2C+Zhen+Zhao%2C+Shu+Wei%2C+Binghong+Wu%2C+Qi+Liu%2C+Hao+Feng%2C+Yang+Li%2C+Siqi+Wang%2C+Lei+Liao%2C+Wei+Shi%2C+Yuliang+Liu%2C+Hao+Liu%2C+Yuan+Xie%2C+Xiang+Bai%2C+and+Can+Huang%2E+Textsquare%3A+Scaling+up+te)
142. Gemini Team. [Gemini: A family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/), 2023. 1, 9, 10
143. InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities. [https://github.com/InternLM/InternLM](https://github.com/InternLM/InternLM), 2023. 1, 2, 7, 8 [↗](https://github.com/InternLM/InternLM)
144. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv.org, 2023. 1, 2
145. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/), 2023. 1, 2
146. Yuxuan Wan, Chaozheng Wang, Yi Dong, Wenxuan Wang, Shuqing Li, Yintong Huo, and Michael R Lyu. Automatically generating ui code from screenshot: A divide-and-conquer-based approach. arXiv preprint arXiv:2406.16386, 2024. 6 — [arXiv:2406.16386](https://arxiv.org/abs/2406.16386) [find](https://scholar.google.com/scholar?q=Yuxuan+Wan%2C+Chaozheng+Wang%2C+Yi+Dong%2C+Wenxuan+Wang%2C+Shuqing+Li%2C+Yintong+Huo%2C+and+Michael+R+Lyu%2E+Automatically+generating+ui+code+from+screenshot%3A+A+divide%2Dand%2Dconquer%2Dbased+approach%2E+arXiv+preprint+arX)
147. Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574, 2023. 8 — [arXiv:2311.07574](https://arxiv.org/abs/2311.07574) [find](https://scholar.google.com/scholar?q=Junke+Wang%2C+Lingchen+Meng%2C+Zejia+Weng%2C+Bo+He%2C+Zuxuan+Wu%2C+and+Yu%2DGang+Jiang%2E+To+see+is+to+believe%3A+Prompting+gpt%2D4v+for+better+visual+instruction+tuning%2E+arXiv+preprint+arXiv%3A2311%2E07574%2C+2023%2E+8+%E2%80%94+%5BarX)
148. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. [Cogvlm: Visual expert for pretrained language models](https://md.research.elamur.ai/free/en/papers/wang-2023-cogvlm-visual-expert-pretrained/), 2023. 2
149. Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision-language models. arXiv preprint arXiv:2312.06109, 2023. 2 — [arXiv:2312.06109](https://arxiv.org/abs/2312.06109) [find](https://scholar.google.com/scholar?q=Haoran+Wei%2C+Lingyu+Kong%2C+Jinyue+Chen%2C+Liang+Zhao%2C+Zheng+Ge%2C+Jinrong+Yang%2C+Jianjian+Sun%2C+Chunrui+Han%2C+and+Xiangyu+Zhang%2E+Vary%3A+Scaling+up+the+vision+vocabulary+for+large+vision%2Dlanguage+models%2E+arXiv+p)
150. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems (NeurIPS), 2022. 8
151. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/), 2023. 2
152. Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. Longvlm: Efficient long video understanding via large language models. arXiv preprint arXiv:2404.03384, 2024. 2 — [arXiv:2404.03384](https://arxiv.org/abs/2404.03384) [find](https://scholar.google.com/scholar?q=Yuetian+Weng%2C+Mingfei+Han%2C+Haoyu+He%2C+Xiaojun+Chang%2C+and+Bohan+Zhuang%2E+Longvlm%3A+Efficient+long+video+understanding+via+large+language+models%2E+arXiv+preprint+arXiv%3A2404%2E03384%2C+2024%2E+2+%E2%80%94+%5BarXiv%3A2404%2E0338)
153. XAI. Grok-1.5 vision preview. 2024. 2, 9 [find](https://scholar.google.com/scholar?q=XAI%2E+Grok%2D1%2E5+vision+preview%2E+2024%2E+2%2C+9)
154. Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava : Parameter-free llava extension from images to videos for video dense captioning, 2024. 6 [find](https://scholar.google.com/scholar?q=Lin+Xu%2C+Yilin+Zhao%2C+Daquan+Zhou%2C+Zhijie+Lin%2C+See+Kiong+Ng%2C+and+Jiashi+Feng%2E+Pllava+%3A+Parameter%2Dfree+llava+extension+from+images+to+videos+for+video+dense+captioning%2C+2024%2E+6)
155. Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. arXiv preprint arXiv:2403.11703, 2024. 2 [find](https://scholar.google.com/scholar?q=Ruyi+Xu%2C+Yuan+Yao%2C+Zonghao+Guo%2C+Junbo+Cui%2C+Zanlin+Ni%2C+Chunjiang+Ge%2C+Tat%2DSeng+Chua%2C+Zhiyuan+Liu%2C+Maosong+Sun%2C+and+Gao+Huang%2E+Llava%2Duhd%3A+an+lmm+perceiving+any+aspect+ratio+and+high%2Dresolution+images%2E+ar)
156. Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. [Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model](https://md.research.elamur.ai/free/en/papers/ye-2023-ureader-universal-ocr-free-visually/). arXiv preprint arXiv:2310.05126, 2023. 2
157. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. [mplug-owl: Modularization empowers large language models with multimodality](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/). arXiv.org, 2023. 2
158. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics (TACL), 2014. 8 [find](https://scholar.google.com/scholar?q=Peter+Young%2C+Alice+Lai%2C+Micah+Hodosh%2C+and+Julia+Hockenmaier%2E+From+image+descriptions+to+visual+denotations%3A+New+similarity+metrics+for+semantic+inference+over+event+descriptions%2E+Transactions+of+the+A)
159. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). arXiv preprint arXiv:2309.12284, 2023. 8
160. Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 2 [find](https://scholar.google.com/scholar?q=Shoubin+Yu%2C+Jaemin+Cho%2C+Prateek+Yadav%2C+and+Mohit+Bansal%2E+Self%2Dchained+image%2Dlanguage+model+for+video+localization+and+question+answering%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29)
161. Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. RLHF-V: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patte… [find](https://scholar.google.com/scholar?q=Tianyu+Yu%2C+Yuan+Yao%2C+Haoye+Zhang%2C+Taiwen+He%2C+Yifeng+Han%2C+Ganqu+Cui%2C+Jinyi+Hu%2C+Zhiyuan+Liu%2C+Hai%2DTao+Zheng%2C+Maosong+Sun%2C+et+al%2E+RLHF%2DV%3A+Towards+trustworthy+mllms+via+behavior+alignment+from+fine%2Dgrained)
162. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. [Mm-vet: Evaluating large multimodal models for integrated capabilities](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/). arXiv preprint arXiv:2308.02490, 2023. 2, 9
163. Tai-Ling Yuan, Zhe Zhu, Kun Xu, Cheng-Jun Li, Tai-Jiang Mu, and Shi-Min Hu. A large chinese text dataset in the wild. Journal of Computer Science and Technology, 34(3): 509–521, 2019. 8 [find](https://scholar.google.com/scholar?q=Tai%2DLing+Yuan%2C+Zhe+Zhu%2C+Kun+Xu%2C+Cheng%2DJun+Li%2C+Tai%2DJiang+Mu%2C+and+Shi%2DMin+Hu%2E+A+large+chinese+text+dataset+in+the+wild%2E+Journal+of+Computer+Science+and+Technology%2C+34%283%29%3A+509%E2%80%93521%2C+2019%2E+8)
164. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline… — [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/)
165. Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy. Contextual object detection with multimodal large language models. arXiv preprint arXiv:2305.18279, 2023. 2 — [arXiv:2305.18279](https://arxiv.org/abs/2305.18279) [find](https://scholar.google.com/scholar?q=Yuhang+Zang%2C+Wei+Li%2C+Jun+Han%2C+Kaiyang+Zhou%2C+and+Chen+Change+Loy%2E+Contextual+object+detection+with+multimodal+large+language+models%2E+arXiv+preprint+arXiv%3A2305%2E18279%2C+2023%2E+2+%E2%80%94+%5BarXiv%3A2305%2E18279%5D%28https%3A)
166. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. [GLM-130b: An open bilingual pre-trained model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/). In Proceedings of the…
167. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. [Sigmoid loss for language image pre-training](https://md.research.elamur.ai/free/en/papers/zhai-2023-sigmoid-loss-language-image-pre/), 2023. 2
168. Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-CLIP: Unlocking the long-text capability of clip. arXiv preprint arXiv:2403.15378, 2024. 2 — [arXiv:2403.15378](https://arxiv.org/abs/2403.15378) [find](https://scholar.google.com/scholar?q=Beichen+Zhang%2C+Pan+Zhang%2C+Xiaoyi+Dong%2C+Yuhang+Zang%2C+and+Jiaqi+Wang%2E+Long%2DCLIP%3A+Unlocking+the+long%2Dtext+capability+of+clip%2E+arXiv+preprint+arXiv%3A2403%2E15378%2C+2024%2E+2+%E2%80%94+%5BarXiv%3A2403%2E15378%5D%28https%3A%2F%2Farxiv%2Eo)
169. Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple llm framework for long-range video question-answering. arXiv preprint arXiv:2312.17235, 2023. 2 [find](https://scholar.google.com/scholar?q=Ce+Zhang%2C+Taixi+Lu%2C+Md+Mohaiminul+Islam%2C+Ziyang+Wang%2C+Shoubin+Yu%2C+Mohit+Bansal%2C+and+Gedas+Bertasius%2E+A+simple+llm+framework+for+long%2Drange+video+question%2Danswering%2E+arXiv+preprint+arXiv%3A2312%2E17235%2C+20)
170. Hang Zhang, Xin Li, and Lidong Bing. [Video-llama: An instruction-tuned audio-visual language model for video understanding](https://md.research.elamur.ai/free/en/papers/zhang-2023-video-llama-instruction-tuned-audio/). arXiv preprint arXiv:2306.02858, 2023. 2
171. Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. [Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition](https://md.research.elamur.ai/free/en/papers/zhang-2023-internlm-xcomposer-vision-language/). arXiv preprint arXiv:2309.15112, 2023. 1, 2, 8
172. Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 6 [find](https://scholar.google.com/scholar?q=Peiyuan+Zhang%2C+Kaichen+Zhang%2C+Bo+Li%2C+Guangtao+Zeng%2C+Jingkang+Yang%2C+Yuanhan+Zhang%2C+Ziyue+Wang%2C+Haoran+Tan%2C+Chunyuan+Li%2C+and+Ziwei+Liu%2E+Long+context+transfer+from+language+to+vision%2E+arXiv+preprint+arXi)
173. Rui Zhang, Yongsheng Zhou, Qianyi Jiang, Qi Song, Nan Li, Kai Zhou, Lei Wang, Dong Wang, Minghui Liao, Mingkun Yang, et al. Icdar 2019 robust reading challenge on reading chinese text on signboard. In International Conference on Document Analysis and Recognition (ICDAR), 2019. 8 [find](https://scholar.google.com/scholar?q=Rui+Zhang%2C+Yongsheng+Zhou%2C+Qianyi+Jiang%2C+Qi+Song%2C+Nan+Li%2C+Kai+Zhou%2C+Lei+Wang%2C+Dong+Wang%2C+Minghui+Liao%2C+Mingkun+Yang%2C+et+al%2E+Icdar+2019+robust+reading+challenge+on+reading+chinese+text+on+signboard%2E+In)
174. Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding, 2024. 2 [find](https://scholar.google.com/scholar?q=Tao+Zhang%2C+Xiangtai+Li%2C+Hao+Fei%2C+Haobo+Yuan%2C+Shengqiong+Wu%2C+Shunping+Ji%2C+Chen+Change+Loy%2C+and+Shuicheng+Yan%2E+Omg%2Dllava%3A+Bridging+image%2Dlevel%2C+object%2Dlevel%2C+pixel%2Dlevel+reasoning+and+understanding%2C+202)
175. Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. Beyond llava-hd: Diving into high-resolution large multimodal models, 2024. 2 [find](https://scholar.google.com/scholar?q=Yi%2DFan+Zhang%2C+Qingsong+Wen%2C+Chaoyou+Fu%2C+Xue+Wang%2C+Zhang+Zhang%2C+Liang+Wang%2C+and+Rong+Jin%2E+Beyond+llava%2Dhd%3A+Diving+into+high%2Dresolution+large+multimodal+models%2C+2024%2E+2)
176. Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. [Mmicl: Empowering vision-language model with multi-modal in-context learning](https://md.research.elamur.ai/free/en/papers/zhao-2023-mmicl-empowering-vision-language-model/). arXiv.org, 2023. 2
177. Wenliang Zhao, Xumin Yu, and Zengyi Qin. Melotts: High-quality multi-lingual multi-accent text-to-speech, 2023. 2, 7 [find](https://scholar.google.com/scholar?q=Wenliang+Zhao%2C+Xumin+Yu%2C+and+Zengyi+Qin%2E+Melotts%3A+High%2Dquality+multi%2Dlingual+multi%2Daccent+text%2Dto%2Dspeech%2C+2023%2E+2%2C+7)
178. Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization. arXiv preprint arXiv:2311.16839, 2023. 6 — [arXiv:2311.16839](https://arxiv.org/abs/2311.16839) [find](https://scholar.google.com/scholar?q=Zhiyuan+Zhao%2C+Bin+Wang%2C+Linke+Ouyang%2C+Xiaoyi+Dong%2C+Jiaqi+Wang%2C+and+Conghui+He%2E+Beyond+hallucinations%3A+Enhancing+lvlms+through+hallucination%2Daware+direct+preference+optimization%2E+arXiv+preprint+arXiv%3A2)
179. Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. MLVU: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264, 2024. 2, 9 [find](https://scholar.google.com/scholar?q=Junjie+Zhou%2C+Yan+Shu%2C+Bo+Zhao%2C+Boya+Wu%2C+Shitao+Xiao%2C+Xi+Yang%2C+Yongping+Xiong%2C+Bo+Zhang%2C+Tiejun+Huang%2C+and+Zheng+Liu%2E+MLVU%3A+A+comprehensive+benchmark+for+multi%2Dtask+long+video+understanding%2E+arXiv+prep)
180. Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.11411, 2024. 6 — [arXiv:2402.11411](https://arxiv.org/abs/2402.11411) [find](https://scholar.google.com/scholar?q=Yiyang+Zhou%2C+Chenhang+Cui%2C+Rafael+Rafailov%2C+Chelsea+Finn%2C+and+Huaxiu+Yao%2E+Aligning+modalities+in+vision+large+language+models+via+preference+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A2402%2E11411%2C+2024%2E+6+%E2%80%94+%5Bar)
181. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. [Minigpt-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/). arXiv.org, 2023. 1, 2

*Updated 1 November 2026*