---
title: "CogVLM2 — paper by Hong et al. (2024) — summary and ideas"
description: "Text-only LLMs cannot use visual input. 91 references, cited by 45 papers here. Among them: 7."
authors: ["Wenyi Hong", "Weihan Wang", "Ming Ding", "Wenmeng Yu", "Qingsong Lv", "Yan Wang", "Yean Cheng", "Shiyu Huang", "Junhui Ji", "Zhao Xue", "Lei Zhao", "Zhuoyi Yang", "Xiaotao Gu", "Xiaohan Zhang", "Guanyu Feng", "Da Yin", "Zihan Wang", "Ji Qi", "Xixuan Song", "Peng Zhang", "Debing Liu", "Bin Xu", "Juanzi Li", "Yuxiao Dong", "Jie Tang"]
year: 2024
venue: "arXiv:2408.16500"
source: "https://arxiv.org/abs/2408.16500"
doi: "10.48550/arxiv.2408.16500"
arxiv: "2408.16500"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hong-2024-cogvlm2-visual-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hong-2024-cogvlm2-visual-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / CogVLM2: Visual Language Models for Image and Video Understanding

# CogVLM2: Visual Language Models for Image and Video Understanding

**Authors:** [Wenyi Hong](https://md.research.elamur.ai/free/en/authors/wenyi-hong/), [Weihan Wang](https://md.research.elamur.ai/free/en/authors/weihan-wang/), [Ming Ding](https://md.research.elamur.ai/free/en/authors/ming-ding/), [Wenmeng Yu](https://md.research.elamur.ai/free/en/authors/wenmeng-yu/), [Qingsong Lv](https://md.research.elamur.ai/free/en/authors/qingsong-lv/), [Yan Wang](https://md.research.elamur.ai/free/en/authors/yan-wang/), Yean Cheng, [Shiyu Huang](https://md.research.elamur.ai/free/en/authors/shiyu-huang/), [Junhui Ji](https://md.research.elamur.ai/free/en/authors/junhui-ji/), Zhao Xue, [Lei Zhao](https://md.research.elamur.ai/free/en/authors/lei-zhao/), [Zhuoyi Yang](https://md.research.elamur.ai/free/en/authors/zhuoyi-yang/), [Xiaotao Gu](https://md.research.elamur.ai/free/en/authors/xiaotao-gu/), [Xiaohan Zhang](https://md.research.elamur.ai/free/en/authors/xiaohan-zhang/), [Guanyu Feng](https://md.research.elamur.ai/free/en/authors/guanyu-feng/), [Da Yin](https://md.research.elamur.ai/free/en/authors/da-yin/), [Zihan Wang](https://md.research.elamur.ai/free/en/authors/zihan-wang/), [Ji Qi](https://md.research.elamur.ai/free/en/authors/ji-qi/), Xixuan Song, Peng Zhang, Debing Liu, [Bin Xu](https://md.research.elamur.ai/free/en/authors/bin-xu/), [Juanzi Li](https://md.research.elamur.ai/free/en/authors/juanzi-li/), [Yuxiao Dong](https://md.research.elamur.ai/free/en/authors/yuxiao-dong/) and [Jie Tang](https://md.research.elamur.ai/free/en/authors/jie-tang/) · **Year:** 2024 · **Venue:** arXiv:2408.16500 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2408.16500) · [PDF](https://arxiv.org/pdf/2408.16500) · [DOI 10.48550/arxiv.2408.16500](https://doi.org/10.48550/arxiv.2408.16500) · [arXiv 2408.16500](https://arxiv.org/abs/2408.16500) · [OpenAlex](https://openalex.org/W4402711432)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [45](https://md.research.elamur.ai/free/en/papers/hong-2024-cogvlm2-visual-language-models/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Text-only LLMs cannot use visual input. Existing vision-language models often have three weaknesses: they connect image features to the LLM only superficially, high-resolution inputs make them expensive, and they lose track of timing in videos.

This report from Zhipu AI and Tsinghua University introduces the CogVLM2 family, which follows VisualGLM, CogVLM and CogAgent. Every model pairs an EVA-CLIP encoder with a new adapter: a 2×2 convolution followed by a SwiGLU block. This adapter shrinks the visual token sequence fourfold with almost no quality loss.

- **CogVLM2** is built on LLaMA3-8B. It keeps visual expert modules in the attention and FFN layers to protect its language abilities.
- **GLM-4V-9B** is built on GLM-4-9B. It drops visual experts and instead relies on joint text-image training to retain language knowledge.
- **CogVLM2-Video** receives video frames tagged with timestamps. It learns from temporal grounding data that the authors generate automatically.

The authors also enlarged and cleaned the pre-training and post-training data. Among open models, the family scores at or near the top on OCR, chart, general and video benchmarks, sometimes beating proprietary systems. The weights are publicly released, giving researchers strong reusable baselines.

## Key points
- **Model sizes and inputs:**
  - CogVLM2 (CogVLM2-LLaMA3-8B) has 19B parameters and accepts images up to 1344×1344.
  - GLM-4V-9B has 13B parameters, uses a 4B-parameter EVA-E ViT and accepts 1120×1120 images. The authors chose a large ViT because it is important for recognizing Chinese characters.
  - CogVLM2-Video has 12B parameters.
- **CogVLM2 results:** 84.2 on TextVQA, 92.3 on DocVQA, 83.3 on VCR and 81.0 on ChartQA.
- **GLM-4V-9B result:** 786 on OCRbench. Its co-training approach caused a slight drop on pure language tasks.
- **CogVLM2-Video results:** 62.3 average on MVBench, which ties VideoChat2_HD_mistral, and 3.41 average on VideoChatGPT-Bench. The stronger GLM-4V-Plus reaches 71.2 on MVBench and 833 on OCRbench and is offered through Zhipu's MaaS platform.
- **Code:** github.com/THUDM/CogVLM2 and github.com/THUDM/GLM-4.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 269 works

## Cite

Cite as: Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, Lei Zhao, Zhuoyi Yang, Xiaotao Gu, Xiaohan Zhang, Guanyu Feng, Da Yin, Zihan Wang, Ji Qi, Xixuan Song, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Yuxiao Dong and Jie Tang. 2024. CogVLM2: Visual Language Models for Image and Video Understanding. arXiv:2408.16500.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hong2024cogvlm2,
  title = {{CogVLM2: Visual Language Models for Image and Video Understanding}},
  author = {Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, Lei Zhao, Zhuoyi Yang, Xiaotao Gu, Xiaohan Zhang, Guanyu Feng, Da Yin, Zihan Wang, Ji Qi, Xixuan Song, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Yuxiao Dong and Jie Tang},
  year = {2024},
  doi = {10.48550/arxiv.2408.16500},
  eprint = {2408.16500},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2408.16500}
}
```

- [← previous Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs](https://md.research.elamur.ai/free/en/papers/singh-2024-tokenization-counts-impact-tokenization-arithmetic/)
- [next → Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws](https://md.research.elamur.ai/free/en/papers/sardana-2024-chinchilla-optimal-accounting-inference-language/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

91 references, 42 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. Acharya, K. Kafle, and C. Kanan. Tallyqa: Answering complex counting questions. In Proc. of Association for the Advancement of Artificial Intelligence, 2019. [find](https://scholar.google.com/scholar?q=M%2E+Acharya%2C+K%2E+Kafle%2C+and+C%2E+Kanan%2E+Tallyqa%3A+Answering+complex+counting+questions%2E+In+Proc%2E+of+Association+for+the+Advancement+of+Artificial+Intelligence%2C+2019%2E)
2. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
3. D. Alexey. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). arXiv preprint arXiv: 2010.11929, 2020.
4. S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. Vqa: Visual question answering. In Proc. of International Conference on Computer Vision, pages 2425–2433, 2015. [find](https://scholar.google.com/scholar?q=S%2E+Antol%2C+A%2E+Agrawal%2C+J%2E+Lu%2C+M%2E+Mitchell%2C+D%2E+Batra%2C+C%2E+L%2E+Zitnick%2C+and+D%2E+Parikh%2E+Vqa%3A+Visual+question+answering%2E+In+Proc%2E+of+International+Conference+on+Computer+Vision%2C+pages+2425%E2%80%932433%2C+2015%2E)
5. J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.[12966, 2023](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/).
6. J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. [Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/). arXiv preprint arXiv:2308.12966, 2023.
7. A. F. Biten, R. Tito, A. Mafla, L. Gomez, M. Rusiñol, C. Jawahar, E. Valveny, and D. Karatzas. Scene text visual question answering. In Proc. of International Conference on Computer Vision, pages 4290–4300, 2019. [find](https://scholar.google.com/scholar?q=A%2E+F%2E+Biten%2C+R%2E+Tito%2C+A%2E+Mafla%2C+L%2E+Gomez%2C+M%2E+Rusi%C3%B1ol%2C+C%2E+Jawahar%2C+E%2E+Valveny%2C+and+D%2E+Karatzas%2E+Scene+text+visual+question+answering%2E+In+Proc%2E+of+International+Conference+on+Computer+Vision%2C+pages+4290)
8. L. Blecher, G. Cucurull, T. Scialom, and R. Stojnic. Nougat: Neural optical understanding for academic documents. arXiv preprint arXiv:2308.13418, 2023. [find](https://scholar.google.com/scholar?q=L%2E+Blecher%2C+G%2E+Cucurull%2C+T%2E+Scialom%2C+and+R%2E+Stojnic%2E+Nougat%3A+Neural+optical+understanding+for+academic+documents%2E+arXiv+preprint+arXiv%3A2308%2E13418%2C+2023%2E)
9. M. Byeon, B. Park, H. Kim, S. Lee, W. Baek, and S. Kim. Coyo-700m: Image-text pair dataset. [https://github.com/kakaobrain/coyo-dataset](https://github.com/kakaobrain/coyo-dataset), 2022. [↗](https://github.com/kakaobrain/coyo-dataset)
10. J. Chen, J. Tang, J. Qin, X. Liang, L. Liu, E. P. Xing, and L. Lin. [Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning](https://md.research.elamur.ai/free/en/papers/chen-2021-geoqa-geometric-question-answering-benchmark/), 2022.
11. L. Chen, J. Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, J. Wang, Y. Qiao, D. Lin, et al. [Are we on the right way for evaluating large vision-language models?](https://md.research.elamur.ai/free/en/papers/chen-2024-right-way-evaluating-large-vision/) arXiv preprint arXiv:2403.20330, 2024.
12. L. Chen, X. Wei, J. Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, B. Lin, Z. Tang, L. Yuan, Y. Qiao, D. Lin, F. Zhao, and J. Wang. Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprint arXiv:2406.04325, 2024. — [arXiv:2406.04325](https://arxiv.org/abs/2406.04325) [find](https://scholar.google.com/scholar?q=L%2E+Chen%2C+X%2E+Wei%2C+J%2E+Li%2C+X%2E+Dong%2C+P%2E+Zhang%2C+Y%2E+Zang%2C+Z%2E+Chen%2C+H%2E+Duan%2C+B%2E+Lin%2C+Z%2E+Tang%2C+L%2E+Yuan%2C+Y%2E+Qiao%2C+D%2E+Lin%2C+F%2E+Zhao%2C+and+J%2E+Wang%2E+Sharegpt4video%3A+Improving+video+understanding+and+generation+with)
13. X. Chen, X. Wang, S. Changpinyo, A. Piergiovanni, P. Padlewski, D. Salz, S. Goodman, A. Grycner, B. Mustafa, L. Beyer, et al. [Pali: A jointly-scaled multilingual language-image model](https://md.research.elamur.ai/free/en/papers/chen-2022-pali-jointly-scaled-multilingual-language/). arXiv preprint arXiv:2209.06794, 2022.
14. Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, B. Li, P. Luo, T. Lu, Y. Qiao, and J. Dai. [Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks](https://md.research.elamur.ai/free/en/papers/chen-2023-internvl-scaling-up-vision/). arXiv preprint arXiv:2312.14238, 2023.
15. S. Y. Gadre, G. Ilharco, A. Fang, J. Hayase, G. Smyrnis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, J. Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=S%2E+Y%2E+Gadre%2C+G%2E+Ilharco%2C+A%2E+Fang%2C+J%2E+Hayase%2C+G%2E+Smyrnis%2C+T%2E+Nguyen%2C+R%2E+Marten%2C+M%2E+Wortsman%2C+D%2E+Ghosh%2C+J%2E+Zhang%2C+et+al%2E+Datacomp%3A+In+search+of+the+next+generation+of+multimodal+datasets%2E+Advances+in+Ne)
16. J. Gao, R. Pi, J. Zhang, J. Ye, W. Zhong, Y. Wang, L. Hong, J. Han, H. Xu, Z. Li, et al. G-llava: Solving geometric problem with multi-modal large language model. arXiv preprint arXiv:2312.11370, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Gao%2C+R%2E+Pi%2C+J%2E+Zhang%2C+J%2E+Ye%2C+W%2E+Zhong%2C+Y%2E+Wang%2C+L%2E+Hong%2C+J%2E+Han%2C+H%2E+Xu%2C+Z%2E+Li%2C+et+al%2E+G%2Dllava%3A+Solving+geometric+problem+with+multi%2Dmodal+large+language+model%2E+arXiv+preprint+arXiv%3A2312%2E11370%2C+2023)
17. T. GLM, A. Zeng, B. Xu, B. Wang, C. Zhang, D. Yin, D. Rojas, G. Feng, H. Zhao, H. Lai, et al. [Chatglm: A family of large language models from glm-130b to glm-4 all tools](https://md.research.elamur.ai/free/en/papers/glm-2024-chatglm-family-large-language-models/). arXiv preprint arXiv:2406.12793, 2024.
18. R. Goyal, S. E. Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fründ, P. Yianilos, M. Mueller-Freitag, F. Hoppe, C. Thurau, I. Bax, and R. Memisevic. The “something something” video database for learning and evaluating visual common sense. In Proc. of International Conferenc… [find](https://scholar.google.com/scholar?q=R%2E+Goyal%2C+S%2E+E%2E+Kahou%2C+V%2E+Michalski%2C+J%2E+Materzynska%2C+S%2E+Westphal%2C+H%2E+Kim%2C+V%2E+Haenel%2C+I%2E+Fr%C3%BCnd%2C+P%2E+Yianilos%2C+M%2E+Mueller%2DFreitag%2C+F%2E+Hoppe%2C+C%2E+Thurau%2C+I%2E+Bax%2C+and+R%2E+Memisevic%2E+The+%E2%80%9Csomething+something%E2%80%9D)
19. K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, J. Sharma, M. Wray, M. Xu, E. Z. Xu, C. Zhao, S. Bansal, D. Batra, V. Cartillier, S. Crane, T. Do, M. Doulaty, A. Erapal… [find](https://scholar.google.com/scholar?q=K%2E+Grauman%2C+A%2E+Westbury%2C+E%2E+Byrne%2C+Z%2E+Chavis%2C+A%2E+Furnari%2C+R%2E+Girdhar%2C+J%2E+Hamburger%2C+H%2E+Jiang%2C+M%2E+Liu%2C+X%2E+Liu%2C+M%2E+Martin%2C+T%2E+Nagarajan%2C+I%2E+Radosavovic%2C+S%2E+K%2E+Ramakrishnan%2C+F%2E+Ryan%2C+J%2E+Sharma%2C+M%2E+Wray%2C+)
20. W. Hong, W. Wang, Q. Lv, J. Xu, W. Yu, J. Ji, Y. Wang, Z. Wang, Y. Dong, M. Ding, et al. [Cogagent: A visual language model for gui agents](https://md.research.elamur.ai/free/en/papers/hong-2023-cogagent-visual-language-model/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281–14290, 2024.
21. Y. Jang, Y. Song, Y. Yu, Y. Kim, and G. Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proc. of Computer Vision and Pattern Recognition, 2017. [find](https://scholar.google.com/scholar?q=Y%2E+Jang%2C+Y%2E+Song%2C+Y%2E+Yu%2C+Y%2E+Kim%2C+and+G%2E+Kim%2E+Tgif%2Dqa%3A+Toward+spatio%2Dtemporal+reasoning+in+visual+question+answering%2E+In+Proc%2E+of+Computer+Vision+and+Pattern+Recognition%2C+2017%2E)
22. K. Kafle, S. Cohen, B. Price, and C. Kanan. Dvqa: Understanding data visualizations via question answering. In Proc. of Computer Vision and Pattern Recognition, 2018. [find](https://scholar.google.com/scholar?q=K%2E+Kafle%2C+S%2E+Cohen%2C+B%2E+Price%2C+and+C%2E+Kanan%2E+Dvqa%3A+Understanding+data+visualizations+via+question+answering%2E+In+Proc%2E+of+Computer+Vision+and+Pattern+Recognition%2C+2018%2E)
23. K. Kafle and C. Kanan. An analysis of visual question answering algorithms. In Proceedings of the IEEE international conference on computer vision, pages 1965–1973, 2017. [find](https://scholar.google.com/scholar?q=K%2E+Kafle+and+C%2E+Kanan%2E+An+analysis+of+visual+question+answering+algorithms%2E+In+Proceedings+of+the+IEEE+international+conference+on+computer+vision%2C+pages+1965%E2%80%931973%2C+2017%2E)
24. S. E. Kahou, V. Michalski, A. Atkinson, A. Kadar, A. Trischler, and Y. Bengio. Figureqa: An annotated figure dataset for visual reasoning, 2018. [find](https://scholar.google.com/scholar?q=S%2E+E%2E+Kahou%2C+V%2E+Michalski%2C+A%2E+Atkinson%2C+A%2E+Kadar%2C+A%2E+Trischler%2C+and+Y%2E+Bengio%2E+Figureqa%3A+An+annotated+figure+dataset+for+visual+reasoning%2C+2018%2E)
25. M. Kazemi, H. Alvari, A. Anand, J. Wu, X. Chen, and R. Soricut. Geomverse: A systematic evaluation of large models for geometric reasoning. arXiv preprint arXiv:2312.12241, 2023. — [arXiv:2312.12241](https://arxiv.org/abs/2312.12241) [find](https://scholar.google.com/scholar?q=M%2E+Kazemi%2C+H%2E+Alvari%2C+A%2E+Anand%2C+J%2E+Wu%2C+X%2E+Chen%2C+and+R%2E+Soricut%2E+Geomverse%3A+A+systematic+evaluation+of+large+models+for+geometric+reasoning%2E+arXiv+preprint+arXiv%3A2312%2E12241%2C+2023%2E+%E2%80%94+%5BarXiv%3A2312%2E12241%5D%28)
26. A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi. [A diagram is worth a dozen images](https://md.research.elamur.ai/free/en/papers/kembhavi-2016-diagram-worth-dozen-images/), 2016.
27. A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi. [A diagram is worth a dozen images](https://md.research.elamur.ai/free/en/papers/kembhavi-2016-diagram-worth-dozen-images/). In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–251. Springer, 2016.
28. A. Kembhavi, M. Seo, D. Schwenk, J. Choi, A. Farhadi, and H. Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proc. of Computer Vision and Pattern Recognition, pages 5376–5384, 2017. [find](https://scholar.google.com/scholar?q=A%2E+Kembhavi%2C+M%2E+Seo%2C+D%2E+Schwenk%2C+J%2E+Choi%2C+A%2E+Farhadi%2C+and+H%2E+Hajishirzi%2E+Are+you+smarter+than+a+sixth+grader%3F+textbook+question+answering+for+multimodal+machine+comprehension%2E+In+Proc%2E+of+Computer+Vis)
29. W. Kim, C. Choi, W. Lee, and W. Rhee. An image grid can be worth a video: Zero-shot video question answering using a vlm, 2024. [find](https://scholar.google.com/scholar?q=W%2E+Kim%2C+C%2E+Choi%2C+W%2E+Lee%2C+and+W%2E+Rhee%2E+An+image+grid+can+be+worth+a+video%3A+Zero%2Dshot+video+question+answering+using+a+vlm%2C+2024%2E)
30. R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al. [Visual genome: Connecting language and vision using crowdsourced dense image annotations](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/). International journal of computer vision, 123:32–73, 2017.
31. J. J. Lau, S. Gayen, A. Ben Abacha, and D. Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1–10, 2018. [find](https://scholar.google.com/scholar?q=J%2E+J%2E+Lau%2C+S%2E+Gayen%2C+A%2E+Ben+Abacha%2C+and+D%2E+Demner%2DFushman%2E+A+dataset+of+clinically+generated+visual+questions+and+answers+about+radiology+images%2E+Scientific+data%2C+5%281%29%3A1%E2%80%9310%2C+2018%2E)
32. C. Li, W. Liu, R. Guo, X. Yin, K. Jiang, Y. Du, Y. Du, L. Zhu, B. Lai, X. Hu, et al. Pp-ocrv3: More attempts for the improvement of ultra lightweight ocr system. arXiv preprint arXiv:2206.03001, 2022. — [arXiv:2206.03001](https://arxiv.org/abs/2206.03001) [find](https://scholar.google.com/scholar?q=C%2E+Li%2C+W%2E+Liu%2C+R%2E+Guo%2C+X%2E+Yin%2C+K%2E+Jiang%2C+Y%2E+Du%2C+Y%2E+Du%2C+L%2E+Zhu%2C+B%2E+Lai%2C+X%2E+Hu%2C+et+al%2E+Pp%2Docrv3%3A+More+attempts+for+the+improvement+of+ultra+lightweight+ocr+system%2E+arXiv+preprint+arXiv%3A2206%2E03001%2C+2022%2E)
33. J. Li, D. Li, S. Savarese, and S. Hoi. [Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). In International conference on machine learning, pages 19730–19742. PMLR, 2023.
34. K. Li, Y. He, Y. Wang, Y. Li, W. Wang, P. Luo, Y. Wang, L. Wang, and Y. Qiao. [Videochat: Chat-centric video understanding](https://md.research.elamur.ai/free/en/papers/li-2023-videochat-chat-centric-video-understanding/). ArXiv, abs/2305.06355, 2023.
35. K. Li, Y. Wang, Y. He, Y. Li, Y. Wang, Y. Liu, Z. Wang, J. Xu, G. Chen, P. Luo, L. Wang, and Y. Qiao. [MVBench: A comprehensive multi-modal video understanding benchmark](https://md.research.elamur.ai/free/en/papers/li-2023-mvbench-comprehensive-multi-modal-video/), 2023.
36. K. Li, Y. Wang, Y. He, Y. Li, Y. Wang, L. Wang, and Y. Qiao. Uniformerv2: Spatiotemporal learning by arming image vits with video uniformer. ArXiv, abs/2211.09552, 2022. — [arXiv:2211.09552](https://arxiv.org/abs/2211.09552) [find](https://scholar.google.com/scholar?q=K%2E+Li%2C+Y%2E+Wang%2C+Y%2E+He%2C+Y%2E+Li%2C+Y%2E+Wang%2C+L%2E+Wang%2C+and+Y%2E+Qiao%2E+Uniformerv2%3A+Spatiotemporal+learning+by+arming+image+vits+with+video+uniformer%2E+ArXiv%2C+abs%2F2211%2E09552%2C+2022%2E+%E2%80%94+%5BarXiv%3A2211%2E09552%5D%28https%3A%2F%2Fa)
37. L. Li, Y. Wang, R. Xu, P. Wang, X. Feng, L. Kong, and Q. Liu. [Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models](https://md.research.elamur.ai/free/en/papers/li-2024-multimodal-arxiv-dataset-improving-scientific/), 2024.
38. Y. Li, Y. Zhang, C. Wang, Z. Zhong, Y. Chen, R. Chu, S. Liu, and J. Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814, 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Li%2C+Y%2E+Zhang%2C+C%2E+Wang%2C+Z%2E+Zhong%2C+Y%2E+Chen%2C+R%2E+Chu%2C+S%2E+Liu%2C+and+J%2E+Jia%2E+Mini%2Dgemini%3A+Mining+the+potential+of+multi%2Dmodality+vision+language+models%2E+arXiv+preprint+arXiv%3A2403%2E18814%2C+2024%2E)
39. K. Q. Lin, P. Zhang, J. Chen, S. Pramanick, D. Gao, A. J. Wang, R. Yan, and M. Z. Shou. Univtg: Towards unified video-language temporal grounding. In Proc. of International Conference on Computer Vision, 2023. [find](https://scholar.google.com/scholar?q=K%2E+Q%2E+Lin%2C+P%2E+Zhang%2C+J%2E+Chen%2C+S%2E+Pramanick%2C+D%2E+Gao%2C+A%2E+J%2E+Wang%2C+R%2E+Yan%2C+and+M%2E+Z%2E+Shou%2E+Univtg%3A+Towards+unified+video%2Dlanguage+temporal+grounding%2E+In+Proc%2E+of+International+Conference+on+Computer+Visi)
40. F. Liu, G. Emerson, and N. Collier. Visual spatial reasoning. Proc. of IEEE International Conference on Automatic Face and Gesture Recognition, 11, 2023. [find](https://scholar.google.com/scholar?q=F%2E+Liu%2C+G%2E+Emerson%2C+and+N%2E+Collier%2E+Visual+spatial+reasoning%2E+Proc%2E+of+IEEE+International+Conference+on+Automatic+Face+and+Gesture+Recognition%2C+11%2C+2023%2E)
41. H. Liu, C. Li, Y. Li, and Y. J. Lee. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/), 2024.
42. H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. [find](https://scholar.google.com/scholar?q=H%2E+Liu%2C+C%2E+Li%2C+Y%2E+Li%2C+B%2E+Li%2C+Y%2E+Zhang%2C+S%2E+Shen%2C+and+Y%2E+J%2E+Lee%2E+Llava%2Dnext%3A+Improved+reasoning%2C+ocr%2C+and+world+knowledge%2C+January+2024%2E)
43. H. Liu, C. Li, Q. Wu, and Y. J. Lee. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). Proc. of Neural Information Processing Systems, 36, 2024.
44. R. Liu, C. Li, H. Tang, Y. Ge, Y. Shan, and G. Li. St-llm: Large language models are effective temporal learners. https://arxiv.org/abs/2404.00308, 2023. — [arXiv:2404.00308](https://arxiv.org/abs/2404.00308) [↗](https://arxiv.org/abs/2404.00308)
45. Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, K. Chen, and D. Lin. [Mmbench: Is your multi-modal model an all-around player?](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/) arXiv:2307.06281, 2023.
46. Y. Liu, Z. Li, B. Yang, C. Li, X. Yin, C. lin Liu, L. Jin, and X. Bai. On the hidden mystery of ocr in large multimodal models, 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Liu%2C+Z%2E+Li%2C+B%2E+Yang%2C+C%2E+Li%2C+X%2E+Yin%2C+C%2E+lin+Liu%2C+L%2E+Jin%2C+and+X%2E+Bai%2E+On+the+hidden+mystery+of+ocr+in+large+multimodal+models%2C+2024%2E)
47. P. Lu, R. Gong, S. Jiang, L. Qiu, S. Huang, X. Liang, and S.-C. Zhu. [Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning](https://md.research.elamur.ai/free/en/papers/lu-2021-inter-gps-interpretable-geometry-problem/). In The Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Con…
48. P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Proc. of Neural Information Processing Systems, 35:2507–2521, 2022. [find](https://scholar.google.com/scholar?q=P%2E+Lu%2C+S%2E+Mishra%2C+T%2E+Xia%2C+L%2E+Qiu%2C+K%2E%2DW%2E+Chang%2C+S%2E%2DC%2E+Zhu%2C+O%2E+Tafjord%2C+P%2E+Clark%2C+and+A%2E+Kalyan%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+science+question+answering%2E+Proc%2E+of+Neural)
49. P. Lu, L. Qiu, K.-W. Chang, Y. N. Wu, S.-C. Zhu, T. Rajpurohit, P. Clark, and A. Kalyan. [Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning](https://md.research.elamur.ai/free/en/papers/lu-2022-dynamic-prompt-learning-policy-gradient/). In Proc. of International Conference on Learning Representations, 2023.
50. P. Lu, L. Qiu, J. Chen, T. Xia, Y. Zhao, W. Zhang, Z. Yu, X. Liang, and S.-C. Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In The 35th Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2021. [find](https://scholar.google.com/scholar?q=P%2E+Lu%2C+L%2E+Qiu%2C+J%2E+Chen%2C+T%2E+Xia%2C+Y%2E+Zhao%2C+W%2E+Zhang%2C+Z%2E+Yu%2C+X%2E+Liang%2C+and+S%2E%2DC%2E+Zhu%2E+Iconqa%3A+A+new+benchmark+for+abstract+diagram+understanding+and+visual+language+reasoning%2E+In+The+35th+Conference+on+N)
51. D. Luo, J. Huang, S. Gong, H. Jin, and Y. Liu. Towards generalisable video moment retrieval: Visual-dynamic injection to image-text pre-training. In Proc. of Computer Vision and Pattern Recognition, 2023. [find](https://scholar.google.com/scholar?q=D%2E+Luo%2C+J%2E+Huang%2C+S%2E+Gong%2C+H%2E+Jin%2C+and+Y%2E+Liu%2E+Towards+generalisable+video+moment+retrieval%3A+Visual%2Ddynamic+injection+to+image%2Dtext+pre%2Dtraining%2E+In+Proc%2E+of+Computer+Vision+and+Pattern+Recognition%2C+2)
52. M. Maaz, H. Rasheed, S. Khan, and F. Khan. Videogpt+: Integrating image and video encoders for enhanced video understanding, 2024. [find](https://scholar.google.com/scholar?q=M%2E+Maaz%2C+H%2E+Rasheed%2C+S%2E+Khan%2C+and+F%2E+Khan%2E+Videogpt%2B%3A+Integrating+image+and+video+encoders+for+enhanced+video+understanding%2C+2024%2E)
53. M. Maaz, H. Rasheed, S. Khan, and F. S. Khan. [Video-chatgpt: Towards detailed video understanding via large vision and language models](https://md.research.elamur.ai/free/en/papers/maaz-2023-video-chatgpt-towards-detailed-video/). In Proc. of IEEE International Conference on Automatic Face and Gesture Recognition, 2024.
54. M. Maaz, H. Rasheed, S. Khan, and F. S. Khan. [Video-chatgpt: Towards detailed video understanding via large vision and language models](https://md.research.elamur.ai/free/en/papers/maaz-2023-video-chatgpt-towards-detailed-video/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024.
55. K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proc. of Computer Vision and Pattern Recognition, pages 3195–3204, 2019. [find](https://scholar.google.com/scholar?q=K%2E+Marino%2C+M%2E+Rastegari%2C+A%2E+Farhadi%2C+and+R%2E+Mottaghi%2E+Ok%2Dvqa%3A+A+visual+question+answering+benchmark+requiring+external+knowledge%2E+In+Proc%2E+of+Computer+Vision+and+Pattern+Recognition%2C+pages+3195%E2%80%933204%2C+)
56. U.-V. Marti and H. Bunke. The iam-database: An english sentence database for offline handwriting recognition. International Journal on Document Analysis and Recognition, 5:39–46, 11 2002. [find](https://scholar.google.com/scholar?q=U%2E%2DV%2E+Marti+and+H%2E+Bunke%2E+The+iam%2Ddatabase%3A+An+english+sentence+database+for+offline+handwriting+recognition%2E+International+Journal+on+Document+Analysis+and+Recognition%2C+5%3A39%E2%80%9346%2C+11+2002%2E)
57. A. Masry, D. Long, J. Q. Tan, S. Joty, and E. Hoque. [ChartQA: A benchmark for question answering about charts with visual and logical reasoning](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/). In Proc. of IEEE International Conference on Automatic Face and Gesture Recognition, pages 2263–2279, Dublin, Ireland, May 2022. Association for Computatio…
58. A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque. [Chartqa: A benchmark for question answering about charts with visual and logical reasoning](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/). arXiv preprint arXiv:2203.10244, 2022.
59. A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque. [Chartqa: A benchmark for question answering about charts with visual and logical reasoning](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/), 2022.
60. M. Mathew, V. Bagal, R. Tito, D. Karatzas, E. Valveny, and C. V. Jawahar. Infographicvqa. In Proc. of IEEE Winter Conference on Applications of Computer Vision, pages 2582–2591, 2022. [find](https://scholar.google.com/scholar?q=M%2E+Mathew%2C+V%2E+Bagal%2C+R%2E+Tito%2C+D%2E+Karatzas%2C+E%2E+Valveny%2C+and+C%2E+V%2E+Jawahar%2E+Infographicvqa%2E+In+Proc%2E+of+IEEE+Winter+Conference+on+Applications+of+Computer+Vision%2C+pages+2582%E2%80%932591%2C+2022%2E)
61. M. Mathew, D. Karatzas, and C. Jawahar. [Docvqa: A dataset for vqa on document images](https://md.research.elamur.ai/free/en/papers/mathew-2020-docvqa-dataset-vqa-document-images/). In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021.
62. M. Mathew, D. Karatzas, and C. V. Jawahar. [Docvqa: A dataset for vqa on document images](https://md.research.elamur.ai/free/en/papers/mathew-2020-docvqa-dataset-vqa-document-images/). In Proc. of IEEE Winter Conference on Applications of Computer Vision, pages 2199–2208, 2021.
63. M. Mathew, D. Karatzas, and C. V. Jawahar. [Docvqa: A dataset for vqa on document images](https://md.research.elamur.ai/free/en/papers/mathew-2020-docvqa-dataset-vqa-document-images/), 2021.
64. A. Mishra, S. Shekhar, A. K. Singh, and A. Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947–952. IEEE, 2019. [find](https://scholar.google.com/scholar?q=A%2E+Mishra%2C+S%2E+Shekhar%2C+A%2E+K%2E+Singh%2C+and+A%2E+Chakraborty%2E+Ocr%2Dvqa%3A+Visual+question+answering+by+reading+text+in+images%2E+In+2019+international+conference+on+document+analysis+and+recognition+%28ICDAR%29%2C+pag)
65. M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, et al. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.05530, 2024.
66. S. Ren, L. Yao, S. Li, X. Sun, and L. Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. arXiv preprint arXiv:2312.02051, 2023. — [arXiv:2312.02051](https://arxiv.org/abs/2312.02051) [find](https://scholar.google.com/scholar?q=S%2E+Ren%2C+L%2E+Yao%2C+S%2E+Li%2C+X%2E+Sun%2C+and+L%2E+Hou%2E+Timechat%3A+A+time%2Dsensitive+multimodal+large+language+model+for+long+video+understanding%2E+arXiv+preprint+arXiv%3A2312%2E02051%2C+2023%2E+%E2%80%94+%5BarXiv%3A2312%2E02051%5D%28https%3A%2F%2F)
67. C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. [Laion-5b: An open large-scale dataset for training next generation image-text models](https://md.research.elamur.ai/free/en/papers/schuhmann-2022-laion-5b-open-large-scale/). Advances in Neural Information Processing Systems, 35:25278–25294, 2022.
68. C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki. [Laion-400m: Open dataset of clip-filtered 400 million image-text pairs](https://md.research.elamur.ai/free/en/papers/schuhmann-2021-laion-400m-open-dataset-clip/). arXiv preprint arXiv:2111.02114, 2021.
69. D. Schwenk, A. Khandelwal, C. Clark, K. Marino, and R. Mottaghi. [A-okvqa: A benchmark for visual question answering using world knowledge](https://md.research.elamur.ai/free/en/papers/schwenk-2022-okvqa-benchmark-visual-question-answering/). In Proc. of European Conference on Computer Vision, page 146–162, 2022.
70. N. Shazeer. Glu variants improve transformer, 2020. [find](https://scholar.google.com/scholar?q=N%2E+Shazeer%2E+Glu+variants+improve+transformer%2C+2020%2E)
71. A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. [find](https://scholar.google.com/scholar?q=A%2E+Singh%2C+V%2E+Natarajan%2C+M%2E+Shah%2C+Y%2E+Jiang%2C+X%2E+Chen%2C+D%2E+Batra%2C+D%2E+Parikh%2C+and+M%2E+Rohrbach%2E+Towards+vqa+models+that+can+read%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+vision+and+pattern+rec)
72. A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach. Towards vqa models that can read. In Proc. of Computer Vision and Pattern Recognition, pages 8317–8326, 2019. [find](https://scholar.google.com/scholar?q=A%2E+Singh%2C+V%2E+Natarajan%2C+M%2E+Shah%2C+Y%2E+Jiang%2C+X%2E+Chen%2C+D%2E+Batra%2C+D%2E+Parikh%2C+and+M%2E+Rohrbach%2E+Towards+vqa+models+that+can+read%2E+In+Proc%2E+of+Computer+Vision+and+Pattern+Recognition%2C+pages+8317%E2%80%938326%2C+2019%2E)
73. Q. Sun, Y. Fang, L. Wu, X. Wang, and Y. Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023. [find](https://scholar.google.com/scholar?q=Q%2E+Sun%2C+Y%2E+Fang%2C+L%2E+Wu%2C+X%2E+Wang%2C+and+Y%2E+Cao%2E+Eva%2Dclip%3A+Improved+training+techniques+for+clip+at+scale%2E+arXiv+preprint+arXiv%3A2303%2E15389%2C+2023%2E)
74. B. J. Tang, A. Boggust, and A. Satyanarayan. VisText: A Benchmark for Semantically Rich Chart Captioning. In Proc. of IEEE International Conference on Automatic Face and Gesture Recognition, 2023. [find](https://scholar.google.com/scholar?q=B%2E+J%2E+Tang%2C+A%2E+Boggust%2C+and+A%2E+Satyanarayan%2E+VisText%3A+A+Benchmark+for+Semantically+Rich+Chart+Captioning%2E+In+Proc%2E+of+IEEE+International+Conference+on+Automatic+Face+and+Gesture+Recognition%2C+2023%2E)
75. S. Tong, E. Brown, P. Wu, S. Woo, M. Middepogu, S. C. Akula, J. Yang, S. Yang, A. Iyer, X. Pan, A. Wang, R. Fergus, Y. LeCun, and S. Xie. [Cambrian-1: A fully open, vision-centric exploration of multimodal llms](https://md.research.elamur.ai/free/en/papers/tong-2024-cambrian-1-fully-open-vision/), 2024.
76. W. Wang, Z. He, W. Hong, Y. Cheng, X. Zhang, J. Qi, S. Huang, B. Xu, Y. Dong, M. Ding, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035, 2024. [find](https://scholar.google.com/scholar?q=W%2E+Wang%2C+Z%2E+He%2C+W%2E+Hong%2C+Y%2E+Cheng%2C+X%2E+Zhang%2C+J%2E+Qi%2C+S%2E+Huang%2C+B%2E+Xu%2C+Y%2E+Dong%2C+M%2E+Ding%2C+et+al%2E+Lvbench%3A+An+extreme+long+video+understanding+benchmark%2E+arXiv+preprint+arXiv%3A2406%2E08035%2C+2024%2E)
77. W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y. Wang, J. Ji, Z. Yang, L. Zhao, X. Song, et al. [Cogvlm: Visual expert for pretrained language models](https://md.research.elamur.ai/free/en/papers/wang-2023-cogvlm-visual-expert-pretrained/). arXiv preprint arXiv:2311.03079, 2023.
78. Z. Wang, L. Wang, T. Wu, T. Li, and G. Wu. Negative sample matters: A renaissance of metric learning for temporal grounding. In Proc. of Association for the Advancement of Artificial Intelligence, 2022. [find](https://scholar.google.com/scholar?q=Z%2E+Wang%2C+L%2E+Wang%2C+T%2E+Wu%2C+T%2E+Li%2C+and+G%2E+Wu%2E+Negative+sample+matters%3A+A+renaissance+of+metric+learning+for+temporal+grounding%2E+In+Proc%2E+of+Association+for+the+Advancement+of+Artificial+Intelligence%2C+202)
79. J. Xiao, X. Shang, A. Yao, and T. seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proc. of Computer Vision and Pattern Recognition, 2021. [find](https://scholar.google.com/scholar?q=J%2E+Xiao%2C+X%2E+Shang%2C+A%2E+Yao%2C+and+T%2E+seng+Chua%2E+Next%2Dqa%3A+Next+phase+of+question%2Danswering+to+explaining+temporal+actions%2E+In+Proc%2E+of+Computer+Vision+and+Pattern+Recognition%2C+2021%2E)
80. L. Xu, Y. Zhao, D. Zhou, Z. Lin, S. K. Ng, and J. Feng. PLLaVA: Parameter-free LLaVA extension from images to videos for video dense captioning, 2024. [find](https://scholar.google.com/scholar?q=L%2E+Xu%2C+Y%2E+Zhao%2C+D%2E+Zhou%2C+Z%2E+Lin%2C+S%2E+K%2E+Ng%2C+and+J%2E+Feng%2E+PLLaVA%3A+Parameter%2Dfree+LLaVA+extension+from+images+to+videos+for+video+dense+captioning%2C+2024%2E)
81. A. Yang, A. Miech, J. Sivic, I. Laptev, and C. Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proc. of International Conference on Computer Vision, 2021. [find](https://scholar.google.com/scholar?q=A%2E+Yang%2C+A%2E+Miech%2C+J%2E+Sivic%2C+I%2E+Laptev%2C+and+C%2E+Schmid%2E+Just+ask%3A+Learning+to+answer+questions+from+millions+of+narrated+videos%2E+In+Proc%2E+of+International+Conference+on+Computer+Vision%2C+2021%2E)
82. Y. Yao, T. Yu, A. Zhang, C. Wang, J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. He, et al. [Minicpm-v: A gpt-4v level mllm on your phone](https://md.research.elamur.ai/free/en/papers/yao-2024-minicpm-v-gpt-4v/). arXiv preprint arXiv:2408.01800, 2024.
83. K. Yi, C. Gan, Y. Li, P. Kohli, J. Wu, A. Torralba, and J. B. Tenenbaum. Clevrer: Collision events for video representation and reasoning. In Proc. of International Conference on Learning Representations, 2020. [find](https://scholar.google.com/scholar?q=K%2E+Yi%2C+C%2E+Gan%2C+Y%2E+Li%2C+P%2E+Kohli%2C+J%2E+Wu%2C+A%2E+Torralba%2C+and+J%2E+B%2E+Tenenbaum%2E+Clevrer%3A+Collision+events+for+video+representation+and+reasoning%2E+In+Proc%2E+of+International+Conference+on+Learning+Representati)
84. S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen. [A survey on multimodal large language models](https://md.research.elamur.ai/free/en/papers/yin-2023-survey-multimodal-large-language-models/). arXiv preprint arXiv:2306.13549, 2023.
85. W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang. [Mm-vet: Evaluating large multimodal models for integrated capabilities](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/). arXiv preprint arXiv:2308.02490, 2023.
86. W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang. [Mm-vet: Evaluating large multimodal models for integrated capabilities](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/). In Proc. of International Conference on Machine Learning, 2024.
87. X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng, Z. Yang, Y. Liu, W. Huang, H. Sun, Y. Su, and W. Chen. [Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/). In Pro…
88. C. Zhang, F. Gao, B. Jia, Y. Zhu, and S.-C. Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. [find](https://scholar.google.com/scholar?q=C%2E+Zhang%2C+F%2E+Gao%2C+B%2E+Jia%2C+Y%2E+Zhu%2C+and+S%2E%2DC%2E+Zhu%2E+Raven%3A+A+dataset+for+relational+and+analogical+visual+reasoning%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR)
89. H. Zhang, X. Li, and L. Bing. [Video-llama: An instruction-tuned audio-visual language model for video understanding](https://md.research.elamur.ai/free/en/papers/zhang-2023-video-llama-instruction-tuned-audio/). arXiv preprint arXiv:2306.02858, 2023.
90. H. Zhang, P. Zhang, X. Hu, Y.-C. Chen, L. Li, X. Dai, L. Wang, L. Yuan, J.-N. Hwang, and J. Gao. Glipv2: Unifying localization and vision-language understanding. Proc. of Neural Information Processing Systems, 35:36067–36080, 2022. [find](https://scholar.google.com/scholar?q=H%2E+Zhang%2C+P%2E+Zhang%2C+X%2E+Hu%2C+Y%2E%2DC%2E+Chen%2C+L%2E+Li%2C+X%2E+Dai%2C+L%2E+Wang%2C+L%2E+Yuan%2C+J%2E%2DN%2E+Hwang%2C+and+J%2E+Gao%2E+Glipv2%3A+Unifying+localization+and+vision%2Dlanguage+understanding%2E+Proc%2E+of+Neural+Information+Processing)
91. T. Zhang, S. Wang, L. Li, G. Zhang, P. Taslakian, S. Rajeswar, J. Fu, B. Liu, and Y. Bengio. Vcr: Visual caption restoration. arXiv preprint arXiv:2406.06462, 2024. — [arXiv:2406.06462](https://arxiv.org/abs/2406.06462) [find](https://scholar.google.com/scholar?q=T%2E+Zhang%2C+S%2E+Wang%2C+L%2E+Li%2C+G%2E+Zhang%2C+P%2E+Taslakian%2C+S%2E+Rajeswar%2C+J%2E+Fu%2C+B%2E+Liu%2C+and+Y%2E+Bengio%2E+Vcr%3A+Visual+caption+restoration%2E+arXiv+preprint+arXiv%3A2406%2E06462%2C+2024%2E+%E2%80%94+%5BarXiv%3A2406%2E06462%5D%28https%3A%2F%2Farxiv%2E)

*Updated 24 October 2026*