---
title: "VideoLLaMA 2 — paper by Cheng et al. (2024)"
description: "Video-LLMs lag behind image-focused multimodal models because they fuse information across… 135 references, cited by 107 papers here."
authors: ["Zesen Cheng", "Sicong Leng", "Hang Zhang", "Yifei Xin", "Xin Li", "Guanzheng Chen", "Yongxin Zhu", "Wenqi Zhang", "Ziyang Luo", "Deli Zhao", "Lidong Bing"]
year: 2024
venue: "arXiv:2406.07476"
source: "https://arxiv.org/abs/2406.07476"
doi: "10.48550/arxiv.2406.07476"
arxiv: "2406.07476"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cheng-2024-videollama-2-advancing-spatial-temporal/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cheng-2024-videollama-2-advancing-spatial-temporal/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs

# VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs

**Authors:** [Zesen Cheng](https://md.research.elamur.ai/free/en/authors/zesen-cheng/), [Sicong Leng](https://md.research.elamur.ai/free/en/authors/sicong-leng/), [Hang Zhang](https://md.research.elamur.ai/free/en/authors/hang-zhang/), Yifei Xin, [Xin Li](https://md.research.elamur.ai/free/en/authors/xin-li/), [Guanzheng Chen](https://md.research.elamur.ai/free/en/authors/guanzheng-chen/), Yongxin Zhu, [Wenqi Zhang](https://md.research.elamur.ai/free/en/authors/wenqi-zhang/), [Ziyang Luo](https://md.research.elamur.ai/free/en/authors/ziyang-luo/), [Deli Zhao](https://md.research.elamur.ai/free/en/authors/deli-zhao/) and [Lidong Bing](https://md.research.elamur.ai/free/en/authors/lidong-bing/) · **Year:** 2024 · **Venue:** arXiv:2406.07476 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2406.07476) · [PDF](https://arxiv.org/pdf/2406.07476) · [DOI 10.48550/arxiv.2406.07476](https://doi.org/10.48550/arxiv.2406.07476) · [arXiv 2406.07476](https://arxiv.org/abs/2406.07476) · [OpenAlex](https://openalex.org/W4399655895)

**Citations:** 11 citations (OpenAlex, 2026-10-14) · [107](https://md.research.elamur.ai/free/en/papers/cheng-2024-videollama-2-advancing-spatial-temporal/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Video-LLMs lag behind image-focused multimodal models because they fuse information across frames poorly and usually ignore the soundtrack. VideoLLaMA 2, from DAMO Academy (Alibaba Group), tackles both weaknesses while keeping the modular two-branch layout of the original VideoLLaMA.

**Vision branch.** A fixed number of frames per video is encoded one by one with CLIP (ViT-L/14). The features then go through a new Spatial-Temporal Convolution (STC) connector, which replaces the earlier Q-former. In it, RegStage convolution blocks surround a 3D convolution that reduces the token count while keeping tokens in spatial-temporal order.

**Audio branch.** BEATs encodes the soundtrack, and a small MLP maps those features into the LLM space. This branch is trained jointly with the rest of the model.

Mistral-Instruct, Mixtral-Instruct and Qwen2-Instruct serve as language decoders. On multiple-choice video QA, open-ended video QA and video captioning, the authors report results that rival other open-source systems and approach some commercial ones. They also report gains on audio-only and audio-visual QA. All checkpoints are public, giving a reusable base for video and audio-visual research.

## Key points
- Visual pipeline: CLIP ViT-L/14 encoder with frames padded and resized to 336×336. The STC connector applies RegStage, then 3D downsampling, then RegStage. Pooling and convolution are used instead of resamplers so that token order is preserved.
- Audio pipeline: audio is converted to fbank spectrograms with 128 frequency bins, encoded by BEATs, and projected by a two-layer MLP. Cross-modal fusion happens only inside the LLM.
- Connector ablation (8 frames, trained on Video-LLaVA data): RegStage combined with 3D Conv (2,2,2) gave the best average of 45.1. Its scores were 45.5 on MV-Bench, 42.2 on EgoSchema and 47.6 on ActivityNet-QA. The best variant without RegStage, 3D Pool (2,2,2), averaged 44.6.
- Evaluation covers MC-VQA, OE-VQA, video captioning, AQA and OE-AVQA. The authors claim the model outperforms open-source models of similar size on multiple benchmarks.
- Code and models are released at github.com/DAMO-NLP-SG/VideoLLaMA2.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) — Company, Hangzhou, China · 201 works · 17,173 citations
- [Alibaba DAMO Academy](https://md.research.elamur.ai/free/en/companies/alibaba-damo-academy/) — Company, Hangzhou, China · 45 works · 6,874 citations
- [Alibaba Group (Cayman Islands)](https://md.research.elamur.ai/free/en/companies/alibaba-group-cayman-islands/) — Company, George Town, Cayman Islands · 36 works · 3,645 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 519 works

## Cite

Cite as: Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao and Lidong Bing. 2024. VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs. arXiv:2406.07476.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{cheng2024videollama,
  title = {{VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs}},
  author = {Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao and Lidong Bing},
  year = {2024},
  doi = {10.48550/arxiv.2406.07476},
  eprint = {2406.07476},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.07476}
}
```

- [← previous MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/juraska-2024-metricx-24-google-submission-wmt/)
- [next → Agentless: Demystifying LLM-based Software Engineering Agents](https://md.research.elamur.ai/free/en/papers/xia-2024-agentless-demystifying-llm-based/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

135 references, 52 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325, 2023. — [arXiv:2301.11325](https://arxiv.org/abs/2301.11325) [find](https://scholar.google.com/scholar?q=Andrea+Agostinelli%2C+Timo+I+Denk%2C+Zal%C3%A1n+Borsos%2C+Jesse+Engel%2C+Mauro+Verzetti%2C+Antoine+Caillon%2C+Qingqing+Huang%2C+Aren+Jansen%2C+Adam+Roberts%2C+Marco+Tagliasacchi%2C+et+al%2E+Musiclm%3A+Generating+music+from+text%2E+)
2. Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K Marks, Chiori Hori, Peter Anderson, et al. Audio visual scene-aware dialog. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7558–7567, 2019. [find](https://scholar.google.com/scholar?q=Huda+Alamri%2C+Vincent+Cartillier%2C+Abhishek+Das%2C+Jue+Wang%2C+Anoop+Cherian%2C+Irfan+Essa%2C+Dhruv+Batra%2C+Tim+K+Marks%2C+Chiori+Hori%2C+Peter+Anderson%2C+et+al%2E+Audio+visual+scene%2Daware+dialog%2E+In+Proceedings+of+the)
3. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock,… — [Flamingo: a Visual Language Model for Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
4. Anthropic. The claude 3 model family: Opus, sonnet, haiku. 2024. [find](https://scholar.google.com/scholar?q=Anthropic%2E+The+claude+3+model+family%3A+Opus%2C+sonnet%2C+haiku%2E+2024%2E)
5. Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny. Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens. arXiv preprint arXiv:2404.03413, 2024. — [arXiv:2404.03413](https://arxiv.org/abs/2404.03413) [find](https://scholar.google.com/scholar?q=Kirolos+Ataallah%2C+Xiaoqian+Shen%2C+Eslam+Abdelrahman%2C+Essam+Sleiman%2C+Deyao+Zhu%2C+Jian+Ding%2C+and+Mohamed+Elhoseiny%2E+Minigpt4%2Dvideo%3A+Advancing+multimodal+llms+for+video+understanding+with+interleaved+visua)
6. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.[12966, 2023](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/).
7. Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021. [find](https://scholar.google.com/scholar?q=Max+Bain%2C+Arsha+Nagrani%2C+G%C3%BCl+Varol%2C+and+Andrew+Zisserman%2E+Frozen+in+time%3A+A+joint+video+and+image+encoder+for+end%2Dto%2Dend+retrieval%2E+In+IEEE+International+Conference+on+Computer+Vision%2C+2021%2E)
8. Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL [https://openai.com/research/video-generation-models-as-world-simulators](https://openai.com/research/video-generation-models-as-world-simulators). [↗](https://openai.com/research/video-generation-models-as-world-simulators)
9. Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. [Emerging properties in self-supervised vision transformers](https://md.research.elamur.ai/free/en/papers/caron-2021-emerging-properties-self-supervised-vision/). In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650–9660, 2021.
10. Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal llm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. [find](https://scholar.google.com/scholar?q=Junbum+Cha%2C+Wooyoung+Kang%2C+Jonghwan+Mun%2C+and+Byungseok+Roh%2E+Honeybee%3A+Locality%2Denhanced+projector+for+multimodal+llm%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recogniti)
11. Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3558–3568, 2021. [find](https://scholar.google.com/scholar?q=Soravit+Changpinyo%2C+Piyush+Sharma%2C+Nan+Ding%2C+and+Radu+Soricut%2E+Conceptual+12m%3A+Pushing+web%2Dscale+image%2Dtext+pre%2Dtraining+to+recognize+long%2Dtail+visual+concepts%2E+In+Proceedings+of+the+IEEE%2FCVF+conferen)
12. David Chen and William B Dolan. [Collecting highly parallel data for paraphrase evaluation](https://md.research.elamur.ai/free/en/papers/chen-2011-collecting-highly-parallel-data-paraphrase/). In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pp. 190–200, 2011.
13. Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for a lite vision-language model, 2024a. [find](https://scholar.google.com/scholar?q=Guiming+Hardy+Chen%2C+Shunian+Chen%2C+Ruifei+Zhang%2C+Junying+Chen%2C+Xiangbo+Wu%2C+Zhiyi+Zhang%2C+Zhihong+Chen%2C+Jianquan+Li%2C+Xiang+Wan%2C+and+Benyou+Wang%2E+Allava%3A+Harnessing+gpt4v%2Dsynthesized+data+for+a+lite+visio)
14. Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721–725. [find](https://scholar.google.com/scholar?q=Honglie+Chen%2C+Weidi+Xie%2C+Andrea+Vedaldi%2C+and+Andrew+Zisserman%2E+Vggsound%3A+A+large%2Dscale+audio%2Dvisual+dataset%2E+In+ICASSP+2020%2D2020+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processin)
15. Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.[15195, 2023a](https://md.research.elamur.ai/free/en/papers/chen-2023-shikra-unleashing-multimodal-llm-s/).
16. Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. [Sharegpt4v: Improving large multi-modal models with better captions](https://md.research.elamur.ai/free/en/papers/chen-2023-sharegpt4v-improving-large-multi-modal/). arXiv preprint arXiv:2311.12793, 2023b.
17. Ling-Hao Chen, Shunlin Lu, Ailing Zeng, Hao Zhang, Benyou Wang, Ruimao Zhang, and Lei Zhang. Motionllm: Understanding human behaviors from human motions and videos. arXiv preprint arXiv:2405.20340, 2024b. — [arXiv:2405.20340](https://arxiv.org/abs/2405.20340) [find](https://scholar.google.com/scholar?q=Ling%2DHao+Chen%2C+Shunlin+Lu%2C+Ailing+Zeng%2C+Hao+Zhang%2C+Benyou+Wang%2C+Ruimao+Zhang%2C+and+Lei+Zhang%2E+Motionllm%3A+Understanding+human+behaviors+from+human+motions+and+videos%2E+arXiv+preprint+arXiv%3A2405%2E20340%2C+20)
18. Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. BEATs: Audio pre-training with acoustic tokenizers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of th… [find](https://scholar.google.com/scholar?q=Sanyuan+Chen%2C+Yu+Wu%2C+Chengyi+Wang%2C+Shujie+Liu%2C+Daniel+Tompkins%2C+Zhuo+Chen%2C+Wanxiang+Che%2C+Xiangzhan+Yu%2C+and+Furu+Wei%2E+BEATs%3A+Audio+pre%2Dtraining+with+acoustic+tokenizers%2E+In+Andreas+Krause%2C+Emma+Brunski)
19. Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. arXiv preprint arXiv:2402.19479, 2024c. — [arXiv:2402.19479](https://arxiv.org/abs/2402.19479) [find](https://scholar.google.com/scholar?q=Tsai%2DShien+Chen%2C+Aliaksandr+Siarohin%2C+Willi+Menapace%2C+Ekaterina+Deyneka%2C+Hsiang%2Dwei+Chao%2C+Byung+Eun+Jeon%2C+Yuwei+Fang%2C+Hsin%2DYing+Lee%2C+Jian+Ren%2C+Ming%2DHsuan+Yang%2C+et+al%2E+Panda%2D70m%3A+Captioning+70m+videos+)
20. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. [Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks](https://md.research.elamur.ai/free/en/papers/chen-2023-internvl-scaling-up-vision/). arXiv preprint arXiv:2312.14238, 2023d.
21. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. [Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models](https://md.research.elamur.ai/free/en/papers/chu-2023-qwen-audio-advancing-universal/). arXiv preprint arXiv:2311.07919, 2023.
22. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. [Qwen2-audio technical report](https://md.research.elamur.ai/free/en/papers/chu-2024-qwen2-audio-technical-report/). arXiv preprint arXiv:2407.10759, 2024.
23. Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. [Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model](https://md.research.elamur.ai/free/en/papers/dong-2024-internlm-xcomposer2-mastering-free/). arXiv preprint arXiv:2401.16420, 2024.
24. Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 736–740. [find](https://scholar.google.com/scholar?q=Konstantinos+Drossos%2C+Samuel+Lipping%2C+and+Tuomas+Virtanen%2E+Clotho%3A+An+audio+captioning+dataset%2E+In+ICASSP+2020%2D2020+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2C+p)
25. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783, 2024.
26. Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206, 2024. — [arXiv:2403.03206](https://arxiv.org/abs/2403.03206) [find](https://scholar.google.com/scholar?q=Patrick+Esser%2C+Sumith+Kulal%2C+Andreas+Blattmann%2C+Rahim+Entezari%2C+Jonas+M%C3%BCller%2C+Harry+Saini%2C+Yam+Levi%2C+Dominik+Lorenz%2C+Axel+Sauer%2C+Frederic+Boesel%2C+et+al%2E+Scaling+rectified+flow+transformers+for+high%2Dre)
27. Chenyou Fan. Egovqa-an egocentric video question answering benchmark dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0–0, 2019. [find](https://scholar.google.com/scholar?q=Chenyou+Fan%2E+Egovqa%2Dan+egocentric+video+question+answering+benchmark+dataset%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Conference+on+Computer+Vision+Workshops%2C+pp%2E+0%E2%80%930%2C+2019%2E)
28. Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-augmented multimodal agent for video understanding. arXiv preprint arXiv:2403.11481, 2024. — [arXiv:2403.11481](https://arxiv.org/abs/2403.11481) [find](https://scholar.google.com/scholar?q=Yue+Fan%2C+Xiaojian+Ma%2C+Rujie+Wu%2C+Yuntao+Du%2C+Jiaqi+Li%2C+Zhi+Gao%2C+and+Qing+Li%2E+Videoagent%3A+A+memory%2Daugmented+multimodal+agent+for+video+understanding%2E+arXiv+preprint+arXiv%3A2403%2E11481%2C+2024%2E+%E2%80%94+%5BarXiv%3A2403)
29. Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. arXiv preprint arXiv:2406.14515, 2024. — [arXiv:2406.14515](https://arxiv.org/abs/2406.14515) [find](https://scholar.google.com/scholar?q=Xinyu+Fang%2C+Kangrui+Mao%2C+Haodong+Duan%2C+Xiangyu+Zhao%2C+Yining+Li%2C+Dahua+Lin%2C+and+Kai+Chen%2E+Mmbench%2Dvideo%3A+A+long%2Dform+multi%2Dshot+benchmark+for+holistic+video+understanding%2E+arXiv+preprint+arXiv%3A2406%2E145)
30. Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. [Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis](https://md.research.elamur.ai/free/en/papers/fu-2024-video-mme-first-ever-comprehensive/). arXiv preprint arXiv:2405.21075, 2024.
31. Thomas Gemma, Teamand Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024. [find](https://scholar.google.com/scholar?q=Thomas+Gemma%2C+Teamand+Mesnard%2C+Cassidy+Hardin%2C+Robert+Dadashi%2C+Surya+Bhupatiraju%2C+Shreya+Pathak%2C+Laurent+Sifre%2C+Morgane+Rivi%C3%A8re%2C+Mihir+Sanjay+Kale%2C+Juliette+Love%2C+et+al%2E+Gemma%3A+Open+models+based+on+ge)
32. Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. [Imagebind: One embedding space to bind them all](https://md.research.elamur.ai/free/en/papers/girdhar-2023-imagebind-one-embedding-space-bind/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180–15190, 2023.
33. Yuan Gong, Jin Yu, and James Glass. Vocalsound: A dataset for improving human vocal sounds recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 151–155. [find](https://scholar.google.com/scholar?q=Yuan+Gong%2C+Jin+Yu%2C+and+James+Glass%2E+Vocalsound%3A+A+dataset+for+improving+human+vocal+sounds+recognition%2E+In+ICASSP+2022%2D2022+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28IC)
34. Gemini Team Google. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
35. Gemini Team Google. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.05530, 2024.
36. Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE i… [find](https://scholar.google.com/scholar?q=Raghav+Goyal%2C+Samira+Ebrahimi+Kahou%2C+Vincent+Michalski%2C+Joanna+Materzynska%2C+Susanne+Westphal%2C+Heuna+Kim%2C+Valentin+Haenel%2C+Ingo+Fruend%2C+Peter+Yianilos%2C+Moritz+Mueller%2DFreitag%2C+et+al%2E+The%22+something+som)
37. Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26584–26595, 2024. [find](https://scholar.google.com/scholar?q=Jiaming+Han%2C+Kaixiong+Gong%2C+Yiyuan+Zhang%2C+Jiaqi+Wang%2C+Kaipeng+Zhang%2C+Dahua+Lin%2C+Yu+Qiao%2C+Peng+Gao%2C+and+Xiangyu+Yue%2E+Onellm%3A+One+framework+to+align+all+modalities+with+language%2E+In+Proceedings+of+the+I)
38. Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. arXiv preprint arXiv:2404.05726, 2024. [find](https://scholar.google.com/scholar?q=Bo+He%2C+Hengduo+Li%2C+Young+Kyun+Jang%2C+Menglin+Jia%2C+Xuefei+Cao%2C+Ashish+Shah%2C+Abhinav+Shrivastava%2C+and+Ser%2DNam+Lim%2E+Ma%2Dlmm%3A+Memory%2Daugmented+large+multimodal+model+for+long%2Dterm+video+understanding%2E+arXiv)
39. Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower llm to grasp video moments. arXiv preprint arXiv:2311.18445, 2(3):9, 2023. — [arXiv:2311.18445](https://arxiv.org/abs/2311.18445) [find](https://scholar.google.com/scholar?q=Bin+Huang%2C+Xin+Wang%2C+Hong+Chen%2C+Zihan+Song%2C+and+Wenwu+Zhu%2E+Vtimellm%3A+Empower+llm+to+grasp+video+moments%2E+arXiv+preprint+arXiv%3A2311%2E18445%2C+2%283%29%3A9%2C+2023%2E+%E2%80%94+%5BarXiv%3A2311%2E18445%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E)
40. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+Mistral+7b)
41. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+et+al%2E+Mixtral+of+expe)
42. Bu Jin, Xinyu Liu, Yupeng Zheng, Pengfei Li, Hao Zhao, Tong Zhang, Yuhang Zheng, Guyue Zhou, and Jingjing Liu. Adapt: Action-aware driving caption transformer. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 7554–7561. [find](https://scholar.google.com/scholar?q=Bu+Jin%2C+Xinyu+Liu%2C+Yupeng+Zheng%2C+Pengfei+Li%2C+Hao+Zhao%2C+Tong+Zhang%2C+Yuhang+Zheng%2C+Guyue+Zhou%2C+and+Jingjing+Liu%2E+Adapt%3A+Action%2Daware+driving+caption+transformer%2E+In+2023+IEEE+International+Conference+on)
43. Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. arXiv preprint arXiv:2311.08046, 2023b. — [arXiv:2311.08046](https://arxiv.org/abs/2311.08046) [find](https://scholar.google.com/scholar?q=Peng+Jin%2C+Ryuichi+Takanobu%2C+Caiwan+Zhang%2C+Xiaochun+Cao%2C+and+Li+Yuan%2E+Chat%2Dunivi%3A+Unified+visual+representation+empowers+large+language+models+with+image+and+video+understanding%2E+arXiv+preprint+arXiv%3A2)
44. Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Bin Chen, Chenyi Lei, An Liu, Chengru Song, Xiaoqiang Lei, et al. Unified language-vision pretraining with dynamic discrete visual tokenization. arXiv preprint arXiv:2309.04669, 2023c. — [arXiv:2309.04669](https://arxiv.org/abs/2309.04669) [find](https://scholar.google.com/scholar?q=Yang+Jin%2C+Kun+Xu%2C+Liwei+Chen%2C+Chao+Liao%2C+Jianchao+Tan%2C+Bin+Chen%2C+Chenyi+Lei%2C+An+Liu%2C+Chengru+Song%2C+Xiaoqiang+Lei%2C+et+al%2E+Unified+language%2Dvision+pretraining+with+dynamic+discrete+visual+tokenization%2E+)
45. Yang Jin, Zhicheng Sun, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, et al. Video-lavit: Unified video-language pre-training with decoupled visual-motional tokenization. arXiv preprint arXiv:2402.03161, 2024. — [arXiv:2402.03161](https://arxiv.org/abs/2402.03161) [find](https://scholar.google.com/scholar?q=Yang+Jin%2C+Zhicheng+Sun%2C+Kun+Xu%2C+Liwei+Chen%2C+Hao+Jiang%2C+Quzhe+Huang%2C+Chengru+Song%2C+Yuliang+Liu%2C+Di+Zhang%2C+Yang+Song%2C+et+al%2E+Video%2Dlavit%3A+Unified+video%2Dlanguage+pre%2Dtraining+with+decoupled+visual%2Dmotion)
46. Raehyuk Jung, Hyojun Go, Jaehyuk Yi, Jiho Jang, Daniel Kim, Jay Suh, Aiden Lee, Cooper Han, Jae Lee, Jeff Kim, et al. Pegasus-v1 technical report. arXiv preprint arXiv:2404.14687, 2024. — [arXiv:2404.14687](https://arxiv.org/abs/2404.14687) [find](https://scholar.google.com/scholar?q=Raehyuk+Jung%2C+Hyojun+Go%2C+Jaehyuk+Yi%2C+Jiho+Jang%2C+Daniel+Kim%2C+Jay+Suh%2C+Aiden+Lee%2C+Cooper+Han%2C+Jae+Lee%2C+Jeff+Kim%2C+et+al%2E+Pegasus%2Dv1+technical+report%2E+arXiv+preprint+arXiv%3A2404%2E14687%2C+2024%2E+%E2%80%94+%5BarXiv%3A2404%2E)
47. Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017. [find](https://scholar.google.com/scholar?q=Will+Kay%2C+Joao+Carreira%2C+Karen+Simonyan%2C+Brian+Zhang%2C+Chloe+Hillier%2C+Sudheendra+Vijayanarasimhan%2C+Fabio+Viola%2C+Tim+Green%2C+Trevor+Back%2C+Paul+Natsev%2C+et+al%2E+The+kinetics+human+action+video+dataset%2E+arXi)
48. Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. [Audiocaps: Generating captions for audios in the wild](https://md.research.elamur.ai/free/en/papers/kim-2019-audiocaps-generating-captions-audios-wild/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers),…
49. Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125, 2023. — [arXiv:2312.14125](https://arxiv.org/abs/2312.14125) [find](https://scholar.google.com/scholar?q=Dan+Kondratyuk%2C+Lijun+Yu%2C+Xiuye+Gu%2C+Jos%C3%A9+Lezama%2C+Jonathan+Huang%2C+Rachel+Hornung%2C+Hartwig+Adam%2C+Hassan+Akbari%2C+Yair+Alon%2C+Vighnesh+Birodkar%2C+et+al%2E+Videopoet%3A+A+large+language+model+for+zero%2Dshot+video)
50. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. [Llava-onevision: Easy visual task transfer](https://md.research.elamur.ai/free/en/papers/li-2024-llava-onevision-easy-visual-task/). arXiv preprint arXiv:2408.03326, 2024a.
51. Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. [Llava-med: Training a large language-and-vision assistant for biomedicine in one day](https://md.research.elamur.ai/free/en/papers/li-2023-llava-med-training-large-language/). Advances in Neural Information Processing Systems, 36, 2024b.
52. Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19108–19118, June 2022a. [find](https://scholar.google.com/scholar?q=Guangyao+Li%2C+Yake+Wei%2C+Yapeng+Tian%2C+Chenliang+Xu%2C+Ji%2DRong+Wen%2C+and+Di+Hu%2E+Learning+to+answer+questions+in+dynamic+audio%2Dvisual+scenarios%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+a)
53. Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19108–19118, 2022b. [find](https://scholar.google.com/scholar?q=Guangyao+Li%2C+Yake+Wei%2C+Yapeng+Tian%2C+Chenliang+Xu%2C+Ji%2DRong+Wen%2C+and+Di+Hu%2E+Learning+to+answer+questions+in+dynamic+audio%2Dvisual+scenarios%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+a)
54. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. [Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). arXiv preprint arXiv:2301.12597, 2023a.
55. KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. [Videochat: Chat-centric video understanding](https://md.research.elamur.ai/free/en/papers/li-2023-videochat-chat-centric-video-understanding/). arXiv preprint arXiv:2305.06355, 2023b.
56. Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. [Mvbench: A comprehensive multi-modal video understanding benchmark](https://md.research.elamur.ai/free/en/papers/li-2023-mvbench-comprehensive-multi-modal-video/). arXiv preprint arXiv:2311.17005, 2023c.
57. Yanwei Li, Chengyao Wang, and Jiaya Jia. [Llama-vid: An image is worth 2 tokens in large language models](https://md.research.elamur.ai/free/en/papers/li-2023-llama-vid-image-worth-2/). arXiv preprint arXiv:2311.17043, 2023d.
58. Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. Tgif: A new dataset and benchmark on animated gif description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4641–4650, 2016. [find](https://scholar.google.com/scholar?q=Yuncheng+Li%2C+Yale+Song%2C+Liangliang+Cao%2C+Joel+Tetreault%2C+Larry+Goldberg%2C+Alejandro+Jaimes%2C+and+Jiebo+Luo%2E+Tgif%3A+A+new+dataset+and+benchmark+on+animated+gif+description%2E+In+Proceedings+of+the+IEEE+Confe)
59. Yunxin Li, Xinyu Chen, Baotian Hu, Longyue Wang, Haoyuan Shi, and Min Zhang. Videovista: A versatile benchmark for video understanding and reasoning, 2024c. [find](https://scholar.google.com/scholar?q=Yunxin+Li%2C+Xinyu+Chen%2C+Baotian+Hu%2C+Longyue+Wang%2C+Haoyuan+Shi%2C+and+Min+Zhang%2E+Videovista%3A+A+versatile+benchmark+for+video+understanding+and+reasoning%2C+2024c%2E)
60. Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. [Video-llava: Learning united visual representation by alignment before projection](https://md.research.elamur.ai/free/en/papers/lin-2023-video-llava-learning-united-visual/). arXiv preprint arXiv:2311.10122, 2023a.
61. Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. [Vila: On pre-training for visual language models](https://md.research.elamur.ai/free/en/papers/lin-2023-vila-pre-training-visual-language/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26689–26699, 2024.
62. Kevin Lin, Faisal Ahmed, Linjie Li, Chung-Ching Lin, Ehsan Azarnasab, Zhengyuan Yang, Jianfeng Wang, Lin Liang, Zicheng Liu, Yumao Lu, et al. Mm-vid: Advancing video understanding with gpt-4v (ision). arXiv preprint arXiv:2310.19773, 2023b. — [arXiv:2310.19773](https://arxiv.org/abs/2310.19773) [find](https://scholar.google.com/scholar?q=Kevin+Lin%2C+Faisal+Ahmed%2C+Linjie+Li%2C+Chung%2DChing+Lin%2C+Ehsan+Azarnasab%2C+Zhengyuan+Yang%2C+Jianfeng+Wang%2C+Lin+Liang%2C+Zicheng+Liu%2C+Yumao+Lu%2C+et+al%2E+Mm%2Dvid%3A+Advancing+video+understanding+with+gpt%2D4v+%28ision%29%2E)
63. Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575, 2023c. — [arXiv:2311.07575](https://arxiv.org/abs/2311.07575) [find](https://scholar.google.com/scholar?q=Ziyi+Lin%2C+Chris+Liu%2C+Renrui+Zhang%2C+Peng+Gao%2C+Longtian+Qiu%2C+Han+Xiao%2C+Han+Qiu%2C+Chen+Lin%2C+Wenqi+Shao%2C+Keqin+Chen%2C+et+al%2E+Sphinx%3A+The+joint+mixing+of+weights%2C+tasks%2C+and+visual+embeddings+for+multi%2Dmodal)
64. Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. Clotho-aqa: A crowdsourced dataset for audio question answering. In 2022 30th European Signal Processing Conference (EUSIPCO), pp. 1140–1144, 2022a. doi: 10.23919/EUSIPCO55093.2022.9909680. — [doi:10.23919/eusipco55093.2022.9909680](https://doi.org/10.23919/eusipco55093.2022.9909680) [find](https://scholar.google.com/scholar?q=Samuel+Lipping%2C+Parthasaarathy+Sudarsanam%2C+Konstantinos+Drossos%2C+and+Tuomas+Virtanen%2E+Clotho%2Daqa%3A+A+crowdsourced+dataset+for+audio+question+answering%2E+In+2022+30th+European+Signal+Processing+Conferenc)
65. Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. Clotho-aqa: A crowdsourced dataset for audio question answering. In 2022 30th European Signal Processing Conference (EUSIPCO), pp. 1140–1144. [find](https://scholar.google.com/scholar?q=Samuel+Lipping%2C+Parthasaarathy+Sudarsanam%2C+Konstantinos+Drossos%2C+and+Tuomas+Virtanen%2E+Clotho%2Daqa%3A+A+crowdsourced+dataset+for+audio+question+answering%2E+In+2022+30th+European+Signal+Processing+Conferenc)
66. Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint arXiv:2402.[08268, 2024a](https://md.research.elamur.ai/free/en/papers/liu-2024-world-model-million-length-video/).
67. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/). arXiv preprint arXiv:2310.03744, 2023a.
68. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). arXiv preprint arXiv:2304.08485, 2023b.
69. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL [https://llava-vl.github.io/blog/2024-01-30-llava-next/](https://llava-vl.github.io/blog/2024-01-30-llava-next/). [↗](https://llava-vl.github.io/blog/2024-01-30-llava-next/)
70. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). Advances in neural information processing systems, 36, 2024c.
71. Jinyi Liu, Yifu Yuan, Jianye Hao, Fei Ni, Lingzhi Fu, Yibin Chen, and Yan Zheng. Enhancing robotic manipulation with ai feedback from multimodal large language models. arXiv preprint arXiv:2402.14245, 2024d. — [arXiv:2402.14245](https://arxiv.org/abs/2402.14245) [find](https://scholar.google.com/scholar?q=Jinyi+Liu%2C+Yifu+Yuan%2C+Jianye+Hao%2C+Fei+Ni%2C+Lingzhi+Fu%2C+Yibin+Chen%2C+and+Yan+Zheng%2E+Enhancing+robotic+manipulation+with+ai+feedback+from+multimodal+large+language+models%2E+arXiv+preprint+arXiv%3A2402%2E14245%2C)
72. Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners. arXiv preprint arXiv:2404.00308, 2024e. — [arXiv:2404.00308](https://arxiv.org/abs/2404.00308) [find](https://scholar.google.com/scholar?q=Ruyang+Liu%2C+Chen+Li%2C+Haoran+Tang%2C+Yixiao+Ge%2C+Ying+Shan%2C+and+Ge+Li%2E+St%2Dllm%3A+Large+language+models+are+effective+temporal+learners%2E+arXiv+preprint+arXiv%3A2404%2E00308%2C+2024e%2E+%E2%80%94+%5BarXiv%3A2404%2E00308%5D%28https%3A%2F%2Fa)
73. Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07207, 2023. — [arXiv:2306.07207](https://arxiv.org/abs/2306.07207) [find](https://scholar.google.com/scholar?q=Ruipu+Luo%2C+Ziwang+Zhao%2C+Min+Yang%2C+Junwei+Dong%2C+Minghui+Qiu%2C+Pengcheng+Lu%2C+Tao+Wang%2C+and+Zhongyu+Wei%2E+Valley%3A+Video+assistant+with+large+language+model+enhanced+ability%2E+arXiv+preprint+arXiv%3A2306%2E07207)
74. Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. [Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration](https://md.research.elamur.ai/free/en/papers/lyu-2023-macaw-llm-multi-modal-language/). arXiv preprint arXiv:2306.09093, 2023.
75. Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. Vista-llama: Reliable video narrator via equal distance to visual tokens. arXiv preprint arXiv:2312.08870, 2023. — [arXiv:2312.08870](https://arxiv.org/abs/2312.08870) [find](https://scholar.google.com/scholar?q=Fan+Ma%2C+Xiaojie+Jin%2C+Heng+Wang%2C+Yuchen+Xian%2C+Jiashi+Feng%2C+and+Yi+Yang%2E+Vista%2Dllama%3A+Reliable+video+narrator+via+equal+distance+to+visual+tokens%2E+arXiv+preprint+arXiv%3A2312%2E08870%2C+2023%2E+%E2%80%94+%5BarXiv%3A2312%2E08)
76. Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. [Video-chatgpt: Towards detailed video understanding via large vision and language models](https://md.research.elamur.ai/free/en/papers/maaz-2023-video-chatgpt-towards-detailed-video/). arXiv preprint arXiv:2306.05424, 2023a.
77. Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. [Video-chatgpt: Towards detailed video understanding via large vision and language models](https://md.research.elamur.ai/free/en/papers/maaz-2023-video-chatgpt-towards-detailed-video/). arXiv preprint arXiv:2306.05424, 2023b.
78. Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. [Egoschema: A diagnostic benchmark for very long-form video language understanding](https://md.research.elamur.ai/free/en/papers/mangalam-2023-egoschema-diagnostic-benchmark-very-long/). Advances in Neural Information Processing Systems, 36, 2024.
79. Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. arXiv preprint arXiv:2303.17395, 2023. [find](https://scholar.google.com/scholar?q=Xinhao+Mei%2C+Chutong+Meng%2C+Haohe+Liu%2C+Qiuqiang+Kong%2C+Tom+Ko%2C+Chengqi+Zhao%2C+Mark+D+Plumbley%2C+Yuexian+Zou%2C+and+Wenwu+Wang%2E+Wavcaps%3A+A+chatgpt%2Dassisted+weakly%2Dlabelled+audio+captioning+dataset+for+audio%2Dl)
80. Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, et al. Snap video: Scaled spatiotemporal transformers for text-to-video synthesis. arXiv preprint arXiv:2402.14797, 2024. — [arXiv:2402.14797](https://arxiv.org/abs/2402.14797) [find](https://scholar.google.com/scholar?q=Willi+Menapace%2C+Aliaksandr+Siarohin%2C+Ivan+Skorokhodov%2C+Ekaterina+Deyneka%2C+Tsai%2DShien+Chen%2C+Anil+Kag%2C+Yuwei+Fang%2C+Aleksei+Stoliar%2C+Elisa+Ricci%2C+Jian+Ren%2C+et+al%2E+Snap+video%3A+Scaled+spatiotemporal+transf)
81. Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen. Tut database for acoustic scene classification and sound event detection. In 2016 24th European Signal Processing Conference (EUSIPCO), pp. 1128–1132, 2016a. doi: 10.1109/EUSIPCO.2016.7760424. — [doi:10.1109/eusipco.2016.7760424](https://doi.org/10.1109/eusipco.2016.7760424) [find](https://scholar.google.com/scholar?q=Annamaria+Mesaros%2C+Toni+Heittola%2C+and+Tuomas+Virtanen%2E+Tut+database+for+acoustic+scene+classification+and+sound+event+detection%2E+In+2016+24th+European+Signal+Processing+Conference+%28EUSIPCO%29%2C+pp%2E+1128%E2%80%93)
82. Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen. TUT database for acoustic scene classification and sound event detection. In 24th European Signal Processing Conference 2016 (EUSIPCO 2016), Budapest, Hungary, 2016b. [find](https://scholar.google.com/scholar?q=Annamaria+Mesaros%2C+Toni+Heittola%2C+and+Tuomas+Virtanen%2E+TUT+database+for+acoustic+scene+classification+and+sound+event+detection%2E+In+24th+European+Signal+Processing+Conference+2016+%28EUSIPCO+2016%29%2C+Buda)
83. Annamaria Mesaros, Toni Heittola, Aleksandr Diment, Benjamin Elizalde, Ankit Shah, Emmanuel Vincent, Bhiksha Raj, and Tuomas Virtanen. Dcase 2017 challenge setup: Tasks, datasets and baseline system. In DCASE 2017-Workshop on Detection and Classification of Acoustic Scenes and Events, 2017. [find](https://scholar.google.com/scholar?q=Annamaria+Mesaros%2C+Toni+Heittola%2C+Aleksandr+Diment%2C+Benjamin+Elizalde%2C+Ankit+Shah%2C+Emmanuel+Vincent%2C+Bhiksha+Raj%2C+and+Tuomas+Virtanen%2E+Dcase+2017+challenge+setup%3A+Tasks%2C+datasets+and+baseline+system%2E+)
84. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023a.
85. OpenAI. Gpt-4v(ision) system card, 2023b. URL [https://openai.com/research/gpt-4v-system-card](https://openai.com/research/gpt-4v-system-card). [↗](https://openai.com/research/gpt-4v-system-card)
86. OpenAI. [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/), 2024. hello-gpt-4o/.
87. Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning. arXiv preprint arXiv:2311.18799, 2… — [arXiv:2311.18799](https://arxiv.org/abs/2311.18799) [find](https://scholar.google.com/scholar?q=Artemis+Panagopoulou%2C+Le+Xue%2C+Ning+Yu%2C+Junnan+Li%2C+Dongxu+Li%2C+Shafiq+Joty%2C+Ran+Xu%2C+Silvio+Savarese%2C+Caiming+Xiong%2C+and+Juan+Carlos+Niebles%2E+X%2Dinstructblip%3A+A+framework+for+aligning+x%2Dmodal+instruction%2D)
88. Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Viorica+Patraucean%2C+Lucas+Smaira%2C+Ankush+Gupta%2C+Adria+Recasens%2C+Larisa+Markeeva%2C+Dylan+Banarse%2C+Skanda+Koppula%2C+Mateusz+Malinowski%2C+Yi+Yang%2C+Carl+Doersch%2C+et+al%2E+Perception+test%3A+A+diagnostic+benchmar)
89. Karol J Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pp. 1015–1018, 2015. [find](https://scholar.google.com/scholar?q=Karol+J+Piczak%2E+Esc%3A+Dataset+for+environmental+sound+classification%2E+In+Proceedings+of+the+23rd+ACM+international+conference+on+Multimedia%2C+pp%2E+1015%E2%80%931018%2C+2015%2E)
90. Team Qwen. Qwen2 technical report. 2024. [find](https://scholar.google.com/scholar?q=Team+Qwen%2E+Qwen2+technical+report%2E+2024%2E)
91. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In International conference on machine learning, pp. 8748–8763, 2021.
92. Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. [Designing network design spaces](https://md.research.elamur.ai/free/en/papers/radosavovic-2020-designing-network-design-spaces/). In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10428–10436, 2020.
93. URL [https://openai.com/index/Santhosh](https://openai.com/index/Santhosh) K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, et al. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. arXiv pr… — [arXiv:2109.08238](https://arxiv.org/abs/2109.08238) [↗](https://openai.com/index/Santhosh)
94. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. [Zero-shot text-to-image generation](https://md.research.elamur.ai/free/en/papers/ramesh-2021-zero-shot-text-image-generation/). In International conference on machine learning, pp. 8821–8831. Pmlr, 2021.
95. Reka team Reka. [Reka core, flash, and edge: A series of powerful multimodal language models](https://md.research.elamur.ai/free/en/papers/ormazabal-2024-reka-core-flash-edge-series/). arXiv preprint arXiv:2404.12387, 2024.
96. Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. arXiv preprint arXiv:2312.02051, 2023. [find](https://scholar.google.com/scholar?q=Shuhuai+Ren%2C+Linli+Yao%2C+Shicheng+Li%2C+Xu+Sun%2C+and+Lu+Hou%2E+Timechat%3A+A+time%2Dsensitive+multimodal+large+language+model+for+long+video+understanding%2E+arXiv+preprint+arXiv%3A2312%2E02051%2C+2023%2E)
97. Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. [Photorealistic text-to-image diffusion models with deep language understanding](https://md.research.elamur.ai/free/en/papers/saharia-2022-photorealistic-text-image-diffusion-models/). Advances in neural information processing systems, 35…
98. Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM International Conference on Multimedia, pp. 1041–1044. Association for Computing Machinery, 2014. ISBN 9781450330633. [find](https://scholar.google.com/scholar?q=Justin+Salamon%2C+Christopher+Jacoby%2C+and+Juan+Pablo+Bello%2E+A+dataset+and+taxonomy+for+urban+sound+research%2E+In+Proceedings+of+the+22nd+ACM+International+Conference+on+Multimedia%2C+pp%2E+1041%E2%80%931044%2E+Associa)
99. Yihua Shao, Hongyi Cai, Wenxin Long, Weiyi Lang, Zhe Wang, Haoran Wu, Yan Wang, Yang Yang, and Zhen Lei. Accidentblip2: Accident detection with multi-view motionblip2. arXiv preprint arXiv:2404.12149, 2024. — [arXiv:2404.12149](https://arxiv.org/abs/2404.12149) [find](https://scholar.google.com/scholar?q=Yihua+Shao%2C+Hongyi+Cai%2C+Wenxin+Long%2C+Weiyi+Lang%2C+Zhe+Wang%2C+Haoran+Wu%2C+Yan+Wang%2C+Yang+Yang%2C+and+Zhen+Lei%2E+Accidentblip2%3A+Accident+detection+with+multi%2Dview+motionblip2%2E+arXiv+preprint+arXiv%3A2404%2E12149%2C)
100. Fangxun Shu, Lei Zhang, Hao Jiang, and Cihang Xie. Audio-visual llm for video understanding. arXiv preprint arXiv:2312.06720, 2023. — [arXiv:2312.06720](https://arxiv.org/abs/2312.06720) [find](https://scholar.google.com/scholar?q=Fangxun+Shu%2C+Lei+Zhang%2C+Hao+Jiang%2C+and+Cihang+Xie%2E+Audio%2Dvisual+llm+for+video+understanding%2E+arXiv+preprint+arXiv%3A2312%2E06720%2C+2023%2E+%E2%80%94+%5BarXiv%3A2312%2E06720%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2312%2E06720%29)
101. Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token to sparse memory for long video understanding. arXiv preprint arXiv:2307.16449, 2023. [find](https://scholar.google.com/scholar?q=Enxin+Song%2C+Wenhao+Chai%2C+Guanhong+Wang%2C+Yucheng+Zhang%2C+Haoyang+Zhou%2C+Feiyang+Wu%2C+Xun+Guo%2C+Tian+Ye%2C+Yan+Lu%2C+Jenq%2DNeng+Hwang%2C+et+al%2E+Moviechat%3A+From+dense+token+to+sparse+memory+for+long+video+understan)
102. Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. [Pandagpt: One model to instruction-follow them all](https://md.research.elamur.ai/free/en/papers/su-2023-pandagpt-one-model-instruction-follow/). arXiv preprint arXiv:2305.16355, 2023.
103. Yunlong Tang, Daiki Shimada, Jing Bi, and Chenliang Xu. Avicuna: Audio-visual llm with interleaver and context-boundary alignment for temporal referential dialogue. arXiv preprint arXiv:2403.16276, 2024. — [arXiv:2403.16276](https://arxiv.org/abs/2403.16276) [find](https://scholar.google.com/scholar?q=Yunlong+Tang%2C+Daiki+Shimada%2C+Jing+Bi%2C+and+Chenliang+Xu%2E+Avicuna%3A+Audio%2Dvisual+llm+with+interleaver+and+context%2Dboundary+alignment+for+temporal+referential+dialogue%2E+arXiv+preprint+arXiv%3A2403%2E16276%2C+20)
104. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
105. Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, et al. Towards generalist biomedical ai. NEJM AI, 1(3):AIoa2300138, 2024. [find](https://scholar.google.com/scholar?q=Tao+Tu%2C+Shekoofeh+Azizi%2C+Danny+Driess%2C+Mike+Schaekermann%2C+Mohamed+Amin%2C+Pi%2DChuan+Chang%2C+Andrew+Carroll%2C+Charles+Lau%2C+Ryutaro+Tanno%2C+Ira+Ktena%2C+et+al%2E+Towards+generalist+biomedical+ai%2E+NEJM+AI%2C+1%283%29%3AAI)
106. Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero-Soriano. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. arXiv preprint arXiv:2312.08578, 2023. — [arXiv:2312.08578](https://arxiv.org/abs/2312.08578) [find](https://scholar.google.com/scholar?q=Jack+Urbanek%2C+Florian+Bordes%2C+Pietro+Astolfi%2C+Mary+Williamson%2C+Vasu+Sharma%2C+and+Adriana+Romero%2DSoriano%2E+A+picture+is+worth+more+than+77+text+tokens%3A+Evaluating+clip%2Dstyle+models+on+dense+captions%2E+arX)
107. Jiawei Wang, Liping Yuan, and Yuchen Zhang. Tarsier: Recipes for training and evaluating large video description models, 2024a. URL https://arxiv.org/abs/2407.00634. — [arXiv:2407.00634](https://arxiv.org/abs/2407.00634) [↗](https://arxiv.org/abs/2407.00634)
108. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. [Cogvlm: Visual expert for pretrained language models](https://md.research.elamur.ai/free/en/papers/wang-2023-cogvlm-visual-expert-pretrained/). arXiv preprint arXiv:2311.03079, 2023a.
109. Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. arXiv preprint arXiv:2403.10517, 2024b. — [arXiv:2403.10517](https://arxiv.org/abs/2403.10517) [find](https://scholar.google.com/scholar?q=Xiaohan+Wang%2C+Yuhui+Zhang%2C+Orr+Zohar%2C+and+Serena+Yeung%2DLevy%2E+Videoagent%3A+Long%2Dform+video+understanding+with+large+language+model+as+agent%2E+arXiv+preprint+arXiv%3A2403%2E10517%2C+2024b%2E+%E2%80%94+%5BarXiv%3A2403%2E10517%5D%28)
110. Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4581–4591, 2019. [find](https://scholar.google.com/scholar?q=Xin+Wang%2C+Jiawei+Wu%2C+Junkun+Chen%2C+Lei+Li%2C+Yuan%2DFang+Wang%2C+and+William+Yang+Wang%2E+Vatex%3A+A+large%2Dscale%2C+high%2Dquality+multilingual+dataset+for+video%2Dand%2Dlanguage+research%2E+In+Proceedings+of+the+IEEE%2FCVF)
111. Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942, 2023b. — [arXiv:2307.06942](https://arxiv.org/abs/2307.06942) [find](https://scholar.google.com/scholar?q=Yi+Wang%2C+Yinan+He%2C+Yizhuo+Li%2C+Kunchang+Li%2C+Jiashuo+Yu%2C+Xin+Ma%2C+Xinhao+Li%2C+Guo+Chen%2C+Xinyuan+Chen%2C+Yaohui+Wang%2C+et+al%2E+Internvid%3A+A+large%2Dscale+video%2Dtext+dataset+for+multimodal+understanding+and+gener)
112. Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation models for multimodal video understanding. arXiv preprint arXiv:2403.[15377, 2024c](https://md.research.elamur.ai/free/en/papers/wang-2024-internvideo2-scaling-foundation-models/).
113. Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for llm reasoning on long videos. arXiv preprint arXiv:2405.19209, 2024d. [find](https://scholar.google.com/scholar?q=Ziyang+Wang%2C+Shoubin+Yu%2C+Elias+Stengel%2DEskin%2C+Jaehong+Yoon%2C+Feng+Cheng%2C+Gedas+Bertasius%2C+and+Mohit+Bansal%2E+Videotree%3A+Adaptive+tree%2Dbased+video+representation+for+llm+reasoning+on+long+videos%2E+arXiv+p)
114. Qi Wu, Yubo Zhao, Yifan Wang, Yu-Wing Tai, and Chi-Keung Tang. Motionllm: Multimodal motion-language learning with large language models. arXiv preprint arXiv:2405.17013, 2024. — [arXiv:2405.17013](https://arxiv.org/abs/2405.17013) [find](https://scholar.google.com/scholar?q=Qi+Wu%2C+Yubo+Zhao%2C+Yifan+Wang%2C+Yu%2DWing+Tai%2C+and+Chi%2DKeung+Tang%2E+Motionllm%3A+Multimodal+motion%2Dlanguage+learning+with+large+language+models%2E+arXiv+preprint+arXiv%3A2405%2E17013%2C+2024%2E+%E2%80%94+%5BarXiv%3A2405%2E17013%5D%28ht)
115. X.AI. Grok-1.5 vision, 2024. URL [https://x.ai/blog/grok-1.5v](https://x.ai/blog/grok-1.5v). [↗](https://x.ai/blog/grok-1.5v)
116. Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9777–9786, 2021. [find](https://scholar.google.com/scholar?q=Junbin+Xiao%2C+Xindi+Shang%2C+Angela+Yao%2C+and+Tat%2DSeng+Chua%2E+Next%2Dqa%3A+Next+phase+of+question%2Danswering+to+explaining+temporal+actions%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+vision+and+patt)
117. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.[12244, 2023a](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/).
118. Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288–5296, 2016. [find](https://scholar.google.com/scholar?q=Jun+Xu%2C+Tao+Mei%2C+Ting+Yao%2C+and+Yong+Rui%2E+Msr%2Dvtt%3A+A+large+video+description+dataset+for+bridging+video+and+language%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+p)
119. Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024a. — [arXiv:2404.16994](https://arxiv.org/abs/2404.16994) [find](https://scholar.google.com/scholar?q=Lin+Xu%2C+Yilin+Zhao%2C+Daquan+Zhou%2C+Zhijie+Lin%2C+See+Kiong+Ng%2C+and+Jiashi+Feng%2E+Pllava%3A+Parameter%2Dfree+llava+extension+from+images+to+videos+for+video+dense+captioning%2E+arXiv+preprint+arXiv%3A2404%2E16994%2C+20)
120. Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. [Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing](https://md.research.elamur.ai/free/en/papers/xu-2024-magpie-alignment-data-synthesis-scratch/). ArXiv, abs/2406.08464, 2024b. URL [https://api.semanticscholar.org/CorpusID:270391432](https://api.semanticscholar.org/CorpusID:270391432). [↗](https://api.semanticscholar.org/CorpusID:270391432)
121. Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kenneth KY Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. arXiv preprint arXiv:2310.01412, 2023b. — [arXiv:2310.01412](https://arxiv.org/abs/2310.01412) [find](https://scholar.google.com/scholar?q=Zhenhua+Xu%2C+Yujia+Zhang%2C+Enze+Xie%2C+Zhen+Zhao%2C+Yong+Guo%2C+Kenneth+KY+Wong%2C+Zhenguo+Li%2C+and+Hengshuang+Zhao%2E+Drivegpt4%3A+Interpretable+end%2Dto%2Dend+autonomous+driving+via+large+language+model%2E+arXiv+preprin)
122. Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. Clip-vip: Adapting pre-trained image-text model to video-language alignment. In The Eleventh International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=Hongwei+Xue%2C+Yuchong+Sun%2C+Bei+Liu%2C+Jianlong+Fu%2C+Ruihua+Song%2C+Houqiang+Li%2C+and+Jiebo+Luo%2E+Clip%2Dvip%3A+Adapting+pre%2Dtrained+image%2Dtext+model+to+video%2Dlanguage+alignment%2E+In+The+Eleventh+International+Conf)
123. Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1686–1697, 2021. [find](https://scholar.google.com/scholar?q=Antoine+Yang%2C+Antoine+Miech%2C+Josef+Sivic%2C+Ivan+Laptev%2C+and+Cordelia+Schmid%2E+Just+ask%3A+Learning+to+answer+questions+from+millions+of+narrated+videos%2E+In+Proceedings+of+the+IEEE%2FCVF+international+confer)
124. Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. Avqa: A dataset for audio-visual question answering on videos. In Proceedings of the 30th ACM international conference on multimedia, pp. 3480–3491, 2022. [find](https://scholar.google.com/scholar?q=Pinci+Yang%2C+Xin+Wang%2C+Xuguang+Duan%2C+Hong+Chen%2C+Runze+Hou%2C+Cong+Jin%2C+and+Wenwu+Zhu%2E+Avqa%3A+A+dataset+for+audio%2Dvisual+question+answering+on+videos%2E+In+Proceedings+of+the+30th+ACM+international+conferenc)
125. Qilang Ye, Zitong Yu, Rui Shao, Xinyu Xie, Philip Torr, and Xiaochun Cao. Cat: Enhancing multimodal large language model to answer questions in dynamic audio-visual scenarios, 2024a. [find](https://scholar.google.com/scholar?q=Qilang+Ye%2C+Zitong+Yu%2C+Rui+Shao%2C+Xinyu+Xie%2C+Philip+Torr%2C+and+Xiaochun+Cao%2E+Cat%3A+Enhancing+multimodal+large+language+model+to+answer+questions+in+dynamic+audio%2Dvisual+scenarios%2C+2024a%2E)
126. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yi Zhou, Junyan Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qiang Qi, Ji Chao Zhang, and Feiyan Huang. [mplug-owl: Modularization empowers large language models with multimodality](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/). arXiv preprint…
127. Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. [mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl2-revolutionizing-multi-modal/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13040–13051, 2024b.
128. Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442, 2019. [find](https://scholar.google.com/scholar?q=Kexin+Yi%2C+Chuang+Gan%2C+Yunzhu+Li%2C+Pushmeet+Kohli%2C+Jiajun+Wu%2C+Antonio+Torralba%2C+and+Joshua+B+Tenenbaum%2E+Clevrer%3A+Collision+events+for+video+representation+and+reasoning%2E+arXiv+preprint+arXiv%3A1910%2E01442%2C)
129. Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.[04652, 2024](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/).
130. Shoubin Yu, Jaehong Yoon, and Mohit Bansal. Crema: Multimodal compositional video reasoning via efficient modular adaptation and fusion. arXiv preprint arXiv:2402.05889, 2024. [find](https://scholar.google.com/scholar?q=Shoubin+Yu%2C+Jaehong+Yoon%2C+and+Mohit+Bansal%2E+Crema%3A+Multimodal+compositional+video+reasoning+via+efficient+modular+adaptation+and+fusion%2E+arXiv+preprint+arXiv%3A2402%2E05889%2C+2024%2E)
131. Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. [Activitynet-qa: A dataset for understanding complex web videos via question answering](https://md.research.elamur.ai/free/en/papers/yu-2019-activitynet-qa-dataset-understanding-complex/). In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 9127–9134, 2019.
132. Hang Zhang, Xin Li, and Lidong Bing. [Video-llama: An instruction-tuned audio-visual language model for video understanding](https://md.research.elamur.ai/free/en/papers/zhang-2023-video-llama-instruction-tuned-audio/). arXiv preprint arXiv:2306.02858, 2023.
133. Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.[04264, 2024](https://md.research.elamur.ai/free/en/papers/zhou-2024-mlvu-benchmarking-multi-task-long/).
134. Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. arXiv preprint arXiv:2310.01852, 2023a. — [arXiv:2310.01852](https://arxiv.org/abs/2310.01852) [find](https://scholar.google.com/scholar?q=Bin+Zhu%2C+Bin+Lin%2C+Munan+Ning%2C+Yang+Yan%2C+Jiaxi+Cui%2C+HongFa+Wang%2C+Yatian+Pang%2C+Wenhao+Jiang%2C+Junwu+Zhang%2C+Zongwei+Li%2C+et+al%2E+Languagebind%3A+Extending+video%2Dlanguage+pretraining+to+n%2Dmodality+by+language%2D)
135. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. [Minigpt-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/). arXiv preprint arXiv:2304.10592, 2023b.

*Updated 2 November 2026*