---
title: "InternVideo2 — paper by Wang et al. (2024) — summary and ideas"
description: "On the data side, they build a 402M-entry multimodal corpus. 127 references, cited by 34 papers here."
authors: ["Yi Wang", "Kunchang Li", "Xinhao Li", "Jiashuo Yu", "Yinan He", "Chenting Wang", "Guo Chen", "Baoqi Pei", "Ziang Yan", "Rongkun Zheng", "Jilan Xu", "Zun Wang", "Yansong Shi", "Tianxiang Jiang", "Songze Li", "Hongjie Zhang", "Yifei Huang", "Yu Qiao", "Yali Wang", "Limin Wang"]
year: 2024
venue: "arXiv:2403.15377"
source: "https://arxiv.org/abs/2403.15377"
doi: "10.48550/arxiv.2403.15377"
arxiv: "2403.15377"
licence: "CC BY-NC-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2024-internvideo2-scaling-foundation-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2024-internvideo2-scaling-foundation-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/) / InternVideo2: Scaling Foundation Models for Multimodal Video Understanding

# InternVideo2: Scaling Foundation Models for Multimodal Video Understanding

**Authors:** [Yi Wang](https://md.research.elamur.ai/free/en/authors/yi-wang/), [Kunchang Li](https://md.research.elamur.ai/free/en/authors/kunchang-li/), Xinhao Li, Jiashuo Yu, [Yinan He](https://md.research.elamur.ai/free/en/authors/yinan-he/), Chenting Wang, [Guo Chen](https://md.research.elamur.ai/free/en/authors/guo-chen/), Baoqi Pei, Ziang Yan, Rongkun Zheng, [Jilan Xu](https://md.research.elamur.ai/free/en/authors/jilan-xu/), [Zun Wang](https://md.research.elamur.ai/free/en/authors/zun-wang/), Yansong Shi, Tianxiang Jiang, Songze Li, Hongjie Zhang, Yifei Huang, [Yu Qiao](https://md.research.elamur.ai/free/en/authors/yu-qiao/), [Yali Wang](https://md.research.elamur.ai/free/en/authors/yali-wang/) and [Limin Wang](https://md.research.elamur.ai/free/en/authors/limin-wang/) · **Year:** 2024 · **Venue:** [arXiv:2403.15377](https://md.research.elamur.ai/free/en/venues/lecture-notes-in-computer-science/) ([Lecture notes in computer science 2024](https://md.research.elamur.ai/free/en/venues/lecture-notes-in-computer-science/2024/)) · **Publisher:** [Springer Science+Business Media](https://md.research.elamur.ai/free/en/publishers/springer-science-business-media/) · **Licence:** [CC BY-NC-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2403.15377) · [PDF](https://arxiv.org/pdf/2403.15377) · [DOI 10.48550/arxiv.2403.15377](https://doi.org/10.48550/arxiv.2403.15377) · [arXiv 2403.15377](https://arxiv.org/abs/2403.15377) · [OpenAlex](https://openalex.org/W4404724811)

**Citations:** 205 citations (OpenAlex, 2026-10-14) · [34](https://md.research.elamur.ai/free/en/papers/wang-2024-internvideo2-scaling-foundation-models/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
InternVideo2 is a family of video foundation models from OpenGVLab (Shanghai AI Laboratory) with Nanjing University and SIAT-CAS. Its goal is a single, general-purpose video encoder that supports low-level perception, semantic alignment with language and audio, and open-ended reasoning. The authors scale a ViT-based video encoder to 6B parameters and train it in three stages:

1. The encoder learns spatiotemporal structure by matching the visible (unmasked) tokens to features from two teacher models.
2. It is aligned with text, audio and speech encoders through contrastive learning.
3. It is connected to an LLM and further tuned by next-token prediction on video instruction data.

On the data side, they build a 402M-entry multimodal corpus. Its new component, InternVid2, consists of semantically segmented clips whose captions merge separate video, audio and speech descriptions. The model reaches state-of-the-art results on more than 60 video and audio tasks, with particular strength in video dialogue and long-video understanding. This makes it a strong candidate backbone for video search, video QA and multimodal assistants.

## Key points
- **Pretraining corpus (402M entries):**
  - 2M videos
  - 50M video-text pairs from WebVid and InternVid
  - 50M video-audio-speech-text pairs (InternVid2)
  - 300M image-text pairs
- **Stage 1 setup:**
  - Teachers are InternVL-6B for semantics and VideoMAEv2-g for motion.
  - 80% of tokens are masked per frame, and an MSE loss is applied only on visible tokens.
  - Input is 8 sparsely sampled frames with 14×14 patches.
- **Caption quality ablation:** Zero-shot text-to-video retrieval on MSR-VTT rises from 24.7 to 34.8 when scene segmentation (SceneDet/AutoShot) is combined with fused video, audio and speech captions.
- **Stage 3 ablation:** Leaving question text out of the QFormer improves out-of-domain results for VideoChat2, for example Egoschema-full rises from 52.9 to 55.8. NextQA drops slightly (79 to 78.6).
- **Stated limitations:**
  - Fixed input resolution and sampling rate, plus heavy token compression.
  - No new architecture is introduced.
  - Caption keyword analysis shows demographic skews, such as 62% male mentions and 0.04% mentions of seniors.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) — Government body, Beijing, China · 68 works · 22,753 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 44 works · 10,162 citations
- [Nanjing University](https://md.research.elamur.ai/free/en/universities/nanjing-university/) — University, Nanjing, China · 28 works · 7,072 citations
- [Shenzhen Institutes of Advanced Technology](https://md.research.elamur.ai/free/en/labs/shenzhen-institutes-of-advanced-technology/) — Research institute, Shenzhen, China · 12 works · 2,431 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 269 works

## Cite

Cite as: Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Chenting Wang, Guo Chen, Baoqi Pei, Ziang Yan, Rongkun Zheng, Jilan Xu, Zun Wang, Yansong Shi, Tianxiang Jiang, Songze Li, Hongjie Zhang, Yifei Huang, Yu Qiao, Yali Wang and Limin Wang. 2024. InternVideo2: Scaling Foundation Models for Multimodal Video Understanding. arXiv:2403.15377.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wang2024internvideo2,
  title = {{InternVideo2: Scaling Foundation Models for Multimodal Video Understanding}},
  author = {Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Chenting Wang, Guo Chen, Baoqi Pei, Ziang Yan, Rongkun Zheng, Jilan Xu, Zun Wang, Yansong Shi, Tianxiang Jiang, Songze Li, Hongjie Zhang, Yifei Huang, Yu Qiao, Yali Wang and Limin Wang},
  year = {2024},
  doi = {10.48550/arxiv.2403.15377},
  eprint = {2403.15377},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.15377}
}
```

- [← previous Trusting Your Evidence: Hallucinate Less with Context-aware Decoding](https://md.research.elamur.ai/free/en/papers/shi-2024-trusting-your-evidence-hallucinate-less/)
- [next → LaMP: When Large Language Models Meet Personalization](https://md.research.elamur.ai/free/en/papers/salemi-2024-lamp-when-large-language-models/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/)

## References

127 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock,…](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
2. Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In ICCV, 2017. [find](https://scholar.google.com/scholar?q=Lisa+Anne+Hendricks%2C+Oliver+Wang%2C+Eli+Shechtman%2C+Josef+Sivic%2C+Trevor+Darrell%2C+and+Bryan+Russell%2E+Localizing+moments+in+video+with+natural+language%2E+In+ICCV%2C+2017%2E)
3. Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, 2021. [find](https://scholar.google.com/scholar?q=Max+Bain%2C+Arsha+Nagrani%2C+G%C3%BCl+Varol%2C+and+Andrew+Zisserman%2E+Frozen+in+time%3A+A+joint+video+and+image+encoder+for+end%2Dto%2Dend+retrieval%2E+In+ICCV%2C+2021%2E)
4. Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. V-JEPA: Latent video prediction for visual representation learning, 2024. URL [https://openreview.net/forum?id=WFYbBOEOtv](https://openreview.net/forum?id=WFYbBOEOtv). [↗](https://openreview.net/forum?id=WFYbBOEOtv)
5. Swarup Ranjan Behera, Krishna Mohan Injeti, Jaya Sai Kiran Patibandla, Praveen Kumar Pokala, and Balakrishna Reddy Pailla. Aquallm: Audio question answering data generation using large language models. arXiv preprint arXiv:2312.17343, 2023. — [arXiv:2312.17343](https://arxiv.org/abs/2312.17343) [find](https://scholar.google.com/scholar?q=Swarup+Ranjan+Behera%2C+Krishna+Mohan+Injeti%2C+Jaya+Sai+Kiran+Patibandla%2C+Praveen+Kumar+Pokala%2C+and+Balakrishna+Reddy+Pailla%2E+Aquallm%3A+Audio+question+answering+data+generation+using+large+language+models)
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In NeurIPS, 2020.
7. Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. arXiv preprint arXiv:2402.15391, 2024. — [arXiv:2402.15391](https://arxiv.org/abs/2402.15391) [find](https://scholar.google.com/scholar?q=Jake+Bruce%2C+Michael+Dennis%2C+Ashley+Edwards%2C+Jack+Parker%2DHolder%2C+Yuge+Shi%2C+Edward+Hughes%2C+Matthew+Lai%2C+Aditi+Mavalankar%2C+Richie+Steigerwald%2C+Chris+Apps%2C+et+al%2E+Genie%3A+Generative+interactive+environment)
8. Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017. [find](https://scholar.google.com/scholar?q=Joao+Carreira+and+Andrew+Zisserman%2E+Quo+vadis%2C+action+recognition%3F+a+new+model+and+the+kinetics+dataset%2E+In+CVPR%2C+2017%2E)
9. João Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. ArXiv, abs/1808.01340, 2018. — [arXiv:1808.01340](https://arxiv.org/abs/1808.01340) [find](https://scholar.google.com/scholar?q=Jo%C3%A3o+Carreira%2C+Eric+Noland%2C+Andras+Banki%2DHorvath%2C+Chloe+Hillier%2C+and+Andrew+Zisserman%2E+A+short+note+about+kinetics%2D600%2E+ArXiv%2C+abs%2F1808%2E01340%2C+2018%2E+%E2%80%94+%5BarXiv%3A1808%2E01340%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1808%2E013)
10. João Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics-700 human action dataset. ArXiv, abs/1907.06987, 2019. — [arXiv:1907.06987](https://arxiv.org/abs/1907.06987) [find](https://scholar.google.com/scholar?q=Jo%C3%A3o+Carreira%2C+Eric+Noland%2C+Chloe+Hillier%2C+and+Andrew+Zisserman%2E+A+short+note+on+the+kinetics%2D700+human+action+dataset%2E+ArXiv%2C+abs%2F1907%2E06987%2C+2019%2E+%E2%80%94+%5BarXiv%3A1907%2E06987%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1907%2E069)
11. David L Chen and William B Dolan. [Collecting highly parallel data for paraphrase evaluation](https://md.research.elamur.ai/free/en/papers/chen-2011-collecting-highly-parallel-data-paraphrase/). In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 190–200. Association for Computational Linguistics, 2011.
12. Guo Chen, Yin-Dong Zheng, Limin Wang, and Tong Lu. Dcan: improving temporal action detection via dual context aggregation. In AAAI, volume 36, pages 248–257, 2022a. [find](https://scholar.google.com/scholar?q=Guo+Chen%2C+Yin%2DDong+Zheng%2C+Limin+Wang%2C+and+Tong+Lu%2E+Dcan%3A+improving+temporal+action+detection+via+dual+context+aggregation%2E+In+AAAI%2C+volume+36%2C+pages+248%E2%80%93257%2C+2022a%2E)
13. Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprint arXiv:2406.04325, 2024a. — [arXiv:2406.04325](https://arxiv.org/abs/2406.04325) [find](https://scholar.google.com/scholar?q=Lin+Chen%2C+Xilin+Wei%2C+Jinsong+Li%2C+Xiaoyi+Dong%2C+Pan+Zhang%2C+Yuhang+Zang%2C+Zehui+Chen%2C+Haodong+Duan%2C+Bin+Lin%2C+Zhenyu+Tang%2C+et+al%2E+Sharegpt4video%3A+Improving+video+understanding+and+generation+with+better+ca)
14. Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. BEATs: Audio pre-training with acoustic tokenizers. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 5178–5193. PMLR, 23–29 Jul 2023a. [find](https://scholar.google.com/scholar?q=Sanyuan+Chen%2C+Yu+Wu%2C+Chengyi+Wang%2C+Shujie+Liu%2C+Daniel+Tompkins%2C+Zhuo+Chen%2C+Wanxiang+Che%2C+Xiangzhan+Yu%2C+and+Furu+Wei%2E+BEATs%3A+Audio+pre%2Dtraining+with+acoustic+tokenizers%2E+In+ICML%2C+volume+202+of+Proceedi)
15. Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang, and Jing Liu. Valor: Vision-audio-language omni-perception pretraining model and dataset. arXiv preprint arXiv:2304.08345, 2023b. — [arXiv:2304.08345](https://arxiv.org/abs/2304.08345) [find](https://scholar.google.com/scholar?q=Sihan+Chen%2C+Xingjian+He%2C+Longteng+Guo%2C+Xinxin+Zhu%2C+Weining+Wang%2C+Jinhui+Tang%2C+and+Jing+Liu%2E+Valor%3A+Vision%2Daudio%2Dlanguage+omni%2Dperception+pretraining+model+and+dataset%2E+arXiv+preprint+arXiv%3A2304%2E08345%2C)
16. Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset. NeurIPS, 36, 2024b. [find](https://scholar.google.com/scholar?q=Sihan+Chen%2C+Handong+Li%2C+Qunbo+Wang%2C+Zijia+Zhao%2C+Mingzhen+Sun%2C+Xinxin+Zhu%2C+and+Jing+Liu%2E+Vast%3A+A+vision%2Daudio%2Dsubtitle%2Dtext+omni%2Dmodality+foundation+model+and+dataset%2E+NeurIPS%2C+36%2C+2024b%2E)
17. Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.08534, 2022b. — [arXiv:2205.08534](https://arxiv.org/abs/2205.08534) [find](https://scholar.google.com/scholar?q=Zhe+Chen%2C+Yuchen+Duan%2C+Wenhai+Wang%2C+Junjun+He%2C+Tong+Lu%2C+Jifeng+Dai%2C+and+Yu+Qiao%2E+Vision+transformer+adapter+for+dense+predictions%2E+arXiv+preprint+arXiv%3A2205%2E08534%2C+2022b%2E+%E2%80%94+%5BarXiv%3A2205%2E08534%5D%28https%3A%2F%2F)
18. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. [Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks](https://md.research.elamur.ai/free/en/papers/chen-2023-internvl-scaling-up-vision/). arXiv preprint arXiv:2312.14238, 20…
19. Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In NeurIPS, 2021. [find](https://scholar.google.com/scholar?q=Bowen+Cheng%2C+Alexander+G%2E+Schwing%2C+and+Alexander+Kirillov%2E+Per%2Dpixel+classification+is+not+all+you+need+for+semantic+segmentation%2E+In+NeurIPS%2C+2021%2E)
20. Feng Cheng, Xizi Wang, Jie Lei, David J. Crandall, Mohit Bansal, and Gedas Bertasius. Vindlu: A recipe for effective video-and-language pretraining. ArXiv, abs/2212.05051, 2022. — [arXiv:2212.05051](https://arxiv.org/abs/2212.05051) [find](https://scholar.google.com/scholar?q=Feng+Cheng%2C+Xizi+Wang%2C+Jie+Lei%2C+David+J%2E+Crandall%2C+Mohit+Bansal%2C+and+Gedas+Bertasius%2E+Vindlu%3A+A+recipe+for+effective+video%2Dand%2Dlanguage+pretraining%2E+ArXiv%2C+abs%2F2212%2E05051%2C+2022%2E+%E2%80%94+%5BarXiv%3A2212%2E05051%5D%28h)
21. Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. [find](https://scholar.google.com/scholar?q=Zesen+Cheng%2C+Sicong+Leng%2C+Hang+Zhang%2C+Yifei+Xin%2C+Xin+Li%2C+Guanzheng+Chen%2C+Yongxin+Zhu%2C+Wenqi+Zhang%2C+Ziyang+Luo%2C+Deli+Zhao%2C+et+al%2E+Videollama+2%3A+Advancing+spatial%2Dtemporal+modeling+and+audio+understandi)
22. Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. [Reproducible scaling laws for contrastive language-image learning](https://md.research.elamur.ai/free/en/papers/cherti-2022-reproducible-scaling-laws-contrastive-language/). In CVPR, pages 2818–2829, 2023.
23. [Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenze…](https://md.research.elamur.ai/free/en/papers/communication-2023-seamlessm4t-massively-multilingual-multimodal-machine/)
24. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. [Instructblip: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/). In NeurIPS, 2023.
25. Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. NeurIPS, 35:16344–16359, 2022. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Dan+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2E+NeurIPS%2C+35%3A16344%E2%80%9316359%2C+2022%2E)
26. Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim M. Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme, Matthias Minderer, Joan Puigcerver,… [find](https://scholar.google.com/scholar?q=Mostafa+Dehghani%2C+Josip+Djolonga%2C+Basil+Mustafa%2C+Piotr+Padlewski%2C+Jonathan+Heek%2C+Justin+Gilmer%2C+Andreas+Steiner%2C+Mathilde+Caron%2C+Robert+Geirhos%2C+Ibrahim+M%2E+Alabdulmohsin%2C+Rodolphe+Jenatton%2C+Lucas+Beye)
27. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). ArXiv, abs/1810.04805, 2018.
28. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). ArXiv, abs/2010.11929, 2020.
29. Danny Driess, F. Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Ho Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Z… [find](https://scholar.google.com/scholar?q=Danny+Driess%2C+F%2E+Xia%2C+Mehdi+S%2E+M%2E+Sajjadi%2C+Corey+Lynch%2C+Aakanksha+Chowdhery%2C+Brian+Ichter%2C+Ayzaan+Wahid%2C+Jonathan+Tompson%2C+Quan+Ho+Vuong%2C+Tianhe+Yu%2C+Wenlong+Huang%2C+Yevgen+Chebotar%2C+Pierre+Sermanet%2C+Da)
30. Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP, pages 736–740. IEEE, 2020. [find](https://scholar.google.com/scholar?q=Konstantinos+Drossos%2C+Samuel+Lipping%2C+and+Tuomas+Virtanen%2E+Clotho%3A+An+audio+captioning+dataset%2E+In+ICASSP%2C+pages+736%E2%80%93740%2E+IEEE%2C+2020%2E)
31. Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019. [find](https://scholar.google.com/scholar?q=Christoph+Feichtenhofer%2C+Haoqi+Fan%2C+Jitendra+Malik%2C+and+Kaiming+He%2E+Slowfast+networks+for+video+recognition%2E+In+ICCV%2C+2019%2E)
32. Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders as spatiotemporal learners. NeurIPS, 2022. [find](https://scholar.google.com/scholar?q=Christoph+Feichtenhofer%2C+Haoqi+Fan%2C+Yanghao+Li%2C+and+Kaiming+He%2E+Masked+autoencoders+as+spatiotemporal+learners%2E+NeurIPS%2C+2022%2E)
33. Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. Violet: End-to-end video-language transformers with masked visual-token modeling. arXiv preprint arXiv:2111.12681, 2021. — [arXiv:2111.12681](https://arxiv.org/abs/2111.12681) [find](https://scholar.google.com/scholar?q=Tsu%2DJui+Fu%2C+Linjie+Li%2C+Zhe+Gan%2C+Kevin+Lin%2C+William+Yang+Wang%2C+Lijuan+Wang%2C+and+Zicheng+Liu%2E+Violet%3A+End%2Dto%2Dend+video%2Dlanguage+transformers+with+masked+visual%2Dtoken+modeling%2E+arXiv+preprint+arXiv%3A2111%2E)
34. Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In ECCV, pages 214–229. Springer, 2020. [find](https://scholar.google.com/scholar?q=Valentin+Gabeur%2C+Chen+Sun%2C+Karteek+Alahari%2C+and+Cordelia+Schmid%2E+Multi%2Dmodal+transformer+for+video+retrieval%2E+In+ECCV%2C+pages+214%E2%80%93229%2E+Springer%2C+2020%2E)
35. J. Gao, Chen Sun, Zhenheng Yang, and Ramakant Nevatia. Tall: Temporal activity localization via language query. In ICCV, 2017. [find](https://scholar.google.com/scholar?q=J%2E+Gao%2C+Chen+Sun%2C+Zhenheng+Yang%2C+and+Ramakant+Nevatia%2E+Tall%3A+Temporal+activity+localization+via+language+query%2E+In+ICCV%2C+2017%2E)
36. Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qianmengke Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans. ArXiv, abs/2305.04790, 2023. — [arXiv:2305.04790](https://arxiv.org/abs/2305.04790) [find](https://scholar.google.com/scholar?q=Tao+Gong%2C+Chengqi+Lyu%2C+Shilong+Zhang%2C+Yudong+Wang%2C+Miao+Zheng%2C+Qianmengke+Zhao%2C+Kuikun+Liu%2C+Wenwei+Zhang%2C+Ping+Luo%2C+and+Kai+Chen%2E+Multimodal%2Dgpt%3A+A+vision+and+language+model+for+dialogue+with+humans%2E+)
37. Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer. arXiv preprint arXiv:2104.01778, 2021. — [arXiv:2104.01778](https://arxiv.org/abs/2104.01778) [find](https://scholar.google.com/scholar?q=Yuan+Gong%2C+Yu%2DAn+Chung%2C+and+James+Glass%2E+Ast%3A+Audio+spectrogram+transformer%2E+arXiv+preprint+arXiv%3A2104%2E01778%2C+2021%2E+%E2%80%94+%5BarXiv%3A2104%2E01778%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2104%2E01778%29)
38. Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The “something something” video database for learning and eva… [find](https://scholar.google.com/scholar?q=Raghav+Goyal%2C+Samira+Ebrahimi+Kahou%2C+Vincent+Michalski%2C+Joanna+Materzynska%2C+Susanne+Westphal%2C+Heuna+Kim%2C+Valentin+Haenel%2C+Ingo+Fr%C3%BCnd%2C+Peter+Yianilos%2C+Moritz+Mueller%2DFreitag%2C+Florian+Hoppe%2C+Christian+T)
39. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. [Making the v in vqa matter: Elevating the role of image understanding in visual question answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). In CVPR, 2017b.
40. Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh K. Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Z. Xu, Chen Zh… [find](https://scholar.google.com/scholar?q=Kristen+Grauman%2C+Andrew+Westbury%2C+Eugene+Byrne%2C+Zachary+Chavis%2C+Antonino+Furnari%2C+Rohit+Girdhar%2C+Jackson+Hamburger%2C+Hao+Jiang%2C+Miao+Liu%2C+Xingyu+Liu%2C+Miguel+Martin%2C+Tushar+Nagarajan%2C+Ilija+Radosavovic%2C)
41. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. [Masked autoencoders are scalable vision learners](https://md.research.elamur.ai/free/en/papers/he-2021-masked-autoencoders-scalable-vision-learners/). In CVPR, 2022.
42. Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, 2015. [find](https://scholar.google.com/scholar?q=Fabian+Caba+Heilbron%2C+Victor+Escorcia%2C+Bernard+Ghanem%2C+and+Juan+Carlos+Niebles%2E+Activitynet%3A+A+large%2Dscale+video+benchmark+for+human+activity+understanding%2E+In+CVPR%2C+2015%2E)
43. J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In ICLR, 2021.
44. Haroon Idrees, Amir R Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. The thumos challenge on action recognition for videos “in the wild”. CVIU, 2017. [find](https://scholar.google.com/scholar?q=Haroon+Idrees%2C+Amir+R+Zamir%2C+Yu%2DGang+Jiang%2C+Alex+Gorban%2C+Ivan+Laptev%2C+Rahul+Sukthankar%2C+and+Mubarak+Shah%2E+The+thumos+challenge+on+action+recognition+for+videos+%E2%80%9Cin+the+wild%E2%80%9D%2E+CVIU%2C+2017%2E)
45. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+Mistral+7b)
46. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759, 2016. [find](https://scholar.google.com/scholar?q=Armand+Joulin%2C+Edouard+Grave%2C+Piotr+Bojanowski%2C+and+Tomas+Mikolov%2E+Bag+of+tricks+for+efficient+text+classification%2E+arXiv+preprint+arXiv%3A1607%2E01759%2C+2016%2E)
47. Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. [Audiocaps: Generating captions for audios in the wild](https://md.research.elamur.ai/free/en/papers/kim-2019-audiocaps-generating-captions-audios-wild/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 119–132, 2019.
48. Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In ICCV, 2017. [find](https://scholar.google.com/scholar?q=Ranjay+Krishna%2C+Kenji+Hata%2C+Frederic+Ren%2C+Li+Fei%2DFei%2C+and+Juan+Carlos+Niebles%2E+Dense%2Dcaptioning+events+in+videos%2E+In+ICCV%2C+2017%2E)
49. Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In ICCV, pages 2556–2563. IEEE, 2011. [find](https://scholar.google.com/scholar?q=Hildegard+Kuehne%2C+Hueihan+Jhuang%2C+Est%C3%ADbaliz+Garrote%2C+Tomaso+Poggio%2C+and+Thomas+Serre%2E+Hmdb%3A+a+large+video+database+for+human+motion+recognition%2E+In+ICCV%2C+pages+2556%E2%80%932563%2E+IEEE%2C+2011%2E)
50. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
51. Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L. Berg. [Tvqa: Localized, compositional video question answering](https://md.research.elamur.ai/free/en/papers/lei-2018-tvqa-localized-compositional-video-question/). In EMNLP, 2018.
52. Jie Lei, Tamara L. Berg, and Mohit Bansal. Qvhighlights: Detecting moments and highlights in videos via natural language queries, 2021. [find](https://scholar.google.com/scholar?q=Jie+Lei%2C+Tamara+L%2E+Berg%2C+and+Mohit+Bansal%2E+Qvhighlights%3A+Detecting+moments+and+highlights+in+videos+via+natural+language+queries%2C+2021%2E)
53. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. [Otter: A multi-modal model with in-context instruction tuning](https://md.research.elamur.ai/free/en/papers/li-2023-otter-multi-modal-model-context/). arXiv preprint arXiv:2305.03726, 2023a.
54. Guangyao Li, Yixin Xu, and Di Hu. Multi-scale attention for audio question answering. arXiv preprint arXiv:2305.17993, 2023b. — [arXiv:2305.17993](https://arxiv.org/abs/2305.17993) [find](https://scholar.google.com/scholar?q=Guangyao+Li%2C+Yixin+Xu%2C+and+Di+Hu%2E+Multi%2Dscale+attention+for+audio+question+answering%2E+arXiv+preprint+arXiv%3A2305%2E17993%2C+2023b%2E+%E2%80%94+%5BarXiv%3A2305%2E17993%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E17993%29)
55. Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. [Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). In ICML, 2022a.
56. Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. [Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/). In ICML, 2022b.
57. Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Limin Wang, and Yu Qiao. Uniformerv2: Spatiotemporal learning by arming image vits with video uniformer. arXiv preprint arXiv:2211.09552, 2022c. — [arXiv:2211.09552](https://arxiv.org/abs/2211.09552) [find](https://scholar.google.com/scholar?q=Kunchang+Li%2C+Yali+Wang%2C+Yinan+He%2C+Yizhuo+Li%2C+Yi+Wang%2C+Limin+Wang%2C+and+Yu+Qiao%2E+Uniformerv2%3A+Spatiotemporal+learning+by+arming+image+vits+with+video+uniformer%2E+arXiv+preprint+arXiv%3A2211%2E09552%2C+2022c%2E+%E2%80%94)
58. KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. [Videochat: Chat-centric video understanding](https://md.research.elamur.ai/free/en/papers/li-2023-videochat-chat-centric-video-understanding/). arXiv preprint arXiv:2305.06355, 2023c.
59. Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. [Mvbench: A comprehensive multi-modal video understanding benchmark](https://md.research.elamur.ai/free/en/papers/li-2023-mvbench-comprehensive-multi-modal-video/). arXiv preprint arXiv:2311.17005, 2023d.
60. Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. arXiv preprint arXiv:2303.16058, 2023e. [find](https://scholar.google.com/scholar?q=Kunchang+Li%2C+Yali+Wang%2C+Yizhuo+Li%2C+Yi+Wang%2C+Yinan+He%2C+Limin+Wang%2C+and+Yu+Qiao%2E+Unmasked+teacher%3A+Towards+training%2Defficient+video+foundation+models%2E+arXiv+preprint+arXiv%3A2303%2E16058%2C+2023e%2E)
61. Tianhao Li and Limin Wang. Learning spatiotemporal features via video and text pair discrimination. CoRR, abs/2001.05691, 2020. URL https://arxiv.org/abs/2001.05691. — [arXiv:2001.05691](https://arxiv.org/abs/2001.05691) [↗](https://arxiv.org/abs/2001.05691)
62. Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video-language temporal grounding. In ICCV, pages 2794–2804, 2023. [find](https://scholar.google.com/scholar?q=Kevin+Qinghong+Lin%2C+Pengchuan+Zhang%2C+Joya+Chen%2C+Shraman+Pramanick%2C+Difei+Gao%2C+Alex+Jinpeng+Wang%2C+Rui+Yan%2C+and+Mike+Zheng+Shou%2E+Univtg%3A+Towards+unified+video%2Dlanguage+temporal+grounding%2E+In+ICCV%2C+pages)
63. Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen. Bmn: Boundary-matching network for temporal action proposal generation, 2019. [find](https://scholar.google.com/scholar?q=Tianwei+Lin%2C+Xiao+Liu%2C+Xin+Li%2C+Errui+Ding%2C+and+Shilei+Wen%2E+Bmn%3A+Boundary%2Dmatching+network+for+temporal+action+proposal+generation%2C+2019%2E)
64. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In ECCV, 2014.
65. Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. Clotho-aqa: A crowdsourced dataset for audio question answering. In EUSIPCO, pages 1140–1144. IEEE, 2022. [find](https://scholar.google.com/scholar?q=Samuel+Lipping%2C+Parthasaarathy+Sudarsanam%2C+Konstantinos+Drossos%2C+and+Tuomas+Virtanen%2E+Clotho%2Daqa%3A+A+crowdsourced+dataset+for+audio+question+answering%2E+In+EUSIPCO%2C+pages+1140%E2%80%931144%2E+IEEE%2C+2022%2E)
66. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/), 2023.
67. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024. [find](https://scholar.google.com/scholar?q=Haotian+Liu%2C+Chunyuan+Li%2C+Yuheng+Li%2C+Bo+Li%2C+Yuanhan+Zhang%2C+Sheng+Shen%2C+and+Yong+Jae+Lee%2E+Llava%2Dnext%3A+Improved+reasoning%2C+ocr%2C+and+world+knowledge%2C+2024%2E)
68. Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yu Duan, and Alex C Kot. Ntu rgb+d 120: A large-scale benchmark for 3d human activity understanding. TPAMI, 2020. [find](https://scholar.google.com/scholar?q=Jun+Liu%2C+Amir+Shahroudy%2C+Mauricio+Perez%2C+Gang+Wang%2C+Ling%2DYu+Duan%2C+and+Alex+C+Kot%2E+Ntu+rgb%2Bd+120%3A+A+large%2Dscale+benchmark+for+3d+human+activity+understanding%2E+TPAMI%2C+2020%2E)
69. Yi Liu, Limin Wang, Yali Wang, Xiao Ma, and Yu Qiao. Fineaction: A fine-grained video dataset for temporal action localization. TIP, 31:6937–6950, 2022. [find](https://scholar.google.com/scholar?q=Yi+Liu%2C+Limin+Wang%2C+Yali+Wang%2C+Xiao+Ma%2C+and+Yu+Qiao%2E+Fineaction%3A+A+fine%2Dgrained+video+dataset+for+temporal+action+localization%2E+TIP%2C+31%3A6937%E2%80%936950%2C+2022%2E)
70. Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. [Swin transformer: Hierarchical vision transformer using shifted windows](https://md.research.elamur.ai/free/en/papers/liu-2021-swin-transformer-hierarchical-vision-transformer/). In ICCV, 2021.
71. Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 2022. [find](https://scholar.google.com/scholar?q=Huaishao+Luo%2C+Lei+Ji%2C+Ming+Zhong%2C+Yang+Chen%2C+Wen+Lei%2C+Nan+Duan%2C+and+Tianrui+Li%2E+Clip4clip%3A+An+empirical+study+of+clip+for+end+to+end+video+clip+retrieval+and+captioning%2E+Neurocomputing%2C+2022%2E)
72. Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Ming-Hui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. ArXiv, abs/2306.07207, 2023. — [arXiv:2306.07207](https://arxiv.org/abs/2306.07207) [find](https://scholar.google.com/scholar?q=Ruipu+Luo%2C+Ziwang+Zhao%2C+Min+Yang%2C+Junwei+Dong%2C+Ming%2DHui+Qiu%2C+Pengcheng+Lu%2C+Tao+Wang%2C+and+Zhongyu+Wei%2E+Valley%3A+Video+assistant+with+large+language+model+enhanced+ability%2E+ArXiv%2C+abs%2F2306%2E07207%2C+2023%2E+%E2%80%94)
73. Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, and Fahad Shahbaz Khan. [Video-chatgpt: Towards detailed video understanding via large vision and language models](https://md.research.elamur.ai/free/en/papers/maaz-2023-video-chatgpt-towards-detailed-video/). ArXiv, abs/2306.05424, 2023.
74. Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, NeurIPS, pages 46212–46244, 2023. [find](https://scholar.google.com/scholar?q=Karttikeya+Mangalam%2C+Raiymbek+Akshulakov%2C+and+Jitendra+Malik%2E+Egoschema%3A+A+diagnostic+benchmark+for+very+long%2Dform+video+language+understanding%2E+In+A%2E+Oh%2C+T%2E+Neumann%2C+A%2E+Globerson%2C+K%2E+Saenko%2C+M%2E+Hardt)
75. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR, 2019. [find](https://scholar.google.com/scholar?q=Kenneth+Marino%2C+Mohammad+Rastegari%2C+Ali+Farhadi%2C+and+Roozbeh+Mottaghi%2E+Ok%2Dvqa%3A+A+visual+question+answering+benchmark+requiring+external+knowledge%2E+In+CVPR%2C+2019%2E)
76. Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D. Plumbley, Yuexian Zou, and Wenwu Wang. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research, 2023. [find](https://scholar.google.com/scholar?q=Xinhao+Mei%2C+Chutong+Meng%2C+Haohe+Liu%2C+Qiuqiang+Kong%2C+Tom+Ko%2C+Chengqi+Zhao%2C+Mark+D%2E+Plumbley%2C+Yuexian+Zou%2C+and+Wenwu+Wang%2E+Wavcaps%3A+A+chatgpt%2Dassisted+weakly%2Dlabelled+audio+captioning+dataset+for+audio%2D)
77. Mathew Monfort, Bolei Zhou, Sarah Adel Bargal, Alex Andonian, Tom Yan, Kandan Ramakrishnan, Lisa M. Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, and Aude Oliva. Moments in time dataset: One million videos for event understanding. TPAMI, 2020. [find](https://scholar.google.com/scholar?q=Mathew+Monfort%2C+Bolei+Zhou%2C+Sarah+Adel+Bargal%2C+Alex+Andonian%2C+Tom+Yan%2C+Kandan+Ramakrishnan%2C+Lisa+M%2E+Brown%2C+Quanfu+Fan%2C+Dan+Gutfreund%2C+Carl+Vondrick%2C+and+Aude+Oliva%2E+Moments+in+time+dataset%3A+One+millio)
78. WonJun Moon, Sangeek Hyun, SuBeen Lee, and Jae-Pil Heo. Correlation-guided query-dependency calibration in video representation learning for temporal grounding. arXiv preprint arXiv:2311.08835, 2023. — [arXiv:2311.08835](https://arxiv.org/abs/2311.08835) [find](https://scholar.google.com/scholar?q=WonJun+Moon%2C+Sangeek+Hyun%2C+SuBeen+Lee%2C+and+Jae%2DPil+Heo%2E+Correlation%2Dguided+query%2Ddependency+calibration+in+video+representation+learning+for+temporal+grounding%2E+arXiv+preprint+arXiv%3A2311%2E08835%2C+2023%2E+)
79. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). ArXiv, abs/2303.08774, 2023a.
80. OpenAI. Gpt-4v(ision) system card. [https://api.semanticscholar.org/CorpusID:263218031](https://api.semanticscholar.org/CorpusID:263218031), 2023b. [↗](https://api.semanticscholar.org/CorpusID:263218031)
81. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. [Dinov2: Learning robust visual features without supervision](https://md.research.elamur.ai/free/en/papers/oquab-2023-dinov2-learning-robust-visual-features/). arXiv preprint arXiv:2304.07193, 2023.
82. Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. In NeurIPS, 2024. [find](https://scholar.google.com/scholar?q=Viorica+Patraucean%2C+Lucas+Smaira%2C+Ankush+Gupta%2C+Adria+Recasens%2C+Larisa+Markeeva%2C+Dylan+Banarse%2C+Skanda+Koppula%2C+Mateusz+Malinowski%2C+Yi+Yang%2C+Carl+Doersch%2C+et+al%2E+Perception+test%3A+A+diagnostic+benchmar)
83. Karol J Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pages 1015–1018, 2015. [find](https://scholar.google.com/scholar?q=Karol+J+Piczak%2E+Esc%3A+Dataset+for+environmental+sound+classification%2E+In+Proceedings+of+the+23rd+ACM+international+conference+on+Multimedia%2C+pages+1015%E2%80%931018%2C+2015%2E)
84. Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In ICCV, 2015. [find](https://scholar.google.com/scholar?q=Bryan+A+Plummer%2C+Liwei+Wang%2C+Chris+M+Cervantes%2C+Juan+C+Caicedo%2C+Julia+Hockenmaier%2C+and+Svetlana+Lazebnik%2E+Flickr30k+entities%3A+Collecting+region%2Dto%2Dphrase+correspondences+for+richer+image%2Dto%2Dsentence+m)
85. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In ICML, 2021.
86. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. [Robust speech recognition via large-scale weak supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/). In ICML, pages 28492–28518. PMLR, 2023.
87. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). JMLR, 2020.
88. Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. A dataset for movie description. In CVPR, pages 3202–3212, 2015. [find](https://scholar.google.com/scholar?q=Anna+Rohrbach%2C+Marcus+Rohrbach%2C+Niket+Tandon%2C+and+Bernt+Schiele%2E+A+dataset+for+movie+description%2E+In+CVPR%2C+pages+3202%E2%80%933212%2C+2015%2E)
89. Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, and Christoph Feichtenhofer. Hiera: A hierarchical vision transformer without the bells-and-whistles, 2023. [find](https://scholar.google.com/scholar?q=Chaitanya+Ryali%2C+Yuan%2DTing+Hu%2C+Daniel+Bolya%2C+Chen+Wei%2C+Haoqi+Fan%2C+Po%2DYao+Huang%2C+Vaibhav+Aggarwal%2C+Arkabandhu+Chowdhury%2C+Omid+Poursaeed%2C+Judy+Hoffman%2C+Jitendra+Malik%2C+Yanghao+Li%2C+and+Christoph+Feichten)
90. Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. — [arXiv:1212.0402](https://arxiv.org/abs/1212.0402) [find](https://scholar.google.com/scholar?q=Khurram+Soomro%2C+Amir+Roshan+Zamir%2C+and+Mubarak+Shah%2E+Ucf101%3A+A+dataset+of+101+human+actions+classes+from+videos+in+the+wild%2E+arXiv+preprint+arXiv%3A1212%2E0402%2C+2012%2E+%E2%80%94+%5BarXiv%3A1212%2E0402%5D%28https%3A%2F%2Farxiv%2Eorg)
91. Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, et al. Generative multimodal models are in-context learners. arXiv preprint arXiv:2312.13286, 2023a. — [arXiv:2312.13286](https://arxiv.org/abs/2312.13286) [find](https://scholar.google.com/scholar?q=Quan+Sun%2C+Yufeng+Cui%2C+Xiaosong+Zhang%2C+Fan+Zhang%2C+Qiying+Yu%2C+Zhengxiong+Luo%2C+Yueze+Wang%2C+Yongming+Rao%2C+Jingjing+Liu%2C+Tiejun+Huang%2C+et+al%2E+Generative+multimodal+models+are+in%2Dcontext+learners%2E+arXiv+pre)
92. Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023b. [find](https://scholar.google.com/scholar?q=Quan+Sun%2C+Yuxin+Fang%2C+Ledell+Wu%2C+Xinlong+Wang%2C+and+Yue+Cao%2E+Eva%2Dclip%3A+Improved+training+techniques+for+clip+at+scale%2E+arXiv+preprint+arXiv%3A2303%2E15389%2C+2023b%2E)
93. Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222, 2023c. — [arXiv:2307.05222](https://arxiv.org/abs/2307.05222) [find](https://scholar.google.com/scholar?q=Quan+Sun%2C+Qiying+Yu%2C+Yufeng+Cui%2C+Fan+Zhang%2C+Xiaosong+Zhang%2C+Yueze+Wang%2C+Hongcheng+Gao%2C+Jingjing+Liu%2C+Tiejun+Huang%2C+and+Xinlong+Wang%2E+Generative+pretraining+in+multimodality%2E+arXiv+preprint+arXiv%3A2307%2E)
94. Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. Eva-clip-18b: Scaling clip to 18 billion parameters. arXiv preprint arXiv:2402.04252, 2024. — [arXiv:2402.04252](https://arxiv.org/abs/2402.04252) [find](https://scholar.google.com/scholar?q=Quan+Sun%2C+Jinsheng+Wang%2C+Qiying+Yu%2C+Yufeng+Cui%2C+Fan+Zhang%2C+Xiaosong+Zhang%2C+and+Xinlong+Wang%2E+Eva%2Dclip%2D18b%3A+Scaling+clip+to+18+billion+parameters%2E+arXiv+preprint+arXiv%3A2402%2E04252%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E0)
95. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
96. Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. NeurIPS, 35:10078–10093, 2022. [find](https://scholar.google.com/scholar?q=Zhan+Tong%2C+Yibing+Song%2C+Jue+Wang%2C+and+Limin+Wang%2E+Videomae%3A+Masked+autoencoders+are+data%2Defficient+learners+for+self%2Dsupervised+video+pre%2Dtraining%2E+NeurIPS%2C+35%3A10078%E2%80%9310093%2C+2022%2E)
97. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). ArXiv, abs/230…
98. Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Bria… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+R%2E+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+Daniel+M%2E+Bikel%2C+Lukas+Blecher%2C+Cristian+)
99. Du Tran, Hong xiu Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In CVPR, 2018. [find](https://scholar.google.com/scholar?q=Du+Tran%2C+Hong+xiu+Wang%2C+Lorenzo+Torresani%2C+Jamie+Ray%2C+Yann+LeCun%2C+and+Manohar+Paluri%2E+A+closer+look+at+spatiotemporal+convolutions+for+action+recognition%2E+In+CVPR%2C+2018%2E)
100. Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Kevin Qinghong Lin, Satoshi Tsutsui, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, et al. All in one: Exploring unified video-language pre-training. In CVPR, pages 6598–6608, 2023a. [find](https://scholar.google.com/scholar?q=Jinpeng+Wang%2C+Yixiao+Ge%2C+Rui+Yan%2C+Yuying+Ge%2C+Kevin+Qinghong+Lin%2C+Satoshi+Tsutsui%2C+Xudong+Lin%2C+Guanyu+Cai%2C+Jianping+Wu%2C+Ying+Shan%2C+et+al%2E+All+in+one%3A+Exploring+unified+video%2Dlanguage+pre%2Dtraining%2E+In+C)
101. Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016. [find](https://scholar.google.com/scholar?q=Limin+Wang%2C+Yuanjun+Xiong%2C+Zhe+Wang%2C+Yu+Qiao%2C+Dahua+Lin%2C+Xiaoou+Tang%2C+and+Luc+Van+Gool%2E+Temporal+segment+networks%3A+Towards+good+practices+for+deep+action+recognition%2E+In+ECCV%2C+2016%2E)
102. Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In CVPR, 2023b. [find](https://scholar.google.com/scholar?q=Limin+Wang%2C+Bingkun+Huang%2C+Zhiyu+Zhao%2C+Zhan+Tong%2C+Yinan+He%2C+Yi+Wang%2C+Yali+Wang%2C+and+Yu+Qiao%2E+Videomae+v2%3A+Scaling+video+masked+autoencoders+with+dual+masking%2E+In+CVPR%2C+2023b%2E)
103. Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Lu Yuan, and Yu-Gang Jiang. Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning. In CVPR, pages 6312–6322, 2023c. [find](https://scholar.google.com/scholar?q=Rui+Wang%2C+Dongdong+Chen%2C+Zuxuan+Wu%2C+Yinpeng+Chen%2C+Xiyang+Dai%2C+Mengchen+Liu%2C+Lu+Yuan%2C+and+Yu%2DGang+Jiang%2E+Masked+video+distillation%3A+Rethinking+masked+feature+modeling+for+self%2Dsupervised+video+represen)
104. Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In ICCV, 2019. [find](https://scholar.google.com/scholar?q=Xin+Wang%2C+Jiawei+Wu%2C+Junkun+Chen%2C+Lei+Li%2C+Yuan%2DFang+Wang%2C+and+William+Yang+Wang%2E+Vatex%3A+A+large%2Dscale%2C+high%2Dquality+multilingual+dataset+for+video%2Dand%2Dlanguage+research%2E+In+ICCV%2C+2019%2E)
105. Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. [Internvideo: General video foundation models via generative and discriminative learning](https://md.research.elamur.ai/free/en/papers/wang-2022-internvideo-general-video-foundation/). arXiv preprint ar…
106. Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942, 2023d. — [arXiv:2307.06942](https://arxiv.org/abs/2307.06942) [find](https://scholar.google.com/scholar?q=Yi+Wang%2C+Yinan+He%2C+Yizhuo+Li%2C+Kunchang+Li%2C+Jiashuo+Yu%2C+Xin+Ma%2C+Xinhao+Li%2C+Guo+Chen%2C+Xinyuan+Chen%2C+Yaohui+Wang%2C+et+al%2E+Internvid%3A+A+large%2Dscale+video%2Dtext+dataset+for+multimodal+understanding+and+gener)
107. Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, et al. mplug-2: A modularized multi-modal foundation model across text, image and video. arXiv preprint arXiv:2302.00402, 2023. — [arXiv:2302.00402](https://arxiv.org/abs/2302.00402) [find](https://scholar.google.com/scholar?q=Haiyang+Xu%2C+Qinghao+Ye%2C+Ming+Yan%2C+Yaya+Shi%2C+Jiabo+Ye%2C+Yuanhong+Xu%2C+Chenliang+Li%2C+Bin+Bi%2C+Qi+Qian%2C+Wei+Wang%2C+et+al%2E+mplug%2D2%3A+A+modularized+multi%2Dmodal+foundation+model+across+text%2C+image+and+video%2E+arX)
108. Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109.14084, 2021. [find](https://scholar.google.com/scholar?q=Hu+Xu%2C+Gargi+Ghosh%2C+Po%2DYao+Huang%2C+Dmytro+Okhonko%2C+Armen+Aghajanyan%2C+Florian+Metze%2C+Luke+Zettlemoyer%2C+and+Christoph+Feichtenhofer%2E+Videoclip%3A+Contrastive+pre%2Dtraining+for+zero%2Dshot+video%2Dtext+understan)
109. Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, pages 5288–5296, 2016. [find](https://scholar.google.com/scholar?q=Jun+Xu%2C+Tao+Mei%2C+Ting+Yao%2C+and+Yong+Rui%2E+Msr%2Dvtt%3A+A+large+video+description+dataset+for+bridging+video+and+language%2E+In+CVPR%2C+pages+5288%E2%80%935296%2C+2016%2E)
110. Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. Video-text modeling with zero-shot transfer from contrastive captioners. ArXiv, abs/2212.04979, 2022. — [arXiv:2212.04979](https://arxiv.org/abs/2212.04979) [find](https://scholar.google.com/scholar?q=Shen+Yan%2C+Tao+Zhu%2C+Zirui+Wang%2C+Yuan+Cao%2C+Mi+Zhang%2C+Soham+Ghosh%2C+Yonghui+Wu%2C+and+Jiahui+Yu%2E+Video%2Dtext+modeling+with+zero%2Dshot+transfer+from+contrastive+captioners%2E+ArXiv%2C+abs%2F2212%2E04979%2C+2022%2E+%E2%80%94+%5BarXi)
111. Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. In ICCV, 2019. [find](https://scholar.google.com/scholar?q=Linjie+Yang%2C+Yuchen+Fan%2C+and+Ning+Xu%2E+Video+instance+segmentation%2E+In+ICCV%2C+2019%2E)
112. Min Yang, Guo Chen, Yin-Dong Zheng, Tong Lu, and Limin Wang. Basictad: an astounding rgb-only baseline for temporal action detection. CVIU, page 103692, 2023. [find](https://scholar.google.com/scholar?q=Min+Yang%2C+Guo+Chen%2C+Yin%2DDong+Zheng%2C+Tong+Lu%2C+and+Limin+Wang%2E+Basictad%3A+an+astounding+rgb%2Donly+baseline+for+temporal+action+detection%2E+CVIU%2C+page+103692%2C+2023%2E)
113. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. [mplug-owl: Modularization empowers large language models with multimodality](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/), 2023…
114. Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. [Coca: Contrastive captioners are image-text foundation models](https://md.research.elamur.ai/free/en/papers/yu-2022-coca-contrastive-captioners-image-text/). arXiv preprint arXiv:2205.01917, 2022.
115. Éloi Zablocki, Hédi Ben-Younes, Patrick Pérez, and Matthieu Cord. Explainability of deep vision-based autonomous driving systems: Review and challenges. IJCV, 130(10):2425–2452, 2022. [find](https://scholar.google.com/scholar?q=%C3%89loi+Zablocki%2C+H%C3%A9di+Ben%2DYounes%2C+Patrick+P%C3%A9rez%2C+and+Matthieu+Cord%2E+Explainability+of+deep+vision%2Dbased+autonomous+driving+systems%3A+Review+and+challenges%2E+IJCV%2C+130%2810%29%3A2425%E2%80%932452%2C+2022%2E)
116. Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In CVPR, pages 16375–16387, 2022. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Jiasen+Lu%2C+Ximing+Lu%2C+Youngjae+Yu%2C+Yanpeng+Zhao%2C+Mohammadreza+Salehi%2C+Aditya+Kusupati%2C+Jack+Hessel%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+Merlot+reserve%3A+Neural+script+knowledge+through+vision+a)
117. Ziyun Zeng, Yixiao Ge, Zhan Tong, Xihui Liu, Shu-Tao Xia, and Ying Shan. Tvtsv2: Learning out-of-the-box spatiotemporal visual representations at scale. arXiv preprint arXiv:2305.14173, 2023. — [arXiv:2305.14173](https://arxiv.org/abs/2305.14173) [find](https://scholar.google.com/scholar?q=Ziyun+Zeng%2C+Yixiao+Ge%2C+Zhan+Tong%2C+Xihui+Liu%2C+Shu%2DTao+Xia%2C+and+Ying+Shan%2E+Tvtsv2%3A+Learning+out%2Dof%2Dthe%2Dbox+spatiotemporal+visual+representations+at+scale%2E+arXiv+preprint+arXiv%3A2305%2E14173%2C+2023%2E+%E2%80%94+%5BarXiv)
118. Biao Zhang and Rico Sennrich. Root mean square layer normalization. NeurIPS, 32, 2019. [find](https://scholar.google.com/scholar?q=Biao+Zhang+and+Rico+Sennrich%2E+Root+mean+square+layer+normalization%2E+NeurIPS%2C+32%2C+2019%2E)
119. Bowen Zhang, Jiahui Yu, Christopher Fifty, Wei Han, Andrew M. Dai, Ruoming Pang, and Fei Sha. Co-training transformer with videos and images improves action recognition. ArXiv, abs/2112.07175, 2021. — [arXiv:2112.07175](https://arxiv.org/abs/2112.07175) [find](https://scholar.google.com/scholar?q=Bowen+Zhang%2C+Jiahui+Yu%2C+Christopher+Fifty%2C+Wei+Han%2C+Andrew+M%2E+Dai%2C+Ruoming+Pang%2C+and+Fei+Sha%2E+Co%2Dtraining+transformer+with+videos+and+images+improves+action+recognition%2E+ArXiv%2C+abs%2F2112%2E07175%2C+2021%2E+%E2%80%94)
120. Hongjie Zhang, Yi Liu, Lu Dong, Yifei Huang, Zhen-Hua Ling, Yali Wang, Limin Wang, and Yu Qiao. Movqa: A benchmark of versatile question-answering for long-form movie understanding. arXiv preprint arXiv:2312.04817, 2023. — [arXiv:2312.04817](https://arxiv.org/abs/2312.04817) [find](https://scholar.google.com/scholar?q=Hongjie+Zhang%2C+Yi+Liu%2C+Lu+Dong%2C+Yifei+Huang%2C+Zhen%2DHua+Ling%2C+Yali+Wang%2C+Limin+Wang%2C+and+Yu+Qiao%2E+Movqa%3A+A+benchmark+of+versatile+question%2Danswering+for+long%2Dform+movie+understanding%2E+arXiv+preprint+arX)
121. Hang Zhao, Antonio Torralba, Lorenzo Torresani, and Zhicheng Yan. Hacs: Human action clips and segments dataset for recognition and temporal localization. In ICCV, 2019. [find](https://scholar.google.com/scholar?q=Hang+Zhao%2C+Antonio+Torralba%2C+Lorenzo+Torresani%2C+and+Zhicheng+Yan%2E+Hacs%3A+Human+action+clips+and+segments+dataset+for+recognition+and+temporal+localization%2E+In+ICCV%2C+2019%2E)
122. Long Zhao, Nitesh B. Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J. Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, Rachel Hornung, Florian Schroff, Ming-Hsuan Yang, David A. Ross, Huisheng Wang, Hartwig Adam, Mikhail Sirotenko, Ting Liu, and Boqing Gong. Videoprism: A foundation… [find](https://scholar.google.com/scholar?q=Long+Zhao%2C+Nitesh+B%2E+Gundavarapu%2C+Liangzhe+Yuan%2C+Hao+Zhou%2C+Shen+Yan%2C+Jennifer+J%2E+Sun%2C+Luke+Friedman%2C+Rui+Qian%2C+Tobias+Weyand%2C+Yue+Zhao%2C+Rachel+Hornung%2C+Florian+Schroff%2C+Ming%2DHsuan+Yang%2C+David+A%2E+Ross%2C)
123. Yanpeng Zhao, Jack Hessel, Youngjae Yu, Ximing Lu, Rowan Zellers, and Yejin Choi. Connecting the dots between audio and text without parallel data through visual knowledge transfer. arXiv preprint arXiv:2112.08995, 2021. [find](https://scholar.google.com/scholar?q=Yanpeng+Zhao%2C+Jack+Hessel%2C+Youngjae+Yu%2C+Ximing+Lu%2C+Rowan+Zellers%2C+and+Yejin+Choi%2E+Connecting+the+dots+between+audio+and+text+without+parallel+data+through+visual+knowledge+transfer%2E+arXiv+preprint+arX)
124. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+et+al%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2E+arXiv+p)
125. Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. arXiv preprint arXiv:2310.01852, 2023a. — [arXiv:2310.01852](https://arxiv.org/abs/2310.01852) [find](https://scholar.google.com/scholar?q=Bin+Zhu%2C+Bin+Lin%2C+Munan+Ning%2C+Yang+Yan%2C+Jiaxi+Cui%2C+HongFa+Wang%2C+Yatian+Pang%2C+Wenhao+Jiang%2C+Junwu+Zhang%2C+Zongwei+Li%2C+et+al%2E+Languagebind%3A+Extending+video%2Dlanguage+pretraining+to+n%2Dmodality+by+language%2D)
126. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. [Minigpt-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/). ArXiv, abs/2304.10592, 2023b.
127. Wentao Zhu, Yufang Huang, Xiufeng Xie, Wenxian Liu, Jincan Deng, Debing Zhang, Zhangyang Wang, and Ji Liu. Autoshot: A short video dataset and state-of-the-art shot boundary detection. In CVPR, pages 2237–2246, 2023c. [find](https://scholar.google.com/scholar?q=Wentao+Zhu%2C+Yufang+Huang%2C+Xiufeng+Xie%2C+Wenxian+Liu%2C+Jincan+Deng%2C+Debing+Zhang%2C+Zhangyang+Wang%2C+and+Ji+Liu%2E+Autoshot%3A+A+short+video+dataset+and+state%2Dof%2Dthe%2Dart+shot+boundary+detection%2E+In+CVPR%2C+pages+)

*Updated 24 October 2026*