---
title: "InternVideo — paper by Wang et al. (2022) — summary and ideas"
description: "Most vision foundation models are pretrained and adapted at the image level. 111 references, cited by 37 papers here."
authors: ["Yi Wang", "Kunchang Li", "Yizhuo Li", "Yinan He", "Bingkun Huang", "Zhiyu Zhao", "Hongjie Zhang", "Jilan Xu", "Yi Liu", "Zun Wang", "Sen Xing", "Guo Chen", "Junting Pan", "Jiashuo Yu", "Yali Wang", "Limin Wang", "Yu Qiao"]
year: 2022
venue: "arXiv:2212.03191"
source: "https://arxiv.org/abs/2212.03191"
doi: "10.48550/arxiv.2212.03191"
arxiv: "2212.03191"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2022-internvideo-general-video-foundation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2022-internvideo-general-video-foundation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) / InternVideo: General Video Foundation Models via Generative and Discriminative Learning

# InternVideo: General Video Foundation Models via Generative and Discriminative Learning

**Authors:** [Yi Wang](https://md.research.elamur.ai/free/en/authors/yi-wang/), [Kunchang Li](https://md.research.elamur.ai/free/en/authors/kunchang-li/), [Yizhuo Li](https://md.research.elamur.ai/free/en/authors/yizhuo-li/), [Yinan He](https://md.research.elamur.ai/free/en/authors/yinan-he/), Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, [Jilan Xu](https://md.research.elamur.ai/free/en/authors/jilan-xu/), [Yi Liu](https://md.research.elamur.ai/free/en/authors/yi-liu/), [Zun Wang](https://md.research.elamur.ai/free/en/authors/zun-wang/), [Sen Xing](https://md.research.elamur.ai/free/en/authors/sen-xing/), [Guo Chen](https://md.research.elamur.ai/free/en/authors/guo-chen/), Junting Pan, Jiashuo Yu, [Yali Wang](https://md.research.elamur.ai/free/en/authors/yali-wang/), [Limin Wang](https://md.research.elamur.ai/free/en/authors/limin-wang/) and [Yu Qiao](https://md.research.elamur.ai/free/en/authors/yu-qiao/) · **Year:** 2022 · **Venue:** arXiv:2212.03191 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2212.03191) · [PDF](https://arxiv.org/pdf/2212.03191) · [DOI 10.48550/arxiv.2212.03191](https://doi.org/10.48550/arxiv.2212.03191) · [arXiv 2212.03191](https://arxiv.org/abs/2212.03191) · [OpenAlex](https://openalex.org/W4310921506)

**Citations:** 91 citations (OpenAlex, 2026-10-14) · [37](https://md.research.elamur.ai/free/en/papers/wang-2022-internvideo-general-video-foundation/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Most vision foundation models are pretrained and adapted at the image level. This makes them a poor fit for video tasks, which depend on motion and temporal structure. Existing video models also tend to specialize in either action understanding or video-language tasks. InternVideo is a family of general-purpose video backbones that combines two self-supervised paradigms:
- **Generative:** masked video modeling (a scaled-up VideoMAE).
- **Discriminative:** video-language contrastive learning, which builds on an image-pretrained ViT extended with local and global spatiotemporal modules.

Both encoders are further refined with supervised action classification and trained separately. They are then merged through lightweight cross-model attention (CMA) layers, while the backbones stay frozen. The authors evaluate the result on a benchmark they assembled from public datasets, grouped into three ability areas: action understanding, video-language alignment, and open-world video use. InternVideo sets new state-of-the-art results across all of them. It also needs far less compute than comparable foundation models. This suggests that dedicated video representations, rather than adapted image features, are worth pursuing as a general tool.

## Key points
- Evaluated on 39 public datasets covering 10 tasks. The model reaches 91.1% top-1 on Kinetics-400 and 77.2% on Something-Something V2.
- With a new pretraining dataset, VideoMAE scales well. Fine-tuned Kinetics-400 accuracy goes from 81.01% (ViT-Base) to 85.35% (Large) and 86.9% (Huge).
- The two encoders are trained independently and fused afterward with CMA modules. The authors report that generative and contrastive features are complementary.
- Training took about 64.5K A100-80G GPU hours, compared with 245.76K TPU v4 hours for CoCa. Power use is roughly 23% of CoCa's.
- In open-set action recognition with a DEAR-style evidential head on ViT-H/16, the model achieves:
  - Open Set AUC of 85.48% (UCF-101 + HMDB-51) and 91.85% (UCF-101 + MiT-v2).
  - Closed-set accuracy of 97.89%.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 137 works · 83,262 citations
- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 65 works · 19,402 citations
- [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) — Government body, Beijing, China · 68 works · 22,753 citations
- [Chinese University of Hong Kong](https://md.research.elamur.ai/free/en/universities/chinese-university-of-hong-kong/) — University, Hong Kong, China · 53 works · 12,921 citations
- [University of Hong Kong](https://md.research.elamur.ai/free/en/universities/university-of-hong-kong/) — University, Pok Fu Lam, China · 44 works · 10,936 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 44 works · 10,162 citations
- [Nanjing University](https://md.research.elamur.ai/free/en/universities/nanjing-university/) — University, Nanjing, China · 28 works · 7,072 citations
- [Australian National University](https://md.research.elamur.ai/free/en/universities/australian-national-university/) — University, Canberra, Australia · 10 works · 6,215 citations
- [Shenzhen Institutes of Advanced Technology](https://md.research.elamur.ai/free/en/labs/shenzhen-institutes-of-advanced-technology/) — Research institute, Shenzhen, China · 12 works · 2,431 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works

## Cite

Cite as: Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang and Yu Qiao. 2022. InternVideo: General Video Foundation Models via Generative and Discriminative Learning. arXiv:2212.03191.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wang2022internvideo,
  title = {{InternVideo: General Video Foundation Models via Generative and Discriminative Learning}},
  author = {Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang and Yu Qiao},
  year = {2022},
  doi = {10.48550/arxiv.2212.03191},
  eprint = {2212.03191},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2212.03191}
}
```

- [← previous Controllable Natural Language Generation with Contrastive Prefixes](https://md.research.elamur.ai/free/en/papers/qian-2022-controllable-natural-language-generation-contrastive/)
- [next → MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data](https://md.research.elamur.ai/free/en/papers/zhao-2022-multihiertt-numerical-reasoning-over-multi/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/)

## References

111 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Boyang Xia, Wenhao Wu, Haoran Wang, Rui Su, Dongliang He, Haosen Yang, Xiaoran Fan, and Wanli Ouyang. Nsnet: Non-saliency suppression sampler for efficient video recognition. In ECCV, 2022. [find](https://scholar.google.com/scholar?q=Boyang+Xia%2C+Wenhao+Wu%2C+Haoran+Wang%2C+Rui+Su%2C+Dongliang+He%2C+Haosen+Yang%2C+Xiaoran+Fan%2C+and+Wanli+Ouyang%2E+Nsnet%3A+Non%2Dsaliency+suppression+sampler+for+efficient+video+recognition%2E+In+ECCV%2C+2022%2E)
2. Alexandros Stergiou and Ronald Poppe. Learn to cycle: Time-consistent feature discovery for action recognition. Pattern Recognition Letters, 141:1–7, 2021. [find](https://scholar.google.com/scholar?q=Alexandros+Stergiou+and+Ronald+Poppe%2E+Learn+to+cycle%3A+Time%2Dconsistent+feature+discovery+for+action+recognition%2E+Pattern+Recognition+Letters%2C+141%3A1%E2%80%937%2C+2021%2E)
3. Lei Wang and Piotr Koniusz. Self-supervising action recognition by statistical moment and subspace descriptors. In ACM International Conference on Multimedia, 2021. [find](https://scholar.google.com/scholar?q=Lei+Wang+and+Piotr+Koniusz%2E+Self%2Dsupervising+action+recognition+by+statistical+moment+and+subspace+descriptors%2E+In+ACM+International+Conference+on+Multimedia%2C+2021%2E)
4. Chenlin Zhang, Jianxin Wu, and Yin Li. Actionformer: Localizing moments of actions with transformers. In eccv, 2022. [find](https://scholar.google.com/scholar?q=Chenlin+Zhang%2C+Jianxin+Wu%2C+and+Yin+Li%2E+Actionformer%3A+Localizing+moments+of+actions+with+transformers%2E+In+eccv%2C+2022%2E)
5. Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 2022. [find](https://scholar.google.com/scholar?q=Huaishao+Luo%2C+Lei+Ji%2C+Ming+Zhong%2C+Yang+Chen%2C+Wen+Lei%2C+Nan+Duan%2C+and+Tianrui+Li%2E+Clip4clip%3A+An+empirical+study+of+clip+for+end+to+end+video+clip+retrieval+and+captioning%2E+Neurocomputing%2C+2022%2E)
6. Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In CVPR, 2022. [find](https://scholar.google.com/scholar?q=Chen+Wei%2C+Haoqi+Fan%2C+Saining+Xie%2C+Chao%2DYuan+Wu%2C+Alan+Yuille%2C+and+Christoph+Feichtenhofer%2E+Masked+feature+prediction+for+self%2Dsupervised+visual+pre%2Dtraining%2E+In+CVPR%2C+2022%2E)
7. Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. [Coca: Contrastive captioners are image-text foundation models](https://md.research.elamur.ai/free/en/papers/yu-2022-coca-contrastive-captioners-image-text/). arXiv preprint arXiv:2205.01917, 2022.
8. Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid. Multiview transformers for video recognition. In CVPR, 2022. [find](https://scholar.google.com/scholar?q=Shen+Yan%2C+Xuehan+Xiong%2C+Anurag+Arnab%2C+Zhichao+Lu%2C+Mi+Zhang%2C+Chen+Sun%2C+and+Cordelia+Schmid%2E+Multiview+transformers+for+video+recognition%2E+In+CVPR%2C+2022%2E)
9. Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. NeurIPS, 2021. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ximing+Lu%2C+Jack+Hessel%2C+Youngjae+Yu%2C+Jae+Sung+Park%2C+Jize+Cao%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+Merlot%3A+Multimodal+neural+script+knowledge+models%2E+NeurIPS%2C+2021%2E)
10. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. [On the opportunities and risks of foundation models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/). arXiv preprint arXiv:2108.07258, 2021.
11. Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. NeurIPS, 2021. [find](https://scholar.google.com/scholar?q=Hassan+Akbari%2C+Liangzhe+Yuan%2C+Rui+Qian%2C+Wei%2DHong+Chuang%2C+Shih%2DFu+Chang%2C+Yin+Cui%2C+and+Boqing+Gong%2E+Vatt%3A+Transformers+for+multimodal+self%2Dsupervised+learning+from+raw+video%2C+audio+and+text%2E+NeurIPS%2C+20)
12. Jing Shao, Siyu Chen, Yangguang Li, Kun Wang, Zhenfei Yin, Yinan He, Jianing Teng, Qinghong Sun, Mengya Gao, Jihao Liu, et al. Intern: A new learning paradigm towards general vision. arXiv preprint arXiv:2111.08687, 2021. — [arXiv:2111.08687](https://arxiv.org/abs/2111.08687) [find](https://scholar.google.com/scholar?q=Jing+Shao%2C+Siyu+Chen%2C+Yangguang+Li%2C+Kun+Wang%2C+Zhenfei+Yin%2C+Yinan+He%2C+Jianing+Teng%2C+Qinghong+Sun%2C+Mengya+Gao%2C+Jihao+Liu%2C+et+al%2E+Intern%3A+A+new+learning+paradigm+towards+general+vision%2E+arXiv+preprint+ar)
13. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In ICML, 2021.
14. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. [Scaling up visual and vision-language representation learning with noisy text supervision](https://md.research.elamur.ai/free/en/papers/jia-2021-scaling-up-visual-vision-language/). In ICML, 2021.
15. Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021. — [arXiv:2111.11432](https://arxiv.org/abs/2111.11432) [find](https://scholar.google.com/scholar?q=Lu+Yuan%2C+Dongdong+Chen%2C+Yi%2DLing+Chen%2C+Noel+Codella%2C+Xiyang+Dai%2C+Jianfeng+Gao%2C+Houdong+Hu%2C+Xuedong+Huang%2C+Boxin+Li%2C+Chunyuan+Li%2C+et+al%2E+Florence%3A+A+new+foundation+model+for+computer+vision%2E+arXiv+prepr)
16. Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-label space. In CVPR, 2022. [find](https://scholar.google.com/scholar?q=Jianwei+Yang%2C+Chunyuan+Li%2C+Pengchuan+Zhang%2C+Bin+Xiao%2C+Ce+Liu%2C+Lu+Yuan%2C+and+Jianfeng+Gao%2E+Unified+contrastive+learning+in+image%2Dtext%2Dlabel+space%2E+In+CVPR%2C+2022%2E)
17. Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. [SimVLM: Simple visual language model pretraining with weak supervision](https://md.research.elamur.ai/free/en/papers/wang-2021-simvlm-simple-visual-language-model/). In ICLR, 2022.
18. Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. [Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework](https://md.research.elamur.ai/free/en/papers/wang-2022-ofa-unifying-architectures-tasks-modalities/). In ICML, 2022.
19. Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442, 2022. [find](https://scholar.google.com/scholar?q=Wenhui+Wang%2C+Hangbo+Bao%2C+Li+Dong%2C+Johan+Bjorck%2C+Zhiliang+Peng%2C+Qiang+Liu%2C+Kriti+Aggarwal%2C+Owais+Khan+Mohammed%2C+Saksham+Singhal%2C+Subhojit+Som%2C+et+al%2E+Image+as+a+foreign+language%3A+Beit+pretraining+for+a)
20. Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In ICLR, 2022. [find](https://scholar.google.com/scholar?q=Hangbo+Bao%2C+Li+Dong%2C+Songhao+Piao%2C+and+Furu+Wei%2E+BEit%3A+BERT+pre%2Dtraining+of+image+transformers%2E+In+ICLR%2C+2022%2E)
21. Paul Barham, Aakanksha Chowdhery, Jeff Dean, Sanjay Ghemawat, Steven Hand, Daniel Hurt, Michael Isard, Hyeontaek Lim, Ruoming Pang, Sudip Roy, et al. Pathways: Asynchronous distributed dataflow for ml. Proceedings of Machine Learning and Systems, 2022. [find](https://scholar.google.com/scholar?q=Paul+Barham%2C+Aakanksha+Chowdhery%2C+Jeff+Dean%2C+Sanjay+Ghemawat%2C+Steven+Hand%2C+Daniel+Hurt%2C+Michael+Isard%2C+Hyeontaek+Lim%2C+Ruoming+Pang%2C+Sudip+Roy%2C+et+al%2E+Pathways%3A+Asynchronous+distributed+dataflow+for+ml)
22. Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-clip: End-to-end multi-grained contrastive learning for video-text retrieval. In ACM International Conference on Multimedia, 2022. [find](https://scholar.google.com/scholar?q=Yiwei+Ma%2C+Guohai+Xu%2C+Xiaoshuai+Sun%2C+Ming+Yan%2C+Ji+Zhang%2C+and+Rongrong+Ji%2E+X%2Dclip%3A+End%2Dto%2Dend+multi%2Dgrained+contrastive+learning+for+video%2Dtext+retrieval%2E+In+ACM+International+Conference+on+Multimedia%2C+)
23. Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In NeurIPS, 2022. [find](https://scholar.google.com/scholar?q=Zhan+Tong%2C+Yibing+Song%2C+Jue+Wang%2C+and+Limin+Wang%2E+VideoMAE%3A+Masked+autoencoders+are+data%2Defficient+learners+for+self%2Dsupervised+video+pre%2Dtraining%2E+In+NeurIPS%2C+2022%2E)
24. Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. All in one: Exploring unified video-language pre-training. arXiv preprint arXiv:2203.07303, 2022. — [arXiv:2203.07303](https://arxiv.org/abs/2203.07303) [find](https://scholar.google.com/scholar?q=Alex+Jinpeng+Wang%2C+Yixiao+Ge%2C+Rui+Yan%2C+Yuying+Ge%2C+Xudong+Lin%2C+Guanyu+Cai%2C+Jianping+Wu%2C+Ying+Shan%2C+Xiaohu+Qie%2C+and+Mike+Zheng+Shou%2E+All+in+one%3A+Exploring+unified+video%2Dlanguage+pre%2Dtraining%2E+arXiv+prep)
25. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. [Masked autoencoders are scalable vision learners](https://md.research.elamur.ai/free/en/papers/he-2021-masked-autoencoders-scalable-vision-learners/). In CVPR, 2022.
26. Tianhao Li and Limin Wang. Learning spatiotemporal features via video and text pair discrimination. CoRR, abs/2001.05691, 2020. — [arXiv:2001.05691](https://arxiv.org/abs/2001.05691) [find](https://scholar.google.com/scholar?q=Tianhao+Li+and+Limin+Wang%2E+Learning+spatiotemporal+features+via+video+and+text+pair+discrimination%2E+CoRR%2C+abs%2F2001%2E05691%2C+2020%2E+%E2%80%94+%5BarXiv%3A2001%2E05691%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2001%2E05691%29)
27. Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017. [find](https://scholar.google.com/scholar?q=Joao+Carreira+and+Andrew+Zisserman%2E+Quo+vadis%2C+action+recognition%3F+a+new+model+and+the+kinetics+dataset%2E+In+CVPR%2C+2017%2E)
28. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). In ICLR, 2021.
29. Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. Multimae: Multi-modal multi-task masked autoencoders. arXiv preprint arXiv:2204.01678, 2022. — [arXiv:2204.01678](https://arxiv.org/abs/2204.01678) [find](https://scholar.google.com/scholar?q=Roman+Bachmann%2C+David+Mizrahi%2C+Andrei+Atanov%2C+and+Amir+Zamir%2E+Multimae%3A+Multi%2Dmodal+multi%2Dtask+masked+autoencoders%2E+arXiv+preprint+arXiv%3A2204%2E01678%2C+2022%2E+%E2%80%94+%5BarXiv%3A2204%2E01678%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F22)
30. Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. Violet: End-to-end video-language transformers with masked visual-token modeling. arXiv preprint arXiv:2111.12681, 2021. — [arXiv:2111.12681](https://arxiv.org/abs/2111.12681) [find](https://scholar.google.com/scholar?q=Tsu%2DJui+Fu%2C+Linjie+Li%2C+Zhe+Gan%2C+Kevin+Lin%2C+William+Yang+Wang%2C+Lijuan+Wang%2C+and+Zicheng+Liu%2E+Violet%3A+End%2Dto%2Dend+video%2Dlanguage+transformers+with+masked+visual%2Dtoken+modeling%2E+arXiv+preprint+arXiv%3A2111%2E)
31. Linjie Li, Zhe Gan, Kevin Lin, Chung-Ching Lin, Zicheng Liu, Ce Liu, and Lijuan Wang. Lavender: Unifying video-language understanding as masked language modeling. arXiv preprint arXiv:2206.07160, 2022. — [arXiv:2206.07160](https://arxiv.org/abs/2206.07160) [find](https://scholar.google.com/scholar?q=Linjie+Li%2C+Zhe+Gan%2C+Kevin+Lin%2C+Chung%2DChing+Lin%2C+Zicheng+Liu%2C+Ce+Liu%2C+and+Lijuan+Wang%2E+Lavender%3A+Unifying+video%2Dlanguage+understanding+as+masked+language+modeling%2E+arXiv+preprint+arXiv%3A2206%2E07160%2C+2022)
32. Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In CVPR, 2022. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Jiasen+Lu%2C+Ximing+Lu%2C+Youngjae+Yu%2C+Yanpeng+Zhao%2C+Mohammadreza+Salehi%2C+Aditya+Kusupati%2C+Jack+Hessel%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+Merlot+reserve%3A+Neural+script+knowledge+through+vision+a)
33. Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015. [find](https://scholar.google.com/scholar?q=Carl+Doersch%2C+Abhinav+Gupta%2C+and+Alexei+A+Efros%2E+Unsupervised+visual+representation+learning+by+context+prediction%2E+In+ICCV%2C+2015%2E)
34. Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In ICCV, 2015. [find](https://scholar.google.com/scholar?q=Xiaolong+Wang+and+Abhinav+Gupta%2E+Unsupervised+learning+of+visual+representations+using+videos%2E+In+ICCV%2C+2015%2E)
35. Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016. [find](https://scholar.google.com/scholar?q=Mehdi+Noroozi+and+Paolo+Favaro%2E+Unsupervised+learning+of+visual+representations+by+solving+jigsaw+puzzles%2E+In+ECCV%2C+2016%2E)
36. Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016. [find](https://scholar.google.com/scholar?q=Richard+Zhang%2C+Phillip+Isola%2C+and+Alexei+A+Efros%2E+Colorful+image+colorization%2E+In+ECCV%2C+2016%2E)
37. Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders as spatiotemporal learners. arXiv preprint arXiv:2205.09113, 2022. — [arXiv:2205.09113](https://arxiv.org/abs/2205.09113) [find](https://scholar.google.com/scholar?q=Christoph+Feichtenhofer%2C+Haoqi+Fan%2C+Yanghao+Li%2C+and+Kaiming+He%2E+Masked+autoencoders+as+spatiotemporal+learners%2E+arXiv+preprint+arXiv%3A2205%2E09113%2C+2022%2E+%E2%80%94+%5BarXiv%3A2205%2E09113%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2205%2E0)
38. Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018. [find](https://scholar.google.com/scholar?q=Zhirong+Wu%2C+Yuanjun+Xiong%2C+Stella+X+Yu%2C+and+Dahua+Lin%2E+Unsupervised+feature+learning+via+non%2Dparametric+instance+discrimination%2E+In+CVPR%2C+2018%2E)
39. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. [Momentum contrast for unsupervised visual representation learning](https://md.research.elamur.ai/free/en/papers/he-2019-momentum-contrast-unsupervised-visual-representation/). In CVPR, 2020.
40. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. [A simple framework for contrastive learning of visual representations](https://md.research.elamur.ai/free/en/papers/chen-2020-simple-framework-contrastive-learning-visual/). In ICML, 2020.
41. Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. NeurIPS, 2020. [find](https://scholar.google.com/scholar?q=Jean%2DBastien+Grill%2C+Florian+Strub%2C+Florent+Altch%C3%A9%2C+Corentin+Tallec%2C+Pierre+Richemond%2C+Elena+Buchatskaya%2C+Carl+Doersch%2C+Bernardo+Avila+Pires%2C+Zhaohan+Guo%2C+Mohammad+Gheshlaghi+Azar%2C+et+al%2E+Bootstrap+you)
42. Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021. [find](https://scholar.google.com/scholar?q=Xinlei+Chen+and+Kaiming+He%2E+Exploring+simple+siamese+representation+learning%2E+In+CVPR%2C+2021%2E)
43. Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020. [find](https://scholar.google.com/scholar?q=Mark+Chen%2C+Alec+Radford%2C+Rewon+Child%2C+Jeffrey+Wu%2C+Heewoo+Jun%2C+David+Luan%2C+and+Ilya+Sutskever%2E+Generative+pretraining+from+pixels%2E+In+ICML%2C+2020%2E)
44. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. [Zero-shot text-to-image generation](https://md.research.elamur.ai/free/en/papers/ramesh-2021-zero-shot-text-image-generation/). In ICML, 2021.
45. Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. Bevt: Bert pretraining of video transformers. In CVPR, 2022. [find](https://scholar.google.com/scholar?q=Rui+Wang%2C+Dongdong+Chen%2C+Zuxuan+Wu%2C+Yinpeng+Chen%2C+Xiyang+Dai%2C+Mengchen+Liu%2C+Yu%2DGang+Jiang%2C+Luowei+Zhou%2C+and+Lu+Yuan%2E+Bevt%3A+Bert+pretraining+of+video+transformers%2E+In+CVPR%2C+2022%2E)
46. Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR, 2020. [find](https://scholar.google.com/scholar?q=Antoine+Miech%2C+Jean%2DBaptiste+Alayrac%2C+Lucas+Smaira%2C+Ivan+Laptev%2C+Josef+Sivic%2C+and+Andrew+Zisserman%2E+End%2Dto%2Dend+learning+of+visual+representations+from+uncurated+instructional+videos%2E+In+CVPR%2C+2020%2E)
47. Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109.14084, 2021. [find](https://scholar.google.com/scholar?q=Hu+Xu%2C+Gargi+Ghosh%2C+Po%2DYao+Huang%2C+Dmytro+Okhonko%2C+Armen+Aghajanyan%2C+Florian+Metze%2C+Luke+Zettlemoyer%2C+and+Christoph+Feichtenhofer%2E+Videoclip%3A+Contrastive+pre%2Dtraining+for+zero%2Dshot+video%2Dtext+understan)
48. Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision-language pre-training for image captioning. In CVPR, 2022. [find](https://scholar.google.com/scholar?q=Xiaowei+Hu%2C+Zhe+Gan%2C+Jianfeng+Wang%2C+Zhengyuan+Yang%2C+Zicheng+Liu%2C+Yumao+Lu%2C+and+Lijuan+Wang%2E+Scaling+up+vision%2Dlanguage+pre%2Dtraining+for+image+captioning%2E+In+CVPR%2C+2022%2E)
49. Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et al. An empirical study of training end-to-end vision-and-language transformers. In CVPR, 2022. [find](https://scholar.google.com/scholar?q=Zi%2DYi+Dou%2C+Yichong+Xu%2C+Zhe+Gan%2C+Jianfeng+Wang%2C+Shuohang+Wang%2C+Lijuan+Wang%2C+Chenguang+Zhu%2C+Pengchuan+Zhang%2C+Lu+Yuan%2C+Nanyun+Peng%2C+et+al%2E+An+empirical+study+of+training+end%2Dto%2Dend+vision%2Dand%2Dlanguage+tr)
50. Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and-language tasks? arXiv preprint arXiv:2107.06383, 2021. [find](https://scholar.google.com/scholar?q=Sheng+Shen%2C+Liunian+Harold+Li%2C+Hao+Tan%2C+Mohit+Bansal%2C+Anna+Rohrbach%2C+Kai%2DWei+Chang%2C+Zhewei+Yao%2C+and+Kurt+Keutzer%2E+How+much+can+clip+benefit+vision%2Dand%2Dlanguage+tasks%3F+arXiv+preprint+arXiv%3A2107%2E06383%2C+)
51. Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783, 2021. — [arXiv:2111.07783](https://arxiv.org/abs/2111.07783) [find](https://scholar.google.com/scholar?q=Lewei+Yao%2C+Runhui+Huang%2C+Lu+Hou%2C+Guansong+Lu%2C+Minzhe+Niu%2C+Hang+Xu%2C+Xiaodan+Liang%2C+Zhenguo+Li%2C+Xin+Jiang%2C+and+Chunjing+Xu%2E+Filip%3A+Fine%2Dgrained+interactive+language%2Dimage+pre%2Dtraining%2E+arXiv+preprint+ar)
52. Chen Sun, Austin Myers, Carl Vondrick, Kevin P. Murphy, and Cordelia Schmid. [Videobert: A joint model for video and language representation learning](https://md.research.elamur.ai/free/en/papers/sun-2019-videobert-joint-model-video-language/). ICCV, 2019.
53. Linchao Zhu and Yi Yang. Actbert: Learning global-local video-text representations. CVPR, 2020. [find](https://scholar.google.com/scholar?q=Linchao+Zhu+and+Yi+Yang%2E+Actbert%3A+Learning+global%2Dlocal+video%2Dtext+representations%2E+CVPR%2C+2020%2E)
54. Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In CVPR, 2021. [find](https://scholar.google.com/scholar?q=Jie+Lei%2C+Linjie+Li%2C+Luowei+Zhou%2C+Zhe+Gan%2C+Tamara+L+Berg%2C+Mohit+Bansal%2C+and+Jingjing+Liu%2E+Less+is+more%3A+Clipbert+for+video%2Dand%2Dlanguage+learning+via+sparse+sampling%2E+In+CVPR%2C+2021%2E)
55. Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, 2021. [find](https://scholar.google.com/scholar?q=Max+Bain%2C+Arsha+Nagrani%2C+G%C3%BCl+Varol%2C+and+Andrew+Zisserman%2E+Frozen+in+time%3A+A+joint+video+and+image+encoder+for+end%2Dto%2Dend+retrieval%2E+In+ICCV%2C+2021%2E)
56. Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Limin Wang, and Yu Qiao. Uniformerv2: Spatiotemporal learning by arming image vits with video uniformer. arXiv preprint arXiv:2211.09552, 2022. — [arXiv:2211.09552](https://arxiv.org/abs/2211.09552) [find](https://scholar.google.com/scholar?q=Kunchang+Li%2C+Yali+Wang%2C+Yinan+He%2C+Yizhuo+Li%2C+Yi+Wang%2C+Limin+Wang%2C+and+Yu+Qiao%2E+Uniformerv2%3A+Spatiotemporal+learning+by+arming+image+vits+with+video+uniformer%2E+arXiv+preprint+arXiv%3A2211%2E09552%2C+2022%2E+%E2%80%94+)
57. Guo Chen, Sen Xing, Zhe Chen, Yi Wang, Kunchang Li, Yizhuo Li, Yi Liu, Jiahao Wang, Yin-Dong Zheng, Bingkun Huang, et al. Internvideo-ego4d: A pack of champion solutions to ego4d challenges. arXiv preprint arXiv:2211.09529, 2022. — [arXiv:2211.09529](https://arxiv.org/abs/2211.09529) [find](https://scholar.google.com/scholar?q=Guo+Chen%2C+Sen+Xing%2C+Zhe+Chen%2C+Yi+Wang%2C+Kunchang+Li%2C+Yizhuo+Li%2C+Yi+Liu%2C+Jiahao+Wang%2C+Yin%2DDong+Zheng%2C+Bingkun+Huang%2C+et+al%2E+Internvideo%2Dego4d%3A+A+pack+of+champion+solutions+to+ego4d+challenges%2E+arXiv+pre)
58. Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In ICCV, 2021. [find](https://scholar.google.com/scholar?q=Anurag+Arnab%2C+Mostafa+Dehghani%2C+Georg+Heigold%2C+Chen+Sun%2C+Mario+Luc%CC%8Cic%CC%81%2C+and+Cordelia+Schmid%2E+Vivit%3A+A+video+vision+transformer%2E+In+ICCV%2C+2021%2E)
59. Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR, 2022. [find](https://scholar.google.com/scholar?q=Ze+Liu%2C+Jia+Ning%2C+Yue+Cao%2C+Yixuan+Wei%2C+Zheng+Zhang%2C+Stephen+Lin%2C+and+Han+Hu%2E+Video+swin+transformer%2E+In+CVPR%2C+2022%2E)
60. Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. NeurIPS, 2021. [find](https://scholar.google.com/scholar?q=Junnan+Li%2C+Ramprasaath+Selvaraju%2C+Akhilesh+Gotmare%2C+Shafiq+Joty%2C+Caiming+Xiong%2C+and+Steven+Chu+Hong+Hoi%2E+Align+before+fuse%3A+Vision+and+language+representation+learning+with+momentum+distillation%2E+Neur)
61. Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang. Bsn: Boundary sensitive network for temporal action proposal generation. In Proceedings of the European conference on computer vision (ECCV), pages 3–19, 2018. [find](https://scholar.google.com/scholar?q=Tianwei+Lin%2C+Xu+Zhao%2C+Haisheng+Su%2C+Chongjing+Wang%2C+and+Ming+Yang%2E+Bsn%3A+Boundary+sensitive+network+for+temporal+action+proposal+generation%2E+In+Proceedings+of+the+European+conference+on+computer+vision+)
62. Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen. Bmn: Boundary-matching network for temporal action proposal generation. In ICCV, 2019. [find](https://scholar.google.com/scholar?q=Tianwei+Lin%2C+Xiao+Liu%2C+Xin+Li%2C+Errui+Ding%2C+and+Shilei+Wen%2E+Bmn%3A+Boundary%2Dmatching+network+for+temporal+action+proposal+generation%2E+In+ICCV%2C+2019%2E)
63. Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. CVPR, 2020. [find](https://scholar.google.com/scholar?q=Elad+Hoffer%2C+Tal+Ben%2DNun%2C+Itay+Hubara%2C+Niv+Giladi%2C+Torsten+Hoefler%2C+and+Daniel+Soudry%2E+Augment+your+batch%3A+Improving+generalization+through+instance+repetition%2E+CVPR%2C+2020%2E)
64. Kunchang Li, Yali Wang, Gao Peng, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatial-temporal representation learning. In ICLR, 2022. [find](https://scholar.google.com/scholar?q=Kunchang+Li%2C+Yali+Wang%2C+Gao+Peng%2C+Guanglu+Song%2C+Yu+Liu%2C+Hongsheng+Li%2C+and+Yu+Qiao%2E+Uniformer%3A+Unified+transformer+for+efficient+spatial%2Dtemporal+representation+learning%2E+In+ICLR%2C+2022%2E)
65. Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016. [find](https://scholar.google.com/scholar?q=Limin+Wang%2C+Yuanjun+Xiong%2C+Zhe+Wang%2C+Yu+Qiao%2C+Dahua+Lin%2C+Xiaoou+Tang%2C+and+Luc+Van+Gool%2E+Temporal+segment+networks%3A+Towards+good+practices+for+deep+action+recognition%2E+In+ECCV%2C+2016%2E)
66. Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. ICCV, 2019. [find](https://scholar.google.com/scholar?q=Antoine+Miech%2C+Dimitri+Zhukov%2C+Jean%2DBaptiste+Alayrac%2C+Makarand+Tapaswi%2C+Ivan+Laptev%2C+and+Josef+Sivic%2E+Howto100m%3A+Learning+a+text%2Dvideo+embedding+by+watching+hundred+million+narrated+video+clips%2E+ICCV%2C)
67. Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. Ava: A video dataset of spatio-temporally localized atomic visual actions. In CVPR, 2018. [find](https://scholar.google.com/scholar?q=Chunhui+Gu%2C+Chen+Sun%2C+David+A+Ross%2C+Carl+Vondrick%2C+Caroline+Pantofaru%2C+Yeqing+Li%2C+Sudheendra+Vijayanarasimhan%2C+George+Toderici%2C+Susanna+Ricco%2C+Rahul+Sukthankar%2C+et+al%2E+Ava%3A+A+video+dataset+of+spatio%2Dt)
68. Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, 2017. [find](https://scholar.google.com/scholar?q=Raghav+Goyal%2C+Samira+Ebrahimi+Kahou%2C+Vincent+Michalski%2C+Joanna+Materzynska%2C+Susanne+Westphal%2C+Heuna+Kim%2C+Valentin+Haenel%2C+Ingo+Fruend%2C+Peter+Yianilos%2C+Moritz+Mueller%2DFreitag%2C+et+al%2E+The%22+something+som)
69. Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint arXiv:1808.01340, 2018. — [arXiv:1808.01340](https://arxiv.org/abs/1808.01340) [find](https://scholar.google.com/scholar?q=Joao+Carreira%2C+Eric+Noland%2C+Andras+Banki%2DHorvath%2C+Chloe+Hillier%2C+and+Andrew+Zisserman%2E+A+short+note+about+kinetics%2D600%2E+arXiv+preprint+arXiv%3A1808%2E01340%2C+2018%2E+%E2%80%94+%5BarXiv%3A1808%2E01340%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
70. Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics-700 human action dataset. arXiv preprint arXiv:1907.06987, 2019. — [arXiv:1907.06987](https://arxiv.org/abs/1907.06987) [find](https://scholar.google.com/scholar?q=Joao+Carreira%2C+Eric+Noland%2C+Chloe+Hillier%2C+and+Andrew+Zisserman%2E+A+short+note+on+the+kinetics%2D700+human+action+dataset%2E+arXiv+preprint+arXiv%3A1907%2E06987%2C+2019%2E+%E2%80%94+%5BarXiv%3A1907%2E06987%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
71. Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. [Laion-400m: Open dataset of clip-filtered 400 million image-text pairs](https://md.research.elamur.ai/free/en/papers/schuhmann-2021-laion-400m-open-dataset-clip/). arXiv preprint arXiv:2111.02114, 2021.
72. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. [Flamingo: a visual language model for few-shot learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/). arXiv preprint arXiv:2204.14198, 2022.
73. Yuan Tian, Yichao Yan, Guangtao Zhai, Guodong Guo, and Zhiyong Gao. Ean: event adaptive network for enhanced action recognition. IJCV, 2022. [find](https://scholar.google.com/scholar?q=Yuan+Tian%2C+Yichao+Yan%2C+Guangtao+Zhai%2C+Guodong+Guo%2C+and+Zhiyong+Gao%2E+Ean%3A+event+adaptive+network+for+enhanced+action+recognition%2E+IJCV%2C+2022%2E)
74. Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019. [find](https://scholar.google.com/scholar?q=Christoph+Feichtenhofer%2C+Haoqi+Fan%2C+Jitendra+Malik%2C+and+Kaiming+He%2E+Slowfast+networks+for+video+recognition%2E+In+ICCV%2C+2019%2E)
75. Zhiwu Qing, Haisheng Su, Weihao Gan, Dongliang Wang, Wei Wu, Xiang Wang, Yu Qiao, Junjie Yan, Changxin Gao, and Nong Sang. Temporal context aggregation network for temporal action proposal refinement. In CVPR, 2021. [find](https://scholar.google.com/scholar?q=Zhiwu+Qing%2C+Haisheng+Su%2C+Weihao+Gan%2C+Dongliang+Wang%2C+Wei+Wu%2C+Xiang+Wang%2C+Yu+Qiao%2C+Junjie+Yan%2C+Changxin+Gao%2C+and+Nong+Sang%2E+Temporal+context+aggregation+network+for+temporal+action+proposal+refinement%2E)
76. Humam Alwassel, Silvio Giancola, and Bernard Ghanem. Tsp: Temporally-sensitive pretraining of video encoders for localization tasks. In ICCV, 2021. [find](https://scholar.google.com/scholar?q=Humam+Alwassel%2C+Silvio+Giancola%2C+and+Bernard+Ghanem%2E+Tsp%3A+Temporally%2Dsensitive+pretraining+of+video+encoders+for+localization+tasks%2E+In+ICCV%2C+2021%2E)
77. Junting Pan, Siyu Chen, Mike Zheng Shou, Yu Liu, Jing Shao, and Hongsheng Li. Actor-context-actor relation network for spatio-temporal action localization. In CVPR, 2021. [find](https://scholar.google.com/scholar?q=Junting+Pan%2C+Siyu+Chen%2C+Mike+Zheng+Shou%2C+Yu+Liu%2C+Jing+Shao%2C+and+Hongsheng+Li%2E+Actor%2Dcontext%2Dactor+relation+network+for+spatio%2Dtemporal+action+localization%2E+In+CVPR%2C+2021%2E)
78. Yutong Feng, Jianwen Jiang, Ziyuan Huang, Zhiwu Qing, Xiang Wang, Shiwei Zhang, Mingqian Tang, and Yue Gao. Relation modeling in spatio-temporal action localization. arXiv preprint arXiv:2106.08061, 2021. — [arXiv:2106.08061](https://arxiv.org/abs/2106.08061) [find](https://scholar.google.com/scholar?q=Yutong+Feng%2C+Jianwen+Jiang%2C+Ziyuan+Huang%2C+Zhiwu+Qing%2C+Xiang+Wang%2C+Shiwei+Zhang%2C+Mingqian+Tang%2C+and+Yue+Gao%2E+Relation+modeling+in+spatio%2Dtemporal+action+localization%2E+arXiv+preprint+arXiv%3A2106%2E08061%2C+2)
79. Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, 2015. [find](https://scholar.google.com/scholar?q=Fabian+Caba+Heilbron%2C+Victor+Escorcia%2C+Bernard+Ghanem%2C+and+Juan+Carlos+Niebles%2E+Activitynet%3A+A+large%2Dscale+video+benchmark+for+human+activity+understanding%2E+In+CVPR%2C+2015%2E)
80. Hang Zhao, Antonio Torralba, Lorenzo Torresani, and Zhicheng Yan. Hacs: Human action clips and segments dataset for recognition and temporal localization. In ICCV, 2019. [find](https://scholar.google.com/scholar?q=Hang+Zhao%2C+Antonio+Torralba%2C+Lorenzo+Torresani%2C+and+Zhicheng+Yan%2E+Hacs%3A+Human+action+clips+and+segments+dataset+for+recognition+and+temporal+localization%2E+In+ICCV%2C+2019%2E)
81. Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In ICCV, 2011. [find](https://scholar.google.com/scholar?q=Hildegard+Kuehne%2C+Hueihan+Jhuang%2C+Est%C3%ADbaliz+Garrote%2C+Tomaso+Poggio%2C+and+Thomas+Serre%2E+Hmdb%3A+a+large+video+database+for+human+motion+recognition%2E+In+ICCV%2C+2011%2E)
82. Haroon Idrees, Amir R Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. The thumos challenge on action recognition for videos “in the wild”. CVIU, 2017. [find](https://scholar.google.com/scholar?q=Haroon+Idrees%2C+Amir+R+Zamir%2C+Yu%2DGang+Jiang%2C+Alex+Gorban%2C+Ivan+Laptev%2C+Rahul+Sukthankar%2C+and+Mubarak+Shah%2E+The+thumos+challenge+on+action+recognition+for+videos+%E2%80%9Cin+the+wild%E2%80%9D%2E+CVIU%2C+2017%2E)
83. Yi Liu, Limin Wang, Yali Wang, Xiao Ma, and Yu Qiao. Fineaction: A fine-grained video dataset for temporal action localization. IEEE Transactions on Image Processing, 2022. [find](https://scholar.google.com/scholar?q=Yi+Liu%2C+Limin+Wang%2C+Yali+Wang%2C+Xiao+Ma%2C+and+Yu+Qiao%2E+Fineaction%3A+A+fine%2Dgrained+video+dataset+for+temporal+action+localization%2E+IEEE+Transactions+on+Image+Processing%2C+2022%2E)
84. Ang Li, Meghana Thotakuri, David A Ross, João Carreira, Alexander Vostrikov, and Andrew Zisserman. The ava-kinetics localized human actions video dataset. arXiv preprint arXiv:2005.00214, 2020. — [arXiv:2005.00214](https://arxiv.org/abs/2005.00214) [find](https://scholar.google.com/scholar?q=Ang+Li%2C+Meghana+Thotakuri%2C+David+A+Ross%2C+Jo%C3%A3o+Carreira%2C+Alexander+Vostrikov%2C+and+Andrew+Zisserman%2E+The+ava%2Dkinetics+localized+human+actions+video+dataset%2E+arXiv+preprint+arXiv%3A2005%2E00214%2C+2020%2E+%E2%80%94+%5BarX)
85. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In ECCV, 2014.
86. Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cnn. In ICCV, 2017. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Georgia+Gkioxari%2C+Piotr+Dollar%2C+and+Ross+Girshick%2E+Mask+r%2Dcnn%2E+In+ICCV%2C+2017%2E)
87. Jiajun Tang, Jin Xia, Xinzhi Mu, Bo Pang, and Cewu Lu. Asynchronous interaction aggregation for action detection. In ECCV, 2020. [find](https://scholar.google.com/scholar?q=Jiajun+Tang%2C+Jin+Xia%2C+Xinzhi+Mu%2C+Bo+Pang%2C+and+Cewu+Lu%2E+Asynchronous+interaction+aggregation+for+action+detection%2E+In+ECCV%2C+2020%2E)
88. Yuqi Liu, Pengfei Xiong, Luhui Xu, Shengming Cao, and Qin Jin. Ts2-net: Token shift and selection transformer for text-video retrieval. In ECCV, 2022. [find](https://scholar.google.com/scholar?q=Yuqi+Liu%2C+Pengfei+Xiong%2C+Luhui+Xu%2C+Shengming+Cao%2C+and+Qin+Jin%2E+Ts2%2Dnet%3A+Token+shift+and+selection+transformer+for+text%2Dvideo+retrieval%2E+In+ECCV%2C+2022%2E)
89. Yicong Hong, Zun Wang, Qi Wu, and Stephen Gould. Bridging the gap between learning in discrete and continuous environments for vision-and-language navigation. In CVPR, 2022. [find](https://scholar.google.com/scholar?q=Yicong+Hong%2C+Zun+Wang%2C+Qi+Wu%2C+and+Stephen+Gould%2E+Bridging+the+gap+between+learning+in+discrete+and+continuous+environments+for+vision%2Dand%2Dlanguage+navigation%2E+In+CVPR%2C+2022%2E)
90. Dong An, Zun Wang, Yangguang Li, Yi Wang, Yicong Hong, Yan Huang, Liang Wang, and Jing Shao. 1st place solutions for rxr-habitat vision-and-language navigation competition (cvpr 2022). arXiv preprint arXiv:2206.11610, 2022. — [arXiv:2206.11610](https://arxiv.org/abs/2206.11610) [find](https://scholar.google.com/scholar?q=Dong+An%2C+Zun+Wang%2C+Yangguang+Li%2C+Yi+Wang%2C+Yicong+Hong%2C+Yan+Huang%2C+Liang+Wang%2C+and+Jing+Shao%2E+1st+place+solutions+for+rxr%2Dhabitat+vision%2Dand%2Dlanguage+navigation+competition+%28cvpr+2022%29%2E+arXiv+preprint+)
91. Xing Cheng, Hezheng Lin, Xiangyu Wu, Fan Yang, and Dong Shen. Improving video-text retrieval by multi-stream corpus alignment and dual softmax loss. arXiv preprint arXiv:2109.04290, 2021. — [arXiv:2109.04290](https://arxiv.org/abs/2109.04290) [find](https://scholar.google.com/scholar?q=Xing+Cheng%2C+Hezheng+Lin%2C+Xiangyu+Wu%2C+Fan+Yang%2C+and+Dong+Shen%2E+Improving+video%2Dtext+retrieval+by+multi%2Dstream+corpus+alignment+and+dual+softmax+loss%2E+arXiv+preprint+arXiv%3A2109%2E04290%2C+2021%2E+%E2%80%94+%5BarXiv%3A210)
92. Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016. [find](https://scholar.google.com/scholar?q=Jun+Xu%2C+Tao+Mei%2C+Ting+Yao%2C+and+Yong+Rui%2E+Msr%2Dvtt%3A+A+large+video+description+dataset+for+bridging+video+and+language%2E+In+CVPR%2C+2016%2E)
93. Zuxuan Wu, Ting Yao, Yanwei Fu, and Yu-Gang Jiang. Deep learning for video classification and captioning. In Frontiers of Multimedia Research. 2017. [find](https://scholar.google.com/scholar?q=Zuxuan+Wu%2C+Ting+Yao%2C+Yanwei+Fu%2C+and+Yu%2DGang+Jiang%2E+Deep+learning+for+video+classification+and+captioning%2E+In+Frontiers+of+Multimedia+Research%2E+2017%2E)
94. Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In ICCV, 2017. [find](https://scholar.google.com/scholar?q=Lisa+Anne+Hendricks%2C+Oliver+Wang%2C+Eli+Shechtman%2C+Josef+Sivic%2C+Trevor+Darrell%2C+and+Bryan+Russell%2E+Localizing+moments+in+video+with+natural+language%2E+In+ICCV%2C+2017%2E)
95. Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. A dataset for movie description. In CVPR, 2015. [find](https://scholar.google.com/scholar?q=Anna+Rohrbach%2C+Marcus+Rohrbach%2C+Niket+Tandon%2C+and+Bernt+Schiele%2E+A+dataset+for+movie+description%2E+In+CVPR%2C+2015%2E)
96. Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In ICCV, 2019. [find](https://scholar.google.com/scholar?q=Xin+Wang%2C+Jiawei+Wu%2C+Junkun+Chen%2C+Lei+Li%2C+Yuan%2DFang+Wang%2C+and+William+Yang+Wang%2E+Vatex%3A+A+large%2Dscale%2C+high%2Dquality+multilingual+dataset+for+video%2Dand%2Dlanguage+research%2E+In+ICCV%2C+2019%2E)
97. David Chen and William B Dolan. [Collecting highly parallel data for paraphrase evaluation](https://md.research.elamur.ai/free/en/papers/chen-2011-collecting-highly-parallel-data-paraphrase/). In ACL, 2011.
98. Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. Tgif: A new dataset and benchmark on animated gif description. In CVPR, 2016. [find](https://scholar.google.com/scholar?q=Yuncheng+Li%2C+Yale+Song%2C+Liangliang+Cao%2C+Joel+Tetreault%2C+Larry+Goldberg%2C+Alejandro+Jaimes%2C+and+Jiebo+Luo%2E+Tgif%3A+A+new+dataset+and+benchmark+on+animated+gif+description%2E+In+CVPR%2C+2016%2E)
99. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. [Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments](https://md.research.elamur.ai/free/en/papers/anderson-2017-vision-language-navigation-interpreting-visually/). In CVPR, 2018.
100. Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav-graph: Vision-and-language navigation in continuous environments. In ECCV, 2020. [find](https://scholar.google.com/scholar?q=Jacob+Krantz%2C+Erik+Wijmans%2C+Arjun+Majumdar%2C+Dhruv+Batra%2C+and+Stefan+Lee%2E+Beyond+the+nav%2Dgraph%3A+Vision%2Dand%2Dlanguage+navigation+in+continuous+environments%2E+In+ECCV%2C+2020%2E)
101. Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. History aware multimodal transformer for vision-and-language navigation. NeurIPS, 2021. [find](https://scholar.google.com/scholar?q=Shizhe+Chen%2C+Pierre%2DLouis+Guhur%2C+Cordelia+Schmid%2C+and+Ivan+Laptev%2E+History+aware+multimodal+transformer+for+vision%2Dand%2Dlanguage+navigation%2E+NeurIPS%2C+2021%2E)
102. Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. ArXiv, abs/2109.08472, 2021. — [arXiv:2109.08472](https://arxiv.org/abs/2109.08472) [find](https://scholar.google.com/scholar?q=Mengmeng+Wang%2C+Jiazheng+Xing%2C+and+Yong+Liu%2E+Actionclip%3A+A+new+paradigm+for+video+action+recognition%2E+ArXiv%2C+abs%2F2109%2E08472%2C+2021%2E+%E2%80%94+%5BarXiv%3A2109%2E08472%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2109%2E08472%29)
103. Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In CVPR, 2022. [find](https://scholar.google.com/scholar?q=Mitchell+Wortsman%2C+Gabriel+Ilharco%2C+Jong+Wook+Kim%2C+Mike+Li%2C+Simon+Kornblith%2C+Rebecca+Roelofs%2C+Raphael+Gontijo+Lopes%2C+Hannaneh+Hajishirzi%2C+Ali+Farhadi%2C+Hongseok+Namkoong%2C+et+al%2E+Robust+fine%2Dtuning+of+z)
104. Youngjae Yu, Jongseok Kim, and Gunhee Kim. A joint sequence fusion model for video question answering and retrieval. In ECCV, 2018. [find](https://scholar.google.com/scholar?q=Youngjae+Yu%2C+Jongseok+Kim%2C+and+Gunhee+Kim%2E+A+joint+sequence+fusion+model+for+video+question+answering+and+retrieval%2E+In+ECCV%2C+2018%2E)
105. Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In CVPR, 2016. [find](https://scholar.google.com/scholar?q=Abhijit+Bendale+and+Terrance+E+Boult%2E+Towards+open+set+deep+networks%2E+In+CVPR%2C+2016%2E)
106. Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016. [find](https://scholar.google.com/scholar?q=Yarin+Gal+and+Zoubin+Ghahramani%2E+Dropout+as+a+bayesian+approximation%3A+Representing+model+uncertainty+in+deep+learning%2E+In+ICML%2C+2016%2E)
107. Ranganath Krishnan, Mahesh Subedar, and Omesh Tickoo. Bar: Bayesian activity recognition using variational inference. arXiv preprint arXiv:1811.03305, 2018. — [arXiv:1811.03305](https://arxiv.org/abs/1811.03305) [find](https://scholar.google.com/scholar?q=Ranganath+Krishnan%2C+Mahesh+Subedar%2C+and+Omesh+Tickoo%2E+Bar%3A+Bayesian+activity+recognition+using+variational+inference%2E+arXiv+preprint+arXiv%3A1811%2E03305%2C+2018%2E+%E2%80%94+%5BarXiv%3A1811%2E03305%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
108. Guangyao Chen, Limeng Qiao, Yemin Shi, Peixi Peng, Jia Li, Tiejun Huang, Shiliang Pu, and Yonghong Tian. Learning open set network with discriminative reciprocal points. In ECCV, 2020. [find](https://scholar.google.com/scholar?q=Guangyao+Chen%2C+Limeng+Qiao%2C+Yemin+Shi%2C+Peixi+Peng%2C+Jia+Li%2C+Tiejun+Huang%2C+Shiliang+Pu%2C+and+Yonghong+Tian%2E+Learning+open+set+network+with+discriminative+reciprocal+points%2E+In+ECCV%2C+2020%2E)
109. Wentao Bao, Qi Yu, and Yu Kong. Evidential deep learning for open set action recognition. In ICCV, 2021. [find](https://scholar.google.com/scholar?q=Wentao+Bao%2C+Qi+Yu%2C+and+Yu+Kong%2E+Evidential+deep+learning+for+open+set+action+recognition%2E+In+ICCV%2C+2021%2E)
110. Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A dataset of 101 human action classes from videos in the wild. Center for Research in Computer Vision, 2012. [find](https://scholar.google.com/scholar?q=Khurram+Soomro%2C+Amir+Roshan+Zamir%2C+and+Mubarak+Shah%2E+A+dataset+of+101+human+action+classes+from+videos+in+the+wild%2E+Center+for+Research+in+Computer+Vision%2C+2012%2E)
111. Mathew Monfort, Bowen Pan, Kandan Ramakrishnan, Alex Andonian, Barry A McNamara, Alex Lascelles, Quanfu Fan, Dan Gutfreund, Rogerio Feris, and Aude Oliva. Multi-moments in time: Learning and interpreting models for multi-action video understanding. TPAMI, 2021. [find](https://scholar.google.com/scholar?q=Mathew+Monfort%2C+Bowen+Pan%2C+Kandan+Ramakrishnan%2C+Alex+Andonian%2C+Barry+A+McNamara%2C+Alex+Lascelles%2C+Quanfu+Fan%2C+Dan+Gutfreund%2C+Rogerio+Feris%2C+and+Aude+Oliva%2E+Multi%2Dmoments+in+time%3A+Learning+and+interpret)

*Updated 24 October 2026*