---
title: "HERO — paper by Li et al. (2020) — summary and ideas"
description: "The authors also release two new benchmarks, How2R and How2QA. 76 references, cited by 65 papers here."
authors: ["Linjie Li", "Yen-Chun Chen", "Yu Cheng", "Zhe Gan", "Licheng Yu", "Jingjing Liu"]
year: 2020
venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/2020.emnlp-main.161/"
doi: "10.18653/v1/2020.emnlp-main.161"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2020-hero-hierarchical-encoder-video-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2020-hero-hierarchical-encoder-video-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/) / HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training

# HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training

**Authors:** [Linjie Li](https://md.research.elamur.ai/free/en/authors/linjie-li/), [Yen-Chun Chen](https://md.research.elamur.ai/free/en/authors/yen-chun-chen/), [Yu Cheng](https://md.research.elamur.ai/free/en/authors/yu-cheng/), [Zhe Gan](https://md.research.elamur.ai/free/en/authors/zhe-gan/), [Licheng Yu](https://md.research.elamur.ai/free/en/authors/licheng-yu/) and [Jingjing Liu](https://md.research.elamur.ai/free/en/authors/jingjing-liu/) · **Year:** 2020 · **Venue:** [Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2020](https://md.research.elamur.ai/free/en/venues/emnlp/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-main.161/) · [PDF](https://aclanthology.org/2020.emnlp-main.161.pdf) · [DOI 10.18653/v1/2020.emnlp-main.161](https://doi.org/10.18653/v1/2020.emnlp-main.161) · [OpenAlex](https://openalex.org/W3105232955)

**Citations:** 403 citations (OpenAlex, 2026-10-14) · [65](https://md.research.elamur.ai/free/en/papers/li-2020-hero-hierarchical-encoder-video-language/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
HERO addresses a gap in video+language pre-training. Earlier models mostly adapted BERT by concatenating subtitles and frames into a single flat sequence, which discards the timing links between the two modalities. Their pre-training objectives were also borrowed from image-text work and ignored the sequential nature of video, and their training data was limited to instructional content.

HERO works in two stages:
- A Cross-modal Transformer fuses each subtitle sentence with the frames that share its timestamps.
- A Temporal Transformer then contextualizes every frame across the whole clip.

Frame features combine ResNet (2D) and SlowFast (3D) representations. Alongside masked language and frame modeling, the authors add two objectives: Video-Subtitle Matching, which covers clip-level matching and moment localization, and Frame Order Modeling, which recovers the order of shuffled frames. Pre-training uses HowTo100M together with a large TV-episode corpus, so the model also sees social interactions and narrative plots.

HERO sets new state-of-the-art results on retrieval, QA, inference and captioning benchmarks. The authors also release two new benchmarks, How2R and How2QA.

## Key points
- On TVR, R@1 roughly doubles compared with XML. HERO also beats prior systems by +3.28% on TVQA (STAGE) and +0.75% on VIOLIN, without using region-level features. On TVC it gains +4.60 on CIDEr over MMT.
- In the val-set ablation with pre-training, the flat BERT-like encoder (F-TRM) reaches TVR R@1 2.69 and TVQA accuracy 49.12. HERO reaches 4.44 and 72.75. Pre-training helps HERO far more than it helps the flat or simple hierarchical baselines.
- The best pre-training setup is MLM + MNCE + FOM + VSM on combined TV and HowTo100M data. FOM mainly helps temporal-reasoning QA, while VSM mainly boosts moment retrieval.
- On single-channel tasks, HERO improves R@1/10/100 on DiDeMo by +0.55/+4.72/+10.65 over XML. On MSR-VTT it improves R@1/5/10 by +1.9/+3.2/+4.9 over HowTo100M-based pre-training. Adding ASR subtitles improves both further.
- Code and the new How2R and How2QA datasets are available at github.com/linjieli222/HERO.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 240 works · 123,537 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 269 works

## Cite

Cite as: Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu and Jingjing Liu. 2020. HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{li2020hero,
  title = {{HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training}},
  author = {Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu and Jingjing Liu},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.161},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.161}
}
```

- [← previous GCAN: Graph-aware Co-Attention Networks for Explainable Fake News Detection on Social Media](https://md.research.elamur.ai/free/en/papers/lu-2020-gcan-graph-aware-co-attention/)
- [next → XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://md.research.elamur.ai/free/en/papers/ponti-2020-xcopa-multilingual-dataset-causal-commonsense/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/)

## References

76 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Chris Alberti, Jeffrey Ling, Michael Collins, and David Reitter. 2019. Fusion of detected objects in text for visual question answering. In EMNLP. [find](https://scholar.google.com/scholar?q=Chris+Alberti%2C+Jeffrey+Ling%2C+Michael+Collins%2C+and+David+Reitter%2E+2019%2E+Fusion+of+detected+objects+in+text+for+visual+question+answering%2E+In+EMNLP%2E)
2. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR. [find](https://scholar.google.com/scholar?q=Peter+Anderson%2C+Xiaodong+He%2C+Chris+Buehler%2C+Damien+Teney%2C+Mark+Johnson%2C+Stephen+Gould%2C+and+Lei+Zhang%2E+2018%2E+Bottom%2Dup+and+top%2Ddown+attention+for+image+captioning+and+visual+question+answering%2E+In+CVPR)
3. Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2017a. Localizing moments in video with natural language. In ICCV. [find](https://scholar.google.com/scholar?q=Lisa+Anne+Hendricks%2C+Oliver+Wang%2C+Eli+Shechtman%2C+Josef+Sivic%2C+Trevor+Darrell%2C+and+Bryan+Russell%2E+2017a%2E+Localizing+moments+in+video+with+natural+language%2E+In+ICCV%2E)
4. Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2017b. Localizing moments in video with natural language. In CVPR. [find](https://scholar.google.com/scholar?q=Lisa+Anne+Hendricks%2C+Oliver+Wang%2C+Eli+Shechtman%2C+Josef+Sivic%2C+Trevor+Darrell%2C+and+Bryan+Russell%2E+2017b%2E+Localizing+moments+in+video+with+natural+language%2E+In+CVPR%2E)
5. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP. [find](https://scholar.google.com/scholar?q=Samuel+R+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+EMNLP%2E)
6. Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, and Jingjing Liu. 2020a. Distilling the knowledge of bert for text generation. In ACL. [find](https://scholar.google.com/scholar?q=Yen%2DChun+Chen%2C+Zhe+Gan%2C+Yu+Cheng%2C+Jingzhou+Liu%2C+and+Jingjing+Liu%2E+2020a%2E+Distilling+the+knowledge+of+bert+for+text+generation%2E+In+ACL%2E)
7. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020b. [Uniter: Universal image-text representation learning](https://md.research.elamur.ai/free/en/papers/chen-2019-uniter-universal-image-text-representation/). In ECCV.
8. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In CVPR. [find](https://scholar.google.com/scholar?q=Jia+Deng%2C+Wei+Dong%2C+Richard+Socher%2C+Li%2DJia+Li%2C+Kai+Li%2C+and+Li+Fei%2DFei%2E+2009%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+CVPR%2E)
9. Michael Denkowski and Alon Lavie. 2014. [Meteor universal: Language specific translation evaluation for any target language](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/). In Proceedings of the ninth workshop on statistical machine translation.
10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In NAACL.
11. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. [Unified language model pre-training for natural language understanding and generation](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/). In NeurIPS.
12. Victor Escorcia, Mattia Soldan, Josef Sivic, Bernard Ghanem, and Bryan Russell. 2019. Temporal localization of moments in video collections with natural language. arXiv preprint arXiv:1907.12763. — [arXiv:1907.12763](https://arxiv.org/abs/1907.12763) [find](https://scholar.google.com/scholar?q=Victor+Escorcia%2C+Mattia+Soldan%2C+Josef+Sivic%2C+Bernard+Ghanem%2C+and+Bryan+Russell%2E+2019%2E+Temporal+localization+of+moments+in+video+collections+with+natural+language%2E+arXiv+preprint+arXiv%3A1907%2E12763%2E+%E2%80%94+%5Ba)
13. Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. 2019. Slowfast networks for video recognition. In ICCV. [find](https://scholar.google.com/scholar?q=Christoph+Feichtenhofer%2C+Haoqi+Fan%2C+Jitendra+Malik%2C+and+Kaiming+He%2E+2019%2E+Slowfast+networks+for+video+recognition%2E+In+ICCV%2E)
14. Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. 2020. Large-scale adversarial training for vision-and-language representation learning. arXiv preprint arXiv:2006.06195. [find](https://scholar.google.com/scholar?q=Zhe+Gan%2C+Yen%2DChun+Chen%2C+Linjie+Li%2C+Chen+Zhu%2C+Yu+Cheng%2C+and+Jingjing+Liu%2E+2020%2E+Large%2Dscale+adversarial+training+for+vision%2Dand%2Dlanguage+representation+learning%2E+arXiv+preprint+arXiv%3A2006%2E06195%2E)
15. Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, and Li Deng. 2017. Semantic compositional networks for visual captioning. In CVPR. [find](https://scholar.google.com/scholar?q=Zhe+Gan%2C+Chuang+Gan%2C+Xiaodong+He%2C+Yunchen+Pu%2C+Kenneth+Tran%2C+Jianfeng+Gao%2C+Lawrence+Carin%2C+and+Li+Deng%2E+2017%2E+Semantic+compositional+networks+for+visual+captioning%2E+In+CVPR%2E)
16. Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. 2017. Tall: Temporal activity localization via language query. In CVPR. [find](https://scholar.google.com/scholar?q=Jiyang+Gao%2C+Chen+Sun%2C+Zhenheng+Yang%2C+and+Ram+Nevatia%2E+2017%2E+Tall%3A+Temporal+activity+localization+via+language+query%2E+In+CVPR%2E)
17. Sergio Guadarrama, Niveda Krishnamoorthy, Girish Malkarnenkar, Subhashini Venugopalan, Raymond Mooney, Trevor Darrell, and Kate Saenko. 2013. Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition. In ICCV. [find](https://scholar.google.com/scholar?q=Sergio+Guadarrama%2C+Niveda+Krishnamoorthy%2C+Girish+Malkarnenkar%2C+Subhashini+Venugopalan%2C+Raymond+Mooney%2C+Trevor+Darrell%2C+and+Kate+Saenko%2E+2013%2E+Youtube2text%3A+Recognizing+and+describing+arbitrary+activit)
18. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+2016%2E+Deep+residual+learning+for+image+recognition%2E+In+CVPR%2E)
19. Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017. Tgif-qa: Toward spatiotemporal reasoning in visual question answering. In CVPR. [find](https://scholar.google.com/scholar?q=Yunseok+Jang%2C+Yale+Song%2C+Youngjae+Yu%2C+Youngjin+Kim%2C+and+Gunhee+Kim%2E+2017%2E+Tgif%2Dqa%3A+Toward+spatiotemporal+reasoning+in+visual+question+answering%2E+In+CVPR%2E)
20. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410. [find](https://scholar.google.com/scholar?q=Rafal+Jozefowicz%2C+Oriol+Vinyals%2C+Mike+Schuster%2C+Noam+Shazeer%2C+and+Yonghui+Wu%2E+2016%2E+Exploring+the+limits+of+language+modeling%2E+arXiv+preprint+arXiv%3A1602%2E02410%2E)
21. Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. 2017. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950. — [arXiv:1705.06950](https://arxiv.org/abs/1705.06950) [find](https://scholar.google.com/scholar?q=Will+Kay%2C+Joao+Carreira%2C+Karen+Simonyan%2C+Brian+Zhang%2C+Chloe+Hillier%2C+Sudheendra+Vijayanarasimhan%2C+Fabio+Viola%2C+Tim+Green%2C+Trevor+Back%2C+Paul+Natsev%2C+et+al%2E+2017%2E+The+kinetics+human+action+video+dataset)
22. Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. [Revealing the dark secrets of bert](https://md.research.elamur.ai/free/en/papers/kovaleva-2019-revealing-dark-secrets-bert/). In EMNLP.
23. Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017. Dense-captioning events in videos. In ICCV. [find](https://scholar.google.com/scholar?q=Ranjay+Krishna%2C+Kenji+Hata%2C+Frederic+Ren%2C+Li+Fei%2DFei%2C+and+Juan+Carlos+Niebles%2E+2017%2E+Dense%2Dcaptioning+events+in+videos%2E+In+ICCV%2E)
24. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In ICLR. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2020%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+In+ICLR%2E)
25. Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. 2018. [Tvqa: Localized, compositional video question answering](https://md.research.elamur.ai/free/en/papers/lei-2018-tvqa-localized-compositional-video-question/). In EMNLP.
26. Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. 2020a. Tvqa+: Spatio-temporal grounding for video question answering. In ACL. [find](https://scholar.google.com/scholar?q=Jie+Lei%2C+Licheng+Yu%2C+Tamara+L+Berg%2C+and+Mohit+Bansal%2E+2020a%2E+Tvqa%2B%3A+Spatio%2Dtemporal+grounding+for+video+question+answering%2E+In+ACL%2E)
27. Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. 2020b. Tvr: A large-scale dataset for video-subtitle moment retrieval. In ECCV. [find](https://scholar.google.com/scholar?q=Jie+Lei%2C+Licheng+Yu%2C+Tamara+L+Berg%2C+and+Mohit+Bansal%2E+2020b%2E+Tvr%3A+A+large%2Dscale+dataset+for+video%2Dsubtitle+moment+retrieval%2E+In+ECCV%2E)
28. Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou. 2020a. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In AAAI. [find](https://scholar.google.com/scholar?q=Gen+Li%2C+Nan+Duan%2C+Yuejian+Fang%2C+Daxin+Jiang%2C+and+Ming+Zhou%2E+2020a%2E+Unicoder%2Dvl%3A+A+universal+encoder+for+vision+and+language+by+cross%2Dmodal+pre%2Dtraining%2E+In+AAAI%2E)
29. Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. [Visualbert: A simple and performant baseline for vision and language](https://md.research.elamur.ai/free/en/papers/li-2019-visualbert-simple-performant-baseline-vision/). arXiv preprint arXiv:1908.03557.
30. Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020b. [Oscar: Object-semantics aligned pre-training for vision-language tasks](https://md.research.elamur.ai/free/en/papers/li-2020-oscar-object-semantics-aligned-pre/). In ECCV.
31. R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2019. Learning deep representations by mutual information estimation and maximization. In ICLR. [find](https://scholar.google.com/scholar?q=R+Devon+Hjelm%2C+Alex+Fedorov%2C+Samuel+Lavoie%2DMarchildon%2C+Karan+Grewal%2C+Phil+Bachman%2C+Adam+Trischler%2C+and+Yoshua+Bengio%2E+2019%2E+Learning+deep+representations+by+mutual+information+estimation+and+maximizat)
32. Chin-Yew Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In ACL.
33. Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. 2020. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.00849. [find](https://scholar.google.com/scholar?q=Zhicheng+Huang%2C+Zhaoyang+Zeng%2C+Bei+Liu%2C+Dongmei+Fu%2C+and+Jianlong+Fu%2E+2020%2E+Pixel%2Dbert%3A+Aligning+image+pixels+with+text+by+deep+multi%2Dmodal+transformers%2E+arXiv+preprint+arXiv%3A2004%2E00849%2E)
34. Jingzhou Liu, Wenhu Chen, Yu Cheng, Zhe Gan, Licheng Yu, Yiming Yang, and Jingjing Liu. 2020. Violin: A large-scale dataset for video-and-language inference. In CVPR. [find](https://scholar.google.com/scholar?q=Jingzhou+Liu%2C+Wenhu+Chen%2C+Yu+Cheng%2C+Zhe+Gan%2C+Licheng+Yu%2C+Yiming+Yang%2C+and+Jingjing+Liu%2E+2020%2E+Violin%3A+A+large%2Dscale+dataset+for+video%2Dand%2Dlanguage+inference%2E+In+CVPR%2E)
35. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
36. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). arXiv preprint arXiv:1910.10683.
37. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+ICLR%2E)
38. Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. 2020. Q-bert: Hessian based ultra low precision quantization of bert. In AAAI. [find](https://scholar.google.com/scholar?q=Sheng+Shen%2C+Zhen+Dong%2C+Jiayu+Ye%2C+Linjian+Ma%2C+Zhewei+Yao%2C+Amir+Gholami%2C+Michael+W+Mahoney%2C+and+Kurt+Keutzer%2E+2020%2E+Q%2Dbert%3A+Hessian+based+ultra+low+precision+quantization+of+bert%2E+In+AAAI%2E)
39. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. [Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks](https://md.research.elamur.ai/free/en/papers/lu-2019-vilbert-pretraining-task-agnostic-visiolinguistic/). In NeurIPS.
40. Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 2020. 12-in-1: Multi-task vision and language representation learning. In CVPR. [find](https://scholar.google.com/scholar?q=Jiasen+Lu%2C+Vedanuj+Goswami%2C+Marcus+Rohrbach%2C+Devi+Parikh%2C+and+Stefan+Lee%2E+2020%2E+12%2Din%2D1%3A+Multi%2Dtask+vision+and+language+representation+learning%2E+In+CVPR%2E)
41. Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Xilin Chen, and Ming Zhou. 2020. Univilm: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.06353. [find](https://scholar.google.com/scholar?q=Huaishao+Luo%2C+Lei+Ji%2C+Botian+Shi%2C+Haoyang+Huang%2C+Nan+Duan%2C+Tianrui+Li%2C+Xilin+Chen%2C+and+Ming+Zhou%2E+2020%2E+Univilm%3A+A+unified+video+and+language+pre%2Dtraining+model+for+multimodal+understanding+and+genera)
42. Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron Courville, and Christopher Pal. 2017. A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering. In CVPR. [find](https://scholar.google.com/scholar?q=Tegan+Maharaj%2C+Nicolas+Ballas%2C+Anna+Rohrbach%2C+Aaron+Courville%2C+and+Christopher+Pal%2E+2017%2E+A+dataset+and+exploration+of+models+for+understanding+video+data+through+fill%2Din%2Dthe%2Dblank+question%2Danswering%2E)
43. Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2020. End-to-end learning of visual representations from uncurated instructional videos. In CVPR. [find](https://scholar.google.com/scholar?q=Antoine+Miech%2C+Jean%2DBaptiste+Alayrac%2C+Lucas+Smaira%2C+Ivan+Laptev%2C+Josef+Sivic%2C+and+Andrew+Zisserman%2E+2020%2E+End%2Dto%2Dend+learning+of+visual+representations+from+uncurated+instructional+videos%2E+In+CVPR%2E)
44. Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV. [find](https://scholar.google.com/scholar?q=Antoine+Miech%2C+Dimitri+Zhukov%2C+Jean%2DBaptiste+Alayrac%2C+Makarand+Tapaswi%2C+Ivan+Laptev%2C+and+Josef+Sivic%2E+2019%2E+Howto100m%3A+Learning+a+text%2Dvideo+embedding+by+watching+hundred+million+narrated+video+clips%2E)
45. Vishvak Murahari, Dhruv Batra, Devi Parikh, and Abhishek Das. 2020. Large-scale pretraining for visual dialog: A simple state-of-the-art baseline. In ECCV. [find](https://scholar.google.com/scholar?q=Vishvak+Murahari%2C+Dhruv+Batra%2C+Devi+Parikh%2C+and+Abhishek+Das%2E+2020%2E+Large%2Dscale+pretraining+for+visual+dialog%3A+A+simple+state%2Dof%2Dthe%2Dart+baseline%2E+In+ECCV%2E)
46. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. [Representation learning with contrastive predictive coding](https://md.research.elamur.ai/free/en/papers/oord-2018-representation-learning-contrastive-predictive-coding/). arXiv preprint arXiv:1807.03748.
47. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. [Scaling neural machine translation](https://md.research.elamur.ai/free/en/papers/ott-2018-scaling-neural-machine-translation/). arXiv preprint arXiv:1806.00187.
48. Yingwei Pan, Tao Mei, Ting Yao, Houqiang Li, and Yong Rui. 2016. Jointly modeling embedding and translation to bridge video and language. In CVPR. [find](https://scholar.google.com/scholar?q=Yingwei+Pan%2C+Tao+Mei%2C+Ting+Yao%2C+Houqiang+Li%2C+and+Yong+Rui%2E+2016%2E+Jointly+modeling+embedding+and+translation+to+bridge+video+and+language%2E+In+CVPR%2E)
49. Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020. [Vl-bert: Pre-training of generic visual-linguistic representations](https://md.research.elamur.ai/free/en/papers/su-2019-vl-bert-pre-training-generic/). In ICLR.
50. Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. 2019a. Contrastive bidirectional transformer for temporal representation learning. arXiv preprint arXiv:1906.05743. — [arXiv:1906.05743](https://arxiv.org/abs/1906.05743) [find](https://scholar.google.com/scholar?q=Chen+Sun%2C+Fabien+Baradel%2C+Kevin+Murphy%2C+and+Cordelia+Schmid%2E+2019a%2E+Contrastive+bidirectional+transformer+for+temporal+representation+learning%2E+arXiv+preprint+arXiv%3A1906%2E05743%2E+%E2%80%94+%5BarXiv%3A1906%2E05743%5D%28ht)
51. Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019b. [Videobert: A joint model for video and language representation learning](https://md.research.elamur.ai/free/en/papers/sun-2019-videobert-joint-model-video-language/). In ICCV.
52. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019c. [Patient knowledge distillation for bert model compression](https://md.research.elamur.ai/free/en/papers/sun-2019-patient-knowledge-distillation-bert-model/). In EMNLP.
53. Hao Tan and Mohit Bansal. 2019. [Lxmert: Learning cross-modality encoder representations from transformers](https://md.research.elamur.ai/free/en/papers/tan-2019-lxmert-learning-cross-modality-encoder/). In EMNLP.
54. Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016. Movieqa: Understanding stories in movies through question-answering. In CVPR. [find](https://scholar.google.com/scholar?q=Makarand+Tapaswi%2C+Yukun+Zhu%2C+Rainer+Stiefelhagen%2C+Antonio+Torralba%2C+Raquel+Urtasun%2C+and+Sanja+Fidler%2E+2016%2E+Movieqa%3A+Understanding+stories+in+movies+through+question%2Danswering%2E+In+CVPR%2E)
55. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NeurIPS.
56. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. [Cider: Consensus-based image description evaluation](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/). In CVPR.
57. Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. 2015. Sequence to sequence-video to text. In ICCV. [find](https://scholar.google.com/scholar?q=Subhashini+Venugopalan%2C+Marcus+Rohrbach%2C+Jeffrey+Donahue%2C+Raymond+Mooney%2C+Trevor+Darrell%2C+and+Kate+Saenko%2E+2015%2E+Sequence+to+sequence%2Dvideo+to+text%2E+In+ICCV%2E)
58. Alex Wang and Kyunghyun Cho. 2019. [Bert has a mouth, and it must speak: Bert as a markov random field language model](https://md.research.elamur.ai/free/en/papers/wang-2019-bert-has-mouth-must-speak/). arXiv preprint arXiv:1902.04094.
59. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In ACL.
60. Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. 2019. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In ICCV. [find](https://scholar.google.com/scholar?q=Xin+Wang%2C+Jiawei+Wu%2C+Junkun+Chen%2C+Lei+Li%2C+Yuan%2DFang+Wang%2C+and+William+Yang+Wang%2E+2019%2E+Vatex%3A+A+large%2Dscale%2C+high%2Dquality+multilingual+dataset+for+video%2Dand%2Dlanguage+research%2E+In+ICCV%2E)
61. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. [find](https://scholar.google.com/scholar?q=Adam+Paszke%2C+Sam+Gross%2C+Soumith+Chintala%2C+Gregory+Chanan%2C+Edward+Yang%2C+Zachary+DeVito%2C+Zeming+Lin%2C+Alban+Desmaison%2C+Luca+Antiga%2C+and+Adam+Lerer%2E+2017%2E+Automatic+differentiation+in+pytorch%2E)
62. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+R+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+NAACL%2E)
63. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+et+al%2E+2016%2E+Google%E2%80%99s+neural+machine+translation+system%3A+Bridg)
64. Qiaolin Xia, Haoyang Huang, Nan Duan, Dongdong Zhang, Lei Ji, Zhifang Sui, Edward Cui, Taroon Bharti, and Ming Zhou. 2020. Xgpt: Cross-modal generative pre-training for image captioning. arXiv preprint arXiv:2003.01473. — [arXiv:2003.01473](https://arxiv.org/abs/2003.01473) [find](https://scholar.google.com/scholar?q=Qiaolin+Xia%2C+Haoyang+Huang%2C+Nan+Duan%2C+Dongdong+Zhang%2C+Lei+Ji%2C+Zhifang+Sui%2C+Edward+Cui%2C+Taroon+Bharti%2C+and+Ming+Zhou%2E+2020%2E+Xgpt%3A+Cross%2Dmodal+generative+pre%2Dtraining+for+image+captioning%2E+arXiv+preprin)
65. Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. 2019. Visual entailment: A novel task for fine-grained image understanding. arXiv preprint arXiv:1901.06706. — [arXiv:1901.06706](https://arxiv.org/abs/1901.06706) [find](https://scholar.google.com/scholar?q=Ning+Xie%2C+Farley+Lai%2C+Derek+Doran%2C+and+Asim+Kadav%2E+2019%2E+Visual+entailment%3A+A+novel+task+for+fine%2Dgrained+image+understanding%2E+arXiv+preprint+arXiv%3A1901%2E06706%2E+%E2%80%94+%5BarXiv%3A1901%2E06706%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
66. Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016a. Msr-vtt: A large video description dataset for bridging video and language. In CVPR. [find](https://scholar.google.com/scholar?q=Jun+Xu%2C+Tao+Mei%2C+Ting+Yao%2C+and+Yong+Rui%2E+2016a%2E+Msr%2Dvtt%3A+A+large+video+description+dataset+for+bridging+video+and+language%2E+In+CVPR%2E)
67. Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016b. Msr-vtt: A large video description dataset for bridging video and language. CVPR. [find](https://scholar.google.com/scholar?q=Jun+Xu%2C+Tao+Mei%2C+Ting+Yao%2C+and+Yong+Rui%2E+2016b%2E+Msr%2Dvtt%3A+A+large+video+description+dataset+for+bridging+video+and+language%2E+CVPR%2E)
68. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). In NeurIPS.
69. Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. 2018a. Mattnet: Modular attention network for referring expression comprehension. In CVPR. [find](https://scholar.google.com/scholar?q=Licheng+Yu%2C+Zhe+Lin%2C+Xiaohui+Shen%2C+Jimei+Yang%2C+Xin+Lu%2C+Mohit+Bansal%2C+and+Tamara+L+Berg%2E+2018a%2E+Mattnet%3A+Modular+attention+network+for+referring+expression+comprehension%2E+In+CVPR%2E)
70. Youngjae Yu, Jongseok Kim, and Gunhee Kim. 2018b. A joint sequence fusion model for video question answering and retrieval. In ECCV. [find](https://scholar.google.com/scholar?q=Youngjae+Yu%2C+Jongseok+Kim%2C+and+Gunhee+Kim%2E+2018b%2E+A+joint+sequence+fusion+model+for+video+question+answering+and+retrieval%2E+In+ECCV%2E)
71. Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In CVPR. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+From+recognition+to+cognition%3A+Visual+commonsense+reasoning%2E+In+CVPR%2E)
72. Luowei Zhou, Yannis Kalantidis, Xinlei Chen, Jason J Corso, and Marcus Rohrbach. 2019. Grounded video description. In CVPR. [find](https://scholar.google.com/scholar?q=Luowei+Zhou%2C+Yannis+Kalantidis%2C+Xinlei+Chen%2C+Jason+J+Corso%2C+and+Marcus+Rohrbach%2E+2019%2E+Grounded+video+description%2E+In+CVPR%2E)
73. Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J Corso, and Jianfeng Gao. 2020. Unified vision-language pre-training for image captioning and vqa. In AAAI. [find](https://scholar.google.com/scholar?q=Luowei+Zhou%2C+Hamid+Palangi%2C+Lei+Zhang%2C+Houdong+Hu%2C+Jason+J+Corso%2C+and+Jianfeng+Gao%2E+2020%2E+Unified+vision%2Dlanguage+pre%2Dtraining+for+image+captioning+and+vqa%2E+In+AAAI%2E)
74. Luowei Zhou, Chenliang Xu, and Jason J Corso. 2018a. Towards automatic learning of procedures from web instructional videos. In AAAI. [find](https://scholar.google.com/scholar?q=Luowei+Zhou%2C+Chenliang+Xu%2C+and+Jason+J+Corso%2E+2018a%2E+Towards+automatic+learning+of+procedures+from+web+instructional+videos%2E+In+AAAI%2E)
75. Luowei Zhou, Yingbo Zhou, Jason J Corso, Richard Socher, and Caiming Xiong. 2018b. End-to-end dense video captioning with masked transformer. In CVPR. [find](https://scholar.google.com/scholar?q=Luowei+Zhou%2C+Yingbo+Zhou%2C+Jason+J+Corso%2C+Richard+Socher%2C+and+Caiming+Xiong%2E+2018b%2E+End%2Dto%2Dend+dense+video+captioning+with+masked+transformer%2E+In+CVPR%2E)
76. Linchao Zhu and Yi Yang. 2020. Actbert: Learning global-local video-text representations. In CVPR. [find](https://scholar.google.com/scholar?q=Linchao+Zhu+and+Yi+Yang%2E+2020%2E+Actbert%3A+Learning+global%2Dlocal+video%2Dtext+representations%2E+In+CVPR%2E)

*Updated 24 October 2026*