---
title: "VL-BERT: Pre-training of Generic Visual-Linguistic — Su 2019"
description: "Vision-and-language tasks such as VQA, visual commonsense reasoning (VCR) and referring expression comprehension have usually relied on task-specific…"
authors: ["Weijie Su", "Xizhou Zhu", "Yue Cao", "Bin Li", "Lewei Lu", "Furu Wei", "Jifeng Dai"]
year: 2019
venue: "arXiv:1908.08530"
source: "https://arxiv.org/abs/1908.08530"
doi: "10.48550/arxiv.1908.08530"
arxiv: "1908.08530"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/su-2019-vl-bert-pre-training-generic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/su-2019-vl-bert-pre-training-generic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / VL-BERT: Pre-training of Generic Visual-Linguistic Representations

# VL-BERT: Pre-training of Generic Visual-Linguistic Representations

**Authors:** [Weijie Su](https://md.research.elamur.ai/free/en/authors/weijie-su/), [Xizhou Zhu](https://md.research.elamur.ai/free/en/authors/xizhou-zhu/), [Yue Cao](https://md.research.elamur.ai/free/en/authors/yue-cao/), [Bin Li](https://md.research.elamur.ai/free/en/authors/bin-li/), [Lewei Lu](https://md.research.elamur.ai/free/en/authors/lewei-lu/), [Furu Wei](https://md.research.elamur.ai/free/en/authors/furu-wei/), Jifeng Dai · **Year:** 2019 · **Venue:** arXiv:1908.08530 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1908.08530) · [PDF](https://arxiv.org/pdf/1908.08530) · [DOI 10.48550/arxiv.1908.08530](https://doi.org/10.48550/arxiv.1908.08530) · [arXiv 1908.08530](https://arxiv.org/abs/1908.08530) · [OpenAlex](https://openalex.org/W2969876226)

**Citations:** 753 citations (openalex-snapshot · 2026-10-14) · [255](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Vision-and-language tasks such as VQA, visual commonsense reasoning (VCR) and referring expression comprehension have usually relied on task-specific architectures. These glue together separately pre-trained image and text networks without any joint pre-training, so they tend to overfit when labelled data is limited. VL-BERT proposes one generic, pre-trainable representation for such tasks. It is a single-stream Transformer in the style of BERT. Its input sequence mixes WordPiece tokens from a sentence with region-of-interest (RoI) features from an image, produced by a Fast R-CNN detector, plus special elements that mark different input formats. Attention between all elements is unrestricted, so the two modalities interact from the first layer onward. The model is pre-trained on Conceptual Captions, where it predicts masked words and masked RoI categories, and on text-only corpora with standard MLM. Fine-tuned on downstream benchmarks, it reportedly matches or surpasses earlier methods, including two-stream designs such as ViLBERT and LXMERT. It also took first place among single models on the VCR leaderboard. This makes it a strong, reusable foundation for multimodal research, and its code is publicly released.

## Key points
- **Pre-training data:** Conceptual Captions supplies the image–caption pairs, and BooksCorpus plus English Wikipedia supply text-only data. Mixing in plain text helps the model handle long, complex sentences.
- **Pre-training tasks:** two objectives are used, masked language modeling with visual clues and masked RoI classification with linguistic clues. The commonly used Sentence-Image Relationship Prediction task was dropped because the authors found it gave no benefit.
- **Detector training and masking:** the Fast R-CNN parameters are trained jointly with the Transformer. RoI masking zeroes out raw image pixels, rather than feature maps, so that no visual information leaks into the prediction target.
- **Pre-training setup:** 16 Tesla V100 GPUs ran 250k iterations with the Adam optimizer. Each 256-sample batch held 128 caption–image pairs and 128 text sequences of up to 64 tokens.
- **Results:** the model was evaluated on VCR, VQA and RefCOCO+ and ranked first as a single model on the VCR leaderboard. Attention maps visualized with BertViz show some heads linking words to their matching image regions. Code: github.com/jackroos/VL-BERT.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

255 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 235 not yet.

- [Flamingo: a Visual Language Model for Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
- [Building Common Ground in Dialogue: A Survey](https://md.research.elamur.ai/free/en/papers/anikina-2025-building-common-ground-dialogue-survey/)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/)
- [UNITER: UNiversal Image-TExt Representation Learning](https://md.research.elamur.ai/free/en/papers/chen-2019-uniter-universal-image-text-representation/)
- [GRAVL-BERT: Graphical Visual-Linguistic Representations for Multimodal Coreference Resolution](https://md.research.elamur.ai/free/en/papers/guo-2022-gravl-bert-graphical-visual-linguistic/)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://md.research.elamur.ai/free/en/papers/huang-2022-layoutlmv3-pre-training-document-ai/)
- [Probing Contextual Language Models for Common Ground with Visual Representations](https://md.research.elamur.ai/free/en/papers/ilharco-2021-probing-contextual-language-models-common/)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://md.research.elamur.ai/free/en/papers/kim-2021-vilt-vision-language-transformer-convolution/)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://md.research.elamur.ai/free/en/papers/li-2020-oscar-object-semantics-aligned-pre/)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/)
- [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://md.research.elamur.ai/free/en/papers/tsimpoukelli-2021-multimodal-few-shot-learning-frozen/)
- [DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation](https://md.research.elamur.ai/free/en/papers/valipour-2023-dylora-parameter-efficient-tuning-pre/)
- [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://md.research.elamur.ai/free/en/papers/wang-2021-simvlm-simple-visual-language-model/)
- [Multimodal Sarcasm Target Identification in Tweets](https://md.research.elamur.ai/free/en/papers/wang-2022-multimodal-sarcasm-target-identification-tweets/)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://md.research.elamur.ai/free/en/papers/wang-2022-ofa-unifying-architectures-tasks-modalities/)
- [LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding](https://md.research.elamur.ai/free/en/papers/xu-2021-layoutlmv2-multi-modal-pre-training/)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://md.research.elamur.ai/free/en/papers/yang-2023-dawn-lmms-preliminary-explorations-gpt/)
- [视觉语言预训练综述](https://md.research.elamur.ai/free/en/papers/yin-2022-vision-language-pre/)
- [Knowledge-Grounded Dialogue Generation with Pre-trained Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2020-knowledge-grounded-dialogue-generation-pre/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{su2019vl,
  title = {{VL-BERT: Pre-training of Generic Visual-Linguistic Representations}},
  author = {Su, Weijie and Zhu, Xizhou and Cao, Yue and Li, Bin and Lu, Lewei and Wei, Furu and Dai, Jifeng},
  year = {2019},
  doi = {10.48550/arxiv.1908.08530},
  eprint = {1908.08530},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1908.08530}
}
```

- [← previous Visualizing and Measuring the Geometry of BERT](https://md.research.elamur.ai/free/en/papers/coenen-2019-visualizing-measuring-geometry-bert/)
- [next → Well-Read Students Learn Better: On the Importance of Pre-training Compact Models](https://md.research.elamur.ai/free/en/papers/turc-2019-well-read-students-learn-better/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/)

## References

47 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Chris Alberti, Jeffrey Ling, Michael Collins, and David Reitter. Fusion of detected objects in text for visual question answering. arXiv preprint arXiv:1908.05054, 2019. [find](https://scholar.google.com/scholar?q=Chris+Alberti%2C+Jeffrey+Ling%2C+Michael+Collins%2C+and+David+Reitter%2E+Fusion+of+detected+objects+in+text+for+visual+question+answering%2E+arXiv+preprint+arXiv%3A1908%2E05054%2C+2019%2E)
2. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6077–6086, 2018. [find](https://scholar.google.com/scholar?q=Peter+Anderson%2C+Xiaodong+He%2C+Chris+Buehler%2C+Damien+Teney%2C+Mark+Johnson%2C+Stephen+Gould%2C+and+Lei+Zhang%2E+Bottom%2Dup+and+top%2Ddown+attention+for+image+captioning+and+visual+question+answering%2E+In+Proceeding)
3. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425–2433, 2015. [find](https://scholar.google.com/scholar?q=Stanislaw+Antol%2C+Aishwarya+Agrawal%2C+Jiasen+Lu%2C+Margaret+Mitchell%2C+Dhruv+Batra%2C+C+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+Vqa%3A+Visual+question+answering%2E+In+Proceedings+of+the+IEEE+international+conference+)
4. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. [find](https://scholar.google.com/scholar?q=Xinlei+Chen%2C+Hao+Fang%2C+Tsung%2DYi+Lin%2C+Ramakrishna+Vedantam%2C+Saurabh+Gupta%2C+Piotr+Dolla%CC%81r%2C+and+C+Lawrence+Zitnick%2E+Microsoft+coco+captions%3A+Data+collection+and+evaluation+server%2E+arXiv+preprint+arXiv%3A15) [our card](https://md.research.elamur.ai/free/en/papers/chen-2015-microsoft-coco-captions-data-collection/)
5. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. [find](https://scholar.google.com/scholar?q=Jia+Deng%2C+Wei+Dong%2C+Richard+Socher%2C+Li%2DJia+Li%2C+Kai+Li%2C+and+Li+Fei%2DFei%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+2009+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pp%2E+248%E2%80%93)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2C+2018%2E)
7. Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference on machine learning, pp. 647–655, 2014. [find](https://scholar.google.com/scholar?q=Jeff+Donahue%2C+Yangqing+Jia%2C+Oriol+Vinyals%2C+Judy+Hoffman%2C+Ning+Zhang%2C+Eric+Tzeng%2C+and+Trevor+Darrell%2E+Decaf%3A+A+deep+convolutional+activation+feature+for+generic+visual+recognition%2E+In+International+con)
8. Difei Gao, Ruiping Wang, Shiguang Shan, and Xilin Chen. From two graphs to n questions: A vqa dataset for compositional reasoning on vision and commonsense. arXiv preprint arXiv:1908.02962, 2019. — [arXiv:1908.02962](https://arxiv.org/abs/1908.02962) [find](https://scholar.google.com/scholar?q=Difei+Gao%2C+Ruiping+Wang%2C+Shiguang+Shan%2C+and+Xilin+Chen%2E+From+two+graphs+to+n+questions%3A+A+vqa+dataset+for+compositional+reasoning+on+vision+and+commonsense%2E+arXiv+preprint+arXiv%3A1908%2E02962%2C+2019%2E+%E2%80%94+%5Ba)
9. Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 1440–1448, 2015. [find](https://scholar.google.com/scholar?q=Ross+Girshick%2E+Fast+r%2Dcnn%2E+In+Proceedings+of+the+IEEE+international+conference+on+computer+vision%2C+pp%2E+1440%E2%80%931448%2C+2015%2E)
10. Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580–587, 2014. [find](https://scholar.google.com/scholar?q=Ross+Girshick%2C+Jeff+Donahue%2C+Trevor+Darrell%2C+and+Jitendra+Malik%2E+Rich+feature+hierarchies+for+accurate+object+detection+and+semantic+segmentation%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vis) [our card](https://md.research.elamur.ai/free/en/papers/girshick-2013-rich-feature-hierarchies-accurate-object/)
11. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6904–6913, 2017. [find](https://scholar.google.com/scholar?q=Yash+Goyal%2C+Tejas+Khot%2C+Douglas+Summers%2DStay%2C+Dhruv+Batra%2C+and+Devi+Parikh%2E+Making+the+v+in+vqa+matter%3A+Elevating+the+role+of+image+understanding+in+visual+question+answering%2E+In+Proceedings+of+the+IE) [our card](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/)
12. Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous detection and segmentation. In European Conference on Computer Vision, pp. 297–312. Springer, 2014. [find](https://scholar.google.com/scholar?q=Bharath+Hariharan%2C+Pablo+Arbela%CC%81ez%2C+Ross+Girshick%2C+and+Jitendra+Malik%2E+Simultaneous+detection+and+segmentation%2E+In+European+Conference+on+Computer+Vision%2C+pp%2E+297%E2%80%93312%2E+Springer%2C+2014%2E)
13. Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. arXiv preprint arXiv:1811.08883, 2018. — [arXiv:1811.08883](https://arxiv.org/abs/1811.08883) [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Ross+Girshick%2C+and+Piotr+Dolla%CC%81r%2E+Rethinking+imagenet+pre%2Dtraining%2E+arXiv+preprint+arXiv%3A1811%2E08883%2C+2018%2E+%E2%80%94+%5BarXiv%3A1811%2E08883%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1811%2E08883%29)
14. Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei. Relation networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3588–3597, 2018. [find](https://scholar.google.com/scholar?q=Han+Hu%2C+Jiayuan+Gu%2C+Zheng+Zhang%2C+Jifeng+Dai%2C+and+Yichen+Wei%2E+Relation+networks+for+object+detection%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+pp%2E+3588%E2%80%933597%2C+20)
15. Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6700–6709, 2019. [find](https://scholar.google.com/scholar?q=Drew+A+Hudson+and+Christopher+D+Manning%2E+Gqa%3A+A+new+dataset+for+real%2Dworld+visual+reasoning+and+compositional+question+answering%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+R) [our card](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/)
16. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2901–2910, 2017. [find](https://scholar.google.com/scholar?q=Justin+Johnson%2C+Bharath+Hariharan%2C+Laurens+van+der+Maaten%2C+Li+Fei%2DFei%2C+C+Lawrence+Zitnick%2C+and+Ross+Girshick%2E+Clevr%3A+A+diagnostic+dataset+for+compositional+language+and+elementary+visual+reasoning%2E+In) [our card](https://md.research.elamur.ai/free/en/papers/johnson-2016-clevr-diagnostic-dataset-compositional-language/)
17. Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787–798, 2014. [find](https://scholar.google.com/scholar?q=Sahar+Kazemzadeh%2C+Vicente+Ordonez%2C+Mark+Matten%2C+and+Tamara+Berg%2E+Referitgame%3A+Referring+to+objects+in+photographs+of+natural+scenes%2E+In+Proceedings+of+the+2014+conference+on+empirical+methods+in+natur) [our card](https://md.research.elamur.ai/free/en/papers/kazemzadeh-2014-referitgame-referring-objects-photographs-natural/)
18. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+Adam%3A+A+method+for+stochastic+optimization%2E+arXiv+preprint+arXiv%3A1412%2E6980%2C+2014%2E)
19. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in neural information processing systems, pp. 3294–3302, 2015. [find](https://scholar.google.com/scholar?q=Ryan+Kiros%2C+Yukun+Zhu%2C+Ruslan+R+Salakhutdinov%2C+Richard+Zemel%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+Skip%2Dthought+vectors%2E+In+Advances+in+neural+information+processing+systems%2C+pp%2E+3294%E2%80%933)
20. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32–73, 2… [find](https://scholar.google.com/scholar?q=Ranjay+Krishna%2C+Yuke+Zhu%2C+Oliver+Groth%2C+Justin+Johnson%2C+Kenji+Hata%2C+Joshua+Kravitz%2C+Stephanie+Chen%2C+Yannis+Kalantidis%2C+Li%2DJia+Li%2C+David+A+Shamma%2C+et+al%2E+Visual+genome%3A+Connecting+language+and+vision+u) [our card](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/)
21. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105, 2012. [find](https://scholar.google.com/scholar?q=Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Geoffrey+E+Hinton%2E+Imagenet+classification+with+deep+convolutional+neural+networks%2E+In+Advances+in+neural+information+processing+systems%2C+pp%2E+1097%E2%80%931105%2C+2012%2E)
22. Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291, 2019. [find](https://scholar.google.com/scholar?q=Guillaume+Lample+and+Alexis+Conneau%2E+Cross%2Dlingual+language+model+pretraining%2E+arXiv+preprint+arXiv%3A1901%2E07291%2C+2019%2E)
23. Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training, 2019a. [find](https://scholar.google.com/scholar?q=Gen+Li%2C+Nan+Duan%2C+Yuejian+Fang%2C+Daxin+Jiang%2C+and+Ming+Zhou%2E+Unicoder%2Dvl%3A+A+universal+encoder+for+vision+and+language+by+cross%2Dmodal+pre%2Dtraining%2C+2019a%2E)
24. Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019b. [find](https://scholar.google.com/scholar?q=Liunian+Harold+Li%2C+Mark+Yatskar%2C+Da+Yin%2C+Cho%2DJui+Hsieh%2C+and+Kai%2DWei+Chang%2E+Visualbert%3A+A+simple+and+performant+baseline+for+vision+and+language%2E+arXiv+preprint+arXiv%3A1908%2E03557%2C+2019b%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2019-visualbert-simple-performant-baseline-vision/)
25. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740–755. Springer, 2014. [find](https://scholar.google.com/scholar?q=Tsung%2DYi+Lin%2C+Michael+Maire%2C+Serge+Belongie%2C+James+Hays%2C+Pietro+Perona%2C+Deva+Ramanan%2C+Piotr+Dolla%CC%81r%2C+and+C+Lawrence+Zitnick%2E+Microsoft+coco%3A+Common+objects+in+context%2E+In+European+conference+on+comput) [our card](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/)
26. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+arXiv+p)
27. Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431–3440, 2015. [find](https://scholar.google.com/scholar?q=Jonathan+Long%2C+Evan+Shelhamer%2C+and+Trevor+Darrell%2E+Fully+convolutional+networks+for+semantic+segmentation%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pp%2E+3431%E2%80%9334)
28. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. arXiv preprint arXiv:1908.02265, 2019. [find](https://scholar.google.com/scholar?q=Jiasen+Lu%2C+Dhruv+Batra%2C+Devi+Parikh%2C+and+Stefan+Lee%2E+Vilbert%3A+Pretraining+task%2Dagnostic+visiolinguistic+representations+for+vision%2Dand%2Dlanguage+tasks%2E+arXiv+preprint+arXiv%3A1908%2E02265%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/lu-2019-vilbert-pretraining-task-agnostic-visiolinguistic/)
29. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Kai+Chen%2C+Greg+Corrado%2C+and+Jeffrey+Dean%2E+Efficient+estimation+of+word+representations+in+vector+space%2E+arXiv+preprint+arXiv%3A1301%2E3781%2C+2013%2E)
30. Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532–1543, 2014. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+Manning%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+2014+conference+on+empirical+methods+in+natural+language+processing+%28E)
31. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+2018%2E)
32. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%2C+2019%2E)
33. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91–99, 2015. [find](https://scholar.google.com/scholar?q=Shaoqing+Ren%2C+Kaiming+He%2C+Ross+Girshick%2C+and+Jian+Sun%2E+Faster+r%2Dcnn%3A+Towards+real%2Dtime+object+detection+with+region+proposal+networks%2E+In+Advances+in+neural+information+processing+systems%2C+pp%2E+91%E2%80%9399%2C+) [our card](https://md.research.elamur.ai/free/en/papers/ren-2015-faster-r-cnn-towards-real/)
34. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556–2565, 2018. [find](https://scholar.google.com/scholar?q=Piyush+Sharma%2C+Nan+Ding%2C+Sebastian+Goodman%2C+and+Radu+Soricut%2E+Conceptual+captions%3A+A+cleaned%2C+hypernymed%2C+image+alt%2Dtext+dataset+for+automatic+image+captioning%2E+In+Proceedings+of+the+56th+Annual+Meeti) [our card](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/)
35. Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Contrastive bidirectional transformer for temporal representation learning. arXiv preprint arXiv:1906.05743, 2019a. — [arXiv:1906.05743](https://arxiv.org/abs/1906.05743) [find](https://scholar.google.com/scholar?q=Chen+Sun%2C+Fabien+Baradel%2C+Kevin+Murphy%2C+and+Cordelia+Schmid%2E+Contrastive+bidirectional+transformer+for+temporal+representation+learning%2E+arXiv+preprint+arXiv%3A1906%2E05743%2C+2019a%2E+%E2%80%94+%5BarXiv%3A1906%2E05743%5D%28ht)
36. Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. arXiv preprint arXiv:1904.01766, 2019b. [find](https://scholar.google.com/scholar?q=Chen+Sun%2C+Austin+Myers%2C+Carl+Vondrick%2C+Kevin+Murphy%2C+and+Cordelia+Schmid%2E+Videobert%3A+A+joint+model+for+video+and+language+representation+learning%2E+arXiv+preprint+arXiv%3A1904%2E01766%2C+2019b%2E) [our card](https://md.research.elamur.ai/free/en/papers/sun-2019-videobert-joint-model-video-language/)
37. Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 2019. [find](https://scholar.google.com/scholar?q=Hao+Tan+and+Mohit+Bansal%2E+Lxmert%3A+Learning+cross%2Dmodality+encoder+representations+from+transformers%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/tan-2019-lxmert-learning-cross-modality-encoder/)
38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998–6008, 2017. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+Attention+is+all+you+need%2E+In+Advances+in+neural+information+processing+sys)
39. Jesse Vig. A multiscale visualization of attention in the transformer model. arXiv preprint arXiv:1906.05714, 2019. URL https://arxiv.org/abs/1906.05714. [↗](https://arxiv.org/abs/1906.05714) [our card](https://md.research.elamur.ai/free/en/papers/vig-2019-multiscale-visualization-attention-transformer-model/)
40. Alex Wang and Kyunghyun Cho. Bert has a mouth, and it must speak: Bert as a markov random field language model. arXiv preprint arXiv:1902.04094, 2019. [find](https://scholar.google.com/scholar?q=Alex+Wang+and+Kyunghyun+Cho%2E+Bert+has+a+mouth%2C+and+it+must+speak%3A+Bert+as+a+markov+random+field+language+model%2E+arXiv+preprint+arXiv%3A1902%2E04094%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-bert-has-mouth-must-speak/)
41. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016. [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+et+al%2E+Google%E2%80%99s+neural+machine+translation+system%3A+Bridging+th)
42. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237, 2019. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Ruslan+Salakhutdinov%2C+and+Quoc+V+Le%2E+Xlnet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+arXiv+preprint+arXiv%3A1906%2E08237%2C+2) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
43. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78, 2014. [find](https://scholar.google.com/scholar?q=Peter+Young%2C+Alice+Lai%2C+Micah+Hodosh%2C+and+Julia+Hockenmaier%2E+From+image+descriptions+to+visual+denotations%3A+New+similarity+metrics+for+semantic+inference+over+event+descriptions%2E+Transactions+of+the+A)
44. Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1307–1315, 2018. [find](https://scholar.google.com/scholar?q=Licheng+Yu%2C+Zhe+Lin%2C+Xiaohui+Shen%2C+Jimei+Yang%2C+Xin+Lu%2C+Mohit+Bansal%2C+and+Tamara+L+Berg%2E+Mattnet%3A+Modular+attention+network+for+referring+expression+comprehension%2E+In+Proceedings+of+the+IEEE+Conference)
45. Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6720–6731, 2019. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+From+recognition+to+cognition%3A+Visual+commonsense+reasoning%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2C)
46. Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4995–5004, 2016. [find](https://scholar.google.com/scholar?q=Yuke+Zhu%2C+Oliver+Groth%2C+Michael+Bernstein%2C+and+Li+Fei%2DFei%2E+Visual7w%3A+Grounded+question+answering+in+images%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pp%2E+4995%E2%80%935) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2015-visual7w-grounded-question-answering-images/)
47. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pp. 19–27, 2015… [find](https://scholar.google.com/scholar?q=Yukun+Zhu%2C+Ryan+Kiros%2C+Rich+Zemel%2C+Ruslan+Salakhutdinov%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+Aligning+books+and+movies%3A+Towards+story%2Dlike+visual+explanations+by+watching+movies+and+re) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/)