--- title: "UNITER: UNiversal Image-TExt Representation — Chen 2019" description: "Most Vision-and-Language (V+L) models build joint image-text representations designed for one task, so they rarely transfer to other tasks." authors: ["Yen-Chun Chen", "Linjie Li", "Licheng Yu", "Ahmed El Kholy", "Faisal Ahmed", "Zhe Gan", "Yu Cheng", "Jingjing Liu"] year: 2019 venue: "arXiv:1909.11740" source: "https://arxiv.org/abs/1909.11740" doi: "10.48550/arxiv.1909.11740" arxiv: "1909.11740" licence: "arXiv — non-exclusive license to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/chen-2019-uniter-universal-image-text-representation/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2019-uniter-universal-image-text-representation/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / UNITER: UNiversal Image-TExt Representation Learning # UNITER: UNiversal Image-TExt Representation Learning **Authors:** [Yen-Chun Chen](https://md.research.elamur.ai/free/en/authors/yen-chun-chen/), [Linjie Li](https://md.research.elamur.ai/free/en/authors/linjie-li/), [Licheng Yu](https://md.research.elamur.ai/free/en/authors/licheng-yu/), [Ahmed El Kholy](https://md.research.elamur.ai/free/en/authors/ahmed-el-kholy/), Faisal Ahmed, [Zhe Gan](https://md.research.elamur.ai/free/en/authors/zhe-gan/), [Yu Cheng](https://md.research.elamur.ai/free/en/authors/yu-cheng/), [Jingjing Liu](https://md.research.elamur.ai/free/en/authors/jingjing-liu/) · **Year:** 2019 · **Venue:** arXiv:1909.11740 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/1909.11740) · [PDF](https://arxiv.org/pdf/1909.11740) · [DOI 10.48550/arxiv.1909.11740](https://doi.org/10.48550/arxiv.1909.11740) · [arXiv 1909.11740](https://arxiv.org/abs/1909.11740) · [OpenAlex](https://openalex.org/W3090449556) **Citations:** 1948 citations (openalex-snapshot · 2026-10-14) · [396](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Most Vision-and-Language (V+L) models build joint image-text representations designed for one task, so they rarely transfer to other tasks. UNITER is a single pre-trained Transformer that produces a general image-text embedding usable across many V+L tasks. Image regions are represented by Faster R-CNN features plus box coordinates, and text by WordPiece tokens with positions. Both are projected into a shared space and processed jointly by one Transformer. Pre-training uses four public image-text corpora and four objectives. It differs from earlier multimodal pre-training in two ways. First, masking is conditional: only one modality is corrupted at a time, so the other remains a complete reference. Second, an Optimal Transport objective explicitly pushes individual words and regions toward alignment. Ablations show that both changes help. After fine-tuning, UNITER outperformed task-specific and earlier pre-trained models on nine datasets covering six task types. This makes it a strong, reusable baseline for multimodal research. ## Key points - **Pre-training data and objectives:** The corpora used for pre-training are Conceptual Captions, SBU Captions, Visual Genome, and COCO. It uses four objectives: - MLM, which masks 15% of words. - MRM, in three variants: MRC, MRFR and MRC-kl. - ITM, for whole image-sentence matching. - WRA, the OT-based word-region alignment task. Each mini-batch trains a single, randomly sampled objective. - **Out-of-domain data helps:** Adding CC and SBU, which contain images and text not seen in downstream tasks, improved results compared with using only COCO and VG. - **Evaluation:** Tasks were VQA, VCR, NLVR 2, Visual Entailment (SNLI-VE), image-text retrieval on Flickr and COCO (including zero-shot), and referring expressions on RefCOCO, RefCOCO+ and RefCOCOg. The model comes in Base and Large sizes. Example results for UNITER-large: - VQA test-dev: about 73.8, compared with about 72.4 for LXMERT. - NLVR 2 dev: about 79.1. - **NLVR 2 adaptation:** This task takes two images per example. Encoding each image-caption pair separately and adding bidirectional attention ("Pair-biattn") gave the best UNITER-base results: 77.18 dev and 77.85 test-P. The "Triplet" setting reached only about 73. - **VCR second-stage pre-training:** Further pre-training on VCR data raised UNITER-base Q→AR on the val split from 53.52 to 57.76. Code is released at github.com/ChenRocks/UNITER. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 396 papers of our library cite it (of the 140673 whose bibliographies we read); 27 of them have a page here, 369 not yet. - [Flamingo: a Visual Language Model for Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/) - [Acquiring Linguistic Knowledge from Multimodal Input](https://md.research.elamur.ai/free/en/papers/amariucai-2023-acquiring-linguistic-knowledge-multimodal-input/) - [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/) - [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://md.research.elamur.ai/free/en/papers/chen-2022-pali-jointly-scaled-multilingual-language/) - [SeamlessM4T: Massively Multilingual & Multimodal Machine Translation](https://md.research.elamur.ai/free/en/papers/communication-2023-seamlessm4t-massively-multilingual-multimodal-machine/) - [GRAVL-BERT: Graphical Visual-Linguistic Representations for Multimodal Coreference Resolution](https://md.research.elamur.ai/free/en/papers/guo-2022-gravl-bert-graphical-visual-linguistic/) - [Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translation](https://md.research.elamur.ai/free/en/papers/guo-2023-bridging-gap-between-synthetic-authentic/) - [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://md.research.elamur.ai/free/en/papers/hessel-2021-clipscore-reference-free-evaluation-metric/) - [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://md.research.elamur.ai/free/en/papers/huang-2022-layoutlmv3-pre-training-document-ai/) - [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://md.research.elamur.ai/free/en/papers/jia-2021-scaling-up-visual-vision-language/) - [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://md.research.elamur.ai/free/en/papers/kim-2021-vilt-vision-language-transformer-convolution/) - [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://md.research.elamur.ai/free/en/papers/li-2020-oscar-object-semantics-aligned-pre/) - [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/) - [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/) - [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://md.research.elamur.ai/free/en/papers/peng-2023-kosmos-2-grounding-multimodal-large/) - [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/) - [Learning Transferable Visual Models From Natural Language Supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/) - [AudioPaLM: A Large Language Model That Can Speak and Listen](https://md.research.elamur.ai/free/en/papers/rubenstein-2023-audiopalm-large-language-model-that/) - [Training data-efficient image transformers & distillation through attention](https://md.research.elamur.ai/free/en/papers/touvron-2020-training-data-efficient-image-transformers/) - [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://md.research.elamur.ai/free/en/papers/wang-2021-simvlm-simple-visual-language-model/) - [Multimodal Sarcasm Target Identification in Tweets](https://md.research.elamur.ai/free/en/papers/wang-2022-multimodal-sarcasm-target-identification-tweets/) - [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://md.research.elamur.ai/free/en/papers/wang-2022-ofa-unifying-architectures-tasks-modalities/) - [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://md.research.elamur.ai/free/en/papers/wu-2023-visual-chatgpt-talking-drawing-editing/) - [LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding](https://md.research.elamur.ai/free/en/papers/xu-2021-layoutlmv2-multi-modal-pre-training/) - [视觉语言预训练综述](https://md.research.elamur.ai/free/en/papers/yin-2022-vision-language-pre/) - [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://md.research.elamur.ai/free/en/papers/yu-2022-coca-contrastive-captioners-image-text/) - [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{chen2019uniter, title = {{UNITER: UNiversal Image-TExt Representation Learning}}, author = {Chen, Yen-Chun and Li, Linjie and Yu, Licheng and Kholy, Ahmed El and Ahmed, Faisal and Gan, Zhe and Cheng, Yu and Liu, Jingjing}, year = {2019}, doi = {10.48550/arxiv.1909.11740}, eprint = {1909.11740}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.1909.11740} } ``` - [← previous Unified Language Model Pre-training for Natural Language Understanding and Generation](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/) - [next → Universal dependencies for Scottish Gaelic: syntax](https://md.research.elamur.ai/free/en/papers/batchelor-2019-universal-dependencies-scottish-gaelic-syntax/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) ## References 60 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Alberti, C., Ling, J., Collins, M., Reitter, D.: Fusion of detected objects in text for visual question answering. In: EMNLP (2019) 2, 4, 11 [find](https://scholar.google.com/scholar?q=Alberti%2C+C%2E%2C+Ling%2C+J%2E%2C+Collins%2C+M%2E%2C+Reitter%2C+D%2E%3A+Fusion+of+detected+objects+in+text+for+visual+question+answering%2E+In%3A+EMNLP+%282019%29+2%2C+4%2C+11) 2. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering. In: CVPR (2018) 4, 18, 21 [find](https://scholar.google.com/scholar?q=Anderson%2C+P%2E%2C+He%2C+X%2E%2C+Buehler%2C+C%2E%2C+Teney%2C+D%2E%2C+Johnson%2C+M%2E%2C+Gould%2C+S%2E%2C+Zhang%2C+L%2E%3A+Bottom%2Dup+and+top%2Ddown+attention+for+image+captioning+and+visual+question+answering%2E+In%3A+CVPR+%282018%29+4%2C+18%2C+21) 3. Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: VQA: Visual Question Answering. In: ICCV (2015) 1 [find](https://scholar.google.com/scholar?q=Antol%2C+S%2E%2C+Agrawal%2C+A%2E%2C+Lu%2C+J%2E%2C+Mitchell%2C+M%2E%2C+Batra%2C+D%2E%2C+Zitnick%2C+C%2EL%2E%2C+Parikh%2C+D%2E%3A+VQA%3A+Visual+Question+Answering%2E+In%3A+ICCV+%282015%29+1) 4. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: ICML (2017) 25 [find](https://scholar.google.com/scholar?q=Arjovsky%2C+M%2E%2C+Chintala%2C+S%2E%2C+Bottou%2C+L%2E%3A+Wasserstein+generative+adversarial+networks%2E+In%3A+ICML+%282017%29+25) 5. Boyd, S., Vandenberghe, L.: Convex optimization. Cambridge university press (2004) 25 [find](https://scholar.google.com/scholar?q=Boyd%2C+S%2E%2C+Vandenberghe%2C+L%2E%3A+Convex+optimization%2E+Cambridge+university+press+%282004%29+25) 6. Cao, J., Gan, Z., Cheng, Y., Yu, L., Chen, Y.C., Liu, J.: Behind the scene: Revealing the secrets of pre-trained vision-and-language models. arXiv preprint arXiv:2005.07310 (2020) 4 — [arXiv:2005.07310](https://arxiv.org/abs/2005.07310) [find](https://scholar.google.com/scholar?q=Cao%2C+J%2E%2C+Gan%2C+Z%2E%2C+Cheng%2C+Y%2E%2C+Yu%2C+L%2E%2C+Chen%2C+Y%2EC%2E%2C+Liu%2C+J%2E%3A+Behind+the+scene%3A+Revealing+the+secrets+of+pre%2Dtrained+vision%2Dand%2Dlanguage+models%2E+arXiv+preprint+arXiv%3A2005%2E07310+%282020%29+4+%E2%80%94+%5BarXiv%3A2005%2E0731) 7. Chen, L., Gan, Z., Cheng, Y., Li, L., Carin, L., Liu, J.: Graph optimal transport for cross-domain alignment. In: ICML (2020) 3 [find](https://scholar.google.com/scholar?q=Chen%2C+L%2E%2C+Gan%2C+Z%2E%2C+Cheng%2C+Y%2E%2C+Li%2C+L%2E%2C+Carin%2C+L%2E%2C+Liu%2C+J%2E%3A+Graph+optimal+transport+for+cross%2Ddomain+alignment%2E+In%3A+ICML+%282020%29+3) 8. Cuturi, M.: Sinkhorn distances: Lightspeed computation of optimal transport. In: NIPS (2013) 26 [find](https://scholar.google.com/scholar?q=Cuturi%2C+M%2E%3A+Sinkhorn+distances%3A+Lightspeed+computation+of+optimal+transport%2E+In%3A+NIPS+%282013%29+26) 9. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: NAACL (2019) 2, 3, 4, 10 [find](https://scholar.google.com/scholar?q=Devlin%2C+J%2E%2C+Chang%2C+M%2EW%2E%2C+Lee%2C+K%2E%2C+Toutanova%2C+K%2E%3A+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In%3A+NAACL+%282019%29+2%2C+3%2C+4%2C+10) 10. Doersch, C., Gupta, A., Efros, A.A.: Unsupervised visual representation learning by context prediction. In: ICCV (2015) 3 [find](https://scholar.google.com/scholar?q=Doersch%2C+C%2E%2C+Gupta%2C+A%2E%2C+Efros%2C+A%2EA%2E%3A+Unsupervised+visual+representation+learning+by+context+prediction%2E+In%3A+ICCV+%282015%29+3) 11. Fukui, A., Park, D.H., Yang, D., Rohrbach, A., Darrell, T., Rohrbach, M.: Multimodal compact bilinear pooling for visual question answering and visual grounding. In: EMNLP (2017) 1 [find](https://scholar.google.com/scholar?q=Fukui%2C+A%2E%2C+Park%2C+D%2EH%2E%2C+Yang%2C+D%2E%2C+Rohrbach%2C+A%2E%2C+Darrell%2C+T%2E%2C+Rohrbach%2C+M%2E%3A+Multimodal+compact+bilinear+pooling+for+visual+question+answering+and+visual+grounding%2E+In%3A+EMNLP+%282017%29+1) 12. Gan, Z., Chen, Y.C., Li, L., Zhu, C., Cheng, Y., Liu, J.: Large-scale adversarial training for vision-and-language representation learning. arXiv preprint arXiv:2006.06195 (2020) 4 — [arXiv:2006.06195](https://arxiv.org/abs/2006.06195) [find](https://scholar.google.com/scholar?q=Gan%2C+Z%2E%2C+Chen%2C+Y%2EC%2E%2C+Li%2C+L%2E%2C+Zhu%2C+C%2E%2C+Cheng%2C+Y%2E%2C+Liu%2C+J%2E%3A+Large%2Dscale+adversarial+training+for+vision%2Dand%2Dlanguage+representation+learning%2E+arXiv+preprint+arXiv%3A2006%2E06195+%282020%29+4+%E2%80%94+%5BarXiv%3A2006%2E06195) 13. Gao, P., Jiang, Z., You, H., Lu, P., Hoi, S.C., Wang, X., Li, H.: Dynamic fusion with intra-and inter-modality attention flow for visual question answering. In: CVPR (2019) 1 [find](https://scholar.google.com/scholar?q=Gao%2C+P%2E%2C+Jiang%2C+Z%2E%2C+You%2C+H%2E%2C+Lu%2C+P%2E%2C+Hoi%2C+S%2EC%2E%2C+Wang%2C+X%2E%2C+Li%2C+H%2E%3A+Dynamic+fusion+with+intra%2Dand+inter%2Dmodality+attention+flow+for+visual+question+answering%2E+In%3A+CVPR+%282019%29+1) 14. Genevay, A., Peyré, G., Cuturi, M.: Learning generative models with sinkhorn divergences. In: AISTATS (2018) 25 [find](https://scholar.google.com/scholar?q=Genevay%2C+A%2E%2C+Peyre%CC%81%2C+G%2E%2C+Cuturi%2C+M%2E%3A+Learning+generative+models+with+sinkhorn+divergences%2E+In%3A+AISTATS+%282018%29+25) 15. Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. In: ICLR (2018) 3 [find](https://scholar.google.com/scholar?q=Gidaris%2C+S%2E%2C+Singh%2C+P%2E%2C+Komodakis%2C+N%2E%3A+Unsupervised+representation+learning+by+predicting+image+rotations%2E+In%3A+ICLR+%282018%29+3) 16. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015) 7 [find](https://scholar.google.com/scholar?q=Hinton%2C+G%2E%2C+Vinyals%2C+O%2E%2C+Dean%2C+J%2E%3A+Distilling+the+knowledge+in+a+neural+network%2E+arXiv+preprint+arXiv%3A1503%2E02531+%282015%29+7) [our card](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/) 17. Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: CVPR (2015) 20, 21 [find](https://scholar.google.com/scholar?q=Karpathy%2C+A%2E%2C+Fei%2DFei%2C+L%2E%3A+Deep+visual%2Dsemantic+alignments+for+generating+image+descriptions%2E+In%3A+CVPR+%282015%29+20%2C+21) [our card](https://md.research.elamur.ai/free/en/papers/karpathy-2014-deep-visual-semantic-alignments-generating/) 18. Kazemzadeh, S., Ordonez, V., Matten, M., Berg, T.: Referitgame: Referring to objects in photographs of natural scenes. In: EMNLP (2014) 2 [find](https://scholar.google.com/scholar?q=Kazemzadeh%2C+S%2E%2C+Ordonez%2C+V%2E%2C+Matten%2C+M%2E%2C+Berg%2C+T%2E%3A+Referitgame%3A+Referring+to+objects+in+photographs+of+natural+scenes%2E+In%3A+EMNLP+%282014%29+2) 19. Kim, J.H., Jun, J., Zhang, B.T.: Bilinear attention networks. In: NeurIPS (2018) 1 [find](https://scholar.google.com/scholar?q=Kim%2C+J%2EH%2E%2C+Jun%2C+J%2E%2C+Zhang%2C+B%2ET%2E%3A+Bilinear+attention+networks%2E+In%3A+NeurIPS+%282018%29+1) 20. Kovaleva, O., Romanov, A., Rogers, A., Rumshisky, A.: Revealing the dark secrets of bert. In: EMNLP (2019) 13 [find](https://scholar.google.com/scholar?q=Kovaleva%2C+O%2E%2C+Romanov%2C+A%2E%2C+Rogers%2C+A%2E%2C+Rumshisky%2C+A%2E%3A+Revealing+the+dark+secrets+of+bert%2E+In%3A+EMNLP+%282019%29+13) [our card](https://md.research.elamur.ai/free/en/papers/kovaleva-2019-revealing-dark-secrets-bert/) 21. Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., et al.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV (2017) 3, 7 Y.-C. Chen et al. [find](https://scholar.google.com/scholar?q=Krishna%2C+R%2E%2C+Zhu%2C+Y%2E%2C+Groth%2C+O%2E%2C+Johnson%2C+J%2E%2C+Hata%2C+K%2E%2C+Kravitz%2C+J%2E%2C+Chen%2C+S%2E%2C+Kalantidis%2C+Y%2E%2C+Li%2C+L%2EJ%2E%2C+Shamma%2C+D%2EA%2E%2C+et+al%2E%3A+Visual+genome%3A+Connecting+language+and+vision+using+crowdsourced+dense+im) [our card](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/) 22. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: Albert: A lite bert for self-supervised learning of language representations. In: ICLR (2020) 3 [find](https://scholar.google.com/scholar?q=Lan%2C+Z%2E%2C+Chen%2C+M%2E%2C+Goodman%2C+S%2E%2C+Gimpel%2C+K%2E%2C+Sharma%2C+P%2E%2C+Soricut%2C+R%2E%3A+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+In%3A+ICLR+%282020%29+3) 23. Lee, K.H., Chen, X., Hua, G., Hu, H., He, X.: Stacked cross attention for image-text matching. In: ECCV (2018) 1, 3, 11, 21 [find](https://scholar.google.com/scholar?q=Lee%2C+K%2EH%2E%2C+Chen%2C+X%2E%2C+Hua%2C+G%2E%2C+Hu%2C+H%2E%2C+He%2C+X%2E%3A+Stacked+cross+attention+for+image%2Dtext+matching%2E+In%3A+ECCV+%282018%29+1%2C+3%2C+11%2C+21) 24. Li, G., Duan, N., Fang, Y., Jiang, D., Zhou, M.: Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In: AAAI (2020) 2, 4, 11 [find](https://scholar.google.com/scholar?q=Li%2C+G%2E%2C+Duan%2C+N%2E%2C+Fang%2C+Y%2E%2C+Jiang%2C+D%2E%2C+Zhou%2C+M%2E%3A+Unicoder%2Dvl%3A+A+universal+encoder+for+vision+and+language+by+cross%2Dmodal+pre%2Dtraining%2E+In%3A+AAAI+%282020%29+2%2C+4%2C+11) 25. Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., Chang, K.W.: Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 (2019) 2, 4, 11, 24 [find](https://scholar.google.com/scholar?q=Li%2C+L%2EH%2E%2C+Yatskar%2C+M%2E%2C+Yin%2C+D%2E%2C+Hsieh%2C+C%2EJ%2E%2C+Chang%2C+K%2EW%2E%3A+Visualbert%3A+A+simple+and+performant+baseline+for+vision+and+language%2E+arXiv+preprint+arXiv%3A1908%2E03557+%282019%29+2%2C+4%2C+11%2C+24) [our card](https://md.research.elamur.ai/free/en/papers/li-2019-visualbert-simple-performant-baseline-vision/) 26. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: ECCV (2014) 3, 7 [find](https://scholar.google.com/scholar?q=Lin%2C+T%2EY%2E%2C+Maire%2C+M%2E%2C+Belongie%2C+S%2E%2C+Hays%2C+J%2E%2C+Perona%2C+P%2E%2C+Ramanan%2C+D%2E%2C+Dolla%CC%81r%2C+P%2E%2C+Zitnick%2C+C%2EL%2E%3A+Microsoft+coco%3A+Common+objects+in+context%2E+In%3A+ECCV+%282014%29+3%2C+7) [our card](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/) 27. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019) 3 [find](https://scholar.google.com/scholar?q=Liu%2C+Y%2E%2C+Ott%2C+M%2E%2C+Goyal%2C+N%2E%2C+Du%2C+J%2E%2C+Joshi%2C+M%2E%2C+Chen%2C+D%2E%2C+Levy%2C+O%2E%2C+Lewis%2C+M%2E%2C+Zettlemoyer%2C+L%2E%2C+Stoyanov%2C+V%2E%3A+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+arXiv+preprint+arXiv%3A1907%2E11692+%28) 28. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: ICLR (2019) 19 [find](https://scholar.google.com/scholar?q=Loshchilov%2C+I%2E%2C+Hutter%2C+F%2E%3A+Decoupled+weight+decay+regularization%2E+In%3A+ICLR+%282019%29+19) 29. Lu, J., Batra, D., Parikh, D., Lee, S.: Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: NeurIPS (2019) 2, 3, 11, 22, 24, 25 [find](https://scholar.google.com/scholar?q=Lu%2C+J%2E%2C+Batra%2C+D%2E%2C+Parikh%2C+D%2E%2C+Lee%2C+S%2E%3A+Vilbert%3A+Pretraining+task%2Dagnostic+visiolinguistic+representations+for+vision%2Dand%2Dlanguage+tasks%2E+In%3A+NeurIPS+%282019%29+2%2C+3%2C+11%2C+22%2C+24%2C+25) [our card](https://md.research.elamur.ai/free/en/papers/lu-2019-vilbert-pretraining-task-agnostic-visiolinguistic/) 30. Lu, J., Goswami, V., Rohrbach, M., Parikh, D., Lee, S.: 12-in-1: Multi-task vision and language representation learning. In: CVPR (2020) 4 [find](https://scholar.google.com/scholar?q=Lu%2C+J%2E%2C+Goswami%2C+V%2E%2C+Rohrbach%2C+M%2E%2C+Parikh%2C+D%2E%2C+Lee%2C+S%2E%3A+12%2Din%2D1%3A+Multi%2Dtask+vision+and+language+representation+learning%2E+In%3A+CVPR+%282020%29+4) 31. Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving jigsaw puzzles. In: ECCV (2016) 3 [find](https://scholar.google.com/scholar?q=Noroozi%2C+M%2E%2C+Favaro%2C+P%2E%3A+Unsupervised+learning+of+visual+representations+by+solving+jigsaw+puzzles%2E+In%3A+ECCV+%282016%29+3) 32. Ordonez, V., Kulkarni, G., Berg, T.L.: Im2text: Describing images using 1 million captioned photographs. In: NeurIPS (2011) 3, 7, 8 [find](https://scholar.google.com/scholar?q=Ordonez%2C+V%2E%2C+Kulkarni%2C+G%2E%2C+Berg%2C+T%2EL%2E%3A+Im2text%3A+Describing+images+using+1+million+captioned+photographs%2E+In%3A+NeurIPS+%282011%29+3%2C+7%2C+8) 33. Ott, M., Edunov, S., Grangier, D., Auli, M.: Scaling neural machine translation. WMT (2018) 19 [find](https://scholar.google.com/scholar?q=Ott%2C+M%2E%2C+Edunov%2C+S%2E%2C+Grangier%2C+D%2E%2C+Auli%2C+M%2E%3A+Scaling+neural+machine+translation%2E+WMT+%282018%29+19) [our card](https://md.research.elamur.ai/free/en/papers/ott-2018-scaling-neural-machine-translation/) 34. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch (2017) 19 [find](https://scholar.google.com/scholar?q=Paszke%2C+A%2E%2C+Gross%2C+S%2E%2C+Chintala%2C+S%2E%2C+Chanan%2C+G%2E%2C+Yang%2C+E%2E%2C+DeVito%2C+Z%2E%2C+Lin%2C+Z%2E%2C+Desmaison%2C+A%2E%2C+Antiga%2C+L%2E%2C+Lerer%2C+A%2E%3A+Automatic+differentiation+in+pytorch+%282017%29+19) 35. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: CVPR (2016) 3 [find](https://scholar.google.com/scholar?q=Pathak%2C+D%2E%2C+Krahenbuhl%2C+P%2E%2C+Donahue%2C+J%2E%2C+Darrell%2C+T%2E%2C+Efros%2C+A%2EA%2E%3A+Context+encoders%3A+Feature+learning+by+inpainting%2E+In%3A+CVPR+%282016%29+3) 36. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep contextualized word representations. In: NAACL (2018) 3 [find](https://scholar.google.com/scholar?q=Peters%2C+M%2EE%2E%2C+Neumann%2C+M%2E%2C+Iyyer%2C+M%2E%2C+Gardner%2C+M%2E%2C+Clark%2C+C%2E%2C+Lee%2C+K%2E%2C+Zettlemoyer%2C+L%2E%3A+Deep+contextualized+word+representations%2E+In%3A+NAACL+%282018%29+3) 37. Peyré, G., Cuturi, M., et al.: Computational optimal transport. Foundations and Trends R in Machine Learning 11(5-6), 355–607 (2019) 3, 6, 25 [find](https://scholar.google.com/scholar?q=Peyre%CC%81%2C+G%2E%2C+Cuturi%2C+M%2E%2C+et+al%2E%3A+Computational+optimal+transport%2E+Foundations+and+Trends+R+in+Machine+Learning+11%285%2D6%29%2C+355%E2%80%93607+%282019%29+3%2C+6%2C+25) 38. Plummer, B.A., Wang, L., Cervantes, C.M., Caicedo, J.C., Hockenmaier, J., Lazebnik, S.: Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In: ICCV (2015) 8 [find](https://scholar.google.com/scholar?q=Plummer%2C+B%2EA%2E%2C+Wang%2C+L%2E%2C+Cervantes%2C+C%2EM%2E%2C+Caicedo%2C+J%2EC%2E%2C+Hockenmaier%2C+J%2E%2C+Lazebnik%2C+S%2E%3A+Flickr30k+entities%3A+Collecting+region%2Dto%2Dphrase+correspondences+for+richer+image%2Dto%2Dsentence+models%2E+In%3A+ICCV+%282) 39. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners (2019) 3 [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+Child%2C+R%2E%2C+Luan%2C+D%2E%2C+Amodei%2C+D%2E%2C+Sutskever%2C+I%2E%3A+Language+models+are+unsupervised+multitask+learners+%282019%29+3) 40. Salimans, T., Zhang, H., Radford, A., Metaxas, D.: Improving GANs using optimal transport. In: ICLR (2018) 25 [find](https://scholar.google.com/scholar?q=Salimans%2C+T%2E%2C+Zhang%2C+H%2E%2C+Radford%2C+A%2E%2C+Metaxas%2C+D%2E%3A+Improving+GANs+using+optimal+transport%2E+In%3A+ICLR+%282018%29+25) 41. Sharma, P., Ding, N., Goodman, S., Soricut, R.: Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: ACL (2018) 3, 7, 8, 25 [find](https://scholar.google.com/scholar?q=Sharma%2C+P%2E%2C+Ding%2C+N%2E%2C+Goodman%2C+S%2E%2C+Soricut%2C+R%2E%3A+Conceptual+captions%3A+A+cleaned%2C+hypernymed%2C+image+alt%2Dtext+dataset+for+automatic+image+captioning%2E+In%3A+ACL+%282018%29+3%2C+7%2C+8%2C+25) [our card](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/) 42. Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., Dai, J.: Vl-bert: Pre-training of generic visual-linguistic representations. In: ICLR (2020) 2, 4, 11, 24, 25 [find](https://scholar.google.com/scholar?q=Su%2C+W%2E%2C+Zhu%2C+X%2E%2C+Cao%2C+Y%2E%2C+Li%2C+B%2E%2C+Lu%2C+L%2E%2C+Wei%2C+F%2E%2C+Dai%2C+J%2E%3A+Vl%2Dbert%3A+Pre%2Dtraining+of+generic+visual%2Dlinguistic+representations%2E+In%3A+ICLR+%282020%29+2%2C+4%2C+11%2C+24%2C+25) [our card](https://md.research.elamur.ai/free/en/papers/su-2019-vl-bert-pre-training-generic/) 43. Suhr, A., Artzi, Y.: Nlvr2 visual bias analysis. arXiv preprint arXiv:1909.10411 (2019) 24 UNITER: UNiversal Image-TExt Representation Learning — [arXiv:1909.10411](https://arxiv.org/abs/1909.10411) [find](https://scholar.google.com/scholar?q=Suhr%2C+A%2E%2C+Artzi%2C+Y%2E%3A+Nlvr2+visual+bias+analysis%2E+arXiv+preprint+arXiv%3A1909%2E10411+%282019%29+24+UNITER%3A+UNiversal+Image%2DTExt+Representation+Learning+%E2%80%94+%5BarXiv%3A1909%2E10411%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1909%2E10411%29) 44. Suhr, A., Zhou, S., Zhang, I., Bai, H., Artzi, Y.: A corpus for reasoning about natural language grounded in photographs. In: ACL (2019) 3, 11 [find](https://scholar.google.com/scholar?q=Suhr%2C+A%2E%2C+Zhou%2C+S%2E%2C+Zhang%2C+I%2E%2C+Bai%2C+H%2E%2C+Artzi%2C+Y%2E%3A+A+corpus+for+reasoning+about+natural+language+grounded+in+photographs%2E+In%3A+ACL+%282019%29+3%2C+11) [our card](https://md.research.elamur.ai/free/en/papers/suhr-2019-corpus-reasoning-natural-language-grounded/) 45. Sun, C., Baradel, F., Murphy, K., Schmid, C.: Contrastive bidirectional transformer for temporal representation learning. arXiv preprint arXiv:1906.05743 (2019) 3 — [arXiv:1906.05743](https://arxiv.org/abs/1906.05743) [find](https://scholar.google.com/scholar?q=Sun%2C+C%2E%2C+Baradel%2C+F%2E%2C+Murphy%2C+K%2E%2C+Schmid%2C+C%2E%3A+Contrastive+bidirectional+transformer+for+temporal+representation+learning%2E+arXiv+preprint+arXiv%3A1906%2E05743+%282019%29+3+%E2%80%94+%5BarXiv%3A1906%2E05743%5D%28https%3A%2F%2Farxiv%2Eor) 46. Sun, C., Myers, A., Vondrick, C., Murphy, K., Schmid, C.: Videobert: A joint model for video and language representation learning. In: ICCV (2019) 3 [find](https://scholar.google.com/scholar?q=Sun%2C+C%2E%2C+Myers%2C+A%2E%2C+Vondrick%2C+C%2E%2C+Murphy%2C+K%2E%2C+Schmid%2C+C%2E%3A+Videobert%3A+A+joint+model+for+video+and+language+representation+learning%2E+In%3A+ICCV+%282019%29+3) [our card](https://md.research.elamur.ai/free/en/papers/sun-2019-videobert-joint-model-video-language/) 47. Tan, H., Bansal, M.: Lxmert: Learning cross-modality encoder representations from transformers. In: EMNLP (2019) 2, 3, 8, 11, 22, 24 [find](https://scholar.google.com/scholar?q=Tan%2C+H%2E%2C+Bansal%2C+M%2E%3A+Lxmert%3A+Learning+cross%2Dmodality+encoder+representations+from+transformers%2E+In%3A+EMNLP+%282019%29+2%2C+3%2C+8%2C+11%2C+22%2C+24) [our card](https://md.research.elamur.ai/free/en/papers/tan-2019-lxmert-learning-cross-modality-encoder/) 48. Trinh, T.H., Luong, M.T., Le, Q.V.: Selfie: Self-supervised pretraining for image embedding. arXiv preprint arXiv:1906.02940 (2019) 3 — [arXiv:1906.02940](https://arxiv.org/abs/1906.02940) [find](https://scholar.google.com/scholar?q=Trinh%2C+T%2EH%2E%2C+Luong%2C+M%2ET%2E%2C+Le%2C+Q%2EV%2E%3A+Selfie%3A+Self%2Dsupervised+pretraining+for+image+embedding%2E+arXiv+preprint+arXiv%3A1906%2E02940+%282019%29+3+%E2%80%94+%5BarXiv%3A1906%2E02940%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1906%2E02940%29) 49. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: NeurIPS (2017) 2, 3, 20 [find](https://scholar.google.com/scholar?q=Vaswani%2C+A%2E%2C+Shazeer%2C+N%2E%2C+Parmar%2C+N%2E%2C+Uszkoreit%2C+J%2E%2C+Jones%2C+L%2E%2C+Gomez%2C+A%2EN%2E%2C+Kaiser%2C+L%2E%2C+Polosukhin%2C+I%2E%3A+Attention+is+all+you+need%2E+In%3A+NeurIPS+%282017%29+2%2C+3%2C+20) 50. Wang, L., Li, Y., Lazebnik, S.: Learning deep structure-preserving image-text embeddings. In: CVPR (2016) 2 [find](https://scholar.google.com/scholar?q=Wang%2C+L%2E%2C+Li%2C+Y%2E%2C+Lazebnik%2C+S%2E%3A+Learning+deep+structure%2Dpreserving+image%2Dtext+embeddings%2E+In%3A+CVPR+%282016%29+2) 51. Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al.: Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016) 4 [find](https://scholar.google.com/scholar?q=Wu%2C+Y%2E%2C+Schuster%2C+M%2E%2C+Chen%2C+Z%2E%2C+Le%2C+Q%2EV%2E%2C+Norouzi%2C+M%2E%2C+Macherey%2C+W%2E%2C+Krikun%2C+M%2E%2C+Cao%2C+Y%2E%2C+Gao%2C+Q%2E%2C+Macherey%2C+K%2E%2C+et+al%2E%3A+Google%E2%80%99s+neural+machine+translation+system%3A+Bridging+the+gap+between+human+and+) 52. Xie, N., Lai, F., Doran, D., Kadav, A.: Visual entailment: A novel task for fine-grained image understanding. arXiv preprint arXiv:1901.06706 (2019) 3, 11 — [arXiv:1901.06706](https://arxiv.org/abs/1901.06706) [find](https://scholar.google.com/scholar?q=Xie%2C+N%2E%2C+Lai%2C+F%2E%2C+Doran%2C+D%2E%2C+Kadav%2C+A%2E%3A+Visual+entailment%3A+A+novel+task+for+fine%2Dgrained+image+understanding%2E+arXiv+preprint+arXiv%3A1901%2E06706+%282019%29+3%2C+11+%E2%80%94+%5BarXiv%3A1901%2E06706%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F19) 53. Xie, Y., Wang, X., Wang, R., Zha, H.: A fast proximal point method for Wasserstein distance. In: arXiv:1802.04307 (2018) 6, 25 — [arXiv:1802.04307](https://arxiv.org/abs/1802.04307) [find](https://scholar.google.com/scholar?q=Xie%2C+Y%2E%2C+Wang%2C+X%2E%2C+Wang%2C+R%2E%2C+Zha%2C+H%2E%3A+A+fast+proximal+point+method+for+Wasserstein+distance%2E+In%3A+arXiv%3A1802%2E04307+%282018%29+6%2C+25+%E2%80%94+%5BarXiv%3A1802%2E04307%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1802%2E04307%29) 54. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., Le, Q.V.: Xlnet: Generalized autoregressive pretraining for language understanding. In: NeurIPS (2019) 3 [find](https://scholar.google.com/scholar?q=Yang%2C+Z%2E%2C+Dai%2C+Z%2E%2C+Yang%2C+Y%2E%2C+Carbonell%2C+J%2E%2C+Salakhutdinov%2C+R%2E%2C+Le%2C+Q%2EV%2E%3A+Xlnet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+In%3A+NeurIPS+%282019%29+3) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/) 55. Yu, L., Lin, Z., Shen, X., Yang, J., Lu, X., Bansal, M., Berg, T.L.: Mattnet: Modular attention network for referring expression comprehension. In: CVPR (2018) 1, 9 [find](https://scholar.google.com/scholar?q=Yu%2C+L%2E%2C+Lin%2C+Z%2E%2C+Shen%2C+X%2E%2C+Yang%2C+J%2E%2C+Lu%2C+X%2E%2C+Bansal%2C+M%2E%2C+Berg%2C+T%2EL%2E%3A+Mattnet%3A+Modular+attention+network+for+referring+expression+comprehension%2E+In%3A+CVPR+%282018%29+1%2C+9) 56. Yu, L., Poirson, P., Yang, S., Berg, A.C., Berg, T.L.: Modeling context in referring expressions. In: ECCV (2016) 3 [find](https://scholar.google.com/scholar?q=Yu%2C+L%2E%2C+Poirson%2C+P%2E%2C+Yang%2C+S%2E%2C+Berg%2C+A%2EC%2E%2C+Berg%2C+T%2EL%2E%3A+Modeling+context+in+referring+expressions%2E+In%3A+ECCV+%282016%29+3) [our card](https://md.research.elamur.ai/free/en/papers/yu-2016-modeling-context-referring-expressions/) 57. Yu, Z., Yu, J., Cui, Y., Tao, D., Tian, Q.: Deep modular co-attention networks for visual question answering. In: CVPR (2019) 11, 19 [find](https://scholar.google.com/scholar?q=Yu%2C+Z%2E%2C+Yu%2C+J%2E%2C+Cui%2C+Y%2E%2C+Tao%2C+D%2E%2C+Tian%2C+Q%2E%3A+Deep+modular+co%2Dattention+networks+for+visual+question+answering%2E+In%3A+CVPR+%282019%29+11%2C+19) 58. Zellers, R., Bisk, Y., Farhadi, A., Choi, Y.: From recognition to cognition: Visual commonsense reasoning. In: CVPR (2019) 3 [find](https://scholar.google.com/scholar?q=Zellers%2C+R%2E%2C+Bisk%2C+Y%2E%2C+Farhadi%2C+A%2E%2C+Choi%2C+Y%2E%3A+From+recognition+to+cognition%3A+Visual+commonsense+reasoning%2E+In%3A+CVPR+%282019%29+3) 59. Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: ECCV (2016) 3 [find](https://scholar.google.com/scholar?q=Zhang%2C+R%2E%2C+Isola%2C+P%2E%2C+Efros%2C+A%2EA%2E%3A+Colorful+image+colorization%2E+In%3A+ECCV+%282016%29+3) 60. Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J.J., Gao, J.: Unified vision-language pre-training for image captioning and vqa. In: AAAI (2020) 2, 4 Y.-C. Chen et al. [find](https://scholar.google.com/scholar?q=Zhou%2C+L%2E%2C+Palangi%2C+H%2E%2C+Zhang%2C+L%2E%2C+Hu%2C+H%2E%2C+Corso%2C+J%2EJ%2E%2C+Gao%2C+J%2E%3A+Unified+vision%2Dlanguage+pre%2Dtraining+for+image+captioning+and+vqa%2E+In%3A+AAAI+%282020%29+2%2C+4+Y%2E%2DC%2E+Chen+et+al%2E)