---
title: "Oscar: Object-Semantics Aligned Pre-training for — Li 2020"
description: "Vision-language pre-training (VLP) models usually feed image region features and text tokens together into a Transformer."
authors: ["Xiujun Li", "Xi Yin", "Chunyuan Li", "Pengchuan Zhang", "Xiaowei Hu", "Lei Zhang", "Lijuan Wang", "Houdong Hu", "Li Dong", "Furu Wei", "Yejin Choi", "Jianfeng Gao"]
year: 2020
venue: "arXiv:2004.06165"
source: "https://arxiv.org/abs/2004.06165"
doi: "10.48550/arxiv.2004.06165"
arxiv: "2004.06165"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2020-oscar-object-semantics-aligned-pre/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2020-oscar-object-semantics-aligned-pre/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

# Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

**Authors:** [Xiujun Li](https://md.research.elamur.ai/free/en/authors/xiujun-li/), Xi Yin, [Chunyuan Li](https://md.research.elamur.ai/free/en/authors/chunyuan-li/), [Pengchuan Zhang](https://md.research.elamur.ai/free/en/authors/pengchuan-zhang/), Xiaowei Hu, [Lei Zhang](https://md.research.elamur.ai/free/en/authors/lei-zhang/), [Lijuan Wang](https://md.research.elamur.ai/free/en/authors/lijuan-wang/), Houdong Hu, [Li Dong](https://md.research.elamur.ai/free/en/authors/li-dong/), [Furu Wei](https://md.research.elamur.ai/free/en/authors/furu-wei/), [Yejin Choi](https://md.research.elamur.ai/free/en/authors/yejin-choi/), [Jianfeng Gao](https://md.research.elamur.ai/free/en/authors/jianfeng-gao/) · **Year:** 2020 · **Venue:** arXiv:2004.06165 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2004.06165) · [PDF](https://arxiv.org/pdf/2004.06165) · [DOI 10.48550/arxiv.2004.06165](https://doi.org/10.48550/arxiv.2004.06165) · [arXiv 2004.06165](https://arxiv.org/abs/2004.06165) · [OpenAlex](https://openalex.org/W3091588028)

**Citations:** 1542 citations (openalex-snapshot · 2026-10-14) · [285](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Vision-language pre-training (VLP) models usually feed image region features and text tokens together into a Transformer. Self-attention must then work out which regions correspond to which words without explicit supervision. Overlapping, over-sampled regions make this harder. Oscar addresses the problem by adding a third input: object tags that a Faster R-CNN detector predicts for the image. Because the tags are words, a BERT-initialized model can easily link them to the caption. They then act as anchors that connect image regions to the linguistic semantic space.

Each training example is a word–tag–region triple. Pre-training uses two objectives:
- a masked token loss over words and tags;
- a contrastive loss that detects whether the tag sequence was swapped.

The model was pre-trained on a public collection of 6.5 million image-text pairs. It was then fine-tuned on seven understanding and generation tasks and reached new state-of-the-art results on six of them. The work shows that cheap, explicit object semantics can make cross-modal alignment learning more efficient and better at generalizing.

## Key points
- **Motivation:** On MS COCO, an image and its caption share at least 1, 2 or 3 objects in 49.7%, 22.2% and 12.9% of cases, respectively.
- **Input design:** Region features are 2048-dimensional Faster R-CNN vectors plus box coordinates, linearly projected to the word-embedding size. Tags are embedded as ordinary word tokens.
- **Pre-training objectives:**
  - Masked Token Loss masks 15% of word and tag tokens.
  - The contrastive loss replaces the tag sequence with a random one 50% of the time and classifies the [CLS] output.
- **Downstream tasks:** VQA, GQA, NLVR2, image-text and text-image retrieval, image captioning, and novel object captioning.
- **Tag ablation (589k pre-training steps):**
  - Both VG and Open Images tags beat a no-tag baseline. For example, VQA dev rose from 70.93 to 71.70 with VG tags, and captioning CIDEr rose from about 115.6 to 123.4.
  - VG tags were slightly better than OI tags, possibly because the VG detector covers a more diverse set of objects.
  - Code and models are released at github.com/microsoft/Oscar.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

285 papers of our library cite it (of the 140673 whose bibliographies we read); 21 of them have a page here, 264 not yet.

- [Flamingo: a Visual Language Model for Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/)
- [SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification](https://md.research.elamur.ai/free/en/papers/fersini-2022-semeval-2022-task-5-multimedia/)
- [GRAVL-BERT: Graphical Visual-Linguistic Representations for Multimodal Coreference Resolution](https://md.research.elamur.ai/free/en/papers/guo-2022-gravl-bert-graphical-visual-linguistic/)
- [Probing Contextual Language Models for Common Ground with Visual Representations](https://md.research.elamur.ai/free/en/papers/ilharco-2021-probing-contextual-language-models-common/)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://md.research.elamur.ai/free/en/papers/jia-2021-scaling-up-visual-vision-language/)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://md.research.elamur.ai/free/en/papers/kim-2021-vilt-vision-language-transformer-convolution/)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/li-2023-evaluating-object-hallucination-large-vision/)
- [Learning Transferable Visual Models From Natural Language Supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/)
- [The Prompt Report: A Systematic Survey of Prompt Engineering Techniques](https://md.research.elamur.ai/free/en/papers/schulhoff-2024-prompt-report-systematic-survey-prompt/)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://md.research.elamur.ai/free/en/papers/tsimpoukelli-2021-multimodal-few-shot-learning-frozen/)
- [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://md.research.elamur.ai/free/en/papers/wang-2021-simvlm-simple-visual-language-model/)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://md.research.elamur.ai/free/en/papers/wang-2022-ofa-unifying-architectures-tasks-modalities/)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://md.research.elamur.ai/free/en/papers/wu-2023-visual-chatgpt-talking-drawing-editing/)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://md.research.elamur.ai/free/en/papers/yang-2023-dawn-lmms-preliminary-explorations-gpt/)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://md.research.elamur.ai/free/en/papers/yao-2022-webshop-towards-scalable-real-world/)
- [视觉语言预训练综述](https://md.research.elamur.ai/free/en/papers/yin-2022-vision-language-pre/)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{li2020oscar,
  title = {{Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks}},
  author = {Li, Xiujun and Yin, Xi and Li, Chunyuan and Zhang, Pengchuan and Hu, Xiaowei and Zhang, Lei and Wang, Lijuan and Hu, Houdong and Dong, Li and Wei, Furu and Choi, Yejin and Gao, Jianfeng},
  year = {2020},
  doi = {10.48550/arxiv.2004.06165},
  eprint = {2004.06165},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2004.06165}
}
```

- [← previous Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports](https://md.research.elamur.ai/free/en/papers/zhang-2020-optimizing-factual-correctness-summary-study/)
- [next → Overview of the IWPT 2020 Shared Task on Parsing into Enhanced Universal Dependencies](https://md.research.elamur.ai/free/en/papers/ud-2020-bouma-overview-iwpt-2020-shared-task/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/)

## References

46 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Agrawal, H., Desai, K., Wang, Y., Chen, X., Jain, R., Johnson, M., Batra, D., Parikh, D., Lee, S., Anderson, P.: nocaps: novel object captioning at scale. In: ICCV (2019) [find](https://scholar.google.com/scholar?q=Agrawal%2C+H%2E%2C+Desai%2C+K%2E%2C+Wang%2C+Y%2E%2C+Chen%2C+X%2E%2C+Jain%2C+R%2E%2C+Johnson%2C+M%2E%2C+Batra%2C+D%2E%2C+Parikh%2C+D%2E%2C+Lee%2C+S%2E%2C+Anderson%2C+P%2E%3A+nocaps%3A+novel+object+captioning+at+scale%2E+In%3A+ICCV+%282019%29)
2. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering. In: CVPR (2018) [find](https://scholar.google.com/scholar?q=Anderson%2C+P%2E%2C+He%2C+X%2E%2C+Buehler%2C+C%2E%2C+Teney%2C+D%2E%2C+Johnson%2C+M%2E%2C+Gould%2C+S%2E%2C+Zhang%2C+L%2E%3A+Bottom%2Dup+and+top%2Ddown+attention+for+image+captioning+and+visual+question+answering%2E+In%3A+CVPR+%282018%29)
3. Brown, P.F., Lai, J.C., Mercer, R.L.: Aligning sentences in parallel corpora. In: Proceedings of the 29th annual meeting on Association for Computational Linguistics (1991) [find](https://scholar.google.com/scholar?q=Brown%2C+P%2EF%2E%2C+Lai%2C+J%2EC%2E%2C+Mercer%2C+R%2EL%2E%3A+Aligning+sentences+in+parallel+corpora%2E+In%3A+Proceedings+of+the+29th+annual+meeting+on+Association+for+Computational+Linguistics+%281991%29) [our card](https://md.research.elamur.ai/free/en/papers/brown-1991-aligning-sentences-parallel-corpora/)
4. Chen, W., Gan, Z., Li, L., Cheng, Y., Wang, W., Liu, J.: Meta module network for compositional visual reasoning. arXiv preprint arXiv:1910.03230 (2019) — [arXiv:1910.03230](https://arxiv.org/abs/1910.03230) [find](https://scholar.google.com/scholar?q=Chen%2C+W%2E%2C+Gan%2C+Z%2E%2C+Li%2C+L%2E%2C+Cheng%2C+Y%2E%2C+Wang%2C+W%2E%2C+Liu%2C+J%2E%3A+Meta+module+network+for+compositional+visual+reasoning%2E+arXiv+preprint+arXiv%3A1910%2E03230+%282019%29+%E2%80%94+%5BarXiv%3A1910%2E03230%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1910%2E)
5. Chen, Y.C., Li, L., Yu, L., Kholy, A.E., Ahmed, F., Gan, Z., Cheng, Y., Liu, J.: Uniter: Learning universal image-text representations. arXiv preprint arXiv:1909.11740 (2019) [find](https://scholar.google.com/scholar?q=Chen%2C+Y%2EC%2E%2C+Li%2C+L%2E%2C+Yu%2C+L%2E%2C+Kholy%2C+A%2EE%2E%2C+Ahmed%2C+F%2E%2C+Gan%2C+Z%2E%2C+Cheng%2C+Y%2E%2C+Liu%2C+J%2E%3A+Uniter%3A+Learning+universal+image%2Dtext+representations%2E+arXiv+preprint+arXiv%3A1909%2E11740+%282019%29) [our card](https://md.research.elamur.ai/free/en/papers/chen-2019-uniter-universal-image-text-representation/)
6. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL (2019) [find](https://scholar.google.com/scholar?q=Devlin%2C+J%2E%2C+Chang%2C+M%2EW%2E%2C+Lee%2C+K%2E%2C+Toutanova%2C+K%2E%3A+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+NAACL+%282019%29)
7. Faghri, F., Fleet, D.J., Kiros, J.R., Fidler, S.: Vse++: Improved visual-semantic embeddings. arXiv preprint arXiv:1707.05612 2(7), 8 (2017) — [arXiv:1707.05612](https://arxiv.org/abs/1707.05612) [find](https://scholar.google.com/scholar?q=Faghri%2C+F%2E%2C+Fleet%2C+D%2EJ%2E%2C+Kiros%2C+J%2ER%2E%2C+Fidler%2C+S%2E%3A+Vse%2B%2B%3A+Improved+visual%2Dsemantic+embeddings%2E+arXiv+preprint+arXiv%3A1707%2E05612+2%287%29%2C+8+%282017%29+%E2%80%94+%5BarXiv%3A1707%2E05612%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1707%2E05612%29)
8. Frome, A., Corrado, G.S., Shlens, J., Bengio, S., Dean, J., Ranzato, M., Mikolov, T.: DeViSE: A deep visual-semantic embedding model. In: NeurIPS (2013) [find](https://scholar.google.com/scholar?q=Frome%2C+A%2E%2C+Corrado%2C+G%2ES%2E%2C+Shlens%2C+J%2E%2C+Bengio%2C+S%2E%2C+Dean%2C+J%2E%2C+Ranzato%2C+M%2E%2C+Mikolov%2C+T%2E%3A+DeViSE%3A+A+deep+visual%2Dsemantic+embedding+model%2E+In%3A+NeurIPS+%282013%29)
9. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In: CVPR (2017) [find](https://scholar.google.com/scholar?q=Goyal%2C+Y%2E%2C+Khot%2C+T%2E%2C+Summers%2DStay%2C+D%2E%2C+Batra%2C+D%2E%2C+Parikh%2C+D%2E%3A+Making+the+V+in+VQA+matter%3A+Elevating+the+role+of+image+understanding+in+visual+question+answering%2E+In%3A+CVPR+%282017%29) [our card](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/)
10. Hao, W., Li, C., Li, X., Carin, L., Gao, J.: Towards learning a generic agent for vision-and-language navigation via pre-training. CVPR (2020) [find](https://scholar.google.com/scholar?q=Hao%2C+W%2E%2C+Li%2C+C%2E%2C+Li%2C+X%2E%2C+Carin%2C+L%2E%2C+Gao%2C+J%2E%3A+Towards+learning+a+generic+agent+for+vision%2Dand%2Dlanguage+navigation+via+pre%2Dtraining%2E+CVPR+%282020%29)
11. Huang, L., Wang, W., Chen, J., Wei, X.Y.: Attention on attention for image captioning. In: ICCV (2019) [find](https://scholar.google.com/scholar?q=Huang%2C+L%2E%2C+Wang%2C+W%2E%2C+Chen%2C+J%2E%2C+Wei%2C+X%2EY%2E%3A+Attention+on+attention+for+image+captioning%2E+In%3A+ICCV+%282019%29)
12. Hudson, D., Manning, C.D.: Learning by abstraction: The neural state machine. In: NeurIPS (2019) [find](https://scholar.google.com/scholar?q=Hudson%2C+D%2E%2C+Manning%2C+C%2ED%2E%3A+Learning+by+abstraction%3A+The+neural+state+machine%2E+In%3A+NeurIPS+%282019%29)
13. Hudson, D.A., Manning, C.D.: GQA: A new dataset for real-world visual reasoning and compositional question answering. arXiv preprint arXiv:1902.09506 (2019) [find](https://scholar.google.com/scholar?q=Hudson%2C+D%2EA%2E%2C+Manning%2C+C%2ED%2E%3A+GQA%3A+A+new+dataset+for+real%2Dworld+visual+reasoning+and+compositional+question+answering%2E+arXiv+preprint+arXiv%3A1902%2E09506+%282019%29) [our card](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/)
14. Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: CVPR (2015) [find](https://scholar.google.com/scholar?q=Karpathy%2C+A%2E%2C+Fei%2DFei%2C+L%2E%3A+Deep+visual%2Dsemantic+alignments+for+generating+image+descriptions%2E+In%3A+CVPR+%282015%29) [our card](https://md.research.elamur.ai/free/en/papers/karpathy-2014-deep-visual-semantic-alignments-generating/)
15. Kiros, R., Salakhutdinov, R., Zemel, R.S.: Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539 (2014) [find](https://scholar.google.com/scholar?q=Kiros%2C+R%2E%2C+Salakhutdinov%2C+R%2E%2C+Zemel%2C+R%2ES%2E%3A+Unifying+visual%2Dsemantic+embeddings+with+multimodal+neural+language+models%2E+arXiv+preprint+arXiv%3A1411%2E2539+%282014%29) [our card](https://md.research.elamur.ai/free/en/papers/kiros-2014-unifying-visual-semantic-embeddings-multimodal/)
16. Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., et al.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision 123(1), 32–73 (2017) [find](https://scholar.google.com/scholar?q=Krishna%2C+R%2E%2C+Zhu%2C+Y%2E%2C+Groth%2C+O%2E%2C+Johnson%2C+J%2E%2C+Hata%2C+K%2E%2C+Kravitz%2C+J%2E%2C+Chen%2C+S%2E%2C+Kalantidis%2C+Y%2E%2C+Li%2C+L%2EJ%2E%2C+Shamma%2C+D%2EA%2E%2C+et+al%2E%3A+Visual+genome%3A+Connecting+language+and+vision+using+crowdsourced+dense+im) [our card](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/)
17. Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Duerig, T., et al.: The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. arXiv preprint arXiv:1811.00982 (2018) — [arXiv:1811.00982](https://arxiv.org/abs/1811.00982) [find](https://scholar.google.com/scholar?q=Kuznetsova%2C+A%2E%2C+Rom%2C+H%2E%2C+Alldrin%2C+N%2E%2C+Uijlings%2C+J%2E%2C+Krasin%2C+I%2E%2C+Pont%2DTuset%2C+J%2E%2C+Kamali%2C+S%2E%2C+Popov%2C+S%2E%2C+Malloci%2C+M%2E%2C+Duerig%2C+T%2E%2C+et+al%2E%3A+The+open+images+dataset+v4%3A+Unified+image+classification%2C+object)
18. Lee, K.H., Chen, X., Hua, G., Hu, H., He, X.: Stacked cross attention for image-text matching. In: ECCV (2018) [find](https://scholar.google.com/scholar?q=Lee%2C+K%2EH%2E%2C+Chen%2C+X%2E%2C+Hua%2C+G%2E%2C+Hu%2C+H%2E%2C+He%2C+X%2E%3A+Stacked+cross+attention+for+image%2Dtext+matching%2E+In%3A+ECCV+%282018%29)
19. Li, G., Duan, N., Fang, Y., Jiang, D., Zhou, M.: Unicoder-VL: A universal encoder for vision and language by cross-modal pre-training. arXiv preprint arXiv:1908.06066 (2019) X. Li, X. Yin, C. Li et al. — [arXiv:1908.06066](https://arxiv.org/abs/1908.06066) [find](https://scholar.google.com/scholar?q=Li%2C+G%2E%2C+Duan%2C+N%2E%2C+Fang%2C+Y%2E%2C+Jiang%2C+D%2E%2C+Zhou%2C+M%2E%3A+Unicoder%2DVL%3A+A+universal+encoder+for+vision+and+language+by+cross%2Dmodal+pre%2Dtraining%2E+arXiv+preprint+arXiv%3A1908%2E06066+%282019%29+X%2E+Li%2C+X%2E+Yin%2C+C%2E+Li+et+al)
20. Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., Chang, K.W.: Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 (2019) [find](https://scholar.google.com/scholar?q=Li%2C+L%2EH%2E%2C+Yatskar%2C+M%2E%2C+Yin%2C+D%2E%2C+Hsieh%2C+C%2EJ%2E%2C+Chang%2C+K%2EW%2E%3A+Visualbert%3A+A+simple+and+performant+baseline+for+vision+and+language%2E+arXiv+preprint+arXiv%3A1908%2E03557+%282019%29) [our card](https://md.research.elamur.ai/free/en/papers/li-2019-visualbert-simple-performant-baseline-vision/)
21. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: Common objects in context. In: ECCV (2014) [find](https://scholar.google.com/scholar?q=Lin%2C+T%2EY%2E%2C+Maire%2C+M%2E%2C+Belongie%2C+S%2E%2C+Hays%2C+J%2E%2C+Perona%2C+P%2E%2C+Ramanan%2C+D%2E%2C+Dolla%CC%81r%2C+P%2E%2C+Zitnick%2C+C%2EL%2E%3A+Microsoft+COCO%3A+Common+objects+in+context%2E+In%3A+ECCV+%282014%29) [our card](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/)
22. Lu, J., Batra, D., Parikh, D., Lee, S.: VilBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: NeurIPS (2019) [find](https://scholar.google.com/scholar?q=Lu%2C+J%2E%2C+Batra%2C+D%2E%2C+Parikh%2C+D%2E%2C+Lee%2C+S%2E%3A+VilBERT%3A+Pretraining+task%2Dagnostic+visiolinguistic+representations+for+vision%2Dand%2Dlanguage+tasks%2E+In%3A+NeurIPS+%282019%29) [our card](https://md.research.elamur.ai/free/en/papers/lu-2019-vilbert-pretraining-task-agnostic-visiolinguistic/)
23. Lu, J., Goswami, V., Rohrbach, M., Parikh, D., Lee, S.: 12-in-1: Multi-Task vision and language representation learning. arXiv preprint arXiv:1912.02315 (2019) — [arXiv:1912.02315](https://arxiv.org/abs/1912.02315) [find](https://scholar.google.com/scholar?q=Lu%2C+J%2E%2C+Goswami%2C+V%2E%2C+Rohrbach%2C+M%2E%2C+Parikh%2C+D%2E%2C+Lee%2C+S%2E%3A+12%2Din%2D1%3A+Multi%2DTask+vision+and+language+representation+learning%2E+arXiv+preprint+arXiv%3A1912%2E02315+%282019%29+%E2%80%94+%5BarXiv%3A1912%2E02315%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
24. Maaten, L.v.d., Hinton, G.: Visualizing data using t-SNE. Journal of machine learning research (2008) [find](https://scholar.google.com/scholar?q=Maaten%2C+L%2Ev%2Ed%2E%2C+Hinton%2C+G%2E%3A+Visualizing+data+using+t%2DSNE%2E+Journal+of+machine+learning+research+%282008%29)
25. Norouzi, M., Mikolov, T., Bengio, S., Singer, Y., Shlens, J., Frome, A., Corrado, G.S., Dean, J.: Zero-shot learning by convex combination of semantic embeddings. arXiv preprint arXiv:1312.5650 (2013) — [arXiv:1312.5650](https://arxiv.org/abs/1312.5650) [find](https://scholar.google.com/scholar?q=Norouzi%2C+M%2E%2C+Mikolov%2C+T%2E%2C+Bengio%2C+S%2E%2C+Singer%2C+Y%2E%2C+Shlens%2C+J%2E%2C+Frome%2C+A%2E%2C+Corrado%2C+G%2ES%2E%2C+Dean%2C+J%2E%3A+Zero%2Dshot+learning+by+convex+combination+of+semantic+embeddings%2E+arXiv+preprint+arXiv%3A1312%2E5650+%282013%29)
26. Ordonez, V., Kulkarni, G., Berg, T.L.: Im2text: Describing images using 1 million captioned photographs. In: NeurIPS (2011) [find](https://scholar.google.com/scholar?q=Ordonez%2C+V%2E%2C+Kulkarni%2C+G%2E%2C+Berg%2C+T%2EL%2E%3A+Im2text%3A+Describing+images+using+1+million+captioned+photographs%2E+In%3A+NeurIPS+%282011%29)
27. Qi, D., Su, L., Song, J., Cui, E., Bharti, T., Sacheti, A.: Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data. arXiv preprint arXiv:2001.07966 (2020) — [arXiv:2001.07966](https://arxiv.org/abs/2001.07966) [find](https://scholar.google.com/scholar?q=Qi%2C+D%2E%2C+Su%2C+L%2E%2C+Song%2C+J%2E%2C+Cui%2C+E%2E%2C+Bharti%2C+T%2E%2C+Sacheti%2C+A%2E%3A+Imagebert%3A+Cross%2Dmodal+pre%2Dtraining+with+large%2Dscale+weak%2Dsupervised+image%2Dtext+data%2E+arXiv+preprint+arXiv%3A2001%2E07966+%282020%29+%E2%80%94+%5BarXiv%3A2001%2E0)
28. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in neural information processing systems. pp. 91–99 (2015) [find](https://scholar.google.com/scholar?q=Ren%2C+S%2E%2C+He%2C+K%2E%2C+Girshick%2C+R%2E%2C+Sun%2C+J%2E%3A+Faster+r%2Dcnn%3A+Towards+real%2Dtime+object+detection+with+region+proposal+networks%2E+In%3A+Advances+in+neural+information+processing+systems%2E+pp%2E+91%E2%80%9399+%282015%29) [our card](https://md.research.elamur.ai/free/en/papers/ren-2015-faster-r-cnn-towards-real/)
29. Ren, Z., Jin, H., Lin, Z., Fang, C., Yuille, A.: Joint image-text representation by gaussian visual-semantic embedding. In: Multimedia (2016) [find](https://scholar.google.com/scholar?q=Ren%2C+Z%2E%2C+Jin%2C+H%2E%2C+Lin%2C+Z%2E%2C+Fang%2C+C%2E%2C+Yuille%2C+A%2E%3A+Joint+image%2Dtext+representation+by+gaussian+visual%2Dsemantic+embedding%2E+In%3A+Multimedia+%282016%29)
30. Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., Goel, V.: Self-critical sequence training for image captioning. In: CVPR (2017) [find](https://scholar.google.com/scholar?q=Rennie%2C+S%2EJ%2E%2C+Marcheret%2C+E%2E%2C+Mroueh%2C+Y%2E%2C+Ross%2C+J%2E%2C+Goel%2C+V%2E%3A+Self%2Dcritical+sequence+training+for+image+captioning%2E+In%3A+CVPR+%282017%29) [our card](https://md.research.elamur.ai/free/en/papers/rennie-2016-self-critical-sequence-training-image/)
31. Sharma, P., Ding, N., Goodman, S., Soricut, R.: Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: Annual Meeting of the Association for Computational Linguistics (2018) [find](https://scholar.google.com/scholar?q=Sharma%2C+P%2E%2C+Ding%2C+N%2E%2C+Goodman%2C+S%2E%2C+Soricut%2C+R%2E%3A+Conceptual+captions%3A+A+cleaned%2C+hypernymed%2C+image+alt%2Dtext+dataset+for+automatic+image+captioning%2E+In%3A+Annual+Meeting+of+the+Association+for+Computation) [our card](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/)
32. Shi, B., Ji, L., Lu, P., Niu, Z., Duan, N.: Knowledge aware semantic concept expansion for image-text matching. In: IJCAI (2019) [find](https://scholar.google.com/scholar?q=Shi%2C+B%2E%2C+Ji%2C+L%2E%2C+Lu%2C+P%2E%2C+Niu%2C+Z%2E%2C+Duan%2C+N%2E%3A+Knowledge+aware+semantic+concept+expansion+for+image%2Dtext+matching%2E+In%3A+IJCAI+%282019%29)
33. Socher, R., Fei-Fei, L.: Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora. In: CVPR (2010) [find](https://scholar.google.com/scholar?q=Socher%2C+R%2E%2C+Fei%2DFei%2C+L%2E%3A+Connecting+modalities%3A+Semi%2Dsupervised+segmentation+and+annotation+of+images+using+unaligned+text+corpora%2E+In%3A+CVPR+%282010%29)
34. Socher, R., Ganjoo, M., Manning, C.D., Ng, A.: Zero-shot learning through cross-modal transfer. In: NeurIPS (2013) [find](https://scholar.google.com/scholar?q=Socher%2C+R%2E%2C+Ganjoo%2C+M%2E%2C+Manning%2C+C%2ED%2E%2C+Ng%2C+A%2E%3A+Zero%2Dshot+learning+through+cross%2Dmodal+transfer%2E+In%3A+NeurIPS+%282013%29)
35. Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., Dai, J.: VL-BERT: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530 (2019) [find](https://scholar.google.com/scholar?q=Su%2C+W%2E%2C+Zhu%2C+X%2E%2C+Cao%2C+Y%2E%2C+Li%2C+B%2E%2C+Lu%2C+L%2E%2C+Wei%2C+F%2E%2C+Dai%2C+J%2E%3A+VL%2DBERT%3A+Pre%2Dtraining+of+generic+visual%2Dlinguistic+representations%2E+arXiv+preprint+arXiv%3A1908%2E08530+%282019%29) [our card](https://md.research.elamur.ai/free/en/papers/su-2019-vl-bert-pre-training-generic/)
36. Suhr, A., Zhou, S., Zhang, A., Zhang, I., Bai, H., Artzi, Y.: A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXiv:1811.00491 (2018) [find](https://scholar.google.com/scholar?q=Suhr%2C+A%2E%2C+Zhou%2C+S%2E%2C+Zhang%2C+A%2E%2C+Zhang%2C+I%2E%2C+Bai%2C+H%2E%2C+Artzi%2C+Y%2E%3A+A+corpus+for+reasoning+about+natural+language+grounded+in+photographs%2E+arXiv+preprint+arXiv%3A1811%2E00491+%282018%29) [our card](https://md.research.elamur.ai/free/en/papers/suhr-2019-corpus-reasoning-natural-language-grounded/)
37. Sun, C., Myers, A., Vondrick, C., Murphy, K., Schmid, C.: VideoBERT: A joint model for video and language representation learning. ICCV (2019) [find](https://scholar.google.com/scholar?q=Sun%2C+C%2E%2C+Myers%2C+A%2E%2C+Vondrick%2C+C%2E%2C+Murphy%2C+K%2E%2C+Schmid%2C+C%2E%3A+VideoBERT%3A+A+joint+model+for+video+and+language+representation+learning%2E+ICCV+%282019%29) [our card](https://md.research.elamur.ai/free/en/papers/sun-2019-videobert-joint-model-video-language/)
38. Tan, H., Bansal, M.: LXMERT: Learning cross-modality encoder representations from transformers. EMNLP (2019) [find](https://scholar.google.com/scholar?q=Tan%2C+H%2E%2C+Bansal%2C+M%2E%3A+LXMERT%3A+Learning+cross%2Dmodality+encoder+representations+from+transformers%2E+EMNLP+%282019%29) [our card](https://md.research.elamur.ai/free/en/papers/tan-2019-lxmert-learning-cross-modality-encoder/)
39. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: NeurIPS (2017) [find](https://scholar.google.com/scholar?q=Vaswani%2C+A%2E%2C+Shazeer%2C+N%2E%2C+Parmar%2C+N%2E%2C+Uszkoreit%2C+J%2E%2C+Jones%2C+L%2E%2C+Gomez%2C+A%2EN%2E%2C+Kaiser%2C+L%2E%2C+Polosukhin%2C+I%2E%3A+Attention+is+all+you+need%2E+In%3A+NeurIPS+%282017%29)
40. Wang, Y., Yang, H., Qian, X., Ma, L., Lu, J., Li, B., Fan, X.: Position focused attention network for image-text matching. arXiv preprint arXiv:1907.09748 (2019) — [arXiv:1907.09748](https://arxiv.org/abs/1907.09748) [find](https://scholar.google.com/scholar?q=Wang%2C+Y%2E%2C+Yang%2C+H%2E%2C+Qian%2C+X%2E%2C+Ma%2C+L%2E%2C+Lu%2C+J%2E%2C+Li%2C+B%2E%2C+Fan%2C+X%2E%3A+Position+focused+attention+network+for+image%2Dtext+matching%2E+arXiv+preprint+arXiv%3A1907%2E09748+%282019%29+%E2%80%94+%5BarXiv%3A1907%2E09748%5D%28https%3A%2F%2Farxiv%2Eorg)
41. Wang, Z., Liu, X., Li, H., Sheng, L., Yan, J., Wang, X., Shao, J.: CAMP: Cross-Modal adaptive message passing for text-image retrieval. In: ICCV (2019) Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks [find](https://scholar.google.com/scholar?q=Wang%2C+Z%2E%2C+Liu%2C+X%2E%2C+Li%2C+H%2E%2C+Sheng%2C+L%2E%2C+Yan%2C+J%2E%2C+Wang%2C+X%2E%2C+Shao%2C+J%2E%3A+CAMP%3A+Cross%2DModal+adaptive+message+passing+for+text%2Dimage+retrieval%2E+In%3A+ICCV+%282019%29+Oscar%3A+Object%2DSemantics+Aligned+Pre%2Dtraining+for)
42. Wu, Q., Shen, C., Liu, L., Dick, A., Van Den Hengel, A.: What value do explicit high level concepts have in vision to language problems? In: CVPR (2016) [find](https://scholar.google.com/scholar?q=Wu%2C+Q%2E%2C+Shen%2C+C%2E%2C+Liu%2C+L%2E%2C+Dick%2C+A%2E%2C+Van+Den+Hengel%2C+A%2E%3A+What+value+do+explicit+high+level+concepts+have+in+vision+to+language+problems%3F+In%3A+CVPR+%282016%29)
43. You, Q., Jin, H., Wang, Z., Fang, C., Luo, J.: Image captioning with semantic attention. In: CVPR (2016) [find](https://scholar.google.com/scholar?q=You%2C+Q%2E%2C+Jin%2C+H%2E%2C+Wang%2C+Z%2E%2C+Fang%2C+C%2E%2C+Luo%2C+J%2E%3A+Image+captioning+with+semantic+attention%2E+In%3A+CVPR+%282016%29)
44. Young, P., Lai, A., Hodosh, M., Hockenmaier, J.: From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2, 67–78 (2014) [find](https://scholar.google.com/scholar?q=Young%2C+P%2E%2C+Lai%2C+A%2E%2C+Hodosh%2C+M%2E%2C+Hockenmaier%2C+J%2E%3A+From+image+descriptions+to+visual+denotations%3A+New+similarity+metrics+for+semantic+inference+over+event+descriptions%2E+Transactions+of+the+Association+f)
45. Zheng, Z., Zheng, L., Garrett, M., Yang, Y., Shen, Y.D.: Dual-path convolutional image-text embedding with instance loss. arXiv preprint arXiv:1711.05535 (2017) — [arXiv:1711.05535](https://arxiv.org/abs/1711.05535) [find](https://scholar.google.com/scholar?q=Zheng%2C+Z%2E%2C+Zheng%2C+L%2E%2C+Garrett%2C+M%2E%2C+Yang%2C+Y%2E%2C+Shen%2C+Y%2ED%2E%3A+Dual%2Dpath+convolutional+image%2Dtext+embedding+with+instance+loss%2E+arXiv+preprint+arXiv%3A1711%2E05535+%282017%29+%E2%80%94+%5BarXiv%3A1711%2E05535%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
46. Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J.J., Gao, J.: Unified vision-language pre-training for image captioning and VQA. AAAI (2020) X. Li, X. Yin, C. Li et al. [find](https://scholar.google.com/scholar?q=Zhou%2C+L%2E%2C+Palangi%2C+H%2E%2C+Zhang%2C+L%2E%2C+Hu%2C+H%2E%2C+Corso%2C+J%2EJ%2E%2C+Gao%2C+J%2E%3A+Unified+vision%2Dlanguage+pre%2Dtraining+for+image+captioning+and+VQA%2E+AAAI+%282020%29+X%2E+Li%2C+X%2E+Yin%2C+C%2E+Li+et+al%2E)