---
title: "ERNIE-ViL — paper by Yu et al. (2020) — summary and ideas"
description: "Most vision-language pre-training models mask sub-words at random. 31 references, cited by 41 papers here."
authors: ["Fei Yu", "Jiji Tang", "Weichong Yin", "Yu Sun", "Hao Tian", "Hua Wu", "Haifeng Wang"]
year: 2020
venue: "arXiv:2006.16934"
source: "https://arxiv.org/abs/2006.16934"
doi: "10.48550/arxiv.2006.16934"
arxiv: "2006.16934"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yu-2020-ernie-vil-knowledge-enhanced/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yu-2020-ernie-vil-knowledge-enhanced/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph

# ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph

**Authors:** [Fei Yu](https://md.research.elamur.ai/free/en/authors/fei-yu/), Jiji Tang, Weichong Yin, [Yu Sun](https://md.research.elamur.ai/free/en/authors/yu-sun/), [Hao Tian](https://md.research.elamur.ai/free/en/authors/hao-tian/), [Hua Wu](https://md.research.elamur.ai/free/en/authors/hua-wu/) and [Haifeng Wang](https://md.research.elamur.ai/free/en/authors/haifeng-wang/) · **Year:** 2020 · **Venue:** arXiv:2006.16934 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2006.16934) · [PDF](https://arxiv.org/pdf/2006.16934) · [DOI 10.48550/arxiv.2006.16934](https://doi.org/10.48550/arxiv.2006.16934) · [arXiv 2006.16934](https://arxiv.org/abs/2006.16934) · [OpenAlex](https://openalex.org/W3038476992)

**Citations:** 105 citations (OpenAlex, 2026-10-14) · [41](https://md.research.elamur.ai/free/en/papers/yu-2020-ernie-vil-knowledge-enhanced/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Most vision-language pre-training models mask sub-words at random. As a result, they give generic words and content-bearing words equal weight and learn fine-grained correspondences between image and text poorly. ERNIE-ViL, from Baidu, addresses this by injecting structured knowledge from scene graphs.

A Scene Graph Parser turns each caption into a graph of objects, their attributes and the relations between them. During pre-training, nodes of each type are masked and must be recovered with help from the image. This gives three Scene Graph Prediction (SGP) objectives: Object, Attribute and Relationship Prediction. The backbone is a two-stream cross-modal Transformer similar to ViLBERT. It takes WordPiece tokens and region features from an object detector.

Pre-training uses Conceptual Captions and SBU Captions, with an optional further stage on the in-domain MS-COCO and Visual-Genome. ERNIE-ViL was tested on five downstream tasks: VQA 2.0, VCR, RefCOCO+, and image and text retrieval on Flickr30K. It outperformed comparable models on all of them and took first place on the VCR leaderboard. The work shows that choosing masking targets by semantic structure is a simple way to improve fine-grained grounding.

## Key points
- On the VCR leaderboard, the model is the best single model and gains 3.7 points absolute on Q→AR over the previous state of the art.
- With only out-of-domain data (CC with 3.0M pairs, SBU with 1.0M pairs), ERNIE-ViL-large beats VLBERT-large by:
  - 6.60% on VCR Q→AR;
  - 1.74% on VQA test-std;
  - 2.40% on both RefCOCO+ testA and testB.
- ERNIE-ViL-base beats Unicoder-VL-base on Flickr30K R@1 by 2.94% for image retrieval and 0.50% for text retrieval.
- Ablation of the base model with BERT initialization shows that SGP raises:
  - VCR dev from 59.06 to 59.92;
  - RefCOCO+ val from 72.81 to 73.50;
  - image retrieval R@1 from 70.74 to 72.96.
  
  Initializing the text stream from ERNIE 2.0 instead lifts VCR to 61.24.
- A vision-conditioned cloze test used 15,000 Flickr30K pairs with 5,000 masked tokens per node type. SGP raised overall ACC@1 from 49.75 to 51.75. Gains by type were +3.08 for relationships, +1.84 for attributes and +1.20 for objects.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Baidu](https://md.research.elamur.ai/free/en/companies/baidu/) — Company, Beijing, China · 40 works · 14,627 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 269 works

## Cite

Cite as: Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu and Haifeng Wang. 2020. ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph. arXiv:2006.16934.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{yu2020ernie,
  title = {{ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph}},
  author = {Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu and Haifeng Wang},
  year = {2020},
  doi = {10.48550/arxiv.2006.16934},
  eprint = {2006.16934},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2006.16934}
}
```

- [← previous Efficient Second-Order TreeCRF for Neural Dependency Parsing](https://md.research.elamur.ai/free/en/papers/zhang-2020-efficient-second-order-treecrf-neural/)
- [next → Revealing the Myth of Higher-Order Inference in Coreference Resolution](https://md.research.elamur.ai/free/en/papers/xu-2020-revealing-myth-higher-order-inference/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/)

## References

31 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anderson, P.; Fernando, B.; Johnson, M.; and Gould, S. 2016. [Spice: Semantic propositional image caption evaluation](https://md.research.elamur.ai/free/en/papers/anderson-2016-spice-semantic-propositional-image-caption/). In European Conference on Computer Vision, 382– 398. Springer.
2. Anderson, P.; He, X.; Buehler, C.; Teney, D.; Johnson, M.; Gould, S.; and Zhang, L. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, 6077–6086. [find](https://scholar.google.com/scholar?q=Anderson%2C+P%2E%3B+He%2C+X%2E%3B+Buehler%2C+C%2E%3B+Teney%2C+D%2E%3B+Johnson%2C+M%2E%3B+Gould%2C+S%2E%3B+and+Zhang%2C+L%2E+2018%2E+Bottom%2Dup+and+top%2Ddown+attention+for+image+captioning+and+visual+question+answering%2E+In+Proceedings+of+the+IEE)
3. Antol, S.; Agrawal, A.; Lu, J.; Mitchell, M.; Batra, D.; Lawrence Zitnick, C.; and Parikh, D. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, 2425–2433. [find](https://scholar.google.com/scholar?q=Antol%2C+S%2E%3B+Agrawal%2C+A%2E%3B+Lu%2C+J%2E%3B+Mitchell%2C+M%2E%3B+Batra%2C+D%2E%3B+Lawrence+Zitnick%2C+C%2E%3B+and+Parikh%2C+D%2E+2015%2E+Vqa%3A+Visual+question+answering%2E+In+Proceedings+of+the+IEEE+international+conference+on+computer+visi)
4. Chen, Y.-C.; Li, L.; Yu, L.; Kholy, A. E.; Ahmed, F.; Gan, Z.; Cheng, Y.; and Liu, J. 2019. [Uniter: Learning universal image-text representations](https://md.research.elamur.ai/free/en/papers/chen-2019-uniter-universal-image-text-representation/). arXiv preprint arXiv:1909.11740 .
5. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805 .
6. Gan, Z.; Chen, Y.-C.; Li, L.; Zhu, C.; Cheng, Y.; and Liu, J. 2020. Large-Scale Adversarial Training for Vision-and-Language Representation Learning. arXiv preprint arXiv:2006.06195 . [find](https://scholar.google.com/scholar?q=Gan%2C+Z%2E%3B+Chen%2C+Y%2E%2DC%2E%3B+Li%2C+L%2E%3B+Zhu%2C+C%2E%3B+Cheng%2C+Y%2E%3B+and+Liu%2C+J%2E+2020%2E+Large%2DScale+Adversarial+Training+for+Vision%2Dand%2DLanguage+Representation+Learning%2E+arXiv+preprint+arXiv%3A2006%2E06195+%2E)
7. Huang, Z.; Zeng, Z.; Liu, B.; Fu, D.; and Fu, J. 2020. Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers. arXiv preprint arXiv:2004.00849 . [find](https://scholar.google.com/scholar?q=Huang%2C+Z%2E%3B+Zeng%2C+Z%2E%3B+Liu%2C+B%2E%3B+Fu%2C+D%2E%3B+and+Fu%2C+J%2E+2020%2E+Pixel%2DBERT%3A+Aligning+Image+Pixels+with+Text+by+Deep+Multi%2DModal+Transformers%2E+arXiv+preprint+arXiv%3A2004%2E00849+%2E)
8. Johnson, J.; Gupta, A.; and Fei-Fei, L. 2018. Image generation from scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1219– 1228. [find](https://scholar.google.com/scholar?q=Johnson%2C+J%2E%3B+Gupta%2C+A%2E%3B+and+Fei%2DFei%2C+L%2E+2018%2E+Image+generation+from+scene+graphs%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+1219%E2%80%93+1228%2E)
9. Johnson, J.; Krishna, R.; Stark, M.; Li, L.-J.; Shamma, D.; Bernstein, M.; and Fei-Fei, L. 2015. Image retrieval using scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, 3668–3678. [find](https://scholar.google.com/scholar?q=Johnson%2C+J%2E%3B+Krishna%2C+R%2E%3B+Stark%2C+M%2E%3B+Li%2C+L%2E%2DJ%2E%3B+Shamma%2C+D%2E%3B+Bernstein%2C+M%2E%3B+and+Fei%2DFei%2C+L%2E+2015%2E+Image+retrieval+using+scene+graphs%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+patter)
10. Kazemzadeh, S.; Ordonez, V.; Matten, M.; and Berg, T. 2014. [Referitgame: Referring to objects in photographs of natural scenes](https://md.research.elamur.ai/free/en/papers/kazemzadeh-2014-referitgame-referring-objects-photographs-natural/). In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 787–798.
11. Krishna, R.; Zhu, Y.; Groth, O.; Johnson, J.; Hata, K.; Kravitz, J.; Chen, S.; Kalantidis, Y.; Li, L.-J.; Shamma, D. A.; et al. 2017. [Visual genome: Connecting language and vision using crowdsourced dense image annotations](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/). International Journal of Computer Vision 123(1): 32–73.
12. Li, G.; Duan, N.; Fang, Y.; Jiang, D.; and Zhou, M. 2019a. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. arXiv preprint arXiv:1908.06066 . — [arXiv:1908.06066](https://arxiv.org/abs/1908.06066) [find](https://scholar.google.com/scholar?q=Li%2C+G%2E%3B+Duan%2C+N%2E%3B+Fang%2C+Y%2E%3B+Jiang%2C+D%2E%3B+and+Zhou%2C+M%2E+2019a%2E+Unicoder%2Dvl%3A+A+universal+encoder+for+vision+and+language+by+cross%2Dmodal+pre%2Dtraining%2E+arXiv+preprint+arXiv%3A1908%2E06066+%2E+%E2%80%94+%5BarXiv%3A1908%2E06066%5D%28)
13. Li, L. H.; Yatskar, M.; Yin, D.; Hsieh, C.-J.; and Chang, K.-W. 2019b. [Visualbert: A simple and performant baseline for vision and language](https://md.research.elamur.ai/free/en/papers/li-2019-visualbert-simple-performant-baseline-vision/). arXiv preprint arXiv:1908.03557 .
14. Li, X.; Yin, X.; Li, C.; Hu, X.; Zhang, P.; Zhang, L.; Wang, L.; Hu, H.; Dong, L.; Wei, F.; et al. 2020. [Oscar: Object-semantics aligned pre-training for vision-language tasks](https://md.research.elamur.ai/free/en/papers/li-2020-oscar-object-semantics-aligned-pre/). arXiv preprint arXiv:2004.06165 .
15. Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In European conference on computer vision, 740–755. Springer.
16. Lu, J.; Batra, D.; Parikh, D.; and Lee, S. 2019. [Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks](https://md.research.elamur.ai/free/en/papers/lu-2019-vilbert-pretraining-task-agnostic-visiolinguistic/). In Advances in Neural Information Processing Systems, 13–23.
17. Ordonez, V.; Kulkarni, G.; and Berg, T. L. 2011. Im2text: Describing images using 1 million captioned photographs. In Advances in neural information processing systems, 1143–1151. [find](https://scholar.google.com/scholar?q=Ordonez%2C+V%2E%3B+Kulkarni%2C+G%2E%3B+and+Berg%2C+T%2E+L%2E+2011%2E+Im2text%3A+Describing+images+using+1+million+captioned+photographs%2E+In+Advances+in+neural+information+processing+systems%2C+1143%E2%80%931151%2E)
18. Radford, A.; Narasimhan, K.; Salimans, T.; and Sutskever, I. 2018. Improving language understanding by generative pre-training. In URL [https://s3-us-west-2.amazonaws.com/openaiassets/research-covers/language](https://s3-us-west-2.amazonaws.com/openaiassets/research-covers/language) unsupervised/language understanding paper.pdf. [↗](https://s3-us-west-2.amazonaws.com/openaiassets/research-covers/language)
19. Sharma, P.; Ding, N.; Goodman, S.; and Soricut, R. 2018. [Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2556–2565.
20. Su, W.; Zhu, X.; Cao, Y.; Li, B.; Lu, L.; Wei, F.; and Dai, J. 2019. [Vl-bert: Pre-training of generic visual-linguistic representations](https://md.research.elamur.ai/free/en/papers/su-2019-vl-bert-pre-training-generic/). arXiv preprint arXiv:1908.08530 .
21. Sun, Y.; Cheng, C.; Zhang, Y.; Zhang, C.; Zheng, L.; Wang, Z.; and Wei, Y. 2020. Circle loss: A unified perspective of pair similarity optimization. arXiv preprint arXiv:2002.10857 . — [arXiv:2002.10857](https://arxiv.org/abs/2002.10857) [find](https://scholar.google.com/scholar?q=Sun%2C+Y%2E%3B+Cheng%2C+C%2E%3B+Zhang%2C+Y%2E%3B+Zhang%2C+C%2E%3B+Zheng%2C+L%2E%3B+Wang%2C+Z%2E%3B+and+Wei%2C+Y%2E+2020%2E+Circle+loss%3A+A+unified+perspective+of+pair+similarity+optimization%2E+arXiv+preprint+arXiv%3A2002%2E10857+%2E+%E2%80%94+%5BarXiv%3A2002%2E108)
22. Sun, Y.; Wang, S.; Li, Y.; Feng, S.; Tian, H.; Wu, H.; and Wang, H. 2019. [Ernie 2.0: A continual pre-training framework for language understanding](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-2-0-continual/). arXiv preprint arXiv:1907.12412 .
23. Tan, H.; and Bansal, M. 2019. [Lxmert: Learning cross-modality encoder representations from transformers](https://md.research.elamur.ai/free/en/papers/tan-2019-lxmert-learning-cross-modality-encoder/). arXiv preprint arXiv:1908.07490 .
24. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in neural information processing systems, 5998–6008.
25. Wu, H.; Mao, J.; Zhang, Y.; Jiang, Y.; Li, L.; Sun, W.; and Ma, W.-Y. 2019. Unified visual-semantic embeddings: Bridging vision and language with structured meaning representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6609–6618. [find](https://scholar.google.com/scholar?q=Wu%2C+H%2E%3B+Mao%2C+J%2E%3B+Zhang%2C+Y%2E%3B+Jiang%2C+Y%2E%3B+Li%2C+L%2E%3B+Sun%2C+W%2E%3B+and+Ma%2C+W%2E%2DY%2E+2019%2E+Unified+visual%2Dsemantic+embeddings%3A+Bridging+vision+and+language+with+structured+meaning+representations%2E+In+Proceedings+of+)
26. Yang, X.; Tang, K.; Zhang, H.; and Cai, J. 2019. Autoencoding scene graphs for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 10685–10694. [find](https://scholar.google.com/scholar?q=Yang%2C+X%2E%3B+Tang%2C+K%2E%3B+Zhang%2C+H%2E%3B+and+Cai%2C+J%2E+2019%2E+Autoencoding+scene+graphs+for+image+captioning%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+10685%E2%80%9310694%2E)
27. Young, P.; Lai, A.; Hodosh, M.; and Hockenmaier, J. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2: 67–78. [find](https://scholar.google.com/scholar?q=Young%2C+P%2E%3B+Lai%2C+A%2E%3B+Hodosh%2C+M%2E%3B+and+Hockenmaier%2C+J%2E+2014%2E+From+image+descriptions+to+visual+denotations%3A+New+similarity+metrics+for+semantic+inference+over+event+descriptions%2E+Transactions+of+the+Asso)
28. Yu, L.; Lin, Z.; Shen, X.; Yang, J.; Lu, X.; Bansal, M.; and Berg, T. L. 2018. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1307–1315. [find](https://scholar.google.com/scholar?q=Yu%2C+L%2E%3B+Lin%2C+Z%2E%3B+Shen%2C+X%2E%3B+Yang%2C+J%2E%3B+Lu%2C+X%2E%3B+Bansal%2C+M%2E%3B+and+Berg%2C+T%2E+L%2E+2018%2E+Mattnet%3A+Modular+attention+network+for+referring+expression+comprehension%2E+In+Proceedings+of+the+IEEE+Conference+on+Compu)
29. Zellers, R.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6720–6731. [find](https://scholar.google.com/scholar?q=Zellers%2C+R%2E%3B+Bisk%2C+Y%2E%3B+Farhadi%2C+A%2E%3B+and+Choi%2C+Y%2E+2019%2E+From+recognition+to+cognition%3A+Visual+commonsense+reasoning%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+67)
30. Zhang, C.; Chao, W.-L.; and Xuan, D. 2019. An empirical study on leveraging scene graphs for visual question answering. arXiv preprint arXiv:1907.12133 . — [arXiv:1907.12133](https://arxiv.org/abs/1907.12133) [find](https://scholar.google.com/scholar?q=Zhang%2C+C%2E%3B+Chao%2C+W%2E%2DL%2E%3B+and+Xuan%2C+D%2E+2019%2E+An+empirical+study+on+leveraging+scene+graphs+for+visual+question+answering%2E+arXiv+preprint+arXiv%3A1907%2E12133+%2E+%E2%80%94+%5BarXiv%3A1907%2E12133%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F190)
31. Zhou, L.; Palangi, H.; Zhang, L.; Hu, H.; Corso, J. J.; and Gao, J. 2019. Unified vision-language pre-training for image captioning and vqa. arXiv preprint arXiv:1909.11059 . — [arXiv:1909.11059](https://arxiv.org/abs/1909.11059) [find](https://scholar.google.com/scholar?q=Zhou%2C+L%2E%3B+Palangi%2C+H%2E%3B+Zhang%2C+L%2E%3B+Hu%2C+H%2E%3B+Corso%2C+J%2E+J%2E%3B+and+Gao%2C+J%2E+2019%2E+Unified+vision%2Dlanguage+pre%2Dtraining+for+image+captioning+and+vqa%2E+arXiv+preprint+arXiv%3A1909%2E11059+%2E+%E2%80%94+%5BarXiv%3A1909%2E11059%5D%28http)

*Updated 24 October 2026*