---
title: "UNIMO — paper by Li et al. (2021) — summary and ideas"
description: "The authors (Baidu) propose UNIMO, a single Transformer pre-trained jointly on three sources: large… 53 references, cited by 68 papers here."
authors: ["Wei Li", "Can Gao", "Guocheng Niu", "Xinyan Xiao", "Hao Liu", "Jiachen Liu", "Hua Wu", "Haifeng Wang"]
year: 2021
venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)"
source: "https://aclanthology.org/2021.acl-long.202/"
doi: "10.18653/v1/2021.acl-long.202"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2021-unimo-unified-modal-understanding-generation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2021-unimo-unified-modal-understanding-generation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/) / UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning

# UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning

**Authors:** [Wei Li](https://md.research.elamur.ai/free/en/authors/wei-li/), [Can Gao](https://md.research.elamur.ai/free/en/authors/can-gao/), Guocheng Niu, [Xinyan Xiao](https://md.research.elamur.ai/free/en/authors/xinyan-xiao/), [Hao Liu](https://md.research.elamur.ai/free/en/authors/hao-liu/), Jiachen Liu, [Hua Wu](https://md.research.elamur.ai/free/en/authors/hua-wu/) and [Haifeng Wang](https://md.research.elamur.ai/free/en/authors/haifeng-wang/) · **Year:** 2021 · **Venue:** [Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2021](https://md.research.elamur.ai/free/en/venues/acl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.acl-long.202/) · [PDF](https://aclanthology.org/2021.acl-long.202.pdf) · [DOI 10.18653/v1/2021.acl-long.202](https://doi.org/10.18653/v1/2021.acl-long.202) · [OpenAlex](https://openalex.org/W3177224328)

**Citations:** 231 citations (OpenAlex, 2026-10-14) · [68](https://md.research.elamur.ai/free/en/papers/li-2021-unimo-unified-modal-understanding-generation/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Existing pre-trained models usually serve only one side. Text-only and vision-only models such as BERT or ResNet cannot handle vision-language inputs. Multi-modal models such as ViLBERT and UNITER learn only from scarce image-text pairs and perform poorly when moved to pure language tasks. The authors (Baidu) propose UNIMO, a single Transformer pre-trained jointly on three sources:

- large non-paired text corpora
- image collections
- image-text pairs

Each modality is learned through masked prediction. Cross-modal contrastive learning (CMCL) then pulls images and texts into a shared semantic space. To make the alignment finer-grained, captions are rewritten to produce extra positive pairs and hard negative pairs. Retrieval also attaches related standalone images and texts to each pair as additional positives.

According to the authors, UNIMO outperforms earlier methods at both understanding and generation, whether the task involves a single modality or several. Ablations indicate that text and visual knowledge strengthen each other. The work matters because one model can be fine-tuned for language-only and vision-language applications alike.

## Key points
- **Input encoding:**
  - Text is tokenized with BPE.
  - Images become region features from Faster R-CNN (pooled ROI features).
  - For a pair, both sequences are concatenated, and the [IMG] and [CLS] outputs serve as the image and text representations.
- **CMCL training signal:** It uses text rewriting to build positive and hard negative image-text pairs, plus text/image retrieval to add weakly related single-modal samples. All of these feed one contrastive loss with a temperature parameter.
- **Reported tasks:** The authors claim the best results among compared methods on image-text retrieval, visual entailment, VQA and image captioning. They also report strong results on language understanding and generation.
- **Ablation:** Removing text-corpus learning ("w/o texts") consistently lowers multi-modal understanding and generation scores. This suggests textual knowledge improves vision-language learning.
- **Availability:** Code and pre-trained models are released in the PaddlePaddle Research GitHub repository (NLP/UNIMO).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Baidu](https://md.research.elamur.ai/free/en/companies/baidu/) — Company, Beijing, China · 40 works · 14,627 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 269 works

## Cite

Cite as: Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu and Haifeng Wang. 2021. UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{li2021unimo,
  title = {{UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning}},
  author = {Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu and Haifeng Wang},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.202},
  url = {https://doi.org/10.18653/v1/2021.acl-long.202}
}
```

- [← previous Muppet: Massive Multi-task Representations with Pre-Finetuning](https://md.research.elamur.ai/free/en/papers/aghajanyan-2021-muppet-massive-multi-task-representations/)
- [next → Want To Reduce Labeling Cost? GPT-3 Can Help](https://md.research.elamur.ai/free/en/papers/wang-2021-want-reduce-labeling-cost-gpt/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/)

## References

53 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. [SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation](https://md.research.elamur.ai/free/en/papers/cer-2017-semeval-2017-task-1-semantic/). In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada.…
2. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020a. [A simple framework for contrastive learning of visual representations](https://md.research.elamur.ai/free/en/papers/chen-2020-simple-framework-contrastive-learning-visual/). In International conference on machine learning, pages 1597–1607. PMLR.
3. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015. [Microsoft coco captions: Data collection and evaluation server](https://md.research.elamur.ai/free/en/papers/chen-2015-microsoft-coco-captions-data-collection/). arXiv preprint arXiv:1504.00325.
4. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020b. [Uniter: Universal image-text representation learning](https://md.research.elamur.ai/free/en/papers/chen-2019-uniter-universal-image-text-representation/). In European Conference on Computer Vision, pages 104–120. Springer.
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
6. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. [Unified language model pre-training for natural language understanding and generation](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/). In Advances in Neural Information Processing Systems, pages 13063–13075.
7. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489–500, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sergey+Edunov%2C+Myle+Ott%2C+Michael+Auli%2C+and+David+Grangier%2E+2018%2E+Understanding+back%2Dtranslation+at+scale%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pag)
8. Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. 2020. Large-scale adversarial training for vision-and-language representation learning. arXiv preprint arXiv:2006.06195. [find](https://scholar.google.com/scholar?q=Zhe+Gan%2C+Yen%2DChun+Chen%2C+Linjie+Li%2C+Chen+Zhu%2C+Yu+Cheng%2C+and+Jingjing+Liu%2E+2020%2E+Large%2Dscale+adversarial+training+for+vision%2Dand%2Dlanguage+representation+learning%2E+arXiv+preprint+arXiv%3A2006%2E06195%2E)
9. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. [Making the v in vqa matter: Elevating the role of image understanding in visual question answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904–6913.
10. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. [Momentum contrast for unsupervised visual representation learning](https://md.research.elamur.ai/free/en/papers/he-2019-momentum-contrast-unsupervised-visual-representation/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738.
11. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+2016%2E+Deep+residual+learning+for+image+recognition%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pages+770%E2%80%93+)
12. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). Advances in neural information processing systems, 28:1693–1701.
13. Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. 2021. Seeing out of the box: End-to-end pre-training for vision-language representation learning. arXiv preprint arXiv:2104.03135. — [arXiv:2104.03135](https://arxiv.org/abs/2104.03135) [find](https://scholar.google.com/scholar?q=Zhicheng+Huang%2C+Zhaoyang+Zeng%2C+Yupan+Huang%2C+Bei+Liu%2C+Dongmei+Fu%2C+and+Jianlong+Fu%2E+2021%2E+Seeing+out+of+the+box%3A+End%2Dto%2Dend+pre%2Dtraining+for+vision%2Dlanguage+representation+learning%2E+arXiv+preprint+arXiv)
14. Yuqi Huo, Manli Zhang, Guangzhen Liu, Haoyu Lu, Yizhao Gao, Guoxing Yang, Jingyuan Wen, Heng Zhang, Baogui Xu, Weihao Zheng, et al. 2021. Wenlan: Bridging vision and language by large-scale multi-modal pre-training. arXiv preprint arXiv:2103.06561. [find](https://scholar.google.com/scholar?q=Yuqi+Huo%2C+Manli+Zhang%2C+Guangzhen+Liu%2C+Haoyu+Lu%2C+Yizhao+Gao%2C+Guoxing+Yang%2C+Jingyuan+Wen%2C+Heng+Zhang%2C+Baogui+Xu%2C+Weihao+Zheng%2C+et+al%2E+2021%2E+Wenlan%3A+Bridging+vision+and+language+by+large%2Dscale+multi%2Dmoda)
15. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. [SpanBERT: Improving pre-training by representing and predicting spans](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/). Transactions of the Association for Computational Linguistics, 8:64–77.
16. Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, et al. 2017. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://github. co… [find](https://scholar.google.com/scholar?q=Ivan+Krasin%2C+Tom+Duerig%2C+Neil+Alldrin%2C+Vittorio+Ferrari%2C+Sami+Abu%2DEl%2DHaija%2C+Alina+Kuznetsova%2C+Hassan+Rom%2C+Jasper+Uijlings%2C+Stefan+Popov%2C+Andreas+Veit%2C+et+al%2E+2017%2E+Openimages%3A+A+public+dataset+for+lar)
17. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. [Visual genome: Connecting language and vision using crowdsourced dense image annotations](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/). International journal of computer vision, 123(1):32…
18. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2017. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84–90. [find](https://scholar.google.com/scholar?q=Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Geoffrey+E+Hinton%2E+2017%2E+Imagenet+classification+with+deep+convolutional+neural+networks%2E+Communications+of+the+ACM%2C+60%286%29%3A84%E2%80%9390%2E)
19. Alon Lavie and Abhaya Agarwal. 2007. [Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments](https://md.research.elamur.ai/free/en/papers/lavie-2007-meteor-automatic-metric-mt-evaluation/). In Proceedings of the second workshop on statistical machine translation, pages 228–231.
20. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the…
21. Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou. 2019a. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. arXiv preprint arXiv:1908.06066. — [arXiv:1908.06066](https://arxiv.org/abs/1908.06066) [find](https://scholar.google.com/scholar?q=Gen+Li%2C+Nan+Duan%2C+Yuejian+Fang%2C+Daxin+Jiang%2C+and+Ming+Zhou%2E+2019a%2E+Unicoder%2Dvl%3A+A+universal+encoder+for+vision+and+language+by+cross%2Dmodal+pre%2Dtraining%2E+arXiv+preprint+arXiv%3A1908%2E06066%2E+%E2%80%94+%5BarXiv%3A1908%2E)
22. Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019b. [Visualbert: A simple and performant baseline for vision and language](https://md.research.elamur.ai/free/en/papers/li-2019-visualbert-simple-performant-baseline-vision/). arXiv preprint arXiv:1908.03557.
23. Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020. [Oscar: Object-semantics aligned pre-training for vision-language tasks](https://md.research.elamur.ai/free/en/papers/li-2020-oscar-object-semantics-aligned-pre/). In European Conference on Computer Vision, pages 121–137. Springer.
24. Huan Lin, Fandong Meng, Jinsong Su, Yongjing Yin, Zhengyuan Yang, Yubin Ge, Jie Zhou, and Jiebo Luo. 2020a. Dynamic context-guided capsule network for multimodal machine translation. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1320–1329. [find](https://scholar.google.com/scholar?q=Huan+Lin%2C+Fandong+Meng%2C+Jinsong+Su%2C+Yongjing+Yin%2C+Zhengyuan+Yang%2C+Yubin+Ge%2C+Jie+Zhou%2C+and+Jiebo+Luo%2E+2020a%2E+Dynamic+context%2Dguided+capsule+network+for+multimodal+machine+translation%2E+In+Proceedings+of)
25. Junyang Lin, An Yang, Yichang Zhang, Jie Liu, Jingren Zhou, and Hongxia Yang. 2020b. Interbert: Vision-and-language interaction for multi-modal pretraining. arXiv preprint arXiv:2003.13198. — [arXiv:2003.13198](https://arxiv.org/abs/2003.13198) [find](https://scholar.google.com/scholar?q=Junyang+Lin%2C+An+Yang%2C+Yichang+Zhang%2C+Jie+Liu%2C+Jingren+Zhou%2C+and+Hongxia+Yang%2E+2020b%2E+Interbert%3A+Vision%2Dand%2Dlanguage+interaction+for+multi%2Dmodal+pretraining%2E+arXiv+preprint+arXiv%3A2003%2E13198%2E+%E2%80%94+%5BarXiv%3A2)
26. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In European conference on computer vision, pages 740–755. Springer.
27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
28. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. [Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks](https://md.research.elamur.ai/free/en/papers/lu-2019-vilbert-pretraining-task-agnostic-visiolinguistic/). In Advances in Neural Information Processing Systems, pages 13–23.
29. Vishvak Murahari, Dhruv Batra, Devi Parikh, and Abhishek Das. 2020. Large-scale pretraining for visual dialog: A simple state-of-the-art baseline. In European Conference on Computer Vision, pages 336– 352. Springer. [find](https://scholar.google.com/scholar?q=Vishvak+Murahari%2C+Dhruv+Batra%2C+Devi+Parikh%2C+and+Abhishek+Das%2E+2020%2E+Large%2Dscale+pretraining+for+visual+dialog%3A+A+simple+state%2Dof%2Dthe%2Dart+baseline%2E+In+European+Conference+on+Computer+Vision%2C+pages+336%E2%80%93)
30. Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 2011. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems, 24:1143–1151. [find](https://scholar.google.com/scholar?q=Vicente+Ordonez%2C+Girish+Kulkarni%2C+and+Tamara+Berg%2E+2011%2E+Im2text%3A+Describing+images+using+1+million+captioned+photographs%2E+Advances+in+neural+information+processing+systems%2C+24%3A1143%E2%80%931151%2E)
31. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). arXiv preprint arXiv:2103.00020.
32. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
33. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
34. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. [CoQA: A conversational question answering challenge](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/). Transactions of the Association for Computational Linguistics, 7:249–266.
35. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2016. [Faster r-cnn: Towards real-time object detection with region proposal networks](https://md.research.elamur.ai/free/en/papers/ren-2015-faster-r-cnn-towards-real/). IEEE transactions on pattern analysis and machine intelligence, 39(6):1137–1149.
36. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389, Lisbon, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alexander+M%2E+Rush%2C+Sumit+Chopra%2C+and+Jason+Weston%2E+2015%2E+A+neural+attention+model+for+abstractive+sentence+summarization%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Language)
37. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li…
38. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. [Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565,…
39. Karen Simonyan and Andrew Zisserman. 2014. [Very deep convolutional networks for large-scale image recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/). arXiv preprint arXiv:1409.1556.
40. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–… [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D%2E+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In)
41. Jinsong Su, Jinchang Chen, Hui Jiang, Chulun Zhou, Huan Lin, Yubin Ge, Qingqiang Wu, and Yongxuan Lai. 2021. Multi-modal neural machine translation with deep semantic interactions. Information Sciences, 554:47–60. [find](https://scholar.google.com/scholar?q=Jinsong+Su%2C+Jinchang+Chen%2C+Hui+Jiang%2C+Chulun+Zhou%2C+Huan+Lin%2C+Yubin+Ge%2C+Qingqiang+Wu%2C+and+Yongxuan+Lai%2E+2021%2E+Multi%2Dmodal+neural+machine+translation+with+deep+semantic+interactions%2E+Information+Science)
42. Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019. [Vl-bert: Pre-training of generic visual-linguistic representations](https://md.research.elamur.ai/free/en/papers/su-2019-vl-bert-pre-training-generic/). arXiv preprint arXiv:1908.08530.
43. Markus Johannes Van Ackeren, Francesca M Barbero, Stefania Mattioni, Roberto Bottini, and Olivier Collignon. 2018. Neuronal populations in the occipital cortex of the blind synchronize to the temporal dynamics of speech. ELife, 7:e31640. [find](https://scholar.google.com/scholar?q=Markus+Johannes+Van+Ackeren%2C+Francesca+M+Barbero%2C+Stefania+Mattioni%2C+Roberto+Bottini%2C+and+Olivier+Collignon%2E+2018%2E+Neuronal+populations+in+the+occipital+cortex+of+the+blind+synchronize+to+the+temporal)
44. Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2020. Unified vision-language pre-training for image captioning and vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13041– 13049. [find](https://scholar.google.com/scholar?q=Luowei+Zhou%2C+Hamid+Palangi%2C+Lei+Zhang%2C+Houdong+Hu%2C+Jason+Corso%2C+and+Jianfeng+Gao%2E+2020%2E+Unified+vision%2Dlanguage+pre%2Dtraining+for+image+captioning+and+vqa%2E+In+Proceedings+of+the+AAAI+Conference+on+Arti)
45. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. [Aligning books and movies: Towards story-like visual explanations by watching movies and reading books](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/). In Proceedings of the IEEE international conference on computer vision, pages 19–…
46. Yu-Siang Wang, Chenxi Liu, Xiaohui Zeng, and Alan Yuille. 2018. Scene graph parsing as dependency parsing. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 397–407, New Orl… [find](https://scholar.google.com/scholar?q=Yu%2DSiang+Wang%2C+Chenxi+Liu%2C+Xiaohui+Zeng%2C+and+Alan+Yuille%2E+2018%2E+Scene+graph+parsing+as+dependency+parsing%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Association+for+Co)
47. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Cola: The corpus of linguistic acceptability (with added annotations). [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R+Bowman%2E+2019%2E+Cola%3A+The+corpus+of+linguistic+acceptability+%28with+added+annotations%29%2E)
48. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+R+Bowman%2E+2017%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+arXiv+preprint+arXiv%3A1704%2E05426%2E)
49. Dongling Xiao, Han Zhang, Yukun Li, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2020. [Ernie-gen: An enhanced multi-flow pre-training and fine-tuning framework for natural language generation](https://md.research.elamur.ai/free/en/papers/xiao-2020-ernie-gen-enhanced-multi/). arXiv preprint arXiv:2001.11314.
50. Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. 2019. Visual entailment: A novel task for fine-grained image understanding. arXiv preprint arXiv:1901.06706. — [arXiv:1901.06706](https://arxiv.org/abs/1901.06706) [find](https://scholar.google.com/scholar?q=Ning+Xie%2C+Farley+Lai%2C+Derek+Doran%2C+and+Asim+Kadav%2E+2019%2E+Visual+entailment%3A+A+novel+task+for+fine%2Dgrained+image+understanding%2E+arXiv+preprint+arXiv%3A1901%2E06706%2E+%E2%80%94+%5BarXiv%3A1901%2E06706%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
51. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). In Advances in neural information processing systems, pages 5753–5763.
52. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78. [find](https://scholar.google.com/scholar?q=Peter+Young%2C+Alice+Lai%2C+Micah+Hodosh%2C+and+Julia+Hockenmaier%2E+2014%2E+From+image+descriptions+to+visual+denotations%3A+New+similarity+metrics+for+semantic+inference+over+event+descriptions%2E+Transactions+of)
53. Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2020. [Ernie-vil: Knowledge enhanced vision-language representations through scene graph](https://md.research.elamur.ai/free/en/papers/yu-2020-ernie-vil-knowledge-enhanced/). arXiv preprint arXiv:2006.16934.

*Updated 24 October 2026*