---
title: "Improving Multimodal Named Entity Recognition via Entity Span"
description: "This work focuses on social media posts, tackling multimodal named entity recognition (MNER) in… 37 references, cited by 60 papers here."
authors: ["Jianfei Yu", "Jing Jiang", "Li Yang", "Rui Xia"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.306/"
doi: "10.18653/v1/2020.acl-main.306"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yu-2020-improving-multimodal-named-entity-recognition/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yu-2020-improving-multimodal-named-entity-recognition/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/) / Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer

# Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer

**Authors:** [Jianfei Yu](https://md.research.elamur.ai/free/en/authors/jianfei-yu/), [Jing Jiang](https://md.research.elamur.ai/free/en/authors/jing-jiang/), [Li Yang](https://md.research.elamur.ai/free/en/authors/li-yang/) and [Rui Xia](https://md.research.elamur.ai/free/en/authors/rui-xia/) · **Year:** 2020 · **Venue:** [Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2020](https://md.research.elamur.ai/free/en/venues/acl/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.306/) · [PDF](https://aclanthology.org/2020.acl-main.306.pdf) · [DOI 10.18653/v1/2020.acl-main.306](https://doi.org/10.18653/v1/2020.acl-main.306) · [OpenAlex](https://openalex.org/W3035448883)

**Citations:** 281 citations (OpenAlex, 2026-10-14) · 60 in our library, of the 141,840 papers whose bibliographies we read

## Summary
This work focuses on social media posts, tackling multimodal named entity recognition (MNER) in settings where a short, noisy post is paired with an image and the system must locate entity mentions and assign types such as PER, LOC, ORG or MISC. The authors identify two weaknesses in earlier MNER systems. First, text shaped the visual features, but images never shaped the word representations. Second, a picture usually spotlights only one or two entities, which pulls attention away from the others. Their Unified Multimodal Transformer (UMT) encodes text with BERT and images with ResNet. A multimodal interaction module, combining cross-modal attention with a visual gate, then yields both image-aware word vectors and word-aware visual vectors. To reduce visual bias, a text-only entity span detection task runs in parallel. A conversion matrix links its span predictions to the final CRF-based MNER labels. UMT sets a new state of the art on two benchmarks. It is presented as the first Transformer applied to MNER.

## Key points
- The task is framed as sequence labeling with the BIO2 schema. Each input is a {sentence, image} pair, and separate CRF layers produce the MNER labels and the auxiliary entity span labels.
- For the visual side, images are resized to 224×224 and passed through a pretrained 152-layer ResNet. The last convolutional layer provides 7×7 = 49 regions, each a 2048-dimensional vector, which are then linearly projected into the BERT space.
- UMT outperforms prior state-of-the-art MNER methods with relative improvements of 3.7% and 3.8% on the two benchmark datasets. It also beats strong text-only baselines such as BERT-CRF.
- In error analysis, roughly 5% of test posts have images unrelated to the text, such as memes, cartoons or metaphorical photos. In those cases, multimodal models tend to do worse than BERT-CRF; for example, "Siri" is tagged as PER because of a face in the image.
- As future work, the authors propose filtering noisy images dynamically and using unlabeled social media posts, since current MNER datasets are small.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Singapore Management University](https://md.research.elamur.ai/free/en/universities/singapore-management-university/) — University, Singapore, Singapore · 9 works · 2,130 citations
- [Nanjing University of Science and Technology](https://md.research.elamur.ai/free/en/universities/nanjing-university-of-science-and-technology/) — University, Nanjing, China · 3 works · 1,544 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (2): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 269 works · [🏷️ Named entity recognition](https://md.research.elamur.ai/free/en/topics/named-entity-recognition/) · 101 works

## Cite

Cite as: Jianfei Yu, Jing Jiang, Li Yang and Rui Xia. 2020. Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yu2020improving,
  title = {{Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer}},
  author = {Jianfei Yu, Jing Jiang, Li Yang and Rui Xia},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.306},
  url = {https://doi.org/10.18653/v1/2020.acl-main.306}
}
```

- [← previous RobBERT: a Dutch RoBERTa-based Language Model](https://md.research.elamur.ai/free/en/papers/delobelle-2020-robbert-dutch-roberta-based-language/)
- [next → Information-Theoretic Probing with Minimum Description Length](https://md.research.elamur.ai/free/en/papers/voita-2020-information-theoretic-probing-minimum-description/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/)

## References

37 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. [Layer normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/). arXiv preprint arXiv:1607.06450.
2. Timothy Baldwin, Marie-Catherine de Marneffe, Bo Han, Young-Bum Kim, Alan Ritter, and Wei Xu. 2015. [Shared tasks of the 2015 workshop on noisy user-generated text: Twitter lexical normalization and named entity recognition](https://md.research.elamur.ai/free/en/papers/baldwin-2015-shared-tasks-2015-workshop-noisy/). In Proceedings of the Workshop on Noisy User-generated Text, pages 126– 135.
3. Hai Leong Chieu and Hwee Tou Ng. 2002. Named entity recognition: a maximum entropy approach using global information. In Proceedings of COLING. [find](https://scholar.google.com/scholar?q=Hai+Leong+Chieu+and+Hwee+Tou+Ng%2E+2002%2E+Named+entity+recognition%3A+a+maximum+entropy+approach+using+global+information%2E+In+Proceedings+of+COLING%2E)
4. Jason PC Chiu and Eric Nichols. 2016. [Named entity recognition with bidirectional lstm-cnns](https://md.research.elamur.ai/free/en/papers/chiu-2015-named-entity-recognition-bidirectional-lstm/). Transactions of the Association for Computational Linguistics, 4:357–370.
5. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. [Natural language processing (almost) from scratch](https://md.research.elamur.ai/free/en/papers/collobert-2011-natural-language-processing-almost-scratch/). Journal of Machine Learning Research, 12:2461– 2505.
6. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. [Neural architectures for named entity recognition](https://md.research.elamur.ai/free/en/papers/lample-2016-neural-architectures-named-entity-recognition/). In Proceedings of NAACL-HLT.
7. Chenliang Li, Aixin Sun, Jianshu Weng, and Qi He. 2014. Tweet segmentation and its application to named entity recognition. IEEE Transactions on knowledge and data engineering, 27(2):558–570. [find](https://scholar.google.com/scholar?q=Chenliang+Li%2C+Aixin+Sun%2C+Jianshu+Weng%2C+and+Qi+He%2E+2014%2E+Tweet+segmentation+and+its+application+to+named+entity+recognition%2E+IEEE+Transactions+on+knowledge+and+data+engineering%2C+27%282%29%3A558%E2%80%93570%2E)
8. Chenliang Li, Jianshu Weng, Qi He, Yuxia Yao, Anwitaman Datta, Aixin Sun, and Bu-Sung Lee. 2012. Twiner: named entity recognition in targeted twitter stream. In Proceedings of SIGIR. [find](https://scholar.google.com/scholar?q=Chenliang+Li%2C+Jianshu+Weng%2C+Qi+He%2C+Yuxia+Yao%2C+Anwitaman+Datta%2C+Aixin+Sun%2C+and+Bu%2DSung+Lee%2E+2012%2E+Twiner%3A+named+entity+recognition+in+targeted+twitter+stream%2E+In+Proceedings+of+SIGIR%2E)
9. Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2018. A survey on deep learning for named entity recognition. arXiv preprint arXiv:1812.09449. [find](https://scholar.google.com/scholar?q=Jing+Li%2C+Aixin+Sun%2C+Jianglei+Han%2C+and+Chenliang+Li%2E+2018%2E+A+survey+on+deep+learning+for+named+entity+recognition%2E+arXiv+preprint+arXiv%3A1812%2E09449%2E)
10. Nut Limsopatham and Nigel Collier. 2016. Bidirectional LSTM for named entity recognition in twitter messages. In Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT). [find](https://scholar.google.com/scholar?q=Nut+Limsopatham+and+Nigel+Collier%2E+2016%2E+Bidirectional+LSTM+for+named+entity+recognition+in+twitter+messages%2E+In+Proceedings+of+the+2nd+Workshop+on+Noisy+User%2Dgenerated+Text+%28WNUT%29%2E)
11. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. [find](https://scholar.google.com/scholar?q=Jia+Deng%2C+Wei+Dong%2C+Richard+Socher%2C+Li%2DJia+Li%2C+Kai+Li%2C+and+Li+Fei%2DFei%2E+2009%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+2009+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pa)
12. Bill Yuchen Lin, Frank F Xu, Zhiyi Luo, and Kenny Zhu. 2017. Multi-channel bilstm-crf model for emerging named entity recognition in social media. In Proceedings of the 3rd Workshop on Noisy User-generated Text. [find](https://scholar.google.com/scholar?q=Bill+Yuchen+Lin%2C+Frank+F+Xu%2C+Zhiyi+Luo%2C+and+Kenny+Zhu%2E+2017%2E+Multi%2Dchannel+bilstm%2Dcrf+model+for+emerging+named+entity+recognition+in+social+media%2E+In+Proceedings+of+the+3rd+Workshop+on+Noisy+User%2Dgene)
13. Leon Derczynski, Diana Maynard, Giuseppe Rizzo, Marieke Van Erp, Genevieve Gorrell, Raphaël Troncy, Johann Petrak, and Kalina Bontcheva. 2015. Analysis of named entity recognition and linking for tweets. Information Processing & Management, 51(2):32–49. [find](https://scholar.google.com/scholar?q=Leon+Derczynski%2C+Diana+Maynard%2C+Giuseppe+Rizzo%2C+Marieke+Van+Erp%2C+Genevieve+Gorrell%2C+Raphae%CC%88l+Troncy%2C+Johann+Petrak%2C+and+Kalina+Bontcheva%2E+2015%2E+Analysis+of+named+entity+recognition+and+linking+for+twe)
14. Dekang Lin and Xiaoyun Wu. 2009. [Phrase clustering for discriminative learning](https://md.research.elamur.ai/free/en/papers/lin-2009-phrase-clustering-discriminative-learning/). In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP.
15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805.
16. Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. [Incorporating non-local information into information extraction systems by gibbs sampling](https://md.research.elamur.ai/free/en/papers/finkel-2005-incorporating-non-local-information-into/). In Proceedings of ACL.
17. Radu Florian, Abe Ittycheriah, Hongyan Jing, and Tong Zhang. 2003. [Named entity recognition through classifier combination](https://md.research.elamur.ai/free/en/papers/florian-2003-named-entity-recognition-through-classifier/). In Proceedings of NAACL.
18. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of CVPR, pages 770–778. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+2016%2E+Deep+residual+learning+for+image+recognition%2E+In+Proceedings+of+CVPR%2C+pages+770%E2%80%93778%2E)
19. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. [Bidirectional lstm-crf models for sequence tagging](https://md.research.elamur.ai/free/en/papers/huang-2015-bidirectional-lstm-crf-models-sequence/). arXiv preprint arXiv:1508.01991.
20. John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of ICML. [find](https://scholar.google.com/scholar?q=John+Lafferty%2C+Andrew+McCallum%2C+and+Fernando+CN+Pereira%2E+2001%2E+Conditional+random+fields%3A+Probabilistic+models+for+segmenting+and+labeling+sequence+data%2E+In+Proceedings+of+ICML%2E)
21. Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. 2018. [Visual attention model for name tagging in multimodal social media](https://md.research.elamur.ai/free/en/papers/lu-2018-visual-attention-model-name-tagging/). In Proceedings of ACL, pages 1990–1999.
22. Gang Luo, Xiaojiang Huang, Chin-Yew Lin, and Zaiqing Nie. 2015. Joint entity recognition and disambiguation. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Gang+Luo%2C+Xiaojiang+Huang%2C+Chin%2DYew+Lin%2C+and+Zaiqing+Nie%2E+2015%2E+Joint+entity+recognition+and+disambiguation%2E+In+Proceedings+of+EMNLP%2E)
23. Xuezhe Ma and Eduard Hovy. 2016. [End-to-end sequence labeling via bi-directional lstm-cnns-crf](https://md.research.elamur.ai/free/en/papers/ma-2016-end-end-sequence-labeling-bi/). In Proceedings of ACL.
24. Diana Maynard, Kalina Bontcheva, and Dominic Rout. 2012. Challenges in developing opinion mining tools for social media. In Proceedings of the@ NLP can u tag usergeneratedcontent. [find](https://scholar.google.com/scholar?q=Diana+Maynard%2C+Kalina+Bontcheva%2C+and+Dominic+Rout%2E+2012%2E+Challenges+in+developing+opinion+mining+tools+for+social+media%2E+In+Proceedings+of+the%40+NLP+can+u+tag+usergeneratedcontent%2E)
25. Seungwhan Moon, Leonardo Neves, and Vitor Carvalho. 2018. Multimodal named entity recognition for short social media posts. In Proceedings of NAACL. [find](https://scholar.google.com/scholar?q=Seungwhan+Moon%2C+Leonardo+Neves%2C+and+Vitor+Carvalho%2E+2018%2E+Multimodal+named+entity+recognition+for+short+social+media+posts%2E+In+Proceedings+of+NAACL%2E)
26. Alexandre Passos, Vineet Kumar, and Andrew McCallum. 2014. [Lexicon infused phrase embeddings for named entity resolution](https://md.research.elamur.ai/free/en/papers/passos-2014-lexicon-infused-phrase-embeddings-named/). In Proceedings of CoNLL.
27. Lev Ratinov and Dan Roth. 2009. [Design challenges and misconceptions in named entity recognition](https://md.research.elamur.ai/free/en/papers/ratinov-2009-design-challenges-misconceptions-named-entity/). In Proceedings of CoNLL.
28. Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. 2011. [Named entity recognition in tweets: an experimental study](https://md.research.elamur.ai/free/en/papers/ritter-2011-named-entity-recognition-tweets-experimental/). In Proceedings of ACL.
29. Alan Ritter, Oren Etzioni, and Sam Clark. 2012. Open domain event extraction from twitter. In Proceedings of SIGKDD. [find](https://scholar.google.com/scholar?q=Alan+Ritter%2C+Oren+Etzioni%2C+and+Sam+Clark%2E+2012%2E+Open+domain+event+extraction+from+twitter%2E+In+Proceedings+of+SIGKDD%2E)
30. Erik F Sang and Jorn Veenstra. 1999. [Representing text chunks](https://md.research.elamur.ai/free/en/papers/sang-1999-representing-text-chunks/). In Proceedings of EACL.
31. Kentaro Torisawa et al. 2007. Exploiting wikipedia as external knowledge for named entity recognition. In Proceedings of EMNLP-CoNLL. [find](https://scholar.google.com/scholar?q=Kentaro+Torisawa+et+al%2E+2007%2E+Exploiting+wikipedia+as+external+knowledge+for+named+entity+recognition%2E+In+Proceedings+of+EMNLP%2DCoNLL%2E)
32. Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. [Multimodal transformer for unaligned multimodal language sequences](https://md.research.elamur.ai/free/en/papers/tsai-2019-multimodal-transformer-unaligned-multimodal-language/). In Proceedings of ACL.
33. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Proceedings of NIPS, pages 5998– 6008.
34. Vikas Yadav and Steven Bethard. 2018. [A survey on recent advances in named entity recognition from deep learning models](https://md.research.elamur.ai/free/en/papers/yadav-2018-survey-recent-advances-named-entity/). In Proceedings of COLING.
35. Jie Yang, Shuailong Liang, and Yue Zhang. 2018. [Design challenges and misconceptions in neural sequence labeling](https://md.research.elamur.ai/free/en/papers/yang-2018-design-challenges-misconceptions-neural-sequence/). In Proceedings of COLING.
36. Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018. Adaptive co-attention network for named entity recognition in tweets. In Proceedings of AAAI, pages 5674–5681. [find](https://scholar.google.com/scholar?q=Qi+Zhang%2C+Jinlan+Fu%2C+Xiaoyu+Liu%2C+and+Xuanjing+Huang%2E+2018%2E+Adaptive+co%2Dattention+network+for+named+entity+recognition+in+tweets%2E+In+Proceedings+of+AAAI%2C+pages+5674%E2%80%935681%2E)
37. GuoDong Zhou and Jian Su. 2002. [Named entity recognition using an hmm-based chunk tagger](https://md.research.elamur.ai/free/en/papers/zhou-2002-named-entity-recognition-hmm-based/). In Proceedings of ACL.

*Updated 24 October 2026*