---
title: "Reasoning with Multimodal Sarcastic Tweets via — Xu 2020"
description: "Sarcasm in social media posts often appears only when a tweet's text is read against its attached picture."
authors: ["Nan Xu", "Zhixiong Zeng", "Wenji Mao"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.349/"
doi: "10.18653/v1/2020.acl-main.349"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2020-reasoning-multimodal-sarcastic-tweets-modeling/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2020-reasoning-multimodal-sarcastic-tweets-modeling/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) / Reasoning with Multimodal Sarcastic Tweets via Modeling Cross-Modality Contrast and Semantic Association

# Reasoning with Multimodal Sarcastic Tweets via Modeling Cross-Modality Contrast and Semantic Association

**Authors:** [Nan Xu](https://md.research.elamur.ai/free/en/authors/nan-xu/), Zhixiong Zeng, [Wenji Mao](https://md.research.elamur.ai/free/en/authors/wenji-mao/) · **Year:** 2020 · **Venue:** Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.349/) · [PDF](https://aclanthology.org/2020.acl-main.349.pdf) · [DOI 10.18653/v1/2020.acl-main.349](https://doi.org/10.18653/v1/2020.acl-main.349) · [OpenAlex](https://openalex.org/W3035565303)

**Citations:** 172 citations (openalex-snapshot · 2026-10-14) · [34](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Sarcasm in social media posts often appears only when a tweet's text is read against its attached picture. For example, a caption praising "perfect" weather may sit next to a photo of pouring rain. Text-only detectors miss such cases. Earlier multimodal systems mainly fused image and text features and did not reason about incongruity between them.

The authors propose D&R Net (Decomposition and Relation Network):
- **D-Net** projects image and text features into a common subspace and into separate modality-specific subspaces. This implicitly captures where the two modalities agree and where they clash.
- **R-Net** uses an attention mechanism between tweet words and adjective-noun pairs (ANPs) detected in the image. It weights ANPs by their detection confidence and explicitly models semantic links across modalities.

On a publicly available multimodal tweet dataset, the model reports state-of-the-art results (F1 80.60, accuracy 84.02). Its attention weights also provide word-level evidence for each decision. The study shows that modelling contrast between modalities, rather than simple fusion, is central to multimodal sarcasm detection.

## Key points
- **Pipeline:**
  - Top-K ANP extraction from each image.
  - Encoding: BiLSTM for text, pre-trained ResNet for images, and max-pooled adjective/noun embeddings for ANPs.
  - D-Net with shared, image and text layers.
  - R-Net with ANP-aware cross-modality attention.
  - A final classifier.
- **Full model:** F1 80.60 and accuracy 84.02.
- **Ablations:**
  - Removing D-Net causes the largest drop, to F1 77.63 (−2.97) and accuracy 82.27.
  - Removing R-Net gives F1 79.90 and accuracy 83.10.
- **ANPs vs. noun-only attributes:** Replacing ANPs with the noun-only attributes of Cai et al. (2019) lowers F1 to 79.52.
- **ANP fusion:** Probability-weighted ANP fusion outperforms MaxPool (79.80) and AvgPool (79.86).
- **Case studies:** Attention visualisations show the model linking text such as "eclipse" or "lovely books" with contradicting visual ANPs, e.g. "rainy clouds" or "bad sign".

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

## Cited by

Cited by

34 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 29 not yet.

- [CofiPara: A Coarse-to-fine Paradigm for Multimodal Sarcasm Target Identification with Large Multimodal Models](https://md.research.elamur.ai/free/en/papers/chen-2024-cofipara-coarse-fine-paradigm-multimodal/)
- [When did you become so smart, oh wise one?! Sarcasm Explanation in Multi-modal Multi-party Dialogues](https://md.research.elamur.ai/free/en/papers/kumar-2022-when-did-you-become-so/)
- [Multi-Modal Sarcasm Detection via Cross-Modal Graph Convolutional Network](https://md.research.elamur.ai/free/en/papers/liang-2022-multi-modal-sarcasm-detection-cross/)
- [Towards Multi-Modal Sarcasm Detection via Hierarchical Congruity Modeling with Knowledge Enhancement](https://md.research.elamur.ai/free/en/papers/liu-2022-multi-modal-sarcasm-detection-hierarchical/)
- [MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection System](https://md.research.elamur.ai/free/en/papers/qin-2023-mmsd2-0-reliable-multi-modal/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xu2020reasoning,
  title = {{Reasoning with Multimodal Sarcastic Tweets via Modeling Cross-Modality Contrast and Semantic Association}},
  author = {Xu, Nan and Zeng, Zhixiong and Mao, Wenji},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.349},
  url = {https://doi.org/10.18653/v1/2020.acl-main.349}
}
```

- [← previous Reasoning with Latent Structure Refinement for Document-Level Relation Extraction](https://md.research.elamur.ai/free/en/papers/nan-2020-reasoning-latent-structure-refinement-document/)
- [next → ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning](https://md.research.elamur.ai/free/en/papers/yu-2020-reclor-reading-comprehension-dataset-requiring/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/)

## References

27 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Silvio Amir, Byron C Wallace, Hao Lyu, Paula Carvalho, and Mario J Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. In Proceedings of CoNLL, pages 167–177. [find](https://scholar.google.com/scholar?q=Silvio+Amir%2C+Byron+C+Wallace%2C+Hao+Lyu%2C+Paula+Carvalho%2C+and+Mario+J+Silva%2E+2016%2E+Modelling+context+with+user+embeddings+for+sarcasm+detection+in+social+media%2E+In+Proceedings+of+CoNLL%2C+pages+167%E2%80%93177%2E) [our card](https://md.research.elamur.ai/free/en/papers/amir-2016-modelling-context-user-embeddings-sarcasm/)
2. David Bamman and Noah A Smith. 2015. Contextualized sarcasm detection on twitter. In Proceedings of ICWSM, pages 574–577. [find](https://scholar.google.com/scholar?q=David+Bamman+and+Noah+A+Smith%2E+2015%2E+Contextualized+sarcasm+detection+on+twitter%2E+In+Proceedings+of+ICWSM%2C+pages+574%E2%80%93577%2E)
3. Christos Baziotis, Athanasiou Nikolaos, Pinelopi Papalampidi, Athanasia Kolovou, Georgios Paraskevopoulos, Nikolaos Ellinas, and Alexandros Potamianos. 2018. Ntua-slp at semeval-2018 task 3: Tracking ironic tweets using ensembles of word and character level attentive rnns. In Proceedings of SemEval, pages 613–621. [find](https://scholar.google.com/scholar?q=Christos+Baziotis%2C+Athanasiou+Nikolaos%2C+Pinelopi+Papalampidi%2C+Athanasia+Kolovou%2C+Georgios+Paraskevopoulos%2C+Nikolaos+Ellinas%2C+and+Alexandros+Potamianos%2E+2018%2E+Ntua%2Dslp+at+semeval%2D2018+task+3%3A+Tracking+)
4. Yitao Cai, Huiyu Cai, and Xiaojun Wan. 2019. Multimodal sarcasm detection in twitter with hierarchical fusion model. In Proceedings of ACL, pages 2506– 2515. [find](https://scholar.google.com/scholar?q=Yitao+Cai%2C+Huiyu+Cai%2C+and+Xiaojun+Wan%2E+2019%2E+Multimodal+sarcasm+detection+in+twitter+with+hierarchical+fusion+model%2E+In+Proceedings+of+ACL%2C+pages+2506%E2%80%93+2515%2E) [our card](https://md.research.elamur.ai/free/en/papers/cai-2019-multi-modal-sarcasm-detection-twitter/)
5. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of ACL-IJCNLP, pages 757–762. [find](https://scholar.google.com/scholar?q=Aditya+Joshi%2C+Vinita+Sharma%2C+and+Pushpak+Bhattacharyya%2E+2015%2E+Harnessing+context+incongruity+for+sarcasm+detection%2E+In+Proceedings+of+ACL%2DIJCNLP%2C+pages+757%E2%80%93762%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2015-harnessing-context-incongruity-sarcasm-detection/)
6. Aditya Joshi, Vaibhav Tripathi, Kevin Patel, Pushpak Bhattacharyya, and Mark Carman. 2016. Are word embedding-based features useful for sarcasm detection? In Proceedings of EMNLP, pages 1006–1011. [find](https://scholar.google.com/scholar?q=Aditya+Joshi%2C+Vaibhav+Tripathi%2C+Kevin+Patel%2C+Pushpak+Bhattacharyya%2C+and+Mark+Carman%2E+2016%2E+Are+word+embedding%2Dbased+features+useful+for+sarcasm+detection%3F+In+Proceedings+of+EMNLP%2C+pages+1006%E2%80%931011%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2016-are-word-embedding-based-features/)
7. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of EMNLP, pages 1746–1751. [find](https://scholar.google.com/scholar?q=Yoon+Kim%2E+2014%2E+Convolutional+neural+networks+for+sentence+classification%2E+In+Proceedings+of+EMNLP%2C+pages+1746%E2%80%931751%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/)
8. Abhijit Mishra, Kuntal Dey, and Pushpak Bhattacharyya. 2017. Learning cognitive features from gaze data for sentiment and sarcasm classification using convolutional neural network. In Proceedings of ACL, pages 377–387. [find](https://scholar.google.com/scholar?q=Abhijit+Mishra%2C+Kuntal+Dey%2C+and+Pushpak+Bhattacharyya%2E+2017%2E+Learning+cognitive+features+from+gaze+data+for+sentiment+and+sarcasm+classification+using+convolutional+neural+network%2E+In+Proceedings+of+A) [our card](https://md.research.elamur.ai/free/en/papers/mishra-2017-learning-cognitive-features-gaze-data/)
9. Lotem Peled and Roi Reichart. 2017. Sarcasm sign: Interpreting sarcasm with sentiment based monolingual machine translation. In Proceedings of ACL, pages 1690–1700. [find](https://scholar.google.com/scholar?q=Lotem+Peled+and+Roi+Reichart%2E+2017%2E+Sarcasm+sign%3A+Interpreting+sarcasm+with+sentiment+based+monolingual+machine+translation%2E+In+Proceedings+of+ACL%2C+pages+1690%E2%80%931700%2E) [our card](https://md.research.elamur.ai/free/en/papers/peled-2017-sarcasm-sign-interpreting-sarcasm-sentiment/)
10. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards multimodal sarcasm detection (an obviously perfect paper). In Proceedings of ACL, pages 4619–4629. [find](https://scholar.google.com/scholar?q=Santiago+Castro%2C+Devamanyu+Hazarika%2C+Vero%CC%81nica+Pe%CC%81rez%2DRosas%2C+Roger+Zimmermann%2C+Rada+Mihalcea%2C+and+Soujanya+Poria%2E+2019%2E+Towards+multimodal+sarcasm+detection+%28an+obviously+perfect+paper%29%2E+In+Proceeding) [our card](https://md.research.elamur.ai/free/en/papers/castro-2019-towards-multimodal-sarcasm-detection-obviously/)
11. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP, pages 1532– 1543. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+EMNLP%2C+pages+1532%E2%80%93+1543%2E)
12. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of CoNLL, pages 107–116. [find](https://scholar.google.com/scholar?q=Dmitry+Davidov%2C+Oren+Tsur%2C+and+Ari+Rappoport%2E+2010%2E+Semi%2Dsupervised+recognition+of+sarcastic+sentences+in+twitter+and+amazon%2E+In+Proceedings+of+CoNLL%2C+pages+107%E2%80%93116%2E) [our card](https://md.research.elamur.ai/free/en/papers/davidov-2010-semi-supervised-recognition-sarcasm-twitter/)
13. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, and Prateek Vij. 2016. A deeper look into sarcastic tweets using deep convolutional neural networks. In Proceedings of COLING, pages 1601–1612. [find](https://scholar.google.com/scholar?q=Soujanya+Poria%2C+Erik+Cambria%2C+Devamanyu+Hazarika%2C+and+Prateek+Vij%2E+2016%2E+A+deeper+look+into+sarcastic+tweets+using+deep+convolutional+neural+networks%2E+In+Proceedings+of+COLING%2C+pages+1601%E2%80%931612%2E) [our card](https://md.research.elamur.ai/free/en/papers/poria-2016-deeper-look-into-sarcastic-tweets/)
14. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceedings of EMNLP, pages 1615–1625. [find](https://scholar.google.com/scholar?q=Bjarke+Felbo%2C+Alan+Mislove%2C+Anders+S%C3%B8gaard%2C+Iyad+Rahwan%2C+and+Sune+Lehmann%2E+2017%2E+Using+millions+of+emoji+occurrences+to+learn+any%2Ddomain+representations+for+detecting+sentiment%2C+emotion+and+sarcasm%2E+I) [our card](https://md.research.elamur.ai/free/en/papers/felbo-2017-millions-emoji-occurrences-learn-any/)
15. Aniruddha Ghosh and Tony Veale. 2017. Magnets for sarcasm: Making sarcasm detection timely, contextual and very personal. In Proceedings of EMNLP, pages 482–491. [find](https://scholar.google.com/scholar?q=Aniruddha+Ghosh+and+Tony+Veale%2E+2017%2E+Magnets+for+sarcasm%3A+Making+sarcasm+detection+timely%2C+contextual+and+very+personal%2E+In+Proceedings+of+EMNLP%2C+pages+482%E2%80%93491%2E) [our card](https://md.research.elamur.ai/free/en/papers/ghosh-2017-magnets-sarcasm-making-sarcasm-detection/)
16. Roberto González-Ibánez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: a closer look. In Proceedings of ACL, pages 581–586. [find](https://scholar.google.com/scholar?q=Roberto+Gonza%CC%81lez%2DIba%CC%81nez%2C+Smaranda+Muresan%2C+and+Nina+Wacholder%2E+2011%2E+Identifying+sarcasm+in+twitter%3A+a+closer+look%2E+In+Proceedings+of+ACL%2C+pages+581%E2%80%93586%2E) [our card](https://md.research.elamur.ai/free/en/papers/ibanez-2011-identifying-sarcasm-twitter-closer-look/)
17. Alex Graves and Jürgen Schmidhuber. 2005. Frame-wise phoneme classification with bidirectional lstm and other neural network architectures. Neural networks, 18(5-6):602–610. [find](https://scholar.google.com/scholar?q=Alex+Graves+and+Ju%CC%88rgen+Schmidhuber%2E+2005%2E+Frame%2Dwise+phoneme+classification+with+bidirectional+lstm+and+other+neural+network+architectures%2E+Neural+networks%2C+18%285%2D6%29%3A602%E2%80%93610%2E)
18. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of CVPR, pages 770–778. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+2016%2E+Deep+residual+learning+for+image+recognition%2E+In+Proceedings+of+CVPR%2C+pages+770%E2%80%93778%2E)
19. Irazú Hernández-Farı́as, José-Miguel Benedı́, and Paolo Rosso. 2015. Applying basic features from sentiment analysis for automatic irony detection. In Proceedings of IbPRIA, pages 337–344. [find](https://scholar.google.com/scholar?q=Irazu%CC%81+Herna%CC%81ndez%2DFar%C4%B1%CC%81as%2C+Jose%CC%81%2DMiguel+Bened%C4%B1%CC%81%2C+and+Paolo+Rosso%2E+2015%2E+Applying+basic+features+from+sentiment+analysis+for+automatic+irony+detection%2E+In+Proceedings+of+IbPRIA%2C+pages+337%E2%80%93344%2E)
20. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of WSDM, pages 97–106. [find](https://scholar.google.com/scholar?q=Ashwin+Rajadesingan%2C+Reza+Zafarani%2C+and+Huan+Liu%2E+2015%2E+Sarcasm+detection+on+twitter%3A+A+behavioral+modeling+approach%2E+In+Proceedings+of+WSDM%2C+pages+97%E2%80%93106%2E)
21. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of EMNLP, pages 704–714. [find](https://scholar.google.com/scholar?q=Ellen+Riloff%2C+Ashequl+Qadir%2C+Prafulla+Surve%2C+Lalindra+De+Silva%2C+Nathan+Gilbert%2C+and+Ruihong+Huang%2E+2013%2E+Sarcasm+as+contrast+between+a+positive+sentiment+and+negative+situation%2E+In+Proceedings+of+EMNL) [our card](https://md.research.elamur.ai/free/en/papers/prag-2013-riloff-sarcasm-as-contrast-between-positive/)
22. Rossano Schifanella, Paloma de Juan, Joel Tetreault, and Liangliang Cao. 2016. Detecting sarcasm in multimodal social platforms. In Proceedings of ACM MM, pages 1136–1145. [find](https://scholar.google.com/scholar?q=Rossano+Schifanella%2C+Paloma+de+Juan%2C+Joel+Tetreault%2C+and+Liangliang+Cao%2E+2016%2E+Detecting+sarcasm+in+multimodal+social+platforms%2E+In+Proceedings+of+ACM+MM%2C+pages+1136%E2%80%931145%2E) [our card](https://md.research.elamur.ai/free/en/papers/schifanella-2016-detecting-sarcasm-multimodal-social-platforms/)
23. Yi Tay, Anh Tuan Luu, Siu Cheung Hui, and Jian Su. 2018. Reasoning with sarcasm by reading in-between. In Proceedings of ACL, pages 1010–1020. [find](https://scholar.google.com/scholar?q=Yi+Tay%2C+Anh+Tuan+Luu%2C+Siu+Cheung+Hui%2C+and+Jian+Su%2E+2018%2E+Reasoning+with+sarcasm+by+reading+in%2Dbetween%2E+In+Proceedings+of+ACL%2C+pages+1010%E2%80%931020%2E) [our card](https://md.research.elamur.ai/free/en/papers/tay-2018-reasoning-sarcasm-reading-between/)
24. Oren Tsur, Dmitry Davidov, and Ari Rappoport. 2010. Icwsm—a great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews. In Proceedings of ICWSM, pages 162–169. [find](https://scholar.google.com/scholar?q=Oren+Tsur%2C+Dmitry+Davidov%2C+and+Ari+Rappoport%2E+2010%2E+Icwsm%E2%80%94a+great+catchy+name%3A+Semi%2Dsupervised+recognition+of+sarcastic+sentences+in+online+product+reviews%2E+In+Proceedings+of+ICWSM%2C+pages+162%E2%80%93169%2E)
25. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS, pages 5998– 6008. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Proceedings+of+NIPS%2C+pages+5998%E2%80%93+6008%2E)
26. Penghui Wei, Wenji Mao, and Chen Guandan. 2019. A topic-aware reinforced model for weakly supervised stance detection. In Proceedings of AAAI, pages 7249–7256. [find](https://scholar.google.com/scholar?q=Penghui+Wei%2C+Wenji+Mao%2C+and+Chen+Guandan%2E+2019%2E+A+topic%2Daware+reinforced+model+for+weakly+supervised+stance+detection%2E+In+Proceedings+of+AAAI%2C+pages+7249%E2%80%937256%2E)
27. Meishan Zhang, Yue Zhang, and Guohong Fu. 2016. Tweet sarcasm detection using deep neural network. In Proceedings of COLING, pages 2449–2460. [find](https://scholar.google.com/scholar?q=Meishan+Zhang%2C+Yue+Zhang%2C+and+Guohong+Fu%2E+2016%2E+Tweet+sarcasm+detection+using+deep+neural+network%2E+In+Proceedings+of+COLING%2C+pages+2449%E2%80%932460%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2016-tweet-sarcasm-detection-deep-neural/)