---
title: "Multi-Modal Sarcasm Detection in Twitter with — Cai 2019"
description: "Sarcasm detection on Twitter has mostly relied on text alone."
authors: ["Yitao Cai", "Huiyu Cai", "Xiaojun Wan"]
year: 2019
venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P19-1239/"
doi: "10.18653/v1/p19-1239"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cai-2019-multi-modal-sarcasm-detection-twitter/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cai-2019-multi-modal-sarcasm-detection-twitter/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / Multi-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model

# Multi-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model

**Authors:** Yitao Cai, Huiyu Cai, [Xiaojun Wan](https://md.research.elamur.ai/free/en/authors/xiaojun-wan/) · **Year:** 2019 · **Venue:** Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P19-1239/) · [PDF](https://aclanthology.org/P19-1239.pdf) · [DOI 10.18653/v1/p19-1239](https://doi.org/10.18653/v1/p19-1239) · [OpenAlex](https://openalex.org/W2951937667)

**Citations:** 377 citations (openalex-snapshot · 2026-10-14) · [73](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Sarcasm detection on Twitter has mostly relied on text alone. However, many tweets include pictures that can reveal or confirm an ironic intent, for example cheerful words about the weather next to a photo of storm clouds. Cai, Cai and Wan address sarcasm detection for tweets that pair text with an image. They build and publicly release a new Twitter dataset for this task.

Their hierarchical fusion model treats three inputs as separate modalities: the text, the image, and a set of predicted image attributes (short descriptive words). The model combines them in three stages:
- Attribute information sets the starting state of a Bi-LSTM that encodes the text.
- Each modality's features are rebuilt with guidance from the others.
- The refined vectors are merged through a learned weighted average before classification.

The full model beats text-only and simple concatenation baselines, and sign tests show the gains are significant. Ablations confirm that every fusion stage and every modality helps. The study shows that structured multimodal fusion, with attributes linking pixels to words, gives better sarcasm detection on social media.

## Key points
- Image features come from ResNet-50 V2 applied to 14×14 regions (196 regions) of a 448×448 image. Their mean serves as the image guidance vector.
- An attribute predictor based on ResNet-101 is pre-trained on ImageNet and fine-tuned on 1000 attributes taken from COCO captions. It outputs five attributes per image, which are encoded with GloVe and weighted by attention.
- The full model scores 0.8018 F-score and 0.8344 accuracy. Removing early fusion lowers F-score to 0.7880, and removing representation fusion lowers it to 0.7902.
- Early fusion works better with attribute vectors than with image vectors (F-score 0.7787 with image vectors). This suggests attributes narrow the gap between text and visual representations.
- Sign tests against the Text(Bi-LSTM), Concat(2) and Concat(3) baselines give p \< 0.05. The error analysis shows the model misses sarcasm that depends on common-sense knowledge, such as recognizing an insulting gesture.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

## Cited by

Cited by

73 papers of our library cite it (of the 140884 whose bibliographies we read); 15 of them have a page here, 58 not yet.

- [CofiPara: A Coarse-to-fine Paradigm for Multimodal Sarcasm Target Identification with Large Multimodal Models](https://md.research.elamur.ai/free/en/papers/chen-2024-cofipara-coarse-fine-paradigm-multimodal/)
- [Linguistic Features in Automatic Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/ducret-2020-linguistic-features-automatic-sarcasm-detection/)
- [A Report on the 2020 Sarcasm Detection Shared Task](https://md.research.elamur.ai/free/en/papers/ghosh-2020-report-2020-sarcasm-detection-shared/)
- [Generalizable Sarcasm Detection is Just Around the Corner, of Course!](https://md.research.elamur.ai/free/en/papers/jang-2024-generalizable-sarcasm-detection-is-just/)
- [When did you become so smart, oh wise one?! Sarcasm Explanation in Multi-modal Multi-party Dialogues](https://md.research.elamur.ai/free/en/papers/kumar-2022-when-did-you-become-so/)
- [Multi-Modal Sarcasm Detection via Cross-Modal Graph Convolutional Network](https://md.research.elamur.ai/free/en/papers/liang-2022-multi-modal-sarcasm-detection-cross/)
- [What Does Your Smile Mean? Jointly Detecting Multi-Modal Sarcasm and Sentiment Using Quantum Probability](https://md.research.elamur.ai/free/en/papers/liu-2021-what-does-your-smile-mean/)
- [Towards Multi-Modal Sarcasm Detection via Hierarchical Congruity Modeling with Knowledge Enhancement](https://md.research.elamur.ai/free/en/papers/liu-2022-multi-modal-sarcasm-detection-hierarchical/)
- [Modeling Intra and Inter-modality Incongruity for Multi-Modal Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/pan-2020-modeling-intra-inter-modality-incongruity/)
- [A Multimodal Corpus for Emotion Recognition in Sarcasm](https://md.research.elamur.ai/free/en/papers/ray-2022-multimodal-corpus-emotion-recognition-sarcasm/)
- [A Novel Hierarchical BERT Architecture for Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/srivastava-2020-novel-hierarchical-bert-architecture-sarcasm/)
- [Dynamic Routing Transformer Network for Multimodal Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/tian-2023-dynamic-routing-transformer-network-multimodal/)
- [Building a Bridge: A Method for Image-Text Sarcasm Detection Without Pretraining on Image-Text Data](https://md.research.elamur.ai/free/en/papers/wang-2020-building-bridge-method-image-text/)
- [Multimodal Sarcasm Target Identification in Tweets](https://md.research.elamur.ai/free/en/papers/wang-2022-multimodal-sarcasm-target-identification-tweets/)
- [Reasoning with Multimodal Sarcastic Tweets via Modeling Cross-Modality Contrast and Semantic Association](https://md.research.elamur.ai/free/en/papers/xu-2020-reasoning-multimodal-sarcastic-tweets-modeling/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{cai2019multi,
  title = {{Multi-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model}},
  author = {Cai, Yitao and Cai, Huiyu and Wan, Xiaojun},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1239},
  url = {https://doi.org/10.18653/v1/p19-1239}
}
```

- [← previous Multi-hop Reading Comprehension through Question Decomposition and Rescoring](https://md.research.elamur.ai/free/en/papers/min-2019-multi-hop-reading-comprehension-through/)
- [next → Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://md.research.elamur.ai/free/en/papers/fabbri-2019-multi-news-large-scale-multi/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/)

## References

28 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Silvio Amir, Byron C. Wallace, Hao Lyu, Paula Carvalho, and Mário J. Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. CoRR, abs/1607.00976. [↗](https://arxiv.org/abs/1607.00976) [our card](https://md.research.elamur.ai/free/en/papers/amir-2016-modelling-context-user-embeddings-sarcasm/)
2. David Bamman and Noah A. Smith. 2015. Contextualized sarcasm detection on twitter. In ICWSM. [find](https://scholar.google.com/scholar?q=David+Bamman+and+Noah+A%2E+Smith%2E+2015%2E+Contextualized+sarcasm+detection+on+twitter%2E+In+ICWSM%2E)
3. Christos Baziotis, Nikos Athanasiou, Pinelopi Papalampidi, Athanasia Kolovou, Georgios Paraskevopoulos, Nikolaos Ellinas, and Alexandros Potamianos. 2018. Ntua-slp at semeval-2018 task 3: Tracking ironic tweets using ensembles of word and character level attentive rnns. arXiv preprint arXiv:1804.06659. [↗](https://arxiv.org/abs/1804.06659)
4. M. Bouazizi and T. Ohtsuki. 2015. Sarcasm detection in twitter: ”all your products are incredibly amazing!!!” - are they really? In 2015 IEEE Global Communications Conference (GLOBECOM), pages 1–6. [find](https://scholar.google.com/scholar?q=M%2E+Bouazizi+and+T%2E+Ohtsuki%2E+2015%2E+Sarcasm+detection+in+twitter%3A+%E2%80%9Dall+your+products+are+incredibly+amazing%21%21%21%E2%80%9D+%2D+are+they+really%3F+In+2015+IEEE+Global+Communications+Conference+%28GLOBECOM%29%2C+pages+1%E2%80%936%2E)
5. Kan Chen, Jiang Wang, Liang-Chieh Chen, Haoyuan Gao, Wei Xu, and Ram Nevatia. 2015. Abc-cnn: An attention based convolutional neural network for visual question answering. arXiv preprint arXiv:1511.05960. [↗](https://arxiv.org/abs/1511.05960)
6. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, CoNLL ’10, pages 107–116, Stroudsburg, PA, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dmitry+Davidov%2C+Oren+Tsur%2C+and+Ari+Rappoport%2E+2010%2E+Semi%2Dsupervised+recognition+of+sarcastic+sentences+in+twitter+and+amazon%2E+In+Proceedings+of+the+Fourteenth+Conference+on+Computational+Natural+Langu) [our card](https://md.research.elamur.ai/free/en/papers/davidov-2010-semi-supervised-recognition-sarcasm-twitter/)
7. Shelly Dews and Ellen Winner. 1995. Muting the meaning a social function of irony. Metaphor and Symbol, 10(1):3–19. Aniruddha Ghosh and Dr. [find](https://scholar.google.com/scholar?q=Shelly+Dews+and+Ellen+Winner%2E+1995%2E+Muting+the+meaning+a+social+function+of+irony%2E+Metaphor+and+Symbol%2C+10%281%29%3A3%E2%80%9319%2E+Aniruddha+Ghosh+and+Dr%2E)
8. Tony Veale. 2016. Frack-ing sarcasm using neural network. In Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 161–169. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tony+Veale%2E+2016%2E+Frack%2Ding+sarcasm+using+neural+network%2E+In+Proceedings+of+the+7th+Workshop+on+Computational+Approaches+to+Subjectivity%2C+Sentiment+and+Social+Media+Analysis%2C+pages+161%E2%80%93169%2E+Associatio)
9. Yue Gu, Kangning Yang, Shiyu Fu, Shuhong Chen, Xinyu Li, and Ivan Marsic. 2018a. Hybrid attention based multimodal network for spoken language classification. In Proceedings of the 27th International Conference on Computational Linguistics, Yue Gu, Kangning Yang, Shiyu Fu, Shuhong Chen, Xinyu Li, and Ivan Marsic. 2018b. Multimodal affective analysis using hierarchical attention strategy with word-level alignment. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2225–2235. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yue+Gu%2C+Kangning+Yang%2C+Shiyu+Fu%2C+Shuhong+Chen%2C+Xinyu+Li%2C+and+Ivan+Marsic%2E+2018a%2E+Hybrid+attention+based+multimodal+network+for+spoken+language+classification%2E+In+Proceedings+of+the+27th+International+)
10. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Identity mappings in deep residual networks. CoRR, abs/1603.05027. [↗](https://arxiv.org/abs/1603.05027)
11. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. [↗](https://doi.org/10.1162/neco.1997.9.8.1735)
12. Yoon Kim. 2014. Convolutional neural networks for sentence classification. CoRR, abs/1408.5882. [↗](https://arxiv.org/abs/1408.5882) [our card](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/)
13. Diederik P. Kingma and Jimmy Ba. 2014. A method for stochastic optimization. abs/1412.6980. Adam: CoRR, Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision – ECCV 2014, pages 740–755, Cham. Springer International Publishing. [↗](https://arxiv.org/abs/1412.6980) [our card](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/)
14. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Conference on Empirical Methods in Natural Language Processing, pages 1532– 1543. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1532%E2%80%93+1543%2E)
15. Soujanya Poria, Erik Cambria, and Alexander Gelbukh. 2015. Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 2539–2544. [find](https://scholar.google.com/scholar?q=Soujanya+Poria%2C+Erik+Cambria%2C+and+Alexander+Gelbukh%2E+2015%2E+Deep+convolutional+neural+network+textual+features+and+multiple+kernel+learning+for+utterance%2Dlevel+multimodal+sentiment+analysis%2E+In+Proceed)
16. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, and Prateek Vij. 2016. A deeper look into sarcastic tweets using deep convolutional neural networks. CoRR, abs/1610.08815. [↗](https://arxiv.org/abs/1610.08815) [our card](https://md.research.elamur.ai/free/en/papers/poria-2016-deeper-look-into-sarcastic-tweets/)
17. Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm detection on czech and english twitter. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 213–223. Dublin City University and Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Toma%CC%81s%CC%8C+Pta%CC%81c%CC%8Cek%2C+Ivan+Habernal%2C+and+Jun+Hong%2E+2014%2E+Sarcasm+detection+on+czech+and+english+twitter%2E+In+Proceedings+of+COLING+2014%2C+the+25th+International+Conference+on+Computational+Linguistics%3A+Tech) [our card](https://md.research.elamur.ai/free/en/papers/ptacek-2014-sarcasm-detection-czech-english-twitter/)
18. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalin-dra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In EMNLP 2013 - 2013 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, pages 704–714. Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Ellen+Riloff%2C+Ashequl+Qadir%2C+Prafulla+Surve%2C+Lalin%2Ddra+De+Silva%2C+Nathan+Gilbert%2C+and+Ruihong+Huang%2E+2013%2E+Sarcasm+as+contrast+between+a+positive+sentiment+and+negative+situation%2E+In+EMNLP+2013+%2D+2013+) [our card](https://md.research.elamur.ai/free/en/papers/prag-2013-riloff-sarcasm-as-contrast-between-positive/)
19. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–252. [find](https://scholar.google.com/scholar?q=Olga+Russakovsky%2C+Jia+Deng%2C+Hao+Su%2C+Jonathan+Krause%2C+Sanjeev+Satheesh%2C+Sean+Ma%2C+Zhiheng+Huang%2C+Andrej+Karpathy%2C+Aditya+Khosla%2C+Michael+Bernstein%2C+Alexander+C%2E+Berg%2C+and+Li+Fei%2DFei%2E+2015%2E+ImageNet+Larg) [our card](https://md.research.elamur.ai/free/en/papers/russakovsky-2014-imagenet-large-scale-visual-recognition/)
20. Rossano Schifanella, Paloma de Juan, Joel R. Tetreault, and Liangliang Cao. 2016. Detecting sarcasm in multimodal social platforms. CoRR, abs/1608.02289. [↗](https://arxiv.org/abs/1608.02289) [our card](https://md.research.elamur.ai/free/en/papers/schifanella-2016-detecting-sarcasm-multimodal-social-platforms/)
21. Yi Tay, Luu Anh Tuan, Siu Cheung Hui, and Jian Su. 2018. Reasoning with sarcasm by reading in-between. CoRR, abs/1805.02856. [↗](https://arxiv.org/abs/1805.02856) [our card](https://md.research.elamur.ai/free/en/papers/tay-2018-reasoning-sarcasm-reading-between/)
22. Haohan Wang, Aaksha Meghawat, Louis-Philippe Morency, and Eric P. Xing. 2016. Select-additive learning: Improving cross-individual generalization in multimodal sentiment analysis. CoRR, abs/1609.05244. [↗](https://arxiv.org/abs/1609.05244)
23. Peng Wang, Qi Wu, Chunhua Shen, and Anton van den Hengel. 2017. The vqa-machine: Learning how to use existing vision algorithms to answer new questions. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn, volume 4. [find](https://scholar.google.com/scholar?q=Peng+Wang%2C+Qi+Wu%2C+Chunhua+Shen%2C+and+Anton+van+den+Hengel%2E+2017%2E+The+vqa%2Dmachine%3A+Learning+how+to+use+existing+vision+algorithms+to+answer+new+questions%2E+In+Proc%2E+IEEE+Conf%2E+Comp%2E+Vis%2E+Patt%2E+Recogn%2C+vo)
24. Chuhan Wu, Fangzhao Wu, Sixing Wu, Junxin Liu, Zhigang Yuan, and Yongfeng Huang. 2018. Thu ngn at semeval-2018 task 3: Tweet irony detection with densely connected lstm and multi-task learning. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 51–56. [find](https://scholar.google.com/scholar?q=Chuhan+Wu%2C+Fangzhao+Wu%2C+Sixing+Wu%2C+Junxin+Liu%2C+Zhigang+Yuan%2C+and+Yongfeng+Huang%2E+2018%2E+Thu+ngn+at+semeval%2D2018+task+3%3A+Tweet+irony+detection+with+densely+connected+lstm+and+multi%2Dtask+learning%2E+In+Pro)
25. Qi Wu, Chunhua Shen, Lingqiao Liu, Anthony Dick, and Anton van den Hengel. 2016. What value do explicit high level concepts have in vision to language problems? In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 203–212. [find](https://scholar.google.com/scholar?q=Qi+Wu%2C+Chunhua+Shen%2C+Lingqiao+Liu%2C+Anthony+Dick%2C+and+Anton+van+den+Hengel%2E+2016%2E+What+value+do+explicit+high+level+concepts+have+in+vision+to+language+problems%3F+In+Proceedings+of+the+IEEE+conference+o)
26. Yang Yang, Jia Jia, Shumei Zhang, Boya Wu, Qicong Chen, Juanzi Li, Chunxiao Xing, and Jie Tang. 2014. How do your friends on social media disclose your emotions? In Twenty-Eighth AAAI Conference on Artificial Intelligence, pages 306–312. [find](https://scholar.google.com/scholar?q=Yang+Yang%2C+Jia+Jia%2C+Shumei+Zhang%2C+Boya+Wu%2C+Qicong+Chen%2C+Juanzi+Li%2C+Chunxiao+Xing%2C+and+Jie+Tang%2E+2014%2E+How+do+your+friends+on+social+media+disclose+your+emotions%3F+In+Twenty%2DEighth+AAAI+Conference+on+Ar)
27. Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. CoRR, abs/1707.07250. [↗](https://arxiv.org/abs/1707.07250) [our card](https://md.research.elamur.ai/free/en/papers/zadeh-2017-tensor-fusion-network-multimodal-sentiment/)
28. Meishan Zhang, Yue Zhang, and Guohong Fu. 2016. Tweet sarcasm detection using deep neural network. In COLING. [find](https://scholar.google.com/scholar?q=Meishan+Zhang%2C+Yue+Zhang%2C+and+Guohong+Fu%2E+2016%2E+Tweet+sarcasm+detection+using+deep+neural+network%2E+In+COLING%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2016-tweet-sarcasm-detection-deep-neural/)