---
title: "Multi-source Semantic Graph-based Multimodal — Jing 2023"
description: "The paper addresses Multimodal Sarcasm Explanation (MuSE), where a model writes a short natural-language sentence explaining why an image-plus-caption…"
authors: ["Liqiang Jing", "Xuemeng Song", "Kun Ouyang", "Mengzhao Jia", "Liqiang Nie"]
year: 2023
venue: "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2023.acl-long.635/"
doi: "10.18653/v1/2023.acl-long.635"
arxiv: "2306.16650"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/jing-2023-multi-source-semantic-graph-based/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/jing-2023-multi-source-semantic-graph-based/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2023](https://md.research.elamur.ai/free/papers/2023/page/4/) / Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generation

# Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generation

**Authors:** Liqiang Jing, Xuemeng Song, Kun Ouyang, Mengzhao Jia, Liqiang Nie · **Year:** 2023 · **Venue:** Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.acl-long.635/) · [PDF](https://aclanthology.org/2023.acl-long.635.pdf) · [DOI 10.18653/v1/2023.acl-long.635](https://doi.org/10.18653/v1/2023.acl-long.635) · [arXiv 2306.16650](https://arxiv.org/abs/2306.16650) · [OpenAlex](https://openalex.org/W4385564954)

**Citations:** 19 citations (openalex-snapshot · 2026-10-14) · 7 in our library, of the 140673 papers whose bibliographies we read

## Summary
The paper addresses Multimodal Sarcasm Explanation (MuSE), where a model writes a short natural-language sentence explaining why an image-plus-caption social media post is sarcastic. The earlier approach, ExMore on the MORE dataset, fed global image features into a BART-based generator. The authors identify three weaknesses in that design: visual features do not fit the text-only semantic space of BART, object-level image details are discarded, and no external knowledge is used. Their system, TEAM, takes a different route. It converts the image into textual object labels (class and attribute) using Faster-RCNN. It then expands the caption and these labels with one-hop neighbours from ConceptNet. Next, it links caption tokens, object labels and knowledge concepts in a multi-source semantic graph processed by a GCN. Finally, a BART decoder generates the explanation. On MORE, TEAM clearly surpasses prior methods. Ablations show that object metadata works better than raw visual features and that the typed graph relations matter. The work suggests that turning vision into text-like semantics helps pretrained language generators reason about multimodal irony.

## Key points
- Evaluation uses the public MORE dataset (Desai et al., 2022). TEAM beats the strongest baseline, ExMore, by reported margins of 28.90 on BLEU-4 and 22.47 on ROUGE-L.
- Only the top-K most confident Faster-RCNN regions are kept. Each region contributes its object class and attribute as text, and its visual vectors are not used.
- External knowledge comes from ConceptNet. Concepts found in the caption and object labels act as anchors, and their one-hop neighbours are retrieved.
- The ablation variants (w/o-Caption, w/-Visual, w/o-Obj, w/o-Graph, w/-FullGraph) all perform worse than the full model. Removing the caption causes the largest drop.
- The authors note two limitations. The model does worse on posts with text inside the image (OCR samples), and it weights all knowledge concepts equally. Code and parameters are available at github.com/LiqiangJing/TEAM.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{jing2023multi,
  title = {{Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generation}},
  author = {Jing, Liqiang and Song, Xuemeng and Ouyang, Kun and Jia, Mengzhao and Nie, Liqiang},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.635},
  eprint = {2306.16650},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.acl-long.635}
}
```

- [← previous mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://md.research.elamur.ai/free/papers/ye-2023-mplug-owl-modularization-empowers-large/)
- [next → Multi-step Jailbreaking Privacy Attacks on ChatGPT](https://md.research.elamur.ai/free/papers/li-2023-multi-step-jailbreaking-privacy-attacks/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/papers/2023/page/4/)

## References

31 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abhijeet Dubey, Aditya Joshi, and Pushpak Bhattacharyya. 2019. Deep models for converting sarcastic utterances into their non sarcastic interpretation. In Proceedings of the ACM India Joint International Conference on Data Science and Management of Data, pages 289–292. ACM. [find](https://scholar.google.com/scholar?q=Abhijeet+Dubey%2C+Aditya+Joshi%2C+and+Pushpak+Bhattacharyya%2E+2019%2E+Deep+models+for+converting+sarcastic+utterances+into+their+non+sarcastic+interpretation%2E+In+Proceedings+of+the+ACM+India+Joint+Internatio)
2. Gavin Abercrombie and Dirk Hovy. 2016. Putting sarcasm detection into context: The effects of class imbalance and manual labelling on supervised machine classification of twitter conversations. In Proceedings of the Association for Computational Linguistics Student Research Workshop, pages 107–113. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Gavin+Abercrombie+and+Dirk+Hovy%2E+2016%2E+Putting+sarcasm+detection+into+context%3A+The+effects+of+class+imbalance+and+manual+labelling+on+supervised+machine+classification+of+twitter+conversations%2E+In+Pro) [our card](https://md.research.elamur.ai/free/papers/abercrombie-2016-putting-sarcasm-detection-into-context/)
3. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rah-wan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1615– 1625. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bjarke+Felbo%2C+Alan+Mislove%2C+Anders+S%C3%B8gaard%2C+Iyad+Rah%2Dwan%2C+and+Sune+Lehmann%2E+2017%2E+Using+millions+of+emoji+occurrences+to+learn+any%2Ddomain+representations+for+detecting+sentiment%2C+emotion+and+sarcasm%2E+) [our card](https://md.research.elamur.ai/free/papers/felbo-2017-millions-emoji-occurrences-learn-any/)
4. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Conference on Computer Vision and Pattern Recognition, pages 6077–6086. IEEE. [find](https://scholar.google.com/scholar?q=Peter+Anderson%2C+Xiaodong+He%2C+Chris+Buehler%2C+Damien+Teney%2C+Mark+Johnson%2C+Stephen+Gould%2C+and+Lei+Zhang%2E+2018%2E+Bottom%2Dup+and+top%2Ddown+attention+for+image+captioning+and+visual+question+answering%2E+In+Conf)
5. Nastaran Babanejad, Heidar Davoudi, Aijun An, and Manos Papagelis. 2020. Affective and contextual embedding for sarcasm detection. In Proceedings of the International Conference on Computational Linguistics, pages 225–243. International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nastaran+Babanejad%2C+Heidar+Davoudi%2C+Aijun+An%2C+and+Manos+Papagelis%2E+2020%2E+Affective+and+contextual+embedding+for+sarcasm+detection%2E+In+Proceedings+of+the+International+Conference+on+Computational+Lingu) [our card](https://md.research.elamur.ai/free/papers/babanejad-2020-affective-contextual-embedding-sarcasm-detection/)
6. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Satanjeev+Banerjee+and+Alon+Lavie%2E+2005%2E+METEOR%3A+an+automatic+metric+for+MT+evaluation+with+improved+correlation+with+human+judgments%2E+In+Proceedings+of+the+Workshop+on+Intrinsic+and+Extrinsic+Evaluat) [our card](https://md.research.elamur.ai/free/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)
7. Deepanway Ghosal, Devamanyu Hazarika, Abhinaba Roy, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. 2020. Kingdom: Knowledge-guided domain adaptation for sentiment analysis. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 3198–3210. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Deepanway+Ghosal%2C+Devamanyu+Hazarika%2C+Abhinaba+Roy%2C+Navonil+Majumder%2C+Rada+Mihalcea%2C+and+Soujanya+Poria%2E+2020%2E+Kingdom%3A+Knowledge%2Dguided+domain+adaptation+for+sentiment+analysis%2E+In+Proceedings+of+the)
8. Debanjan Ghosh, Alexander Richard Fabbri, and Smaranda Muresan. 2017. The role of conversation context for sarcasm detection in online interactions. In Proceedings of the Annual SIGdial Meeting on Discourse and Dialogue, pages 186–196. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Debanjan+Ghosh%2C+Alexander+Richard+Fabbri%2C+and+Smaranda+Muresan%2E+2017%2E+The+role+of+conversation+context+for+sarcasm+detection+in+online+interactions%2E+In+Proceedings+of+the+Annual+SIGdial+Meeting+on+Dis) [our card](https://md.research.elamur.ai/free/papers/ghosh-2017-role-conversation-context-sarcasm-detection/)
9. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, pages 757–762. Association for Computer Linguistics. [find](https://scholar.google.com/scholar?q=Aditya+Joshi%2C+Vinita+Sharma%2C+and+Pushpak+Bhattacharyya%2E+2015%2E+Harnessing+context+incongruity+for+sarcasm+detection%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+Computational+Linguistic) [our card](https://md.research.elamur.ai/free/papers/joshi-2015-harnessing-context-incongruity-sarcasm-detection/)
10. Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations. OpenReview.net. Shivani Kumar, Atharva Kulkarni, Md. [↗](https://doi.org/10.48550/arxiv.1609.02907) [our card](https://md.research.elamur.ai/free/papers/kipf-2016-semi-supervised-classification-graph-convolutional/)
11. Shad Akhtar, and Tanmoy Chakraborty. 2022. When did you become so smart, oh wise one?! sarcasm explanation in multi-modal multi-party dialogues. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 5956–5968. Association for Computational Linguistics. [↗](https://doi.org/10.48448/6ejk-z717) [our card](https://md.research.elamur.ai/free/papers/kumar-2022-when-did-you-become-so/)
12. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 7871–7880. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Yinhan+Liu%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Abdelrahman+Mohamed%2C+Omer+Levy%2C+Veselin+Stoyanov%2C+and+Luke+Zettlemoyer%2E+2020%2E+BART%3A+denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+l) [our card](https://md.research.elamur.ai/free/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
13. Abhijit Mishra, Tarun Tater, and Karthik Sankaranarayanan. 2019. A modular architecture for unsupervised sarcasm generation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing, pages 6143–6153. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Abhijit+Mishra%2C+Tarun+Tater%2C+and+Karthik+Sankaranarayanan%2E+2019%2E+A+modular+architecture+for+unsupervised+sarcasm+generation%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+P) [our card](https://md.research.elamur.ai/free/papers/mishra-2019-modular-architecture-unsupervised-sarcasm-generation/)
14. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 311–318. ACL. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+Computati) [our card](https://md.research.elamur.ai/free/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
15. Lotem Peled and Roi Reichart. 2017. Sarcasm SIGN: interpreting sarcasm with sentiment based monolingual machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1690–1700. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Lotem+Peled+and+Roi+Reichart%2E+2017%2E+Sarcasm+SIGN%3A+interpreting+sarcasm+with+sentiment+based+monolingual+machine+translation%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+Computational+L) [our card](https://md.research.elamur.ai/free/papers/peled-2017-sarcasm-sign-interpreting-sarcasm-sentiment/)
16. Yang Qiao, Liqiang Jing, Xuemeng Song, Xiaolin Chen, Lei Zhu, and Liqiang Nie. 2023. Mutual-enhanced incongruity learning network for multi-modal sarcasm detection. In Thirty-Seventh AAAI Conference on Artificial Intelligence. AAAI Press. [find](https://scholar.google.com/scholar?q=Yang+Qiao%2C+Liqiang+Jing%2C+Xuemeng+Song%2C+Xiaolin+Chen%2C+Lei+Zhu%2C+and+Liqiang+Nie%2E+2023%2E+Mutual%2Denhanced+incongruity+learning+network+for+multi%2Dmodal+sarcasm+detection%2E+In+Thirty%2DSeventh+AAAI+Conference+o)
17. Qintong Li, Piji Li, Zhaochun Ren, Pengjie Ren, and Zhumin Chen. 2022. Knowledge bridging for empathetic dialogue generation. In AAAI Conference on Artificial Intelligence, Conference on Innovative Applications of Artificial Intelligence, The Symposium on Educational Advances in Artificial Intelligence, pages 10993–11001. AAAI Press. [find](https://scholar.google.com/scholar?q=Qintong+Li%2C+Piji+Li%2C+Zhaochun+Ren%2C+Pengjie+Ren%2C+and+Zhumin+Chen%2E+2022%2E+Knowledge+bridging+for+empathetic+dialogue+generation%2E+In+AAAI+Conference+on+Artificial+Intelligence%2C+Conference+on+Innovative+Ap)
18. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing, pages 3980–3990. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nils+Reimers+and+Iryna+Gurevych%2E+2019%2E+Sentence%2Dbert%3A+Sentence+embeddings+using+siamese+bert%2Dnetworks%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+Inte)
19. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+ROUGE%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+Summarization+Branches+Out%2C+pages+74%E2%80%9381%2E+Association+for+Computational+Linguistics%2E)
20. Rossano Schifanella, Paloma de Juan, Joel R. Tetreault, and Liangliang Cao. 2016. Detecting sarcasm in multimodal social platforms. In Proceedings of the ACM Conference on Multimedia Conference, pages 1136–1145. ACM. [find](https://scholar.google.com/scholar?q=Rossano+Schifanella%2C+Paloma+de+Juan%2C+Joel+R%2E+Tetreault%2C+and+Liangliang+Cao%2E+2016%2E+Detecting+sarcasm+in+multimodal+social+platforms%2E+In+Proceedings+of+the+ACM+Conference+on+Multimedia+Conference%2C+pages) [our card](https://md.research.elamur.ai/free/papers/schifanella-2016-detecting-sarcasm-multimodal-social-platforms/)
21. Nayu Liu, Xian Sun, Hongfeng Yu, Wenkai Zhang, and Guangluan Xu. 2020. Multistage fusion with forget gate for multimodal summarization in open-domain videos. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1834–1845. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nayu+Liu%2C+Xian+Sun%2C+Hongfeng+Yu%2C+Wenkai+Zhang%2C+and+Guangluan+Xu%2E+2020%2E+Multistage+fusion+with+forget+gate+for+multimodal+summarization+in+open%2Ddomain+videos%2E+In+Proceedings+of+the+Conference+on+Empiri)
22. Ilya Loshchilov and Frank Hutter. 2017. weight decay regularization in adam. abs/1711.05101. [↗](https://arxiv.org/abs/1711.05101)
23. Fixing CoRR, Lin Ma, Zhengdong Lu, Lifeng Shang, and Hang Li. 2015. Multimodal convolutional neural networks for matching image and sentence. In International Conference on Computer Vision, pages 2623–2631. IEEE. [↗](https://doi.org/10.48550/arxiv.1504.06063)
24. Abdelkader El Mahdaouy, Abdellah El Mekki, Kabil Es-sefar, Nabil El Mamoun, Ismail Berrada, and Ahmed Khoumsi. 2021. Deep multi-task model for sarcasm detection and sentiment analysis in arabic language. In Proceedings of the Arabic Natural Language Processing Workshop, pages 334–339. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2106.12488) [our card](https://md.research.elamur.ai/free/papers/mahdaouy-2021-deep-multi-task-model-sarcasm/)
25. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1073–1083. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1704.04368)
26. Yi Tay, Anh Tuan Luu, Siu Cheung Hui, and Jian Su. 2018. Reasoning with sarcasm by reading in-between. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1010–1020. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yi+Tay%2C+Anh+Tuan+Luu%2C+Siu+Cheung+Hui%2C+and+Jian+Su%2E+2018%2E+Reasoning+with+sarcasm+by+reading+in%2Dbetween%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+1010) [our card](https://md.research.elamur.ai/free/papers/tay-2018-reasoning-sarcasm-reading-between/)
27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, pages 5998–6008. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+Neural+Information+Process)
28. Yiran Xing, Zai Shi, Zhao Meng, Gerhard Lakemeyer, Yunpu Ma, and Roger Wattenhofer. 2021. KM-BART: knowledge enhanced multimodal BART for visual commonsense generation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, pages 525– 535. Association for Computational Linguistics. [↗](https://doi.org/10.18154/rwth-2021-10094)
29. Shaowei Yao and Xiaojun Wan. 2020. Multimodal transformer for multimodal machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 4346–4350. Association for Computational Linguistics. [↗](https://doi.org/10.60692/t3s04-95542)
30. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In International Conference on Learning Representations. OpenRe-view.net. [↗](https://doi.org/10.48550/arxiv.1904.09675)
31. ACL 2023 Responsible NLP Checklist [find](https://scholar.google.com/scholar?q=ACL+2023+Responsible+NLP+Checklist)