---
title: "A Multimodal Corpus for Emotion Recognition in — Ray 2022"
description: "Sarcasm research has mostly focused on detecting whether an utterance is sarcastic."
authors: ["Anupama Ray", "Shubham Mishra", "Apoorva Nunna", "Pushpak Bhattacharyya"]
year: 2022
venue: "Proceedings of the Thirteenth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2022.lrec-1.756/"
arxiv: "2206.02119"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ray-2022-multimodal-corpus-emotion-recognition-sarcasm/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ray-2022-multimodal-corpus-emotion-recognition-sarcasm/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/) / A Multimodal Corpus for Emotion Recognition in Sarcasm

# A Multimodal Corpus for Emotion Recognition in Sarcasm

**Authors:** Anupama Ray, Shubham Mishra, Apoorva Nunna, [Pushpak Bhattacharyya](https://md.research.elamur.ai/free/en/authors/pushpak-bhattacharyya/) · **Year:** 2022 · **Venue:** Proceedings of the Thirteenth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.lrec-1.756/) · [PDF](https://aclanthology.org/2022.lrec-1.756.pdf) · [arXiv 2206.02119](https://arxiv.org/abs/2206.02119) · [OpenAlex](https://openalex.org/W4281762564)

**Citations:** 17 citations (openalex-snapshot · 2026-10-14) · [18](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Sarcasm research has mostly focused on detecting whether an utterance is sarcastic. The emotion behind it, such as frustration, disgust, or ridicule, has received little attention, even though conversational systems need it to respond well. The authors address emotion recognition in sarcastic speech, which they describe as a new task. They start from the emotion-annotated version of MUStARD, a video corpus of sitcom clips. Models trained on it kept disagreeing with the gold labels, so the team re-annotated the data and corrected roughly half of the emotion labels. They then doubled the corpus with new sitcom clips, keeping a 50/50 sarcastic/non-sarcastic balance. Every utterance is labeled with implicit and explicit emotion, valence, arousal, and one of four sarcasm types. The resulting resource, MUStARD++, is benchmarked with text, audio, and video fusion models. These set baselines for emotion classification and valence/arousal regression, and the authors also report gains over the previous state of the art in sarcasm detection. Both data and code are publicly released.

## Key points
- **Label corrections:** The original MUStARD has 690 clips, 345 of them sarcastic. Model-based error analysis flagged 399 disagreements with the labels from Chauhan et al. (2020). Blind re-annotation then confirmed 343 wrong emotion labels: 255 in sarcastic clips and 88 in non-sarcastic ones. Many of the errors were sarcastic utterances marked Neutral.
- **New annotation layers:** MUStARD++ adds valence and arousal scores and sarcasm-type tags following Camp (2012): Propositional, Embedded, Like-prefixed, and Illocutionary. Implicit and explicit emotions are labeled separately.
- **Annotation process:** Seven annotators of varied age, gender, and education labeled perceived emotion, with final labels chosen by majority vote.
- **Benchmark results:** Multiclass emotion F1 remains modest. Implicit emotion reaches about 33, while explicit emotion reaches about 41 (video+text with context). This shows the task is hard.
- **Availability:** Data and code are released at github.com/apoorva-nunna/MUStARD_Plus_Plus. The paper text is CC-BY-NC-4.0, and the summary does not cover the dataset's own license.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

## Cited by

Cited by

18 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 17 not yet.

- [Generalizable Sarcasm Detection is Just Around the Corner, of Course!](https://md.research.elamur.ai/free/en/papers/jang-2024-generalizable-sarcasm-detection-is-just/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ray2022multimodal,
  title = {{A Multimodal Corpus for Emotion Recognition in Sarcasm}},
  author = {Ray, Anupama and Mishra, Shubham and Nunna, Apoorva and Bhattacharyya, Pushpak},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  doi = {10.63317/3d3nbg2do7ny},
  eprint = {2206.02119},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.63317/3d3nbg2do7ny}
}
```

- [← previous A Learning-Based Dependency to Constituency Conversion Algorithm for the Turkish Language](https://md.research.elamur.ai/free/en/papers/an-2022-learning-based-dependency-constituency-conversion/)
- [next → A Paradigm Change for Formal Syntax: Computational Algorithms in the Grammar of English](https://md.research.elamur.ai/free/en/papers/ninio-2022-paradigm-change-formal-syntax-computational/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/)

## References

35 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bedi, M., Kumar, S., Akhtar, M. S., and Chakraborty, T. (2021). Multi-modal sarcasm detection and humor classification in code-mixed conversations. CoRR, abs/2105.09984. [↗](https://arxiv.org/abs/2105.09984)
2. Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., and Narayanan, S. S. (2008). Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335. [↗](https://doi.org/10.1007/s10579-008-9076-6)
3. Cai, Y., Cai, H., and Wan, X. (2019). Multi-modal sarcasm detection in Twitter with hierarchical fusion model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2506–2515, Florence, Italy, July. Association for Computational Linguistics. [↗](https://doi.org/10.60692/gas6m-xj928) [our card](https://md.research.elamur.ai/free/en/papers/cai-2019-multi-modal-sarcasm-detection-twitter/)
4. Camp, E. (2012). Sarcasm, pretense, and the semantics/pragmatics distinction. Noûs, 46(4):587–634. [↗](https://doi.org/10.1111/j.1468-0068.2010.00822.x)
5. Chen, S., Hsu, C., Kuo, C., Huang, T. K., and Ku, L. (2018). Emotionlines: An emotion corpus of multi-party conversations. CoRR, abs/1802.08379. [↗](https://arxiv.org/abs/1802.08379)
6. Cowie, R. and Cornelius, R. R. (2003). Describing the emotional states that are expressed in speech. Speech communication, 40(1-2):5–32. [↗](https://doi.org/10.1016/s0167-6393(02)00071-7)
7. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805)
8. Ekman, P. (1999). Basic emotions. Handbook of cognition and emotion, 98(45-60):16. [find](https://scholar.google.com/scholar?q=Ekman%2C+P%2E+%281999%29%2E+Basic+emotions%2E+Handbook+of+cognition+and+emotion%2C+98%2845%2D60%29%3A16%2E)
9. He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). [↗](https://doi.org/10.48550/arxiv.1512.03385)
10. Jadon, S. and Jasim, M. (2019). Video summarization using keyframe extraction and video skimming. arXiv preprint arXiv:1910.04792. [↗](https://arxiv.org/abs/1910.04792)
11. Joshi, A., Bhattacharyya, P., and Carman, M. J. (2016). Automatic sarcasm detection: A survey. CoRR, abs/1602.03426. [↗](https://arxiv.org/abs/1602.03426) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2016-automatic-sarcasm-detection-survey/)
12. Joshi, A., Bhattacharyya, P., and Carman, M. J. (2017). Automatic sarcasm detection: A survey. ACM Comput. Surv., 50(5), September. [↗](https://doi.org/10.48550/arxiv.1602.03426) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2016-automatic-sarcasm-detection-survey/)
13. Joshi, A., Bhattacharyya, P., and Carman, M. J. (2018). Investigations in Computational Sarcasm. [↗](https://doi.org/10.1007/978-981-10-8396-9)
14. Khodak, M., Saunshi, N., and Vodrahalli, K. (2017). A large self-annotated corpus for sarcasm. CoRR, abs/1704.05579. [↗](https://arxiv.org/abs/1704.05579) [our card](https://md.research.elamur.ai/free/en/papers/khodak-2018-large-self-annotated-corpus-sarcasm/)
15. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2019). BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. CoRR, abs/1910.13461. [↗](https://arxiv.org/abs/1910.13461) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
16. Liu, Y., Albanie, S., Nagrani, A., and Zisserman, A. (2020). Use what you have: Video retrieval using representations from collaborative experts. [↗](https://doi.org/10.48550/arxiv.1907.13487)
17. Maynard, D. G. and Greenwood, M. A. (2014). Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In LREC 2014 Proceedings. [find](https://scholar.google.com/scholar?q=Maynard%2C+D%2E+G%2E+and+Greenwood%2C+M%2E+A%2E+%282014%29%2E+Who+cares+about+sarcastic+tweets%3F+investigating+the+impact+of+sarcasm+on+sentiment+analysis%2E+In+LREC+2014+Proceedings%2E) [our card](https://md.research.elamur.ai/free/en/papers/maynard-2014-who-cares-about-sarcastic-tweets/)
18. McFee, B., Lostanlen, V., McVicar, M., Metsai, A., Balke, S., Thomé, C., Raffel, C., Malek, A., Lee, D., Zalkow, F., Lee, K., Nieto, O., Mason, J., Ellis, D., Yamamoto, R., Seyfarth, S., Battenberg, E., , , Bittner, R., Choi, K., Moore, J., Wei, Z., Hidaka, S., nullmightybofo, Friesch, P., Stöter, F.-R., Hereñú, D., Kim, T., Vollrath, M., and Weiss, A. (2020). librosa/librosa: 0.7.2, January. [find](https://scholar.google.com/scholar?q=McFee%2C+B%2E%2C+Lostanlen%2C+V%2E%2C+McVicar%2C+M%2E%2C+Metsai%2C+A%2E%2C+Balke%2C+S%2E%2C+Thome%CC%81%2C+C%2E%2C+Raffel%2C+C%2E%2C+Malek%2C+A%2E%2C+Lee%2C+D%2E%2C+Zalkow%2C+F%2E%2C+Lee%2C+K%2E%2C+Nieto%2C+O%2E%2C+Mason%2C+J%2E%2C+Ellis%2C+D%2E%2C+Yamamoto%2C+R%2E%2C+Seyfarth%2C+S%2E%2C+Battenberg%2C+)
19. Mohammad, S. M. (2021). Ethics sheet for automatic emotion recognition and sentiment analysis. CoRR, abs/2109.08256. [↗](https://arxiv.org/abs/2109.08256)
20. Oprea, S. and Magdy, W. (2020). iSarcasm: A dataset of intended sarcasm. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1279–1289. [find](https://scholar.google.com/scholar?q=Oprea%2C+S%2E+and+Magdy%2C+W%2E+%282020%29%2E+iSarcasm%3A+A+dataset+of+intended+sarcasm%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+1279%E2%80%931289%2E) [our card](https://md.research.elamur.ai/free/en/papers/oprea-2020-isarcasm-dataset-intended-sarcasm/)
21. Oraby, S., Harrison, V., Reed, L., Hernandez, E., Riloff, E., and Walker, M. (2016). Creating and characterizing a diverse corpus of sarcasm in dialogue. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 31– 41, Los Angeles, September. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Oraby%2C+S%2E%2C+Harrison%2C+V%2E%2C+Reed%2C+L%2E%2C+Hernandez%2C+E%2E%2C+Riloff%2C+E%2E%2C+and+Walker%2C+M%2E+%282016%29%2E+Creating+and+characterizing+a+diverse+corpus+of+sarcasm+in+dialogue%2E+In+Proceedings+of+the+17th+Annual+Meeting+of+t) [our card](https://md.research.elamur.ai/free/en/papers/oraby-2016-creating-characterizing-diverse-corpus-sarcasm/)
22. Pascual, S., Ravanelli, M., Serrà, J., Bonafonte, A., and Bengio, Y. ). Learning Problem-Agnostic Speech Representations from Multiple Self-Supervised Tasks. In INTERSPEECH, 2019, pages 161–165. [find](https://scholar.google.com/scholar?q=Pascual%2C+S%2E%2C+Ravanelli%2C+M%2E%2C+Serra%CC%80%2C+J%2E%2C+Bonafonte%2C+A%2E%2C+and+Bengio%2C+Y%2E+%29%2E+Learning+Problem%2DAgnostic+Speech+Representations+from+Multiple+Self%2DSupervised+Tasks%2E+In+INTERSPEECH%2C+2019%2C+pages+161%E2%80%93165%2E)
23. Plutchick, R. (1980). Emotion: a psychoevolutionary synthesis. New York, Happer & Row. [find](https://scholar.google.com/scholar?q=Plutchick%2C+R%2E+%281980%29%2E+Emotion%3A+a+psychoevolutionary+synthesis%2E+New+York%2C+Happer+%26+Row%2E)
24. Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., and Mihalcea, R. (2018). MELD: A multimodal multi-party dataset for emotion recognition in conversations. CoRR, abs/1810.02508. [↗](https://arxiv.org/abs/1810.02508) [our card](https://md.research.elamur.ai/free/en/papers/poria-2018-meld-multimodal-multi-party-dataset/)
25. Preoţiuc-Pietro, D., Schwartz, H. A., Park, G., Eichstaedt, J., Kern, M., Ungar, L., and Shulman, E. (2016). Modelling valence and arousal in facebook posts. In Proceedings of the 7th workshop on computational approaches to subjectivity, sentiment and social media analysis, pages 9–15. [find](https://scholar.google.com/scholar?q=Preot%CC%A7iuc%2DPietro%2C+D%2E%2C+Schwartz%2C+H%2E+A%2E%2C+Park%2C+G%2E%2C+Eichstaedt%2C+J%2E%2C+Kern%2C+M%2E%2C+Ungar%2C+L%2E%2C+and+Shulman%2C+E%2E+%282016%29%2E+Modelling+valence+and+arousal+in+facebook+posts%2E+In+Proceedings+of+the+7th+workshop+on+com)
26. Raffel, C., Shazeer, N. M., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv, abs/1910.10683. ). [↗](https://arxiv.org/abs/1910.10683)
27. Russell, J. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39:1161– 1178, 12. [↗](https://doi.org/10.1037/h0077714)
28. Sangwan, S., Akhtar, M. S., Behera, P., and Ekbal, A. (2020). I didn’t mean what i wrote! exploring multimodality for sarcasm detection. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–8. [find](https://scholar.google.com/scholar?q=Sangwan%2C+S%2E%2C+Akhtar%2C+M%2E+S%2E%2C+Behera%2C+P%2E%2C+and+Ekbal%2C+A%2E+%282020%29%2E+I+didn%E2%80%99t+mean+what+i+wrote%21+exploring+multimodality+for+sarcasm+detection%2E+In+2020+International+Joint+Conference+on+Neural+Networks+%28IJCN)
29. Schifanella, R., de Juan, P., Tetreault, J., and Cao, L. (2016). Detecting sarcasm in multimodal social platforms. Proceedings of the 24th ACM international conference on Multimedia, Oct. [find](https://scholar.google.com/scholar?q=Schifanella%2C+R%2E%2C+de+Juan%2C+P%2E%2C+Tetreault%2C+J%2E%2C+and+Cao%2C+L%2E+%282016%29%2E+Detecting+sarcasm+in+multimodal+social+platforms%2E+Proceedings+of+the+24th+ACM+international+conference+on+Multimedia%2C+Oct%2E) [our card](https://md.research.elamur.ai/free/en/papers/schifanella-2016-detecting-sarcasm-multimodal-social-platforms/)
30. Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin. [↗](https://doi.org/10.2307/3001968)
31. Zafeiriou, S., Kollias, D., Nicolaou, M., Papaioannou, A., Zhao, G., and Kotsia, I. (2017). Aff-wild: Valence and arousal ‘in-the-wild’ challenge. pages 1980–1987, 07. [find](https://scholar.google.com/scholar?q=Zafeiriou%2C+S%2E%2C+Kollias%2C+D%2E%2C+Nicolaou%2C+M%2E%2C+Papaioannou%2C+A%2E%2C+Zhao%2C+G%2E%2C+and+Kotsia%2C+I%2E+%282017%29%2E+Aff%2Dwild%3A+Valence+and+arousal+%E2%80%98in%2Dthe%2Dwild%E2%80%99+challenge%2E+pages+1980%E2%80%931987%2C+07%2E)
32. Zahiri, S. M. and Choi, J. D. (2017). Emotion detection on TV show transcripts with sequence-based convolutional neural networks. CoRR, abs/1708.04299. [↗](https://arxiv.org/abs/1708.04299)
33. Zvolenszky, Z. (2012). A gricean rearrangement of epithets. [find](https://scholar.google.com/scholar?q=Zvolenszky%2C+Z%2E+%282012%29%2E+A+gricean+rearrangement+of+epithets%2E)
34. Language Resource References Castro, Santiago and Hazarika, Devamanyu and Pérez-Rosas, Verónica and Zimmermann, Roger and Mihalcea, Rada and Poria, Soujanya. (2019). Towards Multimodal Sarcasm Detection (An Obviously Perfect Paper). [find](https://scholar.google.com/scholar?q=Language+Resource+References+Castro%2C+Santiago+and+Hazarika%2C+Devamanyu+and+Pe%CC%81rez%2DRosas%2C+Vero%CC%81nica+and+Zimmermann%2C+Roger+and+Mihalcea%2C+Rada+and+Poria%2C+Soujanya%2E+%282019%29%2E+Towards+Multimodal+Sarcasm+Detec) [our card](https://md.research.elamur.ai/free/en/papers/castro-2019-towards-multimodal-sarcasm-detection-obviously/)
35. Chauhan, Dushyant Singh and Dhanush, SR and Ekbal, Asif and Bhattacharyya, Pushpak. (2020). Sentiment and Emotion help Sarcasm? A Multi-task Learning Framework for Multi-Modal Sarcasm, Sentiment and Emotion Analysis. [find](https://scholar.google.com/scholar?q=Chauhan%2C+Dushyant+Singh+and+Dhanush%2C+SR+and+Ekbal%2C+Asif+and+Bhattacharyya%2C+Pushpak%2E+%282020%29%2E+Sentiment+and+Emotion+help+Sarcasm%3F+A+Multi%2Dtask+Learning+Framework+for+Multi%2DModal+Sarcasm%2C+Sentiment+and+E) [our card](https://md.research.elamur.ai/free/en/papers/chauhan-2020-sentiment-emotion-help-sarcasm-multi/)