---
title: "Sarcasm in Sight and Sound — paper by Bhosale et al. (2023)"
description: "Second, the original data has very uneven coverage of sarcasm types, so the authors… 3.39 points. 50 references, cited by 3 papers here."
authors: ["Swapnil Bhosale", "Abhra Chaudhuri", "Alex Lee Robert Williams", "Divyank Tiwari", "Anjan Dutta", "Xiatian Zhu", "Pushpak Bhattacharyya", "Diptesh Kanojia"]
year: 2023
venue: "arXiv:2310.01430"
source: "https://arxiv.org/abs/2310.01430"
doi: "10.48550/arxiv.2310.01430"
arxiv: "2310.01430"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bhosale-2023-sarcasm-sight-sound-benchmarking-expansion/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bhosale-2023-sarcasm-sight-sound-benchmarking-expansion/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/13/) / Sarcasm in Sight and Sound: Benchmarking and Expansion to Improve Multimodal Sarcasm Detection

# Sarcasm in Sight and Sound: Benchmarking and Expansion to Improve Multimodal Sarcasm Detection

**Authors:** Swapnil Bhosale, Abhra Chaudhuri, Alex Lee Robert Williams, [Divyank Tiwari](https://md.research.elamur.ai/free/en/authors/divyank-tiwari/), Anjan Dutta, Xiatian Zhu, [Pushpak Bhattacharyya](https://md.research.elamur.ai/free/en/authors/pushpak-bhattacharyya/) and [Diptesh Kanojia](https://md.research.elamur.ai/free/en/authors/diptesh-kanojia/) · **Year:** 2023 · **Venue:** arXiv:2310.01430 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.01430) · [PDF](https://arxiv.org/pdf/2310.01430) · [DOI 10.48550/arxiv.2310.01430](https://doi.org/10.48550/arxiv.2310.01430) · [arXiv 2310.01430](https://arxiv.org/abs/2310.01430) · [OpenAlex](https://openalex.org/W4387355348)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/bhosale-2023-sarcasm-sight-sound-benchmarking-expansion/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Sarcasm often cannot be recognized from words alone. Tone of voice and facial expression may be what turns a sentence ironic. This paper revisits MUStARD++, a sarcasm dataset of TV-show clips with text, audio and video. It makes two contributions. First, it re-benchmarks the dataset with modern pretrained encoders in place of the hand-crafted features used before (MFCC, spectrograms, OpenSMILE prosody, ResNet-152 frames). Video is encoded with ViFi-CLIP, speech with Wav2vec 2.0, and the modality features are joined through an existing fusion setup. This raises macro-F1 by more than 2 points over the earlier benchmark. Second, the original data has very uneven coverage of sarcasm types, so the authors build MUStARD++ Balanced. It adds clips from House MD, a show not used in earlier versions, labelled by three annotators with substantial agreement. Training and testing with the new clips adds about 2.4 more macro-F1 points. A controlled experiment shows the gain comes from diversity rather than from simply having more data. The work matters because it sets a stronger baseline and gives a more varied resource for multimodal irony research.

## Key points
- Dataset sizes:
  - MUStARD has 6,365 clips, of which 345 are sarcastic.
  - MUStARD++ has 1,202 clips (601 sarcastic, 601 not).
  - MUStARD++ Balanced has 1,365 clips (691 sarcastic, 674 not), with the new material from House MD.
- Sarcasm types annotated:
  - Propositional (PRO) needs outside context.
  - Illocutionary (ILL) is carried by voice or gesture.
  - Embedded (EMB) has the incongruity inside the text.
  - Like-prefixed (LIKE) is rare; there were too few samples to test it separately.
- Encoders: ViFi-CLIP for video and Wav2vec 2.0 for audio.
  - Better audio and video encoders add 3.39 points with dialogue context and 3.89 points without it.
  - The earlier MUStARD++ trimodal benchmark reached an F1 of about 70.2.
- Distribution-shift test: some of the 952 original training clips were swapped for extension clips, keeping training size fixed (5 runs, majority vote). Class-wise accuracy rose:

  | Class | Before | After |
  |---|---|---|
  | PRO | 0.705 | 0.824 |
  | ILL | 0.484 | 0.545 |
  | EMB | 0.696 | 0.788 |
  | Non-sarcastic | 0.620 | 0.720 |

- Error analysis:
  - Illocutionary sarcasm is the hardest type.
  - Most misclassified items are very short clips (under 2 seconds, e.g. "Nooo"), which carry too little context.

## Interesting ideas
- To show that new data adds diversity and not just volume, the authors swap new samples into the training set without changing its size, then compare on a fixed, type-stratified test set
- The four-way sarcasm taxonomy separates irony visible in the text (embedded), irony that needs situational context (propositional) and irony carried only by delivery (illocutionary)
- Balancing the data by a finer category (sarcasm type) rather than only the binary label improved the minority types and also the non-sarcastic class
- Errors cluster on very short utterances that lack context, and the authors plan to flag or prune them

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): when adding silver data from a new genre, swap at equal training size and evaluate on a frozen stratified test, so the effect of novelty is separated from the effect of size.
- **I2** (frames): the irony module in prag could label which kind of evidence supports an ironic reading and explicitly mark illocutionary-type cases as undecidable from text, instead of guessing.
- **I3** (learn): balance training data by construction or phenomenon type (e.g. rare UD relations, speech-act kinds), not just overall counts, and track per-type accuracy.
- **I4** (answer): treat very short or context-poor inputs as a known low-confidence zone and have readers abstain or ask for context, rather than forcing an interpretation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Indian Institute of Technology Bombay](https://md.research.elamur.ai/free/en/universities/indian-institute-of-technology-bombay/) — University, Mumbai, India · 29 works · 2,104 citations
- [University of Surrey](https://md.research.elamur.ai/free/en/universities/university-of-surrey/) — University, Guildford, United Kingdom · 21 works · 677 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 870 works

🧭 Topics (2): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 972 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works

## Cite

Cite as: Swapnil Bhosale, Abhra Chaudhuri, Alex Lee Robert Williams, Divyank Tiwari, Anjan Dutta, Xiatian Zhu, Pushpak Bhattacharyya and Diptesh Kanojia. 2023. Sarcasm in Sight and Sound: Benchmarking and Expansion to Improve Multimodal Sarcasm Detection. arXiv:2310.01430.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{bhosale2023sarcasm,
  title = {{Sarcasm in Sight and Sound: Benchmarking and Expansion to Improve Multimodal Sarcasm Detection}},
  author = {Swapnil Bhosale, Abhra Chaudhuri, Alex Lee Robert Williams, Divyank Tiwari, Anjan Dutta, Xiatian Zhu, Pushpak Bhattacharyya and Diptesh Kanojia},
  year = {2023},
  doi = {10.48550/arxiv.2310.01430},
  eprint = {2310.01430},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.01430}
}
```

- [← previous ReDASPersuasion at ArAIEval Shared Task: Multilingual and Monolingual Models For Arabic Persuasion Detection](https://md.research.elamur.ai/free/en/papers/qachfar-2023-redaspersuasion-araieval-shared-task-multilingual/)
- [next → The Conversational Discourse Unit: Identification and Its Role in Conversational Turn-taking Management](https://md.research.elamur.ai/free/en/papers/hu-2023-conversational-discourse-unit-identification-its/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/13/)

## References

50 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. S. Castro, D. Hazarika, V. Pérez-Rosas, R. Zimmermann, R. Mihalcea, and S. Poria, “[Towards multimodal sarcasm detection (an _Obviously_ perfect paper)](https://md.research.elamur.ai/free/en/papers/castro-2019-towards-multimodal-sarcasm-detection-obviously/),” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
2. A. Ray, S. Mishra, A. Nunna, and P. Bhattacharyya, “[A multimodal corpus for emotion recognition in sarcasm](https://md.research.elamur.ai/free/en/papers/ray-2022-multimodal-corpus-emotion-recognition-sarcasm/),” in Proceedings of the Thirteenth Language Resources and Evaluation Conference. Marseille, France: European Language Resources Association, 2022.
3. M. Soleymani, D. Garcia, B. Jou, B. Schuller, S.-F. Chang, and M. Pantic, “A survey of multimodal sentiment analysis,” Image and Vision Computing, vol. 65, pp. 3–14, 2017. [↗](https://doi.org/10.1016/j.imavis.2017.08.003)
4. D. Kiela, H. Firooz, A. Mohan, V. Goswami, A. Singh, P. Ringshia, and D. Testuggine, “[The hateful memes challenge: Detecting hate speech in multimodal memes](https://md.research.elamur.ai/free/en/papers/kiela-2020-hateful-memes-challenge-detecting-hate/),” Advances in Neural Information Processing Systems, vol. 33, pp. 2611–2624, 2020. [↗](https://doi.org/10.48550/arxiv.2005.04790)
5. S. Suryawanshi, B. R. Chakravarthi, M. Arcan, and P. Buitelaar, “[Multimodal meme dataset (multioff) for identifying offensive content in image and text](https://md.research.elamur.ai/free/en/papers/suryawanshi-2020-multimodal-meme-dataset-multioff-identifying/),” in Proceedings of the second workshop on trolling, aggression and cyberbullying, 2020, pp. 32–41.
6. P. Guo, Z. Chen, Y. Li, and H. Liu, “Audio-visual fusion network based on conformer for multimodal emotion recognition,” in Artificial Intelligence: Second CAAI International Conference, CICAI 2022, Beijing, China, August 27–28, 2022, Revised Selected Papers, Part II. Springer, 2023, pp. 315–326. [find](https://scholar.google.com/scholar?q=P%2E+Guo%2C+Z%2E+Chen%2C+Y%2E+Li%2C+and+H%2E+Liu%2C+%E2%80%9CAudio%2Dvisual+fusion+network+based+on+conformer+for+multimodal+emotion+recognition%2C%E2%80%9D+in+Artificial+Intelligence%3A+Second+CAAI+International+Conference%2C+CICAI+2022%2C+B)
7. Y. Liu, S. Albanie, A. Nagrani, and A. Zisserman, “Use what you have: Video retrieval using representations from collaborative experts,” arXiv preprint arXiv:1907.13487, 2019. [↗](https://arxiv.org/abs/1907.13487)
8. B. McFee, C. Raffel, D. Liang, D. P. Ellis, M. McVicar, E. Battenberg, and O. Nieto, “librosa: Audio and music signal analysis in python.” in Proceedings of the 14th python in science conference, 2015. [find](https://scholar.google.com/scholar?q=B%2E+McFee%2C+C%2E+Raffel%2C+D%2E+Liang%2C+D%2E+P%2E+Ellis%2C+M%2E+McVicar%2C+E%2E+Battenberg%2C+and+O%2E+Nieto%2C+%E2%80%9Clibrosa%3A+Audio+and+music+signal+analysis+in+python%2E%E2%80%9D+in+Proceedings+of+the+14th+python+in+science+conference%2C+2015)
9. K. He, X. Zhang, S. Ren, and J. Sun, “[Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/),” in CVPR, June 2016. [↗](https://doi.org/10.48550/arxiv.1512.03385)
10. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “[Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/),” in ICML, 2021. [↗](https://doi.org/10.48550/arxiv.2103.00020)
11. H. Rasheed, M. U. Khattak, M. Maaz, S. Khan, and F. S. Khan, “Fine-tuned clip models are efficient video learners,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 6545–6554. [↗](https://doi.org/10.48550/arxiv.2212.03640)
12. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “[Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/),” in Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017. [Online]. Available: [https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) [↗](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)
13. A. Joshi, P. Bhattacharyya, and M. J. Carman, “[Automatic sarcasm detection: A survey](https://md.research.elamur.ai/free/en/papers/joshi-2016-automatic-sarcasm-detection-survey/),” ACM Comput. Surv., vol. 50, no. 5, sep 2017. [Online]. Available: https://doi.org/10.1145/3124420 [↗](https://doi.org/10.1145/3124420)
14. M. Bouazizi and T. Otsuki Ohtsuki, “A pattern-based approach for sarcasm detection on twitter,” IEEE Access, vol. 4, pp. 5477–5488, 2016. [↗](https://doi.org/10.1109/access.2016.2594194)
15. A. Reyes and P. Rosso, “[Mining subjective knowledge from customer reviews: A specific case of irony detection](https://md.research.elamur.ai/free/en/papers/reyes-2011-mining-subjective-knowledge-customer-reviews/),” in Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis (WASSA 2.011). Portland, Oregon: Association for Computational Linguistics, Jun. 2011, pp. 118–124. [Online]. Available: https://aclanthology.org/W11-1715 [↗](https://aclanthology.org/W11-1715/)
16. F. Barbieri, H. Saggion, and F. Ronzano, “[Modelling sarcasm in Twitter, a novel approach](https://md.research.elamur.ai/free/en/papers/barbieri-2014-modelling-sarcasm-twitter-novel-approach/),” in Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis. Baltimore, Maryland: Association for Computational Linguistics, Jun. 2014, pp. 50–58. [Online]. Available: https://aclanthology.org/W14-2609 [↗](https://aclanthology.org/W14-2609/)
17. C. Van Hee, E. Lefever, and V. Hoste, “[SemEval-2018 task 3: Irony detection in English tweets](https://md.research.elamur.ai/free/en/papers/hee-2018-semeval-2018-task-3-irony/),” in Proceedings of the 12th International Workshop on Semantic Evaluation. New Orleans, Louisiana: Association for Computational Linguistics, Jun. 2018, pp. 39–50. [Online]. Available: https://aclanthology.org/S18-1005 [↗](https://aclanthology.org/S18-1005/)
18. R. A. Potamias, G. Siolas, and A. G. Stafylopatis, “[A transformer-based approach to irony and sarcasm detection](https://md.research.elamur.ai/free/en/papers/potamias-2019-transformer-based-approach-irony-sarcasm/),” Neural Computing and Applications, vol. 32, no. 23, pp. 17 309–17 320, jun 2020. [Online]. Available: https://doi.org/10.1007%2Fs00521-020-05102-3 [↗](https://doi.org/10.1007/s00521-020-05102-3)
19. E. Riloff, A. Qadir, P. Surve, L. De Silva, N. Gilbert, and R. Huang, “[Sarcasm as contrast between a positive sentiment and negative situation](https://md.research.elamur.ai/free/en/papers/riloff-2013-sarcasm-as-contrast-between-positive/),” in Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Seattle, Washington, USA: Association for Computational Linguistics, Oct. 2013, pp. 704–714. [Online]. Available: https://aclanthology.org/D13-1066 [↗](https://aclanthology.org/D13-1066/)
20. T. Shangipour ataei, S. Javdan, and B. Minaei-Bidgoli, “[Applying transformers and aspect-based sentiment analysis approaches on sarcasm detection](https://md.research.elamur.ai/free/en/papers/ataei-2020-applying-transformers-aspect-based-sentiment/),” in Proceedings of the Second Workshop on Figurative Language Processing. Online: Association for Computational Linguistics, Jul. 2020, pp. 67–71. [Online]. Available: https://aclanthology.org/2020.figlang-1.9 [↗](https://aclanthology.org/2020.figlang-1.9/)
21. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “[Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/),” 2019.
22. D. Ma, S. Li, X. Zhang, and H. Wang, “Interactive attention networks for aspect-level sentiment classification,” in Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, 2017, pp. 4068–4074. [Online]. Available: https://doi.org/10.24963/ijcai.2017/568 [↗](https://doi.org/10.24963/ijcai.2017/568)
23. I. Abu Farha and W. Magdy, “[Benchmarking transformer-based language models for Arabic sentiment and sarcasm detection](https://md.research.elamur.ai/free/en/papers/farha-2021-benchmarking-transformer-based-language-models/),” in Proceedings of the Sixth Arabic Natural Language Processing Workshop. Kyiv, Ukraine (Virtual): Association for Computational Linguistics, Apr. 2021, pp. 21–31. [Online]. Available: https://aclanthology.org/2021.wanlp-1.3 [↗](https://aclanthology.org/2021.wanlp-1.3/)
24. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, “Unsupervised cross-lingual representation learning at scale,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, Jul. 2020, pp. 8440–8451. [Online]. Available: https://aclanthology.org/2020.acl-main.747 [↗](https://aclanthology.org/2020.acl-main.747/)
25. M. Abdul-Mageed, A. Elmadany, and E. M. B. Nagoudi, “[Arbert & marbert: Deep bidirectional transformers for arabic](https://md.research.elamur.ai/free/en/papers/abdul-mageed-2021-arbert-marbert-deep-bidirectional-transformers/),” 2021. [↗](https://doi.org/10.48550/arxiv.2101.01785)
26. J. Pennington, R. Socher, and C. Manning, “[GloVe: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/),” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar: Association for Computational Linguistics, Oct. 2014, pp. 1532–1543. [Online]. Available: https://aclanthology.org/D14-1162 [↗](https://aclanthology.org/D14-1162/)
27. P. Goel, R. Jain, A. Nayyar, S. Singhal, and M. Srivastava, “Sarcasm detection using deep learning and ensemble learning,” Multimedia Tools and Applications, vol. 81, no. 30, pp. 43 229–43 252, 2022. [↗](https://doi.org/10.1007/s11042-022-12930-z)
28. M. Khodak, N. Saunshi, and K. Vodrahalli, “[A large self-annotated corpus for sarcasm](https://md.research.elamur.ai/free/en/papers/khodak-2018-large-self-annotated-corpus-sarcasm/),” 2018.
29. V. Pérez-Rosas, R. Mihalcea, and L.-P. Morency, “Utterance-level multimodal sentiment analysis,” in Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2013, pp. 973–982. [find](https://scholar.google.com/scholar?q=V%2E+P%C3%A9rez%2DRosas%2C+R%2E+Mihalcea%2C+and+L%2E%2DP%2E+Morency%2C+%E2%80%9CUtterance%2Dlevel+multimodal+sentiment+analysis%2C%E2%80%9D+in+Proceedings+of+the+51st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+L)
30. A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency, “Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos,” arXiv preprint arXiv:1606.06259, 2016. [↗](https://arxiv.org/abs/1606.06259)
31. C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,” Language resources and evaluation, vol. 42, no. 4, pp. 335–359, 2008. [↗](https://doi.org/10.1007/s10579-008-9076-6)
32. S. Poria, E. Cambria, D. Hazarika, N. Majumder, A. Zadeh, and L.-P. Morency, “[Context-dependent sentiment analysis in user-generated videos](https://md.research.elamur.ai/free/en/papers/poria-2017-context-dependent-sentiment-analysis-user/),” in Proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: Long papers), 2017, pp. 873–883.
33. N. Majumder, D. Hazarika, A. Gelbukh, E. Cambria, and S. Poria, “Multimodal sentiment analysis using hierarchical fusion with context modeling,” Knowledge-based systems, vol. 161, pp. 124–133, 2018. [↗](https://doi.org/10.1016/j.knosys.2018.07.041)
34. D. Hazarika, S. Poria, A. Zadeh, E. Cambria, L.-P. Morency, and R. Zimmermann, “[Conversational memory network for emotion recognition in dyadic dialogue videos](https://md.research.elamur.ai/free/en/papers/hazarika-2018-conversational-memory-network-emotion-recognition/),” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). New Orleans, Louisiana: Association for Computational Linguistics, Jun. 2018, pp. 2122–2132. [Online]. Available: https://aclanthology.org/N18-1193 [↗](https://aclanthology.org/N18-1193/)
35. Y. Wang, Y. Shen, Z. Liu, P. P. Liang, A. Zadeh, and L.-P. Morency, “Words can shift: Dynamically adjusting word representations using nonverbal behaviors,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, 2019, pp. 7216–7223. [find](https://scholar.google.com/scholar?q=Y%2E+Wang%2C+Y%2E+Shen%2C+Z%2E+Liu%2C+P%2E+P%2E+Liang%2C+A%2E+Zadeh%2C+and+L%2E%2DP%2E+Morency%2C+%E2%80%9CWords+can+shift%3A+Dynamically+adjusting+word+representations+using+nonverbal+behaviors%2C%E2%80%9D+in+Proceedings+of+the+AAAI+Conference+on+Ar)
36. H. Pham, P. P. Liang, T. Manzini, L.-P. Morency, and B. Póczos, “Found in translation: Learning robust joint representations by cyclic translations between modalities,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, 2019, pp. 6892–6899. [find](https://scholar.google.com/scholar?q=H%2E+Pham%2C+P%2E+P%2E+Liang%2C+T%2E+Manzini%2C+L%2E%2DP%2E+Morency%2C+and+B%2E+P%C3%B3czos%2C+%E2%80%9CFound+in+translation%3A+Learning+robust+joint+representations+by+cyclic+translations+between+modalities%2C%E2%80%9D+in+Proceedings+of+the+AAAI+Conf)
37. P. P. Liang, Z. Liu, A. B. Zadeh, and L.-P. Morency, “Multimodal language analysis with recurrent multistage fusion,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 150–161. [↗](https://doi.org/10.48550/arxiv.1808.03920)
38. Y.-H. H. Tsai, P. P. Liang, A. Zadeh, L.-P. Morency, and R. Salakhutdinov, “Learning factorized multimodal representations,” arXiv preprint arXiv:1806.06176, 2018. [↗](https://arxiv.org/abs/1806.06176)
39. S. Sahay, E. Okur, S. H Kumar, and L. Nachman, “Low rank fusion based transformers for multimodal sequences,” in Second Grand-Challenge and Workshop on Multimodal Language (Challenge-HML). Seattle, USA: Association for Computational Linguistics, Jul. 2020, pp. 29–34. [Online]. Available: https://aclanthology.org/2020.challengehml-1.4 [↗](https://aclanthology.org/2020.challengehml-1.4/)
40. Y.-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, and R. Salakhutdinov, “[Multimodal transformer for unaligned multimodal language sequences](https://md.research.elamur.ai/free/en/papers/tsai-2019-multimodal-transformer-unaligned-multimodal-language/),” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 6558–6569. [Online]. Available: https://aclanthology.org/P19-1656 [↗](https://aclanthology.org/P19-1656/)
41. Z. Liu, Y. Shen, V. B. Lakshminarasimhan, P. P. Liang, A. Bagher Zadeh, and L.-P. Morency, “[Efficient low-rank multimodal fusion with modality-specific factors](https://md.research.elamur.ai/free/en/papers/liu-2018-efficient-low-rank-multimodal-fusion/),” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 2247–2256. [Online]. Available: https://aclanthology.org/P18-1209 [↗](https://aclanthology.org/P18-1209/)
42. A. Bhat and A. Chauhan, “Multimodal sarcasm detection: A survey,” in 2022 IEEE Delhi Section Conference (DELCON), 2022, pp. 1–7. [find](https://scholar.google.com/scholar?q=A%2E+Bhat+and+A%2E+Chauhan%2C+%E2%80%9CMultimodal+sarcasm+detection%3A+A+survey%2C%E2%80%9D+in+2022+IEEE+Delhi+Section+Conference+%28DELCON%29%2C+2022%2C+pp%2E+1%E2%80%937%2E)
43. S. Castro, D. Hazarika, V. Pérez-Rosas, R. Zimmermann, R. Mihalcea, and S. Poria, “[Towards multimodal sarcasm detection (an _obviously_ perfect paper)](https://md.research.elamur.ai/free/en/papers/castro-2019-towards-multimodal-sarcasm-detection-obviously/),” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Florence, Italy: Association for Computational Linguistics, 7 2019.
44. A. Ray, S. Mishra, A. Nunna, and P. Bhattacharyya, “[A multimodal corpus for emotion recognition in sarcasm](https://md.research.elamur.ai/free/en/papers/ray-2022-multimodal-corpus-emotion-recognition-sarcasm/),” in Proceedings of the Thirteenth Language Resources and Evaluation Conference. Marseille, France: European Language Resources Association, Jun. 2022, pp. 6992–7003. [Online]. Available: https://aclanthology.org/2022.lrec-1.756 [↗](https://aclanthology.org/2022.lrec-1.756/)
45. P. P. Liang, Y. Lyu, X. Fan, Z. Wu, Y. Cheng, J. Wu, L. Y. Chen, P. Wu, M. A. Lee, Y. Zhu et al., “Multibench: Multiscale benchmarks for multimodal representation learning,” in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. [↗](https://doi.org/10.48550/arxiv.2107.07502)
46. L.-W. Chen and A. Rudnicky, “Exploring wav2vec 2.0 fine tuning for improved speech emotion recognition,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5. [find](https://scholar.google.com/scholar?q=L%2E%2DW%2E+Chen+and+A%2E+Rudnicky%2C+%E2%80%9CExploring+wav2vec+2%2E0+fine+tuning+for+improved+speech+emotion+recognition%2C%E2%80%9D+in+ICASSP+2023%2D2023+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28I)
47. D. S. Chauhan, D. S R, A. Ekbal, and P. Bhattacharyya, “[Sentiment and emotion help sarcasm? a multi-task learning framework for multi-modal sarcasm, sentiment and emotion analysis](https://md.research.elamur.ai/free/en/papers/chauhan-2020-sentiment-emotion-help-sarcasm-multi/),” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
48. R. W. Picard, “Affective computing mit press,” Cambridge, Massachsusetts, p. 2, 1997. [find](https://scholar.google.com/scholar?q=R%2E+W%2E+Picard%2C+%E2%80%9CAffective+computing+mit+press%2C%E2%80%9D+Cambridge%2C+Massachsusetts%2C+p%2E+2%2C+1997%2E)
49. S. Gururangan, A. Marasović, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith, “Don’t stop pretraining: Adapt language models to domains and tasks,” arXiv preprint arXiv:2004.10964, 2020. [↗](https://arxiv.org/abs/2004.10964)
50. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “[Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/),” arXiv preprint arXiv:1910.13461, 2019. [↗](https://arxiv.org/abs/1910.13461)

*Updated 1 November 2026*