---
title: "All-in-One — paper by Chauhan et al. (2020)"
description: "The authors argue that these properties depend on each other. 25 references, cited by 9 papers here."
authors: ["Dushyant Singh Chauhan", "Dhanush S R", "Asif Ekbal", "Pushpak Bhattacharyya"]
year: 2020
venue: "Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing"
source: "https://aclanthology.org/2020.aacl-main.31/"
doi: "10.18653/v1/2020.aacl-main.31"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chauhan-2020-all-one-deep-attentive-multi/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chauhan-2020-all-one-deep-attentive-multi/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/) / All-in-One: A Deep Attentive Multi-task Learning Framework for Humour, Sarcasm, Offensive, Motivation, and Sentiment on Memes

# All-in-One: A Deep Attentive Multi-task Learning Framework for Humour, Sarcasm, Offensive, Motivation, and Sentiment on Memes

**Authors:** [Dushyant Singh Chauhan](https://md.research.elamur.ai/free/en/authors/dushyant-singh-chauhan/), [Dhanush S R](https://md.research.elamur.ai/free/en/authors/dhanush-s-r/), [Asif Ekbal](https://md.research.elamur.ai/free/en/authors/asif-ekbal/) and [Pushpak Bhattacharyya](https://md.research.elamur.ai/free/en/authors/pushpak-bhattacharyya/) · **Year:** 2020 · **Venue:** [Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing](https://md.research.elamur.ai/free/en/venues/aacl/) ([AACL 2020](https://md.research.elamur.ai/free/en/venues/aacl/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.aacl-main.31/) · [PDF](https://aclanthology.org/2020.aacl-main.31.pdf) · [DOI 10.18653/v1/2020.aacl-main.31](https://doi.org/10.18653/v1/2020.aacl-main.31) · [OpenAlex](https://openalex.org/W3115214080)

**Citations:** 29 citations (OpenAlex, 2026-10-14) · 9 in our library, of the 141,840 papers whose bibliographies we read

## Summary
Memes mix an image with overlaid text and carry several overlapping affective qualities at once. A meme can be funny, sarcastic, offensive and motivational, and it also has an overall sentiment. The authors argue that these properties depend on each other. For example, a sarcastic meme with negative sentiment is likely to also be offensive. They therefore treat all five as one joint problem. Their system is a multi-modal, multi-task network. It combines BERT embeddings of the meme's OCR text (manually corrected) with ResNet-152 image features. On top of this shared encoding sit two attention-like components. The Inter-task Relationship Module (iTRM) learns how much each task supports the others. The Inter-class Relationship Module (iCRM) models links between individual labels across tasks. The outputs of both modules are concatenated and shared by all task heads. The model is evaluated on the Memotion dataset from SemEval 2020. The authors report that it beats both the official baseline and the SemEval 2020 winning system, and that joint training outperforms single-task training. The work shows that modelling dependencies between affect labels pays off on noisy, multi-modal social media content.

## Key points
- Data: the Memotion Analysis dataset (SemEval 2020), evaluated in its three task settings: Task-A, Task-B and Task-C, where Task-C covers graded intensity labels.
- Inputs:
  - Text: 768-dimensional BERT word vectors; the longest text has 189 words.
  - Image: 2048-dimensional avgpool features from ImageNet-pretrained ResNet-152, computed on images resized to 224×224.
- iCRM first collapses each task's classes into a binary yes/no decision. It then backpropagates an auxiliary loss on these confidence scores, which makes the classification hierarchical.
- iTRM attention heatmaps put high weight on the sarcasm–offensive pair and low weight on humour–sarcasm. The authors read this as evidence that humour and sarcasm look alike on the surface but do not help each other.
- Stated limitations: the dataset is small and heavily skewed, and recurring meme templates make it hard to represent memes well.

## Interesting ideas
- Before predicting graded labels, the model first predicts a coarse binary version of each label and trains on it with its own auxiliary loss; this builds a cheap hierarchy that steers the intermediate layers.
- Pairwise class-interaction matrices (humour×sarcasm, sarcasm×offensive, and so on) make visible which label combinations reinforce each other.
- Learned inter-task attention weights serve as a diagnostic showing which auxiliary task actually helps which target, rather than assuming every extra task is useful.
- The error analysis lists concrete items where joint training succeeds and single-task training fails, which grounds the claimed benefit in specific examples.
- The paper observes that the text alone may be ambiguous between sarcastic and motivational, and only the second source of evidence (the image) settles the reading.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (frames): The irony detector in prag could first commit to an explainable "ironic or not" decision and only then grade the strength or type of irony, with each level checked separately.
- **I2** (frames): Explicit co-occurrence tables between speech-act, sentiment and irony labels could serve as inspectable constraints, for example flagging an utterance tagged both "sincere praise" and "sarcastic".
- **I3** (learn): When adding auxiliary training signals such as LLM-teacher labels or extra annotation layers, weights of this kind could show which signals to keep and which to drop as noise.
- **I4** (learn): Every new training component could ship with a small list of instances it fixes and instances it breaks, used next to negative controls.
- **I5** (world): The world model of a text could act as that second source, using the states and events already built to decide whether a literal sentence should be read as ironic.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Indian Institute of Technology Bombay](https://md.research.elamur.ai/free/en/universities/indian-institute-of-technology-bombay/) — University, Mumbai, India · 22 works · 2,018 citations
- [Indian Institute of Technology Patna](https://md.research.elamur.ai/free/en/universities/indian-institute-of-technology-patna/) — University, Patna, India · 10 works · 262 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 798 works · [💭 Social Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/social-psychology/) · 129 works

🧭 Topics (1): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 608 works

## Cite

Cite as: Dushyant Singh Chauhan, Dhanush S R, Asif Ekbal and Pushpak Bhattacharyya. 2020. All-in-One: A Deep Attentive Multi-task Learning Framework for Humour, Sarcasm, Offensive, Motivation, and Sentiment on Memes. Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chauhan2020all,
  title = {{All-in-One: A Deep Attentive Multi-task Learning Framework for Humour, Sarcasm, Offensive, Motivation, and Sentiment on Memes}},
  author = {Dushyant Singh Chauhan, Dhanush S R, Asif Ekbal and Pushpak Bhattacharyya},
  booktitle = {Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.aacl-main.31},
  url = {https://doi.org/10.18653/v1/2020.aacl-main.31}
}
```

- [← previous A Transformer Approach to Contextual Sarcasm Detection in Twitter](https://md.research.elamur.ai/free/en/papers/gregory-2020-transformer-approach-contextual-sarcasm-detection/)
- [next → C-Net: Contextual Network for Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/jena-2020-c-net-contextual-network-sarcasm/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/)

## References

25 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Md Shad Akhtar, Dushyant Chauhan, Deepanway Ghosal, Soujanya Poria, Asif Ekbal, and Pushpak Bhattacharyya. 2019. Multi-task learning for multimodal emotion recognition and sentiment analysis. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 370–379, Minneapolis, Minnesota. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Md+Shad+Akhtar%2C+Dushyant+Chauhan%2C+Deepanway+Ghosal%2C+Soujanya+Poria%2C+Asif+Ekbal%2C+and+Pushpak+Bhattacharyya%2E+2019%2E+Multi%2Dtask+learning+for+multimodal+emotion+recognition+and+sentiment+analysis%2E+In+Proce)
2. Rohan Badlani, Nishit Asnani, and Manan Rai. 2019. Disambiguating sentiment: An ensemble of humour, sarcasm, and hate speech features for sentiment classification. W-NUT 2019, page 337. [find](https://scholar.google.com/scholar?q=Rohan+Badlani%2C+Nishit+Asnani%2C+and+Manan+Rai%2E+2019%2E+Disambiguating+sentiment%3A+An+ensemble+of+humour%2C+sarcasm%2C+and+hate+speech+features+for+sentiment+classification%2E+W%2DNUT+2019%2C+page+337%2E)
3. Francesco Barbieri and Horacio Saggion. 2014. Automatic detection of irony and humour in twitter. In ICCC, pages 155–162. [find](https://scholar.google.com/scholar?q=Francesco+Barbieri+and+Horacio+Saggion%2E+2014%2E+Automatic+detection+of+irony+and+humour+in+twitter%2E+In+ICCC%2C+pages+155%E2%80%93162%2E)
4. Dario Bertero and Pascale Fung. 2016. Multimodal deep neural nets for detecting humor in tv sitcoms. In 2016 IEEE Spoken Language Technology Workshop (SLT), pages 383–390. IEEE. [find](https://scholar.google.com/scholar?q=Dario+Bertero+and+Pascale+Fung%2E+2016%2E+Multimodal+deep+neural+nets+for+detecting+humor+in+tv+sitcoms%2E+In+2016+IEEE+Spoken+Language+Technology+Workshop+%28SLT%29%2C+pages+383%E2%80%93390%2E+IEEE%2E)
5. Paula Carvalho, Luı́s Sarmento, Mário J Silva, and Eugénio De Oliveira. 2009. Clues for detecting irony in user-generated contents: oh...!! it’s” so easy”;-. In Proceedings of the 1st international CIKM workshop on Topic-sentiment analysis for mass opinion, pages 53–56. [find](https://scholar.google.com/scholar?q=Paula+Carvalho%2C+Lu%C4%B1%CC%81s+Sarmento%2C+Ma%CC%81rio+J+Silva%2C+and+Euge%CC%81nio+De+Oliveira%2E+2009%2E+Clues+for+detecting+irony+in+user%2Dgenerated+contents%3A+oh%2E%2E%2E%21%21+it%E2%80%99s%E2%80%9D+so+easy%E2%80%9D%3B%2D%2E+In+Proceedings+of+the+1st+international+)
6. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards multimodal sarcasm detection (an obviously perfect paper). arXiv preprint arXiv:1906.01815. [↗](https://arxiv.org/abs/1906.01815) [our card](https://md.research.elamur.ai/free/en/papers/castro-2019-towards-multimodal-sarcasm-detection-obviously/)
7. Dushyant Singh Chauhan, Md Shad Akhtar, Asif Ekbal, and Pushpak Bhattacharyya. 2019. Context-aware interactive attention for multi-modal sentiment and emotion analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5651–5661, Hong Kong, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dushyant+Singh+Chauhan%2C+Md+Shad+Akhtar%2C+Asif+Ekbal%2C+and+Pushpak+Bhattacharyya%2E+2019%2E+Context%2Daware+interactive+attention+for+multi%2Dmodal+sentiment+and+emotion+analysis%2E+In+Proceedings+of+the+2019+Conf)
8. Dushyant Singh Chauhan, Dhanush S R, Asif Ekbal, and Pushpak Bhattacharyya. 2020. Sentiment and emotion help sarcasm? a multi-task learning framework for multi-modal sarcasm, sentiment and emotion analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4351–4360, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.401) [our card](https://md.research.elamur.ai/free/en/papers/chauhan-2020-sentiment-emotion-help-sarcasm-multi/)
9. Ying Chen, Yilu Zhou, Sencun Zhu, and Heng Xu. 2012. Detecting offensive language in social media to protect adolescent online safety. In 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing, pages 71–80. IEEE. [find](https://scholar.google.com/scholar?q=Ying+Chen%2C+Yilu+Zhou%2C+Sencun+Zhu%2C+and+Heng+Xu%2E+2012%2E+Detecting+offensive+language+in+social+media+to+protect+adolescent+online+safety%2E+In+2012+International+Conference+on+Privacy%2C+Security%2C+Risk+and+T)
10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
11. Deepanway Ghosal, Md Shad Akhtar, Dushyant Singh Chauhan, Soujanya Poria, Asif Ekbal, and Pushpak Bhattacharyya. 2018. Contextual inter-modal attention for multi-modal sentiment analysis. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3454–3466, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Deepanway+Ghosal%2C+Md+Shad+Akhtar%2C+Dushyant+Singh+Chauhan%2C+Soujanya+Poria%2C+Asif+Ekbal%2C+and+Pushpak+Bhattacharyya%2E+2018%2E+Contextual+inter%2Dmodal+attention+for+multi%2Dmodal+sentiment+analysis%2E+In+Proceedin)
12. Raul Gomez, Jaume Gibert, Lluis Gomez, and Dimos-thenis Karatzas. 2020. Exploring hate speech detection in multimodal publications. In The IEEE Winter Conference on Applications of Computer Vision, pages 1470–1478. [find](https://scholar.google.com/scholar?q=Raul+Gomez%2C+Jaume+Gibert%2C+Lluis+Gomez%2C+and+Dimos%2Dthenis+Karatzas%2E+2020%2E+Exploring+hate+speech+detection+in+multimodal+publications%2E+In+The+IEEE+Winter+Conference+on+Applications+of+Computer+Vision%2C+pa)
13. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778. [↗](https://doi.org/10.48550/arxiv.1512.03385)
14. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 757–762. [our card](https://md.research.elamur.ai/free/en/papers/joshi-2015-harnessing-context-incongruity-sarcasm-detection/)
15. Sijie Mai, Haifeng Hu, and Songlong Xing. 2019. Divide, conquer and combine: Hierarchical feature fusion network with local and global perspectives for multimodal affective computing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 481–492. [find](https://scholar.google.com/scholar?q=Sijie+Mai%2C+Haifeng+Hu%2C+and+Songlong+Xing%2E+2019%2E+Divide%2C+conquer+and+combine%3A+Hierarchical+feature+fusion+network+with+local+and+global+perspectives+for+multimodal+affective+computing%2E+In+Proceedings+o)
16. Rada Mihalcea and Stephen Pulman. 2007. Characterizing humour: An exploration of features in humorous texts. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 337–347. Springer. [find](https://scholar.google.com/scholar?q=Rada+Mihalcea+and+Stephen+Pulman%2E+2007%2E+Characterizing+humour%3A+An+exploration+of+features+in+humorous+texts%2E+In+International+Conference+on+Intelligent+Text+Processing+and+Computational+Linguistics%2C+p)
17. Amir H Razavi, Diana Inkpen, Sasha Uritsky, and Stan Matwin. 2010. Offensive language detection using multi-level classification. In Canadian Conference on Artificial Intelligence, pages 16–27. Springer. [find](https://scholar.google.com/scholar?q=Amir+H+Razavi%2C+Diana+Inkpen%2C+Sasha+Uritsky%2C+and+Stan+Matwin%2E+2010%2E+Offensive+language+detection+using+multi%2Dlevel+classification%2E+In+Canadian+Conference+on+Artificial+Intelligence%2C+pages+16%E2%80%9327%2E+Spring)
18. Rossano Schifanella, Paloma de Juan, Joel Tetreault, and Liangliang Cao. 2016. Detecting sarcasm in multimodal social platforms. In Proceedings of the 24th ACM international conference on Multimedia, pages 1136–1145. [↗](https://doi.org/10.1145/2964284.2964321) [our card](https://md.research.elamur.ai/free/en/papers/schifanella-2016-detecting-sarcasm-multimodal-social-platforms/)
19. Chhavi Sharma, Deepesh Bhageria, William Paka, Scott, Srinivas P Y K L, Amitava Das, Tanmoy Chakraborty, and Björn Gambäck. 2020. SemEval-2020 Task 8: Memotion Analysis-The Visuo-Lingual Metaphor! In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2008.03781) [our card](https://md.research.elamur.ai/free/en/papers/sharma-2020-semeval-2020-task-8-memotion/)
20. Jonas Sjöbergh and Kenji Araki. 2007. Recognizing humor without recognizing meaning. In International Workshop on Fuzzy Logic and Applications, pages 469–476. Springer. [find](https://scholar.google.com/scholar?q=Jonas+Sjo%CC%88bergh+and+Kenji+Araki%2E+2007%2E+Recognizing+humor+without+recognizing+meaning%2E+In+International+Workshop+on+Fuzzy+Logic+and+Applications%2C+pages+469%E2%80%93476%2E+Springer%2E)
21. Steve Durairaj Swamy, Shubham Laddha, Basil Abdus-salam, Debayan Datta, and Anupam Jamatia. 2020. Nit-agartala-nlp-team at semeval-2020 task 8: Building multimodal classifiers to tackle internet humor. arXiv preprint arXiv:2005.06943. [↗](https://arxiv.org/abs/2005.06943)
22. Patrycja Swieczkowska, Rafal Rzepka, and Kenji Araki. 2020. Stepwise noise elimination for better motivational and advisory texts classification. Journal of Advanced Computational Intelligence and Intelligent Informatics, 24(1):156–168. [↗](https://doi.org/10.20965/jaciii.2020.p0156)
23. Tony Veale and Yanfen Hao. 2010. Detecting ironic intent in creative comparisons. In ECAI, volume 215, pages 765–770. [↗](https://doi.org/10.3233/978-1-60750-606-5-765)
24. Hao Yang, Yao Deng, Minghan Wang, Ying Qin, and Shiliang Sun. 2020. Humor detection based on paragraph decomposition and bert fine-tuning. [find](https://scholar.google.com/scholar?q=Hao+Yang%2C+Yao+Deng%2C+Minghan+Wang%2C+Ying+Qin%2C+and+Shiliang+Sun%2E+2020%2E+Humor+detection+based+on+paragraph+decomposition+and+bert+fine%2Dtuning%2E)
25. Zixiaofan Yang, Lin Ai, and Julia Hirschberg. 2019. Multimodal indicators of humor in videos. In 2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), pages 538–543. IEEE. [find](https://scholar.google.com/scholar?q=Zixiaofan+Yang%2C+Lin+Ai%2C+and+Julia+Hirschberg%2E+2019%2E+Multimodal+indicators+of+humor+in+videos%2E+In+2019+IEEE+Conference+on+Multimedia+Information+Processing+and+Retrieval+%28MIPR%29%2C+pages+538%E2%80%93543%2E+IEEE%2E)

*Updated 24 October 2026*