---
title: "linus@CHiPSAL 2026 — paper by Regmi et al. (2026)"
description: "Automated moderation of memes is difficult for low-resource, code-mixed languages like Nepali. 19 references, cited by 2 papers here."
authors: ["Sunil Regmi", "Bipesh Subedi", "Saugat Singh", "Suman Shrestha"]
year: 2026
venue: "Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)"
source: "https://aclanthology.org/2026.chipsal-1.27/"
doi: "10.63317/3t7nocwwhq8r"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/regmi-2026-linus-chipsal-2026-multimodal-hate/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/regmi-2026-linus-chipsal-2026-multimodal-hate/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/8/) / linus@CHiPSAL 2026: Multimodal Hate Speech and Sentiment Detection in Low-Resource Memes Using Late-Fusion Hybrid Architecture

# linus@CHiPSAL 2026: Multimodal Hate Speech and Sentiment Detection in Low-Resource Memes Using Late-Fusion Hybrid Architecture

**Authors:** [Sunil Regmi](https://md.research.elamur.ai/free/en/authors/sunil-regmi/), Bipesh Subedi, Saugat Singh and Suman Shrestha · **Year:** 2026 · **Venue:** [Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)](https://md.research.elamur.ai/free/en/venues/chipsal/) ([workshop on Challenges in Processing South Asian Languages 2026](https://md.research.elamur.ai/free/en/venues/chipsal/2026/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.chipsal-1.27/) · [PDF](https://aclanthology.org/2026.chipsal-1.27.pdf) · [DOI 10.63317/3t7nocwwhq8r](https://doi.org/10.63317/3t7nocwwhq8r) · [OpenAlex](https://openalex.org/W7166509002)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/regmi-2026-linus-chipsal-2026-multimodal-hate/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Automated moderation of memes is difficult for low-resource, code-mixed languages like Nepali. Benchmark data is scarce, OCR support is weak, and the text often mixes Devanagari, transliterated English and native English words. For the CHiPSAL 2026 shared task, which focuses on Nepali images with embedded text, the authors developed an approach that handles both binary hate detection (Subtask A) and three-class sentiment classification (Subtask B).

Their model uses late fusion. Visual features come from CLIP ViT-B/32, and the overlaid text is encoded by NepBERTa, a RoBERTa-style model specialised for Nepali. The training sets are small, so they also borrow examples from the other subtask. A heuristic treats negative sentiment as roughly equivalent to hate and positive sentiment as roughly equivalent to non-hate. The authors acknowledge that this equivalence is approximate, since sarcasm can blur it. Training also relied on image augmentations and a class-balanced loss.

On the hidden CodaBench test set, the system reached Macro F1 of 0.8052 for hate detection and 0.6881 for sentiment. The work suggests that pairing a general vision-language encoder with a language-specific text model is a workable recipe for under-resourced meme moderation.

## Key points
- **Architecture:** Late fusion of the CLIP ViT-B/32 image encoder with NepBERTa, using the Rajan/nepbertaTorch subword tokenizer.
- **Cross-task augmentation:**
  - Subtask A training data grew from 1,068 to 1,656 samples by adding non-neutral Subtask B items.
  - Subtask B grew from 1,061 to 2,129 by mapping hate to "Negative" and no-hate to "Positive".
  - Each subtask has a validation split of 133 samples.
- **Official results (CodaBench, hidden test labels):** Macro F1 of 0.8052 on Subtask A and 0.6881 on Subtask B.
- **Image augmentations:** RandomResizedCrop, RandomHorizontalFlip, rotation up to 15°, and strong ColorJitter, applied on top of CLIP-specific normalisation.
- **Limitations noted by the authors:**
  - The label mapping between sentiment and hate is only an approximation, because sarcastic "positive" memes may actually be hateful.
  - Bias, robustness and interpretability are not explicitly addressed.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Kathmandu University](https://md.research.elamur.ai/free/en/universities/kathmandu-university/) — University, Kathmandu, Nepal · 7 works · 20 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (4): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 1,035 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 587 works · [🌱 Low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/) · 287 works

## Cite

Cite as: Sunil Regmi, Bipesh Subedi, Saugat Singh and Suman Shrestha. 2026. linus@CHiPSAL 2026: Multimodal Hate Speech and Sentiment Detection in Low-Resource Memes Using Late-Fusion Hybrid Architecture. Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{regmi2026linus,
  title = {{linus@CHiPSAL 2026: Multimodal Hate Speech and Sentiment Detection in Low-Resource Memes Using Late-Fusion Hybrid Architecture}},
  author = {Sunil Regmi, Bipesh Subedi, Saugat Singh and Suman Shrestha},
  booktitle = {Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)},
  year = {2026},
  doi = {10.63317/3t7nocwwhq8r},
  url = {https://doi.org/10.63317/3t7nocwwhq8r}
}
```

- [← previous Linguistic Steganography via Self-Adjusting Asymmetric Number System](https://md.research.elamur.ai/free/en/papers/liu-2026-linguistic-steganography-self-adjusting-asymmetric/)
- [next → LiveFact: A Dynamic, Time-Aware Benchmark for LLM-Driven Fake News Detection](https://md.research.elamur.ai/free/en/papers/xu-2026-livefact-dynamic-time-aware-benchmark/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/8/)

## References

19 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Greeshma Arya, Mohammad Kamrul Hasan, Ashish Bagwari, Nurhizam Safie, Shayla Islam, Fatima Rayan Awad Ahmed, Aaishani De, Muhammad Attique Khan, and Taher M. Ghazal. 2024. Multimodal hate speech detection in memes using contrastive language-image pre-training. IEEE Access, 12:22359–22375. [find](https://scholar.google.com/scholar?q=Greeshma+Arya%2C+Mohammad+Kamrul+Hasan%2C+Ashish+Bagwari%2C+Nurhizam+Safie%2C+Shayla+Islam%2C+Fatima+Rayan+Awad+Ahmed%2C+Aaishani+De%2C+Muhammad+Attique+Khan%2C+and+Taher+M%2E+Ghazal%2E+2024%2E+Multimodal+hate+speech+detec)
2. Kriti Dubey, Vaishnavi Srivastava, Garima Sharma, Nonita Sharma, Deepak Sharma, Uttam Ghosh, Osama Alfarraj, and Amr Tolba. 2025. Multimodal detection of offensive content in hindi memes. ACM Transactions on Asian and Low-Resource Language Information Processing. [find](https://scholar.google.com/scholar?q=Kriti+Dubey%2C+Vaishnavi+Srivastava%2C+Garima+Sharma%2C+Nonita+Sharma%2C+Deepak+Sharma%2C+Uttam+Ghosh%2C+Osama+Alfarraj%2C+and+Amr+Tolba%2E+2025%2E+Multimodal+detection+of+offensive+content+in+hindi+memes%2E+ACM+Transact)
3. Ahmed El-Sayed and Omar Nasr. 2024. Aast-nlp at multimodal hate speech event detection 2024 : A multimodal approach for classification of text-embedded images based on clip and bert-based models. In CASE. [find](https://scholar.google.com/scholar?q=Ahmed+El%2DSayed+and+Omar+Nasr%2E+2024%2E+Aast%2Dnlp+at+multimodal+hate+speech+event+detection+2024+%3A+A+multimodal+approach+for+classification+of+text%2Dembedded+images+based+on+clip+and+bert%2Dbased+models%2E+In+C)
4. Kazi Toufique Elahi, Tasnuva Binte Rahman, Shakil Shahriar, Samir Sarker, Sajib Kumar Saha Joy, and Faisal Muhammad Shah. 2023a. Explainable multimodal sentiment analysis on bengali memes. [find](https://scholar.google.com/scholar?q=Kazi+Toufique+Elahi%2C+Tasnuva+Binte+Rahman%2C+Shakil+Shahriar%2C+Samir+Sarker%2C+Sajib+Kumar+Saha+Joy%2C+and+Faisal+Muhammad+Shah%2E+2023a%2E+Explainable+multimodal+sentiment+analysis+on+bengali+memes%2E)
5. Kazi Toufique Elahi, Tasnuva Binte Rahman, Shakil Shahriar, Samir Sarker, Sajib Kumar Saha Joy, and Faisal Muhammad Shah. 2023b. Explainable multimodal sentiment analysis on bengali memes. In Proceedings of the International Conference on Computer and Information Technology (ICCIT). [find](https://scholar.google.com/scholar?q=Kazi+Toufique+Elahi%2C+Tasnuva+Binte+Rahman%2C+Shakil+Shahriar%2C+Samir+Sarker%2C+Sajib+Kumar+Saha+Joy%2C+and+Faisal+Muhammad+Shah%2E+2023b%2E+Explainable+multimodal+sentiment+analysis+on+bengali+memes%2E+In+Proceedi)
6. Aishvi Guleria, Kamya Varshney, Garima Pahwa, Shreya Singhal, and Nonita Sharma. 2024. Multimodal sentiment analysis of english and hinglish memes. Multimedia Tools and Applications. [find](https://scholar.google.com/scholar?q=Aishvi+Guleria%2C+Kamya+Varshney%2C+Garima+Pahwa%2C+Shreya+Singhal%2C+and+Nonita+Sharma%2E+2024%2E+Multimodal+sentiment+analysis+of+english+and+hinglish+memes%2E+Multimedia+Tools+and+Applications%2E)
7. Ramesh Kannan and Ratnavel Rajalakshmi. 2022. Multimodal code-mixed Tamil troll meme classification using feature fusion. In Proceedings of the First Workshop on Multimodal Machine Learning in Low-resource Languages, pages 1– 8, IIIT Delhi, New Delhi, India. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Ramesh+Kannan+and+Ratnavel+Rajalakshmi%2E+2022%2E+Multimodal+code%2Dmixed+Tamil+troll+meme+classification+using+feature+fusion%2E+In+Proceedings+of+the+First+Workshop+on+Multimodal+Machine+Learning+in+Low%2Dres)
8. Md. Rezaul Karim, Sumon Kanti Dey, Tanhim Islam, Md. Shajalal, and Bharathi Raja Chakravarthi. 2022. Multimodal hate speech detection from bengali memes and texts. In Proceedings of the International Conference on Speech and Language Technologies for Low-Resource Languages. [find](https://scholar.google.com/scholar?q=Md%2E+Rezaul+Karim%2C+Sumon+Kanti+Dey%2C+Tanhim+Islam%2C+Md%2E+Shajalal%2C+and+Bharathi+Raja+Chakravarthi%2E+2022%2E+Multimodal+hate+speech+detection+from+bengali+memes+and+texts%2E+In+Proceedings+of+the+International+)
9. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In International conference on machine learning, pages 8748–8763. PMLR.
10. Kshitij Rajput, Raghav Kapoor, Kaushal Rai, and Preeti Kaur. 2022. Hate me not: Detecting hate inducing memes in code switched languages. arXiv preprint arXiv:2204.11356. — [arXiv:2204.11356](https://arxiv.org/abs/2204.11356) [find](https://scholar.google.com/scholar?q=Kshitij+Rajput%2C+Raghav+Kapoor%2C+Kaushal+Rai%2C+and+Preeti+Kaur%2E+2022%2E+Hate+me+not%3A+Detecting+hate+inducing+memes+in+code+switched+languages%2E+arXiv+preprint+arXiv%3A2204%2E11356%2E+%E2%80%94+%5BarXiv%3A2204%2E11356%5D%28https%3A%2F%2F)
11. Kengatharaiyer Sarveswaran, Surendrabikram Thapa, Ashwini Vaidya, Tafseer Ahmed Khan, and Bal Krishna Bal. 2026. [Findings of the second workshop on challenges in processing south asian languages (chipsal 2026)](https://md.research.elamur.ai/free/en/papers/sarveswaran-2026-findings-second-workshop-challenges-processing/). In Proceedings of the Second Workshop on Challenges in Processing South Asian Languages (…
12. Surendrabikram Thapa, Shuvam Shiwakoti, Siddhant Bikram Shah, Surabhi Adhikari, Hariram Veeramani, Mehwish Nasim, and Usman Naseem. 2025. Large language models (llm) in computational social science: prospects, current state, and challenges. Social Network Analysis and Mining, 15(1):1–30. [find](https://scholar.google.com/scholar?q=Surendrabikram+Thapa%2C+Shuvam+Shiwakoti%2C+Siddhant+Bikram+Shah%2C+Surabhi+Adhikari%2C+Hariram+Veeramani%2C+Mehwish+Nasim%2C+and+Usman+Naseem%2E+2025%2E+Large+language+models+%28llm%29+in+computational+social+science%3A+p)
13. Surendrabikram Thapa, Shuvam Shiwakoti, Siddhant Bikram Shah, Kritesh Rauniyar, Laxmi Thapa, Surabhi Adhikari, Kristina T. Johnson, Kengatharaiyer Sarveswaran, Bal Krishna Bal, and Usman Naseem. 2026. [Multimodal hate and sentiment understanding in low-resource text-embedded images for online safety…](https://md.research.elamur.ai/free/en/papers/thapa-2026-multimodal-hate-sentiment-understanding-low/)
14. Aidos Konyspay, Pakizar Shamoi, Malika Ziyada, and Zhusup Smambayev. 2025. Meme similarity and emotion detection using multimodal analysis. In 2025 International Conference on Activity and Behavior Computing (ABC), page 1–10. IEEE. [find](https://scholar.google.com/scholar?q=Aidos+Konyspay%2C+Pakizar+Shamoi%2C+Malika+Ziyada%2C+and+Zhusup+Smambayev%2E+2025%2E+Meme+similarity+and+emotion+detection+using+multimodal+analysis%2E+In+2025+International+Conference+on+Activity+and+Behavior+Co)
15. Shreyash Mishra, S Suryavardan, Parth Patwa, Megha Chakraborty, Anku Rani, Aishwarya Reganti, Aman Chadha, Amitava Das, Amit Sheth, Manoj Chinnakotla, Asif Ekbal, and Srijan Kumar. 2023. Memotion 3: Dataset on sentiment and emotion analysis of codemixed hindi-english memes. [find](https://scholar.google.com/scholar?q=Shreyash+Mishra%2C+S+Suryavardan%2C+Parth+Patwa%2C+Megha+Chakraborty%2C+Anku+Rani%2C+Aishwarya+Reganti%2C+Aman+Chadha%2C+Amitava+Das%2C+Amit+Sheth%2C+Manoj+Chinnakotla%2C+Asif+Ekbal%2C+and+Srijan+Kumar%2E+2023%2E+Memotion+3%3A+D)
16. Anil Singh Parihar, Surendrabikram Thapa, and Sushruti Mishra. 2021. Hate speech detection using natural language processing: Applications and challenges. In 2021 5th International Conference on Trends in Electronics and Informatics (ICOEI), pages 1302–1308. IEEE. [find](https://scholar.google.com/scholar?q=Anil+Singh+Parihar%2C+Surendrabikram+Thapa%2C+and+Sushruti+Mishra%2E+2021%2E+Hate+speech+detection+using+natural+language+processing%3A+Applications+and+challenges%2E+In+2021+5th+International+Conference+on+Trend)
17. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, 10. Language Resource References Aashish Bhandari, Siddhant B Shah, Surendrabikram Thapa, Usman Naseem, and Mehwish Nasim. 2023. Crisishatemm: Multimodal analysis of directed and undirected hate speech in text-… [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jong+Wook+Kim%2C+Chris+Hallacy%2C+Aditya+Ramesh%2C+Gabriel+Goh%2C+Sandhini+Agarwal%2C+10%2E+Language+Resource+References+Aashish+Bhandari%2C+Siddhant+B+Shah%2C+Surendrabikram+Thapa%2C+Usman+Naseem%2C+and+Me)
18. Surendrabikram Thapa, Hariram Veeramani, Liang Hu, Qi Zhang, Wei Wang, and Usman Naseem. 2025a. A multimodal prompt-based framework for analyzing code-mixed and low-resource memes. In Proceedings of the International AAAI Conference on Web and Social Media, volume 19, pages 1913–1923. [find](https://scholar.google.com/scholar?q=Surendrabikram+Thapa%2C+Hariram+Veeramani%2C+Liang+Hu%2C+Qi+Zhang%2C+Wei+Wang%2C+and+Usman+Naseem%2E+2025a%2E+A+multimodal+prompt%2Dbased+framework+for+analyzing+code%2Dmixed+and+low%2Dresource+memes%2E+In+Proceedings+of+t)
19. Surendrabikram Thapa, Hariram Veeramani, Imran Razzak, Roy Ka-Wei Lee, and Usman Naseem. 2025b. Cross platform multimodal retrieval augmented distillation for code-switched content understanding. In Companion Proceedings of the ACM on Web Conference 2025, pages 2042– 2051. [find](https://scholar.google.com/scholar?q=Surendrabikram+Thapa%2C+Hariram+Veeramani%2C+Imran+Razzak%2C+Roy+Ka%2DWei+Lee%2C+and+Usman+Naseem%2E+2025b%2E+Cross+platform+multimodal+retrieval+augmented+distillation+for+code%2Dswitched+content+understanding%2E+In+C)

*Updated 2 November 2026*