---
title: "Beyond Benchmark Accuracy: Robustness Evaluation of Hinglish"
description: "Hinglish sentiment classifiers often post excellent test-set scores, but it is unclear whether these reflect real… 0.997 accuracy. 4 references."
authors: ["Chennuru Rahul", "Kolawole Adebayo"]
year: 2026
venue: "Proceedings of the Sixth Workshop on Speech, Vision, and Language Technologies for Dravidian Languages"
source: "https://aclanthology.org/2026.dravidianlangtech-1.2/"
doi: "10.18653/v1/2026.dravidianlangtech-1.2"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rahul-2026-beyond-benchmark-accuracy-robustness-evaluation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rahul-2026-beyond-benchmark-accuracy-robustness-evaluation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/23/) / Beyond Benchmark Accuracy: Robustness Evaluation of Hinglish Sentiment Models

# Beyond Benchmark Accuracy: Robustness Evaluation of Hinglish Sentiment Models

**Authors:** Chennuru Rahul and Kolawole Adebayo · **Year:** 2026 · **Venue:** [Proceedings of the Sixth Workshop on Speech, Vision, and Language Technologies for Dravidian Languages](https://md.research.elamur.ai/free/en/venues/dravidianlangtech/) ([Workshop on Speech, Vision, and Language Technologies for Dravidian Languages 2026](https://md.research.elamur.ai/free/en/venues/dravidianlangtech/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.dravidianlangtech-1.2/) · [PDF](https://aclanthology.org/2026.dravidianlangtech-1.2.pdf) · [DOI 10.18653/v1/2026.dravidianlangtech-1.2](https://doi.org/10.18653/v1/2026.dravidianlangtech-1.2) · [OpenAlex](https://openalex.org/W7166880533)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary

Hinglish sentiment classifiers often post excellent test-set scores, but it is unclear whether these reflect real understanding of sentiment. The authors built a training corpus of 25,543 Hinglish tweets from a public emotion dataset, collapsing its emotion labels into positive, negative and neutral. They cleaned it with deduplication, noise removal and dictionary-based spelling normalisation. A leakage check using SHA-256 hashing and 3-gram Jaccard similarity found no overlap between splits. On this data, fine-tuned XLM-R scores 99.7% in-distribution, compared with 74.6% for mBERT.

The authors then stress-tested XLM-R in two ways. The first was a new 400-sentence adversarial set covering five linguistic phenomena. The second was English TweetEval with no further training. XLM-R drops to 42.5% on the adversarial set and 50.8% on TweetEval, and it handles negation and contrast especially badly. The study argues that headline accuracy can hide reliance on lexical cues. It recommends targeted adversarial and cross-domain testing before sentiment systems are deployed in sensitive, code-mixed settings.

## Key points

- **Data and splits:** the corpus has 25,543 tweets divided into 18,786 train, 2,088 validation and 4,669 test. Classes are skewed: 48.4% negative, 33.3% positive and 18.3% neutral, so a majority-class baseline reaches 48.4%.
- **Models and training:** the models are `bert-base-multilingual-cased` (179M parameters) and `xlm-roberta-base` (278M parameters). Both were trained with AdamW at a learning rate of 2e-5 and batch size 32 for 3 epochs, with checkpoints chosen by validation macro F1.
- **In-distribution results:** XLM-R reaches 0.997 accuracy and 0.996 macro F1, while mBERT reaches 0.746 accuracy and 0.673 macro F1.
- **Adversarial benchmark:** the set has 80 items in each of five categories: negation, contrast, sarcasm, subtle sentiment and true neutral. Items were drafted from templates and then rewritten by two native speakers (Cohen's κ = 0.81). XLM-R's per-category accuracy is 25% for negation, 25% for contrast, 50% for sarcasm, 50% for subtle sentiment and 63% for true neutral. Its overall scores are 42.5% accuracy and 0.328 macro F1.
- **Zero-shot transfer and limitations:** on TweetEval, XLM-R gets 50.8% accuracy and 0.408 macro F1, against a random baseline of 0.333. The authors list three limitations: only two models were tested, no significance testing was done, and the benchmark is small. They plan to release the benchmark and code.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National University of Ireland, Maynooth](https://md.research.elamur.ai/free/en/universities/national-university-of-ireland-maynooth/) — University, Maynooth, Ireland · 5 works · 10 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 972 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Chennuru Rahul and Kolawole Adebayo. 2026. Beyond Benchmark Accuracy: Robustness Evaluation of Hinglish Sentiment Models. Proceedings of the Sixth Workshop on Speech, Vision, and Language Technologies for Dravidian Languages.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{rahul2026beyond,
  title = {{Beyond Benchmark Accuracy: Robustness Evaluation of Hinglish Sentiment Models}},
  author = {Chennuru Rahul and Kolawole Adebayo},
  booktitle = {Proceedings of the Sixth Workshop on Speech, Vision, and Language Technologies for Dravidian Languages},
  year = {2026},
  doi = {10.18653/v1/2026.dravidianlangtech-1.2},
  url = {https://doi.org/10.18653/v1/2026.dravidianlangtech-1.2}
}
```

- [← previous Beyond Accuracy: Alignment and Error Detection across Languages in the Bi-GSM8K Math-Teaching Benchmark](https://md.research.elamur.ai/free/en/papers/park-2026-beyond-accuracy-alignment-error-detection/)
- [next → Beyond Bias Scores: Unmasking Vacuous Neutrality in Small Language Models](https://md.research.elamur.ai/free/en/papers/manduru-2026-beyond-bias-scores-unmasking-vacuous/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/23/)

## References

4 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa-Anke, and Leonardo Neves. 2020. [TweetEval: Unified benchmark and comparative evaluation for tweet classification](https://md.research.elamur.ai/free/en/papers/barbieri-2020-tweeteval-unified-benchmark-comparative-evaluation/). In Findings of EMNLP. DOI: 10.18653/v1/2020.findings-emnlp.148 Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhar…
2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In NAACL. DOI: 10.18653/v1/N19-1423
3. Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. [Generating natural language adversarial examples](https://md.research.elamur.ai/free/en/papers/alzantot-2018-generating-natural-language-adversarial-examples/). In Proceedings of EMNLP, pages 2890–2896.
4. Kumar Ravi and Vadlamani Ravi. 2016. Sentiment classification of Hinglish text. In Proceedings of the 3rd IKDD Conference on Data Science. Parth Patwa, Gustavo Aguilar, Sudipta Kar, Suraj Pandey, Srinivas P Y K L, Björn Gambäck, Tanmoy Chakraborty, Thamar Solorio, and Amitava Das. 2020. SemEval-20… [find](https://scholar.google.com/scholar?q=Kumar+Ravi+and+Vadlamani+Ravi%2E+2016%2E+Sentiment+classification+of+Hinglish+text%2E+In+Proceedings+of+the+3rd+IKDD+Conference+on+Data+Science%2E+Parth+Patwa%2C+Gustavo+Aguilar%2C+Sudipta+Kar%2C+Suraj+Pandey%2C+Srin)

*Updated 1 November 2026*