---
title: "Annotating Indian Regional Biases using Large — Panda 2026"
description: "Regional bias, meaning prejudice tied to someone's geographic origin, is common in Indian social media discussions."
authors: ["Debasmita Panda", "Akash Anil", "Neelesh Kumar Shukla"]
year: 2026
venue: "Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)"
source: "https://aclanthology.org/2026.starsem-conference.16/"
doi: "10.18653/v1/2026.starsem-conference.16"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/panda-2026-annotating-indian-regional-biases-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/panda-2026-annotating-indian-regional-biases-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/) / Annotating Indian Regional Biases using Large Language Models: Evaluation and Analysis

# Annotating Indian Regional Biases using Large Language Models: Evaluation and Analysis

**Authors:** Debasmita Panda, Akash Anil, Neelesh Kumar Shukla · **Year:** 2026 · **Venue:** Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (\*SEM 2026) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.starsem-conference.16/) · [PDF](https://aclanthology.org/2026.starsem-conference.16.pdf) · [DOI 10.18653/v1/2026.starsem-conference.16](https://doi.org/10.18653/v1/2026.starsem-conference.16) · [OpenAlex](https://openalex.org/W7166875015)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14)

## Summary
Regional bias, meaning prejudice tied to someone's geographic origin, is common in Indian social media discussions. Labeled data for it is scarce because human annotation is costly, often inconsistent, and complicated by code-mixing and transliteration. The authors ask whether open-source LLMs can serve as automatic annotators for this task.

They use the IndRegBias corpus of YouTube and Reddit comments, each labeled as regional bias (RB) or non-regional bias (NRB). Nine open-source LLMs were tested in two settings:
- **Zero-shot:** prompting based on chain-of-thought reasoning.
- **Fine-tuned:** low-rank adaptation on a balanced half of the data, evaluated on the other half.

Agreement with the human labels was measured with Cohen's κ, broken down by writing style (English, code-mixed, transliterated).

Zero-shot agreement was weak for nearly all models. Fine-tuning raised κ for every model. On 500 freshly collected YouTube comments, most fine-tuned models also beat their zero-shot versions. Models remained noticeably weaker on code-mixed and transliterated text than on English. The study suggests that targeted domain adaptation can make LLMs practical tools for scaling up bias corpora in multilingual, low-resource settings.

## Key points
- **Dataset:** IndRegBias contains 25,000 comments (13,015 RB, 11,985 NRB).
  - By writing style: 18,222 English, 6,315 code-mixed and 463 transliterated.
  - Hinglish makes up about 87% of the code-mixed portion, and Hindi about 86% of the transliterated portion.
- **Zero-shot results:** Almost all nine LLMs scored κ below 0.49 against the human labels.
- **Out-of-distribution test (500 new YouTube comments):**
  - Krutrim-2 performed best after fine-tuning, with κ rising from 0.5447 to 0.6950.
  - MNemo-12B and Llama-8B climbed from about 0.46 and 0.41 to above 0.60.
  - Qwen-8B, Gemma-27B and DSeek-14B did not improve over their zero-shot results, so six of nine models gained overall.
- **Writing-style gap:** Even after fine-tuning, the best code-mixed κ was 0.694, well below the best English κ of 0.833 (both from Mixtral-MoE). The authors attribute this to the heavy skew toward English and to gaps in the models' pretraining.
- **Resources:** Models were selected to fit on a single H200 GPU (about 141 GB), and the authors have released their code on GitHub.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{panda2026annotating,
  title = {{Annotating Indian Regional Biases using Large Language Models: Evaluation and Analysis}},
  author = {Panda, Debasmita and Anil, Akash and Shukla, Neelesh Kumar},
  booktitle = {Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.starsem-conference.16},
  url = {https://doi.org/10.18653/v1/2026.starsem-conference.16}
}
```

- [← previous Ancient Greek to Modern Greek Machine Translation: A Novel Benchmark and Fine-Tuning Experiments on LLMs and NMT Models](https://md.research.elamur.ai/free/en/papers/mavromatis-2026-ancient-greek-modern-greek-machine/)
- [next → Annotation Matters: Resolving Cross-Corpus Performance Drops in Hebrew Offensive Language Detection](https://md.research.elamur.ai/free/en/papers/aclbulk-2026-hefetz-annotation-matters-resolving-cross-corpus/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/)

## References

21 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Mohammadmasiha Zahedivafa, Juan Diego Bermeo, Maria Korobeynikova, and Fabrizio Gilardi. 2025. Open-source LLMs for text annotation: A practical guide for model setting and fine-tuning. Journal of Computational Social Science, 8(1):17. [find](https://scholar.google.com/scholar?q=Meysam+Alizadeh%2C+Ma%C3%ABl+Kubli%2C+Zeynab+Samei%2C+Shirin+Dehghani%2C+Mohammadmasiha+Zahedivafa%2C+Juan+Diego+Bermeo%2C+Maria+Korobeynikova%2C+and+Fabrizio+Gilardi%2E+2025%2E+Open%2Dsource+LLMs+for+text+annotation%3A+A+pract)
2. Tatiana Anikina, Jan Cegin, Jakub Simko, and Simon Ostermann. 2025. A rigorous evaluation of LLM data generation strategies for low-resource languages. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1142–1158. Association for Computational Li… [find](https://scholar.google.com/scholar?q=Tatiana+Anikina%2C+Jan+Cegin%2C+Jakub+Simko%2C+and+Simon+Ostermann%2E+2025%2E+A+rigorous+evaluation+of+LLM+data+generation+strategies+for+low%2Dresource+languages%2E+In+Proceedings+of+the+2025+Conference+on+Empiric)
3. Shaily Bhatt, Sunipa Dev, Partha Talukdar, Shachi Dave, and Vinodkumar Prabhakaran. 2022. Re-contextualizing fairness in NLP: The case of India. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conferen… [find](https://scholar.google.com/scholar?q=Shaily+Bhatt%2C+Sunipa+Dev%2C+Partha+Talukdar%2C+Shachi+Dave%2C+and+Vinodkumar+Prabhakaran%2E+2022%2E+Re%2Dcontextualizing+fairness+in+NLP%3A+The+case+of+India%2E+In+Proceedings+of+the+2nd+Conference+of+the+Asia%2DPacifi)
4. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. [find](https://scholar.google.com/scholar?q=Jacob+Cohen%2E+1960%2E+A+coefficient+of+agreement+for+nominal+scales%2E+Educational+and+Psychological+Measurement%2C+20%281%29%3A37%E2%80%9346%2E)
5. Sunipa Dev, Jaya Goyal, Dinesh Tewari, Shachi Dave, and Vinodkumar Prabhakaran. 2023. Building socio-culturally inclusive stereotype resources with community engagement. In Advances in Neural Information Processing Systems (NeurIPS), volume 36. [find](https://scholar.google.com/scholar?q=Sunipa+Dev%2C+Jaya+Goyal%2C+Dinesh+Tewari%2C+Shachi+Dave%2C+and+Vinodkumar+Prabhakaran%2E+2023%2E+Building+socio%2Dculturally+inclusive+stereotype+resources+with+community+engagement%2E+In+Advances+in+Neural+Informat)
6. Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. 2023. Is GPT-3 a good data annotator? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11173–11195. Association for Computational Li… [find](https://scholar.google.com/scholar?q=Bosheng+Ding%2C+Chengwei+Qin%2C+Linlin+Liu%2C+Yew+Ken+Chia%2C+Boyang+Li%2C+Shafiq+Joty%2C+and+Lidong+Bing%2E+2023%2E+Is+GPT%2D3+a+good+data+annotator%3F+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Co) [our card](https://md.research.elamur.ai/free/en/papers/ding-2022-gpt-3-good-data-annotator/)
7. Fahim Faisal and Antonios Anastasopoulos. 2023. Geographic and geopolitical biases of language models. In Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL), pages 139–163. [find](https://scholar.google.com/scholar?q=Fahim+Faisal+and+Antonios+Anastasopoulos%2E+2023%2E+Geographic+and+geopolitical+biases+of+language+models%2E+In+Proceedings+of+the+3rd+Workshop+on+Multi%2Dlingual+Representation+Learning+%28MRL%29%2C+pages+139%E2%80%93163%2E)
8. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd-workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30):e2305016120. [find](https://scholar.google.com/scholar?q=Fabrizio+Gilardi%2C+Meysam+Alizadeh%2C+and+Ma%C3%ABl+Kubli%2E+2023%2E+Chatgpt+outperforms+crowd%2Dworkers+for+text%2Dannotation+tasks%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+120%2830%29%3Ae2305016120%2E) [our card](https://md.research.elamur.ai/free/en/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/)
9. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. [find](https://scholar.google.com/scholar?q=Edward+J+Hu%2C+Yelong+Shen%2C+Phillip+Wallis%2C+Zeyuan+Allen%2DZhu%2C+Yuanzhi+Li%2C+Shean+Wang%2C+Lu+Wang%2C+and+Weizhu+Chen%2E+2021%2E+Lora%3A+Low%2Drank+adaptation+of+large+language+models%2E+arXiv+preprint+arXiv%3A2106%2E09685%2E)
10. Suramya Jadhav, Abhay Shanbhag, Amogh Thakurdesai, Ridhima Sinare, and Raviraj Joshi. 2025. On limitations of LLM as annotator for low resource languages. In Proceedings of the International Conference on Natural Language and Speech Processing (ICNLSP 2025). [find](https://scholar.google.com/scholar?q=Suramya+Jadhav%2C+Abhay+Shanbhag%2C+Amogh+Thakurdesai%2C+Ridhima+Sinare%2C+and+Raviraj+Joshi%2E+2025%2E+On+limitations+of+LLM+as+annotator+for+low+resource+languages%2E+In+Proceedings+of+the+International+Conferenc)
11. Khyati Khandelwal, Manuel Tonneau, Andrew M. Bean, Hannah Rose Kirk, and Scott A. Hale. 2024. Indian-BhED: A dataset for measuring India-centric biases in large language models. In Proceedings of the 2024 International Conference on Information Technology for Social Good (GoodIT ’24), pages 231–239.… [find](https://scholar.google.com/scholar?q=Khyati+Khandelwal%2C+Manuel+Tonneau%2C+Andrew+M%2E+Bean%2C+Hannah+Rose+Kirk%2C+and+Scott+A%2E+Hale%2E+2024%2E+Indian%2DBhED%3A+A+dataset+for+measuring+India%2Dcentric+biases+in+large+language+models%2E+In+Proceedings+of+the+)
12. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916. [find](https://scholar.google.com/scholar?q=Takeshi+Kojima%2C+Shixiang+Shane+Gu%2C+Machel+Reid%2C+Yutaka+Matsuo%2C+and+Yusuke+Iwasawa%2E+2022%2E+Large+language+models+are+zero%2Dshot+reasoners%2E+arXiv+preprint+arXiv%3A2205%2E11916%2E) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)
13. Arbi Haza Nasution and Aytuğ Onan. 2024. Chatgpt label: Comparing the quality of human-generated and llm-generated annotations in low-resource language nlp tasks. IEEE Access, 12:71876–71900. [find](https://scholar.google.com/scholar?q=Arbi+Haza+Nasution+and+Aytug%CC%86+Onan%2E+2024%2E+Chatgpt+label%3A+Comparing+the+quality+of+human%2Dgenerated+and+llm%2Dgenerated+annotations+in+low%2Dresource+language+nlp+tasks%2E+IEEE+Access%2C+12%3A71876%E2%80%9371900%2E)
14. Debasmita Panda, Akash Anil, and Neelesh Kumar Shukla. 2026. IndRegBias: A dataset for studying indian regional biases in english and code-mixed social media comments. arXiv preprint arXiv:2601.06477. — [arXiv:2601.06477](https://arxiv.org/abs/2601.06477) [find](https://scholar.google.com/scholar?q=Debasmita+Panda%2C+Akash+Anil%2C+and+Neelesh+Kumar+Shukla%2E+2026%2E+IndRegBias%3A+A+dataset+for+studying+indian+regional+biases+in+english+and+code%2Dmixed+social+media+comments%2E+arXiv+preprint+arXiv%3A2601%2E06477%2E)
15. Maja Pavlović and Massimo Poesio. 2024. The effectiveness of LLMs as annotators: A comparative overview and empirical analysis of direct representation. In Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) @ LREC-COLING 2024, pages 100–110. ELRA and ICCL. [find](https://scholar.google.com/scholar?q=Maja+Pavlovic%CC%81+and+Massimo+Poesio%2E+2024%2E+The+effectiveness+of+LLMs+as+annotators%3A+A+comparative+overview+and+empirical+analysis+of+direct+representation%2E+In+Proceedings+of+the+3rd+Workshop+on+Perspect)
16. Aditya Tomar, Nihar Ranjan Sahoo, and Pushpak Bhattacharyya. 2025. BharatBBQ: A multilingual bias benchmark for question answering in the indian context. arXiv preprint arXiv:2508.07090. [find](https://scholar.google.com/scholar?q=Aditya+Tomar%2C+Nihar+Ranjan+Sahoo%2C+and+Pushpak+Bhattacharyya%2E+2025%2E+BharatBBQ%3A+A+multilingual+bias+benchmark+for+question+answering+in+the+indian+context%2E+arXiv+preprint+arXiv%3A2508%2E07090%2E)
17. Petter Törnberg. 2023. Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning. arXiv preprint arXiv:2304.06588. [find](https://scholar.google.com/scholar?q=Petter+T%C3%B6rnberg%2E+2023%2E+Chatgpt%2D4+outperforms+experts+and+crowd+workers+in+annotating+political+twitter+messages+with+zero%2Dshot+learning%2E+arXiv+preprint+arXiv%3A2304%2E06588%2E) [our card](https://md.research.elamur.ai/free/en/papers/tornberg-2023-chatgpt-4-outperforms-experts-crowd/)
18. Muhammad Uzair Ul Haq, Hussain Ahmad Zafar, and David Broneske. 2025. LLMs as data annotators: How close are we to human performance? arXiv preprint arXiv:2504.15022. — [arXiv:2504.15022](https://arxiv.org/abs/2504.15022) [find](https://scholar.google.com/scholar?q=Muhammad+Uzair+Ul+Haq%2C+Hussain+Ahmad+Zafar%2C+and+David+Broneske%2E+2025%2E+LLMs+as+data+annotators%3A+How+close+are+we+to+human+performance%3F+arXiv+preprint+arXiv%3A2504%2E15022%2E+%E2%80%94+%5BarXiv%3A2504%2E15022%5D%28https%3A%2F%2Farxi)
19. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 24824–24837. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Maarten+Bosma%2C+Brian+Ichter%2C+Fei+Xia%2C+Ed+Chi%2C+Quoc+Le%2C+and+Denny+Zhou%2E+2022%2E+Chain%2Dof%2Dthought+prompting+elicits+reasoning+in+large+language+models%2E+In+Advances) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
20. Lingling Xu, Haoran Xie, S. Joe Qin, Xiaohui Tao, and Fu Lee Wang. 2026. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. IEEE Transactions on Pattern Analysis and Machine Intelligence, PP(99):1–20. [find](https://scholar.google.com/scholar?q=Lingling+Xu%2C+Haoran+Xie%2C+S%2E+Joe+Qin%2C+Xiaohui+Tao%2C+and+Fu+Lee+Wang%2E+2026%2E+Parameter%2Defficient+fine%2Dtuning+methods+for+pretrained+language+models%3A+A+critical+review+and+assessment%2E+IEEE+Transactions+on+)
21. Mingyue Yuan, Jieshan Chen, Zhenchang Xing, and Gelareh Mohammadi. 2025. A case study of scalable content annotation using multi-LLM consensus and human review. In Proceedings of the Generative AI and HCI Workshop at CHI 2025. [find](https://scholar.google.com/scholar?q=Mingyue+Yuan%2C+Jieshan+Chen%2C+Zhenchang+Xing%2C+and+Gelareh+Mohammadi%2E+2025%2E+A+case+study+of+scalable+content+annotation+using+multi%2DLLM+consensus+and+human+review%2E+In+Proceedings+of+the+Generative+AI+and)