---
title: "IndicJR — paper by Pattnayak and Chowdhuri (2026)"
description: "The authors present IndicJR (IJR), an adversarial benchmark covering 12 Indic and South Asian… 29 references, cited by 3 papers here."
authors: ["Priyaranjan Pattnayak", "Sanchari Chowdhuri"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)"
source: "https://aclanthology.org/2026.eacl-industry.50/"
doi: "10.18653/v1/2026.eacl-industry.50"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pattnayak-2026-indicjr-judge-free-benchmark-jailbreak/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pattnayak-2026-indicjr-judge-free-benchmark-jailbreak/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/5/) / IndicJR: A Judge-Free Benchmark of Jailbreak Robustness in South Asian Languages

# IndicJR: A Judge-Free Benchmark of Jailbreak Robustness in South Asian Languages

**Authors:** [Priyaranjan Pattnayak](https://md.research.elamur.ai/free/en/authors/priyaranjan-pattnayak/) and Sanchari Chowdhuri · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-industry.50/) · [PDF](https://aclanthology.org/2026.eacl-industry.50.pdf) · [DOI 10.18653/v1/2026.eacl-industry.50](https://doi.org/10.18653/v1/2026.eacl-industry.50) · [OpenAlex](https://openalex.org/W7140140875)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/pattnayak-2026-indicjr-judge-free-benchmark-jailbreak/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLM safety is usually tested in English, and often only when models must answer in a fixed refusal format. This leaves risks for South Asian users poorly measured, since they often code-switch, mix scripts or type in Latin transliteration. The authors present IndicJR (IJR), an adversarial benchmark covering 12 Indic and South Asian languages. It is scored by deterministic, language-aware rules rather than LLM judges or human raters.

Prompts are built from two sources:
- benign cores taken from 2023 Wikipedia;
- templated harmful requests (chemical, biological, illicit access), each carrying a canary token.

These are wrapped in four jailbreak families and rendered in native, romanized (via AI4Bharat transliteration) and mixed script. Two tracks are contrasted: a JSON track requiring machine-checkable refusals and a FREE track with ordinary answers.

Across 12 models, the refusal contract makes systems look safer than they are. Jailbreak success rate (JSR) is already high under JSON and approaches 1.0 for every model in FREE. English wrappers transfer readily to Indic content, and romanization lowers contract-bound JSR. The work shows that English-only, contract-focused testing can mask real multilingual vulnerabilities.

## Key points
- **Dataset size:** 45,216 prompts in total.
  - JSON track: 42,636 prompts (37,236 attacked-benign, 3,600 clean-benign, 1,800 clean-harmful).
  - FREE track: 2,580 prompts (2,400 / 120 / 60).
  - Languages: the twelve covered are Hindi, Urdu, Punjabi, Nepali, Bengali, Odia, Marathi, Gujarati, Tamil, Telugu, Kannada and Malayalam.
- **Contract gap:** JSON JSR ranges from 0.508 (GPT-4o) to 0.978 (LLaMA 3.3 70B, LLaMA 4 Maverick). In FREE, JSR is 0.93–1.00 for all models, and clean-benign over-refusal drops to about 0, compared with roughly 0.5–0.7 under JSON.
- **Indic specialization:** Sarvam 1 Base is the most vulnerable model despite its Indic focus, with JSON JSR of 0.959 and clean-harmful leakage of 0.393.
- **Orthography and transfer:**
  - Mean JSON JSR falls from 0.755 (native script) to 0.416 (romanized) and 0.488 (mixed).
  - The JSR change correlates with romanization share (ρ ≈ 0.28–0.32).
  - English→Indic attacks exceed 0.58 JSR in every language, with format wrappers often beating instruction wrappers.
- **Validation:**
  - A human audit of 600 samples gave κ ≈ 0.68 (unweighted) and 0.74 (weighted), with schema validity of 95.4%.
  - Lite and full runs agree closely (r > 0.80).
  - The data is released at github.com/IndicJR.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Oracle](https://md.research.elamur.ai/free/en/companies/oracle/) — Company, Austin, United States · 7 works · 1,167 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 517 works

## Cite

Cite as: Priyaranjan Pattnayak and Sanchari Chowdhuri. 2026. IndicJR: A Judge-Free Benchmark of Jailbreak Robustness in South Asian Languages. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{pattnayak2026indicjr,
  title = {{IndicJR: A Judge-Free Benchmark of Jailbreak Robustness in South Asian Languages}},
  author = {Priyaranjan Pattnayak and Sanchari Chowdhuri},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-industry.50},
  url = {https://doi.org/10.18653/v1/2026.eacl-industry.50}
}
```

- [← previous Incentivizing In-depth Reasoning over Long Contexts with Process Advantage Shaping](https://md.research.elamur.ai/free/en/papers/peng-2026-incentivizing-depth-reasoning-over-long/)
- [next → Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Following](https://md.research.elamur.ai/free/en/papers/ren-2026-instructions-are-all-you-need/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/5/)

## References

29 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Muhammad Falensi Azmi, Muhammad Dehan Al Kautsar, Alfan Farizki Wicaksono, and Fajri Koto. 2025. Indosafety: Culturally grounded safety for llms in indonesian languages. Preprint, arXiv:2506.02573. [find](https://scholar.google.com/scholar?q=Muhammad+Falensi+Azmi%2C+Muhammad+Dehan+Al+Kautsar%2C+Alfan+Farizki+Wicaksono%2C+and+Fajri+Koto%2E+2025%2E+Indosafety%3A+Culturally+grounded+safety+for+llms+in+indonesian+languages%2E+Preprint%2C+arXiv%3A2506%2E02573%2E)
2. Kaj Bostrom and Greg Durrett. 2020. [Byte pair encoding is suboptimal for language model pretraining](https://md.research.elamur.ai/free/en/papers/bostrom-2020-byte-pair-encoding-is-suboptimal/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 5296–5307.
3. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, and Eric Wong. 2024. [Jailbreakbench: An open robustness benchmark for jailbreaking large language models](https://md.research.elamur.ai/free/en/papers/chao-2024-jailbreakbench-open-robustness-benchmark-jailbreaking/). ar…
4. Yue Deng, Wenxuan Zhang, Bing Liu, Sinno Jialin Pan, and Lidong Bing. 2024. [Multilingual jailbreak challenges in large language models](https://md.research.elamur.ai/free/en/papers/deng-2023-multilingual-jailbreak-challenges-large-language/). In International Conference on Learning Representations (ICLR). ArXiv:2310.06474.
5. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Alicia Callahan, Anna Chen, Tom Conerly, Christy Dennison, Tyna Eloundou, Davide Eynard, and 1 others. 2022. [Red teaming language models with language models](https://md.research.elamur.ai/free/en/papers/perez-2022-red-teaming-language-models-language/). In Advances in Neural Information Processing Systems (NeurIPS).
6. Hitesh Laxmichand Patel, Amit Agarwal, Arion Das, Bhargava Kumar, Srikant Panda, Priyaranjan Pattnayak, Taki Hasan Rafi, Tejaswini Kumar, and Dong-Kyu Chae. 2025a. Sweeval: Do llms really swear? a safety benchmark for testing limits for enterprise use. In Proceedings of the 2025 Conference of the Na… [find](https://scholar.google.com/scholar?q=Hitesh+Laxmichand+Patel%2C+Amit+Agarwal%2C+Arion+Das%2C+Bhargava+Kumar%2C+Srikant+Panda%2C+Priyaranjan+Pattnayak%2C+Taki+Hasan+Rafi%2C+Tejaswini+Kumar%2C+and+Dong%2DKyu+Chae%2E+2025a%2E+Sweeval%3A+Do+llms+really+swear%3F+a+saf)
7. Kavya Patel, Ruoxi Wang, Ming Zhao, and 1 others. 2025b. Polyguard: A multilingual safety benchmark for large language models. arXiv preprint arXiv:2504.04377. — [arXiv:2504.04377](https://arxiv.org/abs/2504.04377) [find](https://scholar.google.com/scholar?q=Kavya+Patel%2C+Ruoxi+Wang%2C+Ming+Zhao%2C+and+1+others%2E+2025b%2E+Polyguard%3A+A+multilingual+safety+benchmark+for+large+language+models%2E+arXiv+preprint+arXiv%3A2504%2E04377%2E+%E2%80%94+%5BarXiv%3A2504%2E04377%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
8. Priyaranjan Pattnayak, Hitesh Laxmichand Patel, and Amit Agarwal. 2025. Tokenization matters: Improving zero-shot ner for indic languages. Preprint, arXiv:2504.16977. — [arXiv:2504.16977](https://arxiv.org/abs/2504.16977) [find](https://scholar.google.com/scholar?q=Priyaranjan+Pattnayak%2C+Hitesh+Laxmichand+Patel%2C+and+Amit+Agarwal%2E+2025%2E+Tokenization+matters%3A+Improving+zero%2Dshot+ner+for+indic+languages%2E+Preprint%2C+arXiv%3A2504%2E16977%2E+%E2%80%94+%5BarXiv%3A2504%2E16977%5D%28https%3A%2F%2Farxi)
9. Ethan Perez, He Huang, Francis Song, Trevor Cai, Roman Ring, Bowen Chen, Xia Chen, He He, Seung Kim, Thomas Lukasiewicz, and 1 others. 2022. Red-teaming large language models using chain-of-thought. In NeurIPS 2022 Workshop on Critiquing and Correcting Trends in Machine Learning. [find](https://scholar.google.com/scholar?q=Ethan+Perez%2C+He+Huang%2C+Francis+Song%2C+Trevor+Cai%2C+Roman+Ring%2C+Bowen+Chen%2C+Xia+Chen%2C+He+He%2C+Seung+Kim%2C+Thomas+Lukasiewicz%2C+and+1+others%2E+2022%2E+Red%2Dteaming+large+language+models+using+chain%2Dof%2Dthought%2E+I)
10. Ethan Perez, Zifan Wu, Jessica Li, Arijit Ghosh, Peter Liu, Hyung Won Chung, and 1 others. 2024. Polyguard: Multilingual safety for large language models. arXiv preprint arXiv:2402.17572. — [arXiv:2402.17572](https://arxiv.org/abs/2402.17572) [find](https://scholar.google.com/scholar?q=Ethan+Perez%2C+Zifan+Wu%2C+Jessica+Li%2C+Arijit+Ghosh%2C+Peter+Liu%2C+Hyung+Won+Chung%2C+and+1+others%2E+2024%2E+Polyguard%3A+Multilingual+safety+for+large+language+models%2E+arXiv+preprint+arXiv%3A2402%2E17572%2E+%E2%80%94+%5BarXiv%3A240)
11. Divyanshu Kakwani, Anoop Kunchukuttan, Sumanth Golla, and 1 others. 2020. [Indicnlpsuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for indian languages](https://md.research.elamur.ai/free/en/papers/kakwani-2020-indicnlpsuite-monolingual-corpora-evaluation-benchmarks/). arXiv preprint arXiv:2004.00064.
12. Krithika Ramesh, Ankit Kumar, and 1 others. 2022. Indicxtreme: A benchmark for evaluating indic languages in extreme multilingual settings. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Krithika+Ramesh%2C+Ankit+Kumar%2C+and+1+others%2E+2022%2E+Indicxtreme%3A+A+benchmark+for+evaluating+indic+languages+in+extreme+multilingual+settings%2E+In+Proceedings+of+EMNLP%2E)
13. Taku Kudo and John Richardson. 2018. [Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/). arXiv preprint arXiv:1808.06226.
14. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, and Sebastian Ruder. 2021. Good-enough compositional data augmentation and sampling for word-piece tokenization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), pages 3500–3512. [find](https://scholar.google.com/scholar?q=Phillip+Rust%2C+Jonas+Pfeiffer%2C+Ivan+Vulic%CC%81%2C+and+Sebastian+Ruder%2E+2021%2E+Good%2Denough+compositional+data+augmentation+and+sampling+for+word%2Dpiece+tokenization%2E+In+Proceedings+of+the+59th+Annual+Meeting+of)
15. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. [find](https://scholar.google.com/scholar?q=J%2E+Richard+Landis+and+Gary+G%2E+Koch%2E+1977%2E+The+measurement+of+observer+agreement+for+categorical+data%2E+Biometrics%2C+33%281%29%3A159%E2%80%93174%2E)
16. Percy Liang, Rishi Bommasani, Hanlin Zha, and 1 others. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Hanlin+Zha%2C+and+1+others%2E+2022%2E+Holistic+evaluation+of+language+models%2E+arXiv+preprint+arXiv%3A2211%2E09110%2E)
17. Xinyu Liu, Chengyuan Li, Tianyi Zhang, and 1 others. 2024. [Jailbreakbench: An open benchmark for jailbreaking large language models](https://md.research.elamur.ai/free/en/papers/chao-2024-jailbreakbench-open-robustness-benchmark-jailbreaking/). arXiv preprint arXiv:2404.01318.
18. Yash Madhani, Sushane Parthan, Priyanka Bedekar, Gokul N. C., Ruchi Khapra, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh M. Khapra. 2022. Aksharantar: Open indic-language transliteration datasets and models for the next billion users. arXiv preprint arXiv:2205.03018. Includes IndicXlit: a multilin… [find](https://scholar.google.com/scholar?q=Yash+Madhani%2C+Sushane+Parthan%2C+Priyanka+Bedekar%2C+Gokul+N%2E+C%2E%2C+Ruchi+Khapra%2C+Anoop+Kunchukuttan%2C+Pratyush+Kumar%2C+and+Mitesh+M%2E+Khapra%2E+2022%2E+Aksharantar%3A+Open+indic%2Dlanguage+transliteration+datasets+an)
19. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of ACL, pages 1715– 1725.
20. Harman Singh, Nitish Gupta, Shikhar Bharadwaj, Dinesh Tewari, and Partha Talukdar. 2024. Indicgenbench: A multilingual benchmark to evaluate generation capabilities of llms on indic languages. [find](https://scholar.google.com/scholar?q=Harman+Singh%2C+Nitish+Gupta%2C+Shikhar+Bharadwaj%2C+Dinesh+Tewari%2C+and+Partha+Talukdar%2E+2024%2E+Indicgenbench%3A+A+multilingual+benchmark+to+evaluate+generation+capabilities+of+llms+on+indic+languages%2E)
21. Jiayang Song, Yuheng Huang, Zhehua Zhou, and Lei Ma. 2024. Multilingual blending: Llm safety alignment evaluation with language mixture. arXiv preprint arXiv:2407.07342. — [arXiv:2407.07342](https://arxiv.org/abs/2407.07342) [find](https://scholar.google.com/scholar?q=Jiayang+Song%2C+Yuheng+Huang%2C+Zhehua+Zhou%2C+and+Lei+Ma%2E+2024%2E+Multilingual+blending%3A+Llm+safety+alignment+evaluation+with+language+mixture%2E+arXiv+preprint+arXiv%3A2407%2E07342%2E+%E2%80%94+%5BarXiv%3A2407%2E07342%5D%28https%3A%2F%2Fa)
22. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, and 1 others. 2022. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). In Transactions of the Association for Computational Linguistics (TACL).
23. Rui Sun, Chen Huang, Xin Li, and 1 others. 2025. Safedialbench: Evaluating multi-turn dialogue jailbreak attacks on llms. arXiv preprint arXiv:2502.11090. — [arXiv:2502.11090](https://arxiv.org/abs/2502.11090) [find](https://scholar.google.com/scholar?q=Rui+Sun%2C+Chen+Huang%2C+Xin+Li%2C+and+1+others%2E+2025%2E+Safedialbench%3A+Evaluating+multi%2Dturn+dialogue+jailbreak+attacks+on+llms%2E+arXiv+preprint+arXiv%3A2502%2E11090%2E+%E2%80%94+%5BarXiv%3A2502%2E11090%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F25)
24. Wei Wang and 1 others. 2024. All languages matter: On the multilingual safety of llms. In Findings of the Association for Computational Linguistics (ACL). XSAFETY benchmark. [find](https://scholar.google.com/scholar?q=Wei+Wang+and+1+others%2E+2024%2E+All+languages+matter%3A+On+the+multilingual+safety+of+llms%2E+In+Findings+of+the+Association+for+Computational+Linguistics+%28ACL%29%2E+XSAFETY+benchmark%2E)
25. Ishaan Watts, Varun Gumma, Aditya Yadavalli, Vivek Seshadri, Manohar Swaminathan, and Sunayana Sitaram. 2024. PARIKSHA: A large-scale investigation of human-LLM evaluator agreement on multilingual and multi-cultural data. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language… [find](https://scholar.google.com/scholar?q=Ishaan+Watts%2C+Varun+Gumma%2C+Aditya+Yadavalli%2C+Vivek+Seshadri%2C+Manohar+Swaminathan%2C+and+Sunayana+Sitaram%2E+2024%2E+PARIKSHA%3A+A+large%2Dscale+investigation+of+human%2DLLM+evaluator+agreement+on+multilingual+and)
26. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2021. Byt5: Towards a token-free future with pre-trained byte-to-byte models. arXiv preprint arXiv:2105.13626. [find](https://scholar.google.com/scholar?q=Linting+Xue%2C+Aditya+Barua%2C+Noah+Constant%2C+Rami+Al%2DRfou%2C+Sharan+Narang%2C+Mihir+Kale%2C+Adam+Roberts%2C+and+Colin+Raffel%2E+2021%2E+Byt5%3A+Towards+a+token%2Dfree+future+with+pre%2Dtrained+byte%2Dto%2Dbyte+models%2E+arXiv+p)
27. Zheng-Xin Yong and 1 others. 2023. [Low-resource languages jailbreak gpt-4](https://md.research.elamur.ai/free/en/papers/yong-2023-low-resource-languages-jailbreak-gpt/). arXiv preprint arXiv:2310.02446.
28. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. [Defending against neural fake news](https://md.research.elamur.ai/free/en/papers/zellers-2019-defending-against-neural-fake-news/). In Advances in Neural Information Processing Systems (NeurIPS).
29. Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2023. [Safetybench: Evaluating the safety of large language models](https://md.research.elamur.ai/free/en/papers/zhang-2023-safetybench-evaluating-safety-large/). arXiv preprint arXiv:2309.07045.

*Updated 1 November 2026*