--- title: "Do Tokenizers Fail on Informal Hindi Expressions? Evidence" description: "The paper asks whether standard subword tokenizers break apart informal Hindi expressions (slang… 4 references, cited by 2 papers here." authors: ["Manikandan Ravikiran", "Tanmay Tiwari", "Vibhu Gupta", "Rakesh Prakash", "Rohit Saluja", "Shayan Mohanty"] year: 2026 venue: "Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)" source: "https://aclanthology.org/2026.loreslm-1.2/" doi: "10.18653/v1/2026.loreslm-1.2" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/ravikiran-2026-do-tokenizers-fail-informal-hindi/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ravikiran-2026-do-tokenizers-fail-informal-hindi/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/) / Do Tokenizers Fail on Informal Hindi Expressions? Evidence from Static, Downstream, and Robustness Analyses # Do Tokenizers Fail on Informal Hindi Expressions? Evidence from Static, Downstream, and Robustness Analyses **Authors:** [Manikandan Ravikiran](https://md.research.elamur.ai/free/en/authors/manikandan-ravikiran/), [Tanmay Tiwari](https://md.research.elamur.ai/free/en/authors/tanmay-tiwari/), [Vibhu Gupta](https://md.research.elamur.ai/free/en/authors/vibhu-gupta/), Rakesh Prakash, [Rohit Saluja](https://md.research.elamur.ai/free/en/authors/rohit-saluja/) and Shayan Mohanty · **Year:** 2026 · **Venue:** [Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)](https://md.research.elamur.ai/free/en/venues/loreslm/) ([Workshop on Language Models for Low-Resource Languages 2026](https://md.research.elamur.ai/free/en/venues/loreslm/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.loreslm-1.2/) · [PDF](https://aclanthology.org/2026.loreslm-1.2.pdf) · [DOI 10.18653/v1/2026.loreslm-1.2](https://doi.org/10.18653/v1/2026.loreslm-1.2) · [OpenAlex](https://openalex.org/W7140132326) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/ravikiran-2026-do-tokenizers-fail-informal-hindi/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper asks whether standard subword tokenizers break apart informal Hindi expressions (slang, idioms, other lexicalized multi-word phrases) whose meaning holds only as a whole. The authors treat these categories together because they share one problem for segmentation: semantic units do not match word boundaries. Using the HiSlang-4.9k dataset, they compare five tokenizers (multilingual, Indic-focused and byte-level) in three ways: - **Static metrics** for boundary preservation and phrase integrity. - **Downstream expression identification**, framed as BIO sequence labeling under identical training conditions. - **Robustness tests** with orthographic variants, romanization and spelling noise. On clean Devanagari text, Indic-oriented models (MuRIL, IndicBERT) keep expression boundaries better and reach higher span-level F1 than mBERT and XLM-R. Better static preservation tends to go with better downstream results. Under romanization, however, every tokenizer collapses, with phrase integrity close to zero. The work shows that tokenization is an overlooked bottleneck for informal and cross-script Hindi NLP. It also argues that tokenizer evaluations should cover clean, noisy and romanized input. ## Key points - **Dataset:** HiSlang-4.9k contains 4,906 sentences, half with annotated expressions and half without (2,453 each), averaging 15.5 words. The sentences were gathered from film material (screenplays and subtitle files), existing language corpora, and posts on social platforms. - **Annotation:** eight native speakers labeled the data and two linguists reviewed it, with Cohen's κ of 0.97 at sentence level and 0.94 at phrase level. The task uses B-EXP / I-EXP / O tags with an 80/10/10 sentence-level split. - **Byte-level tokenization:** it removes OOV problems but fragments expressions so heavily that downstream performance drops to zero. - **Noise results:** IndicBERT's unusually high span F1 under noise comes from degenerate predictions, not real robustness. This shows that aggregate metrics can hide pathological behavior. - **Limitations:** the study covers Hindi only, and tokenizer, vocabulary and pretraining effects are not separated. The noise is synthetic, and the only task evaluated is sequence labeling. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Colorado Boulder](https://md.research.elamur.ai/free/en/universities/university-of-colorado-boulder/) — University, Boulder, United States · 69 works · 51,069 citations - [Indian Institute of Technology Mandi](https://md.research.elamur.ai/free/en/universities/indian-institute-of-technology-mandi/) — University, Mandi, India · 2 works · 3 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works 🧭 Topics (1): [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) · 124 works ## Cite Cite as: Manikandan Ravikiran, Tanmay Tiwari, Vibhu Gupta, Rakesh Prakash, Rohit Saluja and Shayan Mohanty. 2026. Do Tokenizers Fail on Informal Hindi Expressions? Evidence from Static, Downstream, and Robustness Analyses. Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{ravikiran2026do, title = {{Do Tokenizers Fail on Informal Hindi Expressions? Evidence from Static, Downstream, and Robustness Analyses}}, author = {Manikandan Ravikiran, Tanmay Tiwari, Vibhu Gupta, Rakesh Prakash, Rohit Saluja and Shayan Mohanty}, booktitle = {Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)}, year = {2026}, doi = {10.18653/v1/2026.loreslm-1.2}, url = {https://doi.org/10.18653/v1/2026.loreslm-1.2} } ``` - [← previous Do LLMs Encode Functional Importance of Reasoning Tokens ?](https://md.research.elamur.ai/free/en/papers/singh-2026-do-llms-encode-functional-importance/) - [next → Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models](https://md.research.elamur.ai/free/en/papers/xiang-2026-do-we-need-distinct-representations/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/) ## References 4 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Kalika Bali, Jatin Sharma, Monojit Choudhury, and Yogarshi Vyas. 2014. [“I am borrowing ya mixing?” an analysis of english-hindi code-mixing in facebook](https://md.research.elamur.ai/free/en/papers/bali-2014-i-am-borrowing-ya-mixing/). In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 116– 126, Doha, Qatar. Association for Computational Ling… 2. Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In Proceedings of the 6th International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov+and+Yonatan+Bisk%2E+2018%2E+Synthetic+and+natural+noise+both+break+neural+machine+translation%2E+In+Proceedings+of+the+6th+International+Conference+on+Learning+Representations+%28ICLR%29%2E) 3. Kaj Bostrom and Greg Durrett. 2020. [Byte pair encoding is suboptimal for language model pretraining](https://md.research.elamur.ai/free/en/papers/bostrom-2020-byte-pair-encoding-is-suboptimal/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4617–4624, Online. Association for Computational Linguistics. 4. Maharaj Brahma, N. J. Karthika, Atul Singh, Devaraj Adiga, Smruti Bhate, Ganesh Ramakrishnan, Rohit Saluja, and Maunendra Sankar Desarkar. 2025. Morphtok: Morphologically grounded tokenization for indian languages. In Proceedings of the ICML 2025 Workshop on Tokenization (TokShop). [find](https://scholar.google.com/scholar?q=Maharaj+Brahma%2C+N%2E+J%2E+Karthika%2C+Atul+Singh%2C+Devaraj+Adiga%2C+Smruti+Bhate%2C+Ganesh+Ramakrishnan%2C+Rohit+Saluja%2C+and+Maunendra+Sankar+Desarkar%2E+2025%2E+Morphtok%3A+Morphologically+grounded+tokenization+for+ind) *Updated 31 October 2026*