--- title: "SN-WER — paper by Pattnayak (2026) — summary and ideas" description: "The author proposes SN-WER, a scoring-only method. 12 references. Among them: 0." authors: ["Priyaranjan Pattnayak"] year: 2026 venue: "Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026)" source: "https://aclanthology.org/2026.mellm-1.24/" doi: "10.18653/v1/2026.mellm-1.24" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/pattnayak-2026-sn-wer-script-normalized-wer/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pattnayak-2026-sn-wer-script-normalized-wer/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/43/) / SN-WER: Script-Normalized WER for Multi-Script Indic ASR Evaluation # SN-WER: Script-Normalized WER for Multi-Script Indic ASR Evaluation **Authors:** [Priyaranjan Pattnayak](https://md.research.elamur.ai/free/en/authors/priyaranjan-pattnayak/) · **Year:** 2026 · **Venue:** [Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026)](https://md.research.elamur.ai/free/en/venues/mellm/) ([Workshop on Multilinguality in the Era of Large Language Models 2026](https://md.research.elamur.ai/free/en/venues/mellm/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.mellm-1.24/) · [PDF](https://aclanthology.org/2026.mellm-1.24.pdf) · [DOI 10.18653/v1/2026.mellm-1.24](https://doi.org/10.18653/v1/2026.mellm-1.24) · [OpenAlex](https://openalex.org/W7166806359) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Standard Word Error Rate penalizes ASR output as wrong whenever the hypothesis spells a correct word in a different script from the reference. This is a frequent problem for Indic languages, where models often produce romanized text while benchmark references use native scripts. The author proposes SN-WER, a scoring-only method. It transliterates both reference and hypothesis into the benchmark's native script, applies Unicode, punctuation and digit normalization, and then computes ordinary WER. No retraining or decoding changes are required. The method is tested on Hindi, Bengali, Tamil, Odia and Gujarati using FLEURS and Common Voice v17, with whisper-large-v3, whisper-small and MMS, plus an extra check on Arabic and Urdu. On the curated FLEURS data, SN-WER shrinks inflated model gaps by as much as 12%. On noisier Common Voice, the effect is smaller and less consistent, so real recognition failures remain visible. Controlled perturbation tests show the method removes script-induced penalties while still catching genuine lexical mistakes. The author positions SN-WER as a companion to WER/CER rather than a replacement. It is intended for settings where script does not matter, such as search, indexing, or LLM pipelines. ## Key points - On artificial 0→50% hypothesis romanization, SN-WER absorbs about 67% of the resulting WER inflation. - Under 20–30% lexical corruption, SN-WER tracks WER closely (∆SN/∆WER ≈ 1.09). Shuffled or substituted adversarial hypotheses score near 1.0. - Results barely change across transliterators (ICU, IAST, ITRANS; ∆ \< 0.002) and normalization ablations (∆ \< 0.05). The token collision rate stays below 0.1%. - The size of the correction correlates with the romanization rate (r = 0.81 for Whisper-large). Whisper on Odia FLEURS drops from 1.13 to 1.02. - MMS is the strongest model; on Common Voice its score falls from 0.46 to 0.36. Arabic and Urdu show reductions of 5–9%. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Oracle](https://md.research.elamur.ai/free/en/companies/oracle/) — Company, Austin, United States · 7 works · 1,167 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works 🧭 Topics (1): [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 480 works ## Cite Cite as: Priyaranjan Pattnayak. 2026. SN-WER: Script-Normalized WER for Multi-Script Indic ASR Evaluation. Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{pattnayak2026sn, title = {{SN-WER: Script-Normalized WER for Multi-Script Indic ASR Evaluation}}, author = {Priyaranjan Pattnayak}, booktitle = {Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026)}, year = {2026}, doi = {10.18653/v1/2026.mellm-1.24}, url = {https://doi.org/10.18653/v1/2026.mellm-1.24} } ``` - [← previous SMILE-Next: Teaching Large Language Models to Detect, Classify, and Reason about Laughter](https://md.research.elamur.ai/free/en/papers/mok-2026-smile-next-teaching-large-language/) - [next → SOAPTriage: SOAP-Guided Multi-View Clinical Text Modeling Framework for Automated ESI Prediction](https://md.research.elamur.ai/free/en/papers/wang-2026-soaptriage-soap-guided-multi-view/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/43/) ## References 12 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Ahmed Ali, Walid Magdy, Peter Bell, and Steve Re-nais. 2015. Multi-reference wer for evaluating asr for languages with no orthographic rules. In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pages 576–580. [find](https://scholar.google.com/scholar?q=Ahmed+Ali%2C+Walid+Magdy%2C+Peter+Bell%2C+and+Steve+Re%2Dnais%2E+2015%2E+Multi%2Dreference+wer+for+evaluating+asr+for+languages+with+no+orthographic+rules%2E+In+2015+IEEE+Workshop+on+Automatic+Speech+Recognition+and+) 2. Ahmed M. Ali, Preslav Nakov, Peter Bell, and Steve Renals. 2017. Werd: Using social text spelling variants for evaluating dialectal speech recognition. 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 141–148. [find](https://scholar.google.com/scholar?q=Ahmed+M%2E+Ali%2C+Preslav+Nakov%2C+Peter+Bell%2C+and+Steve+Renals%2E+2017%2E+Werd%3A+Using+social+text+spelling+variants+for+evaluating+dialectal+speech+recognition%2E+2017+IEEE+Automatic+Speech+Recognition+and+Under) 3. Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. [Common voice: A massively-multilingual speech corpus](https://md.research.elamur.ai/free/en/papers/ardila-2019-common-voice-massively-multilingual-speech/). In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 421… 4. Samee Arif, Aamina Jamal Khan, Mustafa Abbas, Agha Ali Raza, and Awais Athar. 2025. WER we stand: Benchmarking Urdu ASR models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 5952–5961, Abu Dhabi, UAE. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Samee+Arif%2C+Aamina+Jamal+Khan%2C+Mustafa+Abbas%2C+Agha+Ali+Raza%2C+and+Awais+Athar%2E+2025%2E+WER+we+stand%3A+Benchmarking+Urdu+ASR+models%2E+In+Proceedings+of+the+31st+International+Conference+on+Computational+Lin) 5. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. [Language (technology) is power: A critical survey of “bias” in NLP](https://md.research.elamur.ai/free/en/papers/blodgett-2020-language-technology-is-power-critical/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454– 5476, Online. Association for Computational Linguistics. 6. Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2023. [Fleurs: Few-shot learning evaluation of universal representations of speech](https://md.research.elamur.ai/free/en/papers/conneau-2022-fleurs-few-shot-learning-evaluation/). In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 798–805. 7. Jesse Emond, Bhuvana Ramabhadran, Brian Roark, Pedro Moreno, and Min Ma. 2018. Transliteration based approaches to improve code-switched speech recognition performance. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 448–455. [find](https://scholar.google.com/scholar?q=Jesse+Emond%2C+Bhuvana+Ramabhadran%2C+Brian+Roark%2C+Pedro+Moreno%2C+and+Min+Ma%2E+2018%2E+Transliteration+based+approaches+to+improve+code%2Dswitched+speech+recognition+performance%2E+In+2018+IEEE+Spoken+Language+Te) 8. Thennal D K, Jesin James, Deepa Padmini Gopinath, and Muhammed Ashraf K. 2025. Advocating character error rate for multilingual ASR evaluation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4941–4950, Albuquerque, New Mexico. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Thennal+D+K%2C+Jesin+James%2C+Deepa+Padmini+Gopinath%2C+and+Muhammed+Ashraf+K%2E+2025%2E+Advocating+character+error+rate+for+multilingual+ASR+evaluation%2E+In+Findings+of+the+Association+for+Computational+Linguis) 9. Shigeki Karita, Richard Sproat, and Haruko Ishikawa. 2023. Lenient evaluation of Japanese speech recognition: Modeling naturally occurring spelling inconsistency. In Proceedings of the Workshop on Computation and Written Language (CAWL 2023), pages 61–70, Toronto, Canada. Association for Computation… [find](https://scholar.google.com/scholar?q=Shigeki+Karita%2C+Richard+Sproat%2C+and+Haruko+Ishikawa%2E+2023%2E+Lenient+evaluation+of+Japanese+speech+recognition%3A+Modeling+naturally+occurring+spelling+inconsistency%2E+In+Proceedings+of+the+Workshop+on+Com) 10. Kavya Manohar and Leena G Pillai. 2024. What is lost in normalization? exploring pitfalls in multilingual ASR model evaluations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10864–10869, Miami, Florida, USA. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Kavya+Manohar+and+Leena+G+Pillai%2E+2024%2E+What+is+lost+in+normalization%3F+exploring+pitfalls+in+multilingual+ASR+model+evaluations%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+L) 11. Andrew C. Morris, Viktoria Maier, and Phil D. Green. 2004. From wer and ril to mer and wil: improved evaluation measures for connected speech recognition. In Interspeech. [find](https://scholar.google.com/scholar?q=Andrew+C%2E+Morris%2C+Viktoria+Maier%2C+and+Phil+D%2E+Green%2E+2004%2E+From+wer+and+ril+to+mer+and+wil%3A+improved+evaluation+measures+for+connected+speech+recognition%2E+In+Interspeech%2E) 12. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. [Robust speech recognition via large-scale weak supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/). In Proceedings of the 40th International Conference on Machine Learning (ICML). *Updated 1 November 2026*