---
title: "Building English ASR model with regional language support (2025)"
description: "For acoustics they propose SplitHead with Attention (SHA). 29 references. Among them: 0."
authors: ["Purvi Agrawal", "Vikas Joshi", "Bharati Patidar", "Ankur Gupta", "Rupesh Kumar Mehta"]
year: 2025
venue: "arXiv:2503.07522"
source: "https://arxiv.org/abs/2503.07522"
doi: "10.48550/arxiv.2503.07522"
arxiv: "2503.07522"
licence: "CC BY-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/agrawal-2025-building-english-asr-model-regional/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/agrawal-2025-building-english-asr-model-regional/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/17/) / Building English ASR model with regional language support

# Building English ASR model with regional language support

**Authors:** Purvi Agrawal, Vikas Joshi, Bharati Patidar, Ankur Gupta and Rupesh Kumar Mehta · **Year:** 2025 · **Venue:** arXiv:2503.07522 · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2503.07522) · [PDF](https://arxiv.org/pdf/2503.07522) · [DOI 10.48550/arxiv.2503.07522](https://doi.org/10.48550/arxiv.2503.07522) · [arXiv 2503.07522](https://arxiv.org/abs/2503.07522) · [OpenAlex](https://openalex.org/W6966514696)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Users of an Indian English (en-IN) speech recognizer often speak Hindi or switch between languages, even when English is their selected setting. Running separate English and Hindi recognizers behind a language identifier costs more compute and breaks down when the identifier is wrong. The Microsoft authors instead upgrade a single hybrid en-IN ASR system so it handles Hindi much better, with no loss on English and no extra compute, so it can directly replace the deployed model. For acoustics they propose SplitHead with Attention (SHA). Shared hidden layers feed two language-specific projection heads that use one common chenone inventory, and a jointly trained self-attention module weights and blends the two heads. Hindi audio is transliterated into Latin script so the English alignment model can label it. Training runs in four stages and can be followed by teacher-student distillation. For language modeling, an English n-gram LM is interpolated with one built from contextually transliterated Hindi text. Compared with a monolingual en-IN baseline, relative word error rate drops 69.3% on Hindi and 5.7% on English.

## Key points
- Motivation from live traffic: about a third of en-IN queries were fully in regional languages, and over 30% of call-center queries involved code-switching.
- SHA combines the two heads' outputs with scalar weights (w_en, w_hi) that self-attention computes over a chunk of lookahead frames. Both heads must share the primary language's chenone set, which is how SHA differs from Shared Hidden Layer (SHL) models.
- Training proceeds in four stages:
  - a data-pooled SingleHead model;
  - a SplitHead model initialized from it, with each head trained on its own language;
  - attention-only training with all other weights frozen, including code-mixed data;
  - full joint fine-tuning.
- Distillation uses a teacher ensemble of a non-streaming pooled model (weight 0.4) and the streaming SHA student itself (0.6). The loss mixes CE and KLD terms with w_KLD = 0.95.
- The Hinglish LM is built by interpolating n-grams rather than training a neural LM, which keeps it light, fast and easy to edit. Its corpora come from news, Wikipedia, social media, podcasts and knowledge-graph entities.

## Interesting ideas
- The secondary language is forced into the primary language's label inventory by running it through the primary aligner after transliteration, so outputs from different branches stay directly comparable and can be summed.
- The gating module is trained alone while everything else is frozen, then the whole model is unfrozen, so the router learns on top of experts that are already competent.
- The student is included in its own teacher ensemble so that distillation from a stronger, non-deployable model cannot drag it far from its prior behavior.
- Non-regression on the primary language is treated as a hard design constraint and checked on separate test sets, rather than optimizing one blended score.
- Transliteration is done in context instead of word by word, since romanized words are ambiguous out of context.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tree): Treebanks in other UD dialects could be mapped into Mova's primary label scheme before mixing, so outputs from parsers trained on different sources can be combined label by label.
- **I2** (learn): When adding a selector that chooses between the perceptron parser and gparse per sentence, Mova could first train only the selector on fixed parsers, then fine-tune jointly.
- **I3** (learn): Mixing the current Mova model into LLM teacher tiers when producing silver data could limit drift and prevent regressions on cases it already handles.
- **I4** (learn): Every new domain or dialect addition should report the core English benchmark beside the new one, as a built-in negative control.
- **I5** (tokens): Romanized foreign names and loanwords in English text could get lemma and morphology decisions that use the surrounding sentence, not a fixed per-word lookup.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 493 works

🧭 Topics (2): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,379 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 519 works

## Cite

Cite as: Purvi Agrawal, Vikas Joshi, Bharati Patidar, Ankur Gupta and Rupesh Kumar Mehta. 2025. Building English ASR model with regional language support. arXiv:2503.07522.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{agrawal2025building,
  title = {{Building English ASR model with regional language support}},
  author = {Purvi Agrawal, Vikas Joshi, Bharati Patidar, Ankur Gupta and Rupesh Kumar Mehta},
  year = {2025},
  doi = {10.48550/arxiv.2503.07522},
  eprint = {2503.07522},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.07522}
}
```

- [← previous Bridging Laughter Across Languages: Generation of Hindi-English Code-mixed Puns](https://md.research.elamur.ai/free/en/papers/asapu-2025-bridging-laughter-across-languages-generation/)
- [next → Building Patient Journeys in Hebrew: A Language Model for Clinical Timeline Extraction](https://md.research.elamur.ai/free/en/papers/hashiloni-2025-building-patient-journeys-hebrew-language/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/17/)

## References

29 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [1] Vikas Joshi, Amit Das, Eric Sun, Rupesh R Mehta, Jinyu Li, and Yifan Gong, “Multiple softmax architecture for streaming multilingual end-to-end ASR systems,” in Proc. Interspeech, 2021, pp. 1767–1771. [find](https://scholar.google.com/scholar?q=%5B1%5D+Vikas+Joshi%2C+Amit+Das%2C+Eric+Sun%2C+Rupesh+R+Mehta%2C+Jinyu+Li%2C+and+Yifan+Gong%2C+%E2%80%9CMultiple+softmax+architecture+for+streaming+multilingual+end%2Dto%2Dend+ASR+systems%2C%E2%80%9D+in+Proc%2E+Interspeech%2C+2021%2C+pp%2E+1767%E2%80%931)
2. [2] Duc Le, Xiaohui Zhang, Weiyi Zheng, Christian Fügen, Geoffrey Zweig, and Michael L Seltzer, “From senones to chenones: Tied context-dependent graphemes for hybrid speech recognition,” in Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019, pp. 457–464. [find](https://scholar.google.com/scholar?q=%5B2%5D+Duc+Le%2C+Xiaohui+Zhang%2C+Weiyi+Zheng%2C+Christian+Fu%CC%88gen%2C+Geoffrey+Zweig%2C+and+Michael+L+Seltzer%2C+%E2%80%9CFrom+senones+to+chenones%3A+Tied+context%2Ddependent+graphemes+for+hybrid+speech+recognition%2C%E2%80%9D+in+Automati)
3. [3] J. Huang, J. Li, D. Yu, L. Deng, and Y. Gong, “Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers,” in Proceedings of ICASSP, 2013, pp. 7304–7308. [find](https://scholar.google.com/scholar?q=%5B3%5D+J%2E+Huang%2C+J%2E+Li%2C+D%2E+Yu%2C+L%2E+Deng%2C+and+Y%2E+Gong%2C+%E2%80%9CCross%2Dlanguage+knowledge+transfer+using+multilingual+deep+neural+network+with+shared+hidden+layers%2C%E2%80%9D+in+Proceedings+of+ICASSP%2C+2013%2C+pp%2E+7304%E2%80%937308%2E)
4. [4] S. Thomas, S. Ganapathy, and H. Hermansky, “Multilingual MLP features for low-resource LVCSR systems,” in Proceedings of ICASSP, 2012, pp. 4269–4272. [find](https://scholar.google.com/scholar?q=%5B4%5D+S%2E+Thomas%2C+S%2E+Ganapathy%2C+and+H%2E+Hermansky%2C+%E2%80%9CMultilingual+MLP+features+for+low%2Dresource+LVCSR+systems%2C%E2%80%9D+in+Proceedings+of+ICASSP%2C+2012%2C+pp%2E+4269%E2%80%934272%2E)
5. [5] Pawel Swietojanski, Arnab Ghoshal, and Steve Renals, “Unsupervised cross-lingual knowledge transfer in dnn-based lvcsr,” in IEEE Spoken Language Technology Workshop (SLT). IEEE, 2012, pp. 246–251. [find](https://scholar.google.com/scholar?q=%5B5%5D+Pawel+Swietojanski%2C+Arnab+Ghoshal%2C+and+Steve+Renals%2C+%E2%80%9CUnsupervised+cross%2Dlingual+knowledge+transfer+in+dnn%2Dbased+lvcsr%2C%E2%80%9D+in+IEEE+Spoken+Language+Technology+Workshop+%28SLT%29%2E+IEEE%2C+2012%2C+pp%2E+246%E2%80%93251%2E)
6. [6] G. Heigold, V. Vanhoucke, A. Senior, P. Nguyen, M. Ranzato, M. Devin, and J. Dean, “Multilingual acoustic models using distributed deep neural networks,” in Proceedings of ICASSP, 2013, pp. 8619–8623. [find](https://scholar.google.com/scholar?q=%5B6%5D+G%2E+Heigold%2C+V%2E+Vanhoucke%2C+A%2E+Senior%2C+P%2E+Nguyen%2C+M%2E+Ranzato%2C+M%2E+Devin%2C+and+J%2E+Dean%2C+%E2%80%9CMultilingual+acoustic+models+using+distributed+deep+neural+networks%2C%E2%80%9D+in+Proceedings+of+ICASSP%2C+2013%2C+pp%2E+8619%E2%80%938)
7. [7] Julius Kunze, Louis Kirsch, Ilia Kurenkov, Andreas Krug, Jens Johannsmeier, and Sebastian Stober, “Transfer learning for speech recognition on a budget,” in Proceedings of the 2nd Workshop on Representation Learning for NLP, Vancouver, Canada, Aug. 2017, pp. 168–177, ACL. [find](https://scholar.google.com/scholar?q=%5B7%5D+Julius+Kunze%2C+Louis+Kirsch%2C+Ilia+Kurenkov%2C+Andreas+Krug%2C+Jens+Johannsmeier%2C+and+Sebastian+Stober%2C+%E2%80%9CTransfer+learning+for+speech+recognition+on+a+budget%2C%E2%80%9D+in+Proceedings+of+the+2nd+Workshop+on+Repr)
8. [8] Yan Huang, Dong Yu, Chaojun Liu, and Yifan Gong, “Multi-accent deep neural network acoustic model with accent-specific top layer using the KLD-regularized model adaptation,” in Proceedings of Interspeech, 2014. [find](https://scholar.google.com/scholar?q=%5B8%5D+Yan+Huang%2C+Dong+Yu%2C+Chaojun+Liu%2C+and+Yifan+Gong%2C+%E2%80%9CMulti%2Daccent+deep+neural+network+acoustic+model+with+accent%2Dspecific+top+layer+using+the+KLD%2Dregularized+model+adaptation%2C%E2%80%9D+in+Proceedings+of+Inte)
9. [9] S. Scanzio, P. Laface, L. Fissore, R. Gemello, and F. Mana, “On the Use of a Multilingual Neural Network Front-End,” in Proceedings of Interspeech, 2008, pp. 2711–2714. [find](https://scholar.google.com/scholar?q=%5B9%5D+S%2E+Scanzio%2C+P%2E+Laface%2C+L%2E+Fissore%2C+R%2E+Gemello%2C+and+F%2E+Mana%2C+%E2%80%9COn+the+Use+of+a+Multilingual+Neural+Network+Front%2DEnd%2C%E2%80%9D+in+Proceedings+of+Interspeech%2C+2008%2C+pp%2E+2711%E2%80%932714%2E)
10. [12] A. Das, K. Kumar, and J. Wu, “Multi-Dialect Speech Recognition in English Using Attention on Ensemble,” in Proceedings of ICASSP, 2021. [find](https://scholar.google.com/scholar?q=%5B12%5D+A%2E+Das%2C+K%2E+Kumar%2C+and+J%2E+Wu%2C+%E2%80%9CMulti%2DDialect+Speech+Recognition+in+English+Using+Attention+on+Ensemble%2C%E2%80%9D+in+Proceedings+of+ICASSP%2C+2021%2E)
11. [13] Surabhi Punjabi, Harish Arsikere, Zeynab Raeesy, Chander Chandak, Nikhil Bhave, Ankish Bansal, Markus Müller, Sergio Murillo, Ariya Rastrow, Sri Garimella, et al., “Streaming end-to-end bilingual ASR systems with joint language identification,” arXiv preprint arXiv:2007.03900, 2020. [↗](https://arxiv.org/abs/2007.03900)
12. [14] Charl Van Heerden, Neil Kleynhans, Etienne Barnard, and Marelie Davel, “Pooling asr data for closely related languages,” 2010. [↗](https://doi.org/10.13140/rg.2.1.2250.1845)
13. [15] Javier Gonzalez-Dominguez, David Eustis, Ignacio Lopez-Moreno, Andrew Senior, Françoise Beaufays, and Pedro J Moreno, “A real-time end-to-end multilingual speech recognition architecture,” IEEE Journal of selected topics in signal processing, vol. 9, no. 4, pp. 749–759, 2014. [↗](https://doi.org/10.1109/jstsp.2014.2364559)
14. [16] Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, Shruti Gupta, Subhash Chandra Bose Gali, Vish Subramanian, and Partha P. Talukdar, “[Muril: Multilingual representations for indian languages](https://md.research.elamur.ai/free/en/papers/khanuja-2021-muril-multilingual-representations-indian-languages/),” CoRR, vol. abs/2103.10730, 2021. [↗](https://arxiv.org/abs/2103.10730)
15. [17] Reinhard Kneser and Volker Steinbiss, “On the dynamic adaptation of stochastic language models,” in 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing. [find](https://scholar.google.com/scholar?q=%5B17%5D+Reinhard+Kneser+and+Volker+Steinbiss%2C+%E2%80%9COn+the+dynamic+adaptation+of+stochastic+language+models%2C%E2%80%9D+in+1993+IEEE+International+Conference+on+Acoustics%2C+Speech%2C+and+Signal+Processing%2E)
16. [18] Dietrich Klakow et al., “Log-linear interpolation of language models.,” in ICSLP. Citeseer, 1998. [find](https://scholar.google.com/scholar?q=%5B18%5D+Dietrich+Klakow+et+al%2E%2C+%E2%80%9CLog%2Dlinear+interpolation+of+language+models%2E%2C%E2%80%9D+in+ICSLP%2E+Citeseer%2C+1998%2E)
17. [19] Cyril Allauzen and Michael Riley, “Bayesian language model interpolation for mobile speech input,” 2011. [find](https://scholar.google.com/scholar?q=%5B19%5D+Cyril+Allauzen+and+Michael+Riley%2C+%E2%80%9CBayesian+language+model+interpolation+for+mobile+speech+input%2C%E2%80%9D+2011%2E)
18. [20] Ngoc Thang Vu, Dau-Cheng Lyu, Jochen Weiner, Dominic Telaar, Tim Schlippe, Fabian Blaicher, Eng-Siong Chng, Tanja Schultz, and Haizhou Li, “A first speech recognition system for mandarin-english code-switch conversational speech,” in Proceedings of ICASSP. IEEE, 2012, pp. 4889–4892. [find](https://scholar.google.com/scholar?q=%5B20%5D+Ngoc+Thang+Vu%2C+Dau%2DCheng+Lyu%2C+Jochen+Weiner%2C+Dominic+Telaar%2C+Tim+Schlippe%2C+Fabian+Blaicher%2C+Eng%2DSiong+Chng%2C+Tanja+Schultz%2C+and+Haizhou+Li%2C+%E2%80%9CA+first+speech+recognition+system+for+mandarin%2Denglish+)
19. [21] Anirudh Raju, Behnam Hedayatnia, Linda Liu, Ankur Gandhe, Chandra Khatri, Angeliki Metallinou, Anu Venkatesh, and Ariya Rastrow, “Contextual language model adaptation for conversational agents,” arXiv preprint:1806.10215, 2018. [↗](https://arxiv.org/abs/1806.10215)
20. [22] Saurabh Garg, Tanmay Parekh, and Preethi Jyothi, “Dual language models for code switched speech recognition,” arXiv preprint arXiv:1711.01048, 2017. [↗](https://arxiv.org/abs/1711.01048)
21. [23] Saurabh Garg, Tanmay Parekh, and Preethi Jyothi, “Code-switched language models using dual RNNs and same-source pretraining,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, Oct.-Nov. 2018, pp. 3078–3083, Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=%5B23%5D+Saurabh+Garg%2C+Tanmay+Parekh%2C+and+Preethi+Jyothi%2C+%E2%80%9CCode%2Dswitched+language+models+using+dual+RNNs+and+same%2Dsource+pretraining%2C%E2%80%9D+in+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural)
22. [10] M. L. Seltzer and J. Droppo, “Multi-task learning in deep neural networks for improved phoneme recognition,” in Proceedings of ICASSP, 2013, pp. 6965–6969. [find](https://scholar.google.com/scholar?q=%5B10%5D+M%2E+L%2E+Seltzer+and+J%2E+Droppo%2C+%E2%80%9CMulti%2Dtask+learning+in+deep+neural+networks+for+improved+phoneme+recognition%2C%E2%80%9D+in+Proceedings+of+ICASSP%2C+2013%2C+pp%2E+6965%E2%80%936969%2E)
23. [24] Khyathi Chandu, Thomas Manzini, Sumeet Singh, and Alan W. Black, “Language informed modeling of code-switched text,” in Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, Melbourne, Australia, July 2018, pp. 92–97, Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=%5B24%5D+Khyathi+Chandu%2C+Thomas+Manzini%2C+Sumeet+Singh%2C+and+Alan+W%2E+Black%2C+%E2%80%9CLanguage+informed+modeling+of+code%2Dswitched+text%2C%E2%80%9D+in+Proceedings+of+the+Third+Workshop+on+Computational+Approaches+to+Linguistic)
24. [11] Austin Waters, Meysam Bastani, Mohamed G. Elfeky, Pedro Moreno, and Xavier Velez, “Towards acoustic model unification across dialects,” in IEEE Workshop on Spoken Language Technology, 2016. [find](https://scholar.google.com/scholar?q=%5B11%5D+Austin+Waters%2C+Meysam+Bastani%2C+Mohamed+G%2E+Elfeky%2C+Pedro+Moreno%2C+and+Xavier+Velez%2C+%E2%80%9CTowards+acoustic+model+unification+across+dialects%2C%E2%80%9D+in+IEEE+Workshop+on+Spoken+Language+Technology%2C+2016%2E)
25. [25] Aditya Patil, Vikas Joshi, Purvi Agrawal, and Rupesh Mehta, “Streaming bilingual end-to-end asr model using attention over multiple softmax,” in 2022 IEEE Spoken Language Technology Workshop (SLT). [find](https://scholar.google.com/scholar?q=%5B25%5D+Aditya+Patil%2C+Vikas+Joshi%2C+Purvi+Agrawal%2C+and+Rupesh+Mehta%2C+%E2%80%9CStreaming+bilingual+end%2Dto%2Dend+asr+model+using+attention+over+multiple+softmax%2C%E2%80%9D+in+2022+IEEE+Spoken+Language+Technology+Workshop+%28SLT)
26. [26] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin, “[Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/),” Advances in neural information processing systems, vol. 30, 2017.
27. [27] A Madhavaraj and AG Ramakrishnan, “Data-pooling and multi-task learning for enhanced performance of speech recognition systems in multiple low resourced languages,” in National Conf. on Communications (NCC). IEEE, 2019, pp. 1–5. [find](https://scholar.google.com/scholar?q=%5B27%5D+A+Madhavaraj+and+AG+Ramakrishnan%2C+%E2%80%9CData%2Dpooling+and+multi%2Dtask+learning+for+enhanced+performance+of+speech+recognition+systems+in+multiple+low+resourced+languages%2C%E2%80%9D+in+National+Conf%2E+on+Communica)
28. [28] Vikas Joshi, Rui Zhao, Rupesh R. Mehta, Kshitiz Kumar, and Jinyu Li, “Transfer Learning Approaches for Streaming End-to-End Speech Recognition System,” in Proceedings of Interspeech 2020, 2020, pp. 2152–2156. [find](https://scholar.google.com/scholar?q=%5B28%5D+Vikas+Joshi%2C+Rui+Zhao%2C+Rupesh+R%2E+Mehta%2C+Kshitiz+Kumar%2C+and+Jinyu+Li%2C+%E2%80%9CTransfer+Learning+Approaches+for+Streaming+End%2Dto%2DEnd+Speech+Recognition+System%2C%E2%80%9D+in+Proceedings+of+Interspeech+2020%2C+2020%2C+p)
29. [29] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al., “[Conformer: Convolution-augmented transformer for speech recognition](https://md.research.elamur.ai/free/en/papers/gulati-2020-conformer-convolution-augmented-transformer-speech/),” arXiv preprint:2005.08100, 2020. [↗](https://arxiv.org/abs/2005.08100)

*Updated 1 November 2026*