---
title: "Svarah: Evaluating English ASR Systems on Indian Accents (2023)"
description: "Those test sets, however, contain mostly native speakers. 15 references, cited by 8 papers here."
authors: ["Tahir Javed", "Sakshi Joshi", "Vignesh Nagarajan", "Sai Sundaresan", "Janki Nawale", "Abhigyan Raman", "Kaushal Bhogale", "Pratyush Kumar", "Mitesh M. Khapra"]
year: 2023
venue: "arXiv:2305.15760"
source: "https://arxiv.org/abs/2305.15760"
doi: "10.48550/arxiv.2305.15760"
arxiv: "2305.15760"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/javed-2023-svarah-evaluating-english-asr-systems/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/javed-2023-svarah-evaluating-english-asr-systems/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/) / Svarah: Evaluating English ASR Systems on Indian Accents

# Svarah: Evaluating English ASR Systems on Indian Accents

**Authors:** Tahir Javed, Sakshi Joshi, Vignesh Nagarajan, Sai Sundaresan, [Janki Nawale](https://md.research.elamur.ai/free/en/authors/janki-nawale/), Abhigyan Raman, Kaushal Bhogale, [Pratyush Kumar](https://md.research.elamur.ai/free/en/authors/pratyush-kumar/) and [Mitesh M. Khapra](https://md.research.elamur.ai/free/en/authors/mitesh-m-khapra/) · **Year:** 2023 · **Venue:** [arXiv:2305.15760](https://md.research.elamur.ai/free/en/venues/interspeech-usb/) ([Interspeech 2023](https://md.research.elamur.ai/free/en/venues/interspeech-usb/2023/)) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2305.15760) · [PDF](https://arxiv.org/pdf/2305.15760) · [DOI 10.48550/arxiv.2305.15760](https://doi.org/10.48550/arxiv.2305.15760) · [arXiv 2305.15760](https://arxiv.org/abs/2305.15760) · [OpenAlex](https://openalex.org/W4385822371)

**Citations:** 14 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/javed-2023-svarah-evaluating-english-asr-systems/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
English ASR systems now report very low WERs on benchmarks such as LibriSpeech. Those test sets, however, contain mostly native speakers. Existing accent collections (Speech Accent Archive, FAE, AccentDB, IITM/NPTEL) cover Indian English only thinly or in narrow domains.

The authors built Svarah, an evaluation set of 9.6 hours of transcribed Indian-accented English. It comes from 117 speakers in 65 districts across 19 states, whose first languages cover 19 of India's 22 constitutionally recognized languages. Recordings were collected remotely with the Karya Android app and mix three kinds of speech:
- reading Wikipedia sentences from nine domains,
- unscripted answers to simple questions about hobbies and subject areas,
- assistant-style commands for groceries, digital payments and government services.

Students segmented and transcribed the audio following Switchboard conventions. The authors compared six open-source models and the Google and Microsoft APIs. Whisper, trained on massive weakly supervised web audio, led the open models. Two open models outperformed both commercial systems and varied little across accents, yet clear headroom remains. The work shows that benchmarks lacking accent diversity overstate robustness.

## Key points
- 117 speakers (54 men, 63 women), 3–5 per native language, roughly balanced across the age bands 18–30, 30–45, 45–60 and 60+, with varied fields and levels of education.
- The read portion draws on 1k Wikipedia sentences from 9 domains, picked via article categories. Each speaker read 5 sentences, and no sentence was given to two speakers.
- The spontaneous portion uses 28 hobby topics plus the 9 domains, with four questions per chosen item. The use-case portion gives five prompts per scenario.
- Prior coverage was thin: SAA has 59 Indian speakers (26 minutes), FAE covers only Hindi and Tamil speakers (0.5 h and 1 h), and AccentDB has 8 speakers.
- The evaluated systems span three paradigms:
  - Whisper (438K hours of weak supervision),
  - Wav2Vec2, HuBERT, WavLM and Data2Vec (self-supervised, fine-tuned on 960 h),
  - the NVIDIA Conformer (24.5K labeled hours),
  - plus Google and Microsoft commercial APIs.

## Interesting ideas
- Recruit evaluation contributors by explicit strata (first language, age, education) so that every subgroup is large enough to score separately.
- Treat low spread of error across subgroups as a robustness signal in its own right, alongside the mean score.
- Elicit three styles of input for one benchmark: scripted reading, free spontaneous talk, and task-oriented commands.
- Give each contributor different items instead of repeating the same ones, which widens vocabulary coverage for a fixed budget.
- Use existing metadata such as Wikipedia categories to sample text that is balanced across domains, without manual curation.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): build test sets stratified by text source and register (fables, textbooks, news, user prose) and report per-stratum results, so that a strong average cannot hide a weak genre.
- **I2** (tree): report LAS for gparse and the perceptron parser per treebank or dialect source together with its standard deviation, and prefer models that stay stable rather than ones that peak on a single treebank.
- **I3** (frames): include imperative transactional sentences (pay X from account Y to Z) next to narrative text, to check that frame and role extraction holds up outside storytelling prose.
- **I4** (learn): when spending LLM-teacher or human effort on evaluation data, favour unique sentences for coverage, and keep duplicated items only where witness agreement is actually needed.
- **I5** (world): draw domain-balanced probe texts by Wikipedia category to measure where the WordNet/Wikidata-backed ontology and verb classes lack coverage.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 452 works

🧭 Topics (2): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 480 works

## Cite

Cite as: Tahir Javed, Sakshi Joshi, Vignesh Nagarajan, Sai Sundaresan, Janki Nawale, Abhigyan Raman, Kaushal Bhogale, Pratyush Kumar and Mitesh M. Khapra. 2023. Svarah: Evaluating English ASR Systems on Indian Accents. arXiv:2305.15760.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{javed2023svarah,
  title = {{Svarah: Evaluating English ASR Systems on Indian Accents}},
  author = {Tahir Javed, Sakshi Joshi, Vignesh Nagarajan, Sai Sundaresan, Janki Nawale, Abhigyan Raman, Kaushal Bhogale, Pratyush Kumar and Mitesh M. Khapra},
  year = {2023},
  doi = {10.48550/arxiv.2305.15760},
  eprint = {2305.15760},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.15760}
}
```

- [← previous Rewarding Chatbots for Real-World Engagement with Millions of Users](https://md.research.elamur.ai/free/en/papers/irvine-2023-rewarding-chatbots-real-world-engagement/)
- [next → AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processing](https://md.research.elamur.ai/free/en/papers/alghamdi-2023-aramus-pushing-limits-data-model/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/)

## References

15 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “[Robust speech recognition via large-scale weak supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/),” CoRR, vol. abs/2212.04356, 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2212.04356 [↗](https://doi.org/10.48550/arxiv.2212.04356)
2. W. Xiong, J. Droppo, X. Huang, F. Seide, M. L. Seltzer, A. Stolcke, D. Yu, and G. Zweig, “Toward human parity in conversational speech recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 12, pp. 2410– 2423, 2017. [↗](https://doi.org/10.1109/taslp.2017.2756440)
3. A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “[wav2vec 2.0: A framework for self-supervised learning of speech representations](https://md.research.elamur.ai/free/en/papers/baevski-2020-wav2vec-2-0-framework-self/),” in Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., 2020. [Online]. Available: [https://proceedings.neurips.cc/paper/2020/hash/92d1e1eb1cd6f9fba3227870bb6d7f07-Abstract.html](https://proceedings.neurips.cc/paper/2020/hash/92d1e1eb1cd6f9fba3227870bb6d7f07-Abstract.html) [↗](https://proceedings.neurips.cc/paper/2020/hash/92d1e1eb1cd6f9fba3227870bb6d7f07-Abstract.html)
4. S. Hollands, D. Blackburn, and H. Christensen, “Evaluating the performance of state-of-the-art ASR systems on non-native english using corpora with extensive language background variation,” in Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022, H. Ko and J. H. L. Hansen, Eds. ISCA, 2022, pp. 3958–3962. [Online]. Available: https://doi.org/10.21437/Interspeech.2022-10433 [↗](https://doi.org/10.21437/interspeech.2022-10433)
5. A. Gulati, J. Qin, C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “[Conformer: Convolution-augmented transformer for speech recognition](https://md.research.elamur.ai/free/en/papers/gulati-2020-conformer-convolution-augmented-transformer-speech/),” in Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020, H. Meng, B. Xu, and T. F. Zheng, Eds. ISCA, 2020, pp. 5036–5040. [Online]. Available: https://doi.org/10.21437/Interspeech.2020-3015 [↗](https://doi.org/10.21437/interspeech.2020-3015)
6. V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An asr corpus based on public domain audio books,” in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206–5210. [find](https://scholar.google.com/scholar?q=V%2E+Panayotov%2C+G%2E+Chen%2C+D%2E+Povey%2C+and+S%2E+Khudanpur%2C+%E2%80%9CLibrispeech%3A+An+asr+corpus+based+on+public+domain+audio+books%2C%E2%80%9D+in+2015+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28IC)
7. J. J. Godfrey, E. Holliman, and J. McDaniel, “Switchboard: telephone speech corpus for research and development,” [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 1, pp. 517–520 vol.1, 1992. [find](https://scholar.google.com/scholar?q=J%2E+J%2E+Godfrey%2C+E%2E+Holliman%2C+and+J%2E+McDaniel%2C+%E2%80%9CSwitchboard%3A+telephone+speech+corpus+for+research+and+development%2C%E2%80%9D+%5BProceedings%5D+ICASSP%2D92%3A+1992+IEEE+International+Conference+on+Acoustics%2C+Speech%2C+and+)
8. L. Drude, J. Heitkaemper, C. Boeddeker, and R. Haeb-Umbach, “Sms-wsj: Database, performance measures, and baseline recipe for multi-channel source separation and recognition,” ArXiv, vol. abs/1910.13934, 2019. [↗](https://arxiv.org/abs/1910.13934)
9. J. Yamagishi, C. Veaux, and K. MacDonald, “Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92),” 2019. 8 https://www.meity.gov.in/9 https://www.cdac.in/index.aspx?id=pune [↗](https://doi.org/10.7488/ds/2645)
10. C. Wang, M. Riviere, A. Lee, A. Wu, C. Talnikar, D. Haziza, M. Williamson, J. Pino, and E. Dupoux, “[VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation](https://md.research.elamur.ai/free/en/papers/wang-2021-voxpopuli-large-scale-multilingual-speech/),” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Online: Association for Computational Linguistics, Aug. 2021, pp. 993–1003. [Online]. Available: https://aclanthology.org/2021. acl-long.80 [↗](https://doi.org/10.60692/73vwj-qzd32)
11. R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “[Common voice: A massively-multilingual speech corpus](https://md.research.elamur.ai/free/en/papers/ardila-2019-common-voice-massively-multilingual-speech/),” in International Conference on Language Resources and Evaluation, 2019. [↗](https://doi.org/10.48550/arxiv.1912.06670)
12. T. Lander, “CSLU: Foreign Accented English Release 1.2,” 2022. [Online]. Available: https://doi.org/10.5683/SP2/K7EQTE [↗](https://doi.org/10.5683/sp2/k7eqte)
13. A. Ahamad, A. Anand, and P. Bhargava, “[Accentdb: A database of non-native english accents to assist neural speech recognition](https://md.research.elamur.ai/free/en/papers/ahamad-2020-accentdb-database-non-native-english/),” in Proceedings of The 12th Language Resources and Evaluation Conference. Marseille, France: European Language Resources Association, May 2020, pp. 5353–5360. [Online]. Available: https://www.aclweb.org/anthology/2020.lrec-1.659 [↗](https://aclanthology.org/2020.lrec-1.659/)
14. B. Abraham, D. Goel, D. Siddarth, K. Bali, M. Chopra, M. Choudhury, P. Joshi, P. Jyothi, S. Sitaram, and V. Seshadri, “Crowdsourcing speech data for low-resource languages from low-income workers,” in Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020, N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, and S. Piperidis, Eds. European Language Resources Association, 2020, pp. 2819–2826. [Online]. Available: https://aclanthology.org/2020.lrec-1.343/ [↗](https://aclanthology.org/2020.lrec-1.343/)
15. T. Javed, K. S. Bhogale, A. Raman, A. Kunchukuttan, P. Kumar, and M. M. Khapra, “Indicsuperb: A speech processing universal performance benchmark for indian languages,” CoRR, vol. abs/2208.11761, 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2208.11761 [↗](https://doi.org/10.48550/arxiv.2208.11761)

*Updated 1 November 2026*