---
title: "Open Universal Arabic ASR Leaderboard — Wang 2024"
description: "The authors build a continuously updated, open leaderboard, modelled on the English Open ASR Leaderboard, for… 31 references, cited by 11 papers here."
authors: ["Yingzhi Wang", "Anas Alhmoud", "Muhammad Alqurishi"]
year: 2024
venue: "arXiv:2412.13788"
source: "https://arxiv.org/abs/2412.13788"
doi: "10.48550/arxiv.2412.13788"
arxiv: "2412.13788"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2024-open-universal-arabic-asr-leaderboard/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2024-open-universal-arabic-asr-leaderboard/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Open Universal Arabic ASR Leaderboard

# Open Universal Arabic ASR Leaderboard

**Authors:** [Yingzhi Wang](https://md.research.elamur.ai/free/en/authors/yingzhi-wang/), Anas Alhmoud and [Muhammad Alqurishi](https://md.research.elamur.ai/free/en/authors/muhammad-alqurishi/) · **Year:** 2024 · **Venue:** arXiv:2412.13788 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2412.13788) · [PDF](https://arxiv.org/pdf/2412.13788) · [DOI 10.48550/arxiv.2412.13788](https://doi.org/10.48550/arxiv.2412.13788) · [arXiv 2412.13788](https://arxiv.org/abs/2412.13788) · [OpenAlex](https://openalex.org/W4415433636)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [11](https://md.research.elamur.ai/free/en/papers/wang-2024-open-universal-arabic-asr-leaderboard/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Arabic speech recognition is moving toward single models that are expected to handle every dialect, but earlier benchmarks tested only a few models, often on one dataset or on corpora covering just one dialect. The authors build a continuously updated, open leaderboard, modelled on the English Open ASR Leaderboard, for general-purpose open-source Arabic ASR systems. They run zero-shot inference over four model families: Whisper variants, NVIDIA's Conformer-CTC, community fine-tuned self-supervised models (wav2vec 2.0, XLS-R, MMS, w2v-BERT, HuBERT) and Seamless M4T. Evaluation uses only multi-dialectal test sets, and models are ranked by average WER, with CER also reported. Beyond accuracy, the paper examines robustness, speaker adaptation, inference speed and memory use. The Conformer-CTC model paired with a 4-gram language model comes out on top, ahead of whisper-large-v3. The benchmark gives the community a shared reference point and a common evaluation protocol for dialect-general Arabic ASR.

## Key points
- The benchmark has five test sets: SADA (Test), Common Voice 18.0 (Test), MASC Test-Clean, MASC Test-Noisy and MGB-2 (Test). All of them are multi-dialectal.
- The leaderboard has 14 entries. nvidia/conformer-ctc-large-ar ranks first at 25.71% average WER with the LM and second at 27.46% with greedy decoding; whisper-large-v3 is third at 29.87%.
- The last-placed entry is speechbrain/wav2vec2-large-ft at 60.15% average WER. whisper-large-v3-turbo (33.30%) scores slightly below v3, a modest cost for its 4 decoder layers instead of 32.
- Seamless M4T supports three Arabic varieties (MSA, Moroccan, Egyptian). It was run once per setting, and only its best score was reported.
- All results are zero-shot: no model was adapted to the benchmark data.

## Interesting ideas
- The authors refuse to judge dialect ability by comparing scores across separate single-dialect corpora, because recording conditions and other corpus traits confound the comparison; they test only on mixed-dialect sets instead.
- The same model appears twice on the leaderboard, once with greedy decoding and once with a language-model beam search, so the gain from the decoding add-on is directly visible.
- MASC is scored as separate clean and noisy splits, a cheap robustness probe that keeps the content distribution constant while degrading the input.
- Models are ranked by an average over several heterogeneous test sets rather than a single headline corpus, which discourages overfitting to one benchmark.
- When a system needs a configuration choice (here, the language setting), the paper reports its best configuration; this is convenient but slightly favours that system.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tree): When measuring how well a parser handles a UD dialect or genre, compare within one treebank that mixes the varieties, rather than across treebanks whose annotation habits differ.
- **I2** (tree): Report perceptron and gparse both bare and with extra constraints or rescoring as separate rows, so each component's contribution can be explained.
- **I3** (tokens): Build a noisy copy of a tagging test set (typos, dropped punctuation, casing changes) to measure how much degradation alone costs the tokenizer and tagger.
- **I4** (learn): Score each training round on an average across several held-out sources, such as school tests, fables and word problems, so a gain on one source cannot hide regressions elsewhere.
- **I5** (answer): If readers are run with several settings, record which setting was picked and how, ideally on a dev split, so oracle selection does not inflate results.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · [🐪 Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/)

## Cite

Cite as: Yingzhi Wang, Anas Alhmoud and Muhammad Alqurishi. 2024. Open Universal Arabic ASR Leaderboard. arXiv:2412.13788.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wang2024open,
  title = {{Open Universal Arabic ASR Leaderboard}},
  author = {Yingzhi Wang, Anas Alhmoud and Muhammad Alqurishi},
  year = {2024},
  doi = {10.48550/arxiv.2412.13788},
  eprint = {2412.13788},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.13788}
}
```

- [← previous Major Entity Identification: A Generalizable Alternative to Coreference Resolution](https://md.research.elamur.ai/free/en/papers/manikantan-2024-major-entity-identification-generalizable-alternative/)
- [next → A Challenge Dataset and Effective Models for Conversational Stance Detection](https://md.research.elamur.ai/free/en/papers/niu-2024-challenge-dataset-effective-models-conversational/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

31 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Dhouib, A. Othman, O. El Ghoul, M. K. Khribi, and A. Al Sinani, “Arabic automatic speech recognition: a systematic literature review,” Applied Sciences, vol. 12, no. 17, p. 8898, 2022. [find](https://scholar.google.com/scholar?q=A%2E+Dhouib%2C+A%2E+Othman%2C+O%2E+El+Ghoul%2C+M%2E+K%2E+Khribi%2C+and+A%2E+Al+Sinani%2C+%E2%80%9CArabic+automatic+speech+recognition%3A+a+systematic+literature+review%2C%E2%80%9D+Applied+Sciences%2C+vol%2E+12%2C+no%2E+17%2C+p%2E+8898%2C+2022%2E)
2. A. A. Abdelhamid, H. A. Alsayadi, I. Hegazy, and Z. T. Fayed, “End-to-end arabic speech recognition: A review,” in Proceedings of the 19th Conference of Language Engineering (ESOLEC’19), Alexandria, Egypt, 2020, pp. 26–30. [find](https://scholar.google.com/scholar?q=A%2E+A%2E+Abdelhamid%2C+H%2E+A%2E+Alsayadi%2C+I%2E+Hegazy%2C+and+Z%2E+T%2E+Fayed%2C+%E2%80%9CEnd%2Dto%2Dend+arabic+speech+recognition%3A+A+review%2C%E2%80%9D+in+Proceedings+of+the+19th+Conference+of+Language+Engineering+%28ESOLEC%E2%80%9919%29%2C+Alexandria%2C+E)
3. A. Ali, P. Bell, J. Glass, Y. Messaoui, H. Mubarak, S. Renals, and Y. Zhang, “The mgb-2 challenge: Arabic multi-dialect broadcast media recognition,” in 2016 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2016, pp. 279–284. [our card](https://md.research.elamur.ai/free/en/papers/ali-2016-mgb-2-challenge-arabic-multi/)
4. S. Alharbi, A. Alowisheq, Z. Tüske, K. Darwish, A. Alrajeh, A. Alrowithi, A. B. Tamran, A. Ibrahim, R. Aloraini, R. Alnajim et al., “Sada: Saudi audio dataset for arabic,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 10 286–10 290. [find](https://scholar.google.com/scholar?q=S%2E+Alharbi%2C+A%2E+Alowisheq%2C+Z%2E+Tu%CC%88ske%2C+K%2E+Darwish%2C+A%2E+Alrajeh%2C+A%2E+Alrowithi%2C+A%2E+B%2E+Tamran%2C+A%2E+Ibrahim%2C+R%2E+Aloraini%2C+R%2E+Alnajim+et+al%2E%2C+%E2%80%9CSada%3A+Saudi+audio+dataset+for+arabic%2C%E2%80%9D+in+ICASSP+2024%2D2024+IEEE+In)
5. M. Al-Fetyani, M. Al-Barham, G. Abandah, A. Alsharkawi, and M. Dawas, “Masc: Massive arabic speech corpus,” in 2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023, pp. 1006–1013. [find](https://scholar.google.com/scholar?q=M%2E+Al%2DFetyani%2C+M%2E+Al%2DBarham%2C+G%2E+Abandah%2C+A%2E+Alsharkawi%2C+and+M%2E+Dawas%2C+%E2%80%9CMasc%3A+Massive+arabic+speech+corpus%2C%E2%80%9D+in+2022+IEEE+Spoken+Language+Technology+Workshop+%28SLT%29%2E+IEEE%2C+2023%2C+pp%2E+1006%E2%80%931013%2E)
6. H. Mubarak, A. Hussein, S. A. Chowdhury, and A. Ali, “Qasr: Qcri aljazeera speech resource–a large scale annotated arabic speech corpus,” arXiv preprint arXiv:2106.13000, 2021. [↗](https://arxiv.org/abs/2106.13000)
7. B. Talafha, K. Kadaoui, S. M. Magdy, M. Habiboullah, C. M. Chafei, A. O. El-Shangiti, H. Zayed, R. Alhamouri, R. Assi, A. Alraeesi et al., “Casablanca: Data and models for multidialectal arabic speech recognition,” arXiv preprint arXiv:2410.04527, 2024. [↗](https://arxiv.org/abs/2410.04527)
8. A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, “Data2vec: A general framework for self-supervised learning in speech, vision and language,” in International Conference on Machine Learning. PMLR, 2022, pp. 1298–1312. [find](https://scholar.google.com/scholar?q=A%2E+Baevski%2C+W%2E%2DN%2E+Hsu%2C+Q%2E+Xu%2C+A%2E+Babu%2C+J%2E+Gu%2C+and+M%2E+Auli%2C+%E2%80%9CData2vec%3A+A+general+framework+for+self%2Dsupervised+learning+in+speech%2C+vision+and+language%2C%E2%80%9D+in+International+Conference+on+Machine+Learning%2E)
9. A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems, vol. 33, pp. 12 449–12 460, 2020. [our card](https://md.research.elamur.ai/free/en/papers/baevski-2020-wav2vec-2-0-framework-self/)
10. W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021. [our card](https://md.research.elamur.ai/free/en/papers/hsu-2021-hubert-self-supervised-speech-representation/)
11. A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu et al., “Conformer: Convolution-augmented transformer for speech recognition,” arXiv preprint arXiv:2005.08100, 2020. [↗](https://arxiv.org/abs/2005.08100) [our card](https://md.research.elamur.ai/free/en/papers/gulati-2020-conformer-convolution-augmented-transformer-speech/)
12. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning. PMLR, 2023, pp. 28 492–28 518. [our card](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/)
13. T. Wolf, “Huggingface’s transformers: State-of-the-art natural language processing,” arXiv preprint arXiv:1910.03771, 2019. [↗](https://arxiv.org/abs/1910.03771) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
14. O. Kuchaiev, J. Li, H. Nguyen, O. Hrinchuk, R. Leary, B. Ginsburg, S. Kriman, S. Beliaev, V. Lavrukhin, J. Cook et al., “Nemo: a toolkit for building ai applications using neural modules,” arXiv preprint arXiv:1909.09577, 2019. [↗](https://arxiv.org/abs/1909.09577)
15. M. Ravanelli, T. Parcollet, A. Moumen, S. de Langen, C. Subakan, P. Plantinga, Y. Wang, P. Mousavi, L. Della Libera, A. Ploujnikov et al., “Open-source conversational ai with speechbrain 1.0,” Journal of Machine Learning Research, vol. 25, no. 333, pp. 1–11, 2024. [find](https://scholar.google.com/scholar?q=M%2E+Ravanelli%2C+T%2E+Parcollet%2C+A%2E+Moumen%2C+S%2E+de+Langen%2C+C%2E+Subakan%2C+P%2E+Plantinga%2C+Y%2E+Wang%2C+P%2E+Mousavi%2C+L%2E+Della+Libera%2C+A%2E+Ploujnikov+et+al%2E%2C+%E2%80%9COpen%2Dsource+conversational+ai+with+speechbrain+1%2E0%2C%E2%80%9D+Journal)
16. B. Talafha, A. Waheed, and M. Abdul-Mageed, “N-shot benchmarking of whisper on diverse arabic speech recognition,” arXiv preprint arXiv:2306.02902, 2023. [↗](https://arxiv.org/abs/2306.02902) [our card](https://md.research.elamur.ai/free/en/papers/talafha-2023-n-shot-benchmarking-whisper-diverse/)
17. A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli, “Unsupervised cross-lingual representation learning for speech recognition,” arXiv preprint arXiv:2006.13979, 2020. [↗](https://arxiv.org/abs/2006.13979)
18. R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” arXiv preprint arXiv:1912.06670, 2019. [↗](https://arxiv.org/abs/1912.06670) [our card](https://md.research.elamur.ai/free/en/papers/ardila-2019-common-voice-massively-multilingual-speech/)
19. A. Ali, S. Vogel, and S. Renals, “Speech recognition challenge in the wild: Arabic mgb-3,” in 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2017, pp. 316–322. [our card](https://md.research.elamur.ai/free/en/papers/ali-2017-speech-recognition-challenge-wild-arabic/)
20. A. Ali, S. Shon, Y. Samih, H. Mubarak, A. Abdelali, J. Glass, S. Renals, and K. Choukri, “The mgb-5 challenge: Recognition and dialect identification of dialectal arabic speech,” in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019, pp. 1026–1033. [find](https://scholar.google.com/scholar?q=A%2E+Ali%2C+S%2E+Shon%2C+Y%2E+Samih%2C+H%2E+Mubarak%2C+A%2E+Abdelali%2C+J%2E+Glass%2C+S%2E+Renals%2C+and+K%2E+Choukri%2C+%E2%80%9CThe+mgb%2D5+challenge%3A+Recognition+and+dialect+identification+of+dialectal+arabic+speech%2C%E2%80%9D+in+2019+IEEE+Automati)
21. A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna, “Fleurs: Few-shot learning evaluation of universal representations of speech,” in 2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023, pp. 798–805. [our card](https://md.research.elamur.ai/free/en/papers/conneau-2022-fleurs-few-shot-learning-evaluation/)
22. A. Abdelali, H. Mubarak, S. Chowdhury, M. Hasanain, B. Mousi, S. Boughorbel, S. Abdaljalil, Y. El Kheir, D. Izham, F. Dalvi et al., “Larabench: Benchmarking arabic ai with large language models,” in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 487–520. [our card](https://md.research.elamur.ai/free/en/papers/abdelali-2024-larabench-benchmarking-arabic-ai-large/)
23. V. Srivastav, S. Majumdar, N. Koluguri, A. Moumen, S. Gandhi et al., “Open automatic speech recognition leaderboard,” [https://huggingface.co/spaces/hf-audio/open](https://huggingface.co/spaces/hf-audio/open) asr leaderboard, 2023. [↗](https://huggingface.co/spaces/hf-audio/open)
24. A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of the 23rd international conference on Machine learning, 2006, pp. 369–376. [find](https://scholar.google.com/scholar?q=A%2E+Graves%2C+S%2E+Ferna%CC%81ndez%2C+F%2E+Gomez%2C+and+J%2E+Schmidhuber%2C+%E2%80%9CConnectionist+temporal+classification%3A+labelling+unsegmented+sequence+data+with+recurrent+neural+networks%2C%E2%80%9D+in+Proceedings+of+the+23rd+internat)
25. A. Graves, “Sequence transduction with recurrent neural networks,” arXiv preprint arXiv:1211.3711, 2012. [↗](https://arxiv.org/abs/1211.3711) [our card](https://md.research.elamur.ai/free/en/papers/graves-2012-sequence-transduction-recurrent-neural-networks/)
26. V. Noroozi, S. Majumdar, A. Kumar, J. Balam, and B. Ginsburg, “Stateful conformer with cache-based inference for streaming automatic speech recognition,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 12 041–12 045. [find](https://scholar.google.com/scholar?q=V%2E+Noroozi%2C+S%2E+Majumdar%2C+A%2E+Kumar%2C+J%2E+Balam%2C+and+B%2E+Ginsburg%2C+%E2%80%9CStateful+conformer+with+cache%2Dbased+inference+for+streaming+automatic+speech+recognition%2C%E2%80%9D+in+ICASSP+2024%2D2024+IEEE+International+Confere)
27. N. Halabi et al., “Arabic speech corpus,” Oxford Text Archive Core Collection, 2016. [find](https://scholar.google.com/scholar?q=N%2E+Halabi+et+al%2E%2C+%E2%80%9CArabic+speech+corpus%2C%E2%80%9D+Oxford+Text+Archive+Core+Collection%2C+2016%2E)
28. V. Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. Fazel-Zarandi et al., “Scaling speech technology to 1,000+ languages,” Journal of Machine Learning Research, vol. 25, no. 97, pp. 1–52, 2024. [our card](https://md.research.elamur.ai/free/en/papers/pratap-2023-scaling-speech-technology-1-000/)
29. Y.-A. Chung, Y. Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y. Wu, “W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,” in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021, pp. 244–250. [find](https://scholar.google.com/scholar?q=Y%2E%2DA%2E+Chung%2C+Y%2E+Zhang%2C+W%2E+Han%2C+C%2E%2DC%2E+Chiu%2C+J%2E+Qin%2C+R%2E+Pang%2C+and+Y%2E+Wu%2C+%E2%80%9CW2v%2Dbert%3A+Combining+contrastive+learning+and+masked+language+modeling+for+self%2Dsupervised+speech+pre%2Dtraining%2C%E2%80%9D+in+2021+IEEE+Aut)
30. L. Barrault, Y.-A. Chung, M. C. Meglioli, D. Dale, N. Dong, P.-A. Duquenne, H. Elsahar, H. Gong, K. Heffernan, J. Hoffman et al., “Seamlessm4t-massively multilingual & multimodal machine translation,” arXiv preprint arXiv:2308.11596, 2023. [↗](https://arxiv.org/abs/2308.11596) [our card](https://md.research.elamur.ai/free/en/papers/communication-2023-seamlessm4t-massively-multilingual-multimodal-machine/)
31. S. A. Chowdhury, A. Hussein, A. Abdelali, and A. Ali, “Towards one model to rule all: Multilingual strategy for dialectal code-switching arabic asr,” arXiv preprint arXiv:2105.14779, 2021. [↗](https://arxiv.org/abs/2105.14779) [our card](https://md.research.elamur.ai/free/en/papers/chowdhury-2021-towards-one-model-rule-all/)