---
title: "N-Shot Benchmarking of Whisper on Diverse — Talafha 2023"
description: "The authors ask how well OpenAI's Whisper handles Arabic beyond the standard benchmarks."
authors: ["Bashar Talafha", "Abdul Waheed", "Muhammad Abdul-Mageed"]
year: 2023
venue: "arXiv:2306.02902"
source: "https://arxiv.org/abs/2306.02902"
doi: "10.48550/arxiv.2306.02902"
arxiv: "2306.02902"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/talafha-2023-n-shot-benchmarking-whisper-diverse/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/talafha-2023-n-shot-benchmarking-whisper-diverse/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / N-Shot Benchmarking of Whisper on Diverse Arabic Speech Recognition

# N-Shot Benchmarking of Whisper on Diverse Arabic Speech Recognition

**Authors:** Bashar Talafha, [Abdul Waheed](https://md.research.elamur.ai/free/en/authors/abdul-waheed/), [Muhammad Abdul-Mageed](https://md.research.elamur.ai/free/en/authors/muhammad-abdul-mageed/) · **Year:** 2023 · **Venue:** arXiv:2306.02902 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2306.02902) · [PDF](https://arxiv.org/pdf/2306.02902) · [DOI 10.48550/arxiv.2306.02902](https://doi.org/10.48550/arxiv.2306.02902) · [arXiv 2306.02902](https://arxiv.org/abs/2306.02902) · [OpenAlex](https://openalex.org/W4385822324)

**Citations:** 11 citations (openalex-snapshot · 2026-10-14) · [19](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The authors ask how well OpenAI's Whisper handles Arabic beyond the standard benchmarks. Arabic is a family of varieties: Modern Standard Arabic (MSA) sits alongside many regional dialects. They test Whisper large-v2 and small, with XLS-R as the baseline, on nearly all public Arabic ASR corpora. These cover Common Voice (three versions), MGB-2, MGB-3, MGB-5 and the Egyptian-accented MSA subset of FLEURS. Whisper is run with no tuning (zero-shot), with small tuning budgets (few-shot) and with full finetuning. The team also built a new evaluation-only corpus, AraYouTube, from soap-opera audio in dialects not covered by earlier work. Native annotators transcribed it.

Zero-shot Whisper already beats fully finetuned XLS-R on every existing dataset. However, its error rates rise sharply on the five new dialects. A model finetuned only on MSA also copes worse with accented and dialectal speech than zero-shot Whisper does. The work shows that strong standard-benchmark scores do not guarantee robustness across language varieties.

## Key points
- Corpora used:
  - Common Voice v6.1, v9.0 and v11.0 (50, 88 and 89 hours).
  - MGB-2: about 1,200 hours of Aljazeera broadcasts, roughly 70% MSA.
  - MGB-3: 4.8 hours of Egyptian YouTube.
  - MGB-5: 10.2 hours of Moroccan speech.
  - FLEURS: 4.39 hours of MSA read by Egyptian speakers.
- AraYouTube is a new test-only resource covering Algerian, Jordanian, Palestinian, UAE and Yemeni dialects.
- Zero-shot Whisper large-v2 word error rates (WER) on the test sets:
  - 15.93 on CV6.1;
  - 34.74 on MGB-2;
  - 82.97 on Moroccan MGB-5;
  - 11.56 on FLEURS.
- Few-shot results improve steadily as the tuning budget doubles from 1h to 16h. On CV6.1, WER falls to 10.95 at 16h. Full finetuning of Whisper large gives 10.81 on CV6.1 and 53.82 on MGB-5.
- XLS-R failed to converge on MGB-2, MGB-5 and FLEURS. With an added language model (XLS-R-LM), it slightly beats Whisper small on CV6.1 (21.62 vs 22.21 WER), even though Whisper saw about 7x more Arabic data.
- Text normalization followed earlier work:
  - punctuation (except % and @), diacritics, Hamzas and Maddas removed;
  - Eastern Arabic numerals mapped to Western ones;
  - Latin characters removed.
- During finetuning, the feature-extraction layer stayed frozen.

## Interesting ideas
- Before computing error rates, apply the same normalization to both references and outputs, because the original annotations were inconsistent (e.g., diacritics present in some utterances and absent in others). Otherwise, scores partly measure annotation noise. — For Mova (learn): When scoring parsers against several treebanks, first map all of them into one UD dialect, so that convention differences are not counted as parser errors.
- Build a small, carefully annotated test set from varieties never seen in training, and keep it strictly for evaluation. This exposes how well a model generalizes rather than how well it memorized. — For Mova (learn): Keep a held-out set of unusual English genres (fables, dialogue, old-style prose) that no witness parser or teacher ever trains on, as a true out-of-distribution check.
- Measure performance at doubling data budgets (1, 2, 4, 8, 16 hours) to get a learning curve instead of a single number. The curve shows where extra data stops paying off. — For Mova (learn): Train gparse on doubling amounts of silver data to see whether additional agreed parses still help or have plateaued.
- Specializing on the standard variety can leave a model less robust on accented input than the untouched base model. The paper therefore compares finetuned models against the zero-shot model as a baseline. — For Mova (tree): After retraining the parser on one treebank, compare it with the previous model on other domains, so that silent regressions are caught.
- A smaller model with an explicit language model added at decoding can match a larger-data model. Extra structured knowledge can thus substitute for raw training volume. — For Mova (tokens): Use lexicon or morphology constraints to rescore tagger candidates, instead of relying only on more training text.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

## Cited by

Cited by

19 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 16 not yet.

- [ADI-20: Arabic Dialect Identification dataset and models](https://md.research.elamur.ai/free/en/papers/elleuch-2025-adi-20-arabic-dialect-identification/)
- [Open Automatic Speech Recognition Models for Classical and Modern Standard Arabic](https://md.research.elamur.ai/free/en/papers/grigoryan-2025-open-automatic-speech-recognition-models/)
- [Open Universal Arabic ASR Leaderboard](https://md.research.elamur.ai/free/en/papers/wang-2024-open-universal-arabic-asr-leaderboard/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{talafha2023n,
  title = {{N-Shot Benchmarking of Whisper on Diverse Arabic Speech Recognition}},
  author = {Talafha, Bashar and Waheed, Abdul and Abdul-Mageed, Muhammad},
  year = {2023},
  doi = {10.48550/arxiv.2306.02902},
  eprint = {2306.02902},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.02902}
}
```

- [← previous MyVoice: Arabic Speech Resource Collaboration Platform](https://md.research.elamur.ai/free/en/papers/elshahawy-2023-myvoice-arabic-speech-resource-collaboration/)
- [next → Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://md.research.elamur.ai/free/en/papers/wang-2023-neural-codec-language-models-zero/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

26 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. S. Liu, A. Mallol-Ragolta, E. Parada-Cabaleiro, K. Qian, X. Jing, A. Kathan, B. Hu, and B. W. Schuller, “Audio self-supervised learning: A survey,” Patterns, vol. 3, no. 12, p. 100616, 2022. [find](https://scholar.google.com/scholar?q=S%2E+Liu%2C+A%2E+Mallol%2DRagolta%2C+E%2E+Parada%2DCabaleiro%2C+K%2E+Qian%2C+X%2E+Jing%2C+A%2E+Kathan%2C+B%2E+Hu%2C+and+B%2E+W%2E+Schuller%2C+%E2%80%9CAudio+self%2Dsupervised+learning%3A+A+survey%2C%E2%80%9D+Patterns%2C+vol%2E+3%2C+no%2E+12%2C+p%2E+100616%2C+2022%2E)
2. M. C. Schiappa, Y. S. Rawat, and M. Shah, “Self-supervised learning for videos: A survey,” ACM Computing Surveys, 2022. [find](https://scholar.google.com/scholar?q=M%2E+C%2E+Schiappa%2C+Y%2E+S%2E+Rawat%2C+and+M%2E+Shah%2C+%E2%80%9CSelf%2Dsupervised+learning+for+videos%3A+A+survey%2C%E2%80%9D+ACM+Computing+Surveys%2C+2022%2E)
3. X. Qiu, T. Sun, Y. Xu, Y. Shao, N. Dai, and X. Huang, “Pre-trained models for natural language processing: A survey,” Science China Technological Sciences, vol. 63, no. 10, pp. 1872–1897, 2020. [find](https://scholar.google.com/scholar?q=X%2E+Qiu%2C+T%2E+Sun%2C+Y%2E+Xu%2C+Y%2E+Shao%2C+N%2E+Dai%2C+and+X%2E+Huang%2C+%E2%80%9CPre%2Dtrained+models+for+natural+language+processing%3A+A+survey%2C%E2%80%9D+Science+China+Technological+Sciences%2C+vol%2E+63%2C+no%2E+10%2C+pp%2E+1872%E2%80%931897%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/)
4. E. Kim, J.-J. Jeon, H. Seo, and H. Kim, “Automatic Pronunciation Assessment using Self-Supervised Speech Representation Learning,” in Proc. Interspeech 2022, 2022, pp. 1411–1415. [find](https://scholar.google.com/scholar?q=E%2E+Kim%2C+J%2E%2DJ%2E+Jeon%2C+H%2E+Seo%2C+and+H%2E+Kim%2C+%E2%80%9CAutomatic+Pronunciation+Assessment+using+Self%2DSupervised+Speech+Representation+Learning%2C%E2%80%9D+in+Proc%2E+Interspeech+2022%2C+2022%2C+pp%2E+1411%E2%80%931415%2E)
5. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” arXiv preprint arXiv:2212.04356, 2022. [↗](https://arxiv.org/abs/2212.04356) [our card](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/)
6. A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino et al., “Xls-r: Self-supervised cross-lingual speech representation learning at scale,” arXiv preprint arXiv:2111.09296, 2021. [↗](https://arxiv.org/abs/2111.09296) [our card](https://md.research.elamur.ai/free/en/papers/babu-2021-xls-r-self-supervised-cross/)
7. D. Wang, X. Wang, and S. Lv, “An overview of end-to-end automatic speech recognition,” Symmetry, vol. 11, no. 8, p. 1018, 2019. [find](https://scholar.google.com/scholar?q=D%2E+Wang%2C+X%2E+Wang%2C+and+S%2E+Lv%2C+%E2%80%9CAn+overview+of+end%2Dto%2Dend+automatic+speech+recognition%2C%E2%80%9D+Symmetry%2C+vol%2E+11%2C+no%2E+8%2C+p%2E+1018%2C+2019%2E)
8. V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210. [find](https://scholar.google.com/scholar?q=V%2E+Panayotov%2C+G%2E+Chen%2C+D%2E+Povey%2C+and+S%2E+Khudanpur%2C+%E2%80%9CLibrispeech%3A+an+asr+corpus+based+on+public+domain+audio+books%2C%E2%80%9D+in+2015+IEEE+international+conference+on+acoustics%2C+speech+and+signal+processing+%28IC)
9. A. Rousseau, P. Deléglise, and Y. Esteve, “Ted-lium: an automatic speech recognition dedicated corpus.” in LREC, 2012, pp. 125– 129. [find](https://scholar.google.com/scholar?q=A%2E+Rousseau%2C+P%2E+Dele%CC%81glise%2C+and+Y%2E+Esteve%2C+%E2%80%9CTed%2Dlium%3A+an+automatic+speech+recognition+dedicated+corpus%2E%E2%80%9D+in+LREC%2C+2012%2C+pp%2E+125%E2%80%93+129%2E)
10. R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” arXiv preprint arXiv:1912.06670, 2019. [↗](https://arxiv.org/abs/1912.06670) [our card](https://md.research.elamur.ai/free/en/papers/ardila-2019-common-voice-massively-multilingual-speech/)
11. R. Olivier and B. Raj, “There is more than one kind of robustness: Fooling whisper with adversarial examples,” arXiv preprint arXiv:2210.17316, 2022. [↗](https://arxiv.org/abs/2210.17316)
12. A. Ahmed, Y. Hifny, K. Shaalan, and S. Toral, “End-to-end lexicon free arabic speech recognition using recurrent neural networks,” in Computational Linguistics, Speech And Image Processing For Arabic Language. World Scientific, 2019, pp. 231– 248. [find](https://scholar.google.com/scholar?q=A%2E+Ahmed%2C+Y%2E+Hifny%2C+K%2E+Shaalan%2C+and+S%2E+Toral%2C+%E2%80%9CEnd%2Dto%2Dend+lexicon+free+arabic+speech+recognition+using+recurrent+neural+networks%2C%E2%80%9D+in+Computational+Linguistics%2C+Speech+And+Image+Processing+For+Arabic+)
13. Y. Belinkov, A. Ali, and J. Glass, “Analyzing phonetic and graphemic representations in end-to-end automatic speech recognition,” arXiv preprint arXiv:1907.04224, 2019. [↗](https://arxiv.org/abs/1907.04224)
14. A. Hussein, S. Watanabe, and A. Ali, “Arabic speech recognition by end-to-end, modular systems and human,” Computer Speech & Language, vol. 71, p. 101272, 2022. [find](https://scholar.google.com/scholar?q=A%2E+Hussein%2C+S%2E+Watanabe%2C+and+A%2E+Ali%2C+%E2%80%9CArabic+speech+recognition+by+end%2Dto%2Dend%2C+modular+systems+and+human%2C%E2%80%9D+Computer+Speech+%26+Language%2C+vol%2E+71%2C+p%2E+101272%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/hussein-2021-arabic-speech-recognition-end-end/)
15. A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems, vol. 33, pp. 12 449–12 460, 2020. [find](https://scholar.google.com/scholar?q=A%2E+Baevski%2C+Y%2E+Zhou%2C+A%2E+Mohamed%2C+and+M%2E+Auli%2C+%E2%80%9Cwav2vec+2%2E0%3A+A+framework+for+self%2Dsupervised+learning+of+speech+representations%2C%E2%80%9D+Advances+in+neural+information+processing+systems%2C+vol%2E+33%2C+pp%2E+12+449%E2%80%93) [our card](https://md.research.elamur.ai/free/en/papers/baevski-2020-wav2vec-2-0-framework-self/)
16. W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021. [find](https://scholar.google.com/scholar?q=W%2E%2DN%2E+Hsu%2C+B%2E+Bolte%2C+Y%2E%2DH%2E+H%2E+Tsai%2C+K%2E+Lakhotia%2C+R%2E+Salakhutdinov%2C+and+A%2E+Mohamed%2C+%E2%80%9CHubert%3A+Self%2Dsupervised+speech+representation+learning+by+masked+prediction+of+hidden+units%2C%E2%80%9D+IEEE%2FACM+Transactions+) [our card](https://md.research.elamur.ai/free/en/papers/hsu-2021-hubert-self-supervised-speech-representation/)
17. Y.-A. Chung, Y. Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y. Wu, “W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,” in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021, pp. 244–250. [find](https://scholar.google.com/scholar?q=Y%2E%2DA%2E+Chung%2C+Y%2E+Zhang%2C+W%2E+Han%2C+C%2E%2DC%2E+Chiu%2C+J%2E+Qin%2C+R%2E+Pang%2C+and+Y%2E+Wu%2C+%E2%80%9CW2v%2Dbert%3A+Combining+contrastive+learning+and+masked+language+modeling+for+self%2Dsupervised+speech+pre%2Dtraining%2C%E2%80%9D+in+2021+IEEE+Aut)
18. A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli, “XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale,” in Proc. Interspeech 2022, 2022, pp. 2278–2282. [find](https://scholar.google.com/scholar?q=A%2E+Babu%2C+C%2E+Wang%2C+A%2E+Tjandra%2C+K%2E+Lakhotia%2C+Q%2E+Xu%2C+N%2E+Goyal%2C+K%2E+Singh%2C+P%2E+von+Platen%2C+Y%2E+Saraf%2C+J%2E+Pino%2C+A%2E+Baevski%2C+A%2E+Conneau%2C+and+M%2E+Auli%2C+%E2%80%9CXLS%2DR%3A+Self%2Dsupervised+Cross%2Dlingual+Speech+Representation) [our card](https://md.research.elamur.ai/free/en/papers/babu-2021-xls-r-self-supervised-cross/)
19. J. Valk and T. Alumäe, “Voxlingua107: a dataset for spoken language recognition,” in 2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 652–658. [find](https://scholar.google.com/scholar?q=J%2E+Valk+and+T%2E+Aluma%CC%88e%2C+%E2%80%9CVoxlingua107%3A+a+dataset+for+spoken+language+recognition%2C%E2%80%9D+in+2021+IEEE+Spoken+Language+Technology+Workshop+%28SLT%29%2E+IEEE%2C+2021%2C+pp%2E+652%E2%80%93658%2E)
20. M. Bakheet, “Improving speech recognition for arabic language using low amounts of labeled data,” 2021. [find](https://scholar.google.com/scholar?q=M%2E+Bakheet%2C+%E2%80%9CImproving+speech+recognition+for+arabic+language+using+low+amounts+of+labeled+data%2C%E2%80%9D+2021%2E)
21. A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna, “Fleurs: Few-shot learning evaluation of universal representations of speech,” arXiv preprint arXiv:2205.12446, 2022. [↗](https://arxiv.org/abs/2205.12446) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2022-fleurs-few-shot-learning-evaluation/)
22. A. Ali, P. Bell, J. Glass, Y. Messaoui, H. Mubarak, S. Renals, and Y. Zhang, “The mgb-2 challenge: Arabic multi-dialect broadcast media recognition,” in 2016 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2016, pp. 279–284. [find](https://scholar.google.com/scholar?q=A%2E+Ali%2C+P%2E+Bell%2C+J%2E+Glass%2C+Y%2E+Messaoui%2C+H%2E+Mubarak%2C+S%2E+Renals%2C+and+Y%2E+Zhang%2C+%E2%80%9CThe+mgb%2D2+challenge%3A+Arabic+multi%2Ddialect+broadcast+media+recognition%2C%E2%80%9D+in+2016+IEEE+Spoken+Language+Technology+Workshop+%28) [our card](https://md.research.elamur.ai/free/en/papers/ali-2016-mgb-2-challenge-arabic-multi/)
23. A. Ali, S. Vogel, and S. Renals, “Speech recognition challenge in the wild: Arabic mgb-3,” in 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2017, pp. 316–322. [find](https://scholar.google.com/scholar?q=A%2E+Ali%2C+S%2E+Vogel%2C+and+S%2E+Renals%2C+%E2%80%9CSpeech+recognition+challenge+in+the+wild%3A+Arabic+mgb%2D3%2C%E2%80%9D+in+2017+IEEE+Automatic+Speech+Recognition+and+Understanding+Workshop+%28ASRU%29%2E+IEEE%2C+2017%2C+pp%2E+316%E2%80%93322%2E) [our card](https://md.research.elamur.ai/free/en/papers/ali-2017-speech-recognition-challenge-wild-arabic/)
24. A. Ali, S. Shon, Y. Samih, H. Mubarak, A. Abdelali, J. Glass, S. Renals, and K. Choukri, “The mgb-5 challenge: Recognition and dialect identification of dialectal arabic speech,” in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019, pp. 1026–1033. [find](https://scholar.google.com/scholar?q=A%2E+Ali%2C+S%2E+Shon%2C+Y%2E+Samih%2C+H%2E+Mubarak%2C+A%2E+Abdelali%2C+J%2E+Glass%2C+S%2E+Renals%2C+and+K%2E+Choukri%2C+%E2%80%9CThe+mgb%2D5+challenge%3A+Recognition+and+dialect+identification+of+dialectal+arabic+speech%2C%E2%80%9D+in+2019+IEEE+Automati)
25. S. A. Chowdhury, A. Hussein, A. Abdelali, and A. Ali, “Towards One Model to Rule All: Multilingual Strategy for Dialectal Code-Switching Arabic ASR,” in Proc. Interspeech 2021, 2021, pp. 2466–2470. [find](https://scholar.google.com/scholar?q=S%2E+A%2E+Chowdhury%2C+A%2E+Hussein%2C+A%2E+Abdelali%2C+and+A%2E+Ali%2C+%E2%80%9CTowards+One+Model+to+Rule+All%3A+Multilingual+Strategy+for+Dialectal+Code%2DSwitching+Arabic+ASR%2C%E2%80%9D+in+Proc%2E+Interspeech+2021%2C+2021%2C+pp%2E+2466%E2%80%932470%2E) [our card](https://md.research.elamur.ai/free/en/papers/chowdhury-2021-towards-one-model-rule-all/)
26. I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017. [↗](https://arxiv.org/abs/1711.05101)