--- title: "Fine-tuning Whisper Across 81 Languages — Singh 2026" description: "The study presents fine-tuning as an inexpensive response to the curse of multilinguality and identifies tokenization as the… 25.6% Spearman. 4 references." authors: ["Shivam Singh", "Alex Warstadt"] year: 2026 venue: "Proceedings of the Society for Computation in Linguistics 2026" source: "https://aclanthology.org/2026.scil-main.37/" doi: "10.18653/v1/2026.scil-main.37" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/singh-2026-fine-tuning-whisper-across-81/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/singh-2026-fine-tuning-whisper-across-81/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/) / Fine-tuning Whisper Across 81 Languages # Fine-tuning Whisper Across 81 Languages **Authors:** Shivam Singh and [Alex Warstadt](https://md.research.elamur.ai/free/en/authors/alex-warstadt/) · **Year:** 2026 · **Venue:** [Proceedings of the Society for Computation in Linguistics 2026](https://md.research.elamur.ai/free/en/venues/scil/) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.scil-main.37/) · [PDF](https://aclanthology.org/2026.scil-main.37.pdf) · [DOI 10.18653/v1/2026.scil-main.37](https://doi.org/10.18653/v1/2026.scil-main.37) · [OpenAlex](https://openalex.org/W7166845071) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Whisper performs well on high-resource languages, but its accuracy drops sharply elsewhere. Zero-shot WER exceeds 50% for Bengali, Tamil and Georgian and reaches about 100% for Khmer, Burmese and Lao. Earlier work showed that monolingual fine-tuning helps, but it covered only a few languages. The authors scale this up by training a separate copy of Whisper large-v3 for each of the 81 languages in FLEURS. They train on the FLEURS training split, keep the checkpoint with the best validation WER, and use two hyperparameter settings found by Bayesian search on Hindi and German. Every language improves, with average relative WER reduction close to 30%. The size of the gain depends heavily on the writing system. Latin- and Cyrillic-script languages benefit most, while Thai, Burmese and Khmer barely change. The authors link this to tokenizer fit: scripts that Whisper's English-centric BPE splits into many short pieces gain less, regardless of pretraining hours. The study presents fine-tuning as an inexpensive response to the curse of multilinguality and identifies tokenization as the main remaining obstacle. The authors also plan to release what they describe as the largest collection of open language-specialized ASR models. ## Key points - Under Config A, mean WER across 81 FLEURS languages falls from 34.8% to 24.6%, and 50 languages end below 25%. Config B gives nearly the same results (mean 25.6%, Spearman ρ ≈ 0.996, mean gap 1.1 pp). - Setup: each language gets its own fine-tuning run on one A100, using AdamW and an encoder learning rate set to 0.3× the decoder rate. Training uses early stopping (patience 3) and a cap of 12 epochs. - Script family drives the outcome: - Latin and Cyrillic languages see 30–45% reductions, with Italian at 2.8%, Spanish at 3.6% and Russian at 5.5%. - Brahmic-script languages improve by about 25% relative but remain at 20–54% WER. - Thai moves only from 82.4% to 70.5%, Burmese improves by about 3%, and Khmer does not improve. - Whisper's BPE compression ratio (characters per token) correlates with relative WER reduction at Spearman ρ ≈ −0.78. The correlation holds at partial ρ ≈ −0.79 after controlling for per-language pretraining duration. - Ongoing work replaces Whisper's tokenizer with language-specific BPE. Early results show large gains for poorly tokenized scripts that remain above 50% WER after standard fine-tuning. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of California San Diego](https://md.research.elamur.ai/free/en/universities/university-of-california-san-diego/) — University, San Diego, United States · 21 works · 45,121 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 7,187 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 7,026 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 7,384 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 7,329 works ## Cite Cite as: Shivam Singh and Alex Warstadt. 2026. Fine-tuning Whisper Across 81 Languages. Proceedings of the Society for Computation in Linguistics 2026. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{singh2026fine, title = {{Fine-tuning Whisper Across 81 Languages}}, author = {Shivam Singh and Alex Warstadt}, booktitle = {Proceedings of the Society for Computation in Linguistics 2026}, year = {2026}, doi = {10.18653/v1/2026.scil-main.37}, url = {https://doi.org/10.18653/v1/2026.scil-main.37} } ``` - [← previous Finding Sense in Nonsense with Generated Contexts: Perspectives from Humans and Language Models](https://md.research.elamur.ai/free/en/papers/olsen-2026-finding-sense-nonsense-generated-contexts/) - [next → Fine-tuning Whisper with Spontaneous Persian Speech (SPS)](https://md.research.elamur.ai/free/en/papers/namdarzadeh-2026-fine-tuning-whisper-spontaneous-persian/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/) ## References 4 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro ceedings of the 58th Annual Meeting of the Associa tion… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettle+moyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dling) 2. Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riber, Ankur Bapna, and 1 others. 2023. FLEURS: Few-shot learn ing evaluation of universal representations of speech. In 2022 IEEE Spoken Language Technology Work shop (SLT), pages 798–805. IEEE. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Min+Ma%2C+Simran+Khanuja%2C+Yu+Zhang%2C+Vera+Axelrod%2C+Siddharth+Dalmia%2C+Jason+Riber%2C+Ankur+Bapna%2C+and+1+others%2E+2023%2E+FLEURS%3A+Few%2Dshot+learn+ing+evaluation+of+universal+representations+of+sp) 3. Yunpeng Liu, Xukui Yang, and Dan Qu. 2024. Ex ploration of Whisper fine-tuning strategies for low-resource ASR. EURASIP Journal on Audio, Speech, and Music Processing, 2024(1):29. [find](https://scholar.google.com/scholar?q=Yunpeng+Liu%2C+Xukui+Yang%2C+and+Dan+Qu%2E+2024%2E+Ex+ploration+of+Whisper+fine%2Dtuning+strategies+for+low%2Dresource+ASR%2E+EURASIP+Journal+on+Audio%2C+Speech%2C+and+Music+Processing%2C+2024%281%29%3A29%2E) 4. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492–28518. PMLR. [our card](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/) *Updated 23 October 2026*