--- title: "ArTST: Arabic Text and Speech Transformer (2023)" description: "The authors built ArTST by keeping the SpeechT5 design: a shared T5-style encoder-decoder with… 5 references, cited by 19 papers here." authors: ["Hawau Olamide Toyin", "Amirbek Djanibekov", "Ajinkya Kulkarni", "Hanan Aldarmaki"] year: 2023 venue: "Proceedings of ArabicNLP 2023" source: "https://aclanthology.org/2023.arabicnlp-1.5/" doi: "10.18653/v1/2023.arabicnlp-1.5" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/toyin-2023-artst-arabic-text-speech-transformer/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/toyin-2023-artst-arabic-text-speech-transformer/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/) / ArTST: Arabic Text and Speech Transformer # ArTST: Arabic Text and Speech Transformer **Authors:** [Hawau Olamide Toyin](https://md.research.elamur.ai/free/en/authors/hawau-olamide-toyin/), [Amirbek Djanibekov](https://md.research.elamur.ai/free/en/authors/amirbek-djanibekov/), [Ajinkya Kulkarni](https://md.research.elamur.ai/free/en/authors/ajinkya-kulkarni/) and [Hanan Aldarmaki](https://md.research.elamur.ai/free/en/authors/hanan-aldarmaki/) · **Year:** 2023 · **Venue:** [Proceedings of ArabicNLP 2023](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2023](https://md.research.elamur.ai/free/en/venues/arabicnlp/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2023.arabicnlp-1.5/) · [PDF](https://aclanthology.org/2023.arabicnlp-1.5.pdf) · [DOI 10.18653/v1/2023.arabicnlp-1.5](https://doi.org/10.18653/v1/2023.arabicnlp-1.5) · [OpenAlex](https://openalex.org/W4389518360) **Citations:** 8 citations (OpenAlex, 2026-10-14) · [19](https://md.research.elamur.ai/free/en/papers/toyin-2023-artst-arabic-text-speech-transformer/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Arabic lacked an open pre-trained model that handles both speech and text as input and output. English SpeechT5, when fine-tuned on Arabic, stayed strongly biased toward English and produced weak results. The authors built ArTST by keeping the SpeechT5 design: a shared T5-style encoder-decoder with separate small pre- and post-networks for each modality. They trained it from scratch on Modern Standard Arabic, using roughly 1K hours of filtered MGB2 broadcast speech with its text. Pre-training combined HuBERT-style masked prediction of discrete speech units with a denoising objective that reconstructs spectrograms. The model was then fine-tuned for ASR, TTS and spoken dialect identification. It matched or beat earlier results on all three tasks, and in ASR it outperformed the much larger multilingual Whisper and MMS. The TTS model pronounced undiacritized text correctly and generalized to unseen MSA sentences. This supports building focused monolingual foundation models for well-resourced languages, and the checkpoints are released for research. ## Key points - **Training data:** MGB2 (Aljazeera recordings, ~1.2K hours, 1.4K speakers). - Overlapping-speech segments and clips longer than 40 s were removed, leaving about 1K hours. - A 200-hour subset was also drawn. - **Speech pre-training targets:** k-means clustering of HuBERT hidden states into 500 classes. Masked spans are 10 frames long. The denoising decoder reconstructs 80-dimensional log mel filterbanks. - **TTS (fine-tuned on ClArTTS and ASC, HiFi-GAN vocoder; MOS from 15 native listeners):** - English SpeechT5: 1.88 on ClArTTS and 1.57 on ASC. - ArTST: 4.11 and 2.93. - ArTST* (first fine-tuned on MGB2 ASR data): 4.31 and 3.44. - Ground truth: 4.64 and 4.31. - **Dialect ID (ADI17):** ArTST reaches 95.08% on dev and 94.18% on test. This beats the best single MGB-5 system (DKU, 94.70/93.80) but trails a 4-system fusion that also used data augmentation (97.40/94.90). - **ASR:** ArTST surpasses Whisper and MMS on MSA. For the SpeechT5 baseline, Buckwalter transliteration worked better than Arabic script. ## Interesting ideas - Dialect classification is treated as generation, with the decoder emitting the class label at its first step, so the same seq2seq machinery serves without a separate classifier head. - Training goes in two stages: first on plentiful but noisy, mismatched data (ASR recordings), then on a small clean set (TTS corpora). This clearly lifted quality when clean data was scarce. - One shared encoder-decoder core sits between light modality-specific pre/post-nets, so different input and output types reuse the same learned representations. - The paper fine-tunes a foreign-language pre-trained model as a baseline, which exposes how much pre-training bias hurts transfer. - Training samples are filtered by overlap tags and a length cap, trading a little data for much less padding and cleaner supervision. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (frames): a speech-act or irony label could be emitted as the first output token of a frame-producing decoder, keeping one model for both labeling and structure. - **I2** (learn): train gparse first on large silver data agreed by witness parsers, then refine on gold treebanks, and measure the gain against a gold-only run. - **I3** (back): parsing (text → graph) and generation (graph → text) could share a backbone with thin task-specific adapters, helping the compressor stay consistent in both directions. - **I4** (learn): add a mismatched-pretraining baseline as a negative control, so improvements credited to in-domain pre-training are actually demonstrated. - **I5** (tree): cap sentence length and drop flagged noisy sentences when batching treebank data for GPU parser training, to cut wasted computation. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Mohamed bin Zayed University of Artificial Intelligence](https://md.research.elamur.ai/free/en/universities/mohamed-bin-zayed-university-of-artificial-intelligence/) — University, Abu Dhabi, United Arab Emirates · 124 works · 5,676 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 452 works 🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works ## Cite Cite as: Hawau Olamide Toyin, Amirbek Djanibekov, Ajinkya Kulkarni and Hanan Aldarmaki. 2023. ArTST: Arabic Text and Speech Transformer. Proceedings of ArabicNLP 2023. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{toyin2023artst, title = {{ArTST: Arabic Text and Speech Transformer}}, author = {Hawau Olamide Toyin, Amirbek Djanibekov, Ajinkya Kulkarni and Hanan Aldarmaki}, booktitle = {Proceedings of ArabicNLP 2023}, year = {2023}, doi = {10.18653/v1/2023.arabicnlp-1.5}, url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.5} } ``` - [← previous SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency](https://md.research.elamur.ai/free/en/papers/zhang-2023-sac3-reliable-hallucination-detection-black/) - [next → ClArTTS: An Open-Source Classical Arabic Text-to-Speech Corpus](https://md.research.elamur.ai/free/en/papers/kulkarni-2023-clartts-open-source-classical-arabic/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/) ## References 5 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, and Furu Wei. 2022. [SpeechT5: Unified-modal encoder-decoder pre-training for spoken language processing](https://md.research.elamur.ai/free/en/papers/ao-2022-speecht5-unified-modal-encoder-decoder/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5723–5738, Dublin, Ireland. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2110.07205) 2. Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. 2022. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300–22312. [find](https://scholar.google.com/scholar?q=Ibrahim+M+Alabdulmohsin%2C+Behnam+Neyshabur%2C+and+Xiaohua+Zhai%2E+2022%2E+Revisiting+neural+scaling+laws+in+language+and+vision%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A22300%E2%80%9322312%2E) 3. Hanan Aldarmaki and Ahmad Ghannam. 2023. [Diacritic Recognition Performance in Arabic ASR](https://md.research.elamur.ai/free/en/papers/aldarmaki-2023-diacritic-recognition-performance-arabic-asr/). In Proc. INTERSPEECH 2023, pages 361–365. 4. Ahmed Ali, Peter Bell, James Glass, Yacine Messaoui, Hamdy Mubarak, Steve Renals, and Yifan Zhang. 2016. [The mgb-2 challenge: Arabic multi-dialect broadcast media recognition](https://md.research.elamur.ai/free/en/papers/ali-2016-mgb-2-challenge-arabic-multi/). In 2016 IEEE Spoken Language Technology Workshop (SLT), pages 279– 284. IEEE. 5. Ahmed Ali, Suwon Shon, Younes Samih, Hamdy Mubarak, Ahmed Abdelali, James Glass, Steve Renals, and Khalid Choukri. 2019. The mgb-5 challenge: Recognition and dialect identification of dialectal arabic speech. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1026–1033. IEEE. [find](https://scholar.google.com/scholar?q=Ahmed+Ali%2C+Suwon+Shon%2C+Younes+Samih%2C+Hamdy+Mubarak%2C+Ahmed+Abdelali%2C+James+Glass%2C+Steve+Renals%2C+and+Khalid+Choukri%2E+2019%2E+The+mgb%2D5+challenge%3A+Recognition+and+dialect+identification+of+dialectal+arabic) *Updated 1 November 2026*