--- title: "How Well Do LLMs Understand Tunisian Arabic? — Mahdi 2025" description: "Industrial LLMs are rarely tested on Tunisian Arabic, a dialect with no standard spelling that is often written in Latin letters (\"Tunizi\") and mixes in…" authors: ["Mohamed Mahdi"] year: 2025 venue: "arXiv:2511.16683" source: "https://arxiv.org/abs/2511.16683" doi: "10.48550/arxiv.2511.16683" arxiv: "2511.16683" licence: "CC BY-SA 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by-sa/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/mahdi-2025-well-llms-understand-tunisian-arabic/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mahdi-2025-well-llms-understand-tunisian-arabic/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) / How Well Do LLMs Understand Tunisian Arabic? # How Well Do LLMs Understand Tunisian Arabic? **Authors:** Mohamed Mahdi · **Year:** 2025 · **Venue:** arXiv:2511.16683 · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2511.16683) · [PDF](https://arxiv.org/pdf/2511.16683) · [DOI 10.48550/arxiv.2511.16683](https://doi.org/10.48550/arxiv.2511.16683) · [arXiv 2511.16683](https://arxiv.org/abs/2511.16683) · [OpenAlex](https://openalex.org/W4416591544) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/mahdi-2025-well-llms-understand-tunisian-arabic/cited-by/) in our library, of the 140884 papers whose bibliographies we read ## Summary Industrial LLMs are rarely tested on Tunisian Arabic, a dialect with no standard spelling that is often written in Latin letters ("Tunizi") and mixes in French and English. The author checks how well current models handle it. He built a small evaluation set by hand. It has 100 comments taken from recent social media posts that are unlikely to be indexed, so older Tunisian corpora that models may have memorized are avoided. Each comment carries four annotations: the Tunizi original, an Arabic-script version, an English translation and a sentiment label. Several popular LLMs were run on three tasks: - transliteration from Latin to Arabic script, scored with CER, Levenshtein distance and LCS similarity; - translation to English, scored with BLEU, METEOR and BERTScore; - three-way sentiment classification, scored with accuracy, F1, Cohen's Kappa and MCC. Gemini and GPT came out on top, while Qwen and Mistral fell well behind. Even the best models stay below their level on well-resourced languages. The author frames the work as a pilot that offers a time-stamped, reproducible baseline and argues for national efforts to build larger dialect datasets. ## Key points - The dataset has 100 Tunizi comments from Facebook, X, YouTube, Instagram and forums, covering several regions, topics and registers, including code-switching. - Sentiment labels are split 37% Positive, 35% Negative and 27% Neutral, and one annotator (the author) labeled everything. - On transliteration, Gemini 2.5 Flash scored best, with CER 0.15, Levenshtein 2.89 and LCS 0.88. - Gemini and GPT lead across all three tasks, and Qwen and Mistral trail considerably. The results are framed as indicative, not statistically conclusive. - All model versions are pinned to November 9, 2025. Data, prompts, scripts and raw model outputs are released at github.com/Mahdyy02/llm-tunisian-language. ## Interesting ideas - Build the test set from fresh, non-indexed text so that scores measure generalization rather than recall of public corpora the model may have seen in pretraining. — For Mova (learn): Keep a held-out test set of newly written texts that no LLM teacher could have seen, and use it as a contamination-free control when grading silver data or teacher tiers. - Annotate one item in several parallel layers (original, normalized script, translation, label) so a single small set feeds several tasks and isolates where failures begin. — For Mova (answer): Give each test text its gold tokens, tree, frames and answer together, so a wrong answer can be traced back to the stage that broke. - Score each task with both surface metrics (BLEU, CER) and meaning-level metrics (BERTScore), because spelling variation can hide correct meaning or fake success. — For Mova (back): Judge the text → graph → text compressor on both exact overlap and meaning preservation, so legitimate paraphrases are not punished. - Report chance-corrected agreement (Cohen's Kappa, MCC) and per-class confusion matrices alongside accuracy, since class imbalance can make plain accuracy look misleadingly good. — For Mova (frames): Evaluate speech-act and irony classifiers with Kappa/MCC and confusion matrices to see which pragmatic classes are actually confused. - Date-stamp model versions and publish raw outputs, so a baseline can be compared fairly later even after the models change. — For Mova (learn): Archive each LLM teacher's raw outputs with version and date, so silver data can be re-audited and re-adjudicated when the teachers are updated. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{mahdi2025how, title = {{How Well Do LLMs Understand Tunisian Arabic?}}, author = {Mahdi, Mohamed}, year = {2025}, doi = {10.48550/arxiv.2511.16683}, eprint = {2511.16683}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2511.16683} } ``` - [← previous EvaCun 2025 Shared Task: Lemmatization and Token Prediction in Akkadian and Sumerian using LLMs](https://md.research.elamur.ai/free/en/papers/aclbulk-2025-gordin-evacun-2025-shared-task-lemmatization/) - [next → Lemmatization as a Classification Task: Results from Arabic across Multiple Genres](https://md.research.elamur.ai/free/en/papers/ctx-2025-saeed-lemmatization-as-classification-task-results/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) ## References 7 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. M. Abdul-Mageed, S. Elbassuoni, J. Doughman, A. Elmadany, E. B. Nagoudi, Y. Zoughby, A. Shaher, I. Gaba, A. Helal, M. El-Razzaz, “DiaLex: A Benchmark for Evaluating Multidialectal Arabic Word Embeddings,” in Proceedings of the Sixth Arabic Natural Language Processing Workshop (WANLP), pp. 11–20, 2021. [find](https://scholar.google.com/scholar?q=M%2E+Abdul%2DMageed%2C+S%2E+Elbassuoni%2C+J%2E+Doughman%2C+A%2E+Elmadany%2C+E%2E+B%2E+Nagoudi%2C+Y%2E+Zoughby%2C+A%2E+Shaher%2C+I%2E+Gaba%2C+A%2E+Helal%2C+M%2E+El%2DRazzaz%2C+%E2%80%9CDiaLex%3A+A+Benchmark+for+Evaluating+Multidialectal+Arabic+Word+Embeddin) 2. E. Gugliotta, M. Dinarelli, “TArC: An Arabizi Tunisian Corpus – Incrementally and Semi-Automatically Collecting a Tunisian Arabish Corpus,” in Proceedings of the 12th Conference on Language Resources and Evaluation (LREC), pp. 6279–6286, 2020. [find](https://scholar.google.com/scholar?q=E%2E+Gugliotta%2C+M%2E+Dinarelli%2C+%E2%80%9CTArC%3A+An+Arabizi+Tunisian+Corpus+%E2%80%93+Incrementally+and+Semi%2DAutomatically+Collecting+a+Tunisian+Arabish+Corpus%2C%E2%80%9D+in+Proceedings+of+the+12th+Conference+on+Language+Resources+) 3. A. Messaoudi, A. Cheikhrouhou, H. Haddad, N. Ferchichi, M. BenHajhmida, A. Korched, M. Naski, F. Ghriss, A. Kerkeni, “TunBERT: Pretrained Contextualized Text Representation for Tunisian Dialect,” arXiv preprint arXiv:2111.13138, 2021. [↗](https://arxiv.org/abs/2111.13138) 4. A. Amara, H. Turki, M. A. Hadj Taieb, M. Ben Aouicha, K. Ellouze, “CTAB: Corpus of Tunisian Arabizi,” Zenodo, 2021, doi:10.5281/zenodo.4781769. [↗](https://doi.org/10.5281/zenodo.4781769) 5. K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, “BLEU: a Method for Automatic Evaluation of Machine Translation,” in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 311–318, 2002. [find](https://scholar.google.com/scholar?q=K%2E+Papineni%2C+S%2E+Roukos%2C+T%2E+Ward%2C+W%2E%2DJ%2E+Zhu%2C+%E2%80%9CBLEU%3A+a+Method+for+Automatic+Evaluation+of+Machine+Translation%2C%E2%80%9D+in+Proceedings+of+the+40th+Annual+Meeting+of+the+Association+for+Computational+Linguistics) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/) 6. S. Banerjee, A. Lavie, “METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments,” in Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65–72, 2005. [find](https://scholar.google.com/scholar?q=S%2E+Banerjee%2C+A%2E+Lavie%2C+%E2%80%9CMETEOR%3A+An+Automatic+Metric+for+MT+Evaluation+with+Improved+Correlation+with+Human+Judgments%2C%E2%80%9D+in+Proceedings+of+the+ACL+Workshop+on+Intrinsic+and+Extrinsic+Evaluation+Measures) [our card](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/) 7. T. Zhang, V. Khashabi, Y. R. Liu, P. Clark, N. A. Smith, “BERTScore: Evaluating Text Generation with BERT,” in International Conference on Learning Representations (ICLR), 2020. [find](https://scholar.google.com/scholar?q=T%2E+Zhang%2C+V%2E+Khashabi%2C+Y%2E+R%2E+Liu%2C+P%2E+Clark%2C+N%2E+A%2E+Smith%2C+%E2%80%9CBERTScore%3A+Evaluating+Text+Generation+with+BERT%2C%E2%80%9D+in+International+Conference+on+Learning+Representations+%28ICLR%29%2C+2020%2E)