--- title: "A Comparative Benchmark of a Moroccan Darija — Assoudi 2025" description: "The paper asks how well general-purpose moderation services handle toxic language in Moroccan Darija." authors: ["Hicham Assoudi"] year: 2025 venue: "arXiv:2505.04640" source: "https://arxiv.org/abs/2505.04640" doi: "10.48550/arxiv.2505.04640" arxiv: "2505.04640" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/assoudi-2025-comparative-benchmark-moroccan-darija-toxicity/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/assoudi-2025-comparative-benchmark-moroccan-darija-toxicity/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/) / A Comparative Benchmark of a Moroccan Darija Toxicity Detection Model (Typica.ai) and Major LLM-Based Moderation APIs (OpenAI, Mistral, Anthropic) # A Comparative Benchmark of a Moroccan Darija Toxicity Detection Model (Typica.ai) and Major LLM-Based Moderation APIs (OpenAI, Mistral, Anthropic) **Authors:** Hicham Assoudi · **Year:** 2025 · **Venue:** arXiv:2505.04640 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2505.04640) · [PDF](https://arxiv.org/pdf/2505.04640) · [DOI 10.48550/arxiv.2505.04640](https://doi.org/10.48550/arxiv.2505.04640) · [arXiv 2505.04640](https://arxiv.org/abs/2505.04640) · [OpenAlex](https://openalex.org/W4416095492) **Citations:** 0 citations (openalex-snapshot · 2026-10-14) · 1 in our library, of the 140884 papers whose bibliographies we read ## Summary The paper asks how well general-purpose moderation services handle toxic language in Moroccan Darija. Darija is a low-resource Arabic dialect with frequent code-switching, unstable spelling and culture-bound insults. The author compares a specialised Typica.ai classifier, a fine-tuned BERT-based transformer, against three commercial moderators: OpenAI omni-moderation-latest, Mistral mistral-moderation-latest and Anthropic claude-3-haiku-20240307 (driven by a prompt). The evaluation data is OMCD_Typica.ai_Mix. It starts from the public OMCD corpus and adds proprietary annotations aimed at harder phenomena such as sarcasm, euphemism and indirect insults. The data is then: - filtered to remove Latin-script-dominated text, - balanced 1:1 by undersampling, - split into train, validation and test. Only the test split is used. Each system's output is reduced to a single toxic or clean decision, and one shared script computes precision, recall, F1 and accuracy on the toxic class. According to the paper, the Typica.ai model outperforms all three APIs. The exact scores are not in the extracted text. The work argues that dialect- and culture-adapted models remain necessary for fair moderation outside high-resource languages. ## Key points - **Dataset size:** OMCD_Typica.ai_Mix has 12,758 Darija comments, with 9,568 for training and 2,552 for validation. The test size is not legible in the extraction; subtraction would give 638. - **Preprocessing:** - Sentences with more than 50% Latin characters are discarded. - Latin characters, symbols and punctuation are removed from the rest. - The classes are balanced 1:1 by keeping every toxic example and sampling an equal number of non-toxic ones. - **How each system's output became a binary decision:** - OpenAI: the global `flagged` field. - Mistral: toxic if any category returns true. - Claude: the basic BLOCK/ALLOW prompt from Anthropic's Content Moderation Cookbook, used without modification. - **Reported outcome:** the Typica.ai classifier beats all three commercial systems. The extracted text does not include the per-metric numbers. - **Released and withheld materials:** the evaluation notebook, test split and prediction outputs are on GitHub (assoudi-typica-ai/darija-toxicity-benchmark). Model weights and training data are not released. - **Stated limitations:** the data includes some non-Moroccan dialects, and toxicity annotation is subjective. ## Interesting ideas - Systems with very different output formats (a global flag, per-category flags, a prompted verdict) are each mapped explicitly to one shared binary decision and then scored by a single script, which keeps the comparison honest. — For Mova (learn): Witness parsers and LLM teacher tiers could each be wrapped in a documented adapter to a common label format, so disagreements are measured by one scorer rather than by mismatched conventions. - A widely cited corpus serves as the trusted backbone, and only the difficult cases general models tend to miss (sarcasm, euphemism, indirect insult) are added on top. — For Mova (frames): An irony and implicature test set could reuse existing benchmark sentences and add targeted indirect-speech items, showing exactly where the prag frames fail. - A simple script-ratio threshold removes inputs outside the model's intended scope before any scoring happens. — For Mova (tokens): A cheap character- or language-ratio gate could keep non-English-dominated lines out of silver data and tagger evaluation, so scope noise is not counted as tagging errors. - The prompt-based competitor is run with the vendor's published baseline prompt, unchanged, to avoid unfairly tuning the comparison either way. — For Mova (learn): When an LLM teacher is the reference, a fixed, publicly documented prompt would make its labels a reproducible baseline rather than a moving target. - Releasing the per-system predictions together with the test split lets others rescore or reanalyse the results without access to the models. — For Mova (learn): Saving every evaluated system's raw outputs alongside the gold data would allow later error analysis and negative-control checks without rerunning expensive parsers. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{assoudi2025comparative, title = {{A Comparative Benchmark of a Moroccan Darija Toxicity Detection Model (Typica.ai) and Major LLM-Based Moderation APIs (OpenAI, Mistral, Anthropic)}}, author = {Assoudi, Hicham}, year = {2025}, doi = {10.48550/arxiv.2505.04640}, eprint = {2505.04640}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2505.04640} } ``` - [← previous A Classifier of Word-Level Variants in Witnesses of Biblical Hebrew Manuscripts](https://md.research.elamur.ai/free/en/papers/findings-2025-stoupak-classifier-word-level-variants-witnesses/) - [next → A Comprehensive Evaluation of Large Language Models on Mental Illnesses in Arabic Context](https://md.research.elamur.ai/free/en/papers/zahran-2025-comprehensive-evaluation-large-language-models/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/) ## References 3 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. M. T. I. Khondaker, A. Waheed, E. M. B. Nagoudi, and M. Abdul-Mageed, “GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali, Eds., Singapore: Association for Computational Linguistics, Dec. 2023, pp. 220–247. doi: 10.18653/v1/2023.emnlp-main.16. F. Faisal, M. M. Rahman, and A. Anastasopoulos, “Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties,” Nov. 17, 2024, arXiv: arXiv:2411.10954. doi: 10.48550/arXiv.2411.10954. R. Rachidi, M. A. Ouassil, M. Errami, B. Cherradi, S. Hamida, and H. Silkan, “Classifying toxicity in the Arabic Moroccan dialect on Instagram: a machine and deep learning approach,” Indonesian Journal of Electrical Engineering and Computer Science, vol. 31, no. 1, Art. no. 1, Jul. 2023, doi: 10.11591/ijeecs.v31.i1.pp588-598. R. Zarnoufi, M. Hajhouj, W. Bachri, H. Jaafar, and M. Abik, “MAOffens: Moroccan Arabic Offensive Language Dataset,” in Arabic Language Processing: From Theory to Practice, B. Hdioud and S. L. Aouragh, Eds., Cham: Springer Nature Switzerland, 2025, pp. 17–29. doi: 10.1007/978-3-031-80438-0_2. A. Ibrahimi and A. Mourhir, “Building a Resource for Offensive Language Detection in Moroccan Dialect: A Human-Labeled Dataset,” Dec. 01, 2023, Social Science Research Network, Rochester, NY: 4637624. Accessed: May 03, 2025. [Online]. Available: https://papers.ssrn.com/abstract=4637624 K. Essefar, H. Ait Baha, A. El Mahdaouy, A. El Mekki, and I. Berrada, “OMCD: Offensive Moroccan Comments Dataset,” Lang Resources & Evaluation, vol. 57, no. 4, pp. 1745–1765, Dec. 2023, doi: 10.1007/s10579-023-09663-2. “assoudi-typica-ai/darija-toxicity-benchmark: Reproducibility materials and evaluation notebook for the paper: A Comparative Benchmark of a Moroccan Darija Toxicity Detection Model (Typica.ai) and Major LLM-Based Moderation APIs.” Accessed: May 04, 2025. [Online]. Available: https://github.com/assoudi-typica-ai/darija-toxicity-benchmark “Upgrading the Moderation API with our new multimodal moderation model.” Accessed: May 03, 2025. [Online]. Available: https://openai.com/index/upgrading-the-moderation-api-with-our-new-multimodal-moderation-model/ “Moderation - OpenAI API.” Accessed: May 03, 2025. [Online]. Available: https://platform.openai.com [↗](https://doi.org/10.18653/v1/2023.emnlp-main.16) 2. “Mistral Moderation API | Mistral AI.” Accessed: May 03, 2025. [Online]. Available: [https://mistral.ai/news/mistral-moderation](https://mistral.ai/news/mistral-moderation) “Moderation | Mistral AI Large Language Models.” Accessed: May 03, 2025. [Online]. Available: https://docs.mistral.ai/capabilities/guardrailing/ “Content moderation,” Anthropic. Accessed: May 03, 2025. [Online]. Available: https://docs.anthropic.com/en/docs/about-claude/use-case-guides/content-moderation “anthropic-cookbook/misc/building_moderation_filter.ipynb at main · [↗](https://mistral.ai/news/mistral-moderation) 3. anthropics/anthropic-cookbook,” GitHub. Accessed: May 03, 2025. [Online]. Available: [https://github.com/anthropics/anthropic-cookbook/blob/main/misc/building_moderation_filter.ipyn](https://github.com/anthropics/anthropic-cookbook/blob/main/misc/building_moderation_filter.ipyn) b “Claude 3.5 Haiku.” Accessed: May 03, 2025. [Online]. Available: https://www.anthropic.com/claude/haiku [↗](https://github.com/anthropics/anthropic-cookbook/blob/main/misc/building_moderation_filter.ipyn)