--- title: "Empirical Evaluation of Pre-trained Language Models for" description: "Moroccan Darija (MD) is the main spoken Arabic variety in Morocco, yet it has few NLP resources… 4 references, cited by 1 paper here." authors: ["Azzedine Aftiss", "Salima Lamsiyah", "Christoph Schommer", "Said Ouatik El Alaoui"] year: 2025 venue: "Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL-4)" source: "https://aclanthology.org/2025.wacl-1.9/" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/aftiss-2025-empirical-evaluation-pre-trained-language/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/aftiss-2025-empirical-evaluation-pre-trained-language/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/14/) / Empirical Evaluation of Pre-trained Language Models for Summarizing Moroccan Darija News Articles # Empirical Evaluation of Pre-trained Language Models for Summarizing Moroccan Darija News Articles **Authors:** Azzedine Aftiss, [Salima Lamsiyah](https://md.research.elamur.ai/free/en/authors/salima-lamsiyah/), [Christoph Schommer](https://md.research.elamur.ai/free/en/authors/christoph-schommer/) and [Said Ouatik El Alaoui](https://md.research.elamur.ai/free/en/authors/said-ouatik-el-alaoui/) · **Year:** 2025 · **Venue:** [Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL-4)](https://md.research.elamur.ai/free/en/venues/wacl/) ([Workshop on Arabic Corpus Linguistics 2025](https://md.research.elamur.ai/free/en/venues/wacl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.wacl-1.9/) · [PDF](https://aclanthology.org/2025.wacl-1.9.pdf) · [OpenAlex](https://openalex.org/W7110486887) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/aftiss-2025-empirical-evaluation-pre-trained-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Moroccan Darija (MD) is the main spoken Arabic variety in Morocco, yet it has few NLP resources, particularly for summarization. The authors extend the earlier GOUD.MA news summarization benchmark. They crawled the goud.ma site again and collected 50,517 new articles across 11 categories, released as GOOD.MA_v2. In this corpus the headline, usually written in Darija, serves as the target summary. Article bodies are written either in MSA alone or in MSA combined with Darija. They compare three groups of models: - **Encoder-only models**, warm-started as Seq2Seq systems: AraBERT, DarijaBERT, DziriBERT. - **Native encoder-decoders**, fine-tuned directly: AraBART, AraT5, mBART. - **Causal LLMs**, prompted zero-shot: GPT-4o mini, Llama 3, Mistral NeMo. Evaluation uses ROUGE and BERTScore. Training on the enlarged data beats the scores originally reported for GOUD.MA, and mBART performs best overall. Zero-shot LLMs and extractive baselines fall behind the fine-tuned models. The authors also caution that overlap-based metrics may undervalue legitimate paraphrases. The data, models and code are open-sourced, giving a reusable base for dialectal Arabic generation. ## Key points - The new data comes from goud.ma, crawled with Scrapy and Selenium between 2022 and October 2024. The crawl yielded 50,517 articles with date, title, content and category, stored as CSV. - The merged training split has 189,805 articles, up from 139,288 in GOUD.MA. Validation and test are the original GOUD.MA sets, with 9,497 articles each. Articles average about 254 tokens and titles about 16.4 tokens. - Encoder-only models were turned into encoder-decoders following Rothe et al. (2020). All fine-tuning ran for 20 epochs with batch size 20 and gradient accumulation of 8. - Extractive baselines were Lead-2, TextRank and SumRank. Metrics were ROUGE-1, ROUGE-2, ROUGE-L and BERTScore. - mBART ranked first, with AraBART and AraT5 close behind. Mistral NeMo hallucinated and sometimes answered in French or Spanish, so its outputs were machine-translated with the Google Translate API before scoring. ## Interesting ideas - News headlines can act as free reference summaries, giving a large supervised summarization set with no manual annotation. This is a cheap way to obtain training and evaluation pairs. - Adding training data while keeping the original validation and test sets fixed isolates the effect of extra data from changes in the benchmark. - Encoder-only checkpoints can be warm-started into encoder-decoder generators, reusing representations trained for understanding as the basis for generation. - Zero-shot LLMs may drift into the wrong output language, and repairing that afterwards with machine translation confounds the comparison. Such repairs should be detected and reported explicitly. - Overlap metrics such as ROUGE can penalize faithful paraphrases. The authors therefore treat their scores as partial and call for human judgment. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (back): headline–body pairs could serve as a test bed for the text → graph → text compressor, checking whether the generated short text keeps the headline's core frame. - **I2** (learn): when silver data agreed by witness parsers is added, scoring always against the same frozen gold split shows whether the new data actually helps. - **I3** (back): the encoder of the gparse transformer parser could initialize a generator, so the analysis and generation sides share representations. - **I4** (learn): LLM teacher tiers should check output language and format before outputs become silver labels, and log any repair step as a possible confound. - **I5** (back): compressor output could be judged by comparing re-parsed frames against the source meaning graph, rather than by n-gram overlap alone. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Luxembourg](https://md.research.elamur.ai/free/en/universities/university-of-luxembourg/) — University, Luxembourg, Luxembourg · 27 works · 337 citations - [Université Ibn-Tofail](https://md.research.elamur.ai/free/en/universities/universite-ibn-tofail/) — University, Kenitra, Morocco · 3 works · 4 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [📝 Summarization](https://md.research.elamur.ai/free/en/topics/summarization/) · 232 works ## Cite Cite as: Azzedine Aftiss, Salima Lamsiyah, Christoph Schommer and Said Ouatik El Alaoui. 2025. Empirical Evaluation of Pre-trained Language Models for Summarizing Moroccan Darija News Articles. Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL-4). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{aftiss2025empirical, title = {{Empirical Evaluation of Pre-trained Language Models for Summarizing Moroccan Darija News Articles}}, author = {Azzedine Aftiss, Salima Lamsiyah, Christoph Schommer and Said Ouatik {El Alaoui}}, booktitle = {Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL-4)}, year = {2025}, url = {https://aclanthology.org/2025.wacl-1.9/} } ``` - [← previous Emergent morpho-phonological representations in self-supervised speech models](https://md.research.elamur.ai/free/en/papers/gauthier-2025-emergent-morpho-phonological-representations-self/) - [next → English Pronunciation Evaluation without Complex Joint Training: LoRA Fine-tuned Speech Multimodal LLM](https://md.research.elamur.ai/free/en/papers/ahn-2025-english-pronunciation-evaluation-complex-joint/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/14/) ## References 4 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Amine Abdaoui, Mohamed Berrimi, Mourad Oussalah, and Abdelouahab Moussaoui. 2021. [Dziribert: a pre-trained language model for the algerian dialect](https://md.research.elamur.ai/free/en/papers/abdaoui-2021-dziribert-pre-trained-language-model/). arXiv preprint arXiv:2109.12346. [↗](https://arxiv.org/abs/2109.12346) 2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774. [↗](https://arxiv.org/abs/2303.08774) 3. Molham Al-Maleh and Said Desouki. 2020. Arabic [https://github.com/ssut/py-googletrans](https://github.com/ssut/py-googletrans) GOOD.MA Models GOOD.MA_v2 R-1 R-2 R-L BERTScore R-1 [↗](https://github.com/ssut/py-googletrans) 4. AraBERT [Antoun et al., 2020] 23.08 8.98 22.06 DarijaBERT [Gaanoun et al., 2024] 19.41 6.64 18.48 DziriBERT [Abdaoui et al., 2021] 17.98 5.83 17.22 -- - R-2 R-L BERTScore 28.47 15.78 25.47 27.79 15.53 24.8 24.16 12.42 21.89 64.79 64.24 63.11 [find](https://scholar.google.com/scholar?q=AraBERT+%5BAntoun+et+al%2E%2C+2020%5D+23%2E08+8%2E98+22%2E06+DarijaBERT+%5BGaanoun+et+al%2E%2C+2024%5D+19%2E41+6%2E64+18%2E48+DziriBERT+%5BAbdaoui+et+al%2E%2C+2021%5D+17%2E98+5%2E83+17%2E22+%2D%2D+%2D+R%2D2+R%2DL+BERTScore+28%2E47+15%2E78+25%2E47+27%2E79+15%2E53) *Updated 31 October 2026*