--- title: "Mizan — paper by Alseelawi and Aljumaily (2026)" description: "Arabic LLM leaderboards are built almost entirely on Modern Standard Arabic (MSA), and strong models now score near the… 12 references." authors: ["Nawar S. Alseelawi", "Mustafa S. Aljumaily"] year: 2026 venue: "arXiv:2609.13980" source: "https://arxiv.org/abs/2609.13980" doi: "10.48550/arxiv.2609.13980" arxiv: "2609.13980" licence: "arXiv — non-exclusive license to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/alseelawi-2026-mizan-national-benchmark-evaluating-large/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/alseelawi-2026-mizan-national-benchmark-evaluating-large/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/37/) / Mizan: A National Benchmark for Evaluating Large Language Models on Iraqi Arabic and the Iraqi Civic Context # Mizan: A National Benchmark for Evaluating Large Language Models on Iraqi Arabic and the Iraqi Civic Context **Authors:** Nawar S. Alseelawi and Mustafa S. Aljumaily · **Year:** 2026 · **Venue:** arXiv:2609.13980 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2609.13980) · [PDF](https://arxiv.org/pdf/2609.13980) · [DOI 10.48550/arxiv.2609.13980](https://doi.org/10.48550/arxiv.2609.13980) · [arXiv 2609.13980](https://arxiv.org/abs/2609.13980) · [OpenAlex](https://openalex.org/W7213257739) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Arabic LLM leaderboards are built almost entirely on Modern Standard Arabic (MSA), and strong models now score near the ceiling on them. They say nothing about how models handle Iraqi Arabic or the Iraqi administrative and civic setting. Mizan fills that gap. It pairs an MSA baseline track with an Iraqi track covering six axes: dialect comprehension, dialect generation, MSA↔Iraqi translation, Iraq-specific knowledge, field extraction from official letters, and safety. Native speakers wrote all 340 items from scratch, and each was reviewed by two people. Answer positions were statistically checked, and every reported score carries a Wilson interval. A pilot run on 27 systems found four things: - MSA saturates, while the Iraqi track spreads models apart. - Every model shows a steady internal gap between the two tracks. - No system extracts document fields well. - Arabic-specialized models are really MSA-specialized. A safety-hardened frontier tier also refused harmless dialect questions. The accompanying platform enforces an integrity protocol, and that protocol was actually used during the study. Mizan matters as a procurement tool for state use of LLMs. ## Key points - The bank has 340 originally authored items, about 50 per axis, which gives intervals of roughly ±10 points per axis. The 150 open-generation items (generation, translation, safety) still await human judging: 4,050 responses, double-judged and checked with Krippendorff's alpha. - The 27 evaluated systems include closed frontier models (one, GPT-6 Astra, tested three days after release), open-weight models of several sizes, and three Arabic models: a commercial, an open-specialized, and a sovereign one. Self-hosted Arabic models ran in 4-bit quantization. - Each model scores 14–18 points lower on the Iraqi track than on MSA. On official-document extraction, all systems fall between 32 and 56. - Two dedicated Arabic models score below a generalist of the same size on the Iraqi track. - The safety-hardened tier refused 12% of Iraqi comprehension items (6/50), against 0% on MSA and 0% for every other system. Mosuli dialect has no coverage yet. A sealed private test tier and a bank of over 1,000 items are planned for v1.0. ## Interesting ideas - Write test items natively rather than translating existing benchmarks, because translated items measure the translator's phrasing and the source culture, not real competence. - Pair a saturated baseline track with a discriminating track and report each system's internal gap between them; the gap stays informative even when absolute ranks are tied. - Put a confidence interval on every published number and treat rankings whose intervals overlap as unsettled rather than declaring a winner. - Write some documents with optional fields deliberately missing, so that a system which hallucinates a value is caught by strict field matching. - Run evaluation under an integrity protocol: dated snapshots that cannot be altered, hash-based certification, a required sign-off by a person before anything goes public, and open withdrawal of flawed runs, followed by full re-verification with run-to-run agreement. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (answer): Write school-test and fable QA items directly in natural English rather than converting them from other datasets, so the readers are graded on real text and not on conversion artifacts. - **I2** (learn): Evaluate each parser or reader version on both an in-domain set and a hard or out-of-domain set, and track the per-version gap as a robustness signal. - **I3** (learn): Attach Wilson intervals to LAS and QA accuracy in evaluation reports, so small test sets do not produce false claims of improvement. - **I4** (world): Add texts that leave a slot (owner, time, quantity) unstated, and check that the world builder leaves it empty instead of filling it in; this works as a built-in negative control. - **I5** (learn): Snapshot the silver data and evaluation outputs with SHA-256 hashes and record retractions, so any reported score can be reproduced and broken runs are visibly withdrawn. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Misan](https://md.research.elamur.ai/free/en/universities/university-of-misan/) — University, Amara, Iraq · 2 works 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works ## Cite Cite as: Nawar S. Alseelawi and Mustafa S. Aljumaily. 2026. Mizan: A National Benchmark for Evaluating Large Language Models on Iraqi Arabic and the Iraqi Civic Context. arXiv:2609.13980. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{alseelawi2026mizan, title = {{Mizan: A National Benchmark for Evaluating Large Language Models on Iraqi Arabic and the Iraqi Civic Context}}, author = {Nawar S. Alseelawi and Mustafa S. Aljumaily}, year = {2026}, doi = {10.48550/arxiv.2609.13980}, eprint = {2609.13980}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2609.13980} } ``` - [← previous Mixture of Heterogeneous Grouped Experts for Language Modeling](https://md.research.elamur.ai/free/en/papers/ma-2026-mixture-heterogeneous-grouped-experts-language/) - [next → MizanQA: A Benchmark for Multi-Answer Moroccan Legal QA](https://md.research.elamur.ai/free/en/papers/bahaj-2026-mizanqa-benchmark-multi-answer-moroccan/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/37/) ## References 12 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Muhammad Abdul-Mageed et al. 2020–2024. The NADI shared tasks: Nuanced Arabic dialect identification (series). Proceedings of the WANLP / ArabicNLP Workshops, Association for Computational Linguistics. Annual shared-task series. [find](https://scholar.google.com/scholar?q=Muhammad+Abdul%2DMageed+et+al%2E+2020%E2%80%932024%2E+The+NADI+shared+tasks%3A+Nuanced+Arabic+dialect+identification+%28series%29%2E+Proceedings+of+the+WANLP+%2F+ArabicNLP+Workshops%2C+Association+for+Computational+Linguistics) 2. Rawan Al-Matham et al. 2025. [BALSAM: A platform for benchmarking Arabic large language models](https://md.research.elamur.ai/free/en/papers/matham-2025-balsam-platform-benchmarking-arabic-large/). Preprint, arXiv:2507.22603. [↗](https://arxiv.org/abs/2507.22603) 3. Ebtesam Almazrouei, Ruxandra Cojocaru, Michele Baldo, Quentin Malartic, Hamza Alobeidli, Daniele Mazzotta, Guilherme Penedo, Giulia Campesan, Mugariya Farooq, Maitha Alhammadi, Julien Launay, and Badreddine Noune. 2023. [AlGhafa evaluation benchmark for Arabic language models](https://md.research.elamur.ai/free/en/papers/almazrouei-2023-alghafa-evaluation-benchmark-arabic-language/). In Proceedings of ArabicNLP 2023, pages 244–275. Association for Computational Linguistics. 4. Malik H. Altakrori, Nizar Habash, Abed Alhakim Freihat, Younes Samih, Kirill Chirkunov, Muhammed AbuOdeh, Radu Florian, Teresa Lynn, Preslav Nakov, and Alham Fikri Aji. 2025. [DialectalArabicMMLU: Benchmarking dialectal capabilities in Arabic and multilingual language models](https://md.research.elamur.ai/free/en/papers/altakrori-2026-dialectalarabicmmlu-benchmarking-dialectal-capabilities-arabic/). Preprint, arXiv:2510.27543. Accepted to LREC 2026. [↗](https://arxiv.org/abs/2510.27543) 5. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Houda+Bouamor%2C+Nizar+Habash%2C+Mohammad+Salameh%2C+Wajdi+Zaghouani%2C+Owen+Rambow%2C+Dana+Abdulrahim%2C+Ossama+Obeid%2C+Salam+Khalifa%2C+Fadhl+Eryani%2C+Alexander+Erdmann%2C+and+Kemal+Oflazer%2E+2018%2E+The+MADAR+Arabic+di) 6. Ali Elfilali et al. 2024. Open Arabic LLM leaderboard (OALL). Hugging Face / Technology Innovation Institute. [find](https://scholar.google.com/scholar?q=Ali+Elfilali+et+al%2E+2024%2E+Open+Arabic+LLM+leaderboard+%28OALL%29%2E+Hugging+Face+%2F+Technology+Innovation+Institute%2E) 7. Leo Gao et al. 2023. A framework for few-shot language model evaluation (lm-evaluation-harness). Version v0.4.0. [find](https://scholar.google.com/scholar?q=Leo+Gao+et+al%2E+2023%2E+A+framework+for+few%2Dshot+language+model+evaluation+%28lm%2Devaluation%2Dharness%29%2E+Version+v0%2E4%2E0%2E) 8. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. [ArabicMMLU: Assessing massive multitask language understanding in Arabic](https://md.research.elamur.ai/free/en/papers/koto-2024-arabicmmlu-assessing-massive-multitask-language/). In Findings of the Association for Computational Linguistics: ACL 2024, pages 5622–5640, Bangkok, Thailand. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2402.12840) 9. Klaus Krippendorff. 2019. Content Analysis: An Introduction to Its Methodology, 4th edition. SAGE Publications. Basel Mousi, Nadir Durrani, Fatema Ahmad, Md. [find](https://scholar.google.com/scholar?q=Klaus+Krippendorff%2E+2019%2E+Content+Analysis%3A+An+Introduction+to+Its+Methodology%2C+4th+edition%2E+SAGE+Publications%2E+Basel+Mousi%2C+Nadir+Durrani%2C+Fatema+Ahmad%2C+Md%2E) 10. Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2025. [AraDiCE: Benchmarks for dialectal and cultural capabilities in LLMs](https://md.research.elamur.ai/free/en/papers/mousi-2025-aradice-benchmarks-dialectal-cultural-capabilities/). In Proceedings of the 31st International Conference on Computational Linguistics (COLING 2025), pages 4186–4218, Abu Dhabi, UAE. Association for Computational Linguistics. 11. Stanford CRFM and Arabic.AI. 2025. HELM Arabic. Stanford Center for Research on Foundation Models. [find](https://scholar.google.com/scholar?q=Stanford+CRFM+and+Arabic%2EAI%2E+2025%2E+HELM+Arabic%2E+Stanford+Center+for+Research+on+Foundation+Models%2E) 12. Edwin B. Wilson. 1927. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209–212. [↗](https://doi.org/10.1080/01621459.1927.10502953) *Updated 1 November 2026*