--- title: "MizanQA: Benchmarking Large Language Models on — Bahaj 2025" description: "Arabic legal NLP lacks good evaluation resources. The one earlier Arabic legal benchmark covers Saudi law and relies largely on translated material.…" authors: ["Adil Bahaj", "Mounir Ghogho"] year: 2025 venue: "arXiv:2508.16357" source: "https://arxiv.org/abs/2508.16357" doi: "10.48550/arxiv.2508.16357" arxiv: "2508.16357" licence: "CC BY-NC-SA 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by-nc-sa/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/bahaj-2025-mizanqa-benchmarking-large-language-models/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bahaj-2025-mizanqa-benchmarking-large-language-models/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) / MizanQA: Benchmarking Large Language Models on Moroccan Legal Question Answering # MizanQA: Benchmarking Large Language Models on Moroccan Legal Question Answering **Authors:** Adil Bahaj, Mounir Ghogho · **Year:** 2025 · **Venue:** arXiv:2508.16357 · **Licence:** [CC BY-NC-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-sa/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2508.16357) · [PDF](https://arxiv.org/pdf/2508.16357) · [DOI 10.48550/arxiv.2508.16357](https://doi.org/10.48550/arxiv.2508.16357) · [arXiv 2508.16357](https://arxiv.org/abs/2508.16357) · [OpenAlex](https://openalex.org/W7077465280) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [5](https://md.research.elamur.ai/free/en/papers/bahaj-2025-mizanqa-benchmarking-large-language-models/cited-by/) in our library, of the 140884 papers whose bibliographies we read ## Summary Arabic legal NLP lacks good evaluation resources. The one earlier Arabic legal benchmark covers Saudi law and relies largely on translated material. Moroccan law is harder still. It is written in Modern Standard Arabic but combines Maliki jurisprudence, local customary law and French legal heritage, and its exams often have several correct options per question. The authors built MizanQA from public Moroccan law exams and MCQ banks, producing more than 1,700 multiple-choice items. The pipeline mixes automated and manual steps: - a legal expert removed material tied to repealed legislation; - page images were converted into structured questions by Gemini-2.0-flash; - every item was checked by hand against guidelines and then assigned to a law category. Because items can have several right answers, the authors propose new metrics: strict exact-set accuracy, a penalized F1-like score, a partial-match score with a penalty for wrong picks, and calibration error at both the option level and the answer-set level. Six LLMs were tested, and even the strongest fully solves fewer than half of the questions. The benchmark shows clear regional knowledge gaps and the need for culturally grounded legal models. ## Key points - The dataset is MizanQA, released as `adlbh/MizanQA-v0` on Hugging Face. It has over 1,700 Arabic MCQs, each with 2 to 16 options, and many items require choosing several correct options. - Six models were evaluated: Allam-2 (7b), Gemini-1.5-flash, Gemini-2.0-flash, Llama-3.3 (70b), Llama-4-maverick (17b) and Llama-4-scout (17b). Each model returned its chosen options plus a confidence for each one. - Gemini-2.0-flash scored best on accuracy, with strict ACC 42.11, PMPA(1) 53.57 and F1-like(1) 64.84. Allam-2 was weakest, with ACC 15.32. - Llama-4-maverick was the best calibrated model (ECE opt 17.64, ECE set 29.10). All scores fell as the penalty for wrong picks increased. - Results varied by area of law: - Models did better on the Law of Obligations and Contracts and on the Constitution, which may be closer to international norms. - Models did worse on the Family Code and Criminal Law, where Islamic jurisprudence plays a larger role. ## Interesting ideas - The authors report three levels of credit for multi-answer items: exact set match, partial credit, and partial credit with an adjustable penalty for each wrong option (α in F1-like, β in PMPA). This separates models that guess broadly from models that are precise. — For Mova (answer): School tests with "choose all that apply" items could be scored on all three levels, which would show whether the reader over-selects or misses answers. - Confidence for a whole answer set is computed as the product of the per-option confidences, assuming the options are independent. Calibration is then checked at both the option level and the set level, which reveals overconfidence that per-option checks alone can hide. — For Mova (learn): Mova's explainable answer confidences could be evaluated with both ECE variants, so we can tell whether trust in single facts carries over to trust in combined conclusions. - Before any modelling, an expert filtered the source material by time and removed questions based on outdated legislation. This keeps the benchmark from penalizing answers that are correct today. — For Mova (world): Facts in the compiled knowledge layer could carry validity periods, so that stale facts are filtered out or flagged when an answer depends on time. - Questions were extracted from page images by a multimodal LLM and then verified by humans against written guidelines. This gives a cheap first draft with a controlled check afterwards. — For Mova (learn): This matches the LLM teacher tiers, where the LLM drafts structured items or labels and human or rule-based checks with explicit guidelines accept them. - Breaking scores down by law category exposed the areas where culture-specific knowledge hurts models, which a single aggregate score would hide. — For Mova (answer): QA results on fables, maths problems and school subjects could be reported by category to locate which knowledge or reasoning type fails. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Law](https://md.research.elamur.ai/free/en/shelves/law/) Topics: [Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · [Legal AI](https://md.research.elamur.ai/free/en/topics/legal-ai/) · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{bahaj2025mizanqa, title = {{MizanQA: Benchmarking Large Language Models on Moroccan Legal Question Answering}}, author = {Bahaj, Adil and Ghogho, Mounir}, year = {2025}, doi = {10.48550/arxiv.2508.16357}, eprint = {2508.16357}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2508.16357} } ``` - [← previous LLM Dependency Parsing with In-Context Rules](https://md.research.elamur.ai/free/en/papers/ginn-2025-llm-dependency-parsing-context-rules/) - [next → MultiProSE: A Multi-label Arabic Dataset for Propaganda, Sentiment, and Emotion Detection](https://md.research.elamur.ai/free/en/papers/al-2025-henaki-multiprose-multi-label-arabic-dataset/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) ## References 18 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. M Saiful Bari, Yazeed Alnumay, Norah A Alzahrani, Nouf M Alotaibi, Hisham A Alyahya, Sultan Al-Rashed, Faisal A Mirza, Shaykhah Z Alsubaie, Hassan A Alahmed, Ghadah Alabduljabbar, and 1 others. 2024. Allam: Large language models for arabic and english. arXiv preprint arXiv:2407.15390. [↗](https://arxiv.org/abs/2407.15390) [our card](https://md.research.elamur.ai/free/en/papers/bari-2024-allam-large-language-models-arabic/) 2. Mohamed Bayan Kmainasi, Ali Ezzat Shahroor, and Amani Al-Ghraibah. 2025. Can large language models predict the outcome of judicial decisions? arXiv e-prints, pages arXiv–2501. [find](https://scholar.google.com/scholar?q=Mohamed+Bayan+Kmainasi%2C+Ali+Ezzat+Shahroor%2C+and+Amani+Al%2DGhraibah%2E+2025%2E+Can+large+language+models+predict+the+outcome+of+judicial+decisions%3F+arXiv+e%2Dprints%2C+pages+arXiv%E2%80%932501%2E) [our card](https://md.research.elamur.ai/free/en/papers/kmainasi-2025-large-language-models-predict-outcome/) 3. Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E Ho. 2024. Large legal fictions: Profiling legal hallucinations in large language models. Journal of Legal Analysis, 16(1):64–93. [find](https://scholar.google.com/scholar?q=Matthew+Dahl%2C+Varun+Magesh%2C+Mirac+Suzgun%2C+and+Daniel+E+Ho%2E+2024%2E+Large+legal+fictions%3A+Profiling+legal+hallucinations+in+large+language+models%2E+Journal+of+Legal+Analysis%2C+16%281%29%3A64%E2%80%9393%2E) [our card](https://md.research.elamur.ai/free/en/papers/dahl-2024-large-legal-fictions-profiling-legal/) 4. Mouath Abu Daoud, Chaimae Abouzahir, Leen Kharouf, Walid Al-Eisawi, Nizar Habash, and Farah E Shamout. 2025. Medarabiq: Benchmarking large language models on arabic medical tasks. arXiv preprint arXiv:2505.03427. [↗](https://arxiv.org/abs/2505.03427) [our card](https://md.research.elamur.ai/free/en/papers/daoud-2025-medarabiq-benchmarking-large-language-models/) 5. Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, and 1 others. 2024. Lawbench: Benchmarking legal knowledge of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7933–7962. [find](https://scholar.google.com/scholar?q=Zhiwei+Fei%2C+Xiaoyu+Shen%2C+Dawei+Zhu%2C+Fengzhe+Zhou%2C+Zhuo+Han%2C+Alan+Huang%2C+Songyang+Zhang%2C+Kai+Chen%2C+Zhixin+Yin%2C+Zongwen+Shen%2C+and+1+others%2E+2024%2E+Lawbench%3A+Benchmarking+legal+knowledge+of+large+language) [our card](https://md.research.elamur.ai/free/en/papers/fei-2023-lawbench-benchmarking-legal-knowledge-large/) 6. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. [↗](https://arxiv.org/abs/2407.21783) 7. Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, and 1 others. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Information Processing Systems, 36:44123–44279. [find](https://scholar.google.com/scholar?q=Neel+Guha%2C+Julian+Nyarko%2C+Daniel+Ho%2C+Christopher+R%C3%A9%2C+Adam+Chilton%2C+Alex+Chohlas%2DWood%2C+Austin+Peters%2C+Brandon+Waldon%2C+Daniel+Rockmore%2C+Diego+Zambrano%2C+and+1+others%2E+2023%2E+Legalbench%3A+A+collaboratively+) [our card](https://md.research.elamur.ai/free/en/papers/guha-2023-legalbench-collaboratively-built-benchmark-measuring/) 8. Faris Hijazi, Somayah Alharbi, Abdulaziz AlHussein, Harethah Shairah, Reem Alzahrani, Hebah Al-shamlan, George Turkiyyah, and Omar Knio. 2024. Arablegaleval: A multitask benchmark for assessing arabic legal knowledge in large language models. In Proceedings of The Second Arabic Natural Language Processing Conference, pages 225–249. [find](https://scholar.google.com/scholar?q=Faris+Hijazi%2C+Somayah+Alharbi%2C+Abdulaziz+AlHussein%2C+Harethah+Shairah%2C+Reem+Alzahrani%2C+Hebah+Al%2Dshamlan%2C+George+Turkiyyah%2C+and+Omar+Knio%2E+2024%2E+Arablegaleval%3A+A+multitask+benchmark+for+assessing+arabic) 9. Chakib Lebaidi Ismail Mellouki. 2021. Issues of equivalence in the moroccan legal text. Journal of University Studies for Inclusive Research, pages 1456–1478. [find](https://scholar.google.com/scholar?q=Chakib+Lebaidi+Ismail+Mellouki%2E+2021%2E+Issues+of+equivalence+in+the+moroccan+legal+text%2E+Journal+of+University+Studies+for+Inclusive+Research%2C+pages+1456%E2%80%931478%2E) 10. Haitao Li, Junjie Chen, Jingli Yang, Qingyao Ai, Wei Jia, Youfeng Liu, Kai Lin, Yueyue Wu, Guozhi Yuan, Yiran Hu, and 1 others. 2024. Legalagentbench: Evaluating llm agents in legal domain. arXiv preprint arXiv:2412.17259. [↗](https://arxiv.org/abs/2412.17259) [our card](https://md.research.elamur.ai/free/en/papers/li-2025-legalagentbench-evaluating-llm-agents-legal/) 11. Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29. [find](https://scholar.google.com/scholar?q=Mahdi+Pakdaman+Naeini%2C+Gregory+Cooper%2C+and+Milos+Hauskrecht%2E+2015%2E+Obtaining+well+calibrated+probabilities+using+bayesian+binning%2E+In+Proceedings+of+the+AAAI+conference+on+artificial+intelligence%2C+vol) 12. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774. [↗](https://arxiv.org/abs/2303.08774) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/) 13. Nicholas Pipitone and Ghita Houir Alami. 2024. Legalbench-rag: A benchmark for retrieval-augmented generation in the legal domain. arXiv preprint arXiv:2408.10343. [↗](https://arxiv.org/abs/2408.10343) 14. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, and 1 others. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. [↗](https://arxiv.org/abs/2308.16149) [our card](https://md.research.elamur.ai/free/en/papers/sengupta-2023-jais-jais-chat-arabic-centric/) 15. Mikolaj Sitarz. 2022. Extending f1 metric, probabilistic approach. arXiv preprint arXiv:2210.11997. [↗](https://arxiv.org/abs/2210.11997) 16. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. [↗](https://arxiv.org/abs/2312.11805) [our card](https://md.research.elamur.ai/free/en/papers/gemini-2023-team-gemini-family-highly-capable-multimodal/) 17. Meta Llama Team. 2025. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation — ai.meta.com. [https://ai.meta.com/blog/llama-4-multimodal-intelligence/](https://ai.meta.com/blog/llama-4-multimodal-intelligence/). [Accessed 05-05-2025]. [↗](https://ai.meta.com/blog/llama-4-multimodal-intelligence/) 18. Lin Yang, Shawn Xu, Andrew Sellergren, Timo Kohlberger, Yuchen Zhou, Ira Ktena, Atilla Kiraly, Faruk Ahmed, Farhad Hormozdiari, Tiam Jaroensri, and 1 others. 2024. Advancing multimodal medical capabilities of gemini. arXiv preprint arXiv:2405.03162. [↗](https://arxiv.org/abs/2405.03162)