---
title: "Are Arabic Benchmarks Reliable? QIMMA's — AlQadi 2026"
description: "Arabic LLM evaluation draws on many benchmarks, but they are uneven."
authors: ["Leen AlQadi", "Ahmed Alzubaidi", "Mohammed Alyafeai", "Hamza Alobeidli", "Maitha Alhammadi", "Shaikha Alsuwaidi", "Omar Alkaabi", "Basma El Amel Boussaha", "Hakim Hacid"]
year: 2026
venue: "arXiv:2604.03395"
source: "https://arxiv.org/abs/2604.03395"
doi: "10.48550/arxiv.2604.03395"
arxiv: "2604.03395"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/alqadi-2026-arabic-benchmarks-reliable-qimma-s/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/alqadi-2026-arabic-benchmarks-reliable-qimma-s/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/) / Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation

# Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation

**Authors:** Leen AlQadi, Ahmed Alzubaidi, Mohammed Alyafeai, [Hamza Alobeidli](https://md.research.elamur.ai/free/en/authors/hamza-alobeidli/), Maitha Alhammadi, Shaikha Alsuwaidi, Omar Alkaabi, Basma El Amel Boussaha, Hakim Hacid · **Year:** 2026 · **Venue:** arXiv:2604.03395 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2604.03395) · [PDF](https://arxiv.org/pdf/2604.03395) · [DOI 10.48550/arxiv.2604.03395](https://doi.org/10.48550/arxiv.2604.03395) · [arXiv 2604.03395](https://arxiv.org/abs/2604.03395) · [OpenAlex](https://openalex.org/W7151452899)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14)

## Summary
Arabic LLM evaluation draws on many benchmarks, but they are uneven. Several are translated from English, many native sets were never carefully checked, coverage is scattered, and outputs are rarely published. QIMMA is a leaderboard that audits its sources before scoring any model. The authors chose 14 mostly native Arabic datasets, split into 109 subsets and over 52k samples. These cover culture, STEM, law, medicine, poetry, safety and code. Every sample was screened by two strong open LLMs, Qwen3-235B-A22B-Instruct and DeepSeek-V3-671B, and humans then reviewed what the models flagged. Flawed items were removed, and the Arabic code prompts were rewritten. The audit found recurring defects even in well-known benchmarks, such as wrong gold answers, cultural bias and malformed questions. Evaluation runs through LightEval and EvalPlus, and per-sample outputs are released. Among instruct models, Jais-2-70B-Chat led narrowly overall and topped four domains. The work matters because it shows that scores on unaudited benchmarks can partly reflect dataset flaws rather than model skill.

## Key points
- The suite has 14 source datasets and 109 subsets with more than 52k samples. About 99% is native Arabic; only the code tasks (3LM versions of HumanEval+ and MBPP+) are not.
- Discard counts differed widely by dataset. ArabicMMLU lost 436 items and kept 13,727, while MizanQA lost 41, FannOrFlop 43 and MedArabiQ 33. Some sets, such as AraTrust and ArabLegalQA, lost nothing.
- The code prompts were rewritten at a high rate: 145 of 164 (88%) in HumanEval+ and 308 of 378 (81%) in MBPP+.
- The two judges were chosen for three reasons: strong Arabic, reliable instruction-following on long prompts, and open weights. They were trained on different data, so they make different mistakes.
- In the results, Jais-2-70B-Chat averaged 65.81 and Qwen2.5-72B-Instruct 65.75. Jais-2-8B-Chat (57.89) was the best model near 8B, and gpt-oss-20b came last (32.10).
- Thinking mode helped coding, where Qwen3.5-27B beat Qwen2.5-72B-Instruct. It did not help generative legal or medical QA.
- Gains from model size flatten past roughly 70B parameters.

## Interesting ideas
- Audit the test set before trusting any score: run two judge models with different training data over every item, and have humans check only the flagged ones, which keeps manual work small. — For Mova (learn): Before running school-course tests or fable QA, we could screen gold answers with two LLM teachers and hand-review only items they question, so failures are not blamed on the reader when the key is wrong.
- Report kept and discarded counts per subset instead of one global figure, so readers can see which sources are noisy. — For Mova (tree): When converting treebanks between UD dialects or filtering silver data, we could log per-treebank rejection rates to spot sources whose annotation conventions clash with ours.
- Turn thinking traces off for tasks with a strict output format, because long reasoning can use up the length budget before the required structure appears. — For Mova (learn): For LLM teacher tiers that must emit structured frames or trees, we could cap or skip reasoning so the teacher returns parseable output instead of truncated text.
- Publish per-sample outputs alongside aggregate scores, so anyone can re-check errors or extend the evaluation. — For Mova (answer): Storing each answer together with its explanation trace for every test item would let us compare reader versions item by item and find regressions behind a stable average.
- Build a named taxonomy of benchmark defects (wrong gold answer, cultural bias, malformed question) and reuse it as a checklist when adding new resources. — For Mova (world): When importing Wikidata or FrameNet facts into the knowledge graph, we could tag suspect entries with a fixed defect category so cleanup can be tracked and prioritised.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{alqadi2026are,
  title = {{Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation}},
  author = {AlQadi, Leen and Alzubaidi, Ahmed and Alyafeai, Mohammed and Alobeidli, Hamza and Alhammadi, Maitha and Alsuwaidi, Shaikha and Alkaabi, Omar and Boussaha, Basma El Amel and Hacid, Hakim},
  year = {2026},
  doi = {10.48550/arxiv.2604.03395},
  eprint = {2604.03395},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2604.03395}
}
```

- [← previous Archaeology at MWE-2026 PARSEME 2.0 Subtasks 1 and 2: Parsing is for Encoders, Paraphrasing is for LLMs](https://md.research.elamur.ai/free/en/papers/roscan-2026-archaeology-mwe-2026-parseme-2/)
- [next → Are Language Models Borrowing-Blind? A Multilingual Evaluation of Loanword Identification across 10 Languages](https://md.research.elamur.ai/free/en/papers/silva-2026-are-language-models-borrowing-blind/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/)

## References

38 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ahmed Abdelali, Hamdy Mubarak, Shammur Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdaljalil, Yassine El Kheir, Daniel Izham, Fahim Dalvi, and 1 others. 2024. LAraBench: Benchmarking Arabic AI with large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 487–520, St. Julian’s, Malta. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ahmed+Abdelali%2C+Hamdy+Mubarak%2C+Shammur+Chowdhury%2C+Maram+Hasanain%2C+Basel+Mousi%2C+Sabri+Boughorbel%2C+Samir+Abdaljalil%2C+Yassine+El+Kheir%2C+Daniel+Izham%2C+Fahim+Dalvi%2C+and+1+others%2E+2024%2E+LAraBench%3A+Benchmark) [our card](https://md.research.elamur.ai/free/en/papers/abdelali-2024-larabench-benchmarking-arabic-ai-large/)
2. Harethah Abu Shairah, Somayah AlHarbi, Abdulaziz AlHussein, and 1 others. 2025. ALARB: An Arabic legal argument reasoning benchmark. Preprint, arXiv:2510.00694. [↗](https://arxiv.org/abs/2510.00694)
3. Inception AI and MBZUAI. 2024. AraGen leaderboard: The first generative tasks leaderboard for arabic. [https://huggingface.co/spaces/inceptiona](https://huggingface.co/spaces/inceptiona) i/Leaderboards. [↗](https://huggingface.co/spaces/inceptiona)
4. Shahad Al-Khalifa, Nadir Durrani, Hend Al-Khalifa, and Firoj Alam. 2025. The landscape of Arabic large language models. Communications of the ACM. [find](https://scholar.google.com/scholar?q=Shahad+Al%2DKhalifa%2C+Nadir+Durrani%2C+Hend+Al%2DKhalifa%2C+and+Firoj+Alam%2E+2025%2E+The+landscape+of+Arabic+large+language+models%2E+Communications+of+the+ACM%2E)
5. Safa AlBallaa, Nora AlTwairesh, Abdulmalik AlSalman, and Sultan Alfarhood. 2025. GATmath and GATLc: Comprehensive benchmarks for evaluating Arabic large language models. PLoS One, 20(9):e0329129. [find](https://scholar.google.com/scholar?q=Safa+AlBallaa%2C+Nora+AlTwairesh%2C+Abdulmalik+AlSalman%2C+and+Sultan+Alfarhood%2E+2025%2E+GATmath+and+GATLc%3A+Comprehensive+benchmarks+for+evaluating+Arabic+large+language+models%2E+PLoS+One%2C+20%289%29%3Ae0329129%2E)
6. Wafa Alghallabi, Ritesh Thawkar, Sara Ghaboura, Ketan More, Omkar Thawakar, Hisham Cholakkal, Salman Khan, and Rao Muhammad Anwer. 2025. Fann or flop: A multigenre, multiera benchmark for arabic poetry understanding in llms. Preprint, arXiv:2505.18152. [↗](https://arxiv.org/abs/2505.18152)
7. Emad A. Alghamdi, Reem I. Masoud, Deema Alnuhait, Afnan Y. Alomairi, Ahmed Ashraf, and Mohamed Zaytoon. 2024. AraTrust: An evaluation of trustworthiness for LLMs in Arabic. Preprint, arXiv:2403.09017. [↗](https://arxiv.org/abs/2403.09017)
8. Rawan Nasser Almatham, Kareem Mohamed Darwish, Raghad Al-Rasheed, Waad Thuwaini Alshammari, Muneera Alhoshan, Amal Almazrua, Asma Al Wazrah, Mais Alheraki, Firoj Alam, Preslav Nakov, and 1 others. 2025. BALSAM: A platform for benchmarking Arabic large language models. In Proceedings of The Third Arabic Natural Language Processing Conference, pages 258–277, Suzhou, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rawan+Nasser+Almatham%2C+Kareem+Mohamed+Darwish%2C+Raghad+Al%2DRasheed%2C+Waad+Thuwaini+Alshammari%2C+Muneera+Alhoshan%2C+Amal+Almazrua%2C+Asma+Al+Wazrah%2C+Mais+Alheraki%2C+Firoj+Alam%2C+Preslav+Nakov%2C+and+1+others%2E+202)
9. Malik H Altakrori and 1 others. 2025. DialectalArabicMMLU: Benchmarking dialectal capabilities in Arabic and multilingual language models. Preprint, arXiv:2510.27543. [↗](https://arxiv.org/abs/2510.27543) [our card](https://md.research.elamur.ai/free/en/papers/altakrori-2026-dialectalarabicmmlu-benchmarking-dialectal-capabilities-arabic/)
10. Fakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed, and Muhammad Abdul-Mageed. 2025. PalmX 2025: The first shared task on benchmarking LLMs on Arabic and Islamic culture. Preprint, arXiv:2509.02550. [↗](https://arxiv.org/abs/2509.02550)
11. Ahmed Alzubaidi, Shaikha Alsuwaidi, Basma El Amel Boussaha, Leen AlQadi, Omar Alkaabi, Mohammed Alyafeai, Hamza Alobeidli, and Hakim Hacid. 2025. Evaluating arabic large language models: A survey of benchmarks, methods, and gaps. Preprint, arXiv:2510.13430. [↗](https://arxiv.org/abs/2510.13430) [our card](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/)
12. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. [find](https://scholar.google.com/scholar?q=Wissam+Antoun%2C+Fady+Baly%2C+and+Hazem+Hajj%2E+2020%2E+Arabert%3A+Transformer%2Dbased+model+for+arabic+language+understanding%2E+In+LREC+2020+Workshop+Language+Resources+and+Evaluation+Conference+11%E2%80%9316+May+2020%2C+p) [our card](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/)
13. Mohamed Anwar, Abdelhakim Freihat, George Ibrahim, Mostafa Awad, Abdelrahman Atef Mohamed Ali Sadallah, Gurpreet Gosal, Gokul Ramakrishnan, Sarath Chandran, Biswajit Mishra, Rituraj Joshi, Ahmed Frikha, Etienne Goffinet, Abhishek Maiti, Ali El Filali, Sarah Al Barri, Samujjwal Ghosh, Rahul Pal, Parvez Mullah, Awantika Shukla, and 41 others. 2025. Jais 2: A family of Arabic-centric open large language models. Technical report, IFM. [find](https://scholar.google.com/scholar?q=Mohamed+Anwar%2C+Abdelhakim+Freihat%2C+George+Ibrahim%2C+Mostafa+Awad%2C+Abdelrahman+Atef+Mohamed+Ali+Sadallah%2C+Gurpreet+Gosal%2C+Gokul+Ramakrishnan%2C+Sarath+Chandran%2C+Biswajit+Mishra%2C+Rituraj+Joshi%2C+Ahmed+Frikh) [our card](https://md.research.elamur.ai/free/en/papers/anwar-2026-jais-2-family-arabic-centric/)
14. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Sebastian+Ruder%2C+and+Dani+Yogatama%2E+2020%2E+On+the+cross%2Dlingual+transferability+of+monolingual+representations%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computati) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/)
15. Adil Bahaj and Mounir Ghogho. 2025. MizanQA: Benchmarking large language models on Moroccan legal question answering. [find](https://scholar.google.com/scholar?q=Adil+Bahaj+and+Mounir+Ghogho%2E+2025%2E+MizanQA%3A+Benchmarking+large+language+models+on+Moroccan+legal+question+answering%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahaj-2025-mizanqa-benchmarking-large-language-models/)
16. Basma El Amel Boussaha, Leen AlQadi, Mugariya Farooq, Shaikha Alsuwaidi, Giulia Campesan, Ahmed Alzubaidi, Mohammed Alyafeai, and Hakim Hacid. 2025. 3LM: Bridging Arabic, STEM, and code through benchmarking. [find](https://scholar.google.com/scholar?q=Basma+El+Amel+Boussaha%2C+Leen+AlQadi%2C+Mugariya+Farooq%2C+Shaikha+Alsuwaidi%2C+Giulia+Campesan%2C+Ahmed+Alzubaidi%2C+Mohammed+Alyafeai%2C+and+Hakim+Hacid%2E+2025%2E+3LM%3A+Bridging+Arabic%2C+STEM%2C+and+code+through+benchm)
17. Stanford CRFM. 2023. HELM arabic: Holistic evaluation of language models for arabic. [https://crfm.stanford.edu/helm/arabic/](https://crfm.stanford.edu/helm/arabic/). [↗](https://crfm.stanford.edu/helm/arabic/)
18. Mouath Abu Daoud, Chaimae Abouzahir, Leen Kharouf, Walid Al-Eisawi, Nizar Habash, and Farah E Shamout. 2025. MedArabiQ: Benchmarking large language models on Arabic medical tasks. arXiv preprint arXiv:2505.03427. [↗](https://arxiv.org/abs/2505.03427) [our card](https://md.research.elamur.ai/free/en/papers/daoud-2025-medarabiq-benchmarking-large-language-models/)
19. Mouath Abu Daoud, Leen Kharouf, Omar El Hajj, Dana El Samad, Mariam Al-Omari, Jihad Mallat, Khaled Saleh, Nizar Habash, and Farah E. Shamout. 2026. Medarabench: Large-scale arabic medical question answering dataset and benchmark. In The Fourteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Mouath+Abu+Daoud%2C+Leen+Kharouf%2C+Omar+El+Hajj%2C+Dana+El+Samad%2C+Mariam+Al%2DOmari%2C+Jihad+Mallat%2C+Khaled+Saleh%2C+Nizar+Habash%2C+and+Farah+E%2E+Shamout%2E+2026%2E+Medarabench%3A+Large%2Dscale+arabic+medical+question+ans) [our card](https://md.research.elamur.ai/free/en/papers/abu-2026-daoud-medarabench-large-scale-arabic-medical/)
20. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. [↗](https://arxiv.org/abs/2412.19437)
21. Ali El Filali, Hamza Alobeidli, Clémentine Fourrier, Basma El Amel Boussaha, Ruxandra Cojocaru, Nathan Habib, and Hakim Hacid. 2024. Open Arabic LLM leaderboard. [https://huggingface.co/s](https://huggingface.co/s) paces/OALL/Open-Arabic-LLM-Leaderboard. [↗](https://huggingface.co/s)
22. Ali El Filali, Manel Aloui, Tarique Husaain, Ahmed Alzubaidi, Basma El Amel Boussaha, Ruxandra Cojocaru, Clémentine Fourrier, Nathan Habib, and Hakim Hacid. 2025. The open Arabic LLM leaderboard 2. [https://huggingface.co/spaces/OALL/Ope](https://huggingface.co/spaces/OALL/Ope) n-Arabic-LLM-Leaderboard. [↗](https://huggingface.co/spaces/OALL/Ope)
23. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. [↗](https://arxiv.org/abs/2407.21783)
24. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of NAACL-HLT, pages 107–112. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Swabha+Swayamdipta%2C+Omer+Levy%2C+Roy+Schwartz%2C+Samuel+Bowman%2C+and+Noah+A+Smith%2E+2018%2E+Annotation+artifacts+in+natural+language+inference+data%2E+In+Proceedings+of+NAACL%2DHLT%2C+pages+107%E2%80%931)
25. Nathan Habib, Clémentine Fourrier, Hynek Kydlíček, Thomas Wolf, and Lewis Tunstall. 2023. Lighteval: A lightweight framework for llm evaluation. [find](https://scholar.google.com/scholar?q=Nathan+Habib%2C+Cl%C3%A9mentine+Fourrier%2C+Hynek+Kydl%C3%AD%C4%8Dek%2C+Thomas+Wolf%2C+and+Lewis+Tunstall%2E+2023%2E+Lighteval%3A+A+lightweight+framework+for+llm+evaluation%2E)
26. Faris Hijazi, Somayah AlHarbi, Abdulaziz AlHussein, Harethah Abu Shairah, Reem AlZahrani, Hebah Al-Shamlan, Omar Knio, and George Turkiyyah. 2024. ArabLegalEval: A multitask benchmark for assessing Arabic legal knowledge in large language models. arXiv preprint arXiv:2408.07983. [↗](https://arxiv.org/abs/2408.07983)
27. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, and 1 others. 2021. Dynabench: Rethinking benchmarking in NLP. arXiv preprint arXiv:2104.14337. [↗](https://arxiv.org/abs/2104.14337)
28. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, and 1 others. 2024. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5622–5640. [find](https://scholar.google.com/scholar?q=Fajri+Koto%2C+Haonan+Li%2C+Sara+Shatnawi%2C+Jad+Doughman%2C+Abdelrahman+Boda+Sadallah%2C+Aisha+Alraeesi%2C+Khalid+Almubarak%2C+Zaid+Alyafeai%2C+Neha+Sengupta%2C+Shady+Shehata%2C+and+1+others%2E+2024%2E+ArabicMMLU%3A+Assessing+) [our card](https://md.research.elamur.ai/free/en/papers/koto-2024-arabicmmlu-assessing-massive-multitask-language/)
29. Malak Mashaabi, Shahad Al-Khalifa, and Hend Al-Khalifa. 2024. A survey of large language models for Arabic language and its dialects. arXiv preprint arXiv:2410.20238. [↗](https://arxiv.org/abs/2410.20238) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)
30. Bilal Mousi and 1 others. 2025. AraDiCE: Benchmarks for dialectal and cultural capabilities in LLMs. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4186–4218, Abu Dhabi, UAE. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bilal+Mousi+and+1+others%2E+2025%2E+AraDiCE%3A+Benchmarks+for+dialectal+and+cultural+capabilities+in+LLMs%2E+In+Proceedings+of+the+31st+International+Conference+on+Computational+Linguistics%2C+pages+4186%E2%80%934218%2C+)
31. Omer Nacar, Serry Taiseer Sibaee, Samar Ahmed, Safa Ben Atitallah, Adel Ammar, Yasser Alhabashi, Abdulrahman S Al-Batati, Arwa Alsehibani, Nour Qandos, Omar Elshehy, and 1 others. 2025. Towards inclusive Arabic LLMs: A culturally aligned benchmark in Arabic large language model evaluation. In Proceedings of the First Workshop on Language Models for Low-Resource Languages, pages 387–401, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Omer+Nacar%2C+Serry+Taiseer+Sibaee%2C+Samar+Ahmed%2C+Safa+Ben+Atitallah%2C+Adel+Ammar%2C+Yasser+Alhabashi%2C+Abdulrahman+S+Al%2DBatati%2C+Arwa+Alsehibani%2C+Nour+Qandos%2C+Omar+Elshehy%2C+and+1+others%2E+2025%2E+Towards+inclus)
32. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bancu, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901. [find](https://scholar.google.com/scholar?q=Yixin+Nie%2C+Adina+Williams%2C+Emily+Dinan%2C+Mohit+Bancu%2C+Jason+Weston%2C+and+Douwe+Kiela%2E+2020%2E+Adversarial+NLI%3A+A+new+benchmark+for+natural+language+understanding%2E+In+Proceedings+of+the+58th+Annual+Meeting)
33. Sebastian Ruder. 2021. Challenges and opportunities in NLP benchmarking. [https://www.ruder.io/n](https://www.ruder.io/n) lp-benchmarking/. [↗](https://www.ruder.io/n)
34. Abdelrahman Sadallah and Fajri Koto. 2025. Commonsense reasoning in Arab culture. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Abdelrahman+Sadallah+and+Fajri+Koto%2E+2025%2E+Commonsense+reasoning+in+Arab+culture%2E+In+Proceedings+of+the+63rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2E)
35. SILMA.AI. 2024. Arabic broad leaderboard (ABL): Next-generation LLM benchmark for evaluating arabic models. [https://silma.ai/arabic-llm-l](https://silma.ai/arabic-llm-l) eaderboard. [↗](https://silma.ai/arabic-llm-l)
36. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, and 1 others. 2025a. Fanar: An arabic-centric multimodal generative ai platform. arXiv preprint arXiv:2501.13944. [↗](https://arxiv.org/abs/2501.13944) [our card](https://md.research.elamur.ai/free/en/papers/abbas-2025-fanar-arabic-centric-multimodal-generative/)
37. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025b. Gemma 3 technical report. Preprint, arXiv:2503.19786. [↗](https://arxiv.org/abs/2503.19786)
38. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. [↗](https://arxiv.org/abs/2505.09388) [our card](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/)