---
title: "The Need for Robust and Inclusive Benchmarks in Evaluating"
description: "Arabic is spoken very widely but makes up a tiny share of multilingual LLM training data, and evaluation work for it has been… 42 references."
authors: ["Lubana Al Rayes", "Ashraf Elnagar"]
year: 2025
venue: "Proceedings of the 8th International Conference on Natural Language and Speech Processing (ICNLSP-2025)"
source: "https://aclanthology.org/2025.icnlsp-1.20/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rayes-2025-need-robust-inclusive-benchmarks-evaluating/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rayes-2025-need-robust-inclusive-benchmarks-evaluating/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/19/) / The Need for Robust and Inclusive Benchmarks in Evaluating LLMs on Arabic Text

# The Need for Robust and Inclusive Benchmarks in Evaluating LLMs on Arabic Text

**Authors:** Lubana Al Rayes and [Ashraf Elnagar](https://md.research.elamur.ai/free/en/authors/ashraf-elnagar/) · **Year:** 2025 · **Venue:** [Proceedings of the 8th International Conference on Natural Language and Speech Processing (ICNLSP-2025)](https://md.research.elamur.ai/free/en/venues/icnlsp/) ([International Conference on Natural Language and Speech Processing 2025](https://md.research.elamur.ai/free/en/venues/icnlsp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.icnlsp-1.20/) · [PDF](https://aclanthology.org/2025.icnlsp-1.20.pdf)

## Summary
Arabic is spoken very widely but makes up a tiny share of multilingual LLM training data, and evaluation work for it has been scattered. This survey brings that work together. Following a PRISMA-style procedure, the authors searched Google Scholar, Elsevier and IEEE Xplore. They screened the results against explicit inclusion rules and kept 26 studies, published between 2022 and 2025, that evaluate LLMs on Arabic text. They sort these benchmarks into four families: general multi-task, knowledge and reasoning, domain-specific, and focused single-task. Each family has its own subcategories. The authors say this structure would also work for other languages.

The review finds the same weaknesses across many benchmarks:
- They mix intrinsic linguistic skills with downstream task performance.
- They test only a few well-known models, mostly GPT variants.
- They rely heavily on multiple-choice formats.
- They neglect dialects and cultural context.
- They rarely annotate data for fairness analysis.

The paper recommends separating intrinsic from extrinsic evaluation, including open and Arabic-centric models, building dialect-rich native data, and studying prompt language in a systematic way. It is a map for anyone who needs to choose or design an Arabic LLM benchmark.

## Key points
- **Data imbalance:** English often makes up over 90% of multilingual LLM training data, while Arabic is typically below 1%.
- **Screening pipeline:** 42 records were found. After removing 2 duplicates, 40 were screened. 4 were dropped at first screening and 10 after full-text review, leaving 26 studies.
- **Multi-task benchmarks:**
  - AraT5/ARGEN casts eight tasks as text-to-text, mostly in MSA.
  - GPTAraEval tests ChatGPT-3.5 and GPT-4 zero-shot on 44 tasks drawn from 60 datasets.
  - LAraBench covers 33 tasks over 61 datasets, including ASR and TTS. It evaluates GPT-4, Jais and Whisper.
- **LAraBench finding:** Even strong LLMs struggled with syntactic and sequence-tagging tasks. The authors partly blame inconsistent output formatting and the lack of dialectal data.
- **Generation and knowledge benchmarks:**
  - Dolphin focuses only on generation, with 200,000 completions from 20,000 prompts, scored by human judges and BLEU/ROUGE-L/COMET.
  - ArabicMMLU has 14,575 MCQs across 40 tasks, taken from regional school exams.
  - AlGhafa has 7,226 MCQs across 45 tasks.

## Interesting ideas
- **Keep intrinsic and extrinsic scores apart.** Measure core language skills (parsing, morphology, semantics) separately from application-level tasks, so a good end score cannot hide shallow processing.
- **Don't trust multiple-choice alone.** MCQ-only tests can inflate results because guessing pays off, and they cannot show whether a model can explain or generate.
- **Prefer native-authored data.** Machine-translated or synthetic benchmark data may carry hidden cultural and linguistic distortions.
- **Tag test items by variety.** Without labels such as dialect or region on test items, nobody can check whether performance is even across groups.
- **Score format failures separately.** Some LLM failures on tagging came from messy output formats rather than missing knowledge.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Report UD/frame/world accuracy as separate diagnostic numbers next to final answer accuracy, so a correct answer reached through a wrong tree is flagged rather than rewarded.
- **I2** (answer): Pair school-test MCQs with open or explanation-required items, and always compare against a random-choice baseline as a negative control.
- **I3** (learn): Build silver data and evaluation sets from text originally written in English rather than translated text, to avoid training the parser and readers on translationese.
- **I4** (tokens): Tag evaluation sentences by register or variety (informal, regional spelling, older prose) so tagger and lemma errors can be broken down by subgroup.
- **I5** (tree): When gparse or an LLM teacher is compared with the perceptron parser, enforce a strict output schema and count format failures apart from real attachment errors.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works

## Cite

Cite as: Lubana Al Rayes and Ashraf Elnagar. 2025. The Need for Robust and Inclusive Benchmarks in Evaluating LLMs on Arabic Text. Proceedings of the 8th International Conference on Natural Language and Speech Processing (ICNLSP-2025).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{rayes2025need,
  title = {{The Need for Robust and Inclusive Benchmarks in Evaluating LLMs on Arabic Text}},
  author = {Lubana {Al Rayes} and Ashraf Elnagar},
  booktitle = {Proceedings of the 8th International Conference on Natural Language and Speech Processing (ICNLSP-2025)},
  year = {2025},
  url = {https://aclanthology.org/2025.icnlsp-1.20/}
}
```

- [← previous The language history of Norwegian Jews – a neglected part of Nordic multilingualism](https://md.research.elamur.ai/free/en/papers/harstad-2025-language-history-norwegian-jews-neglected/)
- [next → The Predicative Infinitive Construct in Phoenician and Punic](https://md.research.elamur.ai/free/en/papers/fofliger-2025-predicative-infinitive-construct-phoenician-punic/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/19/)

## References

42 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Samir Abdaljalil, Hasan Kurban, and Erchin Serpedin. 2025. Halluverse25: Fine-grained multilingual benchmark dataset for llm hallucinations. [find](https://scholar.google.com/scholar?q=Samir+Abdaljalil%2C+Hasan+Kurban%2C+and+Erchin+Serpedin%2E+2025%2E+Halluverse25%3A+Fine%2Dgrained+multilingual+benchmark+dataset+for+llm+hallucinations%2E)
2. Reem Abdel-Salam. 2024. [rematchka at arabicnlu2024: Evaluating large language models for arabic word sense and location sense disambiguation](https://md.research.elamur.ai/free/en/papers/salam-2024-rematchka-arabicnlu2024-evaluating-large-language/). In Proceedings of the Second Arabic Natural Language Processing Conference, pages 383–392.
3. Ahmed Abdelali, Hamdy Mubarak, Shammur Absar Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, et al. 2023. [Larabench: Benchmarking arabic ai with large language models](https://md.research.elamur.ai/free/en/papers/abdelali-2024-larabench-benchmarking-arabic-ai-large/). arXiv preprint arXiv:2305.14982. [↗](https://arxiv.org/abs/2305.14982)
4. Muhammad Al-Barham, Imad Afyouni, Khalid Almubarak, Ayad Turky, Ibrahim Abaker Targio Hashem, Ali Bou Nassif, Ismail Shahin, and Ashraf Elnagar. 2025. Unlocking language boundaries: Araclip-transforming arabic language and image understanding through cross-lingual models. Engineering Applications of Artificial Intelligence, 151:110577. [find](https://scholar.google.com/scholar?q=Muhammad+Al%2DBarham%2C+Imad+Afyouni%2C+Khalid+Almubarak%2C+Ayad+Turky%2C+Ibrahim+Abaker+Targio+Hashem%2C+Ali+Bou+Nassif%2C+Ismail+Shahin%2C+and+Ashraf+Elnagar%2E+2025%2E+Unlocking+language+boundaries%3A+Araclip%2Dtransformi)
5. Mashael Al-Duwais, Hend Al-Khalifa, and Abdulmalik Al-Salman. 2024. A benchmark evaluation of multilingual large language models for arabic cross-lingual named-entity recognition. Electronics, 13(17):3574. [find](https://scholar.google.com/scholar?q=Mashael+Al%2DDuwais%2C+Hend+Al%2DKhalifa%2C+and+Abdulmalik+Al%2DSalman%2E+2024%2E+A+benchmark+evaluation+of+multilingual+large+language+models+for+arabic+cross%2Dlingual+named%2Dentity+recognition%2E+Electronics%2C+13%2817%29%3A)
6. Shahad Al-Khalifa and Hend Al-Khalifa. 2024. [The qiyas benchmark: Measuring chatgpt mathematical and language understanding in arabic](https://md.research.elamur.ai/free/en/papers/khalifa-2024-qiyas-benchmark-measuring-chatgpt-mathematical/). In Proceedings of the 7th International Conference on Natural Language and Speech Processing (ICNLSP 2024), pages 343–351.
7. Abdulmohsen Al-Thubaity, Sakhar Alkhereyf, Hanan Murayshid, Nouf Alshalawi, Maha Omirah, Raghad Alateeq, Rawabi Almutairi, Razan Alsuwailem, Manal Alhassoun, and Imaan Alkhanen. 2023. [Evaluating chatgpt and bard ai on arabic sentiment analysis](https://md.research.elamur.ai/free/en/papers/thubaity-2023-evaluating-chatgpt-bard-ai-arabic/). In Proceedings of ArabicNLP 2023, pages 335–349.
8. Asma Ali Al Wazrah, Afrah Altamimi, Hawra Aljasim, Waad Alshammari, Rawan Al-Matham, Omar Elnashar, Mohamed Amin, and Abdulrahman AlOsaimy. 2025. [Evaluation of large language models on arabic punctuation prediction](https://md.research.elamur.ai/free/en/papers/wazrah-2025-evaluation-large-language-models-arabic/). In Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script, pages 144–154.
9. Emad A Alghamdi, Reem Masoud, Deema Alnuhait, Afnan Y Alomairi, Ahmed Ashraf, and Mohamed Zaytoon. 2025. [Aratrust: An evaluation of trustworthiness for llms in arabic](https://md.research.elamur.ai/free/en/papers/alghamdi-2025-aratrust-evaluation-trustworthiness-llms-arabic/). In Proceedings of the 31st International Conference on Computational Linguistics, pages 8664–8679.
10. Ebtesam Almazrouei, Ruxandra Cojocaru, Michele Baldo, Quentin Malartic, Hamza Alobeidli, Daniele Mazzotta, Guilherme Penedo, Giulia Campesan, Mugariya Farooq, Maitha Alhammadi, et al. 2023. [Alghafa evaluation benchmark for arabic language models](https://md.research.elamur.ai/free/en/papers/almazrouei-2023-alghafa-evaluation-benchmark-arabic-language/). In Proceedings of ArabicNLP 2023, pages 244– 275.
11. Fakhraddin Alwajih, Abdellah El Mekki, Samar Mohamed Magdy, Abdelrahim A Elmadany, Omer Nacar, El Moatez Billah Nagoudi, Reem Abdel-Salam, Hanin Atwany, Youssef Nafea, Abdulfattah Mohammed Yahya, et al. 2025. [Palm: A culturally inclusive and linguistically diverse dataset for arabic llms](https://md.research.elamur.ai/free/en/papers/alwajih-2025-palm-culturally-inclusive-linguistically-diverse/). arXiv preprint arXiv:2503.00151. [↗](https://arxiv.org/abs/2503.00151)
12. Lama Ayash, Hassan Alhuzali, Ashwag Alasmari, and Sultan Aloufi. 2025. Saudiculture: A benchmark for evaluating large language models cultural competence within saudi arabia. arXiv preprint arXiv:2503.17485. [↗](https://arxiv.org/abs/2503.17485)
13. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. [On the opportunities and risks of foundation models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/). arXiv preprint arXiv:2108.07258. [↗](https://arxiv.org/abs/2108.07258)
14. Shamil Chollampatt, Minh Quang Pham, Sathish Reddy Indurthi, and Marco Turchi. 2025. Cross-lingual evaluation of multilingual text generation. In Proceedings of the 31st International Conference on Computational Linguistics, pages 7766–7777. [find](https://scholar.google.com/scholar?q=Shamil+Chollampatt%2C+Minh+Quang+Pham%2C+Sathish+Reddy+Indurthi%2C+and+Marco+Turchi%2E+2025%2E+Cross%2Dlingual+evaluation+of+multilingual+text+generation%2E+In+Proceedings+of+the+31st+International+Conference+on+Co)
15. AbdelRahim Elmadany, Muhammad Abdul-Mageed, et al. 2022. [Arat5: Text-to-text transformers for arabic language generation](https://md.research.elamur.ai/free/en/papers/nagoudi-2022-arat5-text-text-transformers-arabic/). In Proceedings of the 60th annual meeting of the association for computational linguistics (Volume 1: Long papers), pages 628–647.
16. Abdelrahim Elmadany, Ahmed El-Shangiti, Muhammad Abdul-Mageed, et al. 2023. [Dolphin: A challenging and diverse benchmark for arabic nlg](https://md.research.elamur.ai/free/en/papers/nagoudi-2023-dolphin-challenging-diverse-benchmark-arabic/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1404–1422.
17. Ashraf Elnagar, Sane Yagi, Youssef Mansour, Leena Lulu, and Shehdeh Fareh. 2023. A benchmark for evaluating arabic contextualized word embedding models. Information Processing & Management, 60(5):103452. [find](https://scholar.google.com/scholar?q=Ashraf+Elnagar%2C+Sane+Yagi%2C+Youssef+Mansour%2C+Leena+Lulu%2C+and+Shehdeh+Fareh%2E+2023%2E+A+benchmark+for+evaluating+arabic+contextualized+word+embedding+models%2E+Information+Processing+%26+Management%2C+60%285%29%3A1034)
18. Ayushman Gupta, Aryan Singhal, Thomas Law, Veek-shith Rao, Evan Duan, and Ryan Luo Li. 2025. [Can llms verify arabic claims? evaluating the arabic fact-checking abilities of multilingual llms](https://md.research.elamur.ai/free/en/papers/gupta-2025-can-llms-verify-arabic-claims/). In Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script, pages 104–113.
19. Faris Hijazi, Somayah Alharbi, Abdulaziz AlHussein, Harethah Shairah, Reem Alzahrani, Hebah Al-shamlan, George Turkiyyah, and Omar Knio. 2024. [Arablegaleval: A multitask benchmark for assessing arabic legal knowledge in large language models](https://md.research.elamur.ai/free/en/papers/hijazi-2024-arablegaleval-multitask-benchmark-assessing-arabic/). In Proceedings of The Second Arabic Natural Language Processing Conference, pages 225–249.
20. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International conference on machine learning, pages 4411–4421. PMLR. [find](https://scholar.google.com/scholar?q=Junjie+Hu%2C+Sebastian+Ruder%2C+Aditya+Siddhant%2C+Graham+Neubig%2C+Orhan+Firat%2C+and+Melvin+Johnson%2E+2020%2E+Xtreme%3A+A+massively+multilingual+multi%2Dtask+benchmark+for+evaluating+cross%2Dlingual+generalisation%2E+In)
21. Karima Kadaoui, Samar M Magdy, Abdul Waheed, Md Tawkat Islam Khondaker, Ahmed Oumar El-Shangiti, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. [Tarjamat: Evaluation of bard and chatgpt on machine translation of ten arabic varieties](https://md.research.elamur.ai/free/en/papers/kadaoui-2023-tarjamat-evaluation-bard-chatgpt-machine/). arXiv preprint arXiv:2308.03051. [↗](https://arxiv.org/abs/2308.03051)
22. Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. [Gptaraeval: A comprehensive evaluation of chatgpt on arabic nlp](https://md.research.elamur.ai/free/en/papers/khondaker-2023-gptaraeval-comprehensive-evaluation-chatgpt-arabic/). arXiv preprint arXiv:2305.14976. [↗](https://arxiv.org/abs/2305.14976)
23. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, et al. 2024. [Arabicmmlu: Assessing massive multitask language understanding in arabic](https://md.research.elamur.ai/free/en/papers/koto-2024-arabicmmlu-assessing-massive-multitask-language/). arXiv preprint arXiv:2402.12840. [↗](https://arxiv.org/abs/2402.12840)
24. Sang Yun Kwon, Gagan Bhatia, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Beyond english: Evaluating llms for arabic grammatical error correction. arXiv preprint arXiv:2312.08400. [↗](https://arxiv.org/abs/2312.08400)
25. Viet Lai, Nghia Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Nguyen. 2023. [Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning](https://md.research.elamur.ai/free/en/papers/lai-2023-chatgpt-beyond-english-comprehensive-evaluation/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13171–13189.
26. Salima Lamsiyah, Kamyar Zeinalipour, Matthias Brust, Marco Maggini, Pascal Bouvry, Christoph Schommer, et al. 2025. [Arabicsense: A benchmark for evaluating commonsense reasoning in arabic with large language models](https://md.research.elamur.ai/free/en/papers/lamsiyah-2025-arabicsense-benchmark-evaluating-commonsense-reasoning/). In Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL-4), pages 1–11.
27. Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, et al. 2024. [A systematic survey and critical review on evaluating large language models: Challenges, limitations, and recommendations](https://md.research.elamur.ai/free/en/papers/laskar-2024-systematic-survey-critical-review-evaluating/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13785–13816.
28. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, et al. 2020. [Xglue: A new benchmark dataset for cross-lingual pre-training, understanding and generation](https://md.research.elamur.ai/free/en/papers/liang-2020-xglue-new-benchmark-dataset-cross/). arXiv preprint arXiv:2004.01401. [↗](https://arxiv.org/abs/2004.01401)
29. Samar M Magdy, Sang Yun Kwon, Fakhraddin Alwajih, Safaa Abdelfadil, Shady Shehata, and Muhammad Abdul-Mageed. 2025. [Jawaher: A multidialectal dataset of arabic proverbs for llm benchmarking](https://md.research.elamur.ai/free/en/papers/magdy-2025-jawaher-multidialectal-dataset-arabic-proverbs/). arXiv preprint arXiv:2503.00231. [↗](https://arxiv.org/abs/2503.00231)
30. Malak Mashaabi, Shahad Al-Khalifa, and Hend Al-Khalifa. 2025. [A survey of large language models for arabic language and its dialects](https://md.research.elamur.ai/free/en/papers/mashaabi-2024-survey-large-language-models-arabic/). Basel Mousi, Nadir Durrani, Fatema Ahmad, Md.
31. Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2025. [AraDiCE: Benchmarks for dialectal and cultural capabilities in LLMs](https://md.research.elamur.ai/free/en/papers/mousi-2025-aradice-benchmarks-dialectal-cultural-capabilities/). In Proceedings of the 31st International Conference on Computational Linguistics, pages 4186–4218, Abu Dhabi, UAE. Association for Computational Linguistics.
32. Hamdy Mubarak, Hend Al-Khalifa, and Khaloud Suliman Alkhalefah. 2024. [Halwasa: Quantify and analyze hallucinations in large language models: Arabic as a case study](https://md.research.elamur.ai/free/en/papers/mubarak-2024-halwasa-quantify-analyze-hallucinations-large/). In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 8008–8015.
33. Ahmad Mustapha, Hadi Al-Khansa, Hadi Al-Mubasher, Aya Mourad, Ranam Hamoud, Hasan El-Husseini, Marwah Al-Sakkaf, and Mariette Awad. 2024. [Arastem: A native arabic multiple choice question benchmark for evaluating llms knowledge in stem subjects](https://md.research.elamur.ai/free/en/papers/mustapha-2024-arastem-native-arabic-multiple-choice/). arXiv preprint arXiv:2501.00559. [↗](https://arxiv.org/abs/2501.00559)
34. Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, et al. 2024. [Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages](https://md.research.elamur.ai/free/en/papers/myung-2024-blend-benchmark-llms-everyday-knowledge/). Advances in Neural Information Processing Systems, 37:78104–78146.
35. Omer Nacar, Serry Taiseer Sibaee, Samar Ahmed, Safa Ben Atitallah, Adel Ammar, Yasser Alhabashi, Abdulrahman S Al-Batati, Arwa Alsehibani, Nour Qandos, Omar Elshehy, et al. 2025. [Towards inclusive arabic llms: A culturally aligned benchmark in arabic large language model evaluation](https://md.research.elamur.ai/free/en/papers/nacar-2025-inclusive-arabic-llms-culturally-aligned/). In Proceedings of the First Workshop on Language Models for Low-Resource Languages, pages 387–401.
36. Abdulsalam obaid Alharbi, Abdullah Alsuhaibani, Abdulrahman Abdullah Alalawi, Usman Naseem, Shoaib Jameel, Salil Kanhere, and Imran Razzak. 2025. [Evaluating large language models on health-related claims across arabic dialects](https://md.research.elamur.ai/free/en/papers/alharbi-2025-evaluating-large-language-models-health/). In Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script, pages 95–103.
37. Zhaozhi Qian, Faroq Altam, Muhammad Alqurishi, and Riad Souissi. 2024. [Cameleval: Advancing culturally aligned arabic language models and benchmarks](https://md.research.elamur.ai/free/en/papers/qian-2024-cameleval-advancing-culturally-aligned-arabic/). arXiv preprint arXiv:2409.12623. [↗](https://arxiv.org/abs/2409.12623)
38. Haneh Rhel and Dmitri Roussinov. 2025. [Large language models and arabic content: a review](https://md.research.elamur.ai/free/en/papers/rhel-2025-large-language-models-arabic-content/). arXiv preprint arXiv:2505.08004. [↗](https://arxiv.org/abs/2505.08004)
39. Yuemei Xu, Ling Hu, Jiayi Zhao, Zihan Qiu, Kexin Xu, Yuqi Ye, and Hanwen Gu. 2025. A survey on multilingual large language models: Corpora, alignment, and bias. Frontiers of Computer Science, 19(11):1911362. [find](https://scholar.google.com/scholar?q=Yuemei+Xu%2C+Ling+Hu%2C+Jiayi+Zhao%2C+Zihan+Qiu%2C+Kexin+Xu%2C+Yuqi+Ye%2C+and+Hanwen+Gu%2E+2025%2E+A+survey+on+multilingual+large+language+models%3A+Corpora%2C+alignment%2C+and+bias%2E+Frontiers+of+Computer+Science%2C+19%2811%29%3A1)
40. Sane Yagi, Ashraf Elnagar, and Shehdeh Fareh. 2023. A benchmark for evaluating arabic word embedding models. Natural Language Engineering, 29(4):978– 1003. [find](https://scholar.google.com/scholar?q=Sane+Yagi%2C+Ashraf+Elnagar%2C+and+Shehdeh+Fareh%2E+2023%2E+A+benchmark+for+evaluating+arabic+word+embedding+models%2E+Natural+Language+Engineering%2C+29%284%29%3A978%E2%80%93+1003%2E)
41. Sane Yagi, Shehdeh Fareh, Ashraf Elnagar, Mariam Balajeed, Abdalla El-mneizel, and Mohammad Al-Badawi. 2024. Is arabic punctuation rule-governed? Cogent Arts & Humanities, 11(1):2303818. [find](https://scholar.google.com/scholar?q=Sane+Yagi%2C+Shehdeh+Fareh%2C+Ashraf+Elnagar%2C+Mariam+Balajeed%2C+Abdalla+El%2Dmneizel%2C+and+Mohammad+Al%2DBadawi%2E+2024%2E+Is+arabic+punctuation+rule%2Dgoverned%3F+Cogent+Arts+%26+Humanities%2C+11%281%29%3A2303818%2E)
42. Wenli Yang, Lilian Some, Michael Bain, and Byeong Kang. 2025. A comprehensive survey on integrating large language models with knowledge-based methods. Knowledge-Based Systems. [find](https://scholar.google.com/scholar?q=Wenli+Yang%2C+Lilian+Some%2C+Michael+Bain%2C+and+Byeong+Kang%2E+2025%2E+A+comprehensive+survey+on+integrating+large+language+models+with+knowledge%2Dbased+methods%2E+Knowledge%2DBased+Systems%2E)

*Updated 2 November 2026*