---
title: "ALARB: An Arabic Legal Argument Reasoning Benchmark (2025)"
description: "On this data they define three tasks: verdict prediction under different inputs, completion of reasoning… 38 references, cited by 2 papers here."
authors: ["Harethah Abu Shairah", "Somayah AlHarbi", "Abdulaziz AlHussein", "Sameer Alsabea", "Omar Shaqaqi", "Hebah AlShamlan", "Omar Knio", "George Turkiyyah"]
year: 2025
venue: "Proceedings of The Third Arabic Natural Language Processing Conference"
source: "https://aclanthology.org/2025.arabicnlp-main.32/"
doi: "10.18653/v1/2025.arabicnlp-main.32"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shairah-2025-alarb-arabic-legal-argument-reasoning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shairah-2025-alarb-arabic-legal-argument-reasoning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/) / ALARB: An Arabic Legal Argument Reasoning Benchmark

# ALARB: An Arabic Legal Argument Reasoning Benchmark

**Authors:** [Harethah Abu Shairah](https://md.research.elamur.ai/free/en/authors/harethah-abu-shairah/), [Somayah AlHarbi](https://md.research.elamur.ai/free/en/authors/somayah-alharbi/), [Abdulaziz AlHussein](https://md.research.elamur.ai/free/en/authors/abdulaziz-alhussein/), Sameer Alsabea, Omar Shaqaqi, [Hebah AlShamlan](https://md.research.elamur.ai/free/en/authors/hebah-alshamlan/), [Omar Knio](https://md.research.elamur.ai/free/en/authors/omar-knio/) and [George Turkiyyah](https://md.research.elamur.ai/free/en/authors/george-turkiyyah/) · **Year:** 2025 · **Venue:** [Proceedings of The Third Arabic Natural Language Processing Conference](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2025](https://md.research.elamur.ai/free/en/venues/arabicnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.arabicnlp-main.32/) · [PDF](https://aclanthology.org/2025.arabicnlp-main.32.pdf) · [DOI 10.18653/v1/2025.arabicnlp-main.32](https://doi.org/10.18653/v1/2025.arabicnlp-main.32) · [OpenAlex](https://openalex.org/W4416037825)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/shairah-2025-alarb-arabic-legal-argument-reasoning/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most Arabic LLM benchmarks test knowledge or retrieval, often through multiple-choice questions or material translated from English. Few native resources test multistep, open-ended reasoning. ALARB fills this gap for the legal domain. The authors scraped more than 13K commercial court cases from the Saudi Ministry of Justice website, along with eight governing statutes and their implementing regulations. They used an LLM to find the cited articles, then split each case into an itemised list of facts, a step-by-step chain of court reasoning, a short verdict, and links to the relevant articles. Cases were also anonymised.

On this data they define three tasks: verdict prediction under different inputs, completion of reasoning chains, and identification of applicable articles. They benchmark open and closed Arabic-capable LLMs. Fully fine-tuning Gemma-3-12B-it on ALARB instructions lifts it to roughly GPT-4o level on verdict prediction. ALARB therefore serves both as a native Arabic reasoning benchmark and as useful instruction-tuning data for legal reasoning.

## Key points
- The corpus has 13K+ Saudi commercial court cases, each with facts, reasoning steps, verdict and cited articles from 8 statutes with implementing regulations.
- Verdict prediction comes in three variants: facts only, facts + regulations, and facts + court reasoning. Outputs are graded as correct, partial or incorrect.
- Gemma-3-12B-it was fully fine-tuned on 12,012 instruction–output pairs for 4 epochs at a learning rate of 5e-6 and tested on 1,329 cases. Fully correct verdicts rose by +21.5 points on facts only (to 37.3%), +15.7 on facts + regulations (45.3%) and +14.9 on facts + reasoning (65.9%).
- When the final reasoning steps are removed one by one, every model gets worse. The fine-tuned model overtakes GPT-4o once three or more steps are missing.
- On a 100-case sample, reasoning in English raised Gemma's fully correct rate by 9.4 points (to 39%). GPT-4o barely changed.

## Interesting ideas
- The authors extracted cited statutes and articles twice with different LLM prompts and kept the union, trading some precision for higher recall on noisy, inconsistently named references.
- Free narrative case text was rewritten into atomic fact items and an ordered list of single reasoning steps, which makes each inference unit separately checkable.
- The same case is posed with different amounts of supporting input (facts only, plus rules, plus reasoning), so the comparison shows which kind of information drives correct answers.
- The authors cut the last k steps from the gold reasoning chain and plot error against k, measuring how much a model depends on explicit intermediate steps.
- Outputs are scored with a three-way correct/partial/incorrect label instead of a binary judgement, which keeps credit for answers that are nearly right.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): When building silver annotations with LLM teachers, unioning differently prompted runs could catch missed items, with the witness-parser agreement step then filtering out false additions.
- **I2** (world): Breaking a text into atomic facts and ordered steps fits building the world of the text, where each entity state or event could carry a pointer to its source sentence.
- **I3** (answer): QA on fables or word problems could be run with and without explicit world facts or compiled knowledge, showing whether failures come from reading or from reasoning.
- **I4** (learn): A truncated-chain ablation could act as a graded negative control, showing whether Mova's explainable pipeline really uses each step rather than guessing the answer.
- **I5** (answer): School-test and fable-QA evaluation could separate partially right answers, such as the right entity in the wrong state, so the step at fault becomes visible.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Political Science and International Relations](https://md.research.elamur.ai/free/en/shelves/social-sciences/political-science-and-international-relations/) · 182 works

🧭 Topics (4): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [⚖️ Legal AI](https://md.research.elamur.ai/free/en/topics/legal-ai/) · 134 works

## Cite

Cite as: Harethah Abu Shairah, Somayah AlHarbi, Abdulaziz AlHussein, Sameer Alsabea, Omar Shaqaqi, Hebah AlShamlan, Omar Knio and George Turkiyyah. 2025. ALARB: An Arabic Legal Argument Reasoning Benchmark. Proceedings of The Third Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shairah2025alarb,
  title = {{ALARB: An Arabic Legal Argument Reasoning Benchmark}},
  author = {Harethah Abu Shairah, Somayah AlHarbi, Abdulaziz AlHussein, Sameer Alsabea, Omar Shaqaqi, Hebah AlShamlan, Omar Knio and George Turkiyyah},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-main.32},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-main.32}
}
```

- [← previous Agent-Based Uncertainty Awareness Improves Automated Radiology Report Labeling with an Open-Source Large Language Model](https://md.research.elamur.ai/free/en/papers/ben-atya-2025-agent-based-uncertainty-awareness-improves/)
- [next → Annotating Errors in English Learners' Written Language Production: Advancing Automated Written Feedback Systems](https://md.research.elamur.ai/free/en/papers/coyne-2025-annotating-errors-english-learners-written/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/)

## References

38 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Muhammad Abdul-Mageed, Shady Elbassuoni, Jad Doughman, AbdelRahim Elmadany, El Moatez Billah Nagoudi, Yorgo Zoughby, Ahmad Shaher, Iskander Gaba, Ahmed Helal, and Mohammed El-Razzaz. 2021. DiaLex: A benchmark for evaluating multi-dialectal Arabic word embeddings. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 11–20, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Muhammad+Abdul%2DMageed%2C+Shady+Elbassuoni%2C+Jad+Doughman%2C+AbdelRahim+Elmadany%2C+El+Moatez+Billah+Nagoudi%2C+Yorgo+Zoughby%2C+Ahmad+Shaher%2C+Iskander+Gaba%2C+Ahmed+Helal%2C+and+Mohammed+El%2DRazzaz%2E+2021%2E+DiaLex%3A+A+b)
2. Wafa Alghallabi, Ritesh Thawkar, Sara Ghaboura, Ketan More, Omkar Thawakar, Hisham Cholakkal, Salman Khan, and Rao Muhammad Anwer. 2025. [Fann or Flop: A multigenre, multiera benchmark for arabic poetry understanding in LLMs](https://md.research.elamur.ai/free/en/papers/ghallabi-2025-fann-or-flop-multigenre-multiera/). Preprint, arXiv:2505.18152. [↗](https://arxiv.org/abs/2505.18152)
3. Ebtesam Almazrouei, Ruxandra Cojocaru, Michele Baldo, Quentin Malartic, Hamza Alobeidli, Daniele Mazzotta, Guilherme Penedo, Giulia Campesan, Mugariya Farooq, Maitha Alhammadi, Julien Launay, and Badreddine Noune. 2023. [AlGhafa evaluation benchmark for Arabic language models](https://md.research.elamur.ai/free/en/papers/almazrouei-2023-alghafa-evaluation-benchmark-arabic-language/). In Proceedings of ArabicNLP 2023, pages 244–275, Singapore (Hybrid). Association for Computational Linguistics.
4. Norah Alshahrani, Saied Alshahrani, Esma Wali, and Jeanna Matthews. 2024. [Arabic synonym BERT-based adversarial examples for text classification](https://md.research.elamur.ai/free/en/papers/alshahrani-2024-arabic-synonym-bert-based-adversarial/). In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 137–147, St. Julian’s, Malta. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2402.03477)
5. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. [AraBERT: Transformer-based model for Arabic language understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/). In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. [↗](https://doi.org/10.48550/arxiv.2003.00104)
6. Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras. 2022. [LexGLUE: A benchmark dataset for legal language understanding in English](https://md.research.elamur.ai/free/en/papers/chalkidis-2021-lexglue-benchmark-dataset-legal-language/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4310–4330. Association for Computational Linguistics. [↗](https://doi.org/10.2139/ssrn.3936759)
7. Odysseas S. Chlapanis, Dimitrios Galanis, and Ion Androutsopoulos. 2024. LAR-ECHR: A new legal argument reasoning task and dataset for cases of the European court of human rights. In Proceedings of the Natural Legal Language Processing Workshop 2024, pages 267–279, Miami, FL, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Odysseas+S%2E+Chlapanis%2C+Dimitrios+Galanis%2C+and+Ion+Androutsopoulos%2E+2024%2E+LAR%2DECHR%3A+A+new+legal+argument+reasoning+task+and+dataset+for+cases+of+the+European+court+of+human+rights%2E+In+Proceedings+of+th)
8. Ali El Filali, Hamza Alobeidli, Clémentine Fourrier, Basma El Amel Boussaha, Ruxandra Cojocaru, Nathan Habib, and Hakim Hacid. 2024. Open arabic llm leaderboard. [https://huggingface.co/spaces/OALL/Open-Arabic-LLM-Leaderboard](https://huggingface.co/spaces/OALL/Open-Arabic-LLM-Leaderboard). Accessed 4 July 2025. [↗](https://huggingface.co/spaces/OALL/Open-Arabic-LLM-Leaderboard)
9. Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, and Mikel Artetxe. 2024. [Do multilingual language models think better in English?](https://md.research.elamur.ai/free/en/papers/etxaniz-2024-do-multilingual-language-models-think/) In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 550–564, Mexico City, Mexico. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2308.01223)
10. Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, Jidong Ge, and Vincent Ng. 2024. [LawBench: Benchmarking legal knowledge of large language models](https://md.research.elamur.ai/free/en/papers/fei-2023-lawbench-benchmarking-legal-knowledge-large/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7933–7962, Miami, Florida, USA. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2309.16289)
11. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2024. A survey on LLM-as-a-judge. Preprint, arXiv:2411.15594. [↗](https://arxiv.org/abs/2411.15594)
12. Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, and 21 others. 2023. [Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models](https://md.research.elamur.ai/free/en/papers/guha-2023-legalbench-collaboratively-built-benchmark-measuring/). Preprint, arXiv:2308.11462. [↗](https://arxiv.org/abs/2308.11462)
13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.2009.03300)
14. Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021b. [Cuad: An expert-annotated nlp dataset for legal contract review](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-cuad-expert-annotated-nlp-dataset/). Preprint, arXiv:2103.06268. [↗](https://arxiv.org/abs/2103.06268)
15. Faris Hijazi, Somayah Alharbi, Abdulaziz AlHussein, Harethah Shairah, Reem Alzahrani, Hebah Al-shamlan, George Turkiyyah, and Omar Knio. 2024. [ArabLegalEval: A multitask benchmark for assessing Arabic legal knowledge in large language models](https://md.research.elamur.ai/free/en/papers/hijazi-2024-arablegaleval-multitask-benchmark-assessing-arabic/). In Proceedings of the Second Arabic Natural Language Processing Conference, pages 225–249, Bangkok, Thailand. Association for Computational Linguistics.
16. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Song Dingjie, Zhihong Chen, Mosen Alharthi, Bang An, Juncai He, Ziche Liu, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2024. [AceGPT, localizing large language models in Arabic](https://md.research.elamur.ai/free/en/papers/huang-2024-acegpt-localizing-large-language-models/). In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8139–8163, Mexico City, Mexico. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2309.12053)
17. King Salman Global Academy for Arabic Language. 2024. BALSAM index: Benchmark of arabic language ai systems and models. [https://benchmarks.ksaa.gov.sa/b/balsam](https://benchmarks.ksaa.gov.sa/b/balsam). Accessed 4 July 2025. [↗](https://benchmarks.ksaa.gov.sa/b/balsam)
18. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. [ArabicMMLU: Assessing massive multitask language understanding in Arabic](https://md.research.elamur.ai/free/en/papers/koto-2024-arabicmmlu-assessing-massive-multitask-language/). In Findings of the Association for Computational Linguistics: ACL 2024, pages 5622–5640, Bangkok, Thailand. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2402.12840)
19. Sang Kwon, Gagan Bhatia, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Beyond English: Evaluating LLMs for Arabic grammatical error correction. In Proceedings of ArabicNLP 2023, pages 101–119, Singapore (Hybrid). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sang+Kwon%2C+Gagan+Bhatia%2C+El+Moatez+Billah+Nagoudi%2C+and+Muhammad+Abdul%2DMageed%2E+2023%2E+Beyond+English%3A+Evaluating+LLMs+for+Arabic+grammatical+error+correction%2E+In+Proceedings+of+ArabicNLP+2023%2C+pages+101)
20. Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D. Manning, and Daniel E. Ho. 2025. [Hallucination-free? assessing the reliability of leading ai legal research tools](https://md.research.elamur.ai/free/en/papers/magesh-2024-hallucination-free-assessing-reliability-leading/). Journal of Empirical Legal Studies, 22(2):216–242. Basel Mousi, Nadir Durrani, Fatema Ahmad, Md. [↗](https://doi.org/10.1111/jels.12413)
21. Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2025. [AraDiCE: Benchmarks for dialectal and cultural capabilities in LLMs](https://md.research.elamur.ai/free/en/papers/mousi-2025-aradice-benchmarks-dialectal-cultural-capabilities/). In Proceedings of the 31st International Conference on Computational Linguistics, pages 4186–4218, Abu Dhabi, UAE. Association for Computational Linguistics.
22. Shubham Kumar Nigam, Aniket Deroy, Subhankar Maity, and Arnab Bhattacharya. 2024. Rethinking legal judgement prediction in a realistic scenario in the era of large language models. In Proceedings of the Natural Legal Language Processing Workshop 2024, pages 61–80, Miami, FL, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shubham+Kumar+Nigam%2C+Aniket+Deroy%2C+Subhankar+Maity%2C+and+Arnab+Bhattacharya%2E+2024%2E+Rethinking+legal+judgement+prediction+in+a+realistic+scenario+in+the+era+of+large+language+models%2E+In+Proceedings+of+t)
23. Shubham Kumar Nigam, Tanmay Dubey, Govind Sharma, Noel Shallum, Kripabandhu Ghosh, and Arnab Bhattacharya. 2025. LegalSeg: Unlocking the structure of Indian legal judgments through rhetorical role classification. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1129–1144, Albuquerque, New Mexico. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shubham+Kumar+Nigam%2C+Tanmay+Dubey%2C+Govind+Sharma%2C+Noel+Shallum%2C+Kripabandhu+Ghosh%2C+and+Arnab+Bhattacharya%2E+2025%2E+LegalSeg%3A+Unlocking+the+structure+of+Indian+legal+judgments+through+rhetorical+role+cla)
24. Joel Niklaus, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis, and Daniel Ho. 2024. MultiLegalPile: A 689GB multilingual legal corpus. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15077–15094, Bangkok, Thailand. Association for Computational Linguistics. [↗](https://doi.org/10.24451/arbor.19714)
25. Joel Niklaus, Lucia Zheng, Arya D. McCarthy, Christopher Hahn, Brian M Rosen, Peter Henderson, Daniel E. Ho, Garrett Honke, Percy Liang, and Christopher D Manning. 2025. LawInstruct: A resource for studying language model adaptation to the legal domain. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 127–152, Albuquerque, New Mexico. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2404.02127)
26. OpenAI. 2024a. Embeddings. [https://platform.openai.com/docs/guides/embeddings](https://platform.openai.com/docs/guides/embeddings). Accessed: December 2024. [↗](https://platform.openai.com/docs/guides/embeddings)
27. OpenAI. 2024b. GPT-4o (Omni): A Multimodal AI Model. Model announcement. Supports text, vision, audio, video; faster & cheaper than GPT�4 Turbo. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2024b%2E+GPT%2D4o+%28Omni%29%3A+A+Multimodal+AI+Model%2E+Model+announcement%2E+Supports+text%2C+vision%2C+audio%2C+video%3B+faster+%26+cheaper+than+GPT%EF%BF%BD4+Turbo%2E)
28. Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, and 1000[+ others](https://md.research.elamur.ai/free/en/papers/phan-2025-humanity-s-last-exam/). 2025. Humanity’s last exam. Preprint, arXiv:2501.14249. [↗](https://arxiv.org/abs/2501.14249)
29. Zhaozhi Qian, Faroq Altam, Muhammad Alqurishi, and Riad Souissi. 2024. [Cameleval: Advancing culturally aligned arabic language models and benchmarks](https://md.research.elamur.ai/free/en/papers/qian-2024-cameleval-advancing-culturally-aligned-arabic/). Preprint, arXiv:2409.12623. [↗](https://arxiv.org/abs/2409.12623)
30. Gerard Salton, Anawat Wong, and Chung-Shu Yang. 1975. A vector space model for automatic indexing. Communications of the ACM, 18(11):613–620. [↗](https://doi.org/10.1145/361219.361220)
31. Lisa Schut, Yarin Gal, and Sebastian Farquhar. 2025. Do multilingual LLMs think in english? In ICLR 2025 Workshop on Building Trust in Language Models and Applications. [↗](https://doi.org/10.48550/arxiv.2502.15603)
32. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. [GLUE: A multi�task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1804.07461)
33. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint. ArXiv:2505.09388. [↗](https://arxiv.org/abs/2505.09388)
34. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging LLM-as-a-judge with MT-bench and Chatbot Arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc.
35. Lucia Zheng, Neel Guha, Javokhir Arifov, Sarah Zhang, Michal Skreta, Christopher D. Manning, Peter Henderson, and Daniel E. Ho. 2025. [A reasoning-focused legal retrieval benchmark](https://md.research.elamur.ai/free/en/papers/zheng-2025-reasoning-focused-legal-retrieval-benchmark/). In Proceedings of the 2025 Symposium on Computer Science and Law, CSLAW ’25, pages 169–193, New York, NY, USA. Association for Computing Machinery.
36. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2024. [AGIEval: A human�centric benchmark for evaluating foundation models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/). In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2299–2314, Mexico City, Mexico. Association for Computational Linguistics.
37. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Alham Fikri Aji, Zhengzhong Liu, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, and 3 others. 2023. [Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models](https://md.research.elamur.ai/free/en/papers/sengupta-2023-jais-jais-chat-arabic-centric/). CoRR, abs/2308.16149. [↗](https://arxiv.org/abs/2308.16149)
38. Serry Sibaee, Omer Nacar, Adel Ammar, Yasser Al-Habashi, Abdulrahman Al-Batati, and Wadii Boulila. 2025. [From guidelines to practice: A new paradigm for arabic language model evaluation](https://md.research.elamur.ai/free/en/papers/sibaee-2025-guidelines-practice-new-paradigm-arabic/). Preprint, arXiv:2506.01920. [↗](https://arxiv.org/abs/2506.01920)

*Updated 1 November 2026*