---
title: "Beyond Fluency — paper by Al-Sumaidaee et al. (2026)"
description: "The authors build a small, expert-curated benchmark for the Saudi dialect (Najdi/Hijazi). 11 references."
authors: ["Ghassan Al-Sumaidaee", "Sajjad Abdoli", "Ahmed Rashad", "Maxim Legg"]
year: 2026
venue: "arXiv:2608.29990"
source: "https://arxiv.org/abs/2608.29990"
doi: "10.48550/arxiv.2608.29990"
arxiv: "2608.29990"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/al-sumaidaee-2026-fluency-rubric-based-benchmark-evaluating/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/al-sumaidaee-2026-fluency-rubric-based-benchmark-evaluating/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/5/) / Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Models

# Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Models

**Authors:** [Ghassan Al-Sumaidaee](https://md.research.elamur.ai/free/en/authors/ghassan-al-sumaidaee/), [Sajjad Abdoli](https://md.research.elamur.ai/free/en/authors/sajjad-abdoli/), [Ahmed Rashad](https://md.research.elamur.ai/free/en/authors/ahmed-rashad/) and Maxim Legg · **Year:** 2026 · **Venue:** arXiv:2608.29990 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2608.29990) · [PDF](https://arxiv.org/pdf/2608.29990) · [DOI 10.48550/arxiv.2608.29990](https://doi.org/10.48550/arxiv.2608.29990) · [arXiv 2608.29990](https://arxiv.org/abs/2608.29990) · [OpenAlex](https://openalex.org/W7204947814)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Arabic LLM benchmarks mostly measure fluency in Modern Standard Arabic (MSA). Everyday speech, however, is dialectal and carries social meaning that such tests miss. The authors build a small, expert-curated benchmark for the Saudi dialect (Najdi/Hijazi). It has 31 prompts written by native Saudi subject-matter experts, covering idioms, proverbs, pragmatic replies, evaluative words and cultural knowledge, and each prompt comes with an expert ground truth. Scoring happens in two stages. First, atomic, MECE positive criteria are derived from the ground truth alone, before any model output is seen. Second, each model is also penalised through model-specific negative criteria for errors it introduces itself. Four systems were tested: Claude Opus 5, Gemini 3.7, GPT-5.6 and Kimi K3. None scored above 55%. The most frequent failure was distorted register and flattened nuance, not invented facts. The authors argue that dialect-aware human rubrics are needed because automatic or MSA-centred metrics overlook exactly these errors. Prompts, ground truths and scored rubrics are released.

## Key points
- 31 prompts × 4 models = 124 model–prompt evaluations, yielding 466 error instances classified under a nine-category taxonomy (expanded from a six-category scheme, with Language Mixing added during annotation).
- Macro-averages fall in a tight 42.7%–53.1% band. No model passes 55%, and every model scores negatively on at least one prompt.
- Ambiguous Framing accounts for 37.3% of errors and Hallucination for only 11.2%. Omission of key elements and unrequested over-elaboration are the next most common problems.
- Each candidate prompt was pre-tested in a clean, memory-disabled session. Prompts answered perfectly were rewritten or dropped. This check was never used for scoring.
- Acknowledged limitations:
  - no inter-rater agreement figures yet, which matters most for the subjective Ambiguous Framing category;
  - single-turn, first-response protocol;
  - criterion weights set by the experts.

## Interesting ideas
- Fix the grading standard from the reference answer before any output is seen, then add per-system penalty items for errors each system introduces; this keeps the comparison fair while still diagnosing each system.
- Throw out or harden any test item that a quick trial system already solves perfectly, so annotation effort goes only to items that discriminate.
- Pair every prompt with its typical literal misreading, so a confident word-by-word gloss counts as a distinct failure rather than a partial success.
- Report a per-category error profile instead of one leaderboard score, because systems with similar totals can fail in very different ways.
- Penalise unrequested extra content as an error in itself, since added material can carry wrong attributions.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Write rubric items for each test item in advance and freeze them, then record each parser's or reader's own introduced errors separately so repairs can be targeted.
- **I2** (answer): Screen new school-test and fable-QA items with a baseline reader and keep only those that expose real gaps.
- **I3** (frames): Idiom, implicature and irony tests in prag could each carry a "literal trap" reading that acts as a negative control the pragmatics layer must reject.
- **I4** (learn): Break evaluation of witness parsers and LLM teacher tiers down by error type, so choosing which witness to trust depends on its error profile, not just its LAS.
- **I5** (back): When generating text from the meaning graph, flag every statement not supported by a graph node as a negative criterion, even if it is fluent.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 1,485 works

## Cite

Cite as: Ghassan Al-Sumaidaee, Sajjad Abdoli, Ahmed Rashad and Maxim Legg. 2026. Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Models. arXiv:2608.29990.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{alsumaidaee2026beyond,
  title = {{Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Models}},
  author = {Ghassan Al-Sumaidaee, Sajjad Abdoli, Ahmed Rashad and Maxim Legg},
  year = {2026},
  doi = {10.48550/arxiv.2608.29990},
  eprint = {2608.29990},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2608.29990}
}
```

- [← previous Beyond Cultural Knowledge: Evaluating Arabic Cultural Appropriateness of Large Language Models](https://md.research.elamur.ai/free/en/papers/altinisik-2026-cultural-knowledge-evaluating-arabic-cultural/)
- [next → Beyond Multilinguality: Typological Limitations in Multilingual Models for Meitei Language](https://md.research.elamur.ai/free/en/papers/nyalang-2026-beyond-multilinguality-typological-limitations-multilingual/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/5/)

## References

11 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Miro Hodak, David Ellison, Chris Van Buren, Xiaotong Jiang, and Ajay Dholakia. Benchmarking large language models: Opportunities and challenges. In Technology Conference on Performance Evaluation and Benchmarking, pages 77–89. Springer, 2023. [find](https://scholar.google.com/scholar?q=Miro+Hodak%2C+David+Ellison%2C+Chris+Van+Buren%2C+Xiaotong+Jiang%2C+and+Ajay+Dholakia%2E+Benchmarking+large+language+models%3A+Opportunities+and+challenges%2E+In+Technology+Conference+on+Performance+Evaluation+and+)
2. Yanzhu Guo, Guokan Shang, and Chloé Clavel. Benchmarking linguistic diversity of large language models. Transactions of the Association for Computational Linguistics, 13:1507–1526, 2025. [↗](https://doi.org/10.1162/tacl.a.47)
3. Sajjad Abdoli, Ghassan Al-Sumaidaee, Ahmad ElShiekh, Clayton W. Taylor, and Ahmed Rashad. [Benchmarking frontier LLMs on arabic cultural and sociolinguistic knowledge: A cross-evaluation framework with human SME ground truth](https://md.research.elamur.ai/free/en/papers/abdoli-2026-benchmarking-frontier-llms-arabic-cultural/). arXiv preprint arXiv:2607.00139, 2026. [↗](https://arxiv.org/abs/2607.00139)
4. Jiří Milička, Anna Marklová, and Václav Cvrček. Benchmark of stylistic variation in LLM-generated texts. arXiv preprint arXiv:2509.10179, 2025. [↗](https://arxiv.org/abs/2509.10179)
5. Amirhossein Razavi, Mina Soltangheis, Negar Arabzadeh, Sara Salamat, Morteza Zihayat, and Ebrahim Bagheri. Benchmarking prompt sensitivity in large language models. In European Conference on Information Retrieval, pages 303–313. Springer, 2025. [find](https://scholar.google.com/scholar?q=Amirhossein+Razavi%2C+Mina+Soltangheis%2C+Negar+Arabzadeh%2C+Sara+Salamat%2C+Morteza+Zihayat%2C+and+Ebrahim+Bagheri%2E+Benchmarking+prompt+sensitivity+in+large+language+models%2E+In+European+Conference+on+Informati)
6. Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. [Bamboo: A comprehensive benchmark for evaluating long text modeling capacities of large language models](https://md.research.elamur.ai/free/en/papers/dong-2024-bamboo-comprehensive-benchmark-evaluating-long/). In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 2086–2099, 2024.
7. Anni Zou, Wenhao Yu, Hongming Zhang, Kaixin Ma, Deng Cai, Zhuosheng Zhang, Hai Zhao, and Dong Yu. DocBench: A benchmark for evaluating LLM-based document reading systems. In Proceedings of the 4th International Workshop on Knowledge-Augmented Methods for Natural Language Processing, pages 359–373, 2025. [↗](https://doi.org/10.48550/arxiv.2407.10701)
8. Jiageng Wu, Bowen Gu, Ren Zhou, Kevin Xie, Doug Snyder, Yixing Jiang, Valentina Carducci, Richard Wyss, Rishi J Desai, Emily Alsentzer, et al. BRIDGE: Benchmarking large language models for understanding real-world clinical practice texts. Nature Biomedical Engineering, pages 1–16, 2026. [find](https://scholar.google.com/scholar?q=Jiageng+Wu%2C+Bowen+Gu%2C+Ren+Zhou%2C+Kevin+Xie%2C+Doug+Snyder%2C+Yixing+Jiang%2C+Valentina+Carducci%2C+Richard+Wyss%2C+Rishi+J+Desai%2C+Emily+Alsentzer%2C+et+al%2E+BRIDGE%3A+Benchmarking+large+language+models+for+understand)
9. Sajjad Abdoli, Ghassan Al-Sumaidaee, and Ahmed Rashad. Benchmarking frontier text-to-image models on image-description prompts. arXiv preprint arXiv:2608.14976, 2026. [↗](https://arxiv.org/abs/2608.14976)
10. Sajjad Abdoli, Ghassan Al-Sumaidaee, Ahmad ElShiekh, and Ahmed Rashad. Can foundation models hear what made that sound? a tiered benchmark of audio-language models and traditional classifiers for closed-set sound source identification. arXiv preprint arXiv:2608.02397, 2026. [↗](https://arxiv.org/abs/2608.02397)
11. Sajjad Abdoli, Ghassan Al-Sumaidaee, Clayton W. Taylor, Ahmad ElShiekh, and Ahmed Rashad. [Benchmarking commercial ASR systems on code-switching speech: Arabic, persian, and german](https://md.research.elamur.ai/free/en/papers/abdoli-2026-benchmarking-commercial-asr-systems-code/). arXiv preprint arXiv:2605.19069, 2026. [↗](https://arxiv.org/abs/2605.19069)

*Updated 24 October 2026*