--- title: "UI-Level Evaluation of ALLaM 34B — Nacar 2025" description: "The paper examines how well ALLaM 34B, the largest publicly accessible model in SDAIA's Arabic-centric ALLaM family, performs as deployed inside HUMAIN…" authors: ["Omer Nacar"] year: 2025 venue: "arXiv:2508.17378" source: "https://arxiv.org/abs/2508.17378" doi: "10.48550/arxiv.2508.17378" arxiv: "2508.17378" licence: "CC BY-SA 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by-sa/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/nacar-2025-ui-level-evaluation-allam-34b/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nacar-2025-ui-level-evaluation-allam-34b/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/4/) / UI-Level Evaluation of ALLaM 34B: Measuring an Arabic-Centric LLM via HUMAIN Chat # UI-Level Evaluation of ALLaM 34B: Measuring an Arabic-Centric LLM via HUMAIN Chat **Authors:** [Omer Nacar](https://md.research.elamur.ai/free/en/authors/omer-nacar/) · **Year:** 2025 · **Venue:** arXiv:2508.17378 · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2508.17378) · [PDF](https://arxiv.org/pdf/2508.17378) · [DOI 10.48550/arxiv.2508.17378](https://doi.org/10.48550/arxiv.2508.17378) · [arXiv 2508.17378](https://arxiv.org/abs/2508.17378) · [OpenAlex](https://openalex.org/W7077854390) **Citations:** 0 citations (openalex-snapshot · 2026-10-14) · [2](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary The paper examines how well ALLaM 34B, the largest publicly accessible model in SDAIA's Arabic-centric ALLaM family, performs as deployed inside HUMAIN Chat. That service offers no API and no weights, so all testing had to go through the chat interface. The author built 23 prompts covering MSA, five regional dialects (Najdi, Hijazi, Egyptian, Moroccan, Levantine), code-switching and Arabizi, recall of facts, reasoning about arithmetic and time, creative writing tasks, and adversarial safety probes. Each prompt was submitted five times, giving 115 responses. Three LLM judges (GPT-5, Gemini 2.5 Pro, Claude Sonnet-4) scored every response on a 1–5 scale for accuracy, fluency, instruction following, safety, and, where relevant, dialect fidelity. Scores were averaged per judge, then per prompt, then per category, with 95% confidence intervals. Code-switching and generation scored highest, while dialects scored lowest. Levantine and Moroccan were especially weak: the model understands dialectal input but often answers in MSA or even English. The study is useful as a template for evaluating a black-box deployed system and for showing where Arabic dialect coverage falls short. ## Key points - The evaluation set has 23 prompts × 5 runs = 115 outputs, collected only through the HUMAIN Chat web UI. Decoding parameters were not controllable, and observed latency was about 1–3 seconds. - Category means were: - code-switching 4.92 and generation 4.92 - knowledge 4.77, MSA 4.74, reasoning 4.64 - safety 4.54, dialect 4.21 - prompt injection, jailbreak and data exfiltration each exactly 4.20, with zero variance - The dialect heat map shows a different picture from the category table. Najdi, Hijazi and Egyptian score about 3.8 overall, Moroccan about 3.3, and Levantine 2.73. In the weaker dialects, accuracy drops much more than fluency. - Typical failure modes in the qualitative examples: - Najdi weather questions received structured English summaries. - Hijazi questions received long MSA news bulletins. - Egyptian small talk received formulaic MSA self-introductions. - The human check of the LLM judges was limited to a partial review of a subset and is described only qualitatively. Stated limitations include the small prompt pack, dependence on the UI, and reliance on LLM judges. ## Interesting ideas - Each prompt is sent several times and scores come with confidence intervals, so run-to-run instability is visible rather than hidden in a single sample. — For Mova (learn): When an LLM teacher tier labels a sentence, query it several times and accept the label as silver data only if the answers agree, alongside the witness-parser agreement. - Scores are aggregated in a fixed order: raw score → mean over judges → mean over runs of a prompt → category mean with CI. This keeps the many runs of one prompt from outweighing other prompts. — For Mova (learn): Use the same layered averaging in evaluation reports so that sentences or items with many variants do not dominate the totals for a test category. - Fluency and accuracy are scored separately, which exposes a model that produces well-formed text with wrong content (fluent but inaccurate dialect answers). — For Mova (back): Score generated text on two separate axes, surface well-formedness and faithfulness to the source meaning graph, so a smooth but unfaithful realisation cannot pass. - A metric is applied only when it is relevant (dialect fidelity only for dialect prompts), and the overall score averages just the applicable dimensions. — For Mova (answer): Attach metrics to task types, for example numeric correctness only for maths word problems, so irrelevant dimensions do not dilute the scores. - The paper separates understanding a variety from answering in it: the model grasped dialectal input but switched register or language in its reply. — For Mova (back): Add a check that the output's register and language match those of the input or the requested style, and flag drift to formal or foreign phrasing as a distinct error type. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) ## Cited by Cited by 2 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 1 not yet. - [Can Dialects Be Steered Like Languages? Sparse Neurons and Distributed Directions in Arabic LLMs](https://md.research.elamur.ai/free/en/papers/elozeiri-2026-dialects-steered-like-languages-sparse/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{nacar2025ui, title = {{UI-Level Evaluation of ALLaM 34B: Measuring an Arabic-Centric LLM via HUMAIN Chat}}, author = {Nacar, Omer}, year = {2025}, doi = {10.48550/arxiv.2508.17378}, eprint = {2508.17378}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2508.17378} } ``` - [← previous UD-English-CHILDES: A Collected Resource of Gold and Silver Universal Dependencies Trees for Child Language Interactions](https://md.research.elamur.ai/free/en/papers/ud-2025-yang-ud-english-childes-collected-resource/) - [next → UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://md.research.elamur.ai/free/en/papers/qin-2025-ui-tars-pioneering-automated/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/4/) ## References 8 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+) 2. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [↗](https://arxiv.org/abs/2307.09288) 3. Omer Nacar, Serry Taiseer Sibaee, Samar Ahmed, Safa Ben Atitallah, Adel Ammar, Yasser Alhabashi, Abdulrahman S Al-Batati, Arwa Alsehibani, Nour Qandos, Omar Elshehy, et al. Towards inclusive arabic llms: A culturally aligned benchmark in arabic large language model evaluation. In Proceedings of the First Workshop on Language Models for Low-Resource Languages, pages 387–401, 2025. [find](https://scholar.google.com/scholar?q=Omer+Nacar%2C+Serry+Taiseer+Sibaee%2C+Samar+Ahmed%2C+Safa+Ben+Atitallah%2C+Adel+Ammar%2C+Yasser+Alhabashi%2C+Abdulrahman+S+Al%2DBatati%2C+Arwa+Alsehibani%2C+Nour+Qandos%2C+Omar+Elshehy%2C+et+al%2E+Towards+inclusive+arabic+ll) 4. M Saiful Bari, Yazeed Alnumay, Norah A Alzahrani, Nouf M Alotaibi, Hisham A Alyahya, Sultan AlRashed, Faisal A Mirza, Shaykhah Z Alsubaie, Hassan A Alahmed, Ghadah Alabduljabbar, et al. Allam: Large language models for arabic and english. arXiv preprint arXiv:2407.15390, 2024. [↗](https://arxiv.org/abs/2407.15390) [our card](https://md.research.elamur.ai/free/en/papers/bari-2024-allam-large-language-models-arabic/) 5. OpenAI. Gpt-5 is here. [https://openai.com/news/gpt-5](https://openai.com/news/gpt-5), 2025. Accessed: 2025-08-23. [↗](https://openai.com/news/gpt-5) 6. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. [↗](https://arxiv.org/abs/2312.11805) [our card](https://md.research.elamur.ai/free/en/papers/gemini-2023-team-gemini-family-highly-capable-multimodal/) 7. Anthropic. Claude sonnet 4. [https://www.anthropic.com/news/claude-sonnet-4](https://www.anthropic.com/news/claude-sonnet-4), 2025. Accessed: 2025-08-23. [↗](https://www.anthropic.com/news/claude-sonnet-4) 8. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594, 2024. [↗](https://arxiv.org/abs/2411.15594)