---
title: "Evaluating Large Language Models on Health-Related Claims"
description: "The paper asks whether GPT-4 handles health claims reliably when users write in… 0.297. agreement. 18 references, cited by 4 papers here."
authors: ["Abdulsalam obaid Alharbi", "Abdullah Alsuhaibani", "Abdulrahman Abdullah Alalawi", "Usman Naseem", "Shoaib Jameel", "Salil Kanhere", "Imran Razzak"]
year: 2025
venue: "Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script"
source: "https://aclanthology.org/2025.abjadnlp-1.11/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/alharbi-2025-evaluating-large-language-models-health/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/alharbi-2025-evaluating-large-language-models-health/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/) / Evaluating Large Language Models on Health-Related Claims Across Arabic Dialects

# Evaluating Large Language Models on Health-Related Claims Across Arabic Dialects

**Authors:** Abdulsalam obaid Alharbi, Abdullah Alsuhaibani, Abdulrahman Abdullah Alalawi, [Usman Naseem](https://md.research.elamur.ai/free/en/authors/usman-naseem/), [Shoaib Jameel](https://md.research.elamur.ai/free/en/authors/shoaib-jameel/), Salil Kanhere and [Imran Razzak](https://md.research.elamur.ai/free/en/authors/imran-razzak/) · **Year:** 2025 · **Venue:** [Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script](https://md.research.elamur.ai/free/en/venues/abjadnlp/) ([Workshop on NLP for Languages Using Arabic Script 2025](https://md.research.elamur.ai/free/en/venues/abjadnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.abjadnlp-1.11/) · [PDF](https://aclanthology.org/2025.abjadnlp-1.11.pdf)

**Citations:** [4](https://md.research.elamur.ai/free/en/papers/alharbi-2025-evaluating-large-language-models-health/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether GPT-4 handles health claims reliably when users write in different Arabic dialects. The authors took 326 public health claims that experts had already fact-checked, drawn from AraFacts and ArCOV19-Rumors. Each claim is labelled true, false or mixed. Every claim was turned into five queries, ranging from a neutral question to a forceful demand for a supporting article. The queries were then rendered in Saudi, Egyptian, Lebanese and Moroccan Arabic. The model's answers were scored on three things: factual accuracy, consistency of stance across the five query levels, and a new Cultural Sensitivity Score that checks how well answers adapt per dialect.

GPT-4 usually confirmed true claims but almost never handled mixed claims correctly. Its overall accuracy stayed around 54–56% in every dialect. It also became more agreeable as the user's framing grew more insistent. The work matters because health misinformation spreads in everyday dialects, not in Modern Standard Arabic. The framework could also be carried over to other low-resource Abjad/Ajami languages such as Pashto or Hausa.

## Key points
- **Dataset size:** 326 claims × 5 presupposition levels give 1,630 queries per dialect, or 6,520 across the four dialects.
- **Query levels:** the five levels, adopted from Kaur et al. (2023), are neutral, mild presupposition, strong presupposition, writing request and writing demand.
- **Accuracy by claim type:** overall accuracy was 54.05% (Saudi) to 55.58% (Egyptian). True claims scored about 75–78%, while mixed claims scored only 10.77% (Lebanese) to 12.69% (Moroccan).
- **Consistency:** stance consistency peaked for true claims in Saudi (0.472) and was lowest for mixed claims in Lebanese (0.173). Overall consistency ranged from about 0.275 to 0.297.
- **Agreement under pressure:** agreement rose with user pressure. In Saudi it was 37.12% for neutral queries and 54.60% for writing demands; Lebanese writing requests reached 59.59%.

## Interesting ideas
- **Presupposition ladder:** the same claim is asked at five rising levels of user insistence, which exposes whether the system's stance survives pressure. This is a cheap robustness probe for any QA system.
- **Consistency as its own metric:** besides accuracy, the authors measure whether the stance stays stable across reformulations of one item. A model can be right on average yet unstable on individual items.
- **Three-way labels with a "mixed" class:** keeping a category for partly true claims revealed a near-total failure that a binary true/false split would hide.
- **Parallel variants of one item:** the content is held fixed while only the dialect changes, so any difference in output comes from surface variation alone.
- **Agree/disagree/neutral coding of responses:** scoring free-text answers on a small stance scale makes generated output measurable without exact-match references.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (answer): Test sets for fables and school tests could include escalating loaded rephrasings of each question, and readers would be scored on keeping the same answer.
- **I2** (learn): Evaluation could report per-item stance stability across paraphrase sets, acting as a control that catches lucky-but-brittle repairs or rules.
- **I3** (world): The world model could represent claims or states as partially supported, rather than forcing them to true or false, so answers can say "partly".
- **I4** (tree): Parallel sentences in different English varieties or registers could be fed to the parser and gparse to check that the UD trees and LAS stay invariant.
- **I5** (frames): The speech-act layer could tag reader or LLM-teacher outputs with a stance label (affirm, deny, hedge), so agreement is computed over frames rather than strings.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Abdulsalam obaid Alharbi, Abdullah Alsuhaibani, Abdulrahman Abdullah Alalawi, Usman Naseem, Shoaib Jameel, Salil Kanhere and Imran Razzak. 2025. Evaluating Large Language Models on Health-Related Claims Across Arabic Dialects. Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{alharbi2025evaluating,
  title = {{Evaluating Large Language Models on Health-Related Claims Across Arabic Dialects}},
  author = {Abdulsalam obaid Alharbi, Abdullah Alsuhaibani, Abdulrahman Abdullah Alalawi, Usman Naseem, Shoaib Jameel, Salil Kanhere and Imran Razzak},
  booktitle = {Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script},
  year = {2025},
  url = {https://aclanthology.org/2025.abjadnlp-1.11/}
}
```

- [← previous Enrich-on-Graph: Query-Graph Alignment for Complex Reasoning with LLM Enriching](https://md.research.elamur.ai/free/en/papers/li-2025-enrich-graph-query-graph-alignment/)
- [next → Evaluating Numeracy of Language Models as a Natural Language Inference Task](https://md.research.elamur.ai/free/en/papers/mahendra-2025-evaluating-numeracy-language-models-as/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/)

## References

18 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sina Ahmadi, Milind Agarwal, and Antonios Anastasopoulos. 2023. PALI: A language identification benchmark for Perso-Arabic scripts. In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), pages 78–90, Dubrovnik, Croatia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sina+Ahmadi%2C+Milind+Agarwal%2C+and+Antonios+Anastasopoulos%2E+2023%2E+PALI%3A+A+language+identification+benchmark+for+Perso%2DArabic+scripts%2E+In+Tenth+Workshop+on+NLP+for+Similar+Languages%2C+Varieties+and+Dialec)
2. Zien Sheikh Ali, Watheq Mansour, Tamer Elsayed, and Abdulaziz Al-Ali. 2021. Arafacts: The first large arabic dataset of naturally-occurring professionally-verified claims. Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Zien+Sheikh+Ali%2C+Watheq+Mansour%2C+Tamer+Elsayed%2C+and+Abdulaziz+Al%2DAli%2E+2021%2E+Arafacts%3A+The+first+large+arabic+dataset+of+naturally%2Doccurring+professionally%2Dverified+claims%2E+Association+for+Computationa)
3. Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024), pages 130–139. [find](https://scholar.google.com/scholar?q=Proceedings+of+the+Eleventh+Workshop+on+NLP+for+Similar+Languages%2C+Varieties%2C+and+Dialects+%28VarDial+2024%29%2C+pages+130%E2%80%93139%2E)
4. Mariam ALMutairi, Lulwah AlKulaib, Melike Aktas, Sara Alsalamah, and Chang-Tien Lu. 2024. [Synthetic arabic medical dialogues using advanced multi-agent llm techniques](https://md.research.elamur.ai/free/en/papers/almutairi-2024-synthetic-arabic-medical-dialogues-advanced/). In Proceedings of The Second Arabic Natural Language Processing Conference, pages 11–26.
5. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. 2023. [Towards expert-level medical question answering with large language models](https://md.research.elamur.ai/free/en/papers/singhal-2023-towards-expert-level-medical-question/). arXiv preprint arXiv:2305.09617. [↗](https://arxiv.org/abs/2305.09617)
6. Anastasiia Demidova, Hanin Atwany, Nour Rabih, and Sanad Sha’ban. 2024. [Arabic train at nadi 2024 shared task: Llms’ ability to translate arabic dialects into modern standard arabic](https://md.research.elamur.ai/free/en/papers/demidova-2024-arabic-train-nadi-2024-shared/). In Proceedings of The Second Arabic Natural Language Processing Conference, pages 729–734.
7. Surendrabikram Thapa, Kritesh Rauniyar, Hariram Veeramani, Aditya Shah, Imran Razzak, and Usman Naseem. 2024. Did you tell a deadly lie? evaluating large language models for health misinformation identification. In International Conference on Web Information Systems Engineering, pages 391– 405. Springer. [find](https://scholar.google.com/scholar?q=Surendrabikram+Thapa%2C+Kritesh+Rauniyar%2C+Hariram+Veeramani%2C+Aditya+Shah%2C+Imran+Razzak%2C+and+Usman+Naseem%2E+2024%2E+Did+you+tell+a+deadly+lie%3F+evaluating+large+language+models+for+health+misinformation+iden)
8. Fatima Haouari, Maram Hasanain, Reem Suwaileh, and Tamer Elsayed. 2020. [Arcov19-rumors: Arabic covid-19 twitter dataset for misinformation detection](https://md.research.elamur.ai/free/en/papers/haouari-2021-arcov19-rumors-arabic-covid-19/). arXiv preprint arXiv:2010.08768. [↗](https://arxiv.org/abs/2010.08768)
9. R Jairam, G Jyothish, and B Premjith. 2024. A few-shot multi-accented speech classification for indian languages using transformers and llm’s fine-tuning approaches. In Proceedings of the Fourth Workshop on Speech, Vision, and Language Technologies for Dravidian Languages, pages 1–9. [find](https://scholar.google.com/scholar?q=R+Jairam%2C+G+Jyothish%2C+and+B+Premjith%2E+2024%2E+A+few%2Dshot+multi%2Daccented+speech+classification+for+indian+languages+using+transformers+and+llm%E2%80%99s+fine%2Dtuning+approaches%2E+In+Proceedings+of+the+Fourth+Works)
10. Kees Versteegh. 2014. Arabic language. Edinburgh University Press. [find](https://scholar.google.com/scholar?q=Kees+Versteegh%2E+2014%2E+Arabic+language%2E+Edinburgh+University+Press%2E)
11. Yanpeng Ye, Jie Ren, Shaozhou Wang, Yuwei Wan, Imran Razzak, Tong Xie, and Wenjie Zhang. 2024. Construction of functional materials knowledge graph in multidisciplinary materials science via large language model. The Thirty-Eighth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Yanpeng+Ye%2C+Jie+Ren%2C+Shaozhou+Wang%2C+Yuwei+Wan%2C+Imran+Razzak%2C+Tong+Xie%2C+and+Wenjie+Zhang%2E+2024%2E+Construction+of+functional+materials+knowledge+graph+in+multidisciplinary+materials+science+via+large+lan)
12. Navreet Kaur, Monojit Choudhury, and Danish Pruthi. 2023. [Evaluating large language models for health-related queries with presuppositions](https://md.research.elamur.ai/free/en/papers/kaur-2024-evaluating-large-language-models-health/). arXiv preprint arXiv:2312.08800. [↗](https://arxiv.org/abs/2312.08800)
13. Abdullah Khered, Ingy Abdelhalim, Nadine Abdelhalim, Ahmed Soliman, and Riza Theresa Batista-Navarro. 2023. [Unimanc at nadi 2023 shared task: A comparison of various t5-based models for translating arabic dialectical text to modern standard arabic](https://md.research.elamur.ai/free/en/papers/khered-2023-unimanc-nadi-2023-shared-task/). In Proceedings of ArabicNLP 2023, pages 658– 664.
14. Valentin Liévin, Christoffer Egeberg Hother, Andreas Geert Motzfeldt, and Ole Winther. 2024. Can large language models reason about medical questions? Patterns, 5(3). [↗](https://doi.org/10.1016/j.patter.2024.100943)
15. Shen Lingzhi, Yunfei Long, Cai Xiaohao, Chen Guangming, Liu Kang, Imran Razzak, and Shoaib Jameel. 2025. Gamed: Knowledge adaptive multi-experts decoupling for multimodal fake news detection. [↗](https://doi.org/10.5446/70191)
16. Gauri Naik, Sharad Chandakacherla, Shweta Yadav, and Md Shad Akhtar. 2024. No perspective, no perception!! perspective-aware healthcare answer summarization. arXiv preprint arXiv:2406.08881. [↗](https://arxiv.org/abs/2406.08881)
17. Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. [Capabilities of gpt-4 on medical challenge problems](https://md.research.elamur.ai/free/en/papers/nori-2023-capabilities-gpt-4-medical-challenge/). arXiv preprint arXiv:2303.13375. [↗](https://arxiv.org/abs/2303.13375)
18. Viktória Ondrejová and Marek Šuppa. 2024. Can llms handle low-resource dialects? a case study on translation and common sense reasoning in šariš. In [find](https://scholar.google.com/scholar?q=Vikt%C3%B3ria+Ondrejov%C3%A1+and+Marek+%C5%A0uppa%2E+2024%2E+Can+llms+handle+low%2Dresource+dialects%3F+a+case+study+on+translation+and+common+sense+reasoning+in+%C5%A1ari%C5%A1%2E+In)

*Updated 1 November 2026*