---
title: "Can LLMs Verify Arabic Claims? Evaluating the Arabic"
description: "The paper asks how well general-purpose multilingual LLMs can judge Arabic claims as true or false… 18 references, cited by 1 paper here."
authors: ["Ayushman Gupta", "Aryan Singhal", "Thomas Law", "Veekshith Rao", "Evan Duan", "Ryan Luo Li"]
year: 2025
venue: "Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script"
source: "https://aclanthology.org/2025.abjadnlp-1.12/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gupta-2025-can-llms-verify-arabic-claims/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gupta-2025-can-llms-verify-arabic-claims/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/14/) / Can LLMs Verify Arabic Claims? Evaluating the Arabic Fact-Checking Abilities of Multilingual LLMs

# Can LLMs Verify Arabic Claims? Evaluating the Arabic Fact-Checking Abilities of Multilingual LLMs

**Authors:** Ayushman Gupta, Aryan Singhal, Thomas Law, Veekshith Rao, Evan Duan and Ryan Luo Li · **Year:** 2025 · **Venue:** [Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script](https://md.research.elamur.ai/free/en/venues/abjadnlp/) ([Workshop on NLP for Languages Using Arabic Script 2025](https://md.research.elamur.ai/free/en/venues/abjadnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.abjadnlp-1.12/) · [PDF](https://aclanthology.org/2025.abjadnlp-1.12.pdf)

**Citations:** 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks how well general-purpose multilingual LLMs can judge Arabic claims as true or false using only their internal knowledge, without retrieved evidence. The authors filtered the X-fact dataset down to 771 Arabic claims that carry plain true/false labels. They tested four models: Llama 3 8B Instruct, Llama 3 70B Instruct, Gemini 1.0 Pro and GPT-3.5-turbo. Each model was run under four prompting schemes:

- a zero-shot baseline;
- English Chain-of-Thought;
- Self-Consistency over three English reasoning chains;
- Cross-Lingual Prompting (CLP), where the model first works out the task in English and then solves it.

Verification is scored as binary classification, and any answer that is not a clear verdict counts as an error. Zero-shot results were weak (roughly 31–61%). Prompts that make the model reason generally helped. CLP gave the best scores for three of the four models, with Llama 3 8B reaching 86.55%. The work matters because it is an early benchmark of LLM fact-checking outside English and Chinese. One caveat applies: the label distribution is heavily skewed toward false claims.

## Key points
- The data come from the Train, Dev and In-domain Test splits of X-fact (Gupta and Srikumar, 2021). Only claims labelled 'true' or 'false' were kept, giving 771 items: 730 false and 41 true. A model that always answered "false" would score about 94.7%, a comparison the paper does not draw.
- All four models (Llama 3 8B/70B Instruct, Gemini 1.0 Pro, GPT-3.5-turbo) were sampled at temperature 0.7.
- Zero-shot accuracy was 59.01% for Llama 3 8B, 40.21% for Llama 3 70B, 30.60% for Gemini and 60.94% for GPT-3.5-turbo.
- With CLP the scores rose to:
  - 86.55% for Llama 3 8B;
  - 80.42% for Llama 3 70B, double its baseline;
  - 78.21% for GPT-3.5-turbo;
  - 49.41% for Gemini, whose best result (52.53%) came from Self-Consistency instead.
- English CoT and Self-Consistency increased the number of inconclusive answers, up to 81 for Gemini under CoT. CLP reduced them, for example from 21 to 2 for GPT-3.5-turbo.

## Interesting ideas
- The authors log non-answers ("inconclusive") as a separate count while still scoring them as wrong, which shows whether a method changes accuracy or only changes how decisive the model is.
- CLP splits the work into two stages: first restate and understand the task in a pivot language, then solve it. This keeps the comprehension step apart from the reasoning step.
- Self-Consistency takes a majority vote over several sampled reasoning chains. Here it also produced more undecided outcomes, so adding voters does not automatically make a system more decisive.
- The prompting methods form a ladder of increasing complexity (zero-shot → CoT → voting → two-stage), all run on identical items and models. This lets each added component be credited or blamed in isolation.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (answer): School-test and fable-QA readers could report "abstained" apart from "wrong", so we can see whether a change makes a reader more right or merely more willing to commit.
- **I2** (answer): A reader could first build an explicit restatement of the question as a graph query, then answer from the world of the text, making each of the two stages separately inspectable.
- **I3** (learn): When building silver data from several witness parsers, we should track how often the vote ends without a winner, because more witnesses may yield more abstentions rather than cleaner labels.
- **I4** (learn): Evaluations of teacher tiers or repair rules could use the same ladder style, adding one component at a time on a fixed test set.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works

🧭 Topics (3): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works

## Cite

Cite as: Ayushman Gupta, Aryan Singhal, Thomas Law, Veekshith Rao, Evan Duan and Ryan Luo Li. 2025. Can LLMs Verify Arabic Claims? Evaluating the Arabic Fact-Checking Abilities of Multilingual LLMs. Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{gupta2025can,
  title = {{Can LLMs Verify Arabic Claims? Evaluating the Arabic Fact-Checking Abilities of Multilingual LLMs}},
  author = {Ayushman Gupta, Aryan Singhal, Thomas Law, Veekshith Rao, Evan Duan and Ryan Luo Li},
  booktitle = {Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script},
  year = {2025},
  url = {https://aclanthology.org/2025.abjadnlp-1.12/}
}
```

- [← previous BurhanAI at IslamicEval 2025 Shared Task: Combating Hallucinations in LLMs for Islamic Content; Evaluation, Correction, and Retrieval-Based Solution](https://md.research.elamur.ai/free/en/papers/adel-2025-burhanai-islamiceval-2025-shared-task/)
- [next → Can Small-Scale Data Poisoning Exacerbate Dialect-Linked Biases in Large Language Models?](https://md.research.elamur.ai/free/en/papers/abbas-2025-small-scale-data-poisoning-exacerbate/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/14/)

## References

18 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Asaad Alghamdi, Xinyu Duan, Wei Jiang, Zhenhai Wang, Yimeng Wu, Qingrong Xia, Zhefeng Wang, Yi Zheng, Mehdi Rezagholizadeh, Baoxing Huai, Peilun Cheng, and Abbas Ghaddar. 2023. [Aramus: Pushing the limits of data and model scale for arabic natural language processing](https://md.research.elamur.ai/free/en/papers/alghamdi-2023-aramus-pushing-limits-data-model/). Preprint, arXiv:2306.06800. [↗](https://arxiv.org/abs/2306.06800)
2. Saud Althabiti, Mohammad Ammar Alsalka, and Eric Atwell. 2024. [Ta’keed: The first generative fact-checking system for arabic claims](https://md.research.elamur.ai/free/en/papers/althabiti-2024-ta-keed-first-generative-fact/). Preprint, arXiv:2401.14067. [↗](https://arxiv.org/abs/2401.14067)
3. Zaid Alyafeai, Maged S. Alshaibani, Badr AlKhamissi, Hamzah Luqman, Ebrahim Alareqi, and Ali Fadel. 2023. [Taqyim: Evaluating arabic nlp tasks using chatgpt models](https://md.research.elamur.ai/free/en/papers/alyafeai-2023-taqyim-evaluating-arabic-nlp-tasks/). Preprint, arXiv:2306.16322. [↗](https://arxiv.org/abs/2306.16322)
4. Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805. [↗](https://arxiv.org/abs/2312.11805)
5. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. [Arabert: Transformer-based model for arabic language understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/). Preprint, arXiv:2003.00104. [↗](https://arxiv.org/abs/2003.00104)
6. Han Cao, Lingwei Wei, Mengyang Chen, Wei Zhou, and Songlin Hu. 2023. Are large language models good fact checkers: A preliminary study. Preprint, arXiv:2311.17355. [↗](https://arxiv.org/abs/2311.17355)
7. Recep Firat Cekinel, Pinar Karagoz, and Cagri Coltekin. 2024. Cross-lingual learning vs. low-resource fine-tuning: A case study with fact-checking in turkish. Preprint, arXiv:2403.00411. [↗](https://arxiv.org/abs/2403.00411)
8. Tsun-Hin Cheung and Kin-Man Lam. 2023. Factllama: Optimizing instruction-following language models with external knowledge for automated fact-checking. Preprint, arXiv:2309.00240. [↗](https://arxiv.org/abs/2309.00240)
9. Eun Cheol Choi and Emilio Ferrara. 2024. Fact-gpt: Fact-checking augmentation via claim matching with llms. arXiv preprint arXiv:2402.05904. [↗](https://arxiv.org/abs/2402.05904)
10. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T. Al-Natsheh, Samhaa R. El-Beltagy, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Wassim El-Hajj, Mustafa Jarrar, and Hamdy Mubarak. 2021. A panoramic survey of natural language processing in the arab world. Preprint, arXiv:2011.12631. [↗](https://arxiv.org/abs/2011.12631)
11. Mohammad Majd Saad Al Deen, Maren Pielka, Jörn Hees, Bouthaina Soulef Abdou, and Rafet Sifa. 2023. [Improving natural language inference in arabic using transformer models and linguistically informed pre-training](https://md.research.elamur.ai/free/en/papers/deen-2023-improving-natural-language-inference-arabic/). Preprint, arXiv:2307.14666. [↗](https://arxiv.org/abs/2307.14666)
12. Imane Guellil, Houda Saâdane, Faical Azouaou, Billel Gueni, and Damien Nouvel. 2021. [Arabic natural language processing: An overview](https://md.research.elamur.ai/free/en/papers/guellil-2019-arabic-natural-language-processing-overview/). Journal of King Saud University - Computer and Information Sciences, 33(5):497–507.
13. Ashim Gupta and Vivek Srikumar. 2021a. [X-fact: A new benchmark dataset for multilingual fact checking](https://md.research.elamur.ai/free/en/papers/gupta-2021-x-fact-new-benchmark-dataset/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 675–682, Online. Association for Computational Linguistics.
14. Ashim Gupta and Vivek Srikumar. 2021b. [X-fact: A new benchmark dataset for multilingual fact checking](https://md.research.elamur.ai/free/en/papers/gupta-2021-x-fact-new-benchmark-dataset/). Preprint, arXiv:2106.09248. [↗](https://arxiv.org/abs/2106.09248)
15. Emma Hoes, Sacha Altay, and Juan Bermeo. 2023. Leveraging chatgpt for efficient fact-checking. PsyArXiv. April, 3. [↗](https://doi.org/10.31234/osf.io/qnjkf)
16. Xuming Hu, Junzhe Chen, Xiaochuan Li, Yufei Guo, Lijie Wen, Philip S. Yu, and Zhijiang Guo. 2023. Do large language models know about facts? Preprint, arXiv:2310.05177. [↗](https://arxiv.org/abs/2310.05177)
17. Nayeon Lee, Belinda Z Li, Sinong Wang, Wen-tau Yih, Hao Ma, and Madian Khabsa. 2020. Language models as fact-checkers? arXiv preprint arXiv:2006.04102. The claims in the dataset are sourced from https://misbar.com. [↗](https://arxiv.org/abs/2006.04102)
18. MetaAI. 2024. Introducing meta llama 3: The most capable openly available llm to date. [find](https://scholar.google.com/scholar?q=MetaAI%2E+2024%2E+Introducing+meta+llama+3%3A+The+most+capable+openly+available+llm+to+date%2E)

*Updated 1 November 2026*