--- title: "A Knowledge Graph Based Diagnostic Framework for Analyzing" description: "The authors do not build a new detector or metric. 74.6% precision. 1 reference, cited by 1 paper here." authors: ["Najwa Abdullah AlGhamdi", "Sadam Al-Azani", "Kwabena Nuamah", "Alan Bundy"] year: 2026 venue: "Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script" source: "https://aclanthology.org/2026.abjadnlp-1.49/" doi: "10.18653/v1/2026.abjadnlp-1.49" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/alghamdi-2026-knowledge-graph-based-diagnostic-framework/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/alghamdi-2026-knowledge-graph-based-diagnostic-framework/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/9/) / A Knowledge Graph Based Diagnostic Framework for Analyzing Hallucinations in Arabic Machine Reading Comprehension # A Knowledge Graph Based Diagnostic Framework for Analyzing Hallucinations in Arabic Machine Reading Comprehension **Authors:** Najwa Abdullah AlGhamdi, [Sadam Al-Azani](https://md.research.elamur.ai/free/en/authors/sadam-al-azani/), Kwabena Nuamah and Alan Bundy · **Year:** 2026 · **Venue:** [Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script](https://md.research.elamur.ai/free/en/venues/abjadnlp/) ([Workshop on NLP for Languages Using Arabic Script 2026](https://md.research.elamur.ai/free/en/venues/abjadnlp/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.abjadnlp-1.49/) · [PDF](https://aclanthology.org/2026.abjadnlp-1.49.pdf) · [DOI 10.18653/v1/2026.abjadnlp-1.49](https://doi.org/10.18653/v1/2026.abjadnlp-1.49) · [OpenAlex](https://openalex.org/W7140083488) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/alghamdi-2026-knowledge-graph-based-diagnostic-framework/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary LLMs answering questions about a given passage often produce fluent statements that the passage does not actually support. Little of this has been studied for Arabic reading comprehension, and almost none of it for Qur'anic texts, where Classical Arabic, implicit meaning and interpretive sensitivity make errors costly. The authors do not build a new detector or metric. Instead they offer a diagnostic framework that works entirely in Arabic. The framework turns the passage, the question and the answer into subject–relation–object triples and compares them. A question-aware filter decides which content is relevant to what was asked. Supervision is weak: only the passage, question and gold answer are assumed, with no hallucination labels, and humans step in only for ambiguous triples. Applied to model answers on QRCD, the framework shows systematic error patterns. Questions asking for evidence, justification or abstract interpretation trigger far more problem triples than factual ones. This is a contrast that surface similarity scores hide. The work argues for transparent, structured evaluation in low-resource, high-stakes settings. ## Key points - Data: the Qur'anic Reading Comprehension Dataset (QRCD) has 992 passage–question–answer instances over 603 unique passages (about 1.64 questions per passage). Each passage is typically two to five verses, and each instance has a human-annotated gold answer. - Each triple receives one of five labels: supported, missing, hallucinated, off-focus, or gold-only (found in the gold answer but not recoverable from the passage). - The framework extracted 23,388 triples from gold and model answers. Of these, 6,233 were flagged for review because of uncertain question relevance or conflicts between passage support and gold annotations. - Measured against passage-derived reference graphs, triple extraction reached 74.6% precision and 57.7% recall. This limits how far the downstream diagnosis can be trusted. - Typical failures: - hallucinations were plausible generic religious claims absent from the passage; - missing triples were often secondary entities or implicitly expressed relations; - many gold-only triples pointed to interpretive content inside the gold answers themselves. ## Interesting ideas - Comparing answers at the level of individual triples pinpoints exactly which relation is unsupported, instead of grading the whole answer as one unit. - The "gold-only" label marks reference content that the source text itself cannot back, which exposes interpretive or flawed gold annotations rather than blaming the system. - An "off-focus" label separates true statements that ignore what the question asked from statements that are actually false. - Humans adjudicate only the flagged, uncertain items, here about a quarter of all triples, which keeps manual effort small while handling annotation gaps. - Colour-coding the answer graph's edges by label gives reviewers a fast visual overview of where errors cluster within one answer. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (answer): Readers answering fable or school-test questions could check each frame-derived claim of an answer against the world of the text and report which specific fact lacks support. - **I2** (learn): Evaluation could set aside gold items that the world model of the passage cannot derive, treating them as suspect references rather than as parser or reader failures. - **I3** (answer): The answerer could use the question's frame to filter world facts, then report "correct but irrelevant" output as its own category in test diagnostics. - **I4** (learn): Silver data could send to the LLM teacher or a human only those items where witness parsers or world checks disagree, leaving the agreed bulk automatic. - **I5** (world): A debugging view of the text world could colour entities and relations by provenance (stated, inferred, unsupported), making each reasoning step easy to inspect. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Edinburgh](https://md.research.elamur.ai/free/en/universities/university-of-edinburgh/) — University, Edinburgh, United Kingdom · 359 works · 77,352 citations - [King Fahd University of Petroleum and Minerals](https://md.research.elamur.ai/free/en/universities/king-fahd-university-of-petroleum-and-minerals/) — University, Dhahran, Saudi Arabia · 52 works · 4,666 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works 🧭 Topics (3): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 441 works · [🕸️ Knowledge graphs & ontologies](https://md.research.elamur.ai/free/en/topics/knowledge-graphs/) · 192 works ## Cite Cite as: Najwa Abdullah AlGhamdi, Sadam Al-Azani, Kwabena Nuamah and Alan Bundy. 2026. A Knowledge Graph Based Diagnostic Framework for Analyzing Hallucinations in Arabic Machine Reading Comprehension. Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{alghamdi2026knowledge, title = {{A Knowledge Graph Based Diagnostic Framework for Analyzing Hallucinations in Arabic Machine Reading Comprehension}}, author = {Najwa Abdullah AlGhamdi, Sadam Al-Azani, Kwabena Nuamah and Alan Bundy}, booktitle = {Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script}, year = {2026}, doi = {10.18653/v1/2026.abjadnlp-1.49}, url = {https://doi.org/10.18653/v1/2026.abjadnlp-1.49} } ``` - [← previous A Graph Talks, But Who's Listening? Rethinking Evaluations for Graph-Language Models](https://md.research.elamur.ai/free/en/papers/petkar-2026-graph-talks-but-who-s/) - [next → A Morphological Transducer for the Limbu Language](https://md.research.elamur.ai/free/en/papers/singh-2026-morphological-transducer-limbu-language/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/9/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work. 1. Garima Agrawal, Tharindu Kumarage, Zeyad Alghamdi, and Huan Liu. 2023. Can knowledge graphs reduce hallucinations in llms?: A survey. arXiv preprint arXiv:2311.07914. [↗](https://arxiv.org/abs/2311.07914) *Updated 31 October 2026*