--- title: "Burhan at IslamicEval — paper by Basheer et al. (2025)" description: "This paper describes the Burhan team's system for IslamicEval 2025 Subtask 2. 5 references, cited by 1 paper here." authors: ["Mohammad Basheer", "Watheq Mansour", "Abdulhamid Touma", "Ahmad Qadeib Alban"] year: 2025 venue: "Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks" source: "https://aclanthology.org/2025.arabicnlp-sharedtasks.73/" doi: "10.18653/v1/2025.arabicnlp-sharedtasks.73" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/basheer-2025-burhan-islamiceval-fact-augmented-llm/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/basheer-2025-burhan-islamiceval-fact-augmented-llm/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/14/) / Burhan at IslamicEval: Fact-Augmented and LLM-Driven Retrieval for Islamic QA # Burhan at IslamicEval: Fact-Augmented and LLM-Driven Retrieval for Islamic QA **Authors:** Mohammad Basheer, [Watheq Mansour](https://md.research.elamur.ai/free/en/authors/watheq-mansour/), Abdulhamid Touma and Ahmad Qadeib Alban · **Year:** 2025 · **Venue:** [Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2025](https://md.research.elamur.ai/free/en/venues/arabicnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.arabicnlp-sharedtasks.73/) · [PDF](https://aclanthology.org/2025.arabicnlp-sharedtasks.73.pdf) · [DOI 10.18653/v1/2025.arabicnlp-sharedtasks.73](https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.73) · [OpenAlex](https://openalex.org/W4416037779) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/basheer-2025-burhan-islamiceval-fact-augmented-llm/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary This paper describes the Burhan team's system for IslamicEval 2025 Subtask 2. The task is to return a ranked list of up to 20 Qur'anic passages or Sahih Bukhari Hadith that answer a Modern Standard Arabic question. The system should return nothing when no answer exists. A major difficulty is the gap in wording between modern questions and Classical Arabic sources. The authors enrich each passage with explicit facts that GPT-4o extracts: people, places, events, Islamic concepts and legal rulings. They also test adding Tafsir text to the passages. Retrieval uses text-embedding-3-large dense search per collection, and GPT-4o then reranks the candidates. Questions are paraphrased by an LLM, and a GPT-4o binary relevance judge detects questions with no answer. The LLM reranker and the no-answer filter bring the largest gains. On the test set, the best run reaches MAP@10 0.3351. The work shows that LLM-generated fact layers and verification steps can substitute for scarce annotated resources in a low-resource religious-text QA setting. ## Key points - Data: the QPC splits the Qur'an's 114 chapters into 1,266 topical passages, and the SBC holds 2,254 Hadith. Training uses 250 AyaTEC questions with relevance judgments for the QPC only, split 84% train and 16% dev. - Augmentation results (MAP@10, train+dev): plain QPC scores 0.2761, QPC + muyassar 0.2926 and QPC + IE 0.2944. Combining both augmentations drops the score to 0.2878. - Query rewriting: synonym insertion underperformed the baseline with every GPT-4 variant (0.2691–0.2781). Full paraphrasing improved results consistently, with GPT-4.1 best at 0.3065. - Reranking and no-answer detection: a fine-tuned cross-encoder reached 0.3156, while the GPT-4 reranker reached 0.4079. Adding NAD raised the score to 0.4682 with paraphrased queries and 0.4811 without them. - Test set: the best run combined PQ + Dense + RR + NAD over QPC + IE and SBC + IE, scoring MAP@10 0.3351 and MAP_H@5 0.3876. Skipping IE on the Hadith side lowered MAP_H@5 to 0.2936. Paraphrasing helped on test even though it slightly hurt on train+dev. ## Interesting ideas - Turning each short archaic passage into a list of explicit facts (who, where, what event, what ruling) makes implicit references searchable from modern phrasing, bridging a register gap without any parallel data. - Rewriting the whole question beat inserting word-by-word synonyms, which caused query drift; the paper shows the substitution approach injects noise. - A separate binary "does this passage actually answer the question" judge, applied after ranking, lets the system abstain when nothing qualifies. This gave one of the largest gains. - Two individually helpful enrichments hurt when stacked, so each added signal should be ablated alone and in combination rather than assumed additive. - The authors evaluated on train+dev merged for stability, yet the rankings of their two best configurations flipped on test, illustrating how fragile small evaluation sets are. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (world): Each text's world could be stored alongside its original sentences as explicit entity/state facts, so questions in different wording match facts rather than surface strings. - **I2** (answer): Question normalization should rephrase at the level of the whole frame (who did what to whom) rather than swapping individual lemmas for synonyms. - **I3** (answer): Readers could run an explicit support check between the question frame and candidate world facts, returning "no answer" when none pass, instead of always producing the top match. - **I4** (learn): Evaluation could routinely test combinations of silver-data sources or repair rules, since merging helpful ones may degrade results. - **I5** (learn): Small test suites (fables, word problems) should be pooled or bootstrapped, and configuration choices should not rest on tiny margins. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Université Paris Dauphine-PSL](https://md.research.elamur.ai/free/en/universities/universite-paris-dauphine-psl/) — University, Paris, France · 4 works · 16,983 citations - [The University of Queensland](https://md.research.elamur.ai/free/en/universities/the-university-of-queensland/) — University, Brisbane, Australia · 14 works · 4,603 citations - [Syrian Virtual University](https://md.research.elamur.ai/free/en/universities/syrian-virtual-university/) — University, Damascus, Syria · 2 works · 28 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works 🧭 Topics (4): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 441 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 473 works ## Cite Cite as: Mohammad Basheer, Watheq Mansour, Abdulhamid Touma and Ahmad Qadeib Alban. 2025. Burhan at IslamicEval: Fact-Augmented and LLM-Driven Retrieval for Islamic QA. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{basheer2025burhan, title = {{Burhan at IslamicEval: Fact-Augmented and LLM-Driven Retrieval for Islamic QA}}, author = {Mohammad Basheer, Watheq Mansour, Abdulhamid Touma and Ahmad Qadeib Alban}, booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks}, year = {2025}, doi = {10.18653/v1/2025.arabicnlp-sharedtasks.73}, url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.73} } ``` - [← previous Building UD Cairo for Old English in the Classroom](https://md.research.elamur.ai/free/en/papers/levine-2025-building-ud-cairo-old-english/) - [next → BurhanAI at IslamicEval 2025 Shared Task: Combating Hallucinations in LLMs for Islamic Content; Evaluation, Correction, and Retrieval-Based Solution](https://md.research.elamur.ai/free/en/papers/adel-2025-burhanai-islamiceval-2025-shared-task/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/14/) ## References 5 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Hessa Alawwad, Lujain Alawwad, Jamilah Alharbi, and Abdullah Alharbi. 2023. [Ahjl at qur’an qa 2023 shared task: Enhancing passage retrieval using sentence transformer and translation](https://md.research.elamur.ai/free/en/papers/alawwad-2023-ahjl-qur-qa-2023-shared/). In Proceedings of ArabicNLP 2023, pages 702–707. 2. Mohamed Basem, Islam Oshallah, Baraa Hikal, Ali Hamdi, and Ammar Mohamed. 2024. [Optimized quran passage retrieval using an expanded qa dataset and fine-tuned language models](https://md.research.elamur.ai/free/en/papers/basem-2024-optimized-quran-passage-retrieval-using/). In The International Conference of Advanced Computing and Informatics, pages 244–254. Springer. 3. Mohammed Alaa Elkomy and Amany Sarhan. 2024. [Tce at qur’an qa 2023 shared task: Low resource enhanced transformer-based ensemble approach for qur’anic qa](https://md.research.elamur.ai/free/en/papers/elkomy-2023-tce-qur-qa-2023-shared/). arXiv preprint arXiv:2401.13060. [↗](https://arxiv.org/abs/2401.13060) 4. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2022. [find](https://scholar.google.com/scholar?q=Rana+Malhas%2C+Watheq+Mansour%2C+and+Tamer+Elsayed%2E+2022%2E) 5. Qur’an QA 2022: [Overview of the first shared task on question answering over the holy qur’an](https://md.research.elamur.ai/free/en/papers/malhas-2022-qur-qa-2022-overview-first/). In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech *Updated 31 October 2026*