---
title: "TCE at IslamicEval 2025 — paper by ElKoumy et al. (2025)"
description: "This IslamicEval 2025 system detects and checks sacred Islamic text inside answers… 0.898 accuracy. 24 references, cited by 1 paper here."
authors: ["Mohammed ElKoumy", "Khalid Allam", "Ahmad Tamer", "Mohamed Alqablawi"]
year: 2025
venue: "Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks"
source: "https://aclanthology.org/2025.arabicnlp-sharedtasks.71/"
doi: "10.18653/v1/2025.arabicnlp-sharedtasks.71"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/elkoumy-2025-tce-islamiceval-2025-retrieval-augmented/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/elkoumy-2025-tce-islamiceval-2025-retrieval-augmented/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/) / TCE at IslamicEval 2025: Retrieval-Augmented LLMs for Quranic and Hadith Content Identification and Verification

# TCE at IslamicEval 2025: Retrieval-Augmented LLMs for Quranic and Hadith Content Identification and Verification

**Authors:** Mohammed ElKoumy, Khalid Allam, Ahmad Tamer and Mohamed Alqablawi · **Year:** 2025 · **Venue:** [Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2025](https://md.research.elamur.ai/free/en/venues/arabicnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.arabicnlp-sharedtasks.71/) · [PDF](https://aclanthology.org/2025.arabicnlp-sharedtasks.71.pdf) · [DOI 10.18653/v1/2025.arabicnlp-sharedtasks.71](https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.71) · [OpenAlex](https://openalex.org/W4416037755)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/elkoumy-2025-tce-islamiceval-2025-retrieval-augmented/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This IslamicEval 2025 system detects and checks sacred Islamic text inside answers produced by LLMs. Generated text can contain Quranic verses or Hadith that are slightly wrong or entirely invented, and a single altered word or diacritic can change their meaning. The authors addressed two subtasks: locating Quran and Hadith spans (1A) and judging whether each span is authentic (1B). The dataset was small, so they used few-shot prompting of large multilingual models instead of fine-tuning. For 1A, the model is shown typical citation cues and trigger words. It extracts spans from input chunks, and fuzzy alignment then maps those spans back onto the original text. For 1B, the system first retrieves candidate passages from authoritative collections using lexical methods. An LLM then compares the claim with each candidate in turn and stops at the first confirmed match. On the hidden test set, the system reached 0.861 macro-F1 on 1A and 0.898 accuracy on 1B. The results show that cheap lexical retrieval plus a prompted verifier can check text against a reference corpus with useful accuracy.

## Key points
- **Subtask 1A setup:** Qwen3-235B-A22B-Instruct (MoE) was the best model. It scored 0.860 accuracy and 0.765 macro-F1 on validation. Without sentence-aware chunking at 800 characters, it fell to 0.795 accuracy and 0.719 macro-F1. Extracted spans must align with the source text at a fuzzy score of at least 90%, or they are discarded.
- **Quran retrieval (1B):** Candidate verses are found with py_quran. Each query word votes for the verses that contain it. The ranking also uses a proximity score, which rewards words appearing in the right order, and a coverage score. Adjacent hits from the same surah are merged into one continuous segment.
- **Hadith retrieval (1B):** Hadith candidates are ranked with character-level TF-IDF using n-grams of up to 7 characters.
- **Verification results (1B):** GPT-4o with diacritics achieved 0.90 accuracy and 0.92 F1, ahead of all Gemma-4B and Gemma-12B variants. Removing diacritics cost about 2–3 points, for example Gemma-12B dropped from 0.737 to 0.709. Recall stayed above 0.95 in every setting.
- **Error analysis:** Most errors were false positives, where plausible generated text fooled the verifier. Quranic content was harder to verify than Hadith.

## Interesting ideas
- After an LLM proposes spans, the authors re-locate each one in the source with fuzzy matching and drop any that fall below a high threshold, which filters out hallucinated or reworded output.
- Verification runs over ranked candidates one by one and stops as soon as one candidate confirms the claim, which saves calls in the same way a person stops after finding proof.
- The strictness of matching depends on the source type: exact word-for-word identity for Quran, while Hadith tolerates small variations in wording.
- Neighbouring retrieved units from the same source are merged into one contiguous passage before the verifier sees them, so it judges a coherent context rather than fragments.
- Each preprocessing step is tested by switching it off, namely chunking and diacritics, and the drop measures what the step contributes.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Anchor every span or label from the LLM teacher tiers to actual tokens of the input. Reject anything that fails to align before it enters the silver data.
- **I2** (answer): QA readers on fables and tales could check candidate supporting sentences from the world model in ranked order. They would stop at the first sentence that licenses the answer and report it as the explanation.
- **I3** (back): The text → graph → text compressor could be scored with a per-category policy. Names, numbers and quoted speech would require exact reproduction, while ordinary narration could be accepted as a paraphrase.
- **I4** (world): When gathering evidence for an event or entity state, adjacent supporting sentences could be fused into one span before reasoning about event order.
- **I5** (learn): Treat removal of individual steps, such as morphology features or a conversion step, as standard negative controls in evaluation runs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Rensselaer Polytechnic Institute](https://md.research.elamur.ai/free/en/universities/rensselaer-polytechnic-institute/) — University, Troy, United States · 21 works · 4,933 citations
- [Tanta University](https://md.research.elamur.ai/free/en/universities/tanta-university/) — University, Tanta, Egypt · 7 works · 278 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 473 works

## Cite

Cite as: Mohammed ElKoumy, Khalid Allam, Ahmad Tamer and Mohamed Alqablawi. 2025. TCE at IslamicEval 2025: Retrieval-Augmented LLMs for Quranic and Hadith Content Identification and Verification. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{elkoumy2025tce,
  title = {{TCE at IslamicEval 2025: Retrieval-Augmented LLMs for Quranic and Hadith Content Identification and Verification}},
  author = {Mohammed ElKoumy, Khalid Allam, Ahmad Tamer and Mohamed Alqablawi},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.71},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.71}
}
```

- [← previous Targeted Syntactic Evaluation for Grammatical Error Correction](https://md.research.elamur.ai/free/en/papers/koyama-2025-targeted-syntactic-evaluation-grammatical-error/)
- [next → TEDxTN: A Three-way Speech Translation Corpus for Code-Switched Tunisian Arabic - English](https://md.research.elamur.ai/free/en/papers/bougares-2025-tedxtn-three-way-speech-translation/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/)

## References

24 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Saud Althabiti, Mohammad Ammar Alsalka, and Eric Atwell. 2024. [Ta’keed: The first generative fact-checking system for arabic claims](https://md.research.elamur.ai/free/en/papers/althabiti-2024-ta-keed-first-generative-fact/). Preprint, arXiv:2401.14067. [↗](https://arxiv.org/abs/2401.14067)
2. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. [AraBERT: Transformer-based model for Arabic language understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/). In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. [↗](https://doi.org/10.48550/arxiv.2003.00104)
3. Muhammad Huzaifa Bashir, Aqil M. Azmi, Haq Nawaz, Wajdi Zaghouani, Mona Diab, Ala Al-Fuqaha, and Junaid Qadir. 2023. Arabic natural language processing for qur’anic research: a systematic review. Artificial Intelligence Review, 56(7):6801–6854. [↗](https://doi.org/10.1007/s10462-022-10313-2)
4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Preprint, arXiv:2005.14165. [↗](https://arxiv.org/abs/2005.14165)
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
6. Ali Farghaly and Khaled Shaalan. 2009. Arabic natural language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing, 8(4). [↗](https://doi.org/10.1145/1644879.1644881)
7. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). Preprint, arXiv:2407.21783. [↗](https://arxiv.org/abs/2407.21783)
8. Nuno M. Guerreiro, Duarte Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. 2023. [Hallucinations in large multilingual translation models](https://md.research.elamur.ai/free/en/papers/guerreiro-2023-hallucinations-large-multilingual-translation-models/). Preprint, arXiv:2303.16104. [↗](https://arxiv.org/abs/2303.16104)
9. Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022. [A survey on automated fact-checking](https://md.research.elamur.ai/free/en/papers/guo-2022-survey-automated-fact-checking/). Transactions of the Association for Computational Linguistics, 10:178–206. [↗](https://doi.org/10.1162/tacl_a_00454)
10. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. [A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/). ACM Transactions on Information Systems, 43(2):1–55. [↗](https://doi.org/10.1145/3703155)
11. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. [Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/). ACM Comput. Surv., 55(9). [↗](https://doi.org/10.1145/3560815)
12. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2022. [find](https://scholar.google.com/scholar?q=Rana+Malhas%2C+Watheq+Mansour%2C+and+Tamer+Elsayed%2E+2022%2E)
13. Qur’an QA 2022: [Overview of the first shared task on question answering over the holy qur’an](https://md.research.elamur.ai/free/en/papers/malhas-2022-qur-qa-2022-overview-first/). In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 79–87, Marseille, France. European Language Resources Association.
14. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2023. [find](https://scholar.google.com/scholar?q=Rana+Malhas%2C+Watheq+Mansour%2C+and+Tamer+Elsayed%2E+2023%2E)
15. Qur’an QA 2023 [shared task: Overview of passage retrieval and reading comprehension tasks over the holy qur’an](https://md.research.elamur.ai/free/en/papers/malhas-2023-qur-qa-2023-shared-task/). In Proceedings of ArabicNLP 2023, pages 690–701, Singapore (Hybrid). Association for Computational Linguistics.
16. Hamdy Mubarak, Rana Malhas, Watheq Mansour, Abubakr Mohamed, Mahmoud Fawzi, Majd Hawasly, Tamer Elsayed, Kareem Darwish, and Walid Magdy. 2025. [IslamicEval 2025: The First Shared Task of Capturing LLMs Hallucination in Islamic Content](https://md.research.elamur.ai/free/en/papers/mubarak-2025-islamiceval-2025-first-shared-task/). In Proceedings of ArabicNLP 2025, TBD. Association for Computational Linguistics. To appear.
17. OpenAI team. 2024. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). Preprint, arXiv:2303.08774. [↗](https://arxiv.org/abs/2303.08774)
18. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). arXiv preprint arXiv:2203.02155. [↗](https://arxiv.org/abs/2203.02155)
19. Marie C. Platenius, Markus von Detten, Steffen Becker, Wilhelm Schäfer, and Gregor Engels. 2013. A survey of fuzzy service matching approaches in the context of on-the-fly computing. In Proceedings of the 16th International ACM Sigsoft Symposium on Component-Based Software Engineering, CBSE ’13, page 143–152, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Marie+C%2E+Platenius%2C+Markus+von+Detten%2C+Steffen+Becker%2C+Wilhelm+Sch%C3%A4fer%2C+and+Gregor+Engels%2E+2013%2E+A+survey+of+fuzzy+service+matching+approaches+in+the+context+of+on%2Dthe%2Dfly+computing%2E+In+Proceedings+of)
20. Gerard Salton and Christopher Buckley. 1988. Term-weighting approaches in automatic text retrieval. Information Processing & Management, 24(5):513– 523. [↗](https://doi.org/10.1016/0306-4573(88)90021-0)
21. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, and 89 others. 2024. Gemma: Open models based on gemini research and technology. Preprint, arXiv:2403.08295. [↗](https://arxiv.org/abs/2403.08295)
22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA. Curran Associates Inc.
23. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). Preprint, arXiv:2505.09388. [↗](https://arxiv.org/abs/2505.09388)
24. Metric Waleed A. Yousef, Taha M. Madbouly, Omar M. Ibrahime, Ali H. El-Kassas, Ali O. Hassan, and Abdallah R. Albohy. 2018. Pyquran: The python package for quranic analysis. [https://hci-lab.github.io/PyQuran-Private](https://hci-lab.github.io/PyQuran-Private). Ayah Hadith NoAnnotation Value Unique Questions Annotations per Question Ayahs per Question Hadiths per Question 50 4.20 ± 4.30 2.36 ± 3.26 1.52 ± 2.47 Label Distribution 118 76 16 *Table 4: Subtask 1A dataset statistics: annotation density and class distribution for span extraction task.* [↗](https://hci-lab.github.io/PyQuran-Private)

*Updated 1 November 2026*