---
title: "Transformer Tafsir at QIAS 2025 Shared Task: Hybrid"
description: "The paper describes a system for the QIAS 2025 shared task, which tests how well LLMs answer… 18 references, cited by 1 paper here."
authors: ["Muhammad Abu Ahmad", "Mohamad Ballout", "Raia Abu Ahmad", "Elia Bruni"]
year: 2025
venue: "Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks"
source: "https://aclanthology.org/2025.arabicnlp-sharedtasks.124/"
doi: "10.18653/v1/2025.arabicnlp-sharedtasks.124"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ahmad-2025-transformer-tafsir-qias-2025-shared/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ahmad-2025-transformer-tafsir-qias-2025-shared/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/13/) / Transformer Tafsir at QIAS 2025 Shared Task: Hybrid Retrieval-Augmented Generation for Islamic Knowledge Question Answering

# Transformer Tafsir at QIAS 2025 Shared Task: Hybrid Retrieval-Augmented Generation for Islamic Knowledge Question Answering

**Authors:** Muhammad Abu Ahmad, [Mohamad Ballout](https://md.research.elamur.ai/free/en/authors/mohamad-ballout/), [Raia Abu Ahmad](https://md.research.elamur.ai/free/en/authors/raia-abu-ahmad/) and [Elia Bruni](https://md.research.elamur.ai/free/en/authors/elia-bruni/) · **Year:** 2025 · **Venue:** [Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2025](https://md.research.elamur.ai/free/en/venues/arabicnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.arabicnlp-sharedtasks.124/) · [PDF](https://aclanthology.org/2025.arabicnlp-sharedtasks.124.pdf) · [DOI 10.18653/v1/2025.arabicnlp-sharedtasks.124](https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.124) · [OpenAlex](https://openalex.org/W4416037631)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper describes a system for the QIAS 2025 shared task, which tests how well LLMs answer multiple-choice questions on Islamic knowledge in Arabic. Subtask 1 covers inheritance law ('Ilm al-Mawārīth) and Subtask 2 covers general Islamic topics. A central difficulty is the gap between Modern Standard Arabic questions and Classical Arabic source texts.

The authors built a hybrid RAG pipeline without any fine-tuning. Knowledge bases came from the organizers' materials: IslamWeb fatwas for Subtask 1 and a corpus of classical books for Subtask 2. Retrieval runs as a funnel. BM25 works over heavily preprocessed text, a dense retriever (Arabic-Triplet-Matryoshka-V2) works over lightly preprocessed text, and a multilingual miniLMv2 cross-encoder reranks the results. The top passages go into a few-shot prompt with an expert-scholar persona, which is sent to Fanar or Mistral.

Retrieval lifted accuracy in every configuration. The gains were large on general knowledge, up to 25 points, but small on inheritance questions, which mostly require fractional arithmetic. The work shows both the value and the limits of plain retrieval for reasoning-heavy domains.

## Key points
- **Data:**
  - Subtask 1 has 9,450 train, 1,500 validation and 1,000 test MCQs, plus 32,000 IslamWeb fatwas.
  - Subtask 2 has 800 validation and 1,000 test MCQs, drawn from 25 classical books.
  - Each fatwa is one retrieval unit. Books are split into overlapping paragraph chunks.
- **Retrieval funnel:**
  - BM25 keeps the top 1000 candidates.
  - Arabic-Triplet-Matryoshka-V2 narrows these to 200 by cosine similarity.
  - The cross-encoder mmarco-mMiniLMv2-L12-H384-v1 picks the final 5 for the prompt.
- **Preprocessing:** CAMeL Tools handles normalization, dediacritization and tokenization, and NLTK stopword lists are used. Stopword removal and tokenization apply only to the BM25 view.
- **Test accuracy (baseline → +RAG):**

  | Model | Subtask 1 | Subtask 2 |
  |---|---|---|
  | Fanar | 44% → 45% | 55% → 80% |
  | Mistral (mistral-saba-24b) | 35% → 39% | 69% → 79% |

- **By difficulty, best Fanar system (Table 3):**
  - Subtask 1 falls from 53.40% on beginner to 36.00% on advanced questions.
  - Subtask 2 falls from 81.86% to 73.33%.
  - The body text reports slightly different figures for these levels.

## Interesting ideas
- The authors keep two normalizations of each chunk: an aggressive one for lexical matching and a light, meaning-preserving one for neural models. Both are linked to the original text, so any stage can recover the source span.
- Candidates are narrowed in stages, from a wide cheap lexical filter to semantic similarity to an expensive pairwise scorer. This keeps cost low while precision rises at each step.
- The retrieval unit is chosen per source: a whole fatwa stays intact for coherence, while long books are split into overlapping paragraphs so facts are not lost at boundaries.
- The error analysis separates the questions retrieval helps (factual recall) from those it cannot fix (fractional, multi-step inheritance arithmetic). The authors conclude that a symbolic reasoning module is needed.
- Every model is scored both with and without retrieval under one fixed configuration. The domain-specialized model gained less relatively than the general model, which hints at overlap with its pretraining data.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): the en crate could emit a heavy-normalized and a light-normalized view of each token, both with offsets into the raw text, so lemmas and matching never lose traceability to the original span.
- **I2** (answer): fable and tale QA readers could locate supporting sentences the same way, first by lemma overlap, then by frame similarity, then by a full tree-to-question alignment over only a handful of candidates.
- **I3** (world): when compiling knowledge from Wikidata or longer texts, the unit for building a text world could be chosen by document structure, with overlap so entities and events spanning a boundary are not dropped.
- **I4** (answer): maths word problems should go to an explicit symbolic solver over the world state, not to evidence lookup, and the question type could decide the routing.
- **I5** (learn): each new component could be evaluated against an ablated control on the same items, with gains read in light of what the model or teacher already knows.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Osnabrück University](https://md.research.elamur.ai/free/en/universities/osnabruck-university/) — University, Osnabrück, Germany · 7 works · 250 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (3): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 487 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 552 works

## Cite

Cite as: Muhammad Abu Ahmad, Mohamad Ballout, Raia Abu Ahmad and Elia Bruni. 2025. Transformer Tafsir at QIAS 2025 Shared Task: Hybrid Retrieval-Augmented Generation for Islamic Knowledge Question Answering. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ahmad2025transformer,
  title = {{Transformer Tafsir at QIAS 2025 Shared Task: Hybrid Retrieval-Augmented Generation for Islamic Knowledge Question Answering}},
  author = {Muhammad Abu Ahmad, Mohamad Ballout, Raia Abu Ahmad and Elia Bruni},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.124},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.124}
}
```

- [← previous Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties](https://md.research.elamur.ai/free/en/papers/lee-2025-trans-env-framework-evaluating-linguistic/)
- [next → VLM@school — Evaluation of AI image understanding on German middle school knowledge](https://md.research.elamur.ai/free/en/papers/peinl-2025-vlm-school-evaluation-ai-image/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/13/)

## References

18 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abdelrahman Abdallah, Mahmoud Kasem, Mahmoud Abdalla, Mohamed Mahmoud, Mohamed Elkasaby, Yasser Elbendary, and Adam Jatowt. 2024. [Arabicaqa: A comprehensive dataset for arabic question answering](https://md.research.elamur.ai/free/en/papers/abdallah-2024-arabicaqa-comprehensive-dataset-arabic-question/). In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2049–2059.
2. Raghad Al-Rasheed, Abdullah Al Muaddi, Hawra Aljasim, Rawan Al-Matham, Muneera Alhoshan, Asma Al Wazrah, and Abdulrahman AlOsaimy. 2025. [Evaluating RAG pipelines for Arabic lexical information retrieval: A comparative study of embedding and generation models](https://md.research.elamur.ai/free/en/papers/rasheed-2025-evaluating-rag-pipelines-arabic-lexical/). In Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script, pages 155–164, Abu Dhabi, UAE. Association for Computational Linguistics.
3. Jumana Alsubhi, Mohammad D Alahmadi, Ahmed Alhusayni, Ibrahim Aldailami, Israa Hamdine, Ahmad Shabana, Yazeed Iskandar, and Suhayb Khayyat. 2025. [Optimizing rag pipelines for arabic: A systematic analysis of core components](https://md.research.elamur.ai/free/en/papers/alsubhi-2025-optimizing-rag-pipelines-arabic-systematic/). arXiv preprint arXiv:2506.06339. [↗](https://arxiv.org/abs/2506.06339)
4. Muhammad Arslan, Hussam Ghanem, Saba Munawar, and Christophe Cruz. 2024. A survey on rag with llms. Procedia computer science, 246:3781–3790. [find](https://scholar.google.com/scholar?q=Muhammad+Arslan%2C+Hussam+Ghanem%2C+Saba+Munawar%2C+and+Christophe+Cruz%2E+2024%2E+A+survey+on+rag+with+llms%2E+Procedia+computer+science%2C+246%3A3781%E2%80%933790%2E)
5. Steven Bird and Edward Loper. 2004. [NLTK: The natural language toolkit](https://md.research.elamur.ai/free/en/papers/bird-2006-nltk-natural-language-toolkit/). In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214–217, Barcelona, Spain. Association for Computational Linguistics.
6. Abdessalam Bouchekif, Samer Rashwani, Mohammed Ghaly, Mutaz Al-Khatib, Emad Mohamed, Wajdi Zaghouani, Heba Sbahi, Shahd Gaben, and Aiman Erbad. 2025a. [Qias 2025: Overview of the shared task on islamic inheritance reasoning and knowledge assessment](https://md.research.elamur.ai/free/en/papers/bouchekif-2025-qias-2025-overview-shared-task/). In Proceedings of The Third Arabic Natural Language Processing Conference, ArabicNLP 2025, Suzhou, China, November 5–9. Association for Computational Linguistics.
7. Abdessalam Bouchekif, Samer Rashwani, Heba Sbahi, Shahd Gaben, Mutaz Al-Khatib, and Mohammed Ghaly. 2025b. [Assessing large language models on islamic legal reasoning: Evidence from inheritance law evaluation](https://md.research.elamur.ai/free/en/papers/bouchekif-2025-assessing-large-language-models-islamic/). In Proceedings of The Third Arabic Natural Language Processing Conference (Arabic-NLP 2025), Suzhou, China. Association for Computational Linguistics.
8. Hao Cheng, Hao Fang, Xiaodong Liu, and Jianfeng Gao. 2023. Task-aware specialization for efficient and robust dense retrieval for open-domain question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1864–1875, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hao+Cheng%2C+Hao+Fang%2C+Xiaodong+Liu%2C+and+Jianfeng+Gao%2E+2023%2E+Task%2Daware+specialization+for+efficient+and+robust+dense+retrieval+for+open%2Ddomain+question+answering%2E+In+Proceedings+of+the+61st+Annual+Meet)
9. Samhaa R El-Beltagy and Mohamed A Abdallah. 2024. [Exploring retrieval augmented generation in arabic](https://md.research.elamur.ai/free/en/papers/el-beltagy-2024-exploring-retrieval-augmented-generation-arabic/). Procedia Computer Science, 244:296–307.
10. Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński, Genta Indra Winata, Saba Sturua, Saiteja Utpala, Mathieu Ciancone, Marion Schaeffer, Gabriel Sequeira, Diganta Misra, Shreeya Dhakal, Jonathan Rystrøm, Roman Solomatin, and 67 others. 2025. [Mmteb: Massive multilingual text embedding benchmark](https://md.research.elamur.ai/free/en/papers/enevoldsen-2025-mmteb-massive-multilingual-text-embedding/). arXiv preprint arXiv:2502.13595. [↗](https://arxiv.org/abs/2502.13595)
11. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. [Retrieval-augmented generation for large language models: A survey](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/). arXiv preprint arXiv:2312.10997, 2(1). [↗](https://arxiv.org/abs/2312.10997)
12. Nizar Y Habash. 2010. Introduction to Arabic natural language processing. Morgan & Claypool Publishers. [↗](https://doi.org/10.1007/978-3-031-02139-8)
13. Chip Huyen. 2024. AI Engineering: Building Applications with Foundation Models. O’Reilly Media, Incorporated. [find](https://scholar.google.com/scholar?q=Chip+Huyen%2E+2024%2E+AI+Engineering%3A+Building+Applications+with+Foundation+Models%2E+O%E2%80%99Reilly+Media%2C+Incorporated%2E)
14. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. [Dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). In EMNLP (1), pages 6769–6781.
15. Zixuan Ke, Fangkai Jiao, Yifei Ming, Xuan-Phi Nguyen, Austin Xu, Do Xuan Long, Minzhi Li, Chengwei Qin, Peifeng Wang, Silvio Savarese, and 1 others. 2025. A survey of frontiers in llm reasoning: Inference scaling, learning to reason, and agentic systems. arXiv preprint arXiv:2504.09037. [↗](https://arxiv.org/abs/2504.09037)
16. Zahra Khalila, Arbi Haza Nasution, Winda Monika, Aytug Onan, Yohei Murakami, Yasir Bin Ismail Radi, and Noor Mohammad Osmani. 2025. Investigating retrieval-augmented generation in quranic studies: A study of 13 open-source large language models. arXiv preprint arXiv:2503.16581. Cosine similarity; Top candidates = 200; Batch size = 8 for embedding computation Cross-Encoder Reranking: Model: Arabic BERT-based reranker; Final candidates = 5; Temperature = 0.1 for stable rankings Omer Nacar, Anis Koubaa, Serry Sibaee, Yasser Al-Habashi, Adel Ammar, and Wadii Boulila. 2025. [Gate: General arabic text embedding for enhanced semantic textual similarity with matryoshka representation learning and hybrid loss training](https://md.research.elamur.ai/free/en/papers/nacar-2025-gate-general-arabic-text-embedding/). arXiv preprint arXiv:2505.24581. A.2 [↗](https://arxiv.org/abs/2503.16581)
17. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. [CAMeL tools: An open source python toolkit for Arabic natural language processing](https://md.research.elamur.ai/free/en/papers/obeid-2020-camel-tools-open-source-python/). In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
18. Islam Oshallah, Mohamed Basem, Ali Hamdi, and Ammar Mohammed. 2025. Cross-language approach for quranic qa. arXiv preprint arXiv:2501.17449. [↗](https://arxiv.org/abs/2501.17449)

*Updated 1 November 2026*