--- title: "N&N at QIAS 2025 — paper by Alangari and AlShenaifi (2025)" description: "The authors address QIAS 2025 Subtask 2, a shared task in which systems answer Arabic multiple-choice… 89.8% accuracy. 6 references." authors: ["Nourah Alangari", "Nouf AlShenaifi"] year: 2025 venue: "Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks" source: "https://aclanthology.org/2025.arabicnlp-sharedtasks.121/" doi: "10.18653/v1/2025.arabicnlp-sharedtasks.121" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/alangari-2025-n-n-qias-2025-chain/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/alangari-2025-n-n-qias-2025-chain/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/16/) / N&N at QIAS 2025: Chain-of-Thought Ensembles with Retrieval-Augmented framework for Classical Arabic Islamic # N&N at QIAS 2025: Chain-of-Thought Ensembles with Retrieval-Augmented framework for Classical Arabic Islamic **Authors:** [Nourah Alangari](https://md.research.elamur.ai/free/en/authors/nourah-alangari/) and [Nouf AlShenaifi](https://md.research.elamur.ai/free/en/authors/nouf-alshenaifi/) · **Year:** 2025 · **Venue:** [Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2025](https://md.research.elamur.ai/free/en/venues/arabicnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.arabicnlp-sharedtasks.121/) · [PDF](https://aclanthology.org/2025.arabicnlp-sharedtasks.121.pdf) · [DOI 10.18653/v1/2025.arabicnlp-sharedtasks.121](https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.121) · [OpenAlex](https://openalex.org/W4416037624) **Citations:** 1 citations (OpenAlex, 2026-10-14) ## Summary The authors address QIAS 2025 Subtask 2, a shared task in which systems answer Arabic multiple-choice questions drawn from classical Islamic literature. The questions span jurisprudence, the prophetic biography, Qur'anic sciences and Hadith studies, and are graded Beginner, Intermediate or Advanced. Their pipeline has three stages. First, several commercial LLMs answer each question using a standardized Arabic prompt with five chain-of-thought demonstrations taken from the validation set. Second, the three strongest models vote, and an answer is accepted only when at least two agree. Third, if no answer reaches that majority, a retrieval step searches the digitized reference books, reranks the retrieved passages and re-asks GPT-4o with the best ones as context. The combined system scores 89.8% accuracy on the hidden test set, higher than any single model. The work shows that agreement-gated ensembling, with retrieval used only when needed, is a cheap way to strengthen LLMs on specialized, culturally specific domains in languages other than English. ## Key points - Data: 1,400 four-option MCQs (700 validation, 700 test). Supplementary e-books such as Ar-Raḥīq al-Makhtūm and Al-Itqān fī Ulūm al-Qur'ān serve as the retrieval corpus. - Candidate models were GPT-4o, Gemini 2.5-Flash, Qwen-Plus and DeepSeek-V3. Decoding used temperature 0.2, top-p 0.95 and a limit of 512 output tokens. - Reported accuracies: - GPT-4o: about 0.85 / 0.85 / 0.86 across zero-, 3- and 5-shot. - Gemini 2.5: rose from 0.59 to 0.87 once examples were added. - DeepSeek-V3: reached 0.84 at 5-shot. - Qwen-Plus: reached 0.78 at 5-shot. - The R²P retrieval module has three steps: - Dense search uses Arabic-LaBSE embeddings (768-d, mean-pooled, L2-normalized) with a FAISS IndexFlatIP index over chunks of roughly 200 tokens with overlap, returning the top 10 passages. - A hybrid BM25 plus cross-encoder reranker narrows these to 3 passages. - GPT-4o is re-prompted with those 3 passages. - Gemini's weak zero-shot score came from ignoring the letter-only output instruction. Its long responses ran past the token limit and produced empty answers, and few-shot examples fixed this. ## Interesting ideas - Retrieval runs only when the model ensemble disagrees, so the extra cost of evidence lookup is spent on the uncertain cases alone - A strict "at least two of three agree" threshold doubles as an uncertainty detector, not just a way to pick the final answer - Retrieval is split into two stages: a broad dense recall of top-10 passages, then a precise hybrid lexical-plus-neural rerank down to three - Format failures can masquerade as knowledge failures: Gemini's low zero-shot score was really about output compliance, and examples fixed it ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (answer): School-test readers could fall back to searching compiled knowledge or the source text only when their internal reasoning paths give conflicting answers. - **I2** (learn): Silver data from witness parsers could use the same rule, accepting arcs only on 2-of-3 agreement and routing disagreements to an LLM teacher tier or to repair rules. - **I3** (world): Looking up WordNet/FrameNet/Wikidata facts for an entity could gather many candidates cheaply and then rerank them by lexical overlap and context fit before adding them to the text's world. - **I4** (learn): Evaluations using LLM teachers should separate malformed or truncated outputs from wrong answers, so a model is not judged weak because of formatting. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [King Saud University](https://md.research.elamur.ai/free/en/universities/king-saud-university/) — University, Riyadh, Saudi Arabia · 38 works · 527 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works 🧭 Topics (3): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 552 works ## Cite Cite as: Nourah Alangari and Nouf AlShenaifi. 2025. N&N at QIAS 2025: Chain-of-Thought Ensembles with Retrieval-Augmented framework for Classical Arabic Islamic. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{alangari2025n, title = {{N\&N at QIAS 2025: Chain-of-Thought Ensembles with Retrieval-Augmented framework for Classical Arabic Islamic}}, author = {Nourah Alangari and Nouf AlShenaifi}, booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks}, year = {2025}, doi = {10.18653/v1/2025.arabicnlp-sharedtasks.121}, url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.121} } ``` - [← previous MultiMinds at MAHED 2025: Multimodal and Multitask Approaches for Detecting Emotional, Hate, and Offensive Speech in Arabic Content](https://md.research.elamur.ai/free/en/papers/debnath-2025-multiminds-mahed-2025-multimodal-multitask/) - [next → NguyenTriet at MAHED Shared Task: Ensemble of Arabic BERT Models with Hierarchical Prediction and Soft Voting for Text-Based Hope and Hate Detection](https://md.research.elamur.ai/free/en/papers/triet-2025-nguyentriet-mahed-shared-task-ensemble/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/16/) ## References 6 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Zabir Al Nazi, Md Rajib Hossain, and Faisal Al Mamun. 2025. Evaluation of open and closed-source llms for low-resource language with zero-shot, few-shot, and chain-of-thought prompting. Natural Language Processing Journal, 10:100124. [↗](https://doi.org/10.1016/j.nlp.2024.100124) 2. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. [Retrieval-augmented generation for knowledge-intensive nlp tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). In Advances in Neural Information Processing Systems, volume 33, pages 9459– 9474. Curran Associates, Inc. 3. Hamed Meshkin, Joel Zirkle, Ghazal Arabidarrehdor, Anik Chaturbedi, Shilpa Chakravartula, John Mann, Bradlee Thrasher, and Zhihua Li. 2024. Harnessing large language models’ zero-shot and few-shot learning capabilities for regulatory research. Briefings in Bioinformatics, 25(5):bbae354. [↗](https://doi.org/10.1093/bib/bbae354) 4. Ebtehal H. Omoush and Rawan Ghnemat. 2025. Advancing arabic medical question answering systems with rag and llms integration. In Proceedings of the International Conference on New Trends in Computing Sciences (ICTCS), pages 511–516. IEEE. [find](https://scholar.google.com/scholar?q=Ebtehal+H%2E+Omoush+and+Rawan+Ghnemat%2E+2025%2E+Advancing+arabic+medical+question+answering+systems+with+rag+and+llms+integration%2E+In+Proceedings+of+the+International+Conference+on+New+Trends+in+Computing+) 5. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171. [↗](https://arxiv.org/abs/2203.11171) 6. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Proceedings of the Advances in Neural Information Processing Systems, 35:24824–24837. *Updated 1 November 2026*