---
title: "Al-Jawaab at Qur'an QA 2023 Shared Task: Exploring"
description: "The paper describes a system entered in the Qur'an QA 2023 Shared Task, which has two parts… 23 references, cited by 2 papers here."
authors: ["Abdulrezzak Zekiye", "Fadi Amroush"]
year: 2023
venue: "Proceedings of ArabicNLP 2023"
source: "https://aclanthology.org/2023.arabicnlp-1.82/"
doi: "10.18653/v1/2023.arabicnlp-1.82"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zekiye-2023-al-jawaab-qur-qa-2023/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zekiye-2023-al-jawaab-qur-qa-2023/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/) / Al-Jawaab at Qur'an QA 2023 Shared Task: Exploring Embeddings and GPT Models for Passage Retrieval and Reading Comprehension

# Al-Jawaab at Qur'an QA 2023 Shared Task: Exploring Embeddings and GPT Models for Passage Retrieval and Reading Comprehension

**Authors:** Abdulrezzak Zekiye and Fadi Amroush · **Year:** 2023 · **Venue:** [Proceedings of ArabicNLP 2023](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2023](https://md.research.elamur.ai/free/en/venues/arabicnlp/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.arabicnlp-1.82/) · [PDF](https://aclanthology.org/2023.arabicnlp-1.82.pdf) · [DOI 10.18653/v1/2023.arabicnlp-1.82](https://doi.org/10.18653/v1/2023.arabicnlp-1.82) · [OpenAlex](https://openalex.org/W4389518324)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/zekiye-2023-al-jawaab-qur-qa-2023/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper describes a system entered in the Qur'an QA 2023 Shared Task, which has two parts: ranking Qur'anic passages for a Modern Standard Arabic question (Task A) and extracting every answer span from a given passage (Task B). Both tasks include questions with no answer in the text. For Task A, the authors embedded questions and passages with OpenAI's "text-embedding-ada-002" and ranked passages by cosine similarity. A similarity threshold tuned on the dev set decides when to return "no answer." This approach scored below the BM25 baseline. For Task B, they gave GPT-4 a handcrafted Arabic prompt at temperature zero, cleaned the raw output, and kept only answers that could be located verbatim in the passage. This clearly beat a naive whole-passage baseline. The work shows that prompting a strong LLM, combined with strict grounding of its output, can be competitive for span extraction. It also shows that off-the-shelf dense embeddings may lose to lexical retrieval on specialised Arabic text.

## Key points
- Task A data: the Qur'anic passage collection (QPC) has 1,266 thematic passages drawn from the 114 chapters. AyaTEC supplies 199 questions, and QRels lists 1,132 relevant passage-ids. Task B uses QRCD v1.2, which has 1,399 question–passage–answer triplets, about 15% of them unanswerable.
- Task A: the 1536-dimension ada-002 embeddings with a 0.85 threshold reached MAP 0.109 on dev and 0.064 on test. BM25 scored 0.170 and 0.090 on the same sets. On test, MRR was 0.161 against BM25's 0.226.
- Translating questions and passages into English with Google Translate and rerunning retrieval performed noticeably worse than working on the Arabic directly.
- Task B with GPT-4 reached pAP 0.470 on dev and 0.539 on test, compared with 0.255 and 0.327 for the NWPR baseline. Extra post-processing raised the test score to 0.546, roughly a 67% gain. GPT-3.5 did so poorly on dev that it was dropped, falling even below the naive baseline.
- The authors observed prompt-injection-like behaviour, where a question's wording made GPT-4 ignore the instructions. They also name API cost and lack of explainability as limitations.

## Interesting ideas
- Every answer the LLM generates is mapped back to start and end offsets in the passage, and any answer that cannot be found verbatim is thrown away as out-of-passage. This turns a free-form generator into a grounded span extractor.
- "No answer" is produced in two ways: either the model abstains explicitly, or every proposed answer fails the grounding check. This gives a cheap, principled abstention rule.
- The authors report honestly that the dense embedding retriever lost to BM25, and they compare against a trivial whole-passage baseline. Weak, simple baselines thus act as sanity floors for the method.
- Output from a nondeterministic generator arrives in shifting formats, such as quoted text or bulleted lists, so it is normalised before matching. Only after that is it checked against the source.
- Instructions placed in the prompt could be overridden by the content of the question itself. This argues for keeping task instructions and user-supplied text clearly separated.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (answer): readers answering fable or test questions could keep only answers anchored to a concrete span or node in the text, which makes each answer traceable.
- **I2** (learn): LLM teacher labels could be accepted only when they align with the source text, with "nothing survives alignment" recorded as a distinct abstain signal instead of being forced into silver data.
- **I3** (learn): every evaluation could include such a negative control, for example returning the whole sentence or a lexical-overlap pick, so that gains are measured against a dumb floor.
- **I4** (learn): teacher-tier outputs could go through a fixed normalisation step before agreement checks with witness parsers, so that formatting noise does not count as disagreement.
- **I5** (learn): when prompting LLM teachers on raw sentences, the data could be delimited strictly and outputs that drift off-task could be flagged, so that injected text does not contaminate silver labels.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (3): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 441 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 473 works

## Cite

Cite as: Abdulrezzak Zekiye and Fadi Amroush. 2023. Al-Jawaab at Qur'an QA 2023 Shared Task: Exploring Embeddings and GPT Models for Passage Retrieval and Reading Comprehension. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zekiye2023al,
  title = {{Al-Jawaab at Qur'an QA 2023 Shared Task: Exploring Embeddings and GPT Models for Passage Retrieval and Reading Comprehension}},
  author = {Abdulrezzak Zekiye and Fadi Amroush},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.82},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.82}
}
```

- [← previous Acquiring Linguistic Knowledge from Multimodal Input](https://md.research.elamur.ai/free/en/papers/amariucai-2023-acquiring-linguistic-knowledge-multimodal-input/)
- [next → ANLP-RG at NADI 2023 shared task: Machine Translation of Arabic Dialects: A Comparative Study of Transformer Models](https://md.research.elamur.ai/free/en/papers/derouich-2023-anlp-rg-nadi-2023-shared/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/)

## References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Giambattista Amati. 2009. BM25, pages 257–260. Springer US, Boston, MA. [find](https://scholar.google.com/scholar?q=Giambattista+Amati%2E+2009%2E+BM25%2C+pages+257%E2%80%93260%2E+Springer+US%2C+Boston%2C+MA%2E)
2. Rana Malhas and Tamer Elsayed. 2022. Arabic machine reading comprehension on the holy qur’an using cl-arabert. Information Processing Management, 59(6):103068. [↗](https://doi.org/10.1016/j.ipm.2022.103068)
3. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2022. [find](https://scholar.google.com/scholar?q=Rana+Malhas%2C+Watheq+Mansour%2C+and+Tamer+Elsayed%2E+2022%2E)
4. Qur’an QA 2022: [Overview of the First Shared Task on Question Answering over the Holy Qur’an](https://md.research.elamur.ai/free/en/papers/malhas-2022-qur-qa-2022-overview-first/). In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT5) at the 13th Language Resources and Evaluation Conference (LREC 2022), pages 79–87.
5. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2023. [find](https://scholar.google.com/scholar?q=Rana+Malhas%2C+Watheq+Mansour%2C+and+Tamer+Elsayed%2E+2023%2E)
6. Qur’an QA 2023 [Shared Task: Overview of Passage Retrieval and Reading Comprehension Tasks over the Holy Qur’an](https://md.research.elamur.ai/free/en/papers/malhas-2023-qur-qa-2023-shared-task/). In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore.
7. Rana R Malhas. 2023. ARABIC QUESTION ANSWER-ING ON THE HOLY QUR’AN. Ph.D. thesis. [find](https://scholar.google.com/scholar?q=Rana+R+Malhas%2E+2023%2E+ARABIC+QUESTION+ANSWER%2DING+ON+THE+HOLY+QUR%E2%80%99AN%2E+Ph%2ED%2E+thesis%2E)
8. OpenAI. 2023a. Embeddings - openai api. September 12, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023a%2E+Embeddings+%2D+openai+api%2E+September+12%2C+2023%2E)
9. OpenAI. 2023b. New and improved embedding model. September 12, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023b%2E+New+and+improved+embedding+model%2E+September+12%2C+2023%2E)
10. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. [↗](https://doi.org/10.3115/v1/d14-1162)
11. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
12. Faisal Rahutomo, Teruaki Kitasuka, and Masayoshi Aritsugi. 2012. Semantic cosine similarity. In The 7th international student conference on advanced science and technology ICAST, volume 4, page 1. [find](https://scholar.google.com/scholar?q=Faisal+Rahutomo%2C+Teruaki+Kitasuka%2C+and+Masayoshi+Aritsugi%2E+2012%2E+Semantic+cosine+similarity%2E+In+The+7th+international+student+conference+on+advanced+science+and+technology+ICAST%2C+volume+4%2C+page+1%2E)
13. Kenneth Ward Church. 2017. Word2vec. Natural Language Engineering, 23(1):155–162. [find](https://scholar.google.com/scholar?q=Kenneth+Ward+Church%2E+2017%2E+Word2vec%2E+Natural+Language+Engineering%2C+23%281%29%3A155%E2%80%93162%2E)
14. Mihai Surdeanu, Massimiliano Ciaramita, and Hugo Zaragoza. 2011. [Learning to rank answers to non-factoid questions from web collections](https://md.research.elamur.ai/free/en/papers/surdeanu-2011-learning-rank-answers-non-factoid/). Computational linguistics, 37(2):351–383. [↗](https://doi.org/10.1162/coli_a_00051)
15. Maximilian Schreiner. 2023. Gpt-4 architecture, datasets, costs and more leaked. September 12, 2023. [find](https://scholar.google.com/scholar?q=Maximilian+Schreiner%2E+2023%2E+Gpt%2D4+architecture%2C+datasets%2C+costs+and+more+leaked%2E+September+12%2C+2023%2E)
16. Alexis Conneau and Douwe Kiela. 2018. [Senteval: An evaluation toolkit for universal sentence representations](https://md.research.elamur.ai/free/en/papers/conneau-2018-senteval-evaluation-toolkit-universal-sentence/). arXiv preprint arXiv:1803.05449. [↗](https://arxiv.org/abs/1803.05449)
17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805)
18. Kazuaki Kishida. 2005. Property of average precision and its generalization: An examination of evaluation indicator for information retrieval experiments. National Institute of Informatics Tokyo, Japan. [find](https://scholar.google.com/scholar?q=Kazuaki+Kishida%2E+2005%2E+Property+of+average+precision+and+its+generalization%3A+An+examination+of+evaluation+indicator+for+information+retrieval+experiments%2E+National+Institute+of+Informatics+Tokyo%2C+Japa)
19. Rana Malhas and Tamer Elsayed. 2020. AyaTEC: Building a Reusable Verse-based Test Collection for Arabic Question Answering on the Holy Qur’an. ACM Transactions on Asian and Low-Resource Language Information Processing, 19(6):1–21. [find](https://scholar.google.com/scholar?q=Rana+Malhas+and+Tamer+Elsayed%2E+2020%2E+AyaTEC%3A+Building+a+Reusable+Verse%2Dbased+Test+Collection+for+Arabic+Question+Answering+on+the+Holy+Qur%E2%80%99an%2E+ACM+Transactions+on+Asian+and+Low%2DResource+Language+Infor)
20. Marwan N. Swar. 2007. Mushaf Al-Tafseel Al-Mawdoo’ee. Dar Al-Fajr Al-Islami, Damascus. [find](https://scholar.google.com/scholar?q=Marwan+N%2E+Swar%2E+2007%2E+Mushaf+Al%2DTafseel+Al%2DMawdoo%E2%80%99ee%2E+Dar+Al%2DFajr+Al%2DIslami%2C+Damascus%2E)
21. GPT-4 Research Team. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). Technical report.
22. Ellen M Voorhees. 2001. Evaluation by highly relevant documents. In Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, pages 74–82. [find](https://scholar.google.com/scholar?q=Ellen+M+Voorhees%2E+2001%2E+Evaluation+by+highly+relevant+documents%2E+In+Proceedings+of+the+24th+annual+international+ACM+SIGIR+conference+on+Research+and+development+in+information+retrieval%2C+pages+74%E2%80%9382%2E)
23. Abdul Razak Zakieh and Adil Alpkocak. 2021. Classification of medical transcriptions with explanations. Technical report, EasyChair. [find](https://scholar.google.com/scholar?q=Abdul+Razak+Zakieh+and+Adil+Alpkocak%2E+2021%2E+Classification+of+medical+transcriptions+with+explanations%2E+Technical+report%2C+EasyChair%2E)

*Updated 1 November 2026*