---
title: "eRock at Qur'an QA 2022 — paper by Aftab and Malik (2022)"
description: "The paper describes the eRock team's entry to the Qur'an QA 2022 shared task at OSACT5 (LREC… 13 references, cited by 2 papers here."
authors: ["Esha Aftab", "Muhammad Kamran Malik"]
year: 2022
venue: "Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection"
source: "https://aclanthology.org/2022.osact-1.11/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/aftab-2022-erock-qur-qa-2022-contemporary/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/aftab-2022-erock-qur-qa-2022-contemporary/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/8/) / eRock at Qur'an QA 2022: Contemporary Deep Neural Networks for Qur'an based Reading Comprehension Question Answers

# eRock at Qur'an QA 2022: Contemporary Deep Neural Networks for Qur'an based Reading Comprehension Question Answers

**Authors:** Esha Aftab and [Muhammad Kamran Malik](https://md.research.elamur.ai/free/en/authors/muhammad-kamran-malik/) · **Year:** 2022 · **Venue:** [Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection](https://md.research.elamur.ai/free/en/venues/osact/) ([Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks 2022](https://md.research.elamur.ai/free/en/venues/osact/2022/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.osact-1.11/) · [PDF](https://aclanthology.org/2022.osact-1.11.pdf)

**Citations:** [2](https://md.research.elamur.ai/free/en/papers/aftab-2022-erock-qur-qa-2022-contemporary/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper describes the eRock team's entry to the Qur'an QA 2022 shared task at OSACT5 (LREC 2022). The task is closed-domain extractive reading comprehension over QRCD (Qur'anic Reading Comprehension Dataset). Given a question and a Qur'anic passage, a system returns five candidate answer spans ranked by score. Systems are scored with Exact Match (EM), F1 and partial Reciprocal Ranking (pRR).

The authors fine-tune BERT as a baseline. They compare an Arabic-specific pre-trained BERT with multilingual BERT and find the Arabic model stronger. To limit overfitting on the very small training set, they use decoupled weight decay. They also enlarge the training data with AQQAC, a labelled collection of questions and answers about the Arabic Qur'an. From AQQAC they keep only question–answer pairs whose style resembles QRCD, then rebuild a context passage for each pair from the Tanzil plain Qur'an text.

Their best configurations reach 59% pRR on the development split and 31% on the test split. The large gap illustrates how fragile data-hungry neural QA becomes in low-resource, specialised domains.

## Key points
- Task: QRCD, which consists of question–passage–answer tuples. The system outputs 5 ranked answer spans, evaluated with EM, F1 and pRR (partial Reciprocal Ranking).
- Model: fine-tuned BERT taking the question and passage as one pair input separated by [SEP]. The Arabic pre-trained BERT outperformed multilingual BERT.
- Regularisation: decoupled weight decay (Loshchilov & Hutter) to fight overfitting.
- Augmentation: style-filtered tuples from AQQAC, with context passages regenerated from Tanzil's simple Qur'an text. The authors caution that this data was built differently and may not match the test distribution.
- Results: 59% pRR on dev versus 31% pRR on test. Planned future work includes back-translated or synonym-substituted questions and changes to activations, loss or architecture.

## Interesting ideas
- When borrowing an external QA corpus, first filter it to items that resemble the target task's question style, then rebuild each item's context from the canonical source text so inputs look like the real task.
- The authors openly flag that augmented data from a different collection method may not represent the test set, and the dev/test pRR gap bears this out.
- The system returns a ranked list of five candidate spans and is graded with a metric that gives partial credit for overlap and rank, not just exact hits.
- A head-to-head test showed that a language-specific pre-trained encoder beats a multilingual one on a narrow domain.
- The proposed future augmentation rephrases questions via back translation or synonym swaps while keeping the gold answer span fixed.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Silver QA items harvested from other resources could be style-filtered and re-anchored to the original story text before they enter training or evaluation for the fable/tale readers.
- **I2** (learn): Any augmented or teacher-generated set should be scored separately from a held-out, natively collected test split, so that distribution mismatch shows up instead of being hidden.
- **I3** (answer): The readers could emit several ranked answer candidates, each tied to its supporting frames, and be graded with a partial-overlap, rank-aware score to see near misses.
- **I4** (tree): When choosing the encoder behind gparse, a controlled comparison of an English-only versus a multilingual backbone on the same treebank would justify the choice empirically.
- **I5** (answer): Paraphrased variants of school-test or fable questions with unchanged gold answers can check that the frame-based reader is robust to surface wording rather than lexical overlap.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works

🧭 Topics (1): [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 441 works

## Cite

Cite as: Esha Aftab and Muhammad Kamran Malik. 2022. eRock at Qur'an QA 2022: Contemporary Deep Neural Networks for Qur'an based Reading Comprehension Question Answers. Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{aftab2022erock,
  title = {{eRock at Qur'an QA 2022: Contemporary Deep Neural Networks for Qur'an based Reading Comprehension Question Answers}},
  author = {Esha Aftab and Muhammad Kamran Malik},
  booktitle = {Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection},
  year = {2022},
  url = {https://aclanthology.org/2022.osact-1.11/}
}
```

- [← previous Dartmouth at SemEval-2022 Task 6: Detection of Sarcasm](https://md.research.elamur.ai/free/en/papers/lad-2022-dartmouth-semeval-2022-task-6/)
- [next → Examining Political Rhetoric with Epistemic Stance Detection](https://md.research.elamur.ai/free/en/papers/gupta-2022-examining-political-rhetoric-epistemic-stance/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/8/)

## References

13 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Proceedings of the OSACT 2022 Workshop @LREC2022, pages 96–103 Marseille, 20 June 2022 [find](https://scholar.google.com/scholar?q=Proceedings+of+the+OSACT+2022+Workshop+%40LREC2022%2C+pages+96%E2%80%93103+Marseille%2C+20+June+2022)
2. © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 Abstract or genre from a knowledge base e.g. Wikipedia; b) Closed-domain Question Answering (ClosedQA)– where the questions belong to a specific knowledge field or a particular genre e.g. law, education, finance, weather etc. The OpenQA tasks are more prevalent in studies, than closedQA, owing to their wider topic range and numerous resources. The operational pipeline of either type of QA system, may undergo following sequential steps a) Query analysis b) Search and information retrieval and c) Answer formulation. The pipeline subtask ‘Answer Formulation’ further divides the Question Answering systems into two categories: a) Abstractive Question Answering – with well-formed natural language answers abstracted from the relevant text without copying exact phrases from it, b) Extractive Question Answering – with answers derived as a consecutive sequence of tokens from the relevant documents. Abstractive Question Answering is more suitable when complex reasoning is required over multiple paragraphs/documents to answer a question (Chen et al., 2019). Machine Reading Comprehension (MRC) is a sub-task of OpenQA (Ruder, 2021). According to (Chen, 2018) MRC is a task of understanding unstructured text and precisely answer any questions about it. The answer is extracted from the context document by highlighting the start and end of the span. In this work, we attempt at solving the a closed domain MRC task released in 5th workshop on Open-Source Arabic Corpora and Corpora Processing Tools (OS-ACT) in LREC 2022 (Malhas et al., 2022). An annotated Qur’anic Reading Comprehension Dataset (QRCD) is shared at the gitlab repository 3 containing question-answer-passage tuples. The task objective is to extract an answer span against a question, from the provided passage of Quran. It is required to generate 5 possible answer spans ranked according to the scores assigned by our specific Rana Malhas and Tamer Elsayed. Official Repository of Qur’an QA Shared Task. [https://gitlab.com/bigirqu/quranqa.96](https://gitlab.com/bigirqu/quranqa.96) February 2022 model. The performance of the model is evaluated over three metrics; Exact Match (EM), F1 score and partial Reciprocal Ranking (pRR). We contribute to effectively solve the task by adopting following approaches: • Using a BERT (Devlin et al., 2018) pre-trained language model and fine tuning it for the specific dataset QRCD. [↗](https://gitlab.com/bigirqu/quranqa.96)
3. Using regularization technique, Decoupled Weight Decay (Loshchilov and Hutter, 2017), to avoid overfitting. [find](https://scholar.google.com/scholar?q=Using+regularization+technique%2C+Decoupled+Weight+Decay+%28Loshchilov+and+Hutter%2C+2017%29%2C+to+avoid+overfitting%2E)
4. Using two different pre-trained models: a) BERT pre-trained Arabic model, b) BERT pre-trained multilingual model; showing that the first pre-trained model has an edge over the second one and out performs it. • Using data augmentation to increase training data. For this purpose, we used Annotated Corpus of Arabic Al-Quran Question and Answer (AQQAC) (Alqahtani and Atwell, 2018a), filtered it to select the question answer tuples that matched in style to that of QRCD. For each tuple, we took the sequence of Quran verses present in answer and generated their context passage from Quran’s simple text downloaded from Tanzil project 4 . [find](https://scholar.google.com/scholar?q=Using+two+different+pre%2Dtrained+models%3A+a%29+BERT+pre%2Dtrained+Arabic+model%2C+b%29+BERT+pre%2Dtrained+multilingual+model%3B+showing+that+the+first+pre%2Dtrained+model+has+an+edge+over+the+second+one+and+out+perfo)
5. Finally, we analyze that how much a model can hatch improved results using regularization techniques like weight decay and data-augmentation. (Note: the augmented data being collected with an altogether different methodology might not be true representative of the testing data). 2. Related Work Neural Machine Reading Comprehension (MRC) has gained success over answer retrieval from unstructured text. Many statistical systems like AskMSR (Banko et al., 2002) exploit the answer redundancy in the source data. In MRC, the answer might not be redundant or might occur just once, thus leading to undesired results. Thus a model needs to acquire deeper understanding of the question, the context passage and their mutual relationship. The neural models address this need by learning question and context similarities. One such model is BiDAF (Seo et al., 2016) based on Recurrent Neural Network (RNN) and uses bidirectional attention flow to generate query-aware-context representations. It took lead on SQuAD (Rajpurkar et al., 2016) dataset at the time of submission, with Exact Match (EM) and F1 scores of 68% and 77% respectively. [find](https://scholar.google.com/scholar?q=Finally%2C+we+analyze+that+how+much+a+model+can+hatch+improved+results+using+regularization+techniques+like+weight+decay+and+data%2Daugmentation%2E+%28Note%3A+the+augmented+data+being+collected+with+an+altogeth)
6. DrQA (2017) (Chen et al., 2017) is a multi-layer RNN model. It presents pipeline architecture with document retriever, employing TF-IDF algorithm, and the document reader modules. The DrQA reader module showed EM score of 69% and F1 score of 78.8% on SQuAD. [find](https://scholar.google.com/scholar?q=DrQA+%282017%29+%28Chen+et+al%2E%2C+2017%29+is+a+multi%2Dlayer+RNN+model%2E+It+presents+pipeline+architecture+with+document+retriever%2C+employing+TF%2DIDF+algorithm%2C+and+the+document+reader+modules%2E+The+DrQA+reader+modu)
7. QANet (Yu et al., 2018) uses convolution and self-attention mechanisms making it more efficient than counterpart RNN based models. Being faster the model trains on more data generated through back translation. [https://tanzil.net/download/](https://tanzil.net/download/) [↗](https://tanzil.net/download/)
8. Generative Pre-trained Transformer (GPT-1) (Radford et al., 2018) model, with 117M parameters, introduced the approach of pre-training on unlabeled data with unsupervised tasks and fine-tuned on downstream tasks with remarkable results regardless of the limited size of task specific training data. Many more powerful models followed the pursuit. A few to name are BERT (Devlin et al., 2018), XLNET (Yang et al., 2019) and T5 (Raffel et al., 2019). GPT-2 (Radford et al., 2019) and GPT-3 (Brown et al., 2020), the successors of GPT-1, were trained on even larger data and with significantly more parameters, 1.5 billion and 175 billion parameters respectively. [find](https://scholar.google.com/scholar?q=Generative+Pre%2Dtrained+Transformer+%28GPT%2D1%29+%28Radford+et+al%2E%2C+2018%29+model%2C+with+117M+parameters%2C+introduced+the+approach+of+pre%2Dtraining+on+unlabeled+data+with+unsupervised+tasks+and+fine%2Dtuned+on+downs)
9. BERT (Devlin et al., 2018) has multilayer bidirectional transformer encoder architecture that pre-trains on two unsupervised tasks Masked Language Model (MLM) or Next Sequence Prediction(NSP) model. [find](https://scholar.google.com/scholar?q=BERT+%28Devlin+et+al%2E%2C+2018%29+has+multilayer+bidirectional+transformer+encoder+architecture+that+pre%2Dtrains+on+two+unsupervised+tasks+Masked+Language+Model+%28MLM%29+or+Next+Sequence+Prediction%28NSP%29+model%2E)
10. RoBERTa (Liu et al., 2019) an extension of BERT is trained for longer period over longer sequences, with NSP loss removed under the observation that it is only useful if input sequences are individual sentences instead of passages with multiple sentences. A significant improvement is observed in the performance on SQUAD 1.1/2.0, MNLI-m, SST-2 and RACE datasets in comparison to original BERT and even exceeds many of the successors of BERT. [find](https://scholar.google.com/scholar?q=RoBERTa+%28Liu+et+al%2E%2C+2019%29+an+extension+of+BERT+is+trained+for+longer+period+over+longer+sequences%2C+with+NSP+loss+removed+under+the+observation+that+it+is+only+useful+if+input+sequences+are+individual)
11. Dataset DAWQUAS (Ismail and Homsi, 2018) [find](https://scholar.google.com/scholar?q=Dataset+DAWQUAS+%28Ismail+and+Homsi%2C+2018%29)
12. AQQAC (Alqahtani and Atwell, 2018a) 2224 public: 1224 [find](https://scholar.google.com/scholar?q=AQQAC+%28Alqahtani+and+Atwell%2C+2018a%29+2224+public%3A+1224)
13. Arabic SQuAD (Mozannar et al., 2019) ARCD (Mozannar et al., 2019) TyDiQA-GoldP (Clark et al., 2020) Ayatec (Malhas and Elsayed, 2020) [find](https://scholar.google.com/scholar?q=Arabic+SQuAD+%28Mozannar+et+al%2E%2C+2019%29+ARCD+%28Mozannar+et+al%2E%2C+2019%29+TyDiQA%2DGoldP+%28Clark+et+al%2E%2C+2020%29+Ayatec+%28Malhas+and+Elsayed%2C+2020%29)

*Updated 1 November 2026*