--- title: "LARSA22 at Qur'an QA 2022 — paper by Mellah et al. (2022)" description: "The paper describes the LARSA22 team's entry to the Qur'an QA 2022 shared task. 6 references, cited by 1 paper here." authors: ["Youssef Mellah", "Ibtissam Touahri", "Zakaria Kaddari", "Zakaria Haja", "Jamal Berrich", "Toumi Bouchentouf"] year: 2022 venue: "Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection" source: "https://aclanthology.org/2022.osact-1.13/" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/mellah-2022-larsa22-qur-qa-2022-text/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mellah-2022-larsa22-qur-qa-2022-text/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/8/) / LARSA22 at Qur'an QA 2022: Text-to-Text Transformer for Finding Answers to Questions from Qur'an # LARSA22 at Qur'an QA 2022: Text-to-Text Transformer for Finding Answers to Questions from Qur'an **Authors:** [Youssef Mellah](https://md.research.elamur.ai/free/en/authors/youssef-mellah/), Ibtissam Touahri, Zakaria Kaddari, Zakaria Haja, Jamal Berrich and Toumi Bouchentouf · **Year:** 2022 · **Venue:** [Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection](https://md.research.elamur.ai/free/en/venues/osact/) ([Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks 2022](https://md.research.elamur.ai/free/en/venues/osact/2022/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2022.osact-1.13/) · [PDF](https://aclanthology.org/2022.osact-1.13.pdf) **Citations:** [1](https://md.research.elamur.ai/free/en/papers/mellah-2022-larsa22-qur-qa-2022-text/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper describes the LARSA22 team's entry to the Qur'an QA 2022 shared task. Given a question in Modern Standard Arabic and a passage of Qur'anic verses in Classical Arabic, a system must return up to five ranked answer spans taken from the passage. The authors argue that Arabic QA lags behind English because resources are scarce and the language is hard to process. They also note that religious content demands high reliability. Their system treats the task as text-to-text generation. They fine-tune the multilingual mT5 encoder-decoder on the QRCD dataset, feeding it the question concatenated with its passage and training it to output the answer as plain text. Preprocessing is light: some stop words are removed, and the JSON data is converted to TSV. The best model scores an F1 of 0.98 on the development set but only 0.40 on the test set. This gap is the paper's main lesson: strong development numbers on a small, specialized dataset may not carry over to unseen data. The work shows both the appeal and the limits of applying a general multilingual seq2seq model to a niche, low-resource QA domain. ## Key points - The data is QRCD (Qur'anic Reading Comprehension Dataset), built from AyaTEC. It has 1,337 question-passage-answer triplets split into train, dev and test, and it includes both single-answer and multi-answer questions. - The model is mT5, pre-trained on a Common Crawl-based corpus covering 101 languages. AraT5 was considered but its code was not usable when the paper was written. - The input template is `Question : Q Context : P`, and the output is the answer text. - Systems may return up to 5 ranked answers. The main metric is partial Reciprocal Rank (pRR), with Exact Match (EM) and F1@1 also reported for the top answer. - The best result is F1 0.98 on dev and 0.40 on test. Code is published at github.com/mellahysf/Quran-QA. ## Interesting ideas - Extractive span QA can be recast as generation: the question and passage are joined in one fixed textual template, and the model writes out the answer, so a single seq2seq model needs no span-pointer head. - Each system returns a ranked list of up to five candidates, scored with a reciprocal-rank metric that gives partial credit for partial matches, instead of judging a single answer as right or wrong. - Three metrics are reported side by side: strict exact match, token-overlap F1 on the top answer, and a rank-based score. Each one exposes a different kind of error. - The near-perfect dev score next to a much lower test score is a warning about small specialized datasets: development results alone can badly overstate quality. - The questions are in Modern Standard Arabic while the passages are in Classical Arabic, so the system has to bridge a register gap between query and source text. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (answer): A simple templated text-to-text baseline would give Mova's fable and school-test readers an opaque comparison point, showing how much the explainable tree-to-world pipeline gains or loses. - **I2** (answer): Readers could emit several ranked candidate answers, each with its own derivation, so evaluation can reward a correct answer even when it is not ranked first. - **I3** (learn): Evaluation reports could pair strict and lenient scores, making it visible whether failures come from wrong content or only from wrong span boundaries. - **I4** (learn): This supports Mova's use of truly held-out sets and negative controls, so no gain is accepted before it is confirmed on data the system never tuned against. - **I5** (tokens): Tests for the English tokenizer and lemmatizer could deliberately pair modern questions with archaic texts such as old fables, checking that archaic word forms map to the same lemmas as their modern counterparts. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works 🧭 Topics (2): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 441 works ## Cite Cite as: Youssef Mellah, Ibtissam Touahri, Zakaria Kaddari, Zakaria Haja, Jamal Berrich and Toumi Bouchentouf. 2022. LARSA22 at Qur'an QA 2022: Text-to-Text Transformer for Finding Answers to Questions from Qur'an. Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{mellah2022larsa22, title = {{LARSA22 at Qur'an QA 2022: Text-to-Text Transformer for Finding Answers to Questions from Qur'an}}, author = {Youssef Mellah, Ibtissam Touahri, Zakaria Kaddari, Zakaria Haja, Jamal Berrich and Toumi Bouchentouf}, booktitle = {Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection}, year = {2022}, url = {https://aclanthology.org/2022.osact-1.13/} } ``` - [← previous Large-Scale Acoustic Characterization of Singaporean Children's English Pronunciation](https://md.research.elamur.ai/free/en/papers/gu-2022-large-scale-acoustic-characterization-singaporean/) - [next → LISACTeam at SemEval-2022 Task 6: A Transformer based Approach for Intended Sarcasm Detection in English Tweets](https://md.research.elamur.ai/free/en/papers/benlahbib-2022-lisacteam-semeval-2022-task-6/) - [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/8/) ## References 6 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Many studies have been interested in providing reliable linguistic resources for the Holy Qur’an automatic annotation, processing and interpretation. In this context, (Osman et al., 2015) provided a Quranic Dataset that may be exploited by researches aiming to explore the holy Quran. (Dukes et al., 2013) presented the Quranic Arabic Corpus that includes morphological segmentation, part-of-speech, and syntactic analysis annotations. (Al-Salhi and Abdullah, 2022) constructed the ontology of Quranic stories which construction depends on the MappingMaster domain-specific language technology. (Malhas and Elsayed, 2020) introduced AyaTEC, a collection of verse-based questions answering on the Holy Qur’an. The dataset covers 11 topic categories of the Holy Qur’an and includes 207 questions and 1,762 answers covering that target the information needs of both curious and skeptical users. [find](https://scholar.google.com/scholar?q=Many+studies+have+been+interested+in+providing+reliable+linguistic+resources+for+the+Holy+Qur%E2%80%99an+automatic+annotation%2C+processing+and+interpretation%2E+In+this+context%2C+%28Osman+et+al%2E%2C+2015%29+provided+a+Q) 2. Question answering(QA) aims to search for an answer to given questions is one widely investigated task among the most challenging ones in Natural Language Processing (NLP) (Alsubhi et al., 2021). A large volume of QA studies was devoted to the English language. However, it is not the case when it comes to the Arabic language due to its scarce resources. (Alanazi et al., 2021) provided a systematic review of QA research. They have examined English relevant studies and techniques. (Maraoui et al., 2021) have used standardized hadith, narrators, and Tafsir to develop a QA system. The system reached an accuracy of 92%. Current studies Investigated advanced techniques achieving state-of-the-art results. The language pre-trained transformers models helped to achieve significant progress in many NLP tasks. (Xue et al., 2021) used pre-trained multilingual T5 (mT5) as a generative model to generate multilingual question and answer pairs. (Alsubhi et al., 2021) have evaluated the fine-tuned AraBERTv2-base (Antoun et al., 2021a), AraBERTv0.2-large, and AraELECTRA (Antoun et al., 2021b) models using Arabic-SQuAD (Mozannar et al., 2019), ARCD (Mozannar et al., 2019), AQAD (Atef et al., 2020), and TyDiQA-GoldP (Clark et al., 2020) datasets. (Alsaleh et al., 2021) applied AraBERT language model with its two versions to binary classify the QurSim dataset pairs of verses. The best result was AraBERTv0.2 with 92% accuracy. [find](https://scholar.google.com/scholar?q=Question+answering%28QA%29+aims+to+search+for+an+answer+to+given+questions+is+one+widely+investigated+task+among+the+most+challenging+ones+in+Natural+Language+Processing+%28NLP%29+%28Alsubhi+et+al%2E%2C+2021%29%2E+A+la) 3. Imbalanced classification techniques have been applied widely in the field of data mining. It is used to classify the imbalanced classes that are not equal in the number of samples. The problem with imbalanced classes is that the classification performance tends to the class with more samples while the class with few samples will obtain poor performance. This problem can be occurred in the Qur’anic classification due to the difference in the number of verses (B. S. Arkok and Zeki, 2021) (B. [find](https://scholar.google.com/scholar?q=Imbalanced+classification+techniques+have+been+applied+widely+in+the+field+of+data+mining%2E+It+is+used+to+classify+the+imbalanced+classes+that+are+not+equal+in+the+number+of+samples%2E+The+problem+with+i) 4. (B. S. Arkok and Zeki, 2021) applied methods for undersample (RUS), random oversample (ROS), synthetic minority oversampling technique (SMOTE), and random methods to classify the Qur’anic topics that are imbalanced. Many metrics were used in this research to evaluate the experimental results. (B. [find](https://scholar.google.com/scholar?q=%28B%2E+S%2E+Arkok+and+Zeki%2C+2021%29+applied+methods+for+undersample+%28RUS%29%2C+random+oversample+%28ROS%29%2C+synthetic+minority+oversampling+technique+%28SMOTE%29%2C+and+random+methods+to+classify+the+Qur%E2%80%99anic+topics+that+) 5. Arkok and Zeki, 2021) aimed to address balanced classification. They performed the Qur’anic text classification using SVM, Naïve Bayes, KNN, and J48. (Abdelnasser et al., 2014) provided an Arabic QA system for the Holy Quran that retrieves the most relevant verses corresponding to a question, then extracts the passage that contains the answer from the Quran and its interpretation books (Tafseer). Their system reached an accuracy of 85%. (Samy et al., 2019) provided a review of the Arabic Question Answering Systems, their approaches and challenges. 3. Shared Task and DataSet In this section, we introduce the context of our paper which is the “Qur’an QA 2022 Shared Task”, and we describe the given dataset in the task. 3.1 [find](https://scholar.google.com/scholar?q=Arkok+and+Zeki%2C+2021%29+aimed+to+address+balanced+classification%2E+They+performed+the+Qur%E2%80%99anic+text+classification+using+SVM%2C+Na%C3%AFve+Bayes%2C+KNN%2C+and+J48%2E+%28Abdelnasser+et+al%2E%2C+2014%29+provided+an+Arabic+QA+s) 6. Qur’an QA 2022 Shared Task Modern Standard Arabic (MSA) is the most understandable and common language in the Middle East and North Africa (by more than 400M people), and it is the official language used in media such as TV shows and newspapers. For this reason, it is encouraging for the scientific community to exert more effort to build systems specialized in solving situations related to the Arabic language. “Qur’an QA 2022 Shared Task” (Malhas et al, 2022) is shown in the efforts of developing the field of Arabic NLP and it is an attempt to enrich it, especially QA in the Qur’an script context. The task is defined to find the correct answer(s) to a question in each passage (Figure 1), the questions are written in an MSA language, while the passage consists of verses from the Holy Qur’an written in Classical Arabic. [find](https://scholar.google.com/scholar?q=Qur%E2%80%99an+QA+2022+Shared+Task+Modern+Standard+Arabic+%28MSA%29+is+the+most+understandable+and+common+language+in+the+Middle+East+and+North+Africa+%28by+more+than+400M+people%29%2C+and+it+is+the+official+language+u) *Updated 1 November 2026*