---
title: "ArabicaQA: A Comprehensive Dataset for Arabic Question Answering — Abdallah 2024"
description: "Arabic has few large, high-quality resources for machine reading comprehension (MRC) and open-domain QA. Many… 12 references, cited by 20 papers here."
authors: ["Abdelrahman Abdallah", "Mahmoud Kasem", "Mahmoud Abdalla", "Mohamed Mahmoud", "Mohamed Elkasaby", "Yasser Elbendary", "Adam Jatowt"]
year: 2024
venue: "arXiv:2403.17848"
source: "https://arxiv.org/abs/2403.17848"
doi: "10.48550/arxiv.2403.17848"
arxiv: "2403.17848"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/abdallah-2024-arabicaqa-comprehensive-dataset-arabic-question/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/abdallah-2024-arabicaqa-comprehensive-dataset-arabic-question/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / ArabicaQA: A Comprehensive Dataset for Arabic Question Answering

# ArabicaQA: A Comprehensive Dataset for Arabic Question Answering

**Authors:** Abdelrahman Abdallah, Mahmoud Kasem, Mahmoud Abdalla, Mohamed Mahmoud, Mohamed Elkasaby, Yasser Elbendary and [Adam Jatowt](https://md.research.elamur.ai/free/en/authors/adam-jatowt/) · **Year:** 2024 · **Venue:** arXiv:2403.17848 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2403.17848) · [PDF](https://arxiv.org/pdf/2403.17848) · [DOI 10.48550/arxiv.2403.17848](https://doi.org/10.48550/arxiv.2403.17848) · [arXiv 2403.17848](https://arxiv.org/abs/2403.17848) · [OpenAlex](https://openalex.org/W4400528237)

**Citations:** 27 citations (OpenAlex, 2026-10-14) · [20](https://md.research.elamur.ai/free/en/papers/abdallah-2024-arabicaqa-comprehensive-dataset-arabic-question/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Arabic has few large, high-quality resources for machine reading comprehension (MRC) and open-domain QA. Many existing ones are small, rule-based or machine-translated from English, such as Arabic-SQuAD. ArabicaQA fills this gap with a crowd-sourced dataset built on Arabic Wikipedia passages:

- Annotators wrote questions and marked the answer spans in the text.
- A second annotator checked every pair.
- Workers also wrote unanswerable questions designed to look like answerable ones.
- Questions that make sense without the passage were flagged as open-domain and labeled as needing a short or a long answer.

The authors also train AraDPR, a dense bi-encoder retriever built on AraBERT over Arabic Wikipedia. They compare it with TF-IDF, BM25 and a multilingual-BERT DPR, and AraDPR gives the best test retrieval accuracy. Finally, they benchmark several LLMs in a retrieval-augmented setup. Exact-match scores are close to zero, which shows how weak current LLMs remain on Arabic extractive QA. The resource and code are public.

## Key points
- **Dataset size:** 89,095 answerable and 3,701 unanswerable MRC questions, plus 76,266 pairs usable for open-domain QA.
- **Train/dev split:** answerable MRC is 62,186 / 13,483, and unanswerable is 2,596 / 561.
- **Retrieval results (Table 6):** AraDPR reaches 36.10 test accuracy at the smallest cutoff and 71.90 at the largest. The multilingual DPR scores 36.40 / 69.50 and BM25 scores 28.70 / 59.30. Unigram TF-IDF stays near 10 at the smallest cutoff.
- **RAG with the top-1 passage (Table 7):** GPT-3.5 gets the highest F1 at 18.32, followed by Qwen 14B at 16.08. Llama 70B and Mixtral 8x7B stay below 3 F1, and exact match is about 0 for every model.
- **Availability:** data and code are on GitHub (DataScienceUIBK/ArabicaQA). The paper appeared at SIGIR 2024.

## Interesting ideas
- Unanswerable questions were written on purpose to resemble answerable ones, so a model cannot pass just by finding word overlap and must notice when the passage lacks the answer.
- Every question was tagged for whether it stands alone without its passage and whether it needs a short or a long answer, so one collection can serve several tasks.
- Each question–answer pair was checked by a second annotator, a cheap two-step check of pair quality.
- Answers were scored with both strict exact match and a lenient "answer contains gold" measure, which shows that free-form outputs can be roughly right while never matching the gold span exactly.
- Simple sparse baselines (TF-IDF variants, BM25) were kept alongside dense retrievers, so the gain from learned representations can be measured honestly.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Lookalike-but-unanswerable items work well as negative controls in evaluation, because they show whether the reader truly checks the world of the text before answering.
- **I2** (answer): Our fable and school-test QA sets could carry similar tags, so we can measure text-grounded answers separately from answers that rely on background knowledge.
- **I3** (learn): This matches our silver-data agreement idea; a second independent witness, human or LLM tier, could confirm question/answer pairs before they enter training.
- **I4** (answer): Reporting strict and lenient scores side by side would tell us whether errors come from wrong meaning or only from the wording of the answer produced by back-generation.
- **I5** (world): When we pick the passage or knowledge-graph facts for a question, a plain lexical baseline gives a reference point to show that our structured retrieval really adds value.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · [🐪 Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/)

## Institutions

- [Universität Innsbruck](https://md.research.elamur.ai/free/en/universities/universitat-innsbruck/) — University, Innsbruck, Austria · 3 works · 423 citations
- [Assiut University](https://md.research.elamur.ai/free/en/universities/assiut-university/) — University, Asyut, Egypt · 3 works · 234 citations

## Cite

Cite as: Abdelrahman Abdallah, Mahmoud Kasem, Mahmoud Abdalla, Mohamed Mahmoud, Mohamed Elkasaby, Yasser Elbendary and Adam Jatowt. 2024. ArabicaQA: A Comprehensive Dataset for Arabic Question Answering. arXiv:2403.17848.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{abdallah2024arabicaqa,
  title = {{ArabicaQA: A Comprehensive Dataset for Arabic Question Answering}},
  author = {Abdelrahman Abdallah, Mahmoud Kasem, Mahmoud Abdalla, Mohamed Mahmoud, Mohamed Elkasaby, Yasser Elbendary and Adam Jatowt},
  year = {2024},
  doi = {10.48550/arxiv.2403.17848},
  eprint = {2403.17848},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.17848}
}
```

- [← previous XCodeEval: An Execution-based Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval](https://md.research.elamur.ai/free/en/papers/khan-2024-xcodeeval-execution-based-large-scale/)
- [next → Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws](https://md.research.elamur.ai/free/en/papers/sardana-2024-chinchilla-optimal-accounting-inference-language/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

12 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Negin Abadani, Jamshid Mozafari, Afsaneh Fatemi, Mohammd Ali Nematbakhsh, and Arefeh Kazemi. 2021. ParSQuAD: Machine Translated SQuAD dataset for Persian Question Answering. In 2021 7th International Conference on Web Research (ICWR). 163–168. https://doi.org/10.1109/ICWR51868.2021.9443126 [↗](https://doi.org/10.1109/icwr51868.2021.9443126)
2. Abdelrahman Abdallah, Mahmoud Abdalla, Mohamed Elkasaby, Yasser Elbendary, and Adam Jatowt. 2023. Amurd: annotated multilingual receipts dataset for cross-lingual key information extraction and classification. arXiv preprint arXiv:2309.09800 (2023). [↗](https://arxiv.org/abs/2309.09800)
3. Abdelrahman Abdallah and Adam Jatowt. 2023. Generator-retriever-generator: A novel approach to open-domain question answering. arXiv preprint arXiv:2307.11278 (2023). [↗](https://arxiv.org/abs/2307.11278)
4. Abdelrahman Abdallah, Mahmoud Kasem, Mohamed A Hamada, and Shaymaa Sdeek. 2020. Automated question-answer medical model based on deep learning technology. In Proceedings of the 6th International Conference on Engineering & MIS 2020. 1–8. [find](https://scholar.google.com/scholar?q=Abdelrahman+Abdallah%2C+Mahmoud+Kasem%2C+Mohamed+A+Hamada%2C+and+Shaymaa+Sdeek%2E+2020%2E+Automated+question%2Danswer+medical+model+based+on+deep+learning+technology%2E+In+Proceedings+of+the+6th+International+Confe)
5. Abdelrahman Abdallah, Bhawna Piryani, and Adam Jatowt. 2023. Exploring the state of the art in legal QA systems. Journal of Big Data 10, 1 (2023), 127. [find](https://scholar.google.com/scholar?q=Abdelrahman+Abdallah%2C+Bhawna+Piryani%2C+and+Adam+Jatowt%2E+2023%2E+Exploring+the+state+of+the+art+in+legal+QA+systems%2E+Journal+of+Big+Data+10%2C+1+%282023%29%2C+127%2E)
6. Lahsen Abouenour, Karim Bouzouba, and Paolo Rosso. 2010. An evaluated semantic query expansion and structure-based approach for enhancing Arabic question/answering. International Journal on Information and Communication Technologies 3, 3 (2010), 37–51. [find](https://scholar.google.com/scholar?q=Lahsen+Abouenour%2C+Karim+Bouzouba%2C+and+Paolo+Rosso%2E+2010%2E+An+evaluated+semantic+query+expansion+and+structure%2Dbased+approach+for+enhancing+Arabic+question%2Fanswering%2E+International+Journal+on+Informatio)
7. Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva Reddy. 2023. Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering. CoRR abs/2307.16877 (2023). https://doi.org/10.48550/ARXIV.2307.16877 arXiv:2307.16877 [↗](https://doi.org/10.48550/arxiv.2307.16877)
8. Mohammed Akour, Sameer Abufardeh, Kenneth Magel, and Qasem Al-Radaideh. 2011. QArabPro: A rule based question answering system for reading comprehension tests in Arabic. American Journal of Applied Sciences 8, 6 (2011), 652–661. [find](https://scholar.google.com/scholar?q=Mohammed+Akour%2C+Sameer+Abufardeh%2C+Kenneth+Magel%2C+and+Qasem+Al%2DRadaideh%2E+2011%2E+QArabPro%3A+A+rule+based+question+answering+system+for+reading+comprehension+tests+in+Arabic%2E+American+Journal+of+Applied+Sc)
9. Sallam AL-Sarayreh, Azza Mohamed, and Khaled Shaalan. 2023. Challenges and Solutions for Arabic Natural Language Processing in Social Media. In International conference on Variability of the Sun and sun-like stars: from asteroseismology to space weather. Springer, 293–302. [find](https://scholar.google.com/scholar?q=Sallam+AL%2DSarayreh%2C+Azza+Mohamed%2C+and+Khaled+Shaalan%2E+2023%2E+Challenges+and+Solutions+for+Arabic+Natural+Language+Processing+in+Social+Media%2E+In+International+conference+on+Variability+of+the+Sun+and+s)
10. Ali Mohamed Nabil Allam and Mohamed Hassan Haggag. 2012. The question answering systems: A survey. International Journal of Research and Reviews in Information Sciences (IJRRIS) 2, 3 (2012). [find](https://scholar.google.com/scholar?q=Ali+Mohamed+Nabil+Allam+and+Mohamed+Hassan+Haggag%2E+2012%2E+The+question+answering+systems%3A+A+survey%2E+International+Journal+of+Research+and+Reviews+in+Information+Sciences+%28IJRRIS%29+2%2C+3+%282012%29%2E)
11. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B: an open large language model with state-of-the-art performance. (2023). [find](https://scholar.google.com/scholar?q=Ebtesam+Almazrouei%2C+Hamza+Alobeidli%2C+Abdulaziz+Alshamsi%2C+Alessandro+Cappelli%2C+Ruxandra+Cojocaru%2C+Merouane+Debbah%2C+Etienne+Goffinet%2C+Daniel+Heslow%2C+Julien+Launay%2C+Quentin+Malartic%2C+Badreddine+Noune%2C+Ba)
12. Deem AlSaleh, Mashael Bin AlAmir, and Souad Larabi-Marie-Sainte. 2021. SNAD arabic dataset for deep learning. In Intelligent Systems and Applications: Proceedings of the 2020 Intelligent Systems Conference (IntelliSys) Volume 1. Springer, 630–640. arXiv:2310.06825 (2023). [29] Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016. Fasttext. zip: Compressing text classification models. arXiv preprint arXiv:1612.03651 (2016). [30] Uday Kamath, John Liu, and James Whitaker. 2019. Deep learning for NLP and speech recognition. Vol. 84. Springer. [31] Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906 (2020). [32] Arefeh Kazemi, Jamshid Mozafari, and Mohammad Ali Nematbakhsh. 2022. PersianQuAD: The Native Question Answering Dataset for the Persian Language. IEEE Access 10 (2022), 26045–26057. https://doi.org/10.1109/ACCESS. 2022.3157289 [33] Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics 6 (2018), 317–328. [34] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics 7 (2019), 453–466. [35] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33 (2020), 9459–9474. [36] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the Middle: How Language Models Use Long Contexts. CoRR abs/2307.03172 (2023). https://doi.org/10.48550/ARXIV. 2307.03172 arXiv:2307.03172 [37] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019). [38] Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 9802–9822. https://doi.org/10.18653/V1/2023.ACL-LONG.546 [39] Jamshid Mozafari, Arefeh Kazemi, Parham Moradi, Mohammad Ali Nematbakhsh, et al. 2022. PerAnSel: a novel deep neural network-based system for Persian question answering. Computational Intelligence and Neuroscience 2022 (2022). [40] Hussein Mozannar, Karl El Hajal, Elie Maamary, and Hazem Hajj. 2019. Neural Arabic question answering. arXiv preprint arXiv:1906.05394 (2019). [41] Hussein Mozannar, Elie Maamary, Karl El Hajal, and Hazem Hajj. 2019. Neural Arabic Question Answering. In Proceedings of the Fourth Arabic Natural Language Processing Workshop. Association for Computational Linguistics, Florence, Italy, 108–118. https://doi.org/10.18653/v1/W19-4612 [42] Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023 (2016). [43] Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In CoCo@ NIPS. [44] Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. arXiv preprint arXiv:1608.05457 (2016). [45] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116 (2023). arXiv:2306.01116 https://arxiv.org/abs/2306.01116 [46] Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. arXiv preprint arXiv:1806.03822 (2018). [47] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 (2016). [48] Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics 7 (2019), 249–266. [49] Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 conference on empirical methods in natural language processing. 193–203. [50] Ellen Riloff and Michael Thelen. 2000. A rule-based question answering system for reading comprehension tests. In ANLP-NAACL 2000 workshop: reading comprehension tests as evaluation for computer-based language understanding systems. [51] Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022. Improving Passage Retrieval with Zero-Shot Question Generation. (2022). https://arxiv.org/abs/2204.07496 [52] Mohamed Shaheen and Ahmed Magdy Ezzeldin. 2014. Arabic question answering: systems, resources, tools, and future trends. Arabian Journal for Science and Engineering 39 (2014), 4541–4564. [53] Shashi Pal Singh, Ajai Kumar, Hemant Darbari, Lenali Singh, Anshika Rastogi, and Shikha Jain. 2017. Machine translation using deep learning: An overview. In 2017 international conference on computer, communications and electronics (comptelix). IEEE, 162–167. [54] Radu Soricut and Eric Brill. 2006. Automatic question answering using the web: Beyond the factoid. Information Retrieval 9 (2006), 191–206. [55] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023). [56] Omar Trigui, Lamia Hadrich Belguith, and Paolo Rosso. 2010. DefArabicQA: Arabic definition question answering system. In Workshop on Language Resources and Human Language Technologies for Semitic Languages, 7th LREC, Valletta, Malta. 40–45. [57] Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. Newsqa: A machine comprehension dataset. arXiv preprint arXiv:1611.09830 (2016). [58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017). [59] Jiexin Wang, Adam Jatowt, and Masatoshi Yoshikawa. 2022. ArchivalQA: A Large-scale Benchmark Dataset for Open-Domain Question Answering over Historical News Collections. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 3025–3035. [60] Meiyin Wu and Li Chen. 2015. Image recognition based on deep learning. In 2015 Chinese automation congress (CAC). IEEE, 542–546. [61] Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain question answering. In Proceedings of the 2015 conference on empirical methods in natural language processing. 2013–2018. [62] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600 (2018). [63] Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:1810.12885 (2018). [64] Xinyu Zhang, Kelechi Ogueji, Xueguang Ma, and Jimmy Lin. 2023. Toward best practices for training multilingual dense retrieval models. ACM Transactions on Information Systems 42, 2 (2023), 1–33. [65] Xiao Zhang, Heqi Zheng, Yuxiang Nie, Heyan Huang, and Xian-Ling Mao. 2023. SciMRC: Multi-perspective Scientific Machine Reading Comprehension. arXiv preprint arXiv:2306.14149 (2023). [↗](https://doi.org/10.1109/access) [our card](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/)