---
title: "Exploring Retrieval Augmented Generation in Arabic — El-Beltagy 2024"
description: "The authors build a complete Arabic RAG pipeline and evaluate it in two steps. 27 references, cited by 5 papers here."
authors: ["Samhaa R. El-Beltagy", "Mohamed A. Abdallah"]
year: 2024
venue: "arXiv:2408.07425"
source: "https://arxiv.org/abs/2408.07425"
doi: "10.48550/arxiv.2408.07425"
arxiv: "2408.07425"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/el-beltagy-2024-exploring-retrieval-augmented-generation-arabic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/el-beltagy-2024-exploring-retrieval-augmented-generation-arabic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Exploring Retrieval Augmented Generation in Arabic

# Exploring Retrieval Augmented Generation in Arabic

**Authors:** Samhaa R. El-Beltagy and Mohamed A. Abdallah · **Year:** 2024 · **Venue:** arXiv:2408.07425 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2408.07425) · [PDF](https://arxiv.org/pdf/2408.07425) · [DOI 10.48550/arxiv.2408.07425](https://doi.org/10.48550/arxiv.2408.07425) · [arXiv 2408.07425](https://arxiv.org/abs/2408.07425) · [OpenAlex](https://openalex.org/W4403770186)

**Citations:** 17 citations (OpenAlex, 2026-10-14) · [5](https://md.research.elamur.ai/free/en/papers/el-beltagy-2024-exploring-retrieval-augmented-generation-arabic/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Retrieval-augmented generation has been studied mostly for English. It is unclear how well current embedding models and LLMs handle Arabic, which has rich morphology and many dialects that differ sharply from MSA. The authors build a complete Arabic RAG pipeline and evaluate it in two steps. First they compare multilingual semantic embedding models as retrievers. Then they connect the best retriever to several LLMs that act as generators.

They use two QA collections. ARCD is built from Wikipedia. Ar_EduText is a new small set made from high-school textbook passages, with GPT-generated questions and answers that were checked by hand. It also includes Egyptian-dialect rewrites of the questions, so the authors can test what happens when the query dialect differs from the document dialect. In addition, they try automatically rewriting context-dependent ARCD questions so each one stands alone.

Contrary to the authors' expectations, off-the-shelf models cope well. E5-large and BGE are the strongest retrievers. Among open LLMs, Llama3 and Mistral look promising, while GPT-3.5 Turbo leads on the textbook set. The study gives a practical baseline for RAG in a dialect-rich language with limited resources, and the code and data are public.

## Key points
- Ar_EduText was built from six MSA textbook passages and has 158 questions. GPT-4o drafted the questions and GPT-3.5 Turbo drafted the answers, and humans then kept, edited or rejected them. About 75.3% of the Egyptian-dialect question rewrites needed manual fixing.
- ARCD contains 1,395 crowdsourced questions from 155 Wikipedia articles, covering 460 unique paragraphs. Many of its questions depend on an earlier question for context.
- On Ar_EduText, the generators scored as follows (F1 / BLEU / cosine):
  - GPT-3.5 Turbo: 0.59 / 0.33 / 0.95
  - Llama3-70b: 0.43 / 0.20 / 0.91
  - Mixtral: 0.45 / 0.13 / 0.84
  - Mistral-7b: 0.26 / 0.08 / 0.89
  - quantized JAIS-7b: 0.37 / 0.17 / 0.72
- On ARCD, Mistral-7b came first (F1 0.22, cosine 0.83), just ahead of Llama3-70b (0.18, 0.82), with Mixtral third (0.22, 0.79).
- The open models wrapped their answers in extra text. Scoring them required post-processing written separately for each model, and the sample outputs suggest that better prompting could improve results.

## Interesting ideas
- Make a dialect-shifted copy of every test query and retrieve against documents in the standard variety, so robustness to register change becomes a measurable number.
- Rewrite a follow-up question into a self-contained one using only the previous question. Change it only when a vague reference makes it unclear, and never treat proper names as ambiguous.
- Keep both the raw machine-generated item and its human-corrected version side by side, instead of overwriting the draft.
- Evaluate in stages: lock in the best retriever first, then compare generators on top of it, so the two kinds of error do not mix.
- Score answers with several metrics at once (token F1, BLEU, embedding cosine), and extract the core answer from wordy outputs before scoring.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Rewrite evaluation questions into informal or regional English and use them as a control that the extracted frames and answers should not change.
- **I2** (frames): A conservative rule like this could guide coreference over question sequences in fable QA, replacing pronouns with explicit entities before answering.
- **I3** (learn): Storing teacher-tier outputs next to their corrections would show how often the teacher errs and would provide pairs for inducing repair rules.
- **I4** (answer): Score the world-of-text construction separately from the answering reader, so a wrong answer can be traced to one stage.
- **I5** (back): Generated text from the graph → text path could be judged the same way, separating content correctness from surface wording.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [🐪 Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/)

## Institutions

Also on the byline: [New Giza University](https://ror.org/05p2jc137).

## Cite

Cite as: Samhaa R. El-Beltagy and Mohamed A. Abdallah. 2024. Exploring Retrieval Augmented Generation in Arabic. arXiv:2408.07425.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{elbeltagy2024exploring,
  title = {{Exploring Retrieval Augmented Generation in Arabic}},
  author = {Samhaa R. El-Beltagy and Mohamed A. Abdallah},
  year = {2024},
  doi = {10.48550/arxiv.2408.07425},
  eprint = {2408.07425},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2408.07425}
}
```

- [← previous Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Models](https://md.research.elamur.ai/free/en/papers/kim-2024-developing-pragmatic-benchmark-assessing-korean/)
- [next → Generalizable Sarcasm Detection is Just Around the Corner, of Course!](https://md.research.elamur.ai/free/en/papers/jang-2024-generalizable-sarcasm-detection-is-just/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

27 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gao, Yunfan, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. (2024). "Retrieval-Augmented Generation for Large Language Models: A Survey". https://arxiv.org/abs/2312.10997. [↗](https://arxiv.org/abs/2312.10997) [our card](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/)
2. Darwish, Kareem, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R El-Beltagy, et al. (2021). "A panoramic survey of natural language processing in the Arab world". Communications of the ACM 64. ACM New York, NY, USA: 72–81. [find](https://scholar.google.com/scholar?q=Darwish%2C+Kareem%2C+Nizar+Habash%2C+Mourad+Abbas%2C+Hend+Al%2DKhalifa%2C+Huseein+T+Al%2DNatsheh%2C+Houda+Bouamor%2C+Karim+Bouzoubaa%2C+Violetta+Cavalli%2DSforza%2C+Samhaa+R+El%2DBeltagy%2C+et+al%2E+%282021%29%2E+%22A+panoramic+survey+of+)
3. Lewis, Patrick, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, et al. (2020). "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks". In Advances in Neural Information Processing Systems, ed. H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, 33:9459–9474. Curran Associates, Inc. [https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf). [↗](https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf)
4. Huang, Yizheng, and Xiangji Huang. (2024). "A Survey on Retrieval-Augmented Text Generation for Large Language Models". ArXiv abs/2404.10981. https://api.semanticscholar.org/CorpusID:269188036. [↗](https://arxiv.org/abs/2404.10981)
5. Chirkova, Nadezhda, David Rau, Hervé Déjean, Thibault Formal, Stéphane Clinchant, and Vassilina Nikoulina. (2024). "Retrieval-augmented generation in multilingual settings". https://arxiv.org/abs/2407.01463. [↗](https://arxiv.org/abs/2407.01463)
6. Abdelazim, Hazem, Mohamed Tharwat, and Ammar Mohamed. (2023). "Semantic Embeddings for Arabic Retrieval Augmented Generation (ARAG).". International Journal of Advanced Computer Science & Applications 14 (11): 1328 –1333 . [find](https://scholar.google.com/scholar?q=Abdelazim%2C+Hazem%2C+Mohamed+Tharwat%2C+and+Ammar+Mohamed%2E+%282023%29%2E+%22Semantic+Embeddings+for+Arabic+Retrieval+Augmented+Generation+%28ARAG%29%2E%22%2E+International+Journal+of+Advanced+Computer+Science+%26+Applications)
7. Mozannar, Hussein, Karl El Hajal, Elie Maamary, and Hazem Hajj. (2019). "Neural Arabic question answering". arXiv preprint arXiv:1906.05394. [↗](https://arxiv.org/abs/1906.05394)
8. Neelakantan, Arvind, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, et al. (2022). "Text and Code Embeddings by Contrastive Pre-Training". https://arxiv.org/abs/2201.10005. [↗](https://arxiv.org/abs/2201.10005) [our card](https://md.research.elamur.ai/free/en/papers/neelakantan-2022-text-code-embeddings-contrastive-pre/)
9. Alsuhaibani, Mohammed. (2023). "Deep Learning-based Sentence Embeddings using BERT for Textual Entailment". International Journal of Advanced Computer Science and Applications 14. Science and Information (SAI) Organization Limited. [find](https://scholar.google.com/scholar?q=Alsuhaibani%2C+Mohammed%2E+%282023%29%2E+%22Deep+Learning%2Dbased+Sentence+Embeddings+using+BERT+for+Textual+Entailment%22%2E+International+Journal+of+Advanced+Computer+Science+and+Applications+14%2E+Science+and+Informat)
10. Kayid, Amr, and Nils Reimers. (2022). "Cohere’s Multilingual Text Understanding Model is Now Available". [https://cohere.com/blog/multilingual](https://cohere.com/blog/multilingual). December 12. [↗](https://cohere.com/blog/multilingual)
11. Reimers, Nils, and Iryna Gurevych. (2019). "Sentence-bert: Sentence embeddings using siamese bert-networks". arXiv preprint arXiv:1908.10084. [↗](https://arxiv.org/abs/1908.10084) [our card](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/)
12. Sanh, Victor, Lysandre Debut, Julien Chaumond, and Thomas Wolf. (2019). "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter". arXiv preprint arXiv:1910.01108. [↗](https://arxiv.org/abs/1910.01108)
13. Duquenne, Paul-Ambroise, Holger Schwenk, and Beno\^\it Sagot. (2023). "SONAR: sentence-level multimodal and language-agnostic representations". arXiv e-prints: arXiv–2308. [find](https://scholar.google.com/scholar?q=Duquenne%2C+Paul%2DAmbroise%2C+Holger+Schwenk%2C+and+Beno%5C%5E%5Cit+Sagot%2E+%282023%29%2E+%22SONAR%3A+sentence%2Dlevel+multimodal+and+language%2Dagnostic+representations%22%2E+arXiv+e%2Dprints%3A+arXiv%E2%80%932308%2E)
14. Wang, Liang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. (2022). "Text Embeddings by Weakly-Supervised Contrastive Pre-training. CoRR abs/2212.03533 (2022)". [↗](https://arxiv.org/abs/2212.03533) [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-text-embeddings-weakly-supervised-contrastive/)
15. Dumais, Susan T, George W Furnas, Thomas K Landauer, Scott Deerwester, and Richard Harshman. (1988). "Using latent semantic analysis to improve access to textual information". In Proceedings of the SIGCHI conference on Human factors in computing systems, 281–285. [find](https://scholar.google.com/scholar?q=Dumais%2C+Susan+T%2C+George+W+Furnas%2C+Thomas+K+Landauer%2C+Scott+Deerwester%2C+and+Richard+Harshman%2E+%281988%29%2E+%22Using+latent+semantic+analysis+to+improve+access+to+textual+information%22%2E+In+Proceedings+of+the+SI)
16. Deerwester, Scott, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. (1990). "Indexing by latent semantic analysis". Journal of the American society for information science 41. Wiley Online Library: 391–407. [find](https://scholar.google.com/scholar?q=Deerwester%2C+Scott%2C+Susan+T+Dumais%2C+George+W+Furnas%2C+Thomas+K+Landauer%2C+and+Richard+Harshman%2E+%281990%29%2E+%22Indexing+by+latent+semantic+analysis%22%2E+Journal+of+the+American+society+for+information+science+41%2E)
17. Mikolov, Tomas, Greg Corrado, Kai Chen, and Jeffrey Dean. (2013). "word2vec-v1". Proceedings of the International Conference on Learning Representations (ICLR 2013): 1–12. [find](https://scholar.google.com/scholar?q=Mikolov%2C+Tomas%2C+Greg+Corrado%2C+Kai+Chen%2C+and+Jeffrey+Dean%2E+%282013%29%2E+%22word2vec%2Dv1%22%2E+Proceedings+of+the+International+Conference+on+Learning+Representations+%28ICLR+2013%29%3A+1%E2%80%9312%2E)
18. Peters, Matthew E, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. (2018). "Deep Contextualized Word Representations". In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), ed. Marilyn Walker, Heng Ji, and Amanda Stent, 2227–2237. New Orleans, Louisiana: Association for Computational Linguistics. https://aclanthology.org/N18-1202. [↗](https://aclanthology.org/N18-1202/) [our card](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/)
19. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. (2018). "Bert: Pre-training of deep bidirectional transformers for language understanding". arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
20. Soliman, Abu Bakr, Kareem Eissa, and Samhaa R. El-Beltagy. (2017). "AraVec: A set of Arabic Word Embedding Models for use in Arabic NLP". In Procedia Computer Science. [find](https://scholar.google.com/scholar?q=Soliman%2C+Abu+Bakr%2C+Kareem+Eissa%2C+and+Samhaa+R%2E+El%2DBeltagy%2E+%282017%29%2E+%22AraVec%3A+A+set+of+Arabic+Word+Embedding+Models+for+use+in+Arabic+NLP%22%2E+In+Procedia+Computer+Science%2E)
21. Antoun, Wissam, Fady Baly, and Hazem Hajj. (2020). "Arabert: Transformer-based model for arabic language understanding". arXiv preprint arXiv:2003.00104. [↗](https://arxiv.org/abs/2003.00104) [our card](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/)
22. Ollama. (2024). "Ollama Embedding models". [https://ollama.com/blog/embedding-models](https://ollama.com/blog/embedding-models). April. [↗](https://ollama.com/blog/embedding-models)
23. Sengupta, Neha, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, et al. (2023). "Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models". arXiv preprint arXiv:2308.16149. [↗](https://arxiv.org/abs/2308.16149) [our card](https://md.research.elamur.ai/free/en/papers/sengupta-2023-jais-jais-chat-arabic-centric/)
24. Xiao, Shitao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. (2023). "C-Pack: Packaged Resources To Advance General Chinese Embedding". [find](https://scholar.google.com/scholar?q=Xiao%2C+Shitao%2C+Zheng+Liu%2C+Peitian+Zhang%2C+and+Niklas+Muennighoff%2E+%282023%29%2E+%22C%2DPack%3A+Packaged+Resources+To+Advance+General+Chinese+Embedding%22%2E)
25. Jiang, Albert Q, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, et al. (2023). "Mistral 7B". https://arxiv.org/abs/2310.06825. [↗](https://arxiv.org/abs/2310.06825)
26. Dubey, Abhimanyu, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, et al. (2024). "The Llama 3 Herd of Models". https://arxiv.org/abs/2407.21783. [↗](https://arxiv.org/abs/2407.21783)
27. Jiang, Albert Q, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, et al. (2024). "Mixtral of Experts". https://arxiv.org/abs/2401.04088. [↗](https://arxiv.org/abs/2401.04088)