---
title: "Enhancing Arabic Retrieval Augmented Generation through"
description: "The paper examines how the quality of text extraction from Arabic academic PDFs affects retrieval and, through retrieval, the… 13 references."
authors: ["Shadi Saleh", "Belkacem EL Jattari", "Layth Oud", "Maryam Alblooshi", "Bouthaina Lakhdari", "Ali Alnaqbi"]
year: 2025
venue: "Proceedings of the 8th International Conference on Natural Language and Speech Processing (ICNLSP-2025)"
source: "https://aclanthology.org/2025.icnlsp-1.45/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/saleh-2025-enhancing-arabic-retrieval-augmented-generation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/saleh-2025-enhancing-arabic-retrieval-augmented-generation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/17/) / Enhancing Arabic Retrieval Augmented Generation through Language Processing

# Enhancing Arabic Retrieval Augmented Generation through Language Processing

**Authors:** [Shadi Saleh](https://md.research.elamur.ai/free/en/authors/shadi-saleh/), Belkacem EL Jattari, Layth Oud, Maryam Alblooshi, Bouthaina Lakhdari and Ali Alnaqbi · **Year:** 2025 · **Venue:** [Proceedings of the 8th International Conference on Natural Language and Speech Processing (ICNLSP-2025)](https://md.research.elamur.ai/free/en/venues/icnlsp/) ([International Conference on Natural Language and Speech Processing 2025](https://md.research.elamur.ai/free/en/venues/icnlsp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.icnlsp-1.45/) · [PDF](https://aclanthology.org/2025.icnlsp-1.45.pdf) · [OpenAlex](https://openalex.org/W4407397437)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper examines how the quality of text extraction from Arabic academic PDFs affects retrieval and, through retrieval, the answers of a retrieval-augmented chatbot. The authors work with about 12,000 journal articles from the Shamra Academia portal. They compare plain PDF text extraction with PyPDF against the deep-learning OCR system Surya. They then clean the OCR output with a tightly constrained LLM copyediting prompt. Both versions are indexed in Elasticsearch and tested with several classic ranking models.

For evaluation they built a benchmark of 170 real search queries with graded relevance judgments from native-speaker researchers. Cleaner extraction raised P@5 by roughly 8 points for Okapi BM25. The resulting ArabicRAG assistant was compared with GPT-4o answering without retrieval. ArabicRAG produced more fully correct and properly cited answers and far fewer complete failures.

The main lesson is that, for morphologically rich, right-to-left scripts, careful preprocessing comes before any clever modelling. Hallucinated references are a serious risk in academic use.

## Key points
- **Corpus and queries:** The corpus has about 12,000 Arabic PDFs from Shamra Academia.
  - The query pool came from six months of Google/Bing search-console logs (top 100 per month, 600 in total).
  - Experts reduced the pool to 170 queries, split 100 train / 70 test and stratified over 8 subject categories.
- **Extraction comparison:** PyPDF often merged words or dropped letters; in one shown sentence only 6 of 15 words survived. Surya extracted the same sentence correctly but sometimes mangled specialised medical and engineering terms. An LLM correction pass was added to address this.
- **Relevance judgments:** Three judges labelled documents on four levels using the relevation tool. Agreement, measured on 30 re-judged pairs per judge, was 70%. It rose to 78% once the labels were collapsed to binary.
- **Retrieval results (Okapi BM25):** NDCG@5 went from 67.30 to 73.15 and P@5 from 37.97 to 45.50 when switching from PyPDF to Surya. LMDirichlet and LM-JelinekMercer also improved.
- **Chatbot evaluation:** The comparison ran through two Telegram bots.
  - ArabicRAG scored a mean of 2.4 (±0.8) against 1.5 (±1.2) for the GPT-4o baseline.
  - Perfect answers made up 60% versus 40%, and zero-score answers 5% versus 15%.
  - Most ArabicRAG failures were blamed on gaps in corpus coverage.

## Interesting ideas
- An LLM is used only as a restricted OCR fixer: it is told the input comes from OCR, must keep technical terms, must not rephrase or add words, and must output only the corrected text, which curbs meaning drift.
- The test set is built from real search logs, filtered by experts, and split train/test while preserving the category distribution, so evaluation mirrors genuine usage rather than invented questions.
- Annotation reliability is checked by having judges re-label pairs already judged by others, with agreement compared at fine and coarse granularity; disagreement clustered in the middle grades.
- Answers are graded on two separate axes, citation correctness and answer correctness, and the report includes the share of perfect answers, the share of total failures and the spread, not just a mean.
- Failures are traced to the retrieval stage (thin coverage), and a fallback is proposed when retrieval confidence is low.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): A similarly fenced "repair-only" pass could normalise noisy or garbled input before tokenization, without silently changing what the text says.
- **I2** (learn): Evaluation suites for readers could be sampled from real user questions and stratified by domain, so gains are not driven by one over-represented topic.
- **I3** (learn): When several witness parsers vote on silver data, agreement could be measured both on fine labels and on coarsened label groups to see which distinctions are truly unstable.
- **I4** (answer): QA on fables or school tests could score "is the answer right" separately from "is it grounded in the right sentence or event", and report failure rates alongside averages.
- **I5** (world): When the world model or knowledge graph has no good support for a query, the system could flag low confidence or trigger extra lookup instead of producing an ungrounded answer.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (2): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 552 works

## Cite

Cite as: Shadi Saleh, Belkacem EL Jattari, Layth Oud, Maryam Alblooshi, Bouthaina Lakhdari and Ali Alnaqbi. 2025. Enhancing Arabic Retrieval Augmented Generation through Language Processing. Proceedings of the 8th International Conference on Natural Language and Speech Processing (ICNLSP-2025).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{saleh2025enhancing,
  title = {{Enhancing Arabic Retrieval Augmented Generation through Language Processing}},
  author = {Shadi Saleh, Belkacem {EL Jattari}, Layth Oud, Maryam Alblooshi, Bouthaina Lakhdari and Ali Alnaqbi},
  booktitle = {Proceedings of the 8th International Conference on Natural Language and Speech Processing (ICNLSP-2025)},
  year = {2025},
  doi = {10.2139/ssrn.5134287},
  url = {https://doi.org/10.2139/ssrn.5134287}
}
```

- [← previous Enhancing Arabic Morphology Learning via Jigsaw: An Experimental Study in a Islamic Boarding School](https://md.research.elamur.ai/free/en/papers/habib-2025-enhancing-arabic-morphology-learning-jigsaw/)
- [next → Enhancing Quranic Learning: A Multimodal Deep Learning Approach for Arabic Phoneme Recognition](https://md.research.elamur.ai/free/en/papers/kucukmanisa-2025-enhancing-quranic-learning-multimodal-deep/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/17/)

## References

13 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gagan Bhatia, El Moatez Billah Nagoudi, Fakhraddin Alwajih, and Muhammad Abdul-Mageed. 2024. [Qalam: A multimodal LLM for Arabic optical character and handwriting recognition](https://md.research.elamur.ai/free/en/papers/bhatia-2024-qalam-multimodal-llm-arabic-optical/). In Proceedings of The Second Arabic Natural Language Processing Conference, pages 210–224, Bangkok, Thailand. Association for Computational Linguistics.
2. David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. [Latent dirichlet allocation](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/). Journal of machine Learning research, 3(Jan):993–1022.
3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran Associates Inc.
4. H. Bunke, M. Roth, and E.G. Schukat-Talamazzini. 1995. Off-line cursive handwriting recognition using hidden markov models. Pattern Recognition, 28(9):1399–1413. [find](https://scholar.google.com/scholar?q=H%2E+Bunke%2C+M%2E+Roth%2C+and+E%2EG%2E+Schukat%2DTalamazzini%2E+1995%2E+Off%2Dline+cursive+handwriting+recognition+using+hidden+markov+models%2E+Pattern+Recognition%2C+28%289%29%3A1399%E2%80%931413%2E)
5. Fabio Crestani, Mounia Lalmas, Cornelis J. Van Rijsbergen, and Iain Campbell. 1998. “is this document relevant?. . . probably”: a survey of probabilistic models in information retrieval. ACM Comput. Surv., 30(4):528–552. [find](https://scholar.google.com/scholar?q=Fabio+Crestani%2C+Mounia+Lalmas%2C+Cornelis+J%2E+Van+Rijsbergen%2C+and+Iain+Campbell%2E+1998%2E+%E2%80%9Cis+this+document+relevant%3F%2E+%2E+%2E+probably%E2%80%9D%3A+a+survey+of+probabilistic+models+in+information+retrieval%2E+ACM+Comput%2E+S)
6. Dima El Zein and Célia da Costa Pereira. 2022. User’s knowledge and information needs in information retrieval evaluation. In Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization, UMAP ’22, page 170–178, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Dima+El+Zein+and+C%C3%A9lia+da+Costa+Pereira%2E+2022%2E+User%E2%80%99s+knowledge+and+information+needs+in+information+retrieval+evaluation%2E+In+Proceedings+of+the+30th+ACM+Conference+on+User+Modeling%2C+Adaptation+and+Pe)
7. Ehsan Kamalloo, Nouha Dziri, Charles Clarke, and Davood Rafiei. 2023. [Evaluating open-domain question answering in the era of large language models](https://md.research.elamur.ai/free/en/papers/kamalloo-2023-evaluating-open-domain-question-answering/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5591–5606, Toronto, Canada. Association for Computational Linguistics.
8. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193–203, Tampere, Finland. European Association for Machine Translation. [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Christian+Federmann%2E+2023%2E+Large+language+models+are+state%2Dof%2Dthe%2Dart+evaluators+of+translation+quality%2E+In+Proceedings+of+the+24th+Annual+Conference+of+the+European+Association+for+Mach)
9. Sewon Min, Jordan Boyd-Graber, Chris Alberti, Danqi Chen, Eunsol Choi, Michael Collins, Kelvin Guu, Hannaneh Hajishirzi, Kenton Lee, Jennimaria Palomaki, et al. 2021. Neurips 2020 efficientqa competition: Systems, analyses and lessons learned. In NeurIPS 2020 Competition and Demonstration Track, pages 86–111. PMLR. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Jordan+Boyd%2DGraber%2C+Chris+Alberti%2C+Danqi+Chen%2C+Eunsol+Choi%2C+Michael+Collins%2C+Kelvin+Guu%2C+Hannaneh+Hajishirzi%2C+Kenton+Lee%2C+Jennimaria+Palomaki%2C+et+al%2E+2021%2E+Neurips+2020+efficientqa+competit)
10. Fabio Petroni, Federico Siciliano, Fabrizio Silvestri, and Giovanni Trappolini. 2024. IR-RAG @ SIGIR24: Information Retrieval’s Role in RAG Systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page 3036–3039, New York, NY, USA. ACM. [find](https://scholar.google.com/scholar?q=Fabio+Petroni%2C+Federico+Siciliano%2C+Fabrizio+Silvestri%2C+and+Giovanni+Trappolini%2E+2024%2E+IR%2DRAG+%40+SIGIR24%3A+Information+Retrieval%E2%80%99s+Role+in+RAG+Systems%2E+In+Proceedings+of+the+47th+International+ACM+SIGIR+)
11. Jawad Sadek, Fairouz Chakkour, and Farid Meziane. 2012. Arabic Rhetorical Relations Extraction For Answering "Why" and "How to" Questions. In Proceedings of the 17th International Conference on Applications of Natural Language Processing and Information Systems, NLDB’12, page 385–390, Berlin, Heidelberg. Springer. [find](https://scholar.google.com/scholar?q=Jawad+Sadek%2C+Fairouz+Chakkour%2C+and+Farid+Meziane%2E+2012%2E+Arabic+Rhetorical+Relations+Extraction+For+Answering+%22Why%22+and+%22How+to%22+Questions%2E+In+Proceedings+of+the+17th+International+Conference+on+Applic)
12. Simone Teufel. 2007. An Overview of Evaluation Methods in TREC Ad Hoc Information Retrieval and TREC Question Answering, pages 163–186. Springer Netherlands, Dordrecht. [find](https://scholar.google.com/scholar?q=Simone+Teufel%2E+2007%2E+An+Overview+of+Evaluation+Methods+in+TREC+Ad+Hoc+Information+Retrieval+and+TREC+Question+Answering%2C+pages+163%E2%80%93186%2E+Springer+Netherlands%2C+Dordrecht%2E)
13. Chengxiang Zhai and John Lafferty. 2017. A study of smoothing methods for language models applied to ad hoc information retrieval. In ACM SIGIR Forum, volume 51, pages 268–276. ACM New York, NY, USA. [find](https://scholar.google.com/scholar?q=Chengxiang+Zhai+and+John+Lafferty%2E+2017%2E+A+study+of+smoothing+methods+for+language+models+applied+to+ad+hoc+information+retrieval%2E+In+ACM+SIGIR+Forum%2C+volume+51%2C+pages+268%E2%80%93276%2E+ACM+New+York%2C+NY%2C+USA%2E)

*Updated 2 November 2026*