---
title: "101 Billion Arabic Words Dataset — Aloui 2024"
description: "The paper tackles the shortage of large, native Arabic text for pretraining language models."
authors: ["Manel Aloui", "Hasna Chouikhi", "Ghaith Chaabane", "Haithem Kchaou", "Chehir Dhaouadi"]
year: 2024
venue: "arXiv:2405.01590"
source: "https://arxiv.org/abs/2405.01590"
doi: "10.48550/arxiv.2405.01590"
arxiv: "2405.01590"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/aloui-2024-101-billion-arabic-words-dataset/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/aloui-2024-101-billion-arabic-words-dataset/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / 101 Billion Arabic Words Dataset

# 101 Billion Arabic Words Dataset

**Authors:** [Manel Aloui](https://md.research.elamur.ai/free/en/authors/manel-aloui/), [Hasna Chouikhi](https://md.research.elamur.ai/free/en/authors/hasna-chouikhi/), [Ghaith Chaabane](https://md.research.elamur.ai/free/en/authors/ghaith-chaabane/), [Haithem Kchaou](https://md.research.elamur.ai/free/en/authors/haithem-kchaou/), [Chehir Dhaouadi](https://md.research.elamur.ai/free/en/authors/chehir-dhaouadi/) · **Year:** 2024 · **Venue:** arXiv:2405.01590 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2405.01590) · [PDF](https://arxiv.org/pdf/2405.01590) · [DOI 10.48550/arxiv.2405.01590](https://doi.org/10.48550/arxiv.2405.01590) · [arXiv 2405.01590](https://arxiv.org/abs/2405.01590) · [OpenAlex](https://openalex.org/W4396764611)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [12](https://md.research.elamur.ai/free/en/papers/aloui-2024-101-billion-arabic-words-dataset/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper tackles the shortage of large, native Arabic text for pretraining language models. Many Arabic LLMs rely on English or translated material, which can introduce bias and weaken authenticity. Following the RefinedWeb idea that well-filtered web data alone can be enough, the authors mined Common Crawl WET files collected between late 2021 and mid-2022. They kept only Arabic pages and stored the records as JSONL. To decide how much raw text to read by hand, they used Yamane's sample-size formula, which gave about 10,000 documents. Reading that sample exposed the main kinds of noise: stray HTML, odd Unicode, extra whitespace and blank lines, and adult content. A pipeline then filtered and deduplicated the data, mainly at the URL level, running on AWS. This process produced a corpus named the 101 Billion Arabic Words Dataset, which is available on HuggingFace and presented as the biggest Arabic corpus to date. Its quality has not been checked by training a model on it.

## Key points
- The source was 39,502 compressed Common Crawl WET files, about 0.8 PB, covering week 39 of 2021 to week 27 of 2022 in six splits. A typical split held roughly 33,292 sites, of which about 250 were Arabic.
- The manual audit sample of about 10,000 documents came from Yamane's formula with a 1% margin of error, applied to a stated population of about 116.65 billion documents.
- URL filtering had two stages: matching against a list of bad terms, then a second check with the Rust crate rustrict v0.7.24. URL deduplication followed. About 4% of URLs were marked as undesirable.
- For infrastructure, EC2 and S3 were kept in the same AWS region to cut latency and transfer cost, and each process had a bandwidth cap to keep the system stable.
- Analysis of country domains shows that Saudi Arabian sites dominate and North African countries are clearly under-represented. The authors list two limitations: no model was trained to test the data, and URL filtering may miss sensitive content.

## Interesting ideas
- Use a standard survey sample-size formula (Yamane) to choose how many items to inspect by hand, so the audit is cheap but statistically grounded, without picking an arbitrary number — For Mova (learn): Apply the same formula to size manual checks of silver parses or self-induced repair rules, and report the margin of error alongside the measured accuracy.
- Let a small manual sample of raw data decide which cleaning steps to build, instead of starting from a generic cleaning checklist — For Mova (tokens): Read a sized sample of incoming English text first, then add tokenizer and normalization rules only for the noise actually found, such as leftover markup, unusual Unicode and broken whitespace.
- Filter cheaply first and validate with a stricter second tool, so the expensive check runs only on what survives the first pass — For Mova (learn): Gate silver-data candidates with a fast heuristic, then send only the survivors to the multi-parser agreement step to save compute.
- After filtering, examine where the remaining data comes from, as the paper does with country domains, so hidden skews such as missing regions become visible — For Mova (learn): Break silver and training corpora down by source and genre, so that skews (for example, only news and no fables) are found before they bias the parser or readers.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{aloui2024101,
  title = {{101 Billion Arabic Words Dataset}},
  author = {Aloui, Manel and Chouikhi, Hasna and Chaabane, Ghaith and Kchaou, Haithem and Dhaouadi, Chehir},
  year = {2024},
  doi = {10.48550/arxiv.2405.01590},
  eprint = {2405.01590},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.01590}
}
```

- [← previous SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corpora](https://md.research.elamur.ai/free/en/papers/qarah-2024-saudibert-large-language-model-pretrained/)
- [next → Ar-Spider: Text-to-SQL in Arabic](https://md.research.elamur.ai/free/en/papers/almohaimeed-2024-ar-spider-text-sql-arabic/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

29 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Harika Abburi et al. Generative ai text classification using ensemble llm approaches. arXiv preprint arXiv:2309.07755, 2023. [↗](https://arxiv.org/abs/2309.07755)
2. Abdelrahman Abdallah, Mahmoud Kasem, Mahmoud Abdalla, Mohamed Mahmoud, Mohamed Elkasaby, Yasser Elbendary, and Adam Jatowt. Arabicaqa: A comprehensive dataset for arabic question answering. arXiv preprint arXiv:2403.17848, 2024. [↗](https://arxiv.org/abs/2403.17848) [our card](https://md.research.elamur.ai/free/en/papers/abdallah-2024-arabicaqa-comprehensive-dataset-arabic-question/)
3. Ahmed Abdelali et al. Larabench: Benchmarking arabic ai with large language models. 2024. [find](https://scholar.google.com/scholar?q=Ahmed+Abdelali+et+al%2E+Larabench%3A+Benchmarking+arabic+ai+with+large+language+models%2E+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/abdelali-2024-larabench-benchmarking-arabic-ai-large/)
4. Asaad Alghamdi, Xinyu Duan, Wei Jiang, Zhenhai Wang, Yimeng Wu, Qingrong Xia, Zhefeng Wang, Yi Zheng, Mehdi Rezagholizadeh, Baoxing Huai, et al. Aramus: Pushing the limits of data and model scale for arabic natural language processing. arXiv preprint arXiv:2306.06800, 2023. [↗](https://arxiv.org/abs/2306.06800)
5. Zaid Alyafeai and Maged Al-Shaibani. ARBML: Democritizing Arabic natural language processing tools. In Eunjeong L. Park, Masato Hagiwara, Dmitrijs Milajevs, Nelson F. Liu, Geeticka Chauhan, and Liling Tan (eds.), Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), pp. 8–13, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.nlposs-1.2. URL https://aclanthology.org/2020.nlposs-1.2. [↗](https://doi.org/10.18653/v1/2020.nlposs-1.2)
6. Zaid Alyafeai, Khalid Almubarak, Ahmed Ashraf, Deema Alnuhait, Saied Alshahrani, Gubran A. Q. Abdulrahman, Gamil Ahmed, Qais Gawah, Zead Saleh, Mustafa Ghaleb, Yousef Ali, and Maged S. Al-Shaibani. Cidar: Culturally relevant instruction dataset for arabic, 2024. [find](https://scholar.google.com/scholar?q=Zaid+Alyafeai%2C+Khalid+Almubarak%2C+Ahmed+Ashraf%2C+Deema+Alnuhait%2C+Saied+Alshahrani%2C+Gubran+A%2E+Q%2E+Abdulrahman%2C+Gamil+Ahmed%2C+Qais+Gawah%2C+Zead+Saleh%2C+Mustafa+Ghaleb%2C+Yousef+Ali%2C+and+Maged+S%2E+Al%2DShaibani%2E+Ci)
7. Alaaeldin El-Nouby et al. Are large-scale datasets necessary for self-supervised pre-training? 2021. [find](https://scholar.google.com/scholar?q=Alaaeldin+El%2DNouby+et+al%2E+Are+large%2Dscale+datasets+necessary+for+self%2Dsupervised+pre%2Dtraining%3F+2021%2E)
8. William Held et al. A material lens on coloniality in nlp. arXiv preprint arXiv:2311.08391, 2023. [↗](https://arxiv.org/abs/2311.08391)
9. Hanlei Jin et al. A comprehensive survey on process-oriented automatic text summarization with exploration of llm-based methods. arXiv preprint arXiv:2403.02901, 2024. [↗](https://arxiv.org/abs/2403.02901)
10. Pratik Joshi, Christain Barnes, Sebastin Santy, Simran Khanuja, Sanket Shah, Anirudh Srinivasan, Satwik Bhattamishra, Sunayana Sitaram, Monojit Choudhury, and Kalika Bali. Unsung challenges of building and deploying language technologies for low resource language communities. pp. 211–219, 2019. URL https://aclanthology.org/2019.icon-1.25. [↗](https://aclanthology.org/2019.icon-1.25/)
11. Vincent Jung and Lonneke van der Plas. Understanding the effects of language-specific class imbalance in multilingual fine-tuning. 2024. [find](https://scholar.google.com/scholar?q=Vincent+Jung+and+Lonneke+van+der+Plas%2E+Understanding+the+effects+of+language%2Dspecific+class+imbalance+in+multilingual+fine%2Dtuning%2E+2024%2E)
12. Thomas Lancaster. A large language model supported synthesis of contemporary academic integrity research trends. 2024. [find](https://scholar.google.com/scholar?q=Thomas+Lancaster%2E+A+large+language+model+supported+synthesis+of+contemporary+academic+integrity+research+trends%2E+2024%2E)
13. Zhenyu Li et al. Flexkbqa: A flexible llm-powered framework for few-shot knowledge base question answering. 38(17), 2024. [find](https://scholar.google.com/scholar?q=Zhenyu+Li+et+al%2E+Flexkbqa%3A+A+flexible+llm%2Dpowered+framework+for+few%2Dshot+knowledge+base+question+answering%2E+38%2817%29%2C+2024%2E)
14. Alexandre Magueresse, Vincent Carles, and Evan Heetderks. Low-resource languages: A review of past work and future challenges. arXiv preprint arXiv:2006.07264, 2020. [↗](https://arxiv.org/abs/2006.07264)
15. Mike Maxwell and Baden Hughes. Frontiers in linguistic annotation for lower-density languages. 2006. [find](https://scholar.google.com/scholar?q=Mike+Maxwell+and+Baden+Hughes%2E+Frontiers+in+linguistic+annotation+for+lower%2Ddensity+languages%2E+2006%2E)
16. Masayasu Muraoka et al. Cross-lingual transfer of large language model by visually-derived supervision toward low-resource languages. 2023. [find](https://scholar.google.com/scholar?q=Masayasu+Muraoka+et+al%2E+Cross%2Dlingual+transfer+of+large+language+model+by+visually%2Dderived+supervision+toward+low%2Dresource+languages%2E+2023%2E)
17. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 7022–7032, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-34-4. URL https://aclanthology.org/2020.lrec-1.868. [↗](https://aclanthology.org/2020.lrec-1.868/) [our card](https://md.research.elamur.ai/free/en/papers/obeid-2020-camel-tools-open-source-python/)
18. Rasha Obeidat, Marwa Al-Harbi, Mahmoud Al-Ayyoub, and Luay Alawneh. Arquad: An expert-annotated arabic machine reading comprehension dataset. Cognitive Computation, pp. 1–20, 2024. [find](https://scholar.google.com/scholar?q=Rasha+Obeidat%2C+Marwa+Al%2DHarbi%2C+Mahmoud+Al%2DAyyoub%2C+and+Luay+Alawneh%2E+Arquad%3A+An+expert%2Dannotated+arabic+machine+reading+comprehension+dataset%2E+Cognitive+Computation%2C+pp%2E+1%E2%80%9320%2C+2024%2E)
19. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116, 2023. [↗](https://arxiv.org/abs/2306.01116)
20. Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson. Data cards: Purposeful and transparent dataset documentation for responsible ai. 2022. doi: 10.1145/3531146.3533231. URL https://doi.org/10.1145/3531146.3533231. [↗](https://doi.org/10.1145/3531146.3533231)
21. Nathaniel R. Robinson et al. Chatgpt mt: Competitive for high-(but not low-) resource languages. arXiv preprint arXiv:2309.07423, 2023. [↗](https://arxiv.org/abs/2309.07423) [our card](https://md.research.elamur.ai/free/en/papers/robinson-2023-chatgpt-mt-competitive-high-but/)
22. Neha Sengupta and et al. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. 2023. [find](https://scholar.google.com/scholar?q=Neha+Sengupta+and+et+al%2E+Jais+and+jais%2Dchat%3A+Arabic%2Dcentric+foundation+and+instruction%2Dtuned+open+generative+large+language+models%2E+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/sengupta-2023-jais-jais-chat-arabic-centric/)
23. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, et al. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149, 2023. [↗](https://arxiv.org/abs/2308.16149) [our card](https://md.research.elamur.ai/free/en/papers/sengupta-2023-jais-jais-chat-arabic-centric/)
24. Shivalika Singh and et al. Aya dataset: An open-access collection for multilingual instruction tuning. 2024. [find](https://scholar.google.com/scholar?q=Shivalika+Singh+and+et+al%2E+Aya+dataset%3A+An+open%2Daccess+collection+for+multilingual+instruction+tuning%2E+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/singh-2024-aya-dataset-open-access-collection/)
25. Amel Slim and Ahlem Melouah. Low resource arabic dialects transformer neural machine translation improvement through incremental transfer of shared linguistic features. Arabian Journal for Science and Engineering, pp. 1–17, 2024. [find](https://scholar.google.com/scholar?q=Amel+Slim+and+Ahlem+Melouah%2E+Low+resource+arabic+dialects+transformer+neural+machine+translation+improvement+through+incremental+transfer+of+shared+linguistic+features%2E+Arabian+Journal+for+Science+and)
26. Karthik Valmeekam et al. Large language models still can’t plan (a benchmark for llms on planning and reasoning about change). arXiv preprint arXiv:2206.10498, 2022. [↗](https://arxiv.org/abs/2206.10498) [our card](https://md.research.elamur.ai/free/en/papers/valmeekam-2022-planbench-extensible-benchmark-evaluating-large/)
27. Xinyi Wang et al. Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning. 2023. [find](https://scholar.google.com/scholar?q=Xinyi+Wang+et+al%2E+Large+language+models+are+implicitly+topic+models%3A+Explaining+and+finding+good+demonstrations+for+in%2Dcontext+learning%2E+2023%2E)
28. Asma Yamani, Raghad Alziyady, Reem AlYami, Salma Albelali, Leina Albelali, Jawharah Almul-him, Amjad Alsulami, Motaz Alfarraj, and Rabeah Al-Zaidy. The kind dataset: A social collaboration approach for nuanced dialect data collection. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pp. 32–43, 2024. [find](https://scholar.google.com/scholar?q=Asma+Yamani%2C+Raghad+Alziyady%2C+Reem+AlYami%2C+Salma+Albelali%2C+Leina+Albelali%2C+Jawharah+Almul%2Dhim%2C+Amjad+Alsulami%2C+Motaz+Alfarraj%2C+and+Rabeah+Al%2DZaidy%2E+The+kind+dataset%3A+A+social+collaboration+approach+fo)
29. Yifan Yao et al. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. pp. 100211, 2024. [find](https://scholar.google.com/scholar?q=Yifan+Yao+et+al%2E+A+survey+on+large+language+model+%28llm%29+security+and+privacy%3A+The+good%2C+the+bad%2C+and+the+ugly%2E+pp%2E+100211%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/yao-2023-survey-large-language-model-llm/)