--- title: "1.5 billion words Arabic Corpus — paper by El-khair (2016)" description: "Arabic has few corpora that are both large and free. 9 references, cited by 60 papers here." authors: ["Ibrahim Abu El-khair"] year: 2016 venue: "arXiv:1611.04033" source: "https://arxiv.org/abs/1611.04033" doi: "10.48550/arxiv.1611.04033" arxiv: "1611.04033" licence: "CC BY-NC-SA 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by-nc-sa/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/el-khair-2016-1-5-billion-words-arabic/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/el-khair-2016-1-5-billion-words-arabic/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / 1.5 billion words Arabic Corpus # 1.5 billion words Arabic Corpus **Authors:** [Ibrahim Abu El-khair](https://md.research.elamur.ai/free/en/authors/ibrahim-abu-el-khair/) · **Year:** 2016 · **Venue:** arXiv:1611.04033 · **Licence:** [CC BY-NC-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-sa/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/1611.04033) · [PDF](https://arxiv.org/pdf/1611.04033) · [DOI 10.48550/arxiv.1611.04033](https://doi.org/10.48550/arxiv.1611.04033) · [arXiv 1611.04033](https://arxiv.org/abs/1611.04033) · [OpenAlex](https://openalex.org/W4297663785) **Citations:** 71 citations (OpenAlex, 2026-10-14) · [60](https://md.research.elamur.ai/free/en/papers/el-khair-2016-1-5-billion-words-arabic/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Arabic has few corpora that are both large and free. Most existing collections are small or sold through catalogs such as LDC and ELRA. Abu El-khair responds by compiling a free, contemporary Arabic news corpus. Its goal is broad coverage across countries, writing styles, outlets and years. He tested several web-copying tools and kept the two that ran reliably and removed non-text clutter. He then crawled ten online outlets spread across eight Arab countries; nine of them are newspapers and the remaining one is a news agency. The material spans roughly fourteen years. Sources had to meet technical criteria rather than popularity: no overlap with earlier corpora, a long online archive, permission to crawl, and text that can be edited. Every article carries a structured ID. The corpus is released in two markup styles, SGML in the TREC style and XML in the LDC style, and in two encodings, UTF-8 and Windows CP-1256. The result exceeds 1.5 billion words, making it larger than the commercial Arabic Gigaword and roughly double KACST, the biggest free alternative. It gives Arabic IR and NLP researchers a large, multi-topic resource at no cost. ## Key points - The final size is 5,222,973 articles, 1,525,722,252 words and 3,303,723 unique words. It occupies about 10GB in CP-1256 and 16GB in UTF-8. - The ten sources include Alriyadh, youm7, Alyaum, Alqabas, Alittihad, Almustaqbal, Tishreen, Almasryalyoum, Echorouk Online and Saba News Agency. Egypt and KSA contribute two outlets each. - Alriyadh supplies the most words (≈271M, 17.79%). - youm7 supplies the most articles (≈1.03M). - Crawling took place between December 2013 and June 2014. Tools tried and rejected were wget, httrack, Internet Download Manager and cyotek webcopy. The tools kept were MetaProducts Offline Explorer Pro and Visual Web Ripper. - Al-Ahram was left out on purpose because it already appears in Arabic Gigaword. Quality control found broken publication dates in Almustaqbal and Saba News, so those two sites were crawled again. - The paper compares the corpus with earlier Arabic corpora: - Arabic Gigaword 5th ed. has about 1.08B words and costs $6000. - KACST is free and has about 700M words. - ICA, OSAC, KSUCCA and several smaller corpora are also listed. ## Interesting ideas - Choosing sources partly because they do not overlap earlier corpora keeps the new resource independent of existing benchmarks and reduces hidden duplication. - Each document gets an ID built from source code, language code and serial number (e.g. RYD_ARB_0000001), which makes provenance readable straight from the identifier. - The same content is shipped in two markup conventions (TREC-style SGML and LDC-style XML), letting users plug it into whichever toolchain they already have. - Statistics are reported per source, and unique-word totals are deduplicated across sources rather than summed, which shows how much new vocabulary each source really adds. - A separate quality-control pass checked metadata such as publication dates and triggered a re-crawl when it found errors, instead of trusting the harvest as-is. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): Before silver data or a fable/test set enters training, check it for overlap with evaluation material, so that held-out scores mean something. - **I2** (learn): Give every silver sentence an ID that encodes its origin corpus and witness-parser set, so a bad repair rule can be traced back to where its data came from. - **I3** (tree): Export parsed treebanks in several UD dialects or formats from one internal source, so other tools and evaluators can consume them without manual conversion. - **I4** (tokens): Measure how many new lemmas or unknown forms each new text source adds after deduplication, to decide which genres best fill lexicon gaps. - **I5** (world): Validate dates and timestamps in ingested texts before they feed event ordering, and flag or re-fetch documents whose temporal metadata is missing or corrupt. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Umm al-Qura University](https://md.research.elamur.ai/free/en/universities/umm-al-qura-university/) — University, Makkah al Mukarramah, Saudi Arabia · 21 works · 274 citations - [Minia University](https://md.research.elamur.ai/free/en/universities/minia-university/) — University, Minya, Egypt · 3 works · 134 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [📚 Literature and Literary Theory](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/literature-and-literary-theory/) · 541 works 🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works ## Cite Cite as: Ibrahim Abu El-khair. 2016. 1.5 billion words Arabic Corpus. arXiv:1611.04033. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{elkhair20161, title = {{1.5 billion words Arabic Corpus}}, author = {Ibrahim Abu El-khair}, year = {2016}, doi = {10.48550/arxiv.1611.04033}, eprint = {1611.04033}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.1611.04033} } ``` - [← previous Dialogue Act Classification in Domain-Independent Conversations Using a Deep Recurrent Neural Network](https://md.research.elamur.ai/free/en/papers/khanpour-2016-dialogue-act-classification-domain-independent/) - [next → STransE: a novel embedding model of entities and relationships in knowledge bases](https://md.research.elamur.ai/free/en/papers/nguyen-2016-stranse-novel-embedding-model-entities/) - [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) ## References 9 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Abbas, M., & Smaili, K. (2005). Comparison of topic identification methods for Arabic language. Paper presented at the Proceedings of International Conference on Recent Advances in Natural Language Processing, RANLP. [find](https://scholar.google.com/scholar?q=Abbas%2C+M%2E%2C+%26+Smaili%2C+K%2E+%282005%29%2E+Comparison+of+topic+identification+methods+for+Arabic+language%2E+Paper+presented+at+the+Proceedings+of+International+Conference+on+Recent+Advances+in+Natural+Language+Pr) 2. Abbas, M., Smaïli, K., & Berkani, D. (2011). Evaluation of Topic Identification Methods on Arabic Corpora. JDIM, 9(5), 185-192. [find](https://scholar.google.com/scholar?q=Abbas%2C+M%2E%2C+Sma%C3%AFli%2C+K%2E%2C+%26+Berkani%2C+D%2E+%282011%29%2E+Evaluation+of+Topic+Identification+Methods+on+Arabic+Corpora%2E+JDIM%2C+9%285%29%2C+185%2D192%2E) 3. Abdelali, A., Cowie, J., & Soliman, H. (2005). Building a modern standard Arabic corpus. Paper presented at the workshop on computational modeling of lexical acquisition, the split meeting. Croatia, 25-28 July. [find](https://scholar.google.com/scholar?q=Abdelali%2C+A%2E%2C+Cowie%2C+J%2E%2C+%26+Soliman%2C+H%2E+%282005%29%2E+Building+a+modern+standard+Arabic+corpus%2E+Paper+presented+at+the+workshop+on+computational+modeling+of+lexical+acquisition%2C+the+split+meeting%2E+Croatia%2C+2) 4. Abu El-Khair, I. (2003). Effectiveness of document processing techniques for Arabic information retrieval. Ph.D. Dissertation University of Pittsburgh, USA. [find](https://scholar.google.com/scholar?q=Abu+El%2DKhair%2C+I%2E+%282003%29%2E+Effectiveness+of+document+processing+techniques+for+Arabic+information+retrieval%2E+Ph%2ED%2E+Dissertation+University+of+Pittsburgh%2C+USA%2E) 5. Abu El‐Khair, I. (2007). Arabic information retrieval. Annual review of information science and technology, 41(1), 505-533. [find](https://scholar.google.com/scholar?q=Abu+El%E2%80%90Khair%2C+I%2E+%282007%29%2E+Arabic+information+retrieval%2E+Annual+review+of+information+science+and+technology%2C+41%281%29%2C+505%2D533%2E) 6. Abu Salem, H. (1992). A microcomputer based Arabic bibliographic information retrieval system with relational thesauri (Arabic-IRS). Ph. D. Dissertation, Illinois Institute of Technology. [find](https://scholar.google.com/scholar?q=Abu+Salem%2C+H%2E+%281992%29%2E+A+microcomputer+based+Arabic+bibliographic+information+retrieval+system+with+relational+thesauri+%28Arabic%2DIRS%29%2E+Ph%2E+D%2E+Dissertation%2C+Illinois+Institute+of+Technology%2E) 7. Alansary, S., & Nagi, M. (2014). [The International Corpus of Arabic: Compilation, Analysis and Evaluation](https://md.research.elamur.ai/free/en/papers/alansary-2014-international-corpus-arabic-compilation-analysis/). ANLP 2014, 8. 8. Al-Hayat Arabic Corpus. (2001). European Language Resources Association, ELRA Catalog number ELRA-W0030. Retrieved 10/25/2015, from: [http://cata-log.elra.info/product_info.php?prod-ucts_id=632](http://cata-log.elra.info/product_info.php?prod-ucts_id=632) [↗](http://cata-log.elra.info/product_info.php?prod-ucts_id=632) 9. Alrabiah, M. (2012). King Saud University Standard Arabic Language Corpus. [In Arabic]. Retrieved 10/25/2015, from: [http://ksucorpus.ksu.edu.sa/ar](http://ksucorpus.ksu.edu.sa/ar) 10 Al-Sulaiti, L., & Atwell, E. S. (2006). The design of a corpus of contemporary Arabic. International Journal of Corpus Linguistics, 11(2), 135-171. 11 Al-Thubaity, A. O. (2014). A 700M+ Arabic corpus: KACST Arabic corpus design and construction. Language Resources and Evaluation, 1-31. 12 An-Nahar Newspaper Text Corpus. (2001). European Language Resources Association, ELRA Catalog number ELRA-W0027 Retrieved on: 10/25/2015, from: http://cata-log.elra.info/product_info.php?prod-ucts_id=767 13 Arabic Corpus Project. (2008). Knowledge Encyclopedia. [In Arabic]. Retrieved on: 10/25/2015, from: http://www.marefa.org/in-dex.php/ ةيبرعلا_ةريخذلا_عورشم 14 El-Haj, M., & Koulali, R. (2013). KALIMAT a multipurpose Arabic Corpus. Paper presented at the Second Workshop on Arabic Corpus Linguistics (WACL-2), UK. 15 Graff, D. (2003). Arabic Gigaword. Linguistic Data Consortium, Philadelphia. LDC catalog number LDC2003T12. Retrieved on: 10/25/2015, from: https://catalog.ldc.up-enn.edu/LDC2003T12 16 Graff, D. (2007). Arabic Gigaword Third Edition. Linguistic Data Consortium, Philadelphia. LDC catalog number LDC2007T40. Retrieved on: 10/25/2015, from: https://cata-log.ldc.upenn.edu/LDC2007T40 17 Graff, D., Chen, K., Kong, J., & Maeda, K. (2006). Arabic Gigaword Second Edition. Linguistic Data Consortium, Philadelphia. LDC catalog number LDC2006T02. Retrieved on: 10/25/2015, from: https://cata-log.ldc.upenn.edu/LDC2006T02 18 Graff, D., & Walker, K. (2001). Arabic newswire part 1. Linguistic Data Consortium, Philadelphia. LDC catalog number LDC2001T55. Retrieved on: 10/25/2015, from: https://catalog.ldc.up-enn.edu/LDC2001T55 19 Hammo, B., Al-Shargi, F., Yagi, S., & Obeid, N. (2013). Developing Tools for Arabic Corpus for Researchers. Paper presented at the Second Workshop on Arabic Corpus Linguistics (WACL-2), UK. 20 Hasnah, A. (1996). Full Text Processing and Retrieval: Weight Ranking, Text Structuring, and Passage Retrieval for Arabic Documents. Ph. D. Dissertation, Illinois Institute of Technology. 21 Hmeidi, I., Kanaan, G., & Evens, M. (1997). Design and implementation of automatic indexing for information retrieval with Arabic documents. JASIS, 48(10), 867-881. 22 Mansour, M. (2013). The absence of Arabic corpus linguistics: a call for creating an Arabic national corpus. International Journal of Humanities and Social Science, 3(12). 23 MEDAR Evaluation Package. (2010). European Language Resources Association, ELRA Catalog number ELRA-E0040 Retrieved on: 10/25/2015, from: http://cata-log.elra.info/product_info.php?prod-ucts_id=1166 24 NEMLAR Written Corpus. (2003). European Language Resources Association, ELRA Catalog number ELRA-W0042 Retrieved on: 10/25/2015, from: http://cata-log.elra.info/product_info.php?prod-ucts_id=873 25 Parker, R., Graff, D., Chen, K., Kong, J., & Maeda, K. (2009). Arabic Gigaword Fourth Edition. Linguistic Data Consortium, Philadelphia. LDC catalog number LDC2009T30. Retrieved on: 10/25/2015, from: https://cata-log.ldc.upenn.edu/LDC2009T30 26 Parker, R., Graff, D., Chen, K., Kong, J., & Maeda, K. (2011). Arabic Gigaword Fifth Edition. Linguistic Data Consortium, Philadelphia. LDC catalog number LDC2011T11. Retrieved on: 10/25/2015, from: https://cata-log.ldc.upenn.edu/LDC2011T11 27 Saad, M. K., & Ashour, W. (2010). OSAC: Open Source Arabic Corpora. Paper presented at the 6th International Symposium on Electrical and Electronics Engineering and Computer Science, Cyprus. 28 Saleh, Abdul Rahman Al-Haj. (2014). this is the Arabic Linguistic Corpus Project; and this is the Algerian perception of it. [In Arabic]. Algerian news today, 06.25.2014. Retrieved on: 10/25/2015, from: http://www.akhbarelyoum.dz/ar/200243/200 256/109357 29 Saleh, Mahmoud Ismail. (2014). Linguistics of Language Corpora: An Introduction to Arab readers. [In Arabic] Retrieved on: 10/25/2015, from: http://dr-mahmoud-ismail-saleh.blogspot.com/2014/04/blog-post_5.html 30 United Nations. (2015). Frequently Asked Questions (FAQs): official languages of the United Nations. Retrieved on: 25/10/2015, from: http://www.un.org/en/hq/dgacm/faqs.shtml 31 United Nations. (1973). United Nations Resolution number 3190 (D-28). Retrieved on: 25/10/2015, from: http://daccess-dds-ny.un.org/doc/RESOLU-TION/GEN/NR0/279/60/IMG/NR027960.pdf ?OpenElement 32 Zaghouani, W. (2014). Critical survey of the freely available Arabic corpora. Paper presented at the Proceedings of the Workshop on Free/Open-Source Arabic Corpora and Corpora Processing Tools. [↗](http://ksucorpus.ksu.edu.sa/ar) *Updated 2 November 2026*