---
title: "Corpora Preparation and Stopword List Generation for Arabic"
description: "Arabic social media users mostly write in regional dialects, especially Egyptian, while existing Arabic stopword lists cover only… 25 references."
authors: ["Walaa Medhat", "Ahmed H. Yousef", "Hoda Korashy"]
year: 2014
venue: "arXiv:1410.1135"
source: "https://arxiv.org/abs/1410.1135"
doi: "10.48550/arxiv.1410.1135"
arxiv: "1410.1135"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/medhat-2014-corpora-preparation-stopword-list-generation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/medhat-2014-corpora-preparation-stopword-list-generation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/page/4/) / Corpora Preparation and Stopword List Generation for Arabic data in Social Network

# Corpora Preparation and Stopword List Generation for Arabic data in Social Network

**Authors:** Walaa Medhat, [Ahmed H. Yousef](https://md.research.elamur.ai/free/en/authors/ahmed-h-yousef/) and Hoda Korashy · **Year:** 2014 · **Venue:** arXiv:1410.1135 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1410.1135) · [PDF](https://arxiv.org/pdf/1410.1135) · [DOI 10.48550/arxiv.1410.1135](https://doi.org/10.48550/arxiv.1410.1135) · [arXiv 1410.1135](https://arxiv.org/abs/1410.1135) · [OpenAlex](https://openalex.org/W2218947495)

**Citations:** 5 citations (OpenAlex, 2026-10-14)

## Summary
Arabic social media users mostly write in regional dialects, especially Egyptian, while existing Arabic stopword lists cover only Modern Standard Arabic (MSA). The authors built three small sentiment corpora about four Egyptian films shown in August 2014, drawn from Twitter, Facebook comments and the review site elcinema.com. They cleaned the data in stages: removing link-only and spam posts, photo-only and mention-only comments, off-topic tweets and non-Arabic posts. They also converted Franco-arab (Arabic written in Latin letters) into Arabic script and translated embedded English words. Stopword candidates were taken from frequency counts, checked by hand against explicit criteria, and then expanded with Arabic prefixes and suffixes. The result was an Egyptian-dialect list and a purely corpus-based list. Using Naïve Bayes and Decision Tree classifiers with unigram and bigram features, the authors found that general lists that include dialect words work better than MSA-only lists. The corpus-based list lowered accuracy on the lexically richer reviews. This matters because dialect-heavy user text needs resources that MSA tools do not provide.

## Key points
- Facebook comments went from 1459 to 1415, then 1296, then 1261 across the cleaning stages. Tweets went from 1787 to 1069, then 862, then 781. The review corpus has only 32 items.
- Franco-arab makes up about 18% of the Facebook corpus and 3% of the Twitter corpus. It was converted with yamli.com, and the output was revised manually.
- Four stopword options were compared:
  - a merged MSA list built from three published lists;
  - a corpus-based list of the most frequent words;
  - an Egyptian-dialect list;
  - the dialect list combined with the MSA list.
- Decision Tree beat Naïve Bayes on these corpora, which the authors attribute to strong positive-class imbalance. Unigrams beat bigrams with Naïve Bayes, while the two feature types were roughly equal with Decision Tree.
- Training time was reported alongside accuracy. Removing stopwords shortened training, and Decision Tree was consistently slower than Naïve Bayes.

## Interesting ideas
- The cleaning pipeline is a funnel where each filter's surviving item count is recorded, so every data loss can be traced to a specific step.
- Function-word lists are built in three steps: take frequency candidates, keep only those passing explicit criteria (meaningless alone, frequent, not domain-bound), then expand survivors with their affixed variants.
- A list made only from corpus frequency hurt accuracy on lexically rich text, because domain-frequent words like "movie" still carry content.
- Non-standard script and code-switched words are normalized into one written form before any analysis, with a tool proposing candidates and a human confirming them.
- The authors explain the classifier ranking through class imbalance and report cost (training time) next to accuracy.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Silver-data builds could log how many sentences each agreement or quality filter discards, which makes data shrinkage auditable and helps catch filters that are too aggressive.
- **I2** (tokens): The en crate could compile closed-class lexicons (particles, contractions, clitic-like forms) the same way, with a written admission test and generated variants instead of an ad hoc word list.
- **I3** (world): Words that are frequent within a domain should be kept and marked as background topic entities, not dropped, so the world model still knows what the text is about.
- **I4** (tokens): Informal English input (texting spellings, phonetic forms) could get a reversible normalization layer, with candidates logged so that later tagging stays explainable.
- **I5** (learn): Evaluations could always report class balance, a majority-class baseline and runtime, so that a gain caused only by skewed labels is not mistaken for real progress.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Walaa Medhat, Ahmed H. Yousef and Hoda Korashy. 2014. Corpora Preparation and Stopword List Generation for Arabic data in Social Network. arXiv:1410.1135.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{medhat2014corpora,
  title = {{Corpora Preparation and Stopword List Generation for Arabic data in Social Network}},
  author = {Walaa Medhat, Ahmed H. Yousef and Hoda Korashy},
  year = {2014},
  doi = {10.48550/arxiv.1410.1135},
  eprint = {1410.1135},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1410.1135}
}
```

- [← previous BUAP: Evaluating Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment](https://md.research.elamur.ai/free/en/papers/leon-2014-buap-evaluating-compositional-distributional-semantic/)
- [next → MUHIT: A Multilingual Harmonized Dictionary](https://md.research.elamur.ai/free/en/papers/alansary-2014-muhit-multilingual-harmonized-dictionary/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/page/4/)

## References

25 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. H. a. H. K. Walaa Medhat, "Sentiment analysis algorithms and applications: A survey," Ain Shams Engineering Journal, 2014. [↗](https://doi.org/10.1016/j.asej.2014.04.011)
2. B. Pang, L. Lee, and S. Vaithyanathan, "Thumbs up?: sentimentclassification using machine learning techniques," in Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP-2002), 2002. [find](https://scholar.google.com/scholar?q=B%2E+Pang%2C+L%2E+Lee%2C+and+S%2E+Vaithyanathan%2C+%22Thumbs+up%3F%3A+sentimentclassification+using+machine+learning+techniques%2C%22+in+Proceedings+of+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%2D)
3. X. Bai, Padman, R. & Airoldi, "Sentiment Extraction from Unstructured Text Using Tabu Search -Enhanced Markov Blanket," Technical Report CMU-ISRI-04-127, 2004. [find](https://scholar.google.com/scholar?q=X%2E+Bai%2C+Padman%2C+R%2E+%26+Airoldi%2C+%22Sentiment+Extraction+from+Unstructured+Text+Using+Tabu+Search+%2DEnhanced+Markov+Blanket%2C%22+Technical+Report+CMU%2DISRI%2D04%2D127%2C+2004%2E)
4. M. Gamon, "Sentiment classification on customer feedback data: Noisy data, large feature vectors, and the role of linguistic analysis," in Proceedings of COLING'04, Geneva, Switzerland, 2004, pp. 841-847. [find](https://scholar.google.com/scholar?q=M%2E+Gamon%2C+%22Sentiment+classification+on+customer+feedback+data%3A+Noisy+data%2C+large+feature+vectors%2C+and+the+role+of+linguistic+analysis%2C%22+in+Proceedings+of+COLING%2704%2C+Geneva%2C+Switzerland%2C+2004%2C+pp%2E+841%2D)
5. K. Versteegh, Versteegh, C., "The Arabic Language," Columbia University Press, 1997. [find](https://scholar.google.com/scholar?q=K%2E+Versteegh%2C+Versteegh%2C+C%2E%2C+%22The+Arabic+Language%2C%22+Columbia+University+Press%2C+1997%2E)
6. N. Habash, "Introduction to Arabic natural language processing," Synthesis Lectures on Human Language Technologies, vol. 3, 2010. [↗](https://doi.org/10.1007/978-3-031-02139-8)
7. D. C. Mohammed Korayem, and Muhammad Abdul-Mageed2, "Subjectivity and Sentiment Analysis of Arabic: A Survey," AMLTA 2012, CCIS 322, pp. 128–139, 2012. [find](https://scholar.google.com/scholar?q=D%2E+C%2E+Mohammed+Korayem%2C+and+Muhammad+Abdul%2DMageed2%2C+%22Subjectivity+and+Sentiment+Analysis+of+Arabic%3A+A+Survey%2C%22+AMLTA+2012%2C+CCIS+322%2C+pp%2E+128%E2%80%93139%2C+2012%2E)
8. R. Al-Shalabi, Ghasan Kanaan, Jihad M. Jaam, Ahmad Hasnah, and Eyat Hilat, "Stop-word removal algorithm for Arabic language," in 1st International Conference on Information and Communication Technologies: From Theory to Applications, Damascus, 2004, pp. 545-550. [find](https://scholar.google.com/scholar?q=R%2E+Al%2DShalabi%2C+Ghasan+Kanaan%2C+Jihad+M%2E+Jaam%2C+Ahmad+Hasnah%2C+and+Eyat+Hilat%2C+%22Stop%2Dword+removal+algorithm+for+Arabic+language%2C%22+in+1st+International+Conference+on+Information+and+Communication+Technolog)
9. Ibrahim Abu El-Khair, "Effects of stop words elimination for Arabic information retrieval: a comparative study," International Journal of Computing & Information Sciences, vol. 4, pp. 119-133, 2006. [find](https://scholar.google.com/scholar?q=Ibrahim+Abu+El%2DKhair%2C+%22Effects+of+stop+words+elimination+for+Arabic+information+retrieval%3A+a+comparative+study%2C%22+International+Journal+of+Computing+%26+Information+Sciences%2C+vol%2E+4%2C+pp%2E+119%2D133%2C+2006%2E)
10. E. M. S. A. Alajmi, R. R. Darwish, "Toward an ARABIC Stop-Words List Generation," International Journal of Computer Applications, vol. 46, 2012. [find](https://scholar.google.com/scholar?q=E%2E+M%2E+S%2E+A%2E+Alajmi%2C+R%2E+R%2E+Darwish%2C+%22Toward+an+ARABIC+Stop%2DWords+List+Generation%2C%22+International+Journal+of+Computer+Applications%2C+vol%2E+46%2C+2012%2E)
11. L. H. a. B. D. Davison, "Empirical Study of Topic Modeling in Twitter," presented at the 1st Workshop on Social Media Analytics (SOMA ’10), Washington, DC, USA., 2010. [find](https://scholar.google.com/scholar?q=L%2E+H%2E+a%2E+B%2E+D%2E+Davison%2C+%22Empirical+Study+of+Topic+Modeling+in+Twitter%2C%22+presented+at+the+1st+Workshop+on+Social+Media+Analytics+%28SOMA+%E2%80%9910%29%2C+Washington%2C+DC%2C+USA%2E%2C+2010%2E)
12. A. B. a. E. Frank, "Sentiment Knowledge Discovery in Twitter Streaming Data," 2010. [find](https://scholar.google.com/scholar?q=A%2E+B%2E+a%2E+E%2E+Frank%2C+%22Sentiment+Knowledge+Discovery+in+Twitter+Streaming+Data%2C%22+2010%2E)
13. P. P. Alexander Pak, "Twitter Based System: Using Twitter for Disambiguating Sentiment Ambiguous Adjectives," in Proceedings of the 5th International Workshop on Semantic Evaluation, ACL 2010, Uppsala, Sweden, 2010, pp. 436–439. [find](https://scholar.google.com/scholar?q=P%2E+P%2E+Alexander+Pak%2C+%22Twitter+Based+System%3A+Using+Twitter+for+Disambiguating+Sentiment+Ambiguous+Adjectives%2C%22+in+Proceedings+of+the+5th+International+Workshop+on+Semantic+Evaluation%2C+ACL+2010%2C+Uppsala)
14. S. Z. Xiaohua Liu, Furu Wei , Ming Zhou "[Recognizing Named Entities in Tweets](https://md.research.elamur.ai/free/en/papers/liu-2011-recognizing-named-entities-tweets/)," in Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, Portland, Oregon, 2011, pp. 359–367.
15. M. Y. Long Jiang, Ming Zhou, Xiaohua Liu and Tiejun Zhao, "Target-dependent Twitter Sentiment Classification," in Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, Portland, Oregon, 2011, pp. 151–160. [find](https://scholar.google.com/scholar?q=M%2E+Y%2E+Long+Jiang%2C+Ming+Zhou%2C+Xiaohua+Liu+and+Tiejun+Zhao%2C+%22Target%2Ddependent+Twitter+Sentiment+Classification%2C%22+in+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Linguistic)
16. D. M. a. A. Funk, "Automatic detection of political opinions in Tweets," 2011. [find](https://scholar.google.com/scholar?q=D%2E+M%2E+a%2E+A%2E+Funk%2C+%22Automatic+detection+of+political+opinions+in+Tweets%2C%22+2011%2E)
17. J. G. Catie Meador, "Analyzing the Relationship Between Tweets, Box-Office Performance, and Stocks," 2010. [find](https://scholar.google.com/scholar?q=J%2E+G%2E+Catie+Meador%2C+%22Analyzing+the+Relationship+Between+Tweets%2C+Box%2DOffice+Performance%2C+and+Stocks%2C%22+2010%2E)
18. R. B. Alec Go, "Exploiting the Unique Characteristics of Tweets for Sentiment Analysis," 2010. [find](https://scholar.google.com/scholar?q=R%2E+B%2E+Alec+Go%2C+%22Exploiting+the+Unique+Characteristics+of+Tweets+for+Sentiment+Analysis%2C%22+2010%2E)
19. R. B. a. L. H. Alec Go, "Twitter Sentiment Classification using Distant Supervision," 2009. [find](https://scholar.google.com/scholar?q=R%2E+B%2E+a%2E+L%2E+H%2E+Alec+Go%2C+%22Twitter+Sentiment+Classification+using+Distant+Supervision%2C%22+2009%2E)
20. P. D. Mauro Cohen, Sebastian Durandeu, Renzo Navas, Hernán Merlino And Enrique Fernández, "Sentiment Analysis in Microblogging: A Practical Implementation," presented at the CACIC 2011 - XVII CONGRESO ARGENTINO DE CIENCIAS DE LA COMPUTACIÓN, Buenos Aires, Argentina, 2011. [find](https://scholar.google.com/scholar?q=P%2E+D%2E+Mauro+Cohen%2C+Sebastian+Durandeu%2C+Renzo+Navas%2C+Hern%C3%A1n+Merlino+And+Enrique+Fern%C3%A1ndez%2C+%22Sentiment+Analysis+in+Microblogging%3A+A+Practical+Implementation%2C%22+presented+at+the+CACIC+2011+%2D+XVII+CONGRESO)
21. A. H. a. H. K. Walaa Medhat, "A Framework of preparing corpora from Social Network sites for Sentiment Analysis " presented at the International Conference on Information Society (i-Society 2014), London, UK, 2014. [find](https://scholar.google.com/scholar?q=A%2E+H%2E+a%2E+H%2E+K%2E+Walaa+Medhat%2C+%22A+Framework+of+preparing+corpora+from+Social+Network+sites+for+Sentiment+Analysis+%22+presented+at+the+International+Conference+on+Information+Society+%28i%2DSociety+2014%29%2C+Lon)
22. K. N. A. McCallum, "[A Comparison of Event Models for Naive Bayes Text Classification](https://md.research.elamur.ai/free/en/papers/mccallum-1998-comparison-event-models-naive-bayes/)," presented at the AAAI Workshop on Learning for Text Categorization, 1998.
23. J. R. Quinlan, "Induction of Decision Trees," Machine Learning, vol. 1, pp. 81–106, 1986. [↗](https://doi.org/10.1007/bf00116251)
24. C. Z. Charu C. Aggarwal Ed., Mining Text Data. Springer New York Dordrecht Heidelberg London: © Springer Science+Business Media, LLC 2012, 2012, p.^pp. Pages. [find](https://scholar.google.com/scholar?q=C%2E+Z%2E+Charu+C%2E+Aggarwal+Ed%2E%2C+Mining+Text+Data%2E+Springer+New+York+Dordrecht+Heidelberg+London%3A+%C2%A9+Springer+Science%2BBusiness+Media%2C+LLC+2012%2C+2012%2C+p%2E%5Epp%2E+Pages%2E)
25. J. Perkins, Python Text Processing with NLTK 2.0 Cookbook. Birmingham: Packt Publishing Ltd., 2010. [find](https://scholar.google.com/scholar?q=J%2E+Perkins%2C+Python+Text+Processing+with+NLTK+2%2E0+Cookbook%2E+Birmingham%3A+Packt+Publishing+Ltd%2E%2C+2010%2E)

*Updated 1 November 2026*