---
title: "A System for Extracting Sentiment from Large-Scale Arabic"
description: "The paper describes an industrial IBM pipeline for sentiment mining over large volumes of Arabic… 23 references, cited by 2 papers here."
authors: ["Hao Wang", "Vijay R. Bommireddipalli", "Ayman Hanafy", "Mohamed Bahgat", "Sara Noeman", "Ossama S. Emam"]
year: 2015
venue: "arXiv:1511.04661"
source: "https://arxiv.org/abs/1511.04661"
doi: "10.48550/arxiv.1511.04661"
arxiv: "1511.04661"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2015-system-extracting-sentiment-large-scale/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2015-system-extracting-sentiment-large-scale/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/3/) / A System for Extracting Sentiment from Large-Scale Arabic Social Data

# A System for Extracting Sentiment from Large-Scale Arabic Social Data

**Authors:** [Hao Wang](https://md.research.elamur.ai/free/en/authors/hao-wang/), Vijay R. Bommireddipalli, Ayman Hanafy, Mohamed Bahgat, Sara Noeman and [Ossama S. Emam](https://md.research.elamur.ai/free/en/authors/ossama-emam/) · **Year:** 2015 · **Venue:** arXiv:1511.04661 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1511.04661) · [PDF](https://arxiv.org/pdf/1511.04661) · [DOI 10.48550/arxiv.1511.04661](https://doi.org/10.48550/arxiv.1511.04661) · [arXiv 1511.04661](https://arxiv.org/abs/1511.04661) · [OpenAlex](https://openalex.org/W2275421386)

**Citations:** 10 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/wang-2015-system-extracting-sentiment-large-scale/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper describes an industrial IBM pipeline for sentiment mining over large volumes of Arabic social media posts, mainly Twitter, written in Egyptian and Saudi dialects. These dialects have no fixed spelling, mix scripts, and differ from Modern Standard Arabic in morphology and word order. The system runs on IBM BigInsights, a Hadoop-based platform, inside the Social Data Analytics (SDA) product, which previously handled only English. Extraction is written as declarative rules in AQL and executed by SystemT. The authors chose rules over machine learning because domain experts and end users must be able to adapt, maintain and correct the extractor. The pipeline first normalizes characters and words, converts Romanized Arabic back to Arabic script, and then matches purpose-built sentiment lexicons with pattern rules. The system is deliberately tuned for high precision at moderate recall. Its output is joined with user profiles, such as country of origin, so that analysts can see how a specific group feels about a topic.

## Key points
- The platform stack is IBM BigInsights (Hadoop) → SDA → SystemT, with extractors written in AQL, a SQL-like declarative rule language whose cost-based optimizer selects the execution plan.
- Character normalization unifies the Alef variants to bare Alef and resolves Yeh/Alef Maksora and Heh/Teh Marbota confusions. It maps Urdu-style Heh (0x06C1) and Teh (0x06C3) to their Arabic equivalents and strips Tashkeel diacritics (0x064B–0x065F) and Kasheeda (0x0640).
- Letters repeated for emphasis are collapsed until the token matches a known word.
- Romanized "Arabizi" text (e.g., "e7na" → "we") is converted by a character-level model built like phrase-based MT. Spelling variants are mapped to one frequent dialect or MSA form.
- Results are broken down by polarity and originating country. A Saudi Employment use case showed weak positive-class precision, which the authors plan to fix by removing lexicon entries that cause false positives and adding domain rules.

## Interesting ideas
- Normalization runs before any analysis and maps many noisy spellings onto one canonical form, so that downstream lexicons and rules only need to know a single variant.
- Script transliteration is treated as a character-level translation problem, with many surface variants collapsing to a single target spelling.
- Extraction logic is written declaratively and an optimizer chooses how to execute it, which keeps the rules readable and editable by domain experts.
- The system aims for high precision over recall because its outputs drive real decisions. Errors are repaired by pruning the lexicon entries that cause false positives.
- Sentiment output is enriched with author attributes, turning isolated polarity labels into group-level insight.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): A small, explicit table that maps informal English spellings (elongations like "soooo", common misspellings) to canonical tokens would raise lemma coverage while keeping each mapping traceable.
- **I2** (tokens): A similar character-level mapping could restore leetspeak or phonetic chat spellings before tagging, and could log which variant was rewritten so the step stays explainable.
- **I3** (frames): Frame and speech-act patterns in prag could be stated as declarative queries over UD trees rather than hand-coded traversals, making them easier to audit and to extend per domain.
- **I4** (learn): Self-induced repair rules could track which lexicon or rule entry produced each false positive and retire the worst offenders, with precision as the gate for accepting a rule.
- **I5** (world): Attaching evaluations to the entities that hold them in the text world would let readers answer questions about who felt what, instead of reporting only a global polarity.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [IBM](https://md.research.elamur.ai/free/en/companies/ibm/) — Company, Armonk, United States · 139 works · 70,691 citations
- [IBM (Egypt)](https://md.research.elamur.ai/free/en/companies/ibm-egypt/) — Company, Giza, Egypt · 8 works · 301 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 870 works

🧭 Topics (2): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 972 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Hao Wang, Vijay R. Bommireddipalli, Ayman Hanafy, Mohamed Bahgat, Sara Noeman and Ossama S. Emam. 2015. A System for Extracting Sentiment from Large-Scale Arabic Social Data. arXiv:1511.04661.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wang2015system,
  title = {{A System for Extracting Sentiment from Large-Scale Arabic Social Data}},
  author = {Hao Wang, Vijay R. Bommireddipalli, Ayman Hanafy, Mohamed Bahgat, Sara Noeman and Ossama S. Emam},
  year = {2015},
  doi = {10.48550/arxiv.1511.04661},
  eprint = {1511.04661},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1511.04661}
}
```

- [← previous Analyzing and Visualizing Coreference Resolution Errors](https://md.research.elamur.ai/free/en/papers/martschat-2015-analyzing-visualizing-coreference-resolution-errors/)
- [next → CBAS: context based arabic stemmer](https://md.research.elamur.ai/free/en/papers/el-defrawy-2015-cbas-context-based-arabic-stemmer/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/3/)

## References

23 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Korayem, M., Crandall, D., Abdul-Mageed, M.: Subjectivity and sentiment analysis of arabic: A survey. Advanced Machine Learning Technologies and Applications, pp. 128-139. Springer (2012) [find](https://scholar.google.com/scholar?q=Korayem%2C+M%2E%2C+Crandall%2C+D%2E%2C+Abdul%2DMageed%2C+M%2E%3A+Subjectivity+and+sentiment+analysis+of+arabic%3A+A+survey%2E+Advanced+Machine+Learning+Technologies+and+Applications%2C+pp%2E+128%2D139%2E+Springer+%282012%29)
2. [http://bits.blogs.nytimes.com/2014/03/09/the-languages-of-twitter-users](http://bits.blogs.nytimes.com/2014/03/09/the-languages-of-twitter-users) [↗](http://bits.blogs.nytimes.com/2014/03/09/the-languages-of-twitter-users)
3. Pang, B., Lee, L.: Opinion mining and sentiment analysis. Foundations and trends in information retrieval 2, 1-135 (2008) [find](https://scholar.google.com/scholar?q=Pang%2C+B%2E%2C+Lee%2C+L%2E%3A+Opinion+mining+and+sentiment+analysis%2E+Foundations+and+trends+in+information+retrieval+2%2C+1%2D135+%282008%29)
4. Liu, B., Zhang, L.: A survey of opinion mining and sentiment analysis. Mining text data, pp. 415-463. Springer (2012) [find](https://scholar.google.com/scholar?q=Liu%2C+B%2E%2C+Zhang%2C+L%2E%3A+A+survey+of+opinion+mining+and+sentiment+analysis%2E+Mining+text+data%2C+pp%2E+415%2D463%2E+Springer+%282012%29)
5. Agarwal, A., Xie, B., Vovsha, I., Rambow, O., Passonneau, R.: [Sentiment analysis of twitter data](https://md.research.elamur.ai/free/en/papers/agarwal-2011-sentiment-analysis-twitter-data/). In: Proceedings of the Workshop on Languages in Social Media, pp. 30-38. (2011)
6. Neri, F., Aliprandi, C., Capeci, F., Cuadros, M., By, T.: Sentiment analysis on social media. In: Proceedings of the 2012 International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2012), pp. 919-926. IEEE Computer Society, (2012) [find](https://scholar.google.com/scholar?q=Neri%2C+F%2E%2C+Aliprandi%2C+C%2E%2C+Capeci%2C+F%2E%2C+Cuadros%2C+M%2E%2C+By%2C+T%2E%3A+Sentiment+analysis+on+social+media%2E+In%3A+Proceedings+of+the+2012+International+Conference+on+Advances+in+Social+Networks+Analysis+and+Mining+%28A)
7. Wang, H., Can, D., Kazemzadeh, A., Bar, F., Narayanan, S.: A system for real-time twitter sentiment analysis of 2012 us presidential election cycle. In: Proceedings of the ACL 2012 System Demonstrations, pp. 115-120. (2012) [find](https://scholar.google.com/scholar?q=Wang%2C+H%2E%2C+Can%2C+D%2E%2C+Kazemzadeh%2C+A%2E%2C+Bar%2C+F%2E%2C+Narayanan%2C+S%2E%3A+A+system+for+real%2Dtime+twitter+sentiment+analysis+of+2012+us+presidential+election+cycle%2E+In%3A+Proceedings+of+the+ACL+2012+System+Demonstratio)
8. El-Beltagy, S.R., Ali, A.: Open issues in the sentiment analysis of arabic social media: A case study. In: Innovations in Information Technology (IIT), 2013 9th International Conference on, pp. 215-220. (2013) [find](https://scholar.google.com/scholar?q=El%2DBeltagy%2C+S%2ER%2E%2C+Ali%2C+A%2E%3A+Open+issues+in+the+sentiment+analysis+of+arabic+social+media%3A+A+case+study%2E+In%3A+Innovations+in+Information+Technology+%28IIT%29%2C+2013+9th+International+Conference+on%2C+pp%2E+215%2D22)
9. Abdul-Mageed, M., Kübler, S., Diab, M.: [Samar: A system for subjectivity and sentiment analysis of arabic social media](https://md.research.elamur.ai/free/en/papers/mageed-2012-samar-system-subjectivity-sentiment-analysis/). In: Proceedings of the 3rd Workshop in Computational Approaches to Subjectivity and Sentiment Analysis, pp. 19-28. (2012)
10. Al-Subaihin, A.S., Al-Khalifa, H.S.: A System for Sentiment Analysis of Colloquial Arabic Using Human Computation. The Scientific World Journal 2014, (2014) [find](https://scholar.google.com/scholar?q=Al%2DSubaihin%2C+A%2ES%2E%2C+Al%2DKhalifa%2C+H%2ES%2E%3A+A+System+for+Sentiment+Analysis+of+Colloquial+Arabic+Using+Human+Computation%2E+The+Scientific+World+Journal+2014%2C+%282014%29)
11. Mourad, A., Darwish, K.: [Subjectivity and sentiment analysis of modern standard Arabic and Arabic microblogs](https://md.research.elamur.ai/free/en/papers/mourad-2013-subjectivity-sentiment-analysis-modern-standard/). In: Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pp. 55-64. (2013)
12. Abdul-Mageed, M., Diab, M.T., Korayem, M.: [Subjectivity and sentiment analysis of modern standard arabic](https://md.research.elamur.ai/free/en/papers/mageed-2011-subjectivity-sentiment-analysis-modern-standard/). In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pp. 587-591. (2011)
13. Abu-Jbara, A., King, B., Diab, M.T., Radev, D.R.: [Identifying Opinion Subgroups in Arabic Online Discussions](https://md.research.elamur.ai/free/en/papers/jbara-2013-identifying-opinion-subgroups-arabic-online/). In: ACL (2), pp. 829-835. (2013)
14. Shoukry, A., Rafea, A.: Sentence-level Arabic sentiment analysis. In: Collaboration Technologies and Systems (CTS), 2012 International Conference on, pp. 546-550. IEEE, (2012) [find](https://scholar.google.com/scholar?q=Shoukry%2C+A%2E%2C+Rafea%2C+A%2E%3A+Sentence%2Dlevel+Arabic+sentiment+analysis%2E+In%3A+Collaboration+Technologies+and+Systems+%28CTS%29%2C+2012+International+Conference+on%2C+pp%2E+546%2D550%2E+IEEE%2C+%282012%29)
15. Abdul-Mageed, M., Diab, M.: Toward building a large-scale Arabic sentiment lexicon. In: Proceedings of the 6th International Global WordNet Conference, pp. 18-22. (2012) [find](https://scholar.google.com/scholar?q=Abdul%2DMageed%2C+M%2E%2C+Diab%2C+M%2E%3A+Toward+building+a+large%2Dscale+Arabic+sentiment+lexicon%2E+In%3A+Proceedings+of+the+6th+International+Global+WordNet+Conference%2C+pp%2E+18%2D22%2E+%282012%29)
16. Refaee, E., Rieser, V.: [An Arabic twitter corpus for subjectivity and sentiment analysis](https://md.research.elamur.ai/free/en/papers/refaee-2014-arabic-twitter-corpus-subjectivity-sentiment/). In: Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland, may. European Language Resources Association (ELRA). (2014)
17. Abdul-Mageed, M., Diab, M.T.: [AWATIF: A Multi-Genre Corpus for Modern Standard Arabic Subjectivity and Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/mageed-2012-awatif-multi-genre-corpus-modern/). In: LREC, pp. 3907-3914. (2012)
18. Chiticariu, L., Krishnamurthy, R., Li, Y., Raghavan, S., Reiss, F.R., Vaithyanathan, S.: SystemT: an algebraic approach to declarative information extraction. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 128-137. (2010) [find](https://scholar.google.com/scholar?q=Chiticariu%2C+L%2E%2C+Krishnamurthy%2C+R%2E%2C+Li%2C+Y%2E%2C+Raghavan%2C+S%2E%2C+Reiss%2C+F%2ER%2E%2C+Vaithyanathan%2C+S%2E%3A+SystemT%3A+an+algebraic+approach+to+declarative+information+extraction%2E+In%3A+Proceedings+of+the+48th+Annual+Meetin)
19. Li, Y., Reiss, F.R., Chiticariu, L.: SystemT: a declarative information extraction system. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Systems Demonstrations, pp. 109-114. (2011) [find](https://scholar.google.com/scholar?q=Li%2C+Y%2E%2C+Reiss%2C+F%2ER%2E%2C+Chiticariu%2C+L%2E%3A+SystemT%3A+a+declarative+information+extraction+system%2E+In%3A+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Human+Language+T)
20. Fagin, R., Kimelfeld, B., Reiss, F., Vansummeren, S.: Spanners: a formal framework for information extraction. In: Proceedings of the 32nd symposium on Principles of database systems, pp. 37-48. (2013) [find](https://scholar.google.com/scholar?q=Fagin%2C+R%2E%2C+Kimelfeld%2C+B%2E%2C+Reiss%2C+F%2E%2C+Vansummeren%2C+S%2E%3A+Spanners%3A+a+formal+framework+for+information+extraction%2E+In%3A+Proceedings+of+the+32nd+symposium+on+Principles+of+database+systems%2C+pp%2E+37%2D48%2E+%282013)
21. Chiticariu, L., Li, Y., Reiss, F.R.: Rule-Based Information Extraction is Dead! Long Live Rule-Based Information Extraction Systems! In: EMNLP, pp. 827-832. (2013) [find](https://scholar.google.com/scholar?q=Chiticariu%2C+L%2E%2C+Li%2C+Y%2E%2C+Reiss%2C+F%2ER%2E%3A+Rule%2DBased+Information+Extraction+is+Dead%21+Long+Live+Rule%2DBased+Information+Extraction+Systems%21+In%3A+EMNLP%2C+pp%2E+827%2D832%2E+%282013%29)
22. Darwish, K., Magdy, W., Mourad, A.: Language processing for arabic microblog retrieval. In: Proceedings of the 21st ACM international conference on Information and knowledge management, pp. 2427-2430. ACM, (2012) [find](https://scholar.google.com/scholar?q=Darwish%2C+K%2E%2C+Magdy%2C+W%2E%2C+Mourad%2C+A%2E%3A+Language+processing+for+arabic+microblog+retrieval%2E+In%3A+Proceedings+of+the+21st+ACM+international+conference+on+Information+and+knowledge+management%2C+pp%2E+2427%2D2430%2E)
23. Koehn, P., Och, F.J., Marcu, D.: Statistical phrase-based translation. In: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pp. 48-54. Association for Computational Linguistics, (2003) [find](https://scholar.google.com/scholar?q=Koehn%2C+P%2E%2C+Och%2C+F%2EJ%2E%2C+Marcu%2C+D%2E%3A+Statistical+phrase%2Dbased+translation%2E+In%3A+Proceedings+of+the+2003+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics+on+Human+La)

*Updated 30 October 2026*