---
title: "Evaluating Distant Supervision for Subjectivity and Sentiment"
description: "The paper asks whether noisy automatic labels can replace small hand-labelled sets for… 95% accuracy. 25 references, cited by 4 papers here."
authors: ["Eshrag Refaee", "Verena Rieser"]
year: 2014
venue: "Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP)"
source: "https://aclanthology.org/W14-3624/"
doi: "10.3115/v1/w14-3624"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/refaee-2014-evaluating-distant-supervision-subjectivity-sentiment/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/refaee-2014-evaluating-distant-supervision-subjectivity-sentiment/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/page/3/) / Evaluating Distant Supervision for Subjectivity and Sentiment Analysis on Arabic Twitter Feeds

# Evaluating Distant Supervision for Subjectivity and Sentiment Analysis on Arabic Twitter Feeds

**Authors:** [Eshrag Refaee](https://md.research.elamur.ai/free/en/authors/eshrag-refaee/) and [Verena Rieser](https://md.research.elamur.ai/free/en/authors/verena-rieser/) · **Year:** 2014 · **Venue:** [Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP)](https://md.research.elamur.ai/free/en/venues/emnlp-workshop-on-arabic-natural-language-processing-anlp/) ([ANLP 2014](https://md.research.elamur.ai/free/en/venues/emnlp-workshop-on-arabic-natural-language-processing-anlp/2014/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W14-3624/) · [PDF](https://aclanthology.org/W14-3624.pdf) · [DOI 10.3115/v1/w14-3624](https://doi.org/10.3115/v1/w14-3624) · [OpenAlex](https://openalex.org/W2148263003)

**Citations:** 23 citations (OpenAlex, 2026-10-14) · [4](https://md.research.elamur.ai/free/en/papers/refaee-2014-evaluating-distant-supervision-subjectivity-sentiment/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether noisy automatic labels can replace small hand-labelled sets for subjectivity and sentiment analysis (SSA) of Arabic tweets. The motivation is that classifiers trained on limited manual data break down when topics shift over time. The authors collected tweets across a year:
- a manually annotated gold set (3,031 training tweets plus a 1,580-tweet test set gathered later);
- an emoticon-labelled distant supervision (DS) corpus of about 184K tweets;
- a lexicon-labelled DS corpus of about 134K tweets.

In both DS corpora, neutral examples came from Arabic news accounts. SVM classifiers built in WEKA used bag-of-words, morphological and semantic features. They were tested on the later-collected set.

For polar vs. neutral, DS reached about 95% accuracy, while the fully supervised model stayed below 50%. For positive vs. negative, every system hovered near chance. Lexicon-presence labelling did best at 57%. Negative tweets were often predicted as positive. An error analysis points to sarcasm, mixed sentiment, and emoticons flipped by right-to-left typing. The work matters as the first empirical test of DS for Arabic Twitter SSA, and the authors plan to release the corpora.

## Key points
- Corpora: the gold training set has 3,031 multi-dialectal tweets (Feb–Mar 2013), and the held-out test set has 1,580 tweets (autumn 2013). Two native speakers annotated them, with weighted κ = 0.76. Features were extracted with MADA.
- Emoticon DS corpus: 184,013 tweets, labelled via emoticons and some sentiment hashtags. Neutral tweets came from accounts such as BBC-Arabic, Al-Jazeera Arabic and Reuters Arabic.
- Lexicon DS corpus: 134,069 tweets, labelled with a merged lexicon of 4,422 words. The sources were:
  - the Arabic lexicon of Abdul-Mageed and Diab;
  - MPQA machine-translated by Google Translate and hand-cleaned to 2,627 entries;
  - 484 hand-labelled dialectal words.

  Lexicon coverage was 76.62% of tweets, and the automatic labels agreed with a manual dev set at 69.06%.
- Polar vs. neutral accuracy:

  | System | Accuracy |
  |---|---|
  | Emoticon DS | 95.19% |
  | Lexicon-presence | 95.09% |
  | Lexicon-aggregation | 91.09% |
  | Fully supervised | 46.62% |
  | Majority baseline | 53.0% |

- Positive vs. negative accuracy:

  | System | Accuracy |
  |---|---|
  | Lexicon-presence | 57.06% (significantly above emoticon DS) |
  | Lexicon-aggregation | 52.98% |
  | Emoticon DS | 51.25% |
  | Majority baseline | 50.89% |
  | Fully supervised | 49.65% |

## Interesting ideas
- Build the test set from data collected months after the training data, so the score measures robustness to topic drift rather than cross-validation fit.
- Remove the very cues used to assign noisy labels (emoticons, hashtags, matched lexicon words become placeholders), so the learner cannot simply recover the labelling rule.
- Before trusting an automatic labeller, check it against a small hand-labelled dev set (here about 69% agreement). This quantifies how much noise enters the training data.
- Keep only unambiguous instances (all cues point the same way) and drop mixed ones. This trades coverage for cleaner labels and beat score aggregation.
- Inspect misclassified items and tag them for sarcasm, mixed sentiment, or surface-cue inversion, such as emoticons mirrored in right-to-left typing. This shows why surface signals mislead.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Evaluate parsers and readers on texts from a different source or period than the training silver data, as a guard against overfitting to one corpus.
- **I2** (learn): When silver labels come from a rule or a witness parser, mask the triggering evidence during training so the model learns from the rest of the context.
- **I3** (learn): Give every silver-data pipeline a measured agreement score against a gold sample, and compare it with negative controls.
- **I4** (learn): When witness parsers disagree, discard the item instead of averaging votes; the remaining data is smaller but more reliable.
- **I5** (frames): The irony and implicature component could flag a clash between explicit polarity markers and event content, as in "no hope anymore :)", instead of trusting the marker.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Heriot-Watt University](https://md.research.elamur.ai/free/en/universities/heriot-watt-university/) — University, Edinburgh, United Kingdom · 15 works · 5,365 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 870 works

🧭 Topics (3): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 972 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Eshrag Refaee and Verena Rieser. 2014. Evaluating Distant Supervision for Subjectivity and Sentiment Analysis on Arabic Twitter Feeds. Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{refaee2014evaluating,
  title = {{Evaluating Distant Supervision for Subjectivity and Sentiment Analysis on Arabic Twitter Feeds}},
  author = {Eshrag Refaee and Verena Rieser},
  booktitle = {Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP)},
  year = {2014},
  doi = {10.3115/v1/w14-3624},
  url = {https://doi.org/10.3115/v1/w14-3624}
}
```

- [← previous A Human Judgement Corpus and a Metric for Arabic MT Evaluation](https://md.research.elamur.ai/free/en/papers/bouamor-2014-human-judgement-corpus-metric-arabic/)
- [next → Pivot-based multilingual dictionary building using Wiktionary](https://md.research.elamur.ai/free/en/papers/acs-2014-wikt2dict-pivot/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/page/3/)

## References

25 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Muhammad Abdul-Mageed and Mona Diab. 2012. [AWATIF: A multi-genre corpus for modern standard Arabic subjectivity and sentiment analysis](https://md.research.elamur.ai/free/en/papers/mageed-2012-awatif-multi-genre-corpus-modern/). In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey. European Language Resources Association (ELRA).
2. Muhammad Abdul-Mageed, Mona T. Diab, and Mohammed Korayem. 2011. [Subjectivity and sentiment analysis of modern standard Arabic](https://md.research.elamur.ai/free/en/papers/mageed-2011-subjectivity-sentiment-analysis-modern-standard/). In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers - Volume 2, HLT ’11, pages 587–591, Stroudsburg, PA, USA. Association for Computational Linguistics.
3. Muhammad Abdul-Mageed, Sandra Kuebler, and Mona Diab. 2012. [SAMAR: A system for subjectivity and sentiment analysis of Arabic social media](https://md.research.elamur.ai/free/en/papers/mageed-2012-samar-system-subjectivity-sentiment-analysis/). In Proceedings of the 3rd Workshop in Computational Approaches to Subjectivity and Sentiment Analysis, pages 19–28. Association for Computational Linguistics.
4. Rania Al-Sabbagh and Roxana Girju. 2012. [YADAC: Yet another dialectal Arabic corpus](https://md.research.elamur.ai/free/en/papers/sabbagh-2012-yadac-yet-another-dialectal-arabic/). In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey. European Language Resources Association (ELRA).
5. Albert Bifet and Eibe Frank. 2010. Sentiment knowledge discovery in twitter streaming data. In Discovery Science, pages 1–15. Springer. [find](https://scholar.google.com/scholar?q=Albert+Bifet+and+Eibe+Frank%2E+2010%2E+Sentiment+knowledge+discovery+in+twitter+streaming+data%2E+In+Discovery+Science%2C+pages+1%E2%80%9315%2E+Springer%2E)
6. J. Carletta. 1996. [Assessing agreement on classification tasks: the kappa statistic](https://md.research.elamur.ai/free/en/papers/carletta-1996-assessing-agreement-classification-tasks-kappa/). Computational Linguistics, 22(2):249–254. [↗](https://doi.org/10.48550/arxiv.cmp-lg/9602004)
7. Jacob Eisenstein. 2013. [What to do about bad language on the internet](https://md.research.elamur.ai/free/en/papers/eisenstein-2013-what-do-about-bad-language/). In Proceedings of NAACL-HLT, pages 359–369.
8. Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter sentiment classification using distant supervision. CS224N Project Report, Stanford, pages 1–12. [find](https://scholar.google.com/scholar?q=Alec+Go%2C+Richa+Bhayani%2C+and+Lei+Huang%2E+2009%2E+Twitter+sentiment+classification+using+distant+supervision%2E+CS224N+Project+Report%2C+Stanford%2C+pages+1%E2%80%9312%2E)
9. Lichan Hong, Gregorio Convertino, and Ed H Chi. 2011. Language matters in twitter: A large scale study. In ICWSM. [find](https://scholar.google.com/scholar?q=Lichan+Hong%2C+Gregorio+Convertino%2C+and+Ed+H+Chi%2E+2011%2E+Language+matters+in+twitter%3A+A+large+scale+study%2E+In+ICWSM%2E)
10. Ahmed Mourad and Kareem Darwish. 2013. [Subjectivity and sentiment analysis of modern standard Arabic and Arabic microblogs](https://md.research.elamur.ai/free/en/papers/mourad-2013-subjectivity-sentiment-analysis-modern-standard/). WASSA 2013, page 55.
11. Owen Rambow Nizar Habash and Ryan Roth. 2009. MADA+TOKAN: A toolkit for Arabic tokenization, diacritization, morphological disambiguation, pos tagging, stemming and lemmatization. In Khalid Choukri and Bente Maegaard, editors, Proceedings of the Second International Conference on Arabic Language Resources and Tools, Cairo, Egypt, April. The MEDAR Consortium. [find](https://scholar.google.com/scholar?q=Owen+Rambow+Nizar+Habash+and+Ryan+Roth%2E+2009%2E+MADA%2BTOKAN%3A+A+toolkit+for+Arabic+tokenization%2C+diacritization%2C+morphological+disambiguation%2C+pos+tagging%2C+stemming+and+lemmatization%2E+In+Khalid+Choukri+an)
12. A. Pak and P. Paroubek. 2010. [Twitter as a corpus for sentiment analysis and opinion mining](https://md.research.elamur.ai/free/en/papers/pak-2010-twitter-as-corpus-sentiment-analysis/). In Proceedings of LREC.
13. Matthew Purver and Stuart Battersby. 2012. Experimenting with distant supervision for emotion classification. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 482–491, Avignon, France, April. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Matthew+Purver+and+Stuart+Battersby%2E+2012%2E+Experimenting+with+distant+supervision+for+emotion+classification%2E+In+Proceedings+of+the+13th+Conference+of+the+European+Chapter+of+the+Association+for+Compu)
14. Jonathon Read. 2005. Using emoticons to reduce dependency in machine learning techniques for sentiment classification. In Proceedings of the ACL Student Research Workshop, pages 43–48. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jonathon+Read%2E+2005%2E+Using+emoticons+to+reduce+dependency+in+machine+learning+techniques+for+sentiment+classification%2E+In+Proceedings+of+the+ACL+Student+Research+Workshop%2C+pages+43%E2%80%9348%2E+Association+for)
15. Eshrag Refaee and Verena Rieser. 2014. [An Arabic twitter corpus for subjectivity and sentiment analysis](https://md.research.elamur.ai/free/en/papers/refaee-2014-arabic-twitter-corpus-subjectivity-sentiment/). In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland, may. European Language Resources Association (ELRA).
16. Asad Sayeed. 2013. An opinion about opinions about opinions: subjectivity and the aggregate reader. In Proceedings of NAACL-HLT, pages 691–696. [find](https://scholar.google.com/scholar?q=Asad+Sayeed%2E+2013%2E+An+opinion+about+opinions+about+opinions%3A+subjectivity+and+the+aggregate+reader%2E+In+Proceedings+of+NAACL%2DHLT%2C+pages+691%E2%80%93696%2E)
17. Jared Suttles and Nancy Ide. 2013. Distant supervision for emotion classification with discrete binary values. In Computational Linguistics and Intelligent Text Processing, pages 121–136. Springer. [find](https://scholar.google.com/scholar?q=Jared+Suttles+and+Nancy+Ide%2E+2013%2E+Distant+supervision+for+emotion+classification+with+discrete+binary+values%2E+In+Computational+Linguistics+and+Intelligent+Text+Processing%2C+pages+121%E2%80%93136%2E+Springer%2E)
18. Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede. 2011. [Lexicon-based methods for sentiment analysis](https://md.research.elamur.ai/free/en/papers/taboada-2011-lexicon-based-methods-sentiment-analysis/). Computational linguistics, 37(2):267–307. [↗](https://doi.org/10.1162/coli_a_00049)
19. Janyce M. Wiebe, Rebecca F. Bruce, and Thomas P. O’Hara. 1999. [Development and use of a gold-standard data set for subjectivity classifications](https://md.research.elamur.ai/free/en/papers/wiebe-1999-development-use-gold-standard-data/). In Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, ACL ’99, pages 246–253, Stroudsburg, PA, USA. Association for Computational Linguistics.
20. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2009. [Recognizing contextual polarity: An exploration of features for phrase-level sentiment analysis](https://md.research.elamur.ai/free/en/papers/wilson-2009-recognizing-contextual-polarity-exploration-features/). Computational Linguistics, 35(3):399–433. [↗](https://doi.org/10.1162/coli.08-012-r1-06-90)
21. Ian H Witten and Eibe Frank. 2005. Data Mining: Practical machine learning tools and techniques. Morgan Kaufmann. [find](https://scholar.google.com/scholar?q=Ian+H+Witten+and+Eibe+Frank%2E+2005%2E+Data+Mining%3A+Practical+machine+learning+tools+and+techniques%2E+Morgan+Kaufmann%2E)
22. Alecia Wolf. 2000. Emotional expression online: Gender differences in emoticon use. CyberPsychology & Behavior, 3(5):827–833. [↗](https://doi.org/10.1089/10949310050191809)
23. Zheng Yuan and Matthew Purver. 2012. Predicting emotion labels for chinese microblog texts. In Proceedings of the 1st International Workshop on Sentiment Discovery from Affective Data (SDAD), pages 40–47, Bristol, UK, September. [find](https://scholar.google.com/scholar?q=Zheng+Yuan+and+Matthew+Purver%2E+2012%2E+Predicting+emotion+labels+for+chinese+microblog+texts%2E+In+Proceedings+of+the+1st+International+Workshop+on+Sentiment+Discovery+from+Affective+Data+%28SDAD%29%2C+pages+40)
24. Omar F. Zaidan and Chris Callison-Burch. 2013. [Arabic dialect identification](https://md.research.elamur.ai/free/en/papers/zaidan-2014-arabic-dialect-identification/). Computational Linguistics. [↗](https://doi.org/10.1162/coli_a_00169)
25. Ley Zhang, Riddhiman Ghosh, Mohamed Dekhil, Meichun Hsu, and Bing Liu. 2011. Combining lex-iconbased and learning-based methods for twitter sentiment analysis. HP Laboratories, Technical Report HPL-2011, 89. [find](https://scholar.google.com/scholar?q=Ley+Zhang%2C+Riddhiman+Ghosh%2C+Mohamed+Dekhil%2C+Meichun+Hsu%2C+and+Bing+Liu%2E+2011%2E+Combining+lex%2Diconbased+and+learning%2Dbased+methods+for+twitter+sentiment+analysis%2E+HP+Laboratories%2C+Technical+Report+HPL%2D20)

*Updated 1 November 2026*