---
title: "Annotating Targets of Opinions in Arabic using Crowdsourcing"
description: "The authors replace a single hard judgement with two simple crowdsourced steps on Amazon… 25 references, cited by 4 papers here."
authors: ["Noura Farra", "Kathy McKeown", "Nizar Habash"]
year: 2015
venue: "Proceedings of the Second Workshop on Arabic Natural Language Processing"
source: "https://aclanthology.org/W15-3210/"
doi: "10.18653/v1/w15-3210"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/farra-2015-annotating-targets-opinions-arabic-crowdsourcing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/farra-2015-annotating-targets-opinions-arabic-crowdsourcing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/3/) / Annotating Targets of Opinions in Arabic using Crowdsourcing

# Annotating Targets of Opinions in Arabic using Crowdsourcing

**Authors:** [Noura Farra](https://md.research.elamur.ai/free/en/authors/noura-farra/), [Kathy McKeown](https://md.research.elamur.ai/free/en/authors/kathy-mckeown/) and [Nizar Habash](https://md.research.elamur.ai/free/en/authors/nizar-habash/) · **Year:** 2015 · **Venue:** [Proceedings of the Second Workshop on Arabic Natural Language Processing](https://md.research.elamur.ai/free/en/venues/workshop-on-arabic-natural-language-processing/) ([Workshop on Arabic Natural Language Processing 2015](https://md.research.elamur.ai/free/en/venues/workshop-on-arabic-natural-language-processing/2015/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W15-3210/) · [PDF](https://aclanthology.org/W15-3210.pdf) · [DOI 10.18653/v1/w15-3210](https://doi.org/10.18653/v1/w15-3210) · [OpenAlex](https://openalex.org/W2250979019)

**Citations:** 27 citations (OpenAlex, 2026-10-14) · [4](https://md.research.elamur.ai/free/en/papers/farra-2015-annotating-targets-opinions-arabic-crowdsourcing/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses the lack of fine-grained opinion resources for Arabic. No public dataset marked *what* an opinion is about, and earlier fine-grained corpora in other languages relied on slow, costly expert annotators. The authors replace a single hard judgement with two simple crowdsourced steps on Amazon Mechanical Turk. First, workers list the important noun-phrase entities in a comment. Second, workers judge the sentiment the writer expresses toward each entity as positive, negative or neutral. They applied this pipeline to online news-article comments in politics, culture and sports. This genre is noisy, with dialect, weak grammar and misspellings, and topic modeling was used to find the less common domains. Despite the subjectivity of the task, annotators reached majority agreement on polarity for most entities. The authors also found that Arabic morphology and syntax make target boundaries especially variable. The result is a freely downloadable target-and-polarity corpus for Arabic, and the study shows that cheap non-expert workers can produce usable fine-grained sentiment data.

## Key points
- The corpus contains 1177 comments: 596 politics (51%), 382 culture (32%) and 199 sports (17%).
- Sentiment reached majority agreement for 91.8% of the annotated entities.
- In Task 1, three workers list entities. Only phrases named by all three are kept, and when workers overlap partially, the longest agreed phrase is chosen.
- Pilot-driven instructions to list only the *main* entities cut the average from 8 to 6 entities per comment. Task 1 paid 30 cents and took about 2–3 minutes.
- Guidelines were written in MSA so that only Arabic speakers could take part. The opinion holder is assumed to be the comment's author, and targets are not linked to specific opinion expressions.

## Interesting ideas
- Break a hard judgement into two easy questions, "what entities are here?" and then "how does the writer feel about this one?", so that each answer is simple and cheap to check.
- Merge several annotators by keeping only the items all of them named, and resolve partial span overlaps by taking the maximal agreed phrase.
- Treat the opinion target as a real-world entity rather than as the full text span, as when praise of a large turnout is really about the protest.
- Assign polarity per entity, defaulting the holder to the author in commentary, instead of one overall document label.
- Use a pilot round to discover over-generation and fragmented spans, then tighten the instructions before the full run.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): LLM teacher tiers could be asked staged, narrow questions instead of one complex labelling prompt, which makes their silver labels easier to cross-check.
- **I2** (learn): the same intersect-then-maximal-span rule could reconcile entity or argument spans proposed by multiple witness parsers when building silver data.
- **I3** (world): evaluations could be attached to entity nodes in the world of the text, not to surface spans, so different mentions of one entity gather its sentiment.
- **I4** (frames): the prag crate could add an attitude slot (holder, target, polarity) to frames, with the writer as the default holder in opinion genres.
- **I5** (learn): before scaling a teacher prompt or repair rule, run a small pilot and measure how many items it over-produces, as a cheap control for spurious output.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Columbia University](https://md.research.elamur.ai/free/en/universities/columbia-university/) — University, New York, United States · 241 works · 54,053 citations
- [New York University Abu Dhabi](https://md.research.elamur.ai/free/en/universities/new-york-university-abu-dhabi/) — University, Abu Dhabi, United Arab Emirates · 122 works · 4,602 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Noura Farra, Kathy McKeown and Nizar Habash. 2015. Annotating Targets of Opinions in Arabic using Crowdsourcing. Proceedings of the Second Workshop on Arabic Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{farra2015annotating,
  title = {{Annotating Targets of Opinions in Arabic using Crowdsourcing}},
  author = {Noura Farra, Kathy McKeown and Nizar Habash},
  booktitle = {Proceedings of the Second Workshop on Arabic Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/w15-3210},
  url = {https://doi.org/10.18653/v1/w15-3210}
}
```

- [← previous The Influence of Context on Dialogue Act Recognition](https://md.research.elamur.ai/free/en/papers/ribeiro-2015-influence-context-dialogue-act-recognition/)
- [next → Benchmarking Machine Translated Sentiment Analysis for Arabic Tweets](https://md.research.elamur.ai/free/en/papers/refaee-2015-benchmarking-machine-translated-sentiment-analysis/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/3/)

## References

25 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Muhammad Abdul-Mageed and Mona T Diab. 2011. [Subjectivity and sentiment annotation of modern standard Arabic newswire](https://md.research.elamur.ai/free/en/papers/mageed-2011-subjectivity-sentiment-annotation-modern-standard/). In Proceedings of the 5th Linguistic Annotation Workshop, pages 110– 118. Association for Computational Linguistics.
2. David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. [Latent dirichlet allocation](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/). the Journal of machine Learning research, 3:993–1022.
3. Paula Carvalho, Luís Sarmento, Jorge Teixeira, and Mário J Silva. 2011. Liars and saviors in a sentiment annotated corpus of comments to political debates. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pages 564–568. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Paula+Carvalho%2C+Lu%C3%ADs+Sarmento%2C+Jorge+Teixeira%2C+and+M%C3%A1rio+J+Silva%2E+2011%2E+Liars+and+saviors+in+a+sentiment+annotated+corpus+of+comments+to+political+debates%2E+In+Proceedings+of+the+49th+Annual+Meeting+of)
4. Lingjia Deng and Janyce Wiebe. 2015. Mpqa 3.0: An entity/event-level sentiment corpus. [find](https://scholar.google.com/scholar?q=Lingjia+Deng+and+Janyce+Wiebe%2E+2015%2E+Mpqa+3%2E0%3A+An+entity%2Fevent%2Dlevel+sentiment+corpus%2E)
5. Michael Denkowski, Hassan Al-Haj, and Alon Lavie. 2010. [Turker-assisted paraphrasing for English-Arabic machine translation](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pages 66–70. Association for Computational Linguistics.
6. Spence Green and Christopher D Manning. 2010. [Better Arabic parsing: Baselines, evaluations, and analysis](https://md.research.elamur.ai/free/en/papers/green-2010-better-arabic-parsing-baselines-evaluations/). In Proceedings of the 23rd International Conference on Computational Linguistics, pages 394– 402. Association for Computational Linguistics.
7. Nizar Y Habash. 2010. Introduction to Arabic natural language processing. Synthesis Lectures on Human Language Technologies, 3(1):1–187. [↗](https://doi.org/10.1007/978-3-031-02139-8)
8. Chiara Higgins, Elizabeth McGrath, and Lailla Moretto. 2010. [MTurk crowdsourcing: a viable method for rapid discovery of Arabic nicknames?](https://md.research.elamur.ai/free/en/papers/higgins-2010-mturk-crowdsourcing-viable-method-rapid/) In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pages 89–92. Association for Computational Linguistics.
9. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168–177. ACM. [find](https://scholar.google.com/scholar?q=Minqing+Hu+and+Bing+Liu%2E+2004%2E+Mining+and+summarizing+customer+reviews%2E+In+Proceedings+of+the+tenth+ACM+SIGKDD+international+conference+on+Knowledge+discovery+and+data+mining%2C+pages+168%E2%80%93177%2E+ACM%2E)
10. Long Jiang, Mo Yu, Ming Zhou, Xiaohua Liu, and Tiejun Zhao. 2011. Target-dependent twitter sentiment classification. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 151–160. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Long+Jiang%2C+Mo+Yu%2C+Ming+Zhou%2C+Xiaohua+Liu%2C+and+Tiejun+Zhao%2E+2011%2E+Target%2Ddependent+twitter+sentiment+classification%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Ling)
11. Soo-Min Kim and Eduard Hovy. 2006. [Extracting opinions, opinion holders, and topics expressed in online news media text](https://md.research.elamur.ai/free/en/papers/kim-2006-extracting-opinions-opinion-holders-topics/). In Proceedings of the Workshop on Sentiment and Subjectivity in Text, pages 1– 8. Association for Computational Linguistics.
12. Nolan Lawson, Kevin Eustice, Mike Perkowitz, and Meliha Yetisgen-Yildiz. 2010. [Annotating large email datasets for named entity recognition with mechanical turk](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pages 71–79. Association for Computational Linguistics.
13. Andrew K McCallum. 2002. {MALLET: A Machine Learning for Language Toolkit}. [find](https://scholar.google.com/scholar?q=Andrew+K+McCallum%2E+2002%2E+%7BMALLET%3A+A+Machine+Learning+for+Language+Toolkit%7D%2E)
14. Behrang Mohit, Alla Rozovskaya, Nizar Habash, Wajdi Zaghouani, and Ossama Obeid. 2014. [The first QALB shared task on automatic text correction for Arabic](https://md.research.elamur.ai/free/en/papers/mohit-2014-first-qalb-shared-task-automatic/). In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 39–47, Doha, Qatar, October. Association for Computational Linguistics.
15. Maria Pontiki, Haris Papageorgiou, Dimitrios Galanis, Ion Androutsopoulos, John Pavlopoulos, and Suresh Manandhar. 2014. [Semeval-2014 task 4: Aspect based sentiment analysis](https://md.research.elamur.ai/free/en/papers/pontiki-2014-semeval-2014-task-4-aspect/). In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 27–35.
16. Guang Qiu, Bing Liu, Jiajun Bu, and Chun Chen. 2011. [Opinion word expansion and target extraction through double propagation](https://md.research.elamur.ai/free/en/papers/qiu-2011-opinion-word-expansion-target-extraction/). Computational linguistics, 37(1):9–27. [↗](https://doi.org/10.1162/coli_a_00034)
17. Josef Ruppenhofer, Swapna Somasundaran, and Janyce Wiebe. 2008. Finding the sources and targets of subjective expressions. In LREC. [find](https://scholar.google.com/scholar?q=Josef+Ruppenhofer%2C+Swapna+Somasundaran%2C+and+Janyce+Wiebe%2E+2008%2E+Finding+the+sources+and+targets+of+subjective+expressions%2E+In+LREC%2E)
18. Swapna Somasundaran and Janyce Wiebe. 2009. Recognizing stances in online debates. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-Volume 1, pages 226–234. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Swapna+Somasundaran+and+Janyce+Wiebe%2E+2009%2E+Recognizing+stances+in+online+debates%2E+In+Proceedings+of+the+Joint+Conference+of+the+47th+Annual+Meeting+of+the+ACL+and+the+4th+International+Joint+Conferen)
19. Swapna Somasundaran, Josef Ruppenhofer, and Janyce Wiebe. 2008. Discourse level opinion relations: An annotation study. In Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue, pages 129– 137. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Swapna+Somasundaran%2C+Josef+Ruppenhofer%2C+and+Janyce+Wiebe%2E+2008%2E+Discourse+level+opinion+relations%3A+An+annotation+study%2E+In+Proceedings+of+the+9th+SIGdial+Workshop+on+Discourse+and+Dialogue%2C+pages+129%E2%80%93)
20. Veselin Stoyanov and Claire Cardie. 2008. Annotating topics of opinions. In LREC. [find](https://scholar.google.com/scholar?q=Veselin+Stoyanov+and+Claire+Cardie%2E+2008%2E+Annotating+topics+of+opinions%2E+In+LREC%2E)
21. Cigdem Toprak, Niklas Jakob, and Iryna Gurevych. 2010. [Sentence and expression level annotation of opinions in user-generated discourse](https://md.research.elamur.ai/free/en/papers/toprak-2010-sentence-expression-level-annotation-opinions/). In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 575–584. Association for Computational Linguistics.
22. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2-3):165–210. [↗](https://doi.org/10.1007/s10579-005-7880-9)
23. Theresa Ann Wilson. 2008. Fine-grained subjectivity and sentiment analysis: recognizing the intensity, polarity, and attitudes of private states. ProQuest. [find](https://scholar.google.com/scholar?q=Theresa+Ann+Wilson%2E+2008%2E+Fine%2Dgrained+subjectivity+and+sentiment+analysis%3A+recognizing+the+intensity%2C+polarity%2C+and+attitudes+of+private+states%2E+ProQuest%2E)
24. Wajdi Zaghouani, Behrang Mohit, Nizar Habash, Ossama Obeid, Nadi Tomeh, Alla Rozovskaya, Noura Farra, Sarah Alkuhlani, and Kemal Oflazer. 2014. [Large scale Arabic error annotation: Guidelines and framework](https://md.research.elamur.ai/free/en/papers/zaghouani-2014-large-scale-arabic-error-annotation/). In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland, May. European Language Resources Association (ELRA). ACL Anthology Identifier: L14-1721. [↗](https://aclanthology.org/L14-1721/)
25. Omar F Zaidan and Chris Callison-Burch. 2011. [The Arabic online commentary dataset: an annotated dataset of informal Arabic with high dialectal content](https://md.research.elamur.ai/free/en/papers/zaidan-2011-arabic-online-commentary-dataset-annotated/). In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pages 37–41. Association for Computational Linguistics.

*Updated 1 November 2026*