---
title: "Sentence and Expression Level Annotation of Opinions in"
description: "Most expression-level opinion annotation has targeted newswire (MPQA) and meeting dialogs. 22 references, cited by 70 papers here."
authors: ["Cigdem Toprak", "Niklas Jakob", "Iryna Gurevych"]
year: 2010
venue: "Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P10-1059/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/toprak-2010-sentence-expression-level-annotation-opinions/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/toprak-2010-sentence-expression-level-annotation-opinions/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2010](https://md.research.elamur.ai/free/en/papers/2010/) / Sentence and Expression Level Annotation of Opinions in User-Generated Discourse

# Sentence and Expression Level Annotation of Opinions in User-Generated Discourse

**Authors:** Cigdem Toprak, [Niklas Jakob](https://md.research.elamur.ai/free/en/authors/niklas-jakob/) and [Iryna Gurevych](https://md.research.elamur.ai/free/en/authors/iryna-gurevych/) · **Year:** 2010 · **Venue:** [Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2010](https://md.research.elamur.ai/free/en/venues/acl/2010/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P10-1059/) · [PDF](https://aclanthology.org/P10-1059.pdf) · [OpenAlex](https://openalex.org/W2155120841)

**Citations:** 146 citations (OpenAlex, 2026-10-14) · [70](https://md.research.elamur.ai/free/en/papers/toprak-2010-sentence-expression-level-annotation-opinions/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Most expression-level opinion annotation has targeted newswire (MPQA) and meeting dialogs. The few annotated review corpora, such as Hu & Liu (2004) and Zhuang et al. (2006), record only targets and polarities. They do not mark opinion spans, modifiers or anaphoric links, and they report no agreement figures. Toprak, Jakob and Gurevych address this with a corpus of consumer reviews from rateitall and eopinions, many of them about online universities, annotated with a two-stage scheme.

1. **Sentence level.** Each sentence is checked for whether it is relevant to the review's topic and whether it carries an evaluation. An evaluation can be an explicit opinion or a "polar fact", meaning a verifiable statement that still implies judgment.
2. **Expression level.** Evaluative on-topic sentences then receive:
   - opinion expression spans,
   - polarity and strength,
   - modifiers such as negation and intensifiers,
   - targets and holders, linked to their expressions and resolved through anaphora.

A two-annotator study shows that span marking and opinion polarity are fairly reliable. Strength and polar-fact polarity are harder. The resource supports fine-grained tasks like negation-aware polarity and target extraction.

## Key points
- The data comes from the rateitall and eopinions review sites. Examples cover schools such as Capella University, Devry and University of Phoenix.
- Expression-level agreement in Table 4 ranges from 0.74 to 0.97 (raw) and 0.55 to 0.94 (κ) across subtasks. The lowest-agreement attributes are opinion strength and the polarity of polar targets.
- Among annotated modifiers, about 70% are intensifiers and about 20% are negations.
- Polarity differs by evaluation type:
  - 61% of agreed polar-fact targets were negative.
  - Only 35% of agreed opinion expressions were negative.
  - No instances were neutral.
- Strength labels split into 6% weak, 54% average and 40% strong. Most strength disagreements were between weak and average. A third judge consolidates the annotations for the final corpus, which is announced for release on the UKP website.

## Interesting ideas
- **Filter sentences before annotating expressions.** Doing a cheap sentence-level relevance and evaluation pass first narrows the costly span annotation and makes it more consistent. This cascaded design suits any expensive fine-grained labeling project.
- **Annotate polar facts as a category.** Objective statements that still imply praise or criticism get their own label. This captures evaluation that lexicon-based methods miss, and it applies equally to complaints, news comments or trade feedback.
- **Store linked components rather than final scores.** The scheme keeps opinion term, modifier, target and holder as separate units connected by links, including anaphoric ones. Errors can then be traced to a specific component, and new attributes can be added without redesigning the scheme.
- **Use label distributions as findings.** Comparing labels across categories exposed a pattern: reviewers tend to describe negative experiences factually but voice positive ones as explicit opinions. Such analyses can guide both model design and data sampling.
- **Report agreement per subtask.** Computing agreement separately for spans, polarity, strength and modifiers shows which attributes are reliable enough to train on and which need clearer guidelines.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Technische Universität Darmstadt](https://md.research.elamur.ai/free/en/universities/technische-universitat-darmstadt/) — University, Darmstadt, Germany · 76 works · 35,008 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧵 Discourse analysis](https://md.research.elamur.ai/free/en/topics/discourse-analysis/) · 394 works

## Cite

Cite as: Cigdem Toprak, Niklas Jakob and Iryna Gurevych. 2010. Sentence and Expression Level Annotation of Opinions in User-Generated Discourse. Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{toprak2010sentence,
  title = {{Sentence and Expression Level Annotation of Opinions in User-Generated Discourse}},
  author = {Cigdem Toprak, Niklas Jakob and Iryna Gurevych},
  booktitle = {Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics},
  year = {2010},
  url = {https://aclanthology.org/P10-1059/}
}
```

- [← previous Large Scale Parallel Document Mining for Machine Translation](https://md.research.elamur.ai/free/en/papers/uszkoreit-2010-large-scale-parallel-document-mining/)
- [next → Coreference Resolution in a Modular, Entity-Centered Model](https://md.research.elamur.ai/free/en/papers/haghighi-2010-coreference-resolution-modular-entity-centered/)
- [↑ back to papers of 2010](https://md.research.elamur.ai/free/en/papers/2010/)

## References

22 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Nicholas Asher, Farah Benamara, and Yvette Yannick Mathieu. 2008. Distilling opinion in discourse: A preliminary study. In Coling 2008: Companion volume: Posters, pages 7–10, Manchester, UK. [find](https://scholar.google.com/scholar?q=Nicholas+Asher%2C+Farah+Benamara%2C+and+Yvette+Yannick+Mathieu%2E+2008%2E+Distilling+opinion+in+discourse%3A+A+preliminary+study%2E+In+Coling+2008%3A+Companion+volume%3A+Posters%2C+pages+7%E2%80%9310%2C+Manchester%2C+UK%2E)
2. Eric Breck, Yejin Choi, and Claire Cardie. 2007. Identifying expressions of opinion in context. In Proceedings of the Twentieth International Joint Conference on Artificial Intelligence (IJCAI-2007), pages 2683–2688, Hyderabad, India. [find](https://scholar.google.com/scholar?q=Eric+Breck%2C+Yejin+Choi%2C+and+Claire+Cardie%2E+2007%2E+Identifying+expressions+of+opinion+in+context%2E+In+Proceedings+of+the+Twentieth+International+Joint+Conference+on+Artificial+Intelligence+%28IJCAI%2D2007%29%2C+)
3. Xiwen Cheng and Feiyu Xu. 2008. Fine-grained opinion topic and polarity identification. In Proceedings of the 6th International Conference on Language Resources and Evaluation, pages 2710–2714, Mar-rekech, Morocco. [find](https://scholar.google.com/scholar?q=Xiwen+Cheng+and+Feiyu+Xu%2E+2008%2E+Fine%2Dgrained+opinion+topic+and+polarity+identification%2E+In+Proceedings+of+the+6th+International+Conference+on+Language+Resources+and+Evaluation%2C+pages+2710%E2%80%932714%2C+Mar%2Dre)
4. Yejin Choi, Claire Cardie, Ellen Riloff, and Siddharth Patwardhan. 2005. [Identifying sources of opinions with conditional random fields and extraction patterns](https://md.research.elamur.ai/free/en/papers/choi-2005-identifying-sources-opinions-conditional-random/). In HLT ’05: Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 355–362,…
5. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. [find](https://scholar.google.com/scholar?q=Jacob+Cohen%2E+1960%2E+A+coefficient+of+agreement+for+nominal+scales%2E+Educational+and+Psychological+Measurement%2C+20%281%29%3A37%E2%80%9346%2E)
6. Xiaowen Ding, Bing Liu, and Philip S. Yu. 2008. A holistic lexicon-based approach to opinion mining. In Proceedings of the International Conference on Web Search and Web Data Mining, WSDM 2008, pages 231–240, Palo Alto, California, USA. [find](https://scholar.google.com/scholar?q=Xiaowen+Ding%2C+Bing+Liu%2C+and+Philip+S%2E+Yu%2E+2008%2E+A+holistic+lexicon%2Dbased+approach+to+opinion+mining%2E+In+Proceedings+of+the+International+Conference+on+Web+Search+and+Web+Data+Mining%2C+WSDM+2008%2C+pages+)
7. Andrea Esuli and Fabrizio Sebastiani. 2006. Senti-WordNet: A publicly available lexical resource for opinion mining. In Proceedings of the 5th International Conference on Language Resources and Evaluation, pages 417–422, Genova, Italy. [find](https://scholar.google.com/scholar?q=Andrea+Esuli+and+Fabrizio+Sebastiani%2E+2006%2E+Senti%2DWordNet%3A+A+publicly+available+lexical+resource+for+opinion+mining%2E+In+Proceedings+of+the+5th+International+Conference+on+Language+Resources+and+Evalua)
8. Angela Fahrni and Manfred Klenner. 2008. Old wine or warm beer: Target-specific sentiment analysis of adjectives. In Proceedings of the Symposium on Affective Language in Human and Machine, AISB 2008 Convention, pages 60 – 63, Aberdeen, Scotland. [find](https://scholar.google.com/scholar?q=Angela+Fahrni+and+Manfred+Klenner%2E+2008%2E+Old+wine+or+warm+beer%3A+Target%2Dspecific+sentiment+analysis+of+adjectives%2E+In+Proceedings+of+the+Symposium+on+Affective+Language+in+Human+and+Machine%2C+AISB+2008+)
9. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In KDD’04: Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 168–177, Seattle, Washington. [find](https://scholar.google.com/scholar?q=Minqing+Hu+and+Bing+Liu%2E+2004%2E+Mining+and+summarizing+customer+reviews%2E+In+KDD%E2%80%9904%3A+Proceedings+of+the+Tenth+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining%2C+pages+168%E2%80%93177%2C+S)
10. Soo-Min Kim and Eduard Hovy. 2006. [Extracting opinions, opinion holders, and topics expressed in online news media text](https://md.research.elamur.ai/free/en/papers/kim-2006-extracting-opinions-opinion-holders-topics/). In Proceedings of the Workshop on Sentiment and Subjectivity in Text at the joint COLING-ACL Conference, pages 1–8, Sydney, Australia.
11. Klaus Krippendorff. 2004. Content Analysis: An Introduction to Its Methology. Sage Publications, Thousand Oaks, California. [find](https://scholar.google.com/scholar?q=Klaus+Krippendorff%2E+2004%2E+Content+Analysis%3A+An+Introduction+to+Its+Methology%2E+Sage+Publications%2C+Thousand+Oaks%2C+California%2E)
12. Rebecca J. Passonneau. 2004. Computing reliability for coreference. In Proceedings of LREC, volume 4, pages 1503–1506, Lisbon. [find](https://scholar.google.com/scholar?q=Rebecca+J%2E+Passonneau%2E+2004%2E+Computing+reliability+for+coreference%2E+In+Proceedings+of+LREC%2C+volume+4%2C+pages+1503%E2%80%931506%2C+Lisbon%2E)
13. Randolph Quirk, Sidney Greenbaum, Geoffrey Leech, and Jan Svartvik. 1985. A Comprehensive Grammar of the English Language. Longman, New York. [find](https://scholar.google.com/scholar?q=Randolph+Quirk%2C+Sidney+Greenbaum%2C+Geoffrey+Leech%2C+and+Jan+Svartvik%2E+1985%2E+A+Comprehensive+Grammar+of+the+English+Language%2E+Longman%2C+New+York%2E)
14. Ellen Riloff and Janyce Wiebe. 2003. [Learning extraction patterns for subjective expressions](https://md.research.elamur.ai/free/en/papers/riloff-2003-learning-extraction-patterns-subjective-expressions/). In EMNLP-03: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 105–112.
15. Swapna Somasundaran, Josef Ruppenhofer, and Janyce Wiebe. 2008. Discourse level opinion relations: An annotation study. In In Proceedings of SIGdial Workshop on Discourse and Dialogue, pages 129– 137, Columbus, Ohio. [find](https://scholar.google.com/scholar?q=Swapna+Somasundaran%2C+Josef+Ruppenhofer%2C+and+Janyce+Wiebe%2E+2008%2E+Discourse+level+opinion+relations%3A+An+annotation+study%2E+In+In+Proceedings+of+SIGdial+Workshop+on+Discourse+and+Dialogue%2C+pages+129%E2%80%93+137%2C)
16. Veselin Stoyanov and Claire Cardie. 2008. Topic identification for fine-grained opinion analysis. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 817–824, Manchester, UK. [find](https://scholar.google.com/scholar?q=Veselin+Stoyanov+and+Claire+Cardie%2E+2008%2E+Topic+identification+for+fine%2Dgrained+opinion+analysis%2E+In+Proceedings+of+the+22nd+International+Conference+on+Computational+Linguistics+%28Coling+2008%29%2C+pages+)
17. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39:165–210. [find](https://scholar.google.com/scholar?q=Janyce+Wiebe%2C+Theresa+Wilson%2C+and+Claire+Cardie%2E+2005%2E+Annotating+expressions+of+opinions+and+emotions+in+language%2E+Language+Resources+and+Evaluation%2C+39%3A165%E2%80%93210%2E)
18. Theresa Wilson and Janyce Wiebe. 2005. Annotating attributions and private states. In Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky, pages 53–60, Ann Arbor, Michigan. [find](https://scholar.google.com/scholar?q=Theresa+Wilson+and+Janyce+Wiebe%2E+2005%2E+Annotating+attributions+and+private+states%2E+In+Proceedings+of+the+Workshop+on+Frontiers+in+Corpus+Annotations+II%3A+Pie+in+the+Sky%2C+pages+53%E2%80%9360%2C+Ann+Arbor%2C+Michiga)
19. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. [Recognizing contextual polarity in phrase-level sentiment analysis](https://md.research.elamur.ai/free/en/papers/wilson-2005-recognizing-contextual-polarity-phrase-level/). In HLT ’05: Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 347–354, Vancouver, British Columbia, Canada.
20. Theresa Wilson. 2008a. Annotating subjective content in meetings. In Proceedings of the Sixth International Language Resources and Evaluation (LREC’08), Marrakech, Morocco. [find](https://scholar.google.com/scholar?q=Theresa+Wilson%2E+2008a%2E+Annotating+subjective+content+in+meetings%2E+In+Proceedings+of+the+Sixth+International+Language+Resources+and+Evaluation+%28LREC%E2%80%9908%29%2C+Marrakech%2C+Morocco%2E)
21. Theresa Ann Wilson. 2008b. Fine-grained Subjectivity and Sentiment Analysis: Recognizing the Intensity, Polarity, and Attitudes of Private States. Ph.D. thesis, University of Pittsburgh. [find](https://scholar.google.com/scholar?q=Theresa+Ann+Wilson%2E+2008b%2E+Fine%2Dgrained+Subjectivity+and+Sentiment+Analysis%3A+Recognizing+the+Intensity%2C+Polarity%2C+and+Attitudes+of+Private+States%2E+Ph%2ED%2E+thesis%2C+University+of+Pittsburgh%2E)
22. Li Zhuang, Feng Jing, and Xiao-Yan Zhu. 2006. Movie review mining and summarization. In CIKM ’06: Proceedings of the 15th ACM international conference on Information and knowledge management, pages 43–50, Arlington, Virginia, USA. [find](https://scholar.google.com/scholar?q=Li+Zhuang%2C+Feng+Jing%2C+and+Xiao%2DYan+Zhu%2E+2006%2E+Movie+review+mining+and+summarization%2E+In+CIKM+%E2%80%9906%3A+Proceedings+of+the+15th+ACM+international+conference+on+Information+and+knowledge+management%2C+pages+43)

*Updated 24 October 2026*