---
title: "NRC-Canada-2014 — Kiritchenko 2014"
description: "The paper describes the NRC-Canada entry to SemEval-2014 Task 4, Aspect Based Sentiment Analysis, which covers restaurant and laptop reviews."
authors: ["Svetlana Kiritchenko", "Xiaodan Zhu", "Colin Cherry", "Saif Mohammad"]
year: 2014
venue: "Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)"
source: "https://aclanthology.org/S14-2076/"
doi: "10.3115/v1/s14-2076"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kiritchenko-2014-nrc-canada-2014-detecting-aspects/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kiritchenko-2014-nrc-canada-2014-detecting-aspects/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/) / NRC-Canada-2014: Detecting Aspects and Sentiment in Customer Reviews

# NRC-Canada-2014: Detecting Aspects and Sentiment in Customer Reviews

**Authors:** [Svetlana Kiritchenko](https://md.research.elamur.ai/free/en/authors/svetlana-kiritchenko/), [Xiaodan Zhu](https://md.research.elamur.ai/free/en/authors/xiaodan-zhu/), [Colin Cherry](https://md.research.elamur.ai/free/en/authors/colin-cherry/), [Saif Mohammad](https://md.research.elamur.ai/free/en/authors/saif-mohammad/) · **Year:** 2014 · **Venue:** Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/S14-2076/) · [PDF](https://aclanthology.org/S14-2076.pdf) · [DOI 10.3115/v1/s14-2076](https://doi.org/10.3115/v1/s14-2076) · [OpenAlex](https://openalex.org/W2252057809)

**Citations:** 697 citations (openalex-snapshot · 2026-10-14) · [91](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper describes the NRC-Canada entry to SemEval-2014 Task 4, Aspect Based Sentiment Analysis, which covers restaurant and laptop reviews. The task has four parts: finding explicit aspect terms, assigning one of five restaurant aspect categories, and judging the sentiment toward terms and toward categories. Sentiment is one of positive, negative, neutral or conflict.

Aspect terms are found with an in-house semi-Markov tagger. The other three subtasks use supervised SVM classifiers. A large share of the gains comes from resources built from unlabeled review collections: about 184K Yelp restaurant reviews and about 125K Amazon laptop reviews. From these, the authors derived PMI-based sentiment lexicons, a word–aspect association lexicon and Brown word clusters.

The system ranked first in aspect category detection and in category sentiment, and third in term extraction. For term sentiment it placed first on laptops and second on restaurants. The work shows that cheap, automatically induced in-domain lexicons can substantially improve fine-grained sentiment classifiers.

## Key points
- **Unlabeled corpora:**
  - The Yelp restaurant reviews corpus has 183,935 reviews from 58 food-related business categories.
  - The Amazon laptop reviews corpus has 124,712 reviews that mention "laptop" or "notebook".
  - Reviews with 1–2 stars were treated as negative and those with 4–5 stars as positive.
- **Aspect term extraction:**
  - A semi-Markov tagger with O/T tags labels phrases of up to 5 tokens and is trained with structured Passive-Aggressive learning.
  - It reached F1 80.19 on restaurants and 68.57 on laptops, third of 24 teams in both domains.
  - Lower-cased back-off features mattered more than the choice of PA over the perceptron.
- **Aspect term polarity:**
  - A linear SVM (LibSVM) uses surface, lexicon and dependency-parse features from Stanford CoreNLP.
  - Accuracy was 70.49 on laptops and 80.16 on restaurants.
  - Removing the sentiment lexicons cut laptop accuracy to 63.61.
- **Aspect category detection:**
  - Five one-vs-all SVMs reached F1 88.58, first among 21 submissions.
  - The Word–Aspect Association Lexicon added 4.5 F1 points.
- **Aspect category polarity:**
  - The system reached 82.93 accuracy, first among 23 submissions.
  - Lexical resources contributed 8.78 points, mostly through the in-domain Yelp sentiment lexicons.
  - Tweet lexicons and word clusters did not help on this subtask.

## Interesting ideas
- **Star ratings as distant labels:** The authors treated review ratings as noisy polarity labels and scored each word by the difference of its PMI with positive and with negative reviews. This yields large in-domain lexicons without any annotation, and it suits any domain that has rated user content.
- **Separate negated and affirmative lexicons:** Words are scored separately depending on whether they appear between a negation word and the next punctuation mark. For example, "good" scores 1.2 normally but −1.4 when negated. This simple way to model negation carries over to lexicon building in other languages.
- **Bootstrapped word–aspect lexicon:** The trained category classifier labels the large unlabeled corpus, and PMI then links words to categories. This turns a small labeled set into a broad association resource.
- **Feature duplication for multiple categories:** Every feature gets a general copy and a category-specific copy, following Daumé III's domain adaptation trick. A single classifier can then share common sentiment cues while still learning category-specific ones, which is useful whenever classes or subdomains have little data each.
- **Fallback for empty predictions:** If no binary classifier fires for a sentence, it receives the most probable category when that probability is at least 0.4. This cheaply recovers recall in multi-label setups.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

91 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 84 not yet.

- [Recurrent Attention Network on Memory for Aspect Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/chen-2017-recurrent-attention-network-memory-aspect/)
- [Transformation Networks for Target-Oriented Sentiment Classification](https://md.research.elamur.ai/free/en/papers/li-2018-transformation-networks-target-oriented-sentiment/)
- [Arabic aspect sentiment polarity classification using BERT](https://md.research.elamur.ai/free/en/papers/m-2021-abdelgwad-arabic-aspect-sentiment-polarity-classification/)
- [SemEval-2016 Task 6: Detecting Stance in Tweets](https://md.research.elamur.ai/free/en/papers/mohammad-2016-semeval-2016-task-6-detecting/)
- [Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence](https://md.research.elamur.ai/free/en/papers/sun-2019-utilizing-bert-aspect-based-sentiment/)
- [Aspect Level Sentiment Classification with Deep Memory Network](https://md.research.elamur.ai/free/en/papers/tang-2016-aspect-level-sentiment-classification-deep/)
- [Aspect-based Sentiment Classification with Aspect-specific Graph Convolutional Networks](https://md.research.elamur.ai/free/en/papers/zhang-2019-aspect-based-sentiment-classification-aspect/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kiritchenko2014nrc,
  title = {{NRC-Canada-2014: Detecting Aspects and Sentiment in Customer Reviews}},
  author = {Kiritchenko, Svetlana and Zhu, Xiaodan and Cherry, Colin and Mohammad, Saif},
  booktitle = {Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)},
  year = {2014},
  doi = {10.3115/v1/s14-2076},
  url = {https://doi.org/10.3115/v1/s14-2076}
}
```

- [← previous Now We Stronger than Ever: African-American English Syntax in Twitter](https://md.research.elamur.ai/free/en/papers/stewart-2014-now-we-stronger-than-ever/)
- [next → On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/)

## References

21 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Peter F Brown, Peter V Desouza, Robert L Mercer, Vincent J Della Pietra, and Jenifer C Lai. 1992. Class-based n-gram models of natural language. Computational linguistics, 18(4):467–479. [find](https://scholar.google.com/scholar?q=Peter+F+Brown%2C+Peter+V+Desouza%2C+Robert+L+Mercer%2C+Vincent+J+Della+Pietra%2C+and+Jenifer+C+Lai%2E+1992%2E+Class%2Dbased+n%2Dgram+models+of+natural+language%2E+Computational+linguistics%2C+18%284%29%3A467%E2%80%93479%2E)
2. Chih-Chung Chang and Chih-Jen Lin. 2011. LIB-SVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(3):27:1–27:27. [find](https://scholar.google.com/scholar?q=Chih%2DChung+Chang+and+Chih%2DJen+Lin%2E+2011%2E+LIB%2DSVM%3A+A+library+for+support+vector+machines%2E+ACM+Transactions+on+Intelligent+Systems+and+Technology%2C+2%283%29%3A27%3A1%E2%80%9327%3A27%2E)
3. Michael Collins. 2002. Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Michael+Collins%2E+2002%2E+Discriminative+training+methods+for+hidden+markov+models%3A+Theory+and+experiments+with+perceptron+algorithms%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Lan) [our card](https://md.research.elamur.ai/free/en/papers/collins-2002-discriminative-training-methods-hidden-markov/)
4. Koby Crammer and Yoram Singer. 2002. On the algorithmic implementation of multiclass kernel-based vector machines. The Journal of Machine Learning Research, 2:265–292. [find](https://scholar.google.com/scholar?q=Koby+Crammer+and+Yoram+Singer%2E+2002%2E+On+the+algorithmic+implementation+of+multiclass+kernel%2Dbased+vector+machines%2E+The+Journal+of+Machine+Learning+Research%2C+2%3A265%E2%80%93292%2E)
5. Koby Crammer, Ofer Dekel, Joseph Keshet, Shai Shalev-Shwartz, and Yoram Singer. 2006. Online passive-aggressive algorithms. Journal of Machine Learning Research, 7:551–585. [find](https://scholar.google.com/scholar?q=Koby+Crammer%2C+Ofer+Dekel%2C+Joseph+Keshet%2C+Shai+Shalev%2DShwartz%2C+and+Yoram+Singer%2E+2006%2E+Online+passive%2Daggressive+algorithms%2E+Journal+of+Machine+Learning+Research%2C+7%3A551%E2%80%93585%2E)
6. Hal Daumé III. 2007. Frustratingly easy domain adaptation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL ’07, pages 256 – 263. [find](https://scholar.google.com/scholar?q=Hal+Daume%CC%81+III%2E+2007%2E+Frustratingly+easy+domain+adaptation%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+ACL+%E2%80%9907%2C+pages+256+%E2%80%93+263%2E) [our card](https://md.research.elamur.ai/free/en/papers/iii-2007-frustratingly-easy-domain-adaptation/)
7. Berry de Bruijn, Colin Cherry, Svetlana Kiritchenko, Joel Martin, and Xiaodan Zhu. 2011. Machine-learned solutions for three stages of clinical information extraction: the state of the art at i2b2 2010. Journal of the American Medical Informatics Association, 18(5):557–562. [find](https://scholar.google.com/scholar?q=Berry+de+Bruijn%2C+Colin+Cherry%2C+Svetlana+Kiritchenko%2C+Joel+Martin%2C+and+Xiaodan+Zhu%2E+2011%2E+Machine%2Dlearned+solutions+for+three+stages+of+clinical+information+extraction%3A+the+state+of+the+art+at+i2b2+201)
8. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of the International Conference on Language Resources and Evaluation, LREC ’06. [find](https://scholar.google.com/scholar?q=Marie%2DCatherine+de+Marneffe%2C+Bill+MacCartney%2C+and+Christopher+D%2E+Manning%2E+2006%2E+Generating+typed+dependency+parses+from+phrase+structure+parses%2E+In+Proceedings+of+the+International+Conference+on+Langu) [our card](https://md.research.elamur.ai/free/en/papers/map-2006-marneffe-generating-typed-dependency-parses-phrase/)
9. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-speech tagging for Twitter: Annotation, features, and experiments. In Proceedings of the Annual Meeting of the Associatio… [find](https://scholar.google.com/scholar?q=Kevin+Gimpel%2C+Nathan+Schneider%2C+Brendan+O%E2%80%99Connor%2C+Dipanjan+Das%2C+Daniel+Mills%2C+Jacob+Eisenstein%2C+Michael+Heilman%2C+Dani+Yogatama%2C+Jeffrey+Flanigan%2C+and+Noah+A%2E+Smith%2E+2011%2E+Part%2Dof%2Dspeech+tagging+for+Tw) [our card](https://md.research.elamur.ai/free/en/papers/gimpel-2011-part-speech-tagging-twitter-annotation/)
10. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’04, pages 168–177, New York, NY, USA. ACM. [find](https://scholar.google.com/scholar?q=Minqing+Hu+and+Bing+Liu%2E+2004%2E+Mining+and+summarizing+customer+reviews%2E+In+Proceedings+of+the+10th+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining%2C+KDD+%E2%80%9904%2C+pages+168%E2%80%93177%2C+N)
11. Svetlana Kiritchenko, Xiaodan Zhu, and Saif M. Mohammad. 2014. Sentiment analysis of short informal texts. Journal of Artificial Intelligence Research (to appear). [find](https://scholar.google.com/scholar?q=Svetlana+Kiritchenko%2C+Xiaodan+Zhu%2C+and+Saif+M%2E+Mohammad%2E+2014%2E+Sentiment+analysis+of+short+informal+texts%2E+Journal+of+Artificial+Intelligence+Research+%28to+appear%29%2E)
12. Julian McAuley and Jure Leskovec. 2013. Hidden factors and hidden topics: understanding rating dimensions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pages 165–172. ACM. [find](https://scholar.google.com/scholar?q=Julian+McAuley+and+Jure+Leskovec%2E+2013%2E+Hidden+factors+and+hidden+topics%3A+understanding+rating+dimensions+with+review+text%2E+In+Proceedings+of+the+7th+ACM+conference+on+Recommender+systems%2C+pages+165%E2%80%931)
13. Saif M. Mohammad and Peter D. Turney. 2010. Emotions evoked by common words and phrases: Using Mechanical Turk to create an emotion lexicon. In Proceedings of the NAACL-HLT Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, LA, California. [find](https://scholar.google.com/scholar?q=Saif+M%2E+Mohammad+and+Peter+D%2E+Turney%2E+2010%2E+Emotions+evoked+by+common+words+and+phrases%3A+Using+Mechanical+Turk+to+create+an+emotion+lexicon%2E+In+Proceedings+of+the+NAACL%2DHLT+Workshop+on+Computational+A) [our card](https://md.research.elamur.ai/free/en/papers/mohammad-2010-emotions-evoked-common-words-phrases/)
14. Saif M. Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. NRC-Canada: Building the state-of-the-art in sentiment analysis of tweets. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’13, Atlanta, Georgia, USA, June. [find](https://scholar.google.com/scholar?q=Saif+M%2E+Mohammad%2C+Svetlana+Kiritchenko%2C+and+Xiaodan+Zhu%2E+2013%2E+NRC%2DCanada%3A+Building+the+state%2Dof%2Dthe%2Dart+in+sentiment+analysis+of+tweets%2E+In+Proceedings+of+the+International+Workshop+on+Semantic+Evalu) [our card](https://md.research.elamur.ai/free/en/papers/mohammad-2013-nrc-canada-building-state-art/)
15. Olutobi Owoputi, Brendan OConnor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A Smith. 2013. Improved part-of-speech tagging for online conversational text with word clusters. In Proceedings of NAACL-HLT, pages 380–390. [find](https://scholar.google.com/scholar?q=Olutobi+Owoputi%2C+Brendan+OConnor%2C+Chris+Dyer%2C+Kevin+Gimpel%2C+Nathan+Schneider%2C+and+Noah+A+Smith%2E+2013%2E+Improved+part%2Dof%2Dspeech+tagging+for+online+conversational+text+with+word+clusters%2E+In+Proceedings+) [our card](https://md.research.elamur.ai/free/en/papers/owoputi-2013-improved-part-speech-tagging-online/)
16. Maria Pontiki, Dimitrios Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 Task 4: Aspect based sentiment analysis. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’14, Dublin, Ireland, August. [find](https://scholar.google.com/scholar?q=Maria+Pontiki%2C+Dimitrios+Galanis%2C+John+Pavlopoulos%2C+Harris+Papageorgiou%2C+Ion+Androutsopoulos%2C+and+Suresh+Manandhar%2E+2014%2E+SemEval%2D2014+Task+4%3A+Aspect+based+sentiment+analysis%2E+In+Proceedings+of+the+In) [our card](https://md.research.elamur.ai/free/en/papers/pontiki-2014-semeval-2014-task-4-aspect/)
17. M.F. Porter. 1980. An algorithm for suffix stripping. Program, 3:130–137. [find](https://scholar.google.com/scholar?q=M%2EF%2E+Porter%2E+1980%2E+An+algorithm+for+suffix+stripping%2E+Program%2C+3%3A130%E2%80%93137%2E)
18. Sunita Sarawagi and William W Cohen. 2004. Semi-markov conditional random fields for information extraction. In Advances in Neural Information Processing Systems, volume 17, pages 1185–1192. [find](https://scholar.google.com/scholar?q=Sunita+Sarawagi+and+William+W+Cohen%2E+2004%2E+Semi%2Dmarkov+conditional+random+fields+for+information+extraction%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+17%2C+pages+1185%E2%80%931192%2E)
19. Peter Turney and Michael L Littman. 2003. Measuring praise and criticism: Inference of semantic orientation from association. ACM Transactions on Information Systems, 21(4). [find](https://scholar.google.com/scholar?q=Peter+Turney+and+Michael+L+Littman%2E+2003%2E+Measuring+praise+and+criticism%3A+Inference+of+semantic+orientation+from+association%2E+ACM+Transactions+on+Information+Systems%2C+21%284%29%2E)
20. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT ’05, pages 347–354, Stroudsburg, PA, USA. [find](https://scholar.google.com/scholar?q=Theresa+Wilson%2C+Janyce+Wiebe%2C+and+Paul+Hoffmann%2E+2005%2E+Recognizing+contextual+polarity+in+phrase%2Dlevel+sentiment+analysis%2E+In+Proceedings+of+the+Conference+on+Human+Language+Technology+and+Empirical+M) [our card](https://md.research.elamur.ai/free/en/papers/wilson-2005-recognizing-contextual-polarity-phrase-level/)
21. Xiaodan Zhu, Hongyu Guo, Saif M. Mohammad, and Svetlana Kiritchenko. 2014. An empirical study on the effect of negation words on sentiment. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL ’14. [find](https://scholar.google.com/scholar?q=Xiaodan+Zhu%2C+Hongyu+Guo%2C+Saif+M%2E+Mohammad%2C+and+Svetlana+Kiritchenko%2E+2014%2E+An+empirical+study+on+the+effect+of+negation+words+on+sentiment%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+)