---
title: "NRC-Canada: Building the State-of-the-Art in — Mohammad 2013"
description: "The paper describes the NRC-Canada systems for SemEval-2013 Task 2 (Sentiment Analysis in Twitter)."
authors: ["Saif Mohammad", "Svetlana Kiritchenko", "Xiaodan Zhu"]
year: 2013
venue: "Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)"
source: "https://aclanthology.org/S13-2053/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mohammad-2013-nrc-canada-building-state-art/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mohammad-2013-nrc-canada-building-state-art/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2013](https://md.research.elamur.ai/free/en/papers/2013/) / NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets

# NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets

**Authors:** [Saif Mohammad](https://md.research.elamur.ai/free/en/authors/saif-mohammad/), [Svetlana Kiritchenko](https://md.research.elamur.ai/free/en/authors/svetlana-kiritchenko/), [Xiaodan Zhu](https://md.research.elamur.ai/free/en/authors/xiaodan-zhu/) · **Year:** 2013 · **Venue:** Second Joint Conference on Lexical and Computational Semantics (\*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/S13-2053/) · [PDF](https://aclanthology.org/S13-2053.pdf) · [OpenAlex](https://openalex.org/W1589554437)

**Citations:** 451 citations (openalex-snapshot · 2026-10-14) · [209](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper describes the NRC-Canada systems for SemEval-2013 Task 2 (Sentiment Analysis in Twitter). The task has two parts: classifying a whole message as positive, negative, or neutral (message-level), and classifying a marked term inside a message (term-level). Both systems use a linear SVM trained on the organizers' annotated tweets. The features cover surface form (word and character ngrams, all-caps, elongated words, punctuation, emoticons), linguistic information (POS counts, Brown clusters, negated contexts) and sentiment lexicons.

A key contribution is two large tweet-specific lexicons built automatically. Tweets were given pseudo-labels from sentiment-word hashtags or from emoticons, and each word's score is the difference between its PMI with the positive and negative labels. The systems ranked first on tweets in both subtasks. They also transferred well to SMS messages without any retuning. Ablation shows that lexicon features matter most at message level and ngrams matter most at term level. The work matters because it shows how cheap, noisily labeled data can produce sentiment resources that beat or complement hand-built lexicons. Everything is built from freely available resources.

## Key points
- **Message-level results:** macro-averaged F of positive and negative classes was 69.02 on tweets and 68.46 on SMS. Both ranked first out of 48 submissions from 34 teams.
- **Term-level results:**
  - Tweets: 88.93, first among 29 submissions from 23 teams.
  - SMS: 88.00, second, 0.39 points behind the winner.
  - After a post-competition bigram bug fix, scores rose to 89.10 and 88.34.
- **NRC Hashtag Sentiment Lexicon:**
  - Built from about 775,000 tweets collected via Twitter API polling from April to December 2012.
  - Seeds were positive and negative hashtags taken from Roget's Thesaurus.
  - It covers 54,129 unigrams, 316,531 bigrams and 308,808 non-contiguous pairs.
- **Sentiment140 Lexicon:** derived the same way from the 1.6M emoticon-labeled tweets of the sentiment140 corpus. It contains 62,468 unigrams, 677,698 bigrams and 480,010 pairs.
- **Message-level ablation:**
  - Lexicon features were the most influential group, with gains above 8.5 points.
  - Tweets gained mostly from the automatic lexicons, while SMS gained more from the manual ones (NRC Emotion Lexicon, MPQA, Bing Liu Lexicon).
- **Term-level ablation:**
  - Dropping the target's own features costs about 16 points on tweets.
  - Context features add roughly 2–4 points.

## Interesting ideas
- **Distant supervision for lexicons:** treat cheap signals such as hashtags or emoticons as noisy labels, then score each word by PMI(positive) minus PMI(negative). This yields large domain-specific lexicons with graded strength. The approach could be repeated for any language or platform where such markers exist.
- **Negation scope applied to all features:** a negated span runs from a negation word to the next punctuation mark. Words inside it get a `_NEG` suffix, and lexicon-derived polarity features are also marked as negated. This is a simple trick that lets a linear model learn flipped or dampened polarity.
- **Explaining a suspiciously high score:** the authors checked why term-level ngrams were so strong. They found that 85.1% of test tokens had already appeared as training targets, and that targets mostly keep one dominant polarity (ratio 0.808). This kind of overlap analysis is a useful sanity check for any benchmark.
- **Group ablation that reveals redundancy:** removing word ngrams alone or character ngrams alone costs little, but removing both costs a lot. This shows the two groups encode overlapping information, which is useful when pruning feature sets.
- **Cross-genre evaluation without retuning:** the tweet-trained model was applied unchanged to SMS. This gives an honest picture of robustness. It also showed that hand-built lexicons generalize better outside the source domain than tweet-derived ones.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

209 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 199 not yet.

- [DataStories at SemEval-2017 Task 4: Deep LSTM with Attention for Message-level and Topic-based Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/baziotis-2017-datastories-semeval-2017-task-4/)
- [A Retrospective Analysis of the Fake News Challenge Stance-Detection Task](https://md.research.elamur.ai/free/en/papers/hanselowski-2018-retrospective-analysis-fake-news-challenge/)
- [NRC-Canada-2014: Detecting Aspects and Sentiment in Customer Reviews](https://md.research.elamur.ai/free/en/papers/kiritchenko-2014-nrc-canada-2014-detecting-aspects/)
- [Metaphor as a Medium for Emotion: An Empirical Study](https://md.research.elamur.ai/free/en/papers/mohammad-2016-metaphor-as-medium-emotion-empirical/)
- [SemEval-2016 Task 6: Detecting Stance in Tweets](https://md.research.elamur.ai/free/en/papers/mohammad-2016-semeval-2016-task-6-detecting/)
- [Emotion Intensities in Tweets](https://md.research.elamur.ai/free/en/papers/mohammad-2017-emotion-intensities-tweets/)
- [WASSA-2017 Shared Task on Emotion Intensity](https://md.research.elamur.ai/free/en/papers/mohammad-2017-wassa-2017-shared-task-emotion/)
- [SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets](https://md.research.elamur.ai/free/en/papers/patwa-2020-semeval-2020-task-9-overview/)
- [SemEval-2015 Task 10: Sentiment Analysis in Twitter](https://md.research.elamur.ai/free/en/papers/rosenthal-2015-semeval-2015-task-10-sentiment/)
- [Attention-based LSTM for Aspect-level Sentiment Classification](https://md.research.elamur.ai/free/en/papers/wang-2016-attention-based-lstm-aspect-level/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mohammad2013nrc,
  title = {{NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets}},
  author = {Mohammad, Saif and Kiritchenko, Svetlana and Zhu, Xiaodan},
  booktitle = {Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)},
  year = {2013},
  url = {https://aclanthology.org/S13-2053/}
}
```

- [← previous Normalisation of Historical Text Using Context-Sensitive Weighted Levenshtein Distance and Compound Splitting](https://md.research.elamur.ai/free/en/papers/pettersson-2013-normalisation-historical-text-context-sensitive/)
- [next → On the Information Conveyed by Discourse Markers](https://md.research.elamur.ai/free/en/papers/asr-2013-information-conveyed-discourse-markers/)
- [↑ back to papers of 2013](https://md.research.elamur.ai/free/en/papers/2013/)

## References

16 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Chih-Chung Chang and Chih-Jen Lin. 2011. LIB-SVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(3):27:1–27:27. [find](https://scholar.google.com/scholar?q=Chih%2DChung+Chang+and+Chih%2DJen+Lin%2E+2011%2E+LIB%2DSVM%3A+A+library+for+support+vector+machines%2E+ACM+Transactions+on+Intelligent+Systems+and+Technology%2C+2%283%29%3A27%3A1%E2%80%9327%3A27%2E)
2. Cynthia Chew and Gunther Eysenbach. 2010. Pandemics in the Age of Twitter: Content Analysis of Tweets during the 2009 H1N1 Outbreak. PLoS ONE, 5(11):e14118+, November. [find](https://scholar.google.com/scholar?q=Cynthia+Chew+and+Gunther+Eysenbach%2E+2010%2E+Pandemics+in+the+Age+of+Twitter%3A+Content+Analysis+of+Tweets+during+the+2009+H1N1+Outbreak%2E+PLoS+ONE%2C+5%2811%29%3Ae14118%2B%2C+November%2E)
3. R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and Lin C.-J. 2008. LIBLINEAR: A Library for Large Linear Classification. Journal of Machine Learning Research, 9:1871–1874. [find](https://scholar.google.com/scholar?q=R%2E%2DE%2E+Fan%2C+K%2E%2DW%2E+Chang%2C+C%2E%2DJ%2E+Hsieh%2C+X%2E%2DR%2E+Wang%2C+and+Lin+C%2E%2DJ%2E+2008%2E+LIBLINEAR%3A+A+Library+for+Large+Linear+Classification%2E+Journal+of+Machine+Learning+Research%2C+9%3A1871%E2%80%931874%2E)
4. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-Speech Tagging for Twitter: Annotation, Features, and Experiments. In Proceedings of the Annual Meeting of the Associatio… [find](https://scholar.google.com/scholar?q=Kevin+Gimpel%2C+Nathan+Schneider%2C+Brendan+O%E2%80%99Connor%2C+Dipanjan+Das%2C+Daniel+Mills%2C+Jacob+Eisenstein%2C+Michael+Heilman%2C+Dani+Yogatama%2C+Jeffrey+Flanigan%2C+and+Noah+A%2E+Smith%2E+2011%2E+Part%2Dof%2DSpeech+Tagging+for+Tw) [our card](https://md.research.elamur.ai/free/en/papers/gimpel-2011-part-speech-tagging-twitter-annotation/)
5. Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter Sentiment Classification using Distant Supervision. In Final Projects from CS224N for Spring 2008/2009 at The Stanford Natural Language Processing Group. [find](https://scholar.google.com/scholar?q=Alec+Go%2C+Richa+Bhayani%2C+and+Lei+Huang%2E+2009%2E+Twitter+Sentiment+Classification+using+Distant+Supervision%2E+In+Final+Projects+from+CS224N+for+Spring+2008%2F2009+at+The+Stanford+Natural+Language+Processing+)
6. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, KDD ’04, pages 168– 177, New York, NY, USA. ACM. [find](https://scholar.google.com/scholar?q=Minqing+Hu+and+Bing+Liu%2E+2004%2E+Mining+and+summarizing+customer+reviews%2E+In+Proceedings+of+the+tenth+ACM+SIGKDD+international+conference+on+Knowledge+discovery+and+data+mining%2C+KDD+%E2%80%9904%2C+pages+168%E2%80%93+177%2C)
7. Bernard J. Jansen, Mimi Zhang, Kate Sobel, and Abdur Chowdury. 2009. Twitter power: Tweets as electronic word of mouth. Journal of the American Society for Information Science and Technology, 60(11):2169– 2188. [find](https://scholar.google.com/scholar?q=Bernard+J%2E+Jansen%2C+Mimi+Zhang%2C+Kate+Sobel%2C+and+Abdur+Chowdury%2E+2009%2E+Twitter+power%3A+Tweets+as+electronic+word+of+mouth%2E+Journal+of+the+American+Society+for+Information+Science+and+Technology%2C+60%2811%29%3A2)
8. Benjamin Mandel, Aron Culotta, John Boulahanis, Danielle Stark, Bonnie Lewis, and Jeremy Rodrigue. 2012. A demographic analysis of online sentiment during hurricane irene. In Proceedings of the Second Workshop on Language in Social Media, LSM ’12, pages 27–36, Stroudsburg, PA, USA. Association for C… [find](https://scholar.google.com/scholar?q=Benjamin+Mandel%2C+Aron+Culotta%2C+John+Boulahanis%2C+Danielle+Stark%2C+Bonnie+Lewis%2C+and+Jeremy+Rodrigue%2E+2012%2E+A+demographic+analysis+of+online+sentiment+during+hurricane+irene%2E+In+Proceedings+of+the+Second)
9. Saif M. Mohammad and Peter D. Turney. 2010. Emotions Evoked by Common Words and Phrases: Using Mechanical Turk to Create an Emotion Lexicon. In Proceedings of the NAACL-HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, LA, California. [find](https://scholar.google.com/scholar?q=Saif+M%2E+Mohammad+and+Peter+D%2E+Turney%2E+2010%2E+Emotions+Evoked+by+Common+Words+and+Phrases%3A+Using+Mechanical+Turk+to+Create+an+Emotion+Lexicon%2E+In+Proceedings+of+the+NAACL%2DHLT+2010+Workshop+on+Computatio) [our card](https://md.research.elamur.ai/free/en/papers/mohammad-2010-emotions-evoked-common-words-phrases/)
10. Saif Mohammad and Tony Yang. 2011. Tracking Sentiment in Mail: How Genders Differ on Emotional Axes. In Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis (WASSA 2.011), pages 70–79, Portland, Oregon. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Saif+Mohammad+and+Tony+Yang%2E+2011%2E+Tracking+Sentiment+in+Mail%3A+How+Genders+Differ+on+Emotional+Axes%2E+In+Proceedings+of+the+2nd+Workshop+on+Computational+Approaches+to+Subjectivity+and+Sentiment+Analys)
11. Saif Mohammad. 2012. \#Emotional Tweets. In Proceedings of the First Joint Conference on Lexical and Computational Semantics (*SEM), pages 246– 255, Montréal, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Saif+Mohammad%2E+2012%2E+%5C%23Emotional+Tweets%2E+In+Proceedings+of+the+First+Joint+Conference+on+Lexical+and+Computational+Semantics+%28%2ASEM%29%2C+pages+246%E2%80%93+255%2C+Montre%CC%81al%2C+Canada%2E+Association+for+Computational+Li)
12. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up?: Sentiment Classification Using Machine Learning Techniques. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 79–86, Philadelphia, PA. [find](https://scholar.google.com/scholar?q=Bo+Pang%2C+Lillian+Lee%2C+and+Shivakumar+Vaithyanathan%2E+2002%2E+Thumbs+up%3F%3A+Sentiment+Classification+Using+Machine+Learning+Techniques%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Langu) [our card](https://md.research.elamur.ai/free/en/papers/pang-2002-thumbs-up-sentiment-classification-machine/)
13. Marcel Salathé and Shashank Khandelwal. 2011. Assessing vaccination sentiments with online social media: Implications for infectious disease dynamics and control. PLoS Computational Biology, 7(10). [find](https://scholar.google.com/scholar?q=Marcel+Salathe%CC%81+and+Shashank+Khandelwal%2E+2011%2E+Assessing+vaccination+sentiments+with+online+social+media%3A+Implications+for+infectious+disease+dynamics+and+control%2E+PLoS+Computational+Biology%2C+7%2810%29%2E)
14. Sudha Verma, Sarah Vieweg, William Corvey, Leysia Palen, James Martin, Martha Palmer, Aaron Schram, and Kenneth Anderson. 2011. Natural language processing to the rescue? extracting ”situational awareness” tweets during mass emergency. In International AAAI Conference on Weblogs and Social Media. [find](https://scholar.google.com/scholar?q=Sudha+Verma%2C+Sarah+Vieweg%2C+William+Corvey%2C+Leysia+Palen%2C+James+Martin%2C+Martha+Palmer%2C+Aaron+Schram%2C+and+Kenneth+Anderson%2E+2011%2E+Natural+language+processing+to+the+rescue%3F+extracting+%E2%80%9Dsituational+aware)
15. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT ’05, pages 347–354, Stroudsburg, PA, USA. Association for… [find](https://scholar.google.com/scholar?q=Theresa+Wilson%2C+Janyce+Wiebe%2C+and+Paul+Hoffmann%2E+2005%2E+Recognizing+contextual+polarity+in+phrase%2Dlevel+sentiment+analysis%2E+In+Proceedings+of+the+conference+on+Human+Language+Technology+and+Empirical+M) [our card](https://md.research.elamur.ai/free/en/papers/wilson-2005-recognizing-contextual-polarity-phrase-level/)
16. Theresa Wilson, Zornitsa Kozareva, Preslav Nakov, Sara Rosenthal, Veselin Stoyanov, and Alan Ritter. 2013. SemEval-2013 task 2: Sentiment analysis in twitter. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’13, Atlanta, Georgia, USA, June. [find](https://scholar.google.com/scholar?q=Theresa+Wilson%2C+Zornitsa+Kozareva%2C+Preslav+Nakov%2C+Sara+Rosenthal%2C+Veselin+Stoyanov%2C+and+Alan+Ritter%2E+2013%2E+SemEval%2D2013+task+2%3A+Sentiment+analysis+in+twitter%2E+In+Proceedings+of+the+International+Works) [our card](https://md.research.elamur.ai/free/en/papers/nakov-2013-semeval-2013-task-2-sentiment/)