--- title: "Emotions Evoked by Common Words and Phrases — Mohammad 2010" description: "Emotion analysis has had to rely on small, limited lexicons, while polarity resources are much larger." authors: ["Saif Mohammad", "Peter Turney"] year: 2010 venue: "Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text" source: "https://aclanthology.org/W10-0204/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/mohammad-2010-emotions-evoked-common-words-phrases/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mohammad-2010-emotions-evoked-common-words-phrases/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2010](https://md.research.elamur.ai/free/en/papers/2010/) / Emotions Evoked by Common Words and Phrases: Using Mechanical Turk to Create an Emotion Lexicon # Emotions Evoked by Common Words and Phrases: Using Mechanical Turk to Create an Emotion Lexicon **Authors:** [Saif Mohammad](https://md.research.elamur.ai/free/en/authors/saif-mohammad/), [Peter Turney](https://md.research.elamur.ai/free/en/authors/peter-turney/) · **Year:** 2010 · **Venue:** Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/W10-0204/) · [PDF](https://aclanthology.org/W10-0204.pdf) · [OpenAlex](https://openalex.org/W2162010436) **Citations:** 840 citations (openalex-snapshot · 2026-10-14) · [165](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Emotion analysis has had to rely on small, limited lexicons, while polarity resources are much larger. Mohammad and Turney address this gap by crowdsourcing EmoLex, an English lexicon annotated for Plutchik's eight basic emotions (joy, sadness, anger, fear, trust, disgust, surprise, anticipation) and for positive/negative polarity, using Amazon Mechanical Turk. Target terms are frequent unigrams and bigrams from the Macquarie Thesaurus, selected by Google n-gram frequency and supplemented with words from the General Inquirer and the WordNet Affect Lexicon. The key methodological contribution is an automatically generated word-choice question placed before the emotion questions. It filters out annotators who are unfamiliar with a term or answering randomly, and it indicates which sense they are rating. The pilot covers roughly 2,000 terms, each judged by five Turkers, at a total cost under US$500. Agreement is high, and the polarity labels closely match General Inquirer gold labels. The work shows that cheap crowdsourcing can yield reliable affect resources and lays the groundwork for a much larger lexicon. ## Key points - **Data and cost:** EmoLex has 2,081 analyzed terms. Each HIT paid $0.04 and was answered by five different Turkers, more than 1,000 people in total. The whole pilot cost below US$500. - **Term selection:** For each part of speech (nouns, verbs, adjectives, adverbs), the authors took the 200 most frequent unigrams and the 200 most frequent bigrams by Google n-gram corpus frequency. They added the 200 most frequent positive and 200 most frequent negative General Inquirer words, plus WAL emotion words with at most two thesaurus senses. - **Validation against gold labels:** About 90% of GI negative words were marked negative by Turkers, and about 91% of GI positive words were marked positive. - **Agreement:** When polarity is collapsed to two levels (evaluative vs. not), all five annotators agree on more than 50% of terms, and at least four agree on more than 80%. - **Polarity of nouns:** Unigram nouns are rarely negative and often positive. The authors suggest noun concepts may lean positive by default. ## Interesting ideas - **Gating question for quality control:** A multiple-choice synonym question precedes the real task. A random clicker fails it about 75% of the time, so their answers for that term can be discarded. This works for any crowdsourced lexical annotation, including Ukrainian resources. - **Sense disambiguation without definitions:** Thesaurus category head words serve as the answer options. Picking one tells you which sense the annotator has in mind, without long glosses that cost reading time or bias the emotion judgment. - **Automatic question generation from a thesaurus:** Each question uses the target's own category head word as the correct answer and head words of random categories as distractors. This scales easily to any language that has a thesaurus with coarse categories. - **Collect graded, analyze binary:** Annotators rate intensity on four levels (no, weak, moderate, strong), which is later collapsed into two. Collecting finer labels keeps options open and makes the coarser labels more robust. - **Seed known-labeled items into the target set:** Including terms already labeled in existing resources (GI, WAL) gives a built-in quality check against prior gold data at no extra design cost. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 165 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 157 not yet. - [Inducing Domain-Specific Sentiment Lexicons from Unlabeled Corpora](https://md.research.elamur.ai/free/en/papers/hamilton-2016-inducing-domain-specific-sentiment-lexicons/) - [A Retrospective Analysis of the Fake News Challenge Stance-Detection Task](https://md.research.elamur.ai/free/en/papers/hanselowski-2018-retrospective-analysis-fake-news-challenge/) - [NRC-Canada-2014: Detecting Aspects and Sentiment in Customer Reviews](https://md.research.elamur.ai/free/en/papers/kiritchenko-2014-nrc-canada-2014-detecting-aspects/) - [NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets](https://md.research.elamur.ai/free/en/papers/mohammad-2013-nrc-canada-building-state-art/) - [SemEval-2016 Task 6: Detecting Stance in Tweets](https://md.research.elamur.ai/free/en/papers/mohammad-2016-semeval-2016-task-6-detecting/) - [Word Affect Intensities](https://md.research.elamur.ai/free/en/papers/mohammad-2018-word-affect-intensities/) - [From Trial by Fire to Sleep like a Baby: A Lexicon of Anxiety Associations for 20K English Multi-Word Expressions](https://md.research.elamur.ai/free/en/papers/mohammad-2026-trial-fire-sleep-like-baby/) - [Lexicon-Based Methods for Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/taboada-2011-lexicon-based-methods-sentiment-analysis/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{mohammad2010emotions, title = {{Emotions Evoked by Common Words and Phrases: Using Mechanical Turk to Create an Emotion Lexicon}}, author = {Mohammad, Saif and Turney, Peter}, booktitle = {Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text}, year = {2010}, url = {https://aclanthology.org/W10-0204/} } ``` - [← previous Easy-First Dependency Parsing of Modern Hebrew](https://md.research.elamur.ai/free/en/papers/aclbulk-2010-goldberg-easy-first-dependency-parsing-modern/) - [next → End-to-End Coreference Resolution via Hypergraph Partitioning](https://md.research.elamur.ai/free/en/papers/cai-2010-end-end-coreference-resolution-hypergraph/) - [↑ back to papers of 2010](https://md.research.elamur.ai/free/en/papers/2010/) ## References 17 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Cecilia Ovesdotter Alm, Dan Roth, and Richard Sproat. 2005. Emotions from text: Machine learning for text-based emotion prediction. In Proceedings of the Joint Conference on Human Language Technology / Empirical Methods in Natural Language Processing (HLT/EMNLP-2005), pages 579–586, Vancouver, Canad… [find](https://scholar.google.com/scholar?q=Cecilia+Ovesdotter+Alm%2C+Dan+Roth%2C+and+Richard+Sproat%2E+2005%2E+Emotions+from+text%3A+Machine+learning+for+text%2Dbased+emotion+prediction%2E+In+Proceedings+of+the+Joint+Conference+on+Human+Language+Technology+) [our card](https://md.research.elamur.ai/free/en/papers/alm-2005-emotions-text-machine-learning-text/) 2. J.R.L. Bernard, editor. 1986. The Macquarie Thesaurus. Macquarie Library, Sydney, Australia. [find](https://scholar.google.com/scholar?q=J%2ER%2EL%2E+Bernard%2C+editor%2E+1986%2E+The+Macquarie+Thesaurus%2E+Macquarie+Library%2C+Sydney%2C+Australia%2E) 3. Thorsten Brants and Alex Franz. 2006. Web 1t 5-gram version 1. Linguistic Data Consortium. [find](https://scholar.google.com/scholar?q=Thorsten+Brants+and+Alex+Franz%2E+2006%2E+Web+1t+5%2Dgram+version+1%2E+Linguistic+Data+Consortium%2E) 4. Chris Callison-Burch. 2009. Fast, cheap and creative: Evaluating translation quality using amazon’s mechanical turk. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-2009), pages 286–295, Singapore. [find](https://scholar.google.com/scholar?q=Chris+Callison%2DBurch%2E+2009%2E+Fast%2C+cheap+and+creative%3A+Evaluating+translation+quality+using+amazon%E2%80%99s+mechanical+turk%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Processin) [our card](https://md.research.elamur.ai/free/en/papers/burch-2009-fast-cheap-creative-evaluating-translation/) 5. Paul Ekman. 1992. An argument for basic emotions. Cognition and Emotion, 6(3):169–200. [find](https://scholar.google.com/scholar?q=Paul+Ekman%2E+1992%2E+An+argument+for+basic+emotions%2E+Cognition+and+Emotion%2C+6%283%29%3A169%E2%80%93200%2E) [our card](https://md.research.elamur.ai/free/en/papers/ekman-1992-argument-basic-emotions/) 6. Clark Elliott. 1992. The affective reasoner: A process model of emotions in a multi-agent system. Ph.D. thesis, Institute for the Learning Sciences, Northwestern University. [find](https://scholar.google.com/scholar?q=Clark+Elliott%2E+1992%2E+The+affective+reasoner%3A+A+process+model+of+emotions+in+a+multi%2Dagent+system%2E+Ph%2ED%2E+thesis%2C+Institute+for+the+Learning+Sciences%2C+Northwestern+University%2E) 7. Dekang Lin, Shaojun Zhao, Lijuan Qin, and Ming Zhou. 2003. Identifying synonyms among distributionally similar words. In Proceedings of the 18th International Joint Conference on Artificial Intelligence (IJCAI-03), pages 1492–1493, Acapulco, Mexico. [find](https://scholar.google.com/scholar?q=Dekang+Lin%2C+Shaojun+Zhao%2C+Lijuan+Qin%2C+and+Ming+Zhou%2E+2003%2E+Identifying+synonyms+among+distributionally+similar+words%2E+In+Proceedings+of+the+18th+International+Joint+Conference+on+Artificial+Intelligen) 8. A. Lobanova, T. van der Kleij, and J. Spenader. 2010. Defining antonymy: A corpus-based study of opposites by lexico-syntactic patterns. International Journal of Lexicography (in press), 23:19–53. [find](https://scholar.google.com/scholar?q=A%2E+Lobanova%2C+T%2E+van+der+Kleij%2C+and+J%2E+Spenader%2E+2010%2E+Defining+antonymy%3A+A+corpus%2Dbased+study+of+opposites+by+lexico%2Dsyntactic+patterns%2E+International+Journal+of+Lexicography+%28in+press%29%2C+23%3A19%E2%80%9353%2E) 9. Saif Mohammad, Bonnie Dorr, and Codie Dunn. 2008. Computing word-pair antonymy. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-2008), pages 982–991, Waikiki, Hawaii. [find](https://scholar.google.com/scholar?q=Saif+Mohammad%2C+Bonnie+Dorr%2C+and+Codie+Dunn%2E+2008%2E+Computing+word%2Dpair+antonymy%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%2D2008%29%2C+pages+982%E2%80%93991%2C+Waikik) 10. Saif Mohammad, Cody Dunne, and Bonnie Dorr. 2009. Generating high-coverage semantic orientation lexicons from overtly marked words and a thesaurus. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP-2009), pages 599–608, Singapore. [find](https://scholar.google.com/scholar?q=Saif+Mohammad%2C+Cody+Dunne%2C+and+Bonnie+Dorr%2E+2009%2E+Generating+high%2Dcoverage+semantic+orientation+lexicons+from+overtly+marked+words+and+a+thesaurus%2E+In+Proceedings+of+Empirical+Methods+in+Natural+Langu) 11. Alena Neviarouskaya, Helmut Prendinger, and Mitsuru Ishizuka. 2009. Compositionality principle in recognition of fine-grained emotions from text. In Proceedings of the Proceedings of the Third International Conference on Weblogs and Social Media (ICWSM-09), pages 278–281, San Jose, California. [find](https://scholar.google.com/scholar?q=Alena+Neviarouskaya%2C+Helmut+Prendinger%2C+and+Mitsuru+Ishizuka%2E+2009%2E+Compositionality+principle+in+recognition+of+fine%2Dgrained+emotions+from+text%2E+In+Proceedings+of+the+Proceedings+of+the+Third+Interna) 12. R Plutchik. 1980. A general psychoevolutionary theory of emotion. Emotion: Theory, research, and experience, 1(3):3–33. [find](https://scholar.google.com/scholar?q=R+Plutchik%2E+1980%2E+A+general+psychoevolutionary+theory+of+emotion%2E+Emotion%3A+Theory%2C+research%2C+and+experience%2C+1%283%29%3A3%E2%80%9333%2E) 13. Jonathon Read. 2004. Recognising affect in text using pointwise-mutual information. Ph.D. thesis, Department of Informatics, University of Sussex. [find](https://scholar.google.com/scholar?q=Jonathon+Read%2E+2004%2E+Recognising+affect+in+text+using+pointwise%2Dmutual+information%2E+Ph%2ED%2E+thesis%2C+Department+of+Informatics%2C+University+of+Sussex%2E) 14. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Ng. 2008. Cheap and fast - but is it good? Evaluating nonexpert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-2008), pages 254–263, Waikiki, Hawaii. [find](https://scholar.google.com/scholar?q=Rion+Snow%2C+Brendan+O%E2%80%99Connor%2C+Daniel+Jurafsky%2C+and+Andrew+Ng%2E+2008%2E+Cheap+and+fast+%2D+but+is+it+good%3F+Evaluating+nonexpert+annotations+for+natural+language+tasks%2E+In+Proceedings+of+the+Conference+on+Emp) 15. Philip Stone, Dexter C. Dunphy, Marshall S. Smith, Daniel M. Ogilvie, and associates. 1966. The General Inquirer: A Computer Approach to Content Analysis. The MIT Press. [find](https://scholar.google.com/scholar?q=Philip+Stone%2C+Dexter+C%2E+Dunphy%2C+Marshall+S%2E+Smith%2C+Daniel+M%2E+Ogilvie%2C+and+associates%2E+1966%2E+The+General+Inquirer%3A+A+Computer+Approach+to+Content+Analysis%2E+The+MIT+Press%2E) 16. Carlo Strapparava and Alessandro Valitutti. 2004. Wordnet-Affect: An affective extension of WordNet. In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC-2004), pages 1083–1086, Lisbon, Portugal. [find](https://scholar.google.com/scholar?q=Carlo+Strapparava+and+Alessandro+Valitutti%2E+2004%2E+Wordnet%2DAffect%3A+An+affective+extension+of+WordNet%2E+In+Proceedings+of+the+4th+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%2D2004%29) [our card](https://md.research.elamur.ai/free/en/papers/strapparava-2004-wordnet-affect-affective-extension-wordnet/) 17. Peter Turney and Michael Littman. 2003. Measuring praise and criticism: Inference of semantic orientation from association. ACM Transactions on Information Systems (TOIS), 21(4):315–346. [find](https://scholar.google.com/scholar?q=Peter+Turney+and+Michael+Littman%2E+2003%2E+Measuring+praise+and+criticism%3A+Inference+of+semantic+orientation+from+association%2E+ACM+Transactions+on+Information+Systems+%28TOIS%29%2C+21%284%29%3A315%E2%80%93346%2E)