---
title: "Word Affect Intensities — Mohammad 2018"
description: "Human-built lexicons for basic emotions usually record only whether a word is linked to an emotion."
authors: ["Saif Mohammad"]
year: 2018
venue: "Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)"
source: "https://aclanthology.org/L18-1027/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/mohammad-2018-word-affect-intensities/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/mohammad-2018-word-affect-intensities/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2018](https://md.research.elamur.ai/free/papers/2018/page/4/) / Word Affect Intensities

# Word Affect Intensities

**Authors:** [Saif Mohammad](https://md.research.elamur.ai/free/authors/saif-mohammad/) · **Year:** 2018 · **Venue:** Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/L18-1027/) · [PDF](https://aclanthology.org/L18-1027.pdf) · [OpenAlex](https://openalex.org/W2611755161)

**Citations:** 39 citations (openalex-snapshot · 2026-10-14) · [75](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Human-built lexicons for basic emotions usually record only whether a word is linked to an emotion. Automatically induced lexicons give graded scores but are less trustworthy. This paper fills that gap with the NRC Affect Intensity Lexicon (AIL), a manually annotated resource with real-valued intensity scores from 0 to 1 for close to 6,000 English words. It covers four emotions: joy, sadness, anger and fear.

Candidate words came from two sources:
- words already tagged with emotions in the NRC Emotion Lexicon;
- words that appear with emotion hashtags in a large tweet collection more often than chance would predict.

This mix means the lexicon covers both everyday vocabulary and social-media language. Intensities were gathered with best–worst scaling (BWS) over 4-tuples instead of rating scales. Repeat annotation reproduces the scores closely (split-half reliability above 0.91), so the scores are both stable and fine-grained.

The author also aligns AIL with the NRC VAD Lexicon. The three negative emotions look much alike in valence, arousal and dominance, but sadness words come out slightly less dominant. AIL supports emotion analysis and natural language generation.

## Key points
- AIL contains close to 6,000 English entries scored for four emotions. The author states that disgust, anticipation, trust and surprise will be added later.
- Annotators labelled 4-tuples using BWS. The paper cites prior evidence that three annotations for each of 2N tuples (N = number of items) are enough for reliable scores.
- Reliability: split-half reliability exceeds 0.91. Independent re-annotation correlates with the original scores at Spearman 0.92 and Pearson 0.91.
- Comparison with the NRC VAD Lexicon (about 20K words): anger, fear and sadness words have similar average VAD profiles, and sadness words score a bit lower on dominance.
- The lexicon is freely released for non-commercial research use.

## Interesting ideas
- **Comparative judgments instead of rating scales.** Picking the most and least intense item in a 4-tuple implicitly orders five of the six item pairs. This makes each judgment cheap and avoids scale-region bias and drift between annotators. It could work for any graded property, such as formality, toxicity or difficulty.
- **A predictable annotation budget.** The rule of thumb of 2N tuples, each judged three times, lets you plan a crowdsourcing project in advance. This is especially useful when building a similar lexicon for a new language such as Ukrainian.
- **Hashtags as noisy labels for candidate selection.** Words that co-occur with tags like #angry can be used to pick which words to annotate, without trusting the co-occurrence signal for the final scores. This keeps annotation focused on relevant, social-media-specific vocabulary.
- **Hand-label the terms automatic methods get wrong.** Antonyms and loosely related words co-occur with emotion terms and confuse statistical methods. Deliberately including them in manual annotation fixes exactly the weak spot of co-occurrence-based lexicons.
- **Check reliability by re-annotation and cross-check against other resources.** Measuring split-half reliability, then aligning the new lexicon with an existing VAD resource, both validates the scores and surfaces linguistic findings. The sadness–dominance difference is an example of what such a comparison can reveal.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cited by

Cited by

75 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 70 not yet.

- [Affective and Contextual Embedding for Sarcasm Detection](https://md.research.elamur.ai/free/papers/babanejad-2020-affective-contextual-embedding-sarcasm-detection/)
- [Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems](https://md.research.elamur.ai/free/papers/kiritchenko-2018-examining-gender-race-bias-two/)
- [Emotion Intensities in Tweets](https://md.research.elamur.ai/free/papers/mohammad-2017-emotion-intensities-tweets/)
- [Obtaining Reliable Human Ratings of Valence, Arousal, and Dominance for 20,000 English Words](https://md.research.elamur.ai/free/papers/mohammad-2018-obtaining-reliable-human-ratings-valence/)
- [SemEval-2018 Task 1: Affect in Tweets](https://md.research.elamur.ai/free/papers/mohammad-2018-semeval-2018-task-1-affect/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mohammad2018word,
  title = {{Word Affect Intensities}},
  author = {Mohammad, Saif},
  booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)},
  year = {2018},
  doi = {10.63317/58igjzkfg9k5},
  url = {https://doi.org/10.63317/58igjzkfg9k5}
}
```

- [← previous WLV at SemEval-2018 Task 3: Dissecting Tweets in Search of Irony](https://md.research.elamur.ai/free/papers/rohanian-2018-wlv-semeval-2018-task-3/)
- [next → WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-hop Inference](https://md.research.elamur.ai/free/papers/jansen-2018-worldtree-corpus-explanation-graphs-elementary/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/papers/2018/page/4/)

## References

32 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bradley, M. M. and Lang, P. J. (1999). Affective norms for English words (ANEW): Instruction manual and affective ratings. Technical report, The Center for Research in Psychophysiology, University of Florida. [find](https://scholar.google.com/scholar?q=Bradley%2C+M%2E+M%2E+and+Lang%2C+P%2E+J%2E+%281999%29%2E+Affective+norms+for+English+words+%28ANEW%29%3A+Instruction+manual+and+affective+ratings%2E+Technical+report%2C+The+Center+for+Research+in+Psychophysiology%2C+University+of+)
2. Brants, T. and Franz, A. (2006). Web 1t 5-gram version 1. Linguistic Data Consortium. [find](https://scholar.google.com/scholar?q=Brants%2C+T%2E+and+Franz%2C+A%2E+%282006%29%2E+Web+1t+5%2Dgram+version+1%2E+Linguistic+Data+Consortium%2E)
3. Cohen, S. H. (2003). Maximum difference scaling: Improved measures of importance and preference for segmentation. Sawtooth Software, Inc. [find](https://scholar.google.com/scholar?q=Cohen%2C+S%2E+H%2E+%282003%29%2E+Maximum+difference+scaling%3A+Improved+measures+of+importance+and+preference+for+segmentation%2E+Sawtooth+Software%2C+Inc%2E)
4. David, H. A. (1963). The method of paired comparisons. Hafner Publishing Company, New York. [find](https://scholar.google.com/scholar?q=David%2C+H%2E+A%2E+%281963%29%2E+The+method+of+paired+comparisons%2E+Hafner+Publishing+Company%2C+New+York%2E)
5. Ekman, P. (1992). An argument for basic emotions. Cognition and Emotion, 6(3):169–200. [find](https://scholar.google.com/scholar?q=Ekman%2C+P%2E+%281992%29%2E+An+argument+for+basic+emotions%2E+Cognition+and+Emotion%2C+6%283%29%3A169%E2%80%93200%2E) [our card](https://md.research.elamur.ai/free/papers/ekman-1992-argument-basic-emotions/)
6. Flynn, T. N. and Marley, A. A. J. (2014). Best-worst scaling: theory and methods. In Stephane Hess et al., editors, Handbook of Choice Modelling, pages 178–201. Edward Elgar Publishing. [find](https://scholar.google.com/scholar?q=Flynn%2C+T%2E+N%2E+and+Marley%2C+A%2E+A%2E+J%2E+%282014%29%2E+Best%2Dworst+scaling%3A+theory+and+methods%2E+In+Stephane+Hess+et+al%2E%2C+editors%2C+Handbook+of+Choice+Modelling%2C+pages+178%E2%80%93201%2E+Edward+Elgar+Publishing%2E)
7. Francisco, V. and Gervás, P. (2006). Automated mark up of affective information in English texts. In Petr Sojka, et al., editors, Text, Speech and Dialogue, volume 4188 of Lecture Notes in CS, pages 375–382. Springer. [find](https://scholar.google.com/scholar?q=Francisco%2C+V%2E+and+Gerva%CC%81s%2C+P%2E+%282006%29%2E+Automated+mark+up+of+affective+information+in+English+texts%2E+In+Petr+Sojka%2C+et+al%2E%2C+editors%2C+Text%2C+Speech+and+Dialogue%2C+volume+4188+of+Lecture+Notes+in+CS%2C+pages+)
8. Frijda, N. H. (1988). The laws of emotion. American psychologist, 43(5):349. [find](https://scholar.google.com/scholar?q=Frijda%2C+N%2E+H%2E+%281988%29%2E+The+laws+of+emotion%2E+American+psychologist%2C+43%285%29%3A349%2E)
9. Jurgens, D., Mohammad, S. M., Turney, P., and Holyoak, K. (2012). Semeval-2012 task 2: Measuring degrees of relational similarity. In Proceedings of the 6th International Workshop on Semantic Evaluation, pages 356– 364, Montréal, Canada, 7-8 June. [find](https://scholar.google.com/scholar?q=Jurgens%2C+D%2E%2C+Mohammad%2C+S%2E+M%2E%2C+Turney%2C+P%2E%2C+and+Holyoak%2C+K%2E+%282012%29%2E+Semeval%2D2012+task+2%3A+Measuring+degrees+of+relational+similarity%2E+In+Proceedings+of+the+6th+International+Workshop+on+Semantic+Evaluati)
10. Jurgens, D. (2013). Embracing ambiguity: A comparison of annotation methodologies for crowdsourcing word sense labels. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics, Atlanta, GA, USA. [find](https://scholar.google.com/scholar?q=Jurgens%2C+D%2E+%282013%29%2E+Embracing+ambiguity%3A+A+comparison+of+annotation+methodologies+for+crowdsourcing+word+sense+labels%2E+In+Proceedings+of+the+Annual+Conference+of+the+North+American+Chapter+of+the+Asso)
11. Kiritchenko, S. and Mohammad, S. M. (2016). Capturing reliable fine-grained sentiment associations by crowdsourcing and best–worst scaling. In Proceedings of The 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL)… [find](https://scholar.google.com/scholar?q=Kiritchenko%2C+S%2E+and+Mohammad%2C+S%2E+M%2E+%282016%29%2E+Capturing+reliable+fine%2Dgrained+sentiment+associations+by+crowdsourcing+and+best%E2%80%93worst+scaling%2E+In+Proceedings+of+The+15th+Annual+Conference+of+the+North+Am)
12. Kiritchenko, S. and Mohammad, S. M. (2017). Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceedings of The Annual Meeting of the Association for Computational Linguistics (ACL), Vancouver, Canada. [find](https://scholar.google.com/scholar?q=Kiritchenko%2C+S%2E+and+Mohammad%2C+S%2E+M%2E+%282017%29%2E+Best%2Dworst+scaling+more+reliable+than+rating+scales%3A+A+case+study+on+sentiment+intensity+annotation%2E+In+Proceedings+of+The+Annual+Meeting+of+the+Association)
13. Louviere, J. J., Flynn, T. N., and Marley, A. A. J. (2015). Best-Worst Scaling: Theory, Methods and Applications. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Louviere%2C+J%2E+J%2E%2C+Flynn%2C+T%2E+N%2E%2C+and+Marley%2C+A%2E+A%2E+J%2E+%282015%29%2E+Best%2DWorst+Scaling%3A+Theory%2C+Methods+and+Applications%2E+Cambridge+University+Press%2E)
14. Louviere, J. J. (1991). Best-worst scaling: A model for the largest difference judgments. Working Paper. [find](https://scholar.google.com/scholar?q=Louviere%2C+J%2E+J%2E+%281991%29%2E+Best%2Dworst+scaling%3A+A+model+for+the+largest+difference+judgments%2E+Working+Paper%2E)
15. Mohammad, S. M. and Bravo-Marquez, F. (2017a). Emotion intensities in tweets. In Proceedings of the sixth joint conference on lexical and computational semantics (*Sem), Vancouver, Canada. [find](https://scholar.google.com/scholar?q=Mohammad%2C+S%2E+M%2E+and+Bravo%2DMarquez%2C+F%2E+%282017a%29%2E+Emotion+intensities+in+tweets%2E+In+Proceedings+of+the+sixth+joint+conference+on+lexical+and+computational+semantics+%28%2ASem%29%2C+Vancouver%2C+Canada%2E) [our card](https://md.research.elamur.ai/free/papers/mohammad-2017-emotion-intensities-tweets/)
16. Mohammad, S. M. and Bravo-Marquez, F. (2017b). WASSA-2017 shared task on emotion intensity. In Proceedings of the Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA), Copenhagen, Denmark. [find](https://scholar.google.com/scholar?q=Mohammad%2C+S%2E+M%2E+and+Bravo%2DMarquez%2C+F%2E+%282017b%29%2E+WASSA%2D2017+shared+task+on+emotion+intensity%2E+In+Proceedings+of+the+Workshop+on+Computational+Approaches+to+Subjectivity%2C+Sentiment+and+Social+Media+Analy)
17. Mohammad, S. M. and Kiritchenko, S. (2015). Using hashtags to capture fine emotion categories from tweets. Computational Intelligence, 31(2):301–326. [find](https://scholar.google.com/scholar?q=Mohammad%2C+S%2E+M%2E+and+Kiritchenko%2C+S%2E+%282015%29%2E+Using+hashtags+to+capture+fine+emotion+categories+from+tweets%2E+Computational+Intelligence%2C+31%282%29%3A301%E2%80%93326%2E)
18. Mohammad, S. M. and Kiritchenko, S. (2018). Understanding emotions: A dataset of tweets to study interactions between affect categories. In Proceedings of the 11th Edition of the Language Resources and Evaluation Conference (LREC-2018), Miyazaki, Japan. [find](https://scholar.google.com/scholar?q=Mohammad%2C+S%2E+M%2E+and+Kiritchenko%2C+S%2E+%282018%29%2E+Understanding+emotions%3A+A+dataset+of+tweets+to+study+interactions+between+affect+categories%2E+In+Proceedings+of+the+11th+Edition+of+the+Language+Resources+an)
19. Mohammad, S. M. and Turney, P. D. (2010). Emotions evoked by common words and phrases: Using Mechanical Turk to create an emotion lexicon. In Proceedings of the NAACL-HLT Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, LA, California. [find](https://scholar.google.com/scholar?q=Mohammad%2C+S%2E+M%2E+and+Turney%2C+P%2E+D%2E+%282010%29%2E+Emotions+evoked+by+common+words+and+phrases%3A+Using+Mechanical+Turk+to+create+an+emotion+lexicon%2E+In+Proceedings+of+the+NAACL%2DHLT+Workshop+on+Computational+App)
20. Mohammad, S. M. and Turney, P. D. (2013). Crowdsourcing a word–emotion association lexicon. Computational Intelligence, 29(3):436–465. [find](https://scholar.google.com/scholar?q=Mohammad%2C+S%2E+M%2E+and+Turney%2C+P%2E+D%2E+%282013%29%2E+Crowdsourcing+a+word%E2%80%93emotion+association+lexicon%2E+Computational+Intelligence%2C+29%283%29%3A436%E2%80%93465%2E)
21. Mohammad, S. M., Sobhani, P., and Kiritchenko, S. (2017). Stance and sentiment in tweets. Special Section of the ACM Transactions on Internet Technology on Argumentation in Social Media, 17(3). [find](https://scholar.google.com/scholar?q=Mohammad%2C+S%2E+M%2E%2C+Sobhani%2C+P%2E%2C+and+Kiritchenko%2C+S%2E+%282017%29%2E+Stance+and+sentiment+in+tweets%2E+Special+Section+of+the+ACM+Transactions+on+Internet+Technology+on+Argumentation+in+Social+Media%2C+17%283%29%2E)
22. Mohammad, S. M., Bravo-Marquez, F., Salameh, M., and Kiritchenko, S. (2018). Semeval-2018 Task 1: Affect in tweets. In Proceedings of International Workshop on Semantic Evaluation (SemEval-2018), New Orleans, USA. [find](https://scholar.google.com/scholar?q=Mohammad%2C+S%2E+M%2E%2C+Bravo%2DMarquez%2C+F%2E%2C+Salameh%2C+M%2E%2C+and+Kiritchenko%2C+S%2E+%282018%29%2E+Semeval%2D2018+Task+1%3A+Affect+in+tweets%2E+In+Proceedings+of+International+Workshop+on+Semantic+Evaluation+%28SemEval%2D2018%29%2C+New+) [our card](https://md.research.elamur.ai/free/papers/mohammad-2018-semeval-2018-task-1-affect/)
23. Mohammad, S. (2012). \#Emotional Tweets. In Proceedings of the Joint Conference on Lexical and Computational Semantics, pages 246–255, Montréal, Canada. [find](https://scholar.google.com/scholar?q=Mohammad%2C+S%2E+%282012%29%2E+%5C%23Emotional+Tweets%2E+In+Proceedings+of+the+Joint+Conference+on+Lexical+and+Computational+Semantics%2C+pages+246%E2%80%93255%2C+Montre%CC%81al%2C+Canada%2E)
24. Mohammad, S. M. (2018). Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 english words. In Submitted. [find](https://scholar.google.com/scholar?q=Mohammad%2C+S%2E+M%2E+%282018%29%2E+Obtaining+reliable+human+ratings+of+valence%2C+arousal%2C+and+dominance+for+20%2C000+english+words%2E+In+Submitted%2E) [our card](https://md.research.elamur.ai/free/papers/mohammad-2018-obtaining-reliable-human-ratings-valence/)
25. Nakov, P., Rosenthal, S., Kiritchenko, S., Mohammad, S. M., Kozareva, Z., Ritter, A., Stoyanov, V., and Zhu, X. (2016). Developing a successful semeval task in sentiment analysis of twitter and other social media texts. Language Resources and Evaluation, 50(1):35–65. [find](https://scholar.google.com/scholar?q=Nakov%2C+P%2E%2C+Rosenthal%2C+S%2E%2C+Kiritchenko%2C+S%2E%2C+Mohammad%2C+S%2E+M%2E%2C+Kozareva%2C+Z%2E%2C+Ritter%2C+A%2E%2C+Stoyanov%2C+V%2E%2C+and+Zhu%2C+X%2E+%282016%29%2E+Developing+a+successful+semeval+task+in+sentiment+analysis+of+twitter+and+other+)
26. Nielsen, F. Å. (2011). A new ANEW: Evaluation of a word list for sentiment analysis in microblogs. In Proceedings of the ESWC Workshop on ’Making Sense of Microposts’: Big things come in small packages, pages 93–98, Heraklion, Crete. [find](https://scholar.google.com/scholar?q=Nielsen%2C+F%2E+A%CC%8A%2E+%282011%29%2E+A+new+ANEW%3A+Evaluation+of+a+word+list+for+sentiment+analysis+in+microblogs%2E+In+Proceedings+of+the+ESWC+Workshop+on+%E2%80%99Making+Sense+of+Microposts%E2%80%99%3A+Big+things+come+in+small+packag)
27. Orme, B. (2009). Maxdiff analysis: Simple counting, individual-level logit, and HB. Sawtooth Software, Inc. [find](https://scholar.google.com/scholar?q=Orme%2C+B%2E+%282009%29%2E+Maxdiff+analysis%3A+Simple+counting%2C+individual%2Dlevel+logit%2C+and+HB%2E+Sawtooth+Software%2C+Inc%2E)
28. Parrot, W. (2001). Emotions in Social Psychology. Psychology Press. [find](https://scholar.google.com/scholar?q=Parrot%2C+W%2E+%282001%29%2E+Emotions+in+Social+Psychology%2E+Psychology+Press%2E)
29. Plutchik, R. (1980). A general psychoevolutionary theory of emotion. Emotion: Theory, research, and experience, 1(3):3–33. [find](https://scholar.google.com/scholar?q=Plutchik%2C+R%2E+%281980%29%2E+A+general+psychoevolutionary+theory+of+emotion%2E+Emotion%3A+Theory%2C+research%2C+and+experience%2C+1%283%29%3A3%E2%80%9333%2E)
30. Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological review, 110(1):145. [find](https://scholar.google.com/scholar?q=Russell%2C+J%2E+A%2E+%282003%29%2E+Core+affect+and+the+psychological+construction+of+emotion%2E+Psychological+review%2C+110%281%29%3A145%2E)
31. Stone, P., Dunphy, D. C., Smith, M. S., Ogilvie, D. M., and associates. (1966). The General Inquirer: A Computer Approach to Content Analysis. The MIT Press. [find](https://scholar.google.com/scholar?q=Stone%2C+P%2E%2C+Dunphy%2C+D%2E+C%2E%2C+Smith%2C+M%2E+S%2E%2C+Ogilvie%2C+D%2E+M%2E%2C+and+associates%2E+%281966%29%2E+The+General+Inquirer%3A+A+Computer+Approach+to+Content+Analysis%2E+The+MIT+Press%2E)
32. Strapparava, C. and Mihalcea, R. (2007). Semeval-2007 task 14: Affective text. In Proceedings of SemEval-2007, pages 70–74, Prague, Czech Republic. [find](https://scholar.google.com/scholar?q=Strapparava%2C+C%2E+and+Mihalcea%2C+R%2E+%282007%29%2E+Semeval%2D2007+task+14%3A+Affective+text%2E+In+Proceedings+of+SemEval%2D2007%2C+pages+70%E2%80%9374%2C+Prague%2C+Czech+Republic%2E)