---
title: "Best-Worst Scaling More Reliable than — Kiritchenko 2017"
description: "Rating scales are the standard way to collect graded judgments in NLP, but annotators often disagree with each other, contradict their own earlier…"
authors: ["Svetlana Kiritchenko", "Saif Mohammad"]
year: 2017
venue: "Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/P17-2074/"
doi: "10.18653/v1/p17-2074"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kiritchenko-2017-best-worst-scaling-more-reliable/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kiritchenko-2017-best-worst-scaling-more-reliable/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/) / Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation

# Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation

**Authors:** [Svetlana Kiritchenko](https://md.research.elamur.ai/free/en/authors/svetlana-kiritchenko/), [Saif Mohammad](https://md.research.elamur.ai/free/en/authors/saif-mohammad/) · **Year:** 2017 · **Venue:** Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P17-2074/) · [PDF](https://aclanthology.org/P17-2074.pdf) · [DOI 10.18653/v1/p17-2074](https://doi.org/10.18653/v1/p17-2074) · [OpenAlex](https://openalex.org/W2741691725)

**Citations:** 170 citations (openalex-snapshot · 2026-10-14) · [112](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Rating scales are the standard way to collect graded judgments in NLP, but annotators often disagree with each other, contradict their own earlier answers and favour certain parts of the scale. Best–Worst Scaling (BWS) shows annotators 4-tuples and asks only for the highest and lowest item. It has been promoted as more reliable at a similar annotation cost, but this had not been tested directly. Kiritchenko and Mohammad annotated 3,207 English terms (single words and short phrases) for sentiment intensity on CrowdFlower using both methods. They held the total number of annotations equal across the two approaches. Reliability was measured with split-half reliability (SHR): annotations were divided randomly into two halves, scored separately, and the two rankings were correlated. BWS gave significantly more consistent rankings, especially when annotation budgets were small (about 5N or fewer) and for linguistically complex phrases involving negators or modals. The result supports BWS as a cost-effective option for building sentiment lexicons and other graded NLP resources.

## Key points
- **Term set:** 1,621 words from the General Inquirer (Osgood valence subset) plus 1,586 phrases combining those words with negators, modals or degree adverbs.
- **Annotation design:**
  - Rating scale: a 9-point scale from −4 to 4, with 20 ratings per term.
  - BWS: 2N random 4-tuples, each term appearing in 8 tuples, with 10 annotators per tuple.
- **Rating-scale self-consistency:** 180 terms were shown twice to the same workers. In 37% of cases the worker gave a different score, with an average gap of 1.27 points, and inconsistency grew as the time between the two annotations increased.
- **Main reliability result:** With the full data (about 32K annotations per half), SHR was ρ = 0.98 for BWS versus 0.95 for the rating scale (p \< .001). BWS reached 0.95 using only 3N annotations per half, roughly 30% of the rating-scale effort. Reliability depended on the total number of annotations, not on whether 1N, 1.5N or 2N tuples were used.
- **Complex phrases and resources:** The biggest gap was for positive phrases containing a negator (BWS .79 vs. rating scale .17). All data and the scripts for BWS tuple generation and scoring are publicly released.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

112 papers of our library cite it (of the 140884 whose bibliographies we read); 9 of them have a page here, 103 not yet.

- [Summarizing Opinions: Aspect Extraction Meets Sentiment Prediction and They Are Both Weakly Supervised](https://md.research.elamur.ai/free/en/papers/angelidis-2018-summarizing-opinions-aspect-extraction-meets/)
- [Acquiring a Formality-Informed Lexical Resource for Style Analysis](https://md.research.elamur.ai/free/en/papers/eder-2021-acquiring-formality-informed-lexical-resource/)
- [Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://md.research.elamur.ai/free/en/papers/fabbri-2019-multi-news-large-scale-multi/)
- [Hierarchical Transformers for Multi-Document Summarization](https://md.research.elamur.ai/free/en/papers/liu-2019-hierarchical-transformers-multi-document-summarization/)
- [Emotion Intensities in Tweets](https://md.research.elamur.ai/free/en/papers/mohammad-2017-emotion-intensities-tweets/)
- [WASSA-2017 Shared Task on Emotion Intensity](https://md.research.elamur.ai/free/en/papers/mohammad-2017-wassa-2017-shared-task-emotion/)
- [Obtaining Reliable Human Ratings of Valence, Arousal, and Dominance for 20,000 English Words](https://md.research.elamur.ai/free/en/papers/mohammad-2018-obtaining-reliable-human-ratings-valence/)
- [SemEval-2018 Task 1: Affect in Tweets](https://md.research.elamur.ai/free/en/papers/mohammad-2018-semeval-2018-task-1-affect/)
- [Word Affect Intensities](https://md.research.elamur.ai/free/en/papers/mohammad-2018-word-affect-intensities/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kiritchenko2017best,
  title = {{Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation}},
  author = {Kiritchenko, Svetlana and Mohammad, Saif},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-2074},
  url = {https://doi.org/10.18653/v1/p17-2074}
}
```

- [← previous Beam Search Strategies for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/freitag-2017-beam-search-strategies-neural-machine/)
- [next → Bias in polar questions: Evidence from English and German production experiments](https://md.research.elamur.ai/free/en/papers/domaneschi-2017-bias-polar-questions-evidence-english/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/)

## References

20 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jordan J. Louviere, Terry N. Flynn, and A. A. J. Marley. 2015. Best–Worst Scaling: Theory, Methods and Applications. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Jordan+J%2E+Louviere%2C+Terry+N%2E+Flynn%2C+and+A%2E+A%2E+J%2E+Marley%2E+2015%2E+Best%E2%80%93Worst+Scaling%3A+Theory%2C+Methods+and+Applications%2E+Cambridge+University+Press%2E)
2. Hans Baumgartner and Jan-Benedict E.M. Steenkamp. 2001. Response styles in marketing research: A cross-national investigation. Journal of Marketing Research 38(2):143–156. [find](https://scholar.google.com/scholar?q=Hans+Baumgartner+and+Jan%2DBenedict+E%2EM%2E+Steenkamp%2E+2001%2E+Response+styles+in+marketing+research%3A+A+cross%2Dnational+investigation%2E+Journal+of+Marketing+Research+38%282%29%3A143%E2%80%93156%2E)
3. Saif M. Mohammad and Peter D. Turney. 2013. Crowdsourcing a word–emotion association lexicon. Computational Intelligence 29(3):436–465. [find](https://scholar.google.com/scholar?q=Saif+M%2E+Mohammad+and+Peter+D%2E+Turney%2E+2013%2E+Crowdsourcing+a+word%E2%80%93emotion+association+lexicon%2E+Computational+Intelligence+29%283%29%3A436%E2%80%93465%2E)
4. Margaret M. Bradley and Peter J. Lang. 1999. Affective norms for English words (ANEW): Instruction manual and affective ratings. Technical report, The Center for Research in Psychophysiology, University of Florida. [find](https://scholar.google.com/scholar?q=Margaret+M%2E+Bradley+and+Peter+J%2E+Lang%2E+1999%2E+Affective+norms+for+English+words+%28ANEW%29%3A+Instruction+manual+and+affective+ratings%2E+Technical+report%2C+The+Center+for+Research+in+Psychophysiology%2C+Universi)
5. Bryan Orme. 2009. Maxdiff analysis: Simple counting, individual-level logit, and HB. Sawtooth Software, Inc. [find](https://scholar.google.com/scholar?q=Bryan+Orme%2E+2009%2E+Maxdiff+analysis%3A+Simple+counting%2C+individual%2Dlevel+logit%2C+and+HB%2E+Sawtooth+Software%2C+Inc%2E)
6. Stanley Presser and Howard Schuman. 1996. Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. SAGE Publications, Inc. [find](https://scholar.google.com/scholar?q=Stanley+Presser+and+Howard+Schuman%2E+1996%2E+Questions+and+Answers+in+Attitude+Surveys%3A+Experiments+on+Question+Form%2C+Wording%2C+and+Context%2E+SAGE+Publications%2C+Inc%2E)
7. Herbert Aron David. 1963. The method of paired comparisons. Hafner Publishing Company, New York. [find](https://scholar.google.com/scholar?q=Herbert+Aron+David%2E+1963%2E+The+method+of+paired+comparisons%2E+Hafner+Publishing+Company%2C+New+York%2E)
8. Peter Sheridan Dodds, Kameron Decker Harris, Isabel M. Kloumann, Catherine A. Bliss, and Christopher M. Danforth. 2011. Temporal patterns of happiness and information in a global social network: Hedonometrics and Twitter. PloS One 6(12):e26752. [find](https://scholar.google.com/scholar?q=Peter+Sheridan+Dodds%2C+Kameron+Decker+Harris%2C+Isabel+M%2E+Kloumann%2C+Catherine+A%2E+Bliss%2C+and+Christopher+M%2E+Danforth%2E+2011%2E+Temporal+patterns+of+happiness+and+information+in+a+global+social+network%3A+Hedon)
9. Richard Socher, Alex Perelygin, Jean Y. Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). Se… [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Y%2E+Wu%2C+Jason+Chuang%2C+Christopher+D%2E+Manning%2C+Andrew+Y%2E+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treeba)
10. T. N. Flynn and A. A. J. Marley. 2014. Best-worst scaling: theory and methods. In Stephane Hess and Andrew Daly, editors, Handbook of Choice Modelling, Edward Elgar Publishing, pages 178–201. [find](https://scholar.google.com/scholar?q=T%2E+N%2E+Flynn+and+A%2E+A%2E+J%2E+Marley%2E+2014%2E+Best%2Dworst+scaling%3A+theory+and+methods%2E+In+Stephane+Hess+and+Andrew+Daly%2C+editors%2C+Handbook+of+Choice+Modelling%2C+Edward+Elgar+Publishing%2C+pages+178%E2%80%93201%2E)
11. Philip Stone, Dexter C. Dunphy, Marshall S. Smith, Daniel M. Ogilvie, and associates. 1966. The General Inquirer: A Computer Approach to Content Analysis. The MIT Press. [find](https://scholar.google.com/scholar?q=Philip+Stone%2C+Dexter+C%2E+Dunphy%2C+Marshall+S%2E+Smith%2C+Daniel+M%2E+Ogilvie%2C+and+associates%2E+1966%2E+The+General+Inquirer%3A+A+Computer+Approach+to+Content+Analysis%2E+The+MIT+Press%2E)
12. David Jurgens. 2013. Embracing ambiguity: A comparison of annotation methodologies for crowdsourcing word sense labels. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL). Atlanta, Georgia, pages… [find](https://scholar.google.com/scholar?q=David+Jurgens%2E+2013%2E+Embracing+ambiguity%3A+A+comparison+of+annotation+methodologies+for+crowdsourcing+word+sense+labels%2E+In+Proceedings+of+the+Annual+Conference+of+the+North+American+Chapter+of+the+Ass)
13. Carlo Strapparava and Rada Mihalcea. 2007. Semeval-2007 task 14: Affective text. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007). Association for Computational Linguistics, Prague, Czech Republic, pages 70–74. [http://www.aclweb.org/anthology/S/S07/S07-1013](http://www.aclweb.org/anthology/S/S07/S07-1013). [↗](http://www.aclweb.org/anthology/S/S07/S07-1013) [our card](https://md.research.elamur.ai/free/en/papers/strapparava-2007-semeval-2007-task-14-affective/)
14. David Jurgens, Saif M. Mohammad, Peter Turney, and Keith Holyoak. 2012. SemEval-2012 Task 2: Measuring degrees of relational similarity. In Proceedings of the International Workshop on Semantic Evaluation (SemEval). Montréal, Canada, pages 356–364. [http://www.aclweb.org/anthology/S12-1047](http://www.aclweb.org/anthology/S12-1047). [↗](http://www.aclweb.org/anthology/S12-1047)
15. Louis L. Thurstone. 1927. A law of comparative judgment. Psychological review 34(4):273. [find](https://scholar.google.com/scholar?q=Louis+L%2E+Thurstone%2E+1927%2E+A+law+of+comparative+judgment%2E+Psychological+review+34%284%29%3A273%2E)
16. Amy Beth Warriner, Victor Kuperman, and Marc Brysbaert. 2013. Norms of valence, arousal, and dominance for 13,915 English lemmas. Behavior Research Methods 45(4):1191–1207. [find](https://scholar.google.com/scholar?q=Amy+Beth+Warriner%2C+Victor+Kuperman%2C+and+Marc+Brysbaert%2E+2013%2E+Norms+of+valence%2C+arousal%2C+and+dominance+for+13%2C915+English+lemmas%2E+Behavior+Research+Methods+45%284%29%3A1191%E2%80%931207%2E)
17. Svetlana Kiritchenko and Saif M. Mohammad. 2016a. Capturing reliable fine-grained sentiment associations by crowdsourcing and best–worst scaling. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL… [find](https://scholar.google.com/scholar?q=Svetlana+Kiritchenko+and+Saif+M%2E+Mohammad%2E+2016a%2E+Capturing+reliable+fine%2Dgrained+sentiment+associations+by+crowdsourcing+and+best%E2%80%93worst+scaling%2E+In+Proceedings+of+the+Annual+Conference+of+the+North+A)
18. Svetlana Kiritchenko and Saif M. Mohammad. 2016b. The effect of negators, modals, and degree adverbs on sentiment composition. In Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis. San Diego, California, pages 43–52. [http://www.aclweb.or…](http://www.aclweb.or…) [↗](http://www.aclweb.or…)
19. Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of psychology . [find](https://scholar.google.com/scholar?q=Rensis+Likert%2E+1932%2E+A+technique+for+the+measurement+of+attitudes%2E+Archives+of+psychology+%2E)
20. Jordan J. Louviere. 1991. Best-worst scaling: A model for the largest difference judgments. Working Paper. [find](https://scholar.google.com/scholar?q=Jordan+J%2E+Louviere%2E+1991%2E+Best%2Dworst+scaling%3A+A+model+for+the+largest+difference+judgments%2E+Working+Paper%2E)