---
title: "RankME — paper by Novikova et al. (2018) — summary and ideas"
description: "The authors argue that much of this inconsistency comes from task design, not just personal taste. 34 references, cited by 72 papers here."
authors: ["Jekaterina Novikova", "Ondřej Dušek", "Verena Rieser"]
year: 2018
venue: "Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)"
source: "https://aclanthology.org/N18-2012/"
doi: "10.18653/v1/n18-2012"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/novikova-2018-rankme-reliable-human-ratings-natural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/novikova-2018-rankme-reliable-human-ratings-natural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) / RankME: Reliable Human Ratings for Natural Language Generation

# RankME: Reliable Human Ratings for Natural Language Generation

**Authors:** [Jekaterina Novikova](https://md.research.elamur.ai/free/en/authors/jekaterina-novikova/), [Ondřej Dušek](https://md.research.elamur.ai/free/en/authors/ondrej-dusek/) and [Verena Rieser](https://md.research.elamur.ai/free/en/authors/verena-rieser/) · **Year:** 2018 · **Venue:** [Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2018](https://md.research.elamur.ai/free/en/venues/naacl/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N18-2012/) · [PDF](https://aclanthology.org/N18-2012.pdf) · [DOI 10.18653/v1/n18-2012](https://doi.org/10.18653/v1/n18-2012) · [OpenAlex](https://openalex.org/W2793081411)

**Citations:** 18 citations (OpenAlex, 2026-10-14) · [72](https://md.research.elamur.ai/free/en/papers/novikova-2018-rankme-reliable-human-ratings-natural/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Human ratings are the main way to evaluate natural language generation (NLG) systems, but they are often unreliable. Raters disagree on the same output, blur separate criteria together, and absolute scores get used to compare systems. The authors argue that much of this inconsistency comes from task design, not just personal taste. They compared three elicitation methods: 6-point Likert scales, plain magnitude estimation (ME), and their new rank-based magnitude estimation (RankME). RankME joins continuous ME scales with relative ranking of several outputs. They also compared rating all criteria in one task against rating each criterion in its own task. Crowd workers on CrowdFlower rated outputs from three E2E NLG challenge systems. Rating criteria separately reduced the spurious correlation between naturalness and the other criteria. RankME gave the highest inter-annotator agreement and the most discriminative system rankings. Combined with TrueSkill, it scales cheaply to many systems. This matters for error analysis and for building better automatic metrics.

## Key points
- **Data:** the first 100 outputs each from Sheffield NLP, Slug2Slug and the TGen baseline, all seq2seq systems from the E2E NLG challenge. Three crowd workers rated every output on naturalness, informativeness and overall quality.
- **Agreement (ICC, separate-task setup):** for naturalness, RankME reached 0.42, plain ME 0.27 and Likert 0.12. For quality, the scores were 0.68, 0.60 and 0.41 respectively. For informativeness, Likert already agreed well.
- **Separating criteria:** rating criteria in separate tasks pushed naturalness correlations close to zero. Informativeness and quality stayed strongly correlated, probably because raters saw the meaning representation in both tasks.
- **Added vs. missed information:** a Chi-squared test showed the two error types differ significantly (p \< 0.01). They also yield different system rankings, so missed information deserves its own criterion.
- **Rankings and resources:** RankME was the only method to give a clear ranking for overall quality. TrueSkill applied to RankME comparisons reproduced the same rankings for all three criteria. Code and data are at https://github.com/jeknov/RankME.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Heriot-Watt University](https://md.research.elamur.ai/free/en/universities/heriot-watt-university/) — University, Edinburgh, United Kingdom · 10 works · 5,081 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [✍️ Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) · 145 works

## Cite

Cite as: Jekaterina Novikova, Ondřej Dušek and Verena Rieser. 2018. RankME: Reliable Human Ratings for Natural Language Generation. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{novikova2018rankme,
  title = {{RankME: Reliable Human Ratings for Natural Language Generation}},
  author = {Jekaterina Novikova, Ondřej Dušek and Verena Rieser},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-2012},
  url = {https://doi.org/10.18653/v1/n18-2012}
}
```

- [← previous Twitter Universal Dependency Parsing for African-American and Mainstream American English](https://md.research.elamur.ai/free/en/papers/blodgett-2018-twitter-universal-dependency-parsing-african/)
- [next → CASCADE: Contextual Sarcasm Detection in Online Discussion Forums](https://md.research.elamur.ai/free/en/papers/hazarika-2018-cascade-contextual-sarcasm-detection-online/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/)

## References

34 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Nina Dethlefs, Heriberto Cuayáhuitl, Helen Hastie, Verena Rieser, and Oliver Lemon. 2014. Cluster-based prediction of user ratings for stylistic surface realisation. In Proceedings of the European Chapter of the Association for Computational Linguistics (EACL). Gothenburg, Sweden, pages 702–711. ht… [find](https://scholar.google.com/scholar?q=Nina+Dethlefs%2C+Heriberto+Cuaya%CC%81huitl%2C+Helen+Hastie%2C+Verena+Rieser%2C+and+Oliver+Lemon%2E+2014%2E+Cluster%2Dbased+prediction+of+user+ratings+for+stylistic+surface+realisation%2E+In+Proceedings+of+the+European+Ch)
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. [Neural Machine Translation by Jointly Learning to Align and Translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). In 3rd International Conference on Learning Representations (ICLR). San Diego, CA, USA. ArXiv: 1409.0473. http://arxiv.org/abs/1409.0473. [↗](http://arxiv.org/abs/1409.0473)
3. Ellen Gurman Bard, Dan Robertson, and Antonella Sorace. 1996. Magnitude estimation of linguistic acceptability. Language 72:32–68. https://doi.org/10.2307/416793. — [doi:10.2307/416793](https://doi.org/10.2307/416793) [↗](https://doi.org/10.2307/416793)
4. Anja Belz and Eric Kow. 2010. Comparing rating scales and preference judgements in language evaluation. In Proceedings of the 6th International Natural Language Generation Conference (INLG). Trim, Ireland, pages 7–15. [http://aclweb.org/](http://aclweb.org/) anthology/W10-4201. [↗](http://aclweb.org/)
5. Mark Dras. 2015. Evaluating human pairwise preference judgements. Computational Linguistics 41(2):337–345. https://doi.org/10.1162/ COLI_a_00222. [find](https://scholar.google.com/scholar?q=Mark+Dras%2E+2015%2E+Evaluating+human+pairwise+preference+judgements%2E+Computational+Linguistics+41%282%29%3A337%E2%80%93345%2E+https%3A%2F%2Fdoi%2Eorg%2F10%2E1162%2F+COLI%5Fa%5F00222%2E)
6. Ondřej Dušek and Filip Jurčı́ček. 2016. [Sequence-to-sequence generation for spoken dialogue via deep syntax trees and strings](https://md.research.elamur.ai/free/en/papers/dusek-2016-sequence-sequence-generation-spoken-dialogue/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Berlin, Germany, pages 45–51. arXiv:1606.05491. http…
7. Anja Belz and Eric Kow. 2011. Discrete vs. continuous rating scales for language evaluation in NLP. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Short papers. Portland, OR, USA, pages 230–235. [http://aclweb.org/anthology/P11-2040](http://aclweb.org/anthology/P11-2040). [↗](http://aclweb.org/anthology/P11-2040)
8. Ondřej Dušek, Jekaterina Novikova, and Verena Rieser. 2017. Referenceless Quality Estimation for Natural Language Generation. In Proceedings of the 1st Workshop on Learning to Generate Natural Language (LGNL). Sydney, Australia. ArXiv: 1708.01759. http://arxiv.org/abs/1708. 01759. — [arXiv:1708.01759](https://arxiv.org/abs/1708.01759) [↗](http://arxiv.org/abs/1708)
9. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, et al. 2017. [Findings of the 2017 conference on machine translation (WMT17)](https://md.research.elamur.ai/free/en/papers/bojar-2017-findings-2017-conference-machine-translation/). In Proceedings of the Second Conference on Machine Translation (WMT…
10. Albert Gatt and Emiel Krahmer. 2017. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. Journal of Artificial Intelligence Research (JAIR) 60. https://arxiv.org/abs/1703.09902. [↗](https://arxiv.org/abs/1703.09902)
11. Dimitra Gkatzia, Oliver Lemon, and Verena Rieser. 2016. Natural language generation enhances human decision-making with uncertain information. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Berlin, Germany, pages 264– 268. arXiv:1… [find](https://scholar.google.com/scholar?q=Dimitra+Gkatzia%2C+Oliver+Lemon%2C+and+Verena+Rieser%2E+2016%2E+Natural+language+generation+enhances+human+decision%2Dmaking+with+uncertain+information%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Associat)
12. [Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi,…](https://md.research.elamur.ai/free/en/papers/bojar-2016-findings-2016-conference-machine-translation/)
13. Dimitra Gkatzia and Saad Mahamood. 2015. A snapshot of NLG evaluation practices 2005–2014. In Proceedings of the 15th European Workshop on Natural Language Generation (ENLG). Association for Computational Linguistics, Brighton, UK, pages 57– 60. https://doi.org/10.18653/v1/W15-4708. [↗](https://doi.org/10.18653/v1/W15-4708)
14. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. [(Meta-) evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/burch-2007-meta-evaluation-machine-translation/). In Proceedings of the Second Workshop on Statistical Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous Measurement Scales in Hum…
15. Ralf Herbrich, Tom Minka, and Thore Graepel. 2006. Trueskill TM : a Bayesian skill rating system. In Proceedings of the 19th International Conference on Neural Iinformation Processing Systems (NIPS). Vancouver, Canada, pages 569– 576. [http://papers.nips.cc/paper/](http://papers.nips.cc/paper/) 3079-trueskilltm-a-bayesian-skill-ra… [↗](http://papers.nips.cc/paper/)
16. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017a. [Why we need new evaluation metrics for NLG](https://md.research.elamur.ai/free/en/papers/novikova-2017-why-we-need-new-evaluation/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 2231– 2242. [http://aclweb.org/anthology/](http://aclweb.org/anthology/) D17-1237. [↗](http://aclweb.org/anthology/)
17. Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017b. [The E2E dataset: New challenges for end-to-end generation](https://md.research.elamur.ai/free/en/papers/novikova-2017-e2e-dataset-new-challenges-end/). In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue (SIG-DIAL). pages 201–206. [http://aclweb.org/](http://aclweb.org/) anthology/W17-5525. [↗](http://aclweb.org/)
18. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735– 1780. https://doi.org/10.1162/neco. 1997.9.8.1735. [↗](https://doi.org/10.1162/neco)
19. Juraj Juraska, Panagiotis Karagiannis, Kevin K. Bowden, and Marilyn A. Walker. 2018. A Deep Ensemble Model with Slot Alignment for Sequence-to-Sequence Natural Language Generation. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL). To appear. [find](https://scholar.google.com/scholar?q=Juraj+Juraska%2C+Panagiotis+Karagiannis%2C+Kevin+K%2E+Bowden%2C+and+Marilyn+A%2E+Walker%2E+2018%2E+A+Deep+Ensemble+Model+with+Slot+Alignment+for+Sequence%2Dto%2DSequence+Natural+Language+Generation%2E+In+Proceedings+of+t)
20. Jekaterina Novikova, Oliver Lemon, and Verena Rieser. 2016. Crowd-sourcing NLG data: Pictures elicit better data. In Proceedings of the 9th International Natural Language Generation conference (INLG). Edinburgh, UK, pages 265–273. [http://aclweb.org/anthology/W16-6644](http://aclweb.org/anthology/W16-6644). [↗](http://aclweb.org/anthology/W16-6644)
21. Verena Rieser, Oliver Lemon, and Simon Keizer. 2014. Natural language generation as incremental planning under uncertainty: Adaptive information presentation for statistical dialogue systems. IEEE/ACM Transactions on Audio, Speech, and Language Processing 22(5):979–993. https://doi. org/10.1109/TASL… — [doi:10.1109/tasl.2014.2315271](https://doi.org/10.1109/tasl.2014.2315271) [find](https://scholar.google.com/scholar?q=Verena+Rieser%2C+Oliver+Lemon%2C+and+Simon+Keizer%2E+2014%2E+Natural+language+generation+as+incremental+planning+under+uncertainty%3A+Adaptive+information+presentation+for+statistical+dialogue+systems%2E+IEEE%2FACM)
22. Philipp Koehn. 2004. [Statistical significance tests for machine translation evaluation](https://md.research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/). In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (EMNLP). Barcelona, Spain, pages 388–395. [http://aclweb.org/](http://aclweb.org/) anthology/W04-3250. [↗](http://aclweb.org/)
23. Keisuke Sakaguchi, Matt Post, and Benjamin Van Durme. 2014. Efficient elicitation of annotations for human evaluation of machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation (WMT). Baltimore, MD, USA, pages 1–11. [http://aclweb.org/anthology/W14-3301](http://aclweb.org/anthology/W14-3301). [↗](http://aclweb.org/anthology/W14-3301)
24. Philipp Koehn. 2010. Statistical machine translation. Cambridge University Press, Cambridge; New York. http://dx.doi.org/10.1017/ CBO9780511815829. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2E+2010%2E+Statistical+machine+translation%2E+Cambridge+University+Press%2C+Cambridge%3B+New+York%2E+http%3A%2F%2Fdx%2Edoi%2Eorg%2F10%2E1017%2F+CBO9780511815829%2E)
25. Shikhar Sharma, Jing He, Kaheer Suleman, Hannes Schulz, and Philip Bachman. 2016. Natural language generation in dialogue using lexicalized and delexicalized data. CoRR abs/1606.03632. http://arxiv.org/abs/1606.03632. — [arXiv:1606.03632](https://arxiv.org/abs/1606.03632) [↗](http://arxiv.org/abs/1606.03632)
26. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics 33(1):159–174. https://doi.org/ 10.2307/2529310. — [doi:10.2307/2529310](https://doi.org/10.2307/2529310) [find](https://scholar.google.com/scholar?q=J%2E+Richard+Landis+and+Gary+G%2E+Koch%2E+1977%2E+The+measurement+of+observer+agreement+for+categorical+data%2E+Biometrics+33%281%29%3A159%E2%80%93174%2E+https%3A%2F%2Fdoi%2Eorg%2F+10%2E2307%2F2529310%2E+%E2%80%94+%5Bdoi%3A10%2E2307%2F2529310%5D%28https%3A%2F%2Fdoi%2Eor)
27. Advaith Siddharthan, Matthew Green, Kees van Deemter, Chris Mellish, and René van der Wal. 2012. Blogging birds: Generating narratives about reintroduced species to promote public engagement. In Proceedings of the Seventh International Natural Language Generation Conference (INLG). Utica, IL, USA,… [find](https://scholar.google.com/scholar?q=Advaith+Siddharthan%2C+Matthew+Green%2C+Kees+van+Deemter%2C+Chris+Mellish%2C+and+Rene%CC%81+van+der+Wal%2E+2012%2E+Blogging+birds%3A+Generating+narratives+about+reintroduced+species+to+promote+public+engagement%2E+In+Proc)
28. Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. [Towards an automatic turing test: Learning to evaluate dialogue responses](https://md.research.elamur.ai/free/en/papers/lowe-2017-towards-automatic-turing-test-learning/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P…
29. Advaith Siddharthan and Napoleon Katsos. 2012. Offline sentence processing measures for testing readability with users. In Proceedings of the NAACL-HLT 2012 Workshop on Predicting and Improving Text Readability (PITR). Montréal, Canada, pages 17–24. [http://aclweb.org/anthology/](http://aclweb.org/anthology/) W12-2203. [↗](http://aclweb.org/anthology/)
30. Elena Manishina, Bassam Jabaian, Stéphane Huet, and Fabrice Lefevre. 2016. Automatic corpus extension for data-driven natural language generation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC). Portorož, Slovenia, pages 3624–3631. [http://www.lrec…](http://www.lrec…) [↗](http://www.lrec…)
31. Marilyn Walker, Amanda Stent, François Mairesse, and Rashmi Prasad. 2007. Individual and domain adaptation in sentence planning for dialogue. Journal of Artificial Intelligence Research (JAIR) 30(1):413–456. https://doi.org/10. 1613/jair.2329. [find](https://scholar.google.com/scholar?q=Marilyn+Walker%2C+Amanda+Stent%2C+Franc%CC%A7ois+Mairesse%2C+and+Rashmi+Prasad%2E+2007%2E+Individual+and+domain+adaptation+in+sentence+planning+for+dialogue%2E+Journal+of+Artificial+Intelligence+Research+%28JAIR%29+30%281%29%3A)
32. Tsung-Hsien Wen, Milica Gasić, Dongho Kim, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015a. [Stochastic language generation in dialogue using recurrent neural networks with convolutional sentence reranking](https://md.research.elamur.ai/free/en/papers/wen-2015-stochastic-language-generation-dialogue-recurrent/). In Proceedings of the 16th Annual Meeting of the Special Interest Group on…
33. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015b. [Semantically conditioned LSTM-based natural language generation for spoken dialogue systems](https://md.research.elamur.ai/free/en/papers/wen-2015-semantically-conditioned-lstm-based-natural/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Lisbon,…
34. Sam Wiseman, Stuart M. Shieber, and Alexander M. Rush. 2017. [Challenges in data-to-document generation](https://md.research.elamur.ai/free/en/papers/wiseman-2017-challenges-data-document-generation/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017. Copenhagen, Denmark, pages 2253– 2263. https://acl…

*Updated 24 October 2026*