---
title: "Why We Need New Evaluation Metrics for NLG — Novikova 2017"
description: "The authors examine how well automatic NLG evaluation metrics, such as BLEU, agree with human judgements of a… 48 references, cited by 287 papers here."
authors: ["Jekaterina Novikova", "Ondřej Dušek", "Amanda Cercas Curry", "Verena Rieser"]
year: 2017
venue: "Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D17-1238/"
doi: "10.18653/v1/d17-1238"
arxiv: "1707.06875"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/novikova-2017-why-we-need-new-evaluation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/novikova-2017-why-we-need-new-evaluation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/2/) / Why We Need New Evaluation Metrics for NLG

# Why We Need New Evaluation Metrics for NLG

**Authors:** [Jekaterina Novikova](https://md.research.elamur.ai/free/en/authors/jekaterina-novikova/), [Ondřej Dušek](https://md.research.elamur.ai/free/en/authors/ondrej-dusek/), [Amanda Cercas Curry](https://md.research.elamur.ai/free/en/authors/amanda-cercas-curry/) and [Verena Rieser](https://md.research.elamur.ai/free/en/authors/verena-rieser/) · **Year:** 2017 · **Venue:** Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D17-1238/) · [PDF](https://aclanthology.org/D17-1238.pdf) · [DOI 10.18653/v1/d17-1238](https://doi.org/10.18653/v1/d17-1238) · [arXiv 1707.06875](https://arxiv.org/abs/1707.06875)

**Citations:** [287](https://md.research.elamur.ai/free/en/papers/novikova-2017-why-we-need-new-evaluation/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
The authors examine how well automatic NLG evaluation metrics, such as BLEU, agree with human judgements of a system. Particular attention is paid to modern end-to-end generation systems, which learn from data and may produce ungrammatical text. Three generators are compared: RNNLG, TGen and LOLS. They were tested on three crowdsourced corpora for dialogue systems in the restaurant and hotel domains: SFHotel, SFRest and BAGEL. The system outputs were evaluated with 21 metrics. These include word-overlap-based metrics, semantic-similarity metrics, and new grammar-based metrics: readability, number of spelling errors and a parser score. Human ratings were collected separately on three criteria: informativeness, naturalness and quality. The conclusion is that at the sentence level the metrics poorly reflect human judgements, and their behaviour depends on the corpus and the system. At the same time, at the system level they provide a fairly reliable signal and help identify clearly bad outputs. The work argues that NLG needs new metrics, in particular ones that do not depend on reference texts.

## Key points
- In total, 2,460 system outputs were evaluated. Each output was rated by three crowdworkers on a 6-point Likert scale. Inter-annotator agreement is moderate: ICC = 0.45.
- Grammar-based metrics (GBMs) are applied here for the first time to sentence-level NLG evaluation. They do not require reference texts, but they are easy to fool: grammatically correct and readable text unrelated to the input MR will receive a high score.
- The metrics are worst at distinguishing medium-quality outputs from good ones. There are cases where humans give a sentence 6 points, while BLEU, METEOR and other metrics score it very low because of its small overlap with the reference.
- The authors tried combining word-based and grammar-based metrics through ensemble learning. Within a single domain the correlation with human ratings was high, but across domains the model performs poorly.
- The corpora are small: from 404 to 2.3 thousand reference texts per domain. The code, data and detailed analysis results are published on GitHub (jeknov/EMNLP_17_submission).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [✍️ Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/)

## Cite

Cite as: Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry and Verena Rieser. 2017. Why We Need New Evaluation Metrics for NLG. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{novikova2017why,
  title = {{Why We Need New Evaluation Metrics for NLG}},
  author = {Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry and Verena Rieser},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1238},
  eprint = {1707.06875},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d17-1238}
}
```

- [← previous Emotion Intensities in Tweets](https://md.research.elamur.ai/free/en/papers/mohammad-2017-emotion-intensities-tweets/)
- [next → Learning to Translate in Real-time with Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/gu-2017-learning-translate-real-time-neural/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/2/)

## References

48 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anja Belz and Eric Kow. 2011. Discrete vs. continuous rating scales for language evaluation in NLP. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers – Volume 2. Association for Computational Linguistics, Portland, O… [find](https://scholar.google.com/scholar?q=Anja+Belz+and+Eric+Kow%2E+2011%2E+Discrete+vs%2E+continuous+rating+scales+for+language+evaluation+in+NLP%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Human+La)
2. Anja Belz and Ehud Reiter. 2006. Comparing automatic and human evaluation of NLG systems. In Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics. Trento, Italy, pages 313–320. [http://aclweb.org/anthology/E06-1040](http://aclweb.org/anthology/E06-1040). [↗](http://aclweb.org/anthology/E06-1040)
3. Aoife Cahill. 2009. Correlating human and automatic evaluation of a German surface realiser. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers. Association for Computational Linguistics, Suntec, Singapore, pages 97–100. [https://aclweb.org/anthology/P09-2025](https://aclweb.org/anthology/P09-2025). [↗](https://aclweb.org/anthology/P09-2025)
4. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of BLEU in machine translation research. In Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics. Trento, Italy, pages 249–256. [http://aclweb.org/anthology/E06-…](http://aclweb.org/anthology/E06-…) [↗](http://aclweb.org/anthology/E06-…) [our card](https://md.research.elamur.ai/free/en/papers/burch-2006-re-evaluating-role-bleu-machine/)
5. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the Second International Conference on Human Language Technology Research. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, pages 138–145. [http://dl.acm.org/c…](http://dl.acm.org/c…) [↗](http://dl.acm.org/c…)
6. Ondrej Dušek, Jekaterina Novikova, and Verena Rieser. 2017. Referenceless quality estimation for natural language generation. In Proceedings of the 1st Workshop on Learning to Generate Natural Language. [find](https://scholar.google.com/scholar?q=Ondrej+Dus%CC%8Cek%2C+Jekaterina+Novikova%2C+and+Verena+Rieser%2E+2017%2E+Referenceless+quality+estimation+for+natural+language+generation%2E+In+Proceedings+of+the+1st+Workshop+on+Learning+to+Generate+Natural+Langua)
7. Ondřej Dušek and Filip Jurčı́ček. 2015. Training a natural language generator from unaligned data. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Beijing… [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Dus%CC%8Cek+and+Filip+Jurc%CC%8C%C4%B1%CC%81c%CC%8Cek%2E+2015%2E+Training+a+natural+language+generator+from+unaligned+data%2E+In+Proceedings+of+the+53rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+th)
8. Ondřej Dušek and Filip Jurčı́ček. 2016. A context-aware natural language generator for dialogue systems. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue. Association for Computational Linguistics, Los Angeles, CA, USA, pages 185–190. arXiv:1608.0… [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Dus%CC%8Cek+and+Filip+Jurc%CC%8C%C4%B1%CC%81c%CC%8Cek%2E+2016%2E+A+context%2Daware+natural+language+generator+for+dialogue+systems%2E+In+Proceedings+of+the+17th+Annual+Meeting+of+the+Special+Interest+Group+on+Discourse+and+Di)
9. Ondřej Dušek and Filip Jurčı́ček. 2016. Sequence-to-sequence generation for spoken dialogue via deep syntax trees and strings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Berlin, Germany, pages 45–51. arXiv:1606.05491. [http://aclweb.org/anthology/P…](http://aclweb.org/anthology/P…) [↗](http://aclweb.org/anthology/P…)
10. Desmond Elliott and Frank Keller. 2014. Comparing automatic evaluation measures for image description. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Baltimore, MD, USA, pages 452– 457. h… [find](https://scholar.google.com/scholar?q=Desmond+Elliott+and+Frank+Keller%2E+2014%2E+Comparing+automatic+evaluation+measures+for+image+description%2E+In+Proceedings+of+the+52nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volum)
11. Dominic Espinosa, Rajakrishnan Rajkumar, Michael White, and Shoshana Berleant. 2010. Further meta-evaluation of broad-coverage surface realization. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 564–574. htt… [find](https://scholar.google.com/scholar?q=Dominic+Espinosa%2C+Rajakrishnan+Rajkumar%2C+Michael+White%2C+and+Shoshana+Berleant%2E+2010%2E+Further+meta%2Devaluation+of+broad%2Dcoverage+surface+realization%2E+In+Proceedings+of+the+2010+Conference+on+Empirical+M)
12. Rudolf Franz Flesch. 1979. How to write plain English: A book for lawyers and consumers. Harper-Collins. [find](https://scholar.google.com/scholar?q=Rudolf+Franz+Flesch%2E+1979%2E+How+to+write+plain+English%3A+A+book+for+lawyers+and+consumers%2E+Harper%2DCollins%2E)
13. Thomas Francois and Delphine Bernhard, editors. 2014. Recent Advances in Automatic Readability Assessment and Text Simplification, volume 165:2 of International Journal of Applied Linguistics. John Benjamins. http://doi.org/10.1075/itl.165.2. — [doi:10.1075/itl.165.2](https://doi.org/10.1075/itl.165.2) [↗](http://doi.org/10.1075/itl.165.2)
14. Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao, and Bill Dolan. 2015. deltaBLEU: A discriminative metric for generation tasks with intrinsically diverse targets. In Proceedings of the 53rd Annual Meeting of the Association f… [find](https://scholar.google.com/scholar?q=Michel+Galley%2C+Chris+Brockett%2C+Alessandro+Sordoni%2C+Yangfeng+Ji%2C+Michael+Auli%2C+Chris+Quirk%2C+Margaret+Mitchell%2C+Jianfeng+Gao%2C+and+Bill+Dolan%2E+2015%2E+deltaBLEU%3A+A+discriminative+metric+for+generation+task)
15. Jesús Giménez and Lluı́s Màrquez. 2008. A smorgasbord of features for automatic MT evaluation. In Proceedings of the Third Workshop on Statistical Machine Translation. Association for Computational Linguistics, Columbus, OH, USA, pages 195– 198. [http://aclweb.org/anthology/W08-0332](http://aclweb.org/anthology/W08-0332). [↗](http://aclweb.org/anthology/W08-0332)
16. Dimitra Gkatzia, Oliver Lemon, and Verena Rieser. 2016. Natural language generation enhances human decision-making with uncertain information. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Berlin, Germany, pages 264–268. arXiv:16… [find](https://scholar.google.com/scholar?q=Dimitra+Gkatzia%2C+Oliver+Lemon%2C+and+Verena+Rieser%2E+2016%2E+Natural+language+generation+enhances+human+decision%2Dmaking+with+uncertain+information%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Associat)
17. Dimitra Gkatzia and Saad Mahamood. 2015. A snapshot of NLG evaluation practices 2005–2014. In Proceedings of the 15th European Workshop on Natural Language Generation (ENLG). Association for Computational Linguistics, Brighton, UK, pages 57–60. https://doi.org/10.18653/v1/W15-4708. [↗](https://doi.org/10.18653/v1/W15-4708)
18. Aaron L. F. Han, Derek F. Wong, and Lidia S. Chao. 2012. LEPOR: A robust evaluation metric for machine translation with augmented factors. In Proceedings of COLING 2012: Posters. [find](https://scholar.google.com/scholar?q=Aaron+L%2E+F%2E+Han%2C+Derek+F%2E+Wong%2C+and+Lidia+S%2E+Chao%2E+2012%2E+LEPOR%3A+A+robust+evaluation+metric+for+machine+translation+with+augmented+factors%2E+In+Proceedings+of+COLING+2012%3A+Posters%2E)
19. The COLING 2012 Organizing Committee, Mumbai, India, pages 441–450. [http://aclweb.org/anthology/C12-2044](http://aclweb.org/anthology/C12-2044). [↗](http://aclweb.org/anthology/C12-2044)
20. Lushan Han, Abhay Kashyap, Tim Finin, James Mayfield, and Jonathan Weese. 2013. UMBC EBIQUITY-CORE: Semantic textual similarity systems. In Proceedings of the Second Joint Conference on Lexical and Computational Semantics (*SEM). Atlanta, Georgia, volume 1, pages 44–52. [http://aclweb.org/anthology/S…](http://aclweb.org/anthology/S…) [↗](http://aclweb.org/anthology/S…) [our card](https://md.research.elamur.ai/free/en/papers/han-2013-umbc-ebiquity-core-semantic-textual/)
21. Helen Hastie, Heriberto Cuayahuitl, Nina Dethlefs, Simon Keizer, and Xingkun Liu. 2016. Why bother? Is evaluation of NLG in an end-to-end Spoken Dialogue System worth it? In Proceedings of the International Workshop on Spoken Dialogue Systems (IWSDS). Saariselkä, Finland. [find](https://scholar.google.com/scholar?q=Helen+Hastie%2C+Heriberto+Cuayahuitl%2C+Nina+Dethlefs%2C+Simon+Keizer%2C+and+Xingkun+Liu%2E+2016%2E+Why+bother%3F+Is+evaluation+of+NLG+in+an+end%2Dto%2Dend+Spoken+Dialogue+System+worth+it%3F+In+Proceedings+of+the+Interna)
22. Micah Hodosh and Julia Hockenmaier. 2016. Focused evaluation for image description with binary forced-choice tasks. In Proceedings of the 5th Workshop on Vision and Language. Berlin, Germany, pages 19– 28. [http://aclweb.org/anthology/W16-3203](http://aclweb.org/anthology/W16-3203). [↗](http://aclweb.org/anthology/W16-3203)
23. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard H. Hovy. 2013. Learning whom to trust with MACE. In Proceedings of NAACL-HLT. Atlanta, GA, USA, pages 1120–1130. [http://aclweb.org/anthology/N13-1132](http://aclweb.org/anthology/N13-1132). [↗](http://aclweb.org/anthology/N13-1132) [our card](https://md.research.elamur.ai/free/en/papers/hovy-2013-learning-whom-trust-mace/)
24. Min-Yen Kan, Kathleen R. McKeown, and Judith L. Klavans. 2001. Applying natural language generation to indicative summarization. In Proceedings of the 8th European Workshop on Natural Language Generation. Association for Computational Linguistics, Toulouse, France, pages 1–9. https://doi.org/10.3115… [find](https://scholar.google.com/scholar?q=Min%2DYen+Kan%2C+Kathleen+R%2E+McKeown%2C+and+Judith+L%2E+Klavans%2E+2001%2E+Applying+natural+language+generation+to+indicative+summarization%2E+In+Proceedings+of+the+8th+European+Workshop+on+Natural+Language+Generat)
25. Anjuli Kannan and Oriol Vinyals. 2017. Adversarial evaluation of dialogue models. CoRR abs/1701.08198. https://arxiv.org/abs/1701.08198. — [arXiv:1701.08198](https://arxiv.org/abs/1701.08198) [↗](https://arxiv.org/abs/1701.08198)
26. Mert Kilickaya, Aykut Erdem, Nazli Ikizler-Cinbis, and Erkut Erdem. 2017. Re-evaluating automatic metrics for image captioning. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Valencia, Spain.… [find](https://scholar.google.com/scholar?q=Mert+Kilickaya%2C+Aykut+Erdem%2C+Nazli+Ikizler%2DCinbis%2C+and+Erkut+Erdem%2E+2017%2E+Re%2Devaluating+automatic+metrics+for+image+captioning%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Ass)
27. Gerasimos Lampouras and Andreas Vlachos. 2016. Imitation learning for language generation from unaligned data. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. [find](https://scholar.google.com/scholar?q=Gerasimos+Lampouras+and+Andreas+Vlachos%2E+2016%2E+Imitation+learning+for+language+generation+from+unaligned+data%2E+In+Proceedings+of+COLING+2016%2C+the+26th+International+Conference+on+Computational+Linguis)
28. The COLING 2016 Organizing Committee, Osaka, Japan, pages 1101– 1112. [http://aclweb.org/anthology/C16-1105](http://aclweb.org/anthology/C16-1105). [↗](http://aclweb.org/anthology/C16-1105)
29. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. Biometrics 33(1):159–174. https://doi.org/10.2307/2529310. — [doi:10.2307/2529310](https://doi.org/10.2307/2529310) [↗](https://doi.org/10.2307/2529310)
30. Alon Lavie and Abhaya Agarwal. 2007. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation. Association for Computational Linguistics, Prague, Czech Republic, pages 228– 231. http:/… [our card](https://md.research.elamur.ai/free/en/papers/lavie-2007-meteor-automatic-metric-mt-evaluation/)
31. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-04 workshop. Barcelona, Spain, pages 74–81. [http://aclweb.org/anthology/W04-1013](http://aclweb.org/anthology/W04-1013). [↗](http://aclweb.org/anthology/W04-1013) [our card](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/)
32. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natura… [our card](https://md.research.elamur.ai/free/en/papers/liu-2016-how-not-evaluate-your-dialogue/)
33. François Mairesse, Milica Gašić, Filip Jurčı́ček, Simon Keizer, Blaise Thomson, Kai Yu, and Steve Young. 2010. Phrase-based statistical language generation using graphical models and active learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Ass… [find](https://scholar.google.com/scholar?q=Franc%CC%A7ois+Mairesse%2C+Milica+Gas%CC%8Cic%CC%81%2C+Filip+Jurc%CC%8C%C4%B1%CC%81c%CC%8Cek%2C+Simon+Keizer%2C+Blaise+Thomson%2C+Kai+Yu%2C+and+Steve+Young%2E+2010%2E+Phrase%2Dbased+statistical+language+generation+using+graphical+models+and+active+learn)
34. Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2016. What to talk about and how? Selective generation using LSTMs with coarse-to-fine alignment. In Proceedings of NAACL-HLT 2016. San Diego, CA, USA. arXiv:1509.00838. [http://aclweb.org/anthology/N16-1086](http://aclweb.org/anthology/N16-1086). [↗](http://aclweb.org/anthology/N16-1086) [our card](https://md.research.elamur.ai/free/en/papers/mei-2016-what-talk-about-how-selective/)
35. Special Interest Group on Discourse and Dialogue. Saarbrücken, Germany. ArXiv:1706.09254. https://arxiv.org/abs/1706.09254. [↗](https://arxiv.org/abs/1706.09254) [our card](https://md.research.elamur.ai/free/en/papers/novikova-2017-e2e-dataset-new-challenges-end/)
36. Jekaterina Novikova, Oliver Lemon, and Verena Rieser. 2016. Crowd-sourcing NLG data: Pictures elicit better data. In Proceedings of the 9th International Natural Language Generation Conference. Edinburgh, UK, pages 265–273. arXiv:1608.00339. [http://aclweb.org/anthology/W16-2302](http://aclweb.org/anthology/W16-2302). [↗](http://aclweb.org/anthology/W16-2302) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2016-results-wmt16-metrics-shared-task/)
37. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Philadelphia, PA, USA, pages 311–3… [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
38. Ehud Reiter and Anja Belz. 2009. An investigation into the validity of some metrics for automatically evaluating natural language generation systems. Computational Linguistics 35(4):529–558. https://doi.org/10.1162/coli.2009.35.4.35405. [↗](https://doi.org/10.1162/coli.2009.35.4.35405)
39. Verena Rieser, Oliver Lemon, and Simon Keizer. 2014. Natural language generation as incremental planning under uncertainty: Adaptive information presentation for statistical dialogue systems. IEEE/ACM Transactions on Audio, Speech, and Language Processing 22(5):979–993. https://doi.org/10.1109/TASL.… — [doi:10.1109/tasl.2014.2315271](https://doi.org/10.1109/tasl.2014.2315271) [↗](https://doi.org/10.1109/TASL.…)
40. Shikhar Sharma, Jing He, Kaheer Suleman, Hannes Schulz, and Philip Bachman. 2016. Natural language generation in dialogue using lexicalized and delexicalized data. CoRR abs/1606.03632. http://arxiv.org/abs/1606.03632. — [arXiv:1606.03632](https://arxiv.org/abs/1606.03632) [↗](http://arxiv.org/abs/1606.03632)
41. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation of the Americas. Cambridge, MA, USA, pages 223–231. http://mt-arch… [our card](https://md.research.elamur.ai/free/en/papers/snover-2006-study-translation-edit-rate-targeted/)
42. Lucia Specia, Dhwaj Raj, and Marco Turchi. 2010. Machine translation evaluation versus quality estimation. Machine translation 24(1):39–50. https://doi.org/10.1007/s10590-010-9077-2. — [doi:10.1007/s10590-010-9077-2](https://doi.org/10.1007/s10590-010-9077-2) [↗](https://doi.org/10.1007/s10590-010-9077-2)
43. Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2016. There’s no comparison: Reference-less evaluation metrics in grammatical error correction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Austin, TX, USA, pages 2109–2115. arXiv:1610.02124. http… [find](https://scholar.google.com/scholar?q=Courtney+Napoles%2C+Keisuke+Sakaguchi%2C+and+Joel+Tetreault%2E+2016%2E+There%E2%80%99s+no+comparison%3A+Reference%2Dless+evaluation+metrics+in+grammatical+error+correction%2E+In+Proceedings+of+the+2016+Conference+on+Empiri)
44. Amanda Stent, Matthew Marge, and Mohit Singhai. 2005. Evaluating evaluation methods for generation in the presence of variation. In Computational Linguistics and Intelligent Text Processing: 6th International Conference, CICLing 2005, Mexico City, Mexico, February 13-19, 2005. Proceedings. Springer,… — [doi:10.1007/978-3-540-30586-6](https://doi.org/10.1007/978-3-540-30586-6) [find](https://scholar.google.com/scholar?q=Amanda+Stent%2C+Matthew+Marge%2C+and+Mohit+Singhai%2E+2005%2E+Evaluating+evaluation+methods+for+generation+in+the+presence+of+variation%2E+In+Computational+Linguistics+and+Intelligent+Text+Processing%3A+6th+Inter)
45. Jekaterina Novikova, Ondrej Dušek, and Verena Rieser. 2017. The E2E dataset: New challenges for end-to-end generation. In Proceedings of the 18th Annual Meeting of the Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In Proceedin… [our card](https://md.research.elamur.ai/free/en/papers/novikova-2017-e2e-dataset-new-challenges-end/)
46. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Lina Maria Rojas-Barahona, Pei-hao Su, David Vandyke, and Steve J. Young. 2016. Multi-domain neural network language generation for spoken dialogue systems. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Co… [find](https://scholar.google.com/scholar?q=Tsung%2DHsien+Wen%2C+Milica+Gas%CC%8Cic%CC%81%2C+Nikola+Mrks%CC%8Cic%CC%81%2C+Lina+Maria+Rojas%2DBarahona%2C+Pei%2Dhao+Su%2C+David+Vandyke%2C+and+Steve+J%2E+Young%2E+2016%2E+Multi%2Ddomain+neural+network+language+generation+for+spoken+dialogue+sy)
47. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal… [our card](https://md.research.elamur.ai/free/en/papers/wen-2015-semantically-conditioned-lstm-based-natural/)
48. Evan James Williams. 1959. Regression analysis. John Wiley & Sons, New York, NY, USA. [find](https://scholar.google.com/scholar?q=Evan+James+Williams%2E+1959%2E+Regression+analysis%2E+John+Wiley+%26+Sons%2C+New+York%2C+NY%2C+USA%2E)