---
title: "A Comprehensive Assessment of Dialog Evaluation Metrics (2021)"
description: "Researchers have recently proposed many dialog-specific automatic evaluation metrics because… 69 references, cited by 68 papers here."
authors: ["Yi-Ting Yeh", "Maxine Eskenazi", "Shikib Mehri"]
year: 2021
venue: "The First Workshop on Evaluations and Assessments of Neural Conversation Systems"
source: "https://aclanthology.org/2021.eancs-1.3/"
doi: "10.18653/v1/2021.eancs-1.3"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yeh-2021-comprehensive-assessment-dialog-evaluation-metrics/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yeh-2021-comprehensive-assessment-dialog-evaluation-metrics/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / A Comprehensive Assessment of Dialog Evaluation Metrics

# A Comprehensive Assessment of Dialog Evaluation Metrics

**Authors:** Yi-Ting Yeh, [Maxine Eskenazi](https://md.research.elamur.ai/free/en/authors/maxine-eskenazi/) and [Shikib Mehri](https://md.research.elamur.ai/free/en/authors/shikib-mehri/) · **Year:** 2021 · **Venue:** The First Workshop on Evaluations and Assessments of Neural Conversation Systems · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.eancs-1.3/) · [PDF](https://aclanthology.org/2021.eancs-1.3.pdf) · [DOI 10.18653/v1/2021.eancs-1.3](https://doi.org/10.18653/v1/2021.eancs-1.3) · [OpenAlex](https://openalex.org/W3172728711)

**Citations:** 60 citations (OpenAlex, 2026-10-14) · [68](https://md.research.elamur.ai/free/en/papers/yeh-2021-comprehensive-assessment-dialog-evaluation-metrics/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Researchers have recently proposed many dialog-specific automatic evaluation metrics because word-overlap scores such as BLEU track human judgement poorly for conversation. These metrics were introduced around the same time and were each tested on different data, so nobody could tell how they compare. This paper benchmarks 23 metrics (rule-based, embedding-based and learned, both reference-based and reference-free) on 10 human-annotated datasets drawn from USR, GRADE, HolisticEval, FED, DSTC6 and DSTC9. It measures correlation with human scores:
- at the turn level and the dialog/system level,
- for separate quality aspects,
- for different context lengths,
- for simple averaged ensembles of metrics.

No metric wins everywhere. Learned metrics usually perform best on data similar to their training data. Word-overlap metrics can spot clearly bad replies but fail on outputs from strong generators. Averaging several metrics is a cheap way to gain robustness. The released code (DialEvalMetrics) gives future metrics a common test bench, which matters because development-time evaluation drives model and hyperparameter choices.

## Key points
- Scope: 23 metrics, 11 of them reference-free, on 10 quality-annotated datasets. Reference-based metrics (e.g., BLEU, METEOR, BERTScore) could only be scored where a human reference response exists.
- On referenced data, USR, GRADE, DEB and USL-H gave the strongest turn-level correlations. For Spearman correlation at the turn level, the extracted table reports:
  - USR at about 0.42 on USR-TopicalChat.
  - USL-H at about 0.52 on USR-PersonaChat. The authors attribute USL-H's edge over USR here to its VUP grammaticality component.
- Rule-based overlap metrics did unexpectedly well on the USR data but poorly on the GRADE data. GRADE responses come from stronger generators such as Transformer Seq2Seq and DialoGPT, which demand semantic rather than lexical judgement.
- Effect of context length:
  - Most BERT-based reference-free metrics lose accuracy as the context grows.
  - GPT-2-based ones (HolisticEval, DialogRPT, FED) improve.
  - USL-H, DEB and PredictiveEngage stay roughly stable.
- Ensembles: plain averaging of top metrics (USR, GRADE, USL-H, DEB) generally helps. An "All" average over every assessed metric performed surprisingly well across the referenced datasets.

## Interesting ideas
- **Test across generator quality, not just across domains.** Separating annotated data by how strong the response-producing system was shows which metrics only catch obvious failures. The same check applies to any generation metric, such as translation or summarization scorers for Ukrainian.
- **Bucket results by input length.** Grouping samples into context-length intervals and recomputing correlation per bucket exposes length sensitivity that a single overall score hides. This is a cheap diagnostic for any context-dependent scorer.
- **Score each quality aspect against its matching sub-score.** When a metric outputs a specific dimension (e.g., engagingness), compare it with the matching human dimension rather than the overall rating. This gives a fairer picture of multi-faceted metrics.
- **Use naive averaging as an ensemble baseline.** Simply averaging normalized scores from several metrics already beat many individual metrics. It is a strong, almost free baseline before trying learned combinations.
- **Treat in-domain wins with suspicion.** Metrics shone on the datasets they were developed or tuned on, so evaluating on several independently collected annotated sets is a useful guard against overfitting any evaluation tool.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 365 works · 158,048 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works · [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 401 works

## Cite

Cite as: Yi-Ting Yeh, Maxine Eskenazi and Shikib Mehri. 2021. A Comprehensive Assessment of Dialog Evaluation Metrics. The First Workshop on Evaluations and Assessments of Neural Conversation Systems.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yeh2021comprehensive,
  title = {{A Comprehensive Assessment of Dialog Evaluation Metrics}},
  author = {Yi-Ting Yeh, Maxine Eskenazi and Shikib Mehri},
  booktitle = {The First Workshop on Evaluations and Assessments of Neural Conversation Systems},
  year = {2021},
  doi = {10.18653/v1/2021.eancs-1.3},
  url = {https://doi.org/10.18653/v1/2021.eancs-1.3}
}
```

- [← previous Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark](https://md.research.elamur.ai/free/en/papers/niklaus-2021-swiss-judgment-prediction-multilingual-legal/)
- [next → Improving Factual Consistency of Abstractive Summarization via Question Answering](https://md.research.elamur.ai/free/en/papers/nan-2021-improving-factual-consistency-abstractive-summarization/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

69 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. D. Adiwardana, Minh-Thang Luong, David R. So, J. Hall, Noah Fiedel, R. Thoppilan, Z. Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020a. [Towards a human-like open-domain chatbot](https://md.research.elamur.ai/free/en/papers/adiwardana-2020-towards-human-like-open-domain/). ArXiv, abs/2001.09977.
2. Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al. 2019. [The second conversational intelligence challenge (convai2)](https://md.research.elamur.ai/free/en/papers/dinan-2019-second-conversational-intelligence-challenge-convai2/). arXiv preprint arXiv:1902.00098.
3. Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020b. [Towards a human-like open-domain chatbot](https://md.research.elamur.ai/free/en/papers/adiwardana-2020-towards-human-like-open-domain/). arXiv preprint arXiv:2001.09977.
4. Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao, and Bill Dolan. 2015. deltaBLEU: A discriminative metric for generation tasks with intrinsically diverse targets. In Proceedings of the 53rd Annual Meeting of the Association f… [find](https://scholar.google.com/scholar?q=Michel+Galley%2C+Chris+Brockett%2C+Alessandro+Sordoni%2C+Yangfeng+Ji%2C+Michael+Auli%2C+Chris+Quirk%2C+Margaret+Mitchell%2C+Jianfeng+Gao%2C+and+Bill+Dolan%2E+2015%2E+deltaBLEU%3A+A+discriminative+metric+for+generation+task)
5. R Ananthakrishnan, Pushpak Bhattacharyya, M Sasikumar, and Ritesh M Shah. 2007. Some issues in automatic evaluation of english-hindi mt: more blues for bleu. ICON. [find](https://scholar.google.com/scholar?q=R+Ananthakrishnan%2C+Pushpak+Bhattacharyya%2C+M+Sasikumar%2C+and+Ritesh+M+Shah%2E+2007%2E+Some+issues+in+automatic+evaluation+of+english%2Dhindi+mt%3A+more+blues+for+bleu%2E+ICON%2E)
6. Satanjeev Banerjee and Alon Lavie. 2005. [METEOR: An automatic metric for MT evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Ass…
7. Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett, and Bill Dolan. 2020. Dialogue response rank-ingtraining with large-scale human feedback data. In EMNLP. [find](https://scholar.google.com/scholar?q=Xiang+Gao%2C+Yizhe+Zhang%2C+Michel+Galley%2C+Chris+Brockett%2C+and+Bill+Dolan%2E+2020%2E+Dialogue+response+rank%2Dingtraining+with+large%2Dscale+human+feedback+data%2E+In+EMNLP%2E)
8. Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. 2020. [PLATO: Pre-trained dialogue generation model with discrete latent variable](https://md.research.elamur.ai/free/en/papers/bao-2020-plato-pre-trained-dialogue-generation/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 85–96, Online. Association for Computational Linguistics.
9. Sarik Ghazarian, Johnny Wei, Aram Galstyan, and Nanyun Peng. 2019. Better automatic evaluation of open-domain dialogue systems with contextualized embeddings. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 82–89, Minneapolis, Minnesota. Asso… [find](https://scholar.google.com/scholar?q=Sarik+Ghazarian%2C+Johnny+Wei%2C+Aram+Galstyan%2C+and+Nanyun+Peng%2E+2019%2E+Better+automatic+evaluation+of+open%2Ddomain+dialogue+systems+with+contextualized+embeddings%2E+In+Proceedings+of+the+Workshop+on+Methods)
10. Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Langua… [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+van+Merri%C3%ABnboer%2C+Caglar+Gulcehre%2C+Dzmitry+Bahdanau%2C+Fethi+Bougares%2C+Holger+Schwenk%2C+and+Yoshua+Bengio%2E+2014%2E+Learning+phrase+representations+using+RNN+encoder%E2%80%93decoder+for+statistic)
11. Sarik Ghazarian, Ralph Weischedel, Aram Galstyan, and Nanyun Peng. 2020. Predictive engagement: An efficient metric for automatic evaluation of open-domain dialogue systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7789–7796. [find](https://scholar.google.com/scholar?q=Sarik+Ghazarian%2C+Ralph+Weischedel%2C+Aram+Galstyan%2C+and+Nanyun+Peng%2E+2020%2E+Predictive+engagement%3A+An+efficient+metric+for+automatic+evaluation+of+open%2Ddomain+dialogue+systems%2E+In+Proceedings+of+the+AAAI)
12. Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tür. 2019. [find](https://scholar.google.com/scholar?q=Karthik+Gopalakrishnan%2C+Behnam+Hedayatnia%2C+Qinlang+Chen%2C+Anna+Gottardi%2C+Sanjeev+Kwatra%2C+Anu+Venkatesh%2C+Raefer+Gabriel%2C+and+Dilek+Hakkani%2DT%C3%BCr%2E+2019%2E)
13. Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations. In Proc. Interspeech 2019, pages 1891–1895. [find](https://scholar.google.com/scholar?q=Topical%2DChat%3A+Towards+Knowledge%2DGrounded+Open%2DDomain+Conversations%2E+In+Proc%2E+Interspeech+2019%2C+pages+1891%E2%80%931895%2E)
14. Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. 2021. Survey on evaluation methods for dialogue systems. Artificial Intelligence Review, 54(1):755–810. [find](https://scholar.google.com/scholar?q=Jan+Deriu%2C+Alvaro+Rodrigo%2C+Arantxa+Otegi%2C+Guillermo+Echegoyen%2C+Sophie+Rosset%2C+Eneko+Agirre%2C+and+Mark+Cieliebak%2E+2021%2E+Survey+on+evaluation+methods+for+dialogue+systems%2E+Artificial+Intelligence+Review%2C)
15. Jan Deriu, Don Tuggener, Pius von Däniken, Jon Ander Campos, Alvaro Rodrigo, Thiziri Belkacem, Aitor Soroa, Eneko Agirre, and Mark Cieliebak. 2020. Spot the bot: A robust and efficient framework for Jian Guan and Minlie Huang. 2020. UNION: An Unreferenced Metric for Evaluating Open-ended Story Gener… [find](https://scholar.google.com/scholar?q=Jan+Deriu%2C+Don+Tuggener%2C+Pius+von+D%C3%A4niken%2C+Jon+Ander+Campos%2C+Alvaro+Rodrigo%2C+Thiziri+Belkacem%2C+Aitor+Soroa%2C+Eneko+Agirre%2C+and+Mark+Cieliebak%2E+2020%2E+Spot+the+bot%3A+A+robust+and+efficient+framework+for+J)
16. Zekang Li, Jinchao Zhang, Zhengcong Fei, Yang Feng, and Jie Zhou. 2021b. Conversations are not flat: Modeling the intrinsic information flow between dialogue utterances. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zekang+Li%2C+Jinchao+Zhang%2C+Zhengcong+Fei%2C+Yang+Feng%2C+and+Jie+Zhou%2E+2021b%2E+Conversations+are+not+flat%3A+Modeling+the+intrinsic+information+flow+between+dialogue+utterances%2E+In+Proceedings+of+the+59th+Ann)
17. Chulaka Gunasekara, Seokhwan Kim, Luis Fernando D’Haro, Abhinav Rastogi, Yun-Nung Chen, Mihail Eric, Behnam Hedayatnia, Karthik Gopalakrishnan, Yang Liu, Chao-Wei Huang, et al. 2021. Overview of the ninth dialog system technology challenge: Dstc9. Proceedings of the 9th Dialog System Technology Chal… [find](https://scholar.google.com/scholar?q=Chulaka+Gunasekara%2C+Seokhwan+Kim%2C+Luis+Fernando+D%E2%80%99Haro%2C+Abhinav+Rastogi%2C+Yun%2DNung+Chen%2C+Mihail+Eric%2C+Behnam+Hedayatnia%2C+Karthik+Gopalakrishnan%2C+Yang+Liu%2C+Chao%2DWei+Huang%2C+et+al%2E+2021%2E+Overview+of+the+n)
18. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
19. Fenfei Guo, A. Metallinou, Chandra Khatri, Anirudh Raju, Anu Venkatesh, and A. Ram. 2018. Topic-based evaluation for conversational bots. ArXiv, abs/1801.03622. — [arXiv:1801.03622](https://arxiv.org/abs/1801.03622) [find](https://scholar.google.com/scholar?q=Fenfei+Guo%2C+A%2E+Metallinou%2C+Chandra+Khatri%2C+Anirudh+Raju%2C+Anu+Venkatesh%2C+and+A%2E+Ram%2E+2018%2E+Topic%2Dbased+evaluation+for+conversational+bots%2E+ArXiv%2C+abs%2F1801%2E03622%2E+%E2%80%94+%5BarXiv%3A1801%2E03622%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
20. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016a. [How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation](https://md.research.elamur.ai/free/en/papers/liu-2016-how-not-evaluate-your-dialogue/). In Proceedings of the 2016 Conference on Empirical Methods in Natural…
21. Michael Gutmann and Aapo Hyvärinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 297–304. JMLR Workshop and Conference Proceedings. [find](https://scholar.google.com/scholar?q=Michael+Gutmann+and+Aapo+Hyv%C3%A4rinen%2E+2010%2E+Noise%2Dcontrastive+estimation%3A+A+new+estimation+principle+for+unnormalized+statistical+models%2E+In+Proceedings+of+the+Thirteenth+International+Conference+on+Art)
22. Chiori Hori and Takaaki Hori. 2017. End-to-end conversation modeling track in dstc6. arXiv:1706.07440. — [arXiv:1706.07440](https://arxiv.org/abs/1706.07440) [find](https://scholar.google.com/scholar?q=Chiori+Hori+and+Takaaki+Hori%2E+2017%2E+End%2Dto%2Dend+conversation+modeling+track+in+dstc6%2E+arXiv%3A1706%2E07440%2E+%E2%80%94+%5BarXiv%3A1706%2E07440%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1706%2E07440%29)
23. Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016b. [How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation](https://md.research.elamur.ai/free/en/papers/liu-2016-how-not-evaluate-your-dialogue/). arXiv preprint arXiv:1603.08023.
24. Lishan Huang, Zheng Ye, Jinghui Qin, Liang Lin, and Xiaodan Liang. 2020. [GRADE: Automatic graph-enhanced coherence metric for evaluating open-domain dialogue systems](https://md.research.elamur.ai/free/en/papers/huang-2020-grade-automatic-graph-enhanced-coherence/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9230–9240, Online. Associati…
25. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
26. Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. arXiv preprint arXiv:1506.06726. [find](https://scholar.google.com/scholar?q=Ryan+Kiros%2C+Yukun+Zhu%2C+Ruslan+Salakhutdinov%2C+Richard+S+Zemel%2C+Antonio+Torralba%2C+Raquel+Urtasun%2C+and+Sanja+Fidler%2E+2015%2E+Skip%2Dthought+vectors%2E+arXiv+preprint+arXiv%3A1506%2E06726%2E)
27. Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. [Towards an automatic Turing test: Learning to evaluate dialogue responses](https://md.research.elamur.ai/free/en/papers/lowe-2017-towards-automatic-turing-test-learning/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P…
28. Tian Lan, Xian-Ling Mao, Wei Wei, Xiaoyan Gao, and Heyan Huang. 2020. Pone: A novel automatic evaluation metric for open-domain generative dialogue systems. ACM Trans. Inf. Syst., 39(1). [find](https://scholar.google.com/scholar?q=Tian+Lan%2C+Xian%2DLing+Mao%2C+Wei+Wei%2C+Xiaoyan+Gao%2C+and+Heyan+Huang%2E+2020%2E+Pone%3A+A+novel+automatic+evaluation+metric+for+open%2Ddomain+generative+dialogue+systems%2E+ACM+Trans%2E+Inf%2E+Syst%2E%2C+39%281%29%2E)
29. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. [A diversity-promoting objective function for neural conversation models](https://md.research.elamur.ai/free/en/papers/li-2016-diversity-promoting-objective-function-neural/). In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pag…
30. Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. [The Ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems](https://md.research.elamur.ai/free/en/papers/lowe-2015-ubuntu-dialogue-corpus-large-dataset/). In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 285–294, Prague, Czech Republi…
31. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. [DailyDialog: A manually labelled multi-turn dialogue dataset](https://md.research.elamur.ai/free/en/papers/li-2017-dailydialog-manually-labelled-multi-turn/). In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986–995, Taipei, Taiwan. Asian Federati…
32. Joel Mackenzie, Rodger Benham, Matthias Petri, Johanne R Trippas, J Shane Culpepper, and Alistair Moffat. 2020. Cc-news-en: A large english news corpus. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 3077–3084. [find](https://scholar.google.com/scholar?q=Joel+Mackenzie%2C+Rodger+Benham%2C+Matthias+Petri%2C+Johanne+R+Trippas%2C+J+Shane+Culpepper%2C+and+Alistair+Moffat%2E+2020%2E+Cc%2Dnews%2Den%3A+A+large+english+news+corpus%2E+In+Proceedings+of+the+29th+ACM+International+Co)
33. Zekang Li, Jinchao Zhang, Zhengcong Fei, Yang Feng, and Jie Zhou. 2021a. Addressing inquiries about history: An efficient and practical framework for evaluating open-domain chatbot consistency. In Findings Shikib Mehri and Maxine Eskenazi. 2020a. Unsupervised evaluation of interactive dialog with Di… [find](https://scholar.google.com/scholar?q=Zekang+Li%2C+Jinchao+Zhang%2C+Zhengcong+Fei%2C+Yang+Feng%2C+and+Jie+Zhou%2E+2021a%2E+Addressing+inquiries+about+history%3A+An+efficient+and+practical+framework+for+evaluating+open%2Ddomain+chatbot+consistency%2E+In+Fin)
34. Dialogue, pages 225–235, 1st virtual meeting. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dialogue%2C+pages+225%E2%80%93235%2C+1st+virtual+meeting%2E+Association+for+Computational+Linguistics%2E)
35. Shikib Mehri and Maxine Eskenazi. 2020b. [USR: An unsupervised and reference free evaluation metric for dialog generation](https://md.research.elamur.ai/free/en/papers/mehri-2020-usr-unsupervised-reference-free-evaluation/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681–707, Online. Association for Computational Linguistics.
36. Cl’ement Rebuffel, Thomas Scialom, Laure Soulier, Benjamin Piwowarski, Sylvain Lamprier, Jacopo Staiano, Geoffrey Scoutheeten, and P. Gallinari. 2021. Data-questeval: A referenceless metric for data to text semantic evaluation. ArXiv, abs/2104.07555. [find](https://scholar.google.com/scholar?q=Cl%E2%80%99ement+Rebuffel%2C+Thomas+Scialom%2C+Laure+Soulier%2C+Benjamin+Piwowarski%2C+Sylvain+Lamprier%2C+Jacopo+Staiano%2C+Geoffrey+Scoutheeten%2C+and+P%2E+Gallinari%2E+2021%2E+Data%2Dquesteval%3A+A+referenceless+metric+for+data+t)
37. Rostislav Nedelchev, Jens Lehmann, and Ricardo Usbeck. 2020. Language model transformers as evaluators for open-domain dialogues. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6797–6808, Barcelona, Spain (Online). International Committee on Computational Lin… [find](https://scholar.google.com/scholar?q=Rostislav+Nedelchev%2C+Jens+Lehmann%2C+and+Ricardo+Usbeck%2E+2020%2E+Language+model+transformers+as+evaluators+for+open%2Ddomain+dialogues%2E+In+Proceedings+of+the+28th+International+Conference+on+Computational+L)
38. Ehud Reiter. 2018. [A structured review of the validity of BLEU](https://md.research.elamur.ai/free/en/papers/reiter-2018-structured-review-validity-bleu/). Computational Linguistics, 44(3):393– 401.
39. Alan Ritter, Colin Cherry, and William B. Dolan. 2011. [Data-driven response generation in social media](https://md.research.elamur.ai/free/en/papers/ritter-2011-data-driven-response-generation-social/). In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 583–593, Edinburgh, Scotland, UK. Association for Computational Linguistics.
40. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. [Why we need new evaluation metrics for NLG](https://md.research.elamur.ai/free/en/papers/novikova-2017-why-we-need-new-evaluation/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241–2252, Copenhagen, Denmark. Association for Computational Linguistics.
41. Mario Rodríguez-Cantelar, Luis Fernando D’Haro, and Fernando Matía. 2021. Automatic evaluation of non-task oriented dialog systems by using sentence embeddings projections and their dynamics. In Conversational Dialogue Systems for the Next Decade, pages 71–84. Springer. [find](https://scholar.google.com/scholar?q=Mario+Rodr%C3%ADguez%2DCantelar%2C+Luis+Fernando+D%E2%80%99Haro%2C+and+Fernando+Mat%C3%ADa%2E+2021%2E+Automatic+evaluation+of+non%2Dtask+oriented+dialog+systems+by+using+sentence+embeddings+projections+and+their+dynamics%2E+In+Conve)
42. Bo Pang, Erik Nijkamp, Wenjuan Han, Linqi Zhou, Yixian Liu, and Kewei Tu. 2020. Towards holistic and automatic evaluation of open-domain dialogue generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3619–3629, Online. Association for Computati… [find](https://scholar.google.com/scholar?q=Bo+Pang%2C+Erik+Nijkamp%2C+Wenjuan+Han%2C+Linqi+Zhou%2C+Yixian+Liu%2C+and+Kewei+Tu%2E+2020%2E+Towards+holistic+and+automatic+evaluation+of+open%2Ddomain+dialogue+generation%2E+In+Proceedings+of+the+58th+Annual+Meeting+)
43. Ananya B. Sai, Akash Kumar Mohankumar, Siddhartha Arora, and Mitesh M. Khapra. 2020. Improving dialog evaluation with a multi-reference adversarial dataset and large scale pretraining. Transactions of the Association for Computational Linguistics, 8:810–827. [find](https://scholar.google.com/scholar?q=Ananya+B%2E+Sai%2C+Akash+Kumar+Mohankumar%2C+Siddhartha+Arora%2C+and+Mitesh+M%2E+Khapra%2E+2020%2E+Improving+dialog+evaluation+with+a+multi%2Dreference+adversarial+dataset+and+large+scale+pretraining%2E+Transactions+of)
44. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
45. Thomas Scialom, Paul-Alexis Dray, Patrick Gallinari, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, and Alex Wang. 2021. [Questeval: Summarization asks for fact-based evaluation](https://md.research.elamur.ai/free/en/papers/scialom-2021-questeval-summarization-asks-fact-based/). arXiv preprint arXiv:2103.12693.
46. Vitou Phy, Yang Zhao, and Akiko Aizawa. 2020. Deconstruct to reconstruct a configurable evaluation metric for open-domain dialogue systems. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4164–4178, Barcelona, Spain (Online). International Committee on Computa… [find](https://scholar.google.com/scholar?q=Vitou+Phy%2C+Yang+Zhao%2C+and+Akiko+Aizawa%2E+2020%2E+Deconstruct+to+reconstruct+a+configurable+evaluation+metric+for+open%2Ddomain+dialogue+systems%2E+In+Proceedings+of+the+28th+International+Conference+on+Compu)
47. Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. [What makes a good conversation? how controllable attributes affect human judgments](https://md.research.elamur.ai/free/en/papers/see-2019-what-makes-good-conversation-how/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol…
48. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
49. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Dipanjan+Das%2C+and+Ankur+Parikh%2E+2020%2E+BLEURT%3A+Learning+robust+metrics+for+text+generation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+)
50. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67.
51. Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. 2018. [RUSE: Regressor using sentence embeddings for automatic machine translation evaluation](https://md.research.elamur.ai/free/en/papers/shimanaka-2018-ruse-regressor-sentence-embeddings-automatic/). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan B…
52. Chen Zhang, Yiming Chen, Luis Fernando D’Haro, Yan Zhang, Thomas Friedrichs, Grandee Lee, and Haizhou Li. 2021a. Dynaeval: Unifying turn and dialogue level evaluation. In The Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint… [find](https://scholar.google.com/scholar?q=Chen+Zhang%2C+Yiming+Chen%2C+Luis+Fernando+D%E2%80%99Haro%2C+Yan+Zhang%2C+Thomas+Friedrichs%2C+Grandee+Lee%2C+and+Haizhou+Li%2E+2021a%2E+Dynaeval%3A+Unifying+turn+and+dialogue+level+evaluation%2E+In+The+Joint+Conference+of+the+5)
53. Koustuv Sinha, Prasanna Parthasarathi, Jasmine Wang, Ryan Lowe, William L. Hamilton, and Joelle Pineau. 2020. Learning an unreferenced metric for online dialogue evaluation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2430–2441, Online. Associati… [find](https://scholar.google.com/scholar?q=Koustuv+Sinha%2C+Prasanna+Parthasarathi%2C+Jasmine+Wang%2C+Ryan+Lowe%2C+William+L%2E+Hamilton%2C+and+Joelle+Pineau%2E+2020%2E+Learning+an+unreferenced+metric+for+online+dialogue+evaluation%2E+In+Proceedings+of+the+58th)
54. Chen Zhang, Luis Fernando D’Haro, Rafael E Banchs, Thomas Friedrichs, and Haizhou Li. 2021b. Deep am-fm: Toolkit for automatic dialogue evaluation. In Conversational Dialogue Systems for the Next Decade, pages 53–69. Springer. [find](https://scholar.google.com/scholar?q=Chen+Zhang%2C+Luis+Fernando+D%E2%80%99Haro%2C+Rafael+E+Banchs%2C+Thomas+Friedrichs%2C+and+Haizhou+Li%2E+2021b%2E+Deep+am%2Dfm%3A+Toolkit+for+automatic+dialogue+evaluation%2E+In+Conversational+Dialogue+Systems+for+the+Next+Deca)
55. Elior Sulem, Omri Abend, and Ari Rappoport. 2018. [BLEU is not suitable for the evaluation of text simplification](https://md.research.elamur.ai/free/en/papers/sulem-2018-bleu-is-not-suitable-evaluation/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 738–744, Brussels, Belgium. Association for Computational Linguistics.
56. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. [Personalizing dialogue agents: I have a dog, do you have pets too?](https://md.research.elamur.ai/free/en/papers/zhang-2018-personalizing-dialogue-agents-i-have/) In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204– 2213, Melbo…
57. Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2018. Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32. [find](https://scholar.google.com/scholar?q=Chongyang+Tao%2C+Lili+Mou%2C+Dongyan+Zhao%2C+and+Rui+Yan%2E+2018%2E+Ruber%3A+An+unsupervised+method+for+automatic+evaluation+of+open%2Ddomain+dialog+systems%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Inte)
58. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in neural information processing systems, pages 5998–6008.
59. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2019%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+arXiv+preprint+arXiv%3A1904%2E09675%2E)
60. Anu Venkatesh, Chandra Khatri, Ashwin Ram, Fenfei Guo, Raefer Gabriel, Ashish Nagar, Rohit Prasad, Ming Cheng, Behnam Hedayatnia, Angeliki Metallinou, et al. 2018. On evaluating and comparing conversational agents. arXiv preprint arXiv:1801.03625, 4:60–68. — [arXiv:1801.03625](https://arxiv.org/abs/1801.03625) [find](https://scholar.google.com/scholar?q=Anu+Venkatesh%2C+Chandra+Khatri%2C+Ashwin+Ram%2C+Fenfei+Guo%2C+Raefer+Gabriel%2C+Ashish+Nagar%2C+Rohit+Prasad%2C+Ming+Cheng%2C+Behnam+Hedayatnia%2C+Angeliki+Metallinou%2C+et+al%2E+2018%2E+On+evaluating+and+comparing+conversa)
61. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. [Dialogpt: Large-scale generative pre-training for conversational response generation](https://md.research.elamur.ai/free/en/papers/zhang-2019-dialogpt-large-scale-generative-pre/). In ACL, system demonstration.
62. Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017a. [Learning discourse-level diversity for neural dialog models using conditional variational autoencoders](https://md.research.elamur.ai/free/en/papers/zhao-2017-learning-discourse-level-diversity-neural/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 654–664, Vancouver,…
63. Marilyn A. Walker, Diane J. Litman, Candace A. Kamm, and Alicia Abella. 1997. [PARADISE: A framework for evaluating spoken dialogue agents](https://md.research.elamur.ai/free/en/papers/walker-1997-paradise-framework-evaluating-spoken-dialogue/). In 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguisti…
64. Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017b. [Learning discourse-level diversity for neural dialog models using conditional variational autoencoders](https://md.research.elamur.ai/free/en/papers/zhao-2017-learning-discourse-level-diversity-neural/). arXiv preprint arXiv:1703.10960.
65. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,…
66. Wangchunshu Zhou and Ke Xu. 2020. Learning to compare for better training and evaluation of open domain natural language generation models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9717–9724. [find](https://scholar.google.com/scholar?q=Wangchunshu+Zhou+and+Ke+Xu%2E+2020%2E+Learning+to+compare+for+better+training+and+evaluation+of+open+domain+natural+language+generation+models%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intellig)
67. Jiannan Xiang, Yahui Liu, Deng Cai, Huayang Li, Defu Lian, and Lemao Liu. 2021. Assessing dialogue systems with distribution distances. arXiv preprint arXiv:2105.02573. [find](https://scholar.google.com/scholar?q=Jiannan+Xiang%2C+Yahui+Liu%2C+Deng+Cai%2C+Huayang+Li%2C+Defu+Lian%2C+and+Lemao+Liu%2E+2021%2E+Assessing+dialogue+systems+with+distribution+distances%2E+arXiv+preprint+arXiv%3A2105%2E02573%2E)
68. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. [Aligning books and movies: Towards story-like visual explanations by watching movies and reading books](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/). In Proceedings of the IEEE international conference on computer vision, pages 19–…
69. Tsuta Yuma, Naoki Yoshinaga, and Masashi Toyoda. 2020. uBLEU: Uncertainty-aware automatic evaluation method for open-domain dialogue systems. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 199–206, Online. Association for… [find](https://scholar.google.com/scholar?q=Tsuta+Yuma%2C+Naoki+Yoshinaga%2C+and+Masashi+Toyoda%2E+2020%2E+uBLEU%3A+Uncertainty%2Daware+automatic+evaluation+method+for+open%2Ddomain+dialogue+systems%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Associati)

*Updated 24 October 2026*