---
title: "How NOT To Evaluate Your Dialogue System — Liu 2016"
description: "The paper examines whether automatic metrics can be trusted when evaluating unsupervised dialogue models, that… 43 references, cited by 500 papers here."
authors: ["Chia-Wei Liu", "Ryan Lowe", "Iulian Serban", "Mike Noseworthy", "Laurent Charlin", "Joelle Pineau"]
year: 2016
venue: "Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D16-1230/"
doi: "10.18653/v1/d16-1230"
arxiv: "1603.08023"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2016-how-not-evaluate-your-dialogue/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2016-how-not-evaluate-your-dialogue/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/) / How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

# How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

**Authors:** Chia-Wei Liu, [Ryan Lowe](https://md.research.elamur.ai/free/en/authors/ryan-lowe/), [Iulian Serban](https://md.research.elamur.ai/free/en/authors/iulian-vlad-serban/), Mike Noseworthy, [Laurent Charlin](https://md.research.elamur.ai/free/en/authors/laurent-charlin/) and [Joelle Pineau](https://md.research.elamur.ai/free/en/authors/joelle-pineau/) · **Year:** 2016 · **Venue:** Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D16-1230/) · [PDF](https://aclanthology.org/D16-1230.pdf) · [DOI 10.18653/v1/d16-1230](https://doi.org/10.18653/v1/d16-1230) · [arXiv 1603.08023](https://arxiv.org/abs/1603.08023)

**Citations:** [500](https://md.research.elamur.ai/free/en/papers/liu-2016-how-not-evaluate-your-dialogue/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
The paper examines whether automatic metrics can be trusted when evaluating unsupervised dialogue models, that is, chatbots that generate a response based solely on the conversation context. Such systems are often evaluated with metrics from machine translation and summarization. These metrics compare the generated response with a single reference response. The authors collected human judgments of responses produced by several response generation and retrieval models. They then computed the correlation of these judgments with word-overlap metrics (BLEU, METEOR, ROUGE) and with word-embedding-based metrics (Word2Vec). The study was carried out on two corpora: the conversational Twitter corpus and the technical Ubuntu Dialogue Corpus. On Twitter the correlation turned out to be weak, and on Ubuntu it is practically nonexistent. Quantitative and qualitative analysis explains why: a single utterance can have many valid responses, and they may share no words at all with the reference. The paper cautioned the community against the conventional metrics and set the task of developing new ones that are closer to human judgment.

## Key points
- The study examines the word-overlap metrics BLEU-1…4, METEOR and ROUGE-L, as well as the embedding-based Greedy Matching, Embedding Average and Vector Extrema. All of them compare a response with a single reference and do not take the conversation context into account.
- A perfectly appropriate response can receive BLEU = 0 if it shares no words with the reference. The authors illustrate this with an example of a conversation about movies.
- After removing stop words and punctuation, the correlation of BLEU-2 with human judgments becomes even weaker. BLEU and METEOR also depend noticeably on the difference in response lengths.
- Embedding-based metrics reliably distinguish baseline models from state-of-the-art ones, yet they barely correlate with human judgments. They mostly measure how close a response is in topic rather than how appropriate it is.
- What the authors propose: metrics based on whole-sentence representations (for example, skip-thought vectors), weighting words by importance, and learned evaluation models that take the context into account. Until such metrics exist, human evaluation remains indispensable.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/)

## Cite

Cite as: Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin and Joelle Pineau. 2016. How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liu2016how,
  title = {{How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation}},
  author = {Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin and Joelle Pineau},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1230},
  eprint = {1603.08023},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d16-1230}
}
```

- [← previous The LAMBADA dataset: Word prediction requiring a broad discourse context](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/)
- [next → Does String-Based Neural MT Learn Source Syntax?](https://md.research.elamur.ai/free/en/papers/shi-2016-does-string-based-neural-mt/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/)

## References

43 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. R. Artstein, S. Gandhe, J. Gerten, A. Leuski, and D. Traum. 2009. Semi-formal evaluation of conversational characters. In Languages: From Formal to Natural, pages 22–35. Springer. [find](https://scholar.google.com/scholar?q=R%2E+Artstein%2C+S%2E+Gandhe%2C+J%2E+Gerten%2C+A%2E+Leuski%2C+and+D%2E+Traum%2E+2009%2E+Semi%2Dformal+evaluation+of+conversational+characters%2E+In+Languages%3A+From+Formal+to+Natural%2C+pages+22%E2%80%9335%2E+Springer%2E)
2. S. Banerjee and A. Lavie. 2005. METEOR: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. [our card](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)
3. O. Bojar, C. Buck, C. Federmann, B. Haddow, P. Koehn, J. Leveling, C. Monz, P. Pecina, M. Post, H. Saint-Amand, et al. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12–58. Association for Computa… [our card](https://md.research.elamur.ai/free/en/papers/bojar-2014-findings-2014-workshop-statistical-machine/)
4. A. Cahill. 2009. Correlating human and automatic evaluation of a german surface realiser. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 97–100. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=A%2E+Cahill%2E+2009%2E+Correlating+human+and+automatic+evaluation+of+a+german+surface+realiser%2E+In+Proceedings+of+the+ACL%2DIJCNLP+2009+Conference+Short+Papers%2C+pages+97%E2%80%93100%2E+Association+for+Computational+Lin)
5. C. Callison-Burch, M. Osborne, and P. Koehn. 2006. Re-evaluation the role of bleu in machine translation research. In EACL, volume 6, pages 249–256. [find](https://scholar.google.com/scholar?q=C%2E+Callison%2DBurch%2C+M%2E+Osborne%2C+and+P%2E+Koehn%2E+2006%2E+Re%2Devaluation+the+role+of+bleu+in+machine+translation+research%2E+In+EACL%2C+volume+6%2C+pages+249%E2%80%93256%2E)
6. C. Callison-Burch, P. Koehn, C. Monz, K. Peterson, M. Przybocki, and O. F. Zaidan. 2010. Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 17… [our card](https://md.research.elamur.ai/free/en/papers/burch-2010-findings-2010-joint-workshop-statistical/)
7. C. Callison-Burch, P. Koehn, C. Monz, and O. F. Zaidan. 2011. Findings of the 2011 workshop on statistical machine translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 22–64. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/burch-2011-findings-2011-workshop-statistical-machine/)
8. B. Chen and C. Cherry. 2014. A systematic comparison of smoothing techniques for sentence-level bleu. [our card](https://md.research.elamur.ai/free/en/papers/chen-2014-systematic-comparison-smoothing-techniques-sentence/)
9. ACL 2014, page 362. J. Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70(4):213. [find](https://scholar.google.com/scholar?q=ACL+2014%2C+page+362%2E+J%2E+Cohen%2E+1968%2E+Weighted+kappa%3A+Nominal+scale+agreement+provision+for+scaled+disagreement+or+partial+credit%2E+Psychological+bulletin%2C+70%284%29%3A213%2E)
10. D. Espinosa, R. Rajkumar, M. White, and S. Berleant. 2010. Further meta-evaluation of broad-coverage surface realization. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 564–574. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=D%2E+Espinosa%2C+R%2E+Rajkumar%2C+M%2E+White%2C+and+S%2E+Berleant%2E+2010%2E+Further+meta%2Devaluation+of+broad%2Dcoverage+surface+realization%2E+In+Proceedings+of+the+2010+Conference+on+Empirical+Methods+in+Natural+Language)
11. P. W. Foltz, W. Kintsch, and T. K. Landauer. 1998. The measurement of textual coherence with latent semantic analysis. Discourse processes, 25(2-3):285–307. [find](https://scholar.google.com/scholar?q=P%2E+W%2E+Foltz%2C+W%2E+Kintsch%2C+and+T%2E+K%2E+Landauer%2E+1998%2E+The+measurement+of+textual+coherence+with+latent+semantic+analysis%2E+Discourse+processes%2C+25%282%2D3%29%3A285%E2%80%93307%2E)
12. G. Forgues, J. Pineau, J.-M. Larcheveque, and R. Tremblay. 2014. Bootstrapping dialog systems with word embeddings. M. Galley, C. Brockett, A. Sordoni, Y. Ji, M. Auli, C. Quirk, M. l, J. Gao, and B. Dolan. 2015a. deltaBLEU: A discriminative metric for generation tasks with intrinsically diverse targ… [find](https://scholar.google.com/scholar?q=G%2E+Forgues%2C+J%2E+Pineau%2C+J%2E%2DM%2E+Larcheveque%2C+and+R%2E+Tremblay%2E+2014%2E+Bootstrapping+dialog+systems+with+word+embeddings%2E+M%2E+Galley%2C+C%2E+Brockett%2C+A%2E+Sordoni%2C+Y%2E+Ji%2C+M%2E+Auli%2C+C%2E+Quirk%2C+M%2E+l%2C+J%2E+Gao%2C+and+B%2E+D)
13. M. Galley, C. Brockett, A. Sordoni, Y. Ji, M. Auli, C. Quirk, M. Mitchell, J. Gao, and B. Dolan. 2015b. deltableu: A discriminative metric for generation tasks with intrinsically diverse targets. arXiv preprint arXiv:1506.06863. [find](https://scholar.google.com/scholar?q=M%2E+Galley%2C+C%2E+Brockett%2C+A%2E+Sordoni%2C+Y%2E+Ji%2C+M%2E+Auli%2C+C%2E+Quirk%2C+M%2E+Mitchell%2C+J%2E+Gao%2C+and+B%2E+Dolan%2E+2015b%2E+deltableu%3A+A+discriminative+metric+for+generation+tasks+with+intrinsically+diverse+targets%2E+arXi)
14. Y. Graham, N. Mathur, and T. Baldwin. 2015. Accurate evaluation of segment-level machine translation metrics. In Proc. of NAACL-HLT, pages 1183–1191. [our card](https://md.research.elamur.ai/free/en/papers/graham-2015-accurate-evaluation-segment-level-machine/)
15. Citeseer. A. Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850. [find](https://scholar.google.com/scholar?q=Citeseer%2E+A%2E+Graves%2E+2013%2E+Generating+sequences+with+recurrent+neural+networks%2E+arXiv+preprint+arXiv%3A1308%2E0850%2E)
16. S. Hochreiter and J. Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=S%2E+Hochreiter+and+J%2E+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2C+9%288%29%3A1735%E2%80%931780%2E)
17. E. Hovy. 1999. Toward finely differentiated evaluation metrics for machine translation. In Proceedings of the Eagles Workshop on Standards and Evaluation. [find](https://scholar.google.com/scholar?q=E%2E+Hovy%2E+1999%2E+Toward+finely+differentiated+evaluation+metrics+for+machine+translation%2E+In+Proceedings+of+the+Eagles+Workshop+on+Standards+and+Evaluation%2E)
18. K. Jokinen and M. McTear. 2009. [find](https://scholar.google.com/scholar?q=K%2E+Jokinen+and+M%2E+McTear%2E+2009%2E)
19. Spoken Dialogue Systems. Morgan Claypool. C. Kamm. 1995. User interfaces for voice applications. Proceedings of the National Academy of Sciences, 92(22):10031–10037. [find](https://scholar.google.com/scholar?q=Spoken+Dialogue+Systems%2E+Morgan+Claypool%2E+C%2E+Kamm%2E+1995%2E+User+interfaces+for+voice+applications%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+92%2822%29%3A10031%E2%80%9310037%2E)
20. R. Kiros, Y. Zhu, R. R. Salakhutdinov, R. Zemel, R. Urtasun, A. Torralba, and S. Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems, pages 3276–3284. [find](https://scholar.google.com/scholar?q=R%2E+Kiros%2C+Y%2E+Zhu%2C+R%2E+R%2E+Salakhutdinov%2C+R%2E+Zemel%2C+R%2E+Urtasun%2C+A%2E+Torralba%2C+and+S%2E+Fidler%2E+2015%2E+Skip%2Dthought+vectors%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+3276%E2%80%933284%2E)
21. T. K. Landauer and S. T. Dumais. 1997. A solution to plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological review, 104(2):211. [find](https://scholar.google.com/scholar?q=T%2E+K%2E+Landauer+and+S%2E+T%2E+Dumais%2E+1997%2E+A+solution+to+plato%E2%80%99s+problem%3A+The+latent+semantic+analysis+theory+of+acquisition%2C+induction%2C+and+representation+of+knowledge%2E+Psychological+review%2C+104%282%29%3A211%2E)
22. N. Lasguido, S. Sakti, G. Neubig, T. Tomoki, and S. Nakamura. 2014. Utilizing human-to-human conversation examples for a multi domain chat-oriented dialog system. IEICE TRANSACTIONS on Information and Systems, 97(6):1497–1505. [find](https://scholar.google.com/scholar?q=N%2E+Lasguido%2C+S%2E+Sakti%2C+G%2E+Neubig%2C+T%2E+Tomoki%2C+and+S%2E+Nakamura%2E+2014%2E+Utilizing+human%2Dto%2Dhuman+conversation+examples+for+a+multi+domain+chat%2Doriented+dialog+system%2E+IEICE+TRANSACTIONS+on+Information+and)
23. J. Li, M. Galley, C. Brockett, J. Gao, and B. Dolan. 2015. A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055. [our card](https://md.research.elamur.ai/free/en/papers/li-2016-diversity-promoting-objective-function-neural/)
24. J. Li, M. Galley, C. Brockett, J. Gao, and B. Dolan. 2016. A persona-based neural conversation model. arXiv preprint arXiv:1603.06155. [our card](https://md.research.elamur.ai/free/en/papers/li-2016-persona-based-neural-conversation-model/)
25. C.-Y. Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-04 workshop, volume 8. [our card](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/)
26. R. Lowe, N. Pow, I. V. Serban, and J. Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In SIG-DIAL. [our card](https://md.research.elamur.ai/free/en/papers/lowe-2015-ubuntu-dialogue-corpus-large-dataset/)
27. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111– 3119. [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/)
28. J. Mitchell and M. Lapata. 2008. Vector-based models of semantic composition. In ACL, pages 236–244. S. Möller, R. Englert, K. Engelbrecht, V. Hafner, A. Jameson, A. Oulasvirta, A. Raake, and N. Reithinger. 2006. MeMo: towards automatic usability evaluation of spoken dialogue services by user error… [find](https://scholar.google.com/scholar?q=J%2E+Mitchell+and+M%2E+Lapata%2E+2008%2E+Vector%2Dbased+models+of+semantic+composition%2E+In+ACL%2C+pages+236%E2%80%93244%2E+S%2E+Mo%CC%88ller%2C+R%2E+Englert%2C+K%2E+Engelbrecht%2C+V%2E+Hafner%2C+A%2E+Jameson%2C+A%2E+Oulasvirta%2C+A%2E+Raake%2C+and+N%2E+Reit)
29. K. Papineni, S. Roukos, T. Ward, and W. Zhu. 2002a. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on Association for Computational Linguistics (ACL). [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
30. K. Papineni, S. Roukos, T. Ward, J. Henderson, and F. Reeder. 2002b. Corpus-based comprehensive and diagnostic MT evaluation: Initial Arabic, Chinese, French, and Spanish results. In Proceedings of the second international conference on Human Language Technology Research, pages 132–137. [find](https://scholar.google.com/scholar?q=K%2E+Papineni%2C+S%2E+Roukos%2C+T%2E+Ward%2C+J%2E+Henderson%2C+and+F%2E+Reeder%2E+2002b%2E+Corpus%2Dbased+comprehensive+and+diagnostic+MT+evaluation%3A+Initial+Arabic%2C+Chinese%2C+French%2C+and+Spanish+results%2E+In+Proceedings+of+th)
31. E. Reiter and A. Belz. 2009. An investigation into the validity of some metrics for automatically evaluating natural language generation systems. Computational Linguistics, 35(4):529–558. [find](https://scholar.google.com/scholar?q=E%2E+Reiter+and+A%2E+Belz%2E+2009%2E+An+investigation+into+the+validity+of+some+metrics+for+automatically+evaluating+natural+language+generation+systems%2E+Computational+Linguistics%2C+35%284%29%3A529%E2%80%93558%2E)
32. A. Ritter, C. Cherry, and B. Dolan. 2010. Unsupervised modeling of twitter conversations. In North American Chapter of the Association for Computational Linguistics (NAACL). [our card](https://md.research.elamur.ai/free/en/papers/ritter-2010-unsupervised-modeling-twitter-conversations/)
33. A. Ritter, C. Cherry, and W. B. Dolan. 2011. Data-driven response generation in social media. In Proceedings of the conference on empirical methods in natural language processing, pages 583–593. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/ritter-2011-data-driven-response-generation-social/)
34. V. Rus and M. Lintean. 2012. A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metrics. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 157–162, Stroudsburg, PA, USA. Association for Computationa… [find](https://scholar.google.com/scholar?q=V%2E+Rus+and+M%2E+Lintean%2E+2012%2E+A+comparison+of+greedy+and+optimal+assessment+of+natural+language+student+input+using+word%2Dto%2Dword+similarity+metrics%2E+In+Proceedings+of+the+Seventh+Workshop+on+Building+E)
35. J. Schatzmann, K. Georgila, and S. Young. 2005. Quantitative evaluation of user simulation techniques for spoken dialogue systems. In 6th Special Interest Group on Discourse and Dialogue (SIGDIAL). [find](https://scholar.google.com/scholar?q=J%2E+Schatzmann%2C+K%2E+Georgila%2C+and+S%2E+Young%2E+2005%2E+Quantitative+evaluation+of+user+simulation+techniques+for+spoken+dialogue+systems%2E+In+6th+Special+Interest+Group+on+Discourse+and+Dialogue+%28SIGDIAL%29%2E)
36. I. V. Serban, A. Sordoni, Y. Bengio, A. Courville, and J. Pineau. 2015. Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Networks. In AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=I%2E+V%2E+Serban%2C+A%2E+Sordoni%2C+Y%2E+Bengio%2C+A%2E+Courville%2C+and+J%2E+Pineau%2E+2015%2E+Building+End%2DTo%2DEnd+Dialogue+Systems+Using+Generative+Hierarchical+Neural+Networks%2E+In+AAAI+Conference+on+Artificial+Intelligenc)
37. I. V. Serban, A. Sordoni, R. Lowe, L. Charlin, J. Pineau, A. Courville, and Y. Bengio. 2016. A hierarchical latent variable encoder-decoder model for generating dialogues. arXiv preprint arXiv:1605.06069. [our card](https://md.research.elamur.ai/free/en/papers/serban-2016-hierarchical-latent-variable-encoder-decoder/)
38. A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji, M. Mitchell, J. Nie, J. Gao, and B. Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT 2015). [our card](https://md.research.elamur.ai/free/en/papers/sordoni-2015-neural-network-approach-context-sensitive/)
39. A. Stent, M. Marge, and M. Singhai. 2005. Evaluating evaluation methods for generation in the presence of variation. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 341–351. [find](https://scholar.google.com/scholar?q=A%2E+Stent%2C+M%2E+Marge%2C+and+M%2E+Singhai%2E+2005%2E+Evaluating+evaluation+methods+for+generation+in+the+presence+of+variation%2E+In+International+Conference+on+Intelligent+Text+Processing+and+Computational+Lingui)
40. Springer. O. Vinyals and Q. Le. 2015. A neural conversational model. arXiv preprint arXiv:1506.05869. [our card](https://md.research.elamur.ai/free/en/papers/vinyals-2015-neural-conversational-model/)
41. M. Walker, D. Litman, C. Kamm, and A. Abella. 1997. Paradise: A framework for evaluating spoken dialogue agents. In Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics, pages 271–280. [our card](https://md.research.elamur.ai/free/en/papers/walker-1997-paradise-framework-evaluating-spoken-dialogue/)
42. ACL. T.-H. Wen, M. Gasic, N. Mrksic, P.-H. Su, D. Vandyke, and S. Young. 2015. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. arXiv preprint arXiv:1508.01745. [our card](https://md.research.elamur.ai/free/en/papers/wen-2015-semantically-conditioned-lstm-based-natural/)
43. J. Wieting, M. Bansal, K. Gimpel, and K. Livescu. 2015. Towards universal paraphrastic sentence embeddings. CoRR, abs/1511.08198. [our card](https://md.research.elamur.ai/free/en/papers/wieting-2015-towards-universal-paraphrastic-sentence-embeddings/)