---
title: "Is this Dialogue Coherent? Learning from Dialogue Acts and"
description: "They build the Switchboard Coherence (SWBD-Coh) corpus from SWBD-DA telephone… 0.760 MRR. 43 references, cited by 8 papers here."
authors: ["Alessandra Cervone", "Giuseppe Riccardi"]
year: 2020
venue: "Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue"
source: "https://aclanthology.org/2020.sigdial-1.21/"
doi: "10.18653/v1/2020.sigdial-1.21"
arxiv: "2006.10157"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cervone-2020-is-this-dialogue-coherent-learning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cervone-2020-is-this-dialogue-coherent-learning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/) / Is this Dialogue Coherent? Learning from Dialogue Acts and Entities

# Is this Dialogue Coherent? Learning from Dialogue Acts and Entities

**Authors:** [Alessandra Cervone](https://md.research.elamur.ai/free/en/authors/alessandra-cervone/) and [Giuseppe Riccardi](https://md.research.elamur.ai/free/en/authors/giuseppe-riccardi/) · **Year:** 2020 · **Venue:** [Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue](https://md.research.elamur.ai/free/en/venues/sigdial/) ([SIGDIAL 2020](https://md.research.elamur.ai/free/en/venues/sigdial/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.sigdial-1.21/) · [PDF](https://aclanthology.org/2020.sigdial-1.21.pdf) · [DOI 10.18653/v1/2020.sigdial-1.21](https://doi.org/10.18653/v1/2020.sigdial-1.21) · [arXiv 2006.10157](https://arxiv.org/abs/2006.10157) · [OpenAlex](https://openalex.org/W3034410548)

**Citations:** 16 citations (OpenAlex, 2026-10-14) · 8 in our library, of the 141,840 papers whose bibliographies we read

## Summary
Evaluating open-domain dialogue is hard because there is no fixed set of intents or entities, and earlier coherence work scored whole documents or dialogues rather than individual turns. Cervone and Riccardi study how people judge whether a candidate next turn fits the full preceding conversation. They build the Switchboard Coherence (SWBD-Coh) corpus from SWBD-DA telephone dialogues, keeping disfluencies and overlaps. In this corpus, human raters scored next-turn candidates against the complete dialogue history. A regression analysis shows that both entity distribution patterns and Dialogue Acts (DAs) shape these judgments.

They then train biGRU models whose input is deliberately limited to entity features, DA features and speaker-turn features, with no full text. Training uses response selection as weak supervision, and the models are tested on the graded human ratings. Combining entities with DAs works best on both tasks and clearly beats SVM entity-grid baselines. This suggests that dialogue structure, and not only lexical overlap, signals coherence.

## Key points
- **Response-selection data.** The splits contain 740 / 184 / 231 source dialogues, with ten insertion points per dialogue and nine negatives per point. This gives 66,600 training pairs.
- **Two ways to create negatives:**
  - *Internal Swap* takes a later turn from the same conversation and is the harder setting.
  - *External Swap* takes a turn from another conversation and is the easier setting.
- **Best response-selection model.** The biGRU with ent word + DA + turn reaches MRR 0.520 and Acc 77.3 under Internal Swap, and MRR 0.550 and Acc 75.1 under External Swap. The best SVM (ent role + DA) reaches only MRR 0.417 under Internal Swap.
- **Turn coherence rating on SWBD-Coh.**
  - The same combined model trained on External Swap reaches nDCG 0.760, MRR 0.583 and Acc 61.1. A random baseline scores nDCG 0.645.
  - The entity-only model trained on Internal Swap falls below random on MRR (0.395 vs 0.479).
  - Models trained on External Swap beat those trained on Internal Swap in every condition.
- **Syntactic roles add little.** Entity syntactic-role features contribute little once lexical entity features are present. The authors suggest two reasons: spoken syntax is simpler, and parsers trained on written text make more errors on dialogue.
- **Resources released.** The SWBD-Coh corpus and the code are available on GitHub (alecervi/switchboard-coherence-corpus and alecervi/turn-coherence-rating).

## Interesting ideas
- Generating negatives two ways (from the same conversation versus from other conversations) yields a hard and an easy contrast set, which shows whether a model relies on topic drift or on real local fit.
- Restricting the model input to structural signals (entities, DAs, speaker turns) and leaving out raw text turns the model into a probe of how much those structures alone explain.
- Training on a cheap, automatically built ranking task and then testing on a small set of graded human judgments avoids costly supervised annotation while still measuring against real human perception.
- The finding that parser-derived grammatical roles help little on disfluent speech is a warning that tree features trained on written text may not transfer to dialogue.
- Combining what is said (DAs) with what is talked about (entity transitions), plus who speaks, gave the strongest coherence signal.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Mova's negative controls could likewise pair "same text, wrong position" with "foreign text" corruptions, separating real discourse understanding from topic matching.
- **I2** (frames): A coherence check in prag could run only over speech-act labels and frame participants, keeping the judgment explainable and independent of bag-of-words cues.
- **I3** (learn): Silver tasks generated from Mova's own texts could be checked against a small human-rated set using rank metrics such as nDCG instead of binary accuracy.
- **I4** (tree): When Mova parses dialogue or transcripts, it should measure separately how reliable the subject and object relations are, and possibly add spoken-style data to the treebanks used to train gparse.
- **I5** (world): Mova's world of a text could track entity reintroductions alongside each turn's speech act and speaker, then flag turns that fit neither the active entities nor the expected act sequence.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Trento](https://md.research.elamur.ai/free/en/universities/university-of-trento/) — University, Trento, Italy · 62 works · 14,868 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 401 works · [💬 Speech act theory](https://md.research.elamur.ai/free/en/topics/speech-act-theory/) · 160 works

## Cite

Cite as: Alessandra Cervone and Giuseppe Riccardi. 2020. Is this Dialogue Coherent? Learning from Dialogue Acts and Entities. Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{cervone2020is,
  title = {{Is this Dialogue Coherent? Learning from Dialogue Acts and Entities}},
  author = {Alessandra Cervone and Giuseppe Riccardi},
  booktitle = {Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2020},
  doi = {10.18653/v1/2020.sigdial-1.21},
  eprint = {2006.10157},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.sigdial-1.21}
}
```

- [← previous Bridging Resolution: A Survey of the State of the Art](https://md.research.elamur.ai/free/en/papers/kobayashi-2020-bridging-resolution-survey-state-art/)
- [next → Semantic Role Labeling as Syntactic Dependency Parsing](https://md.research.elamur.ai/free/en/papers/shi-2020-semantic-role-labeling-as-syntactic/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/)

## References

43 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Regina Barzilay and Mirella Lapata. 2008. [Modeling local coherence: An entity-based approach](https://md.research.elamur.ai/free/en/papers/barzilay-2008-modeling-local-coherence-entity-based/). Computational Linguistics, 34(1):1–34. [↗](https://doi.org/10.1162/coli.2008.34.1.1)
2. Alessandra Cervone, Evgeny Stepanov, and Giuseppe Riccardi. 2018. Coherence models for dialogue. Proc. Interspeech 2018, pages 1011–1015. [find](https://scholar.google.com/scholar?q=Alessandra+Cervone%2C+Evgeny+Stepanov%2C+and+Giuseppe+Riccardi%2E+2018%2E+Coherence+models+for+dialogue%2E+Proc%2E+Interspeech+2018%2C+pages+1011%E2%80%931015%2E)
3. Elizabeth Clark, Yangfeng Ji, and Noah A Smith. 2018. Neural text generation in stories using entity representations as context. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 2250–2260. [find](https://scholar.google.com/scholar?q=Elizabeth+Clark%2C+Yangfeng+Ji%2C+and+Noah+A+Smith%2E+2018%2E+Neural+text+generation+in+stories+using+entity+representations+as+context%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+)
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. [↗](https://doi.org/10.18653/v1/n19-1423)
5. Micha Elsner and Eugene Charniak. 2011. Disentangling chat with local coherence models. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1179–1189. [find](https://scholar.google.com/scholar?q=Micha+Elsner+and+Eugene+Charniak%2E+2011%2E+Disentangling+chat+with+local+coherence+models%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Human+Language+Techn)
6. Youmna Farag, Helen Yannakoudakis, and Ted Briscoe. 2018. Neural automated essay scoring and coherence modeling for adversarially crafted input. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 263–271. [find](https://scholar.google.com/scholar?q=Youmna+Farag%2C+Helen+Yannakoudakis%2C+and+Ted+Briscoe%2E+2018%2E+Neural+automated+essay+scoring+and+coherence+modeling+for+adversarially+crafted+input%2E+In+Proceedings+of+the+2018+Conference+of+the+North+Amer)
7. Sudeep Gandhe and David Traum. 2016. A semi-automated evaluation metric for dialogue model coherence. Situated Dialog in Speech-Based Human-Computer Interaction, page 217. [↗](https://doi.org/10.1007/978-3-319-21834-2_19)
8. Barbara J Grosz, Scott Weinstein, and Aravind K Joshi. 1995. [Centering: A framework for modeling the local coherence of discourse](https://md.research.elamur.ai/free/en/papers/grosz-1995-centering-framework-modeling-local-coherence/). Computational linguistics, 21(2):203–225.
9. Matthew Henderson, Ivan Vulić, Daniela Gerz, Iñigo Casanueva, Paweł Budzianowski, Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrkšić, and Pei-Hao Su. 2019. Training neural response selection for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5392–5404, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1906.01543)
10. Ryuichiro Higashinaka, Toyomi Meguro, Kenji Imamura, Hiroaki Sugiyama, Toshiro Makino, and Yoshihiro Matsuo. 2014. Evaluating coherence in open domain conversational systems. In Fifteenth Annual Conference of the International Speech Communication Association. [find](https://scholar.google.com/scholar?q=Ryuichiro+Higashinaka%2C+Toyomi+Meguro%2C+Kenji+Imamura%2C+Hiroaki+Sugiyama%2C+Toshiro+Makino%2C+and+Yoshihiro+Matsuo%2E+2014%2E+Evaluating+coherence+in+open+domain+conversational+systems%2E+In+Fifteenth+Annual+Confe)
11. Jerry R Hobbs. 1979. Coherence and coreference. Cognitive science, 3(1):67–90. [↗](https://doi.org/10.1207/s15516709cog0301_4)
12. Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422–446. [↗](https://doi.org/10.1145/582415.582418)
13. Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 133– 142. ACM. [find](https://scholar.google.com/scholar?q=Thorsten+Joachims%2E+2002%2E+Optimizing+search+engines+using+clickthrough+data%2E+In+Proceedings+of+the+eighth+ACM+SIGKDD+international+conference+on+Knowledge+discovery+and+data+mining%2C+pages+133%E2%80%93+142%2E+ACM)
14. Aravind K Joshi and Steve Kuhn. 1979. Centered logic: The role of entity centered sentence representation in natural language inferencing. In IJCAI, pages 435– 439. [find](https://scholar.google.com/scholar?q=Aravind+K+Joshi+and+Steve+Kuhn%2E+1979%2E+Centered+logic%3A+The+role+of+entity+centered+sentence+representation+in+natural+language+inferencing%2E+In+IJCAI%2C+pages+435%E2%80%93+439%2E)
15. Shafiq Joty, Muhammad Tasnim Mohiuddin, and Dat Tien Nguyen. 2018. Coherence modeling of asynchronous conversations: A neural entity grid approach. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 558–568. [↗](https://doi.org/10.48550/arxiv.1805.02275)
16. Seokhwan Kim, Michel Galley, Chulaka Gunasekara, Sungjin Lee, Adam Atkinson, Baolin Peng, Hannes Schulz, Jianfeng Gao, Jinchao Li, Mahmoud Adada, et al. 2019. The eighth dialog system technology challenge. arXiv preprint arXiv:1911.06394. [↗](https://arxiv.org/abs/1911.06394)
17. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. [↗](https://doi.org/10.48550/arxiv.1412.6980)
18. Harshit Kumar, Arvind Agarwal, and Sachindra Joshi. 2019. A practical dialogue-act-driven conversation model for multi-turn response selection. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1980–1989. [find](https://scholar.google.com/scholar?q=Harshit+Kumar%2C+Arvind+Agarwal%2C+and+Sachindra+Joshi%2E+2019%2E+A+practical+dialogue%2Dact%2Ddriven+conversation+model+for+multi%2Dturn+response+selection%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Metho)
19. Mirella Lapata. 2006. [Automatic evaluation of information ordering: Kendall’s tau](https://md.research.elamur.ai/free/en/papers/lapata-2006-automatic-evaluation-information-ordering-kendall/). Computational Linguistics, 32(4):471–484. [↗](https://doi.org/10.1162/coli.2006.32.4.471)
20. Mirella Lapata and Regina Barzilay. 2005. Automatic evaluation of text coherence: Models and representations. In IJCAI, volume 5, pages 1085–1090. [find](https://scholar.google.com/scholar?q=Mirella+Lapata+and+Regina+Barzilay%2E+2005%2E+Automatic+evaluation+of+text+coherence%3A+Models+and+representations%2E+In+IJCAI%2C+volume+5%2C+pages+1085%E2%80%931090%2E)
21. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. [A diversity-promoting objective function for neural conversation models](https://md.research.elamur.ai/free/en/papers/li-2016-diversity-promoting-objective-function-neural/). In Proceedings of NAACL-HLT, pages 110–119. [↗](https://doi.org/10.48550/arxiv.1510.03055)
22. Ryan Lowe, Iulian V Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. On the evaluation of dialogue systems with next utterance classification. In 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, page 264. [find](https://scholar.google.com/scholar?q=Ryan+Lowe%2C+Iulian+V+Serban%2C+Mike+Noseworthy%2C+Laurent+Charlin%2C+and+Joelle+Pineau%2E+2016%2E+On+the+evaluation+of+dialogue+systems+with+next+utterance+classification%2E+In+17th+Annual+Meeting+of+the+Special+I)
23. Ryan Thomas Lowe, Nissan Pow, Iulian Vlad Serban, Laurent Charlin, Chia-Wei Liu, and Joelle Pineau. 2017. Training end-to-end dialogue systems with the ubuntu dialogue corpus. Dialogue & Discourse, 8(1):31–65. [↗](https://doi.org/10.5087/dad.2017.102)
24. Shikib Mehri, Evgeniia Razumovskaia, Tiancheng Zhao, and Maxine Eskenazi. 2019. Pretraining methods for dialog context representation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3836–3845. [find](https://scholar.google.com/scholar?q=Shikib+Mehri%2C+Evgeniia+Razumovskaia%2C+Tiancheng+Zhao%2C+and+Maxine+Eskenazi%2E+2019%2E+Pretraining+methods+for+dialog+context+representation+learning%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Associa)
25. Mohsen Mesgar, Sebastian Bücker, and Iryna Gurevych. 2019. [A neural model for dialogue coherence assessment](https://md.research.elamur.ai/free/en/papers/mesgar-2020-dialogue-coherence-assessment-without-explicit/). arXiv preprint arXiv:1908.08486. [↗](https://arxiv.org/abs/1908.08486)
26. Han Cheol Moon, Muhammad Tasnim Mohiuddin, Shafiq Joty, and Chi Xu. 2019. A unified neural coherence model. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2262–2272. [↗](https://doi.org/10.48550/arxiv.1909.00349)
27. Dat Tien Nguyen and Shafiq Joty. 2017. A neural local coherence model. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1320–1330. [find](https://scholar.google.com/scholar?q=Dat+Tien+Nguyen+and+Shafiq+Joty%2E+2017%2E+A+neural+local+coherence+model%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29%2C+volume+1%2C+pag)
28. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. [Pytorch: An imperative style, high-performance deep learning library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/). In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc. [↗](https://doi.org/10.48550/arxiv.1912.01703)
29. Amruta Purandare and Diane J Litman. 2008. Analyzing dialog coherence using transition patterns in lexical and semantic features. In FLAIRS Conference, pages 195–200. [find](https://scholar.google.com/scholar?q=Amruta+Purandare+and+Diane+J+Litman%2E+2008%2E+Analyzing+dialog+coherence+using+transition+patterns+in+lexical+and+semantic+features%2E+In+FLAIRS+Conference%2C+pages+195%E2%80%93200%2E)
30. Gisela Redeker. 2000. Coherence and structure in text and discourse. Abduction, belief and context in dialogue, pages 233–263. [↗](https://doi.org/10.1075/nlp.1.06red)
31. Harvey Sacks and Gail Jefferson. 1995. Lectures on conversation. Human Studies, 18(2):327–336. [↗](https://doi.org/10.1002/9781444328301)
32. Emanuel A Schegloff. 1968. Sequencing in conversational openings. American anthropologist, 70(6):1075–1095. [find](https://scholar.google.com/scholar?q=Emanuel+A+Schegloff%2E+1968%2E+Sequencing+in+conversational+openings%2E+American+anthropologist%2C+70%286%29%3A1075%E2%80%931095%2E)
33. Iulian Vlad Serban, Ryan Lowe, Laurent Charlin, and Joelle Pineau. 2016. Generative deep neural networks for dialogue: A short review. arXiv preprint arXiv:1611.06216. [↗](https://arxiv.org/abs/1611.06216)
34. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. [Dialogue act modeling for automatic tagging and recognition of conversational speech](https://md.research.elamur.ai/free/en/papers/stolcke-2000-dialogue-act-modeling-automatic-tagging/). Computational linguistics, 26(3):339–373. [↗](https://doi.org/10.1162/089120100561737)
35. Henri Theil. 1961. Economic forecasts and policy. [find](https://scholar.google.com/scholar?q=Henri+Theil%2E+1961%2E+Economic+forecasts+and+policy%2E)
36. David R Traum. 1994. A computational theory of grounding in natural language conversation. Technical report, Rochester Univ NY Dept of Computer Science. [find](https://scholar.google.com/scholar?q=David+R+Traum%2E+1994%2E+A+computational+theory+of+grounding+in+natural+language+conversation%2E+Technical+report%2C+Rochester+Univ+NY+Dept+of+Computer+Science%2E)
37. Svitlana Vakulenko, Maarten de Rijke, Michael Cochez, Vadim Savenkov, and Axel Polleres. 2018. Measuring semantic coherence of a conversation. In International Semantic Web Conference, pages 634– 651. Springer. [↗](https://doi.org/10.48550/arxiv.1806.06411)
38. Anu Venkatesh, Chandra Khatri, Ashwin Ram, Fenfei Guo, Raefer Gabriel, Ashish Nagar, Rohit Prasad, Ming Cheng, Benham Hedayatnia, Angeliki Metalli-nou, Rahul Goel, Shaohua Yang, and Anirudh Raju. 2017. On evaluating and comparing conversational agents. In NIPS 2017 Conversational AI workshop. [find](https://scholar.google.com/scholar?q=Anu+Venkatesh%2C+Chandra+Khatri%2C+Ashwin+Ram%2C+Fenfei+Guo%2C+Raefer+Gabriel%2C+Ashish+Nagar%2C+Rohit+Prasad%2C+Ming+Cheng%2C+Benham+Hedayatnia%2C+Angeliki+Metalli%2Dnou%2C+Rahul+Goel%2C+Shaohua+Yang%2C+and+Anirudh+Raju%2E+2017)
39. Anthony J Viera, Joanne M Garrett, et al. 2005. Understanding interobserver agreement: the kappa statistic. Fam med, 37(5):360–363. [find](https://scholar.google.com/scholar?q=Anthony+J+Viera%2C+Joanne+M+Garrett%2C+et+al%2E+2005%2E+Understanding+interobserver+agreement%3A+the+kappa+statistic%2E+Fam+med%2C+37%285%29%3A360%E2%80%93363%2E)
40. Sanghyun Yi, Rahul Goel, Chandra Khatri, Alessandra Cervone, Tagyoung Chung, Behnam Hedayatnia, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tur. 2019. Towards coherent and engaging spoken dialog response generation using automatic conversation evaluators. In Proceedings of the 12th International Conference on Natural Language Generation, pages 65–75, Tokyo, Japan. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1904.13015)
41. Koichiro Yoshino, Chiori Hori, Julien Perez, Luis Fernando D’Haro, Lazaros Polymenakos, Chulaka Gunasekara, Walter S Lasecki, Jonathan K Kummerfeld, Michel Galley, Chris Brockett, Jianfeng Gao, Bill Dolan, Xiang Gao, Huda Alamari, Tim K. Marks, Devi Parikh, and Dhruv Batra. 2019. Dialog system technology challenge 7. NeurIPS Workshop: The 2nd Conversational AI: Todays Practice and Tomorrows Potential. [find](https://scholar.google.com/scholar?q=Koichiro+Yoshino%2C+Chiori+Hori%2C+Julien+Perez%2C+Luis+Fernando+D%E2%80%99Haro%2C+Lazaros+Polymenakos%2C+Chulaka+Gunasekara%2C+Walter+S+Lasecki%2C+Jonathan+K+Kummerfeld%2C+Michel+Galley%2C+Chris+Brockett%2C+Jianfeng+Gao%2C+Bill+D)
42. Xiangyang Zhou, Lu Li, Daxiang Dong, Yi Liu, Ying Chen, Wayne Xin Zhao, Dianhai Yu, and Hua Wu. 2018. [Multi-turn response selection for chatbots with deep attention matching network](https://md.research.elamur.ai/free/en/papers/zhou-2018-multi-turn-response-selection-chatbots/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1118–1127.
43. Yunxiao Zhou, Man Lan, and Wenting Wang. 2019. Hierarchical intention enhanced network for automatic dialogue coherence assessment. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE. [find](https://scholar.google.com/scholar?q=Yunxiao+Zhou%2C+Man+Lan%2C+and+Wenting+Wang%2E+2019%2E+Hierarchical+intention+enhanced+network+for+automatic+dialogue+coherence+assessment%2E+In+2019+International+Joint+Conference+on+Neural+Networks+%28IJCNN%29%2C+p)

*Updated 24 October 2026*