---
title: "Discourse Coherence in the Wild — Lai 2018"
description: "Coherence models have mostly been tested on polished, professionally edited text such as newswire, using artificial tasks like telling an original…"
authors: ["Alice Lai", "Joel Tetreault"]
year: 2018
venue: "Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue"
source: "https://aclanthology.org/W18-5023/"
doi: "10.18653/v1/w18-5023"
arxiv: "1805.04993"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/lai-2018-discourse-coherence-wild-dataset-evaluation/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/lai-2018-discourse-coherence-wild-dataset-evaluation/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2018](https://md.research.elamur.ai/free/papers/2018/) / Discourse Coherence in the Wild: A Dataset, Evaluation and Methods

# Discourse Coherence in the Wild: A Dataset, Evaluation and Methods

**Authors:** Alice Lai, [Joel Tetreault](https://md.research.elamur.ai/free/authors/joel-tetreault/) · **Year:** 2018 · **Venue:** Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W18-5023/) · [PDF](https://aclanthology.org/W18-5023.pdf) · [DOI 10.18653/v1/w18-5023](https://doi.org/10.18653/v1/w18-5023) · [arXiv 1805.04993](https://arxiv.org/abs/1805.04993) · [OpenAlex](https://openalex.org/W2962790423)

**Citations:** 46 citations (openalex-snapshot · 2026-10-14) · [33](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Coherence models have mostly been tested on polished, professionally edited text such as newswire, using artificial tasks like telling an original document from a shuffled version of its sentences. Lai and Tetreault ask how well these methods work on ordinary writing produced by non-professionals. They built the Grammarly Corpus of Discourse Coherence (GCDC), which covers four everyday genres: forum answers, two collections of emails, and business reviews. Each text received document-level coherence labels from both trained linguists and crowdworkers. The authors then compared seven established coherence algorithms with two new, simple neural models, SentAvg and ParSeq.

Neural approaches generally performed best. Entity-based models did poorly, partly because entity grids built from these texts are very sparse. Scores were far lower than the roughly 90% accuracies reported for sentence ordering. The authors argue that ordering is a misleading proxy for real coherence assessment. They recommend expert annotators over crowdworkers for this task. The data, guidelines, and code are publicly released.

## Key points
- **GCDC structure:** The corpus has four domains, Yahoo (Yahoo Answers L6), Clinton (State Department emails), Enron (Enron Corpus), and Yelp (Yelp Open Dataset). Each domain contains 1,200 texts of 100–300 words, split into 1,000 for training and 200 for testing.
- **Annotation:** Every text was rated on a 3-point scale (low/medium/high) by 3 of 13 expert raters and 5 of 62 qualified Mechanical Turk workers. Consensus labels were derived by averaging the ratings and applying thresholds (≤1.8 is low, >2.2 is high).
- **Agreement:** Expert ICC ranged from about 0.30 to 0.56, while MTurk ICC ranged from about 0.11 to 0.27. Crowdworkers also chose "medium" more often than experts did.
- **Benchmark and baselines:** Seven prior systems were benchmarked, among them EGrid, EGraph, EGridConv, and Clique. The proposed SentAvg and ParSeq are easier to implement than Clique and beat it in most experiments.
- **Order is not the key signal:** SentAvg disregards sentence order yet performed competitively. This suggests that recovering the original order is a different problem from separating low-coherence texts from high-coherence ones.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Discourse analysis](https://md.research.elamur.ai/free/topics/discourse-analysis/)

## Cited by

Cited by

33 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 30 not yet.

- [Automated evaluation of written discourse coherence using GPT-4](https://md.research.elamur.ai/free/papers/naismith-2023-automated-evaluation-written-discourse-coherence/)
- [A Multi-Task Dataset for Assessing Discourse Coherence in Chinese Essays: Structure, Theme, and Logic Analysis](https://md.research.elamur.ai/free/papers/wu-2023-multi-task-dataset-assessing-discourse/)
- [DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence](https://md.research.elamur.ai/free/papers/zhao-2023-discoscore-evaluating-text-generation-bert/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{lai2018discourse,
  title = {{Discourse Coherence in the Wild: A Dataset, Evaluation and Methods}},
  author = {Lai, Alice and Tetreault, Joel},
  booktitle = {Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue},
  year = {2018},
  doi = {10.18653/v1/w18-5023},
  eprint = {1805.04993},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/w18-5023}
}
```

- [← previous Discourse Annotation in the PDTB: The Next Generation](https://md.research.elamur.ai/free/papers/prasad-2018-discourse-annotation-pdtb-next-generation/)
- [next → Discourse Coherence: Concurrent Explicit and Implicit Relations](https://md.research.elamur.ai/free/papers/rohde-2018-discourse-coherence-concurrent-explicit-implicit/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/papers/2018/)

## References

25 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Regina Barzilay and Mirella Lapata. 2005. Modeling local coherence: An entity-based approach. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05). Association for Computational Linguistics, Ann Arbor, Michigan, pages 141–148. https://doi.org/10.3115/12… — [doi:10.3115/1219840.1219858](https://doi.org/10.3115/1219840.1219858) [↗](https://doi.org/10.3115/12…)
2. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735. [↗](https://doi.org/10.1162/neco.1997.9.8.1735)
3. Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Computational Linguistics 34(1):1–34. https://doi.org/10.1162/coli.2008.34.1.1. [↗](https://doi.org/10.1162/coli.2008.34.1.1) [our card](https://md.research.elamur.ai/free/papers/barzilay-2008-modeling-local-coherence-entity-based/)
4. J. Peter Kincaid, Robert P. Fishburne Jr., Richard L. Rogers, and Brad S. Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and Flesch reading ease formula) for navy enlisted personnel. Technical Report, DTIC Document. [find](https://scholar.google.com/scholar?q=J%2E+Peter+Kincaid%2C+Robert+P%2E+Fishburne+Jr%2E%2C+Richard+L%2E+Rogers%2C+and+Brad+S%2E+Chissom%2E+1975%2E+Derivation+of+new+readability+formulas+%28automated+readability+index%2C+fog+count+and+Flesch+reading+ease+formula%29)
5. Jill Burstein, Joel Tetreault, and Slava Andreyev. 2010. Using entity-based features to model coherence in student essays. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Jill+Burstein%2C+Joel+Tetreault%2C+and+Slava+Andreyev%2E+2010%2E+Using+entity%2Dbased+features+to+model+coherence+in+student+essays%2E+In+Human+Language+Technologies%3A+The+2010+Annual+Conference+of+the+North+Ameri)
6. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics 33(1):159–174. [http://www.jstor.org/stable/2529310](http://www.jstor.org/stable/2529310). [↗](http://www.jstor.org/stable/2529310)
7. Jill Burstein, Joel R. Tetreault, and Martin Chodorow. 2013. Holistic discourse coherence annotation for noisy essay writing. D&D 4(2):34–52. [http://dad.uni-bielefeld.de/index.php/dad/article/view/2825](http://dad.uni-bielefeld.de/index.php/dad/article/view/2825). [↗](http://dad.uni-bielefeld.de/index.php/dad/article/view/2825) [our card](https://md.research.elamur.ai/free/papers/burstein-2013-holistic-discourse-coherence-annotation-noisy/)
8. Mirella Lapata. 2003. Probabilistic text structuring: Experiments with sentence ordering. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Sapporo, Japan, pages 545–552. https://doi.org/10.3115/1075096.1075165. [↗](https://doi.org/10.3115/1075096.1075165)
9. Kevin Clark and Christopher D. Manning. 2015. Entity-centric coreference resolution with model stacking. In Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Kevin+Clark+and+Christopher+D%2E+Manning%2E+2015%2E+Entity%2Dcentric+coreference+resolution+with+model+stacking%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2E) [our card](https://md.research.elamur.ai/free/papers/clark-2015-entity-centric-coreference-resolution-model/)
10. Jiwei Li and Eduard Hovy. 2014. A model of coherence based on distributed sentence representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, pages 2039–2048. [http://www.aclweb.org/antholo…](http://www.aclweb.org/antholo…) [↗](http://www.aclweb.org/antholo…)
11. Micha Elsner and Eugene Charniak. 2008. Coreference-inspired coherence modeling. In Proceedings of ACL-08: HLT, Short Papers. Association for Computational Linguistics, Columbus, Ohio, pages 41–44. [http://www.aclweb.org/anthology/P/P08/P08-2011](http://www.aclweb.org/anthology/P/P08/P08-2011). [↗](http://www.aclweb.org/anthology/P/P08/P08-2011) [our card](https://md.research.elamur.ai/free/papers/elsner-2008-coreference-inspired-coherence-modeling/)
12. Jiwei Li and Dan Jurafsky. 2017. Neural net models of open-domain discourse coherence. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, pages 198–209. [https://www.aclweb.org/anthology/D17-1019](https://www.aclweb.org/anthology/D17-1019). [↗](https://www.aclweb.org/anthology/D17-1019) [our card](https://md.research.elamur.ai/free/papers/li-2017-neural-net-models-open-domain/)
13. Micha Elsner and Eugene Charniak. 2011. Extending the entity grid with entity-specific features. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Portland, Oregon, USA, pages 125– 129.… [find](https://scholar.google.com/scholar?q=Micha+Elsner+and+Eugene+Charniak%2E+2011%2E+Extending+the+entity+grid+with+entity%2Dspecific+features%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Human+Langu)
14. Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2011. Automatically evaluating text coherence using discourse relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Portland, Oregon, USA… [find](https://scholar.google.com/scholar?q=Ziheng+Lin%2C+Hwee+Tou+Ng%2C+and+Min%2DYen+Kan%2E+2011%2E+Automatically+evaluating+text+coherence+using+discourse+relations%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Lingui) [our card](https://md.research.elamur.ai/free/papers/lin-2011-automatically-evaluating-text-coherence-discourse/)
15. Vanessa Wei Feng, Ziheng Lin, and Graeme Hirst. 2014. The impact of deep hierarchical discourse structures in the evaluation of text coherence. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers. [find](https://scholar.google.com/scholar?q=Vanessa+Wei+Feng%2C+Ziheng+Lin%2C+and+Graeme+Hirst%2E+2014%2E+The+impact+of+deep+hierarchical+discourse+structures+in+the+evaluation+of+text+coherence%2E+In+Proceedings+of+COLING+2014%2C+the+25th+International+Co)
16. Dublin Swapna Somasundaran, Jill Burstein, and Martin Chodorow. 2014. Lexical chaining for measuring discourse coherence quality in test-taker essays. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers. Dublin City University and Associati… [find](https://scholar.google.com/scholar?q=Dublin+Swapna+Somasundaran%2C+Jill+Burstein%2C+and+Martin+Chodorow%2E+2014%2E+Lexical+chaining+for+measuring+discourse+coherence+quality+in+test%2Dtaker+essays%2E+In+Proceedings+of+COLING+2014%2C+the+25th+Internati) [our card](https://md.research.elamur.ai/free/papers/somasundaran-2014-lexical-chaining-measuring-discourse-coherence/)
17. Annie Louis and Ani Nenkova. 2012. A coherence model based on syntactic patterns. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Association for Computational Linguistics, Jeju Island, Korea, pages 1157–116… [find](https://scholar.google.com/scholar?q=Annie+Louis+and+Ani+Nenkova%2E+2012%2E+A+coherence+model+based+on+syntactic+patterns%2E+In+Proceedings+of+the+2012+Joint+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+Computational+Natu)
18. Mohsen Mesgar and Michael Strube. 2015. Graph-based coherence modeling for assessing readability. In Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics. Association for Computational Linguistics, Denver, Colorado, pages 309–318. [http://www.aclweb.org/anthology/S15-1036…](http://www.aclweb.org/anthology/S15-1036…) [↗](http://www.aclweb.org/anthology/S15-1036…)
19. Dat Tien Nguyen and Shafiq Joty. 2017. A neural local coherence model. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada, pages 1320–1330. [http://aclweb.org/anthology/P17-11…](http://aclweb.org/anthology/P17-11…) [↗](http://aclweb.org/anthology/P17-11…)
20. Mohsen Mesgar and Michael Strube. 2016. Lexical coherence graph modeling using word embeddings. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, San Diego, Cal… [find](https://scholar.google.com/scholar?q=Mohsen+Mesgar+and+Michael+Strube%2E+2016%2E+Lexical+coherence+graph+modeling+using+word+embeddings%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational)
21. Jane Morris and Graeme Hirst. 1991. Lexical cohesion computed by thesaural relations as an indicator of the structure of text. Computational Linguistics 17(1):21–48. [http://dl.acm.org/citation.cfm?id=971738.971740](http://dl.acm.org/citation.cfm?id=971738.971740). [↗](http://dl.acm.org/citation.cfm?id=971738.971740) [our card](https://md.research.elamur.ai/free/papers/morris-1991-lexical-cohesion-computed-thesaural-relations/)
22. Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task. Association for Computati… [find](https://scholar.google.com/scholar?q=Hwee+Tou+Ng%2C+Siew+Mei+Wu%2C+Ted+Briscoe%2C+Christian+Hadiwinoto%2C+Raymond+Hendy+Susanto%2C+and+Christopher+Bryant%2E+2014%2E+The+CoNLL%2D2014+shared+task+on+grammatical+error+correction%2E+In+Proceedings+of+the+Eigh)
23. Ellie Pavlick and Joel Tetreault. 2016. An empirical analysis of formality in online communication. Transactions of the Association for Computational Linguistics 4:61–74. [find](https://scholar.google.com/scholar?q=Ellie+Pavlick+and+Joel+Tetreault%2E+2016%2E+An+empirical+analysis+of+formality+in+online+communication%2E+Transactions+of+the+Association+for+Computational+Linguistics+4%3A61%E2%80%9374%2E) [our card](https://md.research.elamur.ai/free/papers/pavlick-2016-empirical-analysis-formality-online-communication/)
24. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12:2825… [find](https://scholar.google.com/scholar?q=F%2E+Pedregosa%2C+G%2E+Varoquaux%2C+A%2E+Gramfort%2C+V%2E+Michel%2C+B%2E+Thirion%2C+O%2E+Grisel%2C+M%2E+Blondel%2C+P%2E+Prettenhofer%2C+R%2E+Weiss%2C+V%2E+Dubourg%2C+J%2E+Vanderplas%2C+A%2E+Passos%2C+D%2E+Cournapeau%2C+M%2E+Brucher%2C+M%2E+Perrot%2C+and+E%2E+Duc)
25. Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark,… [find](https://scholar.google.com/scholar?q=Nils+Reimers+and+Iryna+Gurevych%2E+2017%2E+Reporting+score+distributions+makes+a+difference%3A+Performance+study+of+LSTM%2Dnetworks+for+sequence+tagging%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Met) [our card](https://md.research.elamur.ai/free/papers/reimers-2017-reporting-score-distributions-makes-difference/)