--- title: "Holistic Discourse Coherence Annotation for — Burstein 2013" description: "Automated essay scoring needs coherence measures that work on noisy student writing, which often contains grammatical errors, misspellings and weak…" authors: ["Jill Burstein", "Joel Tetreault", "Martin Chodorow"] year: 2013 venue: "Dialogue & Discourse Volume 4" source: "https://aclanthology.org/2013.dnd-4.3/" doi: "10.5087/dad.2013.202" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/burstein-2013-holistic-discourse-coherence-annotation-noisy/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/burstein-2013-holistic-discourse-coherence-annotation-noisy/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2013](https://md.research.elamur.ai/free/papers/2013/) / Holistic Discourse Coherence Annotation for Noisy Essay Writing # Holistic Discourse Coherence Annotation for Noisy Essay Writing **Authors:** [Jill Burstein](https://md.research.elamur.ai/free/authors/jill-burstein/), [Joel Tetreault](https://md.research.elamur.ai/free/authors/joel-tetreault/), [Martin Chodorow](https://md.research.elamur.ai/free/authors/martin-chodorow/) · **Year:** 2013 · **Venue:** Dialogue & Discourse Volume 4 · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/2013.dnd-4.3/) · [PDF](https://aclanthology.org/2013.dnd-4.3.pdf) · [DOI 10.5087/dad.2013.202](https://doi.org/10.5087/dad.2013.202) · [OpenAlex](https://openalex.org/W2136262501) **Citations:** 17 citations (openalex-snapshot · 2026-10-14) · [11](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Automated essay scoring needs coherence measures that work on noisy student writing, which often contains grammatical errors, misspellings and weak organization. The features behind these measures should also match the criteria in human scoring rubrics, so that scores can be explained and turned into feedback. Burstein, Tetreault and Chodorow propose a holistic coherence annotation scheme that annotators without deep linguistic training can apply quickly. Annotators read with the aim of judging the text rather than understanding it, then assign an overall coherence rating on a three-point scale. For modeling, these ratings are reduced to a binary low/high label. Most disagreement occurred at the middle level, where readers differ in the inferences they use to bridge gaps in the text. Even so, the binary label reached adequate reliability. Using these labels, the authors trained classifiers on 1555 essays from five assessment types. The models combine entity-grid, type/token, RST, LSA and e-rater grammar-error features. On every data set, the best combination outperformed baselines at detecting low-coherence essays. This shows that inexpensive annotation can support interpretable coherence scoring. ## Key points - The corpus has 1555 essays in five sub-corpora. EP/NNC and Summary/NNC are non-native college-level writing. EP/GL contains graduate-level essays, the PPE set comes from the Professional Proficiency Exam, and EP/6-12 spans grades 6–12. - Score Point 2 ("mostly coherent," with one or two points of confusion) was the hardest level to agree on. It was merged with Score Point 3 into a single "high" class for modeling. - The baselines were Majority Class, full e-rater features and ERGramError features. The best systems' F-scores for the "low" class were 74 (EP/NNC), 57 (Summary/NNC), 63 (EP/GL), 39 (PPE) and 64 (EP/6-12). - The best feature sets usually combined EntityGrid, ERGramError, Type/Token and RST, with maxLSA added for some sets. Summary/NNC was the exception: its best model used only ERGramError and RST. - Each feature type maps to a rubric criterion: - entity-grid transitions reflect how ideas are organized; - type/token reflects repetition of ideas; - RST reflects the discourse units that organize and develop the text; - grammatical errors reflect technical quality. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) Topics: [Discourse analysis](https://md.research.elamur.ai/free/topics/discourse-analysis/) ## Cited by Cited by 11 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 6 not yet. - [Representation Learning for Text-level Discourse Parsing](https://md.research.elamur.ai/free/papers/ji-2014-representation-learning-text-level-discourse/) - [Discourse Coherence in the Wild: A Dataset, Evaluation and Methods](https://md.research.elamur.ai/free/papers/lai-2018-discourse-coherence-wild-dataset-evaluation/) - [Automated evaluation of written discourse coherence using GPT-4](https://md.research.elamur.ai/free/papers/naismith-2023-automated-evaluation-written-discourse-coherence/) - [Lexical Chaining for Measuring Discourse Coherence Quality in Test-taker Essays](https://md.research.elamur.ai/free/papers/somasundaran-2014-lexical-chaining-measuring-discourse-coherence/) - [Joint Syntacto-Discourse Parsing and the Syntacto-Discourse Treebank](https://md.research.elamur.ai/free/papers/zhao-2017-joint-syntacto-discourse-parsing-syntacto/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{burstein2013holistic, title = {{Holistic Discourse Coherence Annotation for Noisy Essay Writing}}, author = {Burstein, Jill and Tetreault, Joel and Chodorow, Martin}, booktitle = {Dialogue \& Discourse Volume 4}, year = {2013}, doi = {10.5087/dad.2013.202}, url = {https://doi.org/10.5087/dad.2013.202} } ``` - [← previous Global Inference for Bridging Anaphora Resolution](https://md.research.elamur.ai/free/papers/hou-2013-global-inference-bridging-anaphora-resolution/) - [next → Implicatures and Nested Beliefs in Approximate Decentralized-POMDPs](https://md.research.elamur.ai/free/papers/vogel-2013-implicatures-nested-beliefs-approximate-decentralized/) - [↑ back to papers of 2013](https://md.research.elamur.ai/free/papers/2013/) ## References 3 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Scheme Class FS Text Type Essay Responses Miltsakaki and Kukich (2000) Higgins et al. (2004) [find](https://scholar.google.com/scholar?q=Scheme+Class+FS+Text+Type+Essay+Responses+Miltsakaki+and+Kukich+%282000%29+Higgins+et+al%2E+%282004%29) 2. FS Essay Responses Wolf and Gibson (2005) [find](https://scholar.google.com/scholar?q=FS+Essay+Responses+Wolf+and+Gibson+%282005%29) 3. FS Wall Street Journal & AP Newswire articles Pitler and Nenkova (2008) Barzilay and Lapata (2008) [find](https://scholar.google.com/scholar?q=FS+Wall+Street+Journal+%26+AP+Newswire+articles+Pitler+and+Nenkova+%282008%29+Barzilay+and+Lapata+%282008%29)