---
title: "Automated evaluation of written discourse — Naismith 2023"
description: "Automatically judging discourse coherence, meaning how well the ideas in a text connect and flow, has long been a weak spot of automated writing…"
authors: ["Ben Naismith", "Phoebe Mulcaire", "Jill Burstein"]
year: 2023
venue: "Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)"
source: "https://aclanthology.org/2023.bea-1.32/"
doi: "10.18653/v1/2023.bea-1.32"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/naismith-2023-automated-evaluation-written-discourse-coherence/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/naismith-2023-automated-evaluation-written-discourse-coherence/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2023](https://md.research.elamur.ai/free/papers/2023/) / Automated evaluation of written discourse coherence using GPT-4

# Automated evaluation of written discourse coherence using GPT-4

**Authors:** Ben Naismith, [Phoebe Mulcaire](https://md.research.elamur.ai/free/authors/phoebe-mulcaire/), [Jill Burstein](https://md.research.elamur.ai/free/authors/jill-burstein/) · **Year:** 2023 · **Venue:** Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.bea-1.32/) · [PDF](https://aclanthology.org/2023.bea-1.32.pdf) · [DOI 10.18653/v1/2023.bea-1.32](https://doi.org/10.18653/v1/2023.bea-1.32) · [OpenAlex](https://openalex.org/W4385572239)

**Citations:** 75 citations (openalex-snapshot · 2026-10-14) · [45](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Automatically judging discourse coherence, meaning how well the ideas in a text connect and flow, has long been a weak spot of automated writing evaluation (AWE). This study tests whether GPT-4 can score coherence in learner writing the way trained human raters do, and explain its scores. The authors built DET-Coh, a set of 500 Writing Sample responses from the Duolingo English Test, balanced by gender, first language and proficiency level. Expert raters scored the responses on a new six-level coherence rubric aligned with CEFR. GPT-4 was queried through the OpenAI API with a prompt containing the task description, the rubric, style guidelines, six scored examples, the writing prompt and the essay. Three setups were tried: score then explanation, explanation then score, and score alone. A regression baseline built on Coh-Metrix-style overlap and similarity features served as the comparison. Every GPT-4 setup matched human scores far better than the baseline. The results suggest LLMs could give interpretable coherence feedback, but the authors warn that the generated rationales may not reflect how GPT-4 actually reached its ratings.

## Key points
- **DET-Coh dataset:** 500 responses from 2022, covering seven L1 groups (Chinese, Arabic, Spanish, Telugu, English, Bengali, Gujarati) and CEFR levels A1–C2. A score of 0 was reserved for blank or bad-faith answers.
- **Human reliability:** Four expert raters took part, and 80 responses were scored by two raters independently. On these, agreement reached 0.72 exact and 0.93 QWK.
- **GPT-4 vs. humans:**
  - QWK was 0.81–0.82 for the conditions that included rationales and 0.78 for rating-only.
  - Spearman ρ reached up to 0.82.
  - Adjacent agreement was 0.95–0.97.
  - Exact agreement was 0.51–0.56.
- **Baseline:** The baseline used local and global argument, noun and stem overlap, coreference overlap and LSA similarity, with leave-one-out linear regression. It reached only 0.39 QWK, ρ 0.47, 0.36 exact and 0.82 adjacent agreement.
- **Qualitative findings:**
  - GPT-4 appeared to give spelling errors too much weight.
  - GPT-4 could also rewrite a learner's essay into a personalized model answer, which both GPT-4 and a human rater scored 5 (CEFR C1).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Discourse analysis](https://md.research.elamur.ai/free/topics/discourse-analysis/)

## Cited by

Cited by

45 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 42 not yet.

- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://md.research.elamur.ai/free/papers/bavaresco-2024-llms-instead-human-judges-large/)
- [Orca 2: Teaching Small Language Models How to Reason](https://md.research.elamur.ai/free/papers/mitra-2023-orca-2-teaching-small-language/)
- [Instruction-Following Evaluation for Large Language Models](https://md.research.elamur.ai/free/papers/zhou-2023-instruction-following-evaluation-large-language/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{naismith2023automated,
  title = {{Automated evaluation of written discourse coherence using GPT-4}},
  author = {Naismith, Ben and Mulcaire, Phoebe and Burstein, Jill},
  booktitle = {Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)},
  year = {2023},
  doi = {10.18653/v1/2023.bea-1.32},
  url = {https://doi.org/10.18653/v1/2023.bea-1.32}
}
```

- [← previous AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://md.research.elamur.ai/free/papers/wu-2023-autogen-enabling-next-gen-llm/)
- [next → Automatic Evaluate Dialogue Appropriateness by Using Dialogue Act](https://md.research.elamur.ai/free/papers/chen-2023-automatic-evaluate-dialogue-appropriateness-dialogue/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/papers/2023/)

## References

35 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater® v.2. The Journal of Technology, Learning and Assessment, 4(3). [find](https://scholar.google.com/scholar?q=Yigal+Attali+and+Jill+Burstein%2E+2006%2E+Automated+essay+scoring+with+e%2Drater%C2%AE+v%2E2%2E+The+Journal+of+Technology%2C+Learning+and+Assessment%2C+4%283%29%2E)
2. Yigal Attali, Andrew Runge, Geoff T. LaFlair, Kevin Yancey, Sarah Goodwin, Yena Park, and Alina A. Von Davier. 2022. The interactive reading task: Transformer-based automatic item generation. Frontiers in Artificial Intelligence, 5. [find](https://scholar.google.com/scholar?q=Yigal+Attali%2C+Andrew+Runge%2C+Geoff+T%2E+LaFlair%2C+Kevin+Yancey%2C+Sarah+Goodwin%2C+Yena+Park%2C+and+Alina+A%2E+Von+Davier%2E+2022%2E+The+interactive+reading+task%3A+Transformer%2Dbased+automatic+item+generation%2E+Frontier)
3. Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Computational Linguistics, 34(1):1–34. [find](https://scholar.google.com/scholar?q=Regina+Barzilay+and+Mirella+Lapata%2E+2008%2E+Modeling+local+coherence%3A+An+entity%2Dbased+approach%2E+Computational+Linguistics%2C+34%281%29%3A1%E2%80%9334%2E) [our card](https://md.research.elamur.ai/free/papers/barzilay-2008-modeling-local-coherence-entity-based/)
4. Daniel Blanchard, Joel R. Tetreault, Derrick Higgins, A. Cahill, and Martin Chodorow. 2013. TOEFL11: A corpus of non-native english. ETS Research Report Series, 2013:15. [find](https://scholar.google.com/scholar?q=Daniel+Blanchard%2C+Joel+R%2E+Tetreault%2C+Derrick+Higgins%2C+A%2E+Cahill%2C+and+Martin+Chodorow%2E+2013%2E+TOEFL11%3A+A+corpus+of+non%2Dnative+english%2E+ETS+Research+Report+Series%2C+2013%3A15%2E)
5. Bren Bridgeman. 2013. Human ratings and automated essay evaluation, pages 243–254. Routledge. [find](https://scholar.google.com/scholar?q=Bren+Bridgeman%2E+2013%2E+Human+ratings+and+automated+essay+evaluation%2C+pages+243%E2%80%93254%2E+Routledge%2E)
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, et al. 2020. Language models are few-shot learners. In Advances in neural information processing systems, volume 33, pages 1877–1901. [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D%2E+Kaplan%2C+Prafulla+Dhariwal%2C+et+al%2E+2020%2E+Language+models+are+few%2Dshot+learners%2E+In+Advances+in+neural+information+processing+systems%2C+vol)
7. Jill Burstein. 2023. Duolingo English Test Responsible AI Standards. Technical report, Duolingo. [find](https://scholar.google.com/scholar?q=Jill+Burstein%2E+2023%2E+Duolingo+English+Test+Responsible+AI+Standards%2E+Technical+report%2C+Duolingo%2E)
8. Jill Burstein, Joel Tetreault, and Martin Chodorow. 2013. Holistic discourse coherence annotation for noisy essay writing. Dialogue & Discourse, 4(2):34–52. [find](https://scholar.google.com/scholar?q=Jill+Burstein%2C+Joel+Tetreault%2C+and+Martin+Chodorow%2E+2013%2E+Holistic+discourse+coherence+annotation+for+noisy+essay+writing%2E+Dialogue+%26+Discourse%2C+4%282%29%3A34%E2%80%9352%2E) [our card](https://md.research.elamur.ai/free/papers/burstein-2013-holistic-discourse-coherence-annotation-noisy/)
9. Marti A. Hearst. 1997. Text tiling: Segmenting text into multi-paragraph subtopic passages. In Computational Linguistics, volume 23, pages 33–64. [find](https://scholar.google.com/scholar?q=Marti+A%2E+Hearst%2E+1997%2E+Text+tiling%3A+Segmenting+text+into+multi%2Dparagraph+subtopic+passages%2E+In+Computational+Linguistics%2C+volume+23%2C+pages+33%E2%80%9364%2E)
10. International Test Commission and Association of Test Publishers (ITC-ATP). 2022. Guidelines for technology-based assessment. Technical report, International Test Commission and Association of Test Publishers, Washington, D.C. [find](https://scholar.google.com/scholar?q=International+Test+Commission+and+Association+of+Test+Publishers+%28ITC%2DATP%29%2E+2022%2E+Guidelines+for+technology%2Dbased+assessment%2E+Technical+report%2C+International+Test+Commission+and+Association+of+Test+Pu)
11. Ramsey Cardwell, Ben Naismith, Geoffrey T. LaFlair, and Steven Nydick. 2023. Duolingo English Test: Technical Manual. [find](https://scholar.google.com/scholar?q=Ramsey+Cardwell%2C+Ben+Naismith%2C+Geoffrey+T%2E+LaFlair%2C+and+Steven+Nydick%2E+2023%2E+Duolingo+English+Test%3A+Technical+Manual%2E)
12. Paraskevas Lagakis and Stavros Demetriadis. 2021. Automated essay scoring: A review of the field. In Proceedings of the 2021 International Conference on Computer, Information and Telecommunication Systems (CITS), pages 1–6. Institute of Electrical and Electronics Engineers. [find](https://scholar.google.com/scholar?q=Paraskevas+Lagakis+and+Stavros+Demetriadis%2E+2021%2E+Automated+essay+scoring%3A+A+review+of+the+field%2E+In+Proceedings+of+the+2021+International+Conference+on+Computer%2C+Information+and+Telecommunication+Sys)
13. Council of Europe. 2001. Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Council of Europe Publishing, Cambridge, UK. [find](https://scholar.google.com/scholar?q=Council+of+Europe%2E+2001%2E+Common+European+Framework+of+Reference+for+Languages%3A+Learning%2C+Teaching%2C+Assessment%2E+Council+of+Europe+Publishing%2C+Cambridge%2C+UK%2E)
14. Alice Lai and Joel Tetreault. 2018. Discourse coherence in the wild: a dataset, evaluation and methods. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 214–223. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alice+Lai+and+Joel+Tetreault%2E+2018%2E+Discourse+coherence+in+the+wild%3A+a+dataset%2C+evaluation+and+methods%2E+In+Proceedings+of+the+19th+Annual+SIGdial+Meeting+on+Discourse+and+Dialogue%2C+pages+214%E2%80%93223%2E+Asso) [our card](https://md.research.elamur.ai/free/papers/lai-2018-discourse-coherence-wild-dataset-evaluation/)
15. Council of Europe. 2020. Common European Framework of Reference for Languages: Learning, teaching, assessment – Companion volume. Council of Europe Publishing, Cambridge, UK. [find](https://scholar.google.com/scholar?q=Council+of+Europe%2E+2020%2E+Common+European+Framework+of+Reference+for+Languages%3A+Learning%2C+teaching%2C+assessment+%E2%80%93+Companion+volume%2E+Council+of+Europe+Publishing%2C+Cambridge%2C+UK%2E)
16. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of NAACL, pages 687– 692. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Luheng+He%2C+and+Luke+Zettlemoyer%2E+2018%2E+Higher%2Dorder+coreference+resolution+with+coarse%2Dto%2Dfine+inference%2E+In+Proceedings+of+NAACL%2C+pages+687%E2%80%93+692%2E+Association+for+Computational+Linguistics) [our card](https://md.research.elamur.ai/free/papers/lee-2018-higher-order-coreference-resolution-coarse/)
17. Department for Science, Technology & Innovation. 2023. A pro-innovation approach to AI regulation. White paper, Crown. [find](https://scholar.google.com/scholar?q=Department+for+Science%2C+Technology+%26+Innovation%2E+2023%2E+A+pro%2Dinnovation+approach+to+AI+regulation%2E+White+paper%2C+Crown%2E)
18. Tim F. McNamara, Ute Knoch, and Jason Fan. 2019. Fairness, Justice and Language Assessment. Oxford University Press, Oxford, UK. [find](https://scholar.google.com/scholar?q=Tim+F%2E+McNamara%2C+Ute+Knoch%2C+and+Jason+Fan%2E+2019%2E+Fairness%2C+Justice+and+Language+Assessment%2E+Oxford+University+Press%2C+Oxford%2C+UK%2E)
19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL, pages 4171– 4186, Minneapolis, Minnesota. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+NAACL%2C+pages+4171%E2%80%93+4186%2C+Min)
20. Atsushi Mizumoto and Masaki Eguchi. 2023. Exploring the potential of using an AI language model for automated essay scoring. Educational Technology Research and Development. [find](https://scholar.google.com/scholar?q=Atsushi+Mizumoto+and+Masaki+Eguchi%2E+2023%2E+Exploring+the+potential+of+using+an+AI+language+model+for+automated+essay+scoring%2E+Educational+Technology+Research+and+Development%2E)
21. Virginia Dignum. 2021. The role and challenges of education for responsible AI. London Review of Education, 19(1):1–11. [find](https://scholar.google.com/scholar?q=Virginia+Dignum%2E+2021%2E+The+role+and+challenges+of+education+for+responsible+AI%2E+London+Review+of+Education%2C+19%281%29%3A1%E2%80%9311%2E)
22. National Governors Association Center for Best Practices and Council of Chief State School Officers. 2010. Common Core State Standards. [find](https://scholar.google.com/scholar?q=National+Governors+Association+Center+for+Best+Practices+and+Council+of+Chief+State+School+Officers%2E+2010%2E+Common+Core+State+Standards%2E)
23. OpenAI. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023%2E+GPT%2D4+technical+report%2E+arXiv+preprint+arXiv%3A2303%2E08774%2E) [our card](https://md.research.elamur.ai/free/papers/openai-2023-gpt-4-technical-report/)
24. Mark D. Shermis and Jill Burstein. 2003. Automated essay scoring: A cross-disciplinary perspective. Routledge, Mahwah, NJ. [find](https://scholar.google.com/scholar?q=Mark+D%2E+Shermis+and+Jill+Burstein%2E+2003%2E+Automated+essay+scoring%3A+A+cross%2Ddisciplinary+perspective%2E+Routledge%2C+Mahwah%2C+NJ%2E)
25. Lance Eliot. 2022. Enraged worries that generative AI ChatGPT spurs students to vastly cheat when writing essays, spawns spellbound attention for AI ethics and AI law. Forbes. [find](https://scholar.google.com/scholar?q=Lance+Eliot%2E+2022%2E+Enraged+worries+that+generative+AI+ChatGPT+spurs+students+to+vastly+cheat+when+writing+essays%2C+spawns+spellbound+attention+for+AI+ethics+and+AI+law%2E+Forbes%2E)
26. Mark D. Shermis and Jill Burstein. 2013. Handbook of automated essay evaluation. Routledge, New York, NY. [find](https://scholar.google.com/scholar?q=Mark+D%2E+Shermis+and+Jill+Burstein%2E+2013%2E+Handbook+of+automated+essay+evaluation%2E+Routledge%2C+New+York%2C+NY%2E)
27. Tyna Eloundo, Sam Manning, Pamela Mishkin, and Daniel Rock. 2023. GPTs are GPTs: An early look at the labor market impact potential of large language models. arXiv preprint axXiv:2303.10130. [find](https://scholar.google.com/scholar?q=Tyna+Eloundo%2C+Sam+Manning%2C+Pamela+Mishkin%2C+and+Daniel+Rock%2E+2023%2E+GPTs+are+GPTs%3A+An+early+look+at+the+labor+market+impact+potential+of+large+language+models%2E+arXiv+preprint+axXiv%3A2303%2E10130%2E)
28. Steve Ferrara and Saed Qunbar. 2022. Validity arguments for AI-based automated scores: Essay scoring as an illustration. Journal of Educational Measurement. [find](https://scholar.google.com/scholar?q=Steve+Ferrara+and+Saed+Qunbar%2E+2022%2E+Validity+arguments+for+AI%2Dbased+automated+scores%3A+Essay+scoring+as+an+illustration%2E+Journal+of+Educational+Measurement%2E)
29. Arthur C. Graesser, Danielle S. McNamara, Max M. Louwerse, and Zhiquiang Cai. 2004. Coh-metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, & Computers, 36(2):193–202. [find](https://scholar.google.com/scholar?q=Arthur+C%2E+Graesser%2C+Danielle+S%2E+McNamara%2C+Max+M%2E+Louwerse%2C+and+Zhiquiang+Cai%2E+2004%2E+Coh%2Dmetrix%3A+Analysis+of+text+on+cohesion+and+language%2E+Behavior+Research+Methods%2C+Instruments%2C+%26+Computers%2C+36%282%29%3A19)
30. Michael A. K. Halliday and Ruqaiya Hasan. 1976. Cohesion in English. Longman, London, UK. [find](https://scholar.google.com/scholar?q=Michael+A%2E+K%2E+Halliday+and+Ruqaiya+Hasan%2E+1976%2E+Cohesion+in+English%2E+Longman%2C+London%2C+UK%2E)
31. Swapna Somasundaran, Jill Burstein, and Martin Chodorow. 2014. Lexical chaining for measuring discourse coherence quality in test-taker essays. In The 25th International Conference on Computational Linguistics (COLING), pages 23–29. [find](https://scholar.google.com/scholar?q=Swapna+Somasundaran%2C+Jill+Burstein%2C+and+Martin+Chodorow%2E+2014%2E+Lexical+chaining+for+measuring+discourse+coherence+quality+in+test%2Dtaker+essays%2E+In+The+25th+International+Conference+on+Computational+Li) [our card](https://md.research.elamur.ai/free/papers/somasundaran-2014-lexical-chaining-measuring-discourse-coherence/)
32. The International Privacy Subcommittee of the ATP Security Committee (ATP). 2021. Artificial intelligence and the testing industry: A primer. Association of Test Publishers. [find](https://scholar.google.com/scholar?q=The+International+Privacy+Subcommittee+of+the+ATP+Security+Committee+%28ATP%29%2E+2021%2E+Artificial+intelligence+and+the+testing+industry%3A+A+primer%2E+Association+of+Test+Publishers%2E)
33. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman. 2023. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. arXiv preprint arXiv:2305.04388. [find](https://scholar.google.com/scholar?q=Miles+Turpin%2C+Julian+Michael%2C+Ethan+Perez%2C+and+Samuel+R+Bowman%2E+2023%2E+Language+models+don%E2%80%99t+always+say+what+they+think%3A+Unfaithful+explanations+in+chain%2Dof%2Dthought+prompting%2E+arXiv+preprint+arXiv%3A2305) [our card](https://md.research.elamur.ai/free/papers/turpin-2023-language-models-don-t-always/)
34. Paul Van den Broek, Charles R. Fletcher, and Kirsten Risden. 2009. Investigations of inferential processes in reading: A theoretical and methodological integration. Discourse Processes, 16:169–180. [find](https://scholar.google.com/scholar?q=Paul+Van+den+Broek%2C+Charles+R%2E+Fletcher%2C+and+Kirsten+Risden%2E+2009%2E+Investigations+of+inferential+processes+in+reading%3A+A+theoretical+and+methodological+integration%2E+Discourse+Processes%2C+16%3A169%E2%80%93180%2E)
35. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS). [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Maarten+Bosma%2C+Brian+Ichter%2C+Fei+Xia%2C+Ed+H%2E+Chi%2C+Quoc+V%2E+Le%2C+and+Denny+Zhou%2E+2022%2E+Chain%2Dof%2Dthought+prompting+elicits+reasoning+in+large+language+models%2E+In+Ad) [our card](https://md.research.elamur.ai/free/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)