---
title: "Modeling Organization in Student Essays — Persing 2010"
description: "The paper tackles automatic scoring of essay organization : how well an essay introduces… 0.5 points. 26 references, cited by 62 papers here."
authors: ["Isaac Persing", "Alan Davis", "Vincent Ng"]
year: 2010
venue: "Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D10-1023/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/persing-2010-modeling-organization-student-essays/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/persing-2010-modeling-organization-student-essays/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2010](https://md.research.elamur.ai/free/en/papers/2010/) / Modeling Organization in Student Essays

# Modeling Organization in Student Essays

**Authors:** [Isaac Persing](https://md.research.elamur.ai/free/en/authors/isaac-persing/), Alan Davis and [Vincent Ng](https://md.research.elamur.ai/free/en/authors/vincent-ng/) · **Year:** 2010 · **Venue:** [Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2010](https://md.research.elamur.ai/free/en/venues/emnlp/2010/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/D10-1023/) · [PDF](https://aclanthology.org/D10-1023.pdf) · [OpenAlex](https://openalex.org/W205093244)

**Citations:** 76 citations (OpenAlex, 2026-10-14) · [62](https://md.research.elamur.ai/free/en/papers/persing-2010-modeling-organization-student-essays/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper tackles automatic scoring of essay *organization*: how well an essay introduces a topic, takes a stance, backs it up and wraps up. Most essay scorers give one holistic grade, and organization has rarely been modeled. The authors annotated 1003 argumentative essays from the International Corpus of Learner English (ICLE) with organization scores from 1 to 4 in half-point steps, and released the data. They first assign discourse roles heuristically, to sentences and then to paragraphs, using paragraph position and the types of sentences a paragraph contains. The resulting label sequences are then used to predict a score in two ways: through heuristic sequence-alignment systems, and through SVM regressors. The SVMs use a linear kernel over flat features, a string kernel, and a newly proposed alignment kernel. Each kernel contributes useful information. Combining all three gives the best system, which significantly beats the strongest heuristic baseline. The work matters because it sets a first baseline and a public benchmark for a discourse-level task that previously had little annotated data.

## Key points
- **Data:** 1003 argumentative ICLE essays covering twelve topics and fifteen native languages. Scores range from 1.0 to 4.0 in half-point steps, and most essays fall at 3.0 (416) or 3.5 (289).
- **Inter-annotator agreement:** 846 essays were scored by two annotators. They agreed exactly only 29% of the time, but came within 0.5 points on 71% and within 1.0 point on 93% of essays. Treating one annotator as gold and the other as predictions gives a mean error of 0.54 and an MSE of 0.50.
- **Label schemas:**
  - Paragraphs get four labels: Introduction, Body, Rebuttal, Conclusion.
  - Sentences get ten labels, such as Thesis, Main Idea, Support, Rebuttal, Solution and Suggestion. These extend the scheme of Burstein et al. (2003).
- **Model comparison:**
  - The best single-kernel model is the linear kernel (Rl nps). Its main gain over the best heuristic (H p) is that it removes the need to pick among 12 heuristic variants.
  - The string and alignment kernels are weak alone, but their combination (Rsa) is the best two-kernel system.
  - The three-kernel model significantly beats H p (p \< 0.05 on S1, p \< 0.01 on S3).
- **Feature analysis:** Backward elimination on the linear-kernel features shows nearest-neighbor features matter most and flat paragraph-sequence features matter least. Removing any feature group still hurts performance.

## Interesting ideas
- **Labels as a symbol string:** Each essay becomes a short string of discourse labels, which can then be compared to other essays with bioinformatics-style sequence alignment. The same trick could suit any task where the *order* of structural units matters, such as dialogue flow or document templates.
- **Stacking heuristic labels:** Without gold paragraph-role annotations, the authors infer sentence roles first and then derive paragraph roles from position plus sentence content. This is a cheap way to get structure for a downstream task when annotating intermediate layers is too costly.
- **Combining weak kernels:** The string and alignment kernels perform poorly on their own but excel when combined. Evaluating complementary representations together, not just individually, can therefore reveal value that ablations of single components would hide.
- **Agreement within a tolerance:** Reporting agreement within 0.5 and 1.0 points, plus error measures, gives a more honest ceiling for fine-grained ordinal scoring than exact agreement alone. This is useful for any subjective graded annotation.
- **Learning to avoid choosing:** Feeding many heuristic signals into a learner can match the best hand-picked heuristic, so no single heuristic has to be selected up front. This is a practical pattern whenever several plausible rule sets compete.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The University of Texas at Dallas](https://md.research.elamur.ai/free/en/universities/the-university-of-texas-at-dallas/) — University, Richardson, United States · 47 works · 5,363 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

## Cite

Cite as: Isaac Persing, Alan Davis and Vincent Ng. 2010. Modeling Organization in Student Essays. Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{persing2010modeling,
  title = {{Modeling Organization in Student Essays}},
  author = {Isaac Persing, Alan Davis and Vincent Ng},
  booktitle = {Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing},
  year = {2010},
  url = {https://aclanthology.org/D10-1023/}
}
```

- [← previous Practical Very Large Scale CRFs](https://md.research.elamur.ai/free/en/papers/lavergne-2010-practical-very-large-scale-crfs/)
- [next → Morphological Annotation of Quranic Arabic](https://md.research.elamur.ai/free/en/papers/dukes-2010-morphological-annotation-quranic-arabic/)
- [↑ back to papers of 2010](https://md.research.elamur.ai/free/en/papers/2010/)

## References

26 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater V.2. Journal of Technology, Learning, and Assessment, 4(3). [find](https://scholar.google.com/scholar?q=Yigal+Attali+and+Jill+Burstein%2E+2006%2E+Automated+essay+scoring+with+e%2Drater+V%2E2%2E+Journal+of+Technology%2C+Learning%2C+and+Assessment%2C+4%283%29%2E)
2. Regina Barzilay and Mirella Lapata. 2005. Modeling local coherence: An entity-based approach. In Proceedings of the ACL, pages 141–148. [find](https://scholar.google.com/scholar?q=Regina+Barzilay+and+Mirella+Lapata%2E+2005%2E+Modeling+local+coherence%3A+An+entity%2Dbased+approach%2E+In+Proceedings+of+the+ACL%2C+pages+141%E2%80%93148%2E)
3. Regina Barzilay and Lillian Lee. 2002. Bootstrapping lexical choice via multiple-sequence alignment. In Proceedings of EMNLP, pages 164–171. [find](https://scholar.google.com/scholar?q=Regina+Barzilay+and+Lillian+Lee%2E+2002%2E+Bootstrapping+lexical+choice+via+multiple%2Dsequence+alignment%2E+In+Proceedings+of+EMNLP%2C+pages+164%E2%80%93171%2E)
4. Regina Barzilay and Lillian Lee. 2003. [Learning to paraphrase: An unsupervised approach using multiple-sequence alignment](https://md.research.elamur.ai/free/en/papers/barzilay-2003-learning-paraphrase-unsupervised-approach-multiple/). In HLT-NAACL 2003: Main Proceedings, pages 16–23.
5. Regina Barzilay and Lillian Lee. 2004. [Catching the drift: Probabilistic content models, with applications to generation and summarization](https://md.research.elamur.ai/free/en/papers/barzilay-2004-catching-drift-probabilistic-content-models/). In HLT-NAACL 2004: Main Proceedings, pages 113–120.
6. Razvan Bunescu and Raymond Mooney. 2005. [A shortest path dependency kernel for relation extraction](https://md.research.elamur.ai/free/en/papers/bunescu-2005-shortest-path-dependency-kernel-relation/). In Proceedings of HLT/EMNLP, pages 724–731.
7. Jill Burstein, Martin Chodorow, and Claudia Leacock. 2004. Automated essay evaluation: The Criterion online writing evaluation service. AI Magazine, 25(3), 27–36. [find](https://scholar.google.com/scholar?q=Jill+Burstein%2C+Martin+Chodorow%2C+and+Claudia+Leacock%2E+2004%2E+Automated+essay+evaluation%3A+The+Criterion+online+writing+evaluation+service%2E+AI+Magazine%2C+25%283%29%2C+27%E2%80%9336%2E)
8. Jill Burstein, Daniel Marcu, and Kevin Knight. 2003. Finding the write stuff: Automatic identification of discourse structure in student essays. IEEE Intelligent Systems, 18(1):32–39. [find](https://scholar.google.com/scholar?q=Jill+Burstein%2C+Daniel+Marcu%2C+and+Kevin+Knight%2E+2003%2E+Finding+the+write+stuff%3A+Automatic+identification+of+discourse+structure+in+student+essays%2E+IEEE+Intelligent+Systems%2C+18%281%29%3A32%E2%80%9339%2E)
9. Michael Collins and Nigel Duffy. 2002. New ranking algorithms for parsing and tagging: Kernels over discrete structures, and the voted perceptron. In Proceedings of the ACL, pages 263–270. [find](https://scholar.google.com/scholar?q=Michael+Collins+and+Nigel+Duffy%2E+2002%2E+New+ranking+algorithms+for+parsing+and+tagging%3A+Kernels+over+discrete+structures%2C+and+the+voted+perceptron%2E+In+Proceedings+of+the+ACL%2C+pages+263%E2%80%93270%2E)
10. Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine Learning, 20(3):273–297. [find](https://scholar.google.com/scholar?q=Corinna+Cortes+and+Vladimir+Vapnik%2E+1995%2E+Support%2Dvector+networks%2E+Machine+Learning%2C+20%283%29%3A273%E2%80%93297%2E)
11. Scott Elliot. 2001. IntelliMetric: From here to validity. Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA. [find](https://scholar.google.com/scholar?q=Scott+Elliot%2E+2001%2E+IntelliMetric%3A+From+here+to+validity%2E+Paper+presented+at+the+annual+meeting+of+the+American+Educational+Research+Association%2C+Seattle%2C+WA%2E)
12. Micha Elsner, Joseph Austerweil, and Eugene Charniak. 2007. [A unified local and global model for discourse coherence](https://md.research.elamur.ai/free/en/papers/elsner-2007-unified-local-global-model-discourse/). In NAACL HLT 2007: Proceedings of the Main Conference, pages 436–443.
13. Sylviane Granger, Estelle Dagneaux, Fanny Meunier, and Magali Paquot. 2009. International Corpus of Learner English (Version 2). Presses universitaires de Louvain. [find](https://scholar.google.com/scholar?q=Sylviane+Granger%2C+Estelle+Dagneaux%2C+Fanny+Meunier%2C+and+Magali+Paquot%2E+2009%2E+International+Corpus+of+Learner+English+%28Version+2%29%2E+Presses+universitaires+de+Louvain%2E)
14. Derrick Higgins, Jill Burstein, Daniel Marcu, and Claudia Gentile. 2004. Evaluating multiple aspects of coherence in student essays. In HLT-NAACL 2004: Main Proceedings, pages 185–192. [find](https://scholar.google.com/scholar?q=Derrick+Higgins%2C+Jill+Burstein%2C+Daniel+Marcu%2C+and+Claudia+Gentile%2E+2004%2E+Evaluating+multiple+aspects+of+coherence+in+student+essays%2E+In+HLT%2DNAACL+2004%3A+Main+Proceedings%2C+pages+185%E2%80%93192%2E)
15. Thorsten Joachims. 1999. Making large-scale SVM learning practical. In Bernhard Scholkopf and Alexander Smola, editors, Advances in Kernel Methods - Support Vector Learning, pages 44–56. MIT Press. [find](https://scholar.google.com/scholar?q=Thorsten+Joachims%2E+1999%2E+Making+large%2Dscale+SVM+learning+practical%2E+In+Bernhard+Scholkopf+and+Alexander+Smola%2C+editors%2C+Advances+in+Kernel+Methods+%2D+Support+Vector+Learning%2C+pages+44%E2%80%9356%2E+MIT+Press%2E)
16. Thomas K. Landauer, Darrell Laham, and Peter W. Foltz. 2003. Automated scoring and annotation of essays with the Intelligent Essay Assessor TM . In Automated Essay Scoring: A Cross-Disciplinary Perspective, pages 87–112. [find](https://scholar.google.com/scholar?q=Thomas+K%2E+Landauer%2C+Darrell+Laham%2C+and+Peter+W%2E+Foltz%2E+2003%2E+Automated+scoring+and+annotation+of+essays+with+the+Intelligent+Essay+Assessor+TM+%2E+In+Automated+Essay+Scoring%3A+A+Cross%2DDisciplinary+Perspe)
17. Huma Lodhi, Craig Saunders, John Shawe-Taylor, Nello Cristianini, and Christopher J. C. H. Watkins. 2002. Text classification using string kernels. Journal of Machine Learning Research, 2:419–444. [find](https://scholar.google.com/scholar?q=Huma+Lodhi%2C+Craig+Saunders%2C+John+Shawe%2DTaylor%2C+Nello+Cristianini%2C+and+Christopher+J%2E+C%2E+H%2E+Watkins%2E+2002%2E+Text+classification+using+string+kernels%2E+Journal+of+Machine+Learning+Research%2C+2%3A419%E2%80%93444%2E)
18. Eleni Miltsakaki and Karen Kukich. 2004. Evaluation of text coherence for electronic essay scoring systems. Natural Language Engineering, 10(1):25–55. [find](https://scholar.google.com/scholar?q=Eleni+Miltsakaki+and+Karen+Kukich%2E+2004%2E+Evaluation+of+text+coherence+for+electronic+essay+scoring+systems%2E+Natural+Language+Engineering%2C+10%281%29%3A25%E2%80%9355%2E)
19. Alessandro Moschitti. 2004. A study on convolution kernels for shallow statistic parsing. In Proceedings of the ACL, pages 335–342. [find](https://scholar.google.com/scholar?q=Alessandro+Moschitti%2E+2004%2E+A+study+on+convolution+kernels+for+shallow+statistic+parsing%2E+In+Proceedings+of+the+ACL%2C+pages+335%E2%80%93342%2E)
20. Saul Ben Needleman and Christian Dennis Wunsch. 1970. A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of Molecular Biology, 48(3):443–453, March. [find](https://scholar.google.com/scholar?q=Saul+Ben+Needleman+and+Christian+Dennis+Wunsch%2E+1970%2E+A+general+method+applicable+to+the+search+for+similarities+in+the+amino+acid+sequence+of+two+proteins%2E+Journal+of+Molecular+Biology%2C+48%283%29%3A443%E2%80%93453)
21. Mark Shermis and Jill Burstein. 2003. Automated Essay Scoring: A Cross-Disciplinary Perspective. Lawrence Erlbaum Associates, Inc., Mahwah, NJ. [find](https://scholar.google.com/scholar?q=Mark+Shermis+and+Jill+Burstein%2E+2003%2E+Automated+Essay+Scoring%3A+A+Cross%2DDisciplinary+Perspective%2E+Lawrence+Erlbaum+Associates%2C+Inc%2E%2C+Mahwah%2C+NJ%2E)
22. Mark Shermis, Jill Burstein, Derrick Higgins, and Klaus Zechner. 2010. Automated essay scoring: Writing assessment and instruction. In International Encyclopedia of Education (3rd edition), pages 20–26. [find](https://scholar.google.com/scholar?q=Mark+Shermis%2C+Jill+Burstein%2C+Derrick+Higgins%2C+and+Klaus+Zechner%2E+2010%2E+Automated+essay+scoring%3A+Writing+assessment+and+instruction%2E+In+International+Encyclopedia+of+Education+%283rd+edition%29%2C+pages+20%E2%80%932)
23. Tony Silva. 1993. Toward an understanding of the distinct nature of L2 writing: The ESL research and its implications. 27(4):657–677. [find](https://scholar.google.com/scholar?q=Tony+Silva%2E+1993%2E+Toward+an+understanding+of+the+distinct+nature+of+L2+writing%3A+The+ESL+research+and+its+implications%2E+27%284%29%3A657%E2%80%93677%2E)
24. Radu Soricut and Daniel Marcu. 2006. Discourse generation using utility-trained coherence models. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 803–810. [find](https://scholar.google.com/scholar?q=Radu+Soricut+and+Daniel+Marcu%2E+2006%2E+Discourse+generation+using+utility%2Dtrained+coherence+models%2E+In+Proceedings+of+the+COLING%2FACL+2006+Main+Conference+Poster+Sessions%2C+pages+803%E2%80%93810%2E)
25. Yannick Versley, Alessandro Moschitti, Massimo Poesio, and Xiaofeng Yang. 2008. [Coreference systems based on kernels methods](https://md.research.elamur.ai/free/en/papers/versley-2008-coreference-systems-based-kernels-methods/). In Proceedings of COL-ING, pages 961–968.
26. Yiming Yang and Jan O. Pedersen. 1997. A comparative study on feature selection in text categorization. In Proceedings of ICML, pages 412–420. [find](https://scholar.google.com/scholar?q=Yiming+Yang+and+Jan+O%2E+Pedersen%2E+1997%2E+A+comparative+study+on+feature+selection+in+text+categorization%2E+In+Proceedings+of+ICML%2C+pages+412%E2%80%93420%2E)

*Updated 24 October 2026*