---
title: "Automated Essay Scoring for Nonnative English Speakers (1999)"
description: "The paper tests whether e-rater, ETS's operational essay scorer, still works when the writers are not… 12 references, cited by 20 papers here."
authors: ["Jill Burstein", "Martin Chodorow"]
year: 1999
venue: "Computer Mediated Language Assessment and Evaluation in Natural Language Processing"
source: "https://aclanthology.org/W99-0411/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/burstein-1999-automated-essay-scoring-nonnative-english/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/burstein-1999-automated-essay-scoring-nonnative-english/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [1999](https://md.research.elamur.ai/free/en/papers/1999/) / Automated Essay Scoring for Nonnative English Speakers

# Automated Essay Scoring for Nonnative English Speakers

**Authors:** [Jill Burstein](https://md.research.elamur.ai/free/en/authors/jill-burstein/) and [Martin Chodorow](https://md.research.elamur.ai/free/en/authors/martin-chodorow/) · **Year:** 1999 · **Venue:** Computer Mediated Language Assessment and Evaluation in Natural Language Processing · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W99-0411/) · [PDF](https://aclanthology.org/W99-0411.pdf) · [OpenAlex](https://openalex.org/W2016195898)

**Citations:** 113 citations (OpenAlex, 2026-10-14) · [20](https://md.research.elamur.ai/free/en/papers/burstein-1999-automated-essay-scoring-nonnative-english/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper tests whether e-rater, ETS's operational essay scorer, still works when the writers are not native English speakers. e-rater builds a separate model for each prompt. It extracts about 52 syntactic, discourse and topical variables from 270 human-scored training essays, and a stepwise linear regression keeps the predictive subset. Syntax comes from an ETS-enhanced CASS chunker. Discourse structure comes from a cue-term lexicon based on Quirk et al. plus positional heuristics. Topic is measured with cosine-based content vector analysis at both the essay level and the argument level. Length measures are deliberately left out.

The authors applied this to two Test of Written English prompts (TWE1, TWE2). Writers were native speakers of Chinese, Arabic or Spanish, plus two small English-speaking groups. On roughly 1,100 cross-validation essays, exact-or-adjacent agreement with the final human score averaged 92.3%, matching operational native-speaker performance. Mean scores differed slightly but significantly by language group, yet agreement did not. The regression chose largely the same features as in the native-speaker models. This suggests the feature set holds up across considerable linguistic variation.

## Key points
- **Operational baseline:** e-rater matches a human reader exactly or adjacently about 92% of the time on a 6-point scale. Its correlation with a single reader is about .73, against .75 between two humans.
- **TWE results:** agreement was 91.1% on TWE1 (n=562) and 93.4% on TWE2 (n=576), with Pearson r of .667 and .718. Always predicting a "4" gives a majority baseline of about 84%.
- **Language-group effects:** group had a significant main effect on scores, and the English groups scored much higher. A Language Group × Reader interaction showed e-rater under-scoring Spanish speakers and over-scoring Chinese speakers relative to humans. χ² tests found no agreement differences across groups. The one exception was a prompt effect for Arabic speakers.
- **Model contents:** TWE models had 7–8 features. At least 4 of the 5 features most common across the 75 operational models reappeared. These were the argument-level and essay-level topical scores, the subjunctive auxiliary count and ratio, and the count of argument development terms.
- **Topical scoring:** essay-level scoring takes a cosine-weighted mean over the 6 nearest training essays. Argument-level scoring compares each argument against 6 per-score "supervectors". It then combines them as (Σ argument scores + number of arguments) / (number of arguments + 1), which nudges essays with more arguments upward.

## Interesting ideas
- **Position-gated discourse markers:** a cue word such as "first" counts as a discourse marker only when it acts as a sentence-level adverbial, not when it modifies a noun. This uses syntax to filter a lexicon instead of trusting surface matches.
- **Rubric-only features:** length was excluded even though it correlates strongly with score, because human graders are not told to use it. This keeps the model's reasons aligned with the actual criteria.
- **Majority-class baseline next to agreement:** reporting that a constant "4" already gets about 84% puts the 92% figure in perspective.
- **Feature-selection stability as a generalization test:** checking whether the same features get selected on a very different writer population is evidence of transfer beyond a single accuracy number.
- **Segment-level scoring after discourse partitioning:** the essay is split into arguments using detected discourse markers, each argument is scored separately, and the results are pooled. This turns a global judgment into explainable local ones.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (frames): connective detection in prag could require the cue to attach as advmod/discourse to the clause root rather than as amod inside a noun phrase, and record that attachment as the explanation.
- **I2** (answer): student-answer graders should score only rubric-backed properties and reject easy proxies like answer length, so every grade can be justified.
- **I3** (learn): every accuracy report should include a trivial constant-output control, so gains are measured against what doing nothing achieves.
- **I4** (learn): when retraining on silver data from new genres or dialects, compare which features or rules get chosen; large shifts flag domain-specific brittleness.
- **I5** (world): fable and story texts could be segmented at discourse connectives into episodes, each assessed or summarized on its own before building the whole-text world state.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Educational Testing Service](https://md.research.elamur.ai/free/en/ngos/educational-testing-service/) — Nonprofit, Princeton, United States · 27 works · 2,212 citations
- [Hunter College](https://md.research.elamur.ai/free/en/universities/hunter-college/) — University, New York, United States · 9 works · 1,245 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (2): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works · [🖍️ Grammatical error correction & learner language](https://md.research.elamur.ai/free/en/topics/grammatical-error-correction/) · 156 works

## Cite

Cite as: Jill Burstein and Martin Chodorow. 1999. Automated Essay Scoring for Nonnative English Speakers. Computer Mediated Language Assessment and Evaluation in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{burstein1999automated,
  title = {{Automated Essay Scoring for Nonnative English Speakers}},
  author = {Jill Burstein and Martin Chodorow},
  booktitle = {Computer Mediated Language Assessment and Evaluation in Natural Language Processing},
  year = {1999},
  doi = {10.3115/1598834.1598847},
  url = {https://doi.org/10.3115/1598834.1598847}
}
```

- [← previous Statistical Properties of Probabilistic Context-Free Grammars](https://md.research.elamur.ai/free/en/papers/chi-1999-statistical-properties-probabilistic-context-free/)
- [next → Corpus Annotation for Parser Evaluation](https://md.research.elamur.ai/free/en/papers/carroll-1999-corpus-annotation-parser-evaluation/)
- [↑ back to papers of 1999](https://md.research.elamur.ai/free/en/papers/1999/)

## References

12 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abney, Steven. (1996) Part-of-speech tagging and partial parsing. In Church, Young and Bloothooft (eds), Corpus-based Methods in Language and Speech. Dordrecht: Kluwer. [↗](https://doi.org/10.1007/978-94-017-1183-8_4)
2. Biber, D. (1993) Using register-diversified corpora for general language studies. Computational Linguistics, 19, 219-241. [find](https://scholar.google.com/scholar?q=Biber%2C+D%2E+%281993%29+Using+register%2Ddiversified+corpora+for+general+language+studies%2E+Computational+Linguistics%2C+19%2C+219%2D241%2E)
3. Breland, Bonner and Kubota (1995), Factors in Performance on Brief, Impromptu Essay Examinations, College Board Report No. 95 -4. [find](https://scholar.google.com/scholar?q=Breland%2C+Bonner+and+Kubota+%281995%29%2C+Factors+in+Performance+on+Brief%2C+Impromptu+Essay+Examinations%2C+College+Board+Report+No%2E+95+%2D4%2E)
4. Bridgeman and McHale (1996). Gender and Ethnic Group Differences on the GMAT Analytical Writing Assessment. ETS RR-96-2. [↗](https://doi.org/10.1002/j.2333-8504.1996.tb01680.x)
5. Burstein, Jill, Karen Kukich, Susanne Wolff, Chi Lu, Martin Chodorow, Lisa Braden-Harder, and Mary Dee Harris (1998). Automated Scoring Using A Hybrid Feature Identification Technique. In the Proceedings of the Annual Meeting of the AsSociation of Computational Linguistics, August, 1998. Montreal, Canada. [find](https://scholar.google.com/scholar?q=Burstein%2C+Jill%2C+Karen+Kukich%2C+Susanne+Wolff%2C+Chi+Lu%2C+Martin+Chodorow%2C+Lisa+Braden%2DHarder%2C+and+Mary+Dee+Harris+%281998%29%2E+Automated+Scoring+Using+A+Hybrid+Feature+Identification+Technique%2E+In+the+Proceedi)
6. Burstein, Jill, Karen Kukich, Susanne Wolff, Chi Lu, & Martin Chodorow (1998). Enriching Automated Scoring Using Discourse Marking. In the Proceedings of the Workshop on Discourse Relations & Discourse Marking, Annual Meeting of the Association of Computational Linguistics, August, 1998. Montreal, Canada. [find](https://scholar.google.com/scholar?q=Burstein%2C+Jill%2C+Karen+Kukich%2C+Susanne+Wolff%2C+Chi+Lu%2C+%26+Martin+Chodorow+%281998%29%2E+Enriching+Automated+Scoring+Using+Discourse+Marking%2E+In+the+Proceedings+of+the+Workshop+on+Discourse+Relations+%26+Discours)
7. Burstein, Jill C., Lisa Braden-Harder, Martin Chodorow, Shuyi Hua, Bruce Kaplan, Karen Kukich, Chi Lu, James Nolan, Don Rock and Susanne Wolff (1998). Computer Analysis of Essay Content for Automated Score Prediction. ETS RR 98-15. [find](https://scholar.google.com/scholar?q=Burstein%2C+Jill+C%2E%2C+Lisa+Braden%2DHarder%2C+Martin+Chodorow%2C+Shuyi+Hua%2C+Bruce+Kaplan%2C+Karen+Kukich%2C+Chi+Lu%2C+James+Nolan%2C+Don+Rock+and+Susanne+Wolff+%281998%29%2E+Computer+Analysis+of+Essay+Content+for+Automated+)
8. Foltz, P. W., W. Kintsch, and T. K. Landauer. (1998). The Measurement of Textual Coherence with Latent Semantic Analysis. Discourse Processes, 25(2&3), 285-307. [↗](https://doi.org/10.1080/01638539809545029)
9. Frase, L. T., Faletti, J., Ginther, A., & Grant, L. (1997). Computer Analysis of the TOEFL Test of Written English (TWE). Educational Testing Service, Princeton, NJ. [↗](https://doi.org/10.1002/j.2333-8504.1998.tb01791.x)
10. Larkey, L. (1998). Automatic Essay Grading Using Text Categorization Techniques. Proceedings of the 21 st A CM-SIGIR Conference on Research and Development in Information Australia, 90-95. [find](https://scholar.google.com/scholar?q=Larkey%2C+L%2E+%281998%29%2E+Automatic+Essay+Grading+Using+Text+Categorization+Techniques%2E+Proceedings+of+the+21+st+A+CM%2DSIGIR+Conference+on+Research+and+Development+in+Information+Australia%2C+90%2D95%2E)
11. Retrieval, Melbourne, Page, E. B. and N. Petersen. (1995). The computer moves into essay grading: updating the ancient test. Phi Delta Kappan. March, 561-565. [find](https://scholar.google.com/scholar?q=Retrieval%2C+Melbourne%2C+Page%2C+E%2E+B%2E+and+N%2E+Petersen%2E+%281995%29%2E+The+computer+moves+into+essay+grading%3A+updating+the+ancient+test%2E+Phi+Delta+Kappan%2E+March%2C+561%2D565%2E)
12. Sheehan, K (1997). A Tree-Based Approach to Proficiency Scaling and Diagnostic Assessment. Journal of Educational Measurement. 34(4), 333-352. Acknowledgements [↗](https://doi.org/10.1111/j.1745-3984.1997.tb00522.x)

*Updated 1 November 2026*