---
title: "Automated Content Scoring of Spoken Responses in an"
description: "The paper addresses automatic scoring of content correctness in spoken answers on ETLA, a 2012… 12 references, cited by 2 papers here."
authors: ["Klaus Zechner", "Xinhao Wang"]
year: 2013
venue: "Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications"
source: "https://aclanthology.org/W13-1709/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zechner-2013-automated-content-scoring-spoken-responses/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zechner-2013-automated-content-scoring-spoken-responses/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2013](https://md.research.elamur.ai/free/en/papers/2013/page/3/) / Automated Content Scoring of Spoken Responses in an Assessment for Teachers of English

# Automated Content Scoring of Spoken Responses in an Assessment for Teachers of English

**Authors:** [Klaus Zechner](https://md.research.elamur.ai/free/en/authors/klaus-zechner/) and [Xinhao Wang](https://md.research.elamur.ai/free/en/authors/xinhao-wang/) · **Year:** 2013 · **Venue:** [Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications](https://md.research.elamur.ai/free/en/venues/bea/) ([BEA 2013](https://md.research.elamur.ai/free/en/venues/bea/2013/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W13-1709/) · [PDF](https://aclanthology.org/W13-1709.pdf) · [OpenAlex](https://openalex.org/W2251574625)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/zechner-2013-automated-content-scoring-spoken-responses/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses automatic scoring of content correctness in spoken answers on ETLA, a 2012 English test for non-native EFL teachers. Its target is items whose answers are only moderately predictable: short and tied to a classroom situation, but still varied. Using the word sequence produced by the ASR front end of SpeechRater, the authors built 15 content features in three families: flexible string matching (regular expressions, keywords), n-grams of words or POS tags (including BLEU), and string edit distance (including WER-style measures). They developed the features on a small in-house corpus of expert-written and read-aloud responses, kept seven of them, and then tested those on pilot data from 2,308 test takers. On recognizer output, individual features correlated with human holistic scores at roughly 0.29–0.48. Adding the features to a linear regression scoring model raised its agreement with human scores from 0.33 to 0.56. This matters because, for such brief answers, fluency and pronunciation cues carry little signal, so content features become the main basis for scoring.

## Key points
- In-house set: 22 items and 1,053 expert-drafted text responses at score levels 1–3. Of these, 789 were used for development, and 264 were recorded (some by two speakers), giving 444 spoken responses for evaluation.
- Pilot set: 14 moderately predictable items, split into asrTrain/asrDev/asrEval/smTrain/smEval with no speaker overlap.
  - Responses average 10.5 words.
  - Human inter-rater correlation is 0.73.
  - ASR word error rate is about 27%, compared with 18% in-house.
- The re_match feature takes three values: 0 for no match, 1 for a partial match, and 2 for an exact match. It was the strongest feature on transcripts in-house (r = 0.789) but lost about a third of its correlation on ASR output.
- Seven features were kept: three from string matching, three from edit distance, and bleu_3. Their correlations were 0.42–0.60 on transcripts and 0.29–0.48 on ASR output. Fluency and pronunciation features stayed below 0.3 on this data.
- Every selected feature correlated slightly better with human analytic content scores than with holistic scores (by 0.01–0.04).

## Interesting ideas
- A pattern-match feature with a middle "partial" level gives credit when the correct answer is embedded in extra material, such as repetitions or self-corrections, instead of failing the whole response
- Features were chosen on three criteria together: predictive strength, low redundancy with other features, and fit with what experts consider content. This avoids piling up correlated signals that measure the same thing
- Each feature was measured twice, on clean human transcripts and on noisy ASR output, and the relative drop was reported. This exposes which features are brittle under realistic input
- The authors checked that a feature tracks the intended construct by showing it aligns better with a narrow content score than with a broad holistic score
- POS-tag n-grams served as a cheap proxy for grammatical well-formedness when comparing against reference answers, alongside BLEU and edit distance

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (answer): student-answer checking in school-course tests could distinguish an exact answer, a correct answer surrounded by noise, and no answer, rather than using a binary match.
- **I2** (learn): when choosing evaluation metrics or repair-rule triggers, drop near-duplicate signals and keep ones a human would accept as meaningful.
- **I3** (tree): compare parser performance on gold-tokenized text against noisy or automatically tokenized text and report the relative degradation per component.
- **I4** (learn): this works as a negative-control style check, confirming that a frame or world metric correlates more with its targeted dimension than with an unrelated overall quality judgment.
- **I5** (back): generated text from the meaning graph could be compared with references at the POS-sequence level as well as the word level, separating grammatical errors from lexical ones.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Educational Testing Service](https://md.research.elamur.ai/free/en/ngos/educational-testing-service/) — Nonprofit, Princeton, United States · 27 works · 2,212 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (2): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,379 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 519 works

## Cite

Cite as: Klaus Zechner and Xinhao Wang. 2013. Automated Content Scoring of Spoken Responses in an Assessment for Teachers of English. Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zechner2013automated,
  title = {{Automated Content Scoring of Spoken Responses in an Assessment for Teachers of English}},
  author = {Klaus Zechner and Xinhao Wang},
  booktitle = {Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications},
  year = {2013},
  url = {https://aclanthology.org/W13-1709/}
}
```

- [← previous Ubertagging: Joint Segmentation and Supertagging for English](https://md.research.elamur.ai/free/en/papers/dridan-2013-ubertagging-joint-segmentation-supertagging-english/)
- [next → DIRA: Dialectal Arabic Information Retrieval Assistant](https://md.research.elamur.ai/free/en/papers/pasha-2013-dira-dialectal-arabic-information-retrieval/)
- [↑ back to papers of 2013](https://md.research.elamur.ai/free/en/papers/2013/page/3/)

## References

12 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abeer Alwan, Yijian Bai, Matt Black, Larry Casey, Matteo Gerosa, Margaret Heritage, Markus Iseli, Barbara Jones, Abe Kazemzadeh, Sungbok Lee, Shrikanth Narayanan, Patti Price, Joseph Tepperman and Shizhen Wang. 2007. A system for technology based assessment of language and literacy in young children: the role of multiple information sources. Proceedings of IEEE International Workshop on Multimedia Signal Processing, 26-30. [find](https://scholar.google.com/scholar?q=Abeer+Alwan%2C+Yijian+Bai%2C+Matt+Black%2C+Larry+Casey%2C+Matteo+Gerosa%2C+Margaret+Heritage%2C+Markus+Iseli%2C+Barbara+Jones%2C+Abe+Kazemzadeh%2C+Sungbok+Lee%2C+Shrikanth+Narayanan%2C+Patti+Price%2C+Joseph+Tepperman+and+Shi)
2. Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater® V.2.0. Journal of Technology, Learning, and Assessment, 4(3): 159-174. [find](https://scholar.google.com/scholar?q=Yigal+Attali+and+Jill+Burstein%2E+2006%2E+Automated+essay+scoring+with+e%2Drater%C2%AE+V%2E2%2E0%2E+Journal+of+Technology%2C+Learning%2C+and+Assessment%2C+4%283%29%3A+159%2D174%2E)
3. Miao Chen and Klaus Zechner. 2011. Computing and evaluating syntactic complexity features for automated scoring of spontaneous non-native speech. Proceedings of ACL, 722-731. [find](https://scholar.google.com/scholar?q=Miao+Chen+and+Klaus+Zechner%2E+2011%2E+Computing+and+evaluating+syntactic+complexity+features+for+automated+scoring+of+spontaneous+non%2Dnative+speech%2E+Proceedings+of+ACL%2C+722%2D731%2E)
4. Peter W. Foltz, Darrell Laham and Thomas K. Landauer. 1999. The intelligent essay assessor: Applications to educational technology. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning, 1(2). [find](https://scholar.google.com/scholar?q=Peter+W%2E+Foltz%2C+Darrell+Laham+and+Thomas+K%2E+Landauer%2E+1999%2E+The+intelligent+essay+assessor%3A+Applications+to+educational+technology%2E+Interactive+Multimedia+Electronic+Journal+of+Computer%2DEnhanced+Learn)
5. Horacio Franco, Harry Bratt, Romain Rossier, Venkata Rao Gadde, Elizabeth Shriberg, Victor Abrash and Kristin Precoda. 2010. EduSpeak®: A speech recognition and pronunciation scoring toolkit for computer-aided language learning applications. Language Testing, 27(3): 401-418. [find](https://scholar.google.com/scholar?q=Horacio+Franco%2C+Harry+Bratt%2C+Romain+Rossier%2C+Venkata+Rao+Gadde%2C+Elizabeth+Shriberg%2C+Victor+Abrash+and+Kristin+Precoda%2E+2010%2E+EduSpeak%C2%AE%3A+A+speech+recognition+and+pronunciation+scoring+toolkit+for+compu)
6. Dharmendra Kanejiya, Arun Kumary and Surendra Prasad. 2003. Automatic evaluation of students' answers using syntactically enhanced LSA. Proceedings of Workshop on Building Educational Applications Using Natural Language Processing, 53-60. [find](https://scholar.google.com/scholar?q=Dharmendra+Kanejiya%2C+Arun+Kumary+and+Surendra+Prasad%2E+2003%2E+Automatic+evaluation+of+students%27+answers+using+syntactically+enhanced+LSA%2E+Proceedings+of+Workshop+on+Building+Educational+Applications+Usi)
7. Claudia Leacock and Martin Chodorow. 2003. C-rater: Automated scoring of short-answer questions. Computers and Humanities,37: 389–405. [find](https://scholar.google.com/scholar?q=Claudia+Leacock+and+Martin+Chodorow%2E+2003%2E+C%2Drater%3A+Automated+scoring+of+short%2Danswer+questions%2E+Computers+and+Humanities%2C37%3A+389%E2%80%93405%2E)
8. Tom Mitchell, Terry Russell, Peter Broomhead and Nicola Aldridge. 2002. Towards robust computerised marking of free-text responses. Proceedings of International Computer Assisted Assessment Conference, 233-249. [find](https://scholar.google.com/scholar?q=Tom+Mitchell%2C+Terry+Russell%2C+Peter+Broomhead+and+Nicola+Aldridge%2E+2002%2E+Towards+robust+computerised+marking+of+free%2Dtext+responses%2E+Proceedings+of+International+Computer+Assisted+Assessment+Conference)
9. Michael Mohler and Rada Mihalcea. 2009. Text-to-text semantic similarity for automatic short answer grading. Proceedings of EACL, 567-575. [find](https://scholar.google.com/scholar?q=Michael+Mohler+and+Rada+Mihalcea%2E+2009%2E+Text%2Dto%2Dtext+semantic+similarity+for+automatic+short+answer+grading%2E+Proceedings+of+EACL%2C+567%2D575%2E)
10. Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu. 2002. [BLEU: A method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). Proceedings of ACL, 311-318.
11. Shasha Xie, Keelan Evanini and Klaus Zechner. 2012. Exploring content features for automated speech scoring. Proceedings of NAACL-HLT, 103-111. [find](https://scholar.google.com/scholar?q=Shasha+Xie%2C+Keelan+Evanini+and+Klaus+Zechner%2E+2012%2E+Exploring+content+features+for+automated+speech+scoring%2E+Proceedings+of+NAACL%2DHLT%2C+103%2D111%2E)
12. Klaus Zechner, Derrick Higgins, Xiaoming Xi and David M. Williamson. 2009. Automatic scoring of non-mative spontaneous speech in tests of spoken English. Speech Communication, 51: 883-895. [find](https://scholar.google.com/scholar?q=Klaus+Zechner%2C+Derrick+Higgins%2C+Xiaoming+Xi+and+David+M%2E+Williamson%2E+2009%2E+Automatic+scoring+of+non%2Dmative+spontaneous+speech+in+tests+of+spoken+English%2E+Speech+Communication%2C+51%3A+883%2D895%2E)

*Updated 2 November 2026*