---
title: "Automated scoring of speaking items in an assessment for"
description: "The paper describes an automatic scorer for the speaking part of a new English test designed for… 27 references, cited by 1 paper here."
authors: ["Klaus Zechner", "Keelan Evanini", "Su-Youn Yoon", "Lawrence Davis", "Xinhao Wang", "Lei Chen", "Chong Min Lee", "Chee Wee Leong"]
year: 2014
venue: "Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications"
source: "https://aclanthology.org/W14-1816/"
doi: "10.3115/v1/w14-1816"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zechner-2014-automated-scoring-speaking-items-assessment/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zechner-2014-automated-scoring-speaking-items-assessment/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/page/3/) / Automated scoring of speaking items in an assessment for teachers of English as a Foreign Language

# Automated scoring of speaking items in an assessment for teachers of English as a Foreign Language

**Authors:** [Klaus Zechner](https://md.research.elamur.ai/free/en/authors/klaus-zechner/), [Keelan Evanini](https://md.research.elamur.ai/free/en/authors/keelan-evanini/), [Su-Youn Yoon](https://md.research.elamur.ai/free/en/authors/su-youn-yoon/), Lawrence Davis, [Xinhao Wang](https://md.research.elamur.ai/free/en/authors/xinhao-wang/), [Lei Chen](https://md.research.elamur.ai/free/en/authors/lei-chen/), Chong Min Lee and [Chee Wee Leong](https://md.research.elamur.ai/free/en/authors/chee-wee-ben-leong/) · **Year:** 2014 · **Venue:** [Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications](https://md.research.elamur.ai/free/en/venues/bea/) ([BEA 2014](https://md.research.elamur.ai/free/en/venues/bea/2014/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W14-1816/) · [PDF](https://aclanthology.org/W14-1816.pdf) · [DOI 10.3115/v1/w14-1816](https://doi.org/10.3115/v1/w14-1816) · [OpenAlex](https://openalex.org/W2250628466)

**Citations:** 15 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/zechner-2014-automated-scoring-speaking-items-assessment/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper describes an automatic scorer for the speaking part of a new English test designed for non-native English speakers who teach EFL. Its 21 items fall into 8 task types. Some are restricted, such as reading aloud or repeating. Others are semi-restricted: the test taker builds a sentence from given words, charts or templates, so many correct answers exist but they cannot be fully listed. The authors built a pipeline on ETS SpeechRater:
- a speech recognizer adapted to accented non-native speech;
- over 100 features covering fluency, pronunciation, prosody, vocabulary, grammar and content;
- a filter that sets aside responses that cannot be scored, such as noisy or non-English ones;
- a separate linear regression model for each task type, each using about ten selected features.

Trained on responses from roughly 1,400 test takers, the system reaches a correlation of 0.73 with expert scores when all 21 item scores of a speaker are summed. Content-related features were the strongest predictors. This matters because semi-restricted spoken answers, which sit between scripted reading and free speech, had rarely been scored automatically.

## Key points
- Item types: restricted (MC, RA, RP) and semi-restricted (IS, KW, CH, KC, VI). Each test taker gave 21 responses, about 9 minutes of audio, double-rated by humans on a 1–3 scale.
- At speaker level (N=272), system–rater correlations were 0.725 and 0.742, against 0.934 between the two human raters. Speakers with more than 7 unscorable responses (28 people) were excluded.
- At item-type level, system–human r ranged from 0.34 (RA) to 0.67 (MC), while human agreement ranged from 0.51 to 0.83. The gap between them was 0.16–0.37.
- Content features correlated with human scores at 0.33–0.67, and grammar features reached up to 0.49. Fluency, pronunciation, prosody and vocabulary features stayed near 0.2, which the authors attribute to very short answers (about 6–14 words).
- More than 7% of responses were unscorable. Recognition word error rate did not clearly predict how well a scoring model performed.

## Interesting ideas
- When the set of valid answers is open but strongly patterned, score content by matching against hand-written regular expressions and by n-gram overlap with answers that humans rated highly, rather than relying on one gold string.
- Train a separate small model for each task type with its own feature subset, instead of one global model, because what signals quality differs between task types.
- Put an explicit "cannot judge" filter before scoring, so that garbage input is flagged instead of receiving a confident wrong score.
- Always report machine–human agreement next to the human–human ceiling, and compare results at both item level and aggregated level, since aggregation hides much item-level noise.
- Prefer features that tolerate errors from the upstream recognizer, because even small errors early in the pipeline can distort content judgments.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (answer): School-test readers could grade short free answers by matching against patterns and high-scoring exemplars, and report which pattern fired as the explanation.
- **I2** (learn): Evaluation and repair-rule induction could be split by construction or question type, so a weakness in one type is not averaged away by the others.
- **I3** (answer): Before answering, a reader could detect inputs it cannot ground, such as unparsable or off-topic questions, and abstain with a stated reason.
- **I4** (learn): Witness-parser agreement could serve as the ceiling for each metric, and accuracy could be reported per item and per document to show where the noise really sits.
- **I5** (frames): Frame and role extraction could be designed to survive local parse errors, for example by checking role fillers against several candidate attachments from the tree.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Educational Testing Service](https://md.research.elamur.ai/free/en/ngos/educational-testing-service/) — Nonprofit, Princeton, United States · 27 works · 2,212 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works

## Cite

Cite as: Klaus Zechner, Keelan Evanini, Su-Youn Yoon, Lawrence Davis, Xinhao Wang, Lei Chen, Chong Min Lee and Chee Wee Leong. 2014. Automated scoring of speaking items in an assessment for teachers of English as a Foreign Language. Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zechner2014automated,
  title = {{Automated scoring of speaking items in an assessment for teachers of English as a Foreign Language}},
  author = {Klaus Zechner, Keelan Evanini, Su-Youn Yoon, Lawrence Davis, Xinhao Wang, Lei Chen, Chong Min Lee and Chee Wee Leong},
  booktitle = {Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications},
  year = {2014},
  doi = {10.3115/v1/w14-1816},
  url = {https://doi.org/10.3115/v1/w14-1816}
}
```

- [← previous Monotheism and the Language of Divine Plurality in the Hebrew Bible and the Dead Sea Scrolls](https://md.research.elamur.ai/free/en/papers/heiser-2014-monotheism-language-divine-plurality-hebrew/)
- [next → Combining Natural and Artificial Examples to Improve Implicit Discourse Relation Identification](https://md.research.elamur.ai/free/en/papers/braud-2014-combining-natural-artificial-examples-improve/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/page/3/)

## References

27 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jared Bernstein, Jian Cheng, and Masanori Suzuki. 2010a. Fluency and structural complexity as predictors of L2 oral proficiency. In Proceedings of Interspeech. [find](https://scholar.google.com/scholar?q=Jared+Bernstein%2C+Jian+Cheng%2C+and+Masanori+Suzuki%2E+2010a%2E+Fluency+and+structural+complexity+as+predictors+of+L2+oral+proficiency%2E+In+Proceedings+of+Interspeech%2E)
2. Jared Bernstein, Alistair Van Moere, and Jian Cheng. 2010b. Validating automated speaking tests. Language Testing, 27(3):355–377. [↗](https://doi.org/10.1177/0265532210364404)
3. Paul Boersma and David Weenick. 2012. Praat: Doing phonetics by computer, version 5.3.32. [http://www.praat.org](http://www.praat.org). [↗](http://www.praat.org)
4. Abhishek Chandel, Abhinav Parate, Maymon Ma-dathingal, Himanshu Pant, Nitendra Rajput, Shajith Ikbal, Om Deshmuck, and Ashish Verma. 2007. Sensei: Spoken language assessment for call center agents. In Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU). [find](https://scholar.google.com/scholar?q=Abhishek+Chandel%2C+Abhinav+Parate%2C+Maymon+Ma%2Ddathingal%2C+Himanshu+Pant%2C+Nitendra+Rajput%2C+Shajith+Ikbal%2C+Om+Deshmuck%2C+and+Ashish+Verma%2E+2007%2E+Sensei%3A+Spoken+language+assessment+for+call+center+agents%2E+In)
5. Lei Chen and Su-Youn Yoon. 2012. Application of structural events detected on ASR outputs for automated speaking assessment. In Proceedings of Interspeech. [find](https://scholar.google.com/scholar?q=Lei+Chen+and+Su%2DYoun+Yoon%2E+2012%2E+Application+of+structural+events+detected+on+ASR+outputs+for+automated+speaking+assessment%2E+In+Proceedings+of+Interspeech%2E)
6. Miao Chen and Klaus Zechner. 2011. Computing and evaluating syntactic complexity features for automated scoring of spontaneous non-native speech. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 722– 731. [find](https://scholar.google.com/scholar?q=Miao+Chen+and+Klaus+Zechner%2E+2011%2E+Computing+and+evaluating+syntactic+complexity+features+for+automated+scoring+of+spontaneous+non%2Dnative+speech%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Assoc)
7. Lei Chen, Klaus Zechner, and Xiaoming Xi. 2009. Improved pronunciation features for construct-driven assessment of non-native spontaneous speech. In Proceedings of NAACL-HLT. [find](https://scholar.google.com/scholar?q=Lei+Chen%2C+Klaus+Zechner%2C+and+Xiaoming+Xi%2E+2009%2E+Improved+pronunciation+features+for+construct%2Ddriven+assessment+of+non%2Dnative+spontaneous+speech%2E+In+Proceedings+of+NAACL%2DHLT%2E)
8. Catia Cucchiarini, Helmer Strik, and Lou Boves. 1997. Automatic evaluation of Dutch pronunciation by using speech recognition technology. In Proceedings of the IEEE Workshop on Auotmatic Speech Recognition and Understanding (ASRU). [find](https://scholar.google.com/scholar?q=Catia+Cucchiarini%2C+Helmer+Strik%2C+and+Lou+Boves%2E+1997%2E+Automatic+evaluation+of+Dutch+pronunciation+by+using+speech+recognition+technology%2E+In+Proceedings+of+the+IEEE+Workshop+on+Auotmatic+Speech+Recogn)
9. Catia Cucchiarini, Helmer Strik, and Lou Boves. 2000. Quantitative assessment of second language learners’ fluency by means of automatic speech recognition technology. Journal of the Acoustical Society of America, 107(2):989–999. [find](https://scholar.google.com/scholar?q=Catia+Cucchiarini%2C+Helmer+Strik%2C+and+Lou+Boves%2E+2000%2E+Quantitative+assessment+of+second+language+learners%E2%80%99+fluency+by+means+of+automatic+speech+recognition+technology%2E+Journal+of+the+Acoustical+Societ)
10. Catia Cucchiarini, Helmer Strik, and Lou Boves. 2002. Quantitative assessment of second language learners’ fluency: Comparisons between read and spontaneous speech. Journal of the Acoustical Society of America, 111(6):2862–2873. [find](https://scholar.google.com/scholar?q=Catia+Cucchiarini%2C+Helmer+Strik%2C+and+Lou+Boves%2E+2002%2E+Quantitative+assessment+of+second+language+learners%E2%80%99+fluency%3A+Comparisons+between+read+and+spontaneous+speech%2E+Journal+of+the+Acoustical+Society+o)
11. Maxine Eskenazi. 2009. An overview of spoken language technology for education. Speech Communication, 51(10):832–844. [↗](https://doi.org/10.1016/j.specom.2009.04.005)
12. Horacio Franco, Leonardo Neumeyer, Vassilios Digalakis, and Orith Ronen. 2000. Combination of machine scores for automatic grading of pronunciation quality. Speech Communication, 30(1-2):121– 130. [↗](https://doi.org/10.1016/s0167-6393(99)00045-x)
13. Derrick Higgins, Xiaoming Xi, Klaus Zechner, and David M. Williamson. 2011. A three-stage approach to the automated scoring of spontaneous spoken responses. Computer Speech and Language, 25(2):282–306. [↗](https://doi.org/10.1016/j.csl.2010.06.001)
14. Florian Hoenig. 2002. Automatic assessment of non-native prosody – Annotation, modelling, and evaluation. In Proceedings of the International Symposium on Automatic Detection of Errors in Pronunciation Training (ISADEPT), pages 21–30, Stockholm, Sweden. [find](https://scholar.google.com/scholar?q=Florian+Hoenig%2E+2002%2E+Automatic+assessment+of+non%2Dnative+prosody+%E2%80%93+Annotation%2C+modelling%2C+and+evaluation%2E+In+Proceedings+of+the+International+Symposium+on+Automatic+Detection+of+Errors+in+Pronunciatio)
15. Je Hun Jeon and Su-Youn Yoon. 2012. Acoustic feature-based non-scorable response detection for an automated speaking proficiency assessment. In Proceedings of Interspeech. [find](https://scholar.google.com/scholar?q=Je+Hun+Jeon+and+Su%2DYoun+Yoon%2E+2012%2E+Acoustic+feature%2Dbased+non%2Dscorable+response+detection+for+an+automated+speaking+proficiency+assessment%2E+In+Proceedings+of+Interspeech%2E)
16. Jack Mostow, Steven F. Roth, Alexander G. Hauptmann, and Matthew Kane. 1994. A prototype reading coach that listens. In Proceedings of the Twelfth National Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Jack+Mostow%2C+Steven+F%2E+Roth%2C+Alexander+G%2E+Hauptmann%2C+and+Matthew+Kane%2E+1994%2E+A+prototype+reading+coach+that+listens%2E+In+Proceedings+of+the+Twelfth+National+Conference+on+Artificial+Intelligence%2E)
17. Leonardo Neumeyer, Horacio Franco, Vassilios Digalakis, and Mitchel Weintraub. 2000. Automatic scoring of pronunciation quality. Speech Communication, 30:83–93. [↗](https://doi.org/10.1016/s0167-6393(99)00046-1)
18. Pearson Education, Inc. 2011. Versant TM Aviation English Test. [http://www.versanttest.com/technology/](http://www.versanttest.com/technology/) VersantAviationEnglishTestValidation. pdf. [↗](http://www.versanttest.com/technology/)
19. Silke Witt. 1999. Use of speech recognition in computer-assisted language learning. Ph.D. thesis, Cambridge University. [find](https://scholar.google.com/scholar?q=Silke+Witt%2E+1999%2E+Use+of+speech+recognition+in+computer%2Dassisted+language+learning%2E+Ph%2ED%2E+thesis%2C+Cambridge+University%2E)
20. Xiaoming Xi, Derrick Higgins, Klaus Zechner, and David M. Williamson. 2008. Automated scoring of spontaneous speech using SpeechRater v1.0. Educational Testing Service Research Report RR-08-62. [find](https://scholar.google.com/scholar?q=Xiaoming+Xi%2C+Derrick+Higgins%2C+Klaus+Zechner%2C+and+David+M%2E+Williamson%2E+2008%2E+Automated+scoring+of+spontaneous+speech+using+SpeechRater+v1%2E0%2E+Educational+Testing+Service+Research+Report+RR%2D08%2D62%2E)
21. Shasha Xie, Keelan Evanini, and Klaus Zechner. 2012. Exploring content features for automated speech scoring. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 103–111, Montréal, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shasha+Xie%2C+Keelan+Evanini%2C+and+Klaus+Zechner%2E+2012%2E+Exploring+content+features+for+automated+speech+scoring%2E+In+Proceedings+of+the+2012+Conference+of+the+North+American+Chapter+of+the+Association+for)
22. Su-Youn Yoon and Suma Bhat. 2012. Assessment of ESL learners’ syntactic competence based on similarity measures. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 600–608, Jeju Island, Korea. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Su%2DYoun+Yoon+and+Suma+Bhat%2E+2012%2E+Assessment+of+ESL+learners%E2%80%99+syntactic+competence+based+on+similarity+measures%2E+In+Proceedings+of+the+2012+Joint+Conference+on+Empirical+Methods+in+Natural+Language+Pr)
23. Su-Youn Yoon, Keelan Evanini, and Klaus Zechner. 2011. Non-scorable response detection for automated speaking proficiency assessment. In Proceedings of NAACL-HLT Workshop on Innovative Use of NLP for Building Educational Applications. [find](https://scholar.google.com/scholar?q=Su%2DYoun+Yoon%2C+Keelan+Evanini%2C+and+Klaus+Zechner%2E+2011%2E+Non%2Dscorable+response+detection+for+automated+speaking+proficiency+assessment%2E+In+Proceedings+of+NAACL%2DHLT+Workshop+on+Innovative+Use+of+NLP+for+)
24. Su-Youn Yoon, Suma Bhat, and Klaus Zechner. 2012. Vocabulary profile as a measure of vocabulary sophistication. In Proceedings of the 7th Workshop on Innovative Use of NLP for Building Educational Applications, NAACL-HLT, Montréal, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Su%2DYoun+Yoon%2C+Suma+Bhat%2C+and+Klaus+Zechner%2E+2012%2E+Vocabulary+profile+as+a+measure+of+vocabulary+sophistication%2E+In+Proceedings+of+the+7th+Workshop+on+Innovative+Use+of+NLP+for+Building+Educational+App)
25. Klaus Zechner and Xinhao Wang. 2013. [Automated content scoring of spoken responses in an assessment for teachers of english](https://md.research.elamur.ai/free/en/papers/zechner-2013-automated-content-scoring-spoken-responses/). In Proceedings of the 8th Workshop on Innovative Use of NLP for Building Educational Applications, NAACL-HLT, Atlanta. Association for Computational Linguistics.
26. Klaus Zechner, Derrick Higgins, Xiaoming Xi, and David M. Williamson. 2009. Automatic scoring of non-native spontaneous speech in tests of spoken English. Speech Communication, 51(10):883–895. [↗](https://doi.org/10.1016/j.specom.2009.04.009)
27. Klaus Zechner, Xiaoming Xi, and Lei Chen. 2011. Evaluating prosodic features for automated scoring of non-native read speech. In Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU). [find](https://scholar.google.com/scholar?q=Klaus+Zechner%2C+Xiaoming+Xi%2C+and+Lei+Chen%2E+2011%2E+Evaluating+prosodic+features+for+automated+scoring+of+non%2Dnative+read+speech%2E+In+Proceedings+of+the+IEEE+Workshop+on+Automatic+Speech+Recognition+and+Un)

*Updated 1 November 2026*