--- title: "Speech- and Text-driven Features for Automated Scoring of" description: "The paper asks how to automatically score one- to two-minute spoken answers in an English proficiency… 4 references, cited by 3 papers here." authors: ["Anastassia Loukina", "Nitin Madnani", "Aoife Cahill"] year: 2017 venue: "Proceedings of the Workshop on Speech-Centric Natural Language Processing" source: "https://aclanthology.org/W17-4609/" doi: "10.18653/v1/w17-4609" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/loukina-2017-speech-text-driven-features-automated/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/loukina-2017-speech-text-driven-features-automated/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/5/) / Speech- and Text-driven Features for Automated Scoring of English Speaking Tasks # Speech- and Text-driven Features for Automated Scoring of English Speaking Tasks **Authors:** Anastassia Loukina, [Nitin Madnani](https://md.research.elamur.ai/free/en/authors/nitin-madnani/) and [Aoife Cahill](https://md.research.elamur.ai/free/en/authors/aoife-cahill/) · **Year:** 2017 · **Venue:** Proceedings of the Workshop on Speech-Centric Natural Language Processing · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W17-4609/) · [PDF](https://aclanthology.org/W17-4609.pdf) · [DOI 10.18653/v1/w17-4609](https://doi.org/10.18653/v1/w17-4609) · [OpenAlex](https://openalex.org/W2760368719) **Citations:** 14 citations (OpenAlex, 2026-10-14) · 3 in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper asks how to automatically score one- to two-minute spoken answers in an English proficiency test, where human raters judge both what was said and how it was said. The authors join two existing ETS components. The first is a content-scoring system built for essays, which they run on 1-best ASR transcripts. The second is a set of 33 acoustic features covering speech rate, pausing, pronunciation and prosody. The data are more than 150,000 responses to 147 questions of two kinds: source-based questions, which rely on given material, and open-ended general questions. Combining both feature sets gives a statistically significant gain over either set alone, but the gain is modest and appears only when all speech feature groups are added together. Text-only models do better on source-based questions. On general questions, speech-only and text-only models perform about equally well. The authors argue that the ASR transcript already carries much of the acoustic information, such as where pauses and hesitations occur. They also note that different proficiency dimensions are strongly correlated. ## Key points - The corpus has 153,461 responses from 33,503 test takers to 147 questions (48 general, 99 source-based). Each response received a holistic human score from 1 to 4. Responses whose ASR transcript had fewer than 10 words were removed. - Recognition used a Kaldi-based ASR with a trigram LM and a 5-layer DNN acoustic model. It was trained on 800 hours of non-native speech from 8,700 speakers, and its WER on a similar corpus is about 30%. - The data were split roughly 70/30 into train and evaluation sets. No test taker appears in both partitions, and the demographics of the two sets were kept similar. - The fusion is simple concatenation of 33 dense speech features with tens of thousands of sparse binary text features. Significance tests used a Bonferroni-corrected α = 0.0125. - Removing pause and disfluency markers from the transcripts significantly hurts the text-only model. It does not affect the combined model, because the speech features carry the same information. ## Interesting ideas - Strip one kind of signal (pause and hesitation tokens) from the input and compare the single-channel and combined models; this shows whether two channels hold redundant information. - Assign all data from one person to a single partition, so the evaluation cannot reward memorizing an individual's quirks. - The authors suspect that a small dense feature block gets drowned out when concatenated with a huge sparse one, and they suggest stacking (separate models joined by a higher-level model) as an alternative. - Report results separately for tasks grounded in provided material and for open-ended tasks, since the value of content features differs sharply between them. - Apply a multiple-comparison correction when testing many model variants against each other. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): Removing one evidence source (such as VerbNet roles or morphology cues) and checking whether the full pipeline still holds steady would show which components are truly independent witnesses. - **I2** (learn): Splitting silver and test data by source document or author, not by sentence, would keep evaluation scores honest. - **I3** (tree): When hand-crafted dense features are mixed into the perceptron parser's sparse feature set, training separate scorers and combining their outputs could stop the dense block from being swamped. - **I4** (answer): Fable-QA and school-test accuracy could be reported separately for questions answerable from the text and for those needing outside knowledge, to show where the world model actually helps. - **I5** (learn): A Bonferroni-style threshold in ablation reports would keep chance gains from being mistaken for real improvements from new repair rules. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Educational Testing Service](https://md.research.elamur.ai/free/en/ngos/educational-testing-service/) — Nonprofit, Princeton, United States · 27 works · 2,212 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (2): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 480 works ## Cite Cite as: Anastassia Loukina, Nitin Madnani and Aoife Cahill. 2017. Speech- and Text-driven Features for Automated Scoring of English Speaking Tasks. Proceedings of the Workshop on Speech-Centric Natural Language Processing. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{loukina2017speech, title = {{Speech- and Text-driven Features for Automated Scoring of English Speaking Tasks}}, author = {Anastassia Loukina, Nitin Madnani and Aoife Cahill}, booktitle = {Proceedings of the Workshop on Speech-Centric Natural Language Processing}, year = {2017}, doi = {10.18653/v1/w17-4609}, url = {https://doi.org/10.18653/v1/w17-4609} } ``` - [← previous Or what?](https://md.research.elamur.ai/free/en/papers/biezma-2017-or-what/) - [next → Towards Full Text Shallow Discourse Relation Annotation: Experiments with Cross-Paragraph Implicit Relations in the PDTB](https://md.research.elamur.ai/free/en/papers/prasad-2017-full-text-shallow-discourse-relation/) - [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/5/) ## References 4 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Jared Bernstein, John De Jong, David B. Pisoni, and Brent Townshend. 2000. Two experiments on automatic scoring of spoken language proficiency. In Proceedings of InStil2000. pages 57–61. [find](https://scholar.google.com/scholar?q=Jared+Bernstein%2C+John+De+Jong%2C+David+B%2E+Pisoni%2C+and+Brent+Townshend%2E+2000%2E+Two+experiments+on+automatic+scoring+of+spoken+language+proficiency%2E+In+Proceedings+of+InStil2000%2E+pages+57%E2%80%9361%2E) 2. Suma Bhat and Su-Youn Yoon. 2015. Automatic assessment of syntactic complexity for spontaneous speech scoring. Speech Communication 67:42–57. [↗](https://doi.org/10.1016/j.specom.2014.09.005) 3. Steven Burrows, Iryna Gurevych, and Benno Stein. 2015. The eras and trends of automatic short answer grading. International Journal of Artificial Intelligence in Education 25(1):60–117. [↗](https://doi.org/10.1007/s40593-014-0026-8) 4. Lei Chen and Klaus Zechner. 2011. Applying rhythm features to automatically assess non-native speech. In Proceedings of Interspeech. pages 1861–1864. [find](https://scholar.google.com/scholar?q=Lei+Chen+and+Klaus+Zechner%2E+2011%2E+Applying+rhythm+features+to+automatically+assess+non%2Dnative+speech%2E+In+Proceedings+of+Interspeech%2E+pages+1861%E2%80%931864%2E) *Updated 1 November 2026*