---
title: "Assessing the Prosody of Non-Native Speakers of English: Measures and Feature Sets (2016)"
description: "The paper addresses automatic rating of how natural the prosody (melody and rhythm) of non-native English speech… 20 references, cited by 7 papers here."
authors: ["Eduardo Coutinho", "Florian Hönig", "Yue Zhang", "Simone Hantke", "Anton Batliner", "Elmar Nöth", "Björn Schuller"]
year: 2016
venue: "Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)"
source: "https://aclanthology.org/L16-1211/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/coutinho-2016-assessing-prosody-non-native-speakers/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/coutinho-2016-assessing-prosody-non-native-speakers/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/) / Assessing the Prosody of Non-Native Speakers of English: Measures and Feature Sets

# Assessing the Prosody of Non-Native Speakers of English: Measures and Feature Sets

**Authors:** Eduardo Coutinho, Florian Hönig, [Yue Zhang](https://md.research.elamur.ai/free/en/authors/yue-zhang/), Simone Hantke, Anton Batliner, Elmar Nöth and Björn Schuller · **Year:** 2016 · **Venue:** [Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2016](https://md.research.elamur.ai/free/en/venues/lrec/2016/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/L16-1211/) · [PDF](https://aclanthology.org/L16-1211.pdf) · [OpenAlex](https://openalex.org/W2576641009)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · 7 in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper addresses automatic rating of how natural the prosody (melody and rhythm) of non-native English speech sounds. Such ratings could support pronunciation-training software and the adaptation of speech recognisers to L2 speakers. The authors built a new corpus of 54 L2 speakers, each reading 11 sentences from two standard phonetics passages. Native English listeners scored every recording for prosodic naturalness. The authors derived the target score in three ways: median, mean, and an evaluator weighted estimator (EWE). They then trained linear Support Vector Regression models on several acoustic feature families. These were partitions of the openSMILE ComParE set plus specialised prosodic, rhythm and tempo features computed from phoneme-recognizer segmentation. Evaluation used strictly speaker-independent cross-validation. Prosodic features worked best. Combining feature families helped only slightly, with late fusion doing better than early fusion. The work shows that text-independent, suprasegmental acoustic cues are enough to assess L2 prosody reasonably well. It also offers a realistic, leakage-free estimate of performance together with a ceiling.

## Key points
- The corpus has 54 speakers with 15 different L1s, mostly German and Chinese. It contains 594 recordings totalling about 1.4 hours. The texts are "The North Wind and the Sun" and "The Rainbow".
- Twenty-seven native raters used a 5-point scale ranging from normal to very unusual. Agreement measured by ICC(2, k) was 0.84–0.96 across sessions and rater-gender subgroups.
- The feature families were:
  - the openSMILE ComParE set (6373 features), split into oS PROS (483), oS SPEC (5500) and oS VQ (390);
  - PROS (3072), built from pseudo-syllable features;
  - RHYT (13), including PVI, Ramus' %V and Deltas, and Varco;
  - DUR (2), consisting of articulation rate and pause percentage.
- Spearman ρ on the EWE target was 0.600 for PROS, 0.542 for oS PROS, 0.526 for DUR alone and 0.411 for RHYT.
  - Late fusion averaged .586 versus .562 for early fusion.
  - The best result, 0.617, came from late fusion of all sets except oS SPEC.
- Two reference points frame the results:
  - Per-speaker averaged gold scores correlate with the EWE target at ρ = 0.823, which serves as an empirical ceiling.
  - Ordinary, non-speaker-independent CV inflates the result to 0.715.

## Interesting ideas
- Rather than averaging raters equally, EWE weights each annotator by how well their scores track the group consensus, which reduces the influence of noisy or inattentive judges.
- The authors contrast speaker-disjoint folds with default random folds (0.617 vs 0.715), which exposes how much a naive split overstates accuracy.
- An empirical ceiling is estimated by checking how well a crude per-speaker average predicts the gold scores, which tells you how much headroom actually remains.
- In late fusion, the meta-regressor is trained on predictions from the inner cross-validation, so the stacker never sees outputs that the base models produced on their own training data.
- Two transparent tempo features come close to sets with thousands of opaque descriptors, so small interpretable baselines are worth checking before scaling up.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Silver-data agreement could weight each witness parser by its correlation with the consensus on held-out sentences, instead of using a flat majority vote.
- **I2** (learn): Evaluation should split by source document or author so that repeated style or vocabulary does not leak across folds, and the gap to a random split should be reported as a leakage diagnostic.
- **I3** (answer): For school-test and fable-QA benchmarks, a simple oracle-style baseline (e.g., per-text or per-question-type averages) could show whether gains are still possible or the task is saturated.
- **I4** (tree): When combining the perceptron parser and gparse with a learned arbiter, the arbiter should be trained on out-of-fold parses to avoid overtrusting whichever parser memorised the training treebank.
- **I5** (learn): Each new learned component should be benchmarked against a handful of explainable hand-built cues, keeping the large model only if it clearly beats them.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Imperial College London](https://md.research.elamur.ai/free/en/universities/imperial-college-london/) — University, London, United Kingdom · 24 works · 8,272 citations
- [Technical University of Munich](https://md.research.elamur.ai/free/en/universities/technical-university-of-munich/) — University, Munich, Germany · 18 works · 48,618 citations
- [University of Liverpool](https://md.research.elamur.ai/free/en/universities/university-of-liverpool/) — University, Liverpool, United Kingdom · 7 works · 539 citations
- [University of Passau](https://md.research.elamur.ai/free/en/universities/university-of-passau/) — University, Passau, Germany · 3 works · 654 citations
- [Friedrich-Alexander-Universität Erlangen-Nürnberg](https://md.research.elamur.ai/free/en/universities/friedrich-alexander-universitat-erlangen-nurnberg/) — University, Erlangen, Germany · 4 works · 26 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 798 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 463 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 213 works

🧭 Topics (2): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 233 works

## Cite

Cite as: Eduardo Coutinho, Florian Hönig, Yue Zhang, Simone Hantke, Anton Batliner, Elmar Nöth and Björn Schuller. 2016. Assessing the Prosody of Non-Native Speakers of English: Measures and Feature Sets. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{coutinho2016assessing,
  title = {{Assessing the Prosody of Non-Native Speakers of English: Measures and Feature Sets}},
  author = {Eduardo Coutinho, Florian Hönig, Yue Zhang, Simone Hantke, Anton Batliner, Elmar Nöth and Björn Schuller},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  year = {2016},
  doi = {10.63317/5gjerdvgynop},
  url = {https://doi.org/10.63317/5gjerdvgynop}
}
```

- [← previous An Arabic-Hebrew parallel corpus of TED talks](https://md.research.elamur.ai/free/en/papers/cettolo-2016-arabic-hebrew-parallel-corpus-ted/)
- [next → Beyond Identity Coreference: Contrasting Indicators of Textual Coherence in English and German](https://md.research.elamur.ai/free/en/papers/kunz-2016-beyond-identity-coreference-contrasting-indicators/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/)

## References

20 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Batliner, A., Buckow, J., Niemann, H., Nöth, E., and Warnke, V. (2000). The prosody module. In W. Wahlster, editor, Verbmobil: Foundations of Speech-to-Speech Translations, pages 106–121. Springer, Berlin. [↗](https://doi.org/10.1007/978-3-662-04230-4_8)
2. Chang, C.-C. and Lin, C.-J. (2011). LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1–27:27. Software available at [http://www.csie.ntu.edu](http://www.csie.ntu.edu). tw/˜cjlin/libsvm. [↗](http://www.csie.ntu.edu)
3. Chen, L. and Zechner, K. (2011). Applying rhythm features to automatically assess non-native speech. In Proc. INTERSPEECH 2011, pages 1861–1864. [find](https://scholar.google.com/scholar?q=Chen%2C+L%2E+and+Zechner%2C+K%2E+%282011%29%2E+Applying+rhythm+features+to+automatically+assess+non%2Dnative+speech%2E+In+Proc%2E+INTERSPEECH+2011%2C+pages+1861%E2%80%931864%2E)
4. Dellwo, V. (2010). Influences of speech rate on the acoustic correlates of speech rhythm: An experimental phonetic study based on acoustic and perceptual evidence. Universität Bonn, Bonn University. Bonn, Germany. [find](https://scholar.google.com/scholar?q=Dellwo%2C+V%2E+%282010%29%2E+Influences+of+speech+rate+on+the+acoustic+correlates+of+speech+rhythm%3A+An+experimental+phonetic+study+based+on+acoustic+and+perceptual+evidence%2E+Universita%CC%88t+Bonn%2C+Bonn+University%2E+)
5. Eyben, F., Weninger, F., Groß, F., and Schuller, B. (2013). Recent Developments in openSMILE, the Munich Open-Source Multimedia Feature Extractor. In Proceedings of the 21st ACM International Conference on Multimedia, MM 2013, pages 835–838, Barcelona, Spain, October. ACM, ACM. [find](https://scholar.google.com/scholar?q=Eyben%2C+F%2E%2C+Weninger%2C+F%2E%2C+Gro%C3%9F%2C+F%2E%2C+and+Schuller%2C+B%2E+%282013%29%2E+Recent+Developments+in+openSMILE%2C+the+Munich+Open%2DSource+Multimedia+Feature+Extractor%2E+In+Proceedings+of+the+21st+ACM+International+Conferen)
6. Grabe, E. and Low, E. L. (2002). Durational variability in speech and the rhythm class hypothesis. Papers in laboratory phonology, 7(515-546). [↗](https://doi.org/10.1515/9783110197105.2.515)
7. Grimm, M. and Kroschel, K. (2005). Evaluation of natural emotions using self assessment manikins. In Proc. ASRU, pages 381–385. IEEE. [find](https://scholar.google.com/scholar?q=Grimm%2C+M%2E+and+Kroschel%2C+K%2E+%282005%29%2E+Evaluation+of+natural+emotions+using+self+assessment+manikins%2E+In+Proc%2E+ASRU%2C+pages+381%E2%80%93385%2E+IEEE%2E)
8. Hönig, F., Batliner, A., Weilhammer, K., and Nöth, E. (2010). Automatic assessment of non-native prosody for English as l2. In Proc. Speech Prosody, Chicago. no pagination. [find](https://scholar.google.com/scholar?q=Ho%CC%88nig%2C+F%2E%2C+Batliner%2C+A%2E%2C+Weilhammer%2C+K%2E%2C+and+No%CC%88th%2C+E%2E+%282010%29%2E+Automatic+assessment+of+non%2Dnative+prosody+for+English+as+l2%2E+In+Proc%2E+Speech+Prosody%2C+Chicago%2E+no+pagination%2E)
9. Hönig, F., Batliner, A., and Nöth, E. (2012a). Automatic assessment of non-native prosody – annotation, modelling and evaluation. In Proceedings of IS-ADEPT, International Symposium on Automatic Detection of Errors in Pronunciation Training, June 6-8, Stockholm, Sweden. [find](https://scholar.google.com/scholar?q=Ho%CC%88nig%2C+F%2E%2C+Batliner%2C+A%2E%2C+and+No%CC%88th%2C+E%2E+%282012a%29%2E+Automatic+assessment+of+non%2Dnative+prosody+%E2%80%93+annotation%2C+modelling+and+evaluation%2E+In+Proceedings+of+IS%2DADEPT%2C+International+Symposium+on+Automatic+Det)
10. Hönig, F., Bocklet, T., Riedhammer, K., Batliner, A., and Nöth, E. (2012b). The automatic assessment of non-native prosody: Combining classical prosodic analysis with acoustic modelling. In Proc. INTERSPEECH 2012, pages 823–826, Portland (Oregon),USA, September. [find](https://scholar.google.com/scholar?q=Ho%CC%88nig%2C+F%2E%2C+Bocklet%2C+T%2E%2C+Riedhammer%2C+K%2E%2C+Batliner%2C+A%2E%2C+and+No%CC%88th%2C+E%2E+%282012b%29%2E+The+automatic+assessment+of+non%2Dnative+prosody%3A+Combining+classical+prosodic+analysis+with+acoustic+modelling%2E+In+Proc%2E+IN)
11. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. [↗](https://doi.org/10.48550/arxiv.1201.0490)
12. Platt, J. C. (1999). Fast training of support vector machines using sequential minimal optimization. In Advances in kernel methods – support vector learning, pages 185– 208. MIT Press. [↗](https://doi.org/10.7551/mitpress/1130.003.0016)
13. Ramus, F. (2002). Acoustic correlates of linguistic rhythm: perspectives. In Proc. of Speech Prosody, pages 115– 120, Aix-en-Provence, France. Laboratoire Parole et Langage. [find](https://scholar.google.com/scholar?q=Ramus%2C+F%2E+%282002%29%2E+Acoustic+correlates+of+linguistic+rhythm%3A+perspectives%2E+In+Proc%2E+of+Speech+Prosody%2C+pages+115%E2%80%93+120%2C+Aix%2Den%2DProvence%2C+France%2E+Laboratoire+Parole+et+Langage%2E)
14. Schuller, B., Steidl, S., Batliner, A., Vinciarelli, A., Scherer, K., Ringeval, F., Chetouani, M., Weninger, F., Eyben, F., Marchi, E., Mortillaro, M., Salamin, H., Polychroniou, A., Valente, F., and Kim, S. (2013). [find](https://scholar.google.com/scholar?q=Schuller%2C+B%2E%2C+Steidl%2C+S%2E%2C+Batliner%2C+A%2E%2C+Vinciarelli%2C+A%2E%2C+Scherer%2C+K%2E%2C+Ringeval%2C+F%2E%2C+Chetouani%2C+M%2E%2C+Weninger%2C+F%2E%2C+Eyben%2C+F%2E%2C+Marchi%2C+E%2E%2C+Mortillaro%2C+M%2E%2C+Salamin%2C+H%2E%2C+Polychroniou%2C+A%2E%2C+Valente%2C+F%2E%2C+and+)
15. The IN-TERSPEECH 2013 Computational Paralinguistics Challenge: Social Signals, Conflict, Emotion, Autism. In Proc. INTERSPEECH 2013, pages 148–152, Lyon, France, August. ISCA, ISCA. [find](https://scholar.google.com/scholar?q=The+IN%2DTERSPEECH+2013+Computational+Paralinguistics+Challenge%3A+Social+Signals%2C+Conflict%2C+Emotion%2C+Autism%2E+In+Proc%2E+INTERSPEECH+2013%2C+pages+148%E2%80%93152%2C+Lyon%2C+France%2C+August%2E+ISCA%2C+ISCA%2E)
16. Schwarz, P., Matejka, P., and Cernocky, J. (2006). Hierarchical structures of neural networks for phoneme recognition. In Proceedings of ICASSP, pages 325–328, Toulouse. [find](https://scholar.google.com/scholar?q=Schwarz%2C+P%2E%2C+Matejka%2C+P%2E%2C+and+Cernocky%2C+J%2E+%282006%29%2E+Hierarchical+structures+of+neural+networks+for+phoneme+recognition%2E+In+Proceedings+of+ICASSP%2C+pages+325%E2%80%93328%2C+Toulouse%2E)
17. Shrout, P. E. and Fleiss, J. L. (1979). Intraclass correlations: uses in assessing rater reliability. Psychological bulletin, 86(2):420. [↗](https://doi.org/10.1037//0033-2909.86.2.420)
18. Smola, A. and Schlkopf, B. (2004). A tutorial on support vector regression. Statistics and Computing, 14(3):199– 222. [↗](https://doi.org/10.1023/b:stco.0000035301.49549.88)
19. Van Compernolle, D. (2001). Recognizing speech of goats, wolves, sheep and ... non-natives. Speech Communication, 35(1):71–79. [↗](https://doi.org/10.1016/s0167-6393(00)00096-0)
20. Weninger, F., Eyben, F., Schuller, B. W., Mortillaro, M., and Scherer, K. R. (2013). On the Acoustics of Emotion in Audio: What Speech, Music and Sound have in Common. Frontiers in Psychology, 4:1–12. [↗](https://doi.org/10.3389/fpsyg.2013.00292)

*Updated 24 October 2026*