---
title: "Experiments of ASR-based mispronunciation detection for"
description: "The paper builds a phone-level mispronunciation detector for Italian learners of English. 26 references, cited by 1 paper here."
authors: ["Nina Hosseini-Kivanani; Roberto Gretter; Marco Matassoni;", "Giuseppe Daniele Falavigna"]
year: 2021
venue: "arXiv:2104.05980"
source: "https://arxiv.org/abs/2104.05980"
doi: "10.48550/arxiv.2104.05980"
arxiv: "2104.05980"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hosseini-kivanani-2021-experiments-asr-based-mispronunciation-detection/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hosseini-kivanani-2021-experiments-asr-based-mispronunciation-detection/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/7/) / Experiments of ASR-based mispronunciation detection for children and adult English learners

# Experiments of ASR-based mispronunciation detection for children and adult English learners

**Authors:** Nina Hosseini-Kivanani; Roberto Gretter; Marco Matassoni; and Giuseppe Daniele Falavigna · **Year:** 2021 · **Venue:** arXiv:2104.05980 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2104.05980) · [PDF](https://arxiv.org/pdf/2104.05980) · [DOI 10.48550/arxiv.2104.05980](https://doi.org/10.48550/arxiv.2104.05980) · [arXiv 2104.05980](https://arxiv.org/abs/2104.05980) · [OpenAlex](https://openalex.org/W3153457231)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper builds a phone-level mispronunciation detector for Italian learners of English. It also aims to make speech recognition more robust to the non-native pronunciations that inflate recognition errors. The system is a Kaldi phone recognizer with a hybrid acoustic model: GMM stages bootstrap TDNNs trained with LF-MMI on i-vector-augmented MFCCs. Two decoding setups are compared. One uses free phone n-grams (orders 1–5). The other is an "error model": the lexicon of known words is expanded with variant pronunciations produced by hand-written rules, which were derived from the errors the 5-gram recognizer made most often. Evaluation uses two phonetically hand-annotated corpora, ChildEn (Italian children) and the Italian part of ISLE (adults). Each utterance is analyzed through three aligned phone transcriptions: canonical, manually corrected, and ASR output. The authors report that the error model separates correct from incorrect sounds and lowers phone error rate compared with the plain n-gram setup. They argue such detectors could support self-directed pronunciation practice in CAPT tools, provided they are adapted to the learner's native language.

## Key points
- **Data:**
  - ChildEn: 5,268 utterances (about 3h28m) from 78 children around ten years old, with 3–4 years of school English.
  - ISLE (adult Italian learners): the abstract reports 5,867 utterances from 46 speakers, while the data section mentions 23 Italian speakers. The full ISLE audio totals about 17h54m.
- **Acoustic model (standard Kaldi recipe):**
  - GMM stages on MFCCs with LDA and speaker adaptive training.
  - 100-dimensional i-vectors appended to the features.
  - TDNNs trained with LF-MMI on mixed child and adult speech.
- **Phone inventory:** it combines English and Italian phones, because the two languages reportedly share only about 40% of their phones. This lets the recognizer output L1-coloured realizations.
- **Error types:** errors are classified as substitution, insertion or deletion. The most frequent errors involve English phones that Italian lacks (e.g., /ax/), which speakers replace with the nearest Italian sound or drop.
- **Results:** the error model improves PER and accuracy over plain phone n-grams on both corpora.

## Interesting ideas
- Comparing three transcriptions pairwise gives three different measurements: canonical vs. manual maps the true errors, canonical vs. system shows what the system flags on unannotated input, and manual vs. system scores detection quality.
- The error rules were bootstrapped in two steps: an unconstrained phone recognizer first revealed the most frequent deviations, which were then frozen into explicit rewrite rules for a constrained decoder.
- Instead of free decoding, the search space is limited to the canonical form plus a small set of rule-generated plausible variants. This keeps outputs interpretable as named deviations from the norm.
- The symbol inventory is widened to cover both the target and the interfering system (English plus Italian phones), so interference can be represented rather than forced into the nearest target category.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Silver-data evaluation could compare the expected/converted analysis, the gold tree and parser output pairwise, separating real annotation divergence from parser mistakes.
- **I2** (learn): Self-induced repair rules could follow the same loop, mining the commonest disagreements of an unconstrained pass and promoting them to explicit, inspectable rules.
- **I3** (tokens): Normalizing noisy or learner text could generate rule-based variant spellings of known lemmas, so each correction carries the rule that explains it.
- **I4** (tree): During UD dialect conversion, a union label inventory could keep source-specific relations visible until they are explicitly mapped, instead of silently collapsing them.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 798 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 463 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 213 works

🧭 Topics (2): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 233 works

## Cite

Cite as: Nina Hosseini-Kivanani; Roberto Gretter; Marco Matassoni; and Giuseppe Daniele Falavigna. 2021. Experiments of ASR-based mispronunciation detection for children and adult English learners. arXiv:2104.05980.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{matassoni2021experiments,
  title = {{Experiments of ASR-based mispronunciation detection for children and adult English learners}},
  author = {Nina Hosseini-Kivanani; Roberto Gretter; Marco Matassoni; and Giuseppe Daniele Falavigna},
  year = {2021},
  doi = {10.48550/arxiv.2104.05980},
  eprint = {2104.05980},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2104.05980}
}
```

- [← previous Expanding the JHU Bible Corpus for Machine Translation of the Indigenous Languages of North America](https://md.research.elamur.ai/free/en/papers/nicolai-2021-expanding-jhu-bible-corpus-machine/)
- [next → Graph Rewriting for Enhanced Universal Dependencies](https://md.research.elamur.ai/free/en/papers/guillaume-2021-graph-rewriting-enhanced-universal-dependencies/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/7/)

## References

26 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Huensch, “The pronunciation teaching practices of university-level graduate teaching assistants of French and Spanish introductory language courses,” Foreign Language Annals, vol. 52, no. 1, pp. 13–31, 2019. [↗](https://doi.org/10.1111/flan.12372)
2. S. M. McCrocklin, “Pronunciation learner autonomy: The potential of Automatic Speech Recognition,” System, vol. 57, no. February, pp. 25–42, 2016. [↗](https://doi.org/10.1016/j.system.2015.12.013)
3. M. A. Peabody, “Methods for pronunciation assessment in computer aided language learning,” Ph.D. dissertation, Massachusetts Institute of Technology, 2011. [find](https://scholar.google.com/scholar?q=M%2E+A%2E+Peabody%2C+%E2%80%9CMethods+for+pronunciation+assessment+in+computer+aided+language+learning%2C%E2%80%9D+Ph%2ED%2E+dissertation%2C+Massachusetts+Institute+of+Technology%2C+2011%2E)
4. G. Fant, Speech sounds and features. The MIT Press, 1973. [find](https://scholar.google.com/scholar?q=G%2E+Fant%2C+Speech+sounds+and+features%2E+The+MIT+Press%2C+1973%2E)
5. S. M. Witt, “Automatic error detection in pronunciation training: Where we are and where we need to go,” Proceedings of the International Symposium on Automatic Detection of Errors in Pronunciation Training (IS ADEPT), vol. 6, pp. 1–8, 2012. [find](https://scholar.google.com/scholar?q=S%2E+M%2E+Witt%2C+%E2%80%9CAutomatic+error+detection+in+pronunciation+training%3A+Where+we+are+and+where+we+need+to+go%2C%E2%80%9D+Proceedings+of+the+International+Symposium+on+Automatic+Detection+of+Errors+in+Pronunciation+Tr)
6. A. Neri, C. Cucchiarini, and W. Strik, “Automatic speech recognition for second language learning: how and why it actually works.” in In Proc. ICPhS, 2003, pp. 1157–1160. [find](https://scholar.google.com/scholar?q=A%2E+Neri%2C+C%2E+Cucchiarini%2C+and+W%2E+Strik%2C+%E2%80%9CAutomatic+speech+recognition+for+second+language+learning%3A+how+and+why+it+actually+works%2E%E2%80%9D+in+In+Proc%2E+ICPhS%2C+2003%2C+pp%2E+1157%E2%80%931160%2E)
7. G. Kawai and K. Hirose, “A call system using speech recognition to teach the pronunciation of japanese tokushuhaku,” in STiLL-Speech Technology in Language Learning, 1998. [find](https://scholar.google.com/scholar?q=G%2E+Kawai+and+K%2E+Hirose%2C+%E2%80%9CA+call+system+using+speech+recognition+to+teach+the+pronunciation+of+japanese+tokushuhaku%2C%E2%80%9D+in+STiLL%2DSpeech+Technology+in+Language+Learning%2C+1998%2E)
8. K. Zechner, “What did they actually say? Agreement and disagreement among transcribers of non-native spontaneous speech responses in an English proficiency test,” in International Workshop on Speech and Language Technology in Education, 2009. [find](https://scholar.google.com/scholar?q=K%2E+Zechner%2C+%E2%80%9CWhat+did+they+actually+say%3F+Agreement+and+disagreement+among+transcribers+of+non%2Dnative+spontaneous+speech+responses+in+an+English+proficiency+test%2C%E2%80%9D+in+International+Workshop+on+Speech+a)
9. R. Downey, H. Farhady, R. Present-Thomas, M. Suzuki, and A. Van Moere, “Evaluation of the usefulness of the versant for english test: A response,” Language Assessment Quarterly, vol. 5, no. 2, pp. 160–167, 2008. [find](https://scholar.google.com/scholar?q=R%2E+Downey%2C+H%2E+Farhady%2C+R%2E+Present%2DThomas%2C+M%2E+Suzuki%2C+and+A%2E+Van+Moere%2C+%E2%80%9CEvaluation+of+the+usefulness+of+the+versant+for+english+test%3A+A+response%2C%E2%80%9D+Language+Assessment+Quarterly%2C+vol%2E+5%2C+no%2E+2%2C+pp%2E+160)
10. S. Witt and S. Young, “Phone-level pronunciation scoring and assessment for interactive language learning,” Speech Communication, vol. 30, no. 2-3, pp. 95–108, feb 2000. [↗](https://doi.org/10.1016/s0167-6393(99)00044-8)
11. H. Ryu, H. Hong, S. Kim, and M. Chung, “Automatic pronunciation assessment of Korean spoken by L2 learners using best feature set selection,” 2016 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2016, 2017. [find](https://scholar.google.com/scholar?q=H%2E+Ryu%2C+H%2E+Hong%2C+S%2E+Kim%2C+and+M%2E+Chung%2C+%E2%80%9CAutomatic+pronunciation+assessment+of+Korean+spoken+by+L2+learners+using+best+feature+set+selection%2C%E2%80%9D+2016+Asia%2DPacific+Signal+and+Information+Processing+Associ)
12. S. M. Witt, “Use of speech recognition in computer-assisted language learning,” Ph.D. dissertation, University of Cambridge Cambridge, United Kingdom, 1999. [find](https://scholar.google.com/scholar?q=S%2E+M%2E+Witt%2C+%E2%80%9CUse+of+speech+recognition+in+computer%2Dassisted+language+learning%2C%E2%80%9D+Ph%2ED%2E+dissertation%2C+University+of+Cambridge+Cambridge%2C+United+Kingdom%2C+1999%2E)
13. O. Franco, Horacio and Neumeyer, Leonardo and Kim, Yoon and Ronen, “Automatic pronunciation scoring for language instruction organization,” in 1997 IEEE international conference on acoustics, speech, and signal processing. IEEE, 1997, pp. 1471–1474. [find](https://scholar.google.com/scholar?q=O%2E+Franco%2C+Horacio+and+Neumeyer%2C+Leonardo+and+Kim%2C+Yoon+and+Ronen%2C+%E2%80%9CAutomatic+pronunciation+scoring+for+language+instruction+organization%2C%E2%80%9D+in+1997+IEEE+international+conference+on+acoustics%2C+speech%2C+)
14. L. Kim, Yoon and Franco, Horacio and Neumeyer, “Automatic pronunciation scoring of specific phone segments for language instruction,” in Fifth European Conference on Speech Communication and Technology, 1997. [find](https://scholar.google.com/scholar?q=L%2E+Kim%2C+Yoon+and+Franco%2C+Horacio+and+Neumeyer%2C+%E2%80%9CAutomatic+pronunciation+scoring+of+specific+phone+segments+for+language+instruction%2C%E2%80%9D+in+Fifth+European+Conference+on+Speech+Communication+and+Technolog)
15. M. Eskenazi, “Using automatic speech processing for foreign language pronunciation tutoring: Some issues and a prototype,” Language Learning and Technology, vol. 2, no. 2, pp. 62–76, 1999. [↗](https://doi.org/10.64152/10125/25043)
16. J. Tao, S. Ghaffarzadegan, L. Chen, and K. Zechner, “Exploring deep learning architectures for automatically grading non-native spontaneous speech,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2016, pp. 6140–6144. [find](https://scholar.google.com/scholar?q=J%2E+Tao%2C+S%2E+Ghaffarzadegan%2C+L%2E+Chen%2C+and+K%2E+Zechner%2C+%E2%80%9CExploring+deep+learning+architectures+for+automatically+grading+non%2Dnative+spontaneous+speech%2C%E2%80%9D+in+2016+IEEE+International+Conference+on+Acoustics%2C)
17. P. Boersma and D. Weenink, “Praat: Doing phonetics by computer (Version 6.0. 14),” Retrieved from (last access: 29.04. 2018), 2016. [find](https://scholar.google.com/scholar?q=P%2E+Boersma+and+D%2E+Weenink%2C+%E2%80%9CPraat%3A+Doing+phonetics+by+computer+%28Version+6%2E0%2E+14%29%2C%E2%80%9D+Retrieved+from+%28last+access%3A+29%2E04%2E+2018%29%2C+2016%2E)
18. D. Jurafsky and J. H. Martin, “N-gram language models,” Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, 2018. [find](https://scholar.google.com/scholar?q=D%2E+Jurafsky+and+J%2E+H%2E+Martin%2C+%E2%80%9CN%2Dgram+language+models%2C%E2%80%9D+Speech+and+Language+Processing%3A+An+Introduction+to+Natural+Language+Processing%2C+Computational+Linguistics%2C+and+Speech+Recognition%2C+2018%2E)
19. F. Miao, Yajie and Zhang, Hao and Metze, “Speaker adaptive training of deep neural network acoustic models using i-vectors,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 23, no. 11, pp. 1938–1949, 2015. [↗](https://doi.org/10.1109/taslp.2015.2457612)
20. M. Madikeri, Srikanth and Dey, Subhadeep and Motlicek, Petr and Ferras, “Implementation of the standard i-vector system for the kaldi speech recognition toolkit,” Idiap, Tech. Rep., 2016. [find](https://scholar.google.com/scholar?q=M%2E+Madikeri%2C+Srikanth+and+Dey%2C+Subhadeep+and+Motlicek%2C+Petr+and+Ferras%2C+%E2%80%9CImplementation+of+the+standard+i%2Dvector+system+for+the+kaldi+speech+recognition+toolkit%2C%E2%80%9D+Idiap%2C+Tech%2E+Rep%2E%2C+2016%2E)
21. D. Povey, V. Peddinti, D. Galvez, P. Ghahremani, V. Manohar, X. Na, Y. Wang, and S. Khudanpur, “Purely sequence-trained neural networks for asr based on lattice-free mmi.” in Interspeech, 2016, pp. 2751–2755. [find](https://scholar.google.com/scholar?q=D%2E+Povey%2C+V%2E+Peddinti%2C+D%2E+Galvez%2C+P%2E+Ghahremani%2C+V%2E+Manohar%2C+X%2E+Na%2C+Y%2E+Wang%2C+and+S%2E+Khudanpur%2C+%E2%80%9CPurely+sequence%2Dtrained+neural+networks+for+asr+based+on+lattice%2Dfree+mmi%2E%E2%80%9D+in+Interspeech%2C+2016%2C+pp%2E+27)
22. A. Batliner, M. Blomberg, S. D’Arcy, D. Elenius, D. Giuliani, M. Gerosa, C. Hacker, M. Russell, S. Steidl, and M. Wong, “The PF STAR children’s speech corpus,” in Ninth European Conference on Speech Communication and Technology, 2005. [find](https://scholar.google.com/scholar?q=A%2E+Batliner%2C+M%2E+Blomberg%2C+S%2E+D%E2%80%99Arcy%2C+D%2E+Elenius%2C+D%2E+Giuliani%2C+M%2E+Gerosa%2C+C%2E+Hacker%2C+M%2E+Russell%2C+S%2E+Steidl%2C+and+M%2E+Wong%2C+%E2%80%9CThe+PF+STAR+children%E2%80%99s+speech+corpus%2C%E2%80%9D+in+Ninth+European+Conference+on+Speech+C)
23. W. Menzel, E. Atwell, P. Bonaventura, D. Herron, P. Howarth, R. Morton, and C. Souter, “The ISLE corpus of non-native spoken English,” in 2nd International Conference on Language Resources and Evaluation, LREC 2000, vol. 2. European Language Resources Association, 2000, pp. 957–964. [find](https://scholar.google.com/scholar?q=W%2E+Menzel%2C+E%2E+Atwell%2C+P%2E+Bonaventura%2C+D%2E+Herron%2C+P%2E+Howarth%2C+R%2E+Morton%2C+and+C%2E+Souter%2C+%E2%80%9CThe+ISLE+corpus+of+non%2Dnative+spoken+English%2C%E2%80%9D+in+2nd+International+Conference+on+Language+Resources+and+Evaluat)
24. S. Browning, “Analysis of Italian children’s English pronunciation,” Report contributed to the EU FP5 PF STAR Project, 2004. [find](https://scholar.google.com/scholar?q=S%2E+Browning%2C+%E2%80%9CAnalysis+of+Italian+children%E2%80%99s+English+pronunciation%2C%E2%80%9D+Report+contributed+to+the+EU+FP5+PF+STAR+Project%2C+2004%2E)
25. K. Kawai, Goh and Hirose, “A CALL system using speech recognition to teach the pronunciation of Japanese tokushuhaku,” in STiLL-Speech Technology in Language Learning, 1998. [find](https://scholar.google.com/scholar?q=K%2E+Kawai%2C+Goh+and+Hirose%2C+%E2%80%9CA+CALL+system+using+speech+recognition+to+teach+the+pronunciation+of+Japanese+tokushuhaku%2C%E2%80%9D+in+STiLL%2DSpeech+Technology+in+Language+Learning%2C+1998%2E)
26. L. Chen, K. Zechner, S. Y. Yoon, K. Evanini, X. Wang, A. Loukina, J. Tao, L. Davis, C. M. Lee, M. Ma, R. Mundkowsky, C. Lu, C. W. Leong, and B. Gyawali, “Automated Scoring of Nonnative Speech Using the SpeechRaterSM v. 5.0 Engine,” ETS Research Report Series, vol. 2018, no. 1, pp. 1–31, dec 2018. [find](https://scholar.google.com/scholar?q=L%2E+Chen%2C+K%2E+Zechner%2C+S%2E+Y%2E+Yoon%2C+K%2E+Evanini%2C+X%2E+Wang%2C+A%2E+Loukina%2C+J%2E+Tao%2C+L%2E+Davis%2C+C%2E+M%2E+Lee%2C+M%2E+Ma%2C+R%2E+Mundkowsky%2C+C%2E+Lu%2C+C%2E+W%2E+Leong%2C+and+B%2E+Gyawali%2C+%E2%80%9CAutomated+Scoring+of+Nonnative+Speech+Using+th)

*Updated 24 October 2026*