---
title: "An End-to-End Mispronunciation Detection System for L2 English"
description: "The authors build an end-to-end MDD system on a hybrid CTC-Attention recognizer: a… 56.02% F1. 19 references, cited by 6 papers here."
authors: ["Bi-Cheng Yan", "Meng-Che Wu", "Hsiao-Tsung Hung", "Berlin Chen"]
year: 2020
venue: "arXiv:2005.11950"
source: "https://arxiv.org/abs/2005.11950"
doi: "10.48550/arxiv.2005.11950"
arxiv: "2005.11950"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yan-2020-end-end-mispronunciation-detection-system/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yan-2020-end-end-mispronunciation-detection-system/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/) / An End-to-End Mispronunciation Detection System for L2 English Speech Leveraging Novel Anti-Phone Modeling

# An End-to-End Mispronunciation Detection System for L2 English Speech Leveraging Novel Anti-Phone Modeling

**Authors:** [Bi-Cheng Yan](https://md.research.elamur.ai/free/en/authors/bi-cheng-yan/), Meng-Che Wu, Hsiao-Tsung Hung and [Berlin Chen](https://md.research.elamur.ai/free/en/authors/berlin-chen/) · **Year:** 2020 · **Venue:** [arXiv:2005.11950](https://md.research.elamur.ai/free/en/venues/interspeech-usb/) ([Interspeech 2020](https://md.research.elamur.ai/free/en/venues/interspeech-usb/2020/)) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2005.11950) · [PDF](https://arxiv.org/pdf/2005.11950) · [DOI 10.48550/arxiv.2005.11950](https://doi.org/10.48550/arxiv.2005.11950) · [arXiv 2005.11950](https://arxiv.org/abs/2005.11950) · [OpenAlex](https://openalex.org/W3097515180)

**Citations:** 40 citations (OpenAlex, 2026-10-14) · [6](https://md.research.elamur.ai/free/en/papers/yan-2020-end-end-mispronunciation-detection-system/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Computer-assisted pronunciation training (CAPT) depends on mispronunciation detection and diagnosis (MDD). Most MDD systems only handle categorical errors, where a learner substitutes, deletes or inserts a whole phone. They miss distortions, where the sound produced lies somewhere between the target L2 phone and an L1 phone. The authors build an end-to-end MDD system on a hybrid CTC-Attention recognizer: a BLSTM encoder, an attention-based ULSTM decoder, and an auxiliary CTC branch, trained with an interpolated loss. They double the phone inventory by adding an "anti-phone" (marked with #) for every canonical phone, so the model can label a distorted realization of a phone without forcing it into another category. Two further steps support this. Training data is augmented by randomly swapping reference labels for anti-phones in correctly pronounced utterances, and training proceeds from a native-speech model to an accented one through transfer learning, with no hand-written phonological rules. On L2-ARCTIC the system reaches 56.02% F1, well above GOP and a coarse single-label variant. The work matters because it shows distortion errors can be represented inside an end-to-end model.

## Key points
- Data:
  - Native-speech pretraining uses TIMIT plus a small part of Librispeech.
  - Accented training and evaluation use the L2-ARCTIC non-native English corpus.
  - The label-shuffling augmentation is applied only to L2-ARCTIC utterances that contain no pronunciation errors.
- Mispronunciation detection F1:
  - CTC-ATT (Anti): 56.02%, with precision 46.57% and recall 70.28%.
  - CTC-ATT (Unk), which uses one coarse anti-label: 44.97%.
  - GOP: 28.31%.
  - These gaps are the 11.05 and 27.71 points reported in the abstract.
- Architecture comparison with anti-phones:
  - Detection F1 is 53.52% for CTC alone, 52.25% for Attention alone and 56.02% for the hybrid.
  - Diagnosis accuracy (DAR) is 32.46%, 37.02% and 40.66% respectively.
- Error breakdown:
  - The ground truth contains 771 non-categorical and 3,310 categorical errors.
  - Phone-specific anti-phones correctly diagnose 9.4% (73) of non-categorical and 33.02% (1,093) of categorical errors.
  - The coarse Unk variant manages 8.4% (65) and 19.63% (650).
- The authors note that diagnosis remains much harder than detection, and that distortion errors are still largely unsolved.

## Interesting ideas
- Give every canonical label its own "near-miss" twin (anti-label), so that a deviant input is marked as a distorted version of a specific category rather than forced into a wrong category or a single catch-all bucket.
- Create training examples for the "deviant" classes by randomly relabeling parts of clean, correct data with anti-labels at a fixed probability, which yields abundant synthetic negatives without needing annotated errors.
- Compare phone-specific anti-labels against one shared unknown label, and break results down by error type; this isolates whether fine-grained rejection classes actually pay off.
- Train in stages: first a model on clean "standard" data, then reuse its encoder to start training on the shifted domain, instead of hand-writing rules for every variety.
- Report detection (is something wrong?) and diagnosis (what exactly is wrong?) as separate metrics, since a system can do well on the first and poorly on the second.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): The tagger could emit labels such as "#VERB" or a "#lemma" flag for nonstandard or misspelled words, saying which category was intended while marking that the form deviates from it.
- **I2** (learn): Silver treebanks could be duplicated with randomly flipped "anti-relation" labels to train a module that flags suspicious arcs, giving cheap negative controls for evaluation.
- **I3** (learn): Whenever a fallback label is added (an unknown frame, an unknown role), it should be tested against per-category fallbacks and scored separately for categorical and distortion-like failures.
- **I4** (tree): gparse could be pretrained on standard UD treebanks, and the encoder then reused to start training on dialect-converted or learner-English data, instead of writing conversion rules for every dialect.
- **I5** (answer): When grading student answers, Mova could score "found an error" separately from "named the right error", making it explicit how well its explanations hold up.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National Taiwan Normal University](https://md.research.elamur.ai/free/en/universities/national-taiwan-normal-university/) — University, Taipei, Taiwan · 3 works · 117 citations
- [Asus](https://md.research.elamur.ai/free/en/companies/asus/) — Company, Taipei, Taiwan · 2 works · 1,627 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 798 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 463 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 213 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works

## Cite

Cite as: Bi-Cheng Yan, Meng-Che Wu, Hsiao-Tsung Hung and Berlin Chen. 2020. An End-to-End Mispronunciation Detection System for L2 English Speech Leveraging Novel Anti-Phone Modeling. arXiv:2005.11950.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{yan2020end,
  title = {{An End-to-End Mispronunciation Detection System for L2 English Speech Leveraging Novel Anti-Phone Modeling}},
  author = {Bi-Cheng Yan, Meng-Che Wu, Hsiao-Tsung Hung and Berlin Chen},
  year = {2020},
  doi = {10.48550/arxiv.2005.11950},
  eprint = {2005.11950},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2005.11950}
}
```

- [← previous Paraphrasing vs Coreferring: Two Sides of the Same Coin](https://md.research.elamur.ai/free/en/papers/meged-2020-paraphrasing-vs-coreferring-two-sides/)
- [next → CxGBERT: BERT meets Construction Grammar](https://md.research.elamur.ai/free/en/papers/madabushi-2020-cxgbert-bert-meets-construction-grammar/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/)

## References

19 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. S. M. Witt and S. J. Young, “Phone-level pronunciation scoring and assessment for interactive language learning,” Speech Communication, vol. 30, pp. 95–108, 2000. [↗](https://doi.org/10.1016/s0167-6393(99)00044-8)
2. W. Hu, et al, “Improved mispronunciation detection with deep neural network trained acoustic models and transfer learning based logistic regression classifiers,” Speech Communication, vol. 67, pp.154–166, 2015. [↗](https://doi.org/10.1016/j.specom.2014.12.008)
3. S. Sudhakara, et al, “An improved goodness of pronunciation (GoP) measure for pronunciation evaluation with DNN-HMM system considering HMM transition probabilities,” Proceedings of the INTERSPEECH, pp. 954–958, 2019. [find](https://scholar.google.com/scholar?q=S%2E+Sudhakara%2C+et+al%2C+%E2%80%9CAn+improved+goodness+of+pronunciation+%28GoP%29+measure+for+pronunciation+evaluation+with+DNN%2DHMM+system+considering+HMM+transition+probabilities%2C%E2%80%9D+Proceedings+of+the+INTERSPEECH%2C+pp)
4. W. Li, et al, “Improving non-native mispronunciation detection and enriching diagnostic feedback with DNN-based speech attribute modeling,” Proceedings of the ICASSP, pp. 6135–6139, 2016. [find](https://scholar.google.com/scholar?q=W%2E+Li%2C+et+al%2C+%E2%80%9CImproving+non%2Dnative+mispronunciation+detection+and+enriching+diagnostic+feedback+with+DNN%2Dbased+speech+attribute+modeling%2C%E2%80%9D+Proceedings+of+the+ICASSP%2C+pp%2E+6135%E2%80%936139%2C+2016%2E)
5. W. Lo, S. Zhang, and H. Meng, “Automatic derivation of phonological rules for mispronunciation detection in a computer-assisted pronunciation training system,” Proceedings of the INTERSPEECH, pp.765–768, 2010. [find](https://scholar.google.com/scholar?q=W%2E+Lo%2C+S%2E+Zhang%2C+and+H%2E+Meng%2C+%E2%80%9CAutomatic+derivation+of+phonological+rules+for+mispronunciation+detection+in+a+computer%2Dassisted+pronunciation+training+system%2C%E2%80%9D+Proceedings+of+the+INTERSPEECH%2C+pp%2E765%E2%80%937)
6. X. Qian, F. K. Soong, and H. Meng, “Discriminative acoustic model for improving mispronunciation detection and diagnosis in computer-aided pronunciation training (CAPT),” in International Speech Communication Association, 2010. [find](https://scholar.google.com/scholar?q=X%2E+Qian%2C+F%2E+K%2E+Soong%2C+and+H%2E+Meng%2C+%E2%80%9CDiscriminative+acoustic+model+for+improving+mispronunciation+detection+and+diagnosis+in+computer%2Daided+pronunciation+training+%28CAPT%29%2C%E2%80%9D+in+International+Speech+Commu)
7. W. Leung, X. Liu and H. Meng, “CNN-RNN-CTC Based End-to-end Mispronunciation Detection and Diagnosis,” Proceedings of the ICASSP, pp. 8132–8136, 2019. [find](https://scholar.google.com/scholar?q=W%2E+Leung%2C+X%2E+Liu+and+H%2E+Meng%2C+%E2%80%9CCNN%2DRNN%2DCTC+Based+End%2Dto%2Dend+Mispronunciation+Detection+and+Diagnosis%2C%E2%80%9D+Proceedings+of+the+ICASSP%2C+pp%2E+8132%E2%80%938136%2C+2019%2E)
8. Y. Feng, et al, “SED-MDD: towards sentence dependent end-to-end mispronunciation detection and diagnosis,” Proceedings of the ICASSP, pp. 3492-3496, 2020 [find](https://scholar.google.com/scholar?q=Y%2E+Feng%2C+et+al%2C+%E2%80%9CSED%2DMDD%3A+towards+sentence+dependent+end%2Dto%2Dend+mispronunciation+detection+and+diagnosis%2C%E2%80%9D+Proceedings+of+the+ICASSP%2C+pp%2E+3492%2D3496%2C+2020)
9. L. Chen, et al, “End-to-End neural network based automated speech scoring,” Proceedings of the ICASSP, pp. 6234–6238, 2018. [find](https://scholar.google.com/scholar?q=L%2E+Chen%2C+et+al%2C+%E2%80%9CEnd%2Dto%2DEnd+neural+network+based+automated+speech+scoring%2C%E2%80%9D+Proceedings+of+the+ICASSP%2C+pp%2E+6234%E2%80%936238%2C+2018%2E)
10. S. Mao, et al, “Unsupervised Discovery of an Extended Phoneme Set in L2 English Speech for Mispronunciation Detection and Diagnosis,” Proceedings of the ICASSP, pp. 6244–6248, 2018. [find](https://scholar.google.com/scholar?q=S%2E+Mao%2C+et+al%2C+%E2%80%9CUnsupervised+Discovery+of+an+Extended+Phoneme+Set+in+L2+English+Speech+for+Mispronunciation+Detection+and+Diagnosis%2C%E2%80%9D+Proceedings+of+the+ICASSP%2C+pp%2E+6244%E2%80%936248%2C+2018%2E)
11. X. Li, et al, “Unsupervised discovery of non-native phonetic patterns in L2 English speech for mispronunciation detection and diagnosis” Proceedings of the INTERSPEECH, pp. 2554–2558, 2018. [find](https://scholar.google.com/scholar?q=X%2E+Li%2C+et+al%2C+%E2%80%9CUnsupervised+discovery+of+non%2Dnative+phonetic+patterns+in+L2+English+speech+for+mispronunciation+detection+and+diagnosis%E2%80%9D+Proceedings+of+the+INTERSPEECH%2C+pp%2E+2554%E2%80%932558%2C+2018%2E)
12. S. Watanabe, et al, “Hybrid CTC/attention architecture for end-to-end speech recognition,” IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240–1253, 2017. [↗](https://doi.org/10.1109/jstsp.2017.2763455)
13. O. Ronen, L. Neumeyer, and H. Franco, “Automatic detection of mispronunciation for language instruction,” Proceedings of Speech Communication and Technology, pp. 649–652, 1997. [find](https://scholar.google.com/scholar?q=O%2E+Ronen%2C+L%2E+Neumeyer%2C+and+H%2E+Franco%2C+%E2%80%9CAutomatic+detection+of+mispronunciation+for+language+instruction%2C%E2%80%9D+Proceedings+of+Speech+Communication+and+Technology%2C+pp%2E+649%E2%80%93652%2C+1997%2E)
14. J. K. Chorowski, et al, “[Attention-based models for speech recognition](https://md.research.elamur.ai/free/en/papers/chorowski-2015-attention-based-models-speech-recognition/),” Proceedings of NIPS, pp. 577–585, 2015. [↗](https://doi.org/10.48550/arxiv.1506.07503)
15. G. Zhao, et al. “L2-ARCTIC: A Non-native English Speech Corpus,” Proceedings of the INTERSPEECH, pp. 2783–2787, 2018. [find](https://scholar.google.com/scholar?q=G%2E+Zhao%2C+et+al%2E+%E2%80%9CL2%2DARCTIC%3A+A+Non%2Dnative+English+Speech+Corpus%2C%E2%80%9D+Proceedings+of+the+INTERSPEECH%2C+pp%2E+2783%E2%80%932787%2C+2018%2E)
16. V. Panayotov, et al, “Librispeech: an asr corpus based on public domain audio books,” Proceedings of the ICASSP, pp. 5206–5210, 2015. [find](https://scholar.google.com/scholar?q=V%2E+Panayotov%2C+et+al%2C+%E2%80%9CLibrispeech%3A+an+asr+corpus+based+on+public+domain+audio+books%2C%E2%80%9D+Proceedings+of+the+ICASSP%2C+pp%2E+5206%E2%80%935210%2C+2015%2E)
17. J. Cho, et al, “Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling,” Proceedings of the SLT, pp. 521–527, 2018. [find](https://scholar.google.com/scholar?q=J%2E+Cho%2C+et+al%2C+%E2%80%9CMultilingual+sequence%2Dto%2Dsequence+speech+recognition%3A+architecture%2C+transfer+learning%2C+and+language+modeling%2C%E2%80%9D+Proceedings+of+the+SLT%2C+pp%2E+521%E2%80%93527%2C+2018%2E)
18. K. Li, X. Qian and H. Meng, “Mispronunciation Detection and Diagnosis in L2 English Speech Using Multidistribution Deep Neural Networks,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, 193–207, 2016. [↗](https://doi.org/10.1109/taslp.2016.2621675)
19. S. Mao, et al, “Applying multitask learning to acoustic-phonemic model for mispronunciation detection and diagnosis in L2 English speech,” Proceedings of the ICASSP, pp. 6254–6258, 2018. [find](https://scholar.google.com/scholar?q=S%2E+Mao%2C+et+al%2C+%E2%80%9CApplying+multitask+learning+to+acoustic%2Dphonemic+model+for+mispronunciation+detection+and+diagnosis+in+L2+English+speech%2C%E2%80%9D+Proceedings+of+the+ICASSP%2C+pp%2E+6254%E2%80%936258%2C+2018%2E)

*Updated 24 October 2026*