---
title: "Towards Robust Mispronunciation Detection and Diagnosis for L2"
description: "Common end-to-end systems pool speech from all accents into one model, and the authors argue this… 30 references, cited by 2 papers here."
authors: ["Shao-Wei Fan Jiang", "Bi-Cheng Yan", "Tien-Hong Lo", "Fu-An Chao", "Berlin Chen"]
year: 2021
venue: "arXiv:2108.11627"
source: "https://arxiv.org/abs/2108.11627"
doi: "10.48550/arxiv.2108.11627"
arxiv: "2108.11627"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jiang-2021-towards-robust-mispronunciation-detection-diagnosis/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jiang-2021-towards-robust-mispronunciation-detection-diagnosis/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/6/) / Towards Robust Mispronunciation Detection and Diagnosis for L2 English Learners with Accent-Modulating Methods

# Towards Robust Mispronunciation Detection and Diagnosis for L2 English Learners with Accent-Modulating Methods

**Authors:** Shao-Wei Fan Jiang, [Bi-Cheng Yan](https://md.research.elamur.ai/free/en/authors/bi-cheng-yan/), Tien-Hong Lo, [Fu-An Chao](https://md.research.elamur.ai/free/en/authors/fu-an-chao/) and [Berlin Chen](https://md.research.elamur.ai/free/en/authors/berlin-chen/) · **Year:** 2021 · **Venue:** [arXiv:2108.11627](https://md.research.elamur.ai/free/en/venues/ieee-automatic-speech-recognition-and-understanding-workshop-asru/) ([ASRU 2021](https://md.research.elamur.ai/free/en/venues/ieee-automatic-speech-recognition-and-understanding-workshop-asru/2021/)) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2108.11627) · [PDF](https://arxiv.org/pdf/2108.11627) · [DOI 10.48550/arxiv.2108.11627](https://doi.org/10.48550/arxiv.2108.11627) · [arXiv 2108.11627](https://arxiv.org/abs/2108.11627) · [OpenAlex](https://openalex.org/W4226046739)

**Citations:** 18 citations (OpenAlex, 2026-10-14) · 2 in our library, of the 141,840 papers whose bibliographies we read

## Summary
Computer-assisted pronunciation training needs mispronunciation detection and diagnosis (MDD) that works for learners of English with many different native-language accents. Common end-to-end systems pool speech from all accents into one model, and the authors argue this weakens the model's ability to tell correct pronunciations from errors. They start from a hybrid CTC-ATT phone recognizer and make its encoder accent-aware in two ways. Accent-modulated concatenation (AMC) appends an accent embedding to every encoder frame. Accent-modulated gating (AMG) uses that embedding to compute a sigmoid gate that rescales each frame. Both versions need the accent label at test time, so they also add "soft" variants (AMC-S, AMG-S). In these, an auxiliary accent classifier is trained jointly with the recognizer, and its hidden summary replaces the true label. Accents then only have to be known for the training data. On L2-ARCTIC, with TIMIT as native-speech data, the accent-aware models beat a plain CTC-ATT baseline and a GOP scoring baseline. AMG-S was the strongest, which makes accent-robust diagnosis practical without asking learners for their background.

## Key points
- The base system is a hybrid CTC-ATT model with a shared encoder, attention decoder and CTC branch. Its loss interpolates the CTC and attention log-posteriors with weight α = 0.3.
- Two fusion mechanisms are proposed:
  - AMC: concatenation of an utterance-level accent embedding with each frame, followed by a linear projection.
  - AMG: a sigmoid gate computed from the frame and accent vector, applied elementwise, followed by ReLU and a projection.
- The soft variants (AMC-S, AMG-S) add a multi-layer bidirectional GRU accent classifier over the encoder states. It is trained jointly with the recognizer, with weight β balancing the two losses.
- The data come from two corpora:
  - TIMIT: 462 training speakers, 3696 utterances, 3.15 h of native speech.
  - L2-ARCTIC: 17 training speakers, 2549 utterances, 2.66 h, plus a test set of 6 speakers, 900 utterances and 0.88 h.
- Evaluation is based on a confusion matrix over correct versus mispronounced phones (CP/MP). For diagnostic accuracy (DAR), the neural models fall roughly between 70.7% and 73.75%. AMG-S is the best of them, though the authors admit DAR is still far from satisfactory.

## Interesting ideas
- Condition every frame on one utterance-level attribute by gating, rather than only concatenating it, so the attribute can amplify or suppress individual feature dimensions.
- Replace a side label that is unavailable at inference with the hidden state of a jointly trained auxiliary classifier, so labels are needed only for training data.
- Recognize what was actually produced without constraints, then compare it with the expected prompt to yield typed errors (substitution, deletion, insertion) rather than a single score.
- The authors warn that merging many sub-populations into one model can blur the distinctions the model needs to make.
- Pair a monotonic-alignment objective (CTC) with a flexible attention decoder, interpolated by a fixed weight, so the rigid objective stabilizes the flexible one.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tree): A gate in gparse computed from a treebank/dialect embedding could let one parser adapt its token representations to each UD dialect instead of averaging conventions.
- **I2** (learn): Train a source/genre classifier alongside the parser on silver data, then feed its hidden summary to the parser so unlabeled input text still gets domain-aware analysis.
- **I3** (answer): When grading student answers, align the produced frame/graph against the expected one and report which roles were swapped, missing or extra, keeping feedback explainable.
- **I4** (learn): Before pooling treebanks or registers into one training set, measure whether pooling hurts per-source accuracy, and add explicit source conditioning when it does.
- **I5** (back): During text generation from the meaning graph, an auxiliary monotonic alignment loss between graph order and output tokens could curb dropped or repeated content.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National Taiwan Normal University](https://md.research.elamur.ai/free/en/universities/national-taiwan-normal-university/) — University, Taipei, Taiwan · 3 works · 117 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 798 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 463 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 213 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works

## Cite

Cite as: Shao-Wei Fan Jiang, Bi-Cheng Yan, Tien-Hong Lo, Fu-An Chao and Berlin Chen. 2021. Towards Robust Mispronunciation Detection and Diagnosis for L2 English Learners with Accent-Modulating Methods. arXiv:2108.11627.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{jiang2021robust,
  title = {{Towards Robust Mispronunciation Detection and Diagnosis for L2 English Learners with Accent-Modulating Methods}},
  author = {Shao-Wei Fan Jiang, Bi-Cheng Yan, Tien-Hong Lo, Fu-An Chao and Berlin Chen},
  year = {2021},
  doi = {10.48550/arxiv.2108.11627},
  eprint = {2108.11627},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2108.11627}
}
```

- [← previous The Accented English Speech Recognition Challenge 2020: Open Datasets, Tracks, Baselines, Results and Methods](https://md.research.elamur.ai/free/en/papers/shi-2021-accented-english-speech-recognition-challenge/)
- [next → A Context-Aware Hierarchical BERT Fusion Network for Multi-turn Dialog Act Detection](https://md.research.elamur.ai/free/en/papers/wu-2021-context-aware-hierarchical-bert-fusion/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/6/)

## References

30 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. K. Evanini and X. Wang, “Automated speech scoring for nonnative middle school students with multiple task types,” in Proceedings of Interspeech, pp. 2435–2439, 2013. [find](https://scholar.google.com/scholar?q=K%2E+Evanini+and+X%2E+Wang%2C+%E2%80%9CAutomated+speech+scoring+for+nonnative+middle+school+students+with+multiple+task+types%2C%E2%80%9D+in+Proceedings+of+Interspeech%2C+pp%2E+2435%E2%80%932439%2C+2013%2E)
2. A. Metallinou and J. Cheng, “Using deep neural networks to improve proficiency assessment for children English language learners,” in Proceedings of Interspeech, pp. 1468–1472, 2014. [find](https://scholar.google.com/scholar?q=A%2E+Metallinou+and+J%2E+Cheng%2C+%E2%80%9CUsing+deep+neural+networks+to+improve+proficiency+assessment+for+children+English+language+learners%2C%E2%80%9D+in+Proceedings+of+Interspeech%2C+pp%2E+1468%E2%80%931472%2C+2014%2E)
3. K. Li et al., “Mispronunciation detection and diagnosis in l2 English speech using multidistribution deep neural networks,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, 193–207, 2016. [↗](https://doi.org/10.1109/taslp.2016.2621675)
4. H. Franco et al., “Automatic pronunciation scoring for language instruction,” in Proceedings of ICASSP, pp. 1471–1474, 1997. [find](https://scholar.google.com/scholar?q=H%2E+Franco+et+al%2E%2C+%E2%80%9CAutomatic+pronunciation+scoring+for+language+instruction%2C%E2%80%9D+in+Proceedings+of+ICASSP%2C+pp%2E+1471%E2%80%931474%2C+1997%2E)
5. S.M. Witt and S.J.Young, “Phone-level pronunciation scoring and assessment for interactive language learning,” Speech Communication, pp. 95–108, 2000. [↗](https://doi.org/10.1016/s0167-6393(99)00044-8)
6. S. Sudhakara et al., “An improved goodness of pronunciation (GoP) measure for pronunciation evaluation with DNN-HMM system considering HMM transition probabilities,” in Proceedings of Interspeech, pp. 954–958, 2019. [find](https://scholar.google.com/scholar?q=S%2E+Sudhakara+et+al%2E%2C+%E2%80%9CAn+improved+goodness+of+pronunciation+%28GoP%29+measure+for+pronunciation+evaluation+with+DNN%2DHMM+system+considering+HMM+transition+probabilities%2C%E2%80%9D+in+Proceedings+of+Interspeech%2C+pp%2E)
7. Z. Wang et al., “A multi-view approach for Mandarin non-native mispronunciation verification,” in Proceedings of ICASSP, pp. 8079–8083, 2020. [↗](https://doi.org/10.48550/arxiv.2009.02573)
8. W. K. Lo et al., “Automatic derivation of phonological rules for mispronunciation detection in a computer-assisted pronunciation training system,” in Proceedings of Interspeech, 2010. [find](https://scholar.google.com/scholar?q=W%2E+K%2E+Lo+et+al%2E%2C+%E2%80%9CAutomatic+derivation+of+phonological+rules+for+mispronunciation+detection+in+a+computer%2Dassisted+pronunciation+training+system%2C%E2%80%9D+in+Proceedings+of+Interspeech%2C+2010%2E)
9. H. Meng et al., “Deriving salient learners’ mispronunciations from cross-language phonological comparisons,” in Proceedings of ASRU, 2007. [find](https://scholar.google.com/scholar?q=H%2E+Meng+et+al%2E%2C+%E2%80%9CDeriving+salient+learners%E2%80%99+mispronunciations+from+cross%2Dlanguage+phonological+comparisons%2C%E2%80%9D+in+Proceedings+of+ASRU%2C+2007%2E)
10. W. Li et al., “Improving non-native mispronunciation detection and enriching diagnostic feedback with DNN-based speech attribute modeling,” in Proceedings of ICASSP, pp. 6135–6139, 2016. [find](https://scholar.google.com/scholar?q=W%2E+Li+et+al%2E%2C+%E2%80%9CImproving+non%2Dnative+mispronunciation+detection+and+enriching+diagnostic+feedback+with+DNN%2Dbased+speech+attribute+modeling%2C%E2%80%9D+in+Proceedings+of+ICASSP%2C+pp%2E+6135%E2%80%936139%2C+2016%2E)
11. WK. Leung et al., “CNN-RNN-CTC based end-to-end mispronunciation detection and diagnosis,” in Proceedings of ICASSP, pp. 8132–8136, 2019. [find](https://scholar.google.com/scholar?q=WK%2E+Leung+et+al%2E%2C+%E2%80%9CCNN%2DRNN%2DCTC+based+end%2Dto%2Dend+mispronunciation+detection+and+diagnosis%2C%E2%80%9D+in+Proceedings+of+ICASSP%2C+pp%2E+8132%E2%80%938136%2C+2019%2E)
12. Y. Feng et al., “SED-MDD: Towards sentence dependent end-to-end mispronunciation detection and diagnosis,” in Proceedings of ICASSP, pp. 3492–3496, 2020. [find](https://scholar.google.com/scholar?q=Y%2E+Feng+et+al%2E%2C+%E2%80%9CSED%2DMDD%3A+Towards+sentence+dependent+end%2Dto%2Dend+mispronunciation+detection+and+diagnosis%2C%E2%80%9D+in+Proceedings+of+ICASSP%2C+pp%2E+3492%E2%80%933496%2C+2020%2E)
13. B. C. Yan et al., “[An end-to-end mispronunciation detection system for L2 English speech leveraging novel anti-phone modeling](https://md.research.elamur.ai/free/en/papers/yan-2020-end-end-mispronunciation-detection-system/),” in Proceedings of Interspeech, pp. 3032–3036, 2020.
14. A. Jain et al., “A multi-accent acoustic model using mixture of experts for speech recognition,” in Proceedings of Interspeech, pp. 779–783, 2019. [find](https://scholar.google.com/scholar?q=A%2E+Jain+et+al%2E%2C+%E2%80%9CA+multi%2Daccent+acoustic+model+using+mixture+of+experts+for+speech+recognition%2C%E2%80%9D+in+Proceedings+of+Interspeech%2C+pp%2E+779%E2%80%93783%2C+2019%2E)
15. V. Pratap et al., “Massively multilingual ASR: 50 languages, 1 model, 1 billion parameters,” in Proceedings of Interspeech, pp. 4751–4755,2020. [find](https://scholar.google.com/scholar?q=V%2E+Pratap+et+al%2E%2C+%E2%80%9CMassively+multilingual+ASR%3A+50+languages%2C+1+model%2C+1+billion+parameters%2C%E2%80%9D+in+Proceedings+of+Interspeech%2C+pp%2E+4751%E2%80%934755%2C2020%2E)
16. D. Vergyri et al., “Automatic speech recognition of multiple accented English data,” in Proceedings of Interspeech, 2010. [find](https://scholar.google.com/scholar?q=D%2E+Vergyri+et+al%2E%2C+%E2%80%9CAutomatic+speech+recognition+of+multiple+accented+English+data%2C%E2%80%9D+in+Proceedings+of+Interspeech%2C+2010%2E)
17. S. Tong et al., “Cross-lingual adaptation of a CTC-based multilingual acoustic model,” Speech Communication, vol. 104, pp. 39–46, 2018. [↗](https://doi.org/10.1016/j.specom.2018.09.001)
18. X. Yang et al., “[Joint modeling of accents and acoustics for multi-accent speech recognition](https://md.research.elamur.ai/free/en/papers/yang-2018-joint-modeling-accents-acoustics-multi/),” in Proceedings of ICASSP, pp.1–5, 2018.
19. A. Jain et al., “Improved accented speech recognition using accent embeddings and multi-task learning,” in Proceedings of Interspeech, pp. 2454–2458, 2018. [find](https://scholar.google.com/scholar?q=A%2E+Jain+et+al%2E%2C+%E2%80%9CImproved+accented+speech+recognition+using+accent+embeddings+and+multi%2Dtask+learning%2C%E2%80%9D+in+Proceedings+of+Interspeech%2C+pp%2E+2454%E2%80%932458%2C+2018%2E)
20. S. Punjabi et al., “Streaming end-to-end bilingual ASR systems with joint language identification,” arXiv preprint arXiv:2007.03900, 2020. [↗](https://arxiv.org/abs/2007.03900)
21. S. Watanabe et al., “Hybrid CTC/attention architecture for end-to-end speech recognition,” IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240–1253, 2017. [↗](https://doi.org/10.1109/jstsp.2017.2763455)
22. A. Graves et al., “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of ICML, vol. 148, 2006. [find](https://scholar.google.com/scholar?q=A%2E+Graves+et+al%2E%2C+%E2%80%9CConnectionist+temporal+classification%3A+labelling+unsegmented+sequence+data+with+recurrent+neural+networks%2C%E2%80%9D+in+Proceedings+of+ICML%2C+vol%2E+148%2C+2006%2E)
23. G. Zhao et al., “L2-ARCTIC: A non-native English speech corpus,” in Proceedings of Interspeech, pp. 2783–2787, 2018. [find](https://scholar.google.com/scholar?q=G%2E+Zhao+et+al%2E%2C+%E2%80%9CL2%2DARCTIC%3A+A+non%2Dnative+English+speech+corpus%2C%E2%80%9D+in+Proceedings+of+Interspeech%2C+pp%2E+2783%E2%80%932787%2C+2018%2E)
24. J.S. Garofolo et al., “DARPA TIMIT: acoustic-phonetic continuous speech corpus CD-ROM, NIST speech disc 1–1.1,” STIN, vol. 93, 1993. [find](https://scholar.google.com/scholar?q=J%2ES%2E+Garofolo+et+al%2E%2C+%E2%80%9CDARPA+TIMIT%3A+acoustic%2Dphonetic+continuous+speech+corpus+CD%2DROM%2C+NIST+speech+disc+1%E2%80%931%2E1%2C%E2%80%9D+STIN%2C+vol%2E+93%2C+1993%2E)
25. G. Pundak et al., “Deep context: end-to-end contextual speech recognition,” in Proceedings of SLT, pp. 418–425, 2018. [find](https://scholar.google.com/scholar?q=G%2E+Pundak+et+al%2E%2C+%E2%80%9CDeep+context%3A+end%2Dto%2Dend+contextual+speech+recognition%2C%E2%80%9D+in+Proceedings+of+SLT%2C+pp%2E+418%E2%80%93425%2C+2018%2E)
26. B.-C. Yan and B. Chen, “End-to-end mispronunciation detection and diagnosis from raw waveforms,” in Proceedings of EUSIPCO, 2021. [find](https://scholar.google.com/scholar?q=B%2E%2DC%2E+Yan+and+B%2E+Chen%2C+%E2%80%9CEnd%2Dto%2Dend+mispronunciation+detection+and+diagnosis+from+raw+waveforms%2C%E2%80%9D+in+Proceedings+of+EUSIPCO%2C+2021%2E)
27. M. Wu et al., “Transformer based end-to-end mispronunciation detection and diagnosis,” in Proceedings of Interspeech, pp. 3954–3958, 2021. [find](https://scholar.google.com/scholar?q=M%2E+Wu+et+al%2E%2C+%E2%80%9CTransformer+based+end%2Dto%2Dend+mispronunciation+detection+and+diagnosis%2C%E2%80%9D+in+Proceedings+of+Interspeech%2C+pp%2E+3954%E2%80%933958%2C+2021%2E)
28. X. Xu et al., “Explore wav2vec 2.0 for mispronunciation detection,” in Proceedings of Interspeech, pp. 4428–4432, 2021. [find](https://scholar.google.com/scholar?q=X%2E+Xu+et+al%2E%2C+%E2%80%9CExplore+wav2vec+2%2E0+for+mispronunciation+detection%2C%E2%80%9D+in+Proceedings+of+Interspeech%2C+pp%2E+4428%E2%80%934432%2C+2021%2E)
29. T.-H. Lo et al., “Improving end-to-end modeling for mispronunciation detection with effective augmentation mechanisms,” in Proceedings of APSIPA ASC, 2021. [↗](https://doi.org/10.48550/arxiv.2110.08731)
30. F.-A. Chao et al., “TENET: A time-reversal enhancement network for noise-robust ASR,” in Proceedings of ASRU, 2021. [find](https://scholar.google.com/scholar?q=F%2E%2DA%2E+Chao+et+al%2E%2C+%E2%80%9CTENET%3A+A+time%2Dreversal+enhancement+network+for+noise%2Drobust+ASR%2C%E2%80%9D+in+Proceedings+of+ASRU%2C+2021%2E)

*Updated 24 October 2026*