---
title: "Transformer-Based Multi-Aspect Multi-Granularity Non-Native"
description: "Gong et al. instead train one model that predicts eight labels together. 27 references, cited by 11 papers here."
authors: ["Yuan Gong", "Ziyi Chen", "Iek-Heng Chu", "Peng Chang", "James Glass"]
year: 2022
venue: "arXiv:2205.03432"
source: "https://arxiv.org/abs/2205.03432"
doi: "10.1109/icassp43922.2022.9746743"
arxiv: "2205.03432"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gong-2022-transformer-based-multi-aspect-multi/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gong-2022-transformer-based-multi-aspect-multi/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / Transformer-Based Multi-Aspect Multi-Granularity Non-Native English Speaker Pronunciation Assessment

# Transformer-Based Multi-Aspect Multi-Granularity Non-Native English Speaker Pronunciation Assessment

**Authors:** Yuan Gong, [Ziyi Chen](https://md.research.elamur.ai/free/en/authors/ziyi-chen/), Iek-Heng Chu, Peng Chang and [James Glass](https://md.research.elamur.ai/free/en/authors/james-glass/) · **Year:** 2022 · **Venue:** [arXiv:2205.03432](https://md.research.elamur.ai/free/en/venues/ieee-international-conference-on-acoustics-speech-and-signal-processing/) ([IEEE International Conference on Acoustics Speech and Signal Processing 2022](https://md.research.elamur.ai/free/en/venues/ieee-international-conference-on-acoustics-speech-and-signal-processing/2022/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://doi.org/10.1109/icassp43922.2022.9746743) · [PDF](https://arxiv.org/pdf/2205.03432) · [DOI 10.1109/icassp43922.2022.9746743](https://doi.org/10.1109/icassp43922.2022.9746743) · [arXiv 2205.03432](https://arxiv.org/abs/2205.03432) · [OpenAlex](https://openalex.org/W4224928163)

**Citations:** 48 citations (OpenAlex, 2026-10-14) · [11](https://md.research.elamur.ai/free/en/papers/gong-2022-transformer-based-multi-aspect-multi/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Tools that score the pronunciation of English learners usually predict a single quality, most often phoneme accuracy, at a single level of detail. Gong et al. instead train one model that predicts eight labels together. These cover phoneme accuracy, word accuracy, word stress and word total, and five utterance-level aspects: accuracy, fluency, completeness, prosody and total.

The model is GOPT, a small Transformer encoder. Its input is a sequence of Goodness of Pronunciation (GOP) features. These come from a public Kaldi TDNN-F acoustic model trained on Librispeech and are force-aligned to the known canonical transcription. Each phoneme position also receives an embedding of the expected phoneme. Five learnable [cls] tokens supply the utterance-level outputs, and every label has its own regression head, so all granularities are learned jointly.

On speechocean762, GOPT beats random forest, SVR, deep-feature transfer and a matched LSTM on most phoneme- and utterance-level metrics. The authors also give the first word-level results on this benchmark. The work matters for two reasons: it runs on fully public resources, and it shows that joint training helps each individual task rather than only bundling them together.

## Key points
- **Data:** speechocean762 has 5,000 utterances from 250 non-native speakers, split 2,500/2,500 between training and test. Five experts annotated every score. Word and utterance scores (0–10) were rescaled to match the phoneme range of 0–2.
- **Input features:**
  - GOP features are 84-dimensional: log phone posterior plus log posterior ratio over 42 phones.
  - They are projected to 24 dimensions and summed with a canonical-phoneme embedding and a positional embedding.
  - The encoder has 3 layers and width 24.
- **Phoneme results (Librispeech acoustic model):**
  - GOPT reaches MSE 0.085 and PCC 0.612.
  - SVR scores 0.160 / 0.450, RF scores 0.130 / 0.440, and the matched LSTM scores 0.089 / 0.591.
- **Utterance and word results:**
  - Utterance PCC is 0.742 for total, 0.753 for fluency and 0.760 for prosody.
  - Utterance completeness stays weak at 0.155.
  - Word-level PCC is 0.533 for accuracy, 0.291 for stress and 0.549 for total.
- **Ablations and robustness:**
  - Joint training beats single-task training.
  - Removing the canonical-phoneme embedding hurts every task.
  - Making the model wider or deeper brings no gain.
  - GOPT improves over SVR with every acoustic model tested, including the PAII internal models PAII-A and PAII-B.

## Interesting ideas
- Rather than building a hierarchy to pool phonemes into an utterance, GOPT prepends one trainable [cls] token per utterance-level label and lets attention link them to the sequence; this is a cheap, flat way to get several sentence-level outputs from a single encoder.
- The loss is averaged inside each granularity and the granularity losses are then summed with equal weight, so levels with many labels or many positions do not drown out the others.
- Word-level labels are copied down to each phoneme during training and averaged back over the word's phonemes at inference, so the unit of labelling is decoupled from the unit the model sees.
- Adding an embedding of what *should* be there (the canonical phoneme) to the measured evidence proved essential, because the same measurement means different things for different expected units.
- The comparison is kept fair: every baseline uses the same acoustic model and features, the LSTM matches GOPT in depth and width, each result is averaged over five seeds, and PCC is used because the labels lean towards high scores.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tree): gparse could prepend a few aspect tokens to predict sentence-level properties (mood, root type, clause count) alongside arcs, giving a sanity signal that is checked against the tree.
- **I2** (learn): when training joint heads for tags, arcs and sentence labels, this per-level averaging is a simple default that keeps the rare sentence-level objective from being swamped by token-level losses.
- **I3** (tokens): a character- or subword-level morphology model could learn word-level features the same way, broadcasting the word label to pieces and averaging their predictions back into one explainable word decision.
- **I4** (tree): feeding the lemma or expected UPOS as an explicit identity embedding next to contextual features could help the parser judge arcs per word type, especially with small treebanks.
- **I5** (learn): evaluation of parser or reader variants should likewise fix the shared inputs, match model capacity and report the spread across seeds, so that a claimed gain is not an artefact of setup or luck.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) — University, Cambridge, United States · 211 works · 69,000 citations
- [MIT Computer Science and Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/mit-computer-science-and-artificial-intelligence-laboratory/) — Research institute, Cambridge, United States · 69 works · 30,953 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 870 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 452 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works

## Cite

Cite as: Yuan Gong, Ziyi Chen, Iek-Heng Chu, Peng Chang and James Glass. 2022. Transformer-Based Multi-Aspect Multi-Granularity Non-Native English Speaker Pronunciation Assessment. arXiv:2205.03432.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{gong2022transformer,
  title = {{Transformer-Based Multi-Aspect Multi-Granularity Non-Native English Speaker Pronunciation Assessment}},
  author = {Yuan Gong, Ziyi Chen, Iek-Heng Chu, Peng Chang and James Glass},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746743},
  eprint = {2205.03432},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1109/icassp43922.2022.9746743}
}
```

- [← previous Semeval-2022 Task 1: CODWOE – Comparing Dictionaries and Word Embeddings](https://md.research.elamur.ai/free/en/papers/mickus-2022-codwoe-dictionaries-embeddings/)
- [next → Morphosyntactic Tagging with Pre-trained Language Models for Arabic and its Dialects](https://md.research.elamur.ai/free/en/papers/inoue-2022-morphosyntactic-tagging-pre-trained-language/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

27 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [1] Maxine Eskenazi, “An overview of spoken language technology for education,” Speech Communication, 2009. [↗](https://doi.org/10.1016/j.specom.2009.04.005)
2. [14] Joseph Tepperman and Shrikanth Narayanan, “Automatic syllable stress detection using prosodic features for pronunciation evaluation of language learners,” in ICASSP, 2005. [find](https://scholar.google.com/scholar?q=%5B14%5D+Joseph+Tepperman+and+Shrikanth+Narayanan%2C+%E2%80%9CAutomatic+syllable+stress+detection+using+prosodic+features+for+pronunciation+evaluation+of+language+learners%2C%E2%80%9D+in+ICASSP%2C+2005%2E)
3. [2] Klaus Zechner, Derrick Higgins, et al., “Automatic scoring of non-native spontaneous speech in tests of spoken english,” Speech Communication, 2009. [↗](https://doi.org/10.1016/j.specom.2009.04.009)
4. [15] Juan Pablo Arias, Nestor Becerra Yoma, and Hiram Vivanco, “Automatic intonation assessment for computer aided language learning,” Speech Communication, 2010. [↗](https://doi.org/10.1016/j.specom.2009.11.001)
5. [3] Silke M Witt, “Automatic error detection in pronunciation training: Where we are and where we need to go,” in International Symposium on Automatic Detection on Errors in Pronunciation Training, 2012. [find](https://scholar.google.com/scholar?q=%5B3%5D+Silke+M+Witt%2C+%E2%80%9CAutomatic+error+detection+in+pronunciation+training%3A+Where+we+are+and+where+we+need+to+go%2C%E2%80%9D+in+International+Symposium+on+Automatic+Detection+on+Errors+in+Pronunciation+Training%2C+20)
6. [16] Kun Li, Xixin Wu, and Helen Meng, “Intonation classification for l2 english speech using multi-distribution deep neural networks,” Computer Speech and Language, 2017. [↗](https://doi.org/10.1016/j.csl.2016.11.006)
7. [4] S.M Witt and S.J Young, “Phone-level pronunciation scoring and assessment for interactive language learning,” Speech Communication, 2000. [↗](https://doi.org/10.1016/s0167-6393(99)00044-8)
8. [17] Ashish Vaswani, Noam Shazeer, et al., “[Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/),” in Advances in Neural Information Processing Systems, 2017.
9. [5] Feng Zhang, Chao Huang, et al., “Automatic mispronunciation detection for mandarin,” in ICASSP, 2008. [find](https://scholar.google.com/scholar?q=%5B5%5D+Feng+Zhang%2C+Chao+Huang%2C+et+al%2E%2C+%E2%80%9CAutomatic+mispronunciation+detection+for+mandarin%2C%E2%80%9D+in+ICASSP%2C+2008%2E)
10. [18] Junbo Zhang, Zhiwen Zhang, et al., “[speechocean762: An open-source non-native english speech corpus for pronunciation assessment](https://md.research.elamur.ai/free/en/papers/zhang-2021-speechocean762-open-source-non-native/),” in Interspeech, 2021.
11. [6] Dean Luo, Yu Qiao, et al., “Analysis and utilization of mllr speaker adaptation technique for learners’ pronunciation evaluation,” in Interspeech, 2009. [find](https://scholar.google.com/scholar?q=%5B6%5D+Dean+Luo%2C+Yu+Qiao%2C+et+al%2E%2C+%E2%80%9CAnalysis+and+utilization+of+mllr+speaker+adaptation+technique+for+learners%E2%80%99+pronunciation+evaluation%2C%E2%80%9D+in+Interspeech%2C+2009%2E)
12. [19] Binghuai Lin, Liyuan Wang, Xiaoli Feng, and Jinsong Zhang, “Automatic scoring at multi-granularity for l2 pronunciation.,” in Interspeech, 2020. [find](https://scholar.google.com/scholar?q=%5B19%5D+Binghuai+Lin%2C+Liyuan+Wang%2C+Xiaoli+Feng%2C+and+Jinsong+Zhang%2C+%E2%80%9CAutomatic+scoring+at+multi%2Dgranularity+for+l2+pronunciation%2E%2C%E2%80%9D+in+Interspeech%2C+2020%2E)
13. [7] Yow-Bang Wang and Lin-Shan Lee, “Improved approaches of modeling and detecting error patterns with empirical analysis for computer-aided pronunciation training,” in ICASSP, 2012. [find](https://scholar.google.com/scholar?q=%5B7%5D+Yow%2DBang+Wang+and+Lin%2DShan+Lee%2C+%E2%80%9CImproved+approaches+of+modeling+and+detecting+error+patterns+with+empirical+analysis+for+computer%2Daided+pronunciation+training%2C%E2%80%9D+in+ICASSP%2C+2012%2E)
14. [20] Xiaoshuo Xu, Yueteng Kang, Songjun Cao, Binghuai Lin, et al., “Explore wav2vec 2.0 for mispronunciation detection,” Interspeech, 2021. [find](https://scholar.google.com/scholar?q=%5B20%5D+Xiaoshuo+Xu%2C+Yueteng+Kang%2C+Songjun+Cao%2C+Binghuai+Lin%2C+et+al%2E%2C+%E2%80%9CExplore+wav2vec+2%2E0+for+mispronunciation+detection%2C%E2%80%9D+Interspeech%2C+2021%2E)
15. [8] Wenping Hu, Yao Qian, et al., “Improved mispronunciation detection with deep neural network trained acoustic models and transfer learning based logistic regression classifiers,” Speech Communication, 2015. [↗](https://doi.org/10.1016/j.specom.2014.12.008)
16. [9] Jiatong Shi, Nan Huo, and Qin Jin, “Context-aware goodness of pronunciation for computer-assisted pronunciation training,” in Interspeech, 2020. [find](https://scholar.google.com/scholar?q=%5B9%5D+Jiatong+Shi%2C+Nan+Huo%2C+and+Qin+Jin%2C+%E2%80%9CContext%2Daware+goodness+of+pronunciation+for+computer%2Dassisted+pronunciation+training%2C%E2%80%9D+in+Interspeech%2C+2020%2E)
17. [10] Joost van Doremalen, C Cucchiarini, and H Strik, “Using non-native error patterns to improve pronunciation verification,” in Interspeech, 2010. [find](https://scholar.google.com/scholar?q=%5B10%5D+Joost+van+Doremalen%2C+C+Cucchiarini%2C+and+H+Strik%2C+%E2%80%9CUsing+non%2Dnative+error+patterns+to+improve+pronunciation+verification%2C%E2%80%9D+in+Interspeech%2C+2010%2E)
18. [11] Catia Cucchiarini, Helmer Strik, and LWJ Boves, “Quantitative assessment of second language learners’ fluency: an automatic approach,” in ICSLP, 1998. [find](https://scholar.google.com/scholar?q=%5B11%5D+Catia+Cucchiarini%2C+Helmer+Strik%2C+and+LWJ+Boves%2C+%E2%80%9CQuantitative+assessment+of+second+language+learners%E2%80%99+fluency%3A+an+automatic+approach%2C%E2%80%9D+in+ICSLP%2C+1998%2E)
19. [12] Catia Cucchiarini, Helmer Strik, and Lou Boves, “Quantitative assessment of second language learners’ fluency by means of automatic speech recognition technology,” The Journal of the Acoustical Society of America, 2000. [find](https://scholar.google.com/scholar?q=%5B12%5D+Catia+Cucchiarini%2C+Helmer+Strik%2C+and+Lou+Boves%2C+%E2%80%9CQuantitative+assessment+of+second+language+learners%E2%80%99+fluency+by+means+of+automatic+speech+recognition+technology%2C%E2%80%9D+The+Journal+of+the+Acoustical+S)
20. [13] Paul Christopher Bagshaw, Automatic prosodic analysis for computer aided pronunciation teaching, Ph.D. thesis, 1994. [find](https://scholar.google.com/scholar?q=%5B13%5D+Paul+Christopher+Bagshaw%2C+Automatic+prosodic+analysis+for+computer+aided+pronunciation+teaching%2C+Ph%2ED%2E+thesis%2C+1994%2E)
21. [21] Binghuai Lin and Liyuan Wang, “Deep feature transfer learning for automatic pronunciation assessment,” Interspeech, 2021. [find](https://scholar.google.com/scholar?q=%5B21%5D+Binghuai+Lin+and+Liyuan+Wang%2C+%E2%80%9CDeep+feature+transfer+learning+for+automatic+pronunciation+assessment%2C%E2%80%9D+Interspeech%2C+2021%2E)
22. [22] Tobias Cincarek, Rainer Gruhn, et al., “Automatic pronunciation scoring of words and sentences independent from the non-native’s first language,” Computer Speech and Language, 2009. [find](https://scholar.google.com/scholar?q=%5B22%5D+Tobias+Cincarek%2C+Rainer+Gruhn%2C+et+al%2E%2C+%E2%80%9CAutomatic+pronunciation+scoring+of+words+and+sentences+independent+from+the+non%2Dnative%E2%80%99s+first+language%2C%E2%80%9D+Computer+Speech+and+Language%2C+2009%2E)
23. [23] Jacob Devlin, Ming-Wei Chang, et al., “[Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/),” ACL, 2018.
24. [24] Vassil Panayotov, Guoguo Chen, et al., “Librispeech: an asr corpus based on public domain audio books,” in ICASSP, 2015. [find](https://scholar.google.com/scholar?q=%5B24%5D+Vassil+Panayotov%2C+Guoguo+Chen%2C+et+al%2E%2C+%E2%80%9CLibrispeech%3A+an+asr+corpus+based+on+public+domain+audio+books%2C%E2%80%9D+in+ICASSP%2C+2015%2E)
25. [25] Ming Tu, Anna Grabek, et al., “Investigating the role of l1 in automatic pronunciation evaluation of l2 speech,” in Interspeech, 2018. [find](https://scholar.google.com/scholar?q=%5B25%5D+Ming+Tu%2C+Anna+Grabek%2C+et+al%2E%2C+%E2%80%9CInvestigating+the+role+of+l1+in+automatic+pronunciation+evaluation+of+l2+speech%2C%E2%80%9D+in+Interspeech%2C+2018%2E)
26. [26] François Hernandez, Vincent Nguyen, et al., “Ted-lium 3: twice as much data and corpus repartition for experiments on speaker adaptation,” in SPECOM, 2018. [find](https://scholar.google.com/scholar?q=%5B26%5D+Franc%CC%A7ois+Hernandez%2C+Vincent+Nguyen%2C+et+al%2E%2C+%E2%80%9CTed%2Dlium+3%3A+twice+as+much+data+and+corpus+repartition+for+experiments+on+speaker+adaptation%2C%E2%80%9D+in+SPECOM%2C+2018%2E)
27. [27] Mitchell P Marcus, Mary Ann Marcinkiewicz, et al., “[Building a large annotated corpus of english: the penn treebank](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/),” Computational Linguistics, 1993. [↗](https://doi.org/10.21236/ada273556)

*Updated 1 November 2026*