---
title: "Proficiency assessment of L2 spoken English using wav2vec 2.0"
description: "The paper tackles automatic grading of spoken English produced by second-language learners. 45 references, cited by 11 papers here."
authors: ["Stefano Banno", "Marco Matassoni"]
year: 2022
venue: "arXiv:2210.13168"
source: "https://arxiv.org/abs/2210.13168"
doi: "10.48550/arxiv.2210.13168"
arxiv: "2210.13168"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/banno-2022-proficiency-assessment-l2-spoken-english/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/banno-2022-proficiency-assessment-l2-spoken-english/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/) / Proficiency assessment of L2 spoken English using wav2vec 2.0

# Proficiency assessment of L2 spoken English using wav2vec 2.0

**Authors:** [Stefano Banno](https://md.research.elamur.ai/free/en/authors/stefano-banno/) and [Marco Matassoni](https://md.research.elamur.ai/free/en/authors/marco-matassoni/) · **Year:** 2022 · **Venue:** [arXiv:2210.13168](https://md.research.elamur.ai/free/en/venues/ieee-spoken-language-technology-workshop-slt/) ([SLT 2022](https://md.research.elamur.ai/free/en/venues/ieee-spoken-language-technology-workshop-slt/2022/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2210.13168) · [PDF](https://arxiv.org/pdf/2210.13168) · [DOI 10.48550/arxiv.2210.13168](https://doi.org/10.48550/arxiv.2210.13168) · [arXiv 2210.13168](https://arxiv.org/abs/2210.13168) · [OpenAlex](https://openalex.org/W4319862401)

**Citations:** 24 citations (OpenAlex, 2026-10-14) · [11](https://md.research.elamur.ai/free/en/papers/banno-2022-proficiency-assessment-l2-spoken-english/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper tackles automatic grading of spoken English produced by second-language learners. Most earlier graders rely either on hand-engineered features, which may miss useful cues, or on ASR transcripts, which carry recognition errors and lose prosodic information. The authors instead fine-tune the self-supervised speech model wav2vec 2.0 directly on raw audio to predict proficiency. They test it on two small datasets. The first is the public ICNALE monologue set, where the task is classifying CEFR level plus a native-speaker class. The second is the non-public TLT-school corpus of Italian schoolchildren, where they regress a holistic score and six analytic subscores. The baseline is a BERT model that reads manual transcripts, and on TLT-school also ASR transcripts. The audio-based approach significantly beats this text-based baseline on both corpora, despite limited training data. This matters because it suggests a single pretrained speech encoder can capture both delivery (fluency, pronunciation) and language-use aspects without a transcription step. That makes it attractive for noisy settings like children's spontaneous non-native speech.

## Key points
- ICNALE spoken monologues: 4332 answers, each 36–69 seconds, on two opinion prompts (part-time jobs; smoking in restaurants). They are split 3898 / 217 / 217 and labelled A2, B1 1, B1 2, B2 or native.
- TLT-school (Trentino, Italy), B1 English spoken subset: 494 answers to 7 small-talk questions. The holistic score is 0–12, obtained by adding six subscores, each rated 0–2, which cover how relevant the answer is, accuracy of form, vocabulary richness and complexity, quality of pronunciation, fluency, and how effectively the speaker communicates.
- The ASR used for the BERT baseline has a WER of 35.9% on all spoken test data and 41.13% on the B1 subset. Manual transcripts keep hesitations and word fragments.
- A Friedman test followed by Nemenyi post-hoc comparisons showed that most subscore pairs differ significantly. Four pairs did not differ: correctness–lexis, correctness–pronunciation, pronunciation–communicative effectiveness, and fluency–communicative effectiveness. The authors note possible rater overlap and halo effect.
- wav2vec 2.0 graders significantly outperform the BERT baseline, whether it reads manual or ASR transcripts.

## Interesting ideas
- Before training one grader per sub-aspect, the authors test statistically whether the human subscores really measure distinct things, using rank-based tests since normality fails.
- Disfluency marks (hesitations, word fragments) are deliberately left in transcripts as weak stand-ins for fluency and pronunciation, instead of being cleaned away.
- The comparison shows how much a pipeline loses when an upstream lossy step (ASR at ~41% WER) feeds a text model, against a model that sees the original signal.
- Noisy items such as mixed-language or off-topic answers were kept rather than filtered, so evaluation reflects realistic conditions.
- Native speakers are added as an extra class alongside learner levels, giving an upper anchor for the scale.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Run the same Friedman/Nemenyi-style check on our multi-axis evaluation scores (e.g., parse vs. frame vs. answer quality) to confirm the axes are not redundant before reporting them separately.
- **I2** (tokens): Have the tokenizer keep and tag disfluency or fragment tokens rather than drop them, so downstream stages can treat them explicitly and explain their effect.
- **I3** (tree): When measuring parser quality on noisy input, report results on both gold and automatically produced input to quantify error propagation from earlier stages.
- **I4** (answer): Keep messy or off-topic items in the school-test and fable QA sets, and score them separately, so the readers are judged on realistic inputs and not only clean ones.
- **I5** (learn): Include clearly native or gold-quality texts as a reference class in evaluations, serving as a positive control next to the negative controls.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Trento](https://md.research.elamur.ai/free/en/universities/university-of-trento/) — University, Trento, Italy · 85 works · 15,524 citations
- [Fondazione Bruno Kessler](https://md.research.elamur.ai/free/en/labs/fondazione-bruno-kessler/) — Research institute, Trento, Italy · 35 works · 43,945 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Developmental and Educational Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/developmental-and-educational-psychology/) · 499 works

🧭 Topics (2): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 480 works

## Cite

Cite as: Stefano Banno and Marco Matassoni. 2022. Proficiency assessment of L2 spoken English using wav2vec 2.0. arXiv:2210.13168.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{banno2022proficiency,
  title = {{Proficiency assessment of L2 spoken English using wav2vec 2.0}},
  author = {Stefano Banno and Marco Matassoni},
  year = {2022},
  doi = {10.48550/arxiv.2210.13168},
  eprint = {2210.13168},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2210.13168}
}
```

- [← previous Data Augmentation using Transformers and Similarity Measures for Improving Arabic Text Classification](https://md.research.elamur.ai/free/en/papers/refai-2022-data-augmentation-using-transformers-similarity/)
- [next → A Dual-Channel Framework for Sarcasm Recognition by Detecting Sentiment Conflict](https://md.research.elamur.ai/free/en/papers/liu-2022-dual-channel-framework-sarcasm-recognition/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/)

## References

45 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [1] M. Zhang, “Contrasting automated and human scoring of essays,” R&D Connections, , no. 21, pp. 1–11, 2013. [find](https://scholar.google.com/scholar?q=%5B1%5D+M%2E+Zhang%2C+%E2%80%9CContrasting+automated+and+human+scoring+of+essays%2C%E2%80%9D+R%26D+Connections%2C+%2C+no%2E+21%2C+pp%2E+1%E2%80%9311%2C+2013%2E)
2. [2] N. Iwashita, A. Brown, T. McNamara, and S. O’Hagan, “Assessed levels of second language speaking proficiency: How distinct?,” Applied Linguistics, vol. 29, no. 1, pp. 24–49, 2008. [↗](https://doi.org/10.1093/applin/amm017)
3. [3] N.H. De Jong, M.P. Steinel, A.F. Florijn, R. Schoonen, and J.H. Hulstijn, “Facets of speaking proficiency,” Studies in Second Language Acquisition, , no. 34, pp. 5–34, 2012. [↗](https://doi.org/10.1017/s0272263111000489)
4. [4] H. Strik and C. Cucchiarini, “Automatic assessment of second language learners’ fluency,” in Proc. International Congress of Phonetic Sciences (ICPhS) 1999, 1999. [find](https://scholar.google.com/scholar?q=%5B4%5D+H%2E+Strik+and+C%2E+Cucchiarini%2C+%E2%80%9CAutomatic+assessment+of+second+language+learners%E2%80%99+fluency%2C%E2%80%9D+in+Proc%2E+International+Congress+of+Phonetic+Sciences+%28ICPhS%29+1999%2C+1999%2E)
5. [5] L. Chen, K. Evanini, and X. Sun, “Assessment of non-native speech using vowel space characteristics,” in Proc. 2010 IEEE Spoken Language Technology Workshop, 2010, pp. 139–144. [find](https://scholar.google.com/scholar?q=%5B5%5D+L%2E+Chen%2C+K%2E+Evanini%2C+and+X%2E+Sun%2C+%E2%80%9CAssessment+of+non%2Dnative+speech+using+vowel+space+characteristics%2C%E2%80%9D+in+Proc%2E+2010+IEEE+Spoken+Language+Technology+Workshop%2C+2010%2C+pp%2E+139%E2%80%93144%2E)
6. [6] E. Coutinho, F. Hönig, Y. Zhang, S. Hantke, A. Batliner, E. Nöth, and B. Schuller, “[Assessing the prosody of non-native speakers of English: Measures and feature sets](https://md.research.elamur.ai/free/en/papers/coutinho-2016-assessing-prosody-non-native-speakers/),” in Proc. 10th International Conference on Language Resources and Evaluation (LREC’16), 2016.
7. [7] S. Bhat and S. Yoon, “Automatic assessment of syntactic complexity for spontaneous speech scoring,” Speech Communication, vol. 67, pp. 42–57, 2015. [↗](https://doi.org/10.1016/j.specom.2014.09.005)
8. [12] L. Chen, J. Tao, S. Ghaffarzadegan, and Y. Qian, “End-to-end neural network based automated speech scoring,” in Proc. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 6234–6238. [find](https://scholar.google.com/scholar?q=%5B12%5D+L%2E+Chen%2C+J%2E+Tao%2C+S%2E+Ghaffarzadegan%2C+and+Y%2E+Qian%2C+%E2%80%9CEnd%2Dto%2Dend+neural+network+based+automated+speech+scoring%2C%E2%80%9D+in+Proc%2E+2018+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing)
9. [13] K. Takai, P. Heracleous, K. Yasuda, and A. Yoneyama, “Deep learning-based automatic pronunciation assessment for second language learners,” in Proc. International Conference on Human-Computer Interaction, 2020, pp. 338–342. [find](https://scholar.google.com/scholar?q=%5B13%5D+K%2E+Takai%2C+P%2E+Heracleous%2C+K%2E+Yasuda%2C+and+A%2E+Yoneyama%2C+%E2%80%9CDeep+learning%2Dbased+automatic+pronunciation+assessment+for+second+language+learners%2C%E2%80%9D+in+Proc%2E+International+Conference+on+Human%2DComputer+Int)
10. [14] S. Cheng, Z. Liu, L. Li, Z. Tang, D. Wang, and T.F. Zheng, “ASR-Free pronunciation assessment,” in Proc. Interspeech 2020, 2020, pp. 3047–3051. [find](https://scholar.google.com/scholar?q=%5B14%5D+S%2E+Cheng%2C+Z%2E+Liu%2C+L%2E+Li%2C+Z%2E+Tang%2C+D%2E+Wang%2C+and+T%2EF%2E+Zheng%2C+%E2%80%9CASR%2DFree+pronunciation+assessment%2C%E2%80%9D+in+Proc%2E+Interspeech+2020%2C+2020%2C+pp%2E+3047%E2%80%933051%2E)
11. [15] K. Kyriakopoulos, K.M. Knill, and M.J.F. Gales, “A deep learning approach to assessing non-native pronunciation of English using phone distances,” in Proc. Interspeech 2018, 2018, pp. 1626–1630. [find](https://scholar.google.com/scholar?q=%5B15%5D+K%2E+Kyriakopoulos%2C+K%2EM%2E+Knill%2C+and+M%2EJ%2EF%2E+Gales%2C+%E2%80%9CA+deep+learning+approach+to+assessing+non%2Dnative+pronunciation+of+English+using+phone+distances%2C%E2%80%9D+in+Proc%2E+Interspeech+2018%2C+2018%2C+pp%2E+1626%E2%80%931630%2E)
12. [16] K. Kyriakopoulos, K.M. Knill, and M.J.F. Gales, “A deep learning approach to automatic characterisation of rhythm in non-native English speech,” in Proc. Interspeech 2019, 2019, pp. 1836–1840. [find](https://scholar.google.com/scholar?q=%5B16%5D+K%2E+Kyriakopoulos%2C+K%2EM%2E+Knill%2C+and+M%2EJ%2EF%2E+Gales%2C+%E2%80%9CA+deep+learning+approach+to+automatic+characterisation+of+rhythm+in+non%2Dnative+English+speech%2C%E2%80%9D+in+Proc%2E+Interspeech+2019%2C+2019%2C+pp%2E+1836%E2%80%931840%2E)
13. [17] X. Wang, K. Evanini, Y. Qian, and M. Mulholland, “Automated scoring of spontaneous speech from young learners of english using transformers,” in 2021 IEEE Spoken Language Technology Workshop (SLT), 2021, pp. 705–712. [find](https://scholar.google.com/scholar?q=%5B17%5D+X%2E+Wang%2C+K%2E+Evanini%2C+Y%2E+Qian%2C+and+M%2E+Mulholland%2C+%E2%80%9CAutomated+scoring+of+spontaneous+speech+from+young+learners+of+english+using+transformers%2C%E2%80%9D+in+2021+IEEE+Spoken+Language+Technology+Workshop+%28SLT)
14. [18] V. Raina, M.J.F. Gales, and K.M. Knill, “Universal adversarial attacks on spoken language assessment systems,” in Proc. Interspeech 2020, 2020, pp. 3855–3859. [find](https://scholar.google.com/scholar?q=%5B18%5D+V%2E+Raina%2C+M%2EJ%2EF%2E+Gales%2C+and+K%2EM%2E+Knill%2C+%E2%80%9CUniversal+adversarial+attacks+on+spoken+language+assessment+systems%2C%E2%80%9D+in+Proc%2E+Interspeech+2020%2C+2020%2C+pp%2E+3855%E2%80%933859%2E)
15. [8] P. Müller, F. De Wet, C. Van Der Walt, and T. Niesler, “Automatically assessing the oral proficiency of proficient L2 speakers.,” in Proc. Workshop on Speech and Language Technology for Education (SLaTE), 2009, pp. 29–32. [find](https://scholar.google.com/scholar?q=%5B8%5D+P%2E+Mu%CC%88ller%2C+F%2E+De+Wet%2C+C%2E+Van+Der+Walt%2C+and+T%2E+Niesler%2C+%E2%80%9CAutomatically+assessing+the+oral+proficiency+of+proficient+L2+speakers%2E%2C%E2%80%9D+in+Proc%2E+Workshop+on+Speech+and+Language+Technology+for+Education)
16. [19] S. Bannò, B. Balusu, M. J. F. Gales, K. M. Knill, and K. Kyriakopoulos, “View-specific assessment of L2 spoken English,” in Proc. Interspeech 2022, 2022, pp. 4471–4475. [find](https://scholar.google.com/scholar?q=%5B19%5D+S%2E+Banno%CC%80%2C+B%2E+Balusu%2C+M%2E+J%2E+F%2E+Gales%2C+K%2E+M%2E+Knill%2C+and+K%2E+Kyriakopoulos%2C+%E2%80%9CView%2Dspecific+assessment+of+L2+spoken+English%2C%E2%80%9D+in+Proc%2E+Interspeech+2022%2C+2022%2C+pp%2E+4471%E2%80%934475%2E)
17. [9] S. Crossley and D. McNamara, “Applications of text analysis tools for spoken response grading,” Language Learning & Technology, vol. 17, no. 2, pp. 171–192, 2013. [↗](https://doi.org/10.64152/10125/44329)
18. [20] Y. Lu, S. Bannò, and M.J.F. Gales, “On assessing and developing spoken ’grammatical error correction’ systems,” in Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), 2022, pp. 51–60. [find](https://scholar.google.com/scholar?q=%5B20%5D+Y%2E+Lu%2C+S%2E+Banno%CC%80%2C+and+M%2EJ%2EF%2E+Gales%2C+%E2%80%9COn+assessing+and+developing+spoken+%E2%80%99grammatical+error+correction%E2%80%99+systems%2C%E2%80%9D+in+Proceedings+of+the+17th+Workshop+on+Innovative+Use+of+NLP+for+Building+Educatio)
19. [10] Y. Wang, M.J.F. Gales, K.M. Knill, K. Kyriakopoulos, A. Malinin, R.C. van Dalen, and M. Rashid, “Towards automatic assessment of spontaneous spoken English,” Speech Communication, vol. 104, pp. 47–56, 2018. [↗](https://doi.org/10.1016/j.specom.2018.09.002)
20. [21] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “[wav2vec 2.0: A framework for self-supervised learning of speech representations](https://md.research.elamur.ai/free/en/papers/baevski-2020-wav2vec-2-0-framework-self/),” in NeurIPS 2020, 2020, pp. 1–12.
21. [11] Z. Liu, G. Xu, T. Liu, W. Fu, Y. Qi, W. Ding, Y. Song, C. Guo, C. Kong, S. Yang, et al., “Dolphin: a spoken language proficiency assessment system for elementary education,” in Proc. The Web Conference 2020, 2020, pp. 2641–2647. [find](https://scholar.google.com/scholar?q=%5B11%5D+Z%2E+Liu%2C+G%2E+Xu%2C+T%2E+Liu%2C+W%2E+Fu%2C+Y%2E+Qi%2C+W%2E+Ding%2C+Y%2E+Song%2C+C%2E+Guo%2C+C%2E+Kong%2C+S%2E+Yang%2C+et+al%2E%2C+%E2%80%9CDolphin%3A+a+spoken+language+proficiency+assessment+system+for+elementary+education%2C%E2%80%9D+in+Proc%2E+The+Web+Conf)
22. [22] W.N. Hsu, A. Sriram, A. Baevski, T. Likhomanenko, Q. Xu, V. Pratap, J. Kahn, A. Lee, R. Collobert, G. Synnaeve, and M. Auli, “Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training,” in Proc. Interspeech 2021, 2021, pp. 721–725. [find](https://scholar.google.com/scholar?q=%5B22%5D+W%2EN%2E+Hsu%2C+A%2E+Sriram%2C+A%2E+Baevski%2C+T%2E+Likhomanenko%2C+Q%2E+Xu%2C+V%2E+Pratap%2C+J%2E+Kahn%2C+A%2E+Lee%2C+R%2E+Collobert%2C+G%2E+Synnaeve%2C+and+M%2E+Auli%2C+%E2%80%9CRobust+wav2vec+2%2E0%3A+Analyzing+domain+shift+in+self%2Dsupervised+pre%2Dtra)
23. [23] S.-W Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y.Y. Lin, A.T. Liu, J. Shi, X. Chang, G.-T. Lin, T.-H. Huang, W.-C. Tseng, K.-T. Lee, D.-R. Liu, Z. Huang, S. Dong, S.-W. Li, S. Watanabe, A. Mohamed, and H.-Y. Lee, “SUPERB: Speech Processing Universal PERformance Benchmark,” in Proc. Interspeech 2021, 2021, pp. 1194–1198. [↗](https://doi.org/10.48550/arxiv.2105.01051)
24. [24] J. Shah, Y.K. Singla, C. Chen, and R. Ratn Shah, “What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structure,” arXiv e-prints, p. arXiv:2101.00387, 2021. [↗](https://arxiv.org/abs/2101.00387)
25. [25] L. Peng, K. Fu, B. Lin, D. Ke, and J. Zhan, “A Study on Fine-Tuning wav2vec2.0 Model for the Task of Mispronunciation Detection and Diagnosis,” in Proc. Interspeech 2021, 2021, pp. 4448–4452. [find](https://scholar.google.com/scholar?q=%5B25%5D+L%2E+Peng%2C+K%2E+Fu%2C+B%2E+Lin%2C+D%2E+Ke%2C+and+J%2E+Zhan%2C+%E2%80%9CA+Study+on+Fine%2DTuning+wav2vec2%2E0+Model+for+the+Task+of+Mispronunciation+Detection+and+Diagnosis%2C%E2%80%9D+in+Proc%2E+Interspeech+2021%2C+2021%2C+pp%2E+4448%E2%80%934452%2E)
26. [26] M. Wu, K. Li, W.-K. Leung, and H. Meng, “Transformer Based End-to-End Mispronunciation Detection and Diagnosis,” in Proc. Interspeech 2021, 2021, pp. 3954–3958. [find](https://scholar.google.com/scholar?q=%5B26%5D+M%2E+Wu%2C+K%2E+Li%2C+W%2E%2DK%2E+Leung%2C+and+H%2E+Meng%2C+%E2%80%9CTransformer+Based+End%2Dto%2DEnd+Mispronunciation+Detection+and+Diagnosis%2C%E2%80%9D+in+Proc%2E+Interspeech+2021%2C+2021%2C+pp%2E+3954%E2%80%933958%2E)
27. [27] X. Xu, Y. Kang, S. Cao, B. Lin, and L. Ma, “Explore wav2vec 2.0 for Mispronunciation Detection,” in Proc. Interspeech 2021, 2021, pp. 4428–4432. [find](https://scholar.google.com/scholar?q=%5B27%5D+X%2E+Xu%2C+Y%2E+Kang%2C+S%2E+Cao%2C+B%2E+Lin%2C+and+L%2E+Ma%2C+%E2%80%9CExplore+wav2vec+2%2E0+for+Mispronunciation+Detection%2C%E2%80%9D+in+Proc%2E+Interspeech+2021%2C+2021%2C+pp%2E+4428%E2%80%934432%2E)
28. [28] E. Kim, J.-J. Jeon, H. Seo, and H. Kim, “Automatic pronunciation assessment using self-supervised speech representation learning,” in Proc. Interspeech 2022, 2022, pp. 1411–1415. [find](https://scholar.google.com/scholar?q=%5B28%5D+E%2E+Kim%2C+J%2E%2DJ%2E+Jeon%2C+H%2E+Seo%2C+and+H%2E+Kim%2C+%E2%80%9CAutomatic+pronunciation+assessment+using+self%2Dsupervised+speech+representation+learning%2C%E2%80%9D+in+Proc%2E+Interspeech+2022%2C+2022%2C+pp%2E+1411%E2%80%931415%2E)
29. [29] J. Devlin, M. Chang, L. Kenton, and K. Toutanova, “[BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/),” arXiv e-prints, p. arXiv:1810.04805, 2018. [↗](https://arxiv.org/abs/1810.04805)
30. [30] S. Ishikawa, “A new horizon in learner corpus studies: The aim of the ICNALE project,” in Corpora and language technologies in teaching, learning and research, G. Weir, S. Ishikawa, and K. Poonpon, Eds., pp. 3–11. University of Strathclyde Press, 2011. [find](https://scholar.google.com/scholar?q=%5B30%5D+S%2E+Ishikawa%2C+%E2%80%9CA+new+horizon+in+learner+corpus+studies%3A+The+aim+of+the+ICNALE+project%2C%E2%80%9D+in+Corpora+and+language+technologies+in+teaching%2C+learning+and+research%2C+G%2E+Weir%2C+S%2E+Ishikawa%2C+and+K%2E+Poonpo)
31. [31] Council of Europe, Common European Framework of Reference for Languages: Learning, Teaching, Assessment, Cambridge University Press, Cambridge, 2001. [find](https://scholar.google.com/scholar?q=%5B31%5D+Council+of+Europe%2C+Common+European+Framework+of+Reference+for+Languages%3A+Learning%2C+Teaching%2C+Assessment%2C+Cambridge+University+Press%2C+Cambridge%2C+2001%2E)
32. [32] Z. Zhou, S. Vajjala, and S.V. Mirnezami, “Experiments on Non-native Speech Assessment and its Consistency,” in NLP4CALL 2019, 2019, pp. 86–92. [find](https://scholar.google.com/scholar?q=%5B32%5D+Z%2E+Zhou%2C+S%2E+Vajjala%2C+and+S%2EV%2E+Mirnezami%2C+%E2%80%9CExperiments+on+Non%2Dnative+Speech+Assessment+and+its+Consistency%2C%E2%80%9D+in+NLP4CALL+2019%2C+2019%2C+pp%2E+86%E2%80%9392%2E)
33. [33] S. Bannò and M. Matassoni, “[Cross-corpora experiments of automatic proficiency assessment and error detection for spoken English](https://md.research.elamur.ai/free/en/papers/banno-2022-cross-corpora-experiments-automatic-proficiency/),” in Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), 2022, pp. 82–91.
34. [34] R. Gretter, M. Matassoni, S. Bannò, and D. Falavigna, “TLT-school: a corpus of non native children speech,” in Proceedings of the 12th Language Resources and Evaluation Conference, 2020. [↗](https://doi.org/10.48550/arxiv.2001.08051)
35. [35] R. Gretter, M. Matassoni, K. Allgaier, S. Tchistiakova, and D. Falavigna, “Automatic assessment of spoken language proficiency of non-native children,” in IEEE International Conference on Acoustics, Speech and Signal Processing, 2019. [find](https://scholar.google.com/scholar?q=%5B35%5D+R%2E+Gretter%2C+M%2E+Matassoni%2C+K%2E+Allgaier%2C+S%2E+Tchistiakova%2C+and+D%2E+Falavigna%2C+%E2%80%9CAutomatic+assessment+of+spoken+language+proficiency+of+non%2Dnative+children%2C%E2%80%9D+in+IEEE+International+Conference+on+Acousti)
36. [36] M. Friedman, “The use of ranks to avoid the assumption of normality implicit in the analysis of variance,” Journal of the American Statistical Association, vol. 32, no. 200, pp. 675–701, 1937. [↗](https://doi.org/10.1080/01621459.1937.10503522)
37. [37] P.B. Nemenyi, Distribution-free multiple comparisons, Ph.D. thesis, Princeton University, 1963. [find](https://scholar.google.com/scholar?q=%5B37%5D+P%2EB%2E+Nemenyi%2C+Distribution%2Dfree+multiple+comparisons%2C+Ph%2ED%2E+thesis%2C+Princeton+University%2C+1963%2E)
38. [38] J.M. Levis, Intelligibility, Oral Communication, and the Teaching of Pronunciation, Cambridge University Press, Cambridge, 2018. [find](https://scholar.google.com/scholar?q=%5B38%5D+J%2EM%2E+Levis%2C+Intelligibility%2C+Oral+Communication%2C+and+the+Teaching+of+Pronunciation%2C+Cambridge+University+Press%2C+Cambridge%2C+2018%2E)
39. [39] M.C. Pennington and P. Rogerson-Revell, English Pronunciation Teaching and Research: Contemporary Perspectives, Palgrave Macmillan, London, 2019. [find](https://scholar.google.com/scholar?q=%5B39%5D+M%2EC%2E+Pennington+and+P%2E+Rogerson%2DRevell%2C+English+Pronunciation+Teaching+and+Research%3A+Contemporary+Perspectives%2C+Palgrave+Macmillan%2C+London%2C+2019%2E)
40. [40] E.-L. Low, Pronunciation for English as an International Language: from Research to Practice, Routledge, London, 2014. [find](https://scholar.google.com/scholar?q=%5B40%5D+E%2E%2DL%2E+Low%2C+Pronunciation+for+English+as+an+International+Language%3A+from+Research+to+Practice%2C+Routledge%2C+London%2C+2014%2E)
41. [41] C.M. Myford and E.W. Wolfe, “Detecting and measuring rater effects using many-facet rasch measurement: Part i,” Journal of Applied Measurement, vol. 4, no. 4, pp. 386–422, 2003. [find](https://scholar.google.com/scholar?q=%5B41%5D+C%2EM%2E+Myford+and+E%2EW%2E+Wolfe%2C+%E2%80%9CDetecting+and+measuring+rater+effects+using+many%2Dfacet+rasch+measurement%3A+Part+i%2C%E2%80%9D+Journal+of+Applied+Measurement%2C+vol%2E+4%2C+no%2E+4%2C+pp%2E+386%E2%80%93422%2C+2003%2E)
42. [42] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush, “[Transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/),” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020, pp. 38–45.
43. [43] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in ICLR 2017, 2017. [↗](https://doi.org/10.48550/arxiv.1711.05101)
44. [44] D.P. Kingma and J. Ba, “[Adam: a method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/),” in International Conference on Learning Representations, 2014. [↗](https://doi.org/10.48550/arxiv.1412.6980)
45. [45] Y. Qian, R. Ubale, M. Mulholland, K. Evanini, and X. Wang, “A prompt-aware neural network approach to content-based scoring of non-native spontaneous speech,” in 2018 IEEE Spoken Language Technology Workshop (SLT), 2018, pp. 979–986. [find](https://scholar.google.com/scholar?q=%5B45%5D+Y%2E+Qian%2C+R%2E+Ubale%2C+M%2E+Mulholland%2C+K%2E+Evanini%2C+and+X%2E+Wang%2C+%E2%80%9CA+prompt%2Daware+neural+network+approach+to+content%2Dbased+scoring+of+non%2Dnative+spontaneous+speech%2C%E2%80%9D+in+2018+IEEE+Spoken+Language+Technol)

*Updated 1 November 2026*