---
title: "Automated Assessment of Spoken Modern Standard Arabic (2009)"
description: "The authors argue that a short, tightly controlled automatic test can replace costly interviews in many… 21 references, cited by 1 paper here."
authors: ["Jian Cheng", "Jared Bernstein", "Ulrike Pado", "Masanori Suzuki"]
year: 2009
venue: "Proceedings of the Fourth Workshop on Innovative Use of NLP for Building Educational Applications"
source: "https://aclanthology.org/W09-2101/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cheng-2009-automated-assessment-spoken-modern-standard/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cheng-2009-automated-assessment-spoken-modern-standard/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2009](https://md.research.elamur.ai/free/en/papers/2009/) / Automated Assessment of Spoken Modern Standard Arabic

# Automated Assessment of Spoken Modern Standard Arabic

**Authors:** [Jian Cheng](https://md.research.elamur.ai/free/en/authors/jian-cheng/), [Jared Bernstein](https://md.research.elamur.ai/free/en/authors/jared-bernstein/), [Ulrike Pado](https://md.research.elamur.ai/free/en/authors/ulrike-pado/) and Masanori Suzuki · **Year:** 2009 · **Venue:** [Proceedings of the Fourth Workshop on Innovative Use of NLP for Building Educational Applications](https://md.research.elamur.ai/free/en/venues/bea/) ([BEA 2009](https://md.research.elamur.ai/free/en/venues/bea/2009/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W09-2101/) · [PDF](https://aclanthology.org/W09-2101.pdf) · [OpenAlex](https://openalex.org/W2023330479)

**Citations:** 33 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Interview-based oral proficiency exams such as the ILR OPI need trained examiners and up to 45 minutes per candidate, so repeated testing is expensive. The authors built and validated the Versant Arabic Test (VAT), a fully automatic test of spoken Modern Standard Arabic (MSA) taken by phone or computer in about 17 minutes. VAT does not try to simulate conversation. Instead it measures "facility": understanding spoken MSA and answering intelligibly in real time.

The test's 69 items are spread over five task types: Readings, Repeats, Sentence Builds, Passage Retellings, and Short Answer Questions. Speech processing extracts the content and acoustic features of each answer. These feed four subscores (Fluency, Pronunciation, Sentence Mastery, Vocabulary), and a weighted average of them gives the Overall score.

Retesting the same people gave r=0.97. VAT scores correlated r=0.87 with scaled ILR OPI ratings, which is close to how well the OPI agrees with itself. The authors argue that a short, tightly controlled automatic test can replace costly interviews in many settings.

## Key points
- The test has 69 items in six sections (Parts A–F). Repeats appear twice, with 15 items each, and the sentences range from 3 to 12 words.
- Items were adapted from real televised broadcast speech, with the vocabulary simplified. Four native speakers wrote them and five dialectally distinct linguists reviewed them. Fifteen speakers from seven countries recorded the prompts.
- Validation against the ILR OPI used N=118 test-takers, with OPI ratings scaled in FACETS to adjust for rater severity. The result was r=0.87, and VAT explained 76% of OPI variance. For comparison, a second OPI administration explained 83% and a second interviewer 81%.
- The training data held 1309 native and 1337 non-native tests. Under 5% of natives scored below 70, while under 10% of learners scored above 70.
- Natives from Egypt, Syria, Iraq, Palestine, Saudi Arabia and Yemen had nearly identical score curves. Moroccans differed only slightly, which suggests national origin does not drive native scores.

## Interesting ideas
- Highly predictable items, such as repeats or answers with a known form, make both recognition and scoring more precise, and narrowing the measured skill this way still predicted most of what a broader, costlier test captures.
- The machine-versus-human correlation is judged against the human ceiling, meaning how well a second interviewer or a second OPI session agrees with the first.
- Several human judgments are combined with FACETS, which models how strict each rater is, instead of taking a plain average.
- The test is sanity-checked through group behaviour: natives should all score high, learners should spread widely, and country of origin should not separate natives.
- Repeat items grow in length on the reasoning that only automatic processing can handle long units, so the longest sentence a person can reproduce shows their skill level.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (answer): School-course tests could prefer item formats with tightly constrained expected answers, so each reader's errors can be traced to a specific step rather than to vague grading.
- **I2** (learn): Evaluation reports should state the agreement between independent annotators or teacher tiers next to Mova's own score, so a gap is read relative to the attainable ceiling.
- **I3** (learn): When building silver data from several witness parsers or LLM teachers, estimate each witness's systematic bias and weight its votes accordingly.
- **I4** (learn): Use similar population checks as negative controls, for example gold-like native texts should score uniformly high and the genre or source of a text should not predict parser accuracy.
- **I5** (tree): Grade parser test sets by sentence length and clause depth, and report the length at which LAS breaks down as a diagnostic of how well structure is handled.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works · [🎓 Education](https://md.research.elamur.ai/free/en/shelves/social-sciences/education/) · 203 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Jian Cheng, Jared Bernstein, Ulrike Pado and Masanori Suzuki. 2009. Automated Assessment of Spoken Modern Standard Arabic. Proceedings of the Fourth Workshop on Innovative Use of NLP for Building Educational Applications.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{cheng2009automated,
  title = {{Automated Assessment of Spoken Modern Standard Arabic}},
  author = {Jian Cheng, Jared Bernstein, Ulrike Pado and Masanori Suzuki},
  booktitle = {Proceedings of the Fourth Workshop on Innovative Use of NLP for Building Educational Applications},
  year = {2009},
  doi = {10.3115/1609843.1609844},
  url = {https://doi.org/10.3115/1609843.1609844}
}
```

- [← previous Detecting Compositionality in Multi-Word Expressions](https://md.research.elamur.ai/free/en/papers/korkontzelos-2009-detecting-compositionality-multi-word-expressions/)
- [next → Inference Rules and their Application to Recognizing Textual Entailment](https://md.research.elamur.ai/free/en/papers/dinu-2009-inference-rules-their-application-recognizing/)
- [↑ back to papers of 2009](https://md.research.elamur.ai/free/en/papers/2009/)

## References

21 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jennifer Balogh and Jared Bernstein. 2007. Workable models of standard performance in English and Spanish. In Y. Matsumoto, D. Oshima, O. Robinson, and P. Sells, editors, Diversity in Language: Perspectives and Implications (CSLI Lecture Notes, 176), 271-292. CSLI, Stanford, CA. [find](https://scholar.google.com/scholar?q=Jennifer+Balogh+and+Jared+Bernstein%2E+2007%2E+Workable+models+of+standard+performance+in+English+and+Spanish%2E+In+Y%2E+Matsumoto%2C+D%2E+Oshima%2C+O%2E+Robinson%2C+and+P%2E+Sells%2C+editors%2C+Diversity+in+Language%3A+Perspe)
2. Jared Bernstein and Isabella Barbier. 2001. Design and development parameters for a rapid automatic screening test for prospective simultaneous interpreters. Interpreting, International Journal of Research and Practice in Interpreting, 5(2): 221-238. [↗](https://doi.org/10.1075/intp.5.2.09ber)
3. Jared Bernstein, Michael Cohen, Hy Murveit, Dmitry Rtischev, and Mitch Weintraub. 1990. Automatic evaluation and training in English pronunciation. In Proceedings of ICSLP, 1185-1188. [find](https://scholar.google.com/scholar?q=Jared+Bernstein%2C+Michael+Cohen%2C+Hy+Murveit%2C+Dmitry+Rtischev%2C+and+Mitch+Weintraub%2E+1990%2E+Automatic+evaluation+and+training+in+English+pronunciation%2E+In+Proceedings+of+ICSLP%2C+1185%2D1188%2E)
4. Linda Crocker and James Algina. 1986. Introduction to Classical & Modern Test Theory. Harcourt Brace Jovanovich, Orland, FL. [find](https://scholar.google.com/scholar?q=Linda+Crocker+and+James+Algina%2E+1986%2E+Introduction+to+Classical+%26+Modern+Test+Theory%2E+Harcourt+Brace+Jovanovich%2C+Orland%2C+FL%2E)
5. Anne Cutler. 2003. Lexical access. In L. Nadel, editor, Encyclopedia of Cognitive Science, volume 2, pp. 858-864. Nature Publishing Group. [find](https://scholar.google.com/scholar?q=Anne+Cutler%2E+2003%2E+Lexical+access%2E+In+L%2E+Nadel%2C+editor%2C+Encyclopedia+of+Cognitive+Science%2C+volume+2%2C+pp%2E+858%2D864%2E+Nature+Publishing+Group%2E)
6. Maxine Eskenazi. 1996. Detection of foreign speakers’ pronunciation errors for second language training – preliminary results. In Proceedings of ICSLP ‘96. [find](https://scholar.google.com/scholar?q=Maxine+Eskenazi%2E+1996%2E+Detection+of+foreign+speakers%E2%80%99+pronunciation+errors+for+second+language+training+%E2%80%93+preliminary+results%2E+In+Proceedings+of+ICSLP+%E2%80%9896%2E)
7. Maxine Eskenazi and Scott Hansma. 1998. The fluency pronunciation trainer. In Proceedings of the STiLL Workshop. [find](https://scholar.google.com/scholar?q=Maxine+Eskenazi+and+Scott+Hansma%2E+1998%2E+The+fluency+pronunciation+trainer%2E+In+Proceedings+of+the+STiLL+Workshop%2E)
8. Horacio Franco, Victor Abrash, Kristin Precoda, Harry Bratt, Raman Rao, John Butzberger, Romain Rossier, and Federico Cesar. 2000. The SRI EduSpeak system: Recognition and pronunciation scoring for language learning. In Proceedings of InSTiLL, 123-128. [find](https://scholar.google.com/scholar?q=Horacio+Franco%2C+Victor+Abrash%2C+Kristin+Precoda%2C+Harry+Bratt%2C+Raman+Rao%2C+John+Butzberger%2C+Romain+Rossier%2C+and+Federico+Cesar%2E+2000%2E+The+SRI+EduSpeak+system%3A+Recognition+and+pronunciation+scoring+for+la)
9. Jörg Jescheniak, Anja Hahne, and Herbert Schriefers. 2003. Information flow in the mental lexicon during speech planning: Evidence from event-related potentials. Cognitive Brain Research, 15(3):858-864. [find](https://scholar.google.com/scholar?q=J%C3%B6rg+Jescheniak%2C+Anja+Hahne%2C+and+Herbert+Schriefers%2E+2003%2E+Information+flow+in+the+mental+lexicon+during+speech+planning%3A+Evidence+from+event%2Drelated+potentials%2E+Cognitive+Brain+Research%2C+15%283%29%3A858%2D86)
10. Willem Levelt. 2001. Spoken word production: A theory of lexical access. Proceedings of the National Academy of Sciences, 98(23):13464-13471. [↗](https://doi.org/10.1073/pnas.231459498)
11. John Linacre. 2003. FACETS Rasch measurement computer program. Winstep, Chicago, IL. [find](https://scholar.google.com/scholar?q=John+Linacre%2E+2003%2E+FACETS+Rasch+measurement+computer+program%2E+Winstep%2C+Chicago%2C+IL%2E)
12. John Linacre, Benjamin Wright, and Mary Lunz. 1990. A Facets model for judgmental scoring. Memo 61. MESA Psychometric Laboratory. University of Chicago. Retrieved April 14, 2009, from [http://http://www.rasch.org/memo61.htm](http://http://www.rasch.org/memo61.htm). [↗](http://http://www.rasch.org/memo61.htm)
13. George Miller and Stephen Isard. 1963. Some perceptual consequences of linguistic rules. Journal of Verbal Learning and Verbal Behavior, 2:217-228. [↗](https://doi.org/10.1016/s0022-5371(63)80087-0)
14. Pearson. 2009. Versant Arabic test – test description and validation summary. Pearson. Retrieved April 14, 2009, from [http://www.ordinate.com/technology/VersantArabic](http://www.ordinate.com/technology/VersantArabic) TestValidation.pdf. [↗](http://www.ordinate.com/technology/VersantArabic)
15. Hagen Soltau, George Saon, Daniel Povy, Lidia Mangu, Brian Kingsbury, Jeff Kuo, Mohamed Omar, and Geoffrey Zweig. 2007. [find](https://scholar.google.com/scholar?q=Hagen+Soltau%2C+George+Saon%2C+Daniel+Povy%2C+Lidia+Mangu%2C+Brian+Kingsbury%2C+Jeff+Kuo%2C+Mohamed+Omar%2C+and+Geoffrey+Zweig%2E+2007%2E)
16. The IBM 2006 GALE Arabic ASR system. In Proceedings of ICASSP 2007, 349-352. [find](https://scholar.google.com/scholar?q=The+IBM+2006+GALE+Arabic+ASR+system%2E+In+Proceedings+of+ICASSP+2007%2C+349%2D352%2E)
17. Brent Townshend, Jared Bernstein, Ognjen Todic & Eryk Warren. 1998. Estimation of Spoken Language Proficiency. In STiLL: Speech Technology in Language Learning, 177-180. [find](https://scholar.google.com/scholar?q=Brent+Townshend%2C+Jared+Bernstein%2C+Ognjen+Todic+%26+Eryk+Warren%2E+1998%2E+Estimation+of+Spoken+Language+Proficiency%2E+In+STiLL%3A+Speech+Technology+in+Language+Learning%2C+177%2D180%2E)
18. Dimitra Vergyri, Arindam Mandal, Wen Wang, Andreas Stolcke, Jing Zheng, Martin Graciarena, David Rybach, Christian Gollan, Ralf Schlüter, Karin Kirchhoff, Arlo Faria, and Nelson Morgan. 2008. Development of the SRI/Nightingale Arabic ASR system. In Proceedings of Interspeech 2008, 1437-1440. [find](https://scholar.google.com/scholar?q=Dimitra+Vergyri%2C+Arindam+Mandal%2C+Wen+Wang%2C+Andreas+Stolcke%2C+Jing+Zheng%2C+Martin+Graciarena%2C+David+Rybach%2C+Christian+Gollan%2C+Ralf+Schl%C3%BCter%2C+Karin+Kirchhoff%2C+Arlo+Faria%2C+and+Nelson+Morgan%2E+2008%2E+Developm)
19. Steve Young, Dan Kershaw, Julian Odell, Dave Ollason, Valtcho Valtchev, and Phil Woodland. 2000. The HTK Book Version 3.0. Cambridge University Press, Cambridge, UK. [find](https://scholar.google.com/scholar?q=Steve+Young%2C+Dan+Kershaw%2C+Julian+Odell%2C+Dave+Ollason%2C+Valtcho+Valtchev%2C+and+Phil+Woodland%2E+2000%2E+The+HTK+Book+Version+3%2E0%2E+Cambridge+University+Press%2C+Cambridge%2C+UK%2E)
20. Klaus Zechner and Xiaoming Xi. 2008. Towards automatic scoring of a test of spoken language with heterogeneous task types. In Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications, 98-106. [find](https://scholar.google.com/scholar?q=Klaus+Zechner+and+Xiaoming+Xi%2E+2008%2E+Towards+automatic+scoring+of+a+test+of+spoken+language+with+heterogeneous+task+types%2E+In+Proceedings+of+the+Third+Workshop+on+Innovative+Use+of+NLP+for+Building+Ed)
21. Klaus Zechner, Derrick Higgins, and Xiaoming Xi. 2007. SpeechRater™: A construct-driven approach to score spontaneous non-native speech. In Proceedings of the Workshop of the ISCA SIG on Speech and Language Technology in Education. [find](https://scholar.google.com/scholar?q=Klaus+Zechner%2C+Derrick+Higgins%2C+and+Xiaoming+Xi%2E+2007%2E+SpeechRater%E2%84%A2%3A+A+construct%2Ddriven+approach+to+score+spontaneous+non%2Dnative+speech%2E+In+Proceedings+of+the+Workshop+of+the+ISCA+SIG+on+Speech+and+La)

*Updated 1 November 2026*