---
title: "Measuring Non-native Speakers' Proficiency of English by Using"
description: "The authors build an automatic pipeline that works from a seed sentence taken from business and… 17 references, cited by 34 papers here."
authors: ["Eiichiro Sumita", "Fumiaki Sugaya", "Seiichi Yamamoto"]
year: 2005
venue: "Proceedings of the Second Workshop on Building Educational Applications Using NLP"
source: "https://aclanthology.org/W05-0210/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sumita-2005-measuring-non-native-speakers-proficiency/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sumita-2005-measuring-non-native-speakers-proficiency/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2005](https://md.research.elamur.ai/free/en/papers/2005/) / Measuring Non-native Speakers' Proficiency of English by Using a Test with Automatically-Generated Fill-in-the-Blank Questions

# Measuring Non-native Speakers' Proficiency of English by Using a Test with Automatically-Generated Fill-in-the-Blank Questions

**Authors:** [Eiichiro Sumita](https://md.research.elamur.ai/free/en/authors/eiichiro-sumita/), [Fumiaki Sugaya](https://md.research.elamur.ai/free/en/authors/fumiaki-sugaya/) and [Seiichi Yamamoto](https://md.research.elamur.ai/free/en/authors/seiichi-yamamoto/) · **Year:** 2005 · **Venue:** [Proceedings of the Second Workshop on Building Educational Applications Using NLP](https://md.research.elamur.ai/free/en/venues/workshop-on-building-educational-applications-using-nlp/) ([Workshop on Building Educational Applications Using NLP 2005](https://md.research.elamur.ai/free/en/venues/workshop-on-building-educational-applications-using-nlp/2005/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W05-0210/) · [PDF](https://aclanthology.org/W05-0210.pdf) · [OpenAlex](https://openalex.org/W1965158952)

**Citations:** 114 citations (OpenAlex, 2026-10-14) · [34](https://md.research.elamur.ai/free/en/papers/sumita-2005-measuring-non-native-speakers-proficiency/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses the high cost of writing fill-in-the-blank (FBQ) English proficiency tests. Good wrong options, which mislead only weaker learners, normally require skilled human writers. The authors build an automatic pipeline that works from a seed sentence taken from business and travel conversation corpora or from Web pages.

The pipeline works in four steps:
- It blanks the verb in the seed sentence.
- It collects same-meaning candidates for the blank from a thesaurus.
- It searches the Web and keeps only candidates whose restored sentence fragment gets no hits at all.
- It builds a four-option item from the correct word and three surviving distracters.

Test-takers' answers are then scored with Item Response Theory (IRT). In the experiments, the estimated proficiency of non-native speakers correlated strongly with their TOEIC scores, while native speakers solved the test almost perfectly. The test could also be shortened by using IRT item information to discard items that tell test-takers apart poorly. This matters because it enables cheap, on-demand testing that can draw on current texts. The authors implemented a mobile-phone version that runs adaptive testing.

## Key points
- Each item has one correct answer and three distracters. The blank position is chosen by "testing knowledge," which in the experiments meant the main verb, following TOEIC-style questions.
- Distracter candidates come from an in-house English thesaurus whose hierarchy follows the Japanese thesaurus Ruigo-Shin-Jiten. Example: for "keep," the candidates "clear" and "share" were rejected, while "guarantee," "promise," and "reserve" were kept.
- Web verification used Google (about four billion pages as of February 2004) under the strictest rule: a candidate survives only if it gets zero hits. Instead of the full sentence, the query is a word window around the blank, running from a content word or sentence start to a content word or sentence end.
- IRT models the probability of a correct answer as an S-shaped logistic function of ability. Item information is used to remove items that do not discriminate between test-takers.
- Questions were also generated from Japan Times news articles and from over 100 Project Gutenberg e-books. A grammar variant keeps the word stem and varies only its inflected form. Listening skills are out of scope.

## Interesting ideas
- The method separates generation from verification: a lexical resource proposes plausible same-category substitutes, and an external evidence source filters out any substitute that still yields an acceptable sentence, which gives cleaner contrastive examples than random corruption.
- Attestation is checked on a short local window bounded by content words rather than on the whole sentence, because whole-sentence lookups almost never match while local fragments capture the key collocation.
- The generated test is validated against groups of known ability: natives should score near the ceiling, and non-native scores should track an external benchmark (TOEIC).
- IRT item information is used to keep only discriminative items, so a much smaller test measures nearly as well as the full one.
- In the grammar variant, the stem stays fixed and only the inflected form changes, so the item tests grammar rather than word meaning.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Build negative-control minimal pairs by swapping a verb for its VerbNet/WordNet neighbours and keeping only unattested swaps, then check that the pipeline flags them as anomalous.
- **I2** (frames): Score verb–argument plausibility for semantic-role and frame decisions by looking up the local predicate–argument window in a large corpus, keeping the evidence as an explainable trace.
- **I3** (answer): Validate auto-generated school-style test items by checking that strong reference readers ace them and that weaker reader configurations score lower in the expected order.
- **I4** (learn): Fit IRT over many parser versions or witness parsers and prune evaluation items that every system passes or fails, yielding a compact, sensitive regression suite.
- **I5** (tokens): Use the morphology generator to produce inflection-variant distracters and test whether the tagger and lemmatizer, plus the downstream tree, reject the wrong surface forms.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Doshisha University](https://md.research.elamur.ai/free/en/universities/doshisha-university/) — University, Kyoto, Japan · 3 works · 149 citations

Also on the byline: [KDDI Research](https://ror.org/05qsqt662).

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Developmental and Educational Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/developmental-and-educational-psychology/) · 499 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works

## Cite

Cite as: Eiichiro Sumita, Fumiaki Sugaya and Seiichi Yamamoto. 2005. Measuring Non-native Speakers' Proficiency of English by Using a Test with Automatically-Generated Fill-in-the-Blank Questions. Proceedings of the Second Workshop on Building Educational Applications Using NLP.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sumita2005measuring,
  title = {{Measuring Non-native Speakers' Proficiency of English by Using a Test with Automatically-Generated Fill-in-the-Blank Questions}},
  author = {Eiichiro Sumita, Fumiaki Sugaya and Seiichi Yamamoto},
  booktitle = {Proceedings of the Second Workshop on Building Educational Applications Using NLP},
  year = {2005},
  doi = {10.3115/1609829.1609839},
  url = {https://doi.org/10.3115/1609829.1609839}
}
```

- [← previous Probabilistic Disambiguation Models for Wide-Coverage HPSG Parsing](https://md.research.elamur.ai/free/en/papers/miyao-2005-probabilistic-disambiguation-models-wide-coverage/)
- [next → Evaluating Machine Translation Output with Automatic Sentence Segmentation](https://md.research.elamur.ai/free/en/papers/matusov-2005-evaluating-machine-translation-output-automatic/)
- [↑ back to papers of 2005](https://md.research.elamur.ai/free/en/papers/2005/)

## References

17 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alderson, Charles. 1996. Do corpora have a role in language assessment? Using Corpora for Language Research, eds. Thomas, J. and Short, M., Longman: 248—259. [find](https://scholar.google.com/scholar?q=Alderson%2C+Charles%2E+1996%2E+Do+corpora+have+a+role+in+language+assessment%3F+Using+Corpora+for+Language+Research%2C+eds%2E+Thomas%2C+J%2E+and+Short%2C+M%2E%2C+Longman%3A+248%E2%80%94259%2E)
2. Brown, J. D. 1993. What are the characteristics of natural cloze tests? Language Testing 10: 93—116. [↗](https://doi.org/10.1177/026553229301000201)
3. Crystal, David. 2003. English as a Global Language, (Second Edition). Cambridge University Press: 212. [find](https://scholar.google.com/scholar?q=Crystal%2C+David%2E+2003%2E+English+as+a+Global+Language%2C+%28Second+Edition%29%2E+Cambridge+University+Press%3A+212%2E)
4. Educational Testing Service 2002. TOEIC koushiki gaido & mondaishu. IIBC: 249. [find](https://scholar.google.com/scholar?q=Educational+Testing+Service+2002%2E+TOEIC+koushiki+gaido+%26+mondaishu%2E+IIBC%3A+249%2E)
5. Embretson, Susan et al. 2000. Item Response Theory for Psychologists. LEA: 371. [find](https://scholar.google.com/scholar?q=Embretson%2C+Susan+et+al%2E+2000%2E+Item+Response+Theory+for+Psychologists%2E+LEA%3A+371%2E)
6. Grefenstette, G. 1999. The WWW as a resource for example-based MT tasks. ASLIB “Translating and the Computer” conference. [find](https://scholar.google.com/scholar?q=Grefenstette%2C+G%2E+1999%2E+The+WWW+as+a+resource+for+example%2Dbased+MT+tasks%2E+ASLIB+%E2%80%9CTranslating+and+the+Computer%E2%80%9D+conference%2E)
7. Irvine, H. S., and Kyllonen, P. C. (2002). Item generation for test development. LEA: 412. [find](https://scholar.google.com/scholar?q=Irvine%2C+H%2E+S%2E%2C+and+Kyllonen%2C+P%2E+C%2E+%282002%29%2E+Item+generation+for+test+development%2E+LEA%3A+412%2E)
8. Izumi, E., and Isahara, H. (2004). Investigation into language learners' acquisition order based on the error analysis of the learner corpus. In Proceedings of Pacific-Asia Conference on Language, Information and Computation (PACLIC) 18 Satellite Workshop on E-Learning, Japan. (in printing) Kikui, G., Sumita, E., Takaezawa, T. and Yamamoto, S., “Creating Corpora for Speech-to-Speech Translation,” Special Session “Multilingual Speech-to-Speech Translation” of EuroSpeech, 2003. [find](https://scholar.google.com/scholar?q=Izumi%2C+E%2E%2C+and+Isahara%2C+H%2E+%282004%29%2E+Investigation+into+language+learners%27+acquisition+order+based+on+the+error+analysis+of+the+learner+corpus%2E+In+Proceedings+of+Pacific%2DAsia+Conference+on+Language%2C+Inf)
9. Kilgarriff, A. and Grefenstette, G. 2003. Special Issue on the WEB as Corpus. Computational Linguistics 29 (3): 333—502. [find](https://scholar.google.com/scholar?q=Kilgarriff%2C+A%2E+and+Grefenstette%2C+G%2E+2003%2E+Special+Issue+on+the+WEB+as+Corpus%2E+Computational+Linguistics+29+%283%29%3A+333%E2%80%94502%2E)
10. Mitkov, Ruslan and Ha, Le An. 2003. Computer-Aided Generation of Multiple-Choice Tests. HLT-NAACL 2003 Workshop: Building Educational Applications Using Natural Language Processing: 17—22. [find](https://scholar.google.com/scholar?q=Mitkov%2C+Ruslan+and+Ha%2C+Le+An%2E+2003%2E+Computer%2DAided+Generation+of+Multiple%2DChoice+Tests%2E+HLT%2DNAACL+2003+Workshop%3A+Building+Educational+Applications+Using+Natural+Language+Processing%3A+17%E2%80%9422%2E)
11. Ohno, S. and Hamanishi, M. 1984. Ruigo-Shin-Jiten, Kadokawa, Tokyo (in Japanese) Shermis, M. D. and Burstein. [find](https://scholar.google.com/scholar?q=Ohno%2C+S%2E+and+Hamanishi%2C+M%2E+1984%2E+Ruigo%2DShin%2DJiten%2C+Kadokawa%2C+Tokyo+%28in+Japanese%29+Shermis%2C+M%2E+D%2E+and+Burstein%2E)
12. J. C. 2003. Automated Essay Scoring. LEA: 238. [find](https://scholar.google.com/scholar?q=J%2E+C%2E+2003%2E+Automated+Essay+Scoring%2E+LEA%3A+238%2E)
13. Tonoike, M., Sato, S., and Utsuro, T. 2004. Answer Validation by Keyword Association. IPSJ, SIGNL, 161: 53—60, (in Japanese). [find](https://scholar.google.com/scholar?q=Tonoike%2C+M%2E%2C+Sato%2C+S%2E%2C+and+Utsuro%2C+T%2E+2004%2E+Answer+Validation+by+Keyword+Association%2E+IPSJ%2C+SIGNL%2C+161%3A+53%E2%80%9460%2C+%28in+Japanese%29%2E)
14. Turney, P.D. 2001. Mining the Web for synonyms: PMI-IR vs. LSA on TOEFL. ECML 2001: 491—502. [find](https://scholar.google.com/scholar?q=Turney%2C+P%2ED%2E+2001%2E+Mining+the+Web+for+synonyms%3A+PMI%2DIR+vs%2E+LSA+on+TOEFL%2E+ECML+2001%3A+491%E2%80%94502%2E)
15. Wainer, Howard et al. 2000. Conputerized Adaptive Testing: A Primer, (Second Edition). LEA: 335. [find](https://scholar.google.com/scholar?q=Wainer%2C+Howard+et+al%2E+2000%2E+Conputerized+Adaptive+Testing%3A+A+Primer%2C+%28Second+Edition%29%2E+LEA%3A+335%2E)
16. Wilson, E. 1997. The Automatic Generation of CALL exercises from general corpora, in eds. Wichmann, A., Fligelstone, S., McEnery, T., Knowles, G., Teaching and Language Corpora, Harlow: Longman:116-130. [find](https://scholar.google.com/scholar?q=Wilson%2C+E%2E+1997%2E+The+Automatic+Generation+of+CALL+exercises+from+general+corpora%2C+in+eds%2E+Wichmann%2C+A%2E%2C+Fligelstone%2C+S%2E%2C+McEnery%2C+T%2E%2C+Knowles%2C+G%2E%2C+Teaching+and+Language+Corpora%2C+Harlow%3A+Longman%3A116%2D13)
17. Yasuda, K., Sugaya, F., Sumita, E., Takezawa, T., Kikui, G. and Yamamoto, S. 2004. [Automatic Measuring of English Language Proficiency using MT Evaluation Technology](https://md.research.elamur.ai/free/en/papers/yasuda-2004-automatic-measuring-english-language-proficiency/), COLING 2004 eLearning for Computational Linguistics and Computational Linguistics for eLearning: 53-60.

*Updated 1 November 2026*