---
title: "Automatic Measuring of English Language Proficiency using MT"
description: "The authors want a fast, computer-run test that does not require experts to write multiple-choice items. 12 references, cited by 2 papers here."
authors: ["Keiji Yasuda", "Fumiaki Sugaya", "Eiichiro Sumita", "Toshiyuki Takezawa", "Genichiro Kikui", "Seiichi Yamamoto"]
year: 2004
venue: "Proceedings of the Workshop on eLearning for Computational Linguistics and Computational Linguistics for eLearning"
source: "https://aclanthology.org/W04-1708/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yasuda-2004-automatic-measuring-english-language-proficiency/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yasuda-2004-automatic-measuring-english-language-proficiency/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2004](https://md.research.elamur.ai/free/en/papers/2004/page/2/) / Automatic Measuring of English Language Proficiency using MT Evaluation Technology

# Automatic Measuring of English Language Proficiency using MT Evaluation Technology

**Authors:** Keiji Yasuda, [Fumiaki Sugaya](https://md.research.elamur.ai/free/en/authors/fumiaki-sugaya/), [Eiichiro Sumita](https://md.research.elamur.ai/free/en/authors/eiichiro-sumita/), [Toshiyuki Takezawa](https://md.research.elamur.ai/free/en/authors/toshiyuki-takezawa/), Genichiro Kikui and [Seiichi Yamamoto](https://md.research.elamur.ai/free/en/authors/seiichi-yamamoto/) · **Year:** 2004 · **Venue:** Proceedings of the Workshop on eLearning for Computational Linguistics and Computational Linguistics for eLearning · **Publisher:** [International Committee on Computational Linguistics (ICCL)](https://md.research.elamur.ai/free/en/publishers/international-committee-on-computational-linguistics-iccl/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W04-1708/) · [PDF](https://aclanthology.org/W04-1708.pdf) · [OpenAlex](https://openalex.org/W1974787691)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/yasuda-2004-automatic-measuring-english-language-proficiency/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether tools built to score machine translation can also grade human learners of English. The authors want a fast, computer-run test that does not require experts to write multiple-choice items. Japanese learners translate Japanese sentences into English. Their output is compared with multiple reference translations using a DP (edit-distance) score, BLEU or NIST. A straight regression line then maps the averaged match score onto TOEIC scores. Before testing, each candidate sentence is kept or dropped based on how well its automatic score tracks the subjects' known TOEIC levels.

Two travel-domain corpora were used: BTEC (510 sentences, 18 subjects) and SLTA1 (330 sentences, 29 subjects). Each sentence had 16 references. With the selected sentences, test-set-level correlation with TOEIC reaches about 0.95. In leave-one-out cross validation, the estimation error is roughly 120 TOEIC points on BTEC and under 100 on SLTA1. The best test length is 30–60 sentences. The work shows that reference-based overlap metrics can serve as cheap, compact proficiency probes.

## Key points
- Three metrics are compared: a DP-based edit-distance score, BLEU (n-grams up to 4) and NIST (up to 5). All three behave in a similar way.
- The data are BTEC (phrasebook-style, about 8 words per sentence, TOEIC range 400s–900s) and SLTA1 (23 interpreted hotel dialogues, about 13 words per sentence, TOEIC range 300s–800s).
- Each sentence has 16 references: one from the corpus English side, plus 15 written by 5 bilinguals (3 each).
- About 20% of BTEC sentences and about 10% of SLTA1 sentences correlate negatively with TOEIC. These are mostly very short, common phrases, or sentences with many valid translations that the references fail to cover.
- After selection, 30–60 sentences give a standard error of about 120 TOEIC points (BTEC) or below 100 (SLTA1). The authors propose automatic paraphrasing as a future way to produce references.

## Interesting ideas
- Each test item is screened by correlating its automatic score with a trusted external measure, and items that fail to discriminate are discarded, which makes a short test stronger than the full pool.
- A low item correlation can point to two different problems: the item is trivial, or the scoring cannot handle the item because its valid answers are too varied.
- A raw overlap metric is turned into a human-meaningful scale by fitting a regression on calibrated subjects, and the result is checked with leave-one-out error rather than correlation alone.
- Many references per sentence reduce the penalty for valid but unusual phrasings, and the authors plan to generate such references through paraphrasing.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Mova's evaluation suites could keep only those sentences whose parser scores track a known ranking of parser versions, giving small but sensitive regression tests.
- **I2** (answer): if automatic grading of a school-test or fable-QA item disagrees with known-good students, the item could be flagged as a likely grader blind spot.
- **I3** (learn): silver-data agreement scores could be mapped onto human-checked accuracy and reported with held-out error, so the numbers mean something concrete.
- **I4** (back): output of the text → graph → text compressor could be scored against several paraphrase references instead of one, so faithful rewordings are not counted as losses.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Advanced Telecommunications Research Institute International](https://md.research.elamur.ai/free/en/labs/advanced-telecommunications-research-institute-international/) — Research institute, Kyoto, Japan · 4 works · 802 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Developmental and Educational Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/developmental-and-educational-psychology/) · 499 works

🧭 Topics (3): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 1,087 works · [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Keiji Yasuda, Fumiaki Sugaya, Eiichiro Sumita, Toshiyuki Takezawa, Genichiro Kikui and Seiichi Yamamoto. 2004. Automatic Measuring of English Language Proficiency using MT Evaluation Technology. Proceedings of the Workshop on eLearning for Computational Linguistics and Computational Linguistics for eLearning.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yasuda2004automatic,
  title = {{Automatic Measuring of English Language Proficiency using MT Evaluation Technology}},
  author = {Keiji Yasuda, Fumiaki Sugaya, Eiichiro Sumita, Toshiyuki Takezawa, Genichiro Kikui and Seiichi Yamamoto},
  booktitle = {Proceedings of the Workshop on eLearning for Computational Linguistics and Computational Linguistics for eLearning},
  year = {2004},
  doi = {10.3115/1610028.1610036},
  url = {https://doi.org/10.3115/1610028.1610036}
}
```

- [← previous Stylistic Fronting in the Ormulum - Scandinavian Syntactic Phenomena in Early Middle English Texts](https://md.research.elamur.ai/free/en/papers/trips-2004-stylistic-fronting-ormulum-scandinavian-syntactic/)
- [next → Intentions, Implicatures and Processing of Complex Questions](https://md.research.elamur.ai/free/en/papers/harabagiu-2004-intentions-implicatures-processing-complex-questions/)
- [↑ back to papers of 2004](https://md.research.elamur.ai/free/en/papers/2004/page/2/)

## References

12 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. CASEC. 2004. Computer Assessment System for English Communication. [http://www.ets.org/toefl/](http://www.ets.org/toefl/). [↗](http://www.ets.org/toefl/)
2. A. Finch, T. Watanabe, and E. Sumita. 2002. “Paraphrasing by Statistical Machine Translation”. In Proceedings of the 1st Forum on Information Technology (FIT2002), volume E-53, pages 187–188. [find](https://scholar.google.com/scholar?q=A%2E+Finch%2C+T%2E+Watanabe%2C+and+E%2E+Sumita%2E+2002%2E+%E2%80%9CParaphrasing+by+Statistical+Machine+Translation%E2%80%9D%2E+In+Proceedings+of+the+1st+Forum+on+Information+Technology+%28FIT2002%29%2C+volume+E%2D53%2C+pages+187%E2%80%93188%2E)
3. G. Kikui, E. Sumita, T. Takezawa, and S. Yamamoto. 2003. “Creating Corpora for Speech-to-Speech Translation”. In Proceedings of EUROSPEECH, pages 381–384. [find](https://scholar.google.com/scholar?q=G%2E+Kikui%2C+E%2E+Sumita%2C+T%2E+Takezawa%2C+and+S%2E+Yamamoto%2E+2003%2E+%E2%80%9CCreating+Corpora+for+Speech%2Dto%2DSpeech+Translation%E2%80%9D%2E+In+Proceedings+of+EUROSPEECH%2C+pages+381%E2%80%93384%2E)
4. NIST. 2002. Automatic Evaluation of Machine Translation Quality Using N-gram Co-Occurence Statistics. [http://www.nist.gov/speech/tests/mt/mt2001/resource/](http://www.nist.gov/speech/tests/mt/mt2001/resource/). [↗](http://www.nist.gov/speech/tests/mt/mt2001/resource/)
5. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 311–318.
6. M. Shimohata and E. Sumita. 2002. “Automatic Paraphrasing Based on Parallel Corpus for Normalization”. In Proceedings of International Conference on Language Resources and Evaluation (LREC), pages 453–457. [find](https://scholar.google.com/scholar?q=M%2E+Shimohata+and+E%2E+Sumita%2E+2002%2E+%E2%80%9CAutomatic+Paraphrasing+Based+on+Parallel+Corpus+for+Normalization%E2%80%9D%2E+In+Proceedings+of+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%29%2C+pages+453)
7. K.-Y. Su, M.-W. Wu, and J.-S. Chang. 1992. A new quantitative quality measure for machine translation systems. In Proceedings of the 14th International Conference on Computational Linguistics(COLING), pages 433– 439. [find](https://scholar.google.com/scholar?q=K%2E%2DY%2E+Su%2C+M%2E%2DW%2E+Wu%2C+and+J%2E%2DS%2E+Chang%2E+1992%2E+A+new+quantitative+quality+measure+for+machine+translation+systems%2E+In+Proceedings+of+the+14th+International+Conference+on+Computational+Linguistics%28COLING%29%2C)
8. T. Takezawa, F. Sugaya, A. Yokoo, and S. Yamamoto. 1999. A new evaluation method for speech translation systems and a case study on ATR-MATRIX from Japanese to English. In Proceeding of Machine Translation Summit (MT Summit), pages 299–307. [find](https://scholar.google.com/scholar?q=T%2E+Takezawa%2C+F%2E+Sugaya%2C+A%2E+Yokoo%2C+and+S%2E+Yamamoto%2E+1999%2E+A+new+evaluation+method+for+speech+translation+systems+and+a+case+study+on+ATR%2DMATRIX+from+Japanese+to+English%2E+In+Proceeding+of+Machine+Transl)
9. T. Takezawa, E. Sumita, F. Sugaya, H. Yamamoto, and S. Yamamoto. 2002. “[Toward a Broad-Coverage Bilingual Corpus for Speech Translation of Travel Conversations in the Real World](https://md.research.elamur.ai/free/en/papers/takezawa-2002-broad-coverage-bilingual-corpus-speech/)”. In Proceedings of International Conference on Language Resources and Evaluation (LREC), pages 147–152.
10. T. Takezawa. 1999. Building a bilingual travel conversation database for speech translation research. In Proceedings of the 2nd International Workshop on East-Asian Language Resources and Evaluation – Oriental CO-COSDA Workshop ’99 –, pages 17–20. [find](https://scholar.google.com/scholar?q=T%2E+Takezawa%2E+1999%2E+Building+a+bilingual+travel+conversation+database+for+speech+translation+research%2E+In+Proceedings+of+the+2nd+International+Workshop+on+East%2DAsian+Language+Resources+and+Evaluation+%E2%80%93)
11. TOEFL. 2004. Test of English as a Foreign Language. [http://www.ets.org/toefl/](http://www.ets.org/toefl/). [↗](http://www.ets.org/toefl/)
12. TOEIC. 2004. Test of English for International Communication. [http://www.ets.org/toeic/](http://www.ets.org/toeic/). [↗](http://www.ets.org/toeic/)

*Updated 31 October 2026*