--- title: "Using Test Suites in Evaluation of Machine — King 1990" description: "An early proposal to evaluate MT by test suites rather than by ratings or error counts. The paper argues that no single evaluation fits all buyers, so it…" authors: ["Margaret King", "Kirsten Falkedal"] year: 1990 venue: "COLING 1990 Volume 2: Papers presented to the 13th International Conference on Computational Linguistics" source: "https://aclanthology.org/C90-2037/" doi: "10.3115/997939.997976" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/king-1990-test-suites-evaluation-machine-translation/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/king-1990-test-suites-evaluation-machine-translation/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [1990](https://md.research.elamur.ai/free/en/papers/1990/) / Using Test Suites in Evaluation of Machine Translation Systems # Using Test Suites in Evaluation of Machine Translation Systems **Authors:** Margaret King, Kirsten Falkedal · **Year:** 1990 · **Venue:** COLING 1990, Volume 2: Papers presented to the 13th International Conference on Computational Linguistics **Link:** [aclanthology.org/C90-2037](https://aclanthology.org/C90-2037/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only [**64** citations](https://openalex.org/works?filter=cites:W2060377713) OpenAlex · 2026-10-14 · [**31** citations in our library](https://md.research.elamur.ai/free/en/papers/king-1990-test-suites-evaluation-machine-translation/cited-by/) of the 140673 papers whose bibliographies we read · [COLING 1990 Volume 2: Papers presented to the 13th International Conference on Computational Linguistics](https://aclanthology.org/volumes/C90-2/) · [licence](https://creativecommons.org/licenses/by-nc-sa/3.0/) - [Publisher's page](https://aclanthology.org/C90-2037/) - [PDF](https://aclanthology.org/C90-2037.pdf) - [DOI 10.3115/997939.997976](https://doi.org/10.3115/997939.997976) - [OpenAlex](https://openalex.org/W2060377713) **Authors (2)** 1. [Margaret King](https://openalex.org/A5112713865) [Semantic Scholar](https://www.semanticscholar.org/author/48505446) · h 11 · 685 citations · 56 works 2. [Kirsten Falkedal](https://openalex.org/A5089544780) [Semantic Scholar](https://www.semanticscholar.org/author/2504135) · h 3 · 209 citations · 6 works Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart. ## What it is about An early proposal to evaluate MT by test suites rather than by ratings or error counts. The paper argues that no single evaluation fits all buyers, so it proposes two suites: one of sentences drawn from a bilingual corpus of the intended use, and one of translational mismatches and ambiguities. Each structure gets two parallel lists, A and B. The developer sees the A list and corrects dictionary coding; the B list is never used for fixes. Its results after each round show the side effects of the fixes. Cases that remain unsolved are classified by the time the maker estimates a change would need. The paper gives a procedure and no measured results. ## Ideas - I1. Use the A and B split for the lexicon work (open problem 2). Fit the merges on the A verses only, and report the B-verse residual after each change as the side-effect check. A merge that raises the B residual is noise. - I2. Keep the paper's rule that every test item has a pre-agreed acceptable output. For Mova, the verse's reference translation by another author can serve as that target, so no new translations are written. - I3. Sort the remaining failures by estimated fix cost (a lexicon entry, an encoder rule, a reader change). That gives the order of work across the eight open problems. ## For Mova The places are `tools/translate` (`meta.rs` for the lexicon, `profile.rs` for the encoder). The A and B lists must be fixed before any merge is tried, and the held-out rule still applies. Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) ## Cite The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.3115/997939.997976](https://doi.org/10.3115/997939.997976) ```bibtex @inproceedings{king1990using, title = {{Using Test Suites in Evaluation of Machine Translation Systems}}, author = {King, Margaret and Falkedal, Kirsten}, booktitle = {COLING 1990 Volume 2: Papers presented to the 13th International Conference on Computational Linguistics}, year = {1990}, doi = {10.3115/997939.997976}, url = {https://doi.org/10.3115/997939.997976} } ``` - [← previous The Hebrew revolution and the revolution of the Hebrew language between the 1880s and the 1930s](https://md.research.elamur.ai/free/en/papers/winther-1990-hebrew-revolution-revolution-hebrew-language/) - [next → Word Association Norms, Mutual Information, and Lexicography](https://md.research.elamur.ai/free/en/papers/church-1990-word-association-norms-mutual-information/) - [↑ back to papers of 1990](https://md.research.elamur.ai/free/en/papers/1990/) ## References 7 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Flickinger, D. et al. HP-NL Test Suite, (as of June 30, 1987). [find](https://scholar.google.com/scholar?q=Flickinger%2C+D%2E+et+al%2E+HP%2DNL+Test+Suite%2C+%28as+of+June+30%2C+1987%29%2E) 2. King, Margaret. A Practical Guide to the Evaluation o/ Machine Translation Systems. ISSCO. Intermediate Report to Suissetra, Feb., 1989. Revised version in preparation. [find](https://scholar.google.com/scholar?q=King%2C+Margaret%2E+A+Practical+Guide+to+the+Evaluation+o%2F+Machine+Translation+Systems%2E+ISSCO%2E+Intermediate+Report+to+Suissetra%2C+Feb%2E%2C+1989%2E+Revised+version+in+preparation%2E) 3. Lehrberger, J. and Bourbeau, L. Machine Translation: Linguistic Characteristics o / M T Systems and General Methodology of Evaluation. John Benjamin. 1988. [find](https://scholar.google.com/scholar?q=Lehrberger%2C+J%2E+and+Bourbeau%2C+L%2E+Machine+Translation%3A+Linguistic+Characteristics+o+%2F+M+T+Systems+and+General+Methodology+of+Evaluation%2E+John+Benjamin%2E+1988%2E) 4. Slocum, J. and Bennet, W. An Evaluation o/ METAL: the LRC Machine Translation System. Second Conference of the European Chapter of the ACL, 1985. [find](https://scholar.google.com/scholar?q=Slocum%2C+J%2E+and+Bennet%2C+W%2E+An+Evaluation+o%2F+METAL%3A+the+LRC+Machine+Translation+System%2E+Second+Conference+of+the+European+Chapter+of+the+ACL%2C+1985%2E) 5. Van Slype, G. Critical Study o/ Methods for Evaluating the Quality o/ Machine Translation Systems. Bureau Marcel Van Dijk, Bruxelles and CCE, 1979. [find](https://scholar.google.com/scholar?q=Van+Slype%2C+G%2E+Critical+Study+o%2F+Methods+for+Evaluating+the+Quality+o%2F+Machine+Translation+Systems%2E+Bureau+Marcel+Van+Dijk%2C+Bruxelles+and+CCE%2C+1979%2E) 6. Vasconcellos, M. (ed.) Technology Translation Strategy. SUNY, 1988. [find](https://scholar.google.com/scholar?q=Vasconcellos%2C+M%2E+%28ed%2E%29+Technology+Translation+Strategy%2E+SUNY%2C+1988%2E) 7. Wilks, Y. and LATSEC Inc. Comparative Translation Quality Analysis, Final Report, Latsec Inc., 1979. as [find](https://scholar.google.com/scholar?q=Wilks%2C+Y%2E+and+LATSEC+Inc%2E+Comparative+Translation+Quality+Analysis%2C+Final+Report%2C+Latsec+Inc%2E%2C+1979%2E+as)