--- title: "An Evaluation Tool for Machine Translation — Nießen 2000" description: "The paper addresses a practical bottleneck in MT research: system variants are run again and again on the same test set, and the outputs need quality…" authors: ["Sonja Nießen", "Franz Josef Och", "Gregor Leusch", "Hermann Ney"] year: 2000 venue: "Proceedings of the Second International Conference on Language Resources and Evaluation (LREC'00)" source: "https://aclanthology.org/L00-1210/" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/en-2000-evaluation-tool-machine-translation-fast/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/en-2000-evaluation-tool-machine-translation-fast/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2000](https://md.research.elamur.ai/free/papers/2000/) / An Evaluation Tool for Machine Translation: Fast Evaluation for MT Research # An Evaluation Tool for Machine Translation: Fast Evaluation for MT Research **Authors:** [Sonja Nießen](https://md.research.elamur.ai/free/authors/sonja-niesen/), [Franz Josef Och](https://md.research.elamur.ai/free/authors/franz-josef-och/), [Gregor Leusch](https://md.research.elamur.ai/free/authors/gregor-leusch/), [Hermann Ney](https://md.research.elamur.ai/free/authors/hermann-ney/) · **Year:** 2000 · **Venue:** Proceedings of the Second International Conference on Language Resources and Evaluation (LREC'00) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/L00-1210/) · [PDF](https://aclanthology.org/L00-1210.pdf) · [OpenAlex](https://openalex.org/W1529616844) **Citations:** 280 citations (openalex-snapshot · 2026-10-14) · [86](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary The paper addresses a practical bottleneck in MT research: system variants are run again and again on the same test set, and the outputs need quality judgements that are cheap, fast and consistent. Plain edit distance (WER) against a single reference is cheap but depends on that one reference. Human scoring (SSER) is more informative, but it is costly and varies between judges. The authors from RWTH Aachen built a semi-automatic tool. It keeps every human-scored translation in an XML database, which is used in four ways: - Scores for outputs seen before are reused. - Scores for unseen outputs are estimated from their closest stored neighbours by edit distance. - Outputs rated perfect serve as extra references. - A GUI shows how a new output differs from similar ones already judged. They also add an information-item error rate, which scores parts of a sentence separately. The tool was used on Verbmobil and EuTrans test sets. Estimated scores came close to real human scores, and evaluation effort fell by roughly 70%. This makes human-quality evaluation practical for everyday system development. ## Key points - **Metrics:** - SSER uses a 0–10 human score per sentence, from "nonsense" to "perfect". Evaluators later said this scale was too fine and suggested six or seven classes. - eSSER replaces missing scores with the average score of the nearest stored candidates by edit distance. - The information error rate (IER) uses the error classes missing, syntax, meaning and other. - Multi-reference WER compares against all stored outputs that received a score of 10. - **Test data:** Verbmobil-147 (appointment-scheduling dialogues) plus two EuTrans test sets ("open" and "closed") from the touristic domain. - **Reliability of estimates:** In leave-one-out validation, the mean absolute score-estimation error was about 1.0–1.4 points on the 0–10 scale. For Verbmobil it was 1.1, over 144 source sentences with about 41 stored translations each. - **Effort saved:** Across 26 hypothesis files, eSSER differed from the true SSER by only 1.2% on average. Only about 29.5% of sentences needed new human judgement, a saving of at least 70% of the work. - **Human cost:** Scoring a new candidate took about 30–60 seconds. Verbmobil sentences accumulated about 6 "perfect" references each, while the EuTrans sets had fewer. ## Interesting ideas - **Cache human judgements by (source, output) pair.** Repeated MT experiments produce many identical or near-identical outputs. Storing scores lets later evaluations reuse them, which keeps judgements consistent over time and cuts annotation cost. The same idea fits any iterative human evaluation, such as LLM output rating. - **Nearest-neighbour score extrapolation with a confidence signal.** Unseen outputs get the score of their most similar judged neighbours. The average normalized distance to those neighbours is reported alongside, as a rough indicator of how trustworthy the estimate is. This gives a cheap proxy metric that comes with its own uncertainty measure. - **References that grow automatically.** System outputs rated perfect become additional references. They are often closer in wording to MT output than free human translations, so WER becomes fairer at no extra cost. - **Sub-sentence information items.** Splitting a source sentence into units of meaning and labelling each one with an error type addresses two problems. Long, partly correct sentences no longer get a single blunt score, and the labels show which kinds of errors dominate. - **Show annotators the diff against judged neighbours.** Colour-marking insertions, deletions and substitutions relative to already scored similar outputs speeds up annotation and anchors new scores to earlier ones. The authors also checked estimation accuracy with leave-one-out, a validation scheme that transfers easily to other settings. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/) ## Cited by Cited by 86 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 74 not yet. - [Automatic MT Error Analysis: Hjerson Helping Addicter](https://md.research.elamur.ai/free/papers/berka-2012-automatic-mt-error-analysis-hjerson/) - [Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments](https://md.research.elamur.ai/free/papers/fomicheva-2019-taking-mt-evaluation-metrics-extremes/) - [BLEU might be Guilty but References are not Innocent](https://md.research.elamur.ai/free/papers/freitag-2020-bleu-might-be-guilty-but/) - [Towards Heterogeneous Automatic MT Error Analysis](https://md.research.elamur.ai/free/papers/gimenez-2008-heterogeneous-automatic-mt-error-analysis/) - [Semi-automatic error analysis for large-scale statistical machine translation](https://md.research.elamur.ai/free/papers/kirchhoff-2007-semi-automatic-error-analysis-large/) - [Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics](https://md.research.elamur.ai/free/papers/lin-2004-automatic-evaluation-machine-translation-quality/) - [ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation](https://md.research.elamur.ai/free/papers/lin-2004-orange-method-evaluating-automatic-evaluation/) - [DTED: Evaluation of Machine Translation Structure Using Dependency Parsing and Tree Edit Distance](https://md.research.elamur.ai/free/papers/mccaffery-2016-dted-evaluation-machine-translation-structure/) - [Discriminative Training and Maximum Entropy Models for Statistical Machine Translation](https://md.research.elamur.ai/free/papers/och-2002-discriminative-training-maximum-entropy-models/) - [Morpho-syntactic Information for Automatic Error Analysis of Statistical Machine Translation Output](https://md.research.elamur.ai/free/papers/popovic-2006-morpho-syntactic-information-automatic-error/) - [A Study of Translation Edit Rate with Targeted Human Annotation](https://md.research.elamur.ai/free/papers/snover-2006-study-translation-edit-rate-targeted/) - [Fluency, Adequacy, or HTER? Exploring Different Human Judgments with a Tunable MT Metric](https://md.research.elamur.ai/free/papers/snover-2009-fluency-adequacy-or-hter-exploring/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{niesen2000evaluation, title = {{An Evaluation Tool for Machine Translation: Fast Evaluation for MT Research}}, author = {Nießen, Sonja and Och, Franz Josef and Leusch, Gregor and Ney, Hermann}, booktitle = {Proceedings of the Second International Conference on Language Resources and Evaluation (LREC'00)}, year = {2000}, doi = {10.63317/56t9t4rchhar}, url = {https://doi.org/10.63317/56t9t4rchhar} } ``` - [← previous Abstract Anaphora Resolution in Danish](https://md.research.elamur.ai/free/papers/navarretta-2000-abstract-anaphora-resolution-danish/) - [next → An interlingua aiming at communication on the Web: How language-independent can it be?](https://md.research.elamur.ai/free/papers/martins-2000-interlingua-aiming-communication-web-how/) - [↑ back to papers of 2000](https://md.research.elamur.ai/free/papers/2000/) ## References 8 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Alshawi, Hiyan, Srinivas Bangalore, and Shona Douglas, 1998. Automatic Acquisition of Hierarchical Transduction Models for Machine Translation. In Proc. 36th Annual Conference of the Association for Computational Linguistics and the 17th International Conference on Computational Linguistics. Montréal, P.Q., Canada. [find](https://scholar.google.com/scholar?q=Alshawi%2C+Hiyan%2C+Srinivas+Bangalore%2C+and+Shona+Douglas%2C+1998%2E+Automatic+Acquisition+of+Hierarchical+Transduction+Models+for+Machine+Translation%2E+In+Proc%2E+36th+Annual+Conference+of+the+Association+for+C) 2. Amengual, J. C., J. M. Benedı́, A. Castaño, A. Marzal, F. Prat, E. Vidal, J. M. Vilar, C. Delogu, A. di Carlo, H. Ney, and S. Vogel, 1996. Example-Based Understanding and Translation Systems (EuTrans): Final Report, Part I. Deliverable of ESPRIT project No. 20268. [find](https://scholar.google.com/scholar?q=Amengual%2C+J%2E+C%2E%2C+J%2E+M%2E+Bened%C4%B1%CC%81%2C+A%2E+Castan%CC%83o%2C+A%2E+Marzal%2C+F%2E+Prat%2C+E%2E+Vidal%2C+J%2E+M%2E+Vilar%2C+C%2E+Delogu%2C+A%2E+di+Carlo%2C+H%2E+Ney%2C+and+S%2E+Vogel%2C+1996%2E+Example%2DBased+Understanding+and+Translation+Systems+%28EuTrans) 3. Nießen, S., S. Vogel, H. Ney, and C. Tillmann, 1998. A DP based Search Algorithm for Statistical Machine Translation. In Proc. 36th Annual Conference of the Association for Computational Linguistics and the 17th International Conference on Computational Linguistics. Montréal, P.Q., Canada. [find](https://scholar.google.com/scholar?q=Nie%C3%9Fen%2C+S%2E%2C+S%2E+Vogel%2C+H%2E+Ney%2C+and+C%2E+Tillmann%2C+1998%2E+A+DP+based+Search+Algorithm+for+Statistical+Machine+Translation%2E+In+Proc%2E+36th+Annual+Conference+of+the+Association+for+Computational+Linguistics+a) 4. Sparck Jones, Karen and Julia R. Galliers, 1996. Evaluating Natural Language Processing Systems: An Analysis and Review. Lecture notes in computer science. Berlin: Springer-Verlag. [find](https://scholar.google.com/scholar?q=Sparck+Jones%2C+Karen+and+Julia+R%2E+Galliers%2C+1996%2E+Evaluating+Natural+Language+Processing+Systems%3A+An+Analysis+and+Review%2E+Lecture+notes+in+computer+science%2E+Berlin%3A+Springer%2DVerlag%2E) 5. Tillmann, C., S. Vogel, H. Ney, H. Sawaf, and A. Zubiaga, 1997. Accelerated DP based Search for Statistical Translation. In Proceedings of the 5th European Conference on Speech Communication and Technology, Rhodes, Greece. [find](https://scholar.google.com/scholar?q=Tillmann%2C+C%2E%2C+S%2E+Vogel%2C+H%2E+Ney%2C+H%2E+Sawaf%2C+and+A%2E+Zubiaga%2C+1997%2E+Accelerated+DP+based+Search+for+Statistical+Translation%2E+In+Proceedings+of+the+5th+European+Conference+on+Speech+Communication+and+Techn) 6. Vidal, E., 1997. Finite-State Speech-to-Speech Translation. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing, Munich, Germany. [find](https://scholar.google.com/scholar?q=Vidal%2C+E%2E%2C+1997%2E+Finite%2DState+Speech%2Dto%2DSpeech+Translation%2E+In+Proceedings+of+the+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing%2C+Munich%2C+Germany%2E) 7. Wahlster, W., 1993. Verbmobil: Translation of Face-to-Face Dialogs. In Proceedings of the MT Summit IV. Kobe, Japan. [↗](https://doi.org/10.22028/d291-24923) 8. White, John S. and Kathryn B. Taylor, 1998. A Task-Oriented Evaluation Metric for Machine Translation. In Proc. First International Conference on Language Resources and Evaluation. Granada, Spain: European Language Resources Association. [find](https://scholar.google.com/scholar?q=White%2C+John+S%2E+and+Kathryn+B%2E+Taylor%2C+1998%2E+A+Task%2DOriented+Evaluation+Metric+for+Machine+Translation%2E+In+Proc%2E+First+International+Conference+on+Language+Resources+and+Evaluation%2E+Granada%2C+Spain%3A+Eur)