--- title: "Linguistic Evaluation of German-English — Avramidis 2019" description: "Generic metrics and human rankings show how good a machine translation system is overall, but not which grammatical constructions it handles badly." authors: ["Eleftherios Avramidis", "Vivien Macketanz", "Ursula Strohriegel", "Hans Uszkoreit"] year: 2019 venue: "Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)" source: "https://aclanthology.org/W19-5351/" doi: "10.18653/v1/w19-5351" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/avramidis-2019-linguistic-evaluation-german-english-machine/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/avramidis-2019-linguistic-evaluation-german-english-machine/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Linguistic Evaluation of German-English Machine Translation Using a Test Suite # Linguistic Evaluation of German-English Machine Translation Using a Test Suite **Authors:** [Eleftherios Avramidis](https://md.research.elamur.ai/free/en/authors/eleftherios-avramidis/), [Vivien Macketanz](https://md.research.elamur.ai/free/en/authors/vivien-macketanz/), Ursula Strohriegel, [Hans Uszkoreit](https://md.research.elamur.ai/free/en/authors/hans-uszkoreit/) · **Year:** 2019 · **Venue:** Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W19-5351/) · [PDF](https://aclanthology.org/W19-5351.pdf) · [DOI 10.18653/v1/w19-5351](https://doi.org/10.18653/v1/w19-5351) · [OpenAlex](https://openalex.org/W2980941473) **Citations:** 14 citations (OpenAlex, 2026-10-14) · [24](https://md.research.elamur.ai/free/en/papers/avramidis-2019-linguistic-evaluation-german-english-machine/cited-by/) in our library, of the 140884 papers whose bibliographies we read ## Summary Generic metrics and human rankings show how good a machine translation system is overall, but not which grammatical constructions it handles badly. The authors address this for German→English. They apply the manually built DFKI test suite to the 16 systems in the WMT19 news translation task. The suite has 5560 source sentences covering 107 mostly grammatical phenomena in 14 categories. Each system output is checked by hand-written positive and negative regular expressions in the TQ-AutoTest tool. Outputs that match neither, or both, are flagged as warnings and resolved by a human annotator, who also refines the rules. Accuracy is the share of correctly translated items per phenomenon, and system differences are checked with a one-tailed Z-test. Even strong neural systems still get roughly a quarter of the items wrong. Idioms, modal constructions, pseudo-clefts, MWE and verb valency are the weakest areas. Compared with WMT18, function words, non-verbal agreement and punctuation improved. The study shows how fine-grained diagnostic testing exposes specific weaknesses that aggregate scores hide. ## Key points - About 10% of outputs were initially flagged as warnings. One linguistics student spent roughly 70 hours reducing this to 3%, leaving 5393 fully valid test items. - Average accuracy across all items was 75.6%. Idioms reached only 11.6% and resultative predicates 17.8%, while MWE and verb valency were around 66%. - Facebook was in the top significance cluster for 11 categories, averaging 87.5% when categories are weighted equally. RWTH and Online-A led when all items are weighted equally. - Overall accuracy improved by about 6% over WMT18. The largest gains were in function words (+12.5%), non-verbal agreement (+9.7%) and punctuation (+8%). - Among systems submitted under the same name in both years, Online-G improved most (+18.7%). UCAM declined in several categories. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · [Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{avramidis2019linguistic, title = {{Linguistic Evaluation of German-English Machine Translation Using a Test Suite}}, author = {Avramidis, Eleftherios and Macketanz, Vivien and Strohriegel, Ursula and Uszkoreit, Hans}, booktitle = {Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)}, year = {2019}, doi = {10.18653/v1/w19-5351}, url = {https://doi.org/10.18653/v1/w19-5351} } ``` - [← previous Discourse Relation Prediction: Revisiting Word Pairs with Convolutional Networks](https://md.research.elamur.ai/free/en/papers/varia-2019-discourse-relation-prediction-revisiting-word/) - [next → Talking about the weather: Two construals of precipitation events in English](https://md.research.elamur.ai/free/en/papers/levin-2019-talking-about-weather-two-construals/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Conclusion and Further Work Ondej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Lucia Specia, Marco Turchi, and Karin Verspoor, editors. 2018a. Proceedings of the Third Conference on Machine Translation. Association for Computational Linguistics, Belgium, Brussels. [find](https://scholar.google.com/scholar?q=Conclusion+and+Further+Work+Ondej+Bojar%2C+Rajen+Chatterjee%2C+Christian+Federmann%2C+Mark+Fishel%2C+Yvette+Graham%2C+Barry+Haddow%2C+Matthias+Huck%2C+Antonio+Jimeno+Yepes%2C+Philipp+Koehn%2C+Christof+Monz%2C+Matteo+Negr)