--- title: "The Effect of Translationese in Machine — Zhang 2019" description: "WMT news test sets are typically built so that half of the source sentences were originally written in the source language and the other half are human…" authors: ["Mike Zhang", "Antonio Toral"] year: 2019 venue: "Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)" source: "https://aclanthology.org/W19-5208/" doi: "10.18653/v1/w19-5208" arxiv: "1906.08069" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/zhang-2019-effect-translationese-machine-translation-test/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/zhang-2019-effect-translationese-machine-translation-test/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2019](https://md.research.elamur.ai/free/papers/2019/page/4/) / The Effect of Translationese in Machine Translation Test Sets # The Effect of Translationese in Machine Translation Test Sets **Authors:** [Mike Zhang](https://md.research.elamur.ai/free/authors/mike-zhang/), [Antonio Toral](https://md.research.elamur.ai/free/authors/antonio-toral/) · **Year:** 2019 · **Venue:** Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W19-5208/) · [PDF](https://aclanthology.org/W19-5208.pdf) · [DOI 10.18653/v1/w19-5208](https://doi.org/10.18653/v1/w19-5208) · [arXiv 1906.08069](https://arxiv.org/abs/1906.08069) · [OpenAlex](https://openalex.org/W2971043182) **Citations:** 67 citations (openalex-snapshot · 2026-10-14) · [82](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary WMT news test sets are typically built so that half of the source sentences were originally written in the source language and the other half are human translations from the target language (translationese). Earlier work on the WMT 2017 Chinese→English set found that the translationese half was easier for MT systems to translate. That finding concerned a single test set and direction. Zhang and Toral therefore test whether the effect holds more broadly. They recompute direct assessment (DA) human evaluation scores for 17 translation directions from the WMT16–18 news tasks, splitting each test set by the original language of the source documents. They then compare system scores and rankings on the full set, the original-source subset (ORG) and the translationese-source subset (TRS). Across directions, scores on TRS are inflated and scores on ORG are deflated relative to the official results, and rankings sometimes shift. The size of this distortion is strongly and negatively correlated with the quality the best systems reach, while language relatedness explains little. These findings matter for fair MT evaluation and for claims of human parity. ## Key points - **Data and method:** The study uses newstest2016–2018 with standardized DA (z-)scores. Ranking agreement between the full set and ORG is measured with Kendall's τ, and system clusters follow WMT's Wilcoxon rank-sum procedure. - **Consistent inflation:** For the top systems, ORG scores fall below the official WMT score and TRS scores rise above it almost everywhere. English→Finnish at WMT18 is a striking case, at roughly −7.7 on ORG versus +8.0 on TRS. - **Rankings:** In Chinese→English, most top systems move position and clusters shift. A perfect τ of 1 appears mainly in directions with few submitted systems. - **Quality correlation:** At WMT18, the best system's ORG score correlates negatively with the gap between the full set and ORG (Pearson R = −0.84 relative, −0.78 absolute, p \< 0.001). Directions translating out of English tend to show larger gaps. - **Language similarity:** Similarity was computed with lang2vec/URIEL features and showed only weak, non-significant correlations (R ≈ −0.11 and −0.15). Code and data are released on GitHub. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/) Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/) ## Cited by Cited by 82 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 75 not yet. - [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://md.research.elamur.ai/free/papers/alves-2024-tower-open-multilingual-large-language/) - [On The Evaluation of Machine Translation Systems Trained With Back-Translation](https://md.research.elamur.ai/free/papers/edunov-2020-evaluation-machine-translation-systems-trained/) - [BLEU might be Guilty but References are not Innocent](https://md.research.elamur.ai/free/papers/freitag-2020-bleu-might-be-guilty-but/) - [IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://md.research.elamur.ai/free/papers/gala-2023-indictrans2-towards-high-quality-accessible/) - [Statistical Power and Translationese in Machine Translation Evaluation](https://md.research.elamur.ai/free/papers/graham-2020-statistical-power-translationese-machine-translation/) - [Facebook AI's WMT21 News Translation Task Submission](https://md.research.elamur.ai/free/papers/tran-2021-facebook-ai-s-wmt21-news/) - [Lost in Translationese? Reducing Translation Effect Using Abstract Meaning Representation](https://md.research.elamur.ai/free/papers/wein-2024-lost-translationese-reducing-translation-effect/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{zhang2019effect, title = {{The Effect of Translationese in Machine Translation Test Sets}}, author = {Zhang, Mike and Toral, Antonio}, booktitle = {Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)}, year = {2019}, doi = {10.18653/v1/w19-5208}, eprint = {1906.08069}, archivePrefix = {arXiv}, url = {https://doi.org/10.18653/v1/w19-5208} } ``` - [← previous The DISRPT 2019 Shared Task on Elementary Discourse Unit Segmentation and Connective Detection](https://md.research.elamur.ai/free/papers/zeldes-2019-disrpt-2019-shared-task-elementary/) - [next → The First Shared Task on Discourse Representation Structure Parsing](https://md.research.elamur.ai/free/papers/abzianidze-2019-first-shared-task-discourse-representation/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/papers/2019/page/4/) ## References 2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Mona Baker. 1993. Corpus linguistics and translation studies: Implications and applications. Text and technology: In honour of John Sinclair, 233:250. [find](https://scholar.google.com/scholar?q=Mona+Baker%2E+1993%2E+Corpus+linguistics+and+translation+studies%3A+Implications+and+applications%2E+Text+and+technology%3A+In+honour+of+John+Sinclair%2C+233%3A250%2E) 2. Conclusion and Future Work Yevgeni Berzak, Chie Nakamura, Suzanne Flynn, and Boris Katz. 2017. Predicting native language from gaze. arXiv preprint arXiv:1704.07398. [find](https://scholar.google.com/scholar?q=Conclusion+and+Future+Work+Yevgeni+Berzak%2C+Chie+Nakamura%2C+Suzanne+Flynn%2C+and+Boris+Katz%2E+2017%2E+Predicting+native+language+from+gaze%2E+arXiv+preprint+arXiv%3A1704%2E07398%2E)