--- title: "Results of the WMT16 Metrics Shared Task — Bojar 2016" description: "The WMT16 Metrics Shared Task measures how well automatic MT evaluation metrics, which compare system outputs to reference translations, agree with human…" authors: ["Ondřej Bojar", "Yvette Graham", "Amir Kamran", "Miloš Stanojević"] year: 2016 venue: "Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers" source: "https://aclanthology.org/W16-2302/" doi: "10.18653/v1/w16-2302" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/bojar-2016-results-wmt16-metrics-shared-task/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bojar-2016-results-wmt16-metrics-shared-task/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / Results of the WMT16 Metrics Shared Task # Results of the WMT16 Metrics Shared Task **Authors:** [Ondřej Bojar](https://md.research.elamur.ai/free/en/authors/ondrej-bojar/), [Yvette Graham](https://md.research.elamur.ai/free/en/authors/yvette-graham/), [Amir Kamran](https://md.research.elamur.ai/free/en/authors/amir-kamran/), Miloš Stanojević · **Year:** 2016 · **Venue:** Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W16-2302/) · [PDF](https://aclanthology.org/W16-2302.pdf) · [DOI 10.18653/v1/w16-2302](https://doi.org/10.18653/v1/w16-2302) · [OpenAlex](https://openalex.org/W2508316494) **Citations:** 103 citations (openalex-snapshot · 2026-10-14) · [72](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary The WMT16 Metrics Shared Task measures how well automatic MT evaluation metrics, which compare system outputs to reference translations, agree with human judgments of quality. Participants scored outputs from the WMT16 News, IT, Tuning and HimL translation systems: 171 outputs across 18 language pairs pairing English with 12 languages, in three domains (news, IT, medical/health). Metrics were assessed at two levels. At system level, they were correlated with official manual rankings. At segment level, they were checked for agreement with humans on individual sentences. This edition introduced three kinds of human gold standard: Relative Ranking (RR) by researchers, crowd-sourced Direct Assessment (DA) of adequacy, and HUME semantic judgments. It also used a new way to compute confidence intervals. To make system-level conclusions more decisive, the organisers generated large sets of hybrid systems. Character-level metrics such as BEER, chrF and CharacTER performed strongly again. Trained metrics also correlated highly with humans, and BLEU was clearly outperformed by many alternatives. The authors argue that MT development should look beyond BLEU and encourage adding new metrics to the Moses scorer. ## Key points - Participation and baselines: 16 metrics came from 9 research groups. For comparison, the organisers also ran several baseline metrics of their own, among them the error-rate measures WER, PER, TER and CDER as well as BLEU, its sentence-level variant SentBLEU, and NIST. - Test sets and official methods: The test sets were newstest2016, it-test2016 (1,000 sentences) and himl2015 (about 340 judged sentences per pair). The official evaluation still relies on RR, using TrueSkill at system level and Kendall's τ at segment level. - Hybrid systems: 10K hybrid systems per language pair were built by combining outputs of two newstest2016 systems. This was applied to DA evaluation only. - Limits of crowd-sourced DA: DA could be collected only for English and Russian as target languages. At least 15 assessments per segment were used, and repeat judgments from the same worker still correlated above 0.9 with judgments from distinct workers. - HUME segment-level results: In the Williams significance test, chrF3 was the sole winner for en-cs and en-ro. Its Pearson r reached .544 on en-cs and .639 on en-ro. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) ## Cited by Cited by 72 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 52 not yet. - [Assessing the Role of Context in Chat Translation Evaluation: Is Context Helpful and Under What Conditions?](https://md.research.elamur.ai/free/en/papers/agrawal-2024-assessing-role-context-chat-translation/) - [Findings of the 2016 Conference on Machine Translation](https://md.research.elamur.ai/free/en/papers/bojar-2016-findings-2016-conference-machine-translation/) - [Rethinking embedding coupling in pre-trained language models](https://md.research.elamur.ai/free/en/papers/chung-2020-rethinking-embedding-coupling-pre-trained/) - [Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments](https://md.research.elamur.ai/free/en/papers/fomicheva-2019-taking-mt-evaluation-metrics-extremes/) - [Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain](https://md.research.elamur.ai/free/en/papers/freitag-2021-results-wmt21-metrics-shared-task/) - [BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation](https://md.research.elamur.ai/free/en/papers/jiang-2022-blonde-automatic-evaluation-metric-document/) - [Discourse Structure in Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/joty-2017-discourse-structure-machine-translation-evaluation/) - [MEANT 2.0: Accurate semantic MT evaluation for any output language](https://md.research.elamur.ai/free/en/papers/lo-2017-meant-2-0-accurate-semantic/) - [YiSi - a Unified Semantic MT Quality Evaluation and Estimation Metric for Languages with Different Levels of Available Resources](https://md.research.elamur.ai/free/en/papers/lo-2019-yisi-unified-semantic-mt-quality/) - [The (Un)Suitability of Automatic Evaluation Metrics for Text Simplification](https://md.research.elamur.ai/free/en/papers/manchego-2021-un-suitability-automatic-evaluation-metrics/) - [Results of the WMT20 Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/mathur-2020-results-wmt20-metrics-shared-task/) - [The E2E Dataset: New Challenges For End-to-End Generation](https://md.research.elamur.ai/free/en/papers/novikova-2017-e2e-dataset-new-challenges-end/) - [Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics](https://md.research.elamur.ai/free/en/papers/perrella-2024-beyond-correlation-interpretable-evaluation-machine/) - [Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!](https://md.research.elamur.ai/free/en/papers/perrella-2024-guardians-machine-translation-meta-evaluation/) - [How Multilingual is Multilingual BERT?](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/) - [chrF++: words helping character n-grams](https://md.research.elamur.ai/free/en/papers/popovic-2017-chrf-words-helping-character-n/) - [A Structured Review of the Validity of BLEU](https://md.research.elamur.ai/free/en/papers/reiter-2018-structured-review-validity-bleu/) - [RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/shimanaka-2018-ruse-regressor-sentence-embeddings-automatic/) - [Paraphrase Generation as Zero-Shot Multilingual Translation: Disentangling Semantic Similarity from Lexical and Syntactic Diversity](https://md.research.elamur.ai/free/en/papers/thompson-2020-paraphrase-generation-as-zero-shot/) - [CharacTer: Translation Edit Rate on Character Level](https://md.research.elamur.ai/free/en/papers/wang-2016-character-translation-edit-rate-character/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{bojar2016results, title = {{Results of the WMT16 Metrics Shared Task}}, author = {Bojar, Ondřej and Graham, Yvette and Kamran, Amir and Stanojević, Miloš}, booktitle = {Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers}, year = {2016}, doi = {10.18653/v1/w16-2302}, url = {https://doi.org/10.18653/v1/w16-2302} } ``` - [← previous Recognizing Implicit Discourse Relations via Repeated Reading: Neural Networks with Multi-Level Attention](https://md.research.elamur.ai/free/en/papers/liu-2016-recognizing-implicit-discourse-relations-repeated/) - [next → Revisiting the Evaluation for Cross Document Event Coreference](https://md.research.elamur.ai/free/en/papers/upadhyay-2016-revisiting-evaluation-cross-document-event/) - [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) ## References 3 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Omri Abend and Ari Rappoport. 2013. Universal Conceptual Cognitive Annotation (UCCA). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 228–238, Sofia, Bulgaria, August. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Omri+Abend+and+Ari+Rappoport%2E+2013%2E+Universal+Conceptual+Cognitive+Annotation+%28UCCA%29%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29) [our card](https://md.research.elamur.ai/free/en/papers/abend-2013-universal-conceptual-cognitive-annotation-ucca/) 2. Alexandra Birch, Barry Haddow, Ondřej Bojar, and Omri Abend. 2016. Hume: Human ucca-based evaluation of machine translation. arXiv preprint arXiv:1607.00030. [↗](https://arxiv.org/abs/1607.00030) [our card](https://md.research.elamur.ai/free/en/papers/birch-2016-hume-human-ucca-based-evaluation/) 3. Ondřej Bojar, Miloš Ercegovčević, Martin Popel, and Omar Zaidan. 2011. A Grain of Salt for the WMT Manual Evaluation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 1–11, Edinburgh, Scotland, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Milos%CC%8C+Ercegovc%CC%8Cevic%CC%81%2C+Martin+Popel%2C+and+Omar+Zaidan%2E+2011%2E+A+Grain+of+Salt+for+the+WMT+Manual+Evaluation%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2C+pages)