---
title: "Results of the WMT20 Metrics Shared Task — Mathur 2020"
description: "The WMT20 Metrics Shared Task benchmarks automatic machine translation evaluation metrics by measuring how closely their scores agree with official human…"
authors: ["Nitika Mathur", "Johnny Wei", "Markus Freitag", "Qingsong Ma", "Ondřej Bojar"]
year: 2020
venue: "Proceedings of the Fifth Conference on Machine Translation"
source: "https://aclanthology.org/2020.wmt-1.77/"
doi: "10.18653/v1/2020.wmt-1.77"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mathur-2020-results-wmt20-metrics-shared-task/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mathur-2020-results-wmt20-metrics-shared-task/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) / Results of the WMT20 Metrics Shared Task

# Results of the WMT20 Metrics Shared Task

**Authors:** [Nitika Mathur](https://md.research.elamur.ai/free/en/authors/nitika-mathur/), [Johnny Wei](https://md.research.elamur.ai/free/en/authors/johnny-wei/), [Markus Freitag](https://md.research.elamur.ai/free/en/authors/markus-freitag/), [Qingsong Ma](https://md.research.elamur.ai/free/en/authors/qingsong-ma/), [Ondřej Bojar](https://md.research.elamur.ai/free/en/authors/ondrej-bojar/) · **Institutions:** [Google](https://md.research.elamur.ai/free/en/companies/google/) · [University of Massachusetts Amherst](https://md.research.elamur.ai/free/en/universities/university-of-massachusetts-amherst/) · [Tencent](https://md.research.elamur.ai/free/en/companies/tencent/) · [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) · [Charles University](https://md.research.elamur.ai/free/en/universities/charles-university/) · [The University of Melbourne](https://md.research.elamur.ai/free/en/universities/the-university-of-melbourne/) · **Year:** 2020 · **Venue:** Proceedings of the Fifth Conference on Machine Translation · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.wmt-1.77/) · [PDF](https://aclanthology.org/2020.wmt-1.77.pdf) · [DOI 10.18653/v1/2020.wmt-1.77](https://doi.org/10.18653/v1/2020.wmt-1.77) · [OpenAlex](https://openalex.org/W3119878165)

**Citations:** 67 citations (OpenAlex, 2026-10-14) · [128](https://md.research.elamur.ai/free/en/papers/mathur-2020-results-wmt20-metrics-shared-task/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
The WMT20 Metrics Shared Task benchmarks automatic machine translation evaluation metrics by measuring how closely their scores agree with official human judgments. Ten teams contributed 27 metrics, four of them reference-free (quality estimation style). The organisers added five baselines, among them BLEU, sentBLEU, TER and chrF computed with SacreBLEU. Metrics scored the outputs of the WMT20 News Translation Task and the parallel corpus filtering task, and they were assessed against Direct Assessment (DA) scores at three granularities: system, segment and, for the first time, whole document. Several additional analyses were run:
- comparing scoring with multiple independent references and with paraphrased references,
- testing how metrics rank human translations,
- identifying systems on which metrics disagree sharply with humans,
- re-annotating disputed cases with professional linguists.

A second reference seldom improved correlation. Lexical metrics consistently undervalued human translations. Reference-free metrics became competitive with reference-based ones. The expert re-annotation tended to side with the metrics, which suggests metrics could help detect unreliable human ratings.

## Key points
- Evaluation covered 208 MT systems in 18 language pairs. Among them were pairs linking English, in both directions, with Tamil, Russian, Polish, Japanese, Inuktitut, German, Czech and Chinese, along with Khmer/Pashto → English from the corpus filtering task.
- Segment-level gold labels (DA RR) were built from pairs of DA scores for the same source segment. A pair was used only when the two scores differed by at least 25 points, and results are reported as Kendall's τ.
- Two independent references were available for en↔de, en↔ru and zh→en. A paraphrased en→de reference set (Freitag et al., 2020) was also provided, and with it every metric ranked human translations above all MT systems.
- COMET-QE recognised high-quality human translations well, whereas BLEU did not. Segment-level correlations were weak for into-English pairs, while out-of-English pairs reached above 0.6.
- The new document-level results gave no clear conclusions, largely because of noise in the human ground truth.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · [Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mathur2020results,
  title = {{Results of the WMT20 Metrics Shared Task}},
  author = {Mathur, Nitika and Wei, Johnny and Freitag, Markus and Ma, Qingsong and Bojar, Ondřej},
  booktitle = {Proceedings of the Fifth Conference on Machine Translation},
  year = {2020},
  doi = {10.18653/v1/2020.wmt-1.77},
  url = {https://doi.org/10.18653/v1/2020.wmt-1.77}
}
```

- [← previous Authorship Attribution for Neural Text Generation](https://md.research.elamur.ai/free/en/papers/uchendu-2020-authorship-attribution-neural-text-generation/)
- [next → End-to-End Slot Alignment and Recognition for Cross-Lingual NLU](https://md.research.elamur.ai/free/en/papers/xu-2020-end-end-slot-alignment-recognition/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/)

## References

49 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Manish Shrivastava Ananya Mukherjee, Hema Ala and Dipti Misra Sharma. 2020. Mee: An automatic metric for evaluation using embeddings for machine translation. (in press). [find](https://scholar.google.com/scholar?q=Manish+Shrivastava+Ananya+Mukherjee%2C+Hema+Ala+and+Dipti+Misra+Sharma%2E+2020%2E+Mee%3A+An+automatic+metric+for+evaluation+using+embeddings+for+machine+translation%2E+%28in+press%29%2E)
2. Loı̈c Barrault, Ondřej Bojar, Marta R Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, et al. 2019. Findings of the 2019 Conference on Machine Translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1–61. [find](https://scholar.google.com/scholar?q=Lo%C4%B1%CC%88c+Barrault%2C+Ondr%CC%8Cej+Bojar%2C+Marta+R+Costa%2Djussa%CC%80%2C+Christian+Federmann%2C+Mark+Fishel%2C+Yvette+Graham%2C+Barry+Haddow%2C+Matthias+Huck%2C+Philipp+Koehn%2C+Shervin+Malmasi%2C+et+al%2E+2019%2E+Findings+of+the+2019+Con) [our card](https://md.research.elamur.ai/free/en/papers/barrault-2019-findings-2019-conference-machine-translation/)
3. Loı̈c Barrault, Magdalena Biesialska, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. 2020. Findings of the 2020 conference on machine translation (wmt20). In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Lo%C4%B1%CC%88c+Barrault%2C+Magdalena+Biesialska%2C+Ondr%CC%8Cej+Bojar%2C+Marta+R%2E+Costa%2Djussa%CC%80%2C+Christian+Federmann%2C+Yvette+Graham%2C+Roman+Grundkiewicz%2C+Barry+Haddow%2C+Matthias+Huck%2C+Eric+Joanis%2C+Tom+Kocmi%2C+Philipp+Koehn%2C+) [our card](https://md.research.elamur.ai/free/en/papers/barrault-2020-findings-2020-conference-machine-translation/)
4. Rachel Bawden, Biao Zhang, Andre Tättar, and Matt Post. 2020. ParBLEU: Augmenting metrics with automatic paraphrases for the WMT’20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation (Volume 2: Shared Task Papers), Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rachel+Bawden%2C+Biao+Zhang%2C+Andre+Ta%CC%88ttar%2C+and+Matt+Post%2E+2020%2E+ParBLEU%3A+Augmenting+metrics+with+automatic+paraphrases+for+the+WMT%E2%80%9920+metrics+shared+task%2E+In+Proceedings+of+the+Fifth+Conference+on+Mach)
5. Ondřej Bojar, Matouš Macháček, Aleš Tamchyna, and Daniel Zeman. 2013. Scratching the surface of possible translations. In Text, Speech, and Dialogue, pages 465–474, Berlin, Heidelberg. Springer Berlin Heidelberg. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Matous%CC%8C+Macha%CC%81c%CC%8Cek%2C+Ales%CC%8C+Tamchyna%2C+and+Daniel+Zeman%2E+2013%2E+Scratching+the+surface+of+possible+translations%2E+In+Text%2C+Speech%2C+and+Dialogue%2C+pages+465%E2%80%93474%2C+Berlin%2C+Heidelberg%2E+Springer+B)
6. Ondřej Bojar, Yvette Graham, and Amir Kamran. 2017. Results of the wmt17 metrics shared task. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 489–513, Copenhagen, Denmark. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Yvette+Graham%2C+and+Amir+Kamran%2E+2017%2E+Results+of+the+wmt17+metrics+shared+task%2E+In+Proceedings+of+the+Second+Conference+on+Machine+Translation%2C+Volume+2%3A+Shared+Task+Papers%2C+pages+489%E2%80%935) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2017-results-wmt17-metrics-shared-task/)
7. Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016. Results of the WMT16 Metrics Shared Task. In Proceedings of the First Conference on Machine Translation, pages 199–231, Berlin, Germany. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Yvette+Graham%2C+Amir+Kamran%2C+and+Milos%CC%8C+Stanojevic%CC%81%2E+2016%2E+Results+of+the+WMT16+Metrics+Shared+Task%2E+In+Proceedings+of+the+First+Conference+on+Machine+Translation%2C+pages+199%E2%80%93231%2C+Berlin%2C) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2016-results-wmt16-metrics-shared-task/)
8. Sheila Castilho, Joss Moorkens, Federico Gaspari, Andy Way, Panayota Georgakopoulou, Maria Gialama, Vilelmini Sosoni, and Rico Sennrich. 2017. Crowdsourcing for nmt evaluation: Professional translators versus the crowd. Translating and the Computer, 39. [find](https://scholar.google.com/scholar?q=Sheila+Castilho%2C+Joss+Moorkens%2C+Federico+Gaspari%2C+Andy+Way%2C+Panayota+Georgakopoulou%2C+Maria+Gialama%2C+Vilelmini+Sosoni%2C+and+Rico+Sennrich%2E+2017%2E+Crowdsourcing+for+nmt+evaluation%3A+Professional+translator)
9. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada. [find](https://scholar.google.com/scholar?q=Qian+Chen%2C+Xiaodan+Zhu%2C+Zhenhua+Ling%2C+Si+Wei%2C+Hui+Jiang%2C+and+Diana+Inkpen%2E+2017%2E+Enhanced+LSTM+for+natural+language+inference%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computat)
10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
11. Markus Dreyer and Daniel Marcu. 2012. HyTER: Meaning-equivalent semantics for translation evaluation. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 162–171, Montréal, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Markus+Dreyer+and+Daniel+Marcu%2E+2012%2E+HyTER%3A+Meaning%2Dequivalent+semantics+for+translation+evaluation%2E+In+Proceedings+of+the+2012+Conference+of+the+North+American+Chapter+of+the+Association+for+Computa)
12. Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 61–71, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Markus+Freitag%2C+David+Grangier%2C+and+Isaac+Caswell%2E+2020%2E+BLEU+might+be+guilty+but+references+are+not+innocent%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing) [our card](https://md.research.elamur.ai/free/en/papers/freitag-2020-bleu-might-be-guilty-but/)
13. Yvette Graham and Timothy Baldwin. 2014. Testing for significance of increased correlation with human judgment. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 172–176, Doha, Qatar. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yvette+Graham+and+Timothy+Baldwin%2E+2014%2E+Testing+for+significance+of+increased+correlation+with+human+judgment%2E+In+Proceedings+of+the+2014+Conference+on+Empirical+Methods+in+Natural+Language+Processin)
14. Yvette Graham, Timothy Baldwin, and Nitika Mathur. 2015. Accurate evaluation of segment-level machine translation metrics. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1183–1191, Denver, Colorado. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Timothy+Baldwin%2C+and+Nitika+Mathur%2E+2015%2E+Accurate+evaluation+of+segment%2Dlevel+machine+translation+metrics%2E+In+Proceedings+of+the+2015+Conference+of+the+North+American+Chapter+of+the+As) [our card](https://md.research.elamur.ai/free/en/papers/graham-2015-accurate-evaluation-segment-level-machine/)
15. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Timothy+Baldwin%2C+Alistair+Moffat%2C+and+Justin+Zobel%2E+2013%2E+Continuous+measurement+scales+in+human+evaluation+of+machine+translation%2E+In+Proceedings+of+the+7th+Linguistic+Annotation+Works)
16. Yvette Graham, Qingsong Ma, Timothy Baldwin, Qun Liu, Carla Parra, and Carolina Scarton. 2017. Improving evaluation of document-level machine translation quality estimation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 356–361, Valencia, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Qingsong+Ma%2C+Timothy+Baldwin%2C+Qun+Liu%2C+Carla+Parra%2C+and+Carolina+Scarton%2E+2017%2E+Improving+evaluation+of+document%2Dlevel+machine+translation+quality+estimation%2E+In+Proceedings+of+the+15th)
17. Yvette Graham, Nitika Mathur, and Timothy Baldwin. 2014. Randomized significance tests in machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 266–274, Baltimore, Maryland, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Nitika+Mathur%2C+and+Timothy+Baldwin%2E+2014%2E+Randomized+significance+tests+in+machine+translation%2E+In+Proceedings+of+the+Ninth+Workshop+on+Statistical+Machine+Translation%2C+pages+266%E2%80%93274%2C+B)
18. Boris Iglewicz and David Caster Hoaglin. 1993. How to detect and handle outliers, volume 16. Asq Press. [find](https://scholar.google.com/scholar?q=Boris+Iglewicz+and+David+Caster+Hoaglin%2E+1993%2E+How+to+detect+and+handle+outliers%2C+volume+16%2E+Asq+Press%2E)
19. Fabio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André F. T. Martins. 2019. OpenKiwi: An open source framework for quality estimation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 117–122, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Fabio+Kepler%2C+Jonay+Tre%CC%81nous%2C+Marcos+Treviso%2C+Miguel+Vera%2C+and+Andre%CC%81+F%2E+T%2E+Martins%2E+2019%2E+OpenKiwi%3A+An+open+source+framework+for+quality+estimation%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+A) [our card](https://md.research.elamur.ai/free/en/papers/kepler-2019-openkiwi-open-source-framework-quality/)
20. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388– 395, Barcelona, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2E+2004%2E+Statistical+significance+tests+for+machine+translation+evaluation%2E+In+Proceedings+of+the+2004+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+388%E2%80%93+395%2C+Barce) [our card](https://md.research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/)
21. Philipp Koehn, Vishrav Chaudhary, Ahmed El-Kishky, Naman Goyal, Peng-Jen Chen, and Francisco Guzmán. 2020. Findings of the wmt 2020 shared task on parallel corpus filtering and alignment. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Vishrav+Chaudhary%2C+Ahmed+El%2DKishky%2C+Naman+Goyal%2C+Peng%2DJen+Chen%2C+and+Francisco+Guzma%CC%81n%2E+2020%2E+Findings+of+the+wmt+2020+shared+task+on+parallel+corpus+filtering+and+alignment%2E+In+Proceedi) [our card](https://md.research.elamur.ai/free/en/papers/koehn-2020-findings-wmt-2020-shared-task/)
22. Samuel Läubli, Sheila Castilho, Graham Neubig, Rico Sennrich, Qinlan Shen, and Antonio Toral. 2020. A set of recommendations for assessing human– machine parity in language translation. Journal of Artificial Intelligence Research, 67:653–672. [find](https://scholar.google.com/scholar?q=Samuel+La%CC%88ubli%2C+Sheila+Castilho%2C+Graham+Neubig%2C+Rico+Sennrich%2C+Qinlan+Shen%2C+and+Antonio+Toral%2E+2020%2E+A+set+of+recommendations+for+assessing+human%E2%80%93+machine+parity+in+language+translation%2E+Journal+of+Ar)
23. Christophe Leys, Christophe Ley, Olivier Klein, Philippe Bernard, and Laurent Licata. 2013. Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. Journal of Experimental Social Psychology, 49(4):764–766. [find](https://scholar.google.com/scholar?q=Christophe+Leys%2C+Christophe+Ley%2C+Olivier+Klein%2C+Philippe+Bernard%2C+and+Laurent+Licata%2E+2013%2E+Detecting+outliers%3A+Do+not+use+standard+deviation+around+the+mean%2C+use+absolute+deviation+around+the+median%2E)
24. Chi-kiu Lo. 2019. YiSi - a unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 507–513, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chi%2Dkiu+Lo%2E+2019%2E+YiSi+%2D+a+unified+semantic+MT+quality+evaluation+and+estimation+metric+for+languages+with+different+levels+of+available+resources%2E+In+Proceedings+of+the+Fourth+Conference+on+Machine+T) [our card](https://md.research.elamur.ai/free/en/papers/lo-2019-yisi-unified-semantic-mt-quality/)
25. Matouš Macháček and Ondřej Bojar. 2014. Results of the WMT14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 293–301, Baltimore, Maryland, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Matous%CC%8C+Macha%CC%81c%CC%8Cek+and+Ondr%CC%8Cej+Bojar%2E+2014%2E+Results+of+the+WMT14+metrics+shared+task%2E+In+Proceedings+of+the+Ninth+Workshop+on+Statistical+Machine+Translation%2C+pages+293%E2%80%93301%2C+Baltimore%2C+Maryland%2C+USA%2E+) [our card](https://md.research.elamur.ai/free/en/papers/machacek-2014-results-wmt14-metrics-shared-task/)
26. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2799–2808, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nitika+Mathur%2C+Timothy+Baldwin%2C+and+Trevor+Cohn%2E+2019%2E+Putting+evaluation+in+context%3A+Contextual+embeddings+improve+machine+translation+evaluation%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Ass)
27. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984–4997, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nitika+Mathur%2C+Timothy+Baldwin%2C+and+Trevor+Cohn%2E+2020%2E+Tangled+up+in+BLEU%3A+Reevaluating+the+evaluation+of+automatic+machine+translation+evaluation+metrics%2E+In+Proceedings+of+the+58th+Annual+Meeting+of) [our card](https://md.research.elamur.ai/free/en/papers/mathur-2020-tangled-up-bleu-reevaluating-evaluation/)
28. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002a. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002a%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+Association+for+Com) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
29. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002b. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL 2002), pages 311– 318, Philadelphia, USA. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002b%2E+BLEU%3A+A+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+Association+for+Com) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
30. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. [find](https://scholar.google.com/scholar?q=Maja+Popovic%CC%81%2E+2015%2E+chrF%3A+character+n%2Dgram+F%2Dscore+for+automatic+MT+evaluation%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages+392%E2%80%93395%2C+Lisbon%2C+Portugal%2E) [our card](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/)
31. Chi-kiu Lo. 2020. Extended study on using pretrained language models and YiSi-1 for machine translation evaluation. In Proceedings of the Fifth Conference on Machine Translation: Shared Task Papers. [find](https://scholar.google.com/scholar?q=Chi%2Dkiu+Lo%2E+2020%2E+Extended+study+on+using+pretrained+language+models+and+YiSi%2D1+for+machine+translation+evaluation%2E+In+Proceedings+of+the+Fifth+Conference+on+Machine+Translation%3A+Shared+Task+Papers%2E)
32. Chi-kiu Lo and Samuel Larkin. 2020. Machine translation reference-less evaluation using YiSi-2 with bilingual mappings of massive multilingual language model. In Proceedings of the Fifth Conference on Machine Translation: Shared Task Papers. [find](https://scholar.google.com/scholar?q=Chi%2Dkiu+Lo+and+Samuel+Larkin%2E+2020%2E+Machine+translation+reference%2Dless+evaluation+using+YiSi%2D2+with+bilingual+mappings+of+massive+multilingual+language+model%2E+In+Proceedings+of+the+Fifth+Conference+on)
33. Qingsong Ma, Ondřej Bojar, and Yvette Graham. 2018. Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 671–688, Belgium, Brussels. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qingsong+Ma%2C+Ondr%CC%8Cej+Bojar%2C+and+Yvette+Graham%2E+2018%2E+Results+of+the+WMT18+metrics+shared+task%3A+Both+characters+and+embeddings+achieve+good+performance%2E+In+Proceedings+of+the+Third+Conference+on+Machin) [our card](https://md.research.elamur.ai/free/en/papers/ma-2018-results-wmt18-metrics-shared-task/)
34. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62–90, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qingsong+Ma%2C+Johnny+Wei%2C+Ondr%CC%8Cej+Bojar%2C+and+Yvette+Graham%2E+2019%2E+Results+of+the+WMT19+metrics+shared+task%3A+Segment%2Dlevel+and+strong+MT+systems+pose+big+challenges%2E+In+Proceedings+of+the+Fourth+Confere) [our card](https://md.research.elamur.ai/free/en/papers/ma-2019-results-wmt19-metrics-shared-task/)
35. Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/w17-4770) [our card](https://md.research.elamur.ai/free/en/papers/popovic-2017-chrf-words-helping-character-n/)
36. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Belgium, Brussels. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Matt+Post%2E+2018%2E+A+call+for+clarity+in+reporting+BLEU+scores%2E+In+Proceedings+of+the+Third+Conference+on+Machine+Translation%3A+Research+Papers%2C+pages+186%E2%80%93+191%2C+Belgium%2C+Brussels%2E+Association+for+Computa) [our card](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/)
37. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020a. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Craig+Stewart%2C+Ana+C+Farinha%2C+and+Alon+Lavie%2E+2020a%2E+COMET%3A+A+neural+framework+for+MT+evaluation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing) [our card](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/)
38. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020b. Unbabel’s participation in the wmt20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Craig+Stewart%2C+Ana+C+Farinha%2C+and+Alon+Lavie%2E+2020b%2E+Unbabel%E2%80%99s+participation+in+the+wmt20+metrics+shared+task%2E+In+Proceedings+of+the+Fifth+Conference+on+Machine+Translation%2C+Online%2E+Assoc) [our card](https://md.research.elamur.ai/free/en/papers/rei-2020-unbabel-s-participation-wmt20-metrics/)
39. Peter J Rousseeuw and Mia Hubert. 2011. Robust statistics for outlier detection. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 1(1):73–79. [find](https://scholar.google.com/scholar?q=Peter+J+Rousseeuw+and+Mia+Hubert%2E+2011%2E+Robust+statistics+for+outlier+detection%2E+Wiley+Interdisciplinary+Reviews%3A+Data+Mining+and+Knowledge+Discovery%2C+1%281%29%3A73%E2%80%9379%2E)
40. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020a. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Dipanjan+Das%2C+and+Ankur+Parikh%2E+2020a%2E+BLEURT%3A+Learning+robust+metrics+for+text+generation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C)
41. Thibault Sellam, Amy Pu, Hyung Won Chung, Sebastian Gehrmann, Qijun Tan, Markus Freitag, Dipanjan Das, and Ankur Parikh. 2020b. Learning to evaluate translation beyond english: Bleurt submissions to the wmt metrics 2020 shared task. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Amy+Pu%2C+Hyung+Won+Chung%2C+Sebastian+Gehrmann%2C+Qijun+Tan%2C+Markus+Freitag%2C+Dipanjan+Das%2C+and+Ankur+Parikh%2E+2020b%2E+Learning+to+evaluate+translation+beyond+english%3A+Bleurt+submissions+to+t) [our card](https://md.research.elamur.ai/free/en/papers/sellam-2020-learning-evaluate-translation-beyond-english/)
42. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the Association for Machine Transaltion in the Americas, pages 223–231. [find](https://scholar.google.com/scholar?q=Matthew+Snover%2C+Bonnie+Dorr%2C+Richard+Schwartz%2C+Linnea+Micciulla%2C+and+John+Makhoul%2E+2006%2E+A+study+of+translation+edit+rate+with+targeted+human+annotation%2E+In+Proceedings+of+the+Association+for+Machine+) [our card](https://md.research.elamur.ai/free/en/papers/snover-2006-study-translation-edit-rate-targeted/)
43. Peter Stanchev, Weiyue Wang, and Hermann Ney. 2019. Eed: Extended edit distance measure for machine translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 514–520, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Peter+Stanchev%2C+Weiyue+Wang%2C+and+Hermann+Ney%2E+2019%2E+Eed%3A+Extended+edit+distance+measure+for+machine+translation%2E+In+Proceedings+of+the+Fourth+Conference+on+Machine+Translation+%28Volume+2%3A+Shared+Task+P)
44. Brian Thompson and Matt Post. 2020. Automatic machine translation evaluation in many languages via zero-shot paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Brian+Thompson+and+Matt+Post%2E+2020%2E+Automatic+machine+translation+evaluation+in+many+languages+via+zero%2Dshot+paraphrasing%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Languag) [our card](https://md.research.elamur.ai/free/en/papers/thompson-2020-automatic-machine-translation-evaluation-many/)
45. Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 113– 123, Belgium, Brussels. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Antonio+Toral%2C+Sheila+Castilho%2C+Ke+Hu%2C+and+Andy+Way%2E+2018%2E+Attaining+the+Unattainable%3F+Reassessing+Claims+of+Human+Parity+in+Neural+Machine+Translation%2E+In+Proceedings+of+the+Third+Conference+on+Machi) [our card](https://md.research.elamur.ai/free/en/papers/toral-2018-attaining-unattainable-reassessing-claims-human/)
46. Tereza Vojtěchová, Michal Novák, Miloš Klouček, and Ondřej Bojar. 2019. SAO WMT19 Test Suite: Machine Translation of Audit Reports. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tereza+Vojte%CC%8Cchova%CC%81%2C+Michal+Nova%CC%81k%2C+Milos%CC%8C+Klouc%CC%8Cek%2C+and+Ondr%CC%8Cej+Bojar%2E+2019%2E+SAO+WMT19+Test+Suite%3A+Machine+Translation+of+Audit+Reports%2E+In+Proceedings+of+the+Fourth+Conference+on+Machine+Translation)
47. Weiyue Wang, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. 2016. CharacTer: Translation edit rate on character level. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 505–510, Berlin, Germany. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Weiyue+Wang%2C+Jan%2DThorsten+Peter%2C+Hendrik+Rosendahl%2C+and+Hermann+Ney%2E+2016%2E+CharacTer%3A+Translation+edit+rate+on+character+level%2E+In+Proceedings+of+the+First+Conference+on+Machine+Translation%3A+Volume+2%2C) [our card](https://md.research.elamur.ai/free/en/papers/wang-2016-character-translation-edit-rate-character/)
48. Evan James Williams. 1959. Regression analysis. wiley. [find](https://scholar.google.com/scholar?q=Evan+James+Williams%2E+1959%2E+Regression+analysis%2E+wiley%2E)
49. Jin Xu, Yinuo Guo, and Junfeng Hu. 2020. Incorporate semantic structures into machine translation evaluation via ucca. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jin+Xu%2C+Yinuo+Guo%2C+and+Junfeng+Hu%2E+2020%2E+Incorporate+semantic+structures+into+machine+translation+evaluation+via+ucca%2E+In+Proceedings+of+the+Fifth+Conference+on+Machine+Translation%2C+Online%2E+Associatio) [our card](https://md.research.elamur.ai/free/en/papers/xu-2020-incorporate-semantic-structures-into-machine/)