---
title: "Unbabel's Participation in the WMT20 Metrics — Rei 2020"
description: "This paper presents Unbabel's submission to the WMT 2020 Metrics Shared Task."
authors: ["Ricardo Rei", "Craig Stewart", "Ana C Farinha", "Alon Lavie"]
year: 2020
venue: "Proceedings of the Fifth Conference on Machine Translation"
source: "https://aclanthology.org/2020.wmt-1.101/"
doi: "10.18653/v1/2020.wmt-1.101"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rei-2020-unbabel-s-participation-wmt20-metrics/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rei-2020-unbabel-s-participation-wmt20-metrics/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/) / Unbabel's Participation in the WMT20 Metrics Shared Task

# Unbabel's Participation in the WMT20 Metrics Shared Task

**Authors:** [Ricardo Rei](https://md.research.elamur.ai/free/en/authors/ricardo-rei/), [Craig Stewart](https://md.research.elamur.ai/free/en/authors/craig-stewart/), [Ana C Farinha](https://md.research.elamur.ai/free/en/authors/ana-c-farinha/), [Alon Lavie](https://md.research.elamur.ai/free/en/authors/alon-lavie/) · **Year:** 2020 · **Venue:** Proceedings of the Fifth Conference on Machine Translation · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.wmt-1.101/) · [PDF](https://aclanthology.org/2020.wmt-1.101.pdf) · [DOI 10.18653/v1/2020.wmt-1.101](https://doi.org/10.18653/v1/2020.wmt-1.101) · [OpenAlex](https://openalex.org/W3118587867)

**Citations:** 31 citations (openalex-snapshot · 2026-10-14) · [67](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
This paper presents Unbabel's submission to the WMT 2020 Metrics Shared Task. It targets the segment-, document- and system-level tracks and the "QE as a Metric" track. The systems are built on the COMET framework. A cross-lingual XLM-RoBERTa encoder embeds the source, MT hypothesis and reference, pooling turns these into sentence vectors, and a feed-forward head predicts quality.

The team trained three Estimator models on different kinds of human judgement: HTER, MQM and Direct Assessment (DA). They also trained a COMET-RANK model on relative rankings derived from DA. The authors add three techniques:
- combining several references at inference time,
- turning segment scores into a document score with a length-weighted average,
- a Kendall's Tau check at system level, to test how well metrics separate strong systems that perform similarly.

Because evaluation uses WMT19 test sets, the reported numbers are not official WMT20 results. On these sets, the models match or beat BLEU, chrF, BERTScore, BLEURT and PRISM, often setting a new state of the art. The results suggest a neural metric that reads the source is a strong, practical MT evaluator. They also suggest that reference quality matters more than the number of references.

## Key points
- **Estimator setup.** The Estimators use XLM-RoBERTa large with larger feed-forward layers (3072 and 1536 units) and a frozen embedding layer. Learning rates shrink by a factor of 0.95 per layer going down. COMET-RANK stays on the base encoder, because larger encoders made its training unstable.
- **Training data:**
  - an HTER corpus of 211K tuples from QT21 plus APE-QUEST,
  - a proprietary MQM corpus of 27K customer-support tuples, from English into 15 languages or variants,
  - WMT DA/DA RR data covering 24 language pairs, using 2017–2018 for training and 2019 for validation.
- **Document level (en-cs, en-de).** Length-weighted "micro-averaging" slightly outperformed a plain mean. The HTER Estimator scored highest, with an average Pearson r of 0.607, compared with 0.598 for DA and 0.577 for MQM.
- **Multiple references.** For de-en, the alternative reference alone gave r=0.34, versus 0.42 with the original reference. For en-de, extra references from Freitag et al. (2020) did not noticeably change segment-level Kendall's Tau.
- **QE as a Metric.** The reference-free variant simply drops the reference input and shrinks the regressor to match.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/)

## Cited by

Cited by

67 papers of our library cite it (of the 140884 whose bibliographies we read); 13 of them have a page here, 54 not yet.

- [Assessing the Role of Context in Chat Translation Evaluation: Is Context Helpful and Under What Conditions?](https://md.research.elamur.ai/free/en/papers/agrawal-2024-assessing-role-context-chat-translation/)
- [Findings of the WMT 2023 Shared Task on Quality Estimation](https://md.research.elamur.ai/free/en/papers/blain-2023-findings-wmt-2023-shared-task/)
- [Quality-Aware Decoding for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/fernandes-2022-quality-aware-decoding-neural-machine/)
- [MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods](https://md.research.elamur.ai/free/en/papers/finkelstein-2023-mbr-qe-finetuning-training-time/)
- [Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain](https://md.research.elamur.ai/free/en/papers/freitag-2021-results-wmt21-metrics-shared-task/)
- [Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/guerreiro-2023-looking-needle-haystack-comprehensive-study/)
- [Crossing the Threshold: Idiomatic Machine Translation through Retrieval Augmentation and Loss Weighting](https://md.research.elamur.ai/free/en/papers/liu-2023-crossing-threshold-idiomatic-machine-translation/)
- [Results of the WMT20 Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/mathur-2020-results-wmt20-metrics-shared-task/)
- [Predict and Fix: A Unified Model for Translation Quality Estimation and Post-Editing](https://md.research.elamur.ai/free/en/papers/modrzejewski-2026-predict-fix-unified-model-translation/)
- [Learning Compact Metrics for MT](https://md.research.elamur.ai/free/en/papers/pu-2021-learning-compact-metrics-mt/)
- [GrammaMT: Improving Machine Translation with Grammar-Informed In-Context Learning](https://md.research.elamur.ai/free/en/papers/ramos-2025-grammamt-improving-machine-translation-grammar/)
- [Are References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/rei-2021-are-references-really-needed-unbabel/)
- [COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/rei-2022-comet-22-unbabel-ist-2022/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{rei2020unbabel,
  title = {{Unbabel's Participation in the WMT20 Metrics Shared Task}},
  author = {Rei, Ricardo and Stewart, Craig and Farinha, Ana C and Lavie, Alon},
  booktitle = {Proceedings of the Fifth Conference on Machine Translation},
  year = {2020},
  doi = {10.18653/v1/2020.wmt-1.101},
  url = {https://doi.org/10.18653/v1/2020.wmt-1.101}
}
```

- [← previous TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/)
- [next → Uncertain Natural Language Inference](https://md.research.elamur.ai/free/en/papers/chen-2020-uncertain-natural-language-inference/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/)

## References

20 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Loı̈c Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1–61, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Lo%C4%B1%CC%88c+Barrault%2C+Ondr%CC%8Cej+Bojar%2C+Marta+R%2E+Costa%2Djussa%CC%80%2C+Christian+Federmann%2C+Mark+Fishel%2C+Yvette+Graham%2C+Barry+Haddow%2C+Matthias+Huck%2C+Philipp+Koehn%2C+Shervin+Malmasi%2C+Christof+Monz%2C+Mathias+Mu%CC%88ller%2C+Sant) [our card](https://md.research.elamur.ai/free/en/papers/barrault-2019-findings-2019-conference-machine-translation/)
2. Aljoscha Burchardt and Arle Lommel. 2014. Practical Guidelines for the Use of MQM in Scientific Research on Translation quality. Quality Translation 21. (access date: 2020-05-26). [find](https://scholar.google.com/scholar?q=Aljoscha+Burchardt+and+Arle+Lommel%2E+2014%2E+Practical+Guidelines+for+the+Use+of+MQM+in+Scientific+Research+on+Translation+quality%2E+Quality+Translation+21%2E+%28access+date%3A+2020%2D05%2D26%29%2E)
3. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.747)
4. Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be Guilty but References are not Innocent. ArXiv, abs/2004.06063. [↗](https://arxiv.org/abs/2004.06063) [our card](https://md.research.elamur.ai/free/en/papers/freitag-2020-bleu-might-be-guilty-but/)
5. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Timothy+Baldwin%2C+Alistair+Moffat%2C+and+Justin+Zobel%2E+2013%2E+Continuous+measurement+scales+in+human+evaluation+of+machine+translation%2E+In+Proceedings+of+the+7th+Linguistic+Annotation+Works)
6. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volum)
7. Julia Ive, Lucia Specia, Sara Szoc, Tom Vanallemeersch, Joachim Van den Bogaert, Eduardo Farah, Christine Maroti, Artur Ventura, and Maxim Khalilov. 2020. A post-editing dataset in the legal domain: Do we underestimate neural machine translation quality? In Proceedings of The 12th Language Resources and Evaluation Conference, pages 3692–3697, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Julia+Ive%2C+Lucia+Specia%2C+Sara+Szoc%2C+Tom+Vanallemeersch%2C+Joachim+Van+den+Bogaert%2C+Eduardo+Farah%2C+Christine+Maroti%2C+Artur+Ventura%2C+and+Maxim+Khalilov%2E+2020%2E+A+post%2Dediting+dataset+in+the+legal+domain%3A+D)
8. Fabio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, António Góis, M. Amin Farajian, António V. Lopes, and André F. T. Martins. 2019. Unbabel’s participation in the WMT19 translation quality estimation shared task. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 78–84, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Fabio+Kepler%2C+Jonay+Tre%CC%81nous%2C+Marcos+Treviso%2C+Miguel+Vera%2C+Anto%CC%81nio+Go%CC%81is%2C+M%2E+Amin+Farajian%2C+Anto%CC%81nio+V%2E+Lopes%2C+and+Andre%CC%81+F%2E+T%2E+Martins%2E+2019%2E+Unbabel%E2%80%99s+participation+in+the+WMT19+translation+quality)
9. Arle Lommel, Aljoscha Burchardt, and Hans Uszkoreit. 2014. Multidimensional Quality Metrics (MQM): A framework for declaring and describing translation quality metrics. Tradumàtica: tecnologies de la traducció, 0:455–463. [find](https://scholar.google.com/scholar?q=Arle+Lommel%2C+Aljoscha+Burchardt%2C+and+Hans+Uszkoreit%2E+2014%2E+Multidimensional+Quality+Metrics+%28MQM%29%3A+A+framework+for+declaring+and+describing+translation+quality+metrics%2E+Traduma%CC%80tica%3A+tecnologies+de+la)
10. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62–90, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qingsong+Ma%2C+Johnny+Wei%2C+Ondr%CC%8Cej+Bojar%2C+and+Yvette+Graham%2E+2019%2E+Results+of+the+WMT19+metrics+shared+task%3A+Segment%2Dlevel+and+strong+MT+systems+pose+big+challenges%2E+In+Proceedings+of+the+Fourth+Confere) [our card](https://md.research.elamur.ai/free/en/papers/ma-2019-results-wmt19-metrics-shared-task/)
11. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984–4997, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nitika+Mathur%2C+Timothy+Baldwin%2C+and+Trevor+Cohn%2E+2020%2E+Tangled+up+in+BLEU%3A+Reevaluating+the+evaluation+of+automatic+machine+translation+evaluation+metrics%2E+In+Proceedings+of+the+58th+Annual+Meeting+of) [our card](https://md.research.elamur.ai/free/en/papers/mathur-2020-tangled-up-bleu-reevaluating-evaluation/)
12. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. [↗](https://doi.org/10.3115/1073083.1073135) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
13. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A Neural Framework for MT Evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Craig+Stewart%2C+Ana+C+Farinha%2C+and+Alon+Lavie%2E+2020%2E+COMET%3A+A+Neural+Framework+for+MT+Evaluation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+) [our card](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/)
14. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.704)
15. Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. 2019. Machine Translation Evaluation with BERT Regressor. arXiv preprint arXiv:1907.12679. [↗](https://arxiv.org/abs/1907.12679)
16. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In In Proceedings of Association for Machine Translation in the Americas, pages 223–231. [find](https://scholar.google.com/scholar?q=Matthew+Snover%2C+Bonnie+Dorr%2C+Richard+Schwartz%2C+Linnea+Micciulla%2C+and+John+Makhoul%2E+2006%2E+A+study+of+translation+edit+rate+with+targeted+human+annotation%2E+In+In+Proceedings+of+Association+for+Machine+T) [our card](https://md.research.elamur.ai/free/en/papers/snover-2006-study-translation-edit-rate-targeted/)
17. Lucia Specia, Kim Harris, Frédéric Blain, Aljoscha Burchardt, Viviven Macketanz, Inguna Skadina, Matteo Negri, , and Marco Turchi. 2017. Translation quality and productivity: A study on rich morphology languages. In Machine Translation Summit XVI, pages 55–71, Nagoya, Japan. [find](https://scholar.google.com/scholar?q=Lucia+Specia%2C+Kim+Harris%2C+Fre%CC%81de%CC%81ric+Blain%2C+Aljoscha+Burchardt%2C+Viviven+Macketanz%2C+Inguna+Skadina%2C+Matteo+Negri%2C+%2C+and+Marco+Turchi%2E+2017%2E+Translation+quality+and+productivity%3A+A+study+on+rich+morphol)
18. Kosuke Takahashi, Katsuhito Sudoh, and Satoshi Nakamura. 2020. Automatic machine translation evaluation using source language inputs and cross-lingual language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3553–3558, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kosuke+Takahashi%2C+Katsuhito+Sudoh%2C+and+Satoshi+Nakamura%2E+2020%2E+Automatic+machine+translation+evaluation+using+source+language+inputs+and+cross%2Dlingual+language+model%2E+In+Proceedings+of+the+58th+Annual)
19. Brian Thompson and Matt Post. 2020. Automatic machine translation evaluation in many languages via zero-shot paraphrasing. ArXiv, abs/2004.14564. [↗](https://arxiv.org/abs/2004.14564) [our card](https://md.research.elamur.ai/free/en/papers/thompson-2020-automatic-machine-translation-evaluation-many/)
20. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+BERTScore%3A+Evaluating+text+generation+with+BERT%2E+In+International+Conference+on+Learning+Representations%2E)