---
title: "Findings of the WMT 2019 Shared Tasks on — Fonseca 2019"
description: "This report describes the WMT 2019 Quality Estimation (QE) shared task."
authors: ["Erick Fonseca", "Lisa Yankovskaya", "André F. T. Martins", "Mark Fishel", "Christian Federmann"]
year: 2019
venue: "Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)"
source: "https://aclanthology.org/W19-5401/"
doi: "10.18653/v1/w19-5401"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/fonseca-2019-findings-wmt-2019-shared-tasks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/fonseca-2019-findings-wmt-2019-shared-tasks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/) / Findings of the WMT 2019 Shared Tasks on Quality Estimation

# Findings of the WMT 2019 Shared Tasks on Quality Estimation

**Authors:** [Erick Fonseca](https://md.research.elamur.ai/free/en/authors/erick-fonseca/), [Lisa Yankovskaya](https://md.research.elamur.ai/free/en/authors/lisa-yankovskaya/), [André F. T. Martins](https://md.research.elamur.ai/free/en/authors/andre-f-t-martins/), [Mark Fishel](https://md.research.elamur.ai/free/en/authors/mark-fishel/), [Christian Federmann](https://md.research.elamur.ai/free/en/authors/christian-federmann/) · **Year:** 2019 · **Venue:** Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W19-5401/) · [PDF](https://aclanthology.org/W19-5401.pdf) · [DOI 10.18653/v1/w19-5401](https://doi.org/10.18653/v1/w19-5401) · [OpenAlex](https://openalex.org/W2971120958)

**Citations:** 90 citations (openalex-snapshot · 2026-10-14) · [83](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
This report describes the WMT 2019 Quality Estimation (QE) shared task. QE means judging machine translation output when only the source sentence and the system's hypothesis are available, with no reference translation. The organizers ran three tracks:

- **Task 1** covered word- and sentence-level QE. Systems tagged words and gaps as OK or BAD and predicted HTER. It used the English-German IT-domain data from the previous year plus a new English-Russian set of Microsoft interface messages.
- **Task 2** covered document-level QE on English-French Amazon product reviews. Systems predicted MQM scores and also located fine-grained error spans, which was a new subtask.
- **Task 3** was new. It treated QE as a reference-free metric, run jointly with the WMT19 metrics task, and scored outputs from the news translation systems against human judgments.

All Task 1 and Task 2 data came from neural MT, so the setup was simpler than in earlier years. Eleven institutions took part. The strongest systems typically relied on pretrained contextual encoders such as BERT and XLM, combined with transfer learning. In Task 3, reference-free metrics usually fell short of the reference-based BLEU and chrF, especially at segment level. This shows both how far reference-free evaluation has come and how much room it still has to improve.

## Key points
- **Word-level evaluation:** the main metric is F1-Mult, the product of the OK and BAD F1 scores, with systems ranked on the source side. MCC was added because BAD labels are rare. Labels were derived using TERCOM and fast_align alignments.
- **Document-level scoring:** MQM penalizes errors by severity, with weights of 1 for minor, 5 for major and 10 for critical, and scores are truncated at 0. Error-span predictions are scored by F1 based on character overlap.
- **Task 3 setup:**
  - Evaluation used Kendall τ at segment level and Pearson correlation at system level.
  - LASER and LogProb served as baselines.
  - Submissions included YiSi-2, UNI, USFD and the ibm1-based metrics.
- **Task 2 participation:** only two teams entered. The organizers plan to lower the entry barrier, for example by releasing preprocessed data or word-level predictions.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/)

## Cited by

Cited by

83 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 71 not yet.

- [Assessing the Role of Context in Chat Translation Evaluation: Is Context Helpful and Under What Conditions?](https://md.research.elamur.ai/free/en/papers/agrawal-2024-assessing-role-context-chat-translation/)
- [Findings of the WMT 2023 Shared Task on Quality Estimation](https://md.research.elamur.ai/free/en/papers/blain-2023-findings-wmt-2023-shared-task/)
- [The Devil Is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/fernandes-2023-devil-is-errors-leveraging-large/)
- [MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset](https://md.research.elamur.ai/free/en/papers/fomicheva-2022-mlqe-pe-multilingual-quality-estimation/)
- [Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain](https://md.research.elamur.ai/free/en/papers/freitag-2021-results-wmt21-metrics-shared-task/)
- [KoBE: Knowledge-Based Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/gekhman-2020-kobe-knowledge-based-machine-translation/)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://md.research.elamur.ai/free/en/papers/guerreiro-2024-xcomet-transparent-machine-translation-evaluation/)
- [TransQuest: Translation Quality Estimation with Cross-lingual Transformers](https://md.research.elamur.ai/free/en/papers/ranasinghe-2020-transquest-translation-quality-estimation-cross/)
- [COMET: A Neural Framework for MT Evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/)
- [Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing](https://md.research.elamur.ai/free/en/papers/thompson-2020-automatic-machine-translation-evaluation-many/)
- [UniTE: Unified Translation Evaluation](https://md.research.elamur.ai/free/en/papers/wan-2022-unite-unified-translation-evaluation/)
- [Detecting Hallucinated Content in Conditional Neural Sequence Generation](https://md.research.elamur.ai/free/en/papers/zhou-2021-detecting-hallucinated-content-conditional-neural/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{fonseca2019findings,
  title = {{Findings of the WMT 2019 Shared Tasks on Quality Estimation}},
  author = {Fonseca, Erick and Yankovskaya, Lisa and Martins, André F. T. and Fishel, Mark and Federmann, Christian},
  booktitle = {Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)},
  year = {2019},
  doi = {10.18653/v1/w19-5401},
  url = {https://doi.org/10.18653/v1/w19-5401}
}
```

- [← previous Findings of the WMT 2019 Shared Task on Parallel Corpus Filtering for Low-Resource Conditions](https://md.research.elamur.ai/free/en/papers/koehn-2019-findings-wmt-2019-shared-task/)
- [next → Fine-Grained Analysis of Propaganda in News Articles](https://md.research.elamur.ai/free/en/papers/martino-2019-fine-grained-analysis-propaganda-news/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/)

## References

14 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. H. Abdi. 2007. Bonferroni and Šidák corrections for multiple comparisons. In N. J. Salkind, editor, Encyclopedia of Measurement and Statistics, pages 103– 107. Sage, Thousand Oaks, CA. [find](https://scholar.google.com/scholar?q=H%2E+Abdi%2E+2007%2E+Bonferroni+and+S%CC%8Cida%CC%81k+corrections+for+multiple+comparisons%2E+In+N%2E+J%2E+Salkind%2C+editor%2C+Encyclopedia+of+Measurement+and+Statistics%2C+pages+103%E2%80%93+107%2E+Sage%2C+Thousand+Oaks%2C+CA%2E)
2. Mikel Artetxe and Holger Schwenk. 2018. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. CoRR, abs/1812.10464. [↗](https://arxiv.org/abs/1812.10464) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2018-massively-multilingual-sentence-embeddings-zero/)
3. Shamil Chollampatt and Hwee Tou Ng. 2018. Neural quality estimation of grammatical error correction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. [find](https://scholar.google.com/scholar?q=Shamil+Chollampatt+and+Hwee+Tou+Ng%2E+2018%2E+Neural+quality+estimation+of+grammatical+error+correction%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+Brussels)
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL. [↗](https://doi.org/10.18653/v1/n19-1423)
5. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of NAACL. [find](https://scholar.google.com/scholar?q=Chris+Dyer%2C+Victor+Chahuneau%2C+and+Noah+A%2E+Smith%2E+2013%2E+A+simple%2C+fast%2C+and+effective+reparameterization+of+IBM+model+2%2E+In+Proceedings+of+NAACL%2E)
6. Yvette Graham, Qingsong Ma, Johnny Tian-Zheng Wei, and Ondřej Bojar. 2019. Results of the WMT19 Metrics Shared Task. In Proceedings of the Fourth Conference on Machine Translation (WMT), Florence, Italy. [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Qingsong+Ma%2C+Johnny+Tian%2DZheng+Wei%2C+and+Ondr%CC%8Cej+Bojar%2E+2019%2E+Results+of+the+WMT19+Metrics+Shared+Task%2E+In+Proceedings+of+the+Fourth+Conference+on+Machine+Translation+%28WMT%29%2C+Florence%2C+It)
7. Julia Ive, Frédéric Blain, and Lucia Specia. 2018. Deepquest: a framework for neural-based quality estimation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3146–3157. [find](https://scholar.google.com/scholar?q=Julia+Ive%2C+Fre%CC%81de%CC%81ric+Blain%2C+and+Lucia+Specia%2E+2018%2E+Deepquest%3A+a+framework+for+neural%2Dbased+quality+estimation%2E+In+Proceedings+of+the+27th+International+Conference+on+Computational+Linguistics%2C+pages)
8. Fábio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André F. T. Martins. 2019. OpenKiwi: An open source framework for quality estimation. In Proceedings of ACL 2019 System Demonstrations. [find](https://scholar.google.com/scholar?q=Fa%CC%81bio+Kepler%2C+Jonay+Tre%CC%81nous%2C+Marcos+Treviso%2C+Miguel+Vera%2C+and+Andre%CC%81+F%2E+T%2E+Martins%2E+2019%2E+OpenKiwi%3A+An+open+source+framework+for+quality+estimation%2E+In+Proceedings+of+ACL+2019+System+Demonstrations%2E) [our card](https://md.research.elamur.ai/free/en/papers/kepler-2019-openkiwi-open-source-framework-quality/)
9. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual Language Model Pretraining. arXiv preprint arXiv:1901.07291. [↗](https://arxiv.org/abs/1901.07291)
10. Chi kiu Lo. 2019. YiSi - A unified semantic MT quality evaluation and estimation metric for languages with Maja Popović. 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. [find](https://scholar.google.com/scholar?q=Chi+kiu+Lo%2E+2019%2E+YiSi+%2D+A+unified+semantic+MT+quality+evaluation+and+estimation+metric+for+languages+with+Maja+Popovic%CC%81%2E+2015%2E+chrf%3A+character+n%2Dgram+f%2Dscore+for+automatic+mt+evaluation%2E+In+Proceedin) [our card](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/)
11. Maja Popović, David Vilar, Eleftherios Avramidis, and Aljoscha Burchardt. 2011. Evaluation without references: Ibm1 scores as evaluation metrics. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 99–103, Edinburgh, Scotland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maja+Popovic%CC%81%2C+David+Vilar%2C+Eleftherios+Avramidis%2C+and+Aljoscha+Burchardt%2E+2011%2E+Evaluation+without+references%3A+Ibm1+scores+as+evaluation+metrics%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+M)
12. Lucia Specia, Gustavo Paetzold, and Carolina Scarton. 2015. Multi-level translation quality prediction with quest++. In Proceedings of ACL-IJCNLP 2015 System Demonstrations, pages 115–120, Beijing, China. Association for Computational Linguistics and The Asian Federation of Natural Language Processing. [find](https://scholar.google.com/scholar?q=Lucia+Specia%2C+Gustavo+Paetzold%2C+and+Carolina+Scarton%2E+2015%2E+Multi%2Dlevel+translation+quality+prediction+with+quest%2B%2B%2E+In+Proceedings+of+ACL%2DIJCNLP+2015+System+Demonstrations%2C+pages+115%E2%80%93120%2C+Beijing%2C+Ch) [our card](https://md.research.elamur.ai/free/en/papers/specia-2015-multi-level-translation-quality-prediction/)
13. Lisa Yankovskaya, Andre Tättar, and Mark Fishel. 2019. Quality Estimation and Translation Metrics via Pre-trained Word and Sentence Embeddings. In Proceedings of the Fourth Conference on Machine Translation (WMT), Florence, Italy. [find](https://scholar.google.com/scholar?q=Lisa+Yankovskaya%2C+Andre+Ta%CC%88ttar%2C+and+Mark+Fishel%2E+2019%2E+Quality+Estimation+and+Translation+Metrics+via+Pre%2Dtrained+Word+and+Sentence+Embeddings%2E+In+Proceedings+of+the+Fourth+Conference+on+Machine+Tran)
14. Alexander Yeh. 2000. More accurate tests for the statistical significance of result differences. In Coling-2000: the 18th Conference on Computational Linguistics, pages 947–953. [↗](https://doi.org/10.48550/arxiv.cs/0008005)