---
title: "MaTESe: Machine Translation Evaluation as a — Perrella 2022"
description: "Learned regression metrics for machine translation, such as COMET, BLEURT and C-SPEC, correlate well with human judgements."
authors: ["Stefano Perrella", "Lorenzo Proietti", "Alessandro Scirè", "Niccolò Campolungo", "Roberto Navigli"]
year: 2022
venue: "Proceedings of the Seventh Conference on Machine Translation (WMT)"
source: "https://aclanthology.org/2022.wmt-1.51/"
doi: "10.18653/v1/2022.wmt-1.51"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/perrella-2022-matese-machine-translation-evaluation-as/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/perrella-2022-matese-machine-translation-evaluation-as/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2022](https://md.research.elamur.ai/free/papers/2022/page/2/) / MaTESe: Machine Translation Evaluation as a Sequence Tagging Problem

# MaTESe: Machine Translation Evaluation as a Sequence Tagging Problem

**Authors:** [Stefano Perrella](https://md.research.elamur.ai/free/authors/stefano-perrella/), [Lorenzo Proietti](https://md.research.elamur.ai/free/authors/lorenzo-proietti/), [Alessandro Scirè](https://md.research.elamur.ai/free/authors/alessandro-scire/), [Niccolò Campolungo](https://md.research.elamur.ai/free/authors/niccolo-campolungo/), [Roberto Navigli](https://md.research.elamur.ai/free/authors/roberto-navigli/) · **Year:** 2022 · **Venue:** Proceedings of the Seventh Conference on Machine Translation (WMT) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.wmt-1.51/) · [PDF](https://aclanthology.org/2022.wmt-1.51.pdf) · [DOI 10.18653/v1/2022.wmt-1.51](https://doi.org/10.18653/v1/2022.wmt-1.51) · [OpenAlex](https://openalex.org/W7126371397)

**Citations:** 3 citations (openalex-snapshot · 2026-10-14) · [29](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary

Learned regression metrics for machine translation, such as COMET, BLEURT and C-SPEC, correlate well with human judgements. However, they output a single number with no explanation. This paper, submitted to the WMT 2022 Metrics Shared Task, brings the MQM human-evaluation protocol into automatic evaluation. MaTESe treats quality estimation as sequence tagging: each token of a candidate translation is labelled as part of a Major error, a Minor error or no error. The sentence score is the sum of span penalties. The model reads the candidate together with its reference, while the MaTESe-QE variant reads the source sentence instead, so it needs no reference. Training uses the public MQM annotations of Freitag et al. (2021a), simplified to severity only, with the three raters' annotations merged per sentence. The authors also propose two span-level diagnostics. On WMT 2021 news data, the metrics match or beat the leading submissions for English→German and come close for Chinese→English. Accuracy falls on out-of-domain TED data. The approach matters because it produces interpretable error locations rather than an opaque score.

## Key points

- **Scoring:** a Major span costs −5 and a Minor span −1. Sentence scores are capped at a minimum of −25, and corpus scores are sentence averages.
- **Pre-processing:**
  - Non-translation, Omission, Source-error and Neutral annotations are dropped.
  - Critical severity becomes Major, and error categories are discarded.
  - When merged spans overlap, the Major span is kept over the Minor one.
- **New span metrics:** TSH (Target Span Hit) and HSH (Hypothesis Span Hit) score how far predicted error spans partially coincide with gold ones, using micro-averaging.
- **Backbones tested:** XLM-RoBERTa, mBART and RoBERTa. XLM-RoBERTa Large clearly outperforms its Base version.
- **Data limitations:** the training data has about 80K annotated sentences but only about 6K unique sources. The authors blame this for the TED drop. MaTESe-QE trails the reference-based MaTESe only slightly.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/)

Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/)

## Cited by

Cited by

29 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 18 not yet.

- [M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluation](https://md.research.elamur.ai/free/papers/feng-2025-m-mad-multidimensional-multi-agent/)
- [The Devil Is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation](https://md.research.elamur.ai/free/papers/fernandes-2023-devil-is-errors-leveraging-large/)
- [Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent](https://md.research.elamur.ai/free/papers/freitag-2023-results-wmt23-metrics-shared-task/)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://md.research.elamur.ai/free/papers/guerreiro-2024-xcomet-transparent-machine-translation-evaluation/)
- [MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task](https://md.research.elamur.ai/free/papers/juraska-2023-metricx-23-google-submission-wmt/)
- [MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task](https://md.research.elamur.ai/free/papers/juraska-2024-metricx-24-google-submission-wmt/)
- [Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models](https://md.research.elamur.ai/free/papers/lu-2024-error-analysis-prompting-enables-human/)
- [Machine Translation Meta Evaluation through Translation Accuracy Challenge Sets](https://md.research.elamur.ai/free/papers/moghe-2025-machine-translation-meta-evaluation-through/)
- [Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics](https://md.research.elamur.ai/free/papers/perrella-2024-beyond-correlation-interpretable-evaluation-machine/)
- [Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!](https://md.research.elamur.ai/free/papers/perrella-2024-guardians-machine-translation-meta-evaluation/)
- [PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation](https://md.research.elamur.ai/free/papers/proietti-2026-pear-pairwise-evaluation-automatic-relative/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{perrella2022matese,
  title = {{MaTESe: Machine Translation Evaluation as a Sequence Tagging Problem}},
  author = {Perrella, Stefano and Proietti, Lorenzo and Scirè, Alessandro and Campolungo, Niccolò and Navigli, Roberto},
  booktitle = {Proceedings of the Seventh Conference on Machine Translation (WMT)},
  year = {2022},
  doi = {10.18653/v1/2022.wmt-1.51},
  url = {https://doi.org/10.18653/v1/2022.wmt-1.51}
}
```

- [← previous Making Italian Parliamentary Records Machine-Actionable: the Construction of the ParlaMint-IT corpus](https://md.research.elamur.ai/free/papers/agnoloni-2022-making-italian-parliamentary-records-machine/)
- [next → MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction](https://md.research.elamur.ai/free/papers/wang-2022-maven-ere-unified-large-scale/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/papers/2022/page/2/)

## References

20 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Satanjeev+Banerjee+and+Alon+Lavie%2E+2005%2E+METEOR%3A+An+automatic+metric+for+MT+evaluation+with+improved+correlation+with+human+judgments%2E+In+Proceedings+of+the+ACL+Workshop+on+Intrinsic+and+Extrinsic+Eva) [our card](https://md.research.elamur.ai/free/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)
2. Hyung Won Chung, Thibault Fevry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2021. Rethinking embedding coupling in pre-trained language models. In International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.2010.12821) [our card](https://md.research.elamur.ai/free/papers/chung-2020-rethinking-embedding-coupling-pre-trained/)
3. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Timothy+Baldwin%2C+Alistair+Moffat%2C+and+Justin+Zobel%2E+2013%2E+Continuous+measurement+scales+in+human+evaluation+of+machine+translation%2E+In+Proceedings+of+the+7th+Linguistic+Annotation+Works)
4. Lifeng Han, Irina Sorokina, Gleb Erofeev, and Serge Gladkoff. 2021. cushLEPOR: customising hLEPOR metric using optuna for higher agreement with human judgments or pre-trained language model LaBSE. In Proceedings of the Sixth Conference on Machine Translation, pages 1014–1023, Online. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2108.09484)
5. Mateusz Krubiński, Erfan Ghadery, Marie-Francine Moens, and Pavel Pecina. 2021a. Just ask! evaluating machine translation by asking and answering questions. In Proceedings of the Sixth Conference on Machine Translation, pages 495–506, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mateusz+Krubin%CC%81ski%2C+Erfan+Ghadery%2C+Marie%2DFrancine+Moens%2C+and+Pavel+Pecina%2E+2021a%2E+Just+ask%21+evaluating+machine+translation+by+asking+and+answering+questions%2E+In+Proceedings+of+the+Sixth+Conference+on+)
6. Mateusz Krubiński, Erfan Ghadery, Marie-Francine Moens, and Pavel Pecina. 2021b. MTEQA at WMT21 metrics shared task. In Proceedings of the Sixth Conference on Machine Translation, pages 1024–1029, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mateusz+Krubin%CC%81ski%2C+Erfan+Ghadery%2C+Marie%2DFrancine+Moens%2C+and+Pavel+Pecina%2E+2021b%2E+MTEQA+at+WMT21+metrics+shared+task%2E+In+Proceedings+of+the+Sixth+Conference+on+Machine+Translation%2C+pages+1024%E2%80%931029%2C+On)
7. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. [↗](https://doi.org/10.1162/tacl_a_00343)
8. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E)
9. Arle Lommel, Aljoscha Burchardt, and Hans Uszkoreit. 2014. Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics. Tradumàtica: tecnologies de la traducció, 0:455–463. [↗](https://doi.org/10.5565/rev/tradumatica.77)
10. Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020. Results of the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 688–725, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nitika+Mathur%2C+Johnny+Wei%2C+Markus+Freitag%2C+Qingsong+Ma%2C+and+Ondr%CC%8Cej+Bojar%2E+2020%2E+Results+of+the+WMT20+metrics+shared+task%2E+In+Proceedings+of+the+Fifth+Conference+on+Machine+Translation%2C+pages+688%E2%80%93725%2C) [our card](https://md.research.elamur.ai/free/papers/mathur-2020-results-wmt20-metrics-shared-task/)
11. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. USA. Association for Machine Translation in the Americas. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+Association+for+Comp) [our card](https://md.research.elamur.ai/free/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
12. Michal Stefanik, Vít Novotný, and Petr Sojka. 2021. Regressive ensemble for machine translation quality evaluation. In Proceedings of the Sixth Conference on Machine Translation, pages 1041–1048, Online. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2109.07242)
13. Kosuke Takahashi, Yoichi Ishibashi, Katsuhito Sudoh, and Satoshi Nakamura. 2021. Multilingual machine translation evaluation metrics fine-tuned on pseudo-negative examples for WMT 2021 metrics task. In Proceedings of the Sixth Conference on Machine Translation, pages 1049–1052, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kosuke+Takahashi%2C+Yoichi+Ishibashi%2C+Katsuhito+Sudoh%2C+and+Satoshi+Nakamura%2E+2021%2E+Multilingual+machine+translation+evaluation+metrics+fine%2Dtuned+on+pseudo%2Dnegative+examples+for+WMT+2021+metrics+task%2E+I)
14. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81+ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+Neural+Information+Process)
15. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/w15-3049) [our card](https://md.research.elamur.ai/free/papers/popovic-2015-chrf-character-n-gram-f/)
16. Lance Ramshaw and Mitch Marcus. 1995. Text chunking using transformation-based learning. In Third Workshop on Very Large Corpora. [find](https://scholar.google.com/scholar?q=Lance+Ramshaw+and+Mitch+Marcus%2E+1995%2E+Text+chunking+using+transformation%2Dbased+learning%2E+In+Third+Workshop+on+Very+Large+Corpora%2E)
17. Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André F. T. Martins, and Alon Lavie. 2021. Are references really needed? unbabel-IST 2021 submission for the metrics shared task. In Proceedings of the Sixth Conference on Machine Translation, pages 1030–1040, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Ana+C+Farinha%2C+Chrysoula+Zerva%2C+Daan+van+Stigt%2C+Craig+Stewart%2C+Pedro+Ramos%2C+Taisiya+Glushkova%2C+Andr%C3%A9+F%2E+T%2E+Martins%2C+and+Alon+Lavie%2E+2021%2E+Are+references+really+needed%3F+unbabel%2DIST+2021+su) [our card](https://md.research.elamur.ai/free/papers/rei-2021-are-references-really-needed-unbabel/)
18. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Craig+Stewart%2C+Ana+C+Farinha%2C+and+Alon+Lavie%2E+2020%2E+COMET%3A+A+neural+framework+for+MT+evaluation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+) [our card](https://md.research.elamur.ai/free/papers/rei-2020-comet-neural-framework-mt-evaluation/)
19. Thibault Sellam, Amy Pu, Hyung Won Chung, Sebastian Gehrmann, Qijun Tan, Markus Freitag, Dipanjan Das, and Ankur Parikh. 2020. Learning to evaluate translation beyond English: BLEURT submissions to the WMT metrics 2020 shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 921–927, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Amy+Pu%2C+Hyung+Won+Chung%2C+Sebastian+Gehrmann%2C+Qijun+Tan%2C+Markus+Freitag%2C+Dipanjan+Das%2C+and+Ankur+Parikh%2E+2020%2E+Learning+to+evaluate+translation+beyond+English%3A+BLEURT+submissions+to+th) [our card](https://md.research.elamur.ai/free/papers/sellam-2020-learning-evaluate-translation-beyond-english/)
20. Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223–231, Cambridge, Massachusetts, [find](https://scholar.google.com/scholar?q=Matthew+Snover%2C+Bonnie+Dorr%2C+Rich+Schwartz%2C+Linnea+Micciulla%2C+and+John+Makhoul%2E+2006%2E+A+study+of+translation+edit+rate+with+targeted+human+annotation%2E+In+Proceedings+of+the+7th+Conference+of+the+Assoc) [our card](https://md.research.elamur.ai/free/papers/snover-2006-study-translation-edit-rate-targeted/)