---
title: "Meta-evaluation of Automatic Machine Translation Metrics"
description: "The paper tackles a practical gap: no one had checked which automatic MT evaluation metrics can be… 2 references, cited by 2 papers here."
authors: ["Paolo Di Natale", "Elena Chiocchetti", "Egon Waldemar Stemle"]
year: 2025
venue: "Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025)"
source: "https://aclanthology.org/2025.clicit-1.38/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/natale-2025-meta-evaluation-automatic-machine-translation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/natale-2025-meta-evaluation-automatic-machine-translation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/13/) / Meta-evaluation of Automatic Machine Translation Metrics between Italian and a Minor Language Variety of German

# Meta-evaluation of Automatic Machine Translation Metrics between Italian and a Minor Language Variety of German

**Authors:** [Paolo Di Natale](https://md.research.elamur.ai/free/en/authors/paolo-di-natale/), [Elena Chiocchetti](https://md.research.elamur.ai/free/en/authors/elena-chiocchetti/) and Egon Waldemar Stemle · **Year:** 2025 · **Venue:** [Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025)](https://md.research.elamur.ai/free/en/venues/clicit/) ([Italian Conference on Computational Linguistics 2025](https://md.research.elamur.ai/free/en/venues/clicit/2025/)) · **Publisher:** [CEUR Workshop Proceedings](https://md.research.elamur.ai/free/en/publishers/ceur-workshop-proceedings/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.clicit-1.38/) · [PDF](https://aclanthology.org/2025.clicit-1.38.pdf) · [OpenAlex](https://openalex.org/W7112301680)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/natale-2025-meta-evaluation-automatic-machine-translation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper tackles a practical gap: no one had checked which automatic MT evaluation metrics can be trusted for Italian and South Tyrolean German. South Tyrolean German is a low-resourced standard variety of German that is co-official in the province of Bolzano/Bozen. There, translated legal and administrative texts can become legally binding, and machine translation is increasingly used, often by non-specialists. Earlier South Tyrolean MT work relied on lexical-overlap scores simply because no localized validation existed.

The authors run a segment-level meta-evaluation on MT@BZ, a corpus of machine-translated bilingual decrees covering both translation directions. Human error tags are converted into quality scores with custom penalty weights. String-based, embedding-based and learned metrics are then compared on how well they agree with these human judgments, across different quality ranges.

Embedding-based metrics do best on high-quality output. Learned neural metrics track humans better when translations are weaker. The study also finds a persistent bias against the minor variety. These results matter for choosing evaluation tools, for planning annotated resources, and for safely using metrics in Minimum Bayes Risk (MBR) decoding.

## Key points
- **Dataset.** MT@BZ is a manually error-annotated legal corpus of decrees, evaluated in both directions, IT→DE and DE→IT. Segments are sentences.
- **Annotation and scoring.**
  - Annotation follows a custom scheme based on the SCATE taxonomy, which separates accuracy errors from fluency errors.
  - Each error type carries a penalty weight. For example, Part of Speech and Untranslated errors cost 25, many other accuracy errors cost 15–20, and Punctuation and Capitalization cost 5.
- **Embedding-based subscores.** Variants built on bert-base-multilingual, bart-large-mnli, deberta-xlarge-mnli and roberta-large-mnli were analysed.
  - On the whole dataset, IT→DE accuracy correlation is about 0.77–0.78.
  - On the mistake-only subset, it falls to about 0.52–0.55.
- **Motivation.**
  - About 350,000 German speakers (69% of the population) have the right to public services in German.
  - Where the two versions diverge, the Italian text prevails.
  - MT systems tend to slip in terminology from other German legal systems, such as *kommunale Steuer* instead of South Tyrolean *Gemeindesteuer*.
- **Main finding.** Which metric family performs best depends on translation quality: embedding-based metrics lead at the high end, learned metrics in lower ranges. Metrics also show bias against minor-variety variants.

## Interesting ideas
- **Stratify meta-evaluation by quality level.** Report correlation on the full set and separately on an error-only subset, because metric rankings can flip between the two and a pooled number hides this.
- **Turn categorical error tags into a graded score with a weighted taxonomy.** Meaning-level accuracy errors cost far more than surface slips like punctuation, which gives a scalar target that reflects what actually matters.
- **Keep the optimizer and the judge apart.** Decoding (as in MBR) and final evaluation should not use the same metric, or even closely correlated metrics of the same type, because the system learns their quirks.
- **Decompose a similarity score into precision and recall.** This shows whether a metric mainly punishes additions or omissions, making an opaque number more diagnosable.
- **Test whether a metric unfairly penalizes regional lexical variants that keep the meaning.** An example is the South Tyrolean legal term versus the Germany-German one, an evaluation axis rarely checked.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Evaluate parser and reader metrics separately on the hard slice (sentences where witness parsers disagree or answers fail), so an aggregate score cannot mask weakness exactly where errors live.
- **I2** (back): Score text→graph→text round trips with weighted error classes, so that losing a semantic role or entity costs much more than a morphology or punctuation difference.
- **I3** (learn): When silver data comes from agreement among witness parsers, keep architecturally similar witnesses from dominating, and evaluate with an independent gold set so shared biases are not rewarded.
- **I4** (back): Report precision and recall of the meaning graph in compressor and generation tests, so additions (hallucinated facts) and omissions (dropped facts) are tracked as distinct failure modes.
- **I5** (world): Map regional or dialectal synonyms to the same ontology concept, and add controls checking that a variant word does not change the extracted entities or the answer.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Masaryk University](https://md.research.elamur.ai/free/en/universities/masaryk-university/) — University, Brno, Czechia · 10 works · 437 citations
- [Eurac Research](https://md.research.elamur.ai/free/en/labs/eurac-research/) — Research institute, Bolzano, Italy · 2 works · 30 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 1,087 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Paolo Di Natale, Elena Chiocchetti and Egon Waldemar Stemle. 2025. Meta-evaluation of Automatic Machine Translation Metrics between Italian and a Minor Language Variety of German. Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{natale2025meta,
  title = {{Meta-evaluation of Automatic Machine Translation Metrics between Italian and a Minor Language Variety of German}},
  author = {Paolo {Di Natale}, Elena Chiocchetti and Egon Waldemar Stemle},
  booktitle = {Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025)},
  year = {2025},
  url = {https://aclanthology.org/2025.clicit-1.38/}
}
```

- [← previous Lemmatization as a Classification Task: Results from Arabic across Multiple Genres](https://md.research.elamur.ai/free/en/papers/saeed-2025-lemmatization-as-classification-task-results/)
- [next → Mind the Dialect: NLP Advancements Uncover Fairness Disparities for Arabic Users in Recommendation Systems](https://md.research.elamur.ai/free/en/papers/alshabanah-2025-mind-dialect-nlp-advancements-uncover/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/13/)

## References

2 references, 2 of them with a card of ours. ↗ leads to the work.

1. in: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), ELRA and ICCL, Torino, Italia, 2024, pp. 3553–3565. URL: https://aclanthology.org/2024.lrec-main.315/. [5] A. Magueresse, V. Carles, E. Heetderks, Low-resource languages: A review of past work and future challenges, 2020. URL: https://arxiv.org/abs/ 2006.07264. arXiv:2006.07264 . [6] R. Knowles, S. Larkin, C.-K. Lo, MSLC24: Further challenges for metrics on a wide landscape of translation quality, in: Proceedings of the Ninth Conference on Machine Translation, Association for Computational Linguistics, Miami, Florida, USA, 2024, pp. 475–491. URL: https://aclanthology.org/2024. wmt-1.34/. doi: 10.18653/v1/2024.wmt-1.34 . [7] V. Dewangan, B. R. S, G. Suri, R. Sonavane, When every token counts: Optimal segmentation for low-resource language models, in: Proceedings of the First Workshop on Language Models for Low-Resource Languages, Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2025, pp. 294–308. URL: https://aclanthology. org/2025.loreslm-1.24/. [8] F. De Camillis, E. W. Stemle, E. Chiocchetti, F. Fernicola, The MT@BZ corpus: machine translation & legal language, in: Proceedings of the 24th Annual Conference of the European Association for Machine Translation, European Association for Machine Translation, Tampere, Finland, 2023, pp. 171– 180. URL: https://aclanthology.org/2023.eamt-1.17/. [9] A. Oliver, S. Alvarez-Vidal, E. Stemle, E. Chiocchetti, Training an NMT system for legal texts of a low-resource language variety south tyrolean German - Italian, in: Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1), European Association for Machine Translation (EAMT), Sheffield, UK, 2024, pp. 573–579. URL: https://aclanthology. org/2024.eamt-1.47/. [10] S. Perrella, L. Proietti, A. Scirè, E. Barba, R. Navigli, Guardians of the machine translation meta-evaluation: Sentinel metrics fall in!, in: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Bangkok, Thailand, 2024, pp. 16216–16244. URL: https://aclanthology.org/2024.acl-long.856/. doi: 10.18653/v1/2024.acl-long.856 . [11] T. Kocmi, C. Federmann, R. Grundkiewicz, M. Junczys-Dowmunt, H. Matsushita, A. Menezes, To ship or not to ship: An extensive evaluation of automatic metrics for machine translation, in: Proceedings of the Sixth Conference on Machine Translation, Association for Computational Linguistics, Online, 2021, pp. 478–494. URL: https://aclanthology.org/2021.wmt-1.57/. [12] M. Freitag, G. Foster, D. Grangier, V. Ratnakar, Q. Tan, W. Macherey, Experts, errors, and context: A large-scale study of human evaluation for machine translation, Transactions of the Association for Computational Linguistics 9 (2021) 1460– 1474. URL: https://aclanthology.org/2021.tacl-1.87/. doi: 10.1162/tacl_a_00437 . [13] F. D. Camillis, La traduzione non professionale nelle istituzioni pubbliche dei territori di lingua minoritaria: il caso di studio dell’amministrazione della Provincia autonoma di Bolzano, Ph.D. thesis, alma, 2021. URL: https://amsdottorato.unibo.it/id/eprint/ 9695/. [14] F. De Camillis, E. Chiocchetti, Machine-translating legal language: error analysis on an italian-german corpus of decrees, Terminology science & research 27 (2024) 1–27. URL: https://journal-eaft-aet.net/ index.php/tsr/article/view/8304/7492. [15] J. O. Alabi, D. I. Adelani, M. Mosbach, D. Klakow, Adapting pre-trained language models to African languages via multilingual adaptive fine-tuning, in: Proceedings of the 29th International Conference on Computational Linguistics, International Committee on Computational Linguistics, Gyeongju, Republic of Korea, 2022, pp. 4336–4349. URL: https://aclanthology.org/2022.coling-1.382/. [16] J. Sun, T. Sellam, E. Clark, T. Vu, T. Dozat, D. Garrette, A. Siddhant, J. Eisenstein, S. Gehrmann, Dialect-robust evaluation of generated text, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Toronto, Canada, 2023, pp. 6010–6028. URL: https://aclanthology.org/2023.acl-long.331/. doi: 10.18653/v1/2023.acl-long.331 . [17] C. Amrhein, N. Moghe, L. Guillou, [ACES: Translation accuracy challenge sets for evaluating machine translation metrics](https://md.research.elamur.ai/free/en/papers/amrhein-2022-aces-translation-accuracy-challenge-sets/), in: Proceedings of the Seventh Conference on Machine Translation (WMT), Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Hybrid), 2022, pp. 479–513. URL: https://aclanthology.org/2022.wmt-1.44/. [18] Y. Yan, T. Wang, C. Zhao, S. Huang, J. Chen, M. Wang, BLEURT has universal translations: An analysis of automatic metrics by minimum risk training, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Toronto, Canada, 2023, pp. 5428–5443. URL: https://aclanthology. org/2023.acl-long.297/. doi: 10.18653/v1/2023. acl-long.297 . [19] P. Fernandes, A. Farinhas, R. Rei, J. G. C. de Souza, P. Ogayo, G. Neubig, A. Martins, Quality-aware decoding for neural machine translation, in: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Seattle, United States, 2022, pp. 1396–1412. URL: https://aclanthology.org/2022.naacl-main.100/. doi: 10. 18653/v1/2022.naacl-main.100 . [20] G. Kovacs, D. Deutsch, M. Freitag, Mitigating metric bias in minimum Bayes risk decoding, in: Proceedings of the Ninth Conference on Machine Translation, Association for Computational Linguistics, Miami, Florida, USA, 2024, pp. 1063–1094. URL: https://aclanthology.org/2024.wmt-1.109/. doi: 10. 18653/v1/2024.wmt-1.109 . [21] J. Pombal, N. M. Guerreiro, R. Rei, A. F. T. Martins, Adding chocolate to mint: Mitigating metric interference in machine translation, 2025. URL: https://arxiv.org/abs/2503.08327. arXiv:2503.08327 . [22] M. Freitag, N. Mathur, D. Deutsch, C.-K. Lo, E. Avramidis, R. Rei, B. Thompson, F. Blain, T. Kocmi, J. Wang, D. I. Adelani, M. Buchicchio, C. Zerva, A. Lavie, Are LLMs breaking MT metrics? results of the WMT24 metrics shared task, in: Proceedings of the Ninth Conference on Machine Translation, Association for Computational Linguistics, Miami, Florida, USA, 2024, pp. 47– 81. URL: https://aclanthology.org/2024.wmt-1.2/. doi: 10.18653/v1/2024.wmt-1.2 . [23] N. Moghe, T. Sherborne, M. Steedman, A. Birch, Extrinsic evaluation of machine translation metrics, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Toronto, Canada, 2023, pp. 13060–13078. URL: https://aclanthology.org/2023.acl-long.730/. doi: 10.18653/v1/2023.acl-long.730 . [24] S. Agrawal, A. Farajian, P. Fernandes, R. Rei, A. F. T. Martins, Assessing the role of context in chat translation evaluation: Is context helpful and under what conditions?, Transactions of the Association for Computational Linguistics 12 (2024) 1250– 1267. URL: https://aclanthology.org/2024.tacl-1.69/. doi: 10.1162/tacl_a_00700 . [25] R. Rei, N. M. Guerreiro, M. Treviso, L. Coheur, A. Lavie, A. Martins, The inside story: Towards better understanding of machine translation neural evaluation metrics, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Association for Computational Linguistics, Toronto, Canada, 2023, pp. 1089–1105. URL: https://aclanthology.org/2023.acl-short.94/. doi: 10. 18653/v1/2023.acl-short.94 . [↗](https://doi.org/10.18653/v1/2024.wmt-1.34)
2. [26] M. Freitag, N. Mathur, C.-k. Lo, E. Avramidis, R. Rei, B. Thompson, T. Kocmi, F. Blain, D. Deutsch, C. Stewart, C. Zerva, S. Castilho, A. Lavie, G. Foster, [Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent](https://md.research.elamur.ai/free/en/papers/freitag-2023-results-wmt23-metrics-shared-task/), in: Proceedings of the Eighth Conference on Machine Translation, Association for Computational Linguistics, Singapore, 2023, pp. 578– 628. URL: https://aclanthology.org/2023.wmt-1.51/. doi: 10.18653/v1/2023.wmt-1.51 . [27] N. Moghe, A. Fazla, C. Amrhein, T. Kocmi, M. Steedman, A. Birch, R. Sennrich, L. Guillou, Machine translation meta evaluation through translation accuracy challenge sets, Computational Linguistics 51 (2025) 73–137. URL: https://aclanthology.org/2025. cl-1.4/. doi: 10.1162/coli_a_00537 . [28] E. Avramidis, S. Manakhimova, V. Macketanz, S. Möller, Machine translation metrics are better in evaluating linguistic errors on LLMs than on encoder-decoder systems, in: Proceedings of the Ninth Conference on Machine Translation, Association for Computational Linguistics, Miami, Florida, USA, 2024, pp. 517–528. URL: https://aclanthology.org/2024.wmt-1.37/. doi: 10.18653/ v1/2024.wmt-1.37 . [29] V. Zouhar, S. Ding, A. Currey, T. Badeka, J. Wang, B. Thompson, Fine-tuned machine translation metrics struggle in unseen domains, in: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Association for Computational Linguistics, Bangkok, Thailand, 2024, pp. 488–500. URL: https://aclanthology.org/2024.acl-short.45/. doi: 10. 18653/v1/2024.acl-short.45 . [30] A. Lommel, S. Gladkoff, A. Melby, S. E. Wright, I. Strandvik, K. Gasova, A. Vaasa, A. Benzo, R. Marazzato Sparano, M. Foresi, J. Innis, L. Han, G. Nenadic, The multi-range theory of translation quality measurement: MQM scoring models and statistical quality control, in: Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 2: Presentations), Association for Machine Translation in the Americas, Chicago, USA, 2024, pp. 75–94. URL: https://aclanthology.org/2024. amta-presentations.6/. [31] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: a method for automatic evaluation of machine translation, in: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 2002, pp. 311–318. URL: https://aclanthology.org/P02-1040/. doi: 10.3115/1073083.1073135 . [32] T. Sellam, D. Das, A. Parikh, BLEURT: Learning robust metrics for text generation, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, 2020, pp. 7881–7892. URL: https://aclanthology.org/2020.acl-main.704/. doi: 10.18653/v1/2020.acl-main.704 . [33] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, Y. Artzi, Bertscore: Evaluating text generation with bert, 2020. URL: https://arxiv.org/abs/1904.09675. arXiv:1904.09675 . [34] M. Popović, chrF: character n-gram F-score for automatic MT evaluation, in: Proceedings of the Tenth Workshop on Statistical Machine Translation, Association for Computational Linguistics, Lisbon, Portugal, 2015, pp. 392–395. URL: https://aclanthology. org/W15-3049/. doi: 10.18653/v1/W15-3049 . [35] R. Rei, C. Stewart, A. C. Farinha, A. Lavie, COMET: A neural framework for MT evaluation, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Online, 2020, pp. 2685–2702. URL: https://aclanthology.org/ 2020.emnlp-main.213/. doi: 10.18653/v1/2020. emnlp-main.213 . [36] R. Rei, M. Treviso, N. M. Guerreiro, C. Zerva, A. C. Farinha, C. Maroti, J. G. C. de Souza, T. Glushkova, D. Alves, L. Coheur, A. Lavie, A. F. T. Martins, CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task, in: Proceedings of the Seventh Conference on Machine Translation (WMT), Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Hybrid), 2022, pp. 634–645. URL: https://aclanthology.org/ 2022.wmt-1.60/. [37] R. Rei, N. M. Guerreiro, J. Pombal, D. van Stigt, M. Treviso, L. Coheur, J. G. C. de Souza, A. Martins, Scaling up CometKiwi: Unbabel-IST 2023 submission for the quality estimation shared task, in: Proceedings of the Eighth Conference on Machine Translation, Association for Computational Linguistics, Singapore, 2023, pp. 841–848. URL: https://aclanthology.org/2023.wmt-1.73/. doi: 10.18653/ v1/2023.wmt-1.73 . [38] J. Juraska, D. Deutsch, M. Finkelstein, M. Freitag, MetricX-24: The Google submission to the WMT 2024 metrics shared task, in: Proceedings of the Ninth Conference on Machine Translation, Association for Computational Linguistics, Miami, Florida, USA, 2024, pp. 492–504. URL: https://aclanthology.org/2024.wmt-1.35/. doi: 10.18653/ v1/2024.wmt-1.35 . [39] M. Snover, B. Dorr, R. Schwartz, L. Micciulla, J. Makhoul, A study of translation edit rate with targeted human annotation, in: Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, Association for Machine Translation in the Americas, Cambridge, Massachusetts, USA, 2006, pp. 223–231. URL: https://aclanthology.org/2006.amta-papers.25/. [40] Y. Wan, D. Liu, B. Yang, H. Zhang, B. Chen, D. Wong, L. Chao, UniTE: Unified translation evaluation, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 8117–8127. URL: https://aclanthology.org/2022.acl-long.558/. doi: 10.18653/v1/2022.acl-long.558 . [41] N. M. Guerreiro, R. Rei, D. v. Stigt, L. Coheur, P. Colombo, A. F. T. Martins, xcomet: Transparent machine translation evaluation through fine-grained error detection, Transactions of the Association for Computational Linguistics 12 (2024) 979– 995. URL: https://aclanthology.org/2024.tacl-1.54/. doi: 10.1162/tacl_a_00683 . [42] V. Zouhar, P. Chen, T. K. Lam, N. Moghe, B. Haddow, Pitfalls and outlooks in using COMET, in: Proceedings of the Ninth Conference on Machine Translation, Association for Computational Linguistics, Miami, Florida, USA, 2024, pp. 1272–1288. URL: https://aclanthology.org/2024.wmt-1.121/. doi: 10. 18653/v1/2024.wmt-1.121 . [43] D. Deutsch, G. Foster, M. Freitag, Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration, in: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Singapore, 2023, pp. 12914–12929. URL: https://aclanthology.org/2023.emnlp-main.798/. doi: 10. 18653/v1/2023.emnlp-main.798 . [44] S. Agrawal, A. Farinhas, R. Rei, A. Martins, Can automatic metrics assess high-quality translations?, in: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Miami, Florida, USA, 2024, pp. 14491–14502. URL: https://aclanthology.org/2024.emnlp-main.802/. doi: 10. 18653/v1/2024.emnlp-main.802 . [↗](https://doi.org/10.18653/v1/2023.wmt-1.51)

*Updated 31 October 2026*