--- title: "Are LLMs Breaking MT Metrics? Results of the — Freitag 2024" description: "LLMs now produce much of the output of modern MT systems." authors: ["Markus Freitag", "Nitika Mathur", "Daniel Deutsch", "Chi-Kiu Lo", "Eleftherios Avramidis", "Ricardo Rei", "Brian Thompson", "Frederic Blain", "Tom Kocmi", "Jiayi Wang", "David Ifeoluwa Adelani", "Marianna Buchicchio", "Chrysoula Zerva", "Alon Lavie"] year: 2024 venue: "Proceedings of the Ninth Conference on Machine Translation" source: "https://aclanthology.org/2024.wmt-1.2/" doi: "10.18653/v1/2024.wmt-1.2" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/freitag-2024-are-llms-breaking-mt-metrics/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/freitag-2024-are-llms-breaking-mt-metrics/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/) / Are LLMs Breaking MT Metrics? Results of the WMT24 Metrics Shared Task # Are LLMs Breaking MT Metrics? Results of the WMT24 Metrics Shared Task **Authors:** [Markus Freitag](https://md.research.elamur.ai/free/en/authors/markus-freitag/), [Nitika Mathur](https://md.research.elamur.ai/free/en/authors/nitika-mathur/), [Daniel Deutsch](https://md.research.elamur.ai/free/en/authors/daniel-deutsch/), [Chi-Kiu Lo](https://md.research.elamur.ai/free/en/authors/chi-kiu-lo/), [Eleftherios Avramidis](https://md.research.elamur.ai/free/en/authors/eleftherios-avramidis/), [Ricardo Rei](https://md.research.elamur.ai/free/en/authors/ricardo-rei/), [Brian Thompson](https://md.research.elamur.ai/free/en/authors/brian-thompson/), [Frederic Blain](https://md.research.elamur.ai/free/en/authors/frederic-blain/), [Tom Kocmi](https://md.research.elamur.ai/free/en/authors/tom-kocmi/), [Jiayi Wang](https://md.research.elamur.ai/free/en/authors/jiayi-wang/), [David Ifeoluwa Adelani](https://md.research.elamur.ai/free/en/authors/david-ifeoluwa-adelani/), Marianna Buchicchio et al. (14 authors) · **Year:** 2024 · **Venue:** Proceedings of the Ninth Conference on Machine Translation · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2024.wmt-1.2/) · [PDF](https://aclanthology.org/2024.wmt-1.2.pdf) · [DOI 10.18653/v1/2024.wmt-1.2](https://doi.org/10.18653/v1/2024.wmt-1.2) · [OpenAlex](https://openalex.org/W4404780957) **Citations:** 13 citations (openalex-snapshot · 2026-10-14) · [93](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary LLMs now produce much of the output of modern MT systems. The WMT24 Metrics Shared Task therefore tested whether automatic MT evaluation metrics still judge such translations reliably. The organizers took paragraph-level outputs from the WMT24 General MT Shared Task for three directions: English→German, English→Spanish (Latin America) and Japanese→Chinese. Expert MQM error annotations, provided by Google and Unbabel, served as the gold standard. Both reference-based and reference-free (QE) metrics were scored on how often they agreed with human judgments. The meta-evaluation dropped Pearson correlation in favour of pairwise ranking accuracy, which better matches how metrics are used in practice. Final ranks came from a weighted average over six tasks. Separately, a challenge-set subtask let external teams probe specific metric weaknesses. Trained neural metrics again clearly beat surface-overlap measures such as BLEU, spBLEU and chrF, and the organizers found no sign that they struggle with LLM-generated translations. The ESA human evaluation confirmed these conclusions, which supports continued use of fine-tuned metrics for modern MT. ## Key points - **Meta-evaluation:** system-level scoring used soft pairwise accuracy, and segment-level scoring used pairwise accuracy with tie calibration. All data and results were computed with and released via MTME. - **Human annotation:** - Google's professional translators annotated en→de (10 raters) and ja→zh (8 raters), using pSxS rater assignment. - Unbabel annotated en→es following its MQM 3.0 typology, with "Critical" errors mapped to Major. - **Winners and scores:** MetricX-24-Hybrid and XCOMET were named the winners. MetaMetrics-MT shared the top rank cluster (average score about 0.72), while BLEU averaged roughly 0.59. - **Challenge sets:** four were submitted: - AfriMTE: African languages, 13 directions. - BioMQM: biomedical domain, 4,641 items. - DFKI: 137,000 items covering more than 100 linguistic phenomena, for en→de and en→ru. - MSLC24: low- to mid-quality MT plus corner cases such as empty strings and wrong or mixed language output. - **Challenge-set findings:** trained metrics may overfit to WMT domains (BioMQM). Extremely low-resource pairs such as English–Luo and English–Twi remain difficult. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) ## Cited by Cited by 93 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 79 not yet. - [Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier](https://md.research.elamur.ai/free/en/papers/dang-2024-aya-expanse-combining-research-breakthroughs/) - [Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet](https://md.research.elamur.ai/free/en/papers/kocmi-2024-findings-wmt24-general-machine-translation/) - [SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages?](https://md.research.elamur.ai/free/en/papers/li-2025-ssa-comet-do-llms-outperform/) - [SiniticMTError: A Machine Translation Dataset with Error Annotations for Sinitic Languages](https://md.research.elamur.ai/free/en/papers/liu-2026-siniticmterror-machine-translation-dataset-error/) - [When the Gold Standard Isn't Necessarily Standard: Challenges of Evaluating the Translation of User-Generated Content](https://md.research.elamur.ai/free/en/papers/nishimwe-2026-when-gold-standard-isn-t/) - [PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation](https://md.research.elamur.ai/free/en/papers/proietti-2026-pear-pairwise-evaluation-automatic-relative/) - [Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains](https://md.research.elamur.ai/free/en/papers/schmidt-2026-who-watches-watchmen-humans-disagree/) - [Remedy-R: Generative Reasoning for Machine Translation Evaluation without Error Annotations](https://md.research.elamur.ai/free/en/papers/tan-2026-remedy-r-generative-reasoning-machine/) - [Diagnosing Translated Benchmarks: An Automated Quality Assurance Study of the EU20 Benchmark Suite](https://md.research.elamur.ai/free/en/papers/thellmann-2026-diagnosing-translated-benchmarks-automated-quality/) - [Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation](https://md.research.elamur.ai/free/en/papers/thellmann-2026-quantifying-impact-translation-errors-multilingual/) - [Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages](https://md.research.elamur.ai/free/en/papers/yari-2026-revisiting-metric-reliability-fine-grained/) - [A Critical Study of Automatic Evaluation in Sign Language Translation](https://md.research.elamur.ai/free/en/papers/yazdani-2026-critical-study-automatic-evaluation-sign/) - [Dynamic Meta-Metrics: Source-Sentence Conditioned Weighting for MT Evaluation](https://md.research.elamur.ai/free/en/papers/zhang-2026-dynamic-meta-metrics-source-sentence/) - [Early-Exit and Instant Confidence Translation Quality Estimation](https://md.research.elamur.ai/free/en/papers/zouhar-2026-early-exit-instant-confidence-translation/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{freitag2024are, title = {{Are LLMs Breaking MT Metrics? Results of the WMT24 Metrics Shared Task}}, author = {Freitag, Markus and Mathur, Nitika and Deutsch, Daniel and Lo, Chi-Kiu and Avramidis, Eleftherios and Rei, Ricardo and Thompson, Brian and Blain, Frederic and Kocmi, Tom and Wang, Jiayi and Adelani, David Ifeoluwa and Buchicchio, Marianna and Zerva, Chrysoula and Lavie, Alon}, booktitle = {Proceedings of the Ninth Conference on Machine Translation}, year = {2024}, doi = {10.18653/v1/2024.wmt-1.2}, url = {https://doi.org/10.18653/v1/2024.wmt-1.2} } ``` - [← previous ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic](https://md.research.elamur.ai/free/en/papers/koto-2024-arabicmmlu-assessing-massive-multitask-language/) - [next → Are We on the Right Way for Evaluating Large Vision-Language Models?](https://md.research.elamur.ai/free/en/papers/chen-2024-right-way-evaluating-large-vision/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/) ## References 3 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Eleftherios Avramidis, Shushen Manakhimova, Vivien Macketanz, and Sebastian Moeller. 2024. Machine translation metrics are better in evaluating linguistic errors on llms than on encoder-decoder systems. In Proceedings of the Ninth Conference on Machine Translation, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Eleftherios+Avramidis%2C+Shushen+Manakhimova%2C+Vivien+Macketanz%2C+and+Sebastian+Moeller%2E+2024%2E+Machine+translation+metrics+are+better+in+evaluating+linguistic+errors+on+llms+than+on+encoder%2Ddecoder+system) 2. David Anugraha, Garry Kuwanto, Lucky Susanto, Derry Tanti Wijaya, and Genta Indra Winata. 2024. Metametrics-MT: Tuning machine translation metametrics via human preference calibration. In Proceedings of the Ninth Conference on Machine Translation, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+Anugraha%2C+Garry+Kuwanto%2C+Lucky+Susanto%2C+Derry+Tanti+Wijaya%2C+and+Genta+Indra+Winata%2E+2024%2E+Metametrics%2DMT%3A+Tuning+machine+translation+metametrics+via+human+preference+calibration%2E+In+Proceedings+) 3. Eleftherios Avramidis and Vivien Macketanz. 2022. Linguistically motivated evaluation of machine translation metrics based on a challenge set. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 514–529, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Eleftherios+Avramidis+and+Vivien+Macketanz%2E+2022%2E+Linguistically+motivated+evaluation+of+machine+translation+metrics+based+on+a+challenge+set%2E+In+Proceedings+of+the+Seventh+Conference+on+Machine+Trans)