---
title: "Quantifying the Impact of Translation — Thellmann 2026"
description: "Benchmarks machine-translated from English are a common way to test LLMs in other languages."
authors: ["Klaudia Thellmann", "Bernhard Stadler", "Michael Färber", "Jens Lehmann"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1916/"
doi: "10.18653/v1/2026.acl-long.1916"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/thellmann-2026-quantifying-impact-translation-errors-multilingual/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/thellmann-2026-quantifying-impact-translation-errors-multilingual/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2026](https://md.research.elamur.ai/free/papers/2026/page/4/) / Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation

# Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation

**Authors:** [Klaudia Thellmann](https://md.research.elamur.ai/free/authors/klaudia-thellmann/), [Bernhard Stadler](https://md.research.elamur.ai/free/authors/bernhard-stadler/), [Michael Färber](https://md.research.elamur.ai/free/authors/michael-farber/), Jens Lehmann · **Year:** 2026 · **Venue:** Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1916/) · [PDF](https://aclanthology.org/2026.acl-long.1916.pdf) · [DOI 10.18653/v1/2026.acl-long.1916](https://doi.org/10.18653/v1/2026.acl-long.1916) · [OpenAlex](https://openalex.org/W7166811754)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14)

## Summary
Benchmarks machine-translated from English are a common way to test LLMs in other languages. Yet the extent to which their translation mistakes distort results has rarely been measured systematically. The authors ask two things. First, do automatic MQM-style error spans match expert human annotations on real benchmark translations? Second, how much do target-side translation errors lower accuracy, as opposed to flaws already present in the English source?

Using EU20 (DeepL translations of MMLU, ARC, HellaSwag, GSM8K and TruthfulQA), they obtained professional span-level annotations for 225 items in nine languages. They compared four LLM annotators and the xCOMET-XXL baseline against this reference, and also tested span localization on Span-ACES Ref. Logistic regressions then relate errors to translated accuracy, controlling for English correctness, fixed effects and source-side anomalies.

GPT-5.2 agreed most closely with the human annotators. Translation errors were linked to accuracy losses of several percentage points, while system rankings stayed essentially unchanged. The practical lesson is that translated leaderboards can remain internally consistent while still systematically understating absolute multilingual performance.

## Key points
- **Human reference:** 225 EU20 items covering DE, DA, FR, IT, RO, ET, HU, SL and LT, with one professional annotator per language. The evaluated annotators were GPT-5.2, GPT-4o-mini, LLaMA-4, Mistral-Large and xCOMET-XXL. GPT-5.2 scored best, with a mean span-level F1 of 0.55 under position-overlap matching.
- **Span-ACES Ref:** a cleaned projection of SPAN-ACES with 1,407 items. Manual review and GPT-5.2 agreed on 165 of 178 validation records. Tolerant span metrics scored well above classic ones, suggesting many disagreements are near-misses on span boundaries.
- **Accuracy impact:** the full-model estimates tie target-side errors to accuracy reductions of roughly 6–8 pp, widening to 6–11 pp for items answered correctly in English. Source-side issues had smaller and less stable effects.
- **Rankings:** setting translation errors to zero in a counterfactual left rankings intact (Spearman's ρ ≈ 1.0). Absolute scores, however, would rise by about 1–4.6 pp.
- **Resources:** data, prompts and code are released at hf.co/btqe and github.com/btqe.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{thellmann2026quantifying,
  title = {{Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation}},
  author = {Thellmann, Klaudia and Stadler, Bernhard and Färber, Michael and Lehmann, Jens},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1916},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1916}
}
```

- [← previous Quantifying the Accuracy and Cost Impact of Design Decisions in Budget-Constrained Agentic LLM Search](https://md.research.elamur.ai/free/papers/mccleary-2026-quantifying-accuracy-cost-impact-design/)
- [next → Ramsa: A Large Sociolinguistically Rich Emirati Arabic Speech Corpus for ASR and TTS](https://md.research.elamur.ai/free/papers/sabbagh-2026-ramsa-large-sociolinguistically-rich-emirati/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/papers/2026/page/4/)

## References

37 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ashish Agrawal, Barah Fazili, and Preethi Jyothi. 2024. Translation errors significantly impact low-resource languages in cross-lingual learning. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 319–329, St. Julian’s, Malta. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ashish+Agrawal%2C+Barah+Fazili%2C+and+Preethi+Jyothi%2E+2024%2E+Translation+errors+significantly+impact+low%2Dresource+languages+in+cross%2Dlingual+learning%2E+In+Proceedings+of+the+18th+Conference+of+the+European+)
2. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020. Translation artifacts in cross-lingual transfer learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7674–7684, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Gorka+Labaka%2C+and+Eneko+Agirre%2E+2020%2E+Translation+artifacts+in+cross%2Dlingual+transfer+learning%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing)
3. Irene Baucells, Javier Aula-Blasco, Iria de Dios-Flores, Silvia Paniagua Suárez, Naiara Perez, Anna Salles, Susana Sotelo Docio, Júlia Falcão, Jose Javier Saiz, Robiert Sepulveda Torres, Jeremy Barnes, Pablo Gamallo, Aitor Gonzalez-Agirre, German Rigau, and Marta Villegas. 2025. IberoBench: A benchmark for LLM evaluation in Iberian languages. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10491–10519, Abu Dhabi, UAE. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Irene+Baucells%2C+Javier+Aula%2DBlasco%2C+Iria+de+Dios%2DFlores%2C+Silvia+Paniagua+Su%C3%A1rez%2C+Naiara+Perez%2C+Anna+Salles%2C+Susana+Sotelo+Docio%2C+J%C3%BAlia+Falc%C3%A3o%2C+Jose+Javier+Saiz%2C+Robiert+Sepulveda+Torres%2C+Jeremy+Barnes)
4. Frederic Blain, Chrysoula Zerva, Ricardo Rei, Nuno M. Guerreiro, Diptesh Kanojia, José G. C. de Souza, Beatriz Silva, Tânia Vaz, Yan Jingxuan, Fatemeh Azadi, Constantin Orasan, and André Martins. 2023. Findings of the WMT 2023 shared task on quality estimation. In Proceedings of the Eighth Conference on Machine Translation, pages 629–653, Singapore. Association for Computational Linguistics. [↗](https://arxiv.org/abs/2406.14267) [our card](https://md.research.elamur.ai/free/papers/blain-2023-findings-wmt-2023-shared-task/)
5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. Preprint, arXiv:2110.14168. [↗](https://arxiv.org/abs/2110.14168) [our card](https://md.research.elamur.ai/free/papers/cobbe-2021-training-verifiers-solve-math-word/)
6. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1809.05053)
7. Martin Fajcik, Martin Docekal, Jan Dolezal, Karel Ondrej, Karel Beneš, Jan Kapsa, Pavel Smrz, Alexander Polok, Michal Hradis, Zuzana Neverilova, Ales Horak, Radoslav Sabol, Michal Stefanik, Adam Jirkovsky, David Adamczyk, Petr Hyner, Jan Hula, and Hynek Kydlicek. 2025. BenCzechMark: A Czech-centric multitask and multimetric benchmark for large language models with duel scoring mechanism. Preprint, arXiv:2412.17933. [↗](https://arxiv.org/abs/2412.17933)
8. Manuel Faysse, Patrick Fernandes, Nuno M. Guerreiro, António Loison, Duarte M. Alves, Caio Corro, Nicolas Boizard, João Alves, Ricardo Rei, Pedro H. Martins, Antoni Bigata Casademunt, François Yvon, André F. T. Martins, Gautier Viaud, Céline Hudelot, and Pierre Colombo. 2025. CroissantLLM: A truly bilingual French-English language model (introducing FrenchBench). Preprint, arXiv:2402.00786. [↗](https://arxiv.org/abs/2402.00786)
9. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation. Transactions of the Association for Computational Linguistics, 9:1460–1474. [↗](https://doi.org/10.1162/tacl_a_00437) [our card](https://md.research.elamur.ai/free/papers/freitag-2021-experts-errors-context-large-scale/)
10. Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frederic Blain, Tom Kocmi, Jiayi Wang, David Ifeoluwa Adelani, Marianna Buchicchio, Chrysoula Zerva, and Alon Lavie. 2024. Are LLMs Breaking MT Metrics? Results of the WMT24 Metrics Shared Task. In Proceedings of the Ninth Conference on Machine Translation, pages 47–81, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Markus+Freitag%2C+Nitika+Mathur%2C+Daniel+Deutsch%2C+Chi%2DKiu+Lo%2C+Eleftherios+Avramidis%2C+Ricardo+Rei%2C+Brian+Thompson%2C+Frederic+Blain%2C+Tom+Kocmi%2C+Jiayi+Wang%2C+David+Ifeoluwa+Adelani%2C+Marianna+Buchicchio%2C+Chrys) [our card](https://md.research.elamur.ai/free/papers/freitag-2024-are-llms-breaking-mt-metrics/)
11. Nuno M. Guerreiro, Ricardo Rei, Daan Van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2024. xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection. Transactions of the Association for Computational Linguistics, 12:979–995. [↗](https://doi.org/10.48550/arxiv.2310.10482) [our card](https://md.research.elamur.ai/free/papers/guerreiro-2024-xcomet-transparent-machine-translation-evaluation/)
12. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.2009.03300) [our card](https://md.research.elamur.ai/free/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
13. Xu Huang, Zhirui Zhang, Xiang Geng, Yichao Du, Jiajun Chen, and Shujian Huang. 2024. Lost in the source language: How large language models evaluate the quality of machine translation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3546–3562, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xu+Huang%2C+Zhirui+Zhang%2C+Xiang+Geng%2C+Yichao+Du%2C+Jiajun+Chen%2C+and+Shujian+Huang%2E+2024%2E+Lost+in+the+source+language%3A+How+large+language+models+evaluate+the+quality+of+machine+translation%2E+In+Findings+of+)
14. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193–203, Tampere, Finland. European Association for Machine Translation. [↗](https://doi.org/10.48550/arxiv.2302.14520)
15. Tom Kocmi, Vilém Zouhar, Eleftherios Avramidis, Roman Grundkiewicz, Marzena Karpinska, Maja Popović, Mrinmaya Sachan, and Mariya Shmatova. 2024. Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation. In Proceedings of the Ninth Conference on Machine Translation, pages 1440–1453, Miami, Florida, USA. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2406.11580) [our card](https://md.research.elamur.ai/free/papers/kocmi-2024-error-span-annotation-balanced-approach/)
16. Klaus Krippendorff. 1995. On the reliability of unitizing continuous data. Sociological Methodology, 25:47–76. [↗](https://doi.org/10.2307/271061)
17. Klaus Krippendorff, Yann Mathet, Stéphane Bouvry, and Antoine Widlöcher. 2016. On the reliability of unitizing textual continua: Further developments. Quality & Quantity, 50(6):2347–2364. [↗](https://doi.org/10.1007/s11135-015-0266-1)
18. Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. 2023. Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 318–327, Singapore. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2307.16039) [our card](https://md.research.elamur.ai/free/papers/lai-2023-okapi-instruction-tuned-large-language/)
19. Alon Lavie, Greg Hanneman, Sweta Agrawal, Diptesh Kanojia, Chi-Kiu Lo, Vilém Zouhar, Frederic Blain, Chrysoula Zerva, Eleftherios Avramidis, Sourabh Deoghare, Archchana Sindhujan, Jiayi Wang, David Ifeoluwa Adelani, Brian Thompson, Tom Kocmi, Markus Freitag, and Daniel Deutsch. 2025. Findings of the WMT25 shared task on automated translation evaluation systems: Linguistic diversity is challenging and references still help. In Proceedings of the Tenth Conference on Machine Translation, pages 436–483, Suzhou, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alon+Lavie%2C+Greg+Hanneman%2C+Sweta+Agrawal%2C+Diptesh+Kanojia%2C+Chi%2DKiu+Lo%2C+Vil%C3%A9m+Zouhar%2C+Frederic+Blain%2C+Chrysoula+Zerva%2C+Eleftherios+Avramidis%2C+Sourabh+Deoghare%2C+Archchana+Sindhujan%2C+Jiayi+Wang%2C+David+If)
20. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Stephanie+Lin%2C+Jacob+Hilton%2C+and+Owain+Evans%2E+2022%2E+TruthfulQA%3A+Measuring+How+Models+Mimic+Human+Falsehoods%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+) [our card](https://md.research.elamur.ai/free/papers/lin-2022-truthfulqa-measuring-how-models-mimic/)
21. Arle Lommel, Serge Gladkoff, Alan Melby, Sue Ellen Wright, Ingemar Strandvik, Katerina Gasova, Angelika Vaasa, Andy Benzo, Romina Marazzato Sparano, Monica Foresi, Johani Innis, Lifeng Han, and Goran Nenadic. 2024. The multi-range theory of translation quality measurement: MQM scoring models and statistical quality control. In Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 2: Presentations), pages 75–94, Chicago, USA. Association for Machine Translation in the Americas. [↗](https://doi.org/10.48550/arxiv.2405.16969)
22. Arle Richard Lommel, Aljoscha Burchardt, and Hans Uszkoreit. 2013. Multidimensional quality metrics: A flexible system for assessing translation quality. In Proceedings of Translating and the Computer 35, London, UK. Aslib. [find](https://scholar.google.com/scholar?q=Arle+Richard+Lommel%2C+Aljoscha+Burchardt%2C+and+Hans+Uszkoreit%2E+2013%2E+Multidimensional+quality+metrics%3A+A+flexible+system+for+assessing+translation+quality%2E+In+Proceedings+of+Translating+and+the+Computer) [our card](https://md.research.elamur.ai/free/papers/lommel-2013-multidimensional-quality-metrics-flexible-system/)
23. Qingyu Lu, Liang Ding, Kanjian Zhang, Jinxia Zhang, and Dacheng Tao. 2025. MQM-APE: Toward high-quality error annotation predictors with automatic post-editing in LLM translation evaluators. In Proceedings of the 31st International Conference on Computational Linguistics, pages 5570–5587, Abu Dhabi, UAE. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2409.14335) [our card](https://md.research.elamur.ai/free/papers/lu-2025-mqm-ape-high-quality-error/)
24. Christian M. Meyer, Margot Mieskes, Christian Stab, and Iryna Gurevych. 2014. DKPro agreement: An open-source Java library for measuring inter-rater agreement. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: System Demonstrations, pages 105–109, Dublin, Ireland. Dublin City University and Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Christian+M%2E+Meyer%2C+Margot+Mieskes%2C+Christian+Stab%2C+and+Iryna+Gurevych%2E+2014%2E+DKPro+agreement%3A+An+open%2Dsource+Java+library+for+measuring+inter%2Drater+agreement%2E+In+Proceedings+of+COLING+2014%2C+the+25th+)
25. Nikita Moghe, Arnisa Fazla, Chantal Amrhein, Tom Kocmi, Mark Steedman, Alexandra Birch, Rico Sennrich, and Liane Guillou. 2025. Machine Translation Meta Evaluation through Translation Accuracy Challenge Sets. Computational Linguistics, pages 1–65. [↗](https://doi.org/10.5167/uzh-265275) [our card](https://md.research.elamur.ai/free/papers/moghe-2025-machine-translation-meta-evaluation-through/)
26. Dan Nielsen. 2023. ScandEval: A benchmark for Scandinavian natural language processing. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 185–201, Tórshavn, Faroe Islands. University of Tartu Library. [↗](https://doi.org/10.48550/arxiv.2304.00906)
27. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525–1534, Berlin, Germany. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Denis+Paperno%2C+Germ%C3%A1n+Kruszewski%2C+Angeliki+Lazaridou%2C+Ngoc+Quan+Pham%2C+Raffaella+Bernardi%2C+Sandro+Pezzelle%2C+Marco+Baroni%2C+Gemma+Boleda%2C+and+Raquel+Fern%C3%A1ndez%2E+2016%2E+The+LAMBADA+dataset%3A+Word+prediction+) [our card](https://md.research.elamur.ai/free/papers/paperno-2016-lambada-dataset-word-prediction-requiring/)
28. ChaeHun Park, Koanho Lee, Hyesu Lim, Jaeseok Kim, Junmo Park, Yu-Jung Heo, Du-Seong Chang, and Jaegul Choo. 2024. Translation deserves better: Analyzing translation artifacts in cross-lingual visual question answering. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5193–5221, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=ChaeHun+Park%2C+Koanho+Lee%2C+Hyesu+Lim%2C+Jaeseok+Kim%2C+Junmo+Park%2C+Yu%2DJung+Heo%2C+Du%2DSeong+Chang%2C+and+Jaegul+Choo%2E+2024%2E+Translation+deserves+better%3A+Analyzing+translation+artifacts+in+cross%2Dlingual+visual+q)
29. Jan Pfister and Andreas Hotho. 2024. SuperGLEBer: German language understanding evaluation benchmark. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7904–7923, Mexico City, Mexico. Association for Computational Linguistics. [↗](https://doi.org/10.48448/fnrg-ry97)
30. Irene Plaza, Nina Melero, Cristina Pozo, Javier Conde, Pedro Reviriego, Marina Mayor-Rocher, and María Grandury. 2024. Spanish and LLM benchmarks: Is MMLU lost in translation? Preprint, arXiv:2406.17789. [↗](https://arxiv.org/abs/2406.17789)
31. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Craig+Stewart%2C+Ana+C+Farinha%2C+and+Alon+Lavie%2E+2020%2E+COMET%3A+A+neural+framework+for+MT+evaluation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+) [our card](https://md.research.elamur.ai/free/papers/rei-2020-comet-neural-framework-mt-evaluation/)
32. Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I. Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Wei-Yin Ko, Sebastian Ruder, Madeline Smith, Antoine Bosselut, Alice Oh, Andre F. T. Martins, Leshem Choshen, and 5 others. 2025. Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation. Preprint, arXiv:2412.03304v2. [↗](https://arxiv.org/abs/2412.03304) [our card](https://md.research.elamur.ai/free/papers/singh-2025-global-mmlu-understanding-addressing-cultural/)
33. Ole Tange. 2024. GNU Parallel 20240722 (’assange’). [find](https://scholar.google.com/scholar?q=Ole+Tange%2E+2024%2E+GNU+Parallel+20240722+%28%E2%80%99assange%E2%80%99%29%2E)
34. Klaudia Thellmann, Bernhard Stadler, Michael Fromm, Jasper Schulze Buschhoff, Alex Jude, Fabio Barth, Johannes Leveling, Nicolas Flores-Herr, Joachim Köhler, René Jäkel, and Mehdi Ali. 2024. Towards Multilingual LLM Evaluation for European Languages. Preprint, arXiv:2410.08928. [↗](https://arxiv.org/abs/2410.08928)
35. Taemin Yeom, Yonghyun Ryu, Yoonjung Choi, and Jinyeong Bak. 2025. Tagged span annotation for detecting translation errors in reasoning LLMs. In Proceedings of the Tenth Conference on Machine Translation, pages 878–886, Suzhou, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Taemin+Yeom%2C+Yonghyun+Ryu%2C+Yoonjung+Choi%2C+and+Jinyeong+Bak%2E+2025%2E+Tagged+span+annotation+for+detecting+translation+errors+in+reasoning+LLMs%2E+In+Proceedings+of+the+Tenth+Conference+on+Machine+Translati)
36. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1905.07830)
37. Chrysoula Zerva, Frederic Blain, José G. C. De Souza, Diptesh Kanojia, Sourabh Deoghare, Nuno M. Guerreiro, Giuseppe Attanasio, Ricardo Rei, Constantin Orasan, Matteo Negri, Marco Turchi, Rajen Chatterjee, Pushpak Bhattacharyya, Markus Freitag, and André Martins. 2024. Findings of the quality estimation shared task at WMT 2024: Are LLMs closing the gap in QE? In Proceedings of the Ninth Conference on Machine Translation, pages 26–76, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chrysoula+Zerva%2C+Frederic+Blain%2C+Jos%C3%A9+G%2E+C%2E+De+Souza%2C+Diptesh+Kanojia%2C+Sourabh+Deoghare%2C+Nuno+M%2E+Guerreiro%2C+Giuseppe+Attanasio%2C+Ricardo+Rei%2C+Constantin+Orasan%2C+Matteo+Negri%2C+Marco+Turchi%2C+Rajen+Chatte)