---
title: "XQ-MEval: A Dataset with Cross-lingual Parallel — Liu 2026"
description: "Multilingual MT systems are usually ranked by averaging automatic metric scores over translation directions."
authors: ["Jingxuan Liu", "Zhi Qu", "Jin Tei", "Hidetaka Kamigaito", "Lemao Liu", "Taro Watanabe"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1846/"
doi: "10.18653/v1/2026.findings-acl.1846"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/liu-2026-xq-meval-dataset-cross-lingual/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/liu-2026-xq-meval-dataset-cross-lingual/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2026](https://md.research.elamur.ai/free/papers/2026/page/5/) / XQ-MEval: A Dataset with Cross-lingual Parallel Quality for Benchmarking Translation Metrics

# XQ-MEval: A Dataset with Cross-lingual Parallel Quality for Benchmarking Translation Metrics

**Authors:** Jingxuan Liu, Zhi Qu, Jin Tei, [Hidetaka Kamigaito](https://md.research.elamur.ai/free/authors/hidetaka-kamigaito/), [Lemao Liu](https://md.research.elamur.ai/free/authors/lemao-liu/), [Taro Watanabe](https://md.research.elamur.ai/free/authors/taro-watanabe/) · **Year:** 2026 · **Venue:** Findings of the Association for Computational Linguistics: ACL 2026 · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1846/) · [PDF](https://aclanthology.org/2026.findings-acl.1846.pdf) · [DOI 10.18653/v1/2026.findings-acl.1846](https://doi.org/10.18653/v1/2026.findings-acl.1846) · [OpenAlex](https://openalex.org/W7166867340)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14) · 1 in our library, of the 140673 papers whose bibliographies we read

## Summary
Multilingual MT systems are usually ranked by averaging automatic metric scores over translation directions. This assumes a metric penalizes a comparable error equally in every language, which had never been tested because no benchmark offered translations of matched quality across languages. The authors built XQ-MEval, a semi-automatic resource for nine English-to-X directions based on Flores. GPT-4o inserts single major MQM-style errors into reference translations, and native speakers filter the results. Zero to five verified errors are then combined, producing pseudo translations at six controlled quality levels. These are packaged as source/translation/reference triplets and sampled into pseudo systems whose true quality is known in advance. Testing nine metrics, the authors found that averaging diverges from human judgment. They are also the first to demonstrate empirically that metric scores are biased across languages: equal-quality output gets different scores in different languages, and scores fall at different rates as quality drops. To address this, they propose Language-specific Global Normalization (LGN), which standardizes scores per direction before averaging. LGN yields statistically significant gains in agreement with human evaluation, supporting fairer multilingual evaluation.

## Key points
- Directions covered: en-zh, en-ja, en-lo, en-vi, en-id, en-fr, en-es, en-si, en-de. These mix high- and low-resource languages, and the source data come from Flores (flores_plus).
- Error budget follows Google's MQM guideline. Each major error costs 5 points, with at most 5 errors (25 points) per sentence.
- Only four purely semantic MQM error types are used, chosen for cross-lingual comparability. They make up about 46.3% of errors in existing MQM datasets.
- LGN applies a per-direction z-score, (score − µ)/σ, before averaging. Paired t-tests confirm the improvement is significant, although the absolute gains are moderate.
- Code and data are on GitHub (zhiqu22/XQ-MEval) under CC BY-SA 4.0. Because the instances are synthetic, the authors also report partial validation on real MQM data.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liu2026xq,
  title = {{XQ-MEval: A Dataset with Cross-lingual Parallel Quality for Benchmarking Translation Metrics}},
  author = {Liu, Jingxuan and Qu, Zhi and Tei, Jin and Kamigaito, Hidetaka and Liu, Lemao and Watanabe, Taro},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1846},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1846}
}
```

- [← previous Word Alignment-Based Evaluation of Uniform Meaning Representations](https://md.research.elamur.ai/free/papers/zeman-2026-word-alignment-based-evaluation-uniform/)
- [next → YallaMorph: A Benchmark for Evaluating Arabic Morphological Generation in Large Language Models](https://md.research.elamur.ai/free/papers/langstream-2026-reda-yallamorph-benchmark-evaluating-arabic-morphological/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/papers/2026/page/5/)

## References

47 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, and Wanli Ouyang. 2024. MT-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7421–7454, Bangkok, Thailand. [find](https://scholar.google.com/scholar?q=Ge+Bai%2C+Jie+Liu%2C+Xingyuan+Bu%2C+Yancheng+He%2C+Jiaheng+Liu%2C+Zhanhui+Zhou%2C+Zhuoran+Lin%2C+Wenbo+Su%2C+Tiezheng+Ge%2C+Bo+Zheng%2C+and+Wanli+Ouyang%2E+2024%2E+MT%2Dbench%2D101%3A+A+fine%2Dgrained+benchmark+for+evaluating+large+) [our card](https://md.research.elamur.ai/free/papers/bai-2024-mt-bench-101-fine-grained/)
2. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine translation (WMT17). In Proceedings of the Second Conference on Machine Translation, pages 169–214, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Rajen+Chatterjee%2C+Christian+Federmann%2C+Yvette+Graham%2C+Barry+Haddow%2C+Shujian+Huang%2C+Matthias+Huck%2C+Philipp+Koehn%2C+Qun+Liu%2C+Varvara+Logacheva%2C+Christof+Monz%2C+Matteo+Negri%2C+Matt+Post%2C+Raph) [our card](https://md.research.elamur.ai/free/papers/bojar-2017-findings-2017-conference-machine-translation/)
3. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, and 2 others. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 131–198, Berlin, Germany. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Rajen+Chatterjee%2C+Christian+Federmann%2C+Yvette+Graham%2C+Barry+Haddow%2C+Matthias+Huck%2C+Antonio+Jimeno+Yepes%2C+Philipp+Koehn%2C+Varvara+Logacheva%2C+Christof+Monz%2C+Matteo+Negri%2C+Aur%C3%A9lie+N%C3%A9v%C3%A9ol%2C+M) [our card](https://md.research.elamur.ai/free/papers/bojar-2016-findings-2016-conference-machine-translation/)
4. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (meta-) evaluation of machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 136–158, Prague, Czech Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chris+Callison%2DBurch%2C+Cameron+Fordyce%2C+Philipp+Koehn%2C+Christof+Monz%2C+and+Josh+Schroeder%2E+2007%2E+%28meta%2D%29+evaluation+of+machine+translation%2E+In+Proceedings+of+the+Second+Workshop+on+Statistical+Machine+T) [our card](https://md.research.elamur.ai/free/papers/burch-2007-meta-evaluation-machine-translation/)
5. Zhe Cao, Zhi Qu, Hidetaka Kamigaito, and Taro Watanabe. 2024. Exploring intrinsic language-specific subspaces in fine-tuning multilingual neural machine translation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21142–21157, Miami, Florida, USA. [↗](https://doi.org/10.48550/arxiv.2409.05224)
6. Liang Chen, Shuming Ma, Dongdong Zhang, Furu Wei, and Baobao Chang. 2023. On the off-target problem of zero-shot multilingual neural machine translation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9542–9558, Toronto, Canada. [↗](https://doi.org/10.60692/8d327-pzf28)
7. Sonia Cristofaro. 2009. Grammatical categories and relations: universality vs. language-specificity and construction-specificity. Language and Linguistics Compass, 3(1):441–479. [↗](https://doi.org/10.1111/j.1749-818x.2008.00111.x)
8. Michael Denkowski and Alon Lavie. 2010. Choosing the right evaluation for machine translation: an examination of annotator and automatic metric performance on human judgment tasks. In Proceedings of the 9th Conference of the Association for Machine Translation in the Americas: Research Papers, Denver, Colorado, USA. Association for Machine Translation in the Americas. [find](https://scholar.google.com/scholar?q=Michael+Denkowski+and+Alon+Lavie%2E+2010%2E+Choosing+the+right+evaluation+for+machine+translation%3A+an+examination+of+annotator+and+automatic+metric+performance+on+human+judgment+tasks%2E+In+Proceedings+of+t)
9. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460–1474. [↗](https://doi.org/10.1162/tacl_a_00437) [our card](https://md.research.elamur.ai/free/papers/freitag-2021-experts-errors-context-large-scale/)
10. Salvador García, Julián Luengo, and Francisco Herrera. 2015. Data Preprocessing in Data Mining, volume 72. Springer Cham. [find](https://scholar.google.com/scholar?q=Salvador+Garc%C3%ADa%2C+Juli%C3%A1n+Luengo%2C+and+Francisco+Herrera%2E+2015%2E+Data+Preprocessing+in+Data+Mining%2C+volume+72%2E+Springer+Cham%2E)
11. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522–538. [↗](https://doi.org/10.1162/tacl_a_00474)
12. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Timothy+Baldwin%2C+Alistair+Moffat%2C+and+Justin+Zobel%2E+2013%2E+Continuous+measurement+scales+in+human+evaluation+of+machine+translation%2E+In+Proceedings+of+the+7th+Linguistic+Annotation+Works)
13. Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2024. xCOMET: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979–995. [↗](https://doi.org/10.48550/arxiv.2310.10482) [our card](https://md.research.elamur.ai/free/papers/guerreiro-2024-xcomet-transparent-machine-translation-evaluation/)
14. Martin Haspelmath. 2010. Comparative concepts and descriptive categories in crosslinguistic studies. Language, 86(3):663–687. [↗](https://doi.org/10.1353/lan.2010.0021)
15. Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. MetricX-24: The Google submission to the WMT 2024 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, pages 492–504, Miami, Florida, USA. [find](https://scholar.google.com/scholar?q=Juraj+Juraska%2C+Daniel+Deutsch%2C+Mara+Finkelstein%2C+and+Markus+Freitag%2E+2024%2E+MetricX%2D24%3A+The+Google+submission+to+the+WMT+2024+metrics+shared+task%2E+In+Proceedings+of+the+Ninth+Conference+on+Machine+Tran) [our card](https://md.research.elamur.ai/free/papers/juraska-2024-metricx-24-google-submission-wmt/)
16. Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. 2023. MetricX-23: The Google submission to the WMT 2023 metrics shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 756–767, Singapore. [find](https://scholar.google.com/scholar?q=Juraj+Juraska%2C+Mara+Finkelstein%2C+Daniel+Deutsch%2C+Aditya+Siddhant%2C+Mehdi+Mirzazadeh%2C+and+Markus+Freitag%2E+2023%2E+MetricX%2D23%3A+The+Google+submission+to+the+WMT+2023+metrics+shared+task%2E+In+Proceedings+of+t) [our card](https://md.research.elamur.ai/free/papers/juraska-2023-metricx-23-google-submission-wmt/)
17. Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1-2):81–93. [↗](https://doi.org/10.1093/biomet/30.1-2.81)
18. Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, Martin Popel, Maja Popović, and 3 others. 2024. Findings of the WMT24 general machine translation shared task: The LLM era is here but MT is not solved yet. In Proceedings of the Ninth Conference on Machine Translation, pages 1–46, Miami, Florida, USA. [find](https://scholar.google.com/scholar?q=Tom+Kocmi%2C+Eleftherios+Avramidis%2C+Rachel+Bawden%2C+Ondr%CC%8Cej+Bojar%2C+Anton+Dvorkovich%2C+Christian+Federmann%2C+Mark+Fishel%2C+Markus+Freitag%2C+Thamme+Gowda%2C+Roman+Grundkiewicz%2C+Barry+Haddow%2C+Marzena+Karpinska%2C+P) [our card](https://md.research.elamur.ai/free/papers/kocmi-2024-findings-wmt24-general-machine-translation/)
19. Tom Kocmi and Christian Federmann. 2023. GEMBA-MQM: Detecting translation quality error spans with GPT-4. In Proceedings of the Eighth Conference on Machine Translation, pages 768–775, Singapore. [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Christian+Federmann%2E+2023%2E+GEMBA%2DMQM%3A+Detecting+translation+quality+error+spans+with+GPT%2D4%2E+In+Proceedings+of+the+Eighth+Conference+on+Machine+Translation%2C+pages+768%E2%80%93775%2C+Singapore%2E) [our card](https://md.research.elamur.ai/free/papers/kocmi-2023-gemba-mqm-detecting-translation-quality/)
20. Philipp Koehn and Christof Monz. 2006. Manual and automatic evaluation of machine translation between European languages. In Proceedings on the Workshop on Statistical Machine Translation, pages 102– 121, New York City. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Philipp+Koehn+and+Christof+Monz%2E+2006%2E+Manual+and+automatic+evaluation+of+machine+translation+between+European+languages%2E+In+Proceedings+on+the+Workshop+on+Statistical+Machine+Translation%2C+pages+102%E2%80%93+) [our card](https://md.research.elamur.ai/free/papers/koehn-2006-manual-automatic-evaluation-machine-translation/)
21. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. [find](https://scholar.google.com/scholar?q=Taku+Kudo+and+John+Richardson%2E+2018%2E+SentencePiece%3A+A+simple+and+language+independent+subword+tokenizer+and+detokenizer+for+neural+text+processing%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+M) [our card](https://md.research.elamur.ai/free/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
22. Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. 2024. MT-eval: A multi-turn capabilities evaluation benchmark for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20153–20177, Miami, Florida, USA. [↗](https://doi.org/10.48550/arxiv.2401.16745) [our card](https://md.research.elamur.ai/free/papers/kwan-2024-mt-eval-multi-turn-capabilities/)
23. Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. HaluEval: A large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449–6464, Singapore. [find](https://scholar.google.com/scholar?q=Junyi+Li%2C+Xiaoxue+Cheng%2C+Xin+Zhao%2C+Jian%2DYun+Nie%2C+and+Ji%2DRong+Wen%2E+2023%2E+HaluEval%3A+A+large%2Dscale+hallucination+evaluation+benchmark+for+large+language+models%2E+In+Proceedings+of+the+2023+Conference+on+E) [our card](https://md.research.elamur.ai/free/papers/li-2023-halueval-large-scale-hallucination-evaluation/)
24. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. [find](https://scholar.google.com/scholar?q=Maja+Popovic%CC%81%2E+2015%2E+chrF%3A+character+n%2Dgram+F%2Dscore+for+automatic+MT+evaluation%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages+392%E2%80%93395%2C+Lisbon%2C+Portugal%2E) [our card](https://md.research.elamur.ai/free/papers/popovic-2015-chrf-character-n-gram-f/)
25. Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. [find](https://scholar.google.com/scholar?q=Maja+Popovic%CC%81%2E+2017%2E+chrF%2B%2B%3A+words+helping+character+n%2Dgrams%2E+In+Proceedings+of+the+Second+Conference+on+Machine+Translation%2C+pages+612%E2%80%93618%2C+Copenhagen%2C+Denmark%2E) [our card](https://md.research.elamur.ai/free/papers/popovic-2017-chrf-words-helping-character-n/)
26. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. [find](https://scholar.google.com/scholar?q=Matt+Post%2E+2018%2E+A+call+for+clarity+in+reporting+BLEU+scores%2E+In+Proceedings+of+the+Third+Conference+on+Machine+Translation%3A+Research+Papers%2C+pages+186%E2%80%93+191%2C+Brussels%2C+Belgium%2E) [our card](https://md.research.elamur.ai/free/papers/post-2018-call-clarity-reporting-bleu-scores/)
27. Zhi Qu, Chenchen Ding, and Taro Watanabe. 2025a. Languages transferred within the encoder: On representation transfer in zero-shot multilingual translation. In Proceedings of Machine Translation Summit XX: Volume 1, pages 81–98, Geneva, Switzerland. European Association for Machine Translation. [↗](https://doi.org/10.48550/arxiv.2406.08092)
28. Zhi Qu, Yiran Wang, Chenchen Ding, Hideki Tanaka, Masao Utiyama, and Taro Watanabe. 2025b. Improving language transfer capability of decoder-only architecture in multilingual neural machine translation. In Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025), pages 178– 195, Suzhuo, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zhi+Qu%2C+Yiran+Wang%2C+Chenchen+Ding%2C+Hideki+Tanaka%2C+Masao+Utiyama%2C+and+Taro+Watanabe%2E+2025b%2E+Improving+language+transfer+capability+of+decoder%2Donly+architecture+in+multilingual+neural+machine+translatio)
29. Zhi Qu, Yiran Wang, Jiannan Mao, Chenchen Ding, Hideki Tanaka, Masao Utiyama, and Taro Watanabe. 2025c. Registering source tokens to target language spaces in multilingual neural machine translation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 21687–21706, Vienna, Austria. [↗](https://doi.org/10.48550/arxiv.2501.02979)
30. Arle Richard Lommel, Aljoscha Burchardt, and Hans Uszkoreit. 2013. Multidimensional quality metrics: a flexible system for assessing translation quality. In Proceedings of Translating and the Computer 35, London, UK. Aslib. [find](https://scholar.google.com/scholar?q=Arle+Richard+Lommel%2C+Aljoscha+Burchardt%2C+and+Hans+Uszkoreit%2E+2013%2E+Multidimensional+quality+metrics%3A+a+flexible+system+for+assessing+translation+quality%2E+In+Proceedings+of+Translating+and+the+Computer) [our card](https://md.research.elamur.ai/free/papers/lommel-2013-multidimensional-quality-metrics-flexible-system/)
31. Zhi Qu and Taro Watanabe. 2022. Adapting to non-centered languages for zero-shot multilingual translation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5251– 5265, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. [↗](https://doi.org/10.48448/vgzs-xz54)
32. Boxuan Lyu, Haiyue Song, Hidetaka Kamigaito, Chenchen Ding, Hideki Tanaka, Masao Utiyama, Kotaro Funakoshi, and Manabu Okumura. 2025. Minimum bayes risk decoding for error span detection in reference-free automatic machine translation evaluation. Preprint, arXiv:2512.07540. [↗](https://arxiv.org/abs/2512.07540)
33. Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022a. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Jos%C3%A9+G%2E+C%2E+de+Souza%2C+Duarte+Alves%2C+Chrysoula+Zerva%2C+Ana+C+Farinha%2C+Taisiya+Glushkova%2C+Alon+Lavie%2C+Luisa+Coheur%2C+and+Andr%C3%A9+F%2E+T%2E+Martins%2E+2022a%2E+COMET%2D22%3A+Unbabel%2DIST+2022+submission+for+t) [our card](https://md.research.elamur.ai/free/papers/rei-2022-comet-22-unbabel-ist-2022/)
34. NLLB Team. 2022. No language left behind: Scaling human-centered machine translation. Preprint, arXiv:2207.04672. [↗](https://arxiv.org/abs/2207.04672)
35. OpenAI. 2024. Gpt-4 technical report. arXiv:2303.08774. [↗](https://arxiv.org/abs/2303.08774) [our card](https://md.research.elamur.ai/free/papers/openai-2023-gpt-4-technical-report/)
36. Preprint, Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. [find](https://scholar.google.com/scholar?q=Preprint%2C+Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+Associatio) [our card](https://md.research.elamur.ai/free/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
37. Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André F. T. Martins, and Alon Lavie. 2021. Are references really needed? unbabel-IST 2021 submission for the metrics shared task. In Proceedings of the Sixth Conference on Machine Translation, pages 1030–1040, Online. Association for Computational Linguistics. Ricardo Rei, Nuno M. Guerreiro, José Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, José G. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Ana+C+Farinha%2C+Chrysoula+Zerva%2C+Daan+van+Stigt%2C+Craig+Stewart%2C+Pedro+Ramos%2C+Taisiya+Glushkova%2C+Andr%C3%A9+F%2E+T%2E+Martins%2C+and+Alon+Lavie%2E+2021%2E+Are+references+really+needed%3F+unbabel%2DIST+2021+su) [our card](https://md.research.elamur.ai/free/papers/rei-2021-are-references-really-needed-unbabel/)
38. C. de Souza, and André F. T. Martins. 2023. Scaling up CometKiwi: Unbabel-IST 2023 submission for the quality estimation shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 841–848, Singapore. [find](https://scholar.google.com/scholar?q=C%2E+de+Souza%2C+and+Andr%C3%A9+F%2E+T%2E+Martins%2E+2023%2E+Scaling+up+CometKiwi%3A+Unbabel%2DIST+2023+submission+for+the+quality+estimation+shared+task%2E+In+Proceedings+of+the+Eighth+Conference+on+Machine+Translation%2C+pa) [our card](https://md.research.elamur.ai/free/papers/rei-2023-scaling-up-cometkiwi-unbabel-ist/)
39. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Craig+Stewart%2C+Ana+C+Farinha%2C+and+Alon+Lavie%2E+2020%2E+COMET%3A+A+neural+framework+for+MT+evaluation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+) [our card](https://md.research.elamur.ai/free/papers/rei-2020-comet-neural-framework-mt-evaluation/)
40. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022b. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2209.06243) [our card](https://md.research.elamur.ai/free/papers/rei-2022-cometkiwi-ist-unbabel-2022-submission/)
41. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Dipanjan+Das%2C+and+Ankur+Parikh%2E+2020%2E+BLEURT%3A+Learning+robust+metrics+for+text+generation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+)
42. Shaomu Tan and Christof Monz. 2025. ReMedy: Learning machine translation evaluation from human preferences with reward modeling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 4370–4387, Suzhou, China. [↗](https://doi.org/10.48448/7fbp-g893)
43. David Vilar, Gregor Leusch, Hermann Ney, and Rafael E. Banchs. 2007. Human evaluation of machine translation through binary system comparisons. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 96–103, Prague, Czech Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+Vilar%2C+Gregor+Leusch%2C+Hermann+Ney%2C+and+Rafael+E%2E+Banchs%2E+2007%2E+Human+evaluation+of+machine+translation+through+binary+system+comparisons%2E+In+Proceedings+of+the+Second+Workshop+on+Statistical+Mac)
44. Pius Von Däniken, Jan Milan Deriu, and Mark Cieliebak. 2025. A measure of the system dependence of automated metrics. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 87–99, Vienna, Austria. [↗](https://doi.org/10.21256/zhaw-36126)
45. Weiqi Wang, Limeng Cui, Xin Liu, Sreyashi Nag, Wenju Xu, Chen Luo, Sheikh Muhammad Sarwar, Yang Li, Hansu Gu, Hui Liu, Changlong Yu, Jiaxin Bai, Yifan Gao, Haiyang Zhang, Qi He, Shuiwang Ji, and Yangqiu Song. 2025. EcomScriptBench: A multi-task benchmark for E-commerce script planning via step-wise intention-driven product association. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1–22, Vienna, Austria. [↗](https://doi.org/10.48550/arxiv.2505.15196)
46. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. [↗](https://doi.org/10.48550/arxiv.2010.11934)
47. Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat. 2021. Share or not? learning to schedule language-specific capacity for multilingual translation. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Biao+Zhang%2C+Ankur+Bapna%2C+Rico+Sennrich%2C+and+Orhan+Firat%2E+2021%2E+Share+or+not%3F+learning+to+schedule+language%2Dspecific+capacity+for+multilingual+translation%2E+In+International+Conference+on+Learning+Repre)