---
title: "Remedy-R: Generative Reasoning for Machine — Tan 2026"
description: "Neural MT metrics such as xCOMET, MetricX and Remedy correlate well with human judgments, but they output only a single number."
authors: ["Shaomu Tan", "Ryosuke Mitani", "Ritvik Choudhary", "Qiyu Wu", "Toshiyuki Sekiya", "Christof Monz"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.364/"
doi: "10.18653/v1/2026.findings-acl.364"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tan-2026-remedy-r-generative-reasoning-machine/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tan-2026-remedy-r-generative-reasoning-machine/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/) / Remedy-R: Generative Reasoning for Machine Translation Evaluation without Error Annotations

# Remedy-R: Generative Reasoning for Machine Translation Evaluation without Error Annotations

**Authors:** Shaomu Tan, Ryosuke Mitani, Ritvik Choudhary, Qiyu Wu, Toshiyuki Sekiya, [Christof Monz](https://md.research.elamur.ai/free/en/authors/christof-monz/) · **Year:** 2026 · **Venue:** Findings of the Association for Computational Linguistics: ACL 2026 · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.364/) · [PDF](https://aclanthology.org/2026.findings-acl.364.pdf) · [DOI 10.18653/v1/2026.findings-acl.364](https://doi.org/10.18653/v1/2026.findings-acl.364) · [OpenAlex](https://openalex.org/W7166883974)

**Citations:** 1 citations (openalex-snapshot · 2026-10-14) · 1 in our library, of the 140673 papers whose bibliographies we read

## Summary
Neural MT metrics such as xCOMET, MetricX and Remedy correlate well with human judgments, but they output only a single number. This makes them hard to interpret, and they can fail on out-of-distribution inputs such as copied sources or perturbed text. Remedy-R is an open generative evaluator that first writes an analysis covering accuracy, fluency and completeness, then gives a score from 0 to 100.

The authors train it with Reinforcement Learning via Verifiable Rewards (PPO). The training signal comes from pairwise human preferences derived from WMT20 MQM scores. The method needs no error-span annotations, no distillation from GPT-4 and no SFT warm-up. The reward checks whether the predicted ranking of two translations matches the human ranking. A Huber-style calibration term then pulls the predicted scores toward the human scores.

Trained on only 60K pairs from two language pairs, Remedy-R is competitive with leading scalar metrics and GPT-4-based judges on WMT22–25 and holds up well on OOD stress tests. Its analyses can also guide a training-free evaluate–revise loop that improves translations. This shows that interpretable, reusable MT evaluation is possible without costly annotation.

## Key points
- **Reward design:** a binary pairwise ranking reward is multiplied by a Huber-shaped calibration penalty (tolerance c=5). Unparsable outputs and ties receive zero reward, and the A/B order is randomized to reduce position bias.
- **Evaluation:** benchmarked on the WMT22–25 metric meta-evaluation, with generalization to languages beyond the training pairs.
- **Remedy-R Agent:** the evaluate–revise loop improved output from Qwen2.5, ALMA-R, GPT-4o-mini and Gemini-2.0-Flash. It covered 11 language pairs beyond the two used in training.
- **Closed LLMs:** on paragraph-level WMT24++, Remedy-R-32B feedback improved GPT-4o-mini about as much as step-by-step self-refinement. For Gemini-2.0-Flash, it recovered a large share of the self-refinement gains.
- **Faithfulness check:** explanations were validated with GPT-4o-mini on 900 WMT22 samples rather than with human raters. Code is available at github.com/Smu-Tan/Remedy-R.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tan2026remedy,
  title = {{Remedy-R: Generative Reasoning for Machine Translation Evaluation without Error Annotations}},
  author = {Tan, Shaomu and Mitani, Ryosuke and Choudhary, Ritvik and Wu, Qiyu and Sekiya, Toshiyuki and Monz, Christof},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.364},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.364}
}
```

- [← previous Reformulate and Create, Don't Translate: Creating Natural Prompts for Underserved Languages](https://md.research.elamur.ai/free/en/papers/simonsen-2026-reformulate-create-don-t-translate/)
- [next → ReNikud: Audio-Supervised Hebrew Grapheme-to-Phoneme Conversion](https://md.research.elamur.ai/free/en/papers/langstream-2026-melichov-renikud-audio-supervised-hebrew-grapheme/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/)

## References

51 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. [↗](https://arxiv.org/abs/2303.08774) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
2. Eleftheria Briakou, Jiaming Luo, Colin Cherry, and Markus Freitag. 2024. Translating step-by-step: Decomposing the translation process for improved translation quality of long-form texts. In Proceedings of the Ninth Conference on Machine Translation, pages 1301–1317. [find](https://scholar.google.com/scholar?q=Eleftheria+Briakou%2C+Jiaming+Luo%2C+Colin+Cherry%2C+and+Markus+Freitag%2E+2024%2E+Translating+step%2Dby%2Dstep%3A+Decomposing+the+translation+process+for+improved+translation+quality+of+long%2Dform+texts%2E+In+Proceedin) [our card](https://md.research.elamur.ai/free/en/papers/briakou-2024-translating-step-step-decomposing-translation/)
3. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113. [↗](https://doi.org/10.48550/arxiv.2204.02311)
4. Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. [↗](https://arxiv.org/abs/2207.04672)
5. Daniel Deutsch, Rotem Dror, and Dan Roth. 2021. A statistical analysis of summarization evaluation metrics using resampling methods. Transactions of the Association for Computational Linguistics, 9:1132– 1146. [↗](https://doi.org/10.1162/tacl_a_00417)
6. Daniel Deutsch, George Foster, and Markus Freitag. 2023. Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12914– 12929. [find](https://scholar.google.com/scholar?q=Daniel+Deutsch%2C+George+Foster%2C+and+Markus+Freitag%2E+2023%2E+Ties+matter%3A+Meta%2Devaluating+modern+metrics+with+pairwise+accuracy+and+tie+calibration%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Meth)
7. Zhaopeng Feng, Shaosheng Cao, Jiahan Ren, Jiayuan Su, Ruizhe Chen, Yan Zhang, Zhe Xu, Yao Hu, Jian Wu, and Zuozhu Liu. 2025a. Mt-r1-zero: Advancing llm-based machine translation via r1-zero-like reinforcement learning. arXiv preprint arXiv:2504.10160. [↗](https://arxiv.org/abs/2504.10160)
8. Zhaopeng Feng, Jiayuan Su, Jiamei Zheng, Jiahan Ren, Yan Zhang, Jian Wu, Hongwei Wang, and Zuozhu Liu. 2025b. M-MAD: Multidimensional multi-agent debate for advanced machine translation evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7084–7107. [find](https://scholar.google.com/scholar?q=Zhaopeng+Feng%2C+Jiayuan+Su%2C+Jiamei+Zheng%2C+Jiahan+Ren%2C+Yan+Zhang%2C+Jian+Wu%2C+Hongwei+Wang%2C+and+Zuozhu+Liu%2E+2025b%2E+M%2DMAD%3A+Multidimensional+multi%2Dagent+debate+for+advanced+machine+translation+evaluation%2E+In) [our card](https://md.research.elamur.ai/free/en/papers/feng-2025-m-mad-multidimensional-multi-agent/)
9. Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André FT Martins, Graham Neubig, Ankush Garg, Jonathan H Clark, Markus Freitag, and Orhan Firat. 2023. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. In Proceedings of the Eighth Conference on Machine Translation, pages 1066–1083. [↗](https://doi.org/10.48550/arxiv.2308.07286) [our card](https://md.research.elamur.ai/free/en/papers/fernandes-2023-devil-is-errors-leveraging-large/)
10. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460–1474. [↗](https://doi.org/10.1162/tacl_a_00437) [our card](https://md.research.elamur.ai/free/en/papers/freitag-2021-experts-errors-context-large-scale/)
11. Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frederic Blain, Tom Kocmi, Jiayi Wang, David Ifeoluwa Adelani, Marianna Buchicchio, Chrysoula Zerva, and Alon Lavie. 2024. Are LLMs breaking MT metrics? results of the WMT24 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, pages 47–81, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Markus+Freitag%2C+Nitika+Mathur%2C+Daniel+Deutsch%2C+Chi%2DKiu+Lo%2C+Eleftherios+Avramidis%2C+Ricardo+Rei%2C+Brian+Thompson%2C+Frederic+Blain%2C+Tom+Kocmi%2C+Jiayi+Wang%2C+David+Ifeoluwa+Adelani%2C+Marianna+Buchicchio%2C+Chrys) [our card](https://md.research.elamur.ai/free/en/papers/freitag-2024-are-llms-breaking-mt-metrics/)
12. Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023. Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent. In Proceedings of the Eighth Conference on Machine Translation, pages 578–628, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Markus+Freitag%2C+Nitika+Mathur%2C+Chi%2Dkiu+Lo%2C+Eleftherios+Avramidis%2C+Ricardo+Rei%2C+Brian+Thompson%2C+Tom+Kocmi%2C+Frederic+Blain%2C+Daniel+Deutsch%2C+Craig+Stewart%2C+Chrysoula+Zerva%2C+Sheila+Castilho%2C+Alon+Lavie%2C+a) [our card](https://md.research.elamur.ai/free/en/papers/freitag-2023-results-wmt23-metrics-shared-task/)
13. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022. Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46–68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Markus+Freitag%2C+Ricardo+Rei%2C+Nitika+Mathur%2C+Chi%2Dkiu+Lo%2C+Craig+Stewart%2C+Eleftherios+Avramidis%2C+Tom+Kocmi%2C+George+Foster%2C+Alon+Lavie%2C+and+Andr%C3%A9+F%2E+T%2E+Martins%2E+2022%2E+Results+of+WMT22+metrics+shared+task%3A) [our card](https://md.research.elamur.ai/free/en/papers/freitag-2022-results-wmt22-metrics-shared-task/)
14. Nuno M Guerreiro, Ricardo Rei, Daan Van Stigt, Luísa Coheur, Pierre Colombo, and André FT Martins. 2024. xcomet: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979–995. [↗](https://doi.org/10.48550/arxiv.2310.10482) [our card](https://md.research.elamur.ai/free/en/papers/guerreiro-2024-xcomet-transparent-machine-translation-evaluation/)
15. Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. 2025. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290. [↗](https://arxiv.org/abs/2503.24290) [our card](https://md.research.elamur.ai/free/en/papers/hu-2025-open-reasoner-zero-open-source/)
16. Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. Metricx-24: The google submission to the wmt 2024 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, pages 492–504. [find](https://scholar.google.com/scholar?q=Juraj+Juraska%2C+Daniel+Deutsch%2C+Mara+Finkelstein%2C+and+Markus+Freitag%2E+2024%2E+Metricx%2D24%3A+The+google+submission+to+the+wmt+2024+metrics+shared+task%2E+In+Proceedings+of+the+Ninth+Conference+on+Machine+Tran) [our card](https://md.research.elamur.ai/free/en/papers/juraska-2024-metricx-24-google-submission-wmt/)
17. Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. 2023. Metricx-23: The google submission to the wmt 2023 metrics shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 756–767. [find](https://scholar.google.com/scholar?q=Juraj+Juraska%2C+Mara+Finkelstein%2C+Daniel+Deutsch%2C+Aditya+Siddhant%2C+Mehdi+Mirzazadeh%2C+and+Markus+Freitag%2E+2023%2E+Metricx%2D23%3A+The+google+submission+to+the+wmt+2023+metrics+shared+task%2E+In+Proceedings+of+t) [our card](https://md.research.elamur.ai/free/en/papers/juraska-2023-metricx-23-google-submission-wmt/)
18. Rebecca Knowles, Samuel Larkin, and Chi-Kiu Lo. 2024. Mslc24: Further challenges for metrics on a wide landscape of translation quality. In Proceedings of the Ninth Conference on Machine Translation, pages 475–491. [find](https://scholar.google.com/scholar?q=Rebecca+Knowles%2C+Samuel+Larkin%2C+and+Chi%2DKiu+Lo%2E+2024%2E+Mslc24%3A+Further+challenges+for+metrics+on+a+wide+landscape+of+translation+quality%2E+In+Proceedings+of+the+Ninth+Conference+on+Machine+Translation%2C+)
19. Tom Kocmi, Ekaterina Artemova, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Konstantin Dranch, Anton Dvorkovich, Sergey Dukanov, Mark Fishel, Markus Freitag, et al. 2025. Findings of the wmt25 general machine translation shared task: Time to stop evaluating on easy test sets. In Proceedings of the Tenth Conference on Machine Translation, pages 355–413. [find](https://scholar.google.com/scholar?q=Tom+Kocmi%2C+Ekaterina+Artemova%2C+Eleftherios+Avramidis%2C+Rachel+Bawden%2C+Ondr%CC%8Cej+Bojar%2C+Konstantin+Dranch%2C+Anton+Dvorkovich%2C+Sergey+Dukanov%2C+Mark+Fishel%2C+Markus+Freitag%2C+et+al%2E+2025%2E+Findings+of+the+wmt25) [our card](https://md.research.elamur.ai/free/en/papers/kocmi-2025-findings-wmt25-general-machine-translation/)
20. Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, et al. 2024. Findings of the wmt24 general machine translation shared task: The llm era is here but mt is not solved yet. In Proceedings of the Ninth Conference on Machine Translation, pages 1–46. [find](https://scholar.google.com/scholar?q=Tom+Kocmi%2C+Eleftherios+Avramidis%2C+Rachel+Bawden%2C+Ondr%CC%8Cej+Bojar%2C+Anton+Dvorkovich%2C+Christian+Federmann%2C+Mark+Fishel%2C+Markus+Freitag%2C+Thamme+Gowda%2C+Roman+Grundkiewicz%2C+et+al%2E+2024%2E+Findings+of+the+wmt24) [our card](https://md.research.elamur.ai/free/en/papers/kocmi-2024-findings-wmt24-general-machine-translation/)
21. Tom Kocmi and Christian Federmann. 2023a. Gemba-mqm: Detecting translation quality error spans with gpt-4. In Proceedings of the Eighth Conference on Machine Translation, pages 768–775. [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Christian+Federmann%2E+2023a%2E+Gemba%2Dmqm%3A+Detecting+translation+quality+error+spans+with+gpt%2D4%2E+In+Proceedings+of+the+Eighth+Conference+on+Machine+Translation%2C+pages+768%E2%80%93775%2E) [our card](https://md.research.elamur.ai/free/en/papers/kocmi-2023-gemba-mqm-detecting-translation-quality/)
22. Tom Kocmi and Christian Federmann. 2023b. Large language models are state-of-the-art evaluators of translation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193–203. [↗](https://doi.org/10.48550/arxiv.2302.14520)
23. Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference on Machine Translation, pages 478–494. [↗](https://doi.org/10.48550/arxiv.2107.10821)
24. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626. [find](https://scholar.google.com/scholar?q=Woosuk+Kwon%2C+Zhuohan+Li%2C+Siyuan+Zhuang%2C+Ying+Sheng%2C+Lianmin+Zheng%2C+Cody+Hao+Yu%2C+Joseph+Gonzalez%2C+Hao+Zhang%2C+and+Ion+Stoica%2E+2023%2E+Efficient+memory+management+for+large+language+model+serving+with+page)
25. Alon Lavie, Greg Hanneman, Sweta Agrawal, Diptesh Kanojia, Chi-Kiu Lo, Vilém Zouhar, Frederic Blain, Chrysoula Zerva, Eleftherios Avramidis, Sourabh Deoghare, et al. 2025. Findings of the wmt25 shared task on automated translation evaluation systems: Linguistic diversity is challenging and references still help. In Proceedings of the Tenth Conference on Machine Translation, pages 436–483. [find](https://scholar.google.com/scholar?q=Alon+Lavie%2C+Greg+Hanneman%2C+Sweta+Agrawal%2C+Diptesh+Kanojia%2C+Chi%2DKiu+Lo%2C+Vil%C3%A9m+Zouhar%2C+Frederic+Blain%2C+Chrysoula+Zerva%2C+Eleftherios+Avramidis%2C+Sourabh+Deoghare%2C+et+al%2E+2025%2E+Findings+of+the+wmt25+shared)
26. Chi-kiu Lo, Samuel Larkin, and Rebecca Knowles. 2023. Metric score landscape challenge (mslc23): Understanding metrics’ performance on a wider landscape of translation quality. In Proceedings of the Eighth Conference on Machine Translation, pages 776–799. [find](https://scholar.google.com/scholar?q=Chi%2Dkiu+Lo%2C+Samuel+Larkin%2C+and+Rebecca+Knowles%2E+2023%2E+Metric+score+landscape+challenge+%28mslc23%29%3A+Understanding+metrics%E2%80%99+performance+on+a+wider+landscape+of+translation+quality%2E+In+Proceedings+of+the+E)
27. Qingyu Lu, Liang Ding, Kanjian Zhang, Jinxia Zhang, and Dacheng Tao. 2025. Mqm-ape: Toward high-quality error annotation predictors with automatic post-editing in llm translation evaluators. In Proceedings of the 31st International Conference on Computational Linguistics, pages 5570–5587. [↗](https://doi.org/10.48550/arxiv.2409.14335) [our card](https://md.research.elamur.ai/free/en/papers/lu-2025-mqm-ape-high-quality-error/)
28. Qingyu Lu, Baopu Qiu, Liang Ding, Kanjian Zhang, Tom Kocmi, and Dacheng Tao. 2024. Error analysis prompting enables human-like translation evaluation in large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 8801–8816. [find](https://scholar.google.com/scholar?q=Qingyu+Lu%2C+Baopu+Qiu%2C+Liang+Ding%2C+Kanjian+Zhang%2C+Tom+Kocmi%2C+and+Dacheng+Tao%2E+2024%2E+Error+analysis+prompting+enables+human%2Dlike+translation+evaluation+in+large+language+models%2E+In+Findings+of+the+Assoc) [our card](https://md.research.elamur.ai/free/en/papers/lu-2024-error-analysis-prompting-enables-human/)
29. Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020. Results of the wmt20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 688–725. [find](https://scholar.google.com/scholar?q=Nitika+Mathur%2C+Johnny+Wei%2C+Markus+Freitag%2C+Qingsong+Ma%2C+and+Ondr%CC%8Cej+Bojar%2E+2020%2E+Results+of+the+wmt20+metrics+shared+task%2E+In+Proceedings+of+the+Fifth+Conference+on+Machine+Translation%2C+pages+688%E2%80%93725%2E) [our card](https://md.research.elamur.ai/free/en/papers/mathur-2020-results-wmt20-metrics-shared-task/)
30. Nikita Moghe, Arnisa Fazla, Chantal Amrhein, Tom Kocmi, Mark Steedman, Alexandra Birch, Rico Sennrich, and Liane Guillou. 2025. Machine translation meta evaluation through translation accuracy challenge sets. Computational Linguistics, 51(1):73–137. [↗](https://doi.org/10.5167/uzh-265275) [our card](https://md.research.elamur.ai/free/en/papers/moghe-2025-machine-translation-meta-evaluation-through/)
31. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. [↗](https://doi.org/10.3115/1073083.1073135) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
32. Stefano Perrella, Lorenzo Proietti, Alessandro Scirè, Edoardo Barba, and Roberto Navigli. 2024. Guardians of the machine translation meta-evaluation: Sentinel metrics fall in! In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16216–16244. [↗](https://doi.org/10.48550/arxiv.2408.13831) [our card](https://md.research.elamur.ai/free/en/papers/perrella-2024-guardians-machine-translation-meta-evaluation/)
33. Maja Popović. 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the tenth workshop on statistical machine translation, pages 392–395. [find](https://scholar.google.com/scholar?q=Maja+Popovic%CC%81%2E+2015%2E+chrf%3A+character+n%2Dgram+f%2Dscore+for+automatic+mt+evaluation%2E+In+Proceedings+of+the+tenth+workshop+on+statistical+machine+translation%2C+pages+392%E2%80%93395%2E) [our card](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/)
34. Lorenzo Proietti, Stefano Perrella, and Roberto Navigli. 2025. Has machine translation evaluation achieved human parity? the human reference and the limits of progress. arXiv preprint arXiv:2506.19571. [↗](https://arxiv.org/abs/2506.19571)
35. Shenbin Qian, Archchana Sindhujan, Minnie Kabra, Diptesh Kanojia, Constantin Orašan, Tharindu Ranasinghe, and Fred Blain. 2024. What do large language models need for machine translation evaluation? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3660–3674. [↗](https://doi.org/10.48550/arxiv.2410.03278)
36. Ricardo Rei, José GC De Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André FT Martins. 2022. Comet-22: Unbabel-ist 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Jos%C3%A9+GC+De+Souza%2C+Duarte+Alves%2C+Chrysoula+Zerva%2C+Ana+C+Farinha%2C+Taisiya+Glushkova%2C+Alon+Lavie%2C+Luisa+Coheur%2C+and+Andr%C3%A9+FT+Martins%2E+2022%2E+Comet%2D22%3A+Unbabel%2Dist+2022+submission+for+the+metr) [our card](https://md.research.elamur.ai/free/en/papers/rei-2022-comet-22-unbabel-ist-2022/)
37. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Craig+Stewart%2C+Ana+C+Farinha%2C+and+Alon+Lavie%2E+2020%2E+COMET%3A+A+neural+framework+for+MT+evaluation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+) [our card](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/)
38. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. [↗](https://arxiv.org/abs/1707.06347)
39. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256. [↗](https://arxiv.org/abs/2409.19256) [our card](https://md.research.elamur.ai/free/en/papers/sheng-2024-hybridflow-flexible-efficient-rlhf-framework/)
40. Richard Sproat, Tianyu Zhao, and Llion Jones. 2025. Transevalnia: Reasoning-based evaluation and ranking of translations. arXiv preprint arXiv:2507.12724. [↗](https://arxiv.org/abs/2507.12724)
41. Shaomu Tan, Ryosuke Mitani, Ritvik Choudhary, and Toshiyuki Sekiya. 2025. Investigating test-time scaling with reranking for machine translation. arXiv preprint arXiv:2509.19020. [↗](https://arxiv.org/abs/2509.19020)
42. Shaomu Tan and Christof Monz. 2025. ReMedy: Learning machine translation evaluation from human preferences with reward modeling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 4370–4387, Suzhou, China. Association for Computational Linguistics. [↗](https://doi.org/10.48448/7fbp-g893)
43. Brian Thompson, Nitika Mathur, Daniel Deutsch, and Huda Khayrallah. 2024. Improving statistical significance in human evaluation of automatic metrics via soft pairwise accuracy. In Proceedings of the Ninth Conference on Machine Translation, pages 1222–1234. [find](https://scholar.google.com/scholar?q=Brian+Thompson%2C+Nitika+Mathur%2C+Daniel+Deutsch%2C+and+Huda+Khayrallah%2E+2024%2E+Improving+statistical+significance+in+human+evaluation+of+automatic+metrics+via+soft+pairwise+accuracy%2E+In+Proceedings+of+the+)
44. Marcos Treviso, Nuno Guerreiro, Sweta Agrawal, Ricardo Rei, José Pombal, Tânia Vaz, Helena Wu, Beatriz Silva, Daan Stigt, and André FT Martins. 2024. xtower: A multilingual llm for explaining and correcting translation errors. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15222–15239. [↗](https://doi.org/10.48550/arxiv.2406.19482)
45. Di Wu, Seth Aycock, and Christof Monz. 2025. Please translate again: Two simple experiments on whether human-like reasoning helps translation. arXiv preprint arXiv:2506.04521. [↗](https://arxiv.org/abs/2506.04521)
46. Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2024a. A paradigm shift in machine translation: Boosting translation performance of large language models. In The Twelfth International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.2309.11674)
47. Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024b. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. In International Conference on Machine Learning, pages 55204– 55224. PMLR. [↗](https://doi.org/10.48550/arxiv.2401.08417) [our card](https://md.research.elamur.ai/free/en/papers/xu-2024-contrastive-preference-optimization-pushing-boundaries/)
48. Wenda Xu, Daniel Deutsch, Mara Finkelstein, Juraj Juraska, Biao Zhang, Zhongtao Liu, William Yang Wang, Lei Li, and Markus Freitag. 2024c. Llmrefine: Pinpointing and refining large language models via fine-grained actionable feedback. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1429–1445. [↗](https://doi.org/10.48550/arxiv.2311.09336)
49. Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yi-Chao Zhang, Yunyang Wan, Yuqi Liu, Zeyu Cui, Zhenru Zhang, Zihan Qiu, Shanghaoran Quan, and Zekun Wang. 2024. Qwen2.5 technical report. ArXiv, abs/2412.15115. [↗](https://arxiv.org/abs/2412.15115)
50. Shijie Zhang, Renhao Li, Songsheng Wang, Philipp Koehn, Min Yang, and Derek F Wong. 2025. Hi-mate: A hierarchical multi-agent framework for machine translation evaluation. arXiv preprint arXiv:2505.16281. [↗](https://arxiv.org/abs/2505.16281)
51. Mao Zheng, Zheng Li, Bingxin Qu, Mingyang Song, Yang Du, Mingrui Sun, and Di Wang. 2025. Hunyuan-mt technical report. arXiv preprint arXiv:2509.05209. [↗](https://arxiv.org/abs/2509.05209)