---
title: "Guardians of the Machine Translation — Perrella 2024"
description: "This paper examines whether the segment-level meta-evaluation used in the WMT Metrics Shared Task ranks MT metrics fairly."
authors: ["Stefano Perrella", "Lorenzo Proietti", "Alessandro Scirè", "Edoardo Barba", "Roberto Navigli"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.856/"
doi: "10.18653/v1/2024.acl-long.856"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/perrella-2024-guardians-machine-translation-meta-evaluation/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/perrella-2024-guardians-machine-translation-meta-evaluation/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2024](https://md.research.elamur.ai/free/papers/2024/page/2/) / Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!

# Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!

**Authors:** [Stefano Perrella](https://md.research.elamur.ai/free/authors/stefano-perrella/), [Lorenzo Proietti](https://md.research.elamur.ai/free/authors/lorenzo-proietti/), [Alessandro Scirè](https://md.research.elamur.ai/free/authors/alessandro-scire/), [Edoardo Barba](https://md.research.elamur.ai/free/authors/edoardo-barba/), [Roberto Navigli](https://md.research.elamur.ai/free/authors/roberto-navigli/) · **Year:** 2024 · **Venue:** Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.856/) · [PDF](https://aclanthology.org/2024.acl-long.856.pdf) · [DOI 10.18653/v1/2024.acl-long.856](https://doi.org/10.18653/v1/2024.acl-long.856) · [OpenAlex](https://openalex.org/W4402671237)

**Citations:** 1 citations (openalex-snapshot · 2026-10-14) · [23](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
This paper examines whether the segment-level meta-evaluation used in the WMT Metrics Shared Task ranks MT metrics fairly. The authors propose *sentinel metrics*, which are probes built to expose weaknesses in the evaluation protocol. Some are trained with deliberately missing information. For example, SENTINEL CAND scores a translation without seeing its source or reference. Others are modified copies of existing metrics, such as continuous versions of the discrete GEMBA-MQM and MaTESe.

Using WMT22/WMT23 data and the official mt-metrics-eval library, the authors identify two flaws:
- **Grouping:** Computing correlations without grouping, or grouping by system, lets trained metrics benefit from spurious features of the input.
- **Tie calibration:** Running pairwise accuracy with tie calibration on the test set itself gives continuous metrics an unfair edge over discrete ones.

They recommend grouping translations by source segment and not calibrating ties on test data. A held-out calibration set does not fully fix the problem either.

The work matters because WMT rankings steer metric and MT research. The authors also find that sentinels correlate strongly with leading metrics. This suggests those metrics may partly rely on spurious cues learned during training.

## Key points
- SENTINEL CAND sees only the candidate translation, yet it lands in the upper half of the WMT23 segment-level ranking, above the COMET and BLEURT-20 baselines.
- The WMT23 final ranking combines three statistics:
  - system-level pairwise accuracy,
  - system- and segment-level Pearson correlation,
  - segment-level pairwise accuracy with tie calibration (acc_eq).
- Under tie calibration, the optimal ϵ for continuous metrics stays at 0 until human ties reach 39% of pairs. As a result, the continuous SENTINEL GEMBA could rank second while the otherwise equivalent GEMBA-MQM ranks second to last.
- The share of human ties in WMT test sets ranges from 15.14% to 53.35%, with the maximum in WMT22 EN→DE.
- In an 80/20 held-out calibration experiment on ZH→EN, rankings shifted with the proportion of ties in the calibration set. Code and sentinel weights are available at github.com/SapienzaNLP/guardians-mt-eval.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/)

## Cited by

Cited by

23 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 18 not yet.

- [Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets](https://md.research.elamur.ai/free/papers/kocmi-2025-findings-wmt25-general-machine-translation/)
- [Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics](https://md.research.elamur.ai/free/papers/perrella-2024-beyond-correlation-interpretable-evaluation-machine/)
- [PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation](https://md.research.elamur.ai/free/papers/proietti-2026-pear-pairwise-evaluation-automatic-relative/)
- [Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains](https://md.research.elamur.ai/free/papers/schmidt-2026-who-watches-watchmen-humans-disagree/)
- [Remedy-R: Generative Reasoning for Machine Translation Evaluation without Error Annotations](https://md.research.elamur.ai/free/papers/tan-2026-remedy-r-generative-reasoning-machine/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{perrella2024guardians,
  title = {{Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!}},
  author = {Perrella, Stefano and Proietti, Lorenzo and Scirè, Alessandro and Barba, Edoardo and Navigli, Roberto},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.856},
  url = {https://doi.org/10.18653/v1/2024.acl-long.856}
}
```

- [← previous GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://md.research.elamur.ai/free/papers/mirzadeh-2024-gsm-symbolic-understanding-limitations-mathematical/)
- [next → GUMsley: Evaluating Entity Salience in Summarization for 12 English Genres](https://md.research.elamur.ai/free/papers/lin-2024-gumsley-evaluating-entity-salience-summarization/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/papers/2024/page/2/)

## References

28 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016. Results of the WMT16 metrics shared task. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 199–231, Berlin, Germany. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Yvette+Graham%2C+Amir+Kamran%2C+and+Milo%C5%A1+Stanojevic%CC%81%2E+2016%2E+Results+of+the+WMT16+metrics+shared+task%2E+In+Proceedings+of+the+First+Conference+on+Machine+Translation%3A+Volume+2%2C+Shared+Task+P) [our card](https://md.research.elamur.ai/free/papers/bojar-2016-results-wmt16-metrics-shared-task/)
2. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dlingu)
3. Daniel Deutsch, Rotem Dror, and Dan Roth. 2021. A statistical analysis of summarization evaluation metrics using resampling methods. Transactions of the Association for Computational Linguistics, 9:1132– 1146. [↗](https://doi.org/10.1162/tacl_a_00417)
4. Daniel Deutsch, George Foster, and Markus Freitag. 2023. Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12914– 12929, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Daniel+Deutsch%2C+George+Foster%2C+and+Markus+Freitag%2E+2023%2E+Ties+matter%3A+Meta%2Devaluating+modern+metrics+with+pairwise+accuracy+and+tie+calibration%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Meth)
5. Sören Dreano, Derek Molloy, and Noel Murphy. 2023. Embed_Llama: Using LLM embeddings for the metrics shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 738–745, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=S%C3%B6ren+Dreano%2C+Derek+Molloy%2C+and+Noel+Murphy%2E+2023%2E+Embed%5FLlama%3A+Using+LLM+embeddings+for+the+metrics+shared+task%2E+In+Proceedings+of+the+Eighth+Conference+on+Machine+Translation%2C+pages+738%E2%80%93745%2C+Singapo)
6. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460–1474. [↗](https://doi.org/10.1162/tacl_a_00437) [our card](https://md.research.elamur.ai/free/papers/freitag-2021-experts-errors-context-large-scale/)
7. Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023. Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent. In Proceedings of the Eighth Conference on Machine Translation, pages 578–628, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Markus+Freitag%2C+Nitika+Mathur%2C+Chi%2Dkiu+Lo%2C+Eleftherios+Avramidis%2C+Ricardo+Rei%2C+Brian+Thompson%2C+Tom+Kocmi%2C+Frederic+Blain%2C+Daniel+Deutsch%2C+Craig+Stewart%2C+Chrysoula+Zerva%2C+Sheila+Castilho%2C+Alon+Lavie%2C+a) [our card](https://md.research.elamur.ai/free/papers/freitag-2023-results-wmt23-metrics-shared-task/)
8. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022. Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46–68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Markus+Freitag%2C+Ricardo+Rei%2C+Nitika+Mathur%2C+Chi%2Dkiu+Lo%2C+Craig+Stewart%2C+Eleftherios+Avramidis%2C+Tom+Kocmi%2C+George+Foster%2C+Alon+Lavie%2C+and+Andr%C3%A9+F%2E+T%2E+Martins%2E+2022%2E+Results+of+WMT22+metrics+shared+task%3A) [our card](https://md.research.elamur.ai/free/papers/freitag-2022-results-wmt22-metrics-shared-task/)
9. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021b. Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain. In Proceedings of the Sixth Conference on Machine Translation, pages 733–774, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Markus+Freitag%2C+Ricardo+Rei%2C+Nitika+Mathur%2C+Chi%2Dkiu+Lo%2C+Craig+Stewart%2C+George+Foster%2C+Alon+Lavie%2C+and+Ondr%CC%8Cej+Bojar%2E+2021b%2E+Results+of+the+WMT21+metrics+shared+task%3A+Evaluating+metrics+with+expert%2Dbas) [our card](https://md.research.elamur.ai/free/papers/freitag-2021-results-wmt21-metrics-shared-task/)
10. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Timothy+Baldwin%2C+Alistair+Moffat%2C+and+Justin+Zobel%2E+2013%2E+Continuous+measurement+scales+in+human+evaluation+of+machine+translation%2E+In+Proceedings+of+the+7th+Linguistic+Annotation+Works)
11. Nuno Miguel Guerreiro, Ricardo Rei, Daan van Stigt, Luísa Coheur, Pierre Colombo, and André F. T. Martins. 2023. xCOMET: Transparent machine translation evaluation through fine-grained error detection. CoRR, abs/2310.10482. [↗](https://arxiv.org/abs/2310.10482) [our card](https://md.research.elamur.ai/free/papers/guerreiro-2024-xcomet-transparent-machine-translation-evaluation/)
12. Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. 2023. MetricX-23: The Google submission to the WMT 2023 metrics shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 756–767, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Juraj+Juraska%2C+Mara+Finkelstein%2C+Daniel+Deutsch%2C+Aditya+Siddhant%2C+Mehdi+Mirzazadeh%2C+and+Markus+Freitag%2E+2023%2E+MetricX%2D23%3A+The+Google+submission+to+the+WMT+2023+metrics+shared+task%2E+In+Proceedings+of+t) [our card](https://md.research.elamur.ai/free/papers/juraska-2023-metricx-23-google-submission-wmt/)
13. M. G. Kendall. 1945. The treatment of ties in ranking problems. Biometrika, 33:239–51. [↗](https://doi.org/10.1093/biomet/33.3.239)
14. Tom Kocmi and Christian Federmann. 2023. GEMBA-MQM: Detecting translation quality error spans with GPT-4. In Proceedings of the Eighth Conference on Machine Translation, pages 768–775, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Christian+Federmann%2E+2023%2E+GEMBA%2DMQM%3A+Detecting+translation+quality+error+spans+with+GPT%2D4%2E+In+Proceedings+of+the+Eighth+Conference+on+Machine+Translation%2C+pages+768%E2%80%93775%2C+Singapore%2E+Asso) [our card](https://md.research.elamur.ai/free/papers/kocmi-2023-gemba-mqm-detecting-translation-quality/)
15. Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference on Machine Translation, pages 478–494, Online. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2107.10821)
16. Tom Kocmi, Vilém Zouhar, Christian Federmann, and Matt Post. 2024. Navigating the metrics maze: Reconciling score magnitudes and accuracies. [↗](https://doi.org/10.48550/arxiv.2401.06760)
17. Philipp Koehn and Christof Monz. 2006. Manual and automatic evaluation of machine translation between European languages. In Proceedings on the Workshop on Statistical Machine Translation, pages 102– 121, New York City. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Philipp+Koehn+and+Christof+Monz%2E+2006%2E+Manual+and+automatic+evaluation+of+machine+translation+between+European+languages%2E+In+Proceedings+on+the+Workshop+on+Statistical+Machine+Translation%2C+pages+102%E2%80%93+) [our card](https://md.research.elamur.ai/free/papers/koehn-2006-manual-automatic-evaluation-machine-translation/)
18. Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2020. On the variance of the adaptive learning rate and beyond. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [↗](https://doi.org/10.48550/arxiv.1908.03265)
19. Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional quality metrics (MQM) : a framework for declaring and describing translation quality metrics. Tradumàtica, (12):455–463. [↗](https://doi.org/10.5565/rev/tradumatica.77)
20. Matouš Macháček and Ondřej Bojar. 2013. Results of the WMT13 metrics shared task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 45–51, Sofia, Bulgaria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Matou%C5%A1+Mach%C3%A1c%CC%8Cek+and+Ondr%CC%8Cej+Bojar%2E+2013%2E+Results+of+the+WMT13+metrics+shared+task%2E+In+Proceedings+of+the+Eighth+Workshop+on+Statistical+Machine+Translation%2C+pages+45%E2%80%9351%2C+Sofia%2C+Bulgaria%2E+Association+)
21. Matouš Macháček and Ondřej Bojar. 2014. Results of the WMT14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 293–301, Baltimore, Maryland, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Matou%C5%A1+Mach%C3%A1c%CC%8Cek+and+Ondr%CC%8Cej+Bojar%2E+2014%2E+Results+of+the+WMT14+metrics+shared+task%2E+In+Proceedings+of+the+Ninth+Workshop+on+Statistical+Machine+Translation%2C+pages+293%E2%80%93301%2C+Baltimore%2C+Maryland%2C+USA%2E+As) [our card](https://md.research.elamur.ai/free/papers/machacek-2014-results-wmt14-metrics-shared-task/)
22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. [↗](https://doi.org/10.3115/1073083.1073135) [our card](https://md.research.elamur.ai/free/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
23. Stefano Perrella, Lorenzo Proietti, Alessandro Scirè, Niccolò Campolungo, and Roberto Navigli. 2022. MaTESe: Machine translation evaluation as a sequence tagging problem. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 569–577, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Stefano+Perrella%2C+Lorenzo+Proietti%2C+Alessandro+Scir%C3%A8%2C+Niccol%C3%B2+Campolungo%2C+and+Roberto+Navigli%2E+2022%2E+MaTESe%3A+Machine+translation+evaluation+as+a+sequence+tagging+problem%2E+In+Proceedings+of+the+Seventh) [our card](https://md.research.elamur.ai/free/papers/perrella-2022-matese-machine-translation-evaluation-as/)
24. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/w15-3049) [our card](https://md.research.elamur.ai/free/papers/popovic-2015-chrf-character-n-gram-f/)
25. Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022a. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Jos%C3%A9+G%2E+C%2E+de+Souza%2C+Duarte+Alves%2C+Chrysoula+Zerva%2C+Ana+C+Farinha%2C+Taisiya+Glushkova%2C+Alon+Lavie%2C+Luisa+Coheur%2C+and+Andr%C3%A9+F%2E+T%2E+Martins%2E+2022a%2E+COMET%2D22%3A+Unbabel%2DIST+2022+submission+for+t) [our card](https://md.research.elamur.ai/free/papers/rei-2022-comet-22-unbabel-ist-2022/)
26. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Craig+Stewart%2C+Ana+C+Farinha%2C+and+Alon+Lavie%2E+2020%2E+COMET%3A+A+neural+framework+for+MT+evaluation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+) [our card](https://md.research.elamur.ai/free/papers/rei-2020-comet-neural-framework-mt-evaluation/)
27. Martins. 2022b. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [↗](https://doi.org/10.48448/9bnx-f091) [our card](https://md.research.elamur.ai/free/papers/rei-2022-cometkiwi-ist-unbabel-2022-submission/)
28. Parker Riley, Daniel Deutsch, George Foster, Viresh Ratnakar, Ali Dabirmoghaddam, and Markus Freitag. 2024. Finding replicable human evaluations via stable ranking probability. [↗](https://doi.org/10.48550/arxiv.2404.01474)