---
title: "How to Select Datapoints for Efficient Human Evaluation of NLG"
description: "They propose output-based selectors: metric average, metric variance, metric consistency, output… 64 references, cited by 16 papers here."
authors: ["Vilém Zouhar", "Peng Cui", "Mrinmaya Sachan"]
year: 2025
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2025.tacl-1.80/"
doi: "10.1162/tacl.a.60"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zouhar-2025-how-select-datapoints-efficient-human/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zouhar-2025-how-select-datapoints-efficient-human/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/) / How to Select Datapoints for Efficient Human Evaluation of NLG Models?

# How to Select Datapoints for Efficient Human Evaluation of NLG Models?

**Authors:** [Vilém Zouhar](https://md.research.elamur.ai/free/en/authors/vilem-zouhar/), [Peng Cui](https://md.research.elamur.ai/free/en/authors/peng-cui/) and [Mrinmaya Sachan](https://md.research.elamur.ai/free/en/authors/mrinmaya-sachan/) · **Year:** 2025 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2025](https://md.research.elamur.ai/free/en/venues/tacl/2025/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.tacl-1.80/) · [PDF](https://aclanthology.org/2025.tacl-1.80.pdf) · [DOI 10.1162/tacl.a.60](https://doi.org/10.1162/tacl.a.60) · [OpenAlex](https://openalex.org/W7114889746)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [16](https://md.research.elamur.ai/free/en/papers/zouhar-2025-how-select-datapoints-efficient-human/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Human judgments remain the most trusted way to compare text generation systems, but annotating whole test sets is too costly, so practitioners usually annotate a random sample. This paper asks how to choose test items that are most informative, so that model rankings from a smaller human-annotated subset match those from the full set. The authors treat this as a budget-constrained selection problem. Under additive cost and utility assumptions it becomes a 0–1 knapsack, which they solve greedily or with ILP solvers when item costs vary. They propose output-based selectors:
- metric average,
- metric variance,
- metric consistency,
- output diversity,
- Item Response Theory.

For cases where system outputs do not yet exist, such as blind shared-task test sets, they distill these selectors into source-based estimators that predict an item's usefulness from the input text alone. Subset quality is measured with soft pairwise accuracy against the full-set ranking. On machine translation (WMT) and summarization (SummEval), variance-, consistency-, diversity- and IRT-based selection beat random sampling. Roughly 70% of annotations suffice to match the full-data result, a meaningful saving for large evaluation campaigns.

## Key points
- The baseline is random subset selection, repeated 100 times with confidence intervals from Student's t distribution. Rankings are compared using soft pairwise accuracy (Thompson et al., 2024) rather than Spearman or Kendall correlation.
- Selecting the hardest items (lowest average metric score) brings little gain and can backfire, because such items tend to be costly to annotate.
- The authors give practical recommendations for choosing a selector:
  - With outputs and a reliable metric, use metric variance or consistency.
  - With outputs but no reliable metric, use output diversity.
  - With only historical data, use source-based diversity estimation.
  - With partial outputs, use an artificial crowd combined with metric variance.
- The method's effectiveness tracks how well automated metrics agree with humans. A key limitation is that selection may hide failure types the metrics fail to detect.
- The released subset2evaluate package includes code and pre-trained item-utility estimators for MT. The authors also claim this is the first cost-aware method for evaluation subset selection.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [ETH Zurich](https://md.research.elamur.ai/free/en/universities/eth-zurich/) — University, Zurich, Switzerland · 72 works · 5,176 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [✍️ Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) · 202 works

## Cite

Cite as: Vilém Zouhar, Peng Cui and Mrinmaya Sachan. 2025. How to Select Datapoints for Efficient Human Evaluation of NLG Models?. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{zouhar2025how,
  title = {{How to Select Datapoints for Efficient Human Evaluation of NLG Models?}},
  author = {Vilém Zouhar, Peng Cui and Mrinmaya Sachan},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.1162/tacl.a.60},
  url = {https://doi.org/10.1162/tacl.a.60}
}
```

- [← previous Diversity-oriented Data Augmentation with Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2025-diversity-oriented-data-augmentation-large/)
- [next → Hunyuan-MT Technical Report](https://md.research.elamur.ai/free/en/papers/zheng-2025-hunyuan-mt-technical-report/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/)

## References

64 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ibrahim Said Ahmad, Antonios Anastasopoulos, Ondřej Bojar, Claudia Borg, Marine Carpuat, Roldano Cattoni, Mauro Cettolo, William Chen, Qianqian Dong, Marcello Federico, Barry Haddow, Dávid Javorský, Mateusz Krubiński, Tsz Kin Lam, Xutai Ma, Prashant Mathur, Evgeny Matusov, Chandresh Maurya, John… [find](https://scholar.google.com/scholar?q=Ibrahim+Said+Ahmad%2C+Antonios+Anastasopoulos%2C+Ondr%CC%8Cej+Bojar%2C+Claudia+Borg%2C+Marine+Carpuat%2C+Roldano+Cattoni%2C+Mauro+Cettolo%2C+William+Chen%2C+Qianqian+Dong%2C+Marcello+Federico%2C+Barry+Haddow%2C+Da%CC%81vid+Javorsky%CC%81)
2. Shir Ashury Tahan, Ariel Gera, Benjamin Sznajder, Leshem Choshen, Liat Ein-Dor, and Eyal Shnarch. 2024. Label-efficient model selection for text generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8384–8402. Associatio… [find](https://scholar.google.com/scholar?q=Shir+Ashury+Tahan%2C+Ariel+Gera%2C+Benjamin+Sznajder%2C+Leshem+Choshen%2C+Liat+Ein%2DDor%2C+and+Eyal+Shnarch%2E+2024%2E+Label%2Defficient+model+selection+for+text+generation%2E+In+Proceedings+of+the+62nd+Annual+Meeting+o)
3. Luca Benedetto, Andrea Cappelli, Roberto Turrin, and Paolo Cremonesi. 2020. R2DE: A NLP approach to estimating IRT parameters of newly generated questions. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, pages 412–421. [find](https://scholar.google.com/scholar?q=Luca+Benedetto%2C+Andrea+Cappelli%2C+Roberto+Turrin%2C+and+Paolo+Cremonesi%2E+2020%2E+R2DE%3A+A+NLP+approach+to+estimating+IRT+parameters+of+newly+generated+questions%2E+In+Proceedings+of+the+Tenth+International+Co)
4. Matthew Byrd and Shashank Srivastava. 2022. Predicting difficulty and discrimination of natural language questions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 119–130. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Matthew+Byrd+and+Shashank+Srivastava%2E+2022%2E+Predicting+difficulty+and+discrimination+of+natural+language+questions%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Lingu)
5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association f… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzma%CC%81n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dling)
6. Daniel Deutsch, Rotem Dror, and Dan Roth. 2021. [A statistical analysis of summarization evaluation metrics using resampling methods](https://md.research.elamur.ai/free/en/papers/deutsch-2021-statistical-analysis-summarization-evaluation-metrics/). Transactions of the Association for Computational Linguistics, 9:1132–1146. https://doi.org/10.1162/tacl a 00417 [↗](https://doi.org/10.1162/tacl)
7. Daniel Deutsch, Rotem Dror, and Dan Roth. 2022. Re-examining system-level correlations of automatic summarization evaluation metrics. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 6038–6052. As… [find](https://scholar.google.com/scholar?q=Daniel+Deutsch%2C+Rotem+Dror%2C+and+Dan+Roth%2E+2022%2E+Re%2Dexamining+system%2Dlevel+correlations+of+automatic+summarization+evaluation+metrics%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Chapte)
8. Daniel Deutsch, George Foster, and Markus Freitag. 2023. [Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration](https://md.research.elamur.ai/free/en/papers/deutsch-2023-ties-matter-meta-evaluating-modern/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12914–12929. Association for Computational Linguistics.
9. Shachar Don-Yehiya, Leshem Choshen, and Omri Abend. 2022. PreQuEL: Quality estimation of machine translation outputs in advance. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11170–11183. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shachar+Don%2DYehiya%2C+Leshem+Choshen%2C+and+Omri+Abend%2E+2022%2E+PreQuEL%3A+Quality+estimation+of+machine+translation+outputs+in+advance%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+L)
10. Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. https://doi.org/10.1162/tacl_a_00373 [↗](https://doi.org/10.1162/tacl_a_00373)
11. Júlia Falcão, Claudia Borg, Nora Aranberri, and Kurt Abela. 2024. COMET for low-resource machine translation evaluation: A case study of English-Maltese and Spanish-Basque. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LR… [find](https://scholar.google.com/scholar?q=Ju%CC%81lia+Falca%CC%83o%2C+Claudia+Borg%2C+Nora+Aranberri%2C+and+Kurt+Abela%2E+2024%2E+COMET+for+low%2Dresource+machine+translation+evaluation%3A+A+case+study+of+English%2DMaltese+and+Spanish%2DBasque%2E+In+Proceedings+of+the+202)
12. Kehua Feng, Keyan Ding, Kede Ma, Zhihua Wang, Qiang Zhang, and Huajun Chen. 2024. Sample-efficient human evaluation of large language models via maximum discrepancy competition. [find](https://scholar.google.com/scholar?q=Kehua+Feng%2C+Keyan+Ding%2C+Kede+Ma%2C+Zhihua+Wang%2C+Qiang+Zhang%2C+and+Huajun+Chen%2E+2024%2E+Sample%2Defficient+human+evaluation+of+large+language+models+via+maximum+discrepancy+competition%2E)
13. Ronald A. Fisher. 1935. The Design of Experiments. Oliver & Boyd, Edinburgh. [find](https://scholar.google.com/scholar?q=Ronald+A%2E+Fisher%2E+1935%2E+The+Design+of+Experiments%2E+Oliver+%26+Boyd%2C+Edinburgh%2E)
14. Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023. [Results of WMT23 metrics shared task: Metrics might be guilty but references…](https://md.research.elamur.ai/free/en/papers/freitag-2023-results-wmt23-metrics-shared-task/)
15. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022. [Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust](https://md.research.elamur.ai/free/en/papers/freitag-2022-results-wmt22-metrics-shared-task/). In Proceedings of the Seventh Confere…
16. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021. [Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain](https://md.research.elamur.ai/free/en/papers/freitag-2021-results-wmt21-metrics-shared-task/). In Proceedings of the Sixth Conference on Machine Transl…
17. Phillip Good. 2000. Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses. Springer. https://doi.org/10.1007/978-1-4757-3235-1 — [doi:10.1007/978-1-4757-3235-1](https://doi.org/10.1007/978-1-4757-3235-1) [↗](https://doi.org/10.1007/978-1-4757-3235-1)
18. Yvette Graham, Timothy Baldwin, and Nitika Mathur. 2015. [Accurate evaluation of segment-level machine translation metrics](https://md.research.elamur.ai/free/en/papers/graham-2015-accurate-evaluation-segment-level-machine/). In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1183–1191. Association f…
19. Q. Huangfu and J. A. J. Hall. 2018. Parallelizing the dual revised simplex method. Mathematical Programming Computation, 10(1):119–142. [find](https://scholar.google.com/scholar?q=Q%2E+Huangfu+and+J%2E+A%2E+J%2E+Hall%2E+2018%2E+Parallelizing+the+dual+revised+simplex+method%2E+Mathematical+Programming+Computation%2C+10%281%29%3A119%E2%80%93142%2E)
20. Ibrahim Jubran, Alaa Maalouf, and Dan Feldman. 2019. Introduction to coresets: Accurate coresets. [find](https://scholar.google.com/scholar?q=Ibrahim+Jubran%2C+Alaa+Maalouf%2C+and+Dan+Feldman%2E+2019%2E+Introduction+to+coresets%3A+Accurate+coresets%2E)
21. Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. [Metricx-24: The google submission to the WMT 2024 metrics shared task](https://md.research.elamur.ai/free/en/papers/juraska-2024-metricx-24-google-submission-wmt/).
22. Richard M. Karp, Raymond E. Miller, and James W. Thatcher. 1975. Reducibility among combinatorial problems. Journal of Symbolic Logic, 40(4). [find](https://scholar.google.com/scholar?q=Richard+M%2E+Karp%2C+Raymond+E%2E+Miller%2C+and+James+W%2E+Thatcher%2E+1975%2E+Reducibility+among+combinatorial+problems%2E+Journal+of+Symbolic+Logic%2C+40%284%29%2E)
23. M. G. Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81–93. [find](https://scholar.google.com/scholar?q=M%2E+G%2E+Kendall%2E+1938%2E+A+new+measure+of+rank+correlation%2E+Biometrika%2C+30%281%2F2%29%3A81%E2%80%9393%2E)
24. Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, Martin Popel, Maja Popov… — [Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet](https://md.research.elamur.ai/free/en/papers/kocmi-2024-findings-wmt24-general-machine-translation/)
25. Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondrej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Kenton Murray, Masaaki Nagata, Martin Popel, Maja Popovic, l ór Steing… [find](https://scholar.google.com/scholar?q=Tom+Kocmi%2C+Eleftherios+Avramidis%2C+Rachel+Bawden%2C+Ondrej+Bojar%2C+Anton+Dvorkovich%2C+Christian+Federmann%2C+Mark+Fishel%2C+Markus+Freitag%2C+Thamme+Gowda%2C+Roman+Grundkiewicz%2C+Barry+Haddow%2C+Marzena+Karpinska%2C+Ph)
26. Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popovi… [find](https://scholar.google.com/scholar?q=Tom+Kocmi%2C+Rachel+Bawden%2C+Ondr%CC%8Cej+Bojar%2C+Anton+Dvorkovich%2C+Christian+Federmann%2C+Mark+Fishel%2C+Thamme+Gowda%2C+Yvette+Graham%2C+Roman+Grundkiewicz%2C+Barry+Haddow%2C+Rebecca+Knowles%2C+Philipp+Koehn%2C+Christof+Mon)
27. Tom Kocmi and Christian Federmann. 2023. [GEMBA-MQM: Detecting translation quality error spans with GPT-4](https://md.research.elamur.ai/free/en/papers/kocmi-2023-gemba-mqm-detecting-translation-quality/).In Proceedings of the Eighth Conference on Machine Translation, pages 768–775. Association for Computational Linguistics.
28. Tom Kocmi, Vilém Zouhar, Eleftherios Avramidis, Roman Grundkiewicz, Marzena Karpinska, Maja Popović, Mrinmaya Sachan, and Mariya Shmatova. 2024c. [Error span annotation: A balanced approach for human evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/kocmi-2024-error-span-annotation-balanced-approach/). In Proceedings of the Ninth Conference on Machine Translation, p…
29. John Patrick Lalor and Pedro Rodriguez. 2023. py-irt: A scalable item response theory library for python. INFORMS Journal on Computing, 35(1):5–13. https://doi.org/10 .1287/ijoc.2022.1250 [find](https://scholar.google.com/scholar?q=John+Patrick+Lalor+and+Pedro+Rodriguez%2E+2023%2E+py%2Dirt%3A+A+scalable+item+response+theory+library+for+python%2E+INFORMS+Journal+on+Computing%2C+35%281%29%3A5%E2%80%9313%2E+https%3A%2F%2Fdoi%2Eorg%2F10+%2E1287%2Fijoc%2E2022%2E1250)
30. John Patrick Lalor, Hao Wu, and Hong Yu. 2019. Learning latent parameters without human response patterns: Item response theory with artificial crowds. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing… [find](https://scholar.google.com/scholar?q=John+Patrick+Lalor%2C+Hao+Wu%2C+and+Hong+Yu%2E+2019%2E+Learning+latent+parameters+without+human+response+patterns%3A+Item+response+theory+with+artificial+crowds%2E+In+Proceedings+of+the+Conference+on+Empirical+Me)
31. Xiang Lisa Li, Farzaan Kaiyom, Evan Zheran Liu, Yifan Mai, Percy Liang, and Tatsunori Hashimoto. 2025. AutoBencher: Towards declarative benchmark construction. [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li%2C+Farzaan+Kaiyom%2C+Evan+Zheran+Liu%2C+Yifan+Mai%2C+Percy+Liang%2C+and+Tatsunori+Hashimoto%2E+2025%2E+AutoBencher%3A+Towards+declarative+benchmark+construction%2E)
32. Yang Li, Jie Ma, Miguel Ballesteros, Yassine Benajiba, and Graham Horwood. 2024. Active evaluation acquisition for efficient LLM benchmarking. [find](https://scholar.google.com/scholar?q=Yang+Li%2C+Jie+Ma%2C+Miguel+Ballesteros%2C+Yassine+Benajiba%2C+and+Graham+Horwood%2E+2024%2E+Active+evaluation+acquisition+for+efficient+LLM+benchmarking%2E)
33. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. [G-eval: NLG evaluation using GPT-4 with better human alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522. Association for Computational Linguistics.
34. Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional quality metrics (MQM): A framework for declaring and describing translation quality metrics. Tradu-màtica, 12:0455–463. [find](https://scholar.google.com/scholar?q=Arle+Lommel%2C+Hans+Uszkoreit%2C+and+Aljoscha+Burchardt%2E+2014%2E+Multidimensional+quality+metrics+%28MQM%29%3A+A+framework+for+declaring+and+describing+translation+quality+metrics%2E+Tradu%2Dma%CC%80tica%2C+12%3A0455%E2%80%93463%2E)
35. Frederic M. Lord and Melvin R. Novick. 2008. Statistical theories of mental test scores. IAP. [find](https://scholar.google.com/scholar?q=Frederic+M%2E+Lord+and+Melvin+R%2E+Novick%2E+2008%2E+Statistical+theories+of+mental+test+scores%2E+IAP%2E)
36. Vânia Mendonça, Ricardo Rei, Luı́sa Coheur, and Alberto Sardinha. 2023. Onception: Active learning with expert advice for real world machine translation. Computational Linguistics, 49(2):325–372. https://doi.org/10.1162/coli_a_00473 [↗](https://doi.org/10.1162/coli_a_00473)
37. Vânia Mendonça, Ricardo Rei, Luisa Coheur, Alberto Sardinha, and Ana Lúcia Santos. 2021. Online Learning meets Machine Translation evaluation: Finding the best systems with the least human effort. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1… [find](https://scholar.google.com/scholar?q=Va%CC%82nia+Mendonc%CC%A7a%2C+Ricardo+Rei%2C+Luisa+Coheur%2C+Alberto+Sardinha%2C+and+Ana+Lu%CC%81cia+Santos%2E+2021%2E+Online+Learning+meets+Machine+Translation+evaluation%3A+Finding+the+best+systems+with+the+least+human+effort%2E+)
38. Toshiaki Nakazawa and Isao Goto, editors. 2024. Proceedings of the Eleventh Workshop on Asian Translation (WAT 2024). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.wat-1.0 — [doi:10.18653/v1/2024.wat-1.0](https://doi.org/10.18653/v1/2024.wat-1.0) [↗](https://doi.org/10.18653/v1/2024.wat-1.0)
39. Nils Reimers and Iryna Gurevych. 2019. [Sentence-BERT: Sentence embeddings using Siamese BERT-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992.…
40. Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. 2024. [MixEval: Deriving wisdom of the crowd from LLM benchmark mixtures](https://md.research.elamur.ai/free/en/papers/ni-2024-mixeval-deriving-wisdom-crowd-llm/).
41. Yvonnick Noel and Bruno Dauvier. 2007. A beta item response model for continuous bounded responses. Applied Psychological Measurement, 31(1):47–73. [find](https://scholar.google.com/scholar?q=Yvonnick+Noel+and+Bruno+Dauvier%2E+2007%2E+A+beta+item+response+model+for+continuous+bounded+responses%2E+Applied+Psychological+Measurement%2C+31%281%29%3A47%E2%80%9373%2E)
42. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: A method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. Association for Computational Linguistics.
43. Dongmin Park, Dimitris Papailiopoulos, and Kangwook Lee. 2022. Active learning is a strong baseline for data subset selection. In Has it Trained Yet? NeurIPS 2022 Workshop. [find](https://scholar.google.com/scholar?q=Dongmin+Park%2C+Dimitris+Papailiopoulos%2C+and+Kangwook+Lee%2E+2022%2E+Active+learning+is+a+strong+baseline+for+data+subset+selection%2E+In+Has+it+Trained+Yet%3F+NeurIPS+2022+Workshop%2E)
44. Stefano Perrella, Lorenzo Proietti, Alessandro Scirè, Edoardo Barba, and Roberto Navigli. 2024. [Guardians of the machine translation meta-evaluation: Sentinel metrics fall in!](https://md.research.elamur.ai/free/en/papers/perrella-2024-guardians-machine-translation-meta-evaluation/) In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1…
45. Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. 2024. tinyBenchmarks: Evaluating LLMs with fewer examples. [find](https://scholar.google.com/scholar?q=Felipe+Maia+Polo%2C+Lucas+Weber%2C+Leshem+Choshen%2C+Yuekai+Sun%2C+Gongjun+Xu%2C+and+Mikhail+Yurochkin%2E+2024%2E+tinyBenchmarks%3A+Evaluating+LLMs+with+fewer+examples%2E)
46. Maja Popović. 2015. [chrF: character n-gram F-score for automatic MT evaluation](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/). In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395. Association for Computational Linguistics.
47. Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. 2020. [COMET: A neural framework for MT evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/). In Proceed-Pedro Rodriguez, Joe Barrow, Alexander Miserlis Hoyle, John P. Lalor, Robin Jia, and Jordan Boyd-Graber. 2021. Evaluation examples are not equally informative: How should that ch…
48. Jie Ruan, Xiao Pu, Mingqi Gao, Xiaojun Wan, and Yuesheng Zhu. 2024. Better than random: Reliable NLG human evaluation with constrained active sampling. [find](https://scholar.google.com/scholar?q=Jie+Ruan%2C+Xiao+Pu%2C+Mingqi+Gao%2C+Xiaojun+Wan%2C+and+Yuesheng+Zhu%2E+2024%2E+Better+than+random%3A+Reliable+NLG+human+evaluation+with+constrained+active+sampling%2E)
49. Belén Saldı́as Fuentes, George Foster, Markus Freitag, and Qijun Tan. 2022. Toward more effective human evaluation for machine translation. In Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval), pages 76–89. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bele%CC%81n+Sald%C4%B1%CC%81as+Fuentes%2C+George+Foster%2C+Markus+Freitag%2C+and+Qijun+Tan%2E+2022%2E+Toward+more+effective+human+evaluation+for+machine+translation%2E+In+Proceedings+of+the+2nd+Workshop+on+Human+Evaluation+of+N)
50. Darcy A. Santor and James O. Ramsay. 1998. Progress in the technology of measurement: Applications of item response models. Psychological Assessment, 10(4):345. [find](https://scholar.google.com/scholar?q=Darcy+A%2E+Santor+and+James+O%2E+Ramsay%2E+1998%2E+Progress+in+the+technology+of+measurement%3A+Applications+of+item+response+models%2E+Psychological+Assessment%2C+10%284%29%3A345%2E)
51. C. Spearman. 1904. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72–101. [find](https://scholar.google.com/scholar?q=C%2E+Spearman%2E+1904%2E+The+proof+and+measurement+of+association+between+two+things%2E+The+American+Journal+of+Psychology%2C+15%281%29%3A72%E2%80%93101%2E)
52. Brian Thompson, Nitika Mathur, Daniel Deutsch, and Huda Khayrallah. 2024. Improving statistical significance in human evaluation of automatic metrics via soft pairwise accuracy. In Proceedings of the Ninth Conference on Machine Translation, pages 1222–1234. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Brian+Thompson%2C+Nitika+Mathur%2C+Daniel+Deutsch%2C+and+Huda+Khayrallah%2E+2024%2E+Improving+statistical+significance+in+human+evaluation+of+automatic+metrics+via+soft+pairwise+accuracy%2E+In+Proceedings+of+the+)
53. Saeed Vahidian, Baharan Mirzasoleiman, and Alexander Cloninger. 2020. Coresets for estimating means and mean square error with limited greedy samples. In Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI), volume 124 of Proceedings of Machine Learning Research, pages… [find](https://scholar.google.com/scholar?q=Saeed+Vahidian%2C+Baharan+Mirzasoleiman%2C+and+Alexander+Cloninger%2E+2020%2E+Coresets+for+estimating+means+and+mean+square+error+with+limited+greedy+samples%2E+In+Proceedings+of+the+36th+Conference+on+Uncertai)
54. Rajan Vivek, Kawin Ethayarajh, Diyi Yang, and Douwe Kiela. 2024. Anchor points: Benchmarking models with much fewer examples. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1576–1601. Association for Compu… [find](https://scholar.google.com/scholar?q=Rajan+Vivek%2C+Kawin+Ethayarajh%2C+Diyi+Yang%2C+and+Douwe+Kiela%2E+2024%2E+Anchor+points%3A+Benchmarking+models+with+much+fewer+examples%2E+In+Proceedings+of+the+18th+Conference+of+the+European+Chapter+of+the+Assoc)
55. Kai Wei, Rishabh Iyer, and Jeff Bilmes. 2015. Submodularity in data subset selection and active learning. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1954–1963. Lille, France. PMLR. [find](https://scholar.google.com/scholar?q=Kai+Wei%2C+Rishabh+Iyer%2C+and+Jeff+Bilmes%2E+2015%2E+Submodularity+in+data+subset+selection+and+active+learning%2E+In+Proceedings+of+the+32nd+International+Conference+on+Machine+Learning%2C+volume+37+of+Proceedi)
56. Mike Wu, Richard L. Davis, Benjamin W. Domingue, Chris Piech, and Noah Goodman. 2020. Variational item response theory: Fast, accurate, and expressive. [find](https://scholar.google.com/scholar?q=Mike+Wu%2C+Richard+L%2E+Davis%2C+Benjamin+W%2E+Domingue%2C+Chris+Piech%2C+and+Noah+Goodman%2E+2020%2E+Variational+item+response+theory%3A+Fast%2C+accurate%2C+and+expressive%2E)
57. Runzhe Zhan, Xuebo Liu, Derek F. Wong, and Lidia S. Chao. 2021. Variance-aware machine translation test sets. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). [find](https://scholar.google.com/scholar?q=Runzhe+Zhan%2C+Xuebo+Liu%2C+Derek+F%2E+Wong%2C+and+Lidia+S%2E+Chao%2E+2021%2E+Variance%2Daware+machine+translation+test+sets%2E+In+Thirty%2Dfifth+Conference+on+Neural+Information+Processing+Systems+Datasets+and+Benchmark)
58. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+BERTScore%3A+Evaluating+text+generation+with+BERT%2E)
59. Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, and Alexandra Olteanu. 2022. Deconstructing NLG evaluation: Evaluation practices, assumptions, and their implications. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computation… [find](https://scholar.google.com/scholar?q=Kaitlyn+Zhou%2C+Su+Lin+Blodgett%2C+Adam+Trischler%2C+Hal+Daume%CC%81+III%2C+Kaheer+Suleman%2C+and+Alexandra+Olteanu%2E+2022%2E+Deconstructing+NLG+evaluation%3A+Evaluation+practices%2C+assumptions%2C+and+their+implications%2E+In)
60. Vilém Zouhar, Pinzhen Chen, Tsz Kin Lam, Nikita Moghe, and Barry Haddow. 2024a. Pitfalls and outlooks in using COMET. In Proceedings of the Ninth Conference on Machine Translation, pages 1272–1288. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Vile%CC%81m+Zouhar%2C+Pinzhen+Chen%2C+Tsz+Kin+Lam%2C+Nikita+Moghe%2C+and+Barry+Haddow%2E+2024a%2E+Pitfalls+and+outlooks+in+using+COMET%2E+In+Proceedings+of+the+Ninth+Conference+on+Machine+Translation%2C+pages+1272%E2%80%931288%2E+A)
61. Vilém Zouhar, Shehzaad Dhuliawala, Wangchunshu Zhou, Nico Daheim, Tom Kocmi, Yuchen Eleanor Jiang, and Mrinmaya Sachan. 2023. Poor man‘s quality estimation: Predicting reference-based MT metrics without the reference. In Proceedings of the 17th Conference of the European Chapter of the Association… [find](https://scholar.google.com/scholar?q=Vile%CC%81m+Zouhar%2C+Shehzaad+Dhuliawala%2C+Wangchunshu+Zhou%2C+Nico+Daheim%2C+Tom+Kocmi%2C+Yuchen+Eleanor+Jiang%2C+and+Mrinmaya+Sachan%2E+2023%2E+Poor+man%E2%80%98s+quality+estimation%3A+Predicting+reference%2Dbased+MT+metrics+with)
62. Vilém Zouhar, Shuoyang Ding, Anna Currey, Tatyana Badeka, Jenyuan Wang, and Brian Thompson. 2024b. [Fine-tuned machine translation metrics struggle in unseen domains](https://md.research.elamur.ai/free/en/papers/zouhar-2024-fine-tuned-machine-translation-metrics/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 488–500. A…
63. Vilém Zouhar, Tom Kocmi, and Mrinmaya Sachan. 2025a. AI-assisted human evaluation of machine translation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4936–49… [find](https://scholar.google.com/scholar?q=Vile%CC%81m+Zouhar%2C+Tom+Kocmi%2C+and+Mrinmaya+Sachan%2E+2025a%2E+AI%2Dassisted+human+evaluation+of+machine+translation%2E+In+Proceedings+of+the+2025+Conference+of+the+Nations+of+the+Americas+Chapter+of+the+Associati)
64. Vilém Zouhar, Maike Züfle, Beni Egressy, Julius Cheng, and Jan Niehues. 2025b. [Early-exit and instant confidence translation quality estimation](https://md.research.elamur.ai/free/en/papers/zouhar-2026-early-exit-instant-confidence-translation/).

*Updated 31 October 2026*