---
title: "Accurate Evaluation of Segment-level Machine — Graham 2015"
description: "Judging how well segment-level MT metrics work has been unreliable."
authors: ["Yvette Graham", "Timothy Baldwin", "Nitika Mathur"]
year: 2015
venue: "Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/N15-1124/"
doi: "10.3115/v1/n15-1124"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/graham-2015-accurate-evaluation-segment-level-machine/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/graham-2015-accurate-evaluation-segment-level-machine/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/) / Accurate Evaluation of Segment-level Machine Translation Metrics

# Accurate Evaluation of Segment-level Machine Translation Metrics

**Authors:** [Yvette Graham](https://md.research.elamur.ai/free/en/authors/yvette-graham/), [Timothy Baldwin](https://md.research.elamur.ai/free/en/authors/timothy-baldwin/), [Nitika Mathur](https://md.research.elamur.ai/free/en/authors/nitika-mathur/) · **Year:** 2015 · **Venue:** Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/N15-1124/) · [PDF](https://aclanthology.org/N15-1124.pdf) · [DOI 10.3115/v1/n15-1124](https://doi.org/10.3115/v1/n15-1124) · [OpenAlex](https://openalex.org/W2294699749)

**Citations:** 92 citations (openalex-snapshot · 2026-10-14) · [63](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Judging how well segment-level MT metrics work has been unreliable. Human judges in WMT agree poorly, pairs judged as ties are dropped, and there was no way to test whether one metric truly beats another. The authors propose a new human evaluation setup built for this purpose. Instead of asking for relative rankings of five outputs, crowd workers give absolute adequacy scores. Many repeat assessments are collected for each translation and averaged, relying on the law of large numbers. Scores are standardized as z-scores, and the number of repeats needed is chosen empirically. Metrics are then compared with Pearson's correlation, and the Williams test checks whether one metric's correlation is significantly higher than another's.

They replicated parts of the WMT-13 human evaluation using this method. Every metric correlated better with humans than the original evaluation reported. METEOR, NLEPOR and SENTBLEU-MOSES were not significantly beaten by any rival, both for Spanish-to-English and for nine language pairs combined. This gives MT researchers a more trustworthy gold standard and a principled way to choose a default metric. The code is released publicly.

## Key points
- In WMT-13, inter-annotator Kappa ranged from 0.075 to 0.324 for crowd workers and from 0.315 to 0.457 for MT researchers.
- The WMT "Kendall's τ" used for segment-level metrics has two problems:
  - It is effectively a weighted average over many separately ranked 5-translation sets.
  - Contradictory human judgments cap the achievable value at roughly ±0.87 to ±0.92.
- About 24% of WMT-13 pairwise judgments were discarded as ties. As a result, metrics got no credit for scoring equal-quality translations similarly.
- The cross-lingual test set has 630 distinct translations: 70 per pair across nine WMT-13 language pairs, with at least 15 crowd assessments each.
  - English-to-Czech was left out because too few Czech-speaking MTurk workers were available.
- On the combined set, METEOR reached the highest Pearson r (0.441) and LEPOR the lowest (0.326). SIMPBLEUP and SIMPBLEUR were also unbeaten overall, but they were outperformed for Spanish-to-English.
- Code for crowd-sourced collection and significance testing is at github.com/ygraham/segment-mteval.

## Interesting ideas
- **Treat disagreement as a signal to average, not noise to remove.** Collect many independent scores per item and use their mean as an estimate of the population's view. This fits any subjective annotation task where judges legitimately differ, such as fluency, toxicity or summary quality.
- **Choose the number of repeats empirically.** Collect two independent rounds of scores and watch how their agreement grows as more assessments are added. This avoids picking an arbitrary standard error and saves annotation budget for extra items.
- **Compare metrics with a dependent-correlation test.** The Williams test accounts for the fact that competing metrics are scored on the same data and correlate with each other. It is useful wherever two automatic scores are compared against one human reference.
- **Declare as "best" every metric that nothing significantly beats.** Counting how many rivals a metric defeats is misleading, because the test's power depends on how strongly metrics correlate with each other. This gives more honest leaderboards.
- **Make scores pool across subsets.** Absolute, z-standardized human scores mean roughly the same thing across language pairs, so data can be combined. This helps when choosing one default tool for several languages, including low-resource ones like Ukrainian.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

63 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 49 not yet.

- [HUME: Human UCCA-Based Evaluation of Machine Translation](https://md.research.elamur.ai/free/en/papers/birch-2016-hume-human-ucca-based-evaluation/)
- [Findings of the 2017 Conference on Machine Translation (WMT17)](https://md.research.elamur.ai/free/en/papers/bojar-2017-findings-2017-conference-machine-translation/)
- [Results of the WMT17 Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/bojar-2017-results-wmt17-metrics-shared-task/)
- [Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments](https://md.research.elamur.ai/free/en/papers/fomicheva-2019-taking-mt-evaluation-metrics-extremes/)
- [Re-evaluating Automatic Summarization with BLEU and 192 Shades of ROUGE](https://md.research.elamur.ai/free/en/papers/graham-2015-re-evaluating-automatic-summarization-bleu/)
- [Fully Character-Level Neural Machine Translation without Explicit Segmentation](https://md.research.elamur.ai/free/en/papers/lee-2017-fully-character-level-neural-machine/)
- [Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performance](https://md.research.elamur.ai/free/en/papers/ma-2018-results-wmt18-metrics-shared-task/)
- [Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges](https://md.research.elamur.ai/free/en/papers/ma-2019-results-wmt19-metrics-shared-task/)
- [Results of the WMT20 Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/mathur-2020-results-wmt20-metrics-shared-task/)
- [KinyCOMET: Automatic Evaluation of Machine Translation Systems for Kinyarwanda–English](https://md.research.elamur.ai/free/en/papers/mazimpaka-2026-kinycomet-automatic-evaluation-machine-translation/)
- [Machine translation Evaluation Eng-Thai MQM Ranking dataset](https://md.research.elamur.ai/free/en/papers/phuangrot-2026-machine-translation-evaluation-eng-thai/)
- [TransQuest: Translation Quality Estimation with Cross-lingual Transformers](https://md.research.elamur.ai/free/en/papers/ranasinghe-2020-transquest-translation-quality-estimation-cross/)
- [Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains](https://md.research.elamur.ai/free/en/papers/schmidt-2026-who-watches-watchmen-humans-disagree/)
- [Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/toral-2018-attaining-unattainable-reassessing-claims-human/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{graham2015accurate,
  title = {{Accurate Evaluation of Segment-level Machine Translation Metrics}},
  author = {Graham, Yvette and Baldwin, Timothy and Mathur, Nitika},
  booktitle = {Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2015},
  doi = {10.3115/v1/n15-1124},
  url = {https://doi.org/10.3115/v1/n15-1124}
}
```

- [← previous A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](https://md.research.elamur.ai/free/en/papers/gal-2015-theoretically-grounded-application-dropout-recurrent/)
- [next → Addressing the Rare Word Problem in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/luong-2015-addressing-rare-word-problem-neural/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/)

## References

17 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. W. Aziz, S. Castilho, and L. Specia. 2012. PET: a tool for post-editing and assessing machine translation. In Proc. of the 8th International Conference on Language Resources and Evaluation (LREC 2012), pages 3982– 3987, Istanbul, Turkey. [find](https://scholar.google.com/scholar?q=W%2E+Aziz%2C+S%2E+Castilho%2C+and+L%2E+Specia%2E+2012%2E+PET%3A+a+tool+for+post%2Dediting+and+assessing+machine+translation%2E+In+Proc%2E+of+the+8th+International+Conference+on+Language+Resources+and+Evaluation+%28LREC+2012%29) [our card](https://md.research.elamur.ai/free/en/papers/aziz-2012-pet-tool-post-editing-assessing/)
2. O. Bojar, C. Buck, C. Callison-Burch, C. Federmann, B. Haddow, P. Koehn, C. Monz, M. Post, R. Soricut, and L. Specia. 2013a. Findings of the 2013 Workshop on Statistical Machine Translation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 1–44, Sofia, Bulgaria. [find](https://scholar.google.com/scholar?q=O%2E+Bojar%2C+C%2E+Buck%2C+C%2E+Callison%2DBurch%2C+C%2E+Federmann%2C+B%2E+Haddow%2C+P%2E+Koehn%2C+C%2E+Monz%2C+M%2E+Post%2C+R%2E+Soricut%2C+and+L%2E+Specia%2E+2013a%2E+Findings+of+the+2013+Workshop+on+Statistical+Machine+Translation%2E+In+Procee) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2013-findings-2013-workshop-statistical-machine/)
3. O. Bojar, C. Buck, C. Callison-Burch, C. Federmann, B. Haddow, P. Koehn, C. Monz, M. Post, R. Soricut, and L. Specia. 2013b. Findings of the 2013 Workshop on Statistical Machine Translation. In Proc. 8th Wkshp. Statistical Machine Translation, pages 1–44, Sofia, Bulgaria. [find](https://scholar.google.com/scholar?q=O%2E+Bojar%2C+C%2E+Buck%2C+C%2E+Callison%2DBurch%2C+C%2E+Federmann%2C+B%2E+Haddow%2C+P%2E+Koehn%2C+C%2E+Monz%2C+M%2E+Post%2C+R%2E+Soricut%2C+and+L%2E+Specia%2E+2013b%2E+Findings+of+the+2013+Workshop+on+Statistical+Machine+Translation%2E+In+Proc%2E+) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2013-findings-2013-workshop-statistical-machine/)
4. O. Bojar, C. Buck, C. Federmann, B. Haddow, P. Koehn, J. Leveling, C. Monz, P. Pecina, M. Post, H. Saint-Amand, R. Soricut, L. Specia, and A. Tamchyna. 2014. Findings of the 2014 Workshop on Statistical Machine Translation. In Proc. 9th Wkshp. Statistical Machine Translation, pages 12–58, Baltimore, USA. [find](https://scholar.google.com/scholar?q=O%2E+Bojar%2C+C%2E+Buck%2C+C%2E+Federmann%2C+B%2E+Haddow%2C+P%2E+Koehn%2C+J%2E+Leveling%2C+C%2E+Monz%2C+P%2E+Pecina%2C+M%2E+Post%2C+H%2E+Saint%2DAmand%2C+R%2E+Soricut%2C+L%2E+Specia%2C+and+A%2E+Tamchyna%2E+2014%2E+Findings+of+the+2014+Workshop+on+Statistic) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2014-findings-2014-workshop-statistical-machine/)
5. C. Callison-Burch, C. Fordyce, P. Koehn, C. Monz, and J. Schroeder. 2007. (Meta-) evaluation of machine translation. In Proc. 2nd Wkshp. Statistical Machine Translation, pages 136–158, Prague, Czech Republic. [find](https://scholar.google.com/scholar?q=C%2E+Callison%2DBurch%2C+C%2E+Fordyce%2C+P%2E+Koehn%2C+C%2E+Monz%2C+and+J%2E+Schroeder%2E+2007%2E+%28Meta%2D%29+evaluation+of+machine+translation%2E+In+Proc%2E+2nd+Wkshp%2E+Statistical+Machine+Translation%2C+pages+136%E2%80%93158%2C+Prague%2C+Czech+R) [our card](https://md.research.elamur.ai/free/en/papers/burch-2007-meta-evaluation-machine-translation/)
6. C. Callison-Burch, C. Fordyce, P. Koehn, C. Monz, and J. Schroeder. 2008. Further meta-evaluation of machine translation. In Proc. 3rd Wkshp. Statistical Machine Translation, pages 70–106, Columbus, USA. [find](https://scholar.google.com/scholar?q=C%2E+Callison%2DBurch%2C+C%2E+Fordyce%2C+P%2E+Koehn%2C+C%2E+Monz%2C+and+J%2E+Schroeder%2E+2008%2E+Further+meta%2Devaluation+of+machine+translation%2E+In+Proc%2E+3rd+Wkshp%2E+Statistical+Machine+Translation%2C+pages+70%E2%80%93106%2C+Columbus%2C+U) [our card](https://md.research.elamur.ai/free/en/papers/burch-2007-meta-evaluation-machine-translation/)
7. C. Callison-Burch, P. Koehn, C. Monz, and J. Schroeder. 2009. Findings of the 2009 Workshop on Statistical Machine Translation. In Proc. 4th Wkshp. Statistical Machine Translation, pages 1–28, Athens, Greece. [find](https://scholar.google.com/scholar?q=C%2E+Callison%2DBurch%2C+P%2E+Koehn%2C+C%2E+Monz%2C+and+J%2E+Schroeder%2E+2009%2E+Findings+of+the+2009+Workshop+on+Statistical+Machine+Translation%2E+In+Proc%2E+4th+Wkshp%2E+Statistical+Machine+Translation%2C+pages+1%E2%80%9328%2C+Athens%2C) [our card](https://md.research.elamur.ai/free/en/papers/burch-2009-findings-2009-workshop-statistical-machine/)
8. M. Denkowski and A. Lavie. 2011. Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. In Proc. 6th Wkshp. Statistical Machine Translation, pages 85–91, Edinburgh, Scotland. [find](https://scholar.google.com/scholar?q=M%2E+Denkowski+and+A%2E+Lavie%2E+2011%2E+Meteor+1%2E3%3A+Automatic+metric+for+reliable+optimization+and+evaluation+of+machine+translation+systems%2E+In+Proc%2E+6th+Wkshp%2E+Statistical+Machine+Translation%2C+pages+85%E2%80%9391%2C) [our card](https://md.research.elamur.ai/free/en/papers/denkowski-2011-meteor-1-3-automatic-metric/)
9. Y. Graham and T. Baldwin. 2014. Testing for significance of increased correlation with human judgment. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 172–176, Doha, Qatar. [find](https://scholar.google.com/scholar?q=Y%2E+Graham+and+T%2E+Baldwin%2E+2014%2E+Testing+for+significance+of+increased+correlation+with+human+judgment%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+172%E2%80%931)
10. Y. Graham, T. Baldwin, A. Moffat, and J. Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proc. 7th Linguistic Annotation Wkshp. & Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. [find](https://scholar.google.com/scholar?q=Y%2E+Graham%2C+T%2E+Baldwin%2C+A%2E+Moffat%2C+and+J%2E+Zobel%2E+2013%2E+Continuous+measurement+scales+in+human+evaluation+of+machine+translation%2E+In+Proc%2E+7th+Linguistic+Annotation+Wkshp%2E+%26+Interoperability+with+Discou)
11. Y. Graham, T. Baldwin, A. Moffat, and J. Zobel. 2014a. Is machine translation getting better over time? In Proceedings of the European Chapter of the Association of Computational Linguistics, pages 443–451, Gothenburg, Sweden. [find](https://scholar.google.com/scholar?q=Y%2E+Graham%2C+T%2E+Baldwin%2C+A%2E+Moffat%2C+and+J%2E+Zobel%2E+2014a%2E+Is+machine+translation+getting+better+over+time%3F+In+Proceedings+of+the+European+Chapter+of+the+Association+of+Computational+Linguistics%2C+pages+44)
12. Y. Graham, N. Mathur, and T. Baldwin. 2014b. Randomized significance tests in machine translation. In Proc. Ninth ACL Wkshp. Statistical Machine Translation, pages 266–74, Baltimore, MD. [find](https://scholar.google.com/scholar?q=Y%2E+Graham%2C+N%2E+Mathur%2C+and+T%2E+Baldwin%2E+2014b%2E+Randomized+significance+tests+in+machine+translation%2E+In+Proc%2E+Ninth+ACL+Wkshp%2E+Statistical+Machine+Translation%2C+pages+266%E2%80%9374%2C+Baltimore%2C+MD%2E)
13. A.L. Han, D.F. Wong, L.S. Chao, Y. Lu, L. He, Y. Wang, and J. Zhou. 2013. A description of tunable machine translation evaluation systems in WMT13 metrics task. In Proceedings of the Eight Workshop on Statistical Machine Translation, pages 414–421, Sofia, Bulgaria. [find](https://scholar.google.com/scholar?q=A%2EL%2E+Han%2C+D%2EF%2E+Wong%2C+L%2ES%2E+Chao%2C+Y%2E+Lu%2C+L%2E+He%2C+Y%2E+Wang%2C+and+J%2E+Zhou%2E+2013%2E+A+description+of+tunable+machine+translation+evaluation+systems+in+WMT13+metrics+task%2E+In+Proceedings+of+the+Eight+Workshop+on)
14. P. Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proc. of Empirical Methods in Natural Language Processing, pages 388–395, Barcelona, Spain. [find](https://scholar.google.com/scholar?q=P%2E+Koehn%2E+2004%2E+Statistical+significance+tests+for+machine+translation+evaluation%2E+In+Proc%2E+of+Empirical+Methods+in+Natural+Language+Processing%2C+pages+388%E2%80%93395%2C+Barcelona%2C+Spain%2E)
15. M. Machacek and O. Bojar. 2014. Results of the wmt14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 293– 301, Baltimore, USA. [find](https://scholar.google.com/scholar?q=M%2E+Machacek+and+O%2E+Bojar%2E+2014%2E+Results+of+the+wmt14+metrics+shared+task%2E+In+Proceedings+of+the+Ninth+Workshop+on+Statistical+Machine+Translation%2C+pages+293%E2%80%93+301%2C+Baltimore%2C+USA%2E) [our card](https://md.research.elamur.ai/free/en/papers/machacek-2014-results-wmt14-metrics-shared-task/)
16. J.H. Steiger. 1980. Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87(2):245. [↗](https://doi.org/10.1037//0033-2909.87.2.245)
17. E.J. Williams. 1959. Regression analysis, volume 14. Wiley, New York, USA. [find](https://scholar.google.com/scholar?q=E%2EJ%2E+Williams%2E+1959%2E+Regression+analysis%2C+volume+14%2E+Wiley%2C+New+York%2C+USA%2E)