---
title: "Testing for Significance of Increased Correlation with Human"
description: "They therefore propose the Williams test (also called Hotelling–Williams), which accounts for how… 17 references, cited by 55 papers here."
authors: ["Yvette Graham", "Timothy Baldwin"]
year: 2014
venue: "Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/D14-1020/"
doi: "10.3115/v1/d14-1020"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/graham-2014-testing-significance-increased-correlation-human/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/graham-2014-testing-significance-increased-correlation-human/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/) / Testing for Significance of Increased Correlation with Human Judgment

# Testing for Significance of Increased Correlation with Human Judgment

**Authors:** [Yvette Graham](https://md.research.elamur.ai/free/en/authors/yvette-graham/) and [Timothy Baldwin](https://md.research.elamur.ai/free/en/authors/timothy-baldwin/) · **Year:** 2014 · **Venue:** [Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2014](https://md.research.elamur.ai/free/en/venues/emnlp/2014/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/D14-1020/) · [PDF](https://aclanthology.org/D14-1020.pdf) · [DOI 10.3115/v1/d14-1020](https://doi.org/10.3115/v1/d14-1020) · [OpenAlex](https://openalex.org/W2250234233)

**Citations:** 65 citations (OpenAlex, 2026-10-14) · [55](https://md.research.elamur.ai/free/en/papers/graham-2014-testing-significance-increased-correlation-human/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
New automatic metrics for machine translation are usually validated by showing a higher correlation with human judgment than BLEU. However, such gains are rarely checked for statistical significance, so some reported improvements may be due to chance. Graham and Baldwin argue that testing each metric's correlation against zero, or using the Fisher r-to-z transformation, is inappropriate here. Both metrics are scored on the same systems, so their correlations are dependent. They therefore propose the Williams test (also called Hotelling–Williams), which accounts for how strongly the two metrics correlate with each other, and they favour Pearson's r over Spearman's ρ as the more sensitive measure. Applied to system-level data from the WMT-12 metrics shared task, the test finds no outright best metric for any language pair. Most metrics also lack sufficient evidence of beating BLEU. The authors release an open-source implementation and recommend that significance testing become standard practice when proposing new MT evaluation metrics.

## Key points
- On WMT-12 Spanish-to-English, AMBER and TERRORCAT have the same Spearman ρ (0.965) but very different Pearson r (0.881 vs. 0.971). Much of this gap comes from the single lowest-scoring system.
- For Spanish-to-English, about 60% of metric pairs differ significantly (one-sided test, p \< 0.05). The top metric, TERRORCAT, is not significantly better than METEOR, SAGAN, SEMPOS or POSF.
- The analysis covers seven WMT-12 language pairs; English-to-Czech is excluded because of missing scores. The share of significant pairwise differences ranges from 3% (Czech-to-English) to 82% (English-to-French).
- Only 34% of metrics significantly outperform BLEU, and no metric beats BLEU across all language pairs.
- The authors caution against ranking metrics by their count of significant wins, because the test's power depends on inter-metric correlation. The code is available at github.com/ygraham/significance-williams.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Yvette Graham and Timothy Baldwin. 2014. Testing for Significance of Increased Correlation with Human Judgment. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{graham2014testing,
  title = {{Testing for Significance of Increased Correlation with Human Judgment}},
  author = {Yvette Graham and Timothy Baldwin},
  booktitle = {Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2014},
  doi = {10.3115/v1/d14-1020},
  url = {https://doi.org/10.3115/v1/d14-1020}
}
```

- [← previous Lexical Chaining for Measuring Discourse Coherence Quality in Test-taker Essays](https://md.research.elamur.ai/free/en/papers/somasundaran-2014-lexical-chaining-measuring-discourse-coherence/)
- [next → Hybrid text simplification using synchronous dependency grammars with hand-written and automatically harvested rules](https://md.research.elamur.ai/free/en/papers/siddharthan-2014-hybrid-text-simplification-synchronous-dependency/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/)

## References

17 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Julio Castillo and Paula Estrella. 2012. Semantic textual similarity for MT evaluation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 52–58, Montréal, Canada. [find](https://scholar.google.com/scholar?q=Julio+Castillo+and+Paula+Estrella%2E+2012%2E+Semantic+textual+similarity+for+MT+evaluation%2E+In+Proceedings+of+the+Seventh+Workshop+on+Statistical+Machine+Translation%2C+pages+52%E2%80%9358%2C+Montre%CC%81al%2C+Canada%2E)
2. Boxing Chen, Roland Kuhn, and George Foster. 2012. Improving AMBER, an MT evaluation metric. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 59–63, Montréal, Canada. [find](https://scholar.google.com/scholar?q=Boxing+Chen%2C+Roland+Kuhn%2C+and+George+Foster%2E+2012%2E+Improving+AMBER%2C+an+MT+evaluation+metric%2E+In+Proceedings+of+the+Seventh+Workshop+on+Statistical+Machine+Translation%2C+pages+59%E2%80%9363%2C+Montre%CC%81al%2C+Canada%2E)
3. Michael Denkowski and Alon Lavie. 2011. [Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems](https://md.research.elamur.ai/free/en/papers/denkowski-2011-meteor-1-3-automatic-metric/). In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 85–91, Edinburgh, UK.
4. Mark Fishel, Rico Sennrich, Maja Popović, and Ondřej Bojar. 2012. TerrorCat: a translation error categorization-based MT quality metric. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 64–70, Montréal, Canada. [find](https://scholar.google.com/scholar?q=Mark+Fishel%2C+Rico+Sennrich%2C+Maja+Popovic%CC%81%2C+and+Ondr%CC%8Cej+Bojar%2E+2012%2E+TerrorCat%3A+a+translation+error+categorization%2Dbased+MT+quality+metric%2E+In+Proceedings+of+the+Seventh+Workshop+on+Statistical+Machine)
5. Ulrich Germann. 2003. Greedy decoding for statistical machine translation in almost linear time. In Proceedings of the 2003 Conference of the North American Chapter of the Assoc. Computational Linguistics on Human Language Technology-Volume 1, pages 1–8, Edmonton, Canada. [find](https://scholar.google.com/scholar?q=Ulrich+Germann%2E+2003%2E+Greedy+decoding+for+statistical+machine+translation+in+almost+linear+time%2E+In+Proceedings+of+the+2003+Conference+of+the+North+American+Chapter+of+the+Assoc%2E+Computational+Linguis)
6. Yvette Graham, Nitika Mathur, and Timothy Baldwin. 2014. Randomized significance tests in machine translation. In Proceedings of the ACL 2014 Ninth Workshop on Statistical Machine Translation, pages 266–274, Baltimore, USA. [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Nitika+Mathur%2C+and+Timothy+Baldwin%2E+2014%2E+Randomized+significance+tests+in+machine+translation%2E+In+Proceedings+of+the+ACL+2014+Ninth+Workshop+on+Statistical+Machine+Translation%2C+pages+2)
7. Philipp Koehn. 2004. [Statistical significance tests for machine translation evaluation](https://md.research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/). In Proceedings of Empirical Methods in Natural Language Processing 2004 (EMNLP 2004), pages 388–395, Barcelona, Spain.
8. Shankar Kumar and William Byrne. 2004. [Minimum Bayes-risk decoding for statistical machine translation](https://md.research.elamur.ai/free/en/papers/kumar-2004-minimum-bayes-risk-decoding-statistical/). In Proceedings of the 4th International Conference on Human Language Technology Research and 5th Annual Meeting of the NAACL (HLT-NAACL 2004), pages 169–176, Boston, USA.
9. Gregor Leusch, Nicola Ueffing, and Hermann Ney. 2003. A novel string-to-string distance measure with applications to machine translation evaluation. In Proceedings 9th Machine Translation Summit (MT Summit IX), pages 240–247, New Orleans, USA. [find](https://scholar.google.com/scholar?q=Gregor+Leusch%2C+Nicola+Ueffing%2C+and+Hermann+Ney%2E+2003%2E+A+novel+string%2Dto%2Dstring+distance+measure+with+applications+to+machine+translation+evaluation%2E+In+Proceedings+9th+Machine+Translation+Summit+%28MT+S)
10. Matouš Macháček and Ondřej Bojar. 2011. Approximating a deep-syntactic metric for MT evaluation and tuning. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 92–98, Edinburgh, UK. [find](https://scholar.google.com/scholar?q=Matous%CC%8C+Macha%CC%81c%CC%8Cek+and+Ondr%CC%8Cej+Bojar%2E+2011%2E+Approximating+a+deep%2Dsyntactic+metric+for+MT+evaluation+and+tuning%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2C+pages+92%E2%80%9398%2C+Ed)
11. Dan Melamed, Ryan Green, and Joseph Turian. 2003. [Precision and recall of machine translation](https://md.research.elamur.ai/free/en/papers/melamed-2003-precision-recall-machine-translation/). In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology (HLT-NAACL 2003) — Short Papers, pages 61–63, Edmonton, Can…
12. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 160–167, Sapporo, Japan. [find](https://scholar.google.com/scholar?q=Franz+Josef+Och%2E+2003%2E+Minimum+error+rate+training+in+statistical+machine+translation%2E+In+Proceedings+of+the+41st+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+160%E2%80%93167%2C+Sappor)
13. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. [BLEU: A method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). Technical Report RC22176 (W0109-022), IBM Research, Thomas J. Watson Research Center.
14. Maja Popovic. 2012. Class error rates for evaluation of machine translation output. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 71–75, Montréal, Canada. [find](https://scholar.google.com/scholar?q=Maja+Popovic%2E+2012%2E+Class+error+rates+for+evaluation+of+machine+translation+output%2E+In+Proceedings+of+the+Seventh+Workshop+on+Statistical+Machine+Translation%2C+pages+71%E2%80%9375%2C+Montre%CC%81al%2C+Canada%2E)
15. Stefan Riezler and John T. Maxwell. 2005. [On some pitfalls in automatic evaluation and significance testing for mt](https://md.research.elamur.ai/free/en/papers/riezler-2005-some-pitfalls-automatic-evaluation-significance/). In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 57–64, Ann Arbor, USA.
16. James H. Steiger. 1980. Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87(2):245. [find](https://scholar.google.com/scholar?q=James+H%2E+Steiger%2E+1980%2E+Tests+for+comparing+elements+of+a+correlation+matrix%2E+Psychological+Bulletin%2C+87%282%29%3A245%2E)
17. Evan J. Williams. 1959. Regression Analysis, volume 14. Wiley, New York, USA. [find](https://scholar.google.com/scholar?q=Evan+J%2E+Williams%2E+1959%2E+Regression+Analysis%2C+volume+14%2E+Wiley%2C+New+York%2C+USA%2E)

*Updated 24 October 2026*