---
title: "A Statistical Analysis of Summarization Evaluation Metrics Using"
description: "Automatic summarization metrics are usually judged by how well their scores correlate with human… 43 references, cited by 69 papers here."
authors: ["Daniel Deutsch", "Rotem Dror", "Dan Roth"]
year: 2021
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2021.tacl-1.67/"
doi: "10.1162/tacl_a_00417"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/deutsch-2021-statistical-analysis-summarization-evaluation-metrics/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/deutsch-2021-statistical-analysis-summarization-evaluation-metrics/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods

# A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods

**Authors:** [Daniel Deutsch](https://md.research.elamur.ai/free/en/authors/daniel-deutsch/), Rotem Dror and [Dan Roth](https://md.research.elamur.ai/free/en/authors/dan-roth/) · **Year:** 2021 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2021](https://md.research.elamur.ai/free/en/venues/tacl/2021/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.tacl-1.67/) · [PDF](https://aclanthology.org/2021.tacl-1.67.pdf) · [DOI 10.1162/tacl\_a\_00417](https://doi.org/10.1162/tacl_a_00417) · [OpenAlex](https://openalex.org/W3211758383)

**Citations:** 46 citations (OpenAlex, 2026-10-14) · [69](https://md.research.elamur.ai/free/en/papers/deutsch-2021-statistical-analysis-summarization-evaluation-metrics/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary

Automatic summarization metrics are usually judged by how well their scores correlate with human quality ratings. However, the field has no standard way to measure how reliable those correlations are, or to tell whether one metric truly beats another. Deutsch, Dror, and Roth address this with resampling statistics. Correlations are computed over a systems × inputs score matrix, so they adapt bootstrapping and permutation testing to resample matrices instead of vectors. Each variant treats systems, inputs, or both as random.

Two simulation experiments compare the variants. Confidence intervals generalize best to held-out data when both systems and inputs are resampled. The matching permutation test is more powerful than its bootstrap counterpart and than Williams' test, the parametric test common in machine translation.

The authors then apply these tools to many metrics on three sets of English human judgments. They find that the confidence intervals are wide, so our knowledge of metric quality is quite uncertain. Few metrics reliably outperform ROUGE. QAEval and BERTScore are notable exceptions in some settings. The approach also carries over to evaluating metrics for other generation tasks.

## Key points

- Three matrix-resampling schemes are proposed: BOOT-SYSTEMS, BOOT-INPUTS, and BOOT-BOTH, along with analogous permutation tests. They are applied to both system-level and summary-level correlations (Pearson, Spearman, Kendall).
- The parametric options, the Fisher transformation and Williams' test, assume normally distributed data. That assumption fails for summarization scores, which motivates the nonparametric approach.
- The human judgments are drawn from three sources: Bhandari et al. (2020), Fabbri et al. (2021), and the TAC 2008 data described by Dang and Owczarzak (2008). The datasets cover single- and multi-document settings, including CNN/DM.
- Recommended setup: BOOT-BOTH for confidence intervals and the corresponding both-variable permutation test for comparing metrics. Williams' test is not advised for summarization metrics because of its low power.
- Published in TACL vol. 9 (2021), pp. 1132–1146. Code is available at github.com/CogComp/stat-analysis-experiments.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Pennsylvania](https://md.research.elamur.ai/free/en/universities/university-of-pennsylvania/) — University, Philadelphia, United States · 170 works · 60,739 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works · [📝 Summarization](https://md.research.elamur.ai/free/en/topics/summarization/) · 159 works

## Cite

Cite as: Daniel Deutsch, Rotem Dror and Dan Roth. 2021. A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{deutsch2021statistical,
  title = {{A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods}},
  author = {Daniel Deutsch, Rotem Dror and Dan Roth},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00417},
  url = {https://doi.org/10.1162/tacl_a_00417}
}
```

- [← previous Intrinsic Bias Metrics Do Not Correlate with Application Bias](https://md.research.elamur.ai/free/en/papers/tarrant-2021-intrinsic-bias-metrics-do-not/)
- [next → End-to-End Training of Neural Retrievers for Open-Domain Question Answering](https://md.research.elamur.ai/free/en/papers/sachan-2021-end-end-training-neural-retrievers/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

43 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Erfan Sadeqi Azer, Daniel Khashabi, Ashish Sabharwal, and Dan Roth. 2020. Not all claims are created equal: Choosing the right statistical approach to assess hypotheses. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5715–5725, Online. Association f… [find](https://scholar.google.com/scholar?q=Erfan+Sadeqi+Azer%2C+Daniel+Khashabi%2C+Ashish+Sabharwal%2C+and+Dan+Roth%2E+2020%2E+Not+all+claims+are+created+equal%3A+Choosing+the+right+statistical+approach+to+assess+hypotheses%2E+In+Proceedings+of+the+58th+Ann)
2. Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. [Re-evaluating evaluation in text summarization](https://md.research.elamur.ai/free/en/papers/bhandari-2020-re-evaluating-evaluation-text-summarization/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347–9359, Online. Association for Computational Linguist…
3. D. Bonett and T. A. Wright. 2000. Sample size requirements for estimating Pearson, Kendall and Spearman correlations. Psychometrika, 65:23–28. https://doi.org/10.1007/BF02294183 [↗](https://doi.org/10.1007/BF02294183)
4. Carlo E. Bonferroni. 1936. Teoria Statistica Delle Classi e Calcolo Delle Probabilita, Libreria internazionale Seeber. [find](https://scholar.google.com/scholar?q=Carlo+E%2E+Bonferroni%2E+1936%2E+Teoria+Statistica+Delle+Classi+e+Calcolo+Delle+Probabilita%2C+Libreria+internazionale+Seeber%2E)
5. Hoa Trang Dang and Karolina Owczarzak. 2008. Overview of the TAC 2008 Update Summarization Task. In Proc. of the Text Analysis Conference (TAC). [find](https://scholar.google.com/scholar?q=Hoa+Trang+Dang+and+Karolina+Owczarzak%2E+2008%2E+Overview+of+the+TAC+2008+Update+Summarization+Task%2E+In+Proc%2E+of+the+Text+Analysis+Conference+%28TAC%29%2E)
6. H. T. Dang and K. Owczarzak. 2009. Overview of the TAC 2009 Summarization track. In Text Analysis Conference (TAC). [find](https://scholar.google.com/scholar?q=H%2E+T%2E+Dang+and+K%2E+Owczarzak%2E+2009%2E+Overview+of+the+TAC+2009+Summarization+track%2E+In+Text+Analysis+Conference+%28TAC%29%2E)
7. Sashka T. Davis, John M. Conroy, and Judith D. Schlesinger. 2012. OCCAMS–An optimal combinatorial covering algorithm for multi-document summarization. In 2012 IEEE 12th International Conference on Data Mining Workshops, pages 454–463. IEEE. [find](https://scholar.google.com/scholar?q=Sashka+T%2E+Davis%2C+John+M%2E+Conroy%2C+and+Judith+D%2E+Schlesinger%2E+2012%2E+OCCAMS%E2%80%93An+optimal+combinatorial+covering+algorithm+for+multi%2Ddocument+summarization%2E+In+2012+IEEE+12th+International+Conference+on+Dat)
8. Michael J. Denkowski and Alon Lavie. 2014. [Meteor universal: Language specific translation evaluation for any target language](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/). In Proceedings of the Ninth Workshop on Statistical Machine Translation, WMT@ACL 2014, June 26–27, 2014, Baltimore, Maryland, USA, pages 376–380. The Association for Compute…
9. Daniel Deutsch, Tania Bedrax-Weiss, and Dan Roth. 2021. [Towards question-answering as an automatic metric for evaluating the content quality of a summary](https://md.research.elamur.ai/free/en/papers/deutsch-2021-question-answering-as-automatic-metric/). Transactions of the Association for Computational Linguistics, 9.
10. Daniel Deutsch and Dan Roth. 2020. SacreROUGE: An open-source library for using and developing summarization evaluation metrics. In Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), pages 120–125, Online. Association for Computational Linguistics. [https://doi.org](https://doi.org) /10.18653/v1/20… [↗](https://doi.org)
11. Rotem Dror, Gili Baumer, Marina Bogomolov, and Roi Reichart. 2017. Replicability analysis for natural language processing: Testing significance with multiple datasets. Transactions of the Association for Computational Linguistics, 5:471–486. https://doi.org/10.1162/tacl_a_00074 [↗](https://doi.org/10.1162/tacl_a_00074)
12. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383–1392. [https://doi.…](https://doi.…) [↗](https://doi.…)
13. Rotem Dror, Lotem Peled-Cohen, Segev Shlomov, and Roi Reichart. 2020. Statistical significance testing for natural language processing. Synthesis Lectures on Human Language Technologies, 13(2):1–116. https://doi.org/10.2200/S00994ED1V01Y202002HLT045 [↗](https://doi.org/10.2200/S00994ED1V01Y202002HLT045)
14. Olive Jean Dunn and Virginia Clark. 1971. Comparison of tests of the equality of dependent correlation coefficients. Journal of the American Statistical Association, 66(336):904–908. [find](https://scholar.google.com/scholar?q=Olive+Jean+Dunn+and+Virginia+Clark%2E+1971%2E+Comparison+of+tests+of+the+equality+of+dependent+correlation+coefficients%2E+Journal+of+the+American+Statistical+Association%2C+66%28336%29%3A904%E2%80%93908%2E)
15. Bradley Efron and Robert Tibshirani. 1993. An Introduction to the Bootstrap. Springer. https://doi.org/10.1080/01621459 .1971.10482369 [↗](https://doi.org/10.1080/01621459)
16. A. R. Fabbri, Wojciech Kryscinski, Bryan McCann, R. Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. https://doi.org/10.1162/tacl a 00373 [↗](https://doi.org/10.1162/tacl)
17. Ronald Aylmer Fisher. 1992. Statistical methods for research workers, Breakthroughs in Statistics, Springer, pages 66–70. https://doi.org/10.1007/978-1-4612-4380-9 6 — [doi:10.1007/978-1-4612-4380-9](https://doi.org/10.1007/978-1-4612-4380-9) [↗](https://doi.org/10.1007/978-1-4612-4380-9)
18. George Giannakopoulos and Vangelis Karkaletsis. 2010. Summarization system evaluation variations based on n-gram graphs. In Proceedings of the Third Text Analysis Conference, TAC 2010, Gaithersburg, Maryland, USA, November 15–16, 2010. NIST. [find](https://scholar.google.com/scholar?q=George+Giannakopoulos+and+Vangelis+Karkaletsis%2E+2010%2E+Summarization+system+evaluation+variations+based+on+n%2Dgram+graphs%2E+In+Proceedings+of+the+Third+Text+Analysis+Conference%2C+TAC+2010%2C+Gaithersburg%2C+M)
19. George Giannakopoulos and Vangelis Karkaletsis. 2013. Summary evaluation: Together we stand NPowER-ed. In Computational Linguistics and Intelligent Text Processing - 14th International Conference, CICLing 2013, Samos, Greece, March 24-30, 2013, Proceedings, Part II, volume 7817 of Lecture Notes in C… — [doi:10.1007/978-3-642](https://doi.org/10.1007/978-3-642) [find](https://scholar.google.com/scholar?q=George+Giannakopoulos+and+Vangelis+Karkaletsis%2E+2013%2E+Summary+evaluation%3A+Together+we+stand+NPowER%2Ded%2E+In+Computational+Linguistics+and+Intelligent+Text+Processing+%2D+14th+International+Conference%2C+CIC)
20. George Giannakopoulos, Vangelis Karkaletsis, George A. Vouros, and Panagiotis Stamatopoulos. 2008. Summarization system evaluation revisited: N-gram graphs. ACM Transactions on Audio, Speech, and Language Processing, 5(3):5:1–5:39. https://doi.org/10.1145/1410358.1410359 — [doi:10.1145/1410358.1410359](https://doi.org/10.1145/1410358.1410359) [↗](https://doi.org/10.1145/1410358.1410359)
21. Dan Gillick and Yang Liu. 2010. [Non-expert evaluation of summarization systems is risky](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In Proceedings of the 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, Los Angeles, USA, June 6, 2010, pages 148–151. Association for Computational Linguistics.
22. Yvette Graham. 2015. [Re-evaluating automatic summarization with BLEU and 192 shades of ROUGE](https://md.research.elamur.ai/free/en/papers/graham-2015-re-evaluating-automatic-summarization-bleu/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 128–137, Lisbon, Portugal, Association for Computational Linguistics. https://doi.org/10.18653/v1/D15-1013 [↗](https://doi.org/10.18653/v1/D15-1013)
23. Yvette Graham and Timothy Baldwin. 2014. [Testing for significance of increased correlation with human judgment](https://md.research.elamur.ai/free/en/papers/graham-2014-testing-significance-increased-correlation-human/). In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 172–176.
24. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/graham-2014-testing-significance-increased-correlation-human/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. https://doi.org/10 .3115/v1/D14-1020
25. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. [Results of the WMT19 Metrics Shared Task: Segment-level and strong MT systems pose big challenges](https://md.research.elamur.ai/free/en/papers/ma-2019-results-wmt19-metrics-shared-task/). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62–90.
26. Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020. [Results of the WMT20 Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/mathur-2020-results-wmt20-metrics-shared-task/). In Proceedings of the Fifth Conference on Machine Translation, pages 688–725.
27. Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of CoNLL, pages 280–290. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+Cicero+Nogueira+dos+Santos%2C+Caglar+Gulcehre%2C+and+Bing+Xiang%2E+2016%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+RNNs+and+beyond%2E+In+Proceedings+of+CoNLL%2C+pag)
28. Eric W. Noreen. 1989. Computer Intensive Methods for Hypothesis Testing: An Introduction. Wiley, New York, 19:21. https://doi.org/10.18653/v1/K16-1028 [↗](https://doi.org/10.18653/v1/K16-1028)
29. Karolina Owczarzak, John M. Conroy, Hoa Trang Dang, and Ani Nenkova. 2012. An assessment of the accuracy of automatic evaluation in summarization. In Proceedings of Workshop on Evaluation Metrics and System Comparison for Automatic Summarization@NACCL-HLT 2012, Montrèal, Canada, June 2012, 2012, pa… [find](https://scholar.google.com/scholar?q=Karolina+Owczarzak%2C+John+M%2E+Conroy%2C+Hoa+Trang+Dang%2C+and+Ani+Nenkova%2E+2012%2E+An+assessment+of+the+accuracy+of+automatic+evaluation+in+summarization%2E+In+Proceedings+of+Workshop+on+Evaluation+Metrics+and+)
30. Karolina Owczarzak and Hoa Trang Dang. 2011. Overview of the TAC 2011 Summarization Track: Guided task and AESOP task. In Proceedings of the Text Analysis Conference (TAC 2011), Gaithersburg, Maryland, USA, November. [find](https://scholar.google.com/scholar?q=Karolina+Owczarzak+and+Hoa+Trang+Dang%2E+2011%2E+Overview+of+the+TAC+2011+Summarization+Track%3A+Guided+task+and+AESOP+task%2E+In+Proceedings+of+the+Text+Analysis+Conference+%28TAC+2011%29%2C+Gaithersburg%2C+Maryland)
31. Kishore Papineni, Salim Roukos, Todd Ward, and Wj Zhu. 2011. [BLEU: A method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In ACL, July, pages 311–318. https://doi.org/10.3115/1073083.1073135 [↗](https://doi.org/10.3115/1073083.1073135)
32. Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. 2017. Learning to score system summaries for better content selection evaluation. In Proceedings of the Workshop on New Frontiers in Summarization, NFiS@EMNLP 2017, Copenhagen, Denmark, September 7, 2017, pages 74–84. Association for Computational… [find](https://scholar.google.com/scholar?q=Maxime+Peyrard%2C+Teresa+Botschen%2C+and+Iryna+Gurevych%2E+2017%2E+Learning+to+score+system+summaries+for+better+content+selection+evaluation%2E+In+Proceedings+of+the+Workshop+on+New+Frontiers+in+Summarization%2C)
33. Peter Rankel, John Conroy, Eric Slud, and Dianne O’Leary. 2011. Ranking human and machine summarization systems. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 467–473, Edinburgh, Scotland, UK. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Peter+Rankel%2C+John+Conroy%2C+Eric+Slud%2C+and+Dianne+O%E2%80%99Leary%2E+2011%2E+Ranking+human+and+machine+summarization+systems%2E+In+Proceedings+of+the+2011+Conference+on+Empirical+Methods+in+Natural+Language+Processi)
34. Peter A. Rankel, John M. Conroy, Hoa Trang Dang, and Ani Nenkova. 2013. A decade of automatic content evaluation of news summaries: Reassessing the state of the art. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 131–136, So… [find](https://scholar.google.com/scholar?q=Peter+A%2E+Rankel%2C+John+M%2E+Conroy%2C+Hoa+Trang+Dang%2C+and+Ani+Nenkova%2E+2013%2E+A+decade+of+automatic+content+evaluation+of+news+summaries%3A+Reassessing+the+state+of+the+art%2E+In+Proceedings+of+the+51st+Annual+)
35. Peter A. Rankel, John M. Conroy, and Judith D. Schlesinger. 2012. Better metrics to automatically predict the quality of a text summary. Algorithms, 5(4):398–420. https://doi.org/10.3390/a5040398 — [doi:10.3390/a5040398](https://doi.org/10.3390/a5040398) [↗](https://doi.org/10.3390/a5040398)
36. Nornadiah Mohd Razali and Yap Bee Wah. 2011. Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling Tests. Journal of Statistical Modeling and Analytics, 2(1):21–33. [find](https://scholar.google.com/scholar?q=Nornadiah+Mohd+Razali+and+Yap+Bee+Wah%2E+2011%2E+Power+comparisons+of+Shapiro%2DWilk%2C+Kolmogorov%2DSmirnov%2C+Lilliefors+and+Anderson%2DDarling+Tests%2E+Journal+of+Statistical+Modeling+and+Analytics%2C+2%281%29%3A21%E2%80%9333%2E)
37. Ori Shapira, David Gabay, Yang Gao, Hadar Ronen, Ramakanth Pasunuru, Mohit Bansal, Yael Amsterdamer, and Ido Dagan. 2019. Crowdsourcing lightweight pyramids for manual summary evaluation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Ori+Shapira%2C+David+Gabay%2C+Yang+Gao%2C+Hadar+Ronen%2C+Ramakanth+Pasunuru%2C+Mohit+Bansal%2C+Yael+Amsterdamer%2C+and+Ido+Dagan%2E+2019%2E+Crowdsourcing+lightweight+pyramids+for+manual+summary+evaluation%2E+In+Proceedin)
38. Samuel Sanford Shapiro and Martin B. Wilk. 1965. An Analysis of variance test for normality (complete samples). Biometrika, 52(3/4):591–611. [find](https://scholar.google.com/scholar?q=Samuel+Sanford+Shapiro+and+Martin+B%2E+Wilk%2E+1965%2E+An+Analysis+of+variance+test+for+normality+%28complete+samples%29%2E+Biometrika%2C+52%283%2F4%29%3A591%E2%80%93611%2E)
39. Stephen Tratz and Eduard H. Hovy. 2008. Summarization evaluation using transformed basic elements. In Proceedings of the First Text Analysis Conference, TAC 2008, Gaithersburg, Maryland, USA, November 17–19, 2008. NIST. [find](https://scholar.google.com/scholar?q=Stephen+Tratz+and+Eduard+H%2E+Hovy%2E+2008%2E+Summarization+evaluation+using+transformed+basic+elements%2E+In+Proceedings+of+the+First+Text+Analysis+Conference%2C+TAC+2008%2C+Gaithersburg%2C+Maryland%2C+USA%2C+November)
40. Frank Wilcoxon. 1992. Individual comparisons by ranking methods. In Breakthroughs in Statistics, pages 196–202, Springer. https://doi.org/10.1093/biomet/52.3-4.591 — [doi:10.1093/biomet/52.3-4.591](https://doi.org/10.1093/biomet/52.3-4.591) [↗](https://doi.org/10.1093/biomet/52.3-4.591)
41. Evan James Williams. 1959. Regression Analysis, volume 14, Wiley. [find](https://scholar.google.com/scholar?q=Evan+James+Williams%2E+1959%2E+Regression+Analysis%2C+volume+14%2C+Wiley%2E)
42. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+BERTScore%3A+Evaluating+text+generation+with+BERT%2E+In+8th+International+Conference+on+Learning+Representations%2C+ICLR+2)
43. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. [MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance](https://md.research.elamur.ai/free/en/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International…

*Updated 24 October 2026*