---
title: "Re-evaluating Evaluation in Text — Bhandari 2020"
description: "ROUGE has been the main way to score summarization systems for about two decades."
authors: ["Manik Bhandari", "Pranav Narayan Gour", "Atabak Ashfaq", "Pengfei Liu", "Graham Neubig"]
year: 2020
venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/2020.emnlp-main.751/"
doi: "10.18653/v1/2020.emnlp-main.751"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bhandari-2020-re-evaluating-evaluation-text-summarization/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bhandari-2020-re-evaluating-evaluation-text-summarization/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) / Re-evaluating Evaluation in Text Summarization

# Re-evaluating Evaluation in Text Summarization

**Authors:** [Manik Bhandari](https://md.research.elamur.ai/free/en/authors/manik-bhandari/), Pranav Narayan Gour, [Atabak Ashfaq](https://md.research.elamur.ai/free/en/authors/atabak-ashfaq/), [Pengfei Liu](https://md.research.elamur.ai/free/en/authors/pengfei-liu/), [Graham Neubig](https://md.research.elamur.ai/free/en/authors/graham-neubig/) · **Year:** 2020 · **Venue:** Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-main.751/) · [PDF](https://aclanthology.org/2020.emnlp-main.751.pdf) · [DOI 10.18653/v1/2020.emnlp-main.751](https://doi.org/10.18653/v1/2020.emnlp-main.751) · [OpenAlex](https://openalex.org/W3093429745)

**Citations:** 147 citations (openalex-snapshot · 2026-10-14) · [109](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
ROUGE has been the main way to score summarization systems for about two decades. Most evidence on how well automatic metrics track human judgment comes from the TAC benchmarks, which are now old and whose summaries are weaker than current neural outputs. The authors test whether those conclusions still hold for modern systems. They collected outputs from 25 strong neural summarizers on CNN/DailyMail (CNNDM) and selected a difficulty-stratified set of test documents. Annotators judged the summaries with the LitePyramids content-selection protocol. Eight automatic metrics were then compared against these judgments at the system level and the summary level, using Pearson correlation and Williams' significance test.

Many findings from TAC reverse on CNNDM. On TAC, MoverScore and JS-2 look strongest. On CNNDM, ROUGE variants generally win. Also, metrics are not always weaker at ranking individual summaries than at ranking systems. The work shows that metric choice depends on dataset and evaluation setting. It argues for multi-dataset meta-evaluation that is updated regularly, possibly through a shared task similar to WMT Metrics. The released dataset is REALSumm.

## Key points
- **Benchmark:** outputs from 25 systems (11 extractive, e.g. BertSumExt, MatchSum; 14 abstractive, e.g. BART, T5, PEGASUS-free set incl. Unilm, Bottom-up) on 100 CNNDM test documents. The documents were sampled from 11,490 by binning on mean metric scores. The data is released as REALSumm.
- **Metrics compared:** ROUGE-1/2/L, BERTScore, MoverScore, SMS, WMS and JS-2, all in recall form except MoverScore. They were tested against TAC-2008 and TAC-2009 human judgments as well as the new CNNDM annotations.
- **System level:** ROUGE-2 correlates best with humans on CNNDM. MoverScore and JS-2 reach only about 0.1 Pearson on extractive systems. When extractive and abstractive systems are mixed, and only the top-k systems are ranked, all metrics lose correlation.
- **Summary vs system level:** MoverScore scores 0.05 at the system level on CNNDM Ext but 0.74 at the summary level. BERTScore, WMS, R-1 and R-L are negatively correlated with humans on TAC-2009.
- **Recommendation:** use R-2 for CNNDM-style data and JS-2 for TAC. When comparing two systems, R-2 is the most reliable for abstractive systems and R-1 for extractive ones.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

109 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 101 not yet.

- [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://md.research.elamur.ai/free/en/papers/fabbri-2022-qafacteval-improved-qa-based-factual/)
- [LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization](https://md.research.elamur.ai/free/en/papers/krishna-2023-longeval-guidelines-human-evaluation-faithfulness/)
- [Generative Judge for Evaluating Alignment](https://md.research.elamur.ai/free/en/papers/li-2023-generative-judge-evaluating-alignment/)
- [Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation](https://md.research.elamur.ai/free/en/papers/liu-2022-revisiting-gold-standard-grounding-summarization/)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://md.research.elamur.ai/free/en/papers/min-2023-factscore-fine-grained-atomic-evaluation/)
- [Frame In, Frame Out: Measuring Framing Bias in LLM-Generated News Summaries](https://md.research.elamur.ai/free/en/papers/pastorino-2026-frame-frame-out-measuring-framing/)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/)
- [BARTScore: Evaluating Generated Text as Text Generation](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{bhandari2020re,
  title = {{Re-evaluating Evaluation in Text Summarization}},
  author = {Bhandari, Manik and Gour, Pranav Narayan and Ashfaq, Atabak and Liu, Pengfei and Neubig, Graham},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.751},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.751}
}
```

- [← previous R^3: Reverse, Retrieve, and Rank for Sarcasm Generation with Commonsense Knowledge](https://md.research.elamur.ai/free/en/papers/chakrabarty-2020-r-textasciicircum3-reverse-retrieve-rank/)
- [next → Reactive Supervision: A New Method for Collecting Sarcasm Data](https://md.research.elamur.ai/free/en/papers/shmueli-2020-reactive-supervision-new-method-collecting/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/)

## References

56 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016. Results of the WMT16 metrics shared task. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 199–231, Berlin, Germany. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Yvette+Graham%2C+Amir+Kamran%2C+and+Milos%CC%8C+Stanojevic%CC%81%2E+2016%2E+Results+of+the+WMT16+metrics+shared+task%2E+In+Proceedings+of+the+First+Conference+on+Machine+Translation%3A+Volume+2%2C+Shared+Task+) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2016-results-wmt16-metrics-shared-task/)
2. Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. 2019. Better rewards yield better summaries: Learning to summarise without references. [find](https://scholar.google.com/scholar?q=Florian+Bo%CC%88hm%2C+Yang+Gao%2C+Christian+M%2E+Meyer%2C+Ori+Shapira%2C+Ido+Dagan%2C+and+Iryna+Gurevych%2E+2019%2E+Better+rewards+yield+better+summaries%3A+Learning+to+summarise+without+references%2E)
3. Arun Tejasvi Chaganty, Stephen Mussman, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evaluation. [find](https://scholar.google.com/scholar?q=Arun+Tejasvi+Chaganty%2C+Stephen+Mussman%2C+and+Percy+Liang%2E+2018%2E+The+price+of+debiasing+automatic+metrics+in+natural+language+evaluation%2E)
4. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675–686, Melbourne, Australia. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Yen%2DChun+Chen+and+Mohit+Bansal%2E+2018%2E+Fast+abstractive+summarization+with+reinforce%2Dselected+sentence+rewriting%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguist) [our card](https://md.research.elamur.ai/free/en/papers/chen-2018-fast-abstractive-summarization-reinforce-selected/)
5. Elizabeth Clark, Asli Celikyilmaz, and Noah A Smith. 2019. Sentence mover’s similarity: Automatic evaluation for multi-sentence texts. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2748–2760. [find](https://scholar.google.com/scholar?q=Elizabeth+Clark%2C+Asli+Celikyilmaz%2C+and+Noah+A+Smith%2E+2019%2E+Sentence+mover%E2%80%99s+similarity%3A+Automatic+evaluation+for+multi%2Dsentence+texts%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+)
6. Hoa Dang and Karolina Owczarzak. 2008. Overview of the tac 2008 update summarization task. In Proceedings of the First Text Analysis Conference (TAC 2008), pages 1 – 16. [find](https://scholar.google.com/scholar?q=Hoa+Dang+and+Karolina+Owczarzak%2E+2008%2E+Overview+of+the+tac+2008+update+summarization+task%2E+In+Proceedings+of+the+First+Text+Analysis+Conference+%28TAC+2008%29%2C+pages+1+%E2%80%93+16%2E)
7. Hoa Dang and Karolina Owczarzak. 2009. Overview of the tac 2009 summarization track. In Proceedings of the First Text Analysis Conference (TAC 2009), pages 1 – 16. [find](https://scholar.google.com/scholar?q=Hoa+Dang+and+Karolina+Owczarzak%2E+2009%2E+Overview+of+the+tac+2009+summarization+track%2E+In+Proceedings+of+the+First+Text+Analysis+Conference+%28TAC+2009%29%2C+pages+1+%E2%80%93+16%2E)
8. Hoa Trang Dang. 2005. Overview of duc 2005. In In Proceedings of the Document Understanding Conf. [find](https://scholar.google.com/scholar?q=Hoa+Trang+Dang%2E+2005%2E+Overview+of+duc+2005%2E+In+In+Proceedings+of+the+Document+Understanding+Conf%2E)
9. Wksp. 2005 (DUC 2005) at the Human Language Technology Conf./Conf. on Empirical Methods in Natural Language Processing (HLT/EMNLP. [find](https://scholar.google.com/scholar?q=Wksp%2E+2005+%28DUC+2005%29+at+the+Human+Language+Technology+Conf%2E%2FConf%2E+on+Empirical+Methods+in+Natural+Language+Processing+%28HLT%2FEMNLP%2E)
10. Hoa Trang Dang. 2006. Overview of duc 2006. In In Proceedings of HLT-NAACL 2006. [find](https://scholar.google.com/scholar?q=Hoa+Trang+Dang%2E+2006%2E+Overview+of+duc+2006%2E+In+In+Proceedings+of+HLT%2DNAACL+2006%2E)
11. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383–1392, Melbourne, A… [find](https://scholar.google.com/scholar?q=Rotem+Dror%2C+Gili+Baumer%2C+Segev+Shlomov%2C+and+Roi+Reichart%2E+2018%2E+The+hitchhiker%E2%80%99s+guide+to+testing+statistical+significance+in+natural+language+processing%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+)
12. Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 45–54, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+David+Grangier%2C+and+Michael+Auli%2E+2018%2E+Controllable+abstractive+summarization%2E+In+Proceedings+of+the+2nd+Workshop+on+Neural+Machine+Translation+and+Generation%2C+pages+45%E2%80%9354%2C+Melbourne%2C+Aus) [our card](https://md.research.elamur.ai/free/en/papers/fan-2018-controllable-abstractive-summarization/)
13. Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098–4109. [find](https://scholar.google.com/scholar?q=Sebastian+Gehrmann%2C+Yuntian+Deng%2C+and+Alexander+Rush%2E+2018%2E+Bottom%2Dup+abstractive+summarization%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+4098%E2%80%934) [our card](https://md.research.elamur.ai/free/en/papers/gehrmann-2018-bottom-up-abstractive-summarization/)
14. Cyril Goutte and Eric Gaussier. 2005. A probabilistic interpretation of precision, recall and f-score, with implication for evaluation. In European Conference on Information Retrieval, pages 345–359. Springer. [find](https://scholar.google.com/scholar?q=Cyril+Goutte+and+Eric+Gaussier%2E+2005%2E+A+probabilistic+interpretation+of+precision%2C+recall+and+f%2Dscore%2C+with+implication+for+evaluation%2E+In+European+Conference+on+Information+Retrieval%2C+pages+345%E2%80%93359%2E+)
15. Yvette Graham. 2015. Re-evaluating automatic summarization with BLEU and 192 shades of ROUGE. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 128–137, Lisbon, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yvette+Graham%2E+2015%2E+Re%2Devaluating+automatic+summarization+with+BLEU+and+192+shades+of+ROUGE%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+128%E2%80%93137%2C+) [our card](https://md.research.elamur.ai/free/en/papers/graham-2015-re-evaluating-automatic-summarization-bleu/)
16. Yvette Graham and Timothy Baldwin. 2014. Testing for significance of increased correlation with human judgment. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 172–176, Doha, Qatar. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yvette+Graham+and+Timothy+Baldwin%2E+2014%2E+Testing+for+significance+of+increased+correlation+with+human+judgment%2E+In+Proceedings+of+the+2014+Conference+on+Empirical+Methods+in+Natural+Language+Processin)
17. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1684–1692. [find](https://scholar.google.com/scholar?q=Karl+Moritz+Hermann%2C+Tomas+Kocisky%2C+Edward+Grefenstette%2C+Lasse+Espeholt%2C+Will+Kay%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+2015%2E+Teaching+machines+to+read+and+comprehend%2E+In+Advances+in+Neural+Information) [our card](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/)
18. Chris Kedzie, Kathleen McKeown, and Hal Daume III. 2018. Content selection in deep learning models of summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1818–1828. [find](https://scholar.google.com/scholar?q=Chris+Kedzie%2C+Kathleen+McKeown%2C+and+Hal+Daume+III%2E+2018%2E+Content+selection+in+deep+learning+models+of+summarization%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Proc)
19. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems, pages 13042–13054. [find](https://scholar.google.com/scholar?q=Li+Dong%2C+Nan+Yang%2C+Wenhui+Wang%2C+Furu+Wei%2C+Xiaodong+Liu%2C+Yu+Wang%2C+Jianfeng+Gao%2C+Ming+Zhou%2C+and+Hsiao%2DWuen+Hon%2E+2019%2E+Unified+language+model+pre%2Dtraining+for+natural+language+understanding+and+generatio) [our card](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/)
20. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388– 395, Barcelona, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2E+2004%2E+Statistical+significance+tests+for+machine+translation+evaluation%2E+In+Proceedings+of+the+2004+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+388%E2%80%93+395%2C+Barce)
21. Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. Bandit-Sum: Extractive summarization as a contextual bandit. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3739–3748, Brussels, Belgium. Association for Computati… [find](https://scholar.google.com/scholar?q=Yue+Dong%2C+Yikang+Shen%2C+Eric+Crawford%2C+Herke+van+Hoof%2C+and+Jackie+Chi+Kit+Cheung%2E+2018%2E+Bandit%2DSum%3A+Extractive+summarization+as+a+contextual+bandit%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+M)
22. Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability. [find](https://scholar.google.com/scholar?q=Klaus+Krippendorff%2E+2011%2E+Computing+krippendorff%E2%80%99s+alpha%2Dreliability%2E)
23. Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In International conference on machine learning, pages 957–966. [find](https://scholar.google.com/scholar?q=Matt+Kusner%2C+Yu+Sun%2C+Nicholas+Kolkin%2C+and+Kilian+Weinberger%2E+2015%2E+From+word+embeddings+to+document+distances%2E+In+International+conference+on+machine+learning%2C+pages+957%E2%80%93966%2E)
24. W Alan Lee Rodgers. 1988. Thirteen ways to look at the correlation coefficient. The American Statistician, 42(1):59–66. [find](https://scholar.google.com/scholar?q=W+Alan+Lee+Rodgers%2E+1988%2E+Thirteen+ways+to+look+at+the+correlation+coefficient%2E+The+American+Statistician%2C+42%281%29%3A59%E2%80%9366%2E)
25. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Yinhan+Liu%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Abdelrahman+Mohamed%2C+Omer+Levy%2C+Ves+Stoyanov%2C+and+Luke+Zettlemoyer%2E+2019%2E+Bart%3A+Denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+langu) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
26. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+Rouge%3A+A+package+for+automatic+evaluation+of+summaries%2E+Text+Summarization+Branches+Out%2E)
27. Chin-Yew Lin, Guihong Cao, Jianfeng Gao, and Jian-Yun Nie. 2006. An information-theoretic approach to automatic evaluation of summaries. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 463– 470, New York City, USA. Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2C+Guihong+Cao%2C+Jianfeng+Gao%2C+and+Jian%2DYun+Nie%2E+2006%2E+An+information%2Dtheoretic+approach+to+automatic+evaluation+of+summaries%2E+In+Proceedings+of+the+Human+Language+Technology+Conference+of+t)
28. Chin-Yew Lin and Franz Josef Och. 2004. ORANGE: a method for evaluating automatic evaluation metrics for machine translation. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 501–507, Geneva, Switzerland. COLING. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin+and+Franz+Josef+Och%2E+2004%2E+ORANGE%3A+a+method+for+evaluating+automatic+evaluation+metrics+for+machine+translation%2E+In+COLING+2004%3A+Proceedings+of+the+20th+International+Conference+on+Comput) [our card](https://md.research.elamur.ai/free/en/papers/lin-2004-orange-method-evaluating-automatic-evaluation/)
29. Yang Liu and Mirella Lapata. 2019a. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730–3740, Hong Kong, China. Asso… [find](https://scholar.google.com/scholar?q=Yang+Liu+and+Mirella+Lapata%2E+2019a%2E+Text+summarization+with+pretrained+encoders%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+International+Joi)
30. Yang Liu and Mirella Lapata. 2019b. Text summarization with pretrained encoders. [find](https://scholar.google.com/scholar?q=Yang+Liu+and+Mirella+Lapata%2E+2019b%2E+Text+summarization+with+pretrained+encoders%2E)
31. Annie Louis and Ani Nenkova. 2013. Automatically assessing machine summary content without a gold standard. Computational Linguistics, 39(2):267– 300. [find](https://scholar.google.com/scholar?q=Annie+Louis+and+Ani+Nenkova%2E+2013%2E+Automatically+assessing+machine+summary+content+without+a+gold+standard%2E+Computational+Linguistics%2C+39%282%29%3A267%E2%80%93+300%2E) [our card](https://md.research.elamur.ai/free/en/papers/louis-2013-automatically-assessing-machine-summary-content/)
32. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60. [find](https://scholar.google.com/scholar?q=Christopher+D%2E+Manning%2C+Mihai+Surdeanu%2C+John+Bauer%2C+Jenny+Finkel%2C+Steven+J%2E+Bethard%2C+and+David+Mc%2DClosky%2E+2014%2E+The+Stanford+CoreNLP+natural+language+processing+toolkit%2E+In+Association+for+Computation) [our card](https://md.research.elamur.ai/free/en/papers/manning-2014-stanford-corenlp-natural-language-processing/)
33. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Ça glar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+Cicero+dos+Santos%2C+C%CC%A7a+glar+Gulc%CC%A7ehre%2C+and+Bing+Xiang%2E+2016%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+rnns+and+beyond%2E)
34. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Paper… [find](https://scholar.google.com/scholar?q=Shashi+Narayan%2C+Shay+B%2E+Cohen%2C+and+Mirella+Lapata%2E+2018%2E+Ranking+sentences+for+extractive+summarization+with+reinforcement+learning%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter) [our card](https://md.research.elamur.ai/free/en/papers/narayan-2018-ranking-sentences-extractive-summarization-reinforcement/)
35. Ani Nenkova and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 145–152, Boston, Massachusetts… [find](https://scholar.google.com/scholar?q=Ani+Nenkova+and+Rebecca+Passonneau%2E+2004%2E+Evaluating+content+selection+in+summarization%3A+The+pyramid+method%2E+In+Proceedings+of+the+Human+Language+Technology+Conference+of+the+North+American+Chapter+of)
36. Jun-Ping Ng and Viktoria Abrecht. 2015. Better summarization evaluation with word embeddings for ROUGE. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1925–1930, Lisbon, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jun%2DPing+Ng+and+Viktoria+Abrecht%2E+2015%2E+Better+summarization+evaluation+with+word+embeddings+for+ROUGE%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages)
37. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241–2252, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jekaterina+Novikova%2C+Ondr%CC%8Cej+Dus%CC%8Cek%2C+Amanda+Cercas+Curry%2C+and+Verena+Rieser%2E+2017%2E+Why+we+need+new+evaluation+metrics+for+NLG%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Lan)
38. Maxime Peyrard. 2019. Studying summarization evaluation metrics in the appropriate scoring range. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5093–5100, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maxime+Peyrard%2E+2019%2E+Studying+summarization+evaluation+metrics+in+the+appropriate+scoring+range%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+5093)
39. Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. 2017. Learning to score system summaries for better content selection evaluation. In Proceedings of the Workshop on New Frontiers in Summarization, pages 74–84, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maxime+Peyrard%2C+Teresa+Botschen%2C+and+Iryna+Gurevych%2E+2017%2E+Learning+to+score+system+summaries+for+better+content+selection+evaluation%2E+In+Proceedings+of+the+Workshop+on+New+Frontiers+in+Summarization%2C)
40. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+2019%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtex)
41. Peter A. Rankel, John M. Conroy, Hoa Trang Dang, and Ani Nenkova. 2013. A decade of automatic content evaluation of news summaries: Reassessing the state of the art. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 131–136, So… [find](https://scholar.google.com/scholar?q=Peter+A%2E+Rankel%2C+John+M%2E+Conroy%2C+Hoa+Trang+Dang%2C+and+Ani+Nenkova%2E+2013%2E+A+decade+of+automatic+content+evaluation+of+news+summaries%3A+Reassessing+the+state+of+the+art%2E+In+Proceedings+of+the+51st+Annual+)
42. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada. Association for Compu… [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J%2E+Liu%2C+and+Christopher+D%2E+Manning%2E+2017%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computati)
43. Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Dipanjan+Das%2C+and+Ankur+P+Parikh%2E+2020%2E+Bleurt%3A+Learning+robust+metrics+for+text+generation%2E+arXiv+preprint+arXiv%3A2004%2E04696%2E)
44. Ori Shapira, David Gabay, Yang Gao, Hadar Ronen, Ramakanth Pasunuru, Mohit Bansal, Yael Amsterdamer, and Ido Dagan. 2019. Crowdsourcing lightweight pyramids for manual summary evaluation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Ori+Shapira%2C+David+Gabay%2C+Yang+Gao%2C+Hadar+Ronen%2C+Ramakanth+Pasunuru%2C+Mohit+Bansal%2C+Yael+Amsterdamer%2C+and+Ido+Dagan%2E+2019%2E+Crowdsourcing+lightweight+pyramids+for+manual+summary+evaluation%2E+In+Proceedin)
45. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International… [find](https://scholar.google.com/scholar?q=Wei+Zhao%2C+Maxime+Peyrard%2C+Fei+Liu%2C+Yang+Gao%2C+Christian+M%2E+Meyer%2C+and+Steffen+Eger%2E+2019%2E+MoverScore%3A+Text+generation+evaluating+with+contextualized+embeddings+and+earth+mover+distance%2E+In+Proceedings+) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/)
46. Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern… [find](https://scholar.google.com/scholar?q=Pauli+Virtanen%2C+Ralf+Gommers%2C+Travis+E%2E+Oliphant%2C+Matt+Haberland%2C+Tyler+Reddy%2C+David+Cournapeau%2C+Evgeni+Burovski%2C+Pearu+Peterson%2C+Warren+Weckesser%2C+Jonathan+Bright%2C+Ste%CC%81fan+J%2E+van+der+Walt%2C+Matthew+Br) [our card](https://md.research.elamur.ai/free/en/papers/virtanen-2019-scipy-1-0-fundamental-algorithms/)
47. Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Extractive summarization as text matching. arXiv preprint arXiv:2004.08795. [find](https://scholar.google.com/scholar?q=Ming+Zhong%2C+Pengfei+Liu%2C+Yiran+Chen%2C+Danqing+Wang%2C+Xipeng+Qiu%2C+and+Xuanjing+Huang%2E+2020%2E+Extractive+summarization+as+text+matching%2E+arXiv+preprint+arXiv%3A2004%2E08795%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhong-2020-extractive-summarization-as-text-matching/)
48. Danqing Wang, Pengfei Liu, Yining Zheng, Xipeng Qiu, and Xuanjing Huang. 2020. Heterogeneous graph neural networks for extractive document summarization. arXiv preprint arXiv:2004.12393. [find](https://scholar.google.com/scholar?q=Danqing+Wang%2C+Pengfei+Liu%2C+Yining+Zheng%2C+Xipeng+Qiu%2C+and+Xuanjing+Huang%2E+2020%2E+Heterogeneous+graph+neural+networks+for+extractive+document+summarization%2E+arXiv+preprint+arXiv%3A2004%2E12393%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2020-heterogeneous-graph-neural-networks-extractive/)
49. Ming Zhong, Pengfei Liu, Danqing Wang, Xipeng Qiu, and Xuan-Jing Huang. 2019. Searching for effective neural extractive summarization: What works and what’s next. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 1049–1058. [find](https://scholar.google.com/scholar?q=Ming+Zhong%2C+Pengfei+Liu%2C+Danqing+Wang%2C+Xipeng+Qiu%2C+and+Xuan%2DJing+Huang%2E+2019%2E+Searching+for+effective+neural+extractive+summarization%3A+What+works+and+what%E2%80%99s+next%2E+In+Proceedings+of+the+57th+Conference)
50. Deyu Zhou, Linsen Guo, and Yulan He. 2018. Neural storyline extraction model for storyline generation from news articles. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1… [find](https://scholar.google.com/scholar?q=Deyu+Zhou%2C+Linsen+Guo%2C+and+Yulan+He%2E+2018%2E+Neural+storyline+extraction+model+for+storyline+generation+from+news+articles%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Ass)
51. Evan J. Williams. 1959. Regression analysis. Wiley, New York, 14. [find](https://scholar.google.com/scholar?q=Evan+J%2E+Williams%2E+1959%2E+Regression+analysis%2E+Wiley%2C+New+York%2C+14%2E)
52. Wonjin Yoon, Yoon Sun Yeo, Minbyul Jeong, Bong-Jun Yi, and Jaewoo Kang. 2020. Learning by semantic similarity makes abstractive summarization better. arXiv preprint arXiv:2002.07767. — [arXiv:2002.07767](https://arxiv.org/abs/2002.07767) [find](https://scholar.google.com/scholar?q=Wonjin+Yoon%2C+Yoon+Sun+Yeo%2C+Minbyul+Jeong%2C+Bong%2DJun+Yi%2C+and+Jaewoo+Kang%2E+2020%2E+Learning+by+semantic+similarity+makes+abstractive+summarization+better%2E+arXiv+preprint+arXiv%3A2002%2E07767%2E+%E2%80%94+%5BarXiv%3A2002%2E077)
53. Haoyu Zhang, Yeyun Gong, Yu Yan, Nan Duan, Jianjun Xu, Ji Wang, Ming Gong, and Ming Zhou. 2019a. Pretraining-based natural language generation for text summarization. arXiv preprint arXiv:1902.09243. [find](https://scholar.google.com/scholar?q=Haoyu+Zhang%2C+Yeyun+Gong%2C+Yu+Yan%2C+Nan+Duan%2C+Jianjun+Xu%2C+Ji+Wang%2C+Ming+Gong%2C+and+Ming+Zhou%2E+2019a%2E+Pretraining%2Dbased+natural+language+generation+for+text+summarization%2E+arXiv+preprint+arXiv%3A1902%2E09243%2E)
54. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+In+International+Conference+on+Learning+Representations%2E)
55. Xingxing Zhang, Mirella Lapata, Furu Wei, and Ming Zhou. 2018. Neural latent extractive document summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 779–784, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xingxing+Zhang%2C+Mirella+Lapata%2C+Furu+Wei%2C+and+Ming+Zhou%2E+2018%2E+Neural+latent+extractive+document+summarization%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processin)
56. Xingxing Zhang, Furu Wei, and Ming Zhou. 2019b. HIBERT: Document level pre-training of hierarchical bidirectional transformers for document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5059–5069, Florence, Italy. Association for Com… [find](https://scholar.google.com/scholar?q=Xingxing+Zhang%2C+Furu+Wei%2C+and+Ming+Zhou%2E+2019b%2E+HIBERT%3A+Document+level+pre%2Dtraining+of+hierarchical+bidirectional+transformers+for+document+summarization%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2019-hibert-document-level-pre-training/)