--- title: "The limits of automatic summarisation according to ROUGE (2017)" description: "This short paper questions how meaningful ROUGE-based progress in extractive summarisation really is. 8 references, cited by 68 papers here." authors: ["Natalie Schluter"] year: 2017 venue: "Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers" source: "https://aclanthology.org/E17-2007/" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/schluter-2017-limits-automatic-summarisation-according-rouge/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/schluter-2017-limits-automatic-summarisation-according-rouge/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/) / The limits of automatic summarisation according to ROUGE # The limits of automatic summarisation according to ROUGE **Authors:** [Natalie Schluter](https://md.research.elamur.ai/free/en/authors/natalie-schluter/) · **Year:** 2017 · **Venue:** [Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2017](https://md.research.elamur.ai/free/en/venues/eacl/2017/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/E17-2007/) · [PDF](https://aclanthology.org/E17-2007.pdf) · [OpenAlex](https://openalex.org/W2741672218) **Citations:** 102 citations (OpenAlex, 2026-10-14) · [68](https://md.research.elamur.ai/free/en/papers/schluter-2017-limits-automatic-summarisation-according-rouge/cited-by/) in our library, of the 141,840 papers whose bibliographies we read ## Summary This short paper questions how meaningful ROUGE-based progress in extractive summarisation really is. It first gives what the author presents as the first formal proof that building an extractive summary that maximises ROUGE-n is NP-hard. The proof is a reduction from the max k-dominating set problem on cubic graphs: vertex neighbourhoods become sentences, and the graph's vertex set becomes the reference summary. Dummy-symbol padding extends the result from ROUGE-1 to any n. Despite this hardness, the empirical comparison tells a different story. Oracle extractive summaries were built greedily and exactly (via an ILP solver) on three benchmarks: duc04 (newswire), echr (court judgments) and wiki (Wikipedia). The two methods reach very similar scores. The best attainable scores are also far below 100% and differ widely between datasets. Averaging over multiple reference summaries makes perfect scores impossible, and human summaries score low against one another. This matters because ROUGE has no natural upper bound. It also offers little reason to expect gains from switching to exact or supervised methods. ## Key points - **Theoretical result:** Theorems 1–2 show that exact oracle extractive summarisation is NP-hard under ROUGE-1 and, more generally, ROUGE-n. - **Datasets and budgets:** - duc04: 30 document sets, with budgets of 665 bytes and 100 words. - echr: 138 pairs from HUDOC, 805-word budget. - wiki: 111 "good article" pairs, 335 budget. - **Tools:** - Evaluation used ROUGE-1.5.5. - Exact oracles were computed with the GLPK solver. - The greedy oracle adds sentences by length-normalised ROUGE gain and clips the final sentence to fit the budget. - **Greedy vs. exact:** Oracle ROUGE-1 is roughly 50 on duc04, about 64 on wiki and about 84 on echr, with greedy and exact within about a point of each other. On duc04, greedy is even slightly better, likely thanks to sentence clipping. - **Human ceiling:** On duc04, scoring one reference summary against the other three yields only 39.92 ROUGE-1 and 9.39 ROUGE-2. In one example set, 136 of 160 required stemmed words can only be partially matched. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Copenhagen](https://md.research.elamur.ai/free/en/universities/university-of-copenhagen/) — University, Copenhagen, Denmark · 78 works · 14,228 citations - [IT University of Copenhagen](https://md.research.elamur.ai/free/en/universities/it-university-of-copenhagen/) — University, Copenhagen, Denmark · 16 works · 2,225 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works 🧭 Topics (1): [📝 Summarization](https://md.research.elamur.ai/free/en/topics/summarization/) · 159 works ## Cite Cite as: Natalie Schluter. 2017. The limits of automatic summarisation according to ROUGE. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{schluter2017limits, title = {{The limits of automatic summarisation according to ROUGE}}, author = {Natalie Schluter}, booktitle = {Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers}, year = {2017}, doi = {10.18653/v1/e17-2007}, url = {https://doi.org/10.18653/v1/e17-2007} } ``` - [← previous Personalized Machine Translation: Preserving Original Author Traits](https://md.research.elamur.ai/free/en/papers/rabinovich-2017-personalized-machine-translation-preserving-original/) - [next → Social Bias in Elicited Natural Language Inferences](https://md.research.elamur.ai/free/en/papers/rudinger-2017-social-bias-elicited-natural-language/) - [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/) ## References 8 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016. [Abstractive sentence summarization with attentive recurrent neural networks](https://md.research.elamur.ai/free/en/papers/chopra-2016-abstractive-sentence-summarization-attentive-recurrent/). In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 93–98, San Di… 2. M.R. Garey and D. S. Johnson. 1979. Computers and Intractability: A Guide to the Theory of NP-Completeness. Freeman and Company. [find](https://scholar.google.com/scholar?q=M%2ER%2E+Garey+and+D%2E+S%2E+Johnson%2E+1979%2E+Computers+and+Intractability%3A+A+Guide+to+the+Theory+of+NP%2DCompleteness%2E+Freeman+and+Company%2E) 3. Dan Gillick and Benoit Favre. 2009. [A scalable global model for summarization](https://md.research.elamur.ai/free/en/papers/gillick-2009-scalable-global-model-summarization/). In Proceedings of ILP, pages 10–18. 4. Yvette Graham. 2015. [Re-evaluating automatic summarization with bleu and 192 shades of rouge](https://md.research.elamur.ai/free/en/papers/graham-2015-re-evaluating-automatic-summarization-bleu/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 128–137, Lisbon, Portugal. 5. Kai Hong, John M. Conroy, Benoit Favre, Alex Kulesza, Hui Lin, and Ani Nenkova. 2014. A repository of state of the art and competitive baseline summaries for generic news summarization. In Proc of LREC. [find](https://scholar.google.com/scholar?q=Kai+Hong%2C+John+M%2E+Conroy%2C+Benoit+Favre%2C+Alex+Kulesza%2C+Hui+Lin%2C+and+Ani+Nenkova%2E+2014%2E+A+repository+of+state+of+the+art+and+competitive+baseline+summaries+for+generic+news+summarization%2E+In+Proc+of+LRE) 6. Chin-Yew Lin and Eduard Hovy. 2003. [Automatic evaluation of summaries using n-gram co-occurrence statistics](https://md.research.elamur.ai/free/en/papers/lin-2003-automatic-evaluation-summaries-n-gram/). In Proceedings of HLT-NAACL, Edmonton, AB, Canada. 7. Chin-Yew Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74–81, Barcelona, Spain. 8. Feifan Liu and Yang Liu. 2008. Correlation between rouge and human evaluation of extractive meeting [find](https://scholar.google.com/scholar?q=Feifan+Liu+and+Yang+Liu%2E+2008%2E+Correlation+between+rouge+and+human+evaluation+of+extractive+meeting) *Updated 24 October 2026*