--- title: "What Have We Achieved on Text Summarization? — Huang 2020" description: "Drawing on the Multidimensional Quality Metric (MQM) framework, the authors define PolyTope, a set… 49 references, cited by 69 papers here." authors: ["Dandan Huang", "Leyang Cui", "Sen Yang", "Guangsheng Bao", "Kun Wang", "Jun Xie", "Yue Zhang"] year: 2020 venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)" source: "https://aclanthology.org/2020.emnlp-main.33/" doi: "10.18653/v1/2020.emnlp-main.33" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/huang-2020-what-have-we-achieved-text/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/huang-2020-what-have-we-achieved-text/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / What Have We Achieved on Text Summarization? # What Have We Achieved on Text Summarization? **Authors:** [Dandan Huang](https://md.research.elamur.ai/free/en/authors/dandan-huang/), [Leyang Cui](https://md.research.elamur.ai/free/en/authors/leyang-cui/), [Sen Yang](https://md.research.elamur.ai/free/en/authors/sen-yang/), [Guangsheng Bao](https://md.research.elamur.ai/free/en/authors/guangsheng-bao/), [Kun Wang](https://md.research.elamur.ai/free/en/authors/kun-wang/), [Jun Xie](https://md.research.elamur.ai/free/en/authors/jun-xie/) and [Yue Zhang](https://md.research.elamur.ai/free/en/authors/yue-zhang/) · **Year:** 2020 · **Venue:** [Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2020](https://md.research.elamur.ai/free/en/venues/emnlp/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-main.33/) · [PDF](https://aclanthology.org/2020.emnlp-main.33.pdf) · [DOI 10.18653/v1/2020.emnlp-main.33](https://doi.org/10.18653/v1/2020.emnlp-main.33) · [OpenAlex](https://openalex.org/W3091889354) **Citations:** 74 citations (OpenAlex, 2026-10-14) · [69](https://md.research.elamur.ai/free/en/papers/huang-2020-what-have-we-achieved-text/cited-by/) in our library, of the 141,840 papers whose bibliographies we read ## Summary ROUGE gains on benchmarks such as CNN/DM say little about which errors summarizers actually make. This paper addresses that gap with a detailed manual error analysis. Drawing on the Multidimensional Quality Metric (MQM) framework, the authors define PolyTope, a set of 8 error categories split between Accuracy and Fluency. Annotators counted errors rather than assigning a single score per summary. They applied it to 10 re-implemented systems on CNN/DM: pre-neural baselines, neural extractive models and abstractive models. Under comparable conditions, extractive systems tend to beat abstractive ones because they stay more faithful to the source. Abstractive models more often omit content or hallucinate. Copy, coverage and hybrid extract-then-generate designs each fix particular weaknesses but introduce limits of their own. Pre-training helps most, especially joint encoder–decoder pre-training, and BART ranks first under both ROUGE and human judgment. The study also shows that ROUGE is reasonably reliable for ranking whole systems but unreliable for judging individual summaries. This makes fine-grained human diagnostics valuable for future evaluation work. ## Key points - **Systems evaluated:** Lead-3, TextRank, Seq2Seq with Attention, SummaRuNNer, Pointer-Generator, Pointer-Generator-with-Coverage, Bottom-Up, BertSumExt, BertSumExtAbs and BART. Annotation covered 150 CNN/DM test documents, giving 1,500 system outputs. - **ROUGE scores:** BART has the highest reported scores (ROUGE-1/2/L of 44.16/21.28/40.90). The plain Seq2Seq model is lowest at 31.33 ROUGE-1. - **Correlation with PolyTope:** At the system level, ROUGE-1/2/L correlate with PolyTope at Pearson 0.78/0.73/0.52. At the instance level the figures drop to 0.40/0.32/0.32. - **Length bias:** Summary length had little effect on PolyTope scores, with correlations of about 0.25 for BertSumExt and 0.27 for BertSumExtAbs. - **Human references and resources:** Gold reference summaries scored 96.41, with 63 Accuracy errors (mostly hallucinated content) and no Fluency errors. The annotated outputs and an evaluation toolkit are released at github.com/hddbang/PolyTope. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Tencent](https://md.research.elamur.ai/free/en/companies/tencent/) — Company, Shenzhen, China · 66 works · 17,945 citations - [Zhejiang University](https://md.research.elamur.ai/free/en/universities/zhejiang-university/) — University, Hangzhou, China · 46 works · 11,542 citations - [Westlake University](https://md.research.elamur.ai/free/en/universities/westlake-university/) — University, Hangzhou, China · 30 works · 7,732 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works 🧭 Topics (1): [📝 Summarization](https://md.research.elamur.ai/free/en/topics/summarization/) · 159 works ## Cite Cite as: Dandan Huang, Leyang Cui, Sen Yang, Guangsheng Bao, Kun Wang, Jun Xie and Yue Zhang. 2020. What Have We Achieved on Text Summarization?. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{huang2020what, title = {{What Have We Achieved on Text Summarization?}}, author = {Dandan Huang, Leyang Cui, Sen Yang, Guangsheng Bao, Kun Wang, Jun Xie and Yue Zhang}, booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)}, year = {2020}, doi = {10.18653/v1/2020.emnlp-main.33}, url = {https://doi.org/10.18653/v1/2020.emnlp-main.33} } ``` - [← previous SIMULEVAL: An Evaluation Toolkit for Simultaneous Translation](https://md.research.elamur.ai/free/en/papers/ma-2020-simuleval-evaluation-toolkit-simultaneous-translation/) - [next → Rethinking embedding coupling in pre-trained language models](https://md.research.elamur.ai/free/en/papers/chung-2020-rethinking-embedding-coupling-pre-trained/) - [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) ## References 49 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Vidhisha Balachandran, Artidoro Pagnoni, Jay Yoon Lee, Dheeraj Rajagopal, Jaime Carbonell, and Yulia Tsvetkov. 2020. Structsum: Incorporating latent and explicit sentence dependencies for single document summarization. arXiv preprint arXiv:2003.00576. — [arXiv:2003.00576](https://arxiv.org/abs/2003.00576) [find](https://scholar.google.com/scholar?q=Vidhisha+Balachandran%2C+Artidoro+Pagnoni%2C+Jay+Yoon+Lee%2C+Dheeraj+Rajagopal%2C+Jaime+Carbonell%2C+and+Yulia+Tsvetkov%2E+2020%2E+Structsum%3A+Incorporating+latent+and+explicit+sentence+dependencies+for+single+docum) 2. Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. 2019. [Better rewards yield better summaries: Learning to summarise without references](https://md.research.elamur.ai/free/en/papers/bohm-2019-better-rewards-yield-better-summaries/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Co… 3. Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. 2018. [Deep communicating agents for abstractive summarization](https://md.research.elamur.ai/free/en/papers/celikyilmaz-2018-deep-communicating-agents-abstractive-summarization/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), p… 4. Jianpeng Cheng and Mirella Lapata. 2016. [Neural summarization by extracting sentences and words](https://md.research.elamur.ai/free/en/papers/cheng-2016-neural-summarization-extracting-sentences-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. 5. James Clarke and Mirella Lapata. 2010. Discourse constraints for document compression. Computational Linguistics, 36(3):411–441. [find](https://scholar.google.com/scholar?q=James+Clarke+and+Mirella+Lapata%2E+2010%2E+Discourse+constraints+for+document+compression%2E+Computational+Linguistics%2C+36%283%29%3A411%E2%80%93441%2E) 6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… 7. Bonnie Dorr, David Zajic, and Richard Schwartz. 2003. [Hedge trimmer: A parse-and-trim approach to headline generation](https://md.research.elamur.ai/free/en/papers/dorr-2003-hedge-trimmer-parse-trim-approach/). In Proceedings of the HLT-NAACL 03 Text Summarization Workshop, pages 1–8. 8. Xiangyu Duan, Hoongfei Yu, Mingming Yin, Min Zhang, Weihua Luo, and Yue Zhang. 2019. Contrastive attention mechanism for abstractive sentence summarization. CoRR, abs/1910.13114. [find](https://scholar.google.com/scholar?q=Xiangyu+Duan%2C+Hoongfei+Yu%2C+Mingming+Yin%2C+Min+Zhang%2C+Weihua+Luo%2C+and+Yue+Zhang%2E+2019%2E+Contrastive+attention+mechanism+for+abstractive+sentence+summarization%2E+CoRR%2C+abs%2F1910%2E13114%2E) 9. Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. [Bottom-up abstractive summarization](https://md.research.elamur.ai/free/en/papers/gehrmann-2018-bottom-up-abstractive-summarization/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098–4109, Brussels, Belgium. Association for Computational Linguistics. 10. David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2003. English gigaword. Linguistic Data Consortium, Philadelphia, 4(1):34. [find](https://scholar.google.com/scholar?q=David+Graff%2C+Junbo+Kong%2C+Ke+Chen%2C+and+Kazuaki+Maeda%2E+2003%2E+English+gigaword%2E+Linguistic+Data+Consortium%2C+Philadelphia%2C+4%281%29%3A34%2E) 11. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. [Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies](https://md.research.elamur.ai/free/en/papers/grusky-2018-newsroom-dataset-1-3-million/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orlean… 12. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. [Incorporating copying mechanism in sequence-to-sequence learning](https://md.research.elamur.ai/free/en/papers/gu-2016-incorporating-copying-mechanism-sequence-sequence/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1631–1640, Berlin, Germany. Association for Computat… 13. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28,… 14. Wan-Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun. 2018. [A unified model for extractive and abstractive summarization using inconsistency loss](https://md.research.elamur.ai/free/en/papers/hsu-2018-unified-model-extractive-abstractive-summarization/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 132–14… 15. Hongyan Jing and Kathleen R. McKeown. 2000. Cut and paste based text summarization. In 1st Meeting of the North American Chapter of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hongyan+Jing+and+Kathleen+R%2E+McKeown%2E+2000%2E+Cut+and+paste+based+text+summarization%2E+In+1st+Meeting+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%2E) 16. Chris Kedzie, Kathleen R. McKeown, and Hal Daumé III. 2018. [Content selection in deep learning models of summarization](https://md.research.elamur.ai/free/en/papers/kedzie-2018-content-selection-deep-learning-models/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 1818–1828. 17. Svetlana Kiritchenko and Saif Mohammad. 2017. [Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation](https://md.research.elamur.ai/free/en/papers/kiritchenko-2017-best-worst-scaling-more-reliable/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 465–470, Vancouver, Canada. As… 18. Wojciech Kryscinski, Nitish Shirish Keskar, Bryan Mc-Cann, Caiming Xiong, and Richard Socher. 2019. [Neural text summarization: A critical evaluation](https://md.research.elamur.ai/free/en/papers/kryscinski-2019-neural-text-summarization-critical-evaluation/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langua… 19. Mirella Lapata and Regina Barzilay. 2005. Automatic evaluation of text coherence: Models and representations. In IJCAI, volume 5, pages 1085–1090. [find](https://scholar.google.com/scholar?q=Mirella+Lapata+and+Regina+Barzilay%2E+2005%2E+Automatic+evaluation+of+text+coherence%3A+Models+and+representations%2E+In+IJCAI%2C+volume+5%2C+pages+1085%E2%80%931090%2E) 20. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. [Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). arXiv preprint arXiv:1910.13461. 21. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. 22. Fei Liu, Jeffrey Flanigan, Sam Thomson, Norman Sadeh, and Noah A. Smith. 2015. [Toward abstractive summarization using semantic representations](https://md.research.elamur.ai/free/en/papers/liu-2015-abstractive-summarization-semantic-representations/). In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 107… 23. Feifan Liu and Yang Liu. 2008. Correlation between ROUGE and human evaluation of extractive meeting summaries. In Proceedings of ACL-08: HLT, Short Papers, pages 201–204, Columbus, Ohio. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Feifan+Liu+and+Yang+Liu%2E+2008%2E+Correlation+between+ROUGE+and+human+evaluation+of+extractive+meeting+summaries%2E+In+Proceedings+of+ACL%2D08%3A+HLT%2C+Short+Papers%2C+pages+201%E2%80%93204%2C+Columbus%2C+Ohio%2E+Association+f) 24. Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. CoRR, abs/1908.08345. [find](https://scholar.google.com/scholar?q=Yang+Liu+and+Mirella+Lapata%2E+2019%2E+Text+summarization+with+pretrained+encoders%2E+CoRR%2C+abs%2F1908%2E08345%2E) 25. Valerie Ruth Mariana. 2014. The multidimensional quality metric (mqm) framework: A new framework for translation quality assessment. [find](https://scholar.google.com/scholar?q=Valerie+Ruth+Mariana%2E+2014%2E+The+multidimensional+quality+metric+%28mqm%29+framework%3A+A+new+framework+for+translation+quality+assessment%2E) 26. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661. [find](https://scholar.google.com/scholar?q=Joshua+Maynez%2C+Shashi+Narayan%2C+Bernd+Bohnet%2C+and+Ryan+McDonald%2E+2020%2E+On+faithfulness+and+factuality+in+abstractive+summarization%2E+arXiv+preprint+arXiv%3A2005%2E00661%2E) 27. Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 404–411, Barcelona, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rada+Mihalcea+and+Paul+Tarau%2E+2004%2E+TextRank%3A+Bringing+order+into+text%2E+In+Proceedings+of+the+2004+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+404%E2%80%93411%2C+Barcelona%2C+Spain%2E+Asso) 28. Rafael Müller, Simon Kornblith, and Geoffrey Hinton. 2019. When does label smoothing help? arXiv preprint arXiv:1906.02629. [find](https://scholar.google.com/scholar?q=Rafael+Mu%CC%88ller%2C+Simon+Kornblith%2C+and+Geoffrey+Hinton%2E+2019%2E+When+does+label+smoothing+help%3F+arXiv+preprint+arXiv%3A1906%2E02629%2E) 29. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. [Summarunner: A recurrent neural network based sequence model for extractive summarization of documents](https://md.research.elamur.ai/free/en/papers/nallapati-2016-summarunner-recurrent-neural-network-based/). In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA., pages 3075–… 30. Ramesh Nallapati, Bowen Zhou, Çağlar dos Santos, Cicero andxiang Gu̇lçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290, Berlin, Germany. A… [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+C%CC%A7ag%CC%86lar+dos+Santos%2C+Cicero+andxiang+GuI%CC%80%E2%80%A1lc%CC%A7ehre%2C+and+Bing+Xiang%2E+2016%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+RNNs+and+beyond%2E+In+Proceedings+of+The+) 31. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. [Ranking sentences for extractive summarization with reinforcement learning](https://md.research.elamur.ai/free/en/papers/narayan-2018-ranking-sentences-extractive-summarization-reinforcement/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Paper… 32. Ani Nenkova and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 145–152, Boston, Massachusetts… [find](https://scholar.google.com/scholar?q=Ani+Nenkova+and+Rebecca+Passonneau%2E+2004%2E+Evaluating+content+selection+in+summarization%3A+The+pyramid+method%2E+In+Proceedings+of+the+Human+Language+Technology+Conference+of+the+North+American+Chapter+of) 33. Joel Larocca Neto, Alex Alves Freitas, and Celso A. A. Kaestner. 2002. Automatic text summarization using a machine learning approach. In Advances in Artificial Intelligence, 16th Brazilian Symposium on Artificial Intelligence, SBIA 2002, Porto de Galinhas/Recife, Brazil, November 11-14, 2002, Proce… [find](https://scholar.google.com/scholar?q=Joel+Larocca+Neto%2C+Alex+Alves+Freitas%2C+and+Celso+A%2E+A%2E+Kaestner%2E+2002%2E+Automatic+text+summarization+using+a+machine+learning+approach%2E+In+Advances+in+Artificial+Intelligence%2C+16th+Brazilian+Symposium+) 34. Jekaterina Novikova, Ondrej Dusek, Amanda Cercas Curry, and Verena Rieser. 2017. [Why we need new evaluation metrics for NLG](https://md.research.elamur.ai/free/en/papers/novikova-2017-why-we-need-new-evaluation/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 2241–2252. 35. L. Page, S. Brin, R. Motwani, and T. Winograd. 1998. [The pagerank citation ranking: Bringing order to the web](https://md.research.elamur.ai/free/en/papers/page-1999-pagerank-citation-ranking-bringing/). In Proceedings of the 7th International World Wide Web Conference, pages 161–172, Brisbane, Australia. 36. Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. 2017. Learning to score system summaries for better content selection evaluation. In Proceedings of the Workshop on New Frontiers in Summarization, pages 74–84, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maxime+Peyrard%2C+Teresa+Botschen%2C+and+Iryna+Gurevych%2E+2017%2E+Learning+to+score+system+summaries+for+better+content+selection+evaluation%2E+In+Proceedings+of+the+Workshop+on+New+Frontiers+in+Summarization%2C) 37. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389, Lisbon, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alexander+M%2E+Rush%2C+Sumit+Chopra%2C+and+Jason+Weston%2E+2015%2E+A+neural+attention+model+for+abstractive+sentence+summarization%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Language) 38. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 -August 4, Volume 1: Long Papers, pages 1073–… [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J%2E+Liu%2C+and+Christopher+D%2E+Manning%2E+2017%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computati) 39. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to summarize from human feedback. arXiv preprint arXiv:2009.01325. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeff+Wu%2C+Daniel+M+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+Christiano%2E+2020%2E+Learning+to+summarize+from+human+feedback%2E+arXiv+preprint+arXiv%3A) 40. Simeng Sun, Ori Shapira, Ido Dagan, and Ani Nenkova. 2019. How to compare summarizers without target length? pitfalls, solutions and re-examination of the neural summarization literature. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 21–29,… [find](https://scholar.google.com/scholar?q=Simeng+Sun%2C+Ori+Shapira%2C+Ido+Dagan%2C+and+Ani+Nenkova%2E+2019%2E+How+to+compare+summarizers+without+target+length%3F+pitfalls%2C+solutions+and+re%2Dexamination+of+the+neural+summarization+literature%2E+In+Proceedin) 41. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. [Modeling coverage for neural machine translation](https://md.research.elamur.ai/free/en/papers/tu-2016-modeling-coverage-neural-machine-translation/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 76– 85, Berlin, Germany. Association for Computational Ling… 42. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems, pages 2692–2700. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Meire+Fortunato%2C+and+Navdeep+Jaitly%2E+2015%2E+Pointer+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+2692%E2%80%932700%2E) 43. Jiacheng Xu and Greg Durrett. 2019. Neural extractive text summarization with syntactic compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3290– 3301, H… [find](https://scholar.google.com/scholar?q=Jiacheng+Xu+and+Greg+Durrett%2E+2019%2E+Neural+extractive+text+summarization+with+syntactic+compression%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9) 44. Fangfang Zhang, Jin-ge Yao, and Rui Yan. 2018. On the abstractiveness of neural document summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 785–790, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Fangfang+Zhang%2C+Jin%2Dge+Yao%2C+and+Rui+Yan%2E+2018%2E+On+the+abstractiveness+of+neural+document+summarization%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages) 45. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2019%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+arXiv+preprint+arXiv%3A1904%2E09675%2E) 46. Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. [Extractive summarization as text matching](https://md.research.elamur.ai/free/en/papers/zhong-2020-extractive-summarization-as-text-matching/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6197–6208, Online. Association for Computational Linguistics. 47. Ming Zhong, Pengfei Liu, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2019. [Searching for effective neural extractive summarization: What works and what’s next](https://md.research.elamur.ai/free/en/papers/zhong-2019-searching-effective-neural-extractive-summarization/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1049–1058, Florence, Italy. Association… 48. Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018. [Neural document summarization by jointly learning to score and select sentences](https://md.research.elamur.ai/free/en/papers/zhou-2018-neural-document-summarization-jointly-learning/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 654– 663, Melbo… 49. Markus Zopf. 2018. Estimating summary quality with pairwise preferences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1687–1696, New Orleans, Louisiana. Association for… [find](https://scholar.google.com/scholar?q=Markus+Zopf%2E+2018%2E+Estimating+summary+quality+with+pairwise+preferences%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+Lan) *Updated 24 October 2026*