---
title: "On Extractive and Abstractive Neural Document — Pilault 2020"
description: "Neural abstractive models struggle with documents several thousand words long."
authors: ["Jonathan Pilault", "Raymond Li", "Sandeep Subramanian", "Chris Pal"]
year: 2020
venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/2020.emnlp-main.748/"
doi: "10.18653/v1/2020.emnlp-main.748"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pilault-2020-extractive-abstractive-neural-document-summarization/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pilault-2020-extractive-abstractive-neural-document-summarization/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / On Extractive and Abstractive Neural Document Summarization with Transformer Language Models

# On Extractive and Abstractive Neural Document Summarization with Transformer Language Models

**Authors:** Jonathan Pilault, [Raymond Li](https://md.research.elamur.ai/free/en/authors/raymond-li/), [Sandeep Subramanian](https://md.research.elamur.ai/free/en/authors/sandeep-subramanian/), Chris Pal · **Year:** 2020 · **Venue:** Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-main.748/) · [PDF](https://aclanthology.org/2020.emnlp-main.748.pdf) · [DOI 10.18653/v1/2020.emnlp-main.748](https://doi.org/10.18653/v1/2020.emnlp-main.748) · [OpenAlex](https://openalex.org/W3098136301)

**Citations:** 183 citations (openalex-snapshot · 2026-10-14) · [76](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Neural abstractive models struggle with documents several thousand words long. Recurrent seq2seq systems handle long-range dependencies poorly, and transformer language models (TLMs) cannot fit such long inputs in memory. The authors propose a two-stage hybrid. First, an extractive sentence pointer network selects salient sentences. Then an autoregressive TLM is trained on documents reordered with delimiter tokens: introduction, extracted sentences, abstract/summary, then the rest of the text. At inference, the model receives the introduction (or the whole document for news and patents) plus the extracted sentences, and it writes the summary. The method is tested on arXiv, PubMed, Newsroom and BigPatent. It is compared with simple baselines, prior state-of-the-art systems, and extractive-only, transformer-only and combined variants. The hybrid gets higher ROUGE than copy-mechanism models and reproduces fewer n-grams from the source, so its outputs resemble human abstracts more closely. The work shows that restructuring the input, with no copy mechanism, can make a plain language model a strong long-document summarizer and a useful baseline.

## Key points
- Pipeline: a sentence pointer network for extraction, followed by a GPT-like TLM conditioned on the reordered input. The authors describe the extractive step as a kind of hard attention.
- The approach also helps on datasets whose full input fits in memory (Newsroom, BigPatent), not only on long arXiv/PubMed papers.
- Transformer-based summaries show n-gram copying statistics closer to human-written abstracts than copy-mechanism baselines.
- In human evaluation, TLM outputs ranked well for coherence and fluency, while purely extractive methods scored higher on informativeness and relevance.
- GPT-2-style models trained naively on web crawl data summarized far worse than the baselines. The paper's own abstract was partly produced by one of its models.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

76 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 71 not yet.

- [DialogSum: A Real-Life Scenario Dialogue Summarization Dataset](https://md.research.elamur.ai/free/en/papers/chen-2021-dialogsum-real-life-scenario-dialogue/)
- [Efficient Attentions for Long Document Summarization](https://md.research.elamur.ai/free/en/papers/huang-2021-efficient-attentions-long-document-summarization/)
- [Recursively Summarizing Books with Human Feedback](https://md.research.elamur.ai/free/en/papers/wu-2021-recursively-summarizing-books-human-feedback/)
- [Big Bird: Transformers for Longer Sequences](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{pilault2020extractive,
  title = {{On Extractive and Abstractive Neural Document Summarization with Transformer Language Models}},
  author = {Pilault, Jonathan and Li, Raymond and Subramanian, Sandeep and Pal, Chris},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.748},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.748}
}
```

- [← previous On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/wang-2020-exposure-bias-hallucination-domain-shift/)
- [next → On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment](https://md.research.elamur.ai/free/en/papers/wang-2020-negative-interference-multilingual-models-findings/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/)

## References

41 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2014%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+arXiv+preprint+arXiv%3A1409%2E0473%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
2. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. CoRR, abs/1603.06393. [find](https://scholar.google.com/scholar?q=Jiatao+Gu%2C+Zhengdong+Lu%2C+Hang+Li%2C+and+Victor+O%2E+K%2E+Li%2E+2016%2E+Incorporating+copying+mechanism+in+sequence%2Dto%2Dsequence+learning%2E+CoRR%2C+abs%2F1603%2E06393%2E) [our card](https://md.research.elamur.ai/free/en/papers/gu-2016-incorporating-copying-mechanism-sequence-sequence/)
3. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. arXiv preprint arXiv:1805.11080. [find](https://scholar.google.com/scholar?q=Yen%2DChun+Chen+and+Mohit+Bansal%2E+2018%2E+Fast+abstractive+summarization+with+reinforce%2Dselected+sentence+rewriting%2E+arXiv+preprint+arXiv%3A1805%2E11080%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2018-fast-abstractive-summarization-reinforce-selected/)
4. Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. arXiv preprint arXiv:1603.08148. [find](https://scholar.google.com/scholar?q=Caglar+Gulcehre%2C+Sungjin+Ahn%2C+Ramesh+Nallapati%2C+Bowen+Zhou%2C+and+Yoshua+Bengio%2E+2016%2E+Pointing+the+unknown+words%2E+arXiv+preprint+arXiv%3A1603%2E08148%2E) [our card](https://md.research.elamur.ai/free/en/papers/gulcehre-2016-pointing-unknown-words/)
5. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. arXiv preprint arXiv:1603.07252. [find](https://scholar.google.com/scholar?q=Jianpeng+Cheng+and+Mirella+Lapata%2E+2016%2E+Neural+summarization+by+extracting+sentences+and+words%2E+arXiv+preprint+arXiv%3A1603%2E07252%2E) [our card](https://md.research.elamur.ai/free/en/papers/cheng-2016-neural-summarization-extracting-sentences-words/)
6. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735– 1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+Comput%2E%2C+9%288%29%3A1735%E2%80%93+1780%2E)
7. Sumit Chopra, Michael Auli, and Alexander M Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 93–98. [find](https://scholar.google.com/scholar?q=Sumit+Chopra%2C+Michael+Auli%2C+and+Alexander+M+Rush%2E+2016%2E+Abstractive+sentence+summarization+with+attentive+recurrent+neural+networks%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter) [our card](https://md.research.elamur.ai/free/en/papers/chopra-2016-abstractive-sentence-summarization-attentive-recurrent/)
8. Wan-Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun. 2018. A unified model for extractive and abstractive summarization using inconsistency loss. arXiv preprint arXiv:1805.06266. [find](https://scholar.google.com/scholar?q=Wan%2DTing+Hsu%2C+Chieh%2DKai+Lin%2C+Ming%2DYing+Lee%2C+Kerui+Min%2C+Jing+Tang%2C+and+Min+Sun%2E+2018%2E+A+unified+model+for+extractive+and+abstractive+summarization+using+inconsistency+loss%2E+arXiv+preprint+arXiv%3A1805%2E06) [our card](https://md.research.elamur.ai/free/en/papers/hsu-2018-unified-model-extractive-abstractive-summarization/)
9. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. CoRR, abs/1804.05685. [find](https://scholar.google.com/scholar?q=Arman+Cohan%2C+Franck+Dernoncourt%2C+Doo+Soon+Kim%2C+Trung+Bui%2C+Seokhwan+Kim%2C+Walter+Chang%2C+and+Nazli+Goharian%2E+2018%2E+A+discourse%2Daware+attention+model+for+abstractive+summarization+of+long+documents%2E+CoRR%2C) [our card](https://md.research.elamur.ai/free/en/papers/cohan-2018-discourse-aware-attention-model-abstractive/)
10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2E)
11. Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of artificial intelligence research, 22:457–479. [find](https://scholar.google.com/scholar?q=Gu%CC%88nes+Erkan+and+Dragomir+R+Radev%2E+2004%2E+Lexrank%3A+Graph%2Dbased+lexical+centrality+as+salience+in+text+summarization%2E+Journal+of+artificial+intelligence+research%2C+22%3A457%E2%80%93479%2E)
12. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Mike+Lewis%2C+and+Yann+Dauphin%2E+2018%2E+Hierarchical+neural+story+generation%2E+arXiv+preprint+arXiv%3A1805%2E04833%2E)
13. Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. 2018. Bottom-up abstractive summarization. arXiv preprint arXiv:1808.10792. [find](https://scholar.google.com/scholar?q=Sebastian+Gehrmann%2C+Yuntian+Deng%2C+and+Alexander+M+Rush%2E+2018%2E+Bottom%2Dup+abstractive+summarization%2E+arXiv+preprint+arXiv%3A1808%2E10792%2E) [our card](https://md.research.elamur.ai/free/en/papers/gehrmann-2018-bottom-up-abstractive-summarization/)
14. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. pages 2672–2680. [find](https://scholar.google.com/scholar?q=Ian+Goodfellow%2C+Jean+Pouget%2DAbadie%2C+Mehdi+Mirza%2C+Bing+Xu%2C+David+Warde%2DFarley%2C+Sherjil+Ozair%2C+Aaron+Courville%2C+and+Yoshua+Bengio%2E+2014%2E+Generative+adversarial+nets%2E+pages+2672%E2%80%932680%2E)
15. Yvette Graham. 2015. Re-evaluating automatic summarization with bleu and 192 shades of rouge. pages 128–137. [find](https://scholar.google.com/scholar?q=Yvette+Graham%2E+2015%2E+Re%2Devaluating+automatic+summarization+with+bleu+and+192+shades+of+rouge%2E+pages+128%E2%80%93137%2E) [our card](https://md.research.elamur.ai/free/en/papers/graham-2015-re-evaluating-automatic-summarization-bleu/)
16. Max Grusky, Mor Naaman, and Yoav Artzi. 2018a. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. arXiv preprint arXiv:1804.11283. [find](https://scholar.google.com/scholar?q=Max+Grusky%2C+Mor+Naaman%2C+and+Yoav+Artzi%2E+2018a%2E+Newsroom%3A+A+dataset+of+1%2E3+million+summaries+with+diverse+extractive+strategies%2E+arXiv+preprint+arXiv%3A1804%2E11283%2E) [our card](https://md.research.elamur.ai/free/en/papers/grusky-2018-newsroom-dataset-1-3-million/)
17. Max Grusky, Mor Naaman, and Yoav Artzi. 2018b. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), p… [find](https://scholar.google.com/scholar?q=Max+Grusky%2C+Mor+Naaman%2C+and+Yoav+Artzi%2E+2018b%2E+Newsroom%3A+A+dataset+of+1%2E3+million+summaries+with+diverse+extractive+strategies%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+t) [our card](https://md.research.elamur.ai/free/en/papers/grusky-2018-newsroom-dataset-1-3-million/)
18. Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2019. Hallucinations in neural machine translation. [find](https://scholar.google.com/scholar?q=Katherine+Lee%2C+Orhan+Firat%2C+Ashish+Agarwal%2C+Clara+Fannjiang%2C+and+David+Sussillo%2E+2019%2E+Hallucinations+in+neural+machine+translation%2E)
19. Chin-Yew Lin. 2004. Looking for a few good metrics: Automatic summarization evaluation-how many samples are enough? In NTCIR. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+Looking+for+a+few+good+metrics%3A+Automatic+summarization+evaluation%2Dhow+many+samples+are+enough%3F+In+NTCIR%2E)
20. Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. arXiv preprint arXiv:1801.10198. [find](https://scholar.google.com/scholar?q=Peter+J+Liu%2C+Mohammad+Saleh%2C+Etienne+Pot%2C+Ben+Goodrich%2C+Ryan+Sepassi%2C+Lukasz+Kaiser%2C+and+Noam+Shazeer%2E+2018%2E+Generating+wikipedia+by+summarizing+long+sequences%2E+arXiv+preprint+arXiv%3A1801%2E10198%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2018-generating-wikipedia-summarizing-long-sequences/)
21. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Hieu+Pham%2C+and+Christopher+D+Manning%2E+2015%2E+Effective+approaches+to+attention%2Dbased+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1508%2E04025%2E)
22. Afonso Mendes, Shashi Narayan, Sebastião Miranda, Zita Marinho, André FT Martins, and Shay B Cohen. 2019. Jointly extracting and compressing documents with summary state representations. arXiv preprint arXiv:1904.02020. [find](https://scholar.google.com/scholar?q=Afonso+Mendes%2C+Shashi+Narayan%2C+Sebastia%CC%83o+Miranda%2C+Zita+Marinho%2C+Andre%CC%81+FT+Martins%2C+and+Shay+B+Cohen%2E+2019%2E+Jointly+extracting+and+compressing+documents+with+summary+state+representations%2E+arXiv+prepr)
23. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. 2017. Mixed precision training. arXiv preprint arXiv:1710.03740. [find](https://scholar.google.com/scholar?q=Paulius+Micikevicius%2C+Sharan+Narang%2C+Jonah+Alben%2C+Gregory+Diamos%2C+Erich+Elsen%2C+David+Garcia%2C+Boris+Ginsburg%2C+Michael+Houston%2C+Oleksii+Kuchaiev%2C+Ganesh+Venkatesh%2C+et+al%2E+2017%2E+Mixed+precision+training%2E)
24. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404–411. [find](https://scholar.google.com/scholar?q=Rada+Mihalcea+and+Paul+Tarau%2E+2004%2E+Textrank%3A+Bringing+order+into+text%2E+In+Proceedings+of+the+2004+conference+on+empirical+methods+in+natural+language+processing%2C+pages+404%E2%80%93411%2E)
25. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Feifei+Zhai%2C+and+Bowen+Zhou%2E+2017%2E+Summarunner%3A+A+recurrent+neural+network+based+sequence+model+for+extractive+summarization+of+documents%2E) [our card](https://md.research.elamur.ai/free/en/papers/nallapati-2016-summarunner-recurrent-neural-network-based/)
26. Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016a. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+Caglar+Gulcehre%2C+Bing+Xiang%2C+et+al%2E+2016a%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+rnns+and+beyond%2E+arXiv+preprint+arXiv%3A1602%2E06023%2E)
27. Ramesh Nallapati, Bowen Zhou, and Mingbo Ma. 2016b. Classify or select: Neural architectures for extractive document summarization. arXiv preprint arXiv:1611.04244. — [arXiv:1611.04244](https://arxiv.org/abs/1611.04244) [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+and+Mingbo+Ma%2E+2016b%2E+Classify+or+select%3A+Neural+architectures+for+extractive+document+summarization%2E+arXiv+preprint+arXiv%3A1611%2E04244%2E+%E2%80%94+%5BarXiv%3A1611%2E04244%5D%28https%3A%2F%2Farxiv%2E)
28. Jun-Ping Ng and Viktoria Abrecht. 2015. Better summarization evaluation with word embeddings for ROUGE. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1925–1930, Lisbon, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jun%2DPing+Ng+and+Viktoria+Abrecht%2E+2015%2E+Better+summarization+evaluation+with+word+embeddings+for+ROUGE%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages)
29. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304. [find](https://scholar.google.com/scholar?q=Romain+Paulus%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2017%2E+A+deep+reinforced+model+for+abstractive+summarization%2E+arXiv+preprint+arXiv%3A1705%2E04304%2E) [our card](https://md.research.elamur.ai/free/en/papers/paulus-2017-deep-reinforced-model-abstractive-summarization/)
30. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
31. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. CoRR, abs/1509.00685. [find](https://scholar.google.com/scholar?q=Alexander+M%2E+Rush%2C+Sumit+Chopra%2C+and+Jason+Weston%2E+2015%2E+A+neural+attention+model+for+abstractive+sentence+summarization%2E+CoRR%2C+abs%2F1509%2E00685%2E)
32. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. CoRR, abs/1704.04368. [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J%2E+Liu%2C+and+Christopher+D%2E+Manning%2E+2017%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+CoRR%2C+abs%2F1704%2E04368%2E)
33. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2015%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+arXiv+preprint+arXiv%3A1508%2E07909%2E)
34. Eva Sharma, Chen Li, and Lu Wang. 2019. Bigpatent: A large-scale dataset for abstractive and coherent summarization. arXiv preprint arXiv:1906.03741. [find](https://scholar.google.com/scholar?q=Eva+Sharma%2C+Chen+Li%2C+and+Lu+Wang%2E+2019%2E+Bigpatent%3A+A+large%2Dscale+dataset+for+abstractive+and+coherent+summarization%2E+arXiv+preprint+arXiv%3A1906%2E03741%2E)
35. Josef Steinberger and Karel Jezek. 2004. Using latent semantic analysis in text summarization and summary evaluation. Proc. ISIM, 4:93–100. [find](https://scholar.google.com/scholar?q=Josef+Steinberger+and+Karel+Jezek%2E+2004%2E+Using+latent+semantic+analysis+in+text+summarization+and+summary+evaluation%2E+Proc%2E+ISIM%2C+4%3A93%E2%80%93100%2E)
36. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. pages 3104–3112. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+pages+3104%E2%80%933112%2E)
37. Lucy Vanderwende, Hisami Suzuki, Chris Brockett, and Ani Nenkova. 2007. Beyond sumbasic: Task-focused summarization with sentence simplification and lexical expansion. Information Processing & Management, 43(6):1606–1618. [find](https://scholar.google.com/scholar?q=Lucy+Vanderwende%2C+Hisami+Suzuki%2C+Chris+Brockett%2C+and+Ani+Nenkova%2E+2007%2E+Beyond+sumbasic%3A+Task%2Dfocused+summarization+with+sentence+simplification+and+lexical+expansion%2E+Information+Processing+%26+Managem)
38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+CoRR%2C+abs%2F1706%2E03762%2E)
39. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. pages 2692–2700. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Meire+Fortunato%2C+and+Navdeep+Jaitly%2E+2015%2E+Pointer+networks%2E+pages+2692%E2%80%932700%2E)
40. Noah Weber, Leena Shekhar, Niranjan Balasubramanian, and Kyunghyun Cho. 2018. Controlling decoding for more abstractive summaries with copy-based networks. arXiv preprint arXiv:1803.07038. — [arXiv:1803.07038](https://arxiv.org/abs/1803.07038) [find](https://scholar.google.com/scholar?q=Noah+Weber%2C+Leena+Shekhar%2C+Niranjan+Balasubramanian%2C+and+Kyunghyun+Cho%2E+2018%2E+Controlling+decoding+for+more+abstractive+summaries+with+copy%2Dbased+networks%2E+arXiv+preprint+arXiv%3A1803%2E07038%2E+%E2%80%94+%5BarXiv%3A18)
41. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In ICML. [find](https://scholar.google.com/scholar?q=Jingqing+Zhang%2C+Yao+Zhao%2C+Mohammad+Saleh%2C+and+Peter+J+Liu%2E+2020%2E+Pegasus%3A+Pre%2Dtraining+with+extracted+gap%2Dsentences+for+abstractive+summarization%2E+In+ICML%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/)