---
title: "HIBERT: Document Level Pre-training of — Zhang 2019"
description: "Neural extractive summarizers usually rely on a hierarchical document encoder trained on sentence labels."
authors: ["Xingxing Zhang", "Furu Wei", "Ming Zhou"]
year: 2019
venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P19-1499/"
doi: "10.18653/v1/p19-1499"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2019-hibert-document-level-pre-training/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2019-hibert-document-level-pre-training/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization

# HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization

**Authors:** [Xingxing Zhang](https://md.research.elamur.ai/free/en/authors/xingxing-zhang/), [Furu Wei](https://md.research.elamur.ai/free/en/authors/furu-wei/), [Ming Zhou](https://md.research.elamur.ai/free/en/authors/ming-zhou/) · **Year:** 2019 · **Venue:** Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P19-1499/) · [PDF](https://aclanthology.org/P19-1499.pdf) · [DOI 10.18653/v1/p19-1499](https://doi.org/10.18653/v1/p19-1499) · [OpenAlex](https://openalex.org/W2962785754)

**Citations:** 377 citations (openalex-snapshot · 2026-10-14) · [69](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Neural extractive summarizers usually rely on a hierarchical document encoder trained on sentence labels. These labels are derived heuristically, for example by greedily maximizing ROUGE against reference summaries, so they are noisy, and complex encoders overfit to them quickly. The authors introduce HIBERT, a two-level Transformer: a sentence encoder builds a vector for each sentence, and a document encoder then contextualizes those vectors. They pre-train it without labels by masking whole sentences and having a decoder regenerate them, which is a document-level counterpart of BERT's masked-word objective. Pre-training runs in two stages, first open-domain on GIGA-CM and then in-domain on the summarization training data. The pre-trained encoder is then fine-tuned to classify sentences for extraction. On CNN/Dailymail and NYT50 it beats an identical model trained from random initialization, outperforms a fine-tuned BERT baseline, and reached state-of-the-art results at the time. The work shows that document encoders, and not only word or sentence encoders, benefit from self-supervised pre-training.

## Key points
- Pre-training improves results over the randomly initialized HeriTransformer by +1.25 ROUGE on CNN/Dailymail and by +2.0 ROUGE on NYT50.
- On the NYT50 test set, HIBERT M reaches ROUGE-1/2/L of 49.47/30.11/41.63, compared with 48.38/29.04/40.53 for BERT and 47.44/28.08/39.56 for HeriTransformer.
- HIBERT S has 54.6M parameters, half of BERT base's 110M, yet beats BERT by about 0.4–0.5 ROUGE on CNNDM.
- Combining open-domain (GIGA-CM) and in-domain pre-training works better than either stage alone, with 43.19 ROUGE-1 on CNNDM validation.
- In a human ranking study on 20 CNNDM test documents, HIBERT M was ranked best in 30% of cases. Its mean rank of 2.15 was second only to the human reference at 1.85, and it was significantly better than Lead3, DCA, Latent and BERT.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

69 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 62 not yet.

- [ETC: Encoding Long and Structured Inputs in Transformers](https://md.research.elamur.ai/free/en/papers/ainslie-2020-etc-encoding-long-structured-inputs/)
- [Re-evaluating Evaluation in Text Summarization](https://md.research.elamur.ai/free/en/papers/bhandari-2020-re-evaluating-evaluation-text-summarization/)
- [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/)
- [Recursively Summarizing Books with Human Feedback](https://md.research.elamur.ai/free/en/papers/wu-2021-recursively-summarizing-books-human-feedback/)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/)
- [Knowledge-Grounded Dialogue Generation with Pre-trained Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2020-knowledge-grounded-dialogue-generation-pre/)
- [Extractive Summarization as Text Matching](https://md.research.elamur.ai/free/en/papers/zhong-2020-extractive-summarization-as-text-matching/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2019hibert,
  title = {{HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization}},
  author = {Zhang, Xingxing and Wei, Furu and Zhou, Ming},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1499},
  url = {https://doi.org/10.18653/v1/p19-1499}
}
```

- [← previous Head-First Linearization with Tree-Structured Representation](https://md.research.elamur.ai/free/en/papers/yu-2019-head-first-linearization-tree-structured/)
- [next → Hierarchical Transformers for Multi-Document Summarization](https://md.research.elamur.ai/free/en/papers/liu-2019-hierarchical-transformers-multi-document-summarization/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/)

## References

45 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450. [find](https://scholar.google.com/scholar?q=Jimmy+Lei+Ba%2C+Jamie+Ryan+Kiros%2C+and+Geoffrey+E+Hinton%2E+2016%2E+Layer+normalization%2E+arXiv+preprint+arXiv%3A1607%2E06450%2E) [our card](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/)
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In In Proceedings of the 3rd International Conference on Learning Representations, San Diego, California. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+In+Proceedings+of+the+3rd+International+Conference+on+Learning+Repre) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
3. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. [find](https://scholar.google.com/scholar?q=Piotr+Bojanowski%2C+Edouard+Grave%2C+Armand+Joulin%2C+and+Tomas+Mikolov%2E+2017%2E+Enriching+word+vectors+with+subword+information%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+5%3A135%E2%80%93146%2E)
4. Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. 2018. Deep communicating agents for abstractive summarization. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), p… [find](https://scholar.google.com/scholar?q=Asli+Celikyilmaz%2C+Antoine+Bosselut%2C+Xiaodong+He%2C+and+Yejin+Choi%2E+2018%2E+Deep+communicating+agents+for+abstractive+summarization%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+t) [our card](https://md.research.elamur.ai/free/en/papers/celikyilmaz-2018-deep-communicating-agents-abstractive-summarization/)
5. Xiuying Chen, Shen Gao, Chongyang Tao, Yan Song, Dongyan Zhao, and Rui Yan. 2018. Iterative document representation learning towards summarization with polishing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4088–4097. Association for Computational… [find](https://scholar.google.com/scholar?q=Xiuying+Chen%2C+Shen+Gao%2C+Chongyang+Tao%2C+Yan+Song%2C+Dongyan+Zhao%2C+and+Rui+Yan%2E+2018%2E+Iterative+document+representation+learning+towards+summarization+with+polishing%2E+In+Proceedings+of+the+2018+Conference)
6. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675–686. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yen%2DChun+Chen+and+Mohit+Bansal%2E+2018%2E+Fast+abstractive+summarization+with+reinforce%2Dselected+sentence+rewriting%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguist) [our card](https://md.research.elamur.ai/free/en/papers/chen-2018-fast-abstractive-summarization-reinforce-selected/)
7. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 484–494, Berlin, Germany. [find](https://scholar.google.com/scholar?q=Jianpeng+Cheng+and+Mirella+Lapata%2E+2016%2E+Neural+summarization+by+extracting+sentences+and+words%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+L) [our card](https://md.research.elamur.ai/free/en/papers/cheng-2016-neural-summarization-extracting-sentences-words/)
8. John M Conroy and Dianne P O’leary. 2001. Text summarization via hidden markov models. In Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, pages 406–407. ACM. [find](https://scholar.google.com/scholar?q=John+M+Conroy+and+Dianne+P+O%E2%80%99leary%2E+2001%2E+Text+summarization+via+hidden+markov+models%2E+In+Proceedings+of+the+24th+annual+international+ACM+SIGIR+conference+on+Research+and+development+in+information+r)
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+BERT%3A+Pre%2Dtraining+of+Deep+Bidirectional+Transformers+for+Language+Understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2E)
10. Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. Banditsum: Extractive summarization as a contextual bandit. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3739–3748. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yue+Dong%2C+Yikang+Shen%2C+Eric+Crawford%2C+Herke+van+Hoof%2C+and+Jackie+Chi+Kit+Cheung%2E+2018%2E+Banditsum%3A+Extractive+summarization+as+a+contextual+bandit%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Me)
11. Greg Durrett, Taylor Berg-Kirkpatrick, and Dan Klein. 2016. Learning-based single-document summarization with compression and anaphoricity constraints. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1998–2008. Association for… [find](https://scholar.google.com/scholar?q=Greg+Durrett%2C+Taylor+Berg%2DKirkpatrick%2C+and+Dan+Klein%2E+2016%2E+Learning%2Dbased+single%2Ddocument+summarization+with+compression+and+anaphoricity+constraints%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the) [our card](https://md.research.elamur.ai/free/en/papers/durrett-2016-learning-based-single-document-summarization/)
12. Elena Filatova and Vasileios Hatzivassiloglou. 2004a. Event-based extractive summarization. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 104–111, Barcelona, Spain. [find](https://scholar.google.com/scholar?q=Elena+Filatova+and+Vasileios+Hatzivassiloglou%2E+2004a%2E+Event%2Dbased+extractive+summarization%2E+In+Text+Summarization+Branches+Out%3A+Proceedings+of+the+ACL%2D04+Workshop%2C+pages+104%E2%80%93111%2C+Barcelona%2C+Spain%2E)
13. Elena Filatova and Vasileios Hatzivassiloglou. 2004b. Event-based extractive summarization. [find](https://scholar.google.com/scholar?q=Elena+Filatova+and+Vasileios+Hatzivassiloglou%2E+2004b%2E+Event%2Dbased+extractive+summarization%2E)
14. Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098–4109. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sebastian+Gehrmann%2C+Yuntian+Deng%2C+and+Alexander+Rush%2E+2018%2E+Bottom%2Dup+abstractive+summarization%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+4098%E2%80%934) [our card](https://md.research.elamur.ai/free/en/papers/gehrmann-2018-bottom-up-abstractive-summarization/)
15. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 315– 323. [find](https://scholar.google.com/scholar?q=Xavier+Glorot%2C+Antoine+Bordes%2C+and+Yoshua+Bengio%2E+2011%2E+Deep+sparse+rectifier+neural+networks%2E+In+Proceedings+of+the+fourteenth+international+conference+on+artificial+intelligence+and+statistics%2C+page)
16. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1631–1640. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Jiatao+Gu%2C+Zhengdong+Lu%2C+Hang+Li%2C+and+Victor+O%2EK%2E+Li%2E+2016%2E+Incorporating+copying+mechanism+in+sequence%2Dto%2Dsequence+learning%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computati) [our card](https://md.research.elamur.ai/free/en/papers/gu-2016-incorporating-copying-mechanism-sequence-sequence/)
17. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+2016%2E+Deep+residual+learning+for+image+recognition%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pages+770%E2%80%93+)
18. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693– 1701. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Karl+Moritz+Hermann%2C+Tomas+Kocisky%2C+Edward+Grefenstette%2C+Lasse+Espeholt%2C+Will+Kay%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+2015%2E+Teaching+machines+to+read+and+comprehend%2E+In+Advances+in+Neural+Information) [our card](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/)
19. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2E)
20. Wan-Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun. 2018. A unified model for extractive and abstractive summarization using inconsistency loss. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 132–14… [find](https://scholar.google.com/scholar?q=Wan%2DTing+Hsu%2C+Chieh%2DKai+Lin%2C+Ming%2DYing+Lee%2C+Kerui+Min%2C+Jing+Tang%2C+and+Min+Sun%2E+2018%2E+A+unified+model+for+extractive+and+abstractive+summarization+using+inconsistency+loss%2E+In+Proceedings+of+the+56th+A) [our card](https://md.research.elamur.ai/free/en/papers/hsu-2018-unified-model-extractive-abstractive-summarization/)
21. Julian Kupiec, Jan Pedersen, and Francine Chen. 1995. A trainable document summarizer. In Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, pages 68–73. ACM. [find](https://scholar.google.com/scholar?q=Julian+Kupiec%2C+Jan+Pedersen%2C+and+Francine+Chen%2E+1995%2E+A+trainable+document+summarizer%2E+In+Proceedings+of+the+18th+annual+international+ACM+SIGIR+conference+on+Research+and+development+in+information+r)
22. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74–81, Barcelona, Spain. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+Rouge%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+Summarization+Branches+Out%3A+Proceedings+of+the+ACL%2D04+Workshop%2C+pages+74%E2%80%9381%2C+Barcelona%2C+Spain%2E)
23. Inderjeet Mani. 2001. Automatic Summarization. John Benjamins Pub Co. [find](https://scholar.google.com/scholar?q=Inderjeet+Mani%2E+2001%2E+Automatic+Summarization%2E+John+Benjamins+Pub+Co%2E)
24. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, pages 55–60. [find](https://scholar.google.com/scholar?q=Christopher+Manning%2C+Mihai+Surdeanu%2C+John+Bauer%2C+Jenny+Finkel%2C+Steven+Bethard%2C+and+David+McClosky%2E+2014%2E+The+stanford+corenlp+natural+language+processing+toolkit%2E+In+Proceedings+of+52nd+annual+meeting) [our card](https://md.research.elamur.ai/free/en/papers/manning-2014-stanford-corenlp-natural-language-processing/)
25. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Greg+S+Corrado%2C+and+Jeff+Dean%2E+2013%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+Advances+in+neural+information+processing+)
26. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In In Proceedings of the 31st AAAI Conference on Artificial Intelligence, pages 3075–3091, San Francisco, California. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Feifei+Zhai%2C+and+Bowen+Zhou%2E+2017%2E+Summarunner%3A+A+recurrent+neural+network+based+sequence+model+for+extractive+summarization+of+documents%2E+In+In+Proceedings+of+the+31st+AAAI+Conferen) [our card](https://md.research.elamur.ai/free/en/papers/nallapati-2016-summarunner-recurrent-neural-network-based/)
27. Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+Caglar+Gulcehre%2C+Bing+Xiang%2C+et+al%2E+2016%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+rnns+and+beyond%2E+arXiv+preprint+arXiv%3A1602%2E06023%2E)
28. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Paper… [find](https://scholar.google.com/scholar?q=Shashi+Narayan%2C+Shay+B%2E+Cohen%2C+and+Mirella+Lapata%2E+2018%2E+Ranking+sentences+for+extractive+summarization+with+reinforcement+learning%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter) [our card](https://md.research.elamur.ai/free/en/papers/narayan-2018-ranking-sentences-extractive-summarization-reinforcement/)
29. Ani Nenkova and Kathleen McKeown. 2011. Automatic summarization. Foundations and Trends in Information Retrieval, 5(2–3):103–233. [find](https://scholar.google.com/scholar?q=Ani+Nenkova+and+Kathleen+McKeown%2E+2011%2E+Automatic+summarization%2E+Foundations+and+Trends+in+Information+Retrieval%2C+5%282%E2%80%933%29%3A103%E2%80%93233%2E)
30. Ani Nenkova, Lucy Vanderwende, and Kathleen McKeown. 2006. A compositional context sensitive multi-document summarizer: exploring the factors that influence summarization. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages… [find](https://scholar.google.com/scholar?q=Ani+Nenkova%2C+Lucy+Vanderwende%2C+and+Kathleen+McKeown%2E+2006%2E+A+compositional+context+sensitive+multi%2Ddocument+summarizer%3A+exploring+the+factors+that+influence+summarization%2E+In+Proceedings+of+the+29th+a)
31. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304. [find](https://scholar.google.com/scholar?q=Romain+Paulus%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2017%2E+A+deep+reinforced+model+for+abstractive+summarization%2E+arXiv+preprint+arXiv%3A1705%2E04304%2E) [our card](https://md.research.elamur.ai/free/en/papers/paulus-2017-deep-reinforced-model-abstractive-summarization/)
32. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+2014+conference+on+empirical+methods+in+natural+language+process)
33. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+the+2018+Conference+of+t)
34. Dragomir Radev, Timothy Allison, Sasha Blair-Goldensohn, John Blitzer, Arda Çelebi, Stanko Dimitrov, Elliott Drabek, Ali Hakim, Wai Lam, Danyu Liu, Jahna Otterbacher, Hong Qi, Horacio Saggion, Simone Teufel, Michael Topper, Adam Winkel, and Zhu Zhang. 2004. Mead - a platform for multidocument multi… [find](https://scholar.google.com/scholar?q=Dragomir+Radev%2C+Timothy+Allison%2C+Sasha+Blair%2DGoldensohn%2C+John+Blitzer%2C+Arda+C%CC%A7elebi%2C+Stanko+Dimitrov%2C+Elliott+Drabek%2C+Ali+Hakim%2C+Wai+Lam%2C+Danyu+Liu%2C+Jahna+Otterbacher%2C+Hong+Qi%2C+Horacio+Saggion%2C+Simone) [our card](https://md.research.elamur.ai/free/en/papers/radev-2004-mead-platform-multidocument-multilingual-text/)
35. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL [https://s3-us-west-2.amazonaws](https://s3-us-west-2.amazonaws). com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf. [↗](https://s3-us-west-2.amazonaws)
36. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada. [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J%2E+Liu%2C+and+Christopher+D%2E+Manning%2E+2017%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computati)
37. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Ling)
38. Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5027–5038. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Emma+Strubell%2C+Patrick+Verga%2C+Daniel+Andor%2C+David+Weiss%2C+and+Andrew+McCallum%2E+2018%2E+Linguistically%2Dinformed+self%2Dattention+for+semantic+role+labeling%2E+In+Proceedings+of+the+2018+Conference+on+Empirica) [our card](https://md.research.elamur.ai/free/en/papers/strubell-2018-linguistically-informed-self-attention-semantic/)
39. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+neural+information+processing+systems%2C+pages+3104%E2%80%933112%2E)
40. Wilson L Taylor. 1953. cloze procedure: A new tool for measuring readability. Journalism Bulletin, 30(4):415–433. [find](https://scholar.google.com/scholar?q=Wilson+L+Taylor%2E+1953%2E+cloze+procedure%3A+A+new+tool+for+measuring+readability%2E+Journalism+Bulletin%2C+30%284%29%3A415%E2%80%93433%2E)
41. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+Neural+Information+Processi)
42. Kristian Woodsend and Mirella Lapata. 2010. Automatic generation of story highlights. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 565–574, Uppsala, Sweden. [find](https://scholar.google.com/scholar?q=Kristian+Woodsend+and+Mirella+Lapata%2E+2010%2E+Automatic+generation+of+story+highlights%2E+In+Proceedings+of+the+48th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+565%E2%80%93574%2C+Uppsala)
43. Jiacheng Xu and Greg Durrett. 2019. Neural extractive text summarization with syntactic compression. arXiv preprint arXiv:1902.00863. [find](https://scholar.google.com/scholar?q=Jiacheng+Xu+and+Greg+Durrett%2E+2019%2E+Neural+extractive+text+summarization+with+syntactic+compression%2E+arXiv+preprint+arXiv%3A1902%2E00863%2E)
44. Xingxing Zhang, Mirella Lapata, Furu Wei, and Ming Zhou. 2018. Neural latent extractive document summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 779–784. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xingxing+Zhang%2C+Mirella+Lapata%2C+Furu+Wei%2C+and+Ming+Zhou%2E+2018%2E+Neural+latent+extractive+document+summarization%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processin)
45. Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018. Neural document summarization by jointly learning to score and select sentences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 654–663. Associ… [find](https://scholar.google.com/scholar?q=Qingyu+Zhou%2C+Nan+Yang%2C+Furu+Wei%2C+Shaohan+Huang%2C+Ming+Zhou%2C+and+Tiejun+Zhao%2E+2018%2E+Neural+document+summarization+by+jointly+learning+to+score+and+select+sentences%2E+In+Proceedings+of+the+56th+Annual+Mee) [our card](https://md.research.elamur.ai/free/en/papers/zhou-2018-neural-document-summarization-jointly-learning/)