---
title: "PRIMERA: Pyramid-based Masked Sentence — Xiao 2022"
description: "Multi-document summarization condenses a cluster of related texts into one summary."
authors: ["Wen Xiao", "Iz Beltagy", "Giuseppe Carenini", "Arman Cohan"]
year: 2022
venue: "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2022.acl-long.360/"
doi: "10.18653/v1/2022.acl-long.360"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xiao-2022-primera-pyramid-based-masked-sentence/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xiao-2022-primera-pyramid-based-masked-sentence/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization

# PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization

**Authors:** [Wen Xiao](https://md.research.elamur.ai/free/en/authors/wen-xiao/), [Iz Beltagy](https://md.research.elamur.ai/free/en/authors/iz-beltagy/), [Giuseppe Carenini](https://md.research.elamur.ai/free/en/authors/giuseppe-carenini/), [Arman Cohan](https://md.research.elamur.ai/free/en/authors/arman-cohan/) · **Year:** 2022 · **Venue:** Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.acl-long.360/) · [PDF](https://aclanthology.org/2022.acl-long.360.pdf) · [DOI 10.18653/v1/2022.acl-long.360](https://doi.org/10.18653/v1/2022.acl-long.360) · [OpenAlex](https://openalex.org/W4285171765)

**Citations:** 122 citations (openalex-snapshot · 2026-10-14) · [82](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Multi-document summarization condenses a cluster of related texts into one summary. Leading systems often depend on graph-based or hierarchical designs tailored to particular datasets, or need extra inputs such as AMR or discourse graphs. Meanwhile, existing pretrained models were built for single documents or cannot generate text (e.g., CDLM). PRIMERA addresses this with a general pretrained model. It joins all documents in a cluster into one long sequence, inserts \<doc-sep> tokens between them and gives those tokens global attention, then processes the input with the Longformer-Encoder-Decoder (LED). For pretraining, the authors adapt PEGASUS's Gap Sentence Generation and add a new way to choose sentences, called Entity Pyramid. This strategy draws on Pyramid Evaluation and prefers sentences built around entities that recur across many documents. The model must then regenerate those masked sentences as a pseudo-summary. Tested on six datasets spanning three domains, PRIMERA beat both pretrained and dataset-specific baselines in most zero-shot, few-shot and fully supervised settings. This makes strong multi-document summarization possible with little labeled data and no custom architecture.

## Key points
- The model is built on LED, whose cost grows linearly with input length. LED was chosen over BigBird-PEGASUS because it allows global attention on individual tokens mid-sequence and has pretrained (not only fine-tuned) checkpoints available.
- Entity Pyramid treats frequently shared entities in the cluster like Summary Content Units. This avoids a weakness of PEGASUS's "Principle" ROUGE-based selection, which tends to favor exact duplicate sentences such as repeated quotes.
- In low-resource settings, PRIMERA improves on the previous state of the art by up to 5 ROUGE-1 points with only 10 training examples.
- Human evaluation, including on DUC2007, compared PRIMERA with LED and PEGASUS. PRIMERA scored best on content and fluency, though PEGASUS matched it on grammaticality.
- Code and pretrained models are released at github.com/allenai/PRIMER. The authors flag the lack of length control in zero-shot use and possible non-factual output as limitations.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

82 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 81 not yet.

- [EpiGator: An Event-based Surveillance System for Infectious Disease Outbreaks](https://md.research.elamur.ai/free/en/papers/wu-2026-epigator-event-based-surveillance-system/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xiao2022primera,
  title = {{PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization}},
  author = {Xiao, Wen and Beltagy, Iz and Carenini, Giuseppe and Cohan, Arman},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.360},
  url = {https://doi.org/10.18653/v1/2022.acl-long.360}
}
```

- [← previous Predicting the Proficiency Level of Nonnative Hebrew Authors](https://md.research.elamur.ai/free/en/papers/lingbulk-2022-nguyen-predicting-proficiency-level-nonnative-hebrew/)
- [next → Probing Classifiers: Promises, Shortcomings, and Advances](https://md.research.elamur.ai/free/en/papers/belinkov-2022-probing-classifiers-promises-shortcomings-advances/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

38 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettlemoyer. 2021. HTLM: hyper-text pre-training and prompting of language models. CoRR, abs/2107.06955. — [arXiv:2107.06955](https://arxiv.org/abs/2107.06955) [find](https://scholar.google.com/scholar?q=Armen+Aghajanyan%2C+Dmytro+Okhonko%2C+Mike+Lewis%2C+Mandar+Joshi%2C+Hu+Xu%2C+Gargi+Ghosh%2C+and+Luke+Zettlemoyer%2E+2021%2E+HTLM%3A+hyper%2Dtext+pre%2Dtraining+and+prompting+of+language+models%2E+CoRR%2C+abs%2F2107%2E06955%2E+%E2%80%94+%5BarX)
2. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. CoRR, abs/2004.05150. [find](https://scholar.google.com/scholar?q=Iz+Beltagy%2C+Matthew+E%2E+Peters%2C+and+Arman+Cohan%2E+2020%2E+Longformer%3A+The+long%2Ddocument+transformer%2E+CoRR%2C+abs%2F2004%2E05150%2E)
3. Jonathan Bragg, Arman Cohan, Kyle Lo, and Iz Beltagy. 2021. Flex: Unifying evaluation for few-shot nlp. arXiv preprint arXiv:2107.07170. — [arXiv:2107.07170](https://arxiv.org/abs/2107.07170) [find](https://scholar.google.com/scholar?q=Jonathan+Bragg%2C+Arman+Cohan%2C+Kyle+Lo%2C+and+Iz+Beltagy%2E+2021%2E+Flex%3A+Unifying+evaluation+for+few%2Dshot+nlp%2E+arXiv+preprint+arXiv%3A2107%2E07170%2E+%E2%80%94+%5BarXiv%3A2107%2E07170%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2107%2E07170%29)
4. Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew Peters, Arie Cattan, and Ido Dagan. 2021. CDLM: Cross-document language modeling. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2648–2662, Punta Cana, Dominican Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Avi+Caciularu%2C+Arman+Cohan%2C+Iz+Beltagy%2C+Matthew+Peters%2C+Arie+Cattan%2C+and+Ido+Dagan%2E+2021%2E+CDLM%3A+Cross%2Ddocument+language+modeling%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+20)
5. Janara Christensen, Mausam, Stephen Soderland, and Oren Etzioni. 2013. Towards coherent multi-document summarization. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1163–1173, Atlanta, Georgia.… [find](https://scholar.google.com/scholar?q=Janara+Christensen%2C+Mausam%2C+Stephen+Soderland%2C+and+Oren+Etzioni%2E+2013%2E+Towards+coherent+multi%2Ddocument+summarization%2E+In+Proceedings+of+the+2013+Conference+of+the+North+American+Chapter+of+the+Associa)
6. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational… [find](https://scholar.google.com/scholar?q=Arman+Cohan%2C+Franck+Dernoncourt%2C+Doo+Soon+Kim%2C+Trung+Bui%2C+Seokhwan+Kim%2C+Walter+Chang%2C+and+Nazli+Goharian%2E+2018%2E+A+discourse%2Daware+attention+model+for+abstractive+summarization+of+long+documents%2E+In+Pr) [our card](https://md.research.elamur.ai/free/en/papers/cohan-2018-discourse-aware-attention-model-abstractive/)
7. Hoa Trang Dang. 2005. Overview of duc 2005. In Proceedings of the document understanding conference, volume 2005, pages 1–12. [find](https://scholar.google.com/scholar?q=Hoa+Trang+Dang%2E+2005%2E+Overview+of+duc+2005%2E+In+Proceedings+of+the+document+understanding+conference%2C+volume+2005%2C+pages+1%E2%80%9312%2E)
8. Hoa Trang Dang and Karolina Owczarzak. 2008. Overview of the tac 2008 update summarization task. Theory and Applications of Categories. [find](https://scholar.google.com/scholar?q=Hoa+Trang+Dang+and+Karolina+Owczarzak%2E+2008%2E+Overview+of+the+tac+2008+update+summarization+task%2E+Theory+and+Applications+of+Categories%2E)
9. Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074–1084, Florence, Italy.… [find](https://scholar.google.com/scholar?q=Alexander+Fabbri%2C+Irene+Li%2C+Tianwei+She%2C+Suyi+Li%2C+and+Dragomir+Radev%2E+2019%2E+Multi%2Dnews%3A+A+large%2Dscale+multi%2Ddocument+summarization+dataset+and+abstractive+hierarchical+model%2E+In+Proceedings+of+the+57t) [our card](https://md.research.elamur.ai/free/en/papers/fabbri-2019-multi-news-large-scale-multi/)
10. Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098–4109, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sebastian+Gehrmann%2C+Yuntian+Deng%2C+and+Alexander+Rush%2E+2018%2E+Bottom%2Dup+abstractive+summarization%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+4098%E2%80%934) [our card](https://md.research.elamur.ai/free/en/papers/gehrmann-2018-bottom-up-abstractive-summarization/)
11. Demian Gholipour Ghalandari, Chris Hokamp, Nghia The Pham, John Glover, and Georgiana Ifrim. 2020. A large-scale multi-document summarization dataset from the Wikipedia current events portal. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1302–1308,… [find](https://scholar.google.com/scholar?q=Demian+Gholipour+Ghalandari%2C+Chris+Hokamp%2C+Nghia+The+Pham%2C+John+Glover%2C+and+Georgiana+Ifrim%2E+2020%2E+A+large%2Dscale+multi%2Ddocument+summarization+dataset+from+the+Wikipedia+current+events+portal%2E+In+Proce)
12. Travis Goodwin, Max Savery, and Dina Demner-Fushman. 2020. Flight of the PEGASUS? comparing transformers on few-shot and zero-shot multi-document abstractive summarization. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5640–5646, Barcelona, Spain (Online). I… [find](https://scholar.google.com/scholar?q=Travis+Goodwin%2C+Max+Savery%2C+and+Dina+Demner%2DFushman%2E+2020%2E+Flight+of+the+PEGASUS%3F+comparing+transformers+on+few%2Dshot+and+zero%2Dshot+multi%2Ddocument+abstractive+summarization%2E+In+Proceedings+of+the+28th+)
13. Xiaotao Gu, Yuning Mao, Jiawei Han, Jialu Liu, You Wu, Cong Yu, Daniel Finnie, Hongkun Yu, Jiaqi Zhai, and Nicholas Zukoski. 2020. Generating representative headlines for news stories. In Proceedings of The Web Conference 2020, WWW ’20, page 1773–1784, New York, NY, USA. Association for Computing Ma… [find](https://scholar.google.com/scholar?q=Xiaotao+Gu%2C+Yuning+Mao%2C+Jiawei+Han%2C+Jialu+Liu%2C+You+Wu%2C+Cong+Yu%2C+Daniel+Finnie%2C+Hongkun+Yu%2C+Jiaqi+Zhai%2C+and+Nicholas+Zukoski%2E+2020%2E+Generating+representative+headlines+for+news+stories%2E+In+Proceedings+)
14. Chris Hokamp, Demian Gholipour Ghalandari, Nghia The Pham, and John Glover. 2020. Dyne: Dynamic ensemble decoding for multi-document summarization. CoRR, abs/2006.08748. — [arXiv:2006.08748](https://arxiv.org/abs/2006.08748) [find](https://scholar.google.com/scholar?q=Chris+Hokamp%2C+Demian+Gholipour+Ghalandari%2C+Nghia+The+Pham%2C+and+John+Glover%2E+2020%2E+Dyne%3A+Dynamic+ensemble+decoding+for+multi%2Ddocument+summarization%2E+CoRR%2C+abs%2F2006%2E08748%2E+%E2%80%94+%5BarXiv%3A2006%2E08748%5D%28https%3A%2F%2Fa)
15. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python. [find](https://scholar.google.com/scholar?q=Matthew+Honnibal%2C+Ines+Montani%2C+Sofie+Van+Lan%2Ddeghem%2C+and+Adriane+Boyd%2E+2020%2E+spaCy%3A+Industrial%2Dstrength+Natural+Language+Processing+in+Python%2E)
16. Hanqi Jin, Tianming Wang, and Xiaojun Wan. 2020. Multi-granularity interaction network for extractive and abstractive multi-document summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6244– 6254, Online. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Hanqi+Jin%2C+Tianming+Wang%2C+and+Xiaojun+Wan%2E+2020%2E+Multi%2Dgranularity+interaction+network+for+extractive+and+abstractive+multi%2Ddocument+summarization%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Ass)
17. Dan Jurafsky and James H. Martin. 2009. Speech and language processing : an introduction to natural language processing, computational linguistics, and speech recognition. Pearson Prentice Hall, Upper Saddle River, N.J. [find](https://scholar.google.com/scholar?q=Dan+Jurafsky+and+James+H%2E+Martin%2E+2009%2E+Speech+and+language+processing+%3A+an+introduction+to+natural+language+processing%2C+computational+linguistics%2C+and+speech+recognition%2E+Pearson+Prentice+Hall%2C+Upper)
18. Yoav Levine, Barak Lenz, Opher Lieber, Omri Abend, Kevin Leyton-Brown, Moshe Tennenholtz, and Yoav Shoham. 2020. Pmi-masking: Principled masking of correlated spans. arXiv preprint arXiv:2010.01825. — [arXiv:2010.01825](https://arxiv.org/abs/2010.01825) [find](https://scholar.google.com/scholar?q=Yoav+Levine%2C+Barak+Lenz%2C+Opher+Lieber%2C+Omri+Abend%2C+Kevin+Leyton%2DBrown%2C+Moshe+Tennenholtz%2C+and+Yoav+Shoham%2E+2020%2E+Pmi%2Dmasking%3A+Principled+masking+of+correlated+spans%2E+arXiv+preprint+arXiv%3A2010%2E01825%2E+%E2%80%94)
19. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Yinhan+Liu%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Abdelrahman+Mohamed%2C+Omer+Levy%2C+Veselin+Stoyanov%2C+and+Luke+Zettlemoyer%2E+2020%2E+BART%3A+Denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+l) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
20. Wei Li, Xinyan Xiao, Jiachen Liu, Hua Wu, Haifeng Wang, and Junping Du. 2020. Leveraging graph to improve abstractive multi-document summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6232–6243, Online. Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Wei+Li%2C+Xinyan+Xiao%2C+Jiachen+Liu%2C+Hua+Wu%2C+Haifeng+Wang%2C+and+Junping+Du%2E+2020%2E+Leveraging+graph+to+improve+abstractive+multi%2Ddocument+summarization%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Ass)
21. Kexin Liao, Logan Lebanoff, and Fei Liu. 2018. Abstract Meaning Representation for multi-document summarization. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1178–1190, Santa Fe, New Mexico, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kexin+Liao%2C+Logan+Lebanoff%2C+and+Fei+Liu%2E+2018%2E+Abstract+Meaning+Representation+for+multi%2Ddocument+summarization%2E+In+Proceedings+of+the+27th+International+Conference+on+Computational+Linguistics%2C+pages) [our card](https://md.research.elamur.ai/free/en/papers/liao-2018-abstract-meaning-representation-multi-document/)
22. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+ROUGE%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+Summarization+Branches+Out%2C+pages+74%E2%80%9381%2C+Barcelona%2C+Spain%2E+Association+for+Computational+Linguistics%2E)
23. Peter J. Liu*, Mohammad Saleh*, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Peter+J%2E+Liu%2A%2C+Mohammad+Saleh%2A%2C+Etienne+Pot%2C+Ben+Goodrich%2C+Ryan+Sepassi%2C+Lukasz+Kaiser%2C+and+Noam+Shazeer%2E+2018%2E+Generating+wikipedia+by+summarizing+long+sequences%2E+In+International+Conference+on+Learn) [our card](https://md.research.elamur.ai/free/en/papers/liu-2018-generating-wikipedia-summarizing-long-sequences/)
24. Yang Liu and Mirella Lapata. 2019a. Hierarchical transformers for multi-document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5070–5081, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yang+Liu+and+Mirella+Lapata%2E+2019a%2E+Hierarchical+transformers+for+multi%2Ddocument+summarization%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+5070%E2%80%935) [our card](https://md.research.elamur.ai/free/en/papers/liu-2019-hierarchical-transformers-multi-document-summarization/)
25. Yang Liu and Mirella Lapata. 2019b. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730–3740, Hong Kong, China. Asso… [find](https://scholar.google.com/scholar?q=Yang+Liu+and+Mirella+Lapata%2E+2019b%2E+Text+summarization+with+pretrained+encoders%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+International+Joi)
26. Yao Lu, Yue Dong, and Laurent Charlin. 2020. Multi-XScience: A large-scale dataset for extreme multi-document summarization of scientific articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8068–8074, Online. Association for Computationa… [find](https://scholar.google.com/scholar?q=Yao+Lu%2C+Yue+Dong%2C+and+Laurent+Charlin%2E+2020%2E+Multi%2DXScience%3A+A+large%2Dscale+dataset+for+extreme+multi%2Ddocument+summarization+of+scientific+articles%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+M)
27. Yuning Mao, Yanru Qu, Yiqing Xie, Xiang Ren, and Jiawei Han. 2020. Multi-document summarization with maximal marginal relevance-guided reinforcement learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1737–1751, Online. Association for C… [find](https://scholar.google.com/scholar?q=Yuning+Mao%2C+Yanru+Qu%2C+Yiqing+Xie%2C+Xiang+Ren%2C+and+Jiawei+Han%2E+2020%2E+Multi%2Ddocument+summarization+with+maximal+marginal+relevance%2Dguided+reinforcement+learning%2E+In+Proceedings+of+the+2020+Conference+on+)
28. Ani Nenkova and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 145–152, Boston, Massachusetts… [find](https://scholar.google.com/scholar?q=Ani+Nenkova+and+Rebecca+Passonneau%2E+2004%2E+Evaluating+content+selection+in+summarization%3A+The+pyramid+method%2E+In+Proceedings+of+the+Human+Language+Technology+Conference+of+the+North+American+Chapter+of)
29. Ramakanth Pasunuru, Mengwen Liu, Mohit Bansal, Sujith Ravi, and Markus Dreyer. 2021. Efficiently summarizing text and graph encodings of multi-document clusters. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn… [find](https://scholar.google.com/scholar?q=Ramakanth+Pasunuru%2C+Mengwen+Liu%2C+Mohit+Bansal%2C+Sujith+Ravi%2C+and+Markus+Dreyer%2E+2021%2E+Efficiently+summarizing+text+and+graph+encodings+of+multi%2Ddocument+clusters%2E+In+Proceedings+of+the+2021+Conference+)
30. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+2020%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtex)
31. Sascha Rothe, Joshua Maynez, and Shashi Narayan. 2021. A thorough evaluation of task-specific pre-training for summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 140–145, Online and Punta Cana, Dominican Republic. Association for Computati… [find](https://scholar.google.com/scholar?q=Sascha+Rothe%2C+Joshua+Maynez%2C+and+Shashi+Narayan%2E+2021%2E+A+thorough+evaluation+of+task%2Dspecific+pre%2Dtraining+for+summarization%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Lang)
32. Andy Su, Difei Su, John M Mulvey, and H Vincent Poor. 2021. Pobrl: Optimizing multi-document summarization by blending reinforcement learning policies. arXiv preprint arXiv:2105.08244. — [arXiv:2105.08244](https://arxiv.org/abs/2105.08244) [find](https://scholar.google.com/scholar?q=Andy+Su%2C+Difei+Su%2C+John+M+Mulvey%2C+and+H+Vincent+Poor%2E+2021%2E+Pobrl%3A+Optimizing+multi%2Ddocument+summarization+by+blending+reinforcement+learning+policies%2E+arXiv+preprint+arXiv%3A2105%2E08244%2E+%E2%80%94+%5BarXiv%3A2105%2E0)
33. Chien-Sheng Wu, Linqing Liu, Wenhao Liu, Pontus Stenetorp, and Caiming Xiong. 2021. Controllable abstractive dialogue summarization with sketch supervision. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5108–5122, Online. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Chien%2DSheng+Wu%2C+Linqing+Liu%2C+Wenhao+Liu%2C+Pontus+Stenetorp%2C+and+Caiming+Xiong%2E+2021%2E+Controllable+abstractive+dialogue+summarization+with+sketch+supervision%2E+In+Findings+of+the+Association+for+Computat)
34. Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek, Krishnan Srinivasan, and Dragomir Radev. 2017. Graph-based neural multi-document summarization. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 452–462, Vancouver, Canada. Association fo… [find](https://scholar.google.com/scholar?q=Michihiro+Yasunaga%2C+Rui+Zhang%2C+Kshitijh+Meelu%2C+Ayush+Pareek%2C+Krishnan+Srinivasan%2C+and+Dragomir+Radev%2E+2017%2E+Graph%2Dbased+neural+multi%2Ddocument+summarization%2E+In+Proceedings+of+the+21st+Conference+on+Co) [our card](https://md.research.elamur.ai/free/en/papers/yasunaga-2017-graph-based-neural-multi-document/)
35. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big bird: Transformers for longer sequences. In Advances in Neural Information Processing Systems, volume 33, pages 17283–17297… [find](https://scholar.google.com/scholar?q=Manzil+Zaheer%2C+Guru+Guruganesh%2C+Kumar+Avinava+Dubey%2C+Joshua+Ainslie%2C+Chris+Alberti%2C+Santiago+Ontanon%2C+Philip+Pham%2C+Anirudh+Ravula%2C+Qifan+Wang%2C+Li+Yang%2C+and+Amr+Ahmed%2E+2020%2E+Big+bird%3A+Transformers+for+) [our card](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/)
36. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 11328–11339. PMLR. [find](https://scholar.google.com/scholar?q=Jingqing+Zhang%2C+Yao+Zhao%2C+Mohammad+Saleh%2C+and+Peter+Liu%2E+2020%2E+PEGASUS%3A+Pre%2Dtraining+with+extracted+gap%2Dsentences+for+abstractive+summarization%2E+In+Proceedings+of+the+37th+International+Conference+on+) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/)
37. Chenguang Zhu, Ziyi Yang, Robert Gmyr, Michael Zeng, and Xuedong Huang. 2021. Leveraging Lead Bias for Zero-Shot Abstractive News Summarization, page 1462–1471. Association for Computing Machinery, New York, NY, USA. [find](https://scholar.google.com/scholar?q=Chenguang+Zhu%2C+Ziyi+Yang%2C+Robert+Gmyr%2C+Michael+Zeng%2C+and+Xuedong+Huang%2E+2021%2E+Leveraging+Lead+Bias+for+Zero%2DShot+Abstractive+News+Summarization%2C+page+1462%E2%80%931471%2E+Association+for+Computing+Machinery%2C+Ne)
38. Yanyan Zou, Xingxing Zhang, Wei Lu, Furu Wei, and Ming Zhou. 2020. Pre-training for abstractive document summarization by reinstating source text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3646–3660, Online. Association for Computational… [find](https://scholar.google.com/scholar?q=Yanyan+Zou%2C+Xingxing+Zhang%2C+Wei+Lu%2C+Furu+Wei%2C+and+Ming+Zhou%2E+2020%2E+Pre%2Dtraining+for+abstractive+document+summarization+by+reinstating+source+text%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Me)