---
title: "Longformer: The Long-Document Transformer — Beltagy 2020"
description: "The authors propose Longformer, an architecture in which full attention is replaced by a combination of two… 60 references, cited by 1,737 papers here."
authors: ["Iz Beltagy", "Matthew E. Peters", "Arman Cohan"]
year: 2020
venue: "arXiv:2004.05150"
source: "https://arxiv.org/abs/2004.05150"
doi: "10.48550/arxiv.2004.05150"
arxiv: "2004.05150"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/) / Longformer: The Long-Document Transformer

# Longformer: The Long-Document Transformer

**Authors:** [Iz Beltagy](https://md.research.elamur.ai/free/en/authors/iz-beltagy/), [Matthew E. Peters](https://md.research.elamur.ai/free/en/authors/matthew-peters/) and [Arman Cohan](https://md.research.elamur.ai/free/en/authors/arman-cohan/) · **Year:** 2020 · **Venue:** arXiv:2004.05150 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2004.05150) · [PDF](https://arxiv.org/pdf/2004.05150) · [DOI 10.48550/arxiv.2004.05150](https://doi.org/10.48550/arxiv.2004.05150) · [arXiv 2004.05150](https://arxiv.org/abs/2004.05150)

**Citations:** [1737](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
The standard Transformer handles long texts poorly. Computation and memory for self-attention grow quadratically with sequence length. As a result, models such as BERT are limited to 512 tokens, and longer documents have to be truncated or split into chunks. The authors propose Longformer, an architecture in which full attention is replaced by a combination of two mechanisms. The first is a local sliding window, including its sparse (dilated) variant. The second is global attention at a few positions chosen for the specific task. Thanks to this, costs grow only linearly. The new mechanism can be built into an already trained model. The authors continued MLM pretraining from the RoBERTa checkpoint and then fine-tuned the model on classification, QA, and coreference resolution. For generative tasks with long inputs, they created LED, an encoder-decoder variant initialized from BART. At the time of publication, Longformer achieved the best results on text8, enwik8, WikiHop, and TriviaQA, and LED on summarization of arXiv papers. Thus, documents of thousands of tokens can be processed in a single pass, without cumbersome task-specific architectures.

## Key points
- In character-level language modeling, the model processes sequences of up to 32K characters on ordinary GPUs and achieves state-of-the-art results on text8 and enwik8.
- In the ablation on WikiHop (dev), the base model with length 4,096 has an accuracy of 73.8. Removing global attention and its separate projections drops accuracy to 65.5 (−8.3). Restricting to 512 tokens with full attention yields 71.7, meaning the gain is not explained by additional pretraining alone.
- In LED, the positional embeddings are extended from 1K to 16K tokens: BART's 1K embeddings were copied 16 times. There are two versions: LED-base (6 layers each in the encoder and decoder) and LED-large (12 each).
- On the arXiv dataset, where the 90th percentile of document length is 14.5K tokens, LED-large (16,384) achieved ROUGE-1/2/L of 46.63 / 19.62 / 41.83. This is slightly better than BigBird (4,096) and Pegasus.
- The code and models are open-sourced by the Allen Institute for AI: github.com/allenai/longformer. The authors note that BigBird later improved results on the leaderboards, but its pretraining required roughly 16 times more compute.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cite

Cite as: Iz Beltagy, Matthew E. Peters and Arman Cohan. 2020. Longformer: The Long-Document Transformer. arXiv:2004.05150.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{beltagy2020longformer,
  title = {{Longformer: The Long-Document Transformer}},
  author = {Iz Beltagy, Matthew E. Peters and Arman Cohan},
  year = {2020},
  doi = {10.48550/arxiv.2004.05150},
  eprint = {2004.05150},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2004.05150}
}
```

- [← previous Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/)
- [next → Language (Technology) is Power: A Critical Survey of “Bias” in NLP](https://md.research.elamur.ai/free/en/papers/blodgett-2020-language-technology-is-power-critical/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/)

## References

60 references, 34 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020. ETC: Encoding long and structured inputs in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processi… [our card](https://md.research.elamur.ai/free/en/papers/ainslie-2020-etc-encoding-long-structured-inputs/)
2. Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. 2018. Character-level language modeling with deeper self-attention. In AAAI. [find](https://scholar.google.com/scholar?q=Rami+Al%2DRfou%2C+Dokook+Choe%2C+Noah+Constant%2C+Mandy+Guo%2C+and+Llion+Jones%2E+2018%2E+Character%2Dlevel+language+modeling+with+deeper+self%2Dattention%2E+In+AAAI%2E)
3. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In ACL. [our card](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/)
4. Jifan Chen, Shih-Ting Lin, and Greg Durrett. 2019. Multi-hop question answering via reasoning chains. arXiv preprint, abs/1910.02610. — [arXiv:1910.02610](https://arxiv.org/abs/1910.02610) [find](https://scholar.google.com/scholar?q=Jifan+Chen%2C+Shih%2DTing+Lin%2C+and+Greg+Durrett%2E+2019%2E+Multi%2Dhop+question+answering+via+reasoning+chains%2E+arXiv+preprint%2C+abs%2F1910%2E02610%2E+%E2%80%94+%5BarXiv%3A1910%2E02610%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1910%2E02610%29)
5. Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. 2018. TVM: An automated end-to-end optimizing compiler for deep learning. In OSDI. [our card](https://md.research.elamur.ai/free/en/papers/chen-2018-tvm-automated-end-end-optimizing/)
6. Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost. arXiv preprint, abs/1604.06174. [find](https://scholar.google.com/scholar?q=Tianqi+Chen%2C+Bing+Xu%2C+Chiyuan+Zhang%2C+and+Carlos+Guestrin%2E+2016%2E+Training+deep+nets+with+sublinear+memory+cost%2E+arXiv+preprint%2C+abs%2F1604%2E06174%2E)
7. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv preprint, abs/1904.10509. [find](https://scholar.google.com/scholar?q=Rewon+Child%2C+Scott+Gray%2C+Alec+Radford%2C+and+Ilya+Sutskever%2E+2019%2E+Generating+long+sequences+with+sparse+transformers%2E+arXiv+preprint%2C+abs%2F1904%2E10509%2E)
8. Christopher Clark and Matt Gardner. 2017. Simple and effective multi-paragraph reading comprehension. In ACL. [our card](https://md.research.elamur.ai/free/en/papers/clark-2017-simple-effective-multi-paragraph-reading/)
9. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does bert look at? an analysis of bert’s attention. arXiv preprint, abs/1906.04341. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Urvashi+Khandelwal%2C+Omer+Levy%2C+and+Christopher+D%2E+Manning%2E+2019%2E+What+does+bert+look+at%3F+an+analysis+of+bert%E2%80%99s+attention%2E+arXiv+preprint%2C+abs%2F1906%2E04341%2E)
10. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In NAACL-HLT 2018. [our card](https://md.research.elamur.ai/free/en/papers/cohan-2018-discourse-aware-attention-model-abstractive/)
11. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In NeurIPS. [find](https://scholar.google.com/scholar?q=Andrew+M+Dai+and+Quoc+V+Le%2E+2015%2E+Semi%2Dsupervised+sequence+learning%2E+In+NeurIPS%2E)
12. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In ACL. [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+Jaime+G%2E+Carbonell%2C+Quoc+V%2E+Le%2C+and+Ruslan+Salakhutdinov%2E+2019%2E+Transformer%2DXL%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+In+ACL%2E)
13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT. [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
14. Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuohang Wang, and Jingjing Liu. 2020. Hierarchical graph network for multi-hop question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8823–8838, Online. Association for Computational L… [find](https://scholar.google.com/scholar?q=Yuwei+Fang%2C+Siqi+Sun%2C+Zhe+Gan%2C+Rohit+Pillai%2C+Shuohang+Wang%2C+and+Jingjing+Liu%2E+2020%2E+Hierarchical+graph+network+for+multi%2Dhop+question+answering%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Meth)
15. Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. [find](https://scholar.google.com/scholar?q=Adam+Fisch%2C+Alon+Talmor%2C+Robin+Jia%2C+Minjoon+Seo%2C+Eunsol+Choi%2C+and+Danqi+Chen%2E+2019%2E)
16. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In MRQA workshop at EMNLP. [our card](https://md.research.elamur.ai/free/en/papers/fisch-2019-mrqa-2019-shared-task-evaluating/)
17. Alexios Gidiotis and Grigorios Tsoumakas. 2020. A divide-and-conquer approach to the summarization of academic articles. ArXiv, abs/2004.06190. — [arXiv:2004.06190](https://arxiv.org/abs/2004.06190) [find](https://scholar.google.com/scholar?q=Alexios+Gidiotis+and+Grigorios+Tsoumakas%2E+2020%2E+A+divide%2Dand%2Dconquer+approach+to+the+summarization+of+academic+articles%2E+ArXiv%2C+abs%2F2004%2E06190%2E+%E2%80%94+%5BarXiv%3A2004%2E06190%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2004%2E06190%29)
18. Michael Glaß, Alfio Massimiliano Gliozzo, Rishav Chakravarti, Anthony Ferritto, Lin Pan, Gaudani Bhargav, Dinesh Garg, and Avirup Sil. 2019. Span selection pre-training for question answering. arXiv preprint, abs/1909.04120. [find](https://scholar.google.com/scholar?q=Michael+Gla%C3%9F%2C+Alfio+Massimiliano+Gliozzo%2C+Rishav+Chakravarti%2C+Anthony+Ferritto%2C+Lin+Pan%2C+Gaudani+Bhargav%2C+Dinesh+Garg%2C+and+Avirup+Sil%2E+2019%2E+Span+selection+pre%2Dtraining+for+question+answering%2E+arXiv+p)
19. Scott Gray, Alec Radford, and Diederik P. Kingma. 2017. Gpu kernels for block-sparse weights. [find](https://scholar.google.com/scholar?q=Scott+Gray%2C+Alec+Radford%2C+and+Diederik+P%2E+Kingma%2E+2017%2E+Gpu+kernels+for+block%2Dsparse+weights%2E)
20. Dirk Groeneveld, Tushar Khot, Mausam, and Ashish Sabhwaral. 2020. A simple yet strong pipeline for HotpotQA. arXiv preprint, abs/2004.06753. [find](https://scholar.google.com/scholar?q=Dirk+Groeneveld%2C+Tushar+Khot%2C+Mausam%2C+and+Ashish+Sabhwaral%2E+2020%2E+A+simple+yet+strong+pipeline+for+HotpotQA%2E+arXiv+preprint%2C+abs%2F2004%2E06753%2E)
21. Ankit Gupta and Jonathan Berant. 2020. Gmat: Global memory augmentation for transformers. ArXiv, abs/2006.03274. — [arXiv:2006.03274](https://arxiv.org/abs/2006.03274) [find](https://scholar.google.com/scholar?q=Ankit+Gupta+and+Jonathan+Berant%2E+2020%2E+Gmat%3A+Global+memory+augmentation+for+transformers%2E+ArXiv%2C+abs%2F2006%2E03274%2E+%E2%80%94+%5BarXiv%3A2006%2E03274%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2006%2E03274%29)
22. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+ACL%2E)
23. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL. [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
24. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In EMNLP-IJCNLP. [our card](https://md.research.elamur.ai/free/en/papers/joshi-2019-bert-coreference-resolution-baselines-analysis/)
25. Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019. SemEval-2019 task 4: Hyperpartisan news detection. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 829–839, Minneapolis, Minnesota, USA… [our card](https://md.research.elamur.ai/free/en/papers/kiesel-2019-semeval-2019-task-4-hyperpartisan/)
26. Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. ICLR. [our card](https://md.research.elamur.ai/free/en/papers/kipf-2016-semi-supervised-classification-graph-convolutional/)
27. Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In ICLR. [our card](https://md.research.elamur.ai/free/en/papers/kitaev-2020-reformer-efficient-transformer/)
28. Olga V. Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of bert. In EMNLP/IJCNLP. [our card](https://md.research.elamur.ai/free/en/papers/kovaleva-2019-revealing-dark-secrets-bert/)
29. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In NAACL. [our card](https://md.research.elamur.ai/free/en/papers/lee-2018-higher-order-coreference-resolution-coarse/)
30. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
31. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint, abs/1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
32. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon… [our card](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/)
33. Matt Mahoney. 2009. Large text compression benchmark. [find](https://scholar.google.com/scholar?q=Matt+Mahoney%2E+2009%2E+Large+text+compression+benchmark%2E)
34. Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. 2016. Wavenet: A generative model for raw audio. In SSW. [our card](https://md.research.elamur.ai/free/en/papers/oord-2016-wavenet-generative-model-raw-audio/)
35. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations. [our card](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/)
36. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL. [our card](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/)
37. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40, Jeju Island, Korea. Association for Computational Linguisti… [our card](https://md.research.elamur.ai/free/en/papers/pradhan-2012-conll-2012-shared-task-modeling/)
38. Jiezhong Qiu, Hao Ma, Omer Levy, Scott Yih, Sinong Wang, and Jie Tang. 2019. Blockwise self-attention for long document understanding. arXiv preprint, abs/1911.02972. [find](https://scholar.google.com/scholar?q=Jiezhong+Qiu%2C+Hao+Ma%2C+Omer+Levy%2C+Scott+Yih%2C+Sinong+Wang%2C+and+Jie+Tang%2E+2019%2E+Blockwise+self%2Dattention+for+long+document+understanding%2E+arXiv+preprint%2C+abs%2F1911%2E02972%2E)
39. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
40. Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In ICLR. [our card](https://md.research.elamur.ai/free/en/papers/rae-2019-compressive-transformers-long-range-sequence/)
41. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, W. Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67. [our card](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)
42. Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2020. Efficient content-based sparse attention with routing transformers. arXiv preprint, abs/2003.05997. [our card](https://md.research.elamur.ai/free/en/papers/roy-2020-efficient-content-based-sparse-attention/)
43. Nan Shao, Yiming Cui, Ting Liu, Shijin Wang, and Guoping Hu. 2020. Is graph structure necessary for multi-hop reasoning? arXiv preprint, abs/2004.03096. — [arXiv:2004.03096](https://arxiv.org/abs/2004.03096) [find](https://scholar.google.com/scholar?q=Nan+Shao%2C+Yiming+Cui%2C+Ting+Liu%2C+Shijin+Wang%2C+and+Guoping+Hu%2E+2020%2E+Is+graph+structure+necessary+for+multi%2Dhop+reasoning%3F+arXiv+preprint%2C+abs%2F2004%2E03096%2E+%E2%80%94+%5BarXiv%3A2004%2E03096%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2004)
44. Sandeep Subramanian, Raymond Li, Jonathan Pilault, and C. Pal. 2020. On extractive and abstractive neural document summarization with transformer language models. In EMNLP. [our card](https://md.research.elamur.ai/free/en/papers/pilault-2020-extractive-abstractive-neural-document-summarization/)
45. Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019. Adaptive attention span in transformers. In ACL. [our card](https://md.research.elamur.ai/free/en/papers/sukhbaatar-2019-adaptive-attention-span-transformers/)
46. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NIPS. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V%2E+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+NIPS%2E)
47. Trieu H. Trinh and Quoc V. Le. 2018. A simple method for commonsense reasoning. arXiv preprint, abs/1806.02847. [find](https://scholar.google.com/scholar?q=Trieu+H%2E+Trinh+and+Quoc+V%2E+Le%2E+2018%2E+A+simple+method+for+commonsense+reasoning%2E+arXiv+preprint%2C+abs%2F1806%2E02847%2E)
48. Ming Tu, Jinke Huang, Xiaodong He, and Bowen Zhou. 2020. Graph sequential network for reasoning over sequences. In NeurIPS Graph Representation Learning workshop. [find](https://scholar.google.com/scholar?q=Ming+Tu%2C+Jinke+Huang%2C+Xiaodong+He%2C+and+Bowen+Zhou%2E+2020%2E+Graph+sequential+network+for+reasoning+over+sequences%2E+In+NeurIPS+Graph+Representation+Learning+workshop%2E)
49. Ming Tu, Kevin Huang, Guangtao Wang, Jing Huang, Xiaodong He, and Bufang Zhou. 2019. Select, answer and explain: Interpretable multi-hop reading comprehension over multiple documents. arXiv preprint, abs/1911.00484. — [arXiv:1911.00484](https://arxiv.org/abs/1911.00484) [find](https://scholar.google.com/scholar?q=Ming+Tu%2C+Kevin+Huang%2C+Guangtao+Wang%2C+Jing+Huang%2C+Xiaodong+He%2C+and+Bufang+Zhou%2E+2019%2E+Select%2C+answer+and+explain%3A+Interpretable+multi%2Dhop+reading+comprehension+over+multiple+documents%2E+arXiv+preprint%2C+)
50. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
51. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. TACL, 6:287–302. [our card](https://md.research.elamur.ai/free/en/papers/welbl-2018-constructing-datasets-multi-hop-reading/)
52. Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convolutions. arXiv preprint, abs/1901.10430. [our card](https://md.research.elamur.ai/free/en/papers/wu-2019-pay-less-attention-lightweight-dynamic/)
53. Qizhe Xie, Zihang Dai, Eduard H. Hovy, Minh-Thang Luong, and Quoc V. Le. 2019. Unsupervised data augmentation for consistency training. arXiv preprint, abs/1904.12848. [our card](https://md.research.elamur.ai/free/en/papers/xie-2019-unsupervised-data-augmentation-consistency-training/)
54. Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shu xin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Li-Wei Wang, and Tie-Yan Liu. 2020. On layer normalization in the transformer architecture. arXiv preprint, abs/2002.04745. [find](https://scholar.google.com/scholar?q=Ruibin+Xiong%2C+Yunchang+Yang%2C+Di+He%2C+Kai+Zheng%2C+Shu+xin+Zheng%2C+Chen+Xing%2C+Huishuai+Zhang%2C+Yanyan+Lan%2C+Li%2DWei+Wang%2C+and+Tie%2DYan+Liu%2E+2020%2E+On+layer+normalization+in+the+transformer+architecture%2E+arXiv+p)
55. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP. [our card](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)
56. Zihao Ye, Qipeng Guo, Quan Gan, Xipeng Qiu, and Zheng Zhang. 2019. BP-Transformer: Modelling long-range context via binary partitioning. arXiv preprint, abs/1911.04070. [find](https://scholar.google.com/scholar?q=Zihao+Ye%2C+Qipeng+Guo%2C+Quan+Gan%2C+Xipeng+Qiu%2C+and+Zheng+Zhang%2E+2019%2E+BP%2DTransformer%3A+Modelling+long%2Drange+context+via+binary+partitioning%2E+arXiv+preprint%2C+abs%2F1911%2E04070%2E)
57. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, C. Alberti, S. Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, L. Yang, and A. Ahmed. 2020. Big bird: Transformers for longer sequences. ArXiv, abs/2007.14062. [our card](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/)
58. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In NeurIPS. [our card](https://md.research.elamur.ai/free/en/papers/zellers-2019-defending-against-neural-fake-news/)
59. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. ICML. [our card](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/)
60. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. ICCV, pages 19–27. [our card](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/)