---
title: "ETC: Encoding Long and Structured Inputs in — Ainslie 2020"
description: "Standard Transformers usually cap inputs at about 512 tokens, because the cost of attention grows quadratically with sequence length."
authors: ["Joshua Ainslie", "Santiago Ontañón", "Chris Alberti", "Vaclav Cvicek", "Zachary Fisher", "Philip Pham", "Anirudh Ravula", "Sumit Sanghai", "Qifan Wang", "Li Yang"]
year: 2020
venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/2020.emnlp-main.19/"
doi: "10.18653/v1/2020.emnlp-main.19"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ainslie-2020-etc-encoding-long-structured-inputs/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ainslie-2020-etc-encoding-long-structured-inputs/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/) / ETC: Encoding Long and Structured Inputs in Transformers

# ETC: Encoding Long and Structured Inputs in Transformers

**Authors:** [Joshua Ainslie](https://md.research.elamur.ai/free/en/authors/joshua-ainslie/), [Santiago Ontañón](https://md.research.elamur.ai/free/en/authors/santiago-ontanon/), [Chris Alberti](https://md.research.elamur.ai/free/en/authors/chris-alberti/), Vaclav Cvicek, Zachary Fisher, [Philip Pham](https://md.research.elamur.ai/free/en/authors/philip-pham/), [Anirudh Ravula](https://md.research.elamur.ai/free/en/authors/anirudh-ravula/), [Sumit Sanghai](https://md.research.elamur.ai/free/en/authors/sumit-sanghai/), [Qifan Wang](https://md.research.elamur.ai/free/en/authors/qifan-wang/), [Li Yang](https://md.research.elamur.ai/free/en/authors/li-yang/) · **Year:** 2020 · **Venue:** Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-main.19/) · [PDF](https://aclanthology.org/2020.emnlp-main.19.pdf) · [DOI 10.18653/v1/2020.emnlp-main.19](https://doi.org/10.18653/v1/2020.emnlp-main.19) · [OpenAlex](https://openalex.org/W3105238007)

**Citations:** 268 citations (openalex-snapshot · 2026-10-14) · [109](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Standard Transformers usually cap inputs at about 512 tokens, because the cost of attention grows quadratically with sequence length. They also have no built-in way to represent hierarchy such as sentences, paragraphs, unordered documents or DOM trees. ETC (Extended Transformer Construction), an encoder-only architecture from Google Research, addresses both problems.

The model splits its input into a long sequence of ordinary tokens and a much smaller global sequence of auxiliary tokens. Long tokens look only at neighbours within a fixed radius plus the global tokens, so cost scales linearly with length. Relative position encodings, following Shaw et al., double as edge labels that express structural links between tokens. A Contrastive Predictive Coding (CPC) pre-training task works like masked language modelling at sentence granularity and teaches the global tokens to act as summaries. ETC can also be initialised from BERT or RoBERTa weights.

On Natural Questions, HotpotQA, WikiHop and OpenKP, single ETC models reached first place on several official leaderboards. This shows that both longer context and explicit structure improve results.

## Key points
- **Attention design:** global-local attention is split into four parts: g2g, g2l, l2g and l2l. Only the l2l part is restricted to a local radius, and inputs of up to 4096 tokens were used.
- **Official leaderboard results** (single models, beating ensemble entries):
  - 1st on NQ long answer (0.7778)
  - 1st on HotpotQA Sup. F1 (0.8909)
  - 1st on WikiHop (0.8225)
  - 1st on OpenKP (0.4205)
  - 3rd on HotpotQA overall and 18th on NQ short answer
- **NQ dev set:** initialising from RoBERTa gave 0.782 / 0.585 (long / short answer), compared with 0.754 / 0.593 for RikiNet. ETC led on long answers but still trailed on short answers.
- **HotpotQA ablation:** dropping CPC and hard g2l masking lowered dev scores from 0.751 / 0.869 to 0.722 / 0.857. On OpenKP, adding visual features gave the largest single improvement.
- **Code and related work:** code is released at goo.gle/research-etc-model. The related BigBird work studies ETC combined with random sparse attention.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

109 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 105 not yet.

- [Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering](https://md.research.elamur.ai/free/en/papers/kim-2021-which-linguist-invented-lightbulb-presupposition/)
- [COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining](https://md.research.elamur.ai/free/en/papers/meng-2021-coco-lm-correcting-contrasting-text/)
- [FNet: Mixing Tokens with Fourier Transforms](https://md.research.elamur.ai/free/en/papers/thorp-2022-fnet-mixing-tokens-fourier-transforms/)
- [InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory](https://md.research.elamur.ai/free/en/papers/xiao-2024-infllm-training-free-long-context/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ainslie2020etc,
  title = {{ETC: Encoding Long and Structured Inputs in Transformers}},
  author = {Ainslie, Joshua and Ontañón, Santiago and Alberti, Chris and Cvicek, Vaclav and Fisher, Zachary and Pham, Philip and Ravula, Anirudh and Sanghai, Sumit and Wang, Qifan and Yang, Li},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.19},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.19}
}
```

- [← previous Estimating POS Annotation Consistency of Different Treebanks in a Language](https://md.research.elamur.ai/free/en/papers/aggarwal-2020-estimating-pos-annotation-consistency-different/)
- [next → Evaluating Amharic Machine Translation](https://md.research.elamur.ai/free/en/papers/hadgu-2020-evaluating-amharic-machine-translation/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/)

## References

44 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Chris Alberti, Kenton Lee, and Michael Collins. 2019. A bert baseline for the natural questions. arXiv preprint arXiv:1901.08634. — [arXiv:1901.08634](https://arxiv.org/abs/1901.08634) [find](https://scholar.google.com/scholar?q=Chris+Alberti%2C+Kenton+Lee%2C+and+Michael+Collins%2E+2019%2E+A+bert+baseline+for+the+natural+questions%2E+arXiv+preprint+arXiv%3A1901%2E08634%2E+%E2%80%94+%5BarXiv%3A1901%2E08634%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1901%2E08634%29)
2. Alexandr Andoni, Piotr Indyk, Thijs Laarhoven, Ilya Razenshteyn, and Ludwig Schmidt. 2015. Practical and optimal lsh for angular distance. In Advances in neural information processing systems, pages 1225– 1233. [find](https://scholar.google.com/scholar?q=Alexandr+Andoni%2C+Piotr+Indyk%2C+Thijs+Laarhoven%2C+Ilya+Razenshteyn%2C+and+Ludwig+Schmidt%2E+2015%2E+Practical+and+optimal+lsh+for+angular+distance%2E+In+Advances+in+neural+information+processing+systems%2C+pages+1)
3. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150v1. [find](https://scholar.google.com/scholar?q=Iz+Beltagy%2C+Matthew+E%2E+Peters%2C+and+Arman+Cohan%2E+2020%2E+Longformer%3A+The+long%2Ddocument+transformer%2E+arXiv+preprint+arXiv%3A2004%2E05150v1%2E)
4. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509. [find](https://scholar.google.com/scholar?q=Rewon+Child%2C+Scott+Gray%2C+Alec+Radford%2C+and+Ilya+Sutskever%2E+2019%2E+Generating+long+sequences+with+sparse+transformers%2E+arXiv+preprint+arXiv%3A1904%2E10509%2E)
5. Gonçalo M Correia, Vlad Niculae, and André FT Martins. 2019. Adaptively sparse transformers. arXiv preprint arXiv:1909.00015. [find](https://scholar.google.com/scholar?q=Gonc%CC%A7alo+M+Correia%2C+Vlad+Niculae%2C+and+Andre%CC%81+FT+Martins%2E+2019%2E+Adaptively+sparse+transformers%2E+arXiv+preprint+arXiv%3A1909%2E00015%2E) [our card](https://md.research.elamur.ai/free/en/papers/correia-2019-adaptively-sparse-transformers/)
6. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860. [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Quoc+V+Le%2C+and+Ruslan+Salakhutdinov%2E+2019%2E+Transformer%2Dxl%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+arXiv+preprint+arXiv%3A1901%2E028)
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2E)
8. Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. 2017. The reversible residual network: Backpropagation without storing activations. In Advances in neural information processing systems, pages 2214–2224. [find](https://scholar.google.com/scholar?q=Aidan+N+Gomez%2C+Mengye+Ren%2C+Raquel+Urtasun%2C+and+Roger+B+Grosse%2E+2017%2E+The+reversible+residual+network%3A+Backpropagation+without+storing+activations%2E+In+Advances+in+neural+information+processing+systems%2C)
9. Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019. Star-transformer. arXiv preprint arXiv:1902.09113. [find](https://scholar.google.com/scholar?q=Qipeng+Guo%2C+Xipeng+Qiu%2C+Pengfei+Liu%2C+Yunfan+Shao%2C+Xiangyang+Xue%2C+and+Zheng+Zhang%2E+2019%2E+Star%2Dtransformer%2E+arXiv+preprint+arXiv%3A1902%2E09113%2E)
10. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. [find](https://scholar.google.com/scholar?q=Aaron+van+den+Oord%2C+Yazhe+Li%2C+and+Oriol+Vinyals%2E+2018%2E+Representation+learning+with+contrastive+predictive+coding%2E+arXiv+preprint+arXiv%3A1807%2E03748%2E) [our card](https://md.research.elamur.ai/free/en/papers/oord-2018-representation-learning-contrastive-predictive-coding/)
11. Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. 2018. Music transformer: Generating music with long-term structure. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Cheng%2DZhi+Anna+Huang%2C+Ashish+Vaswani%2C+Jakob+Uszkoreit%2C+Ian+Simon%2C+Curtis+Hawthorne%2C+Noam+Shazeer%2C+Andrew+M+Dai%2C+Matthew+D+Hoffman%2C+Monica+Dinculescu%2C+and+Douglas+Eck%2E+2018%2E+Music+transformer%3A+Generati)
12. Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. 2019. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507. [find](https://scholar.google.com/scholar?q=Jack+W+Rae%2C+Anna+Potapenko%2C+Siddhant+M+Jayakumar%2C+and+Timothy+P+Lillicrap%2E+2019%2E+Compressive+transformers+for+long%2Drange+sequence+modelling%2E+arXiv+preprint+arXiv%3A1911%2E05507%2E) [our card](https://md.research.elamur.ai/free/en/papers/rae-2019-compressive-transformers-long-range-sequence/)
13. Mandar Joshi, Omer Levy, Daniel S Weld, and Luke Zettlemoyer. 2019. Bert for coreference resolution: Baselines and analysis. arXiv preprint arXiv:1908.09091. [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Omer+Levy%2C+Daniel+S+Weld%2C+and+Luke+Zettlemoyer%2E+2019%2E+Bert+for+coreference+resolution%3A+Baselines+and+analysis%2E+arXiv+preprint+arXiv%3A1908%2E09091%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2019-bert-coreference-resolution-baselines-analysis/)
14. Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451. [find](https://scholar.google.com/scholar?q=Nikita+Kitaev%2C+%C5%81ukasz+Kaiser%2C+and+Anselm+Levskaya%2E+2020%2E+Reformer%3A+The+efficient+transformer%2E+arXiv+preprint+arXiv%3A2001%2E04451%2E) [our card](https://md.research.elamur.ai/free/en/papers/kitaev-2020-reformer-efficient-transformer/)
15. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Tom+Kwiatkowski%2C+Jennimaria+Palomaki%2C+Olivia+Redfield%2C+Michael+Collins%2C+Ankur+Parikh%2C+Chris+Alberti%2C+Danielle+Epstein%2C+Illia+Polosukhin%2C+Jacob+Devlin%2C+Kenton+Lee%2C+et+al%2E+2019%2E+Natural+questions%3A+a+ben) [our card](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
16. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683. [find](https://scholar.google.com/scholar?q=Guokun+Lai%2C+Qizhe+Xie%2C+Hanxiao+Liu%2C+Yiming+Yang%2C+and+Eduard+Hovy%2E+2017%2E+Race%3A+Large%2Dscale+reading+comprehension+dataset+from+examinations%2E+arXiv+preprint+arXiv%3A1704%2E04683%2E) [our card](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/)
17. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2019%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+arXiv+preprint+arXiv%3A1)
18. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J+Liu%2E+2019%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext)
19. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+Squad%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+arXiv+preprint+arXiv%3A1606%2E05250%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
20. Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020. Leveraging pre-trained checkpoints for sequence generation tasks. Transactions of the Association for Computational Linguistics, 8:264–280. [find](https://scholar.google.com/scholar?q=Sascha+Rothe%2C+Shashi+Narayan%2C+and+Aliaksei+Severyn%2E+2020%2E+Leveraging+pre%2Dtrained+checkpoints+for+sequence+generation+tasks%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%3A264%E2%80%93280%2E) [our card](https://md.research.elamur.ai/free/en/papers/rothe-2019-leveraging-pre-trained-checkpoints-sequence/)
21. Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2020. Efficient content-based sparse attention with routing transformers. arXiv preprint arXiv:2003.05997. [find](https://scholar.google.com/scholar?q=Aurko+Roy%2C+Mohammad+Saffar%2C+Ashish+Vaswani%2C+and+David+Grangier%2E+2020%2E+Efficient+content%2Dbased+sparse+attention+with+routing+transformers%2E+arXiv+preprint+arXiv%3A2003%2E05997%2E) [our card](https://md.research.elamur.ai/free/en/papers/roy-2020-efficient-content-based-sparse-attention/)
22. Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61–80. [find](https://scholar.google.com/scholar?q=Franco+Scarselli%2C+Marco+Gori%2C+Ah+Chung+Tsoi%2C+Markus+Hagenbuchner%2C+and+Gabriele+Monfardini%2E+2008%2E+The+graph+neural+network+model%2E+IEEE+Transactions+on+Neural+Networks%2C+20%281%29%3A61%E2%80%9380%2E)
23. Peter Shaw, Philip Massey, Angelica Chen, Francesco Piccinno, and Yasemin Altun. 2019. Generating logical forms from graph representations of text and entities. arXiv preprint arXiv:1905.08407. [find](https://scholar.google.com/scholar?q=Peter+Shaw%2C+Philip+Massey%2C+Angelica+Chen%2C+Francesco+Piccinno%2C+and+Yasemin+Altun%2E+2019%2E+Generating+logical+forms+from+graph+representations+of+text+and+entities%2E+arXiv+preprint+arXiv%3A1905%2E08407%2E)
24. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155. [find](https://scholar.google.com/scholar?q=Peter+Shaw%2C+Jakob+Uszkoreit%2C+and+Ashish+Vaswani%2E+2018%2E+Self%2Dattention+with+relative+position+representations%2E+arXiv+preprint+arXiv%3A1803%2E02155%2E) [our card](https://md.research.elamur.ai/free/en/papers/shaw-2018-self-attention-relative-position-representations/)
25. Dayiheng Liu, Yeyun Gong, Jie Fu, Yu Yan, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, and Nan Duan. 2020. Rikinet: Reading wikipedia pages for natural question answering. arXiv preprint arXiv:2004.14560. [find](https://scholar.google.com/scholar?q=Dayiheng+Liu%2C+Yeyun+Gong%2C+Jie+Fu%2C+Yu+Yan%2C+Jiusheng+Chen%2C+Daxin+Jiang%2C+Jiancheng+Lv%2C+and+Nan+Duan%2E+2020%2E+Rikinet%3A+Reading+wikipedia+pages+for+natural+question+answering%2E+arXiv+preprint+arXiv%3A2004%2E14560)
26. Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. arXiv preprint arXiv:1801.10198. [find](https://scholar.google.com/scholar?q=Peter+J+Liu%2C+Mohammad+Saleh%2C+Etienne+Pot%2C+Ben+Goodrich%2C+Ryan+Sepassi%2C+Lukasz+Kaiser%2C+and+Noam+Shazeer%2E+2018%2E+Generating+wikipedia+by+summarizing+long+sequences%2E+arXiv+preprint+arXiv%3A1801%2E10198%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2018-generating-wikipedia-summarizing-long-sequences/)
27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
28. Sameen Maruf, André FT Martins, and Gholamreza Haffari. 2019. Selective attention for context-aware neural machine translation. arXiv preprint arXiv:1903.08788. [find](https://scholar.google.com/scholar?q=Sameen+Maruf%2C+Andre%CC%81+FT+Martins%2C+and+Gholamreza+Haffari%2E+2019%2E+Selective+attention+for+context%2Daware+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1903%2E08788%2E) [our card](https://md.research.elamur.ai/free/en/papers/maruf-2019-selective-attention-context-aware-neural/)
29. Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. 2018. Document-level neural machine translation with hierarchical attention networks. arXiv preprint arXiv:1809.01576. [find](https://scholar.google.com/scholar?q=Lesly+Miculicich%2C+Dhananjay+Ram%2C+Nikolaos+Pappas%2C+and+James+Henderson%2E+2018%2E+Document%2Dlevel+neural+machine+translation+with+hierarchical+attention+networks%2E+arXiv+preprint+arXiv%3A1809%2E01576%2E) [our card](https://md.research.elamur.ai/free/en/papers/miculicich-2018-document-level-neural-machine-translation/)
30. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. arXiv preprint arXiv:1804.04235. [find](https://scholar.google.com/scholar?q=Noam+Shazeer+and+Mitchell+Stern%2E+2018%2E+Adafactor%3A+Adaptive+learning+rates+with+sublinear+memory+cost%2E+arXiv+preprint+arXiv%3A1804%2E04235%2E)
31. Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang. 2018. Bi-directional block self-attention for fast and memory-efficient sequence modeling. arXiv preprint arXiv:1804.00857. — [arXiv:1804.00857](https://arxiv.org/abs/1804.00857) [find](https://scholar.google.com/scholar?q=Tao+Shen%2C+Tianyi+Zhou%2C+Guodong+Long%2C+Jing+Jiang%2C+and+Chengqi+Zhang%2E+2018%2E+Bi%2Ddirectional+block+self%2Dattention+for+fast+and+memory%2Defficient+sequence+modeling%2E+arXiv+preprint+arXiv%3A1804%2E00857%2E+%E2%80%94+%5BarXiv)
32. Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799. [find](https://scholar.google.com/scholar?q=Sainbayar+Sukhbaatar%2C+Edouard+Grave%2C+Piotr+Bojanowski%2C+and+Armand+Joulin%2E+2019%2E+Adaptive+attention+span+in+transformers%2E+arXiv+preprint+arXiv%3A1905%2E07799%2E) [our card](https://md.research.elamur.ai/free/en/papers/sukhbaatar-2019-adaptive-attention-span-transformers/)
33. Si Sun, Chenyan Xiong, Zhenghao Liu, Zhiyuan Liu, and Jie Bao. 2020. Joint keyphrase chunking and salience ranking with bert. arXiv preprint arXiv:2004.13639. — [arXiv:2004.13639](https://arxiv.org/abs/2004.13639) [find](https://scholar.google.com/scholar?q=Si+Sun%2C+Chenyan+Xiong%2C+Zhenghao+Liu%2C+Zhiyuan+Liu%2C+and+Jie+Bao%2E+2020%2E+Joint+keyphrase+chunking+and+salience+ranking+with+bert%2E+arXiv+preprint+arXiv%3A2004%2E13639%2E+%E2%80%94+%5BarXiv%3A2004%2E13639%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+neural+information+processi)
35. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R+Bowman%2E+2018%2E+Glue%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+arXiv+preprint+arXi) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
36. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287–302. [find](https://scholar.google.com/scholar?q=Johannes+Welbl%2C+Pontus+Stenetorp%2C+and+Sebastian+Riedel%2E+2018%2E+Constructing+datasets+for+multi%2Dhop+reading+comprehension+across+documents%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C) [our card](https://md.research.elamur.ai/free/en/papers/welbl-2018-constructing-datasets-multi-hop-reading/)
37. Lee Xiong, Chuan Hu, Chenyan Xiong, Daniel Campos, and Arnold Overwijk. 2019. Open domain web keyphrase extraction beyond language modeling. arXiv preprint arXiv:1911.02671. [find](https://scholar.google.com/scholar?q=Lee+Xiong%2C+Chuan+Hu%2C+Chenyan+Xiong%2C+Daniel+Campos%2C+and+Arnold+Overwijk%2E+2019%2E+Open+domain+web+keyphrase+extraction+beyond+language+modeling%2E+arXiv+preprint+arXiv%3A1911%2E02671%2E)
38. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5754–5764. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Russ+R+Salakhutdinov%2C+and+Quoc+V+Le%2E+2019%2E+Xlnet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+In+Advances+in+neural+inform) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
39. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Peng+Qi%2C+Saizheng+Zhang%2C+Yoshua+Bengio%2C+William+W+Cohen%2C+Ruslan+Salakhutdinov%2C+and+Christopher+D+Manning%2E+2018%2E+Hotpotqa%3A+A+dataset+for+diverse%2C+explainable+multi%2Dhop+question+answering%2E+) [our card](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)
40. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 14… [find](https://scholar.google.com/scholar?q=Zichao+Yang%2C+Diyi+Yang%2C+Chris+Dyer%2C+Xiaodong+He%2C+Alex+Smola%2C+and+Eduard+Hovy%2E+2016%2E+Hierarchical+attention+networks+for+document+classification%2E+In+Proceedings+of+the+2016+conference+of+the+North+Amer) [our card](https://md.research.elamur.ai/free/en/papers/yang-2016-hierarchical-attention-networks-document-classification/)
41. Zihao Ye, Qipeng Guo, Quan Gan, Xipeng Qiu, and Zheng Zhang. 2019. Bp-transformer: Modelling long-range context via binary partitioning. arXiv preprint arXiv:1911.04070. [find](https://scholar.google.com/scholar?q=Zihao+Ye%2C+Qipeng+Guo%2C+Quan+Gan%2C+Xipeng+Qiu%2C+and+Zheng+Zhang%2E+2019%2E+Bp%2Dtransformer%3A+Modelling+long%2Drange+context+via+binary+partitioning%2E+arXiv+preprint+arXiv%3A1911%2E04070%2E)
42. Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2019. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yang+You%2C+Jing+Li%2C+Sashank+Reddi%2C+Jonathan+Hseu%2C+Sanjiv+Kumar%2C+Srinadh+Bhojanapalli%2C+Xiaodan+Song%2C+James+Demmel%2C+Kurt+Keutzer%2C+and+Cho%2DJui+Hsieh%2E+2019%2E+Large+batch+optimization+for+deep+learning%3A+Trai)
43. Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062. [find](https://scholar.google.com/scholar?q=Manzil+Zaheer%2C+Guru+Guruganesh%2C+Avinava+Dubey%2C+Joshua+Ainslie%2C+Chris+Alberti%2C+Santiago+Ontan%CC%83o%CC%81n%2C+Philip+Pham%2C+Anirudh+Ravula%2C+Qifan+Wang%2C+Li+Yang%2C+et+al%2E+2020%2E+Big+bird%3A+Transformers+for+longer+seque) [our card](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/)
44. Xingxing Zhang, Furu Wei, and Ming Zhou. 2019. Hibert: Document level pre-training of hierarchical bidirectional transformers for document summarization. arXiv preprint arXiv:1905.06566. [find](https://scholar.google.com/scholar?q=Xingxing+Zhang%2C+Furu+Wei%2C+and+Ming+Zhou%2E+2019%2E+Hibert%3A+Document+level+pre%2Dtraining+of+hierarchical+bidirectional+transformers+for+document+summarization%2E+arXiv+preprint+arXiv%3A1905%2E06566%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2019-hibert-document-level-pre-training/)