---
title: "Rethinking Attention with Performers — Choromanski 2020"
description: "The standard Transformer scales poorly to long sequences: the attention mechanism requires time… 71 references, cited by 387 papers here."
authors: ["Krzysztof Choromanski", "Valerii Likhosherstov", "David Dohan", "Xingyou Song", "Andreea Gane", "Tamas Sarlos", "Peter Hawkins", "Jared Davis", "Afroz Mohiuddin", "Lukasz Kaiser", "David Belanger", "Lucy Colwell", "Adrian Weller"]
year: 2020
venue: "arXiv:2009.14794"
source: "https://arxiv.org/abs/2009.14794"
doi: "10.48550/arxiv.2009.14794"
arxiv: "2009.14794"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/choromanski-2020-rethinking-attention-performers/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/choromanski-2020-rethinking-attention-performers/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/) / Rethinking Attention with Performers

# Rethinking Attention with Performers

**Authors:** [Krzysztof Choromanski](https://md.research.elamur.ai/free/en/authors/krzysztof-choromanski/), Valerii Likhosherstov, [David Dohan](https://md.research.elamur.ai/free/en/authors/david-dohan/), Xingyou Song, Andreea Gane, Tamas Sarlos, [Peter Hawkins](https://md.research.elamur.ai/free/en/authors/peter-hawkins/), Jared Davis, Afroz Mohiuddin, [Lukasz Kaiser](https://md.research.elamur.ai/free/en/authors/lukasz-kaiser/), [David Belanger](https://md.research.elamur.ai/free/en/authors/david-belanger/), Lucy Colwell and Adrian Weller · **Year:** 2020 · **Venue:** arXiv:2009.14794 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2009.14794) · [PDF](https://arxiv.org/pdf/2009.14794) · [DOI 10.48550/arxiv.2009.14794](https://doi.org/10.48550/arxiv.2009.14794) · [arXiv 2009.14794](https://arxiv.org/abs/2009.14794) · [OpenAlex](https://openalex.org/W3091156754)

**Citations:** 104 citations (OpenAlex, 2026-10-14) · [387](https://md.research.elamur.ai/free/en/papers/choromanski-2020-rethinking-attention-performers/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The standard Transformer scales poorly to long sequences: the attention mechanism requires time and memory that grow quadratically with the number of tokens L. The authors, from Google, DeepMind, Cambridge and the Alan Turing Institute, propose the Performer. This architecture does not replace softmax attention with a simplified mechanism; instead, it approximates softmax attention itself. The cost grows linearly, and there is no need to assume that the attention matrix is sparse or low-rank. The approach is built on the FAVOR+ method. The softmax kernel is expressed as the expectation of an inner product of random features. These features are always positive and mutually orthogonal, so the estimate remains stable even for small attention values. For the method, the authors prove that the estimate is unbiased or nearly unbiased, that it converges uniformly, and that it has lower variance. The same approach also works with other attention kernels. As a result, the authors were able, for the first time, to compare softmax with alternatives on large-scale tasks that are out of reach for the standard Transformer. The Performer was tested on pixel prediction, text models and protein sequence modeling. The results are competitive with other efficient sparse and dense attention methods. The model is compatible with standard Transformers: a small amount of fine-tuning is enough for the transition.

## Key points
- The complexity of standard attention is O(L²d) in time and O(L² + Ld) in memory. In FAVOR+ it is O(Lrd) and O(Lr + Ld + rd), where r is the number of random features.
- Trigonometric features (sin/cos) can take negative values. This makes the estimate unstable when the true attention values are close to zero, so the authors switched to positive features.
- For the trigonometric orthogonal variant, it is proven that m = Θ(d log d) random projections suffice, and this number does not depend on L. The attention matrix can then be approximated to arbitrary precision in O(Ld² log d) time. For comparison, the LSH approach of Reformer has O(Ld² log L).
- Unidirectional (autoregressive) attention is implemented via prefix sums. FAVOR+ can be combined with reversible layers or clustered attention.
- The code is open source in the google-research repository: the `performer` and `protein_lm` directories.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 430 works · 424,363 citations
- [Columbia University](https://md.research.elamur.ai/free/en/universities/columbia-university/) — University, New York, United States · 241 works · 54,053 citations
- [University of Cambridge](https://md.research.elamur.ai/free/en/universities/university-of-cambridge/) — University, Cambridge, United Kingdom · 243 works · 121,396 citations
- [University of Massachusetts Amherst](https://md.research.elamur.ai/free/en/universities/university-of-massachusetts-amherst/) — University, Amherst Center, United States · 106 works · 66,692 citations
- [Google DeepMind](https://md.research.elamur.ai/free/en/companies/google-deepmind/) — Company, London, United Kingdom · 56 works · 78,880 citations
- [The Alan Turing Institute](https://md.research.elamur.ai/free/en/labs/the-alan-turing-institute/) — Research institute, London, United Kingdom · 29 works · 4,368 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

## Cite

Cite as: Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell and Adrian Weller. 2020. Rethinking Attention with Performers. arXiv:2009.14794.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{choromanski2020rethinking,
  title = {{Rethinking Attention with Performers}},
  author = {Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell and Adrian Weller},
  year = {2020},
  doi = {10.48550/arxiv.2009.14794},
  eprint = {2009.14794},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2009.14794}
}
```

- [← previous CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality](https://md.research.elamur.ai/free/en/papers/yu-2020-ch-sims-chinese-multimodal-sentiment/)
- [next → Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://md.research.elamur.ai/free/en/papers/ravfogel-2020-null-it-out-guarding-protected/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/)

## References

71 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V. Le. Attention augmented convolutional networks. CoRR, abs/1904.09925, 2019. URL http://arxiv.org/abs/1904.09925. [↗](http://arxiv.org/abs/1904.09925)
2. Iz Beltagy, Matthew E. Peters, and Arman Cohan. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). CoRR, abs/2004.05150, 2020. URL https://arxiv.org/abs/2004.05150. [↗](https://arxiv.org/abs/2004.05150)
3. William Chan, Chitwan Saharia, Geoffrey E. Hinton, Mohammad Norouzi, and Navdeep Jaitly. Imputer: Sequence modelling via imputation and dynamic programming. CoRR, abs/2002.08926, 2020. URL https://arxiv.org/abs/2002.08926. — [arXiv:2002.08926](https://arxiv.org/abs/2002.08926) [↗](https://arxiv.org/abs/2002.08926)
4. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. [One billion word benchmark for measuring progress in statistical language modeling](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/). In INTERSPEECH 2014, 15th Annual Conference of the International Speech Communication Association, Singapore, S…
5. Ciprian Chelba, Mia Xu Chen, Ankur Bapna, and Noam Shazeer. Faster transformer decoding: N-gram masked self-attention. CoRR, abs/2001.04589, 2020. URL https://arxiv.org/abs/2001.04589. — [arXiv:2001.04589](https://arxiv.org/abs/2001.04589) [↗](https://arxiv.org/abs/2001.04589)
6. Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George F. Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. The best of both worlds: Combining recent advances in neura… — [The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/chen-2018-best-both-worlds-combining-recent/)
7. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. [Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/). CoRR, abs/1904.10509, 2019. URL http://arxiv.org/abs/1904.10509. [↗](http://arxiv.org/abs/1904.10509)
8. Krzysztof Choromanski, Carlton Downey, and Byron Boots. Initialization matters: Orthogonal predictive state recurrent neural networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. [find](https://scholar.google.com/scholar?q=Krzysztof+Choromanski%2C+Carlton+Downey%2C+and+Byron+Boots%2E+Initialization+matters%3A+Orthogonal+predictive+state+recurrent+neural+networks%2E+In+6th+International+Conference+on+Learning+Representations%2C+ICLR)
9. OpenReview.net, 2018a. URL [https://openreview.net/forum?id=HJJ23bW0b](https://openreview.net/forum?id=HJJ23bW0b). [↗](https://openreview.net/forum?id=HJJ23bW0b)
10. Krzysztof Choromanski, Mark Rowland, Tamás Sarlós, Vikas Sindhwani, Richard E. Turner, and Adrian Weller. The geometry of random features. In International Conference on Artificial Intelligence and Statistics, AISTATS 2018, 9-11 April 2018, Playa Blanca, Lanzarote, Canary Islands, Spain, volume 84 o… [find](https://scholar.google.com/scholar?q=Krzysztof+Choromanski%2C+Mark+Rowland%2C+Tam%C3%A1s+Sarl%C3%B3s%2C+Vikas+Sindhwani%2C+Richard+E%2E+Turner%2C+and+Adrian+Weller%2E+The+geometry+of+random+features%2E+In+International+Conference+on+Artificial+Intelligence+and+St)
11. PMLR, 2018b. URL [http://proceedings.mlr.press/v84/choromanski18a.html](http://proceedings.mlr.press/v84/choromanski18a.html). [↗](http://proceedings.mlr.press/v84/choromanski18a.html)
12. Krzysztof Choromanski, Aldo Pacchiano, Jeffrey Pennington, and Yunhao Tang. KAMA-NNs: Low-dimensional rotation based neural networks. In The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Mach… [find](https://scholar.google.com/scholar?q=Krzysztof+Choromanski%2C+Aldo+Pacchiano%2C+Jeffrey+Pennington%2C+and+Yunhao+Tang%2E+KAMA%2DNNs%3A+Low%2Ddimensional+rotation+based+neural+networks%2E+In+The+22nd+International+Conference+on+Artificial+Intelligence+an)
13. PMLR, 2019a. URL [http://proceedings.mlr.press/v89/choromanski19a.html](http://proceedings.mlr.press/v89/choromanski19a.html). [↗](http://proceedings.mlr.press/v89/choromanski19a.html)
14. Krzysztof Choromanski, Mark Rowland, Wenyu Chen, and Adrian Weller. Unifying orthogonal Monte Carlo methods. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 120… [find](https://scholar.google.com/scholar?q=Krzysztof+Choromanski%2C+Mark+Rowland%2C+Wenyu+Chen%2C+and+Adrian+Weller%2E+Unifying+orthogonal+Monte+Carlo+methods%2E+In+Proceedings+of+the+36th+International+Conference+on+Machine+Learning%2C+ICML+2019%2C+9%2D15+Ju)
15. PMLR, 2019b. URL [http://proceedings.mlr.press/v97/choromanski19a.html](http://proceedings.mlr.press/v97/choromanski19a.html). [↗](http://proceedings.mlr.press/v97/choromanski19a.html)
16. Krzysztof Marcin Choromanski, Mark Rowland, and Adrian Weller. The unreasonable effectiveness of structured random orthogonal embeddings. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA,… [find](https://scholar.google.com/scholar?q=Krzysztof+Marcin+Choromanski%2C+Mark+Rowland%2C+and+Adrian+Weller%2E+The+unreasonable+effectiveness+of+structured+random+orthogonal+embeddings%2E+In+Advances+in+Neural+Information+Processing+Systems+30%3A+Annua)
17. Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. [Fast and accurate deep network learning by exponential linear units (elus)](https://md.research.elamur.ai/free/en/papers/clevert-2015-fast-accurate-deep-network-learning/). In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL [http://arxiv.…](http://arxiv.…) [↗](http://arxiv.…)
18. Qian Cong, Ivan Anishchenko, Sergey Ovchinnikov, and David Baker. Protein interaction networks revealed by proteome coevolution. Science, 365(6449):185–189, 2019. [find](https://scholar.google.com/scholar?q=Qian+Cong%2C+Ivan+Anishchenko%2C+Sergey+Ovchinnikov%2C+and+David+Baker%2E+Protein+interaction+networks+revealed+by+proteome+coevolution%2E+Science%2C+365%286449%29%3A185%E2%80%93189%2C+2019%2E)
19. UniProt Consortium. Uniprot: a worldwide hub of protein knowledge. Nucleic acids research, 47 (D1):D506–D515, 2019. [find](https://scholar.google.com/scholar?q=UniProt+Consortium%2E+Uniprot%3A+a+worldwide+hub+of+protein+knowledge%2E+Nucleic+acids+research%2C+47+%28D1%29%3AD506%E2%80%93D515%2C+2019%2E)
20. Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. Introduction to Algorithms, 3rd Edition. MIT Press, 2009. ISBN 978-0-262-03384-8. URL [http://mitpress.mit.edu/books/introduction-algorithms](http://mitpress.mit.edu/books/introduction-algorithms). [↗](http://mitpress.mit.edu/books/introduction-algorithms)
21. Zihang Dai, Zhilin Yang, Yiming Yang, William W. Cohen, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-XL: Language modeling with longer-term dependency, 2019. URL [https://openreview.net/forum?id=HJePno0cYm](https://openreview.net/forum?id=HJePno0cYm). [↗](https://openreview.net/forum?id=HJePno0cYm)
22. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. [find](https://scholar.google.com/scholar?q=Mostafa+Dehghani%2C+Stephan+Gouws%2C+Oriol+Vinyals%2C+Jakob+Uszkoreit%2C+and+Lukasz+Kaiser%2E+Universal+transformers%2E+In+7th+International+Conference+on+Learning+Representations%2C+ICLR+2019%2C+New+Orleans%2C+LA%2C+USA)
23. OpenReview.net, 2019. URL [https://openreview.net/forum?id=HyzdRiR9Y7](https://openreview.net/forum?id=HyzdRiR9Y7). [↗](https://openreview.net/forum?id=HyzdRiR9Y7)
24. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). CoRR, abs/1810.04805, 2018. URL http://arxiv.org/abs/1810.04805. [↗](http://arxiv.org/abs/1810.04805)
25. Yilun Du, Joshua Meier, Jerry Ma, Rob Fergus, and Alexander Rives. Energy-based models for atomic-resolution protein conformations. arXiv preprint arXiv:2004.13167, 2020. — [arXiv:2004.13167](https://arxiv.org/abs/2004.13167) [find](https://scholar.google.com/scholar?q=Yilun+Du%2C+Joshua+Meier%2C+Jerry+Ma%2C+Rob+Fergus%2C+and+Alexander+Rives%2E+Energy%2Dbased+models+for+atomic%2Dresolution+protein+conformations%2E+arXiv+preprint+arXiv%3A2004%2E13167%2C+2020%2E+%E2%80%94+%5BarXiv%3A2004%2E13167%5D%28https%3A%2F%2F)
26. Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, and Burkhard Rost. End-to-end multitask learning, from protein language to protein features without alignments. bioRxiv, pp. 864405, 2019. [find](https://scholar.google.com/scholar?q=Ahmed+Elnaggar%2C+Michael+Heinzinger%2C+Christian+Dallago%2C+and+Burkhard+Rost%2E+End%2Dto%2Dend+multitask+learning%2C+from+protein+language+to+protein+features+without+alignments%2E+bioRxiv%2C+pp%2E+864405%2C+2019%2E)
27. Roy Frostig, Matthew Johnson, and Chris Leary. Compiling machine learning programs via high-level tracing. In Conference on Machine Learning and Systems 2018, 2018. URL [http://www.sysml.cc/doc/2018/146.pdf](http://www.sysml.cc/doc/2018/146.pdf). [↗](http://www.sysml.cc/doc/2018/146.pdf)
28. Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 3146–3154, 2019. [find](https://scholar.google.com/scholar?q=Jun+Fu%2C+Jing+Liu%2C+Haijie+Tian%2C+Yong+Li%2C+Yongjun+Bao%2C+Zhiwei+Fang%2C+and+Hanqing+Lu%2E+Dual+attention+network+for+scene+segmentation%2E+In+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+CVPR+201)
29. Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. [Conformer: Convolution-augmented transformer for speech recognition](https://md.research.elamur.ai/free/en/papers/gulati-2020-conformer-convolution-augmented-transformer-speech/), 2020.
30. Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer: Generating music with long-term structure. In 7th International Conference on Learning Representations, ICLR 201… [find](https://scholar.google.com/scholar?q=Cheng%2DZhi+Anna+Huang%2C+Ashish+Vaswani%2C+Jakob+Uszkoreit%2C+Ian+Simon%2C+Curtis+Hawthorne%2C+Noam+Shazeer%2C+Andrew+M%2E+Dai%2C+Matthew+D%2E+Hoffman%2C+Monica+Dinculescu%2C+and+Douglas+Eck%2E+Music+transformer%3A+Generating+m)
31. OpenReview.net, 2019. URL [https://openreview.net/forum?id=rJe4ShAcF7](https://openreview.net/forum?id=rJe4ShAcF7). [↗](https://openreview.net/forum?id=rJe4ShAcF7)
32. John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph-based protein design. In Advances in Neural Information Processing Systems, pp. 15794–15805, 2019. [find](https://scholar.google.com/scholar?q=John+Ingraham%2C+Vikas+Garg%2C+Regina+Barzilay%2C+and+Tommi+Jaakkola%2E+Generative+models+for+graph%2Dbased+protein+design%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+15794%E2%80%9315805%2C+2019%2E)
33. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. CoRR, abs/2006.16236, 2020. URL https://arxiv.org/abs/2006.16236. [↗](https://arxiv.org/abs/2006.16236)
34. Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. [Reformer: The efficient transformer](https://md.research.elamur.ai/free/en/papers/kitaev-2020-reformer-efficient-transformer/). In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
35. OpenReview.net, 2020. URL [https://openreview.net/forum?id=rkgNKkHtvB](https://openreview.net/forum?id=rkgNKkHtvB). [↗](https://openreview.net/forum?id=rkgNKkHtvB)
36. Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. [Revealing the dark secrets of bert](https://md.research.elamur.ai/free/en/papers/kovaleva-2019-revealing-dark-secrets-bert/). arXiv preprint arXiv:1908.08593, 2019.
37. Taku Kudo and John Richardson. [Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/). CoRR, abs/1808.06226, 2018. URL http://arxiv.org/abs/1808.06226. [↗](http://arxiv.org/abs/1808.06226)
38. Richard E. Ladner and Michael J. Fischer. Parallel prefix computation. J. ACM, 27(4):831–838, October 1980. ISSN 0004-5411. doi: 10.1145/322217.322232. URL https://doi.org/10.1145/322217.322232. — [doi:10.1145/322217.322232](https://doi.org/10.1145/322217.322232) [↗](https://doi.org/10.1145/322217.322232)
39. Han Lin, Haoxian Chen, Tianyi Zhang, Clément Laroche, and Krzysztof Choromanski. Demystifying orthogonal Monte Carlo and beyond. CoRR, abs/2005.13590, 2020. — [arXiv:2005.13590](https://arxiv.org/abs/2005.13590) [find](https://scholar.google.com/scholar?q=Han+Lin%2C+Haoxian+Chen%2C+Tianyi+Zhang%2C+Cl%C3%A9ment+Laroche%2C+and+Krzysztof+Choromanski%2E+Demystifying+orthogonal+Monte+Carlo+and+beyond%2E+CoRR%2C+abs%2F2005%2E13590%2C+2020%2E+%E2%80%94+%5BarXiv%3A2005%2E13590%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
40. Haoneng Luo, Shiliang Zhang, Ming Lei, and Lei Xie. Simplified self-attention for transformer-based end-to-end speech recognition. CoRR, abs/2005.10463, 2020. URL https://arxiv.org/abs/2005.10463. — [arXiv:2005.10463](https://arxiv.org/abs/2005.10463) [↗](https://arxiv.org/abs/2005.10463)
41. Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R. Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. CoRR, abs/2004.03497, 2020. URL https://arxiv.org/abs/2004.03497. — [arXiv:2004.03497](https://arxiv.org/abs/2004.03497) [↗](https://arxiv.org/abs/2004.03497)
42. Jessica Marcandalli, Brooke Fiala, Sebastian Ols, Michela Perotti, Willem de van der Schueren, Joost Snijder, Edgar Hodge, Mark Benhaim, Rashmi Ravichandran, Lauren Carter, et al. Induction of potent neutralizing antibody responses by a designed protein nanoparticle vaccine for respiratory syncytial… [find](https://scholar.google.com/scholar?q=Jessica+Marcandalli%2C+Brooke+Fiala%2C+Sebastian+Ols%2C+Michela+Perotti%2C+Willem+de+van+der+Schueren%2C+Joost+Snijder%2C+Edgar+Hodge%2C+Mark+Benhaim%2C+Rashmi+Ravichandran%2C+Lauren+Carter%2C+et+al%2E+Induction+of+potent+)
43. Nikita Nangia and Samuel R. Bowman. Listops: A diagnostic dataset for latent tree learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 2-4, 2018, Student Research Workshop, pp… [find](https://scholar.google.com/scholar?q=Nikita+Nangia+and+Samuel+R%2E+Bowman%2E+Listops%3A+A+diagnostic+dataset+for+latent+tree+learning%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Lin)
44. Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine… [find](https://scholar.google.com/scholar?q=Niki+Parmar%2C+Ashish+Vaswani%2C+Jakob+Uszkoreit%2C+Lukasz+Kaiser%2C+Noam+Shazeer%2C+Alexander+Ku%2C+and+Dustin+Tran%2E+Image+transformer%2E+In+Proceedings+of+the+35th+International+Conference+on+Machine+Learning%2C+IC)
45. PMLR, 2018. URL [http://proceedings.mlr.press/v80/parmar18a.html](http://proceedings.mlr.press/v80/parmar18a.html). [↗](http://proceedings.mlr.press/v80/parmar18a.html)
46. Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. [Compressive transformers for long-range sequence modelling](https://md.research.elamur.ai/free/en/papers/rae-2019-compressive-transformers-long-range-sequence/). In International Conference on Learning Representations, 2020. URL [https://openreview.net/forum?id=SylKikSYDH](https://openreview.net/forum?id=SylKikSYDH). [↗](https://openreview.net/forum?id=SylKikSYDH)
47. Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems 20, Proceedings of the Twenty-First Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 3-6, 2007, pp. 1177–1184. [find](https://scholar.google.com/scholar?q=Ali+Rahimi+and+Benjamin+Recht%2E+Random+features+for+large%2Dscale+kernel+machines%2E+In+Advances+in+Neural+Information+Processing+Systems+20%2C+Proceedings+of+the+Twenty%2DFirst+Annual+Conference+on+Neural+Inf)
48. Curran Associates, Inc., 2007. URL [http://papers.nips.cc/paper/3182-random-features-for-large-scale-kernel-machines](http://papers.nips.cc/paper/3182-random-features-for-large-scale-kernel-machines). [↗](http://papers.nips.cc/paper/3182-random-features-for-large-scale-kernel-machines)
49. Alexander Rives, Siddharth Goyal, Joshua Meier, Demi Guo, Myle Ott, C. Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. bioArxiv, 04 2019. doi: 10.1101/622803. — [doi:10.1101/622803](https://doi.org/10.1101/622803) [find](https://scholar.google.com/scholar?q=Alexander+Rives%2C+Siddharth+Goyal%2C+Joshua+Meier%2C+Demi+Guo%2C+Myle+Ott%2C+C%2E+Zitnick%2C+Jerry+Ma%2C+and+Rob+Fergus%2E+Biological+structure+and+function+emerge+from+scaling+unsupervised+learning+to+250+million+pro)
50. Mark Rowland, Jiri Hron, Yunhao Tang, Krzysztof Choromanski, Tamás Sarlós, and Adrian Weller. Orthogonal estimation of Wasserstein distances. In The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings… [find](https://scholar.google.com/scholar?q=Mark+Rowland%2C+Jiri+Hron%2C+Yunhao+Tang%2C+Krzysztof+Choromanski%2C+Tam%C3%A1s+Sarl%C3%B3s%2C+and+Adrian+Weller%2E+Orthogonal+estimation+of+Wasserstein+distances%2E+In+The+22nd+International+Conference+on+Artificial+Intelli)
51. PMLR, 2019. URL [http://proceedings.mlr.press/v89/rowland19a.html](http://proceedings.mlr.press/v89/rowland19a.html). [↗](http://proceedings.mlr.press/v89/rowland19a.html)
52. Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. [Efficient content-based sparse attention with routing transformers](https://md.research.elamur.ai/free/en/papers/roy-2020-efficient-content-based-sparse-attention/). CoRR, abs/2003.05997, 2020. URL https://arxiv.org/abs/2003.05997. [↗](https://arxiv.org/abs/2003.05997)
53. Zhuoran Shen, Mingyuan Zhang, Shuai Yi, Junjie Yan, and Haiyu Zhao. Factorized attention: Self-attention with linear complexities. CoRR, abs/1812.01243, 2018. URL http://arxiv.org/abs/1812.01243. — [arXiv:1812.01243](https://arxiv.org/abs/1812.01243) [↗](http://arxiv.org/abs/1812.01243)
54. Emma Strubell, Ananya Ganesh, and Andrew McCallum. [Energy and policy considerations for deep learning in NLP](https://md.research.elamur.ai/free/en/papers/strubell-2019-energy-policy-considerations-deep-learning/). CoRR, abs/1906.02243, 2019. URL http://arxiv.org/abs/1906.02243. [↗](http://arxiv.org/abs/1906.02243)
55. Yujin Tang, Duong Nguyen, and David Ha. Neuroevolution of self-interpretable agents. CoRR, abs/2003.08165, 2020. URL https://arxiv.org/abs/2003.08165. — [arXiv:2003.08165](https://arxiv.org/abs/2003.08165) [↗](https://arxiv.org/abs/2003.08165)
56. Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. [Long range arena: A benchmark for efficient transformers](https://md.research.elamur.ai/free/en/papers/tay-2020-long-range-arena-benchmark-efficient/). 2021.
57. Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. Transformer dissection: An unified understanding for transformer’s attention via the lens of kernel. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9… [find](https://scholar.google.com/scholar?q=Yao%2DHung+Hubert+Tsai%2C+Shaojie+Bai%2C+Makoto+Yamada%2C+Louis%2DPhilippe+Morency%2C+and+Ruslan+Salakhutdinov%2E+Transformer+dissection%3A+An+unified+understanding+for+transformer%E2%80%99s+attention+via+the+lens+of+kernel%2E)
58. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems 30, pp. 5998–6008. Curran Associates, Inc., 2017. URL http://papers. nips.cc/paper/7181-attent…
59. Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. [find](https://scholar.google.com/scholar?q=Petar+Velickovic%2C+Guillem+Cucurull%2C+Arantxa+Casanova%2C+Adriana+Romero%2C+Pietro+Li%C3%B2%2C+and+Yoshua+Bengio%2E+Graph+attention+networks%2E+In+6th+International+Conference+on+Learning+Representations%2C+ICLR+2018%2C+V)
60. OpenReview.net, 2018. URL [https://openreview.net/forum?id=rJXMpikCZ](https://openreview.net/forum?id=rJXMpikCZ). [↗](https://openreview.net/forum?id=rJXMpikCZ)
61. Jesse Vig. [A multiscale visualization of attention in the transformer model](https://md.research.elamur.ai/free/en/papers/vig-2019-multiscale-visualization-attention-transformer-model/). arXiv preprint arXiv:1906.05714, 2019.
62. Jesse Vig and Yonatan Belinkov. [Analyzing the structure of attention in a transformer language model](https://md.research.elamur.ai/free/en/papers/vig-2019-analyzing-structure-attention-transformer-language/). CoRR, abs/1906.04284, 2019. URL http://arxiv.org/abs/1906.04284. [↗](http://arxiv.org/abs/1906.04284)
63. Jesse Vig, Ali Madani, Lav R. Varshney, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. Bertology meets biology: Interpreting attention in protein language models. CoRR, abs/2006.15222, 2020. URL https://arxiv.org/abs/2006.15222. [↗](https://arxiv.org/abs/2006.15222)
64. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 2692–2700, 2015. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Meire+Fortunato%2C+and+Navdeep+Jaitly%2E+Pointer+networks%2E+In+Advances+in+Neural+Information+Processing+Systems+28%3A+Annual+Conference+on+Neural+Information+Processing+Systems+2015%2C+December)
65. Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. CoRR, abs/2006.04768, 2020. URL https://arxiv.org/abs/2006.04768. [↗](https://arxiv.org/abs/2006.04768)
66. Tong Xiao, Yinqiao Li, Jingbo Zhu, Zhengtao Yu, and Tongran Liu. Sharing attention weights for fast transformer. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 5292–5298. ijcai.org, 2019. doi: 10.24963/… — [doi:10.24963/ijcai.2019/735](https://doi.org/10.24963/ijcai.2019/735) [find](https://scholar.google.com/scholar?q=Tong+Xiao%2C+Yinqiao+Li%2C+Jingbo+Zhu%2C+Zhengtao+Yu%2C+and+Tongran+Liu%2E+Sharing+attention+weights+for+fast+transformer%2E+In+Proceedings+of+the+Twenty%2DEighth+International+Joint+Conference+on+Artificial+Intell)
67. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander J. Smola, and Eduard H. Hovy. [Hierarchical attention networks for document classification](https://md.research.elamur.ai/free/en/papers/yang-2016-hierarchical-attention-networks-document-classification/). In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Sa…
68. Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Toward more efficient training of deep networks. In International Conference on Learning Representations, 2020. URL [https://openreview.net/forum…](https://openreview.net/forum…) [↗](https://openreview.net/forum…)
69. Felix X. Yu, Ananda Theertha Suresh, Krzysztof Marcin Choromanski, Daniel N. Holtmann-Rice, and Sanjiv Kumar. Orthogonal random features. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, p… [find](https://scholar.google.com/scholar?q=Felix+X%2E+Yu%2C+Ananda+Theertha+Suresh%2C+Krzysztof+Marcin+Choromanski%2C+Daniel+N%2E+Holtmann%2DRice%2C+and+Sanjiv+Kumar%2E+Orthogonal+random+features%2E+In+Advances+in+Neural+Information+Processing+Systems+29%3A+Annua)
70. Vinícius Flores Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David P. Reichert, Timothy P. Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, and Peter W. Battaglia. Deep reinforcement learni… [find](https://scholar.google.com/scholar?q=Vin%C3%ADcius+Flores+Zambaldi%2C+David+Raposo%2C+Adam+Santoro%2C+Victor+Bapst%2C+Yujia+Li%2C+Igor+Babuschkin%2C+Karl+Tuyls%2C+David+P%2E+Reichert%2C+Timothy+P%2E+Lillicrap%2C+Edward+Lockhart%2C+Murray+Shanahan%2C+Victoria+Langston%2C)
71. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. [Aligning books and movies: Towards story-like visual explanations by watching movies and reading books](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/). In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Ch…

*Updated 30 October 2026*