---
title: "The Devil in Linear Transformer — paper by Qin et al. (2022)"
description: "First, their analysis shows that the scaling (row normalization) that linear attention inherits from… 30 references, cited by 37 papers here."
authors: ["Zhen Qin", "Xiaodong Han", "Weixuan Sun", "Dongxu Li", "Lingpeng Kong", "Nick Barnes", "Yiran Zhong"]
year: 2022
venue: "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2022.emnlp-main.473/"
doi: "10.18653/v1/2022.emnlp-main.473"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/qin-2022-devil-linear-transformer/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/qin-2022-devil-linear-transformer/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / The Devil in Linear Transformer

# The Devil in Linear Transformer

**Authors:** [Zhen Qin](https://md.research.elamur.ai/free/en/authors/zhen-qin/), [Xiaodong Han](https://md.research.elamur.ai/free/en/authors/xiaodong-han/), [Weixuan Sun](https://md.research.elamur.ai/free/en/authors/weixuan-sun/), [Dongxu Li](https://md.research.elamur.ai/free/en/authors/dongxu-li/), [Lingpeng Kong](https://md.research.elamur.ai/free/en/authors/lingpeng-kong/), [Nick Barnes](https://md.research.elamur.ai/free/en/authors/nick-m-barnes/) and [Yiran Zhong](https://md.research.elamur.ai/free/en/authors/yiran-zhong/) · **Year:** 2022 · **Venue:** [Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2022](https://md.research.elamur.ai/free/en/venues/emnlp/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.emnlp-main.473/) · [PDF](https://aclanthology.org/2022.emnlp-main.473.pdf) · [DOI 10.18653/v1/2022.emnlp-main.473](https://doi.org/10.18653/v1/2022.emnlp-main.473) · [OpenAlex](https://openalex.org/W4385573415)

**Citations:** 30 citations (OpenAlex, 2026-10-14) · [37](https://md.research.elamur.ai/free/en/papers/qin-2022-devil-linear-transformer/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Kernel-based linear transformers reduce attention cost from quadratic to linear in sequence length, but they usually lose accuracy compared with the standard softmax transformer. The authors trace this gap to two causes. First, their analysis shows that the scaling (row normalization) that linear attention inherits from softmax leaves its gradients without an upper bound. Softmax gradients, by contrast, stay within 1/4. The unbounded gradients make training unstable. Second, linear attention spreads its weights almost evenly over long inputs, a problem they call attention dilution, so local structure is lost. They measure this with a locally accumulated attention score. Their model, TransNormer, removes the scaling and adds a normalization layer after attention instead (NormAttention). It also uses block-local DiagAttention in the lower layers, and the whole model keeps linear complexity. On WikiText-103 language modeling, GLUE classification and the Long-Range Arena benchmark, TransNormer outperforms the vanilla transformer and earlier linear variants while running faster. This makes efficient attention a realistic replacement for softmax attention rather than a compromise.

## Key points
- **Gradient analysis:** softmax attention gradients are bounded by 1/4. The linear-attention gradient bound scales as 1/(4|s_ik|), so it can grow arbitrarily large (Proposition 3.1). NormAttention works with either LayerNorm or RMSNorm.
- **Architecture:** in a 12-layer model, the best setup puts DiagAttention in the first 6 layers and NormAttention in the remaining 6. Using only one attention type, or swapping their order, gives higher validation perplexity in the RoBERTa-style pre-training ablation (3.82/3.81 for the chosen T1/T2 configuration).
- **Ablations:** larger DiagAttention blocks help (block size 128 gives ppl 3.72/3.69), but block size 64 was chosen to balance quality against the O(nwd) cost. A GLU feed-forward layer beats a standard FFN (3.82 vs 3.93 for T1). Running both attentions in parallel within a layer doubles compute without a gain.
- **Speed:** on an Nvidia A6000 (48 GB) using Skyformer's benchmark settings, TransNormer trains over 300% faster and runs inference 150% faster than FLASH-quad. The vanilla transformer runs out of memory beyond roughly 3k tokens.
- **Results and code:** both variants (T1, T2) beat all compared methods overall on Long-Range Arena. Code is published at github.com/OpenNLPLab/Transnormer.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Hong Kong](https://md.research.elamur.ai/free/en/universities/university-of-hong-kong/) — University, Pok Fu Lam, China · 89 works · 11,774 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 117 works · 11,281 citations
- [Australian National University](https://md.research.elamur.ai/free/en/universities/australian-national-university/) — University, Canberra, Australia · 23 works · 6,522 citations
- [Sensetime](https://md.research.elamur.ai/free/en/companies/sensetime/) — Company, Beijing, China · 12 works · 2,065 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

## Cite

Cite as: Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes and Yiran Zhong. 2022. The Devil in Linear Transformer. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{qin2022devil,
  title = {{The Devil in Linear Transformer}},
  author = {Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes and Yiran Zhong},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.473},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.473}
}
```

- [← previous IMPLI: Investigating NLI Models' Performance on Figurative Language](https://md.research.elamur.ai/free/en/papers/stowe-2022-impli-investigating-nli-models-performance/)
- [next → VALUE: Understanding Dialect Disparity in NLU](https://md.research.elamur.ai/free/en/papers/ziems-2022-value-understanding-dialect-disparity-nlu/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

30 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. [Layer normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/).
2. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). arXiv preprint arXiv:2004.05150.
3. Yifan Chen, Qi Zeng, Heng Ji, and Yun Yang. 2021. Skyformer: Remodel self-attention with gaussian kernel and nyström method. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual. [find](https://scholar.google.com/scholar?q=Yifan+Chen%2C+Qi+Zeng%2C+Heng+Ji%2C+and+Yun+Yang%2E+2021%2E+Skyformer%3A+Remodel+self%2Dattention+with+gaussian+kernel+and+nystr%C3%B6m+method%2E+In+Advances+in+Neural+Information+Processing+Systems+35%3A+Annual+Conference+)
4. Xuelian Cheng, Huan Xiong, Deng-Ping Fan, Yiran Zhong, Mehrtash Harandi, Tom Drummond, and Zongyuan Ge. 2022a. Implicit motion handling for video camouflaged object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13864–13873. [find](https://scholar.google.com/scholar?q=Xuelian+Cheng%2C+Huan+Xiong%2C+Deng%2DPing+Fan%2C+Yiran+Zhong%2C+Mehrtash+Harandi%2C+Tom+Drummond%2C+and+Zongyuan+Ge%2E+2022a%2E+Implicit+motion+handling+for+video+camouflaged+object+detection%2E+In+Proceedings+of+the+IE)
5. Xuelian Cheng, Yiran Zhong, Mehrtash Harandi, Tom Drummond, Zhiyong Wang, and Zongyuan Ge. 2022b. Deep laparoscopic stereo matching with transformers. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 464–474. Springer. [find](https://scholar.google.com/scholar?q=Xuelian+Cheng%2C+Yiran+Zhong%2C+Mehrtash+Harandi%2C+Tom+Drummond%2C+Zhiyong+Wang%2C+and+Zongyuan+Ge%2E+2022b%2E+Deep+laparoscopic+stereo+matching+with+transformers%2E+In+International+Conference+on+Medical+Image+Comp)
6. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. [Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/). arXiv preprint arXiv:1904.10509.
7. Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. 2020. [Rethinking attention with performers](https://md.research.elamur.ai/free/en/papers/choromanski-2020-rethinking-attention-performers/). arXiv preprint arXiv:2009.14794.
8. Bolin Gao and Lacra Pavel. 2017. On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint arXiv:1704.00805. — [arXiv:1704.00805](https://arxiv.org/abs/1704.00805) [find](https://scholar.google.com/scholar?q=Bolin+Gao+and+Lacra+Pavel%2E+2017%2E+On+the+properties+of+the+softmax+function+with+application+in+game+theory+and+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1704%2E00805%2E+%E2%80%94+%5BarXiv%3A1704%2E00805%5D%28https%3A%2F%2Farx)
9. Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc V Le. 2022. Transformer quality in linear time. arXiv preprint arXiv:2202.10447. [find](https://scholar.google.com/scholar?q=Weizhe+Hua%2C+Zihang+Dai%2C+Hanxiao+Liu%2C+and+Quoc+V+Le%2E+2022%2E+Transformer+quality+in+linear+time%2E+arXiv+preprint+arXiv%3A2202%2E10447%2E)
10. Eric Jang, Shixiang Gu, and Ben Poole. 2016. [Categorical reparameterization with gumbel-softmax](https://md.research.elamur.ai/free/en/papers/jang-2016-categorical-reparameterization-gumbel-softmax/). arXiv preprint arXiv:1611.01144.
11. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156–5165. PMLR. [find](https://scholar.google.com/scholar?q=Angelos+Katharopoulos%2C+Apoorv+Vyas%2C+Nikolaos+Pappas%2C+and+Fran%C3%A7ois+Fleuret%2E+2020%2E+Transformers+are+rnns%3A+Fast+autoregressive+transformers+with+linear+attention%2E+In+International+Conference+on+Machine+L)
12. Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020. [Reformer: The efficient transformer](https://md.research.elamur.ai/free/en/papers/kitaev-2020-reformer-efficient-transformer/). arXiv preprint arXiv:2001.04451.
13. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692.
14. Zexiang Liu, Dong Li, Kaiyue Lu, Zhen Qin, Weixuan Sun, Jiacheng Xu, and Yiran Zhong. 2022. Neural architecture search on efficient transformers and beyond. arXiv preprint arXiv:2207.13955. — [arXiv:2207.13955](https://arxiv.org/abs/2207.13955) [find](https://scholar.google.com/scholar?q=Zexiang+Liu%2C+Dong+Li%2C+Kaiyue+Lu%2C+Zhen+Qin%2C+Weixuan+Sun%2C+Jiacheng+Xu%2C+and+Yiran+Zhong%2E+2022%2E+Neural+architecture+search+on+efficient+transformers+and+beyond%2E+arXiv+preprint+arXiv%3A2207%2E13955%2E+%E2%80%94+%5BarXiv%3A2)
15. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/). 5th International Conference on Learning Representations, ICLR, Toulon, France.
16. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. [fairseq: A fast, extensible toolkit for sequence modeling](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/). arXiv preprint arXiv:1904.01038.
17. Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. 2020. Random feature attention. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Hao+Peng%2C+Nikolaos+Pappas%2C+Dani+Yogatama%2C+Roy+Schwartz%2C+Noah+Smith%2C+and+Lingpeng+Kong%2E+2020%2E+Random+feature+attention%2E+In+International+Conference+on+Learning+Representations%2E)
18. Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. 2022. cosformer: Rethinking softmax in attention. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhen+Qin%2C+Weixuan+Sun%2C+Hui+Deng%2C+Dongxu+Li%2C+Yunshen+Wei%2C+Baohong+Lv%2C+Junjie+Yan%2C+Lingpeng+Kong%2C+and+Yiran+Zhong%2E+2022%2E+cosformer%3A+Rethinking+softmax+in+attention%2E+In+International+Conference+on+Learni)
19. Noam Shazeer. 2020. Glu variants improve transformer. arXiv preprint arXiv:2002.05202. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+2020%2E+Glu+variants+improve+transformer%2E+arXiv+preprint+arXiv%3A2002%2E05202%2E)
20. Jingyu Sun, Guiping Zhong, Dinghao Zhou, Baoxiang Li, and Yiran Zhong. 2022a. Locality matters: A locality-biased linear attention for automatic speech recognition. arXiv preprint arXiv:2203.15609. — [arXiv:2203.15609](https://arxiv.org/abs/2203.15609) [find](https://scholar.google.com/scholar?q=Jingyu+Sun%2C+Guiping+Zhong%2C+Dinghao+Zhou%2C+Baoxiang+Li%2C+and+Yiran+Zhong%2E+2022a%2E+Locality+matters%3A+A+locality%2Dbiased+linear+attention+for+automatic+speech+recognition%2E+arXiv+preprint+arXiv%3A2203%2E15609%2E+%E2%80%94+)
21. Weixuan Sun, Zhen Qin, Hui Deng, Jianyuan Wang, Yi Zhang, Kaihao Zhang, Nick Barnes, Stan Birchfield, Lingpeng Kong, and Yiran Zhong. 2022b. Vicinity vision transformer. arXiv preprint arXiv:2206.10552. — [arXiv:2206.10552](https://arxiv.org/abs/2206.10552) [find](https://scholar.google.com/scholar?q=Weixuan+Sun%2C+Zhen+Qin%2C+Hui+Deng%2C+Jianyuan+Wang%2C+Yi+Zhang%2C+Kaihao+Zhang%2C+Nick+Barnes%2C+Stan+Birchfield%2C+Lingpeng+Kong%2C+and+Yiran+Zhong%2E+2022b%2E+Vicinity+vision+transformer%2E+arXiv+preprint+arXiv%3A2206%2E1055)
22. Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. 2020a. Sparse sinkhorn attention. In International Conference on Machine Learning, pages 9438–9447. PMLR. [find](https://scholar.google.com/scholar?q=Yi+Tay%2C+Dara+Bahri%2C+Liu+Yang%2C+Donald+Metzler%2C+and+Da%2DCheng+Juan%2E+2020a%2E+Sparse+sinkhorn+attention%2E+In+International+Conference+on+Machine+Learning%2C+pages+9438%E2%80%939447%2E+PMLR%2E)
23. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. [Big bird: Transformers for longer sequences](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/). In NeurIPS.
24. Biao Zhang and Rico Sennrich. 2019. Root Mean Square Layer Normalization. In Advances in Neural Information Processing Systems 32, Vancouver, Canada. [find](https://scholar.google.com/scholar?q=Biao+Zhang+and+Rico+Sennrich%2E+2019%2E+Root+Mean+Square+Layer+Normalization%2E+In+Advances+in+Neural+Information+Processing+Systems+32%2C+Vancouver%2C+Canada%2E)
25. Biao Zhang, Ivan Titov, and Rico Sennrich. 2021. Sparse attention with linear units. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6507–6520, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Biao+Zhang%2C+Ivan+Titov%2C+and+Rico+Sennrich%2E+2021%2E+Sparse+attention+with+linear+units%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+6507%E2%80%936520%2C+Online+)
26. Lin Zheng, Chong Wang, and Lingpeng Kong. 2022. Linear complexity randomized self-attention mechanism. arXiv preprint arXiv:2204.04667. — [arXiv:2204.04667](https://arxiv.org/abs/2204.04667) [find](https://scholar.google.com/scholar?q=Lin+Zheng%2C+Chong+Wang%2C+and+Lingpeng+Kong%2E+2022%2E+Linear+complexity+randomized+self%2Dattention+mechanism%2E+arXiv+preprint+arXiv%3A2204%2E04667%2E+%E2%80%94+%5BarXiv%3A2204%2E04667%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2204%2E04667%29)
27. Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. 2022. Audio-visual segmentation. In European Conference on Computer Vision. [find](https://scholar.google.com/scholar?q=Jinxing+Zhou%2C+Jianyuan+Wang%2C+Jiayi+Zhang%2C+Weixuan+Sun%2C+Jing+Zhang%2C+Stan+Birchfield%2C+Dan+Guo%2C+Lingpeng+Kong%2C+Meng+Wang%2C+and+Yiran+Zhong%2E+2022%2E+Audio%2Dvisual+segmentation%2E+In+European+Conference+on+Compu)
28. Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2020b. [Long range arena: A benchmark for efficient transformers](https://md.research.elamur.ai/free/en/papers/tay-2020-long-range-arena-benchmark-efficient/). In International Conference on Learning Representations.
29. Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. 2021. Long-short transformer: Efficient transformers for language and vision. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Chen+Zhu%2C+Wei+Ping%2C+Chaowei+Xiao%2C+Mohammad+Shoeybi%2C+Tom+Goldstein%2C+Anima+Anandkumar%2C+and+Bryan+Catanzaro%2E+2021%2E+Long%2Dshort+transformer%3A+Efficient+transformers+for+language+and+vision%2E+In+Advances+in+N)
30. Michalis K Titsias. 2016. One-vs-each approximation to softmax for scalable estimation of probabilities. arXiv preprint arXiv:1609.07410. — [arXiv:1609.07410](https://arxiv.org/abs/1609.07410) [find](https://scholar.google.com/scholar?q=Michalis+K+Titsias%2E+2016%2E+One%2Dvs%2Deach+approximation+to+softmax+for+scalable+estimation+of+probabilities%2E+arXiv+preprint+arXiv%3A1609%2E07410%2E+%E2%80%94+%5BarXiv%3A1609%2E07410%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1609%2E07410%29)

*Updated 29 October 2026*