---
title: "Transformer Quality in Linear Time — paper by Hua et al. (2022)"
description: "The authors argue that existing efficient-attention alternatives have at least one of three problems… 37 references, cited by 58 papers here."
authors: ["Weizhe Hua", "Zihang Dai", "Hanxiao Liu", "Quoc V. Le"]
year: 2022
venue: "arXiv:2202.10447"
source: "https://arxiv.org/abs/2202.10447"
doi: "10.48550/arxiv.2202.10447"
arxiv: "2202.10447"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hua-2022-transformer-quality-linear-time/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hua-2022-transformer-quality-linear-time/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / Transformer Quality in Linear Time

# Transformer Quality in Linear Time

**Authors:** Weizhe Hua, [Zihang Dai](https://md.research.elamur.ai/free/en/authors/zihang-dai/), [Hanxiao Liu](https://md.research.elamur.ai/free/en/authors/hanxiao-liu/) and [Quoc V. Le](https://md.research.elamur.ai/free/en/authors/quoc-viet-le/) · **Year:** 2022 · **Venue:** arXiv:2202.10447 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2202.10447) · [PDF](https://arxiv.org/pdf/2202.10447) · [DOI 10.48550/arxiv.2202.10447](https://doi.org/10.48550/arxiv.2202.10447) · [arXiv 2202.10447](https://arxiv.org/abs/2202.10447) · [OpenAlex](https://openalex.org/W4226261765)

**Citations:** 41 citations (OpenAlex, 2026-10-14) · [58](https://md.research.elamur.ai/free/en/papers/hua-2022-transformer-quality-linear-time/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Standard Transformers scale quadratically with sequence length. The authors argue that existing efficient-attention alternatives have at least one of three problems: they lose quality against well-tuned baselines, they run slower on GPUs/TPUs than their theoretical cost suggests, or they train slowly in the auto-regressive setting.

The work first redesigns the layer itself. The Gated Attention Unit (GAU) fuses a GLU-style gating block with a lightweight attention module, so a single small attention head that uses squared ReLU instead of softmax is enough. On top of GAU, the authors add mixed chunk attention. It splits the sequence into chunks, applies exact attention within each chunk and cheap linear attention between chunks. This gives linear cost that runs efficiently on accelerators.

The resulting model, FLASH, is tested on masked and auto-regressive language modeling (C4, Wiki-40B, PG-19) at 110M–500M parameters, with contexts from 512 to 8K tokens. It matches the perplexity of an enhanced Transformer++ while training several times faster. This shows that efficient attention can deliver real speed gains, not only theoretical ones.

## Key points
- Over Transformer++, FLASH trains up to 4.9× faster on Wiki-40B, up to 4.8× faster for masked LM on C4, and up to 12.1× cheaper on PG-19. Measurements were made on TPU-v4.
- GAU's attention adds only one small projection (shared dimension s = 128 by default). Two GAU layers replace one Transformer block (MHSA + MLP/GLU) at similar size and speed.
- Ablations show the local quadratic and global linear parts are complementary; removing either clearly hurts quality. Applying the same chunking to Transformer++ (MC-TFM++) costs much more perplexity, e.g., +0.87 vs. +0.11 for FLASH on Wiki-40B at 8K context.
- On TriviaQA (4096-token context, ~110M models), FLASH s=512 with first-to-all attention reaches F1 73.9, versus 74.2 for Transformer+. It cuts pretraining and fine-tuning latency by about 2.8× and 2.7×.
- The default chunk size is 256; larger chunks such as 512 tend to work better once the context exceeds 1024 tokens.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 492 works · 440,463 citations
- [Cornell University](https://md.research.elamur.ai/free/en/universities/cornell-university/) — University, Ithaca, United States · 134 works · 65,159 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Weizhe Hua, Zihang Dai, Hanxiao Liu and Quoc V. Le. 2022. Transformer Quality in Linear Time. arXiv:2202.10447.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hua2022transformer,
  title = {{Transformer Quality in Linear Time}},
  author = {Weizhe Hua, Zihang Dai, Hanxiao Liu and Quoc V. Le},
  year = {2022},
  doi = {10.48550/arxiv.2202.10447},
  eprint = {2202.10447},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2202.10447}
}
```

- [← previous Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets](https://md.research.elamur.ai/free/en/papers/wu-2022-generating-data-mitigate-spurious-correlations/)
- [next → Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints](https://md.research.elamur.ai/free/en/papers/komatsuzaki-2022-sparse-upcycling-training-mixture-experts/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

37 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ba, J. L., Kiros, J. R., and Hinton, G. E. [Layer normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/). arXiv preprint arXiv:1607.06450, 2016.
2. Beltagy, I., Peters, M. E., and Cohan, A. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). arXiv preprint arXiv:2004.05150, 2020.
3. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). arXiv preprint arXiv:2005.14165, 2020.
4. Ho, J., Kalchbrenner, N., Weissenborn, D., and Salimans, T. Axial attention in multidimensional transformers. arXiv preprint arXiv:1912.12180, 2019. — [arXiv:1912.12180](https://arxiv.org/abs/1912.12180) [find](https://scholar.google.com/scholar?q=Ho%2C+J%2E%2C+Kalchbrenner%2C+N%2E%2C+Weissenborn%2C+D%2E%2C+and+Salimans%2C+T%2E+Axial+attention+in+multidimensional+transformers%2E+arXiv+preprint+arXiv%3A1912%2E12180%2C+2019%2E+%E2%80%94+%5BarXiv%3A1912%2E12180%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1912%2E121)
5. Child, R., Gray, S., Radford, A., and Sutskever, I. [Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/). arXiv preprint arXiv:1904.10509, 2019.
6. Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., and Liu, W. Ccnet: Criss-cross attention for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 603–612, 2019. [find](https://scholar.google.com/scholar?q=Huang%2C+Z%2E%2C+Wang%2C+X%2E%2C+Huang%2C+L%2E%2C+Huang%2C+C%2E%2C+Wei%2C+Y%2E%2C+and+Liu%2C+W%2E+Ccnet%3A+Criss%2Dcross+attention+for+semantic+segmentation%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Conference+on+Computer+Vision%2C+pp%2E+)
7. Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. [Rethinking attention with performers](https://md.research.elamur.ai/free/en/papers/choromanski-2020-rethinking-attention-performers/). arXiv preprint arXiv:2009.14794, 2020.
8. Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., and Carreira, J. [Perceiver: General perception with iterative attention](https://md.research.elamur.ai/free/en/papers/jaegle-2021-perceiver-general-perception-iterative-attention/). In International Conference on Machine Learning, pp. 4651–4664. PMLR, 2021.
9. Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., and Salakhutdinov, R. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019. [find](https://scholar.google.com/scholar?q=Dai%2C+Z%2E%2C+Yang%2C+Z%2E%2C+Yang%2C+Y%2E%2C+Carbonell%2C+J%2E%2C+Le%2C+Q%2E+V%2E%2C+and+Salakhutdinov%2C+R%2E+Transformer%2Dxl%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+arXiv+preprint+arXiv%3A1901%2E02860%2C+2019%2E)
10. Joshi, M., Choi, E., Weld, D., and Zettlemoyer, L. [TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601–1611, Vancouver, Canada, July 2017…
11. Dauphin, Y. N., Fan, A., Auli, M., and Grangier, D. [Language modeling with gated convolutional networks](https://md.research.elamur.ai/free/en/papers/dauphin-2016-language-modeling-gated-convolutional-networks/). In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, pp. 933–941. JMLR.org, 2017.
12. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805, 2018.
13. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). arXiv preprint arXiv:2010.11929, 2020.
14. Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., et al. [Glam: Efficient scaling of language models with mixture-of-experts](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/). arXiv preprint arXiv:2112.06905, 2021.
15. Elfwing, S., Uchibe, E., and Doya, K. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3–11, 2018. [find](https://scholar.google.com/scholar?q=Elfwing%2C+S%2E%2C+Uchibe%2C+E%2E%2C+and+Doya%2C+K%2E+Sigmoid%2Dweighted+linear+units+for+neural+network+function+approximation+in+reinforcement+learning%2E+Neural+Networks%2C+107%3A3%E2%80%9311%2C+2018%2E)
16. Gehring, J., Auli, M., Grangier, D., Yarats, D., and Dauphin, Y. N. [Convolutional sequence to sequence learning](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/). In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, pp. 1243–1252. JMLR.org, 2017.
17. Guo, M., Dai, Z., Vrandecic, D., and Al-Rfou, R. Wiki-40b: Multilingual language model dataset. In LREC 2020, 2020. [find](https://scholar.google.com/scholar?q=Guo%2C+M%2E%2C+Dai%2C+Z%2E%2C+Vrandecic%2C+D%2E%2C+and+Al%2DRfou%2C+R%2E+Wiki%2D40b%3A+Multilingual+language+model+dataset%2E+In+LREC+2020%2C+2020%2E)
18. Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. [find](https://scholar.google.com/scholar?q=Hendrycks%2C+D%2E+and+Gimpel%2C+K%2E+Gaussian+error+linear+units+%28gelus%29%2E+arXiv+preprint+arXiv%3A1606%2E08415%2C+2016%2E)
19. Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pp. 5156–5165. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Katharopoulos%2C+A%2E%2C+Vyas%2C+A%2E%2C+Pappas%2C+N%2E%2C+and+Fleuret%2C+F%2E+Transformers+are+rnns%3A+Fast+autoregressive+transformers+with+linear+attention%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+5156%E2%80%935165%2E+)
20. Kitaev, N., Kaiser, Ł., and Levskaya, A. [Reformer: The efficient transformer](https://md.research.elamur.ai/free/en/papers/kitaev-2020-reformer-efficient-transformer/). arXiv preprint arXiv:2001.04451, 2020.
21. Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y.-X., and Yan, X. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in Neural Information Processing Systems, 32:5243–5253, 2019. [find](https://scholar.google.com/scholar?q=Li%2C+S%2E%2C+Jin%2C+X%2E%2C+Xuan%2C+Y%2E%2C+Zhou%2C+X%2E%2C+Chen%2C+W%2E%2C+Wang%2C+Y%2E%2DX%2E%2C+and+Yan%2C+X%2E+Enhancing+the+locality+and+breaking+the+memory+bottleneck+of+transformer+on+time+series+forecasting%2E+Advances+in+Neural+Informat)
22. Liu, H., Dai, Z., So, D. R., and Le, Q. V. Pay attention to mlps. NeurIPS, 2021. [find](https://scholar.google.com/scholar?q=Liu%2C+H%2E%2C+Dai%2C+Z%2E%2C+So%2C+D%2E+R%2E%2C+and+Le%2C+Q%2E+V%2E+Pay+attention+to+mlps%2E+NeurIPS%2C+2021%2E)
23. Narang, S., Chung, H. W., Tay, Y., Fedus, W., Fevry, T., Matena, M., Malkan, K., Fiedel, N., Shazeer, N., Lan, Z., et al. Do transformer modifications transfer across implementations and applications? arXiv preprint arXiv:2102.11972, 2021. [find](https://scholar.google.com/scholar?q=Narang%2C+S%2E%2C+Chung%2C+H%2E+W%2E%2C+Tay%2C+Y%2E%2C+Fedus%2C+W%2E%2C+Fevry%2C+T%2E%2C+Matena%2C+M%2E%2C+Malkan%2C+K%2E%2C+Fiedel%2C+N%2E%2C+Shazeer%2C+N%2E%2C+Lan%2C+Z%2E%2C+et+al%2E+Do+transformer+modifications+transfer+across+implementations+and+applications%3F)
24. Nguyen, T. Q. and Salazar, J. [Transformers without tears: Improving the normalization of self-attention](https://md.research.elamur.ai/free/en/papers/nguyen-2019-transformers-tears-improving-normalization-self/). CoRR, abs/1910.05895, 2019. URL [http://arxiv.org/](http://arxiv.org/) abs/1910.05895. [↗](http://arxiv.org/)
25. Peng, H. et al. Random feature attention. In ICLR, 2021. [find](https://scholar.google.com/scholar?q=Peng%2C+H%2E+et+al%2E+Random+feature+attention%2E+In+ICLR%2C+2021%2E)
26. Rae, J. W., Potapenko, A., Jayakumar, S. M., and Lillicrap, T. P. [Compressive transformers for long-range sequence modelling](https://md.research.elamur.ai/free/en/papers/rae-2019-compressive-transformers-long-range-sequence/). arXiv preprint arXiv:1911.05507, 2019.
27. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67, 2020. URL [http://jmlr.org/papers/v21/20-074.html](http://jmlr.org/papers/v21/20-074.html). [↗](http://jmlr.org/papers/v21/20-074.html)
28. Ramachandran, P., Zoph, B., and Le, Q. V. Searching for activation functions. CoRR, abs/1710.05941, 2017. URL http://arxiv.org/abs/1710.05941. [↗](http://arxiv.org/abs/1710.05941)
29. Ren, H., Dai, H., Dai, Z., Yang, M., Leskovec, J., Schuurmans, D., and Dai, B. Combiner: Full attention transformer with sparse computation cost. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021. URL [https://openreview.net…](https://openreview.net…) [↗](https://openreview.net…)
30. Roy, A., Saffar, M., Vaswani, A., and Grangier, D. [Efficient content-based sparse attention with routing transformers](https://md.research.elamur.ai/free/en/papers/roy-2020-efficient-content-based-sparse-attention/). Transactions of the Association for Computational Linguistics, 9:53–68, 2021.
31. Shazeer, N. GLU variants improve transformer. CoRR, abs/2002.05202, 2020. URL [https://arxiv.org/](https://arxiv.org/) abs/2002.05202. [↗](https://arxiv.org/)
32. So, D. R., Mańke, W., Liu, H., Dai, Z., Shazeer, N., and Le, Q. V. Primer: Searching for efficient transformers for language modeling. NeurIPS, 2021. [find](https://scholar.google.com/scholar?q=So%2C+D%2E+R%2E%2C+Man%CC%81ke%2C+W%2E%2C+Liu%2C+H%2E%2C+Dai%2C+Z%2E%2C+Shazeer%2C+N%2E%2C+and+Le%2C+Q%2E+V%2E+Primer%3A+Searching+for+efficient+transformers+for+language+modeling%2E+NeurIPS%2C+2021%2E)
33. Su, J., Lu, Y., Pan, S., Wen, B., and Liu, Y. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/), 2021.
34. Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. [Lamda: Language models for dialog applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/). arXiv preprint arXiv:2201.08239, 2022.
35. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in neural information processing systems, pp. 5998–6008, 2017.
36. Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020. [find](https://scholar.google.com/scholar?q=Wang%2C+S%2E%2C+Li%2C+B%2E+Z%2E%2C+Khabsa%2C+M%2E%2C+Fang%2C+H%2E%2C+and+Ma%2C+H%2E+Linformer%3A+Self%2Dattention+with+linear+complexity%2E+arXiv+preprint+arXiv%3A2006%2E04768%2C+2020%2E)
37. Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. [Big bird: Transformers for longer sequences](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/). In NeurIPS, 2020.

*Updated 2 November 2026*