---
title: "Generating Long Sequences with Sparse Transformers (2019)"
description: "The authors from OpenAI propose the Sparse Transformer, which replaces the full attention matrix… 28 references, cited by 543 papers here."
authors: ["Rewon Child", "Scott Gray", "Alec Radford", "Ilya Sutskever"]
year: 2019
venue: "arXiv:1904.10509"
source: "https://arxiv.org/abs/1904.10509"
doi: "10.48550/arxiv.1904.10509"
arxiv: "1904.10509"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/) / Generating Long Sequences with Sparse Transformers

# Generating Long Sequences with Sparse Transformers

**Authors:** [Rewon Child](https://md.research.elamur.ai/free/en/authors/rewon-child/), [Scott Gray](https://md.research.elamur.ai/free/en/authors/scott-gray/), [Alec Radford](https://md.research.elamur.ai/free/en/authors/alec-radford/) and [Ilya Sutskever](https://md.research.elamur.ai/free/en/authors/ilya-sutskever/) · **Year:** 2019 · **Venue:** arXiv:1904.10509 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1904.10509) · [PDF](https://arxiv.org/pdf/1904.10509) · [DOI 10.48550/arxiv.1904.10509](https://doi.org/10.48550/arxiv.1904.10509) · [arXiv 1904.10509](https://arxiv.org/abs/1904.10509) · [OpenAlex](https://openalex.org/W2940744433)

**Citations:** 424 citations (OpenAlex, 2026-10-14) · [543](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The classic Transformer scales poorly to long sequences because the time and memory required for self-attention grow quadratically with input length. The authors from OpenAI propose the Sparse Transformer, which replaces the full attention matrix with sparse factorized patterns. The strided pattern suits data with a periodic structure, such as images or music, while the fixed pattern suits text. This substitution reduces complexity to O(n√n), yet any position remains reachable from any other within a few attention steps. In addition, the authors modified the residual block and the initialization so that networks with hundreds of layers can be trained. To save memory, attention matrices are recomputed during the backward pass, and fast GPU kernels were written for training. The same architecture works with raw bytes of text, images, and audio. It set new density modeling records on Enwik8, CIFAR-10, and ImageNet 64x64 and generates samples with coherent global structure. The work showed that attention can be applied to sequences tens of thousands of steps long, and theoretically even up to a million. The idea of sparse attention later became one of the foundations for long-context models.

## Key points
- **CIFAR-10:** a strided model with 59M parameters and 128 layers achieves 2.80 bits per dim, compared with the previous record of 2.85 set by PixelSNAIL. The model also outperforms dense attention (2.82) and trains faster.
- **Enwik8:** a fixed model with 95M parameters and 30 layers, with a context of 12,288, reaches 0.99 bits per byte. This is better than Transformer-XL 88M (1.03) and matches Transformer-XL 277M. The strided pattern performs poorly on text.
- **ImageNet 64x64:** a model with 152M parameters and 48 layers achieves 3.44 bits per dim versus 3.52 for SPN. Training took 7 days on 64 V100 GPUs.
- **Long context:** when the available context is increased during evaluation on Enwik8, compression improves monotonically from 0.9952 to 0.9908. The model therefore genuinely uses long-range dependencies.
- **Audio:** on raw recordings of classical music (5 s, 12 kHz), the model achieves 1.97 bits per byte. Generated samples are up to 65,536 steps long.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

## Cite

Cite as: Rewon Child, Scott Gray, Alec Radford and Ilya Sutskever. 2019. Generating Long Sequences with Sparse Transformers. arXiv:1904.10509.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{child2019generating,
  title = {{Generating Long Sequences with Sparse Transformers}},
  author = {Rewon Child, Scott Gray, Alec Radford and Ilya Sutskever},
  year = {2019},
  doi = {10.48550/arxiv.1904.10509},
  eprint = {1904.10509},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1904.10509}
}
```

- [← previous ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-2-0-continual/)
- [next → ERNIE: Enhanced Representation through Knowledge Integration](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-enhanced-representation-through-knowledge/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/)

## References

28 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Al-Rfou, R., Choe, D., Constant, N., Guo, M., and Jones, L. Character-level language modeling with deeper self-attention. arXiv preprint arXiv:1808.04444, 2018. [find](https://scholar.google.com/scholar?q=Al%2DRfou%2C+R%2E%2C+Choe%2C+D%2E%2C+Constant%2C+N%2E%2C+Guo%2C+M%2E%2C+and+Jones%2C+L%2E+Character%2Dlevel+language+modeling+with+deeper+self%2Dattention%2E+arXiv+preprint+arXiv%3A1808%2E04444%2C+2018%2E)
2. Ba, J. L., Kiros, J. R., and Hinton, G. E. [Layer normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/). arXiv preprint arXiv:1607.06450, 2016.
3. Britz, D., Guan, M. Y., and Luong, M.-T. Efficient attention using a fixed-size memory representation. arXiv preprint arXiv:1707.00110, 2017. [find](https://scholar.google.com/scholar?q=Britz%2C+D%2E%2C+Guan%2C+M%2E+Y%2E%2C+and+Luong%2C+M%2E%2DT%2E+Efficient+attention+using+a+fixed%2Dsize+memory+representation%2E+arXiv+preprint+arXiv%3A1707%2E00110%2C+2017%2E)
4. Chen, T., Xu, B., Zhang, C., and Guestrin, C. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016. [find](https://scholar.google.com/scholar?q=Chen%2C+T%2E%2C+Xu%2C+B%2E%2C+Zhang%2C+C%2E%2C+and+Guestrin%2C+C%2E+Training+deep+nets+with+sublinear+memory+cost%2E+arXiv+preprint+arXiv%3A1604%2E06174%2C+2016%2E)
5. Chen, X., Mishra, N., Rohaninejad, M., and Abbeel, P. Pixelsnail: An improved autoregressive generative model. arXiv preprint arXiv:1712.09763, 2017. — [arXiv:1712.09763](https://arxiv.org/abs/1712.09763) [find](https://scholar.google.com/scholar?q=Chen%2C+X%2E%2C+Mishra%2C+N%2E%2C+Rohaninejad%2C+M%2E%2C+and+Abbeel%2C+P%2E+Pixelsnail%3A+An+improved+autoregressive+generative+model%2E+arXiv+preprint+arXiv%3A1712%2E09763%2C+2017%2E+%E2%80%94+%5BarXiv%3A1712%2E09763%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1712%2E09)
6. Chiu, C.-C. and Raffel, C. Monotonic chunkwise attention. arXiv preprint arXiv:1712.05382, 2017. [find](https://scholar.google.com/scholar?q=Chiu%2C+C%2E%2DC%2E+and+Raffel%2C+C%2E+Monotonic+chunkwise+attention%2E+arXiv+preprint+arXiv%3A1712%2E05382%2C+2017%2E)
7. Dai, Z., Yang, Z., Yang, Y., Cohen, W. W., Carbonell, J., Le, Q. V., and Salakhutdinov, R. Transformer-xl: Language modeling with longer-term dependency. 2018. [find](https://scholar.google.com/scholar?q=Dai%2C+Z%2E%2C+Yang%2C+Z%2E%2C+Yang%2C+Y%2E%2C+Cohen%2C+W%2E+W%2E%2C+Carbonell%2C+J%2E%2C+Le%2C+Q%2E+V%2E%2C+and+Salakhutdinov%2C+R%2E+Transformer%2Dxl%3A+Language+modeling+with+longer%2Dterm+dependency%2E+2018%2E)
8. Dieleman, S., van den Oord, A., and Simonyan, K. The challenge of realistic music generation: modelling raw audio at scale. In Advances in Neural Information Processing Systems, pp. 8000–8010, 2018. [find](https://scholar.google.com/scholar?q=Dieleman%2C+S%2E%2C+van+den+Oord%2C+A%2E%2C+and+Simonyan%2C+K%2E+The+challenge+of+realistic+music+generation%3A+modelling+raw+audio+at+scale%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+8000%E2%80%938010%2C+2018%2E)
9. Gehring, J., Auli, M., Grangier, D., Yarats, D., and Dauphin, Y. N. [Convolutional sequence to sequence learning](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/). arXiv preprint arXiv:1705.03122, 2017.
10. Gruslys, A., Munos, R., Danihelka, I., Lanctot, M., and Graves, A. Memory-efficient backpropagation through time. In Advances in Neural Information Processing Systems, pp. 4125–4133, 2016. [find](https://scholar.google.com/scholar?q=Gruslys%2C+A%2E%2C+Munos%2C+R%2E%2C+Danihelka%2C+I%2E%2C+Lanctot%2C+M%2E%2C+and+Graves%2C+A%2E+Memory%2Defficient+backpropagation+through+time%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+4125%E2%80%934133%2C+2016%2E)
11. He, K., Zhang, X., Ren, S., and Sun, J. Identity mappings in deep residual networks. arXiv preprint arXiv:1603.05027, 2016. [find](https://scholar.google.com/scholar?q=He%2C+K%2E%2C+Zhang%2C+X%2E%2C+Ren%2C+S%2E%2C+and+Sun%2C+J%2E+Identity+mappings+in+deep+residual+networks%2E+arXiv+preprint+arXiv%3A1603%2E05027%2C+2016%2E)
12. Hendrycks, D. and Gimpel, K. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. arXiv preprint arXiv:1606.08415, 2016. [find](https://scholar.google.com/scholar?q=Hendrycks%2C+D%2E+and+Gimpel%2C+K%2E+Bridging+nonlinearities+and+stochastic+regularizers+with+gaussian+error+linear+units%2E+arXiv+preprint+arXiv%3A1606%2E08415%2C+2016%2E)
13. Huang, C.-Z. A., Vaswani, A., Uszkoreit, J., Shazeer, N., Hawthorne, C., Dai, A. M., Hoffman, M. D., and Eck, D. An improved relative self-attention mechanism for transformer with application to music generation. arXiv preprint arXiv:1809.04281, 2018. [find](https://scholar.google.com/scholar?q=Huang%2C+C%2E%2DZ%2E+A%2E%2C+Vaswani%2C+A%2E%2C+Uszkoreit%2C+J%2E%2C+Shazeer%2C+N%2E%2C+Hawthorne%2C+C%2E%2C+Dai%2C+A%2E+M%2E%2C+Hoffman%2C+M%2E+D%2E%2C+and+Eck%2C+D%2E+An+improved+relative+self%2Dattention+mechanism+for+transformer+with+application+to+music)
14. Jozefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., and Wu, Y. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016. [find](https://scholar.google.com/scholar?q=Jozefowicz%2C+R%2E%2C+Vinyals%2C+O%2E%2C+Schuster%2C+M%2E%2C+Shazeer%2C+N%2E%2C+and+Wu%2C+Y%2E+Exploring+the+limits+of+language+modeling%2E+arXiv+preprint+arXiv%3A1602%2E02410%2C+2016%2E)
15. Kingma, D. P. and Dhariwal, P. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, pp. 10236–10245, 2018. [find](https://scholar.google.com/scholar?q=Kingma%2C+D%2E+P%2E+and+Dhariwal%2C+P%2E+Glow%3A+Generative+flow+with+invertible+1x1+convolutions%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+10236%E2%80%9310245%2C+2018%2E)
16. Koutnik, J., Greff, K., Gomez, F., and Schmidhuber, J. A clockwork rnn. arXiv preprint arXiv:1402.3511, 2014. — [arXiv:1402.3511](https://arxiv.org/abs/1402.3511) [find](https://scholar.google.com/scholar?q=Koutnik%2C+J%2E%2C+Greff%2C+K%2E%2C+Gomez%2C+F%2E%2C+and+Schmidhuber%2C+J%2E+A+clockwork+rnn%2E+arXiv+preprint+arXiv%3A1402%2E3511%2C+2014%2E+%E2%80%94+%5BarXiv%3A1402%2E3511%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1402%2E3511%29)
17. Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, L., and Shazeer, N. [Generating wikipedia by summarizing long sequences](https://md.research.elamur.ai/free/en/papers/liu-2018-generating-wikipedia-summarizing-long-sequences/). arXiv preprint arXiv:1801.10198, 2018.
18. Mehri, S., Kumar, K., Gulrajani, I., Kumar, R., Jain, S., Sotelo, J., Courville, A., and Bengio, Y. Samplernn: An unconditional end-to-end neural audio generation model. arXiv preprint arXiv:1612.07837, 2016. — [arXiv:1612.07837](https://arxiv.org/abs/1612.07837) [find](https://scholar.google.com/scholar?q=Mehri%2C+S%2E%2C+Kumar%2C+K%2E%2C+Gulrajani%2C+I%2E%2C+Kumar%2C+R%2E%2C+Jain%2C+S%2E%2C+Sotelo%2C+J%2E%2C+Courville%2C+A%2E%2C+and+Bengio%2C+Y%2E+Samplernn%3A+An+unconditional+end%2Dto%2Dend+neural+audio+generation+model%2E+arXiv+preprint+arXiv%3A1612%2E0783)
19. Menick, J. and Kalchbrenner, N. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. arXiv preprint arXiv:1812.01608, 2018. — [arXiv:1812.01608](https://arxiv.org/abs/1812.01608) [find](https://scholar.google.com/scholar?q=Menick%2C+J%2E+and+Kalchbrenner%2C+N%2E+Generating+high+fidelity+images+with+subscale+pixel+networks+and+multidimensional+upscaling%2E+arXiv+preprint+arXiv%3A1812%2E01608%2C+2018%2E+%E2%80%94+%5BarXiv%3A1812%2E01608%5D%28https%3A%2F%2Farxiv%2Eo)
20. Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaev, O., Venkatesh, G., et al. Mixed precision training. arXiv preprint arXiv:1710.03740, 2017. [find](https://scholar.google.com/scholar?q=Micikevicius%2C+P%2E%2C+Narang%2C+S%2E%2C+Alben%2C+J%2E%2C+Diamos%2C+G%2E%2C+Elsen%2C+E%2E%2C+Garcia%2C+D%2E%2C+Ginsburg%2C+B%2E%2C+Houston%2C+M%2E%2C+Kuchaev%2C+O%2E%2C+Venkatesh%2C+G%2E%2C+et+al%2E+Mixed+precision+training%2E+arXiv+preprint+arXiv%3A1710%2E03740%2C+201)
21. Oord, A. v. d., Kalchbrenner, N., and Kavukcuoglu, K. Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759, 2016. [find](https://scholar.google.com/scholar?q=Oord%2C+A%2E+v%2E+d%2E%2C+Kalchbrenner%2C+N%2E%2C+and+Kavukcuoglu%2C+K%2E+Pixel+recurrent+neural+networks%2E+arXiv+preprint+arXiv%3A1601%2E06759%2C+2016%2E)
22. Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, Ł., Shazeer, N., and Ku, A. Image transformer. arXiv preprint arXiv:1802.05751, 2018. [find](https://scholar.google.com/scholar?q=Parmar%2C+N%2E%2C+Vaswani%2C+A%2E%2C+Uszkoreit%2C+J%2E%2C+Kaiser%2C+%C5%81%2E%2C+Shazeer%2C+N%2E%2C+and+Ku%2C+A%2E+Image+transformer%2E+arXiv+preprint+arXiv%3A1802%2E05751%2C+2018%2E)
23. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training. URL [https://s3-us-west-2.amazonaws](https://s3-us-west-2.amazonaws). com/openai-assets/research-covers/language-unsupervised/language understanding paper. pdf, 2018. [↗](https://s3-us-west-2.amazonaws)
24. Reed, S., Oord, A. v. d., Kalchbrenner, N., Colmenarejo, S. G., Wang, Z., Belov, D., and de Freitas, N. Parallel multiscale autoregressive density estimation. arXiv preprint arXiv:1703.03664, 2017. — [arXiv:1703.03664](https://arxiv.org/abs/1703.03664) [find](https://scholar.google.com/scholar?q=Reed%2C+S%2E%2C+Oord%2C+A%2E+v%2E+d%2E%2C+Kalchbrenner%2C+N%2E%2C+Colmenarejo%2C+S%2E+G%2E%2C+Wang%2C+Z%2E%2C+Belov%2C+D%2E%2C+and+de+Freitas%2C+N%2E+Parallel+multiscale+autoregressive+density+estimation%2E+arXiv+preprint+arXiv%3A1703%2E03664%2C+2017%2E+%E2%80%94+)
25. Salimans, T., Karpathy, A., Chen, X., and Kingma, D. P. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517, 2017. — [arXiv:1701.05517](https://arxiv.org/abs/1701.05517) [find](https://scholar.google.com/scholar?q=Salimans%2C+T%2E%2C+Karpathy%2C+A%2E%2C+Chen%2C+X%2E%2C+and+Kingma%2C+D%2E+P%2E+Pixelcnn%2B%2B%3A+Improving+the+pixelcnn+with+discretized+logistic+mixture+likelihood+and+other+modifications%2E+arXiv+preprint+arXiv%3A1701%2E05517%2C+2017%2E+)
26. Sukhbaatar, S., Weston, J., Fergus, R., et al. End-to-end memory networks. In Advances in neural information processing systems, pp. 2440–2448, 2015. [find](https://scholar.google.com/scholar?q=Sukhbaatar%2C+S%2E%2C+Weston%2C+J%2E%2C+Fergus%2C+R%2E%2C+et+al%2E+End%2Dto%2Dend+memory+networks%2E+In+Advances+in+neural+information+processing+systems%2C+pp%2E+2440%E2%80%932448%2C+2015%2E)
27. Van Den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. [Wavenet: A generative model for raw audio](https://md.research.elamur.ai/free/en/papers/oord-2016-wavenet-generative-model-raw-audio/). CoRR abs/1609.03499, 2016.
28. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, pp. 5998–6008, 2017.

*Updated 29 October 2026*