--- title: "Striped Attention: Faster Ring Attention for Causal Transformers" description: "The authors show this scheme cannot benefit from causal masking. 1 reference, cited by 40 papers here." authors: ["William Brandon", "Aniruddha Nrusimha", "Kevin Qian", "Zachary Ankner", "Tian Jin", "Zhiye Song", "Jonathan Ragan-Kelley"] year: 2023 venue: "arXiv:2311.09431" source: "https://arxiv.org/abs/2311.09431" doi: "10.48550/arxiv.2311.09431" arxiv: "2311.09431" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/brandon-2023-striped-attention-faster-ring-attention/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/brandon-2023-striped-attention-faster-ring-attention/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/11/) / Striped Attention: Faster Ring Attention for Causal Transformers # Striped Attention: Faster Ring Attention for Causal Transformers **Authors:** [William Brandon](https://md.research.elamur.ai/free/en/authors/william-brandon/), [Aniruddha Nrusimha](https://md.research.elamur.ai/free/en/authors/aniruddha-nrusimha/), Kevin Qian, [Zachary Ankner](https://md.research.elamur.ai/free/en/authors/zachary-ankner/), [Tian Jin](https://md.research.elamur.ai/free/en/authors/tian-jin/), Zhiye Song and Jonathan Ragan-Kelley · **Year:** 2023 · **Venue:** arXiv:2311.09431 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2311.09431) · [PDF](https://arxiv.org/pdf/2311.09431) · [DOI 10.48550/arxiv.2311.09431](https://doi.org/10.48550/arxiv.2311.09431) · [arXiv 2311.09431](https://arxiv.org/abs/2311.09431) · [OpenAlex](https://openalex.org/W4388787547) **Citations:** 1 citations (OpenAlex, 2026-10-14) · [40](https://md.research.elamur.ai/free/en/papers/brandon-2023-striped-attention-faster-ring-attention/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Ring Attention (Liu et al., 2023) makes very long contexts trainable by splitting queries, keys and values along the sequence across accelerators arranged in a ring. Key/value blocks circulate between devices while compute overlaps communication. The authors show this scheme cannot benefit from causal masking. Each device holds a contiguous chunk, so after the first round some devices have fully masked tiles and others fully unmasked ones. Each round runs at the pace of the slowest device, so a causal run costs as much as a bidirectional one, even though it needs only about half the arithmetic. Their fix, Striped Attention, reorders tokens so that every device holds positions spread evenly across the sequence (for example, every N-th token). Each device then skips roughly half its work in every round. Because attention is permutation-equivariant, the output stays exact, and the layout is set once before the first layer. Built on the JAX Ring Attention codebase and tested on 1B–7B causal models, it trains up to 1.65× faster on long sequences. This makes very long-context generative models cheaper to train. ## Key points - **Imbalance:** in causal Ring Attention, the work per round is either fully needed or fully discarded depending on the device. Masking therefore saves no wall-clock time. - **Striped layout:** the token permutation is applied once before the model, with no extra per-layer communication, and results remain exact. - **A100 results:** on a server with 8 A100 80GB GPUs, end-to-end training throughput improved by up to 1.45× over Ring Attention at a sequence length of 262,144 (3B model). - **TPU results:** on 16 TPUv4 chips, speedups reached 1.65× at 786,432 tokens (3B model). On 8 TPUv3 chips, speedups reached up to 1.47×. - **Limitations and release:** at short sequences (8,192 tokens on A100), the method is slightly slower than Ring Attention (0.92–0.98×). The code is open source at github.com/exists-forall/striped_attention. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,621 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 183 works ## Cite Cite as: William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song and Jonathan Ragan-Kelley. 2023. Striped Attention: Faster Ring Attention for Causal Transformers. arXiv:2311.09431. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{brandon2023striped, title = {{Striped Attention: Faster Ring Attention for Causal Transformers}}, author = {William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song and Jonathan Ragan-Kelley}, year = {2023}, doi = {10.48550/arxiv.2311.09431}, eprint = {2311.09431}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2311.09431} } ``` - [← previous SALMON: Self-Alignment with Instructable Reward Models](https://md.research.elamur.ai/free/en/papers/sun-2023-salmon-self-alignment-instructable-reward/) - [next → The MiniPile Challenge for Data-Efficient Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-minipile-challenge-data-efficient-language/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/11/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work. 1. 5.3 System optimizations for very long sequence lengths Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. JAX: composable transformations of Python+NumPy programs, 2018. URL [http://github.com/g…](http://github.com/g…) [↗](http://github.com/g…) *Updated 3 November 2026*