---
title: "XAttention: Block Sparse Attention with Antidiagonal Scoring (2025)"
description: "Long-context Transformers are costly because attention grows quadratically with input length, which… 44 references, cited by 58 papers here."
authors: ["Ruyi Xu", "Guangxuan Xiao", "Haofeng Huang", "Junxian Guo", "Song Han"]
year: 2025
venue: "arXiv:2503.16428"
source: "https://arxiv.org/abs/2503.16428"
doi: "10.48550/arxiv.2503.16428"
arxiv: "2503.16428"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2025-xattention-block-sparse-attention-antidiagonal/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2025-xattention-block-sparse-attention-antidiagonal/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) / XAttention: Block Sparse Attention with Antidiagonal Scoring

# XAttention: Block Sparse Attention with Antidiagonal Scoring

**Authors:** Ruyi Xu, [Guangxuan Xiao](https://md.research.elamur.ai/free/en/authors/guangxuan-xiao/), Haofeng Huang, [Junxian Guo](https://md.research.elamur.ai/free/en/authors/junxian-guo/) and [Song Han](https://md.research.elamur.ai/free/en/authors/song-han/) · **Year:** 2025 · **Venue:** arXiv:2503.16428 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2503.16428) · [PDF](https://arxiv.org/pdf/2503.16428) · [DOI 10.48550/arxiv.2503.16428](https://doi.org/10.48550/arxiv.2503.16428) · [arXiv 2503.16428](https://arxiv.org/abs/2503.16428) · [OpenAlex](https://openalex.org/W4415289316)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [58](https://md.research.elamur.ai/free/en/papers/xu-2025-xattention-block-sparse-attention-antidiagonal/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Long-context Transformers are costly because attention grows quadratically with input length, which makes the pre-filling stage a bottleneck. Block-sparse attention can skip unimportant regions of the attention map. However, earlier approaches such as MInference and FlexPrefill spend so much effort estimating which blocks matter that much of the speedup is lost, and simple pooling misses sparse vertical or slash patterns. XAttention is a training-free, plug-and-play alternative. Inside each block, it sums attention values sampled along strided antidiagonals, running from lower-left to upper-right. These lines cross every vertical and diagonal pattern in the block and touch every token. After softmax normalization, the method keeps the smallest set of blocks whose cumulative score exceeds a threshold τ. An optional dynamic-programming search tunes τ separately for each head. Experiments span the language benchmarks RULER and LongBench, the VideoMME benchmark for comprehending video, and VBench for producing video. Quality stays close to full attention while attention computation becomes up to 13.5× faster. This makes sparse prefill practical for long text and multimodal inputs.

## Key points
- Core idea: the sum of strided antidiagonal entries within a block serves as a cheap importance score. It replaces the pooling and vertical-slash search used by MInference and FlexPrefill, whose overhead the authors highlight.
- Block selection: scores are softmax-normalized, and the minimal set of blocks reaching cumulative threshold τ is kept. The pipeline has three stages: scoring, selection, and sparse attention computation.
- Optional per-head threshold tuning uses a DP table D[h][m] and lowers a head's threshold by 10% per step (t ← 0.9·t). The paper presents this step as non-mandatory and examines it in ablations.
- Evaluation covers RULER and LongBench with Llama-3.1-8B-Instruct, VideoMME for video understanding, and VBench prompts for video generation. Results are compared against full attention and sparse baselines.
- Reported speedup is up to 13.5× in attention computation during pre-filling. The authors are from Tsinghua University, MIT, SJTU and NVIDIA, and the code is at github.com/mit-han-lab/x-attention.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 242 works

## Cite

Cite as: Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo and Song Han. 2025. XAttention: Block Sparse Attention with Antidiagonal Scoring. arXiv:2503.16428.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{xu2025xattention,
  title = {{XAttention: Block Sparse Attention with Antidiagonal Scoring}},
  author = {Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo and Song Han},
  year = {2025},
  doi = {10.48550/arxiv.2503.16428},
  eprint = {2503.16428},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.16428}
}
```

- [← previous mmBERT: A Modern Multilingual Encoder with Annealed Language Learning](https://md.research.elamur.ai/free/en/papers/marone-2025-mmbert-modern-multilingual-encoder-annealed/)
- [next → BIG-Bench Extra Hard](https://md.research.elamur.ai/free/en/papers/kazemi-2025-big-bench-extra-hard/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/)

## References

44 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anonymous. Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference. In The Thirteenth International Conference on Learning Representations, 2025. URL [https://openreview.net/forum?id=OfjIlbelrT](https://openreview.net/forum?id=OfjIlbelrT). [↗](https://openreview.net/forum?id=OfjIlbelrT)
2. Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J., and Li, J. [Longbench: A bilingual, multitask benchmark for long context understanding](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/). arXiv preprint arXiv:2308.14508, 2023.
3. Beltagy, I., Peters, M. E., and Cohan, A. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/), 2020. arXiv:2004.05150.
4. Burtsev, M. S., Kuratov, Y., Peganov, A., and Sapunov, G. V. Memory transformer, 2021. URL https://arxiv.org/abs/2006.11527. — [arXiv:2006.11527](https://arxiv.org/abs/2006.11527) [↗](https://arxiv.org/abs/2006.11527)
5. Chen, S., Wong, S., Chen, L., and Tian, Y. Extending context window of large language models via positional interpolation, 2023. arXiv: 2306.15595. [find](https://scholar.google.com/scholar?q=Chen%2C+S%2E%2C+Wong%2C+S%2E%2C+Chen%2C+L%2E%2C+and+Tian%2C+Y%2E+Extending+context+window+of+large+language+models+via+positional+interpolation%2C+2023%2E+arXiv%3A+2306%2E15595%2E)
6. Child, R., Gray, S., Radford, A., and Sutskever, I. [Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/). 2019a.
7. Child, R., Gray, S., Radford, A., and Sutskever, I. [Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/), 2019b. URL https://arxiv.org/abs/1904.10509. [↗](https://arxiv.org/abs/1904.10509)
8. Dao, T. [FlashAttention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/), 2023.
9. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. FlashAttention: Fast and memory-efficient exact attention with IO-awareness, 2022. arXiv:2205.14135. [find](https://scholar.google.com/scholar?q=Dao%2C+T%2E%2C+Fu%2C+D%2E+Y%2E%2C+Ermon%2C+S%2E%2C+Rudra%2C+A%2E%2C+and+Re%CC%81%2C+C%2E+FlashAttention%3A+Fast+and+memory%2Defficient+exact+attention+with+IO%2Dawareness%2C+2022%2E+arXiv%3A2205%2E14135%2E)
10. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
11. Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al. [Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis](https://md.research.elamur.ai/free/en/papers/fu-2024-video-mme-first-ever-comprehensive/). arXiv preprint arXiv:2405.21075, 2024.
12. Gao, Y., Zeng, Z., Du, D., Cao, S., So, H. K.-H., Cao, T., Yang, F., and Yang, M. Seerattention: Learning intrinsic sparse attention in your llms. arXiv preprint arXiv:2410.13276, 2024. [find](https://scholar.google.com/scholar?q=Gao%2C+Y%2E%2C+Zeng%2C+Z%2E%2C+Du%2C+D%2E%2C+Cao%2C+S%2E%2C+So%2C+H%2E+K%2E%2DH%2E%2C+Cao%2C+T%2E%2C+Yang%2C+F%2E%2C+and+Yang%2C+M%2E+Seerattention%3A+Learning+intrinsic+sparse+attention+in+your+llms%2E+arXiv+preprint+arXiv%3A2410%2E13276%2C+2024%2E)
13. Guo, J., Tang, H., Yang, S., Zhang, Z., Liu, Z., and Han, S. Block Sparse Attention. [https://github.com/](https://github.com/) mit-han-lab/Block-Sparse-Attention, 2024. [↗](https://github.com/)
14. Hong, K., Dai, G., Xu, J., Mao, Q., Li, X., Liu, J., Chen, K., Dong, Y., and Wang, Y. Flashdecoding++: Faster large language model inference on gpus, 2024. [find](https://scholar.google.com/scholar?q=Hong%2C+K%2E%2C+Dai%2C+G%2E%2C+Xu%2C+J%2E%2C+Mao%2C+Q%2E%2C+Li%2C+X%2E%2C+Liu%2C+J%2E%2C+Chen%2C+K%2E%2C+Dong%2C+Y%2E%2C+and+Wang%2C+Y%2E+Flashdecoding%2B%2B%3A+Faster+large+language+model+inference+on+gpus%2C+2024%2E)
15. Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg, B. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.[06654, 2024](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/).
16. Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., Wang, Y., Chen, X., Wang, L., Lin, D., Qiao, Y., and Liu, Z. VBench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter… [find](https://scholar.google.com/scholar?q=Huang%2C+Z%2E%2C+He%2C+Y%2E%2C+Yu%2C+J%2E%2C+Zhang%2C+F%2E%2C+Si%2C+C%2E%2C+Jiang%2C+Y%2E%2C+Zhang%2C+Y%2E%2C+Wu%2C+T%2E%2C+Jin%2C+Q%2E%2C+Chanpaisit%2C+N%2E%2C+Wang%2C+Y%2E%2C+Chen%2C+X%2E%2C+Wang%2C+L%2E%2C+Lin%2C+D%2E%2C+Qiao%2C+Y%2E%2C+and+Liu%2C+Z%2E+VBench%3A+Comprehensive+benchmark+suite+)
17. Jiang, H., Li, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L. [Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention](https://md.research.elamur.ai/free/en/papers/jiang-2024-minference-1-0-accelerating-pre/). arXiv preprint arXiv:2407.02490, 2024.
18. Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., Xiong, J., Li, X., Wu, B., Zhang, J., Wu, K., Lin, Q., Yuan, J., Long, Y., Wang, A., Wang, A., Li, C., Huang, D., Yang, F., Tan, H., Wang, H., Song, J., Bai, J., Wu, J., Xue, J., Wang, J., Wang, K., Liu, M., Li, P., Li, S., Wang, W., Yu, W.,… — [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://md.research.elamur.ai/free/en/papers/kong-2024-hunyuanvideo-systematic-framework-large/)
19. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/), 2023.
20. Leviathan, Y., Kalman, M., and Matias, Y. Selective attention improves transformer, 2024. URL https://arxiv.org/abs/2410.02703. perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. — [arXiv:2410.02703](https://arxiv.org/abs/2410.02703) [↗](https://arxiv.org/abs/2410.02703)
21. Wu, W., Wang, Y., Xiao, G., Peng, H., and Fu, Y. [Retrieval head mechanistically explains long-context factuality](https://md.research.elamur.ai/free/en/papers/wu-2024-retrieval-head-mechanistically-explains-long/), 2024.
22. Li, M., Cai, T., Cao, J., Zhang, Q., Cai, H., Bai, J., Jia, Y., Liu, M.-Y., Li, K., and Han, S. Distrifusion: Distributed parallel inference for high-resolution diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. [find](https://scholar.google.com/scholar?q=Li%2C+M%2E%2C+Cai%2C+T%2E%2C+Cao%2C+J%2E%2C+Zhang%2C+Q%2E%2C+Cai%2C+H%2E%2C+Bai%2C+J%2E%2C+Jia%2C+Y%2E%2C+Liu%2C+M%2E%2DY%2E%2C+Li%2C+K%2E%2C+and+Han%2C+S%2E+Distrifusion%3A+Distributed+parallel+inference+for+high%2Dresolution+diffusion+models%2E+In+Proceedings+of+the)
23. Xi, H., Yang, S., Zhao, Y., Xu, C., Li, M., Li, X., Lin, Y., Cai, H., Zhang, J., Li, D., Chen, J., Stoica, I., Keutzer, K., and Han, S. Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity, 2025. URL https://arxiv.org/abs/2502.01776. [↗](https://arxiv.org/abs/2502.01776)
24. Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., and Yuan, L. [Video-llava: Learning united visual representation by alignment before projection](https://md.research.elamur.ai/free/en/papers/lin-2023-video-llava-learning-united-visual/), 2023.
25. Xiao, C., Zhang, P., Han, X., Xiao, G., Lin, Y., Zhang, Z., Liu, Z., and Sun, M. [Infllm: Training-free long-context extrapolation for llms with an efficient context memory](https://md.research.elamur.ai/free/en/papers/xiao-2024-infllm-training-free-long-context/), 2024a. URL https://arxiv.org/abs/ 2402.04617. [↗](https://arxiv.org/abs/)
26. Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for llm compression and acceleration, 2024. [find](https://scholar.google.com/scholar?q=Lin%2C+J%2E%2C+Tang%2C+J%2E%2C+Tang%2C+H%2E%2C+Yang%2C+S%2E%2C+Chen%2C+W%2E%2DM%2E%2C+Wang%2C+W%2E%2DC%2E%2C+Xiao%2C+G%2E%2C+Dang%2C+X%2E%2C+Gan%2C+C%2E%2C+and+Han%2C+S%2E+Awq%3A+Activation%2Daware+weight+quantization+for+llm+compression+and+acceleration%2C+2024%2E)
27. Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, 2023a. [find](https://scholar.google.com/scholar?q=Xiao%2C+G%2E%2C+Lin%2C+J%2E%2C+Seznec%2C+M%2E%2C+Wu%2C+H%2E%2C+Demouth%2C+J%2E%2C+and+Han%2C+S%2E+SmoothQuant%3A+Accurate+and+efficient+post%2Dtraining+quantization+for+large+language+models%2E+In+Proceedings+of+the+40th+International+Confe)
28. Lin*, Y., Tang*, H., Yang*, S., Zhang, Z., Xiao, G., Gan, C., and Han, S. Qserve: W4a8kv4 quantization and system co-design for efficient llm serving. arXiv preprint arXiv:2405.04532, 2024. [find](https://scholar.google.com/scholar?q=Lin%2A%2C+Y%2E%2C+Tang%2A%2C+H%2E%2C+Yang%2A%2C+S%2E%2C+Zhang%2C+Z%2E%2C+Xiao%2C+G%2E%2C+Gan%2C+C%2E%2C+and+Han%2C+S%2E+Qserve%3A+W4a8kv4+quantization+and+system+co%2Ddesign+for+efficient+llm+serving%2E+arXiv+preprint+arXiv%3A2405%2E04532%2C+2024%2E)
29. Liu, H., Zaharia, M., and Abbeel, P. [Ring attention with blockwise transformers for near-infinite context](https://md.research.elamur.ai/free/en/papers/liu-2023-ring-attention-blockwise-transformers-near/), 2023.
30. Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. arXiv, 2023b. [find](https://scholar.google.com/scholar?q=Xiao%2C+G%2E%2C+Tian%2C+Y%2E%2C+Chen%2C+B%2E%2C+Han%2C+S%2E%2C+and+Lewis%2C+M%2E+Efficient+streaming+language+models+with+attention+sinks%2E+arXiv%2C+2023b%2E)
31. Lu, E., Jiang, Z., Liu, J., Du, Y., Jiang, T., Hong, C., Liu, S., He, W., Yuan, E., Wang, Y., Huang, Z., Yuan, H., Xu, S., Xu, X., Lai, G., Chen, Y., Zheng, H., Yan, J., Su, J., Wu, Y., Zhang, N. Y., Yang, Z., Zhou, X., Zhang, M., and Qiu, J. [Moba: Mixture of block attention for long-context llms](https://md.research.elamur.ai/free/en/papers/lu-2025-moba-mixture-block-attention/), 2…
32. Xiao, G., Yin, T., Freeman, W. T., Durand, F., and Han, S. Fastcomposer: Tuning-free multi-subject image generation with localized attention, 2023c. URL https://arxiv.org/abs/2305.10431. — [arXiv:2305.10431](https://arxiv.org/abs/2305.10431) [↗](https://arxiv.org/abs/2305.10431)
33. Xiao, G., Tang, J., Zuo, J., Guo, J., Yang, S., Tang, H., Fu, Y., and Han, S. [Duoattention: Efficient long-context llm inference with retrieval and streaming heads](https://md.research.elamur.ai/free/en/papers/xiao-2024-duoattention-efficient-long-context-llm/). arXiv, 2024b.
34. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023.
35. Oren, M., Hassid, M., Yarden, N., Adi, Y., and Schwartz, R. Transformers are multi-state rnns, 2024. URL https://arxiv.org/abs/2401.06104. [↗](https://arxiv.org/abs/2401.06104)
36. Ye, Z., Lai, R., Lu, R., Lin, C.-Y., Zheng, S., Chen, L., Chen, T., and Ceze, L. Cascade inference: Memory bandwidth efficient shared prefix batch decoding. [https://flashinfer.ai/](https://flashinfer.ai/) 2024/01/08/cascade-inference.html, Jan 2024. URL https://flashinfer.ai/2024/ 01/08/cascade-inference.html. Accessed on 2… [↗](https://flashinfer.ai/)
37. Peebles, W. and Xie, S. [Scalable diffusion models with transformers](https://md.research.elamur.ai/free/en/papers/peebles-2022-scalable-diffusion-models-transformers/), 2023. URL https://arxiv.org/abs/ 2212.09748. [↗](https://arxiv.org/abs/)
38. Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Efficient context window extension of large language models, 2023. [find](https://scholar.google.com/scholar?q=Peng%2C+B%2E%2C+Quesnelle%2C+J%2E%2C+Fan%2C+H%2E%2C+and+Shippole%2C+E%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2C+2023%2E)
39. Yuan, J., Gao, H., Dai, D., Luo, J., Zhao, L., Zhang, Z., Xie, Z., Wei, Y. X., Wang, L., Xiao, Z., Wang, Y., Ruan, C., Zhang, M., Liang, W., and Zeng, W. [Native sparse attention: Hardware-aligned and natively trainable sparse attention](https://md.research.elamur.ai/free/en/papers/yuan-2025-native-sparse-attention-hardware/), 2025. URL https://arxiv.org/abs/ 2502.11089. [↗](https://arxiv.org/abs/)
40. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
41. Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing vision-language model’s Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Alberti, C., Ontañón… — [Big Bird: Transformers for Longer Sequences](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/)
42. Curran Associates, Inc., 2020. [find](https://scholar.google.com/scholar?q=Curran+Associates%2C+Inc%2E%2C+2020%2E)
43. Zhang, P., Chen, Y., Su, R., Ding, H., Stoica, I., Liu, Z., and Zhang, H. Fast video generation with sliding tile attention, 2025. URL https://arxiv.org/abs/ 2502.04507. [↗](https://arxiv.org/abs/)
44. Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., Wang, Z., and Chen, B. H 2 o: Heavy-hitter oracle for efficient generative inference of large language models, 2023. [find](https://scholar.google.com/scholar?q=Zhang%2C+Z%2E%2C+Sheng%2C+Y%2E%2C+Zhou%2C+T%2E%2C+Chen%2C+T%2E%2C+Zheng%2C+L%2E%2C+Cai%2C+R%2E%2C+Song%2C+Z%2E%2C+Tian%2C+Y%2E%2C+Re%CC%81%2C+C%2E%2C+Barrett%2C+C%2E%2C+Wang%2C+Z%2E%2C+and+Chen%2C+B%2E+H+2+o%3A+Heavy%2Dhitter+oracle+for+efficient+generative+inference+of+large+la)

*Updated 29 October 2026*