---
title: "DELTA — paper by Zarch et al. (2026) — summary and ideas"
description: "They build on this with DELTA, a training-free, plug-in sparse attention scheme: Early layers keep full… 43 references, cited by 1 paper here."
authors: ["Hossein Entezari Zarch", "Lei Gao", "Chaoyi Jiang", "Murali Annavaram"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.558/"
doi: "10.18653/v1/2026.findings-acl.558"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zarch-2026-delta-dynamic-layer-aware-token/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zarch-2026-delta-dynamic-layer-aware-token/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/12/) / DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning

# DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning

**Authors:** Hossein Entezari Zarch, Lei Gao, Chaoyi Jiang and [Murali Annavaram](https://md.research.elamur.ai/free/en/authors/murali-annavaram/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.558/) · [PDF](https://aclanthology.org/2026.findings-acl.558.pdf) · [DOI 10.18653/v1/2026.findings-acl.558](https://doi.org/10.18653/v1/2026.findings-acl.558) · [OpenAlex](https://openalex.org/W7166874247)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large reasoning models spend most of their inference time decoding, because every generated token must read the entire, ever-growing KV cache from GPU memory. Sparse attention can lower this cost, but on long reasoning traces existing methods lose accuracy. Eviction can discard tokens that only become relevant later, and selection mistakes accumulate over the derivation. The authors analyze attention in reasoning models and find two patterns: adjacent layers tend to focus on very similar tokens, and token importance shifts only gradually during generation.

They build on this with DELTA, a training-free, plug-in sparse attention scheme:
- Early layers keep full attention.
- A few designated ∆-layers compute full attention and rank tokens by aggregating head-level scores.
- The remaining layers attend only to the selected tokens plus a recency window.

The full KV cache stays in GPU memory. On AIME and GPQA-Diamond, using distilled DeepSeek-R1 models, DELTA matches or exceeds full-attention accuracy and gives up to 1.54× end-to-end speedup. This makes it a low-cost route to faster reasoning-time inference.

## Key points
- **Architecture:** Transformer layers are split into three groups: full-attention initial layers, a small set of ∆-layers that pick salient tokens from attention maps, and sparse layers that reuse those selections. Selection combines head-aware token scoring with a stable recency window.
- **Results:** Compared with state-of-the-art sparse attention methods, DELTA attends to up to 4.25× fewer tokens without losing accuracy. It reaches up to 1.54× end-to-end speedup, with accuracy equal to or better than full attention on AIME and GPQA-Diamond.
- **Motivation:** With full attention in HuggingFace, DeepSeek-R1-Distill-Llama-8B needs over 15 minutes on one A100 to generate 32K tokens for a single AIME problem.
- **Ablation:** On DS-Qwen-7B (Mixed120), the authors vary the page budget K and the recency window L. Tight budgets, such as 64 pages (about 1k tokens), work better with a larger recency window. Larger budgets benefit more from broader coverage of salient tokens.
- **Limitations and code:** DELTA cuts compute and bandwidth but not peak KV memory. It was tested only on distilled DeepSeek-R1 models for math and science QA, and it is sensitive to ∆-layer placement and to (K, L). Code is available at github.com/hoenza/DELTA.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Southern California](https://md.research.elamur.ai/free/en/universities/university-of-southern-california/) — University, Los Angeles, United States · 162 works · 35,836 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Hossein Entezari Zarch, Lei Gao, Chaoyi Jiang and Murali Annavaram. 2026. DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zarch2026delta,
  title = {{DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning}},
  author = {Hossein Entezari Zarch, Lei Gao, Chaoyi Jiang and Murali Annavaram},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.558},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.558}
}
```

- [← previous Defensive Dual Masking for Robust Adversarial Defense](https://md.research.elamur.ai/free/en/papers/yang-2026-defensive-dual-masking-robust-adversarial/)
- [next → DemMA: Dementia Multi-Turn Dialogue Agent with Expert-Guided Reasoning and Action Simulation](https://md.research.elamur.ai/free/en/papers/song-2026-demma-dementia-multi-turn-dialogue/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/12/)

## References

43 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J Nair, Ilya Soloveychik, and Purushotham Kamath. 2024. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems, 6:114–127. [find](https://scholar.google.com/scholar?q=Muhammad+Adnan%2C+Akhil+Arunkumar%2C+Gaurav+Jain%2C+Prashant+J+Nair%2C+Ilya+Soloveychik%2C+and+Purushotham+Kamath%2E+2024%2E+Keyformer%3A+Kv+cache+reduction+through+key+tokens+selection+for+efficient+generative+infer)
2. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). arXiv preprint arXiv:2004.05150.
3. Zefan Cai, Wen Xiao, Hanshi Sun, Cheng Luo, Yikai Zhang, Ke Wan, Yucheng Li, Yeyang Zhou, Li-Wen Chang, Jiuxiang Gu, and 1 others. 2025. Rkv: Redundancy-aware kv cache compression for training-free reasoning models acceleration. arXiv preprint arXiv:2505.24133. — [arXiv:2505.24133](https://arxiv.org/abs/2505.24133) [find](https://scholar.google.com/scholar?q=Zefan+Cai%2C+Wen+Xiao%2C+Hanshi+Sun%2C+Cheng+Luo%2C+Yikai+Zhang%2C+Ke+Wan%2C+Yucheng+Li%2C+Yeyang+Zhou%2C+Li%2DWen+Chang%2C+Jiuxiang+Gu%2C+and+1+others%2E+2025%2E+Rkv%3A+Redundancy%2Daware+kv+cache+compression+for+training%2Dfree+re)
4. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. [Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/). arXiv preprint arXiv:1904.10509.
5. Tri Dao. 2023. [Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). arXiv preprint arXiv:2307.08691.
6. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems, 35:30318– 30332. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Mike+Lewis%2C+Younes+Belkada%2C+and+Luke+Zettlemoyer%2E+2022%2E+Gpt3%2E+int8+%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2E+Advances+in+neural+information+processing+systems%2C+35%3A30318%E2%80%93)
7. Yizhao Gao, Shuming Guo, Shijie Cao, Yuqing Xia, Yu Cheng, Lei Wang, Lingxiao Ma, Yutao Sun, Tianzhu Ye, Li Dong, and 1 others. 2025. Seerattention-r: Sparse attention adaptation for long reasoning. arXiv preprint arXiv:2506.08889. [find](https://scholar.google.com/scholar?q=Yizhao+Gao%2C+Shuming+Guo%2C+Shijie+Cao%2C+Yuqing+Xia%2C+Yu+Cheng%2C+Lei+Wang%2C+Lingxiao+Ma%2C+Yutao+Sun%2C+Tianzhu+Ye%2C+Li+Dong%2C+and+1+others%2E+2025%2E+Seerattention%2Dr%3A+Sparse+attention+adaptation+for+long+reasoning%2E+a)
8. Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2023. [Model tells you what to discard: Adaptive kv cache compression for llms](https://md.research.elamur.ai/free/en/papers/ge-2023-model-tells-discard-adaptive-kv/). arXiv preprint arXiv:2310.01801.
9. Google DeepMind. 2025. Gemini 2.5: Our most intelligent ai model. [https://blog.google/techno](https://blog.google/techno) logy/google-deepmind/gemini-model-thinkin g-updates-march-2025/. Accessed: 2025-09-27. [↗](https://blog.google/techno)
10. Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. [find](https://scholar.google.com/scholar?q=Albert+Gu+and+Tri+Dao%2E+2023%2E+Mamba%3A+Linear%2Dtime+sequence+modeling+with+selective+state+spaces%2E+arXiv+preprint+arXiv%3A2312%2E00752%2E)
11. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
12. MAA AIME. 2025. American invitational mathematics examination-aime 2024, 2024. [find](https://scholar.google.com/scholar?q=MAA+AIME%2E+2025%2E+American+invitational+mathematics+examination%2Daime+2024%2C+2024%2E)
13. Jitai Hao, Yuke Zhu, Tian Wang, Jun Yu, Xin Xin, Bo Zheng, Zhaochun Ren, and Sheng Guo. 2025. Omnikv: Dynamic context selection for efficient long-context llms. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jitai+Hao%2C+Yuke+Zhu%2C+Tian+Wang%2C+Jun+Yu%2C+Xin+Xin%2C+Bo+Zheng%2C+Zhaochun+Ren%2C+and+Sheng+Guo%2E+2025%2E+Omnikv%3A+Dynamic+context+selection+for+efficient+long%2Dcontext+llms%2E+In+The+Thirteenth+International+Confere)
14. Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. [Gqa: Training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/). arXiv preprint arXiv:2305.13245.
15. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874.
16. Junhao Hu, Wenrui Huang, Weidong Wang, Zhenwen Li, Tiancheng Hu, Zhixia Liu, Xusheng Chen, Tao Xie, and Yizhou Shan. 2025. Raas: Reasoning-aware attention sparsity for efficient llm reasoning. arXiv preprint arXiv:2502.11147. [find](https://scholar.google.com/scholar?q=Junhao+Hu%2C+Wenrui+Huang%2C+Weidong+Wang%2C+Zhenwen+Li%2C+Tiancheng+Hu%2C+Zhixia+Liu%2C+Xusheng+Chen%2C+Tao+Xie%2C+and+Yizhou+Shan%2E+2025%2E+Raas%3A+Reasoning%2Daware+attention+sparsity+for+efficient+llm+reasoning%2E+arXiv+p)
17. Hugging Face. 2025. Hugging face. https://huggin gface.co/. Accessed: 2025-09-27. [find](https://scholar.google.com/scholar?q=Hugging+Face%2E+2025%2E+Hugging+face%2E+https%3A%2F%2Fhuggin+gface%2Eco%2F%2E+Accessed%3A+2025%2D09%2D27%2E)
18. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the 29th symposium on operating systems principles, pages 611–626.
19. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. [Snapkv: Llm knows what you are looking for before generation](https://md.research.elamur.ai/free/en/papers/li-2024-snapkv-llm-knows-looking-before/). Advances in Neural Information Processing Systems, 37:22947–22970.
20. Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, and 1 others. 2024a. Retrievalattention: Accelerating long-context llm inference via vector retrieval. arXiv preprint arXiv:2409.10516. [find](https://scholar.google.com/scholar?q=Di+Liu%2C+Meng+Chen%2C+Baotong+Lu%2C+Huiqiang+Jiang%2C+Zhenhua+Han%2C+Qianxi+Zhang%2C+Qi+Chen%2C+Chengruidong+Zhang%2C+Bailu+Ding%2C+Kai+Zhang%2C+and+1+others%2E+2024a%2E+Retrievalattention%3A+Accelerating+long%2Dcontext+llm+inf)
21. Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2023. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information Processing Systems, 36:5… [find](https://scholar.google.com/scholar?q=Zichang+Liu%2C+Aditya+Desai%2C+Fangshuo+Liao%2C+Weitao+Wang%2C+Victor+Xie%2C+Zhaozhuo+Xu%2C+Anastasios+Kyrillidis%2C+and+Anshumali+Shrivastava%2E+2023%2E+Scissorhands%3A+Exploiting+the+persistence+of+importance+hypothesi)
22. Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024b. [Kivi: A tuning-free asymmetric 2bit quantization for kv cache](https://md.research.elamur.ai/free/en/papers/liu-2024-kivi-tuning-free-asymmetric-2bit/). arXiv preprint arXiv:2402.02750.
23. OpenAI. 2025a. Introducing GPT-OSS. [https://openai.com/index/introducing-gpt-oss/](https://openai.com/index/introducing-gpt-oss/). Accessed: 2025-09-27. [↗](https://openai.com/index/introducing-gpt-oss/)
24. OpenAI. 2025b. Introducing openai o3 and o4-mini. [https://openai.com/index/introducing-o3-a](https://openai.com/index/introducing-o3-a) nd-o4-mini/. Accessed: 2025-09-27. [↗](https://openai.com/index/introducing-o3-a)
25. Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz. 2024. Transformers are multistate rnns. arXiv preprint arXiv:2401.06104. — [arXiv:2401.06104](https://arxiv.org/abs/2401.06104) [find](https://scholar.google.com/scholar?q=Matanel+Oren%2C+Michael+Hassid%2C+Nir+Yarden%2C+Yossi+Adi%2C+and+Roy+Schwartz%2E+2024%2E+Transformers+are+multistate+rnns%2E+arXiv+preprint+arXiv%3A2401%2E06104%2E+%E2%80%94+%5BarXiv%3A2401%2E06104%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2401%2E06104%29)
26. Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, and 1 others. 2023. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048. [find](https://scholar.google.com/scholar?q=Bo+Peng%2C+Eric+Alcaide%2C+Quentin+Anthony%2C+Alon+Albalak%2C+Samuel+Arcadinho%2C+Stella+Biderman%2C+Huanqi+Cao%2C+Xin+Cheng%2C+Michael+Chung%2C+Matteo+Grella%2C+and+1+others%2E+2023%2E+Rwkv%3A+Reinventing+rnns+for+the+transfo)
27. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. [Gpqa: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). In First Conference on Language Modeling.
28. Noam Shazeer. 2019. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+2019%2E+Fast+transformer+decoding%3A+One+write%2Dhead+is+all+you+need%2E+arXiv+preprint+arXiv%3A1911%2E02150%2E)
29. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063.
30. Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. 2023. [Retentive network: A successor to transformer for large language models](https://md.research.elamur.ai/free/en/papers/sun-2023-retentive-network-successor-transformer-large/). arXiv preprint arXiv:2307.08621.
31. Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. [Quest: Query-aware sparsity for efficient long-context llm inference](https://md.research.elamur.ai/free/en/papers/tang-2024-quest-query-aware-sparsity-efficient/). arXiv preprint arXiv:2406.10774.
32. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30.
33. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824– 24837.
34. Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023a. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning, pages 38087– 38099. PMLR. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Ji+Lin%2C+Mickael+Seznec%2C+Hao+Wu%2C+Julien+Demouth%2C+and+Song+Han%2E+2023a%2E+Smoothquant%3A+Accurate+and+efficient+post%2Dtraining+quantization+for+large+language+models%2E+In+International+conferen)
35. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023b. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+2023b%2E+Efficient+streaming+language+models+with+attention+sinks%2E+arXiv+preprint+arXiv%3A2309%2E17453%2E)
36. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
37. Lijie Yang, Zhihao Zhang, Zhuofu Chen, Zikun Li, and Zhihao Jia. 2024. Tidaldecode: Fast and accurate llm decoding with position persistent sparse attention. arXiv preprint arXiv:2410.05076. — [arXiv:2410.05076](https://arxiv.org/abs/2410.05076) [find](https://scholar.google.com/scholar?q=Lijie+Yang%2C+Zhihao+Zhang%2C+Zhuofu+Chen%2C+Zikun+Li%2C+and+Zhihao+Jia%2E+2024%2E+Tidaldecode%3A+Fast+and+accurate+llm+decoding+with+position+persistent+sparse+attention%2E+arXiv+preprint+arXiv%3A2410%2E05076%2E+%E2%80%94+%5BarXiv%3A)
38. Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. 2022. [Zeroquant: Efficient and affordable post-training quantization for large-scale transformers](https://md.research.elamur.ai/free/en/papers/yao-2022-zeroquant-efficient-affordable-post-training/). Advances in neural information processing systems, 35:27168– 27183.
39. Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, and 1 others. 2025. Flashinfer: Efficient and customizable attention engine for llm inference serving. arXiv preprint arXiv:2501.01005. [find](https://scholar.google.com/scholar?q=Zihao+Ye%2C+Lequn+Chen%2C+Ruihang+Lai%2C+Wuwei+Lin%2C+Yineng+Zhang%2C+Stephanie+Wang%2C+Tianqi+Chen%2C+Baris+Kasikci%2C+Vinod+Grover%2C+Arvind+Krishnamurthy%2C+and+1+others%2E+2025%2E+Flashinfer%3A+Efficient+and+customizable+a)
40. Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, and 1 others. 2025. [Native sparse attention: Hardware-aligned and natively trainable sparse attention](https://md.research.elamur.ai/free/en/papers/yuan-2025-native-sparse-attention-hardware/). arXiv preprint arXiv:2502.11089.
41. Linan Yue, Yichao Du, Yizhi Wang, Weibo Gao, Fangzhou Yao, Li Wang, Ye Liu, Ziyu Xu, Qi Liu, Shimin Di, and 1 others. 2025. Don’t overthink it: A survey of efficient r1-style large reasoning models. arXiv preprint arXiv:2508.02120. [find](https://scholar.google.com/scholar?q=Linan+Yue%2C+Yichao+Du%2C+Yizhi+Wang%2C+Weibo+Gao%2C+Fangzhou+Yao%2C+Li+Wang%2C+Ye+Liu%2C+Ziyu+Xu%2C+Qi+Liu%2C+Shimin+Di%2C+and+1+others%2E+2025%2E+Don%E2%80%99t+overthink+it%3A+A+survey+of+efficient+r1%2Dstyle+large+reasoning+models%2E+a)
42. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and 1 others. 2020. [Big bird: Transformers for longer sequences](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/). Advances in neural information processing systems, 33:17283–17297.
43. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, and 1 others. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:346… [find](https://scholar.google.com/scholar?q=Zhenyu+Zhang%2C+Ying+Sheng%2C+Tianyi+Zhou%2C+Tianlong+Chen%2C+Lianmin+Zheng%2C+Ruisi+Cai%2C+Zhao+Song%2C+Yuandong+Tian%2C+Christopher+R%C3%A9%2C+Clark+Barrett%2C+and+1+others%2E+2023%2E+H2o%3A+Heavy%2Dhitter+oracle+for+efficient+gene)

*Updated 31 October 2026*