---
title: "RoBSA: RoPE-based Blockwise Sparse Multi-head Latent Attention"
description: "During long-context decoding, attention becomes the main cost in LLM inference. 42 references."
authors: ["Xinyu Shi", "Kairong Luo", "Zhen Zheng", "Wenguang Chen"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.46/"
doi: "10.18653/v1/2026.acl-long.46"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shi-2026-robsa-rope-based-blockwise-sparse/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shi-2026-robsa-rope-based-blockwise-sparse/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/41/) / RoBSA: RoPE-based Blockwise Sparse Multi-head Latent Attention

# RoBSA: RoPE-based Blockwise Sparse Multi-head Latent Attention

**Authors:** Xinyu Shi, Kairong Luo, Zhen Zheng and Wenguang Chen · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.46/) · [PDF](https://aclanthology.org/2026.acl-long.46.pdf) · [DOI 10.18653/v1/2026.acl-long.46](https://doi.org/10.18653/v1/2026.acl-long.46) · [OpenAlex](https://openalex.org/W7166827041)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
During long-context decoding, attention becomes the main cost in LLM inference. For DeepSeek-R1 at a 128k-token context, it takes more than 90% of decoding time. Most sparse-attention techniques are built for standard MHA or GQA. They do not fit Multi-head Latent Attention (MLA), used by DeepSeek-V3, DeepSeek-R1 and Kimi-K2, because MLA caches a compressed latent and applies RoPE to only part of each head.

The authors find two things about MLA:
- Its attention is strongly concentrated on few tokens.
- Scores computed from the small RoPE-carrying slice of query/key dimensions closely match the full scores.

RoBSA uses these cheap RoPE scores to choose blocks of the KV cache and then runs exact attention only on the selected blocks. Layers are grouped as RoPE-sensitive or RoPE-insensitive to decide where the approximation is applied. The method needs no training, plugs into existing models and targets decoding only.

On DeepSeek-V3, evaluated with InfiniteBench and LV-Eval, accuracy stays close to full attention while end-to-end decoding runs up to 2.55× faster. This makes long-context serving of very large MLA models considerably cheaper.

## Key points
- **Observations on DeepSeek-V3:**
  - The top 5% of tokens receive over 0.9 of the attention mass in nearly every layer.
  - Cosine similarity between RoPE-only and full attention scores is close to 1.0 across most layers.
- **Kernel-level results:**
  - Attention latency drops by more than 10× beyond 32k context.
  - At 128k with block size 16, active blocks shrink by up to 16×, to roughly 500 blocks (about 8,000 tokens).
- **End-to-end latency (TPOT):**
  - With θ1 = 0.9 at 128k, TPOT falls from about 175 ms to 60 ms (2.55×).
  - With θ1 = 0.95, speedups are 2.03× at 112k and 2.15× at 128k.
  - At batch size 8, throughput rises 1.79× and TPOT improves 1.77×.
- **Accuracy:**
  - On InfiniteBench, setting (θ1, θ2) = (0.9, 0.99) averages 54.52 versus 55.59 for full attention.
  - Setting (0.9, 0) collapses to 42.86, with Retrieve_KV at 6.6 and Code_run at 12.
  - On LV-Eval (16k–128k), score changes stay within roughly ±1.7 points.
- **Stated limitations:**
  - Prefilling is not accelerated.
  - The method works only for MLA models (mainly the DeepSeek family), not MHA or GQA.
  - Hyperparameters are chosen heuristically.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 325 works · 142,931 citations
- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 333 works · 87,019 citations
- [Tsinghua Shenzhen International Graduate School](https://md.research.elamur.ai/free/en/universities/tsinghua-shenzhen-international-graduate-school/) — University, Shenzhen, China · 28 works · 3,035 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

## Cite

Cite as: Xinyu Shi, Kairong Luo, Zhen Zheng and Wenguang Chen. 2026. RoBSA: RoPE-based Blockwise Sparse Multi-head Latent Attention. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shi2026robsa,
  title = {{RoBSA: RoPE-based Blockwise Sparse Multi-head Latent Attention}},
  author = {Xinyu Shi, Kairong Luo, Zhen Zheng and Wenguang Chen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.46},
  url = {https://doi.org/10.18653/v1/2026.acl-long.46}
}
```

- [← previous RoadMapper: A Multi-Agent System for Roadmap Generation of Solving Complex Research Problems](https://md.research.elamur.ai/free/en/papers/liu-2026-roadmapper-multi-agent-system-roadmap/)
- [next → Robust Language Identification for Romansh Varieties](https://md.research.elamur.ai/free/en/papers/model-2026-robust-language-identification-romansh-varieties/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/41/)

## References

42 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. [Gqa: Training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/). Preprint, arXiv:2305.13245.
2. Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, C… — [SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model](https://md.research.elamur.ai/free/en/papers/allal-2025-smollm2-smol-goes-big-data/)
3. Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2025. [Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks](https://md.research.elamur.ai/free/en/papers/bai-2024-longbench-v2-deeper-understanding/). Preprint, arXiv:2412.15204.
4. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). Preprint, arXiv:2004.05150.
5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 202… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
6. Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, and Beidi Chen. 2024. [Magicpig: Lsh sampling for efficient llm generation](https://md.research.elamur.ai/free/en/papers/chen-2024-magicpig-lsh-sampling-efficient-llm/). Preprint, arXiv:2410.16179.
7. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. [Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/). Preprint, arXiv:1904.10509.
8. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
9. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, and 3290 [other…](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/)
10. Steve Dai, Hasan Genc, Rangharajan Venkatesan, and Brucek Khailany. 2023. Efficient transformer inference with statically structured sparse attention. In 2023 60th ACM/IEEE Design Automation Conference (DAC), pages 1–6. IEEE. [find](https://scholar.google.com/scholar?q=Steve+Dai%2C+Hasan+Genc%2C+Rangharajan+Venkatesan%2C+and+Brucek+Khailany%2E+2023%2E+Efficient+transformer+inference+with+statically+structured+sparse+attention%2E+In+2023+60th+ACM%2FIEEE+Design+Automation+Conferenc)
11. DeepSeek-AI. 2025. Deepseek-v3.2-exp: Boosting long-context efficiency with deepseek sparse attention. [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2E+2025%2E+Deepseek%2Dv3%2E2%2Dexp%3A+Boosting+long%2Dcontext+efficiency+with+deepseek+sparse+attention%2E)
12. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025a. [Deepseek-r1: Incentivizing reasoning capability in l…](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/)
13. DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, and 138 others. 2024. Deepseek-v2: A strong, economical, and efficient… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Aixin+Liu%2C+Bei+Feng%2C+Bin+Wang%2C+Bingxuan+Wang%2C+Bo+Liu%2C+Chenggang+Zhao%2C+Chengqi+Dengr%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+Dejian+Yang%2C+Deli+Chen%2C+Dongjie+Ji%2C+Erhang+Li%2C+Fangyun+Lin%2C+Fuli+Luo%2C)
14. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025b. Deepseek-v3 technical report. Preprint, arXiv:… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+Dejian+Yang%2C+Deli+Chen%2C+Dongjie+Ji%2C+Erhan)
15. Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. Preprint, arXiv:2301.00774. [find](https://scholar.google.com/scholar?q=Elias+Frantar+and+Dan+Alistarh%2E+2023%2E+Sparsegpt%3A+Massive+language+models+can+be+accurately+pruned+in+one%2Dshot%2E+Preprint%2C+arXiv%3A2301%2E00774%2E)
16. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. Gptq: Accurate post-training quantization for generative pre-trained transformers. Preprint, arXiv:2210.17323. [find](https://scholar.google.com/scholar?q=Elias+Frantar%2C+Saleh+Ashkboos%2C+Torsten+Hoefler%2C+and+Dan+Alistarh%2E+2023%2E+Gptq%3A+Accurate+post%2Dtraining+quantization+for+generative+pre%2Dtrained+transformers%2E+Preprint%2C+arXiv%3A2210%2E17323%2E)
17. Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2024. [Model tells you what to discard: Adaptive kv cache compression for llms](https://md.research.elamur.ai/free/en/papers/ge-2023-model-tells-discard-adaptive-kv/). Preprint, arXiv:2310.01801.
18. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). Proceedings of the International Conference on Learning Representations (ICLR).
19. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. [Ruler: What’s the real context size of your long-context language models?](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/) arXiv preprint arXiv:2404.06654.
20. Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. [Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention](https://md.research.elamur.ai/free/en/papers/jiang-2024-minference-1-0-accelerating-pre/). Preprint, arXiv:2407.02490.
21. Greg Kamradt. 2024. Llmtest_needleinahaystack: Doing simple retrieval from llm models at various context lengths to measure accuracy. [https://github.com/gkamradt/LLMTest_NeedleInAHaystack](https://github.com/gkamradt/LLMTest_NeedleInAHaystack). Accessed: 2024-05-23. [↗](https://github.com/gkamradt/LLMTest_NeedleInAHaystack)
22. Tom Kilburn, David BG Edwards, Michael J Lanigan, and Frank H Sumner. 2009. One-level storage system. IRE Transactions on Electronic Computers, (2):223– 235. [find](https://scholar.google.com/scholar?q=Tom+Kilburn%2C+David+BG+Edwards%2C+Michael+J+Lanigan%2C+and+Frank+H+Sumner%2E+2009%2E+One%2Dlevel+storage+system%2E+IRE+Transactions+on+Electronic+Computers%2C+%282%29%3A223%E2%80%93+235%2E)
23. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). Preprint, arXiv:2309.06180.
24. Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, and 6 others. 2025. [Moba: Mixture of block attention for long-context llm…](https://md.research.elamur.ai/free/en/papers/lu-2025-moba-mixture-block-attention/)
25. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others.… — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
26. Noam Shazeer. 2019. Fast transformer decoding: One write-head is all you need. Preprint, arXiv:1911.02150. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+2019%2E+Fast+transformer+decoding%3A+One+write%2Dhead+is+all+you+need%2E+Preprint%2C+arXiv%3A1911%2E02150%2E)
27. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Preprint, arXiv:2104.09864.
28. Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. [Quest: Query-aware sparsity for efficient long-context llm inference](https://md.research.elamur.ai/free/en/papers/tang-2024-quest-query-aware-sparsity-efficient/). Preprint, arXiv:2406.10774.
29. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, and 150 others. 2025. [Kimi k2: Open agentic intelligence](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k2-open-agentic/). Preprint, a…
30. Hanrui Wang, Zhekai Zhang, and Song Han. 2021. Spatten: Efficient sparse attention architecture with cascade token and head pruning. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE. [find](https://scholar.google.com/scholar?q=Hanrui+Wang%2C+Zhekai+Zhang%2C+and+Song+Han%2E+2021%2E+Spatten%3A+Efficient+sparse+attention+architecture+with+cascade+token+and+head+pruning%2E+In+2021+IEEE+International+Symposium+on+High%2DPerformance+Computer+A)
31. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Preprint, arXiv:2201.11903.
32. Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. 2024a. [Infllm: Training-free long-context extrapolation for llms with an efficient context memory](https://md.research.elamur.ai/free/en/papers/xiao-2024-infllm-training-free-long-context/). Preprint, arXiv:2402.04617.
33. Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. 2024b. [Duoattention: Efficient long-context llm inference with retrieval and streaming heads](https://md.research.elamur.ai/free/en/papers/xiao-2024-duoattention-efficient-long-context-llm/). Preprint, arXiv:2410.10819.
34. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024c. Efficient streaming language models with attention sinks. Preprint, arXiv:2309.17453. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+2024c%2E+Efficient+streaming+language+models+with+attention+sinks%2E+Preprint%2C+arXiv%3A2309%2E17453%2E)
35. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025a. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). Preprint, arXiv:2505.09388.
36. Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. 2025b. Parallelizing linear transformers with the delta rule over sequence length. Preprint, arXiv:2406.06484. [find](https://scholar.google.com/scholar?q=Songlin+Yang%2C+Bailin+Wang%2C+Yu+Zhang%2C+Yikang+Shen%2C+and+Yoon+Kim%2E+2025b%2E+Parallelizing+linear+transformers+with+the+delta+rule+over+sequence+length%2E+Preprint%2C+arXiv%3A2406%2E06484%2E)
37. Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. 2025. [Native sparse attention: Hardware-aligned and natively trainable sparse attention](https://md.research.elamur.ai/free/en/papers/yuan-2025-native-sparse-attention-hardware/). In Proceedings…
38. Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, Guohao Dai, Shengen Yan, and Yu Wang. 2024. [Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k](https://md.research.elamur.ai/free/en/papers/yuan-2024-lv-eval-balanced-long-context/). Preprint, arXiv:2402.05136.
39. Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. 2025. Spargeattention: Accurate and training-free sparse attention accelerating any model inference. Preprint, arXiv:2502.18137. [find](https://scholar.google.com/scholar?q=Jintao+Zhang%2C+Chendong+Xiang%2C+Haofeng+Huang%2C+Jia+Wei%2C+Haocheng+Xi%2C+Jun+Zhu%2C+and+Jianfei+Chen%2E+2025%2E+Spargeattention%3A+Accurate+and+training%2Dfree+sparse+attention+accelerating+any+model+inference%2E+Prepr)
40. Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. 2024. ∞bench: Extending long context evaluation beyond 100k tokens. Preprint, arXiv:2402.13718. — [arXiv:2402.13718](https://arxiv.org/abs/2402.13718) [find](https://scholar.google.com/scholar?q=Xinrong+Zhang%2C+Yingfa+Chen%2C+Shengding+Hu%2C+Zihang+Xu%2C+Junhao+Chen%2C+Moo+Khai+Hao%2C+Xu+Han%2C+Zhen+Leng+Thai%2C+Shuo+Wang%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2024%2E+%E2%88%9Ebench%3A+Extending+long+context+evaluation+beyond+)
41. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. H 2 o: Heavy-hitter oracle for efficient generative inference of large language models. Preprint, arXiv:2306.14048. [find](https://scholar.google.com/scholar?q=Zhenyu+Zhang%2C+Ying+Sheng%2C+Tianyi+Zhou%2C+Tianlong+Chen%2C+Lianmin+Zheng%2C+Ruisi+Cai%2C+Zhao+Song%2C+Yuandong+Tian%2C+Christopher+R%C3%A9%2C+Clark+Barrett%2C+Zhangyang+Wang%2C+and+Beidi+Chen%2E+2023%2E+H+2+o%3A+Heavy%2Dhitter+oracl)
42. Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong, and Yu Wang. 2024. A survey on efficient inference for large language models. Preprint, arXiv:2404.14294. [find](https://scholar.google.com/scholar?q=Zixuan+Zhou%2C+Xuefei+Ning%2C+Ke+Hong%2C+Tianyu+Fu%2C+Jiaming+Xu%2C+Shiyao+Li%2C+Yuming+Lou%2C+Luning+Wang%2C+Zhihang+Yuan%2C+Xiuhong+Li%2C+Shengen+Yan%2C+Guohao+Dai%2C+Xiao%2DPing+Zhang%2C+Yuhan+Dong%2C+and+Yu+Wang%2E+2024%2E+A+surve)

*Updated 1 November 2026*