--- title: "Native Sparse Attention — Yuan 2025" description: "Standard softmax attention becomes the main cost in long-context language models." authors: ["Jingyang Yuan", "Huazuo Gao", "Damai Dai", "Junyu Luo", "Liang Zhao", "Zhengyan Zhang", "Zhenda Xie", "Y. X. Wei", "Lean Wang", "Zhiping Xiao", "Yuqing Wang", "Chong Ruan", "Ming Zhang", "Wenfeng Liang", "Wangding Zeng"] year: 2025 venue: "arXiv:2502.11089" source: "https://arxiv.org/abs/2502.11089" doi: "10.48550/arxiv.2502.11089" arxiv: "2502.11089" licence: "arXiv — non-exclusive licence to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/yuan-2025-native-sparse-attention-hardware/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yuan-2025-native-sparse-attention-hardware/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/) / Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention # Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention **Authors:** Jingyang Yuan, [Huazuo Gao](https://md.research.elamur.ai/free/en/authors/huazuo-gao/), [Damai Dai](https://md.research.elamur.ai/free/en/authors/damai-dai/), Junyu Luo, [Liang Zhao](https://md.research.elamur.ai/free/en/authors/liang-zhao/), [Zhengyan Zhang](https://md.research.elamur.ai/free/en/authors/zhengyan-zhang/), [Zhenda Xie](https://md.research.elamur.ai/free/en/authors/zhenda-xie/), Y. X. Wei, [Lean Wang](https://md.research.elamur.ai/free/en/authors/lean-wang/), Zhiping Xiao, Yuqing Wang, [Chong Ruan](https://md.research.elamur.ai/free/en/authors/chong-ruan/) et al. (15 authors) · **Year:** 2025 · **Venue:** arXiv:2502.11089 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2502.11089) · [PDF](https://arxiv.org/pdf/2502.11089) · [DOI 10.48550/arxiv.2502.11089](https://doi.org/10.48550/arxiv.2502.11089) · [arXiv 2502.11089](https://arxiv.org/abs/2502.11089) · [OpenAlex](https://openalex.org/W4407690159) **Citations:** 4 citations (openalex-snapshot · 2026-10-14) · [131](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Standard softmax attention becomes the main cost in long-context language models. The authors estimate it takes 70–80% of decoding latency at 64k tokens. Earlier sparse attention methods often save compute on paper but not in wall-clock time. Many speed up only prefilling or only decoding, clash with GQA/MQA memory layouts, or cannot be trained end to end. The DeepSeek-AI and Peking University team proposes NSA (Natively trainable Sparse Attention). It groups keys and values into contiguous blocks, and each query attends through three gated branches: compressed block summaries for a global view, a learned top-n choice of important blocks for detail, and a sliding window for nearby context. Custom Triton kernels are built around Tensor Core use and balanced arithmetic intensity, so the same sparsity speeds up both training and inference. They pretrained a 27B-parameter transformer on 260B tokens. NSA matched or beat a Full Attention baseline on general, LongBench and chain-of-thought reasoning tests, with large speedups at 64k length. This shows sparsity can be built in from the start of pretraining instead of added afterwards. ## Key points - **Three-branch design:** coarse compressed tokens, selected fine-grained token blocks and sliding-window attention, combined through a gated output. Selection works on contiguous blocks, which keeps it compatible with FlashAttention-style computation and with GQA/MQA. - **Training setup:** a 27B-parameter backbone pretrained on 260B tokens, with kernels benchmarked on A100 GPUs against optimized Triton Full Attention. - **Speed:** on 64k-token sequences, NSA is about 11.6× faster for decoding, 9.0× for forward propagation and 6.0× for backward propagation. The gains grow with sequence length. - **Motivation from prior work:** keeping the top 20% of attention covers only about 70% of attention mass. This suggests that adding sparsity after training can damage structures such as retrieval heads. - **Ablations:** two alternative block-selection strategies gave worse training loss than NSA, even with a 1000-step Full Attention warm start. One used an auxiliary KL loss similar to SeerAttention, and the other a parameter-free heuristic similar to Quest. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [Computing & GPUs](https://md.research.elamur.ai/free/en/shelves/computing/) ## Cited by Cited by 131 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 130 not yet. - [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://md.research.elamur.ai/free/en/papers/deepseek-2025-ai-deepseek-v3-2-pushing/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{yuan2025native, title = {{Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention}}, author = {Yuan, Jingyang and Gao, Huazuo and Dai, Damai and Luo, Junyu and Zhao, Liang and Zhang, Zhengyan and Xie, Zhenda and Wei, Y. X. and Wang, Lean and Xiao, Zhiping and Wang, Yuqing and Ruan, Chong and Zhang, Ming and Liang, Wenfeng and Zeng, Wangding}, year = {2025}, doi = {10.48550/arxiv.2502.11089}, eprint = {2502.11089}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2502.11089} } ``` - [← previous MultiProSE: A Multi-label Arabic Dataset for Propaganda, Sentiment, and Emotion Detection](https://md.research.elamur.ai/free/en/papers/al-2025-henaki-multiprose-multi-label-arabic-dataset/) - [next → Natural Language Processing for Tigrinya: Current State and Future Directions](https://md.research.elamur.ai/free/en/papers/gaim-2025-natural-language-processing-tigrinya-current/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/) ## References 45 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Ainslie%2C+J%2E+Lee%2DThorp%2C+M%2E+de+Jong%2C+Y%2E+Zemlyanskiy%2C+F%2E+Lebr%C3%B3n%2C+and+S%2E+Sanghai%2E+Gqa%3A+Training+generalized+multi%2Dquery+transformer+models+from+multi%2Dhead+checkpoints%2E+arXiv+preprint+arXiv%3A2305%2E13245%2C+) [our card](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/) 2. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. [find](https://scholar.google.com/scholar?q=J%2E+Austin%2C+A%2E+Odena%2C+M%2E+Nye%2C+M%2E+Bosma%2C+H%2E+Michalewski%2C+D%2E+Dohan%2C+E%2E+Jiang%2C+C%2E+Cai%2C+M%2E+Terry%2C+Q%2E+Le%2C+et+al%2E+Program+synthesis+with+large+language+models%2E+arXiv+preprint+arXiv%3A2108%2E07732%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/) 3. Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023. [find](https://scholar.google.com/scholar?q=Y%2E+Bai%2C+X%2E+Lv%2C+J%2E+Zhang%2C+H%2E+Lyu%2C+J%2E+Tang%2C+Z%2E+Huang%2C+Z%2E+Du%2C+X%2E+Liu%2C+A%2E+Zeng%2C+L%2E+Hou%2C+et+al%2E+Longbench%3A+A+bilingual%2C+multitask+benchmark+for+long+context+understanding%2E+arXiv+preprint+arXiv%3A2308%2E14508%2C+) [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/) 4. I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. [find](https://scholar.google.com/scholar?q=I%2E+Beltagy%2C+M%2E+E%2E+Peters%2C+and+A%2E+Cohan%2E+Longformer%3A+The+long%2Ddocument+transformer%2E+arXiv+preprint+arXiv%3A2004%2E05150%2C+2020%2E) 5. G. Chen, H. Shi, J. Li, Y. Gao, X. Ren, Y. Chen, X. Jiang, Z. Li, W. Liu, and C. Huang. Sepllm: Accelerate large language models by compressing one segment into one separator. arXiv preprint arXiv:2412.12094, 2024a. — [arXiv:2412.12094](https://arxiv.org/abs/2412.12094) [find](https://scholar.google.com/scholar?q=G%2E+Chen%2C+H%2E+Shi%2C+J%2E+Li%2C+Y%2E+Gao%2C+X%2E+Ren%2C+Y%2E+Chen%2C+X%2E+Jiang%2C+Z%2E+Li%2C+W%2E+Liu%2C+and+C%2E+Huang%2E+Sepllm%3A+Accelerate+large+language+models+by+compressing+one+segment+into+one+separator%2E+arXiv+preprint+arXiv%3A241) 6. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. [find](https://scholar.google.com/scholar?q=M%2E+Chen%2C+J%2E+Tworek%2C+H%2E+Jun%2C+Q%2E+Yuan%2C+H%2E+P%2E+D%2E+O%2E+Pinto%2C+J%2E+Kaplan%2C+H%2E+Edwards%2C+Y%2E+Burda%2C+N%2E+Joseph%2C+G%2E+Brockman%2C+et+al%2E+Evaluating+large+language+models+trained+on+code%2E+arXiv+preprint+arXiv%3A2107%2E0337) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/) 7. Z. Chen, R. Sadhukhan, Z. Ye, Y. Zhou, J. Zhang, N. Nolte, Y. Tian, M. Douze, L. Bottou, Z. Jia, et al. Magicpig: Lsh sampling for efficient llm generation. arXiv preprint arXiv:2410.16179, 2024b. [find](https://scholar.google.com/scholar?q=Z%2E+Chen%2C+R%2E+Sadhukhan%2C+Z%2E+Ye%2C+Y%2E+Zhou%2C+J%2E+Zhang%2C+N%2E+Nolte%2C+Y%2E+Tian%2C+M%2E+Douze%2C+L%2E+Bottou%2C+Z%2E+Jia%2C+et+al%2E+Magicpig%3A+Lsh+sampling+for+efficient+llm+generation%2E+arXiv+preprint+arXiv%3A2410%2E16179%2C+2024b%2E) 8. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168, 2021. [↗](https://arxiv.org/abs/2110.14168) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/) 9. D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066, 2024. [find](https://scholar.google.com/scholar?q=D%2E+Dai%2C+C%2E+Deng%2C+C%2E+Zhao%2C+R%2E+Xu%2C+H%2E+Gao%2C+D%2E+Chen%2C+J%2E+Li%2C+W%2E+Zeng%2C+X%2E+Yu%2C+Y%2E+Wu%2C+et+al%2E+Deepseekmoe%3A+Towards+ultimate+expert+specialization+in+mixture%2Dof%2Dexperts+language+models%2E+arXiv+preprint+arXiv%3A2) 10. DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. 2024. URL https://arxiv.org/abs/2405.04434 . [↗](https://arxiv.org/abs/2405.04434) 11. DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948 . [↗](https://arxiv.org/abs/2501.12948) [our card](https://md.research.elamur.ai/free/en/papers/deepseek-2025-ai-deepseek-r1-incentivizing-reasoning-capability/) 12. A. Desai, S. Yang, A. Cuadron, A. Klimovic, M. Zaharia, J. E. Gonzalez, and I. Stoica. Hashattention: Semantic sparsity for faster inference. arXiv preprint arXiv:2412.14468, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Desai%2C+S%2E+Yang%2C+A%2E+Cuadron%2C+A%2E+Klimovic%2C+M%2E+Zaharia%2C+J%2E+E%2E+Gonzalez%2C+and+I%2E+Stoica%2E+Hashattention%3A+Semantic+sparsity+for+faster+inference%2E+arXiv+preprint+arXiv%3A2412%2E14468%2C+2024%2E) 13. D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00161, 2019. [find](https://scholar.google.com/scholar?q=D%2E+Dua%2C+Y%2E+Wang%2C+P%2E+Dasigi%2C+G%2E+Stanovsky%2C+S%2E+Singh%2C+and+M%2E+Gardner%2E+Drop%3A+A+reading+comprehension+benchmark+requiring+discrete+reasoning+over+paragraphs%2E+arXiv+preprint+arXiv%3A1903%2E00161%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/) 14. T. Fu, H. Huang, X. Ning, G. Zhang, B. Chen, T. Wu, H. Wang, Z. Huang, S. Li, S. Yan, et al. Moa: Mixture of sparse attention for automatic large language model compression. arXiv preprint arXiv:2406.14909, 2024a. [find](https://scholar.google.com/scholar?q=T%2E+Fu%2C+H%2E+Huang%2C+X%2E+Ning%2C+G%2E+Zhang%2C+B%2E+Chen%2C+T%2E+Wu%2C+H%2E+Wang%2C+Z%2E+Huang%2C+S%2E+Li%2C+S%2E+Yan%2C+et+al%2E+Moa%3A+Mixture+of+sparse+attention+for+automatic+large+language+model+compression%2E+arXiv+preprint+arXiv%3A2406%2E) 15. Y. Fu, Z. Cai, A. Asi, W. Xiong, Y. Dong, and W. Xiao. Not all heads matter: A head-level kv cache compression method with integrated retrieval and reasoning. arXiv preprint arXiv:2410.19258, 2024b. — [arXiv:2410.19258](https://arxiv.org/abs/2410.19258) [find](https://scholar.google.com/scholar?q=Y%2E+Fu%2C+Z%2E+Cai%2C+A%2E+Asi%2C+W%2E+Xiong%2C+Y%2E+Dong%2C+and+W%2E+Xiao%2E+Not+all+heads+matter%3A+A+head%2Dlevel+kv+cache+compression+method+with+integrated+retrieval+and+reasoning%2E+arXiv+preprint+arXiv%3A2410%2E19258%2C+2024b%2E+%E2%80%94) 16. Y. Gao, Z. Zeng, D. Du, S. Cao, H. K.-H. So, T. Cao, F. Yang, and M. Yang. Seerattention: Learning intrinsic sparse attention in your llms. arXiv preprint arXiv:2410.13276, 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Gao%2C+Z%2E+Zeng%2C+D%2E+Du%2C+S%2E+Cao%2C+H%2E+K%2E%2DH%2E+So%2C+T%2E+Cao%2C+F%2E+Yang%2C+and+M%2E+Yang%2E+Seerattention%3A+Learning+intrinsic+sparse+attention+in+your+llms%2E+arXiv+preprint+arXiv%3A2410%2E13276%2C+2024%2E) 17. S. Ge, Y. Zhang, L. Liu, M. Zhang, J. Han, and J. Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Ge%2C+Y%2E+Zhang%2C+L%2E+Liu%2C+M%2E+Zhang%2C+J%2E+Han%2C+and+J%2E+Gao%2E+Model+tells+you+what+to+discard%3A+Adaptive+kv+cache+compression+for+llms%2E+arXiv+preprint+arXiv%3A2310%2E01801%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/ge-2023-model-tells-discard-adaptive-kv/) 18. G. T. Google, P. Georgiev, V. I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. [find](https://scholar.google.com/scholar?q=G%2E+T%2E+Google%2C+P%2E+Georgiev%2C+V%2E+I%2E+Lei%2C+R%2E+Burnell%2C+L%2E+Bai%2C+A%2E+Gulati%2C+G%2E+Tanzer%2C+D%2E+Vincent%2C+Z%2E+Pan%2C+S%2E+Wang%2C+et+al%2E+Gemini+1%2E5%3A+Unlocking+multimodal+understanding+across+millions+of+tokens+of+context%2E) [our card](https://md.research.elamur.ai/free/en/papers/gemini-2024-team-gemini-1-5-unlocking-multimodal/) 19. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. [find](https://scholar.google.com/scholar?q=D%2E+Hendrycks%2C+C%2E+Burns%2C+S%2E+Basart%2C+A%2E+Zou%2C+M%2E+Mazeika%2C+D%2E+Song%2C+and+J%2E+Steinhardt%2E+Measuring+massive+multitask+language+understanding%2E+arXiv+preprint+arXiv%3A2009%2E03300%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/) 20. H. Jiang, Q. Wu, C.-Y. Lin, Y. Yang, and L. Qiu. Llmlingua: Compressing prompts for accelerated inference of large language models. arXiv preprint arXiv:2310.05736, 2023. [find](https://scholar.google.com/scholar?q=H%2E+Jiang%2C+Q%2E+Wu%2C+C%2E%2DY%2E+Lin%2C+Y%2E+Yang%2C+and+L%2E+Qiu%2E+Llmlingua%3A+Compressing+prompts+for+accelerated+inference+of+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E05736%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/jiang-2023-llmlingua-compressing-prompts-accelerated-inference/) 21. H. Jiang, Y. Li, C. Zhang, Q. Wu, X. Luo, S. Ahn, Z. Han, A. H. Abdi, D. Li, C.-Y. Lin, et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. arXiv preprint arXiv:2407.02490, 2024. [find](https://scholar.google.com/scholar?q=H%2E+Jiang%2C+Y%2E+Li%2C+C%2E+Zhang%2C+Q%2E+Wu%2C+X%2E+Luo%2C+S%2E+Ahn%2C+Z%2E+Han%2C+A%2E+H%2E+Abdi%2C+D%2E+Li%2C+C%2E%2DY%2E+Lin%2C+et+al%2E+Minference+1%2E0%3A+Accelerating+pre%2Dfilling+for+long%2Dcontext+llms+via+dynamic+sparse+attention%2E+arXiv+prepri) [our card](https://md.research.elamur.ai/free/en/papers/jiang-2024-minference-1-0-accelerating-pre/) 22. G. Kamradt. LLMTest NeedleInAHaystack. GitHub repository, 2023. URL [https://github.com/gkamradt/LLMTest_NeedleInAHaystack](https://github.com/gkamradt/LLMTest_NeedleInAHaystack) . Accessed: [Insert Access Date Here]. [↗](https://github.com/gkamradt/LLMTest_NeedleInAHaystack) 23. H. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212, 2023. [find](https://scholar.google.com/scholar?q=H%2E+Li%2C+Y%2E+Zhang%2C+F%2E+Koto%2C+Y%2E+Yang%2C+H%2E+Zhao%2C+Y%2E+Gong%2C+N%2E+Duan%2C+and+T%2E+Baldwin%2E+Cmmlu%3A+Measuring+massive+multitask+language+understanding+in+chinese%2E+arXiv+preprint+arXiv%3A2306%2E09212%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-cmmlu-measuring-massive-multitask-language/) 24. Y. Li, Y. Huang, B. Yang, B. Venkitesh, A. Locatelli, H. Ye, T. Cai, P. Lewis, and D. Chen. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469, 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Li%2C+Y%2E+Huang%2C+B%2E+Yang%2C+B%2E+Venkitesh%2C+A%2E+Locatelli%2C+H%2E+Ye%2C+T%2E+Cai%2C+P%2E+Lewis%2C+and+D%2E+Chen%2E+Snapkv%3A+Llm+knows+what+you+are+looking+for+before+generation%2E+arXiv+preprint+arXiv%3A2404%2E14469%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2024-snapkv-llm-knows-looking-before/) 25. G. Liu, C. Li, J. Zhao, C. Zhang, and M. Guo. Clusterkv: Manipulating llm kv cache in semantic space for recallable compression. arXiv preprint arXiv:2412.03213, 2024. — [arXiv:2412.03213](https://arxiv.org/abs/2412.03213) [find](https://scholar.google.com/scholar?q=G%2E+Liu%2C+C%2E+Li%2C+J%2E+Zhao%2C+C%2E+Zhang%2C+and+M%2E+Guo%2E+Clusterkv%3A+Manipulating+llm+kv+cache+in+semantic+space+for+recallable+compression%2E+arXiv+preprint+arXiv%3A2412%2E03213%2C+2024%2E+%E2%80%94+%5BarXiv%3A2412%2E03213%5D%28https%3A%2F%2Farx) 26. J. S. Park, J. C. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein. Generative agents: Interactive simulacra of human behavior. In S. Follmer, J. Han, J. Steimle, and N. H. Riche, editors, Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST 202… [find](https://scholar.google.com/scholar?q=J%2E+S%2E+Park%2C+J%2E+C%2E+O%E2%80%99Brien%2C+C%2E+J%2E+Cai%2C+M%2E+R%2E+Morris%2C+P%2E+Liang%2C+and+M%2E+S%2E+Bernstein%2E+Generative+agents%3A+Interactive+simulacra+of+human+behavior%2E+In+S%2E+Follmer%2C+J%2E+Han%2C+J%2E+Steimle%2C+and+N%2E+H%2E+Riche%2C+edito) 27. B. Peng, J. Quesnelle, H. Fan, and E. Shippole. Yarn: Efficient context window extension of large language models. In ICLR. OpenReview.net, 2024. [find](https://scholar.google.com/scholar?q=B%2E+Peng%2C+J%2E+Quesnelle%2C+H%2E+Fan%2C+and+E%2E+Shippole%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2E+In+ICLR%2E+OpenReview%2Enet%2C+2024%2E) 28. N. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. [find](https://scholar.google.com/scholar?q=N%2E+Shazeer%2E+Fast+transformer+decoding%3A+One+write%2Dhead+is+all+you+need%2E+CoRR%2C+abs%2F1911%2E02150%2C+2019%2E) 29. M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022. [find](https://scholar.google.com/scholar?q=M%2E+Suzgun%2C+N%2E+Scales%2C+N%2E+Sch%C3%A4rli%2C+S%2E+Gehrmann%2C+Y%2E+Tay%2C+H%2E+W%2E+Chung%2C+A%2E+Chowdhery%2C+Q%2E+V%2E+Le%2C+E%2E+H%2E+Chi%2C+D%2E+Zhou%2C+et+al%2E+Challenging+big%2Dbench+tasks+and+whether+chain%2Dof%2Dthought+can+solve+them%2E+arXiv+pr) [our card](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/) 30. J. Tang, Y. Zhao, K. Zhu, G. Xiao, B. Kasikci, and S. Han. Quest: Query-aware sparsity for efficient long-context llm inference. arXiv preprint arXiv:2406.10774, 2024. [find](https://scholar.google.com/scholar?q=J%2E+Tang%2C+Y%2E+Zhao%2C+K%2E+Zhu%2C+G%2E+Xiao%2C+B%2E+Kasikci%2C+and+S%2E+Han%2E+Quest%3A+Query%2Daware+sparsity+for+efficient+long%2Dcontext+llm+inference%2E+arXiv+preprint+arXiv%3A2406%2E10774%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/tang-2024-quest-query-aware-sparsity-efficient/) 31. P. Tillet, H.-T. Kung, and D. Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pages 10–19, 2019. [find](https://scholar.google.com/scholar?q=P%2E+Tillet%2C+H%2E%2DT%2E+Kung%2C+and+D%2E+Cox%2E+Triton%3A+an+intermediate+language+and+compiler+for+tiled+neural+network+computations%2E+In+Proceedings+of+the+3rd+ACM+SIGPLAN+International+Workshop+on+Machine+Learning) [our card](https://md.research.elamur.ai/free/en/papers/tillet-2019-triton-intermediate-language-compiler-tiled/) 32. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017. [find](https://scholar.google.com/scholar?q=A%2E+Vaswani%2C+N%2E+Shazeer%2C+N%2E+Parmar%2C+J%2E+Uszkoreit%2C+L%2E+Jones%2C+A%2E+N%2E+Gomez%2C+L%2E+u%2E+Kaiser%2C+and+I%2E+Polosukhin%2E+Attention+is+all+you+need%2E+Advances+in+Neural+Information+Processing+Systems%2C+2017%2E) 33. Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Wang%2C+X%2E+Ma%2C+G%2E+Zhang%2C+Y%2E+Ni%2C+A%2E+Chandra%2C+S%2E+Guo%2C+W%2E+Ren%2C+A%2E+Arulraj%2C+X%2E+He%2C+Z%2E+Jiang%2C+et+al%2E+Mmlu%2Dpro%3A+A+more+robust+and+challenging+multi%2Dtask+language+understanding+benchmark%2E+arXiv+preprint+arX) [our card](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/) 34. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. [find](https://scholar.google.com/scholar?q=J%2E+Wei%2C+X%2E+Wang%2C+D%2E+Schuurmans%2C+M%2E+Bosma%2C+F%2E+Xia%2C+E%2E+Chi%2C+Q%2E+V%2E+Le%2C+D%2E+Zhou%2C+et+al%2E+Chain%2Dof%2Dthought+prompting+elicits+reasoning+in+large+language+models%2E+Advances+in+neural+information+processing+sys) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/) 35. W. Wu, Z. Pan, C. Wang, L. Chen, Y. Bai, K. Fu, Z. Wang, and H. Xiong. Tokenselect: Efficient long-context inference and length extrapolation for llms via dynamic token-level kv cache selection. arXiv preprint arXiv:2411.02886, 2024. [find](https://scholar.google.com/scholar?q=W%2E+Wu%2C+Z%2E+Pan%2C+C%2E+Wang%2C+L%2E+Chen%2C+Y%2E+Bai%2C+K%2E+Fu%2C+Z%2E+Wang%2C+and+H%2E+Xiong%2E+Tokenselect%3A+Efficient+long%2Dcontext+inference+and+length+extrapolation+for+llms+via+dynamic+token%2Dlevel+kv+cache+selection%2E+arXiv) 36. C. Xiao, P. Zhang, X. Han, G. Xiao, Y. Lin, Z. Zhang, Z. Liu, and M. Sun. Infllm: Training-free long-context extrapolation for llms with an efficient context memory. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. [find](https://scholar.google.com/scholar?q=C%2E+Xiao%2C+P%2E+Zhang%2C+X%2E+Han%2C+G%2E+Xiao%2C+Y%2E+Lin%2C+Z%2E+Zhang%2C+Z%2E+Liu%2C+and+M%2E+Sun%2E+Infllm%3A+Training%2Dfree+long%2Dcontext+extrapolation+for+llms+with+an+efficient+context+memory%2E+In+The+Thirty%2Deighth+Annual+Confer) [our card](https://md.research.elamur.ai/free/en/papers/xiao-2024-infllm-training-free-long-context/) 37. G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023. [find](https://scholar.google.com/scholar?q=G%2E+Xiao%2C+Y%2E+Tian%2C+B%2E+Chen%2C+S%2E+Han%2C+and+M%2E+Lewis%2E+Efficient+streaming+language+models+with+attention+sinks%2E+arXiv+preprint+arXiv%3A2309%2E17453%2C+2023%2E) 38. G. Xiao, J. Tang, J. Zuo, J. Guo, S. Yang, H. Tang, Y. Fu, and S. Han. Duoattention: Efficient long-context llm inference with retrieval and streaming heads. arXiv preprint arXiv:2410.10819, 2024b. [find](https://scholar.google.com/scholar?q=G%2E+Xiao%2C+J%2E+Tang%2C+J%2E+Zuo%2C+J%2E+Guo%2C+S%2E+Yang%2C+H%2E+Tang%2C+Y%2E+Fu%2C+and+S%2E+Han%2E+Duoattention%3A+Efficient+long%2Dcontext+llm+inference+with+retrieval+and+streaming+heads%2E+arXiv+preprint+arXiv%3A2410%2E10819%2C+2024b%2E) [our card](https://md.research.elamur.ai/free/en/papers/xiao-2024-duoattention-efficient-long-context-llm/) 39. M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283–17297, 2020. [find](https://scholar.google.com/scholar?q=M%2E+Zaheer%2C+G%2E+Guruganesh%2C+K%2E+A%2E+Dubey%2C+J%2E+Ainslie%2C+C%2E+Alberti%2C+S%2E+Ontanon%2C+P%2E+Pham%2C+A%2E+Ravula%2C+Q%2E+Wang%2C+L%2E+Yang%2C+et+al%2E+Big+bird%3A+Transformers+for+longer+sequences%2E+Advances+in+neural+information+proc) [our card](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/) 40. E. Zelikman, Y. Wu, J. Mu, and N. D. Goodman. Star: Bootstrapping reasoning with reasoning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 20… [find](https://scholar.google.com/scholar?q=E%2E+Zelikman%2C+Y%2E+Wu%2C+J%2E+Mu%2C+and+N%2E+D%2E+Goodman%2E+Star%3A+Bootstrapping+reasoning+with+reasoning%2E+In+S%2E+Koyejo%2C+S%2E+Mohamed%2C+A%2E+Agarwal%2C+D%2E+Belgrave%2C+K%2E+Cho%2C+and+A%2E+Oh%2C+editors%2C+Advances+in+Neural+Informatio) [our card](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/) 41. F. Zhang, B. Chen, Y. Zhang, J. Keung, J. Liu, D. Zan, Y. Mao, J. Lou, and W. Chen. Repocoder: Repository-level code completion through iterative retrieval and generation. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proces… [find](https://scholar.google.com/scholar?q=F%2E+Zhang%2C+B%2E+Chen%2C+Y%2E+Zhang%2C+J%2E+Keung%2C+J%2E+Liu%2C+D%2E+Zan%2C+Y%2E+Mao%2C+J%2E+Lou%2C+and+W%2E+Chen%2E+Repocoder%3A+Repository%2Dlevel+code+completion+through+iterative+retrieval+and+generation%2E+In+H%2E+Bouamor%2C+J%2E+Pino%2C+and+) 42. K. Zhang, J. Li, G. Li, X. Shi, and Z. Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In L. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vo… [find](https://scholar.google.com/scholar?q=K%2E+Zhang%2C+J%2E+Li%2C+G%2E+Li%2C+X%2E+Shi%2C+and+Z%2E+Jin%2E+Codeagent%3A+Enhancing+code+generation+with+tool%2Dintegrated+agent+systems+for+real%2Dworld+repo%2Dlevel+coding+challenges%2E+In+L%2E+Ku%2C+A%2E+Martins%2C+and+V%2E+Srikumar%2C+) 43. Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661–34710, 2023b. [find](https://scholar.google.com/scholar?q=Z%2E+Zhang%2C+Y%2E+Sheng%2C+T%2E+Zhou%2C+T%2E+Chen%2C+L%2E+Zheng%2C+R%2E+Cai%2C+Z%2E+Song%2C+Y%2E+Tian%2C+C%2E+R%C3%A9%2C+C%2E+Barrett%2C+et+al%2E+H2o%3A+Heavy%2Dhitter+oracle+for+efficient+generative+inference+of+large+language+models%2E+Advances+in+Ne) 44. J. Zhao, Z. Fang, S. Li, S. Yang, and S. He. Buzz: Beehive-structured sparse kv cache with segmented heavy hitters for efficient llm inference. arXiv preprint arXiv:2410.23079, 2024. — [arXiv:2410.23079](https://arxiv.org/abs/2410.23079) [find](https://scholar.google.com/scholar?q=J%2E+Zhao%2C+Z%2E+Fang%2C+S%2E+Li%2C+S%2E+Yang%2C+and+S%2E+He%2E+Buzz%3A+Beehive%2Dstructured+sparse+kv+cache+with+segmented+heavy+hitters+for+efficient+llm+inference%2E+arXiv+preprint+arXiv%3A2410%2E23079%2C+2024%2E+%E2%80%94+%5BarXiv%3A2410%2E230) 45. Z. Zhou, C. Li, X. Chen, S. Wang, Y. Chao, Z. Li, H. Wang, R. An, Q. Shi, Z. Tan, et al. Llm × mapreduce: Simplified long-sequence processing using large language models. arXiv preprint arXiv:2410.09342, 2024. — [arXiv:2410.09342](https://arxiv.org/abs/2410.09342) [find](https://scholar.google.com/scholar?q=Z%2E+Zhou%2C+C%2E+Li%2C+X%2E+Chen%2C+S%2E+Wang%2C+Y%2E+Chao%2C+Z%2E+Li%2C+H%2E+Wang%2C+R%2E+An%2C+Q%2E+Shi%2C+Z%2E+Tan%2C+et+al%2E+Llm+%C3%97+mapreduce%3A+Simplified+long%2Dsequence+processing+using+large+language+models%2E+arXiv+preprint+arXiv%3A2410%2E093)