---
title: "MiniMax Sparse Attention — paper by Lai et al. (2026)"
description: "The authors propose MiniMax Sparse Attention (MSA), a deliberately minimal block-sparse mechanism… 71 references, cited by 2 papers here."
authors: ["Xunhao Lai", "Weiqi Xu", "Yufeng Yang", "Qiaorui Chen", "Yang Xu", "Lunbin Zeng", "Xiaolong Li", "Haohai Sun", "Haichao Zhu", "Vito Zhang", "Jinkai Hu", "Jiayao Li", "Rui Gao", "Zekun Li", "Songquan Zhu", "Jingkai Zhou", "Pengyu Zhao"]
year: 2026
venue: "arXiv:2606.13392"
source: "https://arxiv.org/abs/2606.13392"
doi: "10.48550/arxiv.2606.13392"
arxiv: "2606.13392"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lai-2026-minimax-sparse-attention/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lai-2026-minimax-sparse-attention/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/8/) / MiniMax Sparse Attention

# MiniMax Sparse Attention

**Authors:** Xunhao Lai, Weiqi Xu, Yufeng Yang, Qiaorui Chen, [Yang Xu](https://md.research.elamur.ai/free/en/authors/yang-xu/), Lunbin Zeng, [Xiaolong Li](https://md.research.elamur.ai/free/en/authors/xiaolong-li/), Haohai Sun, Haichao Zhu, Vito Zhang, Jinkai Hu, Jiayao Li, Rui Gao, [Zekun Li](https://md.research.elamur.ai/free/en/authors/zekun-li/), Songquan Zhu, Jingkai Zhou and Pengyu Zhao · **Year:** 2026 · **Venue:** arXiv:2606.13392 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2606.13392) · [PDF](https://arxiv.org/pdf/2606.13392) · [DOI 10.48550/arxiv.2606.13392](https://doi.org/10.48550/arxiv.2606.13392) · [arXiv 2606.13392](https://arxiv.org/abs/2606.13392) · [OpenAlex](https://openalex.org/W7164548514)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 2 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Quadratic softmax attention makes contexts of hundreds of thousands to millions of tokens too costly in production. This matters for agents, repository-scale coding and long-term memory. The authors propose MiniMax Sparse Attention (MSA), a deliberately minimal block-sparse mechanism layered on Grouped Query Attention (GQA).

A small Index Branch adds only two projection matrices. It scores key–value blocks by max-pooling token-level dot products, and each GQA group picks its own Top-k blocks. The block containing the current position is always kept. The Main Branch then runs exact softmax attention restricted to those blocks. Because Top-k selection cannot be differentiated, the indexer learns through a KL objective that matches the group-averaged Main Branch distribution, with gradient detachment and a warmup phase for stability.

Custom GPU kernels turn the theoretical sparsity into real speed. They use an exponent-free Top-k and a KV-outer loop ordering that keeps tensor cores busy.

In a 109B-parameter natively multimodal MoE model, MSA performs about as well as dense GQA at a fraction of the attention cost. The authors note a remaining gap in long-context retrieval.

## Key points
- **Evaluation setup:** a 109B-parameter MoE model trained from scratch on a 3T-token budget with native multimodal data. MSA matches GQA on most pretraining and agentic benchmarks.
- **Efficiency at 1M context:** per-token attention compute drops 28.4×. Wall-clock speedups on H800 are 14.2× for prefill and 7.6× for decoding.
- **Benchmark configuration:** block size B_k = 128 and k = 16, giving 2,048 attended tokens per query. The model uses 64 query heads, 4 KV heads and head dimension 128.
- **Kernel design:** a pre-scheduled chunking scheme with a two-phase combine handles skewed block popularity without atomic operations. LSE computation for the KL loss is fused into the forward pass.
- **Releases:** the inference kernel is at github.com/MiniMax-AI/MSA, and the production model MiniMax-M3 is on Hugging Face. MSA also supports near-lossless conversion of existing GQA checkpoints.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,267 citations
- [Zhejiang University](https://md.research.elamur.ai/free/en/universities/zhejiang-university/) — University, Hangzhou, China · 224 works · 13,587 citations
- [Nvidia](https://md.research.elamur.ai/free/en/companies/nvidia/) — Company, Santa Clara, United States · 37 works · 12,656 citations
- [Huazhong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/huazhong-university-of-science-and-technology/) — University, Wuhan, China · 59 works · 8,310 citations
- [Nvidia (United Kingdom)](https://md.research.elamur.ai/free/en/companies/nvidia-united-kingdom/) — Company, Reading, United Kingdom · 27 works · 11,141 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 297 works

## Cite

Cite as: Xunhao Lai, Weiqi Xu, Yufeng Yang, Qiaorui Chen, Yang Xu, Lunbin Zeng, Xiaolong Li, Haohai Sun, Haichao Zhu, Vito Zhang, Jinkai Hu, Jiayao Li, Rui Gao, Zekun Li, Songquan Zhu, Jingkai Zhou and Pengyu Zhao. 2026. MiniMax Sparse Attention. arXiv:2606.13392.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{lai2026minimax,
  title = {{MiniMax Sparse Attention}},
  author = {Xunhao Lai, Weiqi Xu, Yufeng Yang, Qiaorui Chen, Yang Xu, Lunbin Zeng, Xiaolong Li, Haohai Sun, Haichao Zhu, Vito Zhang, Jinkai Hu, Jiayao Li, Rui Gao, Zekun Li, Songquan Zhu, Jingkai Zhou and Pengyu Zhao},
  year = {2026},
  doi = {10.48550/arxiv.2606.13392},
  eprint = {2606.13392},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2606.13392}
}
```

- [← previous MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling](https://md.research.elamur.ai/free/en/papers/minicpm-team-2026-minicpm-sala-hybridizing-sparse/)
- [next → Mistake Notebook Learning: Batch-Clustered Failures for Training-Free Agent Adaptation](https://md.research.elamur.ai/free/en/papers/su-2026-mistake-notebook-learning-batch-clustered/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/8/)

## References

71 references, 45 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. [GQA: Training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
2. Anthropic. Claude Opus 4.6 and Sonnet 4.6 model card. [https://www.anthropic.com/news/claude-4-6](https://www.anthropic.com/news/claude-4-6) , 2025. [↗](https://www.anthropic.com/news/claude-4-6)
3. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. 𝜏 2 -bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.[07982, 2025](https://md.research.elamur.ai/free/en/papers/barres-2025-tau-2-bench-evaluating-conversational/).
4. Iz Beltagy, Matthew E. Peters, and Arman Cohan. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). arXiv preprint arXiv:2004.05150, 2020.
5. Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, Yao Dou, Jaden Park, Jianfeng Gao, Yong Jae Lee, and Jianwei Yang. Temporalbench: Benchmarking fine-grained temporal understanding for multimodal video models, 2024. URL https://arxi… — [arXiv:2410.10818](https://arxiv.org/abs/2410.10818) [find](https://scholar.google.com/scholar?q=Mu+Cai%2C+Reuben+Tan%2C+Jianrui+Zhang%2C+Bocheng+Zou%2C+Kai+Zhang%2C+Feng+Yao%2C+Fangrui+Zhu%2C+Jing+Gu%2C+Yiwu+Zhong%2C+Yuzhang+Shang%2C+Yao+Dou%2C+Jaden+Park%2C+Jianfeng+Gao%2C+Yong+Jae+Lee%2C+and+Jianwei+Yang%2E+Temporalbench%3A+)
6. Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. MultiPL-E: A scalable and polyglot approach to benchmarking neural code generation.… [find](https://scholar.google.com/scholar?q=Federico+Cassano%2C+John+Gouwar%2C+Daniel+Nguyen%2C+Sydney+Nguyen%2C+Luna+Phipps%2DCostin%2C+Donald+Pinckney%2C+Ming%2DHo+Yee%2C+Yangtian+Zi%2C+Carolyn+Jane+Anderson%2C+Molly+Q%2E+Feldman%2C+Arjun+Guha%2C+Michael+Greenberg%2C+and+)
7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
8. Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. [Rethinking attention with performers](https://md.research.elamur.ai/free/en/papers/choromanski-2020-rethinking-attention-performers/). In International Conference on Learning Representati…
9. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/), 2018. URL https://arxiv.org/abs/1803.05457 . [↗](https://arxiv.org/abs/1803.05457)
10. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/), 2021. URL https://arxiv.org/abs/2110.14168 . [↗](https://arxiv.org/abs/2110.14168)
11. Tri Dao. [FlashAttention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). In International Conference on Learning Representations (ICLR), 2024.
12. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Daniel+Y%2E+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+FlashAttention%3A+Fast+and+memory%2Defficient+exact+attention+with+IO%2Dawareness%2E+In+Advances+in+Neural+Information+Processing+Systems%2C)
13. Tri Dao, Daniel Haziza, Francisco Massa, and Grigory Sizov. Flash-decoding for long-context inference, 2023. [https://crfm.stanford.edu/2023/10/12/flashdecoding.html](https://crfm.stanford.edu/2023/10/12/flashdecoding.html) . [↗](https://crfm.stanford.edu/2023/10/12/flashdecoding.html)
14. DeepSeek-AI. [DeepSeek-V4: Towards highly efficient million-token context intelligence](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2026-deepseek-v4-highly-efficient/), 2026. Technical report (preview).
15. DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenhao Xu, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Erhang Li, Fangqi Zhou, Fangyun Lin, Fucong Dai, Guang… — [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-v3-2-pushing/)
16. Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. [Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis](https://md.research.elamur.ai/free/en/papers/fu-2024-video-mme-first-ever-comprehensive/). arXiv preprint arXiv:2405.21075, 2024.
17. Ling Fu, Zhebin Kuang, Jiajun Song, Mingxin Huang, Biao Yang, Yuzhe Li, Linghao Zhu, Qidi Luo, Xinyu Wang, Hao Lu, Zhang Li, Guozhi Tang, Bin Shan, Chunhui Lin, Qi Liu, Binghong Wu, Hao Feng, Hao Liu, Can Huang, Jingqun Tang, Wei Chen, Lianwen Jin, Yuliang Liu, and Xiang Bai. Ocrbench v2: An improve… — [arXiv:2501.00321](https://arxiv.org/abs/2501.00321) [find](https://scholar.google.com/scholar?q=Ling+Fu%2C+Zhebin+Kuang%2C+Jiajun+Song%2C+Mingxin+Huang%2C+Biao+Yang%2C+Yuzhe+Li%2C+Linghao+Zhu%2C+Qidi+Luo%2C+Xinyu+Wang%2C+Hao+Lu%2C+Zhang+Li%2C+Guozhi+Tang%2C+Bin+Shan%2C+Chunhui+Lin%2C+Qi+Liu%2C+Binghong+Wu%2C+Hao+Feng%2C+Hao+Liu%2C)
18. Google DeepMind. Gemini 3.1 pro. [https://deepmind.google/technologies/gemini/](https://deepmind.google/technologies/gemini/) , 2025. [↗](https://deepmind.google/technologies/gemini/)
19. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. [find](https://scholar.google.com/scholar?q=Albert+Gu+and+Tri+Dao%2E+Mamba%3A+Linear%2Dtime+sequence+modeling+with+selective+state+spaces%2E+arXiv+preprint+arXiv%3A2312%2E00752%2C+2023%2E)
20. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations, 2021. URL [https://openreview.net/forum?id=d7KBjmI3GmQ](https://openreview.net/forum?id=d7KBjmI3GmQ) . [↗](https://openreview.net/forum?id=d7KBjmI3GmQ)
21. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? In First Conference on Language Modeling, 2024. URL — [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/) [https://openreview.net/forum?id=kIoBbc76Sy](https://openreview.net/forum?id=kIoBbc76Sy) . [↗](https://openreview.net/forum?id=kIoBbc76Sy)
22. Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. [MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention](https://md.research.elamur.ai/free/en/papers/jiang-2024-minference-1-0-accelerating-pre/). In Advances in Neural Information Processi…
23. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. [SWE-bench: Can language models resolve real-world GitHub issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) In International Conference on Learning Representations (ICLR), 2024.
24. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. [TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long…
25. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning (ICML), 2020. [find](https://scholar.google.com/scholar?q=Angelos+Katharopoulos%2C+Apoorv+Vyas%2C+Nikolaos+Pappas%2C+and+Fran%C3%A7ois+Fleuret%2E+Transformers+are+RNNs%3A+Fast+autoregressive+transformers+with+linear+attention%2E+In+International+Conference+on+Machine+Learnin)
26. Aniruddha Kembhavi, Michael Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. [A diagram is worth a dozen images](https://md.research.elamur.ai/free/en/papers/kembhavi-2016-diagram-worth-dozen-images/). In European Conference on Computer Vision (ECCV), 2016.
27. Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. [find](https://scholar.google.com/scholar?q=Xunhao+Lai%2C+Jianqiao+Lu%2C+Yao+Luo%2C+Yiyuan+Ma%2C+and+Xun+Zhou%2E+Flexprefill%3A+A+context%2Daware+sparse+attention+mechanism+for+efficient+long%2Dsequence+inference%2E+In+The+Thirteenth+International+Conference+on+)
28. OpenReview.net, 2025. URL [https://openreview.net/forum?id=OfjIlbelrT](https://openreview.net/forum?id=OfjIlbelrT) . [↗](https://openreview.net/forum?id=OfjIlbelrT)
29. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. [SnapKV: LLM knows what you are looking for before generation](https://md.research.elamur.ai/free/en/papers/li-2024-snapkv-llm-knows-looking-before/). In Advances in Neural Information Processing Systems, 2024.
30. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and LINGMING ZHANG. [Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/). In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processi…
31. Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visualwebbench: How far have multimodal LLMs evolved in web page understanding and grounding? In First Conference on Language Modeling, 2024. URL [https://openreview.net/forum?id=egVSgtJJAx](https://openreview.net/forum?id=egVSgtJJAx) . [↗](https://openreview.net/forum?id=egVSgtJJAx)
32. Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Neo Y. Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang, and Jiez… — [MoBA: Mixture of Block Attention for Long-Context LLMs](https://md.research.elamur.ai/free/en/papers/lu-2025-moba-mixture-block-attention/)
33. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. [Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). In The Twelfth International Conference on Learning Representations, 2024.…
34. Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. [EgoSchema: A diagnostic benchmark for very long-form video language understanding](https://md.research.elamur.ai/free/en/papers/mangalam-2023-egoschema-diagnostic-benchmark-very-long/). In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2023.
35. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. [ChartQA: A benchmark for question answering about charts with visual and logical reasoning](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/). In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 202…
36. MiMo, Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, Gang Xie, Hailin Zhang, Hanglong Lv, Hanyu Li, Heyu Chen, Hongshen Xu, Houbin Zhang, Huaqiu Liu, Jiangshan Duo, Jianyu Wei, Jiebao Xiao, Jinhao Dong, Jun Shi, Junhao Hu, Kaina… — [MiMo-V2-Flash Technical Report](https://md.research.elamur.ai/free/en/papers/team-2026-llm-mimo-v2-flash-technical/)
37. MiniMax. [MiniMax-01: Scaling foundation models with lightning attention](https://md.research.elamur.ai/free/en/papers/minimax-2025-minimax-01-scaling-foundation/). arXiv:2501.08313, 2025a.
38. MiniMax. [MiniMax-M1: Scaling test-time compute efficiently with lightning attention](https://md.research.elamur.ai/free/en/papers/minimax-2025-minimax-m1-scaling-test/). arXiv preprint arXiv:2506.13585, 2025b.
39. Moonshot AI. Kimi K2.6: Open agentic foundation model. [https://moonshotai.github.io/Kimi-K2/](https://moonshotai.github.io/Kimi-K2/) , 2026. [↗](https://moonshotai.github.io/Kimi-K2/)
40. OpenAI. Introducing GPT-5. [https://openai.com/gpt-5/](https://openai.com/gpt-5/) , 2025. [↗](https://openai.com/gpt-5/)
41. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Cl… — [gpt-oss-120b & gpt-oss-20b Model Card](https://md.research.elamur.ai/free/en/papers/agarwal-2025-gpt-oss-120b-gpt-oss/)
42. Qwen. Qwen3.5: Accelerating productivity with native multimodal agents, February 2026. URL [https://qwen.ai/blog?id=qwen3.5](https://qwen.ai/blog?id=qwen3.5) . [↗](https://qwen.ai/blog?id=qwen3.5)
43. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. [GPQA: A graduate-level google-proof Q&A benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). arXiv preprint arXiv:2311.12022, 2023.
44. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8732–8740, April 2020. ISSN 2159-5399. doi: 10.1609/aaai.v34i05.6399. URL http://dx.doi.org/10.… [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+34%28)
45. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. [Language models are multilingual chain-of-thought reasoners](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/), 2022. URL https://arxiv.org/abs/2210.03057 . [↗](https://arxiv.org/abs/2210.03057)
46. Haoxiang Sun, Yingqian Chen, Xueyang Wen, Bingchen Hu, Tianyi Shi, Tianyi Wang, Junyi Wu, Wayne Xin Zhou, and Ji-Rong Wen. [Challenging the boundaries of reasoning: An olympiad-level math benchmark for large language models](https://md.research.elamur.ai/free/en/papers/sun-2026-challenging-boundaries-reasoning-olympiad-level/). arXiv preprint arXiv:2503.21380, 2025.
47. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. [Challenging big-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/), 2022. URL https://arxiv.org/abs/2210.09261 . [↗](https://arxiv.org/abs/2210.09261)
48. Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. [Quest: Query-aware sparsity for efficient long-context LLM inference](https://md.research.elamur.ai/free/en/papers/tang-2024-quest-query-aware-sparsity-efficient/). In International Conference on Machine Learning (ICML), 2024.
49. Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, Wentao Li, Enzhe Lu, Weizhou Liu, Yanru Chen, Weixin Xu, Longhui Yu, Yejie Wang, Yu Fan, Longguang Zhong, Enming Yuan, Dehao Zhang, Yizhi Zhang, T. Y. Liu, Haiming Wang, Shengjun… — [Kimi Linear: An Expressive, Efficient Attention Architecture](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-linear-expressive-efficient/)
50. MiniCPM Team, Chaojun Xiao, Yuxuan Li, Xu Han, Yuzhuo Bai, Jie Cai, Haotian Chen, Wentong Chen, Xin Cong, Ganqu Cui, Ning Ding, Shengda Fan, Yewei Fang, Zixuan Fu, Wenyu Guan, Yitong Guan, Junshao Guo, Yufeng Han, Bingxiang He, Yuxiang Huang, Baoxi Ji, Cunliang Kong, Qiuzuo Li, Siyuan Li, Wenhao Li,… — [MiniCPM4: Ultra-Efficient LLMs on End Devices](https://md.research.elamur.ai/free/en/papers/minicpm-team-2025-minicpm4-ultra-efficient-llms/)
51. Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. [Cambrian-1: A fully open, vision-centric exploration of multimodal llms](https://md.research.elamur.ai/free/en/papers/tong-2024-cambrian-1-fully-open-vision/). In A. Globerson, L. Mackey,…
52. Lei Wang, Yu Cheng, Yining Shi, Zhengju Tang, Zhiwen Mo, Wenhao Xie, Lingxiao Ma, Yuqing Xia, Jilong Xue, Fan Yang, and Zhi Yang. Tilelang: A composable tiled programming model for ai systems, 2025. URL https://arxiv.org/abs/2504.17577 . [↗](https://arxiv.org/abs/2504.17577)
53. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. [Mmlu-pro: A more robust and challenging multi-task language understanding benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/). In A…
54. Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, and Danqi Chen. [CharXiv: Charting gaps in realistic chart understanding in multimodal LLMs](https://md.research.elamur.ai/free/en/papers/wang-2024-charxiv-charting-gaps-realistic-chart/). In Advances in Neural Information Processing Syste…
55. Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. [Longvideobench: A benchmark for long-context interleaved video-language understanding](https://md.research.elamur.ai/free/en/papers/wu-2024-longvideobench-benchmark-long-context-interleaved/). In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 28…
56. Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, Song Han, and Maosong Sun. Inf LLM: Training-free long-context extrapolation for LLMs with an efficient context memory. arXiv preprint arXiv:2402.[04617, 2024a](https://md.research.elamur.ai/free/en/papers/xiao-2024-infllm-training-free-long-context/).
57. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations (ICLR), 2024b. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+Efficient+streaming+language+models+with+attention+sinks%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2C+2024b%2E)
58. Guangxuan Xiao, Junxian Guo, Kasra Mazaheri, and Song Han. Optimizing mixture of block attention. arXiv preprint arXiv:2511.11571, 2025. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Junxian+Guo%2C+Kasra+Mazaheri%2C+and+Song+Han%2E+Optimizing+mixture+of+block+attention%2E+arXiv+preprint+arXiv%3A2511%2E11571%2C+2025%2E)
59. Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. [TheAgentCompany: Benchmarking LLM agents on consequential real world tasks](https://md.research.elamur.ai/free/en/papers/xu-2024-theagentcompany-benchmarking-llm-agents/). arXiv preprint arXiv:2412.14161, 2024.
60. Ran Yan, Youhe Jiang, and Binhang Yuan. Flash sparse attention: More efficient natively trainable sparse attention. arXiv preprint arXiv:2508.18224, 2025. [find](https://scholar.google.com/scholar?q=Ran+Yan%2C+Youhe+Jiang%2C+and+Binhang+Yuan%2E+Flash+sparse+attention%3A+More+efficient+natively+trainable+sparse+attention%2E+arXiv+preprint+arXiv%3A2508%2E18224%2C+2025%2E)
61. Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule, 2025. URL https://arxiv.org/abs/2412.06464 . [↗](https://arxiv.org/abs/2412.06464)
62. Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. HELMET: How to evaluate long-context models effectively and thoroughly. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview. net/forum?id=293… [find](https://scholar.google.com/scholar?q=Howard+Yen%2C+Tianyu+Gao%2C+Minmin+Hou%2C+Ke+Ding%2C+Daniel+Fleischer%2C+Peter+Izsak%2C+Moshe+Wasserblat%2C+and+Danqi+Chen%2E+HELMET%3A+How+to+evaluate+long%2Dcontext+models+effectively+and+thoroughly%2E+In+The+Thirteenth+)
63. Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. [Native sparse attention: Hardware-aligned and natively trainable sparse attention](https://md.research.elamur.ai/free/en/papers/yuan-2025-native-sparse-attention-hardware/). In Proceedings of the…
64. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: A massive multi-discipline… — [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/)
65. Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. [Big bird: Transformers for longer sequences](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/). In Advances in Neural Information Processing Systems, 2020.
66. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2O: Heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing… [find](https://scholar.google.com/scholar?q=Zhenyu+Zhang%2C+Ying+Sheng%2C+Tianyi+Zhou%2C+Tianlong+Chen%2C+Lianmin+Zheng%2C+Ruisi+Cai%2C+Zhao+Song%2C+Yuandong+Tian%2C+Christopher+R%C3%A9%2C+Clark+Barrett%2C+Zhangyang+Wang%2C+and+Beidi+Chen%2E+H2O%3A+Heavy%2Dhitter+oracle+for+ef)
67. Weilin Zhao, Zihan Zhou, Zhou Su, Chaojun Xiao, Yuxuan Li, Yanghao Li, Yudi Zhang, Weilun Zhao, Zhen Li, Yuxiang Huang, Ao Sun, Xu Han, and Zhiyuan Liu. Infllm-v2: Dense-sparse switchable attention for seamless short-to-long adaptation. CoRR, abs/2509.24663, 2025a. doi: 10.48550/ARXIV.2509.24663. UR… — [doi:10.48550/arxiv.2509.24663](https://doi.org/10.48550/arxiv.2509.24663) [find](https://scholar.google.com/scholar?q=Weilin+Zhao%2C+Zihan+Zhou%2C+Zhou+Su%2C+Chaojun+Xiao%2C+Yuxuan+Li%2C+Yanghao+Li%2C+Yudi+Zhang%2C+Weilun+Zhao%2C+Zhen+Li%2C+Yuxiang+Huang%2C+Ao+Sun%2C+Xu+Han%2C+and+Zhiyuan+Liu%2E+Infllm%2Dv2%3A+Dense%2Dsparse+switchable+attention+fo)
68. Yilun Zhao, Lujing Xie, Haowei Zhang, Guo Gan, Yitao Long, Zhiyuan Hu, Tongyan Hu, Weiyuan Chen, Chuhan Li, Junyang Song, et al. MMVU: Measuring expert-level multi-discipline video understanding. arXiv preprint arXiv:2501.12380, 2025b. — [arXiv:2501.12380](https://arxiv.org/abs/2501.12380) [find](https://scholar.google.com/scholar?q=Yilun+Zhao%2C+Lujing+Xie%2C+Haowei+Zhang%2C+Guo+Gan%2C+Yitao+Long%2C+Zhiyuan+Hu%2C+Tongyan+Hu%2C+Weiyuan+Chen%2C+Chuhan+Li%2C+Junyang+Song%2C+et+al%2E+MMVU%3A+Measuring+expert%2Dlevel+multi%2Ddiscipline+video+understanding%2E+arXi)
69. Zhipu AI. GLM-5.1: Open foundation models from Zhipu AI. [https://github.com/THUDM/GLM-5](https://github.com/THUDM/GLM-5) , 2026. [↗](https://github.com/THUDM/GLM-5)
70. Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. [Mlvu: Benchmarking multi-task long video understanding](https://md.research.elamur.ai/free/en/papers/zhou-2024-mlvu-benchmarking-multi-task-long/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), page…
71. Terry Yue Zhuo, Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen GONG, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex G… — [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://md.research.elamur.ai/free/en/papers/zhuo-2024-bigcodebench-benchmarking-code-generation-diverse/)

*Updated 2 November 2026*