---
title: "MiniCPM-SALA — paper by Team et al. (2026)"
description: "Full attention in Transformers is costly for very long inputs. 61 references, cited by 1 paper here."
authors: ["MiniCPM Team", "Wenhao An", "Yingfa Chen", "Yewei Fang", "Jiayi Li", "Xin Li", "Yaohui Li", "Yishan Li", "Yuxuan Li", "Biyuan Lin", "Chuan Liu", "Hezi Liu", "Siyuan Liu", "Hongya Lyu", "Yinxu Pan", "Shixin Ren", "Xingyu Shen", "Zhou Su", "Haojun Sun", "Yangang Sun", "Zhen Leng Thai", "Xin Tian", "Rui Wang", "Xiaorong Wang", "Yudong Wang", "Bo Wu", "Xiaoyue Xu", "Dong Xu", "Shuaikang Xue", "Jiawei Yang", "Bowen Zhang", "Jinqian Zhang", "Letian Zhang", "Shengnan Zhang", "Xinyu Zhang", "Xinyuan Zhang", "Zhu Zhang", "Hengyu Zhao", "Jiacheng Zhao", "Zhi Zheng", "Jie Zhou", "Zihan Zhou", "Shuo Wang", "Chaojun Xiao", "Xu Han", "Zhiyuan Liu", "Maosong Sun"]
year: 2026
venue: "arXiv:2602.11761"
source: "https://arxiv.org/abs/2602.11761"
doi: "10.48550/arxiv.2602.11761"
arxiv: "2602.11761"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/minicpm-team-2026-minicpm-sala-hybridizing-sparse/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/minicpm-team-2026-minicpm-sala-hybridizing-sparse/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/8/) / MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling

# MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling

**Authors:** [MiniCPM Team](https://md.research.elamur.ai/free/en/authors/minicpm-team/), Wenhao An, [Yingfa Chen](https://md.research.elamur.ai/free/en/authors/yingfa-chen/), [Yewei Fang](https://md.research.elamur.ai/free/en/authors/yewei-fang/), Jiayi Li, [Xin Li](https://md.research.elamur.ai/free/en/authors/xin-li/), Yaohui Li, [Yishan Li](https://md.research.elamur.ai/free/en/authors/yishan-li/), [Yuxuan Li](https://md.research.elamur.ai/free/en/authors/yuxuan-li/), [Biyuan Lin](https://md.research.elamur.ai/free/en/authors/biyuan-lin/), Chuan Liu, Hezi Liu, Siyuan Liu, Hongya Lyu, [Yinxu Pan](https://md.research.elamur.ai/free/en/authors/yinxu-pan/), Shixin Ren, Xingyu Shen, Zhou Su, Haojun Sun, Yangang Sun, [Zhen Leng Thai](https://md.research.elamur.ai/free/en/authors/zhen-leng-thai/), [Xin Tian](https://md.research.elamur.ai/free/en/authors/xin-tian/), [Rui Wang](https://md.research.elamur.ai/free/en/authors/rui-wang/), Xiaorong Wang, [Yudong Wang](https://md.research.elamur.ai/free/en/authors/yudong-wang/), Bo Wu, Xiaoyue Xu, Dong Xu, Shuaikang Xue, Jiawei Yang, [Bowen Zhang](https://md.research.elamur.ai/free/en/authors/bowen-zhang/), Jinqian Zhang, Letian Zhang, Shengnan Zhang, [Xinyu Zhang](https://md.research.elamur.ai/free/en/authors/xinyu-zhang/), Xinyuan Zhang, Zhu Zhang, Hengyu Zhao, Jiacheng Zhao, [Zhi Zheng](https://md.research.elamur.ai/free/en/authors/zhi-zheng/), [Jie Zhou](https://md.research.elamur.ai/free/en/authors/jie-zhou/), Zihan Zhou, [Shuo Wang](https://md.research.elamur.ai/free/en/authors/shuo-wang/), [Chaojun Xiao](https://md.research.elamur.ai/free/en/authors/chaojun-xiao/), [Xu Han](https://md.research.elamur.ai/free/en/authors/xu-han/), [Zhiyuan Liu](https://md.research.elamur.ai/free/en/authors/zhiyuan-liu/) and [Maosong Sun](https://md.research.elamur.ai/free/en/authors/maosong-sun/) · **Year:** 2026 · **Venue:** arXiv:2602.11761 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2602.11761) · [PDF](https://arxiv.org/pdf/2602.11761) · [DOI 10.48550/arxiv.2602.11761](https://doi.org/10.48550/arxiv.2602.11761) · [arXiv 2602.11761](https://arxiv.org/abs/2602.11761) · [OpenAlex](https://openalex.org/W7128806319)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Full attention in Transformers is costly for very long inputs. Compute grows quadratically with length, and the KV-Cache can reach tens or hundreds of gigabytes at million-token scale. Sparse attention cuts computation but still keeps the whole cache. Linear attention is cheap but compresses context lossily and loses accuracy. MiniCPM-SALA is a 9B model that combines both approaches. About one quarter of its layers use InfLLM-V2 sparse attention, and the rest use Lightning Attention. A layer selection algorithm decides where the sparse layers go. The model also adds HyPE positional encoding, QK-Normalization and output gates.

The model was not trained from scratch. The authors converted an intermediate MiniCPM-4.0 checkpoint through staged continual training, which needed roughly a quarter of the usual budget. General skills stay close to full-attention models such as Qwen3-8B, and long-context scores are stronger. On a single GPU the model runs much faster at long lengths and handles 1M-token inputs that make comparable 8B models run out of memory. This makes long-context inference practical on modest hardware.

## Key points
- **Architecture and training data:**
  - Layers mix InfLLM-V2 (sparse) and Lightning Attention (linear) in a 1:3 ratio.
  - Training runs through several stages: HALO conversion, stable training, short decay, long decay at 32K, 160K and 520K, and SFT.
  - The largest stage uses about 1006.6B tokens.
- **Speed:** On an NVIDIA A6000D at 256K tokens, the model is up to 3.5× faster than Qwen3-8B. Time to first token falls from 180.8 s to 51.6 s.
- **Memory:**
  - The model processes 1024K tokens on both the A6000D and an RTX 5090.
  - On the 5090, Qwen3-8B hits OOM already at 128K without quantization and at 256K with GPTQ INT4.
- **Long-context results:**
  - The average long-context score is 38.97.
  - NoLiMa at 128K reaches 23.86.
  - Although training stopped at 520K, the model extrapolates to 2048K with a score of 81.6, without YaRN. The authors credit NoPE in the sparse layers for this.
  - At 1000K it outperforms Qwen3-Next-80B-A3B-Instruct (86.3 vs 80.3).
- **Release:** Weights are published as openbmb/MiniCPM-SALA on Hugging Face, and code is in the OpenBMB/MiniCPM repository on GitHub.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: MiniCPM Team, Wenhao An, Yingfa Chen, Yewei Fang, Jiayi Li, Xin Li, Yaohui Li, Yishan Li, Yuxuan Li, Biyuan Lin, Chuan Liu, Hezi Liu, Siyuan Liu, Hongya Lyu, Yinxu Pan, Shixin Ren, Xingyu Shen, Zhou Su, Haojun Sun, Yangang Sun, Zhen Leng Thai, Xin Tian, Rui Wang, Xiaorong Wang, Yudong Wang, Bo Wu, Xiaoyue Xu, Dong Xu, Shuaikang Xue, Jiawei Yang, Bowen Zhang, Jinqian Zhang, Letian Zhang, Shengnan Zhang, Xinyu Zhang, Xinyuan Zhang, Zhu Zhang, Hengyu Zhao, Jiacheng Zhao, Zhi Zheng, Jie Zhou, Zihan Zhou, Shuo Wang, Chaojun Xiao, Xu Han, Zhiyuan Liu and Maosong Sun. 2026. MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling. arXiv:2602.11761.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{team2026minicpm,
  title = {{MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling}},
  author = {MiniCPM Team, Wenhao An, Yingfa Chen, Yewei Fang, Jiayi Li, Xin Li, Yaohui Li, Yishan Li, Yuxuan Li, Biyuan Lin, Chuan Liu, Hezi Liu, Siyuan Liu, Hongya Lyu, Yinxu Pan, Shixin Ren, Xingyu Shen, Zhou Su, Haojun Sun, Yangang Sun, Zhen Leng Thai, Xin Tian, Rui Wang, Xiaorong Wang, Yudong Wang, Bo Wu, Xiaoyue Xu, Dong Xu, Shuaikang Xue, Jiawei Yang, Bowen Zhang, Jinqian Zhang, Letian Zhang, Shengnan Zhang, Xinyu Zhang, Xinyuan Zhang, Zhu Zhang, Hengyu Zhao, Jiacheng Zhao, Zhi Zheng, Jie Zhou, Zihan Zhou, Shuo Wang, Chaojun Xiao, Xu Han, Zhiyuan Liu and Maosong Sun},
  year = {2026},
  doi = {10.48550/arxiv.2602.11761},
  eprint = {2602.11761},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2602.11761}
}
```

- [← previous Mind the Gap: Benchmarking LLM Uncertainty and Calibration with Specialty-Aware Clinical QA and Reasoning-Based Behavioural Features](https://md.research.elamur.ai/free/en/papers/testoni-2026-mind-gap-benchmarking-llm-uncertainty/)
- [next → MiniMax Sparse Attention](https://md.research.elamur.ai/free/en/papers/lai-2026-minimax-sparse-attention/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/8/)

## References

61 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. AIME. AIME problems and solutions, 2025. URL [https://artofproblemsolving.com/wiki/index.php/](https://artofproblemsolving.com/wiki/index.php/) AIME_Problems_and_Solutions . [↗](https://artofproblemsolving.com/wiki/index.php/)
2. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. [GQA: Training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://…
3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732, 2021.
4. Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. [LongAlign: A recipe for long context alignment of large language models](https://md.research.elamur.ai/free/en/papers/bai-2024-longalign-recipe-long-context/). In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024. URL https://aclanthology.org/2024.findings-emnlp.74… [↗](https://aclanthology.org/2024.findings-emnlp.74…)
5. Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. [Longwriter: Unleashing 10,000+ word generation from long context LLMs](https://md.research.elamur.ai/free/en/papers/bai-2024-longwriter-unleashing-10-000/). In The Thirteenth International Conference on Learning Representations, 2025. URL [https://openreview.net/forum?id=kQ5s9Yh0WI…](https://openreview.net/forum?id=kQ5s9Yh0WI…) [↗](https://openreview.net/forum?id=kQ5s9Yh0WI…)
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Informa… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+)
7. Curran Associates, Inc., 2020. URL [https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf) . [↗](https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf)
8. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021.
9. Yingfa Chen, Zhen Leng Thai, Zihan Zhou, Zhu Zhang, Xingyu Shen, Shuo Wang, Chaojun Xiao, Xu Han, and Zhiyuan Liu. Hybrid linear attention done right: Efficient distillation and effective architectures for extremely long contexts, 2026. URL https://arxiv.org/abs/2601.22156 . [↗](https://arxiv.org/abs/2601.22156)
10. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. [Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/), 2025. URL https:…
11. OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. [https://github.com/open-compass/opencompass](https://github.com/open-compass/opencompass) , 2023. [↗](https://github.com/open-compass/opencompass)
12. DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. [Deepseek-v3.2: Pushing the frontier of open large language models](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-v3-2-pushing/), 2025. URL https://arxiv.org/abs/2512.02556 . [↗](https://arxiv.org/abs/2512.02556)
13. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023. URL https://arxiv.org/abs/2210.17323 . [↗](https://arxiv.org/abs/2210.17323)
14. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/), 2024. URL https://arxiv.org/abs/2407.21783 . [↗](https://arxiv.org/abs/2407.21783)
15. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024. URL [https://openreview.net/forum?id=tEYskw1VY2](https://openreview.net/forum?id=tEYskw1VY2) . [↗](https://openreview.net/forum?id=tEYskw1VY2)
16. Yuxian Gu, Qinghao Hu, Shang Yang, Haocheng Xi, Junyu Chen, Song Han, and Han Cai. Jet-nemotron: Efficient language model with post neural architecture search, 2025. URL https://arxiv.org/abs/2508. 15884 . [↗](https://arxiv.org/abs/2508)
17. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming – the rise of code intelligence, 2024. URL https://arxiv.org/abs/2401.14196 . [↗](https://arxiv.org/abs/2401.14196)
18. Mutian He and Philip N. Garner. Alleviating forgetfulness of linear attention by hybrid sparse attention and contextualized learnable token eviction, 2025. URL https://arxiv.org/abs/2510.20787 . — [arXiv:2510.20787](https://arxiv.org/abs/2510.20787) [↗](https://arxiv.org/abs/2510.20787)
19. Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-key normalization for transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4246– 4253, 2020. [find](https://scholar.google.com/scholar?q=Alex+Henry%2C+Prudhvi+Raj+Dachapally%2C+Shubham+Shantaram+Pawar%2C+and+Yuxuan+Chen%2E+Query%2Dkey+normalization+for+transformers%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pp%2E+42)
20. Yuichiro Hoshino, Hideyuki Tachibana, Muneyoshi Inahara, and Hiroto Takegawa. Rad: Redundancy-aware distillation for hybrid models via self-speculative decoding, 2025. URL https://arxiv.org/abs/2505. 22135 . [↗](https://arxiv.org/abs/2505)
21. Haowen Hou, Zhiyi Huang, Kaifeng Tan, Rongchang Lu, and Fei Richard Yu. Rwkv-x: A linear complexity hybrid language model, 2025. URL https://arxiv.org/abs/2504.21463 . — [arXiv:2504.21463](https://arxiv.org/abs/2504.21463) [↗](https://arxiv.org/abs/2504.21463)
22. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? In First Conference on Language Modeling, 2024. URL — [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/) [https://openreview.net/forum?id=kIoBbc76Sy](https://openreview.net/forum?id=kIoBbc76Sy) . [↗](https://openreview.net/forum?id=kIoBbc76Sy)
23. Xiang Hu, Jiaqi Leng, Jun Zhao, Kewei Tu, and Wei Wu. Hardware-aligned hierarchical sparse attention for efficient long-term memory access, 2025. URL https://arxiv.org/abs/2504.16795 . — [arXiv:2504.16795](https://arxiv.org/abs/2504.16795) [↗](https://arxiv.org/abs/2504.16795)
24. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. [Livecodebench: Holistic and contamination free evaluation of large language models for code](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/). In The Thirteenth International Conference on Learning Representations, 20…
25. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. [SWE-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=VTF8yNQM66](https://openreview.net/forum?id=VTF8yNQM66) . [↗](https://openreview.net/forum?id=VTF8yNQM66)
26. Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, Wentao Li, et al. [Kimi linear: An expressive, efficient attention architecture](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-linear-expressive-efficient/), 2025. URL https://arxiv.org/abs/2510.26692 . [↗](https://arxiv.org/abs/2510.26692)
27. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. [Cmmlu: Measuring massive multitask language understanding in chinese](https://md.research.elamur.ai/free/en/papers/li-2023-cmmlu-measuring-massive-multitask-language/), 2023.
28. Keyu Li, Junhao Shi, Yang Xiao, Mohan Jiang, Jie Sun, Yunze Wu, Shijie Xia, Xiaojie Cai, Tianze Xu, Weiye Si, Wenjie Li, Dequan Wang, and Pengfei Liu. [Agencybench: Benchmarking the frontiers of autonomous agents in 1m-token real-world contexts](https://md.research.elamur.ai/free/en/papers/li-2026-agencybench-benchmarking-frontiers-autonomous-agents/), 2026. URL https://arxiv.org/abs/2601.11044 . [↗](https://arxiv.org/abs/2601.11044)
29. Yanhong Li, Songlin Yang, Shawn Tan, Mayank Mishra, Rameswar Panda, Jiawei Zhou, and Yoon Kim. Distilling to hybrid attention models via kl-guided layer selection, 2025. URL https://arxiv.org/abs/ 2512.20569 . [↗](https://arxiv.org/abs/)
30. Alexander H. Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, et al. Ministral 3, 2026. URL https://arxiv.org/abs/2601.08584 . [↗](https://arxiv.org/abs/2601.08584)
31. Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code autocompletion systems. In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=pPjZIOuQuF](https://openreview.net/forum?id=pPjZIOuQuF) . [↗](https://openreview.net/forum?id=pPjZIOuQuF)
32. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. [Gaia: a benchmark for general ai assistants](https://md.research.elamur.ai/free/en/papers/mialon-2023-gaia-benchmark-general-ai-assistants/), 2023. URL https://arxiv.org/abs/2311.12983 . [↗](https://arxiv.org/abs/2311.12983)
33. MiniCPM-Team, Chaojun Xiao, Yuxuan Li, Xu Han, Yuzhuo Bai, Jie Cai, Haotian Chen, Wentong Chen, Xin Cong, Ganqu Cui, Ning Ding, et al. [Minicpm4: Ultra-efficient llms on end devices](https://md.research.elamur.ai/free/en/papers/minicpm-team-2025-minicpm4-ultra-efficient-llms/), 2025. URL https://arxiv.org/abs/2506.07900 . [↗](https://arxiv.org/abs/2506.07900)
34. Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A. Rossi, Seunghyun Yoon, and Hinrich Schuetze. Nolima: Long-context evaluation beyond literal matching. In Forty-second International Conference on Machine Learning, 2025. URL [https://openreview.net/forum?id=0OshX1hiSa](https://openreview.net/forum?id=0OshX1hiSa) . [↗](https://openreview.net/forum?id=0OshX1hiSa)
35. NVIDIA, Aarti Basant, Abhijit Khairnar, Abhijit Paithankar, Abhinav Khattar, Adithya Renduchintala, Aditya Malte, Akhiad Bercovich, Akshay Hazare, Alejandra Rico, et al. [Nvidia nemotron nano 2: An accurate and efficient hybrid mamba-transformer reasoning model](https://md.research.elamur.ai/free/en/papers/basant-2025-nvidia-nemotron-nano-2-accurate/), 2025a. URL https://arxiv.org/abs/2508.… [↗](https://arxiv.org/abs/2508.…)
36. NVIDIA, Aaron Blakeman, Aaron Grattafiori, Aarti Basant, Abhibha Gupta, Abhinav Khattar, Adi Renduchintala, Aditya Vavre, Akanksha Shukla, Akhiad Bercovich, Aleksander Ficek, et al. [Nemotron 3 nano: Open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning](https://md.research.elamur.ai/free/en/papers/blakeman-2025-nemotron-3-nano-open-efficient/), 2025b. URL h…
37. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2024. URL https://arxiv.org/abs/2303.08774 . [↗](https://arxiv.org/abs/2303.08774)
38. Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartłomiej Koptyra, Hayden Lau, Jiaju Lin, Kri… [find](https://scholar.google.com/scholar?q=Bo+Peng%2C+Eric+Alcaide%2C+Quentin+Anthony%2C+Alon+Albalak%2C+Samuel+Arcadinho%2C+Stella+Biderman%2C+Huanqi+Cao%2C+Xin+Cheng%2C+Michael+Chung%2C+Leon+Derczynski%2C+Xingjian+Du%2C+Matteo+Grella%2C+Kranthi+Gv%2C+Xuzheng+He%2C+Haow)
39. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=wHBfxhZu1u](https://openreview.net/forum?id=wHBfxhZu1u) . [↗](https://openreview.net/forum?id=wHBfxhZu1u)
40. Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. [ChatDev: Communicative agents for software development](https://md.research.elamur.ai/free/en/papers/qian-2023-chatdev-communicative-agents-software/). In Proceedings of the 62nd Annual Meeting of the Association for Computational…
41. Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Various lengths, constant speed: Efficient language modeling with lightning attention. In Forty-first International Conference on Machine Learning, 2024. URL [https://openreview.net/forum?id=Lwm6TiUP4X](https://openreview.net/forum?id=Lwm6TiUP4X) . [↗](https://openreview.net/forum?id=Lwm6TiUP4X)
42. Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free. In The Thirty-ninth Annual Conference on Neural In… [find](https://scholar.google.com/scholar?q=Zihan+Qiu%2C+Zekun+Wang%2C+Bo+Zheng%2C+Zeyu+Huang%2C+Kaiyue+Wen%2C+Songlin+Yang%2C+Rui+Men%2C+Le+Yu%2C+Fei+Huang%2C+Suozhi+Huang%2C+Dayiheng+Liu%2C+Jingren+Zhou%2C+and+Junyang+Lin%2E+Gated+attention+for+large+language+models%3A+)
43. Qwen Team. Qwen3-Next: Towards Ultimate Training & Inference Efficiency, 2025. URL [https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd) . [↗](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd)
44. Yijia Shao, Yucheng Jiang, Theodore Kanell, Peter Xu, Omar Khattab, and Monica Lam. [Assisting in writing Wikipedia-like articles from scratch with large language models](https://md.research.elamur.ai/free/en/papers/shao-2024-assisting-writing-wikipedia-like-articles/). In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua…
45. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/), 2023. URL https://arxiv.org/abs/2104.09864 . [↗](https://arxiv.org/abs/2104.09864)
46. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. [Challenging big-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). arXiv preprint arXiv:2210.09261, 2022.
47. Falcon LLM Team, Iheb Chaabane, Puneesh Khanna, Suhail Mohmad, Slim Frikha, Shi Hu, Abdalgader Abubaker, Reda Alami, Mikhail Lubinets, Mohamed El Amine Seddik, and Hakim Hacid. Falcon-h1r: Pushing the reasoning frontiers with a hybrid model for efficient test-time scaling, 2026. URL [https://arxiv.o…](https://arxiv.o…) — [arXiv:2601.02346](https://arxiv.org/abs/2601.02346) [↗](https://arxiv.o…)
48. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Sy…
49. Junxiong Wang, Daniele Paliotta, Avner May, Alexander M. Rush, and Tri Dao. The mamba in the llama: Distilling and accelerating hybrid models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp… [find](https://scholar.google.com/scholar?q=Junxiong+Wang%2C+Daniele+Paliotta%2C+Avner+May%2C+Alexander+M%2E+Rush%2C+and+Tri+Dao%2E+The+mamba+in+the+llama%3A+Distilling+and+accelerating+hybrid+models%2E+In+A%2E+Globerson%2C+L%2E+Mackey%2C+D%2E+Belgrave%2C+A%2E+Fan%2C+U%2E+Paque)
50. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. [Mmlu-pro: A more robust and challenging multi-task language understanding benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/). In A…
51. Yudong Wang, Zixuan Fu, Hengyu Zhao, Chen Zhao, Chuyue Zhou, Xinle Lin, Hongya Lyu, Shuaikang Xue, Yi Yi, Yingjiao Wang, Zhi Zheng, Yuzhou Zhang, Jie Zhou, Chaojun Xiao, Xu Han, Zhiyuan Liu, and Maosong Sun. Data science and technology towards agi part i: Tiered data management, 2026. URL https://ar… — [arXiv:2602.09003](https://arxiv.org/abs/2602.09003) [find](https://scholar.google.com/scholar?q=Yudong+Wang%2C+Zixuan+Fu%2C+Hengyu+Zhao%2C+Chen+Zhao%2C+Chuyue+Zhou%2C+Xinle+Lin%2C+Hongya+Lyu%2C+Shuaikang+Xue%2C+Yi+Yi%2C+Yingjiao+Wang%2C+Zhi+Zheng%2C+Yuzhou+Zhang%2C+Jie+Zhou%2C+Chaojun+Xiao%2C+Xu+Han%2C+Zhiyuan+Liu%2C+and+Maoso)
52. Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. [Infllm: Training-free long-context extrapolation for llms with an efficient context memory](https://md.research.elamur.ai/free/en/papers/xiao-2024-infllm-training-free-long-context/). In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advan…
53. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/), 2025a. URL https://arxiv.org/abs/2505.09388 . [↗](https://arxiv.org/abs/2505.09388)
54. Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. In Forty-first International Conference on Machine Learning, 2024a. URL [https://openreview.net/forum?id=ia5XvxFUJT](https://openreview.net/forum?id=ia5XvxFUJT) . [↗](https://openreview.net/forum?id=ia5XvxFUJT)
55. Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL [https://openreview.net/forum?id=y8Rm4VNRPH](https://openreview.net/forum?id=y8Rm4VNRPH) . [↗](https://openreview.net/forum?id=y8Rm4VNRPH)
56. Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. In The Thirteenth International Conference on Learning Representations, 2025b. URL [https://openreview.net/forum?id=r8H7xhYPwz](https://openreview.net/forum?id=r8H7xhYPwz) . [↗](https://openreview.net/forum?id=r8H7xhYPwz)
57. Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. [Native sparse attention: Hardware-aligned and natively trainable sparse attention](https://md.research.elamur.ai/free/en/papers/yuan-2025-native-sparse-attention-hardware/). In Proceedings of th…
58. Weilin Zhao, Zihan Zhou, Zhou Su, Chaojun Xiao, Yuxuan Li, Yanghao Li, Yudi Zhang, Weilun Zhao, Zhen Li, Yuxiang Huang, Ao Sun, Xu Han, and Zhiyuan Liu. Infllm-v2: Dense-sparse switchable attention for seamless short-to-long adaptation, 2025. URL https://arxiv.org/abs/2509.24663 . — [arXiv:2509.24663](https://arxiv.org/abs/2509.24663) [↗](https://arxiv.org/abs/2509.24663)
59. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. [Instruction-following evaluation for large language models](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/), 2023. URL https://arxiv.org/abs/ 2311.07911 . [↗](https://arxiv.org/abs/)
60. Zihan Zhou, Chong Li, Xinyi Chen, Shuo Wang, Yu Chao, Zhili Li, Haoyu Wang, Qi Shi, Zhixing Tan, Xu Han, Xiaodong Shi, Zhiyuan Liu, and Maosong Sun. LLM×MapReduce: Simplified long-sequence processing using large language models. In Proceedings of the 63rd Annual Meeting of the Association for Comput… [find](https://scholar.google.com/scholar?q=Zihan+Zhou%2C+Chong+Li%2C+Xinyi+Chen%2C+Shuo+Wang%2C+Yu+Chao%2C+Zhili+Li%2C+Haoyu+Wang%2C+Qi+Shi%2C+Zhixing+Tan%2C+Xu+Han%2C+Xiaodong+Shi%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+LLM%C3%97MapReduce%3A+Simplified+long%2Dsequence+processing+)
61. Jingwei Zuo, Maksim Velikanov, Ilyas Chahed, Younes Belkada, Dhia Eddine Rhayem, Guillaume Kunsch, Hakim Hacid, Hamza Yous, Brahim Farhat, Ibrahim Khadraoui, Mugariya Farooq, Giulia Campesan, Ruxandra Cojocaru, Yasser Djilali, Shi Hu, Iheb Chaabane, Puneesh Khanna, Mohamed El Amine Seddik, Ngoc Dung… — [Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance](https://md.research.elamur.ai/free/en/papers/zuo-2025-falcon-h1-family-hybrid-head/)

*Updated 31 October 2026*