---
title: "MiniMax-01: Scaling Foundation Models with Lightning Attention"
description: "Most current LLMs and VLMs handle 32K–256K tokens of context. 115 references, cited by 50 papers here."
authors: ["MiniMax", "Aonian Li", "Bangwei Gong", "Bo Yang", "Boji Shan", "Chang Liu", "Cheng Zhu", "Chunhao Zhang", "Congchao Guo", "Da Chen", "Dong Li", "Enwei Jiao", "Gengxin Li", "Guojun Zhang", "Haohai Sun", "Houze Dong", "Jiadai Zhu", "Jiaqi Zhuang", "Jiayuan Song", "Jin Zhu", "Jingtao Han", "Jingyang Li", "Junbin Xie", "Junhao Xu", "Junjie Yan", "Kaishun Zhang", "Kecheng Xiao", "Kexi Kang", "Le Han", "Leyang Wang", "Lianfei Yu", "Liheng Feng", "Lin Zheng", "Linbo Chai", "Long Xing", "Meizhi Ju", "Mingyuan Chi", "Mozhi Zhang", "Peikai Huang", "Pengcheng Niu", "Pengfei Li", "Pengyu Zhao", "Qi Yang", "Qidi Xu", "Qiexiang Wang", "Qin Wang", "Qiuhui Li", "Ruitao Leng", "Shengmin Shi", "Shuqi Yu", "Sichen Li", "Songquan Zhu", "Tao Huang", "Tianrun Liang", "Weigao Sun", "Weixuan Sun", "Weiyu Cheng", "Wenkai Li", "Xiangjun Song", "Xiao Su", "Xiaodong Han", "Xinjie Zhang", "Xinzhu Hou", "Xu Min", "Xun Zou", "Xuyang Shen", "Yan Gong", "Yingjie Zhu", "Yipeng Zhou", "Yiran Zhong", "Yongyi Hu", "Yuanxiang Fan", "Yue Yu", "Yufeng Yang", "Yuhao Li", "Yunan Huang", "Yunji Li", "Yunpeng Huang", "Yunzhi Xu", "Yuxin Mao", "Zehan Li", "Zekang Li", "Zewei Tao", "Zewen Ying", "Zhaoyang Cong", "Zhen Qin", "Zhenhua Fan", "Zhihang Yu", "Zhuo Jiang", "Zijia Wu"]
year: 2025
venue: "arXiv:2501.08313"
source: "https://arxiv.org/abs/2501.08313"
doi: "10.48550/arxiv.2501.08313"
arxiv: "2501.08313"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/minimax-2025-minimax-01-scaling-foundation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/minimax-2025-minimax-01-scaling-foundation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) / MiniMax-01: Scaling Foundation Models with Lightning Attention

# MiniMax-01: Scaling Foundation Models with Lightning Attention

**Authors:** [MiniMax](https://md.research.elamur.ai/free/en/authors/minimax/), [Aonian Li](https://md.research.elamur.ai/free/en/authors/aonian-li/), [Bangwei Gong](https://md.research.elamur.ai/free/en/authors/bangwei-gong/), [Bo Yang](https://md.research.elamur.ai/free/en/authors/bo-yang/), [Boji Shan](https://md.research.elamur.ai/free/en/authors/boji-shan/), [Chang Liu](https://md.research.elamur.ai/free/en/authors/chang-liu/), [Cheng Zhu](https://md.research.elamur.ai/free/en/authors/cheng-zhu/), Chunhao Zhang, Congchao Guo, [Da Chen](https://md.research.elamur.ai/free/en/authors/da-chen/), [Dong Li](https://md.research.elamur.ai/free/en/authors/dong-li/), Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang, Jiayuan Song, Jin Zhu, Jingtao Han, Jingyang Li, Junbin Xie, Junhao Xu, Junjie Yan, Kaishun Zhang, Kecheng Xiao, Kexi Kang, Le Han, Leyang Wang, Lianfei Yu, Liheng Feng, Lin Zheng, Linbo Chai, Long Xing, Meizhi Ju, Mingyuan Chi, Mozhi Zhang, Peikai Huang, Pengcheng Niu, [Pengfei Li](https://md.research.elamur.ai/free/en/authors/pengfei-li/), Pengyu Zhao, Qi Yang, Qidi Xu, Qiexiang Wang, Qin Wang, Qiuhui Li, Ruitao Leng, Shengmin Shi, Shuqi Yu, Sichen Li, Songquan Zhu, Tao Huang, Tianrun Liang, [Weigao Sun](https://md.research.elamur.ai/free/en/authors/weigao-sun/), [Weixuan Sun](https://md.research.elamur.ai/free/en/authors/weixuan-sun/), Weiyu Cheng, [Wenkai Li](https://md.research.elamur.ai/free/en/authors/wenkai-li/), Xiangjun Song, Xiao Su, [Xiaodong Han](https://md.research.elamur.ai/free/en/authors/xiaodong-han/), Xinjie Zhang, Xinzhu Hou, Xu Min, Xun Zou, [Xuyang Shen](https://md.research.elamur.ai/free/en/authors/xuyang-shen/), Yan Gong, Yingjie Zhu, Yipeng Zhou, [Yiran Zhong](https://md.research.elamur.ai/free/en/authors/yiran-zhong/), Yongyi Hu, Yuanxiang Fan, [Yue Yu](https://md.research.elamur.ai/free/en/authors/yue-yu/), Yufeng Yang, Yuhao Li, Yunan Huang, Yunji Li, Yunpeng Huang, Yunzhi Xu, Yuxin Mao, [Zehan Li](https://md.research.elamur.ai/free/en/authors/zehan-li/), Zekang Li, Zewei Tao, Zewen Ying, Zhaoyang Cong, [Zhen Qin](https://md.research.elamur.ai/free/en/authors/zhen-qin/), Zhenhua Fan, Zhihang Yu, Zhuo Jiang and Zijia Wu · **Year:** 2025 · **Venue:** arXiv:2501.08313 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2501.08313) · [PDF](https://arxiv.org/pdf/2501.08313) · [DOI 10.48550/arxiv.2501.08313](https://doi.org/10.48550/arxiv.2501.08313) · [arXiv 2501.08313](https://arxiv.org/abs/2501.08313) · [OpenAlex](https://openalex.org/W4406465766)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [50](https://md.research.elamur.ai/free/en/papers/minimax-2025-minimax-01-scaling-foundation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most current LLMs and VLMs handle 32K–256K tokens of context. That is often too little for tasks like reading whole books, working across entire code repositories, or many-shot in-context learning. The main obstacle is that softmax attention cost grows quadratically with length. MiniMax-01 addresses this with a hybrid architecture built mostly on lightning attention, an I/O-aware linear attention variant, combined with a Mixture of Experts. The authors redesigned their training and inference stack around this architecture. It uses expert parallelism, varlen ring attention, an improved LASP, and custom CUDA kernels. MiniMax-Text-01 was pre-trained on a carefully cleaned and rebalanced corpus and extended to long contexts in three stages. MiniMax-VL-01 adds a lightweight ViT and further multimodal training. Both models perform comparably to GPT-4o and Claude-3.5-Sonnet on standard benchmarks while accepting 20–32 times more context. They are especially strong beyond 200K tokens. The work matters because it is the first deployment of linear attention at this scale, and the weights are openly released.

## Key points
- **Architecture:** 456B total parameters with 45.9B active per token, 32 experts using top-2 routing, and 80 layers. One softmax-attention (GQA) block follows every seven lightning-attention blocks.
- **Context length:** up to 1M tokens during training and extrapolation to 4M tokens at inference. The model was sized to process over 1M tokens on a single 8-GPU, 640GB machine using 8-bit quantization.
- **Efficiency:** end-to-end inference reaches over 75% MFU on Nvidia H20, and prefilling latency is much lower than that of softmax-based models.
- **MiniMax-VL-01:** trained on an additional 512B vision-language tokens in four stages. It scores 68.5 on MMMU and 96.4 on DocVQA but remains weak on OlympiadBench.
- **Limitations and release:** a softmax component (1/8 of layers) still remains, long-context evaluation tasks are artificial, and coding ability is limited. Weights are available at github.com/MiniMax-AI.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: MiniMax, Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang, Jiayuan Song, Jin Zhu, Jingtao Han, Jingyang Li, Junbin Xie, Junhao Xu, Junjie Yan, Kaishun Zhang, Kecheng Xiao, Kexi Kang, Le Han, Leyang Wang, Lianfei Yu, Liheng Feng, Lin Zheng, Linbo Chai, Long Xing, Meizhi Ju, Mingyuan Chi, Mozhi Zhang, Peikai Huang, Pengcheng Niu, Pengfei Li, Pengyu Zhao, Qi Yang, Qidi Xu, Qiexiang Wang, Qin Wang, Qiuhui Li, Ruitao Leng, Shengmin Shi, Shuqi Yu, Sichen Li, Songquan Zhu, Tao Huang, Tianrun Liang, Weigao Sun, Weixuan Sun, Weiyu Cheng, Wenkai Li, Xiangjun Song, Xiao Su, Xiaodong Han, Xinjie Zhang, Xinzhu Hou, Xu Min, Xun Zou, Xuyang Shen, Yan Gong, Yingjie Zhu, Yipeng Zhou, Yiran Zhong, Yongyi Hu, Yuanxiang Fan, Yue Yu, Yufeng Yang, Yuhao Li, Yunan Huang, Yunji Li, Yunpeng Huang, Yunzhi Xu, Yuxin Mao, Zehan Li, Zekang Li, Zewei Tao, Zewen Ying, Zhaoyang Cong, Zhen Qin, Zhenhua Fan, Zhihang Yu, Zhuo Jiang and Zijia Wu. 2025. MiniMax-01: Scaling Foundation Models with Lightning Attention. arXiv:2501.08313.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{minimax2025minimax,
  title = {{MiniMax-01: Scaling Foundation Models with Lightning Attention}},
  author = {MiniMax, Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang, Jiayuan Song, Jin Zhu, Jingtao Han, Jingyang Li, Junbin Xie, Junhao Xu, Junjie Yan, Kaishun Zhang, Kecheng Xiao, Kexi Kang, Le Han, Leyang Wang, Lianfei Yu, Liheng Feng, Lin Zheng, Linbo Chai, Long Xing, Meizhi Ju, Mingyuan Chi, Mozhi Zhang, Peikai Huang, Pengcheng Niu, Pengfei Li, Pengyu Zhao, Qi Yang, Qidi Xu, Qiexiang Wang, Qin Wang, Qiuhui Li, Ruitao Leng, Shengmin Shi, Shuqi Yu, Sichen Li, Songquan Zhu, Tao Huang, Tianrun Liang, Weigao Sun, Weixuan Sun, Weiyu Cheng, Wenkai Li, Xiangjun Song, Xiao Su, Xiaodong Han, Xinjie Zhang, Xinzhu Hou, Xu Min, Xun Zou, Xuyang Shen, Yan Gong, Yingjie Zhu, Yipeng Zhou, Yiran Zhong, Yongyi Hu, Yuanxiang Fan, Yue Yu, Yufeng Yang, Yuhao Li, Yunan Huang, Yunji Li, Yunpeng Huang, Yunzhi Xu, Yuxin Mao, Zehan Li, Zekang Li, Zewei Tao, Zewen Ying, Zhaoyang Cong, Zhen Qin, Zhenhua Fan, Zhihang Yu, Zhuo Jiang and Zijia Wu},
  year = {2025},
  doi = {10.48550/arxiv.2501.08313},
  eprint = {2501.08313},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.08313}
}
```

- [← previous LightThinker: Thinking Step-by-Step Compression](https://md.research.elamur.ai/free/en/papers/zhang-2025-lightthinker-thinking-step-step-compression/)
- [next → WebDancer: Towards Autonomous Information Seeking Agency](https://md.research.elamur.ai/free/en/papers/wu-2025-webdancer-towards-autonomous-information-seeking/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/)

## References

115 references, 66 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. [Phi-4 technical report](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-4-technical-report/). arXiv preprint arXiv:2412.08905, 2024.
2. Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. [Many-shot in-context learning](https://md.research.elamur.ai/free/en/papers/agarwal-2024-many-shot-context-learning/). arXiv preprint arXiv:2404.11018, 2024.
3. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. [GQA: Training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901, 20…
4. Anthropic. Introducing claude 3.5 sonnet, 2024. URL [https://www.anthropic.com/news/claude-3-5-sonnet](https://www.anthropic.com/news/claude-3-5-sonnet) . [↗](https://www.anthropic.com/news/claude-3-5-sonnet)
5. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732, 2021.
6. Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853, 2018. [find](https://scholar.google.com/scholar?q=Alexei+Baevski+and+Michael+Auli%2E+Adaptive+input+representations+for+neural+language+modeling%2E+arXiv+preprint+arXiv%3A1809%2E10853%2C+2018%2E)
7. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+Training+a+helpful+and+harmless+assistant+with+reinforcem)
8. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. [Constitutional AI: Harmlessness from AI feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073, 2022b.
9. Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. [LongBench v2: Towards deeper understanding and reasoning on realistic long-context multitasks](https://md.research.elamur.ai/free/en/papers/bai-2024-longbench-v2-deeper-understanding/). arXiv preprint arXiv:2412.15204, 2024.
10. Iz Beltagy, Matthew E Peters, and Arman Cohan. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). arXiv preprint arXiv:2004.05150, 2020.
11. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. [PIQA: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020.
12. Andrei Z Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pages 21–29. [find](https://scholar.google.com/scholar?q=Andrei+Z+Broder%2E+On+the+resemblance+and+containment+of+documents%2E+In+Proceedings%2E+Compression+and+Complexity+of+SEQUENCES+1997+%28Cat%2E+No%2E+97TB100171%29%2C+pages+21%E2%80%9329%2E)
13. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
14. Jiacheng Chen, Tianhao Liang, Sherman Siu, Zhengqing Wang, Kai Wang, Yubo Wang, Yuansheng Ni, Wang Zhu, Ziyan Jiang, Bohan Lyu, et al. MEGA-Bench: Scaling multimodal evaluation to over 500 real-world tasks. arXiv preprint arXiv:2410.10563, 2024a. — [arXiv:2410.10563](https://arxiv.org/abs/2410.10563) [find](https://scholar.google.com/scholar?q=Jiacheng+Chen%2C+Tianhao+Liang%2C+Sherman+Siu%2C+Zhengqing+Wang%2C+Kai+Wang%2C+Yubo+Wang%2C+Yuansheng+Ni%2C+Wang+Zhu%2C+Ziyan+Jiang%2C+Bohan+Lyu%2C+et+al%2E+MEGA%2DBench%3A+Scaling+multimodal+evaluation+to+over+500+real%2Dworld+)
15. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021.
16. Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, and Feng Zhao. [MindSearch: Mimicking human minds elicits deep ai searcher](https://md.research.elamur.ai/free/en/papers/chen-2024-mindsearch-mimicking-human-minds-elicits/). arXiv preprint arXiv:2407.20183, 2024b.
17. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. [Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/). arXiv preprint arXiv:1904.10509, 2019.
18. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. [PaLM: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research, 24(240):1–113, 2023.
19. Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International Conference on Machine Learning (ICML), pages 4057–4086. PMLR, 2022. [find](https://scholar.google.com/scholar?q=Aidan+Clark%2C+Diego+de+Las+Casas%2C+Aurelia+Guy%2C+Arthur+Mensch%2C+Michela+Paganini%2C+Jordan+Hoffmann%2C+Bogdan+Damoc%2C+Blake+Hechtman%2C+Trevor+Cai%2C+Sebastian+Borgeaud%2C+et+al%2E+Unified+scaling+laws+for+routed+lan)
20. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. [BoolQ: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human…
21. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. [Think you have solved question answering? try ARC, the AI2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457, 2018.
22. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
23. Tri Dao. [FlashAttention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=mZn2Xyh9Ec](https://openreview.net/forum?id=mZn2Xyh9Ec) . [↗](https://openreview.net/forum?id=mZn2Xyh9Ec)
24. Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024. [find](https://scholar.google.com/scholar?q=Tri+Dao+and+Albert+Gu%2E+Transformers+are+ssms%3A+Generalized+models+and+efficient+algorithms+through+structured+state+space+duality%2E+arXiv+preprint+arXiv%3A2405%2E21060%2C+2024%2E)
25. Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 16344–16359, 2022. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Dan+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+FlashAttention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A+1634)
26. Alexandre de Brébisson and Pascal Vincent. A cheap linear attention mechanism with fast lookups and fixed-size representations. arXiv preprint arXiv:1609.05866, 2016. — [arXiv:1609.05866](https://arxiv.org/abs/1609.05866) [find](https://scholar.google.com/scholar?q=Alexandre+de+Br%C3%A9bisson+and+Pascal+Vincent%2E+A+cheap+linear+attention+mechanism+with+fast+lookups+and+fixed%2Dsize+representations%2E+arXiv+preprint+arXiv%3A1609%2E05866%2C+2016%2E+%E2%80%94+%5BarXiv%3A1609%2E05866%5D%28https%3A%2F%2Farxi)
27. DeepSeek-AI. DeepSeek-V3 technical report, 2024. URL https://arxiv.org/abs/2412.19437 . [↗](https://arxiv.org/abs/2412.19437)
28. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, et al. [A survey on in-context learning](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107–1128, 2024.
29. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). In Internationa…
30. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. [DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hum…
31. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. [The Llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783, 2024.
32. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Barret+Zoph%2C+and+Noam+Shazeer%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+Journal+of+Machine+Learning+Research%2C+23%28120%29%3A1%E2%80%9339%2C+2022%2E)
33. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Tom+Dupr%C3%A9+la+Tour%2C+Henk+Tillman%2C+Gabriel+Goh%2C+Rajan+Troll%2C+Alec+Radford%2C+Ilya+Sutskever%2C+Jan+Leike%2C+and+Jeffrey+Wu%2E+Scaling+and+evaluating+sparse+autoencoders%2E+arXiv+preprint+arXiv%3A2406%2E04093)
34. Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. Zamba: A compact 7b SSM hybrid model. arXiv preprint arXiv:2405.16712, 2024. [find](https://scholar.google.com/scholar?q=Paolo+Glorioso%2C+Quentin+Anthony%2C+Yury+Tokpanov%2C+James+Whittington%2C+Jonathan+Pilault%2C+Adam+Ibrahim%2C+and+Beren+Millidge%2E+Zamba%3A+A+compact+7b+SSM+hybrid+model%2E+arXiv+preprint+arXiv%3A2405%2E16712%2C+2024%2E)
35. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010. [find](https://scholar.google.com/scholar?q=Xavier+Glorot+and+Yoshua+Bengio%2E+Understanding+the+difficulty+of+training+deep+feedforward+neural+networks%2E+In+Proceedings+of+the+thirteenth+international+conference+on+artificial+intelligence+and+sta)
36. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024. URL [https://openreview.net/forum?id=tEYskw1VY2](https://openreview.net/forum?id=tEYskw1VY2) . [↗](https://openreview.net/forum?id=tEYskw1VY2)
37. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023. [find](https://scholar.google.com/scholar?q=Suriya+Gunasekar%2C+Yi+Zhang%2C+Jyoti+Aneja%2C+Caio+C%C3%A9sar+Teodoro+Mendes%2C+Allie+Del+Giorno%2C+Sivakanth+Gopi%2C+Mojan+Javaheripi%2C+Piero+Kauffmann%2C+Gustavo+de+Rosa%2C+Olli+Saarikivi%2C+et+al%2E+Textbooks+are+all+you+n)
38. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. [OlympiadBench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems](https://md.research.elamur.ai/free/en/papers/he-2024-olympiadbench-challenging-benchmark-promoting-agi/). arXiv preprint arXiv:2402.14008, 2024a.
39. Yancheng He, Shilong Li, Jiaheng Liu, Yingshui Tan, Weixun Wang, Hui Huang, Xingyuan Bu, Hangyu Guo, Chengwei Hu, Boren Zheng, et al. [Chinese simpleQA: A chinese factuality evaluation for large language models](https://md.research.elamur.ai/free/en/papers/he-2024-chinese-simpleqa-chinese-factuality/). arXiv preprint arXiv:2411.07140, 2024b.
40. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations, 2021a. URL [https://openreview.net/forum?id=d7KBjmI3GmQ](https://openreview.net/forum?id=d7KBjmI3GmQ) . [↗](https://openreview.net/forum?id=d7KBjmI3GmQ)
41. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the MATH dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021b. URL h…
42. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, et al. [Scaling laws for autoregressive generative modeling](https://md.research.elamur.ai/free/en/papers/henighan-2020-scaling-laws-autoregressive-generative-modeling/). arXiv preprint arXiv:2010.14701, 2020.
43. Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. [Scaling laws and interpretability of learning from repeated data](https://md.research.elamur.ai/free/en/papers/hernandez-2022-scaling-laws-interpretability-learning-repeated/). arXiv preprint arXiv:2205.10487, 2022.
44. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). In Proceedings of the 36th International Conference on Neural Information…
45. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.[06654, 2024](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/).
46. Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. Transformer quality in linear time. In International conference on machine learning, pages 9099–9117. PMLR, 2022. [find](https://scholar.google.com/scholar?q=Weizhe+Hua%2C+Zihang+Dai%2C+Hanxiao+Liu%2C+and+Quoc+Le%2E+Transformer+quality+in+linear+time%2E+In+International+conference+on+machine+learning%2C+pages+9099%E2%80%939117%2E+PMLR%2C+2022%2E)
47. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. [GPT-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). arXiv preprint arXiv:2410.21276, 2024.
48. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+Mistral+7B)
49. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. [Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, 2017.
50. G. Kamradt. Llmtest_needleinahaystack, 2023. URL [https://github.com/gkamradt/LLMTest_NeedleInAHaystack](https://github.com/gkamradt/LLMTest_NeedleInAHaystack) . [↗](https://github.com/gkamradt/LLMTest_NeedleInAHaystack)
51. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361, 2020.
52. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. [A diagram is worth a dozen images](https://md.research.elamur.ai/free/en/papers/kembhavi-2016-diagram-worth-dozen-images/). In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–251.
53. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. [Natural questions: a benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Linguistics, 7:…
54. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages… [find](https://scholar.google.com/scholar?q=Katherine+Lee%2C+Daphne+Ippolito%2C+Andrew+Nystrom%2C+Chiyuan+Zhang%2C+Douglas+Eck%2C+Chris+Callison%2DBurch%2C+and+Nicholas+Carlini%2E+Deduplicating+training+data+makes+language+models+better%2E+In+Proceedings+of+the+)
55. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021. URL https://openreview… [find](https://scholar.google.com/scholar?q=Dmitry+Lepikhin%2C+HyoukJoong+Lee%2C+Yuanzhong+Xu%2C+Dehao+Chen%2C+Orhan+Firat%2C+Yanping+Huang%2C+Maxim+Krikun%2C+Noam+Shazeer%2C+and+Zhifeng+Chen%2E+GShard%3A+Scaling+giant+models+with+conditional+computation+and+autom)
56. Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of-experts model. arXiv preprint arXiv:2410.05993, 2024a. — [arXiv:2410.05993](https://arxiv.org/abs/2410.05993) [find](https://scholar.google.com/scholar?q=Dongxu+Li%2C+Yudong+Liu%2C+Haoning+Wu%2C+Yue+Wang%2C+Zhiqi+Shen%2C+Bowen+Qu%2C+Xinyao+Niu%2C+Guoyin+Wang%2C+Bei+Chen%2C+and+Junnan+Li%2E+Aria%3A+An+open+multimodal+native+mixture%2Dof%2Dexperts+model%2E+arXiv+preprint+arXiv%3A2410)
57. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. [BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/). In International conference on machine learning, pages 12888–12900. PMLR, 2022.
58. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. [BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). In International conference on machine learning, pages 19730–19742. PMLR, 2023.
59. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. [From crowdsourced data to high-quality benchmarks: Arena-Hard and BenchBuilder pipeline](https://md.research.elamur.ai/free/en/papers/li-2024-crowdsourced-data-high-quality-benchmarks/). arXiv preprint arXiv:2406.11939, 2024b.
60. Hao Liu, Matei Zaharia, and Pieter Abbeel. [Ring attention with blockwise transformers for near-infinite context](https://md.research.elamur.ai/free/en/papers/liu-2023-ring-attention-blockwise-transformers-near/). In The Twelfth International Conference on Learning Representations, 2024a. URL [https://openreview.net/forum?id=WsRHpHH4s0](https://openreview.net/forum?id=WsRHpHH4s0) . [↗](https://openreview.net/forum?id=WsRHpHH4s0)
61. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. [Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/). In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 21558–21572, 2023.
62. Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. [OCRBench: On the hidden mystery of OCR in large multimodal models](https://md.research.elamur.ai/free/en/papers/liu-2023-ocrbench-hidden-mystery-ocr-large/). Science China Information Sciences, 67(12):220102, 2024b.
63. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL [https://openreview.net/forum?id=Bkg6RiCqY7](https://openreview.net/forum?id=Bkg6RiCqY7) . [↗](https://openreview.net/forum?id=Bkg6RiCqY7)
64. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. [MathVista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). arXiv preprint arXiv:2310.02255, 2023.
65. Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al. MMLongBench-Doc: Benchmarking long-context document understanding with visualizations. arXiv preprint arXiv:2407.01523, 2024. [find](https://scholar.google.com/scholar?q=Yubo+Ma%2C+Yuhang+Zang%2C+Liangyu+Chen%2C+Meiqi+Chen%2C+Yizhu+Jiao%2C+Xinze+Li%2C+Xinyuan+Lu%2C+Ziyu+Liu%2C+Yan+Ma%2C+Xiaoyi+Dong%2C+et+al%2E+MMLongBench%2DDoc%3A+Benchmarking+long%2Dcontext+document+understanding+with+visualiza)
66. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. [ChartQA: A benchmark for question answering about charts with visual and logical reasoning](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/). arXiv preprint arXiv:2203.10244, 2022.
67. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. [DocVQA: A dataset for VQA on document images](https://md.research.elamur.ai/free/en/papers/mathew-2020-docvqa-dataset-vqa-document-images/). In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021.
68. Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162, 2018. [find](https://scholar.google.com/scholar?q=Sam+McCandlish%2C+Jared+Kaplan%2C+Dario+Amodei%2C+and+OpenAI+Dota+Team%2E+An+empirical+model+of+large%2Dbatch+training%2E+arXiv+preprint+arXiv%3A1812%2E06162%2C+2018%2E)
69. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, 2018. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+In+Proceedings+of+the+2018+Conference+on+Empiri)
70. Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358–50376, 2023. [find](https://scholar.google.com/scholar?q=Niklas+Muennighoff%2C+Alexander+Rush%2C+Boaz+Barak%2C+Teven+Le+Scao%2C+Nouamane+Tazi%2C+Aleksandra+Piktus%2C+Sampo+Pyysalo%2C+Thomas+Wolf%2C+and+Colin+A+Raffel%2E+Scaling+data%2Dconstrained+language+models%2E+Advances+in+N)
71. NVIDIA. Transformer engine, 2023. URL [https://github.com/NVIDIA/TransformerEngine](https://github.com/NVIDIA/TransformerEngine) . [↗](https://github.com/NVIDIA/TransformerEngine)
72. Jupinder Parmar, Sanjev Satheesh, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Reuse, don’t retrain: A recipe for continued pretraining of language models. arXiv preprint arXiv:2407.[07263, 2024](https://md.research.elamur.ai/free/en/papers/parmar-2024-reuse-don-t-retrain-recipe/).
73. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data only. In Thirty-seventh Conference on Neural Inform… [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Hamza+Alobeidli%2C+Alessandro+Cappelli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+The+RefinedWeb+dataset+for+Falcon+)
74. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. [The FineWeb datasets: Decanting the web for the finest text data at scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/). In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benc…
75. Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong. [The devil in linear transformer](https://md.research.elamur.ai/free/en/papers/qin-2022-devil-linear-transformer/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7025–7041, 2022a.
76. Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosFormer: Rethinking softmax in attention. In International Conference on Learning Representations, 2022b. URL [https://openreview.net/forum?id=Bl8CQrx2Up4](https://openreview.net/forum?id=Bl8CQrx2Up4) . [↗](https://openreview.net/forum?id=Bl8CQrx2Up4)
77. Zhen Qin, Xiaodong Han, Weixuan Sun, Bowen He, Dong Li, Dongxu Li, Yuchao Dai, Lingpeng Kong, and Yiran Zhong. Toeplitz neural network for sequence modeling. In The Eleventh International Conference on Learning Representations, 2023a. URL [https://openreview.net/forum?id=IxmWsm4xrua](https://openreview.net/forum?id=IxmWsm4xrua) . [↗](https://openreview.net/forum?id=IxmWsm4xrua)
78. Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for sequence modeling. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 33202–33221, 2023b. [find](https://scholar.google.com/scholar?q=Zhen+Qin%2C+Songlin+Yang%2C+and+Yiran+Zhong%2E+Hierarchically+gated+recurrent+neural+network+for+sequence+modeling%2E+In+Proceedings+of+the+37th+International+Conference+on+Neural+Information+Processing+Syste)
79. Zhen Qin, Yuxin Mao, Xuyang Shen, Dong Li, Jing Zhang, Yuchao Dai, and Yiran Zhong. You only scan once: Efficient multi-dimension sequential modeling with lightnet. arXiv preprint arXiv:2405.21022, 2024a. — [arXiv:2405.21022](https://arxiv.org/abs/2405.21022) [find](https://scholar.google.com/scholar?q=Zhen+Qin%2C+Yuxin+Mao%2C+Xuyang+Shen%2C+Dong+Li%2C+Jing+Zhang%2C+Yuchao+Dai%2C+and+Yiran+Zhong%2E+You+only+scan+once%3A+Efficient+multi%2Ddimension+sequential+modeling+with+lightnet%2E+arXiv+preprint+arXiv%3A2405%2E21022%2C+20)
80. Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. [Lightning attention-2: A free lunch for handling unlimited sequence lengths in large language models](https://md.research.elamur.ai/free/en/papers/qin-2024-lightning-attention-2-free/). arXiv preprint arXiv:2401.04658, 2024b.
81. Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Various lengths, constant speed: Efficient language modeling with lightning attention. In International conference on machine learning, pages 41517–41535. PMLR, 2024c. [find](https://scholar.google.com/scholar?q=Zhen+Qin%2C+Weigao+Sun%2C+Dong+Li%2C+Xuyang+Shen%2C+Weixuan+Sun%2C+and+Yiran+Zhong%2E+Various+lengths%2C+constant+speed%3A+Efficient+language+modeling+with+lightning+attention%2E+In+International+conference+on+machine+)
82. Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. HGRN2: Gated linear rnns with state expansion. arXiv preprint arXiv:2404.07904, 2024d. — [arXiv:2404.07904](https://arxiv.org/abs/2404.07904) [find](https://scholar.google.com/scholar?q=Zhen+Qin%2C+Songlin+Yang%2C+Weixuan+Sun%2C+Xuyang+Shen%2C+Dong+Li%2C+Weigao+Sun%2C+and+Yiran+Zhong%2E+HGRN2%3A+Gated+linear+rnns+with+state+expansion%2E+arXiv+preprint+arXiv%3A2404%2E07904%2C+2024d%2E+%E2%80%94+%5BarXiv%3A2404%2E07904%5D%28http)
83. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training Gopher. arXiv preprint arXiv:2112.11446, 2021. [find](https://scholar.google.com/scholar?q=Jack+W+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+et+al%2E+Scaling+language+models%3A+Methods%2C+analysi)
84. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. [Direct preference optimization: your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 53728–53741, 2023…
85. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020. [find](https://scholar.google.com/scholar?q=Samyam+Rajbhandari%2C+Jeff+Rasley%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+Zero%3A+Memory+optimizations+toward+training+trillion+parameter+models%2E+In+SC20%3A+International+Conference+for+High+Performance+Computing)
86. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. [GPQA: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). In First Conference on Language Modeling, 2024. URL [https://openreview.net/forum?id=Ti67584b98](https://openreview.net/forum?id=Ti67584b98) . [↗](https://openreview.net/forum?id=Ti67584b98)
87. Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. [Samba: Simple hybrid state space models for efficient unlimited context language modeling](https://md.research.elamur.ai/free/en/papers/ren-2024-samba-simple-hybrid-state-space/). arXiv preprint arXiv:2406.07522, 2024.
88. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial Winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+WinoGrande%3A+An+adversarial+Winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2C+2021%2E)
89. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EM… [find](https://scholar.google.com/scholar?q=Maarten+Sap%2C+Hannah+Rashkin%2C+Derek+Chen%2C+Ronan+Le+Bras%2C+and+Yejin+Choi%2E+Social+IQa%3A+Commonsense+reasoning+about+social+interactions%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natur)
90. Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. [LAION-400M: Open dataset of CLIP-filtered 400 million image-text pairs](https://md.research.elamur.ai/free/en/papers/schuhmann-2021-laion-400m-open-dataset-clip/). arXiv preprint arXiv:2111.02114, 2021.
91. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)
92. Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. [SCROLLS: Standardized comparison over long language sequences](https://md.research.elamur.ai/free/en/papers/shaham-2022-scrolls-standardized-comparison-over-long/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 12007–1202…
93. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. [DeepSeekMath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300, 2024.
94. Xuyang Shen, Dong Li, Ruitao Leng, Zhen Qin, Weigao Sun, and Yiran Zhong. Scaling laws for linear complexity language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16377–16426, 2024. [find](https://scholar.google.com/scholar?q=Xuyang+Shen%2C+Dong+Li%2C+Ruitao+Leng%2C+Zhen+Qin%2C+Weigao+Sun%2C+and+Yiran+Zhong%2E+Scaling+laws+for+linear+complexity+language+models%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Lang)
95. Yusuxke Shibata, Takuya Kida, Shuichi Fukamachi, Masayuki Takeda, Ayumi Shinohara, Takeshi Shinohara, and Setsuo Arikawa. Byte pair encoding: A text compression scheme that accelerates pattern matching. Technical Report DOI-TR-161, Department of Informatics, Kyushu University, 1999. [find](https://scholar.google.com/scholar?q=Yusuxke+Shibata%2C+Takuya+Kida%2C+Shuichi+Fukamachi%2C+Masayuki+Takeda%2C+Ayumi+Shinohara%2C+Takeshi+Shinohara%2C+and+Setsuo+Arikawa%2E+Byte+pair+encoding%3A+A+text+compression+scheme+that+accelerates+pattern+matchin)
96. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063, 2024.
97. Weigao Sun, Zhen Qin, Dong Li, Xuyang Shen, Yu Qiao, and Yiran Zhong. Linear attention sequence parallelism. arXiv preprint arXiv:2404.02882, 2024. [find](https://scholar.google.com/scholar?q=Weigao+Sun%2C+Zhen+Qin%2C+Dong+Li%2C+Xuyang+Shen%2C+Yu+Qiao%2C+and+Yiran+Zhong%2E+Linear+attention+sequence+parallelism%2E+arXiv+preprint+arXiv%3A2404%2E02882%2C+2024%2E)
98. Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi. A benchmark for learning to translate a new language from one grammar book. In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=tbVWug9f2h](https://openreview.net/forum?id=tbVWug9f2h) . [↗](https://openreview.net/forum?id=tbVWug9f2h)
99. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.05530, 2024a.
100. Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, et al. [Jamba-1.5: Hybrid Transformer-Mamba models at scale](https://md.research.elamur.ai/free/en/papers/lenz-2024-jamba-1-5-hybrid-transformer/). arXiv preprint arXiv:2408.12570, 2024b.
101. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
102. Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, et al. [Michelangelo: Long context evaluations beyond haystacks via latent structure queries](https://md.research.elamur.ai/free/en/papers/vodrahalli-2024-michelangelo-long-context-evaluations-haystacks/). arXiv preprint arXiv:2409.12640, 2024.
103. Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. DeepNet: Scaling transformers to 1,000 layers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024a. [find](https://scholar.google.com/scholar?q=Hongyu+Wang%2C+Shuming+Ma%2C+Li+Dong%2C+Shaohan+Huang%2C+Dongdong+Zhang%2C+and+Furu+Wei%2E+DeepNet%3A+Scaling+transformers+to+1%2C000+layers%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%2C+2024a%2E)
104. Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, and Lidia S Chao. [Learning deep transformer models for machine translation](https://md.research.elamur.ai/free/en/papers/wang-2019-learning-deep-transformer-models-machine/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1810–1822, 2019.
105. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. [MMLU-pro: A more robust and challenging multi-task language understanding benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/). In T…
106. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824–24837, 2022.
107. Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. [Measuring short-form factuality in large language models](https://md.research.elamur.ai/free/en/papers/wei-2024-measuring-short-form-factuality-large/). arXiv preprint arXiv:2411.04368, 2024.
108. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+et+al%2E+Qwen2%2E5+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E15115%2C+2024%2E)
109. Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. [CoCa: Contrastive captioners are image-text foundation models](https://md.research.elamur.ai/free/en/papers/yu-2022-coca-contrastive-captioners-image-text/). Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL [https://openreview.net/forum?id=Ee277P3AYC](https://openreview.net/forum?id=Ee277P3AYC) . [↗](https://openreview.net/forum?id=Ee277P3AYC)
110. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. [MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Patt…
111. Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. MMMU-Pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813, 2024b. [find](https://scholar.google.com/scholar?q=Xiang+Yue%2C+Tianyu+Zheng%2C+Yuansheng+Ni%2C+Yubo+Wang%2C+Kai+Zhang%2C+Shengbang+Tong%2C+Yuxuan+Sun%2C+Botao+Yu%2C+Ge+Zhang%2C+Huan+Sun%2C+et+al%2E+MMMU%2DPro%3A+A+more+robust+multi%2Ddiscipline+multimodal+understanding+benchmar)
112. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. [Big Bird: Transformers for longer sequences](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/). Advances in neural information processing systems, 33:17283–17297, 2020.
113. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computati)
114. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. [Instruction-following evaluation for large language models](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/). arXiv preprint arXiv:2311.07911, 2023.
115. A. Contributors The contributors to the report are listed in alphabetical order as follows: Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang, Jiayua… [find](https://scholar.google.com/scholar?q=A%2E+Contributors+The+contributors+to+the+report+are+listed+in+alphabetical+order+as+follows%3A+Aonian+Li%2C+Bangwei+Gong%2C+Bo+Yang%2C+Boji+Shan%2C+Chang+Liu%2C+Cheng+Zhu%2C+Chunhao+Zhang%2C+Congchao+Guo%2C+Da+Chen%2C+Don)

*Updated 1 November 2026*