---
title: "Kimi K2: Open Agentic Intelligence — Team 2025"
description: "This technical report addresses two obstacles to building agentic LLMs."
authors: ["Kimi Team", "Yifan Bai", "Yiping Bao", "Y. Charles", "Cheng Chen", "Guanduo Chen", "Haiting Chen", "Huarong Chen", "Jiahao Chen", "Ningxin Chen", "Ruijue Chen", "Yanru Chen", "Yuankun Chen", "Yutian Chen", "Zhuofu Chen", "Jialei Cui", "Hao Ding", "Mengnan Dong", "Angang Du", "Chenzhuang Du", "Dikang Du", "Yulun Du", "Yu Fan", "Yichen Feng", "Kelin Fu", "Bofei Gao", "Chenxiao Gao", "Hongcheng Gao", "Peizhong Gao", "Tong Gao", "Yuyao Ge", "Shangyi Geng", "Qizheng Gu", "Xinran Gu", "Longyu Guan", "Haiqing Guo", "Jianhang Guo", "Xiaoru Hao", "Tianhong He", "Weiran He", "Wenyang He", "Yunjia He", "Chao Hong", "Hao Hu", "Yangyang Hu", "Zhenxing Hu", "Weixiao Huang", "Zhiqi Huang", "Zihao Huang", "Tao Jiang", "Zhejun Jiang", "Xinyi Jin", "Yongsheng Kang", "Guokun Lai", "Cheng Li", "Fang Li", "Haoyang Li", "Ming Li", "Wentao Li", "Yang Li", "Yanhao Li", "Yiwei Li", "Zhaowei Li", "Zheming Li", "Hongzhan Lin", "Xiaohan Lin", "Zongyu Lin", "Chengyin Liu", "Chenyu Liu", "Hongzhang Liu", "Jingyuan Liu", "Junqi Liu", "Liang Liu", "Shaowei Liu", "T.Y. Liu", "Tianwei Liu", "Weizhou Liu", "Yangyang Liu", "Yibo Liu", "Yiping Liu", "Yue Liu", "Zhengying Liu", "Enzhe Lu", "Haoyu Lu", "Lijun Lu", "Yashuo Luo", "Shengling Ma", "Xinyu Ma", "Yingwei Ma", "Shaoguang Mao", "Jie Mei", "Xin Men", "Yibo Miao", "Siyuan Pan", "Yebo Peng", "Ruoyu Qin", "Zeyu Qin", "Bowen Qu", "Zeyu Shang", "Lidong Shi", "Shengyuan Shi", "Feifan Song", "Jianlin Su", "Zhengyuan Su", "Lin Sui", "Xinjie Sun", "Flood Sung", "Yunpeng Tai", "Heyi Tang", "Jiawen Tao", "Qifeng Teng", "Chaoran Tian", "Chensi Wang", "Dinglu Wang", "Feng Wang", "Hailong Wang", "Haiming Wang", "Jianzhou Wang", "Jiaxing Wang", "Jinhong Wang", "Shengjie Wang", "Shuyi Wang", "Si Wang", "Xinyuan Wang", "Yao Wang", "Yejie Wang", "Yiqin Wang", "Yuxin Wang", "Yuzhi Wang", "Zhaoji Wang", "Zhengtao Wang", "Zhengtao Wang", "Zhexu Wang", "Chu Wei", "Qianqian Wei", "Haoning Wu", "Wenhao Wu", "Xingzhe Wu", "Yuxin Wu", "Chenjun Xiao", "Jin Xie", "Xiaotong Xie", "Weimin Xiong", "Boyu Xu", "Jinjing Xu", "L.H. Xu", "Lin Xu", "Suting Xu", "Weixin Xu", "Xinran Xu", "Yangchuan Xu", "Ziyao Xu", "Jing Xu", "Jing Xu", "Junjie Yan", "Yuzi Yan", "Hao Yang", "Xiaofei Yang", "Yi Yang", "Ying Yang", "Zhen Yang", "Zhilin Yang", "Zonghan Yang", "Haotian Yao", "Xingcheng Yao", "Wenjie Ye", "Zhuorui Ye", "Bohong Yin", "Longhui Yu", "Enming Yuan", "Hongbang Yuan", "Mengjie Yuan", "Siyu Yuan", "Haobing Zhan", "Dehao Zhang", "Hao Zhang", "Wanlu Zhang", "Xiaobin Zhang", "Yadong Zhang", "Yangkun Zhang", "Yichi Zhang", "Yizhi Zhang", "Yongting Zhang", "Yu Zhang", "Yutao Zhang", "Yutong Zhang", "Zheng Zhang", "Haotian Zhao", "Yikai Zhao", "Zijia Zhao", "Huabin Zheng", "Shaojie Zheng", "Longguang Zhong", "Jianren Zhou", "Xinyu Zhou", "Zaida Zhou", "Jinguo Zhu", "Zhen Zhu", "Weiyu Zhuang", "Xinxing Zu"]
year: 2025
venue: "arXiv:2507.20534"
source: "https://arxiv.org/abs/2507.20534"
doi: "10.48550/arxiv.2507.20534"
arxiv: "2507.20534"
licence: "CC BY-NC-ND 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-nd/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kimi-2025-team-kimi-k2-open-agentic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kimi-2025-team-kimi-k2-open-agentic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/) / Kimi K2: Open Agentic Intelligence

# Kimi K2: Open Agentic Intelligence

**Authors:** [Kimi Team](https://md.research.elamur.ai/free/en/authors/kimi-team/), Yifan Bai, Yiping Bao, Y. Charles, [Cheng Chen](https://md.research.elamur.ai/free/en/authors/cheng-chen/), Guanduo Chen, Haiting Chen, Huarong Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen et al. (200 authors) · **Year:** 2025 · **Venue:** arXiv:2507.20534 · **Licence:** [CC BY-NC-ND 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-nd/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2507.20534) · [PDF](https://arxiv.org/pdf/2507.20534) · [DOI 10.48550/arxiv.2507.20534](https://doi.org/10.48550/arxiv.2507.20534) · [arXiv 2507.20534](https://arxiv.org/abs/2507.20534) · [OpenAlex](https://openalex.org/W7077044503)

**Citations:** 15 citations (openalex-snapshot · 2026-10-14) · [315](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
This technical report addresses two obstacles to building agentic LLMs. High-quality pre-training data is scarce, so the learning signal extracted per token matters more. Examples of multi-step planning and tool use are also rare in natural text. The Kimi Team's answer is Kimi K2, a sparse Mixture-of-Experts model with 1.04T total and 32B active parameters, built on MLA in a design similar to DeepSeek-V3. For pre-training on 15.5T tokens, the team created MuonClip. It extends the Muon optimizer with QK-Clip, which shrinks a head's query and key projection weights after an update whenever that head's largest attention logit exceeds a threshold. This prevents the logit blow-ups that destabilised plain Muon, and the full run had no loss spikes. Post-training then adds a pipeline that generates verified tool-use trajectories in simulated and real environments. It also includes an RL stage that combines verifiable rewards with a self-critique rubric reward. Without extended reasoning, K2 leads open non-thinking models on agentic and coding benchmarks and nears the Claude 4 models. Releasing the checkpoints openly gives the research community a strong base for agent work.

## Key points
- **Architecture and training scale:** an ultra-sparse MoE transformer with 1.04T parameters (32B activated), trained on 15.5T tokens with the MuonClip optimizer.
- **QK-Clip behaviour:**
  - In a mid-scale 9B-active/53B-total model trained with vanilla Muon, max attention logits passed 1000.
  - In K2 with τ = 100, logits stayed at the cap and then settled into a normal range after roughly 30% of training.
- **Benchmark results (non-thinking):**
  - Agentic and software engineering: Tau2-Bench 66.1, ACEBench (En) 76.5, SWE-Bench Verified 65.8, SWE-Bench Multilingual 47.3.
  - Coding, maths and STEM: LiveCodeBench v6 53.7, OJBench 27.1, AIME 2025 49.5, GPQA-Diamond 75.1.
  - LMSYS Arena (17 July 2025): top open-source model and 5th overall.
- **Safety evaluation:**
  - Red-teaming used Promptfoo, comparing K2 with DeepSeek-V3, DeepSeek-R1 and Qwen3-235B-A22B.
  - Base64 attacks had little effect, with pass rates near 100%. Crescendo attacks were the most effective.
- **Known limitations and release:**
  - On hard problems or with vague tool definitions, the model can produce overly long outputs, leading to truncation or incomplete tool calls.
  - Enabling tools unnecessarily can hurt performance on some tasks, and one-shot prompting is weaker than an agentic coding framework for building complete software projects.
  - Base and instruct checkpoints are public (moonshotai/Kimi-K2-Instruct on Hugging Face).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

315 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 312 not yet.

- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://md.research.elamur.ai/free/en/papers/deepseek-2026-ai-deepseek-v4-highly-efficient/)
- [GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models](https://md.research.elamur.ai/free/en/papers/glm-2025-4-glm-4-5-agentic/)
- [Beyond Simple Term Injection: Reasoning Models for Legal Translation in a Non-Dominant Language Variety](https://md.research.elamur.ai/free/en/papers/natale-2026-beyond-simple-term-injection-reasoning/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{team2025kimi,
  title = {{Kimi K2: Open Agentic Intelligence}},
  author = {Team, Kimi and Bai, Yifan and Bao, Yiping and Charles, Y. and Chen, Cheng and Chen, Guanduo and Chen, Haiting and Chen, Huarong and Chen, Jiahao and Chen, Ningxin and Chen, Ruijue and Chen, Yanru and Chen, Yuankun and Chen, Yutian and Chen, Zhuofu and Cui, Jialei and Ding, Hao and Dong, Mengnan and Du, Angang and Du, Chenzhuang and Du, Dikang and Du, Yulun and Fan, Yu and Feng, Yichen and Fu, Kelin and Gao, Bofei and Gao, Chenxiao and Gao, Hongcheng and Gao, Peizhong and Gao, Tong and Ge, Yuyao and Geng, Shangyi and Gu, Qizheng and Gu, Xinran and Guan, Longyu and Guo, Haiqing and Guo, Jianhang and Hao, Xiaoru and He, Tianhong and He, Weiran and He, Wenyang and He, Yunjia and Hong, Chao and Hu, Hao and Hu, Yangyang and Hu, Zhenxing and Huang, Weixiao and Huang, Zhiqi and Huang, Zihao and Jiang, Tao and Jiang, Zhejun and Jin, Xinyi and Kang, Yongsheng and Lai, Guokun and Li, Cheng and Li, Fang and Li, Haoyang and Li, Ming and Li, Wentao and Li, Yang and Li, Yanhao and Li, Yiwei and Li, Zhaowei and Li, Zheming and Lin, Hongzhan and Lin, Xiaohan and Lin, Zongyu and Liu, Chengyin and Liu, Chenyu and Liu, Hongzhang and Liu, Jingyuan and Liu, Junqi and Liu, Liang and Liu, Shaowei and Liu, T.Y. and Liu, Tianwei and Liu, Weizhou and Liu, Yangyang and Liu, Yibo and Liu, Yiping and Liu, Yue and Liu, Zhengying and Lu, Enzhe and Lu, Haoyu and Lu, Lijun and Luo, Yashuo and Ma, Shengling and Ma, Xinyu and Ma, Yingwei and Mao, Shaoguang and Mei, Jie and Men, Xin and Miao, Yibo and Pan, Siyuan and Peng, Yebo and Qin, Ruoyu and Qin, Zeyu and Qu, Bowen and Shang, Zeyu and Shi, Lidong and Shi, Shengyuan and Song, Feifan and Su, Jianlin and Su, Zhengyuan and Sui, Lin and Sun, Xinjie and Sung, Flood and Tai, Yunpeng and Tang, Heyi and Tao, Jiawen and Teng, Qifeng and Tian, Chaoran and Wang, Chensi and Wang, Dinglu and Wang, Feng and Wang, Hailong and Wang, Haiming and Wang, Jianzhou and Wang, Jiaxing and Wang, Jinhong and Wang, Shengjie and Wang, Shuyi and Wang, Si and Wang, Xinyuan and Wang, Yao and Wang, Yejie and Wang, Yiqin and Wang, Yuxin and Wang, Yuzhi and Wang, Zhaoji and Wang, Zhengtao and Wang, Zhengtao and Wang, Zhexu and Wei, Chu and Wei, Qianqian and Wu, Haoning and Wu, Wenhao and Wu, Xingzhe and Wu, Yuxin and Xiao, Chenjun and Xie, Jin and Xie, Xiaotong and Xiong, Weimin and Xu, Boyu and Xu, Jinjing and Xu, L.H. and Xu, Lin and Xu, Suting and Xu, Weixin and Xu, Xinran and Xu, Yangchuan and Xu, Ziyao and Xu, Jing and Xu, Jing and Yan, Junjie and Yan, Yuzi and Yang, Hao and Yang, Xiaofei and Yang, Yi and Yang, Ying and Yang, Zhen and Yang, Zhilin and Yang, Zonghan and Yao, Haotian and Yao, Xingcheng and Ye, Wenjie and Ye, Zhuorui and Yin, Bohong and Yu, Longhui and Yuan, Enming and Yuan, Hongbang and Yuan, Mengjie and Yuan, Siyu and Zhan, Haobing and Zhang, Dehao and Zhang, Hao and Zhang, Wanlu and Zhang, Xiaobin and Zhang, Yadong and Zhang, Yangkun and Zhang, Yichi and Zhang, Yizhi and Zhang, Yongting and Zhang, Yu and Zhang, Yutao and Zhang, Yutong and Zhang, Zheng and Zhao, Haotian and Zhao, Yikai and Zhao, Zijia and Zheng, Huabin and Zheng, Shaojie and Zhong, Longguang and Zhou, Jianren and Zhou, Xinyu and Zhou, Zaida and Zhu, Jinguo and Zhu, Zhen and Zhuang, Weiyu and Zu, Xinxing},
  year = {2025},
  doi = {10.48550/arxiv.2507.20534},
  eprint = {2507.20534},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2507.20534}
}
```

- [← previous Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://md.research.elamur.ai/free/en/papers/kimi-2025-team-kimi-k1-5-scaling-reinforcement/)
- [next → L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/aggarwal-2025-l1-controlling-long-reasoning-model/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/)

## References

92 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Austin et al. Program Synthesis with Large Language Models. 2021. arXiv: 2108.07732 [cs.PL]. URL : https://arxiv.org/abs/2108.07732. [↗](https://arxiv.org/abs/2108.07732) [our card](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/)
2. Yushi Bai et al. LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks. 2025. arXiv: 2412.15204 [cs.CL]. URL : https://arxiv.org/abs/2412.15204. [↗](https://arxiv.org/abs/2412.15204) [our card](https://md.research.elamur.ai/free/en/papers/bai-2024-longbench-v2-deeper-understanding/)
3. Victor Barres et al. τ 2 -Bench: Evaluating Conversational Agents in a Dual-Control Environment. 2025. arXiv: 2506.07982 [cs.AI]. URL : https://arxiv.org/abs/2506.07982. [↗](https://arxiv.org/abs/2506.07982)
4. Stella Biderman et al. “Lessons from the trenches on reproducible evaluation of language models”. In: arXiv preprint arXiv:2405.14782 (2024). [find](https://scholar.google.com/scholar?q=Stella+Biderman+et+al%2E+%E2%80%9CLessons+from+the+trenches+on+reproducible+evaluation+of+language+models%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2405%2E14782+%282024%29%2E)
5. Greg Brockman et al. OpenAI Gym. 2016. arXiv: 1606.01540 [cs.LG]. URL : https://arxiv.org/abs/1606.01540. [↗](https://arxiv.org/abs/1606.01540)
6. Federico Cassano et al. “MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation”. In: IEEE Transactions on Software Engineering 49.7 (2023), pp. 3675–3691. DOI : 10.1109/TSE.2023.3267446. — [doi:10.1109/tse.2023.3267446](https://doi.org/10.1109/tse.2023.3267446) [find](https://scholar.google.com/scholar?q=Federico+Cassano+et+al%2E+%E2%80%9CMultiPL%2DE%3A+A+Scalable+and+Polyglot+Approach+to+Benchmarking+Neural+Code+Generation%E2%80%9D%2E+In%3A+IEEE+Transactions+on+Software+Engineering+49%2E7+%282023%29%2C+pp%2E+3675%E2%80%933691%2E+DOI+%3A+10%2E1109%2FTS)
7. Chen Chen et al. “ACEBench: Who Wins the Match Point in Tool Learning?” In: arXiv e-prints (2025), arXiv– 2501. [find](https://scholar.google.com/scholar?q=Chen+Chen+et+al%2E+%E2%80%9CACEBench%3A+Who+Wins+the+Match+Point+in+Tool+Learning%3F%E2%80%9D+In%3A+arXiv+e%2Dprints+%282025%29%2C+arXiv%E2%80%93+2501%2E)
8. Mark Chen et al. “Evaluating Large Language Models Trained on Code”. In: (2021). arXiv: 2107.03374 [cs.LG]. [find](https://scholar.google.com/scholar?q=Mark+Chen+et+al%2E+%E2%80%9CEvaluating+Large+Language+Models+Trained+on+Code%E2%80%9D%2E+In%3A+%282021%29%2E+arXiv%3A+2107%2E03374+%5Bcs%2ELG%5D%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
9. Peter Clark et al. “Think you have solved question answering? try arc, the ai2 reasoning challenge”. In: arXiv preprint arXiv:1803.05457 (2018). [find](https://scholar.google.com/scholar?q=Peter+Clark+et+al%2E+%E2%80%9CThink+you+have+solved+question+answering%3F+try+arc%2C+the+ai2+reasoning+challenge%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A1803%2E05457+%282018%29%2E)
10. Karl Cobbe et al. Training Verifiers to Solve Math Word Problems. 2021. arXiv: 2110.14168 [cs.LG]. URL : https://arxiv.org/abs/2110.14168. [↗](https://arxiv.org/abs/2110.14168) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
11. DeepSeek-AI. DeepSeek-V3 Technical Report. 2024. arXiv: 2412 . 19437 [cs.CL]. URL : https://arxiv.org/abs/2412.19437. [↗](https://arxiv.org/abs/2412.19437)
12. Mostafa Dehghani et al. “Scaling vision transformers to 22 billion parameters”. In: International conference on machine learning. PMLR. 2023, pp. 7480–7512. [find](https://scholar.google.com/scholar?q=Mostafa+Dehghani+et+al%2E+%E2%80%9CScaling+vision+transformers+to+22+billion+parameters%E2%80%9D%2E+In%3A+International+conference+on+machine+learning%2E+PMLR%2E+2023%2C+pp%2E+7480%E2%80%937512%2E)
13. Guanting Dong et al. Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models. 2024. arXiv: 2406 . 13542 [cs.CL]. URL : https://arxiv.org/abs/2406 . 13542. [↗](https://arxiv.org/abs/2406)
14. Xinrun Du et al. “Supergpqa: Scaling llm evaluation across 285 graduate disciplines”. In: arXiv preprint arXiv:2502.14739 (2025). [find](https://scholar.google.com/scholar?q=Xinrun+Du+et+al%2E+%E2%80%9CSupergpqa%3A+Scaling+llm+evaluation+across+285+graduate+disciplines%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2502%2E14739+%282025%29%2E)
15. Dheeru Dua et al. “DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs”. In: CoRR abs/1903.00161 (2019). arXiv: 1903.00161. URL : http://arxiv.org/abs/1903.00161. [↗](http://arxiv.org/abs/1903.00161) [our card](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)
16. Kazuki Fujii et al. Rewriting Pre-Training Data Boosts LLM Performance in Math and Code. 2025. arXiv: 2505.02881 [cs.LG]. URL : https://arxiv.org/abs/2505.02881. [↗](https://arxiv.org/abs/2505.02881)
17. Paul Gauthier. Aider LLM Leaderboards. [https://aider.chat/docs/leaderboards/](https://aider.chat/docs/leaderboards/). 2025. [↗](https://aider.chat/docs/leaderboards/)
18. Aryo Pradipta Gema et al. “Are we done with mmlu?” In: arXiv preprint arXiv:2406.04127 (2024). [find](https://scholar.google.com/scholar?q=Aryo+Pradipta+Gema+et+al%2E+%E2%80%9CAre+we+done+with+mmlu%3F%E2%80%9D+In%3A+arXiv+preprint+arXiv%3A2406%2E04127+%282024%29%2E)
19. Alex Gu et al. “Cruxeval: A benchmark for code reasoning, understanding and execution”. In: arXiv preprint arXiv:2401.03065 (2024). [find](https://scholar.google.com/scholar?q=Alex+Gu+et+al%2E+%E2%80%9CCruxeval%3A+A+benchmark+for+code+reasoning%2C+understanding+and+execution%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2401%2E03065+%282024%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/gu-2024-cruxeval-benchmark-code-reasoning-understanding/)
20. Daya Guo et al. “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning”. In: arXiv preprint arXiv:2501.12948 (2025). [find](https://scholar.google.com/scholar?q=Daya+Guo+et+al%2E+%E2%80%9CDeepseek%2Dr1%3A+Incentivizing+reasoning+capability+in+llms+via+reinforcement+learning%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2501%2E12948+%282025%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/deepseek-2025-ai-deepseek-r1-incentivizing-reasoning-capability/)
21. Zhicheng Guo et al. “StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models”. In: arXiv preprint arXiv:2403.07714 (2025). [find](https://scholar.google.com/scholar?q=Zhicheng+Guo+et+al%2E+%E2%80%9CStableToolBench%3A+Towards+Stable+Large%2DScale+Benchmarking+on+Tool+Learning+of+Large+Language+Models%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2403%2E07714+%282025%29%2E)
22. Aaron Harlap et al. “Pipedream: Fast and efficient pipeline parallel dnn training”. In: arXiv preprint arXiv:1806.03377 (2018). [find](https://scholar.google.com/scholar?q=Aaron+Harlap+et+al%2E+%E2%80%9CPipedream%3A+Fast+and+efficient+pipeline+parallel+dnn+training%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A1806%2E03377+%282018%29%2E)
23. Y He et al. “Chinese simpleqa: A chinese factuality evaluation for large language models, 2024a”. In: URL https://arxiv.org/abs/2411.07140 (). [↗](https://arxiv.org/abs/2411.07140)
24. Dan Hendrycks et al. “Measuring massive multitask language understanding”. In: arXiv preprint arXiv:2009.03300 (2020). [find](https://scholar.google.com/scholar?q=Dan+Hendrycks+et+al%2E+%E2%80%9CMeasuring+massive+multitask+language+understanding%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2009%2E03300+%282020%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
25. Dan Hendrycks et al. Measuring Mathematical Problem Solving With the MATH Dataset. 2021. arXiv: 2103. 03874 [cs.LG]. URL : https://arxiv.org/abs/2103.03874. [↗](https://arxiv.org/abs/2103.03874) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/)
26. Shengding Hu et al. “Minicpm: Unveiling the potential of small language models with scalable training strategies”. In: arXiv preprint arXiv:2404.06395 (2024). [find](https://scholar.google.com/scholar?q=Shengding+Hu+et+al%2E+%E2%80%9CMinicpm%3A+Unveiling+the+potential+of+small+language+models+with+scalable+training+strategies%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2404%2E06395+%282024%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/hu-2024-minicpm-unveiling-potential-small-language/)
27. Jiaxin Huang et al. “Large language models can self-improve”. In: arXiv preprint arXiv:2210.11610 (2022). [find](https://scholar.google.com/scholar?q=Jiaxin+Huang+et+al%2E+%E2%80%9CLarge+language+models+can+self%2Dimprove%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2210%2E11610+%282022%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/)
28. Siming Huang et al. OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models. 2025. arXiv: 2411.04905 [cs.CL]. URL : https://arxiv.org/abs/2411.04905. [↗](https://arxiv.org/abs/2411.04905)
29. Yanping Huang et al. “Gpipe: Efficient training of giant neural networks using pipeline parallelism”. In: Advances in neural information processing systems 32 (2019). [find](https://scholar.google.com/scholar?q=Yanping+Huang+et+al%2E+%E2%80%9CGpipe%3A+Efficient+training+of+giant+neural+networks+using+pipeline+parallelism%E2%80%9D%2E+In%3A+Advances+in+neural+information+processing+systems+32+%282019%29%2E)
30. Yuzhen Huang et al. C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models. 2023. arXiv: 2305.08322 [cs.CL]. URL : https://arxiv.org/abs/2305.08322. [↗](https://arxiv.org/abs/2305.08322) [our card](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/)
31. Alon Jacovi et al. The FACTS Grounding Leaderboard: Benchmarking LLMs’ Ability to Ground Responses to Long-Form Input. 2025. arXiv: 2501.03200 [cs.CL]. URL : https://arxiv.org/abs/2501.03200. [↗](https://arxiv.org/abs/2501.03200)
32. Naman Jain et al. “Livecodebench: Holistic and contamination free evaluation of large language models for code”. In: arXiv preprint arXiv:2403.07974 (2024). [find](https://scholar.google.com/scholar?q=Naman+Jain+et+al%2E+%E2%80%9CLivecodebench%3A+Holistic+and+contamination+free+evaluation+of+large+language+models+for+code%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2403%2E07974+%282024%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/)
33. Carlos E Jimenez et al. “SWE-bench: Can Language Models Resolve Real-world Github Issues?” In: The Twelfth International Conference on Learning Representations. 2024. URL : [https://openreview.net/forum?id=VTF8yNQM66](https://openreview.net/forum?id=VTF8yNQM66). [↗](https://openreview.net/forum?id=VTF8yNQM66) [our card](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/)
34. Keller Jordan et al. Muon: An optimizer for hidden layers in neural networks. 2024. URL : [https://kellerjordan.github.io/posts/muon/](https://kellerjordan.github.io/posts/muon/). [↗](https://kellerjordan.github.io/posts/muon/)
35. Mandar Joshi et al. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. 2017. arXiv: 1705.03551 [cs.CL]. URL : https://arxiv.org/abs/1705.03551. [↗](https://arxiv.org/abs/1705.03551) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
36. Kimi Team. “Kimi k1. 5: Scaling reinforcement learning with llms”. In: arXiv preprint arXiv:2501.12599 (2025). [find](https://scholar.google.com/scholar?q=Kimi+Team%2E+%E2%80%9CKimi+k1%2E+5%3A+Scaling+reinforcement+learning+with+llms%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2501%2E12599+%282025%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/kimi-2025-team-kimi-k1-5-scaling-reinforcement/)
37. Diederik P. Kingma and Jimmy Ba. “Adam: A Method for Stochastic Optimization”. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. Ed. by Yoshua Bengio and Yann LeCun. 2015. URL : http://arxiv.org/abs/1412.6980. [↗](http://arxiv.org/abs/1412.6980)
38. Satyapriya Krishna et al. Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation. 2025. arXiv: 2409.12941 [cs.CL]. URL : https://arxiv.org/abs/2409.12941. [↗](https://arxiv.org/abs/2409.12941)
39. Joel Lamy-Poirier. “Breadth-first pipeline parallelism”. In: Proceedings of Machine Learning and Systems 5 (2023), pp. 48–67. [find](https://scholar.google.com/scholar?q=Joel+Lamy%2DPoirier%2E+%E2%80%9CBreadth%2Dfirst+pipeline+parallelism%E2%80%9D%2E+In%3A+Proceedings+of+Machine+Learning+and+Systems+5+%282023%29%2C+pp%2E+48%E2%80%9367%2E)
40. Dmitry Lepikhin et al. “Gshard: Scaling giant models with conditional computation and automatic sharding”. In: arXiv preprint arXiv:2006.16668 (2020). [find](https://scholar.google.com/scholar?q=Dmitry+Lepikhin+et+al%2E+%E2%80%9CGshard%3A+Scaling+giant+models+with+conditional+computation+and+automatic+sharding%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2006%2E16668+%282020%29%2E)
41. Haonan Li et al. CMMLU: Measuring massive multitask language understanding in Chinese. 2024. arXiv: 2306.09212 [cs.CL]. URL : https://arxiv.org/abs/2306.09212. [↗](https://arxiv.org/abs/2306.09212) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-cmmlu-measuring-massive-multitask-language/)
42. Jia Li et al. “Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions”. In: Hugging Face repository 13.9 (2024), p. 9. [find](https://scholar.google.com/scholar?q=Jia+Li+et+al%2E+%E2%80%9CNuminamath%3A+The+largest+public+dataset+in+ai4maths+with+860k+pairs+of+competition+math+problems+and+solutions%E2%80%9D%2E+In%3A+Hugging+Face+repository+13%2E9+%282024%29%2C+p%2E+9%2E)
43. Tianle Li et al. “From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline”. In: arXiv preprint arXiv:2406.11939 (2024). [find](https://scholar.google.com/scholar?q=Tianle+Li+et+al%2E+%E2%80%9CFrom+Crowdsourced+Data+to+High%2DQuality+Benchmarks%3A+Arena%2DHard+and+BenchBuilder+Pipeline%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2406%2E11939+%282024%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2024-crowdsourced-data-high-quality-benchmarks/)
44. Bill Yuchen Lin et al. ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning. 2025. arXiv: 2502. 01100 [cs.AI]. URL : https://arxiv.org/abs/2502.01100. [↗](https://arxiv.org/abs/2502.01100)
45. Aixin Liu et al. “Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model”. In: arXiv preprint arXiv:2405.04434 (2024). [find](https://scholar.google.com/scholar?q=Aixin+Liu+et+al%2E+%E2%80%9CDeepseek%2Dv2%3A+A+strong%2C+economical%2C+and+efficient+mixture%2Dof%2Dexperts+language+model%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2405%2E04434+%282024%29%2E)
46. Jiawei Liu et al. “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation”. In: Advances in Neural Information Processing Systems 36 (2023), pp. 21558–21572. [find](https://scholar.google.com/scholar?q=Jiawei+Liu+et+al%2E+%E2%80%9CIs+your+code+generated+by+chatgpt+really+correct%3F+rigorous+evaluation+of+large+language+models+for+code+generation%E2%80%9D%2E+In%3A+Advances+in+Neural+Information+Processing+Systems+36+%282023%29%2C) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/)
47. Jingyuan Liu et al. “Muon is scalable for LLM training”. In: arXiv preprint arXiv:2502.16982 (2025). [find](https://scholar.google.com/scholar?q=Jingyuan+Liu+et+al%2E+%E2%80%9CMuon+is+scalable+for+LLM+training%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2502%2E16982+%282025%29%2E)
48. Ziming Liu et al. “Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiency”. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. SC ’23. ACM, Nov. 2023, pp. 1–13. DOI : 10 . 1145 / 3581784 . 3607073.… [find](https://scholar.google.com/scholar?q=Ziming+Liu+et+al%2E+%E2%80%9CHanayo%3A+Harnessing+Wave%2Dlike+Pipeline+Parallelism+for+Enhanced+Large+Model+Training+Efficiency%E2%80%9D%2E+In%3A+Proceedings+of+the+International+Conference+for+High+Performance+Computing%2C+Netw)
49. Ilya Loshchilov and Frank Hutter. “Decoupled Weight Decay Regularization”. In: International Conference on Learning Representations. 2019. URL : [https://openreview.net/forum?id=Bkg6RiCqY7](https://openreview.net/forum?id=Bkg6RiCqY7). [↗](https://openreview.net/forum?id=Bkg6RiCqY7)
50. Pratyush Maini et al. Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling. 2024. arXiv: 2401.16380 [cs.CL]. URL : https://arxiv.org/abs/2401.16380. [↗](https://arxiv.org/abs/2401.16380)
51. Samuel Miserendino et al. “SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?” In: arXiv preprint arXiv:2502.12115 (2025). [find](https://scholar.google.com/scholar?q=Samuel+Miserendino+et+al%2E+%E2%80%9CSWE%2DLancer%3A+Can+Frontier+LLMs+Earn+%241+Million+from+Real%2DWorld+Freelance+Software+Engineering%3F%E2%80%9D+In%3A+arXiv+preprint+arXiv%3A2502%2E12115+%282025%29%2E)
52. Arindam Mitra et al. “Agentinstruct: Toward generative teaching with agentic flows”. In: arXiv preprint arXiv:2407.03502 (2024). [find](https://scholar.google.com/scholar?q=Arindam+Mitra+et+al%2E+%E2%80%9CAgentinstruct%3A+Toward+generative+teaching+with+agentic+flows%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2407%2E03502+%282024%29%2E)
53. Ivan Moshkov et al. “Aimo-2 winning solution: Building state-of-the-art mathematical reasoning models with openmathreasoning dataset”. In: arXiv preprint arXiv:2504.16891 (2025). [find](https://scholar.google.com/scholar?q=Ivan+Moshkov+et+al%2E+%E2%80%9CAimo%2D2+winning+solution%3A+Building+state%2Dof%2Dthe%2Dart+mathematical+reasoning+models+with+openmathreasoning+dataset%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2504%2E16891+%282025%29%2E)
54. Deepak Narayanan et al. “Efficient large-scale language model training on gpu clusters using megatron-lm”. In: Proceedings of the international conference for high performance computing, networking, storage and analysis. 2021, pp. 1–15. [find](https://scholar.google.com/scholar?q=Deepak+Narayanan+et+al%2E+%E2%80%9CEfficient+large%2Dscale+language+model+training+on+gpu+clusters+using+megatron%2Dlm%E2%80%9D%2E+In%3A+Proceedings+of+the+international+conference+for+high+performance+computing%2C+networking%2C+s)
55. Long Ouyang et al. “Training language models to follow instructions with human feedback”. In: Advances in neural information processing systems 35 (2022), pp. 27730–27744. [find](https://scholar.google.com/scholar?q=Long+Ouyang+et+al%2E+%E2%80%9CTraining+language+models+to+follow+instructions+with+human+feedback%E2%80%9D%2E+In%3A+Advances+in+neural+information+processing+systems+35+%282022%29%2C+pp%2E+27730%E2%80%9327744%2E)
56. Bowen Peng et al. “Yarn: Efficient context window extension of large language models”. In: arXiv preprint arXiv:2309.00071 (2023). [find](https://scholar.google.com/scholar?q=Bowen+Peng+et+al%2E+%E2%80%9CYarn%3A+Efficient+context+window+extension+of+large+language+models%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2309%2E00071+%282023%29%2E)
57. Long Phan et al. Humanity’s Last Exam. 2025. arXiv: 2501.14249 [cs.LG]. URL : https://arxiv.org/abs/2501.14249. [↗](https://arxiv.org/abs/2501.14249) [our card](https://md.research.elamur.ai/free/en/papers/phan-2025-humanity-s-last-exam/)
58. Penghui Qi et al. “Zero bubble pipeline parallelism”. In: arXiv preprint arXiv:2401.10241 (2023). [find](https://scholar.google.com/scholar?q=Penghui+Qi+et+al%2E+%E2%80%9CZero+bubble+pipeline+parallelism%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2401%2E10241+%282023%29%2E)
59. Yujia Qin et al. “Toolllm: Facilitating large language models to master 16000+ real-world apis”. In: arXiv preprint arXiv:2307.16789 (2023). [find](https://scholar.google.com/scholar?q=Yujia+Qin+et+al%2E+%E2%80%9CToolllm%3A+Facilitating+large+language+models+to+master+16000%2B+real%2Dworld+apis%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2307%2E16789+%282023%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/qin-2023-toolllm-facilitating-large-language-models/)
60. Qwen et al. Qwen2.5 Technical Report. 2025. arXiv: 2412.15115 [cs.CL]. URL : https://arxiv.org/abs/2412.15115. [↗](https://arxiv.org/abs/2412.15115)
61. Samyam Rajbhandari et al. “Zero: Memory optimizations toward training trillion parameter models”. In: SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE. 2020, pp. 1–16. [find](https://scholar.google.com/scholar?q=Samyam+Rajbhandari+et+al%2E+%E2%80%9CZero%3A+Memory+optimizations+toward+training+trillion+parameter+models%E2%80%9D%2E+In%3A+SC20%3A+International+Conference+for+High+Performance+Computing%2C+Networking%2C+Storage+and+Analysis%2E+I)
62. David Rein et al. “Gpqa: A graduate-level google-proof q&a benchmark”. In: First Conference on Language Modeling. 2024. [find](https://scholar.google.com/scholar?q=David+Rein+et+al%2E+%E2%80%9CGpqa%3A+A+graduate%2Dlevel+google%2Dproof+q%26a+benchmark%E2%80%9D%2E+In%3A+First+Conference+on+Language+Modeling%2E+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/)
63. Keisuke Sakaguchi et al. “Winogrande: An adversarial winograd schema challenge at scale”. In: Communications of the ACM 64.9 (2021), pp. 99–106. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi+et+al%2E+%E2%80%9CWinogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%E2%80%9D%2E+In%3A+Communications+of+the+ACM+64%2E9+%282021%29%2C+pp%2E+99%E2%80%93106%2E)
64. David Silver and Richard S Sutton. “Welcome to the era of experience”. In: Google AI 1 (2025). [find](https://scholar.google.com/scholar?q=David+Silver+and+Richard+S+Sutton%2E+%E2%80%9CWelcome+to+the+era+of+experience%E2%80%9D%2E+In%3A+Google+AI+1+%282025%29%2E)
65. Ved Sirdeshmukh et al. MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs. 2025. arXiv: 2501 . 17399 [cs.CL]. URL : https://arxiv.org/abs/2501 . 17399. [↗](https://arxiv.org/abs/2501)
66. Giulio Starace et al. “PaperBench: Evaluating AI’s Ability to Replicate AI Research”. In: arXiv preprint arXiv:2504.01848 (2025). [find](https://scholar.google.com/scholar?q=Giulio+Starace+et+al%2E+%E2%80%9CPaperBench%3A+Evaluating+AI%E2%80%99s+Ability+to+Replicate+AI+Research%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2504%2E01848+%282025%29%2E)
67. Hao Sun et al. ZeroSearch: Incentivize the Search Capability of LLMs without Searching. 2025. arXiv: 2505. 04588 [cs.CL]. URL : https://arxiv.org/abs/2505.04588. [↗](https://arxiv.org/abs/2505.04588)
68. Mirac Suzgun et al. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. 2022. arXiv: 2210.09261 [cs.CL]. URL : https://arxiv.org/abs/2210.09261. [↗](https://arxiv.org/abs/2210.09261) [our card](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/)
69. Manveer Singh Tamber et al. “Benchmarking LLM Faithfulness in RAG with Evolving Leaderboards”. In: arXiv preprint arXiv:2505.04847 (2025). [find](https://scholar.google.com/scholar?q=Manveer+Singh+Tamber+et+al%2E+%E2%80%9CBenchmarking+LLM+Faithfulness+in+RAG+with+Evolving+Leaderboards%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2505%2E04847+%282025%29%2E)
70. Gemma Team et al. “Gemma 2: Improving open language models at a practical size”. In: arXiv preprint arXiv:2408.00118 (2024). [find](https://scholar.google.com/scholar?q=Gemma+Team+et+al%2E+%E2%80%9CGemma+2%3A+Improving+open+language+models+at+a+practical+size%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2408%2E00118+%282024%29%2E)
71. LlaMA Team. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation — ai.meta.com. [https://ai.meta.com/blog/llama-4-multimodal-intelligence/](https://ai.meta.com/blog/llama-4-multimodal-intelligence/). [Accessed 15-07-2025]. [↗](https://ai.meta.com/blog/llama-4-multimodal-intelligence/)
72. The Terminal-Bench Team. Terminal-Bench: A Benchmark for AI Agents in Terminal Environments. Apr. 2025. URL : [https://github.com/laude-institute/terminal-bench](https://github.com/laude-institute/terminal-bench). [↗](https://github.com/laude-institute/terminal-bench)
73. Ashish Vaswani et al. “Attention is All you Need”. In: Advances in Neural Information Processing Systems. Ed. by I. Guyon et al. Vol. 30. Curran Associates, Inc., 2017. URL : [https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf). [↗](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)
74. Vectara. Hallucination Evaluation Model (Revision 7437011). 2024. URL : [https://huggingface.co/vectara/hallucination_evaluation_model](https://huggingface.co/vectara/hallucination_evaluation_model). [↗](https://huggingface.co/vectara/hallucination_evaluation_model)
75. Joshua Vendrow et al. “Do large language model benchmarks test reliability?” In: arXiv preprint arXiv:2502.03461 (2025). [find](https://scholar.google.com/scholar?q=Joshua+Vendrow+et+al%2E+%E2%80%9CDo+large+language+model+benchmarks+test+reliability%3F%E2%80%9D+In%3A+arXiv+preprint+arXiv%3A2502%2E03461+%282025%29%2E)
76. Yizhong Wang et al. “Self-instruct: Aligning language models with self-generated instructions”. In: arXiv preprint arXiv:2212.10560 (2022). [find](https://scholar.google.com/scholar?q=Yizhong+Wang+et+al%2E+%E2%80%9CSelf%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2212%2E10560+%282022%29%2E)
77. Yubo Wang et al. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. 2024. arXiv: 2406.01574 [cs.CL]. URL : https://arxiv.org/abs/2406.01574. [↗](https://arxiv.org/abs/2406.01574) [our card](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/)
78. Zhexu Wang et al. OJBench: A Competition Level Code Benchmark For Large Language Models. 2025. arXiv: 2506.16395 [cs.CL]. URL : https://arxiv.org/abs/2506.16395. — [arXiv:2506.16395](https://arxiv.org/abs/2506.16395) [↗](https://arxiv.org/abs/2506.16395)
79. Jason Wei et al. “Measuring short-form factuality in large language models”. In: arXiv preprint arXiv:2411.04368 (2024). [find](https://scholar.google.com/scholar?q=Jason+Wei+et+al%2E+%E2%80%9CMeasuring+short%2Dform+factuality+in+large+language+models%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2411%2E04368+%282024%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/wei-2024-measuring-short-form-factuality-large/)
80. Tianwen Wei et al. CMATH: Can Your Language Model Pass Chinese Elementary School Math Test? 2023. arXiv: 2306.16636 [cs.CL]. URL : https://arxiv.org/abs/2306.16636. [↗](https://arxiv.org/abs/2306.16636)
81. Colin White et al. “LiveBench: A Challenging, Contamination-Free LLM Benchmark”. In: The Thirteenth International Conference on Learning Representations. 2025. [find](https://scholar.google.com/scholar?q=Colin+White+et+al%2E+%E2%80%9CLiveBench%3A+A+Challenging%2C+Contamination%2DFree+LLM+Benchmark%E2%80%9D%2E+In%3A+The+Thirteenth+International+Conference+on+Learning+Representations%2E+2025%2E)
82. Mitchell Wortsman et al. “Small-scale proxies for large-scale transformer training instabilities, 2023”. In: URL https://arxiv.org/abs/2309.14322 (). [↗](https://arxiv.org/abs/2309.14322)
83. Can Xu et al. WizardLM: Empowering large pre-trained language models to follow complex instructions. 2025. arXiv: 2304.12244 [cs.CL]. URL : https://arxiv.org/abs/2304.12244. [↗](https://arxiv.org/abs/2304.12244) [our card](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/)
84. Zhangchen Xu et al. KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding. 2025. arXiv: 2503.02951 [cs.LG]. URL : https://arxiv.org/abs/2503.02951. [↗](https://arxiv.org/abs/2503.02951)
85. John Yang et al. SWE-smith: Scaling Data for Software Engineering Agents. 2025. arXiv: 2504 . 21798 [cs.SE]. URL : https://arxiv.org/abs/2504.21798. — [arXiv:2504.21798](https://arxiv.org/abs/2504.21798) [↗](https://arxiv.org/abs/2504.21798)
86. Shunyu Yao et al. “tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains”. In: arXiv preprint arXiv:2406.12045 (2024). [find](https://scholar.google.com/scholar?q=Shunyu+Yao+et+al%2E+%E2%80%9Ctau%2Dbench%3A+A+Benchmark+for+Tool%2DAgent%2DUser+Interaction+in+Real%2DWorld+Domains%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2406%2E12045+%282024%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/yao-2024-tau-bench-benchmark-tool/)
87. Daoguang Zan et al. “Multi-swe-bench: A multilingual benchmark for issue resolving”. In: arXiv preprint arXiv:2504.02605 (2025). — [arXiv:2504.02605](https://arxiv.org/abs/2504.02605) [find](https://scholar.google.com/scholar?q=Daoguang+Zan+et+al%2E+%E2%80%9CMulti%2Dswe%2Dbench%3A+A+multilingual+benchmark+for+issue+resolving%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2504%2E02605+%282025%29%2E+%E2%80%94+%5BarXiv%3A2504%2E02605%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2504%2E02605%29)
88. Eric Zelikman et al. “Star: Bootstrapping reasoning with reasoning”. In: Advances in Neural Information Processing Systems 35 (2022), pp. 15476–15488. [find](https://scholar.google.com/scholar?q=Eric+Zelikman+et+al%2E+%E2%80%9CStar%3A+Bootstrapping+reasoning+with+reasoning%E2%80%9D%2E+In%3A+Advances+in+Neural+Information+Processing+Systems+35+%282022%29%2C+pp%2E+15476%E2%80%9315488%2E) [our card](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/)
89. Rowan Zellers et al. “Hellaswag: Can a machine really finish your sentence?” In: arXiv preprint arXiv:1905.07830 (2019). [find](https://scholar.google.com/scholar?q=Rowan+Zellers+et+al%2E+%E2%80%9CHellaswag%3A+Can+a+machine+really+finish+your+sentence%3F%E2%80%9D+In%3A+arXiv+preprint+arXiv%3A1905%2E07830+%282019%29%2E)
90. Wanjun Zhong et al. “Agieval: A human-centric benchmark for evaluating foundation models”. In: arXiv preprint arXiv:2304.06364 (2023). [find](https://scholar.google.com/scholar?q=Wanjun+Zhong+et+al%2E+%E2%80%9CAgieval%3A+A+human%2Dcentric+benchmark+for+evaluating+foundation+models%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2304%2E06364+%282023%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/)
91. Jeffrey Zhou et al. “Instruction-Following Evaluation for Large Language Models”. In: ArXiv abs/2311.07911 (2023). URL : https://arxiv.org/abs/2311.07911. [↗](https://arxiv.org/abs/2311.07911) [our card](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/)
92. Qin Zhu et al. AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models. 2025. arXiv: 2502 . 16906 [cs.CL]. URL : https://arxiv.org/abs/2502 . 16906. [↗](https://arxiv.org/abs/2502)