---
title: "DeepSeek-Coder-V2 — DeepSeek-AI 2024"
description: "Open-source code models have lagged behind closed-source offerings, with GPT4-Turbo, Gemini 1.5 Pro and Claude 3 Opus among the leaders."
authors: ["DeepSeek-AI", "Qihao Zhu", "Daya Guo", "Zhihong Shao", "Dejian Yang", "Peiyi Wang", "Runxin Xu", "Y. Wu", "Yukun Li", "Huazuo Gao", "Shirong Ma", "Wangding Zeng", "Xiao Bi", "Zihui Gu", "Hanwei Xu", "Damai Dai", "Kai Dong", "Liyue Zhang", "Yishi Piao", "Zhibin Gou", "Zhenda Xie", "Zhewen Hao", "Bingxuan Wang", "Junxiao Song", "Deli Chen", "Xin Xie", "Kang Guan", "Yuxiang You", "Aixin Liu", "Qiushi Du", "Wenjun Gao", "Xuan Lu", "Qinyu Chen", "Yaohui Wang", "Chengqi Deng", "Jiashi Li", "Chenggang Zhao", "Chong Ruan", "Fuli Luo", "Wenfeng Liang"]
year: 2024
venue: "arXiv:2406.11931"
source: "https://arxiv.org/abs/2406.11931"
doi: "10.48550/arxiv.2406.11931"
arxiv: "2406.11931"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/deepseek-2024-ai-deepseek-coder-v2-breaking-barrier/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/deepseek-2024-ai-deepseek-coder-v2-breaking-barrier/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/) / DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence

# DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence

**Authors:** [DeepSeek-AI](https://md.research.elamur.ai/free/en/authors/deepseek-ai/), [Qihao Zhu](https://md.research.elamur.ai/free/en/authors/qihao-zhu/), [Daya Guo](https://md.research.elamur.ai/free/en/authors/daya-guo/), [Zhihong Shao](https://md.research.elamur.ai/free/en/authors/zhihong-shao/), [Dejian Yang](https://md.research.elamur.ai/free/en/authors/dejian-yang/), [Peiyi Wang](https://md.research.elamur.ai/free/en/authors/peiyi-wang/), [Runxin Xu](https://md.research.elamur.ai/free/en/authors/runxin-xu/), [Y. Wu](https://md.research.elamur.ai/free/en/authors/y-wu/), [Yukun Li](https://md.research.elamur.ai/free/en/authors/yukun-li/), [Huazuo Gao](https://md.research.elamur.ai/free/en/authors/huazuo-gao/), [Shirong Ma](https://md.research.elamur.ai/free/en/authors/shirong-ma/), Wangding Zeng et al. (40 authors) · **Year:** 2024 · **Venue:** arXiv:2406.11931 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2406.11931) · [PDF](https://arxiv.org/pdf/2406.11931) · [DOI 10.48550/arxiv.2406.11931](https://doi.org/10.48550/arxiv.2406.11931) · [arXiv 2406.11931](https://arxiv.org/abs/2406.11931) · [OpenAlex](https://openalex.org/W4399836663)

**Citations:** 51 citations (openalex-snapshot · 2026-10-14) · [144](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Open-source code models have lagged behind closed-source offerings, with GPT4-Turbo, Gemini 1.5 Pro and Claude 3 Opus among the leaders. DeepSeek-Coder-V2 aims to close this gap. It continues pre-training an intermediate DeepSeek-V2 checkpoint, a Mixture-of-Experts model, on 6 trillion additional tokens, for 10.2T tokens in total. The new data mix is about 60% code, 10% mathematics and 30% natural language. Code and math material came from GitHub and CommonCrawl through iterative, fastText-driven recall. This produced 1,170B code-related tokens spanning 338 programming languages and 221B math tokens. The context window was also stretched to 128K tokens.

Alignment combined supervised instruction tuning with GRPO reinforcement learning. A reward model, trained on preferences built from compiler feedback and test cases, guided the RL stage. The smaller variant additionally learned Fill-In-Middle for code completion.

The 236B model equals or surpasses the closed systems on coding and math benchmarks while keeping general language ability near DeepSeek-V2. It is released under a permissive license that allows commercial use. The authors note that instruction following still lags, which limits results on complex tasks like SWE-Bench.

## Key points
- Two MoE sizes: 16B (Lite, 2.4B active parameters) and 236B (21B active). Language coverage grew from 86 to 338, and context from 16K to 128K tokens.
- Code results: 90.2% HumanEval, 76.2% MBPP+ (EvalPlus), 43.4% LiveCodeBench. It is the first open model to exceed 10% on SWE-Bench.
- Math results: 75.7% on MATH (GPT-4o: 76.6%), 94.9% on GSM8K, 53.7% on Math Odyssey. It solved 4/30 AIME 2024 problems with greedy decoding and 5/30 with maj@64.
- Corpus ablation on a 1B model showed clear gains over the old DeepSeek-Coder data after 2T tokens: HumanEval rose from 30.5% to 37.2%, and MBPP from 44.6% to 54.0%.
- General ability includes 79.2 on MMLU and 65.0 on Arena-Hard. Knowledge-heavy tasks dropped relative to DeepSeek-V2 Chat, e.g. TriviaQA fell from 86.7 to 82.3, because less web data was used.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

144 papers of our library cite it (of the 140884 whose bibliographies we read); 13 of them have a page here, 131 not yet.

- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://md.research.elamur.ai/free/en/papers/deepseek-2026-ai-deepseek-v4-highly-efficient/)
- [Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models](https://md.research.elamur.ai/free/en/papers/gao-2024-omni-math-universal-olympiad-level/)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://md.research.elamur.ai/free/en/papers/ge-2024-scaling-synthetic-data-creation-1/)
- [A Survey on Large Language Models for Code Generation](https://md.research.elamur.ai/free/en/papers/jiang-2024-survey-large-language-models-code/)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://md.research.elamur.ai/free/en/papers/li-2024-crowdsourced-data-high-quality-benchmarks/)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://md.research.elamur.ai/free/en/papers/lu-2024-ai-scientist-fully-automated/)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://md.research.elamur.ai/free/en/papers/wang-2025-internvl3-5-advancing-open/)
- [DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search](https://md.research.elamur.ai/free/en/papers/xin-2024-deepseek-prover-v1-5-harnessing/)
- [Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing](https://md.research.elamur.ai/free/en/papers/xu-2024-magpie-alignment-data-synthesis-scratch/)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://md.research.elamur.ai/free/en/papers/yang-2024-qwen2-5-math-technical-report/)
- [The Lessons of Developing Process Reward Models in Mathematical Reasoning](https://md.research.elamur.ai/free/en/papers/zhang-2025-lessons-developing-process-reward-models/)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://md.research.elamur.ai/free/en/papers/zheng-2024-llamafactory-unified-efficient-fine-tuning/)
- [大语言模型的工具使用综述](https://md.research.elamur.ai/free/en/papers/zheng-2025-tool-use-large/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{deepseekai2024deepseek,
  title = {{DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence}},
  author = {DeepSeek-AI and Zhu, Qihao and Guo, Daya and Shao, Zhihong and Yang, Dejian and Wang, Peiyi and Xu, Runxin and Wu, Y. and Li, Yukun and Gao, Huazuo and Ma, Shirong and Zeng, Wangding and Bi, Xiao and Gu, Zihui and Xu, Hanwei and Dai, Damai and Dong, Kai and Zhang, Liyue and Piao, Yishi and Gou, Zhibin and Xie, Zhenda and Hao, Zhewen and Wang, Bingxuan and Song, Junxiao and Chen, Deli and Xie, Xin and Guan, Kang and You, Yuxiang and Liu, Aixin and Du, Qiushi and Gao, Wenjun and Lu, Xuan and Chen, Qinyu and Wang, Yaohui and Deng, Chengqi and Li, Jiashi and Zhao, Chenggang and Ruan, Chong and Luo, Fuli and Liang, Wenfeng},
  year = {2024},
  doi = {10.48550/arxiv.2406.11931},
  eprint = {2406.11931},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.11931}
}
```

- [← previous Deep Learning Based Amharic Chatbot for FAQs in Universities](https://md.research.elamur.ai/free/en/papers/hailu-2024-deep-learning-based-amharic-chatbot/)
- [next → DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search](https://md.research.elamur.ai/free/en/papers/xin-2024-deepseek-prover-v1-5-harnessing/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/)

## References

43 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. L. B. Allal, R. Li, D. Kocetkov, C. Mou, C. Akiki, C. M. Ferrandis, N. Muennighoff, M. Mishra, A. Gu, M. Dey, et al. Santacoder: don’t reach for the stars! arXiv preprint arXiv:2301.03988, 2023. [find](https://scholar.google.com/scholar?q=L%2E+B%2E+Allal%2C+R%2E+Li%2C+D%2E+Kocetkov%2C+C%2E+Mou%2C+C%2E+Akiki%2C+C%2E+M%2E+Ferrandis%2C+N%2E+Muennighoff%2C+M%2E+Mishra%2C+A%2E+Gu%2C+M%2E+Dey%2C+et+al%2E+Santacoder%3A+don%E2%80%99t+reach+for+the+stars%21+arXiv+preprint+arXiv%3A2301%2E03988%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/allal-2023-santacoder-don-t-reach-stars/)
2. A. Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Anthropic%2E+The+claude+3+model+family%3A+Opus%2C+sonnet%2C+haiku%2E+Claude%2D3+Model+Card%2C+2024%2E)
3. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton. Program synthesis with large language models, 2021a. [find](https://scholar.google.com/scholar?q=J%2E+Austin%2C+A%2E+Odena%2C+M%2E+Nye%2C+M%2E+Bosma%2C+H%2E+Michalewski%2C+D%2E+Dohan%2C+E%2E+Jiang%2C+C%2E+Cai%2C+M%2E+Terry%2C+Q%2E+Le%2C+and+C%2E+Sutton%2E+Program+synthesis+with+large+language+models%2C+2021a%2E) [our card](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/)
4. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021b. [find](https://scholar.google.com/scholar?q=J%2E+Austin%2C+A%2E+Odena%2C+M%2E+Nye%2C+M%2E+Bosma%2C+H%2E+Michalewski%2C+D%2E+Dohan%2C+E%2E+Jiang%2C+C%2E+Cai%2C+M%2E+Terry%2C+Q%2E+Le%2C+et+al%2E+Program+synthesis+with+large+language+models%2E+arXiv+preprint+arXiv%3A2108%2E07732%2C+2021b%2E) [our card](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/)
5. M. Bavarian, H. Jun, N. Tezak, J. Schulman, C. McLeavey, J. Tworek, and M. Chen. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255, 2022. [find](https://scholar.google.com/scholar?q=M%2E+Bavarian%2C+H%2E+Jun%2C+N%2E+Tezak%2C+J%2E+Schulman%2C+C%2E+McLeavey%2C+J%2E+Tworek%2C+and+M%2E+Chen%2E+Efficient+training+of+language+models+to+fill+in+the+middle%2E+arXiv+preprint+arXiv%3A2207%2E14255%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/bavarian-2022-efficient-training-language-models-fill/)
6. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. [find](https://scholar.google.com/scholar?q=M%2E+Chen%2C+J%2E+Tworek%2C+H%2E+Jun%2C+Q%2E+Yuan%2C+H%2E+P%2E+d%2E+O%2E+Pinto%2C+J%2E+Kaplan%2C+H%2E+Edwards%2C+Y%2E+Burda%2C+N%2E+Joseph%2C+G%2E+Brockman%2C+et+al%2E+Evaluating+large+language+models+trained+on+code%2E+arXiv+preprint+arXiv%3A2107%2E0337) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
7. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457 . [↗](http://arxiv.org/abs/1803.05457)
8. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [find](https://scholar.google.com/scholar?q=K%2E+Cobbe%2C+V%2E+Kosaraju%2C+M%2E+Bavarian%2C+M%2E+Chen%2C+H%2E+Jun%2C+L%2E+Kaiser%2C+M%2E+Plappert%2C+J%2E+Tworek%2C+J%2E+Hilton%2C+R%2E+Nakano%2C+et+al%2E+Training+verifiers+to+solve+math+word+problems%2E+arXiv+preprint+arXiv%3A2110%2E14168%2C+20) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
9. DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024. [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2E+Deepseek%2Dv2%3A+A+strong%2C+economical%2C+and+efficient+mixture%2Dof%2Dexperts+language+model%2C+2024%2E)
10. Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Dubois%2C+B%2E+Galambosi%2C+P%2E+Liang%2C+and+T%2E+B%2E+Hashimoto%2E+Length%2Dcontrolled+alpacaeval%3A+A+simple+way+to+debias+automatic+evaluators%2E+arXiv+preprint+arXiv%3A2404%2E04475%2C+2024%2E)
11. D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. Li, et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024. [find](https://scholar.google.com/scholar?q=D%2E+Guo%2C+Q%2E+Zhu%2C+D%2E+Yang%2C+Z%2E+Xie%2C+K%2E+Dong%2C+W%2E+Zhang%2C+G%2E+Chen%2C+X%2E+Bi%2C+Y%2E+Wu%2C+Y%2E+Li%2C+et+al%2E+Deepseek%2Dcoder%3A+When+the+large+language+model+meets+programming%E2%80%93the+rise+of+code+intelligence%2E+arXiv+preprint+a)
12. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. [find](https://scholar.google.com/scholar?q=D%2E+Hendrycks%2C+C%2E+Burns%2C+S%2E+Basart%2C+A%2E+Zou%2C+M%2E+Mazeika%2C+D%2E+Song%2C+and+J%2E+Steinhardt%2E+Measuring+massive+multitask+language+understanding%2E+arXiv+preprint+arXiv%3A2009%2E03300%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
13. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. [find](https://scholar.google.com/scholar?q=D%2E+Hendrycks%2C+C%2E+Burns%2C+S%2E+Kadavath%2C+A%2E+Arora%2C+S%2E+Basart%2C+E%2E+Tang%2C+D%2E+Song%2C+and+J%2E+Steinhardt%2E+Measuring+mathematical+problem+solving+with+the+math+dataset%2E+arXiv+preprint+arXiv%3A2103%2E03874%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/)
14. Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023. [find](https://scholar.google.com/scholar?q=Y%2E+Huang%2C+Y%2E+Bai%2C+Z%2E+Zhu%2C+J%2E+Zhang%2C+J%2E+Zhang%2C+T%2E+Su%2C+J%2E+Liu%2C+C%2E+Lv%2C+Y%2E+Zhang%2C+J%2E+Lei%2C+et+al%2E+C%2DEval%3A+A+multi%2Dlevel+multi%2Ddiscipline+chinese+evaluation+suite+for+foundation+models%2E+arXiv+preprint+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/)
15. N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024. [find](https://scholar.google.com/scholar?q=N%2E+Jain%2C+K%2E+Han%2C+A%2E+Gu%2C+W%2E%2DD%2E+Li%2C+F%2E+Yan%2C+T%2E+Zhang%2C+S%2E+Wang%2C+A%2E+Solar%2DLezama%2C+K%2E+Sen%2C+and+I%2E+Stoica%2E+Livecodebench%3A+Holistic+and+contamination+free+evaluation+of+large+language+models+for+code%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/)
16. C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023. [find](https://scholar.google.com/scholar?q=C%2E+E%2E+Jimenez%2C+J%2E+Yang%2C+A%2E+Wettig%2C+S%2E+Yao%2C+K%2E+Pei%2C+O%2E+Press%2C+and+K%2E+Narasimhan%2E+Swe%2Dbench%3A+Can+language+models+resolve+real%2Dworld+github+issues%3F+arXiv+preprint+arXiv%3A2310%2E06770%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/)
17. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In R. Barzilay and M.-Y. Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601… [find](https://scholar.google.com/scholar?q=M%2E+Joshi%2C+E%2E+Choi%2C+D%2E+Weld%2C+and+L%2E+Zettlemoyer%2E+TriviaQA%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+In+R%2E+Barzilay+and+M%2E%2DY%2E+Kan%2C+editors%2C+Proceedings+of+the+55th) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
18. A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jégou, and T. Mikolov. Fasttext. zip: Compressing text classification models. arXiv preprint arXiv:1612.03651, 2016. [find](https://scholar.google.com/scholar?q=A%2E+Joulin%2C+E%2E+Grave%2C+P%2E+Bojanowski%2C+M%2E+Douze%2C+H%2E+J%C3%A9gou%2C+and+T%2E+Mikolov%2E+Fasttext%2E+zip%3A+Compressing+text+classification+models%2E+arXiv+preprint+arXiv%3A1612%2E03651%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/joulin-2016-fasttext-zip-compressing-text-classification/)
19. D. Kocetkov, R. Li, L. Jia, C. Mou, Y. Jernite, M. Mitchell, C. M. Ferrandis, S. Hughes, T. Wolf, D. Bahdanau, et al. The stack: 3 tb of permissively licensed source code. Transactions on Machine Learning Research, 2022. [find](https://scholar.google.com/scholar?q=D%2E+Kocetkov%2C+R%2E+Li%2C+L%2E+Jia%2C+C%2E+Mou%2C+Y%2E+Jernite%2C+M%2E+Mitchell%2C+C%2E+M%2E+Ferrandis%2C+S%2E+Hughes%2C+T%2E+Wolf%2C+D%2E+Bahdanau%2C+et+al%2E+The+stack%3A+3+tb+of+permissively+licensed+source+code%2E+Transactions+on+Machine+Lear)
20. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. P. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research. Trans. Assoc. Comp… [find](https://scholar.google.com/scholar?q=T%2E+Kwiatkowski%2C+J%2E+Palomaki%2C+O%2E+Redfield%2C+M%2E+Collins%2C+A%2E+P%2E+Parikh%2C+C%2E+Alberti%2C+D%2E+Epstein%2C+I%2E+Polosukhin%2C+J%2E+Devlin%2C+K%2E+Lee%2C+K%2E+Toutanova%2C+L%2E+Jones%2C+M%2E+Kelcey%2C+M%2E+Chang%2C+A%2E+M%2E+Dai%2C+J%2E+Uszkoreit%2C+Q%2E+L) [our card](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
21. H. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023a. [find](https://scholar.google.com/scholar?q=H%2E+Li%2C+Y%2E+Zhang%2C+F%2E+Koto%2C+Y%2E+Yang%2C+H%2E+Zhao%2C+Y%2E+Gong%2C+N%2E+Duan%2C+and+T%2E+Baldwin%2E+CMMLU%3A+Measuring+massive+multitask+language+understanding+in+Chinese%2E+arXiv+preprint+arXiv%3A2306%2E09212%2C+2023a%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-cmmlu-measuring-massive-multitask-language/)
22. R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023b. [find](https://scholar.google.com/scholar?q=R%2E+Li%2C+L%2E+B%2E+Allal%2C+Y%2E+Zi%2C+N%2E+Muennighoff%2C+D%2E+Kocetkov%2C+C%2E+Mou%2C+M%2E+Marone%2C+C%2E+Akiki%2C+J%2E+Li%2C+J%2E+Chim%2C+et+al%2E+Starcoder%3A+may+the+source+be+with+you%21+arXiv+preprint+arXiv%3A2305%2E06161%2C+2023b%2E)
23. T. Li, W.-L. Chiang, E. Frick, L. Dunlap, B. Zhu, J. E. Gonzalez, and I. Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024. URL [https://lmsys.org/](https://lmsys.org/) blog/2024-04-19-arena-hard/ . [↗](https://lmsys.org/)
24. J. Liu, C. S. Xia, Y. Wang, and L. Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a. URL [https://openreview.net/for](https://openreview.net/for) um?id=1qvx610Cu7 . [↗](https://openreview.net/for) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/)
25. T. Liu, C. Xu, and J. McAuley. Repobench: Benchmarking repository-level code auto-completion systems. In The Twelfth International Conference on Learning Representations, 2023b. [find](https://scholar.google.com/scholar?q=T%2E+Liu%2C+C%2E+Xu%2C+and+J%2E+McAuley%2E+Repobench%3A+Benchmarking+repository%2Dlevel+code+auto%2Dcompletion+systems%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+2023b%2E)
26. X. Liu, X. Lei, S. Wang, Y. Huang, Z. Feng, B. Wen, J. Cheng, P. Ke, Y. Xu, W. L. Tam, X. Zhang, L. Sun, H. Wang, J. Zhang, M. Huang, Y. Dong, and J. Tang. Alignbench: Benchmarking chinese alignment of large language models. CoRR, abs/2311.18743, 2023c. doi: 10.48550/A RXIV.2311.18743. URL https://d… [find](https://scholar.google.com/scholar?q=X%2E+Liu%2C+X%2E+Lei%2C+S%2E+Wang%2C+Y%2E+Huang%2C+Z%2E+Feng%2C+B%2E+Wen%2C+J%2E+Cheng%2C+P%2E+Ke%2C+Y%2E+Xu%2C+W%2E+L%2E+Tam%2C+X%2E+Zhang%2C+L%2E+Sun%2C+H%2E+Wang%2C+J%2E+Zhang%2C+M%2E+Huang%2C+Y%2E+Dong%2C+and+J%2E+Tang%2E+Alignbench%3A+Benchmarking+chinese+alignment+o)
27. I. Loshchilov and F. Hutter. Decoupled weight decay regularization, 2019. [find](https://scholar.google.com/scholar?q=I%2E+Loshchilov+and+F%2E+Hutter%2E+Decoupled+weight+decay+regularization%2C+2019%2E)
28. A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y. Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Lozhkov%2C+R%2E+Li%2C+L%2E+B%2E+Allal%2C+F%2E+Cassano%2C+J%2E+Lamy%2DPoirier%2C+N%2E+Tazi%2C+A%2E+Tang%2C+D%2E+Pykhtar%2C+J%2E+Liu%2C+Y%2E+Wei%2C+et+al%2E+Starcoder+2+and+the+stack+v2%3A+The+next+generation%2E+arXiv+preprint+arXiv%3A2402%2E19173%2C+20)
29. MAA. American invitational mathematics examination - aime. American Invitational Mathematics Examination - AIME 2024, 2024. URL [https://maa.org/math-competi](https://maa.org/math-competi) tions/american-invitational-mathematics-examination-aime . [↗](https://maa.org/math-competi)
30. Meta. Introducing meta llama 3: The most capable openly available llm to date. [https://ai.meta.com/blog/meta-llama-3/](https://ai.meta.com/blog/meta-llama-3/) , April 2024. [↗](https://ai.meta.com/blog/meta-llama-3/)
31. MistralAI. Codestral. [https://mistral.ai/news/codestral/](https://mistral.ai/news/codestral/) , 2024. Accessed: 2024-05-29. [↗](https://mistral.ai/news/codestral/)
32. Netmind.AI. Odyssey-math. [https://github.com/protagolabs/odyssey-math/tree/main](https://github.com/protagolabs/odyssey-math/tree/main) , 2024. Accessed: April 22, 2024. [↗](https://github.com/protagolabs/odyssey-math/tree/main)
33. OpenAI. Gpt-4 technical report, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
34. B. Peng, J. Quesnelle, H. Fan, and E. Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023. [find](https://scholar.google.com/scholar?q=B%2E+Peng%2C+J%2E+Quesnelle%2C+H%2E+Fan%2C+and+E%2E+Shippole%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2E+arXiv+preprint+arXiv%3A2309%2E00071%2C+2023%2E)
35. M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. [find](https://scholar.google.com/scholar?q=M%2E+Reid%2C+N%2E+Savinov%2C+D%2E+Teplyashin%2C+D%2E+Lepikhin%2C+T%2E+Lillicrap%2C+J%2E%2Db%2E+Alayrac%2C+R%2E+Soricut%2C+A%2E+Lazaridou%2C+O%2E+Firat%2C+J%2E+Schrittwieser%2C+et+al%2E+Gemini+1%2E5%3A+Unlocking+multimodal+understanding+across+million) [our card](https://md.research.elamur.ai/free/en/papers/gemini-2024-team-gemini-1-5-unlocking-multimodal/)
36. B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. [find](https://scholar.google.com/scholar?q=B%2E+Roziere%2C+J%2E+Gehring%2C+F%2E+Gloeckle%2C+S%2E+Sootla%2C+I%2E+Gat%2C+X%2E+E%2E+Tan%2C+Y%2E+Adi%2C+J%2E+Liu%2C+T%2E+Remez%2C+J%2E+Rapin%2C+et+al%2E+Code+llama%3A+Open+foundation+models+for+code%2E+arXiv+preprint+arXiv%3A2308%2E12950%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
37. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. Li, Y. Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. [find](https://scholar.google.com/scholar?q=Z%2E+Shao%2C+P%2E+Wang%2C+Q%2E+Zhu%2C+R%2E+Xu%2C+J%2E+Song%2C+M%2E+Zhang%2C+Y%2E+Li%2C+Y%2E+Wu%2C+and+D%2E+Guo%2E+Deepseekmath%3A+Pushing+the+limits+of+mathematical+reasoning+in+open+language+models%2E+arXiv+preprint+arXiv%3A2402%2E03300%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/)
38. Q. Shi, M. Tang, K. Narasimhan, and S. Yao. Can language models solve olympiad programming? arXiv preprint arXiv:2404.10952, 2024. [find](https://scholar.google.com/scholar?q=Q%2E+Shi%2C+M%2E+Tang%2C+K%2E+Narasimhan%2C+and+S%2E+Yao%2E+Can+language+models+solve+olympiad+programming%3F+arXiv+preprint+arXiv%3A2404%2E10952%2C+2024%2E)
39. M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022. [find](https://scholar.google.com/scholar?q=M%2E+Suzgun%2C+N%2E+Scales%2C+N%2E+Sch%C3%A4rli%2C+S%2E+Gehrmann%2C+Y%2E+Tay%2C+H%2E+W%2E+Chung%2C+A%2E+Chowdhery%2C+Q%2E+V%2E+Le%2C+E%2E+H%2E+Chi%2C+D%2E+Zhou%2C+et+al%2E+Challenging+big%2Dbench+tasks+and+whether+chain%2Dof%2Dthought+can+solve+them%2E+arXiv+pr) [our card](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/)
40. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [find](https://scholar.google.com/scholar?q=H%2E+Touvron%2C+L%2E+Martin%2C+K%2E+Stone%2C+P%2E+Albert%2C+A%2E+Almahairi%2C+Y%2E+Babaei%2C+N%2E+Bashlykov%2C+S%2E+Batra%2C+P%2E+Bhargava%2C+S%2E+Bhosale%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtuned+chat+models%2E+arXiv+preprint+arXiv%3A2)
41. L. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese langu… [find](https://scholar.google.com/scholar?q=L%2E+Xu%2C+H%2E+Hu%2C+X%2E+Zhang%2C+L%2E+Li%2C+C%2E+Cao%2C+Y%2E+Li%2C+Y%2E+Xu%2C+K%2E+Sun%2C+D%2E+Yu%2C+C%2E+Yu%2C+Y%2E+Tian%2C+Q%2E+Dong%2C+W%2E+Liu%2C+B%2E+Shi%2C+Y%2E+Cui%2C+J%2E+Li%2C+J%2E+Zeng%2C+R%2E+Wang%2C+W%2E+Xie%2C+Y%2E+Li%2C+Y%2E+Patterson%2C+Z%2E+Tian%2C+Y%2E+Zhang%2C+H%2E+Zhou%2C+S) [our card](https://md.research.elamur.ai/free/en/papers/xu-2020-clue-chinese-language-understanding/)
42. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. [find](https://scholar.google.com/scholar?q=L%2E+Zheng%2C+W%2E%2DL%2E+Chiang%2C+Y%2E+Sheng%2C+S%2E+Zhuang%2C+Z%2E+Wu%2C+Y%2E+Zhuang%2C+Z%2E+Lin%2C+Z%2E+Li%2C+D%2E+Li%2C+E%2E+P%2E+Xing%2C+H%2E+Zhang%2C+J%2E+E%2E+Gonzalez%2C+and+I%2E+Stoica%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2C+2023%2E)
43. W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan. AGIEval: A human-centric benchmark for evaluating foundation models. CoRR, abs/2304.06364, 2023. doi: 10.48550/arXiv.2304.06364. URL https://doi.org/10.48550/arXiv.2304.06364 . [↗](https://doi.org/10.48550/arXiv.2304.06364) [our card](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/)