---
title: "CodeGemma: Open Code Models Based on Gemma (2024)"
description: "CodeGemma is a family of open code models from Google. 24 references, cited by 61 papers here."
authors: ["CodeGemma Team: Heri Zhao", "Jeffrey Hui", "Joshua Howland", "Nam Nguyen", "Siqi Zuo", "Andrea Hu", "Christopher A. Choquette-Choo", "Jingyue Shen", "Joe Kelley", "Kshitij Bansal", "Luke Vilnis", "Mateo Wirth", "Paul Michel", "Peter Choy", "Pratik Joshi", "Ravin Kumar", "Sarmad Hashmi", "Shubham Agrawal", "Zhitao Gong", "Jane Fine", "Tris Warkentin", "Ale Jakse Hartman", "Bin Ni", "Kathy Korevec", "Kelly Schaefer", "Scott Huffman"]
year: 2024
venue: "arXiv:2406.11409"
source: "https://arxiv.org/abs/2406.11409"
doi: "10.48550/arxiv.2406.11409"
arxiv: "2406.11409"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhao-2024-codegemma-open-code-models-based/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhao-2024-codegemma-open-code-models-based/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / CodeGemma: Open Code Models Based on Gemma

# CodeGemma: Open Code Models Based on Gemma

**Authors:** CodeGemma Team: Heri Zhao, Jeffrey Hui, Joshua Howland, Nam Nguyen, Siqi Zuo, Andrea Hu, [Christopher A. Choquette-Choo](https://md.research.elamur.ai/free/en/authors/christopher-a-choquette-choo/), Jingyue Shen, Joe Kelley, Kshitij Bansal, [Luke Vilnis](https://md.research.elamur.ai/free/en/authors/luke-vilnis/), Mateo Wirth, [Paul Michel](https://md.research.elamur.ai/free/en/authors/paul-michel/), Peter Choy, Pratik Joshi, Ravin Kumar, Sarmad Hashmi, Shubham Agrawal, Zhitao Gong, Jane Fine, Tris Warkentin, Ale Jakse Hartman, Bin Ni, Kathy Korevec, Kelly Schaefer and Scott Huffman · **Year:** 2024 · **Venue:** arXiv:2406.11409 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2406.11409) · [PDF](https://arxiv.org/pdf/2406.11409) · [DOI 10.48550/arxiv.2406.11409](https://doi.org/10.48550/arxiv.2406.11409) · [arXiv 2406.11409](https://arxiv.org/abs/2406.11409) · [OpenAlex](https://openalex.org/W4399795128)

**Citations:** 21 citations (OpenAlex, 2026-10-14) · [61](https://md.research.elamur.ai/free/en/papers/zhao-2024-codegemma-open-code-models-based/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
CodeGemma is a family of open code models from Google. They are built by continuing the training of Gemma checkpoints on large amounts of mostly code data, with the aim of strong coding skills without losing general language and reasoning ability.

There are three variants:
- a 2B model for fast code completion,
- a 7B pretrained model,
- a 7B instruction-tuned model.

Pretraining covered 500B tokens, or 1T tokens for the 2B v1.1 model. It used a fill-in-the-middle (FIM) objective in both PSM and SPM modes. Related files from the same repository were packed into one training example, grouped by dependency graphs and by pairing unit tests with their implementations. Instruction tuning combined math datasets with synthetic code question-answer pairs filtered by an LLM, followed by SFT and RLHF.

The 2B model matches comparable open models on HumanEval Infilling while running much faster. The 7B models clearly beat base Gemma on coding and score well on math among models of similar size. This makes CodeGemma practical both for IDEs and memory-limited setups (2B) and for hosted, quality-focused deployments (7B).

## Key points
- **Releases:** v1.0 (April 9, 2024) included all three models. v1.1 (May 3, 2024) updated the 2B PT and 7B IT models.
- **Training mix and FIM rate:**
  - 2B models: 100% code; 7B models: 80% code and 20% natural language.
  - FIM rate: 80%, except 90% for 2B v1.1.
  - Special tokens: `<|fim_prefix|>`, `<|fim_suffix|>`, `<|fim_middle|>`, `<|file_separator|>`.
- **Infilling speed:** On HumanEval Infilling single-line, CodeGemma 2B reaches 78.41%, and 79.28% in v1.1. It needs 543 s in total, against 990 s for DeepSeek Coder 2B-class.
- **Python coding (HumanEval / MBPP):**

  | Model | HumanEval | MBPP |
  |---|---|---|
  | 7B-IT 1.1 | 60.4% | 55.2% |
  | 2B-PT 1.1 | 37.8% | 49.2% |
  | Gemma 7B PT | 32.3% | 44.4% |
  | Gemma 2B PT | 22.0% | 29.2% |

- **Math reasoning (7B IT 1.1):** 47.3% on GSM8K and 22.3% on MATH, compared with 13.0% GSM8K for Code Llama. Instruction tuning drew on MATH, GSM8K, MathQA and synthetic algebra problems, plus OSS-Instruct-style code data.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

🧭 Topics (1): [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 56 works

## Cite

Cite as: CodeGemma Team: Heri Zhao, Jeffrey Hui, Joshua Howland, Nam Nguyen, Siqi Zuo, Andrea Hu, Christopher A. Choquette-Choo, Jingyue Shen, Joe Kelley, Kshitij Bansal, Luke Vilnis, Mateo Wirth, Paul Michel, Peter Choy, Pratik Joshi, Ravin Kumar, Sarmad Hashmi, Shubham Agrawal, Zhitao Gong, Jane Fine, Tris Warkentin, Ale Jakse Hartman, Bin Ni, Kathy Korevec, Kelly Schaefer and Scott Huffman. 2024. CodeGemma: Open Code Models Based on Gemma. arXiv:2406.11409.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhao2024codegemma,
  title = {{CodeGemma: Open Code Models Based on Gemma}},
  author = {CodeGemma Team: Heri Zhao, Jeffrey Hui, Joshua Howland, Nam Nguyen, Siqi Zuo, Andrea Hu, Christopher A. Choquette-Choo, Jingyue Shen, Joe Kelley, Kshitij Bansal, Luke Vilnis, Mateo Wirth, Paul Michel, Peter Choy, Pratik Joshi, Ravin Kumar, Sarmad Hashmi, Shubham Agrawal, Zhitao Gong, Jane Fine, Tris Warkentin, Ale Jakse Hartman, Bin Ni, Kathy Korevec, Kelly Schaefer and Scott Huffman},
  year = {2024},
  doi = {10.48550/arxiv.2406.11409},
  eprint = {2406.11409},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.11409}
}
```

- [← previous MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning](https://md.research.elamur.ai/free/en/papers/liu-2024-mmc-advancing-multimodal-chart-understanding/)
- [next → Mitigating the Alignment Tax of RLHF](https://md.research.elamur.ai/free/en/papers/lin-2024-mitigating-alignment-tax-rlhf/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

24 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Amini, S. Gabriel, P. Lin, R. Koncel-Kedziorski, Y. Choi, and H. Hajishirzi. [MathQA: Towards interpretable math word problem solving with operation-based formalisms](https://md.research.elamur.ai/free/en/papers/amini-2019-mathqa-interpretable-math-word-problem/), 2019. URL http://arxiv.org/abs/1905.13319 . [↗](http://arxiv.org/abs/1905.13319)
2. J. Austin, A. Odena, M. I. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. J. Cai, M. Terry, Q. V. Le, and C. Sutton. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). CoRR, abs/2108.07732, 2021. URL https://arxiv.org/abs/2108.07732 . have solved question answering? try arc, the ai2 reasoning cha… [↗](https://arxiv.org/abs/2108.07732)
3. J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, S. Wang, W. Wang, S. Wu, B. Xu, J. Xu, A. Yang, H. Yang, J. Yang, S. Yang, Y. Yao, B.… — [Qwen Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
4. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/), 2021a. URL [https://arxiv.org/](https://arxiv.org/) abs/2110.14168v2 . [↗](https://arxiv.org/)
5. M. Bavarian, H. Jun, N. Tezak, J. Schulman, C. McLeavey, J. Tworek, and M. Chen. [Efficient training of language models to fill in the middle](https://md.research.elamur.ai/free/en/papers/bavarian-2022-efficient-training-language-models-fill/), 2022.
6. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). CoRR, abs/2110.14168, 2021b. URL https://arxiv.org/abs/2110.14168 . [↗](https://arxiv.org/abs/2110.14168)
7. Y. Ding, Z. Wang, W. U. Ahmad, H. Ding, M. Tan, N. Jain, M. K. Ramanathan, R. Nallapati, P. Bhatia, D. Roth, and B. Xiang. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion, 2023. [find](https://scholar.google.com/scholar?q=Y%2E+Ding%2C+Z%2E+Wang%2C+W%2E+U%2E+Ahmad%2C+H%2E+Ding%2C+M%2E+Tan%2C+N%2E+Jain%2C+M%2E+K%2E+Ramanathan%2C+R%2E+Nallapati%2C+P%2E+Bhatia%2C+D%2E+Roth%2C+and+B%2E+Xiang%2E+Crosscodeeval%3A+A+diverse+and+multilingual+benchmark+for+cross%2Dfile+code+compl)
8. Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. [PIQA: reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). CoRR, abs/1911.11641, 2019. URL http://arxiv.org/abs/1911. 11641 . D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W. tau Yih, L. Zettle-M. Chen, J. Tworek, H… [↗](http://arxiv.org/abs/1911)
9. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you M. Wirth, M. Sharman, N. Chinaev, N. Thain, O. Bachem, O. Chang, O. Wahltinez, P. Bailey, P. Michel, P. Yotov, P. G. Sessa, R. Chaabouni, R. Comanescu, R. Jana, R. Anil, R. McIlroy, R. Liu, R. Mullins, S.… [find](https://scholar.google.com/scholar?q=P%2E+Clark%2C+I%2E+Cowhey%2C+O%2E+Etzioni%2C+T%2E+Khot%2C+A%2E+Sabharwal%2C+C%2E+Schoenick%2C+and+O%2E+Tafjord%2E+Think+you+M%2E+Wirth%2C+M%2E+Sharman%2C+N%2E+Chinaev%2C+N%2E+Thain%2C+O%2E+Bachem%2C+O%2E+Chang%2C+O%2E+Wahltinez%2C+P%2E+Bailey%2C+P%2E+Michel%2C+P%2E+)
10. D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. K. Li, F. Luo, Y. Xiong, and W. Liang. Deepseek-coder: When the large language model meets programming – the rise of code intelligence, 2024. [find](https://scholar.google.com/scholar?q=D%2E+Guo%2C+Q%2E+Zhu%2C+D%2E+Yang%2C+Z%2E+Xie%2C+K%2E+Dong%2C+W%2E+Zhang%2C+G%2E+Chen%2C+X%2E+Bi%2C+Y%2E+Wu%2C+Y%2E+K%2E+Li%2C+F%2E+Luo%2C+Y%2E+Xiong%2C+and+W%2E+Liang%2E+Deepseek%2Dcoder%3A+When+the+large+language+model+meets+programming+%E2%80%93+the+rise+of+code+)
11. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). CoRR, abs/2009.03300, 2020. URL https://arxiv.org/abs/2009.03300 . [↗](https://arxiv.org/abs/2009.03300)
12. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). NeurIPS, 2021.
13. A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mistral 7b, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Q%2E+Jiang%2C+A%2E+Sablayrolles%2C+A%2E+Mensch%2C+C%2E+Bamford%2C+D%2E+S%2E+Chaplot%2C+D%2E+de+las+Casas%2C+F%2E+Bressand%2C+G%2E+Lengyel%2C+G%2E+Lample%2C+L%2E+Saulnier%2C+L%2E+R%2E+Lavaud%2C+M%2E%2DA%2E+Lachaux%2C+P%2E+Stock%2C+T%2E+L%2E+Scao%2C+T%2E+Lavril%2C+T%2E+W)
14. M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer. [Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). CoRR, abs/1705.03551, 2017. URL http://arxiv.org/abs/1705.03551 . [↗](http://arxiv.org/abs/1705.03551)
15. W. Ling, D. Yogatama, C. Dyer, and P. Blunsom. Program induction by rationale generation : Learning to solve and explain algebraic word problems, 2017. URL [https://arxiv.org/](https://arxiv.org/) abs/1705.04146v3 . [↗](https://arxiv.org/)
16. T. Liu, C. Xu, and J. McAuley. Repobench: Benchmarking repository-level code auto-completion systems, 2023. [find](https://scholar.google.com/scholar?q=T%2E+Liu%2C+C%2E+Xu%2C+and+J%2E+McAuley%2E+Repobench%3A+Benchmarking+repository%2Dlevel+code+auto%2Dcompletion+systems%2C+2023%2E)
17. A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y. Wei, T. Liu, M. Tian, D. Kocetkov, A. Zucker, Y. Belkada, Z. Wang, Q. Liu, D. Abulkhanov, I. Paul, Z. Li, W.-D. Li, M. Risdal, J. Li, J. Zhu, T. Y. Zhuo, E. Zheltonozhskii, N. O. O. Dade, W. Yu, L.… [find](https://scholar.google.com/scholar?q=A%2E+Lozhkov%2C+R%2E+Li%2C+L%2E+B%2E+Allal%2C+F%2E+Cassano%2C+J%2E+Lamy%2DPoirier%2C+N%2E+Tazi%2C+A%2E+Tang%2C+D%2E+Pykhtar%2C+J%2E+Liu%2C+Y%2E+Wei%2C+T%2E+Liu%2C+M%2E+Tian%2C+D%2E+Kocetkov%2C+A%2E+Zucker%2C+Y%2E+Belkada%2C+Z%2E+Wang%2C+Q%2E+Liu%2C+D%2E+Abulkhanov%2C+I%2E+Paul%2C)
18. G. Orlanski, K. Xiao, X. Garcia, J. Hui, J. Howland, J. Malmaud, J. Austin, R. Singh, and M. Catasta. Measuring the impact of programming language distribution. arXiv preprint arXiv:2302.01973, 2023. [find](https://scholar.google.com/scholar?q=G%2E+Orlanski%2C+K%2E+Xiao%2C+X%2E+Garcia%2C+J%2E+Hui%2C+J%2E+Howland%2C+J%2E+Malmaud%2C+J%2E+Austin%2C+R%2E+Singh%2C+and+M%2E+Catasta%2E+Measuring+the+impact+of+programming+language+distribution%2E+arXiv+preprint+arXiv%3A2302%2E01973%2C+2023%2E)
19. B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A. Défossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, and G. Synnaev… — [Code Llama: Open Foundation Models for Code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
20. K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. WINOGRANDE: an adversarial winograd schema challenge at scale. CoRR, abs/1907.10641, 2019. URL http://arxiv.org/abs/1907.10641 . [↗](http://arxiv.org/abs/1907.10641)
21. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan,… [find](https://scholar.google.com/scholar?q=H%2E+Touvron%2C+L%2E+Martin%2C+K%2E+Stone%2C+P%2E+Albert%2C+A%2E+Almahairi%2C+Y%2E+Babaei%2C+N%2E+Bashlykov%2C+S%2E+Batra%2C+P%2E+Bhargava%2C+S%2E+Bhosale%2C+D%2E+Bikel%2C+L%2E+Blecher%2C+C%2E+C%2E+Ferrer%2C+M%2E+Chen%2C+G%2E+Cucurull%2C+D%2E+Esiobu%2C+J%2E+Fernandes%2C)
22. Y. Wei, Z. Wang, J. Liu, Y. Ding, and L. Zhang. Magicoder: Source code is all you need, 2023. URL http://arxiv.org/abs/2312. 02120 . [↗](http://arxiv.org/abs/2312)
23. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush. Huggingface’s transformers: State-of-the-art natural languag… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
24. R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can a machine really finish your sentence?, 2019. [find](https://scholar.google.com/scholar?q=R%2E+Zellers%2C+A%2E+Holtzman%2C+Y%2E+Bisk%2C+A%2E+Farhadi%2C+and+Y%2E+Choi%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F%2C+2019%2E)

*Updated 24 October 2026*