---
title: "CodeGen2 — paper by Nijkamp et al. (2023)"
description: "Training large language models for code is expensive, and practitioners must choose among… 31 references, cited by 59 papers here."
authors: ["Erik Nijkamp", "Hiroaki Hayashi", "Caiming Xiong", "Silvio Savarese", "Yingbo Zhou"]
year: 2023
venue: "arXiv:2305.02309"
source: "https://arxiv.org/abs/2305.02309"
doi: "10.48550/arxiv.2305.02309"
arxiv: "2305.02309"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nijkamp-2023-codegen2-lessons-training-llms-programming/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nijkamp-2023-codegen2-lessons-training-llms-programming/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/) / CodeGen2: Lessons for Training LLMs on Programming and Natural Languages

# CodeGen2: Lessons for Training LLMs on Programming and Natural Languages

**Authors:** Erik Nijkamp, [Hiroaki Hayashi](https://md.research.elamur.ai/free/en/authors/hiroaki-hayashi/), [Caiming Xiong](https://md.research.elamur.ai/free/en/authors/caiming-xiong/), [Silvio Savarese](https://md.research.elamur.ai/free/en/authors/silvio-savarese/) and [Yingbo Zhou](https://md.research.elamur.ai/free/en/authors/yingbo-zhou/) · **Year:** 2023 · **Venue:** arXiv:2305.02309 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2305.02309) · [PDF](https://arxiv.org/pdf/2305.02309) · [DOI 10.48550/arxiv.2305.02309](https://doi.org/10.48550/arxiv.2305.02309) · [arXiv 2305.02309](https://arxiv.org/abs/2305.02309) · [OpenAlex](https://openalex.org/W4368755703)

**Citations:** 35 citations (OpenAlex, 2026-10-14) · [59](https://md.research.elamur.ai/free/en/papers/nijkamp-2023-codegen2-lessons-training-llms-programming/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Training large language models for code is expensive, and practitioners must choose among architectures, objectives, sampling modes and data mixes without clear guidance. The Salesforce Research team behind CodeGen2 tried to fold these choices into one recipe that would yield a single universal model for both program synthesis and program understanding. They tested four hypotheses in a broad set of experiments with 1B-parameter models:
- a Prefix-LM could combine encoder and decoder behaviour;
- causal language modeling, span corruption and infilling could be merged into one objective;
- infill capability could come at no extra compute cost (the "free lunch" claim);
- mixing natural and programming languages would help both domains.

Results were mixed. Prefix-LM showed no measurable advantage over a plain causal decoder, and the free-lunch claim was not supported. However, a simple mixed objective kept zero-shot quality, language mixing looked promising, and multi-epoch training worked well.

The study matters because it openly reports negative findings, distils them into five practical lessons, and releases infill-capable models trained only on permissively licensed data, together with the training code.

## Key points
- Four components were studied for unification: architecture (Prefix-LM vs. causal decoder), learning objective, sampling (left-to-right vs. infill) and data distribution. The findings are condensed into five lessons.
- The proposed objective mixes uncorrupted sequences with within-file span-corruption sequences under next-token prediction. It supports both left-to-right and fill-in-the-middle generation.
- Evaluation covered several benchmarks:
  - HumanEval for zero-shot left-to-right synthesis
  - HumanEval-Infill for infilling
  - XSum for few-shot in-context learning
  - CodeXGLUE (e.g., defect detection) and SuperGLUE for fine-tuned understanding
- The CodeGen2 models are released in 1B, 3.7B, 7B and 16B sizes. Training code is available at github.com/salesforce/CodeGen.
- Training data is restricted to permissively licensed sources. The paper cites The Stack as its permissive code corpus.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Salesforce](https://md.research.elamur.ai/free/en/companies/salesforce/) — Company, San Francisco, United States · 59 works · 18,172 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [🖥️ Computer Science Applications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-science-applications/) · 131 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese and Yingbo Zhou. 2023. CodeGen2: Lessons for Training LLMs on Programming and Natural Languages. arXiv:2305.02309.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{nijkamp2023codegen2,
  title = {{CodeGen2: Lessons for Training LLMs on Programming and Natural Languages}},
  author = {Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese and Yingbo Zhou},
  year = {2023},
  doi = {10.48550/arxiv.2305.02309},
  eprint = {2305.02309},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.02309}
}
```

- [← previous Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions](https://md.research.elamur.ai/free/en/papers/yang-2023-auto-gpt-online-decision-making/)
- [next → Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Better](https://md.research.elamur.ai/free/en/papers/dale-2023-detecting-mitigating-hallucinations-machine-translation/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/)

## References

31 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. Santacoder: don’t reach for the stars! arXiv preprint arXiv:2301.[03988, 2023](https://md.research.elamur.ai/free/en/papers/allal-2023-santacoder-don-t-reach-stars/).
2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732, 2021.
3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). arXiv preprint arXiv:1409.0473, 2014.
4. HumanEval pass@1 pass@10 pass@100 18.83 20.46 28.36 31.78 36.50 47.46 50.41 56.71 75.15 Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. [Efficient training of language models to fill in the middle](https://md.research.elamur.ai/free/en/papers/bavarian-2022-efficient-training-language-models-fill/). arXiv preprint arXiv:2207.14255, 2022.
5. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021a.
8. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021b.
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum…
10. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. [Glm: General language model pretraining with autoregressive blank infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 320–335, 2022.
11. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999, 2022. [find](https://scholar.google.com/scholar?q=Daniel+Fried%2C+Armen+Aghajanyan%2C+Jessy+Lin%2C+Sida+Wang%2C+Eric+Wallace%2C+Freda+Shi%2C+Ruiqi+Zhong%2C+Wen%2Dtau+Yih%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+Incoder%3A+A+generative+model+for+code+infilling+and+synthesis%2E)
12. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+et+al%2E+The+pile%3A+An+800gb+dataset+of+diverse+text+for+language+)
13. Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, et al. The stack: 3 tb of permissively licensed source code. arXiv preprint arXiv:2211.15533, 2022. [find](https://scholar.google.com/scholar?q=Denis+Kocetkov%2C+Raymond+Li%2C+Loubna+Ben+Allal%2C+Jia+Li%2C+Chenghao+Mou%2C+Carlos+Mun%CC%83oz+Ferrandis%2C+Yacine+Jernite%2C+Margaret+Mitchell%2C+Sean+Hughes%2C+Thomas+Wolf%2C+et+al%2E+The+stack%3A+3+tb+of+permissively+license)
14. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. [Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the Assoc…
15. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicola… [find](https://scholar.google.com/scholar?q=Raymond+Li%2C+Loubna+Ben+Allal%2C+Yangtian+Zi%2C+Niklas+Muennighoff%2C+Denis+Kocetkov%2C+Chenghao+Mou%2C+Marc+Marone%2C+Christopher+Akiki%2C+Jia+Li%2C+Jenny+Chim%2C+Qian+Liu%2C+Evgenii+Zheltonozhskii%2C+Terry+Yue+Zhuo%2C+Thoma)
16. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James… — [Competition-Level Code Generation with AlphaCode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/)
17. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664, 2021. [find](https://scholar.google.com/scholar?q=Shuai+Lu%2C+Daya+Guo%2C+Shuo+Ren%2C+Junjie+Huang%2C+Alexey+Svyatkovskiy%2C+Ambrosio+Blanco%2C+Colin+Clement%2C+Dawn+Drain%2C+Daxin+Jiang%2C+Duyu+Tang%2C+et+al%2E+Codexglue%3A+A+machine+learning+benchmark+dataset+for+code+und)
18. Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.[08745, 2018](https://md.research.elamur.ai/free/en/papers/narayan-2018-don-t-give-me-details/).
19. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474, 2022. [find](https://scholar.google.com/scholar?q=Erik+Nijkamp%2C+Bo+Pang%2C+Hiroaki+Hayashi%2C+Lifu+Tu%2C+Huan+Wang%2C+Yingbo+Zhou%2C+Silvio+Savarese%2C+and+Caiming+Xiong%2E+Codegen%3A+An+open+large+language+model+for+code+with+multi%2Dturn+program+synthesis%2E+arXiv+pre)
20. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). arXiv preprint arXiv:2203.02155, 2022.
21. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/). In Proceedings of the 54th Annual Meeting of the Association for Co…
22. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+Ilya+Sutskever%2C+et+al%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+2018%2E)
23. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). J. Mach. Learn. Res., 21(140):1–67, 2020.
24. Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. arXiv preprint arXiv:2205.[05131, 2022a](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/).
25. Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q Tran, David R So, Siamak Shakeri, Xavier Garcia, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, et al. [Transcending scaling laws with 0.1% extra compute](https://md.research.elamur.ai/free/en/papers/tay-2022-transcending-scaling-laws-0-1/). arXiv preprint arXiv:2210.11399, 2022b.
26. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. [Lamda: Language models for dialog applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/). arXiv preprint arXiv:2201.08239, 2022.
27. Priyan Vaithilingam, Tianyi Zhang, and Elena L Glassman. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1–7, 2022. [find](https://scholar.google.com/scholar?q=Priyan+Vaithilingam%2C+Tianyi+Zhang%2C+and+Elena+L+Glassman%2E+Expectation+vs%2E+experience%3A+Evaluating+the+usability+of+code+generation+tools+powered+by+large+language+models%2E+In+CHI+Conference+on+Human+Fact)
28. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). Advances in neural information processing systems, 32, 2019.
29. Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective works best for zero-shot generalization? In International Conference on Machine Learning, pp. 22964–22984. PMLR, 2022. [find](https://scholar.google.com/scholar?q=Thomas+Wang%2C+Adam+Roberts%2C+Daniel+Hesslow%2C+Teven+Le+Scao%2C+Hyung+Won+Chung%2C+Iz+Beltagy%2C+Julien+Launay%2C+and+Colin+Raffel%2E+What+language+model+architecture+and+pretraining+objective+works+best+for+zero%2Ds)
30. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). arXiv preprint arXiv:2206.07682, 2022.
31. Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. arXiv preprint arXiv:2302.03169, 2023. [find](https://scholar.google.com/scholar?q=Sang+Michael+Xie%2C+Shibani+Santurkar%2C+Tengyu+Ma%2C+and+Percy+Liang%2E+Data+selection+for+language+models+via+importance+resampling%2E+arXiv+preprint+arXiv%3A2302%2E03169%2C+2023%2E)

*Updated 2 November 2026*