---
title: "A Systematic Evaluation of Large Language Models of Code (2022)"
description: "It is also unclear how design choices affect performance, such as training on many programming… 33 references, cited by 108 papers here."
authors: ["Frank F. Xu", "Uri Alon", "Graham Neubig", "Vincent J. Hellendoorn"]
year: 2022
venue: "arXiv:2202.13169"
source: "https://arxiv.org/abs/2202.13169"
doi: "10.48550/arxiv.2202.13169"
arxiv: "2202.13169"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2022-systematic-evaluation-large-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2022-systematic-evaluation-large-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / A Systematic Evaluation of Large Language Models of Code

# A Systematic Evaluation of Large Language Models of Code

**Authors:** [Frank F. Xu](https://md.research.elamur.ai/free/en/authors/frank-f-xu/), [Uri Alon](https://md.research.elamur.ai/free/en/authors/uri-alon/), [Graham Neubig](https://md.research.elamur.ai/free/en/authors/graham-neubig/) and Vincent J. Hellendoorn · **Year:** 2022 · **Venue:** arXiv:2202.13169 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2202.13169) · [PDF](https://arxiv.org/pdf/2202.13169) · [DOI 10.48550/arxiv.2202.13169](https://doi.org/10.48550/arxiv.2202.13169) · [arXiv 2202.13169](https://arxiv.org/abs/2202.13169) · [OpenAlex](https://openalex.org/W4221141536)

**Citations:** 17 citations (OpenAlex, 2026-10-14) · [108](https://md.research.elamur.ai/free/en/papers/xu-2022-systematic-evaluation-large-language-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The strongest code language models, such as Codex, are closed. Their weights and training data are unavailable, so researchers cannot fine-tune, inspect or extend them. It is also unclear how design choices affect performance, such as training on many programming languages versus one. The authors compare the largest accessible left-to-right models (Codex, GPT-J, GPT-Neo, GPT-NeoX-20B and CodeParrot) in two ways. The first is functional correctness on the HumanEval benchmark, which tests Python synthesis from natural-language prompts. The second is perplexity on a newly assembled held-out set of GitHub files in 12 programming languages, filtered to avoid overlap with the Pile. To fill a gap in open resources, they also train and release PolyCoder, a family of GPT-2-style models built only from multilingual source code. Codex leads in most languages, not just Python, and outperforms the Pile-trained models. However, PolyCoder 2.7B reaches the lowest perplexity of all models in C, Codex included. Publishing the models openly makes research on code LMs more reproducible and affordable outside large companies.

## Key points
- PolyCoder comes in three sizes (160M, 400M, 2.7B parameters). It was trained on 249GB of code covering 12 languages using a single machine, and is released at github.com/VHellendoorn/Code-LMs.
- The HumanEval setup used 164 prompts with test cases and 100 samples per prompt. Sampling used nucleus sampling (top-p 0.95) at temperatures from 0.2 to 0.8.
- Intrinsic evaluation used 100 random files per language. Perplexity was normalized by Pygments lexer tokens so that models with different tokenizers can be compared. Codex (code-davinci-001) may have seen the test data, so its scores could be inflated.
- In C, perplexity was 2.33 for PolyCoder 2.7B, 2.37 for GPT-NeoX, 2.55 for Codex and 3.69 for GPT-Neo 2.7B. In Python, Codex scored 1.47 versus 3.18 for PolyCoder. CodeParrot was trained only on Python and does poorly in other languages.
- Training longer (100K to 150K steps) helped the 2.7B model but not the 400M one. A high temperature such as 0.8 lowered Pass@100 for the 2.7B model, so temperature should be tuned for each model size.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 598 works · 191,226 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works · [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 168 works

## Cite

Cite as: Frank F. Xu, Uri Alon, Graham Neubig and Vincent J. Hellendoorn. 2022. A Systematic Evaluation of Large Language Models of Code. arXiv:2202.13169.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{xu2022systematic,
  title = {{A Systematic Evaluation of Large Language Models of Code}},
  author = {Frank F. Xu, Uri Alon, Graham Neubig and Vincent J. Hellendoorn},
  year = {2022},
  doi = {10.48550/arxiv.2202.13169},
  eprint = {2202.13169},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2202.13169}
}
```

- [← previous PEER: A Collaborative Language Model](https://md.research.elamur.ai/free/en/papers/schick-2022-peer-collaborative-language-model/)
- [next → ReACC: A Retrieval-Augmented Code Completion Framework](https://md.research.elamur.ai/free/en/papers/lu-2022-reacc-retrieval-augmented-code-completion/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

33 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. [Unified pre-training for program understanding and generation](https://md.research.elamur.ai/free/en/papers/ahmad-2021-unified-pre-training-program-understanding/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2655–2668, Online, J…
2. Miltiadis Allamanis. [The adverse effects of code duplication in machine learning models of code](https://md.research.elamur.ai/free/en/papers/allamanis-2018-adverse-effects-code-duplication-machine/). In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, pp. 143–153, 2019.
3. Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. Structural language models of code. In International Conference on Machine Learning, pp. 245–256. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Uri+Alon%2C+Roy+Sadaka%2C+Omer+Levy%2C+and+Eran+Yahav%2E+Structural+language+models+of+code%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+245%E2%80%93256%2E+PMLR%2C+2020%2E)
4. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732, 2021.
5. Alexei Baevski and Michael Auli. [Adaptive input representations for neural language modeling](https://md.research.elamur.ai/free/en/papers/baevski-2018-adaptive-input-representations-neural-language/). arXiv preprint arXiv:1809.10853, 2018.
6. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. [A neural probabilistic language model](https://md.research.elamur.ai/free/en/papers/bengio-2003-neural-probabilistic-language-model/). Journal of machine learning research, 3(Feb):1137–1155, 2003.
7. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. URL https://doi.org/10.5281/zenodo.5297715. If you use this software, please cite it using these metadata. — [doi:10.5281/zenodo.5297715](https://doi.org/10.5281/zenodo.5297715) [↗](https://doi.org/10.5281/zenodo.5297715)
8. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20B: An open-source autoregressive… — [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/)
9. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). arXiv preprint arXiv:2005.14165, 2020.
10. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021.
11. Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. Advances in Neural Information Processing Systems, 32:7059–7069, 2019. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Guillaume+Lample%2E+Cross%2Dlingual+language+model+pretraining%2E+Advances+in+Neural+Information+Processing+Systems%2C+32%3A7059%E2%80%937069%2C+2019%2E)
12. Aditya Desai, Sumit Gulwani, Vineet Hingorani, Nidhi Jain, Amey Karkare, Mark Marron, and Subhajit Roy. Program synthesis using natural language. In Proceedings of the 38th International Conference on Software Engineering, pp. 345–356, 2016. [find](https://scholar.google.com/scholar?q=Aditya+Desai%2C+Sumit+Gulwani%2C+Vineet+Hingorani%2C+Nidhi+Jain%2C+Amey+Karkare%2C+Mark+Marron%2C+and+Subhajit+Roy%2E+Program+synthesis+using+natural+language%2E+In+Proceedings+of+the+38th+International+Conference+on)
13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805, 2018.
14. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. [Codebert: A pre-trained model for programming and natural languages](https://md.research.elamur.ai/free/en/papers/feng-2020-codebert-pre-trained-model-programming/). arXiv preprint arXiv:2002.08155, 2020.
15. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+et+al%2E+The+pile%3A+An+800gb+dataset+of+diverse+text+for+language+)
16. Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang, et al. Pre-trained models: Past, present and future. AI Open, 2021. [find](https://scholar.google.com/scholar?q=Xu+Han%2C+Zhengyan+Zhang%2C+Ning+Ding%2C+Yuxian+Gu%2C+Xiao+Liu%2C+Yuqi+Huo%2C+Jiezhong+Qiu%2C+Liang+Zhang%2C+Wentao+Han%2C+Minlie+Huang%2C+et+al%2E+Pre%2Dtrained+models%3A+Past%2C+present+and+future%2E+AI+Open%2C+2021%2E)
17. Vincent J Hellendoorn and Premkumar Devanbu. Are deep neural networks the best choice for modeling source code? In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, pp. 763–773, 2017. [find](https://scholar.google.com/scholar?q=Vincent+J+Hellendoorn+and+Premkumar+Devanbu%2E+Are+deep+neural+networks+the+best+choice+for+modeling+source+code%3F+In+Proceedings+of+the+2017+11th+Joint+Meeting+on+Foundations+of+Software+Engineering%2C+pp)
18. Vincent J. Hellendoorn and Anand Ashok Sawant. The growing cost of deep learning for source code. Commun. ACM, 65(1):31–33, dec 2021. ISSN 0001-0782. doi: 10.1145/3501261. URL https://doi.org/10.1145/3501261. — [doi:10.1145/3501261](https://doi.org/10.1145/3501261) [↗](https://doi.org/10.1145/3501261)
19. Abram Hindle, Earl T Barr, Mark Gabel, Zhendong Su, and Premkumar Devanbu. On the naturalness of software. Communications of the ACM, 59(5):122–131, 2016. [find](https://scholar.google.com/scholar?q=Abram+Hindle%2C+Earl+T+Barr%2C+Mark+Gabel%2C+Zhendong+Su%2C+and+Premkumar+Devanbu%2E+On+the+naturalness+of+software%2E+Communications+of+the+ACM%2C+59%285%29%3A122%E2%80%93131%2C+2016%2E)
20. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. [The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/). arXiv preprint arXiv:1904.09751, 2019.
21. Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. [Codesearchnet challenge: Evaluating the state of semantic code search](https://md.research.elamur.ai/free/en/papers/husain-2020-codesearchnet-challenge-evaluating-state-semantic/). arXiv preprint arXiv:1909.09436, 2019.
22. Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning, pp. 5110–5121. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Aditya+Kanade%2C+Petros+Maniatis%2C+Gogul+Balakrishnan%2C+and+Kensen+Shi%2E+Learning+and+evaluating+contextual+embedding+of+source+code%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+5110%E2%80%935121%2E+PMLR%2C+2)
23. Rafael-Michael Karampatsis, Hlib Babii, Romain Robbes, Charles Sutton, and Andrea Janes. Big code!= big vocabulary: Open-vocabulary models for source code. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE), pp. 1073–1085. [find](https://scholar.google.com/scholar?q=Rafael%2DMichael+Karampatsis%2C+Hlib+Babii%2C+Romain+Robbes%2C+Charles+Sutton%2C+and+Andrea+Janes%2E+Big+code%21%3D+big+vocabulary%3A+Open%2Dvocabulary+models+for+source+code%2E+In+2020+IEEE%2FACM+42nd+International+Conferen)
24. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. [Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). arXiv preprint arXiv:1910.13461, 2019.
25. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, MING GONG, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie LIU. CodeXGLUE: A ma… [find](https://scholar.google.com/scholar?q=Shuai+Lu%2C+Daya+Guo%2C+Shuo+Ren%2C+Junjie+Huang%2C+Alexey+Svyatkovskiy%2C+Ambrosio+Blanco%2C+Colin+Clement%2C+Dawn+Drain%2C+Daxin+Jiang%2C+Duyu+Tang%2C+Ge+Li%2C+Lidong+Zhou%2C+Linjun+Shou%2C+Long+Zhou%2C+Michele+Tufano%2C+MING+GO)
26. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2C+2019%2E)
27. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). arXiv preprint arXiv:1910.10683, 2019.
28. Veselin Raychev, Martin Vechev, and Eran Yahav. Code completion with statistical language models. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 419–428, 2014. [find](https://scholar.google.com/scholar?q=Veselin+Raychev%2C+Martin+Vechev%2C+and+Eran+Yahav%2E+Code+completion+with+statistical+language+models%2E+In+Proceedings+of+the+35th+ACM+SIGPLAN+Conference+on+Programming+Language+Design+and+Implementation%2C+p)
29. Rico Sennrich, Barry Haddow, and Alexandra Birch. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). arXiv preprint arXiv:1508.07909, 2015.
30. Lewis Tunstall, Leandro von Werra, and Thomas Wolf. Natural Language Processing with Transformers. ” O’Reilly Media, Inc.”, 2022. [find](https://scholar.google.com/scholar?q=Lewis+Tunstall%2C+Leandro+von+Werra%2C+and+Thomas+Wolf%2E+Natural+Language+Processing+with+Transformers%2E+%E2%80%9D+O%E2%80%99Reilly+Media%2C+Inc%2E%E2%80%9D%2C+2022%2E)
31. Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [https://github.com/kingoflolz/mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax), May 2021. [↗](https://github.com/kingoflolz/mesh-transformer-jax)
32. Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859, 2021. [find](https://scholar.google.com/scholar?q=Yue+Wang%2C+Weishi+Wang%2C+Shafiq+Joty%2C+and+Steven+CH+Hoi%2E+Codet5%3A+Identifier%2Daware+unified+pre%2Dtrained+encoder%2Ddecoder+models+for+code+understanding+and+generation%2E+arXiv+preprint+arXiv%3A2109%2E00859%2C+2021%2E)
33. Daniel Zügner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan Günnemann. Language-agnostic representation learning of source code from structure and context. In International Conference on Learning Representations, 2021. URL [https://openreview.net/for](https://openreview.net/for) um?id=Xh5eMZVONGF. [↗](https://openreview.net/for)

*Updated 2 November 2026*