---
title: "SantaCoder: don't reach for the stars! — Allal 2023"
description: "Most code LLMs are built with little transparency: training data, preprocessing, and design choices are rarely disclosed."
authors: ["Loubna Ben Allal", "Raymond Li", "Denis Kocetkov", "Chenghao Mou", "Christopher Akiki", "Carlos Munoz Ferrandis", "Niklas Muennighoff", "Mayank Mishra", "Alex Gu", "Manan Dey", "Logesh Kumar Umapathi", "Carolyn Jane Anderson", "Yangtian Zi", "Joel Lamy Poirier", "Hailey Schoelkopf", "Sergey Troshin", "Dmitry Abulkhanov", "Manuel Romero", "Michael Lappert", "Francesco De Toni", "Bernardo García del Río", "Qian Liu", "Shamik Bose", "Urvashi Bhattacharyya", "Terry Yue Zhuo", "Ian Yu", "Paulo Villegas", "Marco Zocca", "Sourab Mangrulkar", "David Lansky", "Huu Nguyen", "Danish Contractor", "Luis Villa", "Jia Li", "Dzmitry Bahdanau", "Yacine Jernite", "Sean Hughes", "Daniel Fried", "Arjun Guha", "Harm de Vries", "Leandro von Werra"]
year: 2023
venue: "arXiv:2301.03988"
source: "https://arxiv.org/abs/2301.03988"
doi: "10.48550/arxiv.2301.03988"
arxiv: "2301.03988"
licence: "CC BY-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/allal-2023-santacoder-don-t-reach-stars/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/allal-2023-santacoder-don-t-reach-stars/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / SantaCoder: don't reach for the stars!

# SantaCoder: don't reach for the stars!

**Authors:** [Loubna Ben Allal](https://md.research.elamur.ai/free/en/authors/loubna-ben-allal/), [Raymond Li](https://md.research.elamur.ai/free/en/authors/raymond-li/), Denis Kocetkov, [Chenghao Mou](https://md.research.elamur.ai/free/en/authors/chenghao-mou/), [Christopher Akiki](https://md.research.elamur.ai/free/en/authors/christopher-akiki/), Carlos Munoz Ferrandis, [Niklas Muennighoff](https://md.research.elamur.ai/free/en/authors/niklas-muennighoff/), Mayank Mishra, [Alex Gu](https://md.research.elamur.ai/free/en/authors/alex-gu/), [Manan Dey](https://md.research.elamur.ai/free/en/authors/manan-dey/), [Logesh Kumar Umapathi](https://md.research.elamur.ai/free/en/authors/logesh-kumar-umapathi/), [Carolyn Jane Anderson](https://md.research.elamur.ai/free/en/authors/carolyn-jane-anderson/) et al. (41 authors) · **Year:** 2023 · **Venue:** arXiv:2301.03988 · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2301.03988) · [PDF](https://arxiv.org/pdf/2301.03988) · [DOI 10.48550/arxiv.2301.03988](https://doi.org/10.48550/arxiv.2301.03988) · [arXiv 2301.03988](https://arxiv.org/abs/2301.03988) · [OpenAlex](https://openalex.org/W4315706637)

**Citations:** 49 citations (openalex-snapshot · 2026-10-14) · [115](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary

Most code LLMs are built with little transparency: training data, preprocessing, and design choices are rarely disclosed. This technical report from the open BigCode collaboration documents its work through December 2022 toward responsibly built, openly released code models. The team built a first PII redaction pipeline for The Stack, targeting emails, IP addresses, and secret keys, and tested it on a hand-annotated benchmark. They then trained a series of 1.1B-parameter models on the Python, Java, and JavaScript portions of the PII-redacted The Stack. These runs tested two architectural options, Multi Query Attention (MQA) and Fill-in-the-Middle (FIM), and several data-filtering strategies, with evaluation on MultiPL-E. Stricter near-duplicate removal helped. Keeping only files from repositories with many GitHub stars, a common quality heuristic, clearly hurt results. The resulting model, SantaCoder, matches or beats the much larger InCoder-6.7B and CodeGen-Multi-2.7B on generation and infilling in all three languages. This shows that careful data curation can matter more than model scale.

## Key points

- **Opt-out and PII:** The Stack v1.1 (6.4 TB, 384 languages) honored 9 opt-out requests covering 299 repositories, of which 138 were removed. The PII benchmark contains 400 files drawn from a pre-filtered pool of 4,000 files in 11 languages, labeled on LightTag.
- **Architecture ablations:** Both MQA, which speeds inference at large batch sizes, and FIM, which enables infilling, caused only small drops in downstream scores.
- **Data filters tested:** The team tested four filters:
  - GitHub stars (5+)
  - comments-to-code ratio
  - stronger near-deduplication
  - character-to-token ratio (tokenizer fertility)

  Only the stars filter had a large effect, and it was negative.
- **Docstring generation:** On CodeXGLUE Python docstring generation (smoothed BLEU), the final SantaCoder scored 18.13, compared with 16.04 for InCoder-6.7B and 20.56 for CodeGen-Mono-16B.
- **Release:** All models are published under an OpenRAIL license at hf.co/bigcode.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

115 papers of our library cite it (of the 140884 whose bibliographies we read); 17 of them have a page here, 98 not yet.

- [The Falcon Series of Open Language Models](https://md.research.elamur.ai/free/en/papers/almazrouei-2023-falcon-series-open-language-models/)
- [Llemma: An Open Language Model For Mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/)
- [Qwen Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
- [DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence](https://md.research.elamur.ai/free/en/papers/deepseek-2024-ai-deepseek-coder-v2-breaking-barrier/)
- [CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution](https://md.research.elamur.ai/free/en/papers/gu-2024-cruxeval-benchmark-code-reasoning-understanding/)
- [Qwen2.5-Coder Technical Report](https://md.research.elamur.ai/free/en/papers/hui-2024-qwen2-5-coder-technical-report/)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/)
- [A Survey on Large Language Models for Code Generation](https://md.research.elamur.ai/free/en/papers/jiang-2024-survey-large-language-models-code/)
- [DataComp-LM: In search of the next generation of training sets for language models](https://md.research.elamur.ai/free/en/papers/li-2024-datacomp-lm-search-next-generation/)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/)
- [Crosslingual Generalization through Multitask Finetuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/)
- [MTEB: Massive Text Embedding Benchmark](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-mteb-massive-text-embedding-benchmark/)
- [Generative Representational Instruction Tuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2024-generative-representational-instruction-tuning/)
- [Code Llama: Open Foundation Models for Code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://md.research.elamur.ai/free/en/papers/ustun-2024-aya-model-instruction-finetuned-open/)
- [C-Pack: Packed Resources For General Chinese Embeddings](https://md.research.elamur.ai/free/en/papers/xiao-2023-c-pack-packed-resources/)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://md.research.elamur.ai/free/en/papers/zhuo-2024-bigcodebench-benchmarking-code-generation-diverse/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{allal2023santacoder,
  title = {{SantaCoder: don't reach for the stars!}},
  author = {Allal, Loubna Ben and Li, Raymond and Kocetkov, Denis and Mou, Chenghao and Akiki, Christopher and Ferrandis, Carlos Munoz and Muennighoff, Niklas and Mishra, Mayank and Gu, Alex and Dey, Manan and Umapathi, Logesh Kumar and Anderson, Carolyn Jane and Zi, Yangtian and Poirier, Joel Lamy and Schoelkopf, Hailey and Troshin, Sergey and Abulkhanov, Dmitry and Romero, Manuel and Lappert, Michael and Toni, Francesco De and Río, Bernardo García del and Liu, Qian and Bose, Shamik and Bhattacharyya, Urvashi and Zhuo, Terry Yue and Yu, Ian and Villegas, Paulo and Zocca, Marco and Mangrulkar, Sourab and Lansky, David and Nguyen, Huu and Contractor, Danish and Villa, Luis and Li, Jia and Bahdanau, Dzmitry and Jernite, Yacine and Hughes, Sean and Fried, Daniel and Guha, Arjun and Vries, Harm de and Werra, Leandro von},
  year = {2023},
  doi = {10.48550/arxiv.2301.03988},
  eprint = {2301.03988},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2301.03988}
}
```

- [← previous SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://md.research.elamur.ai/free/en/papers/tang-2023-salmonn-towards-generic-hearing-abilities/)
- [next → Scalable Extraction of Training Data from (Production) Language Models](https://md.research.elamur.ai/free/en/papers/nasr-2023-scalable-extraction-training-data-production/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

44 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Unified pre-training for program understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2655–2668, Online, J… [find](https://scholar.google.com/scholar?q=Wasi+Ahmad%2C+Saikat+Chakraborty%2C+Baishakhi+Ray%2C+and+Kai%2DWei+Chang%2E+Unified+pre%2Dtraining+for+program+understanding+and+generation%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+) [our card](https://md.research.elamur.ai/free/en/papers/ahmad-2021-unified-pre-training-program-understanding/)
2. Toufique Ahmed and Premkumar Devanbu. Multilingual training for software engineering. In Proceedings of the 44th International Conference on Software Engineering. ACM, 2022. doi: 10.1145/3510003.3510049. — [doi:10.1145/3510003.3510049](https://doi.org/10.1145/3510003.3510049) [find](https://scholar.google.com/scholar?q=Toufique+Ahmed+and+Premkumar+Devanbu%2E+Multilingual+training+for+software+engineering%2E+In+Proceedings+of+the+44th+International+Conference+on+Software+Engineering%2E+ACM%2C+2022%2E+doi%3A+10%2E1145%2F3510003%2E35100)
3. Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan… [find](https://scholar.google.com/scholar?q=Ben+Athiwaratkun%2C+Sanjay+Krishna+Gouda%2C+Zijian+Wang%2C+Xiaopeng+Li%2C+Yuchen+Tian%2C+Ming+Tan%2C+Wasi+Uddin+Ahmad%2C+Shiqi+Wang%2C+Qing+Sun%2C+Mingyue+Shang%2C+Sujan+Kumar+Gonugondla%2C+Hantian+Ding%2C+Varun+Kumar%2C+Natha)
4. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. [find](https://scholar.google.com/scholar?q=Jacob+Austin%2C+Augustus+Odena%2C+Maxwell+Nye%2C+Maarten+Bosma%2C+Henryk+Michalewski%2C+David+Dohan%2C+Ellen+Jiang%2C+Carrie+Cai%2C+Michael+Terry%2C+Quoc+Le%2C+et+al%2E+Program+synthesis+with+large+language+models%2E+arXiv+p) [our card](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/)
5. Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle, 2022. URL https://arxiv.org/abs/2207.14255. [↗](https://arxiv.org/abs/2207.14255) [our card](https://md.research.elamur.ai/free/en/papers/bavarian-2022-efficient-training-language-models-fill/)
6. Loubna Ben Allal, Niklas Muennighoff, and Leandro Von Werra. A framework for the evaluation of code generation models. [https://github.com/bigcode-project/](https://github.com/bigcode-project/) bigcode-evaluation-harness, December 2022. [↗](https://github.com/bigcode-project/)
7. Andrei Z Broder. Identifying and filtering near-duplicate documents. In Annual symposium on combinatorial pattern matching, pp. 1–10. Springer, 2000. [find](https://scholar.google.com/scholar?q=Andrei+Z+Broder%2E+Identifying+and+filtering+near%2Dduplicate+documents%2E+In+Annual+symposium+on+combinatorial+pattern+matching%2C+pp%2E+1%E2%80%9310%2E+Springer%2C+2000%2E)
8. Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. A scalable and extensible approach to benchmarking nl2code for 18 programming languag… [find](https://scholar.google.com/scholar?q=Federico+Cassano%2C+John+Gouwar%2C+Daniel+Nguyen%2C+Sydney+Nguyen%2C+Luna+Phipps%2DCostin%2C+Donald+Pinckney%2C+Ming%2DHo+Yee%2C+Yangtian+Zi%2C+Carolyn+Jane+Anderson%2C+Molly+Q+Feldman%2C+Arjun+Guha%2C+Michael+Greenberg%2C+and+A)
9. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… [find](https://scholar.google.com/scholar?q=Mark+Chen%2C+Jerry+Tworek%2C+Heewoo+Jun%2C+Qiming+Yuan%2C+Henrique+Ponde+de+Oliveira+Pinto%2C+Jared+Kaplan%2C+Harri+Edwards%2C+Yuri+Burda%2C+Nicholas+Joseph%2C+Greg+Brockman%2C+Alex+Ray%2C+Raul+Puri%2C+Gretchen+Krueger%2C+Mich) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
10. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… [find](https://scholar.google.com/scholar?q=Aakanksha+Chowdhery%2C+Sharan+Narang%2C+Jacob+Devlin%2C+Maarten+Bosma%2C+Gaurav+Mishra%2C+Adam+Roberts%2C+Paul+Barham%2C+Hyung+Won+Chung%2C+Charles+Sutton%2C+Sebastian+Gehrmann%2C+Parker+Schuh%2C+Kensen+Shi%2C+Sasha+Tsvyashc)
11. Fenia Christopoulou, Gerasimos Lampouras, Milan Gritta, Guchun Zhang, Yinpeng Guo, Zhongqi Li, Qi Zhang, Meng Xiao, Bo Shen, Lin Li, Hao Yu, Li Yan, Pingyi Zhou, Xin Wang, Yuchi Ma, Ignacio Iacobacci, Yasheng Wang, Guangtai Liang, Jiansheng Wei, Xin Jiang, Qianxiang Wang, and Qun Liu. Pangu-coder: P… [find](https://scholar.google.com/scholar?q=Fenia+Christopoulou%2C+Gerasimos+Lampouras%2C+Milan+Gritta%2C+Guchun+Zhang%2C+Yinpeng+Guo%2C+Zhongqi+Li%2C+Qi+Zhang%2C+Meng+Xiao%2C+Bo+Shen%2C+Lin+Li%2C+Hao+Yu%2C+Li+Yan%2C+Pingyi+Zhou%2C+Xin+Wang%2C+Yuchi+Ma%2C+Ignacio+Iacobacci%2C)
12. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1536–1547, Online,… [find](https://scholar.google.com/scholar?q=Zhangyin+Feng%2C+Daya+Guo%2C+Duyu+Tang%2C+Nan+Duan%2C+Xiaocheng+Feng%2C+Ming+Gong%2C+Linjun+Shou%2C+Bing+Qin%2C+Ting+Liu%2C+Daxin+Jiang%2C+and+Ming+Zhou%2E+CodeBERT%3A+A+pre%2Dtrained+model+for+programming+and+natural+language) [our card](https://md.research.elamur.ai/free/en/papers/feng-2020-codebert-pre-trained-model-programming/)
13. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155, 2020b. doi: 10.48550/ARXIV.2002.08155. URL [https://arxiv.org/abs…](https://arxiv.org/abs…) [↗](https://arxiv.org/abs…) [our card](https://md.research.elamur.ai/free/en/papers/feng-2020-codebert-pre-trained-model-programming/)
14. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis, 2022. URL https://arxiv.org/abs/2204.05999. [↗](https://arxiv.org/abs/2204.05999)
15. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB dataset of diverse text for language modeling, 2020. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+Shawn+Presser%2C+and+Connor+Leahy%2E+The+Pile%3A+An+800GB+dataset+of+)
16. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September… — [doi:10.5281/zenodo](https://doi.org/10.5281/zenodo) [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Jason+Phang%2C+Laria+Reynolds%2C+Eric+Tang%2C+Anish+Thite)
17. Vincent J. Hellendoorn, Christian Bird, Earl T. Barr, and Miltiadis Allamanis. Deep Learning Type Inference. In Fse, 2018. [find](https://scholar.google.com/scholar?q=Vincent+J%2E+Hellendoorn%2C+Christian+Bird%2C+Earl+T%2E+Barr%2C+and+Miltiadis+Allamanis%2E+Deep+Learning+Type+Inference%2E+In+Fse%2C+2018%2E)
18. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. arXiv preprint arXiv:2105.09938, 2021. doi: 10.48550/ARXIV.2105. 09938. URL [https://arxiv.or…](https://arxiv.or…) [↗](https://arxiv.or…) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/)
19. Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436, 2019. [find](https://scholar.google.com/scholar?q=Hamel+Husain%2C+Ho%2DHsiang+Wu%2C+Tiferet+Gazit%2C+Miltiadis+Allamanis%2C+and+Marc+Brockschmidt%2E+CodeSearchNet+challenge%3A+Evaluating+the+state+of+semantic+code+search%2E+arXiv+preprint+arXiv%3A1909%2E09436%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/husain-2020-codesearchnet-challenge-evaluating-state-semantic/)
20. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Mapping language to code in programmatic context. arXiv preprint arXiv:1808.09588, 2018. [find](https://scholar.google.com/scholar?q=Srinivasan+Iyer%2C+Ioannis+Konstas%2C+Alvin+Cheung%2C+and+Luke+Zettlemoyer%2E+Mapping+language+to+code+in+programmatic+context%2E+arXiv+preprint+arXiv%3A1808%2E09588%2C+2018%2E)
21. Maliheh Izadi, Roberta Gismondi, and Georgios Gousios. Codefill: Multi-token code completion by jointly learning from structure and naming sequences. In Proceedings of the 44th International Conference on Software Engineering, ICSE ’22, pp. 401–412, New York, NY, USA, 2022. Association for Computing… — [doi:10.1145/3510003.3510172](https://doi.org/10.1145/3510003.3510172) [find](https://scholar.google.com/scholar?q=Maliheh+Izadi%2C+Roberta+Gismondi%2C+and+Georgios+Gousios%2E+Codefill%3A+Multi%2Dtoken+code+completion+by+jointly+learning+from+structure+and+naming+sequences%2E+In+Proceedings+of+the+44th+International+Conferenc)
22. Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020. [find](https://scholar.google.com/scholar?q=Aditya+Kanade%2C+Petros+Maniatis%2C+Gogul+Balakrishnan%2C+and+Kensen+Shi%2E+Learning+and+evaluating+contextual+embedding+of+source+code%2E+In+Proceedings+of+the+37th+International+Conference+on+Machine+Learning)
23. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980. [↗](http://arxiv.org/abs/1412.6980)
24. Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. The Stack: 3 TB of permissively licensed source code. Preprint, 2022. [find](https://scholar.google.com/scholar?q=Denis+Kocetkov%2C+Raymond+Li%2C+Loubna+Ben+Allal%2C+Jia+Li%2C+Chenghao+Mou%2C+Carlos+Mun%CC%83oz+Ferrandis%2C+Yacine+Jernite%2C+Margaret+Mitchell%2C+Sean+Hughes%2C+Thomas+Wolf%2C+Dzmitry+Bahdanau%2C+Leandro+von+Werra%2C+and+Harm+)
25. Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. Spoc: Search-based pseudocode to code. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curr… [find](https://scholar.google.com/scholar?q=Sumith+Kulal%2C+Panupong+Pasupat%2C+Kartik+Chandra%2C+Mina+Lee%2C+Oded+Padon%2C+Alex+Aiken%2C+and+Percy+S+Liang%2E+Spoc%3A+Search%2Dbased+pseudocode+to+code%2E+In+H%2E+Wallach%2C+H%2E+Larochelle%2C+A%2E+Beygelzimer%2C+F%2E+d%27Alche%CC%81%2DBu)
26. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. ArXiv, abs/2211.11501, 2022. [find](https://scholar.google.com/scholar?q=Yuhang+Lai%2C+Chengxi+Li%2C+Yiming+Wang%2C+Tianyi+Zhang%2C+Ruiqi+Zhong%2C+Luke+Zettlemoyer%2C+Scott+Wen+tau+Yih%2C+Daniel+Fried%2C+Sida+Wang%2C+and+Tao+Yu%2E+Ds%2D1000%3A+A+natural+and+reliable+benchmark+for+data+science+cod)
27. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James… [find](https://scholar.google.com/scholar?q=Yujia+Li%2C+David+Choi%2C+Junyoung+Chung%2C+Nate+Kushman%2C+Julian+Schrittwieser%2C+Re%CC%81mi+Leblond%2C+Tom+Eccles%2C+James+Keeling%2C+Felix+Gimeno%2C+Agustin+Dal+Lago%2C+Thomas+Hubert%2C+Peter+Choy%2C+Cyprien+de+Masson+d%E2%80%99Autum) [our card](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/)
28. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664, 2021. [find](https://scholar.google.com/scholar?q=Shuai+Lu%2C+Daya+Guo%2C+Shuo+Ren%2C+Junjie+Huang%2C+Alexey+Svyatkovskiy%2C+Ambrosio+Blanco%2C+Colin+Clement%2C+Dawn+Drain%2C+Daxin+Jiang%2C+Duyu+Tang%2C+et+al%2E+Codexglue%3A+A+machine+learning+benchmark+dataset+for+code+und)
29. Anthony MOI, Nicolas Patry, Pierric Cistac, Pete, Funtowicz Morgan, Sebastian Pütz, Mishig, Bjarte Johansen, Thomas Wolf, Sylvain Gugger, Clement, Julien Chaumond, Lysandre Debut, François Garillot, Luc Georges, dctelus, JC Louis, MarcusGrass, Taufiquzzaman Peyash, 0xflotus, Alan deLevie, Alexande… — [doi:10.5281/zenodo.7298413](https://doi.org/10.5281/zenodo.7298413) [find](https://scholar.google.com/scholar?q=Anthony+MOI%2C+Nicolas+Patry%2C+Pierric+Cistac%2C+Pete%2C+Funtowicz+Morgan%2C+Sebastian+Pu%CC%88tz%2C+Mishig%2C+Bjarte+Johansen%2C+Thomas+Wolf%2C+Sylvain+Gugger%2C+Clement%2C+Julien+Chaumond%2C+Lysandre+Debut%2C+Franc%CC%A7ois+Garillot%2C)
30. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. A conversational paradigm for program synthesis. arXiv preprint, 2022. [find](https://scholar.google.com/scholar?q=Erik+Nijkamp%2C+Bo+Pang%2C+Hiroaki+Hayashi%2C+Lifu+Tu%2C+Huan+Wang%2C+Yingbo+Zhou%2C+Silvio+Savarese%2C+and+Caiming+Xiong%2E+A+conversational+paradigm+for+program+synthesis%2E+arXiv+preprint%2C+2022%2E)
31. Michael Pradel, Georgios Gousios, Jason Liu, and Satish Chandra. TypeWriter: Neural Type Prediction with Search-Based Validation. In Esecfse, 2020. [find](https://scholar.google.com/scholar?q=Michael+Pradel%2C+Georgios+Gousios%2C+Jason+Liu%2C+and+Satish+Chandra%2E+TypeWriter%3A+Neural+Type+Prediction+with+Search%2DBased+Validation%2E+In+Esecfse%2C+2020%2E)
32. Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis, 2020. URL https://arxiv.org/abs/2009.10297. — [arXiv:2009.10297](https://arxiv.org/abs/2009.10297) [↗](https://arxiv.org/abs/2009.10297)
33. Baptiste Roziere, Marie-Anne Lachaux, Marc Szafraniec, and Guillaume Lample. Dobf: A deobfuscation pre-training objective for programming languages. arXiv preprint arXiv:2102.07492, 2021. [find](https://scholar.google.com/scholar?q=Baptiste+Roziere%2C+Marie%2DAnne+Lachaux%2C+Marc+Szafraniec%2C+and+Guillaume+Lample%2E+Dobf%3A+A+deobfuscation+pre%2Dtraining+objective+for+programming+languages%2E+arXiv+preprint+arXiv%3A2102%2E07492%2C+2021%2E)
34. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conf… [find](https://scholar.google.com/scholar?q=Phillip+Rust%2C+Jonas+Pfeiffer%2C+Ivan+Vulic%CC%81%2C+Sebastian+Ruder%2C+and+Iryna+Gurevych%2E+How+good+is+your+tokenizer%3F+on+the+monolingual+performance+of+multilingual+language+models%2E+In+Proceedings+of+the+59th+A) [our card](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/)
35. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagne%CC%81%2C+Alexandra+Sasha+Luccioni%2C+Franc%CC%A7ois+Yvon%2C+Matthias+Galle%CC%81%2C+et+al%2E+Bloom%3A+A+176b%2Dparameter+op)
36. Noam Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150. [↗](http://arxiv.org/abs/1911.02150)
37. Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. Unit test case generation with transformers and focal context. arXiv preprint arXiv:2009.10297, 2020. doi: 10.48550/ARXIV.2009.05617. URL https://arxiv.org/abs/2009.05617. — [doi:10.48550/arxiv.2009.05617](https://doi.org/10.48550/arxiv.2009.05617) [↗](https://arxiv.org/abs/2009.05617)
38. Lewis Tunstall, Leandro Von Werra, and Thomas Wolf. Natural language processing with transformers, revised edition. O’Reilly Media, Sebastopol, CA, June 2022. [find](https://scholar.google.com/scholar?q=Lewis+Tunstall%2C+Leandro+Von+Werra%2C+and+Thomas+Wolf%2E+Natural+language+processing+with+transformers%2C+revised+edition%2E+O%E2%80%99Reilly+Media%2C+Sebastopol%2C+CA%2C+June+2022%2E)
39. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998–6008, 2017. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+Attention+is+all+you+need%2E+In+Advances+in+Neural+Information+Processing+Sys)
40. Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8696–8708, Online and Punta Cana, Dominic… [find](https://scholar.google.com/scholar?q=Yue+Wang%2C+Weishi+Wang%2C+Shafiq+Joty%2C+and+Steven+C%2EH%2E+Hoi%2E+CodeT5%3A+Identifier%2Daware+unified+pre%2Dtrained+encoder%2Ddecoder+models+for+code+understanding+and+generation%2E+In+Proceedings+of+the+2021+Conferenc)
41. Frank F. Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, MAPS 2022, pp. 1–10, New York, NY, USA, 2022a. Association for Computing Machinery. ISB… [find](https://scholar.google.com/scholar?q=Frank+F%2E+Xu%2C+Uri+Alon%2C+Graham+Neubig%2C+and+Vincent+Josua+Hellendoorn%2E+A+systematic+evaluation+of+large+language+models+of+code%2E+In+Proceedings+of+the+6th+ACM+SIGPLAN+International+Symposium+on+Machine+)
42. Frank F. Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, MAPS 2022, pp. 1–10, New York, NY, USA, 2022b. Association for Computing Machinery. ISB… [find](https://scholar.google.com/scholar?q=Frank+F%2E+Xu%2C+Uri+Alon%2C+Graham+Neubig%2C+and+Vincent+Josua+Hellendoorn%2E+A+systematic+evaluation+of+large+language+models+of+code%2E+In+Proceedings+of+the+6th+ACM+SIGPLAN+International+Symposium+on+Machine+)
43. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task, 2018. URL https://arxiv.org/abs/1… [↗](https://arxiv.org/abs/1…) [our card](https://md.research.elamur.ai/free/en/papers/yu-2018-spider-large-scale-human-labeled/)
44. Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning, 2017. URL https://arxiv.org/abs/ 1709.00103. [↗](https://arxiv.org/abs/)