---
title: "CodeS — paper by Zan et al. (2024) — summary and ideas"
description: "The paper introduces NL2Repo, a task in which a model must build a whole software repository from a… 69 references, cited by 7 papers here."
authors: ["Daoguang Zan", "Ailun Yu", "Wei Liu", "Dong Chen", "Bo Shen", "Wei Li", "Yafen Yao", "Yongshun Gong", "Xiaolin Chen", "Bei Guan", "Zhiguang Yang", "Yongji Wang", "Qianxiang Wang", "Lizhen Cui"]
year: 2024
venue: "arXiv:2403.16443"
source: "https://arxiv.org/abs/2403.16443"
doi: "10.48550/arxiv.2403.16443"
arxiv: "2403.16443"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zan-2024-codes-natural-language-code-repository/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zan-2024-codes-natural-language-code-repository/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/) / CodeS: Natural Language to Code Repository via Multi-Layer Sketch

# CodeS: Natural Language to Code Repository via Multi-Layer Sketch

**Authors:** [Daoguang Zan](https://md.research.elamur.ai/free/en/authors/daoguang-zan/), Ailun Yu, [Wei Liu](https://md.research.elamur.ai/free/en/authors/wei-liu/), [Dong Chen](https://md.research.elamur.ai/free/en/authors/dong-chen/), [Bo Shen](https://md.research.elamur.ai/free/en/authors/bo-shen/), [Wei Li](https://md.research.elamur.ai/free/en/authors/wei-li/), Yafen Yao, [Yongshun Gong](https://md.research.elamur.ai/free/en/authors/yongshun-gong/), Xiaolin Chen, Bei Guan, Zhiguang Yang, Yongji Wang, Qianxiang Wang and Lizhen Cui · **Year:** 2024 · **Venue:** arXiv:2403.16443 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2403.16443) · [PDF](https://arxiv.org/pdf/2403.16443) · [DOI 10.48550/arxiv.2403.16443](https://doi.org/10.48550/arxiv.2403.16443) · [arXiv 2403.16443](https://arxiv.org/abs/2403.16443) · [OpenAlex](https://openalex.org/W4393213295)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/zan-2024-codes-natural-language-code-repository/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper introduces NL2Repo, a task in which a model must build a whole software repository from a natural-language requirements document, here a project's README.md. These inputs are long and high-level. The output must stay structurally consistent across folders, files and interfaces, which autoregressive code LLMs struggle with when generating everything in one pass. The authors propose CodeS, which splits the job into three layers. RepoSketcher drafts the directory tree with import hints. FileSketcher writes a skeleton for each file, containing imports and function definitions with empty bodies. SketchFiller then completes each function body. Every stage receives the outputs of the earlier ones. CodeS is implemented in two ways: by prompting existing models (StarCoder2, CodeLlama, DeepSeekCoder, GPT-3.5) and by supervised fine-tuning. For evaluation, the authors build SketchEval, a benchmark of 19 recent GitHub repositories, and SketchBLEU, a repository-level adaptation of CodeBLEU. They also release a VSCode plugin, which 30 participants tested. Benchmark results and user studies indicate that CodeS is effective and practical, and the work makes repository-scale generation measurable.

## Key points
- **SketchEval and SketchBLEU.** SketchEval splits its 19 repositories into Easy (5), Middle (8) and Hard (6). SketchBLEU changes CodeBLEU in four ways:
  - BLEU is computed over all files concatenated.
  - The brevity penalty is softer.
  - A Match_struc term compares n-hop subtrees of a combined directory-plus-syntax tree.
  - Dataflow matching works at repository level via maximum weighted bipartite matching of functions.
- **Fine-tuned CodeS results.** With CodeLlama, overall SketchBLEU grows with model size: 54.39 (7B), 57.23 (13B) and 63.31 (34B). Generating files in dependency order gives small extra gains, such as 63.87 for 34B, mostly on Hard repositories.
- **Transfer to other languages.** A RepoSketcher fine-tuned only on Python (CodeLlama 13B) beats prompt engineering on other languages. Repository-sketch BLEU rises from 46.25 to 57.36 for Java, 36.68 to 50.14 for C++, and 62.88 to 71.57 for Go.
- **User studies.** 30 non-author participants built two real projects, one of them a Blog app, using the plugin. In a survey of 6 users of CodeS Extended, most rated it effective. One rated it ineffective, finding its product design weaker than GitHub Copilot.
- **Availability.** Code, the evaluation scripts and the multilingual evaluation sets are public at github.com/NL2Code/CodeS.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 244 works · 39,359 citations
- [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) — Government body, Beijing, China · 194 works · 23,906 citations
- [Huawei Technologies](https://md.research.elamur.ai/free/en/companies/huawei-technologies/) — Company, Shenzhen, China · 75 works · 12,975 citations
- [Shandong University](https://md.research.elamur.ai/free/en/universities/shandong-university/) — University, Jinan, China · 17 works · 789 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

## Cite

Cite as: Daoguang Zan, Ailun Yu, Wei Liu, Dong Chen, Bo Shen, Wei Li, Yafen Yao, Yongshun Gong, Xiaolin Chen, Bei Guan, Zhiguang Yang, Yongji Wang, Qianxiang Wang and Lizhen Cui. 2024. CodeS: Natural Language to Code Repository via Multi-Layer Sketch. arXiv:2403.16443.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zan2024codes,
  title = {{CodeS: Natural Language to Code Repository via Multi-Layer Sketch}},
  author = {Daoguang Zan, Ailun Yu, Wei Liu, Dong Chen, Bo Shen, Wei Li, Yafen Yao, Yongshun Gong, Xiaolin Chen, Bei Guan, Zhiguang Yang, Yongji Wang, Qianxiang Wang and Lizhen Cui},
  year = {2024},
  doi = {10.48550/arxiv.2403.16443},
  eprint = {2403.16443},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.16443}
}
```

- [← previous ArzEn-MultiGenre: An aligned parallel dataset of Egyptian Arabic song lyrics, novels, and subtitles, with English translations](https://md.research.elamur.ai/free/en/papers/al-sabbagh-2025-arzen-multigenre-aligned-parallel-dataset/)
- [next → Disentangling Singlish Discourse Particles with Task-Driven Representation](https://md.research.elamur.ai/free/en/papers/foo-2024-disentangling-singlish-discourse-particles-task/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/)

## References

69 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Zan D, Chen B, Zhang F, et al. Large language models meet nl2code: A survey[C]//Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023: 7443-7464. [find](https://scholar.google.com/scholar?q=Zan+D%2C+Chen+B%2C+Zhang+F%2C+et+al%2E+Large+language+models+meet+nl2code%3A+A+survey%5BC%5D%2F%2FProceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2E+2023%3A+7443%2D7464%2E)
2. Chen M, Tworek J, Jun H, et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)[J]. arXiv preprint arXiv:2107.03374, 2021.
3. Li Y, Choi D, Chung J, et al. [Competition-level code generation with alphacode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/)[J]. Science, 2022, 378(6624): 1092-1097.
4. Nijkamp E, Pang B, Hayashi H, et al. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis[C]//The Eleventh International Conference on Learning Representations. 2022. [find](https://scholar.google.com/scholar?q=Nijkamp+E%2C+Pang+B%2C+Hayashi+H%2C+et+al%2E+CodeGen%3A+An+Open+Large+Language+Model+for+Code+with+Multi%2DTurn+Program+Synthesis%5BC%5D%2F%2FThe+Eleventh+International+Conference+on+Learning+Representations%2E+2022%2E)
5. Li R, Allal L B, Zi Y, et al. Starcoder: may the source be with you![J]. arXiv preprint arXiv:2305.06161, 2023. [find](https://scholar.google.com/scholar?q=Li+R%2C+Allal+L+B%2C+Zi+Y%2C+et+al%2E+Starcoder%3A+may+the+source+be+with+you%21%5BJ%5D%2E+arXiv+preprint+arXiv%3A2305%2E06161%2C+2023%2E)
6. Lozhkov A, Li R, Allal L B, et al. StarCoder 2 and The Stack v2: The Next Generation[J]. arXiv preprint arXiv:2402.19173, 2024. [find](https://scholar.google.com/scholar?q=Lozhkov+A%2C+Li+R%2C+Allal+L+B%2C+et+al%2E+StarCoder+2+and+The+Stack+v2%3A+The+Next+Generation%5BJ%5D%2E+arXiv+preprint+arXiv%3A2402%2E19173%2C+2024%2E)
7. Roziere B, Gehring J, Gloeckle F, et al. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)[J]. arXiv preprint arXiv:2308.12950, 2023.
8. Guo D, Zhu Q, Yang D, et al. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence[J]. arXiv preprint arXiv:2401.14196, 2024. [find](https://scholar.google.com/scholar?q=Guo+D%2C+Zhu+Q%2C+Yang+D%2C+et+al%2E+DeepSeek%2DCoder%3A+When+the+Large+Language+Model+Meets+Programming%E2%80%93The+Rise+of+Code+Intelligence%5BJ%5D%2E+arXiv+preprint+arXiv%3A2401%2E14196%2C+2024%2E)
9. Achiam J, Adler S, Agarwal S, et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)[J]. arXiv preprint arXiv:2303.08774, 2023.
10. Anthropic. The Claude 3 Model Family: Opus, Sonnet, Haiku. [find](https://scholar.google.com/scholar?q=Anthropic%2E+The+Claude+3+Model+Family%3A+Opus%2C+Sonnet%2C+Haiku%2E)
11. Ridnik T, Kredo D, Friedman I. Code Generation with Alpha-Codium: From Prompt Engineering to Flow Engineering. arXiv preprint arXiv:2401.08500. 2024 Jan 16. [find](https://scholar.google.com/scholar?q=Ridnik+T%2C+Kredo+D%2C+Friedman+I%2E+Code+Generation+with+Alpha%2DCodium%3A+From+Prompt+Engineering+to+Flow+Engineering%2E+arXiv+preprint+arXiv%3A2401%2E08500%2E+2024+Jan+16%2E)
12. Liu T, Jiang Y E, Monath N, et al. Autoregressive Structured Prediction with Language Models[C]//Findings of the Association for Computational Linguistics: EMNLP 2022. 2022: 993-1005. [find](https://scholar.google.com/scholar?q=Liu+T%2C+Jiang+Y+E%2C+Monath+N%2C+et+al%2E+Autoregressive+Structured+Prediction+with+Language+Models%5BC%5D%2F%2FFindings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2022%2E+2022%3A+993%2D1005%2E)
13. Wells C. A generalization of the concept of sketch[J]. Theoretical Computer Science, 1990, 70(1): 159-178. [find](https://scholar.google.com/scholar?q=Wells+C%2E+A+generalization+of+the+concept+of+sketch%5BJ%5D%2E+Theoretical+Computer+Science%2C+1990%2C+70%281%29%3A+159%2D178%2E)
14. Tovey M, Porter S, Newman R. Sketching, concept development and automotive design[J]. Design studies, 2003, 24(2): 135-153. [find](https://scholar.google.com/scholar?q=Tovey+M%2C+Porter+S%2C+Newman+R%2E+Sketching%2C+concept+development+and+automotive+design%5BJ%5D%2E+Design+studies%2C+2003%2C+24%282%29%3A+135%2D153%2E)
15. Zan D, Chen B, Yang D, et al. CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation[J]. Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, 2022: 2369-2375. [find](https://scholar.google.com/scholar?q=Zan+D%2C+Chen+B%2C+Yang+D%2C+et+al%2E+CERT%3A+Continual+Pre%2DTraining+on+Sketches+for+Library%2DOriented+Code+Generation%5BJ%5D%2E+Proceedings+of+the+Thirty%2DFirst+International+Joint+Conference+on+Artificial+Intelligenc)
16. Solar-Lezama A. Program synthesis by sketching[M]. University of California, Berkeley, 2008. [find](https://scholar.google.com/scholar?q=Solar%2DLezama+A%2E+Program+synthesis+by+sketching%5BM%5D%2E+University+of+California%2C+Berkeley%2C+2008%2E)
17. Ellis K, Wong C, Nye M, Sablé-Meyer M, Morales L, Hewitt L, Cary L, Solar-Lezama A, Tenenbaum JB. Dreamcoder: Bootstrapping inductive program synthesis with wake-sleep library learning. InProceedings of the 42nd acm sigplan international conference on programming language design and implementation… [find](https://scholar.google.com/scholar?q=Ellis+K%2C+Wong+C%2C+Nye+M%2C+Sable%CC%81%2DMeyer+M%2C+Morales+L%2C+Hewitt+L%2C+Cary+L%2C+Solar%2DLezama+A%2C+Tenenbaum+JB%2E+Dreamcoder%3A+Bootstrapping+inductive+program+synthesis+with+wake%2Dsleep+library+learning%2E+InProceedings)
18. Levinson R. A general programming language for unified planning and control[J]. Artificial Intelligence, 1995, 76(1-2): 319-375. [find](https://scholar.google.com/scholar?q=Levinson+R%2E+A+general+programming+language+for+unified+planning+and+control%5BJ%5D%2E+Artificial+Intelligence%2C+1995%2C+76%281%2D2%29%3A+319%2D375%2E)
19. Mou L, Li G, Zhang L, et al. Convolutional neural networks over tree structures for programming language processing[C]//Proceedings of the AAAI conference on artificial intelligence. 2016, 30(1). [find](https://scholar.google.com/scholar?q=Mou+L%2C+Li+G%2C+Zhang+L%2C+et+al%2E+Convolutional+neural+networks+over+tree+structures+for+programming+language+processing%5BC%5D%2F%2FProceedings+of+the+AAAI+conference+on+artificial+intelligence%2E+2016%2C+30%281%29%2E)
20. Neelakantan A, Le Q V, Abadi M, et al. Learning a Natural Language Interface with Neural Programmer[C]//International Conference on Learning Representations. 2016. [find](https://scholar.google.com/scholar?q=Neelakantan+A%2C+Le+Q+V%2C+Abadi+M%2C+et+al%2E+Learning+a+Natural+Language+Interface+with+Neural+Programmer%5BC%5D%2F%2FInternational+Conference+on+Learning+Representations%2E+2016%2E)
21. Ouyang L, Wu J, Jiang X, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)[J]. Advances in neural information processing systems, 2022, 35: 27730-27744.
22. Lu S, Guo D, Ren S, et al. CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation[C]//Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). 2021. [find](https://scholar.google.com/scholar?q=Lu+S%2C+Guo+D%2C+Ren+S%2C+et+al%2E+CodeXGLUE%3A+A+Machine+Learning+Benchmark+Dataset+for+Code+Understanding+and+Generation%5BC%5D%2F%2FThirty%2Dfifth+Conference+on+Neural+Information+Processing+Systems+Datasets+and+Bench)
23. Du X, Liu M, Wang K, et al. Evaluating Large Language Models in Class-Level Code Generation[C]//2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE). IEEE Computer Society, 2024: 865-865. [find](https://scholar.google.com/scholar?q=Du+X%2C+Liu+M%2C+Wang+K%2C+et+al%2E+Evaluating+Large+Language+Models+in+Class%2DLevel+Code+Generation%5BC%5D%2F%2F2024+IEEE%2FACM+46th+International+Conference+on+Software+Engineering+%28ICSE%29%2E+IEEE+Computer+Society%2C+2024%3A)
24. Qian C, Cong X, Yang C, et al. Communicative agents for software development[J]. arXiv preprint arXiv:2307.[07924, 2023](https://md.research.elamur.ai/free/en/papers/qian-2023-chatdev-communicative-agents-software/).
25. Yang H, Yue S, He Y. Auto-gpt for online decision making: Benchmarks and additional opinions[J]. arXiv preprint arXiv:2306.02224, 2023. [find](https://scholar.google.com/scholar?q=Yang+H%2C+Yue+S%2C+He+Y%2E+Auto%2Dgpt+for+online+decision+making%3A+Benchmarks+and+additional+opinions%5BJ%5D%2E+arXiv+preprint+arXiv%3A2306%2E02224%2C+2023%2E)
26. [https://github.com/reworkd/AgentGPT](https://github.com/reworkd/AgentGPT) [↗](https://github.com/reworkd/AgentGPT)
27. Davis A M, Bersoff E H, Comer E R. A strategy for comparing alternative software development life cycle models[J]. IEEE Transactions on software Engineering, 1988, 14(10): 1453-1461. [find](https://scholar.google.com/scholar?q=Davis+A+M%2C+Bersoff+E+H%2C+Comer+E+R%2E+A+strategy+for+comparing+alternative+software+development+life+cycle+models%5BJ%5D%2E+IEEE+Transactions+on+software+Engineering%2C+1988%2C+14%2810%29%3A+1453%2D1461%2E)
28. Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. 2023. [CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X](https://md.research.elamur.ai/free/en/papers/zheng-2023-codegeex-pre-trained-model-code/). In Proceedings of the 29th ACM SIGKDD Conference on…
29. Fried D, Aghajanyan A, Lin J, et al. InCoder: A Generative Model for Code Infilling and Synthesis[C]//The Eleventh International Conference on Learning Representations. 2022. [find](https://scholar.google.com/scholar?q=Fried+D%2C+Aghajanyan+A%2C+Lin+J%2C+et+al%2E+InCoder%3A+A+Generative+Model+for+Code+Infilling+and+Synthesis%5BC%5D%2F%2FThe+Eleventh+International+Conference+on+Learning+Representations%2E+2022%2E)
30. Wang Y, Wang W, Joty S, et al. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021: 8696-8708. [find](https://scholar.google.com/scholar?q=Wang+Y%2C+Wang+W%2C+Joty+S%2C+et+al%2E+CodeT5%3A+Identifier%2Daware+Unified+Pre%2Dtrained+Encoder%2DDecoder+Models+for+Code+Understanding+and+Generation%5BC%5D%2F%2FProceedings+of+the+2021+Conference+on+Empirical+Methods+in+)
31. Christopoulou F, Lampouras G, Gritta M, et al. [Pangu-coder: Program synthesis with function-level language modeling](https://md.research.elamur.ai/free/en/papers/christopoulou-2022-pangu-coder-program-synthesis/)[J]. arXiv preprint arXiv:2207.11280, 2022.
32. Dou S, Liu Y, Jia H, et al. StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback[J]. arXiv preprint arXiv:2402.01391, 2024. — [arXiv:2402.01391](https://arxiv.org/abs/2402.01391) [find](https://scholar.google.com/scholar?q=Dou+S%2C+Liu+Y%2C+Jia+H%2C+et+al%2E+StepCoder%3A+Improve+Code+Generation+with+Reinforcement+Learning+from+Compiler+Feedback%5BJ%5D%2E+arXiv+preprint+arXiv%3A2402%2E01391%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E01391%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
33. Luo Z, Xu C, Zhao P, et al. WizardCoder: Empowering Code Large Language Models with Evol-Instruct[C]//The Twelfth International Conference on Learning Representations. 2023. [find](https://scholar.google.com/scholar?q=Luo+Z%2C+Xu+C%2C+Zhao+P%2C+et+al%2E+WizardCoder%3A+Empowering+Code+Large+Language+Models+with+Evol%2DInstruct%5BC%5D%2F%2FThe+Twelfth+International+Conference+on+Learning+Representations%2E+2023%2E)
34. Shen B, Zhang J, Chen T, et al. Pangu-coder2: Boosting large language models for code with ranking feedback[J]. arXiv preprint arXiv:2307.14936, 2023. — [arXiv:2307.14936](https://arxiv.org/abs/2307.14936) [find](https://scholar.google.com/scholar?q=Shen+B%2C+Zhang+J%2C+Chen+T%2C+et+al%2E+Pangu%2Dcoder2%3A+Boosting+large+language+models+for+code+with+ranking+feedback%5BJ%5D%2E+arXiv+preprint+arXiv%3A2307%2E14936%2C+2023%2E+%E2%80%94+%5BarXiv%3A2307%2E14936%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2307%2E1)
35. Zheng W, Sharan SP, Jaiswal AK, Wang K, Xi Y, Xu D, Wang Z. Outline, then details: Syntactically guided coarse-to-fine code generation. InInternational Conference on Machine Learning 2023 Jul 3 (pp. 42403-42419). PMLR. [find](https://scholar.google.com/scholar?q=Zheng+W%2C+Sharan+SP%2C+Jaiswal+AK%2C+Wang+K%2C+Xi+Y%2C+Xu+D%2C+Wang+Z%2E+Outline%2C+then+details%3A+Syntactically+guided+coarse%2Dto%2Dfine+code+generation%2E+InInternational+Conference+on+Machine+Learning+2023+Jul+3+%28pp%2E+4)
36. Zelikman E, Huang Q, Poesia G, Goodman N, Haber N. Parsel: Algorithmic Reasoning with Language Models by Composing Decompositions. Advances in Neural Information Processing Systems. 2024 Feb 13;36. [find](https://scholar.google.com/scholar?q=Zelikman+E%2C+Huang+Q%2C+Poesia+G%2C+Goodman+N%2C+Haber+N%2E+Parsel%3A+Algorithmic+Reasoning+with+Language+Models+by+Composing+Decompositions%2E+Advances+in+Neural+Information+Processing+Systems%2E+2024+Feb+13%3B36%2E)
37. Nguyen N, Nadi S. An empirical evaluation of GitHub copilot’s code suggestions[C]//Proceedings of the 19th International Conference on Mining Software Repositories. 2022: 1-5. [find](https://scholar.google.com/scholar?q=Nguyen+N%2C+Nadi+S%2E+An+empirical+evaluation+of+GitHub+copilot%E2%80%99s+code+suggestions%5BC%5D%2F%2FProceedings+of+the+19th+International+Conference+on+Mining+Software+Repositories%2E+2022%3A+1%2D5%2E)
38. Dakhel A M, Majdinasab V, Nikanjam A, et al. Github copilot ai pair programmer: Asset or liability?[J]. Journal of Systems and Software, 2023, 203: 111734. [find](https://scholar.google.com/scholar?q=Dakhel+A+M%2C+Majdinasab+V%2C+Nikanjam+A%2C+et+al%2E+Github+copilot+ai+pair+programmer%3A+Asset+or+liability%3F%5BJ%5D%2E+Journal+of+Systems+and+Software%2C+2023%2C+203%3A+111734%2E)
39. Yu H, Shen B, Ran D, Zhang J, Zhang Q, Ma Y, Liang G, Li Y, Wang Q, Xie T. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. InProceedings of the 46th IEEE/ACM International Conference on Software Engineering 2024 Feb 6 (pp. 1-12). [find](https://scholar.google.com/scholar?q=Yu+H%2C+Shen+B%2C+Ran+D%2C+Zhang+J%2C+Zhang+Q%2C+Ma+Y%2C+Liang+G%2C+Li+Y%2C+Wang+Q%2C+Xie+T%2E+Codereval%3A+A+benchmark+of+pragmatic+code+generation+with+generative+pre%2Dtrained+models%2E+InProceedings+of+the+46th+IEEE%2FACM+In)
40. Jimenez CE, Yang J, Wettig A, Yao S, Pei K, Press O, Narasimhan KR. [SWE-bench: Can Language Models Resolve Real-world Github Issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/). InThe Twelfth International Conference on Learning Representations 2023 Oct 13.
41. Iyer S, Konstas I, Cheung A, et al. Mapping Language to Code in Programmatic Context[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018: 1643-1652. [find](https://scholar.google.com/scholar?q=Iyer+S%2C+Konstas+I%2C+Cheung+A%2C+et+al%2E+Mapping+Language+to+Code+in+Programmatic+Context%5BC%5D%2F%2FProceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2E+2018%3A+1643%2D1652%2E)
42. Yin P, Deng B, Chen E, et al. [Learning to mine aligned code and natural language pairs from stack overflow](https://md.research.elamur.ai/free/en/papers/yin-2018-learning-mine-aligned-code-natural/)[C]//Proceedings of the 15th international conference on mining software repositories. 2018: 476-486.
43. Du X, Liu M, Wang K, et al. Evaluating Large Language Models in Class-Level Code Generation[C]//2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE). IEEE Computer Society, 2024: 865-865. [find](https://scholar.google.com/scholar?q=Du+X%2C+Liu+M%2C+Wang+K%2C+et+al%2E+Evaluating+Large+Language+Models+in+Class%2DLevel+Code+Generation%5BC%5D%2F%2F2024+IEEE%2FACM+46th+International+Conference+on+Software+Engineering+%28ICSE%29%2E+IEEE+Computer+Society%2C+2024%3A)
44. Ding Y, Wang Z, Ahmad W, et al. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion[J]. Advances in Neural Information Processing Systems, 2024, 36. [find](https://scholar.google.com/scholar?q=Ding+Y%2C+Wang+Z%2C+Ahmad+W%2C+et+al%2E+Crosscodeeval%3A+A+diverse+and+multilingual+benchmark+for+cross%2Dfile+code+completion%5BJ%5D%2E+Advances+in+Neural+Information+Processing+Systems%2C+2024%2C+36%2E)
45. Liu T, Xu C, McAuley J. RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems[C]//The Twelfth International Conference on Learning Representations. 2023. [find](https://scholar.google.com/scholar?q=Liu+T%2C+Xu+C%2C+McAuley+J%2E+RepoBench%3A+Benchmarking+Repository%2DLevel+Code+Auto%2DCompletion+Systems%5BC%5D%2F%2FThe+Twelfth+International+Conference+on+Learning+Representations%2E+2023%2E)
46. Zhang F, Chen B, Zhang Y, et al. RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation[C]//Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023: 2471-2484. [find](https://scholar.google.com/scholar?q=Zhang+F%2C+Chen+B%2C+Zhang+Y%2C+et+al%2E+RepoCoder%3A+Repository%2DLevel+Code+Completion+Through+Iterative+Retrieval+and+Generation%5BC%5D%2F%2FProceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+Language+)
47. Oda Y, Fudaba H, Neubig G, et al. Learning to generate pseudo-code from source code using statistical machine translation[C]//2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2015: 574-584. [find](https://scholar.google.com/scholar?q=Oda+Y%2C+Fudaba+H%2C+Neubig+G%2C+et+al%2E+Learning+to+generate+pseudo%2Dcode+from+source+code+using+statistical+machine+translation%5BC%5D%2F%2F2015+30th+IEEE%2FACM+International+Conference+on+Automated+Software+Engineer)
48. Austin J, Odena A, Nye M, et al. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/)[J]. arXiv preprint arXiv:2108.07732, 2021.
49. Hendrycks D, Basart S, Kadavath S, et al. [Measuring Coding Challenge Competence With APPS](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/)[C]//Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). 2021.
50. Cassano F, Gouwar J, Nguyen D, et al. MultiPL-E: a scalable and polyglot approach to benchmarking neural code generation[J]. IEEE Transactions on Software Engineering, 2023. [find](https://scholar.google.com/scholar?q=Cassano+F%2C+Gouwar+J%2C+Nguyen+D%2C+et+al%2E+MultiPL%2DE%3A+a+scalable+and+polyglot+approach+to+benchmarking+neural+code+generation%5BJ%5D%2E+IEEE+Transactions+on+Software+Engineering%2C+2023%2E)
51. Lai Y, Li C, Wang Y, et al. DS-1000: A natural and reliable benchmark for data science code generation[C]//International Conference on Machine Learning. PMLR, 2023: 18319-18345. [find](https://scholar.google.com/scholar?q=Lai+Y%2C+Li+C%2C+Wang+Y%2C+et+al%2E+DS%2D1000%3A+A+natural+and+reliable+benchmark+for+data+science+code+generation%5BC%5D%2F%2FInternational+Conference+on+Machine+Learning%2E+PMLR%2C+2023%3A+18319%2D18345%2E)
52. Ren S, Guo D, Lu S, et al. Codebleu: a method for automatic evaluation of code synthesis[J]. arXiv preprint arXiv:2009.10297, 2020. [find](https://scholar.google.com/scholar?q=Ren+S%2C+Guo+D%2C+Lu+S%2C+et+al%2E+Codebleu%3A+a+method+for+automatic+evaluation+of+code+synthesis%5BJ%5D%2E+arXiv+preprint+arXiv%3A2009%2E10297%2C+2020%2E)
53. Papineni K, Roukos S, Ward T, et al. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)[C]//Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 2002: 311-318.
54. [https://github.com/hiyouga/LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) [↗](https://github.com/hiyouga/LLaMA-Factory)
55. Wolf T, Debut L, Sanh V, et al. Huggingface’s transformers: State-ofthe-art natural language processing[J]. arXiv preprint arXiv:1910.[03771, 2019](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/).
56. Loshchilov I, Hutter F. Decoupled Weight Decay Regularization[C]//International Conference on Learning Representations. 2018. [find](https://scholar.google.com/scholar?q=Loshchilov+I%2C+Hutter+F%2E+Decoupled+Weight+Decay+Regularization%5BC%5D%2F%2FInternational+Conference+on+Learning+Representations%2E+2018%2E)
57. Kwon W, Li Z, Zhuang S, et al. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/)[C]//Proceedings of the 29th Symposium on Operating Systems Principles. 2023: 611-626.
58. Su J, Ahmed M, Lu Y, et al. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/)[J]. Neurocomputing[J]. Advances in Neural Information Processing Systems, 2024, 568: 127063.
59. Yen H, Gao T, Chen D. [Long-Context Language Modeling with Parallel Context Encoding](https://md.research.elamur.ai/free/en/papers/yen-2024-long-context-language-modeling-parallel/). arXiv preprint arXiv:2402.16617. 2024 Feb 26.
60. Chen Y, Qian S, Tang H, et al. [LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2023-longlora-efficient-fine-tuning-long/)[C]//The Twelfth International Conference on Learning Representations. 2023.
61. Zhang P, Liu Z, Xiao S, Shao N, Ye Q, Dou Z. Soaring from 4K to 400K: Extending LLM’s Context with Activation Beacon. arXiv preprint arXiv:2401.03462. 2024 Jan 7. — [Long Context Compression with Activation Beacon](https://md.research.elamur.ai/free/en/papers/zhang-2024-long-context-compression-activation-beacon/)
62. Guo D, Ren S, Lu S, et al. [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://md.research.elamur.ai/free/en/papers/guo-2021-graphcodebert-pre-training-code-representations/)[C]//International Conference on Learning Representations. 2020.
63. Huang D, Nan Z, Hu X, Jin P, Peng S, Wen Y, Zhang R, Du Z, Guo Q, Pu Y, Chen Y. ANPL: Towards Natural Programming with Interactive Decomposition. Advances in Neural Information Processing Systems. 2024 Feb 13;36. [find](https://scholar.google.com/scholar?q=Huang+D%2C+Nan+Z%2C+Hu+X%2C+Jin+P%2C+Peng+S%2C+Wen+Y%2C+Zhang+R%2C+Du+Z%2C+Guo+Q%2C+Pu+Y%2C+Chen+Y%2E+ANPL%3A+Towards+Natural+Programming+with+Interactive+Decomposition%2E+Advances+in+Neural+Information+Processing+Systems%2E+202)
64. Li J, Li Y, Li G, et al. Skcoder: A sketch-based approach for automatic code generation[C]//2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023: 2124-2135. [find](https://scholar.google.com/scholar?q=Li+J%2C+Li+Y%2C+Li+G%2C+et+al%2E+Skcoder%3A+A+sketch%2Dbased+approach+for+automatic+code+generation%5BC%5D%2F%2F2023+IEEE%2FACM+45th+International+Conference+on+Software+Engineering+%28ICSE%29%2E+IEEE%2C+2023%3A+2124%2D2135%2E)
65. Ge S, Goswami V, Zitnick C L, et al. Creative sketch generation[J]. arXiv preprint arXiv:2011.10039, 2020. [find](https://scholar.google.com/scholar?q=Ge+S%2C+Goswami+V%2C+Zitnick+C+L%2C+et+al%2E+Creative+sketch+generation%5BJ%5D%2E+arXiv+preprint+arXiv%3A2011%2E10039%2C+2020%2E)
66. Lozhkov A, Li R, Allal LB, Cassano F, Lamy-Poirier J, Tazi N, Tang A, Pykhtar D, Liu J, Wei Y, Liu T. StarCoder 2 and The Stack v2: The Next Generation. arXiv preprint arXiv:2402.19173. 2024 Feb 29. [find](https://scholar.google.com/scholar?q=Lozhkov+A%2C+Li+R%2C+Allal+LB%2C+Cassano+F%2C+Lamy%2DPoirier+J%2C+Tazi+N%2C+Tang+A%2C+Pykhtar+D%2C+Liu+J%2C+Wei+Y%2C+Liu+T%2E+StarCoder+2+and+The+Stack+v2%3A+The+Next+Generation%2E+arXiv+preprint+arXiv%3A2402%2E19173%2E+2024+Feb+29%2E)
67. Zhang B, Liu Z, Cherry C, et al. When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method[C]//The Twelfth International Conference on Learning Representations. 2023. [find](https://scholar.google.com/scholar?q=Zhang+B%2C+Liu+Z%2C+Cherry+C%2C+et+al%2E+When+Scaling+Meets+LLM+Finetuning%3A+The+Effect+of+Data%2C+Model+and+Finetuning+Method%5BC%5D%2F%2FThe+Twelfth+International+Conference+on+Learning+Representations%2E+2023%2E)
68. Hall M W, Kennedy K. Efficient call graph analysis[J]. ACM Letters on Programming Languages and Systems (LOPLAS), 1992, 1(3): 227-242. [find](https://scholar.google.com/scholar?q=Hall+M+W%2C+Kennedy+K%2E+Efficient+call+graph+analysis%5BJ%5D%2E+ACM+Letters+on+Programming+Languages+and+Systems+%28LOPLAS%29%2C+1992%2C+1%283%29%3A+227%2D242%2E)
69. Scott Wu. Introducing Devin, the first AI software engineer. [https://www.cognition-labs.com/introducing-devin](https://www.cognition-labs.com/introducing-devin) [↗](https://www.cognition-labs.com/introducing-devin)

*Updated 1 November 2026*