---
title: "NaturalCodeBench — paper by Zhang et al. (2024)"
description: "Popular code-generation benchmarks such as HumanEval, MBPP and DS-1000 mostly test short… 70 references, cited by 6 papers here."
authors: ["Shudan Zhang", "Hanlin Zhao", "Xiao Liu", "Qinkai Zheng", "Zehan Qi", "Xiaotao Gu", "Yuxiao Dong", "Jie Tang"]
year: 2024
venue: "Findings of the Association for Computational Linguistics: ACL 2024"
source: "https://aclanthology.org/2024.findings-acl.471/"
doi: "10.18653/v1/2024.findings-acl.471"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2024-naturalcodebench-examining-coding-performance-mismatch/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2024-naturalcodebench-examining-coding-performance-mismatch/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/) / NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Queries

# NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Queries

**Authors:** [Shudan Zhang](https://md.research.elamur.ai/free/en/authors/shudan-zhang/), [Hanlin Zhao](https://md.research.elamur.ai/free/en/authors/hanlin-zhao/), [Xiao Liu](https://md.research.elamur.ai/free/en/authors/xiao-liu/), [Qinkai Zheng](https://md.research.elamur.ai/free/en/authors/qinkai-zheng/), [Zehan Qi](https://md.research.elamur.ai/free/en/authors/zehan-qi/), [Xiaotao Gu](https://md.research.elamur.ai/free/en/authors/xiaotao-gu/), [Yuxiao Dong](https://md.research.elamur.ai/free/en/authors/yuxiao-dong/) and [Jie Tang](https://md.research.elamur.ai/free/en/authors/jie-tang/) · **Year:** 2024 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2024](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2024](https://md.research.elamur.ai/free/en/venues/findings/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.findings-acl.471/) · [PDF](https://aclanthology.org/2024.findings-acl.471.pdf) · [DOI 10.18653/v1/2024.findings-acl.471](https://doi.org/10.18653/v1/2024.findings-acl.471) · [OpenAlex](https://openalex.org/W4402670580)

**Citations:** 11 citations (OpenAlex, 2026-10-14) · 6 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Popular code-generation benchmarks such as HumanEval, MBPP and DS-1000 mostly test short, well-specified algorithmic or data-science exercises. They say little about how LLMs handle the messier requests developers actually make. NaturalCodeBench (NCB) addresses this gap with 402 Python and Java tasks derived from real user queries to the CodeGeeX coding service. The tasks span six domains, including software engineering, front-end, system administration and AI, and their test inputs include files and other complex structures. Candidate queries were screened first by GPT-3.5 and then by human reviewers for testability, determinism and usefulness. Writing tests for such realistic tasks is expensive, so the authors built a semi-automated pipeline: GPT-4 drafts a reference solution and test cases, and experts fix the errors. This roughly quadruples annotation speed while keeping test coverage similar. The authors evaluated 39 LLMs in a Docker-based execution environment. Models with similar HumanEval scores can differ substantially on NCB, which points to HumanEval overfitting or contamination. Even GPT-4 solves only about half of the tasks.

## Key points
- NCB has 402 problems, split evenly between Python (201) and Java (201) across 6 domains. More than 80% of the collected user queries were discarded during filtering.
- 140 problems (70 Python, 70 Java) are released as a development set. The remaining 262 form a closed test set to limit contamination. An evaluation toolkit is available on GitHub (THUDM/NaturalCodeBench).
- In the pipeline, GPT-4 drafts a solution and test cases (high-coverage plus corner cases), and annotators correct them. Building tests this way took about 40 minutes, compared with 175.9 minutes for fully manual writing, with near-identical coverage.
- Across 39 evaluated LLMs, GPT-4 was the strongest, but its pass rate was only about 53%.
- The authors attribute the mismatch with HumanEval partly to possible contamination of pre-training or instruction data. They cite, for example, GPT-4's own report that 25% of HumanEval appeared in its pre-training corpus.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 333 works · 87,019 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works · [🖥️ Computer Science Applications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-science-applications/) · 102 works

## Cite

Cite as: Shudan Zhang, Hanlin Zhao, Xiao Liu, Qinkai Zheng, Zehan Qi, Xiaotao Gu, Yuxiao Dong and Jie Tang. 2024. NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Queries. Findings of the Association for Computational Linguistics: ACL 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2024naturalcodebench,
  title = {{NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Queries}},
  author = {Shudan Zhang, Hanlin Zhao, Xiao Liu, Qinkai Zheng, Zehan Qi, Xiaotao Gu, Yuxiao Dong and Jie Tang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.471},
  url = {https://doi.org/10.18653/v1/2024.findings-acl.471}
}
```

- [← previous MultiPragEval: Multilingual Pragmatic Evaluation of Large Language Models](https://md.research.elamur.ai/free/en/papers/park-2024-multiprageval-multilingual-pragmatic-evaluation-large/)
- [next → OmniBench: Towards The Future of Universal Omni-Language Models](https://md.research.elamur.ai/free/en/papers/li-2024-omnibench-towards-future-universal-omni/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/)

## References

70 references, 33 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. T. Ahmed and P. Devanbu. Few-shot training llms for project-specific code-summarization. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, ASE ’22, New York, NY, USA, 2023. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=T%2E+Ahmed+and+P%2E+Devanbu%2E+Few%2Dshot+training+llms+for+project%2Dspecific+code%2Dsummarization%2E+In+Proceedings+of+the+37th+IEEE%2FACM+International+Conference+on+Automated+Software+Engineering%2C+ASE+%E2%80%9922%2C+New+Yo)
2. AI@Meta. Llama 3 model card. 2024. [find](https://scholar.google.com/scholar?q=AI%40Meta%2E+Llama+3+model+card%2E+2024%2E)
3. Anonymous. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, 2024. [find](https://scholar.google.com/scholar?q=Anonymous%2E+Wizardcoder%3A+Empowering+code+large+language+models+with+evol%2Dinstruct%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+2024%2E)
4. Anthropic. Claude-2, 2023. [find](https://scholar.google.com/scholar?q=Anthropic%2E+Claude%2D2%2C+2023%2E)
5. Anthropic. Introducing the claude 3 family. claude-3-family, 2023. Accessed: 2024-04-28. [https://www.anthropic.com/news/](https://www.anthropic.com/news/) [↗](https://www.anthropic.com/news/)
6. B. Athiwaratkun, S. K. Gouda, Z. Wang, X. Li, Y. Tian, M. Tan, W. U. Ahmad, S. Wang, Q. Sun, M. Shang, S. K. Gonugondla, H. Ding, V. Kumar, N. Fulton, A. Farahani, S. Jain, R. Giaquinto, H. Qian, M. K. Ramanathan, R. Nallapati, B. Ray, P. Bhatia, S. Sengupta, D. Roth, and B. Xiang. Multi-lingual eva… [find](https://scholar.google.com/scholar?q=B%2E+Athiwaratkun%2C+S%2E+K%2E+Gouda%2C+Z%2E+Wang%2C+X%2E+Li%2C+Y%2E+Tian%2C+M%2E+Tan%2C+W%2E+U%2E+Ahmad%2C+S%2E+Wang%2C+Q%2E+Sun%2C+M%2E+Shang%2C+S%2E+K%2E+Gonugondla%2C+H%2E+Ding%2C+V%2E+Kumar%2C+N%2E+Fulton%2C+A%2E+Farahani%2C+S%2E+Jain%2C+R%2E+Giaquinto%2C+H%2E+Qian%2C+M%2E+K)
7. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/), 2021.
8. J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609, 2023.
9. J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, S. Wang, W. Wang, S. Wu, B. Xu, J. Xu, A. Yang, H. Yang, J. Yang, S. Yang, Y. Yao, B.… — [Qwen Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
10. G. A. Blog. Google gemini: Next generation model. [https://blog.google/technology/](https://blog.google/technology/) ai/google-gemini-next-generation-model-february-2024/, Feb. 2024. [↗](https://blog.google/technology/)
11. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
12. F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M.-H. Yee, Y. Zi, C. J. Anderson, M. Q. Feldman, A. Guha, M. Greenberg, and A. Jangda. Multipl-e: A scalable and extensible approach to benchmarking neural code generation, 2022. [find](https://scholar.google.com/scholar?q=F%2E+Cassano%2C+J%2E+Gouwar%2C+D%2E+Nguyen%2C+S%2E+Nguyen%2C+L%2E+Phipps%2DCostin%2C+D%2E+Pinckney%2C+M%2E%2DH%2E+Yee%2C+Y%2E+Zi%2C+C%2E+J%2E+Anderson%2C+M%2E+Q%2E+Feldman%2C+A%2E+Guha%2C+M%2E+Greenberg%2C+and+A%2E+Jangda%2E+Multipl%2De%3A+A+scalable+and+extensible+)
13. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such,… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
14. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research, 24(240):1–113, 2023.
15. DeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2E+Deepseek+llm%3A+Scaling+open%2Dsource+language+models+with+longtermism%2E+arXiv+preprint+arXiv%3A2401%2E02954%2C+2024%2E)
16. Y. Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2023, page 423–435, New York, NY, USA, 20… [find](https://scholar.google.com/scholar?q=Y%2E+Deng%2C+C%2E+S%2E+Xia%2C+H%2E+Peng%2C+C%2E+Yang%2C+and+L%2E+Zhang%2E+Large+language+models+are+zero%2Dshot+fuzzers%3A+Fuzzing+deep%2Dlearning+libraries+via+large+language+models%2E+In+Proceedings+of+the+32nd+ACM+SIGSOFT+Inter)
17. Y. Deng, C. S. Xia, C. Yang, S. D. Zhang, S. Yang, and L. Zhang. Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt, 2023. [find](https://scholar.google.com/scholar?q=Y%2E+Deng%2C+C%2E+S%2E+Xia%2C+C%2E+Yang%2C+S%2E+D%2E+Zhang%2C+S%2E+Yang%2C+and+L%2E+Zhang%2E+Large+language+models+are+edge%2Dcase+fuzzers%3A+Testing+deep+learning+libraries+via+fuzzgpt%2C+2023%2E)
18. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu…
19. Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang. [GLM: General language model pretraining with autoregressive blank infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/). In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long…
20. A. Eghbali and M. Pradel. Crystalbleu: Precisely and efficiently measuring the similarity of code. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, ASE ’22, New York, NY, USA, 2023. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=A%2E+Eghbali+and+M%2E+Pradel%2E+Crystalbleu%3A+Precisely+and+efficiently+measuring+the+similarity+of+code%2E+In+Proceedings+of+the+37th+IEEE%2FACM+International+Conference+on+Automated+Software+Engineering%2C+ASE+%E2%80%99)
21. M. Evtikhiev, E. Bogomolov, Y. Sokolov, and T. Bryksin. Out of the bleu: How should we assess quality of the code generation models? Journal of Systems and Software, 203:111741, Sept. 2023. [find](https://scholar.google.com/scholar?q=M%2E+Evtikhiev%2C+E%2E+Bogomolov%2C+Y%2E+Sokolov%2C+and+T%2E+Bryksin%2E+Out+of+the+bleu%3A+How+should+we+assess+quality+of+the+code+generation+models%3F+Journal+of+Systems+and+Software%2C+203%3A111741%2C+Sept%2E+2023%2E)
22. D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, S. Yih, L. Zettlemoyer, and M. Lewis. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=D%2E+Fried%2C+A%2E+Aghajanyan%2C+J%2E+Lin%2C+S%2E+Wang%2C+E%2E+Wallace%2C+F%2E+Shi%2C+R%2E+Zhong%2C+S%2E+Yih%2C+L%2E+Zettlemoyer%2C+and+M%2E+Lewis%2E+Incoder%3A+A+generative+model+for+code+infilling+and+synthesis%2E+In+The+Eleventh+Internationa)
23. D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. K. Li, F. Luo, Y. Xiong, and W. Liang. Deepseek-coder: When the large language model meets programming – the rise of code intelligence, 2024. [find](https://scholar.google.com/scholar?q=D%2E+Guo%2C+Q%2E+Zhu%2C+D%2E+Yang%2C+Z%2E+Xie%2C+K%2E+Dong%2C+W%2E+Zhang%2C+G%2E+Chen%2C+X%2E+Bi%2C+Y%2E+Wu%2C+Y%2E+K%2E+Li%2C+F%2E+Luo%2C+Y%2E+Xiong%2C+and+W%2E+Liang%2E+Deepseek%2Dcoder%3A+When+the+large+language+model+meets+programming+%E2%80%93+the+rise+of+code+)
24. D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt. [Measuring coding challenge competence with apps](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/). NeurIPS, 2021.
25. D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt. [Measuring coding challenge competence with apps](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/), 2021.
26. M. Ivanković, G. Petrović, R. Just, and G. Fraser. Code coverage at google. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019, page 955–963, New York, NY, USA, 2019. Association for… [find](https://scholar.google.com/scholar?q=M%2E+Ivankovic%CC%81%2C+G%2E+Petrovic%CC%81%2C+R%2E+Just%2C+and+G%2E+Fraser%2E+Code+coverage+at+google%2E+In+Proceedings+of+the+2019+27th+ACM+Joint+Meeting+on+European+Software+Engineering+Conference+and+Symposium+on+the+Foundat)
27. S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer. Mapping language to code in programmatic context. In E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1643–1652, Brussels, Belgium, Oct.-Nov.… [find](https://scholar.google.com/scholar?q=S%2E+Iyer%2C+I%2E+Konstas%2C+A%2E+Cheung%2C+and+L%2E+Zettlemoyer%2E+Mapping+language+to+code+in+programmatic+context%2E+In+E%2E+Riloff%2C+D%2E+Chiang%2C+J%2E+Hockenmaier%2C+and+J%2E+Tsujii%2C+editors%2C+Proceedings+of+the+2018+Conferenc)
28. A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Q%2E+Jiang%2C+A%2E+Sablayrolles%2C+A%2E+Mensch%2C+C%2E+Bamford%2C+D%2E+S%2E+Chaplot%2C+D%2E+d%2E+l%2E+Casas%2C+F%2E+Bressand%2C+G%2E+Lengyel%2C+G%2E+Lample%2C+L%2E+Saulnier%2C+et+al%2E+Mistral+7b%2E+arXiv+preprint+arXiv%3A2310%2E06825%2C+2023%2E)
29. N. Jiang, K. Liu, T. Lutellier, and L. Tan. Impact of code language models on automated program repair. In Proceedings of the 45th International Conference on Software Engineering, ICSE ’23, page 1430–1442. IEEE Press, 2023. [find](https://scholar.google.com/scholar?q=N%2E+Jiang%2C+K%2E+Liu%2C+T%2E+Lutellier%2C+and+L%2E+Tan%2E+Impact+of+code+language+models+on+automated+program+repair%2E+In+Proceedings+of+the+45th+International+Conference+on+Software+Engineering%2C+ICSE+%E2%80%9923%2C+page+1430)
30. C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. [Swe-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) arXiv preprint arXiv:2310.06770, 2023.
31. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
32. Y. Lai, C. Li, Y. Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W.-T. Yih, D. Fried, S. Wang, and T. Yu. DS-1000: A natural and reliable benchmark for data science code generation. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th Internation… [find](https://scholar.google.com/scholar?q=Y%2E+Lai%2C+C%2E+Li%2C+Y%2E+Wang%2C+T%2E+Zhang%2C+R%2E+Zhong%2C+L%2E+Zettlemoyer%2C+W%2E%2DT%2E+Yih%2C+D%2E+Fried%2C+S%2E+Wang%2C+and+T%2E+Yu%2E+DS%2D1000%3A+A+natural+and+reliable+benchmark+for+data+science+code+generation%2E+In+A%2E+Krause%2C+E%2E+Brunsk)
33. R. Li, L. B. allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. LI, J. Chim, Q. Liu, E. Zheltonozhskii, T. Y. Zhuo, T. Wang, O. Dehaene, J. Lamy-Poirier, J. Monteiro, N. Gontier, M.-H. Yee, L. K. Umapathi, J. Zhu, B. Lipkin, M. Oblokulov, Z. Wang, R. Murthy, J. T. Stillerman,… [find](https://scholar.google.com/scholar?q=R%2E+Li%2C+L%2E+B%2E+allal%2C+Y%2E+Zi%2C+N%2E+Muennighoff%2C+D%2E+Kocetkov%2C+C%2E+Mou%2C+M%2E+Marone%2C+C%2E+Akiki%2C+J%2E+LI%2C+J%2E+Chim%2C+Q%2E+Liu%2C+E%2E+Zheltonozhskii%2C+T%2E+Y%2E+Zhuo%2C+T%2E+Wang%2C+O%2E+Dehaene%2C+J%2E+Lamy%2DPoirier%2C+J%2E+Monteiro%2C+N%2E+Gontie)
34. Y. Li, S. Bubeck, R. Eldan, A. D. Giorno, S. Gunasekar, and Y. T. Lee. [Textbooks are all you need ii: phi-1.5 technical report](https://md.research.elamur.ai/free/en/papers/li-2023-textbooks-all-need-ii-phi/), 2023.
35. Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, T. Hubert, P. Choy, C. de Masson d’Autume, I. Babuschkin, X. Chen, P.-S. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. J. Mankowitz, E. Sutherland Robson, P. Kohli, N. de Frei… — [Competition-Level Code Generation with AlphaCode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/)
36. C.-Y. Lin. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
37. J. Liu, C. S. Xia, Y. Wang, and L. Zhang. [Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/), 2023.
38. X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, et al. [Agentbench: Evaluating llms as agents](https://md.research.elamur.ai/free/en/papers/liu-2023-agentbench-evaluating-llms-agents/). arXiv preprint arXiv:2308.03688, 2023.
39. Z. Liu, C. Chen, J. Wang, X. Che, Y. Huang, J. Hu, and Q. Wang. Fill in the blank: Context-aware automated text input generation for mobile gui testing. In Proceedings of the 45th International Conference on Software Engineering, ICSE ’23, page 1355–1367. IEEE Press, 2023. [find](https://scholar.google.com/scholar?q=Z%2E+Liu%2C+C%2E+Chen%2C+J%2E+Wang%2C+X%2E+Che%2C+Y%2E+Huang%2C+J%2E+Hu%2C+and+Q%2E+Wang%2E+Fill+in+the+blank%3A+Context%2Daware+automated+text+input+generation+for+mobile+gui+testing%2E+In+Proceedings+of+the+45th+International+Confer)
40. S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. Liu. Codexglue: A machine learning benchmark dataset for code understanding and gener… [find](https://scholar.google.com/scholar?q=S%2E+Lu%2C+D%2E+Guo%2C+S%2E+Ren%2C+J%2E+Huang%2C+A%2E+Svyatkovskiy%2C+A%2E+Blanco%2C+C%2E+Clement%2C+D%2E+Drain%2C+D%2E+Jiang%2C+D%2E+Tang%2C+G%2E+Li%2C+L%2E+Zhou%2C+L%2E+Shou%2C+L%2E+Zhou%2C+M%2E+Tufano%2C+M%2E+Gong%2C+M%2E+Zhou%2C+N%2E+Duan%2C+N%2E+Sundaresan%2C+S%2E+K%2E+Deng%2C)
41. Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568, 2023. [find](https://scholar.google.com/scholar?q=Z%2E+Luo%2C+C%2E+Xu%2C+P%2E+Zhao%2C+Q%2E+Sun%2C+X%2E+Geng%2C+W%2E+Hu%2C+C%2E+Tao%2C+J%2E+Ma%2C+Q%2E+Lin%2C+and+D%2E+Jiang%2E+Wizardcoder%3A+Empowering+code+large+language+models+with+evol%2Dinstruct%2E+arXiv+preprint+arXiv%3A2306%2E08568%2C+2023%2E)
42. A. Moradi Dakhel, V. Majdinasab, A. Nikanjam, F. Khomh, M. C. Desmarais, and Z. M. J. Jiang. Github copilot ai pair programmer: Asset or liability? J. Syst. Softw., 203(C), sep 2023. [find](https://scholar.google.com/scholar?q=A%2E+Moradi+Dakhel%2C+V%2E+Majdinasab%2C+A%2E+Nikanjam%2C+F%2E+Khomh%2C+M%2E+C%2E+Desmarais%2C+and+Z%2E+M%2E+J%2E+Jiang%2E+Github+copilot+ai+pair+programmer%3A+Asset+or+liability%3F+J%2E+Syst%2E+Softw%2E%2C+203%28C%29%2C+sep+2023%2E)
43. E. Nijkamp, H. Hayashi, C. Xiong, S. Savarese, and Y. Zhou. Codegen2: Lessons for training llms on programming and natural languages. arXiv preprint arXiv:2305.02309, 2023. [find](https://scholar.google.com/scholar?q=E%2E+Nijkamp%2C+H%2E+Hayashi%2C+C%2E+Xiong%2C+S%2E+Savarese%2C+and+Y%2E+Zhou%2E+Codegen2%3A+Lessons+for+training+llms+on+programming+and+natural+languages%2E+arXiv+preprint+arXiv%3A2305%2E02309%2C+2023%2E)
44. E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=E%2E+Nijkamp%2C+B%2E+Pang%2C+H%2E+Hayashi%2C+L%2E+Tu%2C+H%2E+Wang%2C+Y%2E+Zhou%2C+S%2E+Savarese%2C+and+C%2E+Xiong%2E+Codegen%3A+An+open+large+language+model+for+code+with+multi%2Dturn+program+synthesis%2E+In+The+Eleventh+International+Con)
45. OpenAI, :, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V. Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bogdonoff, O. Boiko… — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
46. OpenAI. Introducing chatgpt, 2022. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Introducing+chatgpt%2C+2022%2E)
47. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). In…
48. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In P. Isabelle, E. Charniak, and D. Lin, editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA, J…
49. Phind. Phind-codellama-34b-v2, 2023. [find](https://scholar.google.com/scholar?q=Phind%2E+Phind%2Dcodellama%2D34b%2Dv2%2C+2023%2E)
50. M. Popović. [chrF: character n-gram F-score for automatic MT evaluation](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/). In O. Bojar, R. Chatterjee, C. Federmann, B. Haddow, C. Hokamp, M. Huck, V. Logacheva, and P. Pecina, editors, Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal, Sept. 2015. A…
51. B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, et al. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950, 2023.
52. B. Roziere, M.-A. Lachaux, L. Chanussot, and G. Lample. [Unsupervised translation of programming languages](https://md.research.elamur.ai/free/en/papers/lachaux-2020-unsupervised-translation-programming-languages/). In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc.
53. B. Roziere, J. M. Zhang, F. Charton, M. Harman, G. Synnaeve, and G. Lample. Leveraging automated unit tests for unsupervised code translation, 2022. [find](https://scholar.google.com/scholar?q=B%2E+Roziere%2C+J%2E+M%2E+Zhang%2C+F%2E+Charton%2C+M%2E+Harman%2C+G%2E+Synnaeve%2C+and+G%2E+Lample%2E+Leveraging+automated+unit+tests+for+unsupervised+code+translation%2C+2022%2E)
54. B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A. Défossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, and G. Synnaev… — [Code Llama: Open Foundation Models for Code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
55. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/), 2023.
56. N. Tran, H. Tran, S. Nguyen, H. Nguyen, and T. N. Nguyen. Does bleu score work for code migration? In Proceedings of the 27th International Conference on Program Comprehension, ICPC ’19, page 165–176. IEEE Press, 2019. [find](https://scholar.google.com/scholar?q=N%2E+Tran%2C+H%2E+Tran%2C+S%2E+Nguyen%2C+H%2E+Nguyen%2C+and+T%2E+N%2E+Nguyen%2E+Does+bleu+score+work+for+code+migration%3F+In+Proceedings+of+the+27th+International+Conference+on+Program+Comprehension%2C+ICPC+%E2%80%9919%2C+page+165%E2%80%93176%2E)
57. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA, 2017. Curran Associates Inc.
58. Y. Wei, C. S. Xia, and L. Zhang. Copiloting the copilots: Fusing large language models with completion engines for automated program repair. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, page 172… [find](https://scholar.google.com/scholar?q=Y%2E+Wei%2C+C%2E+S%2E+Xia%2C+and+L%2E+Zhang%2E+Copiloting+the+copilots%3A+Fusing+large+language+models+with+completion+engines+for+automated+program+repair%2E+In+Proceedings+of+the+31st+ACM+Joint+European+Software+Engi)
59. C. S. Xia, M. Paltenghi, J. L. Tian, M. Pradel, and L. Zhang. Fuzz4all: Universal fuzzing with large language models, 2024. [find](https://scholar.google.com/scholar?q=C%2E+S%2E+Xia%2C+M%2E+Paltenghi%2C+J%2E+L%2E+Tian%2C+M%2E+Pradel%2C+and+L%2E+Zhang%2E+Fuzz4all%3A+Universal+fuzzing+with+large+language+models%2C+2024%2E)
60. C. S. Xia, Y. Wei, and L. Zhang. Automated program repair in the era of large pre-trained language models. In Proceedings of the 45th International Conference on Software Engineering, ICSE ’23, page 1482–1494. IEEE Press, 2023. [find](https://scholar.google.com/scholar?q=C%2E+S%2E+Xia%2C+Y%2E+Wei%2C+and+L%2E+Zhang%2E+Automated+program+repair+in+the+era+of+large+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+45th+International+Conference+on+Software+Engineering%2C+ICSE+%E2%80%9923%2C+page+1)
61. C. S. Xia and L. Zhang. Less training, more repairing please: revisiting automated program repair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, page 959–971, New York, NY,… [find](https://scholar.google.com/scholar?q=C%2E+S%2E+Xia+and+L%2E+Zhang%2E+Less+training%2C+more+repairing+please%3A+revisiting+automated+program+repair+via+zero%2Dshot+learning%2E+In+Proceedings+of+the+30th+ACM+Joint+European+Software+Engineering+Conference+)
62. F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, MAPS 2022, page 1–10, New York, NY, USA, 2022. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=F%2E+F%2E+Xu%2C+U%2E+Alon%2C+G%2E+Neubig%2C+and+V%2E+J%2E+Hellendoorn%2E+A+systematic+evaluation+of+large+language+models+of+code%2E+In+Proceedings+of+the+6th+ACM+SIGPLAN+International+Symposium+on+Machine+Programming%2C+MAP)
63. C. Yang, Y. Deng, R. Lu, J. Yao, J. Liu, R. Jabbarvand, and L. Zhang. White-box compiler fuzzing empowered by large language models, 2023. [find](https://scholar.google.com/scholar?q=C%2E+Yang%2C+Y%2E+Deng%2C+R%2E+Lu%2C+J%2E+Yao%2C+J%2E+Liu%2C+R%2E+Jabbarvand%2C+and+L%2E+Zhang%2E+White%2Dbox+compiler+fuzzing+empowered+by+large+language+models%2C+2023%2E)
64. S. Yang, W.-L. Chiang, L. Zheng, J. E. Gonzalez, and I. Stoica. [Rethinking benchmark and contamination for language models with rephrased samples](https://md.research.elamur.ai/free/en/papers/yang-2023-rethinking-benchmark-contamination-language-models/). arXiv preprint arXiv:2311.04850, 2023.
65. P. Yin, B. Deng, E. Chen, B. Vasilescu, and G. Neubig. [Learning to mine aligned code and natural language pairs from stack overflow](https://md.research.elamur.ai/free/en/papers/yin-2018-learning-mine-aligned-code-natural/). In Proceedings of the 15th International Conference on Mining Software Repositories, MSR ’18, page 476–486, New York, NY, USA, 2018. Association for Computing Machiner…
66. T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, Z. Zhang, and D. Radev. [Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task](https://md.research.elamur.ai/free/en/papers/yu-2018-spider-large-scale-human-labeled/). In E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, editors, Proceedings…
67. J. M. Zelle and R. J. Mooney. [Learning to parse database queries using inductive logic programming](https://md.research.elamur.ai/free/en/papers/zelle-1996-learning-parse-database-queries-using/). In Proceedings of the Thirteenth National Conference on Artificial Intelligence -Volume 2, AAAI’96, page 1050–1055. AAAI Press, 1996.
68. A. Zeng, X. Liu, Z. Du, Z. Wang, H. Lai, M. Ding, Z. Yang, Y. Xu, W. Zheng, X. Xia, W. L. Tam, Z. Ma, Y. Xue, J. Zhai, W. Chen, P. Zhang, Y. Dong, and J. Tang. [Glm-130b: An open bilingual pre-trained model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/), 2023.
69. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/), 2023.
70. Q. Zheng, X. Xia, X. Zou, Y. Dong, S. Wang, Y. Xue, L. Shen, Z. Wang, A. Wang, Y. Li, T. Su, Z. Yang, and J. Tang. [Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x](https://md.research.elamur.ai/free/en/papers/zheng-2023-codegeex-pre-trained-model-code/). In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,…

*Updated 31 October 2026*