---
title: "BioCoder — paper by Tang et al. (2023) — summary and ideas"
description: "BioCoder therefore offers a realistic yardstick for specialized code generation. 64 references, cited by 8 papers here."
authors: ["Xiangru Tang", "Bill Qian", "Rick Gao", "Jiakang Chen", "Xinyun Chen", "Mark Gerstein"]
year: 2023
venue: "arXiv:2308.16458"
source: "https://arxiv.org/abs/2308.16458"
doi: "10.48550/arxiv.2308.16458"
arxiv: "2308.16458"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tang-2023-biocoder-benchmark-bioinformatics-code-generation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tang-2023-biocoder-benchmark-bioinformatics-code-generation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/11/) / BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models

# BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models

**Authors:** [Xiangru Tang](https://md.research.elamur.ai/free/en/authors/xiangru-tang/), [Bill Qian](https://md.research.elamur.ai/free/en/authors/bill-qian/), Rick Gao, Jiakang Chen, [Xinyun Chen](https://md.research.elamur.ai/free/en/authors/xinyun-chen/) and [Mark Gerstein](https://md.research.elamur.ai/free/en/authors/mark-gerstein/) · **Year:** 2023 · **Venue:** [arXiv:2308.16458](https://md.research.elamur.ai/free/en/venues/bioinformatics/) ([Bioinformatics 2023](https://md.research.elamur.ai/free/en/venues/bioinformatics/2023/)) · **Publisher:** [Oxford University Press](https://md.research.elamur.ai/free/en/publishers/oxford-university-press/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2308.16458) · [PDF](https://arxiv.org/pdf/2308.16458) · [DOI 10.48550/arxiv.2308.16458](https://doi.org/10.48550/arxiv.2308.16458) · [arXiv 2308.16458](https://arxiv.org/abs/2308.16458) · [OpenAlex](https://openalex.org/W4400140877)

**Citations:** 30 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/tang-2023-biocoder-benchmark-bioinformatics-code-generation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most LLM code-generation benchmarks use short, generic functions and ignore the domain knowledge and project context that real scientific code needs. BioCoder fills this gap for bioinformatics. The authors mined GitHub repositories cited in peer-reviewed bioinformatics publications, parsed them with AST-based tools, and filtered candidate functions through automated and manual checks. Each problem comes with its full context: imports, class declarations, global variables and helper functions. Tasks from the Rosalind Project were also added. Generated code is run against fuzz tests in Docker containers, and results are reported as Pass@K. Nine models were evaluated, from InCoder and CodeGen to GPT-4, and StarCoder was additionally fine-tuned on BioCoder training data. GPT-3.5 and GPT-4 solved about half of the problems, while smaller open models reached 25% at most. Fine-tuning gave StarCoder consistent gains. The results indicate that strong models need two things: the ability to use long, dependency-heavy prompts and real biological domain knowledge. BioCoder therefore offers a realistic yardstick for specialized code generation.

## Key points
- Dataset: 1,026 Python functions and 1,243 Java methods drawn from 1,720 bioinformatics repositories, plus 253 Rosalind problems. It is split into public (460), hidden (2,269) and similar (460) sets.
- Evaluation relies on a scalable fuzz-testing framework inside Docker. Topic modeling is used to show that the tasks cover the breadth of bioinformatics work.
- Models tested: InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+, InstructCodeT5+, GPT-3.5 and GPT-4. GPT models reach about 50% versus at most 25% for the others. Successful models handle prompts longer than 2,600 tokens.
- Fine-tuning StarCoder on BioCoder data raised Pass@K by more than 3% in every prompt setup and by over 15% in some.
- Acknowledged limitations:
  - possible training-data contamination for closed models;
  - reproducibility issues with evolving APIs;
  - curation bias from using OpenAI GPT to select samples.
- Resources are available at github.com/gersteinlab/biocoder.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 533 works · 443,884 citations
- [Yale University](https://md.research.elamur.ai/free/en/universities/yale-university/) — University, New Haven, United States · 112 works · 14,009 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,888 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 306 works · [🎯 Information Systems and Management](https://md.research.elamur.ai/free/en/shelves/decision-sciences/information-systems-and-management/) · 96 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,394 works · [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 194 works

## Cite

Cite as: Xiangru Tang, Bill Qian, Rick Gao, Jiakang Chen, Xinyun Chen and Mark Gerstein. 2023. BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models. arXiv:2308.16458.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{tang2023biocoder,
  title = {{BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models}},
  author = {Xiangru Tang, Bill Qian, Rick Gao, Jiakang Chen, Xinyun Chen and Mark Gerstein},
  year = {2023},
  doi = {10.48550/arxiv.2308.16458},
  eprint = {2308.16458},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.16458}
}
```

- [← previous Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigation](https://md.research.elamur.ai/free/en/papers/alhafni-2023-advancements-arabic-grammatical-error-detection/)
- [next → Can GPT-4 Perform Neural Architecture Search?](https://md.research.elamur.ai/free/en/papers/zheng-2023-gpt-4-perform-neural-architecture/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/11/)

## References

64 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Allal, L. B., Li, R., Kocetkov, D. et al. (2023). [Santacoder: don’t reach for the stars!](https://md.research.elamur.ai/free/en/papers/allal-2023-santacoder-don-t-reach-stars/) arXiv preprint arXiv:2301.03988.
2. Askell, A., Bai, Y., Chen, A. et al. (2021). A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861. [find](https://scholar.google.com/scholar?q=Askell%2C+A%2E%2C+Bai%2C+Y%2E%2C+Chen%2C+A%2E+et+al%2E+%282021%29%2E+A+general+language+assistant+as+a+laboratory+for+alignment%2E+arXiv+preprint+arXiv%3A2112%2E00861%2E)
3. Athiwaratkun, B., Gouda, S. K., Wang, Z. et al. (2023). [Multi-lingual evaluation of code generation models](https://md.research.elamur.ai/free/en/papers/athiwaratkun-2022-multi-lingual-evaluation-code-generation/).
4. Austin, J., Odena, A., Nye, M. et al. (2021). [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/).
5. Bai, Y., Jones, A., Ndousse, K. et al. (2022). Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. [find](https://scholar.google.com/scholar?q=Bai%2C+Y%2E%2C+Jones%2C+A%2E%2C+Ndousse%2C+K%2E+et+al%2E+%282022%29%2E+Training+a+helpful+and+harmless+assistant+with+reinforcement+learning+from+human+feedback%2E+arXiv+preprint+arXiv%3A2204%2E05862%2E)
6. Barke, S., James, M. B. and Polikarpova, N. (2023). Grounded copilot: How programmers interact with code-generating models. Proceedings of the ACM on Programming Languages, 7(OOPSLA1), 85–111. [find](https://scholar.google.com/scholar?q=Barke%2C+S%2E%2C+James%2C+M%2E+B%2E+and+Polikarpova%2C+N%2E+%282023%29%2E+Grounded+copilot%3A+How+programmers+interact+with+code%2Dgenerating+models%2E+Proceedings+of+the+ACM+on+Programming+Languages%2C+7%28OOPSLA1%29%2C+85%E2%80%93111%2E)
7. Biderman, S., Schoelkopf, H., Anthony, Q. et al. (2023). Pythia: A suite for analyzing large language models across training and scaling. [find](https://scholar.google.com/scholar?q=Biderman%2C+S%2E%2C+Schoelkopf%2C+H%2E%2C+Anthony%2C+Q%2E+et+al%2E+%282023%29%2E+Pythia%3A+A+suite+for+analyzing+large+language+models+across+training+and+scaling%2E)
8. Bommasani, R., Hudson, D. A., Adeli, E. et al. (2022). [On the opportunities and risks of foundation models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/).
9. Brown, T. B., Mann, B., Ryder, N. et al. (2020). [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). arXiv preprint arXiv:2005.14165.
10. Bui, N. D., Le, H., Wang, Y. et al. (2023). Codetf: One-stop transformer library for state-of-the-art code llm. arXiv preprint arXiv:2306.00029. — [arXiv:2306.00029](https://arxiv.org/abs/2306.00029) [find](https://scholar.google.com/scholar?q=Bui%2C+N%2E+D%2E%2C+Le%2C+H%2E%2C+Wang%2C+Y%2E+et+al%2E+%282023%29%2E+Codetf%3A+One%2Dstop+transformer+library+for+state%2Dof%2Dthe%2Dart+code+llm%2E+arXiv+preprint+arXiv%3A2306%2E00029%2E+%E2%80%94+%5BarXiv%3A2306%2E00029%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2306%2E00029%29)
11. Cassano, F., Gouwar, J., Nguyen, D. et al. (2023). MultiPL-E: a scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering, pages 1–17. [find](https://scholar.google.com/scholar?q=Cassano%2C+F%2E%2C+Gouwar%2C+J%2E%2C+Nguyen%2C+D%2E+et+al%2E+%282023%29%2E+MultiPL%2DE%3A+a+scalable+and+polyglot+approach+to+benchmarking+neural+code+generation%2E+IEEE+Transactions+on+Software+Engineering%2C+pages+1%E2%80%9317%2E)
12. BioCoder Chen, B., Zhang, F., Nguyen, A. et al. (2022). Codet: Code generation with generated tests. arXiv preprint arXiv:2207.10397. [find](https://scholar.google.com/scholar?q=BioCoder+Chen%2C+B%2E%2C+Zhang%2C+F%2E%2C+Nguyen%2C+A%2E+et+al%2E+%282022%29%2E+Codet%3A+Code+generation+with+generated+tests%2E+arXiv+preprint+arXiv%3A2207%2E10397%2E)
13. Chen, M., Tworek, J., Jun, H. et al. (2021). [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/).
14. Chen, T. Y., Ho, J. W., Liu, H. et al. (2009). An innovative approach for testing bioinformatics programs using metamorphic testing. BMC Bioinformatics, 10(1), 24. [find](https://scholar.google.com/scholar?q=Chen%2C+T%2E+Y%2E%2C+Ho%2C+J%2E+W%2E%2C+Liu%2C+H%2E+et+al%2E+%282009%29%2E+An+innovative+approach+for+testing+bioinformatics+programs+using+metamorphic+testing%2E+BMC+Bioinformatics%2C+10%281%29%2C+24%2E)
15. Chen, X., Lin, M., SchÃd’rli, N. et al. (2023). Teaching large language models to self-debug. [find](https://scholar.google.com/scholar?q=Chen%2C+X%2E%2C+Lin%2C+M%2E%2C+Sch%C3%83d%E2%80%99rli%2C+N%2E+et+al%2E+%282023%29%2E+Teaching+large+language+models+to+self%2Ddebug%2E)
16. Chowdhery, A., Narang, S., Devlin, J. et al. (2022). [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311.
17. Christopoulou, F., Lampouras, G., Gritta, M. et al. (2022). [PanGu-Coder: program synthesis with function-level language modeling](https://md.research.elamur.ai/free/en/papers/christopoulou-2022-pangu-coder-program-synthesis/). arXiv preprint arXiv:2207.11280.
18. Du, X., Liu, M., Wang, K. et al. (2023). Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. arXiv preprint arXiv:2308.01861. — [arXiv:2308.01861](https://arxiv.org/abs/2308.01861) [find](https://scholar.google.com/scholar?q=Du%2C+X%2E%2C+Liu%2C+M%2E%2C+Wang%2C+K%2E+et+al%2E+%282023%29%2E+Classeval%3A+A+manually%2Dcrafted+benchmark+for+evaluating+llms+on+class%2Dlevel+code+generation%2E+arXiv+preprint+arXiv%3A2308%2E01861%2E+%E2%80%94+%5BarXiv%3A2308%2E01861%5D%28https%3A%2F%2Farxiv)
19. Fried, D., Aghajanyan, A., Lin, J. et al. (2023). Incoder: A generative model for code infilling and synthesis. [find](https://scholar.google.com/scholar?q=Fried%2C+D%2E%2C+Aghajanyan%2C+A%2E%2C+Lin%2C+J%2E+et+al%2E+%282023%29%2E+Incoder%3A+A+generative+model+for+code+infilling+and+synthesis%2E)
20. Gao, L., Tow, J., Biderman, S. et al. (2022). Eleutherai/lm-evaluation-harness: v0.3.0. [find](https://scholar.google.com/scholar?q=Gao%2C+L%2E%2C+Tow%2C+J%2E%2C+Biderman%2C+S%2E+et+al%2E+%282022%29%2E+Eleutherai%2Flm%2Devaluation%2Dharness%3A+v0%2E3%2E0%2E)
21. Hendrycks, D., Basart, S., Kadavath, S. et al. (2021). [Measuring coding challenge competence with apps](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/).
22. NeurIPS. Hoffmann, J., Borgeaud, S., Mensch, A. et al. (2022). [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/).
23. Khlaaf, H., Mishkin, P., Achiam, J. et al. (2022). A hazard analysis framework for code synthesis large language models. arXiv preprint arXiv:2207.14157. — [arXiv:2207.14157](https://arxiv.org/abs/2207.14157) [find](https://scholar.google.com/scholar?q=Khlaaf%2C+H%2E%2C+Mishkin%2C+P%2E%2C+Achiam%2C+J%2E+et+al%2E+%282022%29%2E+A+hazard+analysis+framework+for+code+synthesis+large+language+models%2E+arXiv+preprint+arXiv%3A2207%2E14157%2E+%E2%80%94+%5BarXiv%3A2207%2E14157%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F220)
24. Lai, Y., Li, C., Wang, Y. et al. (2022). DS-1000: a natural and reliable benchmark for data science code generation. ArXiv, abs/2211.11501. [find](https://scholar.google.com/scholar?q=Lai%2C+Y%2E%2C+Li%2C+C%2E%2C+Wang%2C+Y%2E+et+al%2E+%282022%29%2E+DS%2D1000%3A+a+natural+and+reliable+benchmark+for+data+science+code+generation%2E+ArXiv%2C+abs%2F2211%2E11501%2E)
25. Lee, T., Hong, S., Ahn, J. et al. (2023). [Who wrote this code? watermarking for code generation](https://md.research.elamur.ai/free/en/papers/lee-2024-who-wrote-this-code-watermarking/). arXiv preprint arXiv:2305.15060.
26. Li, R., Allal, L. B., Zi, Y. et al. (2023). Starcoder: may the source be with you! Li, Y., Choi, D., Chung, J. et al. (2022). [Competition-level code generation with alphacode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/). arXiv preprint arXiv:2203.07814.
27. Liu, T., Xu, C. and McAuley, J. (2023). [Repobench: Benchmarking repository-level code auto-completion systems](https://md.research.elamur.ai/free/en/papers/liu-2023-repobench-benchmarking-repository-level-code/).
28. Luo, Z., Xu, C., Zhao, P. et al. (2023). Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568. [find](https://scholar.google.com/scholar?q=Luo%2C+Z%2E%2C+Xu%2C+C%2E%2C+Zhao%2C+P%2E+et+al%2E+%282023%29%2E+Wizardcoder%3A+Empowering+code+large+language+models+with+evol%2Dinstruct%2E+arXiv+preprint+arXiv%3A2306%2E08568%2E)
29. Mikolov, T., Chen, K., Corrado, G. et al. (2013). [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/).
30. Mitchell, M., Luccioni, A. S., Lambert, N. et al. (2023). Measuring data. [find](https://scholar.google.com/scholar?q=Mitchell%2C+M%2E%2C+Luccioni%2C+A%2E+S%2E%2C+Lambert%2C+N%2E+et+al%2E+%282023%29%2E+Measuring+data%2E)
31. MOI, A., Patry, N., Cistac, P. et al. (2022). huggingface/tokenizers: Rust 0.13.2. [find](https://scholar.google.com/scholar?q=MOI%2C+A%2E%2C+Patry%2C+N%2E%2C+Cistac%2C+P%2E+et+al%2E+%282022%29%2E+huggingface%2Ftokenizers%3A+Rust+0%2E13%2E2%2E)
32. Muennighoff, N., Wang, T., Sutawika, L. et al. (2023a). [Crosslingual generalization through multitask finetuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/).
33. Muennighoff, N., Liu, Q., Zebaze, A. et al. (2023b). Octopack: Instruction tuning code large language models. arXiv preprint arXiv:2308.07124. [find](https://scholar.google.com/scholar?q=Muennighoff%2C+N%2E%2C+Liu%2C+Q%2E%2C+Zebaze%2C+A%2E+et+al%2E+%282023b%29%2E+Octopack%3A+Instruction+tuning+code+large+language+models%2E+arXiv+preprint+arXiv%3A2308%2E07124%2E)
34. Nijkamp, E., Pang, B., Hayashi, H. et al. (2023a). CodeGen: an open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Nijkamp%2C+E%2E%2C+Pang%2C+B%2E%2C+Hayashi%2C+H%2E+et+al%2E+%282023a%29%2E+CodeGen%3A+an+open+large+language+model+for+code+with+multi%2Dturn+program+synthesis%2E+In+The+Eleventh+International+Conference+on+Learning+Representation)
35. Nijkamp, E., Hayashi, H., Xiong, C. et al. (2023b). [Codegen2: Lessons for training llms on programming and natural languages](https://md.research.elamur.ai/free/en/papers/nijkamp-2023-codegen2-lessons-training-llms-programming/). arXiv preprint.
36. Olausson, T. X., Inala, J. P., Wang, C. et al. (2023). [Demystifying gpt self-repair for code generation](https://md.research.elamur.ai/free/en/papers/olausson-2024-self-repair-silver-bullet-code/). arXiv preprint arXiv:2306.09896.
37. OpenAI (2023). [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
38. Orlanski, G., Xiao, K., Garcia, X. et al. (2023). [Measuring the impact of programming language distribution](https://md.research.elamur.ai/free/en/papers/orlanski-2023-measuring-impact-programming-language-distribution/). In International Conference on Machine Learning, pages 26619–26645.
39. PMLR. Pan, R., Ibrahimzada, A. R., Krishna, R. et al. (2023). [Understanding the effectiveness of large language models in code translation](https://md.research.elamur.ai/free/en/papers/pan-2024-lost-translation-study-bugs-introduced/). arXiv preprint arXiv:2308.03109.
40. Patil, S. G., Zhang, T., Wang, X. et al. (2023). [Gorilla: Large language model connected with massive apis](https://md.research.elamur.ai/free/en/papers/patil-2023-gorilla-large-language-model-connected/). arXiv preprint arXiv:2305.15334.
41. Qin, Y., Liang, S., Ye, Y. et al. (2023). [Toolllm: Facilitating large language models to master 16000+ real-world apis](https://md.research.elamur.ai/free/en/papers/qin-2023-toolllm-facilitating-large-language-models/). arXiv preprint arXiv:2307.16789.
42. Radford, A., Wu, J., Child, R. et al. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8), 9. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+Child%2C+R%2E+et+al%2E+%282019%29%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%2C+9%2E)
43. Russell, P. H., Johnson, R. L., Ananthan, S. et al. (2018). A large-scale analysis of bioinformatics code on github. PloS one, 13(10), e0205898. [find](https://scholar.google.com/scholar?q=Russell%2C+P%2E+H%2E%2C+Johnson%2C+R%2E+L%2E%2C+Ananthan%2C+S%2E+et+al%2E+%282018%29%2E+A+large%2Dscale+analysis+of+bioinformatics+code+on+github%2E+PloS+one%2C+13%2810%29%2C+e0205898%2E)
44. Shen, B., Zhang, J., Chen, T. et al. (2023). Pangu-coder2: Boosting large language models for code with ranking feedback. arXiv preprint arXiv:2307.14936. — [arXiv:2307.14936](https://arxiv.org/abs/2307.14936) [find](https://scholar.google.com/scholar?q=Shen%2C+B%2E%2C+Zhang%2C+J%2E%2C+Chen%2C+T%2E+et+al%2E+%282023%29%2E+Pangu%2Dcoder2%3A+Boosting+large+language+models+for+code+with+ranking+feedback%2E+arXiv+preprint+arXiv%3A2307%2E14936%2E+%E2%80%94+%5BarXiv%3A2307%2E14936%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F23)
45. Shrivastava, D., Kocetkov, D., de Vries, H. et al. (2023). Repofusion: Training code models to understand your repository. [find](https://scholar.google.com/scholar?q=Shrivastava%2C+D%2E%2C+Kocetkov%2C+D%2E%2C+de+Vries%2C+H%2E+et+al%2E+%282023%29%2E+Repofusion%3A+Training+code+models+to+understand+your+repository%2E)
46. Thoppilan, R., Freitas, D. D., Hall, J. et al. (2022). [Lamda: Language models for dialog applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/).
47. Touvron, H., Lavril, T., Izacard, G. et al. (2023). [LLaMA: open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
48. Wang, S., Li, Z., Qian, H. et al. (2022a). Recode: Robustness evaluation of code generation models. arXiv preprint arXiv:2212.10264. [find](https://scholar.google.com/scholar?q=Wang%2C+S%2E%2C+Li%2C+Z%2E%2C+Qian%2C+H%2E+et+al%2E+%282022a%29%2E+Recode%3A+Robustness+evaluation+of+code+generation+models%2E+arXiv+preprint+arXiv%3A2212%2E10264%2E)
49. Wang, Y., Wang, W., Joty, S. et al. (2021). Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. [find](https://scholar.google.com/scholar?q=Wang%2C+Y%2E%2C+Wang%2C+W%2E%2C+Joty%2C+S%2E+et+al%2E+%282021%29%2E+Codet5%3A+Identifier%2Daware+unified+pre%2Dtrained+encoder%2Ddecoder+models+for+code+understanding+and+generation%2E)
50. Wang, Y., Le, H., Gotmare, A. D. et al. (2023a). Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922. [find](https://scholar.google.com/scholar?q=Wang%2C+Y%2E%2C+Le%2C+H%2E%2C+Gotmare%2C+A%2E+D%2E+et+al%2E+%282023a%29%2E+Codet5%2B%3A+Open+code+large+language+models+for+code+understanding+and+generation%2E+arXiv+preprint+arXiv%3A2305%2E07922%2E)
51. Wang, Z., Zhou, S., Fried, D. et al. (2022b). [Execution-based evaluation for open-domain code generation](https://md.research.elamur.ai/free/en/papers/wang-2023-execution-based-evaluation-open-domain/). arXiv preprint arXiv:2212.10481.
52. Wang, Z., Cuenca, G., Zhou, S. et al. (2023b). Mconala: A benchmark for code generation from multiple natural languages. [find](https://scholar.google.com/scholar?q=Wang%2C+Z%2E%2C+Cuenca%2C+G%2E%2C+Zhou%2C+S%2E+et+al%2E+%282023b%29%2E+Mconala%3A+A+benchmark+for+code+generation+from+multiple+natural+languages%2E)
53. Wong, M.-F., Guo, S., Hang, C.-N. et al. (2023). Natural language generation and understanding of big code for ai-assisted programming: A review. Entropy, 25(6), 888. [find](https://scholar.google.com/scholar?q=Wong%2C+M%2E%2DF%2E%2C+Guo%2C+S%2E%2C+Hang%2C+C%2E%2DN%2E+et+al%2E+%282023%29%2E+Natural+language+generation+and+understanding+of+big+code+for+ai%2Dassisted+programming%3A+A+review%2E+Entropy%2C+25%286%29%2C+888%2E)
54. Xu, Q., Hong, F., Li, B. et al. (2023). [On the tool manipulation capability of open-source large language models](https://md.research.elamur.ai/free/en/papers/xu-2023-tool-manipulation-capability-open-source/). arXiv preprint arXiv:2305.16504.
55. Yu, H., Shen, B., Ran, D. et al. (2023). Codereval: A benchmark of pragmatic code generation with generative pre-trained models. arXiv preprint. [find](https://scholar.google.com/scholar?q=Yu%2C+H%2E%2C+Shen%2C+B%2E%2C+Ran%2C+D%2E+et+al%2E+%282023%29%2E+Codereval%3A+A+benchmark+of+pragmatic+code+generation+with+generative+pre%2Dtrained+models%2E+arXiv+preprint%2E)
56. Yuan, Z., Liu, J., Zi, Q. et al. (2023). Evaluating instruction-tuned large language models on code comprehension and generation. arXiv preprint arXiv:2308.01240. — [arXiv:2308.01240](https://arxiv.org/abs/2308.01240) [find](https://scholar.google.com/scholar?q=Yuan%2C+Z%2E%2C+Liu%2C+J%2E%2C+Zi%2C+Q%2E+et+al%2E+%282023%29%2E+Evaluating+instruction%2Dtuned+large+language+models+on+code+comprehension+and+generation%2E+arXiv+preprint+arXiv%3A2308%2E01240%2E+%E2%80%94+%5BarXiv%3A2308%2E01240%5D%28https%3A%2F%2Farxiv%2Eor)
57. Zan, D., Chen, B., Yang, D. et al. (2022a). Cert: Continual pre-training on sketches for library-oriented code generation. [find](https://scholar.google.com/scholar?q=Zan%2C+D%2E%2C+Chen%2C+B%2E%2C+Yang%2C+D%2E+et+al%2E+%282022a%29%2E+Cert%3A+Continual+pre%2Dtraining+on+sketches+for+library%2Doriented+code+generation%2E)
58. Zan, D., Chen, B., Lin, Z. et al. (2022b). When language model meets private library. In Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Zan%2C+D%2E%2C+Chen%2C+B%2E%2C+Lin%2C+Z%2E+et+al%2E+%282022b%29%2E+When+language+model+meets+private+library%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2E)
59. Zan, D., Chen, B., Zhang, F. et al. (2023). Large language models meet nl2code: A survey. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7443–7464. [find](https://scholar.google.com/scholar?q=Zan%2C+D%2E%2C+Chen%2C+B%2E%2C+Zhang%2C+F%2E+et+al%2E+%282023%29%2E+Large+language+models+meet+nl2code%3A+A+survey%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Pap)
60. Zeng, A., Liu, X., Du, Z. et al. (2022). [GLM-130B: an open bilingual pre-trained model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/). arXiv preprint arXiv:2210.02414.
61. Zhang, F., Chen, B., Zhang, Y. et al. (2023a). [find](https://scholar.google.com/scholar?q=Zhang%2C+F%2E%2C+Chen%2C+B%2E%2C+Zhang%2C+Y%2E+et+al%2E+%282023a%29%2E)
62. Zhang, S. D., Tigges, C., Biderman, S. et al. (2023b). [Can transformers learn to solve problems recursively?](https://md.research.elamur.ai/free/en/papers/zhang-2023-transformers-learn-solve-problems-recursively/) arXiv preprint arXiv:2305.14699.
63. Zheng, Q., Xia, X., Zou, X. et al. (2023). Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. [find](https://scholar.google.com/scholar?q=Zheng%2C+Q%2E%2C+Xia%2C+X%2E%2C+Zou%2C+X%2E+et+al%2E+%282023%29%2E+Codegeex%3A+A+pre%2Dtrained+model+for+code+generation+with+multilingual+evaluations+on+humaneval%2Dx%2E)
64. Zhuo, T. Y. (2023). [Large language models are state-of-the-art evaluators of code generation](https://md.research.elamur.ai/free/en/papers/zhuo-2023-ice-score-instructing-large-language/). arXiv preprint arXiv:2304.14317.

*Updated 3 November 2026*