---
title: "Prompting Large Language Models to Tackle the Full Software"
description: "Design is judged by an LLM. 37 references, cited by 8 papers here. Among them: 5."
authors: ["Bowen Li", "Wenhan Wu", "Ziwei Tang", "Lin Shi", "John Yang", "Jinyang Li", "Shunyu Yao", "Chen Qian", "Binyuan Hui", "Qicheng Zhang", "Zhiyin Yu", "He Du", "Ping Yang", "Dahua Lin", "Chao Peng", "Kai Chen"]
year: 2025
venue: "Proceedings of the 31st International Conference on Computational Linguistics"
source: "https://aclanthology.org/2025.coling-main.502/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2025-prompting-large-language-models-tackle/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2025-prompting-large-language-models-tackle/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/) / Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study

# Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study

**Authors:** [Bowen Li](https://md.research.elamur.ai/free/en/authors/bowen-li/), Wenhan Wu, Ziwei Tang, [Lin Shi](https://md.research.elamur.ai/free/en/authors/lin-shi/), [John Yang](https://md.research.elamur.ai/free/en/authors/john-yang/), [Jinyang Li](https://md.research.elamur.ai/free/en/authors/jinyang-li/), [Shunyu Yao](https://md.research.elamur.ai/free/en/authors/shunyu-yao/), [Chen Qian](https://md.research.elamur.ai/free/en/authors/chen-qian/), [Binyuan Hui](https://md.research.elamur.ai/free/en/authors/binyuan-hui/), Qicheng Zhang, [Zhiyin Yu](https://md.research.elamur.ai/free/en/authors/zhiyin-yu/), He Du, Ping Yang, [Dahua Lin](https://md.research.elamur.ai/free/en/authors/dahua-lin/), [Chao Peng](https://md.research.elamur.ai/free/en/authors/chao-peng/) and [Kai Chen](https://md.research.elamur.ai/free/en/authors/kai-chen/) · **Year:** 2025 · **Venue:** [Proceedings of the 31st International Conference on Computational Linguistics](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2025](https://md.research.elamur.ai/free/en/venues/coling/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.coling-main.502/) · [PDF](https://aclanthology.org/2025.coling-main.502.pdf) · [OpenAlex](https://openalex.org/W4392825180)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/li-2025-prompting-large-language-models-tackle/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most coding benchmarks for LLMs test narrow skills, such as producing one file or fixing one repository issue. They do not show whether a model can build software from start to finish. DevEval addresses this gap. It is a case study modelled on the Waterfall process. Starting from a detailed product requirement document (PRD), models must carry out five stages: software design (UML class and sequence diagrams in Mermaid, plus a file-tree architecture), environment setup, implementation of a multi-file codebase, acceptance testing and unit testing.

The authors curated 22 repositories in Python, C/C++, Java and JavaScript. Each stage receives reference inputs, so it can be scored on its own. Design is judged by an LLM. The other stages are scored by executing code and tests.

Every tested model struggles. GPT-4-Turbo scores best but solves under 10% of repository-level implementation. Prompts that add external information or execution feedback help consistently. The study is the first to assess design and environment setup, and it highlights recurring weaknesses such as build files, function arguments and object-oriented code.

## Key points
- **Dataset:** 22 repositories, split into Python (10), C/C++ (5), Java (5) and JavaScript (2). Domains include ML/NLP/CV, web services, databases, algorithms and CLI tools.
- **Environment setup:** dependency files are installed with Conda (Python), Gradle (Java) and NPM (JavaScript). C/C++ is excluded, and only 12 repositories are covered in this stage. GPT-4-Turbo reaches a 41.7% pass rate.
- **Testing:** oracle test scores for generated tests remain below 40%. When the generated tests do execute, coverage reaches up to 79.4%. DeepSeek-Coder-6.7B-Instruct inflated its acceptance-testing score by writing tests that run but check nothing meaningful.
- **Design judging:** GPT-4-Turbo serves as judge, with GPT-3.5-Turbo as the baseline. Excluding ties, the judge agrees with the human majority 79.2% of the time on general principles and 83.2% on faithfulness.
- **Error analysis:** models frequently fail on Makefiles and Gradle files, parameter usage, naming, types, variable scope and file references. They also hallucinate variables. Java and C/C++ prove hardest to implement.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 333 works · 87,019 citations
- [Princeton University](https://md.research.elamur.ai/free/en/universities/princeton-university/) — University, Princeton, United States · 90 works · 39,109 citations
- [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) — Company, Hangzhou, China · 168 works · 15,386 citations
- [University of Hong Kong](https://md.research.elamur.ai/free/en/universities/university-of-hong-kong/) — University, Pok Fu Lam, China · 89 works · 11,774 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 117 works · 11,281 citations
- [Beijing University of Posts and Telecommunications](https://md.research.elamur.ai/free/en/universities/beijing-university-of-posts-and-telecommunications/) — University, Beijing, China · 57 works · 32,843 citations
- [ByteDance](https://md.research.elamur.ai/free/en/companies/bytedance/) — Company, Beijing, China · 41 works · 4,180 citations
- [Nanjing University](https://md.research.elamur.ai/free/en/universities/nanjing-university/) — University, Nanjing, China · 74 works · 7,814 citations
- [Dartmouth College](https://md.research.elamur.ai/free/en/universities/dartmouth-college/) — University, Hanover, United States · 37 works · 42,625 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works · [💻 Software](https://md.research.elamur.ai/free/en/shelves/computer-science/software/) · 92 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, Zhiyin Yu, He Du, Ping Yang, Dahua Lin, Chao Peng and Kai Chen. 2025. Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{li2025prompting,
  title = {{Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study}},
  author = {Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, Zhiyin Yu, He Du, Ping Yang, Dahua Lin, Chao Peng and Kai Chen},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.502/}
}
```

- [← previous PrivacyRestore: Privacy-Preserving Inference in Large Language Models via Privacy Removal and Restoration](https://md.research.elamur.ai/free/en/papers/zeng-2025-privacyrestore-privacy-preserving-inference-large/)
- [next → QCRD: Quality-guided Contrastive Rationale Distillation for Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2025-qcrd-quality-guided-contrastive-rationale/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/)

## References

37 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). Preprint, arXiv:2108.07732.
2. Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, et al. 2023. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXiv:2308.09687. [find](https://scholar.google.com/scholar?q=Maciej+Besta%2C+Nils+Blach%2C+Ales+Kubicek%2C+Robert+Gerstenberger%2C+Lukas+Gianinazzi%2C+Joanna+Gajda%2C+Tomasz+Lehmann%2C+Michal+Podstawski%2C+Hubert+Niewiadomski%2C+Piotr+Nyczyk%2C+et+al%2E+2023%2E+Graph+of+thoughts%3A+Solv)
3. Federico Cassano, John Gouwar, Daniel Nguyen, Sy Duy Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. 2023. Multipl-e: A scalable and polyglot approach to benchmarking neural code genera… [find](https://scholar.google.com/scholar?q=Federico+Cassano%2C+John+Gouwar%2C+Daniel+Nguyen%2C+Sy+Duy+Nguyen%2C+Luna+Phipps%2DCostin%2C+Donald+Pinckney%2C+Ming%2DHo+Yee%2C+Yangtian+Zi%2C+Carolyn+Jane+Anderson%2C+Molly+Q%2E+Feldman%2C+Arjun+Guha%2C+Michael+Greenberg%2C+and+)
4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
5. Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2023. [Metagpt: Meta programming for a multi-agent collaborative framework](https://md.research.elamur.ai/free/en/papers/hong-2023-metagpt-meta-programming-multi/). Preprint, arXiv:2308.00352.
6. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. [Swe-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) arXiv preprint arXiv:2310.06770.
7. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Yih, Daniel Fried, Si yi Wang, and Tao Yu. 2022. Ds-1000: A natural and reliable benchmark for data science code generation. ArXiv, abs/2211.11501. [find](https://scholar.google.com/scholar?q=Yuhang+Lai%2C+Chengxi+Li%2C+Yiming+Wang%2C+Tianyi+Zhang%2C+Ruiqi+Zhong%2C+Luke+Zettlemoyer%2C+Scott+Yih%2C+Daniel+Fried%2C+Si+yi+Wang%2C+and+Tao+Yu%2E+2022%2E+Ds%2D1000%3A+A+natural+and+reliable+benchmark+for+data+science+code)
8. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. [Camel: Communicative agents for" mind" exploration of large scale language model society](https://md.research.elamur.ai/free/en/papers/li-2023-camel-communicative-agents-mind/). arXiv preprint arXiv:2303.17760.
9. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023a. [Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/). arXiv preprint arXiv:2305.01210.
10. Tianyang Liu, Canwen Xu, and Julian McAuley. 2023b. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091. [find](https://scholar.google.com/scholar?q=Tianyang+Liu%2C+Canwen+Xu%2C+and+Julian+McAuley%2E+2023b%2E+Repobench%3A+Benchmarking+repository%2Dlevel+code+auto%2Dcompletion+systems%2E+arXiv+preprint+arXiv%3A2306%2E03091%2E)
11. Yizhou Liu, Pengfei Gao, Xinchen Wang, Jie Liu, Yexuan Shi, Zhao Zhang, and Chao Peng. 2024. Marscode agent: Ai-native automated bug fixing. Preprint, arXiv:2409.00899. — [arXiv:2409.00899](https://arxiv.org/abs/2409.00899) [find](https://scholar.google.com/scholar?q=Yizhou+Liu%2C+Pengfei+Gao%2C+Xinchen+Wang%2C+Jie+Liu%2C+Yexuan+Shi%2C+Zhao+Zhang%2C+and+Chao+Peng%2E+2024%2E+Marscode+agent%3A+Ai%2Dnative+automated+bug+fixing%2E+Preprint%2C+arXiv%3A2409%2E00899%2E+%E2%80%94+%5BarXiv%3A2409%2E00899%5D%28https%3A%2F%2Far)
12. Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2023. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. ArXiv, abs/2310.11248. — [arXiv:2310.11248](https://arxiv.org/abs/2310.11248) [find](https://scholar.google.com/scholar?q=Yangruibo+Ding%2C+Zijian+Wang%2C+Wasi+Uddin+Ahmad%2C+Hantian+Ding%2C+Ming+Tan%2C+Nihal+Jain%2C+Murali+Krishna+Ramanathan%2C+Ramesh+Nallapati%2C+Parminder+Bhatia%2C+Dan+Roth%2C+and+Bing+Xiang%2E+2023%2E+Crosscodeeval%3A+A+diver)
13. Yuliang Liu, Xiangru Tang, Zefan Cai, Junjie Lu, Yichi Zhang, Yan Shao, Zexuan Deng, Helan Hu, Zengxian Yang, Kaikai An, Ruijun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Zheng Li, Liang Chen, Yiming Zong, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yujia Qin, Wangchunshu Zhou, Yilun Zhao, A… — [ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code](https://md.research.elamur.ai/free/en/papers/tang-2023-ml-bench-evaluating-large-language/)
14. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. 2024. Deepseek-coder: When the large language model meets programming – the rise of code intelligence. Preprint, arXiv:2401.14196. [find](https://scholar.google.com/scholar?q=Daya+Guo%2C+Qihao+Zhu%2C+Dejian+Yang%2C+Zhenda+Xie%2C+Kai+Dong%2C+Wentao+Zhang%2C+Guanting+Chen%2C+Xiao+Bi%2C+Y%2E+Wu%2C+Y%2E+K%2E+Li%2C+Fuli+Luo%2C+Yingfei+Xiong%2C+and+Wenfeng+Liang%2E+2024%2E+Deepseek%2Dcoder%3A+When+the+large+language)
15. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664. [find](https://scholar.google.com/scholar?q=Shuai+Lu%2C+Daya+Guo%2C+Shuo+Ren%2C+Junjie+Huang%2C+Alexey+Svyatkovskiy%2C+Ambrosio+Blanco%2C+Colin+Clement%2C+Dawn+Drain%2C+Daxin+Jiang%2C+Duyu+Tang%2C+et+al%2E+2021%2E+Codexglue%3A+A+machine+learning+benchmark+dataset+for+co)
16. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. [Measuring coding challenge competence with apps](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/). NeurIPS.
17. André Miranda and João Pimentel. 2018. On the use of package managers by the c++ open-source community. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing, pages 1483–1491. [find](https://scholar.google.com/scholar?q=Andr%C3%A9+Miranda+and+Jo%C3%A3o+Pimentel%2E+2018%2E+On+the+use+of+package+managers+by+the+c%2B%2B+open%2Dsource+community%2E+In+Proceedings+of+the+33rd+Annual+ACM+Symposium+on+Applied+Computing%2C+pages+1483%E2%80%931491%2E)
18. Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937. [find](https://scholar.google.com/scholar?q=Cheng%2DHan+Chiang+and+Hung%2Dyi+Lee%2E+2023%2E+Can+large+language+models+be+an+alternative+to+human+evaluations%3F+arXiv+preprint+arXiv%3A2305%2E01937%2E)
19. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Niklas Muennighoff, Qian Liu, Qi Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and S. Longpre. 2023. Octopack: Instruction tuning code large… [find](https://scholar.google.com/scholar?q=Sirui+Hong%2C+Mingchen+Zhuge%2C+Jonathan+Chen%2C+Xiawu+Zheng%2C+Yuheng+Cheng%2C+Ceyao+Zhang%2C+Jinlin+Wang%2C+Niklas+Muennighoff%2C+Qian+Liu%2C+Qi+Liu%2C+Armel+Zebaze%2C+Qinkai+Zheng%2C+Binyuan+Hui%2C+Terry+Yue+Zhuo%2C+Swayam+Si)
20. OpenAI. 2023. Openai gpt. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023%2E+Openai+gpt%2E)
21. John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. 2023. [Intercode: Standardizing and benchmarking interactive coding with execution feedback](https://md.research.elamur.ai/free/en/papers/yang-2023-intercode-standardizing-benchmarking-interactive-coding/). ArXiv, abs/2306.14898.
22. Chen Qian, Xin Cong, Wei Liu, Cheng Yang, Weize Chen, Yusheng Su, Yufan Dang, Jiahao Li, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023a. [Communicative agents for software development](https://md.research.elamur.ai/free/en/papers/qian-2023-chatdev-communicative-agents-software/). Preprint, arXiv:2307.07924.
23. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+2023%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+arXiv+preprint+ar)
24. Chen Qian, Yufan Dang, Jiahao Li, Wei Liu, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. 2023b. Experiential co-learning of software-developing agents. arXiv preprint arXiv:2312.17025. [find](https://scholar.google.com/scholar?q=Chen+Qian%2C+Yufan+Dang%2C+Jiahao+Li%2C+Wei+Liu%2C+Weize+Chen%2C+Cheng+Yang%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2023b%2E+Experiential+co%2Dlearning+of+software%2Ddeveloping+agents%2E+arXiv+preprint+arXiv%3A2312%2E17025%2E)
25. Pengcheng Yin, Wen-Ding Li, Kefan Xiao, A. Eashaan Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, and Charles Sutton. 2022. Natural language to code generation in interactive data science notebooks. ArXiv, abs/2212.09248. [find](https://scholar.google.com/scholar?q=Pengcheng+Yin%2C+Wen%2DDing+Li%2C+Kefan+Xiao%2C+A%2E+Eashaan+Rao%2C+Yeming+Wen%2C+Kensen+Shi%2C+Joshua+Howland%2C+Paige+Bailey%2C+Michele+Catasta%2C+Henryk+Michalewski%2C+Oleksandr+Polozov%2C+and+Charles+Sutton%2E+2022%2E+Natural+)
26. Winston W Royce. 1987. Managing the development of large software systems: concepts and techniques. In Proceedings of the 9th international conference on Software Engineering, pages 328–338. [find](https://scholar.google.com/scholar?q=Winston+W+Royce%2E+1987%2E+Managing+the+development+of+large+software+systems%3A+concepts+and+techniques%2E+In+Proceedings+of+the+9th+international+conference+on+Software+Engineering%2C+pages+328%E2%80%93338%2E)
27. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950.
28. Lin Shi, Chiyu Ma, Wenhua Liang, Weicheng Ma, and Soroush Vosoughi. 2024. Judging the judges: A systematic investigation of position bias in pairwise comparative assessments by llms. Preprint, arXiv:2406.07791. — [arXiv:2406.07791](https://arxiv.org/abs/2406.07791) [find](https://scholar.google.com/scholar?q=Lin+Shi%2C+Chiyu+Ma%2C+Wenhua+Liang%2C+Weicheng+Ma%2C+and+Soroush+Vosoughi%2E+2024%2E+Judging+the+judges%3A+A+systematic+investigation+of+position+bias+in+pairwise+comparative+assessments+by+llms%2E+Preprint%2C+arXiv%3A2)
29. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926. [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Qi+Liu%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+2023%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv+preprint+arXiv%3A2305%2E17926%2E)
30. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, a… — [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://md.research.elamur.ai/free/en/papers/wang-2024-openhands-open-platform-ai/)
31. Fengji Zhang, B. Chen, Yue Zhang, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. In Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Fengji+Zhang%2C+B%2E+Chen%2C+Yue+Zhang%2C+Jin+Liu%2C+Daoguang+Zan%2C+Yi+Mao%2C+Jian%2DGuang+Lou%2C+and+Weizhu+Chen%2E+2023%2E+Repocoder%3A+Repository%2Dlevel+code+completion+through+iterative+retrieval+and+generation%2E+In+Confe)
32. Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Autocoderover: Autonomous program improvement. Preprint, arXiv:2404.05427. — [arXiv:2404.05427](https://arxiv.org/abs/2404.05427) [find](https://scholar.google.com/scholar?q=Yuntong+Zhang%2C+Haifeng+Ruan%2C+Zhiyu+Fan%2C+and+Abhik+Roychoudhury%2E+2024%2E+Autocoderover%3A+Autonomous+program+improvement%2E+Preprint%2C+arXiv%3A2404%2E05427%2E+%E2%80%94+%5BarXiv%3A2404%2E05427%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2404%2E05427%29)
33. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023a. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). arXiv preprint arXiv:2306.05685.
34. Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al. 2023b. [Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x](https://md.research.elamur.ai/free/en/papers/zheng-2023-codegeex-pre-trained-model-code/). arXiv preprint arXiv:2303.17568.
35. Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. 2022. Execution-based evaluation for open-domain code generation. In Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Zhiruo+Wang%2C+Shuyan+Zhou%2C+Daniel+Fried%2C+and+Graham+Neubig%2E+2022%2E+Execution%2Dbased+evaluation+for+open%2Ddomain+code+generation%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2E)
36. Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. 2024b. [Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents](https://md.research.elamur.ai/free/en/papers/wang-2023-describe-explain-plan-select-interactive/). Preprint, arXiv:2302.01560.
37. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. [Swe-agent: Agent-computer interfaces enable automated software engineering](https://md.research.elamur.ai/free/en/papers/yang-2024-swe-agent-agent-computer-interfaces/). Preprint, arXiv:2405.15793.

*Updated 1 November 2026*