---
title: "Self-collaboration Code Generation via ChatGPT — Jin; 2023"
description: "In the proposed self-collaboration framework, several instances of the same LLM (ChatGPT… 65 references, cited by 69 papers here."
authors: ["Yihong Dong; Xue Jiang; Zhi Jin;", "Ge Li"]
year: 2023
venue: "arXiv:2304.07590"
source: "https://arxiv.org/abs/2304.07590"
doi: "10.48550/arxiv.2304.07590"
arxiv: "2304.07590"
licence: "arXiv non-exclusive distribution license"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/dong-2023-self-collaboration-code-generation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dong-2023-self-collaboration-code-generation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/) / Self-collaboration Code Generation via ChatGPT

# Self-collaboration Code Generation via ChatGPT

**Authors:** Yihong Dong; Xue Jiang; Zhi Jin; and [Ge Li](https://md.research.elamur.ai/free/en/authors/ge-li/) · **Year:** 2023 · **Venue:** [arXiv:2304.07590](https://md.research.elamur.ai/free/en/venues/acm-transactions-on-software-engineering-and-methodology/) ([ACM Transactions on Software Engineering and Methodology 2023](https://md.research.elamur.ai/free/en/venues/acm-transactions-on-software-engineering-and-methodology/2023/)) · **Publisher:** [Association for Computing Machinery](https://md.research.elamur.ai/free/en/publishers/association-for-computing-machinery/) · **Licence:** [arXiv non-exclusive distribution license](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2304.07590) · [PDF](https://arxiv.org/pdf/2304.07590) · [DOI 10.48550/arxiv.2304.07590](https://doi.org/10.48550/arxiv.2304.07590) · [arXiv 2304.07590](https://arxiv.org/abs/2304.07590) · [OpenAlex](https://openalex.org/W4399557965)

**Citations:** 250 citations (OpenAlex, 2026-10-14) · [69](https://md.research.elamur.ai/free/en/papers/dong-2023-self-collaboration-code-generation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models write code well for simple prompts but often fail on complicated requirements. Human developers handle such complexity by splitting work across a team. This paper brings that idea to a single model. In the proposed self-collaboration framework, several instances of the same LLM (ChatGPT, GPT-3.5) receive role instructions. These instructions assign each agent a specialised job and define how the agents exchange natural-language messages, so they behave like a small, fully automatic development team.

The team structure borrows from software-development methodology (SDM) and has three roles:
- an analyst, who decomposes the requirement and drafts a plan;
- a coder, who writes or revises the program;
- a tester, who reports problems back to the coder.

On several code-generation benchmarks, this setup raised Pass@1 noticeably over a lone ChatGPT agent. Case studies also suggest the team can tackle repository-level tasks that a single agent struggles with. The approach matters because it improves results with prompting alone, with no extra training or hand-built few-shot examples.

## Key points
- **Relative gains:** Pass@1 improved by 29.9%–47.1% relative to the single base LLM agent across the evaluated benchmarks.
- **Workflow:** analysis, coding and testing run in sequence, and each stage can send feedback to the stage before it. Role instructions are supplied only once, when each agent is initialised.
- **Role-playing:** giving the model an expert persona worked better empirically than prompting it directly with no role context.
- **Related methods:** the paper contrasts its framework with reranking approaches (CodeT, Coder-Reviewer) and few-shot refinement methods (Self-planning, Self-debugging).
- **Limitations:** the authors note gaps between current benchmarks and real development work, and suggest adding light human oversight. The code is released at github.com/YihongDong/Self-collaboration-Code-Generation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,267 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (1): [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 168 works

## Cite

Cite as: Yihong Dong; Xue Jiang; Zhi Jin; and Ge Li. 2023. Self-collaboration Code Generation via ChatGPT. arXiv:2304.07590.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{jin2023self,
  title = {{Self-collaboration Code Generation via ChatGPT}},
  author = {Yihong Dong; Xue Jiang; Zhi Jin; and Ge Li},
  year = {2023},
  doi = {10.48550/arxiv.2304.07590},
  eprint = {2304.07590},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.07590}
}
```

- [← previous LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://md.research.elamur.ai/free/en/papers/jiang-2023-llm-blender-ensembling-large-language/)
- [next → Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech](https://md.research.elamur.ai/free/en/papers/huang-2023-chatgpt-better-than-human-annotators/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/)

## References

65 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Pekka Abrahamsson, Outi Salo, Jussi Ronkainen, and Juhani Warsta. 2002. Agile software development methods: Review and analysis. (2002). [find](https://scholar.google.com/scholar?q=Pekka+Abrahamsson%2C+Outi+Salo%2C+Jussi+Ronkainen%2C+and+Juhani+Warsta%2E+2002%2E+Agile+software+development+methods%3A+Review+and+analysis%2E+%282002%29%2E)
2. Chetan Arora, John Grundy, and Mohamed Abdelrazek. 2023. Advancing Requirements Engineering through Generative AI: Assessing the Role of LLMs. CoRR abs/2310.13976 (2023). — [arXiv:2310.13976](https://arxiv.org/abs/2310.13976) [find](https://scholar.google.com/scholar?q=Chetan+Arora%2C+John+Grundy%2C+and+Mohamed+Abdelrazek%2E+2023%2E+Advancing+Requirements+Engineering+through+Generative+AI%3A+Assessing+the+Role+of+LLMs%2E+CoRR+abs%2F2310%2E13976+%282023%29%2E+%E2%80%94+%5BarXiv%3A2310%2E13976%5D%28https%3A%2F%2F)
3. Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. [Program Synthesis with Large Language Models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). CoRR abs/2108.07732 (2021).
4. Kent Beck, Mike Beedle, Arie Van Bennekum, Alistair Cockburn, Ward Cunningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, et al. 2001. Manifesto for agile software development. (2001). [find](https://scholar.google.com/scholar?q=Kent+Beck%2C+Mike+Beedle%2C+Arie+Van+Bennekum%2C+Alistair+Cockburn%2C+Ward+Cunningham%2C+Martin+Fowler%2C+James+Grenning%2C+Jim+Highsmith%2C+Andrew+Hunt%2C+Ron+Jeffries%2C+et+al%2E+2001%2E+Manifesto+for+agile+software+develo)
5. R Meredith Belbin. 2012. Team roles at work. Routledge. [find](https://scholar.google.com/scholar?q=R+Meredith+Belbin%2E+2012%2E+Team+roles+at+work%2E+Routledge%2E)
6. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
7. Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. CodeT: Code Generation with Generated Tests. CoRR abs/2207.10397 (2022). [find](https://scholar.google.com/scholar?q=Bei+Chen%2C+Fengji+Zhang%2C+Anh+Nguyen%2C+Daoguang+Zan%2C+Zeqi+Lin%2C+Jian%2DGuang+Lou%2C+and+Weizhu+Chen%2E+2022%2E+CodeT%3A+Code+Generation+with+Generated+Tests%2E+CoRR+abs%2F2207%2E10397+%282022%29%2E)
8. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mik… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
9. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching Large Language Models to Self-Debug. CoRR abs/2304.05128 (2023). [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Maxwell+Lin%2C+Nathanael+Sch%C3%A4rli%2C+and+Denny+Zhou%2E+2023%2E+Teaching+Large+Language+Models+to+Self%2DDebug%2E+CoRR+abs%2F2304%2E05128+%282023%29%2E)
10. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
11. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang… [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+Albert+Webson%2C+Shixiang+Shane+Gu%2C+Zhuyun+Dai%2C+Mirac+Suzgun%2C+Xiny)
12. Caroline Claus and Craig Boutilier. 1998. The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems. In AAAI/IAAI. AAAI Press / The MIT Press, 746–752. [find](https://scholar.google.com/scholar?q=Caroline+Claus+and+Craig+Boutilier%2E+1998%2E+The+Dynamics+of+Reinforcement+Learning+in+Cooperative+Multiagent+Systems%2E+In+AAAI%2FIAAI%2E+AAAI+Press+%2F+The+MIT+Press%2C+746%E2%80%93752%2E)
13. Tom DeMarco and Tim Lister. 2013. Peopleware: productive projects and teams. Addison-Wesley. [find](https://scholar.google.com/scholar?q=Tom+DeMarco+and+Tim+Lister%2E+2013%2E+Peopleware%3A+productive+projects+and+teams%2E+Addison%2DWesley%2E)
14. Yihong Dong, Jiazheng Ding, Xue Jiang, Zhuo Li, Ge Li, and Zhi Jin. 2023. CodeScore: Evaluating Code Generation by Learning Code Execution. CoRR abs/2301.09043 (2023). — [arXiv:2301.09043](https://arxiv.org/abs/2301.09043) [find](https://scholar.google.com/scholar?q=Yihong+Dong%2C+Jiazheng+Ding%2C+Xue+Jiang%2C+Zhuo+Li%2C+Ge+Li%2C+and+Zhi+Jin%2E+2023%2E+CodeScore%3A+Evaluating+Code+Generation+by+Learning+Code+Execution%2E+CoRR+abs%2F2301%2E09043+%282023%29%2E+%E2%80%94+%5BarXiv%3A2301%2E09043%5D%28https%3A%2F%2Farx)
15. Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, and Ge Li. 2024. Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models. CoRR abs/2402.15938 (2024). [find](https://scholar.google.com/scholar?q=Yihong+Dong%2C+Xue+Jiang%2C+Huanyu+Liu%2C+Zhi+Jin%2C+and+Ge+Li%2E+2024%2E+Generalization+or+Memorization%3A+Data+Contamination+and+Trustworthy+Evaluation+for+Large+Language+Models%2E+CoRR+abs%2F2402%2E15938+%282024%29%2E)
16. Yihong Dong, Ge Li, and Zhi Jin. 2023. CODEP: Grammatical Seq2Seq Model for General-Purpose Code Generation. In ISSTA. ACM, 188–198. [find](https://scholar.google.com/scholar?q=Yihong+Dong%2C+Ge+Li%2C+and+Zhi+Jin%2E+2023%2E+CODEP%3A+Grammatical+Seq2Seq+Model+for+General%2DPurpose+Code+Generation%2E+In+ISSTA%2E+ACM%2C+188%E2%80%93198%2E)
17. Yihong Dong, Kangcheng Luo, Xue Jiang, Zhi Jin, and Ge Li. 2023. PACE: Improving Prompt with Actor-Critic Editing for Large Language Model. CoRR abs/2308.10088 (2023). [find](https://scholar.google.com/scholar?q=Yihong+Dong%2C+Kangcheng+Luo%2C+Xue+Jiang%2C+Zhi+Jin%2C+and+Ge+Li%2E+2023%2E+PACE%3A+Improving+Prompt+with+Actor%2DCritic+Editing+for+Large+Language+Model%2E+CoRR+abs%2F2308%2E10088+%282023%29%2E)
18. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. InCoder: A Generative Model for Code Infilling and Synthesis. CoRR abs/2204.05999 (2022). [find](https://scholar.google.com/scholar?q=Daniel+Fried%2C+Armen+Aghajanyan%2C+Jessy+Lin%2C+Sida+Wang%2C+Eric+Wallace%2C+Freda+Shi%2C+Ruiqi+Zhong%2C+Wen%2Dtau+Yih%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+2022%2E+InCoder%3A+A+Generative+Model+for+Code+Infilling+and+Synt)
19. GitHub. 2022. Copilot. [https://github.com/features/copilot](https://github.com/features/copilot) [↗](https://github.com/features/copilot)
20. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. [Measuring Coding Challenge Competence With APPS](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/). In NeurIPS Datasets and Benchmarks.
21. Huggingface. 2023. HuggingChat. [https://huggingface.co/chat/](https://huggingface.co/chat/) [↗](https://huggingface.co/chat/)
22. IBM. 2023. Dromedary. [https://huggingface.co/ibm/dromedary-65b-lora-delta-v0](https://huggingface.co/ibm/dromedary-65b-lora-delta-v0) [↗](https://huggingface.co/ibm/dromedary-65b-lora-delta-v0)
23. Xue Jiang, Yihong Dong, Lecheng Wang, Qiwei Shang, and Ge Li. 2023. [Self-planning Code Generation with Large Language Model](https://md.research.elamur.ai/free/en/papers/jiang-2023-self-planning-code-generation-large/). CoRR abs/2303.06689 (2023).
24. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. [How Can We Know What Language Models Know](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/). Trans. Assoc. Comput. Linguistics 8 (2020), 423–438.
25. Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2022. Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction. CoRR abs/2209.11515 (2022). — [arXiv:2209.11515](https://arxiv.org/abs/2209.11515) [find](https://scholar.google.com/scholar?q=Sungmin+Kang%2C+Juyeon+Yoon%2C+and+Shin+Yoo%2E+2022%2E+Large+Language+Models+are+Few%2Dshot+Testers%3A+Exploring+LLM%2Dbased+General+Bug+Reproduction%2E+CoRR+abs%2F2209%2E11515+%282022%29%2E+%E2%80%94+%5BarXiv%3A2209%2E11515%5D%28https%3A%2F%2Farxiv%2E)
26. Jon R Katzenbach and Douglas K Smith. 2015. The wisdom of teams: Creating the high-performance organization. Harvard Business Review Press. [find](https://scholar.google.com/scholar?q=Jon+R+Katzenbach+and+Douglas+K+Smith%2E+2015%2E+The+wisdom+of+teams%3A+Creating+the+high%2Dperformance+organization%2E+Harvard+Business+Review+Press%2E)
27. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large Language Models are Zero-Shot Reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). In NeurIPS.
28. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. [CAMEL: Communicative Agents for" Mind" Exploration of Large Scale Language Model Society](https://md.research.elamur.ai/free/en/papers/li-2023-camel-communicative-agents-mind/). CoRR abs/2303.17760 (2023).
29. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas… [find](https://scholar.google.com/scholar?q=Raymond+Li%2C+Loubna+Ben+Allal%2C+Yangtian+Zi%2C+Niklas+Muennighoff%2C+Denis+Kocetkov%2C+Chenghao+Mou%2C+Marc+Marone%2C+Christopher+Akiki%2C+Jia+Li%2C+Jenny+Chim%2C+Qian+Liu%2C+Evgenii+Zheltonozhskii%2C+Terry+Yue+Zhuo%2C+Thoma)
30. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. [Competition-level code generation with alphacode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/). Science 378, 6624 (2022), 1092–1097.
31. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/). ACM Comput. Surv. 55, 9 (2023), 195:1–195:35.
32. LMSYS. 2023. Fastchat. [https://huggingface.co/lmsys/fastchat-t5-3b-v1.0](https://huggingface.co/lmsys/fastchat-t5-3b-v1.0) [↗](https://huggingface.co/lmsys/fastchat-t5-3b-v1.0)
33. LMSYS. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. [https://lmsys.org/blog/2023-03-30-vicuna/](https://lmsys.org/blog/2023-03-30-vicuna/) [↗](https://lmsys.org/blog/2023-03-30-vicuna/)
34. Ian R McChesney and Seamus Gallagher. 2004. Communication and co-ordination practices in software engineering projects. Information and Software Technology 46, 7 (2004), 473–489. [find](https://scholar.google.com/scholar?q=Ian+R+McChesney+and+Seamus+Gallagher%2E+2004%2E+Communication+and+co%2Dordination+practices+in+software+engineering+projects%2E+Information+and+Software+Technology+46%2C+7+%282004%29%2C+473%E2%80%93489%2E)
35. Marvin Minsky. 2007. The emotion machine: Commonsense thinking, artificial intelligence, and the future of the human mind. Simon and Schuster. [find](https://scholar.google.com/scholar?q=Marvin+Minsky%2E+2007%2E+The+emotion+machine%3A+Commonsense+thinking%2C+artificial+intelligence%2C+and+the+future+of+the+human+mind%2E+Simon+and+Schuster%2E)
36. MosaicML. 2023. Introducing MPT-7B: A New Standard for Open-Source, ly Usable LLMs. www.mosaicml.com/blog/mpt-7b [find](https://scholar.google.com/scholar?q=MosaicML%2E+2023%2E+Introducing+MPT%2D7B%3A+A+New+Standard+for+Open%2DSource%2C+ly+Usable+LLMs%2E+www%2Emosaicml%2Ecom%2Fblog%2Fmpt%2D7b)
37. H Penny Nii. 1986. Blackboard Systems. (1986). [find](https://scholar.google.com/scholar?q=H+Penny+Nii%2E+1986%2E+Blackboard+Systems%2E+%281986%29%2E)
38. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. CoRR abs/2203.13474 (2022). [find](https://scholar.google.com/scholar?q=Erik+Nijkamp%2C+Bo+Pang%2C+Hiroaki+Hayashi%2C+Lifu+Tu%2C+Huan+Wang%2C+Yingbo+Zhou%2C+Silvio+Savarese%2C+and+Caiming+Xiong%2E+2022%2E+Codegen%3A+An+open+large+language+model+for+code+with+multi%2Dturn+program+synthesis%2E+CoR)
39. OpenAI. 2022. ChatGPT. [https://openai.com/blog/chatgpt/](https://openai.com/blog/chatgpt/) [↗](https://openai.com/blog/chatgpt/)
40. OpenAI. 2023. [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). CoRR abs/2303.08774 (2023).
41. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Tr… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
42. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Tr… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
43. Kai Petersen, Claes Wohlin, and Dejan Baca. 2009. The Waterfall Model in Large-Scale Development. In PROFES (Lecture years in Business Information Processing, Vol. 32). Springer, 386–400. [find](https://scholar.google.com/scholar?q=Kai+Petersen%2C+Claes+Wohlin%2C+and+Dejan+Baca%2E+2009%2E+The+Waterfall+Model+in+Large%2DScale+Development%2E+In+PROFES+%28Lecture+years+in+Business+Information+Processing%2C+Vol%2E+32%29%2E+Springer%2C+386%E2%80%93400%2E)
44. Laria Reynolds and Kyle McDonell. 2021. [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://md.research.elamur.ai/free/en/papers/reynolds-2021-prompt-programming-large-language-models/). In CHI Extended Abstracts. ACM, 314:1–314:7.
45. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal… — [Code Llama: Open Foundation Models for Code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
46. Nayan B. Ruparelia. 2010. Software development lifecycle models. ACM SIGSOFT Softw. Eng. years 35, 3 (2010), 8–13. [find](https://scholar.google.com/scholar?q=Nayan+B%2E+Ruparelia%2E+2010%2E+Software+development+lifecycle+models%2E+ACM+SIGSOFT+Softw%2E+Eng%2E+years+35%2C+3+%282010%29%2C+8%E2%80%9313%2E)
47. Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. IEEE Trans. Software Eng. 50, 1 (2024), 85–105. [find](https://scholar.google.com/scholar?q=Max+Sch%C3%A4fer%2C+Sarah+Nadi%2C+Aryaz+Eghbali%2C+and+Frank+Tip%2E+2024%2E+An+Empirical+Evaluation+of+Using+Large+Language+Models+for+Automated+Unit+Test+Generation%2E+IEEE+Trans%2E+Software+Eng%2E+50%2C+1+%282024%29%2C+85%E2%80%93105%2E)
48. Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick S. H. Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. 2022. [PEER: A Collaborative Language Model](https://md.research.elamur.ai/free/en/papers/schick-2022-peer-collaborative-language-model/). CoRR abs/2208.11663 (2022).
49. Sijie Shen, Xiang Zhu, Yihong Dong, Qizhi Guo, Yankun Zhen, and Ge Li. 2022. Incorporating domain knowledge through task augmentation for front-end JavaScript code generation. In ESEC/SIGSOFT FSE. ACM, 1533–1543. [find](https://scholar.google.com/scholar?q=Sijie+Shen%2C+Xiang+Zhu%2C+Yihong+Dong%2C+Qizhi+Guo%2C+Yankun+Zhen%2C+and+Ge+Li%2E+2022%2E+Incorporating+domain+knowledge+through+task+augmentation+for+front%2Dend+JavaScript+code+generation%2E+In+ESEC%2FSIGSOFT+FSE%2E+ACM)
50. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. [Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface](https://md.research.elamur.ai/free/en/papers/shen-2023-hugginggpt-solving-ai-tasks-chatgpt/). CoRR abs/2303.17580 (2023).
51. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/). In EMNLP (1). Association for Computational Linguistics, 4222–4235.
52. Stephen W. Smoliar. 1991. Marvin Minsky, The Society of Mind. Artif. Intell. 48, 3 (1991), 349–370. [find](https://scholar.google.com/scholar?q=Stephen+W%2E+Smoliar%2E+1991%2E+Marvin+Minsky%2C+The+Society+of+Mind%2E+Artif%2E+Intell%2E+48%2C+3+%281991%29%2C+349%E2%80%93370%2E)
53. Chang-You Tai, Ziru Chen, Tianshu Zhang, Xiang Deng, and Huan Sun. 2023. Exploring Chain-of-Thought Style Prompting for Text-to-SQL. CoRR abs/2305.14215 (2023). [find](https://scholar.google.com/scholar?q=Chang%2DYou+Tai%2C+Ziru+Chen%2C+Tianshu+Zhang%2C+Xiang+Deng%2C+and+Huan+Sun%2E+2023%2E+Exploring+Chain%2Dof%2DThought+Style+Prompting+for+Text%2Dto%2DSQL%2E+CoRR+abs%2F2305%2E14215+%282023%29%2E)
54. THUDM. 2023. ChatGLM. [https://huggingface.co/THUDM/chatglm-6b](https://huggingface.co/THUDM/chatglm-6b) [↗](https://huggingface.co/THUDM/chatglm-6b)
55. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
56. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). CoRR abs/2201.11903 (2022).
57. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In NeurIPS.
58. Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. 2023. [Visual chatgpt: Talking, drawing and editing with visual foundation models](https://md.research.elamur.ai/free/en/papers/wu-2023-visual-chatgpt-talking-drawing-editing/). CoRR abs/2303.04671 (2023).
59. Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2022. Practical Program Repair in the Era of Large Pre-trained Language Models. CoRR abs/2210.14179 (2022). — [arXiv:2210.14179](https://arxiv.org/abs/2210.14179) [find](https://scholar.google.com/scholar?q=Chunqiu+Steven+Xia%2C+Yuxiang+Wei%2C+and+Lingming+Zhang%2E+2022%2E+Practical+Program+Repair+in+the+Era+of+Large+Pre%2Dtrained+Language+Models%2E+CoRR+abs%2F2210%2E14179+%282022%29%2E+%E2%80%94+%5BarXiv%3A2210%2E14179%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
60. Chunqiu Steven Xia and Lingming Zhang. 2023. Conversational Automated Program Repair. CoRR abs/2301.13246 (2023). — [arXiv:2301.13246](https://arxiv.org/abs/2301.13246) [find](https://scholar.google.com/scholar?q=Chunqiu+Steven+Xia+and+Lingming+Zhang%2E+2023%2E+Conversational+Automated+Program+Repair%2E+CoRR+abs%2F2301%2E13246+%282023%29%2E+%E2%80%94+%5BarXiv%3A2301%2E13246%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2301%2E13246%29)
61. Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models. In ICSE. ACM, 37:1–37:12. [find](https://scholar.google.com/scholar?q=Hao+Yu%2C+Bo+Shen%2C+Dezhi+Ran%2C+Jiaxin+Zhang%2C+Qi+Zhang%2C+Yuchi+Ma%2C+Guangtai+Liang%2C+Ying+Li%2C+Qianxiang+Wang%2C+and+Tao+Xie%2E+2024%2E+CoderEval%3A+A+Benchmark+of+Pragmatic+Code+Generation+with+Generative+Pre%2Dtraine)
62. Eric Zelikman, Qian Huang, Gabriel Poesia, Noah D. Goodman, and Nick Haber. 2022. [Parsel: A Unified Natural Language Framework for Algorithmic Reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2023-parsel-algorithmic-reasoning-language-models/). CoRR abs/2212.10561 (2022).
63. Tianyi Zhang, Tao Yu, Tatsunori B. Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried, and Sida I. Wang. 2022. Coder Reviewer Reranking for Code Generation. CoRR abs/2211.16490 (2022). [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Tao+Yu%2C+Tatsunori+B%2E+Hashimoto%2C+Mike+Lewis%2C+Wen%2Dtau+Yih%2C+Daniel+Fried%2C+and+Sida+I%2E+Wang%2E+2022%2E+Coder+Reviewer+Reranking+for+Code+Generation%2E+CoRR+abs%2F2211%2E16490+%282022%29%2E)
64. Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zi-Yuan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. 2023. [CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X](https://md.research.elamur.ai/free/en/papers/zheng-2023-codegeex-pre-trained-model-code/). CoRR abs/2303.17568.
65. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. [Large Language Models Are Human-Level Prompt Engineers](https://md.research.elamur.ai/free/en/papers/zhou-2022-large-language-models-human-level/). CoRR abs/2211.01910 (2022).

*Updated 2 November 2026*