---
title: "MapCoder — paper by Islam et al. (2024) — summary and ideas"
description: "LLMs handle many natural-language tasks well, yet they still often fail at competition-level programming. 44 references, cited by 62 papers here."
authors: ["Md. Ashraful Islam", "Mohammed Eunus Ali", "Md Rizwan Parvez"]
year: 2024
venue: "arXiv:2405.11403"
source: "https://arxiv.org/abs/2405.11403"
doi: "10.48550/arxiv.2405.11403"
arxiv: "2405.11403"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/islam-2024-mapcoder-multi-agent-code/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/islam-2024-mapcoder-multi-agent-code/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / MapCoder: Multi-Agent Code Generation for Competitive Problem Solving

# MapCoder: Multi-Agent Code Generation for Competitive Problem Solving

**Authors:** [Md. Ashraful Islam](https://md.research.elamur.ai/free/en/authors/md-ashraful-islam/), [Mohammed Eunus Ali](https://md.research.elamur.ai/free/en/authors/mohammed-eunus-ali/) and [Md Rizwan Parvez](https://md.research.elamur.ai/free/en/authors/md-rizwan-parvez/) · **Year:** 2024 · **Venue:** [arXiv:2405.11403](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2405.11403) · [PDF](https://arxiv.org/pdf/2405.11403) · [DOI 10.48550/arxiv.2405.11403](https://doi.org/10.48550/arxiv.2405.11403) · [arXiv 2405.11403](https://arxiv.org/abs/2405.11403) · [OpenAlex](https://openalex.org/W4402671883)

**Citations:** 56 citations (OpenAlex, 2026-10-14) · [62](https://md.research.elamur.ai/free/en/papers/islam-2024-mapcoder-multi-agent-code/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLMs handle many natural-language tasks well, yet they still often fail at competition-level programming. These problems demand careful reading of the statement, multi-step algorithmic reasoning, and code that passes hidden tests. MapCoder addresses this with a prompting framework that mimics how human programmers work. Four LLM agents run in sequence. The first recalls similar solved problems by generating them itself, with no external retriever. The next drafts plans, then a third writes code, and the last debugs it. Each plan receives a confidence score, and the code is checked only against the sample I/O included in the problem statement. A failing solution is repaired for a limited number of rounds. If it still fails, the system moves on to the next-best plan. Unlike Reflexion or AlphaCodium, MapCoder never generates extra test cases. Experiments with ChatGPT, GPT-4 and Gemini Pro covered HumanEval, MBPP, APPS, CodeContests, xCodeEval and related sets. The method reached new state-of-the-art pass@1 scores and stayed strong across programming languages and difficulty levels. This suggests that structured agent pipelines without fine-tuning can substantially improve LLM code generation.

## Key points
- Reported pass@1 with GPT-4: HumanEval 93.9%, MBPP 83.1%, APPS 22.0%, CodeContests 28.5%, xCodeEval 45.3%.
- The Retrieval Agent extends analogical prompting: it self-produces k related problems, plans, code and algorithm tutorials as in-context material. k and the number of debugging rounds t trade accuracy for time.
- In the ablation (ChatGPT, HumanEval), the full pipeline scored 80.0%. Dropping retrieval gave 76.0%. Keeping only the Planning Agent gave 52.0%, which shows that debugging guided by plans matters most.
- On average, a problem costs about 16.7 API calls and roughly 21k tokens. Accuracy improves by about 62% relative to direct prompting.
- Weak spots include dynamic programming, combinatorics, number theory and constructive tasks. The code is open-sourced at github.com/Md-Ashraful-Pramanik/MapCoder.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [🖥️ Computer Science Applications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-science-applications/) · 131 works

🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works · [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 168 works

## Cite

Cite as: Md. Ashraful Islam, Mohammed Eunus Ali and Md Rizwan Parvez. 2024. MapCoder: Multi-Agent Code Generation for Competitive Problem Solving. arXiv:2405.11403.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{islam2024mapcoder,
  title = {{MapCoder: Multi-Agent Code Generation for Competitive Problem Solving}},
  author = {Md. Ashraful Islam, Mohammed Eunus Ali and Md Rizwan Parvez},
  year = {2024},
  doi = {10.48550/arxiv.2405.11403},
  eprint = {2405.11403},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.11403}
}
```

- [← previous Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks](https://md.research.elamur.ai/free/en/papers/andriushchenko-2024-jailbreaking-leading-safety-aligned-llms/)
- [next → MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/juraska-2024-metricx-24-google-submission-wmt/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

44 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. [Unified pre-training for program understanding and generation](https://md.research.elamur.ai/free/en/papers/ahmad-2021-unified-pre-training-program-understanding/). arXiv preprint arXiv:2103.06333.
2. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128. [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Maxwell+Lin%2C+Nathanael+Sch%C3%A4rli%2C+and+Denny+Zhou%2E+2023%2E+Teaching+large+language+models+to+self%2Ddebug%2E+arXiv+preprint+arXiv%3A2304%2E05128%2E)
3. Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. 2023. [Santacoder: don’t reach for the stars!](https://md.research.elamur.ai/free/en/papers/allal-2023-santacoder-don-t-reach-stars/) arXiv preprint arXiv:2301.03988.
4. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311.
5. Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Chris… — [Task-Oriented Dialogue as Dataflow Synthesis](https://md.research.elamur.ai/free/en/papers/andreas-2020-task-oriented-dialogue-as-dataflow/)
6. Yihong Dong, Jiazheng Ding, Xue Jiang, Zhuo Li, Ge Li, and Zhi Jin. 2023a. Codescore: Evaluating code generation by learning code execution. arXiv preprint arXiv:2301.09043. — [arXiv:2301.09043](https://arxiv.org/abs/2301.09043) [find](https://scholar.google.com/scholar?q=Yihong+Dong%2C+Jiazheng+Ding%2C+Xue+Jiang%2C+Zhuo+Li%2C+Ge+Li%2C+and+Zhi+Jin%2E+2023a%2E+Codescore%3A+Evaluating+code+generation+by+learning+code+execution%2E+arXiv+preprint+arXiv%3A2301%2E09043%2E+%E2%80%94+%5BarXiv%3A2301%2E09043%5D%28https)
7. Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2023b. [Self-collaboration code generation via chatgpt](https://md.research.elamur.ai/free/en/papers/dong-2023-self-collaboration-code-generation/).
8. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. [Codebert: A pre-trained model for programming and natural languages](https://md.research.elamur.ai/free/en/papers/feng-2020-codebert-pre-trained-model-programming/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536–1547.
9. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999. [find](https://scholar.google.com/scholar?q=Daniel+Fried%2C+Armen+Aghajanyan%2C+Jessy+Lin%2C+Sida+Wang%2C+Eric+Wallace%2C+Freda+Shi%2C+Ruiqi+Zhong%2C+Wen%2Dtau+Yih%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+2022%2E+Incoder%3A+A+generative+model+for+code+infilling+and+synt)
10. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732.
11. Sumit Gulwani. 2011. [Automating string processing in spreadsheets using input-output examples](https://md.research.elamur.ai/free/en/papers/gulwani-2011-automating-string-processing-spreadsheets-input/). ACM Sigplan Notices, 46(1):317–330.
12. Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. Codet: Code generation with generated tests. arXiv preprint arXiv:2207.10397. [find](https://scholar.google.com/scholar?q=Bei+Chen%2C+Fengji+Zhang%2C+Anh+Nguyen%2C+Daoguang+Zan%2C+Zeqi+Lin%2C+Jian%2DGuang+Lou%2C+and+Weizhu+Chen%2E+2022%2E+Codet%3A+Code+generation+with+generated+tests%2E+arXiv+preprint+arXiv%3A2207%2E10397%2E)
13. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196. [find](https://scholar.google.com/scholar?q=Daya+Guo%2C+Qihao+Zhu%2C+Dejian+Yang%2C+Zhenda+Xie%2C+Kai+Dong%2C+Wentao+Zhang%2C+Guanting+Chen%2C+Xiao+Bi%2C+Y+Wu%2C+YK+Li%2C+et+al%2E+2024%2E+Deepseek%2Dcoder%3A+When+the+large+language+model+meets+programming%E2%80%93the+rise+of+code)
14. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
15. Vincent J. Hellendoorn and Premkumar Devanbu. 2017. Are deep neural networks the best choice for modeling source code? In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, pages 763–773, New York, NY, USA. ACM. [find](https://scholar.google.com/scholar?q=Vincent+J%2E+Hellendoorn+and+Premkumar+Devanbu%2E+2017%2E+Are+deep+neural+networks+the+best+choice+for+modeling+source+code%3F+In+Proceedings+of+the+2017+11th+Joint+Meeting+on+Foundations+of+Software+Engineer)
16. Abram Hindle, Earl T. Barr, Mark Gabel, Zhendong Su, and Premkumar Devanbu. 2016. On the naturalness of software. Commun. ACM, 59(5):122–131. [find](https://scholar.google.com/scholar?q=Abram+Hindle%2C+Earl+T%2E+Barr%2C+Mark+Gabel%2C+Zhendong+Su%2C+and+Premkumar+Devanbu%2E+2016%2E+On+the+naturalness+of+software%2E+Commun%2E+ACM%2C+59%285%29%3A122%E2%80%93131%2E)
17. Dong Huang, Qingwen Bu, Jie M Zhang, Michael Luck, and Heming Cui. 2023. [Agentcoder: Multi-agent-based code generation with iterative testing and optimisation](https://md.research.elamur.ai/free/en/papers/huang-2023-agentcoder-multi-agent-based/). arXiv preprint arXiv:2312.13010.
18. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
19. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021b. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
20. Xue Jiang, Yihong Dong, Lecheng Wang, Qiwei Shang, and Ge Li. 2023b. [Self-planning code generation with large language model](https://md.research.elamur.ai/free/en/papers/jiang-2023-self-planning-code-generation-large/). arXiv preprint arXiv:2303.06689.
21. Emilio Parisotto and Ruslan Salakhutdinov. 2017. Neural map: Structured memory for deep reinforcement learning. arXiv preprint arXiv:1702.08360. [find](https://scholar.google.com/scholar?q=Emilio+Parisotto+and+Ruslan+Salakhutdinov%2E+2017%2E+Neural+map%3A+Structured+memory+for+deep+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1702%2E08360%2E)
22. Md Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. [Retrieval augmented code generation and summarization](https://md.research.elamur.ai/free/en/papers/parvez-2021-retrieval-augmented-code-generation-summarization/). arXiv preprint arXiv:2108.11601.
23. Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. 2023. [xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval](https://md.research.elamur.ai/free/en/papers/khan-2023-xcodeeval-large-scale-multilingual-multitask/). arXiv preprint arXiv:2303.03004.
24. Md Rizwan Parvez, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2018. Building language models for text with named entities. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2373–2383, Melbourne, Australia. Association… [find](https://scholar.google.com/scholar?q=Md+Rizwan+Parvez%2C+Saikat+Chakraborty%2C+Baishakhi+Ray%2C+and+Kai%2DWei+Chang%2E+2018%2E+Building+language+models+for+text+with+named+entities%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Co)
25. Donald E Knuth. 1992. Literate programming. CSLI Lecture Notes, Stanford, CA: Center for the Study of Language and Information (CSLI), 1992. [find](https://scholar.google.com/scholar?q=Donald+E+Knuth%2E+1992%2E+Literate+programming%2E+CSLI+Lecture+Notes%2C+Stanford%2C+CA%3A+Center+for+the+Study+of+Language+and+Information+%28CSLI%29%2C+1992%2E)
26. Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. 2022. [Coderl: Mastering code generation through pretrained models and deep reinforcement learning](https://md.research.elamur.ai/free/en/papers/le-2022-coderl-mastering-code-generation-through/). Advances in Neural Information Processing Systems, 35:21314–21328.
27. Md Rizwan Parvez, Jianfeng Chi, Wasi Uddin Ahmad, Yuan Tian, and Kai-Wei Chang. 2023. Retrieval enhanced data augmentation for question answering on privacy policies. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 201–210, Dubrov… [find](https://scholar.google.com/scholar?q=Md+Rizwan+Parvez%2C+Jianfeng+Chi%2C+Wasi+Uddin+Ahmad%2C+Yuan+Tian%2C+and+Kai%2DWei+Chang%2E+2023%2E+Retrieval+enhanced+data+augmentation+for+question+answering+on+privacy+policies%2E+In+Proceedings+of+the+17th+Confer)
28. Jingyao Li, Pengguang Chen, and Jiaya Jia. 2023. Motcoder: Elevating large language models with modular of thought for challenging programming tasks. arXiv preprint arXiv:2312.15960. — [arXiv:2312.15960](https://arxiv.org/abs/2312.15960) [find](https://scholar.google.com/scholar?q=Jingyao+Li%2C+Pengguang+Chen%2C+and+Jiaya+Jia%2E+2023%2E+Motcoder%3A+Elevating+large+language+models+with+modular+of+thought+for+challenging+programming+tasks%2E+arXiv+preprint+arXiv%3A2312%2E15960%2E+%E2%80%94+%5BarXiv%3A2312%2E159)
29. Oleksandr Polozov and Sumit Gulwani. 2015. Flashmeta: A framework for inductive program synthesis. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, pages 107– 126. [find](https://scholar.google.com/scholar?q=Oleksandr+Polozov+and+Sumit+Gulwani%2E+2015%2E+Flashmeta%3A+A+framework+for+inductive+program+synthesis%2E+In+Proceedings+of+the+2015+ACM+SIGPLAN+International+Conference+on+Object%2DOriented+Programming%2C+Syste)
30. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022a. [Competition-level code generation with alphacode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/). Science, 378(6624):1092–1097.
31. Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017. Abstract syntax networks for code generation and semantic parsing. CoRR, abs/1704.07535. [find](https://scholar.google.com/scholar?q=Maxim+Rabinovich%2C+Mitchell+Stern%2C+and+Dan+Klein%2E+2017%2E+Abstract+syntax+networks+for+code+generation+and+semantic+parsing%2E+CoRR%2C+abs%2F1704%2E07535%2E)
32. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James M… — [Competition-Level Code Generation with AlphaCode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/)
33. Tal Ridnik, Dedy Kredo, and Itamar Friedman. 2024. Code generation with alphacodium: From prompt engineering to flow engineering. arXiv preprint arXiv:2401.08500. [find](https://scholar.google.com/scholar?q=Tal+Ridnik%2C+Dedy+Kredo%2C+and+Itamar+Friedman%2E+2024%2E+Code+generation+with+alphacodium%3A+From+prompt+engineering+to+flow+engineering%2E+arXiv+preprint+arXiv%3A2401%2E08500%2E)
34. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950.
35. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chat-GPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Jiawei+Liu%2C+Chunqiu+Steven+Xia%2C+Yuyao+Wang%2C+and+Lingming+Zhang%2E+2023%2E+Is+your+code+generated+by+chat%2DGPT+really+correct%3F+rigorous+evaluation+of+large+language+models+for+code+generation%2E+In+Thirty%2Dsev)
36. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. 2023. [Reflexion: Language agents with verbal reinforcement learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/). In Thirty-seventh Conference on Neural Information Processing Systems.
37. Zohar Manna and Richard J. Waldinger. 1971. Toward automatic program synthesis. Commun. ACM, 14(3):151–165. [find](https://scholar.google.com/scholar?q=Zohar+Manna+and+Richard+J%2E+Waldinger%2E+1971%2E+Toward+automatic+program+synthesis%2E+Commun%2E+ACM%2C+14%283%29%3A151%E2%80%93165%2E)
38. Kashun Shum, Shizhe Diao, and Tong Zhang. 2023. [Automatic prompt augmentation and selection with chain-of-thought from labeled data](https://md.research.elamur.ai/free/en/papers/shum-2023-automatic-prompt-augmentation-selection-chain/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12113–12139, Singapore. Association for Computational Linguistics.
39. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474. [find](https://scholar.google.com/scholar?q=Erik+Nijkamp%2C+Bo+Pang%2C+Hiroaki+Hayashi%2C+Lifu+Tu%2C+Huan+Wang%2C+Yingbo+Zhou%2C+Silvio+Savarese%2C+and+Caiming+Xiong%2E+2022%2E+Codegen%3A+An+open+large+language+model+for+code+with+multi%2Dturn+program+synthesis%2E+arX)
40. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
41. Carlos Pacheco, Shuvendu K Lahiri, Michael D Ernst, and Thomas Ball. 2007. Feedback-directed random test generation. In 29th International Conference on Software Engineering (ICSE’07), pages 75–84. IEEE. [find](https://scholar.google.com/scholar?q=Carlos+Pacheco%2C+Shuvendu+K+Lahiri%2C+Michael+D+Ernst%2C+and+Thomas+Ball%2E+2007%2E+Feedback%2Ddirected+random+test+generation%2E+In+29th+International+Conference+on+Software+Engineering+%28ICSE%E2%80%9907%29%2C+pages+75%E2%80%9384%2E+IE)
42. Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In EMNLP, pages 8696–8708. [find](https://scholar.google.com/scholar?q=Yue+Wang%2C+Weishi+Wang%2C+Shafiq+Joty%2C+and+Steven+CH+Hoi%2E+2021%2E+Codet5%3A+Identifier%2Daware+unified+pre%2Dtrained+encoder%2Ddecoder+models+for+code+understanding+and+generation%2E+In+EMNLP%2C+pages+8696%E2%80%938708%2E)
43. Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. 2023. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406. [find](https://scholar.google.com/scholar?q=Andy+Zhou%2C+Kai+Yan%2C+Michal+Shlapentokh%2DRothman%2C+Haohan+Wang%2C+and+Yu%2DXiong+Wang%2E+2023%2E+Language+agent+tree+search+unifies+reasoning+acting+and+planning+in+language+models%2E+arXiv+preprint+arXiv%3A2310%2E044)
44. Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, and Graham Neubig. 2023. [Learning to filter context for retrieval-augmented generation](https://md.research.elamur.ai/free/en/papers/wang-2023-learning-filter-context-retrieval-augmented/). arXiv preprint arXiv:2311.08377.

*Updated 2 November 2026*