---
title: "MetaGPT: Meta Programming for A Multi-Agent — Hong 2023"
description: "Multi-agent systems built on large language models (LLMs) handle simple dialogue well but struggle with complex tasks."
authors: ["Sirui Hong", "Mingchen Zhuge", "Jiaqi Chen", "Xiawu Zheng", "Yuheng Cheng", "Ceyao Zhang", "Jinlin Wang", "Zili Wang", "Steven Ka Shing Yau", "Zijuan Lin", "Liyang Zhou", "Chenyu Ran", "Lingfeng Xiao", "Chenglin Wu", "Jürgen Schmidhuber"]
year: 2023
venue: "arXiv:2308.00352"
source: "https://arxiv.org/abs/2308.00352"
doi: "10.48550/arxiv.2308.00352"
arxiv: "2308.00352"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hong-2023-metagpt-meta-programming-multi/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hong-2023-metagpt-meta-programming-multi/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework

# MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework

**Authors:** Sirui Hong, [Mingchen Zhuge](https://md.research.elamur.ai/free/en/authors/mingchen-zhuge/), Jiaqi Chen, [Xiawu Zheng](https://md.research.elamur.ai/free/en/authors/xiawu-zheng/), Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran et al. (15 authors) · **Year:** 2023 · **Venue:** arXiv:2308.00352 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2308.00352) · [PDF](https://arxiv.org/pdf/2308.00352) · [DOI 10.48550/arxiv.2308.00352](https://doi.org/10.48550/arxiv.2308.00352) · [arXiv 2308.00352](https://arxiv.org/abs/2308.00352) · [OpenAlex](https://openalex.org/W4385963839)

**Citations:** 160 citations (openalex-snapshot · 2026-10-14) · [455](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Multi-agent systems built on large language models (LLMs) handle simple dialogue well but struggle with complex tasks. When models are chained naively, one hallucination leads to more, and the final output ends up logically inconsistent. MetaGPT tackles this by taking Standardized Operating Procedures (SOPs) from human teams and encoding them as prompt sequences. The framework simulates a software company. Specialized agents (Product Manager, Architect, Project Manager, Engineer, QA Engineer) work like an assembly line, and each one produces structured artifacts such as PRDs, system designs and interface specifications instead of free-form chat. Agents share these documents through a common message pool, and a publish–subscribe mechanism filters out messages a role doesn't need. The Engineer also uses executable feedback: it runs and debugs its code while generating it. MetaGPT reaches state-of-the-art Pass@1 on HumanEval and MBPP, and it produces more coherent software than chat-based systems such as ChatDev. The work matters because it shows that giving agent teams a human-style workflow structure makes them more reliable on multi-step engineering problems.

## Key points
- Pass@1 scores are 85.9% on HumanEval and 87.7% on MBPP, which the authors report as new state of the art.
- The executable feedback loop alone adds 5.4 percentage points (absolute) on MBPP.
- MetaGPT is compared with AutoGPT, LangChain, AgentVerse and ChatDev on building whole software projects, and the authors report a 100% task completion rate.
- The authors evaluate on their own SoftwareDev set of 70 tasks, which ranges from games (Snake, 2048, Gomoku) to Streamlit/pandas data tools and CRUD apps. Without feedback, an average project has about 4.7 code files and about 190 lines of code, and costs about $1.12 and roughly 500 seconds to generate.
- The code is open source: https://github.com/geekan/MetaGPT.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cited by

Cited by

455 papers of our library cite it (of the 140673 whose bibliographies we read); 19 of them have a page here, 436 not yet.

- [Qwen Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
- [From Persona to Personalization: A Survey on Role-Playing Language Agents](https://md.research.elamur.ai/free/en/papers/chen-2024-persona-personalization-survey-role-playing/)
- [Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecture](https://md.research.elamur.ai/free/en/papers/cui-2023-chatlaw-multi-agent-legal-assistant/)
- [基于大语言模型智能体的代码生成综述](https://md.research.elamur.ai/free/en/papers/dong-2025-code-generation-llm/)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://md.research.elamur.ai/free/en/papers/guo-2024-large-language-model-based/)
- [A Survey on Large Language Models for Code Generation](https://md.research.elamur.ai/free/en/papers/jiang-2024-survey-large-language-models-code/)
- [AgentBench: Evaluating LLMs as Agents](https://md.research.elamur.ai/free/en/papers/liu-2023-agentbench-evaluating-llms-agents/)
- [On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey](https://md.research.elamur.ai/free/en/papers/long-2024-llms-driven-synthetic-data-generation/)
- [A Comprehensive Overview of Large Language Models](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/)
- [ChatDev: Communicative Agents for Software Development](https://md.research.elamur.ai/free/en/papers/qian-2023-chatdev-communicative-agents-software/)
- [Agent Laboratory: Using LLM Agents as Research Assistants](https://md.research.elamur.ai/free/en/papers/schmidgall-2025-agent-laboratory-using-llm/)
- [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://md.research.elamur.ai/free/en/papers/tseng-2024-two-tales-persona-llms-survey/)
- [Executable Code Actions Elicit Better LLM Agents](https://md.research.elamur.ai/free/en/papers/wang-2024-executable-code-actions-elicit-better/)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://md.research.elamur.ai/free/en/papers/wang-2024-openhands-open-platform-ai/)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://md.research.elamur.ai/free/en/papers/wu-2023-autogen-enabling-next-gen-llm/)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://md.research.elamur.ai/free/en/papers/yang-2024-swe-agent-agent-computer-interfaces/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)
- [DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments](https://md.research.elamur.ai/free/en/papers/zheng-2025-deepresearcher-scaling-deep-research-reinforcement/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hong2023metagpt,
  title = {{MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework}},
  author = {Hong, Sirui and Zhuge, Mingchen and Chen, Jiaqi and Zheng, Xiawu and Cheng, Yuheng and Zhang, Ceyao and Wang, Jinlin and Wang, Zili and Yau, Steven Ka Shing and Lin, Zijuan and Zhou, Liyang and Ran, Chenyu and Xiao, Lingfeng and Wu, Chenglin and Schmidhuber, Jürgen},
  year = {2023},
  doi = {10.48550/arxiv.2308.00352},
  eprint = {2308.00352},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.00352}
}
```

- [← previous MEGA: Multilingual Evaluation of Generative AI](https://md.research.elamur.ai/free/en/papers/ahuja-2023-mega-multilingual-evaluation-generative-ai/)
- [next → MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

78 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models. arXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=Elif+Akata%2C+Lion+Schulz%2C+Julian+Coda%2DForno%2C+Seong+Joon+Oh%2C+Matthias+Bethge%2C+and+Eric+Schulz%2E+Playing+repeated+games+with+large+language+models%2E+arXiv+preprint%2C+2023%2E)
2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. [find](https://scholar.google.com/scholar?q=Jacob+Austin%2C+Augustus+Odena%2C+Maxwell+Nye%2C+Maarten+Bosma%2C+Henryk+Michalewski%2C+David+Dohan%2C+Ellen+Jiang%2C+Carrie+Cai%2C+Michael+Terry%2C+Quoc+Le%2C+and+Charles+Sutton%2E+Program+synthesis+with+large+language+mo) [our card](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/)
3. Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 2022. [find](https://scholar.google.com/scholar?q=Anton+Bakhtin%2C+Noam+Brown%2C+Emily+Dinan%2C+Gabriele+Farina%2C+Colin+Flaherty%2C+Daniel+Fried%2C+Andrew+Goff%2C+Jonathan+Gray%2C+Hengyuan+Hu%2C+et+al%2E+Human%2Dlevel+play+in+the+game+of+diplomacy+by+combining+language+m)
4. Robert Balzer. A 15 year perspective on automatic programming. TSE, 1985. [find](https://scholar.google.com/scholar?q=Robert+Balzer%2E+A+15+year+perspective+on+automatic+programming%2E+TSE%2C+1985%2E)
5. R.M. Belbin. Team Roles at Work. Routledge, 2012. URL [https://books.google.co.uk/](https://books.google.co.uk/) books?id=MHIQBAAAQBAJ. [↗](https://books.google.co.uk/)
6. Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. arXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=Tianle+Cai%2C+Xuezhi+Wang%2C+Tengyu+Ma%2C+Xinyun+Chen%2C+and+Denny+Zhou%2E+Large+language+models+as+tool+makers%2E+arXiv+preprint%2C+2023%2E)
7. Harrison Chase. LangChain. [https://github.com/hwchase17/langchain](https://github.com/hwchase17/langchain), 2022. [↗](https://github.com/hwchase17/langchain)
8. Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests, 2022. [find](https://scholar.google.com/scholar?q=Bei+Chen%2C+Fengji+Zhang%2C+Anh+Nguyen%2C+Daoguang+Zan%2C+Zeqi+Lin%2C+Jian%2DGuang+Lou%2C+and+Weizhu+Chen%2E+Codet%3A+Code+generation+with+generated+tests%2C+2022%2E)
9. Jiaqi Chen, Yuxian Jiang, Jiachen Lu, and Li Zhang. S-agents: self-organizing agents in open-ended environment. arXiv preprint, 2024. [find](https://scholar.google.com/scholar?q=Jiaqi+Chen%2C+Yuxian+Jiang%2C+Jiachen+Lu%2C+and+Li+Zhang%2E+S%2Dagents%3A+self%2Dorganizing+agents+in+open%2Dended+environment%2E+arXiv+preprint%2C+2024%2E)
10. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… [find](https://scholar.google.com/scholar?q=Mark+Chen%2C+Jerry+Tworek%2C+Heewoo+Jun%2C+Qiming+Yuan%2C+Henrique+Ponde+de+Oliveira+Pinto%2C+Jared+Kaplan%2C+Harri+Edwards%2C+Yuri+Burda%2C+Nicholas+Joseph%2C+Greg+Brockman%2C+Alex+Ray%2C+Raul+Puri%2C+Gretchen+Krueger%2C+Mich) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
11. Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents, 2023. [find](https://scholar.google.com/scholar?q=Weize+Chen%2C+Yusheng+Su%2C+Jingwei+Zuo%2C+Cheng+Yang%2C+Chenfei+Yuan%2C+Chen+Qian%2C+Chi%2DMin+Chan%2C+Yujia+Qin%2C+Yaxi+Lu%2C+Ruobing+Xie%2C+Zhiyuan+Liu%2C+Maosong+Sun%2C+and+Jie+Zhou%2E+Agentverse%3A+Facilitating+multi%2Dagent+co) [our card](https://md.research.elamur.ai/free/en/papers/chen-2023-agentverse-facilitating-multi-agent/)
12. Xinyun Chen, Chang Liu, and Dawn Song. Execution-guided neural program synthesis. In ICLR, 2018. [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Chang+Liu%2C+and+Dawn+Song%2E+Execution%2Dguided+neural+program+synthesis%2E+In+ICLR%2C+2018%2E)
13. Xinyun Chen, Dawn Song, and Yuandong Tian. Latent execution for neural program synthesis beyond domain-specific languages. NeurIPS, 2021b. [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Dawn+Song%2C+and+Yuandong+Tian%2E+Latent+execution+for+neural+program+synthesis+beyond+domain%2Dspecific+languages%2E+NeurIPS%2C+2021b%2E)
14. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… [find](https://scholar.google.com/scholar?q=Aakanksha+Chowdhery%2C+Sharan+Narang%2C+Jacob+Devlin%2C+Maarten+Bosma%2C+Gaurav+Mishra%2C+Adam+Roberts%2C+Paul+Barham%2C+Hyung+Won+Chung%2C+Charles+Sutton%2C+Sebastian+Gehrmann%2C+Parker+Schuh%2C+Kensen+Shi%2C+Sasha+Tsvyashc)
15. T. DeMarco and T.R. Lister. Peopleware: Productive Projects and Teams. Addison-Wesley, 2013. URL [https://books.google.co.uk/books?id=DVlsAQAAQBAJ](https://books.google.co.uk/books?id=DVlsAQAAQBAJ). [↗](https://books.google.co.uk/books?id=DVlsAQAAQBAJ)
16. Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. Self-collaboration code generation via chatgpt. arXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=Yihong+Dong%2C+Xue+Jiang%2C+Zhi+Jin%2C+and+Ge+Li%2E+Self%2Dcollaboration+code+generation+via+chatgpt%2E+arXiv+preprint%2C+2023%2E)
17. Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate, 2023. [find](https://scholar.google.com/scholar?q=Yilun+Du%2C+Shuang+Li%2C+Antonio+Torralba%2C+Joshua+B%2E+Tenenbaum%2C+and+Igor+Mordatch%2E+Improving+factuality+and+reasoning+in+language+models+through+multiagent+debate%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/du-2023-improving-factuality-reasoning-language-models/)
18. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language models. TACL, 2021. [find](https://scholar.google.com/scholar?q=Yanai+Elazar%2C+Nora+Kassner%2C+Shauli+Ravfogel%2C+Abhilasha+Ravichander%2C+Eduard+Hovy%2C+Hinrich+Schu%CC%88tze%2C+and+Yoav+Goldberg%2E+Measuring+and+improving+consistency+in+pretrained+language+models%2E+TACL%2C+2021%2E)
19. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint, 2020. [find](https://scholar.google.com/scholar?q=Zhangyin+Feng%2C+Daya+Guo%2C+Duyu+Tang%2C+Nan+Duan%2C+Xiaocheng+Feng%2C+Ming+Gong%2C+Linjun+Shou%2C+Bing+Qin%2C+Ting+Liu%2C+Daxin+Jiang%2C+et+al%2E+Codebert%3A+A+pre%2Dtrained+model+for+programming+and+natural+languages%2E+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/feng-2020-codebert-pre-trained-model-programming/)
20. Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=Chrisantha+Fernando%2C+Dylan+Banarse%2C+Henryk+Michalewski%2C+Simon+Osindero%2C+and+Tim+Rockta%CC%88schel%2E+Promptbreeder%3A+Self%2Dreferential+self%2Dimprovement+via+prompt+evolution%2E+arXiv+preprint%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/fernando-2023-promptbreeder-self-referential-self-improvement/)
21. Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017. [find](https://scholar.google.com/scholar?q=Chelsea+Finn%2C+Pieter+Abbeel%2C+and+Sergey+Levine%2E+Model%2Dagnostic+meta%2Dlearning+for+fast+adaptation+of+deep+networks%2E+In+ICML%2C+2017%2E)
22. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint, 2022. [find](https://scholar.google.com/scholar?q=Daniel+Fried%2C+Armen+Aghajanyan%2C+Jessy+Lin%2C+Sida+Wang%2C+Eric+Wallace%2C+Freda+Shi%2C+Ruiqi+Zhong%2C+Wen%2Dtau+Yih%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+Incoder%3A+A+generative+model+for+code+infilling+and+synthesis%2E)
23. Irving John Good. Speculations concerning the first ultraintelligent machine. Adv. Comput., 1965. [find](https://scholar.google.com/scholar?q=Irving+John+Good%2E+Speculations+concerning+the+first+ultraintelligent+machine%2E+Adv%2E+Comput%2E%2C+1965%2E)
24. Rui Hao, Linmei Hu, Weijian Qi, Qingliu Wu, Yirui Zhang, and Liqiang Nie. Chatllm network: More brains, more intelligence. arXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=Rui+Hao%2C+Linmei+Hu%2C+Weijian+Qi%2C+Qingliu+Wu%2C+Yirui+Zhang%2C+and+Liqiang+Nie%2E+Chatllm+network%3A+More+brains%2C+more+intelligence%2E+arXiv+preprint%2C+2023%2E)
25. S. Hochreiter, A. S. Younger, and P. R. Conwell. Learning to learn using gradient descent. In Lecture Notes on Comp. Sci. 2130, Proc. Intl. Conf. on Artificial Neural Networks (ICANN-2001), pp. 87– 94. [find](https://scholar.google.com/scholar?q=S%2E+Hochreiter%2C+A%2E+S%2E+Younger%2C+and+P%2E+R%2E+Conwell%2E+Learning+to+learn+using+gradient+descent%2E+In+Lecture+Notes+on+Comp%2E+Sci%2E+2130%2C+Proc%2E+Intl%2E+Conf%2E+on+Artificial+Neural+Networks+%28ICANN%2D2001%29%2C+pp%2E+87%E2%80%93+94)
26. Springer: Berlin, Heidelberg, 2001. [find](https://scholar.google.com/scholar?q=Springer%3A+Berlin%2C+Heidelberg%2C+2001%2E)
27. Sirui Hong, Yizhang Lin, Bangbang Liu, Binhao Wu, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, Lingyao Zhang, Mingchen Zhuge, et al. Data interpreter: An llm agent for data science. arXiv preprint arXiv:2402.18679, 2024. [find](https://scholar.google.com/scholar?q=Sirui+Hong%2C+Yizhang+Lin%2C+Bangbang+Liu%2C+Binhao+Wu%2C+Danyang+Li%2C+Jiaqi+Chen%2C+Jiayi+Zhang%2C+Jinlin+Wang%2C+Lingyao+Zhang%2C+Mingchen+Zhuge%2C+et+al%2E+Data+interpreter%3A+An+llm+agent+for+data+science%2E+arXiv+preprin)
28. Xue Jiang, Yihong Dong, Lecheng Wang, Qiwei Shang, and Ge Li. Self-planning code generation with large language model. arXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=Xue+Jiang%2C+Yihong+Dong%2C+Lecheng+Wang%2C+Qiwei+Shang%2C+and+Ge+Li%2E+Self%2Dplanning+code+generation+with+large+language+model%2E+arXiv+preprint%2C+2023%2E)
29. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for” mind” exploration of large scale language model society. arXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=Guohao+Li%2C+Hasan+Abed+Al+Kader+Hammoud%2C+Hani+Itani%2C+Dmitrii+Khizbullin%2C+and+Bernard+Ghanem%2E+Camel%3A+Communicative+agents+for%E2%80%9D+mind%E2%80%9D+exploration+of+large+scale+language+model+society%2E+arXiv+preprint%2C+20)
30. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 2022. [find](https://scholar.google.com/scholar?q=Yujia+Li%2C+David+Choi%2C+Junyoung+Chung%2C+Nate+Kushman%2C+Julian+Schrittwieser%2C+Re%CC%81mi+Leblond%2C+Tom+Eccles%2C+James+Keeling%2C+Felix+Gimeno%2C+Agustin+Dal+Lago%2C+et+al%2E+Competition%2Dlevel+code+generation+with+alphac) [our card](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/)
31. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=Tian+Liang%2C+Zhiwei+He%2C+Wenxiang+Jiao%2C+Xing+Wang%2C+Yan+Wang%2C+Rui+Wang%2C+Yujiu+Yang%2C+Zhaopeng+Tu%2C+and+Shuming+Shi%2E+Encouraging+divergent+thinking+in+large+language+models+through+multi%2Dagent+debate%2E+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/liang-2023-encouraging-divergent-thinking-large-language/)
32. Bill Yuchen Lin, Yicheng Fu, Karina Yang, Prithviraj Ammanabrolu, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. arXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=Bill+Yuchen+Lin%2C+Yicheng+Fu%2C+Karina+Yang%2C+Prithviraj+Ammanabrolu%2C+Faeze+Brahman%2C+Shiyu+Huang%2C+Chandra+Bhagavatula%2C+Yejin+Choi%2C+and+Xiang+Ren%2E+Swiftsage%3A+A+generative+agent+with+fast+and+slow+thinking+)
33. Ruibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M Dai, Diyi Yang, and Soroush Vosoughi. Training socially aligned language models in simulated human society. arXiv preprint, 2023a. [find](https://scholar.google.com/scholar?q=Ruibo+Liu%2C+Ruixin+Yang%2C+Chenyan+Jia%2C+Ge+Zhang%2C+Denny+Zhou%2C+Andrew+M+Dai%2C+Diyi+Yang%2C+and+Soroush+Vosoughi%2E+Training+socially+aligned+language+models+in+simulated+human+society%2E+arXiv+preprint%2C+2023a%2E)
34. Yuliang Liu, Xiangru Tang, Zefan Cai, Junjie Lu, Yichi Zhang, Yanjun Shao, Zexuan Deng, Helan Hu, Zengxian Yang, Kaikai An, et al. Ml-bench: Large language models leverage open-source libraries for machine learning tasks. arXiv preprint arXiv:2311.09835, 2023b. [find](https://scholar.google.com/scholar?q=Yuliang+Liu%2C+Xiangru+Tang%2C+Zefan+Cai%2C+Junjie+Lu%2C+Yichi+Zhang%2C+Yanjun+Shao%2C+Zexuan+Deng%2C+Helan+Hu%2C+Zengxian+Yang%2C+Kaikai+An%2C+et+al%2E+Ml%2Dbench%3A+Large+language+models+leverage+open%2Dsource+libraries+for+ma)
35. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=Ziyang+Luo%2C+Can+Xu%2C+Pu+Zhao%2C+Qingfeng+Sun%2C+Xiubo+Geng%2C+Wenxiang+Hu%2C+Chongyang+Tao%2C+Jing+Ma%2C+Qingwei+Lin%2C+and+Daxin+Jiang%2E+Wizardcoder%3A+Empowering+code+large+language+models+with+evol%2Dinstruct%2E+arXiv+p)
36. Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=Potsawee+Manakul%2C+Adian+Liusie%2C+and+Mark+JF+Gales%2E+Selfcheckgpt%3A+Zero%2Dresource+black%2Dbox+hallucination+detection+for+generative+large+language+models%2E+arXiv+preprint%2C+2023%2E)
37. Agile Manifesto. Manifesto for agile software development. Snowbird, UT, 2001. [find](https://scholar.google.com/scholar?q=Agile+Manifesto%2E+Manifesto+for+agile+software+development%2E+Snowbird%2C+UT%2C+2001%2E)
38. John McCarthy. History of lisp. In History of programming languages. 1978. [find](https://scholar.google.com/scholar?q=John+McCarthy%2E+History+of+lisp%2E+In+History+of+programming+languages%2E+1978%2E)
39. Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. arXiv preprint arXiv:2308.07124, 2023. [find](https://scholar.google.com/scholar?q=Niklas+Muennighoff%2C+Qian+Liu%2C+Armel+Zebaze%2C+Qinkai+Zheng%2C+Binyuan+Hui%2C+Terry+Yue+Zhuo%2C+Swayam+Singh%2C+Xiangru+Tang%2C+Leandro+Von+Werra%2C+and+Shayne+Longpre%2E+Octopack%3A+Instruction+tuning+code+large+langua)
40. Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. Lever: Learning to verify language-to-code generation with execution. In ICML, 2023. [find](https://scholar.google.com/scholar?q=Ansong+Ni%2C+Srini+Iyer%2C+Dragomir+Radev%2C+Veselin+Stoyanov%2C+Wen%2Dtau+Yih%2C+Sida+Wang%2C+and+Xi+Victoria+Lin%2E+Lever%3A+Learning+to+verify+language%2Dto%2Dcode+generation+with+execution%2E+In+ICML%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/ni-2023-lever-learning-verify-language-code/)
41. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis, 2023. [find](https://scholar.google.com/scholar?q=Erik+Nijkamp%2C+Bo+Pang%2C+Hiroaki+Hayashi%2C+Lifu+Tu%2C+Huan+Wang%2C+Yingbo+Zhou%2C+Silvio+Savarese%2C+and+Caiming+Xiong%2E+Codegen%3A+An+open+large+language+model+for+code+with+multi%2Dturn+program+synthesis%2C+2023%2E)
42. OpenAI. Gpt-4 technical report, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
43. Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=Joon+Sung+Park%2C+Joseph+C+O%E2%80%99Brien%2C+Carrie+J+Cai%2C+Meredith+Ringel+Morris%2C+Percy+Liang%2C+and+Michael+S+Bernstein%2E+Generative+agents%3A+Interactive+simulacra+of+human+behavior%2E+arXiv+preprint%2C+2023%2E)
44. Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development, 2023. [find](https://scholar.google.com/scholar?q=Chen+Qian%2C+Xin+Cong%2C+Cheng+Yang%2C+Weize+Chen%2C+Yusheng+Su%2C+Juyuan+Xu%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+Communicative+agents+for+software+development%2C+2023%2E)
45. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789, 2023. [find](https://scholar.google.com/scholar?q=Yujia+Qin%2C+Shihao+Liang%2C+Yining+Ye%2C+Kunlun+Zhu%2C+Lan+Yan%2C+Yaxi+Lu%2C+Yankai+Lin%2C+Xin+Cong%2C+Xiangru+Tang%2C+Bill+Qian%2C+et+al%2E+Toolllm%3A+Facilitating+large+language+models+to+master+16000%2B+real%2Dworld+apis%2E+ar) [our card](https://md.research.elamur.ai/free/en/papers/qin-2023-toolllm-facilitating-large-language-models/)
46. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=Baptiste+Rozie%CC%80re%2C+Jonas+Gehring%2C+Fabian+Gloeckle%2C+Sten+Sootla%2C+Itai+Gat%2C+Xiaoqing+Ellen+Tan%2C+Yossi+Adi%2C+Jingyu+Liu%2C+Tal+Remez%2C+Je%CC%81re%CC%81my+Rapin%2C+et+al%2E+Code+llama%3A+Open+foundation+models+for+code%2E+arXi) [our card](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
47. Timo Schick, Jane Dwivedi-Yu, Roberto Dessı̀, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=Timo+Schick%2C+Jane+Dwivedi%2DYu%2C+Roberto+Dess%C4%B1%CC%80%2C+Roberta+Raileanu%2C+Maria+Lomeli%2C+Luke+Zettlemoyer%2C+Nicola+Cancedda%2C+and+Thomas+Scialom%2E+Toolformer%3A+Language+models+can+teach+themselves+to+use+tools%2E+arXi) [our card](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/)
48. J. Schmidhuber. A self-referential weight matrix. In Proceedings of the International Conference on Artificial Neural Networks, Amsterdam, pp. 446–451. Springer, 1993a. [find](https://scholar.google.com/scholar?q=J%2E+Schmidhuber%2E+A+self%2Dreferential+weight+matrix%2E+In+Proceedings+of+the+International+Conference+on+Artificial+Neural+Networks%2C+Amsterdam%2C+pp%2E+446%E2%80%93451%2E+Springer%2C+1993a%2E)
49. J. Schmidhuber. Gödel machines: self-referential universal problem solvers making provably optimal self-improvements. Technical Report IDSIA-19-03, arXiv:cs.LO/0309048 v3, IDSIA, Manno-Lugano, Switzerland, December 2003. — [arXiv:cs.lo/0309048](https://arxiv.org/abs/cs.lo/0309048) [find](https://scholar.google.com/scholar?q=J%2E+Schmidhuber%2E+Go%CC%88del+machines%3A+self%2Dreferential+universal+problem+solvers+making+provably+optimal+self%2Dimprovements%2E+Technical+Report+IDSIA%2D19%2D03%2C+arXiv%3Acs%2ELO%2F0309048+v3%2C+IDSIA%2C+Manno%2DLugano%2C+Switze)
50. J. Schmidhuber. Gödel machines: Fully self-referential optimal universal self-improvers. In B. Goertzel and C. Pennachin (eds.), Artificial General Intelligence, pp. 199–226. Springer Verlag, 2006. Variant available as arXiv:cs.LO/0309048. — [arXiv:cs.lo/0309048](https://arxiv.org/abs/cs.lo/0309048) [find](https://scholar.google.com/scholar?q=J%2E+Schmidhuber%2E+Go%CC%88del+machines%3A+Fully+self%2Dreferential+optimal+universal+self%2Dimprovers%2E+In+B%2E+Goertzel+and+C%2E+Pennachin+%28eds%2E%29%2C+Artificial+General+Intelligence%2C+pp%2E+199%E2%80%93226%2E+Springer+Verlag%2C+2006%2E+V)
51. J. Schmidhuber. Ultimate cognition à la Gödel. Cognitive Computation, 1(2):177–193, 2009. [find](https://scholar.google.com/scholar?q=J%2E+Schmidhuber%2E+Ultimate+cognition+a%CC%80+la+Go%CC%88del%2E+Cognitive+Computation%2C+1%282%29%3A177%E2%80%93193%2C+2009%2E)
52. Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, 1987. [find](https://scholar.google.com/scholar?q=Ju%CC%88rgen+Schmidhuber%2E+Evolutionary+principles+in+self%2Dreferential+learning%2C+or+on+learning+how+to+learn%3A+the+meta%2Dmeta%2D%2E%2E%2E+hook%2E+PhD+thesis%2C+1987%2E)
53. Jürgen Schmidhuber. A ‘self-referential’weight matrix. In ICANN’93: Proceedings of the International Conference on Artificial Neural Networks Amsterdam, The Netherlands 13–16 September 1993 3, 1993b. [find](https://scholar.google.com/scholar?q=Ju%CC%88rgen+Schmidhuber%2E+A+%E2%80%98self%2Dreferential%E2%80%99weight+matrix%2E+In+ICANN%E2%80%9993%3A+Proceedings+of+the+International+Conference+on+Artificial+Neural+Networks+Amsterdam%2C+The+Netherlands+13%E2%80%9316+September+1993+3%2C+1993b%2E)
54. Jürgen Schmidhuber. On learning to think: Algorithmic information theory for novel combinations of reinforcement learning controllers and recurrent neural world models. arXiv preprint, 2015. [find](https://scholar.google.com/scholar?q=Ju%CC%88rgen+Schmidhuber%2E+On+learning+to+think%3A+Algorithmic+information+theory+for+novel+combinations+of+reinforcement+learning+controllers+and+recurrent+neural+world+models%2E+arXiv+preprint%2C+2015%2E)
55. Jürgen Schmidhuber, Jieyu Zhao, and Nicol N Schraudolph. Reinforcement learning with self-modifying policies. In Learning to learn. 1998. [find](https://scholar.google.com/scholar?q=Ju%CC%88rgen+Schmidhuber%2C+Jieyu+Zhao%2C+and+Nicol+N+Schraudolph%2E+Reinforcement+learning+with+self%2Dmodifying+policies%2E+In+Learning+to+learn%2E+1998%2E)
56. Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=Noah+Shinn%2C+Beck+Labash%2C+and+Ashwin+Gopinath%2E+Reflexion%3A+an+autonomous+agent+with+dynamic+memory+and+self%2Dreflection%2E+arXiv+preprint%2C+2023%2E)
57. Marta Skreta, Naruki Yoshikawa, Sebastian Arellano-Rubach, Zhi Ji, Lasse Bjørn Kristensen, Kourosh Darvish, Alán Aspuru-Guzik, Florian Shkurti, and Animesh Garg. Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting. arXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=Marta+Skreta%2C+Naruki+Yoshikawa%2C+Sebastian+Arellano%2DRubach%2C+Zhi+Ji%2C+Lasse+Bj%C3%B8rn+Kristensen%2C+Kourosh+Darvish%2C+Ala%CC%81n+Aspuru%2DGuzik%2C+Florian+Shkurti%2C+and+Animesh+Garg%2E+Errors+are+useful+prompts%3A+Instructio)
58. Elliot Soloway. Learning to program = learning to construct mechanisms and explanations. Communications of the ACM, 1986. [find](https://scholar.google.com/scholar?q=Elliot+Soloway%2E+Learning+to+program+%3D+learning+to+construct+mechanisms+and+explanations%2E+Communications+of+the+ACM%2C+1986%2E)
59. Yashar Talebirad and Amirhossein Nadiri. Multi-agent collaboration: Harnessing the power of intelligent llm agents, 2023. [find](https://scholar.google.com/scholar?q=Yashar+Talebirad+and+Amirhossein+Nadiri%2E+Multi%2Dagent+collaboration%3A+Harnessing+the+power+of+intelligent+llm+agents%2C+2023%2E)
60. Xiangru Tang, Bill Qian, Rick Gao, Jiakang Chen, Xinyun Chen, and Mark Gerstein. Biocoder: A benchmark for bioinformatics code generation with contextual pragmatic knowledge. arXiv preprint arXiv:2308.16458, 2023a. [find](https://scholar.google.com/scholar?q=Xiangru+Tang%2C+Bill+Qian%2C+Rick+Gao%2C+Jiakang+Chen%2C+Xinyun+Chen%2C+and+Mark+Gerstein%2E+Biocoder%3A+A+benchmark+for+bioinformatics+code+generation+with+contextual+pragmatic+knowledge%2E+arXiv+preprint+arXiv%3A2308)
61. Xiangru Tang, Anni Zou, Zhuosheng Zhang, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. Medagents: Large language models as collaborators for zero-shot medical reasoning. arXiv preprint arXiv:2311.10537, 2023b. [find](https://scholar.google.com/scholar?q=Xiangru+Tang%2C+Anni+Zou%2C+Zhuosheng+Zhang%2C+Yilun+Zhao%2C+Xingyao+Zhang%2C+Arman+Cohan%2C+and+Mark+Gerstein%2E+Medagents%3A+Large+language+models+as+collaborators+for+zero%2Dshot+medical+reasoning%2E+arXiv+preprint+ar) [our card](https://md.research.elamur.ai/free/en/papers/tang-2024-medagents-large-language-models-as/)
62. Torantulino et al. Auto-GPT, 2023. [find](https://scholar.google.com/scholar?q=Torantulino+et+al%2E+Auto%2DGPT%2C+2023%2E)
63. Auto-gpt. [https://github.com/Significant-Gravitas/R](https://github.com/Significant-Gravitas/R). J. Waldinger and R. C. T. Lee. PROW: a step toward automatic program writing. In D. E. Walker and L. M. Norton (eds.), Proceedings of the 1st International Joint Conference on Artificial Intelligence (IJCAI), 1969. [↗](https://github.com/Significant-Gravitas/R)
64. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint, 2023a. [find](https://scholar.google.com/scholar?q=Guanzhi+Wang%2C+Yuqi+Xie%2C+Yunfan+Jiang%2C+Ajay+Mandlekar%2C+Chaowei+Xiao%2C+Yuke+Zhu%2C+Linxi+Fan%2C+and+Anima+Anandkumar%2E+Voyager%3A+An+open%2Dended+embodied+agent+with+large+language+models%2E+arXiv+preprint%2C+2023a%2E)
65. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. arXiv preprint, 2023b. [find](https://scholar.google.com/scholar?q=Lei+Wang%2C+Chen+Ma%2C+Xueyang+Feng%2C+Zeyu+Zhang%2C+Hao+Yang%2C+Jingsen+Zhang%2C+Zhiyuan+Chen%2C+Jiakai+Tang%2C+Xu+Chen%2C+Yankai+Lin%2C+et+al%2E+A+survey+on+large+language+model+based+autonomous+agents%2E+arXiv+preprint%2C+2) [our card](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-large-language-model-based/)
66. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint, 2022. [find](https://scholar.google.com/scholar?q=Xuezhi+Wang%2C+Jason+Wei%2C+Dale+Schuurmans%2C+Quoc+Le%2C+Ed+Chi%2C+Sharan+Narang%2C+Aakanksha+Chowdhery%2C+and+Denny+Zhou%2E+Self%2Dconsistency+improves+chain+of+thought+reasoning+in+language+models%2E+arXiv+preprint%2C+2) [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/)
67. Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. arXiv preprint, 2023c. [find](https://scholar.google.com/scholar?q=Zhenhailong+Wang%2C+Shaoguang+Mao%2C+Wenshan+Wu%2C+Tao+Ge%2C+Furu+Wei%2C+and+Heng+Ji%2E+Unleashing+cognitive+synergy+in+large+language+models%3A+A+task%2Dsolving+agent+through+multi%2Dpersona+self%2Dcollaboration%2E+arXiv+)
68. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 2022. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Maarten+Bosma%2C+Fei+Xia%2C+Ed+Chi%2C+Quoc+V+Le%2C+Denny+Zhou%2C+et+al%2E+Chain%2Dof%2Dthought+prompting+elicits+reasoning+in+large+language+models%2E+NeurIPS%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
69. Michael Wooldridge and Nicholas R. Jennings. Pitfalls of agent-oriented development. In Proceedings of the Second International Conference on Autonomous Agents, 1998. URL https://doi.org/10.1145/280765.280867. — [doi:10.1145/280765.280867](https://doi.org/10.1145/280765.280867) [↗](https://doi.org/10.1145/280765.280867)
70. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint, 2022. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Jeffrey+Zhao%2C+Dian+Yu%2C+Nan+Du%2C+Izhak+Shafran%2C+Karthik+Narasimhan%2C+and+Yuan+Cao%2E+React%3A+Synergizing+reasoning+and+acting+in+language+models%2E+arXiv+preprint%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/)
71. Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. Self-taught optimizer (stop): Recursively self-improving code generation. arXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=Eric+Zelikman%2C+Eliana+Lorch%2C+Lester+Mackey%2C+and+Adam+Tauman+Kalai%2E+Self%2Dtaught+optimizer+%28stop%29%3A+Recursively+self%2Dimproving+code+generation%2E+arXiv+preprint%2C+2023%2E)
72. Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models. arXiv preprint, 2023a. [find](https://scholar.google.com/scholar?q=Hongxin+Zhang%2C+Weihua+Du%2C+Jiaming+Shan%2C+Qinhong+Zhou%2C+Yilun+Du%2C+Joshua+B+Tenenbaum%2C+Tianmin+Shu%2C+and+Chuang+Gan%2E+Building+cooperative+embodied+agents+modularly+with+large+language+models%2E+arXiv+prepri)
73. Zhuosheng Zhang, Yao Yao, Aston Zhang, Xiangru Tang, Xinbei Ma, Zhiwei He, Yiming Wang, Mark Gerstein, Rui Wang, Gongshen Liu, et al. Igniting language intelligence: The hitchhiker’s guide from chain-of-thought reasoning to language agents. arXiv preprint arXiv:2311.11797, 2023b. — [arXiv:2311.11797](https://arxiv.org/abs/2311.11797) [find](https://scholar.google.com/scholar?q=Zhuosheng+Zhang%2C+Yao+Yao%2C+Aston+Zhang%2C+Xiangru+Tang%2C+Xinbei+Ma%2C+Zhiwei+He%2C+Yiming+Wang%2C+Mark+Gerstein%2C+Rui+Wang%2C+Gongshen+Liu%2C+et+al%2E+Igniting+language+intelligence%3A+The+hitchhiker%E2%80%99s+guide+from+chain%2D)
74. Xufeng Zhao, Mengdi Li, Cornelius Weber, Muhammad Burhan Hafez, and Stefan Wermter. Chat with the environment: Interactive multimodal perception using large language models. arXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=Xufeng+Zhao%2C+Mengdi+Li%2C+Cornelius+Weber%2C+Muhammad+Burhan+Hafez%2C+and+Stefan+Wermter%2E+Chat+with+the+environment%3A+Interactive+multimodal+perception+using+large+language+models%2E+arXiv+preprint%2C+2023%2E)
75. Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x, 2023. [find](https://scholar.google.com/scholar?q=Qinkai+Zheng%2C+Xiao+Xia%2C+Xu+Zou%2C+Yuxiao+Dong%2C+Shan+Wang%2C+Yufei+Xue%2C+Zihan+Wang%2C+Lei+Shen%2C+Andi+Wang%2C+Yang+Li%2C+Teng+Su%2C+Zhilin+Yang%2C+and+Jie+Tang%2E+Codegeex%3A+A+pre%2Dtrained+model+for+code+generation+with+)
76. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint, 2023a. [find](https://scholar.google.com/scholar?q=Shuyan+Zhou%2C+Frank+F+Xu%2C+Hao+Zhu%2C+Xuhui+Zhou%2C+Robert+Lo%2C+Abishek+Sridhar%2C+Xianyi+Cheng%2C+Yonatan+Bisk%2C+Daniel+Fried%2C+Uri+Alon%2C+et+al%2E+Webarena%3A+A+realistic+web+environment+for+building+autonomous+agent) [our card](https://md.research.elamur.ai/free/en/papers/zhou-2023-webarena-realistic-web-environment-building/)
77. Wangchunshu Zhou, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang, Shi Qiu, Jintian Zhang, Jing Chen, Ruipu Wu, Shuai Wang, et al. Agents: An open-source framework for autonomous language agents. arXiv preprint arXiv:2309.07870, 2023b. [find](https://scholar.google.com/scholar?q=Wangchunshu+Zhou%2C+Yuchen+Eleanor+Jiang%2C+Long+Li%2C+Jialong+Wu%2C+Tiannan+Wang%2C+Shi+Qiu%2C+Jintian+Zhang%2C+Jing+Chen%2C+Ruipu+Wu%2C+Shuai+Wang%2C+et+al%2E+Agents%3A+An+open%2Dsource+framework+for+autonomous+language+agen)
78. Mingchen Zhuge, Haozhe Liu, Francesco Faccio, Dylan R Ashley, Róbert Csordás, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki Irie, et al. Mindstorms in natural language-based societies of mind. arXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=Mingchen+Zhuge%2C+Haozhe+Liu%2C+Francesco+Faccio%2C+Dylan+R+Ashley%2C+Ro%CC%81bert+Csorda%CC%81s%2C+Anand+Gopalakrishnan%2C+Abdullah+Hamdi%2C+Hasan+Abed+Al+Kader+Hammoud%2C+Vincent+Herrmann%2C+Kazuki+Irie%2C+et+al%2E+Mindstorms+in+n)