---
title: "SWE-agent: Agent-Computer Interfaces Enable — Yang 2024"
description: "Language-model agents that work on real software usually go through tools built for humans, such as a plain Linux shell."
authors: ["John Yang", "Carlos E. Jimenez", "Alexander Wettig", "Kilian Lieret", "Shunyu Yao", "Karthik Narasimhan", "Ofir Press"]
year: 2024
venue: "arXiv:2405.15793"
source: "https://arxiv.org/abs/2405.15793"
doi: "10.48550/arxiv.2405.15793"
arxiv: "2405.15793"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yang-2024-swe-agent-agent-computer-interfaces/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2024-swe-agent-agent-computer-interfaces/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering

# SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering

**Authors:** [John Yang](https://md.research.elamur.ai/free/en/authors/john-yang/), [Carlos E. Jimenez](https://md.research.elamur.ai/free/en/authors/carlos-e-jimenez/), [Alexander Wettig](https://md.research.elamur.ai/free/en/authors/alexander-wettig/), Kilian Lieret, [Shunyu Yao](https://md.research.elamur.ai/free/en/authors/shunyu-yao/), [Karthik Narasimhan](https://md.research.elamur.ai/free/en/authors/karthik-narasimhan/), [Ofir Press](https://md.research.elamur.ai/free/en/authors/ofir-press/) · **Year:** 2024 · **Venue:** arXiv:2405.15793 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2405.15793) · [PDF](https://arxiv.org/pdf/2405.15793) · [DOI 10.48550/arxiv.2405.15793](https://doi.org/10.48550/arxiv.2405.15793) · [arXiv 2405.15793](https://arxiv.org/abs/2405.15793) · [OpenAlex](https://openalex.org/W7103754783)

**Citations:** 241 citations (openalex-snapshot · 2026-10-14) · [266](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Language-model agents that work on real software usually go through tools built for humans, such as a plain Linux shell. A shell has many options, makes small edits awkward and says little when an edit goes wrong. The authors argue that LM agents are a separate kind of user that needs its own tooling, which they call an agent-computer interface (ACI). They built SWE-agent, which pairs an LM (mainly GPT-4 Turbo) with a small ACI: search commands with capped output, a file viewer that shows a window of numbered lines, a line-range edit command guarded by a syntax linter, and managed interaction history. The agent works in a ReAct-style loop of thought plus action. The authors refined the interface by studying agent trajectories and by grid-searching configurations on a development set. On the full SWE-bench test set, which is built from real GitHub issues, SWE-agent resolves 12.47% of tasks, compared with 3.8% for the best earlier non-interactive retrieval system. It also reaches 87.7% on HumanEvalFix. The main lesson is that changing the interface, without retraining the model, can bring large gains.

## Key points
- On SWE-bench (2,294 test tasks from 12 Python repositories), SWE-agent with GPT-4 Turbo scores 12.47% pass@1, compared with 3.8% for the earlier RAG baseline. With Claude 3 Opus it scores 10.5%, so the ACI carries over to a different model.
- In ablations on SWE-bench Lite (300 instances), the full ACI solves 10.7 percentage points more tasks than an agent limited to the default Linux shell.
- The four design principles are: simple, well-documented commands; compact actions that do a lot per step; feedback that is informative but short; and guardrails such as rejecting syntactically broken edits.
- Concrete settings: search output is capped at 50 hits (a narrower query is suggested beyond that), and the viewer shows at most 100 lines at a time. Solved tasks take a median of about 12 steps and $1.21.
- Error analysis: 52% of unresolved runs are wrong or overly narrow implementations, and 23.4% come from cascading failed edits. Code, data and a leaderboard are released at swe-agent.com.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Computing & GPUs](https://md.research.elamur.ai/free/en/shelves/computing/)

## Cited by

Cited by

266 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 261 not yet.

- [基于大语言模型智能体的代码生成综述](https://md.research.elamur.ai/free/en/papers/dong-2025-code-generation-llm/)
- [A Survey on Large Language Models for Code Generation](https://md.research.elamur.ai/free/en/papers/jiang-2024-survey-large-language-models-code/)
- [KernelBench: Can LLMs Write Efficient GPU Kernels?](https://md.research.elamur.ai/free/en/papers/ouyang-2025-kernelbench-llms-write-efficient-gpu/)
- [Agent Laboratory: Using LLM Agents as Research Assistants](https://md.research.elamur.ai/free/en/papers/schmidgall-2025-agent-laboratory-using-llm/)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://md.research.elamur.ai/free/en/papers/wang-2024-openhands-open-platform-ai/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{yang2024swe,
  title = {{SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering}},
  author = {Yang, John and Jimenez, Carlos E. and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir},
  year = {2024},
  doi = {10.48550/arxiv.2405.15793},
  eprint = {2405.15793},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.15793}
}
```

- [← previous Survey on Computational Approaches to Implicature](https://md.research.elamur.ai/free/en/papers/anuranjana-2024-survey-computational-approaches-implicature/)
- [next → SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

71 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton. Program synthesis with large language models, 2021. [find](https://scholar.google.com/scholar?q=J%2E+Austin%2C+A%2E+Odena%2C+M%2E+Nye%2C+M%2E+Bosma%2C+H%2E+Michalewski%2C+D%2E+Dohan%2C+E%2E+Jiang%2C+C%2E+Cai%2C+M%2E+Terry%2C+Q%2E+Le%2C+and+C%2E+Sutton%2E+Program+synthesis+with+large+language+models%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/)
2. J. M. Carroll. Human-computer interaction: psychology as a science of design. Annual review of psychology, 48(1):61–83, 1997. [find](https://scholar.google.com/scholar?q=J%2E+M%2E+Carroll%2E+Human%2Dcomputer+interaction%3A+psychology+as+a+science+of+design%2E+Annual+review+of+psychology%2C+48%281%29%3A61%E2%80%9383%2C+1997%2E)
3. F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M.-H. Yee, Y. Zi, C. J. Anderson, M. Q. Feldman, A. Guha, M. Greenberg, and A. Jangda. Multipl-e: A scalable and extensible approach to benchmarking neural code generation, 2022. [find](https://scholar.google.com/scholar?q=F%2E+Cassano%2C+J%2E+Gouwar%2C+D%2E+Nguyen%2C+S%2E+Nguyen%2C+L%2E+Phipps%2DCostin%2C+D%2E+Pinckney%2C+M%2E%2DH%2E+Yee%2C+Y%2E+Zi%2C+C%2E+J%2E+Anderson%2C+M%2E+Q%2E+Feldman%2C+A%2E+Guha%2C+M%2E+Greenberg%2C+and+A%2E+Jangda%2E+Multipl%2De%3A+A+scalable+and+extensible+)
4. S. Chakraborty, Y. Li, M. Irvine, R. Saha, and B. Ray. Entropy guided spectrum based bug localization using statistical language model. arXiv preprint arXiv:1802.06947, 2018. — [arXiv:1802.06947](https://arxiv.org/abs/1802.06947) [find](https://scholar.google.com/scholar?q=S%2E+Chakraborty%2C+Y%2E+Li%2C+M%2E+Irvine%2C+R%2E+Saha%2C+and+B%2E+Ray%2E+Entropy+guided+spectrum+based+bug+localization+using+statistical+language+model%2E+arXiv+preprint+arXiv%3A1802%2E06947%2C+2018%2E+%E2%80%94+%5BarXiv%3A1802%2E06947%5D%28http)
5. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, and J. K. et. al. Evaluating large language models trained on code, 2021. [find](https://scholar.google.com/scholar?q=M%2E+Chen%2C+J%2E+Tworek%2C+H%2E+Jun%2C+Q%2E+Yuan%2C+H%2E+P%2E+de+Oliveira+Pinto%2C+and+J%2E+K%2E+et%2E+al%2E+Evaluating+large+language+models+trained+on+code%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
6. W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, and I. Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024. [find](https://scholar.google.com/scholar?q=W%2E%2DL%2E+Chiang%2C+L%2E+Zheng%2C+Y%2E+Sheng%2C+A%2E+N%2E+Angelopoulos%2C+T%2E+Li%2C+D%2E+Li%2C+H%2E+Zhang%2C+B%2E+Zhu%2C+M%2E+Jordan%2C+J%2E+E%2E+Gonzalez%2C+and+I%2E+Stoica%2E+Chatbot+arena%3A+An+open+platform+for+evaluating+llms+by+human+preference%2C)
7. A. Cooper, R. Reimann, and D. Cronin. About face 3: the essentials of interaction design. John Wiley & Sons, Inc., USA, 2007. ISBN 9780470084113. [find](https://scholar.google.com/scholar?q=A%2E+Cooper%2C+R%2E+Reimann%2C+and+D%2E+Cronin%2E+About+face+3%3A+the+essentials+of+interaction+design%2E+John+Wiley+%26+Sons%2C+Inc%2E%2C+USA%2C+2007%2E+ISBN+9780470084113%2E)
8. Y. Ding, Z. Wang, W. U. Ahmad, H. Ding, M. Tan, N. Jain, M. K. Ramanathan, R. Nallapati, P. Bhatia, D. Roth, and B. Xiang. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks… [find](https://scholar.google.com/scholar?q=Y%2E+Ding%2C+Z%2E+Wang%2C+W%2E+U%2E+Ahmad%2C+H%2E+Ding%2C+M%2E+Tan%2C+N%2E+Jain%2C+M%2E+K%2E+Ramanathan%2C+R%2E+Nallapati%2C+P%2E+Bhatia%2C+D%2E+Roth%2C+and+B%2E+Xiang%2E+Crosscodeeval%3A+A+diverse+and+multilingual+benchmark+for+cross%2Dfile+code+compl)
9. X. Du, M. Liu, K. Wang, H. Wang, J. Liu, Y. Chen, J. Feng, C. Sha, X. Peng, and Y. Lou. Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation, 2023. [find](https://scholar.google.com/scholar?q=X%2E+Du%2C+M%2E+Liu%2C+K%2E+Wang%2C+H%2E+Wang%2C+J%2E+Liu%2C+Y%2E+Chen%2C+J%2E+Feng%2C+C%2E+Sha%2C+X%2E+Peng%2C+and+Y%2E+Lou%2E+Classeval%3A+A+manually%2Dcrafted+benchmark+for+evaluating+llms+on+class%2Dlevel+code+generation%2C+2023%2E)
10. Z. Fan, X. Gao, M. Mirchev, A. Roychoudhury, and S. H. Tan. Automated repair of programs from large language models, 2023. [find](https://scholar.google.com/scholar?q=Z%2E+Fan%2C+X%2E+Gao%2C+M%2E+Mirchev%2C+A%2E+Roychoudhury%2C+and+S%2E+H%2E+Tan%2E+Automated+repair+of+programs+from+large+language+models%2C+2023%2E)
11. R. Fang, R. Bindu, A. Gupta, Q. Zhan, and D. Kang. Llm agents can autonomously hack websites, 2024. [find](https://scholar.google.com/scholar?q=R%2E+Fang%2C+R%2E+Bindu%2C+A%2E+Gupta%2C+Q%2E+Zhan%2C+and+D%2E+Kang%2E+Llm+agents+can+autonomously+hack+websites%2C+2024%2E)
12. T. L. Griffiths. Understanding human intelligence through human limitations. Trends in Cognitive Sciences, 24(11):873–883, 2020. [find](https://scholar.google.com/scholar?q=T%2E+L%2E+Griffiths%2E+Understanding+human+intelligence+through+human+limitations%2E+Trends+in+Cognitive+Sciences%2C+24%2811%29%3A873%E2%80%93883%2C+2020%2E)
13. Y. Gu, Y. Shu, H. Yu, X. Liu, Y. Dong, J. Tang, J. Srinivasa, H. Latapie, and Y. Su. Middleware for llms: Tools are instrumental for language agents in complex environments, 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Gu%2C+Y%2E+Shu%2C+H%2E+Yu%2C+X%2E+Liu%2C+Y%2E+Dong%2C+J%2E+Tang%2C+J%2E+Srinivasa%2C+H%2E+Latapie%2C+and+Y%2E+Su%2E+Middleware+for+llms%3A+Tools+are+instrumental+for+language+agents+in+complex+environments%2C+2024%2E)
14. D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. K. Li, F. Luo, Y. Xiong, and W. Liang. Deepseek-coder: When the large language model meets programming – the rise of code intelligence. CoRR, abs/2401.14196, 2024. URL https://arxiv.org/abs/2401.14196. [↗](https://arxiv.org/abs/2401.14196)
15. D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt. Measuring coding challenge competence with apps, 2021. [find](https://scholar.google.com/scholar?q=D%2E+Hendrycks%2C+S%2E+Basart%2C+S%2E+Kadavath%2C+M%2E+Mazeika%2C+A%2E+Arora%2C+E%2E+Guo%2C+C%2E+Burns%2C+S%2E+Puranik%2C+H%2E+He%2C+D%2E+Song%2C+and+J%2E+Steinhardt%2E+Measuring+coding+challenge+competence+with+apps%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/)
16. S. Holt, M. R. Luyten, and M. van der Schaar. L2MAC: Large language model automatic computer for unbounded code generation. In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=EhrzQwsV4K](https://openreview.net/forum?id=EhrzQwsV4K). [↗](https://openreview.net/forum?id=EhrzQwsV4K)
17. S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, C. Zhang, J. Wang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber. Metagpt: Meta programming for a multi-agent collaborative framework, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Hong%2C+M%2E+Zhuge%2C+J%2E+Chen%2C+X%2E+Zheng%2C+Y%2E+Cheng%2C+C%2E+Zhang%2C+J%2E+Wang%2C+Z%2E+Wang%2C+S%2E+K%2E+S%2E+Yau%2C+Z%2E+Lin%2C+L%2E+Zhou%2C+C%2E+Ran%2C+L%2E+Xiao%2C+C%2E+Wu%2C+and+J%2E+Schmidhuber%2E+Metagpt%3A+Meta+programming+for+a+multi%2Dagent+colla) [our card](https://md.research.elamur.ai/free/en/papers/hong-2023-metagpt-meta-programming-multi/)
18. Q. Huang, J. Vora, P. Liang, and J. Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation, 2024. [find](https://scholar.google.com/scholar?q=Q%2E+Huang%2C+J%2E+Vora%2C+P%2E+Liang%2C+and+J%2E+Leskovec%2E+Mlagentbench%3A+Evaluating+language+agents+on+machine+learning+experimentation%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/huang-2023-mlagentbench-evaluating-language-agents/)
19. N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024. [find](https://scholar.google.com/scholar?q=N%2E+Jain%2C+K%2E+Han%2C+A%2E+Gu%2C+W%2E%2DD%2E+Li%2C+F%2E+Yan%2C+T%2E+Zhang%2C+S%2E+Wang%2C+A%2E+Solar%2DLezama%2C+K%2E+Sen%2C+and+I%2E+Stoica%2E+Livecodebench%3A+Holistic+and+contamination+free+evaluation+of+large+language+models+for+code%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/)
20. C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=VTF8yNQM66. [↗](https://openreview.net/forum?) [our card](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/)
21. R. Just, D. Jalali, and M. D. Ernst. Defects4J: A Database of existing faults to enable controlled testing studies for Java programs. In ISSTA 2014, Proceedings of the 2014 International Symposium on Software Testing and Analysis, pages 437–440, San Jose, CA, USA, July 2014. Tool demo. [find](https://scholar.google.com/scholar?q=R%2E+Just%2C+D%2E+Jalali%2C+and+M%2E+D%2E+Ernst%2E+Defects4J%3A+A+Database+of+existing+faults+to+enable+controlled+testing+studies+for+Java+programs%2E+In+ISSTA+2014%2C+Proceedings+of+the+2014+International+Symposium+on+)
22. S. Kang, J. Yoon, and S. Yoo. Large language models are few-shot testers: Exploring llm-based general bug reproduction, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Kang%2C+J%2E+Yoon%2C+and+S%2E+Yoo%2E+Large+language+models+are+few%2Dshot+testers%3A+Exploring+llm%2Dbased+general+bug+reproduction%2C+2023%2E)
23. R.-M. Karampatsis and C. Sutton. How often do single-statement bugs occur? the manysstubs4j dataset. 2020 IEEE/ACM 17th International Conference on Mining Software Repositories (MSR), pages 573–577, 2019. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 173188438. [↗](https://api.semanticscholar.org/CorpusID:)
24. J. Y. Koh, R. Lo, L. Jang, V. Duvvur, M. C. Lim, P.-Y. Huang, G. Neubig, S. Zhou, R. Salakhutdinov, and D. Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024. [find](https://scholar.google.com/scholar?q=J%2E+Y%2E+Koh%2C+R%2E+Lo%2C+L%2E+Jang%2C+V%2E+Duvvur%2C+M%2E+C%2E+Lim%2C+P%2E%2DY%2E+Huang%2C+G%2E+Neubig%2C+S%2E+Zhou%2C+R%2E+Salakhutdinov%2C+and+D%2E+Fried%2E+Visualwebarena%3A+Evaluating+multimodal+agents+on+realistic+visual+web+tasks%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/koh-2024-visualwebarena-evaluating-multimodal-agents-realistic/)
25. Y. Lai, C. Li, Y. Wang, T. Zhang, R. Zhong, L. Zettlemoyer, S. W. tau Yih, D. Fried, S. Wang, and T. Yu. Ds-1000: A natural and reliable benchmark for data science code generation, 2022. [find](https://scholar.google.com/scholar?q=Y%2E+Lai%2C+C%2E+Li%2C+Y%2E+Wang%2C+T%2E+Zhang%2C+R%2E+Zhong%2C+L%2E+Zettlemoyer%2C+S%2E+W%2E+tau+Yih%2C+D%2E+Fried%2C+S%2E+Wang%2C+and+T%2E+Yu%2E+Ds%2D1000%3A+A+natural+and+reliable+benchmark+for+data+science+code+generation%2C+2022%2E)
26. J. Liu, C. S. Xia, Y. Wang, and L. Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.01210, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Liu%2C+C%2E+S%2E+Xia%2C+Y%2E+Wang%2C+and+L%2E+Zhang%2E+Is+your+code+generated+by+chatgpt+really+correct%3F+rigorous+evaluation+of+large+language+models+for+code+generation%2E+arXiv+preprint+arXiv%3A2305%2E01210%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/)
27. N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. Lost in the middle: How language models use long contexts, 2023. [find](https://scholar.google.com/scholar?q=N%2E+F%2E+Liu%2C+K%2E+Lin%2C+J%2E+Hewitt%2C+A%2E+Paranjape%2C+M%2E+Bevilacqua%2C+F%2E+Petroni%2C+and+P%2E+Liang%2E+Lost+in+the+middle%3A+How+language+models+use+long+contexts%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/)
28. T. Liu, C. Xu, and J. McAuley. Repobench: Benchmarking repository-level code autocompletion systems. In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=pPjZIOuQuF](https://openreview.net/forum?id=pPjZIOuQuF). [↗](https://openreview.net/forum?id=pPjZIOuQuF)
29. Y. Liu, X. Tang, Z. Cai, J. Lu, Y. Zhang, Y. Shao, Z. Deng, H. Hu, K. An, R. Huang, S. Si, S. Chen, H. Zhao, L. Chen, Y. Wang, T. Liu, Z. Jiang, B. Chang, Y. Qin, W. Zhou, Y. Zhao, A. Cohan, and M. Gerstein. Ml-bench: Evaluating large language models for code generation in repository-level machine l… [find](https://scholar.google.com/scholar?q=Y%2E+Liu%2C+X%2E+Tang%2C+Z%2E+Cai%2C+J%2E+Lu%2C+Y%2E+Zhang%2C+Y%2E+Shao%2C+Z%2E+Deng%2C+H%2E+Hu%2C+K%2E+An%2C+R%2E+Huang%2C+S%2E+Si%2C+S%2E+Chen%2C+H%2E+Zhao%2C+L%2E+Chen%2C+Y%2E+Wang%2C+T%2E+Liu%2C+Z%2E+Jiang%2C+B%2E+Chang%2C+Y%2E+Qin%2C+W%2E+Zhou%2C+Y%2E+Zhao%2C+A%2E+Cohan%2C+and+M%2E+Ge)
30. S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. Liu. Codexglue: A machine learning benchmark dataset for code understanding and gener… [find](https://scholar.google.com/scholar?q=S%2E+Lu%2C+D%2E+Guo%2C+S%2E+Ren%2C+J%2E+Huang%2C+A%2E+Svyatkovskiy%2C+A%2E+Blanco%2C+C%2E+Clement%2C+D%2E+Drain%2C+D%2E+Jiang%2C+D%2E+Tang%2C+G%2E+Li%2C+L%2E+Zhou%2C+L%2E+Shou%2C+L%2E+Zhou%2C+M%2E+Tufano%2C+M%2E+Gong%2C+M%2E+Zhou%2C+N%2E+Duan%2C+N%2E+Sundaresan%2C+S%2E+K%2E+Deng%2C)
31. R. T. McCoy, S. Yao, D. Friedman, M. Hardy, and T. L. Griffiths. Embers of autoregression: Understanding large language models through the problem they are trained to solve. arXiv preprint arXiv:2309.13638, 2023. [find](https://scholar.google.com/scholar?q=R%2E+T%2E+McCoy%2C+S%2E+Yao%2C+D%2E+Friedman%2C+M%2E+Hardy%2C+and+T%2E+L%2E+Griffiths%2E+Embers+of+autoregression%3A+Understanding+large+language+models+through+the+problem+they+are+trained+to+solve%2E+arXiv+preprint+arXiv%3A2309%2E)
32. N. Muennighoff, Q. Liu, A. R. Zebaze, Q. Zheng, B. Hui, T. Y. Zhuo, S. Singh, X. Tang, L. V. Werra, and S. Longpre. Octopack: Instruction tuning code large language models. In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=mw1PWNSWZP](https://openreview.net/forum?id=mw1PWNSWZP). [↗](https://openreview.net/forum?id=mw1PWNSWZP)
33. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman. Webgpt: Browser-assisted question-answering with human feedback, 2022. [find](https://scholar.google.com/scholar?q=R%2E+Nakano%2C+J%2E+Hilton%2C+S%2E+Balaji%2C+J%2E+Wu%2C+L%2E+Ouyang%2C+C%2E+Kim%2C+C%2E+Hesse%2C+S%2E+Jain%2C+V%2E+Kosaraju%2C+W%2E+Saunders%2C+X%2E+Jiang%2C+K%2E+Cobbe%2C+T%2E+Eloundou%2C+G%2E+Krueger%2C+K%2E+Button%2C+M%2E+Knight%2C+B%2E+Chess%2C+and+J%2E+Schulman%2E+We) [our card](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)
34. OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V. Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bogdonoff, O. Boiko, M… [find](https://scholar.google.com/scholar?q=OpenAI%2C+J%2E+Achiam%2C+S%2E+Adler%2C+S%2E+Agarwal%2C+L%2E+Ahmad%2C+I%2E+Akkaya%2C+F%2E+L%2E+Aleman%2C+D%2E+Almeida%2C+J%2E+Altenschmidt%2C+S%2E+Altman%2C+S%2E+Anadkat%2C+R%2E+Avila%2C+I%2E+Babuschkin%2C+S%2E+Balaji%2C+V%2E+Balcom%2C+P%2E+Baltescu%2C+H%2E+Bao%2C+M%2E+B) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
35. C. Packer, S. Wooders, K. Lin, V. Fang, S. G. Patil, I. Stoica, and J. E. Gonzalez. Memgpt: Towards llms as operating systems, 2024. [find](https://scholar.google.com/scholar?q=C%2E+Packer%2C+S%2E+Wooders%2C+K%2E+Lin%2C+V%2E+Fang%2C+S%2E+G%2E+Patil%2C+I%2E+Stoica%2C+and+J%2E+E%2E+Gonzalez%2E+Memgpt%3A+Towards+llms+as+operating+systems%2C+2024%2E)
36. O. Press, M. Zhang, S. Min, L. Schmidt, N. Smith, and M. Lewis. Measuring and narrowing the compositionality gap in language models. In H. Bouamor, J. Pino, and K. Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687–5711, Singapore, Dec. 2023. Association… [find](https://scholar.google.com/scholar?q=O%2E+Press%2C+M%2E+Zhang%2C+S%2E+Min%2C+L%2E+Schmidt%2C+N%2E+Smith%2C+and+M%2E+Lewis%2E+Measuring+and+narrowing+the+compositionality+gap+in+language+models%2E+In+H%2E+Bouamor%2C+J%2E+Pino%2C+and+K%2E+Bali%2C+editors%2C+Findings+of+the+Assoc)
37. M. Shao, B. Chen, S. Jancheska, B. Dolan-Gavitt, S. Garg, R. Karri, and M. Shafique. An empirical evaluation of llms for solving offensive security challenges, 2024. [find](https://scholar.google.com/scholar?q=M%2E+Shao%2C+B%2E+Chen%2C+S%2E+Jancheska%2C+B%2E+Dolan%2DGavitt%2C+S%2E+Garg%2C+R%2E+Karri%2C+and+M%2E+Shafique%2E+An+empirical+evaluation+of+llms+for+solving+offensive+security+challenges%2C+2024%2E)
38. W. Shi, R. Xu, Y. Zhuang, Y. Yu, J. Zhang, H. Wu, Y. Zhu, J. Ho, C. Yang, and M. D. Wang. Ehragent: Code empowers large language models for few-shot complex tabular reasoning on electronic health records, 2024. [find](https://scholar.google.com/scholar?q=W%2E+Shi%2C+R%2E+Xu%2C+Y%2E+Zhuang%2C+Y%2E+Yu%2C+J%2E+Zhang%2C+H%2E+Wu%2C+Y%2E+Zhu%2C+J%2E+Ho%2C+C%2E+Yang%2C+and+M%2E+D%2E+Wang%2E+Ehragent%3A+Code+empowers+large+language+models+for+few%2Dshot+complex+tabular+reasoning+on+electronic+health+reco)
39. N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. [find](https://scholar.google.com/scholar?q=N%2E+Shinn%2C+F%2E+Cassano%2C+E%2E+Berman%2C+A%2E+Gopinath%2C+K%2E+Narasimhan%2C+and+S%2E+Yao%2E+Reflexion%3A+Language+agents+with+verbal+reinforcement+learning%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/)
40. D. Sobania, M. Briesch, C. Hanna, and J. Petke. An analysis of the automatic bug fixing performance of chatgpt, 2023. [find](https://scholar.google.com/scholar?q=D%2E+Sobania%2C+M%2E+Briesch%2C+C%2E+Hanna%2C+and+J%2E+Petke%2E+An+analysis+of+the+automatic+bug+fixing+performance+of+chatgpt%2C+2023%2E)
41. A. Sridhar, R. Lo, F. F. Xu, H. Zhu, and S. Zhou. Hierarchical prompting assists large language model on web navigation, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Sridhar%2C+R%2E+Lo%2C+F%2E+F%2E+Xu%2C+H%2E+Zhu%2C+and+S%2E+Zhou%2E+Hierarchical+prompting+assists+large+language+model+on+web+navigation%2C+2023%2E)
42. T. Sumers, S. Yao, K. Narasimhan, and T. L. Griffiths. Cognitive architectures for language agents, 2023. [find](https://scholar.google.com/scholar?q=T%2E+Sumers%2C+S%2E+Yao%2C+K%2E+Narasimhan%2C+and+T%2E+L%2E+Griffiths%2E+Cognitive+architectures+for+language+agents%2C+2023%2E)
43. X. Tang, A. Zou, Z. Zhang, Z. Li, Y. Zhao, X. Zhang, A. Cohan, and M. Gerstein. Medagents: Large language models as collaborators for zero-shot medical reasoning, 2024. [find](https://scholar.google.com/scholar?q=X%2E+Tang%2C+A%2E+Zou%2C+Z%2E+Zhang%2C+Z%2E+Li%2C+Y%2E+Zhao%2C+X%2E+Zhang%2C+A%2E+Cohan%2C+and+M%2E+Gerstein%2E+Medagents%3A+Large+language+models+as+collaborators+for+zero%2Dshot+medical+reasoning%2C+2024%2E)
44. A. Thakur, G. Tsoukalas, Y. Wen, J. Xin, and S. Chaudhuri. An in-context learning agent for formal theorem-proving, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Thakur%2C+G%2E+Tsoukalas%2C+Y%2E+Wen%2C+J%2E+Xin%2C+and+S%2E+Chaudhuri%2E+An+in%2Dcontext+learning+agent+for+formal+theorem%2Dproving%2C+2024%2E)
45. R. Thoppilan, D. D. Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, Y. Li, H. Lee, H. S. Zheng, A. Ghafouri, M. Menegali, Y. Huang, M. Krikun, D. Lepikhin, J. Qin, D. Chen, Y. Xu, Z. Chen, A. Roberts, M. Bosma, V. Zhao, Y. Zhou, C.-C. Chang, I. Krivokon,… [find](https://scholar.google.com/scholar?q=R%2E+Thoppilan%2C+D%2E+D%2E+Freitas%2C+J%2E+Hall%2C+N%2E+Shazeer%2C+A%2E+Kulshreshtha%2C+H%2E%2DT%2E+Cheng%2C+A%2E+Jin%2C+T%2E+Bos%2C+L%2E+Baker%2C+Y%2E+Du%2C+Y%2E+Li%2C+H%2E+Lee%2C+H%2E+S%2E+Zheng%2C+A%2E+Ghafouri%2C+M%2E+Menegali%2C+Y%2E+Huang%2C+M%2E+Krikun%2C+D%2E+Lepikhin%2C) [our card](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/)
46. J. Wang, Y. Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang. Software testing with large language model: Survey, landscape, and vision, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Wang%2C+Y%2E+Huang%2C+C%2E+Chen%2C+Z%2E+Liu%2C+S%2E+Wang%2C+and+Q%2E+Wang%2E+Software+testing+with+large+language+model%3A+Survey%2C+landscape%2C+and+vision%2C+2023%2E)
47. L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, W. X. Zhao, Z. Wei, and J. Wen. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6), Mar. 2024. ISSN 2095-2236. doi: 10.1007/s11704-024-40231-1. URL http://dx.doi.org/1… [find](https://scholar.google.com/scholar?q=L%2E+Wang%2C+C%2E+Ma%2C+X%2E+Feng%2C+Z%2E+Zhang%2C+H%2E+Yang%2C+J%2E+Zhang%2C+Z%2E+Chen%2C+J%2E+Tang%2C+X%2E+Chen%2C+Y%2E+Lin%2C+W%2E+X%2E+Zhao%2C+Z%2E+Wei%2C+and+J%2E+Wen%2E+A+survey+on+large+language+model+based+autonomous+agents%2E+Frontiers+of+Computer) [our card](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-large-language-model-based/)
48. X. Wang, Y. Chen, L. Yuan, Y. Zhang, Y. Li, H. Peng, and H. Ji. Executable code actions elicit better llm agents, 2024. [find](https://scholar.google.com/scholar?q=X%2E+Wang%2C+Y%2E+Chen%2C+L%2E+Yuan%2C+Y%2E+Zhang%2C+Y%2E+Li%2C+H%2E+Peng%2C+and+H%2E+Ji%2E+Executable+code+actions+elicit+better+llm+agents%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2024-executable-code-actions-elicit-better/)
49. Z. Wang, G. Cuenca, S. Zhou, F. F. Xu, and G. Neubig. Mconala: A benchmark for code generation from multiple natural languages, 2023. [find](https://scholar.google.com/scholar?q=Z%2E+Wang%2C+G%2E+Cuenca%2C+S%2E+Zhou%2C+F%2E+F%2E+Xu%2C+and+G%2E+Neubig%2E+Mconala%3A+A+benchmark+for+code+generation+from+multiple+natural+languages%2C+2023%2E)
50. Z. Wang, S. Zhou, D. Fried, and G. Neubig. Execution-based evaluation for open-domain code generation, 2023. [find](https://scholar.google.com/scholar?q=Z%2E+Wang%2C+S%2E+Zhou%2C+D%2E+Fried%2C+and+G%2E+Neubig%2E+Execution%2Dbased+evaluation+for+open%2Ddomain+code+generation%2C+2023%2E)
51. Z. Wang, D. Fried, and G. Neubig. Trove: Inducing verifiable and efficient toolboxes for solving programmatic tasks, 2024. [find](https://scholar.google.com/scholar?q=Z%2E+Wang%2C+D%2E+Fried%2C+and+G%2E+Neubig%2E+Trove%3A+Inducing+verifiable+and+efficient+toolboxes+for+solving+programmatic+tasks%2C+2024%2E)
52. M. Wornow, A. Narayan, K. Opsahl-Ong, Q. McIntyre, N. H. Shah, and C. Re. Automating the enterprise with foundation models, 2024. [find](https://scholar.google.com/scholar?q=M%2E+Wornow%2C+A%2E+Narayan%2C+K%2E+Opsahl%2DOng%2C+Q%2E+McIntyre%2C+N%2E+H%2E+Shah%2C+and+C%2E+Re%2E+Automating+the+enterprise+with+foundation+models%2C+2024%2E)
53. Z. Wu, C. Han, Z. Ding, Z. Weng, Z. Liu, S. Yao, T. Yu, and L. Kong. Os-copilot: Towards generalist computer agents with self-improvement, 2024. [find](https://scholar.google.com/scholar?q=Z%2E+Wu%2C+C%2E+Han%2C+Z%2E+Ding%2C+Z%2E+Weng%2C+Z%2E+Liu%2C+S%2E+Yao%2C+T%2E+Yu%2C+and+L%2E+Kong%2E+Os%2Dcopilot%3A+Towards+generalist+computer+agents+with+self%2Dimprovement%2C+2024%2E)
54. Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, R. Zheng, X. Fan, X. Wang, L. Xiong, Y. Zhou, W. Wang, C. Jiang, Y. Zou, X. Liu, Z. Yin, S. Dou, R. Weng, W. Cheng, Q. Zhang, W. Qin, Y. Zheng, X. Qiu, X. Huang, and T. Gui. The rise and potential of large language… [find](https://scholar.google.com/scholar?q=Z%2E+Xi%2C+W%2E+Chen%2C+X%2E+Guo%2C+W%2E+He%2C+Y%2E+Ding%2C+B%2E+Hong%2C+M%2E+Zhang%2C+J%2E+Wang%2C+S%2E+Jin%2C+E%2E+Zhou%2C+R%2E+Zheng%2C+X%2E+Fan%2C+X%2E+Wang%2C+L%2E+Xiong%2C+Y%2E+Zhou%2C+W%2E+Wang%2C+C%2E+Jiang%2C+Y%2E+Zou%2C+X%2E+Liu%2C+Z%2E+Yin%2C+S%2E+Dou%2C+R%2E+Weng%2C+W%2E+Cheng%2C) [our card](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/)
55. C. S. Xia and L. Zhang. Less training, more repairing please: revisiting automated program repair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 959–971, 2022. [find](https://scholar.google.com/scholar?q=C%2E+S%2E+Xia+and+L%2E+Zhang%2E+Less+training%2C+more+repairing+please%3A+revisiting+automated+program+repair+via+zero%2Dshot+learning%2E+In+Proceedings+of+the+30th+ACM+Joint+European+Software+Engineering+Conference+)
56. C. S. Xia, M. Paltenghi, J. L. Tian, M. Pradel, and L. Zhang. Universal fuzzing via large language models. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2023. [find](https://scholar.google.com/scholar?q=C%2E+S%2E+Xia%2C+M%2E+Paltenghi%2C+J%2E+L%2E+Tian%2C+M%2E+Pradel%2C+and+L%2E+Zhang%2E+Universal+fuzzing+via+large+language+models%2E+In+Proceedings+of+the+46th+IEEE%2FACM+International+Conference+on+Software+Engineering%2C+2023%2E)
57. T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, Y. Liu, Y. Xu, S. Zhou, S. Savarese, C. Xiong, V. Zhong, and T. Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024. [find](https://scholar.google.com/scholar?q=T%2E+Xie%2C+D%2E+Zhang%2C+J%2E+Chen%2C+X%2E+Li%2C+S%2E+Zhao%2C+R%2E+Cao%2C+T%2E+J%2E+Hua%2C+Z%2E+Cheng%2C+D%2E+Shin%2C+F%2E+Lei%2C+Y%2E+Liu%2C+Y%2E+Xu%2C+S%2E+Zhou%2C+S%2E+Savarese%2C+C%2E+Xiong%2C+V%2E+Zhong%2C+and+T%2E+Yu%2E+Osworld%3A+Benchmarking+multimodal+agents+for) [our card](https://md.research.elamur.ai/free/en/papers/xie-2024-osworld-benchmarking-multimodal-agents-open/)
58. A. Z. H. Yang, C. Le Goues, R. Martins, and V. Hellendoorn. Large language models for test-free fault localization. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE ’24, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400702174. doi: 1… — [doi:10.1145/3597503.3623342](https://doi.org/10.1145/3597503.3623342) [find](https://scholar.google.com/scholar?q=A%2E+Z%2E+H%2E+Yang%2C+C%2E+Le+Goues%2C+R%2E+Martins%2C+and+V%2E+Hellendoorn%2E+Large+language+models+for+test%2Dfree+fault+localization%2E+In+Proceedings+of+the+IEEE%2FACM+46th+International+Conference+on+Software+Engineering)
59. J. Yang, A. Prabhakar, K. R. Narasimhan, and S. Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL [https://openreview.net/forum?id=fvKaLF1ns8](https://openreview.net/forum?id=fvKaLF1ns8). [↗](https://openreview.net/forum?id=fvKaLF1ns8)
60. J. Yang, A. Prabhakar, S. Yao, K. Pei, and K. R. Narasimhan. Language agents as hackers: Evaluating cybersecurity skills with capture the flag. In Multi-Agent Security Workshop@ NeurIPS’23, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Yang%2C+A%2E+Prabhakar%2C+S%2E+Yao%2C+K%2E+Pei%2C+and+K%2E+R%2E+Narasimhan%2E+Language+agents+as+hackers%3A+Evaluating+cybersecurity+skills+with+capture+the+flag%2E+In+Multi%2DAgent+Security+Workshop%40+NeurIPS%E2%80%9923%2C+2023%2E)
61. S. Yao, H. Chen, J. Yang, and K. Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Yao%2C+H%2E+Chen%2C+J%2E+Yang%2C+and+K%2E+Narasimhan%2E+Webshop%3A+Towards+scalable+real%2Dworld+web+interaction+with+grounded+language+agents%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/yao-2022-webshop-towards-scalable-real-world/)
62. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. URL [https://openreview.net/forum?id=WE_vluYUL-X](https://openreview.net/forum?id=WE_vluYUL-X). [↗](https://openreview.net/forum?id=WE_vluYUL-X) [our card](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/)
63. P. Yin, W.-D. Li, K. Xiao, A. Rao, Y. Wen, K. Shi, J. Howland, P. Bailey, M. Catasta, H. Michalewski, A. Polozov, and C. Sutton. Natural language to code generation in interactive data science notebooks, 2022. [find](https://scholar.google.com/scholar?q=P%2E+Yin%2C+W%2E%2DD%2E+Li%2C+K%2E+Xiao%2C+A%2E+Rao%2C+Y%2E+Wen%2C+K%2E+Shi%2C+J%2E+Howland%2C+P%2E+Bailey%2C+M%2E+Catasta%2C+H%2E+Michalewski%2C+A%2E+Polozov%2C+and+C%2E+Sutton%2E+Natural+language+to+code+generation+in+interactive+data+science+noteboo)
64. H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y. Ma, G. Liang, Y. Li, T. Xie, and Q. Wang. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In International Conference on Software Engineering, 2023. URL [https://api.semanticscholar.org/CorpusID:256459413](https://api.semanticscholar.org/CorpusID:256459413). [↗](https://api.semanticscholar.org/CorpusID:256459413)
65. Z. Yu, X. Zhang, N. Shang, Y. Huang, C. Xu, Y. Zhao, W. Hu, and Q. Yin. Wavecoder: Widespread and versatile enhanced instruction tuning with refined data generation. arXiv preprint arXiv:2312.14187, 2023. — [arXiv:2312.14187](https://arxiv.org/abs/2312.14187) [find](https://scholar.google.com/scholar?q=Z%2E+Yu%2C+X%2E+Zhang%2C+N%2E+Shang%2C+Y%2E+Huang%2C+C%2E+Xu%2C+Y%2E+Zhao%2C+W%2E+Hu%2C+and+Q%2E+Yin%2E+Wavecoder%3A+Widespread+and+versatile+enhanced+instruction+tuning+with+refined+data+generation%2E+arXiv+preprint+arXiv%3A2312%2E14187%2C+2)
66. E. Zelikman, Q. Huang, G. Poesia, N. D. Goodman, and N. Haber. Parsel: Algorithmic reasoning with language models by composing decompositions, 2022. URL [https://arxiv.org/](https://arxiv.org/) abs/2212.10561. [↗](https://arxiv.org/)
67. E. Zelikman, E. Lorch, L. Mackey, and A. T. Kalai. Self-taught optimizer (stop): Recursively self-improving code generation, 2024. [find](https://scholar.google.com/scholar?q=E%2E+Zelikman%2C+E%2E+Lorch%2C+L%2E+Mackey%2C+and+A%2E+T%2E+Kalai%2E+Self%2Dtaught+optimizer+%28stop%29%3A+Recursively+self%2Dimproving+code+generation%2C+2024%2E)
68. F. Zhang, B. Chen, Y. Zhang, J. Keung, J. Liu, D. Zan, Y. Mao, J.-G. Lou, and W. Chen. Repocoder: Repository-level code completion through iterative retrieval and generation. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL [https://openreview.net/forum?id=q09vTY1…](https://openreview.net/forum?id=q09vTY1…) [↗](https://openreview.net/forum?id=q09vTY1…)
69. S. Zhang, J. Zhang, J. Liu, L. Song, C. Wang, R. Krishna, and Q. Wu. Training language model agents without modifying language models, 2024. [find](https://scholar.google.com/scholar?q=S%2E+Zhang%2C+J%2E+Zhang%2C+J%2E+Liu%2C+L%2E+Song%2C+C%2E+Wang%2C+R%2E+Krishna%2C+and+Q%2E+Wu%2E+Training+language+model+agents+without+modifying+language+models%2C+2024%2E)
70. A. Zhou, K. Yan, M. Shlapentokh-Rothman, H. Wang, and Y.-X. Wang. Language agent tree search unifies reasoning acting and planning in language models, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Zhou%2C+K%2E+Yan%2C+M%2E+Shlapentokh%2DRothman%2C+H%2E+Wang%2C+and+Y%2E%2DX%2E+Wang%2E+Language+agent+tree+search+unifies+reasoning+acting+and+planning+in+language+models%2C+2023%2E)
71. S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, Y. Bisk, D. Fried, U. Alon, and G. Neubig. Webarena: A realistic web environment for building autonomous agents, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Zhou%2C+F%2E+F%2E+Xu%2C+H%2E+Zhu%2C+X%2E+Zhou%2C+R%2E+Lo%2C+A%2E+Sridhar%2C+X%2E+Cheng%2C+Y%2E+Bisk%2C+D%2E+Fried%2C+U%2E+Alon%2C+and+G%2E+Neubig%2E+Webarena%3A+A+realistic+web+environment+for+building+autonomous+agents%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhou-2023-webarena-realistic-web-environment-building/)