--- title: "Training Software Engineering Agents and Verifiers with SWE-Gym" description: "The authors introduce SWE-Gym, which provides 2,438 real GitHub-issue tasks from 11 Python… 3 references, cited by 71 papers here." authors: ["Jiayi Pan; Xingyao Wang; Graham Neubig; Navdeep Jaitly; Heng Ji; Alane Suhr;", "Yizhe Zhang"] year: 2024 venue: "arXiv:2412.21139" source: "https://arxiv.org/abs/2412.21139" doi: "10.48550/arxiv.2412.21139" arxiv: "2412.21139" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/pan-2024-training-software-engineering-agents/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pan-2024-training-software-engineering-agents/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / Training Software Engineering Agents and Verifiers with SWE-Gym # Training Software Engineering Agents and Verifiers with SWE-Gym **Authors:** Jiayi Pan; Xingyao Wang; Graham Neubig; Navdeep Jaitly; Heng Ji; Alane Suhr; and [Yizhe Zhang](https://md.research.elamur.ai/free/en/authors/yizhe-zhang/) · **Year:** 2024 · **Venue:** arXiv:2412.21139 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2412.21139) · [PDF](https://arxiv.org/pdf/2412.21139) · [DOI 10.48550/arxiv.2412.21139](https://doi.org/10.48550/arxiv.2412.21139) · [arXiv 2412.21139](https://arxiv.org/abs/2412.21139) · [OpenAlex](https://openalex.org/W4405957747) **Citations:** 2 citations (OpenAlex, 2026-10-14) · [71](https://md.research.elamur.ai/free/en/papers/pan-2024-training-software-engineering-agents/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Open-weight software engineering (SWE) agents have lagged behind systems built on proprietary models, partly because no environment existed for training them on realistic repository-level work. The authors introduce SWE-Gym, which provides 2,438 real GitHub-issue tasks from 11 Python repositories. Each task includes an executable runtime with dependencies already installed, unit tests, and a natural-language problem statement. The repositories are disjoint from SWE-Bench to avoid contamination. The tests provide reward signals. The authors use these to fine-tune Qwen-2.5 coder models inside two scaffolds: OpenHands, by learning from successful sampled trajectories, and MoatlessTools, through self-improvement via rejection sampling. They also train verifiers, which are outcome-supervised reward models that rank candidate trajectories at inference time. Fine-tuning improves resolution rates by up to 19% absolute. Combining the agents with verifiers reaches 32.0% on SWE-Bench Verified and 26.0% on SWE-Bench Lite, a new best among open-weight agents. Performance keeps rising with more training trajectories and more rollouts, which suggests that sampling compute, not dataset size, is the current limit. The environment, models and trajectories are released publicly. ## Key points - **Released resources:** - SWE-Gym: 2,438 tasks. - SWE-Gym Lite: 230 easier, self-contained tasks for quick prototyping. - SWE-Gym Raw: 64,689 issues from 358 repositories, without executable environments. - **OpenHands result:** A 32B Qwen-2.5 coder fine-tuned on only 491 trajectories improves to 15.3% on SWE-Bench Lite (+12.3) and 20.6% on SWE-Bench Verified (+13.6). - **MoatlessTools result:** Self-improvement through rejection sampling fine-tuning reaches up to 19.7% on SWE-Bench Lite. - **Verifier scaling:** Selecting the best of several rollouts gives roughly log-linear gains. In the MoatlessTools setting with k = 8, a 32B verifier lifts a 32B policy from 19.7% to 26.3% on Lite, while a 7B verifier stops improving after k = 4. - **Data scaling:** Experiments that vary the number of trajectories, unique instances and repositories show no saturation, indicating that task and repository diversity are not yet bottlenecks. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [💻 Software](https://md.research.elamur.ai/free/en/shelves/computer-science/software/) · 119 works 🧭 Topics (1): [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 168 works ## Cite Cite as: Jiayi Pan; Xingyao Wang; Graham Neubig; Navdeep Jaitly; Heng Ji; Alane Suhr; and Yizhe Zhang. 2024. Training Software Engineering Agents and Verifiers with SWE-Gym. arXiv:2412.21139. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{suhr2024training, title = {{Training Software Engineering Agents and Verifiers with SWE-Gym}}, author = {Jiayi Pan; Xingyao Wang; Graham Neubig; Navdeep Jaitly; Heng Ji; Alane Suhr; and Yizhe Zhang}, year = {2024}, doi = {10.48550/arxiv.2412.21139}, eprint = {2412.21139}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2412.21139} } ``` - [← previous Repetition Improves Language Model Embeddings](https://md.research.elamur.ai/free/en/papers/springer-2024-repetition-improves-language-model-embeddings/) - [next → A Primer on the Inner Workings of Transformer-based Language Models](https://md.research.elamur.ai/free/en/papers/ferrando-2024-primer-inner-workings-transformer-based/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) ## References 3 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., et al. Solving rubik’s cube with a robot hand. arXiv preprint arXiv:1910.07113, 2019. 2. Self-Improvement with Reinforcement Learning Despite notable progress, our self-impr… [find](https://scholar.google.com/scholar?q=Akkaya%2C+I%2E%2C+Andrychowicz%2C+M%2E%2C+Chociej%2C+M%2E%2C+Litwin%2C+M%2E%2C+McGrew%2C+B%2E%2C+Petron%2C+A%2E%2C+Paino%2C+A%2E%2C+Plappert%2C+M%2E%2C+Powell%2C+G%2E%2C+Ribas%2C+R%2E%2C+et+al%2E+Solving+rubik%E2%80%99s+cube+with+a+robot+hand%2E+arXiv+preprint+arXiv%3A1910%2E) 2. Badertdinov, I., Trofimova, M., Anapolskiy, Y., Abramov, S., Zainullina, K., Golubev, A., Polezhaev, S., Litvintseva, D., Karasik, S., Fisin, F., Skvortsov, S., Nekrashevich, M., Shevtsov, A., and Yangel, B. Scaling data collection for training software engineering agents. Nebius blog, 2024. 3. Huma… [find](https://scholar.google.com/scholar?q=Badertdinov%2C+I%2E%2C+Trofimova%2C+M%2E%2C+Anapolskiy%2C+Y%2E%2C+Abramov%2C+S%2E%2C+Zainullina%2C+K%2E%2C+Golubev%2C+A%2E%2C+Polezhaev%2C+S%2E%2C+Litvintseva%2C+D%2E%2C+Karasik%2C+S%2E%2C+Fisin%2C+F%2E%2C+Skvortsov%2C+S%2E%2C+Nekrashevich%2C+M%2E%2C+Shevtsov%2C+A%2E%2C+and+Yan) 3. Bai, H., Zhou, Y., Cemri, M., Pan, J., Suhr, A., Levine, S., and Kumar, A. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. ArXiv, abs/2406.11896, 2024. URL [https://api.semanticscholar.org/CorpusID:270562229](https://api.semanticscholar.org/CorpusID:270562229). — [arXiv:2406.11896](https://arxiv.org/abs/2406.11896) [↗](https://api.semanticscholar.org/CorpusID:270562229) *Updated 2 November 2026*