---
title: "Evaluating Agent-based Program Repair at Google (2025)"
description: "The authors build GITS-Eval, a benchmark of 178 fixed bugs from Google's internal issue tracker (GITS). 21 references, cited by 3 papers here."
authors: ["Pat Rondon", "Renyao Wei", "Jose Cambronero", "Jurgen Cito", "Aaron Sun", "Siddhant Sanyam", "Michele Tufano", "Satish Chandra"]
year: 2025
venue: "arXiv:2501.07531"
source: "https://arxiv.org/abs/2501.07531"
doi: "10.48550/arxiv.2501.07531"
arxiv: "2501.07531"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rondon-2025-evaluating-agent-based-program-repair/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rondon-2025-evaluating-agent-based-program-repair/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/15/) / Evaluating Agent-based Program Repair at Google

# Evaluating Agent-based Program Repair at Google

**Authors:** Pat Rondon, Renyao Wei, [Jose Cambronero](https://md.research.elamur.ai/free/en/authors/jose-cambronero/), Jurgen Cito, Aaron Sun, Siddhant Sanyam, [Michele Tufano](https://md.research.elamur.ai/free/en/authors/michele-tufano/) and [Satish Chandra](https://md.research.elamur.ai/free/en/authors/satish-chandra/) · **Year:** 2025 · **Venue:** arXiv:2501.07531 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2501.07531) · [PDF](https://arxiv.org/pdf/2501.07531) · [DOI 10.48550/arxiv.2501.07531](https://doi.org/10.48550/arxiv.2501.07531) · [arXiv 2501.07531](https://arxiv.org/abs/2501.07531) · [OpenAlex](https://openalex.org/W6966995686)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/rondon-2025-evaluating-agent-based-program-repair/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLM-based repair agents have mostly been built and tested on open-source GitHub Python bugs, especially SWE-Bench. It is unclear whether they work inside a large company. The authors build GITS-Eval, a benchmark of 178 fixed bugs from Google's internal issue tracker (GITS). Developers filed 78 of them, and automated tools filed 100: sanitizers (SAN) and a test-order-dependency analyzer (TOD). A multi-phase filtering funnel keeps only bugs with small patches, no multimedia, and reliable, non-flaky tests that expose each bug. It also drops fixes that need "magic constants" the agent could not infer. A separate sample of 2,000 GITS bugs differs from SWE-Bench in language mix, patch size and spread, and how often code terms appear in reports. As a baseline, the authors implement Passerine, a SWE-Agent-style agent adapted to Google's monorepo and tooling. It runs Gemini 1.5 Pro with 20 trajectories per bug. Passerine produces test-passing patches for most machine-reported bugs and about a quarter of human-reported ones. Manual review finds fewer patches equivalent to the real fix. A simple agent already fixes real enterprise bugs, but SWE-Bench scores may not transfer.

## Key points
- GITS-Eval has 178 bugs: 78 human-reported, 50 SAN and 50 TOD. The fix patches date from June 26 to August 30, 2024, and filtering began with about 80k fixed bugs. Patches are limited to under 150 lines.
- Plausible-patch rates (pass bug tests) are 73% for machine-reported bugs (68% TOD, 78% SAN) and 25.6% for human-reported bugs.
- After manual annotation, at least one patch matched the ground-truth fix semantically for 43% of machine-reported bugs (24% TOD, 62% SAN) and 17.9% of human-reported ones.
- Passerine changes its command patterns depending on bug type and benefits from detailed bug reports. This suggests bug reports could be designed with agents in mind.
- Analyzing the agent's runs shows room for efficiency gains, for example by cutting off degenerate trajectories early. The authors make no claims about performance on SWE-Bench or SWE-Bench-Lite.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,888 works · [💻 Software](https://md.research.elamur.ai/free/en/shelves/computer-science/software/) · 150 works · [🌐 Computer Networks and Communications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-networks-and-communications/) · 115 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,394 works · [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 194 works

## Cite

Cite as: Pat Rondon, Renyao Wei, Jose Cambronero, Jurgen Cito, Aaron Sun, Siddhant Sanyam, Michele Tufano and Satish Chandra. 2025. Evaluating Agent-based Program Repair at Google. arXiv:2501.07531.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{rondon2025evaluating,
  title = {{Evaluating Agent-based Program Repair at Google}},
  author = {Pat Rondon, Renyao Wei, Jose Cambronero, Jurgen Cito, Aaron Sun, Siddhant Sanyam, Michele Tufano and Satish Chandra},
  year = {2025},
  doi = {10.48550/arxiv.2501.07531},
  eprint = {2501.07531},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.07531}
}
```

- [← previous Enhanced Arabic-language cyberbullying detection: deep embedding and transformer (BERT) approaches](https://md.research.elamur.ai/free/en/papers/aljohani-2025-enhanced-arabic-language-cyberbullying-detection/)
- [next → Evaluating CxG Generalisation in LLMs via Construction-Based NLI Fine Tuning](https://md.research.elamur.ai/free/en/papers/mackintosh-2025-evaluating-cxg-generalisation-llms-construction/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/15/)

## References

21 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “[Swe-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/)” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [Online]. Avail…
2. J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “[Swe-agent: Agent-computer interfaces enable automated software engineering](https://md.research.elamur.ai/free/en/papers/yang-2024-swe-agent-agent-computer-interfaces/),” arXiv preprint arXiv:2405.15793, 2024.
3. Y. Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 1592–1604. [find](https://scholar.google.com/scholar?q=Y%2E+Zhang%2C+H%2E+Ruan%2C+Z%2E+Fan%2C+and+A%2E+Roychoudhury%2C+%E2%80%9CAutocoderover%3A+Autonomous+program+improvement%2C%E2%80%9D+in+Proceedings+of+the+33rd+ACM+SIGSOFT+International+Symposium+on+Software+Testing+and+Analysis%2C+2024%2C+)
4. I. Bouzenia, P. Devanbu, and M. Pradel, “Repairagent: An autonomous, llm-based agent for program repair,” arXiv preprint arXiv:2403.17134, 2024. — [arXiv:2403.17134](https://arxiv.org/abs/2403.17134) [find](https://scholar.google.com/scholar?q=I%2E+Bouzenia%2C+P%2E+Devanbu%2C+and+M%2E+Pradel%2C+%E2%80%9CRepairagent%3A+An+autonomous%2C+llm%2Dbased+agent+for+program+repair%2C%E2%80%9D+arXiv+preprint+arXiv%3A2403%2E17134%2C+2024%2E+%E2%80%94+%5BarXiv%3A2403%2E17134%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2403%2E17134%29)
5. D. Chen, S. Lin, M. Zeng, D. Zan, J.-G. Wang, A. Cheshkov, J. Sun, H. Yu, G. Dong, A. Aliev et al., “[Coder: Issue resolving with multi-agent and task graphs](https://md.research.elamur.ai/free/en/papers/chen-2024-coder-issue-resolving-multi-agent/),” arXiv preprint arXiv:2406.01304, 2024.
6. M. Tufano, A. Agarwal, J. Jang, R. Z. Moghaddam, and N. Sundaresan, “Autodev: Automated ai-driven development,” arXiv preprint arXiv:2403.08299, 2024. — [arXiv:2403.08299](https://arxiv.org/abs/2403.08299) [find](https://scholar.google.com/scholar?q=M%2E+Tufano%2C+A%2E+Agarwal%2C+J%2E+Jang%2C+R%2E+Z%2E+Moghaddam%2C+and+N%2E+Sundaresan%2C+%E2%80%9CAutodev%3A+Automated+ai%2Ddriven+development%2C%E2%80%9D+arXiv+preprint+arXiv%3A2403%2E08299%2C+2024%2E+%E2%80%94+%5BarXiv%3A2403%2E08299%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2403%2E0)
7. X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh et al., “Opendevin: An open platform for ai software developers as generalist agents,” arXiv preprint arXiv:2407.[16741, 2024](https://md.research.elamur.ai/free/en/papers/wang-2024-openhands-open-platform-ai/).
8. H. Ruan, Y. Zhang, and A. Roychoudhury, “Specrover: Code intent extraction via llms,” arXiv preprint arXiv:2408.02232, 2024. — [arXiv:2408.02232](https://arxiv.org/abs/2408.02232) [find](https://scholar.google.com/scholar?q=H%2E+Ruan%2C+Y%2E+Zhang%2C+and+A%2E+Roychoudhury%2C+%E2%80%9CSpecrover%3A+Code+intent+extraction+via+llms%2C%E2%80%9D+arXiv+preprint+arXiv%3A2408%2E02232%2C+2024%2E+%E2%80%94+%5BarXiv%3A2408%2E02232%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2408%2E02232%29)
9. Y. Liu, P. Gao, X. Wang, C. Peng, and Z. Zhang, “Marscode agent: Ai-native automated bug fixing,” arXiv preprint arXiv:2409.00899, 2024. — [arXiv:2409.00899](https://arxiv.org/abs/2409.00899) [find](https://scholar.google.com/scholar?q=Y%2E+Liu%2C+P%2E+Gao%2C+X%2E+Wang%2C+C%2E+Peng%2C+and+Z%2E+Zhang%2C+%E2%80%9CMarscode+agent%3A+Ai%2Dnative+automated+bug+fixing%2C%E2%80%9D+arXiv+preprint+arXiv%3A2409%2E00899%2C+2024%2E+%E2%80%94+%5BarXiv%3A2409%2E00899%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2409%2E00899%29)
10. R. Potvin and J. Levenberg, “Why google stores billions of lines of code in a single repository,” Communications of the ACM, vol. 59, no. 7, pp. 78–87, 2016. [find](https://scholar.google.com/scholar?q=R%2E+Potvin+and+J%2E+Levenberg%2C+%E2%80%9CWhy+google+stores+billions+of+lines+of+code+in+a+single+repository%2C%E2%80%9D+Communications+of+the+ACM%2C+vol%2E+59%2C+no%2E+7%2C+pp%2E+78%E2%80%9387%2C+2016%2E)
11. Bazel, “Distributed Builds,” [https://bazel.build/basics/distributed-builds](https://bazel.build/basics/distributed-builds), 2024, [Online; accessed October 2024]. [↗](https://bazel.build/basics/distributed-builds)
12. C. Sadowski, K. T. Stolee, and S. Elbaum, “How developers search for code: a case study,” in Proceedings of the 2015 10th joint meeting on foundations of software engineering, 2015, pp. 191–201. [find](https://scholar.google.com/scholar?q=C%2E+Sadowski%2C+K%2E+T%2E+Stolee%2C+and+S%2E+Elbaum%2C+%E2%80%9CHow+developers+search+for+code%3A+a+case+study%2C%E2%80%9D+in+Proceedings+of+the+2015+10th+joint+meeting+on+foundations+of+software+engineering%2C+2015%2C+pp%2E+191%E2%80%93201%2E)
13. S. Saha et al., “Harnessing evolution for multi-hunk program repair,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 2019, pp. 13–24. [find](https://scholar.google.com/scholar?q=S%2E+Saha+et+al%2E%2C+%E2%80%9CHarnessing+evolution+for+multi%2Dhunk+program+repair%2C%E2%80%9D+in+2019+IEEE%2FACM+41st+International+Conference+on+Software+Engineering+%28ICSE%29%2E+IEEE%2C+2019%2C+pp%2E+13%E2%80%9324%2E)
14. V. Sobreira, T. Durieux, F. Madeiral, M. Monperrus, and M. de Almeida Maia, “Dissection of a bug dataset: Anatomy of 395 patches from defects4j,” in 2018 IEEE 25th international conference on software analysis, evolution and reengineering (SANER). IEEE, 2018, pp. 130–140. [find](https://scholar.google.com/scholar?q=V%2E+Sobreira%2C+T%2E+Durieux%2C+F%2E+Madeiral%2C+M%2E+Monperrus%2C+and+M%2E+de+Almeida+Maia%2C+%E2%80%9CDissection+of+a+bug+dataset%3A+Anatomy+of+395+patches+from+defects4j%2C%E2%80%9D+in+2018+IEEE+25th+international+conference+on+software)
15. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao, “[React: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/),” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. [Online]. Available: https://op…
16. X. Wang, Y. Chen, L. Yuan, Y. Zhang, Y. Li, H. Peng, and H. Ji, “[Executable code actions elicit better llm agents](https://md.research.elamur.ai/free/en/papers/wang-2024-executable-code-actions-elicit-better/),” in Proceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2025.
17. A. Greenberg, “Building a distributed build system at google scale,” [https://gotocon.com/dl/goto-chicago-2016/slides/AysyluGreenberg](https://gotocon.com/dl/goto-chicago-2016/slides/AysyluGreenberg) BuildingADistributedBuildSystemAtGoogleScale.pdf, 2016, [Online; accessed October 2024]. [↗](https://gotocon.com/dl/goto-chicago-2016/slides/AysyluGreenberg)
18. K. Wang, D. Rall, G. Tener, V. Gullapalli, X. Huang, and A. Gad, “Smart build targets batching service at Google,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 2021, pp. 160–169. [find](https://scholar.google.com/scholar?q=K%2E+Wang%2C+D%2E+Rall%2C+G%2E+Tener%2C+V%2E+Gullapalli%2C+X%2E+Huang%2C+and+A%2E+Gad%2C+%E2%80%9CSmart+build+targets+batching+service+at+Google%2C%E2%80%9D+in+2021+IEEE%2FACM+43rd+International+Conference+on+Software+Engineering%3A+Software+Engi)
19. Z. Qi, F. Long, S. Achour, and M. Rinard, “An analysis of patch plausibility and correctness for generate-and-validate patch generation systems,” in Proceedings of the 2015 International Symposium on Software Testing and Analysis, 2015, pp. 24–36. [find](https://scholar.google.com/scholar?q=Z%2E+Qi%2C+F%2E+Long%2C+S%2E+Achour%2C+and+M%2E+Rinard%2C+%E2%80%9CAn+analysis+of+patch+plausibility+and+correctness+for+generate%2Dand%2Dvalidate+patch+generation+systems%2C%E2%80%9D+in+Proceedings+of+the+2015+International+Symposium+on+)
20. Y. Xiong, X. Liu, M. Zeng, L. Zhang, and G. Huang, “Identifying patch correctness in test-based program repair,” in Proceedings of the 40th international conference on software engineering, 2018, pp. 789–799. [find](https://scholar.google.com/scholar?q=Y%2E+Xiong%2C+X%2E+Liu%2C+M%2E+Zeng%2C+L%2E+Zhang%2C+and+G%2E+Huang%2C+%E2%80%9CIdentifying+patch+correctness+in+test%2Dbased+program+repair%2C%E2%80%9D+in+Proceedings+of+the+40th+international+conference+on+software+engineering%2C+2018%2C+pp%2E+)
21. B. Yang and J. Yang, “Exploring the differences between plausible and correct patches at fine-grained level,” in 2020 IEEE 2nd International Workshop on Intelligent Bug Fixing (IBF). IEEE, 2020, pp. 1–8. [find](https://scholar.google.com/scholar?q=B%2E+Yang+and+J%2E+Yang%2C+%E2%80%9CExploring+the+differences+between+plausible+and+correct+patches+at+fine%2Dgrained+level%2C%E2%80%9D+in+2020+IEEE+2nd+International+Workshop+on+Intelligent+Bug+Fixing+%28IBF%29%2E+IEEE%2C+2020%2C+pp%2E+1)

*Updated 3 November 2026*