---
title: "Self-Edit: Fault-Aware Code Editor for Code Generation (2023)"
description: "The authors propose Self-Edit, a generate-and-edit pipeline modeled on how human programmers… 22 references, cited by 50 papers here."
authors: ["Kechi Zhang", "Zhuo Li", "Jia Li", "Ge Li", "Zhi Jin"]
year: 2023
venue: "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2023.acl-long.45/"
doi: "10.18653/v1/2023.acl-long.45"
arxiv: "2305.04087"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2023-self-edit-fault-aware-code/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2023-self-edit-fault-aware-code/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / Self-Edit: Fault-Aware Code Editor for Code Generation

# Self-Edit: Fault-Aware Code Editor for Code Generation

**Authors:** [Kechi Zhang](https://md.research.elamur.ai/free/en/authors/kechi-zhang/), [Zhuo Li](https://md.research.elamur.ai/free/en/authors/zhuo-li/), [Jia Li](https://md.research.elamur.ai/free/en/authors/jia-li/), [Ge Li](https://md.research.elamur.ai/free/en/authors/ge-li/) and [Zhi Jin](https://md.research.elamur.ai/free/en/authors/zhi-jin/) · **Year:** 2023 · **Venue:** [Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2023](https://md.research.elamur.ai/free/en/venues/acl/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.acl-long.45/) · [PDF](https://aclanthology.org/2023.acl-long.45.pdf) · [DOI 10.18653/v1/2023.acl-long.45](https://doi.org/10.18653/v1/2023.acl-long.45) · [arXiv 2305.04087](https://arxiv.org/abs/2305.04087) · [OpenAlex](https://openalex.org/W4385572345)

**Citations:** 55 citations (OpenAlex, 2026-10-14) · [50](https://md.research.elamur.ai/free/en/papers/zhang-2023-self-edit-fault-aware-code/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models still solve few competitive programming problems when only one submission per task is allowed. For example, GPT3 reaches roughly 7% pass@1 on APPS-test. The authors propose Self-Edit, a generate-and-edit pipeline modeled on how human programmers debug.

First, a frozen LLM, used as a black box, writes a program from the problem statement. The program is then run on the example test case that comes with the task. The outcome (passed, wrong answer, or an error such as a syntax fault, runtime exception or timeout) is converted into a templated comment. This comment carries details like the failing line, the error message, or expected versus actual output. A comparatively small fault-aware neural editor then takes the problem, the draft code and this comment, and outputs a revised program.

The method was evaluated on APPS and HumanEval with nine LLMs ranging from 110M to 175B parameters. It markedly raises average pass@1 and outperforms reranking-based post-processing while keeping a small, fixed sample budget. This matters because it improves generated code without retraining the LLM or drawing large numbers of candidate samples.

## Key points
- Relative gains in average pass@1 over direct generation are 89% on APPS-dev, 31% on APPS-test and 48% on HumanEval. The HumanEval result is out-of-distribution, which suggests the editor transfers to a different benchmark style.
- Even for the largest model, GPT3-175B (text-davinci-002), the editor lifts APPS-dev pass@1 from 26.6% to 32.4%.
- Execution results fall into three categories, each with its own comment template. On APPS-train, small models such as PyCodeGPT-110M produce SyntaxErrors more often, while GPT3 yields more programs that run normally.
- Prior reranking approaches need heavy sampling (AlphaCode uses about 1 million samples per problem). Examples include the fault-aware neural ranker of Inala et al. (2022), CodeT and AlphaCode-style filtering. The authors present Self-Edit as the first editing-based post-processing method for competitive programming.
- The authors list several limitations:
  - The editor uses small pretrained models.
  - The editor training set is built from only 10 LLM samples per problem.
  - No strict compute comparison with other methods is provided.
  - Multi-round editing suffers from a train/test mismatch.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,267 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [💻 Software](https://md.research.elamur.ai/free/en/shelves/computer-science/software/) · 119 works · [🖥️ Computer Science Applications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-science-applications/) · 131 works

🧭 Topics (1): [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 168 works

## Cite

Cite as: Kechi Zhang, Zhuo Li, Jia Li, Ge Li and Zhi Jin. 2023. Self-Edit: Fault-Aware Code Editor for Code Generation. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2023self,
  title = {{Self-Edit: Fault-Aware Code Editor for Code Generation}},
  author = {Kechi Zhang, Zhuo Li, Jia Li, Ge Li and Zhi Jin},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.45},
  eprint = {2305.04087},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.acl-long.45}
}
```

- [← previous LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions](https://md.research.elamur.ai/free/en/papers/wu-2023-lamini-lm-diverse-herd-distilled/)
- [next → Statistical Rejection Sampling Improves Preference Optimization](https://md.research.elamur.ai/free/en/papers/liu-2023-statistical-rejection-sampling-improves-preference/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

22 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. [Mapping language to code in programmatic context](https://md.research.elamur.ai/free/en/papers/iyer-2018-mapping-language-code-programmatic-context/). arXiv preprint arXiv:1808.09588.
2. Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. 2019. [Spoc: Search-based pseudocode to code](https://md.research.elamur.ai/free/en/papers/kulal-2019-spoc-search-based-pseudocode-code/). Advances in Neural Information Processing Systems, 32.
3. Shuvendu K. Lahiri, Aaditya Naik, Georgios Sakkas, Piali Choudhury, Curtis von Veh, Madanlal Musuvathi, Jeevana Priya Inala, Chenglong Wang, and Jianfeng Gao. 2022. Interactive code generation via test-driven user-intent formalization. CoRR, abs/2208.05950. — [arXiv:2208.05950](https://arxiv.org/abs/2208.05950) [find](https://scholar.google.com/scholar?q=Shuvendu+K%2E+Lahiri%2C+Aaditya+Naik%2C+Georgios+Sakkas%2C+Piali+Choudhury%2C+Curtis+von+Veh%2C+Madanlal+Musuvathi%2C+Jeevana+Priya+Inala%2C+Chenglong+Wang%2C+and+Jianfeng+Gao%2E+2022%2E+Interactive+code+generation+via+tes)
4. Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu-Hong Hoi. 2022. [Coderl: Mastering code generation through pretrained models and deep reinforcement learning](https://md.research.elamur.ai/free/en/papers/le-2022-coderl-mastering-code-generation-through/). In NeurIPS.
5. Jia Li, Ge Li, Yongmin Li, and Zhi Jin. 2023a. Enabling programming thinking in large language models toward code generation. arXiv preprint arXiv:2305.06599. — [arXiv:2305.06599](https://arxiv.org/abs/2305.06599) [find](https://scholar.google.com/scholar?q=Jia+Li%2C+Ge+Li%2C+Yongmin+Li%2C+and+Zhi+Jin%2E+2023a%2E+Enabling+programming+thinking+in+large+language+models+toward+code+generation%2E+arXiv+preprint+arXiv%3A2305%2E06599%2E+%E2%80%94+%5BarXiv%3A2305%2E06599%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
6. Jia Li, Ge Li, Zhuo Li, Zhi Jin, Xing Hu, Kechi Zhang, and Zhiyi Fu. 2022a. Codeeditor: Learning to edit source code with pre-trained models. arXiv preprint arXiv:2210.17040. — [arXiv:2210.17040](https://arxiv.org/abs/2210.17040) [find](https://scholar.google.com/scholar?q=Jia+Li%2C+Ge+Li%2C+Zhuo+Li%2C+Zhi+Jin%2C+Xing+Hu%2C+Kechi+Zhang%2C+and+Zhiyi+Fu%2E+2022a%2E+Codeeditor%3A+Learning+to+edit+source+code+with+pre%2Dtrained+models%2E+arXiv+preprint+arXiv%3A2210%2E17040%2E+%E2%80%94+%5BarXiv%3A2210%2E17040%5D%28http)
7. Jia Li, Yongmin Li, Ge Li, Zhi Jin, Yiyang Hao, and Xing Hu. 2023b. Skcoder: A sketch-based approach for automatic code generation. arXiv preprint arXiv:2302.06144. — [arXiv:2302.06144](https://arxiv.org/abs/2302.06144) [find](https://scholar.google.com/scholar?q=Jia+Li%2C+Yongmin+Li%2C+Ge+Li%2C+Zhi+Jin%2C+Yiyang+Hao%2C+and+Xing+Hu%2E+2023b%2E+Skcoder%3A+A+sketch%2Dbased+approach+for+automatic+code+generation%2E+arXiv+preprint+arXiv%3A2302%2E06144%2E+%E2%80%94+%5BarXiv%3A2302%2E06144%5D%28https%3A%2F%2Farxiv%2E)
8. Jia Li, Chongyang Tao, Huang Hu, Can Xu, Yining Chen, and Daxin Jiang. 2022b. Unsupervised cross-domain adaptation for response selection using self-supervised and adversarial training. In WSDM ’22: The Fifteenth ACM International Conference on Web Search and Data Mining, Virtual Event / Tempe, AZ,… [find](https://scholar.google.com/scholar?q=Jia+Li%2C+Chongyang+Tao%2C+Huang+Hu%2C+Can+Xu%2C+Yining+Chen%2C+and+Daxin+Jiang%2E+2022b%2E+Unsupervised+cross%2Ddomain+adaptation+for+response+selection+using+self%2Dsupervised+and+adversarial+training%2E+In+WSDM+%E2%80%9922%3A+T)
9. Jia Li, Yunfei Zhao, Yongmin Li, Ge Li, and Zhi Jin. 2023c. Towards enhancing in-context learning for code generation. arXiv preprint arXiv:2303.17780. — [arXiv:2303.17780](https://arxiv.org/abs/2303.17780) [find](https://scholar.google.com/scholar?q=Jia+Li%2C+Yunfei+Zhao%2C+Yongmin+Li%2C+Ge+Li%2C+and+Zhi+Jin%2E+2023c%2E+Towards+enhancing+in%2Dcontext+learning+for+code+generation%2E+arXiv+preprint+arXiv%3A2303%2E17780%2E+%E2%80%94+%5BarXiv%3A2303%2E17780%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2303%2E)
10. Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, Jame… — [Competition-Level Code Generation with AlphaCode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/)
11. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. A conversational paradigm for program synthesis. CoRR, abs/2203.13474. [find](https://scholar.google.com/scholar?q=Erik+Nijkamp%2C+Bo+Pang%2C+Hiroaki+Hayashi%2C+Lifu+Tu%2C+Huan+Wang%2C+Yingbo+Zhou%2C+Silvio+Savarese%2C+and+Caiming+Xiong%2E+2022%2E+A+conversational+paradigm+for+program+synthesis%2E+CoRR%2C+abs%2F2203%2E13474%2E)
12. Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2015. Learning to generate pseudo-code from source code using statistical machine translation. In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 57… [find](https://scholar.google.com/scholar?q=Yusuke+Oda%2C+Hiroyuki+Fudaba%2C+Graham+Neubig%2C+Hideaki+Hata%2C+Sakriani+Sakti%2C+Tomoki+Toda%2C+and+Satoshi+Nakamura%2E+2015%2E+Learning+to+generate+pseudo%2Dcode+from+source+code+using+statistical+machine+translati)
13. Richard Yuanzhe Pang and He He. 2021. Text generation by learning from demonstrations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. [find](https://scholar.google.com/scholar?q=Richard+Yuanzhe+Pang+and+He+He%2E+2021%2E+Text+generation+by+learning+from+demonstrations%2E+In+9th+International+Conference+on+Learning+Representations%2C+ICLR+2021%2C+Virtual+Event%2C+Austria%2C+May+3%2D7%2C+2021%2E+Op)
14. Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. 2021. Generate & rank: A multi-task framework for math word problems. In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pa… [find](https://scholar.google.com/scholar?q=Jianhao+Shen%2C+Yichun+Yin%2C+Lin+Li%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Ming+Zhang%2C+and+Qun+Liu%2E+2021%2E+Generate+%26+rank%3A+A+multi%2Dtask+framework+for+math+word+problems%2E+In+Findings+of+the+Association+for+Computation)
15. Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I. Wang. 2022. [Natural language to code translation with execution](https://md.research.elamur.ai/free/en/papers/shi-2022-natural-language-code-translation-execution/). CoRR, abs/2204.11454.
16. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [https://github.com/kingoflolz/](https://github.com/kingoflolz/) mesh-transformer-jax. [↗](https://github.com/kingoflolz/)
17. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
18. Frank F. Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022. [A systematic evaluation of large language models of code](https://md.research.elamur.ai/free/en/papers/xu-2022-systematic-evaluation-large-language-models/). In MAPS@PLDI 2022: 6th ACM SIGPLAN International Symposium on Machine Programming, San Diego, CA, USA, 13 June 2022, pages 1–10. ACM.
19. Pengcheng Yin and Graham Neubig. 2018. Tranx: A transition-based neural abstract syntax parser for semantic parsing and code generation. arXiv preprint arXiv:1810.02720. [find](https://scholar.google.com/scholar?q=Pengcheng+Yin+and+Graham+Neubig%2E+2018%2E+Tranx%3A+A+transition%2Dbased+neural+abstract+syntax+parser+for+semantic+parsing+and+code+generation%2E+arXiv+preprint+arXiv%3A1810%2E02720%2E)
20. Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, and Jian-Guang Lou. 2022. CERT: Continual pre-training on sketches for library-oriented code generation. In The 2022 International Joint Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Daoguang+Zan%2C+Bei+Chen%2C+Dejian+Yang%2C+Zeqi+Lin%2C+Minsu+Kim%2C+Bei+Guan%2C+Yongji+Wang%2C+Weizhu+Chen%2C+and+Jian%2DGuang+Lou%2E+2022%2E+CERT%3A+Continual+pre%2Dtraining+on+sketches+for+library%2Doriented+code+generation%2E+I)
21. Kechi Zhang, Ge Li, Jia Li, Zhuo Li, and Zhi Jin. 2023. Toolcoder: Teach code generation models to use api search tools. ArXiv, abs/2305.04032. — [arXiv:2305.04032](https://arxiv.org/abs/2305.04032) [find](https://scholar.google.com/scholar?q=Kechi+Zhang%2C+Ge+Li%2C+Jia+Li%2C+Zhuo+Li%2C+and+Zhi+Jin%2E+2023%2E+Toolcoder%3A+Teach+code+generation+models+to+use+api+search+tools%2E+ArXiv%2C+abs%2F2305%2E04032%2E+%E2%80%94+%5BarXiv%3A2305%2E04032%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E04032%29)
22. Tianyi Zhang, Tao Yu, Tatsunori B. Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried, and Sida I. Wang. 2022. Coder reviewer reranking for code generation. CoRR, abs/2211.16490. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Tao+Yu%2C+Tatsunori+B%2E+Hashimoto%2C+Mike+Lewis%2C+Wen%2Dtau+Yih%2C+Daniel+Fried%2C+and+Sida+I%2E+Wang%2E+2022%2E+Coder+reviewer+reranking+for+code+generation%2E+CoRR%2C+abs%2F2211%2E16490%2E)

*Updated 2 November 2026*