---
title: "Language Models Can Teach Themselves to Program Better"
description: "Code-generating language models usually learn from human-written code, which contains bugs and… 33 references, cited by 38 papers here."
authors: ["Patrick Haluptzok", "Matthew Bowers", "Adam Tauman Kalai"]
year: 2023
venue: "arXiv:2207.14502"
source: "https://arxiv.org/abs/2207.14502"
doi: "10.48550/arxiv.2207.14502"
arxiv: "2207.14502"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "https://arxiv.org/licenses/nonexclusive-distrib/1.0/license.html"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/haluptzok-2023-language-models-can-teach-themselves/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/haluptzok-2023-language-models-can-teach-themselves/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/) / Language Models Can Teach Themselves to Program Better

# Language Models Can Teach Themselves to Program Better

**Authors:** Patrick Haluptzok, Matthew Bowers and [Adam Tauman Kalai](https://md.research.elamur.ai/free/en/authors/adam-tauman-kalai/) · **Year:** 2023 · **Venue:** arXiv:2207.14502 · **Licence:** [arXiv — non-exclusive licence to distribute](https://arxiv.org/licenses/nonexclusive-distrib/1.0/license.html)

**Links:** [Publisher's page](https://arxiv.org/abs/2207.14502) · [PDF](https://arxiv.org/pdf/2207.14502) · [DOI 10.48550/arxiv.2207.14502](https://doi.org/10.48550/arxiv.2207.14502) · [arXiv 2207.14502](https://arxiv.org/abs/2207.14502) · [OpenAlex](https://openalex.org/W4289445724)

**Citations:** 16 citations (OpenAlex, 2026-10-14) · [38](https://md.research.elamur.ai/free/en/papers/haluptzok-2023-language-models-can-teach-themselves/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Code-generating language models usually learn from human-written code, which contains bugs and often has vague English specifications. Inspired by self-play in systems like AlphaZero, the authors ask whether a model can write its own practice problems and improve by training on them. They use programming puzzles from the P3 dataset (Python Programming Puzzles). In a puzzle, a verifier function f defines the task, and a candidate solution g passes only if executing f on its output returns True. This makes correctness checkable by running code.

The pipeline has four steps. The LM is prompted with unsolved training puzzles and generates new, similar ones. It then attempts solutions to these puzzles. A Python interpreter keeps only solutions that are correct and finish within a time limit. Finally, the model is fine-tuned on the verified pairs. No human-written solutions are used for training, apart from the few-shot prompt examples.

On held-out human-authored test puzzles, GPT-Neo models more than doubled their accuracy. This suggests code LMs paired with an interpreter can create useful training data for themselves.

## Key points
- Generation starts from only 155 training puzzles, shown without solutions. Three GPT-Neo sizes (125M, 1.3B, 2.7B) were tested, and Codex served as an additional puzzle generator.
- The authors release four synthetic datasets of 1M puzzle–solution pairs each, plus code, in the microsoft/PythonProgrammingPuzzles GitHub repository.
- Fine-tuning on self-generated, verified data more than doubles test Pass@k. Codex-generated data helped the Neo models even more. A second iteration added only small gains, suggesting a plateau.
- Ablations show that filtering by the interpreter accounts for much of the improvement. Unverified data helped somewhat, but verified data helped considerably more.
- An analysis using Codex embeddings, UMAP, and an entropy metric found that larger models produce more diverse puzzles. Smaller models' puzzles sat closer to the training set.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 403 works · 162,838 citations
- [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) — University, Cambridge, United States · 274 works · 82,013 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,888 works · [🖥️ Computer Science Applications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-science-applications/) · 184 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works

## Cite

Cite as: Patrick Haluptzok, Matthew Bowers and Adam Tauman Kalai. 2023. Language Models Can Teach Themselves to Program Better. arXiv:2207.14502.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{haluptzok2023language,
  title = {{Language Models Can Teach Themselves to Program Better}},
  author = {Patrick Haluptzok, Matthew Bowers and Adam Tauman Kalai},
  year = {2023},
  doi = {10.48550/arxiv.2207.14502},
  eprint = {2207.14502},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2207.14502}
}
```

- [← previous Improving Language Models via Plug-and-Play Retrieval Feedback](https://md.research.elamur.ai/free/en/papers/yu-2023-improving-language-models-plug-play/)
- [next → Learning Interactive Real-World Simulators](https://md.research.elamur.ai/free/en/papers/yang-2023-learning-interactive-real-world-simulators/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/)

## References

33 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ferran Alet, Javier Lopez-Contreras, James Koppel, Maxwell Nye, Armando Solar-Lezama, Tomas Lozano-Perez, Leslie Kaelbling, and Joshua Tenenbaum. 2021. [A large-scale benchmark for few-shot program induction and synthesis](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). In Proceedings of the 38th International Conference on Machine Learning (Proce…
2. Eser Aygün, Zafarali Ahmed, Ankit Anand, Vlad Firoiu, Xavier Glorot, Laurent Orseau, Doina Precup, and Shibl Mourad. 2020. [Learning to prove from synthetic theorems](https://md.research.elamur.ai/free/en/papers/aygun-2020-learning-prove-synthetic-theorems/). arXiv preprint arXiv:2006.11259 (2020).
3. Matej Balog, Alexander L. Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2017. DeepCoder: Learning to Write Programs. In International Conference on Representation Learning (ICLR). [find](https://scholar.google.com/scholar?q=Matej+Balog%2C+Alexander+L%2E+Gaunt%2C+Marc+Brockschmidt%2C+Sebastian+Nowozin%2C+and+Daniel+Tarlow%2E+2017%2E+DeepCoder%3A+Learning+to+Write+Programs%2E+In+International+Conference+on+Representation+Learning+%28ICLR%29%2E)
4. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. https://doi.org/10.5281/zenodo.5551208 — [doi:10.5281/zenodo.5551208](https://doi.org/10.5281/zenodo.5551208) [↗](https://doi.org/10.5281/zenodo.5551208)
5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
6. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
7. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
8. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training Verifiers to Solve Math Word Problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). (2021). arXiv:2110.14168 [cs.LG]
9. Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sablé-Meyer, Lucas Morales, Luke Hewitt, Luc Cary, Armando Solar-Lezama, and Joshua B. Tenenbaum. 2021. DreamCoder: Bootstrapping Inductive Program Synthesis with Wake-Sleep Library Learning. Association for Computing Machinery, New York, NY, USA, 83… — [doi:10.1145/3453483.3454080](https://doi.org/10.1145/3453483.3454080) [find](https://scholar.google.com/scholar?q=Kevin+Ellis%2C+Catherine+Wong%2C+Maxwell+Nye%2C+Mathias+Sabl%C3%A9%2DMeyer%2C+Lucas+Morales%2C+Luke+Hewitt%2C+Luc+Cary%2C+Armando+Solar%2DLezama%2C+and+Joshua+B%2E+Tenenbaum%2E+2021%2E+DreamCoder%3A+Bootstrapping+Inductive+Program+Sy)
10. Robert M French. 1999. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences 3, 4 (1999), 128–135. [find](https://scholar.google.com/scholar?q=Robert+M+French%2E+1999%2E+Catastrophic+forgetting+in+connectionist+networks%2E+Trends+in+cognitive+sciences+3%2C+4+%281999%29%2C+128%E2%80%93135%2E)
11. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv preprint arXiv:2101.00027 (2020). [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+et+al%2E+2020%2E+The+Pile%3A+An+800GB+Dataset+of+Diverse+Text+for+Lan)
12. Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021. Knowledge Distillation: A Survey. International Journal of Computer Vision 129, 6 (2021), 1789–1819. https://doi.org/10.1007/s11263-021-01453-z — [doi:10.1007/s11263-021-01453-z](https://doi.org/10.1007/s11263-021-01453-z) [↗](https://doi.org/10.1007/s11263-021-01453-z)
13. Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, et al. 2017. [Program synthesis](https://md.research.elamur.ai/free/en/papers/gulwani-2017-program-synthesis/). Foundations and Trends in Programming Languages 4, 1-2 (2017), 1–119.
14. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Steven+Basart%2C+Saurav+Kadavath%2C+Mantas+Mazeika%2C+Akul+Arora%2C+Ethan+Guo%2C+Collin+Burns%2C+Samir+Puranik%2C+Horace+He%2C+Dawn+Song%2C+and+Jacob+Steinhardt%2E+2021%2E)
15. [Measuring Coding Challenge Competence With APPS](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/). (2021). arXiv:2105.09938 [cs.SE]
16. Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. [Distilling the Knowledge in a Neural Network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). CoRR abs/1503.02531 (2015). arXiv:1503.02531 http://arxiv.org/abs/1503.02531 [↗](http://arxiv.org/abs/1503.02531)
17. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
18. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James M… [find](https://scholar.google.com/scholar?q=Yujia+Li%2C+David+Choi%2C+Junyoung+Chung%2C+Nate+Kushman%2C+Julian+Schrittwieser%2C+R%C3%A9mi+Leblond%2C+Tom+Eccles%2C+James+Keeling%2C+Felix+Gimeno%2C+Agustin+Dal+Lago%2C+Thomas+Hubert%2C+Peter+Choy%2C+Cyprien+de+Masson+d%E2%80%99Autume)
19. Leland McInnes, John Healy, and James Melville. 2018. [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://md.research.elamur.ai/free/en/papers/mcinnes-2018-umap-uniform-manifold-approximation-projection/). arXiv e-prints, Article arXiv:1802.03426 (Feb. 2018), arXiv:1802.03426 pages. arXiv:1802.03426 [stat.ML]
20. McInnes, Leland. 2020. GitHub - lmcinnes/umap: Uniform Manifold Approximation and Projection (UMAP). [https://github.com/lmcinnes/umap](https://github.com/lmcinnes/umap). [↗](https://github.com/lmcinnes/umap)
21. Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022. [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://md.research.elamur.ai/free/en/papers/meng-2022-generating-training-data-language-models/). arXiv preprint arXiv:2202.04538 (2022). https://arxiv.org/abs/2202.04538 [↗](https://arxiv.org/abs/2202.04538)
22. Aditya Krishna Menon, Omer Tamuz, Sumit Gulwani, Butler W Lampson, and Adam Kalai. 2013. A Machine Learning Framework for Programming by Example.. In ICML. 187–195. [find](https://scholar.google.com/scholar?q=Aditya+Krishna+Menon%2C+Omer+Tamuz%2C+Sumit+Gulwani%2C+Butler+W+Lampson%2C+and+Adam+Kalai%2E+2013%2E+A+Machine+Learning+Framework+for+Programming+by+Example%2E%2E+In+ICML%2E+187%E2%80%93195%2E)
23. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (201… [find](https://scholar.google.com/scholar?q=F%2E+Pedregosa%2C+G%2E+Varoquaux%2C+A%2E+Gramfort%2C+V%2E+Michel%2C+B%2E+Thirion%2C+O%2E+Grisel%2C+M%2E+Blondel%2C+P%2E+Prettenhofer%2C+R%2E+Weiss%2C+V%2E+Dubourg%2C+J%2E+Vanderplas%2C+A%2E+Passos%2C+D%2E+Cournapeau%2C+M%2E+Brucher%2C+M%2E+Perrot%2C+and+E%2E+Duc)
24. Stanislas Polu and Ilya Sutskever. 2020. [Generative Language Modeling for Automated Theorem Proving](https://md.research.elamur.ai/free/en/papers/polu-2020-generative-language-modeling-automated-theorem/). https://doi.org/10.48550/ARXIV.2009.03393 [↗](https://doi.org/10.48550/ARXIV.2009.03393)
25. Marco Tulio Ribeiro and Scott Lundberg. 2022. Adaptive Testing and Debugging of NLP Models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Dublin, Ireland, 3253–3267. https://doi.org/10.18… [find](https://scholar.google.com/scholar?q=Marco+Tulio+Ribeiro+and+Scott+Lundberg%2E+2022%2E+Adaptive+Testing+and+Debugging+of+NLP+Models%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+P)
26. Baptiste Roziere, Jie M Zhang, Francois Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. 2021. [Leveraging automated unit tests for unsupervised code translation](https://md.research.elamur.ai/free/en/papers/roziere-2022-leveraging-automated-unit-tests-unsupervised/). arXiv preprint arXiv:2110.06773 (2021).
27. Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 (Lugano and Virtual Event, Switzerland) (I… [find](https://scholar.google.com/scholar?q=Sami+Sarsa%2C+Paul+Denny%2C+Arto+Hellas%2C+and+Juho+Leinonen%2E+2022%2E+Automatic+Generation+of+Programming+Exercises+and+Code+Explanations+Using+Large+Language+Models%2E+In+Proceedings+of+the+2022+ACM+Conference)
28. Timo Schick and Hinrich Schütze. 2021. [Generating Datasets with Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/schick-2021-generating-datasets-pretrained-language-models/). arXiv preprint arXiv:2104.07540 (2021). https://arxiv.org/abs/2104.07540 [↗](https://arxiv.org/abs/2104.07540)
29. Tal Schuster, Ashwin Kalyan, Alex Polozov, and Adam Tauman Kalai. 2021. Programming Puzzles. In Thirty-fifth Conference on Neural Information Processing Systems. [https://openreview.net/forum?id=fe_hCc4RBrg](https://openreview.net/forum?id=fe_hCc4RBrg) Richard Shin, Neel Kant, Kavi Gupta, Chris Bender, Brandon Trabucco, Rishabh Singh, and Dawn… [↗](https://openreview.net/forum?id=fe_hCc4RBrg)
30. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research 9 (2008), 2579–2605. [find](https://scholar.google.com/scholar?q=Laurens+van+der+Maaten+and+Geoffrey+Hinton%2E+2008%2E+Visualizing+data+using+t%2DSNE%2E+Journal+of+Machine+Learning+Research+9+%282008%29%2C+2579%E2%80%932605%2E)
31. Peter West, Chandra Bhagavatula, Jack Hessel, Jena D. Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2021. [Symbolic Knowledge Distillation: from General Language Models to Commonsense Models](https://md.research.elamur.ai/free/en/papers/west-2022-symbolic-knowledge-distillation-general-language/). (2021). arXiv:2110.07178 [cs.CL]
32. Yuhuai Wu, Markus Rabe, Wenda Li, Jimmy Ba, Roger Grosse, and Christian Szegedy. 2021. [LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning](https://md.research.elamur.ai/free/en/papers/wu-2021-lime-learning-inductive-bias-primitives/). https://doi.org/10.48550/ARXIV.2101.06223 [↗](https://doi.org/10.48550/ARXIV.2101.06223)
33. Maksym Zavershynskyi, Alexander Skidanov, and Illia Polosukhin. 2018. NAPS: Natural Program Synthesis Dataset. CoRR abs/1807.03168 (2018). arXiv:1807.03168 http://arxiv.org/abs/1807.03168 [↗](http://arxiv.org/abs/1807.03168)

*Updated 3 November 2026*