--- title: "Physics of Language Models — paper by Ye et al. (2024)" description: "Because a program creates the problems, wrong steps and their exact corrections can… 99% accuracy. 3 references, cited by 27 papers here." authors: ["Tian Ye", "Zicheng Xu", "Yuanzhi Li", "Zeyuan Allen-Zhu"] year: 2024 venue: "arXiv:2408.16293" source: "https://arxiv.org/abs/2408.16293" doi: "10.48550/arxiv.2408.16293" arxiv: "2408.16293" licence: "arXiv — non-exclusive licence to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/ye-2024-physics-language-models-part-2-2/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ye-2024-physics-language-models-part-2-2/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/) / Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems # Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems **Authors:** [Tian Ye](https://md.research.elamur.ai/free/en/authors/tian-ye/), [Zicheng Xu](https://md.research.elamur.ai/free/en/authors/zicheng-xu/), [Yuanzhi Li](https://md.research.elamur.ai/free/en/authors/yuanzhi-li/) and [Zeyuan Allen-Zhu](https://md.research.elamur.ai/free/en/authors/zeyuan-allen-zhu/) · **Year:** 2024 · **Venue:** arXiv:2408.16293 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2408.16293) · [PDF](https://arxiv.org/pdf/2408.16293) · [DOI 10.48550/arxiv.2408.16293](https://doi.org/10.48550/arxiv.2408.16293) · [arXiv 2408.16293](https://arxiv.org/abs/2408.16293) · [OpenAlex](https://openalex.org/W4410357133) **Citations:** 4 citations (OpenAlex, 2026-10-14) · [27](https://md.research.elamur.ai/free/en/papers/ye-2024-physics-language-models-part-2-2/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Strong language models still make reasoning slips. They can often fix these when prompted to re-check their work, but not while they are generating. This paper asks whether the ability to catch and repair errors mid-solution can be built in during pretraining. To keep the experiments controlled, the authors use iGSM, a corpus of program-generated grade-school math problems. Because a program creates the problems, wrong steps and their exact corrections can be produced with full reliability, without relying on GPT-4. The authors find three main things: - A GPT2 model pretrained only on clean solutions already "knows" when a step is wrong. A tiny probe can read this out, and regenerating flagged steps ("retry upon regret") beats beam search. - Pretraining on "retry data", where a faulty step is immediately followed by its fix, gives higher accuracy under plain autoregressive decoding than the same amount of error-free data. - They also examine masking, error rate, data preparation and moving such data into fine-tuning. The work matters because it gives practical guidance on building pretraining corpora that teach models to correct themselves. ## Key points - Setting: the iGSM dataset (iGSM-med and iGSM-hard splits), with GPT2-12-12 models (hidden size 768). Each configuration is evaluated on 4096 problems, using 2 seeds and beam size 1 or 4. - Error detection uses V-probing: a rank-8 update on the embedding layer plus a linear head. It reaches over 99% accuracy, far above what prompting-based self-verification achieves. - A second, cheaper probe uses rank-32 embedding updates plus rank-2 query/value updates. It needs only about 0.25% of the pretraining tokens. - Deferring correction data to fine-tuning was tested in two ways: - LoRA with ranks r ∈ {4, …, 256} - Full fine-tuning, treated as continued pretraining on the same number of tokens as pretraining - Motivating example: GPT-4o (2024-08-06) often insists a wrong solution is correct even when told which line contains the error. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 533 works · 443,884 citations - [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 633 works · 195,217 citations - [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 224 works · 241,960 citations - [Mohamed bin Zayed University of Artificial Intelligence](https://md.research.elamur.ai/free/en/universities/mohamed-bin-zayed-university-of-artificial-intelligence/) — University, Abu Dhabi, United Arab Emirates · 145 works · 6,811 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,621 works 🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works ## Cite Cite as: Tian Ye, Zicheng Xu, Yuanzhi Li and Zeyuan Allen-Zhu. 2024. Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems. arXiv:2408.16293. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{ye2024physics, title = {{Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems}}, author = {Tian Ye, Zicheng Xu, Yuanzhi Li and Zeyuan Allen-Zhu}, year = {2024}, doi = {10.48550/arxiv.2408.16293}, eprint = {2408.16293}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2408.16293} } ``` - [← previous Modeling Gender and Dialect Bias in Automatic Speech Recognition](https://md.research.elamur.ai/free/en/papers/harris-2024-modeling-gender-dialect-bias-automatic/) - [next → Position: What Can Large Language Models Tell Us about Time Series Analysis](https://md.research.elamur.ai/free/en/papers/jin-2024-position-large-language-models-tell/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/) ## References 3 references, 3 of them with a card of ours. ↗ leads to the work. 1. Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. [Transformers learn to implement preconditioned gradient descent for in-context learning](https://md.research.elamur.ai/free/en/papers/ahn-2023-transformers-learn-implement-preconditioned-gradient/). Advances in Neural Information Processing Systems, 36, 2024. 2. Zeyuan Allen-Zhu and Yuanzhi Li. [Physics of Language Models: Part 1, Learning Hierarchical Language Structures](https://md.research.elamur.ai/free/en/papers/allen-zhu-2023-physics-language-models-part-1/). ArXiv e-prints, abs/2305.13673, May 2023. Full version available at http://arxiv.org/abs/2305.13673. [↗](http://arxiv.org/abs/2305.13673) 3. Zeyuan Allen-Zhu and Yuanzhi Li. [Physics of Language Models: Part 3.2, Knowledge Manipulation](https://md.research.elamur.ai/free/en/papers/allen-zhu-2023-physics-language-models-part-3-2/). ArXiv e-prints, abs/2309.14402, September 2023. Full version available at http://arxiv.org/abs/2309.14402. [↗](http://arxiv.org/abs/2309.14402) *Updated 3 November 2026*