---
title: "Program-of-Thought Reveals LLM Abstraction Ceilings (2026)"
description: "This paper tests whether LLMs that seem to reason about math actually grasp a problem's structure. 19 references."
authors: ["Mike Zhou", "Fenil Bardoliya", "Vivek Gupta", "Dan Roth"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: EACL 2026"
source: "https://aclanthology.org/2026.findings-eacl.257/"
doi: "10.18653/v1/2026.findings-eacl.257"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhou-2026-program-thought-reveals-llm-abstraction/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhou-2026-program-thought-reveals-llm-abstraction/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/45/) / Program-of-Thought Reveals LLM Abstraction Ceilings

# Program-of-Thought Reveals LLM Abstraction Ceilings

**Authors:** Mike Zhou, Fenil Bardoliya, [Vivek Gupta](https://md.research.elamur.ai/free/en/authors/vivek-gupta/) and [Dan Roth](https://md.research.elamur.ai/free/en/authors/dan-roth/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: EACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-eacl.257/) · [PDF](https://aclanthology.org/2026.findings-eacl.257.pdf) · [DOI 10.18653/v1/2026.findings-eacl.257](https://doi.org/10.18653/v1/2026.findings-eacl.257) · [OpenAlex](https://openalex.org/W7140118366)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
This paper tests whether LLMs that seem to reason about math actually grasp a problem's structure. The criterion is invariance: problems with identical computations but different numbers should be solved equally well. The authors evaluate LLaMA-3.1-8B-Instruct, Mistral-7B-Instruct, Llama-4 Maverick, Qwen-3-8B, GPT-OSS-120B and Deepseek-V3.2-Exp on numerically perturbed GSM8K and MATH items from ReasonAgain. They use Direct, CoT, CoT+Self-Consistency and Program-of-Thought (PoT) prompting. PoT matters because executable code separates failures of abstraction from arithmetic slips. They also LoRA-fine-tune LLaMA-3.1-8B and Mistral-7B on GSM8K Python solutions in two variants: with concrete values, or with names and numbers masked as typed variables. Every model loses substantial accuracy on perturbed variants, and PoT prompting only partly narrows the gap. Fine-tuning makes behaviour more uniform across variants but barely shrinks the gap. The effect also disappears under CoT evaluation and on MATH. Models end up stably wrong, which suggests current supervision teaches solution templates rather than abstraction. The authors argue benchmarks should report the perturbation gap alongside accuracy.

## Key points
- Data: ReasonAgain perturbations comprising 6,701 perturbed plus 1,121 original GSM8K items and 1,513 perturbed plus 268 original MATH items. The authors estimate label noise at about 8% (GSM8K) and 17% (MATH).
- The metric is all-variants accuracy (AVA) and its gap from original accuracy (∆AVA). Example: LLaMA-3.1-8B with CoT on GSM8K falls from 91.92% to 72.50%. With PoT its gap is smaller (13.21). Qwen-3-8B with CoT on MATH shows a gap of 44.63.
- Fine-tuning setup: LoRA with rank 16 and α = 32, 3 epochs, LR 10⁻⁵, effective batch 64. Regimes: PoT–Concrete vs. PoT–Masked.
- Even after training on traces with explicit ceil/fencepost logic, roughly 6–9% of GSM8K (rounding-sensitive items) keeps failing.
- Error decomposition into new, shared and recovered errors shows that fine-tuning consolidates existing mistakes rather than fixing them.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Pennsylvania](https://md.research.elamur.ai/free/en/universities/university-of-pennsylvania/) — University, Philadelphia, United States · 233 works · 65,583 citations
- [Arizona State University](https://md.research.elamur.ai/free/en/universities/arizona-state-university/) — University, Tempe, United States · 54 works · 2,356 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Mike Zhou, Fenil Bardoliya, Vivek Gupta and Dan Roth. 2026. Program-of-Thought Reveals LLM Abstraction Ceilings. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhou2026program,
  title = {{Program-of-Thought Reveals LLM Abstraction Ceilings}},
  author = {Mike Zhou, Fenil Bardoliya, Vivek Gupta and Dan Roth},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.257},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.257}
}
```

- [← previous Program Structure-aware Language Models: Targeted Software Testing beyond Textual Semantics](https://md.research.elamur.ai/free/en/papers/tran-2026-program-structure-aware-language-models/)
- [next → Progressive Re-ranking for Multimodal Retrieval-Augmented Generation via Curriculum Learning](https://md.research.elamur.ai/free/en/papers/min-2026-progressive-re-ranking-multimodal-retrieval/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/45/)

## References

19 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, and 105 others. 2025. [gpt-oss-120b & gpt-oss…](https://md.research.elamur.ai/free/en/papers/agarwal-2025-gpt-oss-120b-gpt-oss/)
2. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research. [find](https://scholar.google.com/scholar?q=Wenhu+Chen%2C+Xueguang+Ma%2C+Xinyi+Wang%2C+and+William+W%2E+Cohen%2E+2023%2E+Program+of+thoughts+prompting%3A+Disentangling+computation+from+reasoning+for+numerical+reasoning+tasks%2E+Transactions+on+Machine+Learning)
3. Accepted November 2023. [find](https://scholar.google.com/scholar?q=Accepted+November+2023%2E)
4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Chris Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
5. DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenhao Xu, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, and 245 others. 2025. [Deepseek-v3.2: Pushing the frontier of open…](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-v3-2-pushing/)
6. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. 2023. [Faith and fate: Limits of transformers on composition…](https://md.research.elamur.ai/free/en/papers/dziri-2023-faith-fate-limits-transformers-compositionality/)
7. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. PAL: Program-aided language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10764–10799. PMLR. [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Pengfei+Liu%2C+Yiming+Yang%2C+Jamie+Callan%2C+and+Graham+Neubig%2E+2023%2E+PAL%3A+Program%2Daided+language+models%2E+In+Proceedings+of+the+40th+International+Conference+o)
8. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 540 o… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
9. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Advances in Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS 2021).
10. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In Proceedings of the 10th International Conference on Learning Representations (ICLR).
11. Meta AI. 2025. Llama 4: Multimodal intelligence. [https://ai.meta.com/blog/](https://ai.meta.com/blog/) llama-4-multimodal-intelligence/. Accessed: 2025-10-05. [↗](https://ai.meta.com/blog/)
12. Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2025. [Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models](https://md.research.elamur.ai/free/en/papers/mirzadeh-2024-gsm-symbolic-understanding-limitations-mathematical/). In Proceedings of the 13th International Conference on Learning Representations (ICLR).
13. Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. 2025. [The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity](https://md.research.elamur.ai/free/en/papers/shojaee-2025-illusion-thinking-understanding-strengths-limitations/). arXiv preprint, arXiv:2506.06941.
14. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In Proceedings of the 11th International Conference on Learning Representations (ICLR).
15. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H Chi, Quoc V Le, and Denny Zhou. 2022. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems (NeurIPS 2022).
16. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
17. Xiaodong Yu, Ben Zhou, Hao Cheng, and Dan Roth. 2024. Reasonagain: Using extractable symbolic programs to evaluate mathematical reasoning. arXiv preprint arXiv:2410.19056. [find](https://scholar.google.com/scholar?q=Xiaodong+Yu%2C+Ben+Zhou%2C+Hao+Cheng%2C+and+Dan+Roth%2E+2024%2E+Reasonagain%3A+Using+extractable+symbolic+programs+to+evaluate+mathematical+reasoning%2E+arXiv+preprint+arXiv%3A2410%2E19056%2E)
18. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V. Le, and Ed H. Chi. 2023. [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/). In Proceedings of the 11th International Conference on Learning Re…
19. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Téléométr Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacr… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)

*Updated 2 November 2026*