---
title: "Language Models as Compilers — paper by Chae et al. (2024)"
description: "Natural-language plans tend to describe logic imprecisely. 26 references, cited by 7 papers here."
authors: ["Hyungjoo Chae", "Yeonghyeon Kim", "Seungone Kim", "Kai Tzu-iunn Ong", "Beong-woo Kwak", "Moohyeon Kim", "Sunghwan Kim", "Taeyoon Kwon", "Jiwan Chung", "Youngjae Yu", "Jinyoung Yeo"]
year: 2024
venue: "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2024.emnlp-main.1253/"
doi: "10.18653/v1/2024.emnlp-main.1253"
arxiv: "2404.02575"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chae-2024-language-models-as-compilers-simulating/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chae-2024-language-models-as-compilers-simulating/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/15/) / Language Models as Compilers: Simulating Pseudocode Execution Improves Algorithmic Reasoning in Language Models

# Language Models as Compilers: Simulating Pseudocode Execution Improves Algorithmic Reasoning in Language Models

**Authors:** Hyungjoo Chae, Yeonghyeon Kim, [Seungone Kim](https://md.research.elamur.ai/free/en/authors/seungone-kim/), Kai Tzu-iunn Ong, Beong-woo Kwak, Moohyeon Kim, Sunghwan Kim, [Taeyoon Kwon](https://md.research.elamur.ai/free/en/authors/taeyoon-kwon/), Jiwan Chung, [Youngjae Yu](https://md.research.elamur.ai/free/en/authors/youngjae-yu/) and [Jinyoung Yeo](https://md.research.elamur.ai/free/en/authors/jinyoung-yeo/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2024](https://md.research.elamur.ai/free/en/venues/emnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.emnlp-main.1253/) · [PDF](https://aclanthology.org/2024.emnlp-main.1253.pdf) · [DOI 10.18653/v1/2024.emnlp-main.1253](https://doi.org/10.18653/v1/2024.emnlp-main.1253) · [arXiv 2404.02575](https://arxiv.org/abs/2404.02575) · [OpenAlex](https://openalex.org/W4404782220)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/chae-2024-language-models-as-compilers-simulating/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models still struggle with algorithmic reasoning, such as completing Dyck sequences or tracking who lies in a chain of statements. Existing remedies fall short in different ways. Chain-of-Thought and Program-of-Thought reason one instance at a time with no separate planning stage. Natural-language plans tend to describe logic imprecisely. The authors propose THINK-AND-EXECUTE, which separates planning from solving.

In the THINK phase, an Instructor LM receives a meta prompt containing hand-annotated pseudocode for three other tasks, plus three sample questions from the target task. It analyses the reasoning the task requires and writes a single task-level pseudocode prompt. Its print() calls specify which intermediate rationales to produce.

In the EXECUTE phase, a Reasoner LM applies that pseudocode to each instance. It simulates the program's execution, generating the intermediate outputs before the final answer.

On seven Big-Bench Hard tasks, the method beats direct prompting, zero-shot CoT, PoT, NL planning, Chain-of-Code and Plan-and-Solve. This suggests that reusable pseudocode guides models better than natural-language instructions or code written separately for each instance.

## Key points
- With GPT-3.5-Turbo, the average zero-shot accuracy is 60.4, compared with 48.0 for zero-shot CoT, 50.3 for Plan-and-Solve, 32.4 for PoT and 28.1 for Chain-of-Code.
- Pseudocode generated by a large model transfers to smaller models. CodeLlama-7B improves from 18.4 (CoT) to 30.8 on average, and CodeLlama-13B from 28.2 to 36.3.
- Gains are strongest on Navigate (96.8 with GPT-3.5-Turbo) and Web of Lies (72.8 vs. 59.2 for CoT). Dyck Languages remains very hard for all methods (at most 8.0).
- The method consumes more tokens than CoT or PoT, but its accuracy-to-cost trade-off lies on the Pareto front.
- A demonstrative experiment on SayCan robot planning yields more accurate plans than the baselines. The main limitation is the reliance on human-annotated meta prompts.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 633 works · 195,218 citations
- [Yonsei University](https://md.research.elamur.ai/free/en/universities/yonsei-university/) — University, Seoul, South Korea · 20 works · 515 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,888 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,948 works

## Cite

Cite as: Hyungjoo Chae, Yeonghyeon Kim, Seungone Kim, Kai Tzu-iunn Ong, Beong-woo Kwak, Moohyeon Kim, Sunghwan Kim, Taeyoon Kwon, Jiwan Chung, Youngjae Yu and Jinyoung Yeo. 2024. Language Models as Compilers: Simulating Pseudocode Execution Improves Algorithmic Reasoning in Language Models. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chae2024language,
  title = {{Language Models as Compilers: Simulating Pseudocode Execution Improves Algorithmic Reasoning in Language Models}},
  author = {Hyungjoo Chae, Yeonghyeon Kim, Seungone Kim, Kai Tzu-iunn Ong, Beong-woo Kwak, Moohyeon Kim, Sunghwan Kim, Taeyoon Kwon, Jiwan Chung, Youngjae Yu and Jinyoung Yeo},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.1253},
  eprint = {2404.02575},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.1253}
}
```

- [← previous Interpreting Multi-band Galaxy Observations with Large Language Model-Based Agents](https://md.research.elamur.ai/free/en/papers/sun-2024-interpreting-multi-band-galaxy-observations/)
- [next → Logging Keystrokes in Writing by English Learners](https://md.research.elamur.ai/free/en/papers/velentzas-2024-logging-keystrokes-writing-english-learners/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/15/)

## References

26 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. OpenAI. 2023. Chatgpt. [https://openai.com/](https://openai.com/) blog/chatgpt. [↗](https://openai.com/)
3. Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. 2023. [Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning](https://md.research.elamur.ai/free/en/papers/pan-2023-logic-lm-empowering-large-language/). ArXiv, abs/2305.12295.
4. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2023. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research. [find](https://scholar.google.com/scholar?q=Wenhu+Chen%2C+Xueguang+Ma%2C+Xinyi+Wang%2C+and+William+W+Cohen%2E+2023%2E+Program+of+thoughts+prompting%3A+Disentangling+computation+from+reasoning+for+numerical+reasoning+tasks%2E+Transactions+on+Machine+Learning+)
5. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950.
6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
7. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. [Challenging big-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). arXiv preprint arXiv:2210.09261.
8. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764–10799. PMLR. [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Pengfei+Liu%2C+Yiming+Yang%2C+Jamie+Callan%2C+and+Graham+Neubig%2E+2023%2E+Pal%3A+Program%2Daided+language+models%2E+In+International+Conference+on+Machine+Learning%2C+page)
9. Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. 2023. Reasoning with language model is planning with world model. ArXiv, abs/2305.14992. [find](https://scholar.google.com/scholar?q=Shibo+Hao%2C+Yi+Gu%2C+Haodi+Ma%2C+Joshua+Jiahua+Hong%2C+Zhen+Wang%2C+Daisy+Zhe+Wang%2C+and+Zhiting+Hu%2E+2023%2E+Reasoning+with+language+model+is+planning+with+world+model%2E+ArXiv%2C+abs%2F2305%2E14992%2E)
10. Haozhe Ji, Pei Ke, Shaohan Huang, Furu Wei, Xiaoyan Zhu, and Minlie Huang. 2020. Language generation with multi-hop reasoning on commonsense knowledge graph. In Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Haozhe+Ji%2C+Pei+Ke%2C+Shaohan+Huang%2C+Furu+Wei%2C+Xiaoyan+Zhu%2C+and+Minlie+Huang%2E+2020%2E+Language+generation+with+multi%2Dhop+reasoning+on+commonsense+knowledge+graph%2E+In+Conference+on+Empirical+Methods+in+Natu)
11. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). Advances in neural information processing systems, 35:22199– 22213.
12. Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2022. Large language models still can’t plan (a benchmark for LLMs on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop. [find](https://scholar.google.com/scholar?q=Karthik+Valmeekam%2C+Alberto+Olmo%2C+Sarath+Sreedharan%2C+and+Subbarao+Kambhampati%2E+2022%2E+Large+language+models+still+can%E2%80%99t+plan+%28a+benchmark+for+LLMs+on+planning+and+reasoning+about+change%29%2E+In+NeurIPS+202)
13. Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon… [find](https://scholar.google.com/scholar?q=Lei+Wang%2C+Wanyu+Xu%2C+Yihuai+Lan%2C+Zhiqiang+Hu%2C+Yunshi+Lan%2C+Roy+Ka%2DWei+Lee%2C+and+Ee%2DPeng+Lim%2E+2023%2E+Plan%2Dand%2Dsolve+prompting%3A+Improving+zero%2Dshot+chain%2Dof%2Dthought+reasoning+by+large+language+models%2E+In+Pr)
14. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171.
15. Chengshu Li, Jacky Liang, Fei Xia, Andy Zeng, Sergey Levine, Dorsa Sadigh, Karol Hausman, Xinyun Chen, Li Fei-Fei, and brian ichter. 2023. Chain of code: Reasoning with a language model-augmented code interpreter. In NeurIPS 2023 Foundation Models for Decision Making Workshop. [find](https://scholar.google.com/scholar?q=Chengshu+Li%2C+Jacky+Liang%2C+Fei+Xia%2C+Andy+Zeng%2C+Sergey+Levine%2C+Dorsa+Sadigh%2C+Karol+Hausman%2C+Xinyun+Chen%2C+Li+Fei%2DFei%2C+and+brian+ichter%2E+2023%2E+Chain+of+code%3A+Reasoning+with+a+language+model%2Daugmented+code)
16. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems.
17. Aman Madaan and Amir Yazdanbakhsh. 2022. [Text and patterns: For effective chain of thought, it takes two to tango](https://md.research.elamur.ai/free/en/papers/madaan-2022-text-patterns-effective-chain-thought/). arXiv preprint arXiv:2209.07686.
18. Nathaniel Weir, Muhammad Khalifa, Linlu Qiu, Orion Weller, and Peter Clark. 2024. Learning to reason via program generation, emulation, and search. [find](https://scholar.google.com/scholar?q=Nathaniel+Weir%2C+Muhammad+Khalifa%2C+Linlu+Qiu%2C+Orion+Weller%2C+and+Peter+Clark%2E+2024%2E+Learning+to+reason+via+program+generation%2C+emulation%2C+and+search%2E)
19. Mayank Mishra, Prince Kumar, Riyaz Bhat, Rudra Murthy V, Danish Contractor, and Srikanth Tamilsel-vam. 2023. Prompting with pseudo-code instructions. arXiv preprint arXiv:2305.11790. [find](https://scholar.google.com/scholar?q=Mayank+Mishra%2C+Prince+Kumar%2C+Riyaz+Bhat%2C+Rudra+Murthy+V%2C+Danish+Contractor%2C+and+Srikanth+Tamilsel%2Dvam%2E+2023%2E+Prompting+with+pseudo%2Dcode+instructions%2E+arXiv+preprint+arXiv%3A2305%2E11790%2E)
20. Dingjun Wu, Jing Zhang, and Xinmei Huang. 2023. Chain of thought prompting elicits knowledge augmentation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6519–6534, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dingjun+Wu%2C+Jing+Zhang%2C+and+Xinmei+Huang%2E+2023%2E+Chain+of+thought+prompting+elicits+knowledge+augmentation%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2023%2C+pages+6519%E2%80%936534%2C+Toro)
21. Niklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. 2024. Octopack: Instruction tuning code large language models. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Niklas+Muennighoff%2C+Qian+Liu%2C+Armel+Randy+Zebaze%2C+Qinkai+Zheng%2C+Binyuan+Hui%2C+Terry+Yue+Zhuo%2C+Swayam+Singh%2C+Xiangru+Tang%2C+Leandro+Von+Werra%2C+and+Shayne+Longpre%2E+2024%2E+Octopack%3A+Instruction+tuning+code+)
22. Eric Zelikman, Qian Huang, Gabriel Poesia, Noah Goodman, and Nick Haber. 2023. [Parsel: Algorithmic reasoning with language models by composing decompositions](https://md.research.elamur.ai/free/en/papers/zelikman-2023-parsel-algorithmic-reasoning-language-models/). Advances in Neural Information Processing Systems, 36:31466–31523.
23. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Com)
24. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. 2022a. [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/). arXiv preprint arXiv:2205.10625.
25. Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur, and Hanie Sedghi. 2022b. [Teaching algorithmic reasoning via in-context learning](https://md.research.elamur.ai/free/en/papers/zhou-2022-teaching-algorithmic-reasoning-context-learning/). arXiv preprint arXiv:2211.09066.
26. Pei Zhou, Jay Pujara, Xiang Ren, Xinyun Chen, Heng-Tze Cheng, Quoc V Le, Ed H Chi, Denny Zhou, Swaroop Mishra, and Huaixiu Steven Zheng. 2024. Self-discover: Large language models self-compose reasoning structures. arXiv preprint arXiv:2402.03620. [find](https://scholar.google.com/scholar?q=Pei+Zhou%2C+Jay+Pujara%2C+Xiang+Ren%2C+Xinyun+Chen%2C+Heng%2DTze+Cheng%2C+Quoc+V+Le%2C+Ed+H+Chi%2C+Denny+Zhou%2C+Swaroop+Mishra%2C+and+Huaixiu+Steven+Zheng%2E+2024%2E+Self%2Ddiscover%3A+Large+language+models+self%2Dcompose+reasoni)

*Updated 3 November 2026*