---
title: "Teaching Algorithmic Reasoning via In-context Learning (2022)"
description: "They propose algorithmic prompting : in-context examples that walk through every step of an… 49 references, cited by 73 papers here."
authors: ["Hattie Zhou", "Azade Nova", "Hugo Larochelle", "Aaron Courville", "Behnam Neyshabur", "Hanie Sedghi"]
year: 2022
venue: "arXiv:2211.09066"
source: "https://arxiv.org/abs/2211.09066"
doi: "10.48550/arxiv.2211.09066"
arxiv: "2211.09066"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhou-2022-teaching-algorithmic-reasoning-context-learning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhou-2022-teaching-algorithmic-reasoning-context-learning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / Teaching Algorithmic Reasoning via In-context Learning

# Teaching Algorithmic Reasoning via In-context Learning

**Authors:** [Hattie Zhou](https://md.research.elamur.ai/free/en/authors/hattie-zhou/), [Azade Nova](https://md.research.elamur.ai/free/en/authors/azade-nova/), Hugo Larochelle, [Aaron Courville](https://md.research.elamur.ai/free/en/authors/aaron-courville/), [Behnam Neyshabur](https://md.research.elamur.ai/free/en/authors/behnam-neyshabur/) and Hanie Sedghi · **Year:** 2022 · **Venue:** arXiv:2211.09066 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2211.09066) · [PDF](https://arxiv.org/pdf/2211.09066) · [DOI 10.48550/arxiv.2211.09066](https://doi.org/10.48550/arxiv.2211.09066) · [arXiv 2211.09066](https://arxiv.org/abs/2211.09066) · [OpenAlex](https://openalex.org/W4319997768)

**Citations:** 23 citations (OpenAlex, 2026-10-14) · [73](https://md.research.elamur.ai/free/en/papers/zhou-2022-teaching-algorithmic-reasoning-context-learning/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Large language models can generalize to new tasks from prompt examples, yet they still fail at algorithmic problems such as parity or multi-digit arithmetic, especially on inputs longer than those in the prompt. Zhou et al. argue that existing rationale-based prompts (scratchpad, chain-of-thought) leave the underlying rules ambiguous. A model may infer the wrong pattern, for instance about when a carry occurs. They propose **algorithmic prompting**: in-context examples that walk through every step of an algorithm with explicit natural-language explanations, so that only one interpretation is possible. The model is not fine-tuned. The work examines four stages:

- learning a single algorithm as a skill
- accumulating several skills in one prompt
- composing simpler skills into harder ones
- calling skills as tools inside math word problems

Evaluation separates in-distribution from out-of-distribution performance by answer length. Algorithmic prompts sharply cut errors on long arithmetic and parity compared with few-shot and chain-of-thought baselines. The study also shows that mixing informal reasoning with rigid algorithmic output in one context causes interference. For research on LLM reasoning, this suggests that longer, more precise demonstrations can translate context length into reliable systematic generalization.

## Key points

- Against the strongest baselines, error drops by roughly 10x on long parity, 9x on addition, 5x on multiplication and 2x on subtraction.
- In ablations, ambiguity-free explanations matter, and mistakes inside algorithmic examples hurt performance noticeably, unlike with other prompting styles.
- Combining natural-language reasoning with algorithmic steps in one prompt produces an interference effect. Adding a solution plan or prompt flags reduces it only partially.
- On GSM8k-Hard, a dialogue-style setup routes addition sub-questions via special tokens to a second model holding the addition prompt.
  - Overall accuracy roughly doubles, from 27.4% to 55.8%.
  - Addition accuracy rises from 61.9% to 98.4%.
  - Logical accuracy falls from 70.6% to 57.7%.
- The tool-use approach also sidesteps context-length limits. Learned algorithms can be invoked as subroutines inside more complex ones.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 492 works · 440,463 citations
- [Université de Montréal](https://md.research.elamur.ai/free/en/universities/universite-de-montreal/) — University, Montreal, Canada · 87 works · 91,597 citations
- [Mila - Quebec Artificial Intelligence Institute](https://md.research.elamur.ai/free/en/labs/mila-quebec-artificial-intelligence-institute/) — Research institute, Montreal, Canada · 65 works · 14,425 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur and Hanie Sedghi. 2022. Teaching Algorithmic Reasoning via In-context Learning. arXiv:2211.09066.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhou2022teaching,
  title = {{Teaching Algorithmic Reasoning via In-context Learning}},
  author = {Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur and Hanie Sedghi},
  year = {2022},
  doi = {10.48550/arxiv.2211.09066},
  eprint = {2211.09066},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2211.09066}
}
```

- [← previous MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction](https://md.research.elamur.ai/free/en/papers/wang-2022-maven-ere-unified-large-scale/)
- [next → True Few-Shot Learning with Prompts–-A Real-World Perspective](https://md.research.elamur.ai/free/en/papers/schick-2022-true-few-shot-learning-prompts/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

49 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anil, C., Wu, Y., Andreassen, A., Lewkowycz, A., Misra, V., Ramasesh, V., Slone, A., Gur-Ari, G., Dyer, E., and Neyshabur, B. (2022). [Exploring length generalization in large language models](https://md.research.elamur.ai/free/en/papers/anil-2022-exploring-length-generalization-large-language/). arXiv preprint arXiv:2207.04901.
2. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
3. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021). [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
4. Chen, X., Liang, C., Yu, A. W., Song, D., and Zhou, D. (2020). Compositional generalization via neural-symbolic stack machines. Advances in Neural Information Processing Systems, 33:1690–1701. [find](https://scholar.google.com/scholar?q=Chen%2C+X%2E%2C+Liang%2C+C%2E%2C+Yu%2C+A%2E+W%2E%2C+Song%2C+D%2E%2C+and+Zhou%2C+D%2E+%282020%29%2E+Compositional+generalization+via+neural%2Dsymbolic+stack+machines%2E+Advances+in+Neural+Information+Processing+Systems%2C+33%3A1690%E2%80%931701%2E)
5. Chiang, T.-R. and Chen, Y.-N. (2018). Semantically-aligned equation generation for solving and reasoning math word problems. arXiv preprint arXiv:1811.00720. [find](https://scholar.google.com/scholar?q=Chiang%2C+T%2E%2DR%2E+and+Chen%2C+Y%2E%2DN%2E+%282018%29%2E+Semantically%2Daligned+equation+generation+for+solving+and+reasoning+math+word+problems%2E+arXiv+preprint+arXiv%3A1811%2E00720%2E)
6. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022). [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311.
7. Cobbe, K., Kosaraju, V., Bavarian, M., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
8. Creswell, A. and Shanahan, M. https://arxiv.org/abs/2208.14271. (2022). — [Faithful Reasoning Using Large Language Models](https://md.research.elamur.ai/free/en/papers/creswell-2022-faithful-reasoning-using-large-language/) [↗](https://arxiv.org/abs/2208.14271)
9. Csordás, R., Irie, K., and Schmidhuber, J. (2021). The devil is in the detail: Simple tricks improve systematic generalization of transformers. In ACL. [find](https://scholar.google.com/scholar?q=Csorda%CC%81s%2C+R%2E%2C+Irie%2C+K%2E%2C+and+Schmidhuber%2C+J%2E+%282021%29%2E+The+devil+is+in+the+detail%3A+Simple+tricks+improve+systematic+generalization+of+transformers%2E+In+ACL%2E)
10. Faldu, K., Sheth, A., Kikani, P., Gaur, M., and Avasthi, A. (2021). Towards tractable mathematical reasoning: Challenges, strategies, and opportunities for solving math word problems. arXiv preprint arXiv:2111.05364. [find](https://scholar.google.com/scholar?q=Faldu%2C+K%2E%2C+Sheth%2C+A%2E%2C+Kikani%2C+P%2E%2C+Gaur%2C+M%2E%2C+and+Avasthi%2C+A%2E+%282021%29%2E+Towards+tractable+mathematical+reasoning%3A+Challenges%2C+strategies%2C+and+opportunities+for+solving+math+word+problems%2E+arXiv+preprint+a)
11. Gordon, J., Lopez-Paz, D., Baroni, M., and Bouchacourt, D. (2019). Permutation equivariant models for compositional generalization in language. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Gordon%2C+J%2E%2C+Lopez%2DPaz%2C+D%2E%2C+Baroni%2C+M%2E%2C+and+Bouchacourt%2C+D%2E+%282019%29%2E+Permutation+equivariant+models+for+compositional+generalization+in+language%2E+In+International+Conference+on+Learning+Representations%2E)
12. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. (2021). [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874.
13. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022). [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). arXiv preprint arXiv:2203.15556.
14. Jiang, A. Q., Li, W., Tworkowski, S., Czechowski, K., Odrzygóźdź, T., Miłoś, P., Wu, Y., and Jamnik, M. (2022). [Thor: Wielding hammers to integrate language models and automated theorem provers](https://md.research.elamur.ai/free/en/papers/jiang-2022-thor-wielding-hammers-integrate-language/). arXiv preprint arXiv:2205.10893.
15. Jones, E. and Steinhardt, J. (2022). Capturing failures of large language models via human cognitive biases. arXiv preprint arXiv:2202.12299. [find](https://scholar.google.com/scholar?q=Jones%2C+E%2E+and+Steinhardt%2C+J%2E+%282022%29%2E+Capturing+failures+of+large+language+models+via+human+cognitive+biases%2E+arXiv+preprint+arXiv%3A2202%2E12299%2E)
16. Jung, J., Qin, L., Welleck, S., Brahman, F., Bhagavatula, C., Bras, R. L., and Choi, Y. (2022). [Maieutic prompting: Logically consistent reasoning with recursive explanations](https://md.research.elamur.ai/free/en/papers/jung-2022-maieutic-prompting-logically-consistent-reasoning/).
17. Kaiser, Ł. and Sutskever, I. (2015). Neural gpus learn algorithms. arXiv preprint arXiv:1511.08228. [find](https://scholar.google.com/scholar?q=Kaiser%2C+%C5%81%2E+and+Sutskever%2C+I%2E+%282015%29%2E+Neural+gpus+learn+algorithms%2E+arXiv+preprint+arXiv%3A1511%2E08228%2E)
18. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361.
19. [Faithful reasoning using large language models](https://md.research.elamur.ai/free/en/papers/creswell-2022-faithful-reasoning-using-large-language/).
20. Kim, J., Hong, G., Kim, K.-m., Kang, J., and Myaeng, S.-H. (2021). Have you seen that number? investigating extrapolation in question answering models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7031–7037. [find](https://scholar.google.com/scholar?q=Kim%2C+J%2E%2C+Hong%2C+G%2E%2C+Kim%2C+K%2E%2Dm%2E%2C+Kang%2C+J%2E%2C+and+Myaeng%2C+S%2E%2DH%2E+%282021%29%2E+Have+you+seen+that+number%3F+investigating+extrapolation+in+question+answering+models%2E+In+Proceedings+of+the+2021+Conference+on+Empiric)
21. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. (2022). [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/).
22. Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al. (2022a). [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). arXiv preprint arXiv:2206.14858.
23. Lewkowycz, A., Andreassen, A., Dohan, D. M., Dyer, E. S., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., Wu, Y., Neyshabur, B., Gur-Ari, G., and Misra, V. (2022b). [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/).
24. Li, W., Yu, L., Wu, Y., and Paulson, L. C. (2020). Isarstep: a benchmark for high-level mathematical reasoning. arXiv preprint arXiv:2006.09265. [find](https://scholar.google.com/scholar?q=Li%2C+W%2E%2C+Yu%2C+L%2E%2C+Wu%2C+Y%2E%2C+and+Paulson%2C+L%2E+C%2E+%282020%29%2E+Isarstep%3A+a+benchmark+for+high%2Dlevel+mathematical+reasoning%2E+arXiv+preprint+arXiv%3A2006%2E09265%2E)
25. Ling, W., Yogatama, D., Dyer, C., and Blunsom, P. (2017). Program induction by rationale generation: Learning to solve and explain algebraic word problems. In ACL. [find](https://scholar.google.com/scholar?q=Ling%2C+W%2E%2C+Yogatama%2C+D%2E%2C+Dyer%2C+C%2E%2C+and+Blunsom%2C+P%2E+%282017%29%2E+Program+induction+by+rationale+generation%3A+Learning+to+solve+and+explain+algebraic+word+problems%2E+In+ACL%2E)
26. Madaan, A. and Yazdanbakhsh, A. (2022). [Text and patterns: For effective chain of thought, it takes two to tango](https://md.research.elamur.ai/free/en/papers/madaan-2022-text-patterns-effective-chain-thought/). arXiv preprint arXiv:2209.07686.
27. Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. (2022). [Rethinking the role of demonstrations: What makes in-context learning work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) arXiv preprint arXiv:2202.12837.
28. Nogueira, R., Jiang, Z., and Lin, J. (2021). [Investigating the limitations of transformers with simple arithmetic tasks](https://md.research.elamur.ai/free/en/papers/nogueira-2021-investigating-limitations-transformers-simple-arithmetic/). arXiv preprint arXiv:2102.13019.
29. Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. (2021). [Show your work: Scratchpads for intermediate computation with language models](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/). arXiv preprint arXiv:2112.00114.
30. Parisi, A., Zhao, Y., and Fiedel, N. (2022). [Talm: Tool augmented language models](https://md.research.elamur.ai/free/en/papers/parisi-2022-talm-tool-augmented-language-models/). arXiv preprint arXiv:2205.12255.
31. Polu, S. and Sutskever, I. (2020). [Generative language modeling for automated theorem proving](https://md.research.elamur.ai/free/en/papers/polu-2020-generative-language-modeling-automated-theorem/). arXiv preprint arXiv:2009.03393.
32. Rabe, M. N., Lee, D., Bansal, K., and Szegedy, C. (2020). Mathematical reasoning via self-supervised skip-tree training. arXiv preprint arXiv:2006.04757. [find](https://scholar.google.com/scholar?q=Rabe%2C+M%2E+N%2E%2C+Lee%2C+D%2E%2C+Bansal%2C+K%2E%2C+and+Szegedy%2C+C%2E+%282020%29%2E+Mathematical+reasoning+via+self%2Dsupervised+skip%2Dtree+training%2E+arXiv+preprint+arXiv%3A2006%2E04757%2E)
33. Razeghi, Y., Logan IV, R. L., Gardner, M., and Singh, S. (2022). Impact of pretraining term frequencies on few-shot reasoning. arXiv preprint arXiv:2202.07206. — [arXiv:2202.07206](https://arxiv.org/abs/2202.07206) [find](https://scholar.google.com/scholar?q=Razeghi%2C+Y%2E%2C+Logan+IV%2C+R%2E+L%2E%2C+Gardner%2C+M%2E%2C+and+Singh%2C+S%2E+%282022%29%2E+Impact+of+pretraining+term+frequencies+on+few%2Dshot+reasoning%2E+arXiv+preprint+arXiv%3A2202%2E07206%2E+%E2%80%94+%5BarXiv%3A2202%2E07206%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
34. Saxton, D., Grefenstette, E., Hill, F., and Kohli, P. (2019). [Analysing mathematical reasoning abilities of neural models](https://md.research.elamur.ai/free/en/papers/saxton-2019-analysing-mathematical-reasoning-abilities-neural/). arXiv preprint arXiv:1904.01557.
35. Thawani, A., Pujara, J., Szekely, P. A., and Ilievski, F. (2021). [Representing numbers in nlp: a survey and a vision](https://md.research.elamur.ai/free/en/papers/thawani-2021-representing-numbers-nlp-survey-vision/). arXiv preprint arXiv:2103.13136.
36. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30.
37. Veličković, P. and Blundell, C. (2021). Neural algorithmic reasoning. Patterns, 2(7):100273. [find](https://scholar.google.com/scholar?q=Velic%CC%8Ckovic%CC%81%2C+P%2E+and+Blundell%2C+C%2E+%282021%29%2E+Neural+algorithmic+reasoning%2E+Patterns%2C+2%287%29%3A100273%2E)
38. Wang, H., Yu, M., Guo, X., Das, R., Xiong, W., and Gao, T. (2019). Do multi-hop readers dream of reasoning chains? arXiv preprint arXiv:1910.14520. [find](https://scholar.google.com/scholar?q=Wang%2C+H%2E%2C+Yu%2C+M%2E%2C+Guo%2C+X%2E%2C+Das%2C+R%2E%2C+Xiong%2C+W%2E%2C+and+Gao%2C+T%2E+%282019%29%2E+Do+multi%2Dhop+readers+dream+of+reasoning+chains%3F+arXiv+preprint+arXiv%3A1910%2E14520%2E)
39. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., and Zhou, D. (2022a). [Rationale-augmented ensembles in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-rationale-augmented-ensembles-language-models/). arXiv preprint arXiv:2207.00747.
40. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., and Zhou, D. (2022b). [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171.
41. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022a). [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). arXiv preprint arXiv:2206.07682.
42. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. (2022b). [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). arXiv preprint arXiv:2201.11903.
43. Welleck, S., Liu, J., Bras, R. L., Hajishirzi, H., Choi, Y., and Cho, K. (2021). [Naturalproofs: Mathematical theorem proving in natural language](https://md.research.elamur.ai/free/en/papers/welleck-2021-naturalproofs-mathematical-theorem-proving-natural/). arXiv preprint arXiv:2104.01112.
44. Wu, Y., Jiang, A. Q., Li, W., Rabe, M. N., Staats, C., Jamnik, M., and Szegedy, C. (2022). [Autoformalization with large language models](https://md.research.elamur.ai/free/en/papers/wu-2022-autoformalization-large-language-models/). arXiv preprint arXiv:2205.12615.
45. Xhonneux, L.-P., Deac, A.-I., Veličković, P., and Tang, J. (2021). How to transfer algorithmic reasoning knowledge to learn new algorithms? Advances in Neural Information Processing Systems, 34:19500–19512. [find](https://scholar.google.com/scholar?q=Xhonneux%2C+L%2E%2DP%2E%2C+Deac%2C+A%2E%2DI%2E%2C+Velic%CC%8Ckovic%CC%81%2C+P%2E%2C+and+Tang%2C+J%2E+%282021%29%2E+How+to+transfer+algorithmic+reasoning+knowledge+to+learn+new+algorithms%3F+Advances+in+Neural+Information+Processing+Systems%2C+34%3A1950)
46. Xu, K., Li, J., Zhang, M., Du, S. S., Kawarabayashi, K.-i., and Jegelka, S. (2019). What can neural networks reason about? arXiv preprint arXiv:1905.13211. [find](https://scholar.google.com/scholar?q=Xu%2C+K%2E%2C+Li%2C+J%2E%2C+Zhang%2C+M%2E%2C+Du%2C+S%2E+S%2E%2C+Kawarabayashi%2C+K%2E%2Di%2E%2C+and+Jegelka%2C+S%2E+%282019%29%2E+What+can+neural+networks+reason+about%3F+arXiv+preprint+arXiv%3A1905%2E13211%2E)
47. Yan, Y., Swersky, K., Koutra, D., Ranganathan, P., and Hashemi, M. (2020). Neural execution engines: Learning to execute subroutines. Advances in Neural Information Processing Systems, 33:17298–17308. [find](https://scholar.google.com/scholar?q=Yan%2C+Y%2E%2C+Swersky%2C+K%2E%2C+Koutra%2C+D%2E%2C+Ranganathan%2C+P%2E%2C+and+Hashemi%2C+M%2E+%282020%29%2E+Neural+execution+engines%3A+Learning+to+execute+subroutines%2E+Advances+in+Neural+Information+Processing+Systems%2C+33%3A17298%E2%80%9317308%2E)
48. Zelikman, E., Wu, Y., and Goodman, N. D. (2022). [Star: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). arXiv preprint arXiv:2203.14465.
49. Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Bousquet, O., Le, Q., and Chi, E. (2022). [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/). arXiv preprint arXiv:2205.10625.

*Updated 2 November 2026*