---
title: "Metacognitive Capabilities of LLMs: An Exploration in"
description: "They test this on math reasoning. 42 references, cited by 40 papers here. Among them: 5."
authors: ["Aniket Didolkar", "Anirudh Goyal", "Nan Rosemary Ke", "Siyuan Guo", "Michal Valko", "Timothy Lillicrap", "Danilo Rezende", "Yoshua Bengio", "Michael Mozer", "Sanjeev Arora"]
year: 2024
venue: "arXiv:2405.12205"
source: "https://arxiv.org/abs/2405.12205"
doi: "10.48550/arxiv.2405.12205"
arxiv: "2405.12205"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/didolkar-2024-metacognitive-capabilities-llms-exploration-mathematical/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/didolkar-2024-metacognitive-capabilities-llms-exploration-mathematical/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/) / Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving

# Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving

**Authors:** [Aniket Didolkar](https://md.research.elamur.ai/free/en/authors/aniket-didolkar/), [Anirudh Goyal](https://md.research.elamur.ai/free/en/authors/anirudh-goyal/), [Nan Rosemary Ke](https://md.research.elamur.ai/free/en/authors/nan-rosemary-ke/), [Siyuan Guo](https://md.research.elamur.ai/free/en/authors/siyuan-guo/), [Michal Valko](https://md.research.elamur.ai/free/en/authors/michal-valko/), [Timothy Lillicrap](https://md.research.elamur.ai/free/en/authors/timothy-lillicrap/), [Danilo Rezende](https://md.research.elamur.ai/free/en/authors/danilo-jimenez-rezende/), [Yoshua Bengio](https://md.research.elamur.ai/free/en/authors/yoshua-bengio/), [Michael Mozer](https://md.research.elamur.ai/free/en/authors/michael-c-mozer/) and [Sanjeev Arora](https://md.research.elamur.ai/free/en/authors/sanjeev-arora/) · **Year:** 2024 · **Venue:** arXiv:2405.12205 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2405.12205) · [PDF](https://arxiv.org/pdf/2405.12205) · [DOI 10.48550/arxiv.2405.12205](https://doi.org/10.48550/arxiv.2405.12205) · [arXiv 2405.12205](https://arxiv.org/abs/2405.12205) · [OpenAlex](https://openalex.org/W4398192835)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [40](https://md.research.elamur.ai/free/en/papers/didolkar-2024-metacognitive-capabilities-llms-exploration-mathematical/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

The authors ask whether large language models have something like metacognitive knowledge, meaning an awareness of which skills a task calls for, and whether that awareness can be used to improve their performance. They test this on math reasoning. GPT-4 is first prompted to tag every training problem in GSM8K and MATH with a fine-grained skill name. It then merges these overly specific tags into a smaller set of broader, human-readable skill categories and relabels the training data with them. The result is a Skill Exemplar Repository: worked question–answer pairs grouped by skill. At test time, a model (GPT-4 or a different LLM) chooses the matching skill from the list for each new problem and receives randomly drawn exemplars of that skill as in-context demonstrations. This skill-guided example selection beats Chain-of-Thought and code-based PAL prompting. The gains carry over to weaker models and to other math datasets. The work suggests that LLMs' self-assigned skill labels are meaningful, and that the approach could extend beyond math.

## Key points

- Skill discovery used GPT-4-0613. About 7,000 GSM8K problems produced roughly 500 raw skill names, which clustering reduced to 22. About 7,500 MATH problems produced around 5,000 raw names, reduced to 117 coarse skills.
- On MATH, skill-based exemplar selection improved accuracy by 11.6% over CoT. With code-generating solutions, it improved by 7.52% over the PAL baseline.
- The MATH repository also helped weaker LLMs. The GSM8K repository improved in-context results on other math datasets.
- An error analysis (also classified by GPT-4) compared the method with topic-based example selection. The skill-based method showed higher success rates for applying the main skill, applying secondary skills, and performing calculations.
- One limitation is that each question receives only one skill, while many problems need several secondary skills. The authors propose hierarchical or multi-skill labeling and fine-tuning on skills as future work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Cambridge](https://md.research.elamur.ai/free/en/universities/university-of-cambridge/) — University, Cambridge, United Kingdom · 287 works · 125,174 citations
- [Princeton University](https://md.research.elamur.ai/free/en/universities/princeton-university/) — University, Princeton, United States · 137 works · 53,849 citations
- [Université de Montréal](https://md.research.elamur.ai/free/en/universities/universite-de-montreal/) — University, Montreal, Canada · 96 works · 92,262 citations
- [Google DeepMind](https://md.research.elamur.ai/free/en/companies/google-deepmind/) — Company, London, United Kingdom · 82 works · 84,829 citations
- [Institut national de recherche en sciences et technologies du numérique](https://md.research.elamur.ai/free/en/labs/institut-national-de-recherche-en-sciences-et-technologies-du-numerique/) — Research institute, Le Chesnay-Rocquencourt, France · 90 works · 26,097 citations
- [Mila - Quebec Artificial Intelligence Institute](https://md.research.elamur.ai/free/en/labs/mila-quebec-artificial-intelligence-institute/) — Research institute, Montreal, Canada · 73 works · 14,704 citations
- [Institut de Recherche en Informatique et Systèmes Aléatoires](https://md.research.elamur.ai/free/en/labs/institut-de-recherche-en-informatique-et-systemes-aleatoires/) — Research institute, Rennes, France · 9 works · 2,021 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [➗ Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/) · 233 works · [➗ Applied Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/applied-mathematics/) · 73 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,948 works

## Cite

Cite as: Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy Lillicrap, Danilo Rezende, Yoshua Bengio, Michael Mozer and Sanjeev Arora. 2024. Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving. arXiv:2405.12205.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{didolkar2024metacognitive,
  title = {{Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving}},
  author = {Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy Lillicrap, Danilo Rezende, Yoshua Bengio, Michael Mozer and Sanjeev Arora},
  year = {2024},
  doi = {10.48550/arxiv.2405.12205},
  eprint = {2405.12205},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.12205}
}
```

- [← previous LitLLM: A Toolkit for Scientific Literature Review](https://md.research.elamur.ai/free/en/papers/agarwal-2024-litllm-toolkit-scientific-literature-review/)
- [next → Reliable, Adaptable, and Attributable Language Models with Retrieval](https://md.research.elamur.ai/free/en/papers/asai-2024-reliable-adaptable-attributable-language-models/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/)

## References

42 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
2. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
3. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403, 2023.
4. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
5. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim-othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
6. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
7. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
8. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. [Mathematical discoveries from program search with large language models](https://md.research.elamur.ai/free/en/papers/paredes-2023-mathematical-discoveries-program-search-large/). Nature, pages 1–3, 2023.
9. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. [Large language models as optimizers](https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-optimizers/). arXiv preprint arXiv:2309.03409, 2023.
10. Mengzhou Hu, Sahar Alkhairy, Ingoo Lee, Rudolf T Pillich, Robin Bachelder, Trey Ideker, and Dexter Pratt. Evaluation of large language models for discovery of gene set function. Research Square, 2023. [find](https://scholar.google.com/scholar?q=Mengzhou+Hu%2C+Sahar+Alkhairy%2C+Ingoo+Lee%2C+Rudolf+T+Pillich%2C+Robin+Bachelder%2C+Trey+Ideker%2C+and+Dexter+Pratt%2E+Evaluation+of+large+language+models+for+discovery+of+gene+set+function%2E+Research+Square%2C+2023%2E)
11. Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. [Solving olympiad geometry without human demonstrations](https://md.research.elamur.ai/free/en/papers/trinh-2024-solving-olympiad-geometry-without-human/). Nature, 625(7995):476–482, 2024.
12. Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. [Learning to solve arithmetic word problems with verb categorization](https://md.research.elamur.ai/free/en/papers/hosseini-2014-learning-solve-arithmetic-word-problems/). In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 523–533, 2014.
13. Subhro Roy, Tim Vieira, and Dan Roth. Reasoning about quantities in natural language. Transactions of the Association for Computational Linguistics, 3:1–13, 2015. [find](https://scholar.google.com/scholar?q=Subhro+Roy%2C+Tim+Vieira%2C+and+Dan+Roth%2E+Reasoning+about+quantities+in+natural+language%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+3%3A1%E2%80%9313%2C+2015%2E)
14. Subhro Roy and Dan Roth. [Solving general arithmetic word problems](https://md.research.elamur.ai/free/en/papers/roy-2015-solving-general-arithmetic-word-problems/). arXiv preprint arXiv:1608.01413, 2016.
15. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. [Are nlp models really able to solve simple math word problems?](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/) arXiv preprint arXiv:2103.07191, 2021.
16. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/), 2021.
17. Joy He-Yueya, Gabriel Poesia, Rose E Wang, and Noah D Goodman. [Solving math word problems by combining language models with symbolic solvers](https://md.research.elamur.ai/free/en/papers/yueya-2023-solving-math-word-problems-combining/). arXiv preprint arXiv:2304.09102, 2023.
18. JH Flavell. Metacognitive aspects of problem solving. In The Nature of Intelligence. Routledge, 1976. [find](https://scholar.google.com/scholar?q=JH+Flavell%2E+Metacognitive+aspects+of+problem+solving%2E+In+The+Nature+of+Intelligence%2E+Routledge%2C+1976%2E)
19. A. Corbett, K. Koedinger, and J. Anderson. Intelligent tutoring systems. In M. Helander T. Landauer and P. Prabhu, editors, Handbook of Human Computer Interaction, pages 849–874. Elsevier Science, Amsterdam, 1997. [find](https://scholar.google.com/scholar?q=A%2E+Corbett%2C+K%2E+Koedinger%2C+and+J%2E+Anderson%2E+Intelligent+tutoring+systems%2E+In+M%2E+Helander+T%2E+Landauer+and+P%2E+Prabhu%2C+editors%2C+Handbook+of+Human+Computer+Interaction%2C+pages+849%E2%80%93874%2E+Elsevier+Science%2C+Ams)
20. K. Koedinger, A. Corbett, and C. Perfetti. The knowledge-learning-instruction framework: Bridging the science-practice chasm to enhance robust student learning, 2012. — [The Knowledge‐Learning‐Instruction Framework: Bridging the Science‐Practice Chasm to Enhance Robust Student Learning](https://md.research.elamur.ai/free/en/papers/koedinger-2012-knowledge-learning-instruction-framework-bridging/)
21. Jason Wei, Xuezhi Wang, Qixuan Liu, Bingtian Yang, Xinchi Dong, Huang Huang, and William Wang. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). arXiv, abs/2201.11903, 2022.
22. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). Advances in neural information processing systems, 35:22199–22213, 2022.
23. Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. [How to train data-efficient llms](https://md.research.elamur.ai/free/en/papers/sachdeva-2024-train-data-efficient-llms/). arXiv preprint arXiv:2402.09668, 2024.
24. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764–10799. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Pengfei+Liu%2C+Yiming+Yang%2C+Jamie+Callan%2C+and+Graham+Neubig%2E+Pal%3A+Program%2Daided+language+models%2E+In+International+Conference+on+Machine+Learning%2C+pages+1076)
25. H. Cen, K. Koedinger, and B. Junker. Learning factors analysis: A general method for cognitive model evaluation and improvement. In M. Ikeda, K. Ashley, and T. Chan, editors, Intelligent Tutoring Systems (volume 4053 of Lec. Notes in Comp. Sci.), pages 164–175. 2006. — [Learning Factors Analysis – A General Method for Cognitive Model Evaluation and Improvement](https://md.research.elamur.ai/free/en/papers/cen-2006-learning-factors-analysis-general-method/)
26. Robert V Lindsey, Mohammad Khajah, and Michael C Mozer. Automatic discovery of cognitive skills to improve the prediction of student learning. Advances in neural information processing systems, 27, 2014. [find](https://scholar.google.com/scholar?q=Robert+V+Lindsey%2C+Mohammad+Khajah%2C+and+Michael+C+Mozer%2E+Automatic+discovery+of+cognitive+skills+to+improve+the+prediction+of+student+learning%2E+Advances+in+neural+information+processing+systems%2C+27%2C+20)
27. Emma Brunskill. Estimating prerequisite structure from noisy data. In Educational Data Mining, pages 217–222, 2011. [find](https://scholar.google.com/scholar?q=Emma+Brunskill%2E+Estimating+prerequisite+structure+from+noisy+data%2E+In+Educational+Data+Mining%2C+pages+217%E2%80%93222%2C+2011%2E)
28. Chen Liang, Jianbo Ye, Shuting Wang, Bart Pursel, and C Lee Giles. Investigating active learning for concept prerequisite learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. [find](https://scholar.google.com/scholar?q=Chen+Liang%2C+Jianbo+Ye%2C+Shuting+Wang%2C+Bart+Pursel%2C+and+C+Lee+Giles%2E+Investigating+active+learning+for+concept+prerequisite+learning%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+vo)
29. Liangming Pan, Chengjiang Li, Juanzi Li, and Jie Tang. Prerequisite relation learning for concepts in moocs. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1447–1456, 2017. [find](https://scholar.google.com/scholar?q=Liangming+Pan%2C+Chengjiang+Li%2C+Juanzi+Li%2C+and+Jie+Tang%2E+Prerequisite+relation+learning+for+concepts+in+moocs%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+)
30. Mayee F. Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. Skill-it! a data-driven skills framework for understanding and training language models, 2023. [find](https://scholar.google.com/scholar?q=Mayee+F%2E+Chen%2C+Nicholas+Roberts%2C+Kush+Bhatia%2C+Jue+Wang%2C+Ce+Zhang%2C+Frederic+Sala%2C+and+Christopher+Re%CC%81%2E+Skill%2Dit%21+a+data%2Ddriven+skills+framework+for+understanding+and+training+language+models%2C+2023%2E)
31. Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen, Anirudh Goyal, and Sanjeev Arora. Skill-mix: a flexible and expandable family of evaluations for ai models, 2023. [find](https://scholar.google.com/scholar?q=Dingli+Yu%2C+Simran+Kaur%2C+Arushi+Gupta%2C+Jonah+Brown%2DCohen%2C+Anirudh+Goyal%2C+and+Sanjeev+Arora%2E+Skill%2Dmix%3A+a+flexible+and+expandable+family+of+evaluations+for+ai+models%2C+2023%2E)
32. Sanjeev Arora and Anirudh Goyal. A theory for emergence of complex skills in language models, 2023. [find](https://scholar.google.com/scholar?q=Sanjeev+Arora+and+Anirudh+Goyal%2E+A+theory+for+emergence+of+complex+skills+in+language+models%2C+2023%2E)
33. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/), 2021.
34. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
35. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/), 2023.
36. Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. [Complexity-based prompting for multi-step reasoning](https://md.research.elamur.ai/free/en/papers/fu-2022-complexity-based-prompting-multi-step/), 2023.
37. Zifan Xu, Haozhu Wang, Dmitriy Bespalov, Peter Stone, and Yanjun Qi. Latent skill discovery for chain-of-thought reasoning, 2023. [find](https://scholar.google.com/scholar?q=Zifan+Xu%2C+Haozhu+Wang%2C+Dmitriy+Bespalov%2C+Peter+Stone%2C+and+Yanjun+Qi%2E+Latent+skill+discovery+for+chain%2Dof%2Dthought+reasoning%2C+2023%2E)
38. Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing english math word problem solvers, 2021. [find](https://scholar.google.com/scholar?q=Shen%2DYun+Miao%2C+Chao%2DChun+Liang%2C+and+Keh%2DYih+Su%2E+A+diverse+corpus+for+evaluating+and+developing+english+math+word+problem+solvers%2C+2021%2E)
39. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Kevin Knight, Ani Nenkova, and Owen Rambow, editors, Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Huma… [find](https://scholar.google.com/scholar?q=Rik+Koncel%2DKedziorski%2C+Subhro+Roy%2C+Aida+Amini%2C+Nate+Kushman%2C+and+Hannaneh+Hajishirzi%2E+MAWPS%3A+A+math+word+problem+repository%2E+In+Kevin+Knight%2C+Ani+Nenkova%2C+and+Owen+Rambow%2C+editors%2C+Proceedings+of+the+)
40. Yifan Zhang, Jingqin Yang, Yang Yuan, and Andrew Chi-Chih Yao. [Cumulative reasoning with large language models](https://md.research.elamur.ai/free/en/papers/zhang-2023-cumulative-reasoning-large-language-models/), 2023.
41. James Zhao, Yuxi Xie, Kenji Kawaguchi, Junxian He, and Michael Xie. Automatic model selection with large language models for reasoning. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 758–783, Singapore, December 202… [find](https://scholar.google.com/scholar?q=James+Zhao%2C+Yuxi+Xie%2C+Kenji+Kawaguchi%2C+Junxian+He%2C+and+Michael+Xie%2E+Automatic+model+selection+with+large+language+models+for+reasoning%2E+In+Houda+Bouamor%2C+Juan+Pino%2C+and+Kalika+Bali%2C+editors%2C+Findings+)
42. Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. [Progressive-hint prompting improves reasoning in large language models](https://md.research.elamur.ai/free/en/papers/zheng-2023-progressive-hint-prompting-improves-reasoning/), 2023. Dataset GSM8K Topic - Pre-Algebra MATH Algebra Inter-Algebra Geometry Number Theory Precalculus Probability

*Updated 3 November 2026*