---
title: "Solving Challenging Math Word Problems Using — Zhou 2023"
description: "Large language models still make many mistakes on competition-level mathematics, both in calculation and in multi-step reasoning."
authors: ["Aojun Zhou", "Ke Wang", "Zimu Lu", "Weikang Shi", "Sichun Luo", "Zipeng Qin", "Shaoqing Lu", "Anya Jia", "Linqi Song", "Mingjie Zhan", "Hongsheng Li"]
year: 2023
venue: "arXiv:2308.07921"
source: "https://arxiv.org/abs/2308.07921"
doi: "10.48550/arxiv.2308.07921"
arxiv: "2308.07921"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "https://arxiv.org/licenses/nonexclusive-distrib/1.0/license.html"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhou-2023-solving-challenging-math-word-problems/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhou-2023-solving-challenging-math-word-problems/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification

# Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification

**Authors:** [Aojun Zhou](https://md.research.elamur.ai/free/en/authors/aojun-zhou/), [Ke Wang](https://md.research.elamur.ai/free/en/authors/ke-wang/), [Zimu Lu](https://md.research.elamur.ai/free/en/authors/zimu-lu/), [Weikang Shi](https://md.research.elamur.ai/free/en/authors/weikang-shi/), [Sichun Luo](https://md.research.elamur.ai/free/en/authors/sichun-luo/), Zipeng Qin, Shaoqing Lu, Anya Jia, [Linqi Song](https://md.research.elamur.ai/free/en/authors/linqi-song/), [Mingjie Zhan](https://md.research.elamur.ai/free/en/authors/mingjie-zhan/), [Hongsheng Li](https://md.research.elamur.ai/free/en/authors/hongsheng-li/) · **Year:** 2023 · **Venue:** arXiv:2308.07921 · **Licence:** [arXiv — non-exclusive licence to distribute](https://arxiv.org/licenses/nonexclusive-distrib/1.0/license.html)

**Links:** [Publisher's page](https://arxiv.org/abs/2308.07921) · [PDF](https://arxiv.org/pdf/2308.07921v1) · [DOI 10.48550/arxiv.2308.07921](https://doi.org/10.48550/arxiv.2308.07921) · [arXiv 2308.07921](https://arxiv.org/abs/2308.07921) · [OpenAlex](https://openalex.org/W4385890378)

**Citations:** 16 citations (openalex-snapshot · 2026-10-14) · [93](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Large language models still make many mistakes on competition-level mathematics, both in calculation and in multi-step reasoning. This paper looks at why OpenAI's GPT-4 Code Interpreter (GPT4-Code) does so much better on such tasks. The authors ran pilot experiments that limit how often the model may run code: no code at all, code only once, or no limit. They found that the gains come mostly from writing code in small steps and from self-debugging, where the model revises its approach after seeing execution results. Building on this, they propose explicit code-based self-verification (CSV). This is a zero-shot prompt that asks the model to write extra code to check its own final answer and to fix the solution when the check fails. The verification outcome (True, Uncertain or False) is then used as a weight in majority voting across sampled solutions. They evaluate on MATH, GSM8K and MMLU-Math. The result is a new state of the art on MATH without any external verifier model, which shows that a model's own code execution can make its math reasoning more reliable.

## Key points
- On MATH, unmodified GPT4-Code reaches 69.7% zero-shot accuracy, compared with 42.2% for plain GPT-4. With CSV and verification-weighted voting, accuracy rises to 84.32%, while the abstract cites 53.9% as the earlier state of the art.
- The paper introduces Code Usage Frequency as a way to compare prompting strategies. Accuracy goes up as the model is allowed, or encouraged, to run code more often.
- In weighted majority voting, each candidate solution counts according to its self-verification state, so a solution checked as True carries more weight than one marked Uncertain or False.
- Unlike verifier approaches such as Cobbe et al. (2021) and Lightman et al. (2023), CSV needs no separate model and no fine-tuning. It is a prompt only.
- The authors release two instruction-following datasets, MATH-code and MMLU-Math-code, intended for fine-tuning open-source models such as LLaMA 2.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/)

## Cited by

Cited by

93 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 82 not yet.

- [Large Language Models for Mathematical Reasoning: Progresses and Challenges](https://md.research.elamur.ai/free/en/papers/ahn-2024-large-language-models-mathematical-reasoning/)
- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2025-towards-reasoning-era-survey-long/)
- [ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving](https://md.research.elamur.ai/free/en/papers/gou-2023-tora-tool-integrated-reasoning-agent/)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-cannot-self/)
- [A Survey on Large Language Models for Code Generation](https://md.research.elamur.ai/free/en/papers/jiang-2024-survey-large-language-models-code/)
- [GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers](https://md.research.elamur.ai/free/en/papers/li-2024-gsm-plus-comprehensive-benchmark-evaluating/)
- [ReFT: Reasoning with Reinforced Fine-Tuning](https://md.research.elamur.ai/free/en/papers/trung-2024-reft-reasoning-reinforced-fine-tuning/)
- [MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning](https://md.research.elamur.ai/free/en/papers/wang-2023-mathcoder-seamless-code-integration-llms/)
- [Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset](https://md.research.elamur.ai/free/en/papers/wang-2024-measuring-multimodal-mathematical-reasoning-math/)
- [MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://md.research.elamur.ai/free/en/papers/zhang-2024-mathverse-multi-modal-llm-truly/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhou2023solving,
  title = {{Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification}},
  author = {Zhou, Aojun and Wang, Ke and Lu, Zimu and Shi, Weikang and Luo, Sichun and Qin, Zipeng and Lu, Shaoqing and Jia, Anya and Song, Linqi and Zhan, Mingjie and Li, Hongsheng},
  year = {2023},
  doi = {10.48550/arxiv.2308.07921},
  eprint = {2308.07921},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.07921}
}
```

- [← previous SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization](https://md.research.elamur.ai/free/en/papers/kim-2023-soda-million-scale-dialogue-distillation/)
- [next → Some Considerations in the Construction of a Historical Language WordNet](https://md.research.elamur.ai/free/en/papers/khan-2023-some-considerations-construction-historical-language/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

29 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. [find](https://scholar.google.com/scholar?q=Rohan+Anil%2C+Andrew+M+Dai%2C+Orhan+Firat%2C+Melvin+Johnson%2C+Dmitry+Lepikhin%2C+Alexandre+Passos%2C+Siamak+Shakeri%2C+Emanuel+Taropa%2C+Paige+Bailey%2C+Zhifeng+Chen%2C+et+al%2E+Palm+2+technical+report%2E+arXiv+preprint+arX) [our card](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/)
2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+)
3. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks, 2022. [find](https://scholar.google.com/scholar?q=Wenhu+Chen%2C+Xueguang+Ma%2C+Xinyi+Wang%2C+and+William+W%2E+Cohen%2E+Program+of+thoughts+prompting%3A+Disentangling+computation+from+reasoning+for+numerical+reasoning+tasks%2C+2022%2E)
4. Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: A theorem-driven question answering dataset, 2023. [find](https://scholar.google.com/scholar?q=Wenhu+Chen%2C+Ming+Yin%2C+Max+Ku%2C+Pan+Lu%2C+Yixin+Wan%2C+Xueguang+Ma%2C+Jianyu+Xu%2C+Xinyi+Wang%2C+and+Tony+Xia%2E+Theoremqa%3A+A+theorem%2Ddriven+question+answering+dataset%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2023-theoremqa-theorem-driven-question-answering/)
5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [find](https://scholar.google.com/scholar?q=Karl+Cobbe%2C+Vineet+Kosaraju%2C+Mohammad+Bavarian%2C+Mark+Chen%2C+Heewoo+Jun%2C+Lukasz+Kaiser%2C+Matthias+Plappert%2C+Jerry+Tworek%2C+Jacob+Hilton%2C+Reiichiro+Nakano%2C+Christopher+Hesse%2C+and+John+Schulman%2E+Training+ve) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
6. Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting for multi-step reasoning. arXiv preprint arXiv:2210.00720, 2022. [find](https://scholar.google.com/scholar?q=Yao+Fu%2C+Hao+Peng%2C+Ashish+Sabharwal%2C+Peter+Clark%2C+and+Tushar+Khot%2E+Complexity%2Dbased+prompting+for+multi%2Dstep+reasoning%2E+arXiv+preprint+arXiv%3A2210%2E00720%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/fu-2022-complexity-based-prompting-multi-step/)
7. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pp. 10764–10799. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Pengfei+Liu%2C+Yiming+Yang%2C+Jamie+Callan%2C+and+Graham+Neubig%2E+Pal%3A+Program%2Daided+language+models%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+10764%E2%80%93)
8. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ArXiv, abs/2009.03300, 2020. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 221516475. [↗](https://api.semanticscholar.org/CorpusID:) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
9. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Collin+Burns%2C+Saurav+Kadavath%2C+Akul+Arora%2C+Steven+Basart%2C+Eric+Tang%2C+Dawn+Song%2C+and+Jacob+Steinhardt%2E+Measuring+mathematical+problem+solving+with+the+math+dataset%2E+arXiv+preprint+arXiv%3A) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/)
10. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. [find](https://scholar.google.com/scholar?q=Jordan+Hoffmann%2C+Sebastian+Borgeaud%2C+Arthur+Mensch%2C+Elena+Buchatskaya%2C+Trevor+Cai%2C+Eliza+Rutherford%2C+Diego+de+Las+Casas%2C+Lisa+Anne+Hendricks%2C+Johannes+Welbl%2C+Aidan+Clark%2C+et+al%2E+Training+compute%2Doptim)
11. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pp. 22199–22213, 2022. [find](https://scholar.google.com/scholar?q=Takeshi+Kojima%2C+Shixiang+%28Shane%29+Gu%2C+Machel+Reid%2C+Yutaka+Matsuo%2C+and+Yusuke+Iwasawa%2E+Large+language+models+are+zero%2Dshot+reasoners%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+35%2C+pp%2E) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)
12. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843–3857, 2022. [find](https://scholar.google.com/scholar?q=Aitor+Lewkowycz%2C+Anders+Andreassen%2C+David+Dohan%2C+Ethan+Dyer%2C+Henryk+Michalewski%2C+Vinay+Ramasesh%2C+Ambrose+Slone%2C+Cem+Anil%2C+Imanol+Schlag%2C+Theo+Gutman%2DSolo%2C+et+al%2E+Solving+quantitative+reasoning+problem) [our card](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/)
13. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5315– 5333, 2023. [find](https://scholar.google.com/scholar?q=Yifei+Li%2C+Zeqi+Lin%2C+Shizhuo+Zhang%2C+Qiang+Fu%2C+Bei+Chen%2C+Jian%2DGuang+Lou%2C+and+Weizhu+Chen%2E+Making+language+models+better+reasoners+with+step%2Daware+verifier%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+t) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-making-language-models-better-reasoners/)
14. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023. [find](https://scholar.google.com/scholar?q=Hunter+Lightman%2C+Vineet+Kosaraju%2C+Yura+Burda%2C+Harri+Edwards%2C+Bowen+Baker%2C+Teddy+Lee%2C+Jan+Leike%2C+John+Schulman%2C+Ilya+Sutskever%2C+and+Karl+Cobbe%2E+Let%E2%80%99s+verify+step+by+step%2E+arXiv+preprint+arXiv%3A2305%2E2005) [our card](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/)
15. Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209.14610, 2022. [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Liang+Qiu%2C+Kai%2DWei+Chang%2C+Ying+Nian+Wu%2C+Song%2DChun+Zhu%2C+Tanmay+Rajpurohit%2C+Peter+Clark%2C+and+Ashwin+Kalyan%2E+Dynamic+prompt+learning+via+policy+gradient+for+semi%2Dstructured+mathematical+reasoning) [our card](https://md.research.elamur.ai/free/en/papers/lu-2022-dynamic-prompt-learning-policy-gradient/)
16. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+et+al%2E+Self%2Drefine%3A+Iterative+refinement+with+self%2Dfeedbac)
17. A. Newell and H.A. Simon. Human Problem Solving. ACS symposium series. Prentice-Hall, 1972. ISBN 9780134454030. URL [https://books.google.com.hk/books?id=h03uAAAAMAAJ](https://books.google.com.hk/books?id=h03uAAAAMAAJ). [↗](https://books.google.com.hk/books?id=h03uAAAAMAAJ)
18. OpenAI. Gpt-4 technical report, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
19. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085, 2022. [find](https://scholar.google.com/scholar?q=Ross+Taylor%2C+Marcin+Kardas%2C+Guillem+Cucurull%2C+Thomas+Scialom%2C+Anthony+Hartshorn%2C+Elvis+Saravia%2C+Andrew+Poulton%2C+Viktor+Kerkez%2C+and+Robert+Stojnic%2E+Galactica%3A+A+large+language+model+for+science%2E+arXiv+) [our card](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/)
20. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtun)
21. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL [https://openreview.net/forum?id=…](https://openreview.net/forum?id=…) [↗](https://openreview.net/forum?id=…) [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/)
22. Yingxu Wang and Vincent Chiew. On the cognitive process of human problem solving. Cognitive Systems Research, 11(1):81–92, 2010. ISSN 1389-0417. [find](https://scholar.google.com/scholar?q=Yingxu+Wang+and+Vincent+Chiew%2E+On+the+cognitive+process+of+human+problem+solving%2E+Cognitive+Systems+Research%2C+11%281%29%3A81%E2%80%9392%2C+2010%2E+ISSN+1389%2D0417%2E)
23. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Proce… [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Maarten+Bosma%2C+brian+ichter%2C+Fei+Xia%2C+Ed+H%2E+Chi%2C+Quoc+V+Le%2C+and+Denny+Zhou%2E+Chain+of+thought+prompting+elicits+reasoning+in+large+language+models%2E+In+Alice+H%2E+) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
24. Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification, 2023. [find](https://scholar.google.com/scholar?q=Yixuan+Weng%2C+Minjun+Zhu%2C+Fei+Xia%2C+Bin+Li%2C+Shizhu+He%2C+Kang+Liu%2C+and+Jun+Zhao%2E+Large+language+models+are+better+reasoners+with+self%2Dverification%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/weng-2023-large-language-models-are-better/)
25. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. ArXiv, abs/2305.10601, 2023. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L%2E+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+ArXiv%2C+abs%2F2305%2E10601%2C)
26. Xu Zhao, Yuxi Xie, Kenji Kawaguchi, Junxian He, and Qizhe Xie. Automatic model selection with large language models for reasoning. arXiv preprint arXiv:2305.14333, 2023. [find](https://scholar.google.com/scholar?q=Xu+Zhao%2C+Yuxi+Xie%2C+Kenji+Kawaguchi%2C+Junxian+He%2C+and+Qizhe+Xie%2E+Automatic+model+selection+with+large+language+models+for+reasoning%2E+arXiv+preprint+arXiv%3A2305%2E14333%2C+2023%2E)
27. Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. Progressive-hint prompting improves reasoning in large language models. arXiv preprint arXiv:2304.09797, 2023. [find](https://scholar.google.com/scholar?q=Chuanyang+Zheng%2C+Zhengying+Liu%2C+Enze+Xie%2C+Zhenguo+Li%2C+and+Yu+Li%2E+Progressive%2Dhint+prompting+improves+reasoning+in+large+language+models%2E+arXiv+preprint+arXiv%3A2304%2E09797%2C+2023%2E)
28. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models, 2023. [find](https://scholar.google.com/scholar?q=Denny+Zhou%2C+Nathanael+Scha%CC%88rli%2C+Le+Hou%2C+Jason+Wei%2C+Nathan+Scales%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Claire+Cui%2C+Olivier+Bousquet%2C+Quoc+Le%2C+and+Ed+Chi%2E+Least%2Dto%2Dmost+prompting+enables+complex+reasoning+in+) [our card](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/)
29. A PPENDIX This Appendix contains two sections. The first section provides the experiment details, including the detailed experiment results on MATH and MMLU datasets. The second section presents some examples of GPT4-Code. [find](https://scholar.google.com/scholar?q=A+PPENDIX+This+Appendix+contains+two+sections%2E+The+first+section+provides+the+experiment+details%2C+including+the+detailed+experiment+results+on+MATH+and+MMLU+datasets%2E+The+second+section+presents+some+)