---
title: "SelfCheck — paper by Miao et al. (2023) — summary and ideas"
description: "Miao, Teh and Rainforth propose SelfCheck, a zero-shot method in which the LLM audits its own… 19 references, cited by 90 papers here."
authors: ["Ning Miao", "Yee Whye Teh", "Tom Rainforth"]
year: 2023
venue: "arXiv:2308.00436"
source: "https://arxiv.org/abs/2308.00436"
doi: "10.48550/arxiv.2308.00436"
arxiv: "2308.00436"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/miao-2023-selfcheck-using-llms-zero-shot/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/miao-2023-selfcheck-using-llms-zero-shot/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/) / SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning

# SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning

**Authors:** Ning Miao, [Yee Whye Teh](https://md.research.elamur.ai/free/en/authors/yee-whye-teh/) and Tom Rainforth · **Year:** 2023 · **Venue:** arXiv:2308.00436 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2308.00436) · [PDF](https://arxiv.org/pdf/2308.00436) · [DOI 10.48550/arxiv.2308.00436](https://doi.org/10.48550/arxiv.2308.00436) · [arXiv 2308.00436](https://arxiv.org/abs/2308.00436) · [OpenAlex](https://openalex.org/W4385963592)

**Citations:** 11 citations (OpenAlex, 2026-10-14) · [90](https://md.research.elamur.ai/free/en/papers/miao-2023-selfcheck-using-llms-zero-shot/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Chain-of-thought prompting lets LLMs solve problems step by step, but long reasoning chains often contain at least one faulty step that ruins the final answer. Earlier verifiers depended on extra resources: fine-tuned checker models, external tools or databases, or hand-written few-shot exemplars tied to one domain. Miao, Teh and Rainforth propose SelfCheck, a zero-shot method in which the LLM audits its own solutions without additional data or training. Asking the model outright whether a step is correct performs poorly. SelfCheck therefore splits the check of each step into separate LLM calls: identify the step's goal, gather the context it depends on, re-derive the step independently, and compare the new version with the original. The per-step verdicts are combined into a confidence score for the whole solution. These scores serve as weights when voting among several sampled solutions. On GSM8K, MathQA and MATH, this weighted voting beats plain majority voting and other baselines. Because SelfCheck needs no fine-tuning or crafted examples, it can be applied directly to reasoning tasks in other domains.

## Key points
- Verifying a step takes four phases: pulling out what the step aims to do, gathering the relevant information, producing the step anew, and comparing the results. The comparison stage labels the regenerated step as supporting, contradicting or unrelated to the original, which gives a step score of 1, −1 or 0.
- Step scores are aggregated into a solution-level confidence w ∈ [0, 1], which is used for weighted voting across multiple candidate solutions.
- Discarding low-confidence solutions lowers the share of incorrect ones by 9% on GSM8K, 22.8% on MathQA and 16.2% on MATH.
- Motivation: GPT-4 with CoT answers only 42.5% of MATH problems correctly. Directly asking an LLM to verify its answer approves it more than 90% of the time, whatever its actual correctness (finding by Ling et al., 2023).
- Code is released at github.com/NingMiao/SelfCheck. The approach is contrasted with Self-Verification (SV), Deductive Verification (DV) and trained verifiers such as a fine-tuned GPT-3 and a deberta-v3-large classifier.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Oxford](https://md.research.elamur.ai/free/en/universities/university-of-oxford/) — University, Oxford, United Kingdom · 122 works · 160,277 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Ning Miao, Yee Whye Teh and Tom Rainforth. 2023. SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning. arXiv:2308.00436.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{miao2023selfcheck,
  title = {{SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning}},
  author = {Ning Miao, Yee Whye Teh and Tom Rainforth},
  year = {2023},
  doi = {10.48550/arxiv.2308.00436},
  eprint = {2308.00436},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.00436}
}
```

- [← previous FireAct: Toward Language Agent Fine-tuning](https://md.research.elamur.ai/free/en/papers/chen-2023-fireact-toward-language-agent-fine/)
- [next → FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets](https://md.research.elamur.ai/free/en/papers/ye-2023-flask-fine-grained-language-model/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/)

## References

19 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-40B: an open large language model with state-of-the-art… [find](https://scholar.google.com/scholar?q=Ebtesam+Almazrouei%2C+Hamza+Alobeidli%2C+Abdulaziz+Alshamsi%2C+Alessandro+Cappelli%2C+Ruxandra+Cojocaru%2C+Merouane+Debbah%2C+Etienne+Goffinet%2C+Daniel+Heslow%2C+Julien+Launay%2C+Quentin+Malartic%2C+Badreddine+Noune%2C+Ba)
2. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. [MathQA: Towards interpretable math word problem solving with operation-based formalisms](https://md.research.elamur.ai/free/en/papers/amini-2019-mathqa-interpretable-math-word-problem/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lin…
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
4. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712, 2023.
5. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311, 2022.
6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
7. Google. Palm 2 technical report. arXiv preprint arXiv:2303.[08774, 2023](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
8. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. [Deberta: Decoding-enhanced bert with disentangled attention](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/). In International Conference on Learning Representations, 2020.
9. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
10. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. On the advance of making language models better reasoners. arXiv preprint arXiv:2206.02336, 2022. — [arXiv:2206.02336](https://arxiv.org/abs/2206.02336) [find](https://scholar.google.com/scholar?q=Yifei+Li%2C+Zeqi+Lin%2C+Shizhuo+Zhang%2C+Qiang+Fu%2C+Bei+Chen%2C+Jian%2DGuang+Lou%2C+and+Weizhu+Chen%2E+On+the+advance+of+making+language+models+better+reasoners%2E+arXiv+preprint+arXiv%3A2206%2E02336%2C+2022%2E+%E2%80%94+%5BarXiv%3A2206%2E)
11. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.[20050, 2023](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/).
12. Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, and Hao Su. [Deductive verification of chain-of-thought reasoning](https://md.research.elamur.ai/free/en/papers/ling-2023-deductive-verification-chain-thought-reasoning/). arXiv preprint arXiv:2306.03872, 2023.
13. Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. [Faithful chain-of-thought reasoning](https://md.research.elamur.ai/free/en/papers/lyu-2023-faithful-chain-thought-reasoning/). arXiv preprint arXiv:2301.13379, 2023.
14. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
15. Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. [Check your facts and try again: Improving large language models with external knowledge and automated feedback](https://md.research.elamur.ai/free/en/papers/peng-2023-check-facts-try-again-improving/). arXiv preprint arXiv:2302.12813, 2023.
16. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
17. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations, 2022.
18. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
19. Yixuan Weng, Minjun Zhu, Shizhu He, Kang Liu, and Jun Zhao. Large language models are reasoners with self-verification. arXiv preprint arXiv:2212.[09561, 2022](https://md.research.elamur.ai/free/en/papers/weng-2023-large-language-models-are-better/).

*Updated 2 November 2026*