---
title: "LLMs cannot find reasoning errors, but can correct them given"
description: "Self-correction in LLMs often fails on reasoning tasks and can even turn right answers into wrong ones. 33 references, cited by 56 papers here."
authors: ["Gladys Tyen", "Hassan Mansoor", "Victor Carbune", "Peter Chen", "Tony Mak"]
year: 2024
venue: "Findings of the Association for Computational Linguistics: ACL 2024"
source: "https://aclanthology.org/2024.findings-acl.826/"
doi: "10.18653/v1/2024.findings-acl.826"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tyen-2024-llms-cannot-find-reasoning-errors/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tyen-2024-llms-cannot-find-reasoning-errors/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / LLMs cannot find reasoning errors, but can correct them given the error location

# LLMs cannot find reasoning errors, but can correct them given the error location

**Authors:** Gladys Tyen, Hassan Mansoor, Victor Carbune, [Peter Chen](https://md.research.elamur.ai/free/en/authors/peter-chen/) and Tony Mak · **Year:** 2024 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2024](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2024](https://md.research.elamur.ai/free/en/venues/findings/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.findings-acl.826/) · [PDF](https://aclanthology.org/2024.findings-acl.826.pdf) · [DOI 10.18653/v1/2024.findings-acl.826](https://doi.org/10.18653/v1/2024.findings-acl.826) · [OpenAlex](https://openalex.org/W4402670356)

**Citations:** 29 citations (OpenAlex, 2026-10-14) · [56](https://md.research.elamur.ai/free/en/papers/tyen-2024-llms-cannot-find-reasoning-errors/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Self-correction in LLMs often fails on reasoning tasks and can even turn right answers into wrong ones. The authors split self-correction into two stages, locating the error and fixing it, to find out which stage is the bottleneck. They built BIG-Bench Mistake, a dataset of Chain-of-Thought traces produced by PaLM 2 Unicorn on five BIG-bench tasks. Each trace is labelled with the step where the first logical error appears, or marked as error-free. They then tested GPT-4-Turbo, GPT-4, GPT-3.5-Turbo, Gemini Pro and PaLM 2 using three few-shot prompting strategies, and all of these models were poor at pinpointing mistakes. When the true error position was supplied through a backtracking procedure, however, the models repaired faulty traces and gained accuracy on every task, while traces that were already correct were barely affected. A small classifier trained on out-of-domain data located errors better than prompting a large model. This suggests that dedicated mistake detectors could make self-correction practical.

## Key points
- BIG-Bench Mistake contains 2186 CoT traces drawn from five tasks: multi-step arithmetic, Dyck languages, logical deduction, word sorting and tracking shuffled objects. It is the first dataset of its type that covers more than mathematics, and it is released on GitHub (WHGTyen/BIG-Bench-Mistake).
- Four tasks were labelled by at least three non-expert annotators each, on 300 sampled traces per task (85% with wrong final answers). Agreement measured by Krippendorff's α was between 0.979 and 0.998. Dyck languages were labelled mostly automatically through pattern matching.
- Trace-level prompting, step-level prompting and CoT step-level prompting all gave weak mistake-location accuracy, even though the errors are objective.
- Providing oracle error locations for backtracking raised downstream accuracy on all five tasks.
- A classifier based on the much smaller PaLM 2 Otter, trained with the target task held out, outperformed 3-shot PaLM 2 Unicorn on four of five tasks. The gains were +22.59 on Dyck languages and +19.67 on tracking shuffled objects, with a small loss of −0.67 on logical deduction. Its accuracy is still below what effective backtracking needs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 353 works · 364,806 citations
- [University of Cambridge](https://md.research.elamur.ai/free/en/universities/university-of-cambridge/) — University, Cambridge, United Kingdom · 155 works · 115,562 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Gladys Tyen, Hassan Mansoor, Victor Carbune, Peter Chen and Tony Mak. 2024. LLMs cannot find reasoning errors, but can correct them given the error location. Findings of the Association for Computational Linguistics: ACL 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tyen2024llms,
  title = {{LLMs cannot find reasoning errors, but can correct them given the error location}},
  author = {Gladys Tyen, Hassan Mansoor, Victor Carbune, Peter Chen and Tony Mak},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.826},
  url = {https://doi.org/10.18653/v1/2024.findings-acl.826}
}
```

- [← previous LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-Training](https://md.research.elamur.ai/free/en/papers/zhu-2024-llama-moe-building-mixture-experts/)
- [next → SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning](https://md.research.elamur.ai/free/en/papers/wang-2024-seaeval-multilingual-foundation-models-cross/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

33 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403.
2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+2022a%2E+Training+a+helpful+and+harmless+assistant+with+rei)
3. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022b. [Constitutional AI: Harmlessness from AI feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073.
4. Angelica Chen, Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Samuel R. Bowman, Kyunghyun Cho, and Ethan Perez. 2024a. Learning from natural language feedback. Transactions on Machine Learning Research. [find](https://scholar.google.com/scholar?q=Angelica+Chen%2C+J%C3%A9r%C3%A9my+Scheurer%2C+Jon+Ander+Campos%2C+Tomasz+Korbak%2C+Jun+Shern+Chan%2C+Samuel+R%2E+Bowman%2C+Kyunghyun+Cho%2C+and+Ethan+Perez%2E+2024a%2E+Learning+from+natural+language+feedback%2E+Transactions+on+Machi)
5. Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2023a. Codet: Code generation with generated tests. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Bei+Chen%2C+Fengji+Zhang%2C+Anh+Nguyen%2C+Daoguang+Zan%2C+Zeqi+Lin%2C+Jian%2DGuang+Lou%2C+and+Weizhu+Chen%2E+2023a%2E+Codet%3A+Code+generation+with+generated+tests%2E+In+The+Eleventh+International+Conference+on+Learning+Re)
6. Pinzhen Chen, Zhicheng Guo, Barry Haddow, and Kenneth Heafield. 2023b. [Iterative translation refinement with large language models](https://md.research.elamur.ai/free/en/papers/chen-2024-iterative-translation-refinement-large-language/). arXiv preprint arXiv:2306.03856.
7. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2024b. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Maxwell+Lin%2C+Nathanael+Sch%C3%A4rli%2C+and+Denny+Zhou%2E+2024b%2E+Teaching+large+language+models+to+self%2Ddebug%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2E)
8. Bhavana Dalvi Mishra, Oyvind Tafjord, and Peter Clark. 2022. Towards teachable reasoning systems: Using a dynamic memory of user feedback for continual system improvement. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9465–9480, Abu Dhabi, United Ar… [find](https://scholar.google.com/scholar?q=Bhavana+Dalvi+Mishra%2C+Oyvind+Tafjord%2C+and+Peter+Clark%2E+2022%2E+Towards+teachable+reasoning+systems%3A+Using+a+dynamic+memory+of+user+feedback+for+continual+system+improvement%2E+In+Proceedings+of+the+2022+C)
9. Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I Liao, Kamilė Lukošiūtė, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. 2023. [The capacity for moral self-correction in large language models](https://md.research.elamur.ai/free/en/papers/ganguli-2023-capacity-moral-self-correction-large/). arXiv preprint arXiv:2302.07459.
10. Andrew F Hayes and Klaus Krippendorff. 2007. Answering the call for a standard reliability measure for coding data. Communication methods and measures, 1(1):77–89. [find](https://scholar.google.com/scholar?q=Andrew+F+Hayes+and+Klaus+Krippendorff%2E+2007%2E+Answering+the+call+for+a+standard+reliability+measure+for+coding+data%2E+Communication+methods+and+measures%2C+1%281%29%3A77%E2%80%9389%2E)
11. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. [Large language models cannot self-correct reasoning yet](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-cannot-self/). arXiv preprint arXiv:2310.01798.
12. Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023. [Language models can solve computer tasks](https://md.research.elamur.ai/free/en/papers/kim-2023-language-models-solve-computer-tasks/). arXiv preprint arXiv:2303.17491.
13. Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. 2023. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267. [find](https://scholar.google.com/scholar?q=Harrison+Lee%2C+Samrat+Phatale%2C+Hassan+Mansoor%2C+Kellie+Lu%2C+Thomas+Mesnard%2C+Colton+Bishop%2C+Victor+Carbune%2C+and+Abhinav+Rastogi%2E+2023%2E+Rlaif%3A+Scaling+reinforcement+learning+from+human+feedback+with+ai+fee)
14. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). arXiv preprint arXiv:2305.20050.
15. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+et+al%2E+2023%2E+Self%2Drefine%3A+Iterative+refinement+with+self%2Df)
16. Aman Madaan, Niket Tandon, Dheeraj Rajagopal, Peter Clark, Yiming Yang, and Eduard Hovy. 2021. Think about it! improving defeasible reasoning by first modeling the question scenario. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6291–6310, Online an… [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Dheeraj+Rajagopal%2C+Peter+Clark%2C+Yiming+Yang%2C+and+Eduard+Hovy%2E+2021%2E+Think+about+it%21+improving+defeasible+reasoning+by+first+modeling+the+question+scenario%2E+In+Proceedings+of)
17. Ning Miao, Yee Whye Teh, and Tom Rainforth. 2024. Selfcheck: Using llms to zero-shot check their own step-by-step reasoning. In ICLR 2024. [find](https://scholar.google.com/scholar?q=Ning+Miao%2C+Yee+Whye+Teh%2C+and+Tom+Rainforth%2E+2024%2E+Selfcheck%3A+Using+llms+to+zero%2Dshot+check+their+own+step%2Dby%2Dstep+reasoning%2E+In+ICLR+2024%2E)
18. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744.
19. Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2023. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. arXiv preprint arXiv:2308.03188. [find](https://scholar.google.com/scholar?q=Liangming+Pan%2C+Michael+Saxon%2C+Wenda+Xu%2C+Deepak+Nathani%2C+Xinyi+Wang%2C+and+William+Yang+Wang%2E+2023%2E+Automatically+correcting+large+language+models%3A+Surveying+the+landscape+of+diverse+self%2Dcorrection+stra)
20. Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023. [Refiner: Reasoning feedback on intermediate representations](https://md.research.elamur.ai/free/en/papers/paul-2023-refiner-reasoning-feedback-intermediate-representations/). arXiv preprint arXiv:2304.01904.
21. William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. [Self-critiquing models for assisting human evaluators](https://md.research.elamur.ai/free/en/papers/saunders-2022-self-critiquing-models-assisting-human/). arXiv preprint arXiv:2206.05802.
22. Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. [Reflexion: Language agents with verbal reinforcement learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/).
23. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). arXiv preprint arXiv:2206.04615.
24. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2023. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). Transactions on Machine Learning…
25. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. Big-bench-hard/cot-prompts. [https://github.com/suzgunmirac/](https://github.com/suzgunmirac/) BIG-Bench-Hard/tree/main/cot-prompts. Accessed: 2023-10-31. [↗](https://github.com/suzgunmirac/)
26. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In ICLR 2023.
27. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc.
28. Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2023. [Generating sequences by learning to self-correct](https://md.research.elamur.ai/free/en/papers/welleck-2022-generating-sequences-learning-self-correct/). In ICLR 2023.
29. Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. 2023. [Large language models are better reasoners with self-verification](https://md.research.elamur.ai/free/en/papers/weng-2023-large-language-models-are-better/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2550–2575, Singapore. Association for Computat…
30. Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, and Qizhe Xie. 2023. [Self-evaluation guided beam search for reasoning](https://md.research.elamur.ai/free/en/papers/xie-2023-self-evaluation-guided-beam-search/). In Thirty-seventh Conference on Neural Information Processing Systems.
31. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. [React: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). arXiv preprint arXiv:2210.03629.
32. Michihiro Yasunaga and Percy Liang. 2020. Graph-based, self-supervised program repair from diagnostic feedback. In International Conference on Machine Learning, pages 10799–10808. PMLR. [find](https://scholar.google.com/scholar?q=Michihiro+Yasunaga+and+Percy+Liang%2E+2020%2E+Graph%2Dbased%2C+self%2Dsupervised+program+repair+from+diagnostic+feedback%2E+In+International+Conference+on+Machine+Learning%2C+pages+10799%E2%80%9310808%2E+PMLR%2E)
33. Weizhe Yuan, Kyunghyun Cho, and Jason Weston. 2024. System-level natural language feedback. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2773–2789, St. Julian’s, Malta. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Weizhe+Yuan%2C+Kyunghyun+Cho%2C+and+Jason+Weston%2E+2024%2E+System%2Dlevel+natural+language+feedback%2E+In+Proceedings+of+the+18th+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Linguisti)

*Updated 24 October 2026*