---
title: "Can Large Language Models Detect Errors in Long"
description: "Reasoning models in the o1 style produce very long chains of thought (CoT). 45 references, cited by 26 papers here."
authors: ["Yancheng He", "Shilong Li", "Jiaheng Liu", "Weixun Wang", "Xingyuan Bu", "Ge Zhang", "Z.Y. Peng", "Zhaoxiang Zhang", "Zhicheng Zheng", "Wenbo Su", "Bo Zheng"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.905/"
doi: "10.18653/v1/2025.acl-long.905"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/he-2025-can-large-language-models-detect/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/he-2025-can-large-language-models-detect/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/) / Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?

# Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?

**Authors:** [Yancheng He](https://md.research.elamur.ai/free/en/authors/yancheng-he/), [Shilong Li](https://md.research.elamur.ai/free/en/authors/shilong-li/), [Jiaheng Liu](https://md.research.elamur.ai/free/en/authors/jiaheng-liu/), [Weixun Wang](https://md.research.elamur.ai/free/en/authors/weixun-wang/), [Xingyuan Bu](https://md.research.elamur.ai/free/en/authors/xingyuan-bu/), [Ge Zhang](https://md.research.elamur.ai/free/en/authors/ge-zhang/), [Z.Y. Peng](https://md.research.elamur.ai/free/en/authors/zhongyuan-peng/), [Zhaoxiang Zhang](https://md.research.elamur.ai/free/en/authors/zhaoxiang-zhang/), Zhicheng Zheng, [Wenbo Su](https://md.research.elamur.ai/free/en/authors/wenbo-su/) and [Bo Zheng](https://md.research.elamur.ai/free/en/authors/bo-zheng/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.905/) · [PDF](https://aclanthology.org/2025.acl-long.905.pdf) · [DOI 10.18653/v1/2025.acl-long.905](https://doi.org/10.18653/v1/2025.acl-long.905) · [OpenAlex](https://openalex.org/W4412944978)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [26](https://md.research.elamur.ai/free/en/papers/he-2025-can-large-language-models-detect/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Reasoning models in the o1 style produce very long chains of thought (CoT). Until this paper, there was no systematic way to check how good those chains are, or whether other models can find the mistakes in them. The authors built DeltaBench to address both gaps. It contains 1,236 problems from Math, Programming, PCB (physics, chemistry, biology) and General Reasoning. Each problem is paired with a long CoT from QwQ-32B-Preview, DeepSeek-R1 or Gemini 2.0 Flash Thinking.

GPT-4 split each response into sections, each covering one sub-task. Human annotators then labelled every section for:
- strategy shifts,
- usefulness,
- correctness, including the first erroneous step, an explanation and a fix,
- whether any reflection was present and whether it was effective.

The authors used these labels in two ways. First, they profiled the reasoning models, which turned out to be redundant and to reflect unproductively. Second, they scored LLM critics and process reward models (PRMs) on locating the faulty sections. Even the strongest systems performed poorly. The benchmark therefore exposes a real bottleneck for verification, reward modelling and self-correction in long-reasoning systems.

## Key points
- **Query pipeline:** Queries were deduplicated with NV-Embed-v2 embeddings and clustered with DBSCAN, giving 17,510 unique queries. These were then sampled for balance across difficulty levels and 48 subcategories.
- **Quality of the reasoning chains:**
  - Basic errors (calculation, syntax, format) make up about 25% of errors for QwQ-32B-Preview and about 23% for Gemini 2.0 Flash Thinking.
  - Around 67.8% of reflections fail to help reach the correct answer.
  - About 27% of sections are redundant.
- **Critic performance:**
  - The best critic, GPT-4-turbo-128k, reaches an F1 of only 40.8%.
  - o1-like critics show no advantage; o1-preview scores below GPT-4o-mini.
- **Self-critique:** Models judge their own output worse than they judge other models' output. DeepSeek-R1 shows a 36% drop.
- **Model size and PRM ranking:**
  - Across Qwen2.5-Instruct sizes, Macro-F1 rises from 3.79 (0.5B) to 26.73 (32B), with clear jumps at 3B→7B and 7B→14B.
  - Among PRMs evaluated with HitRate@k, Qwen2.5-Math-PRM-7B is the strongest at k=1.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) — Company, Hangzhou, China · 168 works · 15,386 citations
- [Institute of Automation](https://md.research.elamur.ai/free/en/labs/institute-of-automation/) — Research institute, Beijing, China · 70 works · 14,251 citations
- [Nanjing University](https://md.research.elamur.ai/free/en/universities/nanjing-university/) — University, Nanjing, China · 74 works · 7,814 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Z.Y. Peng, Zhaoxiang Zhang, Zhicheng Zheng, Wenbo Su and Bo Zheng. 2025. Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{he2025can,
  title = {{Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?}},
  author = {Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Z.Y. Peng, Zhaoxiang Zhang, Zhicheng Zheng, Wenbo Su and Bo Zheng},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.905},
  url = {https://doi.org/10.18653/v1/2025.acl-long.905}
}
```

- [← previous Can Indirect Prompt Injection Attacks Be Detected and Removed?](https://md.research.elamur.ai/free/en/papers/chen-2025-can-indirect-prompt-injection-attacks/)
- [next → CFBench: A Comprehensive Constraints-Following Benchmark for LLMs](https://md.research.elamur.ai/free/en/papers/zhang-2025-cfbench-comprehensive-constraints-following-benchmark/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/)

## References

45 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. 2024a. [Alphamath almost zero: Process supervision without process](https://md.research.elamur.ai/free/en/papers/chen-2024-alphamath-almost-zero-process-supervision/). Preprint, arXiv:2405.03553.
2. Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. 2024b. [Do not think that much for 2+3=? on the overthinking of o1-like llms](https://md.research.elamur.ai/free/en/papers/chen-2024-think-that-much-2-3/). ArXiv, abs/2412.21187.
3. Bytedance-Seed-Foundation-Code-Team Yao Cheng, Jianfeng Chen, Jie Chen, Li Chen, Liyu Chen, Wentao Chen, Zhengyu Chen, Shijie Geng, Aoyan Li, Bowen Li, Bowen Li, Linyi Li, Boyi Liu, Jerry Liu, Kaibo Liu, Qi Liu, Shukai Liu, Si-Han Liu, Tianyi Liu, Tingkai Liu, Yongfei Liu, Rui Long, Jing Mai, Guangh… — [arXiv:2412.00535](https://arxiv.org/abs/2412.00535) [find](https://scholar.google.com/scholar?q=Bytedance%2DSeed%2DFoundation%2DCode%2DTeam+Yao+Cheng%2C+Jianfeng+Chen%2C+Jie+Chen%2C+Li+Chen%2C+Liyu+Chen%2C+Wentao+Chen%2C+Zhengyu+Chen%2C+Shijie+Geng%2C+Aoyan+Li%2C+Bowen+Li%2C+Bowen+Li%2C+Linyi+Li%2C+Boyi+Liu%2C+Jerry+Liu%2C+Kaibo+L)
4. DeepSeek-AI. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2E+2024%2E+Deepseek+llm%3A+Scaling+open%2Dsource+language+models+with+longtermism%2E+arXiv+preprint+arXiv%3A2401%2E02954%2E)
5. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez,… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
6. Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, KDD’96, page 226–231. AAAI Press. [find](https://scholar.google.com/scholar?q=Martin+Ester%2C+Hans%2DPeter+Kriegel%2C+J%C3%B6rg+Sander%2C+and+Xiaowei+Xu%2E+1996%2E+A+density%2Dbased+algorithm+for+discovering+clusters+in+large+spatial+databases+with+noise%2E+In+Proceedings+of+the+Second+Internationa)
7. Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. 2024. [Omni-math: A universal olympiad level ma…](https://md.research.elamur.ai/free/en/papers/gao-2024-omni-math-universal-olympiad-level/)
8. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
9. Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. 2023. Reasoning with language model is planning with world model. Preprint, arXiv:2305.14992. [find](https://scholar.google.com/scholar?q=Shibo+Hao%2C+Yi+Gu%2C+Haodi+Ma%2C+Joshua+Jiahua+Hong%2C+Zhen+Wang%2C+Daisy+Zhe+Wang%2C+and+Zhiting+Hu%2E+2023%2E+Reasoning+with+language+model+is+planning+with+world+model%2E+Preprint%2C+arXiv%3A2305%2E14992%2E)
10. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. 2024a. [Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems](https://md.research.elamur.ai/free/en/papers/he-2024-olympiadbench-challenging-benchmark-promoting-agi/). In…
11. Yancheng He, Shilong Li, Jiaheng Liu, Yingshui Tan, Weixun Wang, Hui Huang, Xingyuan Bu, Hangyu Guo, Chengwei Hu, Boren Zheng, Zhuoran Lin, Xuepeng Liu, Dekai Sun, Shirong Lin, Zhicheng Zheng, Xiaoyong Zhu, Wenbo Su, and Bo Zheng. 2024b. [Chinese simpleqa: A chinese factuality evaluation for large la…](https://md.research.elamur.ai/free/en/papers/he-2024-chinese-simpleqa-chinese-factuality/)
12. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. [Large language models cannot self-correct reasoning yet](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-cannot-self/). arXiv preprint arXiv:2310.01798.
13. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida I. Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. [Livecodebench: Holistic and contamination free evaluation of large language models for code](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/). ArXiv, abs/2403.07974.
14. Tian Lan, Wenwei Zhang, Chen Xu, Heyan Huang, Dahua Lin, Kai Chen, and Xian-Ling Mao. 2024. Criticeval: Evaluating large-scale language model as critic. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Tian+Lan%2C+Wenwei+Zhang%2C+Chen+Xu%2C+Heyan+Huang%2C+Dahua+Lin%2C+Kai+Chen%2C+and+Xian%2DLing+Mao%2E+2024%2E+Criticeval%3A+Evaluating+large%2Dscale+language+model+as+critic%2E+In+The+Thirty%2Deighth+Annual+Conference+on+Neura)
15. Shilong Li, Yancheng He, Hangyu Guo, Xingyuan Bu, Ge Bai, Jie Liu, Jiaheng Liu, Xingwei Qu, Yangguang Li, Wanli Ouyang, et al. 2024a. Graphreader: Building graph-based agent to enhance long-context abilities of large language models. arXiv preprint arXiv:2406.14550. [find](https://scholar.google.com/scholar?q=Shilong+Li%2C+Yancheng+He%2C+Hangyu+Guo%2C+Xingyuan+Bu%2C+Ge+Bai%2C+Jie+Liu%2C+Jiaheng+Liu%2C+Xingwei+Qu%2C+Yangguang+Li%2C+Wanli+Ouyang%2C+et+al%2E+2024a%2E+Graphreader%3A+Building+graph%2Dbased+agent+to+enhance+long%2Dcontext+ab)
16. Shilong Li, Yancheng He, Hui Huang, Xingyuan Bu, Jiaheng Liu, Hangyu Guo, Weixun Wang, Jihao Gu, Wenbo Su, and Bo Zheng. 2024b. 2d-dpo: Scaling direct preference optimization with 2-dimensional supervision. arXiv preprint arXiv:2410.19720. [find](https://scholar.google.com/scholar?q=Shilong+Li%2C+Yancheng+He%2C+Hui+Huang%2C+Xingyuan+Bu%2C+Jiaheng+Liu%2C+Hangyu+Guo%2C+Weixun+Wang%2C+Jihao+Gu%2C+Wenbo+Su%2C+and+Bo+Zheng%2E+2024b%2E+2d%2Ddpo%3A+Scaling+direct+preference+optimization+with+2%2Ddimensional+superv)
17. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). ArXiv, abs/2305.20050.
18. Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. 2025. Zebralogic: On the scaling limits of llms for logical reasoning. [find](https://scholar.google.com/scholar?q=Bill+Yuchen+Lin%2C+Ronan+Le+Bras%2C+Kyle+Richardson%2C+Ashish+Sabharwal%2C+Radha+Poovendran%2C+Peter+Clark%2C+and+Yejin+Choi%2E+2025%2E+Zebralogic%3A+On+the+scaling+limits+of+llms+for+logical+reasoning%2E)
19. Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, and Yujiu Yang. 2024. Criticbench: Benchmarking llms for critique-correct reasoning. arXiv preprint arXiv:2402.14809. [find](https://scholar.google.com/scholar?q=Zicheng+Lin%2C+Zhibin+Gou%2C+Tian+Liang%2C+Ruilin+Luo%2C+Haowei+Liu%2C+and+Yujiu+Yang%2E+2024%2E+Criticbench%3A+Benchmarking+llms+for+critique%2Dcorrect+reasoning%2E+arXiv+preprint+arXiv%3A2402%2E14809%2E)
20. Jiaheng Liu, Ken Deng, Congnan Liu, Jian Yang, Shukai Liu, He Zhu, Peng Zhao, Linzheng Chai, Yanan Wu, Ke Jin, et al. 2024. [M2rc-eval: Massively multilingual repository-level code completion evaluation](https://md.research.elamur.ai/free/en/papers/liu-2025-m2rc-eval-massively-multilingual-repository/). arXiv preprint arXiv:2410.21157.
21. Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, et al. 2025a. [A comprehensive survey on long context language modeling](https://md.research.elamur.ai/free/en/papers/liu-2025-comprehensive-survey-long-context-language/). arXiv preprint arXiv:2503.17407.
22. Wei Liu, Yancheng He, Hui Huang, Chengwei Hu, Jiaheng Liu, Shilong Li, Wenbo Su, and Bo Zheng. 2025b. Air: Complex instruction generation via automatic iterative refinement. arXiv preprint arXiv:2502.17787. [find](https://scholar.google.com/scholar?q=Wei+Liu%2C+Yancheng+He%2C+Hui+Huang%2C+Chengwei+Hu%2C+Jiaheng+Liu%2C+Shilong+Li%2C+Wenbo+Su%2C+and+Bo+Zheng%2E+2025b%2E+Air%3A+Complex+instruction+generation+via+automatic+iterative+refinement%2E+arXiv+preprint+arXiv%3A2502%2E)
23. Liangchen Luo, Zi Lin, Yinxiao Liu, Lei Shu, Yun Zhu, Jingbo Shang, and Lei Meng. 2023. Critique ability of large language models. arXiv preprint arXiv:2310.04815. [find](https://scholar.google.com/scholar?q=Liangchen+Luo%2C+Zi+Lin%2C+Yinxiao+Liu%2C+Lei+Shu%2C+Yun+Zhu%2C+Jingbo+Shang%2C+and+Lei+Meng%2E+2023%2E+Critique+ability+of+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E04815%2E)
24. Kaijing Ma, Xinrun Du, Yunran Wang, Haoran Zhang, Zhoufutu Wen, Xingwei Qu, Jian Yang, Jiaheng Liu, Minghao Liu, Xiang Yue, Wenhao Huang, and Ge Zhang. 2024. Kor-bench: Benchmarking language models on knowledge-orthogonal reasoning tasks. ArXiv, abs/2410.06526. — [arXiv:2410.06526](https://arxiv.org/abs/2410.06526) [find](https://scholar.google.com/scholar?q=Kaijing+Ma%2C+Xinrun+Du%2C+Yunran+Wang%2C+Haoran+Zhang%2C+Zhoufutu+Wen%2C+Xingwei+Qu%2C+Jian+Yang%2C+Jiaheng+Liu%2C+Minghao+Liu%2C+Xiang+Yue%2C+Wenhao+Huang%2C+and+Ge+Zhang%2E+2024%2E+Kor%2Dbench%3A+Benchmarking+language+models+on)
25. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). PREPRINT.
26. OpenAI. 2024. Learning to reason with llms, september 2024. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2024%2E+Learning+to+reason+with+llms%2C+september+2024%2E)
27. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean-Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sotti… — [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/)
28. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. [Gpqa: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). ArXiv, abs/2311.12022.
29. Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao. 2024. [Can language models solve olympiad programming?](https://md.research.elamur.ai/free/en/papers/shi-2024-language-models-solve-olympiad-programming/) ArXiv, abs/2404.10952.
30. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. [Scaling llm test-time compute optimally can be more effective than scaling model parameters](https://md.research.elamur.ai/free/en/papers/snell-2024-scaling-llm-test-time-compute/). arXiv preprint arXiv:2408.03314.
31. Mingyang Song, Zhao yu Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng. 2025. Prmbench: A fine-grained and challenging benchmark for process-level reward models. [find](https://scholar.google.com/scholar?q=Mingyang+Song%2C+Zhao+yu+Su%2C+Xiaoye+Qu%2C+Jiawei+Zhou%2C+and+Yu+Cheng%2E+2025%2E+Prmbench%3A+A+fine%2Dgrained+and+challenging+benchmark+for+process%2Dlevel+reward+models%2E)
32. Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. 2024. On the self-verification limitations of large language models on reasoning and planning tasks. arXiv preprint arXiv:2402.08115. [find](https://scholar.google.com/scholar?q=Kaya+Stechly%2C+Karthik+Valmeekam%2C+and+Subbarao+Kambhampati%2E+2024%2E+On+the+self%2Dverification+limitations+of+large+language+models+on+reasoning+and+planning+tasks%2E+arXiv+preprint+arXiv%3A2402%2E08115%2E)
33. Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. 2024. Judgebench: A benchmark for evaluating llm-based judges. ArXiv, abs/2410.12784. [find](https://scholar.google.com/scholar?q=Sijun+Tan%2C+Siyuan+Zhuang%2C+Kyle+Montgomery%2C+William+Y%2E+Tang%2C+Alejandro+Cuadron%2C+Chenguang+Wang%2C+Raluca+Ada+Popa%2C+and+Ion+Stoica%2E+2024%2E+Judgebench%3A+A+benchmark+for+evaluating+llm%2Dbased+judges%2E+ArXiv%2C+ab)
34. Qwen Team. 2024. Qwen2.5: A party of foundation models. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+2024%2E+Qwen2%2E5%3A+A+party+of+foundation+models%2E)
35. Gladys Tyen, Hassan Mansoor, Peter Chen, Tony Mak, and Victor Cărbune. 2023. Llms cannot find reasoning errors, but can correct them! arXiv preprint arXiv:2311.08516. — [arXiv:2311.08516](https://arxiv.org/abs/2311.08516) [find](https://scholar.google.com/scholar?q=Gladys+Tyen%2C+Hassan+Mansoor%2C+Peter+Chen%2C+Tony+Mak%2C+and+Victor+Ca%CC%86rbune%2E+2023%2E+Llms+cannot+find+reasoning+errors%2C+but+can+correct+them%21+arXiv+preprint+arXiv%3A2311%2E08516%2E+%E2%80%94+%5BarXiv%3A2311%2E08516%5D%28https%3A%2F%2Farx)
36. Zhenyu Wu, Qingkai Zeng, Zhihan Zhang, Zhaoxuan Tan, Chao Shen, and Meng Jiang. 2024. Large language models can self-correct with minimal effort. arXiv preprint arXiv:2405.14092. — [arXiv:2405.14092](https://arxiv.org/abs/2405.14092) [find](https://scholar.google.com/scholar?q=Zhenyu+Wu%2C+Qingkai+Zeng%2C+Zhihan+Zhang%2C+Zhaoxuan+Tan%2C+Chao+Shen%2C+and+Meng+Jiang%2E+2024%2E+Large+language+models+can+self%2Dcorrect+with+minimal+effort%2E+arXiv+preprint+arXiv%3A2405%2E14092%2E+%E2%80%94+%5BarXiv%3A2405%2E14092%5D%28)
37. Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. 2025. [Demystifying long chain-of-thought reasoning in llms](https://md.research.elamur.ai/free/en/papers/yeo-2025-demystifying-long-chain-thought-reasoning/).
38. Alexander Zhang, Marcus Dong, Jiaheng Liu, Wei Zhang, Yejie Wang, Jian Yang, Ge Zhang, Tianyu Liu, Zhongyuan Peng, Yingshui Tan, et al. 2025. Codecriticbench: A holistic code critique benchmark for large language models. arXiv preprint arXiv:2502.16614. — [arXiv:2502.16614](https://arxiv.org/abs/2502.16614) [find](https://scholar.google.com/scholar?q=Alexander+Zhang%2C+Marcus+Dong%2C+Jiaheng+Liu%2C+Wei+Zhang%2C+Yejie+Wang%2C+Jian+Yang%2C+Ge+Zhang%2C+Tianyu+Liu%2C+Zhongyuan+Peng%2C+Yingshui+Tan%2C+et+al%2E+2025%2E+Codecriticbench%3A+A+holistic+code+critique+benchmark+for+la)
39. Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, Raven Yuan, Tuney Zheng, Wei Pang, Xinrun Du, Yiming Liang, Yinghao Ma, Yizhi Li, Ziyang Ma, Bill Lin, Emmanouil Benetos, Huan Yang, Junting Zhou, Kaijing Ma, Minghao Liu, Mor… [find](https://scholar.google.com/scholar?q=Ge+Zhang%2C+Scott+Qu%2C+Jiaheng+Liu%2C+Chenchen+Zhang%2C+Chenghua+Lin%2C+Chou+Leuang+Yu%2C+Danny+Pan%2C+Esther+Cheng%2C+Jie+Liu%2C+Qunshu+Lin%2C+Raven+Yuan%2C+Tuney+Zheng%2C+Wei+Pang%2C+Xinrun+Du%2C+Yiming+Liang%2C+Yinghao+Ma%2C+Yiz)
40. Shudan Zhang, Hanlin Zhao, Xiao Liu, Qinkai Zheng, Zehan Qi, Xiaotao Gu, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. 2024b. Naturalcodebench: Examining coding performance mismatch on humaneval and natural user prompts. ArXiv, abs/2405.04520. — [arXiv:2405.04520](https://arxiv.org/abs/2405.04520) [find](https://scholar.google.com/scholar?q=Shudan+Zhang%2C+Hanlin+Zhao%2C+Xiao+Liu%2C+Qinkai+Zheng%2C+Zehan+Qi%2C+Xiaotao+Gu%2C+Xiaohan+Zhang%2C+Yuxiao+Dong%2C+and+Jie+Tang%2E+2024b%2E+Naturalcodebench%3A+Examining+coding+performance+mismatch+on+humaneval+and+natur)
41. Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2024a. [Processbench: Identifying process errors in mathematical reasoning](https://md.research.elamur.ai/free/en/papers/zheng-2025-processbench-identifying-process-errors-mathematical/). ArXiv, abs/2412.06559.
42. Xin Zheng, Jie Lou, Boxi Cao, Xueru Wen, Yuqiu Ji, Hongyu Lin, Yaojie Lu, Xianpei Han, Debing Zhang, and Le Sun. 2024b. Critic-cot: Boosting the reasoning abilities of large language model via chain-of-thoughts critic. arXiv preprint arXiv:2408.16326. — [arXiv:2408.16326](https://arxiv.org/abs/2408.16326) [find](https://scholar.google.com/scholar?q=Xin+Zheng%2C+Jie+Lou%2C+Boxi+Cao%2C+Xueru+Wen%2C+Yuqiu+Ji%2C+Hongyu+Lin%2C+Yaojie+Lu%2C+Xianpei+Han%2C+Debing+Zhang%2C+and+Le+Sun%2E+2024b%2E+Critic%2Dcot%3A+Boosting+the+reasoning+abilities+of+large+language+model+via+chain%2Do)
43. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied Sanosi Saied, Weizhu Chen, and Nan Duan. 2023. [Agieval: A human-centric benchmark for evaluating foundation models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/). ArXiv, abs/2304.06364.
44. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. 2024. [Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions](https://md.research.elamur.ai/free/en/papers/zhuo-2024-bigcodebench-benchmarking-code-generation-diverse/). arXiv preprint arXiv:2406.15877.
45. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Preprint, arXiv:2305.10601. *Figure 12: The effect of long CoT length.* [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L%2E+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+2023%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+Preprint%2C+arXiv%3A)

*Updated 30 October 2026*