---
title: "Do NOT Think That Much for 2+3=? On the — Chen 2024"
description: "Reasoning models in the o1 family get better accuracy by writing very long chains of thought."
authors: ["Xingyu Chen", "Jiahao Xu", "Tian Liang", "Zhiwei He", "Jianhui Pang", "Dian Yu", "Linfeng Song", "Qiuzhi Liu", "Mengfei Zhou", "Zhuosheng Zhang", "Rui Wang", "Zhaopeng Tu", "Haitao Mi", "Dong Yu"]
year: 2024
venue: "arXiv:2412.21187"
source: "https://arxiv.org/abs/2412.21187"
doi: "10.48550/arxiv.2412.21187"
arxiv: "2412.21187"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2024-think-that-much-2-3/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2024-think-that-much-2-3/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/) / Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs

# Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs

**Authors:** Xingyu Chen, Jiahao Xu, [Tian Liang](https://md.research.elamur.ai/free/en/authors/tian-liang/), [Zhiwei He](https://md.research.elamur.ai/free/en/authors/zhiwei-he/), Jianhui Pang, [Dian Yu](https://md.research.elamur.ai/free/en/authors/dian-yu/), [Linfeng Song](https://md.research.elamur.ai/free/en/authors/linfeng-song/), Qiuzhi Liu, Mengfei Zhou, [Zhuosheng Zhang](https://md.research.elamur.ai/free/en/authors/zhuosheng-zhang/), [Rui Wang](https://md.research.elamur.ai/free/en/authors/rui-wang/), [Zhaopeng Tu](https://md.research.elamur.ai/free/en/authors/zhaopeng-tu/) et al. (14 authors) · **Year:** 2024 · **Venue:** arXiv:2412.21187 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2412.21187) · [PDF](https://arxiv.org/pdf/2412.21187) · [DOI 10.48550/arxiv.2412.21187](https://doi.org/10.48550/arxiv.2412.21187) · [arXiv 2412.21187](https://arxiv.org/abs/2412.21187) · [OpenAlex](https://openalex.org/W4405957960)

**Citations:** 3 citations (openalex-snapshot · 2026-10-14) · [185](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Reasoning models in the o1 family get better accuracy by writing very long chains of thought. This paper argues that much of that extra test-time compute is wasted. The authors call the waste "overthinking": after a model already has the right answer to a trivial question, it keeps producing new solution attempts. They studied QwQ-32B-Preview and DeepSeek-R1 on ASDIV, GSM8K and MATH500. The extra rounds rarely change the answer and seldom try a new approach. They also happen more often on easy problems than on hard ones. The authors add two efficiency metrics to plain accuracy. One looks at outcomes (how many tokens were needed to reach the correct answer) and the other at the process (how varied the reasoning is). To reduce overthinking they use self-training. They take the model's own sampled outputs and cut out redundant solutions, for example keeping everything up to the first correct solution plus one reflection step. They then train on these shorter outputs with SFT or preference optimization. This makes reasoning much shorter while keeping accuracy, which matters for anyone running reasoning LLMs with limited budget or latency.

## Key points
- For the question "2 plus 3", o1-like models used about 1,953% more tokens than conventional LLMs. In one example, QwQ-32B-Preview wrote 13 separate solutions. Solutions were split out of responses with Llama-3.3-70B.
- The first solution round was already correct in more than 92% of correctly answered cases. On ASDIV, for instance, it made up only about 38.7% of QwQ's output.
- Among the training methods, SimPO beat SFT and the other preference-optimization variants, cutting tokens by 22.3% on MATH500. "FCS+Reflection" was the best way to simplify responses, beating plain FCS by 1.4 accuracy points on MATH500.
- The combined approach cut QwQ-32B-Preview's output on MATH500 by 48.6% with no loss of accuracy. Accuracy held on the harder GPQA and AIME sets and improved on ASDIV and GSM8K.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

185 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 176 not yet.

- [L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/aggarwal-2025-l1-controlling-long-reasoning-model/)
- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2025-towards-reasoning-era-survey-long/)
- [OpenThoughts: Data Recipes for Reasoning Models](https://md.research.elamur.ai/free/en/papers/guha-2025-openthoughts-data-recipes-reasoning-models/)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://md.research.elamur.ai/free/en/papers/kimi-2025-team-kimi-k1-5-scaling-reinforcement/)
- [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://md.research.elamur.ai/free/en/papers/li-2025-search-o1-agentic-search-enhanced/)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://md.research.elamur.ai/free/en/papers/liu-2025-understanding-r1-zero-like-training/)
- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://md.research.elamur.ai/free/en/papers/sui-2025-stop-overthinking-survey-efficient-reasoning/)
- [Do Multimodal LLMs Understand Order? Measuring the Fragility of Multimodal Reasoning under Input Order Perturbations](https://md.research.elamur.ai/free/en/papers/wei-2026-do-multimodal-llms-understand-order/)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://md.research.elamur.ai/free/en/papers/zeng-2025-simplerl-zoo-investigating-taming-zero/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chen2024do,
  title = {{Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs}},
  author = {Chen, Xingyu and Xu, Jiahao and Liang, Tian and He, Zhiwei and Pang, Jianhui and Yu, Dian and Song, Linfeng and Liu, Qiuzhi and Zhou, Mengfei and Zhang, Zhuosheng and Wang, Rui and Tu, Zhaopeng and Mi, Haitao and Yu, Dong},
  year = {2024},
  doi = {10.48550/arxiv.2412.21187},
  eprint = {2412.21187},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.21187}
}
```

- [← previous Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://md.research.elamur.ai/free/en/papers/wendler-2024-do-llamas-work-english-latent/)
- [next → Does DetectGPT Fully Utilize Perturbation? Bridging Selective Perturbation to Fine-tuned Contrastive Learning Detector would be Better](https://md.research.elamur.ai/free/en/papers/liu-2024-does-detectgpt-fully-utilize-perturbation/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/)

## References

48 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Int… [find](https://scholar.google.com/scholar?q=Maciej+Besta%2C+Nils+Blach%2C+Ales+Kubicek%2C+Robert+Gerstenberger%2C+Michal+Podstawski%2C+Lukas+Gianinazzi%2C+Joanna+Gajda%2C+Tomasz+Lehmann%2C+Hubert+Niewiadomski%2C+Piotr+Nyczyk%2C+et+al%2E+Graph+of+thoughts%3A+Solving+el)
2. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv:2110.14168, 2021. [find](https://scholar.google.com/scholar?q=Karl+Cobbe%2C+Vineet+Kosaraju%2C+Mohammad+Bavarian%2C+Mark+Chen%2C+Heewoo+Jun%2C+Lukasz+Kaiser%2C+Matthias+Plappert%2C+Jerry+Tworek%2C+Jacob+Hilton%2C+Reiichiro+Nakano%2C+Christopher+Hesse%2C+and+John+Schulman%2E+Training+ve) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
3. Mehul Damani, Idan Shenfeld, Andi Peng, Andreea Bobu, and Jacob Andreas. Learning how hard to think: Input-adaptive allocation of lm computation, 2024. URL https://arxiv.org/abs/2410.04707. [↗](https://arxiv.org/abs/2410.04707)
4. DeepSeek. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. 2025. URL [https://api.semanticscholar.org/CorpusID:275789950](https://api.semanticscholar.org/CorpusID:275789950). [↗](https://api.semanticscholar.org/CorpusID:275789950) [our card](https://md.research.elamur.ai/free/en/papers/deepseek-2025-ai-deepseek-r1-incentivizing-reasoning-capability/)
5. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning, 2024. [find](https://scholar.google.com/scholar?q=Yilun+Du%2C+Shuang+Li%2C+Antonio+Torralba%2C+Joshua+B+Tenenbaum%2C+and+Igor+Mordatch%2E+Improving+factuality+and+reasoning+in+language+models+through+multiagent+debate%2E+In+Forty%2Dfirst+International+Conference+o) [our card](https://md.research.elamur.ai/free/en/papers/du-2023-improving-factuality-reasoning-language-models/)
6. Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=ph04CRkPdC](https://openreview.net/forum?id=ph04CRkPdC). [↗](https://openreview.net/forum?id=ph04CRkPdC)
7. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. [find](https://scholar.google.com/scholar?q=Daya+Guo%2C+Dejian+Yang%2C+Haowei+Zhang%2C+Junxiao+Song%2C+Ruoyu+Zhang%2C+Runxin+Xu%2C+Qihao+Zhu%2C+Shirong+Ma%2C+Peiyi+Wang%2C+Xiao+Bi%2C+et+al%2E+Deepseek%2Dr1%3A+Incentivizing+reasoning+capability+in+llms+via+reinforcement+) [our card](https://md.research.elamur.ai/free/en/papers/deepseek-2025-ai-deepseek-r1-incentivizing-reasoning-capability/)
8. Tingxu Han, Chunrong Fang, Shiyu Zhao, Shiqing Ma, Zhenyu Chen, and Zhenting Wang. Token-budget-aware llm reasoning. arXiv preprint arXiv:2412.18547, 2024. [find](https://scholar.google.com/scholar?q=Tingxu+Han%2C+Chunrong+Fang%2C+Shiyu+Zhao%2C+Shiqing+Ma%2C+Zhenyu+Chen%2C+and+Zhenting+Wang%2E+Token%2Dbudget%2Daware+llm+reasoning%2E+arXiv+preprint+arXiv%3A2412%2E18547%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/han-2024-token-budget-aware-llm-reasoning/)
9. Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space, 2024. URL https://arxiv.org/abs/2412.06769. [↗](https://arxiv.org/abs/2412.06769)
10. David Heineman, Yao Dou, and Wei Xu. Improving minimum bayes risk decoding with multi-prompt. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 22525–22545, 2024. [find](https://scholar.google.com/scholar?q=David+Heineman%2C+Yao+Dou%2C+and+Wei+Xu%2E+Improving+minimum+bayes+risk+decoding+with+multi%2Dprompt%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pp%2E+22525%E2%80%9322545)
11. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In NeurIPS, 2021. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Collin+Burns%2C+Saurav+Kadavath%2C+Akul+Arora%2C+Steven+Basart%2C+Eric+Tang%2C+Dawn+Song%2C+and+Jacob+Steinhardt%2E+Measuring+mathematical+problem+solving+with+the+MATH+dataset%2E+In+NeurIPS%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/)
12. Namgyu Ho, Laura Schmid, and Se-Young Yun. Large language models are reasoning teachers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14852–14882, 2023. [find](https://scholar.google.com/scholar?q=Namgyu+Ho%2C+Laura+Schmid%2C+and+Se%2DYoung+Yun%2E+Large+language+models+are+reasoning+teachers%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Pape) [our card](https://md.research.elamur.ai/free/en/papers/ho-2023-large-language-models-are-reasoning/)
13. Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. When can llms actually correct their own mistakes? a critical survey of self-correction of llms. Transactions of the Association for Computational Linguistics, 12:1417–1440, 2024. [find](https://scholar.google.com/scholar?q=Ryo+Kamoi%2C+Yusen+Zhang%2C+Nan+Zhang%2C+Jiawei+Han%2C+and+Rui+Zhang%2E+When+can+llms+actually+correct+their+own+mistakes%3F+a+critical+survey+of+self%2Dcorrection+of+llms%2E+Transactions+of+the+Association+for+Compu) [our card](https://md.research.elamur.ai/free/en/papers/kamoi-2024-when-can-llms-actually-correct/)
14. Pei Ke, Bosi Wen, Zhuoer Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, et al. Critiquellm: Scaling llm-as-critic for effective and explainable evaluation of large language model generation. corr, abs/2311.18702. detection for generative large langua… [find](https://scholar.google.com/scholar?q=Pei+Ke%2C+Bosi+Wen%2C+Zhuoer+Feng%2C+Xiao+Liu%2C+Xuanyu+Lei%2C+Jiale+Cheng%2C+Shengyuan+Wang%2C+Aohan+Zeng%2C+Yuxiao+Dong%2C+Hongning+Wang%2C+et+al%2E+Critiquellm%3A+Scaling+llm%2Das%2Dcritic+for+effective+and+explainable+evalua)
15. Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment as reward-guided search. In The Twelfth International Conference on Learning Representations, 2024. [find](https://scholar.google.com/scholar?q=Maxim+Khanov%2C+Jirayu+Burapacheep%2C+and+Yixuan+Li%2E+Args%3A+Alignment+as+reward%2Dguided+search%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+2024%2E)
16. Team Kimi, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025. [find](https://scholar.google.com/scholar?q=Team+Kimi%2C+Angang+Du%2C+Bofei+Gao%2C+Bowei+Xing%2C+Changjiu+Jiang%2C+Cheng+Chen%2C+Cheng+Li%2C+Chenjun+Xiao%2C+Chenzhuang+Du%2C+Chonghua+Liao%2C+et+al%2E+Kimi+k1%2E+5%3A+Scaling+reinforcement+learning+with+llms%2E+arXiv+prepri) [our card](https://md.research.elamur.ai/free/en/papers/kimi-2025-team-kimi-k1-5-scaling-reinforcement/)
17. Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, and Yuri Burda. Prover-verifier games improve legibility of llm outputs, 2024. URL https://arxiv.org/abs/2407.13692. — [arXiv:2407.13692](https://arxiv.org/abs/2407.13692) [↗](https://arxiv.org/abs/2407.13692)
18. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199–22213, 2022. [find](https://scholar.google.com/scholar?q=Takeshi+Kojima%2C+Shixiang+Shane+Gu%2C+Machel+Reid%2C+Yutaka+Matsuo%2C+and+Yusuke+Iwasawa%2E+Large+language+models+are+zero%2Dshot+reasoners%2E+Advances+in+neural+information+processing+systems%2C+35%3A+22199%E2%80%9322213%2C+20) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)
19. Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12917, 2024. [find](https://scholar.google.com/scholar?q=Aviral+Kumar%2C+Vincent+Zhuang%2C+Rishabh+Agarwal%2C+Yi+Su%2C+John+D+Co%2DReyes%2C+Avi+Singh%2C+Kate+Baumli%2C+Shariq+Iqbal%2C+Colton+Bishop%2C+Rebecca+Roelofs%2C+et+al%2E+Training+language+models+to+self%2Dcorrect+via+reinfor) [our card](https://md.research.elamur.ai/free/en/papers/kumar-2024-training-language-models-self-correct/)
20. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Yifei+Li%2C+Zeqi+Lin%2C+Shizhuo+Zhang%2C+Qiang+Fu%2C+Bei+Chen%2C+Jian%2DGuang+Lou%2C+and+Weizhu+Chen%2E+Making+language+models+better+reasoners+with+step%2Daware+verifier%2E+In+Anna+Rogers%2C+Jordan+Boyd%2DGraber%2C+and+Naoaki) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-making-language-models-better-reasoners/)
21. Yiwei Li, Peiwen Yuan, Shaoxiong Feng, Boyuan Pan, Xinglin Wang, Bin Sun, Heda Wang, and Kan Li. Escape sky-high cost: Early-stopping self-consistency for multi-step reasoning. In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=ndR8Ytrzhh](https://openreview.net/forum?id=ndR8Ytrzhh). [↗](https://openreview.net/forum?id=ndR8Ytrzhh)
22. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empiri… [find](https://scholar.google.com/scholar?q=Tian+Liang%2C+Zhiwei+He%2C+Wenxiang+Jiao%2C+Xing+Wang%2C+Yan+Wang%2C+Rui+Wang%2C+Yujiu+Yang%2C+Shuming+Shi%2C+and+Zhaopeng+Tu%2E+Encouraging+divergent+thinking+in+large+language+models+through+multi%2Dagent+debate%2E+In+Ya) [our card](https://md.research.elamur.ai/free/en/papers/liang-2023-encouraging-divergent-thinking-large-language/)
23. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=v8L0pN6EOi](https://openreview.net/forum?id=v8L0pN6EOi). [↗](https://openreview.net/forum?id=v8L0pN6EOi) [our card](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/)
24. Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, and Yujiu Yang. CriticBench: Benchmarking LLMs for critique-correct reasoning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 1552–1587, Bangkok, Thailand,… [find](https://scholar.google.com/scholar?q=Zicheng+Lin%2C+Zhibin+Gou%2C+Tian+Liang%2C+Ruilin+Luo%2C+Haowei+Liu%2C+and+Yujiu+Yang%2E+CriticBench%3A+Benchmarking+LLMs+for+critique%2Dcorrect+reasoning%2E+In+Lun%2DWei+Ku%2C+Andre+Martins%2C+and+Vivek+Srikumar+%28eds%2E%29%2C+Fin)
25. Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer. arXiv preprint arXiv:2405.16436, 2024. — [arXiv:2405.16436](https://arxiv.org/abs/2405.16436) [find](https://scholar.google.com/scholar?q=Zhihan+Liu%2C+Miao+Lu%2C+Shenao+Zhang%2C+Boyi+Liu%2C+Hongyi+Guo%2C+Yingxiang+Yang%2C+Jose+Blanchet%2C+and+Zhaoran+Wang%2E+Provably+mitigating+overoptimization+in+rlhf%3A+Your+sft+loss+is+implicitly+an+adversarial+regul)
26. Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592, 2024. [find](https://scholar.google.com/scholar?q=Liangchen+Luo%2C+Yinxiao+Liu%2C+Rosanne+Liu%2C+Samrat+Phatale%2C+Harsh+Lara%2C+Yunxuan+Li%2C+Lei+Shu%2C+Yun+Zhu%2C+Lei+Meng%2C+Jiao+Sun%2C+et+al%2E+Improve+mathematical+reasoning+in+language+models+by+automated+process+sup) [our card](https://md.research.elamur.ai/free/en/papers/luo-2024-improve-mathematical-reasoning-language-models/)
27. Rohin Manvi, Anikait Singh, and Stefano Ermon. Adaptive inference-time compute: Llms can predict if they can do better, even mid-generation, 2024. URL https://arxiv.org/abs/2410.02725. [↗](https://arxiv.org/abs/2410.02725)
28. Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. In Advances in Neural Information Processing Systems (NeurIPS), 2024. [find](https://scholar.google.com/scholar?q=Yu+Meng%2C+Mengzhou+Xia%2C+and+Danqi+Chen%2E+Simpo%3A+Simple+preference+optimization+with+a+reference%2Dfree+reward%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2C+2024%2E)
29. Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. [find](https://scholar.google.com/scholar?q=Shen%2DYun+Miao%2C+Chao%2DChun+Liang%2C+and+Keh%2DYih+Su%2E+A+diverse+corpus+for+evaluating+and+developing+english+math+word+problem+solvers%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Compu)
30. OpenAI. Learning to reason learning-to-reason-with-llms, 2024. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Learning+to+reason+learning%2Dto%2Dreason%2Dwith%2Dllms%2C+2024%2E)
31. Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason E Weston. Iterative reasoning preference optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL [https://openreview.net/forum?id=4XIKfvNYvx](https://openreview.net/forum?id=4XIKfvNYvx). [↗](https://openreview.net/forum?id=4XIKfvNYvx) [our card](https://md.research.elamur.ai/free/en/papers/pang-2024-iterative-reasoning-preference-optimization/)
32. Qwen. Qwq: Reflect deeply on the boundaries of the unknown, November 2024. URL [https://qwenlm.github.io/blog/qwq-32b-preview/](https://qwenlm.github.io/blog/qwq-32b-preview/). [↗](https://qwenlm.github.io/blog/qwq-32b-preview/)
33. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Rafael+Rafailov%2C+Archit+Sharma%2C+Eric+Mitchell%2C+Christopher+D+Manning%2C+Stefano+Ermon%2C+and+Chelsea+Finn%2E+Direct+preference+optimization%3A+Your+language+model+is+secretly+a+reward+model%2E+Advances+in+Neura)
34. Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu. Toward self-improvement of llms via imagination, searching, and criticizing. arXiv preprint arXiv:2404.12253, 2024. [find](https://scholar.google.com/scholar?q=Ye+Tian%2C+Baolin+Peng%2C+Linfeng+Song%2C+Lifeng+Jin%2C+Dian+Yu%2C+Haitao+Mi%2C+and+Dong+Yu%2E+Toward+self%2Dimprovement+of+llms+via+imagination%2C+searching%2C+and+criticizing%2E+arXiv+preprint+arXiv%3A2404%2E12253%2C+2024%2E)
35. Ziyu Wan, Xidong Feng, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training. In Forty-first International Conference on Machine Learning, 2024. [find](https://scholar.google.com/scholar?q=Ziyu+Wan%2C+Xidong+Feng%2C+Muning+Wen%2C+Stephen+Marcus+McAleer%2C+Ying+Wen%2C+Weinan+Zhang%2C+and+Jun+Wang%2E+Alphazero%2Dlike+tree%2Dsearch+can+guide+large+language+model+decoding+and+training%2E+In+Forty%2Dfirst+Interna)
36. Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the As… [find](https://scholar.google.com/scholar?q=Lei+Wang%2C+Wanyu+Xu%2C+Yihuai+Lan%2C+Zhiqiang+Hu%2C+Yunshi+Lan%2C+Roy+Ka%2DWei+Lee%2C+and+Ee%2DPeng+Lim%2E+Plan%2Dand%2Dsolve+prompting%3A+Improving+zero%2Dshot+chain%2Dof%2Dthought+reasoning+by+large+language+models%2E+In+Anna+Rog)
37. Xinglin Wang, Shaoxiong Feng, Yiwei Li, Peiwen Yuan, Yueqi Zhang, Boyuan Pan, Heda Wang, Yao Hu, and Kan Li. Make every penny count: Difficulty-adaptive self-consistency for cost-efficient reasoning, 2024. URL https://arxiv.org/abs/2408.13457. [↗](https://arxiv.org/abs/2408.13457)
38. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023b. URL [https://openreview.net/forum?i…](https://openreview.net/forum?i…) [↗](https://openreview.net/forum?i…) [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/)
39. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022a. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Bosma%2C+Vincent+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+M+Dai%2C+and+Quoc+V+Le%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+In+International+Conference+on+Le)
40. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022b. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Maarten+Bosma%2C+Fei+Xia%2C+Ed+Chi%2C+Quoc+V+Le%2C+Denny+Zhou%2C+et+al%2E+Chain%2Dof%2Dthought+prompting+elicits+reasoning+in+large+language+models%2E+Advances+in+neural+informa) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
41. Ian Wu, Patrick Fernandes, Amanda Bertsch, Seungone Kim, Sina Pakazad, and Graham Neubig. Better instruction-following through minimum bayes risk. arXiv preprint arXiv:2410.02902, 2024. — [arXiv:2410.02902](https://arxiv.org/abs/2410.02902) [find](https://scholar.google.com/scholar?q=Ian+Wu%2C+Patrick+Fernandes%2C+Amanda+Bertsch%2C+Seungone+Kim%2C+Sina+Pakazad%2C+and+Graham+Neubig%2E+Better+instruction%2Dfollowing+through+minimum+bayes+risk%2E+arXiv+preprint+arXiv%3A2410%2E02902%2C+2024%2E+%E2%80%94+%5BarXiv%3A2410%2E)
42. Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, and Bing Qin. Examining inter-consistency of large language models collaboration: An in-depth analysis via debate. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7572–7590, 2023. [find](https://scholar.google.com/scholar?q=Kai+Xiong%2C+Xiao+Ding%2C+Yixin+Cao%2C+Ting+Liu%2C+and+Bing+Qin%2E+Examining+inter%2Dconsistency+of+large+language+models+collaboration%3A+An+in%2Ddepth+analysis+via+debate%2E+In+Findings+of+the+Association+for+Computa)
43. Mayi Xu, Yongqi Li, Ke Sun, and Tieyun Qian. Adaption-of-thought: Learning question difficulty improves large language models for reasoning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 5468–5… [find](https://scholar.google.com/scholar?q=Mayi+Xu%2C+Yongqi+Li%2C+Ke+Sun%2C+and+Tieyun+Qian%2E+Adaption%2Dof%2Dthought%3A+Learning+question+difficulty+improves+large+language+models+for+reasoning%2E+In+Yaser+Al%2DOnaizan%2C+Mohit+Bansal%2C+and+Yun%2DNung+Chen+%28eds%2E%29)
44. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Tom+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+Advances+in+Neural+Informati)
45. Junyi Ye, Jingyi Gu, Xinyun Zhao, Wenpeng Yin, and Guiling Wang. Assessing the creativity of llms in proposing novel solutions to mathematical problems. arXiv preprint arXiv:2410.18336, 2024. [find](https://scholar.google.com/scholar?q=Junyi+Ye%2C+Jingyi+Gu%2C+Xinyun+Zhao%2C+Wenpeng+Yin%2C+and+Guiling+Wang%2E+Assessing+the+creativity+of+llms+in+proposing+novel+solutions+to+mathematical+problems%2E+arXiv+preprint+arXiv%3A2410%2E18336%2C+2024%2E)
46. Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, et al. Self-generated critiques boost reward modeling for language models. arXiv preprint arXiv:2411.16646, 2024. [find](https://scholar.google.com/scholar?q=Yue+Yu%2C+Zhengxing+Chen%2C+Aston+Zhang%2C+Liang+Tan%2C+Chenguang+Zhu%2C+Richard+Yuanzhe+Pang%2C+Yundi+Qian%2C+Xuewei+Wang%2C+Suchin+Gururangan%2C+Chao+Zhang%2C+et+al%2E+Self%2Dgenerated+critiques+boost+reward+modeling+for+l)
47. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488, 2022. [find](https://scholar.google.com/scholar?q=Eric+Zelikman%2C+Yuhuai+Wu%2C+Jesse+Mu%2C+and+Noah+Goodman%2E+Star%3A+Bootstrapping+reasoning+with+reasoning%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A15476%E2%80%9315488%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/)
48. Zirui Zhao, Hanze Dong, Amrita Saha, Caiming Xiong, and Doyen Sahoo. Automatic curriculum expert iteration for reliable llm reasoning, 2024. URL https://arxiv.org/abs/2410.07627. — [arXiv:2410.07627](https://arxiv.org/abs/2410.07627) [↗](https://arxiv.org/abs/2410.07627)