---
title: "Step-Controlled DPO — paper by Lu et al. (2024)"
description: "The authors propose Step-Controlled DPO (SCDPO), which builds step-level error supervision… 56 references, cited by 23 papers here."
authors: ["Zimu Lu", "Aojun Zhou", "Ke Wang", "Houxing Ren", "Weikang Shi", "Junting Pan", "Mingjie Zhan", "Hongsheng Li"]
year: 2024
venue: "arXiv:2407.00782"
source: "https://arxiv.org/abs/2407.00782"
doi: "10.48550/arxiv.2407.00782"
arxiv: "2407.00782"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lu-2024-step-controlled-dpo-leveraging-stepwise/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lu-2024-step-controlled-dpo-leveraging-stepwise/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoning

# Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoning

**Authors:** [Zimu Lu](https://md.research.elamur.ai/free/en/authors/zimu-lu/), [Aojun Zhou](https://md.research.elamur.ai/free/en/authors/aojun-zhou/), [Ke Wang](https://md.research.elamur.ai/free/en/authors/ke-wang/), [Houxing Ren](https://md.research.elamur.ai/free/en/authors/houxing-ren/), [Weikang Shi](https://md.research.elamur.ai/free/en/authors/weikang-shi/), [Junting Pan](https://md.research.elamur.ai/free/en/authors/junting-pan/), [Mingjie Zhan](https://md.research.elamur.ai/free/en/authors/mingjie-zhan/) and [Hongsheng Li](https://md.research.elamur.ai/free/en/authors/hongsheng-li/) · **Year:** 2024 · **Venue:** arXiv:2407.00782 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2407.00782) · [PDF](https://arxiv.org/pdf/2407.00782) · [DOI 10.48550/arxiv.2407.00782](https://doi.org/10.48550/arxiv.2407.00782) · [arXiv 2407.00782](https://arxiv.org/abs/2407.00782) · [OpenAlex](https://openalex.org/W4400373148)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [23](https://md.research.elamur.ai/free/en/papers/lu-2024-step-controlled-dpo-leveraging-stepwise/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Standard DPO for math reasoning labels whole solutions as good or bad based only on the final answer. This coarse signal does not show the model where a reasoning chain breaks down, and human step-level annotation is costly. The authors propose Step-Controlled DPO (SCDPO), which builds step-level error supervision automatically. They start from a fine-tuned model's correct solutions. From various intermediate steps of each solution, they resample continuations at progressively higher temperatures and keep only those that end in a wrong answer. Each faulty continuation is paired with the original correct one, and the shared prefix becomes part of the prompt. This step-controlled data is combined with ordinary answer-based DPO pairs. Across three 7B SFT models using code-integrated or chain-of-thought formats, SCDPO beat plain DPO, including a DPO baseline trained on an equal amount of data. Applied to InternLM2-20B, it produced a model competitive with leading open-source math LLMs. The approach shows that fine-grained preference signals can be obtained without human labelers.

## Key points
- **Naive DPO data:** about 13K preference pairs, generated from the GSM8K and MATH training sets (7.5K questions each). A manual check found only 2 faulty steps out of 369 in correct-answer solutions.
- **7B experiments:** a Mistral-7B fine-tuned on 34K GSM8K and 47K MATH code-integrated solutions, plus MetaMath-Mistral-7B and MathCoder-Mistral-7B. SCDPO improved over DPO on all three.
- **20B model:** InternLM2-20B, fine-tuned on 250K pairs, including 169K APE210K solutions from GPT-4 Code Interpreter. Scores: 88.5% GSM8K, 58.1% MATH, 90.3% CMATH, 80.4% MGSM-zh, 79.3% APE210K. This is roughly 2–3 points above the SFT model on each benchmark.
- **Limitations:** the method does not help with pure-code solutions or image-dependent problems (e.g., geometry).
- **Code release:** data-generation and training code is available at github.com/mathllm/Step-Controlled_DPO.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Chinese University of Hong Kong](https://md.research.elamur.ai/free/en/universities/chinese-university-of-hong-kong/) — University, Hong Kong, China · 126 works · 14,073 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan and Hongsheng Li. 2024. Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoning. arXiv:2407.00782.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{lu2024step,
  title = {{Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoning}},
  author = {Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan and Hongsheng Li},
  year = {2024},
  doi = {10.48550/arxiv.2407.00782},
  eprint = {2407.00782},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.00782}
}
```

- [← previous ResearchTown: Simulator of Human Research Community](https://md.research.elamur.ai/free/en/papers/yu-2024-researchtown-simulator-human-research-community/)
- [next → Tx-LLM: A Large Language Model for Therapeutics](https://md.research.elamur.ai/free/en/papers/chaves-2024-tx-llm-large-language-model/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

56 references, 33 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. [A general theoretical paradigm to understand learning from human preferences](https://md.research.elamur.ai/free/en/papers/azar-2023-general-theoretical-paradigm-understand-learning/). In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR, 2024.
2. Z. Azerbayev, H. Schoelkopf, K. Paster, M. D. Santos, S. McAleer, A. Q. Jiang, J. Deng, S. Biderman, and S. Welleck. [Llemma: An open language model for mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/). arXiv preprint arXiv:2310.10631, 2023.
3. J. Bai, S. Bai, et al. Qwen technical report, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Bai%2C+S%2E+Bai%2C+et+al%2E+Qwen+technical+report%2C+2023%2E)
4. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073, 2022.
5. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
6. Z. Cai, M. Cao, H. Chen, K. Chen, K. Chen, X. Chen, X. Chen, Z. Chen, Z. Chen, P. Chu, et al. [Internlm2 technical report](https://md.research.elamur.ai/free/en/papers/cai-2024-internlm2-technical-report/). arXiv preprint arXiv:2403.17297, 2024.
7. W. Chen, X. Ma, X. Wang, and W. W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022. [find](https://scholar.google.com/scholar?q=W%2E+Chen%2C+X%2E+Ma%2C+X%2E+Wang%2C+and+W%2E+W%2E+Cohen%2E+Program+of+thoughts+prompting%3A+Disentangling+computation+from+reasoning+for+numerical+reasoning+tasks%2E+arXiv+preprint+arXiv%3A2211%2E12588%2C+2022%2E)
8. P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. [find](https://scholar.google.com/scholar?q=P%2E+F%2E+Christiano%2C+J%2E+Leike%2C+T%2E+Brown%2C+M%2E+Martic%2C+S%2E+Legg%2C+and+D%2E+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+neural+information+processing+systems%2C+30%2C+2017%2E)
9. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
10. K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024. [find](https://scholar.google.com/scholar?q=K%2E+Ethayarajh%2C+W%2E+Xu%2C+N%2E+Muennighoff%2C+D%2E+Jurafsky%2C+and+D%2E+Kiela%2E+Kto%3A+Model+alignment+as+prospect+theoretic+optimization%2E+arXiv+preprint+arXiv%3A2402%2E01306%2C+2024%2E)
11. L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764–10799. PMLR, 2023. [find](https://scholar.google.com/scholar?q=L%2E+Gao%2C+A%2E+Madaan%2C+S%2E+Zhou%2C+U%2E+Alon%2C+P%2E+Liu%2C+Y%2E+Yang%2C+J%2E+Callan%2C+and+G%2E+Neubig%2E+Pal%3A+Program%2Daided+language+models%2E+In+International+Conference+on+Machine+Learning%2C+pages+10764%E2%80%9310799%2E+PMLR%2C+2023%2E)
12. Z. Gou, Z. Shao, Y. Gong, Y. Yang, M. Huang, N. Duan, W. Chen, et al. [Tora: A tool-integrated reasoning agent for mathematical problem solving](https://md.research.elamur.ai/free/en/papers/gou-2023-tora-tool-integrated-reasoning-agent/). arXiv preprint arXiv:2309.17452, 2023.
13. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874, 2021.
14. A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Q%2E+Jiang%2C+A%2E+Sablayrolles%2C+A%2E+Roux%2C+A%2E+Mensch%2C+B%2E+Savary%2C+C%2E+Bamford%2C+D%2E+S%2E+Chaplot%2C+D%2E+d%2E+l%2E+Casas%2C+E%2E+B%2E+Hanna%2C+F%2E+Bressand%2C+et+al%2E+Mixtral+of+experts%2E+arXiv+preprint+arXiv%3A2401%2E04088%2C+2024%2E)
15. F. Jiao, C. Qin, Z. Liu, N. F. Chen, and S. Joty. [Learning planning-based reasoning by trajectories collection and process reward synthesizing](https://md.research.elamur.ai/free/en/papers/jiao-2024-learning-planning-based-reasoning-trajectories/). arXiv preprint arXiv:2402.00658, 2024.
16. J. Kreutzer, J. Uyheng, and S. Riezler. [Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning](https://md.research.elamur.ai/free/en/papers/kreutzer-2018-reliability-learnability-human-bandit-feedback/). arXiv preprint arXiv:1805.10627, 2018.
17. H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.[20050, 2023](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/).
18. T. Liu, Y. Zhao, R. Joshi, M. Khalman, M. Saleh, P. J. Liu, and J. Liu. [Statistical rejection sampling improves preference optimization](https://md.research.elamur.ai/free/en/papers/liu-2023-statistical-rejection-sampling-improves-preference/). arXiv preprint arXiv:2309.06657, 2023.
19. P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. [Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). arXiv preprint arXiv:2310.02255, 2023.
20. Z. Lu, A. Zhou, H. Ren, K. Wang, W. Shi, J. Pan, M. Zhan, and H. Li. Mathgenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of llms. arXiv preprint arXiv:2402.16352, 2024. [find](https://scholar.google.com/scholar?q=Z%2E+Lu%2C+A%2E+Zhou%2C+H%2E+Ren%2C+K%2E+Wang%2C+W%2E+Shi%2C+J%2E+Pan%2C+M%2E+Zhan%2C+and+H%2E+Li%2E+Mathgenie%3A+Generating+synthetic+data+with+question+back%2Dtranslation+for+enhancing+mathematical+reasoning+of+llms%2E+arXiv+preprint+ar)
21. H. Luo, Q. Sun, C. Xu, P. Zhao, J. Lou, C. Tao, X. Geng, Q. Lin, S. Chen, and D. Zhang. [Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/). arXiv preprint arXiv:2308.09583, 2023.
22. X. Nguyen, W. Zhang, X. Li, M. M. Aljunied, Q. Tan, L. Cheng, G. Chen, Y. Deng, S. Yang, C. Liu, H. Zhang, and L. Bing. Seallms - large language models for southeast asia. CoRR, abs/2312.[00738, 2023](https://md.research.elamur.ai/free/en/papers/nguyen-2023-seallms-large-language-models-southeast/).
23. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). ArXiv, abs/2303.08774, 2023.
24. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 35:27730–27744, 2022.
25. A. Pal, D. Karkhanis, S. Dooley, M. Roberts, S. Naidu, and C. White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Pal%2C+D%2E+Karkhanis%2C+S%2E+Dooley%2C+M%2E+Roberts%2C+S%2E+Naidu%2C+and+C%2E+White%2E+Smaug%3A+Fixing+failure+modes+of+preference+optimisation+with+dpo%2Dpositive%2E+arXiv+preprint+arXiv%3A2402%2E13228%2C+2024%2E)
26. R. Rafailov, J. Hejna, R. Park, and C. Finn. From r to q: Your language model is secretly a q-function. arXiv preprint arXiv:2404.12358, 2024. [find](https://scholar.google.com/scholar?q=R%2E+Rafailov%2C+J%2E+Hejna%2C+R%2E+Park%2C+and+C%2E+Finn%2E+From+r+to+q%3A+Your+language+model+is+secretly+a+q%2Dfunction%2E+arXiv+preprint+arXiv%3A2404%2E12358%2C+2024%2E)
27. R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). Advances in Neural Information Processing Systems, 36, 2024.
28. R. Ramamurthy, P. Ammanabrolu, K. Brantley, J. Hessel, R. Sifa, C. Bauckhage, H. Hajishirzi, and Y. Choi. [Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization](https://md.research.elamur.ai/free/en/papers/ramamurthy-2022-reinforcement-learning-natural-language-processing/). arXiv preprint arXiv:2210.01241, 2022.
29. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=J%2E+Schulman%2C+F%2E+Wolski%2C+P%2E+Dhariwal%2C+A%2E+Radford%2C+and+O%2E+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)
30. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. Li, Y. Wu, and D. Guo. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300, 2024.
31. F. Shi, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. Vosoughi, H. W. Chung, Y. Tay, S. Ruder, D. Zhou, D. Das, and J. Wei. [Language models are multilingual chain-of-thought reasoners](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/). In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. Op…
32. N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. [find](https://scholar.google.com/scholar?q=N%2E+Stiennon%2C+L%2E+Ouyang%2C+J%2E+Wu%2C+D%2E+Ziegler%2C+R%2E+Lowe%2C+C%2E+Voss%2C+A%2E+Radford%2C+D%2E+Amodei%2C+and+P%2E+F%2E+Christiano%2E+Learning+to+summarize+with+human+feedback%2E+Advances+in+Neural+Information+Processing+Systems%2C+)
33. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
34. L. Tunstall, E. Beeching, N. Lambert, N. Rajani, K. Rasul, Y. Belkada, S. Huang, L. von Werra, C. Fourrier, N. Habib, et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944, 2023. [find](https://scholar.google.com/scholar?q=L%2E+Tunstall%2C+E%2E+Beeching%2C+N%2E+Lambert%2C+N%2E+Rajani%2C+K%2E+Rasul%2C+Y%2E+Belkada%2C+S%2E+Huang%2C+L%2E+von+Werra%2C+C%2E+Fourrier%2C+N%2E+Habib%2C+et+al%2E+Zephyr%3A+Direct+distillation+of+lm+alignment%2E+arXiv+preprint+arXiv%3A2310%2E1694)
35. K. Wang, J. Pan, W. Shi, Z. Lu, M. Zhan, and H. Li. [Measuring multimodal mathematical reasoning with math-vision dataset](https://md.research.elamur.ai/free/en/papers/wang-2024-measuring-multimodal-mathematical-reasoning-math/). arXiv preprint arXiv:2402.14804, 2024.
36. K. Wang, H. Ren, A. Zhou, Z. Lu, S. Luo, W. Shi, R. Zhang, L. Song, M. Zhan, and H. Li. [Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning](https://md.research.elamur.ai/free/en/papers/wang-2023-mathcoder-seamless-code-integration-llms/). arXiv preprint arXiv:2310.03731, 2023.
37. P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui. Math-shepherd: A label-free step-by-step verifier for llms in mathematical reasoning. arXiv preprint arXiv:2312.[08935, 2023](https://md.research.elamur.ai/free/en/papers/wang-2024-math-shepherd-verify-reinforce-llms/).
38. Z. Wang, R. Xia, and P. Liu. Generative ai for math: Part i–mathpile: A billion-token-scale pretraining corpus for math. arXiv preprint arXiv:2312.17120, 2023. [find](https://scholar.google.com/scholar?q=Z%2E+Wang%2C+R%2E+Xia%2C+and+P%2E+Liu%2E+Generative+ai+for+math%3A+Part+i%E2%80%93mathpile%3A+A+billion%2Dtoken%2Dscale+pretraining+corpus+for+math%2E+arXiv+preprint+arXiv%3A2312%2E17120%2C+2023%2E)
39. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824–24837, 2022.
40. T. Wei, J. Luan, W. Liu, S. Dong, and B. Wang. [Cmath: Can your language model pass chinese elementary school math test?](https://md.research.elamur.ai/free/en/papers/wei-2023-cmath-language-model-pass/), 2023.
41. R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229–256, 2004. [find](https://scholar.google.com/scholar?q=R%2E+J%2E+Williams%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Machine+Learning%2C+8%3A229%E2%80%93256%2C+2004%2E)
42. Z. Xi, W. Chen, B. Hong, S. Jin, R. Zheng, W. He, Y. Ding, S. Liu, X. Guo, J. Wang, et al. Training large language models for reasoning through reverse curriculum reinforcement learning. arXiv preprint arXiv:2402.05808, 2024. [find](https://scholar.google.com/scholar?q=Z%2E+Xi%2C+W%2E+Chen%2C+B%2E+Hong%2C+S%2E+Jin%2C+R%2E+Zheng%2C+W%2E+He%2C+Y%2E+Ding%2C+S%2E+Liu%2C+X%2E+Guo%2C+J%2E+Wang%2C+et+al%2E+Training+large+language+models+for+reasoning+through+reverse+curriculum+reinforcement+learning%2E+arXiv+preprin)
43. Y. Xu, X. Liu, X. Liu, Z. Hou, Y. Li, X. Zhang, Z. Wang, A. Zeng, Z. Du, W. Zhao, et al. [Chatglm-math: Improving math problem-solving in large language models with a self-critique pipeline](https://md.research.elamur.ai/free/en/papers/xu-2024-chatglm-math-improving-math/). arXiv preprint arXiv:2404.02893, 2024.
44. L. Yang, H. Yang, W. Cheng, L. Lin, C. Li, Y. Chen, L. Liu, J. Pan, T. Wei, B. Li, L. Zhao, L. Wang, B. Zhu, G. Li, X. Wu, X. Luo, and R. Hu. Skymath: Technical report, 2023. [find](https://scholar.google.com/scholar?q=L%2E+Yang%2C+H%2E+Yang%2C+W%2E+Cheng%2C+L%2E+Lin%2C+C%2E+Li%2C+Y%2E+Chen%2C+L%2E+Liu%2C+J%2E+Pan%2C+T%2E+Wei%2C+B%2E+Li%2C+L%2E+Zhao%2C+L%2E+Wang%2C+B%2E+Zhu%2C+G%2E+Li%2C+X%2E+Wu%2C+X%2E+Luo%2C+and+R%2E+Hu%2E+Skymath%3A+Technical+report%2C+2023%2E)
45. Z. Yang, M. Ding, Q. Lv, Z. Jiang, Z. He, Y. Guo, J. Bai, and J. Tang. Gpt can solve mathematical problems without a calculator. arXiv preprint arXiv:2309.03241, 2023. [find](https://scholar.google.com/scholar?q=Z%2E+Yang%2C+M%2E+Ding%2C+Q%2E+Lv%2C+Z%2E+Jiang%2C+Z%2E+He%2C+Y%2E+Guo%2C+J%2E+Bai%2C+and+J%2E+Tang%2E+Gpt+can+solve+mathematical+problems+without+a+calculator%2E+arXiv+preprint+arXiv%3A2309%2E03241%2C+2023%2E)
46. S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=S%2E+Yao%2C+D%2E+Yu%2C+J%2E+Zhao%2C+I%2E+Shafran%2C+T%2E+Griffiths%2C+Y%2E+Cao%2C+and+K%2E+Narasimhan%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+Advances+in+Neural+Information+Processing+Systems%2C)
47. Yi. A series of large language models trained from scratch by developers at 01-ai. [https://github.com/01-ai/Yi](https://github.com/01-ai/Yi), 2023. [↗](https://github.com/01-ai/Yi)
48. H. Ying, S. Zhang, L. Li, Z. Zhou, Y. Shao, Z. Fei, Y. Ma, J. Hong, K. Liu, Z. Wang, et al. Internlm-math: Open math large language models toward verifiable reasoning. arXiv preprint arXiv:2402.06332, 2024. [find](https://scholar.google.com/scholar?q=H%2E+Ying%2C+S%2E+Zhang%2C+L%2E+Li%2C+Z%2E+Zhou%2C+Y%2E+Shao%2C+Z%2E+Fei%2C+Y%2E+Ma%2C+J%2E+Hong%2C+K%2E+Liu%2C+Z%2E+Wang%2C+et+al%2E+Internlm%2Dmath%3A+Open+math+large+language+models+toward+verifiable+reasoning%2E+arXiv+preprint+arXiv%3A2402%2E06332%2C)
49. L. Yu, W. Jiang, H. Shi, J. Yu, Z. Liu, Y. Zhang, J. T. Kwok, Z. Li, A. Weller, and W. Liu. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). arXiv preprint arXiv:2309.12284, 2023.
50. W. Yuan, R. Y. Pang, K. Cho, S. Sukhbaatar, J. Xu, and J. Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024. [find](https://scholar.google.com/scholar?q=W%2E+Yuan%2C+R%2E+Y%2E+Pang%2C+K%2E+Cho%2C+S%2E+Sukhbaatar%2C+J%2E+Xu%2C+and+J%2E+Weston%2E+Self%2Drewarding+language+models%2E+arXiv+preprint+arXiv%3A2401%2E10020%2C+2024%2E)
51. Z. Yuan, H. Yuan, C. Li, G. Dong, C. Tan, and C. Zhou. [Scaling relationship on learning mathematical reasoning with large language models](https://md.research.elamur.ai/free/en/papers/yuan-2023-scaling-relationship-learning-mathematical-reasoning/). arXiv preprint arXiv:2308.01825, 2023.
52. X. Yue, X. Qu, G. Zhang, Y. Fu, W. Huang, H. Sun, Y. Su, and W. Chen. [Mammoth: Building math generalist models through hybrid instruction tuning](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/). arXiv preprint arXiv:2309.05653, 2023.
53. W. Zhao, M. Shang, Y. Liu, L. Wang, and J. Liu. Ape210k: A large-scale and template-rich dataset of math word problems, 2020. [find](https://scholar.google.com/scholar?q=W%2E+Zhao%2C+M%2E+Shang%2C+Y%2E+Liu%2C+L%2E+Wang%2C+and+J%2E+Liu%2E+Ape210k%3A+A+large%2Dscale+and+template%2Drich+dataset+of+math+word+problems%2C+2020%2E)
54. Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu. [Slic-hf: Sequence likelihood calibration with human feedback](https://md.research.elamur.ai/free/en/papers/zhao-2023-slic-hf-sequence-likelihood-calibration/). arXiv preprint arXiv:2305.10425, 2023.
55. A. Zhou, K. Wang, Z. Lu, W. Shi, S. Luo, Z. Qin, S. Lu, A. Jia, L. Song, M. Zhan, et al. [Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification](https://md.research.elamur.ai/free/en/papers/zhou-2023-solving-challenging-math-word-problems/). arXiv preprint arXiv:2308.07921, 2023.
56. D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. GSM8K MATH Accuracy (%) 0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 Temperatur… [find](https://scholar.google.com/scholar?q=D%2E+M%2E+Ziegler%2C+N%2E+Stiennon%2C+J%2E+Wu%2C+T%2E+B%2E+Brown%2C+A%2E+Radford%2C+D%2E+Amodei%2C+P%2E+Christiano%2C+and+G%2E+Irving%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint+arXiv%3A1909%2E08593%2C+2019%2E+GSM8K+MA)

*Updated 30 October 2026*