--- title: "Step-DPO: Step-wise Preference Optimization for — Lai 2024" description: "Large language models often fail at multi-step math because one slip anywhere in a long derivation ruins the final answer." authors: ["Xin Lai", "Zhuotao Tian", "Yukang Chen", "Senqiao Yang", "Xiangru Peng", "Jiaya Jia"] year: 2024 venue: "arXiv:2406.18629" source: "https://arxiv.org/abs/2406.18629" doi: "10.48550/arxiv.2406.18629" arxiv: "2406.18629" licence: "CC BY-NC-SA 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by-nc-sa/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/lai-2024-step-dpo-step-wise-preference/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lai-2024-step-dpo-step-wise-preference/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs # Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs **Authors:** [Xin Lai](https://md.research.elamur.ai/free/en/authors/xin-lai/), Zhuotao Tian, [Yukang Chen](https://md.research.elamur.ai/free/en/authors/yukang-chen/), Senqiao Yang, Xiangru Peng, [Jiaya Jia](https://md.research.elamur.ai/free/en/authors/jiaya-jia/) · **Year:** 2024 · **Venue:** arXiv:2406.18629 · **Licence:** [CC BY-NC-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-sa/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2406.18629) · [PDF](https://arxiv.org/pdf/2406.18629) · [DOI 10.48550/arxiv.2406.18629](https://doi.org/10.48550/arxiv.2406.18629) · [arXiv 2406.18629](https://arxiv.org/abs/2406.18629) · [OpenAlex](https://openalex.org/W4400141325) **Citations:** 1 citations (openalex-snapshot · 2026-10-14) · [96](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Large language models often fail at multi-step math because one slip anywhere in a long derivation ruins the final answer. Standard Direct Preference Optimization (DPO) helps little here. It compares whole answers, so a rejected solution that is correct except for one faulty step gives a noisy training signal. Models trained this way barely improve at telling good outputs from bad ones. Step-DPO changes the unit of preference from the full answer to a single reasoning step. Given a problem and a prefix of correct steps, the model learns to prefer a correct next step over the first wrong one. The authors also built a low-cost three-stage pipeline (error collection, step localization, rectification). It produces about 10K step-level pairs, and the preferred step is sampled from the model itself. After fewer than 500 training steps, Qwen2-72B-Instruct scores 70.8% on MATH and 94.0% on GSM8K, ahead of several proprietary systems. The work shows that fine-grained process supervision is possible without a separate reward model or a full RLHF pipeline. ## Key points - **Method:** each training example is a math problem, the correct earlier steps, one chosen next step and one rejected next step. The DPO objective is applied only to that next step, not to the whole solution. - **Dataset:** about 10K step-wise preference pairs. Using the model's own corrected steps (in-distribution data) worked better than steps written by humans or GPT-4. - **Efficiency:** with fewer than 500 Step-DPO training steps, models with more than 70B parameters gain close to 3 points of accuracy on MATH. - **Headline result:** Qwen2-72B-Instruct + Step-DPO reaches 70.8% on MATH and 94.0% on GSM8K, beating GPT-4-1106, Claude-3-Opus and Gemini-1.5-Pro. Gains are also reported for Qwen2-7B, Qwen2-57B-A14B, Qwen1.5-32B and Llama3-70B. - **Resources:** code, data and models are released at github.com/dvlab-research/Step-DPO. The authors are from CUHK, HIT (Shenzhen) and SmartMore. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/) ## Cited by Cited by 96 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 95 not yet. - [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2025-towards-reasoning-era-survey-long/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{lai2024step, title = {{Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs}}, author = {Lai, Xin and Tian, Zhuotao and Chen, Yukang and Yang, Senqiao and Peng, Xiangru and Jia, Jiaya}, year = {2024}, doi = {10.48550/arxiv.2406.18629}, eprint = {2406.18629}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2406.18629} } ``` - [← previous StanceEval 2024: The First Arabic Stance Detection Shared Task](https://md.research.elamur.ai/free/en/papers/alturayeif-2024-stanceeval-2024-first-arabic-stance/) - [next → StyleRemix: Interpretable Authorship Obfuscation via Distillation and Perturbation of Style Elements](https://md.research.elamur.ai/free/en/papers/fisher-2024-styleremix-interpretable-authorship-obfuscation-distillation/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) ## References 5 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv:2303.08774, 2023. [find](https://scholar.google.com/scholar?q=Josh+Achiam%2C+Steven+Adler%2C+Sandhini+Agarwal%2C+Lama+Ahmad%2C+Ilge+Akkaya%2C+Florencia+Leoni+Aleman%2C+Diogo+Almeida%2C+Janko+Altenschmidt%2C+Sam+Altman%2C+Shyamal+Anadkat%2C+et+al%2E+Gpt%2D4+technical+report%2E+arXiv%3A2303%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/) 2. Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv:2310.10631, 2023. [find](https://scholar.google.com/scholar?q=Zhangir+Azerbayev%2C+Hailey+Schoelkopf%2C+Keiran+Paster%2C+Marco+Dos+Santos%2C+Stephen+McAleer%2C+Albert+Q+Jiang%2C+Jia+Deng%2C+Stella+Biderman%2C+and+Sean+Welleck%2E+Llemma%3A+An+open+language+model+for+mathematics%2E+arX) [our card](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/) 3. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… [find](https://scholar.google.com/scholar?q=Jinze+Bai%2C+Shuai+Bai%2C+Yunfei+Chu%2C+Zeyu+Cui%2C+Kai+Dang%2C+Xiaodong+Deng%2C+Yang+Fan%2C+Wenbin+Ge%2C+Yu+Han%2C+Fei+Huang%2C+Binyuan+Hui%2C+Luo+Ji%2C+Mei+Li%2C+Junyang+Lin%2C+Runji+Lin%2C+Dayiheng+Liu%2C+Gao+Liu%2C+Chengqiang+Lu%2C+) [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/) 4. Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: process supervision without process. arXiv:2405.03553, 2024. [find](https://scholar.google.com/scholar?q=Guoxin+Chen%2C+Minpeng+Liao%2C+Chengxi+Li%2C+and+Kai+Fan%2E+Alphamath+almost+zero%3A+process+supervision+without+process%2E+arXiv%3A2405%2E03553%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2024-alphamath-almost-zero-process-supervision/) 5. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. NeurIPS, 2017. [find](https://scholar.google.com/scholar?q=Paul+F+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2E+NeurIPS%2C+2017%2E)