---
title: "HDPO — paper by Ding (2026) — summary and ideas"
description: "Two propositions support this design. 33 references, cited by 8 papers here. Among them: 0."
authors: ["Ken Ding"]
year: 2026
venue: "arXiv:2603.23871"
source: "https://arxiv.org/abs/2603.23871"
doi: "10.48550/arxiv.2603.23871"
arxiv: "2603.23871"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ding-2026-hdpo-hybrid-distillation-policy-optimization/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ding-2026-hdpo-hybrid-distillation-policy-optimization/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/) / HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation

# HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation

**Authors:** Ken Ding · **Year:** 2026 · **Venue:** arXiv:2603.23871 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2603.23871) · [PDF](https://arxiv.org/pdf/2603.23871) · [DOI 10.48550/arxiv.2603.23871](https://doi.org/10.48550/arxiv.2603.23871) · [arXiv 2603.23871](https://arxiv.org/abs/2603.23871) · [OpenAlex](https://openalex.org/W7140523709)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/ding-2026-hdpo-hybrid-distillation-policy-optimization/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
When LLMs are fine-tuned for math with RL and binary verifiable rewards (for example, GRPO), "cliff" prompts produce no learning signal. If every rollout in a group fails, the group-normalized advantages are all zero, so the hardest problems never contribute gradients.

HDPO adds a self-distillation term aimed at exactly these prompts. After each GRPO update, it:
1. finds the cliff prompts,
2. lets the same model re-attempt them with the ground-truth answer added to its input,
3. keeps only the correct privileged outputs,
4. moves the unprivileged policy toward the teacher's per-token distribution through a JSD loss weighted by λ.

Two propositions support this design. Because teacher and student share weights, the model-mismatch part of the realizability gap disappears. Keeping only reward-1 samples recovers the policy that is optimal under KL-regularized RL when the threshold is taken to its hard limit.

On OpenMathInstruct-2 with Qwen2.5-Math-1.5B-Instruct, HDPO gives modest gains in pass@4 and pass@8, while pass@1 stays near the baseline. Its main appeal is simplicity: it needs no replay buffer, hint scheduler, curriculum or separate reward model.

## Key points
- **Setup:** Qwen2.5-Math-1.5B-Instruct trained on OpenMathInstruct-2 for about 2000 steps on 8×H200, with 32 prompts per step and 16 generations each.
  - A cliff is a prompt whose total reward is 0.
  - The teacher is either "frozen" (initial weights) or "drifting" (current weights).
  - λ ∈ {0.01, 0.1}.
- **Reported gains over GRPO:** pass@4 improves by +0.8–1.1% and pass@8 by +0.4–1.7%. λ trades exploration (coverage) against exploitation (greedy accuracy).
- **H100 replication:**
  - With λ = 0.1, pass@8 rises by about +1.4–1.5% on both H200 and H100 (for example, 0.8223 → 0.8369 on H100).
  - The cost is lower pass@1: 0.6509 for the baseline versus 0.6343.
  - Gains at λ = 0.01 are close to the noise level.
- **Related approaches:** ReLIFT is the closest prior method. HDPO uses the model's own privileged rollouts instead of external solutions and a single objective instead of a two-phase loop.
- **Disclosure:** the author reports extensive help from Claude with the proofs, the paper text and the code. The experiments themselves were run without LLM involvement.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

## Cite

Cite as: Ken Ding. 2026. HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation. arXiv:2603.23871.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ding2026hdpo,
  title = {{HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation}},
  author = {Ken Ding},
  year = {2026},
  doi = {10.48550/arxiv.2603.23871},
  eprint = {2603.23871},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2603.23871}
}
```

- [← previous H-MEM: Hierarchical Memory for High-Efficiency Long-Term Reasoning in LLM Agents](https://md.research.elamur.ai/free/en/papers/sun-2026-h-mem-hierarchical-memory-high/)
- [next → Justice in Judgment: Unveiling (Hidden) Bias in LLM-assisted Peer Reviews](https://md.research.elamur.ai/free/en/papers/vasu-2026-justice-judgment-unveiling-hidden-bias/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/)

## References

33 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. [On-policy distillation of language models: Learning from self-generated mistakes](https://md.research.elamur.ai/free/en/papers/agarwal-2023-policy-distillation-language-models-learning/). In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2306.13649. [↗](https://arxiv.org/abs/2306.13649)
2. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, et al. [Process reinforcement through implicit rewards](https://md.research.elamur.ai/free/en/papers/cui-2025-process-reinforcement-through-implicit-rewards/). arXiv preprint arXiv:2502.01456, 2025. URL https://arxiv.org/abs/2502.01456. [↗](https://arxiv.org/abs/2502.01456)
3. DeepSeek-AI, Daya Guo, Dejian Yang, et al. [DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948, 2025. URL https://arxiv.org/abs/2501.12948. [↗](https://arxiv.org/abs/2501.12948)
4. Shihan Dou, Muling Wu, Jingwen Xu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. Improving RL exploration for LLM reasoning through retrospective replay. arXiv preprint arXiv:2504.14363, 2025. URL https://arxiv.org/abs/2504.14363. — [arXiv:2504.14363](https://arxiv.org/abs/2504.14363) [↗](https://arxiv.org/abs/2504.14363)
5. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the MATH dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Neural Information Processing Systems, 2021. URL https://arxiv.org/abs/2103. 03874. [↗](https://arxiv.org/abs/2103)
6. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). arXiv preprint arXiv:1503.02531, 2015. URL https://arxiv.org/abs/1503.02531. [↗](https://arxiv.org/abs/1503.02531)
7. Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause. [Reinforcement learning via self-distillation](https://md.research.elamur.ai/free/en/papers/hubotter-2026-reinforcement-learning-self-distillation/). arXiv preprint arXiv:2601.20802, 2026. URL https://arxiv.org/abs/2601.2080… [↗](https://arxiv.org/abs/2601.2080…)
8. Guochao Jiang, Wenfeng Feng, Guofeng Quan, Chuzhan Hao, Yuewei Zhang, Guohua Liu, and Hao Wang. VCRL: Variance-based curriculum reinforcement learning for large language models. arXiv preprint arXiv:2509.19803, 2025. URL https://arxiv.org/abs/ 2509.19803. — [arXiv:2509.19803](https://arxiv.org/abs/2509.19803) [↗](https://arxiv.org/abs/)
9. Hyunjik Kim, George Papamakarios, and Andriy Mnih. The Lipschitz constant of self-attention. In International Conference on Machine Learning, 2021. URL [https://arxiv.org/](https://arxiv.org/) abs/2006.04710. — [arXiv:2006.04710](https://arxiv.org/abs/2006.04710) [↗](https://arxiv.org/)
10. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. [Efficient memory management for large language model serving with PagedAttention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Symposium on Operating Systems Principles, 2023. URL https://arxiv.org/abs/2309.06180. [↗](https://arxiv.org/abs/2309.06180)
11. Thanh-Long V. Le, Myeongho Jeon, Kim Vu, Viet Lai, and Eunho Yang. No prompt left behind: Exploiting zero-variance prompts in LLM reinforcement learning via entropy-guided advantage shaping. arXiv preprint arXiv:2509.21880, 2025. URL https://arxiv.org/abs/2509.21880. — [arXiv:2509.21880](https://arxiv.org/abs/2509.21880) [↗](https://arxiv.org/abs/2509.21880)
12. Long Li, Zhijian Zhou, Jiaran Hao, Jason Klein Liu, Yanting Miao, Wei Pang, Xiaoyu Tan, Wei Chu, Zhe Wang, Shirui Pan, Chao Qu, and Yuan Qi. The choice of divergence: A neglected key to mitigating diversity collapse in reinforcement learning with verifiable reward. arXiv preprint arXiv:2509.07430, 20… — [arXiv:2509.07430](https://arxiv.org/abs/2509.07430) [find](https://scholar.google.com/scholar?q=Long+Li%2C+Zh%C4%B3ian+Zhou%2C+Jiaran+Hao%2C+Jason+Klein+Liu%2C+Yanting+Miao%2C+Wei+Pang%2C+Xiaoyu+Tan%2C+Wei+Chu%2C+Zhe+Wang%2C+Shirui+Pan%2C+Chao+Qu%2C+and+Yuan+Qi%2E+The+choice+of+divergence%3A+A+neglected+key+to+mitigating+dive)
13. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, 2024. [URL https://arxiv.org/abs/ 2305.20050](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). [↗](https://arxiv.org/abs/)
14. Jianhua Lin. Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory, 37(1):145–151, 1991. URL [https://ieeexplore.ieee.org/document/](https://ieeexplore.ieee.org/document/) 61115/. [↗](https://ieeexplore.ieee.org/document/)
15. Huanyu Liu, Jia Li, Chang Yu, Taozhi Chen, Yihong Dong, Lecheng Wang, Xiaolong Hu, and Ge Li. EvoCoT: Overcoming the exploration bottleneck in reinforcement learning. arXiv preprint arXiv:2508.07809, 2025. URL https://arxiv.org/abs/2508.07809. — [arXiv:2508.07809](https://arxiv.org/abs/2508.07809) [↗](https://arxiv.org/abs/2508.07809)
16. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://arxiv.org/abs/1711.05101. [↗](https://arxiv.org/abs/1711.05101)
17. Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Yanhao Li, Bin Cui, and Wentao Zhang. Learning what reinforcement learning can’t: Interleaved online fine-tuning for hardest questions. arXiv preprint arXiv:2506.07527, 2025. URL https://arx… — [arXiv:2506.07527](https://arxiv.org/abs/2506.07527) [find](https://scholar.google.com/scholar?q=Lu+Ma%2C+Hao+Liang%2C+Meiyi+Qiang%2C+Lexiang+Tang%2C+Xiaochen+Ma%2C+Zhen+Hao+Wong%2C+Junbo+Niu%2C+Chengyu+Shen%2C+Runming+He%2C+Yanhao+Li%2C+Bin+Cui%2C+and+Wentao+Zhang%2E+Learning+what+reinforcement+learning+can%E2%80%99t%3A+Interlea)
18. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). In Neural Information Processing Systems, 2022. URL https://arxiv.org/abs/2203.02… [↗](https://arxiv.org/abs/2203.02…)
19. Emiliano Penaloza, Dheeraj Vattikonda, Nicolas Gontier, Alexandre Lacoste, Laurent Charlin, and Massimo Caccia. Privileged information distillation for language models. arXiv preprint arXiv:2602.04942, 2026. URL https://arxiv.org/abs/2602.04942. — [arXiv:2602.04942](https://arxiv.org/abs/2602.04942) [↗](https://arxiv.org/abs/2602.04942)
20. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2305. 18290. [↗](https://arxiv.org/abs/2305)
21. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. URL https://arxiv.org/abs/1707.06347. [↗](https://arxiv.org/abs/1707.06347)
22. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. [DeepSeekMath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300, 2024. URL https://arxiv.org/abs/2402.03300. [↗](https://arxiv.org/abs/2402.03300)
23. Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. [OpenMathInstruct-2: Accelerating AI for math with massive open-source instruction data](https://md.research.elamur.ai/free/en/papers/toshniwal-2024-openmathinstruct-2-accelerating-ai-math/). arXiv preprint arXiv:2410.01560, 2024. URL https://arxiv.org/abs/ 2410.01560. [↗](https://arxiv.org/abs/)
24. Vladimir Vapnik and Akshay Vashist. A new learning paradigm: Learning using privileged information. Neural Networks, 22(5–6):544–557, 2009. doi: 10.1016/j.neunet.2009.06.042. — [doi:10.1016/j.neunet.2009.06.042](https://doi.org/10.1016/j.neunet.2009.06.042) [find](https://scholar.google.com/scholar?q=Vladimir+Vapnik+and+Akshay+Vashist%2E+A+new+learning+paradigm%3A+Learning+using+privileged+information%2E+Neural+Networks%2C+22%285%E2%80%936%29%3A544%E2%80%93557%2C+2009%2E+doi%3A+10%2E1016%2Fj%2Eneunet%2E2009%2E06%2E042%2E+%E2%80%94+%5Bdoi%3A10%2E1016%2Fj%2Eneunet%2E2)
25. Xinyi Wang, Jinyi Han, Zishang Jiang, Tingyun Li, Jiaqing Liang, Sihang Jiang, Zhaoqian Dai, Shuguang Ma, Fei Yu, and Yanghua Xiao. HINT: Helping ineffective rollouts navigate towards effectiveness. arXiv preprint arXiv:2510.09388, 2025. URL [https://arxiv.org/](https://arxiv.org/) abs/2510.09388. — [arXiv:2510.09388](https://arxiv.org/abs/2510.09388) [↗](https://arxiv.org/)
26. Hongling Xu, Qi Zhu, Heyuan Deng, Jinpeng Li, Lu Hou, Yasheng Wang, Lifeng Shang, Ruifeng Xu, and Fei Mi. KDRL: Post-training reasoning LLMs via unified knowledge distillation and reinforcement learning. arXiv preprint arXiv:2506.02208, 2025. URL https://arxiv.org/abs/2506.02208. — [arXiv:2506.02208](https://arxiv.org/abs/2506.02208) [↗](https://arxiv.org/abs/2506.02208)
27. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. [Qwen2.5-Math technical report: Toward mathematical expert model via self-improvement](https://md.research.elamur.ai/free/en/papers/yang-2024-qwen2-5-math-technical-report/). arXiv p…
28. Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125, 2026. URL https://arxiv.org/abs/2602.12125. — [arXiv:2602.12125](https://arxiv.org/abs/2602.12125) [↗](https://arxiv.org/abs/2602.12125)
29. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, et al. [DAPO: An open-source LLM reinforcement learning system at scale](https://md.research.elamur.ai/free/en/papers/yu-2025-dapo-open-source-llm-reinforcement/). arXiv preprint arXiv:2503.14476, 2025. URL https://arxiv.org/abs/2503.14476. [↗](https://arxiv.org/abs/2503.14476)
30. Hongzhi Zhang, Jia Fu, Jingyuan Zhang, Kai Fu, Qi Wang, Fuzheng Zhang, and Guorui Zhou. RLEP: Reinforcement learning with experience replay for LLM reasoning. arXiv preprint arXiv:2507.07451, 2025a. URL https://arxiv.org/abs/2507.07451. — [arXiv:2507.07451](https://arxiv.org/abs/2507.07451) [↗](https://arxiv.org/abs/2507.07451)
31. Xichen Zhang, Sitong Wu, Yinghao Zhu, Haoru Tan, Shaozuo Yu, Ziyi He, and Jiaya Jia. Scaf-GRPO: Scaffolded group relative policy optimization for enhancing LLM reasoning. arXiv preprint arXiv:2510.19807, 2025b. URL https://arxiv.org/abs/2510.19807. — [arXiv:2510.19807](https://arxiv.org/abs/2510.19807) [↗](https://arxiv.org/abs/2510.19807)
32. Zhaoyang Zhang, Shuli Jiang, Yantao Shen, Yuting Zhang, Dhananjay Ram, Shuo Yang, Zhuowen Tu, Wei Xia, and Stefano Soatto. [Reinforcement-aware knowledge distillation for LLM reasoning](https://md.research.elamur.ai/free/en/papers/zhang-2026-reinforcement-aware-knowledge-distillation-llm/). arXiv preprint arXiv:2602.22495, 2026. URL https://arxiv.org/abs/ 2602.22495. [↗](https://arxiv.org/abs/)
33. Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. [Self-distilled reasoner: On-policy self-distillation for large language models](https://md.research.elamur.ai/free/en/papers/zhao-2026-self-distilled-reasoner-policy-self/). arXiv preprint arXiv:2601.18734, 2026. URL https://arxiv.org/abs/2601.18734. [↗](https://arxiv.org/abs/2601.18734)

*Updated 3 November 2026*