---
title: "Unmasking On-Policy Distillation: Where It Helps, Where It Hurts,"
description: "The Apple authors propose an offline, training-free diagnostic instead. 31 references, cited by 7 papers here."
authors: ["Mohammadreza Armandpour", "Fatih Ilhan", "David Harrison", "Ajay Jaiswal", "Duc N.M Hoang", "Fartash Faghri", "Yizhe Zhang", "Minsik Cho", "Mehrdad Farajtabar"]
year: 2026
venue: "arXiv:2605.10889"
source: "https://arxiv.org/abs/2605.10889"
doi: "10.48550/arxiv.2605.10889"
arxiv: "2605.10889"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/armandpour-2026-unmasking-policy-distillation-where-helps/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/armandpour-2026-unmasking-policy-distillation-where-helps/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/) / Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why

# Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why

**Authors:** Mohammadreza Armandpour, [Fatih Ilhan](https://md.research.elamur.ai/free/en/authors/fatih-ilhan/), David Harrison, Ajay Jaiswal, Duc N.M Hoang, [Fartash Faghri](https://md.research.elamur.ai/free/en/authors/fartash-faghri/), [Yizhe Zhang](https://md.research.elamur.ai/free/en/authors/yizhe-zhang/), [Minsik Cho](https://md.research.elamur.ai/free/en/authors/minsik-cho/) and [Mehrdad Farajtabar](https://md.research.elamur.ai/free/en/authors/mehrdad-farajtabar/) · **Year:** 2026 · **Venue:** arXiv:2605.10889 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2605.10889) · [PDF](https://arxiv.org/pdf/2605.10889) · [DOI 10.48550/arxiv.2605.10889](https://doi.org/10.48550/arxiv.2605.10889) · [arXiv 2605.10889](https://arxiv.org/abs/2605.10889) · [OpenAlex](https://openalex.org/W7160898890)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/armandpour-2026-unmasking-policy-distillation-where-helps/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
On-policy distillation gives a student model dense, per-token supervision, either from a larger external teacher or from the student itself conditioned on privileged context such as a correct solution. Choosing among these options usually requires costly training runs whose aggregate scores hide what happens at individual tokens. The Apple authors propose an offline, training-free diagnostic instead. From a tree of student rollouts, they estimate how often each token choice leads to a correct answer. From this they derive an "ideal" logit gradient that most increases success, and they show Dr. GRPO matches this direction in expectation. A targeted-rollout procedure with exponentially growing depth windows keeps compute tied to a chosen budget, even for long reasoning traces. Distillation gradients such as GKD and MiniLLM are then scored by cosine similarity to the ideal. Experiments on Qwen3-0.6B and Qwen3-1.7B show that teacher signals are much more helpful on failed rollouts than on successful ones. The best teacher or context depends on student size and task difficulty, so the authors argue for per-task diagnostics rather than one fixed recipe.

## Key points
- **Setup:** Qwen3-0.6B and Qwen3-1.7B students are evaluated with 8 teacher configurations (self-distillation variants and external teachers) on BoolQ and MMLU. Four AIME 2025 questions with traces of roughly 5K–30K tokens serve as case studies.
- **Failed vs. correct rollouts:** Alignment is consistently higher on incorrect rollouts. For example, the best teacher on AIME question Q0 scored +0.097 on incorrect paths versus −0.011 on correct ones.
- **Context and capacity interact:** A summarized solution roughly doubles alignment for the 1.7B student but slightly hurts the 0.6B student, which needs full step-by-step reasoning. External teachers beat self-distillation only for the larger student, and only on BoolQ.
- **Wrong demonstrations:** Adding an incorrect example (Self-1C1W) hurts on the short-reasoning benchmarks but gives the best alignment on the harder AIME problems.
- **Divergence as a predictor:** Teacher–student divergence (KL, JS, L2) correlates positively with alignment, but only weakly (|ρ| ≈ 0.02–0.04). It is useful as a cheap filter, not as a reliable predictor.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 328 works

## Cite

Cite as: Mohammadreza Armandpour, Fatih Ilhan, David Harrison, Ajay Jaiswal, Duc N.M Hoang, Fartash Faghri, Yizhe Zhang, Minsik Cho and Mehrdad Farajtabar. 2026. Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why. arXiv:2605.10889.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{armandpour2026unmasking,
  title = {{Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why}},
  author = {Mohammadreza Armandpour, Fatih Ilhan, David Harrison, Ajay Jaiswal, Duc N.M Hoang, Fartash Faghri, Yizhe Zhang, Minsik Cho and Mehrdad Farajtabar},
  year = {2026},
  doi = {10.48550/arxiv.2605.10889},
  eprint = {2605.10889},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2605.10889}
}
```

- [← previous TSPO: Breaking the Double Homogenization Dilemma in Multi-turn Search Policy Optimization](https://md.research.elamur.ai/free/en/papers/ma-2026-tspo-breaking-double-homogenization-dilemma/)
- [next → ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios](https://md.research.elamur.ai/free/en/papers/loison-2026-vidore-v3-comprehensive-evaluation-retrieval/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/)

## References

31 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. [On-policy distillation of language models: Learning from self-generated mistakes](https://md.research.elamur.ai/free/en/papers/agarwal-2023-policy-distillation-language-models-learning/). In International Conference on Learning Representations, volume 2024, pages 21246–21263, 2024.
2. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28, 2015. [find](https://scholar.google.com/scholar?q=Samy+Bengio%2C+Oriol+Vinyals%2C+Navdeep+Jaitly%2C+and+Noam+Shazeer%2E+Scheduled+sampling+for+sequence+prediction+with+recurrent+neural+networks%2E+Advances+in+neural+information+processing+systems%2C+28%2C+2015%2E)
3. Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb. [Distillation scaling laws](https://md.research.elamur.ai/free/en/papers/busbridge-2025-distillation-scaling-laws/). arXiv preprint arXiv:2502.08606, 2025.
4. Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4793–4801. [find](https://scholar.google.com/scholar?q=Jang+Hyun+Cho+and+Bharath+Hariharan%2E+On+the+efficacy+of+knowledge+distillation%2E+In+2019+IEEE%2FCVF+International+Conference+on+Computer+Vision+%28ICCV%29%2C+pages+4793%E2%80%934801%2E)
5. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human…
6. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In International Conference on Learning Representations, volume 2024, pages 32694–32717, 2024. [find](https://scholar.google.com/scholar?q=Yuxian+Gu%2C+Li+Dong%2C+Furu+Wei%2C+and+Minlie+Huang%2E+Minillm%3A+Knowledge+distillation+of+large+language+models%2E+In+International+Conference+on+Learning+Representations%2C+volume+2024%2C+pages+32694%E2%80%9332717%2C+2024%2E)
7. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948, 2025.
8. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300, 2020.
9. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. arXiv:1503.[02531, 2015](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/).
10. Distilling the knowledge in a neural network. arXiv preprint Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. [Reinforcement learning via self-distillation](https://md.research.elamur.ai/free/en/papers/hubotter-2026-reinforcement-learning-self-distillation/). arXiv preprint arXiv:2601.…
11. Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. Why does self-distillation (sometimes) degrade the reasoning capability of llms? arXiv preprint arXiv:2603.24472, 2026. — [arXiv:2603.24472](https://arxiv.org/abs/2603.24472) [find](https://scholar.google.com/scholar?q=Jeonghye+Kim%2C+Xufang+Luo%2C+Minbeom+Kim%2C+Sangmook+Lee%2C+Dohyung+Kim%2C+Jiwon+Jeon%2C+Dongsheng+Li%2C+and+Yuqing+Yang%2E+Why+does+self%2Ddistillation+%28sometimes%29+degrade+the+reasoning+capability+of+llms%3F+arXiv+prep)
12. Yoon Kim and Alexander M Rush. [Sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/). In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 1317–1327, 2016.
13. Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, et al. [Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe](https://md.research.elamur.ai/free/en/papers/li-2026-rethinking-policy-distillation-large-language/). arXiv preprint arXiv:2604.13016, 2026.
14. Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, and Radha Poovendran. [Small models struggle to learn from strong reasoners](https://md.research.elamur.ai/free/en/papers/li-2025-small-models-struggle-learn-strong/). In Findings of the Association for Computational Linguistics: ACL 2025, pages 25366–25394, 2025.
15. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, volume 2024, [pages 39578–39601, 2024](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/).
16. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. [Understanding r1-zero-like training: A critical perspective](https://md.research.elamur.ai/free/en/papers/liu-2025-understanding-r1-zero-like-training/). arXiv preprint arXiv:2503.20783, 2025.
17. Kevin Lu and Thinking Machines Lab. On-policy distillation. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.20251026. [https://thinkingmachines.ai/blog/on-policy-distillation](https://thinkingmachines.ai/blog/on-policy-distillation). — [doi:10.64434/tml.20251026](https://doi.org/10.64434/tml.20251026) [↗](https://thinkingmachines.ai/blog/on-policy-distillation)
18. Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. [Improved knowledge distillation via teacher assistant](https://md.research.elamur.ai/free/en/papers/mirzadeh-2019-improved-knowledge-distillation-teacher-assistant/). In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 5191–5198, 2020.
19. Emiliano Penaloza, Dheeraj Vattikonda, Nicolas Gontier, Alexandre Lacoste, Laurent Charlin, and Massimo Caccia. Privileged information distillation for language models. arXiv preprint arXiv:2602.04942, 2026. — [arXiv:2602.04942](https://arxiv.org/abs/2602.04942) [find](https://scholar.google.com/scholar?q=Emiliano+Penaloza%2C+Dheeraj+Vattikonda%2C+Nicolas+Gontier%2C+Alexandre+Lacoste%2C+Laurent+Charlin%2C+and+Massimo+Caccia%2E+Privileged+information+distillation+for+language+models%2E+arXiv+preprint+arXiv%3A2602%2E04942)
20. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300, 2024.
21. Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. [Self-distillation enables continual learning](https://md.research.elamur.ai/free/en/papers/shenfeld-2026-self-distillation-enables-continual-learning/). arXiv preprint arXiv:2601.19897, 2026.
22. Charlie Snell, Dan Klein, and Ruiqi Zhong. [Learning by distilling context](https://md.research.elamur.ai/free/en/papers/snell-2022-learning-distilling-context/). arXiv preprint arXiv:2209.15189, 2022.
23. Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.[14275, 2022](https://md.research.elamur.ai/free/en/papers/uesato-2022-solving-math-word-problems-process/).
24. Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, et al. [Mimo-v2-flash technical report](https://md.research.elamur.ai/free/en/papers/team-2026-llm-mimo-v2-flash-technical/). arXiv preprint arXiv:2601.02780, 2026.
25. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388, 2025.
26. Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125, 2026. — [arXiv:2602.12125](https://arxiv.org/abs/2602.12125) [find](https://scholar.google.com/scholar?q=Wenkai+Yang%2C+Weijie+Liu%2C+Ruobing+Xie%2C+Kai+Yang%2C+Saiyong+Yang%2C+and+Yankai+Lin%2E+Learning+beyond+teacher%3A+Generalized+on%2Dpolicy+distillation+with+reward+extrapolation%2E+arXiv+preprint+arXiv%3A2602%2E12125%2C+20)
27. Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. [On-policy context distillation for language models](https://md.research.elamur.ai/free/en/papers/ye-2026-policy-context-distillation-language-models/). arXiv preprint arXiv:2602.12275, 2026.
28. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. [Dapo: An open-source llm reinforcement learning system at scale](https://md.research.elamur.ai/free/en/papers/yu-2025-dapo-open-source-llm-reinforcement/). Advances in Neural Information Processing Systems, 38:113222–113244, 2026.
29. Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. [Glm-5: from vibe coding to agentic engineering](https://md.research.elamur.ai/free/en/papers/team-2026-5-glm-5-vibe-coding/). arXiv preprint arXiv:2602.15763, 2026.
30. Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. [Self-distilled reasoner: On-policy self-distillation for large language models](https://md.research.elamur.ai/free/en/papers/zhao-2026-self-distilled-reasoner-policy-self/). arXiv preprint arXiv:2601.18734, 2026.
31. A Full AIME 2025 Case Studies A.1 [find](https://scholar.google.com/scholar?q=A+Full+AIME+2025+Case+Studies+A%2E1)

*Updated 3 November 2026*