---
title: "Asymmetric On-Policy Distillation — paper by Jia et al. (2026)"
description: "The authors show three weaknesses in this scheme. 4.09 points. 37 references, cited by 6 papers here."
authors: ["Nan Jia", "Haojin Yang", "Xing Ma", "Jiesong Lian", "Shuailiang Zhang", "Weipeng Zhang", "Ke Zeng", "Xunliang Cai", "Zequn Sun"]
year: 2026
venue: "arXiv:2605.06387"
source: "https://arxiv.org/abs/2605.06387"
doi: "10.48550/arxiv.2605.06387"
arxiv: "2605.06387"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jia-2026-asymmetric-policy-distillation-bridging-exploitation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jia-2026-asymmetric-policy-distillation-bridging-exploitation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/) / Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level

# Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level

**Authors:** Nan Jia, Haojin Yang, [Xing Ma](https://md.research.elamur.ai/free/en/authors/xing-ma/), Jiesong Lian, Shuailiang Zhang, Weipeng Zhang, [Ke Zeng](https://md.research.elamur.ai/free/en/authors/ke-zeng/), [Xunliang Cai](https://md.research.elamur.ai/free/en/authors/xunliang-cai/) and [Zequn Sun](https://md.research.elamur.ai/free/en/authors/zequn-sun/) · **Year:** 2026 · **Venue:** arXiv:2605.06387 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2605.06387) · [PDF](https://arxiv.org/pdf/2605.06387) · [DOI 10.48550/arxiv.2605.06387](https://doi.org/10.48550/arxiv.2605.06387) · [arXiv 2605.06387](https://arxiv.org/abs/2605.06387) · [OpenAlex](https://openalex.org/W7160627520)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [6](https://md.research.elamur.ai/free/en/papers/jia-2026-asymmetric-policy-distillation-bridging-exploitation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
On-policy distillation (OPD) lets a small student learn from its own sampled outputs, with a teacher scoring each token by the gap between their log-probabilities. The authors show three weaknesses in this scheme. Negative advantages have a heavy tail, which makes updates noisy. Most tokens sit near zero advantage and so give almost no gradient. When a mistake is penalized, the freed probability goes to tokens the student already likes, so a correct but unlikely token is rarely promoted. They call this an "exploration black hole," and it is worst for weakly initialized students.

Their fix, asymmetric on-policy distillation (AOPD), keeps the usual policy-gradient update for tokens with positive advantage. For tokens with zero or negative advantage, it instead applies forward KL matching to the teacher's top-K distribution at that prefix.

On competition math benchmarks, AOPD beats OPD across model scales and warm-up lengths. It also keeps policy entropy higher and forgets less during later tool-use training. This makes it a practical upgrade for post-training compact LLMs.

## Key points
- Average improvement over standard OPD is 4.09 points with strong initialization and 8.34 with weak initialization (limited SFT warm-up).
- Evaluation covers AIME 2024, AIME 2025 and HMMT 2025 (Feb), reported as Pass@1 and Pass@4.
- Teacher support is the top-K tokens, with K = 32 as the default.
  - Smaller K speeds up early gains.
  - Larger K gives a higher final score.
  - Going beyond 32 adds little and slows training.
- A threshold τ controls where KL guidance applies.
  - τ = 0 is the default, covering the non-positive region.
  - τ = −1 reduces the method to OPD, and τ = 1 recovers a GKD-style objective.
  - Shrinking the intervened set (τ = −0.2, filtering by answer correctness) lowers results, while guiding only zero-advantage tokens (AOPD-Zero) still helps.
- Roughly 40% of tokens receive KL guidance early in training, falling to about 30% later. Under the same comparison, OPD suffers entropy collapse while AOPD keeps entropy higher.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 335 works · 43,759 citations
- [Nanjing University](https://md.research.elamur.ai/free/en/universities/nanjing-university/) — University, Nanjing, China · 102 works · 8,589 citations
- [Huazhong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/huazhong-university-of-science-and-technology/) — University, Wuhan, China · 64 works · 8,427 citations
- [Meituan](https://md.research.elamur.ai/free/en/companies/meituan/) — Company, Beijing, China · 28 works · 3,965 citations

Also on the byline: [King University](https://ror.org/01evb6z23).

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 328 works

## Cite

Cite as: Nan Jia, Haojin Yang, Xing Ma, Jiesong Lian, Shuailiang Zhang, Weipeng Zhang, Ke Zeng, Xunliang Cai and Zequn Sun. 2026. Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level. arXiv:2605.06387.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{jia2026asymmetric,
  title = {{Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level}},
  author = {Nan Jia, Haojin Yang, Xing Ma, Jiesong Lian, Shuailiang Zhang, Weipeng Zhang, Ke Zeng, Xunliang Cai and Zequn Sun},
  year = {2026},
  doi = {10.48550/arxiv.2605.06387},
  eprint = {2605.06387},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2605.06387}
}
```

- [← previous Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents](https://md.research.elamur.ai/free/en/papers/yu-2026-agentic-memory-learning-unified-long/)
- [next → CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges](https://md.research.elamur.ai/free/en/papers/wang-2026-creativebench-benchmarking-enhancing-machine-creativity/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/)

## References

37 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. 2024. [On-policy distillation of language models: Learning from self-generated mistakes](https://md.research.elamur.ai/free/en/papers/agarwal-2023-policy-distillation-language-models-learning/). In The Twelfth International Conference on Learning Representations.
2. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems, volume 28, pages 1171–1179, Cambridge, MA, USA. MIT Press. [find](https://scholar.google.com/scholar?q=Samy+Bengio%2C+Oriol+Vinyals%2C+Navdeep+Jaitly%2C+and+Noam+Shazeer%2E+2015%2E+Scheduled+sampling+for+sequence+prediction+with+recurrent+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+vol)
3. Jack Chen, Fazhong Liu, Naruto Liu, Yuhan Luo, Erqu Qin, Harry Zheng, Tian Dong, Haojin Zhu, Yan Meng, and Xiao Wang. 2025. Step-wise adaptive integration of supervised fine-tuning and reinforcement learning for task-specific llms. Preprint, arXiv:2505.13026. — [arXiv:2505.13026](https://arxiv.org/abs/2505.13026) [find](https://scholar.google.com/scholar?q=Jack+Chen%2C+Fazhong+Liu%2C+Naruto+Liu%2C+Yuhan+Luo%2C+Erqu+Qin%2C+Harry+Zheng%2C+Tian+Dong%2C+Haojin+Zhu%2C+Yan+Meng%2C+and+Xiao+Wang%2E+2025%2E+Step%2Dwise+adaptive+integration+of+supervised+fine%2Dtuning+and+reinforcement+l)
4. Liang Chen, Xueting Han, Li Shen, Jing Bai, and Kam-Fai Wong. 2026. Beyond two-stage training: Cooperative SFT and RL for LLM reasoning. In Forty-third International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Liang+Chen%2C+Xueting+Han%2C+Li+Shen%2C+Jing+Bai%2C+and+Kam%2DFai+Wong%2E+2026%2E+Beyond+two%2Dstage+training%3A+Cooperative+SFT+and+RL+for+LLM+reasoning%2E+In+Forty%2Dthird+International+Conference+on+Machine+Learning%2E)
5. DeepSeek-AI, Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chengyu Hou, Chenhao Xu, Chenze Shao, Chong Ruan, Conner Sun, and 300 others. 2026. [Deepseek-v4: Towards highly efficient…](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2026-deepseek-v4-highly-efficient/)
6. Yuqian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao. 2026. SRFT: A single-stage method with supervised and reinforcement fine-tuning for reasoning. In The Fourteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yuqian+Fu%2C+Tinghong+Chen%2C+Jiajun+Chai%2C+Xihuai+Wang%2C+Songjun+Tu%2C+Guojun+Yin%2C+Wei+Lin%2C+Qichao+Zhang%2C+Yuanheng+Zhu%2C+and+Dongbin+Zhao%2E+2026%2E+SRFT%3A+A+single%2Dstage+method+with+supervised+and+reinforcement+f)
7. GLM-5-Team, Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, Chenzheng Zhu, Congfeng Yin, Cunxiang Wang, Gengzheng Pan, Hao Zeng, Haoke Zhang, Haoran Wang, Huilong Chen, and 167 others. 2026. [Glm-5: from vibe coding to agentic en…](https://md.research.elamur.ai/free/en/papers/team-2026-5-glm-5-vibe-coding/)
8. Naibin Gu, Chenxu Yang, Qingyi Si, Chuanyu Qin, Dingyu Yao, Peng Fu, Zheng Lin, Weiping Wang, Nan Duan, and Jiaqi Wang. 2026. Co-evolving policy distillation. Preprint, arXiv:2604.27083. [find](https://scholar.google.com/scholar?q=Naibin+Gu%2C+Chenxu+Yang%2C+Qingyi+Si%2C+Chuanyu+Qin%2C+Dingyu+Yao%2C+Peng+Fu%2C+Zheng+Lin%2C+Weiping+Wang%2C+Nan+Duan%2C+and+Jiaqi+Wang%2E+2026%2E+Co%2Devolving+policy+distillation%2E+Preprint%2C+arXiv%3A2604%2E27083%2E)
9. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. MiniLLM: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yuxian+Gu%2C+Li+Dong%2C+Furu+Wei%2C+and+Minlie+Huang%2E+2024%2E+MiniLLM%3A+Knowledge+distillation+of+large+language+models%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2E)
10. Etash Kumar Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Rea Sprague, Ashima Suvarna, Benjamin Feuer, Leon Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, and 32 other… — [OpenThoughts: Data Recipes for Reasoning Models](https://md.research.elamur.ai/free/en/papers/guha-2025-openthoughts-data-recipes-reasoning-models/)
11. Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. 2026. [Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing…](https://md.research.elamur.ai/free/en/papers/he-2025-deepmath-103k-large-scale-challenging/)
12. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). Preprint, arXiv:1503.02531.
13. Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. [Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes](https://md.research.elamur.ai/free/en/papers/hsieh-2023-distilling-step-step-outperforming-larger/). In Findings of the Association for Computati…
14. Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Yinghui Xu, Edoardo M. Ponti, and Ivan Titov. 2026. Blending supervised and reinforcement fine-tuning with prefix sampling. In Forty-third International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Zeyu+Huang%2C+Tianhao+Cheng%2C+Zihan+Qiu%2C+Zili+Wang%2C+Yinghui+Xu%2C+Edoardo+M%2E+Ponti%2C+and+Ivan+Titov%2E+2026%2E+Blending+supervised+and+reinforcement+fine%2Dtuning+with+prefix+sampling%2E+In+Forty%2Dthird+Internationa)
15. Woogyeol Jin, Taywon Min, Yongjin Yang, Dennis Wei, Yi Zhou, Swanand Ravindra Kadhe, Nathalie Baracaldo, and Kimin Lee. 2026. [Entropy-aware on-policy distillation of language models](https://md.research.elamur.ai/free/en/papers/jin-2026-entropy-aware-policy-distillation-language/). In Forty-third International Conference on Machine Learning.
16. Yoon Kim and Alexander M. Rush. 2016. [Sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, Austin, Texas. Association for Computational Linguistics.
17. Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, S. H. Cai, Yuan Cao, Ziwei Chai, Y. Charles, H. S. Che, Cheng Chen, Guanduo Chen, Huarong Chen, Jia Chen, Jianlong Chen, Jun Chen, Kefan Chen, Liang Chen, Ruijue Chen, Xinhao Chen, and 318 others. 2026. [Kimi k2.5: Visual agentic intelligence](https://md.research.elamur.ai/free/en/papers/kimi-team-2026-kimi-k2-5-visual/). Preprint,…
18. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, pages 61…
19. Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan ang Gao, Wenkai Yang, Zhiyuan Liu, and Ning Ding. 2026. [Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe](https://md.research.elamur.ai/free/en/papers/li-2026-rethinking-policy-distillation-large-language/). Preprint, arXiv:2604.13016.
20. Kevin Lu and Thinking Machines Lab. 2025. On-Policy distillation. Thinking Machines Lab: Connectionism. [find](https://scholar.google.com/scholar?q=Kevin+Lu+and+Thinking+Machines+Lab%2E+2025%2E+On%2DPolicy+distillation%2E+Thinking+Machines+Lab%3A+Connectionism%2E)
21. Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Yanhao Li, Bin Cui, and Wentao Zhang. 2026. Learning what reinforcement learning can’t: Interleaved online fine-tuning for hardest questions. In The Fourteenth International Conference on Le… [find](https://scholar.google.com/scholar?q=Lu+Ma%2C+Hao+Liang%2C+Meiyi+Qiang%2C+Lexiang+Tang%2C+Xiaochen+Ma%2C+Zhen+Hao+Wong%2C+Junbo+Niu%2C+Chengyu+Shen%2C+Runming+He%2C+Yanhao+Li%2C+Bin+Cui%2C+and+Wentao+Zhang%2E+2026%2E+Learning+what+reinforcement+learning+can%E2%80%99t%3A+In)
22. Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. 2026. [Self-distillation enables continual learning](https://md.research.elamur.ai/free/en/papers/shenfeld-2026-self-distillation-enables-continual-learning/). In Forty-third International Conference on Machine Learning.
23. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. [Hybridflow: A flexible and efficient rlhf framework](https://md.research.elamur.ai/free/en/papers/sheng-2024-hybridflow-flexible-efficient-rlhf-framework/). In Proceedings of the Twentieth European Conference on Computer Systems, page 1279–1297. ACM.
24. Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. 2023. [ToolAlpaca: Generalized tool learning for language models with 3000 simulated cases](https://md.research.elamur.ai/free/en/papers/tang-2023-toolalpaca-generalized-tool-learning-language/). arXiv preprint arXiv:2306.05301.
25. Xinyu Tang, Yuliang Zhan, Zhixun Li, Xin Zhao, Zhenduo Zhang, Zujie Wen, Zhiqiang Zhang, and Jun Zhou. 2026. [Rethinking sample polarity in reinforcement learning with verifiable rewards](https://md.research.elamur.ai/free/en/papers/tang-2026-rethinking-sample-polarity-reinforcement-learning/). In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers…
26. Yijun Tian, Yikun Han, Xiusi Chen, Wei Wang, and Nitesh V. Chawla. 2025. Beyond answers: Transferring reasoning capabilities to smaller LLMs using multi-teacher knowledge distillation. In Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, WSDM ’25, pages 251–26… [find](https://scholar.google.com/scholar?q=Yijun+Tian%2C+Yikun+Han%2C+Xiusi+Chen%2C+Wei+Wang%2C+and+Nitesh+V%2E+Chawla%2E+2025%2E+Beyond+answers%3A+Transferring+reasoning+capabilities+to+smaller+LLMs+using+multi%2Dteacher+knowledge+distillation%2E+In+Proceedings+)
27. Xiaomi LLM-Core Team, Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, Gang Xie, Hailin Zhang, Hanglong Lv, Hanyu Li, Heyu Chen, Hongshen Xu, Houbin Zhang, Huaqiu Liu, and 107 others. 2026. [Mimo-v2-flash technical report](https://md.research.elamur.ai/free/en/papers/team-2026-llm-mimo-v2-flash-technical/). Preprint…
28. Hongling Xu, Qi Zhu, Heyuan Deng, Jinpeng Li, Lu Hou, Yasheng Wang, Lifeng Shang, Ruifeng Xu, and Fei Mi. 2025. Kdrl: Post-training reasoning llms via unified knowledge distillation and reinforcement learning. Preprint, arXiv:2506.02208. — [arXiv:2506.02208](https://arxiv.org/abs/2506.02208) [find](https://scholar.google.com/scholar?q=Hongling+Xu%2C+Qi+Zhu%2C+Heyuan+Deng%2C+Jinpeng+Li%2C+Lu+Hou%2C+Yasheng+Wang%2C+Lifeng+Shang%2C+Ruifeng+Xu%2C+and+Fei+Mi%2E+2025%2E+Kdrl%3A+Post%2Dtraining+reasoning+llms+via+unified+knowledge+distillation+and+reinforcement+)
29. Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. 2024. A survey on knowledge distillation of large language models. Preprint, arXiv:2402.13116. — [arXiv:2402.13116](https://arxiv.org/abs/2402.13116) [find](https://scholar.google.com/scholar?q=Xiaohan+Xu%2C+Ming+Li%2C+Chongyang+Tao%2C+Tao+Shen%2C+Reynold+Cheng%2C+Jinyang+Li%2C+Can+Xu%2C+Dacheng+Tao%2C+and+Tianyi+Zhou%2E+2024%2E+A+survey+on+knowledge+distillation+of+large+language+models%2E+Preprint%2C+arXiv%3A2402%2E1)
30. Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. 2025. [Learning to reason under off-policy guidance](https://md.research.elamur.ai/free/en/papers/yan-2025-learning-reason-under-off-policy/). In The Thirty-ninth Annual Conference on Neural Information Processing Systems.
31. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). Preprint, arXiv:2505.09388.
32. Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. 2026. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation. Preprint, arXiv:2602.12125. — [arXiv:2602.12125](https://arxiv.org/abs/2602.12125) [find](https://scholar.google.com/scholar?q=Wenkai+Yang%2C+Weijie+Liu%2C+Ruobing+Xie%2C+Kai+Yang%2C+Saiyong+Yang%2C+and+Yankai+Lin%2E+2026%2E+Learning+beyond+teacher%3A+Generalized+on%2Dpolicy+distillation+with+reward+extrapolation%2E+Preprint%2C+arXiv%3A2602%2E12125%2E+%E2%80%94)
33. Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou. 2026a. On-policy RL meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting. In The Fourteenth International Conference on Learning Repres… [find](https://scholar.google.com/scholar?q=Wenhao+Zhang%2C+Yuexiang+Xie%2C+Yuchang+Sun%2C+Yanxi+Chen%2C+Guoyin+Wang%2C+Yaliang+Li%2C+Bolin+Ding%2C+and+Jingren+Zhou%2E+2026a%2E+On%2Dpolicy+RL+meets+off%2Dpolicy+experts%3A+Harmonizing+supervised+fine%2Dtuning+and+reinfor)
34. Xinsen Zhang, Zhenkai Ding, Tianjun Pan, Run Yang, Chun Kang, Xue Xiong, and Jingnan Gu. 2026b. Opsdl: On-policy self-distillation for long-context language models. Preprint, arXiv:2604.17535. — [arXiv:2604.17535](https://arxiv.org/abs/2604.17535) [find](https://scholar.google.com/scholar?q=Xinsen+Zhang%2C+Zhenkai+Ding%2C+Tianjun+Pan%2C+Run+Yang%2C+Chun+Kang%2C+Xue+Xiong%2C+and+Jingnan+Gu%2E+2026b%2E+Opsdl%3A+On%2Dpolicy+self%2Ddistillation+for+long%2Dcontext+language+models%2E+Preprint%2C+arXiv%3A2604%2E17535%2E+%E2%80%94+%5BarXi)
35. Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. 2026. [Self-distilled reasoner: On-policy self-distillation for large language models](https://md.research.elamur.ai/free/en/papers/zhao-2026-self-distilled-reasoner-policy-self/). In Forty-third International Conference on Machine Learning.
36. Binbin Zheng, Xing Ma, Yiheng Liang, Jingqing Ruan, Xiaoliang Fu, Kepeng Lin, Benchang Zhu, Ke Zeng, and Xunliang Cai. 2026. Scope: Signal-calibrated on-policy distillation enhancement with dual-path adaptive weighting. Preprint, arXiv:2604.10688. — [arXiv:2604.10688](https://arxiv.org/abs/2604.10688) [find](https://scholar.google.com/scholar?q=Binbin+Zheng%2C+Xing+Ma%2C+Yiheng+Liang%2C+Jingqing+Ruan%2C+Xiaoliang+Fu%2C+Kepeng+Lin%2C+Benchang+Zhu%2C+Ke+Zeng%2C+and+Xunliang+Cai%2E+2026%2E+Scope%3A+Signal%2Dcalibrated+on%2Dpolicy+distillation+enhancement+with+dual%2Dpath+)
37. Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. 2025. The surprising effectiveness of negative reinforcement in LLM reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Xinyu+Zhu%2C+Mengzhou+Xia%2C+Zhepei+Wei%2C+Wei%2DLin+Chen%2C+Danqi+Chen%2C+and+Yu+Meng%2E+2025%2E+The+surprising+effectiveness+of+negative+reinforcement+in+LLM+reasoning%2E+In+The+Thirty%2Dninth+Annual+Conference+on+Neur)

*Updated 3 November 2026*