---
title: "Learning to Reason under Off-Policy Guidance — Yan 2025"
description: "The authors propose LUFFY, which injects reasoning traces from a stronger model, such as… 65 references, cited by 61 papers here."
authors: ["Jianhao Yan", "Yafu Li", "Zican Hu", "Zhi Wang", "Ganqu Cui", "Xiaoye Qu", "Yu Cheng", "Yue Zhang"]
year: 2025
venue: "arXiv:2504.14945"
source: "https://arxiv.org/abs/2504.14945"
doi: "10.48550/arxiv.2504.14945"
arxiv: "2504.14945"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yan-2025-learning-reason-under-off-policy/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yan-2025-learning-reason-under-off-policy/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) / Learning to Reason under Off-Policy Guidance

# Learning to Reason under Off-Policy Guidance

**Authors:** [Jianhao Yan](https://md.research.elamur.ai/free/en/authors/jianhao-yan/), [Yafu Li](https://md.research.elamur.ai/free/en/authors/yafu-li/), Zican Hu, [Zhi Wang](https://md.research.elamur.ai/free/en/authors/zhi-wang/), [Ganqu Cui](https://md.research.elamur.ai/free/en/authors/ganqu-cui/), [Xiaoye Qu](https://md.research.elamur.ai/free/en/authors/xiaoye-qu/), [Yu Cheng](https://md.research.elamur.ai/free/en/authors/yu-cheng/) and [Yue Zhang](https://md.research.elamur.ai/free/en/authors/yue-zhang/) · **Year:** 2025 · **Venue:** [arXiv:2504.14945](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2025](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2025/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2504.14945) · [PDF](https://arxiv.org/pdf/2504.14945) · [DOI 10.48550/arxiv.2504.14945](https://doi.org/10.48550/arxiv.2504.14945) · [arXiv 2504.14945](https://arxiv.org/abs/2504.14945) · [OpenAlex](https://openalex.org/W4416556500)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [61](https://md.research.elamur.ai/free/en/papers/yan-2025-learning-reason-under-off-policy/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Reinforcement learning with verifiable rewards (RLVR) has driven recent progress in large reasoning models. However, it learns only from samples the model generates itself. As a result, it tends to strengthen behaviors the base model already has rather than teach new ones. The authors propose LUFFY, which injects reasoning traces from a stronger model, such as DeepSeek-R1, into training. Its core is Mixed-Policy GRPO: external traces join the model's own rollouts in the same group when advantages are computed. The model therefore leans on the stronger traces when its own attempts fail and favors its own solutions once they succeed. A second component, policy shaping through regularized importance sampling, boosts the learning signal for rare but important tokens. This keeps entropy from collapsing and discourages shallow copying of the external traces. Across six math benchmarks and several out-of-distribution tasks, LUFFY beats earlier RLVR methods. It can also train weaker base models on which standard on-policy RL fails. The work suggests that external guidance can push models past their initial reasoning limits.

## Key points
- On-policy RLVR baselines are outperformed by about +6.4 points on average across AIME24/25, AMC, OlympiadBench, Minerva and MATH-500, and by over +6.2 points on out-of-distribution tasks. The authors report a new RLVR state of the art with Qwen2.5-Math-7B.
- The Mixed-Policy GRPO objective uses a separate importance ratio against the off-policy model. It comes with a convergence guarantee of rate O(1/√K) under clipped importance weights.
- LUFFY trains LLaMA3.1-8B across a range of difficulty levels. Plain on-policy RL works for this model only on simplified data.
- Generation entropy under on-policy RL drops to near zero after roughly 200 steps. LUFFY keeps entropy higher throughout training, and without policy shaping, mixed-policy training plateaus.
- Code and project page: https://github.com/ElliottYan/LUFFY

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations
- [Chinese University of Hong Kong](https://md.research.elamur.ai/free/en/universities/chinese-university-of-hong-kong/) — University, Hong Kong, China · 159 works · 16,154 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 152 works · 13,164 citations
- [Nanjing University](https://md.research.elamur.ai/free/en/universities/nanjing-university/) — University, Nanjing, China · 99 works · 8,510 citations
- [Westlake University](https://md.research.elamur.ai/free/en/universities/westlake-university/) — University, Hangzhou, China · 51 works · 8,572 citations

Also on the byline: [Suzhou University](https://ror.org/02bz8aa76).

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works

## Cite

Cite as: Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng and Yue Zhang. 2025. Learning to Reason under Off-Policy Guidance. arXiv:2504.14945.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{yan2025learning,
  title = {{Learning to Reason under Off-Policy Guidance}},
  author = {Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng and Yue Zhang},
  year = {2025},
  doi = {10.48550/arxiv.2504.14945},
  eprint = {2504.14945},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2504.14945}
}
```

- [← previous Competitive Programming with Large Reasoning Models](https://md.research.elamur.ai/free/en/papers/el-kishky-2025-competitive-programming-large-reasoning-models/)
- [next → MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL](https://md.research.elamur.ai/free/en/papers/wang-2025-mac-sql-multi-agent-collaborative/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/)

## References

65 references, 37 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. [Openai o1 system card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/). arXiv preprint arXiv:2412.16720, 2024.
2. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948, 2025.
3. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.[12599, 2025](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k1-5-scaling-reinforcement/).
4. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Pro… — [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
5. Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. [https://hkust-nlp.notion.site/simplerl-reason](https://hkust-nlp.notion.site/simplerl-reason), 2025. Notion Blog. [↗](https://hkust-nlp.notion.site/simplerl-reason)
6. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. [Understanding r1-zero-like training: A critical perspective](https://md.research.elamur.ai/free/en/papers/liu-2025-understanding-r1-zero-like-training/). arXiv preprint arXiv:2503.20783, 2025.
7. Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. [Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model](https://md.research.elamur.ai/free/en/papers/hu-2025-open-reasoner-zero-open-source/), 2025.
8. Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: Rl post-training amplifies behaviors learned in pretraining, 2025. [find](https://scholar.google.com/scholar?q=Rosie+Zhao%2C+Alexandru+Meterez%2C+Sham+Kakade%2C+Cengiz+Pehlevan%2C+Samy+Jelassi%2C+and+Eran+Malach%2E+Echo+chamber%3A+Rl+post%2Dtraining+amplifies+behaviors+learned+in+pretraining%2C+2025%2E)
9. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. [Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?](https://md.research.elamur.ai/free/en/papers/yue-2025-reinforcement-learning-really-incentivize-reasoning/), 2025.
10. Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. [Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars](https://md.research.elamur.ai/free/en/papers/gandhi-2025-cognitive-behaviors-that-enable-self/), 2025.
11. Meta AI. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models, September 2024. 15 minute read. [find](https://scholar.google.com/scholar?q=Meta+AI%2E+Llama+3%2E2%3A+Revolutionizing+edge+ai+and+vision+with+open%2C+customizable+models%2C+September+2024%2E+15+minute+read%2E)
12. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/), 2024.
13. Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q. Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. [https://huggingface.co/datasets/Numinamath…](https://huggingface.co/datasets/Numinamath…) [↗](https://huggingface.co/datasets/Numinamath…)
14. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. [Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems](https://md.research.elamur.ai/free/en/papers/he-2024-olympiadbench-challenging-benchmark-promoting-agi/). In Proceedings of the 62nd Annual Meeting of the Assoc…
15. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). Advances in Neural Information Processing Systems, 35:3843–3857, 2022.
16. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874, 2021.
17. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. [Concrete problems in ai safety](https://md.research.elamur.ai/free/en/papers/amodei-2016-concrete-problems-ai-safety/). arXiv preprint arXiv:1606.06565, 2016.
18. Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460– 9471, 2022. [find](https://scholar.google.com/scholar?q=Joar+Skalse%2C+Nikolaus+Howe%2C+Dmitrii+Krasheninnikov%2C+and+David+Krueger%2E+Defining+and+characterizing+reward+gaming%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A9460%E2%80%93+9471%2C+2022%2E)
19. Leo Gao, John Schulman, and Jacob Hilton. [Scaling laws for reward model overoptimization](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/). In International Conference on Machine Learning, pages 10835–10866. PMLR, 2023.
20. Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. [Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild](https://md.research.elamur.ai/free/en/papers/zeng-2025-simplerl-zoo-investigating-taming-zero/), 2025.
21. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)
22. Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton%2C+David+McAllester%2C+Satinder+Singh%2C+and+Yishay+Mansour%2E+Policy+gradient+methods+for+reinforcement+learning+with+function+approximation%2E+Advances+in+neural+information+processing+system)
23. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889–1897. PMLR, 2015. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Sergey+Levine%2C+Pieter+Abbeel%2C+Michael+Jordan%2C+and+Philipp+Moritz%2E+Trust+region+policy+optimization%2E+In+International+conference+on+machine+learning%2C+pages+1889%E2%80%931897%2E+PMLR%2C+2015%2E)
24. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. [Process reinforcement through implicit rewards](https://md.research.elamur.ai/free/en/papers/cui-2025-process-reinforcement-through-implicit-rewards/). arXiv preprint arXiv:2502.01456, 2025.
25. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan D… — [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://md.research.elamur.ai/free/en/papers/yu-2025-dapo-open-source-llm-reinforcement/)
26. Sashank J Reddi, Ahmed Hefny, Suvrit Sra, Barnabas Poczos, and Alex Smola. Stochastic variance reduction for nonconvex optimization. In ICML, pages 314–323, 2016. [find](https://scholar.google.com/scholar?q=Sashank+J+Reddi%2C+Ahmed+Hefny%2C+Suvrit+Sra%2C+Barnabas+Poczos%2C+and+Alex+Smola%2E+Stochastic+variance+reduction+for+nonconvex+optimization%2E+In+ICML%2C+pages+314%E2%80%93323%2C+2016%2E)
27. Wenjia Meng, Qian Zheng, Gang Pan, and Yilong Yin. Off-policy proximal policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 9162–9170, 2023. [find](https://scholar.google.com/scholar?q=Wenjia+Meng%2C+Qian+Zheng%2C+Gang+Pan%2C+and+Yilong+Yin%2E+Off%2Dpolicy+proximal+policy+optimization%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+37%2C+pages+9162%E2%80%939170%2C+2023%2E)
28. Hugging Face. Open r1: A fully open reproduction of deepseek-r1, January 2025. [find](https://scholar.google.com/scholar?q=Hugging+Face%2E+Open+r1%3A+A+fully+open+reproduction+of+deepseek%2Dr1%2C+January+2025%2E)
29. Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. [https://huggingface.co/AI-MO/NuminaMath-1.5](https://github.co… [find](https://scholar.google.com/scholar?q=Jia+LI%2C+Edward+Beeching%2C+Lewis+Tunstall%2C+Ben+Lipkin%2C+Roman+Soletskyi%2C+Shengyi+Costa+Huang%2C+Kashif+Rasul%2C+Longhui+Yu%2C+Albert+Jiang%2C+Ziju+Shen%2C+Zihan+Qin%2C+Bin+Dong%2C+Li+Zhou%2C+Yann+Fleureau%2C+Guillaume+Lam)
30. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. [Qwen2.5-math technical report: Toward mathematical expert model via self-improvement](https://md.research.elamur.ai/free/en/papers/yang-2024-qwen2-5-math-technical-report/), 2024.
31. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+Yang%2C+Jin)
32. Meta Team. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/), 2024.
33. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv:1803.05457v1, 2018.
34. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. [GPQA: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). In First Conference on Language Modeling, 2024.
35. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. [Mmlu-pro: A more robust and challenging multi-task language understanding benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/). arXiv preprint arXiv:2406.01574, 2024.
36. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. [s1: Simple test-time scaling](https://md.research.elamur.ai/free/en/papers/muennighoff-2025-s1-simple-test-time-scaling/). arXiv preprint arXiv:2501.19393, 2025.
37. Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. [Limo: Less is more for reasoning](https://md.research.elamur.ai/free/en/papers/ye-2025-limo-less-more-reasoning/). arXiv preprint arXiv:2502.03387, 2025.
38. Xiao Hu, Xingyu Lu, Liyuan Mao, YiFan Zhang, Tianke Zhang, Bin Wen, Fan Yang, Tingting Gao, and Guorui Zhou. Why distillation can outperform zero-rl: The role of flexible reasoning, 2025. [find](https://scholar.google.com/scholar?q=Xiao+Hu%2C+Xingyu+Lu%2C+Liyuan+Mao%2C+YiFan+Zhang%2C+Tianke+Zhang%2C+Bin+Wen%2C+Fan+Yang%2C+Tingting+Gao%2C+and+Guorui+Zhou%2E+Why+distillation+can+outperform+zero%2Drl%3A+The+role+of+flexible+reasoning%2C+2025%2E)
39. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Sali… — [Phi-4 Technical Report](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-4-technical-report/)
40. Xiaoye Qu, Yafu Li, Zhaochen Su, Weigao Sun, Jianhao Yan, Dongrui Liu, Ganqu Cui, Daizong Liu, Shuxian Liang, Junxian He, Peng Li, Wei Wei, Jing Shao, Chaochao Lu, Yue Zhang, Xian-Sheng Hua, Bowen Zhou, and Yu Cheng. A survey of efficient reasoning for large reasoning models: Language, multimodality… — [A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond](https://md.research.elamur.ai/free/en/papers/qu-2025-survey-efficient-reasoning-large-reasoning/)
41. Yaru Hao, Li Dong, Xun Wu, Shaohan Huang, Zewen Chi, and Furu Wei. On-policy rl with optimal reward baseline, 2025. [find](https://scholar.google.com/scholar?q=Yaru+Hao%2C+Li+Dong%2C+Xun+Wu%2C+Shaohan+Huang%2C+Zewen+Chi%2C+and+Furu+Wei%2E+On%2Dpolicy+rl+with+optimal+reward+baseline%2C+2025%2E)
42. Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. [Ttrl: Test-time reinforcement learning](https://md.research.elamur.ai/free/en/papers/zuo-2025-ttrl-test-time-reinforcement-learning/). arXiv preprint arXiv:2504.16084, 2025.
43. Bo Pang, Hanze Dong, Jiacheng Xu, Silvio Savarese, Yingbo Zhou, and Caiming Xiong. Bolt: Bootstrap long chain-of-thought in language models without distillation. arXiv preprint arXiv:2502.03860, 2025. — [arXiv:2502.03860](https://arxiv.org/abs/2502.03860) [find](https://scholar.google.com/scholar?q=Bo+Pang%2C+Hanze+Dong%2C+Jiacheng+Xu%2C+Silvio+Savarese%2C+Yingbo+Zhou%2C+and+Caiming+Xiong%2E+Bolt%3A+Bootstrap+long+chain%2Dof%2Dthought+in+language+models+without+distillation%2E+arXiv+preprint+arXiv%3A2502%2E03860%2C+2025%2E)
44. Mehdi Fatemi, Banafsheh Rafiee, Mingjie Tang, and Kartik Talamadupula. Concise reasoning via reinforcement learning. arXiv preprint arXiv:2504.05185, 2025. — [arXiv:2504.05185](https://arxiv.org/abs/2504.05185) [find](https://scholar.google.com/scholar?q=Mehdi+Fatemi%2C+Banafsheh+Rafiee%2C+Mingjie+Tang%2C+and+Kartik+Talamadupula%2E+Concise+reasoning+via+reinforcement+learning%2E+arXiv+preprint+arXiv%3A2504%2E05185%2C+2025%2E+%E2%80%94+%5BarXiv%3A2504%2E05185%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
45. Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. [Absolute zero: Reinforced self-play reasoning with zero data](https://md.research.elamur.ai/free/en/papers/zhao-2025-absolute-zero-reinforced-self-play/). arXiv preprint arXiv:2505.03335, 2025.
46. Xuerui Su, Shufang Xie, Guoqing Liu, Yingce Xia, Renqian Luo, Peiran Jin, Zhiming Ma, Yue Wang, Zun Wang, and Yuting Liu. Trust region preference approximation: A simple and stable reinforcement learning algorithm for llm reasoning. arXiv preprint arXiv:2504.04524, 2025. — [arXiv:2504.04524](https://arxiv.org/abs/2504.04524) [find](https://scholar.google.com/scholar?q=Xuerui+Su%2C+Shufang+Xie%2C+Guoqing+Liu%2C+Yingce+Xia%2C+Renqian+Luo%2C+Peiran+Jin%2C+Zhiming+Ma%2C+Yue+Wang%2C+Zun+Wang%2C+and+Yuting+Liu%2E+Trust+region+preference+approximation%3A+A+simple+and+stable+reinforcement+learn)
47. Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. [Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond](https://md.research.elamur.ai/free/en/papers/wen-2025-light-r1-curriculum-sft-dpo/). arXiv preprint arXiv:2503.10460, 2025.
48. Wang Yang, Hongye Jin, Jingfeng Yang, Vipin Chaudhary, and Xiaotian Han. Thinking preference optimization. arXiv preprint arXiv:2502.13173, 2025. — [arXiv:2502.13173](https://arxiv.org/abs/2502.13173) [find](https://scholar.google.com/scholar?q=Wang+Yang%2C+Hongye+Jin%2C+Jingfeng+Yang%2C+Vipin+Chaudhary%2C+and+Xiaotian+Han%2E+Thinking+preference+optimization%2E+arXiv+preprint+arXiv%3A2502%2E13173%2C+2025%2E+%E2%80%94+%5BarXiv%3A2502%2E13173%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2502%2E13173%29)
49. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928–1937. PmLR, 2016. [find](https://scholar.google.com/scholar?q=Volodymyr+Mnih%2C+Adria+Puigdomenech+Badia%2C+Mehdi+Mirza%2C+Alex+Graves%2C+Timothy+Lillicrap%2C+Tim+Harley%2C+David+Silver%2C+and+Koray+Kavukcuoglu%2E+Asynchronous+methods+for+deep+reinforcement+learning%2E+In+Interna)
50. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. [find](https://scholar.google.com/scholar?q=Volodymyr+Mnih%2C+Koray+Kavukcuoglu%2C+David+Silver%2C+Alex+Graves%2C+Ioannis+Antonoglou%2C+Daan+Wierstra%2C+and+Martin+Riedmiller%2E+Playing+atari+with+deep+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1312%2E5602%2C+)
51. Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587–1596. PMLR, 2018. [find](https://scholar.google.com/scholar?q=Scott+Fujimoto%2C+Herke+Hoof%2C+and+David+Meger%2E+Addressing+function+approximation+error+in+actor%2Dcritic+methods%2E+In+International+conference+on+machine+learning%2C+pages+1587%E2%80%931596%2E+PMLR%2C+2018%2E)
52. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. Pmlr, 2018. [find](https://scholar.google.com/scholar?q=Tuomas+Haarnoja%2C+Aurick+Zhou%2C+Pieter+Abbeel%2C+and+Sergey+Levine%2E+Soft+actor%2Dcritic%3A+Off%2Dpolicy+maximum+entropy+deep+reinforcement+learning+with+a+stochastic+actor%2E+In+International+conference+on+machin)
53. Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton%2C+David+McAllester%2C+Satinder+Singh%2C+and+Yishay+Mansour%2E+Policy+gradient+methods+for+reinforcement+learning+with+function+approximation%2E+Advances+in+neural+information+processing+system)
54. Huayu Chen, Kaiwen Zheng, Qinsheng Zhang, Ganqu Cui, Yin Cui, Haotian Ye, Tsung-Yi Lin, Ming-Yu Liu, Jun Zhu, and Haoxiang Wang. Bridging supervised learning and reinforcement learning in math reasoning. arXiv preprint arXiv:2505.18116, 2025. — [arXiv:2505.18116](https://arxiv.org/abs/2505.18116) [find](https://scholar.google.com/scholar?q=Huayu+Chen%2C+Kaiwen+Zheng%2C+Qinsheng+Zhang%2C+Ganqu+Cui%2C+Yin+Cui%2C+Haotian+Ye%2C+Tsung%2DYi+Lin%2C+Ming%2DYu+Liu%2C+Jun+Zhu%2C+and+Haoxiang+Wang%2E+Bridging+supervised+learning+and+reinforcement+learning+in+math+reasoni)
55. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). Advances in Neural Information Processing Systems, 36:53728–53741, 2023.
56. Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501.05444, 2025. — [arXiv:2501.05444](https://arxiv.org/abs/2501.05444) [find](https://scholar.google.com/scholar?q=Yunzhuo+Hao%2C+Jiawei+Gu%2C+Huichen+Will+Wang%2C+Linjie+Li%2C+Zhengyuan+Yang%2C+Lijuan+Wang%2C+and+Yu+Cheng%2E+Can+mllms+reason+in+multimodality%3F+emma%3A+An+enhanced+multimodal+reasoning+benchmark%2E+arXiv+preprint+arX)
57. Tingchen Fu, Jiawei Gu, Yafu Li, Xiaoye Qu, and Yu Cheng. [Scaling reasoning, losing control: Evaluating instruction following in large reasoning models](https://md.research.elamur.ai/free/en/papers/fu-2026-scaling-reasoning-losing-control-evaluating/). arXiv preprint arXiv:2505.14810, 2025.
58. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. [Deep learning scaling is predictable, empirically](https://md.research.elamur.ai/free/en/papers/hestness-2017-deep-learning-scaling-predictable-empirically/). arXiv preprint arXiv:1712.00409, 2017.
59. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. [Scaling laws for autoregressive generative modeling](https://md.research.elamur.ai/free/en/papers/henighan-2020-scaling-laws-autoregressive-generative-modeling/). arXiv preprint arXiv:2010.14701, 2020.
60. Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, 2 edition, 2018. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton+and+Andrew+G+Barto%2E+Reinforcement+Learning%3A+An+Introduction%2E+MIT+Press%2C+2+edition%2C+2018%2E)
61. Philipp Koehn. [Statistical significance tests for machine translation evaluation](https://md.research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/). In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 388–395, 2004.
62. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Pierre Isabelle, Eugene Charniak, and Dekang Lin, editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelp…
63. Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test-time compute without verification or RL is suboptimal. In ICLR 2025 Workshop: VerifAI: AI Verification in the Wild, 2025. [find](https://scholar.google.com/scholar?q=Amrith+Setlur%2C+Nived+Rajaraman%2C+Sergey+Levine%2C+and+Aviral+Kumar%2E+Scaling+test%2Dtime+compute+without+verification+or+RL+is+suboptimal%2E+In+ICLR+2025+Workshop%3A+VerifAI%3A+AI+Verification+in+the+Wild%2C+2025%2E)
64. Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. [Sft memorizes, rl generalizes: A comparative study of foundation model post-training](https://md.research.elamur.ai/free/en/papers/chu-2025-sft-memorizes-rl-generalizes-comparative/), 2025.
65. Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. [Sft or rl? an early investigation into training r1-like reasoning large vision-language models](https://md.research.elamur.ai/free/en/papers/chen-2025-sft-rl-early-investigation-training/). [https://github.com/UCSC-VLAA/VLAA-Thinking](https://github.com/UCSC-VLAA/VLAA-Thinking), 2025. [↗](https://github.com/UCSC-VLAA/VLAA-Thinking)

*Updated 2 November 2026*