---
title: "The Entropy Mechanism of Reinforcement Learning for"
description: "Building on this, they introduce Clip-Cov and KL-Cov, which restrict updates on high-covariance… 60 references, cited by 100 papers here."
authors: ["Ganqu Cui", "Yuchen Zhang", "Jiacheng Chen", "Lifan Yuan", "Zhi Wang", "Yuxin Zuo", "Haozhan Li", "Yuchen Fan", "Huayu Chen", "Weize Chen", "Zhiyuan Liu", "Hao Peng", "Lei Bai", "Wanli Ouyang", "Yu Cheng", "Bowen Zhou", "Ning Ding"]
year: 2025
venue: "arXiv:2505.22617"
source: "https://arxiv.org/abs/2505.22617"
doi: "10.48550/arxiv.2505.22617"
arxiv: "2505.22617"
licence: "CC BY-NC-ND 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-nd/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cui-2025-entropy-mechanism-reinforcement-learning-reasoning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cui-2025-entropy-mechanism-reinforcement-learning-reasoning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/) / The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models

# The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models

**Authors:** [Ganqu Cui](https://md.research.elamur.ai/free/en/authors/ganqu-cui/), [Yuchen Zhang](https://md.research.elamur.ai/free/en/authors/yuchen-zhang/), [Jiacheng Chen](https://md.research.elamur.ai/free/en/authors/jiacheng-chen/), [Lifan Yuan](https://md.research.elamur.ai/free/en/authors/lifan-yuan/), [Zhi Wang](https://md.research.elamur.ai/free/en/authors/zhi-wang/), [Yuxin Zuo](https://md.research.elamur.ai/free/en/authors/yuxin-zuo/), [Haozhan Li](https://md.research.elamur.ai/free/en/authors/haozhan-li/), [Yuchen Fan](https://md.research.elamur.ai/free/en/authors/yuchen-fan/), [Huayu Chen](https://md.research.elamur.ai/free/en/authors/huayu-chen/), [Weize Chen](https://md.research.elamur.ai/free/en/authors/weize-chen/), [Zhiyuan Liu](https://md.research.elamur.ai/free/en/authors/zhiyuan-liu/), [Hao Peng](https://md.research.elamur.ai/free/en/authors/hao-peng/), [Lei Bai](https://md.research.elamur.ai/free/en/authors/lei-bai/), [Wanli Ouyang](https://md.research.elamur.ai/free/en/authors/wanli-ouyang/), [Yu Cheng](https://md.research.elamur.ai/free/en/authors/yu-cheng/), [Bowen Zhou](https://md.research.elamur.ai/free/en/authors/bowen-zhou/) and [Ning Ding](https://md.research.elamur.ai/free/en/authors/ning-ding/) · **Year:** 2025 · **Venue:** arXiv:2505.22617 · **Licence:** [CC BY-NC-ND 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-nd/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2505.22617) · [PDF](https://arxiv.org/pdf/2505.22617) · [DOI 10.48550/arxiv.2505.22617](https://doi.org/10.48550/arxiv.2505.22617) · [arXiv 2505.22617](https://arxiv.org/abs/2505.22617) · [OpenAlex](https://openalex.org/W4416048648)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [100](https://md.research.elamur.ai/free/en/papers/cui-2025-entropy-mechanism-reinforcement-learning-reasoning/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Reinforcement learning with verifiable rewards is now a main way to improve LLM reasoning, but scaling it hits a recurring problem. Without explicit regularization, policy entropy falls to almost zero within the first training steps. The model becomes overconfident, stops exploring and its validation accuracy stalls.

Across many RL runs without entropy or KL terms, the authors found that downstream performance fits R = −a·exp(H) + b. In other words, reward is effectively bought with entropy, and the final ceiling (R = −a + b once H reaches 0) can be predicted early.

They then show theoretically that, for softmax policies, the entropy change between steps follows the covariance between an action's log-probability and its logit update. Under Policy Gradient–style algorithms, that update is proportional to the advantage. So confident, rewarded tokens drain entropy, and in practice this covariance stays mostly positive.

Building on this, they introduce Clip-Cov and KL-Cov, which restrict updates on high-covariance tokens. These methods keep entropy from collapsing, make it tunable, and improve mathematical reasoning results. This matters for spending larger RL compute budgets productively.

## Key points
- In uncontrolled runs, more than 95% of the entropy drop and of the performance gain occurs early in training, after which progress plateaus. Simply adding a standard entropy loss does not solve this.
- The empirical law R = −a·exp(H) + b links validation reward to entropy. The fitted coefficients reflect properties of the model and data and allow early performance forecasting.
- **Clip-Cov** detaches gradients for a small random subset of tokens with positive covariance. **KL-Cov** applies a KL penalty to the tokens with the highest covariance. Both replace the usual clip or PPO-KL terms in the surrogate loss.
- Tuning the clip ratio or the KL coefficient β adjusts entropy levels, as shown on Qwen2.5-7B. KL-Cov gives smoother entropy curves, and only about 10⁻⁴ to 10⁻³ of tokens are affected.
- The main baseline is clip-higher, which the authors interpret as admitting more low-covariance tokens (average covariance ≈ −0.03). Code is available at github.com/PRIME-RL/Entropy-Mechanism-of-RL.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations
- [University of Illinois Urbana-Champaign](https://md.research.elamur.ai/free/en/universities/university-of-illinois-urbana-champaign/) — University, Urbana, United States · 209 works · 71,394 citations
- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,267 citations
- [Chinese University of Hong Kong](https://md.research.elamur.ai/free/en/universities/chinese-university-of-hong-kong/) — University, Hong Kong, China · 159 works · 16,154 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 152 works · 13,166 citations
- [Nanjing University](https://md.research.elamur.ai/free/en/universities/nanjing-university/) — University, Nanjing, China · 99 works · 8,512 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou and Ning Ding. 2025. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv:2505.22617.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{cui2025entropy,
  title = {{The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models}},
  author = {Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou and Ning Ding},
  year = {2025},
  doi = {10.48550/arxiv.2505.22617},
  eprint = {2505.22617},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.22617}
}
```

- [← previous Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation](https://md.research.elamur.ai/free/en/papers/ren-2025-investigating-factual-knowledge-boundary-large/)
- [next → FineWeb2: One Pipeline to Scale Them All — Adapting Pre-Training Data Processing to Every Language](https://md.research.elamur.ai/free/en/papers/penedo-2025-fineweb2-one-pipeline-scale-them/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/)

## References

60 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98): 1–76, 2021. [find](https://scholar.google.com/scholar?q=Alekh+Agarwal%2C+Sham+M+Kakade%2C+Jason+D+Lee%2C+and+Gaurav+Mahajan%2E+On+the+theory+of+policy+gradient+methods%3A+Optimality%2C+approximation%2C+and+distribution+shift%2E+Journal+of+Machine+Learning+Research%2C+22%2898%29)
2. Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. The unreasonable effectiveness of entropy minimization in llm reasoning. arXiv preprint arXiv:2505.15134, 2025. [find](https://scholar.google.com/scholar?q=Shivam+Agarwal%2C+Zimin+Zhang%2C+Lifan+Yuan%2C+Jiawei+Han%2C+and+Hao+Peng%2E+The+unreasonable+effectiveness+of+entropy+minimization+in+llm+reasoning%2E+arXiv+preprint+arXiv%3A2505%2E15134%2C+2025%2E)
3. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024. [find](https://scholar.google.com/scholar?q=Arash+Ahmadian%2C+Chris+Cremer%2C+Matthias+Gall%C3%A9%2C+Marzieh+Fadaee%2C+Julia+Kreutzer%2C+Olivier+Pietquin%2C+Ahmet+%C3%9Cst%C3%BCn%2C+and+Sara+Hooker%2E+Back+to+basics%3A+Revisiting+reinforce+style+optimization+for+learning+from+)
4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
5. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through… — [Process Reinforcement through Implicit Rewards](https://md.research.elamur.ai/free/en/papers/cui-2025-process-reinforcement-through-implicit-rewards/)
6. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chengg… — [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/)
7. Benjamin Eysenbach and Sergey Levine. Maximum entropy rl (provably) solves some robust rl problems. arXiv preprint arXiv:2103.06257, 2021. — [arXiv:2103.06257](https://arxiv.org/abs/2103.06257) [find](https://scholar.google.com/scholar?q=Benjamin+Eysenbach+and+Sergey+Levine%2E+Maximum+entropy+rl+%28provably%29+solves+some+robust+rl+problems%2E+arXiv+preprint+arXiv%3A2103%2E06257%2C+2021%2E+%E2%80%94+%5BarXiv%3A2103%2E06257%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2103%2E06257%29)
8. Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. [Omni-math: A universal olympiad level mathematic benchmark for large language models](https://md.research.elamur.ai/free/en/papers/gao-2024-omni-math-universal-olympiad-level/). arXiv preprint arXiv:2410.07985, 2024.
9. Leo Gao, John Schulman, and Jacob Hilton. [Scaling laws for reward model overoptimization](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/). In International Conference on Machine Learning, 2022.
10. Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pp. 1352–1361. PMLR, 2017. [find](https://scholar.google.com/scholar?q=Tuomas+Haarnoja%2C+Haoran+Tang%2C+Pieter+Abbeel%2C+and+Sergey+Levine%2E+Reinforcement+learning+with+deep+energy%2Dbased+policies%2E+In+International+conference+on+machine+learning%2C+pp%2E+1352%E2%80%931361%2E+PMLR%2C+2017%2E)
11. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861–1870. [find](https://scholar.google.com/scholar?q=Tuomas+Haarnoja%2C+Aurick+Zhou%2C+Pieter+Abbeel%2C+and+Sergey+Levine%2E+Soft+actor%2Dcritic%3A+Off%2Dpolicy+maximum+entropy+deep+reinforcement+learning+with+a+stochastic+actor%2E+In+International+conference+on+machin)
12. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. [OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems](https://md.research.elamur.ai/free/en/papers/he-2024-olympiadbench-challenging-benchmark-promoting-agi/). In Lun-Wei Ku,…
13. Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reaonser series. [https://capricious-hydrogen-41c.notion.site/Skywork-Open-Reaon…](https://capricious-hydrogen-41c.notion.site/Skywork-Open-Reaon…) [↗](https://capricious-hydrogen-41c.notion.site/Skywork-Open-Reaon…)
14. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874, 2021.
15. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. [Deep learning scaling is predictable, empirically](https://md.research.elamur.ai/free/en/papers/hestness-2017-deep-learning-scaling-predictable-empirically/). arXiv preprint arXiv:1712.00409, 2017.
16. Jacob Hilton, Jie Tang, and John Schulman. Scaling laws for single-agent reinforcement learning. arXiv preprint arXiv:2301.13442, 2023. [find](https://scholar.google.com/scholar?q=Jacob+Hilton%2C+Jie+Tang%2C+and+John+Schulman%2E+Scaling+laws+for+single%2Dagent+reinforcement+learning%2E+arXiv+preprint+arXiv%3A2301%2E13442%2C+2023%2E)
17. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). arXiv preprint arXiv:2203.15556, 2022.
18. Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.[03262, 2025](https://md.research.elamur.ai/free/en/papers/hu-2025-reinforce-stabilizing-critic-free-policy/).
19. Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. [Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model](https://md.research.elamur.ai/free/en/papers/hu-2025-open-reasoner-zero-open-source/). arXiv preprint arXiv:2503.24290, 2025.
20. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
21. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. [find](https://scholar.google.com/scholar?q=Saurav+Kadavath%2C+Tom+Conerly%2C+Amanda+Askell%2C+Tom+Henighan%2C+Dawn+Drain%2C+Ethan+Perez%2C+Nicholas+Schiefer%2C+Zac+Hatfield%2DDodds%2C+Nova+DasSarma%2C+Eli+Tran%2DJohnson%2C+et+al%2E+Language+models+%28mostly%29+know+what+th)
22. Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001. [find](https://scholar.google.com/scholar?q=Sham+M+Kakade%2E+A+natural+policy+gradient%2E+Advances+in+neural+information+processing+systems%2C+14%2C+2001%2E)
23. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361, 2020.
24. Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get a baseline for free! In DeepRLStructPred@ICLR, 2019. URL [https://api.semanticscholar.org/CorpusID:198489118](https://api.semanticscholar.org/CorpusID:198489118). [↗](https://api.semanticscholar.org/CorpusID:198489118)
25. Nathan Lambert, Jacob Daniel Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Validad Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini,… — [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://md.research.elamur.ai/free/en/papers/lambert-2024-tulu-3-pushing-frontiers-open/)
26. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). Advances in Neural Information Processing Systems, 35:3843–3857, 2022.
27. Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024. [find](https://scholar.google.com/scholar?q=Jia+Li%2C+Edward+Beeching%2C+Lewis+Tunstall%2C+Ben+Lipkin%2C+Roman+Soletskyi%2C+Shengyi+Huang%2C+Kashif+Rasul%2C+Longhui+Yu%2C+Albert+Q+Jiang%2C+Ziju+Shen%2C+et+al%2E+Numinamath%3A+The+largest+public+dataset+in+ai4maths+with)
28. Jiacai Liu. How does rl policy entropy converge during iteration? [https://zhuanlan.zhihu.com/p/28476703733](https://zhuanlan.zhihu.com/p/28476703733), 2025. URL https://zhuanlan.zhihu.com/p/28476703733. [↗](https://zhuanlan.zhihu.com/p/28476703733)
29. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. [Understanding r1-zero-like training: A critical perspective](https://md.research.elamur.ai/free/en/papers/liu-2025-understanding-r1-zero-like-training/). arXiv preprint arXiv:2503.20783, 2025.
30. Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, et al. Deepscaler: Surpassing o1-preview with a 1.5 b model by scaling rl. Notion Blog, 2025. [find](https://scholar.google.com/scholar?q=Michael+Luo%2C+Sijun+Tan%2C+Justin+Wong%2C+Xiaoxiang+Shi%2C+William+Y+Tang%2C+Manan+Roongta%2C+Colin+Cai%2C+Jeffrey+Luo%2C+Tianjun+Zhang%2C+Li+Erran+Li%2C+et+al%2E+Deepscaler%3A+Surpassing+o1%2Dpreview+with+a+1%2E5+b+model+by+sc)
31. Meta. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/), 2024. URL https://arxiv.org/abs/2407.21783. [↗](https://arxiv.org/abs/2407.21783)
32. Meta-Llama-3.2. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. URL [https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices). MistralAI-NeMo. Mistralai-nemo. URL https://mistral.ai/news/mistral-nemo. MistralAI-Small-3. Mistralai-small-3. URL https://mi… [↗](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices)
33. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928–1937. [find](https://scholar.google.com/scholar?q=Volodymyr+Mnih%2C+Adria+Puigdomenech+Badia%2C+Mehdi+Mirza%2C+Alex+Graves%2C+Timothy+Lillicrap%2C+Tim+Harley%2C+David+Silver%2C+and+Koray+Kavukcuoglu%2E+Asynchronous+methods+for+deep+reinforcement+learning%2E+In+Interna)
34. OpenAI. [Openai o1 system card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/). ArXiv, abs/2412.16720, 2024a.
35. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2024b. URL https://arxiv.org/abs/2303.08774. [↗](https://arxiv.org/abs/2303.08774)
36. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 35:27730–27744, 2022.
37. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xu… [find](https://scholar.google.com/scholar?q=Qwen%2C+An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+Yan)
38. Oleh Rybkin, Michal Nauman, Preston Fu, Charlie Snell, Pieter Abbeel, Sergey Levine, and Aviral Kumar. Value-based deep rl scales predictably. arXiv preprint arXiv:2502.04327, 2025. — [arXiv:2502.04327](https://arxiv.org/abs/2502.04327) [find](https://scholar.google.com/scholar?q=Oleh+Rybkin%2C+Michal+Nauman%2C+Preston+Fu%2C+Charlie+Snell%2C+Pieter+Abbeel%2C+Sergey+Levine%2C+and+Aviral+Kumar%2E+Value%2Dbased+deep+rl+scales+predictably%2E+arXiv+preprint+arXiv%3A2502%2E04327%2C+2025%2E+%E2%80%94+%5BarXiv%3A2502%2E0432)
39. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889–1897. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Sergey+Levine%2C+Pieter+Abbeel%2C+Michael+Jordan%2C+and+Philipp+Moritz%2E+Trust+region+policy+optimization%2E+In+International+conference+on+machine+learning%2C+pp%2E+1889%E2%80%931897%2E)
40. John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q-learning. arXiv preprint arXiv:1704.06440, 2017a. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Xi+Chen%2C+and+Pieter+Abbeel%2E+Equivalence+between+policy+gradients+and+soft+q%2Dlearning%2E+arXiv+preprint+arXiv%3A1704%2E06440%2C+2017a%2E)
41. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017b. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017b%2E)
42. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/), 2024. URL https://arxiv.org/abs/2402.03300. [↗](https://arxiv.org/abs/2402.03300)
43. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. [Hybridflow: A flexible and efficient rlhf framework](https://md.research.elamur.ai/free/en/papers/sheng-2024-hybridflow-flexible-efficient-rlhf-framework/). arXiv preprint arXiv: 2409.19256, 2024.
44. David Silver and Richard S Sutton. Welcome to the era of experience. Google AI, 2025. [find](https://scholar.google.com/scholar?q=David+Silver+and+Richard+S+Sutton%2E+Welcome+to+the+era+of+experience%2E+Google+AI%2C+2025%2E)
45. Richard Sutton. The bitter lesson. Incomplete Ideas (blog), 13(1):38, 2019. [find](https://scholar.google.com/scholar?q=Richard+Sutton%2E+The+bitter+lesson%2E+Incomplete+Ideas+%28blog%29%2C+13%281%29%3A38%2C+2019%2E)
46. Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3:9–44, 1988. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton%2E+Learning+to+predict+by+the+methods+of+temporal+differences%2E+Machine+learning%2C+3%3A9%E2%80%9344%2C+1988%2E)
47. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
48. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.[12599, 2025](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k1-5-scaling-reinforcement/).
49. Marc Toussaint. Robot trajectory optimization using approximate inference. In Proceedings of the 26th annual international conference on machine learning, pp. 1049–1056, 2009. [find](https://scholar.google.com/scholar?q=Marc+Toussaint%2E+Robot+trajectory+optimization+using+approximate+inference%2E+In+Proceedings+of+the+26th+annual+international+conference+on+machine+learning%2C+pp%2E+1049%E2%80%931056%2C+2009%2E)
50. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229–256, 1992. [find](https://scholar.google.com/scholar?q=Ronald+J+Williams%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Machine+learning%2C+8%3A229%E2%80%93256%2C+1992%2E)
51. Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241–268, 1991. [find](https://scholar.google.com/scholar?q=Ronald+J+Williams+and+Jing+Peng%2E+Function+optimization+using+connectionist+reinforcement+learning+algorithms%2E+Connection+Science%2C+3%283%29%3A241%E2%80%93268%2C+1991%2E)
52. Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding, 2025. URL https://arxiv.org/abs/2503.02951. [↗](https://arxiv.org/abs/2503.02951)
53. Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. [Learning to reason under off-policy guidance](https://md.research.elamur.ai/free/en/papers/yan-2025-learning-reason-under-off-policy/). arXiv preprint arXiv:2504.14945, 2025.
54. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Honglin Yu, Weinan… — [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://md.research.elamur.ai/free/en/papers/yu-2025-dapo-open-source-llm-reinforcement/)
55. Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. [Free process rewards without process labels](https://md.research.elamur.ai/free/en/papers/yuan-2024-free-process-rewards-process-labels/). International conference on machine learning, 2025.
56. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. [Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?](https://md.research.elamur.ai/free/en/papers/yue-2025-reinforcement-learning-really-incentivize-reasoning/) ArXiv, abs/2504.13837, 2025.
57. Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. Acecoder: Acing coder rl via automated test-case synthesis, 2025. URL https://arxiv.org/abs/2502.01718. [↗](https://arxiv.org/abs/2502.01718)
58. Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised llm reasoning incentivization. arXiv preprint arXiv:2504.05812, 2025. [find](https://scholar.google.com/scholar?q=Qingyang+Zhang%2C+Haitao+Wu%2C+Changqing+Zhang%2C+Peilin+Zhao%2C+and+Yatao+Bian%2E+Right+question+is+already+half+the+answer%3A+Fully+unsupervised+llm+reasoning+incentivization%2E+arXiv+preprint+arXiv%3A2504%2E05812%2C+2)
59. Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433–1438. Chicago, IL, USA, 2008. [find](https://scholar.google.com/scholar?q=Brian+D+Ziebart%2C+Andrew+L+Maas%2C+J+Andrew+Bagnell%2C+Anind+K+Dey%2C+et+al%2E+Maximum+entropy+inverse+reinforcement+learning%2E+In+Aaai%2C+volume+8%2C+pp%2E+1433%E2%80%931438%2E+Chicago%2C+IL%2C+USA%2C+2008%2E)
60. Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. [Ttrl: Test-time reinforcement learning](https://md.research.elamur.ai/free/en/papers/zuo-2025-ttrl-test-time-reinforcement-learning/). arXiv preprint arXiv:2504.16084, 2025.

*Updated 2 November 2026*