---
title: "Understanding and Preventing Entropy Collapse in RLVR with"
description: "RLVR methods such as GRPO tend to suffer from entropy collapse. 38 references. Among them: 0."
authors: ["Huimin Xu", "Shuai Zhao", "Xiaobao Wu", "Anh Tuan Luu"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.879/"
doi: "10.18653/v1/2026.findings-acl.879"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2026-understanding-preventing-entropy-collapse-rlvr/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2026-understanding-preventing-entropy-collapse-rlvr/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/55/) / Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization

# Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization

**Authors:** Huimin Xu, [Shuai Zhao](https://md.research.elamur.ai/free/en/authors/shuai-zhao/), [Xiaobao Wu](https://md.research.elamur.ai/free/en/authors/xiaobao-wu/) and [Anh Tuan Luu](https://md.research.elamur.ai/free/en/authors/anh-tuan-luu/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.879/) · [PDF](https://aclanthology.org/2026.findings-acl.879.pdf) · [DOI 10.18653/v1/2026.findings-acl.879](https://doi.org/10.18653/v1/2026.findings-acl.879) · [OpenAlex](https://openalex.org/W7166842386)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
RLVR methods such as GRPO tend to suffer from entropy collapse. Early in training the policy becomes nearly deterministic, which limits exploration and destabilizes learning. Existing fixes have drawbacks. Entropy bonuses act bluntly and can inflate entropy late in training. Clipping heuristics like Clip-higher are only approximately on-policy. The authors analyze the problem at the token level using a first-order estimate of how each update changes entropy. They find that updates which lower entropy consistently outweigh those which raise it, and this imbalance explains collapse across existing algorithms. Their method, On-Policy Entropy Flow Optimization (OPEFO), uses an adaptive coefficient λ* to strengthen entropy-raising updates and weaken entropy-lowering ones, aiming for balanced net entropy flow. It stays strictly on-policy, with no reference policy or importance ratios. Across two base models and six math reasoning benchmarks, OPEFO gives the most stable entropy curves and the best scores among the compared baselines. This suggests entropy-flow balancing is a simple, well-grounded way to stabilize reasoning-oriented RL.

## Key points
- Baselines were GRPO, strict on-policy GRPO, Entropy-Reg, Clip-higher, Clip-Cov and KL-Cov. On Qwen2.5-Math-7B, OPEFO reached the following Pass@32 scores:
  - AIME24: 62.4 (GRPO: 59.8)
  - AIME25: 43.3 (GRPO: 38.4)
  - AMC23: 95.6
  - MATH500: 94.1
- On AIME24, OPEFO's advantage holds as k grows, reaching Pass@64 of 68.4 versus 65.6 for GRPO.
- Ablations show weaker results for every simplified variant:
  - A fixed λ of 0.001, set to the average of λ* over training, underperforms the dynamic version.
  - Rescaling only one side (only S+ or only S−) also underperforms full OPEFO.
  - λ* stays positive throughout training and oscillates rather than converging.
- Strict on-policy training adds no overhead. Using one update per rollout instead of GRPO's 8 took about 149 s per batch, versus about 158 s for approximate on-policy methods.
- Code, data and models are released at github.com/Anna7355/Entropy-Flow.
- The authors list several limitations:
  - The analysis relies on a first-order, tabular-softmax approximation.
  - The balancing works locally, at the level of each batch.
  - Evaluation covers only math reasoning.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 212 works · 19,544 citations
- [Nanyang Technological University](https://md.research.elamur.ai/free/en/universities/nanyang-technological-university/) — University, Singapore, Singapore · 184 works · 27,484 citations
- [VinUniversity](https://md.research.elamur.ai/free/en/universities/vinuniversity/) — University, Hanoi, Vietnam · 13 works · 56 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Huimin Xu, Shuai Zhao, Xiaobao Wu and Anh Tuan Luu. 2026. Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xu2026understanding,
  title = {{Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization}},
  author = {Huimin Xu, Shuai Zhao, Xiaobao Wu and Anh Tuan Luu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.879},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.879}
}
```

- [← previous Understanding and Mitigating Spurious Signal Amplification in Test-Time Reinforcement Learning for Math Reasoning](https://md.research.elamur.ai/free/en/papers/yu-2026-understanding-mitigating-spurious-signal-amplification/)
- [next → Understanding Conflicts in Multi-Objective Alignment through Reward Consistency](https://md.research.elamur.ai/free/en/papers/xu-2026-understanding-conflicts-multi-objective-alignment/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/55/)

## References

38 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Leemon Baird and 1 others. 1995. Residual algorithms: Reinforcement learning with function approximation. In Proceedings of the twelfth international conference on machine learning, pages 30–37. [find](https://scholar.google.com/scholar?q=Leemon+Baird+and+1+others%2E+1995%2E+Residual+algorithms%3A+Reinforcement+learning+with+function+approximation%2E+In+Proceedings+of+the+twelfth+international+conference+on+machine+learning%2C+pages+30%E2%80%9337%2E)
2. Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. 2025. [Reasoning with exploration: An entropy perspective](https://md.research.elamur.ai/free/en/papers/cheng-2025-reasoning-exploration-entropy-perspective/). arXiv preprint arXiv:2506.14758.
3. Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, and 1 others. 2025. [The entropy mechanism of reinforcement learning for reasoning language models](https://md.research.elamur.ai/free/en/papers/cui-2025-entropy-mechanism-reinforcement-learning-reasoning/). arXiv preprint arXiv:2505.22617.
4. Jia Deng, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, and Ji-Rong Wen. 2025. Decomposing the entropy-performance exchange: The missing keys to unlocking effective reinforcement learning. arXiv preprint arXiv:2508.02260. — [arXiv:2508.02260](https://arxiv.org/abs/2508.02260) [find](https://scholar.google.com/scholar?q=Jia+Deng%2C+Jie+Chen%2C+Zhipeng+Chen%2C+Wayne+Xin+Zhao%2C+and+Ji%2DRong+Wen%2E+2025%2E+Decomposing+the+entropy%2Dperformance+exchange%3A+The+missing+keys+to+unlocking+effective+reinforcement+learning%2E+arXiv+preprint+ar)
5. Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2017. Accurate, large minibatch sgd: Training imagenet in 1 hour. [find](https://scholar.google.com/scholar?q=Priya+Goyal%2C+Piotr+Doll%C3%A1r%2C+Ross+Girshick%2C+Pieter+Noordhuis%2C+Lukasz+Wesolowski%2C+Aapo+Kyrola%2C+Andrew+Tulloch%2C+Yangqing+Jia%2C+and+Kaiming+He%2E+2017%2E+Accurate%2C+large+minibatch+sgd%3A+Training+imagenet+in+1+ho)
6. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
7. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. Pmlr. [find](https://scholar.google.com/scholar?q=Tuomas+Haarnoja%2C+Aurick+Zhou%2C+Pieter+Abbeel%2C+and+Sergey+Levine%2E+2018%2E+Soft+actor%2Dcritic%3A+Off%2Dpolicy+maximum+entropy+deep+reinforcement+learning+with+a+stochastic+actor%2E+In+International+conference+on+)
8. Yaru Hao, Li Dong, Xun Wu, Shaohan Huang, Zewen Chi, and Furu Wei. 2025a. On-policy rl with optimal reward baseline. arXiv preprint arXiv:2505.23585. — [arXiv:2505.23585](https://arxiv.org/abs/2505.23585) [find](https://scholar.google.com/scholar?q=Yaru+Hao%2C+Li+Dong%2C+Xun+Wu%2C+Shaohan+Huang%2C+Zewen+Chi%2C+and+Furu+Wei%2E+2025a%2E+On%2Dpolicy+rl+with+optimal+reward+baseline%2E+arXiv+preprint+arXiv%3A2505%2E23585%2E+%E2%80%94+%5BarXiv%3A2505%2E23585%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2505%2E23)
9. Zhezheng Hao, Hong Wang, Haoyang Liu, Jian Luo, Jiarui Yu, Hande Dong, Qiang Lin, Can Wang, and Jiawei Chen. 2025b. [Rethinking entropy interventions in rlvr: An entropy change perspective](https://md.research.elamur.ai/free/en/papers/hao-2026-rethinking-entropy-interventions-rlvr-entropy/). arXiv preprint arXiv:2510.10150.
10. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, and 1 others. 2024. [Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems](https://md.research.elamur.ai/free/en/papers/he-2024-olympiadbench-challenging-benchmark-promoting-agi/). In Proceedings of the 62nd Annual Meeting…
11. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874.
12. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. [Openai o1 system card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/). arXiv preprint arXiv:2412.16720.
13. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, and 1 others. 2024. [Tulu 3: Pushing frontiers in open language model post-training](https://md.research.elamur.ai/free/en/papers/lambert-2024-tulu-3-pushing-frontiers-open/). arXiv preprint arXiv:2411.15124.
14. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, and 1 others. 2022. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). Advances in neural information processing systems, 35:3843–3857…
15. Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, and 1 others. 2024. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):… [find](https://scholar.google.com/scholar?q=Jia+Li%2C+Edward+Beeching%2C+Lewis+Tunstall%2C+Ben+Lipkin%2C+Roman+Soletskyi%2C+Shengyi+Huang%2C+Kashif+Rasul%2C+Longhui+Yu%2C+Albert+Q+Jiang%2C+Ziju+Shen%2C+and+1+others%2E+2024%2E+Numinamath%3A+The+largest+public+dataset+in+)
16. Jiacai Liu. 2025. How does rl policy entropy converge during iteration. Zhihu Zhuanlan. [find](https://scholar.google.com/scholar?q=Jiacai+Liu%2E+2025%2E+How+does+rl+policy+entropy+converge+during+iteration%2E+Zhihu+Zhuanlan%2E)
17. Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. 2025. [Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models](https://md.research.elamur.ai/free/en/papers/liu-2025-prorl-prolonged-reinforcement-learning-expands/). arXiv preprint arXiv:2505.24864.
18. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2017%2E+Decoupled+weight+decay+regularization%2E+arXiv+preprint+arXiv%3A1711%2E05101%2E)
19. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928–1937. PmLR. [find](https://scholar.google.com/scholar?q=Volodymyr+Mnih%2C+Adria+Puigdomenech+Badia%2C+Mehdi+Mirza%2C+Alex+Graves%2C+Timothy+Lillicrap%2C+Tim+Harley%2C+David+Silver%2C+and+Koray+Kavukcuoglu%2E+2016%2E+Asynchronous+methods+for+deep+reinforcement+learning%2E+In+I)
20. OpenAI. 2024. Learning to reason with llms. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2024%2E+Learning+to+reason+with+llms%2E)
21. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+2017%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2E)
22. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300.
23. Han Shen. 2025. On entropy control in llm-rl algorithms. arXiv preprint arXiv:2509.03493. — [arXiv:2509.03493](https://arxiv.org/abs/2509.03493) [find](https://scholar.google.com/scholar?q=Han+Shen%2E+2025%2E+On+entropy+control+in+llm%2Drl+algorithms%2E+arXiv+preprint+arXiv%3A2509%2E03493%2E+%E2%80%94+%5BarXiv%3A2509%2E03493%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2509%2E03493%29)
24. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. [Hybridflow: A flexible and efficient rlhf framework](https://md.research.elamur.ai/free/en/papers/sheng-2024-hybridflow-flexible-efficient-rlhf-framework/). In Proceedings of the Twentieth European Conference on Computer Systems, pages 1279–1297.
25. Samuel L Smith and Quoc V Le. 2017. A bayesian perspective on generalization and stochastic gradient descent. arXiv preprint arXiv:1710.06451. [find](https://scholar.google.com/scholar?q=Samuel+L+Smith+and+Quoc+V+Le%2E+2017%2E+A+bayesian+perspective+on+generalization+and+stochastic+gradient+descent%2E+arXiv+preprint+arXiv%3A1710%2E06451%2E)
26. Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton%2C+David+McAllester%2C+Satinder+Singh%2C+and+Yishay+Mansour%2E+1999%2E+Policy+gradient+methods+for+reinforcement+learning+with+function+approximation%2E+Advances+in+neural+information+processing+)
27. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 22 others. 2024. Qwen2.5 technical report. arXiv preprint arXi… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+Yang%2C+Jin)
28. Shihui Yang, Chengfeng Dou, Peidong Guo, Kai Lu, Qiang Ju, Fei Deng, and Rihui Xin. 2025b. Dcpo: Dynamic clipping policy optimization. arXiv preprint arXiv:2509.02333. — [arXiv:2509.02333](https://arxiv.org/abs/2509.02333) [find](https://scholar.google.com/scholar?q=Shihui+Yang%2C+Chengfeng+Dou%2C+Peidong+Guo%2C+Kai+Lu%2C+Qiang+Ju%2C+Fei+Deng%2C+and+Rihui+Xin%2E+2025b%2E+Dcpo%3A+Dynamic+clipping+policy+optimization%2E+arXiv+preprint+arXiv%3A2509%2E02333%2E+%E2%80%94+%5BarXiv%3A2509%2E02333%5D%28https%3A%2F%2Farx)
29. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, and 1 others. 2025. [Dapo: An open-source llm reinforcement learning system at scale](https://md.research.elamur.ai/free/en/papers/yu-2025-dapo-open-source-llm-reinforcement/). arXiv preprint arXiv:2503.14476.
30. Ruipeng Zhang, Ya-Chien Chang, and Sicun Gao. 2025. When maximum entropy misleads policy optimization. arXiv preprint arXiv:2506.05615. — [arXiv:2506.05615](https://arxiv.org/abs/2506.05615) [find](https://scholar.google.com/scholar?q=Ruipeng+Zhang%2C+Ya%2DChien+Chang%2C+and+Sicun+Gao%2E+2025%2E+When+maximum+entropy+misleads+policy+optimization%2E+arXiv+preprint+arXiv%3A2506%2E05615%2E+%E2%80%94+%5BarXiv%3A2506%2E05615%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2506%2E05615%29)
31. Haizhong Zheng, Jiawei Zhao, and Beidi Chen. 2025. Prosperity before collapse: How far can off-policy rl reach with stale data on llms? arXiv preprint arXiv:2510.01161. — [arXiv:2510.01161](https://arxiv.org/abs/2510.01161) [find](https://scholar.google.com/scholar?q=Haizhong+Zheng%2C+Jiawei+Zhao%2C+and+Beidi+Chen%2E+2025%2E+Prosperity+before+collapse%3A+How+far+can+off%2Dpolicy+rl+reach+with+stale+data+on+llms%3F+arXiv+preprint+arXiv%3A2510%2E01161%2E+%E2%80%94+%5BarXiv%3A2510%2E01161%5D%28https%3A%2F%2Far)
32. Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. 2025. The surprising effectiveness of negative reinforcement in llm reasoning. arXiv preprint arXiv:2506.01347. [find](https://scholar.google.com/scholar?q=Xinyu+Zhu%2C+Mengzhou+Xia%2C+Zhepei+Wei%2C+Wei%2DLin+Chen%2C+Danqi+Chen%2C+and+Yu+Meng%2E+2025%2E+The+surprising+effectiveness+of+negative+reinforcement+in+llm+reasoning%2E+arXiv+preprint+arXiv%3A2506%2E01347%2E)
33. Hongze Tan, Jianfei Pan, Jinghao Lin, Tao Chen, Zhihang Zheng, Zhihao Tang, and Haihua Yang. 2025. Gtpo and grpo-s: Token and sequence-level reward shaping with policy entropy. arXiv preprint arXiv:2508.04349. — [arXiv:2508.04349](https://arxiv.org/abs/2508.04349) [find](https://scholar.google.com/scholar?q=Hongze+Tan%2C+Jianfei+Pan%2C+Jinghao+Lin%2C+Tao+Chen%2C+Zhihang+Zheng%2C+Zhihao+Tang%2C+and+Haihua+Yang%2E+2025%2E+Gtpo+and+grpo%2Ds%3A+Token+and+sequence%2Dlevel+reward+shaping+with+policy+entropy%2E+arXiv+preprint+arXiv%3A25)
34. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, and 1 others. 2025. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599. — [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k1-5-scaling-reinforcement/)
35. Jiakang Wang, Runze Liu, Fuzheng Zhang, Xiu Li, and Guorui Zhou. 2025a. Stabilizing knowledge, promoting reasoning: Dual-token constraints for rlvr. arXiv preprint arXiv:2507.15778. — [arXiv:2507.15778](https://arxiv.org/abs/2507.15778) [find](https://scholar.google.com/scholar?q=Jiakang+Wang%2C+Runze+Liu%2C+Fuzheng+Zhang%2C+Xiu+Li%2C+and+Guorui+Zhou%2E+2025a%2E+Stabilizing+knowledge%2C+promoting+reasoning%3A+Dual%2Dtoken+constraints+for+rlvr%2E+arXiv+preprint+arXiv%3A2507%2E15778%2E+%E2%80%94+%5BarXiv%3A2507%2E1577)
36. Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, and 1 others. 2025b. [Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning](https://md.research.elamur.ai/free/en/papers/wang-2025-80-20-rule-high-entropy/). arXiv preprint arXiv:2506.01939.
37. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229–256. [find](https://scholar.google.com/scholar?q=Ronald+J+Williams%2E+1992%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Machine+learning%2C+8%283%29%3A229%E2%80%93256%2E)
38. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.

*Updated 2 November 2026*