---
title: "SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks"
description: "They propose SPPO, which makes this view explicit. 45 references, cited by 8 papers here."
authors: ["Tianyi Wang", "Yixia Li", "Long Li", "Yibiao Chen", "Shaohan Huang", "Yun Chen", "Peng Li", "Yang Liu", "Guanhua Chen"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.356/"
doi: "10.18653/v1/2026.acl-long.356"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2026-sppo-sequence-level-ppo-long/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2026-sppo-sequence-level-ppo-long/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/) / SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks

# SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks

**Authors:** Tianyi Wang, [Yixia Li](https://md.research.elamur.ai/free/en/authors/yixia-li/), [Long Li](https://md.research.elamur.ai/free/en/authors/long-li/), Yibiao Chen, [Shaohan Huang](https://md.research.elamur.ai/free/en/authors/shaohan-huang/), [Yun Chen](https://md.research.elamur.ai/free/en/authors/yun-chen/), [Peng Li](https://md.research.elamur.ai/free/en/authors/peng-li/), [Yang Liu](https://md.research.elamur.ai/free/en/authors/yang-liu/) and [Guanhua Chen](https://md.research.elamur.ai/free/en/authors/guanhua-chen/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.356/) · [PDF](https://aclanthology.org/2026.acl-long.356.pdf) · [DOI 10.18653/v1/2026.acl-long.356](https://doi.org/10.18653/v1/2026.acl-long.356) · [OpenAlex](https://openalex.org/W7166807237)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/wang-2026-sppo-sequence-level-ppo-long/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Reinforcement learning with verifiable rewards is a common way to improve LLM reasoning, but the two main options each have a cost. Token-level PPO depends on a critic and GAE, and these struggle to pass a sparse final reward back through very long Chain-of-Thought outputs. The authors show the critic separates correct from incorrect answers only near the end of a sequence, so the advantage signal fades. GRPO avoids the critic but must sample many responses per prompt, which slows training. The authors argue that GRPO works because it implicitly treats each full response as one action in a contextual bandit. They propose SPPO, which makes this view explicit. A scalar value model, trained with BCE, predicts how likely a prompt is to be solved. The advantage is the binary reward minus this prediction, and it is applied to every token of a single sampled response. On math benchmarks, SPPO beats standard PPO and matches GRPO at a fraction of the compute. This makes RL alignment of reasoning models cheaper.

## Key points
- Evaluated on AIME24/25, AMC23, MATH and Minerva. SPPO reaches GRPO's peak performance with a reported 5.9× training speedup, using one sample per prompt (N = 1).
- With a Decoupled Critic, a small 1.5B value model guides a 7B policy and reduces memory use by 12.8%.
- The "tail effect" diagnosis comes from a PPO critic trained with an 8192-token context. Its value estimates track token position rather than the semantic contribution of each step.
- Additional sparse-reward control experiments cover Hopper, MountainCar, LunarLander, Pendulum and Precision CartPole. In these, SPPO matches or exceeds standard PPO, which fails or becomes unstable on several tasks.
- The method requires verifiable outcomes, so open-ended generation is left for future work. Code is available at github.com/sustech-nlp/SPPO.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 333 works · 87,019 citations
- [Microsoft Research Asia](https://md.research.elamur.ai/free/en/companies/microsoft-research-asia/) — Company, Beijing, China · 73 works · 60,539 citations
- [Beijing University of Posts and Telecommunications](https://md.research.elamur.ai/free/en/universities/beijing-university-of-posts-and-telecommunications/) — University, Beijing, China · 57 works · 32,843 citations
- [Southern University of Science and Technology](https://md.research.elamur.ai/free/en/universities/southern-university-of-science-and-technology/) — University, Shenzhen, China · 19 works · 1,337 citations
- [Shanghai University of Finance and Economics](https://md.research.elamur.ai/free/en/universities/shanghai-university-of-finance-and-economics/) — University, Shanghai, China · 13 works · 825 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Tianyi Wang, Yixia Li, Long Li, Yibiao Chen, Shaohan Huang, Yun Chen, Peng Li, Yang Liu and Guanhua Chen. 2026. SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2026sppo,
  title = {{SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks}},
  author = {Tianyi Wang, Yixia Li, Long Li, Yibiao Chen, Shaohan Huang, Yun Chen, Peng Li, Yang Liu and Guanhua Chen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.356},
  url = {https://doi.org/10.18653/v1/2026.acl-long.356}
}
```

- [← previous Retrieval as Generation: A Unified Framework with Self-Triggered Information Planning](https://md.research.elamur.ai/free/en/papers/li-2026-retrieval-as-generation-unified-framework/)
- [next → Stable and Explainable Personality Trait Evaluation in Large Language Models with Internal Activations](https://md.research.elamur.ai/free/en/papers/ma-2026-stable-explainable-personality-trait-evaluation/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/)

## References

45 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. 2024. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. Preprint, arXiv:2402.14740. [find](https://scholar.google.com/scholar?q=Arash+Ahmadian%2C+Chris+Cremer%2C+Matthias+Gall%C3%A9%2C+Marzieh+Fadaee%2C+Julia+Kreutzer%2C+Olivier+Pietquin%2C+Ahmet+%C3%9Cst%C3%BCn%2C+and+Sara+Hooker%2E+2024%2E+Back+to+basics%3A+Revisiting+reinforce+style+optimization+for+learning)
2. Art of Problem Solving. 2025a. Aime problems and solutions. Accessed: 2026-01-04. [find](https://scholar.google.com/scholar?q=Art+of+Problem+Solving%2E+2025a%2E+Aime+problems+and+solutions%2E+Accessed%3A+2026%2D01%2D04%2E)
3. Art of Problem Solving. 2025b. Amc problems and solutions. Accessed: 2026-01-04. [find](https://scholar.google.com/scholar?q=Art+of+Problem+Solving%2E+2025b%2E+Amc+problems+and+solutions%2E+Accessed%3A+2026%2D01%2D04%2E)
4. Zhoujun Cheng, Shibo Hao, Tianyang Liu, Fan Zhou, Yutao Xie, Feng Yao, Yuexin Bian, Yonghao Zhuang, Nilabjo Dey, Yuheng Zha, Yi Gu, Kun Zhou, Yuqi Wang, Yuan Li, Richard Fan, Jianshu She, Chengqian Gao, Abulhair Saparov, Haonan Li, Taylor W. Killian, Mikhail Yurochkin, Zhengzhong Liu, Eric P. Xing,… — [arXiv:2506.14965](https://arxiv.org/abs/2506.14965) [find](https://scholar.google.com/scholar?q=Zhoujun+Cheng%2C+Shibo+Hao%2C+Tianyang+Liu%2C+Fan+Zhou%2C+Yutao+Xie%2C+Feng+Yao%2C+Yuexin+Bian%2C+Yonghao+Zhuang%2C+Nilabjo+Dey%2C+Yuheng+Zha%2C+Yi+Gu%2C+Kun+Zhou%2C+Yuqi+Wang%2C+Yuan+Li%2C+Richard+Fan%2C+Jianshu+She%2C+Chengqian+Ga)
5. Tianqing Fang, Zhisong Zhang, Xiaoyang Wang, Rui Wang, Can Qin, Yuxuan Wan, Jun-Yu Ma, Ce Zhang, Jiaqi Chen, Xiyun Li, Hongming Zhang, Haitao Mi, and Dong Yu. 2025. Cognitive kernel-pro: A framework for deep research agents and agent foundation models training. Preprint, arXiv:2508.00414. — [arXiv:2508.00414](https://arxiv.org/abs/2508.00414) [find](https://scholar.google.com/scholar?q=Tianqing+Fang%2C+Zhisong+Zhang%2C+Xiaoyang+Wang%2C+Rui+Wang%2C+Can+Qin%2C+Yuxuan+Wan%2C+Jun%2DYu+Ma%2C+Ce+Zhang%2C+Jiaqi+Chen%2C+Xiyun+Li%2C+Hongming+Zhang%2C+Haitao+Mi%2C+and+Dong+Yu%2E+2025%2E+Cognitive+kernel%2Dpro%3A+A+framework+f)
6. Yangyi Fang, Jiaye Lin, Xiaoliang Fu, Cong Qin, Haolin Shi, Chaowen Hu, Lu Pan, Ke Zeng, and Xunliang Cai. 2026a. How to allocate, how to learn? dynamic rollout allocation and advantage modulation for policy optimization. arXiv preprint arXiv:2602.19208. [find](https://scholar.google.com/scholar?q=Yangyi+Fang%2C+Jiaye+Lin%2C+Xiaoliang+Fu%2C+Cong+Qin%2C+Haolin+Shi%2C+Chaowen+Hu%2C+Lu+Pan%2C+Ke+Zeng%2C+and+Xunliang+Cai%2E+2026a%2E+How+to+allocate%2C+how+to+learn%3F+dynamic+rollout+allocation+and+advantage+modulation+for)
7. Yangyi Fang, Jiaye Lin, Xiaoliang Fu, Cong Qin, Haolin Shi, Chang Liu, and Peilin Zhao. 2026b. Proximity-based multi-turn optimization: Practical credit assignment for llm agent training. arXiv preprint arXiv:2602.19225. [find](https://scholar.google.com/scholar?q=Yangyi+Fang%2C+Jiaye+Lin%2C+Xiaoliang+Fu%2C+Cong+Qin%2C+Haolin+Shi%2C+Chang+Liu%2C+and+Peilin+Zhao%2E+2026b%2E+Proximity%2Dbased+multi%2Dturn+optimization%3A+Practical+credit+assignment+for+llm+agent+training%2E+arXiv+prepri)
8. Yiran Guo, Lijie Xu, Jie Liu, Dan Ye, and Shuang Qiu. 2025. Segment policy optimization: Effective segment-level credit assignment in rl for large language models. Preprint, arXiv:2505.23564. — [arXiv:2505.23564](https://arxiv.org/abs/2505.23564) [find](https://scholar.google.com/scholar?q=Yiran+Guo%2C+Lijie+Xu%2C+Jie+Liu%2C+Dan+Ye%2C+and+Shuang+Qiu%2E+2025%2E+Segment+policy+optimization%3A+Effective+segment%2Dlevel+credit+assignment+in+rl+for+large+language+models%2E+Preprint%2C+arXiv%3A2505%2E23564%2E+%E2%80%94+%5BarXiv)
9. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Hongming Zhang, Tianqing Fang, Zhenzhong Lan, and Dong Yu. 2025. [OpenWebVoyager: Building multimodal web agents via iterative real-world exploration, feedback and optimization](https://md.research.elamur.ai/free/en/papers/he-2025-openwebvoyager-building-multimodal-web-agents/). In Proceedings of the 63rd Annual Meeting of the Association for Computatio…
10. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). NeurIPS.
11. Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. 2025. Vineppo: Refining credit assignment in rl training of llms. Preprint, arXiv:2410.01679. [find](https://scholar.google.com/scholar?q=Amirhossein+Kazemnejad%2C+Milad+Aghajohari%2C+Eva+Portelance%2C+Alessandro+Sordoni%2C+Siva+Reddy%2C+Aaron+Courville%2C+and+Nicolas+Le+Roux%2E+2025%2E+Vineppo%3A+Refining+credit+assignment+in+rl+training+of+llms%2E+Prepri)
12. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). Preprint, arXiv:2…
13. Chen Li, Nazhou Liu, and Kai Yang. 2025a. Adaptive group policy optimization: Towards stable training and token-efficient reasoning. Preprint, arXiv:2503.15952. — [arXiv:2503.15952](https://arxiv.org/abs/2503.15952) [find](https://scholar.google.com/scholar?q=Chen+Li%2C+Nazhou+Liu%2C+and+Kai+Yang%2E+2025a%2E+Adaptive+group+policy+optimization%3A+Towards+stable+training+and+token%2Defficient+reasoning%2E+Preprint%2C+arXiv%3A2503%2E15952%2E+%E2%80%94+%5BarXiv%3A2503%2E15952%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
14. Long Li, Xuzheng He, Haozhe Wang, Linlin Wang, and Liang He. 2024. How do humans write code? large models do it the same way too. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, page 4638–4649. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Long+Li%2C+Xuzheng+He%2C+Haozhe+Wang%2C+Linlin+Wang%2C+and+Liang+He%2E+2024%2E+How+do+humans+write+code%3F+large+models+do+it+the+same+way+too%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+)
15. Long Li, Zhijian Zhou, Jiaran Hao, Jason Klein Liu, Yanting Miao, Wei Pang, Xiaoyu Tan, Wei Chu, Zhe Wang, Shirui Pan, Chao Qu, and Yuan Qi. 2026a. The choice of divergence: A neglected key to mitigating diversity collapse in reinforcement learning with verifiable reward. Preprint, arXiv:2509.07430. — [arXiv:2509.07430](https://arxiv.org/abs/2509.07430) [find](https://scholar.google.com/scholar?q=Long+Li%2C+Zhijian+Zhou%2C+Jiaran+Hao%2C+Jason+Klein+Liu%2C+Yanting+Miao%2C+Wei+Pang%2C+Xiaoyu+Tan%2C+Wei+Chu%2C+Zhe+Wang%2C+Shirui+Pan%2C+Chao+Qu%2C+and+Yuan+Qi%2E+2026a%2E+The+choice+of+divergence%3A+A+neglected+key+to+mitigat)
16. Long Li, Zhijian Zhou, Jiangxuan Long, Peiyang Liu, Weidi Xu, Zhe Wang, Shirui Pan, and Chao Qu. 2026b. Sql-astra: Alleviating sparse feedback in agentic sql via column-set matching and trajectory aggregation. Preprint, arXiv:2603.16161. [find](https://scholar.google.com/scholar?q=Long+Li%2C+Zhijian+Zhou%2C+Jiangxuan+Long%2C+Peiyang+Liu%2C+Weidi+Xu%2C+Zhe+Wang%2C+Shirui+Pan%2C+and+Chao+Qu%2E+2026b%2E+Sql%2Dastra%3A+Alleviating+sparse+feedback+in+agentic+sql+via+column%2Dset+matching+and+trajectory+agg)
17. Long Li, Zhijian Zhou, Tianyi Wang, Weidi Xu, Zuming Huang, Wei Chu, Zhe Wang, Shirui Pan, Chao Qu, and Yuan Qi. 2026c. Dyjr: Preserving diversity in reinforcement learning with verifiable rewards via dynamic jensen-shannon replay. Preprint, arXiv:2603.16157. — [arXiv:2603.16157](https://arxiv.org/abs/2603.16157) [find](https://scholar.google.com/scholar?q=Long+Li%2C+Zhijian+Zhou%2C+Tianyi+Wang%2C+Weidi+Xu%2C+Zuming+Huang%2C+Wei+Chu%2C+Zhe+Wang%2C+Shirui+Pan%2C+Chao+Qu%2C+and+Yuan+Qi%2E+2026c%2E+Dyjr%3A+Preserving+diversity+in+reinforcement+learning+with+verifiable+rewards+via)
18. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025. [Understanding r1-zero-like training: A critical perspective](https://md.research.elamur.ai/free/en/papers/liu-2025-understanding-r1-zero-like-training/). Preprint, arXiv:2503.20783.
19. Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. 2025. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. https://pretty-radio -b75.notion.site/DeepScaleR-Surpassing-O 1-Preview-w… [find](https://scholar.google.com/scholar?q=Michael+Luo%2C+Sijun+Tan%2C+Justin+Wong%2C+Xiaoxiang+Shi%2C+William+Y%2E+Tang%2C+Manan+Roongta%2C+Colin+Cai%2C+Jeffrey+Luo%2C+Li+Erran+Li%2C+Raluca+Ada+Popa%2C+and+Ion+Stoica%2E+2025%2E+Deepscaler%3A+Surpassing+o1%2Dpreview+with+a)
20. Weijian Ma, Shizhao Sun, Tianyu Yu, Ruiyu Wang, Tat-Seng Chua, and Jiang Bian. 2026. Thinking with blueprints: Assisting vision-language models in spatial reasoning via structured object representation. Preprint, arXiv:2601.01984. — [arXiv:2601.01984](https://arxiv.org/abs/2601.01984) [find](https://scholar.google.com/scholar?q=Weijian+Ma%2C+Shizhao+Sun%2C+Tianyu+Yu%2C+Ruiyu+Wang%2C+Tat%2DSeng+Chua%2C+and+Jiang+Bian%2E+2026%2E+Thinking+with+blueprints%3A+Assisting+vision%2Dlanguage+models+in+spatial+reasoning+via+structured+object+representatio)
21. Zhiwen Ruan, Yun Chen, Yutao Hou, Peng Li, Yang Liu, and Guanhua Chen. 2025a. Unveiling over-memorization in finetuning llms for reasoning tasks. Preprint, arXiv:2508.04117. — [arXiv:2508.04117](https://arxiv.org/abs/2508.04117) [find](https://scholar.google.com/scholar?q=Zhiwen+Ruan%2C+Yun+Chen%2C+Yutao+Hou%2C+Peng+Li%2C+Yang+Liu%2C+and+Guanhua+Chen%2E+2025a%2E+Unveiling+over%2Dmemorization+in+finetuning+llms+for+reasoning+tasks%2E+Preprint%2C+arXiv%3A2508%2E04117%2E+%E2%80%94+%5BarXiv%3A2508%2E04117%5D%28https)
22. Zhiwen Ruan, Yixia Li, He Zhu, Yun Chen, Peng Li, Yang Liu, and Guanhua Chen. 2025b. Enhancing large language model reasoning via selective critical token fine-tuning. Preprint, arXiv:2510.10974. — [arXiv:2510.10974](https://arxiv.org/abs/2510.10974) [find](https://scholar.google.com/scholar?q=Zhiwen+Ruan%2C+Yixia+Li%2C+He+Zhu%2C+Yun+Chen%2C+Peng+Li%2C+Yang+Liu%2C+and+Guanhua+Chen%2E+2025b%2E+Enhancing+large+language+model+reasoning+via+selective+critical+token+fine%2Dtuning%2E+Preprint%2C+arXiv%3A2510%2E10974%2E+%E2%80%94+%5Ba)
23. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. Preprint, arXiv:1707.06347. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+2017%2E+Proximal+policy+optimization+algorithms%2E+Preprint%2C+arXiv%3A1707%2E06347%2E)
24. Mukai Li, Qingcheng Zeng, Tianqing Fang, Zhenwen Liang, Linfeng Song, Qi Liu, Haitao Mi, and Dong Yu. 2026d. Verified critical step optimization for llm agents. Preprint, arXiv:2602.03412. [find](https://scholar.google.com/scholar?q=Mukai+Li%2C+Qingcheng+Zeng%2C+Tianqing+Fang%2C+Zhenwen+Liang%2C+Linfeng+Song%2C+Qi+Liu%2C+Haitao+Mi%2C+and+Dong+Yu%2E+2026d%2E+Verified+critical+step+optimization+for+llm+agents%2E+Preprint%2C+arXiv%3A2602%2E03412%2E)
25. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). Preprint, arXiv:2402.03300.
26. Xiaozhe Li, Xinyu Fang, Shengyuan Ding, Linyang Li, Haodong Duan, Qingwen Liu, and Kai Chen. 2025b. Np-engine: Empowering optimization reasoning in large language models with verifiable synthetic np problems. Preprint, arXiv:2510.16476. [find](https://scholar.google.com/scholar?q=Xiaozhe+Li%2C+Xinyu+Fang%2C+Shengyuan+Ding%2C+Linyang+Li%2C+Haodong+Duan%2C+Qingwen+Liu%2C+and+Kai+Chen%2E+2025b%2E+Np%2Dengine%3A+Empowering+optimization+reasoning+in+large+language+models+with+verifiable+synthetic+np+p)
27. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. [Hybridflow: A flexible and efficient rlhf framework](https://md.research.elamur.ai/free/en/papers/sheng-2024-hybridflow-flexible-efficient-rlhf-framework/). In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, page 1279–1297. ACM.
28. Zhicong Li, Lingjie Jiang, Yulan Hu, Xingchen Zeng, Yixia Li, Xiangwen Zhang, Guanhua Chen, Zheng Pan, Xin Li, and Yong Liu. 2026e. [No more stale feedback: Co-evolving critics for open-world agent learning](https://md.research.elamur.ai/free/en/papers/li-2026-no-more-stale-feedback-co/). Preprint, arXiv:2601.06794.
29. Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, volume 12. MIT Press. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton%2C+David+McAllester%2C+Satinder+Singh%2C+and+Yishay+Mansour%2E+1999%2E+Policy+gradient+methods+for+reinforcement+learning+with+function+approximation%2E+In+Advances+in+Neural+Information+Processi)
30. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). Preprint, arXiv:2305.20050.
31. Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. 2025. Cppo: Accelerating the training of group relative policy optimization-based reasoning models. Preprint, arXiv:2503.22342. [find](https://scholar.google.com/scholar?q=Zhihang+Lin%2C+Mingbao+Lin%2C+Yuan+Xie%2C+and+Rongrong+Ji%2E+2025%2E+Cppo%3A+Accelerating+the+training+of+group+relative+policy+optimization%2Dbased+reasoning+models%2E+Preprint%2C+arXiv%3A2503%2E22342%2E)
32. Chengbing Wang, Yang Zhang, Wenjie Wang, Xiaoyan Zhao, Fuli Feng, Xiangnan He, and Tat-Seng Chua. 2026a. Think-while-generating: On-the-fly reasoning for personalized long-form generation. Preprint, arXiv:2512.06690. — [arXiv:2512.06690](https://arxiv.org/abs/2512.06690) [find](https://scholar.google.com/scholar?q=Chengbing+Wang%2C+Yang+Zhang%2C+Wenjie+Wang%2C+Xiaoyan+Zhao%2C+Fuli+Feng%2C+Xiangnan+He%2C+and+Tat%2DSeng+Chua%2E+2026a%2E+Think%2Dwhile%2Dgenerating%3A+On%2Dthe%2Dfly+reasoning+for+personalized+long%2Dform+generation%2E+Preprint%2C+a)
33. Chengbing Wang, Wuqiang Zheng, Yang Zhang, Fengbin Zhu, Junyi Cheng, Yi Xie, Wenjie Wang, and Fuli Feng. 2026b. [Perm: Psychology-grounded empathetic reward modeling for large language models](https://md.research.elamur.ai/free/en/papers/wang-2026-perm-psychology-grounded-empathetic-reward/). Preprint, arXiv:2601.10532.
34. Hu Wang, Congbo Ma, Ian Reid, and Mohammad Yaqub. 2025a. Kalman filter enhanced grpo for reinforcement learning-based language model reasoning. Preprint, arXiv:2505.07527. — [arXiv:2505.07527](https://arxiv.org/abs/2505.07527) [find](https://scholar.google.com/scholar?q=Hu+Wang%2C+Congbo+Ma%2C+Ian+Reid%2C+and+Mohammad+Yaqub%2E+2025a%2E+Kalman+filter+enhanced+grpo+for+reinforcement+learning%2Dbased+language+model+reasoning%2E+Preprint%2C+arXiv%3A2505%2E07527%2E+%E2%80%94+%5BarXiv%3A2505%2E07527%5D%28https%3A%2F)
35. He Zhu, Junyou Su, Peng Lai, Ren Ma, Wenjia Zhang, Linyi Yang, and Guanhua Chen. 2026. Anchored supervised fine-tuning. In The Fourteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=He+Zhu%2C+Junyou+Su%2C+Peng+Lai%2C+Ren+Ma%2C+Wenjia+Zhang%2C+Linyi+Yang%2C+and+Guanhua+Chen%2E+2026%2E+Anchored+supervised+fine%2Dtuning%2E+In+The+Fourteenth+International+Conference+on+Learning+Representations%2E)
36. Jiakang Wang, Runze Liu, Lei Lin, Wenping Hu, Xiu Li, Fuzheng Zhang, Guorui Zhou, and Kun Gai. 2025b. Aspo: Asymmetric importance sampling policy optimization. Preprint, arXiv:2510.06062. — [arXiv:2510.06062](https://arxiv.org/abs/2510.06062) [find](https://scholar.google.com/scholar?q=Jiakang+Wang%2C+Runze+Liu%2C+Lei+Lin%2C+Wenping+Hu%2C+Xiu+Li%2C+Fuzheng+Zhang%2C+Guorui+Zhou%2C+and+Kun+Gai%2E+2025b%2E+Aspo%3A+Asymmetric+importance+sampling+policy+optimization%2E+Preprint%2C+arXiv%3A2510%2E06062%2E+%E2%80%94+%5BarXiv%3A251)
37. Wenjie Zhu, Yabin Zhang, Xin Jin, Wenjun Zeng, and Lei Zhang. 2025. Ants: Adaptive negative textual space shaping for ood detection via test-time mllm understanding and reasoning. arXiv preprint arXiv:2509.03951. — [arXiv:2509.03951](https://arxiv.org/abs/2509.03951) [find](https://scholar.google.com/scholar?q=Wenjie+Zhu%2C+Yabin+Zhang%2C+Xin+Jin%2C+Wenjun+Zeng%2C+and+Lei+Zhang%2E+2025%2E+Ants%3A+Adaptive+negative+textual+space+shaping+for+ood+detection+via+test%2Dtime+mllm+understanding+and+reasoning%2E+arXiv+preprint+arXiv)
38. Tianyi Wang, Long Li, Hongcan Guo, Yibiao Chen, Yixia Li, Yong Wang, Yun Chen, and Guanhua Chen. 2026c. Anchored policy optimization: Mitigating exploration collapse via support-constrained rectification. Preprint, arXiv:2602.05717. — [arXiv:2602.05717](https://arxiv.org/abs/2602.05717) [find](https://scholar.google.com/scholar?q=Tianyi+Wang%2C+Long+Li%2C+Hongcan+Guo%2C+Yibiao+Chen%2C+Yixia+Li%2C+Yong+Wang%2C+Yun+Chen%2C+and+Guanhua+Chen%2E+2026c%2E+Anchored+policy+optimization%3A+Mitigating+exploration+collapse+via+support%2Dconstrained+rectificat)
39. Cehua Yang, Dongyu Xiao, Junming Lin, Yuyang Song, Hanxu Yan, Shawn Guo, Wei Zhang, Jian Yang, Mingjie Tang, and Bryan Dai. 2025. Agro-sql: Agentic group-relative optimization with high-fidelity data synthesis. Preprint, arXiv:2512.23366. — [arXiv:2512.23366](https://arxiv.org/abs/2512.23366) [find](https://scholar.google.com/scholar?q=Cehua+Yang%2C+Dongyu+Xiao%2C+Junming+Lin%2C+Yuyang+Song%2C+Hanxu+Yan%2C+Shawn+Guo%2C+Wei+Zhang%2C+Jian+Yang%2C+Mingjie+Tang%2C+and+Bryan+Dai%2E+2025%2E+Agro%2Dsql%3A+Agentic+group%2Drelative+optimization+with+high%2Dfidelity+data+)
40. Jian Yang, Wei Zhang, Jiajun Wu, Junhang Cheng, Tuney Zheng, Fanglin Xu, Weicheng Gu, Lin Jing, Yaxin Du, Joseph Li, Yizhi Li, Yan Xing, Chuan Hao, Ran Tao, Ruihao Gong, Aishan Liu, Zhoujun Li, Mingjie Tang, Chenghua Lin, Siheng Chen, Wayne Xin Zhao, Xianglong Liu, Ming Zhou, Bryan Dai, and Weifeng… [find](https://scholar.google.com/scholar?q=Jian+Yang%2C+Wei+Zhang%2C+Jiajun+Wu%2C+Junhang+Cheng%2C+Tuney+Zheng%2C+Fanglin+Xu%2C+Weicheng+Gu%2C+Lin+Jing%2C+Yaxin+Du%2C+Joseph+Li%2C+Yizhi+Li%2C+Yan+Xing%2C+Chuan+Hao%2C+Ran+Tao%2C+Ruihao+Gong%2C+Aishan+Liu%2C+Zhoujun+Li%2C+Mingji)
41. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli… — [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://md.research.elamur.ai/free/en/papers/yu-2025-dapo-open-source-llm-reinforcement/)
42. Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. 2025. What’s behind ppo’s collapse in long-cot? value optimization holds the secret. Preprint, arXiv:2503.01491. — [arXiv:2503.01491](https://arxiv.org/abs/2503.01491) [find](https://scholar.google.com/scholar?q=Yufeng+Yuan%2C+Yu+Yue%2C+Ruofei+Zhu%2C+Tiantian+Fan%2C+and+Lin+Yan%2E+2025%2E+What%E2%80%99s+behind+ppo%E2%80%99s+collapse+in+long%2Dcot%3F+value+optimization+holds+the+secret%2E+Preprint%2C+arXiv%3A2503%2E01491%2E+%E2%80%94+%5BarXiv%3A2503%2E01491%5D%28https%3A)
43. Xuan Zhang, Ruixiao Li, Zhijian Zhou, Long Li, Yulei Qin, Ke Li, Xing Sun, Xiaoyu Tan, Chao Qu, and Yuan Qi. 2026. Count counts: Motivating exploration in LLM reasoning with count-based intrinsic rewards. In The Fourteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Xuan+Zhang%2C+Ruixiao+Li%2C+Zhijian+Zhou%2C+Long+Li%2C+Yulei+Qin%2C+Ke+Li%2C+Xing+Sun%2C+Xiaoyu+Tan%2C+Chao+Qu%2C+and+Yuan+Qi%2E+2026%2E+Count+counts%3A+Motivating+exploration+in+LLM+reasoning+with+count%2Dbased+intrinsic+rewa)
44. Yuzhong Zhao, Yue Liu, Junpeng Liu, Jingye Chen, Xun Wu, Yaru Hao, Tengchao Lv, Shaohan Huang, Lei Cui, Qixiang Ye, Fang Wan, and Furu Wei. 2025. Geometric-mean policy optimization. Preprint, arXiv:2507.20673. — [arXiv:2507.20673](https://arxiv.org/abs/2507.20673) [find](https://scholar.google.com/scholar?q=Yuzhong+Zhao%2C+Yue+Liu%2C+Junpeng+Liu%2C+Jingye+Chen%2C+Xun+Wu%2C+Yaru+Hao%2C+Tengchao+Lv%2C+Shaohan+Huang%2C+Lei+Cui%2C+Qixiang+Ye%2C+Fang+Wan%2C+and+Furu+Wei%2E+2025%2E+Geometric%2Dmean+policy+optimization%2E+Preprint%2C+arXiv%3A25)
45. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. 2025. [Group sequence policy optimization](https://md.research.elamur.ai/free/en/papers/zheng-2025-group-sequence-policy-optimization/). Preprint, arXiv:2507.18071.

*Updated 31 October 2026*