---
title: "AAPO — paper by Xiong et al. (2026) — summary and ideas"
description: "Group relative advantage estimation, popularized by GRPO, removes PPO's costly value model when post-training LLMs with… 50 references."
authors: ["Jian Xiong", "Jingbo Zhou", "Jingyong Ye", "Qiang Huang", "Dejing Dou"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1131/"
doi: "10.18653/v1/2026.acl-long.1131"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xiong-2026-aapo-enhancing-reasoning-capabilities-llms/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xiong-2026-aapo-enhancing-reasoning-capabilities-llms/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/21/) / AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margin

# AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margin

**Authors:** [Jian Xiong](https://md.research.elamur.ai/free/en/authors/jian-xiong/), [Jingbo Zhou](https://md.research.elamur.ai/free/en/authors/jingbo-zhou/), Jingyong Ye, [Qiang Huang](https://md.research.elamur.ai/free/en/authors/qiang-huang/) and [Dejing Dou](https://md.research.elamur.ai/free/en/authors/dejing-dou/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1131/) · [PDF](https://aclanthology.org/2026.acl-long.1131.pdf) · [DOI 10.18653/v1/2026.acl-long.1131](https://doi.org/10.18653/v1/2026.acl-long.1131) · [OpenAlex](https://openalex.org/W4416256207)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Group relative advantage estimation, popularized by GRPO, removes PPO's costly value model when post-training LLMs with RL for reasoning. It has a weakness, though. When every response sampled for a question gets similar rewards, the normalized advantage collapses toward zero, and the model receives no gradient. The authors observe this for a large and growing share of samples late in training. When rewards differ widely, advantages can instead become noisy and push updates in unhelpful directions.

AAPO (Advantage-Augmented Policy Optimization) addresses this by adding a clipped "advantage margin" to the usual group-normalized advantage. The margin is the reward gap between a response from the policy model and one from a reference model. Like GPG, it optimizes a cross-entropy-style objective directly rather than a surrogate loss. The extra term keeps a meaningful improvement signal even when within-group variance vanishes.

AAPO is reported to beat group-relative baselines on a range of math reasoning benchmarks, using models from multiple families. In AAPO training runs, no step had all-zero advantages. This matters because it makes RL post-training more sample-efficient without extra GPU memory.

## Key points
- The modified advantage is the standard (r − mean)/std term plus clip(r_i − r_ref_i, low, high), with responses scored by rule-based rewards.
- Training dynamics were tracked at group sizes 2, 6 and 10. The vanilla estimator's zero-advantage share rose sharply over training, while AAPO never produced an all-zero-advantage case.
- Removing the clip on the margin made training less stable and lowered scores. Keeping the reference model fixed beat updating it: average 56.7 vs 55.4 for AAPO-1.5B and 52.4 vs 51.0 for AAPO-7B.
- Compute cost per step versus GRPO:
  - 1.5B model: 826 s vs 567 s, with 52.2 GiB of memory for both.
  - 7B model: 384 s vs 299 s, with 78.9 vs 78.7 GiB of memory.
  - The overhead comes from sampling with the reference model. The authors suggest vLLM could reduce it.
- Limitations: AAPO reduces but does not eliminate zero advantages. Code is published at github.com/JianxXiong/AAPO.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 202 works · 22,509 citations
- [Baidu](https://md.research.elamur.ai/free/en/companies/baidu/) — Company, Beijing, China · 84 works · 17,681 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Jian Xiong, Jingbo Zhou, Jingyong Ye, Qiang Huang and Dejing Dou. 2026. AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margin. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xiong2026aapo,
  title = {{AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margin}},
  author = {Jian Xiong, Jingbo Zhou, Jingyong Ye, Qiang Huang and Dejing Dou},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1131},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1131}
}
```

- [← previous AAbAAC: An Annotated Corpus for Autoimmunity Information Extraction](https://md.research.elamur.ai/free/en/papers/maury-2026-aabaac-annotated-corpus-autoimmunity-information/)
- [next → ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development](https://md.research.elamur.ai/free/en/papers/yang-2026-abc-bench-benchmarking-agentic-backend/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/21/)

## References

50 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, and 1 others. 2021. [On the opportunities and risks of foundation models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/). arXiv preprint arXiv:2108.07258.
2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in Neural Information Processing Systems.
3. Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, and 1 others. 2025. An empirical study on eliciting and improving r1-like reasoning models. arXiv preprint arXiv:2503.04548. — [arXiv:2503.04548](https://arxiv.org/abs/2503.04548) [find](https://scholar.google.com/scholar?q=Zhipeng+Chen%2C+Yingqian+Min%2C+Beichen+Zhang%2C+Jie+Chen%2C+Jinhao+Jiang%2C+Daixuan+Cheng%2C+Wayne+Xin+Zhao%2C+Zheng+Liu%2C+Xu+Miao%2C+Yang+Lu%2C+and+1+others%2E+2025%2E+An+empirical+study+on+eliciting+and+improving+r1%2Dlike)
4. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, and 1 others. 2023. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research.
5. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Paul+F+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+2017%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+Neural+Information+Processing+Systems%2E)
6. Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. 2025. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.02546. — [arXiv:2504.02546](https://arxiv.org/abs/2504.02546) [find](https://scholar.google.com/scholar?q=Xiangxiang+Chu%2C+Hailang+Huang%2C+Xiao+Zhang%2C+Fei+Wei%2C+and+Yong+Wang%2E+2025%2E+Gpg%3A+A+simple+and+strong+reinforcement+learning+baseline+for+model+reasoning%2E+arXiv+preprint+arXiv%3A2504%2E02546%2E+%E2%80%94+%5BarXiv%3A2504%2E02)
7. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, and 1 others. 2025. [Process reinforcement through implicit rewards](https://md.research.elamur.ai/free/en/papers/cui-2025-process-reinforcement-through-implicit-rewards/). arXiv preprint arXiv:2502.01456.
8. Quy-Anh Dang and Chris Ngo. 2025. Reinforcement learning for reasoning in small llms: What works and what doesn’t. arXiv preprint arXiv:2503.16219. [find](https://scholar.google.com/scholar?q=Quy%2DAnh+Dang+and+Chris+Ngo%2E+2025%2E+Reinforcement+learning+for+reasoning+in+small+llms%3A+What+works+and+what+doesn%E2%80%99t%2E+arXiv+preprint+arXiv%3A2503%2E16219%2E)
9. Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna. 2021. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. Synthese. [find](https://scholar.google.com/scholar?q=Tom+Everitt%2C+Marcus+Hutter%2C+Ramana+Kumar%2C+and+Victoria+Krakovna%2E+2021%2E+Reward+tampering+problems+and+solutions+in+reinforcement+learning%3A+A+causal+influence+diagram+perspective%2E+Synthese%2E)
10. Nathan Habib, Clémentine Fourrier, Hynek Kydlíček, Thomas Wolf, and Lewis Tunstall. 2023. Lighteval: A lightweight framework for llm evaluation. [find](https://scholar.google.com/scholar?q=Nathan+Habib%2C+Cl%C3%A9mentine+Fourrier%2C+Hynek+Kydl%C3%ADc%CC%8Cek%2C+Thomas+Wolf%2C+and+Lewis+Tunstall%2E+2023%2E+Lighteval%3A+A+lightweight+framework+for+llm+evaluation%2E)
11. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In The Twelfth International Conference on Learning Representations.
12. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). Advances in Neural Information Processing Systems (Datasets and Benchmarks Track).
13. Jian Hu. 2025. [Reinforce++: A simple and efficient approach for aligning large language models](https://md.research.elamur.ai/free/en/papers/hu-2025-reinforce-stabilizing-critic-free-policy/). arXiv preprint arXiv:2501.03262.
14. Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. 2025. [Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model](https://md.research.elamur.ai/free/en/papers/hu-2025-open-reasoner-zero-open-source/). arXiv preprint arXiv:2503.24290.
15. Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, and 1 others. 2024. [Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai](https://md.research.elamur.ai/free/en/papers/huang-2024-olympicarena-benchmarking-multi-discipline-cognitive/). Advances in Neural Information Processing Systems.
16. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. [Livecodebench: Holistic and contamination free evaluation of large language models for code](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/). arXiv preprint arXiv:2403.07974.
17. Diederik P Kingma and Jimmy Ba. 2015. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). In The Third International Conference on Learning Representations.
18. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the 29th symposium on operating systems principles.
19. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. Deepseek-r1 incentivizes reasoning in llms through rein… [find](https://scholar.google.com/scholar?q=Daya+Guo%2C+Dejian+Yang%2C+Haowei+Zhang%2C+Junxiao+Song%2C+Peiyi+Wang%2C+Qihao+Zhu%2C+Runxin+Xu%2C+Ruoyu+Zhang%2C+Shirong+Ma%2C+Xiao+Bi%2C+Xiaokang+Zhang%2C+Xingkai+Yu%2C+Yu+Wu%2C+Z%2E+F%2E+Wu%2C+Zhibin+Gou%2C+Zhihong+Shao%2C+Zhuoshu+Li)
20. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, and 1 others. 2022. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). Advances in Neural Information Processing Systems.
21. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Tuomas+Haarnoja%2C+Aurick+Zhou%2C+Pieter+Abbeel%2C+and+Sergey+Levine%2E+2018%2E+Soft+actor%2Dcritic%3A+Off%2Dpolicy+maximum+entropy+deep+reinforcement+learning+with+a+stochastic+actor%2E+In+International+Conference+on+)
22. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). In The Twelfth International Conference on Learning Representations.
23. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. [Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/). ACM computing surveys.
24. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025. [Understanding r1-zero-like training: A critical perspective](https://md.research.elamur.ai/free/en/papers/liu-2025-understanding-r1-zero-like-training/). arXiv preprint arXiv:2503.20783.
25. OpenAI. 2024. Introducing OpenAI O1 Preview. [https://openai.com/index/](https://openai.com/index/) introducing-openai-o1-preview/. [↗](https://openai.com/index/)
26. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems.
27. Alexander Pan, Kush Bhatia, and Jacob Steinhardt. 2022. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint arXiv:2201.03544. [find](https://scholar.google.com/scholar?q=Alexander+Pan%2C+Kush+Bhatia%2C+and+Jacob+Steinhardt%2E+2022%2E+The+effects+of+reward+misspecification%3A+Mapping+and+mitigating+misaligned+models%2E+arXiv+preprint+arXiv%3A2201%2E03544%2E)
28. Qwen Team. 2024. QwQ: Reflect Deeply on the Boundaries of the Unknown. [https://qwenlm.github.io/blog/qwq-32b-preview/](https://qwenlm.github.io/blog/qwq-32b-preview/). [↗](https://qwenlm.github.io/blog/qwq-32b-preview/)
29. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, and 1 others. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+Ilya+Sutskever%2C+and+1+others%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
30. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). Advances in Neural Information Processing Systems.
31. Herbert Robbins and David Siegmund. 1971. A convergence theorem for non negative almost supermartin-gales and some applications. In Optimizing methods in statistics. Elsevier. [find](https://scholar.google.com/scholar?q=Herbert+Robbins+and+David+Siegmund%2E+1971%2E+A+convergence+theorem+for+non+negative+almost+supermartin%2Dgales+and+some+applications%2E+In+Optimizing+methods+in+statistics%2E+Elsevier%2E)
32. Rebecca Roelofs, Vaishaal Shankar, Benjamin Recht, Sara Fridovich-Keil, Moritz Hardt, John Miller, and Ludwig Schmidt. 2019. A meta-analysis of overfitting in machine learning. Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Rebecca+Roelofs%2C+Vaishaal+Shankar%2C+Benjamin+Recht%2C+Sara+Fridovich%2DKeil%2C+Moritz+Hardt%2C+John+Miller%2C+and+Ludwig+Schmidt%2E+2019%2E+A+meta%2Danalysis+of+overfitting+in+machine+learning%2E+Advances+in+Neural+Info)
33. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015. Trust region policy optimization. In International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Sergey+Levine%2C+Pieter+Abbeel%2C+Michael+Jordan%2C+and+Philipp+Moritz%2E+2015%2E+Trust+region+policy+optimization%2E+In+International+Conference+on+Machine+Learning%2E)
34. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2016. High-dimensional continuous control using generalized advantage estimation. In The Fourth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Philipp+Moritz%2C+Sergey+Levine%2C+Michael+Jordan%2C+and+Pieter+Abbeel%2E+2016%2E+High%2Ddimensional+continuous+control+using+generalized+advantage+estimation%2E+In+The+Fourth+International+Conferenc)
35. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+2017%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2E)
36. Claude E Shannon. 1948. A mathematical theory of communication. The Bell system technical journal. [find](https://scholar.google.com/scholar?q=Claude+E+Shannon%2E+1948%2E+A+mathematical+theory+of+communication%2E+The+Bell+system+technical+journal%2E)
37. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, and 1 others. 2024. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300.
38. Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. 2024. [Preference ranking optimization for human alignment](https://md.research.elamur.ai/free/en/papers/song-2023-preference-ranking-optimization-human-alignment/). In Proceedings of the AAAI Conference on Artificial Intelligence.
39. Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton%2C+David+McAllester%2C+Satinder+Singh%2C+and+Yishay+Mansour%2E+1999%2E+Policy+gradient+methods+for+reinforcement+learning+with+function+approximation%2E+Advances+in+Neural+Information+Processing+)
40. RUCAIBoxSTILL Team. 2025. Still-3-1.5b-preview: Enhancing slow thinking abilities of small models through reinforcement learning. [https://github.com/RUCAIBox/Slow_Thinking_with_LLMs](https://github.com/RUCAIBox/Slow_Thinking_with_LLMs). [↗](https://github.com/RUCAIBox/Slow_Thinking_with_LLMs)
41. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
42. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems.
43. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning. [find](https://scholar.google.com/scholar?q=Ronald+J+Williams%2E+1992%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Machine+learning%2E)
44. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, and 1 others. 2024. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122. — [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://md.research.elamur.ai/free/en/papers/yang-2024-qwen2-5-math-technical-report/)
45. Chaojian Yu, Bo Han, Li Shen, Jun Yu, Chen Gong, Mingming Gong, and Tongliang Liu. 2022. Understanding robust overfitting of adversarial training and beyond. In International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Chaojian+Yu%2C+Bo+Han%2C+Li+Shen%2C+Jun+Yu%2C+Chen+Gong%2C+Mingming+Gong%2C+and+Tongliang+Liu%2E+2022%2E+Understanding+robust+overfitting+of+adversarial+training+and+beyond%2E+In+International+Conference+on+Machine+Lea)
46. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, and 1 others. 2025. [Dapo: An open-source llm reinforcement learning system at scale](https://md.research.elamur.ai/free/en/papers/yu-2025-dapo-open-source-llm-reinforcement/). arXiv preprint arXiv:2503.14476.
47. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. 2025. [Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?](https://md.research.elamur.ai/free/en/papers/yue-2025-reinforcement-learning-really-incentivize-reasoning/) Advances in Neural Information Processing Systems.
48. Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. 2025. [Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild](https://md.research.elamur.ai/free/en/papers/zeng-2025-simplerl-zoo-investigating-taming-zero/). In Proceedings of the Conference On Language Modeling.
49. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, and 1 others. 2023. [A survey of large language models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/). arXiv preprint arXiv:2303.18223.
50. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. [find](https://scholar.google.com/scholar?q=Daniel+M+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+2019%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint+ar)

*Updated 2 November 2026*