---
title: "VAPO — paper by Yue et al. (2025) — summary and ideas"
description: "ByteDance Seed's VAPO argues that a learned value model has more potential, because it gives… 30 references, cited by 67 papers here."
authors: ["Yu Yue", "Yufeng Yuan", "Qiying Yu", "Xiaochen Zuo", "Ruofei Zhu", "Wenyuan Xu", "Jiaze Chen", "Chengyi Wang", "TianTian Fan", "Zhengyin Du", "Xiangpeng Wei", "Xiangyu Yu", "Gaohong Liu", "Juncai Liu", "Lingjun Liu", "Haibin Lin", "Zhiqi Lin", "Bole Ma", "Chi Zhang", "Mofan Zhang", "Wang Zhang", "Hang Zhu", "Ru Zhang", "Xin Liu", "Mingxuan Wang", "Yonghui Wu", "Lin Yan"]
year: 2025
venue: "arXiv:2504.05118"
source: "https://arxiv.org/abs/2504.05118"
doi: "10.48550/arxiv.2504.05118"
arxiv: "2504.05118"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yue-2025-vapo-efficient-reliable-reinforcement-learning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yue-2025-vapo-efficient-reliable-reinforcement-learning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) / VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks

# VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks

**Authors:** [Yu Yue](https://md.research.elamur.ai/free/en/authors/yu-yue/), [Yufeng Yuan](https://md.research.elamur.ai/free/en/authors/yufeng-yuan/), [Qiying Yu](https://md.research.elamur.ai/free/en/authors/qiying-yu/), [Xiaochen Zuo](https://md.research.elamur.ai/free/en/authors/xiaochen-zuo/), [Ruofei Zhu](https://md.research.elamur.ai/free/en/authors/ruofei-zhu/), [Wenyuan Xu](https://md.research.elamur.ai/free/en/authors/wenyuan-xu/), [Jiaze Chen](https://md.research.elamur.ai/free/en/authors/jiaze-chen/), [Chengyi Wang](https://md.research.elamur.ai/free/en/authors/chengyi-wang/), [TianTian Fan](https://md.research.elamur.ai/free/en/authors/tiantian-fan/), [Zhengyin Du](https://md.research.elamur.ai/free/en/authors/zhengyin-du/), [Xiangpeng Wei](https://md.research.elamur.ai/free/en/authors/xiangpeng-wei/), Xiangyu Yu, Gaohong Liu, Juncai Liu, [Lingjun Liu](https://md.research.elamur.ai/free/en/authors/lingjun-liu/), [Haibin Lin](https://md.research.elamur.ai/free/en/authors/haibin-lin/), Zhiqi Lin, Bole Ma, [Chi Zhang](https://md.research.elamur.ai/free/en/authors/chi-zhang/), Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, [Xin Liu](https://md.research.elamur.ai/free/en/authors/xin-liu/), [Mingxuan Wang](https://md.research.elamur.ai/free/en/authors/mingxuan-wang/), [Yonghui Wu](https://md.research.elamur.ai/free/en/authors/yonghui-wu/) and Lin Yan · **Year:** 2025 · **Venue:** arXiv:2504.05118 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2504.05118) · [PDF](https://arxiv.org/pdf/2504.05118) · [DOI 10.48550/arxiv.2504.05118](https://doi.org/10.48550/arxiv.2504.05118) · [arXiv 2504.05118](https://arxiv.org/abs/2504.05118) · [OpenAlex](https://openalex.org/W4416118517)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [67](https://md.research.elamur.ai/free/en/papers/yue-2025-vapo-efficient-reliable-reinforcement-learning/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most RL work on long chain-of-thought reasoning uses critic-free methods such as GRPO and DAPO. These skip the value model and estimate advantages from group rewards. ByteDance Seed's VAPO argues that a learned value model has more potential, because it gives per-token credit assignment and lower-variance estimates, but only if three problems are fixed: biased value estimates over long trajectories, widely varying response lengths, and sparse verifier rewards.

VAPO starts from PPO and adds a new Length-adaptive GAE, where λ depends on response length. It combines this with six existing techniques: Value-Pretraining and Decoupled-GAE from VC-PPO, Clip-Higher and Token-level Loss from DAPO, a positive-example LM loss in the style of SIL, and Group-Sampling from GRPO.

Training starts from the Qwen2.5-32B base model with no SFT data. VAPO reaches 60.4 on AIME 2024 in about 5,000 steps, more than 10 points above DAPO and DeepSeek-R1-Zero-Qwen-32B. Repeated runs did not crash. The results show that critic-based RL can beat critic-free methods on hard reasoning tasks.

## Key points
- **AIME24 scores** (all on Qwen2.5-32B):
  - vanilla PPO: 5, because the value model collapses and responses shrink
  - GRPO (DeepSeek-R1-Zero-Qwen-32B): 47
  - DAPO: 50
  - VAPO: 60.4
- **Efficiency and stability:** VAPO matches DAPO's score using roughly 60% of DAPO's update steps. Three independent runs each peaked at 60–61, with steady entropy.
- **Ablations:**
  - removing Value-Pretraining caps the score near 11
  - removing Decoupled-GAE costs 27 points
  - Length-adaptive GAE adds 15 points
  - removing Clip-Higher limits the score to 46
  - Token-level Loss adds 7 points
  - the positive-example LM loss adds about 6 points
  - Group-Sampling adds 5 points
- **Setup:** each sampling round uses 512 prompts with 16 samples per prompt, and the mini-batch size is 512. Evaluation is the average pass rate over 32 samples, with top-p 0.7 and temperature 1.0.
- **Identified challenges:**
  - initializing the value model from a reward model introduces bias
  - fixed λ=0.95 makes the terminal reward fade over long sequences
  - response lengths vary widely
  - verifier rewards are sparse

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, Xin Liu, Mingxuan Wang, Yonghui Wu and Lin Yan. 2025. VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks. arXiv:2504.05118.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{yue2025vapo,
  title = {{VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks}},
  author = {Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, Xin Liu, Mingxuan Wang, Yonghui Wu and Lin Yan},
  year = {2025},
  doi = {10.48550/arxiv.2504.05118},
  eprint = {2504.05118},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2504.05118}
}
```

- [← previous MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL](https://md.research.elamur.ai/free/en/papers/wang-2025-mac-sql-multi-agent-collaborative/)
- [next → xLAM: A Family of Large Action Models to Empower AI Agent Systems](https://md.research.elamur.ai/free/en/papers/zhang-2025-xlam-family-large-action-models/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/)

## References

30 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms, 2024. URL https://arxiv.org/abs/2402.14740. [↗](https://arxiv.org/abs/2402.14740)
2. Anthropic. Claude 3.5 sonnet, 2024. URL [https://www.anthropic.com/news/claude-3-5-sonnet](https://www.anthropic.com/news/claude-3-5-sonnet). [↗](https://www.anthropic.com/news/claude-3-5-sonnet)
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
4. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research, 24(240):1–113, 2023.
5. Google DeepMind. Gemini 2.0 flash thinking, 2024. URL [https://deepmind.google/technologies/gemini/flash-thinking/](https://deepmind.google/technologies/gemini/flash-thinking/). [↗](https://deepmind.google/technologies/gemini/flash-thinking/)
6. DeepSeek-AI. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/), 2025. URL https://arxiv.org/abs/2501.12948. [↗](https://arxiv.org/abs/2501.12948)
7. Ron Good and Harold J. Fletcher. Reporting explained variance. Journal of Research in Science Teaching, 18(1): 1–7, 1981. doi: https://doi.org/10.1002/tea.3660180102. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/tea.3660180102. — [doi:10.1002/tea.3660180102](https://doi.org/10.1002/tea.3660180102) [↗](https://doi.org/10.1002/tea.3660180102)
8. Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.[03262, 2025](https://md.research.elamur.ai/free/en/papers/hu-2025-reinforce-stabilizing-critic-free-policy/).
9. Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. [Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model](https://md.research.elamur.ai/free/en/papers/hu-2025-open-reasoner-zero-open-source/), 2025. URL https://arxiv.org/abs/2503.24290. [↗](https://arxiv.org/abs/2503.24290)
10. Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 REINFORCE samples, get a baseline for free! In Deep Reinforcement Learning Meets Structured Prediction, ICLR 2019 Workshop, New Orleans, Louisiana, United States, May 6, 2019. OpenReview.net, 2019. URL [https://openreview.net/forum?id=r1lgTGL5DE](https://openreview.net/forum?id=r1lgTGL5DE). [↗](https://openreview.net/forum?id=r1lgTGL5DE)
11. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+et+al%2E+Deepseek%2Dv3+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E19437%2C+2024%2E)
12. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. [Understanding r1-zero-like training: A critical perspective](https://md.research.elamur.ai/free/en/papers/liu-2025-understanding-r1-zero-like-training/), 2025. URL https://arxiv.org/abs/2503.20783. [↗](https://arxiv.org/abs/2503.20783)
13. Zhiyu Mei, Wei Fu, Kaiwei Li, Guangju Wang, Huanchen Zhang, and Yi Wu. Real: Efficient rlhf training of large language models with parameter reallocation. In Proceedings of the Eighth Conference on Machine Learning and Systems, MLSys 2025, Santa Clara, CA, USA, May 12-15, 2025. mlsys.org, 2025. [find](https://scholar.google.com/scholar?q=Zhiyu+Mei%2C+Wei+Fu%2C+Kaiwei+Li%2C+Guangju+Wang%2C+Huanchen+Zhang%2C+and+Yi+Wu%2E+Real%3A+Efficient+rlhf+training+of+large+language+models+with+parameter+reallocation%2E+In+Proceedings+of+the+Eighth+Conference+on+Ma)
14. Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3878–3887. PMLR, 10–15 Jul 2018. URL https:/… [find](https://scholar.google.com/scholar?q=Junhyuk+Oh%2C+Yijie+Guo%2C+Satinder+Singh%2C+and+Honglak+Lee%2E+Self%2Dimitation+learning%2E+In+Jennifer+Dy+and+Andreas+Krause%2C+editors%2C+Proceedings+of+the+35th+International+Conference+on+Machine+Learning%2C+volum)
15. OpenAI. GPT4 technical report. arXiv preprint arXiv:2303.[08774, 2023](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
16. OpenAI. Learning to reason learning-to-reason-with-llms/. with [find](https://scholar.google.com/scholar?q=OpenAI%2E+Learning+to+reason+learning%2Dto%2Dreason%2Dwith%2Dllms%2F%2E+with)
17. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 35:27730–27744, 2022.
18. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 35:27730–27744, 2022. llms, 20…
19. Qwen. Qwq-32b: Embracing the power of reinforcement learning, 2024. URL [https://qwenlm.github.io/blog/qwq-32b/](https://qwenlm.github.io/blog/qwq-32b/). [↗](https://qwenlm.github.io/blog/qwq-32b/)
20. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Philipp+Moritz%2C+Sergey+Levine%2C+Michael+Jordan%2C+and+Pieter+Abbeel%2E+High%2Ddimensional+continuous+control+using+generalized+advantage+estimation%2E+arXiv+preprint+arXiv%3A1506%2E02438%2C+2015%2E)
21. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)
22. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300, 2024.
23. Wei Shen, Guanlin Liu, Zheng Wu, Ruofei Zhu, Qingping Yang, Chao Xin, Yu Yue, and Lin Yan. Exploring data scaling trends and effects in reinforcement learning from human feedback. arXiv preprint arXiv:2503.22230, 2025. [find](https://scholar.google.com/scholar?q=Wei+Shen%2C+Guanlin+Liu%2C+Zheng+Wu%2C+Ruofei+Zhu%2C+Qingping+Yang%2C+Chao+Xin%2C+Yu+Yue%2C+and+Lin+Yan%2E+Exploring+data+scaling+trends+and+effects+in+reinforcement+learning+from+human+feedback%2E+arXiv+preprint+arXiv)
24. Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton%2C+Andrew+G+Barto%2C+et+al%2E+Reinforcement+learning%3A+An+introduction%2C+volume+1%2E+MIT+press+Cambridge%2C+1998%2E)
25. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
26. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.[12599, 2025](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k1-5-scaling-reinforcement/).
27. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Info…
28. XAI. Grok 3 beta — the age of reasoning agents, 2024. URL [https://x.ai/news/grok-3](https://x.ai/news/grok-3). [↗](https://x.ai/news/grok-3)
29. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan D… — [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://md.research.elamur.ai/free/en/papers/yu-2025-dapo-open-source-llm-reinforcement/)
30. Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What’s behind ppo’s collapse in long-cot? value optimization holds the secret, 2025. URL https://arxiv.org/abs/2503.01491. — [arXiv:2503.01491](https://arxiv.org/abs/2503.01491) [↗](https://arxiv.org/abs/2503.01491)

*Updated 2 November 2026*