---
title: "Exploring Data Scaling Trends and Effects in Reinforcement"
description: "This ByteDance Seed study asks why adding more prompt data to PPO-based RLHF stops improving… 33 references, cited by 12 papers here."
authors: ["Wei Shen", "Guanlin Liu", "Zheng Wu", "Ruofei Zhu", "Qingping Yang", "Chao Xin", "Yu Yue", "Lin Yan"]
year: 2025
venue: "arXiv:2503.22230"
source: "https://arxiv.org/abs/2503.22230"
doi: "10.48550/arxiv.2503.22230"
arxiv: "2503.22230"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shen-2025-exploring-data-scaling-trends-effects/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shen-2025-exploring-data-scaling-trends-effects/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/9/) / Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback

# Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback

**Authors:** [Wei Shen](https://md.research.elamur.ai/free/en/authors/wei-shen/), [Guanlin Liu](https://md.research.elamur.ai/free/en/authors/guanlin-liu/), [Zheng Wu](https://md.research.elamur.ai/free/en/authors/zheng-wu/), [Ruofei Zhu](https://md.research.elamur.ai/free/en/authors/ruofei-zhu/), Qingping Yang, [Chao Xin](https://md.research.elamur.ai/free/en/authors/chao-xin/), [Yu Yue](https://md.research.elamur.ai/free/en/authors/yu-yue/) and [Lin Yan](https://md.research.elamur.ai/free/en/authors/lin-yan/) · **Year:** 2025 · **Venue:** arXiv:2503.22230 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2503.22230) · [PDF](https://arxiv.org/pdf/2503.22230) · [DOI 10.48550/arxiv.2503.22230](https://doi.org/10.48550/arxiv.2503.22230) · [arXiv 2503.22230](https://arxiv.org/abs/2503.22230) · [OpenAlex](https://openalex.org/W6891837642)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [12](https://md.research.elamur.ai/free/en/papers/shen-2025-exploring-data-scaling-trends-effects/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This ByteDance Seed study asks why adding more prompt data to PPO-based RLHF stops improving large language models. Prior work focused mostly on algorithms. The authors trace the plateau to two causes: reward hacking, and a collapse in response diversity when the policy mainly picks up coarse differences between answers. Their pipeline has three steps: SFT, training three reward signals, and PPO. The signals are a Bradley-Terry model, a pairwise GenRM, and Reasoning Task Verifiers (RTV) such as code sandboxes. GenRM judges responses against ground-truth answers for reasoning tasks. For other tasks, it compares against a Best-of-N response chosen by the BT model from SFT samples. On the data side, Pre-PPO keeps the prompts that the model finds hard, since these are less prone to hacking and contain subtle response differences. A curriculum also places math and coding prompts first. Tested at two model sizes, these changes improved overall RLHF results. The work shows that how prompts are curated matters as much as algorithm design.

## Key points
- The authors identify two bottlenecks to RLHF data scaling: reward hacking, and shrinking diversity of model responses.
- Robustness to reward hacking ranks as follows: RTV is strongest, GenRM with ground truth comes second, and GenRM relying on SFT Best-of-N references comes last. RTV and ground-truth GenRM also detect fine-grained distinctions better.
- Pre-PPO selects a subset of challenging prompts for training; the framework figure shows about 10% being selected. A two-stage schedule trains on code and math prompts first.
- Without these changes, baseline RLHF performed only about as well as SFT Bo5, meaning five SFT samples reranked by the reward model.
- In the discussion, the authors suggest that long chain-of-thought (O1-style) RL works by turning fine-grained differences into coarse ones. They also propose CAI-style datasets to make GenRM without ground truth more sensitive to subtle differences.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

## Cite

Cite as: Wei Shen, Guanlin Liu, Zheng Wu, Ruofei Zhu, Qingping Yang, Chao Xin, Yu Yue and Lin Yan. 2025. Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback. arXiv:2503.22230.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{shen2025exploring,
  title = {{Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback}},
  author = {Wei Shen, Guanlin Liu, Zheng Wu, Ruofei Zhu, Qingping Yang, Chao Xin, Yu Yue and Lin Yan},
  year = {2025},
  doi = {10.48550/arxiv.2503.22230},
  eprint = {2503.22230},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.22230}
}
```

- [← previous Empowering GraphRAG with Knowledge Filtering and Integration](https://md.research.elamur.ai/free/en/papers/guo-2025-empowering-graphrag-knowledge-filtering-integration/)
- [next → from Benign import Toxic: Jailbreaking the Language Model via Adversarial Metaphors](https://md.research.elamur.ai/free/en/papers/yan-2025-benign-import-toxic-jailbreaking-language/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/9/)

## References

33 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
2. AI Anthropic. Introducing claude, 2023. URL [https://www.anthropic.com/news/introducing-claude](https://www.anthropic.com/news/introducing-claude). [↗](https://www.anthropic.com/news/introducing-claude)
3. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+Training+a+helpful+and+harmless+assistant+with+reinforcem)
4. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073, 2022.
5. Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.02743, 2023. — [arXiv:2310.02743](https://arxiv.org/abs/2310.02743) [find](https://scholar.google.com/scholar?q=Thomas+Coste%2C+Usman+Anwar%2C+Robert+Kirk%2C+and+David+Krueger%2E+Reward+model+ensembles+help+mitigate+overoptimization%2E+arXiv+preprint+arXiv%3A2310%2E02743%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E02743%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310)
6. Chengqian Gao, Haonan Li, Liu Liu, Zeke Xie, Peilin Zhao, and Zhiqiang Xu. Principled data selection for alignment: The hidden risks of difficult examples. arXiv preprint arXiv:2502.09650, 2025. — [arXiv:2502.09650](https://arxiv.org/abs/2502.09650) [find](https://scholar.google.com/scholar?q=Chengqian+Gao%2C+Haonan+Li%2C+Liu+Liu%2C+Zeke+Xie%2C+Peilin+Zhao%2C+and+Zhiqiang+Xu%2E+Principled+data+selection+for+alignment%3A+The+hidden+risks+of+difficult+examples%2E+arXiv+preprint+arXiv%3A2502%2E09650%2C+2025%2E+%E2%80%94+%5Bar)
7. Leo Gao, John Schulman, and Jacob Hilton. [Scaling laws for reward model overoptimization](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/). In International Conference on Machine Learning, pages 10835–10866. PMLR, 2023.
8. Lin Gui, Cristina Gârbacea, and Victor Veitch. Bonbon alignment for large language models and the sweetness of best-of-n sampling. arXiv preprint arXiv:2406.00832, 2024. — [arXiv:2406.00832](https://arxiv.org/abs/2406.00832) [find](https://scholar.google.com/scholar?q=Lin+Gui%2C+Cristina+G%C3%A2rbacea%2C+and+Victor+Veitch%2E+Bonbon+alignment+for+large+language+models+and+the+sweetness+of+best%2Dof%2Dn+sampling%2E+arXiv+preprint+arXiv%3A2406%2E00832%2C+2024%2E+%E2%80%94+%5BarXiv%3A2406%2E00832%5D%28https%3A%2F%2Fa)
9. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948, 2025.
10. Zhenyu Hou, Pengfan Du, Yilin Niu, Zhengxiao Du, Aohan Zeng, Xiao Liu, Minlie Huang, Hongning Wang, Jie Tang, and Yuxiao Dong. Does rlhf scale? exploring the impacts from data, model, and method. arXiv preprint arXiv:2412.06000, 2024. — [arXiv:2412.06000](https://arxiv.org/abs/2412.06000) [find](https://scholar.google.com/scholar?q=Zhenyu+Hou%2C+Pengfan+Du%2C+Yilin+Niu%2C+Zhengxiao+Du%2C+Aohan+Zeng%2C+Xiao+Liu%2C+Minlie+Huang%2C+Hongning+Wang%2C+Jie+Tang%2C+and+Yuxiao+Dong%2E+Does+rlhf+scale%3F+exploring+the+impacts+from+data%2C+model%2C+and+method%2E+arXi)
11. Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. [Llm-blender: Ensembling large language models with pairwise ranking and generative fusion](https://md.research.elamur.ai/free/en/papers/jiang-2023-llm-blender-ensembling-large-language/). arXiv preprint arXiv:2306.02561, 2023.
12. Adam Tauman Kalai and Santosh S Vempala. Calibrated language models must hallucinate. In Proceedings of the 56th Annual ACM Symposium on Theory of Computing, pages 160–171, 2024. [find](https://scholar.google.com/scholar?q=Adam+Tauman+Kalai+and+Santosh+S+Vempala%2E+Calibrated+language+models+must+hallucinate%2E+In+Proceedings+of+the+56th+Annual+ACM+Symposium+on+Theory+of+Computing%2C+pages+160%E2%80%93171%2C+2024%2E)
13. Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. [Understanding the effects of rlhf on llm generalisation and diversity](https://md.research.elamur.ai/free/en/papers/kirk-2023-understanding-effects-rlhf-llm-generalisation/). arXiv preprint arXiv:2310.06452, 2023.
14. Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. [Reward design with language models](https://md.research.elamur.ai/free/en/papers/kwon-2023-reward-design-language-models/). arXiv preprint arXiv:2303.00001, 2023.
15. Nathan Lambert, Thomas Krendl Gilbert, and Tom Zick. The history and risks of reinforcement learning and human feedback. arXiv e-prints, pages arXiv–2310, 2023. [find](https://scholar.google.com/scholar?q=Nathan+Lambert%2C+Thomas+Krendl+Gilbert%2C+and+Tom+Zick%2E+The+history+and+risks+of+reinforcement+learning+and+human+feedback%2E+arXiv+e%2Dprints%2C+pages+arXiv%E2%80%932310%2C+2023%2E)
16. Xuefeng Li, Haoyang Zou, and Pengfei Liu. [Limr: Less is more for rl scaling](https://md.research.elamur.ai/free/en/papers/li-2025-limr-less-more-rl-scaling/). arXiv preprint arXiv:2502.11886, 2025.
17. Ziniu Li, Tian Xu, Yushun Zhang, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient method for aligning large language models. arXiv preprint arXiv:2310.10505, 2023. [find](https://scholar.google.com/scholar?q=Ziniu+Li%2C+Tian+Xu%2C+Yushun+Zhang%2C+Yang+Yu%2C+Ruoyu+Sun%2C+and+Zhi%2DQuan+Luo%2E+Remax%3A+A+simple%2C+effective%2C+and+efficient+method+for+aligning+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E10505%2C+2023%2E)
18. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+et+al%2E+Deepseek%2Dv3+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E19437%2C+2024%2E)
19. Lev McKinney, Yawen Duan, David Krueger, and Adam Gleave. On the fragility of learned reward functions. arXiv preprint arXiv:2301.03652, 2023. — [arXiv:2301.03652](https://arxiv.org/abs/2301.03652) [find](https://scholar.google.com/scholar?q=Lev+McKinney%2C+Yawen+Duan%2C+David+Krueger%2C+and+Adam+Gleave%2E+On+the+fragility+of+learned+reward+functions%2E+arXiv+preprint+arXiv%3A2301%2E03652%2C+2023%2E+%E2%80%94+%5BarXiv%3A2301%2E03652%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2301%2E03652%29)
20. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
21. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). arXiv preprint arXiv:2203.02155, 2022.
22. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
23. Silviu Pitis. Failure modes of learning reward models for llms and other sequence models. In ICML 2023 Workshop The Many Facets of Preference-Based Learning, 2023. [find](https://scholar.google.com/scholar?q=Silviu+Pitis%2E+Failure+modes+of+learning+reward+models+for+llms+and+other+sequence+models%2E+In+ICML+2023+Workshop+The+Many+Facets+of+Preference%2DBased+Learning%2C+2023%2E)
24. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2305.18290. [↗](https://arxiv.org/abs/2305.18290)
25. Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Ramé, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, et al. [Bond: Aligning llms with best-of-n distillation](https://md.research.elamur.ai/free/en/papers/sessa-2024-bond-aligning-llms-best-n/). arXiv preprint arXiv:2407.14622, 2024.
26. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300, 2024.
27. Wei Shen and Chuheng Zhang. Policy filtration in rlhf to fine-tune llm for code generation. arXiv preprint arXiv:2409.06957, 2024. — [arXiv:2409.06957](https://arxiv.org/abs/2409.06957) [find](https://scholar.google.com/scholar?q=Wei+Shen+and+Chuheng+Zhang%2E+Policy+filtration+in+rlhf+to+fine%2Dtune+llm+for+code+generation%2E+arXiv+preprint+arXiv%3A2409%2E06957%2C+2024%2E+%E2%80%94+%5BarXiv%3A2409%2E06957%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2409%2E06957%29)
28. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
29. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
30. Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilities. Microsoft Auton. Syst. Robot. Res, 2:20, 2023. [find](https://scholar.google.com/scholar?q=Sai+Vemprala%2C+Rogerio+Bonatti%2C+Arthur+Bucker%2C+and+Ashish+Kapoor%2E+Chatgpt+for+robotics%3A+Design+principles+and+model+abilities%2E+Microsoft+Auton%2E+Syst%2E+Robot%2E+Res%2C+2%3A20%2C+2023%2E)
31. Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. [Secrets of rlhf in large language models part ii: Reward modeling](https://md.research.elamur.ai/free/en/papers/wang-2024-secrets-rlhf-large-language-models/). arXiv preprint arXiv:2401.06080, 2024.
32. Yuan Wu et al. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024. [find](https://scholar.google.com/scholar?q=Yuan+Wu+et+al%2E+Back+to+basics%3A+Revisiting+reinforce+style+optimization+for+learning+from+human+feedback+in+llms%2E+arXiv+preprint+arXiv%3A2402%2E14740%2C+2024%2E)
33. Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. [Generative verifiers: Reward modeling as next-token prediction](https://md.research.elamur.ai/free/en/papers/zhang-2024-generative-verifiers-reward-modeling-as/). arXiv preprint arXiv:2408.15240, 2024.

*Updated 3 November 2026*