---
title: "OpenRLHF: An Easy-to-use, Scalable and — Hu 2024"
description: "Training LLMs with RLHF and RLVR, especially with PPO, is dominated by text generation."
authors: ["Jian Hu", "Xibin Wu", "Wei Shen", "Jason Klein Liu", "Zilin Zhu", "Weixun Wang", "Songlin Jiang", "Haoran Wang", "Hao Chen", "Bin Chen", "Weikai Fang", "Xianyu", "Yu Cao", "Haotian Xu", "Yiming Liu"]
year: 2024
venue: "arXiv:2405.11143"
source: "https://arxiv.org/abs/2405.11143"
doi: "10.48550/arxiv.2405.11143"
arxiv: "2405.11143"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hu-2024-openrlhf-easy-use-scalable/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hu-2024-openrlhf-easy-use-scalable/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework

# OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework

**Authors:** Jian Hu, [Xibin Wu](https://md.research.elamur.ai/free/en/authors/xibin-wu/), [Wei Shen](https://md.research.elamur.ai/free/en/authors/wei-shen/), Jason Klein Liu, Zilin Zhu, Weixun Wang, Songlin Jiang, [Haoran Wang](https://md.research.elamur.ai/free/en/authors/haoran-wang/), [Hao Chen](https://md.research.elamur.ai/free/en/authors/hao-chen/), [Bin Chen](https://md.research.elamur.ai/free/en/authors/bin-chen/), Weikai Fang, Xianyu et al. (15 authors) · **Year:** 2024 · **Venue:** arXiv:2405.11143 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2405.11143) · [PDF](https://arxiv.org/pdf/2405.11143) · [DOI 10.48550/arxiv.2405.11143](https://doi.org/10.48550/arxiv.2405.11143) · [arXiv 2405.11143](https://arxiv.org/abs/2405.11143) · [OpenAlex](https://openalex.org/W6959934489)

**Citations:** 1 citations (openalex-snapshot · 2026-10-14) · [131](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Training LLMs with RLHF and RLVR, especially with PPO, is dominated by text generation. The authors say this phase can take more than 90% of runtime when models produce long chain-of-thought outputs. Existing tools also force a trade-off. TRL, DeepSpeed-Chat and ColossalChat are approachable but weak at inference optimization, while Nemo-aligner, ChatLearn and verl are fast but tightly coupled and hard to learn. OpenRLHF is an open-source framework that aims to offer both speed and simplicity. Ray schedules GPUs into two roles. vLLM-based rollout engines generate responses, and DeepSpeed ZeRO engines compute log-probabilities and train HuggingFace Transformers models. Scaling relies on a "3D" combination of tensor parallelism (DeepSpeed AutoTP), data parallelism and ring-attention sequence parallelism. Asynchronous message passing between components reduces idle GPU time and makes agent-style RL easier to add. The paper reports speedups of 1.22× to 1.68× over state-of-the-art frameworks across model sizes, with a relatively small codebase. This makes it a practical entry point for alignment experiments on modest research infrastructure.

## Key points
- Stack: Ray handles orchestration, vLLM handles rollout and DeepSpeed ZeRO handles training. The code is at github.com/OpenRLHF/OpenRLHF and is used by CMU, MIT, Microsoft and HKUST.
- RLVR test on GSM8K with GRPO (one epoch): OpenRLHF took 1,657 s versus 5,189 s for TRL, about 3.1× faster.
- RLHF test with PPO on 1,024 prompts: OpenRLHF took 236.8 s versus 855.09 s for DeepSpeed-Chat, about 3.6× faster. In a 7B model with 2K context, it ran 1.56× faster than the comparison framework (30.3 s vs 47.3 s).
- The core code is about 8,523 lines, compared with 19,071 for TRL and 32,325 for verl. The framework also supports SFT, DPO, reward models and PRM.
- Limitations: there is no support for vision-language or other multimodal models. The framework depends on upstream Ray, vLLM and DeepSpeed releases, and its peak performance may trail well-funded industrial frameworks.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

131 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 126 not yet.

- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2025-towards-reasoning-era-survey-long/)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://md.research.elamur.ai/free/en/papers/lambert-2024-tulu-3-pushing-frontiers-open/)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://md.research.elamur.ai/free/en/papers/liu-2025-understanding-r1-zero-like-training/)
- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://md.research.elamur.ai/free/en/papers/wang-2025-80-20-rule-high-entropy/)
- [Demystifying Long Chain-of-Thought Reasoning in LLMs](https://md.research.elamur.ai/free/en/papers/yeo-2025-demystifying-long-chain-thought-reasoning/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hu2024openrlhf,
  title = {{OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework}},
  author = {Hu, Jian and Wu, Xibin and Shen, Wei and Liu, Jason Klein and Zhu, Zilin and Wang, Weixun and Jiang, Songlin and Wang, Haoran and Chen, Hao and Chen, Bin and Fang, Weikai and Xianyu and Cao, Yu and Xu, Haotian and Liu, Yiming},
  year = {2024},
  doi = {10.48550/arxiv.2405.11143},
  eprint = {2405.11143},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.11143}
}
```

- [← previous OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://md.research.elamur.ai/free/en/papers/wang-2024-openhands-open-platform-ai/)
- [next → Optimized Quran Passage Retrieval Using an Expanded QA Dataset and Fine-Tuned Language Models](https://md.research.elamur.ai/free/en/papers/basem-2024-optimized-quran-passage-retrieval-using/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

30 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. [find](https://scholar.google.com/scholar?q=Paul+F+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+neural+information+processing+systems%2C+30%2C+2017%2E)
2. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F+Christiano%2E+Learning+to+summarize+with+human+feedback%2E+Advances+in+Neural+Infor)
3. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. [find](https://scholar.google.com/scholar?q=Daya+Guo%2C+Dejian+Yang%2C+Haowei+Zhang%2C+Junxiao+Song%2C+Ruoyu+Zhang%2C+Runxin+Xu%2C+Qihao+Zhu%2C+Shirong+Ma%2C+Peiyi+Wang%2C+Xiao+Bi%2C+et+al%2E+Deepseek%2Dr1%3A+Incentivizing+reasoning+capability+in+llms+via+reinforcement+) [our card](https://md.research.elamur.ai/free/en/papers/deepseek-2025-ai-deepseek-r1-incentivizing-reasoning-capability/)
4. Wei Shen, Guanlin Liu, Zheng Wu, Ruofei Zhu, Qingping Yang, Chao Xin, Yu Yue, and Lin Yan. Exploring data scaling trends and effects in reinforcement learning from human feedback. arXiv preprint arXiv:2503.22230, 2025. [find](https://scholar.google.com/scholar?q=Wei+Shen%2C+Guanlin+Liu%2C+Zheng+Wu%2C+Ruofei+Zhu%2C+Qingping+Yang%2C+Chao+Xin%2C+Yu+Yue%2C+and+Lin+Yan%2E+Exploring+data+scaling+trends+and+effects+in+reinforcement+learning+from+human+feedback%2E+arXiv+preprint+arXiv)
5. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. [find](https://scholar.google.com/scholar?q=Josh+Achiam%2C+Steven+Adler%2C+Sandhini+Agarwal%2C+Lama+Ahmad%2C+Ilge+Akkaya%2C+Florencia+Leoni+Aleman%2C+Diogo+Almeida%2C+Janko+Altenschmidt%2C+Sam+Altman%2C+Shyamal+Anadkat%2C+et+al%2E+Gpt%2D4+technical+report%2E+arXiv+prepr) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
6. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. [find](https://scholar.google.com/scholar?q=Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Andy+Jones%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Nova+DasSarma%2C+et+al%2E+A+general+language+assistant+as+a+laboratory+for+alignment%2E+a)
7. Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, et al. Deepspeed-chat: Easy, fast and affordable rlhf training of chatgpt-like models at all scales. arXiv preprint arXiv:2308.01320, 2023. — [arXiv:2308.01320](https://arxiv.org/abs/2308.01320) [find](https://scholar.google.com/scholar?q=Zhewei+Yao%2C+Reza+Yazdani+Aminabadi%2C+Olatunji+Ruwase%2C+Samyam+Rajbhandari%2C+Xiaoxia+Wu%2C+Ammar+Ahmad+Awan%2C+Jeff+Rasley%2C+Minjia+Zhang%2C+Conglong+Li%2C+Connor+Holmes%2C+et+al%2E+Deepspeed%2Dchat%3A+Easy%2C+fast+and+affo)
8. Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. Trl: Transformer reinforcement learning. [https://github.com/huggingface/trl](https://github.com/huggingface/trl), 2020. [↗](https://github.com/huggingface/trl)
9. Shenggui Li, Hongxin Liu, Zhengda Bian, Jiarui Fang, Haichen Huang, Yuliang Liu, Boxiang Wang, and Yang You. Colossal-ai: A unified deep learning system for large-scale parallel training. In Proceedings of the 52nd International Conference on Parallel Processing, pages 766–775, 2023. [find](https://scholar.google.com/scholar?q=Shenggui+Li%2C+Hongxin+Liu%2C+Zhengda+Bian%2C+Jiarui+Fang%2C+Haichen+Huang%2C+Yuliang+Liu%2C+Boxiang+Wang%2C+and+Yang+You%2E+Colossal%2Dai%3A+A+unified+deep+learning+system+for+large%2Dscale+parallel+training%2E+In+Proceedin)
10. Gerald Shen, Zhilin Wang, Olivier Delalleau, Jiaqi Zeng, Yi Dong, Daniel Egert, Shengyang Sun, Jimmy Zhang, Sahil Jain, Ali Taghibakhshi, Markel Sanz Ausin, Ashwath Aithal, and Oleksii Kuchaiev. Nemo-aligner: Scalable toolkit for efficient model alignment, 2024. [find](https://scholar.google.com/scholar?q=Gerald+Shen%2C+Zhilin+Wang%2C+Olivier+Delalleau%2C+Jiaqi+Zeng%2C+Yi+Dong%2C+Daniel+Egert%2C+Shengyang+Sun%2C+Jimmy+Zhang%2C+Sahil+Jain%2C+Ali+Taghibakhshi%2C+Markel+Sanz+Ausin%2C+Ashwath+Aithal%2C+and+Oleksii+Kuchaiev%2E+Nemo%2D)
11. alibaba. chatlearn. [https://github.com/alibaba/ChatLearn](https://github.com/alibaba/ChatLearn), 2017. [↗](https://github.com/alibaba/ChatLearn)
12. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024. [find](https://scholar.google.com/scholar?q=Guangming+Sheng%2C+Chi+Zhang%2C+Zilingfeng+Ye%2C+Xibin+Wu%2C+Wang+Zhang%2C+Ru+Zhang%2C+Yanghua+Peng%2C+Haibin+Lin%2C+and+Chuan+Wu%2E+Hybridflow%3A+A+flexible+and+efficient+rlhf+framework%2E+arXiv+preprint+arXiv%3A+2409%2E19256) [our card](https://md.research.elamur.ai/free/en/papers/sheng-2024-hybridflow-flexible-efficient-rlhf-framework/)
13. Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. In International conference on machine learning, pages 3053–3062. PMLR, 2018. [find](https://scholar.google.com/scholar?q=Eric+Liang%2C+Richard+Liaw%2C+Robert+Nishihara%2C+Philipp+Moritz%2C+Roy+Fox%2C+Ken+Goldberg%2C+Joseph+Gonzalez%2C+Michael+Jordan%2C+and+Ion+Stoica%2E+Rllib%3A+Abstractions+for+distributed+reinforcement+learning%2E+In+Inter)
14. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pierric+Cistac%2C+Tim+Rault%2C+R%C3%A9mi+Louf%2C+Morgan+Funtowicz%2C+Joe+Davison%2C+Sam+Shleifer%2C+Patrick+von+Platen%2C+Clara+M) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
15. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)
16. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. [find](https://scholar.google.com/scholar?q=Long+Ouyang%2C+Jeffrey+Wu%2C+Xu+Jiang%2C+Diogo+Almeida%2C+Carroll+Wainwright%2C+Pamela+Mishkin%2C+Chong+Zhang%2C+Sandhini+Agarwal%2C+Katarina+Slama%2C+Alex+Ray%2C+et+al%2E+Training+language+models+to+follow+instructions+wi)
17. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [find](https://scholar.google.com/scholar?q=Karl+Cobbe%2C+Vineet+Kosaraju%2C+Mohammad+Bavarian%2C+Mark+Chen%2C+Heewoo+Jun%2C+Lukasz+Kaiser%2C+Matthias+Plappert%2C+Jerry+Tworek%2C+Jacob+Hilton%2C+Reiichiro+Nakano%2C+et+al%2E+Training+verifiers+to+solve+math+word+prob) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
18. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. [find](https://scholar.google.com/scholar?q=Daniel+M+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint+arXiv%3A19)
19. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023. [find](https://scholar.google.com/scholar?q=Rafael+Rafailov%2C+Archit+Sharma%2C+Eric+Mitchell%2C+Christopher+D+Manning%2C+Stefano+Ermon%2C+and+Chelsea+Finn%2E+Direct+preference+optimization%3A+Your+language+model+is+secretly+a+reward+model%2E+Advances+in+Neura)
20. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. [find](https://scholar.google.com/scholar?q=Woosuk+Kwon%2C+Zhuohan+Li%2C+Siyuan+Zhuang%2C+Ying+Sheng%2C+Lianmin+Zheng%2C+Cody+Hao+Yu%2C+Joseph+E%2E+Gonzalez%2C+Hao+Zhang%2C+and+Ion+Stoica%2E+Efficient+memory+management+for+large+language+model+serving+with+pagedat)
21. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020. [find](https://scholar.google.com/scholar?q=Samyam+Rajbhandari%2C+Jeff+Rasley%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+Zero%3A+Memory+optimizations+toward+training+trillion+parameter+models%2E+In+SC20%3A+International+Conference+for+High+Performance+Computing)
22. Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. [https://github.com/openai/baselines](https://github.com/openai/baselines), 2017. Ad- [↗](https://github.com/openai/baselines)
23. Wei Shen, Jian Hu, Pengyu Zhao, Xiaonan He, and Lichang Chen. vanced tricks for training large language models with proximal policy optimization. [https://swtheking.notion.site/eb7b2d1891f44b3a84e7396d19d39e6f?v=01bcb084210149488d730064cbabc99f&pvs=74](https://swtheking.notion.site/eb7b2d1891f44b3a84e7396d19d39e6f?v=01bcb084210149488d730064cbabc99f&pvs=74), 2024. Notion Blog. [↗](https://swtheking.notion.site/eb7b2d1891f44b3a84e7396d19d39e6f?v=01bcb084210149488d730064cbabc99f&pvs=74)
24. Shashank Mohan Jain. Hugging face. In Introduction to transformers for NLP: With the hugging face library and models to solve problems, pages 51–67. Springer, 2022. [find](https://scholar.google.com/scholar?q=Shashank+Mohan+Jain%2E+Hugging+face%2E+In+Introduction+to+transformers+for+NLP%3A+With+the+hugging+face+library+and+models+to+solve+problems%2C+pages+51%E2%80%9367%2E+Springer%2C+2022%2E)
25. Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01889, 2023. [find](https://scholar.google.com/scholar?q=Hao+Liu%2C+Matei+Zaharia%2C+and+Pieter+Abbeel%2E+Ring+attention+with+blockwise+transformers+for+near%2Dinfinite+context%2E+arXiv+preprint+arXiv%3A2310%2E01889%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-ring-attention-blockwise-transformers-near/)
26. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. [find](https://scholar.google.com/scholar?q=Qiying+Yu%2C+Zheng+Zhang%2C+Ruofei+Zhu%2C+Yufeng+Yuan%2C+Xiaochen+Zuo%2C+Yu+Yue%2C+Weinan+Dai%2C+Tiantian+Fan%2C+Gaohong+Liu%2C+Lingjun+Liu%2C+et+al%2E+Dapo%3A+An+open%2Dsource+llm+reinforcement+learning+system+at+scale%2E+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/yu-2025-dapo-open-source-llm-reinforcement/)
27. Sagar Imambi, Kolla Bhanu Prakash, and GR Kanagachidambaresan. Pytorch. Programming with TensorFlow: solution for edge computing applications, pages 87–104, 2021. [find](https://scholar.google.com/scholar?q=Sagar+Imambi%2C+Kolla+Bhanu+Prakash%2C+and+GR+Kanagachidambaresan%2E+Pytorch%2E+Programming+with+TensorFlow%3A+solution+for+edge+computing+applications%2C+pages+87%E2%80%93104%2C+2021%2E)
28. Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277, 2023. [find](https://scholar.google.com/scholar?q=Yanli+Zhao%2C+Andrew+Gu%2C+Rohan+Varma%2C+Liang+Luo%2C+Chien%2DChin+Huang%2C+Min+Xu%2C+Less+Wright%2C+Hamid+Shojanazeri%2C+Myle+Ott%2C+Sam+Shleifer%2C+et+al%2E+Pytorch+fsdp%3A+experiences+on+scaling+fully+sharded+data+parallel)
29. Kezhao Liu, Jason Klein Liu, Mingtao Chen, and Yiming Liu. Rethinking kl regularization in rlhf: From value estimation to gradient optimization. arXiv preprint arXiv:2510.01555, 2025. — [arXiv:2510.01555](https://arxiv.org/abs/2510.01555) [find](https://scholar.google.com/scholar?q=Kezhao+Liu%2C+Jason+Klein+Liu%2C+Mingtao+Chen%2C+and+Yiming+Liu%2E+Rethinking+kl+regularization+in+rlhf%3A+From+value+estimation+to+gradient+optimization%2E+arXiv+preprint+arXiv%3A2510%2E01555%2C+2025%2E+%E2%80%94+%5BarXiv%3A2510%2E01)
30. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=Hunter+Lightman%2C+Vineet+Kosaraju%2C+Yuri+Burda%2C+Harrison+Edwards%2C+Bowen+Baker%2C+Teddy+Lee%2C+Jan+Leike%2C+John+Schulman%2C+Ilya+Sutskever%2C+and+Karl+Cobbe%2E+Let%E2%80%99s+verify+step+by+step%2E+In+The+Twelfth+Internationa) [our card](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/)