---
title: "OpenR — paper by Wang et al. (2024) — summary and ideas"
description: "OpenAI's o1 suggested that LLMs reason better when they use extra compute at inference and learn… 33 references, cited by 48 papers here."
authors: ["Jun Wang", "Meng Fang", "Ziyu Wan", "Muning Wen", "Jiachen Zhu", "Anjie Liu", "Ziqin Gong", "Yan Song", "Lei Chen", "Lionel M. Ni", "Linyi Yang", "Ying Wen", "Weinan Zhang"]
year: 2024
venue: "arXiv:2410.09671"
source: "https://arxiv.org/abs/2410.09671"
doi: "10.48550/arxiv.2410.09671"
arxiv: "2410.09671"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2024-openr-open-source-framework-advanced/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2024-openr-open-source-framework-advanced/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/) / OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models

# OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models

**Authors:** [Jun Wang](https://md.research.elamur.ai/free/en/authors/jun-wang/), [Meng Fang](https://md.research.elamur.ai/free/en/authors/meng-fang/), Ziyu Wan, [Muning Wen](https://md.research.elamur.ai/free/en/authors/muning-wen/), [Jiachen Zhu](https://md.research.elamur.ai/free/en/authors/jiachen-zhu/), Anjie Liu, Ziqin Gong, [Yan Song](https://md.research.elamur.ai/free/en/authors/yan-song/), [Lei Chen](https://md.research.elamur.ai/free/en/authors/lei-chen/), [Lionel M. Ni](https://md.research.elamur.ai/free/en/authors/lionel-ming-shuan-ni/), [Linyi Yang](https://md.research.elamur.ai/free/en/authors/linyi-yang/), [Ying Wen](https://md.research.elamur.ai/free/en/authors/ying-wen/) and [Weinan Zhang](https://md.research.elamur.ai/free/en/authors/weinan-zhang/) · **Year:** 2024 · **Venue:** arXiv:2410.09671 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2410.09671) · [PDF](https://arxiv.org/pdf/2410.09671) · [DOI 10.48550/arxiv.2410.09671](https://doi.org/10.48550/arxiv.2410.09671) · [arXiv 2410.09671](https://arxiv.org/abs/2410.09671) · [OpenAlex](https://openalex.org/W4403570038)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [48](https://md.research.elamur.ai/free/en/papers/wang-2024-openr-open-source-framework-advanced/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
OpenAI's o1 suggested that LLMs reason better when they use extra compute at inference and learn through reinforcement learning. However, no open tooling combined these ideas in one place. OpenR is an open-source platform that brings the needed parts together: automatic creation of process-supervision data, training of process reward models (PRMs), online and offline RL for the policy model, and search-based decoding at test time. The framework treats reasoning as a Markov decision process. Each action adds a new reasoning step or gives the final answer, and the PRM scores the intermediate steps. That score can steer search and also serve as a reward signal for RL. The authors train PRMs on PRM800K, Math-Shepherd and their own synthetic MATH-APS set, then test on the MATH benchmark. Pairing a PRM with guided search lifted test-time reasoning performance by about 10%. The main contribution is shared code, models and datasets, so others can reproduce and extend o1-style reasoning research.

## Key points
- OpenR combines data acquisition, RL training (online and offline), PRM-based process supervision and non-autoregressive, search-based decoding in a single open framework. The authors describe it as the first open project to explore o1's core methods with RL.
- Evaluation on the MATH dataset shows that PRMs combined with guided search raise reasoning performance by roughly 10%.
- Process supervision data comes from PRM800K, Math-Shepherd and the newly generated MATH-APS dataset. Synthetic labeling reduces reliance on expensive human annotation.
- Case studies compare test-time methods. Best-of-N and Beam Search solved sample problems correctly, while plain CoT made errors. The authors attribute this to the wider search space of the first two methods.
- Limitations include mid-sized base models, small-scale experiments and limited process-supervision data. Code, models and data are released at https://openreasoner.github.io.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 232 works · 20,305 citations
- [University College London](https://md.research.elamur.ai/free/en/universities/university-college-london/) — University, London, United Kingdom · 119 works · 23,545 citations
- [Westlake University](https://md.research.elamur.ai/free/en/universities/westlake-university/) — University, Hangzhou, China · 60 works · 8,829 citations
- [The Hong Kong University of Science and Technology (Guangzhou)](https://md.research.elamur.ai/free/en/universities/the-hong-kong-university-of-science-and-technology-guangzhou/) — University, Guangzhou, China · 73 works · 1,936 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,621 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,948 works

## Cite

Cite as: Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M. Ni, Linyi Yang, Ying Wen and Weinan Zhang. 2024. OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models. arXiv:2410.09671.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wang2024openr,
  title = {{OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models}},
  author = {Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M. Ni, Linyi Yang, Ying Wen and Weinan Zhang},
  year = {2024},
  doi = {10.48550/arxiv.2410.09671},
  eprint = {2410.09671},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.09671}
}
```

- [← previous mDPO: Conditional Preference Optimization for Multimodal Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2024-mdpo-conditional-preference-optimization-multimodal/)
- [next → Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback](https://md.research.elamur.ai/free/en/papers/ivison-2024-unpacking-dpo-ppo-disentangling-best/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/)

## References

33 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. OpenAI. Learning to reason with learning-to-reason-with-llms/, 2024. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Learning+to+reason+with+learning%2Dto%2Dreason%2Dwith%2Dllms%2F%2C+2024%2E)
2. Daniel Kahneman. Thinking, Fast and Slow. Farrar, Straus and Giroux, New York, 2011. [find](https://scholar.google.com/scholar?q=Daniel+Kahneman%2E+Thinking%2C+Fast+and+Slow%2E+Farrar%2C+Straus+and+Giroux%2C+New+York%2C+2011%2E)
3. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.[11903, 2022](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/).
4. Maxwell Nye, Anders J Andreassen, Guy Gur-Ari, Henryk Michalewski, David Dohan, Jackie Jiang, John Schulman, William Fedus, and Charles Sutton. [Show your work: Scratchpads for intermediate computation with language models](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/). arXiv preprint arXiv:2112.00114, 2021.
5. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35: 27730–27744, 2022.
6. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making large language models better reasoners with step-aware verifier. arXiv preprint arXiv:2206.02336, 2022. — [arXiv:2206.02336](https://arxiv.org/abs/2206.02336) [find](https://scholar.google.com/scholar?q=Yifei+Li%2C+Zeqi+Lin%2C+Shizhuo+Zhang%2C+Qiang+Fu%2C+Bei+Chen%2C+Jian%2DGuang+Lou%2C+and+Weizhu+Chen%2E+Making+large+language+models+better+reasoners+with+step%2Daware+verifier%2E+arXiv+preprint+arXiv%3A2206%2E02336%2C+2022%2E+%E2%80%94)
7. Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. [Improve mathematical reasoning in language models by automated process supervision](https://md.research.elamur.ai/free/en/papers/luo-2024-improve-mathematical-reasoning-language-models/). arXiv preprint arXiv:2406.06592, 2024.
8. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. [Scaling llm test-time compute optimally can be more effective than scaling model parameters](https://md.research.elamur.ai/free/en/papers/snell-2024-scaling-llm-test-time-compute/). arXiv preprint arXiv:2408.03314, 2024. llms. [https://openai.com/index/](https://openai.com/index/) [↗](https://openai.com/index/)
9. Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.[00724, 2024](https://md.research.elamur.ai/free/en/papers/wu-2024-inference-scaling-laws-empirical-analysis/).
10. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361, 2020.
11. Xidong Feng, Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training. In ICML 2024, 2024. [find](https://scholar.google.com/scholar?q=Xidong+Feng%2C+Ziyu+Wan%2C+Muning+Wen%2C+Ying+Wen%2C+Weinan+Zhang%2C+and+Jun+Wang%2E+Alphazero%2Dlike+tree%2Dsearch+can+guide+large+language+model+decoding+and+training%2E+In+ICML+2024%2C+2024%2E)
12. Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992, 2023. [find](https://scholar.google.com/scholar?q=Shibo+Hao%2C+Yi+Gu%2C+Haodi+Ma%2C+Joshua+Jiahua+Hong%2C+Zhen+Wang%2C+Daisy+Zhe+Wang%2C+and+Zhiting+Hu%2E+Reasoning+with+language+model+is+planning+with+world+model%2E+arXiv+preprint+arXiv%3A2305%2E14992%2C+2023%2E)
13. Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. arXiv preprint arXiv:2310.02226, 2023. [find](https://scholar.google.com/scholar?q=Sachin+Goyal%2C+Ziwei+Ji%2C+Ankit+Singh+Rawat%2C+Aditya+Krishna+Menon%2C+Sanjiv+Kumar%2C+and+Vaishnavh+Nagarajan%2E+Think+before+you+speak%3A+Training+language+models+with+pause+tokens%2E+arXiv+preprint+arXiv%3A2310%2E02)
14. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021a.
15. Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.[14275, 2022](https://md.research.elamur.ai/free/en/papers/uesato-2022-solving-math-word-problems-process/).
16. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.[20050, 2023](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/).
17. Fei Yu, Anningzhe Gao, and Benyou Wang. [Ovm, outcome-supervised value models for planning in mathematical reasoning](https://md.research.elamur.ai/free/en/papers/yu-2023-ovm-outcome-supervised-value-models/). In Findings of the Association for Computational Linguistics: NAACL 2024, pages 858–875, 2024.
18. Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. [Generative verifiers: Reward modeling as next-token prediction](https://md.research.elamur.ai/free/en/papers/zhang-2024-generative-verifiers-reward-modeling-as/). arXiv preprint arXiv:2408.15240, 2024.
19. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. [Star: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). Advances in Neural Information Processing Systems, 35:15476–15488, 2022.
20. Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. [Quiet-star: Language models can teach themselves to think before speaking](https://md.research.elamur.ai/free/en/papers/al-2024-quiet-star-language-models-teach/). arXiv preprint arXiv:2403.09629, 2024.
21. Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. [Math-shepherd: Verify and reinforce llms step-by-step without human annotations](https://md.research.elamur.ai/free/en/papers/wang-2024-math-shepherd-verify-reinforce-llms/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa…
22. Zihan Wang, Yunxuan Li, Yuexin Wu, Liangchen Luo, Le Hou, Hongkun Yu, and Jingbo Shang. [Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision](https://md.research.elamur.ai/free/en/papers/wang-2024-multi-step-problem-solving-through/). arXiv preprint arXiv:2402.02658, 2024b.
23. Richard Bellman. Dynamic programming and stochastic control processes. Information and control, 1(3):228–239, 1958. [find](https://scholar.google.com/scholar?q=Richard+Bellman%2E+Dynamic+programming+and+stochastic+control+processes%2E+Information+and+control%2C+1%283%29%3A228%E2%80%93239%2C+1958%2E)
24. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). NeurIPS, 2021.
25. Martijn Van Otterlo and Marco Wiering. Reinforcement learning and markov decision processes. In Reinforcement learning: State-of-the-art, pages 3–42. Springer, 2012. [find](https://scholar.google.com/scholar?q=Martijn+Van+Otterlo+and+Marco+Wiering%2E+Reinforcement+learning+and+markov+decision+processes%2E+In+Reinforcement+learning%3A+State%2Dof%2Dthe%2Dart%2C+pages+3%E2%80%9342%2E+Springer%2C+2012%2E)
26. Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. In International Conference on Machine Learning, pages 3676–3713. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Thomas+Carta%2C+Cl%C3%A9ment+Romac%2C+Thomas+Wolf%2C+Sylvain+Lamprier%2C+Olivier+Sigaud%2C+and+Pierre%2DYves+Oudeyer%2E+Grounding+large+language+models+in+interactive+environments+with+online+reinforcement+learning%2E+In+)
27. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)
28. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300, 2024.
29. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Philipp+Moritz%2C+Sergey+Levine%2C+Michael+Jordan%2C+and+Pieter+Abbeel%2E+High%2Ddimensional+continuous+control+using+generalized+advantage+estimation%2E+arXiv+preprint+arXiv%3A1506%2E02438%2C+2015%2E)
30. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021b.
31. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171, 2022.
32. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.[12122, 2024](https://md.research.elamur.ai/free/en/papers/yang-2024-qwen2-5-math-technical-report/).
33. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/), 2023.

*Updated 3 November 2026*