---
title: "OVM, Outcome-supervised Value Models for Planning in"
description: "Building on that, they propose OVM, a value model trained this way and used for… 84.7% accuracy. 29 references, cited by 73 papers here."
authors: ["Fei Yu", "Anningzhe Gao", "Benyou Wang"]
year: 2023
venue: "arXiv:2311.09724"
source: "https://arxiv.org/abs/2311.09724"
doi: "10.48550/arxiv.2311.09724"
arxiv: "2311.09724"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yu-2023-ovm-outcome-supervised-value-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yu-2023-ovm-outcome-supervised-value-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/) / OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning

# OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning

**Authors:** [Fei Yu](https://md.research.elamur.ai/free/en/authors/fei-yu/), [Anningzhe Gao](https://md.research.elamur.ai/free/en/authors/anningzhe-gao/) and [Benyou Wang](https://md.research.elamur.ai/free/en/authors/benyou-wang/) · **Year:** 2023 · **Venue:** [arXiv:2311.09724](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2023](https://md.research.elamur.ai/free/en/venues/findings/2023/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2311.09724) · [PDF](https://arxiv.org/pdf/2311.09724) · [DOI 10.48550/arxiv.2311.09724](https://doi.org/10.48550/arxiv.2311.09724) · [arXiv 2311.09724](https://arxiv.org/abs/2311.09724) · [OpenAlex](https://openalex.org/W4401042684)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [73](https://md.research.elamur.ai/free/en/papers/yu-2023-ovm-outcome-supervised-value-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLMs solving multi-step math problems often make an early mistake that carries through to a wrong final answer. Guided decoding tries to prevent this by scoring partial solutions step by step and keeping only the most promising candidates. The authors argue that a partial path should be judged by its chance of reaching a correct answer (a value, in planning terms) rather than by whether the steps so far are correct (a reward). They show theoretically that a verifier trained with outcome supervision, where only the final answer's correctness labels every step, already estimates this value. Building on that, they propose OVM, a value model trained this way and used for beam-style search during generation. It needs no costly step-level annotations and no simulation at inference time. On GSM8K and Game of 24, OVM beats or matches process-supervised and complete-path verification baselines while sampling fewer paths. This suggests that cheap outcome labels can power effective planning for LLM reasoning.

## Key points
- On GSM8K, OVM-7B reaches 84.7% accuracy, the best reported result among LLMs up to 13B parameters. It uses no extra datasets, GPT-4 or code execution.
- On Game of 24, OVM-7B achieves a 78.7% success rate while visiting only 20 nodes per step. Greedy decoding reaches 11%, and majority voting over 100 paths reaches 11.7%.
- The theory shows that the loss-minimizing output of an outcome-supervised model for a partial path equals the expected probability of a correct final answer, which is exactly the value.
- Process labels (logical correctness) and outcome-oriented PRM-O labels agree 98.6% of the time on GSM8K but only 56.9% on Game of 24. This explains why the process-supervised PRM is weaker on Game of 24.
- Stated limitations:
  - the extra model increases memory use and slows inference;
  - outcome supervision may require larger training data;
  - generalization was not tested.
- Code is available at github.com/FreedomIntelligence/OVM.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Chinese University of Hong Kong](https://md.research.elamur.ai/free/en/universities/chinese-university-of-hong-kong/) — University, Hong Kong, China · 159 works · 16,154 citations
- [Chinese University of Hong Kong, Shenzhen](https://md.research.elamur.ai/free/en/universities/chinese-university-of-hong-kong-shenzhen/) — University, Shenzhen, China · 56 works · 3,212 citations
- [Shenzhen Research Institute of Big Data](https://md.research.elamur.ai/free/en/universities/shenzhen-research-institute-of-big-data/) — University, Shenzhen, China · 20 works · 2,016 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Fei Yu, Anningzhe Gao and Benyou Wang. 2023. OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning. arXiv:2311.09724.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{yu2023ovm,
  title = {{OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning}},
  author = {Fei Yu, Anningzhe Gao and Benyou Wang},
  year = {2023},
  doi = {10.48550/arxiv.2311.09724},
  eprint = {2311.09724},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.09724}
}
```

- [← previous Language models are not naysayers: An analysis of language models on negation benchmarks](https://md.research.elamur.ai/free/en/papers/truong-2023-language-models-naysayers-analysis-language/)
- [next → Automated Annotation with Generative AI Requires Validation](https://md.research.elamur.ai/free/en/papers/pangakis-2023-automated-annotation-generative-ai-requires/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/)

## References

29 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. 2023. [Sparks of artificial general intelligence: Early experiments with GPT-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). CoRR, ab…
2. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). CoRR, abs/2110.14168.
3. Antonia Creswell, Murray Shanahan, and Irina Higgins. 2023. Selection-inference: Exploiting large language models for interpretable logical reasoning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. [find](https://scholar.google.com/scholar?q=Antonia+Creswell%2C+Murray+Shanahan%2C+and+Irina+Higgins%2E+2023%2E+Selection%2Dinference%3A+Exploiting+large+language+models+for+interpretable+logical+reasoning%2E+In+The+Eleventh+International+Conference+on+Learn)
4. Tri Dao. 2023. [Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). abs/2307.08691.
5. CoRR, Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. In NeurIPS. [find](https://scholar.google.com/scholar?q=CoRR%2C+Tri+Dao%2C+Daniel+Y%2E+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+2022%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2E+In+NeurIPS%2E)
6. Xidong Feng, Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang, and Jun Wang. 2023. Alphazero-like tree-search can guide large language model decoding and training. CoRR, abs/2309.17179. [find](https://scholar.google.com/scholar?q=Xidong+Feng%2C+Ziyu+Wan%2C+Muning+Wen%2C+Ying+Wen%2C+Weinan+Zhang%2C+and+Jun+Wang%2E+2023%2E+Alphazero%2Dlike+tree%2Dsearch+can+guide+large+language+model+decoding+and+training%2E+CoRR%2C+abs%2F2309%2E17179%2E)
7. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. [Tora: A tool-integrated reasoning agent for mathematical problem solving](https://md.research.elamur.ai/free/en/papers/gou-2023-tora-tool-integrated-reasoning-agent/). CoRR, abs/2309.17452.
8. Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. 2023. Reasoning with language model is planning with world model. CoRR, abs/2305.14992. [find](https://scholar.google.com/scholar?q=Shibo+Hao%2C+Yi+Gu%2C+Haodi+Ma%2C+Joshua+Jiahua+Hong%2C+Zhen+Wang%2C+Daisy+Zhe+Wang%2C+and+Zhiting+Hu%2E+2023%2E+Reasoning+with+language+model+is+planning+with+world+model%2E+CoRR%2C+abs%2F2305%2E14992%2E)
9. Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. [Large language models can self-improve](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/). CoRR, abs/2210.11610.
10. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. CoRR, abs/2210.03350. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Muru+Zhang%2C+Sewon+Min%2C+Ludwig+Schmidt%2C+Noah+A%2E+Smith%2C+and+Mike+Lewis%2E+2022%2E+Measuring+and+narrowing+the+compositionality+gap+in+language+models%2E+CoRR%2C+abs%2F2210%2E03350%2E)
11. Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. 2021. Generate & rank: A multi-task framework for math word problems. In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pa… [find](https://scholar.google.com/scholar?q=Jianhao+Shen%2C+Yichun+Yin%2C+Lin+Li%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Ming+Zhang%2C+and+Qun+Liu%2E+2021%2E+Generate+%26+rank%3A+A+multi%2Dtask+framework+for+math+word+problems%2E+In+Findings+of+the+Association+for+Computation)
12. Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton+and+Andrew+G+Barto%2E+2018%2E+Reinforcement+learning%3A+An+introduction%2E+MIT+press%2E)
13. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. [Challenging big-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). In Findings of the Association for Computational Linguistics…
14. Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. 2023. Discriminator-guided multi-step reasoning with language models. CoRR, abs/2305.14934. — [arXiv:2305.14934](https://arxiv.org/abs/2305.14934) [find](https://scholar.google.com/scholar?q=Muhammad+Khalifa%2C+Lajanugen+Logeswaran%2C+Moontae+Lee%2C+Honglak+Lee%2C+and+Lu+Wang%2E+2023%2E+Discriminator%2Dguided+multi%2Dstep+reasoning+with+language+models%2E+CoRR%2C+abs%2F2305%2E14934%2E+%E2%80%94+%5BarXiv%3A2305%2E14934%5D%28https%3A%2F%2F)
15. Jonathan Uesato, Nate Kushman, Ramana Kumar, H. Francis Song, Noah Y. Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. [Solving math word problems with process- and outcome-based feedback](https://md.research.elamur.ai/free/en/papers/uesato-2022-solving-math-word-problems-process/). CoRR, abs/2211.14275.
16. Chengpeng Li, Zheng Yuan, Hongyi Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang, and Chang Zhou. 2023a. Query and response augmentation cannot help out-of-domain math reasoning generalization. CoRR, abs/2310.05506. [find](https://scholar.google.com/scholar?q=Chengpeng+Li%2C+Zheng+Yuan%2C+Hongyi+Yuan%2C+Guanting+Dong%2C+Keming+Lu%2C+Jiancan+Wu%2C+Chuanqi+Tan%2C+Xiang+Wang%2C+and+Chang+Zhou%2E+2023a%2E+Query+and+response+augmentation+cannot+help+out%2Dof%2Ddomain+math+reasoning+ge)
17. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In NeurIPS.
18. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2022. On the advance of making language models better reasoners. CoRR, abs/2206.02336. — [arXiv:2206.02336](https://arxiv.org/abs/2206.02336) [find](https://scholar.google.com/scholar?q=Yifei+Li%2C+Zeqi+Lin%2C+Shizhuo+Zhang%2C+Qiang+Fu%2C+Bei+Chen%2C+Jian%2DGuang+Lou%2C+and+Weizhu+Chen%2E+2022%2E+On+the+advance+of+making+language+models+better+reasoners%2E+CoRR%2C+abs%2F2206%2E02336%2E+%E2%80%94+%5BarXiv%3A2206%2E02336%5D%28http)
19. Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. 2022. [Large language models are better reasoners with self-verification](https://md.research.elamur.ai/free/en/papers/weng-2023-large-language-models-are-better/). CoRR, abs/2212.09561.
20. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2023b. [Making language models better reasoners with step-aware verifier](https://md.research.elamur.ai/free/en/papers/li-2023-making-language-models-better-reasoners/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Cana…
21. Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, and Qizhe Xie. 2023. [Decomposition enhances reasoning via self-evaluation guided decoding](https://md.research.elamur.ai/free/en/papers/xie-2023-self-evaluation-guided-beam-search/). CoRR, abs/2305.00633.
22. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). CoRR, abs/2305.20050.
23. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. [Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/). CoRR, abs/2308.09583.
24. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. CoRR, abs/2305.10601. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L%2E+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+2023%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+CoRR%2C+abs%2F2305%2E1)
25. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). CoRR, abs/2309.12284.
26. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. 2023. [Scaling relationship on learning mathematical reasoning with large language models](https://md.research.elamur.ai/free/en/papers/yuan-2023-scaling-relationship-learning-mathematical-reasoning/). CoRR, abs/2308.01825.
27. Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, and Hongsheng Li. 2023. [Solving challenging math word problems using GPT-4 code interpreter with code-based self-verification](https://md.research.elamur.ai/free/en/papers/zhou-2023-solving-challenging-math-word-problems/). CoRR, abs/2308.07921.
28. Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang. 2023. [Solving math word problems via cooperative reasoning induced language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-solving-math-word-problems-cooperative/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa…
29. Notation q Q S s i a m y S (1:t) S (1:t) K b n Φ f (1:t) v θ PRM ORM OVM [find](https://scholar.google.com/scholar?q=Notation+q+Q+S+s+i+a+m+y+S+%281%3At%29+S+%281%3At%29+K+b+n+%CE%A6+f+%281%3At%29+v+%CE%B8+PRM+ORM+OVM)

*Updated 2 November 2026*