---
title: "Rewarding Progress — paper by Setlur et al. (2024)"
description: "Process reward models (PRMs) score every step of a reasoning trace instead of only the final answer. 46 references, cited by 90 papers here."
authors: ["Amrith Setlur", "Chirag Nagpal", "Adam Fisch", "Xinyang Geng", "Jacob Eisenstein", "Rishabh Agarwal", "Alekh Agarwal", "Jonathan Berant", "Aviral Kumar"]
year: 2024
venue: "arXiv:2410.08146"
source: "https://arxiv.org/abs/2410.08146"
doi: "10.48550/arxiv.2410.08146"
arxiv: "2410.08146"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/setlur-2024-rewarding-progress-scaling-automated-process/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/setlur-2024-rewarding-progress-scaling-automated-process/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning

# Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning

**Authors:** [Amrith Setlur](https://md.research.elamur.ai/free/en/authors/amrith-setlur/), Chirag Nagpal, [Adam Fisch](https://md.research.elamur.ai/free/en/authors/adam-fisch/), [Xinyang Geng](https://md.research.elamur.ai/free/en/authors/xinyang-geng/), [Jacob Eisenstein](https://md.research.elamur.ai/free/en/authors/jacob-eisenstein/), [Rishabh Agarwal](https://md.research.elamur.ai/free/en/authors/rishabh-agarwal/), [Alekh Agarwal](https://md.research.elamur.ai/free/en/authors/alekh-agarwal/), [Jonathan Berant](https://md.research.elamur.ai/free/en/authors/jonathan-berant/) and [Aviral Kumar](https://md.research.elamur.ai/free/en/authors/aviral-kumar/) · **Year:** 2024 · **Venue:** arXiv:2410.08146 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2410.08146) · [PDF](https://arxiv.org/pdf/2410.08146) · [DOI 10.48550/arxiv.2410.08146](https://doi.org/10.48550/arxiv.2410.08146) · [arXiv 2410.08146](https://arxiv.org/abs/2410.08146) · [OpenAlex](https://openalex.org/W4403365357)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [90](https://md.research.elamur.ai/free/en/papers/setlur-2024-rewarding-progress-scaling-automated-process/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Process reward models (PRMs) score every step of a reasoning trace instead of only the final answer. Step labels written by humans are expensive, though, and earlier automatically labelled PRMs gave only marginal gains over outcome reward models (ORMs). This work asks what a step-level reward should actually measure. The authors' answer is that a step should earn credit for progress: how much it raises or lowers the chance of eventually reaching a correct answer. In RL terms this is the step's advantage, and it is computed under a separate "prover" policy rather than the base policy itself. Their theory says useful provers must be able to separate good steps from bad ones produced by the base model while staying broadly aligned with it. Provers that are too strong or too weak fail this test, and even a prover weaker than the base model can improve it. The authors train process advantage verifiers (PAVs) on rollout data collected automatically and test them with Gemma2 models on math reasoning. PAVs make both test-time search and online RL clearly more accurate and more efficient than ORMs, which makes automated dense supervision practical.

## Key points
- **Progress as reward:** A step's process reward is the change in the prover's success probability before and after the step. Using the base policy's own advantages adds nothing, because it produces the same policy-gradient updates as outcome-only reward.
- **Prover choice:** Best-of-K versions of the base policy with a moderate K > 1 work well as provers. A prover that succeeds from almost any step gives every step roughly the same score and cannot tell them apart.
- **Test-time search:** Beam search guided by PAVs is more than 8% more accurate than ORM re-ranking and 1.5–5× more compute-efficient.
- **Online RL:** Using PAVs as dense rewards gives 5–6× better sample efficiency and more than 6% higher accuracy than ORM-only rewards. It also improves Pass@N and solves hard problems that the SFT policy cannot solve even with N = 256 samples.
- **Setup:** Experiments use Gemma2 2B, 9B and 27B. Training data comes from "seed" traces sampled from the prover plus partial rollouts from their prefixes, which give Q-value estimates.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 492 works · 440,463 citations
- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 598 works · 191,226 citations
- [Google DeepMind](https://md.research.elamur.ai/free/en/companies/google-deepmind/) — Company, London, United Kingdom · 72 works · 84,326 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant and Aviral Kumar. 2024. Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning. arXiv:2410.08146.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{setlur2024rewarding,
  title = {{Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning}},
  author = {Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant and Aviral Kumar},
  year = {2024},
  doi = {10.48550/arxiv.2410.08146},
  eprint = {2410.08146},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.08146}
}
```

- [← previous PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competition](https://md.research.elamur.ai/free/en/papers/tsoukalas-2024-putnambench-evaluating-neural-theorem-provers/)
- [next → Exploring the Potential of ChatGPT on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse Relations](https://md.research.elamur.ai/free/en/papers/chan-2024-exploring-potential-chatgpt-sentence-level/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

46 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Agarwal, S. M. Kakade, J. D. Lee, and G. Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1–76, 2021. [find](https://scholar.google.com/scholar?q=A%2E+Agarwal%2C+S%2E+M%2E+Kakade%2C+J%2E+D%2E+Lee%2C+and+G%2E+Mahajan%2E+On+the+theory+of+policy+gradient+methods%3A+Optimality%2C+approximation%2C+and+distribution+shift%2E+Journal+of+Machine+Learning+Research%2C+22%2898%29%3A1%E2%80%9376%2C+202)
2. T. Anthony, Z. Tian, and D. Barber. [Thinking fast and slow with deep learning and tree search](https://md.research.elamur.ai/free/en/papers/anthony-2017-thinking-fast-slow-deep-learning/). Advances in neural information processing systems, 30, 2017.
3. M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pages 1471–1479, 2016. [find](https://scholar.google.com/scholar?q=M%2E+Bellemare%2C+S%2E+Srinivasan%2C+G%2E+Ostrovski%2C+T%2E+Schaul%2C+D%2E+Saxton%2C+and+R%2E+Munos%2E+Unifying+count%2Dbased+exploration+and+intrinsic+motivation%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+14)
4. X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fu, et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. [find](https://scholar.google.com/scholar?q=X%2E+Bi%2C+D%2E+Chen%2C+G%2E+Chen%2C+S%2E+Chen%2C+D%2E+Dai%2C+C%2E+Deng%2C+H%2E+Ding%2C+K%2E+Dong%2C+Q%2E+Du%2C+Z%2E+Fu%2C+et+al%2E+Deepseek+llm%3A+Scaling+open%2Dsource+language+models+with+longtermism%2E+arXiv+preprint+arXiv%3A2401%2E02954%2C+2024%2E)
5. J. D. Chang, K. Brantley, R. Ramamurthy, D. Misra, and W. Sun. Learning to generate better than your llm. arXiv preprint arXiv:2306.11816, 2023. — [arXiv:2306.11816](https://arxiv.org/abs/2306.11816) [find](https://scholar.google.com/scholar?q=J%2E+D%2E+Chang%2C+K%2E+Brantley%2C+R%2E+Ramamurthy%2C+D%2E+Misra%2C+and+W%2E+Sun%2E+Learning+to+generate+better+than+your+llm%2E+arXiv+preprint+arXiv%3A2306%2E11816%2C+2023%2E+%E2%80%94+%5BarXiv%3A2306%2E11816%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2306%2E11816%29)
6. K.-W. Chang, A. Krishnamurthy, A. Agarwal, H. Daumé III, and J. Langford. Learning to search better than your teacher. In International Conference on Machine Learning, pages 2058–2066. PMLR, 2015. [find](https://scholar.google.com/scholar?q=K%2E%2DW%2E+Chang%2C+A%2E+Krishnamurthy%2C+A%2E+Agarwal%2C+H%2E+Daum%C3%A9+III%2C+and+J%2E+Langford%2E+Learning+to+search+better+than+your+teacher%2E+In+International+Conference+on+Machine+Learning%2C+pages+2058%E2%80%932066%2E+PMLR%2C+2015%2E)
7. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021a.
8. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021b.
9. M. Collins. [Discriminative reranking for natural language parsing](https://md.research.elamur.ai/free/en/papers/collins-2000-discriminative-reranking-natural-language-parsing/). In Proceedings of the International Conference on Machine Learning, 2000.
10. Gemma Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+T%2E+Mesnard%2C+C%2E+Hardin%2C+R%2E+Dadashi%2C+S%2E+Bhupatiraju%2C+S%2E+Pathak%2C+L%2E+Sifre%2C+M%2E+Rivi%C3%A8re%2C+M%2E+S%2E+Kale%2C+J%2E+Love%2C+et+al%2E+Gemma%3A+Open+models+based+on+gemini+research+and+technology%2E+arXiv+preprint+a)
11. M. Germain, K. Gregor, I. Murray, and H. Larochelle. Made: Masked autoencoder for distribution estimation. In International conference on machine learning, pages 881–889. PMLR, 2015. [find](https://scholar.google.com/scholar?q=M%2E+Germain%2C+K%2E+Gregor%2C+I%2E+Murray%2C+and+H%2E+Larochelle%2E+Made%3A+Masked+autoencoder+for+distribution+estimation%2E+In+International+conference+on+machine+learning%2C+pages+881%E2%80%93889%2E+PMLR%2C+2015%2E)
12. A. Havrilla, Y. Du, S. C. Raparthy, C. Nalmpantis, J. Dwivedi-Yu, M. Zhuravinskyi, E. Hambro, S. Sukhbaatar, and R. Raileanu. [Teaching large language models to reason with reinforcement learning](https://md.research.elamur.ai/free/en/papers/havrilla-2024-teaching-large-language-models-reason/). arXiv preprint arXiv:2403.04642, 2024.
13. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). NeurIPS, 2021.
14. G. Hinton. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). arXiv preprint arXiv:1503.02531, 2015.
15. A. Hosseini, X. Yuan, N. Malkin, A. Courville, A. Sordoni, and R. Agarwal. [V-star: Training verifiers for self-taught reasoners](https://md.research.elamur.ai/free/en/papers/hosseini-2024-v-star-training-verifiers-self/). arXiv preprint arXiv:2402.06457, 2024.
16. H. Hwang, D. Kim, S. Kim, S. Ye, and M. Seo. Self-explore to avoid the pit: Improving the reasoning capabilities of language models with fine-grained rewards. arXiv preprint arXiv:2404.10346, 2024. — [arXiv:2404.10346](https://arxiv.org/abs/2404.10346) [find](https://scholar.google.com/scholar?q=H%2E+Hwang%2C+D%2E+Kim%2C+S%2E+Kim%2C+S%2E+Ye%2C+and+M%2E+Seo%2E+Self%2Dexplore+to+avoid+the+pit%3A+Improving+the+reasoning+capabilities+of+language+models+with+fine%2Dgrained+rewards%2E+arXiv+preprint+arXiv%3A2404%2E10346%2C+2024%2E+%E2%80%94+)
17. S. Kakade and J. Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pages 267–274, 2002. [find](https://scholar.google.com/scholar?q=S%2E+Kakade+and+J%2E+Langford%2E+Approximately+optimal+approximate+reinforcement+learning%2E+In+Proceedings+of+the+Nineteenth+International+Conference+on+Machine+Learning%2C+pages+267%E2%80%93274%2C+2002%2E)
18. S. M. Kakade. A natural policy gradient. In Advances in neural information processing systems, volume 14. Advances in neural information processing systems, 2001a. [find](https://scholar.google.com/scholar?q=S%2E+M%2E+Kakade%2E+A+natural+policy+gradient%2E+In+Advances+in+neural+information+processing+systems%2C+volume+14%2E+Advances+in+neural+information+processing+systems%2C+2001a%2E)
19. S. M. Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001b. [find](https://scholar.google.com/scholar?q=S%2E+M%2E+Kakade%2E+A+natural+policy+gradient%2E+Advances+in+neural+information+processing+systems%2C+14%2C+2001b%2E)
20. A. Kazemnejad, M. Aghajohari, E. Portelance, A. Sordoni, S. Reddy, A. Courville, and N. L. Roux. Vineppo: Unlocking rl potential for llm reasoning through refined credit assignment. arXiv preprint arXiv:2410.[01679, 2024](https://md.research.elamur.ai/free/en/papers/kazemnejad-2024-vineppo-refining-credit-assignment-rl/).
21. H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.[20050, 2023](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/).
22. L. Luo, Y. Liu, R. Liu, S. Phatale, H. Lara, Y. Li, L. Shu, Y. Zhu, L. Meng, J. Sun, et al. [Improve mathematical reasoning in language models by automated process supervision](https://md.research.elamur.ai/free/en/papers/luo-2024-improve-mathematical-reasoning-language-models/). arXiv preprint arXiv:2406.06592, 2024.
23. Q. Ma, H. Zhou, T. Liu, J. Yuan, P. Liu, Y. You, and H. Yang. Let’s reward step by step: Step-level reward model as the navigators for reasoning. arXiv preprint arXiv:2310.10080, 2023. [find](https://scholar.google.com/scholar?q=Q%2E+Ma%2C+H%2E+Zhou%2C+T%2E+Liu%2C+J%2E+Yuan%2C+P%2E+Liu%2C+Y%2E+You%2C+and+H%2E+Yang%2E+Let%E2%80%99s+reward+step+by+step%3A+Step%2Dlevel+reward+model+as+the+navigators+for+reasoning%2E+arXiv+preprint+arXiv%3A2310%2E10080%2C+2023%2E)
24. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al. [Webgpt: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/). arXiv preprint arXiv:2112.09332, 2021.
25. A. Y. Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, volume 99, pages 278–287, 1999. [find](https://scholar.google.com/scholar?q=A%2E+Y%2E+Ng%2C+D%2E+Harada%2C+and+S%2E+Russell%2E+Policy+invariance+under+reward+transformations%3A+Theory+and+application+to+reward+shaping%2E+In+ICML%2C+volume+99%2C+pages+278%E2%80%93287%2C+1999%2E)
26. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
27. R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). arXiv preprint arXiv:2305.18290, 2023.
28. S. Ross and J. A. Bagnell. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:1406.5979, 2014. [find](https://scholar.google.com/scholar?q=S%2E+Ross+and+J%2E+A%2E+Bagnell%2E+Reinforcement+and+imitation+learning+via+interactive+no%2Dregret+learning%2E+arXiv+preprint+arXiv%3A1406%2E5979%2C+2014%2E)
29. A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, and R. Hadsell. Policy distillation. arXiv preprint arXiv:1511.06295, 2015. [find](https://scholar.google.com/scholar?q=A%2E+A%2E+Rusu%2C+S%2E+G%2E+Colmenarejo%2C+C%2E+Gulcehre%2C+G%2E+Desjardins%2C+J%2E+Kirkpatrick%2C+R%2E+Pascanu%2C+V%2E+Mnih%2C+K%2E+Kavukcuoglu%2C+and+R%2E+Hadsell%2E+Policy+distillation%2E+arXiv+preprint+arXiv%3A1511%2E06295%2C+2015%2E)
30. A. Setlur, S. Garg, X. Geng, N. Garg, V. Smith, and A. Kumar. Rl on incorrect synthetic data scales the efficiency of llm math reasoning by eight-fold. arXiv preprint arXiv:2406.14532, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Setlur%2C+S%2E+Garg%2C+X%2E+Geng%2C+N%2E+Garg%2C+V%2E+Smith%2C+and+A%2E+Kumar%2E+Rl+on+incorrect+synthetic+data+scales+the+efficiency+of+llm+math+reasoning+by+eight%2Dfold%2E+arXiv+preprint+arXiv%3A2406%2E14532%2C+2024%2E)
31. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. Li, Y. Wu, and D. Guo. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300, 2024.
32. A. Singh, J. D. Co-Reyes, R. Agarwal, A. Anand, P. Patil, P. J. Liu, J. Harrison, J. Lee, K. Xu, A. Parisi, et al. [Beyond human data: Scaling self-training for problem-solving with language models](https://md.research.elamur.ai/free/en/papers/singh-2023-human-data-scaling-self-training/). arXiv preprint arXiv:2312.06585, 2023a.
33. I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Prog-prompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11523–11530. [find](https://scholar.google.com/scholar?q=I%2E+Singh%2C+V%2E+Blukis%2C+A%2E+Mousavian%2C+A%2E+Goyal%2C+D%2E+Xu%2C+J%2E+Tremblay%2C+D%2E+Fox%2C+J%2E+Thomason%2C+and+A%2E+Garg%2E+Prog%2Dprompt%3A+Generating+situated+robot+task+plans+using+large+language+models%2E+In+2023+IEEE+Internati)
34. C. Snell, J. Lee, K. Xu, and A. Kumar. [Scaling llm test-time compute optimally can be more effective than scaling model parameters](https://md.research.elamur.ai/free/en/papers/snell-2024-scaling-llm-test-time-compute/). arXiv preprint arXiv:2408.03314, 2024.
35. W. Sun, A. Venkatraman, G. J. Gordon, B. Boots, and J. A. Bagnell. Deeply aggrevated: Differentiable imitation learning for sequential prediction. In International conference on machine learning, pages 3309–3318. PMLR, 2017. [find](https://scholar.google.com/scholar?q=W%2E+Sun%2C+A%2E+Venkatraman%2C+G%2E+J%2E+Gordon%2C+B%2E+Boots%2C+and+J%2E+A%2E+Bagnell%2E+Deeply+aggrevated%3A+Differentiable+imitation+learning+for+sequential+prediction%2E+In+International+conference+on+machine+learning%2C+page)
36. R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. The MIT Press, second edition, 2018. [find](https://scholar.google.com/scholar?q=R%2E+S%2E+Sutton+and+A%2E+G%2E+Barto%2E+Reinforcement+learning%3A+An+introduction%2E+The+MIT+Press%2C+second+edition%2C+2018%2E)
37. R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999. [find](https://scholar.google.com/scholar?q=R%2E+S%2E+Sutton%2C+D%2E+McAllester%2C+S%2E+Singh%2C+and+Y%2E+Mansour%2E+Policy+gradient+methods+for+reinforcement+learning+with+function+approximation%2E+Advances+in+neural+information+processing+systems%2C+12%2C+1999%2E)
38. F. Tajwar, A. Singh, A. Sharma, R. Rafailov, J. Schneider, T. Xie, S. Ermon, C. Finn, and A. Kumar. Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data, 2024. [find](https://scholar.google.com/scholar?q=F%2E+Tajwar%2C+A%2E+Singh%2C+A%2E+Sharma%2C+R%2E+Rafailov%2C+J%2E+Schneider%2C+T%2E+Xie%2C+S%2E+Ermon%2C+C%2E+Finn%2C+and+A%2E+Kumar%2E+Preference+Fine%2DTuning+of+LLMs+Should+Leverage+Suboptimal%2C+On%2DPolicy+Data%2C+2024%2E)
39. J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.[14275, 2022](https://md.research.elamur.ai/free/en/papers/uesato-2022-solving-math-word-problems-process/).
40. P. Wang, L. Li, Z. Shao, R. X. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui. [Math-shepherd: Verify and reinforce llms step-by-step without human annotations](https://md.research.elamur.ai/free/en/papers/wang-2024-math-shepherd-verify-reinforce-llms/), 2024.
41. R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256, 1992. [find](https://scholar.google.com/scholar?q=R%2E+J%2E+Williams%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Machine+learning%2C+8%283%2D4%29%3A229%E2%80%93256%2C+1992%2E)
42. Y. Wu, Z. Sun, S. Li, S. Welleck, and Y. Yang. An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.[00724, 2024](https://md.research.elamur.ai/free/en/papers/wu-2024-inference-scaling-laws-empirical-analysis/).
43. F. Yu, A. Gao, and B. Wang. Outcome-supervised verifiers for planning in mathematical reasoning. arXiv preprint arXiv:2311.[09724, 2023](https://md.research.elamur.ai/free/en/papers/yu-2023-ovm-outcome-supervised-value-models/).
44. Z. Yuan, H. Yuan, C. Li, G. Dong, C. Tan, and C. Zhou. [Scaling relationship on learning mathematical reasoning with large language models](https://md.research.elamur.ai/free/en/papers/yuan-2023-scaling-relationship-learning-mathematical-reasoning/). arXiv preprint arXiv:2308.01825, 2023.
45. E. Zelikman, Y. Wu, J. Mu, and N. Goodman. [Star: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). Advances in Neural Information Processing Systems, 35:15476–15488, 2022.
46. L. Zhang, A. Hosseini, H. Bansal, M. Kazemi, A. Kumar, and R. Agarwal. [Generative verifiers: Reward modeling as next-token prediction](https://md.research.elamur.ai/free/en/papers/zhang-2024-generative-verifiers-reward-modeling-as/). arXiv preprint arXiv:2408.15240, 2024.

*Updated 2 November 2026*