--- title: "Let's Verify Step by Step — Lightman 2023" description: "Language models that reason in a chain of thought still make logical slips, and one bad step can ruin a whole solution." authors: ["Hunter Lightman", "Vineet Kosaraju", "Yura Burda", "Harri Edwards", "Bowen Baker", "Teddy Lee", "Jan Leike", "John Schulman", "Ilya Sutskever", "Karl Cobbe"] year: 2023 venue: "arXiv:2305.20050" source: "https://arxiv.org/abs/2305.20050" doi: "10.48550/arxiv.2305.20050" arxiv: "2305.20050" licence: "arXiv — non-exclusive licence to distribute" licence_url: "https://arxiv.org/licenses/nonexclusive-distrib/1.0/license.html" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) / Let's Verify Step by Step # Let's Verify Step by Step **Authors:** Hunter Lightman, [Vineet Kosaraju](https://md.research.elamur.ai/free/en/authors/vineet-kosaraju/), Yura Burda, [Harri Edwards](https://md.research.elamur.ai/free/en/authors/harri-edwards/), [Bowen Baker](https://md.research.elamur.ai/free/en/authors/bowen-baker/), [Teddy Lee](https://md.research.elamur.ai/free/en/authors/teddy-lee/), [Jan Leike](https://md.research.elamur.ai/free/en/authors/jan-leike/), [John Schulman](https://md.research.elamur.ai/free/en/authors/john-schulman/), [Ilya Sutskever](https://md.research.elamur.ai/free/en/authors/ilya-sutskever/), [Karl Cobbe](https://md.research.elamur.ai/free/en/authors/karl-cobbe/) · **Year:** 2023 · **Venue:** arXiv:2305.20050 · **Licence:** [arXiv — non-exclusive licence to distribute](https://arxiv.org/licenses/nonexclusive-distrib/1.0/license.html) **Links:** [Publisher's page](https://arxiv.org/abs/2305.20050) · [PDF](https://arxiv.org/pdf/2305.20050v1) · [DOI 10.48550/arxiv.2305.20050](https://doi.org/10.48550/arxiv.2305.20050) · [arXiv 2305.20050](https://arxiv.org/abs/2305.20050) · [OpenAlex](https://openalex.org/W4379089709) **Citations:** 31 citations (openalex-snapshot · 2026-10-14) · [1328](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Language models that reason in a chain of thought still make logical slips, and one bad step can ruin a whole solution. The paper asks which kind of feedback produces a more trustworthy reward model for catching these errors. One option is outcome supervision (ORM), which judges only the final answer. The other is process supervision (PRM), which judges every intermediate step. The OpenAI team took GPT-4 models, further trained them on the MathMix math corpus, and had human annotators label each step of generated MATH solutions as positive, negative or neutral. Each reward model then picked the best answer from many sampled solutions. To get more value from labelling, annotators were mostly shown solutions that the current PRM rated highly even though their final answer was wrong. This is a form of active learning. The step-level verifier clearly outperformed the outcome-based one, solving 78.2% of a held-out MATH subset. Controlled smaller-scale experiments showed the same trend. The released PRM800K dataset makes research on step-level feedback much more accessible. The results also suggest that rewarding sound reasoning, not just correct answers, helps both accuracy and alignment. ## Key points - **PRM800K**: about 800K human step-level labels covering 75K model-generated solutions to 12K MATH problems. Evaluation uses 500 MATH test problems kept out of training. - **Best-of-N evaluation**: the PRM scores a solution as the product of its per-step correctness probabilities. The best GPT-4-based PRM reaches 78.2% on the MATH subset, clearly ahead of the ORM. - **Active learning**: showing annotators "convincing wrong-answer" solutions made process supervision about 2.6× more data-efficient. - **Cheaper ablations**: small models, pretrained with about 200× less compute, were supervised by the large PRM instead of by humans. This allowed a fair comparison of ORM and PRM without new annotation. - **Relation to Uesato et al. (2022)**: that study found the two methods roughly equal on GSM8K. This paper argues the gap appears only with larger amounts of process feedback and on harder problems. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/) ## Cited by Cited by 1328 papers of our library cite it (of the 140673 whose bibliographies we read); 61 of them have a page here, 1267 not yet. - [Many-Shot In-Context Learning](https://md.research.elamur.ai/free/en/papers/agarwal-2024-many-shot-context-learning/) - [Large Language Models for Mathematical Reasoning: Progresses and Challenges](https://md.research.elamur.ai/free/en/papers/ahn-2024-large-language-models-mathematical-reasoning/) - [Llemma: An Open Language Model For Mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/) - [Qwen Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/) - [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://md.research.elamur.ai/free/en/papers/brown-2024-large-language-monkeys-scaling-inference/) - [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://md.research.elamur.ai/free/en/papers/burns-2023-weak-strong-generalization-eliciting-strong/) - [InternLM2 Technical Report](https://md.research.elamur.ai/free/en/papers/cai-2024-internlm2-technical-report/) - [AlphaMath Almost Zero: Process Supervision without Process](https://md.research.elamur.ai/free/en/papers/chen-2024-alphamath-almost-zero-process-supervision/) - [Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs](https://md.research.elamur.ai/free/en/papers/chen-2024-think-that-much-2-3/) - [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2025-towards-reasoning-era-survey-long/) - [FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios](https://md.research.elamur.ai/free/en/papers/chern-2023-factool-factuality-detection-generative-ai/) - [Process Reinforcement through Implicit Rewards](https://md.research.elamur.ai/free/en/papers/cui-2025-process-reinforcement-through-implicit-rewards/) - [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deepseek-2025-ai-deepseek-r1-incentivizing-reasoning-capability/) - [JudgeMeNot: Personalizing Large Language Models to Emulate Judicial Reasoning in Hebrew](https://md.research.elamur.ai/free/en/papers/findings-2026-razumenko-judgemenot-personalizing-large-language-models/) - [Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs](https://md.research.elamur.ai/free/en/papers/gandhi-2025-cognitive-behaviors-that-enable-self/) - [ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving](https://md.research.elamur.ai/free/en/papers/gou-2023-tora-tool-integrated-reasoning-agent/) - [rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking](https://md.research.elamur.ai/free/en/papers/guan-2025-rstar-math-small-llms-can/) - [V-STaR: Training Verifiers for Self-Taught Reasoners](https://md.research.elamur.ai/free/en/papers/hosseini-2024-v-star-training-verifiers-self/) - [OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework](https://md.research.elamur.ai/free/en/papers/hu-2024-openrlhf-easy-use-scalable/) - [Large Language Models Can Self-Improve](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/) - [Large Language Models Cannot Self-Correct Reasoning Yet](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-cannot-self/) - [Survey of Hallucination in Natural Language Generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/) - [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://md.research.elamur.ai/free/en/papers/kimi-2025-team-kimi-k1-5-scaling-reinforcement/) - [Training Language Models to Self-Correct via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/kumar-2024-training-language-models-self-correct/) - [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://md.research.elamur.ai/free/en/papers/lambert-2024-tulu-3-pushing-frontiers-open/) - [Generative Judge for Evaluating Alignment](https://md.research.elamur.ai/free/en/papers/li-2023-generative-judge-evaluating-alignment/) - [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://md.research.elamur.ai/free/en/papers/li-2024-llms-judges-comprehensive-survey-llm/) - [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://md.research.elamur.ai/free/en/papers/li-2025-search-o1-agentic-search-enhanced/) - [大语言模型的幻觉问题研究综述](https://md.research.elamur.ai/free/en/papers/liu-2024-hallucinations-large-language/) - [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/) - [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/) - [Improve Mathematical Reasoning in Language Models by Automated Process Supervision](https://md.research.elamur.ai/free/en/papers/luo-2024-improve-mathematical-reasoning-language-models/) - [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://md.research.elamur.ai/free/en/papers/microsoft-2025-phi-4-mini-technical-report/) - [s1: Simple test-time scaling](https://md.research.elamur.ai/free/en/papers/muennighoff-2025-s1-simple-test-time-scaling/) - [OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text](https://md.research.elamur.ai/free/en/papers/paster-2023-openwebmath-open-dataset-high-quality/) - [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/) - [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://md.research.elamur.ai/free/en/papers/snell-2024-scaling-llm-test-time-compute/) - [To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning](https://md.research.elamur.ai/free/en/papers/sprague-2024-cot-cot-chain-thought-helps/) - [ReFT: Reasoning with Reinforced Fine-Tuning](https://md.research.elamur.ai/free/en/papers/trung-2024-reft-reasoning-reinforced-fine-tuning/) - [Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts](https://md.research.elamur.ai/free/en/papers/wang-2024-interpretable-preferences-multi-objective-reward/) - [Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations](https://md.research.elamur.ai/free/en/papers/wang-2024-math-shepherd-verify-reinforce-llms/) - [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://md.research.elamur.ai/free/en/papers/wang-2025-internvl3-5-advancing-open/) - [Large Language Models are Better Reasoners with Self-Verification](https://md.research.elamur.ai/free/en/papers/weng-2023-large-language-models-are-better/) - [Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models](https://md.research.elamur.ai/free/en/papers/wu-2024-inference-scaling-laws-empirical-analysis/) - [The Rise and Potential of Large Language Model Based Agents: A Survey](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/) - [Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/xie-2025-logic-rl-unleashing-llm/) - [How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark](https://md.research.elamur.ai/free/en/papers/yang-2025-llm-reasoning-distracted-irrelevant-context/) - [Qwen3 Technical Report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/) - [Demystifying Long Chain-of-Thought Reasoning in LLMs](https://md.research.elamur.ai/free/en/papers/yeo-2025-demystifying-long-chain-thought-reasoning/) - [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/) - [Scaling Relationship on Learning Mathematical Reasoning with Large Language Models](https://md.research.elamur.ai/free/en/papers/yuan-2023-scaling-relationship-learning-mathematical-reasoning/) - [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://md.research.elamur.ai/free/en/papers/zeng-2025-simplerl-zoo-investigating-taming-zero/) - [Instruction Tuning for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/zhang-2023-instruction-tuning-large-language-models/) - [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2023-siren-s-song-ai-ocean/) - [Generative Verifiers: Reward Modeling as Next-Token Prediction](https://md.research.elamur.ai/free/en/papers/zhang-2024-generative-verifiers-reward-modeling-as/) - [ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search](https://md.research.elamur.ai/free/en/papers/zhang-2024-rest-mcts-llm-self-training/) - [The Lessons of Developing Process Reward Models in Mathematical Reasoning](https://md.research.elamur.ai/free/en/papers/zhang-2025-lessons-developing-process-reward-models/) - [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/) - [ProcessBench: Identifying Process Errors in Mathematical Reasoning](https://md.research.elamur.ai/free/en/papers/zheng-2025-processbench-identifying-process-errors-mathematical/) - [Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification](https://md.research.elamur.ai/free/en/papers/zhou-2023-solving-challenging-math-word-problems/) - [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://md.research.elamur.ai/free/en/papers/zhu-2025-internvl3-exploring-advanced-training/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{lightman2023let, title = {{Let's Verify Step by Step}}, author = {Lightman, Hunter and Kosaraju, Vineet and Burda, Yura and Edwards, Harri and Baker, Bowen and Lee, Teddy and Leike, Jan and Schulman, John and Sutskever, Ilya and Cobbe, Karl}, year = {2023}, doi = {10.48550/arxiv.2305.20050}, eprint = {2305.20050}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2305.20050} } ``` - [← previous LENS: A Learnable Evaluation Metric for Text Simplification](https://md.research.elamur.ai/free/en/papers/maddela-2023-lens-learnable-evaluation-metric-text/) - [next → LEVER: Learning to Verify Language-to-Code Generation with Execution](https://md.research.elamur.ai/free/en/papers/ni-2023-lever-learning-verify-language-code/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) ## References 26 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. [find](https://scholar.google.com/scholar?q=A%2E+Askell%2C+Y%2E+Bai%2C+A%2E+Chen%2C+D%2E+Drain%2C+D%2E+Ganguli%2C+T%2E+Henighan%2C+A%2E+Jones%2C+N%2E+Joseph%2C+B%2E+Mann%2C+N%2E+DasSarma%2C+et+al%2E+A+general+language+assistant+as+a+laboratory+for+alignment%2E+arXiv+preprint+arXiv%3A2112%2E0) 2. S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Bubeck%2C+V%2E+Chandrasekaran%2C+R%2E+Eldan%2C+J%2E+Gehrke%2C+E%2E+Horvitz%2C+E%2E+Kamar%2C+P%2E+Lee%2C+Y%2E+T%2E+Lee%2C+Y%2E+Li%2C+S%2E+Lundberg%2C+et+al%2E+Sparks+of+artificial+general+intelligence%3A+Early+experiments+with+gpt%2D4%2E+arXiv+pr) [our card](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/) 3. P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. [find](https://scholar.google.com/scholar?q=P%2E+F%2E+Christiano%2C+J%2E+Leike%2C+T%2E+Brown%2C+M%2E+Martic%2C+S%2E+Legg%2C+and+D%2E+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+neural+information+processing+systems%2C+30%2C+2017%2E) 4. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [find](https://scholar.google.com/scholar?q=K%2E+Cobbe%2C+V%2E+Kosaraju%2C+M%2E+Bavarian%2C+M%2E+Chen%2C+H%2E+Jun%2C+L%2E+Kaiser%2C+M%2E+Plappert%2C+J%2E+Tworek%2C+J%2E+Hilton%2C+R%2E+Nakano%2C+et+al%2E+Training+verifiers+to+solve+math+word+problems%2E+arXiv+preprint+arXiv%3A2110%2E14168%2C+20) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/) 5. A. Cotra. Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover. [https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to](https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to), 2022. [↗](https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to) 6. A. Creswell, M. Shanahan, and I. Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712, 2022. [find](https://scholar.google.com/scholar?q=A%2E+Creswell%2C+M%2E+Shanahan%2C+and+I%2E+Higgins%2E+Selection%2Dinference%3A+Exploiting+large+language+models+for+interpretable+logical+reasoning%2E+arXiv+preprint+arXiv%3A2205%2E09712%2C+2022%2E) 7. T. Everitt, V. Krakovna, L. Orseau, M. Hutter, and S. Legg. Reinforcement learning with a corrupted reward channel. arXiv preprint arXiv:1705.08417, 2017. — [arXiv:1705.08417](https://arxiv.org/abs/1705.08417) [find](https://scholar.google.com/scholar?q=T%2E+Everitt%2C+V%2E+Krakovna%2C+L%2E+Orseau%2C+M%2E+Hutter%2C+and+S%2E+Legg%2E+Reinforcement+learning+with+a+corrupted+reward+channel%2E+arXiv+preprint+arXiv%3A1705%2E08417%2C+2017%2E+%E2%80%94+%5BarXiv%3A1705%2E08417%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F17) 8. L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. arXiv preprint arXiv:2210.10760, 2022. [find](https://scholar.google.com/scholar?q=L%2E+Gao%2C+J%2E+Schulman%2C+and+J%2E+Hilton%2E+Scaling+laws+for+reward+model+overoptimization%2E+arXiv+preprint+arXiv%3A2210%2E10760%2C+2022%2E) 9. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. [find](https://scholar.google.com/scholar?q=D%2E+Hendrycks%2C+C%2E+Burns%2C+S%2E+Kadavath%2C+A%2E+Arora%2C+S%2E+Basart%2C+E%2E+Tang%2C+D%2E+Song%2C+and+J%2E+Steinhardt%2E+Measuring+mathematical+problem+solving+with+the+math+dataset%2E+arXiv+preprint+arXiv%3A2103%2E03874%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/) 10. T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022. [find](https://scholar.google.com/scholar?q=T%2E+Kojima%2C+S%2E+S%2E+Gu%2C+M%2E+Reid%2C+Y%2E+Matsuo%2C+and+Y%2E+Iwasawa%2E+Large+language+models+are+zero%2Dshot+reasoners%2E+arXiv+preprint+arXiv%3A2205%2E11916%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/) 11. A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858, 2022. [find](https://scholar.google.com/scholar?q=A%2E+Lewkowycz%2C+A%2E+Andreassen%2C+D%2E+Dohan%2C+E%2E+Dyer%2C+H%2E+Michalewski%2C+V%2E+Ramasesh%2C+A%2E+Slone%2C+C%2E+Anil%2C+I%2E+Schlag%2C+T%2E+Gutman%2DSolo%2C+et+al%2E+Solving+quantitative+reasoning+problems+with+language+models%2E+arXiv+pr) [our card](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/) 12. Y. Li, Z. Lin, S. Zhang, Q. Fu, B. Chen, J.-G. Lou, and W. Chen. On the advance of making language models better reasoners. arXiv preprint arXiv:2206.02336, 2022. — [arXiv:2206.02336](https://arxiv.org/abs/2206.02336) [find](https://scholar.google.com/scholar?q=Y%2E+Li%2C+Z%2E+Lin%2C+S%2E+Zhang%2C+Q%2E+Fu%2C+B%2E+Chen%2C+J%2E%2DG%2E+Lou%2C+and+W%2E+Chen%2E+On+the+advance+of+making+language+models+better+reasoners%2E+arXiv+preprint+arXiv%3A2206%2E02336%2C+2022%2E+%E2%80%94+%5BarXiv%3A2206%2E02336%5D%28https%3A%2F%2Farxiv%2Eor) 13. J. Maynez, S. Narayan, B. Bohnet, and R. McDonald. On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661, 2020. [find](https://scholar.google.com/scholar?q=J%2E+Maynez%2C+S%2E+Narayan%2C+B%2E+Bohnet%2C+and+R%2E+McDonald%2E+On+faithfulness+and+factuality+in+abstractive+summarization%2E+arXiv+preprint+arXiv%3A2005%2E00661%2C+2020%2E) 14. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021. [find](https://scholar.google.com/scholar?q=R%2E+Nakano%2C+J%2E+Hilton%2C+S%2E+Balaji%2C+J%2E+Wu%2C+L%2E+Ouyang%2C+C%2E+Kim%2C+C%2E+Hesse%2C+S%2E+Jain%2C+V%2E+Kosaraju%2C+W%2E+Saunders%2C+et+al%2E+Webgpt%3A+Browser%2Dassisted+question%2Danswering+with+human+feedback%2E+arXiv+preprint+arXiv%3A211) [our card](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/) 15. E. Nichols, L. Gao, and R. Gomez. Collaborative storytelling with large-scale neural language models. In Proceedings of the 13th ACM SIGGRAPH Conference on Motion, Interaction and Games, pages 1–10, 2020. [find](https://scholar.google.com/scholar?q=E%2E+Nichols%2C+L%2E+Gao%2C+and+R%2E+Gomez%2E+Collaborative+storytelling+with+large%2Dscale+neural+language+models%2E+In+Proceedings+of+the+13th+ACM+SIGGRAPH+Conference+on+Motion%2C+Interaction+and+Games%2C+pages+1%E2%80%9310%2C+2) 16. M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021. [find](https://scholar.google.com/scholar?q=M%2E+Nye%2C+A%2E+J%2E+Andreassen%2C+G%2E+Gur%2DAri%2C+H%2E+Michalewski%2C+J%2E+Austin%2C+D%2E+Bieber%2C+D%2E+Dohan%2C+A%2E+Lewkowycz%2C+M%2E+Bosma%2C+D%2E+Luan%2C+et+al%2E+Show+your+work%3A+Scratchpads+for+intermediate+computation+with+language+mod) [our card](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/) 17. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2E+arXiv+preprint+arXiv%3A2303%2E08774%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/) 18. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. [find](https://scholar.google.com/scholar?q=L%2E+Ouyang%2C+J%2E+Wu%2C+X%2E+Jiang%2C+D%2E+Almeida%2C+C%2E+L%2E+Wainwright%2C+P%2E+Mishkin%2C+C%2E+Zhang%2C+S%2E+Agarwal%2C+K%2E+Slama%2C+A%2E+Ray%2C+et+al%2E+Training+language+models+to+follow+instructions+with+human+feedback%2E+arXiv+preprint) 19. J. Shen, Y. Yin, L. Li, L. Shang, X. Jiang, M. Zhang, and Q. Liu. Generate & rank: A multi-task framework for math word problems. arXiv preprint arXiv:2109.03034, 2021. [find](https://scholar.google.com/scholar?q=J%2E+Shen%2C+Y%2E+Yin%2C+L%2E+Li%2C+L%2E+Shang%2C+X%2E+Jiang%2C+M%2E+Zhang%2C+and+Q%2E+Liu%2E+Generate+%26+rank%3A+A+multi%2Dtask+framework+for+math+word+problems%2E+arXiv+preprint+arXiv%3A2109%2E03034%2C+2021%2E) 20. N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. [find](https://scholar.google.com/scholar?q=N%2E+Stiennon%2C+L%2E+Ouyang%2C+J%2E+Wu%2C+D%2E+Ziegler%2C+R%2E+Lowe%2C+C%2E+Voss%2C+A%2E+Radford%2C+D%2E+Amodei%2C+and+P%2E+F%2E+Christiano%2E+Learning+to+summarize+with+human+feedback%2E+Advances+in+Neural+Information+Processing+Systems%2C+) 21. A. Stuhlmüller and J. Byun. Supervise process, not outcomes. [https://ought.org/updates/2022-04-06-process](https://ought.org/updates/2022-04-06-process), 2022. [↗](https://ought.org/updates/2022-04-06-process) 22. J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022. [find](https://scholar.google.com/scholar?q=J%2E+Uesato%2C+N%2E+Kushman%2C+R%2E+Kumar%2C+F%2E+Song%2C+N%2E+Siegel%2C+L%2E+Wang%2C+A%2E+Creswell%2C+G%2E+Irving%2C+and+I%2E+Higgins%2E+Solving+math+word+problems+with+process%2Dand+outcome%2Dbased+feedback%2E+arXiv+preprint+arXiv%3A2211%2E1427) [our card](https://md.research.elamur.ai/free/en/papers/uesato-2022-solving-math-word-problems-process/) 23. X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. [find](https://scholar.google.com/scholar?q=X%2E+Wang%2C+J%2E+Wei%2C+D%2E+Schuurmans%2C+Q%2E+Le%2C+E%2E+Chi%2C+and+D%2E+Zhou%2E+Self%2Dconsistency+improves+chain+of+thought+reasoning+in+language+models%2E+arXiv+preprint+arXiv%3A2203%2E11171%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/) 24. J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. [find](https://scholar.google.com/scholar?q=J%2E+Wei%2C+X%2E+Wang%2C+D%2E+Schuurmans%2C+M%2E+Bosma%2C+E%2E+Chi%2C+Q%2E+Le%2C+and+D%2E+Zhou%2E+Chain+of+thought+prompting+elicits+reasoning+in+large+language+models%2E+arXiv+preprint+arXiv%3A2201%2E11903%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/) 25. E. Zelikman, Y. Wu, J. Mu, and N. Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35: 15476–15488, 2022. [find](https://scholar.google.com/scholar?q=E%2E+Zelikman%2C+Y%2E+Wu%2C+J%2E+Mu%2C+and+N%2E+Goodman%2E+Star%3A+Bootstrapping+reasoning+with+reasoning%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A+15476%E2%80%9315488%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/) 26. D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. [find](https://scholar.google.com/scholar?q=D%2E+M%2E+Ziegler%2C+N%2E+Stiennon%2C+J%2E+Wu%2C+T%2E+B%2E+Brown%2C+A%2E+Radford%2C+D%2E+Amodei%2C+P%2E+Christiano%2C+and+G%2E+Irving%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint+arXiv%3A1909%2E08593%2C+2019%2E)