--- title: "Scaling LLM Test-Time Compute Optimally can be — Snell 2024" description: "The paper asks how much an LLM can improve on difficult prompts when it gets extra computation at inference time, and how that budget is best spent." authors: ["Charlie Snell", "Jaehoon Lee", "Kelvin Xu", "Aviral Kumar"] year: 2024 venue: "arXiv:2408.03314" source: "https://arxiv.org/abs/2408.03314" doi: "10.48550/arxiv.2408.03314" arxiv: "2408.03314" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/snell-2024-scaling-llm-test-time-compute/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/snell-2024-scaling-llm-test-time-compute/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters # Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters **Authors:** [Charlie Snell](https://md.research.elamur.ai/free/en/authors/charlie-snell/), [Jaehoon Lee](https://md.research.elamur.ai/free/en/authors/jaehoon-lee/), [Kelvin Xu](https://md.research.elamur.ai/free/en/authors/kelvin-xu/), [Aviral Kumar](https://md.research.elamur.ai/free/en/authors/aviral-kumar/) · **Year:** 2024 · **Venue:** arXiv:2408.03314 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2408.03314) · [PDF](https://arxiv.org/pdf/2408.03314) · [DOI 10.48550/arxiv.2408.03314](https://doi.org/10.48550/arxiv.2408.03314) · [arXiv 2408.03314](https://arxiv.org/abs/2408.03314) · [OpenAlex](https://openalex.org/W4406492615) **Citations:** 21 citations (openalex-snapshot · 2026-10-14) · [503](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary The paper asks how much an LLM can improve on difficult prompts when it gets extra computation at inference time, and how that budget is best spent. The authors use the MATH benchmark and PaLM 2 models fine-tuned for two specific capabilities. They study two ways to spend the extra compute. The first is search guided by a process-based reward model (PRM), which scores each individual solution step. The second is having the model revise its own earlier answers one after another. Which approach works best depends on how hard a question is for the base model. Revisions help most on easier questions, while parallel sampling or tree search pays off on harder ones. Their "compute-optimal" policy picks the approach for each prompt based on estimated difficulty, and it matches best-of-N with roughly four times less compute. With total FLOPs held equal, a small model given extra inference compute beats a ~14× larger model on easy and medium questions. On the hardest questions, extra pretraining still wins. These results help decide how to split compute between training and inference. ## Key points - **Setup:** PaLM 2-S* was fine-tuned separately as a revision model and as a PRM verifier, and evaluated on MATH. The authors fine-tuned because simply prompting off-the-shelf models does not produce useful revisions. Questions were grouped into five difficulty bins based on how often the base model solves them. - **Efficiency gain:** Choosing the strategy per prompt makes test-time scaling 2–4× more efficient than a best-of-N baseline, for both revisions and PRM search. - **Test-time vs. pretraining:** The FLOPs-matched comparison against a ~14× larger model used three ratios of inference tokens to pretraining tokens (R = 0.16, 0.79, 22). Extra test-time compute wins at low R and on difficulty bins 1–3. Extra pretraining wins at high R and on bins 4–5. - **Limitations:** Estimating question difficulty itself costs a lot of inference compute. Gains on the hardest problems stay small, and PRM tree search was not tried together with revisions. - **Future direction:** The authors suggest distilling the outputs of test-time compute back into the base model, creating a self-improvement loop. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/) ## Cited by Cited by 503 papers of our library cite it (of the 140673 whose bibliographies we read); 21 of them have a page here, 482 not yet. - [Fanar: An Arabic-Centric Multimodal Generative AI Platform](https://md.research.elamur.ai/free/en/papers/abbas-2025-fanar-arabic-centric-multimodal-generative/) - [L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/aggarwal-2025-l1-controlling-long-reasoning-model/) - [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://md.research.elamur.ai/free/en/papers/chen-2024-expanding-performance-boundaries-open/) - [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2025-towards-reasoning-era-survey-long/) - [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training](https://md.research.elamur.ai/free/en/papers/chu-2025-sft-memorizes-rl-generalizes-comparative/) - [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deepseek-2025-ai-deepseek-r1-incentivizing-reasoning-capability/) - [Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs](https://md.research.elamur.ai/free/en/papers/gandhi-2025-cognitive-behaviors-that-enable-self/) - [rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking](https://md.research.elamur.ai/free/en/papers/guan-2025-rstar-math-small-llms-can/) - [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://md.research.elamur.ai/free/en/papers/kimi-2025-team-kimi-k1-5-scaling-reinforcement/) - [Training Language Models to Self-Correct via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/kumar-2024-training-language-models-self-correct/) - [METAL: A Multi-Agent Framework for Chart Generation with Test-Time Scaling](https://md.research.elamur.ai/free/en/papers/li-2025-metal-multi-agent-framework-chart/) - [s1: Simple test-time scaling](https://md.research.elamur.ai/free/en/papers/muennighoff-2025-s1-simple-test-time-scaling/) - [Beyond Simple Term Injection: Reasoning Models for Legal Translation in a Non-Dominant Language Variety](https://md.research.elamur.ai/free/en/papers/natale-2026-beyond-simple-term-injection-reasoning/) - [mSCoRe: A Multilingual and Scalable Benchmark for Skill-based Commonsense Reasoning](https://md.research.elamur.ai/free/en/papers/ngo-2026-mscore-multilingual-scalable-benchmark-skill/) - [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://md.research.elamur.ai/free/en/papers/sui-2025-stop-overthinking-survey-efficient-reasoning/) - [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://md.research.elamur.ai/free/en/papers/wang-2025-internvl3-5-advancing-open/) - [NegNLI-BR: A Brazilian Portuguese Benchmark for Negation in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/westhelle-2026-negnli-br-brazilian-portuguese-benchmark/) - [LIMO: Less is More for Reasoning](https://md.research.elamur.ai/free/en/papers/ye-2025-limo-less-more-reasoning/) - [Demystifying Long Chain-of-Thought Reasoning in LLMs](https://md.research.elamur.ai/free/en/papers/yeo-2025-demystifying-long-chain-thought-reasoning/) - [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/) - [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://md.research.elamur.ai/free/en/papers/zhu-2025-internvl3-exploring-advanced-training/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{snell2024scaling, title = {{Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters}}, author = {Snell, Charlie and Lee, Jaehoon and Xu, Kelvin and Kumar, Aviral}, year = {2024}, doi = {10.48550/arxiv.2408.03314}, eprint = {2408.03314}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2408.03314} } ``` - [← previous SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corpora](https://md.research.elamur.ai/free/en/papers/qarah-2024-saudibert-large-language-model-pretrained/) - [next → Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://md.research.elamur.ai/free/en/papers/ge-2024-scaling-synthetic-data-creation-1/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) ## References 51 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Training revision models with synthetic data. Coming soon, 2024. [find](https://scholar.google.com/scholar?q=Training+revision+models+with+synthetic+data%2E+Coming+soon%2C+2024%2E) 2. C. Andrieu, N. De Freitas, A. Doucet, and M. I. Jordan. An introduction to mcmc for machine learning. 2003. [find](https://scholar.google.com/scholar?q=C%2E+Andrieu%2C+N%2E+De+Freitas%2C+A%2E+Doucet%2C+and+M%2E+I%2E+Jordan%2E+An+introduction+to+mcmc+for+machine+learning%2E+2003%2E) 3. R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, E. Chu, J. H. Clark, L. E. Shafey, Y. Huang, K. Meier-Hellstern, G. Mishra, E. Moreira, M. Omernick, K. Robinson, S. Ruder, Y. Tay, K. Xiao, Y. Xu, Y. Zhang, G. H. Abrego, J. Ahn, J. Austin,… [find](https://scholar.google.com/scholar?q=R%2E+Anil%2C+A%2E+M%2E+Dai%2C+O%2E+Firat%2C+M%2E+Johnson%2C+D%2E+Lepikhin%2C+A%2E+Passos%2C+S%2E+Shakeri%2C+E%2E+Taropa%2C+P%2E+Bailey%2C+Z%2E+Chen%2C+E%2E+Chu%2C+J%2E+H%2E+Clark%2C+L%2E+E%2E+Shafey%2C+Y%2E+Huang%2C+K%2E+Meier%2DHellstern%2C+G%2E+Mishra%2C+E%2E+Moreira%2C+M%2E+) [our card](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/) 4. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellit… [find](https://scholar.google.com/scholar?q=Y%2E+Bai%2C+S%2E+Kadavath%2C+S%2E+Kundu%2C+A%2E+Askell%2C+J%2E+Kernion%2C+A%2E+Jones%2C+A%2E+Chen%2C+A%2E+Goldie%2C+A%2E+Mirhoseini%2C+C%2E+McKinnon%2C+C%2E+Chen%2C+C%2E+Olsson%2C+C%2E+Olah%2C+D%2E+Hernandez%2C+D%2E+Drain%2C+D%2E+Ganguli%2C+D%2E+Li%2C+E%2E+Tran%2DJohnson%2C) [our card](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/) 5. C. Blakeney, M. Paul, B. W. Larsen, S. Owen, and J. Frankle. Does your data spark joy? performance gains from domain upsampling at the end of training, 2024. URL https://arxiv.org/abs/ 2406.03476 . [↗](https://arxiv.org/abs/) 6. G. Chen, M. Liao, C. Li, and K. Fan. Alphamath almost zero: process supervision without process, 2024. [find](https://scholar.google.com/scholar?q=G%2E+Chen%2C+M%2E+Liao%2C+C%2E+Li%2C+and+K%2E+Fan%2E+Alphamath+almost+zero%3A+process+supervision+without+process%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2024-alphamath-almost-zero-process-supervision/) 7. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, 2021. [find](https://scholar.google.com/scholar?q=K%2E+Cobbe%2C+V%2E+Kosaraju%2C+M%2E+Bavarian%2C+M%2E+Chen%2C+H%2E+Jun%2C+L%2E+Kaiser%2C+M%2E+Plappert%2C+J%2E+Tworek%2C+J%2E+Hilton%2C+R%2E+Nakano%2C+C%2E+Hesse%2C+and+J%2E+Schulman%2E+Training+verifiers+to+solve+math+word+problems%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/) 8. Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch. Improving factuality and reasoning in language models through multiagent debate, 2023. [find](https://scholar.google.com/scholar?q=Y%2E+Du%2C+S%2E+Li%2C+A%2E+Torralba%2C+J%2E+B%2E+Tenenbaum%2C+and+I%2E+Mordatch%2E+Improving+factuality+and+reasoning+in+language+models+through+multiagent+debate%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/du-2023-improving-factuality-reasoning-language-models/) 9. J. S. B. T. Evans. Heuristic and analytic processes in reasoning. British Journal of Psychology, 75(4): 451–468, 1984. [find](https://scholar.google.com/scholar?q=J%2E+S%2E+B%2E+T%2E+Evans%2E+Heuristic+and+analytic+processes+in+reasoning%2E+British+Journal+of+Psychology%2C+75%284%29%3A+451%E2%80%93468%2C+1984%2E) 10. X. Feng, Z. Wan, M. Wen, S. M. McAleer, Y. Wen, W. Zhang, and J. Wang. Alphazero-like tree-search can guide large language model decoding and training, 2024. [find](https://scholar.google.com/scholar?q=X%2E+Feng%2C+Z%2E+Wan%2C+M%2E+Wen%2C+S%2E+M%2E+McAleer%2C+Y%2E+Wen%2C+W%2E+Zhang%2C+and+J%2E+Wang%2E+Alphazero%2Dlike+tree%2Dsearch+can+guide+large+language+model+decoding+and+training%2C+2024%2E) 11. L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. Pal: Program-aided language models, 2023. URL https://arxiv.org/abs/2211.10435 . [↗](https://arxiv.org/abs/2211.10435) 12. S. Goyal, Z. Ji, A. S. Rawat, A. K. Menon, S. Kumar, and V. Nagarajan. Think before you speak: Training language models with pause tokens, 2024. URL https://arxiv.org/abs/2310.02226 . [↗](https://arxiv.org/abs/2310.02226) 13. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. [find](https://scholar.google.com/scholar?q=D%2E+Hendrycks%2C+C%2E+Burns%2C+S%2E+Kadavath%2C+A%2E+Arora%2C+S%2E+Basart%2C+E%2E+Tang%2C+D%2E+Song%2C+and+J%2E+Steinhardt%2E+Measuring+mathematical+problem+solving+with+the+math+dataset%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/) 14. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre. Training compute-opt… [find](https://scholar.google.com/scholar?q=J%2E+Hoffmann%2C+S%2E+Borgeaud%2C+A%2E+Mensch%2C+E%2E+Buchatskaya%2C+T%2E+Cai%2C+E%2E+Rutherford%2C+D%2E+de+Las+Casas%2C+L%2E+A%2E+Hendricks%2C+J%2E+Welbl%2C+A%2E+Clark%2C+T%2E+Hennigan%2C+E%2E+Noland%2C+K%2E+Millican%2C+G%2E+van+den+Driessche%2C+B%2E+Damoc%2C+A) 15. J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou. Large language models cannot self-correct reasoning yet, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Huang%2C+X%2E+Chen%2C+S%2E+Mishra%2C+H%2E+S%2E+Zheng%2C+A%2E+W%2E+Yu%2C+X%2E+Song%2C+and+D%2E+Zhou%2E+Large+language+models+cannot+self%2Dcorrect+reasoning+yet%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-cannot-self/) 16. A. L. Jones. Scaling scaling laws with board games, 2021. URL https://arxiv.org/abs/2104. 03113 . [↗](https://arxiv.org/abs/2104) 17. D. Kahneman. Maps of bounded rationality: Psychology for behavioral economics. The American Economic Review, 93(5):1449–1475, 2003. [find](https://scholar.google.com/scholar?q=D%2E+Kahneman%2E+Maps+of+bounded+rationality%3A+Psychology+for+behavioral+economics%2E+The+American+Economic+Review%2C+93%285%29%3A1449%E2%80%931475%2C+2003%2E) 18. D. Kahneman. Thinking, fast and slow. Farrar, Straus and Giroux, New York, first paperback edition edition, 2013. [find](https://scholar.google.com/scholar?q=D%2E+Kahneman%2E+Thinking%2C+fast+and+slow%2E+Farrar%2C+Straus+and+Giroux%2C+New+York%2C+first+paperback+edition+edition%2C+2013%2E) 19. L. Kocsis and C. Szepesv’ari. Bandit based monte-carlo planning. In European conference on machine learning, pages 282–293. Springer, 2006. [find](https://scholar.google.com/scholar?q=L%2E+Kocsis+and+C%2E+Szepesv%E2%80%99ari%2E+Bandit+based+monte%2Dcarlo+planning%2E+In+European+conference+on+machine+learning%2C+pages+282%E2%80%93293%2E+Springer%2C+2006%2E) 20. A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra. Solving quantitative reasoning problems with language models, 2022. [find](https://scholar.google.com/scholar?q=A%2E+Lewkowycz%2C+A%2E+Andreassen%2C+D%2E+Dohan%2C+E%2E+Dyer%2C+H%2E+Michalewski%2C+V%2E+Ramasesh%2C+A%2E+Slone%2C+C%2E+Anil%2C+I%2E+Schlag%2C+T%2E+Gutman%2DSolo%2C+Y%2E+Wu%2C+B%2E+Neyshabur%2C+G%2E+Gur%2DAri%2C+and+V%2E+Misra%2E+Solving+quantitative+reasoning) [our card](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/) 21. Y. Li, Z. Lin, S. Zhang, Q. Fu, B. Chen, J.-G. Lou, and W. Chen. Making large language models better reasoners with step-aware verifier, 2023. [find](https://scholar.google.com/scholar?q=Y%2E+Li%2C+Z%2E+Lin%2C+S%2E+Zhang%2C+Q%2E+Fu%2C+B%2E+Chen%2C+J%2E%2DG%2E+Lou%2C+and+W%2E+Chen%2E+Making+large+language+models+better+reasoners+with+step%2Daware+verifier%2C+2023%2E) 22. H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step, 2023. [find](https://scholar.google.com/scholar?q=H%2E+Lightman%2C+V%2E+Kosaraju%2C+Y%2E+Burda%2C+H%2E+Edwards%2C+B%2E+Baker%2C+T%2E+Lee%2C+J%2E+Leike%2C+J%2E+Schulman%2C+I%2E+Sutskever%2C+and+K%2E+Cobbe%2E+Let%E2%80%99s+verify+step+by+step%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/) 23. A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark. Self-refine: Iterative refinement with self-feedback, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Madaan%2C+N%2E+Tandon%2C+P%2E+Gupta%2C+S%2E+Hallinan%2C+L%2E+Gao%2C+S%2E+Wiegreffe%2C+U%2E+Alon%2C+N%2E+Dziri%2C+S%2E+Prabhumoye%2C+Y%2E+Yang%2C+S%2E+Gupta%2C+B%2E+P%2E+Majumder%2C+K%2E+Hermann%2C+S%2E+Welleck%2C+A%2E+Yazdanbakhsh%2C+and+P%2E+Clark%2E+Self%2Drefi) 24. N. McAleese, R. Pokorny, J. F. Cerón Uribe, E. Nitishinskaya, M. Trębacz, and J. Leike. Llm critics help catch llm bugs. OpenAI, 2024. [find](https://scholar.google.com/scholar?q=N%2E+McAleese%2C+R%2E+Pokorny%2C+J%2E+F%2E+Cer%C3%B3n+Uribe%2C+E%2E+Nitishinskaya%2C+M%2E+Tr%C4%99bacz%2C+and+J%2E+Leike%2E+Llm+critics+help+catch+llm+bugs%2E+OpenAI%2C+2024%2E) 25. OpenAI. Gpt-4 technical report, 2024. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/) 26. Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, S. Zhao, L. Hong, R. Tian, R. Xie, J. Zhou, M. Gerstein, D. Li, Z. Liu, and M. Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023. URL https://arxiv.org/abs/2307.16789 . [↗](https://arxiv.org/abs/2307.16789) [our card](https://md.research.elamur.ai/free/en/papers/qin-2023-toolllm-facilitating-large-language-models/) 27. C. Qu, S. Dai, X. Wei, H. Cai, S. Wang, D. Yin, J. Xu, and J.-R. Wen. Tool learning with large language models: A survey, 2024. URL https://arxiv.org/abs/2405.17935 . [↗](https://arxiv.org/abs/2405.17935) 28. Y. Qu, T. Zhang, N. Garg, and A. Kumar. Recursive introspection: Teaching foundation models how to self-improve. 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Qu%2C+T%2E+Zhang%2C+N%2E+Garg%2C+and+A%2E+Kumar%2E+Recursive+introspection%3A+Teaching+foundation+models+how+to+self%2Dimprove%2E+2024%2E) 29. N. Sardana and J. Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws, 2023. [find](https://scholar.google.com/scholar?q=N%2E+Sardana+and+J%2E+Frankle%2E+Beyond+chinchilla%2Doptimal%3A+Accounting+for+inference+in+language+model+scaling+laws%2C+2023%2E) 30. W. Saunders, C. Yeh, J. Wu, S. Bills, L. Ouyang, J. Ward, and J. Leike. Self-critiquing models for assisting human evaluators, 2022. [find](https://scholar.google.com/scholar?q=W%2E+Saunders%2C+C%2E+Yeh%2C+J%2E+Wu%2C+S%2E+Bills%2C+L%2E+Ouyang%2C+J%2E+Ward%2C+and+J%2E+Leike%2E+Self%2Dcritiquing+models+for+assisting+human+evaluators%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/saunders-2022-self-critiquing-models-assisting-human/) 31. A. Setlur, S. Garg, X. Geng, N. Garg, V. Smith, and A. Kumar. Rl on incorrect synthetic data scales the efficiency of llm math reasoning by eight-fold. arXiv preprint arXiv:2406.14532, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Setlur%2C+S%2E+Garg%2C+X%2E+Geng%2C+N%2E+Garg%2C+V%2E+Smith%2C+and+A%2E+Kumar%2E+Rl+on+incorrect+synthetic+data+scales+the+efficiency+of+llm+math+reasoning+by+eight%2Dfold%2E+arXiv+preprint+arXiv%3A2406%2E14532%2C+2024%2E) 32. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. [find](https://scholar.google.com/scholar?q=Z%2E+Shao%2C+P%2E+Wang%2C+Q%2E+Zhu%2C+R%2E+Xu%2C+J%2E+Song%2C+X%2E+Bi%2C+H%2E+Zhang%2C+M%2E+Zhang%2C+Y%2E+K%2E+Li%2C+Y%2E+Wu%2C+and+D%2E+Guo%2E+Deepseekmath%3A+Pushing+the+limits+of+mathematical+reasoning+in+open+language+models%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/) 33. A. Sharma, S. Keh, E. Mitchell, C. Finn, K. Arora, and T. Kollar. A critical evaluation of ai feedback for aligning large language models, 2024. URL https://arxiv.org/abs/2402.12366 . — [arXiv:2402.12366](https://arxiv.org/abs/2402.12366) [↗](https://arxiv.org/abs/2402.12366) 34. N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. [find](https://scholar.google.com/scholar?q=N%2E+Shinn%2C+F%2E+Cassano%2C+E%2E+Berman%2C+A%2E+Gopinath%2C+K%2E+Narasimhan%2C+and+S%2E+Yao%2E+Reflexion%3A+Language+agents+with+verbal+reinforcement+learning%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/) 35. A. Singh, J. D. Co-Reyes, R. Agarwal, A. Anand, P. Patil, X. Garcia, P. J. Liu, J. Harrison, J. Lee, K. Xu, A. Parisi, A. Kumar, A. Alemi, A. Rizkowsky, A. Nova, B. Adlam, B. Bohnet, G. Elsayed, H. Sedghi, I. Mordatch, I. Simpson, I. Gur, J. Snoek, J. Pennington, J. Hron, K. Kenealy, K. Swersky, K.… [find](https://scholar.google.com/scholar?q=A%2E+Singh%2C+J%2E+D%2E+Co%2DReyes%2C+R%2E+Agarwal%2C+A%2E+Anand%2C+P%2E+Patil%2C+X%2E+Garcia%2C+P%2E+J%2E+Liu%2C+J%2E+Harrison%2C+J%2E+Lee%2C+K%2E+Xu%2C+A%2E+Parisi%2C+A%2E+Kumar%2C+A%2E+Alemi%2C+A%2E+Rizkowsky%2C+A%2E+Nova%2C+B%2E+Adlam%2C+B%2E+Bohnet%2C+G%2E+Elsayed%2C+H%2E+Se) [our card](https://md.research.elamur.ai/free/en/papers/singh-2023-human-data-scaling-self-training/) 36. C. Snell, E. Wallace, D. Klein, and S. Levine. Predicting emergent capabilities by finetuning. Conference on Language Modeling 2024, 2024. [find](https://scholar.google.com/scholar?q=C%2E+Snell%2C+E%2E+Wallace%2C+D%2E+Klein%2C+and+S%2E+Levine%2E+Predicting+emergent+capabilities+by+finetuning%2E+Conference+on+Language+Modeling+2024%2C+2024%2E) 37. K. Stechly, M. Marquez, and S. Kambhampati. Gpt-4 doesn’t know it’s wrong: An analysis of iterative prompting for reasoning problems, 2023. [find](https://scholar.google.com/scholar?q=K%2E+Stechly%2C+M%2E+Marquez%2C+and+S%2E+Kambhampati%2E+Gpt%2D4+doesn%E2%80%99t+know+it%E2%80%99s+wrong%3A+An+analysis+of+iterative+prompting+for+reasoning+problems%2C+2023%2E) 38. R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. Second edition, 2018. [find](https://scholar.google.com/scholar?q=R%2E+S%2E+Sutton+and+A%2E+G%2E+Barto%2E+Reinforcement+learning%3A+An+introduction%2E+Second+edition%2C+2018%2E) 39. G. Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. [find](https://scholar.google.com/scholar?q=G%2E+Team%2E+Gemini+1%2E5%3A+Unlocking+multimodal+understanding+across+millions+of+tokens+of+context%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/gemini-2024-team-gemini-1-5-unlocking-multimodal/) 40. Y. Tian, B. Peng, L. Song, L. Jin, D. Yu, H. Mi, and D. Yu. Toward self-improvement of llms via imagination, searching, and criticizing, 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Tian%2C+B%2E+Peng%2C+L%2E+Song%2C+L%2E+Jin%2C+D%2E+Yu%2C+H%2E+Mi%2C+and+D%2E+Yu%2E+Toward+self%2Dimprovement+of+llms+via+imagination%2C+searching%2C+and+criticizing%2C+2024%2E) 41. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan,… [find](https://scholar.google.com/scholar?q=H%2E+Touvron%2C+L%2E+Martin%2C+K%2E+Stone%2C+P%2E+Albert%2C+A%2E+Almahairi%2C+Y%2E+Babaei%2C+N%2E+Bashlykov%2C+S%2E+Batra%2C+P%2E+Bhargava%2C+S%2E+Bhosale%2C+D%2E+Bikel%2C+L%2E+Blecher%2C+C%2E+C%2E+Ferrer%2C+M%2E+Chen%2C+G%2E+Cucurull%2C+D%2E+Esiobu%2C+J%2E+Fernandes%2C) 42. J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins. Solving math word problems with process- and outcome-based feedback, 2022. [find](https://scholar.google.com/scholar?q=J%2E+Uesato%2C+N%2E+Kushman%2C+R%2E+Kumar%2C+F%2E+Song%2C+N%2E+Siegel%2C+L%2E+Wang%2C+A%2E+Creswell%2C+G%2E+Irving%2C+and+I%2E+Higgins%2E+Solving+math+word+problems+with+process%2D+and+outcome%2Dbased+feedback%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/uesato-2022-solving-math-word-problems-process/) 43. K. Valmeekam, M. Marquez, and S. Kambhampati. Can large language models really improve by self-critiquing their own plans?, 2023. [find](https://scholar.google.com/scholar?q=K%2E+Valmeekam%2C+M%2E+Marquez%2C+and+S%2E+Kambhampati%2E+Can+large+language+models+really+improve+by+self%2Dcritiquing+their+own+plans%3F%2C+2023%2E) 44. P. Villalobos and D. Atkinson. Trading off compute in training and inference, 2023. URL [https://epochai.org/blog/trading-off-compute-in-training-and-inference](https://epochai.org/blog/trading-off-compute-in-training-and-inference) . Accessed: 2024-07-03. [↗](https://epochai.org/blog/trading-off-compute-in-training-and-inference) 45. P. Wang, L. Li, Z. Shao, R. X. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations, 2023. [find](https://scholar.google.com/scholar?q=P%2E+Wang%2C+L%2E+Li%2C+Z%2E+Shao%2C+R%2E+X%2E+Xu%2C+D%2E+Dai%2C+Y%2E+Li%2C+D%2E+Chen%2C+Y%2E+Wu%2C+and+Z%2E+Sui%2E+Math%2Dshepherd%3A+Verify+and+reinforce+llms+step%2Dby%2Dstep+without+human+annotations%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2024-math-shepherd-verify-reinforce-llms/) 46. R. Wang, E. Zelikman, G. Poesia, Y. Pu, N. Haber, and N. D. Goodman. Hypothesis search: Inductive reasoning with language models, 2024. URL https://arxiv.org/abs/2309.05660 . [↗](https://arxiv.org/abs/2309.05660) 47. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Wei%2C+X%2E+Wang%2C+D%2E+Schuurmans%2C+M%2E+Bosma%2C+B%2E+Ichter%2C+F%2E+Xia%2C+E%2E+Chi%2C+Q%2E+Le%2C+and+D%2E+Zhou%2E+Chain%2Dof%2Dthought+prompting+elicits+reasoning+in+large+language+models%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/) 48. S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Yao%2C+D%2E+Yu%2C+J%2E+Zhao%2C+I%2E+Shafran%2C+T%2E+L%2E+Griffiths%2C+Y%2E+Cao%2C+and+K%2E+Narasimhan%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2C+2023%2E) 49. Z. Yuan, H. Yuan, C. Li, G. Dong, K. Lu, C. Tan, C. Zhou, and J. Zhou. Scaling relationship on learning mathematical reasoning with large language models, 2023. [find](https://scholar.google.com/scholar?q=Z%2E+Yuan%2C+H%2E+Yuan%2C+C%2E+Li%2C+G%2E+Dong%2C+K%2E+Lu%2C+C%2E+Tan%2C+C%2E+Zhou%2C+and+J%2E+Zhou%2E+Scaling+relationship+on+learning+mathematical+reasoning+with+large+language+models%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/yuan-2023-scaling-relationship-learning-mathematical-reasoning/) 50. E. Zelikman, Y. Wu, J. Mu, and N. D. Goodman. Star: Bootstrapping reasoning with reasoning, 2022. [find](https://scholar.google.com/scholar?q=E%2E+Zelikman%2C+Y%2E+Wu%2C+J%2E+Mu%2C+and+N%2E+D%2E+Goodman%2E+Star%3A+Bootstrapping+reasoning+with+reasoning%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/) 51. E. Zelikman, G. Harik, Y. Shao, V. Jayasiri, N. Haber, and N. D. Goodman. Quiet-star: Language models can teach themselves to think before speaking, 2024. URL https://arxiv.org/abs/ 2403.09629 . [↗](https://arxiv.org/abs/) [our card](https://md.research.elamur.ai/free/en/papers/al-2024-quiet-star-language-models-teach/)