---
title: "Think Hard Only When Needed — paper by Suh et al. (2026)"
description: "The authors propose THROW, a two-stage pipeline aimed at streaming, batch-1 inference. 0.9% accuracy. 22 references."
authors: ["Hyewon Suh", "Chaojian Li", "Cheng-Jhih Shih", "Zheng Wang", "Kejing Xia", "Yonggan Fu", "Yingyan Celine Lin"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: EACL 2026"
source: "https://aclanthology.org/2026.findings-eacl.315/"
doi: "10.18653/v1/2026.findings-eacl.315"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/suh-2026-think-hard-only-when-needed/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/suh-2026-think-hard-only-when-needed/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/45/) / Think Hard Only When Needed: A Hybrid Best-of-N and Beam Search for Efficient Test-Time Compute

# Think Hard Only When Needed: A Hybrid Best-of-N and Beam Search for Efficient Test-Time Compute

**Authors:** Hyewon Suh, Chaojian Li, Cheng-Jhih Shih, [Zheng Wang](https://md.research.elamur.ai/free/en/authors/zheng-wang/), Kejing Xia, [Yonggan Fu](https://md.research.elamur.ai/free/en/authors/yonggan-fu/) and [Yingyan Celine Lin](https://md.research.elamur.ai/free/en/authors/yingyan-celine-lin/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: EACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-eacl.315/) · [PDF](https://aclanthology.org/2026.findings-eacl.315.pdf) · [DOI 10.18653/v1/2026.findings-eacl.315](https://doi.org/10.18653/v1/2026.findings-eacl.315) · [OpenAlex](https://openalex.org/W7140128666)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Test-time compute lets small language models reason better by spending extra inference effort, but the two standard strategies have opposite costs. Best-of-N runs many full trajectories in parallel and wastes tokens on paths that fail early. PRM-guided beam search steers generation more carefully but is slow because it decodes and scores step by step. The authors propose THROW, a two-stage pipeline aimed at streaming, batch-1 inference. In Stage I it samples short partial trajectories, as Best-of-N would, and scores them with a Process Reward Model. Those scores form an indicator that labels each query as easy or hard. In Stage II, easy queries get their weak branches cut, while hard queries get beam-search-style expansion of their most promising branches. Tested on MATH500, AMC23 and AIME24, THROW is much faster and uses far fewer tokens than both baselines. Its accuracy stays near that of beam search. This matters for running capable reasoning on memory- and latency-constrained devices.

## Key points
- Speed: THROW runs on average 1.54× faster than Best-of-N and 14.38× faster than beam search in wall-clock time.
- Tokens: it generates 35.7% fewer tokens than Best-of-N and 80.4% fewer than beam search.
- Accuracy: with Qwen2.5-1.5B-Instruct, it loses only about 0.9% accuracy compared with beam search.
- Difficulty indicator: the PRM-based score correlates with oracle pass@1 accuracy, with absolute Pearson values from 0.679 to 0.801 across benchmarks. AIME24 shows the weakest correlation because many of its problems are unsolvable.
- Ablations on MATH500:
  - THROW's Stage II branch selection beats random selection by about 5.2% on average (e.g., 0.674 vs 0.604 at N=8).
  - Stage I length was tested from 150 to 550 tokens, and a moderate length gave the best trade-off.
  - The approach depends on PRM signals and is limited to verifier-based setups.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Georgia Institute of Technology](https://md.research.elamur.ai/free/en/universities/georgia-institute-of-technology/) — University, Atlanta, United States · 133 works · 24,497 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

## Cite

Cite as: Hyewon Suh, Chaojian Li, Cheng-Jhih Shih, Zheng Wang, Kejing Xia, Yonggan Fu and Yingyan Celine Lin. 2026. Think Hard Only When Needed: A Hybrid Best-of-N and Beam Search for Efficient Test-Time Compute. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{suh2026think,
  title = {{Think Hard Only When Needed: A Hybrid Best-of-N and Beam Search for Efficient Test-Time Compute}},
  author = {Hyewon Suh, Chaojian Li, Cheng-Jhih Shih, Zheng Wang, Kejing Xia, Yonggan Fu and Yingyan Celine Lin},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.315},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.315}
}
```

- [← previous Think Faster Than Words: Efficient LLM Chain-of-Thought Reasoning via Dynamic Shortcut Decoding](https://md.research.elamur.ai/free/en/papers/liu-2026-think-faster-than-words-efficient/)
- [next → Think Less, Know More: State-Aware Reasoning Compression with Knowledge Guidance for Efficient Reasoning](https://md.research.elamur.ai/free/en/papers/sui-2026-think-less-know-more-state/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/45/)

## References

22 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. AI-MO. 2024. AIME 2024 dataset. //huggingface.co/datasets/AI-MO/ aimo-validation-aime. https: [find](https://scholar.google.com/scholar?q=AI%2DMO%2E+2024%2E+AIME+2024+dataset%2E+%2F%2Fhuggingface%2Eco%2Fdatasets%2FAI%2DMO%2F+aimo%2Dvalidation%2Daime%2E+https%3A)
2. Edward Beeching, Lewis Tunstall, and Sasha Rush. Scaling test-time compute with open models. [find](https://scholar.google.com/scholar?q=Edward+Beeching%2C+Lewis+Tunstall%2C+and+Sasha+Rush%2E+Scaling+test%2Dtime+compute+with+open+models%2E)
3. Wei Chen, Zhiyuan Li, Zhen Guo, and Yikang Shen. 2024. Octo-planner: On-device language model for planner-action agents. arXiv preprint arXiv:2406.18082. — [arXiv:2406.18082](https://arxiv.org/abs/2406.18082) [find](https://scholar.google.com/scholar?q=Wei+Chen%2C+Zhiyuan+Li%2C+Zhen+Guo%2C+and+Yikang+Shen%2E+2024%2E+Octo%2Dplanner%3A+On%2Ddevice+language+model+for+planner%2Daction+agents%2E+arXiv+preprint+arXiv%3A2406%2E18082%2E+%E2%80%94+%5BarXiv%3A2406%2E18082%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F240)
4. Mehul Damani, Idan Shenfeld, Andi Peng, Andreea Bobu, and Jacob Andreas. 2025. Learning how hard to think: Input-adaptive allocation of LM computation. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Mehul+Damani%2C+Idan+Shenfeld%2C+Andi+Peng%2C+Andreea+Bobu%2C+and+Jacob+Andreas%2E+2025%2E+Learning+how+hard+to+think%3A+Input%2Dadaptive+allocation+of+LM+computation%2E+In+The+Thirteenth+International+Conference+on+Le)
5. Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. 2024. RLHF workflow: From reward modeling to online RLHF. arXiv preprint arXiv:2405.07863. [find](https://scholar.google.com/scholar?q=Hanze+Dong%2C+Wei+Xiong%2C+Bo+Pang%2C+Haoxiang+Wang%2C+Han+Zhao%2C+Yingbo+Zhou%2C+Nan+Jiang%2C+Doyen+Sahoo%2C+Caiming+Xiong%2C+and+Tong+Zhang%2E+2024%2E+RLHF+workflow%3A+From+reward+modeling+to+online+RLHF%2E+arXiv+preprint+ar)
6. Yichao Fu, Junda Chen, Yonghao Zhuang, Zheyu Fu, Ion Stoica, and Hao Zhang. 2025. Reasoning without self-doubt: More efficient chain-of-thought through certainty probing. In ICLR 2025 Workshop on Foundation Models in the Wild. [find](https://scholar.google.com/scholar?q=Yichao+Fu%2C+Junda+Chen%2C+Yonghao+Zhuang%2C+Zheyu+Fu%2C+Ion+Stoica%2C+and+Hao+Zhang%2E+2025%2E+Reasoning+without+self%2Ddoubt%3A+More+efficient+chain%2Dof%2Dthought+through+certainty+probing%2E+In+ICLR+2025+Workshop+on+Foun)
7. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
8. Jujie He, Tianwen Wei, Rui Yan, Jiacai Liu, Chaojie Wang, Yimeng Gan, Shiwen Tu, Chris Yuhao Liu, Liang Zeng, Xiaokun Wang, Boyang Wang, Yongcong Li, Fuxiang Zhang, Jiacheng Xu, Bo An, Yang Liu, and Yahui Zhou. 2024. Skywork-o1 open series. [find](https://scholar.google.com/scholar?q=Jujie+He%2C+Tianwen+Wei%2C+Rui+Yan%2C+Jiacai+Liu%2C+Chaojie+Wang%2C+Yimeng+Gan%2C+Shiwen+Tu%2C+Chris+Yuhao+Liu%2C+Liang+Zeng%2C+Xiaokun+Wang%2C+Boyang+Wang%2C+Yongcong+Li%2C+Fuxiang+Zhang%2C+Jiacheng+Xu%2C+Bo+An%2C+Yang+Liu%2C+and+Y)
9. Chengsong Huang, Langlin Huang, Jixuan Leng, Jiacheng Liu, and Jiaxin Huang. 2025. Efficient test-time scaling via self-calibration. Preprint, arXiv:2503.00031. — [arXiv:2503.00031](https://arxiv.org/abs/2503.00031) [find](https://scholar.google.com/scholar?q=Chengsong+Huang%2C+Langlin+Huang%2C+Jixuan+Leng%2C+Jiacheng+Liu%2C+and+Jiaxin+Huang%2E+2025%2E+Efficient+test%2Dtime+scaling+via+self%2Dcalibration%2E+Preprint%2C+arXiv%3A2503%2E00031%2E+%E2%80%94+%5BarXiv%3A2503%2E00031%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
10. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). In The Twelfth International Conference on Learning Representations.
11. Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. 2025. [Can 1b llm surpass 405b llm? rethinking compute-optimal test-time scaling](https://md.research.elamur.ai/free/en/papers/liu-2025-1b-llm-surpass-405b-llm/). Preprint, arXiv:2502.06703.
12. Rohin Manvi, Anikait Singh, and Stefano Ermon. 2024. [Adaptive inference-time compute: Llms can predict if they can do better, even mid-generation](https://md.research.elamur.ai/free/en/papers/manvi-2024-adaptive-inference-time-compute-llms/). Preprint, arXiv:2410.02725. math-ai. 2025.
13. AMC 2023 dataset. [https://huggingface.co/datasets/math-ai/amc23](https://huggingface.co/datasets/math-ai/amc23). [↗](https://huggingface.co/datasets/math-ai/amc23)
14. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=Qwen%2C+%3A%2C+An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+)
15. Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2025. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Charlie+Victor+Snell%2C+Jaehoon+Lee%2C+Kelvin+Xu%2C+and+Aviral+Kumar%2E+2025%2E+Scaling+LLM+test%2Dtime+compute+optimally+can+be+more+effective+than+scaling+parameters+for+reasoning%2E+In+The+Thirteenth+Internation)
16. Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024. [Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations](https://md.research.elamur.ai/free/en/papers/wang-2024-math-shepherd-verify-reinforce-llms/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper…
17. Yiming Wang, Pei Zhang, Siyuan Huang, Baosong Yang, Zhuosheng Zhang, Fei Huang, and Rui Wang. 2025. Sampling-efficient test-time scaling: Self-estimating the best-of-n sampling in early decoding. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Yiming+Wang%2C+Pei+Zhang%2C+Siyuan+Huang%2C+Baosong+Yang%2C+Zhuosheng+Zhang%2C+Fei+Huang%2C+and+Rui+Wang%2E+2025%2E+Sampling%2Defficient+test%2Dtime+scaling%3A+Self%2Destimating+the+best%2Dof%2Dn+sampling+in+early+decoding%2E+In+T)
18. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems.
19. Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. 2024. Scaling inference computation: Compute-optimal inference for problem-solving with language models. In The 4th Workshop on Mathematical Reasoning and AI at NeurIPS’24. [find](https://scholar.google.com/scholar?q=Yangzhen+Wu%2C+Zhiqing+Sun%2C+Shanda+Li%2C+Sean+Welleck%2C+and+Yiming+Yang%2E+2024%2E+Scaling+inference+computation%3A+Compute%2Doptimal+inference+for+problem%2Dsolving+with+language+models%2E+In+The+4th+Workshop+on+Math)
20. Wei Xiong, Hanning Zhang, Nan Jiang, and Tong Zhang. 2024. An implementation of generative prm. [https://huggingface.co/RLHFlow/](https://huggingface.co/RLHFlow/) Llama3.1-8B-PRM-Deepseek-Data. [↗](https://huggingface.co/RLHFlow/)
21. Chenxu Yang, Qingyi Si, Yongjie Duan, Zheliang Zhu, Chenyu Zhu, Qiaowei Li, Minghui Chen, Zheng Lin, and Weiping Wang. 2025. Dynamic early exit in reasoning models. Preprint, arXiv:2504.15895. [find](https://scholar.google.com/scholar?q=Chenxu+Yang%2C+Qingyi+Si%2C+Yongjie+Duan%2C+Zheliang+Zhu%2C+Chenyu+Zhu%2C+Qiaowei+Li%2C+Minghui+Chen%2C+Zheng+Lin%2C+and+Weiping+Wang%2E+2025%2E+Dynamic+early+exit+in+reasoning+models%2E+Preprint%2C+arXiv%3A2504%2E15895%2E)
22. Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. [The lessons of developing process reward models in mathematical reasoning](https://md.research.elamur.ai/free/en/papers/zhang-2025-lessons-developing-process-reward-models/). In Findings of the Association for Computational Linguistics: ACL 2025, pages 10495–10516, Vienna…

*Updated 1 November 2026*