--- title: "Do LLMs Overthink Basic Math Reasoning? Benchmarking the" description: "The paper also proposes the Overthinking Score, the harmonic mean of accuracy and normalized token… 1 reference, cited by 3 papers here." authors: ["Gaurav Srivastava", "Aafiya Shamshad Hussain", "Sriram Srinivasan", "Xuan Wang"] year: 2026 venue: "Findings of the Association for Computational Linguistics: ACL 2026" source: "https://aclanthology.org/2026.findings-acl.1285/" doi: "10.18653/v1/2026.findings-acl.1285" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/srivastava-2026-do-llms-overthink-basic-math/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/srivastava-2026-do-llms-overthink-basic-math/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/5/) / Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models # Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models **Authors:** [Gaurav Srivastava](https://md.research.elamur.ai/free/en/authors/gaurav-srivastava/), Aafiya Shamshad Hussain, Sriram Srinivasan and [Xuan Wang](https://md.research.elamur.ai/free/en/authors/xuan-wang/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1285/) · [PDF](https://aclanthology.org/2026.findings-acl.1285.pdf) · [DOI 10.18653/v1/2026.findings-acl.1285](https://doi.org/10.18653/v1/2026.findings-acl.1285) · [OpenAlex](https://openalex.org/W4415314084) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/srivastava-2026-do-llms-overthink-basic-math/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper investigates "overthinking": LLMs, especially reasoning-tuned ones, produce very long chains of thought on trivial arithmetic. These chains often do not improve correctness and can even hurt it. To measure this, the authors built LLMThinkBench. It generates fresh test instances on the fly for 14 deterministic basic-math tasks, including sorting, comparison, arithmetic operations, max/min, mean/median/mode and odd/even counts, which limits contamination. Answers are extracted with a hierarchical parser. The paper also proposes the Overthinking Score, the harmonic mean of accuracy and normalized token efficiency, so a model only scores well if it is both correct and concise. The authors evaluated 53 LLMs, covering standard, reasoning and quantized variants under different reasoning budgets. Reasoning models used far more tokens yet often lost accuracy, and they degraded sharply when output length was capped. Larger reasoning budgets brought little or no benefit. Efficiency-aware rankings favored mid-sized instruction-tuned models. The benchmark is released as a PyPI package with a public leaderboard, which gives a reusable standard for judging reasoning efficiency alongside accuracy. ## Key points - **Token use vs. accuracy:** reasoning models averaged about 6,780 output tokens versus 378 for standard models, roughly 18× more. - **Collapse under limits:** with a 1,024-token cap, accuracy fell from 72% to 44%. - **Reasoning effort:** GPT-5, o3 and o4-mini peaked at low effort, and medium or high effort only added tokens without improving accuracy. - **Overthinking Score rankings:** - GPT-4.1-mini scored 0.930 and GPT-4.1 scored 0.927. - Qwen3-14B (0.727) beat Qwen3-32B (0.668). - Phi-4-reasoning scored 0.352 against 0.863 for Phi-4. - **Causes of wasted tokens:** about 5,000 hand-annotated responses showed four main patterns: redundant verification, self-contradiction, irrelevant exploration and stopping failures. The authors trace these mainly to CoT supervision. Concise prompts and tools (calculator, Python) narrowed the gap but did not close it. - **Quantization:** compression barely affected large models (Qwen2.5-32B: 73.08% → 72.67%) but hurt tiny ones badly (Qwen2.5-0.5B: 21.31% → 12.77%). - **Availability:** the benchmark installs with `pip install llmthinkbench`. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Virginia Tech](https://md.research.elamur.ai/free/en/universities/virginia-tech/) — University, Blacksburg, United States · 36 works · 43,036 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works 🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works ## Cite Cite as: Gaurav Srivastava, Aafiya Shamshad Hussain, Sriram Srinivasan and Xuan Wang. 2026. Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models. Findings of the Association for Computational Linguistics: ACL 2026. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{srivastava2026do, title = {{Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models}}, author = {Gaurav Srivastava, Aafiya Shamshad Hussain, Sriram Srinivasan and Xuan Wang}, booktitle = {Findings of the Association for Computational Linguistics: ACL 2026}, year = {2026}, doi = {10.18653/v1/2026.findings-acl.1285}, url = {https://doi.org/10.18653/v1/2026.findings-acl.1285} } ``` - [← previous Do Large Language Models Reflect Demographic Pluralism in Safety?](https://md.research.elamur.ai/free/en/papers/naseem-2026-do-large-language-models-reflect/) - [next → Do Not Step Into the Same River Twice: Learning to Reason from Trial and Error](https://md.research.elamur.ai/free/en/papers/tang-2026-do-not-step-into-same/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/5/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. This work was supported by the NSF \#2442253, NSF NAIRR Pilot with PSC Neocortex and NCSA Delta, Cisco Research, NVIDIA, Amazon, the Commonwealth Cyber Initiative, the Amazon–Virginia Tech Center for Efficient and Robust Machine Learning, the Sanghani Center for AI and Data Analytics at Virginia Tech… [find](https://scholar.google.com/scholar?q=This+work+was+supported+by+the+NSF+%5C%232442253%2C+NSF+NAIRR+Pilot+with+PSC+Neocortex+and+NCSA+Delta%2C+Cisco+Research%2C+NVIDIA%2C+Amazon%2C+the+Commonwealth+Cyber+Initiative%2C+the+Amazon%E2%80%93Virginia+Tech+Center+for+) *Updated 31 October 2026*