--- title: "Over-Reasoning and Redundant Calculation of Large Language" description: "The authors build GSM8K-Zero from GSM8K. 3 references, cited by 28 papers here. Among them: 5." authors: ["Cheng-Han Chiang", "Hung-yi Lee"] year: 2024 venue: "Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)" source: "https://aclanthology.org/2024.eacl-short.15/" doi: "10.18653/v1/2024.eacl-short.15" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/chiang-2024-over-reasoning-redundant-calculation-large/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chiang-2024-over-reasoning-redundant-calculation-large/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / Over-Reasoning and Redundant Calculation of Large Language Models # Over-Reasoning and Redundant Calculation of Large Language Models **Authors:** Cheng-Han Chiang and [Hung-yi Lee](https://md.research.elamur.ai/free/en/authors/hung-yi-lee/) · **Year:** 2024 · **Venue:** [Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2024](https://md.research.elamur.ai/free/en/venues/eacl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2024.eacl-short.15/) · [PDF](https://aclanthology.org/2024.eacl-short.15.pdf) · [DOI 10.18653/v1/2024.eacl-short.15](https://doi.org/10.18653/v1/2024.eacl-short.15) · [OpenAlex](https://openalex.org/W4411638675) **Citations:** 5 citations (OpenAlex, 2026-10-14) · [28](https://md.research.elamur.ai/free/en/papers/chiang-2024-over-reasoning-redundant-calculation-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Chain-of-thought (CoT) reasoning helps LLMs on math problems, but it also makes outputs longer, slower and more expensive. This paper asks whether models can tell when step-by-step work is actually needed. The authors build GSM8K-Zero from GSM8K. They drop each question's final query and replace it with a ChatGPT-generated question about a quantity already stated in the text, so no arithmetic is required. GPT-4 then filters the generated pairs. An answer counts as redundant if a regular expression finds the operators ×, + or = in it. Seven RLHF-trained models were tested zero-shot. Most of them still produced unnecessary calculations, and answers containing calculations were usually less accurate. The authors trace this partly to reward modeling. When GPT-4 and ChatGPT act as proxy reward models, they favour long, calculation-heavy answers even when those answers are wrong. The study highlights a previously unexamined failure mode: over-reasoning on trivial inputs. It argues for training methods that teach LLMs when step-by-step reasoning is warranted. ## Key points - **Dataset construction:** GSM8K-Zero was built from 3,500 sampled GSM8K training questions and contains 2,978 question–answer pairs. A manual check of 250 pairs suggests about 85% are valid. It is released on GitHub and Hugging Face (`dcml0714/GSM8K-Zero`). - **Redundancy rates:** Llama-2-chat models (7b, 13b, 70b) included calculations in over 80% of responses. Claude-2 did so in about 75%, ChatGPT in about 47%, PaLM (text-bison-001) in about 29% and GPT-4 in about 12%. - **Accuracy:** Several models stayed near or below 50% on this essentially extractive task, including PaLM (~41%) and Llama-2-13b/7b (~40%). For most models, accuracy fell sharply when calculations appeared; for example, ChatGPT dropped from ~97% to ~61%. Typical errors were hallucinated variables and arithmetic mistakes. - **Prompting to skip CoT:** Explicitly allowing models to skip CoT reduced redundancy, for instance ChatGPT to ~26% and Llama-2-7b-chat to ~33%, but did not eliminate it. - **Proxy reward models:** As proxy reward models, ChatGPT consistently preferred long but wrong answers over short correct ones. GPT-4 chose the short correct answer 61% of the time but still picked the long wrong one in 34% of cases. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [National Taiwan University](https://md.research.elamur.ai/free/en/universities/national-taiwan-university/) — University, Taipei, Taiwan · 29 works · 2,116 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works 🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works ## Cite Cite as: Cheng-Han Chiang and Hung-yi Lee. 2024. Over-Reasoning and Redundant Calculation of Large Language Models. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{chiang2024over, title = {{Over-Reasoning and Redundant Calculation of Large Language Models}}, author = {Cheng-Han Chiang and Hung-yi Lee}, booktitle = {Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)}, year = {2024}, doi = {10.18653/v1/2024.eacl-short.15}, url = {https://doi.org/10.18653/v1/2024.eacl-short.15} } ``` - [← previous Me LLaMA: Foundation Large Language Models for Medical Applications](https://md.research.elamur.ai/free/en/papers/xie-2024-me-llama-foundation-large-language/) - [next → SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety](https://md.research.elamur.ai/free/en/papers/rottger-2024-safetyprompts-systematic-review-open-datasets/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) ## References 3 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/) 2. Yuvanesh Anand, Zach Nussbaum, Brandon Duderstadt, Benjamin Schmidt, and Andriy Mulyar. 2023. [find](https://scholar.google.com/scholar?q=Yuvanesh+Anand%2C+Zach+Nussbaum%2C+Brandon+Duderstadt%2C+Benjamin+Schmidt%2C+and+Andriy+Mulyar%2E+2023%2E) 3. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan \#\# Acknowledgements [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+%5C%23%5C%23+Acknowledgements) *Updated 30 October 2026*