--- title: "Do LLMs Really Think Step-by-step In Implicit Reasoning? (2024)" description: "Yu tests this with synthetic addition/subtraction chains of 3–5 steps, where every… 100% accuracy. 11 references, cited by 9 papers here." authors: ["Yijiong Yu"] year: 2024 venue: "arXiv:2411.15862" source: "https://arxiv.org/abs/2411.15862" doi: "10.48550/arxiv.2411.15862" arxiv: "2411.15862" licence: "CC BY-SA 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by-sa/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/yu-2024-llms-really-think-step-step/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yu-2024-llms-really-think-step-step/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/11/) / Do LLMs Really Think Step-by-step In Implicit Reasoning? # Do LLMs Really Think Step-by-step In Implicit Reasoning? **Authors:** Yijiong Yu · **Year:** 2024 · **Venue:** arXiv:2411.15862 · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2411.15862) · [PDF](https://arxiv.org/pdf/2411.15862) · [DOI 10.48550/arxiv.2411.15862](https://doi.org/10.48550/arxiv.2411.15862) · [arXiv 2411.15862](https://arxiv.org/abs/2411.15862) · [OpenAlex](https://openalex.org/W4404986869) **Citations:** 1 citations (OpenAlex, 2026-10-14) · [9](https://md.research.elamur.ai/free/en/papers/yu-2024-llms-really-think-step-step/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Implicit chain-of-thought means answering multi-step problems without writing out the intermediate steps. It promises faster, cheaper inference, but it is unclear whether models reason internally the way explicit CoT does. Yu tests this with synthetic addition/subtraction chains of 3–5 steps, where every intermediate value is known. At every layer, the hidden state of the last prompt token is captured. A separate probe per layer then tries to recover each step's result. Two settings are compared: Qwen2.5-72B-Instruct told to answer directly, and a Mistral-7B model fine-tuned to internalize CoT (Deng et al., 2024). - **Prompted Qwen:** the probes reliably find only the input value and the final answer. Intermediate results are barely detectable, which suggests learned shortcuts rather than genuine stepwise computation. - **Trained Mistral:** it does compute steps one after another across layers. On longer chains, however, it runs out of layers before finishing. - **Format changes:** both models lose much accuracy when equations are reversed or values are divided by 10. The author concludes that implicit CoT is fragile and that explicit CoT remains necessary. ## Key points - **Data and probing setup:** - 2000 generated problems, all values kept in the range 0–19. - 1600 problems train the probes and 400 test them. - Each probe is a 1-layer MLP acting as a 20-class classifier, trained for 8 epochs per layer. - **Models:** - Qwen2.5-72B-Instruct, quantized to 4-bit, with its 80 layers averaged pairwise into 40. - mistral-7b-internal-CoT, trained on GSM8k, with 32 layers left uncompressed. - **Layer limit in Mistral:** the 2nd step emerges around layer 20 for 3-step problems and around layer 23 for 4-step ones. Steps 4–5 of 5-step problems are never reached. - **Implicit-CoT accuracy on original / reversed / scaled problems:** | Model | 3-step | 5-step | |---|---|---| | Mistral | 78.2 / 13.7 / 10.0 | 4.8 / 1.8 / 5.4 | | Qwen | 99.8 / 89.0 / 90.1 | 65.2 / 14.9 / 59.1 | - **Explicit CoT baseline:** Qwen2.5 models at 1.5b, 7b and 72b, prompted to reason step by step, reach close to 100% accuracy regardless of format or step count. The only limitation the author notes is that just arithmetic tasks were tested. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works 🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works ## Cite Cite as: Yijiong Yu. 2024. Do LLMs Really Think Step-by-step In Implicit Reasoning?. arXiv:2411.15862. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{yu2024do, title = {{Do LLMs Really Think Step-by-step In Implicit Reasoning?}}, author = {Yijiong Yu}, year = {2024}, doi = {10.48550/arxiv.2411.15862}, eprint = {2411.15862}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2411.15862} } ``` - [← previous Challenges to Evaluating the Generalization of Coreference Resolution Models: A Measurement Modeling Perspective](https://md.research.elamur.ai/free/en/papers/porada-2024-challenges-evaluating-generalization-coreference-resolution/) - [next → EgyBERT: A Large Language Model Pretrained on Egyptian Dialect Corpora](https://md.research.elamur.ai/free/en/papers/qarah-2024-egybert-large-language-model-pretrained/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/11/) ## References 11 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Zeyuan Allen-Zhu and Yuanzhi Li. [Physics of language models: Part 3.2, knowledge manipulation](https://md.research.elamur.ai/free/en/papers/allen-zhu-2023-physics-language-models-part-3-2/). Preprint, arxiv:2309.14402 [cs]. 2. Yuntian Deng, Yejin Choi, and Stuart Shieber. 2024. From explicit CoT to implicit CoT: Learning to internalize CoT step by step. Preprint, arxiv:2405.14838 [cs]. [find](https://scholar.google.com/scholar?q=Yuntian+Deng%2C+Yejin+Choi%2C+and+Stuart+Shieber%2E+2024%2E+From+explicit+CoT+to+implicit+CoT%3A+Learning+to+internalize+CoT+step+by+step%2E+Preprint%2C+arxiv%3A2405%2E14838+%5Bcs%5D%2E) 3. Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber. 2023. Implicit chain of thought reasoning via knowledge distillation. Preprint, arxiv:2311.01460. [find](https://scholar.google.com/scholar?q=Yuntian+Deng%2C+Kiran+Prasad%2C+Roland+Fernandez%2C+Paul+Smolensky%2C+Vishrav+Chaudhary%2C+and+Stuart+Shieber%2E+2023%2E+Implicit+chain+of+thought+reasoning+via+knowledge+distillation%2E+Preprint%2C+arxiv%3A2311%2E01460%2E) 4. Daniel Kahneman. 2011. Thinking, fast and slow. Farrar, Straus and Giroux. [find](https://scholar.google.com/scholar?q=Daniel+Kahneman%2E+2011%2E+Thinking%2C+fast+and+slow%2E+Farrar%2C+Straus+and+Giroux%2E) 5. Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, and Pengfei Liu. 2024. [O1 Replication Journey: A Strategic Progress Report – Part 1](https://md.research.elamur.ai/free/en/papers/qin-2024-o1-replication-journey-strategic-progress/). arXiv preprint. ArXiv:2410.18982. 6. Qwen Team. 2024. Qwen2.5: A party of foundation models! Section: blog. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+2024%2E+Qwen2%2E5%3A+A+party+of+foundation+models%21+Section%3A+blog%2E) 7. Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokked transformers are implicit reasoners: A mechanistic journey to the edge of generalization. Preprint, arxiv:2405.15071. [find](https://scholar.google.com/scholar?q=Boshi+Wang%2C+Xiang+Yue%2C+Yu+Su%2C+and+Huan+Sun%2E+Grokked+transformers+are+implicit+reasoners%3A+A+mechanistic+journey+to+the+edge+of+generalization%2E+Preprint%2C+arxiv%3A2405%2E15071%2E) 8. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processi… 9. Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning? Preprint, arxiv:2402.16837. [find](https://scholar.google.com/scholar?q=Sohee+Yang%2C+Elena+Gribovskaya%2C+Nora+Kassner%2C+Mor+Geva%2C+and+Sebastian+Riedel%2E+Do+large+language+models+latently+perform+multi%2Dhop+reasoning%3F+Preprint%2C+arxiv%3A2402%2E16837%2E) 10. Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. 2024. [Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process](https://md.research.elamur.ai/free/en/papers/ye-2024-physics-language-models-part-2/). 11. Zihan Yu, Liang He, Zhen Wu, Xinyu Dai, and Jiajun Chen. 2023. Towards Better Chain-of-Thought Prompting Strategies: A Survey. arXiv preprint. ArXiv:2310.04959. — [arXiv:2310.04959](https://arxiv.org/abs/2310.04959) [find](https://scholar.google.com/scholar?q=Zihan+Yu%2C+Liang+He%2C+Zhen+Wu%2C+Xinyu+Dai%2C+and+Jiajun+Chen%2E+2023%2E+Towards+Better+Chain%2Dof%2DThought+Prompting+Strategies%3A+A+Survey%2E+arXiv+preprint%2E+ArXiv%3A2310%2E04959%2E+%E2%80%94+%5BarXiv%3A2310%2E04959%5D%28https%3A%2F%2Farxiv%2Eorg%2F) *Updated 2 November 2026*