---
title: "Enhancing the Reasoning Capabilities of Small Language Models"
description: "The authors propose Solution Guidance (SG), in which the model only outlines what to do to solve a… 30 references, cited by 7 papers here."
authors: ["Jing Bi", "Yuting Wu", "Weiwei Xing", "Zhenjie Wei"]
year: 2025
venue: "Proceedings of the 31st International Conference on Computational Linguistics"
source: "https://aclanthology.org/2025.coling-main.609/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bi-2025-enhancing-reasoning-capabilities-small-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bi-2025-enhancing-reasoning-capabilities-small-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/10/) / Enhancing the Reasoning Capabilities of Small Language Models via Solution Guidance Fine-Tuning

# Enhancing the Reasoning Capabilities of Small Language Models via Solution Guidance Fine-Tuning

**Authors:** Jing Bi, Yuting Wu, Weiwei Xing and Zhenjie Wei · **Year:** 2025 · **Venue:** [Proceedings of the 31st International Conference on Computational Linguistics](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2025](https://md.research.elamur.ai/free/en/venues/coling/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.coling-main.609/) · [PDF](https://aclanthology.org/2025.coling-main.609.pdf) · [OpenAlex](https://openalex.org/W4405433048)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/bi-2025-enhancing-reasoning-capabilities-small-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Small language models (under 10 billion parameters) reason poorly with Chain-of-Thought (CoT). CoT fine-tuning helps, but it needs large annotated datasets. It also mixes logic with arithmetic, so one early mistake can carry through the remaining steps and produce noisy, wrong outputs. The authors propose Solution Guidance (SG), in which the model only outlines what to do to solve a problem, as a short ordered list of sub-goals, without performing any calculations. GPT-4o acts as a teacher and writes SG examples for questions drawn from GSM8K. One SLM is fine-tuned on these examples (SGFT) to produce guidance. The guidance and the original question are then given as a prompt to a second, unmodified SLM, which computes the answer. On math and common-sense benchmarks, this plug-and-play pipeline beats conventional CoT fine-tuning while keeping the models' general abilities. It needs only a small fraction of the training data and runs on one consumer-grade GPU, which makes it practical when resources are limited.

## Key points
- SG examples consist of 2 to 6 calculation-free steps generated by GPT-4o. The questions were sampled from a 2,000-question subset of the GSM8K training set.
- According to the authors, 1,000 SG examples beat 30,000 CoT examples, so SGFT needs roughly 3% of the data that CoT fine-tuning requires.
- In the ablation study, the best setup paired an SG-fine-tuned guidance model with an untrained answering model. Using a CoT-fine-tuned model as the answerer gave worse results than using the original model.
- During fine-tuning, training with context plus few-shot examples (up to 3-shot) worked best. Zero-shot training without context was consistently weaker.
- The method also works for common-sense questions, illustrated with a CommonsenseQA example. Code and data are available at github.com/BiJings/SGFT.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Beijing Jiaotong University](https://md.research.elamur.ai/free/en/universities/beijing-jiaotong-university/) — University, Beijing, China · 28 works · 1,754 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Jing Bi, Yuting Wu, Weiwei Xing and Zhenjie Wei. 2025. Enhancing the Reasoning Capabilities of Small Language Models via Solution Guidance Fine-Tuning. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{bi2025enhancing,
  title = {{Enhancing the Reasoning Capabilities of Small Language Models via Solution Guidance Fine-Tuning}},
  author = {Jing Bi, Yuting Wu, Weiwei Xing and Zhenjie Wei},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.609/}
}
```

- [← previous Empowering Bengali Education with AI: Solving Bengali Math Word Problems through Transformer Models](https://md.research.elamur.ai/free/en/papers/era-2025-empowering-bengali-education-ai-solving/)
- [next → EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checking](https://md.research.elamur.ai/free/en/papers/wei-2025-equibench-benchmarking-large-language-models/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/10/)

## References

30 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Chr… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
2. In our work, we aimed to enhance the model’s reasoning capabilities while preserving its original abilities, so we chose not to focus on smaller-scale models. Some researchers have found that even smaller models, when fine-tuned, can develop cer-Gawon Choi and Hyemin Ahn. 2024. [Can only llms tain ma…](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
3. Shizhe Diao, Pengcheng Wang, Yong Lin, Xiang Liu, and Tong Zhang. 2023. Active prompting with chain-of-thought for large language models. In Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shizhe+Diao%2C+Pengcheng+Wang%2C+Yong+Lin%2C+Xiang+Liu%2C+and+Tong+Zhang%2E+2023%2E+Active+prompting+with+chain%2Dof%2Dthought+for+large+language+models%2E+In+Annual+Meeting+of+the+Association+for+Computational+Linguis)
4. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. [A survey for in-context learning](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/). ArXiv, abs/2301.00234.
5. Yao Fu, Hao-Chun Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023. [Specializing smaller language models towards multi-step reasoning](https://md.research.elamur.ai/free/en/papers/fu-2023-specializing-smaller-language-models-towards/). ArXiv, abs/2301.12726.
6. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346– 361. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Daniel+Khashabi%2C+Elad+Segal%2C+Tushar+Khot%2C+Dan+Roth%2C+and+Jonathan+Berant%2E+2021%2E+Did+aristotle+use+a+laptop%3F+a+question+answering+benchmark+with+implicit+reasoning+strategies%2E+Transactions+of+)
7. Namgyu Ho, Laura Schmid, and Se-Young Yun. 2022. [Large language models are reasoning teachers](https://md.research.elamur.ai/free/en/papers/ho-2023-large-language-models-are-reasoning/). In Annual Meeting of the Association for Computational Linguistics.
8. J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). ArXiv, abs/2106.09685.
9. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). ArXiv, abs/2205.11916.
10. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson E. Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, John Kernion, Kamil.e Lukovsiut.e, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Sa… — [Measuring Faithfulness in Chain-of-Thought Reasoning](https://md.research.elamur.ai/free/en/papers/lanham-2023-measuring-faithfulness-chain-thought-reasoning/)
11. Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jingu Qian, Baolin Peng, Yi Mao, Wenhu Chen, and Xifeng Yan. 2022. [Explanations from large language models make small reasoners better](https://md.research.elamur.ai/free/en/papers/li-2022-explanations-large-language-models-make/). ArXiv, abs/2210.06726.
12. Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang. 2022. A survey of deep learning for mathematical reasoning. ArXiv, abs/2212.10535. [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Liang+Qiu%2C+Wenhao+Yu%2C+Sean+Welleck%2C+and+Kai%2DWei+Chang%2E+2022%2E+A+survey+of+deep+learning+for+mathematical+reasoning%2E+ArXiv%2C+abs%2F2212%2E10535%2E)
13. Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2022. [Teaching small language models to reason](https://md.research.elamur.ai/free/en/papers/magister-2023-teaching-small-language-models-reason/). ArXiv, abs/2212.08410.
14. Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and A. Kalyan. 2022. [Lila: A unified benchmark for mathematical reasoning](https://md.research.elamur.ai/free/en/papers/mishra-2022-lila-unified-benchmark-mathematical-reasoning/). ArXiv, abs/2210.17517.
15. Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. 2024. Lisa: Layer-wise importance sampling for memory-efficient large language model fine-tuning. ArXiv, abs/2403.17919. [find](https://scholar.google.com/scholar?q=Rui+Pan%2C+Xiang+Liu%2C+Shizhe+Diao%2C+Renjie+Pi%2C+Jipeng+Zhang%2C+Chi+Han%2C+and+Tong+Zhang%2E+2024%2E+Lisa%3A+Layer%2Dwise+importance+sampling+for+memory%2Defficient+large+language+model+fine%2Dtuning%2E+ArXiv%2C+abs%2F2403%2E179)
16. Arkil Patel, S. Bhattamishra, and Navin Goyal. 2021. [Are nlp models really able to solve simple math word problems?](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/) In North American Chapter of the Association for Computational Linguistics.
17. Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. 2022. [Reasoning with language model prompting: A survey](https://md.research.elamur.ai/free/en/papers/qiao-2022-reasoning-language-model-prompting-survey/). ArXiv, abs/2212.09597.
18. Subhro Roy and Dan Roth. 2016. [Solving general arithmetic word problems](https://md.research.elamur.ai/free/en/papers/roy-2015-solving-general-arithmetic-word-problems/). ArXiv, abs/1608.01413.
19. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. [Commonsenseqa: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). ArXiv, abs/1811.00937.
20. Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Bria… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
21. Miles Turpin, Julian Michael, Ethan Perez, and Sam Bowman. 2023. [Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting](https://md.research.elamur.ai/free/en/papers/turpin-2023-language-models-don-t-always/). ArXiv, abs/2305.04388.
22. J. Wang, Qiushi Sun, Nuo Chen, Xiang Lorraine Li, and Ming Gao. 2023. [Boosting language models reasoning with chain-of-knowledge prompting](https://md.research.elamur.ai/free/en/papers/wang-2024-boosting-language-models-reasoning-chain/). ArXiv, abs/2306.06427.
23. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Huai hsin Chi, and Denny Zhou. 2022. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). ArXiv, abs/2203.11171.
24. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. 2022. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). ArXiv, abs/2201.11903.
25. Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, E. Cambria, Xiaodong Liu, Jianfeng Gao, and Furu Wei. 2022. [Language models as inductive reasoners](https://md.research.elamur.ai/free/en/papers/yang-2024-language-models-as-inductive-reasoners/). ArXiv, abs/2212.10923.
26. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. ArXiv, abs/2305.10601. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L%2E+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+2023%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+ArXiv%2C+abs%2F2305%2E)
27. Team Glm Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Ming yue Liu, Minlie Huang, Peng Zhang, Qinka… — [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://md.research.elamur.ai/free/en/papers/glm-2024-chatglm-family-large-language-models/)
28. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alexander J. Smola. 2022. [Automatic chain of thought prompting in large language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-automatic-chain-thought-prompting-large/). ArXiv, abs/2210.03493.
29. Qihuang Zhong, Kang Wang, Ziyang Xu, Juhua Liu, Liang Ding, Bo Du, and Dacheng Tao. 2024. Achieving \>97% on gsm8k: Deeply understanding the problems makes llms better reasoners. ArXiv, abs/2404.14963. — [arXiv:2404.14963](https://arxiv.org/abs/2404.14963) [find](https://scholar.google.com/scholar?q=Qihuang+Zhong%2C+Kang+Wang%2C+Ziyang+Xu%2C+Juhua+Liu%2C+Liang+Ding%2C+Bo+Du%2C+and+Dacheng+Tao%2E+2024%2E+Achieving+%5C%3E97%25+on+gsm8k%3A+Deeply+understanding+the+problems+makes+llms+better+reasoners%2E+ArXiv%2C+abs%2F2404%2E14963)
30. Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. 2023. [A survey on model compression for large language models](https://md.research.elamur.ai/free/en/papers/zhu-2024-survey-model-compression-large-language/). ArXiv, abs/2308.07633.

*Updated 2 November 2026*