---
title: "MARD — paper by Yang et al. (2026) — summary and ideas"
description: "The authors first compare base Qwen models with their DeepSeek-R1-distilled versions (1.5B, 7B, 14B, 32B), measuring… 40 references."
authors: ["Wenqi Yang", "Jianjun Li", "Zhibo Zhang", "Mingqian Ding", "Yushen Fang"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1749/"
doi: "10.18653/v1/2026.acl-long.1749"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yang-2026-mard-module-aware-reasoning-distillation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2026-mard-module-aware-reasoning-distillation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/40/) / MARD: Module-Aware Reasoning Distillation for Language Models with Adaptive Supervision

# MARD: Module-Aware Reasoning Distillation for Language Models with Adaptive Supervision

**Authors:** Wenqi Yang, [Jianjun Li](https://md.research.elamur.ai/free/en/authors/jianjun-li/), [Zhibo Zhang](https://md.research.elamur.ai/free/en/authors/zhibo-zhang/), Mingqian Ding and Yushen Fang · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1749/) · [PDF](https://aclanthology.org/2026.acl-long.1749.pdf) · [DOI 10.18653/v1/2026.acl-long.1749](https://doi.org/10.18653/v1/2026.acl-long.1749) · [OpenAlex](https://openalex.org/W7166849129)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Compact language models struggle with multi-step reasoning. Existing reasoning distillation methods supervise every Transformer component equally, although components contribute unevenly. The authors first compare base Qwen models with their DeepSeek-R1-distilled versions (1.5B, 7B, 14B, 32B), measuring layer-wise ℓ2-norm changes in each projection matrix. Changes cluster in FFN projections and the attention output projection (o_proj), mainly in deeper layers. FFN dominates in small models, while o_proj gains weight as scale grows. Building on this, MARD freezes the student backbone and adds lightweight bottleneck adapters only at these residual write-back points, in layers chosen by teacher saliency. Training mixes token-level and module-level distillation using teacher reasoning traces prepared offline. A meta-learned controller rebalances attention versus FFN supervision according to problem difficulty. On mathematical reasoning benchmarks, MARD reportedly beats strong fine-tuning and reasoning-aware distillation baselines with little overhead. Ablations support both targeted placement and adaptive weighting. The work matters for cheaper reasoning models and gives interpretable hints about where reasoning sits inside Transformers.

## Key points
- **Localization analysis:** Base Qwen models at 1.5B/7B/14B/32B were compared with their DeepSeek-R1 distilled counterparts. Q/K/V projections changed far less than FFN (up/gate/down_proj) and o_proj, and shallow layers changed little compared with deep ones.
- **Layer-placement ablation:** Three adapter placements were compared: first three layers, last three layers, and saliency-selected layers. Shallow placement helped little or inconsistently, and the last-three-layer setting beat it. Saliency-guided selection was best with the same adapter count. Activation heatmaps over 20 GSM8K problems show reasoning spread over a band of middle-to-upper layers.
- **Attention-submodule ablation:** On GSM8K with a 1.5B student, Q, K, V and O placements were each tested with and without FFN adapters. FFN supervision proved essential, and O+FFN gave the largest gains.
- **Cross-architecture robustness:** Appendix F reports a Llama teacher → Qwen student setting, with only minor degradation.
- **Limitations and availability:** The evaluation focuses mostly on math. Bi-level meta-optimization adds compute and memory cost, and the method needs access to a strong teacher. Code is at github.com/wqyang24/MARD. The provided extract omits the main results tables, so exact benchmark scores are not available here.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Huazhong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/huazhong-university-of-science-and-technology/) — University, Wuhan, China · 59 works · 8,310 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 297 works

## Cite

Cite as: Wenqi Yang, Jianjun Li, Zhibo Zhang, Mingqian Ding and Yushen Fang. 2026. MARD: Module-Aware Reasoning Distillation for Language Models with Adaptive Supervision. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yang2026mard,
  title = {{MARD: Module-Aware Reasoning Distillation for Language Models with Adaptive Supervision}},
  author = {Wenqi Yang, Jianjun Li, Zhibo Zhang, Mingqian Ding and Yushen Fang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1749},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1749}
}
```

- [← previous MARCH: Evaluating the Intersection of Ambiguity Interpretation and Multi-hop Inference](https://md.research.elamur.ai/free/en/papers/park-2026-march-evaluating-intersection-ambiguity-interpretation/)
- [next → Market-Bench: Benchmarking Large Language Models on Economic and Trade Competition](https://md.research.elamur.ai/free/en/papers/zheng-2026-market-bench-benchmarking-large-language/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/40/)

## References

40 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Shivam Adarsh, Kumar Shridhar, Nicholas Monath, and Mrinmaya Sachan. 2025. Siked: Self-guided iterative knowledge distillation for mathematical reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 9868–9880. [find](https://scholar.google.com/scholar?q=Shivam+Adarsh%2C+Kumar+Shridhar%2C+Nicholas+Monath%2C+and+Mrinmaya+Sachan%2E+2025%2E+Siked%3A+Self%2Dguided+iterative+knowledge+distillation+for+mathematical+reasoning%2E+In+Findings+of+the+Association+for+Computatio)
2. Jing Bi, Yuting Wu, Weiwei Xing, and Zhenjie Wei. 2025. [Enhancing the reasoning capabilities of small language models via solution guidance fine-tuning](https://md.research.elamur.ai/free/en/papers/bi-2025-enhancing-reasoning-capabilities-small-language/). In Proceedings of the 31st International Conference on Computational Linguistics, pages 9074–9084.
3. Giannis Chatziveroglou, Richard Yun, and Maura Kelleher. 2025. Exploring llm reasoning through controlled prompt variations. arXiv preprint arXiv:2504.02111. — [arXiv:2504.02111](https://arxiv.org/abs/2504.02111) [find](https://scholar.google.com/scholar?q=Giannis+Chatziveroglou%2C+Richard+Yun%2C+and+Maura+Kelleher%2E+2025%2E+Exploring+llm+reasoning+through+controlled+prompt+variations%2E+arXiv+preprint+arXiv%3A2504%2E02111%2E+%E2%80%94+%5BarXiv%3A2504%2E02111%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
4. Xinghao Chen, Zhijing Sun, Guo Wenjin, Miaoran Zhang, Yanjun Chen, Yirong Sun, Hui Su, Yijie Pan, Dietrich Klakow, Wenjie Li, and 1 others. 2025. Unveiling the key factors for distilling chain-of-thought reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 15094–… [find](https://scholar.google.com/scholar?q=Xinghao+Chen%2C+Zhijing+Sun%2C+Guo+Wenjin%2C+Miaoran+Zhang%2C+Yanjun+Chen%2C+Yirong+Sun%2C+Hui+Su%2C+Yijie+Pan%2C+Dietrich+Klakow%2C+Wenjie+Li%2C+and+1+others%2E+2025%2E+Unveiling+the+key+factors+for+distilling+chain%2Dof%2Dthou)
5. Fengxiang Cheng, Haoxuan Li, Fenrong Liu, Robert van Rooij, Kun Zhang, and Zhouchen Lin. 2025. Empowering llms with logical reasoning: A comprehensive survey. arXiv preprint arXiv:2502.15652. [find](https://scholar.google.com/scholar?q=Fengxiang+Cheng%2C+Haoxuan+Li%2C+Fenrong+Liu%2C+Robert+van+Rooij%2C+Kun+Zhang%2C+and+Zhouchen+Lin%2E+2025%2E+Empowering+llms+with+logical+reasoning%3A+A+comprehensive+survey%2E+arXiv+preprint+arXiv%3A2502%2E15652%2E)
6. Yihong Dong, Yuchen Liu, Xue Jiang, Bin Gu, Zhi Jin, and Ge Li. 2025. Rethinking repetition problems of llms in code generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 965–985. [find](https://scholar.google.com/scholar?q=Yihong+Dong%2C+Yuchen+Liu%2C+Xue+Jiang%2C+Bin+Gu%2C+Zhi+Jin%2C+and+Ge+Li%2E+2025%2E+Rethinking+repetition+problems+of+llms+in+code+generation%2E+In+Proceedings+of+the+63rd+Annual+Meeting+of+the+Association+for+Comput)
7. Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuohang Wang, and Jingjing Liu. 2020. [Hierarchical graph network for multi-hop question answering](https://md.research.elamur.ai/free/en/papers/fang-2020-hierarchical-graph-network-multi-hop/). In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 8823–8838.
8. Tao Feng, Yicheng Li, Li Chenglin, Hao Chen, Fei Yu, and Yin Zhang. 2024. [Teaching small language models reasoning through counterfactual distillation](https://md.research.elamur.ai/free/en/papers/feng-2024-teaching-small-language-models-reasoning/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5831–5842.
9. Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023. [Specializing smaller language models towards multi-step reasoning](https://md.research.elamur.ai/free/en/papers/fu-2023-specializing-smaller-language-models-towards/). In International Conference on Machine Learning, pages 10421–10430. PMLR.
10. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, and 1 others. 2025. Deepseekr1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638. [find](https://scholar.google.com/scholar?q=Daya+Guo%2C+Dejian+Yang%2C+Haowei+Zhang%2C+Junxiao+Song%2C+Peiyi+Wang%2C+Qihao+Zhu%2C+Runxin+Xu%2C+Ruoyu+Zhang%2C+Shirong+Ma%2C+Xiao+Bi%2C+and+1+others%2E+2025%2E+Deepseekr1+incentivizes+reasoning+in+llms+through+reinforceme)
11. Pei Guo, Wangjie You, Juntao Li, Yan Bowen, and Min Zhang. 2024. Exploring reversal mathematical reasoning ability for large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 13671–13685. [find](https://scholar.google.com/scholar?q=Pei+Guo%2C+Wangjie+You%2C+Juntao+Li%2C+Yan+Bowen%2C+and+Min+Zhang%2E+2024%2E+Exploring+reversal+mathematical+reasoning+ability+for+large+language+models%2E+In+Findings+of+the+Association+for+Computational+Linguisti)
12. Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, Xinyuan Wang, Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, and 1 others. 2024. Llm reasoners: New evaluation, library, and analysis of step-by-step reasoning with large language models. arXiv preprint arXiv:2404.05221. — [arXiv:2404.05221](https://arxiv.org/abs/2404.05221) [find](https://scholar.google.com/scholar?q=Shibo+Hao%2C+Yi+Gu%2C+Haotian+Luo%2C+Tianyang+Liu%2C+Xiyan+Shao%2C+Xinyuan+Wang%2C+Shuhua+Xie%2C+Haodi+Ma%2C+Adithya+Samavedhi%2C+Qiyue+Gao%2C+and+1+others%2E+2024%2E+Llm+reasoners%3A+New+evaluation%2C+library%2C+and+analysis+of+s)
13. Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, and Roberta Raileanu. 2024. [Glore: When, where, and how to improve llm reasoning via global and local refinements](https://md.research.elamur.ai/free/en/papers/havrilla-2024-glore-where-improve-llm-reasoning/). arXiv preprint arXiv:2402.10963.
14. Xingwei He, Yeyun Gong, A-Long Jin, Weizhen Qi, Hang Zhang, Jian Jiao, Bartuer Zhou, Biao Cheng, Sm Yiu, and Nan Duan. 2022. Metric-guided distillation: Distilling knowledge from the metric to ranker and retriever for generative commonsense reasoning. In Proceedings of the 2022 Conference on Empiric… [find](https://scholar.google.com/scholar?q=Xingwei+He%2C+Yeyun+Gong%2C+A%2DLong+Jin%2C+Weizhen+Qi%2C+Hang+Zhang%2C+Jian+Jiao%2C+Bartuer+Zhou%2C+Biao+Cheng%2C+Sm+Yiu%2C+and+Nan+Duan%2E+2022%2E+Metric%2Dguided+distillation%3A+Distilling+knowledge+from+the+metric+to+ranker+)
15. Lanxiang Hu, Tajana Rosing, and Hao Zhang. 2025. Trimllm: Progressive layer dropping for domain-specific llms. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 667–681. [find](https://scholar.google.com/scholar?q=Lanxiang+Hu%2C+Tajana+Rosing%2C+and+Hao+Zhang%2E+2025%2E+Trimllm%3A+Progressive+layer+dropping+for+domain%2Dspecific+llms%2E+In+Proceedings+of+the+63rd+Annual+Meeting+of+the+Association+for+Computational+Linguistic)
16. Jie Huang and Kevin Chen-Chuan Chang. 2023. [Towards reasoning in large language models: A survey](https://md.research.elamur.ai/free/en/papers/huang-2023-reasoning-large-language-models-survey/). In Findings of the association for computational linguistics: ACL 2023, pages 1049–1065.
17. Jinghan Jia, Hadi Reisizadeh, Chongyu Fan, Nathalie Baracaldo, Mingyi Hong, and Sijia Liu. 2025. Epic: Towards lossless speedup for reasoning training through edge-preserving cot condensation. arXiv preprint arXiv:2506.04205. — [arXiv:2506.04205](https://arxiv.org/abs/2506.04205) [find](https://scholar.google.com/scholar?q=Jinghan+Jia%2C+Hadi+Reisizadeh%2C+Chongyu+Fan%2C+Nathalie+Baracaldo%2C+Mingyi+Hong%2C+and+Sijia+Liu%2E+2025%2E+Epic%3A+Towards+lossless+speedup+for+reasoning+training+through+edge%2Dpreserving+cot+condensation%2E+arXiv+p)
18. Minki Kang, Seanie Lee, Jinheon Baek, Kenji Kawaguchi, and Sung Ju Hwang. 2023. [Knowledge-augmented reasoning distillation for small language models in knowledge-intensive tasks](https://md.research.elamur.ai/free/en/papers/kang-2023-knowledge-augmented-reasoning-distillation-small/). Advances in Neural Information Processing Systems, 36:48573– 48602.
19. Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Eric Tang, Sumanth Hegde, Kourosh Hakhamaneshi, Shishir G Patil, Matei Zaharia, and 1 others. 2025a. Llms can easily learn to reason from demonstrations structure, not content, is what matters! arXiv preprint arXiv:2502.07374. [find](https://scholar.google.com/scholar?q=Dacheng+Li%2C+Shiyi+Cao%2C+Tyler+Griggs%2C+Shu+Liu%2C+Xiangxi+Mo%2C+Eric+Tang%2C+Sumanth+Hegde%2C+Kourosh+Hakhamaneshi%2C+Shishir+G+Patil%2C+Matei+Zaharia%2C+and+1+others%2E+2025a%2E+Llms+can+easily+learn+to+reason+from+demo)
20. Yiwei Li, Peiwen Yuan, Shaoxiong Feng, Boyuan Pan, Bin Sun, Xinglin Wang, Heda Wang, and Kan Li. 2024a. Turning dust into gold: Distilling complex reasoning capabilities from llms by leveraging negative data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18591– 1… [find](https://scholar.google.com/scholar?q=Yiwei+Li%2C+Peiwen+Yuan%2C+Shaoxiong+Feng%2C+Boyuan+Pan%2C+Bin+Sun%2C+Xinglin+Wang%2C+Heda+Wang%2C+and+Kan+Li%2E+2024a%2E+Turning+dust+into+gold%3A+Distilling+complex+reasoning+capabilities+from+llms+by+leveraging+negati)
21. Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, and Radha Poovendran. 2025b. [Small models struggle to learn from strong reasoners](https://md.research.elamur.ai/free/en/papers/li-2025-small-models-struggle-learn-strong/). arXiv preprint arXiv:2502.12143.
22. Zhiming Li, Yushi Cao, Xiufeng Xu, Junzhe Jiang, Xu Liu, Yon Shin Teo, Shang-Wei Lin, and Yang Liu. 2024b. Llms for relational reasoning: How far are we? In Proceedings of the 1st International Workshop on Large Language Models for Code, pages 119–126. [find](https://scholar.google.com/scholar?q=Zhiming+Li%2C+Yushi+Cao%2C+Xiufeng+Xu%2C+Junzhe+Jiang%2C+Xu+Liu%2C+Yon+Shin+Teo%2C+Shang%2DWei+Lin%2C+and+Yang+Liu%2E+2024b%2E+Llms+for+relational+reasoning%3A+How+far+are+we%3F+In+Proceedings+of+the+1st+International+Worksh)
23. Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao, Wenwei Zhang, Songyang Zhang, and Kai Chen. 2025a. Are your llms capable of stable reasoning? In Findings of the Association for Computational Linguistics: ACL 2025, pages 17594–17632. [find](https://scholar.google.com/scholar?q=Junnan+Liu%2C+Hongwei+Liu%2C+Linchen+Xiao%2C+Ziyi+Wang%2C+Kuikun+Liu%2C+Songyang+Gao%2C+Wenwei+Zhang%2C+Songyang+Zhang%2C+and+Kai+Chen%2E+2025a%2E+Are+your+llms+capable+of+stable+reasoning%3F+In+Findings+of+the+Association)
24. Zitao Liu, Ying Zheng, Zhibo Yin, Jiahao Chen, Tianqiao Liu, Mi Tian, and Weiqi Luo. 2025b. Arith-meticgpt: empowering small-size large language models with advanced arithmetic skills. Machine Learning, 114(1):24. [find](https://scholar.google.com/scholar?q=Zitao+Liu%2C+Ying+Zheng%2C+Zhibo+Yin%2C+Jiahao+Chen%2C+Tianqiao+Liu%2C+Mi+Tian%2C+and+Weiqi+Luo%2E+2025b%2E+Arith%2Dmeticgpt%3A+empowering+small%2Dsize+large+language+models+with+advanced+arithmetic+skills%2E+Machine+Learnin)
25. Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2023. [Teaching small language models to reason](https://md.research.elamur.ai/free/en/papers/magister-2023-teaching-small-language-models-reason/). In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 2: short papers), pages 1773–1781.
26. Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, and 1 others. 2023. [Orca 2: Teaching small language models how to reason](https://md.research.elamur.ai/free/en/papers/mitra-2023-orca-2-teaching-small-language/). arXiv preprint arXiv:2311.11045.
27. Xuefei Ning, Zifu Wang, Shiyao Li, Zinan Lin, Peiran Yao, Tianyu Fu, Matthew Blaschko, Guohao Dai, Huazhong Yang, and Yu Wang. 2024. Can llms learn by teaching for better reasoning? a preliminary study. Advances in Neural Information Processing Systems, 37:71188–71239. [find](https://scholar.google.com/scholar?q=Xuefei+Ning%2C+Zifu+Wang%2C+Shiyao+Li%2C+Zinan+Lin%2C+Peiran+Yao%2C+Tianyu+Fu%2C+Matthew+Blaschko%2C+Guohao+Dai%2C+Huazhong+Yang%2C+and+Yu+Wang%2E+2024%2E+Can+llms+learn+by+teaching+for+better+reasoning%3F+a+preliminary+stud)
28. Chen Qian, Dongrui Liu, Haochen Wen, Zhen Bai, Yong Liu, and Jing Shao. 2025. Demystifying reasoning dynamics with mutual information: Thinking tokens are information peaks in llm reasoning. arXiv preprint arXiv:2506.02867. — [arXiv:2506.02867](https://arxiv.org/abs/2506.02867) [find](https://scholar.google.com/scholar?q=Chen+Qian%2C+Dongrui+Liu%2C+Haochen+Wen%2C+Zhen+Bai%2C+Yong+Liu%2C+and+Jing+Shao%2E+2025%2E+Demystifying+reasoning+dynamics+with+mutual+information%3A+Thinking+tokens+are+information+peaks+in+llm+reasoning%2E+arXiv+pre)
29. Leonardo Ranaldi, Marco Valentino, and Andre Freitas. 2025. [Improving chain-of-thought reasoning via quasi-symbolic abstractions](https://md.research.elamur.ai/free/en/papers/ranaldi-2025-improving-chain-thought-reasoning-quasi/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 17222–17240.
30. Jie Shao and Jianxin Wu. 2025. in the large language models? arXiv:2505.20993. — [arXiv:2505.20993](https://arxiv.org/abs/2505.20993) [find](https://scholar.google.com/scholar?q=Jie+Shao+and+Jianxin+Wu%2E+2025%2E+in+the+large+language+models%3F+arXiv%3A2505%2E20993%2E+%E2%80%94+%5BarXiv%3A2505%2E20993%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2505%2E20993%29)
31. Who reasons arXiv preprint Aman Sharma and Paras Chopra. 2025. Think just enough: Sequence-level entropy as a confidence signal for llm reasoning. arXiv preprint arXiv:2510.08146. [find](https://scholar.google.com/scholar?q=Who+reasons+arXiv+preprint+Aman+Sharma+and+Paras+Chopra%2E+2025%2E+Think+just+enough%3A+Sequence%2Dlevel+entropy+as+a+confidence+signal+for+llm+reasoning%2E+arXiv+preprint+arXiv%3A2510%2E08146%2E)
32. Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2023. [Distilling reasoning capabilities into smaller language models](https://md.research.elamur.ai/free/en/papers/shridhar-2023-distilling-reasoning-capabilities-into-smaller/). Findings of the Association for Computational Linguistics: ACL 2023, pages 7059–7073.
33. Yijun Tian, Yikun Han, Xiusi Chen, Wei Wang, and Nitesh V Chawla. 2025. Beyond answers: Transferring reasoning capabilities to smaller llms using multi-teacher knowledge distillation. In Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, pages 251– 260. [find](https://scholar.google.com/scholar?q=Yijun+Tian%2C+Yikun+Han%2C+Xiusi+Chen%2C+Wei+Wang%2C+and+Nitesh+V+Chawla%2E+2025%2E+Beyond+answers%3A+Transferring+reasoning+capabilities+to+smaller+llms+using+multi%2Dteacher+knowledge+distillation%2E+In+Proceedings+o)
34. Chengyu Wang, Taolin Zhang, Richang Hong, and Jun Huang. 2025. A short survey on small reasoning models: Training, inference, applications and research directions. arXiv preprint arXiv:2504.09100. — [arXiv:2504.09100](https://arxiv.org/abs/2504.09100) [find](https://scholar.google.com/scholar?q=Chengyu+Wang%2C+Taolin+Zhang%2C+Richang+Hong%2C+and+Jun+Huang%2E+2025%2E+A+short+survey+on+small+reasoning+models%3A+Training%2C+inference%2C+applications+and+research+directions%2E+arXiv+preprint+arXiv%3A2504%2E09100%2E+%E2%80%94+%5B)
35. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long p… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2023%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+In+Proce)
36. Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, and 1 others. 2025a. [Towards large reasoning models: A survey of reinforced reasoning with large language models](https://md.research.elamur.ai/free/en/papers/xu-2025-towards-large-reasoning-models-survey/). arXiv preprint arXiv:2501.09686.
37. Shuyao Xu, Cheng Peng, Jiangxuan Long, Weidi Xu, Wei Chu, and Yuan Qi. 2025b. [Harnessing negative signals: Reinforcement distillation from teacher data for llm reasoning](https://md.research.elamur.ai/free/en/papers/xu-2026-harnessing-negative-signals-reinforcement-distillation/). arXiv preprint arXiv:2505.24850.
38. Bohao Yang, Chen Tang, Kun Zhao, Chenghao Xiao, and Chenghua Lin. 2024. Effective distillation of table-based reasoning ability from llms. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 5538–5550. [find](https://scholar.google.com/scholar?q=Bohao+Yang%2C+Chen+Tang%2C+Kun+Zhao%2C+Chenghao+Xiao%2C+and+Chenghua+Lin%2E+2024%2E+Effective+distillation+of+table%2Dbased+reasoning+ability+from+llms%2E+In+Proceedings+of+the+2024+Joint+International+Conference+on+)
39. Ruiwen Zhou, Wenyue Hua, Liangming Pan, Sitao Cheng, Xiaobao Wu, En Yu, and William Yang Wang. 2025. [Rulearena: A benchmark for rule-guided reasoning with llms in real-world scenarios](https://md.research.elamur.ai/free/en/papers/zhou-2025-rulearena-benchmark-rule-guided-reasoning/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),…
40. Xuekai Zhu, Biqing Qi, Kaiyan Zhang, Xinwei Long, Zhouhan Lin, and Bowen Zhou. 2024. Pad: Program-aided distillation can teach small models reasoning better than chain-of-thought fine-tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Xuekai+Zhu%2C+Biqing+Qi%2C+Kaiyan+Zhang%2C+Xinwei+Long%2C+Zhouhan+Lin%2C+and+Bowen+Zhou%2E+2024%2E+Pad%3A+Program%2Daided+distillation+can+teach+small+models+reasoning+better+than+chain%2Dof%2Dthought+fine%2Dtuning%2E+In+Proce)

*Updated 2 November 2026*