---
title: "Enhancing Long-Chain Reasoning Distillation through"
description: "The authors propose ORION (errOr-aware self-ReflectION), which lets the student adapt the teacher's… 45 references, cited by 1 paper here."
authors: ["Zhuoyang Wu", "Xinze Li", "Zhenghao Liu", "Yukun Yan", "Zhiyuan Liu", "Minghe Yu", "Cheng Yang", "Yu Gu", "Ge Yu", "Maosong Sun"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.72/"
doi: "10.18653/v1/2026.findings-acl.72"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wu-2026-enhancing-long-chain-reasoning-distillation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wu-2026-enhancing-long-chain-reasoning-distillation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/12/) / Enhancing Long-Chain Reasoning Distillation through Error-Aware Self-Reflection

# Enhancing Long-Chain Reasoning Distillation through Error-Aware Self-Reflection

**Authors:** [Zhuoyang Wu](https://md.research.elamur.ai/free/en/authors/zhuoyang-wu/), [Xinze Li](https://md.research.elamur.ai/free/en/authors/xinze-li/), [Zhenghao Liu](https://md.research.elamur.ai/free/en/authors/zhenghao-liu/), [Yukun Yan](https://md.research.elamur.ai/free/en/authors/yukun-yan/), [Zhiyuan Liu](https://md.research.elamur.ai/free/en/authors/zhiyuan-liu/), [Minghe Yu](https://md.research.elamur.ai/free/en/authors/minghe-yu/), [Cheng Yang](https://md.research.elamur.ai/free/en/authors/cheng-hong-yang/), [Yu Gu](https://md.research.elamur.ai/free/en/authors/yu-gu/), [Ge Yu](https://md.research.elamur.ai/free/en/authors/ge-yu/) and [Maosong Sun](https://md.research.elamur.ai/free/en/authors/maosong-sun/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.72/) · [PDF](https://aclanthology.org/2026.findings-acl.72.pdf) · [DOI 10.18653/v1/2026.findings-acl.72](https://doi.org/10.18653/v1/2026.findings-acl.72) · [OpenAlex](https://openalex.org/W4415272049)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/wu-2026-enhancing-long-chain-reasoning-distillation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Distilling long Chain-of-Thought (CoT) traces from strong reasoning models into small models often works poorly. The teacher's lengthy trajectories exceed what the student can absorb, and corrections written by the teacher can add noise. The authors propose ORION (errOr-aware self-ReflectION), which lets the student adapt the teacher's reasoning to its own level. The pipeline has three steps:

1. The student samples answers at several decoding temperatures to surface its typical mistakes.
2. Given these mistakes and the teacher's long CoT, the student reflects and rewrites the CoT into a trajectory suited to itself.
3. SFT data are built by pairing each question and its sampled wrong solutions as input with the refined CoT as the target.

On several math reasoning benchmarks, ORION beats every baseline by more than 2%. Training entropy is lower and steadier, outputs are shorter and less redundant, and the generated CoTs are rated as more coherent. The work suggests that fitting supervision to the student's capacity matters as much as the teacher's quality.

## Key points
- Teacher traces come from a Reasoning Language Model such as DeepSeek-R1. ORION is compared against vanilla SFT on human-annotated solutions and against direct long-form CoT distillation.
- ORION reaches the lowest perplexity of the compared models when vanilla LLMs act as judges. In GPT-4 preference judgments it earns roughly double the win rate of the alternatives.
- Error analysis used 500 wrongly answered test items with Qwen3-8B. Reasoning errors were the largest category, about 46.5% in the figure. ORION fixed around 41% of them, the best correction rate among the compared models.
- An ablation without the sampled student errors (ORION w/o Solution Error) corrects fewer reasoning and understanding mistakes. This confirms that the self-generated errors contribute.
- Code is released at github.com/NEUIR/ORION. The authors note two limitations: refinement by an LLM may itself introduce errors, and sampling multiple student trajectories adds compute cost.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 333 works · 87,019 citations
- [Beijing University of Posts and Telecommunications](https://md.research.elamur.ai/free/en/universities/beijing-university-of-posts-and-telecommunications/) — University, Beijing, China · 57 works · 32,843 citations
- [Northeastern University](https://md.research.elamur.ai/free/en/universities/northeastern-university-cn/) — University, Shenyang, China · 42 works · 1,940 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Zhuoyang Wu, Xinze Li, Zhenghao Liu, Yukun Yan, Zhiyuan Liu, Minghe Yu, Cheng Yang, Yu Gu, Ge Yu and Maosong Sun. 2026. Enhancing Long-Chain Reasoning Distillation through Error-Aware Self-Reflection. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wu2026enhancing,
  title = {{Enhancing Long-Chain Reasoning Distillation through Error-Aware Self-Reflection}},
  author = {Zhuoyang Wu, Xinze Li, Zhenghao Liu, Yukun Yan, Zhiyuan Liu, Minghe Yu, Cheng Yang, Yu Gu, Ge Yu and Maosong Sun},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.72},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.72}
}
```

- [← previous Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks](https://md.research.elamur.ai/free/en/papers/yamaguchi-2026-enhancing-linguistic-competence-language-models/)
- [next → Enhancing the Safety of Medical Vision-Language Models by Synthetic Demonstrations](https://md.research.elamur.ai/free/en/papers/xue-2026-enhancing-safety-medical-vision-language/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/12/)

## References

45 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 2025. AIME. aime problems and solutions. Accessed: September 23, 2025. 2025. AMC12. amc 12 problems and solutions. Accessed: September 23, 2025. [find](https://scholar.google.com/scholar?q=2025%2E+AIME%2E+aime+problems+and+solutions%2E+Accessed%3A+September+23%2C+2025%2E+2025%2E+AMC12%2E+amc+12+problems+and+solutions%2E+Accessed%3A+September+23%2C+2025%2E)
2. Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. 2023. Learning from mistakes makes llm better reasoner. ArXiv preprint, abs/2310.20689. [find](https://scholar.google.com/scholar?q=Shengnan+An%2C+Zexiong+Ma%2C+Zeqi+Lin%2C+Nanning+Zheng%2C+Jian%2DGuang+Lou%2C+and+Weizhu+Chen%2E+2023%2E+Learning+from+mistakes+makes+llm+better+reasoner%2E+ArXiv+preprint%2C+abs%2F2310%2E20689%2E)
3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
4. Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. 2024. [Do not think that much for 2+ 3=?](https://md.research.elamur.ai/free/en/papers/chen-2024-think-that-much-2-3/) on the overthinking of o1-like llms. ArXiv preprint, abs/2412.21187.
5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). ArXiv preprint, abs/2110.14168.
6. DeepSeek-AI, Daya Guo, Dejian Yang, and Haowei Zhang. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). ArXiv preprint, abs/2510.12948.
7. Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023. [Specializing smaller language models towards multi-step reasoning](https://md.research.elamur.ai/free/en/papers/fu-2023-specializing-smaller-language-models-towards/). In Proceedings of the ICML, pages 10421–10430.
8. Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, and et al. Yibo Miao. 2024. [Omni-math: A universal olympiad level mathematic benchmark for large language models](https://md.research.elamur.ai/free/en/papers/gao-2024-omni-math-universal-olympiad-level/). ArXiv preprint, abs/2410.07985.
9. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. PAL: program-aided language models. In Proceedings of the ICML, pages 10764– 10799. [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Pengfei+Liu%2C+Yiming+Yang%2C+Jamie+Callan%2C+and+Graham+Neubig%2E+2023%2E+PAL%3A+program%2Daided+language+models%2E+In+Proceedings+of+the+ICML%2C+pages+10764%E2%80%93+10799%2E)
10. Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob J. Hyndman, and Pablo Montero-Manso. 2021. Monash time series forecasting archive. ArXiv preprint, abs/2105.06643. — [arXiv:2105.06643](https://arxiv.org/abs/2105.06643) [find](https://scholar.google.com/scholar?q=Rakshitha+Godahewa%2C+Christoph+Bergmeir%2C+Geoffrey+I%2E+Webb%2C+Rob+J%2E+Hyndman%2C+and+Pablo+Montero%2DManso%2E+2021%2E+Monash+time+series+forecasting+archive%2E+ArXiv+preprint%2C+abs%2F2105%2E06643%2E+%E2%80%94+%5BarXiv%3A2105%2E06643%5D%28ht)
11. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and Abhishek Kadian. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). ArXiv preprint, abs/2407.21783.
12. Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. [The false promise of imitating proprietary llms](https://md.research.elamur.ai/free/en/papers/gudibande-2023-false-promise-imitating-proprietary-llms/). ArXiv preprint, abs/2305.15717.
13. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). ArXiv preprint, abs/2103.03874.
14. Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. [Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes](https://md.research.elamur.ai/free/en/papers/hsieh-2023-distilling-step-step-outperforming-larger/). In Proceedings of ACL Findings, pages 8003–…
15. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In Proceedings of ICLR.
16. Hugging Face. 2025. Open r1: A fully open reproduction of deepseek-r1. [find](https://scholar.google.com/scholar?q=Hugging+Face%2E+2025%2E+Open+r1%3A+A+fully+open+reproduction+of+deepseek%2Dr1%2E)
17. Kushal Jain, Piyushi Goyal, and Kumar Shridhar. 2025. Undo: Understanding distillation as optimization. ArXiv preprint, abs/2504.02521. [find](https://scholar.google.com/scholar?q=Kushal+Jain%2C+Piyushi+Goyal%2C+and+Kumar+Shridhar%2E+2025%2E+Undo%3A+Understanding+distillation+as+optimization%2E+ArXiv+preprint%2C+abs%2F2504%2E02521%2E)
18. Chengpeng Li, Zheng Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang, and Chang Zhou. 2023. Query and response augmentation cannot help out-of-domain math reasoning generalization. ArXiv preprint, abs/2310.05506. [find](https://scholar.google.com/scholar?q=Chengpeng+Li%2C+Zheng+Yuan%2C+Guanting+Dong%2C+Keming+Lu%2C+Jiancan+Wu%2C+Chuanqi+Tan%2C+Xiang+Wang%2C+and+Chang+Zhou%2E+2023%2E+Query+and+response+augmentation+cannot+help+out%2Dof%2Ddomain+math+reasoning+generalization%2E+)
19. Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Eric Tang, Sumanth Hegde, Kourosh Hakhamaneshi, Shishir G Patil, Matei Zaharia, et al. 2025a. Llms can easily learn to reason from demonstrations structure, not content, is what matters! ArXiv preprint, abs/2502.07374. [find](https://scholar.google.com/scholar?q=Dacheng+Li%2C+Shiyi+Cao%2C+Tyler+Griggs%2C+Shu+Liu%2C+Xiangxi+Mo%2C+Eric+Tang%2C+Sumanth+Hegde%2C+Kourosh+Hakhamaneshi%2C+Shishir+G+Patil%2C+Matei+Zaharia%2C+et+al%2E+2025a%2E+Llms+can+easily+learn+to+reason+from+demonstrati)
20. Xiaochuan Li, Zichun Yu, and Chenyan Xiong. 2024a. Montessori-instruct: Generate influential training data tailored for student learning. ArXiv preprint, abs/2410.14208. — [arXiv:2410.14208](https://arxiv.org/abs/2410.14208) [find](https://scholar.google.com/scholar?q=Xiaochuan+Li%2C+Zichun+Yu%2C+and+Chenyan+Xiong%2E+2024a%2E+Montessori%2Dinstruct%3A+Generate+influential+training+data+tailored+for+student+learning%2E+ArXiv+preprint%2C+abs%2F2410%2E14208%2E+%E2%80%94+%5BarXiv%3A2410%2E14208%5D%28https%3A%2F%2Fa)
21. Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, and Radha Poovendran. 2025b. Small models struggle to learn from strong reasoners. ArXiv preprint, abs/2502.12143. [find](https://scholar.google.com/scholar?q=Yuetai+Li%2C+Xiang+Yue%2C+Zhangchen+Xu%2C+Fengqing+Jiang%2C+Luyao+Niu%2C+Bill+Yuchen+Lin%2C+Bhaskar+Ramasubramanian%2C+and+Radha+Poovendran%2E+2025b%2E+Small+models+struggle+to+learn+from+strong+reasoners%2E+ArXiv+prepri)
22. Zhuochun Li, Yuelyu Ji, Rui Meng, and Daqing He. 2024b. Learning from committee: Reasoning distillation from a mixture of teachers with peer-review. ArXiv preprint, abs/2410.03663. [find](https://scholar.google.com/scholar?q=Zhuochun+Li%2C+Yuelyu+Ji%2C+Rui+Meng%2C+and+Daqing+He%2E+2024b%2E+Learning+from+committee%3A+Reasoning+distillation+from+a+mixture+of+teachers+with+peer%2Dreview%2E+ArXiv+preprint%2C+abs%2F2410%2E03663%2E)
23. Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao, Wenwei Zhang, Songyang Zhang, and Kai Chen. 2024. Are your llms capable of stable reasoning? ArXiv preprint, abs/2412.13147. [find](https://scholar.google.com/scholar?q=Junnan+Liu%2C+Hongwei+Liu%2C+Linchen+Xiao%2C+Ziyi+Wang%2C+Kuikun+Liu%2C+Songyang+Gao%2C+Wenwei+Zhang%2C+Songyang+Zhang%2C+and+Kai+Chen%2E+2024%2E+Are+your+llms+capable+of+stable+reasoning%3F+ArXiv+preprint%2C+abs%2F2412%2E13147%2E)
24. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023a. [Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/). ArXiv preprint, abs/2308.09583.
25. Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023b. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. ArXiv preprint, abs/2308.08747. [find](https://scholar.google.com/scholar?q=Yun+Luo%2C+Zhen+Yang%2C+Fandong+Meng%2C+Yafu+Li%2C+Jie+Zhou%2C+and+Yue+Zhang%2E+2023b%2E+An+empirical+study+of+catastrophic+forgetting+in+large+language+models+during+continual+fine%2Dtuning%2E+ArXiv+preprint%2C+abs%2F2308)
26. OpenAI, Josh Achiam, Steven Adler, and Sandhini Agarwal. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). ArXiv preprint, abs/2303.08774.
27. Zhuoshi Pan, Yu Li, Honglin Lin, Qizhi Pei, Zinan Tang, Wei Wu, Chenlin Ming, H Vicky Zhao, Conghui He, and Lijun Wu. 2025. Lemma: Learning from errors for mathematical advancement in llms. ArXiv preprint, abs/2503.17439. [find](https://scholar.google.com/scholar?q=Zhuoshi+Pan%2C+Yu+Li%2C+Honglin+Lin%2C+Qizhi+Pei%2C+Zinan+Tang%2C+Wei+Wu%2C+Chenlin+Ming%2C+H+Vicky+Zhao%2C+Conghui+He%2C+and+Lijun+Wu%2E+2025%2E+Lemma%3A+Learning+from+errors+for+mathematical+advancement+in+llms%2E+ArXiv+prep)
28. Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. 2023. [Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages](https://md.research.elamur.ai/free/en/papers/qin-2023-cross-lingual-prompting-improving-zero/). In Proceedings of EMNLP, pages 2695–2709.
29. Yongqi Tong, Dawei Li, Sizhe Wang, Yujia Wang, Fei Teng, and Jingbo Shang. 2024. [Can llms learn from previous mistakes?](https://md.research.elamur.ai/free/en/papers/tong-2024-can-llms-learn-previous-mistakes/) investigating llms’ errors to boost for reasoning. ArXiv preprint, abs/2403.20046.
30. Renxi Wang, Haonan Li, Xudong Han, Yixuan Zhang, and Timothy Baldwin. 2024a. Learning from failure: Integrating negative examples when fine-tuning large language models as agents. ArXiv preprint, abs/2402.11651. — [arXiv:2402.11651](https://arxiv.org/abs/2402.11651) [find](https://scholar.google.com/scholar?q=Renxi+Wang%2C+Haonan+Li%2C+Xudong+Han%2C+Yixuan+Zhang%2C+and+Timothy+Baldwin%2E+2024a%2E+Learning+from+failure%3A+Integrating+negative+examples+when+fine%2Dtuning+large+language+models+as+agents%2E+ArXiv+preprint%2C+abs%2F)
31. Yanbo Wang, Yongcan Yu, Jian Liang, and Ran He. 2025. A comprehensive survey on trustworthiness in reasoning with large language models. ArXiv preprint, abs/2509.03871. [find](https://scholar.google.com/scholar?q=Yanbo+Wang%2C+Yongcan+Yu%2C+Jian+Liang%2C+and+Ran+He%2E+2025%2E+A+comprehensive+survey+on+trustworthiness+in+reasoning+with+large+language+models%2E+ArXiv+preprint%2C+abs%2F2509%2E03871%2E)
32. Zihao Wang, Anji Liu, Haowei Lin, Jiaqi Li, Xiaojian Ma, and Yitao Liang. 2024b. [Rat: Retrieval augmented thoughts elicit context-aware reasoning in long-horizon generation](https://md.research.elamur.ai/free/en/papers/wang-2024-rat-retrieval-augmented-thoughts-elicit/). ArXiv preprint, abs/2403.05313.
33. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Proceedings of NeurIPS.
34. Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. 2024. Qurating: Selecting high-quality data for training language models. ArXiv preprint, abs/2402.09739. [find](https://scholar.google.com/scholar?q=Alexander+Wettig%2C+Aatmik+Gupta%2C+Saumya+Malik%2C+and+Danqi+Chen%2E+2024%2E+Qurating%3A+Selecting+high%2Dquality+data+for+training+language+models%2E+ArXiv+preprint%2C+abs%2F2402%2E09739%2E)
35. An Yang, Anfeng Li, Baosong Yang, and et al. Beichen Zhang. 2025a. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). ArXiv preprint, abs/2505.09388.
36. An Yang, Baosong Yang, Beichen Zhang, and Binyuan Hui. 2024a. Qwen2.5 technical report. ArXiv preprint, abs/2412.15115. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+and+Binyuan+Hui%2E+2024a%2E+Qwen2%2E5+technical+report%2E+ArXiv+preprint%2C+abs%2F2412%2E15115%2E)
37. Ling Yang, Zhaochen Yu, Tianjun Zhang, Minkai Xu, Joseph E Gonzalez, Bin Cui, and Shuicheng Yan. 2024b. Supercorrect: Advancing small llm reasoning with thought template distillation and self-correction. ArXiv preprint, abs/2410.09008. — [arXiv:2410.09008](https://arxiv.org/abs/2410.09008) [find](https://scholar.google.com/scholar?q=Ling+Yang%2C+Zhaochen+Yu%2C+Tianjun+Zhang%2C+Minkai+Xu%2C+Joseph+E+Gonzalez%2C+Bin+Cui%2C+and+Shuicheng+Yan%2E+2024b%2E+Supercorrect%3A+Advancing+small+llm+reasoning+with+thought+template+distillation+and+self%2Dcorrecti)
38. Ling Yang, Zhaochen Yu, Tianjun Zhang, Minkai Xu, Joseph E. Gonzalez, Bin Cui, and Shuicheng Yan. 2025b. Supercorrect: Advancing small LLM reasoning with thought template distillation and self-correction. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Ling+Yang%2C+Zhaochen+Yu%2C+Tianjun+Zhang%2C+Minkai+Xu%2C+Joseph+E%2E+Gonzalez%2C+Bin+Cui%2C+and+Shuicheng+Yan%2E+2025b%2E+Supercorrect%3A+Advancing+small+LLM+reasoning+with+thought+template+distillation+and+self%2Dcorrect)
39. Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. 2025. [Limo: Less is more for reasoning](https://md.research.elamur.ai/free/en/papers/ye-2025-limo-less-more-reasoning/). ArXiv preprint, abs/2502.03387.
40. Huifeng Yin, Yu Zhao, Minghao Wu, Xuanfan Ni, Bo Zeng, Hao Wang, Tianqi Shi, Liangying Shao, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. 2025. [Marco-o1 v2: Towards widening the distillation bottleneck for reasoning models](https://md.research.elamur.ai/free/en/papers/yin-2025-marco-o1-v2-widening/). ArXiv preprint, abs/2503.01461.
41. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, and Xi Victoria Lin. 2022. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). ArXiv preprint, abs/2205.01068.
42. Yongheng Zhang, Qiguang Chen, Jingxuan Zhou, Peng Wang, Jiasheng Si, Jin Wang, Wenpeng Lu, and Libo Qin. 2024. Wrong-of-thought: An integrated reasoning framework with multi-perspective verification and wrong information. In Proceedings of EMNLP Findings, pages 6644–6653. [find](https://scholar.google.com/scholar?q=Yongheng+Zhang%2C+Qiguang+Chen%2C+Jingxuan+Zhou%2C+Peng+Wang%2C+Jiasheng+Si%2C+Jin+Wang%2C+Wenpeng+Lu%2C+and+Libo+Qin%2E+2024%2E+Wrong%2Dof%2Dthought%3A+An+integrated+reasoning+framework+with+multi%2Dperspective+verification+a)
43. Wangchunshu Zhou, Canwen Xu, and Julian McAuley. 2021. Bert learns to teach: Knowledge distillation with meta learning. ArXiv preprint, abs/2106.04570. [find](https://scholar.google.com/scholar?q=Wangchunshu+Zhou%2C+Canwen+Xu%2C+and+Julian+McAuley%2E+2021%2E+Bert+learns+to+teach%3A+Knowledge+distillation+with+meta+learning%2E+ArXiv+preprint%2C+abs%2F2106%2E04570%2E)
44. Dataset OpenR1-Math-220k (2025) GSM-Hard (2023) MATH500 (2021) AIME24 (2024) AMC23 (2024) [find](https://scholar.google.com/scholar?q=Dataset+OpenR1%2DMath%2D220k+%282025%29+GSM%2DHard+%282023%29+MATH500+%282021%29+AIME24+%282024%29+AMC23+%282024%29)
45. Train 10,000 -- -- Test -1,319 500 30 40 Models Qwen2.5-7B Qwen3-8B Llama3.1-8B Correct Num 30,219 27,135 51,896 Ratio 0.61 0.71 0.60 *Table 4: Refinement Statistics for Different Student Models.* *Table 3: Data Statistics.* [find](https://scholar.google.com/scholar?q=Train+10%2C000+%2D%2D+%2D%2D+Test+%2D1%2C319+500+30+40+Models+Qwen2%2E5%2D7B+Qwen3%2D8B+Llama3%2E1%2D8B+Correct+Num+30%2C219+27%2C135+51%2C896+Ratio+0%2E61+0%2E71+0%2E60+%2ATable+4%3A+Refinement+Statistics+for+Different+Student+Models%2E%2A+%2ATa)

*Updated 1 November 2026*