---
title: "Skill-Aware Data Selection and Fine-Tuning for Data-Efficient"
description: "The authors propose a skill-centric distillation framework for math reasoning built on a hierarchical skill tree taken from… 27 references."
authors: ["Lechen Zhang", "Yunxiang Zhang", "Wei Hu", "Lu Wang"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/2026.acl-short.49/"
doi: "10.18653/v1/2026.acl-short.49"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2026-skill-aware-data-selection-fine/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2026-skill-aware-data-selection-fine/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/50/) / Skill-Aware Data Selection and Fine-Tuning for Data-Efficient Reasoning Distillation

# Skill-Aware Data Selection and Fine-Tuning for Data-Efficient Reasoning Distillation

**Authors:** [Lechen Zhang](https://md.research.elamur.ai/free/en/authors/lechen-zhang/), [Yunxiang Zhang](https://md.research.elamur.ai/free/en/authors/yunxiang-zhang/), [Wei Hu](https://md.research.elamur.ai/free/en/authors/wei-hu/) and [Lu Wang](https://md.research.elamur.ai/free/en/authors/lu-wang/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-short.49/) · [PDF](https://aclanthology.org/2026.acl-short.49.pdf) · [DOI 10.18653/v1/2026.acl-short.49](https://doi.org/10.18653/v1/2026.acl-short.49) · [OpenAlex](https://openalex.org/W7166850477)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Distilling reasoning from large models such as DeepSeek-R1 into smaller LLMs usually relies on large SFT datasets in which every example counts the same. That approach ignores both which skills a problem requires and what the student model already knows. The authors propose a skill-centric distillation framework for math reasoning built on a hierarchical skill tree taken from Instruct-SkillMix. First, Qwen2.5-32B-Instruct labels each problem in a 100K OpenMathReasoning pool by walking the tree from the root down to the relevant leaf skills. Second, the student is evaluated on the pool, and training examples are drawn with probability inversely tied to the student's accuracy on each skill. Third, the full skill chain is inserted before each solution, so the model learns to name the skills it needs before it starts reasoning. With only 1K selected examples, this beats random selection on Qwen3-4B and Qwen3-8B. The gains concentrate on weaker skills, which makes per-skill performance more even. The work shows that careful, model-adaptive data selection can count for more than data volume.

## Key points
- **Gains over random 1K SFT:** +1.6 Avg@8 on Qwen3-4B (68.4 vs. 66.8) and +1.4 on Qwen3-8B (69.5 vs. 68.1). Evaluation used five benchmarks: AMC23, AIME2024, AIME2025, MATH L5 and OlympiadBench.
- **Full-corpus fine-tuning hurts:** training on all 100K examples drops accuracy below the base model (58.1 vs. 67.1 for Qwen3-4B).
- **Sampling ablation:** an accuracy exponent of T = 1.0 works well, and more aggressive settings saturate.
- **Skill-chain ablation:** the full chain (72.9) beats root-only (72.2) and leaf-only (72.7) skill labels.
- **Generalization (Appendix C):** the method also works with the EvalTree taxonomy and with R1-Distill-Llama-8B. It outperforms the LIMO, s1, Light-R1 and Select2Reason baselines. Code and data are released at github.com/orange0629/skill-data-selection.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Michigan](https://md.research.elamur.ai/free/en/universities/university-of-michigan/) — University, Ann Arbor, United States · 86 works · 15,493 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 297 works

## Cite

Cite as: Lechen Zhang, Yunxiang Zhang, Wei Hu and Lu Wang. 2026. Skill-Aware Data Selection and Fine-Tuning for Data-Efficient Reasoning Distillation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2026skill,
  title = {{Skill-Aware Data Selection and Fine-Tuning for Data-Efficient Reasoning Distillation}},
  author = {Lechen Zhang, Yunxiang Zhang, Wei Hu and Lu Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-short.49},
  url = {https://doi.org/10.18653/v1/2026.acl-short.49}
}
```

- [← previous Skill Extraction from Resumes and Job Offers across Six Languages](https://md.research.elamur.ai/free/en/papers/rodriguez-2026-skill-extraction-resumes-job-offers/)
- [next → SKILL-IR-Discourse: A Large, Annotated Corpus of Argumentation and Domain Discourse on International Relations](https://md.research.elamur.ai/free/en/papers/wolska-2026-skill-ir-discourse-large-annotated/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/50/)

## References

27 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mayee F Chen, Nicholas Roberts, Kush Bhatia, Jue WANG, Ce Zhang, Frederic Sala, and Christopher Re. 2023. Skill-it! a data-driven skills framework for understanding and training language models. In Thirty-seventh Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Mayee+F+Chen%2C+Nicholas+Roberts%2C+Kush+Bhatia%2C+Jue+WANG%2C+Ce+Zhang%2C+Frederic+Sala%2C+and+Christopher+Re%2E+2023%2E+Skill%2Dit%21+a+data%2Ddriven+skills+framework+for+understanding+and+training+language+models%2E+In+Th)
2. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 81 others. 2025. [Deepseek-r1: Incentivizing reasoning capability in llm…](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/)
3. Aniket Rajiv Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy P Lillicrap, Danilo Jimenez Rezende, Yoshua Bengio, Michael Curtis Mozer, and Sanjeev Arora. 2024. Metacognitive capabilities of LLMs: An exploration in mathematical problem solving. In The Thirty-eighth Annual… [find](https://scholar.google.com/scholar?q=Aniket+Rajiv+Didolkar%2C+Anirudh+Goyal%2C+Nan+Rosemary+Ke%2C+Siyuan+Guo%2C+Michal+Valko%2C+Timothy+P+Lillicrap%2C+Danilo+Jimenez+Rezende%2C+Yoshua+Bengio%2C+Michael+Curtis+Mozer%2C+and+Sanjeev+Arora%2E+2024%2E+Metacognitiv)
4. Robert Gagne. 1962. The acquisition of knowledge. Psychological Review, 69:355–365. [find](https://scholar.google.com/scholar?q=Robert+Gagne%2E+1962%2E+The+acquisition+of+knowledge%2E+Psychological+Review%2C+69%3A355%E2%80%93365%2E)
5. Shousheng Jia Haosheng Zou, Xiaowei Lv and Xiangzheng Zhang. 2024. 360-llama-factory. [find](https://scholar.google.com/scholar?q=Shousheng+Jia+Haosheng+Zou%2C+Xiaowei+Lv+and+Xiangzheng+Zhang%2E+2024%2E+360%2Dllama%2Dfactory%2E)
6. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. 2024. [OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems](https://md.research.elamur.ai/free/en/papers/he-2024-olympiadbench-challenging-benchmark-promoting-agi/). In Proce…
7. Yinghui He, Abhishek Panigrahi, Yong Lin, and Sanjeev Arora. 2026. STAT: Skill-targeted adaptive training. In The Fourteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yinghui+He%2C+Abhishek+Panigrahi%2C+Yong+Lin%2C+and+Sanjeev+Arora%2E+2026%2E+STAT%3A+Skill%2Dtargeted+adaptive+training%2E+In+The+Fourteenth+International+Conference+on+Learning+Representations%2E)
8. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). NeurIPS.
9. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). Preprint, arXiv:1503.02531.
10. Simran Kaur, Simon Park, Anirudh Goyal, and Sanjeev Arora. 2025. Instruct-skillmix: A powerful pipeline for LLM instruction tuning. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Simran+Kaur%2C+Simon+Park%2C+Anirudh+Goyal%2C+and+Sanjeev+Arora%2E+2025%2E+Instruct%2Dskillmix%3A+A+powerful+pipeline+for+LLM+instruction+tuning%2E+In+The+Thirteenth+International+Conference+on+Learning+Representatio)
11. Jiazheng Li, Lu Yu, Qing Cui, Zhiqiang Zhang, JUN ZHOU, Yanfang Ye, and Chuxu Zhang. 2025a. MASS: Mathematical data selection via skill graphs for pretraining large language models. In Forty-second International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Jiazheng+Li%2C+Lu+Yu%2C+Qing+Cui%2C+Zhiqiang+Zhang%2C+JUN+ZHOU%2C+Yanfang+Ye%2C+and+Chuxu+Zhang%2E+2025a%2E+MASS%3A+Mathematical+data+selection+via+skill+graphs+for+pretraining+large+language+models%2E+In+Forty%2Dsecond+In)
12. Yang Li, Youssef Emad, Karthik Padthe, Jack Lanchantin, Weizhe Yuan, Thao Nguyen, Jason Weston, Shang-Wen Li, Dong Wang, Ilia Kulikov, and Xian Li. 2025b. Naturalthoughts: Selecting and distilling reasoning traces for general reasoning tasks. Preprint, arXiv:2507.01921. — [arXiv:2507.01921](https://arxiv.org/abs/2507.01921) [find](https://scholar.google.com/scholar?q=Yang+Li%2C+Youssef+Emad%2C+Karthik+Padthe%2C+Jack+Lanchantin%2C+Weizhe+Yuan%2C+Thao+Nguyen%2C+Jason+Weston%2C+Shang%2DWen+Li%2C+Dong+Wang%2C+Ilia+Kulikov%2C+and+Xian+Li%2E+2025b%2E+Naturalthoughts%3A+Selecting+and+distilling+rea)
13. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open-Rev…
14. Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. 2025. Aimo-2 winning solution: Building state-of-the-art mathematical reasoning models with openmathreasoning dataset. arXiv preprint arXiv:2504.16891. [find](https://scholar.google.com/scholar?q=Ivan+Moshkov%2C+Darragh+Hanley%2C+Ivan+Sorokin%2C+Shubham+Toshniwal%2C+Christof+Henkel%2C+Benedikt+Schifferer%2C+Wei+Du%2C+and+Igor+Gitman%2E+2025%2E+Aimo%2D2+winning+solution%3A+Building+state%2Dof%2Dthe%2Dart+mathematical+reas)
15. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. [s1: Simple test-time scaling](https://md.research.elamur.ai/free/en/papers/muennighoff-2025-s1-simple-test-time-scaling/). Preprint, arXiv:2501.19393.
16. Shousheng Jia, and Xiangzheng Zhang. 2025. Lightr1: Curriculum SFT, DPO and RL for long COT from scratch and beyond. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), pages 318– 327, Vienna, Austria. Association for Computational L… [find](https://scholar.google.com/scholar?q=Shousheng+Jia%2C+and+Xiangzheng+Zhang%2E+2025%2E+Lightr1%3A+Curriculum+SFT%2C+DPO+and+RL+for+long+COT+from+scratch+and+beyond%2E+In+Proceedings+of+the+63rd+Annual+Meeting+of+the+Association+for+Computational+Ling)
17. Richard T. White. 1973. Research into learning hierarchies. Review of Educational Research, 43(3):361– 375. [find](https://scholar.google.com/scholar?q=Richard+T%2E+White%2E+1973%2E+Research+into+learning+hierarchies%2E+Review+of+Educational+Research%2C+43%283%29%3A361%E2%80%93+375%2E)
18. Cehao Yang, Xueyuan Lin, Xiaojun Wu, Chengjin Xu, Xuhui Jiang, Honghao Liu, Hui Xiong, and Jian Guo. 2025a. [Select2reason: Efficient instruction-tuning data selection for long-cot reasoning](https://md.research.elamur.ai/free/en/papers/yang-2026-select2reason-efficient-instruction-tuning-data/). Preprint, arXiv:2505.17266.
19. Zheyuan Yang, Lyuhao Chen, Arman Cohan, and Yilun Zhao. 2025b. Table-r1: Inference-time scaling for table reasoning. Preprint, arXiv:2505.23621. — [arXiv:2505.23621](https://arxiv.org/abs/2505.23621) [find](https://scholar.google.com/scholar?q=Zheyuan+Yang%2C+Lyuhao+Chen%2C+Arman+Cohan%2C+and+Yilun+Zhao%2E+2025b%2E+Table%2Dr1%3A+Inference%2Dtime+scaling+for+table+reasoning%2E+Preprint%2C+arXiv%3A2505%2E23621%2E+%E2%80%94+%5BarXiv%3A2505%2E23621%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2505%2E23621%29)
20. Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. 2025. [LIMO: Less is more for reasoning](https://md.research.elamur.ai/free/en/papers/ye-2025-limo-less-more-reasoning/). In Second Conference on Language Modeling.
21. Zhiyuan Zeng, Yizhong Wang, Hannaneh Hajishirzi, and Pang Wei Koh. 2025. Evaltree: Profiling language model weaknesses via hierarchical capability trees. In Second Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Zhiyuan+Zeng%2C+Yizhong+Wang%2C+Hannaneh+Hajishirzi%2C+and+Pang+Wei+Koh%2E+2025%2E+Evaltree%3A+Profiling+language+model+weaknesses+via+hierarchical+capability+trees%2E+In+Second+Conference+on+Language+Modeling%2E)
22. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. [Llamafactory: Unified efficient fine-tuning of 100+ language models](https://md.research.elamur.ai/free/en/papers/zheng-2024-llamafactory-unified-efficient-fine-tuning/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations),…
23. Qwen Team. 2024. Qwen2.5: A party of foundation models. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+2024%2E+Qwen2%2E5%3A+A+party+of+foundation+models%2E)
24. Qwen Team. 2025. Qwen3: Think Deeper, Act Faster \| Qwen. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+2025%2E+Qwen3%3A+Think+Deeper%2C+Act+Faster+%5C%7C+Qwen%2E)
25. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. [Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter](https://md.research.elamur.ai/free/en/papers/sanh-2019-distilbert-distilled-version-bert-smaller/). Preprint, arXiv:1910.01108.
26. Lin Sun, Guangxiang Zhao, Xiaoqi Jian, Yuhan Wu, Weihong Lin, Yongfu Zhu, Change Jia, Linglin Zhang, Jinzhu Wu, Junfeng Ran, Sai er Hu, Zihan Jiang, Junting Zhou, Wenrui Liu, Bin Cui, Tong Yang, and Xiangzheng Zhang. 2025. [Tinyr1-32b-preview: Boosting accuracy with branch-merge distillation](https://md.research.elamur.ai/free/en/papers/sun-2025-tinyr1-32b-preview-boosting-accuracy/). Preprin…
27. Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, [find](https://scholar.google.com/scholar?q=Liang+Wen%2C+Yunke+Cai%2C+Fenrui+Xiao%2C+Xin+He%2C+Qi+An%2C+Zhenyu+Duan%2C+Yimin+Du%2C+Junchen+Liu%2C+Lifu+Tang%2C+Xiaowei+Lv%2C+Haosheng+Zou%2C+Yongchao+Deng%2C)

*Updated 2 November 2026*