---
title: "Calibrated Progressive Distillation: Co-Designing Curriculum and"
description: "The paper addresses white-box knowledge distillation (KD) of LLMs. 47 references. Among them: 0."
authors: ["Mengxiang Zhang", "Lingyuan Liu"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.335/"
doi: "10.18653/v1/2026.findings-acl.335"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2026-calibrated-progressive-distillation-co-designing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2026-calibrated-progressive-distillation-co-designing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/25/) / Calibrated Progressive Distillation: Co-Designing Curriculum and Target Mixing for Knowledge Distillation of Large Language Models

# Calibrated Progressive Distillation: Co-Designing Curriculum and Target Mixing for Knowledge Distillation of Large Language Models

**Authors:** [Mengxiang Zhang](https://md.research.elamur.ai/free/en/authors/mengxiang-zhang/) and [Lingyuan Liu](https://md.research.elamur.ai/free/en/authors/lingyuan-liu/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.335/) · [PDF](https://aclanthology.org/2026.findings-acl.335.pdf) · [DOI 10.18653/v1/2026.findings-acl.335](https://doi.org/10.18653/v1/2026.findings-acl.335) · [OpenAlex](https://openalex.org/W7166860707)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper addresses white-box knowledge distillation (KD) of LLMs. In this setting, the teacher–student capacity gap causes mode averaging or collapse. Existing remedies are curriculum learning and blending teacher and student distributions into the target, but they are typically tuned separately. The authors argue that ordering data uniformly by sample count gives an uneven difficulty trajectory, and that heuristic mixing ratios are disconnected from the data.

Their framework, Calibrated Progressive Distillation (CPD), scores each example by the untrained student's cross-entropy loss. Samples are ranked easy-to-hard into a Difficulty Profile, and each epoch's subset is chosen so that average difficulty rises evenly. In parallel, the student's weight in the distillation target and the temperature grow linearly across epochs. Supervision therefore moves from teacher-dominated toward student-aware signals.

Theoretical analysis shows bounded gradients and an implicit shift of emphasis from easy to hard samples. Experiments cover Qwen2.5, GPT-2 and OpenLLaMA2 on instruction following, math and code. CPD beats strong KD baselines and cuts runtime by over 10%, making LLM compression cheaper and more stable.

## Key points
- CPD is compared with KLD, RKL, on-policy GKD, SRKL, DA-KD and ABKD on five instruction-following benchmarks. The compression settings are Qwen2.5 1.5B→0.5B and OpenLLaMA2 7B→3B.
- Building the profile for Dolly takes a one-off forward pass of about 16 minutes on one A100. Training then needs 17.7% fewer iterations, and the total of 254 minutes is 87.5% of the baseline's 290 minutes.
- Advantages are largest in low-resource settings with small data ratios. CPD also holds up across different teacher–student capacity gaps.
- Limitation: token-level cross-entropy is a weak difficulty signal for math reasoning and code generation, where gains are limited. The authors suggest execution-based feedback such as test outcomes or answer checks.
- The Difficulty Profile is static. Per-epoch subsets still overlap by more than 80% with dynamically re-profiled ones, and dynamic profiling gave negligible gains at more than double the cost.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Hong Kong](https://md.research.elamur.ai/free/en/universities/university-of-hong-kong/) — University, Pok Fu Lam, China · 118 works · 13,496 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 297 works

## Cite

Cite as: Mengxiang Zhang and Lingyuan Liu. 2026. Calibrated Progressive Distillation: Co-Designing Curriculum and Target Mixing for Knowledge Distillation of Large Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2026calibrated,
  title = {{Calibrated Progressive Distillation: Co-Designing Curriculum and Target Mixing for Knowledge Distillation of Large Language Models}},
  author = {Mengxiang Zhang and Lingyuan Liu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.335},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.335}
}
```

- [← previous CAKT-Agent: A Context-Adaptive Knowledge Tracing Agent Based on MoE](https://md.research.elamur.ai/free/en/papers/long-2026-cakt-agent-context/)
- [next → Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference](https://md.research.elamur.ai/free/en/papers/zhouxuwen-2026-calibrated-speculative-decoding-frequency-guided/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/25/)

## References

47 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. 2024. [On-policy distillation of language models: Learning from self-generated mistakes](https://md.research.elamur.ai/free/en/papers/agarwal-2023-policy-distillation-language-models-learning/). In The Twelfth International Conference on Learning Representations.
2. Abi Aryan, Aakash Kumar Nain, Andrew McMahon, Lucas Augusto Meyer, and Harpreet Singh Sahota. 2023. The costly dilemma: generalization, evaluation and cost-optimal deployment of large language models. arXiv preprint arXiv:2308.08061. — [arXiv:2308.08061](https://arxiv.org/abs/2308.08061) [find](https://scholar.google.com/scholar?q=Abi+Aryan%2C+Aakash+Kumar+Nain%2C+Andrew+McMahon%2C+Lucas+Augusto+Meyer%2C+and+Harpreet+Singh+Sahota%2E+2023%2E+The+costly+dilemma%3A+generalization%2C+evaluation+and+cost%2Doptimal+deployment+of+large+language+models%2E)
3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732.
4. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. [Curriculum learning](https://md.research.elamur.ai/free/en/papers/bengio-2009-curriculum-learning/). In Proceedings of the 26th annual international conference on machine learning, pages 41–48.
5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
6. Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See [https://vicuna.lmsys](https://vicuna.lmsys). org (accessed 14 April 2023), 2(3):6. [↗](https://vicuna.lmsys)
7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
8. Together Computer. 2023. Redpajama-data: An open source recipe to reproduce llama training dataset. [find](https://scholar.google.com/scholar?q=Together+Computer%2E+2023%2E+Redpajama%2Ddata%3A+An+open+source+recipe+to+reproduce+llama+training+dataset%2E)
9. Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm. [find](https://scholar.google.com/scholar?q=Mike+Conover%2C+Matt+Hayes%2C+Ankit+Mathur%2C+Jianwei+Xie%2C+Jun+Wan%2C+Sam+Shah%2C+Ali+Ghodsi%2C+Patrick+Wendell%2C+Matei+Zaharia%2C+and+Reynold+Xin%2E+2023%2E+Free+dolly%3A+Introducing+the+world%E2%80%99s+first+truly+open+instruct)
10. Shen Gao, Zhengliang Shi, Minghang Zhu, Bowen Fang, Xin Xin, Pengjie Ren, Zhumin Chen, Jun Ma, and Zhaochun Ren. 2024. Confucius: Iterative tool learning from introspection feedback by easy-to-difficult curriculum. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18… [find](https://scholar.google.com/scholar?q=Shen+Gao%2C+Zhengliang+Shi%2C+Minghang+Zhu%2C+Bowen+Fang%2C+Xin+Xin%2C+Pengjie+Ren%2C+Zhumin+Chen%2C+Jun+Ma%2C+and+Zhaochun+Ren%2E+2024%2E+Confucius%3A+Iterative+tool+learning+from+introspection+feedback+by+easy%2Dto%2Ddifficu)
11. Xinyang Geng and Hao Liu. 2023. Openllama: An open reproduction of llama. [find](https://scholar.google.com/scholar?q=Xinyang+Geng+and+Hao+Liu%2E+2023%2E+Openllama%3A+An+open+reproduction+of+llama%2E)
12. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023. [Minillm: Knowledge distillation of large language models](https://md.research.elamur.ai/free/en/papers/gu-2023-minillm-policy-distillation-large-language/). arXiv preprint arXiv:2306.08543.
13. Changyi He, Yifu Ding, Jinyang Guo, Ruihao Gong, Haotong Qin, and Xianglong Liu. 2025. Da-kd: Difficulty-aware knowledge distillation for efficient large language models. In Forty-second International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Changyi+He%2C+Yifu+Ding%2C+Jinyang+Guo%2C+Ruihao+Gong%2C+Haotong+Qin%2C+and+Xianglong+Liu%2E+2025%2E+Da%2Dkd%3A+Difficulty%2Daware+knowledge+distillation+for+efficient+large+language+models%2E+In+Forty%2Dsecond+International)
14. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874.
15. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568. [find](https://scholar.google.com/scholar?q=Ziyang+Luo%2C+Can+Xu%2C+Pu+Zhao%2C+Qingfeng+Sun%2C+Xiubo+Geng%2C+Wenxiang+Hu%2C+Chongyang+Tao%2C+Jing+Ma%2C+Qingwei+Lin%2C+and+Daxin+Jiang%2E+2023%2E+Wizardcoder%3A+Empowering+code+large+language+models+with+evol%2Dinstruct%2E+a)
16. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). arXiv preprint arXiv:1503.02531.
17. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
18. Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022. [Unnatural instructions: Tuning language models with (almost) no human labor](https://md.research.elamur.ai/free/en/papers/honovich-2023-unnatural-instructions-tuning-language-models/). arXiv preprint arXiv:2212.09689.
19. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). ICLR, 1(2):3.
20. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. — [Qwen2.5-Coder Technical Report](https://md.research.elamur.ai/free/en/papers/hui-2024-qwen2-5-coder-technical-report/)
21. Jongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding, Luming Liang, Ilya Zharkov, and Se-Young Yun. 2025. [Distillm-2: A contrastive approach boosts the distillation of llms](https://md.research.elamur.ai/free/en/papers/ko-2025-distillm-2-contrastive-approach-boosts/). arXiv preprint arXiv:2503.07067.
22. Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. 2024. [Distillm: Towards streamlined distillation for large language models](https://md.research.elamur.ai/free/en/papers/ko-2024-distillm-towards-streamlined-distillation-large/). arXiv preprint arXiv:2402.03898.
23. Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. 2024. [Superfiltering: Weak-to-strong data filtering for fast instruction-tuning](https://md.research.elamur.ai/free/en/papers/li-2024-superfiltering-weak-strong-data-filtering/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page…
24. Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang. 2023. Curriculum temperature for knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1504–1512. [find](https://scholar.google.com/scholar?q=Zheng+Li%2C+Xiang+Li%2C+Lingfeng+Yang%2C+Borui+Zhao%2C+Renjie+Song%2C+Lei+Luo%2C+Jun+Li%2C+and+Jian+Yang%2E+2023%2E+Curriculum+temperature+for+knowledge+distillation%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial)
25. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+et+al%2E+2024%2E+Deepseek%2Dv3+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E19437%2E)
26. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. [Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/). Advances in Neural Information Processing Systems, 36:21558–21572.
27. Lingyuan Liu and Mengxiang Zhang. 2025. Being strong progressively! enhancing knowledge distillation of large language models through a curriculum learning framework. arXiv preprint arXiv:2506.05695. [find](https://scholar.google.com/scholar?q=Lingyuan+Liu+and+Mengxiang+Zhang%2E+2025%2E+Being+strong+progressively%21+enhancing+knowledge+distillation+of+large+language+models+through+a+curriculum+learning+framework%2E+arXiv+preprint+arXiv%3A2506%2E05695%2E)
28. Ayan Sengupta, Shantanu Dixit, Md Shad Akhtar, and Tanmoy Chakraborty. 2023. A good learner can teach better: Teacher-student collaborative knowledge distillation. In Proceedings of the The Twelfth International Conference on Learning Representations, Virtual Event, pages 25–29. [find](https://scholar.google.com/scholar?q=Ayan+Sengupta%2C+Shantanu+Dixit%2C+Md+Shad+Akhtar%2C+and+Tanmoy+Chakraborty%2E+2023%2E+A+good+learner+can+teach+better%3A+Teacher%2Dstudent+collaborative+knowledge+distillation%2E+In+Proceedings+of+the+The+Twelfth+In)
29. Makoto Shing, Kou Misaki, Han Bao, Sho Yokoi, and Takuya Akiba. 2025. Taid: Temporally adaptive interpolated distillation for efficient knowledge transfer in language models. arXiv preprint arXiv:2501.16937. — [arXiv:2501.16937](https://arxiv.org/abs/2501.16937) [find](https://scholar.google.com/scholar?q=Makoto+Shing%2C+Kou+Misaki%2C+Han+Bao%2C+Sho+Yokoi%2C+and+Takuya+Akiba%2E+2025%2E+Taid%3A+Temporally+adaptive+interpolated+distillation+for+efficient+knowledge+transfer+in+language+models%2E+arXiv+preprint+arXiv%3A2501)
30. Qwen Team. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2407.10671. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+2024%2E+Qwen2%2E+5+technical+report%2E+arXiv+preprint+arXiv%3A2407%2E10671%2E)
31. Ren Jie Tee and Mengmi Zhang. 2023. Integrating curricula with replays: Its effects on continual learning. In Proceedings of the AAAI Symposium Series, volume 1, pages 109–116. [find](https://scholar.google.com/scholar?q=Ren+Jie+Tee+and+Mengmi+Zhang%2E+2023%2E+Integrating+curricula+with+replays%3A+Its+effects+on+continual+learning%2E+In+Proceedings+of+the+AAAI+Symposium+Series%2C+volume+1%2C+pages+109%E2%80%93116%2E)
32. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
33. Guanghui Wang, Zhiyong Yang, Zitai Wang, Shi Wang, Qianqian Xu, and Qingming Huang. 2025. Abkd: Pursuing a proper allocation of the probability mass in knowledge distillation via alpha-beta-divergence. arXiv preprint arXiv:2505.04560. — [arXiv:2505.04560](https://arxiv.org/abs/2505.04560) [find](https://scholar.google.com/scholar?q=Guanghui+Wang%2C+Zhiyong+Yang%2C+Zitai+Wang%2C+Shi+Wang%2C+Qianqian+Xu%2C+and+Qingming+Huang%2E+2025%2E+Abkd%3A+Pursuing+a+proper+allocation+of+the+probability+mass+in+knowledge+distillation+via+alpha%2Dbeta%2Ddivergence)
34. Xin Wang, Yudong Chen, and Wenwu Zhu. 2021. A survey on curriculum learning. IEEE transactions on pattern analysis and machine intelligence, 44(9):4555–4576. [find](https://scholar.google.com/scholar?q=Xin+Wang%2C+Yudong+Chen%2C+and+Wenwu+Zhu%2E+2021%2E+A+survey+on+curriculum+learning%2E+IEEE+transactions+on+pattern+analysis+and+machine+intelligence%2C+44%289%29%3A4555%E2%80%934576%2E)
35. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022a. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2022a%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+arXiv+p)
36. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. 2022b. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arXiv:2204.07705, 2… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Anjana+Arunkumar%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Atharva+Naik%2C+David+Stap%2C+et+al%2E+2022b%2E+Benchmarking+gen)
37. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824–24837.
38. Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. 2023. [F-divergence minimization for sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/wen-2023-f-divergence-minimization-sequence-level/). arXiv preprint arXiv:2307.15190.
39. Liuyu Xiang, Guiguang Ding, and Jungong Han. 2020. Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16, pages 247–263. Springer. [find](https://scholar.google.com/scholar?q=Liuyu+Xiang%2C+Guiguang+Ding%2C+and+Jungong+Han%2E+2020%2E+Learning+from+multiple+experts%3A+Self%2Dpaced+knowledge+distillation+for+long%2Dtailed+classification%2E+In+Computer+Vision%E2%80%93ECCV+2020%3A+16th+European+Confere)
40. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2024. [Wizardlm: Empowering large pre-trained language models to follow complex instructions](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/). In The Twelfth International Conference on Learning Representations.
41. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. 2024a. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122. — [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://md.research.elamur.ai/free/en/papers/yang-2024-qwen2-5-math-technical-report/)
42. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. 2024b. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122. — [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://md.research.elamur.ai/free/en/papers/yang-2024-qwen2-5-math-technical-report/)
43. Chuanpeng Yang, Yao Zhu, Wang Lu, Yidong Wang, Qian Chen, Chenlong Gao, Bingjie Yan, and Yiqiang Chen. 2024c. Survey on knowledge distillation for large language models: methods, evaluation, and application. ACM Transactions on Intelligent Systems and Technology. [find](https://scholar.google.com/scholar?q=Chuanpeng+Yang%2C+Yao+Zhu%2C+Wang+Lu%2C+Yidong+Wang%2C+Qian+Chen%2C+Chenlong+Gao%2C+Bingjie+Yan%2C+and+Yiqiang+Chen%2E+2024c%2E+Survey+on+knowledge+distillation+for+large+language+models%3A+methods%2C+evaluation%2C+and+appli)
44. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). arXiv preprint arXiv:2309.12284.
45. Mengxiang Zhang and Lingyuan Liu. 2025. Staged knowledge distillation through least-to-most prompting: Optimizing teacher guidance via difficulty-aware training. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 8489–8501. [find](https://scholar.google.com/scholar?q=Mengxiang+Zhang+and+Lingyuan+Liu%2E+2025%2E+Staged+knowledge+distillation+through+least%2Dto%2Dmost+prompting%3A+Optimizing+teacher+guidance+via+difficulty%2Daware+training%2E+In+Findings+of+the+Association+for+Com)
46. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in neural information processing systems, 36:46595–46623.
47. Qingqing Zhu, Xiuying Chen, Pengfei Wu, JunFei Liu, and Dongyan Zhao. 2021. Combining curriculum learning and knowledge distillation for dialogue generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1284–1295. [find](https://scholar.google.com/scholar?q=Qingqing+Zhu%2C+Xiuying+Chen%2C+Pengfei+Wu%2C+JunFei+Liu%2C+and+Dongyan+Zhao%2E+2021%2E+Combining+curriculum+learning+and+knowledge+distillation+for+dialogue+generation%2E+In+Findings+of+the+Association+for+Computa)

*Updated 2 November 2026*