---
title: "Dual-Space Knowledge Distillation for Large Language Models"
description: "The proposed fix is Dual-Space Knowledge Distillation (DSKD). 38 references, cited by 31 papers here."
authors: ["Songming Zhang", "Xue Zhang", "Zengkui Sun", "Yufeng Chen", "Jinan Xu"]
year: 2024
venue: "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2024.emnlp-main.1010/"
doi: "10.18653/v1/2024.emnlp-main.1010"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2024-dual-space-knowledge-distillation-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2024-dual-space-knowledge-distillation-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / Dual-Space Knowledge Distillation for Large Language Models

# Dual-Space Knowledge Distillation for Large Language Models

**Authors:** [Songming Zhang](https://md.research.elamur.ai/free/en/authors/songming-zhang/), [Xue Zhang](https://md.research.elamur.ai/free/en/authors/xue-zhang/), [Zengkui Sun](https://md.research.elamur.ai/free/en/authors/zengkui-sun/), [Yufeng Chen](https://md.research.elamur.ai/free/en/authors/yufeng-chen/) and [Jinan Xu](https://md.research.elamur.ai/free/en/authors/jinan-xu/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2024](https://md.research.elamur.ai/free/en/venues/emnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.emnlp-main.1010/) · [PDF](https://aclanthology.org/2024.emnlp-main.1010.pdf) · [DOI 10.18653/v1/2024.emnlp-main.1010](https://doi.org/10.18653/v1/2024.emnlp-main.1010) · [OpenAlex](https://openalex.org/W4404781935)

**Citations:** 8 citations (OpenAlex, 2026-10-14) · [31](https://md.research.elamur.ai/free/en/papers/zhang-2024-dual-space-knowledge-distillation-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Standard white-box knowledge distillation for LLMs trains a small student to match the teacher's token-level output distributions. However, each distribution is produced by that model's own prediction head, so the two sit in different output spaces. The authors argue this mismatch has two costs. First, the student stays dissimilar to the teacher in both hidden representations and distributions. Second, distillation breaks down when the models use different vocabularies. A toy simulation with 2-D vectors supports the first claim: with a shared prediction head, the student's states and the KL loss get much closer to the teacher's.

The proposed fix is Dual-Space Knowledge Distillation (DSKD). Linear projectors map each model's final hidden states into the other's representation space, so both distributions come from one shared head, in the student space and again in the teacher space. For mismatched tokenizers, a cross-model attention (CMA) mechanism learns to align tokens across the two sequences. On instruction-following benchmarks, DSKD beats the conventional framework under several distance functions and outperforms earlier cross-vocabulary KD methods. This makes distillation between arbitrary LLM pairs feasible.

## Key points
- **Simulation design:** trainable "student" and frozen "teacher" 2-D vectors were distilled for 1000 iterations, with loss curves averaged over 100 runs. Separate heads left KL divergence well above zero, while a shared head converged faster and closer to zero.
- **Compatibility:** DSKD works with various distance functions, including KL, reverse KL and JS divergence. It does not enlarge the student, unlike SimKD, which reuses the teacher's head at inference.
- **GPT-4 evaluation:** on 100 random Dolly test instructions, GPT-4 preferred responses from TinyLLaMA trained with DSKD over the vanilla framework in most cases.
- **Representation similarity:** across 1000 training samples, L1 distances between teacher and student representation structures were much smaller with DSKD than with vanilla KD or SFT. Structure was measured via cosine similarity and normalized inner product.
- **Limitation and code:** DSKD-CMA still trails same-vocabulary DSKD slightly, which the authors attribute to token alignment errors. Code is available at github.com/songmzhang/DSKD.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Beijing Jiaotong University](https://md.research.elamur.ai/free/en/universities/beijing-jiaotong-university/) — University, Beijing, China · 22 works · 1,754 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 242 works

## Cite

Cite as: Songming Zhang, Xue Zhang, Zengkui Sun, Yufeng Chen and Jinan Xu. 2024. Dual-Space Knowledge Distillation for Large Language Models. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2024dual,
  title = {{Dual-Space Knowledge Distillation for Large Language Models}},
  author = {Songming Zhang, Xue Zhang, Zengkui Sun, Yufeng Chen and Jinan Xu},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.1010},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.1010}
}
```

- [← previous Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Efforts](https://md.research.elamur.ai/free/en/papers/tan-2024-personalized-pieces-efficient-personalized-large/)
- [next → InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output](https://md.research.elamur.ai/free/en/papers/zhang-2024-internlm-xcomposer-2-5/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

38 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. 2024. [On-policy distillation of language models: Learning from self-generated mistakes](https://md.research.elamur.ai/free/en/papers/agarwal-2023-policy-distillation-language-models-learning/). In The Twelfth International Conference on Learning Representations.
2. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609.
3. Nicolas Boizard, Kevin El-Haddad, Céline Hudelot, and Pierre Colombo. 2024. [Towards cross-tokenizer distillation: the universal logit distillation loss for llms](https://md.research.elamur.ai/free/en/papers/boizard-2024-towards-cross-tokenizer-distillation-universal/). arXiv preprint arXiv:2402.12030.
4. Defang Chen, Jian-Ping Mei, Hailin Zhang, Can Wang, Yan Feng, and Chun Chen. 2022. Knowledge distillation with the reused teacher classifier. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11933–11942. [find](https://scholar.google.com/scholar?q=Defang+Chen%2C+Jian%2DPing+Mei%2C+Hailin+Zhang%2C+Can+Wang%2C+Yan+Feng%2C+and+Chun+Chen%2E+2022%2E+Knowledge+distillation+with+the+reused+teacher+classifier%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+visi)
5. Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, and Jingjing Liu. 2020. Distilling knowledge learned in BERT for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7893–7905, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yen%2DChun+Chen%2C+Zhe+Gan%2C+Yu+Cheng%2C+Jingzhou+Liu%2C+and+Jingjing+Liu%2E+2020%2E+Distilling+knowledge+learned+in+BERT+for+text+generation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Compu)
6. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023. URL [https://lmsys.org/blog/2023-03-30-vicuna](https://lmsys.org/blog/2023-03-30-vicuna), 3(5). [↗](https://lmsys.org/blog/2023-03-30-vicuna)
7. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research, 24(240):1–113.
8. Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023. [Specializing smaller language models towards multi-step reasoning](https://md.research.elamur.ai/free/en/papers/fu-2023-specializing-smaller-language-models-towards/). In International Conference on Machine Learning, pages 10421–10430. PMLR.
9. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yuxian+Gu%2C+Li+Dong%2C+Furu+Wei%2C+and+Minlie+Huang%2E+2023%2E+Minillm%3A+Knowledge+distillation+of+large+language+models%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2E)
10. Chin-Yew Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pages 74–81.
11. Yang Liu, Sheng Shen, and Mirella Lapata. 2021. Noisy self-knowledge distillation for text summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 692–703, Online. Association for Computat… [find](https://scholar.google.com/scholar?q=Yang+Liu%2C+Sheng+Shen%2C+and+Mirella+Lapata%2E+2021%2E+Noisy+self%2Dknowledge+distillation+for+text+summarization%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association+for+Com)
12. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv:2303.08774.
13. Preprint, Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). arXiv preprint arXiv:1503.02531.
14. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
15. Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2023. [Unnatural instructions: Tuning language models with (almost) no human labor](https://md.research.elamur.ai/free/en/papers/honovich-2023-unnatural-instructions-tuning-language-models/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14409–14428.
16. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+2019%2E+Distilbert%2C+a+distilled+version+of+bert%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E+arXiv+preprint+arXiv%3A1910%2E01108%2E)
17. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023%2E+Mist)
18. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. [Tinybert: Distilling bert for natural language understanding](https://md.research.elamur.ai/free/en/papers/jiao-2020-tinybert-distilling-bert-natural-language/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163–4174.
19. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361.
20. Yoon Kim and Alexander M Rush. 2016. [Sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/). arXiv preprint arXiv:1606.07947.
21. Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. 2024. [Distillm: Towards streamlined distillation for large language models](https://md.research.elamur.ai/free/en/papers/ko-2024-distillm-towards-streamlined-distillation-large/). arXiv preprint arXiv:2402.03898.
22. Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi. 2023. [Symbolic chain-of-thought distillation: Small models can also “think” step-by-step](https://md.research.elamur.ai/free/en/papers/li-2023-symbolic-chain-thought-distillation-small/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2…
23. Alexander Lin, Jeremy Wohlwend, Howard Chen, and Tao Lei. 2020. Autoregressive knowledge distillation through imitation learning. arXiv preprint arXiv:2009.07253. [find](https://scholar.google.com/scholar?q=Alexander+Lin%2C+Jeremy+Wohlwend%2C+Howard+Chen%2C+and+Tao+Lei%2E+2020%2E+Autoregressive+knowledge+distillation+through+imitation+learning%2E+arXiv+preprint+arXiv%3A2009%2E07253%2E)
24. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. [Patient knowledge distillation for bert model compression](https://md.research.elamur.ai/free/en/papers/sun-2019-patient-knowledge-distillation-bert-model/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 43…
25. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. [Mobilebert: a compact task-agnostic bert for resource-limited devices](https://md.research.elamur.ai/free/en/papers/sun-2020-mobilebert-compact-task-agnostic-bert/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2158–2170.
26. Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Tie-Yan Liu. 2019. Multilingual neural machine translation with knowledge distillation. arXiv preprint arXiv:1902.10461. — [arXiv:1902.10461](https://arxiv.org/abs/1902.10461) [find](https://scholar.google.com/scholar?q=Xu+Tan%2C+Yi+Ren%2C+Di+He%2C+Tao+Qin%2C+Zhou+Zhao%2C+and+Tie%2DYan+Liu%2E+2019%2E+Multilingual+neural+machine+translation+with+knowledge+distillation%2E+arXiv+preprint+arXiv%3A1902%2E10461%2E+%E2%80%94+%5BarXiv%3A1902%2E10461%5D%28https%3A%2F%2Farx)
27. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
28. Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. 2024. Knowledge fusion of large language models. arXiv preprint arXiv:2401.10491. [find](https://scholar.google.com/scholar?q=Fanqi+Wan%2C+Xinting+Huang%2C+Deng+Cai%2C+Xiaojun+Quan%2C+Wei+Bi%2C+and+Shuming+Shi%2E+2024%2E+Knowledge+fusion+of+large+language+models%2E+arXiv+preprint+arXiv%3A2401%2E10491%2E)
29. Fusheng Wang, Jianhao Yan, Fandong Meng, and Jie Zhou. 2021a. Selective knowledge distillation for neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume… [find](https://scholar.google.com/scholar?q=Fusheng+Wang%2C+Jianhao+Yan%2C+Fandong+Meng%2C+and+Jie+Zhou%2E+2021a%2E+Selective+knowledge+distillation+for+neural+machine+translation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computat)
30. Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. 2021b. Minilmv2: Multi-head self-attention relation distillation for compressing pre-trained transformers. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2140–2151. [find](https://scholar.google.com/scholar?q=Wenhui+Wang%2C+Hangbo+Bao%2C+Shaohan+Huang%2C+Li+Dong%2C+and+Furu+Wei%2E+2021b%2E+Minilmv2%3A+Multi%2Dhead+self%2Dattention+relation+distillation+for+compressing+pre%2Dtrained+transformers%2E+In+Findings+of+the+Association)
31. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776–5788. [find](https://scholar.google.com/scholar?q=Wenhui+Wang%2C+Furu+Wei%2C+Li+Dong%2C+Hangbo+Bao%2C+Nan+Yang%2C+and+Ming+Zhou%2E+2020%2E+Minilm%3A+Deep+self%2Dattention+distillation+for+task%2Dagnostic+compression+of+pre%2Dtrained+transformers%2E+Advances+in+Neural+Inform)
32. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In The 61st Annual Meeting Of The Association For Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2023%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+In+The+6)
33. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. 2022. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arXiv:2204.07705, 2. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Anjana+Arunkumar%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Atharva+Naik%2C+David+Stap%2C+et+al%2E+2022%2E+Benchmarking+gene)
34. Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. 2023. [f-divergence minimization for sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/wen-2023-f-divergence-minimization-sequence-level/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10817– 10834.
35. Taiqiang Wu, Chaofan Tao, Jiahao Wang, Zhe Zhao, and Ngai Wong. 2024. [Rethinking kullback-leibler divergence in knowledge distillation for large language models](https://md.research.elamur.ai/free/en/papers/wu-2025-rethinking-kullback-leibler-divergence-knowledge/). arXiv preprint arXiv:2404.02657.
36. Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. 2024. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116. — [arXiv:2402.13116](https://arxiv.org/abs/2402.13116) [find](https://scholar.google.com/scholar?q=Xiaohan+Xu%2C+Ming+Li%2C+Chongyang+Tao%2C+Tao+Shen%2C+Reynold+Cheng%2C+Jinyang+Li%2C+Can+Xu%2C+Dacheng+Tao%2C+and+Tianyi+Zhou%2E+2024%2E+A+survey+on+knowledge+distillation+of+large+language+models%2E+arXiv+preprint+arXiv%3A2)
37. Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. [Tinyllama: An open-source small language model](https://md.research.elamur.ai/free/en/papers/zhang-2024-tinyllama-open-source-small-language/). arXiv preprint arXiv:2401.02385.
38. Songming Zhang, Yunlong Liang, Shuaibo Wang, Yufeng Chen, Wenjuan Han, Jian Liu, and Jinan Xu. 2023. Towards understanding and improving knowledge distillation for neural machine translation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P… [find](https://scholar.google.com/scholar?q=Songming+Zhang%2C+Yunlong+Liang%2C+Shuaibo+Wang%2C+Yufeng+Chen%2C+Wenjuan+Han%2C+Jian+Liu%2C+and+Jinan+Xu%2E+2023%2E+Towards+understanding+and+improving+knowledge+distillation+for+neural+machine+translation%2E+In+Proce)

*Updated 30 October 2026*