--- title: "DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs" description: "The authors argue this ignores how the choice of loss interacts with the type of data. 1 reference, cited by 32 papers here." authors: ["Jongwoo Ko", "Tianyi Chen", "Sungnyun Kim", "Tianyu Ding", "Luming Liang", "Ilya Zharkov", "Se-Young Yun"] year: 2025 venue: "arXiv:2503.07067" source: "https://arxiv.org/abs/2503.07067" doi: "10.48550/arxiv.2503.07067" arxiv: "2503.07067" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/ko-2025-distillm-2-contrastive-approach-boosts/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ko-2025-distillm-2-contrastive-approach-boosts/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/) / DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs # DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs **Authors:** [Jongwoo Ko](https://md.research.elamur.ai/free/en/authors/jongwoo-ko/), [Tianyi Chen](https://md.research.elamur.ai/free/en/authors/tianyi-chen/), [Sungnyun Kim](https://md.research.elamur.ai/free/en/authors/sungnyun-kim/), Tianyu Ding, Luming Liang, Ilya Zharkov and [Se-Young Yun](https://md.research.elamur.ai/free/en/authors/se-young-yun/) · **Year:** 2025 · **Venue:** arXiv:2503.07067 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2503.07067) · [PDF](https://arxiv.org/pdf/2503.07067) · [DOI 10.48550/arxiv.2503.07067](https://doi.org/10.48550/arxiv.2503.07067) · [arXiv 2503.07067](https://arxiv.org/abs/2503.07067) · [OpenAlex](https://openalex.org/W6891914185) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [32](https://md.research.elamur.ai/free/en/papers/ko-2025-distillm-2-contrastive-approach-boosts/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Knowledge distillation (KD) for LLMs usually applies the same divergence to teacher-generated outputs (TGOs) and student-generated outputs (SGOs). The authors argue this ignores how the choice of loss interacts with the type of data. A naive contrastive alternative, DPKD, plugs the teacher into DPO in place of the reference model. They find it prone to reward hacking: it pushes student-response likelihood down excessively and produces degenerate text. DistiLLM-2 extends DistiLLM with three components: - **Contrastive objective:** skew KL (SKL) on teacher responses raises their likelihood, while skew reverse KL (SRKL) on the student's own responses lowers theirs. - **Data curation:** SGOs are collected once per epoch in batches, which allows fast generation with vLLM. - **Curriculum:** the skew coefficient α adapts per sample, and the SRKL weight β grows over training. The method reaches state-of-the-art results for small models on instruction-following, math reasoning and code generation. It also transfers to other uses: building reference models for preference alignment, vision-language distillation, recovering quantized models, and training speculative-decoding drafters. This makes it a practical recipe for building compact models. ## Key points - The training loss is a β-weighted mix of SKL (computed on teacher responses) and SRKL (computed on student responses). The paper was published at ICML 2025, and the code is at github.com/jongwooko/distillm-2. - The DPKD failure is illustrated by negative log-likelihood. Student responses reached an NLL of 91.25, while teacher responses were fitted only to 20.29. - For VLM distillation, the teacher was LLaVA-1.5-7B and the student TinyLLaVA-1.4B, trained on 83K prompts from RLAIF-V-Dataset. On OK-VQA and TextVQA, DistiLLM-2 averaged +2.01% over GKD and +4.61% over DistiLLM. - In speculative decoding, a Llama-68m drafter trained with DistiLLM-2 delivered the highest speedups. It reached about ×1.97 with Phi3-medium and ×1.84 with Phi3.5-mini as verifiers, beating drafters trained with SFT, GKD and DistiLLM. - For a 4-bit quantized Phi-3.5-mini, LoRA-based recovery via DistiLLM-2 gave the best average score among the KD baselines. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works 🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 297 works ## Cite Cite as: Jongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding, Luming Liang, Ilya Zharkov and Se-Young Yun. 2025. DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs. arXiv:2503.07067. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{ko2025distillm, title = {{DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs}}, author = {Jongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding, Luming Liang, Ilya Zharkov and Se-Young Yun}, year = {2025}, doi = {10.48550/arxiv.2503.07067}, eprint = {2503.07067}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2503.07067} } ``` - [← previous Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward](https://md.research.elamur.ai/free/en/papers/zhang-2025-direct-preference-optimization-video-large/) - [next → Evaluating Language Models as Synthetic Data Generators](https://md.research.elamur.ai/free/en/papers/kim-2025-evaluating-language-models-as-synthetic/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. This paper presents work whose goal is to advance the field of machine learning, specifically efficiency and efficacy of LLMs-based system. We also have shown four possible applications (but not limited to) as follows: Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree,… — [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/) *Updated 30 October 2026*