---
title: "Rethinking Kullback-Leibler Divergence in Knowledge Distillation"
description: "Because LLM distillation runs for only a few epochs, the authors propose Adaptive KL (AKL). 48 references, cited by 44 papers here."
authors: ["Taiqiang Wu", "Chaofan Tao", "Jiahao Wang", "Runming Yang", "Zhe Zhao", "Ngai Wong"]
year: 2025
venue: "Proceedings of the 31st International Conference on Computational Linguistics"
source: "https://aclanthology.org/2025.coling-main.383/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wu-2025-rethinking-kullback-leibler-divergence-knowledge/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wu-2025-rethinking-kullback-leibler-divergence-knowledge/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) / Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models

# Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models

**Authors:** [Taiqiang Wu](https://md.research.elamur.ai/free/en/authors/taiqiang-wu/), [Chaofan Tao](https://md.research.elamur.ai/free/en/authors/chaofan-tao/), [Jiahao Wang](https://md.research.elamur.ai/free/en/authors/jiahao-wang/), [Runming Yang](https://md.research.elamur.ai/free/en/authors/runming-yang/), [Zhe Zhao](https://md.research.elamur.ai/free/en/authors/zhe-zhao/) and [Ngai Wong](https://md.research.elamur.ai/free/en/authors/ngai-wong/) · **Year:** 2025 · **Venue:** [Proceedings of the 31st International Conference on Computational Linguistics](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2025](https://md.research.elamur.ai/free/en/venues/coling/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.coling-main.383/) · [PDF](https://aclanthology.org/2025.coling-main.383.pdf) · [OpenAlex](https://openalex.org/W4393968834)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [44](https://md.research.elamur.ai/free/en/papers/wu-2025-rethinking-kullback-leibler-divergence-knowledge/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Distilling large language models usually means training a student to match the teacher's next-token distribution with a Kullback-Leibler loss. Several recent methods prefer reverse KL (RKL) to forward KL (FKL), arguing that RKL is mode-seeking and FKL is mean-seeking. Wu et al. show that this argument assumes continuous distributions and a unimodal, Gaussian-like student. Neither condition holds for discrete softmax outputs over a vocabulary.

Using gradient analysis and toy experiments, the authors prove that both losses reach their optimum only when the student reproduces the teacher's distribution exactly. Given enough epochs, the two losses therefore converge to the same solution. The real difference lies in early training. FKL first fits the high-probability head of the distribution, while RKL first fits the low-probability tail.

Because LLM distillation runs for only a few epochs, the authors propose Adaptive KL (AKL). AKL blends FKL and RKL using weights derived from how teacher and student differ on head versus tail tokens. Across Rouge-L, GPT-4 ratings and human judgments, AKL outperforms the baselines without adding parameters. This corrects a common misconception and provides an inexpensive loss that can directly replace existing ones.

## Key points
- On discrete toy distributions, FKL and RKL started from the same initial student, trained with Adam at learning rate 0.1, both matched the target after 200 epochs. Convergence appeared after roughly 50 or more epochs, far more than the approximately 10 used in typical LLM KD.
- Baselines included SFT, SeqKD, FKL, RKL and a plain FKL+RKL sum. Experiments used GPT-2 and LLaMA-family models, including a 6.7B teacher, with TinyLLaMA 1.1B as one of the students.
- On the 8 Dolly test subtasks, FKL was strongest on Closed QA and Summary, while RKL was strongest on Brain Storming. FKL+RKL usually fell in between: on Classification it scored 48.00, compared with 47.01 for FKL and 51.39 for RKL. AKL even exceeded the teacher on Information Extraction and Creative Writing.
- GPT-4 scored responses for diversity and quality, and five human experts made blind pairwise comparisons. In these comparisons, AKL won or tied against every baseline in more than 75% of cases on both criteria.
- Additional evaluations on Winogrande, OpenBookQA, BoolQ and ARC also favored AKL. Models larger than the tested ones, such as LLaMA-2 70B, were not evaluated.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 333 works · 87,019 citations
- [University of Hong Kong](https://md.research.elamur.ai/free/en/universities/university-of-hong-kong/) — University, Pok Fu Lam, China · 89 works · 11,774 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 242 works

## Cite

Cite as: Taiqiang Wu, Chaofan Tao, Jiahao Wang, Runming Yang, Zhe Zhao and Ngai Wong. 2025. Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wu2025rethinking,
  title = {{Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models}},
  author = {Taiqiang Wu, Chaofan Tao, Jiahao Wang, Runming Yang, Zhe Zhao and Ngai Wong},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.383/}
}
```

- [← previous Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy](https://md.research.elamur.ai/free/en/papers/liu-2025-skywork-reward-v2-scaling/)
- [next → Unfamiliar Finetuning Examples Control How Language Models Hallucinate](https://md.research.elamur.ai/free/en/papers/kang-2025-unfamiliar-finetuning-examples-control-how/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/)

## References

48 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. 2024. [On-policy distillation of language models: Learning from self-generated mistakes](https://md.research.elamur.ai/free/en/papers/agarwal-2023-policy-distillation-language-models-learning/).
2. Ibtihel Amara, Nazanin Sepahvand, Brett H Meyer, Warren J Gross, and James J Clark. 2022. Bd-kd: Balancing the divergences for online knowledge distillation. [find](https://scholar.google.com/scholar?q=Ibtihel+Amara%2C+Nazanin+Sepahvand%2C+Brett+H+Meyer%2C+Warren+J+Gross%2C+and+James+J+Clark%2E+2022%2E+Bd%2Dkd%3A+Balancing+the+divergences+for+online+knowledge+distillation%2E)
3. Akhter Al Amin and Kazi Sinthia Kabir. 2022. A disability lens towards biases in GPT-3 generated open-ended languages. CoRR, abs/2206.11993. — [arXiv:2206.11993](https://arxiv.org/abs/2206.11993) [find](https://scholar.google.com/scholar?q=Akhter+Al+Amin+and+Kazi+Sinthia+Kabir%2E+2022%2E+A+disability+lens+towards+biases+in+GPT%2D3+generated+open%2Dended+languages%2E+CoRR%2C+abs%2F2206%2E11993%2E+%E2%80%94+%5BarXiv%3A2206%2E11993%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2206%2E11993%29)
4. Alan Chan, Hugo Silva, Sungsu Lim, Tadashi Kozuno, A. Rupam Mahmood, and Martha White. 2022. Greedification operators for policy optimization: Investigating forward and reverse KL divergences. J. Mach. Learn. Res., 23:253:1–253:79. [find](https://scholar.google.com/scholar?q=Alan+Chan%2C+Hugo+Silva%2C+Sungsu+Lim%2C+Tadashi+Kozuno%2C+A%2E+Rupam+Mahmood%2C+and+Martha+White%2E+2022%2E+Greedification+operators+for+policy+optimization%3A+Investigating+forward+and+reverse+KL+divergences%2E+J%2E+Mach)
5. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. 2(3):6. [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E+Gonzalez%2C+et+al%2E+2023%2E+Vicuna%3A+An+open%2Dsource+chatbot+impressing+gpt%2D4+wi)
6. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). arXiv preprint arXiv:1905.10044.
7. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457.
8. Wojciech M Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant Jayakumar, Grzegorz Swirszcz, and Max Jaderberg. 2019. Distilling policy distillation. In The 22nd international conference on artificial intelligence and statistics, pages 1331–1340. PMLR. [find](https://scholar.google.com/scholar?q=Wojciech+M+Czarnecki%2C+Razvan+Pascanu%2C+Simon+Osindero%2C+Siddhant+Jayakumar%2C+Grzegorz+Swirszcz%2C+and+Max+Jaderberg%2E+2019%2E+Distilling+policy+distillation%2E+In+The+22nd+international+conference+on+artificial)
9. P Kingma Diederik. 2014. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). (No Title).
10. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Ani… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Alain+Le+Noac%E2%80%99h%2C+Haonan+Li%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Chris+Oci)
11. Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. 2019. Openwebtext corpus. [find](https://scholar.google.com/scholar?q=Aaron+Gokaslan%2C+Vanya+Cohen%2C+Ellie+Pavlick%2C+and+Stefanie+Tellex%2E+2019%2E+Openwebtext+corpus%2E)
12. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023. [Knowledge distillation of large language models](https://md.research.elamur.ai/free/en/papers/gu-2023-minillm-policy-distillation-large-language/). CoRR, abs/2306.08543.
13. Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). CoRR, abs/1503.02531.
14. Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2023. [Unnatural instructions: Tuning language models with (almost) no human labor](https://md.research.elamur.ai/free/en/papers/honovich-2023-unnatural-instructions-tuning-language-models/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14409–14428, Toronto, Canada. Associat…
15. Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. 2024. Billm: Pushing the limit of post-training quantization for llms. [find](https://scholar.google.com/scholar?q=Wei+Huang%2C+Yangdong+Liu%2C+Haotong+Qin%2C+Ying+Li%2C+Shiming+Zhang%2C+Xianglong+Liu%2C+Michele+Magno%2C+and+Xiaojuan+Qi%2E+2024%2E+Billm%3A+Pushing+the+limit+of+post%2Dtraining+quantization+for+llms%2E)
16. Harold Jeffreys. 1946. An invariant form for the prior probability in estimation problems. Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, 186(1007):453–461. [find](https://scholar.google.com/scholar?q=Harold+Jeffreys%2E+1946%2E+An+invariant+form+for+the+prior+probability+in+estimation+problems%2E+Proceedings+of+the+Royal+Society+of+London%2E+Series+A%2E+Mathematical+and+Physical+Sciences%2C+186%281007%29%3A453%E2%80%93461%2E)
17. Gyeongman Kim, Doohyuk Jang, and Eunho Yang. 2024. Promptkd: Distilling student-friendly knowledge for generative language models via prompt tuning. arXiv preprint arXiv:2402.12842. [find](https://scholar.google.com/scholar?q=Gyeongman+Kim%2C+Doohyuk+Jang%2C+and+Eunho+Yang%2E+2024%2E+Promptkd%3A+Distilling+student%2Dfriendly+knowledge+for+generative+language+models+via+prompt+tuning%2E+arXiv+preprint+arXiv%3A2402%2E12842%2E)
18. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
19. Minsoo Kim, Sihwa Lee, Janghwan Lee, Sukjin Hong, Du-Seong Chang, Wonyong Sung, and Jungwook Choi. 2023. Token-scaled logit distillation for ternary weight generative language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems… [find](https://scholar.google.com/scholar?q=Minsoo+Kim%2C+Sihwa+Lee%2C+Janghwan+Lee%2C+Sukjin+Hong%2C+Du%2DSeong+Chang%2C+Wonyong+Sung%2C+and+Jungwook+Choi%2E+2023%2E+Token%2Dscaled+logit+distillation+for+ternary+weight+generative+language+models%2E+In+Advances+in+N)
20. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2021%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2E)
21. Yoon Kim and Alexander M Rush. 2016. [Sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327.
22. Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhijun Tu, Kai Han, Hailin Hu, and Dacheng Tao. 2024. A survey on transformer compression. [find](https://scholar.google.com/scholar?q=Yehui+Tang%2C+Yunhe+Wang%2C+Jianyuan+Guo%2C+Zhijun+Tu%2C+Kai+Han%2C+Hailin+Hu%2C+and+Dacheng+Tao%2E+2024%2E+A+survey+on+transformer+compression%2E)
23. Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. 2024. [Distillm: Towards streamlined distillation for large language models](https://md.research.elamur.ai/free/en/papers/ko-2024-distillm-towards-streamlined-distillation-large/). arXiv preprint arXiv:2402.03898.
24. Chaofan Tao, Lu Hou, Haoli Bai, Jiansheng Wei, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. 2023. Structured pruning for efficient generative pre-trained language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10880– 10895. [find](https://scholar.google.com/scholar?q=Chaofan+Tao%2C+Lu+Hou%2C+Haoli+Bai%2C+Jiansheng+Wei%2C+Xin+Jiang%2C+Qun+Liu%2C+Ping+Luo%2C+and+Ngai+Wong%2E+2023%2E+Structured+pruning+for+efficient+generative+pre%2Dtrained+language+models%2E+In+Findings+of+the+Associatio)
25. Daphne Koller and Nir Friedman. 2009. Probabilistic Graphical Models - Principles and Techniques. MIT Press. [find](https://scholar.google.com/scholar?q=Daphne+Koller+and+Nir+Friedman%2E+2009%2E+Probabilistic+Graphical+Models+%2D+Principles+and+Techniques%2E+MIT+Press%2E)
26. Hyoje Lee, Yeachan Park, Hyun Seo, and Myungjoo Kang. 2023. Self-knowledge distillation via dropout. Computer Vision and Image Understanding, 233:103720. [find](https://scholar.google.com/scholar?q=Hyoje+Lee%2C+Yeachan+Park%2C+Hyun+Seo%2C+and+Myungjoo+Kang%2E+2023%2E+Self%2Dknowledge+distillation+via+dropout%2E+Computer+Vision+and+Image+Understanding%2C+233%3A103720%2E)
27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/).
28. Li Lucy and David Bamman. 2021. [Gender and representation bias in GPT-3 generated stories](https://md.research.elamur.ai/free/en/papers/lucy-2021-gender-representation-bias-gpt-3/). In Proceedings of the Third Workshop on Narrative Understanding, pages 48–55, Virtual. Association for Computational Linguistics.
29. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+arXiv+preprint+arXiv%3A1809%2E02789%2E)
30. Tom Minka et al. 2005. Divergence measures and message passing. Technical report, Technical report, Microsoft Research. [find](https://scholar.google.com/scholar?q=Tom+Minka+et+al%2E+2005%2E+Divergence+measures+and+message+passing%2E+Technical+report%2C+Technical+report%2C+Microsoft+Research%2E)
31. Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. 2024. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Niklas+Muennighoff%2C+Alexander+Rush%2C+Boaz+Barak%2C+Teven+Le+Scao%2C+Nouamane+Tazi%2C+Aleksandra+Piktus%2C+Sampo+Pyysalo%2C+Thomas+Wolf%2C+and+Colin+A+Raffel%2E+2024%2E+Scaling+data%2Dconstrained+language+models%2E+Advance)
32. A Tuan Nguyen, Toan Tran, Yarin Gal, Philip HS Torr, and Atılım Güneş Baydin. 2021. Kl guided domain adaptation. arXiv preprint arXiv:2106.07780. — [arXiv:2106.07780](https://arxiv.org/abs/2106.07780) [find](https://scholar.google.com/scholar?q=A+Tuan+Nguyen%2C+Toan+Tran%2C+Yarin+Gal%2C+Philip+HS+Torr%2C+and+At%C4%B1l%C4%B1m+G%C3%BCnes%CC%A7+Baydin%2E+2021%2E+Kl+guided+domain+adaptation%2E+arXiv+preprint+arXiv%3A2106%2E07780%2E+%E2%80%94+%5BarXiv%3A2106%2E07780%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2106%2E07780)
33. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). abs/2303.08774.
34. CoRR, Igal Sason and Sergio Verdú. 2016. f -divergence inequalities. IEEE Transactions on Information Theory, 62(11):5973–6006. [find](https://scholar.google.com/scholar?q=CoRR%2C+Igal+Sason+and+Sergio+Verd%C3%BA%2E+2016%2E+f+%2Ddivergence+inequalities%2E+IEEE+Transactions+on+Information+Theory%2C+62%2811%29%3A5973%E2%80%936006%2E)
35. Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. 2022. Compression of generative pre-trained language models via quantization. pages 4821–4836. [find](https://scholar.google.com/scholar?q=Chaofan+Tao%2C+Lu+Hou%2C+Wei+Zhang%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Qun+Liu%2C+Ping+Luo%2C+and+Ngai+Wong%2E+2022%2E+Compression+of+generative+pre%2Dtrained+language+models+via+quantization%2E+pages+4821%E2%80%934836%2E)
36. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [find](https://scholar.google.com/scholar?q=Rohan+Taori%2C+Ishaan+Gulrajani%2C+Tianyi+Zhang%2C+Yann+Dubois%2C+Xuechen+Li%2C+Carlos+Guestrin%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+2023%2E+Stanford+alpaca%3A+An+instruction%2Dfollowing+llama+model%2E)
37. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). CoRR, a…
38. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/).
39. Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. 2023a. Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints. [find](https://scholar.google.com/scholar?q=Chaoqi+Wang%2C+Yibo+Jiang%2C+Chenghao+Yang%2C+Han+Liu%2C+and+Yuxin+Chen%2E+2023a%2E+Beyond+reverse+kl%3A+Generalizing+direct+preference+optimization+with+diverse+divergence+constraints%2E)
40. Jiahao Wang, Songyang Zhang, Yong Liu, Taiqiang Wu, Yujiu Yang, Xihui Liu, Kai Chen, Ping Luo, and Dahua Lin. 2023b. Riformer: Keep your vision backbone effective but removing token mixer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14443–14452. [find](https://scholar.google.com/scholar?q=Jiahao+Wang%2C+Songyang+Zhang%2C+Yong+Liu%2C+Taiqiang+Wu%2C+Yujiu+Yang%2C+Xihui+Liu%2C+Kai+Chen%2C+Ping+Luo%2C+and+Dahua+Lin%2E+2023b%2E+Riformer%3A+Keep+your+vision+backbone+effective+but+removing+token+mixer%2E+In+Proceedi)
41. Wenxiao Wang, Wei Chen, Yicong Luo, Yongliu Long, Zhengkai Lin, Liye Zhang, Binbin Lin, Deng Cai, and Xiaofei He. 2024. Model compression and efficient inference for [large language models: A survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/).
42. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Atharva+Naik%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Anjana+Arunkumar%2C+David+Stap%2C+Eshaan+Pathak%2C+Giannis+Karaman)
43. Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. 2023. [f-divergence minimization for sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/wen-2023-f-divergence-minimization-sequence-level/).
44. Taiqiang Wu, Cheng Hou, Zhe Zhao, Shanshan Lao, Jiayi Li, Ngai Wong, and Yujiu Yang. 2023. Weight-inherited distillation for task-agnostic BERT compression. CoRR, abs/2305.09098. [find](https://scholar.google.com/scholar?q=Taiqiang+Wu%2C+Cheng+Hou%2C+Zhe+Zhao%2C+Shanshan+Lao%2C+Jiayi+Li%2C+Ngai+Wong%2C+and+Yujiu+Yang%2E+2023%2E+Weight%2Dinherited+distillation+for+task%2Dagnostic+BERT+compression%2E+CoRR%2C+abs%2F2305%2E09098%2E)
45. Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2023. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694. [find](https://scholar.google.com/scholar?q=Mengzhou+Xia%2C+Tianyu+Gao%2C+Zhiyuan+Zeng%2C+and+Danqi+Chen%2E+2023%2E+Sheared+llama%3A+Accelerating+language+model+pre%2Dtraining+via+structured+pruning%2E+arXiv+preprint+arXiv%3A2310%2E06694%2E)
46. Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. [Tinyllama: An open-source small language model](https://md.research.elamur.ai/free/en/papers/zhang-2024-tinyllama-open-source-small-language/).
47. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. [OPT: open pre-t…](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/)
48. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+BERT%2E+In+8th+International+Conference+on+Learning+Representations%2C+ICLR+2)

*Updated 30 October 2026*