---
title: "Beyond Logits — paper by Gong et al. (2025)"
description: "The authors propose feature dynamics distillation (FDD), which adds two terms to the standard… 52 references, cited by 5 papers here."
authors: ["Guoqiang Gong", "Jiaxing Wang", "Jin Xu", "Deping Xiang", "Zicheng Zhang", "Leqi Shen", "Yifeng Zhang", "Junhua Shu", "Zhaolong Xing", "Zhen Chen", "Pengzhang Liu", "Ke Zhang"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.1125/"
doi: "10.18653/v1/2025.acl-long.1125"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gong-2025-beyond-logits-aligning-feature-dynamics/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gong-2025-beyond-logits-aligning-feature-dynamics/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/10/) / Beyond Logits: Aligning Feature Dynamics for Effective Knowledge Distillation

# Beyond Logits: Aligning Feature Dynamics for Effective Knowledge Distillation

**Authors:** Guoqiang Gong, Jiaxing Wang, [Jin Xu](https://md.research.elamur.ai/free/en/authors/jin-xu/), Deping Xiang, Zicheng Zhang, Leqi Shen, Yifeng Zhang, Junhua Shu, Zhaolong Xing, [Zhen Chen](https://md.research.elamur.ai/free/en/authors/zhen-chen/), Pengzhang Liu and Ke Zhang · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.1125/) · [PDF](https://aclanthology.org/2025.acl-long.1125.pdf) · [DOI 10.18653/v1/2025.acl-long.1125](https://doi.org/10.18653/v1/2025.acl-long.1125) · [OpenAlex](https://openalex.org/W4412944760)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · 5 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most knowledge distillation (KD) for LLMs trains a small student to imitate only the teacher's final output distribution. This paper argues that this discards useful signal. Transformers with residual connections can be read as discretized ODEs, with layer indices acting as time steps. Under that view, a student is a coarser approximation of the same feature evolution, so it should follow the teacher's whole internal trajectory.

The authors propose feature dynamics distillation (FDD), which adds two terms to the standard KL-based KD loss:
- **Layer-wise term:** KL divergence between intermediate-layer predictions on a fixed teacher-to-student layer mapping.
- **Layer-delta term:** cosine distance between finite differences of those predictions across adjacent layers.

Teacher and student hidden sizes differ, so each model's own pretrained LM head projects its hidden states into vocabulary space, in the style of the Logit Lens. This replaces a learned projection matrix. On instruction-following benchmarks (Dolly, S-NI, UnNI), both added terms help, and combining all three gives the best student. This suggests that internal dynamics are a valuable distillation target for GPT-like models, not just BERT-style ones.

## Key points
- **Loss design:** FDD combines output KD with two extra terms. Trajectory matching uses KL divergence on intermediate-layer distributions, and derivative matching uses cosine distance on layer-to-layer changes.
- **Layer mapping:** student layer *j* is aligned with teacher layer K·*j*, where K is the ratio of teacher to student depth.
- **Ablation averages:** output KD plus the delta term reached 20.52. Using all three losses gave the top average of 22.16, indicating the two new terms are complementary.
- **Projection choice:** the reused pretrained LM head was compared against a randomly initialized projection (as in TinyBERT-style methods) and a pretrained-projection variant, evaluated on Dolly, S-NI and UnNI.
- **Limitations:** experiments used teachers up to 13B and students up to 7B parameters. Because the method relies on LM-head projection into a shared vocabulary space, it may be hard to apply across mismatched architectures or vocabularies.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 333 works · 87,019 citations
- [University of Oxford](https://md.research.elamur.ai/free/en/universities/university-of-oxford/) — University, Oxford, United Kingdom · 104 works · 157,183 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 242 works

## Cite

Cite as: Guoqiang Gong, Jiaxing Wang, Jin Xu, Deping Xiang, Zicheng Zhang, Leqi Shen, Yifeng Zhang, Junhua Shu, Zhaolong Xing, Zhen Chen, Pengzhang Liu and Ke Zhang. 2025. Beyond Logits: Aligning Feature Dynamics for Effective Knowledge Distillation. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{gong2025beyond,
  title = {{Beyond Logits: Aligning Feature Dynamics for Effective Knowledge Distillation}},
  author = {Guoqiang Gong, Jiaxing Wang, Jin Xu, Deping Xiang, Zicheng Zhang, Leqi Shen, Yifeng Zhang, Junhua Shu, Zhaolong Xing, Zhen Chen, Pengzhang Liu and Ke Zhang},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1125},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1125}
}
```

- [← previous ArabicSense: A Benchmark for Evaluating Commonsense Reasoning in Arabic with Large Language Models](https://md.research.elamur.ai/free/en/papers/lamsiyah-2025-arabicsense-benchmark-evaluating-commonsense-reasoning/)
- [next → Can LLMs Help Create Grammar?: Automating Grammar Creation for Endangered Languages with In-Context Learning](https://md.research.elamur.ai/free/en/papers/spencer-2025-can-llms-help-create-grammar/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/10/)

## References

52 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. 2024. [On-policy distillation of language models: Learning from self-generated mistakes](https://md.research.elamur.ai/free/en/papers/agarwal-2023-policy-distillation-language-models-learning/). In International Conference on Learning Representations.
2. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. [Layer normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/). In 1607.06450.
3. Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. [Eliciting latent predictions from transformers with the tuned lens](https://md.research.elamur.ai/free/en/papers/belrose-2023-eliciting-latent-predictions-transformers-tuned/). arXiv preprint arXiv:2303.08112.
4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
5. Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. 2018. Neural ordinary differential equations. Advances in neural information processing systems, 31. [find](https://scholar.google.com/scholar?q=Ricky+TQ+Chen%2C+Yulia+Rubanova%2C+Jesse+Bettencourt%2C+and+David+K+Duvenaud%2E+2018%2E+Neural+ordinary+differential+equations%2E+Advances+in+neural+information+processing+systems%2C+31%2E)
6. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+and+Eric+P%2E+Xing%2E+2023%2E+Vicuna%3A+An+open%2Dsource+cha)
7. Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2024. [Dola: Decoding by contrasting layers improves factuality in large language models](https://md.research.elamur.ai/free/en/papers/chuang-2023-dola-decoding-contrasting-layers-improves/). In International Conference on Learning Representations.
8. Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2023. [Analyzing transformers in embedding space](https://md.research.elamur.ai/free/en/papers/dar-2023-analyzing-transformers-embedding-space/). In Association for Computational Linguistics.
9. DeepSeek-AI-Group. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2DGroup%2E+2024%2E+Deepseek%2Dv3+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E19437%2E)
10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Conference of the North American Chapter of the Association for Computational Linguistics.
11. Subhabrata Dutta, Tanya Gautam, Soumen Chakrabarti, and Tanmoy Chakraborty. 2021. Redesigning the transformer architecture with insights from multi-particle dynamical systems. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Subhabrata+Dutta%2C+Tanya+Gautam%2C+Soumen+Chakrabarti%2C+and+Tanmoy+Chakraborty%2E+2021%2E+Redesigning+the+transformer+architecture+with+insights+from+multi%2Dparticle+dynamical+systems%2E+In+Advances+in+Neural+In)
12. Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed A Aly, Beidi Chen, and Carole-Jean Wu. 2024. [Layerskip: Enabling early exit inference and self-speculative decoding](https://md.research.elamur.ai/free/en/papers/elhoushi-2024-layerskip-enabling-early-exit-inference/). In Association for Comput…
13. Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. Koala: A dialogue model for academic research. [find](https://scholar.google.com/scholar?q=Xinyang+Geng%2C+Arnav+Gudibande%2C+Hao+Liu%2C+Eric+Wallace%2C+Pieter+Abbeel%2C+Sergey+Levine%2C+and+Dawn+Song%2E+2023%2E+Koala%3A+A+dialogue+model+for+academic+research%2E)
14. Xinyang Geng and Hao Liu. 2023. Openllama: An open reproduction of llama. [find](https://scholar.google.com/scholar?q=Xinyang+Geng+and+Hao+Liu%2E+2023%2E+Openllama%3A+An+open+reproduction+of+llama%2E)
15. Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. [Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space](https://md.research.elamur.ai/free/en/papers/geva-2022-transformer-feed-forward-layers-build/). In Conference on Empirical Methods in Natural Language Processing.
16. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. Minillm: Knowledge distillation of large language models. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yuxian+Gu%2C+Li+Dong%2C+Furu+Wei%2C+and+Minlie+Huang%2E+2024%2E+Minillm%3A+Knowledge+distillation+of+large+language+models%2E+In+International+Conference+on+Learning+Representations%2E)
17. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In Conference on Computer Vision and Pattern Recognition. sizes. In Findings of the Association for Computational Linguistics.
18. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations.
19. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. Tiny{bert}: Distilling {bert} for natural language understanding. In Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Xiaoqi+Jiao%2C+Yichun+Yin%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Xiao+Chen%2C+Linlin+Li%2C+Fang+Wang%2C+and+Qun+Liu%2E+2020%2E+Tiny%7Bbert%7D%3A+Distilling+%7Bbert%7D+for+natural+language+understanding%2E+In+Conference+on+Empirical+Metho)
20. Yoon Kim and Alexander M. Rush. 2016. [Sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/). In Conference on Empirical Methods in Natural Language Processing.
21. Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. 2024. [Distillm: Towards streamlined distillation for large language models](https://md.research.elamur.ai/free/en/papers/ko-2024-distillm-towards-streamlined-distillation-large/). In International Conference on Machine Learning.
22. Bei Li, Quan Du, Tao Zhou, Shuhan Zhou, Xin Zeng, 2 Tong Xiao1, and Jingbo Zhu. 2022. Ode transformer: An ordinary differential equation-inspired model for neural machine translation. In Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bei+Li%2C+Quan+Du%2C+Tao+Zhou%2C+Shuhan+Zhou%2C+Xin+Zeng%2C+2+Tong+Xiao1%2C+and+Jingbo+Zhu%2E+2022%2E+Ode+transformer%3A+An+ordinary+differential+equation%2Dinspired+model+for+neural+machine+translation%2E+In+Association+f)
23. Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. 2023. Less is more: Task-aware layer-wise distillation for language model compression. In International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Chen+Liang%2C+Simiao+Zuo%2C+Qingru+Zhang%2C+Pengcheng+He%2C+Weizhu+Chen%2C+and+Tuo+Zhao%2E+2023%2E+Less+is+more%3A+Task%2Daware+layer%2Dwise+distillation+for+language+model+compression%2E+In+International+Conference+on+Mac)
24. Alexander Lin, Jeremy Wohlwend, Howard Chen, and Tao Lei. 2020. Autoregressive knowledge distillation through imitation learning. In Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Alexander+Lin%2C+Jeremy+Wohlwend%2C+Howard+Chen%2C+and+Tao+Lei%2E+2020%2E+Autoregressive+knowledge+distillation+through+imitation+learning%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2E)
25. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out.
26. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). arXiv preprint arXiv:1503.02531.
27. Jiaheng Liu, Chenchen Zhang, Jinyang Guo, Yuanxing Zhang, Haoran Que, Ken Deng, Zhiqi Bai, Jie Liu, Ge Zhang, Jiakai Wang, Yanan Wu, Congnan Liu, Wenbo Su, Jiamang Wang, Lin Qu, and Bo Zheng. 2024. [Ddk: Distilling domain knowledge for efficient large language models](https://md.research.elamur.ai/free/en/papers/liu-2024-ddk-distilling-domain-knowledge-efficient/). In Advances in Neural Informatio…
28. Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2023. [Unnatural instructions: Tuning language models with (almost) no human labor](https://md.research.elamur.ai/free/en/papers/honovich-2023-unnatural-instructions-tuning-language-models/). In Annual Meeting of the Association for Computational Linguistics.
29. Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2019. Understanding and improving transformer from a multi-particle dynamic system point of view. arXiv preprint arXiv:1906.02762. — [arXiv:1906.02762](https://arxiv.org/abs/1906.02762) [find](https://scholar.google.com/scholar?q=Yiping+Lu%2C+Zhuohan+Li%2C+Di+He%2C+Zhiqing+Sun%2C+Bin+Dong%2C+Tao+Qin%2C+Liwei+Wang%2C+and+Tie%2DYan+Liu%2E+2019%2E+Understanding+and+improving+transformer+from+a+multi%2Dparticle+dynamic+system+point+of+view%2E+arXiv+prepr)
30. Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong… [find](https://scholar.google.com/scholar?q=Cheng%2DYu+Hsieh%2C+Chun%2DLiang+Li%2C+Chih%2DKuan+Yeh%2C+Hootan+Nakhost%2C+Yasuhisa+Fujii%2C+Alex+Ratner%2C+Ranjay+Krishna%2C+Chen%2DYu+Lee%2C+and+Tomas+Pfister%2E+2023%2E+Distilling+step%2Dby%2Dstep%21+outperforming+larger+language+)
31. Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. 2024. Compact language models via pruning and knowledge distillation. Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Saurav+Muralidharan%2C+Sharath+Turuvekere+Sreenivas%2C+Raviraj+Joshi%2C+Marcin+Chochowski%2C+Mostofa+Patwary%2C+Mohammad+Shoeybi%2C+Bryan+Catanzaro%2C+Jan+Kautz%2C+and+Pavlo+Molchanov%2E+2024%2E+Compact+language+models+v)
32. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Annual Meeting of the Association for Computational Linguistics. nostalgebraist. 2020. Interpreting gpt:… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A%2E+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2023%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+In+Annu)
33. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Ku… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Atharva+Naik%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Anjana+Arunkumar%2C+David+Stap%2C+Eshaan+Pathak%2C+Giannis+Karaman)
34. OpenAI. 2022. Introducing chatgpt. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2022%2E+Introducing+chatgpt%2E)
35. OpenAI. 2024. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2412.19437.
36. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. [Instruction tuning with gpt-4](https://md.research.elamur.ai/free/en/papers/peng-2023-instruction-tuning-gpt-4/). arXiv preprint arXiv:2304.03277.
37. Qwen. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=Qwen%2E+2024%2E+Qwen2%2E5+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E15115%2E)
38. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
39. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In Workshop on Energy Efficient Machine Learning and Cognitive Computing - Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+2019%2E+Distilbert%2C+a+distilled+version+of+bert%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E+In+Workshop+on+Energy+Efficient+Machine+Learning+and)
40. Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. 2025. Layer by layer: Uncovering hidden representations in language models. arXiv preprint arXiv:2502.02013. [find](https://scholar.google.com/scholar?q=Oscar+Skean%2C+Md+Rifat+Arefin%2C+Dan+Zhao%2C+Niket+Patel%2C+Jalal+Naghiyev%2C+Yann+LeCun%2C+and+Ravid+Shwartz%2DZiv%2E+2025%2E+Layer+by+layer%3A+Uncovering+hidden+representations+in+language+models%2E+arXiv+preprint+arXiv)
41. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca). [↗](https://github.com/tatsu-lab/stanford_alpaca)
42. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
43. Hale F Trotter. 1959. On the product of semi-groups of operators. Proceedings of the American Mathematical Society, 10(4):545–551. [find](https://scholar.google.com/scholar?q=Hale+F+Trotter%2E+1959%2E+On+the+product+of+semi%2Dgroups+of+operators%2E+Proceedings+of+the+American+Mathematical+Society%2C+10%284%29%3A545%E2%80%93551%2E)
44. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems.
45. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Wenhui+Wang%2C+Furu+Wei%2C+Li+Dong%2C+Hangbo+Bao%2C+Nan+Yang%2C+and+Ming+Zhou%2E+2020%2E+Minilm%3A+Deep+self%2Dattention+distillation+for+task%2Dagnostic+compression+of+pre%2Dtrained+transformers%2E+In+Advances+in+Neural+Inf)
46. Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. 2023. [f-divergence minimization for sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/wen-2023-f-divergence-minimization-sequence-level/). In Annual Meeting of the Association for Computational Linguistics.
47. Taiqiang Wu, Chaofan Tao, Jiahao Wang, Runming Yang, Zhe Zhao, and Ngai Wong. 2025. [Rethinking kullback-leibler divergence in knowledge distillation for large language models](https://md.research.elamur.ai/free/en/papers/wu-2025-rethinking-kullback-leibler-divergence-knowledge/). In International Conference on Computational Linguistics.
48. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2024. [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/). In The Twelfth International Conference on Learning Representations.
49. Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024a. [Tinyllama: An open-source small language model](https://md.research.elamur.ai/free/en/papers/zhang-2024-tinyllama-open-source-small-language/). arXiv preprint arXiv:2401.02385.
50. Songming Zhang, Xue Zhang, Zengkui Sun, Yufeng Chen, and Jinan Xu. 2024b. [Dual-space knowledge distillation for large language models](https://md.research.elamur.ai/free/en/papers/zhang-2024-dual-space-knowledge-distillation-large/). In Conference on Empirical Methods in Natural Language Processing, pages 18164–18181.
51. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Advances in Neural Information Processing Systems.
52. Qihuang Zhong, Liang Ding, Li Shen, Juhua Liu, Bo Du, and Dacheng Tao. 2024. Revisiting knowledge distillation for autoregressive language models. In Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qihuang+Zhong%2C+Liang+Ding%2C+Li+Shen%2C+Juhua+Liu%2C+Bo+Du%2C+and+Dacheng+Tao%2E+2024%2E+Revisiting+knowledge+distillation+for+autoregressive+language+models%2E+In+Association+for+Computational+Linguistics%2E)

*Updated 1 November 2026*