---
title: "Multi-Granularity Semantic Revision for Large Language Model"
description: "The paper addresses white-box knowledge distillation (KD) for auto-regressive LLMs, where a compact… 44 references, cited by 2 papers here."
authors: ["Xiaoyu Liu", "Yun Zhang", "Wei Li", "Simiao Li", "Xudong Huang", "Hanting Chen", "Yehui Tang", "Jie Hu", "Zhiwei Xiong", "Yunhe Wang"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.212/"
doi: "10.18653/v1/2026.acl-long.212"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2026-multi-granularity-semantic-revision-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2026-multi-granularity-semantic-revision-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/8/) / Multi-Granularity Semantic Revision for Large Language Model Distillation

# Multi-Granularity Semantic Revision for Large Language Model Distillation

**Authors:** [Xiaoyu Liu](https://md.research.elamur.ai/free/en/authors/xiaoyu-liu/), Yun Zhang, [Wei Li](https://md.research.elamur.ai/free/en/authors/wei-li/), Simiao Li, Xudong Huang, [Hanting Chen](https://md.research.elamur.ai/free/en/authors/hanting-chen/), [Yehui Tang](https://md.research.elamur.ai/free/en/authors/yehui-tang/), Jie Hu, Zhiwei Xiong and [Yunhe Wang](https://md.research.elamur.ai/free/en/authors/yunhe-wang/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.212/) · [PDF](https://aclanthology.org/2026.acl-long.212.pdf) · [DOI 10.18653/v1/2026.acl-long.212](https://doi.org/10.18653/v1/2026.acl-long.212) · [OpenAlex](https://openalex.org/W4400703153)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/liu-2026-multi-granularity-semantic-revision-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses white-box knowledge distillation (KD) for auto-regressive LLMs, where a compact student learns from the full output distributions of a larger teacher. The authors identify two weaknesses in recent methods such as GKD, MiniLLM and Distillm. First, training on student-generated sequences lets the student's own mistakes contaminate the training signal. Second, standard divergence losses have trouble focusing on the informative part of very large vocabulary distributions.

Their remedy works at three granularities:
- **Sequence level:** a correction-and-regeneration step finds the first token where student and teacher diverge most, substitutes the teacher's choice, and lets generation continue from that point.
- **Token level:** a clipped KL variant uses a small learnable network to keep only the high-density region of the teacher's distribution plus the target class.
- **Span level:** phrase spans from an external chunker are used to align intra-span probability relations between teacher and student.

Across four model families from 0.1B to 13B parameters, the combined method beats prior KD approaches. This matters for building smaller deployable models. However, the dependence on a chunker is a practical concern for less-resourced languages such as Ukrainian.

## Key points
- **Components:**
  - SCRG (sequence correction and re-generation) works with both on-policy (GKD-style) and off-policy (Distillm-style) sampling schedules.
  - DAC-KL (distribution adaptive clipping Kullback-Leibler) is the token-level loss.
  - Span priors provide the span-level alignment.
- **Models tested:** LLAMA2, Open-LLAMA2, OPT and GPT2 series, spanning 0.1B–13B parameters.
- **Ablations:**
  - SCRG improves on existing student-generation strategies under both sampling regimes.
  - DAC-KL outperforms alternative distillation losses.
  - DAC-KL works best when it covers both the high-density classes and the target class.
- **Limitations:**
  - Spans come from external chunkers such as SpaCy, NLTK, or JieBa for Chinese.
  - For low-resource languages, the authors suggest using models like GPT-4 to generate spans.
  - Experiments covered only limited linguistic domains.
- **Venue:** ACL 2026 (Long Papers); the work was carried out mainly at Huawei together with USTC and HKUST (GZ).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Hong Kong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/hong-kong-university-of-science-and-technology/) — University, Hong Kong, China · 142 works · 25,136 citations
- [University of Science and Technology of China](https://md.research.elamur.ai/free/en/universities/university-of-science-and-technology-of-china/) — University, Hefei, China · 108 works · 112,698 citations
- [Huawei Technologies](https://md.research.elamur.ai/free/en/companies/huawei-technologies/) — Company, Shenzhen, China · 75 works · 12,975 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 242 works

## Cite

Cite as: Xiaoyu Liu, Yun Zhang, Wei Li, Simiao Li, Xudong Huang, Hanting Chen, Yehui Tang, Jie Hu, Zhiwei Xiong and Yunhe Wang. 2026. Multi-Granularity Semantic Revision for Large Language Model Distillation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liu2026multi,
  title = {{Multi-Granularity Semantic Revision for Large Language Model Distillation}},
  author = {Xiaoyu Liu, Yun Zhang, Wei Li, Simiao Li, Xudong Huang, Hanting Chen, Yehui Tang, Jie Hu, Zhiwei Xiong and Yunhe Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.212},
  url = {https://doi.org/10.18653/v1/2026.acl-long.212}
}
```

- [← previous MTQE.en-he: Machine Translation Quality Estimation for English-Hebrew](https://md.research.elamur.ai/free/en/papers/rosenbaum-2026-mtqe-en-he-machine-translation/)
- [next → MultiFileTest: A Multi-File-Level LLM Unit Test Generation Benchmark and Impact of Error Fixing Mechanisms](https://md.research.elamur.ai/free/en/papers/wang-2026-multifiletest-multi-file-level-llm/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/8/)

## References

44 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Romero Adriana, Ballas Nicolas, K Samira Ebrahimi, Chassang Antoine, Gatta Carlo, and Bengio Yoshua. 2015. Fitnets: Hints for thin deep nets. Proc. ICLR, 2(3):1. [find](https://scholar.google.com/scholar?q=Romero+Adriana%2C+Ballas+Nicolas%2C+K+Samira+Ebrahimi%2C+Chassang+Antoine%2C+Gatta+Carlo%2C+and+Bengio+Yoshua%2E+2015%2E+Fitnets%3A+Hints+for+thin+deep+nets%2E+Proc%2E+ICLR%2C+2%283%29%3A1%2E)
3. Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. 2024. [On-policy distillation of language models: Learning from self-generated mistakes](https://md.research.elamur.ai/free/en/papers/agarwal-2023-policy-distillation-language-models-learning/). In The Twelfth International Conference on Learning Representations.
4. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28. [find](https://scholar.google.com/scholar?q=Samy+Bengio%2C+Oriol+Vinyals%2C+Navdeep+Jaitly%2C+and+Noam+Shazeer%2E+2015%2E+Scheduled+sampling+for+sequence+prediction+with+recurrent+neural+networks%2E+Advances+in+neural+information+processing+systems%2C+28%2E)
5. TomB. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Thomas Henighan, Rewon Child, Aditya Ramesh, DanielM. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
6. Hongzhan Chen, Xiaojun Quan, Hehong Chen, Ming Yan, and Ji Zhang. 2024. [Knowledge distillation for closed-source language models](https://md.research.elamur.ai/free/en/papers/chen-2024-knowledge-distillation-black-box-large/). arXiv preprint arXiv:2401.07013.
7. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, and 1 others. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See [https://vicuna.lmsys](https://vicuna.lmsys). org (accessed 14 April 2023), 2(3):6. [↗](https://vicuna.lmsys)
8. Free Dolly. 2023. Introducing the world’s first truly open instruction-tuned llm. databricks. com. [find](https://scholar.google.com/scholar?q=Free+Dolly%2E+2023%2E+Introducing+the+world%E2%80%99s+first+truly+open+instruction%2Dtuned+llm%2E+databricks%2E+com%2E)
9. Xinyang Geng and Hao Liu. 2023. Openllama: An open reproduction of llama. URL: [https://github.com/openlm-research/open_llama](https://github.com/openlm-research/open_llama). [↗](https://github.com/openlm-research/open_llama)
10. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yuxian+Gu%2C+Li+Dong%2C+Furu+Wei%2C+and+Minlie+Huang%2E+2023%2E+Minillm%3A+Knowledge+distillation+of+large+language+models%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2E)
11. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). arXiv preprint arXiv:1503.02531.
12. Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022. [Unnatural instructions: Tuning language models with (almost) no human labor](https://md.research.elamur.ai/free/en/papers/honovich-2023-unnatural-instructions-tuning-language-models/). arXiv preprint arXiv:2212.09689.
13. Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. [Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes](https://md.research.elamur.ai/free/en/papers/hsieh-2023-distilling-step-step-outperforming-larger/). arXiv preprint arXiv:2305.02301.
14. Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. 2023. Lion: Adversarial distillation of closed-source large language model. arXiv preprint arXiv:2305.12870. — [arXiv:2305.12870](https://arxiv.org/abs/2305.12870) [find](https://scholar.google.com/scholar?q=Yuxin+Jiang%2C+Chunkit+Chan%2C+Mingyang+Chen%2C+and+Wei+Wang%2E+2023%2E+Lion%3A+Adversarial+distillation+of+closed%2Dsource+large+language+model%2E+arXiv+preprint+arXiv%3A2305%2E12870%2E+%E2%80%94+%5BarXiv%3A2305%2E12870%5D%28https%3A%2F%2Farxiv%2E)
15. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. [Tinybert: Distilling bert for natural language understanding](https://md.research.elamur.ai/free/en/papers/jiao-2020-tinybert-distilling-bert-natural-language/). arXiv preprint arXiv:1909.10351.
16. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361.
17. Minsang Kim and Seungjun Baek. 2024. Measuring sample importance in data pruning for training llms from a data compression perspective. arXiv preprint arXiv:2406.14124. — [arXiv:2406.14124](https://arxiv.org/abs/2406.14124) [find](https://scholar.google.com/scholar?q=Minsang+Kim+and+Seungjun+Baek%2E+2024%2E+Measuring+sample+importance+in+data+pruning+for+training+llms+from+a+data+compression+perspective%2E+arXiv+preprint+arXiv%3A2406%2E14124%2E+%E2%80%94+%5BarXiv%3A2406%2E14124%5D%28https%3A%2F%2Far)
18. Yoon Kim and Alexander M Rush. 2016. [Sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/). arXiv preprint arXiv:1606.07947.
19. Tibor Kiss and Jan Strunk. 2006. [Unsupervised multilingual sentence boundary detection](https://md.research.elamur.ai/free/en/papers/kiss-2006-unsupervised-multilingual-sentence-boundary-detection/). Computational linguistics, 32(4):485–525.
20. Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. 2024. [Distillm: Towards streamlined distillation for large language models](https://md.research.elamur.ai/free/en/papers/ko-2024-distillm-towards-streamlined-distillation-large/). arXiv preprint arXiv:2402.03898.
21. Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. 2023. Less is more: Task-aware layer-wise distillation for language model compression. In International Conference on Machine Learning, pages 20852–20867. PMLR. [find](https://scholar.google.com/scholar?q=Chen+Liang%2C+Simiao+Zuo%2C+Qingru+Zhang%2C+Pengcheng+He%2C+Weizhu+Chen%2C+and+Tuo+Zhao%2E+2023%2E+Less+is+more%3A+Task%2Daware+layer%2Dwise+distillation+for+language+model+compression%2E+In+International+Conference+on+Mac)
22. Kevin J Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, and Lawrence Carin. 2020. Mixkd: Towards efficient distillation of large-scale language models. arXiv preprint arXiv:2011.00593. — [arXiv:2011.00593](https://arxiv.org/abs/2011.00593) [find](https://scholar.google.com/scholar?q=Kevin+J+Liang%2C+Weituo+Hao%2C+Dinghan+Shen%2C+Yufan+Zhou%2C+Weizhu+Chen%2C+Changyou+Chen%2C+and+Lawrence+Carin%2E+2020%2E+Mixkd%3A+Towards+efficient+distillation+of+large%2Dscale+language+models%2E+arXiv+preprint+arXiv%3A20)
23. Alexander Lin, Jeremy Wohlwend, Howard Chen, and Tao Lei. 2020. Autoregressive knowledge distillation through imitation learning. arXiv preprint arXiv:2009.07253. [find](https://scholar.google.com/scholar?q=Alexander+Lin%2C+Jeremy+Wohlwend%2C+Howard+Chen%2C+and+Tao+Lei%2E+2020%2E+Autoregressive+knowledge+distillation+through+imitation+learning%2E+arXiv+preprint+arXiv%3A2009%2E07253%2E)
24. Chin-Yew Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pages 74–81.
25. Chang Liu, Chongyang Tao, Jiazhan Feng, and Dongyan Zhao. 2022. Multi-granularity structural knowledge distillation for language model compression. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1001–1011. [find](https://scholar.google.com/scholar?q=Chang+Liu%2C+Chongyang+Tao%2C+Jiazhan+Feng%2C+and+Dongyan+Zhao%2E+2022%2E+Multi%2Dgranularity+structural+knowledge+distillation+for+language+model+compression%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Ass)
26. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 35:27730–27744.
27. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. [Instruction tuning with gpt-4](https://md.research.elamur.ai/free/en/papers/peng-2023-instruction-tuning-gpt-4/). arXiv preprint arXiv:2304.03277.
28. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI b… [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+Language+models+are+unsupervised+multitask+learners%2E+Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amode)
29. Mrigank Raman, Pranav Mani, Davis Liang, and Zachary Lipton. 2023. For distillation, tokens are not all you need. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following. [find](https://scholar.google.com/scholar?q=Mrigank+Raman%2C+Pranav+Mani%2C+Davis+Liang%2C+and+Zachary+Lipton%2E+2023%2E+For+distillation%2C+tokens+are+not+all+you+need%2E+In+NeurIPS+2023+Workshop+on+Instruction+Tuning+and+Instruction+Following%2E)
30. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+2019%2E+Distilbert%2C+a+distilled+version+of+bert%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E+arXiv+preprint+arXiv%3A1910%2E01108%2E)
31. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. [Patient knowledge distillation for bert model compression](https://md.research.elamur.ai/free/en/papers/sun-2019-patient-knowledge-distillation-bert-model/). arXiv preprint arXiv:1908.09355.
32. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [find](https://scholar.google.com/scholar?q=Rohan+Taori%2C+Ishaan+Gulrajani%2C+Tianyi+Zhang%2C+Yann+Dubois%2C+Xuechen+Li%2C+Carlos+Guestrin%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+2023%2E+Stanford+alpaca%3A+An+instruction%2Dfollowing+llama+model%2E)
33. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth’ee Lacroix, Baptiste Rozi‘ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. Hugo Touvron… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
34. Fusheng Wang, Jianhao Yan, Fandong Meng, and Jie Zhou. 2021. Selective knowledge distillation for neural machine translation. arXiv preprint arXiv:2105.12967. [find](https://scholar.google.com/scholar?q=Fusheng+Wang%2C+Jianhao+Yan%2C+Fandong+Meng%2C+and+Jie+Zhou%2E+2021%2E+Selective+knowledge+distillation+for+neural+machine+translation%2E+arXiv+preprint+arXiv%3A2105%2E12967%2E)
35. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022a. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2022a%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+arXiv+p)
36. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, and 1 others. 2022b. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. arXiv preprint arX… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Anjana+Arunkumar%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Atharva+Naik%2C+David+Stap%2C+and+1+others%2E+2022b%2E+Super%2Dnat)
37. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, EdH. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/).
38. Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. 2023. [f-divergence minimization for sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/wen-2023-f-divergence-minimization-sequence-level/). arXiv preprint arXiv:2307.15190.
39. Mengyang Yuan, Bo Lang, and Fengnan Quan. 2024. Student-friendly knowledge distillation. Knowledge-Based Systems, 296:111915. [find](https://scholar.google.com/scholar?q=Mengyang+Yuan%2C+Bo+Lang%2C+and+Fengnan+Quan%2E+2024%2E+Student%2Dfriendly+knowledge+distillation%2E+Knowledge%2DBased+Systems%2C+296%3A111915%2E)
40. Songming Zhang, Xue Zhang, Zengkui Sun, Yufeng Chen, and Jinan Xu. 2024. [Dual-space knowledge distillation for large language models](https://md.research.elamur.ai/free/en/papers/zhang-2024-dual-space-knowledge-distillation-large/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18164–18181.
41. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer… — [OPT: Open Pre-trained Transformer Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/)
42. Yuanchi Zhang, Peng Li, Maosong Sun, and Yang Liu. 2023. Continual knowledge distillation for neural machine translation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7978– 7996. [find](https://scholar.google.com/scholar?q=Yuanchi+Zhang%2C+Peng+Li%2C+Maosong+Sun%2C+and+Yang+Liu%2E+2023%2E+Continual+knowledge+distillation+for+neural+machine+translation%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational)
43. Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. 2022. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 11953–11962. [find](https://scholar.google.com/scholar?q=Borui+Zhao%2C+Quan+Cui%2C+Renjie+Song%2C+Yiyu+Qiu%2C+and+Jiajun+Liang%2E+2022%2E+Decoupled+knowledge+distillation%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+computer+vision+and+pattern+recognition%2C+pages+11953)
44. Qihuang Zhong, Liang Ding, Li Shen, Juhua Liu, Bo Du, and Dacheng Tao. 2024. Revisiting knowledge distillation for autoregressive language models. arXiv preprint arXiv:2402.11890. [find](https://scholar.google.com/scholar?q=Qihuang+Zhong%2C+Liang+Ding%2C+Li+Shen%2C+Juhua+Liu%2C+Bo+Du%2C+and+Dacheng+Tao%2E+2024%2E+Revisiting+knowledge+distillation+for+autoregressive+language+models%2E+arXiv+preprint+arXiv%3A2402%2E11890%2E)

*Updated 31 October 2026*