---
title: "f-Divergence Minimization for Sequence-Level Knowledge"
description: "To address both, they introduce f-DISTILL, which treats sequence-level distillation as minimizing a… 62 references, cited by 73 papers here."
authors: ["Yuqiao Wen", "Zichao Li", "Wenyu Du", "Lili Mou"]
year: 2023
venue: "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2023.acl-long.605/"
doi: "10.18653/v1/2023.acl-long.605"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wen-2023-f-divergence-minimization-sequence-level/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wen-2023-f-divergence-minimization-sequence-level/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / f-Divergence Minimization for Sequence-Level Knowledge Distillation

# f-Divergence Minimization for Sequence-Level Knowledge Distillation

**Authors:** Yuqiao Wen, [Zichao Li](https://md.research.elamur.ai/free/en/authors/zichao-li/), Wenyu Du and [Lili Mou](https://md.research.elamur.ai/free/en/authors/lili-mou/) · **Year:** 2023 · **Venue:** [Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2023](https://md.research.elamur.ai/free/en/venues/acl/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.acl-long.605/) · [PDF](https://aclanthology.org/2023.acl-long.605.pdf) · [DOI 10.18653/v1/2023.acl-long.605](https://doi.org/10.18653/v1/2023.acl-long.605) · [OpenAlex](https://openalex.org/W4385570255)

**Citations:** 18 citations (OpenAlex, 2026-10-14) · [73](https://md.research.elamur.ai/free/en/papers/wen-2023-f-divergence-minimization-sequence-level/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper tackles knowledge distillation (KD) for text generation, where a compact student model must reproduce the output distribution of a much larger teacher. The authors argue that the standard KL-based approach (SeqKD) makes the student spread its probability too thinly across the teacher's outputs, a failure called mode averaging. Reverse-KL-style objectives such as ENGINE have the opposite flaw: the student fixates on a few high-probability regions, called mode collapsing. To address both, they introduce f-DISTILL, which treats sequence-level distillation as minimizing a general f-divergence. The framework yields four variants: KL, RKL, JS and TVD. The authors show that SeqKD and ENGINE are approximations of the KL and RKL cases. They also derive step-wise, word-level decompositions that make the intractable sequence-level losses computable, either exactly or as an upper bound. Teacher samples are drawn offline to reduce training cost. On four generation benchmarks, f-DISTILL beats existing distribution-matching KD methods. The symmetric losses (JS, TVD) perform best, which suggests that balancing the two failure modes produces better small models for deployment.

## Key points
- Four f-DISTILL variants are proposed: KL, reverse KL (RKL), Jensen–Shannon (JS) and total variation distance (TVD). SeqKD and ENGINE emerge as approximations of the KL and RKL variants.
- The evaluation covers four tasks:
  - DART for data-to-text generation
  - XSum for summarization
  - WMT16 EN-RO for machine translation
  - Commonsense Dialogue for dialogue generation
- The KL variant uses the teacher's soft labels instead of SeqKD's hard beam-search sequences, which gives the student more information.
- f-DISTILL also adds gains on top of representation-matching KD methods. Code is available at github.com/MANGA-UOFA/fdistill.
- The authors note several limitations:
  - Training is slower than with SeqKD or ENGINE, although inference speed is unaffected.
  - The experiments took about 2000 GPU hours.
  - No multi-run statistics are reported.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [McGill University](https://md.research.elamur.ai/free/en/universities/mcgill-university/) — University, Montreal, Canada · 51 works · 10,476 citations
- [University of Hong Kong](https://md.research.elamur.ai/free/en/universities/university-of-hong-kong/) — University, Pok Fu Lam, China · 44 works · 10,936 citations
- [Canadian Institute for Advanced Research](https://md.research.elamur.ai/free/en/labs/canadian-institute-for-advanced-research/) — Research institute, Toronto, Canada · 17 works · 16,274 citations
- [University of Alberta](https://md.research.elamur.ai/free/en/universities/university-of-alberta/) — University, Edmonton, Canada · 23 works · 2,655 citations
- [Alberta Machine Intelligence Institute](https://md.research.elamur.ai/free/en/labs/alberta-machine-intelligence-institute/) — Research institute, Alberta, Canada · 2 works · 229 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 77 works

## Cite

Cite as: Yuqiao Wen, Zichao Li, Wenyu Du and Lili Mou. 2023. f-Divergence Minimization for Sequence-Level Knowledge Distillation. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wen2023f,
  title = {{f-Divergence Minimization for Sequence-Level Knowledge Distillation}},
  author = {Yuqiao Wen, Zichao Li, Wenyu Du and Lili Mou},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.605},
  url = {https://doi.org/10.18653/v1/2023.acl-long.605}
}
```

- [← previous ChatHaruhi: Reviving Anime Character in Reality via Large Language Model](https://md.research.elamur.ai/free/en/papers/li-2023-chatharuhi-reviving-anime-character-reality/)
- [next → Skywork: A More Open Bilingual Foundation Model](https://md.research.elamur.ai/free/en/papers/wei-2023-skywork-more-open-bilingual/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

62 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. S. M. Ali and S. D. Silvey. 1966. A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society, 28(1):131–142. [find](https://scholar.google.com/scholar?q=S%2E+M%2E+Ali+and+S%2E+D%2E+Silvey%2E+1966%2E+A+general+class+of+coefficients+of+divergence+of+one+distribution+from+another%2E+Journal+of+the+Royal+Statistical+Society%2C+28%281%29%3A131%E2%80%93142%2E)
2. Satanjeev Banerjee and Alon Lavie. 2005. [METEOR: An automatic metric for MT evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72.
3. Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. 2020. [PLATO: Pre-trained dialogue generation model with discrete latent variable](https://md.research.elamur.ai/free/en/papers/bao-2020-plato-pre-trained-dialogue-generation/). In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 85–96.
4. Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. [Findings of the 2019 Conference on Machine Translation (…](https://md.research.elamur.ai/free/en/papers/barrault-2019-findings-2019-conference-machine-translation/)
5. Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning. Springer. [find](https://scholar.google.com/scholar?q=Christopher+M%2E+Bishop%2E+2006%2E+Pattern+Recognition+and+Machine+Learning%2E+Springer%2E)
6. [Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi,…](https://md.research.elamur.ai/free/en/papers/bojar-2016-findings-2016-conference-machine-translation/)
7. Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model compression. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, page 535–541. [find](https://scholar.google.com/scholar?q=Cristian+Bucilua%CC%8C%2C+Rich+Caruana%2C+and+Alexandru+Niculescu%2DMizil%2E+2006%2E+Model+compression%2E+In+Proceedings+of+the+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining%2C+page+535%E2%80%93541%2E)
8. Chun Fan, Jiwei Li, Tianwei Zhang, Xiang Ao, Fei Wu, Yuxian Meng, and Xiaofei Sun. 2021. Layer-wise model pruning based on mutual information. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 3079–3090. [find](https://scholar.google.com/scholar?q=Chun+Fan%2C+Jiwei+Li%2C+Tianwei+Zhang%2C+Xiang+Ao%2C+Fei+Wu%2C+Yuxian+Meng%2C+and+Xiaofei+Sun%2E+2021%2E+Layer%2Dwise+model+pruning+based+on+mutual+information%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+)
9. Gongfan Fang, Yifan Bao, Jie Song, Xinchao Wang, Donglin Xie, Chengchao Shen, and Mingli Song. 2021. Mosaicking to distill: Knowledge distillation from out-of-domain data. In Advances in Neural Information Processing Systems, pages 11920–11932. [find](https://scholar.google.com/scholar?q=Gongfan+Fang%2C+Yifan+Bao%2C+Jie+Song%2C+Xinchao+Wang%2C+Donglin+Xie%2C+Chengchao+Shen%2C+and+Mingli+Song%2E+2021%2E+Mosaicking+to+distill%3A+Knowledge+distillation+from+out%2Dof%2Ddomain+data%2E+In+Advances+in+Neural+Inform)
10. Jonathan Frankle and Michael Carbin. 2018. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jonathan+Frankle+and+Michael+Carbin%2E+2018%2E+The+lottery+ticket+hypothesis%3A+Finding+sparse%2C+trainable+neural+networks%2E+In+International+Conference+on+Learning+Representations%2E)
11. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Ian+Goodfellow%2C+Jean+Pouget%2DAbadie%2C+Mehdi+Mirza%2C+Bing+Xu%2C+David+Warde%2DFarley%2C+Sherjil+Ozair%2C+Aaron+Courville%2C+and+Yoshua+Bengio%2E+2014%2E+Generative+adversarial+nets%2E+In+Advances+in+Neural+Information+Pr)
12. Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. 2018. [Non-autoregressive neural machine translation](https://md.research.elamur.ai/free/en/papers/gu-2017-non-autoregressive-neural-machine-translation/). In International Conference on Learning Representations.
13. Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). arXiv preprint arXiv:1503.02531.
14. Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah Smith. 2020. [Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation](https://md.research.elamur.ai/free/en/papers/kasai-2020-deep-encoder-shallow-decoder-reevaluating/). In International Conference on Learning Representations.
15. Moniba Keymanesh, Adrian Benton, and Mark Dredze. 2022. What makes data-to-text generation hard for pretrained language models? In Proceedings of the Workshop on Natural Language Generation, Evaluation, and Metrics, pages 539–554. [find](https://scholar.google.com/scholar?q=Moniba+Keymanesh%2C+Adrian+Benton%2C+and+Mark+Dredze%2E+2022%2E+What+makes+data%2Dto%2Dtext+generation+hard+for+pretrained+language+models%3F+In+Proceedings+of+the+Workshop+on+Natural+Language+Generation%2C+Evaluatio)
16. Yoon Kim and Alexander M. Rush. 2016. [Sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/). In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1317–1327.
17. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+International+Conference+on+Learning+Representations%2E)
18. Rémi Lebret, David Grangier, and Michael Auli. 2016. [Neural text generation from structured data with application to the biography domain](https://md.research.elamur.ai/free/en/papers/lebret-2016-neural-text-generation-structured-data/). In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1203–1213.
19. Yann LeCun, John Denker, and Sara Solla. 1989. Optimal brain damage. In Advances in Neural Information Processing Systems, pages 598–605. [find](https://scholar.google.com/scholar?q=Yann+LeCun%2C+John+Denker%2C+and+Sara+Solla%2E+1989%2E+Optimal+brain+damage%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+598%E2%80%93605%2E)
20. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the Annual Meeting of the Asso…
21. Jianquan Li, Xiaokang Liu, Honghong Zhao, Ruifeng Xu, Min Yang, and Yaohong Jin. 2020. BERT-EMD: Many-to-many layer mapping for BERT compression with earth mover’s distance. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 3009–3018. [find](https://scholar.google.com/scholar?q=Jianquan+Li%2C+Xiaokang+Liu%2C+Honghong+Zhao%2C+Ruifeng+Xu%2C+Min+Yang%2C+and+Yaohong+Jin%2E+2020%2E+BERT%2DEMD%3A+Many%2Dto%2Dmany+layer+mapping+for+BERT+compression+with+earth+mover%E2%80%99s+distance%2E+In+Proceedings+of+the+Conf)
22. Chenyang Huang, Hao Zhou, Osmar R. Zaïane, Lili Mou, and Lei Li. 2022. Non-autoregressive translation with layer-wise prediction and deep supervision. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 10776–10784. [find](https://scholar.google.com/scholar?q=Chenyang+Huang%2C+Hao+Zhou%2C+Osmar+R%2E+Za%C3%AFane%2C+Lili+Mou%2C+and+Lei+Li%2E+2022%2E+Non%2Dautoregressive+translation+with+layer%2Dwise+prediction+and+deep+supervision%2E+In+Proceedings+of+the+AAAI+Conference+on+Artifici)
23. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. [A diversity-promoting objective function for neural conversation models](https://md.research.elamur.ai/free/en/papers/li-2016-diversity-promoting-objective-function-neural/). In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1…
24. Xiaoqi Jiao, Huating Chang, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2021. Improving task-agnostic BERT distillation with layer mapping search. Neurocomputing, 461:194–203. [find](https://scholar.google.com/scholar?q=Xiaoqi+Jiao%2C+Huating+Chang%2C+Yichun+Yin%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Xiao+Chen%2C+Linlin+Li%2C+Fang+Wang%2C+and+Qun+Liu%2E+2021%2E+Improving+task%2Dagnostic+BERT+distillation+with+layer+mapping+search%2E+Neurocomputing)
25. Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016b. [Deep reinforcement learning for dialogue generation](https://md.research.elamur.ai/free/en/papers/li-2016-deep-reinforcement-learning-dialogue-generation/). In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1192–1202.
26. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP, pages 4163– 4174. [find](https://scholar.google.com/scholar?q=Xiaoqi+Jiao%2C+Yichun+Yin%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Xiao+Chen%2C+Linlin+Li%2C+Fang+Wang%2C+and+Qun+Liu%2E+2020%2E+TinyBERT%3A+Distilling+BERT+for+natural+language+understanding%2E+In+Findings+of+the+Association+for+C)
27. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, pages 4582–4597. [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+Internat)
28. Zhuoran Li, Chunming Hu, Xiaohui Guo, Junfan Chen, Wenyi Qin, and Richong Zhang. 2022. An unsupervised multiple-task and multiple-teacher model for cross-lingual named entity recognition. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 170–179. [find](https://scholar.google.com/scholar?q=Zhuoran+Li%2C+Chunming+Hu%2C+Xiaohui+Guo%2C+Junfan+Chen%2C+Wenyi+Qin%2C+and+Richong+Zhang%2E+2022%2E+An+unsupervised+multiple%2Dtask+and+multiple%2Dteacher+model+for+cross%2Dlingual+named+entity+recognition%2E+In+Proceedin)
29. Alexander Lin, Jeremy Wohlwend, Howard Chen, and Tao Lei. 2020. Autoregressive knowledge distillation through imitation learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 6121–6133. [find](https://scholar.google.com/scholar?q=Alexander+Lin%2C+Jeremy+Wohlwend%2C+Howard+Chen%2C+and+Tao+Lei%2E+2020%2E+Autoregressive+knowledge+distillation+through+imitation+learning%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Langu)
30. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81.
31. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. [How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation](https://md.research.elamur.ai/free/en/papers/liu-2016-how-not-evaluate-your-dialogue/). In Proceedings of the Conference on Empirical Methods in Natural Langua…
32. Liyuan Liu, Xiang Ren, Jingbo Shang, Xiaotao Gu, Jian Peng, and Jiawei Han. 2018. Efficient contextualized representation: Language model pruning for sequence labeling. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1215–1225. [find](https://scholar.google.com/scholar?q=Liyuan+Liu%2C+Xiang+Ren%2C+Jingbo+Shang%2C+Xiaotao+Gu%2C+Jian+Peng%2C+and+Jiawei+Han%2E+2018%2E+Efficient+contextualized+representation%3A+Language+model+pruning+for+sequence+labeling%2E+In+Proceedings+of+the+Conferenc)
33. Christos Louizos, Max Welling, and Diederik P Kingma. 2018. Learning sparse neural networks through L 0 regularization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Christos+Louizos%2C+Max+Welling%2C+and+Diederik+P+Kingma%2E+2018%2E+Learning+sparse+neural+networks+through+L+0+regularization%2E+In+International+Conference+on+Learning+Representations%2E)
34. [Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, C…](https://md.research.elamur.ai/free/en/papers/nan-2020-dart-open-domain-structured-data/)
35. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1797–1807. [find](https://scholar.google.com/scholar?q=Shashi+Narayan%2C+Shay+B%2E+Cohen%2C+and+Mirella+Lapata%2E+2018%2E+Don%E2%80%99t+give+me+the+details%2C+just+the+summary%21+Topic%2Daware+convolutional+neural+networks+for+extreme+summarization%2E+In+Proceedings+of+the+Confere)
36. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [BLEU: A method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 311–318.
37. Romain Paulus, Caiming Xiong, and Richard Socher. 2018. [A deep reinforced model for abstractive summarization](https://md.research.elamur.ai/free/en/papers/paulus-2017-deep-reinforced-model-abstractive-summarization/). In International Conference on Learning Representations.
38. Maja Popović. 2015. [chrF: character n-gram F-score for automatic MT evaluation](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/). In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395.
39. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. [Exploring the limits of transfer learning with a unified text-to-text Transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67.
40. Igal Sason and Sergio Verdú. 2016. f -divergence inequalities. IEEE Transactions on Information Theory, 62(11):5973–6006. [find](https://scholar.google.com/scholar?q=Igal+Sason+and+Sergio+Verd%C3%BA%2E+2016%2E+f+%2Ddivergence+inequalities%2E+IEEE+Transactions+on+Information+Theory%2C+62%2811%29%3A5973%E2%80%936006%2E)
41. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 7881– 7892. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Dipanjan+Das%2C+and+Ankur+Parikh%2E+2020%2E+BLEURT%3A+Learning+robust+metrics+for+text+generation%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages)
42. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1715–1725.
43. Chenze Shao, Xuanfu Wu, and Yang Feng. 2022. One reference is not enough: Diverse distillation with reference selection for non-autoregressive translation. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pa… [find](https://scholar.google.com/scholar?q=Chenze+Shao%2C+Xuanfu+Wu%2C+and+Yang+Feng%2E+2022%2E+One+reference+is+not+enough%3A+Diverse+distillation+with+reference+selection+for+non%2Dautoregressive+translation%2E+In+Proceedings+of+the+Conference+of+the+Nort)
44. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of the International Conference on Machine Learning, pages 4596–4604. [find](https://scholar.google.com/scholar?q=Noam+Shazeer+and+Mitchell+Stern%2E+2018%2E+Adafactor%3A+Adaptive+learning+rates+with+sublinear+memory+cost%2E+In+Proceedings+of+the+International+Conference+on+Machine+Learning%2C+pages+4596%E2%80%934604%2E)
45. Sam Shleifer and Alexander M Rush. 2020. Pre-trained summarization distillation. arXiv preprint arXiv:2010.13002. — [arXiv:2010.13002](https://arxiv.org/abs/2010.13002) [find](https://scholar.google.com/scholar?q=Sam+Shleifer+and+Alexander+M+Rush%2E+2020%2E+Pre%2Dtrained+summarization+distillation%2E+arXiv+preprint+arXiv%3A2010%2E13002%2E+%E2%80%94+%5BarXiv%3A2010%2E13002%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2010%2E13002%29)
46. Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. [A study of translation edit rate with targeted human annotation](https://md.research.elamur.ai/free/en/papers/snover-2006-study-translation-edit-rate-targeted/). In Proceedings of the Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223–231.
47. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. [Patient knowledge distillation for BERT model compression](https://md.research.elamur.ai/free/en/papers/sun-2019-patient-knowledge-distillation-bert-model/). In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing, pages 4323–4332.
48. Chuanxin Tang, Yucheng Zhao, Guangting Wang, Chong Luo, Wenxuan Xie, and Wenjun Zeng. 2022. Sparse MLP for image recognition: Is self-attention really necessary? In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2344–2351. [find](https://scholar.google.com/scholar?q=Chuanxin+Tang%2C+Yucheng+Zhao%2C+Guangting+Wang%2C+Chong+Luo%2C+Wenxuan+Xie%2C+and+Wenjun+Zeng%2E+2022%2E+Sparse+MLP+for+image+recognition%3A+Is+self%2Dattention+really+necessary%3F+In+Proceedings+of+the+AAAI+Conference+)
49. Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019. Distilling task-specific knowledge from BERT into simple neural networks. arXiv preprint arXiv:1903.12136. [find](https://scholar.google.com/scholar?q=Raphael+Tang%2C+Yao+Lu%2C+Linqing+Liu%2C+Lili+Mou%2C+Olga+Vechtomova%2C+and+Jimmy+Lin%2E+2019%2E+Distilling+task%2Dspecific+knowledge+from+BERT+into+simple+neural+networks%2E+arXiv+preprint+arXiv%3A1903%2E12136%2E)
50. Lifu Tu, Richard Yuanzhe Pang, Sam Wiseman, and Kevin Gimpel. 2020. ENGINE: Energy-based inference networks for non-autoregressive machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 2819–2826. [find](https://scholar.google.com/scholar?q=Lifu+Tu%2C+Richard+Yuanzhe+Pang%2C+Sam+Wiseman%2C+and+Kevin+Gimpel%2E+2020%2E+ENGINE%3A+Energy%2Dbased+inference+networks+for+non%2Dautoregressive+machine+translation%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Asso)
51. Bolin Wei, Shuai Lu, Lili Mou, Hao Zhou, Pascal Poupart, Ge Li, and Zhi Jin. 2019. Why do neural dialog systems generate short and meaningless replies? A comparison between dialog and translation. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing, pages 7290–7… [find](https://scholar.google.com/scholar?q=Bolin+Wei%2C+Shuai+Lu%2C+Lili+Mou%2C+Hao+Zhou%2C+Pascal+Poupart%2C+Ge+Li%2C+and+Zhi+Jin%2E+2019%2E+Why+do+neural+dialog+systems+generate+short+and+meaningless+replies%3F+A+comparison+between+dialog+and+translation%2E+In+)
52. Yuqiao Wen, Yongchang Hao, Yanshuai Cao, and Lili Mou. 2023. An equal-size hard EM algorithm for diverse dialogue generation. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yuqiao+Wen%2C+Yongchang+Hao%2C+Yanshuai+Cao%2C+and+Lili+Mou%2E+2023%2E+An+equal%2Dsize+hard+EM+algorithm+for+diverse+dialogue+generation%2E+In+International+Conference+on+Learning+Representations%2E)
53. Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. 2021. One teacher is enough? Pre-trained language model distillation from multiple teachers. In Findings of the Association for Computational Linguistics: ACL-IJCNLP, pages 4408–4413. [find](https://scholar.google.com/scholar?q=Chuhan+Wu%2C+Fangzhao+Wu%2C+and+Yongfeng+Huang%2E+2021%2E+One+teacher+is+enough%3F+Pre%2Dtrained+language+model+distillation+from+multiple+teachers%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+A)
54. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020b. [DIALOGPT : Large-scale generative pre-training for conversational response generation](https://md.research.elamur.ai/free/en/papers/zhang-2019-dialogpt-large-scale-generative-pre/). In Proceedings of the Annual Meeting of the Association for Computational Linguist…
55. Miaoyun Zhao, Yulai Cong, Shuyang Dai, and Lawrence Carin. 2020. Bridging maximum likelihood and adversarial learning via α-divergence. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6901–6908. [find](https://scholar.google.com/scholar?q=Miaoyun+Zhao%2C+Yulai+Cong%2C+Shuyang+Dai%2C+and+Lawrence+Carin%2E+2020%2E+Bridging+maximum+likelihood+and+adversarial+learning+via+%CE%B1%2Ddivergence%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence)
56. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. [MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance](https://md.research.elamur.ai/free/en/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/). In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Co…
57. Pei Zhou, Karthik Gopalakrishnan, Behnam Hedayatnia, Seokhwan Kim, Jay Pujara, Xiang Ren, Yang Liu, and Dilek Hakkani-Tur. 2021. Commonsense-focused dialogues for response generation: An empirical study. In Proceedings of the Annual Meeting of the Special Interest Group on Discourse and Dialogue, pa… [find](https://scholar.google.com/scholar?q=Pei+Zhou%2C+Karthik+Gopalakrishnan%2C+Behnam+Hedayatnia%2C+Seokhwan+Kim%2C+Jay+Pujara%2C+Xiang+Ren%2C+Yang+Liu%2C+and+Dilek+Hakkani%2DTur%2E+2021%2E+Commonsense%2Dfocused+dialogues+for+response+generation%3A+An+empirical+stu)
58. Ze Yang, Linjun Shou, Ming Gong, Wutao Lin, and Daxin Jiang. 2020. Model compression with two-stage multi-teacher knowledge distillation for web question answering system. In Proceedings of the International Conference on Web Search and Data Mining, page 690–698. [find](https://scholar.google.com/scholar?q=Ze+Yang%2C+Linjun+Shou%2C+Ming+Gong%2C+Wutao+Lin%2C+and+Daxin+Jiang%2E+2020%2E+Model+compression+with+two%2Dstage+multi%2Dteacher+knowledge+distillation+for+web+question+answering+system%2E+In+Proceedings+of+the+Intern)
59. Hongxu Yin, Pavlo Molchanov, Jose M. Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K. Jha, and Jan Kautz. 2020. Dreaming to distill: Data-free knowledge transfer via DeepInversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8715–8724. [find](https://scholar.google.com/scholar?q=Hongxu+Yin%2C+Pavlo+Molchanov%2C+Jose+M%2E+Alvarez%2C+Zhizhong+Li%2C+Arun+Mallya%2C+Derek+Hoiem%2C+Niraj+K%2E+Jha%2C+and+Jan+Kautz%2E+2020%2E+Dreaming+to+distill%3A+Data%2Dfree+knowledge+transfer+via+DeepInversion%2E+In+Proceedi)
60. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020a. [PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/). In Proceedings of the International Conference on Machine Learning, pages 11328–11339.
61. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2019%2E+BERTScore%3A+Evaluating+text+generation+with+BERT%2E+In+International+Conference+on+Learning+Representations%2E)
62. Yivan Zhang, Gang Niu, and Masashi Sugiyama. 2021. Learning noise transition matrix from only noisy labels via total variation regularization. In Proceedings of the International Conference on Machine Learning, pages 12501–12512. [find](https://scholar.google.com/scholar?q=Yivan+Zhang%2C+Gang+Niu%2C+and+Masashi+Sugiyama%2E+2021%2E+Learning+noise+transition+matrix+from+only+noisy+labels+via+total+variation+regularization%2E+In+Proceedings+of+the+International+Conference+on+Machine)

*Updated 24 October 2026*