---
title: "Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs — Boizard 2024"
description: "Drawing on optimal transport, the authors introduce a new objective they name the Universal Logit Distillation… 66 references, cited by 34 papers here."
authors: ["Nicolas Boizard", "Kevin El Haddad", "Celine Hudelot", "Pierre Colombo"]
year: 2024
venue: "arXiv:2402.12030 · Transactions on Machine Learning Research, January 2025"
source: "https://arxiv.org/abs/2402.12030"
doi: "10.48550/arxiv.2402.12030"
arxiv: "2402.12030"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/boizard-2024-towards-cross-tokenizer-distillation-universal/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/boizard-2024-towards-cross-tokenizer-distillation-universal/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs

# Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs

**Authors:** Nicolas Boizard, Kevin El Haddad, Celine Hudelot and [Pierre Colombo](https://md.research.elamur.ai/free/en/authors/pierre-colombo/) · **Year:** 2024 · **Venue:** arXiv:2402.12030 · Transactions on Machine Learning Research, January 2025 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.12030) · [PDF](https://arxiv.org/pdf/2402.12030) · [DOI 10.48550/arxiv.2402.12030](https://doi.org/10.48550/arxiv.2402.12030) · [arXiv 2402.12030](https://arxiv.org/abs/2402.12030) · [OpenAlex](https://openalex.org/W4392019320)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 34 in our library, of the 140,884 papers whose bibliographies we read

## Summary

Logit-based knowledge distillation usually relies on KL divergence. This requires the teacher and the student to share one vocabulary, but LLM families rarely share tokenizers. As a result, practitioners mostly fall back on fine-tuning the student on text the teacher has generated. Drawing on optimal transport, the authors introduce a new objective they name the Universal Logit Distillation (ULD) loss. It uses a closed-form Wasserstein distance to compare the output distributions of teacher and student even when their vocabularies differ. The method treats both models as black boxes, needing only their logits, so it also works across architectures. They test it with a LLama teacher and smaller students such as Pythia-410m, Bloomz-560m and the encoder-decoder MT0-580m. Tasks cover extractive QA, generative QA and summarization, on SQuAD, QED, PubMedQA and DIALOGSum. ULD consistently beats distillation from teacher-generated text. It reaches comparable quality with roughly half the data or half the student size, and it curbs overfitting. Code, weights and generated datasets are released, making distillation feasible between arbitrary model families.

## Key points

- Tokenizer mismatch is substantial: for example, the Bloomz tokenizer covers only 30.03% of Mistral's vocabulary, so KL-based logit distillation cannot be applied directly.
- In the dataset-size ablation (LLama → Pythia-410m, five runs per ratio), ULD wins at every ratio tested (25%, 50%, 75% and 100% of the data). With only 50% of the data, it outperforms teacher-text training on the full set.
- Pythia-410m trained with ULD matches Pythia-1b trained on teacher-generated text on QED and DIALOGSum.
- Cross-architecture transfer (LLama decoder → MT0-580m encoder-decoder), teacher-text vs ULD:
  - QED F1: 55.63 → 56.01
  - PubMedQA BERTScore: 27.56 → 30.19
  - DIALOGSum Rouge-Lsum: 23.22 → 23.92
- Resources: code is at github.com/Nicolas-BZRD/llm-recipes, and models and datasets are on Hugging Face (Nicolas-BZRD).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/)

## Institutions

- [Université Paris-Saclay](https://md.research.elamur.ai/free/en/universities/universite-paris-saclay/) — University, Gif-sur-Yvette, France · 26 works · 5,075 citations
- [CentraleSupélec](https://md.research.elamur.ai/free/en/labs/centralesupelec/) — Research institute, Gif-sur-Yvette, France · 7 works · 1,242 citations
- [Mathématiques et Informatique pour la Complexité et les Systèmes](https://md.research.elamur.ai/free/en/labs/mathematiques-et-informatique-pour-la-complexite-et-les-systemes/) — Research institute, Gif-sur-Yvette · 2 works · 260 citations

Also on the byline: [University of Mons](https://ror.org/02qnnz951).

## Cite

Cite as: Nicolas Boizard, Kevin El Haddad, Celine Hudelot and Pierre Colombo. 2024. Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs. arXiv:2402.12030 · Transactions on Machine Learning Research, January 2025.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{boizard2024cross,
  title = {{Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs}},
  author = {Nicolas Boizard, Kevin {El Haddad}, Celine Hudelot and Pierre Colombo},
  year = {2024},
  doi = {10.48550/arxiv.2402.12030},
  eprint = {2402.12030},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.12030}
}
```

- [← previous Maverick: Efficient and Accurate Coreference Resolution Defying Recent Trends](https://md.research.elamur.ai/free/en/papers/martinelli-2024-maverick-efficient-accurate-coreference-resolution/)
- [next → Is Sarcasm Detection A Step-by-Step Reasoning Process in Large Language Models?](https://md.research.elamur.ai/free/en/papers/yao-2024-sarcasm-detection-step-step-reasoning/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

66 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models.… [our card](https://md.research.elamur.ai/free/en/papers/almazrouei-2023-falcon-series-open-language-models/)
2. Hareesh Bahuleyan, Lili Mou, Hao Zhou, and Olga Vechtomova. Stochastic wasserstein autoencoder for probabilistic sentence generation. 2018. [find](https://scholar.google.com/scholar?q=Hareesh+Bahuleyan%2C+Lili+Mou%2C+Hao+Zhou%2C+and+Olga+Vechtomova%2E+Stochastic+wasserstein+autoencoder+for+probabilistic+sentence+generation%2E+2018%2E)
3. Rishabh Bhardwaj, Tushar Vaidya, and Soujanya Poria. Knot: Knowledge distillation using optimal transport for solving nlp tasks, 2022. [find](https://scholar.google.com/scholar?q=Rishabh+Bhardwaj%2C+Tushar+Vaidya%2C+and+Soujanya+Poria%2E+Knot%3A+Knowledge+distillation+using+optimal+transport+for+solving+nlp+tasks%2C+2022%2E)
4. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+Aviya+Skowron%2C+Lintang+Sutaw)
5. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. Gpt-neox-20b: An open-source autoregressive… [our card](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/)
6. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information, 2017. URL https://arxiv.org/abs/1607.04606. [↗](https://arxiv.org/abs/1607.04606) [our card](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/)
7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. volume 33, pp. 1877–1901, 2020. [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
8. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023. [our card](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/)
9. Cristian Buciluundefined, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’06, pp. 535–541, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933395. doi: 1… — [doi:10.1145/1150402.1150464](https://doi.org/10.1145/1150402.1150464) [find](https://scholar.google.com/scholar?q=Cristian+Buciluundefined%2C+Rich+Caruana%2C+and+Alexandru+Niculescu%2DMizil%2E+Model+compression%2E+In+Proceedings+of+the+12th+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining%2C+KDD+%E2%80%9906)
10. Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. Dialogsum: A real-life scenario dialogue summarization dataset, 2021. [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-dialogsum-real-life-scenario-dialogue/)
11. Pierre Colombo, Guillaume Staerman, Chloé Clavel, and Pablo Piantanida. Automatic text evaluation through the lens of Wasserstein barycenters. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Langu… [find](https://scholar.google.com/scholar?q=Pierre+Colombo%2C+Guillaume+Staerman%2C+Chlo%C3%A9+Clavel%2C+and+Pablo+Piantanida%2E+Automatic+text+evaluation+through+the+lens+of+Wasserstein+barycenters%2E+In+Marie%2DFrancine+Moens%2C+Xuanjing+Huang%2C+Lucia+Specia%2C+an)
12. Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. volume 26, 2013. [find](https://scholar.google.com/scholar?q=Marco+Cuturi%2E+Sinkhorn+distances%3A+Lightspeed+computation+of+optimal+transport%2E+volume+26%2C+2013%2E)
13. Sayantan Dasgupta, Trevor Cohn, and Timothy Baldwin. Cost-effective distillation of large language models. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 7346–7354, 2023. [find](https://scholar.google.com/scholar?q=Sayantan+Dasgupta%2C+Trevor+Cohn%2C+and+Timothy+Baldwin%2E+Cost%2Deffective+distillation+of+large+language+models%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2023%2C+pp%2E+7346%E2%80%937354%2C+2023%2E)
14. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Artidoro+Pagnoni%2C+Ari+Holtzman%2C+and+Luke+Zettlemoyer%2E+Qlora%3A+Efficient+finetuning+of+quantized+llms%2C+2023%2E)
15. Zilin Du, Haoxin Li, Xu Guo, and Boyang Li. Training on synthetic data beats real data in multimodal relation extraction, 2023. [find](https://scholar.google.com/scholar?q=Zilin+Du%2C+Haoxin+Li%2C+Xu+Guo%2C+and+Boyang+Li%2E+Training+on+synthetic+data+beats+real+data+in+multimodal+relation+extraction%2C+2023%2E)
16. Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, Ievgen Redko, Antoine Rolet, Antony Schutz, Vivien Segu… [find](https://scholar.google.com/scholar?q=R%C3%A9mi+Flamary%2C+Nicolas+Courty%2C+Alexandre+Gramfort%2C+Mokhtar+Z%2E+Alaya%2C+Aur%C3%A9lie+Boisbunon%2C+Stanislas+Chambon%2C+Laetitia+Chapel%2C+Adrien+Corenflos%2C+Kilian+Fatras%2C+Nemo+Fournier%2C+L%C3%A9o+Gautheron%2C+Nathalie+T%2EH%2E+)
17. Nuno M Guerreiro, Pierre Colombo, Pablo Piantanida, and André FT Martins. Optimal transport for unsupervised hallucination detection in neural machine translation. 2022. [find](https://scholar.google.com/scholar?q=Nuno+M+Guerreiro%2C+Pierre+Colombo%2C+Pablo+Piantanida%2C+and+Andr%C3%A9+FT+Martins%2E+Optimal+transport+for+unsupervised+hallucination+detection+in+neural+machine+translation%2E+2022%2E)
18. Nan He, Hanyu Lai, Chenyang Zhao, Zirui Cheng, Junting Pan, Ruoyu Qin, Ruofan Lu, Rui Lu, Yunchen Zhang, Gangming Zhao, Zhaohui Hou, Zhiyuan Huang, Shaoqing Lu, Ding Liang, and Mingjie Zhan. Teacherlm: Teaching to fish rather than giving the fish, language modeling likewise. Number arXiv:2310.19019.… — [doi:10.48550/arxiv.2310.19019](https://doi.org/10.48550/arxiv.2310.19019) [find](https://scholar.google.com/scholar?q=Nan+He%2C+Hanyu+Lai%2C+Chenyang+Zhao%2C+Zirui+Cheng%2C+Junting+Pan%2C+Ruoyu+Qin%2C+Ruofan+Lu%2C+Rui+Lu%2C+Yunchen+Zhang%2C+Gangming+Zhao%2C+Zhaohui+Hou%2C+Zhiyuan+Huang%2C+Shaoqing+Lu%2C+Ding+Liang%2C+and+Mingjie+Zhan%2E+Teacherlm)
19. Xuanli He, Islam Nassar, Jamie Kiros, Gholamreza Haffari, and Mohammad Norouzi. Generate, Annotate, and Learn: NLP with Synthetic Text. volume 10, pp. 826–842, 08 2022. doi: 10.1162/tacl_a_00492. URL https://doi.org/10.1162/tacl_a_00492. [↗](https://doi.org/10.1162/tacl_a_00492)
20. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. Number arXiv:1503.02531arXiv:1503.02531. arXiv, March 2015. doi: 10.48550/arXiv.1503.02531. URL http://arxiv.org/abs/1503.02531. arXiv:1503.02531 [cs, stat]. [↗](http://arxiv.org/abs/1503.02531) [our card](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/)
21. Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes, 2023. [our card](https://md.research.elamur.ai/free/en/papers/hsieh-2023-distilling-step-step-outperforming-larger/)
22. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
23. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
24. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding, 2020. [find](https://scholar.google.com/scholar?q=Xiaoqi+Jiao%2C+Yichun+Yin%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Xiao+Chen%2C+Linlin+Li%2C+Fang+Wang%2C+and+Qun+Liu%2E+Tinybert%3A+Distilling+bert+for+natural+language+understanding%2C+2020%2E)
25. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering, 2019. [our card](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/)
26. Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, B… [find](https://scholar.google.com/scholar?q=Dhiraj+Kalamkar%2C+Dheevatsa+Mudigere%2C+Naveen+Mellempudi%2C+Dipankar+Das%2C+Kunal+Banerjee%2C+Sasikanth+Avancha%2C+Dharma+Teja+Vooturi%2C+Nataraj+Jammalamadaka%2C+Jianyu+Huang%2C+Hector+Yuen%2C+Jiyan+Yang%2C+Jongsoo+Park)
27. Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1317–1327, Austin, Texas, November 2016. Association for Computational Linguistics. do… [our card](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/)
28. Alina Kramchaninova and Arne Defauw. Synthetic data generation for multilingual domain-adaptable question answering systems. In Helena Moniz, Lieve Macken, Andrew Rufener, Loïc Barrault, Marta R. Costa-jussà, Christophe Declercq, Maarit Koponen, Ellie Kemp, Spyridon Pilos, Mikel L. Forcada, Carolina… [find](https://scholar.google.com/scholar?q=Alina+Kramchaninova+and+Arne+Defauw%2E+Synthetic+data+generation+for+multilingual+domain%2Dadaptable+question+answering+systems%2E+In+Helena+Moniz%2C+Lieve+Macken%2C+Andrew+Rufener%2C+Lo%C3%AFc+Barrault%2C+Marta+R%2E+Cost)
29. Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares, and Michael Collins. Qed: A framework and dataset for explanations in question answering, 2020. [find](https://scholar.google.com/scholar?q=Matthew+Lamm%2C+Jennimaria+Palomaki%2C+Chris+Alberti%2C+Daniel+Andor%2C+Eunsol+Choi%2C+Livio+Baldini+Soares%2C+and+Michael+Collins%2E+Qed%3A+A+framework+and+dataset+for+explanations+in+question+answering%2C+2020%2E)
30. VI Lcvenshtcin. Binary coors capable or ‘correcting deletions, insertions, and reversals. In Soviet Physics-Doklady, volume 10, 1966. [find](https://scholar.google.com/scholar?q=VI+Lcvenshtcin%2E+Binary+coors+capable+or+%E2%80%98correcting+deletions%2C+insertions%2C+and+reversals%2E+In+Soviet+Physics%2DDoklady%2C+volume+10%2C+1966%2E)
31. Brian Lester, Joshua Yurtsever, Siamak Shakeri, and Noah Constant. Reducing retraining by recycling parameter-efficient prompts, 2022. [find](https://scholar.google.com/scholar?q=Brian+Lester%2C+Joshua+Yurtsever%2C+Siamak+Shakeri%2C+and+Noah+Constant%2E+Reducing+retraining+by+recycling+parameter%2Defficient+prompts%2C+2022%2E)
32. Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274–19286. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Yaniv+Leviathan%2C+Matan+Kalman%2C+and+Yossi+Matias%2E+Fast+inference+from+transformers+via+speculative+decoding%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+19274%E2%80%9319286%2E+PMLR%2C+2023%2E)
33. Tianda Li, Yassir El Mesbahi, Ivan Kobyzev, Ahmad Rashid, Atif Mahmud, Nithin Anchuri, Habib Hajimo-lahoseini, Yang Liu, and Mehdi Rezagholizadeh. dis, 2021. [find](https://scholar.google.com/scholar?q=Tianda+Li%2C+Yassir+El+Mesbahi%2C+Ivan+Kobyzev%2C+Ahmad+Rashid%2C+Atif+Mahmud%2C+Nithin+Anchuri%2C+Habib+Hajimo%2Dlahoseini%2C+Yang+Liu%2C+and+Mehdi+Rezagholizadeh%2E+dis%2C+2021%2E)
34. Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013. [↗](https://aclanthology.org/W04-1013) [our card](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/)
35. Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models, 2023. [our card](https://md.research.elamur.ai/free/en/papers/ma-2023-llm-pruner-structural-pruning-large/)
36. Alireza Mohammadshahi, Vassilina Nikoulina, Alexandre Berard, Caroline Brun, James Henderson, and Laurent Besacier. Small-100: Introducing shallow multilingual machine translation model for low-resource languages. Number arXiv:2210.11621arXiv:2210.11621. arXiv, October 2022. doi: 10.48550/arXiv.2210… [find](https://scholar.google.com/scholar?q=Alireza+Mohammadshahi%2C+Vassilina+Nikoulina%2C+Alexandre+Berard%2C+Caroline+Brun%2C+James+Henderson%2C+and+Laurent+Besacier%2E+Small%2D100%3A+Introducing+shallow+multilingual+machine+translation+model+for+low%2Dresour)
37. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel… [our card](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/)
38. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training la… [our card](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
39. Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. volume 11, pp. 355–607. Now Publishers, Inc., 2019. [find](https://scholar.google.com/scholar?q=Gabriel+Peyr%C3%A9%2C+Marco+Cuturi%2C+et+al%2E+Computational+optimal+transport%3A+With+applications+to+data+science%2E+volume+11%2C+pp%2E+355%E2%80%93607%2E+Now+Publishers%2C+Inc%2E%2C+2019%2E)
40. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text, 2016. [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
41. Mrigank Raman, Pranav Mani, Davis Liang, and Zachary Lipton. For distillation, tokens are not all you need. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023. URL [https://openreview.net/forum?id=2fc5GOPYip](https://openreview.net/forum?id=2fc5GOPYip). [↗](https://openreview.net/forum?id=2fc5GOPYip)
42. Vipula Rawte, Amit Sheth, and Amitava Das. A survey of hallucination in large foundation models. 2023. [our card](https://md.research.elamur.ai/free/en/papers/rawte-2023-survey-hallucination-large-foundation-models/)
43. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+Distilbert%2C+a+distilled+version+of+bert%3A+smaller%2C+faster%2C+cheaper+and+lighter%2C+2020%2E)
44. Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Da… [our card](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
45. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation, 2021. [our card](https://md.research.elamur.ai/free/en/papers/shuster-2021-retrieval-augmentation-reduces-hallucination-conversation/)
46. Marina Sokolova, Nathalie Japkowicz, and Stan Szpakowicz. Beyond accuracy, f-score and roc: A family of discriminant measures for performance evaluation. volume Vol. 4304, pp. 1015–1021, 01 2006. ISBN 978-3-540-49787-5. doi: 10.1007/11941439_114. — [doi:10.1007/11941439_114](https://doi.org/10.1007/11941439_114) [find](https://scholar.google.com/scholar?q=Marina+Sokolova%2C+Nathalie+Japkowicz%2C+and+Stan+Szpakowicz%2E+Beyond+accuracy%2C+f%2Dscore+and+roc%3A+A+family+of+discriminant+measures+for+performance+evaluation%2E+volume+Vol%2E+4304%2C+pp%2E+1015%E2%80%931021%2C+01+2006%2E+ISBN)
47. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: a compact task-agnostic bert for resource-limited devices, 2020. [our card](https://md.research.elamur.ai/free/en/papers/sun-2020-mobilebert-compact-task-agnostic-bert/)
48. Inar Timiryasov and Jean-Loup Tastet. Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty, 2023. [find](https://scholar.google.com/scholar?q=Inar+Timiryasov+and+Jean%2DLoup+Tastet%2E+Baby+llama%3A+knowledge+distillation+from+an+ensemble+of+teachers+trained+on+a+small+dataset+with+no+performance+penalty%2C+2023%2E)
49. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. Number arXiv:2… [our card](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)
50. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+Dan+Bikel%2C+Lukas+Blecher%2C+Cristian+Canton+Fe)
51. Asahi Ushio, Fernando Alva-Manchego, and Jose Camacho-Collados. An empirical comparison of lm-based question and answer generation methods. 2023. [find](https://scholar.google.com/scholar?q=Asahi+Ushio%2C+Fernando+Alva%2DManchego%2C+and+Jose+Camacho%2DCollados%2E+An+empirical+comparison+of+lm%2Dbased+question+and+answer+generation+methods%2E+2023%2E)
52. Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009. [find](https://scholar.google.com/scholar?q=C%C3%A9dric+Villani+et+al%2E+Optimal+transport%3A+old+and+new%2C+volume+338%2E+Springer%2C+2009%2E)
53. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers, 2020a. [find](https://scholar.google.com/scholar?q=Wenhui+Wang%2C+Furu+Wei%2C+Li+Dong%2C+Hangbo+Bao%2C+Nan+Yang%2C+and+Ming+Zhou%2E+Minilm%3A+Deep+self%2Dattention+distillation+for+task%2Dagnostic+compression+of+pre%2Dtrained+transformers%2C+2020a%2E)
54. Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. Minilmv2: Multi-head self-attention relation distillation for compressing pretrained transformers, 2021. [find](https://scholar.google.com/scholar?q=Wenhui+Wang%2C+Hangbo+Bao%2C+Shaohan+Huang%2C+Li+Dong%2C+and+Furu+Wei%2E+Minilmv2%3A+Multi%2Dhead+self%2Dattention+relation+distillation+for+compressing+pretrained+transformers%2C+2021%2E)
55. Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a few examples: A survey on few-shot learning. volume 53, pp. 1–34. ACM New York, NY, USA, 2020b. [find](https://scholar.google.com/scholar?q=Yaqing+Wang%2C+Quanming+Yao%2C+James+T+Kwok%2C+and+Lionel+M+Ni%2E+Generalizing+from+a+few+examples%3A+A+survey+on+few%2Dshot+learning%2E+volume+53%2C+pp%2E+1%E2%80%9334%2E+ACM+New+York%2C+NY%2C+USA%2C+2020b%2E)
56. Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, and Alham Fikri Aji. Lamini-lm: A diverse herd of distilled models from large-scale instructions, 2023. [find](https://scholar.google.com/scholar?q=Minghao+Wu%2C+Abdul+Waheed%2C+Chiyu+Zhang%2C+Muhammad+Abdul%2DMageed%2C+and+Alham+Fikri+Aji%2E+Lamini%2Dlm%3A+A+diverse+herd+of+distilled+models+from+large%2Dscale+instructions%2C+2023%2E)
57. Hongteng Xu, Wenlin Wang, Wei Liu, and Lawrence Carin. Distilled wasserstein learning for word embedding and topic modeling, 2018. [find](https://scholar.google.com/scholar?q=Hongteng+Xu%2C+Wenlin+Wang%2C+Wei+Liu%2C+and+Lawrence+Carin%2E+Distilled+wasserstein+learning+for+word+embedding+and+topic+modeling%2C+2018%2E)
58. Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Jia-Jun Li, Nora Bradford, Branda Sun, Tran Bao Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang, Nanyun Peng, Zhou Yu, and Mark Warschauer. Fantastic questions and where to find them: Fairytaleqa – an au… [our card](https://md.research.elamur.ai/free/en/papers/xu-2022-fantastic-questions-where-find-them/)
59. Jianbo Ye, Panruo Wu, James Z. Wang, and Jia Li. Fast discrete distribution clustering using wasserstein barycenter with sparse support, 2017. [find](https://scholar.google.com/scholar?q=Jianbo+Ye%2C+Panruo+Wu%2C+James+Z%2E+Wang%2C+and+Jia+Li%2E+Fast+discrete+distribution+clustering+using+wasserstein+barycenter+with+sparse+support%2C+2017%2E)
60. Qinyuan Ye, Iz Beltagy, Matthew Peters, Xiang Ren, and Hannaneh Hajishirzi. FiD-ICL: A fusion-in-decoder approach for efficient in-context learning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Qinyuan+Ye%2C+Iz+Beltagy%2C+Matthew+Peters%2C+Xiang+Ren%2C+and+Hannaneh+Hajishirzi%2E+FiD%2DICL%3A+A+fusion%2Din%2Ddecoder+approach+for+efficient+in%2Dcontext+learning%2E+In+Anna+Rogers%2C+Jordan+Boyd%2DGraber%2C+and+Naoaki+Okaz)
61. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained tr… [our card](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/)
62. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL [https://openreview.net/forum?id=SkeHuCVFDr](https://openreview.net/forum?id=SkeHuCVFDr). [↗](https://openreview.net/forum?id=SkeHuCVFDr)
63. Jiachen Zhao, Wenlong Zhao, Andrew Drozdov, Benjamin Rozonoyer, Md Arafat Sultan, Jay-Yoon Lee, Mohit Iyyer, and Andrew McCallum. Multistage collaborative knowledge distillation from large language models. Number arXiv:2311.08640. arXiv, November 2023a. URL http://arxiv.org/abs/2311.08640. arXiv:231… — [arXiv:2311.08640](https://arxiv.org/abs/2311.08640) [↗](http://arxiv.org/abs/2311.08640)
64. Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Experiences on scaling fully sharded dat… [find](https://scholar.google.com/scholar?q=Yanli+Zhao%2C+Andrew+Gu%2C+Rohan+Varma%2C+Liang+Luo%2C+Chien%2DChin+Huang%2C+Min+Xu%2C+Less+Wright%2C+Hamid+Shojanazeri%2C+Myle+Ott%2C+Sam+Shleifer%2C+Alban+Desmaison%2C+Can+Balioglu%2C+Pritam+Damania%2C+Bernard+Nguyen%2C+Geeta+Ch)
65. Tianxun Zhou and Keng-Hwee Chiam. Synthetic data generation method for data-free knowledge distillation in regression neural networks. volume 227, pp. 120327. Elsevier, 2023. [find](https://scholar.google.com/scholar?q=Tianxun+Zhou+and+Keng%2DHwee+Chiam%2E+Synthetic+data+generation+method+for+data%2Dfree+knowledge+distillation+in+regression+neural+networks%2E+volume+227%2C+pp%2E+120327%2E+Elsevier%2C+2023%2E)
66. Yubo Zhuang, Xiaohui Chen, and Yun Yang. Wasserstein k-means for clustering probability distributions, 2022. [find](https://scholar.google.com/scholar?q=Yubo+Zhuang%2C+Xiaohui+Chen%2C+and+Yun+Yang%2E+Wasserstein+k%2Dmeans+for+clustering+probability+distributions%2C+2022%2E)