--- title: "Cross-Tokenizer LLM Distillation through a Byte-Level Interface" description: "The authors propose Byte-Level Distillation (BLD), a simple baseline that needs no alignment because… 1 reference, cited by 5 papers here." authors: ["Avyav Kumar Singh", "Yen-Chen Wu", "Alexandru Cioba", "Alberto Bernacchia", "Davide Buffelli"] year: 2026 venue: "Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U)" source: "https://aclanthology.org/2026.customnlp4u-1.9/" doi: "10.18653/v1/2026.customnlp4u-1.9" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/singh-2026-cross-tokenizer-llm-distillation-through/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/singh-2026-cross-tokenizer-llm-distillation-through/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/) / Cross-Tokenizer LLM Distillation through a Byte-Level Interface # Cross-Tokenizer LLM Distillation through a Byte-Level Interface **Authors:** Avyav Kumar Singh, Yen-Chen Wu, Alexandru Cioba, Alberto Bernacchia and Davide Buffelli · **Year:** 2026 · **Venue:** [Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U)](https://md.research.elamur.ai/free/en/venues/customnlp4u/) ([Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual 2026](https://md.research.elamur.ai/free/en/venues/customnlp4u/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.customnlp4u-1.9/) · [PDF](https://aclanthology.org/2026.customnlp4u-1.9.pdf) · [DOI 10.18653/v1/2026.customnlp4u-1.9](https://doi.org/10.18653/v1/2026.customnlp4u-1.9) · [OpenAlex](https://openalex.org/W7166831533) **Citations:** 0 citations (OpenAlex, 2026-10-14) · 5 in our library, of the 149,439 papers whose bibliographies we read ## Summary Standard knowledge distillation assumes that the teacher and student share one vocabulary, because the student learns by matching the teacher's next-token distribution. When the tokenizers differ, existing cross-tokenizer distillation (CTD) methods have to align the two vocabularies through heuristics, which adds complexity. The authors propose Byte-Level Distillation (BLD), a simple baseline that needs no alignment because bytes are a representation every tokenizer shares. The teacher's token probabilities are converted into byte-level probabilities using a fast approximation from Vieira et al. (2025). The student gets a small learnable byte-level head, placed alongside its usual token head, and distillation happens through this byte interface. Once training ends, the extra head is discarded. The method was tested in three settings, with models from 1B to 8B parameters: - moving Llama3.2 3B to the Qwen2 tokenizer; - converting Llama3.2 3B to byte-level tokenization; - distilling OpenMath2-Llama3.1-8B into Gemma2 2B. BLD matches, and on some benchmarks beats, more elaborate methods such as MinED, ALM + SFT and DSKD. Its instruction following is clearly weaker, however. No method wins everywhere, so CTD remains an open problem. ## Key points - **Architecture and training:** The byte head consists of 10 parallel linear projections onto a 260-symbol vocabulary (256 bytes plus 4 special tokens). This lets each token position predict up to 10 bytes. The backbone is tuned with LoRA (rank 64, query/value projections only), and new embeddings are initialised with Fast Vocabulary Transfer (FVT). - **Llama3.2 3B → Qwen2 tokenizer:** BLD scores best on PiQA (75.68) and AGI-ZH (35.97). Its IFEval score is only 30.58, versus 62.83 for MinED and 58.51 for ALM + SFT. - **BPE-to-byte transfer:** This setting is much harder. All methods drop sharply, with MMLU about 21 points and ARC-C about 13 points below the original model. BLD leads only on PiQA (67.52), and only narrowly. - **OpenMath2-Llama3.1-8B → Gemma2 2B:** BLD reaches 62.55 on GSM8K, ahead of ALM + SFT (61.56), SFT (59.29) and the base model (51.48). On MATH, SFT is better (22.40 vs. 20.08). - **Gap to the teacher:** The teacher scores 87.26 on GSM8K and 37.60 on MATH, so every student still falls far short. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions Also on the byline: [Cambridge College](https://ror.org/036qj5r13). 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works 🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) · 124 works ## Cite Cite as: Avyav Kumar Singh, Yen-Chen Wu, Alexandru Cioba, Alberto Bernacchia and Davide Buffelli. 2026. Cross-Tokenizer LLM Distillation through a Byte-Level Interface. Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{singh2026cross, title = {{Cross-Tokenizer LLM Distillation through a Byte-Level Interface}}, author = {Avyav Kumar Singh, Yen-Chen Wu, Alexandru Cioba, Alberto Bernacchia and Davide Buffelli}, booktitle = {Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U)}, year = {2026}, doi = {10.18653/v1/2026.customnlp4u-1.9}, url = {https://doi.org/10.18653/v1/2026.customnlp4u-1.9} } ``` - [← previous Contrastive Learning with Narrative Twins for Modeling Story Salience](https://md.research.elamur.ai/free/en/papers/sterner-2026-contrastive-learning-narrative-twins-modeling/) - [next → CURE-Med: Curriculum-Informed Reinforcement Learning for Multilingual Medical Reasoning](https://md.research.elamur.ai/free/en/papers/onyame-2026-cure-med-curriculum-informed-reinforcement/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. Nicolas Boizard, Kevin El Haddad, CELINE HUDE-LOT, and Pierre Colombo. 2025. [Towards cross-tokenizer distillation: the universal logit distillation loss for LLMs](https://md.research.elamur.ai/free/en/papers/boizard-2024-towards-cross-tokenizer-distillation-universal/). Transactions on Machine Learning Research. *Updated 30 October 2026*