---
title: "CTPD: Cross Tokenizer Preference Distillation (2026)"
description: "They justify the design through importance sampling. 25 references, cited by 8 papers here."
authors: ["Truong Nguyen", "Phi Van Dat", "Ngan Nguyen", "Linh Ngo Van", "Trung Le", "Thanh Hong Nguyen"]
year: 2026
venue: "arXiv:2601.11865"
source: "https://arxiv.org/abs/2601.11865"
doi: "10.48550/arxiv.2601.11865"
arxiv: "2601.11865"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nguyen-2026-ctpd-cross-tokenizer-preference-distillation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nguyen-2026-ctpd-cross-tokenizer-preference-distillation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/) / CTPD: Cross Tokenizer Preference Distillation

# CTPD: Cross Tokenizer Preference Distillation

**Authors:** [Truong Nguyen](https://md.research.elamur.ai/free/en/authors/truong-nguyen/), Phi Van Dat, [Ngan Nguyen](https://md.research.elamur.ai/free/en/authors/ngan-nguyen/), [Linh Ngo Van](https://md.research.elamur.ai/free/en/authors/linh-ngo-van/), [Trung Le](https://md.research.elamur.ai/free/en/authors/trung-le/) and Thanh Hong Nguyen · **Year:** 2026 · **Venue:** arXiv:2601.11865 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2601.11865) · [PDF](https://arxiv.org/pdf/2601.11865) · [DOI 10.48550/arxiv.2601.11865](https://doi.org/10.48550/arxiv.2601.11865) · [arXiv 2601.11865](https://arxiv.org/abs/2601.11865) · [OpenAlex](https://openalex.org/W7125106540)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/nguyen-2026-ctpd-cross-tokenizer-preference-distillation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Small language models are cheaper to deploy but harder to align with human preferences. One remedy is to distill alignment from a larger, already-aligned teacher, so the expensive alignment step is paid only once. The richest form of this is white-box distillation, which uses the teacher's logits rather than its generated text. It breaks down, however, when teacher and student split text into different tokens. The only earlier white-box preference-distillation study assumed a shared vocabulary.

CTPD addresses the mismatched-tokenizer case. Its key observation is that both tokenizations describe the same characters. It therefore segments each sequence into spans where teacher and student token groups cover identical character intervals. Teacher log-probabilities are then mapped onto the student's spans without adding trainable parameters. On this basis, the authors move TIS-DPO's token importance weights to the span level and use the teacher itself as the DPO reference policy. They justify the design through importance sampling.

On six benchmarks, CTPD outperforms both preference-alignment and classic distillation baselines. This makes any strong teacher usable for aligning compact models, whatever its tokenizer.

## Key points
- **Three components:** Aligned Span Projection, a cross-tokenizer version of TIS-DPO with span-level importance weights, and a Teacher-Anchored Reference that uses the teacher as π_ref.
- **Evaluation:** the Language Model Evaluation Harness on ARC, HellaSwag, Winogrande, MMLU, TruthfulQA and GSM8k.
- **Main results:** average scores beat TIS-DPO by +1.26 and +0.66 points in the two reported settings. The largest gains are on GSM8k (+3.16) and TruthfulQA (+2.85).
- **Ablations (one table labelled Llama3.1-8B):**
  - Random importance weights sharply hurt performance.
  - Length-averaged, student-estimated and teacher–student-estimated weights all fall short of the proposed scheme.
  - Using the student instead of the teacher as reference also lowers scores.
- **Training setup:**
  - One epoch per stage with AdamW, batch size 16, on 8× NVIDIA H100-80GB.
  - CTPD uses lr 1e-6 and β = 0.1, with weights clipped to [−0.5, 1.5].
  - Code is available at github.com/dinhtruongng/CTPD.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 242 works · [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) · 119 works

## Cite

Cite as: Truong Nguyen, Phi Van Dat, Ngan Nguyen, Linh Ngo Van, Trung Le and Thanh Hong Nguyen. 2026. CTPD: Cross Tokenizer Preference Distillation. arXiv:2601.11865.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{nguyen2026ctpd,
  title = {{CTPD: Cross Tokenizer Preference Distillation}},
  author = {Truong Nguyen, Phi {Van Dat}, Ngan Nguyen, Linh Ngo Van, Trung Le and Thanh Hong Nguyen},
  year = {2026},
  doi = {10.48550/arxiv.2601.11865},
  eprint = {2601.11865},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2601.11865}
}
```

- [← previous ATLAS: Orchestrating Heterogeneous Models and Tools for Multi-Domain Complex Reasoning](https://md.research.elamur.ai/free/en/papers/wu-2026-atlas-orchestrating-heterogeneous-models-tools/)
- [next → DeepSeek-OCR 2: Visual Causal Flow](https://md.research.elamur.ai/free/en/papers/wei-2026-deepseek-ocr-2-visual/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/)

## References

25 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bai, Y.; Jones, A.; Ndousse, K.; Askell, A.; Chen, A.; Das-Sarma, N.; Drain, D.; Fort, S.; Ganguli, D.; Henighan, T.; Joseph, N.; Kadavath, S.; Kernion, J.; Conerly, T.; El-Showk, S.; Elhage, N.; Hatfield-Dodds, Z.; Hernandez, D.; Hume, T.; Johnston, S.; Kravec, S.; Lovitt, L.; Nanda, N.; Olsson, C.… [find](https://scholar.google.com/scholar?q=Bai%2C+Y%2E%3B+Jones%2C+A%2E%3B+Ndousse%2C+K%2E%3B+Askell%2C+A%2E%3B+Chen%2C+A%2E%3B+Das%2DSarma%2C+N%2E%3B+Drain%2C+D%2E%3B+Fort%2C+S%2E%3B+Ganguli%2C+D%2E%3B+Henighan%2C+T%2E%3B+Joseph%2C+N%2E%3B+Kadavath%2C+S%2E%3B+Kernion%2C+J%2E%3B+Conerly%2C+T%2E%3B+El%2DShowk%2C+S%2E%3B+Elhage%2C+N%2E%3B+Hatf)
2. Beeching, E.; Lambert, N.; Tunstall, L.; Rajani, N.; and von Werra, L. 2023. Open LLM Leaderboard. [https://huggingface.co/spaces/HuggingFaceH4/](https://huggingface.co/spaces/HuggingFaceH4/) open llm leaderboard. [↗](https://huggingface.co/spaces/HuggingFaceH4/)
3. Boizard, N.; El Haddad, K.; Hudelot, C.; and Colombo, P. 2024. [Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs](https://md.research.elamur.ai/free/en/papers/boizard-2024-towards-cross-tokenizer-distillation-universal/).
4. Christiano, P. F.; Leike, J.; Brown, T. B.; Martic, M.; Legg, S.; and Amodei, D. 2017. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems, volume 30. [find](https://scholar.google.com/scholar?q=Christiano%2C+P%2E+F%2E%3B+Leike%2C+J%2E%3B+Brown%2C+T%2E+B%2E%3B+Martic%2C+M%2E%3B+Legg%2C+S%2E%3B+and+Amodei%2C+D%2E+2017%2E+Deep+Reinforcement+Learning+from+Human+Preferences%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+)
5. Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018. [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv:1803.05457.
6. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J. 2021. [Training Verifiers to Solve Math Word Problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv:2110.14168.
7. Cui, X.; Zhu, M.; Qin, Y.; Xie, L.; Zhou, W.; and Li, H. 2025. Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models. In Proceedings of the AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Cui%2C+X%2E%3B+Zhu%2C+M%2E%3B+Qin%2C+Y%2E%3B+Xie%2C+L%2E%3B+Zhou%2C+W%2E%3B+and+Li%2C+H%2E+2025%2E+Multi%2DLevel+Optimal+Transport+for+Universal+Cross%2DTokenizer+Knowledge+Distillation+on+Language+Models%2E+In+Proceedings+of+the+AAAI+Confere)
8. Gao, S.; Wan, F.; Guo, J.; Quan, X.; and Wang, Q. 2025. Advantage-Guided Distillation for Preference Alignment in Small Language Models. arXiv:2502.17927. — [arXiv:2502.17927](https://arxiv.org/abs/2502.17927) [find](https://scholar.google.com/scholar?q=Gao%2C+S%2E%3B+Wan%2C+F%2E%3B+Guo%2C+J%2E%3B+Quan%2C+X%2E%3B+and+Wang%2C+Q%2E+2025%2E+Advantage%2DGuided+Distillation+for+Preference+Alignment+in+Small+Language+Models%2E+arXiv%3A2502%2E17927%2E+%E2%80%94+%5BarXiv%3A2502%2E17927%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F25)
9. Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2021. [Measuring Massive Multitask Language Understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv:2009.03300.
10. Hinton, G.; Vinyals, O.; and Dean, J. 2015. [Distilling the Knowledge in a Neural Network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). arXiv preprint arXiv:1503.02531.
11. Hsieh, C.-Y.; Li, C.-L.; Yeh, C.-K.; Nakhost, H.; Fujii, Y.; Ratner, A.; Krishna, R.; Lee, C.-Y.; and Pfister, T. 2023. [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://md.research.elamur.ai/free/en/papers/hsieh-2023-distilling-step-step-outperforming-larger/). arXiv:2305.02301.
12. Kloek, T.; and van Dijk, H. K. 1978. Bayesian Estimates of Equation System Parameters: An Application of Integration by Monte Carlo. Econometrica: Journal of the Econometric Society, 46(1): 1–19. [find](https://scholar.google.com/scholar?q=Kloek%2C+T%2E%3B+and+van+Dijk%2C+H%2E+K%2E+1978%2E+Bayesian+Estimates+of+Equation+System+Parameters%3A+An+Application+of+Integration+by+Monte+Carlo%2E+Econometrica%3A+Journal+of+the+Econometric+Society%2C+46%281%29%3A+1%E2%80%9319%2E)
13. Lin, S.; Hilton, J.; and Evans, O. 2022. [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). arXiv:2109.07958.
14. Liu, A.; Bai, H.; Lu, Z.; Sun, Y.; Kong, X.; Wang, S.; Shan, J.; Jose, A. M.; Liu, X.; Wen, L.; Yu, P. S.; and Cao, M. 2025. TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights. arXiv:2410.04350. — [arXiv:2410.04350](https://arxiv.org/abs/2410.04350) [find](https://scholar.google.com/scholar?q=Liu%2C+A%2E%3B+Bai%2C+H%2E%3B+Lu%2C+Z%2E%3B+Sun%2C+Y%2E%3B+Kong%2C+X%2E%3B+Wang%2C+S%2E%3B+Shan%2C+J%2E%3B+Jose%2C+A%2E+M%2E%3B+Liu%2C+X%2E%3B+Wen%2C+L%2E%3B+Yu%2C+P%2E+S%2E%3B+and+Cao%2C+M%2E+2025%2E+TIS%2DDPO%3A+Token%2Dlevel+Importance+Sampling+for+Direct+Preference+Optimization)
15. Loshchilov, I.; and Hutter, F. 2019. Decoupled Weight Decay Regularization. arXiv:1711.05101. [find](https://scholar.google.com/scholar?q=Loshchilov%2C+I%2E%3B+and+Hutter%2C+F%2E+2019%2E+Decoupled+Weight+Decay+Regularization%2E+arXiv%3A1711%2E05101%2E)
16. OpenAI. 2023. [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv:2303.08774.
17. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P.; Leike, J.; and Lowe, R. 2022. [Training language models to follow instruction…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
18. Rafailov, R.; Sharma, A.; Mitchell, E.; Ermon, S.; Manning, C. D.; and Finn, C. 2024. [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). arXiv:2305.18290.
19. Ren, M.; Zeng, W.; Yang, B.; and Urtasun, R. 2019. Learning to Reweight Examples for Robust Deep Learning. arXiv:1803.09050. — [arXiv:1803.09050](https://arxiv.org/abs/1803.09050) [find](https://scholar.google.com/scholar?q=Ren%2C+M%2E%3B+Zeng%2C+W%2E%3B+Yang%2C+B%2E%3B+and+Urtasun%2C+R%2E+2019%2E+Learning+to+Reweight+Examples+for+Robust+Deep+Learning%2E+arXiv%3A1803%2E09050%2E+%E2%80%94+%5BarXiv%3A1803%2E09050%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1803%2E09050%29)
20. Sakaguchi, K.; Bras, R. L.; Bhagavatula, C.; and Choi, Y. 2019. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. arXiv:1907.10641. [find](https://scholar.google.com/scholar?q=Sakaguchi%2C+K%2E%3B+Bras%2C+R%2E+L%2E%3B+Bhagavatula%2C+C%2E%3B+and+Choi%2C+Y%2E+2019%2E+WinoGrande%3A+An+Adversarial+Winograd+Schema+Challenge+at+Scale%2E+arXiv%3A1907%2E10641%2E)
21. Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347. [find](https://scholar.google.com/scholar?q=Schulman%2C+J%2E%3B+Wolski%2C+F%2E%3B+Dhariwal%2C+P%2E%3B+Radford%2C+A%2E%3B+and+Klimov%2C+O%2E+2017%2E+Proximal+Policy+Optimization+Algorithms%2E+arXiv%3A1707%2E06347%2E)
22. Stiennon, N.; Ouyang, L.; Wu, J.; Ziegler, D. M.; Lowe, R.; Voss, C.; Radford, A.; Amodei, D.; and Christiano, P. 2022. Learning to summarize from human feedback. arXiv:2009.01325. Sutawika, L.; Gao, L.; Schoelkopf, H.; Biderman, S.; Tow, J.; Abbasi, B.; ben fattori; Lovering, C.; farzanehnakhaee70;… [find](https://scholar.google.com/scholar?q=Stiennon%2C+N%2E%3B+Ouyang%2C+L%2E%3B+Wu%2C+J%2E%3B+Ziegler%2C+D%2E+M%2E%3B+Lowe%2C+R%2E%3B+Voss%2C+C%2E%3B+Radford%2C+A%2E%3B+Amodei%2C+D%2E%3B+and+Christiano%2C+P%2E+2022%2E+Learning+to+summarize+from+human+feedback%2E+arXiv%3A2009%2E01325%2E+Sutawika%2C+L%2E%3B+Gao%2C+)
23. Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? arXiv:1905.07830. [find](https://scholar.google.com/scholar?q=Zellers%2C+R%2E%3B+Holtzman%2C+A%2E%3B+Bisk%2C+Y%2E%3B+Farhadi%2C+A%2E%3B+and+Choi%2C+Y%2E+2019%2E+HellaSwag%3A+Can+a+Machine+Really+Finish+Your+Sentence%3F+arXiv%3A1905%2E07830%2E)
24. Zeng, Y.; Liu, G.; Ma, W.; Yang, N.; Zhang, H.; and Wang, J. 2024. Token-level direct preference optimization. In Forty-first International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Zeng%2C+Y%2E%3B+Liu%2C+G%2E%3B+Ma%2C+W%2E%3B+Yang%2C+N%2E%3B+Zhang%2C+H%2E%3B+and+Wang%2C+J%2E+2024%2E+Token%2Dlevel+direct+preference+optimization%2E+In+Forty%2Dfirst+International+Conference+on+Machine+Learning%2E)
25. Zhang, X.; Zhang, S.; Liang, Y.; Meng, F.; Chen, Y.; Xu, J.; and Zhou, J. 2025. A Dual-Space Framework for General Knowledge Distillation of Large Language Models. arXiv:2504.11426. — [arXiv:2504.11426](https://arxiv.org/abs/2504.11426) [find](https://scholar.google.com/scholar?q=Zhang%2C+X%2E%3B+Zhang%2C+S%2E%3B+Liang%2C+Y%2E%3B+Meng%2C+F%2E%3B+Chen%2C+Y%2E%3B+Xu%2C+J%2E%3B+and+Zhou%2C+J%2E+2025%2E+A+Dual%2DSpace+Framework+for+General+Knowledge+Distillation+of+Large+Language+Models%2E+arXiv%3A2504%2E11426%2E+%E2%80%94+%5BarXiv%3A2504%2E1142)

*Updated 31 October 2026*