---
title: "AdaptBPE: From General Purpose to Specialized Tokenizers"
description: "Liyanage and Yvon propose AdaptBPE, a post-hoc procedure that edits an existing… 46.5 accuracy. 45 references, cited by 1 paper here."
authors: ["Vijini Pilana Liyanage", "François Yvon"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.eacl-long.119/"
doi: "10.18653/v1/2026.eacl-long.119"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liyanage-2026-adaptbpe-general-purpose-specialized-tokenizers/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liyanage-2026-adaptbpe-general-purpose-specialized-tokenizers/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/9/) / AdaptBPE: From General Purpose to Specialized Tokenizers

# AdaptBPE: From General Purpose to Specialized Tokenizers

**Authors:** Vijini Pilana Liyanage and [François Yvon](https://md.research.elamur.ai/free/en/authors/francois-yvon/) · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-long.119/) · [PDF](https://aclanthology.org/2026.eacl-long.119.pdf) · [DOI 10.18653/v1/2026.eacl-long.119](https://doi.org/10.18653/v1/2026.eacl-long.119) · [OpenAlex](https://openalex.org/W7140126971)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
General-purpose BPE tokenizers in multilingual LLMs carry large vocabularies. When a model is specialized to a few languages or a single domain, many tokens become dead weight, and segmentations of in-domain text are longer than necessary. Liyanage and Yvon propose AdaptBPE, a post-hoc procedure that edits an existing merge list without touching model weights or the original training data. It starts from the first N merges of a pretrained tokenizer, such as those of BLOOM or Llama-3. It then greedily swaps rarely used merges for later merges that shorten an adaptation corpus. Some merges are marked as "virtual" (scaffold) units, which are undone after tokenization. The result is a proper merge list that stays compatible with the original LLM.

Experiments cover medical text (PubMed, EMEA), FLORES translation and zero-shot classification across several languages. At equal vocabulary size, AdaptBPE compresses better than truncation and frequency baselines. Its translation quality is close to that of the full vocabulary. This offers a cheap, plug-and-play way to shrink the active vocabulary and reduce inference cost for specialized deployments.

## Key points
- **Baselines:** AdaptBPE is compared with First k, First k>0 and Top k merge selection, plus the full vocabulary. The main budget is N = 15k merges, with 30k for bilingual MT.
- **Perplexity:** On PubMed with N = 15k, perplexity drops well below the full-vocabulary level.
  - BLOOM: 18.2 → 11.4.
  - Llama-3: 12.6 → 8.0.
- **EMEA translation:** On 5-shot English→French medical translation, BLEU with AdaptBPE matches or slightly exceeds the full tokenizer.
  - BLOOM: 42.9 vs 42.9.
  - Llama-3: 45.6 vs 45.4.
- **FLORES translation:** For eng–fra and eng–oci with BLOOM and Llama-3, AdaptBPE stays within roughly 0.3–1.1 BLEU of the full vocabulary. It beats all baselines.
- **Zero-shot classification:** With fine-tuned Llama-3, trimming hurts English most (54.9 → 46.5 accuracy). The effect is minor for most other languages. Code and data are released at github.com/vijini/Adapt-BPE.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Centre National de la Recherche Scientifique](https://md.research.elamur.ai/free/en/labs/centre-national-de-la-recherche-scientifique/) — Research institute, Paris, France · 169 works · 62,488 citations
- [Sorbonne Université](https://md.research.elamur.ai/free/en/universities/sorbonne-universite/) — University, Paris, France · 34 works · 7,389 citations
- [Institut Systèmes Intelligents et de Robotique](https://md.research.elamur.ai/free/en/labs/institut-systemes-intelligents-et-de-robotique/) — Research institute, Paris, France · 8 works · 206 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) · 124 works

## Cite

Cite as: Vijini Pilana Liyanage and François Yvon. 2026. AdaptBPE: From General Purpose to Specialized Tokenizers. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liyanage2026adaptbpe,
  title = {{AdaptBPE: From General Purpose to Specialized Tokenizers}},
  author = {Vijini Pilana Liyanage and François Yvon},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.119},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.119}
}
```

- [← previous Active Learning for Corpus Refinement: Cost-Effective Preprocessing to Improve Validity of Applied Quantitative Text Analysis](https://md.research.elamur.ai/free/en/papers/steglich-2026-active-learning-corpus-refinement-cost/)
- [next → Adaptive Data Flywheel: Applying MAPE Control Loops to AI Agent Improvement](https://md.research.elamur.ai/free/en/papers/shukla-2026-adaptive-data-flywheel-applying-mape/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/9/)

## References

45 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Haonan Gao, and En-Shiun Annie Lee. 2024. [SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects](https://md.research.elamur.ai/free/en/papers/adelani-2023-sib-200-simple-inclusive-big/). In Proceedings of the 18th Conference of the Eu…
2. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, and 29 others. 2023. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). Preprint, arXiv:2309.16609.
3. Thomas Bauwens and Pieter Delobelle. 2024. BPE-knockout: Pruning pre-existing BPE tokenisers with backwards-compatible morphological semi-supervision. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vo… [find](https://scholar.google.com/scholar?q=Thomas+Bauwens+and+Pieter+Delobelle%2E+2024%2E+BPE%2Dknockout%3A+Pruning+pre%2Dexisting+BPE+tokenisers+with+backwards%2Dcompatible+morphological+semi%2Dsupervision%2E+In+Proceedings+of+the+2024+Conference+of+the+Nort)
4. Martin Berglund and Brink van der Merwe. 2023. Formalizing BPE tokenization. Electronic Proceedings in Theoretical Computer Science, 388:16–27. [find](https://scholar.google.com/scholar?q=Martin+Berglund+and+Brink+van+der+Merwe%2E+2023%2E+Formalizing+BPE+tokenization%2E+Electronic+Proceedings+in+Theoretical+Computer+Science%2C+388%3A16%E2%80%9327%2E)
5. Workshop BigScience, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sa… [find](https://scholar.google.com/scholar?q=Workshop+BigScience%2C+Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+Jonathan+Tow%2C+A)
6. Nikolay Bogoychev, Pinzhen Chen, Barry Haddow, and Alexandra Birch. 2024. The ups and downs of large language model inference with vocabulary trimming by language heuristics. In Proceedings of the Fifth Workshop on Insights from Negative Results in NLP, pages 148–153, Mexico City, Mexico. Associatio… [find](https://scholar.google.com/scholar?q=Nikolay+Bogoychev%2C+Pinzhen+Chen%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2024%2E+The+ups+and+downs+of+large+language+model+inference+with+vocabulary+trimming+by+language+heuristics%2E+In+Proceedings+of+the+Fif)
7. Kaj Bostrom and Greg Durrett. 2020. [Byte pair encoding is suboptimal for language model pretraining](https://md.research.elamur.ai/free/en/papers/bostrom-2020-byte-pair-encoding-is-suboptimal/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4617–4624, Online. Association for Computational Linguistics.
8. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, a… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretchen)
9. Pavel Chizhov, Catherine Arnett, Elizaveta Korotkova, and Ivan P. Yamshchikov. 2024. BPE gets picky: Efficient vocabulary refinement during tokenizer training. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16587–16604, Miami, Florida, USA. Associati… [find](https://scholar.google.com/scholar?q=Pavel+Chizhov%2C+Catherine+Arnett%2C+Elizaveta+Korotkova%2C+and+Ivan+P%2E+Yamshchikov%2E+2024%2E+BPE+gets+picky%3A+Efficient+vocabulary+refinement+during+tokenizer+training%2E+In+Proceedings+of+the+2024+Conference+on)
10. Marco Cognetta, Tatsuya Hiraoka, Rico Sennrich, Yuval Pinter, and Naoaki Okazaki. 2024. An analysis of BPE vocabulary trimming in neural machine translation. In Proceedings of the Fifth Workshop on Insights from Negative Results in NLP, pages 48–50, Mexico City, Mexico. Association for Computational… [find](https://scholar.google.com/scholar?q=Marco+Cognetta%2C+Tatsuya+Hiraoka%2C+Rico+Sennrich%2C+Yuval+Pinter%2C+and+Naoaki+Okazaki%2E+2024%2E+An+analysis+of+BPE+vocabulary+trimming+in+neural+machine+translation%2E+In+Proceedings+of+the+Fifth+Workshop+on+In)
11. Thomas M. Cover and Joy A. Thomas. 1991. Elememts of Infomation Theory. John Wiley and Sons. [find](https://scholar.google.com/scholar?q=Thomas+M%2E+Cover+and+Joy+A%2E+Thomas%2E+1991%2E+Elememts+of+Infomation+Theory%2E+John+Wiley+and+Sons%2E)
12. Severino Da Dalt, Joan Llop, Irene Baucells, Marc Pamies, Yishi Xu, Aitor Gonzalez-Agirre, and Marta Villegas. 2024. FLOR: On the effectiveness of language adaptation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLI… [find](https://scholar.google.com/scholar?q=Severino+Da+Dalt%2C+Joan+Llop%2C+Irene+Baucells%2C+Marc+Pamies%2C+Yishi+Xu%2C+Aitor+Gonzalez%2DAgirre%2C+and+Marta+Villegas%2E+2024%2E+FLOR%3A+On+the+effectiveness+of+language+adaptation%2E+In+Proceedings+of+the+2024+Joint)
13. Konstantin Dobler and Gerard de Melo. 2023. [FOCUS: Effective embedding initialization for monolingual specialization of multilingual models](https://md.research.elamur.ai/free/en/papers/dobler-2023-focus-effective-embedding-initialization-monolingual/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13440–13454, Singapore. Association for Computational Linguist…
14. Zhili Feng, Tanya Marwah, Lester Mackey, David Alvarez-Melis, and Nicolo Fusi. 2024. Adapting language models via token translation. In Workshop on Machine Learning and Compression, NeurIPS 2024. [find](https://scholar.google.com/scholar?q=Zhili+Feng%2C+Tanya+Marwah%2C+Lester+Mackey%2C+David+Alvarez%2DMelis%2C+and+Nicolo+Fusi%2E+2024%2E+Adapting+language+models+via+token+translation%2E+In+Workshop+on+Machine+Learning+and+Compression%2C+NeurIPS+2024%2E)
15. Philip Gage. 1994. A new algorithm for data compression. Computer Users Journal, 12(2):23–38. [find](https://scholar.google.com/scholar?q=Philip+Gage%2E+1994%2E+A+new+algorithm+for+data+compression%2E+Computer+Users+Journal%2C+12%282%29%3A23%E2%80%9338%2E)
16. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
17. Jue Hou, Anisia Katinskaia, Anh-Duc Vu, and Roman Yangarber. 2023. Effects of sub-word segmentation on performance of transformer language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7413–7425, Singapore. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Jue+Hou%2C+Anisia+Katinskaia%2C+Anh%2DDuc+Vu%2C+and+Roman+Yangarber%2E+2023%2E+Effects+of+sub%2Dword+segmentation+on+performance+of+transformer+language+models%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Me)
18. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations.
19. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Alain+Le+Noac%E2%80%99h%2C+Haonan+Li%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Chris+Oci)
20. Ayyoob Imani, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André Martins, François Yvon, and Hinrich Schütze. 2023. Glot500: Scaling multilingual corpora and language models to 500 languages. In Proceedings of the 61st Annual Meeti… [find](https://scholar.google.com/scholar?q=Ayyoob+Imani%2C+Peiqin+Lin%2C+Amir+Hossein+Kargaran%2C+Silvia+Severini%2C+Masoud+Jalili+Sabet%2C+Nora+Kassner%2C+Chunlan+Ma%2C+Helmut+Schmid%2C+Andr%C3%A9+Martins%2C+Fran%C3%A7ois+Yvon%2C+and+Hinrich+Sch%C3%BCtze%2E+2023%2E+Glot500%3A+Scalin)
21. Leonidas Gee, Leonardo Rigutini, Marco Ernandes, and Andrea Zugarini. 2023. Multi-word tokenization for sequence compression. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 612–621, Singapore. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Leonidas+Gee%2C+Leonardo+Rigutini%2C+Marco+Ernandes%2C+and+Andrea+Zugarini%2E+2023%2E+Multi%2Dword+tokenization+for+sequence+compression%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+Lang)
22. Taku Kudo. 2018. [Subword regularization: Improving neural network translation models with multiple subword candidates](https://md.research.elamur.ai/free/en/papers/kudo-2018-subword-regularization-improving-neural-network/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational…
23. Leonidas Gee, Andrea Zugarini, Leonardo Rigutini, and Paolo Torroni. 2022. Fast vocabulary transfer for language model compression. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 409– 416, Abu Dhabi, UAE. Association for Computational… [find](https://scholar.google.com/scholar?q=Leonidas+Gee%2C+Andrea+Zugarini%2C+Leonardo+Rigutini%2C+and+Paolo+Torroni%2E+2022%2E+Fast+vocabulary+transfer+for+language+model+compression%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natura)
24. Saibo Geng, Nathan Ranchin, Yunzhen Yao, Maxime Peyrard, Chris Wendler, Michael Gastpar, and Robert West. 2025. Zip2zip: Inference-time adaptive vocabularies for language models via token compression. In ES-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models. [find](https://scholar.google.com/scholar?q=Saibo+Geng%2C+Nathan+Ranchin%2C+Yunzhen+Yao%2C+Maxime+Peyrard%2C+Chris+Wendler%2C+Michael+Gastpar%2C+and+Robert+West%2E+2025%2E+Zip2zip%3A+Inference%2Dtime+adaptive+vocabularies+for+language+models+via+token+compression%2E)
25. Omer Goldman, Avi Caciularu, Matan Eyal, Kris Cao, Idan Szpektor, and Reut Tsarfaty. 2024. Unpacking tokenization: Evaluating text compression and its correlation with model performance. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2274–2286, Bangkok, Thailand. Assoc… [find](https://scholar.google.com/scholar?q=Omer+Goldman%2C+Avi+Caciularu%2C+Matan+Eyal%2C+Kris+Cao%2C+Idan+Szpektor%2C+and+Reut+Tsarfaty%2E+2024%2E+Unpacking+tokenization%3A+Evaluating+text+compression+and+its+correlation+with+model+performance%2E+In+Findings+o)
26. Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gérard Dupont, Stella Biderman, Anna Rogers, Loub… — [The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2023-bigscience-roots-corpus-1-6tb/)
27. Yuxi Li, Yi Liu, Gelei Deng, Ying Zhang, Wenjia Song, Ling Shi, Kailong Wang, Yuekang Li, Yang Liu, and Haoyu Wang. 2024. Glitch tokens in large language models: Categorization taxonomy and effective detection. Proceedings of the ACM Software Engineering Conference, 1(FSE). [find](https://scholar.google.com/scholar?q=Yuxi+Li%2C+Yi+Liu%2C+Gelei+Deng%2C+Ying+Zhang%2C+Wenjia+Song%2C+Ling+Shi%2C+Kailong+Wang%2C+Yuekang+Li%2C+Yang+Liu%2C+and+Haoyu+Wang%2E+2024%2E+Glitch+tokens+in+large+language+models%3A+Categorization+taxonomy+and+effective+)
28. Haoran Lian, Yizhe Xiong, Jianwei Niu, Shasha Mo, Zhenpeng Su, Zijia Lin, Hui Chen, Jungong Han, and Guiguang Ding. 2025. Scaffold-BPE: Enhancing byte pair encoding for large language models with simple and effective scaffold token removal. In Proceedings of the AAAI Conference on Artificial Intelli… [find](https://scholar.google.com/scholar?q=Haoran+Lian%2C+Yizhe+Xiong%2C+Jianwei+Niu%2C+Shasha+Mo%2C+Zhenpeng+Su%2C+Zijia+Lin%2C+Hui+Chen%2C+Jungong+Han%2C+and+Guiguang+Ding%2E+2025%2E+Scaffold%2DBPE%3A+Enhancing+byte+pair+encoding+for+large+language+models+with+simp)
29. Peiqin Lin, Shaoxiong Ji, Jörg Tiedemann, André F. T. Martins, and Hinrich Schütze. 2024. [Mala-500: Massive language adaptation of large language models](https://md.research.elamur.ai/free/en/papers/lin-2024-mala-500-massive-language-adaptation/). CoRR, abs/2401.13303.
30. Chengyuan Liu, Shihang Wang, Lizhi Qing, Kun Kuang, Yangyang Kang, Changlong Sun, and Fei Wu. 2024. Gold panning in vocabulary: An adaptive method for vocabulary expansion of domain-specific LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7442–7… [find](https://scholar.google.com/scholar?q=Chengyuan+Liu%2C+Shihang+Wang%2C+Lizhi+Qing%2C+Kun+Kuang%2C+Yangyang+Kang%2C+Changlong+Sun%2C+and+Fei+Wu%2E+2024%2E+Gold+panning+in+vocabulary%3A+An+adaptive+method+for+vocabulary+expansion+of+domain%2Dspecific+LLMs%2E+In+)
31. Siyang Liu, Naihao Deng, Sahand Sabour, Yilin Jia, Minlie Huang, and Rada Mihalcea. 2023. Task-adaptive tokenization: Enhancing long-form text generation efficacy in mental health and beyond. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15264– 1528… [find](https://scholar.google.com/scholar?q=Siyang+Liu%2C+Naihao+Deng%2C+Sahand+Sabour%2C+Yilin+Jia%2C+Minlie+Huang%2C+and+Rada+Mihalcea%2E+2023%2E+Task%2Dadaptive+tokenization%3A+Enhancing+long%2Dform+text+generation+efficacy+in+mental+health+and+beyond%2E+In+Proce)
32. Yinquan Lu, Wenhao Zhu, Lei Li, Yu Qiao, and Fei Yuan. 2024. LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languages. CoRR, abs/2407.05975. [find](https://scholar.google.com/scholar?q=Yinquan+Lu%2C+Wenhao+Zhu%2C+Lei+Li%2C+Yu+Qiao%2C+and+Fei+Yuan%2E+2024%2E+LLaMAX%3A+Scaling+Linguistic+Horizons+of+LLM+by+Enhancing+Translation+Capabilities+Beyond+100+Languages%2E+CoRR%2C+abs%2F2407%2E05975%2E)
33. Manuel Mager, Arturo Oncevay, Elisabeth Mager, Katharina Kann, and Thang Vu. 2022. BPE vs. morphological segmentation: A case study on machine translation of four polysynthetic languages. In Findings of the Association for Computational Linguistics: ACL 2022, pages 961–971, Dublin, Ireland. Associat… [find](https://scholar.google.com/scholar?q=Manuel+Mager%2C+Arturo+Oncevay%2C+Elisabeth+Mager%2C+Katharina+Kann%2C+and+Thang+Vu%2E+2022%2E+BPE+vs%2E+morphological+segmentation%3A+A+case+study+on+machine+translation+of+four+polysynthetic+languages%2E+In+Findings+)
34. Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. 2022. [WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models](https://md.research.elamur.ai/free/en/papers/minixhofer-2022-wechsel-effective-initialization-subword-embeddings/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hum…
35. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, and 20 o… [find](https://scholar.google.com/scholar?q=NLLB+Team%2C+Marta+R%2E+Costa%2Djuss%C3%A0%2C+James+Cross%2C+Onur+%C3%87elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+Jean+Maillard%2C+Anna+Sun%2C+Skyler+Wang%2C+Guillaume+We)
36. Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. 2019. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, volum… [find](https://scholar.google.com/scholar?q=Yaniv+Ovadia%2C+Emily+Fertig%2C+Jie+Ren%2C+Zachary+Nado%2C+D%2E+Sculley%2C+Sebastian+Nowozin%2C+Joshua+Dillon%2C+Balaji+Lakshminarayanan%2C+and+Jasper+Snoek%2E+2019%2E+Can+you+trust+your+model%27s+uncertainty%3F+evaluating+pre)
37. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
38. Matt Post. 2018. [A call for clarity in reporting BLEU scores](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/). In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics.
39. Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2020. [BPE-dropout: Simple and effective subword regularization](https://md.research.elamur.ai/free/en/papers/provilkov-2020-bpe-dropout-simple-effective-subword/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1882–1892, Online. Association for Computational Linguistics.
40. François Remy, Pieter Delobelle, Hayastan Avetisyan, Alfiya Khabibullina, Miryam de Lhoneux, and Thomas Demeester. 2024. Trans-tokenization and cross-lingual vocabulary transfers: Language adaptation of LLMs for low-resource NLP. In First Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Fran%C3%A7ois+Remy%2C+Pieter+Delobelle%2C+Hayastan+Avetisyan%2C+Alfiya+Khabibullina%2C+Miryam+de+Lhoneux%2C+and+Thomas+Demeester%2E+2024%2E+Trans%2Dtokenization+and+cross%2Dlingual+vocabulary+transfers%3A+Language+adaptation+)
41. Vin Sachidananda, Jason Kessler, and Yi-An Lai. 2021. Efficient domain adaptation of language models via adaptive tokenization. In Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing, pages 155–165, Virtual. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Vin+Sachidananda%2C+Jason+Kessler%2C+and+Yi%2DAn+Lai%2E+2021%2E+Efficient+domain+adaptation+of+language+models+via+adaptive+tokenization%2E+In+Proceedings+of+the+Second+Workshop+on+Simple+and+Efficient+Natural+La)
42. Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, and 22 others.… — [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
43. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin…
44. Pranaydeep Singh, Eneko Agirre, Gorka Azkune, Orphee De Clercq, and Els Lefever. 2025. EnerGIZAr: Leveraging GIZA++ for effective tokenizer initialization. In Findings of the Association for Computational Linguistics: ACL 2025, pages 2124–2137, Vienna, Austria. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Pranaydeep+Singh%2C+Eneko+Agirre%2C+Gorka+Azkune%2C+Orphee+De+Clercq%2C+and+Els+Lefever%2E+2025%2E+EnerGIZAr%3A+Leveraging+GIZA%2B%2B+for+effective+tokenizer+initialization%2E+In+Findings+of+the+Association+for+Computati)
45. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Sl… [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Thomas+Mesnard%2C+Cassidy+Hardin%2C+Robert+Dadashi%2C+Surya+Bhupatiraju%2C+Shreya+Pathak%2C+Laurent+Sifre%2C+Morgane+Rivi%C3%A8re%2C+Mihir+Sanjay+Kale%2C+Juliette+Love%2C+Pouya+Tafti%2C+L%C3%A9onard+Hussenot%2C+Pier+Gius)

*Updated 31 October 2026*