---
title: "MUTANT: A Recipe for Multilingual Tokenizer Design (2026)"
description: "The authors propose MUTANT, a reproducible procedure for designing multilingual tokenizers, and use it to build MUTANT-Indic. 63 references."
authors: ["Souvik Rana", "Ashish Kulkarni", "Arul Menezes", "Chandra Khatri", "Shubham Agarwal"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.2146/"
doi: "10.18653/v1/2026.acl-long.2146"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rana-2026-mutant-recipe-multilingual-tokenizer-design/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rana-2026-mutant-recipe-multilingual-tokenizer-design/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/43/) / MUTANT: A Recipe for Multilingual Tokenizer Design

# MUTANT: A Recipe for Multilingual Tokenizer Design

**Authors:** Souvik Rana, Ashish Kulkarni, [Arul Menezes](https://md.research.elamur.ai/free/en/authors/arul-menezes/), Chandra Khatri and [Shubham Agarwal](https://md.research.elamur.ai/free/en/authors/shubham-agarwal/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.2146/) · [PDF](https://aclanthology.org/2026.acl-long.2146.pdf) · [DOI 10.18653/v1/2026.acl-long.2146](https://doi.org/10.18653/v1/2026.acl-long.2146) · [OpenAlex](https://openalex.org/W4416019584)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Tokenizers built mainly for English break Indian-language text into many small pieces. The LLaMA-4 tokenizer, for instance, needs roughly 10.5 tokens per word for Oriya. This raises training and inference costs and treats languages unequally. The authors propose MUTANT, a reproducible procedure for designing multilingual tokenizers, and use it to build MUTANT-Indic. It covers English, the 22 scheduled languages of India, and code. The recipe has three parts:
- It filters and normalizes a varied corpus drawn from web text, Wikipedia, math and code.
- It sets per-script vocabulary budgets by adjusting the proportions of the training data, rather than gluing separately trained tokenizers together.
- It trains byte-level BPE in two phases. The first allows merges only inside words; the second permits merges across word boundaries to capture multiword expressions.

MUTANT-Indic cuts average fertility by 39.5% compared with LLaMA4 and by 18% compared with Sutra, the strongest prior tokenizer. It also gives 44% higher inference throughput than LLaMA4 while keeping English and Indic benchmark results comparable. The work shows that careful tokenizer design yields cheaper and more equitable multilingual LLMs.

## Key points
- **Evaluation scope:** 22 Indic languages, English and code. The authors released an evaluation framework (github.com/ola-krutrim/MUTANT) and a dataset (krutrim-ai-labs/MUTANT on Hugging Face).
- **Two-stage training:** The curriculum follows the SuperBPE approach. The best switch from subword to cross-word merges comes at 85–90% of the target vocabulary. This two-stage design beat the single-stage BoundlessBPE on fertility for the top 10 Indic languages and English.
- **Vocabulary allocation:** Matching the training corpus to the desired per-script budgets worked better than explicitly merging script-specific vocabularies. The merged approach caused interference and inconsistent segmentation across scripts.
- **Ablation findings:**
  - Gains plateau beyond about 10 GB of Stage 1 training data.
  - Vocabularies larger than 200K bring no notable fertility gain.
  - NFKC normalization gives small but consistent improvements.
- **Limitations:**
  - Downstream tests use models of at most 1B parameters, in both pretraining-from-scratch and continual-pretraining settings.
  - Morphology-aware pre-tokenization was studied but left out of the final tokenizer because of its high inference latency.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 882 works · [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) · 124 works

## Cite

Cite as: Souvik Rana, Ashish Kulkarni, Arul Menezes, Chandra Khatri and Shubham Agarwal. 2026. MUTANT: A Recipe for Multilingual Tokenizer Design. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{rana2026mutant,
  title = {{MUTANT: A Recipe for Multilingual Tokenizer Design}},
  author = {Souvik Rana, Ashish Kulkarni, Arul Menezes, Chandra Khatri and Shubham Agarwal},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2146},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2146}
}
```

- [← previous Muslim: A Deployed Arabic Voice AI Platform for Grounded Islamic Knowledge](https://md.research.elamur.ai/free/en/papers/elnawasany-2026-muslim-deployed-arabic-voice-ai/)
- [next → MVP: Enhancing Video Large Language Models via Self-supervised Masked Video Prediction](https://md.research.elamur.ai/free/en/papers/sun-2026-mvp-enhancing-video-large-language/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/43/)

## References

63 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, and 1 others. 2025. [Phi-4-reasoning technical report](https://md.research.elamur.ai/free/en/papers/abdin-2025-phi-4-reasoning-technical-report/). arXiv preprint arXiv:2504.21318.
2. ADA Sci. 2024. Multilingual tokenization efficiency in large language models: A study on Indian languages. Lattice, 5(2). [find](https://scholar.google.com/scholar?q=ADA+Sci%2E+2024%2E+Multilingual+tokenization+efficiency+in+large+language+models%3A+A+study+on+Indian+languages%2E+Lattice%2C+5%282%29%2E)
3. AI4Bharat. 2022. IndicBERT: A multilingual AL-BERT model for Indian languages. [https://huggingface.co/ai4bharat/indic-bert](https://huggingface.co/ai4bharat/indic-bert). [↗](https://huggingface.co/ai4bharat/indic-bert)
4. Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, and 1 others. 2024. [Tokenizer choice for llm training: Negligible or crucial?](https://md.research.elamur.ai/free/en/papers/ali-2024-tokenizer-choice-llm-training-negligible/) In Findings of the Association for Computational Linguistics: NAACL 202…
5. Maharaj Brahma, NJ Karthika, Atul Singh, Devaraj Adiga, Smruti Bhate, Ganesh Ramakrishnan, Rohit Saluja, and Maunendra Sankar Desarkar. 2025. Morphtok: Morphologically grounded tokenization for indian languages. arXiv preprint arXiv:2504.10335. — [arXiv:2504.10335](https://arxiv.org/abs/2504.10335) [find](https://scholar.google.com/scholar?q=Maharaj+Brahma%2C+NJ+Karthika%2C+Atul+Singh%2C+Devaraj+Adiga%2C+Smruti+Bhate%2C+Ganesh+Ramakrishnan%2C+Rohit+Saluja%2C+and+Maunendra+Sankar+Desarkar%2E+2025%2E+Morphtok%3A+Morphologically+grounded+tokenization+for+indian)
6. Jie Chen, Zhipeng Chen, Jiapeng Wang, Kun Zhou, Yutao Zhu, Jinhao Jiang, Yingqian Min, Wayne Xin Zhao, Zhicheng Dou, Jiaxin Mao, and 1 others. 2024. [Towards effective and efficient continual pre-training of large language models](https://md.research.elamur.ai/free/en/papers/chen-2025-effective-efficient-continual-pre-training/). arXiv preprint arXiv:2407.18743.
7. Pavel Chizhov, Catherine Arnett, Elizaveta Korotkova, and Ivan P. Yamshchikov. 2024. BPE gets picky: Efficient vocabulary refinement during tokenizer training. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16587–16604. Association for Computational… [find](https://scholar.google.com/scholar?q=Pavel+Chizhov%2C+Catherine+Arnett%2C+Elizaveta+Korotkova%2C+and+Ivan+P%2E+Yamshchikov%2E+2024%2E+BPE+gets+picky%3A+Efficient+vocabulary+refinement+during+tokenizer+training%2E+In+Proceedings+of+the+2024+Conference+on)
8. Gautier Dagan, Gabriel Synnaeve, and Baptiste Roziere. 2024. [Getting the most out of your tokenizer for pre-training and domain adaptation](https://md.research.elamur.ai/free/en/papers/dagan-2024-getting-most-out-tokenizer-pre/). arXiv preprint arXiv:2402.01035.
9. Anh Dang, Limor Raviv, and Lukas Galke. 2024. Tokenization and morphology in multilingual language models: A comparative analysis of mT5 and ByT5. arXiv preprint arXiv:2410.11627. [find](https://scholar.google.com/scholar?q=Anh+Dang%2C+Limor+Raviv%2C+and+Lukas+Galke%2E+2024%2E+Tokenization+and+morphology+in+multilingual+language+models%3A+A+comparative+analysis+of+mT5+and+ByT5%2E+arXiv+preprint+arXiv%3A2410%2E11627%2E)
10. DeepSeek. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). [https://github.com/deepseek-ai/DeepSeek-R1](https://github.com/deepseek-ai/DeepSeek-R1). [↗](https://github.com/deepseek-ai/DeepSeek-R1)
11. Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2023. [Towards leaving no indic language behind: Building monolingual corpora, benchmark and models for indic languages](https://md.research.elamur.ai/free/en/papers/doddapaneni-2023-leaving-no-indic-language-behind/). Preprint, arXiv:2212.05409.
12. Abhimanyu Dubey and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
13. Philip Gage. 1994. A new algorithm for data compression. The C Users Journal, 12(2):23–38. [find](https://scholar.google.com/scholar?q=Philip+Gage%2E+1994%2E+A+new+algorithm+for+data+compression%2E+The+C+Users+Journal%2C+12%282%29%3A23%E2%80%9338%2E)
14. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Alain+Le+Noac%E2%80%99h%2C+Haonan+Li%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Chris+Oci)
15. Jonas Geiping, Alex Stein, Manli Shu, Khalid Saifullah, Yuxin Wen, and Tom Goldstein. 2024. Coercing llms to do and reveal (almost) anything. arXiv preprint arXiv:2402.14020. — [arXiv:2402.14020](https://arxiv.org/abs/2402.14020) [find](https://scholar.google.com/scholar?q=Jonas+Geiping%2C+Alex+Stein%2C+Manli+Shu%2C+Khalid+Saifullah%2C+Yuxin+Wen%2C+and+Tom+Goldstein%2E+2024%2E+Coercing+llms+to+do+and+reveal+%28almost%29+anything%2E+arXiv+preprint+arXiv%3A2402%2E14020%2E+%E2%80%94+%5BarXiv%3A2402%2E14020%5D%28http)
16. Gemma Team. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. [find](https://scholar.google.com/scholar?q=Gemma+Team%2E+2025%2E+Gemma+3+technical+report%2E+arXiv+preprint+arXiv%3A2503%2E19786%2E)
17. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
18. Shuhao Gu, Mengdi Zhao, Bowen Zhang, Liangdong Wang, Jijie Li, and Guang Liu. 2024. Retok: Replacing tokenizer to enhance representation efficiency in large language model. Preprint, arXiv:2410.04335. — [arXiv:2410.04335](https://arxiv.org/abs/2410.04335) [find](https://scholar.google.com/scholar?q=Shuhao+Gu%2C+Mengdi+Zhao%2C+Bowen+Zhang%2C+Liangdong+Wang%2C+Jijie+Li%2C+and+Guang+Liu%2E+2024%2E+Retok%3A+Replacing+tokenizer+to+enhance+representation+efficiency+in+large+language+model%2E+Preprint%2C+arXiv%3A2410%2E04335%2E)
19. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
20. Jonathan Hayase, Alisa Liu, Yejin Choi, Sewoong Oh, and Noah A Smith. 2024. Data mixture inference attack: Bpe tokenizers reveal training data compositions. Advances in Neural Information Processing Systems, 37:8956–8983. [find](https://scholar.google.com/scholar?q=Jonathan+Hayase%2C+Alisa+Liu%2C+Yejin+Choi%2C+Sewoong+Oh%2C+and+Noah+A+Smith%2E+2024%2E+Data+mixture+inference+attack%3A+Bpe+tokenizers+reveal+training+data+compositions%2E+Advances+in+Neural+Information+Processing+S)
21. David Heineman, Valentin Hofmann, Ian Magnusson, Yuling Gu, Noah A Smith, Hannaneh Hajishirzi, Kyle Lo, and Jesse Dodge. 2025. Signal and noise: A framework for reducing uncertainty in language model evaluation. arXiv preprint arXiv:2508.13144. — [arXiv:2508.13144](https://arxiv.org/abs/2508.13144) [find](https://scholar.google.com/scholar?q=David+Heineman%2C+Valentin+Hofmann%2C+Ian+Magnusson%2C+Yuling+Gu%2C+Noah+A+Smith%2C+Hannaneh+Hajishirzi%2C+Kyle+Lo%2C+and+Jesse+Dodge%2E+2025%2E+Signal+and+noise%3A+A+framework+for+reducing+uncertainty+in+language+model+)
22. Shashank Mohan Jain. 2022. Hugging face. In Introduction to transformers for NLP: With the hugging face library and models to solve problems, pages 51–67. Springer. [find](https://scholar.google.com/scholar?q=Shashank+Mohan+Jain%2E+2022%2E+Hugging+face%2E+In+Introduction+to+transformers+for+NLP%3A+With+the+hugging+face+library+and+models+to+solve+problems%2C+pages+51%E2%80%9367%2E+Springer%2E)
23. Divyanshu Kakwani and 1 others. 2020. Monolingual corpora, evaluation benchmarks and pre-trained models for 11 major Indian languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4948–4961. [find](https://scholar.google.com/scholar?q=Divyanshu+Kakwani+and+1+others%2E+2020%2E+Monolingual+corpora%2C+evaluation+benchmarks+and+pre%2Dtrained+models+for+11+major+Indian+languages%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMN)
24. N J Karthika, Maharaj Brahma, Rohit Saluja, Ganesh Ramakrishnan, and Maunendra Sankar Desarkar. 2025a. Multilingual tokenization through the lens of indian languages: Challenges and insights. arXiv preprint arXiv:2506.17789. [find](https://scholar.google.com/scholar?q=N+J+Karthika%2C+Maharaj+Brahma%2C+Rohit+Saluja%2C+Ganesh+Ramakrishnan%2C+and+Maunendra+Sankar+Desarkar%2E+2025a%2E+Multilingual+tokenization+through+the+lens+of+indian+languages%3A+Challenges+and+insights%2E+arXiv+pr)
25. N J Karthika, Maharaj Brahma, Rohit Saluja, Ganesh Ramakrishnan, and Maunendra Sankar Desarkar. 2025b. Multilingual tokenization through the lens of indian languages: Challenges and insights. Preprint, arXiv:2506.17789. [find](https://scholar.google.com/scholar?q=N+J+Karthika%2C+Maharaj+Brahma%2C+Rohit+Saluja%2C+Ganesh+Ramakrishnan%2C+and+Maunendra+Sankar+Desarkar%2E+2025b%2E+Multilingual+tokenization+through+the+lens+of+indian+languages%3A+Challenges+and+insights%2E+Preprint)
26. Mohammed Safi Ur Rahman Khan, Priyam Mehta, Ananth Sankar, Umashankar Kumaravelan, Sumanth Doddapaneni, Suriyaprasaad G, Varun Balan G, Sparsh Jain, Anoop Kunchukuttan, Pratyush Kumar, Raj Dabre, and Mitesh M. Khapra. 2024. [Indicllmsuite: A blueprint for creating pre-training and fine-tuning dataset…](https://md.research.elamur.ai/free/en/papers/khan-2024-indicllmsuite-blueprint-creating-pre-training/)
27. Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, and 1 others. 2022. The stack: 3 tb of permissively licensed source code. Transactions on Machine Learning Research. [find](https://scholar.google.com/scholar?q=Denis+Kocetkov%2C+Raymond+Li%2C+Loubna+Ben+Allal%2C+Jia+Li%2C+Chenghao+Mou%2C+Carlos+Mu%C3%B1oz+Ferrandis%2C+Yacine+Jernite%2C+Margaret+Mitchell%2C+Sean+Hughes%2C+Thomas+Wolf%2C+and+1+others%2E+2022%2E+The+stack%3A+3+tb+of+permissi)
28. Taku Kudo. 2018. [Subword regularization: Improving neural network translation models with multiple subword candidates](https://md.research.elamur.ai/free/en/papers/kudo-2018-subword-regularization-improving-neural-network/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational…
29. Taku Kudo and John Richardson. 2018. [SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71.
30. Anoop Kunchukuttan. 2020. The IndicNLP Library. [https://github.com/anoopkunchukuttan/](https://github.com/anoopkunchukuttan/) indic_nlp_library/blob/master/docs/ indicnlp.pdf. [↗](https://github.com/anoopkunchukuttan/)
31. Sander Land and Max Bartolo. 2024. [Fishing for magikarp: Automatically detecting under-trained tokens in large language models](https://md.research.elamur.ai/free/en/papers/land-2024-fishing-magikarp-automatically-detecting-under/). Preprint, arXiv:2405.05417.
32. Haoran Lian, Yizhe Xiong, Jianwei Niu, Shasha Mo, Zhenpeng Su, Zijia Lin, Hui Chen, Peng Liu, Jungong Han, and Guiguang Ding. 2024. Scaffold-BPE: Enhancing byte pair encoding for large language models with simple and effective scaffold token removal. arXiv preprint arXiv:2404.17808. — [arXiv:2404.17808](https://arxiv.org/abs/2404.17808) [find](https://scholar.google.com/scholar?q=Haoran+Lian%2C+Yizhe+Xiong%2C+Jianwei+Niu%2C+Shasha+Mo%2C+Zhenpeng+Su%2C+Zijia+Lin%2C+Hui+Chen%2C+Peng+Liu%2C+Jungong+Han%2C+and+Guiguang+Ding%2E+2024%2E+Scaffold%2DBPE%3A+Enhancing+byte+pair+encoding+for+large+language+models)
33. Qwen. 2024. Qwen3 technical report. qwenlm.github.io/blog/qwen3/. [find](https://scholar.google.com/scholar?q=Qwen%2E+2024%2E+Qwen3+technical+report%2E+qwenlm%2Egithub%2Eio%2Fblog%2Fqwen3%2F%2E)
34. Alisa Liu, Jonathan Hayase, Valentin Hofmann, Sewoong Oh, Noah A. Smith, and Yejin Choi. 2025a. Superbpe: Space travel for language models. Preprint, arXiv:2503.13423. — [arXiv:2503.13423](https://arxiv.org/abs/2503.13423) [find](https://scholar.google.com/scholar?q=Alisa+Liu%2C+Jonathan+Hayase%2C+Valentin+Hofmann%2C+Sewoong+Oh%2C+Noah+A%2E+Smith%2C+and+Yejin+Choi%2E+2025a%2E+Superbpe%3A+Space+travel+for+language+models%2E+Preprint%2C+arXiv%3A2503%2E13423%2E+%E2%80%94+%5BarXiv%3A2503%2E13423%5D%28https%3A%2F%2Farx)
35. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, and 1 others. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. [find](https://scholar.google.com/scholar?q=Jack+W+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+and+1+others%2E+2021%2E+Scaling+language+models%3A+Met)
36. Alisa Liu, Jonathan Hayase, Valentin Hofmann, Sewoong Oh, Noah A. Smith, and Yejin Choi. 2025b. SuperBPE: Space travel for language models. arXiv preprint arXiv:2503.13423. — [arXiv:2503.13423](https://arxiv.org/abs/2503.13423) [find](https://scholar.google.com/scholar?q=Alisa+Liu%2C+Jonathan+Hayase%2C+Valentin+Hofmann%2C+Sewoong+Oh%2C+Noah+A%2E+Smith%2C+and+Yejin+Choi%2E+2025b%2E+SuperBPE%3A+Space+travel+for+language+models%2E+arXiv+preprint+arXiv%3A2503%2E13423%2E+%E2%80%94+%5BarXiv%3A2503%2E13423%5D%28https%3A)
37. Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, and 1 others. 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173. [find](https://scholar.google.com/scholar?q=Anton+Lozhkov%2C+Raymond+Li%2C+Loubna+Ben+Allal%2C+Federico+Cassano%2C+Joel+Lamy%2DPoirier%2C+Nouamane+Tazi%2C+Ao+Tang%2C+Dmytro+Pykhtar%2C+Jiawei+Liu%2C+Yuxiang+Wei%2C+and+1+others%2E+2024%2E+Starcoder+2+and+the+stack+v2%3A+The)
38. Daniil Maksymenko and Oleksii Turuta. 2025. Tokenization efficiency of current foundational large language models for the Ukrainian language. PMC Digital Health, 12380774. [find](https://scholar.google.com/scholar?q=Daniil+Maksymenko+and+Oleksii+Turuta%2E+2025%2E+Tokenization+efficiency+of+current+foundational+large+language+models+for+the+Ukrainian+language%2E+PMC+Digital+Health%2C+12380774%2E)
39. Meta. 2025. The llama 4 herd: The beginning of a new era of natively multimodal intelligence. [https://ai.meta.com/blog/](https://ai.meta.com/blog/) llama-4-multimodal-intelligence/. [↗](https://ai.meta.com/blog/)
40. Sabrina J Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y Lee, Benoît Sagot, and 1 others. 2021. [Between words and characters: A brief history of open-vocabulary modeling and tokenization in nlp](https://md.research.elamur.ai/free/en/papers/mielke-2021-between-words-characters-brief-history/). arXiv preprint arXiv:2112.10508.
41. Mistral. 2024. Mistral nemo. [https://mistral.ai/](https://mistral.ai/) news/mistral-nemo. [↗](https://mistral.ai/)
42. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, and 21 others. 2025. 2 ol… [find](https://scholar.google.com/scholar?q=Team+OLMo%2C+Pete+Walsh%2C+Luca+Soldaini%2C+Dirk+Groeneveld%2C+Kyle+Lo%2C+Shane+Arora%2C+Akshita+Bhagia%2C+Yuling+Gu%2C+Shengyi+Huang%2C+Matt+Jordan%2C+Nathan+Lambert%2C+Dustin+Schwenk%2C+Oyvind+Tafjord%2C+Taira+Anderson%2C+Davi)
43. OpenAI. 2025. [gpt-oss-120b & gpt-oss-20b model card](https://md.research.elamur.ai/free/en/papers/agarwal-2025-gpt-oss-120b-gpt-oss/). Preprint, arXiv:2508.10925.
44. Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, and 1 others. 2024. [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/). arXiv preprint arXiv:2406.17557.
45. Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2023. [Language model tokenizers introduce unfairness between languages](https://md.research.elamur.ai/free/en/papers/petrov-2023-language-model-tokenizers-introduce-unfairness/). Advances in neural information processing systems, 36:36963– 36990. https://
46. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, and 1 others. 2018. Improving language understanding by generative pre-training. TODO. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+Ilya+Sutskever%2C+and+1+others%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+TODO%2E)
47. Varshini Reddy, Craig W Schmidt, Yuval Pinter, and Chris Tanner. 2025. How much is enough? the diminishing returns of tokenization training data. arXiv preprint arXiv:2502.20273. — [arXiv:2502.20273](https://arxiv.org/abs/2502.20273) [find](https://scholar.google.com/scholar?q=Varshini+Reddy%2C+Craig+W+Schmidt%2C+Yuval+Pinter%2C+and+Chris+Tanner%2E+2025%2E+How+much+is+enough%3F+the+diminishing+returns+of+tokenization+training+data%2E+arXiv+preprint+arXiv%3A2502%2E20273%2E+%E2%80%94+%5BarXiv%3A2502%2E20273%5D%28)
48. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. [How good is your tokenizer? on the monolingual performance of multilingual language models](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join…
49. Sarvam. 2024. Sarvam-1: A 2b parameter language model optimized for indian languages. [https://www.sarvam.ai/blogs/sarvam-1](https://www.sarvam.ai/blogs/sarvam-1). [↗](https://www.sarvam.ai/blogs/sarvam-1)
50. Sarvam AI. 2025. Arc-challenge-indic. [https://huggingface.co/datasets/sarvamai/](https://huggingface.co/datasets/sarvamai/) arc-challenge-indic. [↗](https://huggingface.co/datasets/sarvamai/)
51. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, and 1 others. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+and+1+others%2E+2022%2E+Bloom%3A+A+176b%2Dpa)
52. Craig W. Schmidt, Varshini Reddy, Chris Tanner, and Yuval Pinter. 2024. Boundless byte pair encoding: Breaking the pre-tokenization barrier. arXiv preprint arXiv:2504.00178. — [arXiv:2504.00178](https://arxiv.org/abs/2504.00178) [find](https://scholar.google.com/scholar?q=Craig+W%2E+Schmidt%2C+Varshini+Reddy%2C+Chris+Tanner%2C+and+Yuval+Pinter%2E+2024%2E+Boundless+byte+pair+encoding%3A+Breaking+the+pre%2Dtokenization+barrier%2E+arXiv+preprint+arXiv%3A2504%2E00178%2E+%E2%80%94+%5BarXiv%3A2504%2E00178%5D%28https)
53. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Li…
54. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725.
55. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. Preprint, arXiv:1701.06538. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2C+Azalia+Mirhoseini%2C+Krzysztof+Maziarz%2C+Andy+Davis%2C+Quoc+Le%2C+Geoffrey+Hinton%2C+and+Jeff+Dean%2E+2017%2E+Outrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2E+Preprint)
56. Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021. Fast WordPiece tokenization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2089–2103. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xinying+Song%2C+Alex+Salcianu%2C+Yang+Song%2C+Dave+Dopson%2C+and+Denny+Zhou%2E+2021%2E+Fast+WordPiece+tokenization%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages)
57. Sagar Tamang and Dibya Jyoti Bora. 2024. Evaluating tokenizer performance of large language models across official indian languages. arXiv preprint arXiv:2411.12240. — [arXiv:2411.12240](https://arxiv.org/abs/2411.12240) [find](https://scholar.google.com/scholar?q=Sagar+Tamang+and+Dibya+Jyoti+Bora%2E+2024%2E+Evaluating+tokenizer+performance+of+large+language+models+across+official+indian+languages%2E+arXiv+preprint+arXiv%3A2411%2E12240%2E+%E2%80%94+%5BarXiv%3A2411%2E12240%5D%28https%3A%2F%2Farxiv)
58. Hugo Touvron and 1 others. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
59. Sshubam Verma, Mohammed Safi Ur Rahman Khan, Vishwajeet Kumar, Rudra Murthy, and Jaydeep Sen. 2024. Milu: A multi-task indic language understanding benchmark. arXiv preprint arXiv: 2411.02538. [find](https://scholar.google.com/scholar?q=Sshubam+Verma%2C+Mohammed+Safi+Ur+Rahman+Khan%2C+Vishwajeet+Kumar%2C+Rudra+Murthy%2C+and+Jaydeep+Sen%2E+2024%2E+Milu%3A+A+multi%2Dtask+indic+language+understanding+benchmark%2E+arXiv+preprint+arXiv%3A+2411%2E02538%2E)
60. Wenhao Xue and 1 others. 2024. [Getting the most out of your tokenizer for pre-training and domain adaptation](https://md.research.elamur.ai/free/en/papers/dagan-2024-getting-most-out-tokenizer-pre/). arXiv preprint arXiv:2402.01035.
61. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
62. Fan Zhou, Zengzhi Wang, Nikhil Ranjan, Zhoujun Cheng, Liping Tang, Guowei He, Zhengzhong Liu, and Eric P Xing. 2025. Megamath: Pushing the limits of open math corpora. arXiv preprint arXiv:2504.02807. [find](https://scholar.google.com/scholar?q=Fan+Zhou%2C+Zengzhi+Wang%2C+Nikhil+Ranjan%2C+Zhoujun+Cheng%2C+Liping+Tang%2C+Guowei+He%2C+Zhengzhong+Liu%2C+and+Eric+P+Xing%2E+2025%2E+Megamath%3A+Pushing+the+limits+of+open+math+corpora%2E+arXiv+preprint+arXiv%3A2504%2E02807%2E)
63. Vilém Zouhar, Clara Meister, Juan Luis Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. 2023. Tokenization and the noiseless channel. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5184–5207. [find](https://scholar.google.com/scholar?q=Vil%C3%A9m+Zouhar%2C+Clara+Meister%2C+Juan+Luis+Gastaldi%2C+Li+Du%2C+Mrinmaya+Sachan%2C+and+Ryan+Cotterell%2E+2023%2E+Tokenization+and+the+noiseless+channel%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+)

*Updated 2 November 2026*