---
title: "FLEXITOKENS — paper by Owodunni et al. (2026)"
description: "The authors propose FLEXITOKENS, built on the hourglass-style byte-level transformer of Nawrot et… 52 references, cited by 3 papers here."
authors: ["Abraham Toluwase Owodunni", "Orevaoghene Ahia", "Sachin Kumar"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.848/"
doi: "10.18653/v1/2026.findings-acl.848"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/owodunni-2026-flexitokens-flexible-tokenization-evolving-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/owodunni-2026-flexitokens-flexible-tokenization-evolving-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/5/) / FLEXITOKENS: Flexible Tokenization for Evolving Language Models

# FLEXITOKENS: Flexible Tokenization for Evolving Language Models

**Authors:** [Abraham Toluwase Owodunni](https://md.research.elamur.ai/free/en/authors/abraham-toluwase-owodunni/), [Orevaoghene Ahia](https://md.research.elamur.ai/free/en/authors/orevaoghene-ahia/) and [Sachin Kumar](https://md.research.elamur.ai/free/en/authors/sachin-kumar/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.848/) · [PDF](https://aclanthology.org/2026.findings-acl.848.pdf) · [DOI 10.18653/v1/2026.findings-acl.848](https://doi.org/10.18653/v1/2026.findings-acl.848) · [OpenAlex](https://openalex.org/W4415143708)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/owodunni-2026-flexitokens-flexible-tokenization-evolving-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Subword tokenizers stay fixed when a language model is finetuned. As a result, text from new domains, unseen languages or unfamiliar scripts gets split into too many pieces, which hurts accuracy and inflates compute costs. Byte-level models with a learned segmentation module are meant to fix this. In practice, though, they are usually trained with a binomial regularizer that pins the compression rate to a single preset value, so they end up rigid in a different way.

The authors propose FLEXITOKENS, built on the hourglass-style byte-level transformer of Nawrot et al. (2023). It replaces the binomial term with a hinge-like loss that only penalizes the boundary rate when it falls outside an allowed interval. A single boundary predictor is shared across languages, with a separate rate setting per language.

They test it on multilingual and domain-specific benchmarks: WikiANN NER, XNLI, SIB-200, sentiment, medical abstracts, irony, code-switching and ILI. FLEXITOKENS fragments text less and outperforms BPE and gradient-based baselines by up to 10 points. This makes it relevant for adapting models to under-served languages and scripts.

## Key points
- **Loss design:** the boundary loss is zero whenever the boundary rate k/N stays between β and α, where β = α − λσ and σ is the per-language spread of tokenization rates. Outside that band, the loss pushes the rate back.
- **Architecture:** a lightweight tokenization module with a Gumbel-sigmoid boundary predictor feeds a pooled LM block. An upsampling module then predicts over a 256-byte vocabulary.
- **Unseen script:** on XNLI, Urdu (out-of-distribution) rises from 54.11 with BPE to about 57.6, a gain of roughly 3 points. Average XNLI accuracy goes from 65.42 to 66.31.
- **NER:** WikiANN covers languages including Ukrainian (uk). The average F1 for FLEXITOKENS is about 66.6, versus 61.39 for BPE.
- **Compression ablation and code:** pushing compression from 3× to 5× and 10× slightly lowers accuracy, while a larger tokenization module (3,12,3) helps. Code is released at github.com/skai-research/flexitokens.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 353 works · 200,043 citations
- [The Ohio State University](https://md.research.elamur.ai/free/en/universities/the-ohio-state-university/) — University, Columbus, United States · 82 works · 18,502 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) · 119 works

## Cite

Cite as: Abraham Toluwase Owodunni, Orevaoghene Ahia and Sachin Kumar. 2026. FLEXITOKENS: Flexible Tokenization for Evolving Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{owodunni2026flexitokens,
  title = {{FLEXITOKENS: Flexible Tokenization for Evolving Language Models}},
  author = {Abraham Toluwase Owodunni, Orevaoghene Ahia and Sachin Kumar},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.848},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.848}
}
```

- [← previous FinReporting: An Agentic Workflow for Localized Reporting of Cross-Jurisdiction Financial Disclosure](https://md.research.elamur.ai/free/en/papers/zhang-2026-finreporting-agentic-workflow-localized-reporting/)
- [next → Forest Before Trees: Latent Superposition for Efficient Visual Reasoning](https://md.research.elamur.ai/free/en/papers/wang-2026-forest-before-trees-latent-superposition/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/5/)

## References

52 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O Alabi, Yanke Mao, Haonan Gao, and Annie En-Shiun Lee. 2023. [Sib-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects](https://md.research.elamur.ai/free/en/papers/adelani-2023-sib-200-simple-inclusive-big/). arXiv preprint arXiv:2309.07445.
2. Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Valentin Hofmann, Tomasz Limisiewicz, Yulia Tsvetkov, and Noah A Smith. 2024. [Magnet: Improving the multilingual fairness of language models with adaptive gradient-based tokenization](https://md.research.elamur.ai/free/en/papers/ahia-2024-magnet-improving-multilingual-fairness-language/). Advances in Neural Information Processing Systems, 37:47790–47814.
3. Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023. [Do all languages cost the same? tokenization in the era of commercial language models](https://md.research.elamur.ai/free/en/papers/ahia-2023-do-all-languages-cost-same/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9904…
4. Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. 2018. Character-level language modeling with deeper self-attention. In AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Rami+Al%2DRfou%2C+Dokook+Choe%2C+Noah+Constant%2C+Mandy+Guo%2C+and+Llion+Jones%2E+2018%2E+Character%2Dlevel+language+modeling+with+deeper+self%2Dattention%2E+In+AAAI+Conference+on+Artificial+Intelligence%2E)
5. Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. 2022. [Adapting pre-trained language models to African languages via multilingual adaptive fine-tuning](https://md.research.elamur.ai/free/en/papers/alabi-2022-adapting-pre-trained-language-models/). In Proceedings of the 29th International Conference on Computational Linguistics, pages 4336–4349, Gyeongju, Republic…
6. Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, and 1 others. 2024. [Tokenizer choice for llm training: Negligible or crucial?](https://md.research.elamur.ai/free/en/papers/ali-2024-tokenizer-choice-llm-training-negligible/) In Findings of the Association for Computational Linguistics: NAACL 202…
7. Sukjun Hwang, Brandon Wang, and Albert Gu. 2025. Dynamic chunking for end-to-end hierarchical sequence modeling. arXiv preprint arXiv:2507.07955. clapAI. 2024. Multilingualsentiment: A multilingual sentiment classification dataset. [find](https://scholar.google.com/scholar?q=Sukjun+Hwang%2C+Brandon+Wang%2C+and+Albert+Gu%2E+2025%2E+Dynamic+chunking+for+end%2Dto%2Dend+hierarchical+sequence+modeling%2E+arXiv+preprint+arXiv%3A2507%2E07955%2E+clapAI%2E+2024%2E+Multilingualsentiment%3A+A+multilingual+se)
8. Sander Land and Max Bartolo. 2024. [Fishing for magikarp: Automatically detecting under-trained tokens in large language models](https://md.research.elamur.ai/free/en/papers/land-2024-fishing-magikarp-automatically-detecting-under/). arXiv preprint arXiv:2405.05417.
9. Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022. [Canine: Pre-training an efficient tokenization-free encoder for language representation](https://md.research.elamur.ai/free/en/papers/clark-2022-canine-pre-training-efficient-tokenization/). Transactions of the Association for Computational Linguistics, 10:73–91.
10. Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. arXiv preprint arXiv:1809.05053. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Guillaume+Lample%2C+Ruty+Rinott%2C+Adina+Williams%2C+Samuel+R+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+Xnli%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+arXiv+preprint+arXi)
11. Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, and 1 others. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. [find](https://scholar.google.com/scholar?q=Marta+R+Costa%2DJuss%C3%A0%2C+James+Cross%2C+Onur+%C3%87elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+Jean+Maillard%2C+and+1+others%2E+2022%2E+No+language+left+behind%3A+Sc)
12. Gautier Dagan, Gabriel Synnaeve, and Baptiste Rozière. 2024. [Getting the most out of your tokenizer for pre-training and domain adaptation](https://md.research.elamur.ai/free/en/papers/dagan-2024-getting-most-out-tokenizer-pre/). In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.
13. Björn Deiseroth, Manuel Brack, Patrick Schramowski, Kristian Kersting, and Samuel Weinbach. 2024. T-free: Subword tokenizer-free generative llms via sparse representations for memory-efficient embeddings. arXiv preprint arXiv:2406.19223. [find](https://scholar.google.com/scholar?q=Bj%C3%B6rn+Deiseroth%2C+Manuel+Brack%2C+Patrick+Schramowski%2C+Kristian+Kersting%2C+and+Samuel+Weinbach%2E+2024%2E+T%2Dfree%3A+Subword+tokenizer%2Dfree+generative+llms+via+sparse+representations+for+memory%2Defficient+embeddi)
14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies,…
15. William Fleshman and Benjamin Van Durme. 2023. Toucan: Token-aware character level language modeling. arXiv preprint arXiv:2311.08620. — [arXiv:2311.08620](https://arxiv.org/abs/2311.08620) [find](https://scholar.google.com/scholar?q=William+Fleshman+and+Benjamin+Van+Durme%2E+2023%2E+Toucan%3A+Token%2Daware+character+level+language+modeling%2E+arXiv+preprint+arXiv%3A2311%2E08620%2E+%E2%80%94+%5BarXiv%3A2311%2E08620%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E08620%29)
16. Jonas Geiping, Alex Stein, Manli Shu, Khalid Saifullah, Yuxin Wen, and Tom Goldstein. 2024. Coercing llms to do and reveal (almost) anything. arXiv preprint arXiv:2402.14020. — [arXiv:2402.14020](https://arxiv.org/abs/2402.14020) [find](https://scholar.google.com/scholar?q=Jonas+Geiping%2C+Alex+Stein%2C+Manli+Shu%2C+Khalid+Saifullah%2C+Yuxin+Wen%2C+and+Tom+Goldstein%2E+2024%2E+Coercing+llms+to+do+and+reveal+%28almost%29+anything%2E+arXiv+preprint+arXiv%3A2402%2E14020%2E+%E2%80%94+%5BarXiv%3A2402%2E14020%5D%28http)
17. Nathan Godey, Roman Castagné, Éric de la Clergerie, and Benoît Sagot. 2022. MANTa: Efficient gradient-based tokenization for end-to-end robust language modeling. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2859– 2870, Abu Dhabi, United Arab Emirates. Association f… [find](https://scholar.google.com/scholar?q=Nathan+Godey%2C+Roman+Castagn%C3%A9%2C+%C3%89ric+de+la+Clergerie%2C+and+Beno%C3%AEt+Sagot%2E+2022%2E+MANTa%3A+Efficient+gradient%2Dbased+tokenization+for+end%2Dto%2Dend+robust+language+modeling%2E+In+Findings+of+the+Association+for+Com)
18. Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850. [find](https://scholar.google.com/scholar?q=Alex+Graves%2E+2013%2E+Generating+sequences+with+recurrent+neural+networks%2E+arXiv+preprint+arXiv%3A1308%2E0850%2E)
19. Tomasz Limisiewicz, Terra Blevins, Hila Gonen, Orevaoghene Ahia, and Luke Zettlemoyer. 2024. [MYTE: Morphology-driven byte encoding for better and fairer multilingual language modeling](https://md.research.elamur.ai/free/en/papers/limisiewicz-2024-myte-morphology-driven-byte-encoding/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),…
20. Siyang Liu, Naihao Deng, Sahand Sabour, Yilin Jia, Minlie Huang, and Rada Mihalcea. 2023. Task-adaptive tokenization: Enhancing long-form text generation efficacy in mental health and beyond. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15264– 1528… [find](https://scholar.google.com/scholar?q=Siyang+Liu%2C+Naihao+Deng%2C+Sahand+Sabour%2C+Yilin+Jia%2C+Minlie+Huang%2C+and+Rada+Mihalcea%2E+2023%2E+Task%2Dadaptive+tokenization%3A+Enhancing+long%2Dform+text+generation+efficacy+in+mental+health+and+beyond%2E+In+Proce)
21. Jonas Lotz, Elizabeth Salesky, Phillip Rust, and Desmond Elliott. 2023. Text rendering strategies for pixel language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10155–10172, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jonas+Lotz%2C+Elizabeth+Salesky%2C+Phillip+Rust%2C+and+Desmond+Elliott%2E+2023%2E+Text+rendering+strategies+for+pixel+language+models%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+Langu)
22. Horse Main. 2025. H-net - scaling laws (byte). [find](https://scholar.google.com/scholar?q=Horse+Main%2E+2025%2E+H%2Dnet+%2D+scaling+laws+%28byte%29%2E)
23. Benjamin Minixhofer, Tyler Murray, Tomasz Limisiewicz, Anna Korhonen, Luke Zettlemoyer, Noah A Smith, Edoardo M Ponti, Luca Soldaini, and Valentin Hofmann. 2025. Bolmo: Byteifying the next generation of language models. arXiv preprint arXiv:2512.15586. [find](https://scholar.google.com/scholar?q=Benjamin+Minixhofer%2C+Tyler+Murray%2C+Tomasz+Limisiewicz%2C+Anna+Korhonen%2C+Luke+Zettlemoyer%2C+Noah+A+Smith%2C+Edoardo+M+Ponti%2C+Luca+Soldaini%2C+and+Valentin+Hofmann%2E+2025%2E+Bolmo%3A+Byteifying+the+next+generation+)
24. Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. 2022. [WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models](https://md.research.elamur.ai/free/en/papers/minixhofer-2022-wechsel-effective-initialization-subword-embeddings/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hum…
25. Benjamin Minixhofer, Edoardo Maria Ponti, and Ivan Vulić. 2024. Zero-shot tokenizer transfer. arXiv preprint arXiv:2405.07883. — [arXiv:2405.07883](https://arxiv.org/abs/2405.07883) [find](https://scholar.google.com/scholar?q=Benjamin+Minixhofer%2C+Edoardo+Maria+Ponti%2C+and+Ivan+Vulic%CC%81%2E+2024%2E+Zero%2Dshot+tokenizer+transfer%2E+arXiv+preprint+arXiv%3A2405%2E07883%2E+%E2%80%94+%5BarXiv%3A2405%2E07883%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2405%2E07883%29)
26. Md Mofijul Islam, Gustavo Aguilar, Pragaash Ponnusamy, Clint Solomon Mathialagan, Chengyuan Ma, and Chenlei Guo. 2022. A vocabulary-free multilingual neural tokenizer for end-to-end task learning. In Proceedings of the 7th Workshop on Representation Learning for NLP, pages 91–99, Dublin, Ireland. As… [find](https://scholar.google.com/scholar?q=Md+Mofijul+Islam%2C+Gustavo+Aguilar%2C+Pragaash+Ponnusamy%2C+Clint+Solomon+Mathialagan%2C+Chengyuan+Ma%2C+and+Chenlei+Guo%2E+2022%2E+A+vocabulary%2Dfree+multilingual+neural+tokenizer+for+end%2Dto%2Dend+task+learning%2E+In+)
27. Piotr Nawrot, Jan Chorowski, Adrian Lancucki, and Edoardo Maria Ponti. 2023. Efficient transformers with dynamic token pooling. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6403–6417, Toronto, Canada. Association for Comput… [find](https://scholar.google.com/scholar?q=Piotr+Nawrot%2C+Jan+Chorowski%2C+Adrian+Lancucki%2C+and+Edoardo+Maria+Ponti%2E+2023%2E+Efficient+transformers+with+dynamic+token+pooling%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computa)
28. Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A Haggag, Alfonso Amayuelas, and 1 others. 2024. [Include: Evaluating multilingual language understanding with regional knowledge](https://md.research.elamur.ai/free/en/papers/romanou-2024-include-evaluating-multilingual-language-understanding/). arXiv preprint arXiv:241…
29. Piotr Nawrot, Szymon Tworkowski, Michał Tyrolski, Lukasz Kaiser, Yuhuai Wu, Christian Szegedy, and Henryk Michalewski. 2022. Hierarchical transformers are more efficient language models. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1559–1571, Seattle, United States… [find](https://scholar.google.com/scholar?q=Piotr+Nawrot%2C+Szymon+Tworkowski%2C+Micha%C5%82+Tyrolski%2C+Lukasz+Kaiser%2C+Yuhuai+Wu%2C+Christian+Szegedy%2C+and+Henryk+Michalewski%2E+2022%2E+Hierarchical+transformers+are+more+efficient+language+models%2E+In+Findings+o)
30. Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, and Desmond Elliott. 2023. Language modelling with pixels. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Phillip+Rust%2C+Jonas+F%2E+Lotz%2C+Emanuele+Bugliarello%2C+Elizabeth+Salesky%2C+Miryam+de+Lhoneux%2C+and+Desmond+Elliott%2E+2023%2E+Language+modelling+with+pixels%2E+In+The+Eleventh+International+Conference+on+Learning)
31. Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, and Srinivasan Iyer. 2024. Byte latent transformer: Patches scale better than tokens. Preprint, arXiv:2412.09871. [find](https://scholar.google.com/scholar?q=Artidoro+Pagnoni%2C+Ram+Pasunuru%2C+Pedro+Rodriguez%2C+John+Nguyen%2C+Benjamin+Muller%2C+Margaret+Li%2C+Chunting+Zhou%2C+Lili+Yu%2C+Jason+Weston%2C+Luke+Zettlemoyer%2C+Gargi+Ghosh%2C+Mike+Lewis%2C+Ari+Holtzman%2C+and+Srinivasa)
32. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. [Cross-lingual name tagging and linking for 282 languages](https://md.research.elamur.ai/free/en/papers/pan-2017-cross-lingual-name-tagging-linking/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada. A…
33. Chanjun Park, Sugyeong Eo, Hyeonseok Moon, and Heuiseok Lim. 2021. Should we find another model?: Improving neural machine translation performance with ONE-piece tokenization method without model modification. In Proceedings of the 2021 Conference of the North American Chapter of the Association for… [find](https://scholar.google.com/scholar?q=Chanjun+Park%2C+Sugyeong+Eo%2C+Hyeonseok+Moon%2C+and+Heuiseok+Lim%2E+2021%2E+Should+we+find+another+model%3F%3A+Improving+neural+machine+translation+performance+with+ONE%2Dpiece+tokenization+method+without+model+modi)
34. Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, and 1 others. 2024a. [The fineweb datasets: Decanting the web for the finest text data at scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/). Advances in Neural Information Processing Systems, 37:30811–30849.
35. Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Martin Jaggi, Leandro von Werra, and Thomas Wolf. 2024b. Fineweb2: A sparkling update with 1000s of languages. [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Hynek+Kydl%C3%ADc%CC%8Cek%2C+Vinko+Sabolc%CC%8Cec%2C+Bettina+Messmer%2C+Negar+Foroutan%2C+Martin+Jaggi%2C+Leandro+von+Werra%2C+and+Thomas+Wolf%2E+2024b%2E+Fineweb2%3A+A+sparkling+update+with+1000s+of+languages%2E)
36. Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2023. [Language model tokenizers introduce unfairness between languages](https://md.research.elamur.ai/free/en/papers/petrov-2023-language-model-tokenizers-introduce-unfairness/). Advances in neural information processing systems, 36:36963– 36990.
37. Omid Rohanian, Shiva Taslimipoor, Richard Evans, and Ruslan Mitkov. 2018. [Wlv at semeval-2018 task 3: Dissecting tweets in search of irony](https://md.research.elamur.ai/free/en/papers/rohanian-2018-wlv-semeval-2018-task-3/). In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 553–559.
38. Vin Sachidananda, Jason Kessler, and Yi-An Lai. 2021. Efficient domain adaptation of language models via adaptive tokenization. In Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing, pages 155–165, Virtual. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Vin+Sachidananda%2C+Jason+Kessler%2C+and+Yi%2DAn+Lai%2E+2021%2E+Efficient+domain+adaptation+of+language+models+via+adaptive+tokenization%2E+In+Proceedings+of+the+Second+Workshop+on+Simple+and+Efficient+Natural+La)
39. Elizabeth Salesky, Neha Verma, Philipp Koehn, and Matt Post. 2023. Multilingual pixel representations for translation and effective cross-lingual transfer. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13845–13861, Singapore. Association for Computa… [find](https://scholar.google.com/scholar?q=Elizabeth+Salesky%2C+Neha+Verma%2C+Philipp+Koehn%2C+and+Matt+Post%2E+2023%2E+Multilingual+pixel+representations+for+translation+and+effective+cross%2Dlingual+transfer%2E+In+Proceedings+of+the+2023+Conference+on+Emp)
40. Ashutosh Sathe, Divyanshu Aggarwal, and Sunayana Sitaram. 2025. Improving consistency in LLM inference using probabilistic tokenization. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4766–4778, Albuquerque, New Mexico. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ashutosh+Sathe%2C+Divyanshu+Aggarwal%2C+and+Sunayana+Sitaram%2E+2025%2E+Improving+consistency+in+LLM+inference+using+probabilistic+tokenization%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+N)
41. Tim Schopf, Daniel Braun, and Florian Matthes. 2022. Evaluating unsupervised text classification: zero-shot and similarity-based approaches. In Proceedings of the 2022 6th International Conference on Natural Language Processing and Information Retrieval, pages 6–15. [find](https://scholar.google.com/scholar?q=Tim+Schopf%2C+Daniel+Braun%2C+and+Florian+Matthes%2E+2022%2E+Evaluating+unsupervised+text+classification%3A+zero%2Dshot+and+similarity%2Dbased+approaches%2E+In+Proceedings+of+the+2022+6th+International+Conference+on+)
42. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin…
43. Aaditya K. Singh and DJ Strouse. 2024. [Tokenization counts: the impact of tokenization on arithmetic in frontier llms](https://md.research.elamur.ai/free/en/papers/singh-2024-tokenization-counts-impact-tokenization-arithmetic/). Preprint, arXiv:2402.14903.
44. Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, and 1 others. 2024. [Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation](https://md.research.elamur.ai/free/en/papers/singh-2025-global-mmlu-understanding-addressing-cultural/)…
45. Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. 2024. [Scaling laws with vocabulary: Larger models deserve larger vocabularies](https://md.research.elamur.ai/free/en/papers/tao-2024-scaling-laws-vocabulary-larger-models/). Preprint, arXiv:2407.13623.
46. Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2022. [Charformer: Fast character transformers via gradient-based subword tokenization](https://md.research.elamur.ai/free/en/papers/tay-2021-charformer-fast-character-transformers-gradient/). In International Conference on Learning Representations.
47. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=J%C3%B6rg+Tiedemann%2E+2012%2E+Parallel+data%2C+tools+and+interfaces+in+OPUS%2E+In+Proceedings+of+the+Eighth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9912%29%2C+pages+2214%E2%80%932218%2C+Istanbul%2C+Turk)
48. Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, and Alexander M Rush. 2024. [Mambabyte: Token-free selective state space model](https://md.research.elamur.ai/free/en/papers/wang-2024-mambabyte-token-free-selective-state/). In First Conference on Language Modeling.
49. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306. [find](https://scholar.google.com/scholar?q=Linting+Xue%2C+Aditya+Barua%2C+Noah+Constant%2C+Rami+Al%2DRfou%2C+Sharan+Narang%2C+Mihir+Kale%2C+Adam+Roberts%2C+and+Colin+Raffel%2E+2022%2E+ByT5%3A+Towards+a+token%2Dfree+future+with+pre%2Dtrained+byte%2Dto%2Dbyte+models%2E+Transac)
50. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
51. Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. 2023. Megabyte: Predicting million-byte sequences with multiscale transformers. Advances in Neural Information Processing Systems, 36:78808–78823. [find](https://scholar.google.com/scholar?q=Lili+Yu%2C+D%C3%A1niel+Simig%2C+Colin+Flaherty%2C+Armen+Aghajanyan%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+2023%2E+Megabyte%3A+Predicting+million%2Dbyte+sequences+with+multiscale+transformers%2E+Advances+in+Neural+Informatio)
52. Marcos Zampieri, Preslav Nakov, Nikola Ljubešić, Jörg Tiedemann, Shervin Malmasi, and Ahmed Ali, editors. 2018. Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018). Association for Computational Linguistics, Santa Fe, New Mexico, USA. [find](https://scholar.google.com/scholar?q=Marcos+Zampieri%2C+Preslav+Nakov%2C+Nikola+Ljube%C5%A1ic%CC%81%2C+J%C3%B6rg+Tiedemann%2C+Shervin+Malmasi%2C+and+Ahmed+Ali%2C+editors%2E+2018%2E+Proceedings+of+the+Fifth+Workshop+on+NLP+for+Similar+Languages%2C+Varieties+and+Dialects+)

*Updated 31 October 2026*