---
title: "The Tokenizer Tax Across 25 European Languages: Domain"
description: "The paper measures the \"tokenizer tax,\" meaning how many more tokens per word a language needs… 22 references, cited by 3 papers here."
authors: ["Volodymyr Ovcharov"]
year: 2026
venue: "arXiv:2605.24718"
source: "https://arxiv.org/abs/2605.24718"
doi: "10.48550/arxiv.2605.24718"
arxiv: "2605.24718"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ovcharov-2026-tokenizer-tax-across-25-european/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ovcharov-2026-tokenizer-tax-across-25-european/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/) / The Tokenizer Tax Across 25 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty

# The Tokenizer Tax Across 25 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty

**Authors:** Volodymyr Ovcharov · **Year:** 2026 · **Venue:** arXiv:2605.24718 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2605.24718) · [PDF](https://arxiv.org/pdf/2605.24718) · [DOI 10.48550/arxiv.2605.24718](https://doi.org/10.48550/arxiv.2605.24718) · [arXiv 2605.24718](https://arxiv.org/abs/2605.24718) · [OpenAlex](https://openalex.org/W7162424077)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/ovcharov-2026-tokenizer-tax-across-25-european/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper measures the "tokenizer tax," meaning how many more tokens per word a language needs than English. Content is held fixed by using parallel text throughout. Ten foundation models are covered. Seven are called through AWS Bedrock. Three others (GPT-4o, Gemma 2, DeepSeek V3) are counted with their public tokenizers, a shortcut checked against API counts and found to agree within 1.8%. Fertility is computed on concatenated blocks of about 6,000 characters from SIB-200 in 25 European languages. Ukrainian legal, news and Wikipedia text serve to test whether results change by register. Further experiments cover how words are split into subwords, few-shot SIB-200 classification in four Slavic languages, and the ULP grammar benchmark. English costs about 1.23 tokens per word, while Greek and Maltese reach roughly 3.1. Ukrainian, at 2.66, costs 15–18% more than its Slavic relatives, which the author links to scarcer pre-training data. Model rankings stay the same across registers. Whether few-shot prompting helps depends on the model, not the language. The results inform API budgeting, model routing and tokenizer design, and all measurements are released publicly.

## Key points
- **Cost by language family:** Romance languages sit at 1.5–1.7 tokens/word, Germanic at 1.7–1.9, Slavic at 2.2–2.5 and Uralic/Baltic at 2.7–3.0. Ukrainian requires about 2.2× the English cost per API call. The author ties this to 2–6× less pre-training data than Polish.
- **Domain invariance:** The six locally measured models rank identically on news and Wikipedia text (Spearman ρ = 1.0). The spread between cheapest and most expensive model is 1.68× on news, 1.58× on Wikipedia and 1.61× on legal text. Wikipedia is the most expensive register and news the cheapest.
- **Subword splitting:** Qwen3 breaks "відповідальність" into 9 pieces, while Gemma 2 needs only 4. DeepSeek V3 has a 128K vocabulary yet still falls among the high-fertility models. The author concludes that the share of Cyrillic-specific merges matters more than vocabulary size.
- **Few-shot and grammar results:** Few-shot prompting lowers Llama 4 Maverick by 8.4 pp on average across the four languages and raises Nemotron by 10.5 pp. Fertility correlates only weakly with ULP grammar accuracy (ρ = −0.43, n = 7). Nemotron does well on classification but scores worst on grammar.
- **Release and cost:** The data is published as the HuggingFace dataset `overthelex/tokenizer-fertility-map`, with scripts in the GitHub repo `overthelex/SecondLayer`. All experiments together cost $4.05.

## Interesting ideas
- Comparing systems on aligned parallel items separates the effect of the language from the effect of the content, which makes any cross-condition comparison cleaner.
- Before relying on a cheap local measurement, the author checks it against the expensive ground truth on a small overlap set and reports the gap.
- A rank correlation across registers tests whether the ordering of systems is stable, which shows whether one benchmark is enough.
- Checking whether subword splits line up with morpheme boundaries gives a mechanistic explanation for an efficiency gap, beyond reporting a single number.
- The paper treats the effect of few-shot prompting as a property of each model that must be verified per model and task.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Evaluation could use paired items, such as the same fable in original and paraphrased wording, so that score differences reflect phrasing rather than story content.
- **I2** (learn): Silver labels from the witness parsers could be validated against an LLM teacher tier on a small shared sample, with the agreement rate reported before the cheaper source is trusted at scale.
- **I3** (tree): Spearman ρ between perceptron parser and gparse variants across different English treebanks would show whether a single dev set can safely guide model choice.
- **I4** (tokens): The lemmatizer and morphology analyzer could be scored on how well their segmentations match derivational boundaries such as -ness or -ation, which would show where lemmas break.
- **I5** (learn): Each LLM teacher tier should get its own few-shot versus zero-shot check per annotation task, instead of one prompt policy applied to all teachers.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works · [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) · 119 works

## Cite

Cite as: Volodymyr Ovcharov. 2026. The Tokenizer Tax Across 25 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty. arXiv:2605.24718.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ovcharov2026tokenizer,
  title = {{The Tokenizer Tax Across 25 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty}},
  author = {Volodymyr Ovcharov},
  year = {2026},
  doi = {10.48550/arxiv.2605.24718},
  eprint = {2605.24718},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2605.24718}
}
```

- [← previous The Reasoning Trap: How Enhancing LLM Reasoning Amplifies Tool Hallucination](https://md.research.elamur.ai/free/en/papers/yin-2026-reasoning-trap-how-enhancing-llm/)
- [next → Time-RA: Towards Time Series Reasoning for Anomaly Diagnosis with LLM Feedback](https://md.research.elamur.ai/free/en/papers/yang-2026-time-ra-time-series-reasoning/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/)

## References

22 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abadji, J., Ortiz Suarez, P., Romary, L., and Sagot, B. [Towards a Cleaner Document-Oriented Multilingual Crawled Corpus](https://md.research.elamur.ai/free/en/papers/abadji-2022-towards-cleaner-document-oriented-multilingual/). Proceedings of LREC, pages 4344–4355, 2022. arXiv:2201.06642.
2. Adelani, D. I., et al. [SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects](https://md.research.elamur.ai/free/en/papers/adelani-2023-sib-200-simple-inclusive-big/). Proceedings of EACL, pages 226–245, 2024. arXiv:2309.07445.
3. Ahia, O., Kumar, S., Gonen, H., Kasai, J., Mortensen, D. R., Smith, N. A., and Tsvetkov, Y. [Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models](https://md.research.elamur.ai/free/en/papers/ahia-2023-do-all-languages-cost-same/). Proceedings of EMNLP, pages 9904–9923, 2023. arXiv:2305.13707.
4. Brown, T. B., et al. [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). NeurIPS, 2020. arXiv:2005.14165.
5. Chaplynskyi, D. Ukrainian Brown Corpus and evaluation of multilingual models. Proceedings of the Second Ukrainian NLP Workshop, 2023. [find](https://scholar.google.com/scholar?q=Chaplynskyi%2C+D%2E+Ukrainian+Brown+Corpus+and+evaluation+of+multilingual+models%2E+Proceedings+of+the+Second+Ukrainian+NLP+Workshop%2C+2023%2E)
6. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. Unsupervised Cross-lingual Representation Learning at Scale. Proceedings of ACL, pages 8440–8451, 2020. arXiv:1911.02116. [find](https://scholar.google.com/scholar?q=Conneau%2C+A%2E%2C+Khandelwal%2C+K%2E%2C+Goyal%2C+N%2E%2C+Chaudhary%2C+V%2E%2C+Wenzek%2C+G%2E%2C+Guzm%C3%A1n%2C+F%2E%2C+Grave%2C+E%2E%2C+Ott%2C+M%2E%2C+Zettlemoyer%2C+L%2E%2C+and+Stoyanov%2C+V%2E+Unsupervised+Cross%2Dlingual+Representation+Learning+at+Scale%2E+Procee)
7. FrancophonIA. EU Acts in Ukrainian: Multilingual Legal Translation Corpus. HuggingFace Datasets, 2022. [find](https://scholar.google.com/scholar?q=FrancophonIA%2E+EU+Acts+in+Ukrainian%3A+Multilingual+Legal+Translation+Corpus%2E+HuggingFace+Datasets%2C+2022%2E)
8. Galeshchuk, S. ULP: Ukrainian Language Proficiency Benchmark for LLMs. HuggingFace Datasets, 2024. [find](https://scholar.google.com/scholar?q=Galeshchuk%2C+S%2E+ULP%3A+Ukrainian+Language+Proficiency+Benchmark+for+LLMs%2E+HuggingFace+Datasets%2C+2024%2E)
9. Lai, V. D., Ngo, N. T., Veyseh, A. P. B., Man, H., Dernoncourt, F., Bui, T., and Nguyen, T. H. [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://md.research.elamur.ai/free/en/papers/lai-2023-chatgpt-beyond-english-comprehensive-evaluation/). Findings of EMNLP, 2023. arXiv:2304.05613.
10. Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. [What Makes Good In-Context Examples for GPT-3?](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/) Proceedings of DeeLIO Workshop at ACL, 2022. arXiv:2101.06804.
11. Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) Proceedings of EMNLP, 2022. arXiv:2202.12837.
12. Nguyen, T., Nguyen, H., Lai, V. D., Man, H., Ngo, N. T., Dernoncourt, F., Rossi, R. A., and Nguyen, T. H. [CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages](https://md.research.elamur.ai/free/en/papers/nguyen-2023-culturax-cleaned-enormous-multilingual-dataset/). Proceedings of ACL: System Demonstrations, pages 228–237, 2024. arXiv:2309.09400.
13. Niklaus, J., Chalkidis, I., and Stürmer, M. [LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain](https://md.research.elamur.ai/free/en/papers/niklaus-2023-lextreme-multi-lingual-multi-task/). Findings of EACL, pages 1721–1734, 2024. arXiv:2301.13126.
14. Ovcharov, V. Tokenizer Fertility and Zero-Shot Performance of Foundation Models on Ukrainian Legal Text: A Comparative Study. arXiv:2605.14890, 2026. [find](https://scholar.google.com/scholar?q=Ovcharov%2C+V%2E+Tokenizer+Fertility+and+Zero%2DShot+Performance+of+Foundation+Models+on+Ukrainian+Legal+Text%3A+A+Comparative+Study%2E+arXiv%3A2605%2E14890%2C+2026%2E)
15. Petrov, A., La Malfa, E., Torr, P., and Bibi, A. [Language Model Tokenizers Introduce Unfairness Between Languages](https://md.research.elamur.ai/free/en/papers/petrov-2023-language-model-tokenizers-introduce-unfairness/). NeurIPS, 2023. arXiv:2305.15425.
16. Rust, P., Pfeiffer, J., Vulić, I., Ruder, S., and Gurevych, I. [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/). Proceedings of ACL, pages 3118–3135, 2021. arXiv:2012.15613.
17. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). NeurIPS, 2022. arXiv:2201.11903.
18. Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S. [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/). Proceedings of ICML, 2021. arXiv:2102.09690.
19. Clark, J. H., Garrette, D., Turc, I., and Wieting, J. [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://md.research.elamur.ai/free/en/papers/clark-2022-canine-pre-training-efficient-tokenization/). Transactions of the Association for Computational Linguistics, 10:73–91, 2022. arXiv:2103.06874.
20. Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., and Smith, N. A. Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. Proceedings of ACL, pages 8342–8360, 2020. arXiv:2004.10964. [find](https://scholar.google.com/scholar?q=Gururangan%2C+S%2E%2C+Marasovi%C4%87%2C+A%2E%2C+Swayamdipta%2C+S%2E%2C+Lo%2C+K%2E%2C+Beltagy%2C+I%2E%2C+Downey%2C+D%2E%2C+and+Smith%2C+N%2E+A%2E+Don%E2%80%99t+Stop+Pretraining%3A+Adapt+Language+Models+to+Domains+and+Tasks%2E+Proceedings+of+ACL%2C+pages+8342%E2%80%93836)
21. Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S., and Raffel, C. ByT5: Towards a Token-Free Future with Pre-Trained Byte-to-Byte Models. Transactions of the Association for Computational Linguistics, 10:291–306, 2022. arXiv:2105.13626. [find](https://scholar.google.com/scholar?q=Xue%2C+L%2E%2C+Barua%2C+A%2E%2C+Constant%2C+N%2E%2C+Al%2DRfou%2C+R%2E%2C+Narang%2C+S%2E%2C+and+Raffel%2C+C%2E+ByT5%3A+Towards+a+Token%2DFree+Future+with+Pre%2DTrained+Byte%2Dto%2DByte+Models%2E+Transactions+of+the+Association+for+Computational+Ling)
22. Zheng, B., Dong, L., Huang, S., Singhal, S., Che, W., Liu, T., Song, X., and Wei, F. Allocating Large Vocabulary Capacity for Cross-Lingual Language Model Pre-Training. Proceedings of EMNLP, pages 4684–4694, 2021. arXiv:2109.07306. [find](https://scholar.google.com/scholar?q=Zheng%2C+B%2E%2C+Dong%2C+L%2E%2C+Huang%2C+S%2E%2C+Singhal%2C+S%2E%2C+Che%2C+W%2E%2C+Liu%2C+T%2E%2C+Song%2C+X%2E%2C+and+Wei%2C+F%2E+Allocating+Large+Vocabulary+Capacity+for+Cross%2DLingual+Language+Model+Pre%2DTraining%2E+Proceedings+of+EMNLP%2C+pages+468)

*Updated 1 November 2026*