---
title: "TokCollate — paper by Variš et al. (2026) — summary and ideas"
description: "The authors (ÚFAL, Charles University) release TokCollate, an MIT-licensed toolkit with two parts that can also run separately… 32 references."
authors: ["Dušan Variš", "Abishek Stephen", "Jindřich Libovický"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)"
source: "https://aclanthology.org/2026.acl-demo.41/"
doi: "10.18653/v1/2026.acl-demo.41"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/varis-2026-tokcollate-comprehensive-tool-tokenizer-evaluation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/varis-2026-tokcollate-comprehensive-tool-tokenizer-evaluation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/46/) / TokCollate: A Comprehensive Tool for Tokenizer Evaluation and Visualization across Languages

# TokCollate: A Comprehensive Tool for Tokenizer Evaluation and Visualization across Languages

**Authors:** Dušan Variš, Abishek Stephen and [Jindřich Libovický](https://md.research.elamur.ai/free/en/authors/jindrich-libovicky/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-demo.41/) · [PDF](https://aclanthology.org/2026.acl-demo.41.pdf) · [DOI 10.18653/v1/2026.acl-demo.41](https://doi.org/10.18653/v1/2026.acl-demo.41) · [OpenAlex](https://openalex.org/W7166875799)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Tokenizers trained mainly on English and other high-resource languages often split text in less-resourced languages into many more units. This raises compute and API costs for those speakers and can lower model quality, a problem known as the token premium. Existing evaluation tools cover only some metrics, rarely support structured cross-language comparison, and lack interactive exploration.

The authors (ÚFAL, Charles University) release TokCollate, an MIT-licensed toolkit with two parts that can also run separately:
- a Python CLI that scores pre-tokenized outputs according to a YAML configuration;
- a React + TypeScript web visualizer backed by an Express.js server.

The toolkit computes per-language intrinsic metrics and cross-lingual metrics over parallel text. A language knowledge base built from FLORES+ lets users filter and aggregate results by language family, script, region, speaker count and resource tier. A case study shows that writing system, tokenizer training and morphology interact in complex ways. For Tamil, for example, apparently efficient tokens may not match morpheme boundaries. Overall, the tool makes it easier to see which language communities a given tokenizer puts at a disadvantage.

## Key points
- **Monolingual metrics:** corpus token count (CTC), fertility, characters per token, and Rényi efficiency and Shannon entropy (reusing the Zouhar et al. scorer).
- **Cross-lingual metrics:** vocabulary overlap, Jensen-Shannon divergence, Eflomal alignment score, a fast cross-lingual PMI approximation, and length ratio.
- **Language knowledge base:** covers roughly 200 FLORES+ languages, identified by ISO 639-3, ISO 15,924 and Glottolog codes. Speaker counts come from WikiData, and resource levels are assigned using Joshi et al.'s (2020) classification into six tiers.
- **Tamil case study:** Llama-4 and DeepSeek-V3 average about 4–5 bytes per token with a Rényi efficiency of about 0.7, yet may still produce grapheme-level tokens.
- **Availability:** code at github.com/ufal/TokCollate, with a live demo online. Supervised metrics such as MorphScore against UD treebanks are largely out of scope.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Charles University](https://md.research.elamur.ai/free/en/universities/charles-university/) — University, Prague, Czechia · 133 works · 20,686 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works · [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) · 119 works

## Cite

Cite as: Dušan Variš, Abishek Stephen and Jindřich Libovický. 2026. TokCollate: A Comprehensive Tool for Tokenizer Evaluation and Visualization across Languages. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{varis2026tokcollate,
  title = {{TokCollate: A Comprehensive Tool for Tokenizer Evaluation and Visualization across Languages}},
  author = {Dušan Variš, Abishek Stephen and Jindřich Libovický},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-demo.41},
  url = {https://doi.org/10.18653/v1/2026.acl-demo.41}
}
```

- [← previous To Write or to Automate Linguistic Prompts, That Is the Question](https://md.research.elamur.ai/free/en/papers/torron-2026-write-or-automate-linguistic-prompts/)
- [next → ToMELP: A Theory-of-Mind Benchmark for Route-Controlled Persuasion under the Elaboration Likelihood Model](https://md.research.elamur.ai/free/en/papers/wang-2026-tomelp-theory-mind-benchmark-route/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/46/)

## References

32 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023. [Do all languages cost the same? tokenization in the era of commercial language models](https://md.research.elamur.ai/free/en/papers/ahia-2023-do-all-languages-cost-same/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9904…
2. Part I, volume 13451 of Lecture Notes in Computer Science, pages 545–554. Springer. 01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang,… — [Yi: Open Foundation Models by 01.AI](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/)
3. Matthias Gallé. 2019. Investigating the effectiveness of BPE: The power of shorter sequences. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1375–1381, Hong Kong… [find](https://scholar.google.com/scholar?q=Matthias+Gall%C3%A9%2E+2019%2E+Investigating+the+effectiveness+of+BPE%3A+The+power+of+shorter+sequences%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+Inte)
4. Catherine Arnett, Tyler A. Chang, and Benjamin K. Bergen. 2024. A bit of a problem: Measurement disparities in dataset sizes across languages. In Proceedings of the Annual Meeting of the Special Interest Group on Under-Resourced Languages. [find](https://scholar.google.com/scholar?q=Catherine+Arnett%2C+Tyler+A%2E+Chang%2C+and+Benjamin+K%2E+Bergen%2E+2024%2E+A+bit+of+a+problem%3A+Measurement+disparities+in+dataset+sizes+across+languages%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Special+Inter)
5. Catherine Arnett, Marisa Hudspeth, and Brendan O’Connor. 2025. Evaluating Morphological Alignment of Tokenizers in 70 Languages. In Proceedings of the ICML 2025 Tokenization Workshop (TokShop). [find](https://scholar.google.com/scholar?q=Catherine+Arnett%2C+Marisa+Hudspeth%2C+and+Brendan+O%E2%80%99Connor%2E+2025%2E+Evaluating+Morphological+Alignment+of+Tokenizers+in+70+Languages%2E+In+Proceedings+of+the+ICML+2025+Tokenization+Workshop+%28TokShop%29%2E)
6. Khuyagbaatar Batsuren, Ekaterina Vylomova, Verna Dankers, Tsetsuukhei Delgerbaatar, Omri Uzan, Yuval Pinter, and Gábor Bella. 2024. Evaluating subword tokenization: Alien subword composition and oov generalization challenge. Preprint, arXiv:2404.13292. — [arXiv:2404.13292](https://arxiv.org/abs/2404.13292) [find](https://scholar.google.com/scholar?q=Khuyagbaatar+Batsuren%2C+Ekaterina+Vylomova%2C+Verna+Dankers%2C+Tsetsuukhei+Delgerbaatar%2C+Omri+Uzan%2C+Yuval+Pinter%2C+and+G%C3%A1bor+Bella%2E+2024%2E+Evaluating+subword+tokenization%3A+Alien+subword+composition+and+oov+g)
7. Thomas Bauwens. 2025. Tktkt: the tokeniser toolkit. [find](https://scholar.google.com/scholar?q=Thomas+Bauwens%2E+2025%2E+Tktkt%3A+the+tokeniser+toolkit%2E)
8. Lisa Beinborn and Yuval Pinter. 2023. Analyzing cognitive plausibility of subword tokenization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4478–4486, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Lisa+Beinborn+and+Yuval+Pinter%2E+2023%2E+Analyzing+cognitive+plausibility+of+subword+tokenization%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+4478%E2%80%9344)
9. Iaroslav Chelombitko, Egor Safronov, and Aleksey Komissarov. 2024. Qtok: A comprehensive framework for evaluating multilingual tokenizer quality in large language models. CoRR, abs/2410.12989. — [arXiv:2410.12989](https://arxiv.org/abs/2410.12989) [find](https://scholar.google.com/scholar?q=Iaroslav+Chelombitko%2C+Egor+Safronov%2C+and+Aleksey+Komissarov%2E+2024%2E+Qtok%3A+A+comprehensive+framework+for+evaluating+multilingual+tokenizer+quality+in+large+language+models%2E+CoRR%2C+abs%2F2410%2E12989%2E+%E2%80%94+%5BarXi)
10. Marco Cognetta, Vilém Zouhar, Sangwhan Moon, and Naoaki Okazaki. 2024. Two counterexamples to tokenization and the noiseless channel. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16897–16906, Torin… [find](https://scholar.google.com/scholar?q=Marco+Cognetta%2C+Vil%C3%A9m+Zouhar%2C+Sangwhan+Moon%2C+and+Naoaki+Okazaki%2E+2024%2E+Two+counterexamples+to+tokenization+and+the+noiseless+channel%2E+In+Proceedings+of+the+2024+Joint+International+Conference+on+Compu)
11. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. Deepseek-v3 technical report. Preprint, arXiv:2… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+Dejian+Yang%2C+Deli+Chen%2C+Dongjie+Ji%2C+Erhan)
12. Miguel Domingo, Mercedes García-Martínez, Alexandre Helle, Francisco Casacuberta, and Manuel Herranz. 2019. How much does tokenization affect neural machine translation? In Computational Linguistics and Intelligent Text Processing - 20th International Conference, CICLing 2019, La Rochelle, France, A… [find](https://scholar.google.com/scholar?q=Miguel+Domingo%2C+Mercedes+Garc%C3%ADa%2DMart%C3%ADnez%2C+Alexandre+Helle%2C+Francisco+Casacuberta%2C+and+Manuel+Herranz%2E+2019%2E+How+much+does+tokenization+affect+neural+machine+translation%3F+In+Computational+Linguistics+a)
13. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
14. Katharina Hämmerl, Tomasz Limisiewicz, Jindřich Libovický, and Alexander Fraser. 2025. Beyond literal token overlap: Token alignability for multilinguality. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language… [find](https://scholar.google.com/scholar?q=Katharina+H%C3%A4mmerl%2C+Tomasz+Limisiewicz%2C+Jindr%CC%8Cich+Libovick%C3%BD%2C+and+Alexander+Fraser%2E+2025%2E+Beyond+literal+token+overlap%3A+Token+alignability+for+multilinguality%2E+In+Proceedings+of+the+2025+Conference+of+t)
15. Vojtěch John, Zdeněk Žabokrtský, and Benjamin Reeves. 2026. Modeling word-internal structures: Morphological segmentation across 58 languages. In Proceedings of the Workshop on Structured Linguistic Data and Evaluation (SLiDE) at LREC-COLING 2026, pages 180–190, Palma, Mallorca, Spain. [find](https://scholar.google.com/scholar?q=Vojte%CC%8Cch+John%2C+Zdene%CC%8Ck+%C5%BDabokrtsk%C3%BD%2C+and+Benjamin+Reeves%2E+2026%2E+Modeling+word%2Dinternal+structures%3A+Morphological+segmentation+across+58+languages%2E+In+Proceedings+of+the+Workshop+on+Structured+Linguistic)
16. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. [The state and fate of linguistic diversity and inclusion in the NLP world](https://md.research.elamur.ai/free/en/papers/joshi-2020-state-fate-linguistic-diversity-inclusion/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Comp…
17. Taku Kudo and John Richardson. 2018. [SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ…
18. Jindřich Libovický and Jindřich Helcl. 2024. Lexically grounded subword segmentation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7403–7420, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jindr%CC%8Cich+Libovick%C3%BD+and+Jindr%CC%8Cich+Helcl%2E+2024%2E+Lexically+grounded+subword+segmentation%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+7403%E2%80%937420%2C+Miam)
19. Tomasz Limisiewicz, Jiří Balhar, and David Mareček. 2023. Tokenization impacts multilingual language modeling: Assessing vocabulary allocation and overlap across languages. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5661–5681, Toronto, Canada. Association for Com… [find](https://scholar.google.com/scholar?q=Tomasz+Limisiewicz%2C+Jir%CC%8C%C3%AD+Balhar%2C+and+David+Marec%CC%8Cek%2E+2023%2E+Tokenization+impacts+multilingual+language+modeling%3A+Assessing+vocabulary+allocation+and+overlap+across+languages%2E+In+Findings+of+the+Associ)
20. Clara Meister. 2025. Tokeval: A tokenizer analysis suite. [find](https://scholar.google.com/scholar?q=Clara+Meister%2E+2025%2E+Tokeval%3A+A+tokenizer+analysis+suite%2E)
21. Meta AI. 2025. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation. [https://ai.meta.com/blog/](https://ai.meta.com/blog/) llama-4-multimodal-intelligence/. Blog post. [↗](https://ai.meta.com/blog/)
22. Mistral AI. 2025. Mistral medium 3. [https://mistral.ai/news/mistral-medium-3](https://mistral.ai/news/mistral-medium-3). Blog post. [↗](https://mistral.ai/news/mistral-medium-3)
23. Sebastian Nordhoff and Harald Hammarström. 2011. Glottolog/langdoc: Defining dialects, languages, and language families as collections of resources. In First International Workshop on Linked Science 2011-In conjunction with the International Semantic Web Conference (ISWC 2011). [find](https://scholar.google.com/scholar?q=Sebastian+Nordhoff+and+Harald+Hammarstr%C3%B6m%2E+2011%2E+Glottolog%2Flangdoc%3A+Defining+dialects%2C+languages%2C+and+language+families+as+collections+of+resources%2E+In+First+International+Workshop+on+Linked+Science+2)
24. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. [The fineweb datasets: Decanting the web for the finest text data at scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/). In The Thirty-eight Conference on Neural Information Processing Systems Datasets a…
25. Juan Antonio Perez-Ortiz, Felipe Sánchez-Martínez, Víctor M. Sánchez-Cartagena, Miquel Esplà-Gomis, Aaron Galiano Jimenez, Antoni Oliver, Claudi Aventín-Boya, Alejandro Pardos, Cristina Valdés, Jusèp Loís Sans Socasau, and Juan Pablo Martínez. 2024. Expanding the FLORES+ multilingual benchmark with… [find](https://scholar.google.com/scholar?q=Juan+Antonio+Perez%2DOrtiz%2C+Felipe+S%C3%A1nchez%2DMart%C3%ADnez%2C+V%C3%ADctor+M%2E+S%C3%A1nchez%2DCartagena%2C+Miquel+Espl%C3%A0%2DGomis%2C+Aaron+Galiano+Jimenez%2C+Antoni+Oliver%2C+Claudi+Avent%C3%ADn%2DBoya%2C+Alejandro+Pardos%2C+Cristina+Vald%C3%A9s%2C+Jus%C3%A8p+)
26. Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi. 2023. [Language model tokenizers introduce unfairness between languages](https://md.research.elamur.ai/free/en/papers/petrov-2023-language-model-tokenizers-introduce-unfairness/). In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA…
27. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. [How good is your tokenizer? on the monolingual performance of multilingual language models](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join…
28. Craig W Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner. 2024. [Tokenization is more than compression](https://md.research.elamur.ai/free/en/papers/schmidt-2024-tokenization-is-more-than-compression/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 678–702, Miami, Florida, USA. Association for Compu…
29. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/team-2025-4-glm-4-5-agentic/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin…
30. Omri Uzan, Craig W. Schmidt, Chris Tanner, and Yuval Pinter. 2024. Greed is all you need: An evaluation of tokenizer inference methods. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 813–822, Bangkok, Thailand. Association f… [find](https://scholar.google.com/scholar?q=Omri+Uzan%2C+Craig+W%2E+Schmidt%2C+Chris+Tanner%2C+and+Yuval+Pinter%2E+2024%2E+Greed+is+all+you+need%3A+An+evaluation+of+tokenizer+inference+methods%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Association+for)
31. Denny Vrandecic and Markus Krötzsch. 2014. [Wikidata: a free collaborative knowledgebase](https://md.research.elamur.ai/free/en/papers/vrandecic-2014-wikidata-free-collaborative-knowledgebase/). Commun. ACM, 57(10):78–85. xAI. 2024. Grok-2 beta release. [https://x.ai/news/](https://x.ai/news/) grok-2. Blog post. [↗](https://x.ai/news/)
32. Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. 2023. Tokenization and the noiseless channel. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5184–5207, Toronto, Canada. Association for… [find](https://scholar.google.com/scholar?q=Vil%C3%A9m+Zouhar%2C+Clara+Meister%2C+Juan+Gastaldi%2C+Li+Du%2C+Mrinmaya+Sachan%2C+and+Ryan+Cotterell%2E+2023%2E+Tokenization+and+the+noiseless+channel%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+C)

*Updated 1 November 2026*