---
title: "Tokenization Is More Than Compression — Schmidt 2024"
description: "It builds its vocabulary top-down, pruning tokens to reduce corpus token count (CTC). 53 references, cited by 42 papers here."
authors: ["Craig W Schmidt", "Varshini Reddy", "Haoran Zhang", "Alec Alameddine", "Omri Uzan", "Yuval Pinter", "Chris Tanner"]
year: 2024
venue: "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2024.emnlp-main.40/"
doi: "10.18653/v1/2024.emnlp-main.40"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/schmidt-2024-tokenization-is-more-than-compression/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/schmidt-2024-tokenization-is-more-than-compression/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Tokenization Is More Than Compression

# Tokenization Is More Than Compression

**Authors:** Craig W Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, [Yuval Pinter](https://md.research.elamur.ai/free/en/authors/yuval-pinter/) and [Chris Tanner](https://md.research.elamur.ai/free/en/authors/chris-tanner/) · **Year:** 2024 · **Venue:** Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.emnlp-main.40/) · [PDF](https://aclanthology.org/2024.emnlp-main.40.pdf) · [DOI 10.18653/v1/2024.emnlp-main.40](https://doi.org/10.18653/v1/2024.emnlp-main.40) · [OpenAlex](https://openalex.org/W4404783106)

**Citations:** 24 citations (OpenAlex, 2026-10-14) · [42](https://md.research.elamur.ai/free/en/papers/schmidt-2024-tokenization-is-more-than-compression/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
A common explanation for why Byte-Pair Encoding (BPE) works well is that it compresses text into fewer tokens. The authors test this idea directly with PathPiece, a new lossless byte-level tokenizer. Given a vocabulary, PathPiece finds the segmentation with the smallest possible number of tokens by computing a shortest path over a DAG of bytes. It builds its vocabulary top-down, pruning tokens to reduce corpus token count (CTC).

The authors treat tokenization as three stages: pre-tokenization, vocabulary construction and segmentation. They vary choices at each stage for PathPiece, BPE, WordPiece, Unigram and SaGe, training 64 English language models on The Pile.

Across these settings, lower CTC shows no clear link to better downstream accuracy, so compression alone does not explain what makes a tokenizer effective. Two other factors matter more:
- **Pre-tokenization**, such as space handling and splitting digits.
- **Initialization from a BPE vocabulary**, for top-down vocabulary methods.

These results give practical guidance for tokenizer design. All vocabularies and model weights are publicly released.

## Key points
- **Models trained:** 54 at 350M parameters, 6 at 1.3B and 4 at 2.4B. The larger models used a single vocabulary size of 40,960 and were trained on 200B tokens.
- **PathPiece segmentation:** runs in O(nL) time with a maximum token width of L = 16 bytes. It requires all 256 single-byte tokens to be in the vocabulary.
- **Main comparisons:** 18 tokenizer configurations were compared on average accuracy over 10 downstream tasks.
  - Vocabulary size had little effect within the tested range.
  - The five best tokenizers did not differ significantly from one another.
  - Tokenizer rankings shifted with model size.
- **Initial vocabularies:** BPE-based starting vocabularies outperformed n-gram ones for PathPiece and SaGe, and the n-gram start was significantly worse.
- **Cost and limitations:** training used about 138,432 A100 GPU hours, and all experiments were on English text only.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/)

## Institutions

- [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) — University, Cambridge, United States · 125 works · 60,031 citations
- [Harvard University](https://md.research.elamur.ai/free/en/universities/harvard-university/) — University, Cambridge, United States · 64 works · 35,670 citations
- [Ben-Gurion University of the Negev](https://md.research.elamur.ai/free/en/universities/ben-gurion-university-of-the-negev/) — University, Beersheba, Israel · 34 works · 2,748 citations

## Cite

Cite as: Craig W Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter and Chris Tanner. 2024. Tokenization Is More Than Compression. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{schmidt2024tokenization,
  title = {{Tokenization Is More Than Compression}},
  author = {Craig W Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter and Chris Tanner},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.40},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.40}
}
```

- [← previous MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models](https://md.research.elamur.ai/free/en/papers/kwan-2024-mt-eval-multi-turn-capabilities/)
- [next → Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs](https://md.research.elamur.ai/free/en/papers/singh-2024-tokenization-counts-impact-tokenization-arithmetic/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

53 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Schulze Buschhoff, Charvi Jain, Alexander Arno Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Samuel Weinbach, Ra… [our card](https://md.research.elamur.ai/free/en/papers/ali-2024-tokenizer-choice-llm-training-negligible/)
2. Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. [our card](https://md.research.elamur.ai/free/en/papers/amini-2019-mathqa-interpretable-math-word-problem/)
3. Thomas Bauwens and Pieter Delobelle. 2024. BPE-knockout: Pruning pre-existing BPE tokenisers with backwards-compatible morphological semi-supervision. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vo… [find](https://scholar.google.com/scholar?q=Thomas+Bauwens+and+Pieter+Delobelle%2E+2024%2E+BPE%2Dknockout%3A+Pruning+pre%2Dexisting+BPE+tokenisers+with+backwards%2Dcompatible+morphological+semi%2Dsupervision%2E+In+Proceedings+of+the+2024+Conference+of+the+Nort)
4. Stella Biderman, Kieran Bicheno, and Leo Gao. 2022. Datasheet for the pile. CoRR, abs/2201.07311. — [arXiv:2201.07311](https://arxiv.org/abs/2201.07311) [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Kieran+Bicheno%2C+and+Leo+Gao%2E+2022%2E+Datasheet+for+the+pile%2E+CoRR%2C+abs%2F2201%2E07311%2E+%E2%80%94+%5BarXiv%3A2201%2E07311%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2201%2E07311%29)
5. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. Piqa: Reasoning about physical commonsense in natural language. [our card](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/)
6. Kaj Bostrom and Greg Durrett. 2020. Byte pair encoding is suboptimal for language model pretraining. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4617–4624, Online. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/bostrom-2020-byte-pair-encoding-is-suboptimal/)
7. Gerlof Bouma. 2009. Normalized (pointwise) mutual information in collocation extraction. Proceedings of GSCL, 30:31–40. [find](https://scholar.google.com/scholar?q=Gerlof+Bouma%2E+2009%2E+Normalized+%28pointwise%29+mutual+information+in+collocation+extraction%2E+Proceedings+of+GSCL%2C+30%3A31%E2%80%9340%2E)
8. Ana Brassard, Benjamin Heinzerling, Pride Kavumba, and Kentaro Inui. 2022. Copa-sse: Semi-structured explanations for commonsense reasoning. [find](https://scholar.google.com/scholar?q=Ana+Brassard%2C+Benjamin+Heinzerling%2C+Pride+Kavumba%2C+and+Kentaro+Inui%2E+2022%2E+Copa%2Dsse%3A+Semi%2Dstructured+explanations+for+commonsense+reasoning%2E)
9. Pavel Chizhov, Catherine Arnett, Elizaveta Korotkova, and Ivan P. Yamshchikov. 2024. Bpe gets picky: Efficient vocabulary refinement during tokenizer training. [find](https://scholar.google.com/scholar?q=Pavel+Chizhov%2C+Catherine+Arnett%2C+Elizaveta+Korotkova%2C+and+Ivan+P%2E+Yamshchikov%2E+2024%2E+Bpe+gets+picky%3A+Efficient+vocabulary+refinement+during+tokenizer+training%2E)
10. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457. [find](https://scholar.google.com/scholar?q=Peter+Clark%2C+Isaac+Cowhey%2C+Oren+Etzioni%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+Carissa+Schoenick%2C+and+Oyvind+Tafjord%2E+2018%2E+Think+you+have+solved+question+answering%3F+try+arc%2C+the+ai2+reasoning+challenge%2E+ArX)
11. Marco Cognetta, Vilém Zouhar, Sangwhan Moon, and Naoaki Okazaki. 2024. Two counterexamples to tokenization and the noiseless channel. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16897–16906, Torin… [find](https://scholar.google.com/scholar?q=Marco+Cognetta%2C+Vil%C3%A9m+Zouhar%2C+Sangwhan+Moon%2C+and+Naoaki+Okazaki%2E+2024%2E+Two+counterexamples+to+tokenization+and+the+noiseless+channel%2E+In+Proceedings+of+the+2024+Joint+International+Conference+on+Compu)
12. Pavlos S. Efraimidis. 2010. Weighted random sampling over data streams. CoRR, abs/1012.0256. — [arXiv:1012.0256](https://arxiv.org/abs/1012.0256) [find](https://scholar.google.com/scholar?q=Pavlos+S%2E+Efraimidis%2E+2010%2E+Weighted+random+sampling+over+data+streams%2E+CoRR%2C+abs%2F1012%2E0256%2E+%E2%80%94+%5BarXiv%3A1012%2E0256%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1012%2E0256%29)
13. Philip Gage. 1994. A new algorithm for data compression. C Users J., 12(2):23–38. [find](https://scholar.google.com/scholar?q=Philip+Gage%2E+1994%2E+A+new+algorithm+for+data+compression%2E+C+Users+J%2E%2C+12%282%29%3A23%E2%80%9338%2E)
14. Matthias Gallé. 2019. Investigating the effectiveness of BPE: The power of shorter sequences. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1375–1381, Hong Kong… [find](https://scholar.google.com/scholar?q=Matthias+Gall%C3%A9%2E+2019%2E+Investigating+the+effectiveness+of+BPE%3A+The+power+of+shorter+sequences%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+Inte)
15. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+Shawn+Presser%2C+and+Connor+Leahy%2E+2020%2E+The+pile%3A+An+800gb+datas)
16. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Ani… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Alain+Le+Noac%E2%80%99h%2C+Haonan+Li%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Chris+Oci)
17. Omer Goldman, Avi Caciularu, Matan Eyal, Kris Cao, Idan Szpektor, and Reut Tsarfaty. 2024. Unpacking tokenization: Evaluating text compression and its correlation with model performance. [find](https://scholar.google.com/scholar?q=Omer+Goldman%2C+Avi+Caciularu%2C+Matan+Eyal%2C+Kris+Cao%2C+Idan+Szpektor%2C+and+Reut+Tsarfaty%2E+2024%2E+Unpacking+tokenization%3A+Evaluating+text+compression+and+its+correlation+with+model+performance%2E)
18. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
19. Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. 2021. Superbizarre is not superb: Derivational morphology improves BERT’s interpretation of complex words. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferenc… [our card](https://md.research.elamur.ai/free/en/papers/hofmann-2021-superbizarre-derivational/)
20. Valentin Hofmann, Hinrich Schuetze, and Janet Pierrehumbert. 2022. An embarrassingly simple method to mitigate undesirable properties of pretrained language model tokenizers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 38… [find](https://scholar.google.com/scholar?q=Valentin+Hofmann%2C+Hinrich+Schuetze%2C+and+Janet+Pierrehumbert%2E+2022%2E+An+embarrassingly+simple+method+to+mitigate+undesirable+properties+of+pretrained+language+model+tokenizers%2E+In+Proceedings+of+the+60t)
21. Cassandra L Jacobs and Yuval Pinter. 2022. Lost in space marking. arXiv preprint arXiv:2208.01561. — [arXiv:2208.01561](https://arxiv.org/abs/2208.01561) [find](https://scholar.google.com/scholar?q=Cassandra+L+Jacobs+and+Yuval+Pinter%2E+2022%2E+Lost+in+space+marking%2E+arXiv+preprint+arXiv%3A2208%2E01561%2E+%E2%80%94+%5BarXiv%3A2208%2E01561%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2208%2E01561%29)
22. Jean Kaddour. 2023. The minipile challenge for data-efficient language models. [find](https://scholar.google.com/scholar?q=Jean+Kaddour%2E+2023%2E+The+minipile+challenge+for+data%2Defficient+language+models%2E)
23. Edward Gow-Smith, Dylan Phelps, Harish Tayyar Madabushi, Carolina Scarton, and Aline Villavicencio. 2024. Word boundary information isn’t useful for encoder language models. In Proceedings of the 9th Workshop on Representation Learning for NLP (RepL4NLP-2024), pages 118–135, Bangkok, Thailand. Assoc… [find](https://scholar.google.com/scholar?q=Edward+Gow%2DSmith%2C+Dylan+Phelps%2C+Harish+Tayyar+Madabushi%2C+Carolina+Scarton%2C+and+Aline+Villavicencio%2E+2024%2E+Word+boundary+information+isn%E2%80%99t+useful+for+encoder+language+models%2E+In+Proceedings+of+the+9th+)
24. Edward Gow-Smith, Harish Tayyar Madabushi, Carolina Scarton, and Aline Villavicencio. 2022. Improving tokenisation by alternative treatment of spaces. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11430–11443, Abu Dhabi, United Arab Emirates. Associ… [find](https://scholar.google.com/scholar?q=Edward+Gow%2DSmith%2C+Harish+Tayyar+Madabushi%2C+Carolina+Scarton%2C+and+Aline+Villavicencio%2E+2022%2E+Improving+tokenisation+by+alternative+treatment+of+spaces%2E+In+Proceedings+of+the+2022+Conference+on+Empirica)
25. Gregory Grefenstette. 1999. Tokenization, pages 117– 133. Springer Netherlands, Dordrecht. [find](https://scholar.google.com/scholar?q=Gregory+Grefenstette%2E+1999%2E+Tokenization%2C+pages+117%E2%80%93+133%2E+Springer+Netherlands%2C+Dordrecht%2E)
26. Ximena Gutierrez-Vasques, Christian Bentz, Olga Sozinova, and Tanja Samardzic. 2021. From characters to words: the turning point of BPE merges. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3454–3468, Online. Associ… [find](https://scholar.google.com/scholar?q=Ximena+Gutierrez%2DVasques%2C+Christian+Bentz%2C+Olga+Sozinova%2C+and+Tanja+Samardzic%2E+2021%2E+From+characters+to+words%3A+the+turning+point+of+BPE+merges%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Ch)
27. Xuanli He, Gholamreza Haffari, and Mohammad Norouzi. 2020. Dynamic programming encoding for subword segmentation in neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3042–3051, Online. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Xuanli+He%2C+Gholamreza+Haffari%2C+and+Mohammad+Norouzi%2E+2020%2E+Dynamic+programming+encoding+for+subword+segmentation+in+neural+machine+translation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Associa)
28. Stav Klein and Reut Tsarfaty. 2020. Getting the \#\#life out of living: How adequate are word-pieces for modelling complex morphology? In Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 204–209, Online. Association for Computational… [find](https://scholar.google.com/scholar?q=Stav+Klein+and+Reut+Tsarfaty%2E+2020%2E+Getting+the+%5C%23%5C%23life+out+of+living%3A+How+adequate+are+word%2Dpieces+for+modelling+complex+morphology%3F+In+Proceedings+of+the+17th+SIGMORPHON+Workshop+on+Computational+R)
29. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational… [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-subword-regularization-improving-neural-network/)
30. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ… [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
31. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. [our card](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/)
32. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In 13th International Conference on the Principles of Knowledge Representation and Reasoning, KR 2012, Proceedings of the International Conference on Knowledge Representation and Reasoning, pages 552–561. I… [find](https://scholar.google.com/scholar?q=Hector+J%2E+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+winograd+schema+challenge%2E+In+13th+International+Conference+on+the+Principles+of+Knowledge+Representation+and+Reasoning%2C+KR+2012%2C+Pro)
33. Haoran Lian, Yizhe Xiong, Jianwei Niu, Shasha Mo, Zhenpeng Su, Zijia Lin, Peng Liu, Hui Chen, and Guiguang Ding. 2024. Scaffold-bpe: Enhancing byte pair encoding with simple and effective scaffold token removal. [find](https://scholar.google.com/scholar?q=Haoran+Lian%2C+Yizhe+Xiong%2C+Jianwei+Niu%2C+Shasha+Mo%2C+Zhenpeng+Su%2C+Zijia+Lin%2C+Peng+Liu%2C+Hui+Chen%2C+and+Guiguang+Ding%2E+2024%2E+Scaffold%2Dbpe%3A+Enhancing+byte+pair+encoding+with+simple+and+effective+scaffold+tok)
34. Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa. 2023. XLM-V: Overcoming the vocabulary bottleneck in multilingual masked language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p… [find](https://scholar.google.com/scholar?q=Davis+Liang%2C+Hila+Gonen%2C+Yuning+Mao%2C+Rui+Hou%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Luke+Zettlemoyer%2C+and+Madian+Khabsa%2E+2023%2E+XLM%2DV%3A+Overcoming+the+vocabulary+bottleneck+in+multilingual+masked+language+)
35. Tomasz Limisiewicz, Jiří Balhar, and David Mareček. 2023. Tokenization impacts multilingual language modeling: Assessing vocabulary allocation and overlap across languages. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5661–5681, Toronto, Canada. Association for Com… [find](https://scholar.google.com/scholar?q=Tomasz+Limisiewicz%2C+Jir%CC%8C%C3%AD+Balhar%2C+and+David+Marec%CC%8Cek%2E+2023%2E+Tokenization+impacts+multilingual+language+modeling%3A+Assessing+vocabulary+allocation+and+overlap+across+languages%2E+In+Findings+of+the+Associ)
36. Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y. Lee, Benoît Sagot, and Samson Tan. 2021. Between words and characters: A brief history of open-vocabulary modeling and tokenization in nlp. [our card](https://md.research.elamur.ai/free/en/papers/mielke-2021-between-words-characters-brief-history/)
37. Tomas Mikolov, Ilya Sutskever, Anoop Deoras, Hai Son Le, Stefan Kombrink, and Jan Honza Černocký. 2011. Subword language modeling with neural networks. Preprint available at: [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:46542477. [↗](https://api.semanticscholar.org/)
38. Anselmo Peñas, Eduard Hovy, Pamela Forner, Álvaro Rodrigo, Richard Sutcliffe, and Roser Morante. 2013. Qa4mre 2011-2013: Overview of question answering for machine reading evaluation. In CLEF 2013, LNCS 8138, pages 303–320. [find](https://scholar.google.com/scholar?q=Anselmo+Pe%C3%B1as%2C+Eduard+Hovy%2C+Pamela+Forner%2C+%C3%81lvaro+Rodrigo%2C+Richard+Sutcliffe%2C+and+Roser+Morante%2E+2013%2E+Qa4mre+2011%2D2013%3A+Overview+of+question+answering+for+machine+reading+evaluation%2E+In+CLEF+2013%2C+LN)
39. Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2020. BPE-dropout: Simple and effective subword regularization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1882–1892, Online. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/provilkov-2020-bpe-dropout-simple-effective-subword/)
40. Jonne Saleva and Constantine Lignos. 2023. What changes when you randomly choose BPE merge operations? not much. In Proceedings of the Fourth Workshop on Insights from Negative Results in NLP, pages 59–66, Dubrovnik, Croatia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jonne+Saleva+and+Constantine+Lignos%2E+2023%2E+What+changes+when+you+randomly+choose+BPE+merge+operations%3F+not+much%2E+In+Proceedings+of+the+Fourth+Workshop+on+Insights+from+Negative+Results+in+NLP%2C+pages+5)
41. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152. [find](https://scholar.google.com/scholar?q=Mike+Schuster+and+Kaisuke+Nakajima%2E+2012%2E+Japanese+and+korean+voice+search%2E+In+2012+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2C+pages+5149%E2%80%935152%2E)
42. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin… [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/)
43. Jasdeep Singh, Bryan McCann, Richard Socher, and Caiming Xiong. 2019. BERT is not an interlingua and the bias of tokenization. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 47–55, Hong Kong, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jasdeep+Singh%2C+Bryan+McCann%2C+Richard+Socher%2C+and+Caiming+Xiong%2E+2019%2E+BERT+is+not+an+interlingua+and+the+bias+of+tokenization%2E+In+Proceedings+of+the+2nd+Workshop+on+Deep+Learning+Approaches+for+Low%2DRe)
44. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. [our card](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)
45. Omri Uzan, Craig W. Schmidt, Chris Tanner, and Yuval Pinter. 2024. Greed is all you need: An evaluation of tokenizer inference methods. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 813–822, Bangkok, Thailand. Association f… [find](https://scholar.google.com/scholar?q=Omri+Uzan%2C+Craig+W%2E+Schmidt%2C+Chris+Tanner%2C+and+Yuval+Pinter%2E+2024%2E+Greed+is+all+you+need%3A+An+evaluation+of+tokenizer+inference+methods%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Association+for)
46. A. Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory, 13(2):260–269. [find](https://scholar.google.com/scholar?q=A%2E+Viterbi%2E+1967%2E+Error+bounds+for+convolutional+codes+and+an+asymptotically+optimum+decoding+algorithm%2E+IEEE+Transactions+on+Information+Theory%2C+13%282%29%3A260%E2%80%93269%2E)
47. Jeffrey S. Vitter. 1985. Random sampling with a reservoir. ACM Transactions on Mathematical Software, 11(1):37–57. [find](https://scholar.google.com/scholar?q=Jeffrey+S%2E+Vitter%2E+1985%2E+Random+sampling+with+a+reservoir%2E+ACM+Transactions+on+Mathematical+Software%2C+11%281%29%3A37%E2%80%9357%2E)
48. Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. ArXiv, abs/1707.06209. [our card](https://md.research.elamur.ai/free/en/papers/welbl-2017-crowdsourcing-multiple-choice-science-questions/)
49. F Wilcoxon. 1945. Individual comparisons by ranking methods. biom. bull., 1, 80–83. [find](https://scholar.google.com/scholar?q=F+Wilcoxon%2E+1945%2E+Individual+comparisons+by+ranking+methods%2E+biom%2E+bull%2E%2C+1%2C+80%E2%80%9383%2E)
50. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V%2E+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+Jeff+Klingner%2C+Apurva+Shah%2C+Melvin+Johnson%2C+Xiaobing+Liu%2C+%C5%81uk)
51. Shaked Yehezkel and Yuval Pinter. 2023. Incorporating context into subword vocabularies. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 623–635, Dubrovnik, Croatia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shaked+Yehezkel+and+Yuval+Pinter%2E+2023%2E+Incorporating+context+into+subword+vocabularies%2E+In+Proceedings+of+the+17th+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Linguistics%2C)
52. Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. 2023a. Tokenization and the noiseless channel. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5184–5207, Toronto, Canada. Association for… [find](https://scholar.google.com/scholar?q=Vil%C3%A9m+Zouhar%2C+Clara+Meister%2C+Juan+Gastaldi%2C+Li+Du%2C+Mrinmaya+Sachan%2C+and+Ryan+Cotterell%2E+2023a%2E+Tokenization+and+the+noiseless+channel%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+)
53. Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Tim Vieira, Mrinmaya Sachan, and Ryan Cotterell. 2023b. A formal perspective on byte-pair encoding. In Findings of the Association for Computational Linguistics: ACL 2023, pages 598–614, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Vil%C3%A9m+Zouhar%2C+Clara+Meister%2C+Juan+Gastaldi%2C+Li+Du%2C+Tim+Vieira%2C+Mrinmaya+Sachan%2C+and+Ryan+Cotterell%2E+2023b%2E+A+formal+perspective+on+byte%2Dpair+encoding%2E+In+Findings+of+the+Association+for+Computational+)