---
title: "Tokenization — definition and key papers — Elamur Research"
description: "Tokenization is the step in text processing that splits a stream of characters into units called tokens, such as words, punctuation marks…"
canonical: "https://research.elamur.ai/free/en/concepts/tokenization/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/concepts/tokenization/)

[home](https://md.research.elamur.ai/free/en/) / [concepts](https://md.research.elamur.ai/free/en/concepts/) / Tokenization

# 🪙 Tokenization

Concept · 30 key papers · [all concepts](https://md.research.elamur.ai/free/en/concepts/)

Part of: [🗨️ Natural language processing](https://md.research.elamur.ai/free/en/concepts/natural-language-processing/)

Tokenization is the step in text processing that splits a stream of characters into units called tokens, such as words, punctuation marks, numbers or subword pieces. These tokens become the basic items that later analysis or a language model counts, tags and predicts.

Also: word segmentation, lexical analysis, pre-tokenization, subword segmentation, token segmentation

## What it is

Almost every program that reads text first has to decide where one unit ends and the next begins. In [natural language processing](https://md.research.elamur.ai/free/en/concepts/natural-language-processing/) this step is called tokenization. The sentence "Dr. Smith's car didn't start." looks simple. A tokenizer still has to decide several things:

- whether "Dr." keeps its period;
- whether "Smith's" is one token or two;
- whether "didn't" becomes "did" + "n't".

The Penn Treebank convention, for instance, splits don't into do + n't ([Mielke et al. 2021](https://md.research.elamur.ai/free/en/papers/mielke-2021-between-words-characters-brief-history/)).

The difficult cases fall into a few groups:

- Clitics and contractions: English 's, n't, 're; French l'homme.
- Hyphenated forms such as state-of-the-art, and numbers with internal punctuation such as 3.14, 1,000 or 12/05/2023.
- URLs, e-mail addresses, hashtags and emoticons, which naive rules break apart.
- Languages without spaces between words, such as Chinese and Japanese. Here tokenization becomes a real search problem, because most characters can occur in several positions inside a word ([Xue 2003](https://md.research.elamur.ai/free/en/papers/xue-2003-chinese-word-segmentation-as-character/)).
- Languages that attach clitics to their hosts, such as Arabic. In these languages tokenization is tangled up with morphological analysis ([Habash & Rambow 2005](https://md.research.elamur.ai/free/en/papers/habash-2005-arabic-tokenization-part-speech-tagging/)).

A related task is sentence segmentation: deciding which periods, question marks or line breaks end a sentence. It is often solved together with tokenization ([Straka et al. 2016](https://md.research.elamur.ai/free/en/papers/straka-2016-udpipe-trainable-pipeline-processing-conll/)).

Tokens should be kept apart from types. A token is one occurrence in running text, and a type is a distinct form. "the cat saw the dog" has five tokens but four types. The ratio between the two is the [type–token ratio](https://md.research.elamur.ai/free/en/concepts/type-token-ratio/), and how often types recur is described by [Zipf's law](https://md.research.elamur.ai/free/en/concepts/zipfs-law/).

## The key idea

At bottom, a tokenizer makes a binary decision at every gap between characters: boundary or no boundary. Older systems made that decision with hand-written rules, such as regular expressions, abbreviation lists and exceptions for clitics. The Moses tokenizer used in machine translation is the classic example ([Kudo & Richardson 2018](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)).

Learned systems turn the decision into classification. UDPipe gives each character one of three labels: no boundary, token boundary, or sentence boundary. It learns these labels with a bidirectional GRU, trained only by aligning raw text with its tokenized version ([Straka et al. 2016](https://md.research.elamur.ai/free/en/papers/straka-2016-udpipe-trainable-pipeline-processing-conll/)). In Chinese word segmentation the same idea appears as position tags. Each character is marked as word-initial, word-internal, word-final, or a single-character word ([Xue 2003](https://md.research.elamur.ai/free/en/papers/xue-2003-chinese-word-segmentation-as-character/)).

A second key idea is that the "right" segmentation depends on its purpose. For language models the goal is a vocabulary that is small but open, so every string can be represented. A standard way to measure how much a tokenizer splits words is fertility:

```
fertility = T / W
```

Here T is the number of tokens the tokenizer produces for a text, and W is the number of whitespace-delimited words in the same text. A value near 1 means words stay whole, and higher values mean heavier fragmentation. Lower fertility in monolingual tokenizers went along with better downstream results in several languages ([Rust et al. 2021](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/)).

## Try it: Mova’s tokens against GPT-2’s

Type a sentence and see how two tokenizers cut it. [Mova](https://md.research.elamur.ai/free/en/projects/mova/)’s English tokenizer follows the Universal Dependencies conventions: words, punctuation and clitics (`did` + `n’t`), abbreviations kept whole (`U.S.`). GPT-2’s byte-pair encoding cuts by frequency: common words stay whole, a leading space is part of the token, rare words fall into pieces. Both run in your browser; the text is not sent anywhere.

Six of the examples, token by token (Mova’s tokenizer, then GPT-2’s BPE; ␣ marks a space that belongs to the token):

Mova’s tokenizer: a table-driven automaton with dictionaries learned from the English Web Treebank and the ESL spoken treebank of Universal Dependencies (CC BY-SA 4.0). GPT-2: OpenAI’s vocabulary (© 2019 OpenAI, Modified MIT License), its first 12,000 merges of 50,000 — a word the full vocabulary keeps whole can split here.

## History and founding papers

### Rule-based tokenizers and the word problem

Early English pipelines treated tokenization as nearly solved: split on whitespace, then peel off punctuation by rule. Languages written without spaces showed how much was hidden in that assumption. Sproat, Shih, Gale and Chang cast Chinese segmentation as a search over weighted finite-state transducers. Their model combined a dictionary with sub-models for derived words, personal names and transliterated foreign names, and was built for a Mandarin text-to-speech system ([Sproat et al. 1996](https://md.research.elamur.ai/free/en/papers/sproat-1996-stochastic-finite-state-word-segmentation/)). Their work also showed that human annotators disagree about where Chinese words end ([Sproat et al. 1996](https://md.research.elamur.ai/free/en/papers/sproat-1996-stochastic-finite-state-word-segmentation/)).

### Shared benchmarks and character tagging

The SIGHAN bakeoffs made Chinese segmentation comparable across groups. The first, in 2003, offered four corpora, each with its own segmentation standard, plus closed and open tracks ([Sproat & Emerson 2003](https://md.research.elamur.ai/free/en/papers/sproat-2003-first-international-chinese-word-segmentation/)). Further rounds followed in 2005 ([Emerson 2005](https://md.research.elamur.ai/free/en/papers/emerson-2005-second-international-chinese-word-segmentation/)) and 2006, the latter adding named entity recognition ([Levow 2006](https://md.research.elamur.ai/free/en/papers/levow-2006-third-international-chinese-language-processing/)). Across the rounds, out-of-vocabulary words stayed the main weakness ([Emerson 2005](https://md.research.elamur.ai/free/en/papers/emerson-2005-second-international-chinese-word-segmentation/)).

Xue's reformulation of segmentation as character tagging with a maximum entropy model became the dominant approach ([Xue 2003](https://md.research.elamur.ai/free/en/papers/xue-2003-chinese-word-segmentation-as-character/)). Zhang and Clark then trained segmentation and part-of-speech tagging jointly with a single perceptron, which reduced the errors that a pipeline passes from one step to the next ([Zhang & Clark 2008](https://md.research.elamur.ai/free/en/papers/zhang-2008-joint-word-segmentation-pos-tagging/)). Chang, Galley and Manning found that a higher segmentation F-score does not reliably give better translations. Consistency and granularity of the units mattered more ([Chang et al. 2008](https://md.research.elamur.ai/free/en/papers/chang-2008-optimizing-chinese-word-segmentation-machine/)).

For Arabic, Habash and Rambow handled tokenization, tagging and morphological disambiguation in one step, choosing among the outputs of a morphological analyzer ([Habash & Rambow 2005](https://md.research.elamur.ai/free/en/papers/habash-2005-arabic-tokenization-part-speech-tagging/)).

### Unsupervised and neural segmentation

Bayesian models learned word boundaries from unsegmented text or phonemic transcripts. Goldwater, Griffiths and Johnson showed that bigram context matters once search is done properly ([Goldwater et al. 2006](https://md.research.elamur.ai/free/en/papers/goldwater-2006-contextual-dependencies-unsupervised-word-segmentation/)). Mochihashi, Yamada and Ueda nested a character-level Pitman-Yor model inside a word-level one ([Mochihashi et al. 2009](https://md.research.elamur.ai/free/en/papers/mochihashi-2009-bayesian-unsupervised-word-segmentation-nested/)). Neural taggers replaced hand-built feature templates with character embeddings ([Zheng et al. 2013](https://md.research.elamur.ai/free/en/papers/zheng-2013-deep-learning-chinese-word-segmentation/)), and later with LSTM memory ([Chen et al. 2015](https://md.research.elamur.ai/free/en/papers/chen-2015-long-short-term-memory-neural/)).

### Segmenting above the word

Segmentation also happens at larger scales. Choi's C99 split documents into topical blocks using ranked sentence similarities ([Choi 2000](https://md.research.elamur.ai/free/en/papers/choi-2000-advances-domain-independent-linear-text/)). Utiyama and Isahara cast the same task as a probabilistic search with a [minimum description length](https://md.research.elamur.ai/free/en/concepts/minimum-description-length/) prior ([Utiyama & Isahara 2001](https://md.research.elamur.ai/free/en/papers/utiyama-2001-statistical-model-domain-independent-text/)). Pevzner and Hearst diagnosed biases in the Pk metric used for such tasks and proposed WindowDiff ([Pevzner & Hearst 2002](https://md.research.elamur.ai/free/en/papers/pevzner-2002-critique-improvement-evaluation-metric-text/)). Koshorek and colleagues made topic segmentation supervised by using Wikipedia tables of contents as labels ([Koshorek et al. 2018](https://md.research.elamur.ai/free/en/papers/koshorek-2018-text-segmentation-as-supervised-learning/)).

In speech translation, the system's sentence boundaries rarely match the reference. Matusov and colleagues re-segmented MT output so that standard metrics such as [BLEU](https://md.research.elamur.ai/free/en/concepts/bleu/) could still be applied ([Matusov et al. 2005](https://md.research.elamur.ai/free/en/papers/matusov-2005-evaluating-machine-translation-output-automatic/)).

### The step to subwords

Neural [machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) exposed the cost of closed word vocabularies. Sennrich, Haddow and Birch adapted byte pair encoding to split rare words into subword units, so the model could translate an open vocabulary ([Sennrich et al. 2016](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/)). Kudo and Richardson's SentencePiece learned such vocabularies directly from raw text. It treats whitespace as an ordinary symbol, which makes the process reversible and removes the need for language-specific pre-tokenizers ([Kudo & Richardson 2018](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)). BPEmb then released pretrained subword embeddings for 275 languages ([Heinzerling & Strube 2018](https://md.research.elamur.ai/free/en/papers/heinzerling-2018-bpemb-tokenization-free-pre-trained/)). The algorithms themselves are covered under [subword tokenization](https://md.research.elamur.ai/free/en/concepts/subword-tokenization/).

## How it is used today

### Treebank pipelines and multiword tokens

In [Universal Dependencies](https://md.research.elamur.ai/free/en/concepts/universal-dependencies/), tokenization is the first layer of annotation. UD separates the orthographic token from the syntactic words it contains. Spanish "del" or German "zum" is a multiword token that is split into two words, and each word gets its own [lemma](https://md.research.elamur.ai/free/en/concepts/lemma/) and node in the tree. The mapping between tokens and [word forms](https://md.research.elamur.ai/free/en/concepts/word-form/) is therefore not one-to-one ([Mielke et al. 2021](https://md.research.elamur.ai/free/en/papers/mielke-2021-between-words-characters-brief-history/)).

UDPipe 1.2 finds sentence and token boundaries jointly with a character-level network. It splits multiword tokens with a dictionary and suffix rules extracted from the treebank, then passes the result to [part-of-speech tagging](https://md.research.elamur.ai/free/en/concepts/part-of-speech-tagging/) and [dependency parsing](https://md.research.elamur.ai/free/en/concepts/dependency-parsing/) ([Straka & Straková 2017](https://md.research.elamur.ai/free/en/papers/straka-2017-tokenizing-pos-tagging-lemmatizing-parsing/)). It was the baseline system of the CoNLL 2017 shared task on parsing from raw text ([Straka & Straková 2017](https://md.research.elamur.ai/free/en/papers/straka-2017-tokenizing-pos-tagging-lemmatizing-parsing/)).

### Language models

Every [large language model](https://md.research.elamur.ai/free/en/concepts/large-language-models/) built on the [Transformer](https://md.research.elamur.ai/free/en/concepts/transformer/) reads subword ids, so its vocabulary shapes its cost and its coverage. Varying the tokenizer while keeping all else fixed changed both downstream accuracy and training compute. An English-centric tokenizer used for multilingual training cost up to 68% more compute ([Ali et al. 2024](https://md.research.elamur.ai/free/en/papers/ali-2024-tokenizer-choice-llm-training-negligible/)).

The same sentence can need very different numbers of tokens in different languages. For one widely used commercial tokenizer, Bulgarian text took about 2.6 times as many tokens as its English translation ([Petrov et al. 2023](https://md.research.elamur.ai/free/en/papers/petrov-2023-language-model-tokenizers-introduce-unfairness/)). Content-matched comparisons across 22 languages found gaps of up to roughly five times. The gaps were linked to script and encoding as well as to the amount of training data ([Ahia et al. 2023](https://md.research.elamur.ai/free/en/papers/ahia-2023-do-all-languages-cost-same/)). Because commercial APIs charge per token, such gaps translate into price differences. This concern is central to [multilingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) and to [low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/).

### Tokenizer-free models and beyond text

Some encoders skip the tokenizer altogether. CANINE reads Unicode characters through hash embeddings and downsampling ([Clark et al. 2022](https://md.research.elamur.ai/free/en/papers/clark-2022-canine-pre-training-efficient-tokenization/)). Charformer learns soft subword blocks over bytes with a gradient-based module ([Tay et al. 2021](https://md.research.elamur.ai/free/en/papers/tay-2021-charformer-fast-character-transformers-gradient/)).

The notion of tokenization has also spread to other kinds of data. FAST compresses chunks of robot actions with a discrete cosine transform before discretizing them, which makes high-frequency control learnable by next-token prediction ([Pertsch et al. 2025](https://md.research.elamur.ai/free/en/papers/pertsch-2025-fast-efficient-action-tokenization-vision/)).

## Open questions

- No single unit is best. A survey of open-vocabulary modeling concludes that some requirements conflict, for example robustness to noise versus computation that is equally efficient across languages ([Mielke et al. 2021](https://md.research.elamur.ai/free/en/papers/mielke-2021-between-words-characters-brief-history/)).
- Intrinsic metrics are weak guides. Fertility and parity only partly predict model quality ([Ali et al. 2024](https://md.research.elamur.ai/free/en/papers/ali-2024-tokenizer-choice-llm-training-negligible/)). Higher segmentation F-scores did not reliably improve translation ([Chang et al. 2008](https://md.research.elamur.ai/free/en/papers/chang-2008-optimizing-chinese-word-segmentation-machine/)).
- Fairness across languages and scripts is unsolved. Even byte-level encodings are unequal across scripts ([Petrov et al. 2023](https://md.research.elamur.ai/free/en/papers/petrov-2023-language-model-tokenizers-introduce-unfairness/)).
- [Morphology](https://md.research.elamur.ai/free/en/concepts/morphology/) that is not built by adding pieces in a row, such as the root-and-pattern words of [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/), fits poorly with contiguous subword units. This limitation is acknowledged even for learned segmenters ([Tay et al. 2021](https://md.research.elamur.ai/free/en/papers/tay-2021-charformer-fast-character-transformers-gradient/); [Clark et al. 2022](https://md.research.elamur.ai/free/en/papers/clark-2022-canine-pre-training-efficient-tokenization/)).
- Reversibility matters. Most tokenizers normalize text in ways that cannot be undone ([Mielke et al. 2021](https://md.research.elamur.ai/free/en/papers/mielke-2021-between-words-characters-brief-history/)). Tools such as SentencePiece and UDPipe instead record enough information to rebuild the original string ([Kudo & Richardson 2018](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/); [Straka & Straková 2017](https://md.research.elamur.ai/free/en/papers/straka-2017-tokenizing-pos-tagging-lemmatizing-parsing/)).
- What counts as a word in unsegmented scripts remains a matter of convention. The bakeoffs used several incompatible standards ([Sproat & Emerson 2003](https://md.research.elamur.ai/free/en/papers/sproat-2003-first-international-chinese-word-segmentation/)), and annotation inconsistency still affects evaluation ([Levow 2006](https://md.research.elamur.ai/free/en/papers/levow-2006-third-international-chinese-language-processing/)).

Papers on these problems are collected under [tokenization and segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/).

## Key papers

The works this page cites, the most cited first; citations as counted by OpenAlex plus the papers of this library that cite the work, read 2026-10-24.

- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/) — Taku Kudo, John Richardson · 2018 · 4,471 citations
- [Neural Machine Translation of Rare Words with Subword Units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/) — Rico Sennrich, Barry Haddow, Alexandra Birch · 2016 · 3,980 citations
- [Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe](https://md.research.elamur.ai/free/en/papers/straka-2017-tokenizing-pos-tagging-lemmatizing-parsing/) — Milan Straka, Jana Straková · 2017 · 811 citations
- [Advances in domain independent linear text segmentation](https://md.research.elamur.ai/free/en/papers/choi-2000-advances-domain-independent-linear-text/) — Freddy Y. Y. Choi · 2000 · 694 citations
- [Arabic Tokenization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop](https://md.research.elamur.ai/free/en/papers/habash-2005-arabic-tokenization-part-speech-tagging/) — Nizar Habash, Owen Rambow · 2005 · 635 citations
- [A Critique and Improvement of an Evaluation Metric for Text Segmentation](https://md.research.elamur.ai/free/en/papers/pevzner-2002-critique-improvement-evaluation-metric-text/) — Lev Pevzner, Marti A. Hearst · 2002 · 595 citations
- [Chinese Word Segmentation as Character Tagging](https://md.research.elamur.ai/free/en/papers/xue-2003-chinese-word-segmentation-as-character/) — Nianwen Xue · 2003 · 505 citations
- [UDPipe: Trainable Pipeline for Processing CoNLL-U Files Performing Tokenization, Morphological Analysis, POS Tagging and Parsing](https://md.research.elamur.ai/free/en/papers/straka-2016-udpipe-trainable-pipeline-processing-conll/) — Milan Straka, Jan Hajič, Jana Straková · 2016 · 478 citations
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/) — Phillip Rust, Jonas Pfeiffer, Ivan Vulić et al. · 2021 · 444 citations
- [Optimizing Chinese Word Segmentation for Machine Translation Performance](https://md.research.elamur.ai/free/en/papers/chang-2008-optimizing-chinese-word-segmentation-machine/) — Pi-Chuan Chang, Michel Galley, Christopher D. Manning · 2008 · 420 citations
- [A Stochastic Finite-State Word-Segmentation Algorithm for Chinese](https://md.research.elamur.ai/free/en/papers/sproat-1996-stochastic-finite-state-word-segmentation/) — Richard W. Sproat, Chilin Shih, William Gale et al. · 1996 · 413 citations
- [The Third International Chinese Language Processing Bakeoff: Word Segmentation and Named Entity Recognition](https://md.research.elamur.ai/free/en/papers/levow-2006-third-international-chinese-language-processing/) — Gina-Anne Levow · 2006 · 391 citations
- [Contextual Dependencies in Unsupervised Word Segmentation](https://md.research.elamur.ai/free/en/papers/goldwater-2006-contextual-dependencies-unsupervised-word-segmentation/) — Sharon Goldwater, Thomas L. Griffiths, Mark Johnson · 2006 · 372 citations
- [A Statistical Model for Domain-Independent Text Segmentation](https://md.research.elamur.ai/free/en/papers/utiyama-2001-statistical-model-domain-independent-text/) — Masao Utiyama, Hitoshi Isahara · 2001 · 361 citations
- [Deep Learning for Chinese Word Segmentation and POS Tagging](https://md.research.elamur.ai/free/en/papers/zheng-2013-deep-learning-chinese-word-segmentation/) — Xiaoqing Zheng, Hanyang Chen, Tianyu Xu · 2013 · 342 citations
- [Long Short-Term Memory Neural Networks for Chinese Word Segmentation](https://md.research.elamur.ai/free/en/papers/chen-2015-long-short-term-memory-neural/) — Xinchi Chen, Xipeng Qiu, Chenxi Zhu et al. · 2015 · 336 citations
- [Bayesian Unsupervised Word Segmentation with Nested Pitman-Yor Language Modeling](https://md.research.elamur.ai/free/en/papers/mochihashi-2009-bayesian-unsupervised-word-segmentation-nested/) — Daichi Mochihashi, Takeshi Yamada, Naonori Ueda · 2009 · 312 citations
- [Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://md.research.elamur.ai/free/en/papers/clark-2022-canine-pre-training-efficient-tokenization/) — Jonathan H. Clark, Dan Garrette, Iulia Turc et al. · 2022 · 297 citations
- [The First International Chinese Word Segmentation Bakeoff](https://md.research.elamur.ai/free/en/papers/sproat-2003-first-international-chinese-word-segmentation/) — Richard Sproat, Thomas Emerson · 2003 · 286 citations
- [BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages](https://md.research.elamur.ai/free/en/papers/heinzerling-2018-bpemb-tokenization-free-pre-trained/) — Benjamin Heinzerling, Michael Strube · 2018 · 257 citations
- [Text Segmentation as a Supervised Learning Task](https://md.research.elamur.ai/free/en/papers/koshorek-2018-text-segmentation-as-supervised-learning/) — Omri Koshorek, Adir Cohen, Noam Mor et al. · 2018 · 228 citations
- [Language Model Tokenizers Introduce Unfairness Between Languages](https://md.research.elamur.ai/free/en/papers/petrov-2023-language-model-tokenizers-introduce-unfairness/) — Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr et al. · 2023 · 197 citations
- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://md.research.elamur.ai/free/en/papers/mielke-2021-between-words-characters-brief-history/) — Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky et al. · 2021 · 190 citations
- [Joint Word Segmentation and POS Tagging Using a Single Perceptron](https://md.research.elamur.ai/free/en/papers/zhang-2008-joint-word-segmentation-pos-tagging/) — Yue Zhang, Stephen Clark · 2008 · 187 citations
- [Charformer: Fast Character Transformers via Gradient-based Subword Tokenization](https://md.research.elamur.ai/free/en/papers/tay-2021-charformer-fast-character-transformers-gradient/) — Yi Tay, Vinh Q. Tran, Sebastian Ruder et al. · 2021 · 181 citations
- [Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models](https://md.research.elamur.ai/free/en/papers/ahia-2023-do-all-languages-cost-same/) — Orevaoghene Ahia, Sachin Kumar, Hila Gonen et al. · 2023 · 164 citations
- [Evaluating Machine Translation Output with Automatic Sentence Segmentation](https://md.research.elamur.ai/free/en/papers/matusov-2005-evaluating-machine-translation-output-automatic/) — Evgeny Matusov, Gregor Leusch, Oliver Bender et al. · 2005 · 147 citations
- [The Second International Chinese Word Segmentation Bakeoff](https://md.research.elamur.ai/free/en/papers/emerson-2005-second-international-chinese-word-segmentation/) — Thomas Emerson · 2005 · 129 citations
- [FAST: Efficient Action Tokenization for Vision-Language-Action Models](https://md.research.elamur.ai/free/en/papers/pertsch-2025-fast-efficient-action-tokenization-vision/) — Karl Pertsch, Kyle Stachowicz, Brian Ichter et al. · 2025 · 108 citations
- [Tokenizer Choice For LLM Training: Negligible or Crucial?](https://md.research.elamur.ai/free/en/papers/ali-2024-tokenizer-choice-llm-training-negligible/) — Mehdi Ali, Michael Fromm, Klaudia Thellmann et al. · 2024 · 102 citations

## Related concepts

- [Subword tokenization](https://md.research.elamur.ai/free/en/concepts/subword-tokenization/) — Subword tokenization is a way of splitting text into units smaller than words but usually larger than single characters. A vocabulary of…
- [Lemmatization](https://md.research.elamur.ai/free/en/concepts/lemmatization/) — Lemmatization is the task of reducing an inflected or variant word form to its lemma, the canonical dictionary form under which it would be…
- [Stemming](https://md.research.elamur.ai/free/en/concepts/stemming/) — Stemming is the reduction of inflected or derived word forms to a common stem, usually by stripping affixes with rules or statistics, so…
- [Word form](https://md.research.elamur.ai/free/en/concepts/word-form/) — A word form is one concrete written or spoken shape of a word as it appears in text, such as walks or walked. Each form realises a lexeme…

## More papers

Topics: [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) · [🧩 Morphology](https://md.research.elamur.ai/free/en/topics/morphology/) · [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

Shelves: [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/)

Updated 24 October 2026

- [← previous concept TF-IDF](https://md.research.elamur.ai/free/en/concepts/tf-idf/)
- [next concept → Topic modeling](https://md.research.elamur.ai/free/en/concepts/topic-modeling/)
- [↑ back to concepts](https://md.research.elamur.ai/free/en/concepts/)