--- title: "SentencePiece — Kudo 2018" description: "Neural text systems such as NMT still often depend on preprocessing that is tied to specific languages." authors: ["Taku Kudo", "John Richardson"] year: 2018 venue: "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations" source: "https://aclanthology.org/D18-2012/" doi: "10.18653/v1/d18-2012" arxiv: "1808.06226" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/) / SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing # SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing **Authors:** [Taku Kudo](https://md.research.elamur.ai/free/en/authors/taku-kudo/), [John Richardson](https://md.research.elamur.ai/free/en/authors/john-richardson/) · **Year:** 2018 · **Venue:** Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/D18-2012/) · [PDF](https://aclanthology.org/D18-2012.pdf) · [DOI 10.18653/v1/d18-2012](https://doi.org/10.18653/v1/d18-2012) · [arXiv 1808.06226](https://arxiv.org/abs/1808.06226) · [OpenAlex](https://openalex.org/W2885185669) **Citations:** 2869 citations (openalex-snapshot · 2026-10-14) · [1588](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Neural text systems such as NMT still often depend on preprocessing that is tied to specific languages. Examples include the rule-based Moses tokenizer and the separate word segmenters needed for Chinese, Japanese and Korean, which make multilingual and reproducible pipelines harder to build. Kudo and Richardson present SentencePiece, an open-source subword tokenizer and detokenizer written in C++ with Python and TensorFlow bindings. It learns BPE or unigram language model vocabularies directly from raw sentences, so no pre-tokenization is needed. Whitespace is treated as an ordinary symbol (U+2581), which makes tokenization fully reversible. The tool also handles vocabulary-to-id mapping and NFKC-based normalization, which users can customize. Everything, including the normalization rules, is stored in a single self-contained model file. In English-Japanese experiments on KFTT with GNMT, subword models beat word-level baselines. Training on raw text matched or beat pre-tokenized training, and the gain was largest when Japanese was the target language. This matters because a single tool can now preprocess any language, including unsegmented scripts and morphologically rich languages such as Ukrainian, with no hand-written rules. ## Key points - Supports two segmentation algorithms (BPE and unigram LM). BPE segmentation uses a binary heap, which brings its cost down from O(N²) to O(N log N). Users set the final vocabulary size (`--vocab_size`) rather than a number of merge operations. - Lossless design: decoding an encoded normalized string returns exactly that normalized string. Detokenization only needs to join the pieces and turn the meta symbol back into spaces. - On KFTT (440k training sentences), shared 8k SentencePiece vocabularies reached 29.55 BLEU for ja→en and 21.62 for en→ja. The 80k word baselines scored 28.24 and 20.06. Adding pre-tokenization lowered en→ja to 20.86. - On raw Japanese text, SentencePiece trained in 217.3 s versus 528.0 s for subword-nmt. It segmented in 5.9 s versus 216.2 s. - Released under Apache 2 at github.com/google/sentencepiece. Models are serialized as Protocol buffers, and the API supports on-the-fly subword sampling for subword regularization. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Computing & GPUs](https://md.research.elamur.ai/free/en/shelves/computing/) Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) ## Cited by Cited by 1588 papers of our library cite it (of the 140673 whose bibliographies we read); 85 of them have a page here, 1503 not yet. - [A unified approach to sentence segmentation of punctuated text in many languages](https://md.research.elamur.ai/free/en/papers/acl-2021-wicks-unified-approach-sentence-segmentation-punctuated/) - [Semitic Root Encoding: Tokenization Based on the Templatic Morphology of Semitic Languages in NMT](https://md.research.elamur.ai/free/en/papers/aclbulk-2025-hatch-semitic-root-encoding-tokenization-based/) - [Unified Pre-training for Program Understanding and Generation](https://md.research.elamur.ai/free/en/papers/ahmad-2021-unified-pre-training-program-understanding/) - [Yi: Open Foundation Models by 01.AI](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/) - [Niyamika at BHASHA Task 1: Word-Level Transliteration for English-Hindi Mixed Text in Grammar Correction Using MT5](https://md.research.elamur.ai/free/en/papers/ambaliya-2025-niyamika-bhasha-task-1-word/) - [ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics](https://md.research.elamur.ai/free/en/papers/amrhein-2022-aces-translation-accuracy-challenge-sets/) - [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/) - [On the Cross-lingual Transferability of Monolingual Representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/) - [語彙置換継続事前学習による日英バイリンガルモデルの構築と評価](https://md.research.elamur.ai/free/en/papers/asaba-2024-building-evaluating-japanese-english/) - [Program Synthesis with Large Language Models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/) - [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://md.research.elamur.ai/free/en/papers/bandarkar-2023-belebele-benchmark-parallel-reading-comprehension/) - [Simple, Scalable Adaptation for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/bapna-2019-simple-scalable-adaptation-neural-machine/) - [Findings of the 2019 Conference on Machine Translation (WMT19)](https://md.research.elamur.ai/free/en/papers/barrault-2019-findings-2019-conference-machine-translation/) - [Automatic Evaluation and Analysis of Idioms in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/baziotis-2023-automatic-evaluation-analysis-idioms-neural/) - [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/) - [Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus](https://md.research.elamur.ai/free/en/papers/caswell-2020-language-id-wild-unexpected/) - [Rethinking embedding coupling in pre-trained language models](https://md.research.elamur.ai/free/en/papers/chung-2020-rethinking-embedding-coupling-pre-trained/) - [Seamless: Multilingual Expressive and Streaming Speech Translation](https://md.research.elamur.ai/free/en/papers/communication-2023-seamless-multilingual-expressive-streaming-speech/) - [SeamlessM4T: Massively Multilingual & Multimodal Machine Translation](https://md.research.elamur.ai/free/en/papers/communication-2023-seamlessm4t-massively-multilingual-multimodal-machine/) - [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://md.research.elamur.ai/free/en/papers/cui-2023-efficient-effective-text-encoding-chinese/) - [BERTje: A Dutch BERT Model](https://md.research.elamur.ai/free/en/papers/de-2019-vries-bertje-dutch-bert-model/) - [Moshi: a speech-text foundation model for real-time dialogue](https://md.research.elamur.ai/free/en/papers/defossez-2024-moshi-speech-text-foundation-model/) - [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/) - [What Helps Transformers Recognize Conversational Structure? Importance of Context, Punctuation, and Labels in Dialog Act Recognition](https://md.research.elamur.ai/free/en/papers/elasko-2021-what-helps-transformers-recognize-conversational/) - [MoVoC: Morphology-Aware Subword Construction for Ge'ez Script Languages](https://md.research.elamur.ai/free/en/papers/findings-2025-teklehaymanot-movoc-morphology-aware-subword-construction/) - [MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods](https://md.research.elamur.ai/free/en/papers/finkelstein-2023-mbr-qe-finetuning-training-time/) - [Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust](https://md.research.elamur.ai/free/en/papers/freitag-2022-results-wmt22-metrics-shared-task/) - [Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities](https://md.research.elamur.ai/free/en/papers/fujii-2024-continual-pre-training-cross-lingual/) - [IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://md.research.elamur.ai/free/en/papers/gala-2023-indictrans2-towards-high-quality-accessible/) - [Gemini: A Family of Highly Capable Multimodal Models](https://md.research.elamur.ai/free/en/papers/gemini-2023-team-gemini-family-highly-capable-multimodal/) - [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://md.research.elamur.ai/free/en/papers/gu-2020-domain-specific-language-model-pretraining/) - [Cross-Domain Label-Adaptive Stance Detection](https://md.research.elamur.ai/free/en/papers/hardalov-2021-cross-domain-label-adaptive-stance/) - [Language Is Not All You Need: Aligning Perception with Language Models](https://md.research.elamur.ai/free/en/papers/huang-2023-language-all-need-aligning-perception/) - [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://md.research.elamur.ai/free/en/papers/inan-2023-llama-guard-llm-based-input/) - [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/) - [CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs](https://md.research.elamur.ai/free/en/papers/kishky-2020-ccaligned-massive-collection-cross-lingual/) - [MADLAD-400: A Multilingual And Document-Level Large Audited Dataset](https://md.research.elamur.ai/free/en/papers/kudugunta-2023-madlad-400-multilingual-document-level/) - [Data Augmentation using Pre-trained Transformer Models](https://md.research.elamur.ai/free/en/papers/kumar-2020-data-augmentation-using-pre-trained/) - [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://md.research.elamur.ai/free/en/papers/labrak-2024-biomistral-collection-open-source-pretrained/) - [Unsupervised Translation of Programming Languages](https://md.research.elamur.ai/free/en/papers/lachaux-2020-unsupervised-translation-programming-languages/) - [Can language models learn from explanations in context?](https://md.research.elamur.ai/free/en/papers/lampinen-2022-can-language-models-learn-explanations/) - [Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinya](https://md.research.elamur.ai/free/en/papers/langstream-2026-teklehaymanot-expanding-lexicon-ge-ez-based/) - [Hierarchical Transformers for Multi-Document Summarization](https://md.research.elamur.ai/free/en/papers/liu-2019-hierarchical-transformers-multi-document-summarization/) - [XQ-MEval: A Dataset with Cross-lingual Parallel Quality for Benchmarking Translation Metrics](https://md.research.elamur.ai/free/en/papers/liu-2026-xq-meval-dataset-cross-lingual/) - [Findings of the AmericasNLP 2021 Shared Task on Open Machine Translation for Indigenous Languages of the Americas](https://md.research.elamur.ai/free/en/papers/mager-2021-findings-americasnlp-2021-shared-task/) - [Playing with Words at the National Library of Sweden -- Making a Swedish BERT](https://md.research.elamur.ai/free/en/papers/malmsten-2020-playing-words-national-library-sweden/) - [CamemBERT: a Tasty French Language Model](https://md.research.elamur.ai/free/en/papers/martin-2019-camembert-tasty-french-language-model/) - [EuroLLM: Multilingual Language Models for Europe](https://md.research.elamur.ai/free/en/papers/martins-2024-eurollm-multilingual-language-models-europe/) - [Automatic Prediction of Prominence and Boundary Strength from Text](https://md.research.elamur.ai/free/en/papers/mas-2026-automatic-prediction-prominence-boundary-strength/) - [ParaCLEAN: Improving Translation Quality through Systematic Parallel Data Cleaning](https://md.research.elamur.ai/free/en/papers/mash-2026-paraclean-improving-translation-quality-through/) - [Teaching language models to support answers with verified quotes](https://md.research.elamur.ai/free/en/papers/menick-2022-teaching-language-models-support-answers/) - [Machine Translation Meta Evaluation through Translation Accuracy Challenge Sets](https://md.research.elamur.ai/free/en/papers/moghe-2025-machine-translation-meta-evaluation-through/) - [Adaptive Machine Translation with Large Language Models](https://md.research.elamur.ai/free/en/papers/moslem-2023-adaptive-machine-translation-large-language/) - [When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/muller-2021-when-being-unseen-mbert-is/) - [Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark](https://md.research.elamur.ai/free/en/papers/niklaus-2021-swiss-judgment-prediction-multilingual-legal/) - [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://md.research.elamur.ai/free/en/papers/park-2023-linear-representation-hypothesis-geometry-large/) - [A Recipe For Arbitrary Text Style Transfer with Large Language Models](https://md.research.elamur.ai/free/en/papers/reif-2021-recipe-arbitrary-text-style-transfer/) - [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://md.research.elamur.ai/free/en/papers/rothe-2019-leveraging-pre-trained-checkpoints-sequence/) - [Code Llama: Open Foundation Models for Code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/) - [AudioPaLM: A Large Language Model That Can Speak and Listen](https://md.research.elamur.ai/free/en/papers/rubenstein-2023-audiopalm-large-language-model-that/) - [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://md.research.elamur.ai/free/en/papers/schwenk-2019-wikimatrix-mining-135m-parallel-sentences/) - [Diacritic Restoration for Low-Resource Indigenous Languages: Case Study with Bribri and Cook Islands Māori](https://md.research.elamur.ai/free/en/papers/solano-2026-diacritic-restoration-low-resource-indigenous/) - [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/) - [Synthetic Instruction Generation for Low-Resource Nordic Languages: Viability and Limitations in LLM Instruction-Tuning](https://md.research.elamur.ai/free/en/papers/stenlund-2026-synthetic-instruction-generation-low-resource/) - [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://md.research.elamur.ai/free/en/papers/sun-2020-mobilebert-compact-task-agnostic-bert/) - [Multilingual Translation from Denoising Pre-Training](https://md.research.elamur.ai/free/en/papers/tang-2021-multilingual-translation-denoising-pre-training/) - [Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing](https://md.research.elamur.ai/free/en/papers/thompson-2020-automatic-machine-translation-evaluation-many/) - [Paraphrase Generation as Zero-Shot Multilingual Translation: Disentangling Semantic Similarity from Lexical and Syntactic Diversity](https://md.research.elamur.ai/free/en/papers/thompson-2020-paraphrase-generation-as-zero-shot/) - [LaMDA: Language Models for Dialog Applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/) - [The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT](https://md.research.elamur.ai/free/en/papers/tiedemann-2020-tatoeba-translation-challenge-realistic-data/) - [Facebook AI's WMT21 News Translation Task Submission](https://md.research.elamur.ai/free/en/papers/tran-2021-facebook-ai-s-wmt21-news/) - [Multimodal Few-Shot Learning with Frozen Language Models](https://md.research.elamur.ai/free/en/papers/tsimpoukelli-2021-multimodal-few-shot-learning-frozen/) - [Multilingual is not enough: BERT for Finnish](https://md.research.elamur.ai/free/en/papers/virtanen-2019-multilingual-enough-bert-finnish/) - [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://md.research.elamur.ai/free/en/papers/wang-2021-simvlm-simple-visual-language-model/) - [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://md.research.elamur.ai/free/en/papers/wang-2021-voxpopuli-large-scale-multilingual-speech/) - [Are All Languages Created Equal in Multilingual BERT?](https://md.research.elamur.ai/free/en/papers/wu-2020-are-all-languages-created-equal/) - [BloombergGPT: A Large Language Model for Finance](https://md.research.elamur.ai/free/en/papers/wu-2023-bloomberggpt-large-language-model-finance/) - [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://md.research.elamur.ai/free/en/papers/xu-2020-clue-chinese-language-understanding/) - [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/) - [Multilingual Universal Sentence Encoder for Semantic Retrieval](https://md.research.elamur.ai/free/en/papers/yang-2020-multilingual-universal-sentence-encoder-semantic/) - [Baichuan 2: Open Large-scale Language Models](https://md.research.elamur.ai/free/en/papers/yang-2023-baichuan-2-open-large-scale/) - [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/) - [Big Bird: Transformers for Longer Sequences](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/) - [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://md.research.elamur.ai/free/en/papers/zhang-2020-improving-massively-multilingual-neural-machine/) - [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{kudo2018sentencepiece, title = {{SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing}}, author = {Kudo, Taku and Richardson, John}, booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations}, year = {2018}, doi = {10.18653/v1/d18-2012}, eprint = {1808.06226}, archivePrefix = {arXiv}, url = {https://doi.org/10.18653/v1/d18-2012} } ``` - [← previous Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks](https://md.research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/) - [next → Sentences with Gapping: Parsing and Reconstructing Elided Predicates](https://md.research.elamur.ai/free/en/papers/ud-2018-schuster-sentences-gapping-parsing-reconstructing-elided/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/) ## References 3 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2017. Unsupervised neural machine translation. arXive preprint arXiv:1710.11041. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Gorka+Labaka%2C+Eneko+Agirre%2C+and+Kyunghyun+Cho%2E+2017%2E+Unsupervised+neural+machine+translation%2E+arXive+preprint+arXiv%3A1710%2E11041%2E) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2017-unsupervised-neural-machine-translation/) 2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2014%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+arXiv+preprint+arXiv%3A1409%2E0473%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/) 3. Michael Denkowski and Graham Neubig. 2017. Stronger baselines for trustable results in neural machine translation. Proc. of Workshop on Neural Machine Translation. *Table 2: Segmentation performance. KFTT corpus (440k* [find](https://scholar.google.com/scholar?q=Michael+Denkowski+and+Graham+Neubig%2E+2017%2E+Stronger+baselines+for+trustable+results+in+neural+machine+translation%2E+Proc%2E+of+Workshop+on+Neural+Machine+Translation%2E+%2ATable+2%3A+Segmentation+performance%2E+) [our card](https://md.research.elamur.ai/free/en/papers/denkowski-2017-stronger-baselines-trustable-results-neural/)