--- title: "Tokenization & segmentation — papers — Elamur Research" description: "Tokenization & segmentation: 77 papers of the Elamur Research library, the most cited first." canonical: "https://research.elamur.ai/free/en/topics/tokenization/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/topics/tokenization/) [home](https://md.research.elamur.ai/free/en/) / [topics](https://md.research.elamur.ai/free/en/topics/) / Tokenization & segmentation # ✂️ Tokenization & segmentation 77 papers · page 1 of 2 · 1–25 · the most cited first · [all topics](https://md.research.elamur.ai/free/en/topics/) Cutting text into words and pieces: tokenizers, subwords, word and sentence segmentation. 💡 Explained: [Minimum description length](https://md.research.elamur.ai/free/en/concepts/minimum-description-length/) · [Subword tokenization](https://md.research.elamur.ai/free/en/concepts/subword-tokenization/) ## Papers (77) **1** [2](https://md.research.elamur.ai/free/en/topics/tokenization/page/2/) [next →](https://md.research.elamur.ai/free/en/topics/tokenization/page/2/) 1. **[Enriching Word Vectors with Subword Information](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/) [Transactions of the Association for Computational Linguistics (TACL)](https://md.research.elamur.ai/free/en/venues/tacl/)**Piotr Bojanowski, Edouard Grave, Armand Joulin, et al.*2017 · 11,848 citations* 2. **[SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**Taku Kudo, John Richardson*2018 · 4,471 citations* 3. **[Neural Machine Translation of Rare Words with Subword Units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Rico Sennrich, Barry Haddow, Alexandra Birch*2016 · 3,980 citations* 4. **[Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://md.research.elamur.ai/free/en/papers/kudo-2018-subword-regularization-improving-neural-network/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Taku Kudo*2018 · 1,566 citations* 5. [**Arabic Tokenization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop**Nizar Habash, Owen Rambow*2005 · 635 citations*](https://md.research.elamur.ai/free/en/papers/habash-2005-arabic-tokenization-part-speech-tagging/) 6. [**Unsupervised Multilingual Sentence Boundary Detection**Tibor Kiss, Jan Strunk*2006 · 523 citations*](https://md.research.elamur.ai/free/en/papers/kiss-2006-unsupervised-multilingual-sentence-boundary-detection/) 7. **[Chinese Word Segmentation as Character Tagging](https://md.research.elamur.ai/free/en/papers/xue-2003-chinese-word-segmentation-as-character/) [Conference on Computational Linguistics and Speech Processing (ROCLING)](https://md.research.elamur.ai/free/en/venues/rocling/)**Nianwen Xue*2003 · 505 citations* 8. [**UDPipe: Trainable Pipeline for Processing CoNLL-U Files Performing Tokenization, Morphological Analysis, POS Tagging and Parsing**Milan Straka, Jan Hajič, Jana Straková*2016 · 478 citations*](https://md.research.elamur.ai/free/en/papers/straka-2016-udpipe-trainable-pipeline-processing-conll/) 9. **[How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Phillip Rust, Jonas Pfeiffer, Ivan Vulić, et al.*2021 · 444 citations* 10. **[Optimizing Chinese Word Segmentation for Machine Translation Performance](https://md.research.elamur.ai/free/en/papers/chang-2008-optimizing-chinese-word-segmentation-machine/) [Conference on Machine Translation (WMT)](https://md.research.elamur.ai/free/en/venues/wmt/)**Pi-Chuan Chang, Michel Galley, Christopher D. Manning*2008 · 420 citations* 11. **[A Stochastic Finite-State Word-Segmentation Algorithm for Chinese](https://md.research.elamur.ai/free/en/papers/sproat-1996-stochastic-finite-state-word-segmentation/) [Computational Linguistics](https://md.research.elamur.ai/free/en/venues/cl/)**Richard W. Sproat, Chilin Shih, William Gale, et al.*1996 · 413 citations* 12. **[The Third International Chinese Language Processing Bakeoff: Word Segmentation and Named Entity Recognition](https://md.research.elamur.ai/free/en/papers/levow-2006-third-international-chinese-language-processing/) [SIGHAN Workshop on Chinese Language Processing](https://md.research.elamur.ai/free/en/venues/sighan-workshop-on-chinese-language-processing/)**Gina-Anne Levow*2006 · 391 citations* 13. **[BPE-Dropout: Simple and Effective Subword Regularization](https://md.research.elamur.ai/free/en/papers/provilkov-2020-bpe-dropout-simple-effective-subword/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Ivan Provilkov, Dmitrii Emelianenko, Elena Voita*2020 · 389 citations* 14. [**Tokenization as the Initial Phase in NLP**Jonathan J. Webster, Chunyu Kit*1992 · 382 citations*](https://md.research.elamur.ai/free/en/papers/webster-1992-tokenization-as-initial-phase-nlp/) 15. **[Contextual Dependencies in Unsupervised Word Segmentation](https://md.research.elamur.ai/free/en/papers/goldwater-2006-contextual-dependencies-unsupervised-word-segmentation/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Sharon Goldwater, Thomas L. Griffiths, Mark Johnson*2006 · 372 citations* 16. **[Deep Learning for Chinese Word Segmentation and POS Tagging](https://md.research.elamur.ai/free/en/papers/zheng-2013-deep-learning-chinese-word-segmentation/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**Xiaoqing Zheng, Hanyang Chen, Tianyu Xu*2013 · 342 citations* 17. **[Long Short-Term Memory Neural Networks for Chinese Word Segmentation](https://md.research.elamur.ai/free/en/papers/chen-2015-long-short-term-memory-neural/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**Xinchi Chen, Xipeng Qiu, Chenxi Zhu, et al.*2015 · 336 citations* 18. **[Distant Supervision for Relation Extraction beyond the Sentence Boundary](https://md.research.elamur.ai/free/en/papers/quirk-2017-distant-supervision-relation-extraction-beyond/) [Conference of the European Chapter of the Association for Computational Linguistics (EACL)](https://md.research.elamur.ai/free/en/venues/eacl/)**Chris Quirk, Hoifung Poon*2017 · 318 citations* 19. **[Bayesian Unsupervised Word Segmentation with Nested Pitman-Yor Language Modeling](https://md.research.elamur.ai/free/en/papers/mochihashi-2009-bayesian-unsupervised-word-segmentation-nested/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Daichi Mochihashi, Takeshi Yamada, Naonori Ueda*2009 · 312 citations* 20. **[Byte Pair Encoding is Suboptimal for Language Model Pretraining](https://md.research.elamur.ai/free/en/papers/bostrom-2020-byte-pair-encoding-is-suboptimal/) [Findings of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/findings/)**Kaj Bostrom, Greg Durrett*2020 · 309 citations* 21. **[Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://md.research.elamur.ai/free/en/papers/clark-2022-canine-pre-training-efficient-tokenization/) [Transactions of the Association for Computational Linguistics (TACL)](https://md.research.elamur.ai/free/en/venues/tacl/)**Jonathan H. Clark, Dan Garrette, Iulia Turc, et al.*2022 · 297 citations* 22. **[The First International Chinese Word Segmentation Bakeoff](https://md.research.elamur.ai/free/en/papers/sproat-2003-first-international-chinese-word-segmentation/) [SIGHAN Workshop on Chinese Language Processing](https://md.research.elamur.ai/free/en/venues/sighan-workshop-on-chinese-language-processing/)**Richard Sproat, Thomas Emerson*2003 · 286 citations* 23. **[BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages](https://md.research.elamur.ai/free/en/papers/heinzerling-2018-bpemb-tokenization-free-pre-trained/) [Language Resources and Evaluation Conference (LREC)](https://md.research.elamur.ai/free/en/venues/lrec/)**Benjamin Heinzerling, Michael Strube*2018 · 257 citations* 24. **[Neural Machine Translation with Byte-Level Subwords](https://md.research.elamur.ai/free/en/papers/wang-2019-neural-machine-translation-byte-level/) [Proceedings of the AAAI Conference on Artificial Intelligence](https://md.research.elamur.ai/free/en/venues/proceedings-of-the-aaai-conference-on-artificial-intelligence/)**Changhan Wang, Kyunghyun Cho, Jiatao Gu*2019 · 228 citations* 25. **[Language Model Tokenizers Introduce Unfairness Between Languages](https://md.research.elamur.ai/free/en/papers/petrov-2023-language-model-tokenizers-introduce-unfairness/) [Conference on Neural Information Processing Systems (NeurIPS)](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/)**Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, et al.*2023 · 197 citations* **1** [2](https://md.research.elamur.ai/free/en/topics/tokenization/page/2/) [next →](https://md.research.elamur.ai/free/en/topics/tokenization/page/2/) Updated 24 October 2026 - [← previous topic Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) - [next topic → Treebanks](https://md.research.elamur.ai/free/en/topics/treebanks/) - [↑ back to topics](https://md.research.elamur.ai/free/en/topics/)