--- title: "Tokenization & segmentation — papers, page 2" description: "Tokenization & segmentation: 68 papers of the Elamur Research library · page 2 of 2 · 26–68, the most cited first." canonical: "https://research.elamur.ai/free/en/topics/tokenization/page/2/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/topics/tokenization/page/2/) [home](https://md.research.elamur.ai/free/en/) / [topics](https://md.research.elamur.ai/free/en/topics/) / [Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) / page 2 # ✂️ Tokenization & segmentation 68 papers · page 2 of 2 · 26–68 · the most cited first · [all topics](https://md.research.elamur.ai/free/en/topics/) Cutting text into words and pieces: tokenizers, subwords, word and sentence segmentation. [← prev](https://md.research.elamur.ai/free/en/topics/tokenization/) [1](https://md.research.elamur.ai/free/en/topics/tokenization/) **2** 26. [**Adaptive Sentence Boundary Disambiguation**David D. Palmer, Marti A. Hearst*1994 · 91 citations*](https://md.research.elamur.ai/free/en/papers/palmer-1994-adaptive-sentence-boundary-disambiguation/) 27. [**WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models**Benjamin Minixhofer, Fabian Paischer, Navid Rekabsaz*2022 · 90 citations*](https://md.research.elamur.ai/free/en/papers/minixhofer-2022-wechsel-effective-initialization-subword-embeddings/) 28. [**Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP**Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky, et al.*2021 · 89 citations*](https://md.research.elamur.ai/free/en/papers/mielke-2021-between-words-characters-brief-history/) 29. [**PySBD: Pragmatic Sentence Boundary Disambiguation**Nipun Sadvilkar, Mark Neumann*2020 · 89 citations*](https://md.research.elamur.ai/free/en/papers/sadvilkar-2020-pysbd-pragmatic-sentence-boundary-disambiguation/) 30. [**Sentence Boundary Detection: A Long Solved Problem?**Jonathon Read, Rebecca Dridan, Stephan Oepen, et al.*2012 · 81 citations*](https://md.research.elamur.ai/free/en/papers/read-2012-sentence-boundary-detection-long-solved/) 31. [**Tokenization Is More Than Compression**Craig W Schmidt, Varshini Reddy, Haoran Zhang, et al.*2024 · 66 citations*](https://md.research.elamur.ai/free/en/papers/schmidt-2024-tokenization-is-more-than-compression/) 32. [**Evaluating Various Tokenizers for Arabic Text Classification**Zaid Alyafeai, Maged S. Al-shaibani, Mustafa Ghaleb, et al.*2021 · 66 citations*](https://md.research.elamur.ai/free/en/papers/alyafeai-2021-evaluating-various-tokenizers-arabic-text/) 33. [**Experiments on Sentence Boundary Detection**Mark Stevenson, Robert Gaizauskas*2000 · 63 citations*](https://md.research.elamur.ai/free/en/papers/stevenson-2000-experiments-sentence-boundary-detection/) 34. [**Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs**Aaditya K. Singh, DJ Strouse*2024 · 53 citations*](https://md.research.elamur.ai/free/en/papers/singh-2024-tokenization-counts-impact-tokenization-arithmetic/) 35. [**SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models**Xin Zhang, Dong Zhang, Shimin Li, et al.*2023 · 51 citations*](https://md.research.elamur.ai/free/en/papers/zhang-2023-speechtokenizer-unified-speech-tokenizer-speech/) 36. [**Getting the most out of your tokenizer for pre-training and domain adaptation**Gautier Dagan, Gabriel Synnaeve, Baptiste Roziere*2024 · 44 citations*](https://md.research.elamur.ai/free/en/papers/dagan-2024-getting-most-out-tokenizer-pre/) 37. [**Tokenization: Returning to a Long Solved Problem – A Survey, Contrastive Experiment, Recommendations, and Toolkit**Rebecca Dridan, Stephan Oepen*2012 · 43 citations*](https://md.research.elamur.ai/free/en/papers/dridan-2012-tokenization-returning-long-solved-problem/) 38. [**Word Segmentation, Unknown-word Resolution, and Morphological Agreement in a Hebrew Parsing System**Yoav Goldberg, Michael Elhadad*2013 · 39 citations*](https://md.research.elamur.ai/free/en/papers/goldberg-2013-word-segmentation-unknown-word-resolution/) 39. [**FAST: Efficient Action Tokenization for Vision-Language-Action Models**Karl Pertsch, Kyle Stachowicz, Brian Ichter, et al.*2025 · 38 citations*](https://md.research.elamur.ai/free/en/papers/pertsch-2025-fast-efficient-action-tokenization-vision/) 40. [**A unified approach to sentence segmentation of punctuated text in many languages**Rachel Wicks, Matt Post*2021 · 38 citations*](https://md.research.elamur.ai/free/en/papers/wicks-2021-unified-approach-sentence-segmentation-punctuated/) 41. [**Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs**Nicolas Boizard, Kevin El Haddad, Celine Hudelot, et al.*2024 · 34 citations*](https://md.research.elamur.ai/free/en/papers/boizard-2024-towards-cross-tokenizer-distillation-universal/) 42. [**On the Proper Treatment of Tokenization in Psycholinguistics**Mario Giulianelli, Luca Malagutti, Juan Luis Gastaldi, et al.*2024 · 23 citations*](https://md.research.elamur.ai/free/en/papers/giulianelli-2024-proper-treatment-tokenization-psycholinguistics/) 43. [**Simultaneous Tokenization and Part-Of-Speech Tagging for Arabic without a Morphological Analyzer**Seth Kulick*2010 · 17 citations*](https://md.research.elamur.ai/free/en/papers/kulick-2010-simultaneous-tokenization-part-speech-tagging/) 44. [**Optimizing the Size of Subword Vocabularies in Dialect Classification**Vani Kanjirangat, Tanja Samardžić, Ljiljana Dolamic, et al.*2023 · 11 citations*](https://md.research.elamur.ai/free/en/papers/kanjirangat-2023-optimizing-size-subword-vocabularies-dialect/) 45. [**On the Importance of Tokenization in Arabic Embedding Models**Mohamed Alkaoud, Mairaj Syed*2020 · 11 citations*](https://md.research.elamur.ai/free/en/papers/alkaoud-2020-importance-tokenization-arabic-embedding-models/) 46. [**A Cross-Task Flexible Transition Model for Arabic Tokenization, Affix Detection, Affix Labeling, POS Tagging, and Dependency Parsing**Stephen Tratz*2013 · 7 citations*](https://md.research.elamur.ai/free/en/papers/tratz-2013-cross-task-flexible-transition-model/) 47. [**Sentence Boundary Detection on Line Breaks in Japanese**Yuta Hayashibe, Kensuke Mitsuzawa*2020 · 6 citations*](https://md.research.elamur.ai/free/en/papers/hayashibe-2020-sentence-boundary-detection-line-breaks/) 48. [**Exploring Tokenization Strategies and Vocabulary Sizes for Enhanced Arabic Language Models**Mohamed Taher Alrefaie, Nour Eldin Morsy, Nada Samir*2024 · 5 citations*](https://md.research.elamur.ai/free/en/papers/alrefaie-2024-exploring-tokenization-strategies-vocabulary-sizes/) 49. [**The Effect of Automatic Tokenization, Vocalization, Stemming, and POS Tagging on Arabic Dependency Parsing**Emad Mohamed*2011 · 5 citations*](https://md.research.elamur.ai/free/en/papers/mohamed-2011-effect-automatic-tokenization-vocalization-stemming/) 50. [**MoVoC: Morphology-Aware Subword Construction for Ge'ez Script Languages**Hailay Kidu Teklehaymanot, Dren Fazlija, Wolfgang Nejdl*2025 · 4 citations*](https://md.research.elamur.ai/free/en/papers/teklehaymanot-2025-movoc-morphology-aware-subword-construction/) 51. [**MoVoC: Morphology-Aware Subword Construction for Geez Script Languages**Hailay Kidu Teklehaymanot, Dren Fazlija, Wolfgang Nejdl*2025 · 4 citations*](https://md.research.elamur.ai/free/en/papers/teklehaymanot-2025-movoc-morphology-aware-subword-construction-geez/) 52. [**Word Segmentation for Akkadian Cuneiform**Timo Homburg, Christian Chiarcos*2016 · 4 citations*](https://md.research.elamur.ai/free/en/papers/homburg-2016-word-segmentation-akkadian-cuneiform/) 53. [**Do Diacritics Matter? Evaluating the Impact of Arabic Diacritics on Tokenization and LLM Benchmarks**Go Inoue, Bashar Alhafni, Nizar Habash, et al.*2026 · 3 citations*](https://md.research.elamur.ai/free/en/papers/inoue-2026-do-diacritics-matter-evaluating-impact/) 54. [**Modelling the Morphology of Verbal Paradigms: A Case Study in the Tokenization of Turkish and Hebrew**Giuseppe Samo, Paola Merlo*2026 · 2 citations*](https://md.research.elamur.ai/free/en/papers/samo-2026-modelling-morphology-verbal-paradigms-case/) 55. [**Arabic Sentence Segmentation Across Genres and Punctuation Conditions**Mohammed Elkholy, Khalid N. Elmadani, Nizar Habash, et al.*2026 · 1 citation*](https://md.research.elamur.ai/free/en/papers/elkholy-2026-arabic-sentence-segmentation-across-genres/) 56. [**Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMs**Yara Alakeel, Chatrine Qwaider, Hanan Aldarmaki, et al.*2026 · 1 citation*](https://md.research.elamur.ai/free/en/papers/alakeel-2026-morphemes-borders-evaluating-root-pattern/) 57. [**Morphemes without Borders: Evaluating Root–Pattern Morphology in Arabic Tokenizers and LLMs**Yara Yousif Alakeel, Chatrine Qwaider, Hanan Aldarmaki, et al.*2026 · 1 citation*](https://md.research.elamur.ai/free/en/papers/alakeel-2026-morphemes-without-borders-evaluating-root/) 58. [**Tokenisation in Machine Translation Does Matter: The impact of different tokenisation approaches for Maltese**Kurt Abela, Kurt Micallef, Marc Tanti, et al.*2024 · 1 citation*](https://md.research.elamur.ai/free/en/papers/abela-2024-tokenisation-machine-translation-does-matter/) 59. [**A Subword Embedding Approach for Variation Detection in Luxembourgish User Comments**Anne-Marie Lutgen, Alistair Plum, Christoph Purschke*2026*](https://md.research.elamur.ai/free/en/papers/lutgen-2026-subword-embedding-approach-variation-detection/) 60. [**Bringing Mapudungun into the Modern MT Ecosystem: Morphology-Aware Tokenization for NLLB-200 Fine-Tuning**Isaac Thompson, Brandon Rogers, Eric Ringger*2026*](https://md.research.elamur.ai/free/en/papers/thompson-2026-bringing-mapudungun-into-modern-mt/) 61. [**LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation**Hailay Kidu Teklehaymanot, Dren Fazlija, Wolfgang Nejdl*2026*](https://md.research.elamur.ai/free/en/papers/teklehaymanot-2026-lgse-lexically-grounded-subword-embedding/) 62. [**SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Models**Dong Liu, Yanxuan Yu*2026*](https://md.research.elamur.ai/free/en/papers/liu-2026-semtoken-semantic-aware-tokenization-efficient/) 63. [**The Impact of Tokenization Algorithms on Hungarian Language Model Performance**Mátyás Osváth, Máté Norbert Molnár, Roland Gunics, et al.*2026*](https://md.research.elamur.ai/free/en/papers/osvath-2026-impact-tokenization-algorithms-hungarian-language/) 64. [**Toxic Subword Pruning for Dialogue Response Generation on Large Language Models**Hongyuan Adam Lu, Wai Lam*2026*](https://md.research.elamur.ai/free/en/papers/lu-2026-toxic-subword-pruning-dialogue-response/) 65. [**Word predictability estimates from language models are not robust to tokenizer vocabulary**Kien Nguyen, Suhas Arehalli*2026*](https://md.research.elamur.ai/free/en/papers/nguyen-2026-word-predictability-estimates-language-models/) 66. [**AraToken: Optimizing Arabic Tokenization with Normalization Pipeline and Language Extension for Qwen3**Mark Kashirskiy, Artiom Lipinski, Ilya Makarov*2025*](https://md.research.elamur.ai/free/en/papers/kashirskiy-2025-aratoken-optimizing-arabic-tokenization-normalization/) 67. [**Semitic Root Encoding: Tokenization Based on the Templatic Morphology of Semitic Languages in NMT**Brendan T. Hatch, Stephen D. Richardson*2025*](https://md.research.elamur.ai/free/en/papers/hatch-2025-semitic-root-encoding-tokenization-based/) 68. [**Review of the First International Ancient Chinese Word Segmentation and POS Tagging Bakeoff (第一届古代汉语分词和词性标注国际评测)**Bin Li (李斌), Yiguo Yuan (袁义国), Jingya Lu (芦靖雅), et al.*2023*](https://md.research.elamur.ai/free/en/papers/li-2023-first-international-ancient/) [← prev](https://md.research.elamur.ai/free/en/topics/tokenization/) [1](https://md.research.elamur.ai/free/en/topics/tokenization/) **2**