--- title: "Tokenization as the Initial Phase in NLP — Webster 1992" description: "It defines the notions of word and token from two points of view, that of the lexicographer and that of a working implementation. Cited by 19 papers here." authors: ["Jonathan J. Webster", "Chunyu Kit"] year: 1992 venue: "COLING 1992 Volume 4: The 14th International Conference on Computational Linguistics" source: "https://aclanthology.org/C92-4173/" doi: "10.3115/992424.992434" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/webster-1992-tokenization-as-initial-phase-nlp/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/webster-1992-tokenization-as-initial-phase-nlp/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [1992](https://md.research.elamur.ai/free/en/papers/1992/) / Tokenization as the Initial Phase in NLP # Tokenization as the Initial Phase in NLP **Authors:** Jonathan J. Webster and [Chunyu Kit](https://md.research.elamur.ai/free/en/authors/chunyu-kit/) · **Year:** 1992 · **Venue:** COLING 1992 Volume 4: The 14th International Conference on Computational Linguistics **Link:** [aclanthology.org/C92-4173](https://aclanthology.org/C92-4173/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only [**362** citations](https://openalex.org/works?filter=cites:W2037450062) OpenAlex, 2026-10-14 · [**19** citations in our library](https://md.research.elamur.ai/free/en/papers/webster-1992-tokenization-as-initial-phase-nlp/cited-by/) of the 140,884 papers whose bibliographies we read · [COLING 1992 Volume 4: The 14th International Conference on Computational Linguistics](https://aclanthology.org/volumes/C92-4/) · [licence](https://creativecommons.org/licenses/by-nc-sa/3.0/) - [Publisher's page](https://aclanthology.org/C92-4173/) - [PDF](https://aclanthology.org/C92-4173.pdf) - [DOI 10.3115/992424.992434](https://doi.org/10.3115/992424.992434) - [OpenAlex](https://openalex.org/W2037450062) **Authors (2)** 1. [Jonathan J. Webster](https://openalex.org/A5031246043) [Semantic Scholar](https://www.semanticscholar.org/author/144623301) · h 16 · 1,684 citations · 123 works 2. [Chunyu Kit](https://openalex.org/A5069506463) [ORCID](https://orcid.org/0000-0002-6445-7400) · [Semantic Scholar](https://www.semanticscholar.org/author/1780916) · h 22 · 1,886 citations · 107 works Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart. ## What it is about The paper argues that tokenization, the first step of NLP, is a real problem with its own definitions. It defines the notions of word and token from two points of view, that of the lexicographer and that of a working implementation. Automatic segmentation of Chinese words serves as the main illustration. For English, the authors develop practical methods to identify compound tokens, such as idioms, phrasal verbs and fixed expressions, so that each of them is read as one unit. The paper gives no evaluation table, so its value for Mova is in the definitions and in the list of compound types. ## Ideas - I3. A count probe: list the candidate compound tokens in EWT test that the dictionary covers, and report how many of them the tokenizer currently splits. This gives the size of the problem before any change. ## For Mova - I1. Keep UD tokens as they are, but mark compound units (phrasal verbs, fixed expressions such as *in spite of*) as spans from a list in the in-code dictionary, and pass the span to the parser as a feature; measure on EWT test the LAS of `fixed`, `compound:prt` and `flat` arcs with `en parse-eval`, with and without the span feature. Joining them into single tokens would break UD tokenization, so the span is the safe form of the paper's idea. - I2. Write the definition of a token used in Mova as a short rule list in [languages/en/19-lexicon-mwe-register.md](https://mova.elamur.ai/free/en/docs/languages/en/19-lexicon-mwe-register/), and check it against the EWT tokens that the paper's criteria would split or join. The code is `en/src/tok.rs` and `en/src/dict.rs`, with the command `en tok-eval`. The chapter is [languages/en/19-lexicon-mwe-register.md](https://mova.elamur.ai/free/en/docs/languages/en/19-lexicon-mwe-register/). The paper's method is for Chinese and for an early lexicon, so only the definitions and the idea of compound tokens carry over. Measure first on EWT test, with the count of joined tokens and the exact-match rate, and check that the result is the same in two runs. Topics[Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) ## Cite Cite as Jonathan J. Webster and Chunyu Kit. 1992. Tokenization as the Initial Phase in NLP. COLING 1992 Volume 4: The 14th International Conference on Computational Linguistics. The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.3115/992424.992434](https://doi.org/10.3115/992424.992434) ```bibtex @inproceedings{webster1992tokenization, title = {{Tokenization as the Initial Phase in NLP}}, author = {Jonathan J. Webster and Chunyu Kit}, booktitle = {COLING 1992 Volume 4: The 14th International Conference on Computational Linguistics}, year = {1992}, doi = {10.3115/992424.992434}, url = {https://doi.org/10.3115/992424.992434} } ``` - [← previous Inside-Outside Reestimation From Partially Bracketed Corpora](https://md.research.elamur.ai/free/en/papers/pereira-1992-inside-outside-reestimation-partially-bracketed/) - [next → A Problem for RST: The Need for Multi-Level Discourse Analysis](https://md.research.elamur.ai/free/en/papers/moore-1992-problem-rst-need-multi-level/) - [↑ back to papers of 1992](https://md.research.elamur.ai/free/en/papers/1992/)