---
title: "Mova: text → tokens — papers it applies — Elamur Research"
description: "3 papers Mova applies at the stage «text → tokens»: Splitting the text into words with their part of speech, lemma and features: the tagger and its…"
canonical: "https://research.elamur.ai/free/en/projects/mova/tokens/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/projects/mova/tokens/)

[home](https://md.research.elamur.ai/free/en/) / [projects](https://md.research.elamur.ai/free/en/projects/) / [Mova](https://md.research.elamur.ai/free/en/projects/mova/) / [Main list](https://md.research.elamur.ai/free/en/projects/mova/main/) / text → tokens

Mova · papers by stage

# text → tokens

Splitting the text into words with their part of speech, lemma and features: the tagger and its lexicon. The 3 works of [Mova](https://mova.elamur.ai/free/en/)'s own list that this stage uses; measured marks a work whose effect Mova measured in its own experiments (1 here). Each title opens our card: the summary, how Mova uses the work, its references and who cites it. What the stage does is on [Mova's pipeline page](https://mova.elamur.ai/free/en/); every stage is on [the main list](https://md.research.elamur.ai/free/en/projects/mova/main/).

- [TnT – A Statistical Part-of-Speech Tagger](https://md.research.elamur.ai/free/en/papers/brants-2000-tnt-statistical-part-speech-tagger/)measured Thorsten Brants · 2000 **What Mova took:** `en/src/tag.rs` — a direct reimplementation: trigram Markov model over tag classes with a dense transition table, deleted-interpolation smoothing, a word-to-tags lexicon for known words, and suffix models…
- [Part-of-Speech Tagging Guidelines for the Penn Treebank Project (3rd Revision)](https://md.research.elamur.ai/free/en/papers/santorini-1990-part-of-speech-tagging-guidelines-penn-treebank/) Beatrice Santorini · 1990 **What Mova took:** `en/seeds/` — the English grammar seeds (short declarative rule notes with conditions, exceptions, examples and sources) cite the guidelines as the authority for PTB tag decisions; about 43 seed files reference it, mostly under…
- [A Good Part-of-Speech Tagger in about 200 Lines of Python](https://md.research.elamur.ai/free/en/papers/honnibal-2013-good-part-of-speech-tagger/) Matthew Honnibal · 2013 **What Mova took:** `en/src/ptag.rs` — the Penn Treebank tagger follows this recipe: an averaged perceptron (Collins 2002) with a greedy left-to-right pass, as described in the article.