--- title: "Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe — Straka 2017" description: "The paper describes UDPipe 1.1 (the shared-task baseline) and UDPipe 1.2… 1 reference, cited by 244 papers here." authors: ["Milan Straka", "Jana Straková"] year: 2017 venue: "Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies" source: "https://aclanthology.org/K17-3009/" doi: "10.18653/v1/k17-3009" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/straka-2017-tokenizing-pos-tagging-lemmatizing-parsing/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/straka-2017-tokenizing-pos-tagging-lemmatizing-parsing/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/) / Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe # Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe **Authors:** [Milan Straka](https://md.research.elamur.ai/free/en/authors/milan-straka/) and [Jana Straková](https://md.research.elamur.ai/free/en/authors/jana-strakova/) · **Year:** 2017 · **Venue:** Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies **Link:** [aclanthology.org/K17-3009](https://aclanthology.org/K17-3009/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) [**567** citations](https://openalex.org/works?filter=cites:W2741029840) OpenAlex, 2026-10-14 · [**244** citations in our library](https://md.research.elamur.ai/free/en/papers/straka-2017-tokenizing-pos-tagging-lemmatizing-parsing/cited-by/) of the 140,884 papers whose bibliographies we read · [Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies](https://aclanthology.org/volumes/K17-3/) · [licence](https://creativecommons.org/licenses/by/4.0/) - [Publisher's page](https://aclanthology.org/K17-3009/) - [PDF](https://aclanthology.org/K17-3009.pdf) - [DOI 10.18653/v1/k17-3009](https://doi.org/10.18653/v1/k17-3009) - [OpenAlex](https://openalex.org/W2741029840) **Authors (2)** 1. [Milan Straka](https://openalex.org/A5060081515) [ORCID](https://orcid.org/0000-0003-3295-5576) · [Semantic Scholar](https://www.semanticscholar.org/author/2757959) · h 22 · 3,074 citations · 127 works 2. [Jana Straková](https://openalex.org/A5047678040) [ORCID](https://orcid.org/0000-0003-0075-2408) · [Semantic Scholar](https://www.semanticscholar.org/author/2285735) · h 12 · 1,660 citations · 56 works Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart. ## What it is about The paper describes UDPipe 1.1 (the shared-task baseline) and UDPipe 1.2 (the participant system, which placed 8th of 33). Tokenisation and sentence splitting come from one character-level GRU; multi-word tokens are split by a dictionary plus suffix rules; the tagger uses a suffix-based morphological guesser with an averaged perceptron; lemmas are chosen as separate rules by a second tagger; and the parser is the transition-based Parsito with a search-based oracle and a default beam of 5. Tuning steps that helped include a larger tokenizer (dimension 24 to 64) for hard tokenisation, adding other treebanks of the same language to small training sets, and joint segmentation with parsing for three treebanks that lack punctuation. The search-based oracle raises the arc-standard static-oracle LAS from 68.27 to 68.87 on the authors' data. The full model has a median size of about 13 MB and runs at about 1,700 words per second on one CPU core. ## For Mova - I1. en/src/tok.rs: keep the exact spacing in MISC fields (SpacesBefore, SpacesAfter, TokenOffset, as this paper defines them) and measure the round-trip rate of raw text through tokenisation and back on EWT dev. - I2. en/src/parse.rs: train the arc-standard perceptron with a search-based oracle (the best gold-consistent state in the beam, not only the static move) and measure LAS on EWT dev through `en parse-eval`, twice. - I3. en/src/parse.rs: retrain on EWT plus a second English treebank whose licence allows training (check [licence/DATA.md](https://mova.elamur.ai/free/en/docs/licence/DATA/)), and plot dev LAS against the added size with `en parse-eval`, to see whether small treebanks gain as they did here. - I4. en/src/morph.rs: add suffix-rule splitting for contractions that the dictionary misses, applied only when a rule does not fire too often on the training data, and measure LAS on ATIS (currently 63.0, lowercase queries). The ideas touch the tokenizer `en/src/tok.rs`, the parser `en/src/parse.rs` and the lexicon in `en/src/morph.rs`, and they use `en parse-eval`, `en ud-eval` and the ATIS data. The chapters are `languages/en/07-punctuation-capitals-numbers` (contractions, spacing) and `languages/en/19-lexicon-mwe-register` (multi-word tokens). Measure first I2, because a change to the oracle is cheap to test and the paper reports a clear gain for it. ## Institutions [🎓**Charles University** Prague · Czechia 99 works · 18,879 citations](https://md.research.elamur.ai/free/en/universities/charles-university/) Topics[Part-of-speech tagging](https://md.research.elamur.ai/free/en/topics/part-of-speech-tagging/) ## Cite Cite as Milan Straka and Jana Straková. 2017. Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe. Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/k17-3009](https://doi.org/10.18653/v1/k17-3009) ```bibtex @inproceedings{straka2017tokenizing, title = {{Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe}}, author = {Milan Straka and Jana Straková}, booktitle = {Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies}, year = {2017}, doi = {10.18653/v1/k17-3009}, url = {https://doi.org/10.18653/v1/k17-3009} } ``` - [← previous Gated Self-Matching Networks for Reading Comprehension and Question Answering](https://md.research.elamur.ai/free/en/papers/wang-2017-gated-self-matching-networks-reading/) - [next → Semi-supervised sequence tagging with bidirectional language models](https://md.research.elamur.ai/free/en/papers/peters-2017-semi-supervised-sequence-tagging-bidirectional/) - [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Glob-aly normalized transition-based neural networks. In Association for Computational Linguistic. http://arxiv.org/abs/1603.06042. [↗](http://arxiv.org/abs/1603.06042) [our card](https://md.research.elamur.ai/free/en/papers/andor-2016-globally-normalized-transition-based-neural/)