---
title: "Multilingual Part-of-Speech Tagging with — Plank 2016"
description: "The paper examines how well bi-LSTM part-of-speech taggers work outside English."
authors: ["Barbara Plank", "Anders Søgaard", "Yoav Goldberg"]
year: 2016
venue: "arXiv:1604.05529"
source: "https://arxiv.org/abs/1604.05529"
doi: "10.48550/arxiv.1604.05529"
arxiv: "1604.05529"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/plank-2016-multilingual-part-speech-tagging-bidirectional/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/plank-2016-multilingual-part-speech-tagging-bidirectional/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss

# Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss

**Authors:** [Barbara Plank](https://md.research.elamur.ai/free/en/authors/barbara-plank/), [Anders Søgaard](https://md.research.elamur.ai/free/en/authors/anders-sogaard/), [Yoav Goldberg](https://md.research.elamur.ai/free/en/authors/yoav-goldberg/) · **Year:** 2016 · **Venue:** arXiv:1604.05529 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1604.05529) · [PDF](https://arxiv.org/pdf/1604.05529) · [DOI 10.48550/arxiv.1604.05529](https://doi.org/10.48550/arxiv.1604.05529) · [arXiv 1604.05529](https://arxiv.org/abs/1604.05529) · [OpenAlex](https://openalex.org/W2338266296)

**Citations:** 33 citations (openalex-snapshot · 2026-10-14) · [168](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The paper examines how well bi-LSTM part-of-speech taggers work outside English. It tests many languages, several input granularities, small training sets and noisy labels. The authors use a hierarchical bi-LSTM. A lower sequence bi-LSTM builds a word vector from character or unicode byte embeddings. That vector is joined with a word embedding and passed to a context bi-LSTM that assigns tags. They also propose a multi-task variant, FREQBIN, which predicts each token's tag and a binned log frequency of the token in the training data. The extra objective pushes the network to represent rare and common words differently. The models are tested on 22 UD v1.2 treebanks and the WSJ and compared with TnT (an HMM tagger) and a CRF tagger. Combining word and character inputs gives the best representation, and FREQBIN reaches state-of-the-art accuracy. It helps most with out-of-vocabulary words in morphologically rich languages. The work matters because neural taggers turn out to need less training data than commonly assumed, though they degrade more under heavy label noise.

## Key points
- Data: UD v1.2 with 17 universal POS tags, using every language with at least 60k tokens (22 languages), plus WSJ with 45 tags. Baselines are TnT and a crfsuite-based CRF. Code is released as `bplank/bilstm-aux` (built on CNN/pycnn).
- Macro-average accuracy: TnT 94.61, CRF 94.27, word-only bi-LSTM 92.37, w + ~c with POLYGLOT 96.50 and FREQBIN 96.52. FREQBIN is best on 12 of 22 languages.
- On OOV tokens, FREQBIN raises average accuracy from 83.48 to 87.98. Gains are strongest for Arabic, Farsi, Hebrew and Finnish. Most of the benefit from sub-token input falls on rare words in Slavic and non-Indo-European languages.
- The words-plus-characters model beats TnT on every language except Indonesian, with gains above 2 points for Hebrew and Slovene. On WSJ it scores 97.22, comparable to earlier neural taggers.
- TnT wins with very little data. The bi-LSTM becomes competitive at about 500 training sentences, or about 100 for non-Indo-European languages. When more than 30% of labels are corrupted, the bi-LSTM loses more accuracy than TnT.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

Topics: [Part-of-speech tagging](https://md.research.elamur.ai/free/en/topics/part-of-speech-tagging/)

## Cited by

Cited by

168 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 150 not yet.

- [What can we learn from Semantic Tagging?](https://md.research.elamur.ai/free/en/papers/abdou-2018-what-can-we-learn-semantic/)
- [JW300: A Wide-Coverage Parallel Corpus for Low-Resource Languages](https://md.research.elamur.ai/free/en/papers/agic-2019-jw300-wide-coverage-parallel-corpus/)
- [Multi-view and multi-task training of RST discourse parsers](https://md.research.elamur.ai/free/en/papers/braud-2016-multi-view-multi-task-training/)
- [Controllable Paraphrase Generation with a Syntactic Exemplar](https://md.research.elamur.ai/free/en/papers/chen-2019-controllable-paraphrase-generation-syntactic-exemplar/)
- [Restoring Hebrew Diacritics Without a Dictionary](https://md.research.elamur.ai/free/en/papers/findings-2022-gershuni-restoring-hebrew-diacritics-without-dictionary/)
- [Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP](https://md.research.elamur.ai/free/en/papers/goot-2021-massive-choice-ample-tasks-machamp/)
- [A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios](https://md.research.elamur.ai/free/en/papers/hedderich-2021-survey-recent-approaches-natural-language/)
- [Modeling Speech Acts in Asynchronous Conversations: A Neural-CRF Approach](https://md.research.elamur.ai/free/en/papers/joty-2018-modeling-speech-acts-asynchronous-conversations/)
- [Simple Models for Word Formation in English Slang](https://md.research.elamur.ai/free/en/papers/kulkarni-2018-simple-models-word-formation-english/)
- [Whit's the Richt Pairt o Speech: PoS tagging for Scots](https://md.research.elamur.ai/free/en/papers/lameris-2021-whit-s-richt-pairt-o/)
- [Noise-Robust Morphological Disambiguation for Dialectal Arabic](https://md.research.elamur.ai/free/en/papers/naacl-2018-zalmout-noise-robust-morphological-disambiguation-dialectal/)
- [DyNet: The Dynamic Neural Network Toolkit](https://md.research.elamur.ai/free/en/papers/neubig-2017-dynet-dynamic-neural-network-toolkit/)
- [Neural Semantic Parsing by Character-based Translation: Experiments with Abstract Meaning Representations](https://md.research.elamur.ai/free/en/papers/noord-2017-neural-semantic-parsing-character-based/)
- [Annotation, Modelling and Analysis of Fine-Grained Emotions on a Stance and Sentiment Detection Corpus](https://md.research.elamur.ai/free/en/papers/schuff-2017-annotation-modelling-analysis-fine-grained/)
- [BRAGD: Constrained Multi-Label POS Tagging for Faroese](https://md.research.elamur.ai/free/en/papers/simonsen-2026-bragd-constrained-multi-label-pos/)
- [Stanford's Graph-based Neural Dependency Parser at the CoNLL 2017 Shared Task](https://md.research.elamur.ai/free/en/papers/ud-2017-dozat-stanford-s-graph-based-neural/)
- [Universal Dependencies Parsing for Colloquial Singaporean English](https://md.research.elamur.ai/free/en/papers/ud-2017-wang-universal-dependencies-parsing-colloquial-singaporean/)
- [A Survey on Recent Advances in Named Entity Recognition from Deep Learning models](https://md.research.elamur.ai/free/en/papers/yadav-2018-survey-recent-advances-named-entity/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{plank2016multilingual,
  title = {{Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss}},
  author = {Plank, Barbara and Søgaard, Anders and Goldberg, Yoav},
  year = {2016},
  doi = {10.48550/arxiv.1604.05529},
  eprint = {1604.05529},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1604.05529}
}
```

- [← previous Multi30K: Multilingual English-German Image Descriptions](https://md.research.elamur.ai/free/en/papers/elliott-2016-multi30k-multilingual-english-german-image/)
- [next → Multilingual Projection for Parsing Truly Low-Resource Languages](https://md.research.elamur.ai/free/en/papers/agic-2016-multilingual-projection-parsing-truly-low/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/)

## References

29 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed Word Representations for Multilingual NLP. In CoNLL. [find](https://scholar.google.com/scholar?q=Rami+Al%2DRfou%2C+Bryan+Perozzi%2C+and+Steven+Skiena%2E+2013%2E+Polyglot%3A+Distributed+Word+Representations+for+Multilingual+NLP%2E+In+CoNLL%2E) [our card](https://md.research.elamur.ai/free/en/papers/rfou-2013-polyglot-distributed-word-representations-multilingual/)
2. Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2015. Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs. In EMNLP. [find](https://scholar.google.com/scholar?q=Miguel+Ballesteros%2C+Chris+Dyer%2C+and+Noah+A%2E+Smith%2E+2015%2E+Improved+Transition%2Dbased+Parsing+by+Modeling+Characters+instead+of+Words+with+LSTMs%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/ballesteros-2015-improved-transition-based-parsing-modeling/)
3. Thorsten Brants. 2000. Tnt: a statistical part-of-speech tagger. In Proceedings of the sixth conference on Applied natural language processing. [find](https://scholar.google.com/scholar?q=Thorsten+Brants%2E+2000%2E+Tnt%3A+a+statistical+part%2Dof%2Dspeech+tagger%2E+In+Proceedings+of+the+sixth+conference+on+Applied+natural+language+processing%2E) [our card](https://md.research.elamur.ai/free/en/papers/brants-2000-tnt-statistical-part-speech-tagger/)
4. Hao Cheng, Hao Fang, and Mari Ostendorf. 2015. Open-Domain Name Error Detection using a Multi-Task RNN. In EMNLP. [find](https://scholar.google.com/scholar?q=Hao+Cheng%2C+Hao+Fang%2C+and+Mari+Ostendorf%2E+2015%2E+Open%2DDomain+Name+Error+Detection+using+a+Multi%2DTask+RNN%2E+In+EMNLP%2E)
5. Kyunghyun Cho. 2015. Natural language understanding with distributed representation. ArXiv, abs/1511.07916. — [arXiv:1511.07916](https://arxiv.org/abs/1511.07916) [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2E+2015%2E+Natural+language+understanding+with+distributed+representation%2E+ArXiv%2C+abs%2F1511%2E07916%2E+%E2%80%94+%5BarXiv%3A1511%2E07916%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1511%2E07916%29)
6. Grzegorz Chrupała. 2013. Text segmentation with character-level text embeddings. In Workshop on Deep Learning for Audio, Speech and Language Processing, ICML. [find](https://scholar.google.com/scholar?q=Grzegorz+Chrupa%C5%82a%2E+2013%2E+Text+segmentation+with+character%2Dlevel+text+embeddings%2E+In+Workshop+on+Deep+Learning+for+Audio%2C+Speech+and+Language+Processing%2C+ICML%2E)
7. Michael Collins. 2002. Discriminative training methods for Hidden Markov Models. In EMNLP. [find](https://scholar.google.com/scholar?q=Michael+Collins%2E+2002%2E+Discriminative+training+methods+for+Hidden+Markov+Models%2E+In+EMNLP%2E)
8. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537. [find](https://scholar.google.com/scholar?q=Ronan+Collobert%2C+Jason+Weston%2C+Le%CC%81on+Bottou%2C+Michael+Karlen%2C+Koray+Kavukcuoglu%2C+and+Pavel+Kuksa%2E+2011%2E+Natural+language+processing+%28almost%29+from+scratch%2E+The+Journal+of+Machine+Learning+Research%2C+12%3A2) [our card](https://md.research.elamur.ai/free/en/papers/collobert-2011-natural-language-processing-almost-scratch/)
9. Jeffrey L Elman. 1990. Finding structure in time. Cognitive science, 14(2):179–211. [find](https://scholar.google.com/scholar?q=Jeffrey+L+Elman%2E+1990%2E+Finding+structure+in+time%2E+Cognitive+science%2C+14%282%29%3A179%E2%80%93211%2E)
10. Dan Gillick, Cliff Brunk, Oriol Vinyals, and Amarnag Subramanya. 2016. Multilingual language processing from bytes. In NAACL. [find](https://scholar.google.com/scholar?q=Dan+Gillick%2C+Cliff+Brunk%2C+Oriol+Vinyals%2C+and+Amarnag+Subramanya%2E+2016%2E+Multilingual+language+processing+from+bytes%2E+In+NAACL%2E)
11. Yoav Goldberg. 2015. A primer on neural network models for natural language processing. ArXiv, abs/1510.00726. [find](https://scholar.google.com/scholar?q=Yoav+Goldberg%2E+2015%2E+A+primer+on+neural+network+models+for+natural+language+processing%2E+ArXiv%2C+abs%2F1510%2E00726%2E)
12. Alex Graves and Jürgen Schmidhuber. 2005. Frame-wise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks, 18(5):602–610. [find](https://scholar.google.com/scholar?q=Alex+Graves+and+Ju%CC%88rgen+Schmidhuber%2E+2005%2E+Frame%2Dwise+phoneme+classification+with+bidirectional+lstm+and+other+neural+network+architectures%2E+Neural+Networks%2C+18%285%29%3A602%E2%80%93610%2E)
13. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2E)
14. Tobias Horsmann, Nicolai Erbs, and Torsten Zesch. 2015. Fast or accurate?–a comparative evaluation of pos tagging models. In Proceedings of the International Conference of the German Society for Computational Linguistics and Language Technology. [find](https://scholar.google.com/scholar?q=Tobias+Horsmann%2C+Nicolai+Erbs%2C+and+Torsten+Zesch%2E+2015%2E+Fast+or+accurate%3F%E2%80%93a+comparative+evaluation+of+pos+tagging+models%2E+In+Proceedings+of+the+International+Conference+of+the+German+Society+for+Compu)
15. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991. [find](https://scholar.google.com/scholar?q=Zhiheng+Huang%2C+Wei+Xu%2C+and+Kai+Yu%2E+2015%2E+Bidirectional+LSTM%2DCRF+models+for+sequence+tagging%2E+arXiv+preprint+arXiv%3A1508%2E01991%2E) [our card](https://md.research.elamur.ai/free/en/papers/huang-2015-bidirectional-lstm-crf-models-sequence/)
16. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2015. Character-aware neural language models. arXiv preprint arXiv:1508.06615. [find](https://scholar.google.com/scholar?q=Yoon+Kim%2C+Yacine+Jernite%2C+David+Sontag%2C+and+Alexander+M+Rush%2E+2015%2E+Character%2Daware+neural+language+models%2E+arXiv+preprint+arXiv%3A1508%2E06615%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/)
17. E. Kiperwasser and Y. Goldberg. 2016. Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations. ArXiv e-prints. [find](https://scholar.google.com/scholar?q=E%2E+Kiperwasser+and+Y%2E+Goldberg%2E+2016%2E+Simple+and+Accurate+Dependency+Parsing+Using+Bidirectional+LSTM+Feature+Representations%2E+ArXiv+e%2Dprints%2E) [our card](https://md.research.elamur.ai/free/en/papers/narrative-2016-kiperwasser-simple-accurate-dependency-parsing-bidirectional/)
18. Matthieu Labeau, Kevin Löser, and Alexandre Allauzen. 2015. Non-lexical neural architecture for fine-grained pos tagging. In EMNLP. [find](https://scholar.google.com/scholar?q=Matthieu+Labeau%2C+Kevin+Lo%CC%88ser%2C+and+Alexandre+Allauzen%2E+2015%2E+Non%2Dlexical+neural+architecture+for+fine%2Dgrained+pos+tagging%2E+In+EMNLP%2E)
19. Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In EMNLP. [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Chris+Dyer%2C+Alan+W+Black%2C+Isabel+Trancoso%2C+Ramon+Fermandez%2C+Silvio+Amir%2C+Luis+Marujo%2C+and+Tiago+Luis%2E+2015%2E+Finding+function+in+form%3A+Compositional+character+models+for+open+vocabulary+word) [our card](https://md.research.elamur.ai/free/en/papers/ling-2015-finding-function-form-compositional-character/)
20. Pengfei Liu, Shafiq Joty, and Helen Meng. 2015. Fine-grained opinion mining with recurrent neural networks and word embeddings. In EMNLP. [find](https://scholar.google.com/scholar?q=Pengfei+Liu%2C+Shafiq+Joty%2C+and+Helen+Meng%2E+2015%2E+Fine%2Dgrained+opinion+mining+with+recurrent+neural+networks+and+word+embeddings%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2015-fine-grained-opinion-mining-recurrent/)
21. Christopher D Manning. 2011. Part-of-speech tagging from 97% to 100%: is it time for some linguistics? In Computational Linguistics and Intelligent Text Processing. Springer. [find](https://scholar.google.com/scholar?q=Christopher+D+Manning%2E+2011%2E+Part%2Dof%2Dspeech+tagging+from+97%25+to+100%25%3A+is+it+time+for+some+linguistics%3F+In+Computational+Linguistics+and+Intelligent+Text+Processing%2E+Springer%2E) [our card](https://md.research.elamur.ai/free/en/papers/enud-2011-manning-part-speech-tagging-97-100/)
22. Joakim Nivre, Željko Agić, Maria Jesus Aranzabe, Masayuki Asahara, Aitziber Atutxa, Miguel Ballesteros, John Bauer, Kepa Bengoetxea, Riyaz Ahmad Bhat, Cristina Bosco, Sam Bowman, Giuseppe G. A. Celano, Miriam Connor, Marie-Catherine de Marneffe, Arantza Diaz de Ilarraza, Kaja Dobrovoljc, Timothy D… [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Z%CC%8Celjko+Agic%CC%81%2C+Maria+Jesus+Aranzabe%2C+Masayuki+Asahara%2C+Aitziber+Atutxa%2C+Miguel+Ballesteros%2C+John+Bauer%2C+Kepa+Bengoetxea%2C+Riyaz+Ahmad+Bhat%2C+Cristina+Bosco%2C+Sam+Bowman%2C+Giuseppe+G%2E+A%2E+Cela)
23. Barbara Plank, Dirk Hovy, Ryan McDonald, and Anders Søgaard. 2014. Adapting taggers to twitter using not-so-distant supervision. In COLING. [find](https://scholar.google.com/scholar?q=Barbara+Plank%2C+Dirk+Hovy%2C+Ryan+McDonald%2C+and+Anders+S%C3%B8gaard%2E+2014%2E+Adapting+taggers+to+twitter+using+not%2Dso%2Ddistant+supervision%2E+In+COLING%2E)
24. Cicero D Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In ICML. [find](https://scholar.google.com/scholar?q=Cicero+D+Santos+and+Bianca+Zadrozny%2E+2014%2E+Learning+character%2Dlevel+representations+for+part%2Dof%2Dspeech+tagging%2E+In+ICML%2E)
25. Ilya Sutskever, James Martens, and Geoffrey E Hinton. 2011. Generating text with recurrent neural networks. In ICML. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+James+Martens%2C+and+Geoffrey+E+Hinton%2E+2011%2E+Generating+text+with+recurrent+neural+networks%2E+In+ICML%2E)
26. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In NIPS. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+NIPS%2E)
27. Peilu Wang, Yao Qian, Frank K. Soong, Lei He, and Hai Zhao. 2015. Part-of-speech tagging with bidirectional long short-term memory recurrent neural network. pre-print, abs/1510.06168. — [arXiv:1510.06168](https://arxiv.org/abs/1510.06168) [find](https://scholar.google.com/scholar?q=Peilu+Wang%2C+Yao+Qian%2C+Frank+K%2E+Soong%2C+Lei+He%2C+and+Hai+Zhao%2E+2015%2E+Part%2Dof%2Dspeech+tagging+with+bidirectional+long+short%2Dterm+memory+recurrent+neural+network%2E+pre%2Dprint%2C+abs%2F1510%2E06168%2E+%E2%80%94+%5BarXiv%3A1510%2E06)
28. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, pages 649–657. [find](https://scholar.google.com/scholar?q=Xiang+Zhang%2C+Junbo+Zhao%2C+and+Yann+LeCun%2E+2015%2E+Character%2Dlevel+convolutional+networks+for+text+classification%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+649%E2%80%93657%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/)
29. Jie Zhou and Wei Xu. 2015. End-to-end learning of semantic role labeling using recurrent neural networks. In ACL. [find](https://scholar.google.com/scholar?q=Jie+Zhou+and+Wei+Xu%2E+2015%2E+End%2Dto%2Dend+learning+of+semantic+role+labeling+using+recurrent+neural+networks%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhou-2015-end-end-learning-semantic-role/)