---
title: "Polyglot: Distributed Word Representations — Al-Rfou' 2013"
description: "Most NLP systems rely on hand-engineered, language-specific features and are built mainly for English, so porting them to other languages is expensive."
authors: ["Rami Al-Rfou'", "Bryan Perozzi", "Steven Skiena"]
year: 2013
venue: "Proceedings of the Seventeenth Conference on Computational Natural Language Learning"
source: "https://aclanthology.org/W13-3520/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rfou-2013-polyglot-distributed-word-representations-multilingual/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rfou-2013-polyglot-distributed-word-representations-multilingual/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2013](https://md.research.elamur.ai/free/en/papers/2013/) / Polyglot: Distributed Word Representations for Multilingual NLP

# Polyglot: Distributed Word Representations for Multilingual NLP

**Authors:** [Rami Al-Rfou'](https://md.research.elamur.ai/free/en/authors/rami-al-rfou/), Bryan Perozzi, Steven Skiena · **Year:** 2013 · **Venue:** Proceedings of the Seventeenth Conference on Computational Natural Language Learning · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W13-3520/) · [PDF](https://aclanthology.org/W13-3520.pdf)

**Citations:** [192](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Most NLP systems rely on hand-engineered, language-specific features and are built mainly for English, so porting them to other languages is expensive. Learned word embeddings offer an alternative, but they are costly to train and had mostly been studied on English. The authors trained distributed word representations for 117 languages, each with more than 10,000 Wikipedia articles, using every language's Wikipedia as the corpus. The model follows the SENNA-style ranking approach. A neural network scores a window of words and receives a hinge-loss penalty unless it ranks the real window above a copy whose central word was randomly swapped, by a margin. Normalization is deliberately light: case is kept and digits become #. Vocabularies are capped at 100,000 frequent tokens. Optimizations contributed to Theano made the training feasible. A neural PoS tagger that uses only these embeddings as features, trained on the universal tagset, is competitive with the TnT tagger and even beats it for English and Danish. The public release gives researchers ready-made features for languages, including low-resource ones, where they lack expertise.

## Key points
- Resource: embeddings for 117 Wikipedia languages, with vocabularies of up to 100,000 words each. Out-of-vocabulary words are mapped to a single ⟨UNK⟩ token.
- Because case is preserved, the English model keeps "Apple" near IT companies (e.g., Dell) and "apple" near fruits and vegetables.
- PoS tagger setup:
  - window of 2 words on each side, hidden layer of 300, learning rate 0.3;
  - evaluated on the 12-tag universal tagset of Petrov et al. (2012);
  - German and Czech exceed 98.5% accuracy on known words;
  - Slovene, which had the least data, is weakest at 93.46%.
- Replacing the embedding initialization with random vectors lowers accuracy for every language tested. The largest drops are Slovene (−2.68%, only 27,284 training examples) and Bulgarian (−2.01%); the smallest is English (−0.25%).
- Main weaknesses are domain mismatch and handling of unknown words. Word-type coverage of the PoS datasets ranges from 60.68% (German) to 83.67% (Slovene), and the tagger uses no character-level information.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Part-of-speech tagging](https://md.research.elamur.ai/free/en/topics/part-of-speech-tagging/)

## Cited by

Cited by

192 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 178 not yet.

- [Towards a Cleaner Document-Oriented Multilingual Crawled Corpus](https://md.research.elamur.ai/free/en/papers/abadji-2022-towards-cleaner-document-oriented-multilingual/)
- [Evaluating Word Embeddings on Low-Resource Languages](https://md.research.elamur.ai/free/en/papers/aclbulk-2020-stringham-evaluating-word-embeddings-low-resource/)
- [Massively Multilingual Word Embeddings](https://md.research.elamur.ai/free/en/papers/ammar-2016-massively-multilingual-word-embeddings/)
- [Tailoring Continuous Word Representations for Dependency Parsing](https://md.research.elamur.ai/free/en/papers/bansal-2014-tailoring-continuous-word-representations-dependency/)
- [Neural Surface Realization for Italian](https://md.research.elamur.ai/free/en/papers/basile-2018-neural-surface-realization-italian/)
- [Multi-view and multi-task training of RST discourse parsers](https://md.research.elamur.ai/free/en/papers/braud-2016-multi-view-multi-task-training/)
- [Improving Coreference Resolution by Learning Entity-Level Distributed Representations](https://md.research.elamur.ai/free/en/papers/clark-2016-improving-coreference-resolution-learning-entity/)
- [Learning Word Vectors for 157 Languages](https://md.research.elamur.ai/free/en/papers/grave-2018-learning-word-vectors-157-languages/)
- [Whit's the Richt Pairt o Speech: PoS tagging for Scots](https://md.research.elamur.ai/free/en/papers/lameris-2021-whit-s-richt-pairt-o/)
- [Linguistic Regularities in Sparse and Explicit Word Representations](https://md.research.elamur.ai/free/en/papers/levy-2014-linguistic-regularities-sparse-explicit-word/)
- [Can we obtain significant success in RST discourse parsing by using Large Language Models?](https://md.research.elamur.ai/free/en/papers/maekawa-2024-can-we-obtain-significant-success/)
- [The Third Multilingual Surface Realisation Shared Task (SR'20): Overview and Evaluation Results](https://md.research.elamur.ai/free/en/papers/mille-2020-third-multilingual-surface-realisation-shared/)
- [Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss](https://md.research.elamur.ai/free/en/papers/plank-2016-multilingual-part-speech-tagging-bidirectional/)
- [On the Limitations of Unsupervised Bilingual Dictionary Induction](https://md.research.elamur.ai/free/en/papers/sogaard-2018-limitations-unsupervised-bilingual-dictionary-induction/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{alrfou2013polyglot,
  title = {{Polyglot: Distributed Word Representations for Multilingual NLP}},
  author = {Al-Rfou', Rami and Perozzi, Bryan and Skiena, Steven},
  booktitle = {Proceedings of the Seventeenth Conference on Computational Natural Language Learning},
  year = {2013},
  url = {https://aclanthology.org/W13-3520/}
}
```

- [← previous Paraphrase-Driven Learning for Open Question Answering](https://md.research.elamur.ai/free/en/papers/fader-2013-paraphrase-driven-learning-open-question/)
- [next → POS Tagging for Historical Texts with Sparse Training Data](https://md.research.elamur.ai/free/en/papers/aclbulk-2013-bollmann-pos-tagging-historical-texts-sparse/)
- [↑ back to papers of 2013](https://md.research.elamur.ai/free/en/papers/2013/)

## References

45 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Susana Afonso, Eckhard Bick, Renato Haber, and Diana Santos. 2002. Floresta sintá (c) tica”: a treebank for portuguese. In Proc. of the Third Intern. Conf. on Language Resources and Evaluation (LREC), pages 1698–1703. [find](https://scholar.google.com/scholar?q=Susana+Afonso%2C+Eckhard+Bick%2C+Renato+Haber%2C+and+Diana+Santos%2E+2002%2E+Floresta+sinta%CC%81+%28c%29+tica%E2%80%9D%3A+a+treebank+for+portuguese%2E+In+Proc%2E+of+the+Third+Intern%2E+Conf%2E+on+Language+Resources+and+Evaluation+%28LREC%29) [our card](https://md.research.elamur.ai/free/en/papers/afonso-2002-floresta-sinta-c-tica-treebank/)
2. Rami Al-Rfou’ and Steven Skiena. 2012. Speedread: A fast named entity recognition pipeline. In Proceedings of the 24th International Conference on Computational Linguistics (Coling 2012), pages 53– 61, Mumbai, India, December. [find](https://scholar.google.com/scholar?q=Rami+Al%2DRfou%E2%80%99+and+Steven+Skiena%2E+2012%2E+Speedread%3A+A+fast+named+entity+recognition+pipeline%2E+In+Proceedings+of+the+24th+International+Conference+on+Computational+Linguistics+%28Coling+2012%29%2C+pages+53%E2%80%93+61)
3. Coling 2012 Organizing Committee. [find](https://scholar.google.com/scholar?q=Coling+2012+Organizing+Committee%2E)
4. Eduard Bejček, Jarmila Panevová, Jan Popelka, Pavel Straňák, Magda Ševčı́ková, Jan Štěpánek, and Zdeněk Žabokrtský. 2012. Prague Dependency Treebank 2.5 – a revisited version of PDT 2.0. In Proceedings of COLING 2012, pages 231–246, Mumbai, India, December. [find](https://scholar.google.com/scholar?q=Eduard+Bejc%CC%8Cek%2C+Jarmila+Panevova%CC%81%2C+Jan+Popelka%2C+Pavel+Stran%CC%8Ca%CC%81k%2C+Magda+S%CC%8Cevc%CC%8C%C4%B1%CC%81kova%CC%81%2C+Jan+S%CC%8Cte%CC%8Cpa%CC%81nek%2C+and+Zdene%CC%8Ck+Z%CC%8Cabokrtsky%CC%81%2E+2012%2E+Prague+Dependency+Treebank+2%2E5+%E2%80%93+a+revisited+version+of+PDT+2%2E0%2E+)
5. The COLING 2012 Organizing Committee. [find](https://scholar.google.com/scholar?q=The+COLING+2012+Organizing+Committee%2E)
6. Yoshua Bengio and J-S Senecal. 2008. Adaptive importance sampling to accelerate training of a neural probabilistic language model. Neural Networks, IEEE Transactions on, 19(4):713–722. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio+and+J%2DS+Senecal%2E+2008%2E+Adaptive+importance+sampling+to+accelerate+training+of+a+neural+probabilistic+language+model%2E+Neural+Networks%2C+IEEE+Transactions+on%2C+19%284%29%3A713%E2%80%93722%2E)
7. Y. Bengio, H. Schwenk, J.S. Senécal, F. Morin, and J.L. Gauvain. 2006. Neural probabilistic language models. Innovations in Machine Learning, pages 137– 186. [find](https://scholar.google.com/scholar?q=Y%2E+Bengio%2C+H%2E+Schwenk%2C+J%2ES%2E+Sene%CC%81cal%2C+F%2E+Morin%2C+and+J%2EL%2E+Gauvain%2E+2006%2E+Neural+probabilistic+language+models%2E+Innovations+in+Machine+Learning%2C+pages+137%E2%80%93+186%2E)
8. Y. Bengio, J. Louradour, R. Collobert, and J. Weston. 2009. Curriculum learning. In International Conference on Machine Learning, ICML. [find](https://scholar.google.com/scholar?q=Y%2E+Bengio%2C+J%2E+Louradour%2C+R%2E+Collobert%2C+and+J%2E+Weston%2E+2009%2E+Curriculum+learning%2E+In+International+Conference+on+Machine+Learning%2C+ICML%2E) [our card](https://md.research.elamur.ai/free/en/papers/bengio-2009-curriculum-learning/)
9. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. 2010. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June. Oral… [find](https://scholar.google.com/scholar?q=James+Bergstra%2C+Olivier+Breuleux%2C+Fre%CC%81de%CC%81ric+Bastien%2C+Pascal+Lamblin%2C+Razvan+Pascanu%2C+Guillaume+Desjardins%2C+Joseph+Turian%2C+David+Warde%2DFarley%2C+and+Yoshua+Bengio%2E+2010%2E+Theano%3A+a+CPU+and+GPU+math+expre)
10. John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspondence learning. In Conference on Empirical Methods in Natural Language Processing, Sydney, Australia. [find](https://scholar.google.com/scholar?q=John+Blitzer%2C+Ryan+McDonald%2C+and+Fernando+Pereira%2E+2006%2E+Domain+adaptation+with+structural+correspondence+learning%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+Sydney%2C+Australia)
11. Léon Bottou. 1991. Stochastic gradient learning in neural networks. In Proceedings of Neuro-Nı̂mes 91, Nimes, France. EC2. [find](https://scholar.google.com/scholar?q=Le%CC%81on+Bottou%2E+1991%2E+Stochastic+gradient+learning+in+neural+networks%2E+In+Proceedings+of+Neuro%2DN%C4%B1%CC%82mes+91%2C+Nimes%2C+France%2E+EC2%2E)
12. Sabine Brants, Stefanie Dipper, Silvia Hansen, Wolfgang Lezius, and George Smith. 2002. The tiger treebank. In IN PROCEEDINGS OF THE WORK-SHOP ON TREEBANKS AND LINGUISTIC THEO-RIES, pages 24–41. [find](https://scholar.google.com/scholar?q=Sabine+Brants%2C+Stefanie+Dipper%2C+Silvia+Hansen%2C+Wolfgang+Lezius%2C+and+George+Smith%2E+2002%2E+The+tiger+treebank%2E+In+IN+PROCEEDINGS+OF+THE+WORK%2DSHOP+ON+TREEBANKS+AND+LINGUISTIC+THEO%2DRIES%2C+pages+24%E2%80%9341%2E)
13. Thorsten Brants. 2000. Tnt: a statistical part-of-speech tagger. In Proceedings of the sixth conference on Applied natural language processing, pages 224–231. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Thorsten+Brants%2E+2000%2E+Tnt%3A+a+statistical+part%2Dof%2Dspeech+tagger%2E+In+Proceedings+of+the+sixth+conference+on+Applied+natural+language+processing%2C+pages+224%E2%80%93231%2E+Association+for+Computational+Linguistics) [our card](https://md.research.elamur.ai/free/en/papers/brants-2000-tnt-statistical-part-speech-tagger/)
14. Peter F Brown, Peter V Desouza, Robert L Mercer, Vincent J Della Pietra, and Jenifer C Lai. 1992. Class-based n-gram models of natural language. Computational linguistics, 18(4):467–479. [find](https://scholar.google.com/scholar?q=Peter+F+Brown%2C+Peter+V+Desouza%2C+Robert+L+Mercer%2C+Vincent+J+Della+Pietra%2C+and+Jenifer+C+Lai%2E+1992%2E+Class%2Dbased+n%2Dgram+models+of+natural+language%2E+Computational+linguistics%2C+18%284%29%3A467%E2%80%93479%2E)
15. Minmin Chen, Zhixiang Xu, Kilian Weinberger, and Fei Sha. 2012. Marginalized denoising autoencoders for domain adaptation. In John Langford and Joelle Pineau, editors, Proceedings of the 29th International Conference on Machine Learning (ICML-12), ICML ’12, pages 767–774. ACM, New York, NY, USA, Jul… [find](https://scholar.google.com/scholar?q=Minmin+Chen%2C+Zhixiang+Xu%2C+Kilian+Weinberger%2C+and+Fei+Sha%2E+2012%2E+Marginalized+denoising+autoencoders+for+domain+adaptation%2E+In+John+Langford+and+Joelle+Pineau%2C+editors%2C+Proceedings+of+the+29th+Internat)
16. Yanqing Chen, Bryan Perozzi, Rami Al-Rfou’, and Steven Skiena. 2013. The expressive power of word embeddings. CoRR, abs/1301.3226. — [arXiv:1301.3226](https://arxiv.org/abs/1301.3226) [find](https://scholar.google.com/scholar?q=Yanqing+Chen%2C+Bryan+Perozzi%2C+Rami+Al%2DRfou%E2%80%99%2C+and+Steven+Skiena%2E+2013%2E+The+expressive+power+of+word+embeddings%2E+CoRR%2C+abs%2F1301%2E3226%2E+%E2%80%94+%5BarXiv%3A1301%2E3226%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1301%2E3226%29)
17. R. Collobert and J. Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In International Conference on Machine Learning, ICML. [find](https://scholar.google.com/scholar?q=R%2E+Collobert+and+J%2E+Weston%2E+2008%2E+A+unified+architecture+for+natural+language+processing%3A+Deep+neural+networks+with+multitask+learning%2E+In+International+Conference+on+Machine+Learning%2C+ICML%2E)
18. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. J. Mach. Learn. Res., 12:2493–2537, November. [find](https://scholar.google.com/scholar?q=Ronan+Collobert%2C+Jason+Weston%2C+Le%CC%81on+Bottou%2C+Michael+Karlen%2C+Koray+Kavukcuoglu%2C+and+Pavel+Kuksa%2E+2011%2E+Natural+language+processing+%28almost%29+from+scratch%2E+J%2E+Mach%2E+Learn%2E+Res%2E%2C+12%3A2493%E2%80%932537%2C+November%2E) [our card](https://md.research.elamur.ai/free/en/papers/collobert-2011-natural-language-processing-almost-scratch/)
19. Ronan Collobert. 2011. Deep learning for efficient discriminative parsing. In AISTATS. [find](https://scholar.google.com/scholar?q=Ronan+Collobert%2E+2011%2E+Deep+learning+for+efficient+discriminative+parsing%2E+In+AISTATS%2E)
20. Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc Le, Mark Mao, Marc’Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Ng. 2012. Large scale distributed deep networks. In P. Bartlett, F.C.N. Pereira, C.J.C. Burges, L. Bottou, and K.Q. Weinberger, editors, Advance… [find](https://scholar.google.com/scholar?q=Jeffrey+Dean%2C+Greg+Corrado%2C+Rajat+Monga%2C+Kai+Chen%2C+Matthieu+Devin%2C+Quoc+Le%2C+Mark+Mao%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Andrew+Senior%2C+Paul+Tucker%2C+Ke+Yang%2C+and+Andrew+Ng%2E+2012%2E+Large+scale+distributed+deep+netwo)
21. Sašo Džeroski, Tomaž Erjavec, Nina Ledinek, Petr Pajas, Zdenek Žabokrtsky, and Andreja Žele. 2006. Towards a slovene dependency treebank. In Proc. of the Fifth Intern. Conf. on Language Resources and Evaluation (LREC). [find](https://scholar.google.com/scholar?q=Sas%CC%8Co+Dz%CC%8Ceroski%2C+Tomaz%CC%8C+Erjavec%2C+Nina+Ledinek%2C+Petr+Pajas%2C+Zdenek+Z%CC%8Cabokrtsky%2C+and+Andreja+Z%CC%8Cele%2E+2006%2E+Towards+a+slovene+dependency+treebank%2E+In+Proc%2E+of+the+Fifth+Intern%2E+Conf%2E+on+Language+Resources)
22. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Domain adaptation for large-scale sentiment classification: A deep learning approach. In Proceedings of the Twenty-eight International Conference on Machine Learning (ICML’11), volume 27, pages 97–110, June. [find](https://scholar.google.com/scholar?q=Xavier+Glorot%2C+Antoine+Bordes%2C+and+Yoshua+Bengio%2E+2011%2E+Domain+adaptation+for+large%2Dscale+sentiment+classification%3A+A+deep+learning+approach%2E+In+Proceedings+of+the+Twenty%2Deight+International+Conferenc)
23. Jan Hajič, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Antònia Martı́, Lluı́s Màrquez, Adam Meyers, Joakim Nivre, Sebastian Padó, Jan Štěpánek, Pavel Straňák, Mihai Surdeanu, Nianwen Xue, and Yi Zhang. 2009. The CoNLL-2009 shared task: Syntactic and semantic dependenc… [find](https://scholar.google.com/scholar?q=Jan+Hajic%CC%8C%2C+Massimiliano+Ciaramita%2C+Richard+Johansson%2C+Daisuke+Kawahara%2C+Maria+Anto%CC%80nia+Mart%C4%B1%CC%81%2C+Llu%C4%B1%CC%81s+Ma%CC%80rquez%2C+Adam+Meyers%2C+Joakim+Nivre%2C+Sebastian+Pado%CC%81%2C+Jan+S%CC%8Cte%CC%8Cpa%CC%81nek%2C+Pavel+Stran%CC%8Ca%CC%81k%2C+Mihai+Sur) [our card](https://md.research.elamur.ai/free/en/papers/hajic-2009-conll-2009-shared-task-syntactic/)
24. Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing crosslingual distributed representations of words. In Proceedings of COLING 2012, pages 1459–1474, Mumbai, India, December. [find](https://scholar.google.com/scholar?q=Alexandre+Klementiev%2C+Ivan+Titov%2C+and+Binod+Bhattarai%2E+2012%2E+Inducing+crosslingual+distributed+representations+of+words%2E+In+Proceedings+of+COLING+2012%2C+pages+1459%E2%80%931474%2C+Mumbai%2C+India%2C+December%2E)
25. The COLING 2012 Organizing Committee. [find](https://scholar.google.com/scholar?q=The+COLING+2012+Organizing+Committee%2E)
26. Terry Koo, Xavier Carreras, and Michael Collins. 2008. Simple semi-supervised dependency parsing. In In Proc. ACL/HLT. [find](https://scholar.google.com/scholar?q=Terry+Koo%2C+Xavier+Carreras%2C+and+Michael+Collins%2E+2008%2E+Simple+semi%2Dsupervised+dependency+parsing%2E+In+In+Proc%2E+ACL%2FHLT%2E) [our card](https://md.research.elamur.ai/free/en/papers/koo-2008-simple-semi-supervised-dependency-parsing/)
27. Matthias Trautner Kromann. 2003. The danish dependency treebank and the dtag treebank tool. In Proceedings of the Second Workshop on Treebanks and Linguistic Theories (TLT), page 217. [find](https://scholar.google.com/scholar?q=Matthias+Trautner+Kromann%2E+2003%2E+The+danish+dependency+treebank+and+the+dtag+treebank+tool%2E+In+Proceedings+of+the+Second+Workshop+on+Treebanks+and+Linguistic+Theories+%28TLT%29%2C+page+217%2E)
28. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313–330. [find](https://scholar.google.com/scholar?q=Mitchell+P+Marcus%2C+Mary+Ann+Marcinkiewicz%2C+and+Beatrice+Santorini%2E+1993%2E+Building+a+large+annotated+corpus+of+english%3A+The+penn+treebank%2E+Computational+linguistics%2C+19%282%29%3A313%E2%80%93330%2E) [our card](https://md.research.elamur.ai/free/en/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
29. T. Mikolov, M. Karafiát, L. Burget, J. Cernocky, and S. Khudanpur. 2010. Recurrent neural network based language model. Proceedings of Interspeech. [find](https://scholar.google.com/scholar?q=T%2E+Mikolov%2C+M%2E+Karafia%CC%81t%2C+L%2E+Burget%2C+J%2E+Cernocky%2C+and+S%2E+Khudanpur%2E+2010%2E+Recurrent+neural+network+based+language+model%2E+Proceedings+of+Interspeech%2E)
30. Andriy Mnih and Geoffrey E Hinton. 2009. A scalable hierarchical distributed language model. Advances in neural information processing systems, 21:1081– 1088. [find](https://scholar.google.com/scholar?q=Andriy+Mnih+and+Geoffrey+E+Hinton%2E+2009%2E+A+scalable+hierarchical+distributed+language+model%2E+Advances+in+neural+information+processing+systems%2C+21%3A1081%E2%80%93+1088%2E)
31. Frederic Morin and Yoshua Bengio. 2005. Hierarchical probabilistic neural network language model. In Proceedings of the international workshop on artificial intelligence and statistics, pages 246–252. [find](https://scholar.google.com/scholar?q=Frederic+Morin+and+Yoshua+Bengio%2E+2005%2E+Hierarchical+probabilistic+neural+network+language+model%2E+In+Proceedings+of+the+international+workshop+on+artificial+intelligence+and+statistics%2C+pages+246%E2%80%93252%2E)
32. Roberto Navigli and Simone Paolo Ponzetto. 2010. Babelnet: Building a very large multilingual semantic network. In Proceedings of the 48th annual meeting of the association for computational linguistics, pages 216–225. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Roberto+Navigli+and+Simone+Paolo+Ponzetto%2E+2010%2E+Babelnet%3A+Building+a+very+large+multilingual+semantic+network%2E+In+Proceedings+of+the+48th+annual+meeting+of+the+association+for+computational+linguisti)
33. Joakim Nivre, Jens Nilsson, and Johan Hall. 2006. Talbanken05: A swedish treebank with phrase structure and dependency annotation. In Proceedings of the fifth International Conference on Language Resources and Evaluation (LREC), pages 1392–1395. [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Jens+Nilsson%2C+and+Johan+Hall%2E+2006%2E+Talbanken05%3A+A+swedish+treebank+with+phrase+structure+and+dependency+annotation%2E+In+Proceedings+of+the+fifth+International+Conference+on+Language+Reso)
34. Joakim Nivre, Johan Hall, Sandra Kübler, Ryan Mc-Donald, Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007. [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Johan+Hall%2C+Sandra+Ku%CC%88bler%2C+Ryan+Mc%2DDonald%2C+Jens+Nilsson%2C+Sebastian+Riedel%2C+and+Deniz+Yuret%2E+2007%2E)
35. The CoNLL 2007 shared task on dependency parsing. In Proceedings of the CoNLL Shared Task Session of EMNLP-CoNLL 2007, pages 915–932, Prague, Czech Republic, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=The+CoNLL+2007+shared+task+on+dependency+parsing%2E+In+Proceedings+of+the+CoNLL+Shared+Task+Session+of+EMNLP%2DCoNLL+2007%2C+pages+915%E2%80%93932%2C+Prague%2C+Czech+Republic%2C+June%2E+Association+for+Computational+Lingui)
36. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Eight International C… [find](https://scholar.google.com/scholar?q=Slav+Petrov%2C+Dipanjan+Das%2C+and+Ryan+McDonald%2E+2012%2E+A+universal+part%2Dof%2Dspeech+tagset%2E+In+Nicoletta+Calzolari+%28Conference+Chair%29%2C+Khalid+Choukri%2C+Thierry+Declerck%2C+Mehmet+Ug%CC%86ur+Dog%CC%86an%2C+Bente+Maegaard%2C)
37. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the Sixteenth Conference on Computational Natural Language Learning (CoNLL 2012), Jeju, Korea. [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Alessandro+Moschitti%2C+Nianwen+Xue%2C+Olga+Uryupina%2C+and+Yuchen+Zhang%2E+2012%2E+CoNLL%2D2012+shared+task%3A+Modeling+multilingual+unrestricted+coreference+in+OntoNotes%2E+In+Proceedings+of+the+Six) [our card](https://md.research.elamur.ai/free/en/papers/pradhan-2012-conll-2012-shared-task-modeling/)
38. David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 2002. Learning representations by backpropagating errors. Cognitive modeling, 1:213. [find](https://scholar.google.com/scholar?q=David+E+Rumelhart%2C+Geoffrey+E+Hinton%2C+and+Ronald+J+Williams%2E+2002%2E+Learning+representations+by+backpropagating+errors%2E+Cognitive+modeling%2C+1%3A213%2E)
39. Lu Shuxiang. 2004. The Contemporary Chinese Dictionary (Xiandai Hanyu Cidian). Commercial Press. [find](https://scholar.google.com/scholar?q=Lu+Shuxiang%2E+2004%2E+The+Contemporary+Chinese+Dictionary+%28Xiandai+Hanyu+Cidian%29%2E+Commercial+Press%2E)
40. Kiril Simov, Petya Osenova, Milena Slavcheva, Sia Kolkovska, Elisaveta Balabanova, Dimitar Doikoff, Krassimira Ivanova, Er Simov, and Milen Kouylekov. 2002. Building a linguistically interpreted corpus of bulgarian: the bultreebank. In In: Proceedings of LREC 2002, Canary Islands. [find](https://scholar.google.com/scholar?q=Kiril+Simov%2C+Petya+Osenova%2C+Milena+Slavcheva%2C+Sia+Kolkovska%2C+Elisaveta+Balabanova%2C+Dimitar+Doikoff%2C+Krassimira+Ivanova%2C+Er+Simov%2C+and+Milen+Kouylekov%2E+2002%2E+Building+a+linguistically+interpreted+corpu)
41. Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and Christopher D. Manning. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Advances in Neural Information Processing Systems 24. [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Eric+H%2E+Huang%2C+Jeffrey+Pennington%2C+Andrew+Y%2E+Ng%2C+and+Christopher+D%2E+Manning%2E+2011%2E+Dynamic+pooling+and+unfolding+recursive+autoencoders+for+paraphrase+detection%2E+In+Advances+in+Neural+)
42. Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Brody+Huval%2C+Christopher+D%2E+Manning%2C+and+Andrew+Y%2E+Ng%2E+2012%2E+Semantic+compositionality+through+recursive+matrix%2Dvector+spaces%2E+In+Proceedings+of+the+2012+Conference+on+Empirical+Method)
43. Oscar Täckström, Ryan McDonald, and Jakob Uszkoreit. 2012. Cross-lingual word clusters for direct transfer of linguistic structure. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 477–487. Asso… [find](https://scholar.google.com/scholar?q=Oscar+Ta%CC%88ckstro%CC%88m%2C+Ryan+McDonald%2C+and+Jakob+Uszkoreit%2E+2012%2E+Cross%2Dlingual+word+clusters+for+direct+transfer+of+linguistic+structure%2E+In+Proceedings+of+the+2012+Conference+of+the+North+American+Chapte) [our card](https://md.research.elamur.ai/free/en/papers/tackstrom-2012-cross-lingual-word-clusters-direct/)
44. J. Turian, L. Ratinov, and Y. Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 384–394. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=J%2E+Turian%2C+L%2E+Ratinov%2C+and+Y%2E+Bengio%2E+2010%2E+Word+representations%3A+a+simple+and+general+method+for+semi%2Dsupervised+learning%2E+In+Proceedings+of+the+48th+Annual+Meeting+of+the+Association+for+Computation) [our card](https://md.research.elamur.ai/free/en/papers/turian-2010-word-representations-simple-general-method/)
45. Leonoor Van der Beek, Gosse Bouma, Rob Malouf, and Gertjan Van Noord. 2002. The alpino dependency treebank. Language and Computers, 45(1):8– 22. [find](https://scholar.google.com/scholar?q=Leonoor+Van+der+Beek%2C+Gosse+Bouma%2C+Rob+Malouf%2C+and+Gertjan+Van+Noord%2E+2002%2E+The+alpino+dependency+treebank%2E+Language+and+Computers%2C+45%281%29%3A8%E2%80%93+22%2E)