---
title: "On the Relation between Linguistic Typology and (Limitations of)"
description: "The paper asks why supposedly language-independent language models perform so unevenly across… 63 references, cited by 54 papers here."
authors: ["Daniela Gerz", "Ivan Vulić", "Edoardo Maria Ponti", "Roi Reichart", "Anna Korhonen"]
year: 2018
venue: "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D18-1029/"
doi: "10.18653/v1/d18-1029"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gerz-2018-relation-between-linguistic-typology-limitations/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gerz-2018-relation-between-linguistic-typology-limitations/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) / On the Relation between Linguistic Typology and (Limitations of) Multilingual Language Modeling

# On the Relation between Linguistic Typology and (Limitations of) Multilingual Language Modeling

**Authors:** [Daniela Gerz](https://md.research.elamur.ai/free/en/authors/daniela-gerz/), [Ivan Vulić](https://md.research.elamur.ai/free/en/authors/ivan-vulic/), [Edoardo Maria Ponti](https://md.research.elamur.ai/free/en/authors/edoardo-maria-ponti/), [Roi Reichart](https://md.research.elamur.ai/free/en/authors/roi-reichart/) and [Anna Korhonen](https://md.research.elamur.ai/free/en/authors/anna-korhonen/) · **Year:** 2018 · **Venue:** [Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2018](https://md.research.elamur.ai/free/en/venues/emnlp/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D18-1029/) · [PDF](https://aclanthology.org/D18-1029.pdf) · [DOI 10.18653/v1/d18-1029](https://doi.org/10.18653/v1/d18-1029) · [OpenAlex](https://openalex.org/W2890225082)

**Citations:** 58 citations (OpenAlex, 2026-10-14) · [54](https://md.research.elamur.ai/free/en/papers/gerz-2018-relation-between-linguistic-typology-limitations/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary

The paper asks why supposedly language-independent language models perform so unevenly across languages, and whether linguistic typology can explain it. The authors built a new LM benchmark of 50 languages, chosen by stratified sampling to cover isolating, fusional, introflexive and agglutinative morphology. On it they evaluated n-gram, neural (LSTM) and character-aware (Char-CNN-LSTM) models. They argue that the common fixed-vocabulary setup, which maps rare words to \<UNK>, makes perplexity look unrealistically good for morphologically rich languages. They therefore use a full-vocabulary setup in which every training word is kept and only unseen test words count as OOV.

The coarse morphological type of a language predicts absolute LM performance. Type-to-token ratio is the corpus statistic most strongly tied to perplexity. Fine-grained WALS features (fusion, inflectional synthesis, exponence, flexivity) help explain this, because they drive the number of rare word forms and how ambiguous character n-grams are. Character information always helps, but the gain varies by language. The study argues that future multilingual LMs should be informed by typology, especially for languages like Tamil or Korean.

## Key points

- **New dataset:** 50 typologically diverse languages. Previous multilingual LM sets reached at most 21 EU languages (Europarl), and the most typologically diverse earlier set had only 10 languages (Vania and Lopez, 2017).
- **Effect of the vocabulary setup:** a Finnish model reported at perplexity ≈20 with a 5k fixed vocabulary reaches ≈2,000 under the full-vocabulary setup.
- **One-way ANOVA on typological features:**
  - fusion has a weak effect (η² = 0.09) on performance;
  - inflectional synthesis has a medium effect (η² = 0.21);
  - exponence shows weak links to the LSTM vs. Char-CNN-LSTM perplexity gap (η² = 0.07 for nouns, 0.04 for verbs).
- **Strongest corpus predictor:** type-to-token ratio (TTR), measured by Pearson's correlation with LM performance.
- **Availability:** code and data are released at people.ds.cam.ac.uk/dsg40/lmmrl.html.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Cambridge](https://md.research.elamur.ai/free/en/universities/university-of-cambridge/) — University, Cambridge, United Kingdom · 155 works · 115,562 citations
- [Technion – Israel Institute of Technology](https://md.research.elamur.ai/free/en/universities/technion-israel-institute-of-technology/) — University, Haifa, Israel · 28 works · 4,941 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 467 works · [🧩 Morphology](https://md.research.elamur.ai/free/en/topics/morphology/) · 277 works

## Cite

Cite as: Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Roi Reichart and Anna Korhonen. 2018. On the Relation between Linguistic Typology and (Limitations of) Multilingual Language Modeling. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{gerz2018relation,
  title = {{On the Relation between Linguistic Typology and (Limitations of) Multilingual Language Modeling}},
  author = {Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Roi Reichart and Anna Korhonen},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1029},
  url = {https://doi.org/10.18653/v1/d18-1029}
}
```

- [← previous Crowdsourcing Question-Answer Meaning Representations](https://md.research.elamur.ai/free/en/papers/michael-2018-crowdsourcing-question-answer-meaning-representations/)
- [next → Neural Models of Factuality](https://md.research.elamur.ai/free/en/papers/rudinger-2018-neural-models-factuality/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/)

## References

63 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Oliver Adams, Adam Makarucha, Graham Neubig, Steven Bird, and Trevor Cohn. 2017. Cross-lingual word embeddings for low-resource language modeling. In Proceedings of EACL, pages 937–947. [find](https://scholar.google.com/scholar?q=Oliver+Adams%2C+Adam+Makarucha%2C+Graham+Neubig%2C+Steven+Bird%2C+and+Trevor+Cohn%2E+2017%2E+Cross%2Dlingual+word+embeddings+for+low%2Dresource+language+modeling%2E+In+Proceedings+of+EACL%2C+pages+937%E2%80%93947%2E)
2. Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. [Polyglot: Distributed word representations for multilingual NLP](https://md.research.elamur.ai/free/en/papers/rfou-2013-polyglot-distributed-word-representations-multilingual/). In Proceedings of CoNLL, pages 183–192.
3. Dik Bakker. 2010. Language sampling. In The Oxford handbook of linguistic typology, pages 100–127. Oxford University Press. [find](https://scholar.google.com/scholar?q=Dik+Bakker%2E+2010%2E+Language+sampling%2E+In+The+Oxford+handbook+of+linguistic+typology%2C+pages+100%E2%80%93127%2E+Oxford+University+Press%2E)
4. Emily M. Bender. 2013. Linguistic fundamentals for natural language processing: 100 essentials from morphology and syntax. Morgan & Claypool Publishers. [find](https://scholar.google.com/scholar?q=Emily+M%2E+Bender%2E+2013%2E+Linguistic+fundamentals+for+natural+language+processing%3A+100+essentials+from+morphology+and+syntax%2E+Morgan+%26+Claypool+Publishers%2E)
5. Toms Bergmanis, Katharina Kann, Hinrich Schütze, and Sharon Goldwater. 2017. Training data augmentation for low-resource morphological inflection. In Proceedings of CoNLL, pages 31–39. [find](https://scholar.google.com/scholar?q=Toms+Bergmanis%2C+Katharina+Kann%2C+Hinrich+Sch%C3%BCtze%2C+and+Sharon+Goldwater%2E+2017%2E+Training+data+augmentation+for+low%2Dresource+morphological+inflection%2E+In+Proceedings+of+CoNLL%2C+pages+31%E2%80%9339%2E)
6. Balthasar Bickel and Johanna Nichols. 2013. Inflectional Synthesis of the Verb. Max Planck Institute for Evolutionary Anthropology, Leipzig. [find](https://scholar.google.com/scholar?q=Balthasar+Bickel+and+Johanna+Nichols%2E+2013%2E+Inflectional+Synthesis+of+the+Verb%2E+Max+Planck+Institute+for+Evolutionary+Anthropology%2C+Leipzig%2E)
7. Alexandra Birch, Miles Osborne, and Philipp Koehn. 2008. Predicting success in machine translation. In Proceedings of EMNLP, pages 745–754. [find](https://scholar.google.com/scholar?q=Alexandra+Birch%2C+Miles+Osborne%2C+and+Philipp+Koehn%2E+2008%2E+Predicting+success+in+machine+translation%2E+In+Proceedings+of+EMNLP%2C+pages+745%E2%80%93754%2E)
8. Steven Bird. 2011. Bootstrapping the language archive: New prospects for natural language processing in preserving linguistic heritage. Linguistic Issues in Language Technology, 6(4). [find](https://scholar.google.com/scholar?q=Steven+Bird%2E+2011%2E+Bootstrapping+the+language+archive%3A+New+prospects+for+natural+language+processing+in+preserving+linguistic+heritage%2E+Linguistic+Issues+in+Language+Technology%2C+6%284%29%2E)
9. Ondřej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013. [Findings of the 2013 Workshop on Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/bojar-2013-findings-2013-workshop-statistical-machine/). In Proceedings of the 8th Workshop on Statistical Machine Translati…
10. Jan A. Botha and Phil Blunsom. 2014. Compositional morphology for word representations and language modelling. In Proceedings of ICML, pages 1899– 1907. [find](https://scholar.google.com/scholar?q=Jan+A%2E+Botha+and+Phil+Blunsom%2E+2014%2E+Compositional+morphology+for+word+representations+and+language+modelling%2E+In+Proceedings+of+ICML%2C+pages+1899%E2%80%93+1907%2E)
11. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, and Phillipp Koehn. 2013. [One billion word benchmark for measuring progress in statistical language modeling](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/). In Proceedings of IN-TERPSEECH, pages 2635–2639.
12. Xie Chen, Xunying Liu, Yanmin Qian, MJF Gales, and Philip C Woodland. 2016. CUED-RNNLM: An open-source toolkit for efficient training and evaluation of recurrent neural network language models. In Proceedings of ICASSP, pages 6000 –6004. [find](https://scholar.google.com/scholar?q=Xie+Chen%2C+Xunying+Liu%2C+Yanmin+Qian%2C+MJF+Gales%2C+and+Philip+C+Woodland%2E+2016%2E+CUED%2DRNNLM%3A+An+open%2Dsource+toolkit+for+efficient+training+and+evaluation+of+recurrent+neural+network+language+models%2E+In+Pro)
13. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. [find](https://scholar.google.com/scholar?q=Matthew+S%2E+Dryer+and+Martin+Haspelmath%2C+editors%2E+2013%2E+WALS+Online%2E+Max+Planck+Institute+for+Evolutionary+Anthropology%2C+Leipzig%2E)
14. Nicholas Evans and Stephen C. Levinson. 2009. The myth of language universals: Language diversity and its importance for cognitive science. Behavioral and Brain Sciences, 32(5):429–448. [find](https://scholar.google.com/scholar?q=Nicholas+Evans+and+Stephen+C%2E+Levinson%2E+2009%2E+The+myth+of+language+universals%3A+Language+diversity+and+its+importance+for+cognitive+science%2E+Behavioral+and+Brain+Sciences%2C+32%285%29%3A429%E2%80%93448%2E)
15. Katja Filippova, Enrique Alfonseca, Carlos A. Colmenares, Lukasz Kaiser, and Oriol Vinyals. 2015. [Sentence compression by deletion with LSTMs](https://md.research.elamur.ai/free/en/papers/filippova-2015-sentence-compression-deletion-lstms/). In Proceedings of EMNLP, pages 360–368.
16. Ankur Gandhe, Florian Metze, and Ian Lane. 2014. Neural network language models for low resource languages. In Proceedings of INTERSPEECH, pages 2615–2619. [find](https://scholar.google.com/scholar?q=Ankur+Gandhe%2C+Florian+Metze%2C+and+Ian+Lane%2E+2014%2E+Neural+network+language+models+for+low+resource+languages%2E+In+Proceedings+of+INTERSPEECH%2C+pages+2615%E2%80%932619%2E)
17. Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Jason Naradowsky, Roi Reichart, and Anna Korhonen. 2018. [Language modeling for morphologically rich languages: Character-aware modeling for word-level prediction](https://md.research.elamur.ai/free/en/papers/gerz-2018-language-modeling-morphologically-rich-languages/). Transactions of the ACL, 6:451– 465.
18. Joshua T. Goodman. 2001. A bit of progress in language modeling. Computer Speech & Language, 15(4):403–434. [find](https://scholar.google.com/scholar?q=Joshua+T%2E+Goodman%2E+2001%2E+A+bit+of+progress+in+language+modeling%2E+Computer+Speech+%26+Language%2C+15%284%29%3A403%E2%80%93434%2E)
19. Edouard Grave, Moustapha Cissé, and Armand Joulin. 2017. Unbounded cache model for online language modeling with open vocabulary. In Proceedings of NIPS, pages 6044–6054. [find](https://scholar.google.com/scholar?q=Edouard+Grave%2C+Moustapha+Ciss%C3%A9%2C+and+Armand+Joulin%2E+2017%2E+Unbounded+cache+model+for+online+language+modeling+with+open+vocabulary%2E+In+Proceedings+of+NIPS%2C+pages+6044%E2%80%936054%2E)
20. Derek Greene and Padraig Cunningham. 2006. Practical solutions to the problem of diagonal dominance in kernel document clustering. In Proceedings of ICML, pages 377–384. [find](https://scholar.google.com/scholar?q=Derek+Greene+and+Padraig+Cunningham%2E+2006%2E+Practical+solutions+to+the+problem+of+diagonal+dominance+in+kernel+document+clustering%2E+In+Proceedings+of+ICML%2C+pages+377%E2%80%93384%2E)
21. Martin Haspelmath and Andrea Sims. 2013. Understanding morphology. [find](https://scholar.google.com/scholar?q=Martin+Haspelmath+and+Andrea+Sims%2E+2013%2E+Understanding+morphology%2E)
22. Ryan Cotterell and Jason Eisner. 2017. Probabilistic typology: Deep generative models of vowel inventories. In Proceedings of ACL, pages 1182–1192. [find](https://scholar.google.com/scholar?q=Ryan+Cotterell+and+Jason+Eisner%2E+2017%2E+Probabilistic+typology%3A+Deep+generative+models+of+vowel+inventories%2E+In+Proceedings+of+ACL%2C+pages+1182%E2%80%931192%2E)
23. Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. [Scalable modified Kneser-Ney language model estimation](https://md.research.elamur.ai/free/en/papers/heafield-2013-scalable-modified-kneser-ney-language/). In Proceedings of ACL, pages 690–696.
24. Ryan Cotterell, Sebastian J. Mielke, Jason Eisner, and Brian Roark. 2018. [Are all languages equally hard to language-model?](https://md.research.elamur.ai/free/en/papers/cotterell-2018-are-all-languages-equally-hard/) In Proceedings of NAACL-HLT.
25. Aurelie Herbelot and Marco Baroni. 2017. High-risk learning: acquiring new word vectors from tiny data. In Proceedings of EMNLP, pages 304–309. [find](https://scholar.google.com/scholar?q=Aurelie+Herbelot+and+Marco+Baroni%2E+2017%2E+High%2Drisk+learning%3A+acquiring+new+word+vectors+from+tiny+data%2E+In+Proceedings+of+EMNLP%2C+pages+304%E2%80%93309%2E)
26. Ryan Cotterell, Thomas Müller, Alexander Fraser, and Hinrich Schütze. 2015. Labeled morphological segmentation with semi-Markov models. In Proceedings of CoNLL, pages 164–174. [find](https://scholar.google.com/scholar?q=Ryan+Cotterell%2C+Thomas+M%C3%BCller%2C+Alexander+Fraser%2C+and+Hinrich+Sch%C3%BCtze%2E+2015%2E+Labeled+morphological+segmentation+with+semi%2DMarkov+models%2E+In+Proceedings+of+CoNLL%2C+pages+164%E2%80%93174%2E)
27. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of ACL, pages 328–339. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+Proceedings+of+ACL%2C+pages+328%E2%80%93339%2E)
28. Joachim Daiber. 2018. Typologically Robust Statistical Machine Translation. Ph.D. thesis, University of Amsterdam. [find](https://scholar.google.com/scholar?q=Joachim+Daiber%2E+2018%2E+Typologically+Robust+Statistical+Machine+Translation%2E+Ph%2ED%2E+thesis%2C+University+of+Amsterdam%2E)
29. Matthew S Dryer. 1989. Large linguistic areas and language sampling. Studies in Language. International Journal sponsored by the Foundation “Foundations of Language”, 13(2):257–292. [find](https://scholar.google.com/scholar?q=Matthew+S+Dryer%2E+1989%2E+Large+linguistic+areas+and+language+sampling%2E+Studies+in+Language%2E+International+Journal+sponsored+by+the+Foundation+%E2%80%9CFoundations+of+Language%E2%80%9D%2C+13%282%29%3A257%E2%80%93292%2E)
30. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. In Proceedings of ICML. [find](https://scholar.google.com/scholar?q=Rafal+Jozefowicz%2C+Oriol+Vinyals%2C+Mike+Schuster%2C+Noam+Shazeer%2C+and+Yonghui+Wu%2E+2016%2E+Exploring+the+limits+of+language+modeling%2E+In+Proceedings+of+ICML%2E)
31. Dan Jurafsky and James H. Martin. 2017. Speech and Language Processing, volume 3. Pearson. [find](https://scholar.google.com/scholar?q=Dan+Jurafsky+and+James+H%2E+Martin%2E+2017%2E+Speech+and+Language+Processing%2C+volume+3%2E+Pearson%2E)
32. Kazuya Kawakami, Chris Dyer, and Phil Blunsom. 2017. Learning to create and reuse words in open-vocabulary neural language modeling. In Proceedings of ACL, pages 1492–1502. [find](https://scholar.google.com/scholar?q=Kazuya+Kawakami%2C+Chris+Dyer%2C+and+Phil+Blunsom%2E+2017%2E+Learning+to+create+and+reuse+words+in+open%2Dvocabulary+neural+language+modeling%2E+In+Proceedings+of+ACL%2C+pages+1492%E2%80%931502%2E)
33. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2016. [Character-aware neural language models](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/). In Proceedings of AAAI, pages 2741– 2749.
34. Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for M-gram language modeling. In Proceedings of ICASSP, pages 181–184. [find](https://scholar.google.com/scholar?q=Reinhard+Kneser+and+Hermann+Ney%2E+1995%2E+Improved+backing%2Doff+for+M%2Dgram+language+modeling%2E+In+Proceedings+of+ICASSP%2C+pages+181%E2%80%93184%2E)
35. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of the 10th Machine Translation Summit, pages 79–86. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2E+2005%2E+Europarl%3A+A+parallel+corpus+for+statistical+machine+translation%2E+In+Proceedings+of+the+10th+Machine+Translation+Summit%2C+pages+79%E2%80%9386%2E)
36. Yann LeCun, Bernhard E. Boser, John S. Denker, Donnie Henderson, Richard E. Howard, Wayne E. Hubbard, and Lawrence D. Jackel. 1989. Handwritten digit recognition with a back-propagation network. In Proceedings of NIPS, pages 396–404. [find](https://scholar.google.com/scholar?q=Yann+LeCun%2C+Bernhard+E%2E+Boser%2C+John+S%2E+Denker%2C+Donnie+Henderson%2C+Richard+E%2E+Howard%2C+Wayne+E%2E+Hubbard%2C+and+Lawrence+D%2E+Jackel%2E+1989%2E+Handwritten+digit+recognition+with+a+back%2Dpropagation+network%2E+In+Pr)
37. Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2017. Recurrent additive networks. CoRR, abs/1705.07393. — [arXiv:1705.07393](https://arxiv.org/abs/1705.07393) [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Omer+Levy%2C+and+Luke+Zettlemoyer%2E+2017%2E+Recurrent+additive+networks%2E+CoRR%2C+abs%2F1705%2E07393%2E+%E2%80%94+%5BarXiv%3A1705%2E07393%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1705%2E07393%29)
38. Wang Ling, Tiago Luís, Luís Marujo, Ramón Fernández Astudillo, Silvio Amir, Chris Dyer, Alan W. Black, and Isabel Trancoso. 2015. [Finding function in form: Compositional character models for open vocabulary word representation](https://md.research.elamur.ai/free/en/papers/ling-2015-finding-function-form-compositional-character/). In Proceedings of EMNLP, pages 1520–1530.
39. Minh-Thang Luong and Christopher D. Manning. 2016. [Achieving open vocabulary neural machine translation with hybrid word-character models](https://md.research.elamur.ai/free/en/papers/luong-2016-achieving-open-vocabulary-neural-machine/). In Proceedings of ACL, pages 1054–1063.
40. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. [Learning word vectors for sentiment analysis](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/). In Proceedings of ACL, pages 142–150.
41. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. [Building a large annotated corpus of English: The Penn Treebank](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/). Computational Linguistics, 19(2):313–330.
42. John McWhorter. 2001. The world’s simplest grammars are Creole grammars. Linguistic Typology, 5(2):125–66. [find](https://scholar.google.com/scholar?q=John+McWhorter%2E+2001%2E+The+world%E2%80%99s+simplest+grammars+are+Creole+grammars%2E+Linguistic+Typology%2C+5%282%29%3A125%E2%80%9366%2E)
43. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proceedings of INTERSPEECH, pages 1045–1048. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Martin+Karafi%C3%A1t%2C+Lukas+Burget%2C+Jan+Cernocky%CC%80%2C+and+Sanjeev+Khudanpur%2E+2010%2E+Recurrent+neural+network+based+language+model%2E+In+Proceedings+of+INTERSPEECH%2C+pages+1045%E2%80%931048%2E)
44. Yasumasa Miyamoto and Kyunghyun Cho. 2016. Gated word-character recurrent language model. In Proceedings of EMNLP, pages 1992–1997. [find](https://scholar.google.com/scholar?q=Yasumasa+Miyamoto+and+Kyunghyun+Cho%2E+2016%2E+Gated+word%2Dcharacter+recurrent+language+model%2E+In+Proceedings+of+EMNLP%2C+pages+1992%E2%80%931997%2E)
45. Thomas Müller, Hinrich Schütze, and Helmut Schmid. 2012. A comparative investigation of morphological language modeling for the languages of the European Union. In Proceedings of NAACL-HLT, pages 386–395. [find](https://scholar.google.com/scholar?q=Thomas+M%C3%BCller%2C+Hinrich+Sch%C3%BCtze%2C+and+Helmut+Schmid%2E+2012%2E+A+comparative+investigation+of+morphological+language+modeling+for+the+languages+of+the+European+Union%2E+In+Proceedings+of+NAACL%2DHLT%2C+pages+386%E2%80%93)
46. Helen O’Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, and Anna Korhonen. 2016. Survey on the use of typological information in natural language processing. In Proceedings of COLING, pages 1297– 1308. [find](https://scholar.google.com/scholar?q=Helen+O%E2%80%99Horan%2C+Yevgeni+Berzak%2C+Ivan+Vulic%CC%81%2C+Roi+Reichart%2C+and+Anna+Korhonen%2E+2016%2E+Survey+on+the+use+of+typological+information+in+natural+language+processing%2E+In+Proceedings+of+COLING%2C+pages+1297%E2%80%93+13)
47. Michael Paul, Hirofumi Yamamoto, Eiichiro Sumita, and Satoshi Nakamura. 2009. On the importance of pivot language selection for statistical machine translation. In Proceedings of NAACL-HLT, pages 221–224. [find](https://scholar.google.com/scholar?q=Michael+Paul%2C+Hirofumi+Yamamoto%2C+Eiichiro+Sumita%2C+and+Satoshi+Nakamura%2E+2009%2E+On+the+importance+of+pivot+language+selection+for+statistical+machine+translation%2E+In+Proceedings+of+NAACL%2DHLT%2C+pages+221%E2%80%93)
48. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In Proceedings of NAACL-HLT, pages 2227–2237.
49. Yuval Pinter, Robert Guthrie, and Jacob Eisenstein. 2017. Mimicking word embeddings using subword RNNs. In Proceedings of EMNLP, pages 102–112. [find](https://scholar.google.com/scholar?q=Yuval+Pinter%2C+Robert+Guthrie%2C+and+Jacob+Eisenstein%2E+2017%2E+Mimicking+word+embeddings+using+subword+RNNs%2E+In+Proceedings+of+EMNLP%2C+pages+102%E2%80%93112%2E)
50. Frans Plank. 2017. Split morphology: How agglutination and flexion mix. Linguistic Typology, 21(2017):1–62. [find](https://scholar.google.com/scholar?q=Frans+Plank%2E+2017%2E+Split+morphology%3A+How+agglutination+and+flexion+mix%2E+Linguistic+Typology%2C+21%282017%29%3A1%E2%80%9362%2E)
51. Jay M. Ponte and W. Bruce Croft. 1998. [A language modeling approach to information retrieval](https://md.research.elamur.ai/free/en/papers/ponte-1998-language-modeling-approach-information/). In Proceedings of SIGIR, pages 275–281.
52. Edoardo Maria Ponti, Helen O’Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. 2018. [Modeling language variation and universals: A survey on typological linguistics for natural language processing](https://md.research.elamur.ai/free/en/papers/ponti-2019-modeling-language-variation-universals-survey/). CoRR, abs/1807.00914.
53. Edoardo Maria Ponti, Ivan Vulić, and Anna Korhonen. 2017. Decoding sentiment from distributed representations of sentences. In Proceedings of *SEM, pages 22–32. [find](https://scholar.google.com/scholar?q=Edoardo+Maria+Ponti%2C+Ivan+Vulic%CC%81%2C+and+Anna+Korhonen%2E+2017%2E+Decoding+sentiment+from+distributed+representations+of+sentences%2E+In+Proceedings+of+%2ASEM%2C+pages+22%E2%80%9332%2E)
54. Ofir Press and Lior Wolf. 2017. [Using the output embedding to improve language models](https://md.research.elamur.ai/free/en/papers/press-2017-output-embedding-improve-language-models/). In Proceedings of EACL, pages 157–163.
55. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of EMNLP, pages 379–389. [find](https://scholar.google.com/scholar?q=Alexander+M%2E+Rush%2C+Sumit+Chopra%2C+and+Jason+Weston%2E+2015%2E+A+neural+attention+model+for+abstractive+sentence+summarization%2E+In+Proceedings+of+EMNLP%2C+pages+379%E2%80%93389%2E)
56. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. [Highway networks](https://md.research.elamur.ai/free/en/papers/srivastava-2015-highway-networks/). In Proceedings of the ICML Deep Learning Workshop.
57. Martin Sundermeyer, Hermann Ney, and Ralf Schluter. 2015. From feedforward to recurrent LSTM neural networks for language modeling. IEEE Transactions on Audio, Speech and Language Processing, 23(3):517–529. [find](https://scholar.google.com/scholar?q=Martin+Sundermeyer%2C+Hermann+Ney%2C+and+Ralf+Schluter%2E+2015%2E+From+feedforward+to+recurrent+LSTM+neural+networks+for+language+modeling%2E+IEEE+Transactions+on+Audio%2C+Speech+and+Language+Processing%2C+23%283%29%3A51)
58. Clara Vania and Adam Lopez. 2017. From characters to words to in between: Do we capture morphology? In Proceedings of ACL, pages 2016–2027. [find](https://scholar.google.com/scholar?q=Clara+Vania+and+Adam+Lopez%2E+2017%2E+From+characters+to+words+to+in+between%3A+Do+we+capture+morphology%3F+In+Proceedings+of+ACL%2C+pages+2016%E2%80%932027%2E)
59. Ivan Vulić, Nikola Mrkšić, Roi Reichart, Diarmuid Ó Séaghdha, Steve Young, and Anna Korhonen. 2017. Morph-fitting: Fine-tuning word vector spaces with simple language-specific rules. In Proceedings of ACL, pages 56–68. [find](https://scholar.google.com/scholar?q=Ivan+Vulic%CC%81%2C+Nikola+Mrk%C5%A1ic%CC%81%2C+Roi+Reichart%2C+Diarmuid+%C3%93+S%C3%A9aghdha%2C+Steve+Young%2C+and+Anna+Korhonen%2E+2017%2E+Morph%2Dfitting%3A+Fine%2Dtuning+word+vector+spaces+with+simple+language%2Dspecific+rules%2E+In+Proceedings+)
60. Tian Wang and Kyunghyun Cho. 2016. Larger-context language modelling with recurrent neural network. In Proceedings of ACL, pages 1319–1329. [find](https://scholar.google.com/scholar?q=Tian+Wang+and+Kyunghyun+Cho%2E+2016%2E+Larger%2Dcontext+language+modelling+with+recurrent+neural+network%2E+In+Proceedings+of+ACL%2C+pages+1319%E2%80%931329%2E)
61. Hamed Zamani and W. Bruce Croft. 2016. Embedding-based query language models. In Proceedings of IC-TIR, pages 147–156. [find](https://scholar.google.com/scholar?q=Hamed+Zamani+and+W%2E+Bruce+Croft%2E+2016%2E+Embedding%2Dbased+query+language+models%2E+In+Proceedings+of+IC%2DTIR%2C+pages+147%E2%80%93156%2E)
62. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2015. [Recurrent neural network regularization](https://md.research.elamur.ai/free/en/papers/zaremba-2014-recurrent-neural-network-regularization/). In Proceedings of ICLR (Conference Papers).
63. George Kingsley Zipf. 1949. Human behavior and the principle of least effort: An introduction to human ecology. [find](https://scholar.google.com/scholar?q=George+Kingsley+Zipf%2E+1949%2E+Human+behavior+and+the+principle+of+least+effort%3A+An+introduction+to+human+ecology%2E)

*Updated 24 October 2026*