---
title: "What Do Language Representations Really Represent? (2019)"
description: "Multilingual neural models can learn a vector for each language. 17 references, cited by 40 papers here."
authors: ["Johannes Bjerva", "Robert Östling", "Maria Han Veiga", "Jörg Tiedemann", "Isabelle Augenstein"]
year: 2019
venue: "Computational Linguistics"
source: "https://aclanthology.org/J19-2006/"
doi: "10.1162/coli_a_00351"
licence: "CC BY-NC-ND 4.0 (Computational Linguistics, MIT Press)"
licence_url: "https://creativecommons.org/licenses/by-nc-nd/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bjerva-2019-what-do-language-representations-really/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bjerva-2019-what-do-language-representations-really/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/6/) / What Do Language Representations Really Represent?

# What Do Language Representations Really Represent?

**Authors:** [Johannes Bjerva](https://md.research.elamur.ai/free/en/authors/johannes-bjerva/), [Robert Östling](https://md.research.elamur.ai/free/en/authors/robert-ostling/), Maria Han Veiga, [Jörg Tiedemann](https://md.research.elamur.ai/free/en/authors/jorg-tiedemann/) and [Isabelle Augenstein](https://md.research.elamur.ai/free/en/authors/isabelle-augenstein/) · **Year:** 2019 · **Venue:** [Computational Linguistics](https://md.research.elamur.ai/free/en/venues/cl/) ([Computational Linguistics 2019](https://md.research.elamur.ai/free/en/venues/cl/2019/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY-NC-ND 4.0 (Computational Linguistics, MIT Press)](https://creativecommons.org/licenses/by-nc-nd/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/J19-2006/) · [PDF](https://aclanthology.org/J19-2006.pdf) · [DOI 10.1162/coli\_a\_00351](https://doi.org/10.1162/coli_a_00351) · [OpenAlex](https://openalex.org/W2908834655)

**Citations:** 8 citations (OpenAlex, 2026-10-14) · [40](https://md.research.elamur.ai/free/en/papers/bjerva-2019-what-do-language-representations-really/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Multilingual neural models can learn a vector for each language. Earlier work, notably Rabinovich, Ordan, and Wintner (2017), judged such vectors mainly by how well they recover family trees. Bjerva et al. ask what kind of similarity these vectors actually capture. They use English translations from the Europarl corpus, each labelled with its original EU source language. On this data they train a 2-layer LSTM language model that receives a language vector at every time step. Inputs come at several abstraction levels: raw text, function words, POS, phrase structure and dependency relations. The learned language distances are compared with three references: genetic distance, geographical distance between speaker communities, and a new structural distance. Spearman correlations show the vectors align best with structural similarity, not genealogy. A causal analysis supports this and suggests genetic relatedness acts as a confound. The study warns against validating language embeddings with one convenient benchmark and shows how NLP and typology can inform each other.

## Key points
- Data: the same Europarl-derived English translations used by Rabinovich et al. (2017), with the faint source-language signal acting as the training cue.
- Five representation types are compared: Raw, Func, POS, Phrase and Deprel. Genetic distances are taken from Serva and Petroni (2008) phylogenetic trees.
- The new structural measure profiles each language by dependency relations together with head and dependent POS. This yields 8,607-dimensional normalized vectors, richer than the approach of Chen and Gerdes (2017).
- Ward clustering on the structural vectors departs from genealogy in several ways. Romanian is set apart from the other Romance languages, English groups with North Germanic, and Latvian/Lithuanian cluster with Estonian/Finnish.
- In the causal network built with IC* (p \< 0.001), the only link between similarity measures and representations connects structural similarity to the raw model. No link reaches genetic similarity.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Copenhagen](https://md.research.elamur.ai/free/en/universities/university-of-copenhagen/) — University, Copenhagen, Denmark · 115 works · 15,292 citations
- [University of Zurich](https://md.research.elamur.ai/free/en/universities/university-of-zurich/) — University, Zurich, Switzerland · 78 works · 12,260 citations
- [University of Helsinki](https://md.research.elamur.ai/free/en/universities/university-of-helsinki/) — University, Helsinki, Finland · 50 works · 5,318 citations
- [Stockholm University](https://md.research.elamur.ai/free/en/universities/stockholm-university/) — University, Stockholm, Sweden · 21 works · 1,832 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 870 works

## Cite

Cite as: Johannes Bjerva, Robert Östling, Maria Han Veiga, Jörg Tiedemann and Isabelle Augenstein. 2019. What Do Language Representations Really Represent?. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{bjerva2019what,
  title = {{What Do Language Representations Really Represent?}},
  author = {Johannes Bjerva, Robert Östling, Maria Han Veiga, Jörg Tiedemann and Isabelle Augenstein},
  journal = {Computational Linguistics},
  year = {2019},
  doi = {10.1162/coli\_a\_00351},
  url = {https://doi.org/10.1162/coli_a_00351}
}
```

- [← previous A Neural Entity Coreference Resolution Review](https://md.research.elamur.ai/free/en/papers/stylianou-2019-neural-entity-coreference-resolution-review/)
- [next → “Caption” as a Coherence Relation: Evidence and Implications](https://md.research.elamur.ai/free/en/papers/alikhani-2019-caption-as-coherence-relation-evidence/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/6/)

## References

17 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Malaviya, Chaitanya, Graham Neubig, and Patrick Littell. 2017. [Learning language representations for typology prediction](https://md.research.elamur.ai/free/en/papers/malaviya-2017-learning-language-representations-typology-prediction/). In EMNLP, pages 2519–2525.
2. Nivre, Joakim, et al. 2017. Universal Dependencies 2.1. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Charles University. Östling, Robert. 2015. Word order typology through multilingual word alignment. In ACL-IJCNLP, pages 205–211. Östling, Robert and Jo… [find](https://scholar.google.com/scholar?q=Nivre%2C+Joakim%2C+et+al%2E+2017%2E+Universal+Dependencies+2%2E1%2E+LINDAT%2FCLARIN+digital+library+at+the+Institute+of+Formal+and+Applied+Linguistics+%28U%CC%81FAL%29%2C+Charles+University%2E+O%CC%88stling%2C+Robert%2E+2015%2E+Word+order)
3. Pearl, Judea. 2009. Causality, Cambridge University Press. Ponti, Edoardo Maria, Helen O’Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. 2018. Modeling language variation and universals: A survey on typological linguistics for natural language… — [arXiv:1807.00914](https://arxiv.org/abs/1807.00914) [find](https://scholar.google.com/scholar?q=Pearl%2C+Judea%2E+2009%2E+Causality%2C+Cambridge+University+Press%2E+Ponti%2C+Edoardo+Maria%2C+Helen+O%E2%80%99Horan%2C+Yevgeni+Berzak%2C+Ivan+Vulic%CC%81%2C+Roi+Reichart%2C+Thierry+Poibeau%2C+Ekaterina+Shutova%2C+and+Anna+Korhonen%2E+2018%2E+)
4. Rabinovich, Ella, Noam Ordan, and Shuly Wintner. 2017. Found in translation: Reconstructing phylogenetic language trees from translations. In ACL. Roberts, Seán and James Winters. 2013. Linguistic diversity and traffic accidents: Lessons from statistical studies of cultural traits. PloS One, 8(8):e… [find](https://scholar.google.com/scholar?q=Rabinovich%2C+Ella%2C+Noam+Ordan%2C+and+Shuly+Wintner%2E+2017%2E+Found+in+translation%3A+Reconstructing+phylogenetic+language+trees+from+translations%2E+In+ACL%2E+Roberts%2C+Sea%CC%81n+and+James+Winters%2E+2013%2E+Linguistic+di)
5. Serva, Maurizio and Filippo Petroni. 2008. Indo-European languages tree by Levenshtein distance. EPL, 81(6):68005. Straka, Milan, Jan Hajic, and Jana Straková. 2016. UD-Pipe: Trainable pipeline for processing CoNLL-U files performing tokenization, morphological analysis, POS tagging and parsing. In… [find](https://scholar.google.com/scholar?q=Serva%2C+Maurizio+and+Filippo+Petroni%2E+2008%2E+Indo%2DEuropean+languages+tree+by+Levenshtein+distance%2E+EPL%2C+81%286%29%3A68005%2E+Straka%2C+Milan%2C+Jan+Hajic%2C+and+Jana+Strakova%CC%81%2E+2016%2E+UD%2DPipe%3A+Trainable+pipeline+for+p)
6. Velupillai, Viveka. 2012. An Introduction to Linguistic Typology, John Benjamins Publishing. [find](https://scholar.google.com/scholar?q=Velupillai%2C+Viveka%2E+2012%2E+An+Introduction+to+Linguistic+Typology%2C+John+Benjamins+Publishing%2E)
7. Verma, Thomas and Judea Pearl. 1992. An algorithm for deciding if a set of observed independencies has a causal explanation. In Proceedings of the 8th Conference on Uncertainty and Artificial Intelligence, pages 323–330. [find](https://scholar.google.com/scholar?q=Verma%2C+Thomas+and+Judea+Pearl%2E+1992%2E+An+algorithm+for+deciding+if+a+set+of+observed+independencies+has+a+causal+explanation%2E+In+Proceedings+of+the+8th+Conference+on+Uncertainty+and+Artificial+Intellig)
8. Vinyals, Oriol, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In NIPS. [find](https://scholar.google.com/scholar?q=Vinyals%2C+Oriol%2C+Lukasz+Kaiser%2C+Terry+Koo%2C+Slav+Petrov%2C+Ilya+Sutskever%2C+and+Geoffrey+Hinton%2E+2015%2E+Grammar+as+a+foreign+language%2E+In+NIPS%2E)
9. Ammar, Waleed, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah Smith. 2016. [Many languages, one parser](https://md.research.elamur.ai/free/en/papers/ammar-2016-many-languages-one-parser/). TACL, 4:431–444.
10. Bjerva, Johannes and Isabelle Augenstein. 2018. From phonology to syntax: Unsupervised linguistic typology at different levels with language embeddings. In NAACL-HLT. [find](https://scholar.google.com/scholar?q=Bjerva%2C+Johannes+and+Isabelle+Augenstein%2E+2018%2E+From+phonology+to+syntax%3A+Unsupervised+linguistic+typology+at+different+levels+with+language+embeddings%2E+In+NAACL%2DHLT%2E)
11. Chen, Xinying and Kim Gerdes. 2017. Classifying languages by dependency structure. Typologies of delexicalized universal dependency treebanks. In DepLing, pages 54–63. [find](https://scholar.google.com/scholar?q=Chen%2C+Xinying+and+Kim+Gerdes%2E+2017%2E+Classifying+languages+by+dependency+structure%2E+Typologies+of+delexicalized+universal+dependency+treebanks%2E+In+DepLing%2C+pages+54%E2%80%9363%2E)
12. Cotterell, Ryan and Jason Eisner. 2017. Probabilistic typology: Deep generative models of vowel inventories. In ACL. [find](https://scholar.google.com/scholar?q=Cotterell%2C+Ryan+and+Jason+Eisner%2E+2017%2E+Probabilistic+typology%3A+Deep+generative+models+of+vowel+inventories%2E+In+ACL%2E)
13. Dunn, Michael, Simon J. Greenhill, Stephen C. Levinson, and Russell D. Gray. 2011. Evolved structure of language shows lineage-specific trends in word-order universals. Nature, 473(7345):79–82. [find](https://scholar.google.com/scholar?q=Dunn%2C+Michael%2C+Simon+J%2E+Greenhill%2C+Stephen+C%2E+Levinson%2C+and+Russell+D%2E+Gray%2E+2011%2E+Evolved+structure+of+language+shows+lineage%2Dspecific+trends+in+word%2Dorder+universals%2E+Nature%2C+473%287345%29%3A79%E2%80%9382%2E)
14. Gellerstam, Martin. 1986. Translationese in Swedish novels translated from English. Translation Studies in Scandinavia, 1:88–95. [find](https://scholar.google.com/scholar?q=Gellerstam%2C+Martin%2E+1986%2E+Translationese+in+Swedish+novels+translated+from+English%2E+Translation+Studies+in+Scandinavia%2C+1%3A88%E2%80%9395%2E)
15. Gerz, Daniela, Ivan Vulic, Edoardo Maria Ponti, Roi Reichart, and Anna Korhonen. 2018. [On the relation between linguistic typology and (limitations of) multilingual language modeling](https://md.research.elamur.ai/free/en/papers/gerz-2018-relation-between-linguistic-typology-limitations/). In EMNLP. Hammarström, Harald, Robert Forkel, and Martin Haspelmath. 2017. Glottolog 3.0. Jena: Max Planck Institut…
16. Haspelmath, Martin. 2001. Language Typology and Language Universals: An International Handbook, volume 20, Walter de Gruyter. Johnson, Melvin, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeff… [find](https://scholar.google.com/scholar?q=Haspelmath%2C+Martin%2E+2001%2E+Language+Typology+and+Language+Universals%3A+An+International+Handbook%2C+volume+20%2C+Walter+de+Gruyter%2E+Johnson%2C+Melvin%2C+Mike+Schuster%2C+Quoc+V%2E+Le%2C+Maxim+Krikun%2C+Yonghui+Wu%2C+Zhif)
17. Koehn, Philipp. 2005. Europarl: A parallel corpus for statistical machine translation. In MT Summit X. [find](https://scholar.google.com/scholar?q=Koehn%2C+Philipp%2E+2005%2E+Europarl%3A+A+parallel+corpus+for+statistical+machine+translation%2E+In+MT+Summit+X%2E)

*Updated 29 October 2026*