---
title: "The Geometry of Multilingual Language Model Representations"
description: "Chang, Tu and Bergen study this in XLM-R. 30 references, cited by 55 papers here. Among them: 27."
authors: ["Tyler A. Chang", "Zhuowen Tu", "Benjamin K. Bergen"]
year: 2022
venue: "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2022.emnlp-main.9/"
doi: "10.18653/v1/2022.emnlp-main.9"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chang-2022-geometry-multilingual-language-model-representations/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chang-2022-geometry-multilingual-language-model-representations/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / The Geometry of Multilingual Language Model Representations

# The Geometry of Multilingual Language Model Representations

**Authors:** [Tyler A. Chang](https://md.research.elamur.ai/free/en/authors/tyler-a-chang/), [Zhuowen Tu](https://md.research.elamur.ai/free/en/authors/zhuowen-tu/) and [Benjamin K. Bergen](https://md.research.elamur.ai/free/en/authors/benjamin-bergen/) · **Year:** 2022 · **Venue:** [Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2022](https://md.research.elamur.ai/free/en/venues/emnlp/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.emnlp-main.9/) · [PDF](https://aclanthology.org/2022.emnlp-main.9.pdf) · [DOI 10.18653/v1/2022.emnlp-main.9](https://doi.org/10.18653/v1/2022.emnlp-main.9) · [OpenAlex](https://openalex.org/W4385573436)

**Citations:** 27 citations (OpenAlex, 2026-10-14) · [55](https://md.research.elamur.ai/free/en/papers/chang-2022-geometry-multilingual-language-model-representations/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Multilingual language models are often described as mapping all languages into one common space. They also keep language-specific signals, and it has been unclear how both coexist geometrically. Chang, Tu and Bergen study this in XLM-R. They feed it OSCAR web text in 88 languages and use SVD to build a mean-shifted (affine) subspace for each language at every layer. They test these subspaces in three ways: by how projecting onto them affects masked-token perplexity, by direct distance measures between subspaces, and by low-dimensional visualizations.

Once language means are removed, the subspaces of different languages turn out to be largely similar, especially in middle layers. The differences between the means lie along language-sensitive directions. These directions carry information such as vocabulary and group languages by family, and shifting representations by language means is enough to make the model predict tokens in another language. Separate language-neutral directions encode token position and part-of-speech, and they are roughly orthogonal to the language-sensitive ones. This geometry helps explain cross-lingual transfer. It may also guide which axes to align when adapting models across languages.

## Key points
- Each language subspace was derived from about 262K contextual token vectors (512 OSCAR sequences of 512 tokens each). Its size was set to cover 90% of variance, giving a median of 335 dimensions out of 768.
- Projecting onto a language's own subspace barely raised perplexity. Projecting onto another language's subspace raised it sharply. Re-centering that subspace on the original language's mean recovered most of the loss, especially in middle layers.
- In layer eight, the Spanish subspace and the Chinese subspace are separated by about as much as a subspace rotated by roughly two degrees or scaled by 1.53x.
- Projections onto language-neutral axes show token position as spirals, curves and even a near-perfect torus, with part-of-speech clusters on separate axes. These structures stay fairly stable across middle layers.
- The authors list several limitations:
  - The language sample is skewed: 52 of the 88 languages are Indo-European, with only six Austronesian, two African and no Native American languages.
  - Only XLM-R was analyzed.
  - Code is released at github.com/tylerachang/multilingual-geometry.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of California San Diego](https://md.research.elamur.ai/free/en/universities/university-of-california-san-diego/) — University, San Diego, United States · 34 works · 51,167 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 465 works

## Cite

Cite as: Tyler A. Chang, Zhuowen Tu and Benjamin K. Bergen. 2022. The Geometry of Multilingual Language Model Representations. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chang2022geometry,
  title = {{The Geometry of Multilingual Language Model Representations}},
  author = {Tyler A. Chang, Zhuowen Tu and Benjamin K. Bergen},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.9},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.9}
}
```

- [← previous Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering](https://md.research.elamur.ai/free/en/papers/liu-2022-rainier-reinforced-knowledge-introspector-commonsense/)
- [next → UniMorph 4.0: Universal Morphology](https://md.research.elamur.ai/free/en/papers/batsuren-2022-unimorph-4/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

30 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Harald Hammarström, Robert Forkel, Martin Haspelmath, and Sebastian Bank. 2021. Glottolog 4.5. Max Planck Institute for Evolutionary Anthropology, Leipzig. [find](https://scholar.google.com/scholar?q=Harald+Hammarstr%C3%B6m%2C+Robert+Forkel%2C+Martin+Haspelmath%2C+and+Sebastian+Bank%2E+2021%2E+Glottolog+4%2E5%2E+Max+Planck+Institute+for+Evolutionary+Anthropology%2C+Leipzig%2E)
2. Julien Abadji, Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. 2021. Ungoliant: An optimized pipeline for the generation of a very large-scale multilingual web corpus. In Proceedings of the Workshop on Challenges in the Management of Large Corpora, pages 1–9. [find](https://scholar.google.com/scholar?q=Julien+Abadji%2C+Pedro+Javier+Ortiz+Su%C3%A1rez%2C+Laurent+Romary%2C+and+Beno%C3%AEt+Sagot%2E+2021%2E+Ungoliant%3A+An+optimized+pipeline+for+the+generation+of+a+very+large%2Dscale+multilingual+web+corpus%2E+In+Proceedings+of+t)
3. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pages 4411–4421. [find](https://scholar.google.com/scholar?q=Junjie+Hu%2C+Sebastian+Ruder%2C+Aditya+Siddhant%2C+Graham+Neubig%2C+Orhan+Firat%2C+and+Melvin+Johnson%2E+2020%2E+XTREME%3A+A+massively+multilingual+multi%2Dtask+benchmark+for+evaluating+cross%2Dlingual+generalisation%2E+In)
4. Silvére Bonnabel and Rodolphe Sepulchre. 2009. Riemannian metric and geometric mean for positive semidefinite matrices of fixed rank. SIAM Journal on Matrix Analysis and Applications, 31:1055–1070. [find](https://scholar.google.com/scholar?q=Silv%C3%A9re+Bonnabel+and+Rodolphe+Sepulchre%2E+2009%2E+Riemannian+metric+and+geometric+mean+for+positive+semidefinite+matrices+of+fixed+rank%2E+SIAM+Journal+on+Matrix+Analysis+and+Applications%2C+31%3A1055%E2%80%931070%2E)
5. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. [What does BERT learn about the structure of language?](https://md.research.elamur.ai/free/en/papers/jawahar-2019-what-does-bert-learn-about/) In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
6. Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. 2021. Isotropy in the contextual embedding space: Clusters and manifolds. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Xingyu+Cai%2C+Jiaji+Huang%2C+Yuchen+Bian%2C+and+Kenneth+Church%2E+2021%2E+Isotropy+in+the+contextual+embedding+space%3A+Clusters+and+manifolds%2E+In+International+Conference+on+Learning+Representations%2E)
7. Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Multilingual alignment of contextual word representations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Steven+Cao%2C+Nikita+Kitaev%2C+and+Dan+Klein%2E+2020%2E+Multilingual+alignment+of+contextual+word+representations%2E+In+International+Conference+on+Learning+Representations%2E)
8. Ethan A. Chi, John Hewitt, and Christopher D. Manning. 2020. Finding universal grammatical relations in multilingual BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5564–5577, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ethan+A%2E+Chi%2C+John+Hewitt%2C+and+Christopher+D%2E+Manning%2E+2020%2E+Finding+universal+grammatical+relations+in+multilingual+BERT%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computationa)
9. Rochelle Choenni and Ekaterina Shutova. 2020. What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological properties. arXiv, abs/2009.12862. — [arXiv:2009.12862](https://arxiv.org/abs/2009.12862) [find](https://scholar.google.com/scholar?q=Rochelle+Choenni+and+Ekaterina+Shutova%2E+2020%2E+What+does+it+mean+to+be+language%2Dagnostic%3F+Probing+multilingual+sentence+encoders+for+typological+properties%2E+arXiv%2C+abs%2F2009%2E12862%2E+%E2%80%94+%5BarXiv%3A2009%2E12862%5D%28)
10. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020a. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association f… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020a%2E+Unsupervised+cross%2Dling)
11. Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020b. [Emerging cross-lingual structure in pretrained language models](https://md.research.elamur.ai/free/en/papers/conneau-2020-emerging-cross-lingual-structure-pretrained/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6022–6034, Online. Association for Computational Ling…
12. Sneha Kudugunta, Ankur Bapna, Isaac Caswell, and Orhan Firat. 2019. [Investigating multilingual NMT representations at scale](https://md.research.elamur.ai/free/en/papers/kudugunta-2019-investigating-multilingual-nmt-representations-scale/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCN…
13. Saurabh Kulshreshtha, Jose Luis Redondo Garcia, and Ching-Yun Chang. 2020. Cross-lingual alignment methods for multilingual BERT: A comparative study. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 933–942, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Saurabh+Kulshreshtha%2C+Jose+Luis+Redondo+Garcia%2C+and+Ching%2DYun+Chang%2E+2020%2E+Cross%2Dlingual+alignment+methods+for+multilingual+BERT%3A+A+comparative+study%2E+In+Findings+of+the+Association+for+Computational+)
14. Sheng Liang, Philipp Dufter, and Hinrich Schütze. 2021. Locating language-specific information in contextualized embeddings. arXiv, abs/2109.08040. — [arXiv:2109.08040](https://arxiv.org/abs/2109.08040) [find](https://scholar.google.com/scholar?q=Sheng+Liang%2C+Philipp+Dufter%2C+and+Hinrich+Sch%C3%BCtze%2E+2021%2E+Locating+language%2Dspecific+information+in+contextualized+embeddings%2E+arXiv%2C+abs%2F2109%2E08040%2E+%E2%80%94+%5BarXiv%3A2109%2E08040%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2109%2E0804)
15. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Mi… [find](https://scholar.google.com/scholar?q=Yaobo+Liang%2C+Nan+Duan%2C+Yeyun+Gong%2C+Ning+Wu%2C+Fenfei+Guo%2C+Weizhen+Qi%2C+Ming+Gong%2C+Linjun+Shou%2C+Daxin+Jiang%2C+Guihong+Cao%2C+Xiaodong+Fan%2C+Ruofei+Zhang%2C+Rahul+Agrawal%2C+Edward+Cui%2C+Sining+Wei%2C+Taroon+Bharti%2C+)
16. Jindřich Libovický, Rudolf Rosa, and Alexander Fraser. 2020. [On the language neutrality of pre-trained multilingual representations](https://md.research.elamur.ai/free/en/papers/libovicky-2020-language-neutrality-pre-trained-multilingual/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1663–1674, Online. Association for Computational Linguistics.
17. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyano… — [Few-shot Learning with Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+C)
19. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. [Universal Dependencies v2: An evergrowing multilingual treebank collection](https://md.research.elamur.ai/free/en/papers/nivre-2020-universal-dependencies-v2/). In Proceedings of the 12th Language Resources and Evaluat…
20. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. [How multilingual is multilingual BERT?](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/) In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001, Florence, Italy. Association for Computational Linguistics.
21. Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017. SVCCA: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Maithra+Raghu%2C+Justin+Gilmer%2C+Jason+Yosinski%2C+and+Jascha+Sohl%2DDickstein%2E+2017%2E+SVCCA%3A+Singular+vector+canonical+correlation+analysis+for+deep+learning+dynamics+and+interpretability%2E+In+Conference+on+N)
22. Sara Rajaee and Mohammad Taher Pilehvar. 2022. An isotropy analysis in the multilingual BERT embedding space. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1309–1316, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sara+Rajaee+and+Mohammad+Taher+Pilehvar%2E+2022%2E+An+isotropy+analysis+in+the+multilingual+BERT+embedding+space%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2022%2C+pages+1309%E2%80%931316%2C+D)
23. Taraka Rama, Lisa Beinborn, and Steffen Eger. 2020. Probing multilingual BERT for genetic and typological signals. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1214–1228, Barcelona, Spain (Online). International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Taraka+Rama%2C+Lisa+Beinborn%2C+and+Steffen+Eger%2E+2020%2E+Probing+multilingual+BERT+for+genetic+and+typological+signals%2E+In+Proceedings+of+the+28th+International+Conference+on+Computational+Linguistics%2C+pag)
24. Naomi Saphra and Adam Lopez. 2019. [Understanding learning dynamics of language models with SVCCA](https://md.research.elamur.ai/free/en/papers/saphra-2019-understanding-learning-dynamics-language-models/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3257–3267, Minn…
25. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. [BERT rediscovers the classical NLP pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics.
26. Elena Voita, Rico Sennrich, and Ivan Titov. 2019. [The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives](https://md.research.elamur.ai/free/en/papers/voita-2019-bottom-up-evolution-representations-transformer/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Join…
27. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
28. Shijie Wu and Mark Dredze. 2020. [Are all languages created equal in multilingual BERT?](https://md.research.elamur.ai/free/en/papers/wu-2020-are-all-languages-created-equal/) In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120–130, Online. Association for Computational Linguistics.
29. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. [mT5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling…
30. Wei Zhao, Steffen Eger, Johannes Bjerva, and Isabelle Augenstein. 2021. Inducing language-agnostic multilingual representations. In Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics, pages 229–240, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wei+Zhao%2C+Steffen+Eger%2C+Johannes+Bjerva%2C+and+Isabelle+Augenstein%2E+2021%2E+Inducing+language%2Dagnostic+multilingual+representations%2E+In+Proceedings+of+%2ASEM+2021%3A+The+Tenth+Joint+Conference+on+Lexical+and+)

*Updated 24 October 2026*