---
title: "Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing — paper by Wang et al. (2019)"
description: "The authors propose Cross-Lingual BERT Transformation (CLBT), an off-line method that maps contextualized… 35 references, cited by 58 papers here."
authors: ["Yuxuan Wang", "Wanxiang Che", "Jiang Guo", "Yijia Liu", "Ting Liu"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1575/"
doi: "10.18653/v1/d19-1575"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2019-cross-lingual-bert-transformation-zero/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2019-cross-lingual-bert-transformation-zero/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing

# Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing

**Authors:** [Yuxuan Wang](https://md.research.elamur.ai/free/en/authors/yuxuan-wang/), [Wanxiang Che](https://md.research.elamur.ai/free/en/authors/wanxiang-che/), [Jiang Guo](https://md.research.elamur.ai/free/en/authors/jiang-guo/), [Yijia Liu](https://md.research.elamur.ai/free/en/authors/yijia-liu/) and [Ting Liu](https://md.research.elamur.ai/free/en/authors/ting-liu/) · **Year:** 2019 · **Venue:** [Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2019](https://md.research.elamur.ai/free/en/venues/emnlp/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1575/) · [PDF](https://aclanthology.org/D19-1575.pdf) · [DOI 10.18653/v1/d19-1575](https://doi.org/10.18653/v1/d19-1575) · [OpenAlex](https://openalex.org/W2970850076)

**Citations:** 95 citations (OpenAlex, 2026-10-14) · [58](https://md.research.elamur.ai/free/en/papers/wang-2019-cross-lingual-bert-transformation-zero/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Zero-shot cross-lingual dependency parsing needs lexical features that carry over between languages. Earlier transfer parsers relied on static cross-lingual embeddings, and training cross-lingual language models such as XLM from scratch requires large parallel corpora and heavy compute. The authors propose Cross-Lingual BERT Transformation (CLBT), an off-line method that maps contextualized embeddings of target languages from multilingual BERT (mBERT) into the space of English BERT (enBERT). Tokens in 10,000 Europarl sentence pairs per language are aligned with fast_align at the WordPiece level. A linear map is then fitted on the aligned token vectors, either as a closed-form orthogonal SVD solution or through unconstrained gradient descent (GD). The mapped embeddings feed a Biaffine graph-based parser that is trained only on the English UD v2.2 treebank and applied directly to 17 target languages. CLBT raises averaged LAS by 2.91 points over the previous static-embedding state of the art. It also rivals XLM on shared languages while using a small fraction of its parallel data. This makes adding a new language pair cheap, typically within hours.

## Key points
- **Setup:** English is the source and 17 languages are targets, all drawn from UD v2.2 and limited to languages with available Europarl data. The baselines are FT-SVD (FastText plus SVD, the previous state of the art), raw mBERT, and XLM (XNLI-15 model).
- **Main results:** Averaged target LAS is 60.63 for FT-SVD, 60.53 for mBERT, 63.09 for CLBT (SVD), and 63.54 for CLBT (GD). Combining CLBT with FT-SVD adds a further 0.33 (GD) or 0.51 (SVD) LAS.
- **Comparison with XLM:** CLBT beats XLM on French, Spanish and Bulgarian but trails it on German. XLM used 0.2M–13.1M parallel sentences per language, whereas CLBT used 10,000 (about 0.4M tokens).
- **Training data size:** With SVD, most of the 13 tested languages peak at about 5,000 parallel sentences. For Germanic and Romance languages, just 100 sentences give near-comparable LAS.
- **Analysis and code:** A t-SNE analysis on SENSEVAL2 data for *nature*/*naturaleza* shows that the transformation brings same-sense tokens across languages closer together. Code is available at github.com/WangYuxuan93/CLBT.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) — University, Cambridge, United States · 159 works · 65,157 citations
- [Harbin Institute of Technology](https://md.research.elamur.ai/free/en/universities/harbin-institute-of-technology/) — University, Harbin, China · 76 works · 27,918 citations
- [MIT Computer Science and Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/mit-computer-science-and-artificial-intelligence-laboratory/) — Research institute, Cambridge, United States · 54 works · 29,441 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 467 works · [🌳 Dependency parsing](https://md.research.elamur.ai/free/en/topics/dependency-parsing/) · 120 works

## Cite

Cite as: Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu and Ting Liu. 2019. Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2019cross,
  title = {{Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing}},
  author = {Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu and Ting Liu},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1575},
  url = {https://doi.org/10.18653/v1/d19-1575}
}
```

- [← previous Generating Sentences from Disentangled Syntactic and Semantic Spaces](https://md.research.elamur.ai/free/en/papers/bao-2019-generating-sentences-disentangled-syntactic-semantic/)
- [next → LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://md.research.elamur.ai/free/en/papers/zen-2019-libritts-corpus-derived-librispeech-text/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

35 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Wasi Uddin Ahmad, Zhisong Zhang, Xuezhe Ma, Eduard H. Hovy, Kai-Wei Chang, and Nanyun Peng. 2018. Near or far, wide range zero-shot cross-lingual dependency parsing. CoRR, abs/1811.00570. [find](https://scholar.google.com/scholar?q=Wasi+Uddin+Ahmad%2C+Zhisong+Zhang%2C+Xuezhe+Ma%2C+Eduard+H%2E+Hovy%2C+Kai%2DWei+Chang%2C+and+Nanyun+Peng%2E+2018%2E+Near+or+far%2C+wide+range+zero%2Dshot+cross%2Dlingual+dependency+parsing%2E+CoRR%2C+abs%2F1811%2E00570%2E)
2. David Alvarez-Melis and Tommi Jaakkola. 2018. Gromov-Wasserstein alignment of word embedding spaces. In Proc. of EMNLP, pages 1881–1890. [our card](https://md.research.elamur.ai/free/en/papers/melis-2018-gromov-wasserstein-alignment-word-embedding/)
3. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. [our card](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/)
4. Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng, and Ting Liu. 2018. Towards better ud parsing: Deep contextualized word embeddings, ensemble, and treebank concatenation. In Proc. of CoNLL, pages 55–64. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wanxiang+Che%2C+Yijia+Liu%2C+Yuxuan+Wang%2C+Bo+Zheng%2C+and+Ting+Liu%2E+2018%2E+Towards+better+ud+parsing%3A+Deep+contextualized+word+embeddings%2C+ensemble%2C+and+treebank+concatenation%2E+In+Proc%2E+of+CoNLL%2C+pages+55%E2%80%9364)
5. Xilun Chen and Claire Cardie. 2018. Unsupervised multilingual word embeddings. In Proc. of EMNLP, pages 261–270. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xilun+Chen+and+Claire+Cardie%2E+2018%2E+Unsupervised+multilingual+word+embeddings%2E+In+Proc%2E+of+EMNLP%2C+pages+261%E2%80%93270%2E+Association+for+Computational+Linguistics%2E)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
7. Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In Proc. of ICLR. [our card](https://md.research.elamur.ai/free/en/papers/dozat-2016-deep-biaffine-attention-neural-dependency/)
8. Chris Dyer, Adam Lopez, Juri Ganitkevitch, Jonathan Weese, Ferhan Ture, Phil Blunsom, Hendra Setiawan, Vladimir Eidelman, and Philip Resnik. 2010. cdec: A decoder, alignment, and learning framework for finite-state and context-free translation models. In Proc. of the ACL 2010 System Demonstrations,… [our card](https://md.research.elamur.ai/free/en/papers/dyer-2010-cdec-decoder-alignment-learning-framework/)
9. Philip Edmonds and Scott Cotton. 2001. Senseval-2: Overview. In The Proc. of the Second International Workshop on Evaluating Word Sense Disambiguation Systems, SENSEVAL ’01, pages 1–5, Stroudsburg, PA, USA. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/edmonds-2001-senseval-2-overview/)
10. Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. 2015. Cross-lingual dependency parsing based on distributed representations. In Proc. of ACL and IJCNLP, pages 1234–1244. [our card](https://md.research.elamur.ai/free/en/papers/guo-2015-cross-lingual-dependency-parsing-based/)
11. Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. 2016. A representation learning framework for multi-source transfer parsing. In Proc. of AAAI, pages 2734–2740. [find](https://scholar.google.com/scholar?q=Jiang+Guo%2C+Wanxiang+Che%2C+David+Yarowsky%2C+Haifeng+Wang%2C+and+Ting+Liu%2E+2016%2E+A+representation+learning+framework+for+multi%2Dsource+transfer+parsing%2E+In+Proc%2E+of+AAAI%2C+pages+2734%E2%80%932740%2E)
12. Suzana Ilić, Edison Marrese-Taylor, Jorge Balazs, and Yutaka Matsuo. 2018. Deep contextualized word representations for detecting sarcasm and irony. In Proc. of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 2–7. Association for Computationa… [our card](https://md.research.elamur.ai/free/en/papers/ilic-2018-deep-contextualized-word-representations-detecting/)
13. Diederick P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proc. of ICLR. [find](https://scholar.google.com/scholar?q=Diederick+P+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+Proc%2E+of+ICLR%2E)
14. Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing crosslingual distributed representations of words. In Proc. of COLING, pages 1459–1474. [our card](https://md.research.elamur.ai/free/en/papers/klementiev-2012-inducing-crosslingual-distributed-representations-words/)
15. Tomáš Kočiský, Karl Moritz Hermann, and Phil Blunsom. 2014. Learning bilingual word representations by marginalizing alignments. In Proc. of ACL, pages 224–229. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Toma%CC%81s%CC%8C+Koc%CC%8Cisky%CC%81%2C+Karl+Moritz+Hermann%2C+and+Phil+Blunsom%2E+2014%2E+Learning+bilingual+word+representations+by+marginalizing+alignments%2E+In+Proc%2E+of+ACL%2C+pages+224%E2%80%93229%2E+Association+for+Computational+Lingu)
16. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proc. of the tenth Machine Translation Summit, pages 79–86, Phuket, Thailand. AAMT. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2E+2005%2E+Europarl%3A+A+parallel+corpus+for+statistical+machine+translation%2E+In+Proc%2E+of+the+tenth+Machine+Translation+Summit%2C+pages+79%E2%80%9386%2C+Phuket%2C+Thailand%2E+AAMT%2E)
17. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. CoRR, abs/1901.07291. [find](https://scholar.google.com/scholar?q=Guillaume+Lample+and+Alexis+Conneau%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E+CoRR%2C+abs%2F1901%2E07291%2E)
18. Guillaume Lample, Alexis Conneau, Marc’Aurelio Ranzato, Ludovic Denoyer, and Herv Jgou. 2018. Word translation without parallel data. In Proc. of ICLR. [find](https://scholar.google.com/scholar?q=Guillaume+Lample%2C+Alexis+Conneau%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Ludovic+Denoyer%2C+and+Herv+Jgou%2E+2018%2E+Word+translation+without+parallel+data%2E+In+Proc%2E+of+ICLR%2E)
19. Angeliki Lazaridou, Georgiana Dinu, and Marco Baroni. 2015. Hubness and pollution: Delving into cross-space mapping for zero-shot learning. In Proc. of ACL and IJCNLP, pages 270–280. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/lazaridou-2015-hubness-pollution-delving-into-cross/)
20. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems 30, pages 6294–6305. Curran Associates, Inc. [our card](https://md.research.elamur.ai/free/en/papers/mccann-2017-learned-translation-contextualized-word-vectors/)
21. Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, Claudia Bedini, Núria Bertomeu Castelló, and Jungmee Lee. 2013. Universal dependency annotation for multilingual parsing. In Proc. of ACL, pag… [our card](https://md.research.elamur.ai/free/en/papers/mcdonald-2013-universal-dependency-annotation-multilingual-parsing/)
22. Ryan McDonald, Slav Petrov, and Keith Hall. 2011. Multi-source transfer of delexicalized dependency parsers. In Proc. of EMNLP, pages 62–72. [our card](https://md.research.elamur.ai/free/en/papers/mcdonald-2011-multi-source-transfer-delexicalized-dependency/)
23. Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. CoRR, abs/1309.4168. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Quoc+V%2E+Le%2C+and+Ilya+Sutskever%2E+2013%2E+Exploiting+similarities+among+languages+for+machine+translation%2E+CoRR%2C+abs%2F1309%2E4168%2E)
24. Phoebe Mulcaire, Jungo Kasai, and Noah Smith. 2019. Polyglot contextual representations improve crosslingual transfer. In Proc. of NAACL. [find](https://scholar.google.com/scholar?q=Phoebe+Mulcaire%2C+Jungo+Kasai%2C+and+Noah+Smith%2E+2019%2E+Polyglot+contextual+representations+improve+crosslingual+transfer%2E+In+Proc%2E+of+NAACL%2E)
25. Joakim Nivre, Mitchell Abrams, Željko Agić, and et al. 2018. Universal dependencies 2.2. LIN-DAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Mitchell+Abrams%2C+Z%CC%8Celjko+Agic%CC%81%2C+and+et+al%2E+2018%2E+Universal+dependencies+2%2E2%2E+LIN%2DDAT%2FCLARIN+digital+library+at+the+Institute+of+Formal+and+Applied+Linguistics+%28U%CC%81FAL%29%2C+Faculty+of+Mathema)
26. Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In Proc. of ACL, pages 1756–1765. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/peters-2017-semi-supervised-sequence-tagging-bidirectional/)
27. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL, pages 2227–2237. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/)
28. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proc. of ACL, pages 4996–5001. [our card](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/)
29. Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2018. Cross-lingual transfer learning for multilingual task oriented dialog. CoRR, abs/1810.13327. [our card](https://md.research.elamur.ai/free/en/papers/schuster-2019-cross-lingual-transfer-learning-multilingual/)
30. Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In Proc. of NAACL, pages 1599–1613. [our card](https://md.research.elamur.ai/free/en/papers/schuster-2019-cross-lingual-alignment-contextual-word/)
31. Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In Proc. of ICLR. [our card](https://md.research.elamur.ai/free/en/papers/smith-2017-offline-bilingual-word-vectors-orthogonal/)
32. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of bert. In Proc. of EMNLP. [our card](https://md.research.elamur.ai/free/en/papers/wu-2019-beto-bentz-becas-surprising-cross/)
33. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V%2E+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+Jeff+Klingner%2C+Apurva+Shah%2C+Melvin+Johnson%2C+Xiaobing+Liu%2C+Luk)
34. Min Xiao and Yuhong Guo. 2014. Distributed word representation learning for cross-lingual dependency parsing. In Proc. of CoNLL, pages 119–129. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Min+Xiao+and+Yuhong+Guo%2E+2014%2E+Distributed+word+representation+learning+for+cross%2Dlingual+dependency+parsing%2E+In+Proc%2E+of+CoNLL%2C+pages+119%E2%80%93129%2E+Association+for+Computational+Linguistics%2E)
35. Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. In Proc. of NAACL, pages 1006–1011. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/xing-2015-normalized-word-embedding-orthogonal-transform/)

*Updated 24 October 2026*