---
title: "Multilingual Constituency Parsing with — Kitaev 2019"
description: "The paper asks how much unsupervised pre-training helps constituency parsing across languages, and whether one parser can serve many languages without a…"
authors: ["Nikita Kitaev", "Steven Cao", "Dan Klein"]
year: 2019
venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P19-1340/"
doi: "10.18653/v1/p19-1340"
arxiv: "1812.11760"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kitaev-2019-multilingual-constituency-parsing-self-attention/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kitaev-2019-multilingual-constituency-parsing-self-attention/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / Multilingual Constituency Parsing with Self-Attention and Pre-Training

# Multilingual Constituency Parsing with Self-Attention and Pre-Training

**Authors:** [Nikita Kitaev](https://md.research.elamur.ai/free/en/authors/nikita-kitaev/), Steven Cao, [Dan Klein](https://md.research.elamur.ai/free/en/authors/dan-klein/) · **Year:** 2019 · **Venue:** Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P19-1340/) · [PDF](https://aclanthology.org/P19-1340.pdf) · [DOI 10.18653/v1/p19-1340](https://doi.org/10.18653/v1/p19-1340) · [arXiv 1812.11760](https://arxiv.org/abs/1812.11760) · [OpenAlex](https://openalex.org/W2963643701)

**Citations:** 200 citations (openalex-snapshot · 2026-10-14) · [156](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The paper asks how much unsupervised pre-training helps constituency parsing across languages, and whether one parser can serve many languages without a huge parameter cost. The authors extend their earlier self-attentive span-based parser (Kitaev and Klein, 2018). It scores labeled spans with an MLP classifier and decodes with a CKY variant. They feed it fastText, ELMo or BERT representations, then fine-tune on treebanks for English (WSJ), Chinese (Chinese Treebank 5.1) and the nine SPMRL 2013/2014 languages. For ten languages they also train one jointly fine-tuned model. It shares multilingual BERT and the self-attention layers, and adds only a small per-language span classifier. Contextual pre-training improves results for every language, and higher-capacity models do better. The shared model is about ten times smaller than separate per-language parsers and loses little accuracy. Pairwise experiments show that small treebanks can benefit from large ones. The work sets new state-of-the-art results for all 11 languages. This makes it a practical blueprint for compact multilingual parsers, and the code and models are public (self-attentive-parser).

## Key points
- On the WSJ dev set, F1 rises from 93.61 (no pre-training) and 93.72 (fastText) to 95.21 (ELMo), 95.32 (BERT BASE) and 95.70 (BERT LARGE). A 4-model ensemble reaches 95.87.
- English-only BERT beats multilingual BERT by under 0.3 F1 (95.24 vs. 94.97). Even BERT pre-trained on Chinese Wikipedia roughly matches the treebank-only baseline on English (93.57 F1).
- The joint ten-language model has 189M parameters, compared with 1,851M for separate models. Its average F1 is 91.12 vs. 91.40, a 3.2% relative error increase, and it actually improves Swedish.
- The largest treebanks (German, English, Korean) work best as auxiliary languages. Small ones (Swedish, French, Polish) gain the most from joint training. Language pairing is driven more by annotation conventions than by linguistic kinship.
- Test-set results are 95.8 F1 on WSJ (ensemble) and 91.75 F1 on Chinese Treebank 5.1. On SPMRL, both the joint and per-language models beat all prior systems on every language.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Treebanks](https://md.research.elamur.ai/free/en/topics/treebanks/)

## Cited by

Cited by

156 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 142 not yet.

- [Tetra-Tagging: Word-Synchronous Parsing with Linear-Time Inference](https://md.research.elamur.ai/free/en/papers/acl-2020-kitaev-tetra-tagging-word-synchronous-parsing/)
- [Extracting Headless MWEs from Dependency Parse Trees: Parsing, Tagging, and Joint Modeling Approaches](https://md.research.elamur.ai/free/en/papers/acl-2020-shi-extracting-headless-mwes-dependency-parse/)
- [Anaphoric Zero Pronoun Identification: A Multilingual Approach](https://md.research.elamur.ai/free/en/papers/aloraini-2020-anaphoric-zero-pronoun-identification-multilingual/)
- [Penn-Helsinki Parsed Corpus of Early Modern English: First Parsing Results and Analysis](https://md.research.elamur.ai/free/en/papers/findings-2022-kulick-penn-helsinki-parsed-corpus-early/)
- [Parsing Gapping Constructions Based on Grammatical and Semantic Roles](https://md.research.elamur.ai/free/en/papers/kato-2020-parsing-gapping-constructions-based-grammatical/)
- [Parsing Early Modern English for Linguistic Search](https://md.research.elamur.ai/free/en/papers/kulick-2022-parsing-early-modern-english-linguistic/)
- [FlauBERT: Unsupervised Language Model Pre-training for French](https://md.research.elamur.ai/free/en/papers/le-2019-flaubert-unsupervised-language-model-pre/)
- [Semantic Role Labeling as Syntactic Dependency Parsing](https://md.research.elamur.ai/free/en/papers/shi-2020-semantic-role-labeling-as-syntactic/)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://md.research.elamur.ai/free/en/papers/sinha-2021-masked-language-modeling-distributional-hypothesis/)
- [75 Languages, 1 Model: Parsing Universal Dependencies Universally](https://md.research.elamur.ai/free/en/papers/ud-2019-kondratyuk-75-languages-1-model-parsing/)
- [Graph-to-Graph Transformer for Transition-based Dependency Parsing](https://md.research.elamur.ai/free/en/papers/ud-2020-mohammadshahi-graph-graph-transformer-transition-based/)
- [BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://md.research.elamur.ai/free/en/papers/wang-2019-bert-has-mouth-must-speak/)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
- [Multilingual Coreference Resolution in Multiparty Dialogue](https://md.research.elamur.ai/free/en/papers/zheng-2023-multilingual-coreference-resolution-multiparty-dialogue/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kitaev2019multilingual,
  title = {{Multilingual Constituency Parsing with Self-Attention and Pre-Training}},
  author = {Kitaev, Nikita and Cao, Steven and Klein, Dan},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1340},
  eprint = {1812.11760},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1340}
}
```

- [← previous Multilingual and Multi-Aspect Hate Speech Analysis](https://md.research.elamur.ai/free/en/papers/ousidhoum-2019-multilingual-multi-aspect-hate-speech/)
- [next → Multilingual is not enough: BERT for Finnish](https://md.research.elamur.ai/free/en/papers/virtanen-2019-multilingual-enough-bert-finnish/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/)

## References

21 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. An empirical investigation of statistical significance in NLP. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 995–1005, Jeju Island, Korea.… [find](https://scholar.google.com/scholar?q=Taylor+Berg%2DKirkpatrick%2C+David+Burkett%2C+and+Dan+Klein%2E+2012%2E+An+empirical+investigation+of+statistical+significance+in+NLP%2E+In+Proceedings+of+the+2012+Joint+Conference+on+Empirical+Methods+in+Natural+)
2. Anders Björkelund, Ozlem Cetinoglu, Agnieszka Faleńska, Richárd Farkas, Thomas Mueller, Wolfgang Seeker, and Zsolt Szántó. 2014. The IMS-Wrocław-Szeged-CIS entry at the SPMRL 2014 shared task: Reranking and morphosyntax meet unlabeled data. In Proceedings of the First Joint Workshop on Statisti… [find](https://scholar.google.com/scholar?q=Anders+Bjo%CC%88rkelund%2C+Ozlem+Cetinoglu%2C+Agnieszka+Falen%CC%81ska%2C+Richa%CC%81rd+Farkas%2C+Thomas+Mueller%2C+Wolfgang+Seeker%2C+and+Zsolt+Sza%CC%81nto%CC%81%2E+2014%2E+The+IMS%2DWroc%C5%82aw%2DSzeged%2DCIS+entry+at+the+SPMRL+2014+shared+task%3A+Re)
3. Anders Björkelund, Ozlem Cetinoglu, Richárd Farkas, Thomas Mueller, and Wolfgang Seeker. 2013. (Re)ranking meets morphosyntax: State-of-the-art results from the SPMRL 2013 shared task. In Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages, pages 135–145, Se… [find](https://scholar.google.com/scholar?q=Anders+Bjo%CC%88rkelund%2C+Ozlem+Cetinoglu%2C+Richa%CC%81rd+Farkas%2C+Thomas+Mueller%2C+and+Wolfgang+Seeker%2E+2013%2E+%28Re%29ranking+meets+morphosyntax%3A+State%2Dof%2Dthe%2Dart+results+from+the+SPMRL+2013+shared+task%2E+In+Proceeding)
4. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. [find](https://scholar.google.com/scholar?q=Piotr+Bojanowski%2C+Edouard+Grave%2C+Armand+Joulin%2C+and+Tomas+Mikolov%2E+2017%2E+Enriching+word+vectors+with+subword+information%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+5%3A135%E2%80%93146%2E)
5. Do Kook Choe and Eugene Charniak. 2016. Parsing as language modeling. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2331–2336. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Do+Kook+Choe+and+Eugene+Charniak%2E+2016%2E+Parsing+as+language+modeling%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+2331%E2%80%932336%2E+Association+for+Comput)
6. Maximin Coavoux and Benoit Crabbé. 2017. Multilingual lexicalized constituency parsing with word-level auxiliary tasks. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 331–336. Association for Computationa… [find](https://scholar.google.com/scholar?q=Maximin+Coavoux+and+Benoit+Crabbe%CC%81%2E+2017%2E+Multilingual+lexicalized+constituency+parsing+with+word%2Dlevel+auxiliary+tasks%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Associatio)
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018a. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805 [cs]. ArXiv: 1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018a%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv%3A1810%2E04805+%5Bcs%5D%2E+ArXiv%3A+1810%2E04805%2E)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018b. BERT: Pre-training of deep bidirectional transformers for language understanding. [https://github.com/google-research/bert/](https://github.com/google-research/bert/) blob/master/multilingual.md. [↗](https://github.com/google-research/bert/)
9. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 199–209. Association for Computationa… [find](https://scholar.google.com/scholar?q=Chris+Dyer%2C+Adhiguna+Kuncoro%2C+Miguel+Ballesteros%2C+and+Noah+A%2E+Smith%2E+2016%2E+Recurrent+neural+network+grammars%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of+the+Association+for)
10. Daniel Fried and Dan Klein. 2018. Policy gradient as a proxy for dynamic oracles in constituency parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 469–476. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Daniel+Fried+and+Dan+Klein%2E+2018%2E+Policy+gradient+as+a+proxy+for+dynamic+oracles+in+constituency+parsing%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Vo)
11. Daniel Fried, Mitchell Stern, and Dan Klein. 2017. Improving neural parsing by disentangling model combination and reranking effects. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 161–166. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Daniel+Fried%2C+Mitchell+Stern%2C+and+Dan+Klein%2E+2017%2E+Improving+neural+parsing+by+disentangling+model+combination+and+reranking+effects%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+C)
12. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volum)
13. Vidur Joshi, Matthew Peters, and Mark Hopkins. 2018. Extending a parser to distant domains using a few dozen partially annotated examples. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1190–1199. Association for Computationa… [find](https://scholar.google.com/scholar?q=Vidur+Joshi%2C+Matthew+Peters%2C+and+Mark+Hopkins%2E+2018%2E+Extending+a+parser+to+distant+domains+using+a+few+dozen+partially+annotated+examples%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+) [our card](https://md.research.elamur.ai/free/en/papers/acl-2018-joshi-extending-parser-distant-domains-few/)
14. Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2676–2686. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nikita+Kitaev+and+Dan+Klein%2E+2018%2E+Constituency+parsing+with+a+self%2Dattentive+encoder%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers) [our card](https://md.research.elamur.ai/free/en/papers/kitaev-2018-constituency-parsing-self-attentive-encoder/)
15. Jiangming Liu and Yue Zhang. 2017. In-order transition-based constituent parsing. Transactions of the Association for Computational Linguistics, 5:413–424. [find](https://scholar.google.com/scholar?q=Jiangming+Liu+and+Yue+Zhang%2E+2017%2E+In%2Dorder+transition%2Dbased+constituent+parsing%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+5%3A413%E2%80%93424%2E)
16. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the 11th Language Resources and Evaluation Conference, Miyazaki, Japan. European Language Resource Association. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Edouard+Grave%2C+Piotr+Bojanowski%2C+Christian+Puhrsch%2C+and+Armand+Joulin%2E+2018%2E+Advances+in+pre%2Dtraining+distributed+word+representations%2E+In+Proceedings+of+the+11th+Language+Resources+and) [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2017-advances-pre-training-distributed-word/)
17. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+the+2018+Conference+of+t)
18. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
19. Djamé Seddah, Reut Tsarfaty, Sandra Kübler, Marie Candito, Jinho D. Choi, Richárd Farkas, Jennifer Foster, Iakes Goenaga, Koldo Gojenola Gal-letebeitia, Yoav Goldberg, Spence Green, Nizar Habash, Marco Kuhlmann, Wolfgang Maier, Joakim Nivre, Adam Przepiórkowski, Ryan Roth, Wolfgang Seeker, Yanni… [find](https://scholar.google.com/scholar?q=Djame%CC%81+Seddah%2C+Reut+Tsarfaty%2C+Sandra+Ku%CC%88bler%2C+Marie+Candito%2C+Jinho+D%2E+Choi%2C+Richa%CC%81rd+Farkas%2C+Jennifer+Foster%2C+Iakes+Goenaga%2C+Koldo+Gojenola+Gal%2Dletebeitia%2C+Yoav+Goldberg%2C+Spence+Green%2C+Nizar+Habash%2C+M) [our card](https://md.research.elamur.ai/free/en/papers/seddah-2013-overview-spmrl-2013-shared-task/)
20. Zhiyang Teng and Yue Zhang. 2018. Two local models for neural constituent parsing. In Proceedings of the 27th International Conference on Computational Linguistics, pages 119–132. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zhiyang+Teng+and+Yue+Zhang%2E+2018%2E+Two+local+models+for+neural+constituent+parsing%2E+In+Proceedings+of+the+27th+International+Conference+on+Computational+Linguistics%2C+pages+119%E2%80%93132%2E+Association+for+Comp)
21. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V%2E+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+Jeff+Klingner%2C+Apurva+Shah%2C+Melvin+Johnson%2C+Xiaobing+Liu%2C+%C5%81uk)