---
title: "Constituency Parsing with a Self-Attentive Encoder — Kitaev 2018"
description: "The authors keep the chart-based span decoder of Stern et al. (2017a), with changes from Gaddy et al.… 93.55 F1. 19 references, cited by 283 papers here."
authors: ["Nikita Kitaev", "Dan Klein"]
year: 2018
venue: "arXiv:1805.01052"
source: "https://arxiv.org/abs/1805.01052"
doi: "10.48550/arxiv.1805.01052"
arxiv: "1805.01052"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kitaev-2018-constituency-parsing-self-attentive-encoder/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kitaev-2018-constituency-parsing-self-attentive-encoder/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/) / Constituency Parsing with a Self-Attentive Encoder

# Constituency Parsing with a Self-Attentive Encoder

**Authors:** [Nikita Kitaev](https://md.research.elamur.ai/free/en/authors/nikita-kitaev/) and [Dan Klein](https://md.research.elamur.ai/free/en/authors/dan-klein/) · **Year:** 2018 · **Venue:** arXiv:1805.01052 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1805.01052) · [PDF](https://arxiv.org/pdf/1805.01052) · [DOI 10.48550/arxiv.1805.01052](https://doi.org/10.48550/arxiv.1805.01052) · [arXiv 1805.01052](https://arxiv.org/abs/1805.01052) · [OpenAlex](https://openalex.org/W2963084773)

**Citations:** 476 citations (OpenAlex, 2026-10-14) · [283](https://md.research.elamur.ai/free/en/papers/kitaev-2018-constituency-parsing-self-attentive-encoder/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
This paper asks whether a strong constituency parser needs a recurrent (LSTM) encoder. The authors keep the chart-based span decoder of Stern et al. (2017a), with changes from Gaddy et al. (2018), and replace the encoder with a stack of self-attention layers adapted from the Transformer of Vaswani et al. (2017). Attention shows how information moves between words, so the authors use it to study the model. They find it relies mostly on word position rather than word content. Keeping content and position signals in separate channels, which they call a "factored" encoder, improves accuracy. They also compare several word-level inputs: external POS tags, character embeddings of prefixes and suffixes, a character LSTM, and pre-trained ELMo vectors. On the Penn Treebank WSJ test set, the best model without external data scores 93.55 F1. With ELMo it scores 95.13 F1, and both were the best single-model results at the time. The parser also beats earlier published scores on 8 of 9 SPMRL languages. The work shows that attention-based encoders can match or beat RNNs for syntactic parsing and are easier to inspect.

## Key points
- On the WSJ development set, the self-attentive encoder (8 layers, 8 heads) scores 92.67 F1, compared with 92.24 for the LSTM encoder of Gaddy et al. (2018) using the same decoder.
- Development F1 rises at each step: factored attention gives 93.15, adding a CharLSTM gives 93.61, and adding ELMo gives 95.21. The ELMo result came from a smaller 4-layer encoder.
- WSJ test results are 93.55 F1 for the CharLSTM model using WSJ data only and 95.13 F1 for the ELMo model.
- On SPMRL 2013/2014, the parser sets new best results for 8 of 9 languages. Swedish is the exception, where an ensemble with reranking (Björkelund et al., 2014) stays ahead. ELMo was tested only on English.
- Training the WSJ-only model took about 18 hours on one Tesla K80 GPU. Code and English models are published at github.com/nikitakit/self-attentive-parser.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [📐 Syntax & grammar formalisms](https://md.research.elamur.ai/free/en/topics/syntax-grammar/)

## Institutions

- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 107 works · 205,102 citations

## Cite

Cite as: Nikita Kitaev and Dan Klein. 2018. Constituency Parsing with a Self-Attentive Encoder. arXiv:1805.01052.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{kitaev2018constituency,
  title = {{Constituency Parsing with a Self-Attentive Encoder}},
  author = {Nikita Kitaev and Dan Klein},
  year = {2018},
  doi = {10.48550/arxiv.1805.01052},
  eprint = {1805.01052},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1805.01052}
}
```

- [← previous Colorless Green Recurrent Networks Dream Hierarchically](https://md.research.elamur.ai/free/en/papers/gulordava-2018-colorless-green-recurrent-networks-dream/)
- [next → Findings of the 2018 Conference on Machine Translation (WMT18)](https://md.research.elamur.ai/free/en/papers/bojar-2018-findings-2018-conference-machine-translation/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/)

## References

19 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer Normalization. arXiv:1607.06450 [cs, stat]. ArXiv: 1607.06450. [our card](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/)
2. Anders Björkelund, Ozlem Cetinoglu, Agnieszka Faleńska, Richárd Farkas, Thomas Mueller, Wolfgang Seeker, and Zsolt Szántó. 2014. The IMS-Wrocław-Szeged-CIS entry at the SPMRL 2014 shared task: Reranking and morphosyntax meet unlabeled data. In Proceedings of the First Joint Workshop on Statisti… [find](https://scholar.google.com/scholar?q=Anders+Bjo%CC%88rkelund%2C+Ozlem+Cetinoglu%2C+Agnieszka+Falen%CC%81ska%2C+Richa%CC%81rd+Farkas%2C+Thomas+Mueller%2C+Wolfgang+Seeker%2C+and+Zsolt+Sza%CC%81nto%CC%81%2E+2014%2E+The+IMS%2DWroc%C5%82aw%2DSzeged%2DCIS+entry+at+the+SPMRL+2014+shared+task%3A+Re)
3. Anders Björkelund, Ozlem Cetinoglu, Richárd Farkas, Thomas Mueller, and Wolfgang Seeker. 2013. (Re)ranking meets morphosyntax: State-of-the-art results from the SPMRL 2013 shared task. In Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages, pages 135–145, Se… [find](https://scholar.google.com/scholar?q=Anders+Bjo%CC%88rkelund%2C+Ozlem+Cetinoglu%2C+Richa%CC%81rd+Farkas%2C+Thomas+Mueller%2C+and+Wolfgang+Seeker%2E+2013%2E+%28Re%29ranking+meets+morphosyntax%3A+State%2Dof%2Dthe%2Dart+results+from+the+SPMRL+2013+shared+task%2E+In+Proceeding)
4. Do Kook Choe and Eugene Charniak. 2016. Parsing as language modeling. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2331–2336. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Do+Kook+Choe+and+Eugene+Charniak%2E+2016%2E+Parsing+as+language+modeling%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+2331%E2%80%932336%2E+Association+for+Comput)
5. Maximin Coavoux and Benoit Crabbé. 2017. Multilingual lexicalized constituency parsing with word-level auxiliary tasks. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 331–336. Association for Computationa… [find](https://scholar.google.com/scholar?q=Maximin+Coavoux+and+Benoit+Crabbe%CC%81%2E+2017%2E+Multilingual+lexicalized+constituency+parsing+with+word%2Dlevel+auxiliary+tasks%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Associatio)
6. James Cross and Liang Huang. 2016. Span-based constituency parsing with a structure-label system and provably optimal dynamic oracles. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1–11. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/cross-2016-span-based-constituency-parsing-structure/)
7. Greg Durrett and Dan Klein. 2015. Neural CRF parsing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 302–312. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Greg+Durrett+and+Dan+Klein%2E+2015%2E+Neural+CRF+parsing%2E+In+Proceedings+of+the+53rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+7th+International+Joint+Conference+on+Natural+L)
8. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 199–209. Association for Computationa… [find](https://scholar.google.com/scholar?q=Chris+Dyer%2C+Adhiguna+Kuncoro%2C+Miguel+Ballesteros%2C+and+Noah+A%2E+Smith%2E+2016%2E+Recurrent+neural+network+grammars%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of+the+Association+for)
9. Daniel Fried, Mitchell Stern, and Dan Klein. 2017. Improving neural parsing by disentangling model combination and reranking effects. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 161–166. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Daniel+Fried%2C+Mitchell+Stern%2C+and+Dan+Klein%2E+2017%2E+Improving+neural+parsing+by+disentangling+model+combination+and+reranking+effects%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+C)
10. David Gaddy, Mitchell Stern, and Dan Klein. 2018. What’s going on in neural constituency parsers? An analysis. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+Gaddy%2C+Mitchell+Stern%2C+and+Dan+Klein%2E+2018%2E+What%E2%80%99s+going+on+in+neural+constituency+parsers%3F+An+analysis%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Association+fo)
11. David Hall, Greg Durrett, and Dan Klein. 2014. Less grammar, more features. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, volume 1, pages 228–237. [find](https://scholar.google.com/scholar?q=David+Hall%2C+Greg+Durrett%2C+and+Dan+Klein%2E+2014%2E+Less+grammar%2C+more+features%2E+In+Proceedings+of+the+52nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+volume+1%2C+pages+228%E2%80%93237%2E)
12. Jiangming Liu and Yue Zhang. 2017. In-order transition-based constituent parsing. Transactions of the Association for Computational Linguistics, 5:413–424. [our card](https://md.research.elamur.ai/free/en/papers/liu-2017-order-transition-based-constituent-parsing/)
13. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno… [our card](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/)
14. Djamé Seddah, Reut Tsarfaty, Sandra Kübler, Marie Candito, Jinho D. Choi, Richárd Farkas, Jennifer Foster, Iakes Goenaga, Koldo Gojenola Gal-letebeitia, Yoav Goldberg, Spence Green, Nizar Habash, Marco Kuhlmann, Wolfgang Maier, Joakim Nivre, Adam Przepiórkowski, Ryan Roth, Wolfgang Seeker, Yanni… [our card](https://md.research.elamur.ai/free/en/papers/seddah-2013-overview-spmrl-2013-shared-task/)
15. Mitchell Stern, Jacob Andreas, and Dan Klein. 2017a. A minimal span-based neural constituency parser. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 818–827. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/stern-2017-minimal-span-based-neural-constituency/)
16. Mitchell Stern, Daniel Fried, and Dan Klein. 2017b. Effective inference for generative neural parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1695–1700. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mitchell+Stern%2C+Daniel+Fried%2C+and+Dan+Klein%2E+2017b%2E+Effective+inference+for+generative+neural+parsing%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+)
17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Process… [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
18. Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In Advances in Neural Information Processing Systems 28, pages 2755– 2763. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+%C5%81ukasz+Kaiser%2C+Terry+Koo%2C+Slav+Petrov%2C+Ilya+Sutskever%2C+and+Geoffrey+Hinton%2E+2015%2E+Grammar+as+a+foreign+language%2E+In+Advances+in+Neural+Information+Processing+Systems+28%2C+pages+2755%E2%80%93+276)
19. Symbol Description N N d model h d k d v d f f [find](https://scholar.google.com/scholar?q=Symbol+Description+N+N+d+model+h+d+k+d+v+d+f+f)