---
title: "What Do Recurrent Neural Network Grammars Learn About"
description: "RNNGs are generative models that build phrase-structure trees top-down using a… 93.6 F1 on PTB. 33 references, cited by 73 papers here."
authors: ["Adhiguna Kuncoro", "Miguel Ballesteros", "Lingpeng Kong", "Chris Dyer", "Graham Neubig", "Noah A. Smith"]
year: 2016
venue: "arXiv:1611.05774"
source: "https://arxiv.org/abs/1611.05774"
doi: "10.48550/arxiv.1611.05774"
arxiv: "1611.05774"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kuncoro-2016-recurrent-neural-network-grammars-learn/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kuncoro-2016-recurrent-neural-network-grammars-learn/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / What Do Recurrent Neural Network Grammars Learn About Syntax?

# What Do Recurrent Neural Network Grammars Learn About Syntax?

**Authors:** [Adhiguna Kuncoro](https://md.research.elamur.ai/free/en/authors/adhiguna-kuncoro/), [Miguel Ballesteros](https://md.research.elamur.ai/free/en/authors/miguel-ballesteros/), [Lingpeng Kong](https://md.research.elamur.ai/free/en/authors/lingpeng-kong/), [Chris Dyer](https://md.research.elamur.ai/free/en/authors/chris-dyer/), [Graham Neubig](https://md.research.elamur.ai/free/en/authors/graham-neubig/) and [Noah A. Smith](https://md.research.elamur.ai/free/en/authors/noah-a-smith/) · **Year:** 2016 · **Venue:** arXiv:1611.05774 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1611.05774) · [PDF](https://arxiv.org/pdf/1611.05774) · [DOI 10.48550/arxiv.1611.05774](https://doi.org/10.48550/arxiv.1611.05774) · [arXiv 1611.05774](https://arxiv.org/abs/1611.05774) · [OpenAlex](https://openalex.org/W2950191064)

**Citations:** 10 citations (OpenAlex, 2026-10-14) · [73](https://md.research.elamur.ai/free/en/papers/kuncoro-2016-recurrent-neural-network-grammars-learn/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Neural parsers perform well, but it is hard to tell what syntactic knowledge they actually acquire. This paper treats recurrent neural network grammars (RNNGs) as objects of linguistic inquiry. RNNGs are generative models that build phrase-structure trees top-down using a stack, a buffer and an action history. On the English PTB (§02–21 train, §24 dev, §23 test), the authors train variants with components removed. They evaluate these on constituency parsing, on dependency parsing after conversion to Stanford dependencies, and on language modeling. They then add a gated attention mechanism to the composition function (GA-RNNG) to reveal which children a phrase vector depends on. A label-free version (U-GA-RNNG) tests how much nonterminal categories matter. Three findings emerge:
- Explicit composition on the stack is the decisive ingredient.
- Attention largely recovers lexical heads, mostly agreeing with hand-written head rules but sometimes spreading weight over several children, e.g. in conjunctions.
- Phrase vectors learned without labels still cluster into traditional categories.

These results support the endocentricity hypothesis and suggest that bracketing carries most of the syntactic information.

## Key points
- Removing the stack hurts most: PTB §23 F1 falls from 93.3 (baseline RNNG) to 92.5. Dropping the buffer or the action history barely matters.
- A stack-only RNNG without ensembling reached 93.6 F1 on PTB. After conversion, it achieved 95.8 UAS and 94.6 LAS on PTB SD, the best supervised results reported at the time. GA-RNNG scored 93.5 F1.
- RNNGs beat linearized-tree models: 91.2 vs. 88.3 F1 against Vinyals et al. (2015) in the discriminative setting, and 93.3 vs. 92.6 against Choe and Charniak (2016) in the generative setting.
- In U-GA-RNNG, PP and SBAR phrases opening with preposition-like words cluster together. Phrases built on *of* and *because* form their own groups.
- Code and pretrained models are released at github.com/clab/rnng (interpreting-rnng).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 492 works · 440,463 citations
- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 598 works · 191,226 citations
- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 402 works · 212,763 citations
- [IBM](https://md.research.elamur.ai/free/en/companies/ibm/) — Company, Armonk, United States · 154 works · 72,606 citations
- [Google DeepMind](https://md.research.elamur.ai/free/en/companies/google-deepmind/) — Company, London, United Kingdom · 72 works · 84,326 citations
- [IBM Research - Thomas J. Watson Research Center](https://md.research.elamur.ai/free/en/labs/ibm-research-thomas-j-watson-research-center/) — Research institute, Yorktown Heights, United States · 55 works · 49,277 citations
- [University of Pittsburgh](https://md.research.elamur.ai/free/en/universities/university-of-pittsburgh/) — University, Pittsburgh, United States · 65 works · 23,401 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🧠 Neuroscience](https://md.research.elamur.ai/free/en/shelves/neuroscience/) · 186 works · [🧠 Cognitive Neuroscience](https://md.research.elamur.ai/free/en/shelves/neuroscience/cognitive-neuroscience/) · 185 works

🧭 Topics (1): [📐 Syntax & grammar formalisms](https://md.research.elamur.ai/free/en/topics/syntax-grammar/) · 221 works

## Cite

Cite as: Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig and Noah A. Smith. 2016. What Do Recurrent Neural Network Grammars Learn About Syntax?. arXiv:1611.05774.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{kuncoro2016what,
  title = {{What Do Recurrent Neural Network Grammars Learn About Syntax?}},
  author = {Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig and Noah A. Smith},
  year = {2016},
  doi = {10.48550/arxiv.1611.05774},
  eprint = {1611.05774},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1611.05774}
}
```

- [← previous Sentiment Lexicons for Arabic Social Media](https://md.research.elamur.ai/free/en/papers/mohammad-2016-sentiment-lexicons-arabic-social-media/)
- [next → ConvKN at SemEval-2016 Task 3: Answer and Question Selection for Question Answering on Arabic and English Fora](https://md.research.elamur.ai/free/en/papers/cedeno-2016-convkn-semeval-2016-task-3/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/)

## References

33 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. [Globally normalized transition-based neural networks](https://md.research.elamur.ai/free/en/papers/andor-2016-globally-normalized-transition-based-neural/). In Proc. of ACL.
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). In Proc. of ICLR.
3. David Chiang and Daniel M. Bikel. 2002. Recovering latent information in treebanks. In Proc. of COL-ING. [find](https://scholar.google.com/scholar?q=David+Chiang+and+Daniel+M%2E+Bikel%2E+2002%2E+Recovering+latent+information+in+treebanks%2E+In+Proc%2E+of+COL%2DING%2E)
4. Do Kook Choe and Eugene Charniak. 2016. Parsing as language modeling. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Do+Kook+Choe+and+Eugene+Charniak%2E+2016%2E+Parsing+as+language+modeling%2E+In+Proc%2E+of+EMNLP%2E)
5. Noam Chomsky. 1970. Remarks on nominalization. In Readings in English Transformational Grammar. [find](https://scholar.google.com/scholar?q=Noam+Chomsky%2E+1970%2E+Remarks+on+nominalization%2E+In+Readings+in+English+Transformational+Grammar%2E)
6. Noam Chomsky. 1993. A Minimalist Program for Linguistic Theory. [find](https://scholar.google.com/scholar?q=Noam+Chomsky%2E+1993%2E+A+Minimalist+Program+for+Linguistic+Theory%2E)
7. Michael Collins. 1997. [Three generative, lexicalised models for statistical parsing](https://md.research.elamur.ai/free/en/papers/collins-1997-three-generative-lexicalised-models-statistical/). In Proc. of EACL.
8. Marie-Catherine De Marneffe and Christopher D. Manning. 2008. Stanford typed dependencies manual. [find](https://scholar.google.com/scholar?q=Marie%2DCatherine+De+Marneffe+and+Christopher+D%2E+Manning%2E+2008%2E+Stanford+typed+dependencies+manual%2E)
9. Marie-Catherine De Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. [Generating typed dependency parses from phrase structure parses](https://md.research.elamur.ai/free/en/papers/marneffe-2006-generating-typed-dependency-parses-phrase/). In Proc. of LREC.
10. Timothy Dozat and Christopher D. Manning. 2016. [Deep biaffine attention for neural dependency parsing](https://md.research.elamur.ai/free/en/papers/dozat-2016-deep-biaffine-attention-neural-dependency/). arXiv:1611.01734.
11. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In Proc. of NAACL. [find](https://scholar.google.com/scholar?q=Chris+Dyer%2C+Adhiguna+Kuncoro%2C+Miguel+Ballesteros%2C+and+Noah+A%2E+Smith%2E+2016%2E+Recurrent+neural+network+grammars%2E+In+Proc%2E+of+NAACL%2E)
12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2E)
13. Ray Jackendoff. 1977. X’ Syntax. [find](https://scholar.google.com/scholar?q=Ray+Jackendoff%2E+1977%2E+X%E2%80%99+Syntax%2E)
14. T. Florian Jaeger. 2010. Redundancy and reduction: Speakers manage syntactic information density. Cognitive Psychology, 61(1). [find](https://scholar.google.com/scholar?q=T%2E+Florian+Jaeger%2E+2010%2E+Redundancy+and+reduction%3A+Speakers+manage+syntactic+information+density%2E+Cognitive+Psychology%2C+61%281%29%2E)
15. Janne Bondi Johannessen. 1998. Coordination. [find](https://scholar.google.com/scholar?q=Janne+Bondi+Johannessen%2E+1998%2E+Coordination%2E)
16. Richard Johansson and Pierre Nugues. 2007. [Extended constituent-to-dependency conversion for English](https://md.research.elamur.ai/free/en/papers/johansson-2007-extended-constituent-dependency-conversion-english/). In Proc. of NODALIDA.
17. Mark Johnson. 1998. [PCFG models of linguistic tree representations](https://md.research.elamur.ai/free/en/papers/johnson-1998-pcfg-models-linguistic-tree-representations/). Computational Linguistics.
18. Andrej Karpathy, Justin Johnson, and Fei-Fei Li. 2015. [Visualizing and understanding recurrent networks](https://md.research.elamur.ai/free/en/papers/karpathy-2015-visualizing-understanding-recurrent-networks/). arXiv:1506.02078.
19. Edward L. Keenan. 1987. Multiply-headed noun phrases. Linguistic Inquiry, 18(3):481–490. [find](https://scholar.google.com/scholar?q=Edward+L%2E+Keenan%2E+1987%2E+Multiply%2Dheaded+noun+phrases%2E+Linguistic+Inquiry%2C+18%283%29%3A481%E2%80%93490%2E)
20. Eliyahu Kiperwasser and Yoav Goldberg. 2016. [Simple and accurate dependency parsing using bidirectional LSTM feature representations](https://md.research.elamur.ai/free/en/papers/kiperwasser-2016-simple-accurate-dependency-parsing-bidirectional/). TACL.
21. Dan Klein and Christopher D. Manning. 2002. [A generative constituent-context model for improved grammar induction](https://md.research.elamur.ai/free/en/papers/klein-2002-generative-constituent-context-model-improved/). In Proc. of ACL.
22. Dan Klein and Christopher D. Manning. 2003. [Accurate unlexicalized parsing](https://md.research.elamur.ai/free/en/papers/klein-2003-accurate-unlexicalized-parsing/). In Proc. of ACL.
23. Lingpeng Kong and Noah A. Smith. 2014. [An empirical comparison of parsing methods for stanford dependencies](https://md.research.elamur.ai/free/en/papers/kong-2014-empirical-comparison-parsing-methods-stanford/). arXiv:1404.4314.
24. Jiwei Li, Dan Jurafsky, and Eduard Hovy. 2015. [When are tree structures necessary for deep learning of representations?](https://md.research.elamur.ai/free/en/papers/li-2015-when-are-tree-structures-necessary/) In Proc. of EMNLP.
25. Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. [Visualizing and understanding neural models in NLP](https://md.research.elamur.ai/free/en/papers/li-2016-visualizing-understanding-neural-models-nlp/). In Proc. of NAACL.
26. Tom M. Mitchell. 1980. The need for biases in learning generalizations. [find](https://scholar.google.com/scholar?q=Tom+M%2E+Mitchell%2E+1980%2E+The+need+for+biases+in+learning+generalizations%2E)
27. Fernando Pereira and Yves Schabes. 1992. [Inside-outside reestimation from partially bracketed corpora](https://md.research.elamur.ai/free/en/papers/pereira-1992-inside-outside-reestimation-partially-bracketed/). In Proc. of ACL.
28. Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. 2006. [Learning accurate, compact, and interpretable tree annotation](https://md.research.elamur.ai/free/en/papers/petrov-2006-learning-accurate-compact-interpretable-tree/). In Proc. of COLING-ACL.
29. Federico Sangati and Willem Zuidema. 2009. Unsupervised methods for head assignments. In Proc. of EACL. [find](https://scholar.google.com/scholar?q=Federico+Sangati+and+Willem+Zuidema%2E+2009%2E+Unsupervised+methods+for+head+assignments%2E+In+Proc%2E+of+EACL%2E)
30. Xing Shi, Inkit Padhi, and Kevin Knight. 2016. [Does string-based neural MT learn source syntax?](https://md.research.elamur.ai/free/en/papers/shi-2016-does-string-based-neural-mt/) In Proc. of EMNLP.
31. Laurens van der Maaten and Geoffrey E. Hinton. 2008. Visualizing high-dimensional data using tsne. Journal of Machine Learning Research, 9. [find](https://scholar.google.com/scholar?q=Laurens+van+der+Maaten+and+Geoffrey+E%2E+Hinton%2E+2008%2E+Visualizing+high%2Ddimensional+data+using+tsne%2E+Journal+of+Machine+Learning+Research%2C+9%2E)
32. Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In NIPS. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Lukasz+Kaiser%2C+Terry+Koo%2C+Slav+Petrov%2C+Ilya+Sutskever%2C+and+Geoffrey+Hinton%2E+2015%2E+Grammar+as+a+foreign+language%2E+In+NIPS%2E)
33. Sam Wiseman and Alexander M. Rush. 2016. Sequence-to-sequence learning as beam-search optimization. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Sam+Wiseman+and+Alexander+M%2E+Rush%2E+2016%2E+Sequence%2Dto%2Dsequence+learning+as+beam%2Dsearch+optimization%2E+In+Proc%2E+of+EMNLP%2E)

*Updated 2 November 2026*