--- title: "Deep RNNs Encode Soft Hierarchical Syntax — Blevins 2018" description: "This short paper asks whether deep recurrent networks trained on ordinary NLP objectives learn… 1 reference, cited by 62 papers here." authors: ["Terra Blevins", "Omer Levy", "Luke Zettlemoyer"] year: 2018 venue: "Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)" source: "https://aclanthology.org/P18-2003/" doi: "10.18653/v1/p18-2003" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/blevins-2018-deep-rnns-encode-soft-hierarchical/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/blevins-2018-deep-rnns-encode-soft-hierarchical/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/) / Deep RNNs Encode Soft Hierarchical Syntax # Deep RNNs Encode Soft Hierarchical Syntax **Authors:** [Terra Blevins](https://md.research.elamur.ai/free/en/authors/terra-blevins/), [Omer Levy](https://md.research.elamur.ai/free/en/authors/omer-levy/) and [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) · **Year:** 2018 · **Venue:** [Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2018](https://md.research.elamur.ai/free/en/venues/acl/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/P18-2003/) · [PDF](https://aclanthology.org/P18-2003.pdf) · [DOI 10.18653/v1/p18-2003](https://doi.org/10.18653/v1/p18-2003) · [OpenAlex](https://openalex.org/W2801092741) **Citations:** 116 citations (OpenAlex, 2026-10-14) · [62](https://md.research.elamur.ai/free/en/papers/blevins-2018-deep-rnns-encode-soft-hierarchical/cited-by/) in our library, of the 141,840 papers whose bibliographies we read ## Summary This short paper asks whether deep recurrent networks trained on ordinary NLP objectives learn syntactic structure that they were never directly taught. The authors probe four pre-trained multi-layer LSTM models: a Stanford dependency parser (English Web Treebank, UD), DeepSRL (CoNLL-2012), an OpenNMT encoder (WMT-14 English-German), and forward/backward language models (CoNLL-2012). From each layer's word vectors, a small feed-forward classifier predicts four labels: the POS tag and the constituent labels one, two, and three levels above the word. A second probe tests whether a pair of word vectors indicates a dependency arc in UD. All four kinds of supervision produce representations that beat per-word majority baselines. The only exception is POS for the parser, which already receives gold tags as input. The layers also form a soft hierarchy: tasks higher in the tree reach peak accuracy in deeper layers. The results suggest that deep RNNs build up abstract syntax internally. This may explain why they work without hand-made linguistic features, and why ELMo-style layer mixing helps. ## Key points - The probe is a single 300-unit ReLU hidden layer with softmax. It is trained on the CoNLL-2012 development set and tested on its test set, so it never sees the RNNs' training data. - In 11 of 16 model–task combinations, accuracy rises up to some layer and then falls. For SRL, parent-label accuracy is highest at the layer just above where POS is best, and the grandparent and great-grandparent labels reach their best one layer higher again. - The language model matches the dependency parser on the constituent tasks at layer 2. After that layer, the language model declines while the parser keeps improving. - On dependency arc prediction, the parser's top layer reaches about 95%. The SRL, MT, and LM models peak roughly 12–20 points above their input layers, while GloVe stays at chance (0.50). - Baseline task scores of the probed models: parser UAS 91.5 / LAS 82.18; MT BLEU 21.37; LM perplexity 50.56 (forward) and 51.24 (backward) on CoNLL-2012. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 263 works · 180,581 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works 🧭 Topics (1): [📐 Syntax & grammar formalisms](https://md.research.elamur.ai/free/en/topics/syntax-grammar/) · 161 works ## Cite Cite as: Terra Blevins, Omer Levy and Luke Zettlemoyer. 2018. Deep RNNs Encode Soft Hierarchical Syntax. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{blevins2018deep, title = {{Deep RNNs Encode Soft Hierarchical Syntax}}, author = {Terra Blevins, Omer Levy and Luke Zettlemoyer}, booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)}, year = {2018}, doi = {10.18653/v1/p18-2003}, url = {https://doi.org/10.18653/v1/p18-2003} } ``` - [← previous Tensor2Tensor for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/vaswani-2018-tensor2tensor-neural-machine-translation/) - [next → Integrating Transformer and Paraphrase Rules for Sentence Simplification](https://md.research.elamur.ai/free/en/papers/zhao-2018-integrating-transformer-paraphrase-rules-sentence/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. Roee Aharoni and Yoav Goldberg. 2017. [Towards string-to-tree neural machine translation](https://md.research.elamur.ai/free/en/papers/aharoni-2017-string-tree-neural-machine-translation/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, pages 132–140. https://doi.org/10.18653/v1/P17-2021… [↗](https://doi.org/10.18653/v1/P17-2021…) *Updated 24 October 2026*