--- title: "Neural language models as psycholinguistic — Futrell 2019" description: "Futrell and colleagues test whether neural language models track incremental syntactic state." authors: ["Richard Futrell", "Ethan Wilcox", "Takashi Morita", "Peng Qian", "Miguel Ballesteros", "Roger Levy"] year: 2019 venue: "Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)" source: "https://aclanthology.org/N19-1004/" doi: "10.18653/v1/n19-1004" arxiv: "1903.03260" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/futrell-2019-neural-language-models-as-psycholinguistic/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/futrell-2019-neural-language-models-as-psycholinguistic/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / Neural language models as psycholinguistic subjects: Representations of syntactic state # Neural language models as psycholinguistic subjects: Representations of syntactic state **Authors:** [Richard Futrell](https://md.research.elamur.ai/free/en/authors/richard-futrell/), [Ethan Wilcox](https://md.research.elamur.ai/free/en/authors/ethan-wilcox/), [Takashi Morita](https://md.research.elamur.ai/free/en/authors/takashi-morita/), [Peng Qian](https://md.research.elamur.ai/free/en/authors/peng-qian/), [Miguel Ballesteros](https://md.research.elamur.ai/free/en/authors/miguel-ballesteros/), [Roger Levy](https://md.research.elamur.ai/free/en/authors/roger-levy/) · **Year:** 2019 · **Venue:** Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/N19-1004/) · [PDF](https://aclanthology.org/N19-1004.pdf) · [DOI 10.18653/v1/n19-1004](https://doi.org/10.18653/v1/n19-1004) · [arXiv 1903.03260](https://arxiv.org/abs/1903.03260) · [OpenAlex](https://openalex.org/W2921890305) **Citations:** 144 citations (openalex-snapshot · 2026-10-14) · [147](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Futrell and colleagues test whether neural language models track incremental syntactic state. This is the information a stack-based parser would hold, such as knowing that an open subordinate clause still needs a main clause. They borrow methods from psycholinguistics and treat per-word surprisal as a stand-in for human reading time. Surprisal is measured on hand-built sentence sets that target subordinate clauses and two garden-path ambiguities: NP/Z and main verb/reduced relative (MV/RR). Condition contrasts are analysed with linear mixed-effects regression. Four English models are compared. JRNN (One Billion Word Benchmark) and GRNN (Wikipedia) are large LSTMs. An RNNG and a matched TinyLSTM are both trained on about 1 million Penn Treebank tokens. All models show basic evidence of representing syntactic state. Only the large-data models, and in some tests the RNNG, use subtler lexical cues such as verb transitivity or participle morphology. Unlike humans, the models' garden-path effects shrink rather than grow as the ambiguous region lengthens. The work offers a reusable targeted-evaluation framework and shows that both training-data scale and explicit syntactic structure shape what models learn. ## Key points - **Models and training data:** - JRNN: LSTM, about 800M tokens. - GRNN: LSTM, about 90M tokens. - RNNG: about 1M Penn Treebank tokens; surprisal estimated with word-synchronous beam search (action beam 100, word beam 10). - TinyLSTM: about 1M Penn Treebank tokens. - **Subordinate clauses:** the authors define two measures. - Matrix licensing effect: lower surprisal for a main clause after a subordinator. - No-matrix penalty: higher surprisal when the main clause is missing. - Only GRNN and the RNNG showed the penalty. - **MV/RR garden path (29 items):** reducing the relative clause adds the following surprisal at the disambiguating verb: | Model | Added surprisal | |---|---| | TinyLSTM | 0.3 bits | | JRNN | 1.9 bits | | RNNG | 2.8 bits | | GRNN | 3.6 bits | An unambiguous participle (e.g., "given") lowers but does not remove this effect, so models treat morphology as only a noisy cue. - **Fine-grained cues:** TinyLSTM failed every fine-grained test. The RNNG missed the verb-transitivity effect in NP/Z sentences. - **Released materials:** items, results and analysis scripts are at github.com/langprocgroup/nn_syntactic_state. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) Topics: [Treebanks](https://md.research.elamur.ai/free/en/topics/treebanks/) ## Cited by Cited by 147 papers of our library cite it (of the 140884 whose bibliographies we read); 12 of them have a page here, 135 not yet. - [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/) - [Assessing the Agreement Competence of Large Language Models](https://md.research.elamur.ai/free/en/papers/garcia-2025-assessing-agreement-competence-large-language/) - [SyntaxGym: An Online Platform for Targeted Evaluation of Language Models](https://md.research.elamur.ai/free/en/papers/gauthier-2020-syntaxgym-online-platform-targeted-evaluation/) - [Expectations over Unspoken Alternatives Predict Pragmatic Inferences](https://md.research.elamur.ai/free/en/papers/hu-2023-expectations-over-unspoken-alternatives-predict/) - [A fine-grained comparison of pragmatic language understanding in humans and language models](https://md.research.elamur.ai/free/en/papers/hu-2023-fine-grained-comparison-pragmatic-language/) - [Prompting is not a substitute for probability measurements in large language models](https://md.research.elamur.ai/free/en/papers/hu-2023-prompting-is-not-substitute-probability/) - [Can language models capture syntactic associations without surface cues? A case study of reflexive anaphor licensing in English control constructions](https://md.research.elamur.ai/free/en/papers/lee-2022-can-language-models-capture-syntactic/) - [Word predictability estimates from language models are not robust to tokenizer vocabulary](https://md.research.elamur.ai/free/en/papers/nguyen-2026-word-predictability-estimates-language-models/) - [Harnessing the linguistic signal to predict scalar inferences](https://md.research.elamur.ai/free/en/papers/schuster-2020-harnessing-linguistic-signal-predict-scalar/) - [Similar Predictions, Different Processes: A Multi-Level Comparison of Human and Multimodal LLM Language Prediction](https://md.research.elamur.ai/free/en/papers/wang-2026-similar-predictions-different-processes-multi/) - [Toward Cognitive Alignment in Large Language Models: Integrating Linguistic Theory and Human Data](https://md.research.elamur.ai/free/en/papers/zaghouani-2026-cognitive-alignment-large-language-models/) - [SemEval-2022 Task 3: PreTENS — Evaluating Neural Networks on Presuppositional Semantic Knowledge](https://md.research.elamur.ai/free/en/papers/zamparelli-2022-semeval-2022-task-3-pretens/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{futrell2019neural, title = {{Neural language models as psycholinguistic subjects: Representations of syntactic state}}, author = {Futrell, Richard and Wilcox, Ethan and Morita, Takashi and Qian, Peng and Ballesteros, Miguel and Levy, Roger}, booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)}, year = {2019}, doi = {10.18653/v1/n19-1004}, eprint = {1903.03260}, archivePrefix = {arXiv}, url = {https://doi.org/10.18653/v1/n19-1004} } ``` - [← previous Neural Generative Rhetorical Structure Parsing](https://md.research.elamur.ai/free/en/papers/mabona-2019-neural-generative-rhetorical-structure-parsing/) - [next → Neural Legal Judgment Prediction in English](https://md.research.elamur.ai/free/en/papers/chalkidis-2019-neural-legal-judgment-prediction-english/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) ## References 4 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. R. Haraald Baayen, D.J. Davidson, and D.M. Bates. 2008. Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4):390–412. [find](https://scholar.google.com/scholar?q=R%2E+Haraald+Baayen%2C+D%2EJ%2E+Davidson%2C+and+D%2EM%2E+Bates%2E+2008%2E+Mixed%2Deffects+modeling+with+crossed+random+effects+for+subjects+and+items%2E+Journal+of+Memory+and+Language%2C+59%284%29%3A390%E2%80%93412%2E) 2. Dale J. Barr, Roger P. Levy, Christoph Scheepers, and Harry J Tily. 2013. Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3):255–278. [find](https://scholar.google.com/scholar?q=Dale+J%2E+Barr%2C+Roger+P%2E+Levy%2C+Christoph+Scheepers%2C+and+Harry+J+Tily%2E+2013%2E+Random+effects+structure+for+confirmatory+hypothesis+testing%3A+Keep+it+maximal%2E+Journal+of+Memory+and+Language%2C+68%283%29%3A255%E2%80%93278%2E) 3. Jean-Philippe Bernardy and Shalom Lappin. 2017. Using deep neural networks to learn syntactic agreement. Linguistic Issues in Language Technology, 15:1–15. [find](https://scholar.google.com/scholar?q=Jean%2DPhilippe+Bernardy+and+Shalom+Lappin%2E+2017%2E+Using+deep+neural+networks+to+learn+syntactic+agreement%2E+Linguistic+Issues+in+Language+Technology%2C+15%3A1%E2%80%9315%2E) 4. Thomas G. Bever. 1970. The cognitive basis for linguistic structures. In J. R. Hayes, editor, Cognition and the Development of Language. Wiley, New York. [find](https://scholar.google.com/scholar?q=Thomas+G%2E+Bever%2E+1970%2E+The+cognitive+basis+for+linguistic+structures%2E+In+J%2E+R%2E+Hayes%2C+editor%2C+Cognition+and+the+Development+of+Language%2E+Wiley%2C+New+York%2E)