---
title: "On the Predictive Power of Neural Language Models for Human"
description: "This study asks which type of language model best reflects the expectations people form while… 23 references, cited by 94 papers here."
authors: ["Ethan Gotlieb Wilcox", "Jon Gauthier", "Jennifer Hu", "Peng Qian", "Roger Levy"]
year: 2020
venue: "arXiv:2006.01912"
source: "https://arxiv.org/abs/2006.01912"
doi: "10.48550/arxiv.2006.01912"
arxiv: "2006.01912"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wilcox-2020-predictive-power-neural-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wilcox-2020-predictive-power-neural-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior

# On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior

**Authors:** [Ethan Gotlieb Wilcox](https://md.research.elamur.ai/free/en/authors/ethan-wilcox/), [Jon Gauthier](https://md.research.elamur.ai/free/en/authors/jon-gauthier/), [Jennifer Hu](https://md.research.elamur.ai/free/en/authors/jennifer-hu/), [Peng Qian](https://md.research.elamur.ai/free/en/authors/peng-qian/) and [Roger Levy](https://md.research.elamur.ai/free/en/authors/roger-levy/) · **Year:** 2020 · **Venue:** arXiv:2006.01912 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2006.01912) · [PDF](https://arxiv.org/pdf/2006.01912) · [DOI 10.48550/arxiv.2006.01912](https://doi.org/10.48550/arxiv.2006.01912) · [arXiv 2006.01912](https://arxiv.org/abs/2006.01912) · [OpenAlex](https://openalex.org/W3033254023)

**Citations:** 105 citations (OpenAlex, 2026-10-14) · [94](https://md.research.elamur.ai/free/en/papers/wilcox-2020-predictive-power-neural-language-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This study asks which type of language model best reflects the expectations people form while reading, as seen in word-by-word reading times. The authors trained more than two dozen models from four families: LSTM, RNNG (supervised with Penn Treebank-style parses), GPT-2 Transformers trained from scratch (word-level and BPE variants) and a Kneser-Ney 5-gram. Each was trained on four nested BLLIP subsets of increasing size. Each model's surprisal was entered into regressions on human reading data from the Dundee eye-tracking corpus, Brown self-paced reading and Natural Stories, with word length and frequency as controls. Fit was scored as the per-token log-likelihood gain over a baseline (∆LogLik). Surprisal related to reading time in a near-linear way, and lower perplexity usually meant better fit. Architecture still mattered. At comparable perplexity, Transformers and n-grams did better than LSTMs and RNNGs, especially on eye movements. Syntactic generalization ability explained nothing extra once perplexity was accounted for. This implies that modeling naturalistic reading and modeling targeted syntactic tests may require different approaches.

## Key points
- Training data: BLLIP-XS (100K tokens), BLLIP-SM (5M), BLLIP-MD (14M) and BLLIP-LG (42M). Each smaller set is a subset of the larger ones, and 1–3 random seeds were trained per configuration.
- Evaluation: surprisal of the current word and preceding words (to capture spillover) was used to predict gaze duration in Dundee and self-paced reading times in Brown and Natural Stories (SPRT). Natural Stories deliberately includes rare constructions.
- At equal perplexity, GPT-2 and the 5-gram show higher psychometric predictive power than LSTM and RNNG. The 12-layer GPT-2 with BPE trained on larger data scores best on all three corpora.
- Syntactic Generalization (SG) scores, taken from Hu et al. (2020), did not significantly improve ∆LogLik prediction beyond perplexity in a stepwise regression (p > 0.26 for every corpus).
- Code and data: github.com/wilcoxeg/neural-networks-read-times.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) — University, Cambridge, United States · 243 works · 76,506 citations
- [Harvard University](https://md.research.elamur.ai/free/en/universities/harvard-university/) — University, Cambridge, United States · 126 works · 45,130 citations

📚 Shelves (8): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,649 works · [💭 Developmental and Educational Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/developmental-and-educational-psychology/) · 523 works · [🧠 Neuroscience](https://md.research.elamur.ai/free/en/shelves/neuroscience/) · 186 works · [🧠 Cognitive Neuroscience](https://md.research.elamur.ai/free/en/shelves/neuroscience/cognitive-neuroscience/) · 185 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧒 Psycholinguistics & language acquisition](https://md.research.elamur.ai/free/en/topics/psycholinguistics/) · 108 works

## Cite

Cite as: Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian and Roger Levy. 2020. On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior. arXiv:2006.01912.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wilcox2020predictive,
  title = {{On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior}},
  author = {Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian and Roger Levy},
  year = {2020},
  doi = {10.48550/arxiv.2006.01912},
  eprint = {2006.01912},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2006.01912}
}
```

- [← previous Multimodal Transformer for Multimodal Machine Translation](https://md.research.elamur.ai/free/en/papers/yao-2020-multimodal-transformer-multimodal-machine-translation/)
- [next → SpellGCN: Incorporating Phonological and Visual Similarities into Language Models for Chinese Spelling Check](https://md.research.elamur.ai/free/en/papers/cheng-2020-spellgcn-incorporating-phonological-visual-similarities/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/)

## References

23 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aurnhammer, C., & Frank, S. (2019). Comparing gated and simple recurrent neural network architectures as models of human sentence processing. In Proceedings of the 41st annual conference of the cognitive science society (pp. 112– 118). [find](https://scholar.google.com/scholar?q=Aurnhammer%2C+C%2E%2C+%26+Frank%2C+S%2E+%282019%29%2E+Comparing+gated+and+simple+recurrent+neural+network+architectures+as+models+of+human+sentence+processing%2E+In+Proceedings+of+the+41st+annual+conference+of+the+cognit)
2. Charniak, E., Blaheta, D., Ge, N., Hall, K., Hale, J., & Johnson, M. (2000). BLLIP 1987–89 WSJ corpus release 1. Linguistic Data Consortium, Philadelphia, 36. [find](https://scholar.google.com/scholar?q=Charniak%2C+E%2E%2C+Blaheta%2C+D%2E%2C+Ge%2C+N%2E%2C+Hall%2C+K%2E%2C+Hale%2C+J%2E%2C+%26+Johnson%2C+M%2E+%282000%29%2E+BLLIP+1987%E2%80%9389+WSJ+corpus+release+1%2E+Linguistic+Data+Consortium%2C+Philadelphia%2C+36%2E)
3. Dyer, C., Kuncoro, A., Ballesteros, M., & Smith, N. A. (2016). Recurrent neural network grammars. In Proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: Human language technologies, volume 1 (long papers). [find](https://scholar.google.com/scholar?q=Dyer%2C+C%2E%2C+Kuncoro%2C+A%2E%2C+Ballesteros%2C+M%2E%2C+%26+Smith%2C+N%2E+A%2E+%282016%29%2E+Recurrent+neural+network+grammars%2E+In+Proceedings+of+the+2018+conference+of+the+north+american+chapter+of+the+association+for+computation)
4. Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14(2), 179-211. doi: 10.1207/s15516709cog1402\\ 1 Fossum, V., & Levy, R. (2012). Sequential vs. hierarchical syntactic models of human incremental sentence processing. In Proceedings of the 3rd workshop on cognitive modeling and compu… — [doi:10.1207/s15516709cog1402\](https://doi.org/10.1207/s15516709cog1402\) [find](https://scholar.google.com/scholar?q=Elman%2C+J%2E+L%2E+%281990%29%2E+Finding+structure+in+time%2E+Cognitive+Science%2C+14%282%29%2C+179%2D211%2E+doi%3A+10%2E1207%2Fs15516709cog1402%5C%5C+1+Fossum%2C+V%2E%2C+%26+Levy%2C+R%2E+%282012%29%2E+Sequential+vs%2E+hierarchical+syntactic+models+of+huma)
5. Frank, S. L., & Bod, R. (2011). Insensitivity of the human sentence-processing system to hierarchical structure. Psychological science, 22(6), 829–834. [find](https://scholar.google.com/scholar?q=Frank%2C+S%2E+L%2E%2C+%26+Bod%2C+R%2E+%282011%29%2E+Insensitivity+of+the+human+sentence%2Dprocessing+system+to+hierarchical+structure%2E+Psychological+science%2C+22%286%29%2C+829%E2%80%93834%2E)
6. Frank, S. L., Otten, L. J., Galli, G., & Vigliocco, G. (2015). The ERP response to the amount of information conveyed by words in sentences. Brain and Language, 140, 1–11. [find](https://scholar.google.com/scholar?q=Frank%2C+S%2E+L%2E%2C+Otten%2C+L%2E+J%2E%2C+Galli%2C+G%2E%2C+%26+Vigliocco%2C+G%2E+%282015%29%2E+The+ERP+response+to+the+amount+of+information+conveyed+by+words+in+sentences%2E+Brain+and+Language%2C+140%2C+1%E2%80%9311%2E)
7. Futrell, R., Gibson, E., Tily, H., Blank, I., Vishnevetsky, A., Piantadosi, S. T., & Fedorenko, E. (2017). The natural stories corpus. arXiv preprint arXiv:1708.05763. [find](https://scholar.google.com/scholar?q=Futrell%2C+R%2E%2C+Gibson%2C+E%2E%2C+Tily%2C+H%2E%2C+Blank%2C+I%2E%2C+Vishnevetsky%2C+A%2E%2C+Piantadosi%2C+S%2E+T%2E%2C+%26+Fedorenko%2C+E%2E+%282017%29%2E+The+natural+stories+corpus%2E+arXiv+preprint+arXiv%3A1708%2E05763%2E)
8. Futrell, R., Wilcox, E., Morita, T., & Levy, R. (2018). [RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency](https://md.research.elamur.ai/free/en/papers/futrell-2018-rnns-psycholinguistic-subjects-syntactic-state/).
9. Goodkind, A., & Bicknell, K. (2018). [Predictive power of word surprisal for reading times is a linear function of language model quality](https://md.research.elamur.ai/free/en/papers/goodkind-2018-predictive-power-word-surprisal-reading/). In Proceedings of the 8th workshop on cognitive modeling and computational linguistics (CMCL 2018) (pp. 10–18).
10. Hale, J. (2001). [A probabilistic Earley parser as a psycholinguistic model](https://md.research.elamur.ai/free/en/papers/hale-2001-probabilistic-earley-parser-as-psycholinguistic/). In Proceedings of the second meeting of the north american chapter of the association for computational linguistics on language technologies (pp. 1–8).
11. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735-1780. doi: 10 .1162/neco.1997.9.8.1735 Hu, J., Gauthier, J., Qian, P., Wilcox, E., & Levy, R. P. (2020). A systematic assessment of syntactic generalization in neural language models. In Proceedings of t… [find](https://scholar.google.com/scholar?q=Hochreiter%2C+S%2E%2C+%26+Schmidhuber%2C+J%2E+%281997%29%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2C+9%288%29%2C+1735%2D1780%2E+doi%3A+10+%2E1162%2Fneco%2E1997%2E9%2E8%2E1735+Hu%2C+J%2E%2C+Gauthier%2C+J%2E%2C+Qian%2C+P%2E%2C+Wilcox%2C+E%2E%2C+%26+Levy%2C+R%2E+P%2E+%282020%29)
12. Kennedy, A. (2003). The Dundee Corpus [cd-rom]. Psychology Department, University of Dundee. [find](https://scholar.google.com/scholar?q=Kennedy%2C+A%2E+%282003%29%2E+The+Dundee+Corpus+%5Bcd%2Drom%5D%2E+Psychology+Department%2C+University+of+Dundee%2E)
13. Levy, R. (2008). Expectation-based syntactic comprehension. Cognition, 106(3), 1126–1177. [find](https://scholar.google.com/scholar?q=Levy%2C+R%2E+%282008%29%2E+Expectation%2Dbased+syntactic+comprehension%2E+Cognition%2C+106%283%29%2C+1126%E2%80%931177%2E)
14. Marcus, M. P., Marcinkiewicz, M. A., & Santorini, B. (1993). [Building a large annotated corpus of English: The Penn Treebank](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/). Computational Linguistics, 19, 313–330.
15. Marvin, R., & Linzen, T. (2018, October-November). Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 1192–1202). Brussels, Belgium: Association for Computational Linguistics. Retrieved from [https://www.acl…](https://www.acl…) [↗](https://www.acl…)
16. Merkx, D., & Frank, S. L. (2020). Comparing transformers and rnns on predicting human sentence processing data. arXiv preprint arXiv:2005.09471. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., De-Vito, Z., . . . Lerer, A. (2017). Automatic differentiation in PyTorch. In Neural information… — [arXiv:2005.09471](https://arxiv.org/abs/2005.09471) [find](https://scholar.google.com/scholar?q=Merkx%2C+D%2E%2C+%26+Frank%2C+S%2E+L%2E+%282020%29%2E+Comparing+transformers+and+rnns+on+predicting+human+sentence+processing+data%2E+arXiv+preprint+arXiv%3A2005%2E09471%2E+Paszke%2C+A%2E%2C+Gross%2C+S%2E%2C+Chintala%2C+S%2E%2C+Chanan%2C+G%2E%2C+Yang%2C+)
17. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+Child%2C+R%2E%2C+Luan%2C+D%2E%2C+Amodei%2C+D%2E%2C+%26+Sutskever%2C+I%2E+%282019%29%2E+Language+models+are+unsupervised+multitask+learners%2E)
18. Sennrich, R., Haddow, B., & Birch, A. (2015). [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). arXiv preprint arXiv:1508.07909.
19. Smith, N. J., & Levy, R. (2008). Optimal processing times in reading: a formal model and empirical investigation. In Proceedings of the 30th annual meeting of the Cognitive Science Society (pp. 595–600). [find](https://scholar.google.com/scholar?q=Smith%2C+N%2E+J%2E%2C+%26+Levy%2C+R%2E+%282008%29%2E+Optimal+processing+times+in+reading%3A+a+formal+model+and+empirical+investigation%2E+In+Proceedings+of+the+30th+annual+meeting+of+the+Cognitive+Science+Society+%28pp%2E+595%E2%80%9360)
20. Washington, DC. Smith, N. J., & Levy, R. (2013). The effect of word predictability on reading time is logarithmic. Cognition, 128(3), 302–319. [find](https://scholar.google.com/scholar?q=Washington%2C+DC%2E+Smith%2C+N%2E+J%2E%2C+%26+Levy%2C+R%2E+%282013%29%2E+The+effect+of+word+predictability+on+reading+time+is+logarithmic%2E+Cognition%2C+128%283%29%2C+302%E2%80%93319%2E)
21. Staub, A., & Clifton, C. (2006). Syntactic prediction in language comprehension: Evidence from either . . . or. Journal of Experimental Psychology: Learning, Memory, & Cognition, 32(2), 425–436. [find](https://scholar.google.com/scholar?q=Staub%2C+A%2E%2C+%26+Clifton%2C+C%2E+%282006%29%2E+Syntactic+prediction+in+language+comprehension%3A+Evidence+from+either+%2E+%2E+%2E+or%2E+Journal+of+Experimental+Psychology%3A+Learning%2C+Memory%2C+%26+Cognition%2C+32%282%29%2C+425%E2%80%93436%2E)
22. Stolcke, A. (2002). SRILM – an extensible language modeling toolkit. In Seventh international conference on spoken language processing. [find](https://scholar.google.com/scholar?q=Stolcke%2C+A%2E+%282002%29%2E+SRILM+%E2%80%93+an+extensible+language+modeling+toolkit%2E+In+Seventh+international+conference+on+spoken+language+processing%2E)
23. Stowe, L. A. (1986). Parsing wh-constructions: Evidence for on-line gap location. Language & Cognitive Processes, 1(3), 227–245. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., . . . Polosukhin, I. (2017). [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In I. Guyon et al. (Eds.), Advances i…

*Updated 2 November 2026*