---
title: "Predictive power of word surprisal for — Goodkind 2018"
description: "Surprisal theory links reading difficulty to a word's negative log probability in context."
authors: ["Adam Goodkind", "Klinton Bicknell"]
year: 2018
venue: "Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018)"
source: "https://aclanthology.org/W18-0102/"
doi: "10.18653/v1/w18-0102"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/goodkind-2018-predictive-power-word-surprisal-reading/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/goodkind-2018-predictive-power-word-surprisal-reading/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/) / Predictive power of word surprisal for reading times is a linear function of language model quality

# Predictive power of word surprisal for reading times is a linear function of language model quality

**Authors:** Adam Goodkind, Klinton Bicknell · **Year:** 2018 · **Venue:** Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W18-0102/) · [PDF](https://aclanthology.org/W18-0102.pdf) · [DOI 10.18653/v1/w18-0102](https://doi.org/10.18653/v1/w18-0102) · [OpenAlex](https://openalex.org/W2795342569)

**Citations:** 180 citations (openalex-snapshot · 2026-10-14) · [114](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Surprisal theory links reading difficulty to a word's negative log probability in context. Researchers usually estimate these probabilities with computational language models, which predict upcoming words less well than humans do. This raises the question of whether weaker models distort conclusions about surprisal. Goodkind and Bicknell trained seven language models of varying quality on the Google One Billion Word Benchmark: n-gram models, an LSTM, and two LSTM/5-gram interpolations. They used each model's surprisal estimates to predict first-pass gaze durations in the English Dundee Corpus, fitting generalized additive mixed-effects models with both linear and nonlinear surprisal terms. As perplexity dropped, the models' gain in fit to reading times grew along a tight linear trend. Across all model types, the estimated size of the surprisal effect stayed roughly the same, and its shape looked approximately linear. These results suggest that surprisal findings based on weaker language models are not systematically biased. They also suggest that better language models yield better cognitive predictions.

## Key points
- **Models compared:** seven in total.
  - Four kenlm n-gram models (2-gram to 5-gram) with Modified Kneser-Ney smoothing; unigram counts served only as a frequency control.
  - One word-level LSTM.
  - Two LSTM+5-gram interpolations: one with perplexity-optimal weights of 0.71/0.29, and one with equal weights.
- **Training data:** the "1b corpus" (about 0.8 billion words, vocabulary of roughly 800,000).
- **Perplexity on Dundee (all tokens):** from 73.39 for the optimal interpolation to 290.88 for the bigram model; the LSTM scored 113.27 and the 5-gram 168.98.
- **Reading data:** Dundee Corpus, 10 readers of editorials from The Independent. After exclusions, 289,726 gaze durations remained, and 175 out-of-vocabulary tokens were dropped.
- **Nonlinear GAMM fit:** the gain in log-likelihood ranked models the same way as perplexity, from 297 for the optimal interpolation to 166 for the bigram model. The surprisal curves stayed close to linear, extending the earlier findings of Smith & Levy (2013).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

114 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 110 not yet.

- [Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?](https://md.research.elamur.ai/free/en/papers/oh-2023-why-does-surprisal-larger-transformer/)
- [Determinants of Hesitations and Repetitions in Hindi Spontaneous Speech](https://md.research.elamur.ai/free/en/papers/sharma-2026-determinants-hesitations-repetitions-hindi-spontaneous/)
- [Quantifying the cross-linguistic effects of syncretism on agreement attraction](https://md.research.elamur.ai/free/en/papers/turk-2026-quantifying-cross-linguistic-effects-syncretism/)
- [Testing the Predictions of Surprisal Theory in 11 Languages](https://md.research.elamur.ai/free/en/papers/wilcox-2023-testing-predictions-surprisal-theory-11/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{goodkind2018predictive,
  title = {{Predictive power of word surprisal for reading times is a linear function of language model quality}},
  author = {Goodkind, Adam and Bicknell, Klinton},
  booktitle = {Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018)},
  year = {2018},
  doi = {10.18653/v1/w18-0102},
  url = {https://doi.org/10.18653/v1/w18-0102}
}
```

- [← previous Predicting Factuality of Reporting and Bias of News Media Sources](https://md.research.elamur.ai/free/en/papers/baly-2018-predicting-factuality-reporting-bias-news/)
- [next → Primary and secondary discourse connectives: definitions and lexicons](https://md.research.elamur.ai/free/en/papers/danlos-2018-primary-secondary-discourse-connectives-definitions/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/)

## References

19 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Monsalve, I. F., Frank, S. L., & Vigliocco, G. (2012). Lexical surprisal as a general predictor of reading time. In Proceedings of the 13th conference of the european chapter of the association for computational linguistics (pp. 398–408). [find](https://scholar.google.com/scholar?q=Monsalve%2C+I%2E+F%2E%2C+Frank%2C+S%2E+L%2E%2C+%26+Vigliocco%2C+G%2E+%282012%29%2E+Lexical+surprisal+as+a+general+predictor+of+reading+time%2E+In+Proceedings+of+the+13th+conference+of+the+european+chapter+of+the+association+for+co)
2. Bicknell, K., & Levy, R. (2010). A rational model of eye movement control in reading. In J. Hajivc, S. Carberry, S. Clark, & J. Nivre (Eds.), Proceedings of the 48th annual meeting of the association for computational linguistics (acl) (pp. 1168– 1178). Uppsala, Sweden: Association for Computational… [find](https://scholar.google.com/scholar?q=Bicknell%2C+K%2E%2C+%26+Levy%2C+R%2E+%282010%29%2E+A+rational+model+of+eye+movement+control+in+reading%2E+In+J%2E+Hajivc%2C+S%2E+Carberry%2C+S%2E+Clark%2C+%26+J%2E+Nivre+%28Eds%2E%29%2C+Proceedings+of+the+48th+annual+meeting+of+the+association+)
3. R Core Team. (2013). R: A language and environment for statistical computing [Computer software manual]. Vienna, Austria. Retrieved from [http://www.R-project.org/Chelba](http://www.R-project.org/Chelba), C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., & Robinson, T. (2013). One billion word benchmark for measuring pro… [↗](http://www.R-project.org/Chelba) [our card](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/)
4. Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3), 379–423. Retrieved from http://dx.doi.org/10.1002/ j.1538-7305.1948.tb01338.x doi: 10.1002/j.1538-7305.1948.tb01338.x — [doi:10.1002/j.1538-7305.1948.tb01338.x](https://doi.org/10.1002/j.1538-7305.1948.tb01338.x) [find](https://scholar.google.com/scholar?q=Shannon%2C+C%2E+E%2E+%281948%29%2E+A+mathematical+theory+of+communication%2E+Bell+System+Technical+Journal%2C+27%283%29%2C+379%E2%80%93423%2E+Retrieved+from+http%3A%2F%2Fdx%2Edoi%2Eorg%2F10%2E1002%2F+j%2E1538%2D7305%2E1948%2Etb01338%2Ex+doi%3A+10%2E1002%2Fj%2E1538%2D7)
5. Demberg, V., & Keller, F. (2008). Data from eye-tracking corpora as evidence for theories of syntactic processing complexity. Cognition, 109(2), 193– 210. [find](https://scholar.google.com/scholar?q=Demberg%2C+V%2E%2C+%26+Keller%2C+F%2E+%282008%29%2E+Data+from+eye%2Dtracking+corpora+as+evidence+for+theories+of+syntactic+processing+complexity%2E+Cognition%2C+109%282%29%2C+193%E2%80%93+210%2E)
6. Smith, N., & Levy, R. (2011). Cloze but no cigar: The complex relationship between cloze, corpus, and subjective probabilities in language processing. In Proceedings of the cognitive science society (Vol. 33). [find](https://scholar.google.com/scholar?q=Smith%2C+N%2E%2C+%26+Levy%2C+R%2E+%282011%29%2E+Cloze+but+no+cigar%3A+The+complex+relationship+between+cloze%2C+corpus%2C+and+subjective+probabilities+in+language+processing%2E+In+Proceedings+of+the+cognitive+science+society+%28)
7. Fossum, V., & Levy, R. (2012). Sequential vs. hierarchical syntactic models of human incremental sentence processing. In Proceedings of the 3rd workshop on cognitive modeling and computational linguistics (pp. 61–69). [find](https://scholar.google.com/scholar?q=Fossum%2C+V%2E%2C+%26+Levy%2C+R%2E+%282012%29%2E+Sequential+vs%2E+hierarchical+syntactic+models+of+human+incremental+sentence+processing%2E+In+Proceedings+of+the+3rd+workshop+on+cognitive+modeling+and+computational+linguis)
8. Smith, N., & Levy, R. (2013). The effect of word predictability on reading time is logarithmic. Cognition, 128(3), 302–319. [find](https://scholar.google.com/scholar?q=Smith%2C+N%2E%2C+%26+Levy%2C+R%2E+%282013%29%2E+The+effect+of+word+predictability+on+reading+time+is+logarithmic%2E+Cognition%2C+128%283%29%2C+302%E2%80%93319%2E)
9. Frank, S. L., & Bod, R. (2011). Insensitivity of the human sentence-processing system to hierarchical structure. Psychological science, 22(6), 829–834. [find](https://scholar.google.com/scholar?q=Frank%2C+S%2E+L%2E%2C+%26+Bod%2C+R%2E+%282011%29%2E+Insensitivity+of+the+human+sentence%2Dprocessing+system+to+hierarchical+structure%2E+Psychological+science%2C+22%286%29%2C+829%E2%80%93834%2E)
10. Stolcke, A., Zheng, J., Wang, W., & Abrash, V. (2011). Srilm at sixteen: Update and outlook. In Proceedings of ieee automatic speech recognition and understanding workshop (Vol. 5). [find](https://scholar.google.com/scholar?q=Stolcke%2C+A%2E%2C+Zheng%2C+J%2E%2C+Wang%2C+W%2E%2C+%26+Abrash%2C+V%2E+%282011%29%2E+Srilm+at+sixteen%3A+Update+and+outlook%2E+In+Proceedings+of+ieee+automatic+speech+recognition+and+understanding+workshop+%28Vol%2E+5%29%2E)
11. Frank, S. L., Monsalve, I. F., Thompson, R. L., & Vigliocco, G. (2013). Reading time data for evaluating broad-coverage models of english sentence processing. Behavior Research Methods, 45(4), 1182–1190. [find](https://scholar.google.com/scholar?q=Frank%2C+S%2E+L%2E%2C+Monsalve%2C+I%2E+F%2E%2C+Thompson%2C+R%2E+L%2E%2C+%26+Vigliocco%2C+G%2E+%282013%29%2E+Reading+time+data+for+evaluating+broad%2Dcoverage+models+of+english+sentence+processing%2E+Behavior+Research+Methods%2C+45%284%29%2C+1182%E2%80%9311)
12. Taylor, W. L. (1953). ” Cloze procedure”: a new tool for measuring readability. Journalism quarterly. [find](https://scholar.google.com/scholar?q=Taylor%2C+W%2E+L%2E+%281953%29%2E+%E2%80%9D+Cloze+procedure%E2%80%9D%3A+a+new+tool+for+measuring+readability%2E+Journalism+quarterly%2E)
13. Hale, J. (2001). A probabilistic Earley parser as a psycholinguistic model. In Proceedings of the second meeting of the north american chapter of the association for computational linguistics on language technologies (pp. 1–8). [find](https://scholar.google.com/scholar?q=Hale%2C+J%2E+%282001%29%2E+A+probabilistic+Earley+parser+as+a+psycholinguistic+model%2E+In+Proceedings+of+the+second+meeting+of+the+north+american+chapter+of+the+association+for+computational+linguistics+on+langu) [our card](https://md.research.elamur.ai/free/en/papers/hale-2001-probabilistic-earley-parser-as-psycholinguistic/)
14. Wood, S. N. (2004). Stable and efficient multiple smoothing parameter estimation for generalized additive models. Journal of the American Statistical Association, 99(467), 673-686. [find](https://scholar.google.com/scholar?q=Wood%2C+S%2E+N%2E+%282004%29%2E+Stable+and+efficient+multiple+smoothing+parameter+estimation+for+generalized+additive+models%2E+Journal+of+the+American+Statistical+Association%2C+99%28467%29%2C+673%2D686%2E)
15. Wood, S. N. (2017). Generalized additive models: An introduction with R (2nd ed.). Chapman and Hall/CRC. [find](https://scholar.google.com/scholar?q=Wood%2C+S%2E+N%2E+%282017%29%2E+Generalized+additive+models%3A+An+introduction+with+R+%282nd+ed%2E%29%2E+Chapman+and+Hall%2FCRC%2E)
16. Heafield, K., Pouzyrevsky, I., Clark, J. H., & Koehn, P. (2013, August). Scalable modified Kneser-Ney language model estimation. In Proceedings of the 51st annual meeting of the association for computational linguistics (pp. 690–696). Sofia, Bulgaria. Retrieved from [https://kheafield.com/papers/](https://kheafield.com/papers/) edi… [↗](https://kheafield.com/papers/)
17. Kennedy, A., Hill, R., & Pynte, J. (2003). The dundee corpus. In Proceedings of the 12th european conference on eye movement. [find](https://scholar.google.com/scholar?q=Kennedy%2C+A%2E%2C+Hill%2C+R%2E%2C+%26+Pynte%2C+J%2E+%282003%29%2E+The+dundee+corpus%2E+In+Proceedings+of+the+12th+european+conference+on+eye+movement%2E)
18. Levy, R. (2008). Expectation-based syntactic comprehension. Cognition, 106(3), 1126–1177. [find](https://scholar.google.com/scholar?q=Levy%2C+R%2E+%282008%29%2E+Expectation%2Dbased+syntactic+comprehension%2E+Cognition%2C+106%283%29%2C+1126%E2%80%931177%2E)
19. Mitchell, J., Lapata, M., Demberg, V., & Keller, F. (2010). Syntactic and semantic factors in processing difficulty: An integrated measure. In Proceedings of the 48th annual meeting of the association for computational linguistics (pp. 196–206). [find](https://scholar.google.com/scholar?q=Mitchell%2C+J%2E%2C+Lapata%2C+M%2E%2C+Demberg%2C+V%2E%2C+%26+Keller%2C+F%2E+%282010%29%2E+Syntactic+and+semantic+factors+in+processing+difficulty%3A+An+integrated+measure%2E+In+Proceedings+of+the+48th+annual+meeting+of+the+association)