---
title: "Why Does Surprisal From Larger Transformer-Based — Oh 2023"
description: "Classic psycholinguistic studies found that language models with lower perplexity produce surprisal values that better predict human reading behavior."
authors: ["Byung-Doh Oh", "William Schuler"]
year: 2023
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2023.tacl-1.20/"
doi: "10.1162/tacl_a_00548"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/oh-2023-why-does-surprisal-larger-transformer/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/oh-2023-why-does-surprisal-larger-transformer/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/) / Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?

# Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?

**Authors:** [Byung-Doh Oh](https://md.research.elamur.ai/free/en/authors/byung-doh-oh/), [William Schuler](https://md.research.elamur.ai/free/en/authors/william-schuler/) · **Year:** 2023 · **Venue:** Transactions of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.tacl-1.20/) · [PDF](https://aclanthology.org/2023.tacl-1.20.pdf) · [DOI 10.1162/tacl\_a\_00548](https://doi.org/10.1162/tacl_a_00548) · [OpenAlex](https://openalex.org/W4361766487)

**Citations:** 105 citations (OpenAlex, 2026-10-14) · [106](https://md.research.elamur.ai/free/en/papers/oh-2023-why-does-surprisal-larger-transformer/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Classic psycholinguistic studies found that language models with lower perplexity produce surprisal values that better predict human reading behavior. Earlier work by Oh et al. (2022), using only four GPT-2 sizes, hinted at the reverse. This paper tests that reversal systematically and tries to explain it. The authors computed word surprisal from GPT-2, GPT-Neo and OPT variants. They measured how much each surprisal predictor improved a baseline linear mixed-effects model of self-paced reading times (Natural Stories Corpus) and go-past eye-tracking durations (Dundee Corpus).

Across families, larger and lower-perplexity models fit human data worse, following a monotonic log-linear trend. Residual analysis shows that big models assign very low surprisal to open-class words such as nouns, adjectives and named entities, so regressions underpredict human slowdowns there. Their inflated surprisal coefficients then cause overprediction on function words like modals and conjunctions. The authors link this to large models memorizing content from massive training data. They advise caution, and smaller-model baselines, when using pre-trained LMs as cognitive models.

## Key points
- **Models:** 17 decoder-only Transformer LMs.
  - GPT-2: 4 sizes, about 124M–1558M parameters.
  - GPT-Neo family: 5 models, 125M up to GPT-NeoX 20B, including GPT-J 6B.
  - OPT: 8 sizes, 125M–66B; the 175B model was skipped for compute reasons.
- **Data:**
  - Natural Stories: 181 readers, 10 stories, 770,102 filtered observations.
  - Dundee: 10 readers, 67 editorials, 195,507 observations.
  - Each dataset was split into exploratory and held-out halves, so only a single significance test was needed.
- **Main result:** fit (ΔLL) decreases strictly monotonically as perplexity drops for all GPT-Neo and OPT variants on both corpora. This replicates the GPT-2 finding.
- **Error analysis:** differences in mean squared error between model sizes concentrate on nouns and adjectives (named entities, predicative adjectives), with compensatory overprediction on function words.
- **Resources:** code is released at github.com/byungdoh/llm_surprisal.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [Psycholinguistics & language acquisition](https://md.research.elamur.ai/free/en/topics/psycholinguistics/)

## Institutions

- [The Ohio State University](https://md.research.elamur.ai/free/en/universities/the-ohio-state-university/) — University, Columbus, United States · 46 works · 16 163 citations

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{oh2023why,
  title = {{Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?}},
  author = {Oh, Byung-Doh and Schuler, William},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.1162/tacl\_a\_00548},
  url = {https://doi.org/10.1162/tacl_a_00548}
}
```

- [← previous MADLAD-400: A Multilingual And Document-Level Large Audited Dataset](https://md.research.elamur.ai/free/en/papers/kudugunta-2023-madlad-400-multilingual-document-level/)
- [next → AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors](https://md.research.elamur.ai/free/en/papers/chen-2023-agentverse-facilitating-multi-agent/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/)

## References

48 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Suhas Arehalli, Brian Dillon, and Tal Linzen. 2022. Syntactic surprisal from neural models predicts, but underestimates, human processing difficulty from syntactic ambiguities. In Proceedings of the 26th Conference on Computational Natural Language Learning, pages 301–313. [find](https://scholar.google.com/scholar?q=Suhas+Arehalli%2C+Brian+Dillon%2C+and+Tal+Linzen%2E+2022%2E+Syntactic+surprisal+from+neural+models+predicts%2C+but+underestimates%2C+human+processing+difficulty+from+syntactic+ambiguities%2E+In+Proceedings+of+the+2)
2. Christoph Aurnhammer and Stefan L. Frank. 2019. Comparing gated and simple recurrent neural network architectures as models of human sentence processing. In Proceedings of the 41st Annual Meeting of the Cognitive Science Society, pages 112–118. nition, 109(2):193–210. https://doi.org/10 .1016/j.cogn… [find](https://scholar.google.com/scholar?q=Christoph+Aurnhammer+and+Stefan+L%2E+Frank%2E+2019%2E+Comparing+gated+and+simple+recurrent+neural+network+architectures+as+models+of+human+sentence+processing%2E+In+Proceedings+of+the+41st+Annual+Meeting+of+t)
3. Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1):1–48. https://doi.org/10.18637/jss.v067.i01 [↗](https://doi.org/10.18637/jss.v067.i01) [our card](https://md.research.elamur.ai/free/en/papers/bates-2015-fitting-linear-mixed-effects-models/)
4. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. GPT-NeoX-20B: An open-source autoregr… [find](https://scholar.google.com/scholar?q=Sid+Black%2C+Stella+Biderman%2C+Eric+Hallahan%2C+Quentin+Anthony%2C+Leo+Gao%2C+Laurence+Golding%2C+Horace+He%2C+Connor+Leahy%2C+Kyle+McDonell%2C+Jason+Phang%2C+Michael+Pieler%2C+Usvsn+Sai+Prashanth%2C+Shivanshu+Purohit%2C+Lari) [our card](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/)
5. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large scale autoregressive language modeling with Mesh-Tensorflow. Zenodo. https://doi.org/10.5281/zenodo.5297715 — [doi:10.5281/zenodo.5297715](https://doi.org/10.5281/zenodo.5297715) [↗](https://doi.org/10.5281/zenodo.5297715)
6. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. arXiv preprint, arXiv:2202.07646v2. https://doi.org/10.48550/arXiv.2202.07646 [↗](https://doi.org/10.48550/arXiv.2202.07646)
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional Transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+Transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
8. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 199–209. https://doi.org/10.18653 /v1… [find](https://scholar.google.com/scholar?q=Chris+Dyer%2C+Adhiguna+Kuncoro%2C+Miguel+Ballesteros%2C+and+Noah+A%2E+Smith%2E+2016%2E+Recurrent+neural+network+grammars%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of+the+Association+for)
9. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Ja… [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Neel+Nanda%2C+Catherine+Olsson%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Zac+Hatfield%2DDodds%2C+Da)
10. Jeffrey L. Elman. 1991. Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning, 7:195–225. https://doi.org/10.1007/978-1-4615 -4008-3 5 — [doi:10.1007/978-1-4615](https://doi.org/10.1007/978-1-4615) [↗](https://doi.org/10.1007/978-1-4615)
11. Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Langu… [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+van+Merrie%CC%88nboer%2C+Caglar+Gulcehre%2C+Dzmitry+Bahdanau%2C+Fethi+Bougares%2C+Holger+Schwenk%2C+and+Yoshua+Bengio%2E+2014%2E+Learning+phrase+representations+using+RNN+encoder%E2%80%93decoder+for+statisti)
12. Richard Futrell, Edward Gibson, Harry J. Tily, Idan Blank, Anastasia Vishnevetsky, Steven Piantadosi, and Evelina Fedorenko. 2021. The Natural Stories corpus: A reading-time corpus of English texts containing rare syntactic constructions. Language Resources and Evaluation, 55(1):63–77. [https://doi.o…](https://doi.o…) [↗](https://doi.o…)
13. Christian Clark and William Schuler. 2022. Evidence for composition operations in broad-coverage sentence processing. In 35th Annual Conference on Human Sentence Processing. [find](https://scholar.google.com/scholar?q=Christian+Clark+and+William+Schuler%2E+2022%2E+Evidence+for+composition+operations+in+broad%2Dcoverage+sentence+processing%2E+In+35th+Annual+Conference+on+Human+Sentence+Processing%2E)
14. Edward Gibson. 2000. The Dependency Locality Theory: A distance-based theory of linguistic complexity. In Image, Language, Brain: Papers from the First Mind Articulation Project Symposium, pages 95–126, Cambridge, MA. MIT Press. [find](https://scholar.google.com/scholar?q=Edward+Gibson%2E+2000%2E+The+Dependency+Locality+Theory%3A+A+distance%2Dbased+theory+of+linguistic+complexity%2E+In+Image%2C+Language%2C+Brain%3A+Papers+from+the+First+Mind+Articulation+Project+Symposium%2C+pages+95%E2%80%9312)
15. Vera Demberg and Frank Keller. 2008. Data from eye-tracking corpora as evidence for theories of syntactic processing complexity. [find](https://scholar.google.com/scholar?q=Vera+Demberg+and+Frank+Keller%2E+2008%2E+Data+from+eye%2Dtracking+corpora+as+evidence+for+theories+of+syntactic+processing+complexity%2E)
16. Cog-Adam Goodkind and Klinton Bicknell. 2018. Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics, pages 10–18. https://doi.org/10.18653/v1/W18-0102 pages 141–150.… [↗](https://doi.org/10.18653/v1/W18-0102) [our card](https://md.research.elamur.ai/free/en/papers/goodkind-2018-predictive-power-word-surprisal-reading/)
17. Philip N. Johnson-Laird. 1983. Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness. Harvard University Press, Cambridge, MA. [find](https://scholar.google.com/scholar?q=Philip+N%2E+Johnson%2DLaird%2E+1983%2E+Mental+Models%3A+Towards+a+Cognitive+Science+of+Language%2C+Inference%2C+and+Consciousness%2E+Harvard+University+Press%2C+Cambridge%2C+MA%2E)
18. Alan Kennedy, Robin Hill, and Joël Pynte. 2003. The Dundee Corpus. In Proceedings of the 12th European Conference on Eye Movement. [find](https://scholar.google.com/scholar?q=Alan+Kennedy%2C+Robin+Hill%2C+and+Joe%CC%88l+Pynte%2E+2003%2E+The+Dundee+Corpus%2E+In+Proceedings+of+the+12th+European+Conference+on+Eye+Movement%2E)
19. Michael Hahn, Richard Futrell, Edward Gibson, and Roger P. Levy. 2022. A resource-rational model of human processing of recursive linguistic structure. Proceedings of the National Academy of Sciences, 119(43):e2122602119. https://doi.org/10.1073/pnas.2122602119, PubMed: 36260742 — [doi:10.1073/pnas.2122602119](https://doi.org/10.1073/pnas.2122602119) [↗](https://doi.org/10.1073/pnas.2122602119)
20. Tatsuki Kuribayashi, Yohei Oseki, Ana Brassard, and Kentaro Inui. 2022. Context limitations make neural language models more human-like. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Tatsuki+Kuribayashi%2C+Yohei+Oseki%2C+Ana+Brassard%2C+and+Kentaro+Inui%2E+2022%2E+Context+limitations+make+neural+language+models+more+human%2Dlike%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+N)
21. John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics on Language Technologies, pages 1–8. https://doi.org/10.3115/1073336.1073357 [↗](https://doi.org/10.3115/1073336.1073357) [our card](https://md.research.elamur.ai/free/en/papers/hale-2001-probabilistic-earley-parser-as-psycholinguistic/)
22. Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryo Yoshida, Masayuki Asahara, and Kentaro Inui. 2021. Lower perplexity is not always human-like. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language P… [find](https://scholar.google.com/scholar?q=Tatsuki+Kuribayashi%2C+Yohei+Oseki%2C+Takumi+Ito%2C+Ryo+Yoshida%2C+Masayuki+Asahara%2C+and+Kentaro+Inui%2E+2021%2E+Lower+perplexity+is+not+always+human%2Dlike%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Associa)
23. John Hale, Chris Dyer, Adhiguna Kuncoro, and Jonathan Brennan. 2018. Finding syntax in human encephalography with beam search. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 2727–2736. https://doi.org/10 .18653/v1/P18-1254 [find](https://scholar.google.com/scholar?q=John+Hale%2C+Chris+Dyer%2C+Adhiguna+Kuncoro%2C+and+Jonathan+Brennan%2E+2018%2E+Finding+syntax+in+human+encephalography+with+beam+search%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computat)
24. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. https://doi.org/10.1016/j.cognition .2007.05.006, PubMed: 17662975 — [doi:10.1016/j.cognition](https://doi.org/10.1016/j.cognition) [↗](https://doi.org/10.1016/j.cognition)
25. Yiding Hao, Simon Mendelsohn, Rachel Sterneck, Randi Martinez, and Robert Frank. 2020. Probabilistic predictions of people perusing: Evaluating metrics of language model performance for psycholinguistic modeling. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pag… [find](https://scholar.google.com/scholar?q=Yiding+Hao%2C+Simon+Mendelsohn%2C+Rachel+Sterneck%2C+Randi+Martinez%2C+and+Robert+Frank%2E+2020%2E+Probabilistic+predictions+of+people+perusing%3A+Evaluating+metrics+of+language+model+performance+for+psycholinguist)
26. Richard L. Lewis, Shravan Vasishth, and Julie A. Van Dyke. 2006. Computational principles of working memory in sentence comprehension. Trends in Cognitive Science, 10(10):447–454. https://doi.org/10.1016/j.tics.2006.08.007, PubMed: 16949330 [↗](https://doi.org/10.1016/j.tics.2006.08.007)
27. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735, PubMed: 9377276 [↗](https://doi.org/10.1162/neco.1997.9.8.1735)
28. Danny Merkx and Stefan L. Frank. 2021. Human sentence processing: Recurrence or attention? In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 12–22. https://doi.org/10.18653/v1/2021.cmcl-1.2 [↗](https://doi.org/10.18653/v1/2021.cmcl-1.2)
29. Nora Hollenstein and Lisa Beinborn. 2021. Relative importance in sentence processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Luan Nguyen, Marten van Schijndel, and William… [find](https://scholar.google.com/scholar?q=Nora+Hollenstein+and+Lisa+Beinborn%2E+2021%2E+Relative+importance+in+sentence+processing%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11th+Internatio)
30. Byung-Doh Oh, Christian Clark, and William Schuler. 2021. Surprisal estimators for human reading times need character models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3… [find](https://scholar.google.com/scholar?q=Byung%2DDoh+Oh%2C+Christian+Clark%2C+and+William+Schuler%2E+2021%2E+Surprisal+estimators+for+human+reading+times+need+character+models%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computati)
31. Marten van Schijndel and Tal Linzen. 2021. Single-stage prediction models do not explain the magnitude of syntactic disambiguation difficulty. Cognitive Science, 45(6):e12988. https://doi.org/10.1111/cogs.12988, PubMed: 34170031 — [doi:10.1111/cogs.12988](https://doi.org/10.1111/cogs.12988) [↗](https://doi.org/10.1111/cogs.12988)
32. Byung-Doh Oh, Christian Clark, and William Schuler. 2022. Comparison of structural parsers and neural language models as surprisal estimators. Frontiers in Artificial Intelligence, 5:777963. https://doi.org/10.3389/frai .2022.777963, PubMed: 35310956 — [doi:10.3389/frai](https://doi.org/10.3389/frai) [↗](https://doi.org/10.3389/frai)
33. Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2021. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45)… — [doi:10.1073/pnas](https://doi.org/10.1073/pnas) [find](https://scholar.google.com/scholar?q=Martin+Schrimpf%2C+Idan+Asher+Blank%2C+Greta+Tuckute%2C+Carina+Kauf%2C+Eghbal+A%2E+Hosseini%2C+Nancy+Kanwisher%2C+Joshua+B%2E+Tenenbaum%2C+and+Evelina+Fedorenko%2E+2021%2E+The+neural+architecture+of+language%3A+Integrative+m)
34. Byung-Doh Oh and William Schuler. 2022. Entropy- and distance-based predictors from GPT-2 attention patterns predict reading times over and above GPT-2 surprisal. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9324–9334. [find](https://scholar.google.com/scholar?q=Byung%2DDoh+Oh+and+William+Schuler%2E+2022%2E+Entropy%2D+and+distance%2Dbased+predictors+from+GPT%2D2+attention+patterns+predict+reading+times+over+and+above+GPT%2D2+surprisal%2E+In+Proceedings+of+the+2022+Conference)
35. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1715–1725. https://doi.org/10.18653/v1/P16 -1162 [↗](https://doi.org/10.18653/v1/P16) [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/)
36. Alice Mado Proverbio, Stefania Lilli, Carlo Semenza, and Alberto Zani. 2001. ERP indexes of functional differences in brain activation during proper and common names retrieval. Neuropsychologia, 39(8):815–827. https://doi.org/10.1016/S0028-3932(01)00003-3, PubMed: 11369405 — [doi:10.1016/s0028-3932(01)00003-3](https://doi.org/10.1016/s0028-3932%2801%2900003-3) [↗](https://doi.org/10.1016/S0028-3932(01)00003-3)
37. Cory Shain, Idan Asher Blank, Marten van Schijndel, William Schuler, and Evelina Fedorenko. 2020. fMRI reveals language-specific predictive coding during naturalistic sentence comprehension. Neuropsychologia, 138:107307. https://doi.org/10.1016/j .neuropsychologia.2019.107307, PubMed: 31874149 — [doi:10.1016/j](https://doi.org/10.1016/j) [find](https://scholar.google.com/scholar?q=Cory+Shain%2C+Idan+Asher+Blank%2C+Marten+van+Schijndel%2C+William+Schuler%2C+and+Evelina+Fedorenko%2E+2020%2E+fMRI+reveals+language%2Dspecific+predictive+coding+during+naturalistic+sentence+comprehension%2E+Neuropsyc)
38. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Technical Report. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Technical+Report%2E)
39. Soo Hyun Ryu and Richard L. Lewis. 2021. Accounting for agreement phenomena in sentence comprehension with Transformer language models: Effects of similarity-based interference on surprisal and attention. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 61–71… [find](https://scholar.google.com/scholar?q=Soo+Hyun+Ryu+and+Richard+L%2E+Lewis%2E+2021%2E+Accounting+for+agreement+phenomena+in+sentence+comprehension+with+Transformer+language+models%3A+Effects+of+similarity%2Dbased+interference+on+surprisal+and+attent)
40. Soo Hyun Ryu and Richard L. Lewis. 2022. Using Transformer language model to integrate surprisal, entropy, and working memory retrieval accounts of sentence processing. In 35th Annual Conference on Human Sentence Processing. [find](https://scholar.google.com/scholar?q=Soo+Hyun+Ryu+and+Richard+L%2E+Lewis%2E+2022%2E+Using+Transformer+language+model+to+integrate+surprisal%2C+entropy%2C+and+working+memory+retrieval+accounts+of+sentence+processing%2E+In+35th+Annual+Conference+on+Hu)
41. Cory Shain and William Schuler. 2021. Continuous-time deconvolutional regression for psycholinguistic modeling. Cognition, 215:104735. https://doi.org/10.1016/j .cognition.2021.104735, PubMed: 34303182 — [doi:10.1016/j](https://doi.org/10.1016/j) [find](https://scholar.google.com/scholar?q=Cory+Shain+and+William+Schuler%2E+2021%2E+Continuous%2Dtime+deconvolutional+regression+for+psycholinguistic+modeling%2E+Cognition%2C+215%3A104735%2E+https%3A%2F%2Fdoi%2Eorg%2F10%2E1016%2Fj+%2Ecognition%2E2021%2E104735%2C+PubMed%3A+3430318)
42. Cory Shain, Marten van Schijndel, and William Schuler. 2018. Deep syntactic annotations for broad-coverage psycholinguistic modeling. In Workshop on Linguistic and Neuro-Cognitive Resources. [find](https://scholar.google.com/scholar?q=Cory+Shain%2C+Marten+van+Schijndel%2C+and+William+Schuler%2E+2018%2E+Deep+syntactic+annotations+for+broad%2Dcoverage+psycholinguistic+modeling%2E+In+Workshop+on+Linguistic+and+Neuro%2DCognitive+Resources%2E)
43. Claude Elwood Shannon. 1948. A mathematical theory of communication. Bell System Technical Journal, 27:379–423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability on reading Victor Sanh, Lysandre Debut, Julien Chaumond, and… — [doi:10.1002/j.1538-7305.1948.tb01338.x](https://doi.org/10.1002/j.1538-7305.1948.tb01338.x) [↗](https://doi.org/10.1002/j.1538-7305.1948.tb01338.x)
44. Alan D. Thompson. 2022. What’s in my AI? A comprehensive analysis of datasets used to train GPT-1, GPT-2, GPT-3, GPT-NeoX-20B, Megatron-11B, MT-NLG, and Gopher. Life-Architect.ai Report. the emotional valence of names: An ERP study. Brain and Language, 125(1):118–127. https://doi.org/10.1016/j.bandl… — [doi:10.1016/j.bandl](https://doi.org/10.1016/j.bandl) [↗](https://doi.org/10.1016/j.bandl…)
45. Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger P. Levy. 2020. On the predictive power of neural language models for human real-time comprehension behavior. In Proceedings of the 42nd Annual Meeting of the Cognitive Science Society, pages 1707–1713. [find](https://scholar.google.com/scholar?q=Ethan+Gotlieb+Wilcox%2C+Jon+Gauthier%2C+Jennifer+Hu%2C+Peng+Qian%2C+and+Roger+P%2E+Levy%2E+2020%2E+On+the+predictive+power+of+neural+language+models+for+human+real%2Dtime+comprehension+behavior%2E+In+Proceedings+of+the)
46. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+Neural+Information+Process) [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
47. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: Open pre-trai… [find](https://scholar.google.com/scholar?q=Susan+Zhang%2C+Stephen+Roller%2C+Naman+Goyal%2C+Mikel+Artetxe%2C+Moya+Chen%2C+Shuohui+Chen%2C+Christopher+Dewan%2C+Mona+Diab%2C+Xian+Li%2C+Xi+Victoria+Lin%2C+Todor+Mihaylov%2C+Myle+Ott%2C+Sam+Shleifer%2C+Kurt+Shuster%2C+Daniel+S) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/)
48. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 billion parameter autoregressive language model. [https://github.com/kingoflolz/mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax) Lin Wang, Zude Zhu, Marcel Bastiaansen, Peter Hagoort, and Yufang Yang. 2013. Recognizing [↗](https://github.com/kingoflolz/mesh-transformer-jax)