---
title: "Memory efficiency and resource-rational encoding in sentence"
description: "The authors propose a resource-rational way to limit working memory (WM) in Transformers. 87 references."
authors: ["Weijie Xu", "Brian Dillon", "Richard Futrell"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1550/"
doi: "10.18653/v1/2026.acl-long.1550"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2026-memory-efficiency-resource-rational-encoding/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2026-memory-efficiency-resource-rational-encoding/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/41/) / Memory efficiency and resource-rational encoding in sentence processing

# Memory efficiency and resource-rational encoding in sentence processing

**Authors:** [Weijie Xu](https://md.research.elamur.ai/free/en/authors/weijie-xu/), [Brian Dillon](https://md.research.elamur.ai/free/en/authors/brian-dillon/) and [Richard Futrell](https://md.research.elamur.ai/free/en/authors/richard-futrell/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1550/) · [PDF](https://aclanthology.org/2026.acl-long.1550.pdf) · [DOI 10.18653/v1/2026.acl-long.1550](https://doi.org/10.18653/v1/2026.acl-long.1550) · [OpenAlex](https://openalex.org/W7166811300)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary

Language models used as models of human reading usually have far more memory for context than people do. The authors propose a resource-rational way to limit working memory (WM) in Transformers. Instead of restricting what attention can reach, they make stored information less precise. They add Gaussian noise to the self-attention value vectors of a GPT-2 small model. The noise precision shrinks exponentially as the distance between query and key grows, and each attention head learns its own decay rate. Training on wikitext-103 uses a hybrid objective that combines next-word prediction with an explicit penalty on total encoding precision, weighted by λ. Two main findings emerge:

* **Better fit to reading times.** Constrained models' surprisal predicts human reading times better than unconstrained models.
* **Compressed predictions.** Tighter constraints make next-word predictions more compressed and categorical, with coarser distinctions between contexts.

This offers a testable prediction for psycholinguistics. It also separates how memory is retrieved from how memories are represented, and it links WM limits to quantization in engineering.

## Key points

- Noise model: the precision of value vector *k* seen from position *q* is τ₀·e^(−θ(q−k)), and the decay rate θ is trained separately for each head.
- Lower-layer attention heads learned faster precision decay than higher layers.
- Fit to reading times follows a Goldilocks pattern. It peaks at a moderate constraint, for example λ = 0.01 on the SPR NSC corpus, rather than improving steadily as λ increases.
- Setup: GPT-2 small trained on English wikitext-103. Decay depends only on linear distance, so it ignores hierarchical structure.
- Suggested future work includes crosslinguistic tests, other model sizes, and State Space Models for truly incremental processing.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Massachusetts Amherst](https://md.research.elamur.ai/free/en/universities/university-of-massachusetts-amherst/) — University, Amherst Center, United States · 114 works · 68,731 citations
- [University of California, Irvine](https://md.research.elamur.ai/free/en/universities/university-of-california-irvine/) — University, Irvine, United States · 43 works · 36,565 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🧠 Neuroscience](https://md.research.elamur.ai/free/en/shelves/neuroscience/) · 186 works · [🧠 Cognitive Neuroscience](https://md.research.elamur.ai/free/en/shelves/neuroscience/cognitive-neuroscience/) · 185 works

🧭 Topics (1): [🧒 Psycholinguistics & language acquisition](https://md.research.elamur.ai/free/en/topics/psycholinguistics/) · 108 works

## Cite

Cite as: Weijie Xu, Brian Dillon and Richard Futrell. 2026. Memory efficiency and resource-rational encoding in sentence processing. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xu2026memory,
  title = {{Memory efficiency and resource-rational encoding in sentence processing}},
  author = {Weijie Xu, Brian Dillon and Richard Futrell},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1550},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1550}
}
```

- [← previous Memory Dial: A Training Framework for Controllable Memorization in Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2026-memory-dial-training-framework-controllable/)
- [next → Memory-Driven Role-Playing: Evaluation and Enhancement of Persona Knowledge Utilization in LLMs](https://md.research.elamur.ai/free/en/papers/wang-2026-memory-driven-role-playing-evaluation/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/41/)

## References

87 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. George A Alvarez. 2011. Representing multiple objects as an ensemble enhances visual cognition. Trends in Cognitive Sciences, 15(3):122–131. [find](https://scholar.google.com/scholar?q=George+A+Alvarez%2E+2011%2E+Representing+multiple+objects+as+an+ensemble+enhances+visual+cognition%2E+Trends+in+Cognitive+Sciences%2C+15%283%29%3A122%E2%80%93131%2E)
2. John Robert Anderson. 1990. The Adaptive Character of Thought. Psychology Press. [find](https://scholar.google.com/scholar?q=John+Robert+Anderson%2E+1990%2E+The+Adaptive+Character+of+Thought%2E+Psychology+Press%2E)
3. Aryaman Arora, Dan Jurafsky, and Christopher Potts. 2024. [CausalGym: Benchmarking causal interpretability methods on linguistic tasks](https://md.research.elamur.ai/free/en/papers/arora-2024-causalgym-benchmarking-causal-interpretability-methods/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14638–14663.
4. R Harald Baayen, Douglas J Davidson, and Douglas M Bates. 2008. Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4):390–412. [find](https://scholar.google.com/scholar?q=R+Harald+Baayen%2C+Douglas+J+Davidson%2C+and+Douglas+M+Bates%2E+2008%2E+Mixed%2Deffects+modeling+with+crossed+random+effects+for+subjects+and+items%2E+Journal+of+Memory+and+Language%2C+59%284%29%3A390%E2%80%93412%2E)
5. Brian Bartek, Richard L Lewis, Shravan Vasishth, and Mason R Smith. 2011. In search of on-line locality effects in sentence comprehension. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37(5):1178. [find](https://scholar.google.com/scholar?q=Brian+Bartek%2C+Richard+L+Lewis%2C+Shravan+Vasishth%2C+and+Mason+R+Smith%2E+2011%2E+In+search+of+on%2Dline+locality+effects+in+sentence+comprehension%2E+Journal+of+Experimental+Psychology%3A+Learning%2C+Memory%2C+and+Cog)
6. Christopher J Bates and Robert A Jacobs. 2020. Efficient data compression in perception and perceptual memory. Psychological Review, 127(5):891. [find](https://scholar.google.com/scholar?q=Christopher+J+Bates+and+Robert+A+Jacobs%2E+2020%2E+Efficient+data+compression+in+perception+and+perceptual+memory%2E+Psychological+Review%2C+127%285%29%3A891%2E)
7. Paul M Bays, Raquel FG Catalao, and Masud Husain. 2009. The precision of visual working memory is set by allocation of a shared resource. Journal of Vision, 9(10):7–7. [find](https://scholar.google.com/scholar?q=Paul+M+Bays%2C+Raquel+FG+Catalao%2C+and+Masud+Husain%2E+2009%2E+The+precision+of+visual+working+memory+is+set+by+allocation+of+a+shared+resource%2E+Journal+of+Vision%2C+9%2810%29%3A7%E2%80%937%2E)
8. Paul M Bays, Sebastian Schneegans, Wei Ji Ma, and Timothy F Brady. 2024. Representation and computation in visual working memory. Nature Human Behaviour, pages 1–19. [find](https://scholar.google.com/scholar?q=Paul+M+Bays%2C+Sebastian+Schneegans%2C+Wei+Ji+Ma%2C+and+Timothy+F+Brady%2E+2024%2E+Representation+and+computation+in+visual+working+memory%2E+Nature+Human+Behaviour%2C+pages+1%E2%80%9319%2E)
9. Veronica Boyce, Richard Futrell, and Roger P Levy. 2020. Maze made easy: Better and easier measurement of incremental processing difficulty. Journal of Memory and Language, 111:104082. [find](https://scholar.google.com/scholar?q=Veronica+Boyce%2C+Richard+Futrell%2C+and+Roger+P+Levy%2E+2020%2E+Maze+made+easy%3A+Better+and+easier+measurement+of+incremental+processing+difficulty%2E+Journal+of+Memory+and+Language%2C+111%3A104082%2E)
10. Veronica Boyce and Roger Levy. 2023. A-maze of Natural Stories: Comprehension and surprisal in the Maze task. Glossa Psycholinguistics, 2(1). [find](https://scholar.google.com/scholar?q=Veronica+Boyce+and+Roger+Levy%2E+2023%2E+A%2Dmaze+of+Natural+Stories%3A+Comprehension+and+surprisal+in+the+Maze+task%2E+Glossa+Psycholinguistics%2C+2%281%29%2E)
11. Timothy F Brady, Maria M Robinson, and Jamal R Williams. 2024. Noisy and hierarchical visual memory across timescales. Nature Reviews Psychology, pages 1–17. [find](https://scholar.google.com/scholar?q=Timothy+F+Brady%2C+Maria+M+Robinson%2C+and+Jamal+R+Williams%2E+2024%2E+Noisy+and+hierarchical+visual+memory+across+timescales%2E+Nature+Reviews+Psychology%2C+pages+1%E2%80%9317%2E)
12. Timothy F Brady and Joshua B Tenenbaum. 2013. A probabilistic model of visual working memory: Incorporating higher order regularities into working memory capacity estimates. Psychological Review, 120(1):85. [find](https://scholar.google.com/scholar?q=Timothy+F+Brady+and+Joshua+B+Tenenbaum%2E+2013%2E+A+probabilistic+model+of+visual+working+memory%3A+Incorporating+higher+order+regularities+into+working+memory+capacity+estimates%2E+Psychological+Review%2C+120%28)
13. Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King. 2023. Evidence of a predictive coding hierarchy in the human brain listening to speech. Nature Human Behaviour, 7(3):430–441. [find](https://scholar.google.com/scholar?q=Charlotte+Caucheteux%2C+Alexandre+Gramfort%2C+and+Jean%2DR%C3%A9mi+King%2E+2023%2E+Evidence+of+a+predictive+coding+hierarchy+in+the+human+brain+listening+to+speech%2E+Nature+Human+Behaviour%2C+7%283%29%3A430%E2%80%93441%2E)
14. Noam Chomsky. 1957. Syntactic structures. Mouton de Gruyter. [find](https://scholar.google.com/scholar?q=Noam+Chomsky%2E+1957%2E+Syntactic+structures%2E+Mouton+de+Gruyter%2E)
15. Christian Clark, Byung-Doh Oh, and William Schuler. 2025. Linear recency bias during training improves transformers’ fit to reading times. In Proceedings of the 31st International Conference on Computational Linguistics, pages 7735–7747. [find](https://scholar.google.com/scholar?q=Christian+Clark%2C+Byung%2DDoh+Oh%2C+and+William+Schuler%2E+2025%2E+Linear+recency+bias+during+training+improves+transformers%E2%80%99+fit+to+reading+times%2E+In+Proceedings+of+the+31st+International+Conference+on+Comput)
16. Charles Clifton Jr, Adrian Staub, and Keith Rayner. 2007. Eye movements in reading words and sentences. Eye movements, pages 341–371. [find](https://scholar.google.com/scholar?q=Charles+Clifton+Jr%2C+Adrian+Staub%2C+and+Keith+Rayner%2E+2007%2E+Eye+movements+in+reading+words+and+sentences%2E+Eye+movements%2C+pages+341%E2%80%93371%2E)
17. Tri Dao and Albert Gu. 2024. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Proceedings of the 41st International Conference on Machine Learning, pages 10041–10071. [find](https://scholar.google.com/scholar?q=Tri+Dao+and+Albert+Gu%2E+2024%2E+Transformers+are+SSMs%3A+Generalized+models+and+efficient+algorithms+through+structured+state+space+duality%2E+In+Proceedings+of+the+41st+International+Conference+on+Machine+L)
18. Andrea De Varda and Marco Marelli. 2024. Locally biased transformers better align with human reading times. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 30– 36. [find](https://scholar.google.com/scholar?q=Andrea+De+Varda+and+Marco+Marelli%2E+2024%2E+Locally+biased+transformers+better+align+with+human+reading+times%2E+In+Proceedings+of+the+Workshop+on+Cognitive+Modeling+and+Computational+Linguistics%2C+pages+30)
19. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. LLM.int8(): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318– 30332. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Mike+Lewis%2C+Younes+Belkada%2C+and+Luke+Zettlemoyer%2E+2022%2E+LLM%2Eint8%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A30318%E2%80%93+30)
20. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Fine-tuning of Quantized LLMs. In Advances in Neural Information Processing Systems, volume 36, pages 10088–10115. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Artidoro+Pagnoni%2C+Ari+Holtzman%2C+and+Luke+Zettlemoyer%2E+2023%2E+QLoRA%3A+Efficient+Fine%2Dtuning+of+Quantized+LLMs%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+36%2C+pages+10088%E2%80%931)
21. Stanley Dubinsky and Kemp Williams. 1995. Recategorization of prepositions as complementizers: The case of temporal prepositions in english. Linguistic Inquiry, 26(1):125–137. [find](https://scholar.google.com/scholar?q=Stanley+Dubinsky+and+Kemp+Williams%2E+1995%2E+Recategorization+of+prepositions+as+complementizers%3A+The+case+of+temporal+prepositions+in+english%2E+Linguistic+Inquiry%2C+26%281%29%3A125%E2%80%93137%2E)
22. Jeffrey L Elman. 1990. Finding structure in time. Cognitive Science, 14(2):179–211. [find](https://scholar.google.com/scholar?q=Jeffrey+L+Elman%2E+1990%2E+Finding+structure+in+time%2E+Cognitive+Science%2C+14%282%29%3A179%E2%80%93211%2E)
23. Michael C Frank and Noah D Goodman. 2025. Cognitive modeling using artificial intelligence. Annual Review of Psychology, 77. [find](https://scholar.google.com/scholar?q=Michael+C+Frank+and+Noah+D+Goodman%2E+2025%2E+Cognitive+modeling+using+artificial+intelligence%2E+Annual+Review+of+Psychology%2C+77%2E)
24. Stefan L Frank and Rens Bod. 2011. Insensitivity of the human sentence-processing system to hierarchical structure. Psychological Science, 22(6):829–834. [find](https://scholar.google.com/scholar?q=Stefan+L+Frank+and+Rens+Bod%2E+2011%2E+Insensitivity+of+the+human+sentence%2Dprocessing+system+to+hierarchical+structure%2E+Psychological+Science%2C+22%286%29%3A829%E2%80%93834%2E)
25. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. [find](https://scholar.google.com/scholar?q=Elias+Frantar%2C+Saleh+Ashkboos%2C+Torsten+Hoefler%2C+and+Dan+Alistarh%2E+2022%2E+GPTQ%3A+Accurate+post%2Dtraining+quantization+for+generative+pre%2Dtrained+transformers%2E+arXiv+preprint+arXiv%3A2210%2E17323%2E)
26. Richard Futrell, Edward Gibson, and Roger P Levy. 2020. Lossy-context surprisal: An information-theoretic model of memory effects in sentence processing. Cognitive Science, 44(3):e12814. [find](https://scholar.google.com/scholar?q=Richard+Futrell%2C+Edward+Gibson%2C+and+Roger+P+Levy%2E+2020%2E+Lossy%2Dcontext+surprisal%3A+An+information%2Dtheoretic+model+of+memory+effects+in+sentence+processing%2E+Cognitive+Science%2C+44%283%29%3Ae12814%2E)
27. Richard Futrell, Edward Gibson, Harry J Tily, Idan Blank, Anastasia Vishnevetsky, Steven T Piantadosi, and Evelina Fedorenko. 2021. The Natural Stories corpus: A reading-time corpus of English texts containing rare syntactic constructions. Language Resources and Evaluation, 55:63–77. [find](https://scholar.google.com/scholar?q=Richard+Futrell%2C+Edward+Gibson%2C+Harry+J+Tily%2C+Idan+Blank%2C+Anastasia+Vishnevetsky%2C+Steven+T+Piantadosi%2C+and+Evelina+Fedorenko%2E+2021%2E+The+Natural+Stories+corpus%3A+A+reading%2Dtime+corpus+of+English+texts+c)
28. Richard Futrell and Kyle Mahowald. 2025. How linguistics learned to stop worrying and love the language models. arXiv preprint arXiv:2501.17047. — [arXiv:2501.17047](https://arxiv.org/abs/2501.17047) [find](https://scholar.google.com/scholar?q=Richard+Futrell+and+Kyle+Mahowald%2E+2025%2E+How+linguistics+learned+to+stop+worrying+and+love+the+language+models%2E+arXiv+preprint+arXiv%3A2501%2E17047%2E+%E2%80%94+%5BarXiv%3A2501%2E17047%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2501%2E17047%29)
29. Samuel J Gershman, Eric J Horvitz, and Joshua B Tenenbaum. 2015. Computational rationality: A converging paradigm for intelligence in brains, minds, and machines. Science, 349(6245):273–278. [find](https://scholar.google.com/scholar?q=Samuel+J+Gershman%2C+Eric+J+Horvitz%2C+and+Joshua+B+Tenenbaum%2E+2015%2E+Computational+rationality%3A+A+converging+paradigm+for+intelligence+in+brains%2C+minds%2C+and+machines%2E+Science%2C+349%286245%29%3A273%E2%80%93278%2E)
30. Edward Gibson. 1998. Linguistic complexity: Locality of syntactic dependencies. Cognition, 68(1):1–76. [find](https://scholar.google.com/scholar?q=Edward+Gibson%2E+1998%2E+Linguistic+complexity%3A+Locality+of+syntactic+dependencies%2E+Cognition%2C+68%281%29%3A1%E2%80%9376%2E)
31. Edward Gibson, Neal Pearlmutter, Enriqueta Canseco-Gonzalez, and Gregory Hickok. 1996. Recency preference in the human sentence processing mechanism. Cognition, 59(1):23–59. [find](https://scholar.google.com/scholar?q=Edward+Gibson%2C+Neal+Pearlmutter%2C+Enriqueta+Canseco%2DGonzalez%2C+and+Gregory+Hickok%2E+1996%2E+Recency+preference+in+the+human+sentence+processing+mechanism%2E+Cognition%2C+59%281%29%3A23%E2%80%9359%2E)
32. Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, and 1 others. 2022. Shared computational principles for language processing in humans and deep language models. Nature Neuroscience, 25(3):369–380. [find](https://scholar.google.com/scholar?q=Ariel+Goldstein%2C+Zaid+Zada%2C+Eliav+Buchnik%2C+Mariano+Schain%2C+Amy+Price%2C+Bobbi+Aubrey%2C+Samuel+A+Nastase%2C+Amir+Feder%2C+Dotan+Emanuel%2C+Alon+Cohen%2C+and+1+others%2E+2022%2E+Shared+computational+principles+for+lan)
33. Laura Gwilliams, Alec Marantz, David Poeppel, and Jean-Rémi King. 2025. Hierarchical dynamic coding coordinates speech comprehension in the human brain. Proceedings of the National Academy of Sciences, 122(42):e2422097122. [find](https://scholar.google.com/scholar?q=Laura+Gwilliams%2C+Alec+Marantz%2C+David+Poeppel%2C+and+Jean%2DR%C3%A9mi+King%2E+2025%2E+Hierarchical+dynamic+coding+coordinates+speech+comprehension+in+the+human+brain%2E+Proceedings+of+the+National+Academy+of+Sciences)
34. Michael Hahn, Judith Degen, and Richard Futrell. 2021. Modeling word and morpheme order in natural language as an efficient trade-off of memory and surprisal. Psychological Review, 128(4):726. [find](https://scholar.google.com/scholar?q=Michael+Hahn%2C+Judith+Degen%2C+and+Richard+Futrell%2E+2021%2E+Modeling+word+and+morpheme+order+in+natural+language+as+an+efficient+trade%2Doff+of+memory+and+surprisal%2E+Psychological+Review%2C+128%284%29%3A726%2E)
35. Michael Hahn and Richard Futrell. 2019. Estimating predictive rate–distortion curves via neural variational inference. Entropy, 21(7):640. [find](https://scholar.google.com/scholar?q=Michael+Hahn+and+Richard+Futrell%2E+2019%2E+Estimating+predictive+rate%E2%80%93distortion+curves+via+neural+variational+inference%2E+Entropy%2C+21%287%29%3A640%2E)
36. Michael Hahn, Richard Futrell, Roger Levy, and Edward Gibson. 2022. A resource-rational model of human processing of recursive linguistic structure. Proceedings of the National Academy of Sciences, 119(43):e2122602119. [find](https://scholar.google.com/scholar?q=Michael+Hahn%2C+Richard+Futrell%2C+Roger+Levy%2C+and+Edward+Gibson%2E+2022%2E+A+resource%2Drational+model+of+human+processing+of+recursive+linguistic+structure%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+11)
37. John Hale. 2001. [A probabilistic Earley parser as a psycholinguistic model](https://md.research.elamur.ai/free/en/papers/hale-2001-probabilistic-earley-parser-as-psycholinguistic/). In Second Meeting of the North American Chapter of the Association for Computational Linguistics.
38. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735– 1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+J%C3%BCrgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2C+9%288%29%3A1735%E2%80%93+1780%2E)
39. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 1725–1744. [find](https://scholar.google.com/scholar?q=Jennifer+Hu%2C+Jon+Gauthier%2C+Peng+Qian%2C+Ethan+Wilcox%2C+and+Roger+Levy%2E+2020%2E+A+systematic+assessment+of+syntactic+generalization+in+neural+language+models%2E+In+Proceedings+of+the+58th+annual+meeting+of+th)
40. Andrej Karpathy. 2023. nanoGPT. [https://github.com/karpathy/nanoGPT](https://github.com/karpathy/nanoGPT). [↗](https://github.com/karpathy/nanoGPT)
41. Maayan Keshev, Mandy Cartner, Aya Meltzer-Asscher, and Brian Dillon. 2025. A working memory model of sentence processing as binding morphemes to syntactic positions. Topics in Cognitive Science, 17(1):88– 105. [find](https://scholar.google.com/scholar?q=Maayan+Keshev%2C+Mandy+Cartner%2C+Aya+Meltzer%2DAsscher%2C+and+Brian+Dillon%2E+2025%2E+A+working+memory+model+of+sentence+processing+as+binding+morphemes+to+syntactic+positions%2E+Topics+in+Cognitive+Science%2C+17%281%29)
42. Tatsuki Kuribayashi, Yohei Oseki, Ana Brassard, and Kentaro Inui. 2022. Context limitations make neural language models more human-like. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10421–10436. [find](https://scholar.google.com/scholar?q=Tatsuki+Kuribayashi%2C+Yohei+Oseki%2C+Ana+Brassard%2C+and+Kentaro+Inui%2E+2022%2E+Context+limitations+make+neural+language+models+more+human%2Dlike%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+N)
43. Tatsuki Kuribayashi, Yohei Oseki, Souhaib Ben Taieb, Kentaro Inui, and Timothy Baldwin. 2025. [Large language models are human-like internally](https://md.research.elamur.ai/free/en/papers/kuribayashi-2025-large-language-models-are-human/). Transactions of the Association for Computational Linguistics, 13:1743–1766.
44. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. [find](https://scholar.google.com/scholar?q=Roger+Levy%2E+2008%2E+Expectation%2Dbased+syntactic+comprehension%2E+Cognition%2C+106%283%29%3A1126%E2%80%931177%2E)
45. Richard L Lewis. 1996. Interference in short-term memory: The magical number two (or three) in sentence processing. Journal of Psycholinguistic Research, 25(1):93–115. [find](https://scholar.google.com/scholar?q=Richard+L+Lewis%2E+1996%2E+Interference+in+short%2Dterm+memory%3A+The+magical+number+two+%28or+three%29+in+sentence+processing%2E+Journal+of+Psycholinguistic+Research%2C+25%281%29%3A93%E2%80%93115%2E)
46. Richard L Lewis, Andrew Howes, and Satinder Singh. 2014. Computational rationality: Linking mechanism and behavior through bounded utility maximization. Topics in Cognitive Science, 6(2):279–311. [find](https://scholar.google.com/scholar?q=Richard+L+Lewis%2C+Andrew+Howes%2C+and+Satinder+Singh%2E+2014%2E+Computational+rationality%3A+Linking+mechanism+and+behavior+through+bounded+utility+maximization%2E+Topics+in+Cognitive+Science%2C+6%282%29%3A279%E2%80%93311%2E)
47. Richard L Lewis and Shravan Vasishth. 2005. An activation-based model of sentence processing as skilled memory retrieval. Cognitive Science, 29:375– 419. [find](https://scholar.google.com/scholar?q=Richard+L+Lewis+and+Shravan+Vasishth%2E+2005%2E+An+activation%2Dbased+model+of+sentence+processing+as+skilled+memory+retrieval%2E+Cognitive+Science%2C+29%3A375%E2%80%93+419%2E)
48. Richard L Lewis, Shravan Vasishth, and Julie A Van Dyke. 2006. Computational principles of working memory in sentence comprehension. Trends in Cognitive Sciences, 10(10):447–454. [find](https://scholar.google.com/scholar?q=Richard+L+Lewis%2C+Shravan+Vasishth%2C+and+Julie+A+Van+Dyke%2E+2006%2E+Computational+principles+of+working+memory+in+sentence+comprehension%2E+Trends+in+Cognitive+Sciences%2C+10%2810%29%3A447%E2%80%93454%2E)
49. Falk Lieder and Thomas L Griffiths. 2020. Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources. Behavioral and Brain Sciences, 43:e1. [find](https://scholar.google.com/scholar?q=Falk+Lieder+and+Thomas+L+Griffiths%2E+2020%2E+Resource%2Drational+analysis%3A+Understanding+human+cognition+as+the+optimal+use+of+limited+computational+resources%2E+Behavioral+and+Brain+Sciences%2C+43%3Ae1%2E)
50. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. In Proceedings of Machine Learning and Systems, volume 6, pages 87–100. [find](https://scholar.google.com/scholar?q=Ji+Lin%2C+Jiaming+Tang%2C+Haotian+Tang%2C+Shang+Yang%2C+Wei%2DMing+Chen%2C+Wei%2DChen+Wang%2C+Guangxuan+Xiao%2C+Xingyu+Dang%2C+Chuang+Gan%2C+and+Song+Han%2E+2024%2E+AWQ%3A+Activation%2Daware+weight+quantization+for+on%2Ddevice+LLM+c)
51. Steven G Luke and Kiel Christianson. 2018. The provo corpus: A large eye-tracking corpus with predictability norms. Behavior Research Methods, 50(2):826– 833. [find](https://scholar.google.com/scholar?q=Steven+G+Luke+and+Kiel+Christianson%2E+2018%2E+The+provo+corpus%3A+A+large+eye%2Dtracking+corpus+with+predictability+norms%2E+Behavior+Research+Methods%2C+50%282%29%3A826%E2%80%93+833%2E)
52. Wei Ji Ma, Jeffrey M Beck, Peter E Latham, and Alexandre Pouget. 2006. Bayesian inference with probabilistic population codes. Nature Neuroscience, 9(11):1432–1438. [find](https://scholar.google.com/scholar?q=Wei+Ji+Ma%2C+Jeffrey+M+Beck%2C+Peter+E+Latham%2C+and+Alexandre+Pouget%2E+2006%2E+Bayesian+inference+with+probabilistic+population+codes%2E+Nature+Neuroscience%2C+9%2811%29%3A1432%E2%80%931438%2E)
53. Wei Ji Ma, Masud Husain, and Paul M Bays. 2014. Changing concepts of working memory. Nature Neuroscience, 17(3):347–356. [find](https://scholar.google.com/scholar?q=Wei+Ji+Ma%2C+Masud+Husain%2C+and+Paul+M+Bays%2E+2014%2E+Changing+concepts+of+working+memory%2E+Nature+Neuroscience%2C+17%283%29%3A347%E2%80%93356%2E)
54. Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. 2024. Dissociating language and thought in large language models. Trends in Cognitive Sciences. [find](https://scholar.google.com/scholar?q=Kyle+Mahowald%2C+Anna+A+Ivanova%2C+Idan+A+Blank%2C+Nancy+Kanwisher%2C+Joshua+B+Tenenbaum%2C+and+Evelina+Fedorenko%2E+2024%2E+Dissociating+language+and+thought+in+large+language+models%2E+Trends+in+Cognitive+Sciences%2E)
55. David Marr. 1982. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. Henry Holt and Co., Inc., New York, NY, USA. [find](https://scholar.google.com/scholar?q=David+Marr%2E+1982%2E+Vision%3A+A+Computational+Investigation+into+the+Human+Representation+and+Processing+of+Visual+Information%2E+Henry+Holt+and+Co%2E%2C+Inc%2E%2C+New+York%2C+NY%2C+USA%2E)
56. Brian McElree. 2006. Accessing recent events. Psychology of Learning and Motivation, 46:155–200. [find](https://scholar.google.com/scholar?q=Brian+McElree%2E+2006%2E+Accessing+recent+events%2E+Psychology+of+Learning+and+Motivation%2C+46%3A155%E2%80%93200%2E)
57. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/). arXiv preprint arXiv:1609.07843.
58. George A Miller. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2):81. [find](https://scholar.google.com/scholar?q=George+A+Miller%2E+1956%2E+The+magical+number+seven%2C+plus+or+minus+two%3A+Some+limits+on+our+capacity+for+processing+information%2E+Psychological+Review%2C+63%282%29%3A81%2E)
59. Masato Mita, Ryo Yoshida, and Yohei Oseki. 2025. [Developmentally-plausible working memory shapes a critical period for language acquisition](https://md.research.elamur.ai/free/en/papers/mita-2025-developmentally-plausible-working-memory-shapes/). arXiv preprint arXiv:2502.04795.
60. Dennis Norris. 2006. The Bayesian reader: Explaining word recognition as an optimal Bayesian decision process. Psychological Review, 113(2):327. [find](https://scholar.google.com/scholar?q=Dennis+Norris%2E+2006%2E+The+Bayesian+reader%3A+Explaining+word+recognition+as+an+optimal+Bayesian+decision+process%2E+Psychological+Review%2C+113%282%29%3A327%2E)
61. Klaus Oberauer. 2009. Design for a working memory. Psychology of Learning and Motivation, 51:45–100. [find](https://scholar.google.com/scholar?q=Klaus+Oberauer%2E+2009%2E+Design+for+a+working+memory%2E+Psychology+of+Learning+and+Motivation%2C+51%3A45%E2%80%93100%2E)
62. Byung-Doh Oh and Tal Linzen. 2025. To model human linguistic prediction, make LLMs less superhuman. arXiv preprint arXiv:2510.05141. — [arXiv:2510.05141](https://arxiv.org/abs/2510.05141) [find](https://scholar.google.com/scholar?q=Byung%2DDoh+Oh+and+Tal+Linzen%2E+2025%2E+To+model+human+linguistic+prediction%2C+make+LLMs+less+superhuman%2E+arXiv+preprint+arXiv%3A2510%2E05141%2E+%E2%80%94+%5BarXiv%3A2510%2E05141%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2510%2E05141%29)
63. Byung-Doh Oh and William Schuler. 2022. Entropy-and distance-based predictors from GPT-2 attention patterns predict reading times over and above GPT-2 surprisal. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9324–9334. [find](https://scholar.google.com/scholar?q=Byung%2DDoh+Oh+and+William+Schuler%2E+2022%2E+Entropy%2Dand+distance%2Dbased+predictors+from+GPT%2D2+attention+patterns+predict+reading+times+over+and+above+GPT%2D2+surprisal%2E+In+Proceedings+of+the+2022+Conference+)
64. Byung-Doh Oh and William Schuler. 2023. [Why does surprisal from larger transformer-based language models provide a poorer fit to human reading times?](https://md.research.elamur.ai/free/en/papers/oh-2023-why-does-surprisal-larger-transformer/) Transactions of the Association for Computational Linguistics, 11:336–350.
65. William Timkey and Tal Linzen. 2023. A language model with limited memory capacity captures interference in human sentence processing. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8705–8720. [find](https://scholar.google.com/scholar?q=William+Timkey+and+Tal+Linzen%2E+2023%2E+A+language+model+with+limited+memory+capacity+captures+interference+in+human+sentence+processing%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMN)
66. John C Trueswell, Michael K Tanenhaus, and Susan M Garnsey. 1994. Semantic influences on parsing: Use of thematic role information in syntactic ambiguity resolution. Journal of Memory and Language, 33(3):285–318. [find](https://scholar.google.com/scholar?q=John+C+Trueswell%2C+Michael+K+Tanenhaus%2C+and+Susan+M+Garnsey%2E+1994%2E+Semantic+influences+on+parsing%3A+Use+of+thematic+role+information+in+syntactic+ambiguity+resolution%2E+Journal+of+Memory+and+Language%2C+33)
67. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+and+1+others%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%3A9%2E)
68. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30.
69. Tamar I Regev, Colton Casto, Eghbal A Hosseini, Markus Adamek, Anthony L Ritaccio, Jon T Willie, Peter Brunner, and Evelina Fedorenko. 2024. Neural populations in the language network differ in the size of their temporal receptive windows. Nature Human Behaviour, 8(10):1924–1942. [find](https://scholar.google.com/scholar?q=Tamar+I+Regev%2C+Colton+Casto%2C+Eghbal+A+Hosseini%2C+Markus+Adamek%2C+Anthony+L+Ritaccio%2C+Jon+T+Willie%2C+Peter+Brunner%2C+and+Evelina+Fedorenko%2E+2024%2E+Neural+populations+in+the+language+network+differ+in+the+si)
70. Jesse Vig and Yonatan Belinkov. 2019. [Analyzing the structure of attention in a transformer language model](https://md.research.elamur.ai/free/en/papers/vig-2019-analyzing-structure-attention-transformer-language/). In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63–76, Florence, Italy.
71. Peter J Rousseeuw. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20:53–65. [find](https://scholar.google.com/scholar?q=Peter+J+Rousseeuw%2E+1987%2E+Silhouettes%3A+a+graphical+aid+to+the+interpretation+and+validation+of+cluster+analysis%2E+Journal+of+Computational+and+Applied+Mathematics%2C+20%3A53%E2%80%9365%2E)
72. Soo Hyun Ryu and Richard L Lewis. 2021. Accounting for agreement phenomena in sentence comprehension with transformer language models: Effects of similarity-based interference on surprisal and attention. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 61– 71… [find](https://scholar.google.com/scholar?q=Soo+Hyun+Ryu+and+Richard+L+Lewis%2E+2021%2E+Accounting+for+agreement+phenomena+in+sentence+comprehension+with+transformer+language+models%3A+Effects+of+similarity%2Dbased+interference+on+surprisal+and+attenti)
73. Soo Hyun Ryu and Richard L Lewis. 2025. Memory for prediction: A transformer-based theory of sentence processing. Journal of Memory and Language, 145:104670. [find](https://scholar.google.com/scholar?q=Soo+Hyun+Ryu+and+Richard+L+Lewis%2E+2025%2E+Memory+for+prediction%3A+A+transformer%2Dbased+theory+of+sentence+processing%2E+Journal+of+Memory+and+Language%2C+145%3A104670%2E)
74. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and 1 others. 2023. Findings of the babylm challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of the BabyL… [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Aaron+Mueller%2C+Leshem+Choshen%2C+Ethan+Wilcox%2C+Chengxu+Zhuang%2C+Juan+Ciro%2C+Rafael+Mosquera%2C+Bhargavi+Paranjabe%2C+Adina+Williams%2C+Tal+Linzen%2C+and+1+others%2E+2023%2E+Findings+of+the+babylm+chall)
75. Ethan G Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P Levy. 2023. [Testing the predictions of surprisal theory in 11 languages](https://md.research.elamur.ai/free/en/papers/wilcox-2023-testing-predictions-surprisal-theory-11/). Transactions of the Association for Computational Linguistics, 11:1451–1470.
76. Jacqueline S Sachs. 1974. Memory in reading and listening to discourse. Memory & Cognition, 2(1):95–100. [find](https://scholar.google.com/scholar?q=Jacqueline+S+Sachs%2E+1974%2E+Memory+in+reading+and+listening+to+discourse%2E+Memory+%26+Cognition%2C+2%281%29%3A95%E2%80%93100%2E)
77. Weijie Xu, Jason Chon, Tianran Liu, and Richard Futrell. 2023. The linearity of the effect of surprisal on reading times across languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15711–15721. [find](https://scholar.google.com/scholar?q=Weijie+Xu%2C+Jason+Chon%2C+Tianran+Liu%2C+and+Richard+Futrell%2E+2023%2E+The+linearity+of+the+effect+of+surprisal+on+reading+times+across+languages%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A)
78. Elizabeth R Schotter and Brian Dillon. 2025. A beginner’s guide to eye tracking for psycholinguistic studies of reading. Behavior Research Methods, 57(2):68. [find](https://scholar.google.com/scholar?q=Elizabeth+R+Schotter+and+Brian+Dillon%2E+2025%2E+A+beginner%E2%80%99s+guide+to+eye+tracking+for+psycholinguistic+studies+of+reading%2E+Behavior+Research+Methods%2C+57%282%29%3A68%2E)
79. Weijie Xu and Richard Futrell. 2026. Strategic resource allocation in memory encoding: An efficiency principle shaping language processing. Journal of Memory and Language, 146:104706. [find](https://scholar.google.com/scholar?q=Weijie+Xu+and+Richard+Futrell%2E+2026%2E+Strategic+resource+allocation+in+memory+encoding%3A+An+efficiency+principle+shaping+language+processing%2E+Journal+of+Memory+and+Language%2C+146%3A104706%2E)
80. Herbert A Simon. 1955. A behavioral model of rational choice. The Quarterly Journal of Economics, pages 99–118. [find](https://scholar.google.com/scholar?q=Herbert+A+Simon%2E+1955%2E+A+behavioral+model+of+rational+choice%2E+The+Quarterly+Journal+of+Economics%2C+pages+99%E2%80%93118%2E)
81. Noga Zaslavsky, Charles Kemp, Terry Regier, and Naftali Tishby. 2018. Efficient compression in color naming and its evolution. Proceedings of the National Academy of Sciences, 115(31):7937–7942. [find](https://scholar.google.com/scholar?q=Noga+Zaslavsky%2C+Charles+Kemp%2C+Terry+Regier%2C+and+Naftali+Tishby%2E+2018%2E+Efficient+compression+in+color+naming+and+its+evolution%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+115%2831%29%3A7937%E2%80%937942%2E)
82. Garrett Smith and Shravan Vasishth. 2020. A principled approach to feature selection in models of sentence processing. Cognitive Science, 44(12):e12918. [find](https://scholar.google.com/scholar?q=Garrett+Smith+and+Shravan+Vasishth%2E+2020%2E+A+principled+approach+to+feature+selection+in+models+of+sentence+processing%2E+Cognitive+Science%2C+44%2812%29%3Ae12918%2E)
83. Robyn Speer. 2022. rspeer/wordfreq: v3.0. [find](https://scholar.google.com/scholar?q=Robyn+Speer%2E+2022%2E+rspeer%2Fwordfreq%3A+v3%2E0%2E)
84. Junyuan Zhao, Ruimin Gao, and Jonathan R Brennan. 2025. Decoding the neural dynamics of headed syntactic structure building. Journal of Neuroscience, 45(17). [find](https://scholar.google.com/scholar?q=Junyuan+Zhao%2C+Ruimin+Gao%2C+and+Jonathan+R+Brennan%2E+2025%2E+Decoding+the+neural+dynamics+of+headed+syntactic+structure+building%2E+Journal+of+Neuroscience%2C+45%2817%29%2E)
85. Susanne Still. 2014. Information bottleneck approach to predictive inference. Entropy, 16(2):968–989. [find](https://scholar.google.com/scholar?q=Susanne+Still%2E+2014%2E+Information+bottleneck+approach+to+predictive+inference%2E+Entropy%2C+16%282%29%3A968%E2%80%93989%2E)
86. A James V Stone. 2018. Principles of neural information theory: Computational neuroscience and metabolic efficiency. Sebtel Press. [find](https://scholar.google.com/scholar?q=A+James+V+Stone%2E+2018%2E+Principles+of+neural+information+theory%3A+Computational+neuroscience+and+metabolic+efficiency%2E+Sebtel+Press%2E)
87. Training procedure. Models are trained on Wikitext-103 dataset (Merity et al., 2016) with 16000 training steps. The training procedure Model training with all items from the inanimate category (i.e., cross-category distance). Then, the separatability between the two categories of a linguistic featur… [find](https://scholar.google.com/scholar?q=Training+procedure%2E+Models+are+trained+on+Wikitext%2D103+dataset+%28Merity+et+al%2E%2C+2016%29+with+16000+training+steps%2E+The+training+procedure+Model+training+with+all+items+from+the+inanimate+category+%28i%2Ee%2E%2C+)

*Updated 2 November 2026*