---
title: "Revisiting the Uniform Information Density Hypothesis (2021)"
description: "The uniform information density (UID) hypothesis claims that people prefer utterances in which… 61 references, cited by 62 papers here."
authors: ["Clara Meister", "Tiago Pimentel", "Patrick Haller", "Lena Jäger", "Ryan Cotterell", "Roger Levy"]
year: 2021
venue: "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2021.emnlp-main.74/"
doi: "10.18653/v1/2021.emnlp-main.74"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/meister-2021-revisiting-uniform-information-density-hypothesis/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/meister-2021-revisiting-uniform-information-density-hypothesis/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Revisiting the Uniform Information Density Hypothesis

# Revisiting the Uniform Information Density Hypothesis

**Authors:** [Clara Meister](https://md.research.elamur.ai/free/en/authors/clara-meister/), [Tiago Pimentel](https://md.research.elamur.ai/free/en/authors/tiago-pimentel/), Patrick Haller, Lena Jäger, [Ryan Cotterell](https://md.research.elamur.ai/free/en/authors/ryan-cotterell/) and [Roger Levy](https://md.research.elamur.ai/free/en/authors/roger-levy/) · **Year:** 2021 · **Venue:** [Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2021](https://md.research.elamur.ai/free/en/venues/emnlp/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.emnlp-main.74/) · [PDF](https://aclanthology.org/2021.emnlp-main.74.pdf) · [DOI 10.18653/v1/2021.emnlp-main.74](https://doi.org/10.18653/v1/2021.emnlp-main.74) · [OpenAlex](https://openalex.org/W3204890648)

**Citations:** 61 citations (OpenAlex, 2026-10-14) · [62](https://md.research.elamur.ai/free/en/papers/meister-2021-revisiting-uniform-information-density-hypothesis/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The uniform information density (UID) hypothesis claims that people prefer utterances in which information, measured as surprisal, is spread evenly. Most earlier evidence comes from language production. Two questions have stayed open: how evenness should be measured, and over what span it should hold (phrase, sentence, document or the whole language). The authors test UID from the comprehension side. They use sentence-level reading times from reading corpora, including eye-tracking data, and sentence acceptability judgments. They fit regression models whose predictor is the sum of surprisals raised to a power k, scanning a fine-grained range of k. They then compare competing ways of quantifying uniformity.

For reading times, a linear surprisal effect fits well, but values of k slightly above 1 fit about as well and cannot be excluded. Acceptability shows a clearer super-linear pattern: sentences with uneven information are rated worse. Measures based on a language-wide mean surprisal beat local ones, which supports the channel-capacity view of UID. The work gives a principled basis for picking UID metrics in psycholinguistics and NLP.

## Key points
- **Core predictor:** the predictor is the sum of surprisal^k over a sentence.
  - For acceptability, correlation rises when k > 1 (Fig. 1).
  - For reading times, k slightly above 1 matches or beats k = 1, most visibly in the Provo corpus.
- **Analysis design:** the analysis works at the sentence level and counts re-reading after the first pass. A finer k grid replaces the coarser quadratic test of Smith and Levy (2013).
- **Operationalizations compared:** three families were tested:
  - super-linear surprisal;
  - deviation from a target mean µ_c (global variability);
  - differences between neighbouring surprisals (local variability, after Collins 2014).
  
  The super-linear form explains psychometric data best.
- **Scope of the mean:** the target mean was computed at sentence, document and language level, with µ_lang estimated over WikiText 103. Higher-level means predicted reading times better in every corpus.
- **Robustness check and limitations:** replacing surprisal with negative raw probability (prompted by Brothers and Kuperberg 2021) gave worse fits.
  - The data cover only written English, plus Dutch in the appendix.
  - Code: github.com/rycolab/revisiting-uid.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Cambridge](https://md.research.elamur.ai/free/en/universities/university-of-cambridge/) — University, Cambridge, United Kingdom · 155 works · 115,562 citations
- [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) — University, Cambridge, United States · 159 works · 65,157 citations
- [University of Zurich](https://md.research.elamur.ai/free/en/universities/university-of-zurich/) — University, Zurich, Switzerland · 50 works · 11,138 citations
- [ETH Zurich](https://md.research.elamur.ai/free/en/universities/eth-zurich/) — University, Zurich, Switzerland · 36 works · 4,534 citations
- [University of Potsdam](https://md.research.elamur.ai/free/en/universities/university-of-potsdam/) — University, Potsdam, Germany · 36 works · 5,236 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🧠 Neuroscience](https://md.research.elamur.ai/free/en/shelves/neuroscience/) · 95 works · [🧠 Cognitive Neuroscience](https://md.research.elamur.ai/free/en/shelves/neuroscience/cognitive-neuroscience/) · 95 works

🧭 Topics (1): [🧒 Psycholinguistics & language acquisition](https://md.research.elamur.ai/free/en/topics/psycholinguistics/) · 66 works

## Cite

Cite as: Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell and Roger Levy. 2021. Revisiting the Uniform Information Density Hypothesis. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{meister2021revisiting,
  title = {{Revisiting the Uniform Information Density Hypothesis}},
  author = {Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell and Roger Levy},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.74},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.74}
}
```

- [← previous Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization](https://md.research.elamur.ai/free/en/papers/gan-2021-exploring-underexplored-limitations-cross-domain/)
- [next → DocNLI: A Large-scale Dataset for Document-level Natural Language Inference](https://md.research.elamur.ai/free/en/papers/yin-2021-docnli-large-scale-dataset-document/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

61 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Matthew Aylett and Alice Turk. 2004. The smooth signal redundancy hypothesis: A functional explanation for relationships between redundancy, prosodic prominence, and duration in spontaneous speech. Language and Speech, 47(1):31–56. [find](https://scholar.google.com/scholar?q=Matthew+Aylett+and+Alice+Turk%2E+2004%2E+The+smooth+signal+redundancy+hypothesis%3A+A+functional+explanation+for+relationships+between+redundancy%2C+prosodic+prominence%2C+and+duration+in+spontaneous+speech%2E+La)
2. Alan Bell, Daniel Jurafsky, Eric Fosler-Lussier, Cynthia Girand, Michelle Gregory, and Daniel Gildea. 2003. Effects of disfluencies, predictability, and utterance position on word form variation in English conversation. The Journal of the Acoustical Society of America, 113(2):1001–1024. [find](https://scholar.google.com/scholar?q=Alan+Bell%2C+Daniel+Jurafsky%2C+Eric+Fosler%2DLussier%2C+Cynthia+Girand%2C+Michelle+Gregory%2C+and+Daniel+Gildea%2E+2003%2E+Effects+of+disfluencies%2C+predictability%2C+and+utterance+position+on+word+form+variation+in+En)
3. Jelke Bloem. 2016. Testing the processing hypothesis of word order variation using a probabilistic language model. In Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity (CL4LC), pages 174–185. [find](https://scholar.google.com/scholar?q=Jelke+Bloem%2E+2016%2E+Testing+the+processing+hypothesis+of+word+order+variation+using+a+probabilistic+language+model%2E+In+Proceedings+of+the+Workshop+on+Computational+Linguistics+for+Linguistic+Complexity)
4. The COLING 2016 Organizing Committee. [find](https://scholar.google.com/scholar?q=The+COLING+2016+Organizing+Committee%2E)
5. Trevor Brothers and Gina R. Kuperberg. 2021. Word predictability effects are linear, not logarithmic: Implications for probabilistic models of sentence comprehension. Journal of Memory and Language, 116:104174. [find](https://scholar.google.com/scholar?q=Trevor+Brothers+and+Gina+R%2E+Kuperberg%2E+2021%2E+Word+predictability+effects+are+linear%2C+not+logarithmic%3A+Implications+for+probabilistic+models+of+sentence+comprehension%2E+Journal+of+Memory+and+Language%2C+1)
6. Harm Brouwer, Hartmut Fitz, and John Hoeks. 2010. Modeling the noun phrase versus sentence coordination ambiguity in Dutch: Evidence from surprisal theory. In Proceedings of the 2010 Workshop on Cognitive Modeling and Computational Linguistics, pages 72–80. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Harm+Brouwer%2C+Hartmut+Fitz%2C+and+John+Hoeks%2E+2010%2E+Modeling+the+noun+phrase+versus+sentence+coordination+ambiguity+in+Dutch%3A+Evidence+from+surprisal+theory%2E+In+Proceedings+of+the+2010+Workshop+on+Cogni)
7. Michael Xavier Collins. 2014. Information density and dependency length as complementary cognitive models. Journal of Psycholinguistic Research, 43(5):651–681. [find](https://scholar.google.com/scholar?q=Michael+Xavier+Collins%2E+2014%2E+Information+density+and+dependency+length+as+complementary+cognitive+models%2E+Journal+of+Psycholinguistic+Research%2C+43%285%29%3A651%E2%80%93681%2E)
8. Uschi Cop, Nicolas Dirix, Denis Drieghe, and Wouter Duyck. 2017. Presenting GECO: An eyetracking corpus of monolingual and bilingual sentence reading. Behavior Research Methods, 49(2):602–615. [find](https://scholar.google.com/scholar?q=Uschi+Cop%2C+Nicolas+Dirix%2C+Denis+Drieghe%2C+and+Wouter+Duyck%2E+2017%2E+Presenting+GECO%3A+An+eyetracking+corpus+of+monolingual+and+bilingual+sentence+reading%2E+Behavior+Research+Methods%2C+49%282%29%3A602%E2%80%93615%2E)
9. Christophe Coupé, Yoon Mi Oh, Dan Dediu, and François Pellegrino. 2019. Different languages, similar encoding efficiency: Comparable information rates across the human communicative niche. Science Advances, 5(9). [find](https://scholar.google.com/scholar?q=Christophe+Coupe%CC%81%2C+Yoon+Mi+Oh%2C+Dan+Dediu%2C+and+Franc%CC%A7ois+Pellegrino%2E+2019%2E+Different+languages%2C+similar+encoding+efficiency%3A+Comparable+information+rates+across+the+human+communicative+niche%2E+Science+A)
10. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988. Association for Comput… [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Quoc+Le%2C+and+Ruslan+Salakhutdinov%2E+2019%2E+Transformer%2DXL%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+In+Proceedings+of+the+57th+Annu)
11. Wietse de Vries and Malvina Nissim. 2021. [As good as new. how to successfully recycle English GPT-2 to make models for other languages](https://md.research.elamur.ai/free/en/papers/vries-2021-as-good-as-new-how/). In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 836–846. Association for Computational Linguistics.
12. Vera Demberg and Frank Keller. 2008. Data from eye-tracking corpora as evidence for theories of syntactic processing complexity. Cognition, 109(2):193–210. [find](https://scholar.google.com/scholar?q=Vera+Demberg+and+Frank+Keller%2E+2008%2E+Data+from+eye%2Dtracking+corpora+as+evidence+for+theories+of+syntactic+processing+complexity%2E+Cognition%2C+109%282%29%3A193%E2%80%93210%2E)
13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
14. August Fenk and Gertraud Fenk. 1980. Konstanz im Kurzzeitgedächtnis - Konstanz im sprachlichen Informationsfluß? Zeitschrift für Experimentelle und Angewandte Psychologie, 27(3):400–414. [find](https://scholar.google.com/scholar?q=August+Fenk+and+Gertraud+Fenk%2E+1980%2E+Konstanz+im+Kurzzeitgeda%CC%88chtnis+%2D+Konstanz+im+sprachlichen+Informationsflu%C3%9F%3F+Zeitschrift+fu%CC%88r+Experimentelle+und+Angewandte+Psychologie%2C+27%283%29%3A400%E2%80%93414%2E)
15. Irene Fernandez Monsalve, Stefan L. Frank, and Gabriella Vigliocco. 2012. Lexical surprisal as a general predictor of reading time. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 398–408. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Irene+Fernandez+Monsalve%2C+Stefan+L%2E+Frank%2C+and+Gabriella+Vigliocco%2E+2012%2E+Lexical+surprisal+as+a+general+predictor+of+reading+time%2E+In+Proceedings+of+the+13th+Conference+of+the+European+Chapter+of+the)
16. Victoria Fossum and Roger Levy. 2012. Sequential vs. hierarchical syntactic models of human incremental sentence processing. In Proceedings of the 3rd Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2012), pages 61–69. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Victoria+Fossum+and+Roger+Levy%2E+2012%2E+Sequential+vs%2E+hierarchical+syntactic+models+of+human+incremental+sentence+processing%2E+In+Proceedings+of+the+3rd+Workshop+on+Cognitive+Modeling+and+Computational+)
17. A. F. Frank and T. Jaeger. 2008. Speaking rationally: Uniform information density as an optimal strategy for language production. In the Annual Meeting of the Cognitive Science Society. [find](https://scholar.google.com/scholar?q=A%2E+F%2E+Frank+and+T%2E+Jaeger%2E+2008%2E+Speaking+rationally%3A+Uniform+information+density+as+an+optimal+strategy+for+language+production%2E+In+the+Annual+Meeting+of+the+Cognitive+Science+Society%2E)
18. Stefan L. Frank and Rens Bod. 2011. Insensitivity of the human sentence-processing system to hierarchical structure. Psychological Science, 22(6):829– 834. [find](https://scholar.google.com/scholar?q=Stefan+L%2E+Frank+and+Rens+Bod%2E+2011%2E+Insensitivity+of+the+human+sentence%2Dprocessing+system+to+hierarchical+structure%2E+Psychological+Science%2C+22%286%29%3A829%E2%80%93+834%2E)
19. Stefan L. Frank, Leun J. Otten, Giulia Galli, and Gabriella Vigliocco. 2013. Word surprisal predicts n400 amplitude during reading. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 878–883. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Stefan+L%2E+Frank%2C+Leun+J%2E+Otten%2C+Giulia+Galli%2C+and+Gabriella+Vigliocco%2E+2013%2E+Word+surprisal+predicts+n400+amplitude+during+reading%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+Association+for+Com)
20. Richard Futrell, Edward Gibson, Harry J. Tily, Idan Blank, Anastasia Vishnevetsky, Steven Piantadosi, and Evelina Fedorenko. 2018. The Natural Stories Corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Associa… [find](https://scholar.google.com/scholar?q=Richard+Futrell%2C+Edward+Gibson%2C+Harry+J%2E+Tily%2C+Idan+Blank%2C+Anastasia+Vishnevetsky%2C+Steven+Piantadosi%2C+and+Evelina+Fedorenko%2E+2018%2E+The+Natural+Stories+Corpus%2E+In+Proceedings+of+the+Eleventh+Internatio)
21. Dmitriy Genzel and Eugene Charniak. 2002. [Entropy rate constancy in text](https://md.research.elamur.ai/free/en/papers/genzel-2002-entropy-rate-constancy-text/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 199–206. Association for Computational Linguistics.
22. Adam Goodkind and Klinton Bicknell. 2018a. [Predictive power of word surprisal for reading times is a linear function of language model quality](https://md.research.elamur.ai/free/en/papers/goodkind-2018-predictive-power-word-surprisal-reading/). In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018), pages 10–18.
23. Adam Goodkind and Klinton Bicknell. 2018b. [Predictive power of word surprisal for reading times is a linear function of language model quality](https://md.research.elamur.ai/free/en/papers/goodkind-2018-predictive-power-word-surprisal-reading/). In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018), pages 10–18. Association for Computational Linguistics.
24. John Hale. 2001. [A probabilistic Earley parser as a psycholinguistic model](https://md.research.elamur.ai/free/en/papers/hale-2001-probabilistic-earley-parser-as-psycholinguistic/). In Second Meeting of the North American Chapter of the Association for Computational Linguistics.
25. Kenneth Heafield. 2011. [KenLM: Faster and smaller language model queries](https://md.research.elamur.ai/free/en/papers/heafield-2011-kenlm-faster-smaller-language-model/). In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187–197. Association for Computational Linguistics.
26. T. Jaeger. 2010. Redundancy and reduction: Speakers manage syntactic information density. Cognitive Psychology, 61:23–62. [find](https://scholar.google.com/scholar?q=T%2E+Jaeger%2E+2010%2E+Redundancy+and+reduction%3A+Speakers+manage+syntactic+information+density%2E+Cognitive+Psychology%2C+61%3A23%E2%80%9362%2E)
27. Ayush Jain, Vishal Singh, Sidharth Ranjan, Rajakrishnan Rajkumar, and Sumeet Agarwal. 2018. Uniform Information Density effects on syntactic choice in Hindi. In Proceedings of the Workshop on Linguistic Complexity and Natural Language Processing, pages 38–48. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Ayush+Jain%2C+Vishal+Singh%2C+Sidharth+Ranjan%2C+Rajakrishnan+Rajkumar%2C+and+Sumeet+Agarwal%2E+2018%2E+Uniform+Information+Density+effects+on+syntactic+choice+in+Hindi%2E+In+Proceedings+of+the+Workshop+on+Linguist)
28. Alan Kennedy, Robin Hill, and Joel Pynte. 2003. The Dundee Corpus. In Proceedings of the 12th European Conference on Eye Movements. [find](https://scholar.google.com/scholar?q=Alan+Kennedy%2C+Robin+Hill%2C+and+Joel+Pynte%2E+2003%2E+The+Dundee+Corpus%2E+In+Proceedings+of+the+12th+European+Conference+on+Eye+Movements%2E)
29. Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryo Yoshida, Masayuki Asahara, and Kentaro Inui. 2021. Lower perplexity is not always human-like. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language P… [find](https://scholar.google.com/scholar?q=Tatsuki+Kuribayashi%2C+Yohei+Oseki%2C+Takumi+Ito%2C+Ryo+Yoshida%2C+Masayuki+Asahara%2C+and+Kentaro+Inui%2E+2021%2E+Lower+perplexity+is+not+always+human%2Dlike%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Associa)
30. Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive Science, 41(5):1202–1241. [find](https://scholar.google.com/scholar?q=Jey+Han+Lau%2C+Alexander+Clark%2C+and+Shalom+Lappin%2E+2017%2E+Grammaticality%2C+acceptability%2C+and+probability%3A+A+probabilistic+view+of+linguistic+knowledge%2E+Cognitive+Science%2C+41%285%29%3A1202%E2%80%931241%2E)
31. Roger Levy. 2005. Probabilistic Models of Word Order and Syntactic Discontinuity. Ph.D. thesis, Stanford University. [find](https://scholar.google.com/scholar?q=Roger+Levy%2E+2005%2E+Probabilistic+Models+of+Word+Order+and+Syntactic+Discontinuity%2E+Ph%2ED%2E+thesis%2C+Stanford+University%2E)
32. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. [find](https://scholar.google.com/scholar?q=Roger+Levy%2E+2008%2E+Expectation%2Dbased+syntactic+comprehension%2E+Cognition%2C+106%283%29%3A1126%E2%80%931177%2E)
33. Roger Levy and T. Florian Jaeger. 2007. Speakers optimize information density through syntactic reduction. In Advances in Neural Information Processing Systems, volume 19. MIT Press. [find](https://scholar.google.com/scholar?q=Roger+Levy+and+T%2E+Florian+Jaeger%2E+2007%2E+Speakers+optimize+information+density+through+syntactic+reduction%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+19%2E+MIT+Press%2E)
34. Roger P. Levy. 2018. Communicative efficiency, Uniform Information Density, and the Rational Speech Act Theory. In 40th Annual Meeting of the Cognitive Science Society, pages 684–689. Cognitive Science Society. [find](https://scholar.google.com/scholar?q=Roger+P%2E+Levy%2E+2018%2E+Communicative+efficiency%2C+Uniform+Information+Density%2C+and+the+Rational+Speech+Act+Theory%2E+In+40th+Annual+Meeting+of+the+Cognitive+Science+Society%2C+pages+684%E2%80%93689%2E+Cognitive+Scienc)
35. Steven G. Luke and Kiel Christianson. 2018. The Provo Corpus: A large eye-tracking corpus with predictability norms. Behavior Research Methods, 50(2):826–833. [find](https://scholar.google.com/scholar?q=Steven+G%2E+Luke+and+Kiel+Christianson%2E+2018%2E+The+Provo+Corpus%3A+A+large+eye%2Dtracking+corpus+with+predictability+norms%2E+Behavior+Research+Methods%2C+50%282%29%3A826%E2%80%93833%2E)
36. Kyle Mahowald, Evelina Fedorenko, Steven T. Piantadosi, and Edward Gibson. 2013. Info/information theory: Speakers choose shorter words in predictive contexts. Cognition, 126(2):313–318. [find](https://scholar.google.com/scholar?q=Kyle+Mahowald%2C+Evelina+Fedorenko%2C+Steven+T%2E+Piantadosi%2C+and+Edward+Gibson%2E+2013%2E+Info%2Finformation+theory%3A+Speakers+choose+shorter+words+in+predictive+contexts%2E+Cognition%2C+126%282%29%3A313%E2%80%93318%2E)
37. Clara Meister, Ryan Cotterell, and Tim Vieira. 2020. [If beam search is the answer, what was the question?](https://md.research.elamur.ai/free/en/papers/meister-2020-if-beam-search-is-answer/) In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2173–2185. Association for Computational Linguistics.
38. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/). In 5th International Conference on Learning Representations.
39. Jeff Mitchell, Mirella Lapata, Vera Demberg, and Frank Keller. 2010. Syntactic and semantic factors in processing difficulty: An integrated measure. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 196– 206. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jeff+Mitchell%2C+Mirella+Lapata%2C+Vera+Demberg%2C+and+Frank+Keller%2E+2010%2E+Syntactic+and+semantic+factors+in+processing+difficulty%3A+An+integrated+measure%2E+In+Proceedings+of+the+48th+Annual+Meeting+of+the+As)
40. Hermann Ney, Ute Essen, and Reinhard Kneser. 1994. On structuring probabilistic dependences in stochastic language modelling. Computer Speech and Language, 8(1):1–38. [find](https://scholar.google.com/scholar?q=Hermann+Ney%2C+Ute+Essen%2C+and+Reinhard+Kneser%2E+1994%2E+On+structuring+probabilistic+dependences+in+stochastic+language+modelling%2E+Computer+Speech+and+Language%2C+8%281%29%3A1%E2%80%9338%2E)
41. Irene Nikkarinen, Tiago Pimentel, Damián Blasi, and Ryan Cotterell. 2021. Modelling the unigram distribution. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3721– 3729. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Irene+Nikkarinen%2C+Tiago+Pimentel%2C+Damia%CC%81n+Blasi%2C+and+Ryan+Cotterell%2E+2021%2E+Modelling+the+unigram+distribution%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL%2DIJCNLP+2021%2C+pages+3721)
42. François Pellegrino, Ioana Chitoran, Egidio Marsico, and Christophe Coupé. 2011. A cross-language perspective on speech information rate. Language, 87(3):539–558. [find](https://scholar.google.com/scholar?q=Franc%CC%A7ois+Pellegrino%2C+Ioana+Chitoran%2C+Egidio+Marsico%2C+and+Christophe+Coupe%CC%81%2E+2011%2E+A+cross%2Dlanguage+perspective+on+speech+information+rate%2E+Language%2C+87%283%29%3A539%E2%80%93558%2E)
43. Dirk Pijpops, Dirk Speelman, Stefan Grondelaers, and Freek Van de Velde. 2018. Comparing explanations for the complexity principle: evidence from argument realization. Language and Cognition, 10(3):514–543. [find](https://scholar.google.com/scholar?q=Dirk+Pijpops%2C+Dirk+Speelman%2C+Stefan+Grondelaers%2C+and+Freek+Van+de+Velde%2E+2018%2E+Comparing+explanations+for+the+complexity+principle%3A+evidence+from+argument+realization%2E+Language+and+Cognition%2C+10%283%29%3A51)
44. Tiago Pimentel, Clara Meister, Elizabeth Salesky, Simone Teufel, Damián Blasi, and Ryan Cotterell. 2021. A surprisal–duration trade-off across and within the world’s languages. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational L… [find](https://scholar.google.com/scholar?q=Tiago+Pimentel%2C+Clara+Meister%2C+Elizabeth+Salesky%2C+Simone+Teufel%2C+Damia%CC%81n+Blasi%2C+and+Ryan+Cotterell%2E+2021%2E+A+surprisal%E2%80%93duration+trade%2Doff+across+and+within+the+world%E2%80%99s+languages%2E+In+Proceedings+of+the+)
45. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
46. Keith Rayner. 1998. Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124(3):372–422. [find](https://scholar.google.com/scholar?q=Keith+Rayner%2E+1998%2E+Eye+movements+in+reading+and+information+processing%3A+20+years+of+research%2E+Psychological+Bulletin%2C+124%283%29%3A372%E2%80%93422%2E)
47. Alfréd Rényi. 1961. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics, pages 547–561. University of California Press. [find](https://scholar.google.com/scholar?q=Alfre%CC%81d+Re%CC%81nyi%2E+1961%2E+On+measures+of+entropy+and+information%2E+In+Proceedings+of+the+Fourth+Berkeley+Symposium+on+Mathematical+Statistics+and+Probability%2C+Volume+1%3A+Contributions+to+the+Theory+of+Stati)
48. Martin Schrimpf, Idan Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kanwisher, Joshua Tenenbaum, and Evelina Fedorenko. 2020. The neural architecture of language: Integrative reverse-engineering converges on a model for predictive processing. BioRxiv. [find](https://scholar.google.com/scholar?q=Martin+Schrimpf%2C+Idan+Blank%2C+Greta+Tuckute%2C+Carina+Kauf%2C+Eghbal+A+Hosseini%2C+Nancy+Kanwisher%2C+Joshua+Tenenbaum%2C+and+Evelina+Fedorenko%2E+2020%2E+The+neural+architecture+of+language%3A+Integrative+reverse%2Deng)
49. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1715–1725. Association for Computational Linguistics.
50. Claude E. Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379–423. [find](https://scholar.google.com/scholar?q=Claude+E%2E+Shannon%2E+1948%2E+A+mathematical+theory+of+communication%2E+The+Bell+System+Technical+Journal%2C+27%283%29%3A379%E2%80%93423%2E)
51. Les Sikos, Clayton Greenberg, Heiner Drenhaus, and Matthew W Crocker. 2017. Information density of encodings: The role of syntactic variation in comprehension. In 39th Annual Meeting of the Cognitive Science Society. Cognitive Science Society. [find](https://scholar.google.com/scholar?q=Les+Sikos%2C+Clayton+Greenberg%2C+Heiner+Drenhaus%2C+and+Matthew+W+Crocker%2E+2017%2E+Information+density+of+encodings%3A+The+role+of+syntactic+variation+in+comprehension%2E+In+39th+Annual+Meeting+of+the+Cognitive+)
52. Processing, pages 5191–5202. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Processing%2C+pages+5191%E2%80%935202%2E+Association+for+Computational+Linguistics%2E)
53. Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. 2020. On the predictive power of neural language models for human real-time comprehension behavior. In Proceedings of the 42nd Annual Meeting of the Cognitive Science Society, pages 1707–1713. Cognitive Science Society. [find](https://scholar.google.com/scholar?q=Ethan+Gotlieb+Wilcox%2C+Jon+Gauthier%2C+Jennifer+Hu%2C+Peng+Qian%2C+and+Roger+Levy%2E+2020%2E+On+the+predictive+power+of+neural+language+models+for+human+real%2Dtime+comprehension+behavior%2E+In+Proceedings+of+the+42)
54. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
55. Meilin Zhan and Roger Levy. 2018. Comparing theories of speaker choice using a model of classifier production in Mandarin Chinese. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1997–2005. Assoc… [find](https://scholar.google.com/scholar?q=Meilin+Zhan+and+Roger+Levy%2E+2018%2E+Comparing+theories+of+speaker+choice+using+a+model+of+classifier+production+in+Mandarin+Chinese%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+o)
56. Meilin Zhan and Roger P. Levy. 2019. Availability-based production predicts speakers’ real-time choices of Mandarin classifiers. In 41st Annual Meeting of the Cognitive Science Society, pages 1268–1274. Cognitive Science Society. [find](https://scholar.google.com/scholar?q=Meilin+Zhan+and+Roger+P%2E+Levy%2E+2019%2E+Availability%2Dbased+production+predicts+speakers%E2%80%99+real%2Dtime+choices+of+Mandarin+classifiers%2E+In+41st+Annual+Meeting+of+the+Cognitive+Science+Society%2C+pages+1268%E2%80%93127)
57. Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3):302–319. [find](https://scholar.google.com/scholar?q=Nathaniel+J%2E+Smith+and+Roger+Levy%2E+2013%2E+The+effect+of+word+predictability+on+reading+time+is+logarithmic%2E+Cognition%2C+128%283%29%3A302%E2%80%93319%2E)
58. Sascha Topolinski and Fritz Strack. 2009. The architecture of intuition: fluency and affect determine intuitive judgments of semantic and visual coherence and judgments of grammaticality in artificial grammar learning. Journal of Experimental Psychology: General, 138(1):39. [find](https://scholar.google.com/scholar?q=Sascha+Topolinski+and+Fritz+Strack%2E+2009%2E+The+architecture+of+intuition%3A+fluency+and+affect+determine+intuitive+judgments+of+semantic+and+visual+coherence+and+judgments+of+grammaticality+in+artificial)
59. Fatemeh Torabi Asr and Vera Demberg. 2015. Uniform surprisal at the level of discourse relations: Negation markers and discourse connective omission. In Proceedings of the 11th International Conference on Computational Semantics, pages 118–128. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Fatemeh+Torabi+Asr+and+Vera+Demberg%2E+2015%2E+Uniform+surprisal+at+the+level+of+discourse+relations%3A+Negation+markers+and+discourse+connective+omission%2E+In+Proceedings+of+the+11th+International+Conferenc)
60. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R%2E+Bowman%2E+2019%2E+Neural+network+acceptability+judgments%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%3A625%E2%80%93641%2E)
61. Jason Wei, Clara Meister, and Ryan Cotterell. 2021. A cognitive regularizer for language modeling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Clara+Meister%2C+and+Ryan+Cotterell%2E+2021%2E+A+cognitive+regularizer+for+language+modeling%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+10)

*Updated 24 October 2026*