---
title: "How to Compute the Probability of a Word — Pimentel 2024"
description: "Researchers who use language models to measure surprisal or perplexity usually care about words. 49 references, cited by 36 papers here."
authors: ["Tiago Pimentel", "Clara Meister"]
year: 2024
venue: "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2024.emnlp-main.1020/"
doi: "10.18653/v1/2024.emnlp-main.1020"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pimentel-2024-how-compute-probability-word/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pimentel-2024-how-compute-probability-word/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / How to Compute the Probability of a Word

# How to Compute the Probability of a Word

**Authors:** [Tiago Pimentel](https://md.research.elamur.ai/free/en/authors/tiago-pimentel/) and [Clara Meister](https://md.research.elamur.ai/free/en/authors/clara-meister/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2024](https://md.research.elamur.ai/free/en/venues/emnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.emnlp-main.1020/) · [PDF](https://aclanthology.org/2024.emnlp-main.1020.pdf) · [DOI 10.18653/v1/2024.emnlp-main.1020](https://doi.org/10.18653/v1/2024.emnlp-main.1020) · [OpenAlex](https://openalex.org/W4404782137)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [36](https://md.research.elamur.ai/free/en/papers/pimentel-2024-how-compute-probability-word/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Researchers who use language models to measure surprisal or perplexity usually care about words. Most modern LMs, however, predict subword tokens, and turning subword probabilities into word probabilities is harder than it looks. Pimentel and Meister formalise words, tokenisers and the distributions over them. They then derive correct formulas for a word's probability in context. With end-of-word-marking tokenisers, multiplying the probabilities of a word's subwords is enough. With beginning-of-word (bow) tokenisers, which GPT models, Pythia and Mistral use, the calculation must also account for whether the next token starts a new word. Many recent psycholinguistic studies skipped this step. The fix adds no cost, since it fits within one forward pass. The authors re-ran two earlier analyses with corrected probabilities: one on reading-based sentence comprehension and one on word lengths and communicative efficiency. The original conclusions held, but the measured quantities shifted in statistically significant ways. The method is therefore not a minor detail for empirical studies.

## Key points
- The common "bug" in bow-tokeniser LMs is corrected by renormalising with the probability mass of bow-marked tokens, both after the target word and after its context. The paper's Figure 1 gives this formula, and it comes as a pip package, `wordsprobability` (code: tpimentelms/probability-of-a-word).
- Prior work hit by this bug includes studies by Wilcox et al. (2023a), Shain et al. (2024), Oh and Schuler (2023b) and Pimentel et al. (2023a). Concurrent work by Oh and Schuler (2024) found a similar fix independently.
- On sentence comprehension, corrected surprisal predicted reading data only slightly better for individual models. Pooled across all models, the improvement was consistent and significant on all four datasets (α \< 0.01, permutation tests). The same pattern held in seven other languages.
- In the word-length study on English Wiki-40B, the correction lowered correlations for both channel capacity hypothesis predictors (Piantadosi et al.; Pimentel et al.). Zipf's frequency-based account remained the best predictor of word lengths.
- Limitations include tokeniser corner cases the paper does not cover, such as Spanish "¿". The theory also assumes autoregressive models, and the empirical check covers only two studies.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧒 Psycholinguistics & language acquisition](https://md.research.elamur.ai/free/en/topics/psycholinguistics/) · 102 works

## Cite

Cite as: Tiago Pimentel and Clara Meister. 2024. How to Compute the Probability of a Word. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{pimentel2024how,
  title = {{How to Compute the Probability of a Word}},
  author = {Tiago Pimentel and Clara Meister},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.1020},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.1020}
}
```

- [← previous CausalGym: Benchmarking causal interpretability methods on linguistic tasks](https://md.research.elamur.ai/free/en/papers/arora-2024-causalgym-benchmarking-causal-interpretability-methods/)
- [next → Introducing v0.5 of the AI Safety Benchmark from MLCommons](https://md.research.elamur.ai/free/en/papers/vidgen-2024-introducing-v0-5-ai-safety/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

49 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Christian Bentz and Ramon Ferrer-i-Cancho. 2016. Zipf’s law of abbreviation as a language universal. In Proceedings of the Leiden Workshop on Capturing Phylogenetic Algorithms for Linguistics. Universität Tübingen. [find](https://scholar.google.com/scholar?q=Christian+Bentz+and+Ramon+Ferrer%2Di%2DCancho%2E+2016%2E+Zipf%E2%80%99s+law+of+abbreviation+as+a+language+universal%2E+In+Proceedings+of+the+Leiden+Workshop+on+Capturing+Phylogenetic+Algorithms+for+Linguistics%2E+Univers)
2. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. 2023. Pythia: A suite for analyzing large language models across trainin… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+Aviya+Skowron%2C+Lintang+Sutaw)
3. Kris Cao and Laura Rimell. 2021. You should evaluate your language model on marginal likelihood over tokenisations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2104–2114, Online and Punta Cana, Dominican Republic. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Kris+Cao+and+Laura+Rimell%2E+2021%2E+You+should+evaluate+your+language+model+on+marginal+likelihood+over+tokenisations%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Proce)
4. Thomas M. Cover and Joy A. Thomas. 2006. Elements of Information Theory, second edition. Wiley-Interscience. [find](https://scholar.google.com/scholar?q=Thomas+M%2E+Cover+and+Joy+A%2E+Thomas%2E+2006%2E+Elements+of+Information+Theory%2C+second+edition%2E+Wiley%2DInterscience%2E)
5. Robert M. W. Dixon and Alexandra Y. Aikhenvald. 2003. Word: a typological framework. In Word: A Cross-linguistic Typology, page 1–41. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Robert+M%2E+W%2E+Dixon+and+Alexandra+Y%2E+Aikhenvald%2E+2003%2E+Word%3A+a+typological+framework%2E+In+Word%3A+A+Cross%2Dlinguistic+Typology%2C+page+1%E2%80%9341%2E+Cambridge+University+Press%2E)
6. Li Du, Lucas Torroba Hennigen, Tiago Pimentel, Clara Meister, Jason Eisner, and Ryan Cotterell. 2023. A measure-theoretic characterization of tight language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9744–9770, To… [find](https://scholar.google.com/scholar?q=Li+Du%2C+Lucas+Torroba+Hennigen%2C+Tiago+Pimentel%2C+Clara+Meister%2C+Jason+Eisner%2C+and+Ryan+Cotterell%2E+2023%2E+A+measure%2Dtheoretic+characterization+of+tight+language+models%2E+In+Proceedings+of+the+61st+Annual+M)
7. Richard Futrell, Edward Gibson, Harry J. Tily, Idan Blank, Anastasia Vishnevetsky, Steven Piantadosi, and Evelina Fedorenko. 2018. The natural stories corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Richard+Futrell%2C+Edward+Gibson%2C+Harry+J%2E+Tily%2C+Idan+Blank%2C+Anastasia+Vishnevetsky%2C+Steven+Piantadosi%2C+and+Evelina+Fedorenko%2E+2018%2E+The+natural+stories+corpus%2E+In+Proceedings+of+the+Eleventh+Internatio)
8. Matthias Gallé. 2019. Investigating the effectiveness of BPE: The power of shorter sequences. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1375–1381, Hong Kong… [find](https://scholar.google.com/scholar?q=Matthias+Gall%C3%A9%2E+2019%2E+Investigating+the+effectiveness+of+BPE%3A+The+power+of+shorter+sequences%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+Inte)
9. Edward Gibson, Richard Futrell, Steven T. Piantadosi, Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger Levy. 2019. How efficiency shapes human language. Trends in Cognitive Sciences, 23(5):389– 407. [find](https://scholar.google.com/scholar?q=Edward+Gibson%2C+Richard+Futrell%2C+Steven+T%2E+Piantadosi%2C+Isabelle+Dautriche%2C+Kyle+Mahowald%2C+Leon+Bergen%2C+and+Roger+Levy%2E+2019%2E+How+efficiency+shapes+human+language%2E+Trends+in+Cognitive+Sciences%2C+23%285%29%3A38)
10. Mario Giulianelli, Luca Malagutti, Juan Luis Gastaldi, Brian DuSell, Tim Vieira, and Ryan Cotterell. 2024. [On the proper treatment of tokenization in psycholinguistics](https://md.research.elamur.ai/free/en/papers/giulianelli-2024-proper-treatment-tokenization-psycholinguistics/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA. Association for Com…
11. Adam Goodkind and Klinton Bicknell. 2018. [Predictive power of word surprisal for reading times is a linear function of language model quality](https://md.research.elamur.ai/free/en/papers/goodkind-2018-predictive-power-word-surprisal-reading/). In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018), pages 10–18, Salt Lake City, Utah. Association for Comput…
12. Mandy Guo, Zihang Dai, Denny Vrandečić, and Rami Al-Rfou. 2020. Wiki-40B: Multilingual language model dataset. In Proceedings of the 12th Language Resources and Evaluation Conference, page 2440–2452, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Mandy+Guo%2C+Zihang+Dai%2C+Denny+Vrandec%CC%8Cic%CC%81%2C+and+Rami+Al%2DRfou%2E+2020%2E+Wiki%2D40B%3A+Multilingual+language+model+dataset%2E+In+Proceedings+of+the+12th+Language+Resources+and+Evaluation+Conference%2C+page+2440%E2%80%932452)
13. John Hale. 2001. [A probabilistic Earley parser as a psycholinguistic model](https://md.research.elamur.ai/free/en/papers/hale-2001-probabilistic-earley-parser-as-psycholinguistic/). In Second Meeting of the North American Chapter of the Association for Computational Linguistics, pages 1–8.
14. Tracy Alan Hall and Ursula Kleinhenz. 1999. Studies on the phonological word. John Benjamins. [find](https://scholar.google.com/scholar?q=Tracy+Alan+Hall+and+Ursula+Kleinhenz%2E+1999%2E+Studies+on+the+phonological+word%2E+John+Benjamins%2E)
15. Martin Haspelmath. 2023. Defining the word. WORD, 69(3):283–297. [find](https://scholar.google.com/scholar?q=Martin+Haspelmath%2E+2023%2E+Defining+the+word%2E+WORD%2C+69%283%29%3A283%E2%80%93297%2E)
16. Davis Howes. 1968. Zipf’s law and Miller’s random-monkey model. The American Journal of Psychology, 81(2):269–272. [find](https://scholar.google.com/scholar?q=Davis+Howes%2E+1968%2E+Zipf%E2%80%99s+law+and+Miller%E2%80%99s+random%2Dmonkey+model%2E+The+American+Journal+of+Psychology%2C+81%282%29%3A269%E2%80%93272%2E)
17. Marcel Adam Just, Patricia A. Carpenter, and Jacqueline D. Woolley. 1982. Paradigms and processes in reading comprehension. Journal of Experimental Psychology: General, 111(2):228–238. [find](https://scholar.google.com/scholar?q=Marcel+Adam+Just%2C+Patricia+A%2E+Carpenter%2C+and+Jacqueline+D%2E+Woolley%2E+1982%2E+Paradigms+and+processes+in+reading+comprehension%2E+Journal+of+Experimental+Psychology%3A+General%2C+111%282%29%3A228%E2%80%93238%2E)
18. Alan Kennedy, Robin Hill, and Joel Pynte. 2003. The Dundee corpus. In Proceedings of the 12th European Conference on Eye Movements. [find](https://scholar.google.com/scholar?q=Alan+Kennedy%2C+Robin+Hill%2C+and+Joel+Pynte%2E+2003%2E+The+Dundee+corpus%2E+In+Proceedings+of+the+12th+European+Conference+on+Eye+Movements%2E)
19. Taku Kudo and John Richardson. 2018. [SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ…
20. Natalia Levshina. 2022. Frequency, informativity and word length: Insights from typologically diverse corpora. Entropy, 24(2). [find](https://scholar.google.com/scholar?q=Natalia+Levshina%2E+2022%2E+Frequency%2C+informativity+and+word+length%3A+Insights+from+typologically+diverse+corpora%2E+Entropy%2C+24%282%29%2E)
21. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. [find](https://scholar.google.com/scholar?q=Roger+Levy%2E+2008%2E+Expectation%2Dbased+syntactic+comprehension%2E+Cognition%2C+106%283%29%3A1126%E2%80%931177%2E)
22. Marina Nespor and Irene Vogel. 2007. Prosodic Phonology: With a New Foreword. De Gruyter Mouton, Berlin, Boston. [find](https://scholar.google.com/scholar?q=Marina+Nespor+and+Irene+Vogel%2E+2007%2E+Prosodic+Phonology%3A+With+a+New+Foreword%2E+De+Gruyter+Mouton%2C+Berlin%2C+Boston%2E)
23. Byung-Doh Oh, Christian Clark, and William Schuler. 2022. Comparison of structural parsers and neural language models as surprisal estimators. Frontiers in Artificial Intelligence, 5. [find](https://scholar.google.com/scholar?q=Byung%2DDoh+Oh%2C+Christian+Clark%2C+and+William+Schuler%2E+2022%2E+Comparison+of+structural+parsers+and+neural+language+models+as+surprisal+estimators%2E+Frontiers+in+Artificial+Intelligence%2C+5%2E)
24. Byung-Doh Oh and William Schuler. 2023a. Transformer-based language model surprisal predicts human reading times best with about two billion training tokens. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1915–1921, Singapore. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Byung%2DDoh+Oh+and+William+Schuler%2E+2023a%2E+Transformer%2Dbased+language+model+surprisal+predicts+human+reading+times+best+with+about+two+billion+training+tokens%2E+In+Findings+of+the+Association+for+Computa)
25. Byung-Doh Oh and William Schuler. 2023b. [Why does surprisal from larger transformer-based language models provide a poorer fit to human reading times?](https://md.research.elamur.ai/free/en/papers/oh-2023-why-does-surprisal-larger-transformer/) Transactions of the Association for Computational Linguistics, 11:336–350.
26. Byung-Doh Oh and William Schuler. 2024. Leading whitespaces of language models’ subword vocabulary poses a confound for calculating word probabilities. arXiv preprint arXiv:2406.10851. — [arXiv:2406.10851](https://arxiv.org/abs/2406.10851) [find](https://scholar.google.com/scholar?q=Byung%2DDoh+Oh+and+William+Schuler%2E+2024%2E+Leading+whitespaces+of+language+models%E2%80%99+subword+vocabulary+poses+a+confound+for+calculating+word+probabilities%2E+arXiv+preprint+arXiv%3A2406%2E10851%2E+%E2%80%94+%5BarXiv%3A2406%2E1)
27. Steven T. Piantadosi, Harry Tily, and Edward Gibson. 2011. Word lengths are optimized for efficient communication. Proceedings of the National Academy of Sciences, 108(9):3526–3529. [find](https://scholar.google.com/scholar?q=Steven+T%2E+Piantadosi%2C+Harry+Tily%2C+and+Edward+Gibson%2E+2011%2E+Word+lengths+are+optimized+for+efficient+communication%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+108%289%29%3A3526%E2%80%933529%2E)
28. Tiago Pimentel, Clara Meister, Ethan Wilcox, Kyle Mahowald, and Ryan Cotterell. 2023a. Revisiting the optimality of word lengths. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2240–2255, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tiago+Pimentel%2C+Clara+Meister%2C+Ethan+Wilcox%2C+Kyle+Mahowald%2C+and+Ryan+Cotterell%2E+2023a%2E+Revisiting+the+optimality+of+word+lengths%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+)
29. Tiago Pimentel, Clara Meister, Ethan G. Wilcox, Roger P. Levy, and Ryan Cotterell. 2023b. On the effect of anticipation on reading times. Transactions of the Association for Computational Linguistics, 11:1624–1642. [find](https://scholar.google.com/scholar?q=Tiago+Pimentel%2C+Clara+Meister%2C+Ethan+G%2E+Wilcox%2C+Roger+P%2E+Levy%2C+and+Ryan+Cotterell%2E+2023b%2E+On+the+effect+of+anticipation+on+reading+times%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C)
30. Roger Levy and T. Florian Jaeger. 2007. Speakers optimize information density through syntactic reduction. In Advances in Neural Information Processing Systems, volume 19. MIT Press. [find](https://scholar.google.com/scholar?q=Roger+Levy+and+T%2E+Florian+Jaeger%2E+2007%2E+Speakers+optimize+information+density+through+syntactic+reduction%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+19%2E+MIT+Press%2E)
31. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%3A9%2E)
32. Steven G. Luke and Kiel Christianson. 2018. The Provo corpus: A large eye-tracking corpus with predictability norms. Behavior Research Methods, 50(2):826– 833. [find](https://scholar.google.com/scholar?q=Steven+G%2E+Luke+and+Kiel+Christianson%2E+2018%2E+The+Provo+corpus%3A+A+large+eye%2Dtracking+corpus+with+predictability+norms%2E+Behavior+Research+Methods%2C+50%282%29%3A826%E2%80%93+833%2E)
33. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin…
34. Stephan C. Meylan and Thomas L. Griffiths. 2021. The challenges of large-scale, web-based language datasets: Word length and predictability revisited. Cognitive Science, 45(6):e12983. [find](https://scholar.google.com/scholar?q=Stephan+C%2E+Meylan+and+Thomas+L%2E+Griffiths%2E+2021%2E+The+challenges+of+large%2Dscale%2C+web%2Dbased+language+datasets%3A+Word+length+and+predictability+revisited%2E+Cognitive+Science%2C+45%286%29%3Ae12983%2E)
35. Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y. Lee, Benoît Sagot, and Samson Tan. 2021. [Between words and characters: A brief history of open-vocabulary modeling and tokenization in NLP](https://md.research.elamur.ai/free/en/papers/mielke-2021-between-words-characters-brief-history/). arXiv preprint arXiv:2112.10508.
36. Cory Shain. 2019. A large-scale study of the effects of word frequency and predictability in naturalistic reading. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), page… [find](https://scholar.google.com/scholar?q=Cory+Shain%2E+2019%2E+A+large%2Dscale+study+of+the+effects+of+word+frequency+and+predictability+in+naturalistic+reading%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Associatio)
37. Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cotterell, and Roger Levy. 2024. Large-scale evidence for logarithmic effects of word predictability on reading time. Proceedings of the National Academy of Sciences, 121(10):e2307876121. [find](https://scholar.google.com/scholar?q=Cory+Shain%2C+Clara+Meister%2C+Tiago+Pimentel%2C+Ryan+Cotterell%2C+and+Roger+Levy%2E+2024%2E+Large%2Dscale+evidence+for+logarithmic+effects+of+word+predictability+on+reading+time%2E+Proceedings+of+the+National+Academ)
38. Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2022. [mGPT: Few-shot learners go multilingual](https://md.research.elamur.ai/free/en/papers/shliazhko-2022-mgpt-few-shot-learners-go/). arXiv preprint arXiv:2204.07580.
39. Noam Siegelman, Sascha Schroeder, Cengiz Acartürk, Hee-Don Ahn, Svetlana Alexeeva, Simona Amenta, Raymond Bertram, Rolando Bonandrini, Marc Brysbaert, Daria Chernova, Sara Maria Da Fonseca, Nicolas Dirix, Wouter Duyck, Argyro Fella, Ram Frost, Carolina A. Gattei, Areti Kalaitzi, Nayoung Kwon, Kaidi… [find](https://scholar.google.com/scholar?q=Noam+Siegelman%2C+Sascha+Schroeder%2C+Cengiz+Acart%C3%BCrk%2C+Hee%2DDon+Ahn%2C+Svetlana+Alexeeva%2C+Simona+Amenta%2C+Raymond+Bertram%2C+Rolando+Bonandrini%2C+Marc+Brysbaert%2C+Daria+Chernova%2C+Sara+Maria+Da+Fonseca%2C+Nicolas+Di)
40. Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. [Transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). In Proceedings of the 2020 Conference on Empirical Me…
41. George K. Zipf. 1935. The Psychobiology of Language. London: Routledge. [find](https://scholar.google.com/scholar?q=George+K%2E+Zipf%2E+1935%2E+The+Psychobiology+of+Language%2E+London%3A+Routledge%2E)
42. Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. 2023. Tokenization and the noiseless channel. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5184–5207, Toronto, Canada. Association for… [find](https://scholar.google.com/scholar?q=Vil%C3%A9m+Zouhar%2C+Clara+Meister%2C+Juan+Gastaldi%2C+Li+Du%2C+Mrinmaya+Sachan%2C+and+Ryan+Cotterell%2E+2023%2E+Tokenization+and+the+noiseless+channel%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+C)
43. Nathaniel J. Smith and Roger Levy. 2008. Optimal processing times in reading: a formal model and empirical investigation. In Proceedings of the Cognitive Science Society, volume 30, pages 595–600. [find](https://scholar.google.com/scholar?q=Nathaniel+J%2E+Smith+and+Roger+Levy%2E+2008%2E+Optimal+processing+times+in+reading%3A+a+formal+model+and+empirical+investigation%2E+In+Proceedings+of+the+Cognitive+Science+Society%2C+volume+30%2C+pages+595%E2%80%93600%2E)
44. Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3):302–319. [find](https://scholar.google.com/scholar?q=Nathaniel+J%2E+Smith+and+Roger+Levy%2E+2013%2E+The+effect+of+word+predictability+on+reading+time+is+logarithmic%2E+Cognition%2C+128%283%29%3A302%E2%80%93319%2E)
45. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. [LLaMA: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv pr…
46. Ethan Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. 2020. On the predictive power of neural language models for human real-time comprehension behavior. In Proceedings of the Cognitive Science Society. [find](https://scholar.google.com/scholar?q=Ethan+Wilcox%2C+Jon+Gauthier%2C+Jennifer+Hu%2C+Peng+Qian%2C+and+Roger+Levy%2E+2020%2E+On+the+predictive+power+of+neural+language+models+for+human+real%2Dtime+comprehension+behavior%2E+In+Proceedings+of+the+Cognitive+)
47. Ethan Wilcox, Clara Meister, Ryan Cotterell, and Tiago Pimentel. 2023a. Language model quality correlates with psychometric predictive power in multiple languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7503–7511, Singapore. Association for C… [find](https://scholar.google.com/scholar?q=Ethan+Wilcox%2C+Clara+Meister%2C+Ryan+Cotterell%2C+and+Tiago+Pimentel%2E+2023a%2E+Language+model+quality+correlates+with+psychometric+predictive+power+in+multiple+languages%2E+In+Proceedings+of+the+2023+Conferenc)
48. Ethan Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P. Levy. 2023b. [Testing the predictions of surprisal theory in 11 languages](https://md.research.elamur.ai/free/en/papers/wilcox-2023-testing-predictions-surprisal-theory-11/). Transactions of the Association for Computational Linguistics, 11:1451–1470.
49. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pier%2Dric+Cistac%2C+Tim+Rault%2C+R%C3%A9mi+Louf%2C+Morgan+Funtowicz%2C)

*Updated 29 October 2026*