---
title: "What Goes Into a LM Acceptability Judgment? Rethinking the"
description: "The authors propose MORCELA, which uses the same inputs (LM log probability, unigram log… 41 references, cited by 8 papers here."
authors: ["Lindia Tjuatja", "Graham Neubig", "Tal Linzen", "Sophie Hao"]
year: 2025
venue: "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.naacl-long.109/"
doi: "10.18653/v1/2025.naacl-long.109"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tjuatja-2025-what-goes-into-lm-acceptability/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tjuatja-2025-what-goes-into-lm-acceptability/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/) / What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Length

# What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Length

**Authors:** [Lindia Tjuatja](https://md.research.elamur.ai/free/en/authors/lindia-tjuatja/), [Graham Neubig](https://md.research.elamur.ai/free/en/authors/graham-neubig/), [Tal Linzen](https://md.research.elamur.ai/free/en/authors/tal-linzen/) and Sophie Hao · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2025](https://md.research.elamur.ai/free/en/venues/naacl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.naacl-long.109/) · [PDF](https://aclanthology.org/2025.naacl-long.109.pdf) · [DOI 10.18653/v1/2025.naacl-long.109](https://doi.org/10.18653/v1/2025.naacl-long.109) · [OpenAlex](https://openalex.org/W4411119540)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/tjuatja-2025-what-goes-into-lm-acceptability/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Comparing language model probabilities with human acceptability ratings requires a linking function. Raw sentence probabilities fall for longer sentences and for sentences containing rare words, while human judgments are largely unaffected by these factors. The standard linking function, SLOR, applies the same fixed correction for length and unigram frequency to every model.

The authors propose MORCELA, which uses the same inputs (LM log probability, unigram log probability and token length) but adds two learnable weights: β for frequency and γ for length. These weights are estimated separately for each model by linear regression on human ratings. The authors test it on English judgments from Sprouse et al. (2013), using decoder-only transformers from the Pythia and OPT families.

MORCELA tracks human scores significantly better than SLOR. The fitted weights show that SLOR overcorrects for both confounds. Larger models need relatively less frequency adjustment, which the authors attribute to their better prediction of rare words in context. The main implication is that LM–human agreement has probably been underestimated, and that corrections should be calibrated per model rather than fixed in advance.

## Key points
- MORCELA scores a sentence as (p − βu + γ)/ℓ. β and γ are derived from a regression of human ratings on p/ℓ, u/ℓ and 1/ℓ. SLOR is the special case with β = 1 and γ = 0.
- Models tested: all eight Pythia sizes (70M–12B, trained on The Pile) and OPT models from 125M to 30B (excluding the two largest). Both families were trained on about 300B tokens.
- OPT's training corpus is not public, so unigram frequencies for OPT were approximated from 100,000 sequences generated by OPT-30B.
- Data: 1,450 sentences from Linguistic Inquiry examples, rated on a 1–7 Likert scale by US-based AMT workers and z-normalized per participant (Sprouse et al., 2013).
- Every model still requires substantial frequency correction, and a clear gap remains between the best LM–human correlation and inter-annotator agreement. The study covers English only.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 539 works · 179,533 citations
- [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 201 works · 100,180 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

## Cite

Cite as: Lindia Tjuatja, Graham Neubig, Tal Linzen and Sophie Hao. 2025. What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Length. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tjuatja2025what,
  title = {{What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Length}},
  author = {Lindia Tjuatja, Graham Neubig, Tal Linzen and Sophie Hao},
  booktitle = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.109},
  url = {https://doi.org/10.18653/v1/2025.naacl-long.109}
}
```

- [← previous VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cues](https://md.research.elamur.ai/free/en/papers/zhang-2025-vlm2-bench-closer-look-how/)
- [next → When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoning](https://md.research.elamur.ai/free/en/papers/singhi-2025-solve-verify-compute-optimal-problem/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/)

## References

41 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Hirotugu Akaike. 1974. A new look at the statistical model identification. IEEE transactions on automatic control, 19(6):716–723. [find](https://scholar.google.com/scholar?q=Hirotugu+Akaike%2E+1974%2E+A+new+look+at+the+statistical+model+identification%2E+IEEE+transactions+on+automatic+control%2C+19%286%29%3A716%E2%80%93723%2E)
2. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Conferenc… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Gregory+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+et+al%2E+2023%2E+Pythia%3A)
3. Noam Chomsky. 1957. Syntactic Structures, 1 edition. Mouton, The Hague, Netherlands. [find](https://scholar.google.com/scholar?q=Noam+Chomsky%2E+1957%2E+Syntactic+Structures%2C+1+edition%2E+Mouton%2C+The+Hague%2C+Netherlands%2E)
4. Jeffrey L. Elman. 1990. Finding Structure in Time. Cognitive Science, 14(2):179–211. [find](https://scholar.google.com/scholar?q=Jeffrey+L%2E+Elman%2E+1990%2E+Finding+Structure+in+Time%2E+Cognitive+Science%2C+14%282%29%3A179%E2%80%93211%2E)
5. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+et+al%2E+2020%2E+The+pile%3A+An+800gb+dataset+of+diverse+text+for+lan)
6. Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. [SyntaxGym: An Online Platform for Targeted Evaluation of Language Models](https://md.research.elamur.ai/free/en/papers/gauthier-2020-syntaxgym-online-platform-targeted-evaluation/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70–76, Online. Association for…
7. Grant Goodall. 2021. Sentence Acceptability Experiments: What, How, and Why, page 7–38. Cambridge Handbooks in Language and Linguistics. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Grant+Goodall%2E+2021%2E+Sentence+Acceptability+Experiments%3A+What%2C+How%2C+and+Why%2C+page+7%E2%80%9338%2E+Cambridge+Handbooks+in+Language+and+Linguistics%2E+Cambridge+University+Press%2E)
8. Adam Goodkind and Klinton Bicknell. 2018. [Predictive power of word surprisal for reading times is a linear function of language model quality](https://md.research.elamur.ai/free/en/papers/goodkind-2018-predictive-power-word-surprisal-reading/). In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018), pages 10–18, Salt Lake City, UT. Association for Computat…
9. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. [Colorless green recurrent networks dream hierarchically](https://md.research.elamur.ai/free/en/papers/gulordava-2018-colorless-green-recurrent-networks-dream/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume…
10. Katharina Kann, Sascha Rothe, and Katja Filippova. 2018. Sentence-level fluency evaluation: References help, but can be spared! In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 313–323, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Katharina+Kann%2C+Sascha+Rothe%2C+and+Katja+Filippova%2E+2018%2E+Sentence%2Dlevel+fluency+evaluation%3A+References+help%2C+but+can+be+spared%21+In+Proceedings+of+the+22nd+Conference+on+Computational+Natural+Language+)
11. Dhruv Kumar, Lili Mou, Lukasz Golab, and Olga Vechtomova. 2020. Iterative edit-based unsupervised sentence simplification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7918–7928, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dhruv+Kumar%2C+Lili+Mou%2C+Lukasz+Golab%2C+and+Olga+Vechtomova%2E+2020%2E+Iterative+edit%2Dbased+unsupervised+sentence+simplification%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computationa)
12. Jey Han Lau, Carlos Armendariz, Shalom Lappin, Matthew Purver, and Chang Shu. 2020. [How Furiously Can Colorless Green Ideas Sleep? Sentence Acceptability in Context](https://md.research.elamur.ai/free/en/papers/lau-2020-how-furiously-can-colorless-green/). Transactions of the Association for Computational Linguistics, 8:296–310.
13. Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, Acceptability, and Probability: A Probabilistic View of Linguistic Knowledge. Cognitive Science, 41(5):1202–1241. [find](https://scholar.google.com/scholar?q=Jey+Han+Lau%2C+Alexander+Clark%2C+and+Shalom+Lappin%2E+2017%2E+Grammaticality%2C+Acceptability%2C+and+Probability%3A+A+Probabilistic+View+of+Linguistic+Knowledge%2E+Cognitive+Science%2C+41%285%29%3A1202%E2%80%931241%2E)
14. Cara Su-Yi Leong and Tal Linzen. 2023. Language models can learn exceptions to syntactic rules. In Proceedings of the Society for Computation in Linguistics 2023, pages 133–144, Amherst, MA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Cara+Su%2DYi+Leong+and+Tal+Linzen%2E+2023%2E+Language+models+can+learn+exceptions+to+syntactic+rules%2E+In+Proceedings+of+the+Society+for+Computation+in+Linguistics+2023%2C+pages+133%E2%80%93144%2C+Amherst%2C+MA%2E+Associati)
15. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. [Assessing the ability of LSTMs to learn syntax-sensitive dependencies](https://md.research.elamur.ai/free/en/papers/linzen-2016-assessing-ability-lstms-learn-syntax/). Transactions of the Association for Computational Linguistics, 4:521–535.
16. Jiayi Lu, Jonathan Merchan, Lian Wang, and Judith Degen. 2024. Can syntactic log-odds ratio predict acceptability and satiation? In Proceedings of the Society for Computation in Linguistics 2024, pages 10–19, Irvine, CA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jiayi+Lu%2C+Jonathan+Merchan%2C+Lian+Wang%2C+and+Judith+Degen%2E+2024%2E+Can+syntactic+log%2Dodds+ratio+predict+acceptability+and+satiation%3F+In+Proceedings+of+the+Society+for+Computation+in+Linguistics+2024%2C+page)
17. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rebecca+Marvin+and+Tal+Linzen%2E+2018%2E+Targeted+syntactic+evaluation+of+language+models%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1192%E2%80%931202%2C+Bruss)
18. Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell, and Roger Levy. 2021. [Revisiting the Uniform Information Density hypothesis](https://md.research.elamur.ai/free/en/papers/meister-2021-revisiting-uniform-information-density-hypothesis/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 963– 980, Online and Punta Cana, Dominican Republic…
19. Kanishka Misra and Kyle Mahowald. 2024. [Language models learn rare phenomena from less rare phenomena: The case of the missing aanns](https://md.research.elamur.ai/free/en/papers/misra-2024-language-models-learn-rare-phenomena/). arXiv preprint arXiv:2403.19827.
20. Peter Norvig. 2017. On Chomsky and the Two Cultures of Statistical Learning. In Wolfgang Pietsch, Jörg Wernecke, and Maximilian Ott, editors, Berechen-barkeit der Welt? Philosophie und Wissenschaft im Zeitalter von Big Data, pages 61–83. Springer Fachmedien, Wiesbaden, Germany. [find](https://scholar.google.com/scholar?q=Peter+Norvig%2E+2017%2E+On+Chomsky+and+the+Two+Cultures+of+Statistical+Learning%2E+In+Wolfgang+Pietsch%2C+J%C3%B6rg+Wernecke%2C+and+Maximilian+Ott%2C+editors%2C+Berechen%2Dbarkeit+der+Welt%3F+Philosophie+und+Wissenschaft+im)
21. Byung-Doh Oh and William Schuler. 2023. [Why does surprisal from larger transformer-based language models provide a poorer fit to human reading times?](https://md.research.elamur.ai/free/en/papers/oh-2023-why-does-surprisal-larger-transformer/) Transactions of the Association for Computational Linguistics, 11:336–350.
22. Byung-Doh Oh, Shisen Yue, and William Schuler. 2024. Frequency explains the inverse correlation of large language models’ size, training data amount, and surprisal’s fit to reading times. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics,… [find](https://scholar.google.com/scholar?q=Byung%2DDoh+Oh%2C+Shisen+Yue%2C+and+William+Schuler%2E+2024%2E+Frequency+explains+the+inverse+correlation+of+large+language+models%E2%80%99+size%2C+training+data+amount%2C+and+surprisal%E2%80%99s+fit+to+reading+times%2E+In+Proceedin)
23. Adam Pauls and Dan Klein. 2012. Large-scale syntactic language modeling with treelets. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 959–968, Jeju Island, Korea. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Adam+Pauls+and+Dan+Klein%2E+2012%2E+Large%2Dscale+syntactic+language+modeling+with+treelets%2E+In+Proceedings+of+the+50th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers)
24. Fernando Pereira. 2000. Formal Grammar and Information Theory: Together Again? Philosophical Transactions: Mathematical, Physical and Engineering Sciences, 358(1769):1239–1253. [find](https://scholar.google.com/scholar?q=Fernando+Pereira%2E+2000%2E+Formal+Grammar+and+Information+Theory%3A+Together+Again%3F+Philosophical+Transactions%3A+Mathematical%2C+Physical+and+Engineering+Sciences%2C+358%281769%29%3A1239%E2%80%931253%2E)
25. Willard Van Orman Quine. 1960. Word and Object. MIT Press, Cambridge, MA. [find](https://scholar.google.com/scholar?q=Willard+Van+Orman+Quine%2E+1960%2E+Word+and+Object%2E+MIT+Press%2C+Cambridge%2C+MA%2E)
26. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. [Recipes for building an open-domain chatbot](https://md.research.elamur.ai/free/en/papers/roller-2021-recipes-building-open-domain-chatbot/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational…
27. Carson T Schütze. 2016. The empirical base of linguistics: Grammaticality judgments and linguistic methodology. Language Science Press. [find](https://scholar.google.com/scholar?q=Carson+T+Sch%C3%BCtze%2E+2016%2E+The+empirical+base+of+linguistics%3A+Grammaticality+judgments+and+linguistic+methodology%2E+Language+Science+Press%2E)
28. Gideon Schwarz. 1978. Estimating the dimension of a model. Annals of Statistics, 6:461–464. [find](https://scholar.google.com/scholar?q=Gideon+Schwarz%2E+1978%2E+Estimating+the+dimension+of+a+model%2E+Annals+of+Statistics%2C+6%3A461%E2%80%93464%2E)
29. Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3):302–319. [find](https://scholar.google.com/scholar?q=Nathaniel+J%2E+Smith+and+Roger+Levy%2E+2013%2E+The+effect+of+word+predictability+on+reading+time+is+logarithmic%2E+Cognition%2C+128%283%29%3A302%E2%80%93319%2E)
30. Antonella Sorace and Frank Keller. 2005. Gradience in linguistic data. Lingua, 115(11):1497–1524. Data in Theoretical Linguistics. [find](https://scholar.google.com/scholar?q=Antonella+Sorace+and+Frank+Keller%2E+2005%2E+Gradience+in+linguistic+data%2E+Lingua%2C+115%2811%29%3A1497%E2%80%931524%2E+Data+in+Theoretical+Linguistics%2E)
31. Jon Sprouse. 2007. Continuous acceptability, categorical grammaticality, and experimental syntax. Biolinguistics. [find](https://scholar.google.com/scholar?q=Jon+Sprouse%2E+2007%2E+Continuous+acceptability%2C+categorical+grammaticality%2C+and+experimental+syntax%2E+Biolinguistics%2E)
32. Jon Sprouse. 2013. Acceptability judgments. [find](https://scholar.google.com/scholar?q=Jon+Sprouse%2E+2013%2E+Acceptability+judgments%2E)
33. Jon Sprouse. 2015. Three open questions in experimental syntax. Linguistics Vanguard, 1(1):89–100. [find](https://scholar.google.com/scholar?q=Jon+Sprouse%2E+2015%2E+Three+open+questions+in+experimental+syntax%2E+Linguistics+Vanguard%2C+1%281%29%3A89%E2%80%93100%2E)
34. Jon Sprouse, Carson T Schütze, and Diogo Almeida. 2013. A comparison of informal and formal acceptability judgments using a random sample from linguistic inquiry 2001–2010. Lingua, 134:219–248. [find](https://scholar.google.com/scholar?q=Jon+Sprouse%2C+Carson+T+Sch%C3%BCtze%2C+and+Diogo+Almeida%2E+2013%2E+A+comparison+of+informal+and+formal+acceptability+judgments+using+a+random+sample+from+linguistic+inquiry+2001%E2%80%932010%2E+Lingua%2C+134%3A219%E2%80%93248%2E)
35. Jon Sprouse, Beracah Yankama, Sagar Indurkhya, Sandiway Fong, and Robert C. Berwick. 2018. Colorless green ideas do sleep furiously: Gradient acceptability and the nature of the grammar. The Linguistic Review, 35(3):575–599. [find](https://scholar.google.com/scholar?q=Jon+Sprouse%2C+Beracah+Yankama%2C+Sagar+Indurkhya%2C+Sandiway+Fong%2C+and+Robert+C%2E+Berwick%2E+2018%2E+Colorless+green+ideas+do+sleep+furiously%3A+Gradient+acceptability+and+the+nature+of+the+grammar%2E+The+Linguisti)
36. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is All you Need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems 30, pages 5998–6008, Long Beach, CA, USA. Curran Associates, Inc.
37. Héctor Vázquez Martínez, Annika Lea Heuser, Charles Yang, and Jordan Kodner. 2023. Evaluating neural language models as cognitive models of language acquisition. In Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP, pages 48–64, Singapore. Association for Computational… [find](https://scholar.google.com/scholar?q=H%C3%A9ctor+V%C3%A1zquez+Mart%C3%ADnez%2C+Annika+Lea+Heuser%2C+Charles+Yang%2C+and+Jordan+Kodner%2E+2023%2E+Evaluating+neural+language+models+as+cognitive+models+of+language+acquisition%2E+In+Proceedings+of+the+1st+GenBench+Wor)
38. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://md.research.elamur.ai/free/en/papers/warstadt-2020-blimp-benchmark-linguistic-minimal-pairs/). Transactions of the Association for Computational Linguistics, 8:377– 392.
39. Ethan Gotlieb Wilcox, Pranali Vani, and Roger P Levy. 2021. A targeted assessment of incremental processing in neural languagemodels and humans. arXiv preprint arXiv:2106.03232. — [arXiv:2106.03232](https://arxiv.org/abs/2106.03232) [find](https://scholar.google.com/scholar?q=Ethan+Gotlieb+Wilcox%2C+Pranali+Vani%2C+and+Roger+P+Levy%2E+2021%2E+A+targeted+assessment+of+incremental+processing+in+neural+languagemodels+and+humans%2E+arXiv+preprint+arXiv%3A2106%2E03232%2E+%E2%80%94+%5BarXiv%3A2106%2E03232%5D%28h)
40. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. [OPT: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068.
41. Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021. [A robustly optimized BERT pre-training approach with post-training](https://md.research.elamur.ai/free/en/papers/zhuang-2021-robustly-optimized-bert-pre-training/). In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1218–1227, Huhhot, China. Chinese Information Processing Society of China.

*Updated 30 October 2026*