---
title: "Manipulating language models' training data to study syntactic"
description: "The English passive is productive, yet a few verbs that look transitive resist it ( last , cost , have ). 84 references, cited by 7 papers here."
authors: ["Cara Su-Yi Leong", "Tal Linzen"]
year: 2024
venue: "arXiv:2407.04593"
source: "https://arxiv.org/abs/2407.04593"
doi: "10.48550/arxiv.2407.04593"
arxiv: "2407.04593"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/leong-2024-manipulating-language-models-training-data/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/leong-2024-manipulating-language-models-training-data/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/) / Manipulating language models' training data to study syntactic constraint learning: the case of English passivization

# Manipulating language models' training data to study syntactic constraint learning: the case of English passivization

**Authors:** Cara Su-Yi Leong and [Tal Linzen](https://md.research.elamur.ai/free/en/authors/tal-linzen/) · **Year:** 2024 · **Venue:** arXiv:2407.04593 · **Publisher:** [Elsevier BV](https://md.research.elamur.ai/free/en/publishers/elsevier-bv/) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2407.04593) · [PDF](https://arxiv.org/pdf/2407.04593) · [DOI 10.48550/arxiv.2407.04593](https://doi.org/10.48550/arxiv.2407.04593) · [arXiv 2407.04593](https://arxiv.org/abs/2407.04593) · [OpenAlex](https://openalex.org/W7137892280)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/leong-2024-manipulating-language-models-training-data/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The English passive is productive, yet a few verbs that look transitive resist it (*last*, *cost*, *have*). This raises a learnability puzzle known as Baker's Paradox: how does a learner tell a form that is missing because it is ungrammatical from one that is merely unattested by chance? The authors use neural language models trained on roughly human-scale input (100 million words of English) as stand-ins for learners. They first collect human acceptability ratings for active/passive sentences built on verbs the literature flags as resisting passivization. They then derive model judgments by comparing probabilities over minimal pairs. Model and human verb-level passivizability agree closely, more so than with simple frequency-based baselines. Next, they retrain models on edited corpora that manipulate two candidate cues: how rare a verb's passive is relative to its active (entrenchment) and how affected its object is (affectedness). Both cues independently shift passivizability, with no interaction. Even combined, they do not account for the whole effect. The study shows how targeted corpus editing can test causal claims about what learners extract from input.

## Key points
- Human ratings (Experiment 1A) used verbs listed as passive-resistant by Bach, Levin, Postal and Zwicky. Model judgments correlated with these ratings at r = 0.9, well above frequency-only models (Experiment 1B).
- Experiment 2A tested entrenchment by making a highly passivizable verb much rarer in the passive in the training corpus. Its passives then became less acceptable to the model.
- Experiment 2B tested affectedness by pairing an unpassivizable verb with arguments typical of a canonically passivizable verb. This raised the verb's passivizability, although the size of the effect depended on which verb was used.
- Experiment 3 injected a novel verb that appears only in actives, varying both its frequency and its semantic contexts. Frequency and affectedness had additive effects with no interaction.
- Materials, judgment data and analysis scripts are published at github.com/craaaa/exceptions.

## Interesting ideas
- Counterfactual corpus surgery: retrain the same learner on corpora that differ in one targeted property, then attribute any change in behavior to that property.
- Nonce-verb injection: add an invented verb with fully controlled frequency and argument types, so that generalization is separated from memorization of familiar lemmas.
- Graded judgments from minimal-pair probability comparisons, aggregated per verb and correlated with human ratings, yield a gradient score instead of a binary right/wrong verdict.
- Affectedness (the object undergoes a change of state, location or existence) works as a semantic predictor of whether the passive is licensed.
- Always benchmark the learner against plain frequency baselines, to show that it captures more than counting.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): remove, for example, all passive tokens of selected verbs from the silver training data and retrain the perceptron parser, using the result as a causal check of which constructions it truly learned from data.
- **I2** (frames): seed texts with a made-up verb whose objects are clearly changed or unchanged, and check whether role assignment follows the argument semantics rather than a lookup of known lemmas.
- **I3** (tree): score active/passive variants of the same sentence with gparse's confidence and correlate per-verb scores with a small human-rated set, as a calibration test.
- **I4** (back): store an affectedness flag in the compiled verb-class knowledge, so the generator never produces a passive like "*One hour was lasted by the meeting".
- **I5** (learn): add a lemma-frequency baseline as a negative control in evaluations, so that apparent gains are not just frequency effects.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 201 works · 100,180 citations

📚 Shelves (6): [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Developmental and Educational Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/developmental-and-educational-psychology/) · 499 works · [🧠 Neuroscience](https://md.research.elamur.ai/free/en/shelves/neuroscience/) · 165 works · [🧠 Cognitive Neuroscience](https://md.research.elamur.ai/free/en/shelves/neuroscience/cognitive-neuroscience/) · 164 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works

## Cite

Cite as: Cara Su-Yi Leong and Tal Linzen. 2024. Manipulating language models' training data to study syntactic constraint learning: the case of English passivization. arXiv:2407.04593.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{leong2024manipulating,
  title = {{Manipulating language models' training data to study syntactic constraint learning: the case of English passivization}},
  author = {Cara Su-Yi Leong and Tal Linzen},
  year = {2024},
  doi = {10.48550/arxiv.2407.04593},
  eprint = {2407.04593},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.04593}
}
```

- [← previous Low Resource Question Answering: An Amharic Benchmarking Dataset](https://md.research.elamur.ai/free/en/papers/taffa-2024-low-resource-question-answering-amharic/)
- [next → More Labels or Cases? Assessing Label Variation in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/gruber-2024-more-labels-or-cases-assessing/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/)

## References

84 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al., 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). ArXiv preprint abs/2303.08774. URL: https://arxiv.org/abs/2303.08774. [↗](https://arxiv.org/abs/2303.08774)
2. Ambridge, B., Arnon, I., Bekman, D., 2023. He was run-over by a bus: Passive – but not pseudo-passive – sentences are rated as more acceptable when the subject is highly affected. New data from Hebrew, and a meta-analytic synthesis across English, Balinese, Hebrew, Indonesian and Mandarin. Glossa Psycholinguistics 2. doi:10.5070/G6011177. [↗](https://doi.org/10.5070/g6011177)
3. Ambridge, B., Bidgood, A., Pine, J.M., Rowland, C.F., Freudenthal, D., 2016. Is Passive Syntax Semantically Constrained? Evidence From Adult Grammaticality Judgment and Comprehension Studies. Cognitive Science 40, 1435–1459. doi:10.1111/cogs.12277. [↗](https://doi.org/10.1111/cogs.12277)
4. Ambridge, B., Bidgood, A., Twomey, K.E., Pine, J.M., Rowland, C.F., Freudenthal, D., 2015. Preemption versus Entrenchment: Towards a Construction-General Solution to the Problem of the Retreat from Verb Argument Structure Overgeneralization. PLoS ONE 10, e0123723. doi:10.1371/journal.pone.0123723. [↗](https://doi.org/10.1371/journal.pone.0123723)
5. Armeni, K., Honey, C., Linzen, T., 2022. Characterizing verbatim short-term memory in neural language models, in: Fokkens, A., Srikumar, V. (Eds.), Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Hybrid). pp. 405–424. doi:10.18653/v1/2022.conll-1.28. [↗](https://doi.org/10.18653/v1/2022.conll-1.28)
6. Aryawibawa, I.N., Ambridge, B., 2018. Is Syntax Semantically Constrained? Evidence From a Grammaticality Judgment Study of Indonesian. Cognitive Science 42, 3135–3148. doi:10.1111/cogs.12697. [↗](https://doi.org/10.1111/cogs.12697)
7. Bach, E.W., 1980. In Defense of Passive. arXiv:25001027. [find](https://scholar.google.com/scholar?q=Bach%2C+E%2EW%2E%2C+1980%2E+In+Defense+of+Passive%2E+arXiv%3A25001027%2E)
8. Bakeman, R., Mcarthur, D., 1996. Picturing repeated measures: Comments on Loftus, Morrison, and others. Behavior Research Methods, Instruments & Computers 28, 584–589. doi:10.3758/BF03200546. [↗](https://doi.org/10.3758/bf03200546)
9. Baker, C.L., 1979. Syntactic Theory and the Projection Problem. Linguistic Inquiry 10, 533–581. arXiv:4178133. [find](https://scholar.google.com/scholar?q=Baker%2C+C%2EL%2E%2C+1979%2E+Syntactic+Theory+and+the+Projection+Problem%2E+Linguistic+Inquiry+10%2C+533%E2%80%93581%2E+arXiv%3A4178133%2E)
10. Baroni, M., 2022. On the proper role of linguistically-oriented deep net analysis in linguistic theorizing. arXiv:2106.08694. [↗](https://arxiv.org/abs/2106.08694)
11. Beavers, J., 2011. On affectedness. Natural Language & Linguistic Theory 29, 335–370. arXiv:41475291. [find](https://scholar.google.com/scholar?q=Beavers%2C+J%2E%2C+2011%2E+On+affectedness%2E+Natural+Language+%26+Linguistic+Theory+29%2C+335%E2%80%93370%2E+arXiv%3A41475291%2E)
12. Bidgood, A., Pine, J.M., Rowland, C.F., Ambridge, B., 2020. Syntactic Representations Are Both Abstract and Semantically Constrained: Evidence From Children’s and Adults’ Comprehension and Production/Priming of the English Passive. Cognitive Science 44, e12892. doi:10.1111/cogs.12892. Linguistics and Philosophy 3, 297–341. [↗](https://doi.org/10.1111/cogs.12892)
13. Boyd, J.K., Goldberg, A.E., 2011. Learning What NOT to Say: The Role of Statistical Preemption and Categorization in A-Adjective Production. Language 87, 55–83. doi:10. 1353/lan.2011.0012. [↗](https://doi.org/10.1353/lan.2011.0012)
14. Braine, M.D.S., Brooks, P.J., 1995. Verb Argument Structure and the Problem of Avoiding an Overgeneral Grammar, in: Tomasello, M., Merriman, W.E. (Eds.), Beyond Names for Things: Young Children’s Acquisition of Verbs. L. Erlbaum, Hillsdale, N.J, pp. 353–376. [find](https://scholar.google.com/scholar?q=Braine%2C+M%2ED%2ES%2E%2C+Brooks%2C+P%2EJ%2E%2C+1995%2E+Verb+Argument+Structure+and+the+Problem+of+Avoiding+an+Overgeneral+Grammar%2C+in%3A+Tomasello%2C+M%2E%2C+Merriman%2C+W%2EE%2E+%28Eds%2E%29%2C+Beyond+Names+for+Things%3A+Young+Children%E2%80%99s+Acqu)
15. Brooks, P.J., Tomasello, M., 1999. Young children learn to produce passives with nonce verbs. Developmental Psychology 35, 29. doi:10.1037/0012-1649.35.1.29. [↗](https://doi.org/10.1037/0012-1649.35.1.29)
16. Brown, R., 1973. A First Language: The Early Stages. Harvard University Press., Cambridge, MA. [find](https://scholar.google.com/scholar?q=Brown%2C+R%2E%2C+1973%2E+A+First+Language%3A+The+Early+Stages%2E+Harvard+University+Press%2E%2C+Cambridge%2C+MA%2E)
17. Chouinard, M.M., Clark, E.V., 2003. Adult reformulations of child errors as negative evidence. Journal of Child Language 30, 637–669. doi:10.1017/S0305000903005701. [↗](https://doi.org/10.1017/s0305000903005701)
18. Clark, E.V., 1987. The principle of contrast: A constraint on language acquisition, in: Mechanisms of Language Aquisition. Lawrence Erlbaum Associates, Inc, Hillsdale, NJ, US, pp. 1–33. [find](https://scholar.google.com/scholar?q=Clark%2C+E%2EV%2E%2C+1987%2E+The+principle+of+contrast%3A+A+constraint+on+language+acquisition%2C+in%3A+Mechanisms+of+Language+Aquisition%2E+Lawrence+Erlbaum+Associates%2C+Inc%2C+Hillsdale%2C+NJ%2C+US%2C+pp%2E+1%E2%80%9333%2E)
19. Comrie, B., 1988. Passive and voice, in: Shibatani, M. (Ed.), Passive and Voice. John Benjamins Publishing Company. Typological Studies in Language, pp. 9–24. doi:10. 1075/tsl.16.04com. [find](https://scholar.google.com/scholar?q=Comrie%2C+B%2E%2C+1988%2E+Passive+and+voice%2C+in%3A+Shibatani%2C+M%2E+%28Ed%2E%29%2C+Passive+and+Voice%2E+John+Benjamins+Publishing+Company%2E+Typological+Studies+in+Language%2C+pp%2E+9%E2%80%9324%2E+doi%3A10%2E+1075%2Ftsl%2E16%2E04com%2E)
20. Comrie, B., Cole, P., Sadock, J.M., 1977. In Defense of Spontaneous Demotion: The Impersonal Passive, in: Grammatical Relations. Brill. chapter Grammatical Relations, pp. 47–58. doi:10.1163/9789004368866_004. [↗](https://doi.org/10.1163/9789004368866_004)
21. Constantinescu, I., Pimentel, T., Cotterell, R., Warstadt, A., 2024. Investigating Critical Period Effects in Language Acquisition through Neural Language Models. doi:10.48550/ arXiv.2407.19325, arXiv:2407.19325. [↗](https://arxiv.org/abs/2407.19325)
22. Dankers, V., Lucas, C., Titov, I., 2022. [Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/dankers-2022-can-transformer-be-too-compositional/), in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Dublin, Ireland. pp. 3608–3626. doi:10.18653/ v1/2022.acl-long.252. [↗](https://aclanthology.org/2022.acl-long.252/)
23. Darmasetiyawan, I.M.S., Messenger, K., Ambridge, B., 2022. Is Passive Priming Really Impervious to Verb Semantics? A High-Powered Replication of Messenger Et al. (2012). Collabra: Psychology 8, 31055. doi:10.1525/collabra.31055. [↗](https://doi.org/10.1525/collabra.31055)
24. Demuth, K., 2011. The role of frequency in language acquisition, in: The Role of Frequency in Language Acquisition. De Gruyter Mouton, pp. 383–388. doi:10.1515/ 9783110977905.383. [find](https://scholar.google.com/scholar?q=Demuth%2C+K%2E%2C+2011%2E+The+role+of+frequency+in+language+acquisition%2C+in%3A+The+Role+of+Frequency+in+Language+Acquisition%2E+De+Gruyter+Mouton%2C+pp%2E+383%E2%80%93388%2E+doi%3A10%2E1515%2F+9783110977905%2E383%2E)
25. Fox, D., Grodzinsky, Y., 1998. Children’s Passive: A View from the By-Phrase. Linguistic Inquiry 29, 311–332. arXiv:4179020. [↗](https://doi.org/10.1162/002438998553761)
26. Frank, M.C., 2023. Bridging the data gap between children and large language models. Trends in Cognitive Sciences 27, 990–992. [↗](https://doi.org/10.1016/j.tics.2023.08.007)
27. Gibson, E., Piantadosi, S.T., Brink, K., Bergen, L., Lim, E., Saxe, R., 2013. A Noisy-Channel Account of Crosslinguistic Word-Order Variation. Psychological Science 24, 1079–1088. doi:10.1177/0956797612463705. [↗](https://doi.org/10.1177/0956797612463705)
28. Gilkerson, J., Richards, J.A., Warren, S.F., Montgomery, J.K., Greenwood, C.R., Kimbrough, O.D., Hansen, J.H.L., Paul, T.D., 2017. Mapping the Early Language Environment Using All-Day Recordings and Automated Analysis. American Journal of Speech-Language Pathology 26, 248–265. doi:10.1044/2016_AJSLP-15-0169. [↗](https://doi.org/10.1044/2016_ajslp-15-0169)
29. Gokaslan, A., Cohen, V., 2019. OpenWebText corpus. [find](https://scholar.google.com/scholar?q=Gokaslan%2C+A%2E%2C+Cohen%2C+V%2E%2C+2019%2E+OpenWebText+corpus%2E)
30. Goldberg, A.E., 1995. Constructions: A Construction Grammar Approach to Argument Structure. Cognitive Theory of Language and Culture, University of Chicago Press, Chicago. [find](https://scholar.google.com/scholar?q=Goldberg%2C+A%2EE%2E%2C+1995%2E+Constructions%3A+A+Construction+Grammar+Approach+to+Argument+Structure%2E+Cognitive+Theory+of+Language+and+Culture%2C+University+of+Chicago+Press%2C+Chicago%2E)
31. Goldberg, A.E., 2006. Constructions at Work: The Nature of Generalization in Language. Oxford Linguistics. 1. publ ed., Oxford University Press, Oxford. [find](https://scholar.google.com/scholar?q=Goldberg%2C+A%2EE%2E%2C+2006%2E+Constructions+at+Work%3A+The+Nature+of+Generalization+in+Language%2E+Oxford+Linguistics%2E+1%2E+publ+ed%2E%2C+Oxford+University+Press%2C+Oxford%2E)
32. Gordon, P., Chafetz, J., 1990. Verb-based versus class-based accounts of actionality effects in children’s comprehension of passives. Cognition 36, 227–254. doi:10.1016/ 0010-0277(90)90058-R. [↗](https://doi.org/10.1016/0010-0277(90)90058-r)
33. Gulordava, K., Bojanowski, P., Grave, E., Linzen, T., Baroni, M., 2018. [Colorless Green Recurrent Networks Dream Hierarchically](https://md.research.elamur.ai/free/en/papers/gulordava-2018-colorless-green-recurrent-networks-dream/), in: Walker, M., Ji, H., Stent, A. (Eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Association for Computational Linguistics, New Orleans, Louisiana. pp. 1195–1205. doi:10.18653/v1/N18-1108. [↗](https://doi.org/10.18653/v1/n18-1108)
34. Hawkins, R., Yamakoshi, T., Griffiths, T., Goldberg, A., 2020. Investigating representations of verb bias in neural language models, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Online. pp. 4653–4663. doi:10.18653/v1/2020.emnlp-main.376. [↗](https://doi.org/10.18653/v1/2020.emnlp-main.376)
35. Heafield, K., 2011. [KenLM: Faster and smaller language model queries](https://md.research.elamur.ai/free/en/papers/heafield-2011-kenlm-faster-smaller-language-model/), in: Proceedings of the Sixth Workshop on Statistical Machine Translation, Association for Computational Linguistics, Edinburgh, Scotland. pp. 187–197.
36. Honnibal, M., Montani, I., Van Landeghem, S., Boyd, A., 2020. spaCy: Industrial-strength natural language processing in python. [find](https://scholar.google.com/scholar?q=Honnibal%2C+M%2E%2C+Montani%2C+I%2E%2C+Van+Landeghem%2C+S%2E%2C+Boyd%2C+A%2E%2C+2020%2E+spaCy%3A+Industrial%2Dstrength+natural+language+processing+in+python%2E)
37. Huang, K.J., Arehalli, S., Kugemoto, M., Muxica, C., Prasad, G., Dillon, B., Linzen, T., 2024. Large-scale benchmark yields no evidence that language model surprisal explains syntactic disambiguation difficulty. Journal of Memory and Language 137, 104510. [↗](https://doi.org/10.1016/j.jml.2024.104510)
38. Jumelet, J., Denić, M., Szymanik, J., Hupkes, D., Steinert-Threlkeld, S., 2021. Language Models Use Monotonicity to Assess NPI Licensing. doi:10.48550/arXiv.2105.13818, arXiv:2105.13818. [↗](https://doi.org/10.48550/arxiv.2105.13818)
39. Keenan, E.L., Dryer, M.S., 2007. Passive in the world’s languages, in: Shopen, T. (Ed.), Language Typology and Syntactic Description. 2 ed.. Cambridge University Press, pp. 325–361. doi:10.1017/CBO9780511619427.006. [↗](https://doi.org/10.1017/cbo9780511619427.006)
40. Kingma, D.P., Ba, J., 2015. [Adam: A Method for Stochastic Optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/), in: arXiv:1412.6980 [Cs]. arXiv:1412.6980. [↗](https://arxiv.org/abs/1412.6980)
41. Lake, B.M., Ullman, T.D., Tenenbaum, J.B., Gershman, S.J., 2017. Building machines that learn and think like people. Behavioral and Brain Sciences 40, e253. doi:10.1017/ S0140525X16001837. [↗](https://doi.org/10.1017/s0140525x16001837)
42. Lakretz, Y., Hupkes, D., Vergallito, A., Marelli, M., Baroni, M., Dehaene, S., 2021. Mechanisms for handling nested dependencies in neural-network language models and humans. Cognition 213, 104699. doi:10.1016/j.cognition.2021.104699. [↗](https://doi.org/10.1016/j.cognition.2021.104699)
43. Lau, J.H., Clark, A., Lappin, S., 2017. Grammaticality, Acceptability, and Probability: A Probabilistic View of Linguistic Knowledge. Cognitive Science 41, 1202–1241. doi:10. 1111/cogs.12414. [↗](https://doi.org/10.1111/cogs.12414)
44. Leong, C.S.Y., Linzen, T., 2023. Language Models Can Learn Exceptions to Syntactic Rules, in: Proceedings of the Society for Computation in Linguistics, University of Massachusetts Amherst, Amherst. doi:10.7275/H25Z-0Y75. [↗](https://doi.org/10.7275/h25z-0y75)
45. Levin, B., 1993. English Verb Classes and Alternations: A Preliminary Investigation. University of Chicago Press, Chicago. [find](https://scholar.google.com/scholar?q=Levin%2C+B%2E%2C+1993%2E+English+Verb+Classes+and+Alternations%3A+A+Preliminary+Investigation%2E+University+of+Chicago+Press%2C+Chicago%2E)
46. Levy, R., 2008. A noisy-channel model of rational human sentence comprehension under uncertain input, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP ’08, Association for Computational Linguistics, Honolulu, Hawaii. p. 234. doi:10.3115/1613715.1613749. [↗](https://doi.org/10.3115/1613715.1613749)
47. Linzen, T., 2020. [How Can We Accelerate Progress Towards Human-like Linguistic Generalization?](https://md.research.elamur.ai/free/en/papers/linzen-2020-how-can-we-accelerate-progress/), in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online. pp. 5210–5217. doi:10.18653/v1/2020.acl-main.465. [↗](https://doi.org/10.18653/v1/2020.acl-main.465)
48. Linzen, T., Baroni, M., 2021. Syntactic structure from deep learning. Annual Review of Linguistics 7, 195–212. [↗](https://doi.org/10.1146/annurev-linguistics-032020-051035)
49. Linzen, T., Dupoux, E., Goldberg, Y., 2016. [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://md.research.elamur.ai/free/en/papers/linzen-2016-assessing-ability-lstms-learn-syntax/). Transactions of the Association for Computational Linguistics 4, 521–535. doi:10.1162/tacl_a_00115. [↗](https://doi.org/10.1162/tacl_a_00115)
50. Liu, L., Ambridge, B., 2021. Balancing information-structure and semantic constraints on construction choice: Building a computational model of passive and passive-like constructions in Mandarin Chinese. Cognitive Linguistics 32, 349–388. doi:10.1515/ cog-2019-0100. [↗](https://doi.org/10.1515/cog-2019-0100)
51. MacWhinney, B., 2000. The CHILDES Project: Tools for Analyzing Talk. Third Edition. Lawrence Erlbaum Associates, Mahwah, NJ. [find](https://scholar.google.com/scholar?q=MacWhinney%2C+B%2E%2C+2000%2E+The+CHILDES+Project%3A+Tools+for+Analyzing+Talk%2E+Third+Edition%2E+Lawrence+Erlbaum+Associates%2C+Mahwah%2C+NJ%2E)
52. Maratsos, M., 1985. Semantic restrictions on children’s passives. Cognition 19, 167–191. doi:10.1016/0010-0277(85)90017-4. [↗](https://doi.org/10.1016/0010-0277(85)90017-4)
53. Marvin, R., Linzen, T., 2018. Targeted Syntactic Evaluation of Language Models, in: Riloff, E., Chiang, D., Hockenmaier, J., Tsujii, J. (Eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Brussels, Belgium. pp. 1192–1202. doi:10.18653/v1/D18-1151. [↗](https://doi.org/10.18653/v1/d18-1151)
54. McCoy, R.T., Min, J., Linzen, T., 2020. BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance, in: Alishahi, A., Belinkov, Y., Chrupała, G., Hupkes, D., Pinter, Y., Sajjad, H. (Eds.), Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, Association for Computational Linguistics, Online. pp. 217– 227. URL: https://aclanthology.org/2020.blackboxnlp-1.21, doi:10.18653/v1/ 2020.blackboxnlp-1.21. [↗](https://aclanthology.org/2020.blackboxnlp-1.21/)
55. Messenger, K., Branigan, H.P., McLean, J.F., Sorace, A., 2012. Is young children’s passive syntax semantically constrained? Evidence from syntactic priming. Journal of Memory and Language 66, 568–587. doi:10.1016/j.jml.2012.03.008. [↗](https://doi.org/10.1016/j.jml.2012.03.008)
56. Misra, K., Mahowald, K., 2024. Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNs. arXiv:2403.19827. OpenAI, Hurst, A., Lerer, A., Goucher, A.P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A.J., Welihinda, A., Hayes, A., Radford, A., M adry, A., Baker-Whitcomb, A., Beutel, A., Borzunov, A., Carney, A., Chow, A., Kirillov, A., Nichol, A., Paino, A., Renzin, A., Passos, A.T., Kirillov, A., Christakis, A., Conneau, A., Kamali, A., Jabri, A., Moyer, A., Tam, A., Crookes, A., Tootoochian, A., Tootoonchian, A., Kumar, A., Vallone, A., Karpathy, A., Braunstein, A., Cann, A., Codispoti, A., Galu, A., Kondrich, A., Tulloch, A., Mishchenko, A., Baek, A., Jiang, A., Pelisse, A., Woodford, A., Gosalia, A., Dhar, A., Pantuliano, A., Nayak, A., Oliver, A., Zoph, B., Ghorbani, B., Leimberger, B., Rossen, B., Sokolowsky, B., Wang, B., Zweig, B., Hoover, B., Samic, B., McGrew, B., Spero, B., Giertler, B., Cheng, B., Lightcap, B., Walkin, B., Quinn, B., Guarraci, B., Hsu, B., Kellogg, B., Eastman, B., Lugaresi, C., Wainwright, C., Bassin, C., Hudson, C., Chu, C., Nelson, C., Li, C., Shern, C.J., Conger, C., Barette, C., Voss, C., Ding, C., Lu, C., Zhang, C., Beaumont, C., Hallacy, C., Koch, C., Gibson, C., Kim, C., Choi, C., McLeavey, C., Hesse, C., Fischer, C., Winter, C., Czarnecki, C., Jarvis, C., Wei, C., Koumouzelis, C., Sherburn, D., Kappler, D., Levin, D., Levy, D., Carr, D., Farhi, D., Mely, D., Robinson, D., Sasaki, D., Jin, D., Valladares, D., Tsipras, D., Li, D., Nguyen, D.P., Findlay, D., Oiwoh, E., Wong, E., Asdar, E., Proehl, E., Yang, E., Antonow, E., Kramer, E., Peterson, E., Sigler, E., Wallace, E., Brevdo, E., Mays, E., Khorasani, F., Such, F.P., Raso, F., Zhang, F., von Lohmann, F., Sulit, F., Goh, G., Oden, G., Salmon, G., Starace, G., Brockman, G., Salman, H., Bao, H., Hu, H., Wong, H., Wang, H., Schmidt, H., Whitney, H., Jun, H., Kirchner, H., Pinto, H.P.d.O., Ren, H., Chang, H., Chung, H.W., Kivlichan, I., O’Connell, I., O’Connell, I., Osband, I., Silber, I., Sohl, I., Okuyucu, I., Lan, I., Kostrikov, I., Sutskever, I., Kanitscheider, I., Gulrajani, I., Coxon, J., Menick, J., Pachocki, J., Aung, J., Betker, J., Crooks, J., Lennon, J., Kiros, J., Leike, J., Park, J., Kwon, J., Phang, J., Teplitz, J., Wei, J., Wolfe, J., Chen, J., Harris, J., Varavva, J., Lee, J.G., Shieh, J., Lin, J., Yu, J., Weng, J., Tang, J., Yu, J., Jang, J., Candela, J.Q., Beutler, J., Landers, J., Parish, J., Heidecke, J., Schulman, J., Lachman, J., McKay, J., Uesato, J., Ward, J., Kim, J.W., Huizinga, J., Sitkin, J., Kraaijeveld, J., Gross, J., Kaplan, J., Snyder, J., Achiam, J., Jiao, J., Lee, J., Zhuang, J., Harriman, J., Fricke, K., Hayashi, K., Singhal, K., Shi, K., Karthik, K., Wood, K., Rimbach, K., Hsu, K., Nguyen, K., Gu-Lemberg, K., Button, K., Liu, K., Howe, K., Muthukumar, K., Luther, K., Ahmad, L., Kai, L., Itow, L., Workman, L., Pathak, L., Chen, L., Jing, L., Guy, L., Fedus, L., Zhou, L., Mamitsuka, L., Weng, L., McCallum, L., Held, L., Ouyang, L., Feuvrier, L., Zhang, L., Kondraciuk, L., Kaiser, L., Hewitt, L., Metz, L., Doshi, L., Aflak, M., Simens, M., Boyd, M., Thompson, M., Dukhan, M., Chen, M., Gray, M., Hudnall, M., Zhang, M., Aljubeh, M., Litwin, M., Zeng, M., Johnson, M., Shetty, M., Gupta, M., Shah, M., Yatbaz, M., Yang, M.J., Zhong, M., Glaese, M., Chen, M., Janner, M., Lampe, M., Petrov, M., Wu, M., Wang, M., Fradin, M., Pokrass, M., Castro, M., de Castro, M.O.T., Pavlov, M., Brundage, M., Wang, M., Khan, M., Murati, M., Bavarian, M., Lin, M., Yesildal, M., Soto, N., Gimelshein, N., Cone, N., Staudacher, N., Summers, N., LaFontaine, N., Chowdhury, N., Ryder, N., Stathas, N., Turley, N., Tezak, N., Felix, N., Kudige, N., Keskar, N., Deutsch, N., Bundick, N., Puckett, N., Nachum, O., Okelola, O., Boiko, O., Murk, O., Jaffe, O., Watkins, O., Godement, O., Campbell-Moore, O., Chao, P., McMillan, P., Belov, P., Su, P., Bak, P., Bakkum, P., Deng, P., Dolan, P., Hoeschele, P., Welinder, P., Tillet, P., Pronin, P., Tillet, P., Dhariwal, P., Yuan, Q., Dias, R., Lim, R., Arora, R., Troll, R., Lin, R., Lopes, R.G., Puri, R., Miyara, R., Leike, R., Gaubert, R., Zamani, R., Wang, R., Donnelly, R., Honsby, R., Smith, R., Sahai, R., Ramchandani, R., Huet, R., Carmichael, R., Zellers, R., Chen, R., Chen, R., Nigmatullin, R., Cheu, R., Jain, S., Altman, S., Schoenholz, S., Toizer, S., Miserendino, S., Agarwal, S., Culver, S., Ethersmith, S., Gray, S., Grove, S., Metzger, S., Hermani, S., Jain, S., Zhao, S., Wu, S., Jomoto, S., Wu, S., Shuaiqi, Xia, Phene, S., Papay, S., Narayanan, S., Coffey, S., Lee, S., Hall, S., Balaji, S., Broda, T., Stramer, T., Xu, T., Gogineni, T., Christianson, T., Sanders, T., Patwardhan, T., Cunninghman, T., Degry, T., Dimson, T., Raoux, T., Shadwell, T., Zheng, T., Underwood, T., Markov, T., Sherbakov, T., Rubin, T., Stasi, T., Kaftan, T., Heywood, T., Peterson, T., Walters, T., Eloundou, T., Qi, V., Moeller, V., Monaco, V., Kuo, V., Fomenko, V., Chang, W., Zheng, W., Zhou, W., Manassra, W., Sheu, W., Zaremba, W., Patil, Y., Qian, Y., Kim, Y., Cheng, Y., Zhang, Y., He, Y., Zhang, Y., Jin, Y., Dai, Y., Malkov, Y., 2024. [GPT-4o System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). doi:10.48550/arXiv.2410.21276, arXiv:2410.21276. [↗](https://doi.org/10.48550/arxiv.2410.21276)
57. Papadimitriou, I., Chi, E.A., Futrell, R., Mahowald, K., 2021. Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERT, in: Merlo, P., Tiedemann, J., Tsarfaty, R. (Eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Association for Computational Linguistics, Online. pp. 2522–2532. doi:10.18653/v1/2021.eacl-main.215. [↗](https://doi.org/10.18653/v1/2021.eacl-main.215)
58. Papadimitriou, I., Lopez, K., Jurafsky, D., 2023. Multilingual BERT has an accent: Evaluating English influences on fluency in multilingual models, in: Vlachos, A., Augenstein, I. (Eds.), Findings of the Association for Computational Linguistics: EACL 2023, Association for Computational Linguistics, Dubrovnik, Croatia. pp. 1194–1200. doi:10.18653/v1/2023.findings-eacl.89. [↗](https://doi.org/10.18653/v1/2023.findings-eacl.89)
59. Patil, A., Jumelet, J., Chiu, Y.Y., Lapastora, A., Shen, P., Wang, L., Willrich, C., Steinert-Threlkeld, S., 2024. [Filtered Corpus Training (FiCT) Shows that Language Models can Generalize from Indirect Evidence](https://md.research.elamur.ai/free/en/papers/patil-2024-filtered-corpus-training-fict-shows/). arXiv:2405.15750. [↗](https://arxiv.org/abs/2405.15750)
60. Perfors, A., Tenenbaum, J.B., Wonnacott, E., 2010. Variability, negative evidence, and the acquisition of verb argument constructions. Journal of Child Language 37, 607–642. doi:10.1017/S0305000910000012. [↗](https://doi.org/10.1017/s0305000910000012)
61. Pinker, S., 1989. Learnability and Cognition: The Acquisition of Argument Structure. Learning, Development, and Conceptual Change. 1. paperback ed., 4. print ed., MIT Press, Cambridge, Mass. [find](https://scholar.google.com/scholar?q=Pinker%2C+S%2E%2C+1989%2E+Learnability+and+Cognition%3A+The+Acquisition+of+Argument+Structure%2E+Learning%2C+Development%2C+and+Conceptual+Change%2E+1%2E+paperback+ed%2E%2C+4%2E+print+ed%2E%2C+MIT+Press%2C+Cambridge%2C+Mass%2E)
62. Pinker, S., Lebeaux, D.S., Frost, L.A., 1987. Productivity and constraints in the acquisition of the passive. Cognition 26, 195–267. doi:10.1016/S0010-0277(87)80001-X. [↗](https://doi.org/10.1016/s0010-0277(87)80001-x)
63. Postal, P.M., 2004. Skeptical Linguistic Essays. Oxford University Press, Oxford ; New York. [↗](https://doi.org/10.1093/oso/9780195166712.001.0001)
64. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., 2019. Language Models Are Unsupervised Multitask Learners. Technical Report. OpenAI. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+Child%2C+R%2E%2C+Luan%2C+D%2E%2C+Amodei%2C+D%2E%2C+Sutskever%2C+I%2E%2C+2019%2E+Language+Models+Are+Unsupervised+Multitask+Learners%2E+Technical+Report%2E+OpenAI%2E)
65. Regier, T., Gahl, S., 2004. Learning the unlearnable: The role of missing evidence. Cognition 93, 147–155. doi:10.1016/j.cognition.2003.12.003. [↗](https://doi.org/10.1016/j.cognition.2003.12.003)
66. Reisinger, D., Rudinger, R., Ferraro, F., Harman, C., Rawlins, K., Van Durme, B., 2015. [Semantic Proto-Roles](https://md.research.elamur.ai/free/en/papers/reisinger-2015-semantic-proto-roles/). Transactions of the Association for Computational Linguistics 3, 475–488. doi:10.1162/tacl_a_00152. [↗](https://doi.org/10.1162/tacl_a_00152)
67. Roland, D., Dick, F., Elman, J.L., 2007. Frequency of basic English grammatical structures: A corpus analysis. Journal of Memory and Language 57, 348–379. doi:10.1016/j.jml. 2007.03.002. [↗](https://doi.org/10.1016/j.jml)
68. Saffran, J.R., Aslin, R.N., Newport, E.L., 1996. Statistical Learning by 8-Month-Old Infants. Science 274, 1926–1928. doi:10.1126/science.274.5294.1926. [↗](https://doi.org/10.1126/science.274.5294.1926)
69. Spearman, C., 1910. Correlation Calculated from Faulty Data. British Journal of Psychology, 1904-1920 3, 271–295. doi:10.1111/j.2044-8295.1910.tb00206.x. [↗](https://doi.org/10.1111/j.2044-8295.1910.tb00206.x)
70. Sprouse, J., Almeida, D., 2017. Design sensitivity and statistical power in acceptability judgment experiments. Glossa: a journal of general linguistics 2. doi:10.5334/gjgl.236. [↗](https://doi.org/10.5334/gjgl.236)
71. Theakston, A.L., 2004. The role of entrenchment in children’s and adults’ performance on grammaticality judgment tasks. Cognitive Development 19, 15–34. doi:10.1016/j. cogdev.2003.08.001. [find](https://scholar.google.com/scholar?q=Theakston%2C+A%2EL%2E%2C+2004%2E+The+role+of+entrenchment+in+children%E2%80%99s+and+adults%E2%80%99+performance+on+grammaticality+judgment+tasks%2E+Cognitive+Development+19%2C+15%E2%80%9334%2E+doi%3A10%2E1016%2Fj%2E+cogdev%2E2003%2E08%2E001%2E)
72. Thompson, S.P., Newport, E.L., 2007. Statistical Learning of Syntax: The Role of Transitional Probability. Language Learning and Development 3, 1–42. doi:10.1207/ s15473341lld0301_1. [↗](https://doi.org/10.1080/15475440709336999)
73. Timkey, W., Linzen, T., 2023. A language model with limited memory capacity captures interference in human sentence processing, in: Bouamor, H., Pino, J., Bali, K. (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Computational Linguistics, Singapore. pp. 8705–8720. doi:10.18653/v1/ 2023.findings-emnlp.582. [↗](https://aclanthology.org/2023.findings-emnlp.582/)
74. Tjuatja, L., Neubig, G., Linzen, T., Hao, S., 2025. [What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Length](https://md.research.elamur.ai/free/en/papers/tjuatja-2025-what-goes-into-lm-acceptability/), in: Chiruzzo, L., Ritter, A., Wang, L. (Eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Association for Computational Linguistics, Albuquerque, New Mexico. pp. 2173–2186. doi:10.18653/v1/2025.naacl-long.109. [↗](https://doi.org/10.18653/v1/2025.naacl-long.109)
75. Tomasello, M., 2000. Do young children have adult syntactic competence? Cognition 74, 209–253. doi:10.1016/S0010-0277(99)00069-4. [↗](https://doi.org/10.1016/s0010-0277(99)00069-4)
76. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al., 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). ArXiv preprint abs/2307.09288. URL: https://arxiv.org/abs/2307.09288. [↗](https://arxiv.org/abs/2307.09288)
77. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I., 2017. [Attention Is All You Need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/), in: arXiv:1706.03762 [Cs]. arXiv:1706.03762. [↗](https://arxiv.org/abs/1706.03762)
78. Vong, W.K., Wang, W., Orhan, A.E., Lake, B.M., 2024. Grounded language acquisition through the eyes and ears of a single child. Science doi:10.1126/science.adi1374. [↗](https://doi.org/10.1126/science.adi1374)
79. Warstadt, A., Bowman, S.R., 2022. [What artificial neural networks can tell us about human language acquisition](https://md.research.elamur.ai/free/en/papers/warstadt-2022-artificial-neural-networks-tell-us/), in: Algebraic Structures in Natural Language. CRC Press, pp. 17– 60. [↗](https://doi.org/10.1201/9781003205388-2)
80. Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., Cotterell, R., 2023. Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora, in: Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., Cotterell, R. (Eds.), Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, Association for Computational Linguistics, Singapore. pp. 1–34. doi:10.18653/v1/2023.conll-babylm.1. [↗](https://doi.org/10.18653/v1/2023.conll-babylm.1)
81. Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S.F., Bowman, S.R., 2020. [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://md.research.elamur.ai/free/en/papers/warstadt-2020-blimp-benchmark-linguistic-minimal-pairs/). Transactions of the Association for Computational Linguistics 8, 377–392. doi:10.1162/tacl_a_00321. [↗](https://doi.org/10.1162/tacl_a_00321)
82. Wei, J., Garrette, D., Linzen, T., Pavlick, E., 2021. [Frequency Effects on Syntactic Rule Learning in Transformers](https://md.research.elamur.ai/free/en/papers/wei-2021-frequency-effects-syntactic-rule-learning/), in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Online and Punta Cana, Dominican Republic. pp. 932–948. doi:10.18653/v1/2021.emnlp-main. 72. [↗](https://doi.org/10.18653/v1/2021.emnlp-main)
83. Wilcox, E.G., Hu, M.Y., Mueller, A., Warstadt, A., Choshen, L., Zhuang, C., Williams, A., Cotterell, R., Linzen, T., 2025. Bigger is not always better: The importance of human-scale language modeling for psycholinguistics. Journal of Memory and Language 144, 104650. [↗](https://doi.org/10.1016/j.jml.2025.104650)
84. Zwicky, A.M., 1987. Slashes in the passive. Linguistics 25. doi:10.1515/ling.1987.25. 4.639. [↗](https://doi.org/10.1515/ling.1987.25)

*Updated 1 November 2026*