---
title: "Derivational Morphology Reveals Analogical Generalization in"
description: "The work offers a data-grounded way to probe the mechanisms behind LLM behavior. 120 references, cited by 25 papers here."
authors: ["Valentin Hofmann", "Leonie Weissweiler", "David Mortensen", "Hinrich Schütze", "Janet Pierrehumbert"]
year: 2024
venue: "arXiv:2411.07990"
source: "https://arxiv.org/abs/2411.07990"
doi: "10.48550/arxiv.2411.07990"
arxiv: "2411.07990"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hofmann-2024-derivational-morphology-reveals-analogical-generalization/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hofmann-2024-derivational-morphology-reveals-analogical-generalization/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/) / Derivational Morphology Reveals Analogical Generalization in Large Language Models

# Derivational Morphology Reveals Analogical Generalization in Large Language Models

**Authors:** [Valentin Hofmann](https://md.research.elamur.ai/free/en/authors/valentin-hofmann/), [Leonie Weissweiler](https://md.research.elamur.ai/free/en/authors/leonie-weissweiler/), [David Mortensen](https://md.research.elamur.ai/free/en/authors/david-r-mortensen/), [Hinrich Schütze](https://md.research.elamur.ai/free/en/authors/hinrich-schutze/) and [Janet Pierrehumbert](https://md.research.elamur.ai/free/en/authors/janet-pierrehumbert/) · **Year:** 2024 · **Venue:** [arXiv:2411.07990](https://md.research.elamur.ai/free/en/venues/proceedings-of-the-national-academy-of-sciences/) ([Proceedings of the National Academy of Sciences 2024](https://md.research.elamur.ai/free/en/venues/proceedings-of-the-national-academy-of-sciences/2024/)) · **Publisher:** [National Academy of Sciences](https://md.research.elamur.ai/free/en/publishers/national-academy-of-sciences/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2411.07990) · [PDF](https://arxiv.org/pdf/2411.07990) · [DOI 10.48550/arxiv.2411.07990](https://doi.org/10.48550/arxiv.2411.07990) · [arXiv 2411.07990](https://arxiv.org/abs/2411.07990) · [OpenAlex](https://openalex.org/W4410229834)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [25](https://md.research.elamur.ai/free/en/papers/hofmann-2024-derivational-morphology-reveals-analogical-generalization/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether large language models generalize linguistic patterns by applying abstract rules or by drawing analogies to stored examples. Earlier work mostly studied highly regular syntax, where the two explanations make identical predictions. The authors therefore turn to a more variable phenomenon: English adjective nominalization with -ity versus -ness, looking at adjectives that already carry a suffix. They train two cognitive models on adjective–derivative pairs found in the Pile, which is GPT-J's training corpus. One is a rule-based learner (MGL) and the other an exemplar-based analogy model (GCM). Both models are then compared with GPT-J on nonce adjectives that never appear in the Pile. For regular classes, every model matches GPT-J equally well. For variable classes, the token-based analogical model fits best. GPT-J is also sensitive to the frequency of individual words, even for regular forms, which argues against a rule mechanism. A further finding is that GPT-J generalizes over tokens, whereas humans generalize over types, and this makes its choices less human-like. The work offers a data-grounded way to probe the mechanisms behind LLM behavior.

## Key points
- The study covers four adjective classes. Two are regular: -able, which prefers -ity, and -ish, which prefers -ness. Two are variable: -ive and -ous.
- For each class, 50 nonce adjectives were generated with UniPseudo, drawing on CELEX, MorphoLex and MorphoQuantics. Each nonce adjective and both of its possible derivatives were checked to have zero frequency in the Pile.
- GPT-J's preference was measured by comparing summed log probabilities of the two derivatives, averaged over 12 prompts.
- On the regular classes, MGL and GCM agree with GPT-J identically, at .893 for -able and .997 for -ish. On -ive and -ous, the token-based GCM gives the best match.
- Compared with all the cognitive models, the predictions of GPT-J and GPT-4 are less human-like. The models also show no sign of decomposing words into morpheme types.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 633 works · 195,217 citations
- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 422 works · 213,538 citations
- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 317 works · 106,633 citations
- [University of Oxford](https://md.research.elamur.ai/free/en/universities/university-of-oxford/) — University, Oxford, United Kingdom · 138 works · 161,448 citations
- [Ludwig-Maximilians-Universität München](https://md.research.elamur.ai/free/en/universities/ludwig-maximilians-universitat-munchen/) — University, Munich, Germany · 116 works · 57,070 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,673 works · [💭 Developmental and Educational Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/developmental-and-educational-psychology/) · 533 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works · [🧩 Morphology](https://md.research.elamur.ai/free/en/topics/morphology/) · 421 works

## Cite

Cite as: Valentin Hofmann, Leonie Weissweiler, David Mortensen, Hinrich Schütze and Janet Pierrehumbert. 2024. Derivational Morphology Reveals Analogical Generalization in Large Language Models. arXiv:2411.07990.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hofmann2024derivational,
  title = {{Derivational Morphology Reveals Analogical Generalization in Large Language Models}},
  author = {Valentin Hofmann, Leonie Weissweiler, David Mortensen, Hinrich Schütze and Janet Pierrehumbert},
  year = {2024},
  doi = {10.48550/arxiv.2411.07990},
  eprint = {2411.07990},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2411.07990}
}
```

- [← previous Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challenges](https://md.research.elamur.ai/free/en/papers/ding-2024-data-augmentation-using-large-language/)
- [next → Detection and Measurement of Syntactic Templates in Generated Text](https://md.research.elamur.ai/free/en/papers/shaib-2024-detection-measurement-syntactic-templates-generated/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/)

## References

120 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
2. Gemini Team. [Gemini: A Family of Highly Capable Multimodal Models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). Preprint, arXiv 2312.11805, 2023.
3. OpenAI. [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). Preprint, arXiv 2303.08774, 2024.
4. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. [LLaMA: Open and Efficient Foundation Language Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). Preprint, arXi…
5. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
6. Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, W… [find](https://scholar.google.com/scholar?q=Dirk+Groeneveld%2C+Iz+Beltagy%2C+Pete+Walsh%2C+Akshita+Bhagia%2C+Rodney+Kinney%2C+Oyvind+Tafjord%2C+Ananya+Harsh+Jha%2C+Hamish+Ivison%2C+Ian+Magnusson%2C+Yizhong+Wang%2C+Shane+Arora%2C+David+Atkinson%2C+Russell+Authur%2C+Khyat)
7. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
8. Robert Dale. GPT-3: What’s It Good for? Natural Language Engineering, 27(1):113–118, 2021. [find](https://scholar.google.com/scholar?q=Robert+Dale%2E+GPT%2D3%3A+What%E2%80%99s+It+Good+for%3F+Natural+Language+Engineering%2C+27%281%29%3A113%E2%80%93118%2C+2021%2E)
9. Hubert Haider. Is Chat-GPT a Grammatically Competent Informant? Preprint, lingbuzz 007285, 2023. [find](https://scholar.google.com/scholar?q=Hubert+Haider%2E+Is+Chat%2DGPT+a+Grammatically+Competent+Informant%3F+Preprint%2C+lingbuzz+007285%2C+2023%2E)
10. Emily M. Bender and Alexander Koller. “[Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data](https://md.research.elamur.ai/free/en/papers/bender-2020-climbing-nlu-meaning-form-understanding/)”. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198, 2020.
11. Vittoria Dentella, Elliot Murphy, Gary Marcus, and Evelina Leivada. Testing AI Performance on Less Frequent Aspects of Language Reveals Insensitivity to Underlying Meaning. Preprint, arXiv 2302.12313, 2023. — [arXiv:2302.12313](https://arxiv.org/abs/2302.12313) [find](https://scholar.google.com/scholar?q=Vittoria+Dentella%2C+Elliot+Murphy%2C+Gary+Marcus%2C+and+Evelina+Leivada%2E+Testing+AI+Performance+on+Less+Frequent+Aspects+of+Language+Reveals+Insensitivity+to+Underlying+Meaning%2E+Preprint%2C+arXiv+2302%2E12313%2C)
12. Roni Katzir. Why Large Language Models Are Poor Theories of Human Linguistic Cognition: A Reply to Piantadosi (2023). Preprint, lingbuzz 007190, 2023. [find](https://scholar.google.com/scholar?q=Roni+Katzir%2E+Why+Large+Language+Models+Are+Poor+Theories+of+Human+Linguistic+Cognition%3A+A+Reply+to+Piantadosi+%282023%29%2E+Preprint%2C+lingbuzz+007190%2C+2023%2E)
13. Leonie Weissweiler, Valentin Hofmann, Anjali Kantharuban, Anna Cai, Ritam Dutt, Amey Hengle, Anubha Kabra, Atharva Kulkarni, Abhishek Vijayakumar, Haofei Yu, Hinrich Schütze, Kemal Oflazer, and David Mortensen. “Counting the Bugs in ChatGPT’s Wugs: A Multilingual Investigation into the Morphological… — [Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model](https://md.research.elamur.ai/free/en/papers/weissweiler-2023-chatgpt-wugs-morphology/)
14. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. “[Colorless Green Recurrent Networks Dream Hierarchically](https://md.research.elamur.ai/free/en/papers/gulordava-2018-colorless-green-recurrent-networks-dream/)”. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1195–…
15. Coleman Haley. “This is a BERT. Now There Are Several of Them. Can They Generalize to Novel Words?”. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 333–341, 2020. [find](https://scholar.google.com/scholar?q=Coleman+Haley%2E+%E2%80%9CThis+is+a+BERT%2E+Now+There+Are+Several+of+Them%2E+Can+They+Generalize+to+Novel+Words%3F%E2%80%9D%2E+In+Proceedings+of+the+Third+BlackboxNLP+Workshop+on+Analyzing+and+Interpreting+Neural+Networks+for+)
16. Najoung Kim and Paul Smolensky. “Testing for Grammatical Category Abstraction in Neural Language Models”. In Proceedings of the Society for Computation in Linguistics 2021, pages 467–470, 2021. [find](https://scholar.google.com/scholar?q=Najoung+Kim+and+Paul+Smolensky%2E+%E2%80%9CTesting+for+Grammatical+Category+Abstraction+in+Neural+Language+Models%E2%80%9D%2E+In+Proceedings+of+the+Society+for+Computation+in+Linguistics+2021%2C+pages+467%E2%80%93470%2C+2021%2E)
17. Rowan Hall Maudslay and Ryan Cotterell. “Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing”. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 124–131, 2021. [find](https://scholar.google.com/scholar?q=Rowan+Hall+Maudslay+and+Ryan+Cotterell%2E+%E2%80%9CDo+Syntactic+Probes+Probe+Syntax%3F+Experiments+with+Jabberwocky+Probing%E2%80%9D%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association)
18. Jason Wei, Dan Garrette, Tal Linzen, and Ellie Pavlick. “[Frequency Effects on Syntactic Rule Learning in Transformers](https://md.research.elamur.ai/free/en/papers/wei-2021-frequency-effects-syntactic-rule-learning/)”. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 932–948, 2021.
19. R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. [How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN](https://md.research.elamur.ai/free/en/papers/mccoy-2023-how-much-do-language-models/). Transactions of the Association for Computational Linguistics, 11:652–670, 2023.
20. Noam Chomsky. Aspects of the Theory of Syntax. MIT Press, Cambridge, MA, 1965. [find](https://scholar.google.com/scholar?q=Noam+Chomsky%2E+Aspects+of+the+Theory+of+Syntax%2E+MIT+Press%2C+Cambridge%2C+MA%2C+1965%2E)
21. Noam Chomsky and Morris Halle. The Sound Pattern of English. Harper & Row, New York, NY, 1968. [find](https://scholar.google.com/scholar?q=Noam+Chomsky+and+Morris+Halle%2E+The+Sound+Pattern+of+English%2E+Harper+%26+Row%2C+New+York%2C+NY%2C+1968%2E)
22. Robert M. Nosofsky. Attention, Similarity, and the Identification-Categorization Relationship. Journal of Experimental Psychology: General, 115(1):39–57, 1986. [find](https://scholar.google.com/scholar?q=Robert+M%2E+Nosofsky%2E+Attention%2C+Similarity%2C+and+the+Identification%2DCategorization+Relationship%2E+Journal+of+Experimental+Psychology%3A+General%2C+115%281%29%3A39%E2%80%9357%2C+1986%2E)
23. Robert M. Nosofsky. Similarity, Frequency, and Category Representations. Journal of Experimental Psychology: Learning, Memory, and Cognition, 14(1):54–65, 1988. [find](https://scholar.google.com/scholar?q=Robert+M%2E+Nosofsky%2E+Similarity%2C+Frequency%2C+and+Category+Representations%2E+Journal+of+Experimental+Psychology%3A+Learning%2C+Memory%2C+and+Cognition%2C+14%281%29%3A54%E2%80%9365%2C+1988%2E)
24. Dedre Gentner, Keith Holyoak, and Boicho Kokinov. The Analogical Mind: Perspectives from Cognitive Science. MIT Press, Cambridge, MA, 2001. [find](https://scholar.google.com/scholar?q=Dedre+Gentner%2C+Keith+Holyoak%2C+and+Boicho+Kokinov%2E+The+Analogical+Mind%3A+Perspectives+from+Cognitive+Science%2E+MIT+Press%2C+Cambridge%2C+MA%2C+2001%2E)
25. Joshua B. Tenenbaum and Thomas L. Griffiths. Generalization, Similarity, and Bayesian Inference. Behavioral and Brain Sciences, 24(4):629–640, 2001. [find](https://scholar.google.com/scholar?q=Joshua+B%2E+Tenenbaum+and+Thomas+L%2E+Griffiths%2E+Generalization%2C+Similarity%2C+and+Bayesian+Inference%2E+Behavioral+and+Brain+Sciences%2C+24%284%29%3A629%E2%80%93640%2C+2001%2E)
26. Royal Skousen. Analogical Modeling of Language. Kluwer, Dordrecht, 1989. [find](https://scholar.google.com/scholar?q=Royal+Skousen%2E+Analogical+Modeling+of+Language%2E+Kluwer%2C+Dordrecht%2C+1989%2E)
27. Keith Johnson. “Speech Perception Without Speaker Normalization: An Exemplar Model”. In Keith Johnson and John W. Mullennix, editors, Talker Variability in Speech Processing, pages 145–165. 1997. [find](https://scholar.google.com/scholar?q=Keith+Johnson%2E+%E2%80%9CSpeech+Perception+Without+Speaker+Normalization%3A+An+Exemplar+Model%E2%80%9D%2E+In+Keith+Johnson+and+John+W%2E+Mullennix%2C+editors%2C+Talker+Variability+in+Speech+Processing%2C+pages+145%E2%80%93165%2E+1997%2E)
28. Lisa Dawdy-Hesterberg and Janet Pierrehumbert. Learnability and Generalisation of Arabic Broken Plural Nouns. Language, Cognition and Neuroscience, 29(10):1268–1282, 2014. [find](https://scholar.google.com/scholar?q=Lisa+Dawdy%2DHesterberg+and+Janet+Pierrehumbert%2E+Learnability+and+Generalisation+of+Arabic+Broken+Plural+Nouns%2E+Language%2C+Cognition+and+Neuroscience%2C+29%2810%29%3A1268%E2%80%931282%2C+2014%2E)
29. Simon Todd, Janet Pierrehumbert, and Jennifer Hay. Word Frequency Effects in Sound Change as a Consequence of Perceptual Asymmetries: An Exemplar-Based Model. Cognition, 185:1–20, 2019. [find](https://scholar.google.com/scholar?q=Simon+Todd%2C+Janet+Pierrehumbert%2C+and+Jennifer+Hay%2E+Word+Frequency+Effects+in+Sound+Change+as+a+Consequence+of+Perceptual+Asymmetries%3A+An+Exemplar%2DBased+Model%2E+Cognition%2C+185%3A1%E2%80%9320%2C+2019%2E)
30. Ben Ambridge. Against Stored Abstractions: A Radical Exemplar Model of Language Acquisition. First Language, 40(5-6):509–559, 2020. [find](https://scholar.google.com/scholar?q=Ben+Ambridge%2E+Against+Stored+Abstractions%3A+A+Radical+Exemplar+Model+of+Language+Acquisition%2E+First+Language%2C+40%285%2D6%29%3A509%E2%80%93559%2C+2020%2E)
31. Péter Rácz, Clay Beckner, Jennifer B. Hay, and Janet Pierrehumbert. Morphological Convergence as On-Line Lexical Analogy. Language, 96(4):735–770, 2020. [find](https://scholar.google.com/scholar?q=P%C3%A9ter+R%C3%A1cz%2C+Clay+Beckner%2C+Jennifer+B%2E+Hay%2C+and+Janet+Pierrehumbert%2E+Morphological+Convergence+as+On%2DLine+Lexical+Analogy%2E+Language%2C+96%284%29%3A735%E2%80%93770%2C+2020%2E)
32. Péter Rácz and Ágnes Lukács. Lexical and Social Effects on the Learning and Integration of Inflectional Morphology. Cognitive Science, 48(8):e13483, 2024. [find](https://scholar.google.com/scholar?q=P%C3%A9ter+R%C3%A1cz+and+%C3%81gnes+Luk%C3%A1cs%2E+Lexical+and+Social+Effects+on+the+Learning+and+Integration+of+Inflectional+Morphology%2E+Cognitive+Science%2C+48%288%29%3Ae13483%2C+2024%2E)
33. Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raf. [Emergent and Predictable Memorization in Large Language Models](https://md.research.elamur.ai/free/en/papers/biderman-2023-emergent-predictable-memorization-large-language/). Preprint, arXiv 2304.11158, 2023.
34. Boxi Cao, Qiaoyu Tang, Hongyu Lin, Xianpei Han, Jiawei Chen, Tianshu Wang, and Le Sun. Retentive or Forgetful? Diving into the Knowledge Memorizing Mechanism of Language Models. Preprint, arXiv 2305.09144, 2023. [find](https://scholar.google.com/scholar?q=Boxi+Cao%2C+Qiaoyu+Tang%2C+Hongyu+Lin%2C+Xianpei+Han%2C+Jiawei+Chen%2C+Tianshu+Wang%2C+and+Le+Sun%2E+Retentive+or+Forgetful%3F+Diving+into+the+Knowledge+Memorizing+Mechanism+of+Language+Models%2E+Preprint%2C+arXiv+2305%2E0)
35. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. “Quantifying Memorization Across Neural Language Models”. In International Conference on Learning Representations 2023, 2023. [find](https://scholar.google.com/scholar?q=Nicholas+Carlini%2C+Daphne+Ippolito%2C+Matthew+Jagielski%2C+Katherine+Lee%2C+Florian+Tramer%2C+and+Chiyuan+Zhang%2E+%E2%80%9CQuantifying+Memorization+Across+Neural+Language+Models%E2%80%9D%2E+In+International+Conference+on+Learnin)
36. Steven Pinker and Alan Prince. On Language and Connectionism: Analysis of a Parallel Distributed Processing Model of Language Acquisition. Cognition, 28(1-2):73–193, 1988. [find](https://scholar.google.com/scholar?q=Steven+Pinker+and+Alan+Prince%2E+On+Language+and+Connectionism%3A+Analysis+of+a+Parallel+Distributed+Processing+Model+of+Language+Acquisition%2E+Cognition%2C+28%281%2D2%29%3A73%E2%80%93193%2C+1988%2E)
37. Steven Pinker and Alan Prince. “Regular and Irregular Morphology and the Psychological Status of Rules of Grammar”. In Annual Meeting of the Berkeley Linguistics Society, pages 230–251, 1991. [find](https://scholar.google.com/scholar?q=Steven+Pinker+and+Alan+Prince%2E+%E2%80%9CRegular+and+Irregular+Morphology+and+the+Psychological+Status+of+Rules+of+Grammar%E2%80%9D%2E+In+Annual+Meeting+of+the+Berkeley+Linguistics+Society%2C+pages+230%E2%80%93251%2C+1991%2E)
38. Sandeep Prasada and Steven Pinker. Generalisation of Regular and Irregular Morphological Patterns. Language and Cognitive Processes, 8(1):1–56, 1993. [find](https://scholar.google.com/scholar?q=Sandeep+Prasada+and+Steven+Pinker%2E+Generalisation+of+Regular+and+Irregular+Morphological+Patterns%2E+Language+and+Cognitive+Processes%2C+8%281%29%3A1%E2%80%9356%2C+1993%2E)
39. Ulrike Hahn and Nick Chater. Similarity and Rules: Distinct? Exhaustive? Empirically Distinguishable? Cognition, 65(2-3):197–230, 1998. [find](https://scholar.google.com/scholar?q=Ulrike+Hahn+and+Nick+Chater%2E+Similarity+and+Rules%3A+Distinct%3F+Exhaustive%3F+Empirically+Distinguishable%3F+Cognition%2C+65%282%2D3%29%3A197%E2%80%93230%2C+1998%2E)
40. Emmanuel M. Pothos. The Rules versus Similarity Distinction. Behavioral and Brain Sciences, 28(1): 1–14, 2005. [find](https://scholar.google.com/scholar?q=Emmanuel+M%2E+Pothos%2E+The+Rules+versus+Similarity+Distinction%2E+Behavioral+and+Brain+Sciences%2C+28%281%29%3A+1%E2%80%9314%2C+2005%2E)
41. Sabine Arndt-Lappe. “Word-Formation and Analogy”. In Peter O. Müller, Ingeborg Ohnheiser, Susan Olsen, and Franz Rainer, editors, Word-Formation: An International Handbook of the Languages of Europe, pages 822–841. 2014. [find](https://scholar.google.com/scholar?q=Sabine+Arndt%2DLappe%2E+%E2%80%9CWord%2DFormation+and+Analogy%E2%80%9D%2E+In+Peter+O%2E+M%C3%BCller%2C+Ingeborg+Ohnheiser%2C+Susan+Olsen%2C+and+Franz+Rainer%2C+editors%2C+Word%2DFormation%3A+An+International+Handbook+of+the+Languages+of+Europe%2C+)
42. David E. Rumelhart and James L. McClelland. On Learning the Past Tenses of English Verbs. Psycholinguistics: Critical Concepts in Psychology, 4:216–271, 1986. [find](https://scholar.google.com/scholar?q=David+E%2E+Rumelhart+and+James+L%2E+McClelland%2E+On+Learning+the+Past+Tenses+of+English+Verbs%2E+Psycholinguistics%3A+Critical+Concepts+in+Psychology%2C+4%3A216%E2%80%93271%2C+1986%2E)
43. David C Plaut and James L McClelland. “Generalization With Componential Attractors: Word and Nonword Reading in an Attractor Network”’. In Proceedings of the 15th Annual Conference of the Cognitive Science Society, pages 824–829, 1993. [find](https://scholar.google.com/scholar?q=David+C+Plaut+and+James+L+McClelland%2E+%E2%80%9CGeneralization+With+Componential+Attractors%3A+Word+and+Nonword+Reading+in+an+Attractor+Network%E2%80%9D%E2%80%99%2E+In+Proceedings+of+the+15th+Annual+Conference+of+the+Cognitive+Sc)
44. David E Rumelhart and Peter M Todd. “Learning and Connectionist Representations”. In David E. Meyer and Sylvan Kornblum, editors, Attention and Performance XIV: Synergies in Experimental Psychology, Artificial Intelligence, and Cognitive Neuroscience, pages 3–30. 1993. [find](https://scholar.google.com/scholar?q=David+E+Rumelhart+and+Peter+M+Todd%2E+%E2%80%9CLearning+and+Connectionist+Representations%E2%80%9D%2E+In+David+E%2E+Meyer+and+Sylvan+Kornblum%2C+editors%2C+Attention+and+Performance+XIV%3A+Synergies+in+Experimental+Psychology%2C+A)
45. David C Plaut, James McClelland, Mark S. Seidenberg, and Karalyn Patterson. Understanding Normal and Impaired Word Reading: Computational Principles in Quasi-Regular Domains. Psychological Review, 103(1):56–115, 1996. [find](https://scholar.google.com/scholar?q=David+C+Plaut%2C+James+McClelland%2C+Mark+S%2E+Seidenberg%2C+and+Karalyn+Patterson%2E+Understanding+Normal+and+Impaired+Word+Reading%3A+Computational+Principles+in+Quasi%2DRegular+Domains%2E+Psychological+Review%2C+103)
46. David C. Plaut and Laura M. Gonnerman. Are Non-Semantic Morphological Effects Incompatible with a Distributed Connectionist Approach to Lexical Processing? Language and Cognitive Processes, 15(4-5): 445–485, 2000. [find](https://scholar.google.com/scholar?q=David+C%2E+Plaut+and+Laura+M%2E+Gonnerman%2E+Are+Non%2DSemantic+Morphological+Effects+Incompatible+with+a+Distributed+Connectionist+Approach+to+Lexical+Processing%3F+Language+and+Cognitive+Processes%2C+15%284%2D5%29%3A+4)
47. Laura M. Gonnerman, Mark S. Seidenberg, and Elaine S. Andersen. Graded Semantic and Phonological Similarity Effects in Priming: Evidence for a Distributed Connectionist Approach to Morphology. Journal of Experimental Psychology: General, 136(2):323–345, 2007. [find](https://scholar.google.com/scholar?q=Laura+M%2E+Gonnerman%2C+Mark+S%2E+Seidenberg%2C+and+Elaine+S%2E+Andersen%2E+Graded+Semantic+and+Phonological+Similarity+Effects+in+Priming%3A+Evidence+for+a+Distributed+Connectionist+Approach+to+Morphology%2E+Journal)
48. Mark Aronoff. Word Formation in Generative Grammar. MIT Press, Cambridge, MA, 1976. [find](https://scholar.google.com/scholar?q=Mark+Aronoff%2E+Word+Formation+in+Generative+Grammar%2E+MIT+Press%2C+Cambridge%2C+MA%2C+1976%2E)
49. Laurie Bauer. Morphological Productivity. Cambridge University Press, Cambridge, UK, 2001. [find](https://scholar.google.com/scholar?q=Laurie+Bauer%2E+Morphological+Productivity%2E+Cambridge+University+Press%2C+Cambridge%2C+UK%2C+2001%2E)
50. Martin Haspelmath and Andrea Sims. Understanding Morphology. Routledge, Oxford, UK, 2010. [find](https://scholar.google.com/scholar?q=Martin+Haspelmath+and+Andrea+Sims%2E+Understanding+Morphology%2E+Routledge%2C+Oxford%2C+UK%2C+2010%2E)
51. Laurie Bauer, Rochelle Lieber, and Ingo Plag. The Oxford Reference Guide to English Morphology. Oxford University Press, Oxford, UK, 2013. [find](https://scholar.google.com/scholar?q=Laurie+Bauer%2C+Rochelle+Lieber%2C+and+Ingo+Plag%2E+The+Oxford+Reference+Guide+to+English+Morphology%2E+Oxford+University+Press%2C+Oxford%2C+UK%2C+2013%2E)
52. Robert Beard. “Derivation”. In Andrew Spencer and Arnold M. Zwicky, editors, The Handbook of Morphology, pages 44–65, 2017. [find](https://scholar.google.com/scholar?q=Robert+Beard%2E+%E2%80%9CDerivation%E2%80%9D%2E+In+Andrew+Spencer+and+Arnold+M%2E+Zwicky%2C+editors%2C+The+Handbook+of+Morphology%2C+pages+44%E2%80%9365%2C+2017%2E)
53. Frank Anshen and Mark Aronoff. Producing Morphologically Complex Words. Linguistics, 26(4): 641–656, 1988. [find](https://scholar.google.com/scholar?q=Frank+Anshen+and+Mark+Aronoff%2E+Producing+Morphologically+Complex+Words%2E+Linguistics%2C+26%284%29%3A+641%E2%80%93656%2C+1988%2E)
54. R. Harald Baayen and Antoinette Renouf. Chronicling the Times: Productive Lexical Innovations in an English Newspaper. Language, 72(1):69–96, 1996. [find](https://scholar.google.com/scholar?q=R%2E+Harald+Baayen+and+Antoinette+Renouf%2E+Chronicling+the+Times%3A+Productive+Lexical+Innovations+in+an+English+Newspaper%2E+Language%2C+72%281%29%3A69%E2%80%9396%2C+1996%2E)
55. Frank Anshen and Mark Aronoff. Using Dictionaries to Study the Mental Lexicon. Brain and Language, 68(1-2):16–26, 1999. [find](https://scholar.google.com/scholar?q=Frank+Anshen+and+Mark+Aronoff%2E+Using+Dictionaries+to+Study+the+Mental+Lexicon%2E+Brain+and+Language%2C+68%281%2D2%29%3A16%E2%80%9326%2C+1999%2E)
56. Mark Lindsay. Rival Suffixes: Synonymy, Competition, and the Emergence of Productivity. Mediterranean Morphology Meetings, 8(0):192–203, 2012. [find](https://scholar.google.com/scholar?q=Mark+Lindsay%2E+Rival+Suffixes%3A+Synonymy%2C+Competition%2C+and+the+Emergence+of+Productivity%2E+Mediterranean+Morphology+Meetings%2C+8%280%29%3A192%E2%80%93203%2C+2012%2E)
57. Sabine Arndt-Lappe. Analogy in Suffix Rivalry: The Case of English -Ity and -Ness. English Language & Linguistics, 18(3):497–548, 2014. [find](https://scholar.google.com/scholar?q=Sabine+Arndt%2DLappe%2E+Analogy+in+Suffix+Rivalry%3A+The+Case+of+English+%2DIty+and+%2DNess%2E+English+Language+%26+Linguistics%2C+18%283%29%3A497%E2%80%93548%2C+2014%2E)
58. Joan Bresnan, Anna Cueni, Tatiana Nikitina, and R Harald Baayen. “Predicting the Dative Alternation”. In Gerlof Bouma, Irene Krämer, and Joost Zwarts, editors, Cognitive Foundations of Interpretation, pages 69–94. 2007. [find](https://scholar.google.com/scholar?q=Joan+Bresnan%2C+Anna+Cueni%2C+Tatiana+Nikitina%2C+and+R+Harald+Baayen%2E+%E2%80%9CPredicting+the+Dative+Alternation%E2%80%9D%2E+In+Gerlof+Bouma%2C+Irene+Kr%C3%A4mer%2C+and+Joost+Zwarts%2C+editors%2C+Cognitive+Foundations+of+Interpretation%2C)
59. Michael Walsh, Bernd Möbius, Travis Wade, and Hinrich Schütze. Multilevel Exemplar Theory. Cognitive Science, 34(4):537–582, 2010. [find](https://scholar.google.com/scholar?q=Michael+Walsh%2C+Bernd+M%C3%B6bius%2C+Travis+Wade%2C+and+Hinrich+Sch%C3%BCtze%2E+Multilevel+Exemplar+Theory%2E+Cognitive+Science%2C+34%284%29%3A537%E2%80%93582%2C+2010%2E)
60. Joan Bresnan. Formal Grammar, Usage Probabilities, and Auxiliary Contraction. Language, 97(1): 108–150, 2021. [find](https://scholar.google.com/scholar?q=Joan+Bresnan%2E+Formal+Grammar%2C+Usage+Probabilities%2C+and+Auxiliary+Contraction%2E+Language%2C+97%281%29%3A+108%E2%80%93150%2C+2021%2E)
61. Daniel Edmiston. A Systematic Analysis of Morphological Content in BERT Models for Multiple Languages. Preprint, arXiv 2004.03032, 2020. — [arXiv:2004.03032](https://arxiv.org/abs/2004.03032) [find](https://scholar.google.com/scholar?q=Daniel+Edmiston%2E+A+Systematic+Analysis+of+Morphological+Content+in+BERT+Models+for+Multiple+Languages%2E+Preprint%2C+arXiv+2004%2E03032%2C+2020%2E+%E2%80%94+%5BarXiv%3A2004%2E03032%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2004%2E03032%29)
62. Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. “DagoBERT: Generating Derivational Morphology with a Pretrained Language Model”. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3848–3861, 2020. [find](https://scholar.google.com/scholar?q=Valentin+Hofmann%2C+Janet+Pierrehumbert%2C+and+Hinrich+Sch%C3%BCtze%2E+%E2%80%9CDagoBERT%3A+Generating+Derivational+Morphology+with+a+Pretrained+Language+Model%E2%80%9D%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+)
63. Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. “Superbizarre Is Not Superb: Derivational Morphology Improves BERT’s Interpretation of Complex Words”. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pages 3594[–3608, 2021](https://md.research.elamur.ai/free/en/papers/hofmann-2021-superbizarre-derivational/).
64. Ron Sun. “Introduction to Computational Cognitive Modeling”. In Ron Sun, editor, The Cambridge Handbook of Computational Psychology, pages 3–19. 2008. [find](https://scholar.google.com/scholar?q=Ron+Sun%2E+%E2%80%9CIntroduction+to+Computational+Cognitive+Modeling%E2%80%9D%2E+In+Ron+Sun%2C+editor%2C+The+Cambridge+Handbook+of+Computational+Psychology%2C+pages+3%E2%80%9319%2E+2008%2E)
65. Robert Wilson and Anne Collins. Ten Simple Rules for the Computational Modeling of Behavioral Data. eLife, 8:e49547, 2019. [find](https://scholar.google.com/scholar?q=Robert+Wilson+and+Anne+Collins%2E+Ten+Simple+Rules+for+the+Computational+Modeling+of+Behavioral+Data%2E+eLife%2C+8%3Ae49547%2C+2019%2E)
66. Adrian Brasoveanu and Jakub Dotlačil. Computational Cognitive Modeling and Linguistic Theory. Springer, Cham, 2020. [find](https://scholar.google.com/scholar?q=Adrian+Brasoveanu+and+Jakub+Dotlac%CC%8Cil%2E+Computational+Cognitive+Modeling+and+Linguistic+Theory%2E+Springer%2C+Cham%2C+2020%2E)
67. Ekin Akyurek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. “Towards Tracing Knowledge in Language Models Back to the Training Data”. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2429–2446, 2022. [find](https://scholar.google.com/scholar?q=Ekin+Akyurek%2C+Tolga+Bolukbasi%2C+Frederick+Liu%2C+Binbin+Xiong%2C+Ian+Tenney%2C+Jacob+Andreas%2C+and+Kelvin+Guu%2E+%E2%80%9CTowards+Tracing+Knowledge+in+Language+Models+Back+to+the+Training+Data%E2%80%9D%2E+In+Findings+of+the+Asso)
68. Xiaochuang Han and Yulia Tsvetkov. ORCA: Interpreting Prompted Language Models via Locating Supporting Data Evidence in the Ocean of Pretraining Data. Preprint, arXiv 2205.12600, 2022. — [arXiv:2205.12600](https://arxiv.org/abs/2205.12600) [find](https://scholar.google.com/scholar?q=Xiaochuang+Han+and+Yulia+Tsvetkov%2E+ORCA%3A+Interpreting+Prompted+Language+Models+via+Locating+Supporting+Data+Evidence+in+the+Ocean+of+Pretraining+Data%2E+Preprint%2C+arXiv+2205%2E12600%2C+2022%2E+%E2%80%94+%5BarXiv%3A2205%2E1)
69. Yasaman Razeghi, Robert Logan IV, Matt Gardner, and Sameer Singh. “[Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning](https://md.research.elamur.ai/free/en/papers/razeghi-2022-impact-pretraining-term-frequencies-few/)”. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 840–854, 2022.
70. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schütze, and Yoav Goldberg. Measuring Causal Effects of Data Statistics on Language Model’s ‘Factual’ Predictions. Preprint, arXiv 2207.14251, 2023. — [arXiv:2207.14251](https://arxiv.org/abs/2207.14251) [find](https://scholar.google.com/scholar?q=Yanai+Elazar%2C+Nora+Kassner%2C+Shauli+Ravfogel%2C+Amir+Feder%2C+Abhilasha+Ravichander%2C+Marius+Mosbach%2C+Yonatan+Belinkov%2C+Hinrich+Sch%C3%BCtze%2C+and+Yoav+Goldberg%2E+Measuring+Causal+Effects+of+Data+Statistics+on+Lan)
71. Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [https://github.com/kingoflolz/mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax), 2021. [↗](https://github.com/kingoflolz/mesh-transformer-jax)
72. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. Preprint, arXiv 2101.00027, 2020. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+Shawn+Presser%2C+and+Connor+Leahy%2E+The+Pile%3A+An+800GB+Dataset+of+)
73. Adam Albright and Bruce Hayes. “[Modeling English Past Tense Intuitions with Minimal Generalization](https://md.research.elamur.ai/free/en/papers/albright-2002-modeling-english-past-tense-intuitions/)”. In Proceedings of the ACL-02 Workshop on Morphological and Phonological Learning, pages 58–69, 2002.
74. Adam Albright and Bruce Hayes. Rules vs. Analogy in English Past Tenses: A Computational/Experimental Study. Cognition, 90(2):119–161, 2003. [find](https://scholar.google.com/scholar?q=Adam+Albright+and+Bruce+Hayes%2E+Rules+vs%2E+Analogy+in+English+Past+Tenses%3A+A+Computational%2FExperimental+Study%2E+Cognition%2C+90%282%29%3A119%E2%80%93161%2C+2003%2E)
75. Robert M. Nosofsky. Relations between Exemplar-Similarity and Likelihood Models of Classification. Journal of Mathematical Psychology, 34(4):393–418, 1990. [find](https://scholar.google.com/scholar?q=Robert+M%2E+Nosofsky%2E+Relations+between+Exemplar%2DSimilarity+and+Likelihood+Models+of+Classification%2E+Journal+of+Mathematical+Psychology%2C+34%284%29%3A393%E2%80%93418%2C+1990%2E)
76. Boris New, Jessica Bourgin, Julien Barra, and Christophe Pallier. UniPseudo: A Universal Pseudoword Generator. Quarterly Journal of Experimental Psychology, 77(2):278–286, 2024. [find](https://scholar.google.com/scholar?q=Boris+New%2C+Jessica+Bourgin%2C+Julien+Barra%2C+and+Christophe+Pallier%2E+UniPseudo%3A+A+Universal+Pseudoword+Generator%2E+Quarterly+Journal+of+Experimental+Psychology%2C+77%282%29%3A278%E2%80%93286%2C+2024%2E)
77. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen… [find](https://scholar.google.com/scholar?q=Jack+W%2E+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+Eliza+Rutherford%2C+Tom+Hennigan%2C+Jacob+Menick%2C+A)
78. Mary L. Hare, Marilyn Ford, and William D. Marslen-Wilson. “Ambiguity and Frequency Effects in Regular Verb Inflection”. In Joan Bybee and Paul Hopper, editors, Frequency and the Emergence of Linguistic Structure, pages 181–200. 2001. [find](https://scholar.google.com/scholar?q=Mary+L%2E+Hare%2C+Marilyn+Ford%2C+and+William+D%2E+Marslen%2DWilson%2E+%E2%80%9CAmbiguity+and+Frequency+Effects+in+Regular+Verb+Inflection%E2%80%9D%2E+In+Joan+Bybee+and+Paul+Hopper%2C+editors%2C+Frequency+and+the+Emergence+of+Linguist)
79. Sabine Arndt-Lappe and Mirjam Ernestus. “Morpho-Phonological Alternations: The Role of Lexical Storage”. In V. Pirelli, I. Plag, and W.U. Dressler, editors, Word Knowledge and Word Usage: A Cross-Disciplinary Guide to the Mental Lexicon, pages 191–227. 2020. [find](https://scholar.google.com/scholar?q=Sabine+Arndt%2DLappe+and+Mirjam+Ernestus%2E+%E2%80%9CMorpho%2DPhonological+Alternations%3A+The+Role+of+Lexical+Storage%E2%80%9D%2E+In+V%2E+Pirelli%2C+I%2E+Plag%2C+and+W%2EU%2E+Dressler%2C+editors%2C+Word+Knowledge+and+Word+Usage%3A+A+Cross%2DDisc)
80. Sture Holm. A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics, 6(2):65–70, 1979. [find](https://scholar.google.com/scholar?q=Sture+Holm%2E+A+Simple+Sequentially+Rejective+Multiple+Test+Procedure%2E+Scandinavian+Journal+of+Statistics%2C+6%282%29%3A65%E2%80%9370%2C+1979%2E)
81. George Lakoff. Irregularity in Syntax. Holt, Rinehart and Winston, New York City, NY, 1970. [find](https://scholar.google.com/scholar?q=George+Lakoff%2E+Irregularity+in+Syntax%2E+Holt%2C+Rinehart+and+Winston%2C+New+York+City%2C+NY%2C+1970%2E)
82. Greville Corbett, Andrew Hippisley, Dunstan Brown, and Paul Marriott. “Frequency, Regularity, and the Paradigm”. In Joan Bybee and Paul Hopper, editors, Frequency and the Emergence of Linguistic Structure, pages 201–228. 2001. [find](https://scholar.google.com/scholar?q=Greville+Corbett%2C+Andrew+Hippisley%2C+Dunstan+Brown%2C+and+Paul+Marriott%2E+%E2%80%9CFrequency%2C+Regularity%2C+and+the+Paradigm%E2%80%9D%2E+In+Joan+Bybee+and+Paul+Hopper%2C+editors%2C+Frequency+and+the+Emergence+of+Linguistic+Struc)
83. Joan Bybee. From Usage to Grammar: The Mind’s Response to Repetition. Language, 82(4):711–733, 2006. [find](https://scholar.google.com/scholar?q=Joan+Bybee%2E+From+Usage+to+Grammar%3A+The+Mind%E2%80%99s+Response+to+Repetition%2E+Language%2C+82%284%29%3A711%E2%80%93733%2C+2006%2E)
84. Erez Lieberman, Jean-Baptiste Michel, Joe Jackson, Tina Tang, and Martin A. Nowak. Quantifying the Revolutionary Dynamics of Language. Nature, 449(7163):713–716, 2007. [find](https://scholar.google.com/scholar?q=Erez+Lieberman%2C+Jean%2DBaptiste+Michel%2C+Joe+Jackson%2C+Tina+Tang%2C+and+Martin+A%2E+Nowak%2E+Quantifying+the+Revolutionary+Dynamics+of+Language%2E+Nature%2C+449%287163%29%3A713%E2%80%93716%2C+2007%2E)
85. Robert Daland, Andrea D. Sims, and Janet Pierrehumbert. “Much Ado about Nothing: A Social Network Model of Russian Paradigmatic Gaps”. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 936–943, 2007. [find](https://scholar.google.com/scholar?q=Robert+Daland%2C+Andrea+D%2E+Sims%2C+and+Janet+Pierrehumbert%2E+%E2%80%9CMuch+Ado+about+Nothing%3A+A+Social+Network+Model+of+Russian+Paradigmatic+Gaps%E2%80%9D%2E+In+Proceedings+of+the+45th+Annual+Meeting+of+the+Association+of+C)
86. Aleksandar Petrov, Philip H.S. Torr, and Adel Bibi. “When do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations”. In International Conference on Learning Representations 2024, 2024. [find](https://scholar.google.com/scholar?q=Aleksandar+Petrov%2C+Philip+H%2ES%2E+Torr%2C+and+Adel+Bibi%2E+%E2%80%9CWhen+do+Prompting+and+Prefix%2DTuning+Work%3F+A+Theory+of+Capabilities+and+Limitations%E2%80%9D%2E+In+International+Conference+on+Learning+Representations+2024%2C+)
87. Joan Bybee. Regular Morphology and the Lexicon. Language and Cognitive Processes, 10(5):425–455, 1995. [find](https://scholar.google.com/scholar?q=Joan+Bybee%2E+Regular+Morphology+and+the+Lexicon%2E+Language+and+Cognitive+Processes%2C+10%285%29%3A425%E2%80%93455%2C+1995%2E)
88. Janet Pierrehumbert. Stochastic Phonology. Glot International, 5(6):195–207, 2001. [find](https://scholar.google.com/scholar?q=Janet+Pierrehumbert%2E+Stochastic+Phonology%2E+Glot+International%2C+5%286%29%3A195%E2%80%93207%2C+2001%2E)
89. Janet Pierrehumbert. Phonetic Diversity, Statistical Learning, and the Acquisition of Phonology. Language and Speech, 46(2-3):115–154, 2003. [find](https://scholar.google.com/scholar?q=Janet+Pierrehumbert%2E+Phonetic+Diversity%2C+Statistical+Learning%2C+and+the+Acquisition+of+Phonology%2E+Language+and+Speech%2C+46%282%2D3%29%3A115%E2%80%93154%2C+2003%2E)
90. Ian R. McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, Andrew Gritsevskiy, Daniel Wurgaft, Derik Kauffman, Gabriel Recchia, Jiacheng Liu, Joe Cavanagh, Max Weiss, Sicong Huang, The Floating Droid, Tom Tseng… — [Inverse Scaling: When Bigger Isn't Better](https://md.research.elamur.ai/free/en/papers/mckenzie-2023-inverse-scaling-bigger-isn-t/)
91. Gregory Stump. Rule Conflation in an Inferential-Realizational Theory of Morphotactics. Acta Linguistica Academica, 64(1):79–124, 2017. [find](https://scholar.google.com/scholar?q=Gregory+Stump%2E+Rule+Conflation+in+an+Inferential%2DRealizational+Theory+of+Morphotactics%2E+Acta+Linguistica+Academica%2C+64%281%29%3A79%E2%80%93124%2C+2017%2E)
92. Gregory Stump. Some Sources of Apparent Gaps in Derivational Paradigms. Morphology, 29(2):271–292, 2019. [find](https://scholar.google.com/scholar?q=Gregory+Stump%2E+Some+Sources+of+Apparent+Gaps+in+Derivational+Paradigms%2E+Morphology%2C+29%282%29%3A271%E2%80%93292%2C+2019%2E)
93. Janet Pierrehumbert. The Statistical Basis of an Unnatural Alternation,. In Louis Goldstein, Douglas H. Whalen, and Catherine T. Best, editors, Laboratory Phonology 8, pages 81–106. 2006. [find](https://scholar.google.com/scholar?q=Janet+Pierrehumbert%2E+The+Statistical+Basis+of+an+Unnatural+Alternation%2C%2E+In+Louis+Goldstein%2C+Douglas+H%2E+Whalen%2C+and+Catherine+T%2E+Best%2C+editors%2C+Laboratory+Phonology+8%2C+pages+81%E2%80%93106%2E+2006%2E)
94. Tanja Säily. Variation in Morphological Productivity in the BNC: Sociolinguistic and Methodological Considerations. Corpus Linguistics and Linguistic Theory, 7(1):119–141, 2011. [find](https://scholar.google.com/scholar?q=Tanja+S%C3%A4ily%2E+Variation+in+Morphological+Productivity+in+the+BNC%3A+Sociolinguistic+and+Methodological+Considerations%2E+Corpus+Linguistics+and+Linguistic+Theory%2C+7%281%29%3A119%E2%80%93141%2C+2011%2E)
95. Tanja Säily. Sociolinguistic Variation in English Derivational Productivity: Studies and Methods in Diachronic Corpus Linguistics. Société Néophilologique de Helsinki, Helsinki, 2014. [find](https://scholar.google.com/scholar?q=Tanja+S%C3%A4ily%2E+Sociolinguistic+Variation+in+English+Derivational+Productivity%3A+Studies+and+Methods+in+Diachronic+Corpus+Linguistics%2E+Soci%C3%A9t%C3%A9+N%C3%A9ophilologique+de+Helsinki%2C+Helsinki%2C+2014%2E)
96. Tanja Säily. Sociolinguistic Variation in Morphological Productivity in Eighteenth-Century English. Corpus Linguistics and Linguistic Theory, 12(1):129–151, 2016. [find](https://scholar.google.com/scholar?q=Tanja+S%C3%A4ily%2E+Sociolinguistic+Variation+in+Morphological+Productivity+in+Eighteenth%2DCentury+English%2E+Corpus+Linguistics+and+Linguistic+Theory%2C+12%281%29%3A129%E2%80%93151%2C+2016%2E)
97. Janet Pierrehumbert. Phonological Representation: Beyond Abstract versus Episodic. Annual Review of Linguistics, 2(1):33–52, 2016. [find](https://scholar.google.com/scholar?q=Janet+Pierrehumbert%2E+Phonological+Representation%3A+Beyond+Abstract+versus+Episodic%2E+Annual+Review+of+Linguistics%2C+2%281%29%3A33%E2%80%9352%2C+2016%2E)
98. Howard C Nusbaum, David B Pisoni, and Christopher K Davis. Sizing up the Hoosier Mental Lexicon. Research on Spoken Language Processing Report, 10(3):357–376, 1984. [find](https://scholar.google.com/scholar?q=Howard+C+Nusbaum%2C+David+B+Pisoni%2C+and+Christopher+K+Davis%2E+Sizing+up+the+Hoosier+Mental+Lexicon%2E+Research+on+Spoken+Language+Processing+Report%2C+10%283%29%3A357%E2%80%93376%2C+1984%2E)
99. Jeremy M. Needle, Janet Pierrehumbert, and Jennifer B. Hay. “Phonotactic and Morphological Effects in the Acceptability of Pseudowords”. In Andrea D. Sims, Adam Ussishkin, Jeff Parker, and Samantha Wray, editors, Morphological Diversity and Linguistic Cognition, pages 79–112. 2022. [find](https://scholar.google.com/scholar?q=Jeremy+M%2E+Needle%2C+Janet+Pierrehumbert%2C+and+Jennifer+B%2E+Hay%2E+%E2%80%9CPhonotactic+and+Morphological+Effects+in+the+Acceptability+of+Pseudowords%E2%80%9D%2E+In+Andrea+D%2E+Sims%2C+Adam+Ussishkin%2C+Jeff+Parker%2C+and+Samantha+Wr)
100. Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. “Predicting the Growth of Morphological Families from Social and Linguistic Factors”. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7273–7283, 2020. [find](https://scholar.google.com/scholar?q=Valentin+Hofmann%2C+Janet+Pierrehumbert%2C+and+Hinrich+Sch%C3%BCtze%2E+%E2%80%9CPredicting+the+Growth+of+Morphological+Families+from+Social+and+Linguistic+Factors%E2%80%9D%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Assoc)
101. Valentin Hofmann, Hinrich Schütze, and Janet Pierrehumbert. “A Graph Auto-Encoder Model of Derivational Morphology. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1127–1138, 2020. [find](https://scholar.google.com/scholar?q=Valentin+Hofmann%2C+Hinrich+Sch%C3%BCtze%2C+and+Janet+Pierrehumbert%2E+%E2%80%9CA+Graph+Auto%2DEncoder+Model+of+Derivational+Morphology%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Lingu)
102. Ethan G. Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P. Levy. [Testing the Predictions of Surprisal Theory in 11 Languages](https://md.research.elamur.ai/free/en/papers/wilcox-2023-testing-predictions-surprisal-theory-11/). Transactions of the Association for Computational Linguistics, 11:1451–1470, 2023.
103. Marc Brysbaert, Paweł Mandera, Samantha F. McCormick, and Emmanuel Keuleers. Word Prevalence Norms for 62,000 English Lemmas. Behavior Research Methods, 51(2):467–479, 2019. [find](https://scholar.google.com/scholar?q=Marc+Brysbaert%2C+Pawe%C5%82+Mandera%2C+Samantha+F%2E+McCormick%2C+and+Emmanuel+Keuleers%2E+Word+Prevalence+Norms+for+62%2C000+English+Lemmas%2E+Behavior+Research+Methods%2C+51%282%29%3A467%E2%80%93479%2C+2019%2E)
104. Kathleen Rastle, Matthew H. Davis, and Boris New. The Broth in My Brother’s Brothel: Morpho-Orthographic Segmentation in Visual Word Recognition. Psychonomic Bulletin and Review, 11(6): 1090–1098, 2004. [find](https://scholar.google.com/scholar?q=Kathleen+Rastle%2C+Matthew+H%2E+Davis%2C+and+Boris+New%2E+The+Broth+in+My+Brother%E2%80%99s+Brothel%3A+Morpho%2DOrthographic+Segmentation+in+Visual+Word+Recognition%2E+Psychonomic+Bulletin+and+Review%2C+11%286%29%3A+1090%E2%80%931098%2C+200)
105. Marcus Taft. Morphological Decomposition and the Reverse Base Frequency Effect. The Quarterly Journal of Experimental Psychology, 57(4):745–765, 2004. [find](https://scholar.google.com/scholar?q=Marcus+Taft%2E+Morphological+Decomposition+and+the+Reverse+Base+Frequency+Effect%2E+The+Quarterly+Journal+of+Experimental+Psychology%2C+57%284%29%3A745%E2%80%93765%2C+2004%2E)
106. Elisabeth Beyersmann, Johannes C. Ziegler, and Jonathan Grainger. Differences in the Processing of Prefixes and Suffixes Revealed by a Letter-Search Task. Scientific Studies of Reading, 19(5):360–373, 2015. [find](https://scholar.google.com/scholar?q=Elisabeth+Beyersmann%2C+Johannes+C%2E+Ziegler%2C+and+Jonathan+Grainger%2E+Differences+in+the+Processing+of+Prefixes+and+Suffixes+Revealed+by+a+Letter%2DSearch+Task%2E+Scientific+Studies+of+Reading%2C+19%285%29%3A360%E2%80%93373%2C)
107. Vittoria Dentella, Fritz Günther, and Evelina Leivada. Systematic Testing of Three Language Models Reveals Low Language Accuracy, Absence of Response Stability, and a Yes-Response Bias. Proceedings of the National Academy of Sciences, 120(51):e2309583120, 2023. [find](https://scholar.google.com/scholar?q=Vittoria+Dentella%2C+Fritz+G%C3%BCnther%2C+and+Evelina+Leivada%2E+Systematic+Testing+of+Three+Language+Models+Reveals+Low+Language+Accuracy%2C+Absence+of+Response+Stability%2C+and+a+Yes%2DResponse+Bias%2E+Proceedings+of)
108. Jennifer Hu and Roger Levy. “[Prompting is Not a Substitute for Probability Measurements in Large Language Models](https://md.research.elamur.ai/free/en/papers/hu-2023-prompting-is-not-substitute-probability/)”. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5040–5060, 2023.
109. Jennifer Hu, Kyle Mahowald, Gary Lupyan, Anna Ivanova, and Roger Levy. Language Models Align with Human Judgments on Key Grammatical Constructions. Proceedings of the National Academy of Sciences, 121(36):e2400917121, 2024. [find](https://scholar.google.com/scholar?q=Jennifer+Hu%2C+Kyle+Mahowald%2C+Gary+Lupyan%2C+Anna+Ivanova%2C+and+Roger+Levy%2E+Language+Models+Align+with+Human+Judgments+on+Key+Grammatical+Constructions%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+121)
110. Willard V. Quine. Word and Object. MIT Press, Cambridge, MA, 1960. [find](https://scholar.google.com/scholar?q=Willard+V%2E+Quine%2E+Word+and+Object%2E+MIT+Press%2C+Cambridge%2C+MA%2C+1960%2E)
111. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. [Faith and Fate: Limits of Transformers on Compositionality](https://md.research.elamur.ai/free/en/papers/dziri-2023-faith-fate-limits-transformers-compositionality/).…
112. R. Harald Baayen, Richard Piepenbrock, and Leon Gulikers. The CELEX Lexical Database (CD-ROM). Linguistic Data Consortium, Philadelphia, PA, 1995. [find](https://scholar.google.com/scholar?q=R%2E+Harald+Baayen%2C+Richard+Piepenbrock%2C+and+Leon+Gulikers%2E+The+CELEX+Lexical+Database+%28CD%2DROM%29%2E+Linguistic+Data+Consortium%2C+Philadelphia%2C+PA%2C+1995%2E)
113. Claudia H. Sánchez-Gutiérrez, Hugo Mailhot, S. Hélène Deacon, and Maximiliano A. Wilson. [MorphoLex: A Derivational Morphological Database for 70,000 English Words](https://md.research.elamur.ai/free/en/papers/gutierrez-2018-sanchezgutierrez-morpholex/). Behavior Research Methods, 50(4):1568–1580, 2018.
114. Jacqueline Laws and Chris Ryder. Getting the Measure of Derivational Morphology in Adult Speech: A Corpus Analysis using MorphoQuantics. Language Studies Working Papers, 6:3–17, 2014. [find](https://scholar.google.com/scholar?q=Jacqueline+Laws+and+Chris+Ryder%2E+Getting+the+Measure+of+Derivational+Morphology+in+Adult+Speech%3A+A+Corpus+Analysis+using+MorphoQuantics%2E+Language+Studies+Working+Papers%2C+6%3A3%E2%80%9317%2C+2014%2E)
115. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
116. Harald Baayen and Rochelle Lieber. Productivity and English Derivation: A Corpus-Based Study. Linguistics, 29(5):801–844, 1991. [find](https://scholar.google.com/scholar?q=Harald+Baayen+and+Rochelle+Lieber%2E+Productivity+and+English+Derivation%3A+A+Corpus%2DBased+Study%2E+Linguistics%2C+29%285%29%3A801%E2%80%93844%2C+1991%2E)
117. Kilem Li Gwet. Computing Inter-Rater Reliability and Its Variance in the Presence of High Agreement. British Journal of Mathematical and Statistical Psychology, 61(1):29–48, 2008. [find](https://scholar.google.com/scholar?q=Kilem+Li+Gwet%2E+Computing+Inter%2DRater+Reliability+and+Its+Variance+in+the+Presence+of+High+Agreement%2E+British+Journal+of+Mathematical+and+Statistical+Psychology%2C+61%281%29%3A29%E2%80%9348%2C+2008%2E)
118. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. [Pre-Train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/). ACM Computing Surveys, 55(9):1–35, September 2023.
119. David Crystal. The Cambridge Encyclopedia of the English Language. Cambridge University Press, Cambridge, UK, 1997. [find](https://scholar.google.com/scholar?q=David+Crystal%2E+The+Cambridge+Encyclopedia+of+the+English+Language%2E+Cambridge+University+Press%2C+Cambridge%2C+UK%2C+1997%2E)
120. Ingo Plag. Word-Formation in English. Cambridge University Press, Cambridge, UK, 2003. Supporting Information Derivative Statistics [find](https://scholar.google.com/scholar?q=Ingo+Plag%2E+Word%2DFormation+in+English%2E+Cambridge+University+Press%2C+Cambridge%2C+UK%2C+2003%2E+Supporting+Information+Derivative+Statistics)

*Updated 3 November 2026*