---
title: "Anything Goes? A Crosslinguistic Study of (Im)possible"
description: "This study tests that objection beyond English, the only language covered by earlier work (Kallini et al.… 86 references, cited by 7 papers here."
authors: ["Xiulin Yang", "Tatsuya Aoyama", "Yuekun Yao", "Ethan Gotlieb Wilcox"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.1264/"
doi: "10.18653/v1/2025.acl-long.1264"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yang-2025-anything-goes-crosslinguistic-study-im/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2025-anything-goes-crosslinguistic-study-im/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/9/) / Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs

# Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs

**Authors:** [Xiulin Yang](https://md.research.elamur.ai/free/en/authors/xiulin-yang/), [Tatsuya Aoyama](https://md.research.elamur.ai/free/en/authors/tatsuya-aoyama/), Yuekun Yao and [Ethan Gotlieb Wilcox](https://md.research.elamur.ai/free/en/authors/ethan-wilcox/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.1264/) · [PDF](https://aclanthology.org/2025.acl-long.1264.pdf) · [DOI 10.18653/v1/2025.acl-long.1264](https://doi.org/10.18653/v1/2025.acl-long.1264) · [OpenAlex](https://openalex.org/W4412889906)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/yang-2025-anything-goes-crosslinguistic-study-im/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
A common objection to using language models as models of human acquisition is that they supposedly learn any input pattern equally well, natural or not. This study tests that objection beyond English, the only language covered by earlier work (Kallini et al., 2024). The authors built two sentence-aligned parallel corpora from OPUS sources, OPUS12 and OPUS30, so that content is held constant across languages. They trained GPT-2 small on 12 attested languages from 4 families, on "impossible" versions of each (shuffled, reversed or otherwise deterministically reordered tokens), and on noun-phrase word orders that Greenberg's Universal 20 marks as typologically unattested.

Within each language, GPT-2 generally reaches lower perplexity on the real language than on its impossible variants, though some locally shuffled variants come close. Across languages, however, the two classes do not separate cleanly. For unattested NP orders, perplexity alone shows no preference for attested orders, but a targeted generalization test does. The authors conclude that LMs carry soft, human-like inductive biases that are weaker than those of human learners. This refutes the "anything goes" view while still showing limits.

## Key points
- **Corpora:**
  - OPUS12 covers 12 languages, with about 1M sentences and roughly 10.1M English words. That word count is comparable to a child's input over the first 2–5 years.
  - OPUS30 covers 30 languages, with 48K sentences and about 0.7M words, and serves as an extra test set.
  - Both corpora draw on NLLB, TED2020, the Bible, OpenSubtitles and CCAligned.
- **Setup:**
  - GPT-2 small was trained per language with 3 seeds and 1200 training steps.
  - Models were scored by geometric mean perplexity on a parallel 10K-sentence test split.
  - Each language got its own tokenizer, rather than a shared multilingual vocabulary, to avoid favoring Latin-script languages.
- **Replication check:** A scaled-down (10M-word) rerun of Kallini et al. (2024) reproduced their ranking exactly (Spearman's ρ = 1).
- **Separability:** Perplexity separated attested from impossible languages with a score of 0.75. This is above chance but not perfect.
- **Confound:** Normalizing NP order seems to lower entropy, which may explain why some unattested NP orders get lower perplexity. Code and data are released at github.com/picol-georgetown/multilingual-LM.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Saarland University](https://md.research.elamur.ai/free/en/universities/saarland-university/) — University, Saarbrücken, Germany · 148 works · 18,287 citations
- [Georgetown University](https://md.research.elamur.ai/free/en/universities/georgetown-university/) — University, Washington, United States · 88 works · 4,732 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

## Cite

Cite as: Xiulin Yang, Tatsuya Aoyama, Yuekun Yao and Ethan Gotlieb Wilcox. 2025. Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yang2025anything,
  title = {{Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs}},
  author = {Xiulin Yang, Tatsuya Aoyama, Yuekun Yao and Ethan Gotlieb Wilcox},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1264},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1264}
}
```

- [← previous Agent-as-Judge for Factual Summarization of Long Narratives](https://md.research.elamur.ai/free/en/papers/jeong-2025-agent-as-judge-factual-summarization/)
- [next → AraHealthQA 2025: The First Shared Task on Arabic Health Question Answering](https://md.research.elamur.ai/free/en/papers/alhuzali-2025-arahealthqa-2025-first-shared-task/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/9/)

## References

86 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. [AraGPT2: Pre-trained transformer for Arabic language generation](https://md.research.elamur.ai/free/en/papers/antoun-2021-aragpt2-pre-trained-transformer-arabic/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 196–207, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
2. Catherine Arnett and Benjamin Bergen. 2025. [Why do language models perform worse for morphologically complex languages?](https://md.research.elamur.ai/free/en/papers/arnett-2025-why-do-language-models-perform/) In Proceedings of the 31st International Conference on Computational Linguistics, pages 6607–6623, Abu Dhabi, UAE. Association for Computational Linguistics.
3. Florbela Barreto, António Branco, Eduardo Ferreira, Amália Mendes, Maria Fernanda Bacelar do Nascimento, Filipe Nunes, and João Ricardo Silva. 2006. Open resources and tools for the shallow processing of Portuguese: The TagShare project. In Proceedings of the Fifth International Conference on Langua… [find](https://scholar.google.com/scholar?q=Florbela+Barreto%2C+Ant%C3%B3nio+Branco%2C+Eduardo+Ferreira%2C+Am%C3%A1lia+Mendes%2C+Maria+Fernanda+Bacelar+do+Nascimento%2C+Filipe+Nunes%2C+and+Jo%C3%A3o+Ricardo+Silva%2E+2006%2E+Open+resources+and+tools+for+the+shallow+processing)
4. Johan J Bolhuis, Stephen Crain, Sandiway Fong, and Andrea Moro. 2024. Three reasons why AI doesn’t model human language. Nature, 627(8004):489. [find](https://scholar.google.com/scholar?q=Johan+J+Bolhuis%2C+Stephen+Crain%2C+Sandiway+Fong%2C+and+Andrea+Moro%2E+2024%2E+Three+reasons+why+AI+doesn%E2%80%99t+model+human+language%2E+Nature%2C+627%288004%29%3A489%2E)
5. Nadav Borenstein, Anej Svete, Robin Chan, Josef Valvoda, Franz Nowak, Isabelle Augenstein, Eleanor Chodroff, and Ryan Cotterell. 2024. What languages are easy to language-model? a perspective from learning probabilistic regular languages. In Proceedings of the 62nd Annual Meeting of the Association… [find](https://scholar.google.com/scholar?q=Nadav+Borenstein%2C+Anej+Svete%2C+Robin+Chan%2C+Josef+Valvoda%2C+Franz+Nowak%2C+Isabelle+Augenstein%2C+Eleanor+Chodroff%2C+and+Ryan+Cotterell%2E+2024%2E+What+languages+are+easy+to+language%2Dmodel%3F+a+perspective+from+lea)
6. Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Benjamin K. Bergen. 2024. Goldfish: Monolingual language models for 350 languages. Preprint, arXiv:2408.10441. [find](https://scholar.google.com/scholar?q=Tyler+A%2E+Chang%2C+Catherine+Arnett%2C+Zhuowen+Tu%2C+and+Benjamin+K%2E+Bergen%2E+2024%2E+Goldfish%3A+Monolingual+language+models+for+350+languages%2E+Preprint%2C+arXiv%3A2408%2E10441%2E)
7. Noam Chomsky and Andrea Moro. 2022. The secrets of words. MIT Press. [find](https://scholar.google.com/scholar?q=Noam+Chomsky+and+Andrea+Moro%2E+2022%2E+The+secrets+of+words%2E+MIT+Press%2E)
8. Noam Chomsky, Lan Roberts, and Jeffrey Watumull. 2023. Noam Chomsky: The false promise of Chat-GPT. The New York Times. Accessed: 2024-12-16. [find](https://scholar.google.com/scholar?q=Noam+Chomsky%2C+Lan+Roberts%2C+and+Jeffrey+Watumull%2E+2023%2E+Noam+Chomsky%3A+The+false+promise+of+Chat%2DGPT%2E+The+New+York+Times%2E+Accessed%3A+2024%2D12%2D16%2E)
9. Christos Christodouloupoulos and Mark Steedman. 2015. [A massively parallel corpus: the Bible in 100 languages](https://md.research.elamur.ai/free/en/papers/christodouloupoulos-2014-massively-parallel-corpus-bible-100/). Language resources and evaluation, 49:375–395.
10. Guglielmo Cinque. 2005. Deriving Greenberg’s Universal 20 and its exceptions. Linguistic inquiry, 36(3):315–332. [find](https://scholar.google.com/scholar?q=Guglielmo+Cinque%2E+2005%2E+Deriving+Greenberg%E2%80%99s+Universal+20+and+its+exceptions%2E+Linguistic+inquiry%2C+36%283%29%3A315%E2%80%93332%2E)
11. Ryan Cotterell, Sabrina J. Mielke, Jason Eisner, and Brian Roark. 2018. [Are all languages equally hard to language-model?](https://md.research.elamur.ai/free/en/papers/cotterell-2018-are-all-languages-equally-hard/) In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages…
12. Jennifer Culbertson, Julie Franck, Guillaume Braquet, Magda Barrera Navarro, and Inbal Arnon. 2020. A learning bias for word order harmony: Evidence from speakers of non-harmonic languages. Cognition, 204:104392. [find](https://scholar.google.com/scholar?q=Jennifer+Culbertson%2C+Julie+Franck%2C+Guillaume+Braquet%2C+Magda+Barrera+Navarro%2C+and+Inbal+Arnon%2E+2020%2E+A+learning+bias+for+word+order+harmony%3A+Evidence+from+speakers+of+non%2Dharmonic+languages%2E+Cognition%2C)
13. Jennifer Culbertson and Elissa L Newport. 2015. Harmonic biases in child learners: In support of language universals. Cognition, 139:71–82. [find](https://scholar.google.com/scholar?q=Jennifer+Culbertson+and+Elissa+L+Newport%2E+2015%2E+Harmonic+biases+in+child+learners%3A+In+support+of+language+universals%2E+Cognition%2C+139%3A71%E2%80%9382%2E)
14. Jennifer Culbertson and Elissa L Newport. 2017. Innovation of word order harmony across development. Open Mind, 1(2):91–100. [find](https://scholar.google.com/scholar?q=Jennifer+Culbertson+and+Elissa+L+Newport%2E+2017%2E+Innovation+of+word+order+harmony+across+development%2E+Open+Mind%2C+1%282%29%3A91%E2%80%93100%2E)
15. Jennifer Culbertson, Paul Smolensky, and Géraldine Legendre. 2012. Learning biases predict a word order universal. Cognition, 122(3):306–329. [find](https://scholar.google.com/scholar?q=Jennifer+Culbertson%2C+Paul+Smolensky%2C+and+G%C3%A9raldine+Legendre%2E+2012%2E+Learning+biases+predict+a+word+order+universal%2E+Cognition%2C+122%283%29%3A306%E2%80%93329%2E)
16. Christine Cuskley, Rebecca Woods, and Molly Flaherty. 2024. The limitations of large language models for understanding human language and cognition. Open Mind, 8:1058–1083. [find](https://scholar.google.com/scholar?q=Christine+Cuskley%2C+Rebecca+Woods%2C+and+Molly+Flaherty%2E+2024%2E+The+limitations+of+large+language+models+for+understanding+human+language+and+cognition%2E+Open+Mind%2C+8%3A1058%E2%80%931083%2E)
17. Forrest Davis and Marten van Schijndel. 2020. Recurrent neural network language models always learn English-like relative clause attachment. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1979–1990, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Forrest+Davis+and+Marten+van+Schijndel%2E+2020%2E+Recurrent+neural+network+language+models+always+learn+English%2Dlike+relative+clause+attachment%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Associatio)
18. Iria de Dios-Flores, Juan Garcia Amboage, and Marcos Garcia. 2023. Dependency resolution at the syntax-semantics interface: psycholinguistic and computational insights on control dependencies. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… [find](https://scholar.google.com/scholar?q=Iria+de+Dios%2DFlores%2C+Juan+Garcia+Amboage%2C+and+Marcos+Garcia%2E+2023%2E+Dependency+resolution+at+the+syntax%2Dsemantics+interface%3A+psycholinguistic+and+computational+insights+on+control+dependencies%2E+In+Proc)
19. Rodolfo Delmonte, Antonella Bristot, and Sara Tonelli. 2007. VIT-Venice Italian Treebank: Syntactic and quantitative features. In Sixth International Workshop on Treebanks and Linguistic Theories, volume 1, pages 43–54. Northern European Association for Language Technol. [find](https://scholar.google.com/scholar?q=Rodolfo+Delmonte%2C+Antonella+Bristot%2C+and+Sara+Tonelli%2E+2007%2E+VIT%2DVenice+Italian+Treebank%3A+Syntactic+and+quantitative+features%2E+In+Sixth+International+Workshop+on+Treebanks+and+Linguistic+Theories%2C+vol)
20. Vittoria Dentella, Fritz Guenther, and Evelina Leivada. 2024. Language in vivo vs. in silico: Size matters but larger language models still do not comprehend language on a par with humans. arXiv preprint arXiv:2404.14883. — [arXiv:2404.14883](https://arxiv.org/abs/2404.14883) [find](https://scholar.google.com/scholar?q=Vittoria+Dentella%2C+Fritz+Guenther%2C+and+Evelina+Leivada%2E+2024%2E+Language+in+vivo+vs%2E+in+silico%3A+Size+matters+but+larger+language+models+still+do+not+comprehend+language+on+a+par+with+humans%2E+arXiv+prepr)
21. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
22. Matthew S Dryer. 1992. The Greenbergian word order correlations. Language, 68(1):81–138. [find](https://scholar.google.com/scholar?q=Matthew+S+Dryer%2E+1992%2E+The+Greenbergian+word+order+correlations%2E+Language%2C+68%281%29%3A81%E2%80%93138%2E)
23. Stefan Dumitrescu. 2024. GPT-Neo Romanian 780M. Hugging Face. Available at: [https://huggingface.co/dumitrescustefan/](https://huggingface.co/dumitrescustefan/) gpt-neo-romanian-780m. [↗](https://huggingface.co/dumitrescustefan/)
24. Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzmán, and Philipp Koehn. 2020. [CCAligned: A massive collection of cross-lingual web-document pairs](https://md.research.elamur.ai/free/en/papers/kishky-2020-ccaligned-massive-collection-cross-lingual/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020), pages 5960–5969, Online. Association for Computa…
25. Danny Fox and Roni Katzir. 2024. Large language models and theoretical linguistics. Theoretical Linguistics, 50(1-2):71–76. [find](https://scholar.google.com/scholar?q=Danny+Fox+and+Roni+Katzir%2E+2024%2E+Large+language+models+and+theoretical+linguistics%2E+Theoretical+Linguistics%2C+50%281%2D2%29%3A71%E2%80%9376%2E)
26. Richard Futrell and Kyle Mahowald. 2025. How linguistics learned to stop worrying and love the language models. arXiv preprint arXiv:2501.17047. — [arXiv:2501.17047](https://arxiv.org/abs/2501.17047) [find](https://scholar.google.com/scholar?q=Richard+Futrell+and+Kyle+Mahowald%2E+2025%2E+How+linguistics+learned+to+stop+worrying+and+love+the+language+models%2E+arXiv+preprint+arXiv%3A2501%2E17047%2E+%E2%80%94+%5BarXiv%3A2501%2E17047%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2501%2E17047%29)
27. Dedre Gentner and Melissa Bowerman. 2009. Why some spatial semantic categories are harder to learn than others: The typological prevalence hypothesis. Crosslinguistic approaches to the psychology of language: Research in the tradition of Dan Isaac Slobin, 465:480. [find](https://scholar.google.com/scholar?q=Dedre+Gentner+and+Melissa+Bowerman%2E+2009%2E+Why+some+spatial+semantic+categories+are+harder+to+learn+than+others%3A+The+typological+prevalence+hypothesis%2E+Crosslinguistic+approaches+to+the+psychology+of+l)
28. Edward Gibson, Richard Futrell, Steven P Piantadosi, Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger Levy. 2019. How efficiency shapes human language. Trends in cognitive sciences, 23(5):389– 407. [find](https://scholar.google.com/scholar?q=Edward+Gibson%2C+Richard+Futrell%2C+Steven+P+Piantadosi%2C+Isabelle+Dautriche%2C+Kyle+Mahowald%2C+Leon+Bergen%2C+and+Roger+Levy%2E+2019%2E+How+efficiency+shapes+human+language%2E+Trends+in+cognitive+sciences%2C+23%285%29%3A389)
29. David Gil. 2008. How complex are isolating languages. Language. [find](https://scholar.google.com/scholar?q=David+Gil%2E+2008%2E+How+complex+are+isolating+languages%2E+Language%2E)
30. Omer Goldman, Avi Caciularu, Matan Eyal, Kris Cao, Idan Szpektor, and Reut Tsarfaty. 2024. Unpacking tokenization: Evaluating text compression and its correlation with model performance. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2274–2286, Bangkok, Thailand. Assoc… [find](https://scholar.google.com/scholar?q=Omer+Goldman%2C+Avi+Caciularu%2C+Matan+Eyal%2C+Kris+Cao%2C+Idan+Szpektor%2C+and+Reut+Tsarfaty%2E+2024%2E+Unpacking+tokenization%3A+Evaluating+text+compression+and+its+correlation+with+model+performance%2E+In+Findings+o)
31. Adam Goodkind and Klinton Bicknell. 2018. [Predictive power of word surprisal for reading times is a linear function of language model quality](https://md.research.elamur.ai/free/en/papers/goodkind-2018-predictive-power-word-surprisal-reading/). In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018), pages 10–18, Salt Lake City, Utah. Association for Comput…
32. Joseph H Greenberg et al. 1963. Some universals of grammar with particular reference to the order of meaningful elements. Universals of language, 2:73– 113. [find](https://scholar.google.com/scholar?q=Joseph+H+Greenberg+et+al%2E+1963%2E+Some+universals+of+grammar+with+particular+reference+to+the+order+of+meaningful+elements%2E+Universals+of+language%2C+2%3A73%E2%80%93+113%2E)
33. Yeb Havinga. 2023. GPT Neo 1.3B pre-trained on cleaned Dutch mC4. Accessed: 2024-12-10. iGeniusAI. 2024. Italia-9b-instruct-v0.1. [https://huggingface.co/iGeniusAI/](https://huggingface.co/iGeniusAI/) Italia-9B-Instruct-v0.1. Accessed: 2024-06-17. [↗](https://huggingface.co/iGeniusAI/)
34. Tamar Johnson, Kexin Gao, Kenny Smith, Hugh Rabagliati, and Jennifer Culbertson. 2021. Investigating the effects of i-complexity and e-complexity on the learnability of morphological systems. J. Lang. Model., 9. [find](https://scholar.google.com/scholar?q=Tamar+Johnson%2C+Kexin+Gao%2C+Kenny+Smith%2C+Hugh+Rabagliati%2C+and+Jennifer+Culbertson%2E+2021%2E+Investigating+the+effects+of+i%2Dcomplexity+and+e%2Dcomplexity+on+the+learnability+of+morphological+systems%2E+J%2E+Lang%2E)
35. John E Joseph and Frederick J Newmeyer. 2012. ‘All languages are equally complex’. Historiographia linguistica, 39. [find](https://scholar.google.com/scholar?q=John+E+Joseph+and+Frederick+J+Newmeyer%2E+2012%2E+%E2%80%98All+languages+are+equally+complex%E2%80%99%2E+Historiographia+linguistica%2C+39%2E)
36. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, Valencia, Spain. Associatio… [find](https://scholar.google.com/scholar?q=Armand+Joulin%2C+Edouard+Grave%2C+Piotr+Bojanowski%2C+and+Tomas+Mikolov%2E+2017%2E+Bag+of+tricks+for+efficient+text+classification%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Associati)
37. Julie Kallini, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, and Christopher Potts. 2024. [Mission: Impossible language models](https://md.research.elamur.ai/free/en/papers/kallini-2024-mission-impossible-language-models/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14691–14714, Bangkok, Thailand. Association…
38. Roni Katzir. 2023. Why large language models are poor theories of human linguistic cognition: A reply to Piantadosi. Biolinguistics, 17:1–12. [find](https://scholar.google.com/scholar?q=Roni+Katzir%2E+2023%2E+Why+large+language+models+are+poor+theories+of+human+linguistic+cognition%3A+A+reply+to+Piantadosi%2E+Biolinguistics%2C+17%3A1%E2%80%9312%2E)
39. H. Toprak Kesgin, M. Kaan Yuce, Eren Dogan, M. Ege-men Uzun, Atahan Uz, H. Emre Seyrek, Ahmed Zeer, and M. Fatih Amasyali. 2024. Introducing cosmosGPT: Monolingual training for Turkish language models. In 2024 International Conference on INnovations in Intelligent SysTems and Applications (INISTA),… [find](https://scholar.google.com/scholar?q=H%2E+Toprak+Kesgin%2C+M%2E+Kaan+Yuce%2C+Eren+Dogan%2C+M%2E+Ege%2Dmen+Uzun%2C+Atahan+Uz%2C+H%2E+Emre+Seyrek%2C+Ahmed+Zeer%2C+and+M%2E+Fatih+Amasyali%2E+2024%2E+Introducing+cosmosGPT%3A+Monolingual+training+for+Turkish+language+models)
40. Najoung Kim and Tal Linzen. 2020. COGS: A compositional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9087–9105, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Najoung+Kim+and+Tal+Linzen%2E+2020%2E+COGS%3A+A+compositional+generalization+challenge+based+on+semantic+interpretation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Proces)
41. Christo Kirov and Ryan Cotterell. 2018. [Recurrent neural networks in linguistic theory: Revisiting Pinker and Prince (1988) and the past tense debate](https://md.research.elamur.ai/free/en/papers/kirov-2018-recurrent-neural-networks-linguistic-theory/). Transactions of the Association for Computational Linguistics, 6:651–665.
42. Alexander Koplenig and Sascha Wolfer. 2023. Languages with more speakers tend to be harder to (machine-) learn. Scientific Reports, 13(1):18521. [find](https://scholar.google.com/scholar?q=Alexander+Koplenig+and+Sascha+Wolfer%2E+2023%2E+Languages+with+more+speakers+tend+to+be+harder+to+%28machine%2D%29+learn%2E+Scientific+Reports%2C+13%281%29%3A18521%2E)
43. Alexander Koplenig, Sascha Wolfer, and Peter Meyer. 2023. A large quantitative analysis of written language challenges the idea that all languages are equally complex. Scientific Reports, 13(1):15351. [find](https://scholar.google.com/scholar?q=Alexander+Koplenig%2C+Sascha+Wolfer%2C+and+Peter+Meyer%2E+2023%2E+A+large+quantitative+analysis+of+written+language+challenges+the+idea+that+all+languages+are+equally+complex%2E+Scientific+Reports%2C+13%281%29%3A15351%2E)
44. Tatsuki Kuribayashi, Yohei Oseki, and Timothy Baldwin. 2024. Psychometric predictive power of large language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1983–2005, Mexico City, Mexico. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tatsuki+Kuribayashi%2C+Yohei+Oseki%2C+and+Timothy+Baldwin%2E+2024%2E+Psychometric+predictive+power+of+large+language+models%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+NAACL+2024%2C+pages+198)
45. Tatsuki Kuribayashi, Yohei Oseki, Souhaib Ben Taieb, Kentaro Inui, and Timothy Baldwin. 2025. [Large language models are human-like internally](https://md.research.elamur.ai/free/en/papers/kuribayashi-2025-large-language-models-are-human/). arXiv preprint arXiv:2502.01615.
46. Nur Lan, Emmanuel Chemla, and Roni Katzir. 2024. Large language models and the argument from the poverty of the stimulus. Linguistic Inquiry, pages 1–56. [find](https://scholar.google.com/scholar?q=Nur+Lan%2C+Emmanuel+Chemla%2C+and+Roni+Katzir%2E+2024%2E+Large+language+models+and+the+argument+from+the+poverty+of+the+stimulus%2E+Linguistic+Inquiry%2C+pages+1%E2%80%9356%2E)
47. Julien Launay, E.l. Tommasone, Baptiste Pannier, François Boniface, Amélie Chatelain, Alessandro Cappelli, Iacopo Poli, and Djamé Seddah. 2022. PAGnol: An extra-large French generative model. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4275–4284, Marseille, F… [find](https://scholar.google.com/scholar?q=Julien+Launay%2C+E%2El%2E+Tommasone%2C+Baptiste+Pannier%2C+Fran%C3%A7ois+Boniface%2C+Am%C3%A9lie+Chatelain%2C+Alessandro+Cappelli%2C+Iacopo+Poli%2C+and+Djam%C3%A9+Seddah%2E+2022%2E+PAGnol%3A+An+extra%2Dlarge+French+generative+model%2E+In+Proce)
48. Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa. 2023. [XLM-V: Overcoming the vocabulary bottleneck in multilingual masked language models](https://md.research.elamur.ai/free/en/papers/liang-2023-xlm-v-overcoming-vocabulary-bottleneck/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p…
49. Pierre Lison and Jörg Tiedemann. 2016. [OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles](https://md.research.elamur.ai/free/en/papers/lison-2016-opensubtitles2016-extracting-large-parallel-corpora/). In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 923–929, Portorož, Slovenia. European Language Resources Association (EL…
50. Ricardo Lopes, Joao Magalhaes, and David Semedo. 2024. GlórIA: A generative and open large language model for Portuguese. In Proceedings of the 16th International Conference on Computational Processing of Portuguese - Vol. 1, pages 441–453, Santiago de Compostela, Galicia/Spain. Association for Comp… [find](https://scholar.google.com/scholar?q=Ricardo+Lopes%2C+Joao+Magalhaes%2C+and+David+Semedo%2E+2024%2E+Gl%C3%B3rIA%3A+A+generative+and+open+large+language+model+for+Portuguese%2E+In+Proceedings+of+the+16th+International+Conference+on+Computational+Processin)
51. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. [Building a large annotated corpus of English: The Penn Treebank](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/). Computational Linguistics, 19(2):313–330.
52. John H McWhorter. 2001. The worlds simplest grammars are creole grammars. Linguistic Typology, 5(2-3):125–166. [find](https://scholar.google.com/scholar?q=John+H+McWhorter%2E+2001%2E+The+worlds+simplest+grammars+are+creole+grammars%2E+Linguistic+Typology%2C+5%282%2D3%29%3A125%E2%80%93166%2E)
53. John H McWhorter. 2011. Linguistic simplicity and complexity: Why do languages undress?, volume 1. Walter de Gruyter. [find](https://scholar.google.com/scholar?q=John+H+McWhorter%2E+2011%2E+Linguistic+simplicity+and+complexity%3A+Why+do+languages+undress%3F%2C+volume+1%2E+Walter+de+Gruyter%2E)
54. Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell, and Roger Levy. 2021. [Revisiting the Uniform Information Density hypothesis](https://md.research.elamur.ai/free/en/papers/meister-2021-revisiting-uniform-information-density-hypothesis/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 963– 980, Online and Punta Cana, Dominican Republic…
55. Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. 2019. [What kind of language is hard to language-model?](https://md.research.elamur.ai/free/en/papers/mielke-2019-what-kind-language-is-hard/) In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4975–4989, Florence, Italy. Association for Computational Linguistic…
56. Raphaël Millière. 2024. [Language models as models of language](https://md.research.elamur.ai/free/en/papers/milliere-2024-language-models-models-language/). arXiv preprint arXiv:2408.07144.
57. Jeff Mitchell and Jeffrey Bowers. 2020. Priorless recurrent networks learn curiously. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5147–5158, Barcelona, Spain (Online). International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jeff+Mitchell+and+Jeffrey+Bowers%2E+2020%2E+Priorless+recurrent+networks+learn+curiously%2E+In+Proceedings+of+the+28th+International+Conference+on+Computational+Linguistics%2C+pages+5147%E2%80%935158%2C+Barcelona%2C+Spai)
58. Andrea Moro. 2023. Embodied syntax: impossible languages and the irreducible difference between humans and machines. Sistemi intelligenti, 35(2):321– 328. [find](https://scholar.google.com/scholar?q=Andrea+Moro%2E+2023%2E+Embodied+syntax%3A+impossible+languages+and+the+irreducible+difference+between+humans+and+machines%2E+Sistemi+intelligenti%2C+35%282%29%3A321%E2%80%93+328%2E)
59. Andrea Moro, Matteo Greco, and Stefano F Cappa. 2023. Large languages, impossible languages and human brains. Cortex, 167:82–85. [find](https://scholar.google.com/scholar?q=Andrea+Moro%2C+Matteo+Greco%2C+and+Stefano+F+Cappa%2E+2023%2E+Large+languages%2C+impossible+languages+and+human+brains%2E+Cortex%2C+167%3A82%E2%80%9385%2E)
60. Frederick J Newmeyer. 2005. Possible and probable languages. Oxford: Oxford. [find](https://scholar.google.com/scholar?q=Frederick+J+Newmeyer%2E+2005%2E+Possible+and+probable+languages%2E+Oxford%3A+Oxford%2E)
61. Frederick J Newmeyer. 2021. Complexity and relative complexity in generative grammar. Frontiers in Communication, 6:614352. [find](https://scholar.google.com/scholar?q=Frederick+J+Newmeyer%2E+2021%2E+Complexity+and+relative+complexity+in+generative+grammar%2E+Frontiers+in+Communication%2C+6%3A614352%2E)
62. Byung-Doh Oh and William Schuler. 2023a. Transformer-based language model surprisal predicts human reading times best with about two billion training tokens. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1915–1921, Singapore. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Byung%2DDoh+Oh+and+William+Schuler%2E+2023a%2E+Transformer%2Dbased+language+model+surprisal+predicts+human+reading+times+best+with+about+two+billion+training+tokens%2E+In+Findings+of+the+Association+for+Computa)
63. Byung-Doh Oh and William Schuler. 2023b. [Why does surprisal from larger transformer-based language models provide a poorer fit to human reading times?](https://md.research.elamur.ai/free/en/papers/oh-2023-why-does-surprisal-larger-transformer/) Transactions of the Association for Computational Linguistics, 11:336–350.
64. Malte Ostendorff. 2023. Gpt2-xl-wechsel-german. [https://huggingface.co/malteos/](https://huggingface.co/malteos/) gpt2-xl-wechsel-german. [↗](https://huggingface.co/malteos/)
65. Joe Pater. 2019. Generative linguistics and neural networks at 60: Foundation, friction, and fusion. Language, 95(1):e41–e74. [find](https://scholar.google.com/scholar?q=Joe+Pater%2E+2019%2E+Generative+linguistics+and+neural+networks+at+60%3A+Foundation%2C+friction%2C+and+fusion%2E+Language%2C+95%281%29%3Ae41%E2%80%93e74%2E)
66. Lisa Pearl and Benjamin Mis. 2011. How far can indirect evidence take us? Anaphoric one revisited. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 33. [find](https://scholar.google.com/scholar?q=Lisa+Pearl+and+Benjamin+Mis%2E+2011%2E+How+far+can+indirect+evidence+take+us%3F+Anaphoric+one+revisited%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Cognitive+Science+Society%2C+volume+33%2E)
67. Steven T Piantadosi. 2023. Modern language models refute Chomsky’s approach to language. From fieldwork to linguistic theory: A tribute to Dan Everett, pages 353–414. [find](https://scholar.google.com/scholar?q=Steven+T+Piantadosi%2E+2023%2E+Modern+language+models+refute+Chomsky%E2%80%99s+approach+to+language%2E+From+fieldwork+to+linguistic+theory%3A+A+tribute+to+Dan+Everett%2C+pages+353%E2%80%93414%2E)
68. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. [Stanza: A Python natural language processing toolkit for many human languages](https://md.research.elamur.ai/free/en/papers/qi-2020-stanza-python-natural-language-processing/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101–108, Online.…
69. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
70. Nils Reimers and Iryna Gurevych. 2020. [Making monolingual sentence embeddings multilingual using knowledge distillation](https://md.research.elamur.ai/free/en/papers/reimers-2020-making-monolingual-sentence-embeddings-multilingual/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512–4525, Online. Association for Computational Linguistics.
71. Jenny Saffran, Marc Hauser, Rebecca Seibel, Joshua Kapfhamer, Fritz Tsao, and Fiery Cushman. 2008. Grammatical pattern learning by human infants and cotton-top tamarin monkeys. Cognition, 107(2):479– 500. [find](https://scholar.google.com/scholar?q=Jenny+Saffran%2C+Marc+Hauser%2C+Rebecca+Seibel%2C+Joshua+Kapfhamer%2C+Fritz+Tsao%2C+and+Fiery+Cushman%2E+2008%2E+Grammatical+pattern+learning+by+human+infants+and+cotton%2Dtop+tamarin+monkeys%2E+Cognition%2C+107%282%29%3A479%E2%80%93+)
72. Craig W Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner. 2024. [Tokenization is more than compression](https://md.research.elamur.ai/free/en/papers/schmidt-2024-tokenization-is-more-than-compression/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 678–702, Miami, Florida, USA. Association for Compu…
73. Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 2021. [CCMatrix: Mining billions of high-quality parallel sentences on the web](https://md.research.elamur.ai/free/en/papers/schwenk-2019-ccmatrix-mining-billions-high-quality/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Co…
74. Jenny L Singleton and Elissa L Newport. 2004. When learners surpass their models: The acquisition of American sign language from inconsistent input. Cognitive psychology, 49(4):370–407. [find](https://scholar.google.com/scholar?q=Jenny+L+Singleton+and+Elissa+L+Newport%2E+2004%2E+When+learners+surpass+their+models%3A+The+acquisition+of+American+sign+language+from+inconsistent+input%2E+Cognitive+psychology%2C+49%284%29%3A370%E2%80%93407%2E)
75. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=J%C3%B6rg+Tiedemann%2E+2012%2E+Parallel+data%2C+tools+and+interfaces+in+OPUS%2E+In+Proceedings+of+the+Eighth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9912%29%2C+pages+2214%E2%80%932218%2C+Istanbul%2C+Turk)
76. Ianthi-Maria Tsimpli and Neil Smith. 1995. Mind of a Savant: Language, Learning and Modularity. Blackwell. [find](https://scholar.google.com/scholar?q=Ianthi%2DMaria+Tsimpli+and+Neil+Smith%2E+1995%2E+Mind+of+a+Savant%3A+Language%2C+Learning+and+Modularity%2E+Blackwell%2E)
77. Eleftheria Tsipidi, Franz Nowak, Ryan Cotterell, Ethan Wilcox, Mario Giulianelli, and Alex Warstadt. 2024. Surprise! Uniform Information Density isn’t the whole story: Predicting surprisal contours in long-form discourse. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language… [find](https://scholar.google.com/scholar?q=Eleftheria+Tsipidi%2C+Franz+Nowak%2C+Ryan+Cotterell%2C+Ethan+Wilcox%2C+Mario+Giulianelli%2C+and+Alex+Warstadt%2E+2024%2E+Surprise%21+Uniform+Information+Density+isn%E2%80%99t+the+whole+story%3A+Predicting+surprisal+contours+in)
78. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. 2023. Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of the… [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Aaron+Mueller%2C+Leshem+Choshen%2C+Ethan+Wilcox%2C+Chengxu+Zhuang%2C+Juan+Ciro%2C+Rafael+Mosquera%2C+Bhargavi+Paranjabe%2C+Adina+Williams%2C+Tal+Linzen%2C+and+Ryan+Cotterell%2E+2023%2E+Findings+of+the+BabyLM)
79. Ethan G. Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P. Levy. 2023. [Testing the predictions of surprisal theory in 11 languages](https://md.research.elamur.ai/free/en/papers/wilcox-2023-testing-predictions-surprisal-theory-11/). Transactions of the Association for Computational Linguistics, 11:1451–1470.
80. Michał Wojczulis and Dariusz Kłeczek. 2021. papu-GaPT2 - Polish GPT2 language model. [find](https://scholar.google.com/scholar?q=Micha%C5%82+Wojczulis+and+Dariusz+K%C5%82eczek%2E+2021%2E+papu%2DGaPT2+%2D+Polish+GPT2+language+model%2E)
81. Tianyang Xu, Tatsuki Kuribayashi, Yohei Oseki, Ryan Cotterell, and Alex Warstadt. 2025. [Can language models learn typologically implausible languages?](https://md.research.elamur.ai/free/en/papers/xu-2026-can-language-models-learn-typologically/) arXiv preprint arXiv:2502.12317.
82. Naiwen Xue, Fei Xia, Fu-Dong Chiou, and Marta Palmer. 2005. The Penn Chinese treebank: Phrase structure annotation of a large corpus. Natural language engineering, 11(2):207–238. [find](https://scholar.google.com/scholar?q=Naiwen+Xue%2C+Fei+Xia%2C+Fu%2DDong+Chiou%2C+and+Marta+Palmer%2E+2005%2E+The+Penn+Chinese+treebank%3A+Phrase+structure+annotation+of+a+large+corpus%2E+Natural+language+engineering%2C+11%282%29%3A207%E2%80%93238%2E)
83. Yuekun Yao and Alexander Koller. 2022. Structural generalization is hard for sequence-to-sequence models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5048–5062, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yuekun+Yao+and+Alexander+Koller%2E+2022%2E+Structural+generalization+is+hard+for+sequence%2Dto%2Dsequence+models%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pag)
84. Dmitry Zmitrovich, Aleksandr Abramov, Andrey Kalmykov, Vitaly Kadulin, Maria Tikhonova, Ekaterina Taktasheva, Danil Astafurov, Mark Baushenko, Artem Snegirev, Tatiana Shavrina, Sergei S. Markov, Vladislav Mikhailov, and Alena Fenogenova. 2024. [A family of pretrained transformer language models for R…](https://md.research.elamur.ai/free/en/papers/zmitrovich-2024-family-pretrained-transformer-language-models/)
85. Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. 2023. Tokenization and the noiseless channel. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5184–5207, Toronto, Canada. Association for… [find](https://scholar.google.com/scholar?q=Vil%C3%A9m+Zouhar%2C+Clara+Meister%2C+Juan+Gastaldi%2C+Li+Du%2C+Mrinmaya+Sachan%2C+and+Ryan+Cotterell%2E+2023%2E+Tokenization+and+the+noiseless+channel%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+C)
86. A Experiment Results of Replicating Kallini et al. (2024) [find](https://scholar.google.com/scholar?q=A+Experiment+Results+of+Replicating+Kallini+et+al%2E+%282024%29)

*Updated 31 October 2026*