---
title: "Vocabulary Shapes Cross-Lingual Variation of Word-Order"
description: "Using ten European languages drawn from a multilingual parallel corpus, the authors construct a continuous spectrum of… 84 references."
authors: ["Jonas Mayer Martins", "Jaap Jumelet", "Viola Priesemann", "Lisa Beinborn"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1510/"
doi: "10.18653/v1/2026.acl-long.1510"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/martins-2026-vocabulary-shapes-cross-lingual-variation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/martins-2026-vocabulary-shapes-cross-lingual-variation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/47/) / Vocabulary Shapes Cross-Lingual Variation of Word-Order Learnability in Language Models

# Vocabulary Shapes Cross-Lingual Variation of Word-Order Learnability in Language Models

**Authors:** [Jonas Mayer Martins](https://md.research.elamur.ai/free/en/authors/jonas-mayer-martins/), [Jaap Jumelet](https://md.research.elamur.ai/free/en/authors/jaap-jumelet/), Viola Priesemann and [Lisa Beinborn](https://md.research.elamur.ai/free/en/authors/lisa-beinborn/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1510/) · [PDF](https://aclanthology.org/2026.acl-long.1510.pdf) · [DOI 10.18653/v1/2026.acl-long.1510](https://doi.org/10.18653/v1/2026.acl-long.1510) · [OpenAlex](https://openalex.org/W7166837659)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper asks why languages such as Czech tolerate flexible word order while languages like English do not, and whether this affects how easily a language model can learn them. Using ten European languages drawn from a multilingual parallel corpus, the authors construct a continuous spectrum of artificially reordered versions of each language. They shuffle whole words, not subwords, using fixed permutations per sentence length drawn from the Mallows permutation model. A single parameter θ moves the order from intact, through local and full scrambling, to complete reversal. Because whole words are moved, vocabulary, morphology and global text entropy stay the same, and only word-order regularity changes. Transformer language models trained on these variants show higher surprisal as order becomes more irregular. Reversing sentences has only a small effect. The usual split into free- and fixed-order languages does not account for the differences between languages. Vocabulary statistics do: Zipf-based coverage, sentence length and simple morphology proxies. The study suggests that vocabulary structure must be considered when interpreting surprisal in shuffling experiments.

## Key points
- Word order is controlled by the Mallows order parameter θ, with these settings:
  - large positive values keep the original order;
  - θ = 0 makes all permutations equally likely;
  - strongly negative values approach full sentence reversal.
- Data: ten languages selected from a high-quality parallel corpus covering 21 European languages, including Czech and Finnish (free order) and English and French (fixed order).
- Scale: 1400 models were trained for the θ scan and 600 for the vocabulary-size scan, on a single A100 GPU with 40 GB, totalling about 150 kcore-hours.
- Morphological-complexity proxies matter most for explaining robustness to strongly scrambled word order. Larger vocabularies reduce the surprisal penalty for some languages.
- Stated limitations:
  - only European languages are covered;
  - model size is small;
  - free/fixed typology is treated as binary;
  - only global test-set surprisal is evaluated;
  - there is no comparison with human learners.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Groningen](https://md.research.elamur.ai/free/en/universities/university-of-groningen/) — University, Groningen, Netherlands · 70 works · 5,142 citations
- [University of Göttingen](https://md.research.elamur.ai/free/en/universities/university-of-gottingen/) — University, Göttingen, Germany · 17 works · 158 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works

## Cite

Cite as: Jonas Mayer Martins, Jaap Jumelet, Viola Priesemann and Lisa Beinborn. 2026. Vocabulary Shapes Cross-Lingual Variation of Word-Order Learnability in Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{martins2026vocabulary,
  title = {{Vocabulary Shapes Cross-Lingual Variation of Word-Order Learnability in Language Models}},
  author = {Jonas Mayer Martins, Jaap Jumelet, Viola Priesemann and Lisa Beinborn},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1510},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1510}
}
```

- [← previous VLCE: A Knowledge-Enhanced Framework for Image Description in Disaster Assessment](https://md.research.elamur.ai/free/en/papers/rahman-2026-vlce-knowledge-enhanced-framework-image/)
- [next → VocalRep: Structure-Aware Vocal Representations for Multimodal Generation](https://md.research.elamur.ai/free/en/papers/shen-2026-vocalrep-structure-aware-vocal-representations/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/47/)

## References

84 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bas Aarts. 2011. Oxford Modern English Grammar, 1st edition. Oxford University Press, New York. [find](https://scholar.google.com/scholar?q=Bas+Aarts%2E+2011%2E+Oxford+Modern+English+Grammar%2C+1st+edition%2E+Oxford+University+Press%2C+New+York%2E)
2. Mostafa Abdou, Vinit Ravishankar, Artur Kulmizev, and Anders Søgaard. 2022. Word order does matter and shuffled language models know it. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6907–6919, Dublin, Ireland. Association f… [find](https://scholar.google.com/scholar?q=Mostafa+Abdou%2C+Vinit+Ravishankar%2C+Artur+Kulmizev%2C+and+Anders+S%C3%B8gaard%2E+2022%2E+Word+order+does+matter+and+shuffled+language+models+know+it%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+fo)
3. Catherine Arnett and Benjamin K. Bergen. 2025. [Why do language models perform worse for morphologically complex languages?](https://md.research.elamur.ai/free/en/papers/arnett-2025-why-do-language-models-perform/) In Proceedings of the 31st International Conference on Computational Linguistics, pages 6607–6623, Abu Dhabi, UAE. Association for Computational Linguistics.
4. Emi Baylor, Esther Ploeger, and Johannes Bjerva. 2024. [Multilingual gradient word-order typology from universal dependencies](https://md.research.elamur.ai/free/en/papers/baylor-2024-multilingual-gradient-word-order-typology/). In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 42–49, St. Julian’s, Malta. Ass…
5. Lisa Beinborn and Yuval Pinter. 2023. Analyzing cognitive plausibility of subword tokenization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4478–4486, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Lisa+Beinborn+and+Yuval+Pinter%2E+2023%2E+Analyzing+cognitive+plausibility+of+subword+tokenization%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+4478%E2%80%9344)
6. Arianna Bisazza, Ahmet Üstün, and Stephan Sportel. 2021. [On the difficulty of translating free-order case-marking languages](https://md.research.elamur.ai/free/en/papers/bisazza-2021-difficulty-translating-free-order-case/). In Transactions of the Association for Computational Linguistics, volume 9, pages 1233–1248, Cambridge. MIT Press.
7. Noam Chomsky. 2023. Noam Chomsky: The false promise of ChatGPT. The New York Times. [find](https://scholar.google.com/scholar?q=Noam+Chomsky%2E+2023%2E+Noam+Chomsky%3A+The+false+promise+of+ChatGPT%2E+The+New+York+Times%2E)
8. Leshem Choshen and Omri Abend. 2019. Automatically extracting challenge sets for non-local phenomena in neural machine translation. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 291–303, Hong Kong, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Leshem+Choshen+and+Omri+Abend%2E+2019%2E+Automatically+extracting+challenge+sets+for+non%2Dlocal+phenomena+in+neural+machine+translation%2E+In+Proceedings+of+the+23rd+Conference+on+Computational+Natural+Langu)
9. Thomas Hikaru Clark, Clara Meister, Tiago Pimentel, Michael Hahn, Ryan Cotterell, Richard Futrell, and Roger Levy. 2023. [A cross-linguistic pressure for uniform information density in word order](https://md.research.elamur.ai/free/en/papers/clark-2023-cross-linguistic-pressure-uniform-information/). In Transactions of the Association for Computational Linguistics, volume 11, pages 1048–1065, Cambridge.
10. Ryan Cotterell, Sebastian J. Mielke, Jason Eisner, and Brian Roark. 2018. [Are all languages equally hard to language-model?](https://md.research.elamur.ai/free/en/papers/cotterell-2018-are-all-languages-equally-hard/) In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), page…
11. Marta Crispino, Cristina Mollica, Valerio Astuti, and Luca Tardella. 2023. Efficient and accurate inference for mixtures of Mallows models with Spearman distance. Statistics and Computing, 33(5):98. [find](https://scholar.google.com/scholar?q=Marta+Crispino%2C+Cristina+Mollica%2C+Valerio+Astuti%2C+and+Luca+Tardella%2E+2023%2E+Efficient+and+accurate+inference+for+mixtures+of+Mallows+models+with+Spearman+distance%2E+Statistics+and+Computing%2C+33%285%29%3A98%2E)
12. William Croft. 2002. Typology and Universals, 2nd edition. Cambridge University Press, Cambridge. [find](https://scholar.google.com/scholar?q=William+Croft%2E+2002%2E+Typology+and+Universals%2C+2nd+edition%2E+Cambridge+University+Press%2C+Cambridge%2E)
13. Marion Di Marco and Alexander Fraser. 2024. Subword segmentation in LLMs: Looking at inflection and consistency. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12050–12060, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Marion+Di+Marco+and+Alexander+Fraser%2E+2024%2E+Subword+segmentation+in+LLMs%3A+Looking+at+inflection+and+consistency%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Language+Processi)
14. Richard Diehl Martinez, David Demitri Africa, Yuval Weiss, Suchir Salhan, Ryan Daniels, and Paula Buttery. 2025. Pico: A modular framework for hypothesis-driven small language model research. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrat… [find](https://scholar.google.com/scholar?q=Richard+Diehl+Martinez%2C+David+Demitri+Africa%2C+Yuval+Weiss%2C+Suchir+Salhan%2C+Ryan+Daniels%2C+and+Paula+Buttery%2E+2025%2E+Pico%3A+A+modular+framework+for+hypothesis%2Ddriven+small+language+model+research%2E+In+Proce)
15. Michael Hahn and Yang Xu. 2022. Crosslinguistic word order variation reflects evolutionary pressures of dependency and information locality. Proceedings of the National Academy of Sciences, 119(24):e2122604119. [find](https://scholar.google.com/scholar?q=Michael+Hahn+and+Yang+Xu%2E+2022%2E+Crosslinguistic+word+order+variation+reflects+evolutionary+pressures+of+dependency+and+information+locality%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+119%2824%29%3Ae2)
16. Matthew Dryer and Martin Haspelmath. 2024. The World Atlas of Language Structures online - Order of subject, object and verb. [find](https://scholar.google.com/scholar?q=Matthew+Dryer+and+Martin+Haspelmath%2E+2024%2E+The+World+Atlas+of+Language+Structures+online+%2D+Order+of+subject%2C+object+and+verb%2E)
17. John Hale. 2001. [A probabilistic Earley parser as a psycholinguistic model](https://md.research.elamur.ai/free/en/papers/hale-2001-probabilistic-earley-parser-as-psycholinguistic/). In Second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies 2001 - NAACL ’01, pages 1–8, Pittsburgh, Pennsylvania. Association for Computational Linguistics.
18. Nadine El-Naggar, Tatsuki Kuribayashi, and Ted Briscoe. 2025a. GCG-based artificial languages for evaluating inductive biases of neural language models. In Proceedings of the 29th Conference on Computational Natural Language Learning, pages 540– 556, Vienna, Austria. Association for Computational Li… [find](https://scholar.google.com/scholar?q=Nadine+El%2DNaggar%2C+Tatsuki+Kuribayashi%2C+and+Ted+Briscoe%2E+2025a%2E+GCG%2Dbased+artificial+languages+for+evaluating+inductive+biases+of+neural+language+models%2E+In+Proceedings+of+the+29th+Conference+on+Comput)
19. Nadine El-Naggar, Tatsuki Kuribayashi, and Ted Briscoe. 2025b. Which word orders facilitate length generalization in LMs? An investigation with GCG-based artificial languages. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 35587–35601, Suzhou, China.… [find](https://scholar.google.com/scholar?q=Nadine+El%2DNaggar%2C+Tatsuki+Kuribayashi%2C+and+Ted+Briscoe%2E+2025b%2E+Which+word+orders+facilitate+length+generalization+in+LMs%3F+An+investigation+with+GCG%2Dbased+artificial+languages%2E+In+Proceedings+of+the+20)
20. William Feller. 1968. An Introduction to Probability Theory and its Applications I, 3rd edition. Wiley series in probability and mathematical statistics. Wiley, New York. [find](https://scholar.google.com/scholar?q=William+Feller%2E+1968%2E+An+Introduction+to+Probability+Theory+and+its+Applications+I%2C+3rd+edition%2E+Wiley+series+in+probability+and+mathematical+statistics%2E+Wiley%2C+New+York%2E)
21. Michael A. Fligner and Joseph S. Verducci. 1986. Distance based ranking models. Journal of the Royal Statistical Society: Series B (Methodological), 48(3):359–369. [find](https://scholar.google.com/scholar?q=Michael+A%2E+Fligner+and+Joseph+S%2E+Verducci%2E+1986%2E+Distance+based+ranking+models%2E+Journal+of+the+Royal+Statistical+Society%3A+Series+B+%28Methodological%29%2C+48%283%29%3A359%E2%80%93369%2E)
22. Víctor Franco-Sánchez, Arnau Martí-Llobet, and Ramon Ferrer-i-Cancho. 2024. Swap distance minimization beyond entropy minimization in word order variation. Preprint, arXiv:2404.14192. — [arXiv:2404.14192](https://arxiv.org/abs/2404.14192) [find](https://scholar.google.com/scholar?q=V%C3%ADctor+Franco%2DS%C3%A1nchez%2C+Arnau+Mart%C3%AD%2DLlobet%2C+and+Ramon+Ferrer%2Di%2DCancho%2E+2024%2E+Swap+distance+minimization+beyond+entropy+minimization+in+word+order+variation%2E+Preprint%2C+arXiv%3A2404%2E14192%2E+%E2%80%94+%5BarXiv%3A2404%2E14)
23. Richard Futrell, Edward Gibson, and Roger P. Levy. 2020. Lossy-context surprisal: An information-theoretic model of memory effects in sentence processing. Cognitive Science, 44(3):e12814. [find](https://scholar.google.com/scholar?q=Richard+Futrell%2C+Edward+Gibson%2C+and+Roger+P%2E+Levy%2E+2020%2E+Lossy%2Dcontext+surprisal%3A+An+information%2Dtheoretic+model+of+memory+effects+in+sentence+processing%2E+Cognitive+Science%2C+44%283%29%3Ae12814%2E)
24. Richard Futrell and Kyle Mahowald. 2025. How Linguistics Learned to Stop Worrying and Love the Language Models. Behavioral and Brain Sciences, pages 1–98. [find](https://scholar.google.com/scholar?q=Richard+Futrell+and+Kyle+Mahowald%2E+2025%2E+How+Linguistics+Learned+to+Stop+Worrying+and+Love+the+Language+Models%2E+Behavioral+and+Brain+Sciences%2C+pages+1%E2%80%9398%2E)
25. Joseph H. Greenberg. 1990. Some universals of grammar with particular reference to the order of meaningful elements. In Keith Denning and Suzanne Kemmer, editors, On Language: Selected Writings of Joseph H. Greenberg, pages 40–70. Stanford University Press. [find](https://scholar.google.com/scholar?q=Joseph+H%2E+Greenberg%2E+1990%2E+Some+universals+of+grammar+with+particular+reference+to+the+order+of+meaningful+elements%2E+In+Keith+Denning+and+Suzanne+Kemmer%2C+editors%2C+On+Language%3A+Selected+Writings+of+Jos)
26. Coleman Haley and Colin Wilson. 2021. Deep neural networks easily learn unnatural infixation and reduplication patterns. In Proceedings of the Society for Computation in Linguistics 2021, pages 427–433, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Coleman+Haley+and+Colin+Wilson%2E+2021%2E+Deep+neural+networks+easily+learn+unnatural+infixation+and+reduplication+patterns%2E+In+Proceedings+of+the+Society+for+Computation+in+Linguistics+2021%2C+pages+427%E2%80%9343)
27. Harald Hammarström, Robert Forkel, Martin Haspelmath, and Sebastian Bank. 2025. Glottolog 5.2. [find](https://scholar.google.com/scholar?q=Harald+Hammarstr%C3%B6m%2C+Robert+Forkel%2C+Martin+Haspelmath%2C+and+Sebastian+Bank%2E+2025%2E+Glottolog+5%2E2%2E)
28. Robert T. Harms. 1997. Estonian Grammar, 1st edition. Taylor and Francis, London. [find](https://scholar.google.com/scholar?q=Robert+T%2E+Harms%2E+1997%2E+Estonian+Grammar%2C+1st+edition%2E+Taylor+and+Francis%2C+London%2E)
29. Martin Harris and Nigel Vincent. 2012. Romance Languages, 1st edition. Routledge Language Family Series. Taylor and Francis, Hoboken. [find](https://scholar.google.com/scholar?q=Martin+Harris+and+Nigel+Vincent%2E+2012%2E+Romance+Languages%2C+1st+edition%2E+Routledge+Language+Family+Series%2E+Taylor+and+Francis%2C+Hoboken%2E)
30. Martin Haspelmath. 2023. Defining the word. WORD, 69(3):283–297. [find](https://scholar.google.com/scholar?q=Martin+Haspelmath%2E+2023%2E+Defining+the+word%2E+WORD%2C+69%283%29%3A283%E2%80%93297%2E)
31. John A. Hawkins. 2014. Cross-Linguistic Variation and Efficiency, 1st edition. Oxford University Press, Oxford. [find](https://scholar.google.com/scholar?q=John+A%2E+Hawkins%2E+2014%2E+Cross%2DLinguistic+Variation+and+Efficiency%2C+1st+edition%2E+Oxford+University+Press%2C+Oxford%2E)
32. Jack Hessel and Alexandra Schofield. 2021. How effective is BERT without word ordering? Implications for language understanding and data privacy. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language… [find](https://scholar.google.com/scholar?q=Jack+Hessel+and+Alexandra+Schofield%2E+2021%2E+How+effective+is+BERT+without+word+ordering%3F+Implications+for+language+understanding+and+data+privacy%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Assoc)
33. Philip Holmes and Ian Hinchliffe. 2013. Swedish: A Comprehensive Grammar, 3rd edition. Routledge, London. [find](https://scholar.google.com/scholar?q=Philip+Holmes+and+Ian+Hinchliffe%2E+2013%2E+Swedish%3A+A+Comprehensive+Grammar%2C+3rd+edition%2E+Routledge%2C+London%2E)
34. Johannes Kabatek. 2022. Manual of Brazilian Portuguese Linguistics, 1st edition. Number 21 in Manuals of Romance Linguistics. Walter de Gruyter GmbH, Berlin/Boston. [find](https://scholar.google.com/scholar?q=Johannes+Kabatek%2E+2022%2E+Manual+of+Brazilian+Portuguese+Linguistics%2C+1st+edition%2E+Number+21+in+Manuals+of+Romance+Linguistics%2E+Walter+de+Gruyter+GmbH%2C+Berlin%2FBoston%2E)
35. Ximena Gutierrez-Vasques, Christian Bentz, and Tanja Samardžić. 2023. [Languages through the looking glass of BPE compression](https://md.research.elamur.ai/free/en/papers/vasques-2023-languages-through-looking-glass-bpe/). Computational Linguistics, 49(4):943–1001.
36. Julie Kallini, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, and Christopher Potts. 2024. [Mission: Impossible language models](https://md.research.elamur.ai/free/en/papers/kallini-2024-mission-impossible-language-models/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14691–14714, Bangkok, Thailand. Association…
37. Michael Hahn, Dan Jurafsky, and Richard Futrell. 2020. Universals of word order reflect optimization of grammars for efficient communication. Proceedings of the National Academy of Sciences, 117(5):2347–2353. [find](https://scholar.google.com/scholar?q=Michael+Hahn%2C+Dan+Jurafsky%2C+and+Richard+Futrell%2E+2020%2E+Universals+of+word+order+reflect+optimization+of+grammars+for+efficient+communication%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+117%285%29%3A23)
38. Julie Kallini and Christopher Potts. 2025. Language models as tools for investigating the distinction between possible and impossible natural languages. Preprint, arXiv:2512.09394. [find](https://scholar.google.com/scholar?q=Julie+Kallini+and+Christopher+Potts%2E+2025%2E+Language+models+as+tools+for+investigating+the+distinction+between+possible+and+impossible+natural+languages%2E+Preprint%2C+arXiv%3A2512%2E09394%2E)
39. Fred Karlsson. 2017. Finnish: A Comprehensive Grammar, 1st edition. Routledge, Abingdon, Oxon. [find](https://scholar.google.com/scholar?q=Fred+Karlsson%2E+2017%2E+Finnish%3A+A+Comprehensive+Grammar%2C+1st+edition%2E+Routledge%2C+Abingdon%2C+Oxon%2E)
40. Roni Katzir. 2023. Why large language models are poor theories of human linguistic cognition: A reply to Piantadosi. Biolinguistics, 17:e13153. [find](https://scholar.google.com/scholar?q=Roni+Katzir%2E+2023%2E+Why+large+language+models+are+poor+theories+of+human+linguistic+cognition%3A+A+reply+to+Piantadosi%2E+Biolinguistics%2C+17%3Ae13153%2E)
41. Maurice G. Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81. [find](https://scholar.google.com/scholar?q=Maurice+G%2E+Kendall%2E+1938%2E+A+new+measure+of+rank+correlation%2E+Biometrika%2C+30%281%2F2%29%3A81%2E)
42. Istvan Kenesei, Robert M. Vago, and Anna Fenyvesi. 2002. Hungarian, 1st edition. Routledge, London. [find](https://scholar.google.com/scholar?q=Istvan+Kenesei%2C+Robert+M%2E+Vago%2C+and+Anna+Fenyvesi%2E+2002%2E+Hungarian%2C+1st+edition%2E+Routledge%2C+London%2E)
43. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79–86, Phuket, Thailand. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2E+2005%2E+Europarl%3A+A+parallel+corpus+for+statistical+machine+translation%2E+In+Proceedings+of+Machine+Translation+Summit+X%3A+Papers%2C+pages+79%E2%80%9386%2C+Phuket%2C+Thailand%2E)
44. Alexander Koplenig, Sascha Wolfer, and Peter Meyer. 2023. A large quantitative analysis of written language challenges the idea that all languages are equally complex. Scientific Reports, 13(1):15351. [find](https://scholar.google.com/scholar?q=Alexander+Koplenig%2C+Sascha+Wolfer%2C+and+Peter+Meyer%2E+2023%2E+A+large+quantitative+analysis+of+written+language+challenges+the+idea+that+all+languages+are+equally+complex%2E+Scientific+Reports%2C+13%281%29%3A15351%2E)
45. Evelina Leivada, Raquel Montero, Paolo Morosi, Natalia Moskvina, Tamara Serrano, Marcel Aguilar, and Fritz Guenther. 2025. Large language model probabilities cannot distinguish between possible and impossible language. Preprint, arXiv:2509.15114. — [arXiv:2509.15114](https://arxiv.org/abs/2509.15114) [find](https://scholar.google.com/scholar?q=Evelina+Leivada%2C+Raquel+Montero%2C+Paolo+Morosi%2C+Natalia+Moskvina%2C+Tamara+Serrano%2C+Marcel+Aguilar%2C+and+Fritz+Guenther%2E+2025%2E+Large+language+model+probabilities+cannot+distinguish+between+possible+and+im)
46. Natalia Levshina, Savithry Namboodiripad, Marc Allassonnière-Tang, Mathew Kramer, Luigi Talamo, Annemarie Verkerk, Sasha Wilmoth, Gabriela Garrido Rodriguez, Timothy Michael Gupton, Evan Kidd, Zoey Liu, Chiara Naccarato, Rachel Nordlinger, Anastasia Panova, and Natalia Stoynova. 2023. Why we need a… [find](https://scholar.google.com/scholar?q=Natalia+Levshina%2C+Savithry+Namboodiripad%2C+Marc+Allassonni%C3%A8re%2DTang%2C+Mathew+Kramer%2C+Luigi+Talamo%2C+Annemarie+Verkerk%2C+Sasha+Wilmoth%2C+Gabriela+Garrido+Rodriguez%2C+Timothy+Michael+Gupton%2C+Evan+Kidd%2C+Zoey+Li)
47. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. [find](https://scholar.google.com/scholar?q=Roger+Levy%2E+2008%2E+Expectation%2Dbased+syntactic+comprehension%2E+Cognition%2C+106%283%29%3A1126%E2%80%931177%2E)
48. Siqi Liu, Jianwei Yan, and Haitao Liu. 2025. The complexity trade-off between morphological richness and word order freedom in Romance languages: A quantitative perspective. Zeitschrift für romanische Philologie, 141(2):323–349. [find](https://scholar.google.com/scholar?q=Siqi+Liu%2C+Jianwei+Yan%2C+and+Haitao+Liu%2E+2025%2E+The+complexity+trade%2Doff+between+morphological+richness+and+word+order+freedom+in+Romance+languages%3A+A+quantitative+perspective%2E+Zeitschrift+f%C3%BCr+romanische)
49. Tom Lundskaer-Nielsen and Philip Holmes. 2015. Danish: A Comprehensive Grammar, 2nd edition. Routledge, London. [find](https://scholar.google.com/scholar?q=Tom+Lundskaer%2DNielsen+and+Philip+Holmes%2E+2015%2E+Danish%3A+A+Comprehensive+Grammar%2C+2nd+edition%2E+Routledge%2C+London%2E)
50. David J. C. MacKay. 2019. Information Theory, Inference, and Learning Algorithms, 22nd edition. Cambridge University Press, Cambridge. [find](https://scholar.google.com/scholar?q=David+J%2E+C%2E+MacKay%2E+2019%2E+Information+Theory%2C+Inference%2C+and+Learning+Algorithms%2C+22nd+edition%2E+Cambridge+University+Press%2C+Cambridge%2E)
51. James D. Naughton. 2008. Czech: An Essential Grammar, 2nd edition. Essential grammars. Routledge, London. [find](https://scholar.google.com/scholar?q=James+D%2E+Naughton%2E+2008%2E+Czech%3A+An+Essential+Grammar%2C+2nd+edition%2E+Essential+grammars%2E+Routledge%2C+London%2E)
52. Julie Nijs, Freek Van de Velde, and Hubert Cuyckens. 2025. Is word order responsive to morphology? Disentangling cause and effect in morphosyntactic change in five Western European languages. Entropy, 27(1):53. [find](https://scholar.google.com/scholar?q=Julie+Nijs%2C+Freek+Van+de+Velde%2C+and+Hubert+Cuyckens%2E+2025%2E+Is+word+order+responsive+to+morphology%3F+Disentangling+cause+and+effect+in+morphosyntactic+change+in+five+Western+European+languages%2E+Entropy%2C)
53. Isabel Papadimitriou, Richard Futrell, and Kyle Mahowald. 2022. When classifying grammatical role, BERT doesn’t care about word order... except when it matters. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 636–643, Dublin,… [find](https://scholar.google.com/scholar?q=Isabel+Papadimitriou%2C+Richard+Futrell%2C+and+Kyle+Mahowald%2E+2022%2E+When+classifying+grammatical+role%2C+BERT+doesn%E2%80%99t+care+about+word+order%2E%2E%2E+except+when+it+matters%2E+In+Proceedings+of+the+60th+Annual+Meeti)
54. Isabel Papadimitriou and Jacob Prince. 2025. Vocabulary embeddings organize linguistic structure early in language model training. Preprint, arXiv:2510.07613. — [arXiv:2510.07613](https://arxiv.org/abs/2510.07613) [find](https://scholar.google.com/scholar?q=Isabel+Papadimitriou+and+Jacob+Prince%2E+2025%2E+Vocabulary+embeddings+organize+linguistic+structure+early+in+language+model+training%2E+Preprint%2C+arXiv%3A2510%2E07613%2E+%E2%80%94+%5BarXiv%3A2510%2E07613%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
55. Thang Pham, Trung Bui, Long Mai, and Anh Nguyen. 2021. [Out of order: How important is the sequential order of words in a sentence in Natural Language Understanding tasks?](https://md.research.elamur.ai/free/en/papers/pham-2021-out-order-how-important-is/) In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1145–1160, Online. Association for Computati…
56. Steven T. Piantadosi. 2014. Zipf’s word frequency law in natural language: A critical review and future directions. Psychonomic Bulletin & Review, 21(5):1112–1130. [find](https://scholar.google.com/scholar?q=Steven+T%2E+Piantadosi%2E+2014%2E+Zipf%E2%80%99s+word+frequency+law+in+natural+language%3A+A+critical+review+and+future+directions%2E+Psychonomic+Bulletin+%26+Review%2C+21%285%29%3A1112%E2%80%931130%2E)
57. Steven T. Piantadosi. 2024. Modern language models refute Chomsky’s approach to language. In From Fieldwork to Linguistic Theory, 1st edition, number 15 in Empirically Oriented Theoretical Morphology and Syntax, pages 353–414. Language Science Press, Berlin. [find](https://scholar.google.com/scholar?q=Steven+T%2E+Piantadosi%2E+2024%2E+Modern+language+models+refute+Chomsky%E2%80%99s+approach+to+language%2E+In+From+Fieldwork+to+Linguistic+Theory%2C+1st+edition%2C+number+15+in+Empirically+Oriented+Theoretical+Morphology+)
58. Steven T. Piantadosi, Harry Tily, and Edward Gibson. 2012. The communicative function of ambiguity in language. Cognition, 122(3):280–291. [find](https://scholar.google.com/scholar?q=Steven+T%2E+Piantadosi%2C+Harry+Tily%2C+and+Edward+Gibson%2E+2012%2E+The+communicative+function+of+ambiguity+in+language%2E+Cognition%2C+122%283%29%3A280%E2%80%93291%2E)
59. Esther Ploeger, Wessel Poelman, Miryam de Lhoneux, and Johannes Bjerva. 2024. What is “typological diversity” in NLP? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5681–5700, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Esther+Ploeger%2C+Wessel+Poelman%2C+Miryam+de+Lhoneux%2C+and+Johannes+Bjerva%2E+2024%2E+What+is+%E2%80%9Ctypological+diversity%E2%80%9D+in+NLP%3F+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Language+Pro)
60. Colin L. Mallows. 1957. Non-null ranking models. I. Biometrika, 44(1/2):114–130. [find](https://scholar.google.com/scholar?q=Colin+L%2E+Mallows%2E+1957%2E+Non%2Dnull+ranking+models%2E+I%2E+Biometrika%2C+44%281%2F2%29%3A114%E2%80%93130%2E)
61. Esther Ploeger, Wessel Poelman, Andreas Holck Høeg-Petersen, Anders Schlichtkrull, Miryam De Lhoneux, and Johannes Bjerva. 2025. [A principled framework for evaluating on typologically diverse languages](https://md.research.elamur.ai/free/en/papers/ploeger-2026-principled-framework-evaluating-typologically-diverse/). Computational Linguistics, pages 1–36.
62. Sebastian J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. 2019. [What kind of language is hard to language-model?](https://md.research.elamur.ai/free/en/papers/mielke-2019-what-kind-language-is-hard/) In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4975–4989, Florence, Italy. Association for Computational Linguist…
63. Wessel Poelman, Thomas Bauwens, and Miryam de Lhoneux. 2025. [Confounding factors in relating model performance to morphology](https://md.research.elamur.ai/free/en/papers/poelman-2025-confounding-factors-relating-model-performance/). In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 7273–7298, Suzhou, China. Association for Computational Linguistics.
64. Edoardo Maria Ponti, Helen O’Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. 2019. [Modeling language variation and universals: A survey on typological linguistics for natural language processing](https://md.research.elamur.ai/free/en/papers/ponti-2019-modeling-language-variation-universals-survey/). Computational Linguistics, 45(3):559–601.
65. Wenpin Tang. 2019. Mallows ranking models: Maximum likelihood estimate and regeneration. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6125–6134. Proceedings of Machine Learning Research. [find](https://scholar.google.com/scholar?q=Wenpin+Tang%2E+2019%2E+Mallows+ranking+models%3A+Maximum+likelihood+estimate+and+regeneration%2E+In+Proceedings+of+the+36th+International+Conference+on+Machine+Learning%2C+volume+97+of+Proceedings+of+Machine+Le)
66. Dace Praulinš. 2012. Latvian: An Essential Grammar, 1st edition. Essential grammars. Routledge, London. [find](https://scholar.google.com/scholar?q=Dace+Praulin%C5%A1%2E+2012%2E+Latvian%3A+An+Essential+Grammar%2C+1st+edition%2E+Essential+grammars%2E+Routledge%2C+London%2E)
67. Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. 2024. [Scaling laws with vocabulary: Larger models deserve larger vocabularies](https://md.research.elamur.ai/free/en/papers/tao-2024-scaling-laws-vocabulary-larger-models/). In Advances in Neural Information Processing System, volume 37, pages 114147–114179. Curran Associates, Inc.
68. Geoffrey Sampson. 2009. A linguistic axiom challenged. In Geoffrey Sampson, David Gil, and Peter Trudgill, editors, Language Complexity as an Evolving Variable, pages 1–18. Oxford University PressOxford. [find](https://scholar.google.com/scholar?q=Geoffrey+Sampson%2E+2009%2E+A+linguistic+axiom+challenged%2E+In+Geoffrey+Sampson%2C+David+Gil%2C+and+Peter+Trudgill%2C+editors%2C+Language+Complexity+as+an+Evolving+Variable%2C+pages+1%E2%80%9318%2E+Oxford+University+PressOxfo)
69. Geoffrey Sampson, David Gil, and Peter Trudgill, editors. 2009. Language Complexity as an Evolving Variable. Oxford University PressOxford, New York. [find](https://scholar.google.com/scholar?q=Geoffrey+Sampson%2C+David+Gil%2C+and+Peter+Trudgill%2C+editors%2E+2009%2E+Language+Complexity+as+an+Evolving+Variable%2E+Oxford+University+PressOxford%2C+New+York%2E)
70. Daniel J. Schad, Antje Nuthmann, and Ralf Engbert. 2010. Eye movements during reading of randomly shuffled text. Vision Research, 50(23):2600–2616. [find](https://scholar.google.com/scholar?q=Daniel+J%2E+Schad%2C+Antje+Nuthmann%2C+and+Ralf+Engbert%2E+2010%2E+Eye+movements+during+reading+of+randomly+shuffled+text%2E+Vision+Research%2C+50%2823%29%3A2600%E2%80%932616%2E)
71. Thomas Schürmann and Peter Grassberger. 1996. Entropy estimation of symbol sequences. Chaos: An Interdisciplinary Journal of Nonlinear Science, 6(3):414–427. [find](https://scholar.google.com/scholar?q=Thomas+Sch%C3%BCrmann+and+Peter+Grassberger%2E+1996%2E+Entropy+estimation+of+symbol+sequences%2E+Chaos%3A+An+Interdisciplinary+Journal+of+Nonlinear+Science%2C+6%283%29%3A414%E2%80%93427%2E)
72. Chen Shani, Yuval Reif, Nathan Roll, Dan Jurafsky, and Ekaterina Shutova. 2026. The roots of performance disparity in multilingual language models: Intrinsic modeling difficulty or design choices? Preprint, arXiv:2601.07220. [find](https://scholar.google.com/scholar?q=Chen+Shani%2C+Yuval+Reif%2C+Nathan+Roll%2C+Dan+Jurafsky%2C+and+Ekaterina+Shutova%2E+2026%2E+The+roots+of+performance+disparity+in+multilingual+language+models%3A+Intrinsic+modeling+difficulty+or+design+choices%3F+Pre)
73. Kushal Tatariya, Wessel Poelman, and Miryam De Lhoneux. 2025. On the interplay between positional encodings, morphological complexity, and word order flexibility. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapt… [find](https://scholar.google.com/scholar?q=Kushal+Tatariya%2C+Wessel+Poelman%2C+and+Miryam+De+Lhoneux%2E+2025%2E+On+the+interplay+between+positional+encodings%2C+morphological+complexity%2C+and+word+order+flexibility%2E+In+Proceedings+of+the+14th+Internatio)
74. Tianyang Xu, Tatsuki Kuribayashi, Yohei Oseki, Ryan Cotterell, and Alex Warstadt. 2025. [Can language models learn typologically implausible languages?](https://md.research.elamur.ai/free/en/papers/xu-2026-can-language-models-learn-typologically/) Preprint, arXiv:2502.12317.
75. Claude E. Shannon. 1948. A mathematical theory of communication. Bell System Technical Journal, 27(3):379–423. [find](https://scholar.google.com/scholar?q=Claude+E%2E+Shannon%2E+1948%2E+A+mathematical+theory+of+communication%2E+Bell+System+Technical+Journal%2C+27%283%29%3A379%E2%80%93423%2E)
76. Xiulin Yang, Tatsuya Aoyama, Yuekun Yao, and Ethan Wilcox. 2025. [Anything goes? A crosslinguistic study of (im)possible language learning in LMs](https://md.research.elamur.ai/free/en/papers/yang-2025-anything-goes-crosslinguistic-study-im/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 26058–26077, Vienna, Austria. As…
77. Anna Siewierska, editor. 2010. Constituent Order in the Languages of Europe, 1st edition. Number Eurotyp 20-1 in Empirical Approaches to Language Typology. De Gruyter, Berlin. [find](https://scholar.google.com/scholar?q=Anna+Siewierska%2C+editor%2E+2010%2E+Constituent+Order+in+the+Languages+of+Europe%2C+1st+edition%2E+Number+Eurotyp+20%2D1+in+Empirical+Approaches+to+Language+Typology%2E+De+Gruyter%2C+Berlin%2E)
78. George Kingsley Zipf. 1935. The Psycho-Biology of Language: An Introduction to Dynamic Philology, 1st edition. Houghton Mifflin, Boston, Massachusetts. [find](https://scholar.google.com/scholar?q=George+Kingsley+Zipf%2E+1935%2E+The+Psycho%2DBiology+of+Language%3A+An+Introduction+to+Dynamic+Philology%2C+1st+edition%2E+Houghton+Mifflin%2C+Boston%2C+Massachusetts%2E)
79. Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3):302–319. [find](https://scholar.google.com/scholar?q=Nathaniel+J%2E+Smith+and+Roger+Levy%2E+2013%2E+The+effect+of+word+predictability+on+reading+time+is+logarithmic%2E+Cognition%2C+128%283%29%3A302%E2%80%93319%2E)
80. Taiga Someya, Anej Svete, Brian DuSell, Timothy J. O’Donnell, Mario Giulianelli, and Ryan Cotterell. 2025. Information locality as an inductive bias for neural language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2… [find](https://scholar.google.com/scholar?q=Taiga+Someya%2C+Anej+Svete%2C+Brian+DuSell%2C+Timothy+J%2E+O%E2%80%99Donnell%2C+Mario+Giulianelli%2C+and+Ryan+Cotterell%2E+2025%2E+Information+locality+as+an+inductive+bias+for+neural+language+models%2E+In+Proceedings+of+the+6)
81. Helena Sulkala and Merja Karjalainen. 2012. Finnish, 1st edition. Descriptive grammars. Routledge, London. [find](https://scholar.google.com/scholar?q=Helena+Sulkala+and+Merja+Karjalainen%2E+2012%2E+Finnish%2C+1st+edition%2E+Descriptive+grammars%2E+Routledge%2C+London%2E)
82. Peter Svenonius. 2025. Word order universals and their relationship to structure. Annual Review of Linguistics, 11(1):137–162. [find](https://scholar.google.com/scholar?q=Peter+Svenonius%2E+2025%2E+Word+order+universals+and+their+relationship+to+structure%2E+Annual+Review+of+Linguistics%2C+11%281%29%3A137%E2%80%93162%2E)
83. George Kingsley Zipf. 1949. Human Behavior and the Principle of Least Effort: An Introduction to Human Ecology. Addison-Wesley Press., Cambridge. [find](https://scholar.google.com/scholar?q=George+Kingsley+Zipf%2E+1949%2E+Human+Behavior+and+the+Principle+of+Least+Effort%3A+An+Introduction+to+Human+Ecology%2E+Addison%2DWesley+Press%2E%2C+Cambridge%2E)
84. Imry Ziv, Nur Lan, Emmanuel Chemla, and Roni Katzir. 2025. [Biasless language models learn unnaturally: How LLMs fail to distinguish the possible from the impossible](https://md.research.elamur.ai/free/en/papers/ziv-2026-biasless-language-models-learn-unnaturally/). Preprint, arXiv:2510.07178.

*Updated 1 November 2026*