--- title: "The role of grammar in transition-probabilities of subsequent" description: "Zipf's law for word frequencies can be produced by sample-space reducing (SSR) processes, in which each new word narrows the set of…" authors: ["Rudolf Hanel", "Stefan Thurner"] year: 2018 venue: "arXiv:1812.10991" source: "https://arxiv.org/abs/1812.10991" doi: "10.48550/arxiv.1812.10991" arxiv: "1812.10991" licence: "arXiv — non-exclusive license to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/hanel-2018-role-grammar-transition-probabilities-subsequent/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hanel-2018-role-grammar-transition-probabilities-subsequent/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/6/) / The role of grammar in transition-probabilities of subsequent words in English text # The role of grammar in transition-probabilities of subsequent words in English text **Authors:** Rudolf Hanel and Stefan Thurner · **Year:** 2018 · **Venue:** [arXiv:1812.10991](https://md.research.elamur.ai/free/en/venues/plos-one/) ([PLoS ONE 2018](https://md.research.elamur.ai/free/en/venues/plos-one/2018/)) · **Publisher:** [Public Library of Science](https://md.research.elamur.ai/free/en/publishers/public-library-of-science/) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/1812.10991) · [PDF](https://arxiv.org/pdf/1812.10991) · [DOI 10.48550/arxiv.1812.10991](https://doi.org/10.48550/arxiv.1812.10991) · [arXiv 1812.10991](https://arxiv.org/abs/1812.10991) · [OpenAlex](https://openalex.org/W2907228773) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Zipf's law for word frequencies can be produced by sample-space reducing (SSR) processes, in which each new word narrows the set of words that may follow. Pure SSR processes, however, yield triangular word-to-word transition matrices. Real English text has matrices that look almost random, close to a Bernoulli model with the same word frequencies. Hanel and Thurner resolve this mismatch by treating text generation as two stacked processes. An SSR process first picks the words of a narrative. A toy grammar then rearranges them locally. Each lexicon word gets one of N_g grammatical labels (for example S, P, O), and a "neutral" label marks words that fuse with the next labelled word. The authors call this the grammatically ordered SSR (goSSR) model. They test it on English novels from Project Gutenberg and compare it with the Simon, random typewriting and Bernoulli models using matrix-level statistics. Only three to five labels are needed to keep Zipf's law and to produce realistic transition matrices. Because local reordering hides the structure of word selection, adjacent-word statistics reveal little about how meaning is built. ## Key points - A pure SSR process with W ordered states, restarted repeatedly, visits state i with probability proportional to 1/i, which is exactly Zipf's law. Its transition matrix is triangular. - The goSSR model assigns each word one of N_g roughly equally frequent labels, plus a neutral label 0 with probability p_n. Neutral words merge with the next non-neutral word into one unit. - Using N_g = 3–5 labels is enough to make goSSR transition matrices resemble those of real text. The figures compare N_g = 2 with N_g = 5 and p_n = 0 with p_n = 0.2. - Among the example books, "Five little peppers and how they grew" contains about 4,800 distinct words, roughly 88,000 word transitions and about 5,200 sentences. - Two test statistics are used: - cos(A), the cosine between the all-ones vector v and Av, which serves as a proxy for the largest eigenvalue; - skew(A), the summed antisymmetry A_ij − A_ji over the upper triangle. ## Interesting ideas - The null model shuffles every word in the text but puts sentence ends back at their original positions, so sentence-length structure survives while word order is destroyed. - Baselines are fitted fairly: the Simon model's new-word probability is adjusted over time so its output matches the target text's vocabulary size and length. - Choosing content and ordering it are modelled as separate stages, and the ordering step largely hides the selection structure from bigram statistics. - Function-like "neutral" words are glued to the next content word and moved as one block. - Whole transition matrices are compared with a few cheap scalars: convergence toward the stationary distribution (cos) and directional asymmetry (skew). ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): Use such sentence-preserving shuffles as a negative control. If a metric still rewards shuffled input, it is measuring frequencies rather than structure. - **I2** (learn): When comparing parsers or generators against trivial baselines, match the baselines on length and vocabulary so differences reflect structure rather than size effects. - **I3** (back): Generation can first select content from the world graph and then linearize it with grammatical slot rules. This also argues that adjacency statistics are a poor test of meaning preservation in the compressor. - **I4** (tree): This mirrors attaching det/case/aux to their content head. Reordering or comparing at the level of such head-plus-function chunks could simplify UD dialect conversion and word-order checks. - **I5** (back): Compute cos and skew on generated versus original text corpora as a quick sanity check that generation or translation output has natural sequential statistics. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Santa Fe Institute](https://md.research.elamur.ai/free/en/labs/santa-fe-institute/) — Research institute, Santa Fe, United States · 3 works · 124 citations - [Complexity Science Hub](https://md.research.elamur.ai/free/en/ngos/complexity-science-hub/) — Nonprofit, Vienna, Austria · 2 works · 235 citations Also on the byline: [Medical University of Vienna](https://ror.org/05n3x4p02) · [International Institute for Applied Systems Analysis](https://ror.org/02wfhk785). 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works 🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works ## Cite Cite as: Rudolf Hanel and Stefan Thurner. 2018. The role of grammar in transition-probabilities of subsequent words in English text. arXiv:1812.10991. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{hanel2018role, title = {{The role of grammar in transition-probabilities of subsequent words in English text}}, author = {Rudolf Hanel and Stefan Thurner}, year = {2018}, doi = {10.48550/arxiv.1812.10991}, eprint = {1812.10991}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.1812.10991} } ``` - [← previous The Relevance of Text and Speech Features in Automatic Non-native English Accent Identification](https://md.research.elamur.ai/free/en/papers/vajjala-2018-relevance-text-speech-features-automatic/) - [next → The semantic transparency of English compound nouns](https://md.research.elamur.ai/free/en/papers/schafer-2018-semantic-transparency-english-compound/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/6/) *Updated 24 October 2026*