---
title: "The distribution of information content in English sentences (2016)"
description: "The paper asks how information is spread across word positions in an English sentence. 38 references."
authors: ["Shuiyuan Yu", "Jin Cong", "Junying Liang", "Haitao Liu"]
year: 2016
venue: "arXiv:1609.07681"
source: "https://arxiv.org/abs/1609.07681"
doi: "10.48550/arxiv.1609.07681"
arxiv: "1609.07681"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yu-2016-distribution-information-content-english-sentences/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yu-2016-distribution-information-content-english-sentences/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/) / The distribution of information content in English sentences

# The distribution of information content in English sentences

**Authors:** Shuiyuan Yu, Jin Cong, Junying Liang and Haitao Liu · **Year:** 2016 · **Venue:** arXiv:1609.07681 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1609.07681) · [PDF](https://arxiv.org/pdf/1609.07681) · [DOI 10.48550/arxiv.1609.07681](https://doi.org/10.48550/arxiv.1609.07681) · [arXiv 1609.07681](https://arxiv.org/abs/1609.07681) · [OpenAlex](https://openalex.org/W2524886565)

**Citations:** 4 citations (OpenAlex, 2026-10-14)

## Summary
The paper asks how information is spread across word positions in an English sentence. Two existing views disagree. Constant Entropy Rate (CER) and Uniform Information Density (UID) predict an even spread. Earlier psycholinguistic work instead points to a "sentence-final effect." The authors used part of the British National Corpus (BNC) and treated each sentence length from 3 to 50 words separately. For every position they computed a positional entropy, a unigram entropy conditioned only on position. They also tracked several related statistics: the number of distinct types, the power-law exponent fitted to the words' frequency profile, the average length and average frequency of words, and the proportions of high-, medium- and low-frequency words. All measures form a three-step staircase. The first position has the least information, the final position has the most, and the medial positions stay statistically flat. The authors draw three conclusions. CER and UID do not explain the medial plateau. A word's context should not be reduced to the words before it. Sentence context behaves holistically rather than adding up word by word. These points matter for how language models define context.

## Key points
- **Data:** a BNC subset of 17,293 texts, with about 34.85M tokens, 328,935 types and about 1.96M sentences. Word rank-frequency follows a power law (adjusted R² = 0.9992).
- **Positional entropy:** H(X) = −Σ pᵢ log₂ pᵢ, computed over the words seen at a given position. It relates to Shannon's F1. The paper decomposes it as H(X) = I(X,C) + H(X|C).
- **Flat medial positions:** these show no significant trend. For length 15, correlation is 0.4770 with F = 2.3569 (p = 0.1633). For lengths 30 and 45, p is 0.7731 and 0.7104.
- **Positions excluded from the medial test:** the 2nd, 3rd, antepenultimate and penultimate positions were left out because they behave differently.
- **Frequency classes:** the top 100 words by rank (mostly function words), the bottom 100 (rare), and everything else (medium). Final positions are dominated by low-frequency content words, which matches Behaghel's observation about German.

## Interesting ideas
- Bin sentences by exact length before comparing positions, because a given position plays different roles in short and long sentences; this removes a hidden confound from any per-position statistic.
- Use a context-free, position-conditioned entropy as a measurable proxy when true contextual information is too hard to estimate, and read the gap through H(X) = I(X,C) + H(X|C).
- Test whether the middle of a sequence is flat by combining a correlation with position and an F-test, after excluding the unstable positions next to the edges.
- Treat crude frequency-rank classes (top 100 versus the rest) as a cheap function-word versus content-word signal; it needs no annotation yet still tracks informativeness.
- The argument that a word's context is the whole sentence, not just its left history, challenges purely Markov/n-gram views of context.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Report parser and tagger error rates by position within length buckets, so that apparent "position effects" in evaluation are not artifacts of length mixing.
- **I2** (tokens): Position-conditioned lexical priors could inform unknown-word guessing, for example by expecting rare content words more often at sentence end.
- **I3** (learn): Use this as a negative-control check that a confidence score or silver-agreement rate does not drift with token position when it should not.
- **I4** (world): A rank-based salience cue, combined with the finding that sentence-final words carry the topic, could help decide which entities to prioritise as new or focal in the world model.
- **I5** (tree): This supports whole-sentence (bidirectional) features in gparse rather than left-to-right-only cues when attaching late, topic-bearing content words.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🧠 Neuroscience](https://md.research.elamur.ai/free/en/shelves/neuroscience/) · 95 works · [🧠 Cognitive Neuroscience](https://md.research.elamur.ai/free/en/shelves/neuroscience/cognitive-neuroscience/) · 95 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works

## Cite

Cite as: Shuiyuan Yu, Jin Cong, Junying Liang and Haitao Liu. 2016. The distribution of information content in English sentences. arXiv:1609.07681.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{yu2016distribution,
  title = {{The distribution of information content in English sentences}},
  author = {Shuiyuan Yu, Jin Cong, Junying Liang and Haitao Liu},
  year = {2016},
  doi = {10.48550/arxiv.1609.07681},
  eprint = {1609.07681},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1609.07681}
}
```

- [← previous Sarcasm Detection : Building a Contextual Hierarchy](https://md.research.elamur.ai/free/en/papers/bali-2016-sarcasm-detection-building-contextual-hierarchy/)
- [next → The material variance of the Dead Sea Scrolls: On texts and artefacts](https://md.research.elamur.ai/free/en/papers/tigchelaar-2016-material-variance-dead-sea-scrolls/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/)

## References

38 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. van Son R, Koopmans-van Beinum FJ, Pols LCW (1998) Efficiency as an organizing principle of natural speech. ICSLP98, p 0. [find](https://scholar.google.com/scholar?q=van+Son+R%2C+Koopmans%2Dvan+Beinum+FJ%2C+Pols+LCW+%281998%29+Efficiency+as+an+organizing+principle+of+natural+speech%2E+ICSLP98%2C+p+0%2E)
2. van Son R, van Santen J (2005) Duration and spectral balance of intervocalic consonants: A case for efficient communication. Speech Communication 47: 100–123. [↗](https://doi.org/10.1016/j.specom.2005.06.005)
3. Bolinger D (1963) Length, vowel, juncture. Linguistics 1: 5–29. [↗](https://doi.org/10.1515/ling.1963.1.1.5)
4. Lieberman P (1963) Some effects of semantic and grammatical context on the production and perception of speech. Lang Speech 6: 172–187. [↗](https://doi.org/10.1177/002383096300600306)
5. Aylett MP (1999) Stochastic suprasegmentals: Relationships between redundancy, prosodic structure and syllabic duration. Proceedings of the 1999 International Conference on Spoken Language Processing pp. 289-292. [find](https://scholar.google.com/scholar?q=Aylett+MP+%281999%29+Stochastic+suprasegmentals%3A+Relationships+between+redundancy%2C+prosodic+structure+and+syllabic+duration%2E+Proceedings+of+the+1999+International+Conference+on+Spoken+Language+Processing+)
6. Pluymaekers M, Ernestusb M, Baayen RH (2005) Lexical frequency and acoustic reduction in spoken Dutch. J Acoust Soc Am 118: 2561–2569. [↗](https://doi.org/10.1121/1.2011150)
7. Bell A, Brenier J, Gregory M, Girand C, Jurafsky D. (2009) Predictability effects on durations of content and function words in conversational English. Journal of Memory and Language 60: 92–111. [↗](https://doi.org/10.1016/j.jml.2008.06.003)
8. Jaeger TF (2006) Redundancy and syntactic reduction in spontaneous speech. PhD thesis (Stanford University, Palo Alto, CA). [find](https://scholar.google.com/scholar?q=Jaeger+TF+%282006%29+Redundancy+and+syntactic+reduction+in+spontaneous+speech%2E+PhD+thesis+%28Stanford+University%2C+Palo+Alto%2C+CA%29%2E)
9. Levy R, Jaeger TF (2007) Speakers optimize information density through syntactic reduction. Advances in Neural Information Processing Systems 19, eds Schölkopf B, Platt J, Hoffman T (MIT Press, Cambridge, MA), pp 849–856. [↗](https://doi.org/10.7551/mitpress/7503.003.0111)
10. Aylett M, Turk A (2004) The smooth signal redundancy hypothesis: A functional explanation for relationships between redundancy, prosodic prominence, and duration in spontaneous speech. Lang Speech 47:31–56. [↗](https://doi.org/10.1177/00238309040470010201)
11. Mahowald K, Fedorenko E, Piantadosi ST, Gibson E. (2013) Info/information theory: speakers choose shorter words in predictive contexts. Cognition 126:313-8. [↗](https://doi.org/10.1016/j.cognition.2012.09.010)
12. Frank AF, Jaeger TF (2008) Speaking rationally: Uniform information density as an optimal strategy for language production. The 31st Annual Meeting of the Cognitive Science Society, eds Love BC, McRae K, Sloutsky VM (Cognitive Science Society, Austin, TX), pp 939–944. [find](https://scholar.google.com/scholar?q=Frank+AF%2C+Jaeger+TF+%282008%29+Speaking+rationally%3A+Uniform+information+density+as+an+optimal+strategy+for+language+production%2E+The+31st+Annual+Meeting+of+the+Cognitive+Science+Society%2C+eds+Love+BC%2C+McRae)
13. Manin D (2006) Experiments on predictability of word in context and information rate in natural language. Journal of Information Processes 6:229–236. [↗](https://doi.org/10.48550/arxiv.cs/0612136)
14. Piantadosi ST, Tily H, Gibson E (2011) Word lengths are optimized for efficient communication. Proc Natl Acad Sci USA 108:3526–3529. [↗](https://doi.org/10.1073/pnas.1012551108)
15. Pellegrino F, Coupé C, Marsico E (2011) Across-language perspective on speech information rate. Language 87: 539–558. [find](https://scholar.google.com/scholar?q=Pellegrino+F%2C+Coup%C3%A9+C%2C+Marsico+E+%282011%29+Across%2Dlanguage+perspective+on+speech+information+rate%2E+Language+87%3A+539%E2%80%93558%2E)
16. Hale J (2003) The information conveyed by words in sentences. J Psycholinguist Res 32:101–123. [↗](https://doi.org/10.1023/a:1022492123056)
17. Maurlts L, Perfors A, Navarro D (2010) Why are some word orders more common than others? A uniform information density account. Advances in Neural Information Process Systems 23: 1585-1593. [find](https://scholar.google.com/scholar?q=Maurlts+L%2C+Perfors+A%2C+Navarro+D+%282010%29+Why+are+some+word+orders+more+common+than+others%3F+A+uniform+information+density+account%2E+Advances+in+Neural+Information+Process+Systems+23%3A+1585%2D1593%2E)
18. Tomlin RS (1986) Basic word order: functional principles (Croom Helm, London). [find](https://scholar.google.com/scholar?q=Tomlin+RS+%281986%29+Basic+word+order%3A+functional+principles+%28Croom+Helm%2C+London%29%2E)
19. Genzel D, Charniak E (2002) [Entropy rate constancy in text](https://md.research.elamur.ai/free/en/papers/genzel-2002-entropy-rate-constancy-text/). Proceedings of the 40th annual meeting of the association for Computational linguistics pp. 199–206.
20. Shannon C (1948) A mathematical theory of communications. Bell Systems Technical Journal 27: 623 – 656. [find](https://scholar.google.com/scholar?q=Shannon+C+%281948%29+A+mathematical+theory+of+communications%2E+Bell+Systems+Technical+Journal+27%3A+623+%E2%80%93+656%2E)
21. Behaghel O (1909) Beziehungen zwischen Umfang und Reihenfolge von Satzgliedern. Indogermanische Forschungen 25: 110-142. [↗](https://doi.org/10.1515/9783110242652.110)
22. Rubenstein H, Pickett JM (1958) Intelligibility of Words in Sentences. J Acoust Soc Am 30: 670-670. [find](https://scholar.google.com/scholar?q=Rubenstein+H%2C+Pickett+JM+%281958%29+Intelligibility+of+Words+in+Sentences%2E+J+Acoust+Soc+Am+30%3A+670%2D670%2E)
23. Rubenstein H, Pickett JM (1957) Word Intelligibility and Position in Sentence. J Acoust Soc Am 29: 1263-1263. [find](https://scholar.google.com/scholar?q=Rubenstein+H%2C+Pickett+JM+%281957%29+Word+Intelligibility+and+Position+in+Sentence%2E+J+Acoust+Soc+Am+29%3A+1263%2D1263%2E)
24. Ball P (2011) How words get the message across. Nature News, 10.1038/news.2011.40. Available at http://www.nature.com/news/2011/110124/full/news.2011.40.html. [↗](https://doi.org/10.1038/news.2011.40)
25. Griffiths TL (2011). Rethinking language: How probabilities shape the words we use. Proc Natl Acad Sci USA 108: 3825-3826. [↗](https://doi.org/10.1073/pnas.1100760108)
26. Bybee J (2007) Frequency of Use and the Organization of Language (Oxford University Press, New York). [↗](https://doi.org/10.1093/acprof:oso/9780195301571.001.0001)
27. Bybee J (2003) Mechanisms of change in grammaticization: The role of frequency. The Handbook of Historical Linguistics eds Joseph BD, Janda RD (Blackwell, Oxford), pp 602–623. [↗](https://doi.org/10.1002/9780470756393.ch19)
28. Hopper PJ, Traugott EC (1993) Grammaticalization (Cambridge University Press, Cambridge). [find](https://scholar.google.com/scholar?q=Hopper+PJ%2C+Traugott+EC+%281993%29+Grammaticalization+%28Cambridge+University+Press%2C+Cambridge%29%2E)
29. Bybee J, Hopper P (2001) Introduction to frequency and the emergence of linguistic structure. Frequency and the Emergence of Linguistic Structure eds Bybee J, Hopper P (John Benjamins, Amsterdam), pp 1–24. [↗](https://doi.org/10.1075/tsl.45.01byb)
30. Haspelmath M (2008) Creating economical morphosyntactic patterns in language change. Language universals and language change ed Good J (Oxford University Press, Oxford), pp 185–214. [find](https://scholar.google.com/scholar?q=Haspelmath+M+%282008%29+Creating+economical+morphosyntactic+patterns+in+language+change%2E+Language+universals+and+language+change+ed+Good+J+%28Oxford+University+Press%2C+Oxford%29%2C+pp+185%E2%80%93214%2E)
31. Haspelmath M (2008) Frequency vs. iconicity in explaining grammatical asymmetries. Cogn Linguist 19: 1–33. [↗](https://doi.org/10.1515/cog.2008.001)
32. Liu H (2008) The complexity of Chinese syntactic dependency networks. Physica A 387:3048--3058. [↗](https://doi.org/10.1016/j.physa.2008.01.069)
33. Solé RV, Corominas-Murtra B, Valverde S, Steels L (2010) Language networks: Their structure, function and evolution. Complexity 15: 20-26. [↗](https://doi.org/10.1002/cplx.20305)
34. Clear J (1993) The British National Corpus (MIT Press, Cambridge, MA). [find](https://scholar.google.com/scholar?q=Clear+J+%281993%29+The+British+National+Corpus+%28MIT+Press%2C+Cambridge%2C+MA%29%2E)
35. Paninski L (2003) Estimation of entropy and mutual information. Neural Comput 15: 1191–1253. [↗](https://doi.org/10.1162/089976603321780272)
36. Shannon C (1951) Prediction and entropy of printed English. The Bell System Technical Journal 30: 50-64. [↗](https://doi.org/10.1002/j.1538-7305.1951.tb01366.x)
37. Schurmann T, Grassberger P (1996) Entropy estimation of symbol sequences. Chaos 6: 414–427. [↗](https://doi.org/10.1063/1.166191)
38. Ferrer-i-Cancho R, Dębowski Ł, Moscoso del Prado Martín F (2013) Constant conditional entropy and related hypotheses. Journal of Statistical Mechanics L07001. (2013-07-19, the first version) [↗](https://doi.org/10.1088/1742-5468/2013/07/l07001)

*Updated 24 October 2026*