---
title: "Squibs — paper by Fan (2010) — summary and ideas"
description: "Studies of English texts often find that words occurring only once (hapax legomena) make up about half of the vocabulary… 12 references."
authors: ["Fengxiang Fan"]
year: 2010
venue: "Computational Linguistics"
source: "https://aclanthology.org/J10-4003/"
doi: "10.1162/coli_a_00013"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/fan-2010-squibs-asymptotic-model-english-hapax/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/fan-2010-squibs-asymptotic-model-english-hapax/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2010](https://md.research.elamur.ai/free/en/papers/2010/page/3/) / Squibs: An Asymptotic Model for the English Hapax/Vocabulary Ratio

# Squibs: An Asymptotic Model for the English Hapax/Vocabulary Ratio

**Authors:** Fengxiang Fan · **Year:** 2010 · **Venue:** [Computational Linguistics](https://md.research.elamur.ai/free/en/venues/cl/) ([Computational Linguistics 2010](https://md.research.elamur.ai/free/en/venues/cl/2010/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/J10-4003/) · [PDF](https://aclanthology.org/J10-4003.pdf) · [DOI 10.1162/coli\_a\_00013](https://doi.org/10.1162/coli_a_00013) · [OpenAlex](https://openalex.org/W2070296328)

**Citations:** 9 citations (OpenAlex, 2026-10-14)

## Summary
Studies of English texts often find that words occurring only once (hapax legomena) make up about half of the vocabulary, whatever the text size. Fan asks why this hapax/vocabulary ratio (HVR) looks constant and whether a larger corpus would lower it. He split the British National Corpus (BNC), about 100 million words, into roughly 4,200-word blocks. The text was tokenized and lemmatized, with case ignored and vocabulary counted as lemmas. He then built random cumulative samples and tracked vocabulary size, hapax count and HVR as the samples grew.

HVR turns out not to be constant. It follows a U-shape: it falls at first, bottoms out at a "Point Of Return" (POR) of a few million words, then rises slowly. A manual audit of BNC hapaxes shows the cause. Most of them are names, codes, numerals, foreign items and typos, and such strings keep accumulating without limit. A simulation that mixes a novel's vocabulary with random strings suggests HVR tends toward 1 as corpus size grows. Very large corpora therefore add mostly noise and do not cure sparseness.

## Key points
- **Data and sampling:**
  - The BNC was cut into 23,709 blocks.
  - These were regrouped by sampling without replacement into ten sets of about 10M tokens each.
  - Across sets, mean V = 103,588.9, mean H = 42,384.6 and mean final HVR = 0.4091.
  - The HVR minimum averaged about 0.3928.
  - POR values ranged from 1,515,629 to 4,382,688 tokens, with a mean of about 2.96M.
- **Full BNC:**
  - V = 346,578 and H = 154,403.
  - HVR starts at 0.4583, drops to 0.3899 at POR = 3,820,340 tokens, and ends at 0.4455.
- **Audit of 1,000 random BNC hapaxes:**
  - 232 were dictionary words: 137 general, 93 technical and 2 Early Modern.
  - 718 were out-of-dictionary: 416 names, 161 alphanumeric strings, 123 numerals, 16 foreign words and 2 interjections.
  - 50 were typos.
- **Hapax length:** hapaxes average 7.74 characters, compared with 7.29 for other words. The longest BNC word, a 104-character string, is itself a hapax.
- **Simulation:**
  - The source was *Through the Looking-Glass* (30,566 tokens, 2,754 types).
  - Each 100-token sample held 94 drawn words plus 6 random strings of 6–9 characters.
  - After about 64M tokens, HVR reached 0.9991, with POR at 20,300 tokens.

## Interesting ideas
- Instead of quoting one HVR number, the author follows the whole growth curve and uses its minimum as a sign of saturation; this makes it possible to tell when core vocabulary is covered and when growth is mostly noise.
- A random sample of singletons is sorted by hand into dictionary words, names, alphanumeric codes, numerals, foreign words and typos, which shows what "rare words" really are.
- The hypothesis is tested with a controlled toy world: a closed vocabulary from one novel plus a known rate of injected random strings, so the expected curve can be predicted and checked.
- The POR was measured over ten independent random orderings of the same blocks, which exposes how much the result depends on sampling order.
- Singletons tend to be longer than recurring words, and many look like character noise, so word length is a cheap cue for out-of-dictionary material.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Plot the hapax curve while growing the silver corpus, and treat passing the POR as the point where more raw text adds names and junk rather than new core lexicon.
- **I2** (tokens): Run the same audit on unknown tokens from the tokenizer and lemmatizer, and send names, numbers, codes and probable typos to dedicated handlers instead of growing the lexicon.
- **I3** (learn): Inject synthetic random strings into evaluation texts as a negative control, and verify that coverage metrics and lexicon updates react as predicted rather than absorbing the noise.
- **I4** (learn): Report corpus-growth and learning curves over several random shufflings, so that conclusions about data sufficiency are not artifacts of one ordering.
- **I5** (tokens): Combine length and character-shape features in the tagger to flag likely codes, foreign strings or garbage before attempting lemmatization.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Dalian Maritime University](https://md.research.elamur.ai/free/en/universities/dalian-maritime-university/) — University, Dalian, China · 2 works · 20 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [📚 Literature and Literary Theory](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/literature-and-literary-theory/) · 519 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works

## Cite

Cite as: Fengxiang Fan. 2010. Squibs: An Asymptotic Model for the English Hapax/Vocabulary Ratio. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{fan2010squibs,
  title = {{Squibs: An Asymptotic Model for the English Hapax/Vocabulary Ratio}},
  author = {Fengxiang Fan},
  journal = {Computational Linguistics},
  year = {2010},
  doi = {10.1162/coli\_a\_00013},
  url = {https://doi.org/10.1162/coli_a_00013}
}
```

- [← previous Hedge Detection Using the RelHunter Approach](https://md.research.elamur.ai/free/en/papers/fernandes-2010-hedge-detection-relhunter-approach/)
- [next → A Detailed, Accurate, Extensive, Available English Lexical Database](https://md.research.elamur.ai/free/en/papers/kilgarriff-2010-detailed-accurate-extensive-available-english/)
- [↑ back to papers of 2010](https://md.research.elamur.ai/free/en/papers/2010/page/3/)

## References

12 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Baayen, Harald. 1996. The effects of lexical specialization on the growth curve of the vocabulary. Computational Linguistics, 22(4):455–480. [find](https://scholar.google.com/scholar?q=Baayen%2C+Harald%2E+1996%2E+The+effects+of+lexical+specialization+on+the+growth+curve+of+the+vocabulary%2E+Computational+Linguistics%2C+22%284%29%3A455%E2%80%93480%2E)
2. Baayen, Harald. 2001. Word Frequency Distributions. Kluwer Academic Publishers, Dordrecht. [↗](https://doi.org/10.1007/978-94-010-0844-0)
3. Biber, Douglas, Stig Johansson, Geoffrey Leech, Susan Conrad, and Edward Finegan. 1999. Longman Grammar of Spoken and Written English. Pearson Education Limited, Harlow, Essex. [find](https://scholar.google.com/scholar?q=Biber%2C+Douglas%2C+Stig+Johansson%2C+Geoffrey+Leech%2C+Susan+Conrad%2C+and+Edward+Finegan%2E+1999%2E+Longman+Grammar+of+Spoken+and+Written+English%2E+Pearson+Education+Limited%2C+Harlow%2C+Essex%2E)
4. Holmes, David. 1991. Vocabulary richness and the prophetic voice. Literary and Linguistic Computing, 6(4):259–268. [↗](https://doi.org/10.1093/llc/6.4.259)
5. Jurafsky, Daniel and James Martin. 2000. Speech and Language Processing, an Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Prentice-Hall, Upper Saddle River. [find](https://scholar.google.com/scholar?q=Jurafsky%2C+Daniel+and+James+Martin%2E+2000%2E+Speech+and+Language+Processing%2C+an+Introduction+to+Natural+Language+Processing%2C+Computational+Linguistics%2C+and+Speech+Recognition%2E+Prentice%2DHall%2C+Upper+Saddle+)
6. Kennedy, Graeme. 1998. An Introduction to Corpus Linguistics. Addison Wesley, London. [find](https://scholar.google.com/scholar?q=Kennedy%2C+Graeme%2E+1998%2E+An+Introduction+to+Corpus+Linguistics%2E+Addison+Wesley%2C+London%2E)
7. Kornai, A. 2002. How many words are there? Glottometrics, 4:61–86. [find](https://scholar.google.com/scholar?q=Kornai%2C+A%2E+2002%2E+How+many+words+are+there%3F+Glottometrics%2C+4%3A61%E2%80%9386%2E)
8. Kučera, H. and W. Francis. 1967. Computational Analysis of Present-Day American English. Brown University Press, Providence, RI. [find](https://scholar.google.com/scholar?q=Kuc%CC%8Cera%2C+H%2E+and+W%2E+Francis%2E+1967%2E+Computational+Analysis+of+Present%2DDay+American+English%2E+Brown+University+Press%2C+Providence%2C+RI%2E)
9. Manning, Christopher and Hinrich Schütze. 2001. Foundations of Statistical Natural Language Processing. The MIT Press, Cambridge, MA. [find](https://scholar.google.com/scholar?q=Manning%2C+Christopher+and+Hinrich+Schu%CC%88tze%2E+2001%2E+Foundations+of+Statistical+Natural+Language+Processing%2E+The+MIT+Press%2C+Cambridge%2C+MA%2E)
10. Popescu, I. I. and G. Altmann. 2008. Hapax legomena and language typology. Journal of Quantitative Linguistics, 15(4):370–378. [↗](https://doi.org/10.1080/09296170802326699)
11. Popescu, I. I., J. Mačutek, and G. Altmann. 2009. Aspects of word frequencies. RAM, Lüdenscheid. [find](https://scholar.google.com/scholar?q=Popescu%2C+I%2E+I%2E%2C+J%2E+Mac%CC%8Cutek%2C+and+G%2E+Altmann%2E+2009%2E+Aspects+of+word+frequencies%2E+RAM%2C+Lu%CC%88denscheid%2E)
12. Tweedie, Fiona and Harald Baayen. 1998. How variable may a constant be? Measure of lexical richness in perspective. Journal of Quantitative Linguistics, 32(5):323–352. [find](https://scholar.google.com/scholar?q=Tweedie%2C+Fiona+and+Harald+Baayen%2E+1998%2E+How+variable+may+a+constant+be%3F+Measure+of+lexical+richness+in+perspective%2E+Journal+of+Quantitative+Linguistics%2C+32%285%29%3A323%E2%80%93352%2E)

*Updated 1 November 2026*