--- title: "Accurate Methods for the Statistics of — Dunning 1993" description: "Statistical text analysis often relies on tests such as Pearson's χ² and z-scores, which assume that counts are roughly normally distributed." authors: ["Ted Dunning"] year: 1993 source: "https://aclanthology.org/J93-1003/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/dunning-1993-accurate-methods-statistics-surprise-coincidence/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dunning-1993-accurate-methods-statistics-surprise-coincidence/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [1993](https://md.research.elamur.ai/free/en/papers/1993/) / Accurate Methods for the Statistics of Surprise and Coincidence # Accurate Methods for the Statistics of Surprise and Coincidence **Authors:** Ted Dunning · **Year:** 1993 · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/J93-1003/) · [PDF](https://aclanthology.org/J93-1003.pdf) · [OpenAlex](https://openalex.org/W2116780029) **Citations:** 2661 citations (openalex-snapshot · 2026-10-14) · [399](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Statistical text analysis often relies on tests such as Pearson's χ² and z-scores, which assume that counts are roughly normally distributed. Dunning shows that this assumption fails for rare events, and rare events make up much of real text. In Reuters newswire, words rarer than one per 50,000 tokens account for roughly a fifth to a third of running text. When the expected count is small, the normal approximation inflates significance by amounts that vary widely, so the errors cannot simply be corrected afterwards. As an alternative, the paper proposes the generalized log-likelihood ratio statistic (−2 log λ), built on binomial and multinomial models of word counts. It is easy to compute, behaves well on small samples, and lets rare and common phenomena be compared directly. Where χ² is valid, the two measures give nearly the same values. In a bigram-association experiment, χ² ranks one-off co-occurrences at the top. The log-likelihood ranking instead surfaces meaningful collocations such as "the swiss," "can be" and "previous year." The measure became a standard tool for collocation and term extraction. ## Key points - The usual rule of thumb is that the normal approximation to the binomial holds when np(1−p) > 5. Below 1 it breaks down badly: for n = 100, normal estimates overstate significance about 40× at np = 0.1 and about 4×10²⁰ at np = 0.01. - In a half-million-word Reuters sample, everyday words such as "landscape" or "winter" appeared fewer than five times. In a 50,000-word sample, their expected count would fall below one. - Ranking bigrams by χ² gives the maximum score (31777) to pairs seen only once. Under log-likelihood, the top pair is "the swiss" with −2 log λ = 270.72. - The paper contrasts its approach with the IBM machine-translation group, which relied on corpora of about 1 billion words plus the Canadian Hansards, as well as with the count-based mutual information and z-scores used by Gale and Church. - Software implementing the method was to be distributed through the Consortium for Lexical Research. Suggested future work includes Fisher's exact test and Poisson-based models. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) ## Cited by Cited by 399 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 379 not yet. - [Detecting Compositionality in Multi-Word Expressions](https://md.research.elamur.ai/free/en/papers/aclbulk-2009-korkontzelos-detecting-compositionality-multi-word-expressions/) - [Distributional Memory: A General Framework for Corpus-Based Semantics](https://md.research.elamur.ai/free/en/papers/baroni-2010-distributional-memory-general-framework-corpus/) - [Unsupervised Learning of Contextual Role Knowledge for Coreference Resolution](https://md.research.elamur.ai/free/en/papers/bean-2004-unsupervised-learning-contextual-role-knowledge/) - [Text: now in 2D! A framework for lexical expansion with contextual similarity](https://md.research.elamur.ai/free/en/papers/biemann-2013-text-now-2d-framework-lexical/) - [Unsupervised Multilingual Sentence Boundary Detection](https://md.research.elamur.ai/free/en/papers/cl-2006-kiss-unsupervised-multilingual-sentence-boundary-detection/) - [Survey: Multiword Expression Processing: A Survey](https://md.research.elamur.ai/free/en/papers/constant-2017-survey-multiword-expression-processing-survey/) - [A Statistical Approach to Anaphora Resolution](https://md.research.elamur.ai/free/en/papers/ge-1998-statistical-approach-anaphora-resolution/) - [Unrestricted Bridging Resolution](https://md.research.elamur.ai/free/en/papers/hou-2018-unrestricted-bridging-resolution/) - [Automatic Identification of Non-compositional Phrases](https://md.research.elamur.ai/free/en/papers/lin-1999-automatic-identification-non-compositional-phrases/) - [The Automated Acquisition of Topic Signatures for Text Summarization](https://md.research.elamur.ai/free/en/papers/lin-2000-automated-acquisition-topic-signatures-text/) - [Automatic Thesaurus Construction for Modern Hebrew](https://md.research.elamur.ai/free/en/papers/lingbulk-2018-liebeskind-automatic-thesaurus-construction-modern-hebrew/) - [Improved Statistical Machine Translation Using Monolingually-Derived Paraphrases](https://md.research.elamur.ai/free/en/papers/marton-2009-improved-statistical-machine-translation-monolingually/) - [Using Linguistic Features to Improve the Generalization Capability of Neural Coreference Resolvers](https://md.research.elamur.ai/free/en/papers/moosavi-2018-linguistic-features-improve-generalization-capability/) - [Dependency-Based Construction of Semantic Space Models](https://md.research.elamur.ai/free/en/papers/pado-2007-dependency-based-construction-semantic-space/) - [Automatic induction of FrameNet lexical units](https://md.research.elamur.ai/free/en/papers/pennacchiotti-2008-automatic-induction-framenet-lexical-units/) - [Automatic Identification of Word Translations from Unrelated English and German Corpora](https://md.research.elamur.ai/free/en/papers/rapp-1999-automatic-identification-word-translations-unrelated/) - [Social Bias in Elicited Natural Language Inferences](https://md.research.elamur.ai/free/en/papers/rudinger-2017-social-bias-elicited-natural-language/) - [Transparent Semantic Change Detection with Dependency-Based Profiles](https://md.research.elamur.ai/free/en/papers/tat-2026-transparent-semantic-change-detection-dependency/) - [Summarizing Scientific Articles: Experiments with Relevance and Rhetorical Status](https://md.research.elamur.ai/free/en/papers/teufel-2002-summarizing-scientific-articles-experiments-relevance/) - [Antecedent Selection Techniques for High-Recall Coreference Resolution](https://md.research.elamur.ai/free/en/papers/versley-2007-antecedent-selection-techniques-high-recall/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{dunning1993accurate, title = {{Accurate Methods for the Statistics of Surprise and Coincidence}}, author = {Dunning, Ted}, year = {1993}, url = {https://aclanthology.org/J93-1003/} } ``` - [← previous A Program for Aligning Sentences in Bilingual Corpora](https://md.research.elamur.ai/free/en/papers/gale-1993-program-aligning-sentences-bilingual-corpora/) - [next → Aligning Sentences in Bilingual Corpora Using Lexical Information](https://md.research.elamur.ai/free/en/papers/chen-1993-aligning-sentences-bilingual-corpora-lexical/) - [↑ back to papers of 1993](https://md.research.elamur.ai/free/en/papers/1993/) ## References 9 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Bradley, James V. (1968). Distribution-Free Statistical Tests. Prentice Hall. [find](https://scholar.google.com/scholar?q=Bradley%2C+James+V%2E+%281968%29%2E+Distribution%2DFree+Statistical+Tests%2E+Prentice+Hall%2E) 2. Brown, Peter E; Cocke, John; Della Pietra, Stephen A.; Della Pietra, Vincent J.; Jelinek, Frederick; Lafferty, John D.; Mercer, Robert L.; and Roossin, Paul S. (1989). "A statistical approach to machine translation." Technical Report RC 14773 (\#66226), IBM Research Division. [find](https://scholar.google.com/scholar?q=Brown%2C+Peter+E%3B+Cocke%2C+John%3B+Della+Pietra%2C+Stephen+A%2E%3B+Della+Pietra%2C+Vincent+J%2E%3B+Jelinek%2C+Frederick%3B+Lafferty%2C+John+D%2E%3B+Mercer%2C+Robert+L%2E%3B+and+Roossin%2C+Paul+S%2E+%281989%29%2E+%22A+statistical+approach+to+machi) 3. Church, Ken W.; Gale, William A.; Hanks, Patrick; and Hindle, Donald (1989). "Parsing, word associations and typical predicate-argument relations." In Proceedings, International Workshop on Parsing Technologies, CMU. [find](https://scholar.google.com/scholar?q=Church%2C+Ken+W%2E%3B+Gale%2C+William+A%2E%3B+Hanks%2C+Patrick%3B+and+Hindle%2C+Donald+%281989%29%2E+%22Parsing%2C+word+associations+and+typical+predicate%2Dargument+relations%2E%22+In+Proceedings%2C+International+Workshop+on+Parsing+Te) 4. Dumais, S.; Furnas, G.; Landauer, T.; Deerwester, S.; and Harshman, R. (1988). "Using latent semantic analysis to improve access to textual information." In Proceedings, CHI '88. 281-285. [find](https://scholar.google.com/scholar?q=Dumais%2C+S%2E%3B+Furnas%2C+G%2E%3B+Landauer%2C+T%2E%3B+Deerwester%2C+S%2E%3B+and+Harshman%2C+R%2E+%281988%29%2E+%22Using+latent+semantic+analysis+to+improve+access+to+textual+information%2E%22+In+Proceedings%2C+CHI+%2788%2E+281%2D285%2E) 5. Gale, William A., and Church, Ken W. k(~ A .., B) 29114 29121 29159 29077 29152 29154 29155 29163 29165 29158 29027 A the at the be months increased 1986 level again the as B swiss the volume the the the the the the increased the kji ~j kji Pji - qj = ~kjlogpj J (1993). "A program for aligning sente… [find](https://scholar.google.com/scholar?q=Gale%2C+William+A%2E%2C+and+Church%2C+Ken+W%2E+k%28%7E+A+%2E%2E%2C+B%29+29114+29121+29159+29077+29152+29154+29155+29163+29165+29158+29027+A+the+at+the+be+months+increased+1986+level+again+the+as+B+swiss+the+volume+the+the+) [our card](https://md.research.elamur.ai/free/en/papers/gale-1993-program-aligning-sentences-bilingual-corpora/) 6. Gale, William A., and Church, Ken W. (in press). "Identifying word correspondences in parallel texts." McDonald, James E.; Plate, Tony; and Schvaneveldt, Roger (1990). "Using Pathfinder to extract semantic information from text." In Pathfinder Associative Networks: Studies in Knowledge Organization,… [find](https://scholar.google.com/scholar?q=Gale%2C+William+A%2E%2C+and+Church%2C+Ken+W%2E+%28in+press%29%2E+%22Identifying+word+correspondences+in+parallel+texts%2E%22+McDonald%2C+James+E%2E%3B+Plate%2C+Tony%3B+and+Schvaneveldt%2C+Roger+%281990%29%2E+%22Using+Pathfinder+to+extract+sem) 7. Mood, A. M.; Graybill, E A.; and Boes, D. C. (1974). Introduction to the Theory of Statistics. McGraw Hill. [find](https://scholar.google.com/scholar?q=Mood%2C+A%2E+M%2E%3B+Graybill%2C+E+A%2E%3B+and+Boes%2C+D%2E+C%2E+%281974%29%2E+Introduction+to+the+Theory+of+Statistics%2E+McGraw+Hill%2E) 8. Schvaneveldt, Roger, ed. (1990). Pathfinder Associative Networks: Studies in Knowledge Organization. Ablex. [find](https://scholar.google.com/scholar?q=Schvaneveldt%2C+Roger%2C+ed%2E+%281990%29%2E+Pathfinder+Associative+Networks%3A+Studies+in+Knowledge+Organization%2E+Ablex%2E) 9. Salton, Gerald, and McGill, M. J. (1983). Introduction to Modern Information Retrieval. McGraw Hill. [find](https://scholar.google.com/scholar?q=Salton%2C+Gerald%2C+and+McGill%2C+M%2E+J%2E+%281983%29%2E+Introduction+to+Modern+Information+Retrieval%2E+McGraw+Hill%2E)