---
title: "A Comparison of Event Models for Naive Bayes Text Classification"
description: "Two different probabilistic classifiers were both being called \"naive Bayes\" in text categorization… 22 references, cited by 151 papers here."
authors: ["Andrew McCallum", "Kamal Nigam"]
year: 1998
venue: "AAAI-98 Workshop on Learning for Text Categorization"
source: "https://aaai.org/papers/041-ws98-05-007/"
licence: "author copy (kamalnigam.com) — read only"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mccallum-1998-comparison-event-models-naive-bayes/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mccallum-1998-comparison-event-models-naive-bayes/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [1998](https://md.research.elamur.ai/free/en/papers/1998/) / A Comparison of Event Models for Naive Bayes Text Classification

# A Comparison of Event Models for Naive Bayes Text Classification

**Authors:** [Andrew McCallum](https://md.research.elamur.ai/free/en/authors/andrew-mccallum/) and [Kamal Nigam](https://md.research.elamur.ai/free/en/authors/kamal-nigam/) · **Year:** 1998 · **Venue:** AAAI-98 Workshop on Learning for Text Categorization · **Licence:** author copy (kamalnigam.com) — read only

**Links:** [Publisher's page](https://aaai.org/papers/041-ws98-05-007/) · [PDF](http://www.kamalnigam.com/papers/multinomial-aaaiws98.pdf) · [OpenAlex](https://openalex.org/W1550206324)

**Citations:** 3,173 citations (OpenAlex, 2026-10-14) · [151](https://md.research.elamur.ai/free/en/papers/mccallum-1998-comparison-event-models-naive-bayes/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Two different probabilistic classifiers were both being called "naive Bayes" in text categorization research. One is the multi-variate Bernoulli model, which treats a document as a binary vector recording which vocabulary words appear. The other is the multinomial model, a unigram language model that counts how often each word occurs. McCallum and Nigam give a shared mixture-model derivation of both, showing where their assumptions and parameter estimates differ. They then test both on five corpora drawn from Web pages, UseNet postings and Reuters newswire, varying vocabulary size through mutual-information feature selection. The Bernoulli variant sometimes wins when the vocabulary is very small. The multinomial generally pulls ahead as vocabulary grows and almost always wins when each model is given its best vocabulary size. Averaged over datasets, it reduces error by 27%. This matters because many real categories cover diverse topics and need large vocabularies, so the paper gives practitioners a clear reason to prefer word counts over presence/absence features.

## Key points
- **The two models:**
  - The Bernoulli model multiplies probabilities over the whole vocabulary, including explicit terms for words missing from a document.
  - The multinomial model uses only the words that appear, weighted by their counts, and includes document length as the number of draws.
- **Smoothing and priors:** Both models smooth word estimates with a Laplacean (add-one) prior. Class priors are maximum-likelihood estimates.
- **Feature selection:** Words are ranked by average mutual information with the class variable to shrink the vocabulary.
- **Main result:** The multinomial cuts error by 27% on average across the five corpora, and by more than 50% in some cases. Data includes Reuters, Yahoo! and the Market Guide Industry Sector hierarchy.
- **Reuters behaviour:** Reuters tasks often peak with small vocabularies. The authors' scores there exceed those of Joachims (1998), who mixed event models between feature selection and classification.

## Interesting ideas
- **Compare models at their own best settings.** Each model is evaluated at its own optimal vocabulary size, not only at a shared fixed size. This gives a fairer head-to-head comparison for any method that depends on a tunable feature budget.
- **Token normalization combined with counts.** All digit strings are collapsed into one token. Its presence is useless, since every article is dated, but its frequency separates earnings reports from other news. This trick could help with Ukrainian trade or financial texts.
- **Keep non-text features out of the word vocabulary.** Metadata features such as a message's recipient count should not compete with words for the same probability mass. They can be modelled as separate Bernoulli variables next to the word multinomial, optionally with a weight, which is a clean way to fuse metadata with text.
- **Absent words still carry evidence.** In the multinomial, a missing high-probability class word lowers that class's likelihood implicitly. This is a useful argument against assuming explicit absence features are always needed.
- **Model document length explicitly.** Making length part of the generative story, and possibly class-dependent, matters for corpora with very uneven document lengths. Under the Bernoulli model, long documents tend to switch on many words regardless of their class.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 539 works · 179,533 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

💡 Concepts (1): [💡 Text classification](https://md.research.elamur.ai/free/en/concepts/text-classification/) · 50 works

## Cite

Cite as: Andrew McCallum and Kamal Nigam. 1998. A Comparison of Event Models for Naive Bayes Text Classification. AAAI-98 Workshop on Learning for Text Categorization.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mccallum1998comparison,
  title = {{A Comparison of Event Models for Naive Bayes Text Classification}},
  author = {Andrew McCallum and Kamal Nigam},
  booktitle = {AAAI-98 Workshop on Learning for Text Categorization},
  year = {1998},
  url = {https://aaai.org/papers/041-ws98-05-007/}
}
```

- [← previous The Berkeley FrameNet Project](https://md.research.elamur.ai/free/en/papers/baker-1998-berkeley-framenet/)
- [next → The Berkeley FrameNet Project](https://md.research.elamur.ai/free/en/papers/baker-1998-berkeley-framenet-project/)
- [↑ back to papers of 1998](https://md.research.elamur.ai/free/en/papers/1998/)

## References

22 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. John Wiley, 1991. M. Craven, D. DiPasquo, D. Freitag, A. McCallum, T. Mitchell, K. Nigam, and S. Slattery. Learning to extract symbolic knowledge from the World Wide Web. In AAAI-98, 1998. P. Domingos and M. Pazzani. On the optimalit… [find](https://scholar.google.com/scholar?q=Thomas+M%2E+Cover+and+Joy+A%2E+Thomas%2E+Elements+of+Information+Theory%2E+John+Wiley%2C+1991%2E+M%2E+Craven%2C+D%2E+DiPasquo%2C+D%2E+Freitag%2C+A%2E+McCallum%2C+T%2E+Mitchell%2C+K%2E+Nigam%2C+and+S%2E+Slattery%2E+Learning+to+extract+symbol)
2. Jerome H. Friedman. On bias, variance, 0/1 - loss, and the curse-of-dimensionality. Data Mining and Knowledge Discovery, 1:55–77, 1997. Louise Guthrie and Elbert Walker. Document classification by machine: Theory and practice. In Proceedings of COLING-94, 1994. [find](https://scholar.google.com/scholar?q=Jerome+H%2E+Friedman%2E+On+bias%2C+variance%2C+0%2F1+%2D+loss%2C+and+the+curse%2Dof%2Ddimensionality%2E+Data+Mining+and+Knowledge+Discovery%2C+1%3A55%E2%80%9377%2C+1997%2E+Louise+Guthrie+and+Elbert+Walker%2E+Document+classification+by+mac)
3. Thorsten Joachims. A probabilistic analysis of the Rocchio algorithm with TFIDF for text categorization. In ICML-97, 1997. [find](https://scholar.google.com/scholar?q=Thorsten+Joachims%2E+A+probabilistic+analysis+of+the+Rocchio+algorithm+with+TFIDF+for+text+categorization%2E+In+ICML%2D97%2C+1997%2E)
4. Thorsten Joachims. [Text categorization with Support Vector Machines: Learning with many relevant features](https://md.research.elamur.ai/free/en/papers/joachims-1998-text-categorization-support-vector-machines/). In ECML-98, 1998.
5. T. Kalt and W. B. Croft. A new probabilistic model of text classification and retrieval. Technical Report IR-78, University of Massachusetts Center for Intelligent Information Retrieval, 1996. [http://ciir.cs.umass.edu/publications/index.shtml](http://ciir.cs.umass.edu/publications/index.shtml). [↗](http://ciir.cs.umass.edu/publications/index.shtml)
6. Daphne Koller and Mehran Sahami. Hierarchically classifying documents using very few words. In Proceedings of the Fourteenth International Conference on Machine Learning, 1997. [find](https://scholar.google.com/scholar?q=Daphne+Koller+and+Mehran+Sahami%2E+Hierarchically+classifying+documents+using+very+few+words%2E+In+Proceedings+of+the+Fourteenth+International+Conference+on+Machine+Learning%2C+1997%2E)
7. Pat Langley, Wayne Iba, and Kevin Thompson. An analysis of Bayesian classifiers. In AAAI-92, 1992. [find](https://scholar.google.com/scholar?q=Pat+Langley%2C+Wayne+Iba%2C+and+Kevin+Thompson%2E+An+analysis+of+Bayesian+classifiers%2E+In+AAAI%2D92%2C+1992%2E)
8. Leah S. Larkey and W. Bruce Croft. Combining classifiers in text categorization. In SIGIR-96, 1996. [find](https://scholar.google.com/scholar?q=Leah+S%2E+Larkey+and+W%2E+Bruce+Croft%2E+Combining+classifiers+in+text+categorization%2E+In+SIGIR%2D96%2C+1996%2E)
9. D. Lewis and W. Gale. A sequential algorithm for training text classifiers. In SIGIR-94, 1994. [find](https://scholar.google.com/scholar?q=D%2E+Lewis+and+W%2E+Gale%2E+A+sequential+algorithm+for+training+text+classifiers%2E+In+SIGIR%2D94%2C+1994%2E)
10. David D. Lewis. An evaluation of phrasal and clustered representations on a text categorization task. In SIGIR-92, 1992. [find](https://scholar.google.com/scholar?q=David+D%2E+Lewis%2E+An+evaluation+of+phrasal+and+clustered+representations+on+a+text+categorization+task%2E+In+SIGIR%2D92%2C+1992%2E)
11. David Lewis. Naive (bayes) at forty: The independence asssumption in information retrieval. In ECML’98: Tenth European Conference On Machine Learning, 1998. [find](https://scholar.google.com/scholar?q=David+Lewis%2E+Naive+%28bayes%29+at+forty%3A+The+independence+asssumption+in+information+retrieval%2E+In+ECML%E2%80%9998%3A+Tenth+European+Conference+On+Machine+Learning%2C+1998%2E)
12. Hang Li and Kenji Yamanishi. Document classification using a finite mixture model. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics, 1997. [find](https://scholar.google.com/scholar?q=Hang+Li+and+Kenji+Yamanishi%2E+Document+classification+using+a+finite+mixture+model%2E+In+Proceedings+of+the+35th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+1997%2E)
13. Ray Liere and Prasad Tadepalli. Active learning with committees for text categorization. In AAAI-97, 1997. [find](https://scholar.google.com/scholar?q=Ray+Liere+and+Prasad+Tadepalli%2E+Active+learning+with+committees+for+text+categorization%2E+In+AAAI%2D97%2C+1997%2E)
14. Andrew McCallum, Ronald Rosenfeld, Tom Mitchell, and Andrew Ng. Improving text clasification by shrinkage in a hierarchy of classes. In ICML-98, 1998. [find](https://scholar.google.com/scholar?q=Andrew+McCallum%2C+Ronald+Rosenfeld%2C+Tom+Mitchell%2C+and+Andrew+Ng%2E+Improving+text+clasification+by+shrinkage+in+a+hierarchy+of+classes%2E+In+ICML%2D98%2C+1998%2E)
15. Tom M. Mitchell. Machine Learning. WCB/McGraw-Hill, 1997. [find](https://scholar.google.com/scholar?q=Tom+M%2E+Mitchell%2E+Machine+Learning%2E+WCB%2FMcGraw%2DHill%2C+1997%2E)
16. Kamal Nigam, Andrew McCallum, Sebastian Thrun, and Tom Mitchell. Learning to classify text from labeled and unlabeled documents. In AAAI-98, 1998. [find](https://scholar.google.com/scholar?q=Kamal+Nigam%2C+Andrew+McCallum%2C+Sebastian+Thrun%2C+and+Tom+Mitchell%2E+Learning+to+classify+text+from+labeled+and+unlabeled+documents%2E+In+AAAI%2D98%2C+1998%2E)
17. S. E. Robertson and K. Sparck-Jones. Relevance weighting of search terms. Journal of the American Society for Information Science, 27:129–146, 1976. [find](https://scholar.google.com/scholar?q=S%2E+E%2E+Robertson+and+K%2E+Sparck%2DJones%2E+Relevance+weighting+of+search+terms%2E+Journal+of+the+American+Society+for+Information+Science%2C+27%3A129%E2%80%93146%2C+1976%2E)
18. Mehran Sahami, Susan Dumais, David Heckerman, and Eric Horvitz. A bayesian approach to filtering junk e-mail. In AAAI-98 Workshop on Learning for Text Categorization, 1998. [find](https://scholar.google.com/scholar?q=Mehran+Sahami%2C+Susan+Dumais%2C+David+Heckerman%2C+and+Eric+Horvitz%2E+A+bayesian+approach+to+filtering+junk+e%2Dmail%2E+In+AAAI%2D98+Workshop+on+Learning+for+Text+Categorization%2C+1998%2E)
19. Mehran Sahami. Learning limited dependence Bayesian classifiers. In KDD-96: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, pages 335–338. AAAI Press, 1996. [find](https://scholar.google.com/scholar?q=Mehran+Sahami%2E+Learning+limited+dependence+Bayesian+classifiers%2E+In+KDD%2D96%3A+Proceedings+of+the+Second+International+Conference+on+Knowledge+Discovery+and+Data+Mining%2C+pages+335%E2%80%93338%2E+AAAI+Press%2C+1996%2E)
20. V. Vapnik. Estimations of dependences based on statistical data. Springer Publisher, 1982. [find](https://scholar.google.com/scholar?q=V%2E+Vapnik%2E+Estimations+of+dependences+based+on+statistical+data%2E+Springer+Publisher%2C+1982%2E)
21. Yiming Yang and Jan Pederson. Feature selection in statistical learning of text categorization. In ICML-97, 1997. acq Multinomial Multi-variate Bernoulli Precision/Recall Breakeven Point 1000 [find](https://scholar.google.com/scholar?q=Yiming+Yang+and+Jan+Pederson%2E+Feature+selection+in+statistical+learning+of+text+categorization%2E+In+ICML%2D97%2C+1997%2E+acq+Multinomial+Multi%2Dvariate+Bernoulli+Precision%2FRecall+Breakeven+Point+1000)
22. Vocabulary Size crude Multinomial Multi-variate Bernoulli Precision/Recall Breakeven Point 1000 Vocabulary Size grain Multinomial Multi-variate Bernoulli Precision/Recall Breakeven Point 1000 Vocabulary Size trade Multinomial Multi-variate Bernoulli Precision/Recall Breakeven Point 1000 Vocabulary S… [find](https://scholar.google.com/scholar?q=Vocabulary+Size+crude+Multinomial+Multi%2Dvariate+Bernoulli+Precision%2FRecall+Breakeven+Point+1000+Vocabulary+Size+grain+Multinomial+Multi%2Dvariate+Bernoulli+Precision%2FRecall+Breakeven+Point+1000+Vocabula)

*Updated 29 October 2026*