---
title: "Baselines and Bigrams — Wang 2012"
description: "Naive Bayes and linear SVMs are common baselines in text classification, but their accuracy depends strongly on the model variant, the feature set and…"
authors: ["Sida Wang", "Christopher Manning"]
year: 2012
venue: "Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/P12-2018/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2012-baselines-bigrams-simple-good-sentiment/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2012-baselines-bigrams-simple-good-sentiment/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2012](https://md.research.elamur.ai/free/en/papers/2012/) / Baselines and Bigrams: Simple, Good Sentiment and Topic Classification

# Baselines and Bigrams: Simple, Good Sentiment and Topic Classification

**Authors:** [Sida Wang](https://md.research.elamur.ai/free/en/authors/sida-wang/), [Christopher Manning](https://md.research.elamur.ai/free/en/authors/christopher-manning/) · **Year:** 2012 · **Venue:** Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P12-2018/) · [PDF](https://aclanthology.org/P12-2018.pdf) · [OpenAlex](https://openalex.org/W2154359981)

**Citations:** 944 citations (openalex-snapshot · 2026-10-14) · [172](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Naive Bayes and linear SVMs are common baselines in text classification, but their accuracy depends strongly on the model variant, the feature set and the dataset. Many published comparisons therefore rely on weak versions of them. Wang and Manning test these choices systematically on nine binary tasks. Six are sentiment or subjectivity tasks with short snippets or full reviews (RT-s, MPQA, CR, Subj., RT-2k, IMDB), and three are 20-newsgroups topic pairs. They compare binarized multinomial NB, an L2-regularized L2-loss SVM (LIBLINEAR) and a new hybrid called NBSVM, each with unigrams or unigrams plus bigrams. NBSVM is an SVM trained on features scaled by NB log-count ratios, with its weights pulled toward their mean magnitude. The main findings are:
- Bigrams reliably help sentiment classification.
- MNB beats SVM on short snippets, while SVM wins on long reviews.
- NBSVM is strong everywhere and often matches or exceeds more complex published systems, such as recursive autoencoders and tree-based CRFs.

This matters because it gives a cheap, reproducible baseline that sophisticated models must clearly beat.

## Key points
- **Results:** NBSVM-bi scores 91.22% on IMDB, 89.45% on RT-2k, 79.4% on RT-s and 81.8% on CR. These match or exceed earlier systems that use lexicons, reversal rules, parse trees or Wikipedia pretraining.
- **Snippets versus long reviews:** On snippets, MNB outperforms the linear SVM on every dataset except MPQA, even with roughly 9k training examples. On the long-review sets RT-2k and IMDB, the SVM is clearly stronger than MNB.
- **Bigrams:** Adding bigrams always improved the sentiment results. On the newsgroup topic tasks the effect was mixed and small, and adding trigrams slightly hurt.
- **Settings:** Every reported run shares one set of hyperparameters: α = 1, with NBSVM using C = 1 and β = 0.25 and the SVM using C = 0.1. No stopword lists, lexicons or external resources are used.
- **NB variants:** Multivariate Bernoulli NB is less stable than MNB and can be up to 10% worse. Binarizing counts gives about 1% over raw counts on long documents and makes little difference on snippets.

## Interesting ideas
- **Generative weights as discriminative features:** Multiplying binary features by NB log-count ratios before training a discriminative model is a cheap hybrid. It could boost any linear classifier, for example on low-resource Ukrainian text data.
- **Interpolation as regularization:** Mixing the learned SVM weights with a constant equal to their average magnitude means the model follows NB unless the SVM is confident. Values of β between ¼ and ½ were robust, so a similar shrinkage trick could stabilize other models.
- **Strong baselines first:** Tuning simple bag-of-features baselines properly can overturn claims that structure-sensitive models are required. New methods for sentiment or classification should be benchmarked against such tuned baselines.
- **Report significance thresholds per dataset:** Listing the accuracy gap needed for significance at p \< 0.05 for each dataset makes it obvious which improvements are real. This is a good habit for any benchmark table.
- **Choose features by task type:** Bigrams capture modified words such as negated verbs and adjectives, which matters for sentiment but much less for topic labelling. Feature choices should therefore follow the task rather than habit.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

172 papers of our library cite it (of the 140884 whose bibliographies we read); 18 of them have a page here, 154 not yet.

- [CITE: A Corpus of Image-Text Discourse Relations](https://md.research.elamur.ai/free/en/papers/alikhani-2019-cite-corpus-image-text-discourse/)
- [Applying Transformers and Aspect-based Sentiment Analysis approaches on Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/ataei-2020-applying-transformers-aspect-based-sentiment/)
- [Deep Unordered Composition Rivals Syntactic Methods for Text Classification](https://md.research.elamur.ai/free/en/papers/iyyer-2015-deep-unordered-composition-rivals-syntactic/)
- [Effective Use of Word Order for Text Categorization with Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/johnson-2015-effective-use-word-order-text/)
- [FastText.zip: Compressing text classification models](https://md.research.elamur.ai/free/en/papers/joulin-2016-fasttext-zip-compressing-text-classification/)
- [Recurrent Convolutional Neural Networks for Discourse Compositionality](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-convolutional-neural-networks-discourse/)
- [Convolutional Neural Networks for Sentence Classification](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/)
- [Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer](https://md.research.elamur.ai/free/en/papers/li-2018-delete-retrieve-generate-simple-approach/)
- [Adversarial Training Methods for Semi-Supervised Text Classification](https://md.research.elamur.ai/free/en/papers/miyato-2016-adversarial-training-methods-semi-supervised/)
- [Representation Learning with Contrastive Predictive Coding](https://md.research.elamur.ai/free/en/papers/oord-2018-representation-learning-contrastive-predictive-coding/)
- [A Transformer-based approach to Irony and Sarcasm detection](https://md.research.elamur.ai/free/en/papers/potamias-2019-transformer-based-approach-irony-sarcasm/)
- [Learning to Generate Reviews and Discovering Sentiment](https://md.research.elamur.ai/free/en/papers/radford-2017-learning-generate-reviews-discovering-sentiment/)
- [Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification](https://md.research.elamur.ai/free/en/papers/tang-2014-learning-sentiment-specific-word-embedding/)
- [Document Modeling with Gated Recurrent Neural Network for Sentiment Classification](https://md.research.elamur.ai/free/en/papers/tang-2015-document-modeling-gated-recurrent-neural/)
- [Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment](https://md.research.elamur.ai/free/en/papers/tsuchiya-2018-performance-impact-caused-hidden-bias/)
- [Hierarchical Attention Networks for Document Classification](https://md.research.elamur.ai/free/en/papers/yang-2016-hierarchical-attention-networks-document-classification/)
- [Depression and Self-Harm Risk Assessment in Online Forums](https://md.research.elamur.ai/free/en/papers/yates-2017-depression-self-harm-risk-assessment/)
- [Knowledge-Enriched Transformer for Emotion Detection in Textual Conversations](https://md.research.elamur.ai/free/en/papers/zhong-2019-knowledge-enriched-transformer-emotion-detection/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2012baselines,
  title = {{Baselines and Bigrams: Simple, Good Sentiment and Topic Classification}},
  author = {Wang, Sida and Manning, Christopher},
  booktitle = {Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2012},
  url = {https://aclanthology.org/P12-2018/}
}
```

- [← previous Automatic MT Error Analysis: Hjerson Helping Addicter](https://md.research.elamur.ai/free/en/papers/berka-2012-automatic-mt-error-analysis-hjerson/)
- [next → Batch Tuning Strategies for Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/cherry-2012-batch-tuning-strategies-statistical-machine/)
- [↑ back to papers of 2012](https://md.research.elamur.ai/free/en/papers/2012/)

## References

20 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. R. Collobert and J. Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of ICML. [find](https://scholar.google.com/scholar?q=R%2E+Collobert+and+J%2E+Weston%2E+2008%2E+A+unified+architecture+for+natural+language+processing%3A+Deep+neural+networks+with+multitask+learning%2E+In+Proceedings+of+ICML%2E)
2. George E. Dahl, Ryan P. Adams, and Hugo Larochelle. 2012. Training restricted boltzmann machines on word observations. arXiv:1202.5695v1 [cs.LG]. — [arXiv:1202.5695](https://arxiv.org/abs/1202.5695) [find](https://scholar.google.com/scholar?q=George+E%2E+Dahl%2C+Ryan+P%2E+Adams%2C+and+Hugo+Larochelle%2E+2012%2E+Training+restricted+boltzmann+machines+on+word+observations%2E+arXiv%3A1202%2E5695v1+%5Bcs%2ELG%5D%2E+%E2%80%94+%5BarXiv%3A1202%2E5695%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1202%2E5695%29)
3. Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. LIBLINEAR: A library for large linear classification. Journal of Machine Learning Research, 9:1871–1874, June. [find](https://scholar.google.com/scholar?q=Rong%2DEn+Fan%2C+Kai%2DWei+Chang%2C+Cho%2DJui+Hsieh%2C+Xiang%2DRui+Wang%2C+and+Chih%2DJen+Lin%2E+2008%2E+LIBLINEAR%3A+A+library+for+large+linear+classification%2E+Journal+of+Machine+Learning+Research%2C+9%3A1871%E2%80%931874%2C+June%2E)
4. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings ACM SIGKDD, pages 168–177. [find](https://scholar.google.com/scholar?q=Minqing+Hu+and+Bing+Liu%2E+2004%2E+Mining+and+summarizing+customer+reviews%2E+In+Proceedings+ACM+SIGKDD%2C+pages+168%E2%80%93177%2E)
5. Alistair Kennedy and Diana Inkpen. 2006. Sentiment classification of movie reviews using contextual valence shifters. Computational Intelligence, 22. [find](https://scholar.google.com/scholar?q=Alistair+Kennedy+and+Diana+Inkpen%2E+2006%2E+Sentiment+classification+of+movie+reviews+using+contextual+valence+shifters%2E+Computational+Intelligence%2C+22%2E)
6. Simon Lacoste-Julien, Fei Sha, and Michael I. Jordan. 2008. DiscLDA: Discriminative learning for dimensionality reduction and classification. In Proceedings of NIPS, pages 897–904. [find](https://scholar.google.com/scholar?q=Simon+Lacoste%2DJulien%2C+Fei+Sha%2C+and+Michael+I%2E+Jordan%2E+2008%2E+DiscLDA%3A+Discriminative+learning+for+dimensionality+reduction+and+classification%2E+In+Proceedings+of+NIPS%2C+pages+897%E2%80%93904%2E)
7. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Andrew+L%2E+Maas%2C+Raymond+E%2E+Daly%2C+Peter+T%2E+Pham%2C+Dan+Huang%2C+Andrew+Y%2E+Ng%2C+and+Christopher+Potts%2E+2011%2E+Learning+word+vectors+for+sentiment+analysis%2E+In+Proceedings+of+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/)
8. Justin Martineau and Tim Finin. 2009. Delta tfidf: An improved feature space for sentiment analysis. In Proceedings of ICWSM. [find](https://scholar.google.com/scholar?q=Justin+Martineau+and+Tim+Finin%2E+2009%2E+Delta+tfidf%3A+An+improved+feature+space+for+sentiment+analysis%2E+In+Proceedings+of+ICWSM%2E)
9. Andrew McCallum and Kamal Nigam. 1998. A comparison of event models for naive bayes text classification. In AAAI-98 Workshop, pages 41–48. [find](https://scholar.google.com/scholar?q=Andrew+McCallum+and+Kamal+Nigam%2E+1998%2E+A+comparison+of+event+models+for+naive+bayes+text+classification%2E+In+AAAI%2D98+Workshop%2C+pages+41%E2%80%9348%2E)
10. Vangelis Metsis, Ion Androutsopoulos, and Georgios Paliouras. 2006. Spam filtering with naive bayes - which naive bayes? In Proceedings of CEAS. [find](https://scholar.google.com/scholar?q=Vangelis+Metsis%2C+Ion+Androutsopoulos%2C+and+Georgios+Paliouras%2E+2006%2E+Spam+filtering+with+naive+bayes+%2D+which+naive+bayes%3F+In+Proceedings+of+CEAS%2E)
11. Karo Moilanen and Stephen Pulman. 2007. Sentiment composition. In Proceedings of RANLP, pages 378– 382, September 27-29. [find](https://scholar.google.com/scholar?q=Karo+Moilanen+and+Stephen+Pulman%2E+2007%2E+Sentiment+composition%2E+In+Proceedings+of+RANLP%2C+pages+378%E2%80%93+382%2C+September+27%2D29%2E)
12. Tetsuji Nakagawa, Kentaro Inui, and Sadao Kurohashi. 2010. Dependency tree-based sentiment classification using CRFs with hidden variables. In Proceedings of ACL:HLT. [find](https://scholar.google.com/scholar?q=Tetsuji+Nakagawa%2C+Kentaro+Inui%2C+and+Sadao+Kurohashi%2E+2010%2E+Dependency+tree%2Dbased+sentiment+classification+using+CRFs+with+hidden+variables%2E+In+Proceedings+of+ACL%3AHLT%2E)
13. Andrew Y Ng and Michael I Jordan. 2002. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. In Proceedings of NIPS, volume 2, pages 841–848. [find](https://scholar.google.com/scholar?q=Andrew+Y+Ng+and+Michael+I+Jordan%2E+2002%2E+On+discriminative+vs%2E+generative+classifiers%3A+A+comparison+of+logistic+regression+and+naive+bayes%2E+In+Proceedings+of+NIPS%2C+volume+2%2C+pages+841%E2%80%93848%2E)
14. Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Bo+Pang+and+Lillian+Lee%2E+2004%2E+A+sentimental+education%3A+Sentiment+analysis+using+subjectivity+summarization+based+on+minimum+cuts%2E+In+Proceedings+of+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/pang-2004-sentimental-education-sentiment-analysis-subjectivity/)
15. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Bo+Pang+and+Lillian+Lee%2E+2005%2E+Seeing+stars%3A+Exploiting+class+relationships+for+sentiment+categorization+with+respect+to+rating+scales%2E+In+Proceedings+of+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/pang-2005-seeing-stars-exploiting-class-relationships/)
16. Jason D. Rennie, Lawrence Shih, Jaime Teevan, and David R. Karger. 2003. Tackling the poor assumptions of naive bayes text classifiers. In Proceedings of ICML, pages 616–623. [find](https://scholar.google.com/scholar?q=Jason+D%2E+Rennie%2C+Lawrence+Shih%2C+Jaime+Teevan%2C+and+David+R%2E+Karger%2E+2003%2E+Tackling+the+poor+assumptions+of+naive+bayes+text+classifiers%2E+In+Proceedings+of+ICML%2C+pages+616%E2%80%93623%2E)
17. Greg Schohn and David Cohn. 2000. Less is more: Active learning with support vector machines. In Proceedings of ICML, pages 839–846. [find](https://scholar.google.com/scholar?q=Greg+Schohn+and+David+Cohn%2E+2000%2E+Less+is+more%3A+Active+learning+with+support+vector+machines%2E+In+Proceedings+of+ICML%2C+pages+839%E2%80%93846%2E)
18. Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011. Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Jeffrey+Pennington%2C+Eric+H%2E+Huang%2C+Andrew+Y%2E+Ng%2C+and+Christopher+D%2E+Manning%2E+2011%2E+Semi%2DSupervised+Recursive+Autoencoders+for+Predicting+Sentiment+Distributions%2E+In+Proceedings+of+EMNL) [our card](https://md.research.elamur.ai/free/en/papers/socher-2011-semi-supervised-recursive-autoencoders-predicting/)
19. Casey Whitelaw, Navendu Garg, and Shlomo Argamon. 2005. Using appraisal taxonomies for sentiment analysis. In Proceedings of CIKM-05. [find](https://scholar.google.com/scholar?q=Casey+Whitelaw%2C+Navendu+Garg%2C+and+Shlomo+Argamon%2E+2005%2E+Using+appraisal+taxonomies+for+sentiment+analysis%2E+In+Proceedings+of+CIKM%2D05%2E)
20. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2-3):165–210. [find](https://scholar.google.com/scholar?q=Janyce+Wiebe%2C+Theresa+Wilson%2C+and+Claire+Cardie%2E+2005%2E+Annotating+expressions+of+opinions+and+emotions+in+language%2E+Language+Resources+and+Evaluation%2C+39%282%2D3%29%3A165%E2%80%93210%2E)