--- title: "Text Categorization with Support Vector — Joachims 1998" description: "Joachims investigates whether Support Vector Machines suit supervised text categorization, where each category is treated as its own binary decision over…" authors: ["Thorsten Joachims"] year: 1998 venue: "ECML 1998" source: "https://link.springer.com/chapter/10.1007/BFb0026683" doi: "10.1007/bfb0026683" licence: "author copy (cs.cornell.edu, LS-8 tech report 23) — read only" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/joachims-1998-text-categorization-support-vector-machines/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/joachims-1998-text-categorization-support-vector-machines/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [1998](https://md.research.elamur.ai/free/en/papers/1998/) / Text Categorization with Support Vector Machines: Learning with Many Relevant Features # Text Categorization with Support Vector Machines: Learning with Many Relevant Features **Authors:** [Thorsten Joachims](https://md.research.elamur.ai/free/en/authors/thorsten-joachims/) · **Year:** 1998 · **Venue:** ECML 1998 · **Licence:** author copy (cs.cornell.edu, LS-8 tech report 23) — read only **Links:** [Publisher's page](https://doi.org/10.1007/bfb0026683) · [PDF](https://www.cs.cornell.edu/people/tj/publications/joachims_98a.pdf) · [DOI 10.1007/bfb0026683](https://doi.org/10.1007/bfb0026683) · [OpenAlex](https://openalex.org/W2149684865) **Citations:** 7999 citations (openalex-snapshot · 2026-10-14) · [303](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Joachims investigates whether Support Vector Machines suit supervised text categorization, where each category is treated as its own binary decision over bag-of-words vectors with tf-idf weighting. He first argues from the properties of text. Feature spaces are huge (over 10,000 word stems), but almost no features are useless: a naive Bayes model trained only on the lowest-ranked words still beats chance. Document vectors are sparse, and many tasks are linearly separable. Because SVMs control capacity through the margin rather than the number of features, they should cope with this setting without aggressive feature selection. The experiments use Reuters-21578 (ModApte split) and the Ohsumed corpus. SVMs with polynomial and RBF kernels, trained with SVM light, are compared against naive Bayes, Rocchio, k-NN and C4.5, with each baseline given generous tuning. SVMs win on both collections for every parameter choice tested. A VC-dimension-based heuristic also selects good kernel settings automatically. The work established SVMs as a robust, low-tuning standard for text classification. ## Key points - **Data:** On Reuters ModApte there are 9603 training and 3299 test documents, 90 categories and 9962 terms. On Ohsumed (1991 abstracts) there are 10,000 training and 10,000 test documents, 23 MeSH "diseases" categories and 15561 terms. - **Reuters results** (microaveraged precision/recall-breakeven): - Naive Bayes reaches 72.0, Rocchio 79.9, C4.5 79.4 and k-NN 82.3. - With automatically chosen parameters, the polynomial SVM reaches 86.0 and the RBF SVM 86.4. - The RBF SVM beats k-NN on 63 of 90 categories, with 19 ties. - **Ohsumed results:** - k-NN reaches 59.1, naive Bayes 57.0, Rocchio 56.6 and C4.5 50.0 (C4.5 overfits with more than 500 features). - The SVMs reach 65.9 (polynomial) and 66.0 (RBF). - The RBF SVM beats k-NN on all 23 categories. - **Robustness:** No overfitting appears even with degree-5 polynomials using all features, so separate feature selection becomes unnecessary. - **Cost:** Training time is similar to C4.5 but slower than naive Bayes, Rocchio and k-NN. Classification is faster than k-NN. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Computing & GPUs](https://md.research.elamur.ai/free/en/shelves/computing/) ## Cited by Cited by 303 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 283 not yet. - [Noun Phrase Chunking in Hebrew: Influence of Lexical and Morphological Features](https://md.research.elamur.ai/free/en/papers/aclbulk-2006-goldberg-noun-phrase-chunking-hebrew-influence/) - [A Comparative Study of Feature Selection Methods for Dialectal Arabic Sentiment Classification Using Support Vector Machine](https://md.research.elamur.ai/free/en/papers/al-2019-harbi-comparative-study-feature-selection-methods/) - [Modeling Local Coherence: An Entity-Based Approach](https://md.research.elamur.ai/free/en/papers/barzilay-2008-modeling-local-coherence-entity-based/) - [The Form is the Substance: Classification of Genres in Text](https://md.research.elamur.ai/free/en/papers/dewdney-2001-form-is-substance-classification-genres/) - [Modeling Global Syntactic Variation in English Using Dialect Classification](https://md.research.elamur.ai/free/en/papers/dunn-2019-modeling-global-syntactic-variation-english/) - [Finding Variants for Construction-Based Dialectometry: A Corpus-Based Approach to Regional CxGs](https://md.research.elamur.ai/free/en/papers/dunn-2021-finding-variants-construction-based-dialectometry/) - [FastText.zip: Compressing text classification models](https://md.research.elamur.ai/free/en/papers/joulin-2016-fasttext-zip-compressing-text-classification/) - [Classifying Dialogue Acts in One-on-One Live Chats](https://md.research.elamur.ai/free/en/papers/kim-2010-classifying-dialogue-acts-one-one/) - [Chunking with Support Vector Machines](https://md.research.elamur.ai/free/en/papers/kudo-2001-chunking-support-vector-machines/) - [RCV1: A New Benchmark Collection for Text Categorization Research](https://md.research.elamur.ai/free/en/papers/lewis-2004-rcv1-benchmark-collection-text-categorization/) - [Multilingual Annotation and Disambiguation of Discourse Connectives for Machine Translation](https://md.research.elamur.ai/free/en/papers/meyer-2011-multilingual-annotation-disambiguation-discourse-connectives/) - [Finding Deceptive Opinion Spam by Any Stretch of the Imagination](https://md.research.elamur.ai/free/en/papers/ott-2011-finding-deceptive-opinion-spam-any/) - [Thumbs up? Sentiment Classification using Machine Learning Techniques](https://md.research.elamur.ai/free/en/papers/pang-2002-thumbs-up-sentiment-classification-machine/) - [Towards Robust Semantic Role Labeling](https://md.research.elamur.ai/free/en/papers/pradhan-2008-robust-semantic-role-labeling/) - [Reading Level Assessment Using Support Vector Machines and Statistical Language Models](https://md.research.elamur.ai/free/en/papers/schwarm-2005-reading-level-assessment-support-vector/) - [Hierarchical Attention Networks for Document Classification](https://md.research.elamur.ai/free/en/papers/yang-2016-hierarchical-attention-networks-document-classification/) - [A New Dataset and Method for Automatically Grading ESOL Texts](https://md.research.elamur.ai/free/en/papers/yannakoudakis-2011-new-dataset-method-automatically-grading/) - [Using “Annotator Rationales” to Improve Machine Learning for Text Categorization](https://md.research.elamur.ai/free/en/papers/zaidan-2007-annotator-rationales-improve-machine-learning/) - [A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification](https://md.research.elamur.ai/free/en/papers/zhang-2017-sensitivity-analysis-practitioners-guide-convolutional/) - [Exploring Various Knowledge in Relation Extraction](https://md.research.elamur.ai/free/en/papers/zhou-2005-exploring-various-knowledge-relation-extraction/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{joachims1998text, title = {{Text Categorization with Support Vector Machines: Learning with Many Relevant Features}}, author = {Joachims, Thorsten}, booktitle = {ECML 1998}, year = {1998}, doi = {10.1007/bfb0026683}, url = {https://doi.org/10.1007/bfb0026683} } ``` - [← previous Syntactic and Semantic Transfer with F-Structures](https://md.research.elamur.ai/free/en/papers/dorna-1998-syntactic-semantic-transfer-f-structures/) - [next → The Anatomy of a Large-Scale Hypertextual Web Search Engine](https://md.research.elamur.ai/free/en/papers/brin-1998-anatomy-large-scale-hypertextual-web/) - [↑ back to papers of 1998](https://md.research.elamur.ai/free/en/papers/1998/) ## References 11 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. C. Cortes and V. Vapnik. Support-vector networks. Machine Learning , 20:273{ 297, November 1995. [find](https://scholar.google.com/scholar?q=C%2E+Cortes+and+V%2E+Vapnik%2E+Support%2Dvector+networks%2E+Machine+Learning+%2C+20%3A273%7B+297%2C+November+1995%2E) 2. T. Joachims. A probabilistic analysis of the rocchio algorithm with t df for text categorization. In International Conference on Machine Learning (ICML) , 1997. [find](https://scholar.google.com/scholar?q=T%2E+Joachims%2E+A+probabilistic+analysis+of+the+rocchio+algorithm+with+t+df+for+text+categorization%2E+In+International+Conference+on+Machine+Learning+%28ICML%29+%2C+1997%2E) 3. T. Joachims. Text categorization with support vector machines: Learning with many relevant features. Technical Report 23, Universitat Dortmund, LS VIII, 1997. [find](https://scholar.google.com/scholar?q=T%2E+Joachims%2E+Text+categorization+with+support+vector+machines%3A+Learning+with+many+relevant+features%2E+Technical+Report+23%2C+Universit%7Fat+Dortmund%2C+LS+VIII%2C+1997%2E) 4. J. Kivinen, M. Warmuth, and P. Auer. The perceptron algorithm vs. winnow: Linear vs. logarithmic mistake bounds when few input variables are relevant. In Conference on Computational Learning Theory , 1995. [find](https://scholar.google.com/scholar?q=J%2E+Kivinen%2C+M%2E+Warmuth%2C+and+P%2E+Auer%2E+The+perceptron+algorithm+vs%2E+winnow%3A+Linear+vs%2E+logarithmic+mistake+bounds+when+few+input+variables+are+relevant%2E+In+Conference+on+Computational+Learning+Theory+%2C+) 5. T. Mitchell. Machine Learning . McGraw-Hill, 1997. [find](https://scholar.google.com/scholar?q=T%2E+Mitchell%2E+Machine+Learning+%2E+McGraw%2DHill%2C+1997%2E) 6. J. R. Quinlan. C4.5: Programs for Machine Learning . Morgan Kaufmann, 1993. [find](https://scholar.google.com/scholar?q=J%2E+R%2E+Quinlan%2E+C4%2E5%3A+Programs+for+Machine+Learning+%2E+Morgan+Kaufmann%2C+1993%2E) [our card](https://md.research.elamur.ai/free/en/papers/quinlan-1993-c45-programs-machine-learning/) 7. J. Rocchio. Relevance feedback in information retrieval. In G. Salton, editor, The SMART Retrieval System: Experiments in Automatic Document Processing , pages 313{323. Prentice-Hall Inc., 1971. [find](https://scholar.google.com/scholar?q=J%2E+Rocchio%2E+Relevance+feedback+in+information+retrieval%2E+In+G%2E+Salton%2C+editor%2C+The+SMART+Retrieval+System%3A+Experiments+in+Automatic+Document+Processing+%2C+pages+313%7B323%2E+Prentice%2DHall+Inc%2E%2C+1971%2E) 8. G. Salton and C. Buckley. Term weighting approaches in automatic text retrieval. Information Processing and Management , 24(5):513{523, 1988. [find](https://scholar.google.com/scholar?q=G%2E+Salton+and+C%2E+Buckley%2E+Term+weighting+approaches+in+automatic+text+retrieval%2E+Information+Processing+and+Management+%2C+24%285%29%3A513%7B523%2C+1988%2E) 9. Vladimir N. Vapnik. The Nature of Statistical Learning Theory . Springer, New York, 1995. [find](https://scholar.google.com/scholar?q=Vladimir+N%2E+Vapnik%2E+The+Nature+of+Statistical+Learning+Theory+%2E+Springer%2C+New+York%2C+1995%2E) 10. Y. Yang. An evaluation of statistical approaches to text categorization. Technical Report CMU-CS-97-127, Carnegie Mellon University, April 1997. [find](https://scholar.google.com/scholar?q=Y%2E+Yang%2E+An+evaluation+of+statistical+approaches+to+text+categorization%2E+Technical+Report+CMU%2DCS%2D97%2D127%2C+Carnegie+Mellon+University%2C+April+1997%2E) 11. Y. Yang and J. Pedersen. A comparative study on feature selection in text categorization. In International Conference on Machine Learning (ICML) , 1997. This article was processed using the L A TEX macro package with LLNCS style [find](https://scholar.google.com/scholar?q=Y%2E+Yang+and+J%2E+Pedersen%2E+A+comparative+study+on+feature+selection+in+text+categorization%2E+In+International+Conference+on+Machine+Learning+%28ICML%29+%2C+1997%2E+This+article+was+processed+using+the+L+A+TEX+ma)