---
title: "Chunking with Support Vector Machines — Kudo 2001"
description: "The paper addresses chunking: splitting a token sequence into non-overlapping base phrases and assigning each a grammatical class."
authors: ["Taku Kudo", "Yuji Matsumoto"]
year: 2001
venue: "Second Meeting of the North American Chapter of the Association for Computational Linguistics"
source: "https://aclanthology.org/N01-1025/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kudo-2001-chunking-support-vector-machines/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kudo-2001-chunking-support-vector-machines/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2001](https://md.research.elamur.ai/free/en/papers/2001/) / Chunking with Support Vector Machines

# Chunking with Support Vector Machines

**Authors:** [Taku Kudo](https://md.research.elamur.ai/free/en/authors/taku-kudo/), [Yuji Matsumoto](https://md.research.elamur.ai/free/en/authors/yuji-matsumoto/) · **Year:** 2001 · **Venue:** Second Meeting of the North American Chapter of the Association for Computational Linguistics · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/N01-1025/) · [PDF](https://aclanthology.org/N01-1025.pdf) · [OpenAlex](https://openalex.org/W1988995507)

**Citations:** 520 citations (openalex-snapshot · 2026-10-14) · [127](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper addresses chunking: splitting a token sequence into non-overlapping base phrases and assigning each a grammatical class. Chunking is treated as tagging each token with a chunk label. The authors argue that HMM and Maximum Entropy models depend on hand-picked feature sets and feature combinations. SVMs instead generalize well in very high-dimensional spaces, and a polynomial kernel lets them consider feature conjunctions implicitly.

Each token is classified from a fixed window of context: neighbouring words, their POS tags, and the chunk labels already assigned. Tagging runs either left-to-right or right-to-left. The multi-class problem is decomposed into pairwise binary SVMs. Eight systems, built with different chunk representations and parsing directions, are combined by weighted voting. Some of the weights come from SVM theory (VC bound, Leave-One-Out bound) rather than only from cross validation.

On WSJ-based data, the combined system reaches F 94.22 (baseNP-S), 95.77 (baseNP-L) and 93.91 (CoNLL-2000 chunking). These scores beat earlier MBL/ME-based voting systems, which shows that margin-based learners handle rich, sparse NLP features well.

## Key points
- Datasets: baseNP-S, baseNP-L and the CoNLL-2000 chunking data (WSJ corpus).
- Best F-scores with weighted voting:
  - baseNP-S: 94.22 (P 94.15%, R 94.29%)
  - baseNP-L: 95.77
  - chunking: 93.91
- Comparison with prior work:
  - Single representations already give 93.76–94.11 on baseNP-S and 95.29–95.34 on baseNP-L.
  - Tjong Kim Sang et al. report 93.86 and 94.90 on the same sets.
- Several chunk representations are described: IOB1, IOB2, IOE1, IOE2 and Start/End (B/I/O/E/S). Voting results are reported for the IOB/IOE variants, with forward and backward parsing.
- The authors' CoNLL-2000 shared-task entry scored 93.48 with IOB2-F (beam width 5). The re-run in this paper got 93.46 with deterministic tagging (width 1) and TinySVM.

## Interesting ideas
- **Diversity from label encodings.** The same learner is trained on different encodings of the same chunks (IOB vs IOE variants, forward vs backward direction), and the outputs are then voted. This gives ensemble diversity cheaply, without needing different algorithms. It could transfer to NER or segmentation tasks for Ukrainian.
- **Theory-derived voting weights.** Each system's vote is weighted by its estimated error from the VC bound or the Leave-One-Out bound, computed from the trained model itself. The VC-bound weights performed about as well as cross-validation weights at no extra training cost. This is attractive when compute is limited.
- **Pairwise decomposition to cut training cost.** SVM training cost grows faster than linearly with data size. Many small pairwise classifiers can therefore be cheaper to train than a few one-vs-all models, and they may also be more accurate.
- **Kernel instead of manual feature engineering.** A polynomial kernel covers all feature combinations up to a chosen degree. This replaces heuristic selection of conjunctions over words, POS tags and previous labels.
- **Previous predictions as features.** Labels already assigned to neighbouring tokens are fed back in as features. Reversing the parsing direction then yields a second, complementary model from the same feature design.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Part-of-speech tagging](https://md.research.elamur.ai/free/en/topics/part-of-speech-tagging/)

## Cited by

Cited by

127 papers of our library cite it (of the 140884 whose bibliographies we read); 9 of them have a page here, 118 not yet.

- [Modeling Local Coherence: An Entity-Based Approach](https://md.research.elamur.ai/free/en/papers/barzilay-2008-modeling-local-coherence-entity-based/)
- [Natural Language Processing (almost) from Scratch](https://md.research.elamur.ai/free/en/papers/collobert-2011-natural-language-processing-almost-scratch/)
- [A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks](https://md.research.elamur.ai/free/en/papers/hashimoto-2016-joint-many-task-model-growing/)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://md.research.elamur.ai/free/en/papers/huang-2015-bidirectional-lstm-crf-models-sequence/)
- [Japanese Zero Pronoun Resolution based on Ranking Rules and Machine Learning](https://md.research.elamur.ai/free/en/papers/isozaki-2003-japanese-zero-pronoun-resolution-based/)
- [Modeling Joint Entity and Relation Extraction with Table Representation](https://md.research.elamur.ai/free/en/papers/miwa-2014-modeling-joint-entity-relation-extraction/)
- [Shallow Semantic Parsing using Support Vector Machines](https://md.research.elamur.ai/free/en/papers/pradhan-2004-shallow-semantic-parsing-support-vector/)
- [Towards Robust Semantic Role Labeling](https://md.research.elamur.ai/free/en/papers/pradhan-2008-robust-semantic-role-labeling/)
- [A Classifier-Based Parser with Linear Run-Time Complexity](https://md.research.elamur.ai/free/en/papers/sagae-2005-classifier-based-parser-linear-run/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kudo2001chunking,
  title = {{Chunking with Support Vector Machines}},
  author = {Kudo, Taku and Matsumoto, Yuji},
  booktitle = {Second Meeting of the North American Chapter of the Association for Computational Linguistics},
  year = {2001},
  doi = {10.3115/1073336.1073361},
  url = {https://doi.org/10.3115/1073336.1073361}
}
```

- [← previous Building a Discourse-Tagged Corpus in the Framework of Rhetorical Structure Theory](https://md.research.elamur.ai/free/en/papers/carlson-2001-building-discourse-tagged-corpus-framework/)
- [next → Classifying the Semantic Relations in Noun Compounds via a Domain-Specific Lexical Hierarchy](https://md.research.elamur.ai/free/en/papers/aclbulk-2001-rosario-classifying-semantic-relations-noun-compounds/)
- [↑ back to papers of 2001](https://md.research.elamur.ai/free/en/papers/2001/)

## References

22 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Erin L. Allwein, Robert E. Schapire, and Yoram Singer. 2000. Reducing multiclass to binary: A unifying approach for margin classifiers. In International Conf. on Machine Learning (ICML), pages 9–16. [find](https://scholar.google.com/scholar?q=Erin+L%2E+Allwein%2C+Robert+E%2E+Schapire%2C+and+Yoram+Singer%2E+2000%2E+Reducing+multiclass+to+binary%3A+A+unifying+approach+for+margin+classifiers%2E+In+International+Conf%2E+on+Machine+Learning+%28ICML%29%2C+pages+9%E2%80%9316%2E)
2. Eric Brill. 1995. Transformation-Based Error-Driven Learning and Natural Language Processing: A Case Study in Part-of-Speech Tagging. Computational Linguistics, 21(4). [find](https://scholar.google.com/scholar?q=Eric+Brill%2E+1995%2E+Transformation%2DBased+Error%2DDriven+Learning+and+Natural+Language+Processing%3A+A+Case+Study+in+Part%2Dof%2DSpeech+Tagging%2E+Computational+Linguistics%2C+21%284%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/brill-1995-transformation-based-error-driven-learning/)
3. Oliver Chapelle and Vladimir Vapnik. 2000. Model selection for support vector machines. In Advances in Neural Information Processing Systems 12. Cambridge, Mass: MIT Press. [find](https://scholar.google.com/scholar?q=Oliver+Chapelle+and+Vladimir+Vapnik%2E+2000%2E+Model+selection+for+support+vector+machines%2E+In+Advances+in+Neural+Information+Processing+Systems+12%2E+Cambridge%2C+Mass%3A+MIT+Press%2E)
4. C. Cortes and Vladimir N. Vapnik. 1995. Support Vector Networks. Machine Learning, 20:273– 297. [find](https://scholar.google.com/scholar?q=C%2E+Cortes+and+Vladimir+N%2E+Vapnik%2E+1995%2E+Support+Vector+Networks%2E+Machine+Learning%2C+20%3A273%E2%80%93+297%2E)
5. T. G. Dietterich and G. Bakiri. 1995. Solving multi-class learning problems via error-correcting output codes. Journal of Artificial Intelligence Research, 2:263–286. [find](https://scholar.google.com/scholar?q=T%2E+G%2E+Dietterich+and+G%2E+Bakiri%2E+1995%2E+Solving+multi%2Dclass+learning+problems+via+error%2Dcorrecting+output+codes%2E+Journal+of+Artificial+Intelligence+Research%2C+2%3A263%E2%80%93286%2E)
6. Yoav Freund and Robert E. Schapire. 1996. Experiments with a new boosting algorithm. In International Conference on Machine Learning (ICML), pages 148–146. [find](https://scholar.google.com/scholar?q=Yoav+Freund+and+Robert+E%2E+Schapire%2E+1996%2E+Experiments+with+a+new+boosting+algorithm%2E+In+International+Conference+on+Machine+Learning+%28ICML%29%2C+pages+148%E2%80%93146%2E)
7. Thorsten Joachims. 1998. Text Categorization with Support Vector Machines: Learning with Many Relevant Features. In European Conference on Machine Learning (ECML). [find](https://scholar.google.com/scholar?q=Thorsten+Joachims%2E+1998%2E+Text+Categorization+with+Support+Vector+Machines%3A+Learning+with+Many+Relevant+Features%2E+In+European+Conference+on+Machine+Learning+%28ECML%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/joachims-1998-text-categorization-support-vector-machines/)
8. Ulrich H.-G Kreßel. 1999. Pairwise Classification and Support Vector Machines. In Advances in Kernel Mathods. MIT Press. [find](https://scholar.google.com/scholar?q=Ulrich+H%2E%2DG+Kre%C3%9Fel%2E+1999%2E+Pairwise+Classification+and+Support+Vector+Machines%2E+In+Advances+in+Kernel+Mathods%2E+MIT+Press%2E)
9. Taku Kudo and Yuji Matsumoto. 2000a. Japanese Dependency Structure Analysis Based on Support Vector Machines. In Empirical Methods in Natural Language Processing and Very Large Corpora, pages 18–25. [find](https://scholar.google.com/scholar?q=Taku+Kudo+and+Yuji+Matsumoto%2E+2000a%2E+Japanese+Dependency+Structure+Analysis+Based+on+Support+Vector+Machines%2E+In+Empirical+Methods+in+Natural+Language+Processing+and+Very+Large+Corpora%2C+pages+18%E2%80%9325%2E)
10. Taku Kudo and Yuji Matsumoto. 2000b. Use of Support Vector Learning for Chunk Identification. In Proceedings of the 4th Conference on CoNLL-2000 and LLL-2000, pages 142–144. [find](https://scholar.google.com/scholar?q=Taku+Kudo+and+Yuji+Matsumoto%2E+2000b%2E+Use+of+Support+Vector+Learning+for+Chunk+Identification%2E+In+Proceedings+of+the+4th+Conference+on+CoNLL%2D2000+and+LLL%2D2000%2C+pages+142%E2%80%93144%2E)
11. Lance A. Ramshaw and Mitchell P. Marcus. 1995. Text chunking using transformation-based learning. In Proceedings of the 3rd Workshop on Very Large Corpora, pages 88–94. [find](https://scholar.google.com/scholar?q=Lance+A%2E+Ramshaw+and+Mitchell+P%2E+Marcus%2E+1995%2E+Text+chunking+using+transformation%2Dbased+learning%2E+In+Proceedings+of+the+3rd+Workshop+on+Very+Large+Corpora%2C+pages+88%E2%80%9394%2E) [our card](https://md.research.elamur.ai/free/en/papers/ramshaw-1995-text-chunking-transformation-based-learning/)
12. Manabu Sassano and Takehito Utsuro. 2000. Named Entity Chunking Techniques in Supervised Learning for Japanese Named Entity Recognition. In Proceedings of COLING 2000, pages 705–711. [find](https://scholar.google.com/scholar?q=Manabu+Sassano+and+Takehito+Utsuro%2E+2000%2E+Named+Entity+Chunking+Techniques+in+Supervised+Learning+for+Japanese+Named+Entity+Recognition%2E+In+Proceedings+of+COLING+2000%2C+pages+705%E2%80%93711%2E)
13. Robert E. Schapire, Yoav Freund, Peter Bartlett, and Wee Sun Lee. 1997. Boosting the margin: a new explanation for the effectiveness of voting methods. In International Conference on Machine Learning (ICML), pages 322–330. [find](https://scholar.google.com/scholar?q=Robert+E%2E+Schapire%2C+Yoav+Freund%2C+Peter+Bartlett%2C+and+Wee+Sun+Lee%2E+1997%2E+Boosting+the+margin%3A+a+new+explanation+for+the+effectiveness+of+voting+methods%2E+In+International+Conference+on+Machine+Learning+)
14. Hirotoshi Taira and Masahiko Haruno. 1999. Feature Selection in SVM Text Categorization. In AAAI-99. [find](https://scholar.google.com/scholar?q=Hirotoshi+Taira+and+Masahiko+Haruno%2E+1999%2E+Feature+Selection+in+SVM+Text+Categorization%2E+In+AAAI%2D99%2E)
15. Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 Shared Task: Chunking. In Proceedings of CoNLL-2000 and LLL-2000, pages 127–132. [find](https://scholar.google.com/scholar?q=Erik+F%2E+Tjong+Kim+Sang+and+Sabine+Buchholz%2E+2000%2E+Introduction+to+the+CoNLL%2D2000+Shared+Task%3A+Chunking%2E+In+Proceedings+of+CoNLL%2D2000+and+LLL%2D2000%2C+pages+127%E2%80%93132%2E) [our card](https://md.research.elamur.ai/free/en/papers/sang-2000-introduction-conll-2000-shared-task/)
16. Erik F. Tjong Kim Sang and Jorn Veenstra. 1999. Representing text chunks. In Proceedings of EACL’99, pages 173–179. [find](https://scholar.google.com/scholar?q=Erik+F%2E+Tjong+Kim+Sang+and+Jorn+Veenstra%2E+1999%2E+Representing+text+chunks%2E+In+Proceedings+of+EACL%E2%80%9999%2C+pages+173%E2%80%93179%2E) [our card](https://md.research.elamur.ai/free/en/papers/sang-1999-representing-text-chunks/)
17. Erik F. Tjong Kim Sang, Walter Daelemans, Hervé Déjean, Rob Koeling, Yuval Krymolowski, Vasin Punyakanok, and Dan Roth. 2000. Applying system combination to base noun phrase identification. In Proceedings of COLING 2000, pages 857–863. [find](https://scholar.google.com/scholar?q=Erik+F%2E+Tjong+Kim+Sang%2C+Walter+Daelemans%2C+Herve%CC%81+De%CC%81jean%2C+Rob+Koeling%2C+Yuval+Krymolowski%2C+Vasin+Punyakanok%2C+and+Dan+Roth%2E+2000%2E+Applying+system+combination+to+base+noun+phrase+identification%2E+In+Proce)
18. Erik F. Tjong Kim Sang. 2000a. Noun phrase recognition by system combination. In Proceedings of ANLP-NAACL 2000, pages 50–55. [find](https://scholar.google.com/scholar?q=Erik+F%2E+Tjong+Kim+Sang%2E+2000a%2E+Noun+phrase+recognition+by+system+combination%2E+In+Proceedings+of+ANLP%2DNAACL+2000%2C+pages+50%E2%80%9355%2E)
19. Erik F. Tjong Kim Sang. 2000b. Text Chunking by System Combination. In Proceedings of CoNLL-2000 and LLL-2000, pages 151–153. [find](https://scholar.google.com/scholar?q=Erik+F%2E+Tjong+Kim+Sang%2E+2000b%2E+Text+Chunking+by+System+Combination%2E+In+Proceedings+of+CoNLL%2D2000+and+LLL%2D2000%2C+pages+151%E2%80%93153%2E)
20. Kiyotaka Uchimoto, Qing Ma, Masaki Murata, Hiromi Ozaku, and Hitoshi Isahara. 2000. Named Entity Extraction Based on A Maximum Entropy Model and Transformation Rules. In Processing of the ACL 2000. [find](https://scholar.google.com/scholar?q=Kiyotaka+Uchimoto%2C+Qing+Ma%2C+Masaki+Murata%2C+Hiromi+Ozaku%2C+and+Hitoshi+Isahara%2E+2000%2E+Named+Entity+Extraction+Based+on+A+Maximum+Entropy+Model+and+Transformation+Rules%2E+In+Processing+of+the+ACL+2000%2E)
21. Hans van Halteren. 2000. Chunking with WPDV Models. In Proceedings of CoNLL-2000 and LLL-2000, pages 154–156. [find](https://scholar.google.com/scholar?q=Hans+van+Halteren%2E+2000%2E+Chunking+with+WPDV+Models%2E+In+Proceedings+of+CoNLL%2D2000+and+LLL%2D2000%2C+pages+154%E2%80%93156%2E)
22. Vladimir N. Vapnik. 1998. Statistical Learning Theory. Wiley-Interscience. [find](https://scholar.google.com/scholar?q=Vladimir+N%2E+Vapnik%2E+1998%2E+Statistical+Learning+Theory%2E+Wiley%2DInterscience%2E)