---
title: "Feature-Rich Part-of-Speech Tagging with a — Toutanova 2003"
description: "The paper asks whether an English part-of-speech tagger improves when each local decision can see tags on both sides, not only the preceding ones."
authors: ["Kristina Toutanova", "Dan Klein", "Christopher D. Manning", "Yoram Singer"]
year: 2003
venue: "Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics"
source: "https://aclanthology.org/N03-1033/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/toutanova-2003-feature-rich-part-speech-tagging/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/toutanova-2003-feature-rich-part-speech-tagging/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2003](https://md.research.elamur.ai/free/en/papers/2003/) / Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network

# Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network

**Authors:** [Kristina Toutanova](https://md.research.elamur.ai/free/en/authors/kristina-toutanova/), [Dan Klein](https://md.research.elamur.ai/free/en/authors/dan-klein/), [Christopher D. Manning](https://md.research.elamur.ai/free/en/authors/christopher-d-manning/), [Yoram Singer](https://md.research.elamur.ai/free/en/authors/yoram-singer/) · **Year:** 2003 · **Venue:** Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/N03-1033/) · [PDF](https://aclanthology.org/N03-1033.pdf) · [OpenAlex](https://openalex.org/W1996430422)

**Citations:** 2844 citations (OpenAlex, 2026-10-14) · [715](https://md.research.elamur.ai/free/en/papers/toutanova-2003-feature-rich-part-speech-tagging/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper asks whether an English part-of-speech tagger improves when each local decision can see tags on both sides, not only the preceding ones. The authors build a cyclic dependency network. At every position, a local conditional loglinear (maxent) model predicts the current tag from the neighbouring tags on the left and right plus the surrounding words. Such a cyclic graph does not define a proper joint distribution. Even so, the product of the local scores can be maximised exactly with a Viterbi-like dynamic program.

On top of this structure they add rich lexical templates, including features over several consecutive words, detailed unknown-word features, and a Gaussian (quadratic) prior for regularisation. The prior lets them keep rare features without overfitting. On the Penn Treebank WSJ, the tagger reaches 97.24% per-token accuracy and 56.34% whole-sentence accuracy, the best single automatically learned tagger the authors knew of. The broader point is that bidirectional conditioning, heavy lexicalisation and proper regularisation work well together for sparse sequence-labelling tasks.

## Key points
- Results on the Penn Treebank WSJ: 97.24% tag accuracy and 56.34% of sentences tagged fully correctly.
- Error reduction over previous work, on the same data splits:
  - 4.4% relative to the voted perceptron of Collins (2002);
  - 12.1% relative to the earlier loglinear tagger of Toutanova and Manning (2000).
- The tagger also beats the combination tagger of Brill and Wu (1998), which reported 97.16% with more training data.
- Effect of regularisation on the development set:
  - Smaller model: 5.3% fewer errors overall and 20.1% fewer on unknown words.
  - Large feature-rich model: 16.2% fewer errors after convergence, and 5.8% fewer than the unsmoothed model's best early-stopped result.
- Training speed: smoothed models converged in 80 and 134 conjugate-gradient iterations, versus 335 and 370 for unsmoothed ones. The largest models used several hundred thousand features and took about 25 minutes per iteration on a 2GHz PC.

## Interesting ideas
- **Condition on both neighbours.** Each local classifier predicts a label from both the left and right context, and decoding uses a modified Viterbi search. This avoids "observation bias" cases such as *will to fight*, where a purely left-to-right model loses useful evidence. The trick transfers to any sequence labeller, such as NER, morphological tagging or chunking.
- **Prior instead of frequency cutoffs.** A Gaussian prior replaces feature-frequency cutoffs and early stopping, so very rare features can be kept and still help. It also shifts weight toward general features, which benefits unknown words. This is valuable for morphologically rich languages like Ukrainian, where many word forms are rare.
- **Multiword lexical templates.** Templates that look jointly at several consecutive words let the model learn fixed idiomatic sequences automatically. This is a cheap way to recover what hand-written idiom lists used to provide.
- **Training-curve diagnostic.** Plotting dev accuracy against training iterations shows that regularised models improve almost steadily while unregularised ones peak and then decline. This is a simple check of whether a model needs early stopping or stronger regularisation.
- **Report sentence-level accuracy.** Whole-sentence accuracy is reported alongside per-token accuracy. It exposes differences that per-token figures hide once token accuracy approaches its ceiling.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Part-of-speech tagging](https://md.research.elamur.ai/free/en/topics/part-of-speech-tagging/)

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 248 works · 214 804 citations
- [Hebrew University of Jerusalem](https://md.research.elamur.ai/free/en/universities/hebrew-university-of-jerusalem/) — University, Jerusalem, Israel · 42 works · 8 354 citations

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{toutanova2003feature,
  title = {{Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network}},
  author = {Toutanova, Kristina and Klein, Dan and Manning, Christopher D. and Singer, Yoram},
  booktitle = {Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2003},
  doi = {10.3115/1073445.1073478},
  url = {https://doi.org/10.3115/1073445.1073478}
}
```

- [← previous Accurate Unlexicalized Parsing](https://md.research.elamur.ai/free/en/papers/aclbulk-2003-klein-accurate-unlexicalized-parsing/)
- [next → Head-Driven Statistical Models for Natural Language Parsing](https://md.research.elamur.ai/free/en/papers/cl-2003-collins-head-driven-statistical-models-natural/)
- [↑ back to papers of 2003](https://md.research.elamur.ai/free/en/papers/2003/)

## References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Steven Abney, Robert E. Schapire, and Yoram Singer. 1999. Boosting applied to tagging and PP attachment. In EMNLP/VLC 1999, pages 38–45. [find](https://scholar.google.com/scholar?q=Steven+Abney%2C+Robert+E%2E+Schapire%2C+and+Yoram+Singer%2E+1999%2E+Boosting+applied+to+tagging+and+PP+attachment%2E+In+EMNLP%2FVLC+1999%2C+pages+38%E2%80%9345%2E)
2. Thorsten Brants. 2000. TnT – a statistical part-of-speech tagger. In ANLP 6, pages 224–231. [find](https://scholar.google.com/scholar?q=Thorsten+Brants%2E+2000%2E+TnT+%E2%80%93+a+statistical+part%2Dof%2Dspeech+tagger%2E+In+ANLP+6%2C+pages+224%E2%80%93231%2E) [our card](https://md.research.elamur.ai/free/en/papers/brants-2000-tnt-statistical-part-speech-tagger/)
3. Eric Brill and Jun Wu. 1998. Classifier combination for improved lexical disambiguation. In ACL 36/COLING 17, pages 191–195. [find](https://scholar.google.com/scholar?q=Eric+Brill+and+Jun+Wu%2E+1998%2E+Classifier+combination+for+improved+lexical+disambiguation%2E+In+ACL+36%2FCOLING+17%2C+pages+191%E2%80%93195%2E)
4. Eric Brill. 1995. Transformation-based error-driven learning and natural language processing: A case study in part-of-speech tagging. Computational Linguistics, 21(4):543–565. [find](https://scholar.google.com/scholar?q=Eric+Brill%2E+1995%2E+Transformation%2Dbased+error%2Ddriven+learning+and+natural+language+processing%3A+A+case+study+in+part%2Dof%2Dspeech+tagging%2E+Computational+Linguistics%2C+21%284%29%3A543%E2%80%93565%2E) [our card](https://md.research.elamur.ai/free/en/papers/brill-1995-transformation-based-error-driven-learning/)
5. Eugene Charniak, Curtis Hendrickson, Neil Jacobson, and Mike Perkowitz. 1993. Equations for part-of-speech tagging. In AAAI 11, pages 784–789. [find](https://scholar.google.com/scholar?q=Eugene+Charniak%2C+Curtis+Hendrickson%2C+Neil+Jacobson%2C+and+Mike+Perkowitz%2E+1993%2E+Equations+for+part%2Dof%2Dspeech+tagging%2E+In+AAAI+11%2C+pages+784%E2%80%93789%2E)
6. Stanley F. Chen and Ronald Rosenfeld. 2000. A survey of smoothing techniques for maximum entropy models. IEEE Transactions on Speech and Audio Processing, 8(1):37–50. [find](https://scholar.google.com/scholar?q=Stanley+F%2E+Chen+and+Ronald+Rosenfeld%2E+2000%2E+A+survey+of+smoothing+techniques+for+maximum+entropy+models%2E+IEEE+Transactions+on+Speech+and+Audio+Processing%2C+8%281%29%3A37%E2%80%9350%2E)
7. Kenneth W. Church. 1988. A stochastic parts program and noun phrase parser for unrestricted text. In ANLP 2, pages 136–143. [find](https://scholar.google.com/scholar?q=Kenneth+W%2E+Church%2E+1988%2E+A+stochastic+parts+program+and+noun+phrase+parser+for+unrestricted+text%2E+In+ANLP+2%2C+pages+136%E2%80%93143%2E) [our card](https://md.research.elamur.ai/free/en/papers/church-1988-stochastic-parts-program-noun-phrase/)
8. Michael Collins. 2002. Discriminative training methods for Hidden Markov Models: Theory and experiments with perceptron algorithms. In EMNLP 2002. [find](https://scholar.google.com/scholar?q=Michael+Collins%2E+2002%2E+Discriminative+training+methods+for+Hidden+Markov+Models%3A+Theory+and+experiments+with+perceptron+algorithms%2E+In+EMNLP+2002%2E) [our card](https://md.research.elamur.ai/free/en/papers/collins-2002-discriminative-training-methods-hidden-markov/)
9. Robert G. Cowell, A. Philip Dawid, Steffen L. Lauritzen, and David J. Spiegelhalter. 1999. Probabilistic Networks and Expert Systems. Springer-Verlag, New York. [find](https://scholar.google.com/scholar?q=Robert+G%2E+Cowell%2C+A%2E+Philip+Dawid%2C+Steffen+L%2E+Lauritzen%2C+and+David+J%2E+Spiegelhalter%2E+1999%2E+Probabilistic+Networks+and+Expert+Systems%2E+Springer%2DVerlag%2C+New+York%2E)
10. David Heckerman, David Maxwell Chickering, Christopher Meek, Robert Rounthwaite, and Carl Myers Kadie. 2000. Dependency networks for inference, collaborative filtering and data visualization. Journal of Machine Learning Research, 1(1):49–75. [find](https://scholar.google.com/scholar?q=David+Heckerman%2C+David+Maxwell+Chickering%2C+Christopher+Meek%2C+Robert+Rounthwaite%2C+and+Carl+Myers+Kadie%2E+2000%2E+Dependency+networks+for+inference%2C+collaborative+filtering+and+data+visualization%2E+Journal+)
11. Mark Johnson, Stuart Geman, Stephen Canon, Zhiyi Chi, and Stefan Riezler. 1999. Estimators for stochastic “unification-based” grammars. In ACL 37, pages 535–541. [find](https://scholar.google.com/scholar?q=Mark+Johnson%2C+Stuart+Geman%2C+Stephen+Canon%2C+Zhiyi+Chi%2C+and+Stefan+Riezler%2E+1999%2E+Estimators+for+stochastic+%E2%80%9Cunification%2Dbased%E2%80%9D+grammars%2E+In+ACL+37%2C+pages+535%E2%80%93541%2E)
12. Dan Klein and Christopher D. Manning. 2002. Conditional structure versus conditional estimation in NLP models. In EMNLP 2002, pages 9–16. [find](https://scholar.google.com/scholar?q=Dan+Klein+and+Christopher+D%2E+Manning%2E+2002%2E+Conditional+structure+versus+conditional+estimation+in+NLP+models%2E+In+EMNLP+2002%2C+pages+9%E2%80%9316%2E)
13. John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML-2001, pages 282–289. [find](https://scholar.google.com/scholar?q=John+Lafferty%2C+Andrew+McCallum%2C+and+Fernando+Pereira%2E+2001%2E+Conditional+random+fields%3A+Probabilistic+models+for+segmenting+and+labeling+sequence+data%2E+In+ICML%2D2001%2C+pages+282%E2%80%93289%2E)
14. Sang-Zoo Lee, Jun ichi Tsujii, and Hae-Chang Rim. 2000. Part-of-speech tagging based on Hidden Markov Model assuming joint independence. In ACL 38, pages 263–169. [find](https://scholar.google.com/scholar?q=Sang%2DZoo+Lee%2C+Jun+ichi+Tsujii%2C+and+Hae%2DChang+Rim%2E+2000%2E+Part%2Dof%2Dspeech+tagging+based+on+Hidden+Markov+Model+assuming+joint+independence%2E+In+ACL+38%2C+pages+263%E2%80%93169%2E)
15. Mitchell P. Marcus, Beatrice Santorini, and Mary A. Marcinkiewicz. 1994. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19:313– 330. [find](https://scholar.google.com/scholar?q=Mitchell+P%2E+Marcus%2C+Beatrice+Santorini%2C+and+Mary+A%2E+Marcinkiewicz%2E+1994%2E+Building+a+large+annotated+corpus+of+English%3A+The+Penn+Treebank%2E+Computational+Linguistics%2C+19%3A313%E2%80%93+330%2E) [our card](https://md.research.elamur.ai/free/en/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
16. Ian Marshall. 1987. Tag selection using probabilistic methods. In Roger Garside, Geoffrey Sampson, and Geoffrey Leech, editors, The Computational analysis of English: a corpus-based approach, pages 42–65. Longman, London. [find](https://scholar.google.com/scholar?q=Ian+Marshall%2E+1987%2E+Tag+selection+using+probabilistic+methods%2E+In+Roger+Garside%2C+Geoffrey+Sampson%2C+and+Geoffrey+Leech%2C+editors%2C+The+Computational+analysis+of+English%3A+a+corpus%2Dbased+approach%2C+pages+42)
17. Adwait Ratnaparkhi. 1996. A maximum entropy model for part-of-speech tagging. In EMNLP 1, pages 133–142. [find](https://scholar.google.com/scholar?q=Adwait+Ratnaparkhi%2E+1996%2E+A+maximum+entropy+model+for+part%2Dof%2Dspeech+tagging%2E+In+EMNLP+1%2C+pages+133%E2%80%93142%2E) [our card](https://md.research.elamur.ai/free/en/papers/ratnaparkhi-1996-maximum-entropy-model-part-speech/)
18. Scott M. Thede and Mary P. Harper. 1999. Second-order hidden Markov model for part-of-speech tagging. In ACL 37, pages 175–182. [find](https://scholar.google.com/scholar?q=Scott+M%2E+Thede+and+Mary+P%2E+Harper%2E+1999%2E+Second%2Dorder+hidden+Markov+model+for+part%2Dof%2Dspeech+tagging%2E+In+ACL+37%2C+pages+175%E2%80%93182%2E)
19. Kristina Toutanova and Christopher Manning. 2000. Enriching the knowledge sources used in a maximum entropy part-of-speech tagger. In EMNLP/VLC 1999, pages 63–71. [find](https://scholar.google.com/scholar?q=Kristina+Toutanova+and+Christopher+Manning%2E+2000%2E+Enriching+the+knowledge+sources+used+in+a+maximum+entropy+part%2Dof%2Dspeech+tagger%2E+In+EMNLP%2FVLC+1999%2C+pages+63%E2%80%9371%2E) [our card](https://md.research.elamur.ai/free/en/papers/toutanova-2000-enriching-knowledge-sources-used-maximum/)
20. Tong Zhang and Frank J. Oles. 2001. Text categorization based on regularized linear classification methods. Information Retrieval, 4:5–31. [find](https://scholar.google.com/scholar?q=Tong+Zhang+and+Frank+J%2E+Oles%2E+2001%2E+Text+categorization+based+on+regularized+linear+classification+methods%2E+Information+Retrieval%2C+4%3A5%E2%80%9331%2E)