--- title: "Enriching the Knowledge Sources Used in a — Toutanova 2000" description: "This work asks whether a statistical part-of-speech tagger can gain accuracy by drawing on richer kinds of evidence instead of reusing the standard…" authors: ["Kristina Toutanova", "Christopher D. Manning"] year: 2000 venue: "2000 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora" source: "https://aclanthology.org/W00-1308/" doi: "10.3115/1117794.1117802" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/toutanova-2000-enriching-knowledge-sources-used-maximum/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/toutanova-2000-enriching-knowledge-sources-used-maximum/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2000](https://md.research.elamur.ai/free/en/papers/2000/) / Enriching the Knowledge Sources Used in a Maximum Entropy Part-of-Speech Tagger # Enriching the Knowledge Sources Used in a Maximum Entropy Part-of-Speech Tagger **Authors:** [Kristina Toutanova](https://md.research.elamur.ai/free/en/authors/kristina-toutanova/), [Christopher D. Manning](https://md.research.elamur.ai/free/en/authors/christopher-d-manning/) · **Year:** 2000 · **Venue:** 2000 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/W00-1308/) · [PDF](https://aclanthology.org/W00-1308.pdf) · [DOI 10.3115/1117794.1117802](https://doi.org/10.3115/1117794.1117802) · [OpenAlex](https://openalex.org/W2003458432) **Citations:** 939 citations (openalex-snapshot · 2026-10-14) · [143](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary This work asks whether a statistical part-of-speech tagger can gain accuracy by drawing on richer kinds of evidence instead of reusing the standard n-gram-style cues that most taggers share. The authors build a maximum entropy (log-linear) tagger trained with Improved Iterative Scaling on the WSJ portion of the Penn Treebank. Its baseline features closely follow Ratnaparkhi (1996). They then add a small set of linguistically motivated, partly non-local features targeting three error sources: - capitalization cues for unknown words, - confusions between verb forms (VB/VBP, VBD/VBN), - the RP/IN/RB ambiguity of words like *up* or *out*. Each addition gives a modest but consistent gain. The final model reaches 96.86% overall accuracy and 86.91% on unseen words, compared with 96.72% and 84.5% for the baseline. The extra features number very few, so the model barely grows. The work matters because it shows that targeted, linguistically informed features can still move performance when competing taggers seem to have converged to near-identical accuracy. ## Key points - **Data split:** WSJ sections 0–20 were used for training (about 1.06M tokens), 21–22 for development, and 23–24 for the final test (111,221 tokens, 2.59% unknown). - **Baseline setup:** - Words seen fewer than 7 times count as rare and receive special affix and shape features. - Feature cutoffs are 5 for general templates and 45 for rare ones. - Decoding uses a beam of size 5. - A tag dictionary is augmented with the systematic VBD/VBN and VB/VBP ambiguities. - **Capitalization features (with prefix features removed):** accuracy rose to 96.76% overall and 86.76% on unknown words, which cuts unknown-word error by about 15% relative to the baseline. - **Verb-form features:** VB/VBP confusions fell by 23.1% and VBD/VBN confusions by 12.3%, giving 96.83% overall and 86.87% on unknown words. - **Particle features:** these use verb–particle pairs mined from the training data and raised RP accuracy from 41.5% to 44.3%. The authors informally estimate that the Treebank itself marks this distinction only about 85% correctly. ## Interesting ideas - **Classify errors before engineering features.** The authors sort tagger errors into four groups: annotation inconsistency, unknown-word issues, systematic ambiguities with clear contextual cues, and semantically subtle distinctions. They then spend effort only where gains are realistic. This kind of triage would help prioritize feature or data work for any tagger, including a Ukrainian one. - **Prune templates through ablation.** Templates were added one at a time and checked against the dev set. This showed that word features more than one position away added nothing, so dropping them kept the model lean without losing accuracy. - **Use non-local, structure-motivated predicates.** One example looks back up to three positions for a verb known from the training data to take the current word as a particle. Maximum entropy models absorb such overlapping cues without assuming independence, and the same trick suits other long-range dependencies. - **Hard-code known lexical ambiguities into the tag dictionary.** If a verb form was seen only as VBD, VBN is still allowed as a candidate (and likewise VB/VBP). This cheaply protects against training-data gaps in any closed-class ambiguity pattern. - **Measure the annotation ceiling.** A quick informal check of how reliably the treebank marks a distinction tells you how much improvement is attainable at all. It is a useful sanity check before chasing small gains on a noisy category. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) Topics: [Treebanks](https://md.research.elamur.ai/free/en/topics/treebanks/) · [Part-of-speech tagging](https://md.research.elamur.ai/free/en/topics/part-of-speech-tagging/) ## Cited by Cited by 143 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 136 not yet. - [A Probabilistic Morphological Analyzer for Syriac](https://md.research.elamur.ai/free/en/papers/aclbulk-2010-mcclanahan-probabilistic-morphological-analyzer-syriac/) - [Domain-Dependent Detection of Light Verb Constructions](https://md.research.elamur.ai/free/en/papers/aclbulk-2011-nagy-domain-dependent-detection-light-verb/) - [Investigating Data Contamination in Modern Benchmarks for Large Language Models](https://md.research.elamur.ai/free/en/papers/deng-2024-investigating-data-contamination-modern-benchmarks/) - [An Accurate Arabic Root-Based Lemmatizer for Information Retrieval Purposes](https://md.research.elamur.ai/free/en/papers/langstream-2012-el-shishtawy-accurate-arabic-root-based-lemmatizer/) - [Discovering Implicit Discourse Relations Through Brown Cluster Pair Representation and Coreference Patterns](https://md.research.elamur.ai/free/en/papers/rutherford-2014-discovering-implicit-discourse-relations-through/) - [Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network](https://md.research.elamur.ai/free/en/papers/toutanova-2003-feature-rich-part-speech-tagging/) - [A Hierarchical Neural Model for Learning Sequences of Dialogue Acts](https://md.research.elamur.ai/free/en/papers/tran-2017-hierarchical-neural-model-learning-sequences/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{toutanova2000enriching, title = {{Enriching the Knowledge Sources Used in a Maximum Entropy Part-of-Speech Tagger}}, author = {Toutanova, Kristina and Manning, Christopher D.}, booktitle = {2000 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora}, year = {2000}, doi = {10.3115/1117794.1117802}, url = {https://doi.org/10.3115/1117794.1117802} } ``` - [← previous Discriminative Reranking for Natural Language Parsing](https://md.research.elamur.ai/free/en/papers/collins-2000-discriminative-reranking-natural-language-parsing/) - [next → Experiments on Sentence Boundary Detection](https://md.research.elamur.ai/free/en/papers/aclbulk-2000-stevenson-experiments-sentence-boundary-detection/) - [↑ back to papers of 2000](https://md.research.elamur.ai/free/en/papers/2000/) ## References 10 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Baker, C. L. 1995. English Syntax. Cambridge, MA: MIT Press, 2 nd edition. nerger, Adam L., Della Pietra, Stephen A., and Della Pietra, Vincent J. 1996. A Maximum Entropy Approach to Natural Language Processing. Computational Linguistics 22:39-71. [find](https://scholar.google.com/scholar?q=Baker%2C+C%2E+L%2E+1995%2E+English+Syntax%2E+Cambridge%2C+MA%3A+MIT+Press%2C+2+nd+edition%2E+nerger%2C+Adam+L%2E%2C+Della+Pietra%2C+Stephen+A%2E%2C+and+Della+Pietra%2C+Vincent+J%2E+1996%2E+A+Maximum+Entropy+Approach+to+Natural+Language+) [our card](https://md.research.elamur.ai/free/en/papers/berger-1996-maximum-entropy-approach-natural-language/) 2. Brants, Thorsten. 2000. TnT - A Statistical Part-of-Speech Tagger. In Proceedings of the Sixth Applied Natural Language Processing Conference (ANLP 2000), Seattle, WA, pp. 224-23 I. [find](https://scholar.google.com/scholar?q=Brants%2C+Thorsten%2E+2000%2E+TnT+%2D+A+Statistical+Part%2Dof%2DSpeech+Tagger%2E+In+Proceedings+of+the+Sixth+Applied+Natural+Language+Processing+Conference+%28ANLP+2000%29%2C+Seattle%2C+WA%2C+pp%2E+224%2D23+I%2E) [our card](https://md.research.elamur.ai/free/en/papers/brants-2000-tnt-statistical-part-speech-tagger/) 3. Brill, Eric. 1994. Some Advances in Transformation-Based Part of Speech Tagging. Proceedings of AAAL Vol. 1, pp. 722-727. [find](https://scholar.google.com/scholar?q=Brill%2C+Eric%2E+1994%2E+Some+Advances+in+Transformation%2DBased+Part+of+Speech+Tagging%2E+Proceedings+of+AAAL+Vol%2E+1%2C+pp%2E+722%2D727%2E) 4. Charniak, Eugene. 2000. A Maximum-Entropy-Inspired Parser. Proceedings of the I st Meeting of the North American Chapter of the Association for Computational Linguistics, pp. 132-139. [find](https://scholar.google.com/scholar?q=Charniak%2C+Eugene%2E+2000%2E+A+Maximum%2DEntropy%2DInspired+Parser%2E+Proceedings+of+the+I+st+Meeting+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%2C+pp%2E+132%2D139%2E) [our card](https://md.research.elamur.ai/free/en/papers/charniak-2000-maximum-entropy-inspired-parser/) 5. Jelinek, Frederick. 1997. Statistical Methods for Speech Recognition. Cambridge, MA: MIT Press. [find](https://scholar.google.com/scholar?q=Jelinek%2C+Frederick%2E+1997%2E+Statistical+Methods+for+Speech+Recognition%2E+Cambridge%2C+MA%3A+MIT+Press%2E) 6. Manning, Christopher D. and Hinrich Schiitze. 1999. Foundations of Statistical Natural Language Processing. Cambridge, MA: MIT Press. [find](https://scholar.google.com/scholar?q=Manning%2C+Christopher+D%2E+and+Hinrich+Schiitze%2E+1999%2E+Foundations+of+Statistical+Natural+Language+Processing%2E+Cambridge%2C+MA%3A+MIT+Press%2E) 7. Mikheev, Andrei. 1999. Periods, Capitalized Words, etc. Ms., University of Edinburgh. Available at: http:llwww.ltg.ed.ac.ukl-mikheevlpapers.html [find](https://scholar.google.com/scholar?q=Mikheev%2C+Andrei%2E+1999%2E+Periods%2C+Capitalized+Words%2C+etc%2E+Ms%2E%2C+University+of+Edinburgh%2E+Available+at%3A+http%3Allwww%2Eltg%2Eed%2Eac%2Eukl%2Dmikheevlpapers%2Ehtml) 8. Ratnaparkhi, Adwait. 1996. A maximum entropy model for part-of-speech tagging. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, University of Pennsylvania, pp. 133--142. [find](https://scholar.google.com/scholar?q=Ratnaparkhi%2C+Adwait%2E+1996%2E+A+maximum+entropy+model+for+part%2Dof%2Dspeech+tagging%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+University+of+Pennsylvania%2C+pp%2E+133) [our card](https://md.research.elamur.ai/free/en/papers/ratnaparkhi-1996-maximum-entropy-model-part-speech/) 9. Ratnaparkhi, Adwait. 1998. Maximum Entropy Models for Natural Language Ambiguity Resolution. PhD Thesis, University of Pennsylvania. [find](https://scholar.google.com/scholar?q=Ratnaparkhi%2C+Adwait%2E+1998%2E+Maximum+Entropy+Models+for+Natural+Language+Ambiguity+Resolution%2E+PhD+Thesis%2C+University+of+Pennsylvania%2E) 10. Samuelsson, Christer and Atro Voutilainen. 1997. Comparing a Linguistic and a Stochastic Tagger. In Proceedings of the 25 th Annual Meeting of the Association for Computational Linguistics, pp. 246-253. [find](https://scholar.google.com/scholar?q=Samuelsson%2C+Christer+and+Atro+Voutilainen%2E+1997%2E+Comparing+a+Linguistic+and+a+Stochastic+Tagger%2E+In+Proceedings+of+the+25+th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pp%2E+246%2D25)