--- title: "A Maximum Entropy Model for — Ratnaparkhi 1996" description: "The paper addresses English part-of-speech tagging, where previous taggers evaluated on the Wall St." authors: ["Adwait Ratnaparkhi"] year: 1996 venue: "Conference on Empirical Methods in Natural Language Processing" source: "https://aclanthology.org/W96-0213/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/ratnaparkhi-1996-maximum-entropy-model-part-speech/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ratnaparkhi-1996-maximum-entropy-model-part-speech/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [1996](https://md.research.elamur.ai/free/en/papers/1996/) / A Maximum Entropy Model for Part-Of-Speech Tagging # A Maximum Entropy Model for Part-Of-Speech Tagging **Authors:** [Adwait Ratnaparkhi](https://md.research.elamur.ai/free/en/authors/adwait-ratnaparkhi/) · **Year:** 1996 · **Venue:** Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/W96-0213/) · [PDF](https://aclanthology.org/W96-0213.pdf) · [OpenAlex](https://openalex.org/W1773803948) **Citations:** 1254 citations (openalex-snapshot · 2026-10-14) · [300](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary The paper addresses English part-of-speech tagging, where previous taggers evaluated on the Wall St. Journal portion of the Penn Treebank had achieved roughly 96.5% accuracy. Ratnaparkhi asks whether richer use of context can push this further. He builds a Maximum Entropy (log-linear) tagger that treats tagging as predicting each tag from a "history": the surrounding words within two positions, the two previous tags, and, for rare or unseen words, spelling cues such as prefixes, suffixes, digits, capitals and hyphens. Parameters are estimated with Generalized Iterative Scaling. Decoding uses a beam search, optionally restricted by a tag dictionary. On unseen test data the tagger scores 96.6%, matching or slightly beating Markov-model, decision-tree and transformation-based taggers, and it does so without word classes or elaborate smoothing. The study also finds that word-specific features add little, because annotation in the treebank is inconsistent across annotators. Training and testing on a single annotator's data improves accuracy by about 0.5%. This suggests that corpus noise, more than modelling power, limits WSJ tagging accuracy. ## Key points - Data: WSJ section of the Penn Treebank, split into contiguous parts. - Training: 40,000 sentences / 962,687 words. - Development: 8,000 / 192,826. - Test: 5,485 / 133,805. - Unknown words: 6,107 in development and 3,546 in test. - Baseline on development data with the tag dictionary: 96.43% word accuracy, 86.23% on unknown words, 47.55% whole-sentence accuracy. Without the dictionary, word accuracy is 96.31%. - Final result on the held-out test set: 96.6%. Comparable systems score about 96.5%, including Brill's TBL (85% on unknown words) and Magerman's SDT. - Settings and cost: - Words seen fewer than 5 times count as "rare." - Features occurring fewer than 10 times are discarded, except current-word features. - Beam size is 5. - Training took about 24 hours on an IBM RS/6000. - Word-specific features gave little gain. Tag distributions for words like "about" differ by annotator, and a single-annotator setup yields about 0.5% higher accuracy. ## Interesting ideas - **Rare words stand in for unknown words.** Spelling features are learned only from infrequent training words, which mimic the out-of-vocabulary items seen at test time. This is a cheap way to train OOV handling, and it could transfer to morphologically rich languages like Ukrainian. - **Feature templates with a count cutoff.** Yes/no context templates are instantiated automatically from the training data, and rare features are simply dropped. This replaces complex smoothing with one easy-to-tune threshold, which suits any log-linear or feature-based classifier. - **Tag dictionary as a speed tool.** Known words are allowed only the tags they appeared with in training. The accuracy gain is tiny, but the search space shrinks a lot. This trade-off is useful wherever decoding speed matters. - **Analysing errors by annotator.** Comparing tag distributions per annotator exposes inconsistencies in the gold standard and explains why accuracy plateaus. This is a valuable check before chasing small gains on any treebank, including UD corpora. - **Probabilistic outputs for downstream use.** Because the tagger produces a probability for each decision, it can be plugged into larger probabilistic systems such as parsers or chunkers. Rule-based approaches like TBL cannot offer this. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) Topics: [Treebanks](https://md.research.elamur.ai/free/en/topics/treebanks/) · [Part-of-speech tagging](https://md.research.elamur.ai/free/en/topics/part-of-speech-tagging/) ## Cited by Cited by 300 papers of our library cite it (of the 140884 whose bibliographies we read); 32 of them have a page here, 268 not yet. - [Maximum Entropy Based Restoration of Arabic Diacritics](https://md.research.elamur.ai/free/en/papers/aclbulk-2006-zitouni-maximum-entropy-based-restoration-arabic/) - [Part-of-Speech Tagging for Middle English through Alignment and Projection of Parallel Diachronic Texts](https://md.research.elamur.ai/free/en/papers/aclbulk-2007-moon-part-speech-tagging-middle-english/) - [Identifying Cascading Errors using Constraints in Dependency Parsing](https://md.research.elamur.ai/free/en/papers/aclbulk-2015-ng-identifying-cascading-errors-constraints-dependency/) - [Part-of-Speech Tagging for Historical English](https://md.research.elamur.ai/free/en/papers/aclbulk-2016-yang-part-speech-tagging-historical-english/) - [Validation Issues induced by an Automatic Pre-Annotation Mechanism in the Building of Non-projective Dependency Treebanks](https://md.research.elamur.ai/free/en/papers/bank-2014-lacroix-validation-issues-induced-automatic-pre/) - [Tailoring Continuous Word Representations for Dependency Parsing](https://md.research.elamur.ai/free/en/papers/bansal-2014-tailoring-continuous-word-representations-dependency/) - [Intricacies of Collins' Parsing Model](https://md.research.elamur.ai/free/en/papers/bikel-2004-intricacies-collins-parsing-model/) - [Domain Adaptation with Structural Correspondence Learning](https://md.research.elamur.ai/free/en/papers/blitzer-2006-domain-adaptation-structural-correspondence-learning/) - [TnT – A Statistical Part-of-Speech Tagger](https://md.research.elamur.ai/free/en/papers/brants-2000-tnt-statistical-part-speech-tagger/) - [Head-Driven Statistical Models for Natural Language Parsing](https://md.research.elamur.ai/free/en/papers/cl-2003-collins-head-driven-statistical-models-natural/) - [Three Generative, Lexicalised Models for Statistical Parsing](https://md.research.elamur.ai/free/en/papers/collins-1997-three-generative-lexicalised-models-statistical/) - [Natural Language Processing (almost) from Scratch](https://md.research.elamur.ai/free/en/papers/collobert-2011-natural-language-processing-almost-scratch/) - [Online Learning of Approximate Dependency Parsing Algorithms](https://md.research.elamur.ai/free/en/papers/continual-2006-mcdonald-online-learning-approximate-dependency-parsing/) - [Paraphrase Identification as Probabilistic Quasi-Synchronous Recognition](https://md.research.elamur.ai/free/en/papers/das-2009-paraphrase-identification-as-probabilistic-quasi/) - [Probabilistic Frame-Semantic Parsing](https://md.research.elamur.ai/free/en/papers/das-2010-probabilistic-frame-semantic-parsing/) - [Semi-Supervised Frame-Semantic Parsing for Unknown Predicates](https://md.research.elamur.ai/free/en/papers/das-2011-semi-supervised-frame-semantic-parsing/) - [Frame-Semantic Parsing](https://md.research.elamur.ai/free/en/papers/das-2014-frame-semantic-parsing/) - [Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics?](https://md.research.elamur.ai/free/en/papers/enud-2011-manning-part-speech-tagging-97-100/) - [Structured Perceptron with Inexact Search](https://md.research.elamur.ai/free/en/papers/huang-2012-structured-perceptron-inexact-search/) - [Bidirectional LSTM-CRF Models for Sequence Tagging](https://md.research.elamur.ai/free/en/papers/huang-2015-bidirectional-lstm-crf-models-sequence/) - [Simple Semi-supervised Dependency Parsing](https://md.research.elamur.ai/free/en/papers/koo-2008-simple-semi-supervised-dependency-parsing/) - [A Class of Submodular Functions for Document Summarization](https://md.research.elamur.ai/free/en/papers/lin-2011-class-submodular-functions-document-summarization/) - [Learning Morphology with Morfette](https://md.research.elamur.ai/free/en/papers/lingbulk-2008-chrupala-learning-morphology-morfette/) - [Early results for Named Entity Recognition with Conditional Random Fields, Feature Induction and Web-Enhanced Lexicons](https://md.research.elamur.ai/free/en/papers/mccallum-2003-early-results-named-entity-recognition/) - [Analyzing the Impact of Spelling Errors on POS-Tagging and Chunking in Learner English](https://md.research.elamur.ai/free/en/papers/mizumoto-2017-analyzing-impact-spelling-errors-pos/) - [Online Large-Margin Training of Dependency Parsers](https://md.research.elamur.ai/free/en/papers/narrative-2005-mcdonald-online-large-margin-training-dependency/) - [Chinese Part-of-Speech Tagging: One-at-a-Time or All-at-Once? Word-Based or Character-Based?](https://md.research.elamur.ai/free/en/papers/ng-2004-chinese-part-speech-tagging-one/) - [Improved Part-of-Speech Tagging for Online Conversational Text with Word Clusters](https://md.research.elamur.ai/free/en/papers/owoputi-2013-improved-part-speech-tagging-online/) - [Cross-lingual Word Clusters for Direct Transfer of Linguistic Structure](https://md.research.elamur.ai/free/en/papers/tackstrom-2012-cross-lingual-word-clusters-direct/) - [Enriching the Knowledge Sources Used in a Maximum Entropy Part-of-Speech Tagger](https://md.research.elamur.ai/free/en/papers/toutanova-2000-enriching-knowledge-sources-used-maximum/) - [Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network](https://md.research.elamur.ai/free/en/papers/toutanova-2003-feature-rich-part-speech-tagging/) - [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{ratnaparkhi1996maximum, title = {{A Maximum Entropy Model for Part-Of-Speech Tagging}}, author = {Ratnaparkhi, Adwait}, booktitle = {Conference on Empirical Methods in Natural Language Processing}, year = {1996}, url = {https://aclanthology.org/W96-0213/} } ``` - [← previous A Maximum Entropy Approach to Natural Language Processing](https://md.research.elamur.ai/free/en/papers/berger-1996-maximum-entropy-approach-natural-language/) - [next → A Stochastic Finite-State Word-Segmentation Algorithm for Chinese](https://md.research.elamur.ai/free/en/papers/sproat-1996-stochastic-finite-state-word-segmentation/) - [↑ back to papers of 1996](https://md.research.elamur.ai/free/en/papers/1996/) ## References 13 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. ARPA. 1994. Proceedings of the Human Language Technology Workshop. [find](https://scholar.google.com/scholar?q=ARPA%2E+1994%2E+Proceedings+of+the+Human+Language+Technology+Workshop%2E) 2. Adam Berger, Stephen A. Della Pietra, and Vincent J. Della Pietra. 1996. A Maximum Entropy Approach to Natural Language Processing. Computational Linguistics, 22(1):39-71. [find](https://scholar.google.com/scholar?q=Adam+Berger%2C+Stephen+A%2E+Della+Pietra%2C+and+Vincent+J%2E+Della+Pietra%2E+1996%2E+A+Maximum+Entropy+Approach+to+Natural+Language+Processing%2E+Computational+Linguistics%2C+22%281%29%3A39%2D71%2E) [our card](https://md.research.elamur.ai/free/en/papers/berger-1996-maximum-entropy-approach-natural-language/) 3. Eric Brill. 1994. Some Advances in Transformation-Based Part of Speech Tagging. In Proceedings off the Twelfth National Conference on Artificial Intelligence, volume 1, pages 722-727. [find](https://scholar.google.com/scholar?q=Eric+Brill%2E+1994%2E+Some+Advances+in+Transformation%2DBased+Part+of+Speech+Tagging%2E+In+Proceedings+off+the+Twelfth+National+Conference+on+Artificial+Intelligence%2C+volume+1%2C+pages+722%2D727%2E) 4. Peter F Brown, Vincent DellaPietra, Peter V deSouza, Jennifer C Lai, and Robert L Mercer. 1992. Class-Based n-gram Models of Natural Language. Computational Linguistics, 18(4). [find](https://scholar.google.com/scholar?q=Peter+F+Brown%2C+Vincent+DellaPietra%2C+Peter+V+deSouza%2C+Jennifer+C+Lai%2C+and+Robert+L+Mercer%2E+1992%2E+Class%2DBased+n%2Dgram+Models+of+Natural+Language%2E+Computational+Linguistics%2C+18%284%29%2E) 5. J. N. Darroch and D. Ratcliff. 1972. Generalized Iterative Scaling for Log-Linear Models. The Annals of Mathematical Statistics, 43(5) :1470-1480. [find](https://scholar.google.com/scholar?q=J%2E+N%2E+Darroch+and+D%2E+Ratcliff%2E+1972%2E+Generalized+Iterative+Scaling+for+Log%2DLinear+Models%2E+The+Annals+of+Mathematical+Statistics%2C+43%285%29+%3A1470%2D1480%2E) 6. Steven Della Pietra, Vincent Della Pietra, and John Lafferty. 1995. Inducing Features of Random Fields. Technical Report CMU-CS95-144, School of Computer Science, Carnegie-Mellon University. [find](https://scholar.google.com/scholar?q=Steven+Della+Pietra%2C+Vincent+Della+Pietra%2C+and+John+Lafferty%2E+1995%2E+Inducing+Features+of+Random+Fields%2E+Technical+Report+CMU%2DCS95%2D144%2C+School+of+Computer+Science%2C+Carnegie%2DMellon+University%2E) 7. F Jelinek, J Lafferty, D Magerman, R Mercer, A Ratnaparkhi, and S Roukos. 1994. Decision Tree Parsing using a Hidden Derivational Model. In Proceedings of the Human Language Technology Workshop (ARP, 1994), pages 272-277. [find](https://scholar.google.com/scholar?q=F+Jelinek%2C+J+Lafferty%2C+D+Magerman%2C+R+Mercer%2C+A+Ratnaparkhi%2C+and+S+Roukos%2E+1994%2E+Decision+Tree+Parsing+using+a+Hidden+Derivational+Model%2E+In+Proceedings+of+the+Human+Language+Technology+Workshop+%28ARP%2C+) 8. Ray Lau, Ronald Rosenfeld, and Salim Roukos. 1993. Adaptive Language Modeling Using The Maximum Entropy Principle. In Proceedings of the Human Language Technology Workshop, pages 108-113. ARPA. [find](https://scholar.google.com/scholar?q=Ray+Lau%2C+Ronald+Rosenfeld%2C+and+Salim+Roukos%2E+1993%2E+Adaptive+Language+Modeling+Using+The+Maximum+Entropy+Principle%2E+In+Proceedings+of+the+Human+Language+Technology+Workshop%2C+pages+108%2D113%2E+ARPA%2E) 9. David M. Magerman. 1995. Statistical Decision-Tree Models for Parsing. In Proceedings of the 33rd Annual Meeting of the ACL. [find](https://scholar.google.com/scholar?q=David+M%2E+Magerman%2E+1995%2E+Statistical+Decision%2DTree+Models+for+Parsing%2E+In+Proceedings+of+the+33rd+Annual+Meeting+of+the+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/magerman-1995-statistical-decision-tree-models-parsing/) 10. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Mareinkiewicz. 1994. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19(2):313-330. [find](https://scholar.google.com/scholar?q=Mitchell+P%2E+Marcus%2C+Beatrice+Santorini%2C+and+Mary+Ann+Mareinkiewicz%2E+1994%2E+Building+a+large+annotated+corpus+of+English%3A+the+Penn+Treebank%2E+Computational+Linguistics%2C+19%282%29%3A313%2D330%2E) [our card](https://md.research.elamur.ai/free/en/papers/bank-1993-marcus-building-large-annotated-corpus-english/) 11. Bernard Merialdo. 1994. Tagging English Text with a Probabilistic Model. Computational Linguistics, 20(2):155-172. [find](https://scholar.google.com/scholar?q=Bernard+Merialdo%2E+1994%2E+Tagging+English+Text+with+a+Probabilistic+Model%2E+Computational+Linguistics%2C+20%282%29%3A155%2D172%2E) [our card](https://md.research.elamur.ai/free/en/papers/merialdo-1994-tagging-english-text-probabilistic-model/) 12. Adwait Ratnaparkhi, Jeff Reynar, and Salim Roukos. 1994. A Maximum Entropy Model for Prepositional Phrase Attachment. In Proceedings of the Human Language Technology Workshop (ARP, 1994), pages 250-255. T°tal W°rd Accuracy 1 9 I 6 Unkn°wn . 6 W°rd 3 % Accuracy 85.56% Table 11: Performance of Special… [find](https://scholar.google.com/scholar?q=Adwait+Ratnaparkhi%2C+Jeff+Reynar%2C+and+Salim+Roukos%2E+1994%2E+A+Maximum+Entropy+Model+for+Prepositional+Phrase+Attachment%2E+In+Proceedings+of+the+Human+Language+Technology+Workshop+%28ARP%2C+1994%29%2C+pages+250%2D25) [our card](https://md.research.elamur.ai/free/en/papers/aclbulk-1994-ratnaparkhi-maximum-entropy-model-prepositional-phrase/) 13. Ralph Weischedel, Marie Meteer, Richard Schwartz, Lance Ramshaw, and Jeff Palmucci. 1993. Coping With Ambiguity and Unknown Words through Probabilistic Models. Computational Linguistics, 19(2):359-382. Sentence AccuracY47.51% I [find](https://scholar.google.com/scholar?q=Ralph+Weischedel%2C+Marie+Meteer%2C+Richard+Schwartz%2C+Lance+Ramshaw%2C+and+Jeff+Palmucci%2E+1993%2E+Coping+With+Ambiguity+and+Unknown+Words+through+Probabilistic+Models%2E+Computational+Linguistics%2C+19%282%29%3A359%2D382)