--- title: "A Maximum Entropy Approach to Natural Language — Berger 1996" description: "Statistical NLP models have two linked tasks: decide which statistics of the data matter (feature selection), then build a predictive model that respects…" authors: ["Adam L. Berger", "Stephen A. Della Pietra", "Vincent J. Della Pietra"] year: 1996 venue: "Computational Linguistics" source: "https://aclanthology.org/J96-1002/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/berger-1996-maximum-entropy-approach-natural-language/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/berger-1996-maximum-entropy-approach-natural-language/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [1996](https://md.research.elamur.ai/free/en/papers/1996/) / A Maximum Entropy Approach to Natural Language Processing # A Maximum Entropy Approach to Natural Language Processing **Authors:** Adam L. Berger, [Stephen A. Della Pietra](https://md.research.elamur.ai/free/en/authors/stephen-a-della-pietra/), [Vincent J. Della Pietra](https://md.research.elamur.ai/free/en/authors/vincent-j-della-pietra/) · **Year:** 1996 · **Venue:** Computational Linguistics · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/J96-1002/) · [PDF](https://aclanthology.org/J96-1002.pdf) · [OpenAlex](https://openalex.org/W2096175520) **Citations:** 3065 citations (openalex-snapshot · 2026-10-14) · [568](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Statistical NLP models have two linked tasks: decide which statistics of the data matter (feature selection), then build a predictive model that respects them (model selection). Berger and the Della Pietras handle both with the maximum entropy principle. Among all conditional distributions p(y|x) whose expected feature values match those in the training sample, the method chooses the one that is otherwise most uniform. They show this solution belongs to an exponential (log-linear) family with one weight per constraint. It also coincides with the maximum-likelihood model in that family, so entropy-based and likelihood-based reasoning lead to the same answer. They describe an iterative algorithm for fitting the weights, plus a greedy procedure that adds features one at a time from a large candidate pool, with shortcuts that make it practical. They test the framework on components of IBM's Candide French–English translation system: translation-sense disambiguation, sentence segmentation and word reordering. The paper helped make log-linear "MaxEnt" classifiers a standard NLP tool. ## Key points - Features are binary functions of (context x, output y). A constraint is a separate idea: it requires the model's expected feature value to equal the value observed in the training data. - The solution has the form p(y|x) ∝ exp(Σ λᵢ fᵢ(x, y)). Maximizing entropy under the constraints and maximizing likelihood within this exponential family give the same model. - Features are chosen automatically and incrementally from template-generated candidates. At each step the algorithm adds the feature that most improves likelihood, using approximations to keep the search fast. - On French NOUN de NOUN reordering (e.g., *taux d'intérêt* → "interest rate"), the model was trained on 10,000 Hansard corpus events and ended up with 358 constraints. - On 71,555 test phrases, overall accuracy rose from 70.2% (a baseline that never swaps) to 80.4%. Accuracy on phrases that should be swapped went from 0% to 49.2%, while accuracy on phrases that should stay in order fell from 100% to 93.5%. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 568 papers of our library cite it (of the 140673 whose bibliographies we read); 37 of them have a page here, 531 not yet. - [Maximum Entropy Based Restoration of Arabic Diacritics](https://md.research.elamur.ai/free/en/papers/aclbulk-2006-zitouni-maximum-entropy-based-restoration-arabic/) - [Learning Recursive Segments for Discourse Parsing](https://md.research.elamur.ai/free/en/papers/afantenos-2010-learning-recursive-segments-discourse-parsing/) - [Discourse parsing for multi-party chat dialogues](https://md.research.elamur.ai/free/en/papers/afantenos-2015-discourse-parsing-multi-party-chat/) - [Experiments with a Multilanguage Non-Projective Dependency Parser](https://md.research.elamur.ai/free/en/papers/attardi-2006-experiments-multilanguage-non-projective-dependency/) - [Dialogue Act Modeling in a Complex Task-Oriented Domain](https://md.research.elamur.ai/free/en/papers/boyer-2010-dialogue-act-modeling-complex-task/) - [Extending BART to Provide a Coreference Resolution System for German](https://md.research.elamur.ai/free/en/papers/broscheit-2010-extending-bart-provide-coreference-resolution/) - [An Expectation Maximization Approach to Pronoun Resolution](https://md.research.elamur.ai/free/en/papers/cherry-2005-expectation-maximization-approach-pronoun-resolution/) - [Feature Forest Models for Probabilistic HPSG Parsing](https://md.research.elamur.ai/free/en/papers/cl-2008-miyao-feature-forest-models-probabilistic-hpsg/) - [Discriminative Reranking for Natural Language Parsing](https://md.research.elamur.ai/free/en/papers/collins-2000-discriminative-reranking-natural-language-parsing/) - [Clause Restructuring for Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/collins-2005-clause-restructuring-statistical-machine-translation/) - [Joint Determination of Anaphoricity and Coreference Resolution using Integer Programming](https://md.research.elamur.ai/free/en/papers/denis-2007-joint-determination-anaphoricity-coreference-resolution/) - [The Form is the Substance: Classification of Genres in Text](https://md.research.elamur.ai/free/en/papers/dewdney-2001-form-is-substance-classification-genres/) - [Maximum Entropy Models for FrameNet Classification](https://md.research.elamur.ai/free/en/papers/fleischman-2003-maximum-entropy-models-framenet-classification/) - [Named Entity Recognition through Classifier Combination](https://md.research.elamur.ai/free/en/papers/florian-2003-named-entity-recognition-through-classifier/) - [Identifying Agreement and Disagreement in Conversational Speech: Use of Bayesian Networks to Model Pragmatic Dependencies](https://md.research.elamur.ai/free/en/papers/galley-2004-identifying-agreement-disagreement-conversational-speech/) - [Probabilistic Coreference in Information Extraction](https://md.research.elamur.ai/free/en/papers/kehler-1997-probabilistic-coreference-information-extraction/) - [Creating a Coreference Resolution System for Polish](https://md.research.elamur.ai/free/en/papers/kopec-2012-creating-coreference-resolution-system-polish/) - [A Mention-Synchronous Coreference Resolution Algorithm Based On the Bell Tree](https://md.research.elamur.ai/free/en/papers/luo-2004-mention-synchronous-coreference-resolution-algorithm/) - [Multi-Lingual Coreference Resolution With Syntactic Features](https://md.research.elamur.ai/free/en/papers/luo-2005-multi-lingual-coreference-resolution-syntactic/) - [Coreference or Not: A Twin Model for Coreference Resolution](https://md.research.elamur.ai/free/en/papers/luo-2007-coreference-or-not-twin-model/) - [A Comparison of Algorithms for Maximum Entropy Parameter Estimation](https://md.research.elamur.ai/free/en/papers/malouf-2002-comparison-algorithms-maximum-entropy-parameter/) - [Did It Happen? The Pragmatic Complexity of Veridicality Assessment](https://md.research.elamur.ai/free/en/papers/marneffe-2012-did-it-happen-pragmatic-complexity/) - [Learning Noun Phrase Anaphoricity to Improve Conference Resolution: Issues in Representation and Optimization](https://md.research.elamur.ai/free/en/papers/ng-2004-learning-noun-phrase-anaphoricity-improve/) - [Machine Learning for Coreference Resolution: From Local Classification to Global Ranking](https://md.research.elamur.ai/free/en/papers/ng-2005-machine-learning-coreference-resolution-local/) - [Graph-Cut-Based Anaphoricity Determination for Coreference Resolution](https://md.research.elamur.ai/free/en/papers/ng-2009-graph-cut-based-anaphoricity-determination/) - [Supervised Noun Phrase Coreference Research: The First Fifteen Years](https://md.research.elamur.ai/free/en/papers/ng-2010-supervised-noun-phrase-coreference-research/) - [BESTCUT: A Graph Algorithm for Coreference Resolution](https://md.research.elamur.ai/free/en/papers/nicolae-2006-bestcut-graph-algorithm-coreference-resolution/) - [Discriminative Training and Maximum Entropy Models for Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/och-2002-discriminative-training-maximum-entropy-models/) - [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://md.research.elamur.ai/free/en/papers/pereyra-2017-regularizing-neural-networks-penalizing-confident/) - [Integer Linear Programming for Discourse Parsing](https://md.research.elamur.ai/free/en/papers/perret-2016-integer-linear-programming-discourse-parsing/) - [Exploiting Semantic Role Labeling, WordNet and Wikipedia for Coreference Resolution](https://md.research.elamur.ai/free/en/papers/ponzetto-2006-exploiting-semantic-role-labeling-wordnet/) - [Semantic Role Labeling for Coreference Resolution](https://md.research.elamur.ai/free/en/papers/ponzetto-2006-semantic-role-labeling-coreference-resolution/) - [Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech](https://md.research.elamur.ai/free/en/papers/prag-2000-stolcke-dialogue-act-modeling-automatic-tagging/) - [Active Learning Literature Survey](https://md.research.elamur.ai/free/en/papers/settles-2009-active-learning-literature-survey/) - [Hedge Classification in Biomedical Texts with a Weakly Supervised Selection of Keywords](https://md.research.elamur.ai/free/en/papers/szarvas-2008-hedge-classification-biomedical-texts-weakly/) - [Chinese Syntactic Reordering for Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/wang-2007-chinese-syntactic-reordering-statistical-machine/) - [A Twin-Candidate Model for Learning-Based Anaphora Resolution](https://md.research.elamur.ai/free/en/papers/yang-2008-twin-candidate-model-learning-based/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @article{berger1996maximum, title = {{A Maximum Entropy Approach to Natural Language Processing}}, author = {Berger, Adam L. and Pietra, Stephen A. Della and Pietra, Vincent J. Della}, journal = {Computational Linguistics}, year = {1996}, doi = {10.5555/234285.234289}, url = {https://doi.org/10.5555/234285.234289} } ``` - [next → A Unified Theory of Irony and Its Computational Formalization](https://md.research.elamur.ai/free/en/papers/utsumi-1996-unified-theory-irony-its-computational/) - [↑ back to papers of 1996](https://md.research.elamur.ai/free/en/papers/1996/) ## References 22 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Bahl, L.; Brown, P.; de Souza, P.; and Mercer, R. (1989). A tree-based statistical language model for natural language speech recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing, 37(7). [find](https://scholar.google.com/scholar?q=Bahl%2C+L%2E%3B+Brown%2C+P%2E%3B+de+Souza%2C+P%2E%3B+and+Mercer%2C+R%2E+%281989%29%2E+A+tree%2Dbased+statistical+language+model+for+natural+language+speech+recognition%2E+IEEE+Transactions+on+Acoustics%2C+Speech%2C+and+Signal+Processing) 2. Berger, A.; Brown, P.; Della Pietra, S.; Della Pietra, V.; Gillett, J.; Lafferty, J.; Printz, H.; and Urea, L. (1994). The Candide System for Machine Translation. In Proceedings, ARPA Conference on Human Language Technology, Plainsborough, New Jersey. [find](https://scholar.google.com/scholar?q=Berger%2C+A%2E%3B+Brown%2C+P%2E%3B+Della+Pietra%2C+S%2E%3B+Della+Pietra%2C+V%2E%3B+Gillett%2C+J%2E%3B+Lafferty%2C+J%2E%3B+Printz%2C+H%2E%3B+and+Urea%2C+L%2E+%281994%29%2E+The+Candide+System+for+Machine+Translation%2E+In+Proceedings%2C+ARPA+Conference+on+Hu) 3. Black, E.; Jelinek, E; Lafferty, J.; Magerman, D.; Mercer, R.; and Roukos, S. (1992). Towards History-based Grammars: Using Richer Models for Probabilistic Parsing. In Proceedings, DARPA Speech and Natural Language Workshop, Arden House, New York. [find](https://scholar.google.com/scholar?q=Black%2C+E%2E%3B+Jelinek%2C+E%3B+Lafferty%2C+J%2E%3B+Magerman%2C+D%2E%3B+Mercer%2C+R%2E%3B+and+Roukos%2C+S%2E+%281992%29%2E+Towards+History%2Dbased+Grammars%3A+Using+Richer+Models+for+Probabilistic+Parsing%2E+In+Proceedings%2C+DARPA+Speech+and+Na) 4. Brown, D. (1959). A Note on Approximations to Discrete Probability Distributions. Information and Control, 2:386-392. [find](https://scholar.google.com/scholar?q=Brown%2C+D%2E+%281959%29%2E+A+Note+on+Approximations+to+Discrete+Probability+Distributions%2E+Information+and+Control%2C+2%3A386%2D392%2E) 5. Brown, P.; Della Pietra, S.; Della Pietra, V.; and Mercer, R. (1993). The Mathematics of Statistical Machine Translation: Parameter Estimation. Computational Linguistics, 19(2):263--311. [find](https://scholar.google.com/scholar?q=Brown%2C+P%2E%3B+Della+Pietra%2C+S%2E%3B+Della+Pietra%2C+V%2E%3B+and+Mercer%2C+R%2E+%281993%29%2E+The+Mathematics+of+Statistical+Machine+Translation%3A+Parameter+Estimation%2E+Computational+Linguistics%2C+19%282%29%3A263%2D%2D311%2E) 6. Brown, P.; Cocke, J.; Della Pietra, S.; Della Pietra, V.; Jelinek, E; Lafferty, J.; Mercer, R.; and Roossin, P. (1990). A Statistical Approach to Machine Translation. Computational Linguistics, 16:79-85. [find](https://scholar.google.com/scholar?q=Brown%2C+P%2E%3B+Cocke%2C+J%2E%3B+Della+Pietra%2C+S%2E%3B+Della+Pietra%2C+V%2E%3B+Jelinek%2C+E%3B+Lafferty%2C+J%2E%3B+Mercer%2C+R%2E%3B+and+Roossin%2C+P%2E+%281990%29%2E+A+Statistical+Approach+to+Machine+Translation%2E+Computational+Linguistics%2C+16%3A79%2D) 7. Brown, P.; Della Pietra, V.; de Souza, P.; and Mercer, R. (1990). Class-based N-Gram Models of Natural Language. Proceedings, IBM Natural Language ITL, 283-298. [find](https://scholar.google.com/scholar?q=Brown%2C+P%2E%3B+Della+Pietra%2C+V%2E%3B+de+Souza%2C+P%2E%3B+and+Mercer%2C+R%2E+%281990%29%2E+Class%2Dbased+N%2DGram+Models+of+Natural+Language%2E+Proceedings%2C+IBM+Natural+Language+ITL%2C+283%2D298%2E) 8. Brown, P.; Della Pietra, S.; Della Pietra, V.; and Mercer, R. (1991). A Statistical Approach to Sense Disambiguation in Machine Translation. In Proceedings, DARPA Speech and Natural Language Workshop, 146-151. [find](https://scholar.google.com/scholar?q=Brown%2C+P%2E%3B+Della+Pietra%2C+S%2E%3B+Della+Pietra%2C+V%2E%3B+and+Mercer%2C+R%2E+%281991%29%2E+A+Statistical+Approach+to+Sense+Disambiguation+in+Machine+Translation%2E+In+Proceedings%2C+DARPA+Speech+and+Natural+Language+Workshop%2C) 9. Cover, T. and Thomas, J. (1991). Elements of Information Theory. John Wiley & Sons. [find](https://scholar.google.com/scholar?q=Cover%2C+T%2E+and+Thomas%2C+J%2E+%281991%29%2E+Elements+of+Information+Theory%2E+John+Wiley+%26+Sons%2E) 10. Csiszdr, I. (1975). I-Divergence Geometry of Probability Distributions and Minimization Problems, The Annals of Probability, 3(1):146-158. ibid. (1989). A Geometric Interpretation of Darroch and Ratcliff's Generalized Iterative Scaling. The Annals of Statistics, 17(3):1409-1413. [find](https://scholar.google.com/scholar?q=Csiszdr%2C+I%2E+%281975%29%2E+I%2DDivergence+Geometry+of+Probability+Distributions+and+Minimization+Problems%2C+The+Annals+of+Probability%2C+3%281%29%3A146%2D158%2E+ibid%2E+%281989%29%2E+A+Geometric+Interpretation+of+Darroch+and+Ratcl) 11. Csisz~ir, L. and Tusn~idy, G. (1984). Information Geometry and Alternating Minimization Procedures. Statistics & Decisions, Supplemental Issue, no. 1, 205-237. [find](https://scholar.google.com/scholar?q=Csisz%7Eir%2C+L%2E+and+Tusn%7Eidy%2C+G%2E+%281984%29%2E+Information+Geometry+and+Alternating+Minimization+Procedures%2E+Statistics+%26+Decisions%2C+Supplemental+Issue%2C+no%2E+1%2C+205%2D237%2E) 12. Darroch, J. N. and Ratcliff, D. (1972). Generalized Iterative Scaling for Log-linear Models. Annals of Mathematical Statistics, no. 43, 1470-1480. [find](https://scholar.google.com/scholar?q=Darroch%2C+J%2E+N%2E+and+Ratcliff%2C+D%2E+%281972%29%2E+Generalized+Iterative+Scaling+for+Log%2Dlinear+Models%2E+Annals+of+Mathematical+Statistics%2C+no%2E+43%2C+1470%2D1480%2E) 13. Della Pietra, S.; Della Pietra, V.; Gillett, J.; Lafferty, J.; Printz, H.; and Urea, L. (1994). "Inference and Estimation of a Long-range Trigram Model." In Lecture Notes in Artificfal Intelligence, 862. Springer-Verlag, 78-92. [find](https://scholar.google.com/scholar?q=Della+Pietra%2C+S%2E%3B+Della+Pietra%2C+V%2E%3B+Gillett%2C+J%2E%3B+Lafferty%2C+J%2E%3B+Printz%2C+H%2E%3B+and+Urea%2C+L%2E+%281994%29%2E+%22Inference+and+Estimation+of+a+Long%2Drange+Trigram+Model%2E%22+In+Lecture+Notes+in+Artificfal+Intelligence%2C+8) 14. Della Pietra, S.; Della Pietra, V.; and Lafferty, J. (1995). "Inducing features of random fields" CMU Technical Report CMU-CS-95-144. [find](https://scholar.google.com/scholar?q=Della+Pietra%2C+S%2E%3B+Della+Pietra%2C+V%2E%3B+and+Lafferty%2C+J%2E+%281995%29%2E+%22Inducing+features+of+random+fields%22+CMU+Technical+Report+CMU%2DCS%2D95%2D144%2E) 15. Dempster, A. P.; Laird, N. M.; and Rubin, D. B. (1977). Maximum Likelihood from Incomplete Data via the EM Algorithm. Journal of the Royal Statistical Society, 39(B):1-38. [find](https://scholar.google.com/scholar?q=Dempster%2C+A%2E+P%2E%3B+Laird%2C+N%2E+M%2E%3B+and+Rubin%2C+D%2E+B%2E+%281977%29%2E+Maximum+Likelihood+from+Incomplete+Data+via+the+EM+Algorithm%2E+Journal+of+the+Royal+Statistical+Society%2C+39%28B%29%3A1%2D38%2E) 16. Guiasu, S. and Shenitzer, A. (1985). The Principle of Maximum Entropy. The Mathematical Intelligencer , 7(1). [find](https://scholar.google.com/scholar?q=Guiasu%2C+S%2E+and+Shenitzer%2C+A%2E+%281985%29%2E+The+Principle+of+Maximum+Entropy%2E+The+Mathematical+Intelligencer+%2C+7%281%29%2E) 17. Jaynes, E. T. (1990) "Notes on Present Status and Future Prospects." In Maximum Entropy and Bayesian Methods, edited by W. T. Grandy and L. H. Schick. Kluwer, 1-13. [find](https://scholar.google.com/scholar?q=Jaynes%2C+E%2E+T%2E+%281990%29+%22Notes+on+Present+Status+and+Future+Prospects%2E%22+In+Maximum+Entropy+and+Bayesian+Methods%2C+edited+by+W%2E+T%2E+Grandy+and+L%2E+H%2E+Schick%2E+Kluwer%2C+1%2D13%2E) 18. Jelinek, F. and Mercer, R. L. (1980). Interpolated Estimation of Markov Source Parameters from Sparse Data. In Proceedings, Workshop on Pattern Recognition in Practice, Amsterdam, The Netherlands. [find](https://scholar.google.com/scholar?q=Jelinek%2C+F%2E+and+Mercer%2C+R%2E+L%2E+%281980%29%2E+Interpolated+Estimation+of+Markov+Source+Parameters+from+Sparse+Data%2E+In+Proceedings%2C+Workshop+on+Pattern+Recognition+in+Practice%2C+Amsterdam%2C+The+Netherlands%2E) 19. Lucassen, J. and Mercer, R. (1984). An Information Theoretic Approach to Automatic Determination of Phonemic Baseforms. In Proceedings, IEEE International Conference on Acoustics, Speech and Signal Processing, San Diego, CA, 42.5.1-42.5.4. [find](https://scholar.google.com/scholar?q=Lucassen%2C+J%2E+and+Mercer%2C+R%2E+%281984%29%2E+An+Information+Theoretic+Approach+to+Automatic+Determination+of+Phonemic+Baseforms%2E+In+Proceedings%2C+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Pr) 20. Merialdo, B. (1990). Tagging Text with a Probabilistic Model. In Proceedings, IBM Natural Language ITL, Paris, France, 161-172. [find](https://scholar.google.com/scholar?q=Merialdo%2C+B%2E+%281990%29%2E+Tagging+Text+with+a+Probabilistic+Model%2E+In+Proceedings%2C+IBM+Natural+Language+ITL%2C+Paris%2C+France%2C+161%2D172%2E) 21. Nddas, A.; Mercer, R.; Bahl, L.; Bakis, R.; Cohen, P.; Cole, A.-; Jelinek, F.; and Lewis, B. (1981). Continuous Speech Recognition with Automatically Selected Acoustic Prototypes Obtained by either Bootstrapping or Clustering. In Proceedings, IEEE International Conference on Acoustics, Speech and Si… [find](https://scholar.google.com/scholar?q=Nddas%2C+A%2E%3B+Mercer%2C+R%2E%3B+Bahl%2C+L%2E%3B+Bakis%2C+R%2E%3B+Cohen%2C+P%2E%3B+Cole%2C+A%2E%2D%3B+Jelinek%2C+F%2E%3B+and+Lewis%2C+B%2E+%281981%29%2E+Continuous+Speech+Recognition+with+Automatically+Selected+Acoustic+Prototypes+Obtained+by+either+Bo) 22. Sokolnikoff, I. S. and Redheffer, R. M. (1966). Mathematics of Physics and Modern Engineering, Second Edition, McGraw-Hill Book Company. [find](https://scholar.google.com/scholar?q=Sokolnikoff%2C+I%2E+S%2E+and+Redheffer%2C+R%2E+M%2E+%281966%29%2E+Mathematics+of+Physics+and+Modern+Engineering%2C+Second+Edition%2C+McGraw%2DHill+Book+Company%2E)