---
title: "Discriminative Training and Maximum Entropy — Och 2002"
description: "Classical statistical MT follows the source-channel scheme."
authors: ["Franz Josef Och", "Hermann Ney"]
year: 2002
venue: "Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P02-1038/"
doi: "10.3115/1073083.1073133"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/och-2002-discriminative-training-maximum-entropy-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/och-2002-discriminative-training-maximum-entropy-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2002](https://md.research.elamur.ai/free/en/papers/2002/) / Discriminative Training and Maximum Entropy Models for Statistical Machine Translation

# Discriminative Training and Maximum Entropy Models for Statistical Machine Translation

**Authors:** [Franz Josef Och](https://md.research.elamur.ai/free/en/authors/franz-josef-och/), [Hermann Ney](https://md.research.elamur.ai/free/en/authors/hermann-ney/) · **Year:** 2002 · **Venue:** Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P02-1038/) · [PDF](https://aclanthology.org/P02-1038.pdf) · [DOI 10.3115/1073083.1073133](https://doi.org/10.3115/1073083.1073133) · [OpenAlex](https://openalex.org/W2154124206)

**Citations:** 1034 citations (openalex-snapshot · 2026-10-14) · [427](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Classical statistical MT follows the source-channel scheme. A target language model and an inverse translation model are trained separately and multiplied with equal weight. That combination is only optimal if both models are exact, and it gives no clean way to add more knowledge. Och and Ney instead model the posterior probability of the translation directly with a log-linear (maximum entropy) model. Every knowledge source becomes a weighted feature function of the source sentence, the target sentence and hidden variables such as alignments. The source-channel model is the special case where two log-probability features both have weight 1. They build this on the alignment template system and train the weights with GIS. The normalization sum is approximated over n-best lists, and the criterion is adapted to several reference translations. Tuning only the weights of the four baseline features already lowers every error measure. Adding a word penalty, a class-based five-gram LM and a conventional dictionary feature improves results further. The framework turns MT into an extensible feature-combination problem and became the basis of later log-linear SMT systems.

## Key points
- Baseline features are a word trigram LM plus three alignment template components: the template alignment model, the template model and the lexicon model.
- Moving from equal weights to trained weights reduced SER from 86.9 to 81.7, WER from 42.8 to 40.2, and raised BLEU from 43.9 to 49.7.
- The full system (ME+WP+CLM+MX) reached SER 77.8, WER 38.4, mWER 31.9, BLEU 55.2, SSER 28.8 and IER 30.9.
- GIS converged after roughly 4000 iterations, with no significant overfitting on the test set.
- The learned scaling factors show large weights for the word penalty (about 2.6–3.0) and the conventional dictionary (2.92). Adding a new feature also shifted the weights of the existing ones.

## Interesting ideas
- **Treat existing component models as log-features with learned weights.** Any pipeline that multiplies separately trained probability models can gain from tuning their relative weights. This holds for ASR, OCR post-correction and tagging alike.
- **Let a curated resource earn its weight.** A feature that counts matches with a hand-made dictionary lets training decide how much to trust it compared with automatically learned lexicons. This is a cheap way to inject manually built lexical resources into a statistical system.
- **Split features by how they fit into search.** Features that decompose locally, such as the word penalty and n-gram LMs, go into dynamic-programming search. Global ones, such as dictionary counts, are applied only when rescoring n-best lists. This keeps decoding tractable while still allowing rich features.
- **Approximate an intractable normalizer with an n-best list.** Summing over a set of highly probable hypotheses instead of all outputs makes discriminative training feasible for structured outputs.
- **Evaluate with both automatic and human-judged measures.** Reporting SER, WER, PER, mWER and BLEU next to SSER and IER shows whether automatic gains match judged translation quality. This is a useful habit when evaluating any generation system.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

427 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 410 not yet.

- [An Expectation Maximization Approach to Pronoun Resolution](https://md.research.elamur.ai/free/en/papers/cherry-2005-expectation-maximization-approach-pronoun-resolution/)
- [A Hierarchical Phrase-Based Model for Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/chiang-2005-hierarchical-phrase-based-model-statistical/)
- [Hierarchical Phrase-Based Translation](https://md.research.elamur.ai/free/en/papers/chiang-2007-hierarchical-phrase-based-translation/)
- [Discriminative Reranking for Natural Language Parsing](https://md.research.elamur.ai/free/en/papers/collins-2000-discriminative-reranking-natural-language-parsing/)
- [Spelling Normalization of Historical Documents by Using a Machine Translation Approach](https://md.research.elamur.ai/free/en/papers/domingo-2018-spelling-normalization-historical-documents-machine/)
- [Loosely Tree-Based Alignment for Machine Translation](https://md.research.elamur.ai/free/en/papers/gildea-2003-loosely-tree-based-alignment-machine/)
- [Feature-Rich Translation by Quasi-Synchronous Lattice Parsing](https://md.research.elamur.ai/free/en/papers/gimpel-2009-feature-rich-translation-quasi-synchronous/)
- [Modelling pronominal anaphora in statistical machine translation](https://md.research.elamur.ai/free/en/papers/hardmeier-2010-modelling-pronominal-anaphora-statistical-machine/)
- [Statistical Syntax-Directed Translation with Extended Domain of Locality](https://md.research.elamur.ai/free/en/papers/huang-2006-statistical-syntax-directed-translation-extended/)
- [A trainable transfer-based MT approach for languages with limited resources](https://md.research.elamur.ai/free/en/papers/lavie-2004-trainable-transfer-based-mt-approach/)
- [Tree-to-String Alignment Template for Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/liu-2006-tree-string-alignment-template-statistical/)
- [Improved Statistical Machine Translation Using Monolingually-Derived Paraphrases](https://md.research.elamur.ai/free/en/papers/marton-2009-improved-statistical-machine-translation-monolingually/)
- [Improving Machine Translation Performance by Exploiting Non-Parallel Corpora](https://md.research.elamur.ai/free/en/papers/munteanu-2005-improving-machine-translation-performance-exploiting/)
- [The Web as a Parallel Corpus](https://md.research.elamur.ai/free/en/papers/resnik-2003-web-as-parallel-corpus/)
- [Inferring Shallow-Transfer Machine Translation Rules from Small Parallel Corpora](https://md.research.elamur.ai/free/en/papers/sanchez-2014-martinez-inferring-shallow-transfer-machine-translation/)
- [Topological Ordering of Function Words in Hierarchical Phrase-based Translation](https://md.research.elamur.ai/free/en/papers/setiawan-2009-topological-ordering-function-words-hierarchical/)
- [Improvements in Phrase-Based Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/zens-2004-improvements-phrase-based-statistical-machine/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{och2002discriminative,
  title = {{Discriminative Training and Maximum Entropy Models for Statistical Machine Translation}},
  author = {Och, Franz Josef and Ney, Hermann},
  booktitle = {Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics},
  year = {2002},
  doi = {10.3115/1073083.1073133},
  url = {https://doi.org/10.3115/1073083.1073133}
}
```

- [← previous Dialogue Act Recognition with Bayesian Networks for Dutch Dialogues](https://md.research.elamur.ai/free/en/papers/keizer-2002-dialogue-act-recognition-bayesian-networks/)
- [next → Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms](https://md.research.elamur.ai/free/en/papers/collins-2002-discriminative-training-methods-hidden-markov/)
- [↑ back to papers of 2002](https://md.research.elamur.ai/free/en/papers/2002/)

## References

15 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. L. R. Bahl, P. F. Brown, P. V. de Souza, and R. L. Mercer. 1986. Maximum mutual information estimation of hidden markov model parameters. In Proc. Int. Conf. on Acoustics, Speech, and Signal Processing, pages 49–52, Tokyo, Japan, April. [find](https://scholar.google.com/scholar?q=L%2E+R%2E+Bahl%2C+P%2E+F%2E+Brown%2C+P%2E+V%2E+de+Souza%2C+and+R%2E+L%2E+Mercer%2E+1986%2E+Maximum+mutual+information+estimation+of+hidden+markov+model+parameters%2E+In+Proc%2E+Int%2E+Conf%2E+on+Acoustics%2C+Speech%2C+and+Signal+Processin)
2. A. L. Berger, S. A. Della Pietra, and V. J. Della Pietra. 1996. A maximum entropy approach to natural language processing. Computational Linguistics, 22(1):39–72, March. [find](https://scholar.google.com/scholar?q=A%2E+L%2E+Berger%2C+S%2E+A%2E+Della+Pietra%2C+and+V%2E+J%2E+Della+Pietra%2E+1996%2E+A+maximum+entropy+approach+to+natural+language+processing%2E+Computational+Linguistics%2C+22%281%29%3A39%E2%80%9372%2C+March%2E) [our card](https://md.research.elamur.ai/free/en/papers/berger-1996-maximum-entropy-approach-natural-language/)
3. P. Beyerlein. 1997. Discriminative model combination. In Proc. of the IEEE Workshop on Automatic Speech Recognition and Understanding, pages 238– 245, Santa Barbara, CA, December. [find](https://scholar.google.com/scholar?q=P%2E+Beyerlein%2E+1997%2E+Discriminative+model+combination%2E+In+Proc%2E+of+the+IEEE+Workshop+on+Automatic+Speech+Recognition+and+Understanding%2C+pages+238%E2%80%93+245%2C+Santa+Barbara%2C+CA%2C+December%2E)
4. P. F. Brown, S. A. Della Pietra, V. J. Della Pietra, and R. L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311. [find](https://scholar.google.com/scholar?q=P%2E+F%2E+Brown%2C+S%2E+A%2E+Della+Pietra%2C+V%2E+J%2E+Della+Pietra%2C+and+R%2E+L%2E+Mercer%2E+1993%2E+The+mathematics+of+statistical+machine+translation%3A+Parameter+estimation%2E+Computational+Linguistics%2C+19%282%29%3A263%E2%80%93311%2E)
5. J. N. Darroch and D. Ratcliff. 1972. Generalized iterative scaling for log-linear models. Annals of Mathematical Statistics, 43:1470–1480. [find](https://scholar.google.com/scholar?q=J%2E+N%2E+Darroch+and+D%2E+Ratcliff%2E+1972%2E+Generalized+iterative+scaling+for+log%2Dlinear+models%2E+Annals+of+Mathematical+Statistics%2C+43%3A1470%E2%80%931480%2E)
6. B. H. Juang, W. Chou, and C. H. Lee. 1995. Statistical and discriminative methods for speech recognition. In A. J. R. Ayuso and J. M. L. Soler, editors, Speech Recognition and Coding - New Advances and Trends. Springer Verlag, Berlin, Germany. [find](https://scholar.google.com/scholar?q=B%2E+H%2E+Juang%2C+W%2E+Chou%2C+and+C%2E+H%2E+Lee%2E+1995%2E+Statistical+and+discriminative+methods+for+speech+recognition%2E+In+A%2E+J%2E+R%2E+Ayuso+and+J%2E+M%2E+L%2E+Soler%2C+editors%2C+Speech+Recognition+and+Coding+%2D+New+Advances+an)
7. H. Ney. 1995. On the probabilistic-interpretation of neural-network classifiers and discriminative training criteria. IEEE Trans. on Pattern Analysis and Machine Intelligence, 17(2):107–119, February. [find](https://scholar.google.com/scholar?q=H%2E+Ney%2E+1995%2E+On+the+probabilistic%2Dinterpretation+of+neural%2Dnetwork+classifiers+and+discriminative+training+criteria%2E+IEEE+Trans%2E+on+Pattern+Analysis+and+Machine+Intelligence%2C+17%282%29%3A107%E2%80%93119%2C+February%2E)
8. S. Nießen, F. J. Och, G. Leusch, and H. Ney. 2000. An evaluation tool for machine translation: Fast evaluation for MT research. In Proc. of the Second Int. Conf. on Language Resources and Evaluation (LREC), pages 39–45, Athens, Greece, May. [find](https://scholar.google.com/scholar?q=S%2E+Nie%C3%9Fen%2C+F%2E+J%2E+Och%2C+G%2E+Leusch%2C+and+H%2E+Ney%2E+2000%2E+An+evaluation+tool+for+machine+translation%3A+Fast+evaluation+for+MT+research%2E+In+Proc%2E+of+the+Second+Int%2E+Conf%2E+on+Language+Resources+and+Evaluation+%28) [our card](https://md.research.elamur.ai/free/en/papers/en-2000-evaluation-tool-machine-translation-fast/)
9. F. J. Och, C. Tillmann, and H. Ney. 1999. Improved alignment models for statistical machine translation. In Proc. of the Joint SIGDAT Conf. on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 20–28, University of Maryland, College Park, MD, June. [find](https://scholar.google.com/scholar?q=F%2E+J%2E+Och%2C+C%2E+Tillmann%2C+and+H%2E+Ney%2E+1999%2E+Improved+alignment+models+for+statistical+machine+translation%2E+In+Proc%2E+of+the+Joint+SIGDAT+Conf%2E+on+Empirical+Methods+in+Natural+Language+Processing+and+Very) [our card](https://md.research.elamur.ai/free/en/papers/och-1999-improved-alignment-models-statistical-machine/)
10. K. A. Papineni, S. Roukos, and R. T. Ward. 1997. Feature-based language understanding. In European Conf. on Speech Communication and Technology, pages 1435–1438, Rhodes, Greece, September. [find](https://scholar.google.com/scholar?q=K%2E+A%2E+Papineni%2C+S%2E+Roukos%2C+and+R%2E+T%2E+Ward%2E+1997%2E+Feature%2Dbased+language+understanding%2E+In+European+Conf%2E+on+Speech+Communication+and+Technology%2C+pages+1435%E2%80%931438%2C+Rhodes%2C+Greece%2C+September%2E)
11. K. A. Papineni, S. Roukos, and R. T. Ward. 1998. Maximum likelihood and discriminative training of direct translation models. In Proc. Int. Conf. on Acoustics, Speech, and Signal Processing, pages 189–192, Seattle, WA, May. [find](https://scholar.google.com/scholar?q=K%2E+A%2E+Papineni%2C+S%2E+Roukos%2C+and+R%2E+T%2E+Ward%2E+1998%2E+Maximum+likelihood+and+discriminative+training+of+direct+translation+models%2E+In+Proc%2E+Int%2E+Conf%2E+on+Acoustics%2C+Speech%2C+and+Signal+Processing%2C+pages+189)
12. K. A. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. 2001. Bleu: a method for automatic evaluation of machine translation. Technical Report RC22176 (W0109-022), IBM Research Division, Thomas J. Watson Research Center, Yorktown Heights, NY, September. [find](https://scholar.google.com/scholar?q=K%2E+A%2E+Papineni%2C+S%2E+Roukos%2C+T%2E+Ward%2C+and+W%2E%2DJ%2E+Zhu%2E+2001%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+Technical+Report+RC22176+%28W0109%2D022%29%2C+IBM+Research+Division%2C+Thomas+J%2E+Watson+R) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
13. J. Peters and D. Klakow. 1999. Compact maximum entropy language models. In Proc. of the IEEE Workshop on Automatic Speech Recognition and Understanding, Keystone, CO, December. [find](https://scholar.google.com/scholar?q=J%2E+Peters+and+D%2E+Klakow%2E+1999%2E+Compact+maximum+entropy+language+models%2E+In+Proc%2E+of+the+IEEE+Workshop+on+Automatic+Speech+Recognition+and+Understanding%2C+Keystone%2C+CO%2C+December%2E)
14. R. Schlüter and H. Ney. 2001. Model-based MCE bound to the true Bayes’ error. IEEE Signal Processing Letters, 8(5):131–133, May. [find](https://scholar.google.com/scholar?q=R%2E+Schlu%CC%88ter+and+H%2E+Ney%2E+2001%2E+Model%2Dbased+MCE+bound+to+the+true+Bayes%E2%80%99+error%2E+IEEE+Signal+Processing+Letters%2C+8%285%29%3A131%E2%80%93133%2C+May%2E)
15. W. Wahlster. 1993. Verbmobil: Translation of face-to-face dialogs. In Proc. of MT Summit IV, pages 127– 135, Kobe, Japan, July. [find](https://scholar.google.com/scholar?q=W%2E+Wahlster%2E+1993%2E+Verbmobil%3A+Translation+of+face%2Dto%2Dface+dialogs%2E+In+Proc%2E+of+MT+Summit+IV%2C+pages+127%E2%80%93+135%2C+Kobe%2C+Japan%2C+July%2E)