--- title: "Improved Statistical Alignment Models — Och 2000" description: "Och and Ney address two problems in statistical word alignment for machine translation: there was no established way to evaluate alignment models…" authors: ["Franz Josef Och", "Hermann Ney"] year: 2000 venue: "Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics" source: "https://aclanthology.org/P00-1056/" doi: "10.3115/1075218.1075274" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/och-2000-improved-statistical-alignment-models/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/och-2000-improved-statistical-alignment-models/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2000](https://md.research.elamur.ai/free/en/papers/2000/) / Improved Statistical Alignment Models # Improved Statistical Alignment Models **Authors:** [Franz Josef Och](https://md.research.elamur.ai/free/en/authors/franz-josef-och/), [Hermann Ney](https://md.research.elamur.ai/free/en/authors/hermann-ney/) · **Year:** 2000 · **Venue:** Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/P00-1056/) · [PDF](https://aclanthology.org/P00-1056.pdf) · [DOI 10.3115/1075218.1075274](https://doi.org/10.3115/1075218.1075274) · [OpenAlex](https://openalex.org/W1973923101) **Citations:** 1001 citations (openalex-snapshot · 2026-10-14) · [466](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Och and Ney address two problems in statistical word alignment for machine translation: there was no established way to evaluate alignment models directly, and the models themselves needed improvement. They propose evaluating a model by comparing its Viterbi alignment against a manually produced reference alignment. Annotators mark both certain ("sure") and ambiguous ("possible") links, and the comparison yields precision, recall and an alignment error rate (AER). Using this measure, they compare IBM-1 through IBM-5 and the HMM alignment model on German–English (Verbmobil) and French–English (Hansards) data. They also test several training refinements: - smoothing of alignment and fertility parameters, - word-class dependencies, - different alignment subsets for EM count collection, - bootstrapping schemes. Fertility-based models with a first-order alignment dependency clearly beat the simple IBM-1 and IBM-2. Starting IBM-3 training from the HMM instead of IBM-2 further improves alignment quality. Combining the Viterbi alignments from both translation directions can raise precision or recall, depending on the merging method. The work matters because it provides a reproducible, automatic way to judge design decisions in alignment modelling. ## Key points - Six models are compared: - IBM-1 and IBM-2 use zero-order alignment. - The HMM uses first-order alignment. - IBM-3 and IBM-4 add explicit fertility. - IBM-5 is a non-deficient reformulation of IBM-4. An "empty word" absorbs source words that have no counterpart. - Evaluation compares the Viterbi alignment with a hand-made reference containing sure and possible links, producing precision, recall and AER. - Training uses EM over a parallel corpus. - IBM-1, IBM-2 and the HMM allow exact count collection; the HMM uses the Baum–Welch algorithm. - For IBM-3 to IBM-5, counts are gathered only over a small set of promising alignments. Three options are compared: the Viterbi alignment alone, its neighbours (one move or swap away), and an even larger set that includes "pegged" alignments. - Using the HMM rather than IBM-2 as the starting point for IBM-3 to IBM-5 significantly improves alignment quality. - Word classes are learned automatically with a separate statistical clustering procedure (Kneser & Ney). They condition the alignment probabilities in the HMM, IBM-4 and IBM-5. ## Interesting ideas - **Intrinsic evaluation against a gold standard with graded confidence.** Separating sure from possible links lets the reference reflect genuine annotator uncertainty without penalising either choice. The same scheme could work for evaluating other ambiguous structure-prediction tasks, such as aligning Ukrainian–English parallel data or matching entities across datasets. - **Length-aware fertility smoothing.** Fertility estimates are interpolated with a general distribution that depends only on word length in letters, since longer (e.g. German compound) words tend to align to more words. The specific estimate dominates for frequent words and the general one for rare words. This is a cheap backoff trick for any morphologically rich language with sparse vocabulary, Ukrainian included. - **Interpolation with a uniform distribution, tuned on held-out data.** Alignment probabilities are mixed with a constant distribution, and the mixing weights are chosen to maximise alignment quality on a validation set. Tuning smoothing for the downstream metric rather than likelihood is broadly reusable. - **Bootstrapping complex models from better simple ones.** Initialising an expensive model from a stronger intermediate model (HMM instead of IBM-2) improves the final result. This is a reminder that training pipelines and initialisation order deserve explicit experiments. - **Symmetrising directional outputs.** Merging the alignments from both translation directions by intersection, union, or a refined in-between rule lets you trade precision against recall. The same applies to combining outputs of any two asymmetric predictors. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 466 papers of our library cite it (of the 140673 whose bibliographies we read); 31 of them have a page here, 435 not yet. - [Word Sense Disambiguation Improves Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/chan-2007-word-sense-disambiguation-improves-statistical/) - [A Hierarchical Phrase-Based Model for Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/chiang-2005-hierarchical-phrase-based-model-statistical/) - [Hierarchical Phrase-Based Translation](https://md.research.elamur.ai/free/en/papers/chiang-2007-hierarchical-phrase-based-translation/) - [Czech-English Dependency Tree-based Machine Translation](https://md.research.elamur.ai/free/en/papers/cmejrek-2003-czech-english-dependency-tree-based/) - [Constructing Corpora for the Development and Evaluation of Paraphrase Systems](https://md.research.elamur.ai/free/en/papers/cohn-2008-constructing-corpora-development-evaluation-paraphrase/) - [Survey: Multiword Expression Processing: A Survey](https://md.research.elamur.ai/free/en/papers/constant-2017-survey-multiword-expression-processing-survey/) - [Learning to Simplify Sentences Using Wikipedia](https://md.research.elamur.ai/free/en/papers/coster-2011-learning-simplify-sentences-wikipedia/) - [Unsupervised Construction of Large Paraphrase Corpora: Exploiting Massively Parallel News Sources](https://md.research.elamur.ai/free/en/papers/dolan-2004-unsupervised-construction-large-paraphrase-corpora/) - [Automatically Constructing a Corpus of Sentential Paraphrases](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/) - [Paraphrase-Driven Learning for Open Question Answering](https://md.research.elamur.ai/free/en/papers/fader-2013-paraphrase-driven-learning-open-question/) - [What's in a translation rule?](https://md.research.elamur.ai/free/en/papers/galley-2004-what-s-translation-rule/) - [Jointly Learning to Align and Translate with Transformer Models](https://md.research.elamur.ai/free/en/papers/garg-2019-jointly-learning-align-translate-transformer/) - [Loosely Tree-Based Alignment for Machine Translation](https://md.research.elamur.ai/free/en/papers/gildea-2003-loosely-tree-based-alignment-machine/) - [Tuning as Ranking](https://md.research.elamur.ai/free/en/papers/hopkins-2011-tuning-as-ranking/) - [Learning a Lexical Simplifier Using Wikipedia](https://md.research.elamur.ai/free/en/papers/horn-2014-learning-lexical-simplifier-wikipedia/) - [Attention is Not Only a Weight: Analyzing Transformers with Vector Norms](https://md.research.elamur.ai/free/en/papers/kobayashi-2020-attention-is-not-only-weight/) - [Learning Domain-Specific Transfer Rules: An Experiment with Korean to English Translation](https://md.research.elamur.ai/free/en/papers/lavoie-2002-learning-domain-specific-transfer-rules/) - [Tree-to-String Alignment Template for Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/liu-2006-tree-string-alignment-template-statistical/) - [N-gram-based Machine Translation](https://md.research.elamur.ai/free/en/papers/marino-2006-n-gram-based-machine-translation/) - [Soft Syntactic Constraints for Hierarchical Phrased-Based Translation](https://md.research.elamur.ai/free/en/papers/marton-2008-soft-syntactic-constraints-hierarchical-phrased/) - [Improved Statistical Machine Translation Using Monolingually-Derived Paraphrases](https://md.research.elamur.ai/free/en/papers/marton-2009-improved-statistical-machine-translation-monolingually/) - [Grammar-Based Concept Alignment for Domain-Specific Machine Translation](https://md.research.elamur.ai/free/en/papers/masciolini-2021-grammar-based-concept-alignment-domain/) - [Coreference in Prague Czech-English Dependency Treebank](https://md.research.elamur.ai/free/en/papers/nedoluzhko-2016-coreference-prague-czech-english-dependency/) - [Cross-lingual Coreference Resolution of Pronouns](https://md.research.elamur.ai/free/en/papers/novak-2014-cross-lingual-coreference-resolution-pronouns/) - [Projection-based Coreference Resolution Using Deep Syntax](https://md.research.elamur.ai/free/en/papers/novak-2017-projection-based-coreference-resolution-deep/) - [Monolingual Machine Translation for Paraphrase Generation](https://md.research.elamur.ai/free/en/papers/quirk-2004-monolingual-machine-translation-paraphrase-generation/) - [Translation-Based Projection for Multilingual Coreference Resolution](https://md.research.elamur.ai/free/en/papers/rahman-2012-translation-based-projection-multilingual-coreference/) - [Maximum Entropy Based Phrase Reordering Model for Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/xiong-2006-maximum-entropy-based-phrase-reordering/) - [Inducing Multilingual Text Analysis Tools via Robust Projection across Aligned Corpora](https://md.research.elamur.ai/free/en/papers/yarowsky-2001-inducing-multilingual-text-analysis-tools/) - [Investigating Explicitation of Discourse Connectives in Translation using Automatic Annotations](https://md.research.elamur.ai/free/en/papers/yung-2023-investigating-explicitation-discourse-connectives-translation/) - [Cross-Language Parser Adaptation between Related Languages](https://md.research.elamur.ai/free/en/papers/zeman-2008-cross-language-parser-adaptation-between/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{och2000improved, title = {{Improved Statistical Alignment Models}}, author = {Och, Franz Josef and Ney, Hermann}, booktitle = {Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics}, year = {2000}, doi = {10.3115/1075218.1075274}, url = {https://doi.org/10.3115/1075218.1075274} } ``` - [← previous Importance of Pronominal Anaphora Resolution in Question Answering Systems](https://md.research.elamur.ai/free/en/papers/vicedo-2000-importance-pronominal-anaphora-resolution-question/) - [next → Integrating Subject Field Codes into WordNet](https://md.research.elamur.ai/free/en/papers/magnini-2000-integrating-subject-field-codes-into/) - [↑ back to papers of 2000](https://md.research.elamur.ai/free/en/papers/2000/)