---
title: "Machine translation — papers and Mova's translator"
description: "Machine translation in plain words, from rules to neural networks, Mova's deterministic translator through a meta-world, and our cards of the papers on MT."
canonical: "https://research.elamur.ai/free/en/topics/machine-translation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/topics/machine-translation/)

[home](https://md.research.elamur.ai/free/en/) / [topics](https://md.research.elamur.ai/free/en/topics/) / Machine translation

# 🔁 Machine translation

718 papers · page 1 of 29 · the most cited first · [all topics](https://md.research.elamur.ai/free/en/topics/)

Translating text from one language to another by program.

💡 Explained: [BLEU score](https://md.research.elamur.ai/free/en/concepts/bleu/) · [Transformer and attention](https://md.research.elamur.ai/free/en/concepts/transformer/)

## What it is

Machine translation, MT for short, translates text from one language to another by program. The first systems were rules written by hand: dictionaries and grammars of transfer from one language's structure to another's, or an interlingua that belongs to no language. Statistical systems then learned translation tables from aligned texts, and neural networks now translate whole sentences at once.

Translation is hard because languages cut the world differently: words have no exact match, word order changes, and style matters as much as meaning. Systems are measured with people's judgments and with automatic scores such as BLEU that compare the output with human translations.

## How Mova does it

Mova translates without a neural network: a text is read down its language's stack into a tree, frames and a world, then into a meta-world that belongs to no language, and written out by the target language's encoder with the target author's delta. The test is byte for byte on verse-aligned Bibles, and what is lost is measured in bits at every stage.

- [translate — the translator through a meta-world](https://mova.elamur.ai/free/en/docs/tools/translate/)
- [roundtrip — where each layer loses information](https://mova.elamur.ai/free/en/docs/tools/roundtrip/)
- [The machine translation shelf of the paper cards](https://md.research.elamur.ai/free/en/shelves/mtrans/)

A survey · about 19 minutes to read · every paper named here has its own page in this library

**In this article**

1. [What it is](#a-what-it-is)
2. [History and milestones](#a-history-and-milestones)
3. [Main problems and tasks](#a-main-problems-and-tasks)
4. [Methods and schools](#a-methods-and-schools)
5. [Datasets, resources and benchmarks](#a-datasets-resources-and-benchmarks)
6. [Landmark papers in this library](#a-landmark-papers-in-this-library)
7. [People](#a-people)
8. [Institutions and places](#a-institutions-and-places)
9. [Open questions](#a-open-questions)
10. [Neighbouring topics](#a-neighbouring-topics)
11. [Key works in this library](#a-key-works-in-this-library)

Machine translation (MT) is the use of computer programs to render text or speech in one human language as text or speech in another. It is one of the oldest problems in computing: proposals to mechanise translation date back to the late 1940s, and the first public demonstration of a translating program took place in 1954. For decades the problem served as a proving ground for whole schools of thought about language — hand-written grammars and dictionaries, probabilistic models estimated from bilingual text, and finally neural networks trained end to end. Each wave reshaped not only translation but natural language processing as a whole; the attention mechanism and the Transformer architecture, now at the heart of large language models, were first developed for translation.

This topic gathers the papers in this library on translation systems and on everything that surrounds them: word and sentence alignment, decoding and tuning, the parallel corpora that feed the models, and, prominently, the question of how translation quality should be measured. Evaluation is unusually central here. Automatic metrics such as BLEU, METEOR, chrF and COMET, the annual shared tasks of the Conference on Machine Translation (WMT), and the debates about whether machines have reached human parity make up a large share of the most cited work.

The collection spans five decades, from early rule-based and interlingua research to multilingual neural systems and translation with large language models. Its centre of gravity lies in the statistical and early neural eras, with a renewed surge of papers in the most recent years on low-resource languages, document-level translation and evaluation with learned metrics.

## What it is

At its simplest, a translation system maps a source sentence to a target sentence that preserves its meaning and reads naturally in the target language. These two demands — **adequacy** (faithfulness to the source) and **fluency** (well-formedness in the target) — have shaped both system design and evaluation since the field began. Real translation also involves register, terminology, cultural reference, document-level coherence and the many cases where languages simply do not line up word for word.

Several core notions recur across all eras:

- **Parallel corpora (bitexts)**: collections of texts paired with their translations, aligned at the document, sentence and often word level. They are the raw material of every data-driven system.
- **Alignment**: the correspondence between units of the source and the target — sentences, phrases or words. Learning it, explicitly or implicitly, is the heart of translation modelling.
- **Reordering**: languages differ in word order (English subject–verb–object against Japanese subject–object–verb, German verb-final subordinate clauses), so a system must decide where each piece goes.
- **Decoding**: searching among the very large number of possible outputs for the one the model scores highest, usually with beam search.
- **Evaluation**: judging output quality, either by human assessment or by automatic metrics that compare system output with reference translations.

Machine translation differs from its neighbours in its output: unlike classification tasks, it produces open-ended text, and there are always many correct answers. This links it closely to [text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) and summarisation, which inherited much of its methodology, and to [multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/), which shares its concern with many languages but usually aims at analysis rather than generation. Several techniques first developed for translation — phrase-based models, paraphrase extraction from bitexts, sequence-to-sequence learning — were later applied to monolingual rewriting tasks such as paraphrasing, simplification and error correction, and some of those papers stand on this topic too.

## History and milestones

Landmark papers on this topic: the 12 most cited, in time The 12 most cited papers on this topic span 2002–2020; the dot grows with the citations.

**The numbers**

| Paper | Citations |
| --- | --- |
| [2002 — Bleu: a Method for Automatic Evaluation of Machine Translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/) | 32,065 |
| [2004 — Statistical Significance Tests for Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/) | 2,444 |
| [2005 — METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/) | 5,483 |
| [2014 — Neural Machine Translation by Jointly Learning to Align and Translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/) | 18,908 |
| [2014 — On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/) | 7,734 |
| [2014 — Meteor Universal: Language Specific Translation Evaluation for Any Target Language](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/) | 2,392 |
| [2015 — chrF: character n-gram F-score for automatic MT evaluation](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/) | 2,105 |
| [2016 — Neural Machine Translation of Rare Words with Subword Units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/) | 3,409 |
| [2017 — Attention Is All You Need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/) | 16,413 |
| [2017 — OpenNMT: Open-Source Toolkit for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/klein-2017-opennmt-open-source-toolkit-neural/) | 2,460 |
| [2018 — A Call for Clarity in Reporting BLEU Scores](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/) | 4,130 |
| [2020 — COMET: A Neural Framework for MT Evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/) | 1,784 |

### Rule-based and interlingua systems

The first generation of MT systems, from the 1950s to the 1980s, relied on bilingual dictionaries and hand-written rules. Direct systems replaced words and applied local fixes; transfer systems parsed the source, transformed its structure into a target structure, and generated output; interlingua systems aimed to map the source into a language-neutral meaning representation from which any target could be produced. The 1966 ALPAC report in the United States judged early progress disappointing and sharply reduced funding, but rule-based systems such as SYSTRAN remained in commercial use for decades. Work from this period in the library includes Bonnie Dorr's formal account of [translation divergences](https://md.research.elamur.ai/free/en/papers/dorr-1994-machine-translation-divergences-formal-description/) — systematic structural mismatches, such as one language expressing with a verb what another expresses with a preposition — and a review of [the ARPA MT evaluation methodologies](https://md.research.elamur.ai/free/en/papers/white-1994-arpa-mt-evaluation-methodologies-evolution/) from the early 1990s.

### The statistical turn

In the late 1980s a group at IBM proposed treating translation as a problem of probability estimation from bilingual text. Their paper [A Statistical Approach to Language Translation](https://md.research.elamur.ai/free/en/papers/brown-1988-statistical-approach-language-translation/) introduced the noisy-channel view: a target sentence is chosen by combining a language model, which judges fluency, with a translation model, which judges how well the target accounts for the source. The family of IBM word-alignment models that followed, trained on the Canadian parliamentary Hansards, defined the field for the next decade. Practical prerequisites were solved at the same time, notably the length-based [sentence alignment program of Gale and Church](https://md.research.elamur.ai/free/en/papers/gale-1993-program-aligning-sentences-bilingual-corpora/).

Around 2000 the unit of translation moved from words to phrases. The alignment template approach of Och, Tillmann and Ney, [refinements of the alignment models](https://md.research.elamur.ai/free/en/papers/och-2000-improved-statistical-alignment-models/) together with a gold-standard alignment error rate, the [joint phrase model of Marcu and Wong](https://md.research.elamur.ai/free/en/papers/marcu-2002-phrase-based-joint-probability-model/), and the [log-linear framework](https://md.research.elamur.ai/free/en/papers/och-2002-discriminative-training-maximum-entropy-models/) that replaced the fixed noisy-channel product with weighted feature functions led to phrase-based statistical MT. The open-source Moses toolkit (2007) made this architecture the standard research baseline. In the same period, the BLEU metric (2002) gave researchers a cheap, repeatable way to measure progress, and annual shared tasks — NIST evaluations and, from 2006, the Workshop on Statistical Machine Translation — turned the field into a sequence of public benchmarks.

### Syntax and hierarchy

A parallel line of work sought to bring linguistic structure into statistical models. [Yamada and Knight](https://md.research.elamur.ai/free/en/papers/yamada-2001-syntax-based-statistical-translation-model/) modelled translation as operations on a parse tree; [Chiang's hierarchical phrase-based model](https://md.research.elamur.ai/free/en/papers/chiang-2005-hierarchical-phrase-based-model-statistical/) learned a synchronous grammar with gaps from unannotated bitext; [Galley and colleagues](https://md.research.elamur.ai/free/en/papers/galley-2006-scalable-inference-training-context-rich/) scaled up syntactic rule extraction; [dependency treelet translation](https://md.research.elamur.ai/free/en/papers/quirk-2005-dependency-treelet-translation-syntactically-informed/) used source dependency parses. Others kept the phrase-based engine but reordered the input with rules, as in [clause restructuring for German](https://md.research.elamur.ai/free/en/papers/collins-2005-clause-restructuring-statistical-machine-translation/).

### The neural turn

From 2013 neural networks entered translation, first as features inside statistical systems — such as the [neural network joint model](https://md.research.elamur.ai/free/en/papers/devlin-2014-fast-robust-neural-network-joint/) — and then as complete systems. [Recurrent continuous translation models](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-continuous-translation-models/) showed that a single network could condition a language model on a source sentence. Encoder–decoder models of 2014 compressed the source into a vector, and the [attention mechanism of Bahdanau, Cho and Bengio](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/) let the decoder look back at every source position. Within two to three years neural MT overtook phrase-based systems in shared tasks and commercial deployment. The Transformer (2017), built entirely on attention, became the standard architecture; subword segmentation, back-translation of monolingual data and multilingual training followed.

### Multilinguality, evaluation and large language models

Since about 2018 the field has moved in several directions at once: massively multilingual models covering a hundred languages or more, unsupervised translation from monolingual text, document-level and context-aware translation, and a reckoning with evaluation, as claims of human parity met careful critique and neural metrics replaced string overlap. Most recently, general-purpose large language models have become competitive translators, raising new questions about prompting, hallucination and how to evaluate systems that may already exceed the references they are compared with.

Papers on this topic by year 718 dated papers from 1976 to 2026; the busiest year is 2026 with 128, and 54% of them appeared in 2018 or later.

**The numbers**

| Year | Papers |
| --- | --- |
| 2026 | 128 |
| 2025 | 11 |
| 2024 | 28 |
| 2023 | 36 |
| 2022 | 16 |
| 2021 | 17 |
| 2020 | 41 |
| 2019 | 52 |
| 2018 | 59 |
| 2017 | 50 |
| 2016 | 38 |
| 2015 | 14 |
| 2014 | 14 |
| 2013 | 11 |
| 2012 | 19 |
| 2011 | 8 |
| 2010 | 11 |
| 2009 | 14 |
| 2008 | 8 |
| 2007 | 19 |
| 2006 | 16 |
| 2005 | 15 |
| 2004 | 12 |
| 2003 | 8 |
| 2002 | 7 |
| 2001 | 2 |
| 2000 | 5 |
| 1999 | 4 |
| 1998 | 1 |
| 1997 | 4 |
| 1996 | 2 |
| 1995 | 5 |
| 1994 | 11 |
| 1993 | 2 |
| 1992 | 3 |
| 1991 | 5 |
| 1990 | 5 |
| 1989 | 4 |
| 1988 | 4 |
| 1987 | 1 |
| 1986 | 3 |
| 1985 | 1 |
| 1982 | 2 |
| 1981 | 1 |
| 1976 | 1 |

## Main problems and tasks

The core task is sentence-level translation of text, but the field divides into many sub-problems and settings:

- **Word and sentence alignment**: recovering correspondences in parallel text, both as a training step and as a task in itself.
- **Reordering**: placing translated material in the right order, especially between structurally distant languages.
- **Vocabulary and rare words**: handling names, numbers, morphological variants and words never seen in training. In neural systems this led to work on [large target vocabularies](https://md.research.elamur.ai/free/en/papers/jean-2015-very-large-target-vocabulary-neural/), [copying via alignment pointers](https://md.research.elamur.ai/free/en/papers/luong-2015-addressing-rare-word-problem-neural/), subword units and [character-level models](https://md.research.elamur.ai/free/en/papers/lee-2017-fully-character-level-neural-machine/).
- **Domain adaptation**: transferring a system trained on parliamentary or news text to medicine, law, software or conversation; see [Koehn and Schroeder's experiments](https://md.research.elamur.ai/free/en/papers/koehn-2007-experiments-domain-adaptation-statistical-machine/) and the [survey of adaptation for neural MT](https://md.research.elamur.ai/free/en/papers/chu-2018-survey-domain-adaptation-neural-machine/).
- **Low-resource translation**: building systems when parallel data is scarce, through transfer, multilinguality, monolingual data and data augmentation.
- **Document-level translation**: keeping pronouns, terminology, tense and style consistent across sentences.
- **Speech translation and simultaneous translation**: translating spoken input, sometimes while the speaker is still talking.
- **Multimodal translation**: using images alongside text, as in the [shared task on multimodal MT](https://md.research.elamur.ai/free/en/papers/specia-2016-shared-task-multimodal-machine-translation/).
- **Quality estimation**: predicting the quality of a translation without a reference, starting from [sentence-level estimation](https://md.research.elamur.ai/free/en/papers/specia-2009-estimating-sentence-level-quality-machine/).
- **Automatic post-editing**: correcting the output of a black-box MT system, a WMT task since the mid-2010s.
- **Evaluation**: designing metrics and human protocols that can rank systems reliably.
- **Bias and robustness**: detecting and reducing stereotyped choices (for example of grammatical gender), hallucinated content and sensitivity to noisy input.

## Methods and schools

### Rule-based and example-based MT

Rule-based systems encode linguistic knowledge in dictionaries and transfer grammars written by experts. They give predictable, controllable output but are expensive to build and brittle outside their coverage. Example-based MT, proposed in Japan in the 1980s, translates by analogy with stored examples, an idea that in some ways anticipated phrase-based statistics.

### Statistical machine translation

Statistical MT estimates all model components from data. The IBM models and the HMM aligner learn word correspondences with expectation maximisation; [alignment by agreement](https://md.research.elamur.ai/free/en/papers/liang-2006-alignment-agreement/) showed that training two directional aligners jointly beats intersecting them afterwards. Phrase-based systems extract phrase pairs consistent with the word alignment and score them with relative frequencies; [factored models](https://md.research.elamur.ai/free/en/papers/koehn-2007-factored-translation-models/) extended phrases with lemmas, part-of-speech tags and morphology. A log-linear model combines translation, reordering and language-model features, and the weights are tuned on development data — at first with minimum error rate training, later with [batch tuning methods](https://md.research.elamur.ai/free/en/papers/cherry-2012-batch-tuning-strategies-statistical-machine/) able to handle thousands of features. Language models themselves grew to web scale, as in the Google work on [very large n-gram models with "stupid backoff"](https://md.research.elamur.ai/free/en/papers/brants-2007-large-language-models-machine-translation/). Decoding moved from maximum a posteriori search towards [minimum Bayes-risk decoding](https://md.research.elamur.ai/free/en/papers/kumar-2004-minimum-bayes-risk-decoding-statistical/), which picks the output that minimises the risk, averaged over competing hypotheses, measured by the evaluation metric of interest.

The main debate within the statistical school concerned syntax: whether to model hierarchical structure (Hiero, tree-to-string and string-to-tree systems) or to keep flat phrases and add syntactic information as features or preprocessing. Syntax helped most for distant pairs such as Chinese–English and Japanese–English.

### Neural machine translation

Neural MT models the whole conditional distribution of the target given the source with a single network trained end to end. The main families are:

- **Recurrent encoder–decoders with attention**, the first dominant design, studied in Cho et al.'s [analysis of encoder–decoder properties](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/) and refined with [coverage models](https://md.research.elamur.ai/free/en/papers/tu-2016-modeling-coverage-neural-machine-translation/) that discourage over- and under-translation.
- **Convolutional models**, such as the [convolutional encoder of Gehring et al.](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-encoder-model-neural-machine/).
- **Transformers**, built entirely on self-attention. Work in the library examines how to [scale their training](https://md.research.elamur.ai/free/en/papers/ott-2018-scaling-neural-machine-translation/), [train very deep encoders](https://md.research.elamur.ai/free/en/papers/wang-2019-learning-deep-transformer-models-machine/), and [which tricks transfer between architectures](https://md.research.elamur.ai/free/en/papers/chen-2018-best-both-worlds-combining-recent/).
- **Non-autoregressive models**, which generate all target tokens in parallel for speed, starting from the [Non-Autoregressive Transformer](https://md.research.elamur.ai/free/en/papers/gu-2017-non-autoregressive-neural-machine-translation/).

Training methods beyond maximum likelihood include [minimum risk training](https://md.research.elamur.ai/free/en/papers/shen-2015-minimum-risk-training-neural-machine/), reinforcement learning, and [sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/), which compresses a large teacher into a small student. Monolingual data enters through [language-model fusion](https://md.research.elamur.ai/free/en/papers/gulcehre-2015-using-monolingual-corpora-neural-machine/), back-translation and its [iterative](https://md.research.elamur.ai/free/en/papers/hoang-2018-iterative-back-translation-neural-machine/) and [tagged](https://md.research.elamur.ai/free/en/papers/caswell-2019-tagged-back-translation/) variants. Decoding research covers [beam search strategies](https://md.research.elamur.ai/free/en/papers/freitag-2017-beam-search-strategies-neural-machine/), [lexically constrained decoding](https://md.research.elamur.ai/free/en/papers/post-2018-fast-lexically-constrained-decoding-dynamic/) and [terminology constraints](https://md.research.elamur.ai/free/en/papers/dinu-2019-training-neural-machine-translation-apply/).

### Multilingual, unsupervised and LLM-based translation

Multilingual systems share parameters across many language pairs. Early designs [shared a single attention module](https://md.research.elamur.ai/free/en/papers/firat-2016-multi-way-multilingual-neural-machine/) across language-specific encoders and decoders; later ones used one model for all pairs, enabling [zero-shot translation](https://md.research.elamur.ai/free/en/papers/firat-2016-zero-resource-translation-multi-lingual/) between pairs never seen together and, at Google, [a single model for over a hundred languages](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/). Unsupervised MT learns from monolingual corpora only, with [denoising and back-translation in a shared latent space](https://md.research.elamur.ai/free/en/papers/lample-2017-unsupervised-machine-translation-using-monolingual/) or with [statistical methods](https://md.research.elamur.ai/free/en/papers/artetxe-2018-unsupervised-statistical-machine-translation/). The newest school uses general large language models as translators, studied in work on [prompting](https://md.research.elamur.ai/free/en/papers/zhang-2023-prompting-large-language-model-machine/), [in-context example selection](https://md.research.elamur.ai/free/en/papers/agrawal-2023-context-examples-selection-machine-translation/), [multilingual evaluation of LLMs](https://md.research.elamur.ai/free/en/papers/zhu-2024-multilingual-machine-translation-large-language/) and [document-level translation with LLMs](https://md.research.elamur.ai/free/en/papers/wang-2023-document-level-machine-translation-large/).

### The evaluation debate

A long-running argument concerns what automatic metrics actually measure. String-overlap metrics such as BLEU are cheap and reproducible, but they penalise valid paraphrases and lose their discriminating power as systems improve. Critics have documented [inconsistent BLEU reporting](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/), reviewed [its validity](https://md.research.elamur.ai/free/en/papers/reiter-2018-structured-review-validity-bleu/), and shown that metric rankings can be [distorted by outlier systems](https://md.research.elamur.ai/free/en/papers/mathur-2020-tangled-up-bleu-reevaluating-evaluation/). The WMT metrics tasks of the 2020s concluded that [neural metrics are more reliable](https://md.research.elamur.ai/free/en/papers/freitag-2022-results-wmt22-metrics-shared-task/), while human evaluation itself moved from crowd-sourced scores towards expert error annotation.

## Datasets, resources and benchmarks

Translation research depends on a small number of public data sources and recurring benchmarks:

| Resource | What it is |
| --- | --- |
| Canadian Hansards | English–French parliamentary proceedings, the training data of the IBM models; a large aligned release accompanied [Gale and Church's aligner](https://md.research.elamur.ai/free/en/papers/gale-1993-program-aligning-sentences-bilingual-corpora/) |
| Europarl | Proceedings of the European Parliament in many EU languages, the standard SMT training corpus of the 2000s |
| UN corpora | [MultiUN](https://md.research.elamur.ai/free/en/papers/eisele-2010-multiun-multilingual-corpus-united-nation/) and [the United Nations Parallel Corpus v1.0](https://md.research.elamur.ai/free/en/papers/ziemski-2016-united-nations-parallel-corpus-v1/), in the six official UN languages |
| TED talks | Subtitles collected in [WIT3](https://md.research.elamur.ai/free/en/papers/cettolo-2012-wit3-web-inventory-transcribed-translated/), basis of the IWSLT evaluations |
| MuST-C | [A speech translation corpus](https://md.research.elamur.ai/free/en/papers/gangi-2019-must-c-multilingual-speech-translation/) built from English talks and their translations |
| WMT news test sets | Freshly translated news used each year for the shared task |
| FLORES | [Evaluation sets for low-resource pairs](https://md.research.elamur.ai/free/en/papers/guzman-2019-flores-evaluation-datasets-low-resource/), first Nepali–English and Sinhala–English |
| WinoMT | [A challenge set for gender bias](https://md.research.elamur.ai/free/en/papers/stanovsky-2019-evaluating-gender-bias-machine-translation/) |

The WMT findings papers in the library — for [2013](https://md.research.elamur.ai/free/en/papers/bojar-2013-findings-2013-workshop-statistical-machine/), [2014](https://md.research.elamur.ai/free/en/papers/bojar-2014-findings-2014-workshop-statistical-machine/), [2015](https://md.research.elamur.ai/free/en/papers/bojar-2015-findings-2015-workshop-statistical-machine/), [2016](https://md.research.elamur.ai/free/en/papers/bojar-2016-findings-2016-conference-machine-translation/), [2017](https://md.research.elamur.ai/free/en/papers/bojar-2017-findings-2017-conference-machine-translation/), [2018](https://md.research.elamur.ai/free/en/papers/bojar-2018-findings-2018-conference-machine-translation/), [2019](https://md.research.elamur.ai/free/en/papers/barrault-2019-findings-2019-conference-machine-translation/), [2020](https://md.research.elamur.ai/free/en/papers/barrault-2020-findings-2020-conference-machine-translation/) and [2024](https://md.research.elamur.ai/free/en/papers/kocmi-2024-findings-wmt24-general-machine-translation/) — document how the benchmark evolved: new language pairs, test sets built only from original source text to avoid translationese, a shift from relative ranking to direct assessment, document context for judges, and test suites that probe specific phenomena.

Open-source toolkits are resources in their own right: [OpenNMT](https://md.research.elamur.ai/free/en/papers/klein-2017-opennmt-open-source-toolkit-neural/), [Marian](https://md.research.elamur.ai/free/en/papers/junczys-2018-dowmunt-marian-fast-neural-machine-translation/), [cdec](https://md.research.elamur.ai/free/en/papers/dyer-2010-cdec-decoder-alignment-learning-framework/) and, for quality estimation, [QuEst](https://md.research.elamur.ai/free/en/papers/specia-2013-quest-translation-quality-estimation-framework/).

## Landmark papers in this library

Citations of the papers on this topic, by when they were published Papers published in 2015–2019 gather 38% of the 224,213 citations of the papers on this topic.

**The numbers**

| Published | Papers and their citations |
| --- | --- |
| 2025–2029 | 139 papers, 291 citations |
| 2020–2024 | 138 papers, 17,508 citations |
| 2015–2019 | 213 papers, 86,242 citations |
| 2010–2014 | 63 papers, 43,890 citations |
| 2005–2009 | 72 papers, 26,496 citations |
| 2000–2004 | 34 papers, 45,364 citations |
| 1995–1999 | 16 papers, 1,472 citations |
| 1990–1994 | 26 papers, 2,273 citations |
| 1985–1989 | 13 papers, 649 citations |
| 1980–1984 | 3 papers, 28 citations |
| 1975–1979 | 1 papers, 0 citations |

### BLEU (2002)

[Papineni, Roukos, Ward and Zhu](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/) proposed scoring a translation by its modified n-gram precision against one or more references, with counts capped by the reference and a penalty for overly short output, all aggregated over a whole test corpus. Its ranking of systems agreed with human judges, and it was fast enough to run after every experiment. BLEU became the default metric for two decades and is by far the most cited paper here; nearly every later metric defines itself against it.

### Attention for neural translation (2014)

[Bahdanau, Cho and Bengio](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/) removed the bottleneck of encoding a sentence into one fixed vector. A bidirectional recurrent encoder produced one annotation per source word, and the decoder learned soft weights over these annotations at every step. Quality on long sentences improved dramatically, and the learned weights resembled word alignments. The attention mechanism introduced here became the basis of the Transformer.

### On the Properties of Neural Machine Translation (2014)

[Cho and colleagues](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/) analysed early encoder–decoder translators and found that their quality fell steeply with sentence length and with the number of unknown words. The diagnosis set the research agenda that attention, large vocabularies and subword units then addressed.

### METEOR (2005)

[Banerjee and Lavie](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/) built a metric on an explicit alignment between hypothesis and reference, matching exact words first and then stems and synonyms, combining precision and a heavier-weighted recall, and penalising fragmented matches. It correlated better with human judgments at the sentence level than BLEU. Successors include [Meteor 1.3](https://md.research.elamur.ai/free/en/papers/denkowski-2011-meteor-1-3-automatic-metric/) and [Meteor Universal](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/), which derives its resources for any language from the training bitext.

### A Call for Clarity in Reporting BLEU (2018)

[Matt Post](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/) showed that "BLEU" in published papers hides a family of configurations, chiefly in how references are tokenised, and that these choices alone shift scores by up to almost two points. He released SacreBLEU, which handles reference processing internally and prints a signature of its settings; it is now the standard way to report BLEU.

### chrF (2015) and chrF++ (2017)

[Maja Popović](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/) proposed an F-score over character n-grams that needs no tokeniser or language-specific resources and suits morphologically rich targets. [chrF++](https://md.research.elamur.ai/free/en/papers/popovic-2017-chrf-words-helping-character-n/) added word unigrams and bigrams. Both are widely reported alongside BLEU.

### COMET (2020)

[Rei, Stewart, Farinha and Lavie](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/) trained a regression model over cross-lingual pretrained encodings of the source, hypothesis and reference to predict human quality scores. COMET marked the shift from string matching to learned metrics; later versions such as [COMET-22](https://md.research.elamur.ai/free/en/papers/rei-2022-comet-22-unbabel-ist-2022/) and [xCOMET](https://md.research.elamur.ai/free/en/papers/guerreiro-2024-xcomet-transparent-machine-translation-evaluation/) added error-span detection.

### Improved Statistical Alignment Models (2000)

[Och and Ney](https://md.research.elamur.ai/free/en/papers/och-2000-improved-statistical-alignment-models/) introduced evaluation of word alignment against human annotation with "sure" and "possible" links and the alignment error rate, and used it to compare the IBM models and the HMM aligner systematically. Their training refinements became the GIZA++ toolkit, used to align parallel corpora for more than a decade.

### Discriminative Training and Maximum Entropy Models (2002)

[Och and Ney](https://md.research.elamur.ai/free/en/papers/och-2002-discriminative-training-maximum-entropy-models/) replaced the equal-weight product of the noisy channel with a log-linear combination of arbitrary features whose weights are learned. This made statistical MT extensible: any new knowledge source could be added as a feature, and the framework underlay phrase-based and syntax-based systems until the neural era.

### A Hierarchical Phrase-Based Model (2005)

[David Chiang](https://md.research.elamur.ai/free/en/papers/chiang-2005-hierarchical-phrase-based-model-statistical/) allowed phrase pairs to contain gaps filled by other phrases, forming a synchronous context-free grammar learned from unannotated bitext. The Hiero model captured long-distance reordering without a parser and outperformed a strong phrase-based system on Chinese–English. It became a standard alternative to flat phrase-based translation.

### Recurrent Continuous Translation Models (2013)

[Kalchbrenner and Blunsom](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-continuous-translation-models/) translated using only continuous vectors: a convolutional sentence encoder conditioned a recurrent target language model, with no phrase tables or explicit alignments. It was among the first demonstrations that a purely neural model could learn translation, a year before the encoder–decoder papers.

### Sequence-Level Knowledge Distillation (2016)

[Kim and Rush](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/) trained compact students on the beam-search outputs of large teachers. The students came close to teacher quality, decoded well without beam search, and could be pruned heavily. Sequence-level distillation became a standard tool, notably for training non-autoregressive models.

### Achieving Human Parity on Chinese to English News Translation (2018)

[Hassan and colleagues at Microsoft](https://md.research.elamur.ai/free/en/papers/hassan-2018-achieving-human-parity-automatic-chinese/) defined human parity as a statistically insignificant difference between human judgments of machine and professional translations, and reported reaching it on WMT 2017 Chinese–English news. The claim provoked careful re-examination: [Läubli, Sennrich and Volk](https://md.research.elamur.ai/free/en/papers/laubli-2018-has-machine-translation-achieved-human/) and [Toral and colleagues](https://md.research.elamur.ai/free/en/papers/toral-2018-attaining-unattainable-reassessing-claims-human/) showed that document context, professional raters and original-language source text change the verdict.

### Experts, Errors, and Context (2021)

[Freitag and colleagues](https://md.research.elamur.ai/free/en/papers/freitag-2021-experts-errors-context-large-scale/) had professional translators annotate errors in top WMT 2020 systems using the Multidimensional Quality Metrics (MQM) framework, with full document context. The expert rankings differed sharply from crowd ratings, clearly favoured human translation, and agreed better with embedding-based metrics than with crowd scores. MQM annotation has since become the reference for metric evaluation at WMT.

### Massively multilingual NMT (2019)

[Arivazhagan and colleagues](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/) trained one Transformer on more than a hundred languages paired with English, using web-scale data with highly unequal amounts per language. Low-resource languages gained through transfer, while high-resource ones suffered from interference unless capacity grew — a trade-off that still frames multilingual modelling.

### Unsupervised Machine Translation Using Monolingual Corpora Only (2017)

[Lample, Conneau, Denoyer and Ranzato](https://md.research.elamur.ai/free/en/papers/lample-2017-unsupervised-machine-translation-using-monolingual/) trained a translator without any parallel sentences, starting from an unsupervised bilingual lexicon and alternating denoising with back-translation in a shared latent space. Together with the [parallel work of Artetxe and colleagues](https://md.research.elamur.ai/free/en/papers/artetxe-2017-unsupervised-neural-machine-translation/), it opened a research line aimed at languages with no bitext.

## People

Authors with the most papers on this topic Philipp Koehn leads with 31 papers; the 12 authors shown wrote 235 of the papers on this topic between them, counting each paper once per author.

**The numbers**

| Author | Papers here · their citations |
| --- | --- |
| [Philipp Koehn](https://md.research.elamur.ai/free/en/authors/philipp-koehn/) | 31 papers · 14,205 citations |
| [Barry Haddow](https://md.research.elamur.ai/free/en/authors/barry-haddow/) | 24 papers · 11,891 citations |
| [Ondřej Bojar](https://md.research.elamur.ai/free/en/authors/ondrej-bojar/) | 24 papers · 7,668 citations |
| [André F. T. Martins](https://md.research.elamur.ai/free/en/authors/andre-f-t-martins/) | 23 papers · 3,924 citations |
| [Rico Sennrich](https://md.research.elamur.ai/free/en/authors/rico-sennrich/) | 20 papers · 8,272 citations |
| [Markus Freitag](https://md.research.elamur.ai/free/en/authors/markus-freitag/) | 20 papers · 3,336 citations |
| [Lucia Specia](https://md.research.elamur.ai/free/en/authors/lucia-specia/) | 19 papers · 4,465 citations |
| [Chris Callison-Burch](https://md.research.elamur.ai/free/en/authors/chris-callison-burch/) | 16 papers · 4,863 citations |
| [Christian Federmann](https://md.research.elamur.ai/free/en/authors/christian-federmann/) | 15 papers · 7,163 citations |
| [Christof Monz](https://md.research.elamur.ai/free/en/authors/christof-monz/) | 15 papers · 6,683 citations |
| [Kyunghyun Cho](https://md.research.elamur.ai/free/en/authors/kyunghyun-cho/) | 14 papers · 32,171 citations |
| [Alon Lavie](https://md.research.elamur.ai/free/en/authors/alon-lavie/) | 14 papers · 11,718 citations |

- [Philipp Koehn](https://md.research.elamur.ai/free/en/authors/philipp-koehn/), now at Johns Hopkins University, is the most prolific author on this topic. His papers here range from [factored translation models](https://md.research.elamur.ai/free/en/papers/koehn-2007-factored-translation-models/) and [domain adaptation](https://md.research.elamur.ai/free/en/papers/koehn-2007-experiments-domain-adaptation-statistical-machine/) to the [Edinburgh IWSLT system](https://md.research.elamur.ai/free/en/papers/koehn-2005-edinburgh-system-description-2005-iwslt/) and many years of WMT findings; he led the development of Moses.
- [Ondřej Bojar](https://md.research.elamur.ai/free/en/authors/ondrej-bojar/) of Charles University is first author of most WMT findings papers of the 2010s and a central organiser of the shared tasks.
- [Barry Haddow](https://md.research.elamur.ai/free/en/authors/barry-haddow/) of the University of Edinburgh co-authored the WMT findings and a [survey of low-resource MT](https://md.research.elamur.ai/free/en/papers/haddow-2022-survey-low-resource-machine-translation/).
- [Franz Josef Och](https://md.research.elamur.ai/free/en/authors/franz-josef-och/) and [Hermann Ney](https://md.research.elamur.ai/free/en/authors/hermann-ney/) of RWTH Aachen University laid the foundations of phrase-based statistical MT with the [alignment template approach](https://md.research.elamur.ai/free/en/papers/och-1999-improved-alignment-models-statistical-machine/), [alignment models](https://md.research.elamur.ai/free/en/papers/och-2000-improved-statistical-alignment-models/) and [log-linear training](https://md.research.elamur.ai/free/en/papers/och-2002-discriminative-training-maximum-entropy-models/); Och later led translation work at Google.
- [Kyunghyun Cho](https://md.research.elamur.ai/free/en/authors/kyunghyun-cho/), now at New York University, co-authored the founding papers of neural MT — [attention](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/), [encoder–decoder analysis](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/), [large vocabularies](https://md.research.elamur.ai/free/en/papers/jean-2015-very-large-target-vocabulary-neural/) and [multilingual translation](https://md.research.elamur.ai/free/en/papers/firat-2016-multi-way-multilingual-neural-machine/) — with [Yoshua Bengio](https://md.research.elamur.ai/free/en/authors/yoshua-bengio/) and [Dzmitry Bahdanau](https://md.research.elamur.ai/free/en/authors/dzmitry-bahdanau/).
- [Rico Sennrich](https://md.research.elamur.ai/free/en/authors/rico-sennrich/) of the University of Zurich contributed subword and back-translation methods through the [Edinburgh WMT systems](https://md.research.elamur.ai/free/en/papers/sennrich-2016-edinburgh-neural-machine-translation-systems/), [politeness control](https://md.research.elamur.ai/free/en/papers/sennrich-2016-controlling-politeness-neural-machine-translation/) and work on [context-aware translation](https://md.research.elamur.ai/free/en/papers/voita-2018-context-aware-neural-machine-translation/).
- [Alon Lavie](https://md.research.elamur.ai/free/en/authors/alon-lavie/) of Carnegie Mellon University co-created [METEOR](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/) and later [COMET](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/), bridging two generations of metrics.
- [Ricardo Rei](https://md.research.elamur.ai/free/en/authors/ricardo-rei/) and [André F. T. Martins](https://md.research.elamur.ai/free/en/authors/andre-f-t-martins/), working in Lisbon, developed the COMET family of learned metrics; Martins also co-authored the [Marian toolkit](https://md.research.elamur.ai/free/en/papers/junczys-2018-dowmunt-marian-fast-neural-machine-translation/).
- [Markus Freitag](https://md.research.elamur.ai/free/en/authors/markus-freitag/), now at ETH Zurich, led the [expert MQM study](https://md.research.elamur.ai/free/en/papers/freitag-2021-experts-errors-context-large-scale/), the WMT metrics tasks of [2022](https://md.research.elamur.ai/free/en/papers/freitag-2022-results-wmt22-metrics-shared-task/) and [2023](https://md.research.elamur.ai/free/en/papers/freitag-2023-results-wmt23-metrics-shared-task/), and [work on references](https://md.research.elamur.ai/free/en/papers/freitag-2020-bleu-might-be-guilty-but/) as a source of metric error.
- [Lucia Specia](https://md.research.elamur.ai/free/en/authors/lucia-specia/) of Imperial College London pioneered [quality estimation](https://md.research.elamur.ai/free/en/papers/specia-2009-estimating-sentence-level-quality-machine/) and [multimodal translation](https://md.research.elamur.ai/free/en/papers/specia-2016-shared-task-multimodal-machine-translation/).
- [Kevin Knight](https://md.research.elamur.ai/free/en/authors/kevin-knight/) introduced [syntax-based statistical translation](https://md.research.elamur.ai/free/en/papers/yamada-2001-syntax-based-statistical-translation-model/) and scaled up [syntactic rule extraction](https://md.research.elamur.ai/free/en/papers/galley-2006-scalable-inference-training-context-rich/) with [Daniel Marcu](https://md.research.elamur.ai/free/en/authors/daniel-marcu/).
- [Chris Callison-Burch](https://md.research.elamur.ai/free/en/authors/chris-callison-burch/) showed how to [mine paraphrases from bilingual corpora](https://md.research.elamur.ai/free/en/papers/bannard-2005-paraphrasing-bilingual-parallel-corpora/) and applied translation methods to [simplification](https://md.research.elamur.ai/free/en/papers/xu-2016-optimizing-statistical-machine-translation-text/).
- [Orhan Firat](https://md.research.elamur.ai/free/en/authors/orhan-firat/) is a leading author on multilingual NMT, from [shared attention](https://md.research.elamur.ai/free/en/papers/firat-2016-multi-way-multilingual-neural-machine/) to [massively multilingual models at Google](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/).
- [Matt Post](https://md.research.elamur.ai/free/en/authors/matt-post/), now at Microsoft, wrote the [SacreBLEU paper](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/) and work on [constrained decoding](https://md.research.elamur.ai/free/en/papers/post-2018-fast-lexically-constrained-decoding-dynamic/).
- [Graham Neubig](https://md.research.elamur.ai/free/en/authors/graham-neubig/) of Carnegie Mellon University contributed to [rapid adaptation to new languages](https://md.research.elamur.ai/free/en/papers/neubig-2018-rapid-adaptation-neural-machine-translation/) and [discrete lexicons in NMT](https://md.research.elamur.ai/free/en/papers/arthur-2016-incorporating-discrete-translation-lexicons-into/).

Other frequent contributors include [Christian Federmann](https://md.research.elamur.ai/free/en/authors/christian-federmann/), [Christof Monz](https://md.research.elamur.ai/free/en/authors/christof-monz/) of the University of Amsterdam, [Yvette Graham](https://md.research.elamur.ai/free/en/authors/yvette-graham/) of Trinity College Dublin, [Alexandra Birch](https://md.research.elamur.ai/free/en/authors/alexandra-birch/), [Tom Kocmi](https://md.research.elamur.ai/free/en/authors/tom-kocmi/), [George Foster](https://md.research.elamur.ai/free/en/authors/george-foster/), [Marcello Federico](https://md.research.elamur.ai/free/en/authors/marcello-federico/), [Marcin Junczys-Dowmunt](https://md.research.elamur.ai/free/en/authors/marcin-junczys-dowmunt/) and [Rachel Bawden](https://md.research.elamur.ai/free/en/authors/rachel-bawden/), whose work on [discourse phenomena](https://md.research.elamur.ai/free/en/papers/bawden-2017-evaluating-discourse-phenomena-neural-machine/) helped establish document-level evaluation.

## Institutions and places

The authors on this topic are concentrated in a handful of research centres that have shaped MT for decades. Among current affiliations, Microsoft leads, followed by the University of Edinburgh — home of the Moses project and of successive WMT-winning systems — Carnegie Mellon University, Johns Hopkins University and Charles University in Prague, which hosts much of the WMT organisation. Other important centres are the University of Amsterdam, Google, ETH Zurich and the University of Zurich, Imperial College London, and a Lisbon cluster (Instituto de Telecomunicações, Instituto Superior Técnico, University of Lisbon) associated with learned evaluation metrics. Historically, IBM Research, RWTH Aachen, the Information Sciences Institute at the University of Southern California and Fondazione Bruno Kessler in Trento were also major sources of work.

Where the authors on this topic work now University of Edinburgh is the current home of authors of 53 papers on this topic; counted are the authors with a page here, by their latest institution.

**The numbers**

| Institution | Papers |
| --- | --- |
| [University of Edinburgh](https://md.research.elamur.ai/free/en/universities/university-of-edinburgh/) | 53 papers |
| [Microsoft (United States)](https://md.research.elamur.ai/free/en/companies/microsoft/) | 51 papers |
| [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) | 49 papers |
| [Johns Hopkins University](https://md.research.elamur.ai/free/en/universities/johns-hopkins-university/) | 48 papers |
| [University of Cape Town](https://md.research.elamur.ai/free/en/universities/university-of-cape-town/) | 47 papers |
| [Charles University](https://md.research.elamur.ai/free/en/universities/charles-university/) | 36 papers |
| [Google (United States)](https://md.research.elamur.ai/free/en/companies/google/) | 34 papers |
| [University of Amsterdam](https://md.research.elamur.ai/free/en/universities/university-of-amsterdam/) | 32 papers |
| [ETH Zurich](https://md.research.elamur.ai/free/en/universities/eth-zurich/) | 29 papers |
| [Imperial College London](https://md.research.elamur.ai/free/en/universities/imperial-college-london/) | 27 papers |

By country, the United States accounts for the largest share of authors, followed by the United Kingdom, China, Germany and Switzerland; Czechia, Canada and the Netherlands are also prominent, reflecting the European strength of the field built around Europarl, EU-funded projects and the WMT campaign.

Countries of the authors on this topic United States comes first: authors working there now wrote 319 papers on this topic.

**The numbers**

| Country | Papers |
| --- | --- |
| United States | 319 papers |
| United Kingdom | 131 papers |
| China | 99 papers |
| Germany | 70 papers |
| Switzerland | 67 papers |
| Czechia | 62 papers |
| Canada | 60 papers |
| Netherlands | 55 papers |
| South Africa | 47 papers |
| France | 40 papers |

The papers appear mostly in the ACL family of venues: the annual Conference (formerly Workshop) on Machine Translation, the main ACL and EMNLP conferences, Transactions of the ACL and the journal Computational Linguistics. Language-resource papers often appear at LREC, and the European Association for Machine Translation conference and dedicated workshops on low-resource MT and spoken language translation are well represented.

Where the papers on this topic were published Proceedings of the Fifteenth Language Resources and Evaluation Conference carries the most, 22 of 718.

**The numbers**

| Venue | Papers |
| --- | --- |
| Proceedings of the Fifteenth Language Resources and Evaluation Conference | 22 papers |
| Computational Linguistics | 21 papers |
| Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 1) | 19 papers |
| Transactions of the Association for Computational Linguistics | 15 papers |
| Proceedings for the Ninth Workshop on Technologies for Machine Translation of Low Resource Languages (LoResMT 2026) | 12 papers |
| Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing | 12 papers |
| Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics | 12 papers |
| Proceedings of the Second Conference on Machine Translation | 12 papers |
| Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 2) | 10 papers |
| Proceedings of the Eighth Conference on Machine Translation | 10 papers |

## Open questions

- **Evaluation at high quality.** When systems approach or exceed reference quality, reference-based metrics lose meaning. How to evaluate reliably — with expert error annotation, source-based judgments or learned metrics — and how to avoid metrics that can be gamed remain open; recent WMT metrics tasks ask whether [LLM output breaks existing metrics](https://md.research.elamur.ai/free/en/papers/freitag-2024-are-llms-breaking-mt-metrics/).
- **Human parity and its meaning.** Parity claims depend heavily on the evaluation protocol, rater expertise, document context and the direction in which the test set was originally written.
- **Low-resource and endangered languages.** Most of the world's languages still lack adequate parallel data. Community-driven work such as [participatory research for African languages](https://md.research.elamur.ai/free/en/papers/nekoto-2020-participatory-research-low-resourced-machine/), transfer between [related languages](https://md.research.elamur.ai/free/en/papers/nguyen-2017-transfer-learning-across-low-resource/) and evaluation sets such as FLORES are early steps.
- **Document and discourse.** Sentence-by-sentence translation mishandles pronouns, ellipsis and lexical cohesion; [context-aware models](https://md.research.elamur.ai/free/en/papers/voita-2019-when-good-translation-is-wrong/) and [hierarchical attention over documents](https://md.research.elamur.ai/free/en/papers/miculicich-2018-document-level-neural-machine-translation/) help, but evaluation of document-level quality is still immature.
- **Hallucination and robustness.** Neural systems can produce fluent text unrelated to the source; see [the curious case of hallucinations](https://md.research.elamur.ai/free/en/papers/raunak-2021-curious-case-hallucinations-neural-machine/) and [hallucinations in large multilingual models](https://md.research.elamur.ai/free/en/papers/guerreiro-2023-hallucinations-large-multilingual-translation-models/).
- **Bias.** Systems often default to stereotyped gender when the source is ambiguous, as shown by [WinoMT](https://md.research.elamur.ai/free/en/papers/stanovsky-2019-evaluating-gender-bias-machine-translation/) and surveyed in [Gender Bias in Machine Translation](https://md.research.elamur.ai/free/en/papers/savoldi-2021-gender-bias-machine-translation/).
- **Multilingual capacity.** How to add languages without degrading existing ones, and how to allocate model capacity across hundreds of languages.
- **The role of general LLMs.** Whether dedicated translation models will remain competitive with general large language models, and how to control terminology, style and faithfulness in prompted translation.

## Neighbouring topics

Machine translation sits at a crossroads. Its largest overlap is with [Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/): metrics and human evaluation protocols developed for MT became templates for summarisation, generation and dialogue. [Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) and [Low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/) share its concern with data scarcity and transfer across languages. Work on Arabic and Hebrew translation, including [Arabic dialects](https://md.research.elamur.ai/free/en/papers/zbib-2012-machine-translation-arabic-dialects/) and [Arabic preprocessing](https://md.research.elamur.ai/free/en/papers/habash-2006-arabic-preprocessing-schemes-statistical-machine/), links the topic to [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) and [Hebrew & Aramaic](https://md.research.elamur.ai/free/en/topics/hebrew-aramaic/).

Speech translation connects it to [Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/), and multimodal translation to [Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/). Linguistic structure enters through [Morphology](https://md.research.elamur.ai/free/en/topics/morphology/), [Syntax & grammar formalisms](https://md.research.elamur.ai/free/en/topics/syntax-grammar/), [Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/), [Word sense disambiguation](https://md.research.elamur.ai/free/en/topics/word-sense-disambiguation/) — as in [WSD for statistical MT](https://md.research.elamur.ai/free/en/papers/carpuat-2007-improving-statistical-machine-translation-word/) — and [Idioms & multiword expressions](https://md.research.elamur.ai/free/en/topics/multiword-expressions/). Document-level translation draws on [Coreference resolution](https://md.research.elamur.ai/free/en/topics/coreference-resolution/) and [Discourse analysis](https://md.research.elamur.ai/free/en/topics/discourse-analysis/). Synchronous grammars link it to [Semantic parsing & meaning representation](https://md.research.elamur.ai/free/en/topics/semantic-parsing/). Neural MT gave rise to much of modern [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) and [Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/), and questions of fairness tie it to [Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/). Related monolingual tasks such as paraphrasing and simplification, and translation-inspired approaches to [Grammatical error correction](https://md.research.elamur.ai/free/en/topics/grammatical-error-correction/), also borrow its methods.

Neighbouring topics: papers shared Evaluation & benchmarks shares the most: 193 of the 718 papers here (27%) stand on both.

**The numbers**

| Topic | Papers shared |
| --- | --- |
| [Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) | 193 shared · 27% |
| [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) | 49 shared · 7% |
| [Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) | 46 shared · 6% |
| [Low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/) | 44 shared · 6% |
| [Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) | 17 shared · 2% |
| [Coreference resolution](https://md.research.elamur.ai/free/en/topics/coreference-resolution/) | 12 shared · 2% |
| [Morphology](https://md.research.elamur.ai/free/en/topics/morphology/) | 10 shared · 1% |
| [Semantic parsing & meaning representation](https://md.research.elamur.ai/free/en/topics/semantic-parsing/) | 10 shared · 1% |
| [Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) | 10 shared · 1% |
| [Discourse analysis](https://md.research.elamur.ai/free/en/topics/discourse-analysis/) | 9 shared · 1% |

Within the library, most papers on this topic also stand on the [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) shelf, with many on [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) and [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/).

The shelves this topic stands on Machine translation: 531 of the 718 papers here.

**The numbers**

| Shelf | Papers |
| --- | --- |
| [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) | 531 papers |
| [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) | 377 papers |
| [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) | 303 papers |
| [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/) | 15 papers |
| [Other](https://md.research.elamur.ai/free/en/shelves/other/) | 11 papers |
| [Computing & GPUs](https://md.research.elamur.ai/free/en/shelves/computing/) | 5 papers |
| [Law](https://md.research.elamur.ai/free/en/shelves/law/) | 3 papers |
| [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/) | 1 paper |
| [Society & ethics](https://md.research.elamur.ai/free/en/shelves/society/) | 1 paper |

## Key works in this library

- [BLEU](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/) — the n-gram precision metric that defined MT evaluation for two decades.
- [Jointly Learning to Align and Translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/) — introduced attention for translation.
- [On the Properties of Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/) — diagnosed the length and vocabulary limits of early encoder–decoders.
- [A Statistical Approach to Language Translation](https://md.research.elamur.ai/free/en/papers/brown-1988-statistical-approach-language-translation/) — the IBM proposal that launched statistical MT.
- [Aligning Sentences in Bilingual Corpora](https://md.research.elamur.ai/free/en/papers/gale-1993-program-aligning-sentences-bilingual-corpora/) — length-based sentence alignment.
- [Improved Statistical Alignment Models](https://md.research.elamur.ai/free/en/papers/och-2000-improved-statistical-alignment-models/) — alignment error rate and a systematic comparison of alignment models.
- [Discriminative Training and Maximum Entropy Models](https://md.research.elamur.ai/free/en/papers/och-2002-discriminative-training-maximum-entropy-models/) — the log-linear framework.
- [A Hierarchical Phrase-Based Model for SMT](https://md.research.elamur.ai/free/en/papers/chiang-2005-hierarchical-phrase-based-model-statistical/) — synchronous grammars learned without a parser.
- [A Syntax-based Statistical Translation Model](https://md.research.elamur.ai/free/en/papers/yamada-2001-syntax-based-statistical-translation-model/) — tree-to-string translation.
- [Factored Translation Models](https://md.research.elamur.ai/free/en/papers/koehn-2007-factored-translation-models/) — linguistic annotation inside phrase-based MT.
- [Large Language Models in Machine Translation](https://md.research.elamur.ai/free/en/papers/brants-2007-large-language-models-machine-translation/) — trillion-token n-gram models and stupid backoff.
- [METEOR](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/) — alignment-based metric with stems and synonyms.
- [chrF](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/) — character n-gram F-score.
- [A Call for Clarity in Reporting BLEU](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/) — SacreBLEU and reproducible scoring.
- [COMET](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/) — a learned neural metric.
- [Experts, Errors, and Context](https://md.research.elamur.ai/free/en/papers/freitag-2021-experts-errors-context-large-scale/) — MQM expert evaluation of top systems.
- [OpenNMT](https://md.research.elamur.ai/free/en/papers/klein-2017-opennmt-open-source-toolkit-neural/) — open-source neural MT toolkit.
- [Sequence-Level Knowledge Distillation](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/) — compressing translation models.
- [Modeling Coverage for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/tu-2016-modeling-coverage-neural-machine-translation/) — tracking which source words have been translated.
- [Scaling Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/ott-2018-scaling-neural-machine-translation/) — fast large-batch Transformer training.
- [Non-Autoregressive Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/gu-2017-non-autoregressive-neural-machine-translation/) — parallel decoding.
- [Achieving Human Parity (Chinese–English news)](https://md.research.elamur.ai/free/en/papers/hassan-2018-achieving-human-parity-automatic-chinese/) — the human-parity claim and its protocol.
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://md.research.elamur.ai/free/en/papers/lample-2017-unsupervised-machine-translation-using-monolingual/) — translation without parallel data.
- [Massively Multilingual NMT in the Wild](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/) — one model for over a hundred languages.
- [Evaluating Gender Bias in Machine Translation](https://md.research.elamur.ai/free/en/papers/stanovsky-2019-evaluating-gender-bias-machine-translation/) — the WinoMT challenge set.

## Papers

**1** [2](https://md.research.elamur.ai/free/en/topics/machine-translation/page/2/) … [29](https://md.research.elamur.ai/free/en/topics/machine-translation/page/29/) [next →](https://md.research.elamur.ai/free/en/topics/machine-translation/page/2/)

- [Bleu: a Method for Automatic Evaluation of Machine Translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/) — Kishore Papineni, Salim Roukos, Todd Ward, et al. · 2002 · 32 065 citations
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/) — Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio · 2014 · 18 908 citations
- [Attention Is All You Need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/) — Ashish Vaswani, Noam Shazeer, Niki Parmar, et al. · 2017 · 16 413 citations
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/) — Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, et al. · 2014 · 7 734 citations
- [METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/) — Satanjeev Banerjee, Alon Lavie · 2005 · 5 483 citations
- [A Call for Clarity in Reporting BLEU Scores](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/) — Matt Post · 2018 · 4 130 citations
- [Neural Machine Translation of Rare Words with Subword Units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/) — Rico Sennrich, Barry Haddow, Alexandra Birch · 2016 · 3 409 citations
- [OpenNMT: Open-Source Toolkit for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/klein-2017-opennmt-open-source-toolkit-neural/) — Guillaume Klein, Yoon Kim, Yuntian Deng, et al. · 2017 · 2 460 citations
- [Statistical Significance Tests for Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/) — Philipp Koehn · 2004 · 2 444 citations
- [Meteor Universal: Language Specific Translation Evaluation for Any Target Language](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/) — Michael Denkowski, Alon Lavie · 2014 · 2 392 citations
- [chrF: character n-gram F-score for automatic MT evaluation](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/) — Maja Popović · 2015 · 2 105 citations
- [COMET: A Neural Framework for MT Evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/) — Ricardo Rei, Craig Stewart, Ana C Farinha, et al. · 2020 · 1 784 citations
- [A Hierarchical Phrase-Based Model for Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/chiang-2005-hierarchical-phrase-based-model-statistical/) — David Chiang · 2005 · 1 683 citations
- [Recurrent Continuous Translation Models](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-continuous-translation-models/) — Nal Kalchbrenner, Phil Blunsom · 2013 · 1 597 citations
- [Effective Approaches to Attention-based Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/) — Thang Luong, Hieu Pham, Christopher D. Manning · 2015 · 1 474 citations
- [Improved Statistical Alignment Models](https://md.research.elamur.ai/free/en/papers/och-2000-improved-statistical-alignment-models/) — Franz Josef Och, Hermann Ney · 2000 · 1 467 citations
- [Discriminative Training and Maximum Entropy Models for Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/och-2002-discriminative-training-maximum-entropy-models/) — Franz Josef Och, Hermann Ney · 2002 · 1 461 citations
- [Sequence-Level Knowledge Distillation](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/) — Yoon Kim, Alexander M. Rush · 2016 · 1 440 citations
- [A Program for Aligning Sentences in Bilingual Corpora](https://md.research.elamur.ai/free/en/papers/gale-1993-program-aligning-sentences-bilingual-corpora/) — William A. Gale, Kenneth W. Church · 1993 · 1 284 citations
- [A Syntax-based Statistical Translation Model](https://md.research.elamur.ai/free/en/papers/yamada-2001-syntax-based-statistical-translation-model/) — Kenji Yamada, Kevin Knight · 2001 · 1 018 citations
- [On Using Very Large Target Vocabulary for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/jean-2015-very-large-target-vocabulary-neural/) — Sébastien Jean, Kyunghyun Cho, Roland Memisevic, et al. · 2015 · 999 citations
- [Optimizing Statistical Machine Translation for Text Simplification](https://md.research.elamur.ai/free/en/papers/xu-2016-optimizing-statistical-machine-translation-text/) — Wei Xu, Courtney Napoles, Ellie Pavlick, et al. · 2016 · 952 citations
- [Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics](https://md.research.elamur.ai/free/en/papers/lin-2004-automatic-evaluation-machine-translation-quality/) — Chin-Yew Lin, Franz Josef Och · 2004 · 937 citations
- [Findings of the 2016 Conference on Machine Translation](https://md.research.elamur.ai/free/en/papers/bojar-2016-findings-2016-conference-machine-translation/) — Ondřej Bojar, Rajen Chatterjee, Christian Federmann, et al. · 2016 · 936 citations
- [Modeling Coverage for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/tu-2016-modeling-coverage-neural-machine-translation/) — Zhaopeng Tu, Zhengdong Lu, Yang Liu, et al. · 2016 · 927 citations

**1** [2](https://md.research.elamur.ai/free/en/topics/machine-translation/page/2/) … [29](https://md.research.elamur.ai/free/en/topics/machine-translation/page/29/) [next →](https://md.research.elamur.ai/free/en/topics/machine-translation/page/2/)

- [← previous topic Low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/)
- [next topic → Morphology](https://md.research.elamur.ai/free/en/topics/morphology/)
- [↑ back to topics](https://md.research.elamur.ai/free/en/topics/)