---
title: "BLEU score — definition and key papers"
description: "BLEU score (Bilingual Evaluation Understudy) is an automatic metric for machine translation."
canonical: "https://research.elamur.ai/free/en/concepts/bleu/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/concepts/bleu/)

[home](https://md.research.elamur.ai/free/en/) / [concepts](https://md.research.elamur.ai/free/en/concepts/) / BLEU score

# 🎯 BLEU score

Concept · 26 key papers · [all concepts](https://md.research.elamur.ai/free/en/concepts/)

BLEU score (Bilingual Evaluation Understudy) is an automatic metric for machine translation. It rates a candidate text by how many of its word n-grams also occur in one or more human reference translations, and it penalises candidates that are too short.

Also: BLEU, Bleu, BLEU-4, IBM BLEU, Bilingual Evaluation Understudy

## What it is

BLEU estimates translation quality without a human judge. It compares system output with reference translations and counts shared word sequences. The score runs from 0 to 1 and is often reported multiplied by 100. A higher score means more overlap with the references. BLEU is meant to be computed over a whole test corpus, not a single sentence. It was proposed as a cheap, repeatable stand-in for human evaluation of machine translation ([Papineni et al. 2002](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)).

The metric measures surface agreement only. It does not check whether meaning is preserved, whether grammar is correct, or whether a different but valid wording was used. Its value rests on the assumption that, averaged over many sentences, overlap with references tracks human preferences. Much later work has tested and questioned that assumption; the main findings are summarised under Open questions ([Reiter 2018](https://md.research.elamur.ai/free/en/papers/reiter-2018-structured-review-validity-bleu/)).

## The key idea

BLEU combines three components: clipped ("modified") n-gram precision, a geometric mean across n-gram orders, and a brevity penalty.

```
BLEU = BP · exp( Σₙ wₙ · log pₙ ),   n = 1 … N

pₙ = Σ min(count_cand(g), max_ref count(g)) / Σ count_cand(g)

BP = 1             if c > r
BP = exp(1 − r/c)  if c ≤ r
```

In these formulas, g is an n-gram of length n in the candidate. count_cand(g) is how often g appears in the candidate. max_ref count(g) is the largest number of times g appears in any single reference. Taking the minimum of the two is called clipping. It stops a system from gaining credit by repeating a common word: "the the the the" cannot earn more matches for "the" than a reference actually contains.

The sums run over the corpus, so pₙ is the modified precision for order n. N is the highest order, usually 4, which gives the common variant BLEU-4. The weights wₙ are usually uniform (1/N), which turns the exponentiated weighted sum of logs into a geometric mean.

The brevity penalty BP depends on two lengths. c is the total candidate length. r is the effective reference length, usually the reference length closest to each candidate, summed over the corpus. Precision alone would reward very short outputs, so BP shrinks the score when the candidate is shorter than the references. For instance, a five-word candidate against a six-word reference gets BP = exp(1 − 6/5) ≈ 0.82.

A worked example uses the reference "the cat is on the mat" and the candidate "the cat sat on the mat".

| Order | Matched / candidate n-grams | pₙ |
| --- | --- | --- |
| 1 | the, the, cat, on, mat → 5/6 | 0.833 |
| 2 | the cat, on the, the mat → 3/5 | 0.600 |
| 3 | on the mat → 1/4 | 0.250 |
| 4 | none → 0/3 | 0 |

Both sentences have six words, so BP = 1. With N = 3, the product 0.833 × 0.6 × 0.25 is 0.125; its cube root is 0.5, so BLEU-3 = 0.5. With N = 4, a single zero precision makes the whole product zero.

This is why sentence-level BLEU needs smoothing and why the metric is defined at corpus level. Smoothing means adding small pseudo-counts so that one empty n-gram order does not force the score to zero.

## History and founding papers

BLEU was introduced by Papineni, Roukos, Ward and Zhu at IBM in 2002 ([Papineni et al. 2002](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)). It became the dominant automatic measure in machine translation research. More than a decade later, automatic evaluation for most target languages still depended on it ([Denkowski & Lavie 2014](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/)). Its n-gram counting idea soon spread to other tasks.

Lin and Hovy turned BLEU's clipped precision into a recall measure for summary evaluation. They found that unmodified BLEU correlated with human rankings on single-document summaries from DUC 2001 (the Document Understanding Conference evaluation). The Spearman ρ was 0.66 using one reference and 0.82 using three ([Lin & Hovy 2003](https://md.research.elamur.ai/free/en/papers/lin-2003-automatic-evaluation-summaries-n-gram/)). That line of work led to the ROUGE package ([Lin 2004](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/)).

Critiques of BLEU appeared early and set the agenda for successor metrics:

- Lin and Och listed several weaknesses: a fixed maximum n, a brevity penalty in place of true recall, word order captured only within n-grams of length up to N, and frequent zero scores for single sentences. They proposed subsequence-based alternatives ([Lin & Och 2004](https://md.research.elamur.ai/free/en/papers/lin-2004-automatic-evaluation-machine-translation-quality/)).
- Banerjee and Lavie designed METEOR around similar complaints, notably missing recall and the absence of explicit word-to-word matching. Their answer combined direct recall, staged word alignment and a fragmentation penalty ([Banerjee & Lavie 2005](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)).
- Koehn showed how bootstrap resampling, including a paired version for comparing two systems, can test whether a BLEU difference is statistically significant ([Koehn 2004](https://md.research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/)). Riezler and Maxwell catalogued further pitfalls in testing practice ([Riezler & Maxwell 2005](https://md.research.elamur.ai/free/en/papers/riezler-2005-some-pitfalls-automatic-evaluation-significance/)).
- Callison-Burch, Osborne and Koehn argued that a BLEU improvement is neither necessary nor sufficient for better human-judged translation. They also showed BLEU underrating a rule-based system ([Callison-Burch et al. 2006](https://md.research.elamur.ai/free/en/papers/burch-2006-re-evaluating-role-bleu-machine/)).

## How it is used today

BLEU remains a standard reported number in translation papers and shared tasks. It is also used as an optimisation target. Phrase-based systems commonly tuned feature weights with minimum error rate training (MERT), which searches for the weights that maximise an automatic score such as BLEU on development data. In one such system, the tuning setup made the output too short. This badly hurt scores under NIST, a separate n-gram metric that weights matches by how informative they are ([Koehn et al. 2005](https://md.research.elamur.ai/free/en/papers/koehn-2005-edinburgh-system-description-2005-iwslt/)).

Shared tasks paired BLEU with human ratings. In the 2006 European-language task, BLEU was computed against a single reference. Averaged per language pair, it ran from 28.33 for English–French, the easiest pair, to 14.01 for English–German, the hardest. The same task found that BLEU undervalued a rule-based commercial system on in-domain text ([Koehn & Monz 2006](https://md.research.elamur.ai/free/en/papers/koehn-2006-manual-automatic-evaluation-machine-translation/)).

Because tokenisation and casing change the score, organisers often normalise outputs themselves ([Koehn & Monz 2006](https://md.research.elamur.ai/free/en/papers/koehn-2006-manual-automatic-evaluation-machine-translation/)). Post called for clearer reporting of exactly how BLEU numbers are produced ([Post 2018](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/)).

Several later metrics now appear next to BLEU, and often in place of it:

- METEOR aligns words in stages: exact forms, then Porter stems, then WordNet synonyms. It scores the alignment with recall-weighted unigram matching plus a fragmentation penalty. It was tested on LDC TIDES 2003 data, which are Arabic and Chinese news translated into English and distributed by the Linguistic Data Consortium. Its segment-level Pearson correlations with human judgments were 0.347 (Arabic) and 0.331 (Chinese). Exact matching alone gave 0.312 and 0.293, and the full metric clearly beat unigram precision, recall or F1 used on their own ([Banerjee & Lavie 2005](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)). Later versions re-tuned its parameters per language ([Lavie & Agarwal 2007](https://md.research.elamur.ai/free/en/papers/lavie-2007-meteor-automatic-metric-mt-evaluation/)). Version 1.3 added paraphrase matching and function-word weighting ([Denkowski & Lavie 2011](https://md.research.elamur.ai/free/en/papers/denkowski-2011-meteor-1-3-automatic-metric/)). Meteor Universal extended it to new languages. It was tested on Russian data from WMT13, the 2013 Workshop on Statistical Machine Translation. There it reached a sentence-level Kendall τ (a rank correlation with human pairwise rankings) of 0.128, against 0.068 for BLEU ([Denkowski & Lavie 2014](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/)).
- chrF computes an F-score over character n-grams, which suits morphologically rich languages ([Popović 2015](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/)). chrF++ adds word unigrams and bigrams to character n-grams up to length 6. It uses the recall weight β = 2 rather than the β = 3 favoured for the original chrF ([Popović 2015](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/)), because heavier recall weighting can make tuned output too long ([Popović 2017](https://md.research.elamur.ai/free/en/papers/popovic-2017-chrf-words-helping-character-n/)).
- COMET is a neural framework that learns to predict human quality judgments ([Rei et al. 2020](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/)). Its 2022 ensemble outperformed BLEU and chrF in correlation with MQM annotations on WMT 2021 news data ([Rei et al. 2022](https://md.research.elamur.ai/free/en/papers/rei-2022-comet-22-unbabel-ist-2022/)). MQM (Multidimensional Quality Metrics) is human annotation of error spans with severity levels. xCOMET adds error-span tagging, so the metric shows where a translation fails ([Guerreiro et al. 2024](https://md.research.elamur.ai/free/en/papers/guerreiro-2024-xcomet-transparent-machine-translation-evaluation/)).

Outside translation, BLEU-4 is still a common baseline in image captioning. Reference-free scoring with CLIPScore complements it rather than replacing it: an information-gain analysis shows CLIPScore adding information beyond reference-based metrics such as BLEU-4. On New York Times news captions, reference-based metrics do far better: BLEU-4 reached 91% accuracy where CLIPScore reached 65% ([Hessel et al. 2021](https://md.research.elamur.ai/free/en/papers/hessel-2021-clipscore-reference-free-evaluation-metric/)).

Its use elsewhere has been strongly contested. For dialogue response generation, BLEU showed weak or no correlation with human judgments ([Liu et al. 2016](https://md.research.elamur.ai/free/en/papers/liu-2016-how-not-evaluate-your-dialogue/)). In natural language generation, word-overlap metrics including BLEU correlated only weakly with human ratings ([Novikova et al. 2017](https://md.research.elamur.ai/free/en/papers/novikova-2017-why-we-need-new-evaluation/)).

## Open questions

The main open issue is validity: how far BLEU reflects what people value. A structured review concluded that the evidence supports BLEU for diagnostic, system-level evaluation of MT systems. It does not support BLEU for judging individual texts, for other NLG tasks, or for testing scientific hypotheses ([Reiter 2018](https://md.research.elamur.ai/free/en/papers/reiter-2018-structured-review-validity-bleu/)). Mathur, Baldwin and Cohn showed that meta-evaluation can itself mislead. A few outlier systems can inflate Pearson correlations, and small BLEU differences between systems often disagree with human judgments ([Mathur et al. 2020](https://md.research.elamur.ai/free/en/papers/mathur-2020-tangled-up-bleu-reevaluating-evaluation/)).

Specific blind spots are well documented:

- Valid wording variation goes unrewarded. Only 21 of 10,728 pairs of independent human references in NIST 2004 data were identical ([Kauchak & Barzilay 2006](https://md.research.elamur.ai/free/en/papers/kauchak-2006-paraphrasing-automatic-evaluation/)).
- Long-range word-order errors are barely penalised. A Japanese–English output that swaps cause and effect still scored a single-sentence BLEU of 0.740 on the 0–1 scale. An order-based Kendall τ for the same output was −0.236 ([Isozaki et al. 2010](https://md.research.elamur.ai/free/en/papers/isozaki-2010-automatic-evaluation-translation-quality-distant/)).
- Recall is missing, and the brevity penalty is only a partial substitute ([Banerjee & Lavie 2005](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)).
- Single-sentence scores are unstable ([Lin & Och 2004](https://md.research.elamur.ai/free/en/papers/lin-2004-automatic-evaluation-machine-translation-quality/)).
- Small but critical errors change few n-grams. Examples are wrong numbers, wrong named entities and flipped negation. The COMET-22 authors note that neural metrics also often miss such errors, and they added word-level error tagging to catch them ([Rei et al. 2022](https://md.research.elamur.ai/free/en/papers/rei-2022-comet-22-unbabel-ist-2022/)).

A further question is whether a metric that ranks outputs well also makes a good training objective. The evidence from Meteor tuning suggests the two goals can diverge ([Denkowski & Lavie 2011](https://md.research.elamur.ai/free/en/papers/denkowski-2011-meteor-1-3-automatic-metric/)).

As learned metrics take over, BLEU's remaining advantages are transparency, speed and comparability with decades of earlier results. How to weigh those against weaker agreement with human judgment is still debated.

## Key papers

The works this page cites, the most cited first; citations as counted by OpenAlex plus the papers of this library that cite the work, read 2026-10-23.

- [Bleu: a Method for Automatic Evaluation of Machine Translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/) — Kishore Papineni, Salim Roukos, Todd Ward et al. · 2002 · 32,065 citations
- [ROUGE: A Package for Automatic Evaluation of Summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/) — Chin-Yew Lin · 2004 · 5,641 citations
- [METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/) — Satanjeev Banerjee, Alon Lavie · 2005 · 5,483 citations
- [A Call for Clarity in Reporting BLEU Scores](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/) — Matt Post · 2018 · 4,130 citations
- [Statistical Significance Tests for Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/) — Philipp Koehn · 2004 · 2,444 citations
- [Meteor Universal: Language Specific Translation Evaluation for Any Target Language](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/) — Michael Denkowski, Alon Lavie · 2014 · 2,392 citations
- [chrF: character n-gram F-score for automatic MT evaluation](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/) — Maja Popović · 2015 · 2,105 citations
- [Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics](https://md.research.elamur.ai/free/en/papers/lin-2003-automatic-evaluation-summaries-n-gram/) — Chin-Yew Lin, Eduard Hovy · 2003 · 1,967 citations
- [COMET: A Neural Framework for MT Evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/) — Ricardo Rei, Craig Stewart, Ana C Farinha et al. · 2020 · 1,784 citations
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://md.research.elamur.ai/free/en/papers/hessel-2021-clipscore-reference-free-evaluation-metric/) — Jack Hessel, Ari Holtzman, Maxwell Forbes et al. · 2021 · 1,498 citations
- [Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics](https://md.research.elamur.ai/free/en/papers/lin-2004-automatic-evaluation-machine-translation-quality/) — Chin-Yew Lin, Franz Josef Och · 2004 · 937 citations
- [chrF++: words helping character n-grams](https://md.research.elamur.ai/free/en/papers/popovic-2017-chrf-words-helping-character-n/) — Maja Popović · 2017 · 749 citations
- [Edinburgh System Description for the 2005 IWSLT Speech Translation Evaluation](https://md.research.elamur.ai/free/en/papers/koehn-2005-edinburgh-system-description-2005-iwslt/) — Philipp Koehn, Amittai Axelrod, Alexandra Birch Mayne et al. · 2005 · 545 citations
- [Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems](https://md.research.elamur.ai/free/en/papers/denkowski-2011-meteor-1-3-automatic-metric/) — Michael Denkowski, Alon Lavie · 2011 · 541 citations
- [A Structured Review of the Validity of BLEU](https://md.research.elamur.ai/free/en/papers/reiter-2018-structured-review-validity-bleu/) — Ehud Reiter · 2018 · 538 citations
- [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://md.research.elamur.ai/free/en/papers/liu-2016-how-not-evaluate-your-dialogue/) — Chia-Wei Liu, Ryan Lowe, Iulian Serban et al. · 2016 · 500 citations
- [Automatic Evaluation of Translation Quality for Distant Language Pairs](https://md.research.elamur.ai/free/en/papers/isozaki-2010-automatic-evaluation-translation-quality-distant/) — Hideki Isozaki, Tsutomu Hirao, Kevin Duh et al. · 2010 · 461 citations
- [COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/rei-2022-comet-22-unbabel-ist-2022/) — Ricardo Rei, José G. C. de Souza, Duarte Alves et al. · 2022 · 433 citations
- [Manual and Automatic Evaluation of Machine Translation between European Languages](https://md.research.elamur.ai/free/en/papers/koehn-2006-manual-automatic-evaluation-machine-translation/) — Philipp Koehn, Christof Monz · 2006 · 433 citations
- [METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments](https://md.research.elamur.ai/free/en/papers/lavie-2007-meteor-automatic-metric-mt-evaluation/) — Alon Lavie, Abhaya Agarwal · 2007 · 387 citations
- [Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics](https://md.research.elamur.ai/free/en/papers/mathur-2020-tangled-up-bleu-reevaluating-evaluation/) — Nitika Mathur, Timothy Baldwin, Trevor Cohn · 2020 · 385 citations
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://md.research.elamur.ai/free/en/papers/guerreiro-2024-xcomet-transparent-machine-translation-evaluation/) — Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt et al. · 2024 · 333 citations
- [Paraphrasing for Automatic Evaluation](https://md.research.elamur.ai/free/en/papers/kauchak-2006-paraphrasing-automatic-evaluation/) — David Kauchak, Regina Barzilay · 2006 · 321 citations
- [On Some Pitfalls in Automatic Evaluation and Significance Testing for MT](https://md.research.elamur.ai/free/en/papers/riezler-2005-some-pitfalls-automatic-evaluation-significance/) — Stefan Riezler, John T. Maxwell · 2005 · 318 citations
- [Re-evaluating the Role of Bleu in Machine Translation Research](https://md.research.elamur.ai/free/en/papers/burch-2006-re-evaluating-role-bleu-machine/) — Chris Callison-Burch, Miles Osborne, Philipp Koehn · 2006 · 292 citations
- [Why We Need New Evaluation Metrics for NLG](https://md.research.elamur.ai/free/en/papers/novikova-2017-why-we-need-new-evaluation/) — Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry et al. · 2017 · 287 citations

## Related concepts

- [Language model](https://md.research.elamur.ai/free/en/concepts/language-model/) — A language model is a probability distribution over sequences of tokens (words, subwords or characters). It is usually factored into…
- [Transformer and attention](https://md.research.elamur.ai/free/en/concepts/transformer/) — The Transformer is a neural network architecture that represents a sequence by letting each element gather information from all others…
- [Semantic similarity](https://md.research.elamur.ai/free/en/concepts/semantic-similarity/) — Semantic similarity is a graded measure of how close two linguistic units, such as words, concepts, sentences or documents, are in meaning.…
- [Subword tokenization](https://md.research.elamur.ai/free/en/concepts/subword-tokenization/) — Subword tokenization is a way of splitting text into units smaller than words but usually larger than single characters. A vocabulary of…

## More papers

Topics: [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/)

Shelves: [🌐 Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) · [💬 Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

- [next concept → Child-directed speech](https://md.research.elamur.ai/free/en/concepts/child-directed-speech/)
- [↑ back to concepts](https://md.research.elamur.ai/free/en/concepts/)