---
title: "chrF — definition and key papers — Elamur Research"
description: "The character n-gram F-score (chrF) is an automatic metric for machine translation and other text generation."
canonical: "https://research.elamur.ai/free/en/concepts/chrf/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/concepts/chrf/)

[home](https://md.research.elamur.ai/free/en/) / [concepts](https://md.research.elamur.ai/free/en/concepts/) / chrF

# 🔡 chrF

Concept · 27 key papers · [all concepts](https://md.research.elamur.ai/free/en/concepts/)

Part of: [🗨️ Natural language processing](https://md.research.elamur.ai/free/en/concepts/natural-language-processing/)

The character n-gram F-score (chrF) is an automatic metric for machine translation and other text generation. It compares a system output with a human reference by counting shared character sequences and combines character precision and recall into a recall-weighted F-score.

Also: chrF2, chrF3, chrF+, chrF++, character n-gram F-score

## What it is

chrF is a way to score a translation automatically. You give it a machine output and one or more human reference translations. It returns a number that is higher the more short character sequences the two texts share. It belongs to the family of overlap metrics in [machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) evaluation, next to the older [BLEU score](https://md.research.elamur.ai/free/en/concepts/bleu/) ([Papineni et al. 2002](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)) and METEOR ([Denkowski & Lavie 2014](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/)).

The difference is the unit of comparison. BLEU matches whole words. chrF matches character n-grams, such as "hou", "ous" and "use". A translation that gets a word's stem right but its ending wrong therefore still earns partial credit. Maja Popović proposed the metric in 2015. She presented it as a simple scorer that needs no external tools or linguistic resources and does not depend on the language or on how the text is tokenized ([Popović 2015](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/)).

## The key idea

### The F-beta formula

For every n-gram order n from 1 up to a maximum N, the hypothesis and the reference are cut into overlapping character n-grams. Counts are clipped as in BLEU, so a hypothesis n-gram is matched at most as often as it occurs in the reference ([Papineni et al. 2002](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)). Precision and recall are then averaged over the orders and combined:

```
chrP = (1/N) · Σₙ matchesₙ / hyp_ngramsₙ
chrR = (1/N) · Σₙ matchesₙ / ref_ngramsₙ
chrFβ = (1 + β²) · chrP · chrR / (β² · chrP + chrR)
```

The symbols mean the following:

- matchesₙ is the clipped number of character n-grams of order n that the hypothesis shares with the reference.
- hyp_ngramsₙ and ref_ngramsₙ are the totals of such n-grams in the hypothesis and in the reference.
- chrP and chrR are the averaged character precision and recall.
- β is the weight that sets how much more recall matters than precision, following the classic F-measure of van Rijsbergen (1979).

The 2015 paper tested 4-, 6- and 10-grams. It also tried a variable length tied to the average word length. 6-grams worked best, and treating the space as a separate character gave no gain ([Popović 2015](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/)).

The paper compared β = 1 with β = 3, and chrF3 correlated best with the human rankings of the time ([Popović 2015](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/)). The Conference on Machine Translation (WMT) then switched to direct assessment, in which human raters give each translation an absolute quality score. β was re-tested against these scores. β = 2 and β = 3 tied on average, and β = 2 became the default because a heavier recall weight can push tuned systems towards overly long outputs ([Popović 2017](https://md.research.elamur.ai/free/en/papers/popovic-2017-chrf-words-helping-character-n/)).

Length bias from tuning can also run the other way. The Edinburgh system for the 2005 International Workshop on Spoken Language Translation (IWSLT) was tuned towards BLEU, which favours precision. Its output came out too short, and this badly hurt its scores under NIST, a related n-gram metric ([Koehn et al. 2005](https://md.research.elamur.ai/free/en/papers/koehn-2005-edinburgh-system-description-2005-iwslt/)).

### Why characters help with rich morphology

In languages with rich [morphology](https://md.research.elamur.ai/free/en/concepts/morphology/), one lemma appears in many [word forms](https://md.research.elamur.ai/free/en/concepts/word-form/). A word-level metric treats "domu" and "dom" in Czech as unrelated. Character n-grams see that they share most of their material. The 2015 study covered Hindi, Czech, Russian, German, French, Spanish and English. It argued that an inexpensive character-based metric is especially valuable when translating into morphologically rich languages ([Popović 2015](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/)).

The same intuition underlies character n-gram representations outside evaluation: CHARAGRAM embeddings handled rare words, misspellings and morphological variants well ([Wieting et al. 2016](https://md.research.elamur.ai/free/en/papers/wieting-2016-charagram-embedding-words-sentences-character/)).

### A small worked example

Take the reference "houses" and the hypothesis "house". To keep the arithmetic short, use only orders 1 and 2 (real chrF goes to 6).

| Order | Hypothesis n-grams | Reference n-grams | Matches | Precision | Recall |
| --- | --- | --- | --- | --- | --- |
| 1 | h o u s e (5) | h o u s e s (6) | 5 | 5/5 | 5/6 |
| 2 | ho ou us se (4) | ho ou us se es (5) | 4 | 4/4 | 4/5 |

Averaging gives chrP = 1 and chrR ≈ 0.817. With β = 2, chrF2 = 5 · 1 · 0.817 / (4 · 1 + 0.817) ≈ 0.85.

An exact-match word metric such as BLEU would give this pair zero, because "house" ≠ "houses". chrF instead registers a near miss that is penalised mainly on the recall side.

### chrF++: adding word n-grams

In 2017 Popović scored character and word n-gram orders separately against direct assessment on WMT-15 and WMT-16 data. Character 3- and 4-grams were the strongest single orders. Word unigrams and bigrams correlated well, while word 3- and 4-grams tended to underrate good translations ([Popović 2017](https://md.research.elamur.ai/free/en/papers/popovic-2017-chrf-words-helping-character-n/)).

chrF++ therefore averages character 1–6-grams with word 1–2-grams under uniform weights. On that WMT-15/16 data, this raised segment-level correlation with direct assessment above plain chrF. A unigram-only variant, chrF+, did best for English→Russian, though that test set was small ([Popović 2017](https://md.research.elamur.ai/free/en/papers/popovic-2017-chrf-words-helping-character-n/)).

## History and founding papers

Before 2015, character n-grams appeared only inside larger combined metrics such as BEER and MTeRater. They had not been tested as a metric in their own right ([Popović 2015](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/)). The field's reference points were BLEU, with its clipped word precisions combined by a geometric mean ([Papineni et al. 2002](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)), and paired bootstrap resampling for judging whether score differences are significant ([Koehn 2004](https://md.research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/)). A year earlier, Meteor Universal had tried to carry a richer metric to languages without stemmers or WordNet ([Denkowski & Lavie 2014](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/)).

The founding paper scored all WMT 12–14 systems. In system-level Pearson correlation with human judgments on WMT 14, chrF3 reached 0.857, against 0.845 for BLEU. For translation out of English, its average segment-level Kendall τ rank correlation beat even the best WMT 14 metrics ([Popović 2015](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/)). Compared direction by direction, chrF3 agreed with human judgments better than BLEU in 80% of translation directions ([Popović 2015](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/)).

Meta-evaluation itself was changing during these years. Graham and colleagues replaced relative rankings with repeated absolute adequacy scores. They also used the Williams test, which tells whether one metric's correlation with humans is significantly higher than a rival's ([Graham et al. 2015](https://md.research.elamur.ai/free/en/papers/graham-2015-accurate-evaluation-segment-level-machine/)). chrF++ came out of re-testing chrF against direct assessment scores, the newer official WMT judgment type ([Popović 2017](https://md.research.elamur.ai/free/en/papers/popovic-2017-chrf-words-helping-character-n/)). Discourse-aware metrics explored a different axis. DISCOTKparty added discourse-tree similarity to lexical metrics and ranked first in the WMT14 metrics task at both system and segment level, as a later journal account describes ([Joty et al. 2017](https://md.research.elamur.ai/free/en/papers/joty-2017-discourse-structure-machine-translation-evaluation/)).

## How it is used today

### In WMT and toolkits

By WMT18, chrF++ was one of the standard n-gram baselines against which learned metrics such as RUSE were compared ([Shimanaka et al. 2018](https://md.research.elamur.ai/free/en/papers/shimanaka-2018-ruse-regressor-sentence-embeddings-automatic/)). Popović released a Python script that reads raw text and supports multiple references. It uses β = 2 by default and reports segment scores plus micro- and macro-averaged document scores ([Popović 2017](https://md.research.elamur.ai/free/en/papers/popovic-2017-chrf-words-helping-character-n/)).

Later meta-evaluations scored chrF side by side with BLEU computed through the standardised sacreBLEU implementation ([Mathur et al. 2020](https://md.research.elamur.ai/free/en/papers/mathur-2020-tangled-up-bleu-reevaluating-evaluation/)). One reanalysis of WMT 2019 found that BLEU and TER make wrong pairwise system decisions more often than chrF. Its authors recommended replacing BLEU with chrF, YiSi-1 or ESIM ([Mathur et al. 2020](https://md.research.elamur.ai/free/en/papers/mathur-2020-tangled-up-bleu-reevaluating-evaluation/)). chrF also sits inside composite baselines: one domain-robustness study averaged BLEU, chrF and TER as its surface-form reference point ([Zouhar et al. 2024](https://md.research.elamur.ai/free/en/papers/zouhar-2024-fine-tuned-machine-translation-metrics/)).

### Beside neural metrics

Trained neural metrics now set the pace in [evaluation benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/). xCOMET reported state-of-the-art results at segment, system and error-span level ([Guerreiro et al. 2024](https://md.research.elamur.ai/free/en/papers/guerreiro-2024-xcomet-transparent-machine-translation-evaluation/)). LLM judges have long trailed trained metrics such as MetricX and xCOMET, most clearly per segment ([Feng et al. 2025](https://md.research.elamur.ai/free/en/papers/feng-2025-m-mad-multidimensional-multi-agent/)). This line runs from multilingual zero-shot paraphrase scoring with Prism ([Thompson & Post 2020](https://md.research.elamur.ai/free/en/papers/thompson-2020-automatic-machine-translation-evaluation-many/)) and unified source/reference evaluators such as UniTE ([Wan et al. 2022](https://md.research.elamur.ai/free/en/papers/wan-2022-unite-unified-translation-evaluation/)) to span-tagging metrics such as MaTESe ([Perrella et al. 2022](https://md.research.elamur.ai/free/en/papers/perrella-2022-matese-machine-translation-evaluation-as/)) and xCOMET ([Guerreiro et al. 2024](https://md.research.elamur.ai/free/en/papers/guerreiro-2024-xcomet-transparent-machine-translation-evaluation/)).

[Large language models](https://md.research.elamur.ai/free/en/concepts/large-language-models/) have added generative evaluators:

- AutoMQM, a prompt that has the model list and grade error spans ([Fernandes et al. 2023](https://md.research.elamur.ai/free/en/papers/fernandes-2023-devil-is-errors-leveraging-large/))
- EAPrompt (Error Analysis Prompting), which first identifies errors and then counts them ([Lu et al. 2024](https://md.research.elamur.ai/free/en/papers/lu-2024-error-analysis-prompting-enables-human/))
- M-MAD, a multi-agent debate method ([Feng et al. 2025](https://md.research.elamur.ai/free/en/papers/feng-2025-m-mad-multidimensional-multi-agent/))
- Remedy-R, an evaluator trained with reinforcement learning rather than prompted ([Tan et al. 2026](https://md.research.elamur.ai/free/en/papers/tan-2026-remedy-r-generative-reasoning-machine/))

chrF remains useful because it is deterministic, fast and free of a trained model. That matters for [low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/), for reproducibility, and as a sanity check next to neural scores. Overlap metrics also travel to neighbouring tasks. Older MT metrics such as BLEU, TER and METEOR, used as the only features, made a strong paraphrase-identification baseline ([Madnani et al. 2012](https://md.research.elamur.ai/free/en/papers/madnani-2012-re-examining-machine-translation-metrics/)).

## Open questions

- Semantic depth. Lexical metrics, chrF included, lack the semantic depth needed to judge non-literal translation of slang, idioms and poetry ([Tian et al. 2026](https://md.research.elamur.ai/free/en/papers/tian-2026-beyond-literal-mapping-benchmarking-improving/)). In chat translation, reference-based neural metrics correlated best with human Multidimensional Quality Metrics (MQM) scores, which are derived from annotated errors ([Agrawal et al. 2024](https://md.research.elamur.ai/free/en/papers/agrawal-2024-assessing-role-context-chat-translation/)).
- Small score gaps. In WMT 2019 data, a 1–2 point BLEU gain matched a real human-judged improvement only about half the time ([Mathur et al. 2020](https://md.research.elamur.ai/free/en/papers/mathur-2020-tangled-up-bleu-reevaluating-evaluation/)). Whether chrF differences of similar size are more trustworthy is not settled, so significance testing remains advisable ([Koehn 2004](https://md.research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/)).
- Test-set artefacts. Some past WMT test sets were partly reverse-created, with their sources translated from the target side. When these sets were split into forward- and reverse-created halves, the correlation between the BLEU system rankings on the two halves fell as low as 0.106 ([Graham et al. 2020](https://md.research.elamur.ai/free/en/papers/graham-2020-statistical-power-translationese-machine-translation/)). Benchmark contamination can also inflate surface-overlap scores ([Tan et al. 2026](https://md.research.elamur.ai/free/en/papers/tan-2026-when-flores-bloomz-wrong-cross/)). A string-matching metric cannot detect either problem by itself.
- Modality and granularity. Speech translation is still scored mostly with text-only metrics ([Zarzu & Zouhar 2026](https://md.research.elamur.ai/free/en/papers/zarzu-2026-hurdles-automatic-metric-speech-translation/)). Error-based human schemes measure effort in ways a single F-score does not capture. One example, tested on Arabic, orders post-editing error classes by the mental effort needed to fix them ([Temnikova et al. 2016](https://md.research.elamur.ai/free/en/papers/temnikova-2016-applying-cognitive-machine-translation-evaluation/)).
- The role of β and word n-grams. It is unclear whether these settings should change for scripts written without spaces, or for systems built on [subword tokenization](https://md.research.elamur.ai/free/en/concepts/subword-tokenization/). The question has received little systematic study since the 2017 re-tuning ([Popović 2017](https://md.research.elamur.ai/free/en/papers/popovic-2017-chrf-words-helping-character-n/)).

## Key papers

The works this page cites, the most cited first; citations as counted by OpenAlex plus the papers of this library that cite the work, read 2026-10-24.

- [Bleu: a Method for Automatic Evaluation of Machine Translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/) — Kishore Papineni, Salim Roukos, Todd Ward et al. · 2002 · 32,134 citations
- [Statistical Significance Tests for Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/) — Philipp Koehn · 2004 · 2,444 citations
- [Meteor Universal: Language Specific Translation Evaluation for Any Target Language](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/) — Michael Denkowski, Alon Lavie · 2014 · 2,399 citations
- [chrF: character n-gram F-score for automatic MT evaluation](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/) — Maja Popović · 2015 · 2,107 citations
- [chrF++: words helping character n-grams](https://md.research.elamur.ai/free/en/papers/popovic-2017-chrf-words-helping-character-n/) — Maja Popović · 2017 · 750 citations
- [Edinburgh System Description for the 2005 IWSLT Speech Translation Evaluation](https://md.research.elamur.ai/free/en/papers/koehn-2005-edinburgh-system-description-2005-iwslt/) — Philipp Koehn, Amittai Axelrod, Alexandra Birch Mayne et al. · 2005 · 545 citations
- [Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics](https://md.research.elamur.ai/free/en/papers/mathur-2020-tangled-up-bleu-reevaluating-evaluation/) — Nitika Mathur, Timothy Baldwin, Trevor Cohn · 2020 · 386 citations
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://md.research.elamur.ai/free/en/papers/guerreiro-2024-xcomet-transparent-machine-translation-evaluation/) — Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt et al. · 2024 · 333 citations
- [Re-examining Machine Translation Metrics for Paraphrase Identification](https://md.research.elamur.ai/free/en/papers/madnani-2012-re-examining-machine-translation-metrics/) — Nitin Madnani, Joel Tetreault, Martin Chodorow · 2012 · 272 citations
- [Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing](https://md.research.elamur.ai/free/en/papers/thompson-2020-automatic-machine-translation-evaluation-many/) — Brian Thompson, Matt Post · 2020 · 259 citations
- [Charagram: Embedding Words and Sentences via Character n-grams](https://md.research.elamur.ai/free/en/papers/wieting-2016-charagram-embedding-words-sentences-character/) — John Wieting, Mohit Bansal, Kevin Gimpel et al. · 2016 · 241 citations
- [The Devil Is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/fernandes-2023-devil-is-errors-leveraging-large/) — Patrick Fernandes, Daniel Deutsch, Mara Finkelstein et al. · 2023 · 156 citations
- [Accurate Evaluation of Segment-level Machine Translation Metrics](https://md.research.elamur.ai/free/en/papers/graham-2015-accurate-evaluation-segment-level-machine/) — Yvette Graham, Timothy Baldwin, Nitika Mathur · 2015 · 156 citations
- [Statistical Power and Translationese in Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/graham-2020-statistical-power-translationese-machine-translation/) — Yvette Graham, Barry Haddow, Philipp Koehn · 2020 · 150 citations
- [RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/shimanaka-2018-ruse-regressor-sentence-embeddings-automatic/) — Hiroki Shimanaka, Tomoyuki Kajiwara, Mamoru Komachi · 2018 · 141 citations
- [Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models](https://md.research.elamur.ai/free/en/papers/lu-2024-error-analysis-prompting-enables-human/) — Qingyu Lu, Baopu Qiu, Liang Ding et al. · 2024 · 135 citations
- [UniTE: Unified Translation Evaluation](https://md.research.elamur.ai/free/en/papers/wan-2022-unite-unified-translation-evaluation/) — Yu Wan, Dayiheng Liu, Baosong Yang et al. · 2022 · 100 citations
- [Discourse Structure in Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/joty-2017-discourse-structure-machine-translation-evaluation/) — Shafiq Joty, Francisco Guzmán, Lluís Màrquez et al. · 2017 · 40 citations
- [Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains](https://md.research.elamur.ai/free/en/papers/zouhar-2024-fine-tuned-machine-translation-metrics/) — Vilém Zouhar, Shuoyang Ding, Anna Currey et al. · 2024 · 32 citations
- [MaTESe: Machine Translation Evaluation as a Sequence Tagging Problem](https://md.research.elamur.ai/free/en/papers/perrella-2022-matese-machine-translation-evaluation-as/) — Stefano Perrella, Lorenzo Proietti, Alessandro Scirè et al. · 2022 · 32 citations
- [M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/feng-2025-m-mad-multidimensional-multi-agent/) — Zhaopeng Feng, Jiayuan Su, Jiamei Zheng et al. · 2025 · 24 citations
- [Assessing the Role of Context in Chat Translation Evaluation: Is Context Helpful and Under What Conditions?](https://md.research.elamur.ai/free/en/papers/agrawal-2024-assessing-role-context-chat-translation/) — Sweta Agrawal, Amin Farajian, Patrick Fernandes et al. · 2024 · 8 citations
- [Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation](https://md.research.elamur.ai/free/en/papers/tian-2026-beyond-literal-mapping-benchmarking-improving/) — Yanzhi Tian, Cunxiang Wang, Zeming Liu et al. · 2026 · 5 citations
- [Applying the Cognitive Machine Translation Evaluation Approach to Arabic](https://md.research.elamur.ai/free/en/papers/temnikova-2016-applying-cognitive-machine-translation-evaluation/) — Irina Temnikova, Wajdi Zaghouani, Stephan Vogel et al. · 2016 · 3 citations
- [Remedy-R: Generative Reasoning for Machine Translation Evaluation without Error Annotations](https://md.research.elamur.ai/free/en/papers/tan-2026-remedy-r-generative-reasoning-machine/) — Shaomu Tan, Ryosuke Mitani, Ritvik Choudhary et al. · 2026 · 2 citations
- [When Flores Bloomz Wrong: Cross-Direction Contamination in Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/tan-2026-when-flores-bloomz-wrong-cross/) — David Tan, Pinzhen Chen, Josef van Genabith et al. · 2026 · 2 citations
- [Hurdles of Automatic Metric for Speech Translation Evaluation](https://md.research.elamur.ai/free/en/papers/zarzu-2026-hurdles-automatic-metric-speech-translation/) — Victor Eugen Zarzu, Vilem Zouhar · 2026 · 1 citation

## Related concepts

- [BLEU score](https://md.research.elamur.ai/free/en/concepts/bleu/) — BLEU score (Bilingual Evaluation Understudy) is an automatic metric for machine translation. It rates a candidate text by how many of its…
- [Morphology](https://md.research.elamur.ai/free/en/concepts/morphology/) — Morphology is the branch of linguistics that studies the internal structure of words. It covers how words are built from meaningful units…

## More papers

Topics: [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/)

Shelves: [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/)

Updated 24 October 2026

- [← previous concept Child-directed speech](https://md.research.elamur.ai/free/en/concepts/child-directed-speech/)
- [next concept → clf: classifier](https://md.research.elamur.ai/free/en/concepts/ud-clf/)
- [↑ back to concepts](https://md.research.elamur.ai/free/en/concepts/)