---
title: "Fluency, Adequacy, or HTER? Exploring — Snover 2009"
description: "Automatic MT metrics are judged by how well they agree with human assessments."
authors: ["Matthew Snover", "Nitin Madnani", "Bonnie Dorr", "Richard Schwartz"]
year: 2009
venue: "Proceedings of the Fourth Workshop on Statistical Machine Translation"
source: "https://aclanthology.org/W09-0441/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/snover-2009-fluency-adequacy-or-hter-exploring/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/snover-2009-fluency-adequacy-or-hter-exploring/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2009](https://md.research.elamur.ai/free/papers/2009/) / Fluency, Adequacy, or HTER? Exploring Different Human Judgments with a Tunable MT Metric

# Fluency, Adequacy, or HTER? Exploring Different Human Judgments with a Tunable MT Metric

**Authors:** [Matthew Snover](https://md.research.elamur.ai/free/authors/matthew-snover/), [Nitin Madnani](https://md.research.elamur.ai/free/authors/nitin-madnani/), [Bonnie Dorr](https://md.research.elamur.ai/free/authors/bonnie-dorr/), [Richard Schwartz](https://md.research.elamur.ai/free/authors/richard-schwartz/) · **Year:** 2009 · **Venue:** Proceedings of the Fourth Workshop on Statistical Machine Translation · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W09-0441/) · [PDF](https://aclanthology.org/W09-0441.pdf)

**Citations:** [115](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Automatic MT metrics are judged by how well they agree with human assessments. However, Fluency, Adequacy and HTER capture different aspects of quality, and a single metric with fixed weights cannot reveal those differences. The authors introduce TER-Plus (TERp), which extends Translation Edit Rate (TER) in two ways. First, it adds new alignment operations: stem matches via the Porter stemmer, synonym matches via WordNet, and paraphrase substitutions scored with probabilities from a phrase table. Second, it makes the cost of every edit type a tunable parameter, optimised to maximise correlation with a chosen human judgment. TERp performed strongly in NIST Metrics MATR 2008, achieving the best average rank for both Pearson and Spearman correlation. The authors then tuned it separately to Fluency/Adequacy data and to GALE HTER data and compared the learned costs. HTER treats edits quite differently from the other two judgments, and Fluency and Adequacy also differ from each other. This shows that the choice of human judgment shapes what a "good" metric rewards.

## Key points
- TERp keeps TER's operations (match, insertion, deletion, substitution, shift) and adds stem match, synonym match and phrase substitution. Shifts are also allowed for stems, synonyms and paraphrases.
- The cost of a phrase substitution has four free parameters (w1–w4). It combines the paraphrase's log probability, its raw probability and the internal edit cost of aligning the two phrases. The cost is clipped so it never goes below zero.
- The baselines were BLEU (IBM version, case-insensitive, 4-grams, plus a BLEU-2 variant), METEOR and TER.
- The HTER data came from GALE 2007 (Phase 2.0): 6 system-combination outputs (3 Arabic→English, 3 Chinese→English), totalling 16,267 segments in 1,568 documents. The Fluency/Adequacy set had 7,452 segments in 900 documents.
- Learned costs differed by judgment:
  - **Adequacy and Fluency:** insertions are cheap, deletions are expensive, and stem and synonym matches are free.
  - **HTER:** insertion and deletion costs are balanced, stem and synonym edits are penalised, and shifts are costly.
  - **Fluency vs. Adequacy:** Fluency's shift cost is more than double Adequacy's, and Fluency is friendlier to paraphrases.

## Interesting ideas
- **Tuned weights as a diagnostic tool.** Fitting an interpretable metric to different human judgments, then reading the learned weights, turns metric tuning into a way of probing what each annotation scheme actually values. This approach could be used to compare annotation guidelines in other generation or labelling tasks.
- **Scaling a cost by resource confidence.** A paraphrase's probability discounts its alignment cost, so the metric trusts confident paraphrases more. Any matcher that uses noisy external resources (lexicons, synonym lists) could weight them by confidence in a similar way.
- **Interpretable over black-box parameters.** Each tunable parameter corresponds to a nameable edit type, so the tuned metric stays explainable. This is useful for any evaluation metric that needs to be justified to stakeholders.
- **Metrics as extrinsic tests of other resources.** The authors observe that pivot-method paraphrase probabilities seem poorly suited to deciding whether a paraphrase marks a correct translation. In general, plugging a resource into a correlation-tuned metric can serve as an external test of that resource's quality.
- **Separating evaluation from optimisation.** A metric that works well for final evaluation may not work well as a tuning objective, for example when it must rank outputs from systems already optimised for it. Evaluation setups should test these two roles separately.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/)

## Cited by

Cited by

115 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 107 not yet.

- [Findings of the 2013 Workshop on Statistical Machine Translation](https://md.research.elamur.ai/free/papers/bojar-2013-findings-2013-workshop-statistical-machine/)
- [Findings of the 2009 Workshop on Statistical Machine Translation](https://md.research.elamur.ai/free/papers/burch-2009-findings-2009-workshop-statistical-machine/)
- [Meteor Universal: Language Specific Translation Evaluation for Any Target Language](https://md.research.elamur.ai/free/papers/denkowski-2014-meteor-universal-language-specific-translation/)
- [Submodular Optimization-based Diverse Paraphrasing and its Effectiveness in Data Augmentation](https://md.research.elamur.ai/free/papers/kumar-2019-submodular-optimization-based-diverse-paraphrasing/)
- [Dear Sir or Madam, May I Introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://md.research.elamur.ai/free/papers/rao-2018-dear-sir-or-madam-may/)
- [Semantic Structural Evaluation for Text Simplification](https://md.research.elamur.ai/free/papers/sulem-2018-semantic-structural-evaluation-text-simplification/)
- [Learning to Simplify Sentences with Quasi-Synchronous Grammar and Integer Programming](https://md.research.elamur.ai/free/papers/woodsend-2011-learning-simplify-sentences-quasi-synchronous/)
- [RED: A Reference Dependency Based MT Evaluation Metric](https://md.research.elamur.ai/free/papers/yu-2014-red-reference-dependency-based-mt/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{snover2009fluency,
  title = {{Fluency, Adequacy, or HTER? Exploring Different Human Judgments with a Tunable MT Metric}},
  author = {Snover, Matthew and Madnani, Nitin and Dorr, Bonnie and Schwartz, Richard},
  booktitle = {Proceedings of the Fourth Workshop on Statistical Machine Translation},
  year = {2009},
  url = {https://aclanthology.org/W09-0441/}
}
```

- [← previous Findings of the 2009 Workshop on Statistical Machine Translation](https://md.research.elamur.ai/free/papers/burch-2009-findings-2009-workshop-statistical-machine/)
- [next → Global Learning of Noun Phrase Anaphoricity in Coreference Resolution via Label Propagation](https://md.research.elamur.ai/free/papers/zhou-2009-global-learning-noun-phrase-anaphoricity/)
- [↑ back to papers of 2009](https://md.research.elamur.ai/free/papers/2009/)

## References

18 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL 2005 Workshop on Intrinsic and Extrinsic Evaulation Measures for MT and/or Summarization. [find](https://scholar.google.com/scholar?q=Satanjeev+Banerjee+and+Alon+Lavie%2E+2005%2E+METEOR%3A+An+Automatic+Metric+for+MT+Evaluation+with+Improved+Correlation+with+Human+Judgments%2E+In+Proceedings+of+the+ACL+2005+Workshop+on+Intrinsic+and+Extrinsi) [our card](https://md.research.elamur.ai/free/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)
2. Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with Bilingual Parallel Corpora. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL 2005), pages 597–604, Ann Arbor, Michigan, June. [find](https://scholar.google.com/scholar?q=Colin+Bannard+and+Chris+Callison%2DBurch%2E+2005%2E+Paraphrasing+with+Bilingual+Parallel+Corpora%2E+In+Proceedings+of+the+43rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL+2005%29%2C+pages) [our card](https://md.research.elamur.ai/free/papers/bannard-2005-paraphrasing-bilingual-parallel-corpora/)
3. Chris Callison-Burch. 2008. Syntactic constraints on paraphrases extracted from parallel corpora. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 196– 205, Honolulu, Hawaii, October. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chris+Callison%2DBurch%2E+2008%2E+Syntactic+constraints+on+paraphrases+extracted+from+parallel+corpora%2E+In+Proceedings+of+the+2008+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+196%E2%80%93+) [our card](https://md.research.elamur.ai/free/papers/burch-2008-syntactic-constraints-paraphrases-extracted-parallel/)
4. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. MIT Press. [http://www.cogsci.princeton.edu/˜wn](http://www.cogsci.princeton.edu/˜wn) [2000, September 7]. [↗](http://www.cogsci.princeton.edu/˜wn)
5. David Kauchak and Regina Barzilay. 2006. Paraphrasing for Automatic Evaluation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL, pages 455– 462. [find](https://scholar.google.com/scholar?q=David+Kauchak+and+Regina+Barzilay%2E+2006%2E+Paraphrasing+for+Automatic+Evaluation%2E+In+Proceedings+of+the+Human+Language+Technology+Conference+of+the+North+American+Chapter+of+the+ACL%2C+pages+455%E2%80%93+462%2E) [our card](https://md.research.elamur.ai/free/papers/kauchak-2006-paraphrasing-automatic-evaluation/)
6. Gregor Leusch, Nicola Ueffing, and Hermann Ney. 2006. CDER: Efficient MT Evaluation Using Block Movements. In Proceedings of the 11th Conferenceof the European Chapter of the Association for Computational Linguistics (EACL 2006). [find](https://scholar.google.com/scholar?q=Gregor+Leusch%2C+Nicola+Ueffing%2C+and+Hermann+Ney%2E+2006%2E+CDER%3A+Efficient+MT+Evaluation+Using+Block+Movements%2E+In+Proceedings+of+the+11th+Conferenceof+the+European+Chapter+of+the+Association+for+Computati) [our card](https://md.research.elamur.ai/free/papers/leusch-2006-cder-efficient-mt-evaluation-block/)
7. V. I. Levenshtein. 1966. Binary Codes Capable of Correcting Deletions, Insertions, and Reversals. Soviet Physics Doklady, 10:707–710. [find](https://scholar.google.com/scholar?q=V%2E+I%2E+Levenshtein%2E+1966%2E+Binary+Codes+Capable+of+Correcting+Deletions%2C+Insertions%2C+and+Reversals%2E+Soviet+Physics+Doklady%2C+10%3A707%E2%80%93710%2E)
8. Daniel Lopresti and Andrew Tomkins. 1997. Block edit models for approximate string matching. Theoretical Computer Science, 181(1):159–179, July. [↗](https://doi.org/10.1016/s0304-3975(96)00268-x)
9. Nitin Madnani, Necip Fazil Ayan, Philip Resnik, and Bonnie J. Dorr. 2007. Using paraphrases for parameter tuning in statistical machine translation. In Proceedings of the Workshop on Statistical Machine Translation, Prague, Czech Republic, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nitin+Madnani%2C+Necip+Fazil+Ayan%2C+Philip+Resnik%2C+and+Bonnie+J%2E+Dorr%2E+2007%2E+Using+paraphrases+for+parameter+tuning+in+statistical+machine+translation%2E+In+Proceedings+of+the+Workshop+on+Statistical+Machi)
10. Nitin Madnani, Philip Resnik, Bonnie J. Dorr, and Richard Schwartz. 2008. Are Multiple Reference Translations Necessary? Investigating the Value of Paraphrased Reference Translations in Parameter Optimization. In Proceedings of the Eighth Conference of the Association for Machine Translation in the Americas, October. [find](https://scholar.google.com/scholar?q=Nitin+Madnani%2C+Philip+Resnik%2C+Bonnie+J%2E+Dorr%2C+and+Richard+Schwartz%2E+2008%2E+Are+Multiple+Reference+Translations+Necessary%3F+Investigating+the+Value+of+Paraphrased+Reference+Translations+in+Parameter+Opti)
11. S. Niessen, F.J. Och, G. Leusch, and H. Ney. 2000. An evaluation tool for machine translation: Fast evaluation for MT research. In Proceedings of the 2nd International Conference on Language Resources and Evaluation (LREC-2000), pages 39–45. [find](https://scholar.google.com/scholar?q=S%2E+Niessen%2C+F%2EJ%2E+Och%2C+G%2E+Leusch%2C+and+H%2E+Ney%2E+2000%2E+An+evaluation+tool+for+machine+translation%3A+Fast+evaluation+for+MT+research%2E+In+Proceedings+of+the+2nd+International+Conference+on+Language+Resources) [our card](https://md.research.elamur.ai/free/papers/en-2000-evaluation-tool-machine-translation-fast/)
12. Joseph Olive. 2005. Global Autonomous Language Exploitation (GALE). DARPA/IPTO Proposer Information Pamphlet. [find](https://scholar.google.com/scholar?q=Joseph+Olive%2E+2005%2E+Global+Autonomous+Language+Exploitation+%28GALE%29%2E+DARPA%2FIPTO+Proposer+Information+Pamphlet%2E)
13. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Traslation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+Method+for+Automatic+Evaluation+of+Machine+Traslation%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+Association+for+Compu)
14. Martin F. Porter. 1980. An algorithm for suffic stripping. Program, 14(3):130–137. [find](https://scholar.google.com/scholar?q=Martin+F%2E+Porter%2E+1980%2E+An+algorithm+for+suffic+stripping%2E+Program%2C+14%283%29%3A130%E2%80%93137%2E)
15. Mark Przybocki, Kay Peterson, and Sebastian Bronsart. 2008. Official results of the NIST 2008 ”Metrics for MAchine TRanslation” Challenge (MetricsMATR08). [http://nist.gov/speech/tests/metricsmatr/2008/results/](http://nist.gov/speech/tests/metricsmatr/2008/results/), October. [↗](http://nist.gov/speech/tests/metricsmatr/2008/results/)
16. Antti-Veikko Rosti, Spyros Matsoukas, and Richard Schwartz. 2007. Improved word-level system combination for machine translation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 312–319, Prague, Czech Republic, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Antti%2DVeikko+Rosti%2C+Spyros+Matsoukas%2C+and+Richard+Schwartz%2E+2007%2E+Improved+word%2Dlevel+system+combination+for+machine+translation%2E+In+Proceedings+of+the+45th+Annual+Meeting+of+the+Association+of+Comput)
17. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of Association for Machine Translation in the Americas. [find](https://scholar.google.com/scholar?q=Matthew+Snover%2C+Bonnie+Dorr%2C+Richard+Schwartz%2C+Linnea+Micciulla%2C+and+John+Makhoul%2E+2006%2E+A+Study+of+Translation+Edit+Rate+with+Targeted+Human+Annotation%2E+In+Proceedings+of+Association+for+Machine+Tran) [our card](https://md.research.elamur.ai/free/papers/snover-2006-study-translation-edit-rate-targeted/)
18. Liang Zhou, Chon-Yew Lin, and Eduard Hovy. 2006. Re-evaluating Machine Translation Results with Paraphrase Support. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006), pages 77–84. [find](https://scholar.google.com/scholar?q=Liang+Zhou%2C+Chon%2DYew+Lin%2C+and+Eduard+Hovy%2E+2006%2E+Re%2Devaluating+Machine+Translation+Results+with+Paraphrase+Support%2E+In+Proceedings+of+the+2006+Conference+on+Empirical+Methods+in+Natural+Language+Proce)