--- title: "Meteor 1.3: Automatic Metric for Reliable — Denkowski 2011" description: "Meteor 1.3 is the CMU entry to the WMT 2011 shared tasks on automatic evaluation metrics and on tunable metrics." authors: ["Michael Denkowski", "Alon Lavie"] year: 2011 venue: "Proceedings of the Sixth Workshop on Statistical Machine Translation" source: "https://aclanthology.org/W11-2107/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/denkowski-2011-meteor-1-3-automatic-metric/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/denkowski-2011-meteor-1-3-automatic-metric/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2011](https://md.research.elamur.ai/free/en/papers/2011/) / Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems # Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems **Authors:** [Michael Denkowski](https://md.research.elamur.ai/free/en/authors/michael-denkowski/), [Alon Lavie](https://md.research.elamur.ai/free/en/authors/alon-lavie/) · **Year:** 2011 · **Venue:** Proceedings of the Sixth Workshop on Statistical Machine Translation · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/W11-2107/) · [PDF](https://aclanthology.org/W11-2107.pdf) · [OpenAlex](https://openalex.org/W2116492146) **Citations:** 344 citations (openalex-snapshot · 2026-10-14) · [197](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Meteor 1.3 is the CMU entry to the WMT 2011 shared tasks on automatic evaluation metrics and on tunable metrics. It addresses three weaknesses of earlier Meteor versions: poor handling of punctuation, noisy paraphrase matches, and treating every word as equally important. The authors built new resources for all five WMT target languages: - a text normalizer modeled on the Moses tokenizer, - paraphrase tables filtered with a simple substring rule, - function-word lists derived from word frequency in large monolingual corpora. A new δ weight lets the score count content words differently from function words. Parameters were fit by exhaustive search against human judgments. This produced three variants: Ranking (fit to WMT 2009+2010 rankings), Adequacy (fit to NIST adequacy data) and Tuning (balanced by hand). On held-out data, 1.3 beats 1.2 for English, French and Spanish rankings and for adequacy. Czech, German and H-TER show overfitting. In MERT on Urdu-English, the balanced Tuning version beats BLEU-tuned systems on BLEU, TER and Meteor 1.2. The specialized variants, in contrast, tune badly. The work shows that a metric good for judging outputs is not automatically good as an optimization target. ## Key points - **Paraphrase filtering.** Pivot-based paraphrase pairs are discarded whenever either phrase is contained entirely within its counterpart. For example, the English table shrinks from 6.24M to 5.27M pairs and the Czech table from 756K to 684K, which also reduces memory use and loading time. - **Function-word lists.** Every word with relative frequency of at least 10⁻³ in pooled Europarl, news and news-commentary data is listed as a function word. This gives, for example, 93 words for English (836M-word corpus) and 66 for Spanish. Punctuation consistently lands near the top of these lists. - **Scoring.** Alignment combines exact, stem (Snowball), synonym (WordNet, English only) and paraphrase matchers. The score is a weighted F-mean multiplied by a fragmentation penalty, controlled by the parameters α, β, γ, δ and the matcher weights. In all Ranking parameter sets δ is above 0.5, so content words always count more. - **Ranking results (Kendall's τ, tune WMT09 → test WMT10).** English test τ rises from 0.320 to 0.343. Czech and German get worse on test data despite gains on tuning data. - **MERT results.** The setup was Z-MERT on a Moses Urdu-English system, with 3 runs per metric. On newswire, Meteor-tune reached 24.89 BLEU, compared with 23.67 when tuning directly to BLEU. It also scored better TER (69.54 vs 72.48). Meteor-rank collapsed to 19.28 BLEU and 94.64 TER. ## Interesting ideas - **Normalize meaning-equivalent punctuation before scoring.** Joining acronyms (U.N. → UN) and splitting hyphenated words lets differently styled references and hypotheses match. This is a cheap preprocessing step that could help any string-overlap metric, including evaluation of Ukrainian text with inconsistent hyphenation or abbreviations. - **Clean pivot paraphrases with a substring filter.** Dropping pairs where one side merely contains the other removes "word accumulation" noise and shrinks the table at the same time. This is a trivial, language-independent rule worth applying to any automatically mined paraphrase or synonym resource. - **Use frequency-derived function-word lists instead of hand-made stopword lists.** A single relative-frequency threshold on a large corpus yields a list for any language without manual work. Weighting content words above function words then sharpens distinctions between near-identical outputs. - **Tuning targets need balance, not maximum correlation.** Parameter sets that maximize correlation with human judgments can be gamed by the optimizer and push it toward degenerate outputs. When a metric will serve as a training or optimization objective, choose balanced weights (precision vs recall, content vs function words) and validate end-to-end with several independent metrics. - **Average over repeated optimization runs.** Repeating MERT 3 times per metric and reporting the averages controls for optimizer variance. This is good practice for any comparison where the training procedure itself is noisy. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) ## Cited by Cited by 197 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 180 not yet. - [The First Surface Realisation Shared Task: Overview and Evaluation Results](https://md.research.elamur.ai/free/en/papers/belz-2011-first-surface-realisation-shared-task/) - [Automatic MT Error Analysis: Hjerson Helping Addicter](https://md.research.elamur.ai/free/en/papers/berka-2012-automatic-mt-error-analysis-hjerson/) - [Findings of the 2011 Workshop on Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/burch-2011-findings-2011-workshop-statistical-machine/) - [Findings of the 2012 Workshop on Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/burch-2012-findings-2012-workshop-statistical-machine/) - [Meteor Universal: Language Specific Translation Evaluation for Any Target Language](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/) - [Accurate Evaluation of Segment-level Machine Translation Metrics](https://md.research.elamur.ai/free/en/papers/graham-2015-accurate-evaluation-segment-level-machine/) - [Achieving Human Parity on Automatic Chinese to English News Translation](https://md.research.elamur.ai/free/en/papers/hassan-2018-achieving-human-parity-automatic-chinese/) - [Discourse Structure in Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/joty-2017-discourse-structure-machine-translation-evaluation/) - [The NarrativeQA Reading Comprehension Challenge](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/) - [Politeness Transfer: A Tag and Generate Approach](https://md.research.elamur.ai/free/en/papers/madaan-2020-politeness-transfer-tag-generate-approach/) - [Sarcasm SIGN: Interpreting Sarcasm with Sentiment Based Monolingual Machine Translation](https://md.research.elamur.ai/free/en/papers/peled-2017-sarcasm-sign-interpreting-sarcasm-sentiment/) - [COMET: A Neural Framework for MT Evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/) - [Nematus: a Toolkit for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/sennrich-2017-nematus-toolkit-neural-machine-translation/) - [Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches](https://md.research.elamur.ai/free/en/papers/storks-2019-recent-advances-natural-language-inference/) - [Semantic Structural Evaluation for Text Simplification](https://md.research.elamur.ai/free/en/papers/sulem-2018-semantic-structural-evaluation-text-simplification/) - [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/) - [RED: A Reference Dependency Based MT Evaluation Metric](https://md.research.elamur.ai/free/en/papers/yu-2014-red-reference-dependency-based-mt/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{denkowski2011meteor, title = {{Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems}}, author = {Denkowski, Michael and Lavie, Alon}, booktitle = {Proceedings of the Sixth Workshop on Statistical Machine Translation}, year = {2011}, url = {https://aclanthology.org/W11-2107/} } ``` - [← previous MEANT: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility based on semantic roles](https://md.research.elamur.ai/free/en/papers/lo-2011-meant-inexpensive-high-accuracy-semi/) - [next → Mining Subjective Knowledge from Customer Reviews: A Specific Case of Irony Detection](https://md.research.elamur.ai/free/en/papers/reyes-2011-mining-subjective-knowledge-customer-reviews/) - [↑ back to papers of 2011](https://md.research.elamur.ai/free/en/papers/2011/) ## References 20 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proc. of ACL WIEEMMTS 2005. [find](https://scholar.google.com/scholar?q=Satanjeev+Banerjee+and+Alon+Lavie%2E+2005%2E+METEOR%3A+An+Automatic+Metric+for+MT+Evaluation+with+Improved+Correlation+with+Human+Judgments%2E+In+Proc%2E+of+ACL+WIEEMMTS+2005%2E) [our card](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/) 2. Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proc. of ACL2005. [find](https://scholar.google.com/scholar?q=Colin+Bannard+and+Chris+Callison%2DBurch%2E+2005%2E+Paraphrasing+with+bilingual+parallel+corpora%2E+In+Proc%2E+of+ACL2005%2E) [our card](https://md.research.elamur.ai/free/en/papers/bannard-2005-paraphrasing-bilingual-parallel-corpora/) 3. Chris Callison-Burch, Philipp Koehn, Christof Monz, Kay Peterson, Mark Przybocki, and Omar Zaidan. 2010. Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation. In Proc. of ACL WMT/MetricsMATR 2010. [find](https://scholar.google.com/scholar?q=Chris+Callison%2DBurch%2C+Philipp+Koehn%2C+Christof+Monz%2C+Kay+Peterson%2C+Mark+Przybocki%2C+and+Omar+Zaidan%2E+2010%2E+Findings+of+the+2010+Joint+Workshop+on+Statistical+Machine+Translation+and+Metrics+for+Machine+) [our card](https://md.research.elamur.ai/free/en/papers/burch-2010-findings-2010-joint-workshop-statistical/) 4. Michael Denkowski and Alon Lavie. 2010a. Choosing the Right Evaluation for Machine Translation: an Examination of Annotator and Automatic Metric Performance on Human Judgment Tasks. In Proc. of AMTA 2010. [find](https://scholar.google.com/scholar?q=Michael+Denkowski+and+Alon+Lavie%2E+2010a%2E+Choosing+the+Right+Evaluation+for+Machine+Translation%3A+an+Examination+of+Annotator+and+Automatic+Metric+Performance+on+Human+Judgment+Tasks%2E+In+Proc%2E+of+AMTA+2) 5. Michael Denkowski and Alon Lavie. 2010b. METEOR-NEXT and the METEOR Paraphrase Tables: Improve Evaluation Support for Five Target Languages. In Proc. of ACL WMT/MetricsMATR 2010. δ 0.75 0.70 0.55 0.50 w exact 1.00 1.00 1.00 1.00 w stem 0.60 1.00 0.20 0.50 w syn 0.80 0.60 0.60 0.50 w par 0.60 0.80 0.80 0.50 [find](https://scholar.google.com/scholar?q=Michael+Denkowski+and+Alon+Lavie%2E+2010b%2E+METEOR%2DNEXT+and+the+METEOR+Paraphrase+Tables%3A+Improve+Evaluation+Support+for+Five+Target+Languages%2E+In+Proc%2E+of+ACL+WMT%2FMetricsMATR+2010%2E+%CE%B4+0%2E75+0%2E70+0%2E55+0%2E50) 6. Qin Gao and Stephan Vogel. 2008. Parallel Implementations of Word Alignment Tool. In Proc. of ACL WSETQANLP 2008. [find](https://scholar.google.com/scholar?q=Qin+Gao+and+Stephan+Vogel%2E+2008%2E+Parallel+Implementations+of+Word+Alignment+Tool%2E+In+Proc%2E+of+ACL+WSETQANLP+2008%2E) 7. Hieu Hoang, Alexandra Birch, Chris Callison-burch, Richard Zens, Rwth Aachen, Alexandra Constantin, Marcello Federico, Nicola Bertoldi, Chris Dyer, Brooke Cowan, Wade Shen, Christine Moran, and Ondej Bojar. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Proc. of ACL 2007. [find](https://scholar.google.com/scholar?q=Hieu+Hoang%2C+Alexandra+Birch%2C+Chris+Callison%2Dburch%2C+Richard+Zens%2C+Rwth+Aachen%2C+Alexandra+Constantin%2C+Marcello+Federico%2C+Nicola+Bertoldi%2C+Chris+Dyer%2C+Brooke+Cowan%2C+Wade+Shen%2C+Christine+Moran%2C+and+Ondej+) 8. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical Phrase-Based Translation. In Proc. of NAACL/HLT 2003. [↗](https://doi.org/10.21236/ada461156) 9. Philipp Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In Proc. of MT Summit 2005. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2E+2005%2E+Europarl%3A+A+Parallel+Corpus+for+Statistical+Machine+Translation%2E+In+Proc%2E+of+MT+Summit+2005%2E) 10. Zhifei Li, Chris Callison-Burch, Chris Dyer, Juri Ganitkevitch, Sanjeev Khudanpur, Lane Schwartz, Wren Thornton, Jonathan Weese, and Omar Zaidan. 2009. Joshua: An Open Source Toolkit for Parsing-based Machine Translation. In Proc. of WMT 2009. [find](https://scholar.google.com/scholar?q=Zhifei+Li%2C+Chris+Callison%2DBurch%2C+Chris+Dyer%2C+Juri+Ganitkevitch%2C+Sanjeev+Khudanpur%2C+Lane+Schwartz%2C+Wren+Thornton%2C+Jonathan+Weese%2C+and+Omar+Zaidan%2E+2009%2E+Joshua%3A+An+Open+Source+Toolkit+for+Parsing%2Dbased) [our card](https://md.research.elamur.ai/free/en/papers/li-2009-joshua-open-source-toolkit-parsing/) 11. George Miller and Christiane Fellbaum. 2007. WordNet. [http://wordnet.princeton.edu/](http://wordnet.princeton.edu/). [↗](http://wordnet.princeton.edu/) 12. Joseph Olive. 2005. Global Autonomous Language Exploitation (GALE). DARPA/IPTO Proposer Information Pamphlet. [find](https://scholar.google.com/scholar?q=Joseph+Olive%2E+2005%2E+Global+Autonomous+Language+Exploitation+%28GALE%29%2E+DARPA%2FIPTO+Proposer+Information+Pamphlet%2E) 13. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proc. of ACL 2002. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+BLEU%3A+a+Method+for+Automatic+Evaluation+of+Machine+Translation%2E+In+Proc%2E+of+ACL+2002%2E) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/) 14. Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2009. English Gigaword Fourth Edition. Linguistic Data Consortium, LDC2009T13. [↗](https://doi.org/10.35111/y4px-6y07) 15. Martin Porter. 2001. Snowball: A language for stemming algorithms. [http://snowball.tartarus.org/texts/](http://snowball.tartarus.org/texts/). [↗](http://snowball.tartarus.org/texts/) 16. NIST Open Machine Translation 2009 Evaluation. [http://www.itl.nist.gov/iad/mig/tests/mt/2009/](http://www.itl.nist.gov/iad/mig/tests/mt/2009/). [↗](http://www.itl.nist.gov/iad/mig/tests/mt/2009/) 17. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proc. of AMTA 2006. [find](https://scholar.google.com/scholar?q=Matthew+Snover%2C+Bonnie+Dorr%2C+Richard+Schwartz%2C+Linnea+Micciulla%2C+and+John+Makhoul%2E+2006%2E+A+Study+of+Translation+Edit+Rate+with+Targeted+Human+Annotation%2E+In+Proc%2E+of+AMTA+2006%2E) [our card](https://md.research.elamur.ai/free/en/papers/snover-2006-study-translation-edit-rate-targeted/) 18. Andreas Stolke. 2002. SRILM - an Extensible Language Modeling Toolkit. In Proc. of ICSLP 2002. [find](https://scholar.google.com/scholar?q=Andreas+Stolke%2E+2002%2E+SRILM+%2D+an+Extensible+Language+Modeling+Toolkit%2E+In+Proc%2E+of+ICSLP+2002%2E) 19. C. van Rijsbergen, 1979. Information Retrieval, chapter 7. 2nd edition. [find](https://scholar.google.com/scholar?q=C%2E+van+Rijsbergen%2C+1979%2E+Information+Retrieval%2C+chapter+7%2E+2nd+edition%2E) 20. Omar F. Zaidan. 2009. Z-MERT: A Fully Configurable Open Source Tool for Minimum Error Rate Training of Machine Translation Systems. The Prague Bulletin of Mathematical Linguistics. [↗](https://doi.org/10.2478/v10108-009-0018-2)