---
title: "Results of the WMT17 Metrics Shared Task — Bojar 2017"
description: "The WMT17 Metrics Shared Task measures how well automatic MT evaluation metrics, which compare system output against reference translations, can replace…"
authors: ["Ondřej Bojar", "Yvette Graham", "Amir Kamran"]
year: 2017
venue: "Proceedings of the Second Conference on Machine Translation"
source: "https://aclanthology.org/W17-4755/"
doi: "10.18653/v1/w17-4755"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bojar-2017-results-wmt17-metrics-shared-task/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bojar-2017-results-wmt17-metrics-shared-task/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/) / Results of the WMT17 Metrics Shared Task

# Results of the WMT17 Metrics Shared Task

**Authors:** [Ondřej Bojar](https://md.research.elamur.ai/free/en/authors/ondrej-bojar/), [Yvette Graham](https://md.research.elamur.ai/free/en/authors/yvette-graham/), [Amir Kamran](https://md.research.elamur.ai/free/en/authors/amir-kamran/) · **Year:** 2017 · **Venue:** Proceedings of the Second Conference on Machine Translation · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W17-4755/) · [PDF](https://aclanthology.org/W17-4755.pdf) · [DOI 10.18653/v1/w17-4755](https://doi.org/10.18653/v1/w17-4755) · [OpenAlex](https://openalex.org/W2915756181)

**Citations:** 118 citations (openalex-snapshot · 2026-10-14) · [92](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The WMT17 Metrics Shared Task measures how well automatic MT evaluation metrics, which compare system output against reference translations, can replace human quality judgements. The organizers distributed outputs of systems from the WMT17 News Translation Task, the Neural MT training task, and the health-domain HimL project, covering 16 language pairs. Participants returned two kinds of scores: one for a whole test set (system level) and one for each sentence (segment level). These scores were correlated with two human gold standards. The first was Direct Assessment (DA), which now replaces relative ranking as the official WMT method. The second was the semantics-oriented HUME score. Pearson correlation was the main statistic, and Kendall's τ was used where sentence-level DA data were too sparse. The strongest metrics exceeded 0.95 average Pearson correlation at system level, while sentence-level agreement typically fell between 0.4 and 0.6. Metrics that use character-level information again performed well, and both trained and untrained approaches reached the top. The findings help researchers choose reliable, publicly available evaluation tools.

## Key points
- Seven baselines (NIST, BLEU, SentBLEU, CDER, TER, PER and WER) served as comparison points for the 14 metrics submitted by 8 research groups.
- The data comprised newstest2017 and himltest2017:
  - newstest2017 has about 3,000 sentences per direction (2,001 for Chinese and Latvian).
  - himltest2017 has about 300 sentences each for English→Czech/German/Polish/Romanian.
  - Together they yielded 166 real system outputs, plus 10K hybrid systems per language pair.
- Top performers included chrF, CharacTer, BEER, BLEND and the new MEANT 2.0. MEANT 2.0 also led on the small HUME segment-level evaluation.
- On self-reported speed, most good metrics were fast. Only MEANT 2.0 was notably slower, needing up to about one second per sentence.
- A data leak affected English→Polish HUME: 324 of 340 sentences in one system's file had appeared in the training data. Among the trained metrics, only AutoDA was affected.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/)

## Cited by

Cited by

92 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 75 not yet.

- [Assessing the Role of Context in Chat Translation Evaluation: Is Context Helpful and Under What Conditions?](https://md.research.elamur.ai/free/en/papers/agrawal-2024-assessing-role-context-chat-translation/)
- [Findings of the 2017 Conference on Machine Translation (WMT17)](https://md.research.elamur.ai/free/en/papers/bojar-2017-findings-2017-conference-machine-translation/)
- [Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments](https://md.research.elamur.ai/free/en/papers/fomicheva-2019-taking-mt-evaluation-metrics-extremes/)
- [Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain](https://md.research.elamur.ai/free/en/papers/freitag-2021-results-wmt21-metrics-shared-task/)
- [YiSi - a Unified Semantic MT Quality Evaluation and Estimation Metric for Languages with Different Levels of Available Resources](https://md.research.elamur.ai/free/en/papers/lo-2019-yisi-unified-semantic-mt-quality/)
- [Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performance](https://md.research.elamur.ai/free/en/papers/ma-2018-results-wmt18-metrics-shared-task/)
- [LENS: A Learnable Evaluation Metric for Text Simplification](https://md.research.elamur.ai/free/en/papers/maddela-2023-lens-learnable-evaluation-metric-text/)
- [The (Un)Suitability of Automatic Evaluation Metrics for Text Simplification](https://md.research.elamur.ai/free/en/papers/manchego-2021-un-suitability-automatic-evaluation-metrics/)
- [Results of the WMT20 Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/mathur-2020-results-wmt20-metrics-shared-task/)
- [KinyCOMET: Automatic Evaluation of Machine Translation Systems for Kinyarwanda–English](https://md.research.elamur.ai/free/en/papers/mazimpaka-2026-kinycomet-automatic-evaluation-machine-translation/)
- [Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics](https://md.research.elamur.ai/free/en/papers/perrella-2024-beyond-correlation-interpretable-evaluation-machine/)
- [COMET: A Neural Framework for MT Evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/)
- [Are References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/rei-2021-are-references-really-needed-unbabel/)
- [COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/rei-2022-comet-22-unbabel-ist-2022/)
- [Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains](https://md.research.elamur.ai/free/en/papers/schmidt-2026-who-watches-watchmen-humans-disagree/)
- [RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/shimanaka-2018-ruse-regressor-sentence-embeddings-automatic/)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://md.research.elamur.ai/free/en/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{bojar2017results,
  title = {{Results of the WMT17 Metrics Shared Task}},
  author = {Bojar, Ondřej and Graham, Yvette and Kamran, Amir},
  booktitle = {Proceedings of the Second Conference on Machine Translation},
  year = {2017},
  doi = {10.18653/v1/w17-4755},
  url = {https://doi.org/10.18653/v1/w17-4755}
}
```

- [← previous Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging](https://md.research.elamur.ai/free/en/papers/reimers-2017-reporting-score-distributions-makes-difference/)
- [next → Revisiting Selectional Preferences for Coreference Resolution](https://md.research.elamur.ai/free/en/papers/heinzerling-2017-revisiting-selectional-preferences-coreference-resolution/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/)

## References

33 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Omri Abend and Ari Rappoport. 2013. Universal Conceptual Cognitive Annotation (UCCA). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 228–238, Sofia, Bulgaria, August. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Omri+Abend+and+Ari+Rappoport%2E+2013%2E+Universal+Conceptual+Cognitive+Annotation+%28UCCA%29%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29) [our card](https://md.research.elamur.ai/free/en/papers/abend-2013-universal-conceptual-cognitive-annotation-ucca/)
2. Melania Duma and Wolfgang Menzel. 2017. Uhh submission to the wmt17 metrics shared task. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Tasks Papers, Copenhagen, Denmark, September. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Melania+Duma+and+Wolfgang+Menzel%2E+2017%2E+Uhh+submission+to+the+wmt17+metrics+shared+task%2E+In+Proceedings+of+the+Second+Conference+on+Machine+Translation%2C+Volume+2%3A+Shared+Tasks+Papers%2C+Copenhagen%2C+Denm)
3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473. [↗](https://arxiv.org/abs/1409.0473) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
4. Yvette Graham and Timothy Baldwin. 2014. Testing for Significance of Increased Correlation with Human Judgment. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 172–176, Doha, Qatar, October. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yvette+Graham+and+Timothy+Baldwin%2E+2014%2E+Testing+for+Significance+of+Increased+Correlation+with+Human+Judgment%2E+In+Proceedings+of+the+2014+Conference+on+Empirical+Methods+in+Natural+Language+Processin)
5. Alexandra Birch, Omri Abend, Ondřej Bojar, and Barry Haddow. 2016. HUME: Human UCCA-Based Evaluation of Machine Translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1264– 1274, Austin, Texas, November. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1607.00030) [our card](https://md.research.elamur.ai/free/en/papers/birch-2016-hume-human-ucca-based-evaluation/)
6. Yvette Graham and Qun Liu. 2016. Achieving Accurate Conclusions in Evaluation of Automatic Machine Translation Metrics. In Proceedings of the 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yvette+Graham+and+Qun+Liu%2E+2016%2E+Achieving+Accurate+Conclusions+in+Evaluation+of+Automatic+Machine+Translation+Metrics%2E+In+Proceedings+of+the+15th+Annual+Conference+of+the+North+American+Chapter+of+th)
7. Ondřej Bojar, Miloš Ercegovčević, Martin Popel, and Omar Zaidan. 2011. A Grain of Salt for the WMT Manual Evaluation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 1–11, Edinburgh, Scotland, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Milos%CC%8C+Ercegovc%CC%8Cevic%CC%81%2C+Martin+Popel%2C+and+Omar+Zaidan%2E+2011%2E+A+Grain+of+Salt+for+the+WMT+Manual+Evaluation%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2C+pages)
8. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous Measurement Scales in Human Evaluation of Machine Translation. In Proceedings of the 7th Linguistic Annotation Workshop & Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Timothy+Baldwin%2C+Alistair+Moffat%2C+and+Justin+Zobel%2E+2013%2E+Continuous+Measurement+Scales+in+Human+Evaluation+of+Machine+Translation%2E+In+Proceedings+of+the+7th+Linguistic+Annotation+Works)
9. Ondřej Bojar, Christian Federmann, Barry Haddow, Philipp Koehn, Matt Post, and Lucia Specia. 2016a. Ten Years of WMT Evaluation Campaigns: Lessons Learnt. In Proceedings of the LREC 2016 Workshop Translation Evaluation From Fragmented Tools and Data Sets to an Integrated Ecosystem, pages 27–34, Portorose, Slovenia, 5. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Christian+Federmann%2C+Barry+Haddow%2C+Philipp+Koehn%2C+Matt+Post%2C+and+Lucia+Specia%2E+2016a%2E+Ten+Years+of+WMT+Evaluation+Campaigns%3A+Lessons+Learnt%2E+In+Proceedings+of+the+LREC+2016+Workshop+Tra)
10. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2014. Is Machine Translation Getting Better over Time? In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 443–451, Ondřej Bojar, Yvette Graham, , and Amir Kamran Miloš Stanojević. 2016b. Results of the WMT16 Metrics Shared Task . In Proceedings of the First Conference on Machine Translation, Gothenburg, Sweden, April. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Timothy+Baldwin%2C+Alistair+Moffat%2C+and+Justin+Zobel%2E+2014%2E+Is+Machine+Translation+Getting+Better+over+Time%3F+In+Proceedings+of+the+14th+Conference+of+the+European+Chapter+of+the+Associati) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2016-results-wmt16-metrics-shared-task/)
11. Matouš Macháček and Ondřej Bojar. 2013. Results of the WMT13 Metrics Shared Task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 45–51, Sofia, Bulgaria, August. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Matous%CC%8C+Macha%CC%81c%CC%8Cek+and+Ondr%CC%8Cej+Bojar%2E+2013%2E+Results+of+the+WMT13+Metrics+Shared+Task%2E+In+Proceedings+of+the+Eighth+Workshop+on+Statistical+Machine+Translation%2C+pages+45%E2%80%9351%2C+Sofia%2C+Bulgaria%2C+August%2E+As)
12. Yvette Graham, Nitika Mathur, and Timothy Baldwin. 2015. Accurate Evaluation of Segment-level Machine Translation Metrics. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics Human Language Technologies, Denver, Colorado. [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Nitika+Mathur%2C+and+Timothy+Baldwin%2E+2015%2E+Accurate+Evaluation+of+Segment%2Dlevel+Machine+Translation+Metrics%2E+In+Proceedings+of+the+2015+Conference+of+the+North+American+Chapter+of+the+As) [our card](https://md.research.elamur.ai/free/en/papers/graham-2015-accurate-evaluation-segment-level-machine/)
13. David Mareček, Ondřej Bojar, Ondřej Hübsch, Rudolf Rosa, and Dusan Varis. 2017. Cuni experiments for wmt17 metrics task. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Tasks Papers, Copenhagen, Denmark, September. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+Marec%CC%8Cek%2C+Ondr%CC%8Cej+Bojar%2C+Ondr%CC%8Cej+Hu%CC%88bsch%2C+Rudolf+Rosa%2C+and+Dusan+Varis%2E+2017%2E+Cuni+experiments+for+wmt17+metrics+task%2E+In+Proceedings+of+the+Second+Conference+on+Machine+Translation%2C+Volume+2%3A+S)
14. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2016. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, FirstView:1–28, 1. [↗](https://doi.org/10.1017/s1351324915000339)
15. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Haji, Christopher Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016), pages 1659–1666, Paris, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Marie%2DCatherine+de+Marneffe%2C+Filip+Ginter%2C+Yoav+Goldberg%2C+Jan+Haji%2C+Christopher+Manning%2C+Ryan+McDonald%2C+Slav+Petrov%2C+Sampo+Pyysalo%2C+Natalia+Silveira%2C+Reut+Tsarfaty%2C+and+Daniel+Zeman%2E+201) [our card](https://md.research.elamur.ai/free/en/papers/ud-2016-nivre-universal-dependencies-v1-multilingual-treebank/)
16. Jan Hajič. 2004. Complex Corpus Annotation: The Prague Dependency Treebank. In Insight into Slovak and Czech Corpus Linguistics, Bratislava, Slovakia. Jazykovedný ústav Ľ. Štúra, SAV. [find](https://scholar.google.com/scholar?q=Jan+Hajic%CC%8C%2E+2004%2E+Complex+Corpus+Annotation%3A+The+Prague+Dependency+Treebank%2E+In+Insight+into+Slovak+and+Czech+Corpus+Linguistics%2C+Bratislava%2C+Slovakia%2E+Jazykovedny%CC%81+u%CC%81stav+L%CC%8C%2E+S%CC%8Ctu%CC%81ra%2C+SAV%2E)
17. Jindřich Helcl and Jindřich Libovický. 2017. Neural Monkey: An open-source tool for sequence learning. The Prague Bulletin of Mathematical Linguistics, 107:5–17. [↗](https://doi.org/10.1515/pralin-2017-0001)
18. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, pages 311–318. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+BLEU%3A+A+Method+for+Automatic+Evaluation+of+Machine+Translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+on+Association+for+Computat) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
19. Philipp Koehn and Christof Monz. 2006. Manual and Automatic Evaluation of Machine Translation Between European Languages. In Proceedings of the Workshop on Statistical Machine Translation, StatMT ’06, pages 102–121, Stroudsburg, PA, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Philipp+Koehn+and+Christof+Monz%2E+2006%2E+Manual+and+Automatic+Evaluation+of+Machine+Translation+Between+European+Languages%2E+In+Proceedings+of+the+Workshop+on+Statistical+Machine+Translation%2C+StatMT+%E2%80%9906%2C) [our card](https://md.research.elamur.ai/free/en/papers/koehn-2006-manual-automatic-evaluation-machine-translation/)
20. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portugal, September. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maja+Popovic%CC%81%2E+2015%2E+chrF%3A+character+n%2Dgram+F%2Dscore+for+automatic+MT+evaluation%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+Lisboa%2C+Portugal%2C+September%2E+Association+for+C) [our card](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/)
21. Gregor Leusch, Nicola Ueffing, and Hermann Ney. 2006. CDER: Efficient MT Evaluation Using Block Movements. In In Proceedings of EACL, pages 241– 248. [find](https://scholar.google.com/scholar?q=Gregor+Leusch%2C+Nicola+Ueffing%2C+and+Hermann+Ney%2E+2006%2E+CDER%3A+Efficient+MT+Evaluation+Using+Block+Movements%2E+In+In+Proceedings+of+EACL%2C+pages+241%E2%80%93+248%2E) [our card](https://md.research.elamur.ai/free/en/papers/leusch-2006-cder-efficient-mt-evaluation-block/)
22. Maja Popović. 2017. chrf++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Tasks Papers, Copenhagen, Denmark, September. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maja+Popovic%CC%81%2E+2017%2E+chrf%2B%2B%3A+words+helping+character+n%2Dgrams%2E+In+Proceedings+of+the+Second+Conference+on+Machine+Translation%2C+Volume+2%3A+Shared+Tasks+Papers%2C+Copenhagen%2C+Denmark%2C+September%2E+Association) [our card](https://md.research.elamur.ai/free/en/papers/popovic-2017-chrf-words-helping-character-n/)
23. Chi-kiu Lo, Philipp Dowling, and Dekai Wu. 2015. Improving evaluation and optimization of MT systems against MEANT. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portugal, September. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chi%2Dkiu+Lo%2C+Philipp+Dowling%2C+and+Dekai+Wu%2E+2015%2E+Improving+evaluation+and+optimization+of+MT+systems+against+MEANT%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+Lisboa%2C+Por)
24. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In In Proceedings of Association for Machine Translation in the Americas, pages 223–231. [find](https://scholar.google.com/scholar?q=Matthew+Snover%2C+Bonnie+Dorr%2C+Richard+Schwartz%2C+Linnea+Micciulla%2C+and+John+Makhoul%2E+2006%2E+A+study+of+translation+edit+rate+with+targeted+human+annotation%2E+In+In+Proceedings+of+Association+for+Machine+T) [our card](https://md.research.elamur.ai/free/en/papers/snover-2006-study-translation-edit-rate-targeted/)
25. Chi-kiu Lo. 2017. Meant 2.0: Accurate semantic mt evaluation for any output language. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Tasks Papers, Copenhagen, Denmark, September. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chi%2Dkiu+Lo%2E+2017%2E+Meant+2%2E0%3A+Accurate+semantic+mt+evaluation+for+any+output+language%2E+In+Proceedings+of+the+Second+Conference+on+Machine+Translation%2C+Volume+2%3A+Shared+Tasks+Papers%2C+Copenhagen%2C+Denmark) [our card](https://md.research.elamur.ai/free/en/papers/lo-2017-meant-2-0-accurate-semantic/)
26. Miloš Stanojević and Khalil Sima’an. 2015. BEER 1.1: ILLC UvA submission to metrics and tuning task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portugal, September. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Milos%CC%8C+Stanojevic%CC%81+and+Khalil+Sima%E2%80%99an%2E+2015%2E+BEER+1%2E1%3A+ILLC+UvA+submission+to+metrics+and+tuning+task%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+Lisboa%2C+Portugal%2C+Septem)
27. Qingsong Ma, Yvette Graham, Shugen Wang, and Qun Liu. 2017. Blend: a novel combined mt metric based on direct assessment casict-dcu submission to wmt17 metrics task. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Tasks Papers, Copenhagen, Denmark, September. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qingsong+Ma%2C+Yvette+Graham%2C+Shugen+Wang%2C+and+Qun+Liu%2E+2017%2E+Blend%3A+a+novel+combined+mt+metric+based+on+direct+assessment+casict%2Ddcu+submission+to+wmt17+metrics+task%2E+In+Proceedings+of+the+Second+Confe)
28. Andre Tättar and Mark Fishel. 2017. bleu2vec: the painfully familiar metric on continuous vector space steroids. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Tasks Papers, Copenhagen, Denmark, September. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Andre+Ta%CC%88ttar+and+Mark+Fishel%2E+2017%2E+bleu2vec%3A+the+painfully+familiar+metric+on+continuous+vector+space+steroids%2E+In+Proceedings+of+the+Second+Conference+on+Machine+Translation%2C+Volume+2%3A+Shared+Tasks)
29. Matouš Macháček and Ondřej Bojar. 2014. Results of the WMT14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 293–301, Baltimore, MD, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Matous%CC%8C+Macha%CC%81c%CC%8Cek+and+Ondr%CC%8Cej+Bojar%2E+2014%2E+Results+of+the+WMT14+metrics+shared+task%2E+In+Proceedings+of+the+Ninth+Workshop+on+Statistical+Machine+Translation%2C+pages+293%E2%80%93301%2C+Baltimore%2C+MD%2C+USA%2E+Associ) [our card](https://md.research.elamur.ai/free/en/papers/machacek-2014-results-wmt14-metrics-shared-task/)
30. Yolanda Vazquez-Alvarez and Mark Huckvale. 2002. The reliability of the ITU-t p.85 standard for the evaluation of text-to-speech systems. In Proc. of IC-SLP - INTERSPEECH. [find](https://scholar.google.com/scholar?q=Yolanda+Vazquez%2DAlvarez+and+Mark+Huckvale%2E+2002%2E+The+reliability+of+the+ITU%2Dt+p%2E85+standard+for+the+evaluation+of+text%2Dto%2Dspeech+systems%2E+In+Proc%2E+of+IC%2DSLP+%2D+INTERSPEECH%2E)
31. Weiyue Wang, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. 2016. CharacTer: Translation Edit Rate on Character Level. In Proceedings of the First Conference on Machine Translation, Berlin, Germany, August. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Weiyue+Wang%2C+Jan%2DThorsten+Peter%2C+Hendrik+Rosendahl%2C+and+Hermann+Ney%2E+2016%2E+CharacTer%3A+Translation+Edit+Rate+on+Character+Level%2E+In+Proceedings+of+the+First+Conference+on+Machine+Translation%2C+Berlin%2C+G) [our card](https://md.research.elamur.ai/free/en/papers/wang-2016-character-translation-edit-rate-character/)
32. Evan James Williams. 1959. Regression analysis, volume 14. Wiley New York. [find](https://scholar.google.com/scholar?q=Evan+James+Williams%2E+1959%2E+Regression+analysis%2C+volume+14%2E+Wiley+New+York%2E)
33. Hui Yu, Qingsong Ma, Xiaofeng Wu, and Qun Liu. 2015. CASICT-DCU Participation in WMT2015 Metrics Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portugal, September. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hui+Yu%2C+Qingsong+Ma%2C+Xiaofeng+Wu%2C+and+Qun+Liu%2E+2015%2E+CASICT%2DDCU+Participation+in+WMT2015+Metrics+Task%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+Lisboa%2C+Portugal%2C+Septem)