--- title: "Findings of the 2010 Joint Workshop on — Callison-Burch 2010" description: "This report covers the WMT10 shared tasks, run jointly with NIST's MetricsMATR." authors: ["Chris Callison-Burch", "Philipp Koehn", "Christof Monz", "Kay Peterson", "Mark Przybocki", "Omar Zaidan"] year: 2010 venue: "Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR" source: "https://aclanthology.org/W10-1703/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/burch-2010-findings-2010-joint-workshop-statistical/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/burch-2010-findings-2010-joint-workshop-statistical/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2010](https://md.research.elamur.ai/free/en/papers/2010/) / Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation # Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation **Authors:** [Chris Callison-Burch](https://md.research.elamur.ai/free/en/authors/chris-callison-burch/), [Philipp Koehn](https://md.research.elamur.ai/free/en/authors/philipp-koehn/), [Christof Monz](https://md.research.elamur.ai/free/en/authors/christof-monz/), [Kay Peterson](https://md.research.elamur.ai/free/en/authors/kay-peterson/), [Mark Przybocki](https://md.research.elamur.ai/free/en/authors/mark-przybocki/), [Omar Zaidan](https://md.research.elamur.ai/free/en/authors/omar-zaidan/) · **Year:** 2010 · **Venue:** Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/W10-1703/) · [PDF](https://aclanthology.org/W10-1703.pdf) **Citations:** [117](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary This report covers the WMT10 shared tasks, run jointly with NIST's MetricsMATR. It asks how well current machine translation systems perform, whether combining their outputs helps, and how closely automatic metrics track human opinion. There were three tasks: translation between English and German, Spanish, French and Czech; system combination; and metric evaluation. The organizers supplied training corpora, a fresh news test set and two open-source baseline toolkits. They then ran a large manual evaluation, which they treat as the primary measure of quality, covering 104 MT systems and 41 combination entries. The resulting human rankings served as the reference for scoring 26 automatic metrics. For the first time, the organizers also recruited non-expert judges through Amazon's Mechanical Turk. Individual crowd workers agreed less with one another than experts did, but the system rankings they produced correlated strongly with expert rankings, and filtering out unreliable workers improved the match further. All translations, judgments and scores were released publicly. The work matters for benchmarking, cheaper human evaluation and metric validation. ## Key points - Participation grew by about 50% over the previous year, to 33 groups from 29 institutions. - The organizers received 153 primary and 28 secondary submissions. - Two commercial systems and two online systems were also evaluated, using their web interfaces. - The test data came from 119 news articles collected in mid-December 2009 from Czech, English, French, German and Spanish outlets. - The agency CEET translated them directly, without going through a pivot language. - Training resources included Europarl, News Commentary, United Nations, the 10⁹ Word Parallel Corpus and CzEng, plus large monolingual news data for language modeling. - More than 120 annotators contributed 337 hours of manual evaluation, producing: - 37,884 ranking labels, - 7,255 edited translations, - 6,905 yes/no judgments. - Nine teams entered system combination, supported by a 25-article dev set and 20 n-best lists. - Google output was left out of the combination inputs, and combined systems beat their components more often than in 2009. ## Interesting ideas - **Calibrate crowd workers against experts before merging their data.** The organizers cut the worst 300 MTurk workers, at which point crowd–expert kappa (0.449) roughly matched expert–expert kappa (0.439). Using expert agreement as the target gives a principled stopping point for any crowdsourced annotation project. - **Judge workers by an internal signal instead of expert overlap.** Workers were scored by how often they preferred the human reference translation over system output (RPR). Methods that filtered or weighted workers by this score performed well with no expert data at all. A comparable built-in sanity check could be designed for other labeling tasks. - **Remove a dominant system from ensemble inputs.** Excluding an unusually strong component (Google) made it possible to see whether combination methods really add value. This is a useful control when benchmarking ensembles in any field. - **Give combination participants a dedicated dev set and n-best lists.** Releasing these alongside the primary outputs lets downstream teams tune their methods properly. This is a cheap design choice for any multi-stage shared task. - **Treat human judgment as ground truth for metrics.** Collecting human system rankings first and then measuring how well each automatic metric correlates with them is a reusable way to validate metrics for new languages or tasks. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) ## Cited by Cited by 117 papers of our library cite it (of the 140673 whose bibliographies we read); 21 of them have a page here, 96 not yet. - [Findings of the 2021 Conference on Machine Translation (WMT21)](https://md.research.elamur.ai/free/en/papers/akhbardeh-2021-findings-2021-conference-machine-translation/) - [PET: a Tool for Post-editing and Assessing Machine Translation](https://md.research.elamur.ai/free/en/papers/aziz-2012-pet-tool-post-editing-assessing/) - [Findings of the 2019 Conference on Machine Translation (WMT19)](https://md.research.elamur.ai/free/en/papers/barrault-2019-findings-2019-conference-machine-translation/) - [Findings of the 2020 Conference on Machine Translation (WMT20)](https://md.research.elamur.ai/free/en/papers/barrault-2020-findings-2020-conference-machine-translation/) - [Findings of the 2013 Workshop on Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/bojar-2013-findings-2013-workshop-statistical-machine/) - [Findings of the 2014 Workshop on Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/bojar-2014-findings-2014-workshop-statistical-machine/) - [Findings of the 2015 Workshop on Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/bojar-2015-findings-2015-workshop-statistical-machine/) - [Findings of the 2016 Conference on Machine Translation](https://md.research.elamur.ai/free/en/papers/bojar-2016-findings-2016-conference-machine-translation/) - [Findings of the 2017 Conference on Machine Translation (WMT17)](https://md.research.elamur.ai/free/en/papers/bojar-2017-findings-2017-conference-machine-translation/) - [Findings of the 2018 Conference on Machine Translation (WMT18)](https://md.research.elamur.ai/free/en/papers/bojar-2018-findings-2018-conference-machine-translation/) - [Findings of the 2011 Workshop on Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/burch-2011-findings-2011-workshop-statistical-machine/) - [Findings of the 2012 Workshop on Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/burch-2012-findings-2012-workshop-statistical-machine/) - [Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems](https://md.research.elamur.ai/free/en/papers/denkowski-2011-meteor-1-3-automatic-metric/) - [Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments](https://md.research.elamur.ai/free/en/papers/fomicheva-2019-taking-mt-evaluation-metrics-extremes/) - [Statistical Power and Translationese in Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/graham-2020-statistical-power-translationese-machine-translation/) - [Learning Whom to Trust with MACE](https://md.research.elamur.ai/free/en/papers/hovy-2013-learning-whom-trust-mace/) - [Discourse Structure in Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/joty-2017-discourse-structure-machine-translation-evaluation/) - [Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet](https://md.research.elamur.ai/free/en/papers/kocmi-2024-findings-wmt24-general-machine-translation/) - [Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets](https://md.research.elamur.ai/free/en/papers/kocmi-2025-findings-wmt25-general-machine-translation/) - [MEANT: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility based on semantic roles](https://md.research.elamur.ai/free/en/papers/lo-2011-meant-inexpensive-high-accuracy-semi/) - [From Human to Automatic Error Classification for Machine Translation Output](https://md.research.elamur.ai/free/en/papers/popovic-2011-human-automatic-error-classification-machine/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{callisonburch2010findings, title = {{Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation}}, author = {Callison-Burch, Chris and Koehn, Philipp and Monz, Christof and Peterson, Kay and Przybocki, Mark and Zaidan, Omar}, booktitle = {Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR}, year = {2010}, url = {https://aclanthology.org/W10-1703/} } ``` - [← previous Extending BART to Provide a Coreference Resolution System for German](https://md.research.elamur.ai/free/en/papers/broscheit-2010-extending-bart-provide-coreference-resolution/) - [next → Fine-Grain Morphological Analyzer and Part-of-Speech Tagger for Arabic Text](https://md.research.elamur.ai/free/en/papers/lingbulk-2010-sawalha-fine-grain-morphological-analyzer-part/) - [↑ back to papers of 2010](https://md.research.elamur.ai/free/en/papers/2010/) ## References 2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Alexandre Allauzen, Josep M. Crego, lknur Durgar El-Kahlout, and Francois Yvon. 2010. Limsi’s statistical translation systems for wmt’10. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 29–34, Uppsala, Sweden, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alexandre+Allauzen%2C+Josep+M%2E+Crego%2C+lknur+Durgar+El%2DKahlout%2C+and+Francois+Yvon%2E+2010%2E+Limsi%E2%80%99s+statistical+translation+systems+for+wmt%E2%80%9910%2E+In+Proceedings+of+the+Joint+Fifth+Workshop+on+Statistical+Mach) 2. Loı̈c Barrault. 2010. Many: Open source mt system combination at wmt’10. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation 16 [http://www.statmt.org/wmt09/results](http://www.statmt.org/wmt09/results). html K exp -filtered Label count 2,583 2,488 1,578 627 12,570 12,870 9,197 13,169 [↗](http://www.statmt.org/wmt09/results)