---
title: "Findings of the 2011 Workshop on — Callison-Burch 2011"
description: "This report documents the outcomes of the WMT11 shared tasks: translation between English and Czech, French, German and Spanish, combining outputs of…"
authors: ["Chris Callison-Burch", "Philipp Koehn", "Christof Monz", "Omar Zaidan"]
year: 2011
venue: "Proceedings of the Sixth Workshop on Statistical Machine Translation"
source: "https://aclanthology.org/W11-2103/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/burch-2011-findings-2011-workshop-statistical-machine/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/burch-2011-findings-2011-workshop-statistical-machine/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2011](https://md.research.elamur.ai/free/papers/2011/) / Findings of the 2011 Workshop on Statistical Machine Translation

# Findings of the 2011 Workshop on Statistical Machine Translation

**Authors:** [Chris Callison-Burch](https://md.research.elamur.ai/free/authors/chris-callison-burch/), [Philipp Koehn](https://md.research.elamur.ai/free/authors/philipp-koehn/), [Christof Monz](https://md.research.elamur.ai/free/authors/christof-monz/), [Omar Zaidan](https://md.research.elamur.ai/free/authors/omar-zaidan/) · **Year:** 2011 · **Venue:** Proceedings of the Sixth Workshop on Statistical Machine Translation · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W11-2103/) · [PDF](https://aclanthology.org/W11-2103.pdf)

**Citations:** [135](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
This report documents the outcomes of the WMT11 shared tasks: translation between English and Czech, French, German and Spanish, combining outputs of several MT systems, and evaluation metrics judged against human preferences. The organizers built new news test sets from 110 articles that were professionally translated. They released parallel and monolingual training corpora, development data and two open-source baseline toolkits. They then ran a large manual evaluation covering 148 primary system entries and 41 system combination entries, and used the human rankings to measure how well 21 automatic metrics agree with people. Two tasks were new this year. The first was translating Haitian Creole SMS messages sent to an emergency service after the 2010 earthquake into English, a low-resource and informal setting. The second was a pilot "tunable metrics" task, in which a fixed Urdu-English system was optimized toward different metrics. In that pilot, the system tuned on BLEU was judged about as good as systems tuned on newer metrics. All data, outputs and human judgments are public, which makes the workshop a shared benchmark resource.

## Key points
- The evaluation covered 148 primary entries, including 46 outputs crawled from online, commercial and rule-based systems, plus 60 contrastive entries and 41 system combination entries. Correlation with human judgments was measured for 21 metrics.
- Test data came from 110 news articles from December 2010, drawn from Czech, English, French, German and Spanish outlets. The agency CEET translated them directly, without a pivot language.
- Training resources included Europarl, News Commentary, the United Nations corpus, the 10⁹ French-English corpus and CzEng. Moses-style phrase-based and parsing-based baseline toolkits were also provided.
- For system combination, the data was split into a tuning set and a held-out test set:
  - European pairs: 1,003 tuning segments and 2,000 test segments.
  - Haitian Creole: 674 tuning segments and 600 test segments.
- In the tunable metrics pilot (Urdu-English):
  - The BLEU-tuned system scored 0.79 on the ≥others measure, roughly level with CMU-METEOR, RWTH-CDER and others.
  - SHEFFIELD-ROSE produced overly long output and scored only 0.05.
  - Ties were frequent: 38% of comparisons overall and 47% without the reference, against 21% for European pairs.

## Interesting ideas
- **Fix the system, vary only the tuning metric.** This isolates the metric's effect on output quality. It is a clean way to test whether a metric is useful in practice, beyond just correlating with human rankings.
- **Treat crawled third-party outputs as unconstrained.** Online and commercial systems may have seen the test data, so they are labelled and anonymized separately. Any benchmark that mixes open and black-box systems needs this kind of contamination caution.
- **Check rankings under both win-only and win-or-tie scoring.** Rankings shifted noticeably between >others and ≥others, so a high tie rate is itself a signal. It suggests annotators may be unable to separate low-quality outputs, and conclusions should be weakened accordingly.
- **Split a tuning portion off the test set for downstream combiners.** Releasing that portion with references gives second-stage systems fair tuning data while keeping a truly held-out evaluation part.
- **Build a featured task from real crisis messages.** Using actual SMS text from a disaster gives a realistic testbed for low-resource, noisy language. The same approach could suit under-resourced languages or domains in other projects.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/)

Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/)

## Cited by

Cited by

135 papers of our library cite it (of the 140673 whose bibliographies we read); 22 of them have a page here, 113 not yet.

- [Findings of the 2021 Conference on Machine Translation (WMT21)](https://md.research.elamur.ai/free/papers/akhbardeh-2021-findings-2021-conference-machine-translation/)
- [Findings of the 2019 Conference on Machine Translation (WMT19)](https://md.research.elamur.ai/free/papers/barrault-2019-findings-2019-conference-machine-translation/)
- [Findings of the 2020 Conference on Machine Translation (WMT20)](https://md.research.elamur.ai/free/papers/barrault-2020-findings-2020-conference-machine-translation/)
- [Findings of the 2013 Workshop on Statistical Machine Translation](https://md.research.elamur.ai/free/papers/bojar-2013-findings-2013-workshop-statistical-machine/)
- [Findings of the 2014 Workshop on Statistical Machine Translation](https://md.research.elamur.ai/free/papers/bojar-2014-findings-2014-workshop-statistical-machine/)
- [Findings of the 2015 Workshop on Statistical Machine Translation](https://md.research.elamur.ai/free/papers/bojar-2015-findings-2015-workshop-statistical-machine/)
- [Findings of the 2016 Conference on Machine Translation](https://md.research.elamur.ai/free/papers/bojar-2016-findings-2016-conference-machine-translation/)
- [Findings of the 2017 Conference on Machine Translation (WMT17)](https://md.research.elamur.ai/free/papers/bojar-2017-findings-2017-conference-machine-translation/)
- [Findings of the 2018 Conference on Machine Translation (WMT18)](https://md.research.elamur.ai/free/papers/bojar-2018-findings-2018-conference-machine-translation/)
- [Findings of the 2012 Workshop on Statistical Machine Translation](https://md.research.elamur.ai/free/papers/burch-2012-findings-2012-workshop-statistical-machine/)
- [Integrating shallow-transfer rules into phrase-based statistical machine translation](https://md.research.elamur.ai/free/papers/cartagena-2011-integrating-shallow-transfer-rules-into/)
- [WIT3: Web Inventory of Transcribed and Translated Talks](https://md.research.elamur.ai/free/papers/cettolo-2012-wit3-web-inventory-transcribed-translated/)
- [A Systematic Comparison of Smoothing Techniques for Sentence-Level BLEU](https://md.research.elamur.ai/free/papers/chen-2014-systematic-comparison-smoothing-techniques-sentence/)
- [Meteor Universal: Language Specific Translation Evaluation for Any Target Language](https://md.research.elamur.ai/free/papers/denkowski-2014-meteor-universal-language-specific-translation/)
- [Statistical Power and Translationese in Machine Translation Evaluation](https://md.research.elamur.ai/free/papers/graham-2020-statistical-power-translationese-machine-translation/)
- [Discourse Structure in Machine Translation Evaluation](https://md.research.elamur.ai/free/papers/joty-2017-discourse-structure-machine-translation-evaluation/)
- [Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet](https://md.research.elamur.ai/free/papers/kocmi-2024-findings-wmt24-general-machine-translation/)
- [Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets](https://md.research.elamur.ai/free/papers/kocmi-2025-findings-wmt25-general-machine-translation/)
- [Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses](https://md.research.elamur.ai/free/papers/lowe-2017-towards-automatic-turing-test-learning/)
- [Translation of "It" in a Deep Syntax Framework](https://md.research.elamur.ai/free/papers/novak-2013-translation-it-deep-syntax-framework/)
- [SemEval-2012 Task 1: English Lexical Simplification](https://md.research.elamur.ai/free/papers/specia-2012-semeval-2012-task-1-english/)
- [Beyond BLEU: Training Neural Machine Translation with Semantic Similarity](https://md.research.elamur.ai/free/papers/wieting-2019-beyond-bleu-training-neural-machine/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{callisonburch2011findings,
  title = {{Findings of the 2011 Workshop on Statistical Machine Translation}},
  author = {Callison-Burch, Chris and Koehn, Philipp and Monz, Christof and Zaidan, Omar},
  booktitle = {Proceedings of the Sixth Workshop on Statistical Machine Translation},
  year = {2011},
  url = {https://aclanthology.org/W11-2103/}
}
```

- [← previous Fill-up versus interpolation methods for phrase-based SMT adaptation](https://md.research.elamur.ai/free/papers/bisazza-2011-fill-up-versus-interpolation-methods/)
- [next → From Human to Automatic Error Classification for Machine Translation Output](https://md.research.elamur.ai/free/papers/popovic-2011-human-automatic-error-classification-machine/)
- [↑ back to papers of 2011](https://md.research.elamur.ai/free/papers/2011/)

## References

80 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Vera Aleksic and Gregor Thurmair. 2011. Personal Translator at WMT2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Vera+Aleksic+and+Gregor+Thurmair%2E+2011%2E+Personal+Translator+at+WMT2011%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
2. Alexandre Allauzen, Hélène Bonneau-Maynard, Hai-Son Le, Aurélien Max, Guillaume Wisniewski, François Yvon, Gilles Adda, Josep Maria Crego, Adrien Lardilleux, Thomas Lavergne, and Artem Sokolov. 2011. LIMSI @ WMT11. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Alexandre+Allauzen%2C+He%CC%81le%CC%80ne+Bonneau%2DMaynard%2C+Hai%2DSon+Le%2C+Aure%CC%81lien+Max%2C+Guillaume+Wisniewski%2C+Franc%CC%A7ois+Yvon%2C+Gilles+Adda%2C+Josep+Maria+Crego%2C+Adrien+Lardilleux%2C+Thomas+Lavergne%2C+and+Artem+Sokolov%2E+20)
3. Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater v.2.0. Journal of Technology, Learning, and Assessment, 4(3):159–174. [find](https://scholar.google.com/scholar?q=Yigal+Attali+and+Jill+Burstein%2E+2006%2E+Automated+essay+scoring+with+e%2Drater+v%2E2%2E0%2E+Journal+of+Technology%2C+Learning%2C+and+Assessment%2C+4%283%29%3A159%E2%80%93174%2E)
4. Eleftherios Avramidis, Maja Popović, David Vilar, and Aljoscha Burchardt. 2011. Evaluate with confidence [http://statmt.org/wmt11/results.html](http://statmt.org/wmt11/results.html) estimation: Machine ranking of translation outputs using grammatical features. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [↗](http://statmt.org/wmt11/results.html)
5. Wilker Aziz, Miguel Rios, and Lucia Specia. 2011. Shallow semantic trees for SMT. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Wilker+Aziz%2C+Miguel+Rios%2C+and+Lucia+Specia%2E+2011%2E+Shallow+semantic+trees+for+SMT%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
6. Kathryn Baker, Michael Bloodgood, Chris Callison-Burch, Bonnie Dorr, Scott Miller, Christine Piatko, Nathaniel W. Filardo, and Lori Levin. 2010. Semantically-informed syntactic machine translation: A tree-grafting approach. In Proceedings of AMTA. [find](https://scholar.google.com/scholar?q=Kathryn+Baker%2C+Michael+Bloodgood%2C+Chris+Callison%2DBurch%2C+Bonnie+Dorr%2C+Scott+Miller%2C+Christine+Piatko%2C+Nathaniel+W%2E+Filardo%2C+and+Lori+Levin%2E+2010%2E+Semantically%2Dinformed+syntactic+machine+translation%3A+A+)
7. Loı̈c Barrault. 2011. MANY improvements for WMT’11. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Lo%C4%B1%CC%88c+Barrault%2E+2011%2E+MANY+improvements+for+WMT%E2%80%9911%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
8. Ergun Bicici and Deniz Yuret. 2011. RegMT system for machine translation, system combination, and evaluation. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Ergun+Bicici+and+Deniz+Yuret%2E+2011%2E+RegMT+system+for+machine+translation%2C+system+combination%2C+and+evaluation%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
9. Ondřej Bojar and Aleš Tamchyna. 2011. Improving translation model by monolingual data. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar+and+Ales%CC%8C+Tamchyna%2E+2011%2E+Improving+translation+model+by+monolingual+data%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
10. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (Meta-) evaluation of machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation (WMT07), Prague, Czech Republic. [find](https://scholar.google.com/scholar?q=Chris+Callison%2DBurch%2C+Cameron+Fordyce%2C+Philipp+Koehn%2C+Christof+Monz%2C+and+Josh+Schroeder%2E+2007%2E+%28Meta%2D%29+evaluation+of+machine+translation%2E+In+Proceedings+of+the+Second+Workshop+on+Statistical+Machine+T) [our card](https://md.research.elamur.ai/free/papers/burch-2007-meta-evaluation-machine-translation/)
11. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008. Further meta-evaluation of machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation (WMT08), Colmbus, Ohio. [find](https://scholar.google.com/scholar?q=Chris+Callison%2DBurch%2C+Cameron+Fordyce%2C+Philipp+Koehn%2C+Christof+Monz%2C+and+Josh+Schroeder%2E+2008%2E+Further+meta%2Devaluation+of+machine+translation%2E+In+Proceedings+of+the+Third+Workshop+on+Statistical+Machi)
12. Chris Callison-Burch, Philipp Koehn, Christof Monz, and Josh Schroeder. 2009. Findings of the 2009 workshop on statistical machine translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation (WMT09), Athens, Greece. [find](https://scholar.google.com/scholar?q=Chris+Callison%2DBurch%2C+Philipp+Koehn%2C+Christof+Monz%2C+and+Josh+Schroeder%2E+2009%2E+Findings+of+the+2009+workshop+on+statistical+machine+translation%2E+In+Proceedings+of+the+Fourth+Workshop+on+Statistical+Mac) [our card](https://md.research.elamur.ai/free/papers/burch-2009-findings-2009-workshop-statistical-machine/)
13. Chris Callison-Burch, Philipp Koehn, Christof Monz, Kay Peterson, Mark Przybocki, and Omar F. Zaidan. 2010. Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation (WMT10), Up… [find](https://scholar.google.com/scholar?q=Chris+Callison%2DBurch%2C+Philipp+Koehn%2C+Christof+Monz%2C+Kay+Peterson%2C+Mark+Przybocki%2C+and+Omar+F%2E+Zaidan%2E+2010%2E+Findings+of+the+2010+joint+workshop+on+statistical+machine+translation+and+metrics+for+machi) [our card](https://md.research.elamur.ai/free/papers/burch-2010-findings-2010-joint-workshop-statistical/)
14. Boxing Chen and Roland Kuhn. 2011. Amber: A modified bleu, enhanced ranking metric. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Boxing+Chen+and+Roland+Kuhn%2E+2011%2E+Amber%3A+A+modified+bleu%2C+enhanced+ranking+metric%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
15. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurment, 20(1):37–46. [find](https://scholar.google.com/scholar?q=Jacob+Cohen%2E+1960%2E+A+coefficient+of+agreement+for+nominal+scales%2E+Educational+and+Psychological+Measurment%2C+20%281%29%3A37%E2%80%9346%2E)
16. Antonio M. Corbı́-Bellot, Mikel L. Forcada, Sergio Ortiz-Rojas, Juan Antonio Pérez-Ortiz, Gema Ramı́rez-Sánchez, Felipe Sánchez-Martı́nez, Iñaki Alegria, Aingeru Mayor, and Kepa Sarasola. 2005. An open-source shallow-transfer machine translation engine for the romance languages of Spain. In Proc… [find](https://scholar.google.com/scholar?q=Antonio+M%2E+Corb%C4%B1%CC%81%2DBellot%2C+Mikel+L%2E+Forcada%2C+Sergio+Ortiz%2DRojas%2C+Juan+Antonio+Pe%CC%81rez%2DOrtiz%2C+Gema+Ram%C4%B1%CC%81rez%2DSa%CC%81nchez%2C+Felipe+Sa%CC%81nchez%2DMart%C4%B1%CC%81nez%2C+In%CC%83aki+Alegria%2C+Aingeru+Mayor%2C+and+Kepa+Sarasola%2E+2005%2E+An)
17. Marta R. Costa-jussà and Rafael E. Banchs. 2011. The BM-I2R Haitian-Créole-to-English translation system description for the WMT 2011 evaluation campaign. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Marta+R%2E+Costa%2Djussa%CC%80+and+Rafael+E%2E+Banchs%2E+2011%2E+The+BM%2DI2R+Haitian%2DCre%CC%81ole%2Dto%2DEnglish+translation+system+description+for+the+WMT+2011+evaluation+campaign%2E+In+Proceedings+of+the+Sixth+Workshop+on+Sta)
18. Daniel Dahlmeier, Chang Liu, and Hwee Tou Ng. 2011. TESLA at WMT 2011: Translation evaluation and tunable metric. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Daniel+Dahlmeier%2C+Chang+Liu%2C+and+Hwee+Tou+Ng%2E+2011%2E+TESLA+at+WMT+2011%3A+Translation+evaluation+and+tunable+metric%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
19. Michael Denkowski and Alon Lavie. 2011a. Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Michael+Denkowski+and+Alon+Lavie%2E+2011a%2E+Meteor+1%2E3%3A+Automatic+metric+for+reliable+optimization+and+evaluation+of+machine+translation+systems%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machi) [our card](https://md.research.elamur.ai/free/papers/denkowski-2011-meteor-1-3-automatic-metric/)
20. Michael Denkowski and Alon Lavie. 2011b. METEOR-Tuned Phrase-Based SMT: CMU French-English and Haitian-English Systems for WMT 2011. Technical Report CMU-LTI-11-011, Language Technologies Institute, Carnegie Mellon University. [find](https://scholar.google.com/scholar?q=Michael+Denkowski+and+Alon+Lavie%2E+2011b%2E+METEOR%2DTuned+Phrase%2DBased+SMT%3A+CMU+French%2DEnglish+and+Haitian%2DEnglish+Systems+for+WMT+2011%2E+Technical+Report+CMU%2DLTI%2D11%2D011%2C+Language+Technologies+Institute%2C+C)
21. Chris Dyer, Kevin Gimpel, Jonathan H. Clark, and Noah A. Smith. 2011. The CMU-ARK German-English translation system. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Chris+Dyer%2C+Kevin+Gimpel%2C+Jonathan+H%2E+Clark%2C+and+Noah+A%2E+Smith%2E+2011%2E+The+CMU%2DARK+German%2DEnglish+translation+system%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
22. Vladimir Eidelman, Kristy Hollingshead, and Philip Resnik. 2011. Noisy SMS machine translation in low-density languages. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Vladimir+Eidelman%2C+Kristy+Hollingshead%2C+and+Philip+Resnik%2E+2011%2E+Noisy+SMS+machine+translation+in+low%2Ddensity+languages%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
23. Christian Federmann and Sabine Hunsicker. 2011. Stochastic parse tree selection for an existing RBMT system. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Christian+Federmann+and+Sabine+Hunsicker%2E+2011%2E+Stochastic+parse+tree+selection+for+an+existing+RBMT+system%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
24. Robert Frederking, Alexander Rudnicky, and Christopher Hogan. 1997. Interactive speech translation in the DIPLOMAT project. In Proceedings of the ACL-1997 Workshop on Spoken Language Translation. [find](https://scholar.google.com/scholar?q=Robert+Frederking%2C+Alexander+Rudnicky%2C+and+Christopher+Hogan%2E+1997%2E+Interactive+speech+translation+in+the+DIPLOMAT+project%2E+In+Proceedings+of+the+ACL%2D1997+Workshop+on+Spoken+Language+Translation%2E)
25. Markus Freitag, Gregor Leusch, Joern Wuebker, Stephan Peitz, Hermann Ney, Teresa Herrmann, Jan Niehues, Alex Waibel, Alexandre Allauzen, Gilles Adda, Josep Maria Crego, Bianka Buschbeck, Tonio Wandmacher, and Jean Senellart. 2011. Joint WMT submission of the QUAERO project. In Proceedings of the Six… [find](https://scholar.google.com/scholar?q=Markus+Freitag%2C+Gregor+Leusch%2C+Joern+Wuebker%2C+Stephan+Peitz%2C+Hermann+Ney%2C+Teresa+Herrmann%2C+Jan+Niehues%2C+Alex+Waibel%2C+Alexandre+Allauzen%2C+Gilles+Adda%2C+Josep+Maria+Crego%2C+Bianka+Buschbeck%2C+Tonio+Wandmac)
26. Yoko Futagi, Paul Deane, Martin Chodorow, and Joel Tetreault. 2008. A computational approach to detecting collocation errors in the writing of non-native speakers of English. Computer Assisted Language Learning Journal. [find](https://scholar.google.com/scholar?q=Yoko+Futagi%2C+Paul+Deane%2C+Martin+Chodorow%2C+and+Joel+Tetreault%2E+2008%2E+A+computational+approach+to+detecting+collocation+errors+in+the+writing+of+non%2Dnative+speakers+of+English%2E+Computer+Assisted+Languag)
27. Jesús González-Rubio and Francisco Casacuberta. 2011. The UPV-PRHLT combination system for WMT 2011. [find](https://scholar.google.com/scholar?q=Jesu%CC%81s+Gonza%CC%81lez%2DRubio+and+Francisco+Casacuberta%2E+2011%2E+The+UPV%2DPRHLT+combination+system+for+WMT+2011%2E)
28. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
29. Greg Hanneman and Alon Lavie. 2011. CMU syntax-based machine translation at WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Greg+Hanneman+and+Alon+Lavie%2E+2011%2E+CMU+syntax%2Dbased+machine+translation+at+WMT+2011%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
30. Christian Hardmeier, Jörg Tiedemann, Markus Saers, Marcello Federico, and Mathur Prashant. 2011. The Uppsala-FBK systems at WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Christian+Hardmeier%2C+Jo%CC%88rg+Tiedemann%2C+Markus+Saers%2C+Marcello+Federico%2C+and+Mathur+Prashant%2E+2011%2E+The+Uppsala%2DFBK+systems+at+WMT+2011%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Trans)
31. Kenneth Heafield and Alon Lavie. 2011. CMU system combination in WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Kenneth+Heafield+and+Alon+Lavie%2E+2011%2E+CMU+system+combination+in+WMT+2011%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
32. Teresa Herrmann, Mohammed Mediani, Jan Niehues, and Alex Waibel. 2011. The Karlsruhe Institute of Technology translation systems for the WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Teresa+Herrmann%2C+Mohammed+Mediani%2C+Jan+Niehues%2C+and+Alex+Waibel%2E+2011%2E+The+Karlsruhe+Institute+of+Technology+translation+systems+for+the+WMT+2011%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+M)
33. Sanjika Hewavitharana, Nguyen Bach, Qin Gao, Vamshi Ambati, and Stephan Vogel. 2011. CMU Haitian Creole-English translation system for WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Sanjika+Hewavitharana%2C+Nguyen+Bach%2C+Qin+Gao%2C+Vamshi+Ambati%2C+and+Stephan+Vogel%2E+2011%2E+CMU+Haitian+Creole%2DEnglish+translation+system+for+WMT+2011%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Mac)
34. Maria Holmqvist, Sara Stymne, and Lars Ahrenberg. 2011. Experiments with word alignment, normalization and clause reordering for SMT between English and German. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Maria+Holmqvist%2C+Sara+Stymne%2C+and+Lars+Ahrenberg%2E+2011%2E+Experiments+with+word+alignment%2C+normalization+and+clause+reordering+for+SMT+between+English+and+German%2E+In+Proceedings+of+the+Sixth+Workshop+on)
35. Chang Hu, Philip Resnik, Yakov Kronrod, Vladimir Eidelman, Olivia Buzek, and Benjamin B. Bederson. 2011. The value of monolingual crowdsourcing in a real-world translation scenario: Simulation using Haitian Creole emergency SMS messages. In Proceedings of the Sixth Workshop on Statistical Machine Tr… [find](https://scholar.google.com/scholar?q=Chang+Hu%2C+Philip+Resnik%2C+Yakov+Kronrod%2C+Vladimir+Eidelman%2C+Olivia+Buzek%2C+and+Benjamin+B%2E+Bederson%2E+2011%2E+The+value+of+monolingual+crowdsourcing+in+a+real%2Dworld+translation+scenario%3A+Simulation+using+H)
36. Matthias Huck, Joern Wuebker, Christoph Schmidt, Markus Freitag, Stephan Peitz, Daniel Stein, Arnaud Dagnelies, Saab Mansour, Gregor Leusch, and Hermann Ney. 2011. The RWTH Aachen machine translation system for WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Matthias+Huck%2C+Joern+Wuebker%2C+Christoph+Schmidt%2C+Markus+Freitag%2C+Stephan+Peitz%2C+Daniel+Stein%2C+Arnaud+Dagnelies%2C+Saab+Mansour%2C+Gregor+Leusch%2C+and+Hermann+Ney%2E+2011%2E+The+RWTH+Aachen+machine+translation+)
37. Maxim Khalilov and Khalil Sima’an. 2011. ILLC-UvA translation system for EMNLP-WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Maxim+Khalilov+and+Khalil+Sima%E2%80%99an%2E+2011%2E+ILLC%2DUvA+translation+system+for+EMNLP%2DWMT+2011%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
38. Philipp Koehn and Christof Monz. 2006. Manual and automatic evaluation of machine translation between European languages. In Proceedings of NAACL 2006 Workshop on Statistical Machine Translation, New York, New York. [find](https://scholar.google.com/scholar?q=Philipp+Koehn+and+Christof+Monz%2E+2006%2E+Manual+and+automatic+evaluation+of+machine+translation+between+European+languages%2E+In+Proceedings+of+NAACL+2006+Workshop+on+Statistical+Machine+Translation%2C+New+) [our card](https://md.research.elamur.ai/free/papers/koehn-2006-manual-automatic-evaluation-machine-translation/)
39. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proc… [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Hieu+Hoang%2C+Alexandra+Birch%2C+Chris+Callison%2DBurch%2C+Marcello+Federico%2C+Nicola+Bertoldi%2C+Brooke+Cowan%2C+Wade+Shen%2C+Christine+Moran%2C+Richard+Zens%2C+Chris+Dyer%2C+Ondrej+Bojar%2C+Alexandra+Consta)
40. Oliver Lacey-Hall. 2011. The guardian’s poverty matters blog: How remote teams can help the rapid response to disasters, March. [find](https://scholar.google.com/scholar?q=Oliver+Lacey%2DHall%2E+2011%2E+The+guardian%E2%80%99s+poverty+matters+blog%3A+How+remote+teams+can+help+the+rapid+response+to+disasters%2C+March%2E)
41. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33:159–174. [find](https://scholar.google.com/scholar?q=J%2E+Richard+Landis+and+Gary+G%2E+Koch%2E+1977%2E+The+measurement+of+observer+agreement+for+categorical+data%2E+Biometrics%2C+33%3A159%E2%80%93174%2E)
42. Gregor Leusch and Hermann Ney. 2009. Edit distances with block movements and error rate confidence estimates. Machine Translation, 23:129–140. [find](https://scholar.google.com/scholar?q=Gregor+Leusch+and+Hermann+Ney%2E+2009%2E+Edit+distances+with+block+movements+and+error+rate+confidence+estimates%2E+Machine+Translation%2C+23%3A129%E2%80%93140%2E)
43. Gregor Leusch, Markus Freitag, and Hermann Ney. 2011. The RWTH system combination system for WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Gregor+Leusch%2C+Markus+Freitag%2C+and+Hermann+Ney%2E+2011%2E+The+RWTH+system+combination+system+for+WMT+2011%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
44. William Lewis, Robert Munro, and Stephan Vogel. 2011. Crisis MT: Developing a cookbook for MT in crisis situations. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=William+Lewis%2C+Robert+Munro%2C+and+Stephan+Vogel%2E+2011%2E+Crisis+MT%3A+Developing+a+cookbook+for+MT+in+crisis+situations%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
45. William D. Lewis. 2010. Haitian Creole: How to build and ship an MT engine from scratch in 4 days, 17hours, & 30 minutes. In Proceedings of EAMT 2010. [find](https://scholar.google.com/scholar?q=William+D%2E+Lewis%2E+2010%2E+Haitian+Creole%3A+How+to+build+and+ship+an+MT+engine+from+scratch+in+4+days%2C+17hours%2C+%26+30+minutes%2E+In+Proceedings+of+EAMT+2010%2E)
46. Zhifei Li, Chris Callison-Burch, Chris Dyer, Juri Ganitkevitch, Ann Irvine, Sanjeev Khudanpur, Lane Schwartz, Wren Thornton, Ziyuan Wang, Jonathan Weese, and Omar Zaidan. 2010. Joshua 2.0: A toolkit for parsing-based machine translation with syntax, semirings, discriminative training and other goodi… [find](https://scholar.google.com/scholar?q=Zhifei+Li%2C+Chris+Callison%2DBurch%2C+Chris+Dyer%2C+Juri+Ganitkevitch%2C+Ann+Irvine%2C+Sanjeev+Khudanpur%2C+Lane+Schwartz%2C+Wren+Thornton%2C+Ziyuan+Wang%2C+Jonathan+Weese%2C+and+Omar+Zaidan%2E+2010%2E+Joshua+2%2E0%3A+A+toolkit+f)
47. Ding Liu and Daniel Gildea. 2005. Syntactic features for evaluation of machine translation. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 25–32. [find](https://scholar.google.com/scholar?q=Ding+Liu+and+Daniel+Gildea%2E+2005%2E+Syntactic+features+for+evaluation+of+machine+translation%2E+In+Proceedings+of+the+ACL+Workshop+on+Intrinsic+and+Extrinsic+Evaluation+Measures+for+Machine+Translation+an) [our card](https://md.research.elamur.ai/free/papers/liu-2005-syntactic-features-evaluation-machine-translation/)
48. Chang Liu, Daniel Dahlmeier, and Hwee Tou Ng. 2011. Better evaluation metrics lead to better machine translation. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Chang+Liu%2C+Daniel+Dahlmeier%2C+and+Hwee+Tou+Ng%2E+2011%2E+Better+evaluation+metrics+lead+to+better+machine+translation%2E+In+Proceedings+of+EMNLP%2E)
49. Verónica López-Ludeña and Rubén San-Segundo. 2011. UPM system for the translation task. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Vero%CC%81nica+Lo%CC%81pez%2DLuden%CC%83a+and+Rube%CC%81n+San%2DSegundo%2E+2011%2E+UPM+system+for+the+translation+task%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
50. Matouš Macháček and Ondřej Bojar. 2011. Approximating a deep-syntactic metric for MT evaluation and tuning. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Matous%CC%8C+Macha%CC%81c%CC%8Cek+and+Ondr%CC%8Cej+Bojar%2E+2011%2E+Approximating+a+deep%2Dsyntactic+metric+for+MT+evaluation+and+tuning%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
51. David Mareček, Rudolf Rosa, Petra Galuščáková, and Ondřej Bojar. 2011. Two-step translation with grammatical post-processing. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=David+Marec%CC%8Cek%2C+Rudolf+Rosa%2C+Petra+Galus%CC%8Cc%CC%8Ca%CC%81kova%CC%81%2C+and+Ondr%CC%8Cej+Bojar%2E+2011%2E+Two%2Dstep+translation+with+grammatical+post%2Dprocessing%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translat)
52. Robert Munro. 2010. Crowdsourced translation for emergency response in Haiti: the global collaboration of local knowledge. In Proceedings of the AMTA Workshop on Collaborative Crowdsourcing for Translation. [find](https://scholar.google.com/scholar?q=Robert+Munro%2E+2010%2E+Crowdsourced+translation+for+emergency+response+in+Haiti%3A+the+global+collaboration+of+local+knowledge%2E+In+Proceedings+of+the+AMTA+Workshop+on+Collaborative+Crowdsourcing+for+Transl)
53. Douglas W. Oard and Franz Josef Och. 2003. Rapid-response machine translation for unexpected languages. In Proceedings of MT Summit IX. [find](https://scholar.google.com/scholar?q=Douglas+W%2E+Oard+and+Franz+Josef+Och%2E+2003%2E+Rapid%2Dresponse+machine+translation+for+unexpected+languages%2E+In+Proceedings+of+MT+Summit+IX%2E)
54. Douglas W. Oard. 2003. The surprise language exercises. ACM Transactions on Asian Language Information Processing, 2(2):79–84. [find](https://scholar.google.com/scholar?q=Douglas+W%2E+Oard%2E+2003%2E+The+surprise+language+exercises%2E+ACM+Transactions+on+Asian+Language+Information+Processing%2C+2%282%29%3A79%E2%80%9384%2E)
55. Franz Josef Och. 2003. Minimum error rate training for statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL-2003), Sapporo, Japan. [find](https://scholar.google.com/scholar?q=Franz+Josef+Och%2E+2003%2E+Minimum+error+rate+training+for+statistical+machine+translation%2E+In+Proceedings+of+the+41st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%2D2003%29%2C+Sapporo%2C+)
56. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL-2002), Philadelphia, Pennsylvania. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+A+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+Association+for+Comp) [our card](https://md.research.elamur.ai/free/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
57. Kristen Parton, Joel Tetreault, Nitin Madnani, and Martin Chodorow. 2011. E-rating machine translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Kristen+Parton%2C+Joel+Tetreault%2C+Nitin+Madnani%2C+and+Martin+Chodorow%2E+2011%2E+E%2Drating+machine+translation%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
58. Martin Popel, David Mareček, Nathan Green, and Zdenêk Ẑabokrtský. 2011. Influence of parser choice on dependency-based MT. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Martin+Popel%2C+David+Marec%CC%8Cek%2C+Nathan+Green%2C+and+Zdene%CC%82k+Z%CC%82abokrtsky%CC%81%2E+2011%2E+Influence+of+parser+choice+on+dependency%2Dbased+MT%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
59. Maja Popović, David Vilar, Eleftherios Avramidis, and Aljoscha Burchardt. 2011. Evaluation without references: IBM1 scores as evaluation metrics. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Maja+Popovic%CC%81%2C+David+Vilar%2C+Eleftherios+Avramidis%2C+and+Aljoscha+Burchardt%2E+2011%2E+Evaluation+without+references%3A+IBM1+scores+as+evaluation+metrics%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+M)
60. Maja Popović. 2011. Morphemes and POS tags for ngram based evaluation metrics. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Maja+Popovic%CC%81%2E+2011%2E+Morphemes+and+POS+tags+for+ngram+based+evaluation+metrics%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
61. Marion Potet, Raphaël Rubino, Benjamin Lecouteux, Stéphane Huet, Laurent Besacier, Hervé Blanchon, and Fabrice Lefèvre. 2011. The LIGA (LIG/LIA) machine translation system for WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Marion+Potet%2C+Raphae%CC%88l+Rubino%2C+Benjamin+Lecouteux%2C+Ste%CC%81phane+Huet%2C+Laurent+Besacier%2C+Herve%CC%81+Blanchon%2C+and+Fabrice+Lefe%CC%80vre%2E+2011%2E+The+LIGA+%28LIG%2FLIA%29+machine+translation+system+for+WMT+2011%2E+In+Proceed)
62. Mark Przybocki, Kay Peterson, and Sebastian Bronsart. 2008. Official results of the NIST 2008 “Metrics for MAchine TRanslation” challenge (Metrics-MATR08). In AMTA-2008 workshop on Metrics for Machine Translation, Honolulu, Hawaii. [find](https://scholar.google.com/scholar?q=Mark+Przybocki%2C+Kay+Peterson%2C+and+Sebastian+Bronsart%2E+2008%2E+Official+results+of+the+NIST+2008+%E2%80%9CMetrics+for+MAchine+TRanslation%E2%80%9D+challenge+%28Metrics%2DMATR08%29%2E+In+AMTA%2D2008+workshop+on+Metrics+for+Machine)
63. Miguel Rios, Wilker Aziz, and Lucia Specia. 2011. TINE: A metric to assess MT adequacy. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Miguel+Rios%2C+Wilker+Aziz%2C+and+Lucia+Specia%2E+2011%2E+TINE%3A+A+metric+to+assess+MT+adequacy%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
64. Christian Rishøj and Anders Søgaard. 2011. Factored translation with unsupervised word clusters. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Christian+Rish%C3%B8j+and+Anders+S%C3%B8gaard%2E+2011%2E+Factored+translation+with+unsupervised+word+clusters%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
65. Antti-Veikko Rosti, Bing Zhang, Spyros Matsoukas, and Richard Schwartz. 2011. Expected BLEU training for graphs: BBN system description for WMT11 system combination task. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Antti%2DVeikko+Rosti%2C+Bing+Zhang%2C+Spyros+Matsoukas%2C+and+Richard+Schwartz%2E+2011%2E+Expected+BLEU+training+for+graphs%3A+BBN+system+description+for+WMT11+system+combination+task%2E+In+Proceedings+of+the+Sixth+W)
66. Vı́ctor M. Sánchez-Cartagena, Felipe Sánchez-Martı́nez, and Juan Antonio Pérez-Ortiz. 2011. The Universitat d’Alacant hybrid machine translation system for WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=V%C4%B1%CC%81ctor+M%2E+Sa%CC%81nchez%2DCartagena%2C+Felipe+Sa%CC%81nchez%2DMart%C4%B1%CC%81nez%2C+and+Juan+Antonio+Pe%CC%81rez%2DOrtiz%2E+2011%2E+The+Universitat+d%E2%80%99Alacant+hybrid+machine+translation+system+for+WMT+2011%2E+In+Proceedings+of+the+Sixth+Wor)
67. Holger Schwenk, Patrik Lambert, Loı̈c Barrault, Christophe Servan, Sadaf Abdul-Rauf, Haithem Afli, and Kashif Shah. 2011. LIUM’s SMT machine translation systems for WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Holger+Schwenk%2C+Patrik+Lambert%2C+Lo%C4%B1%CC%88c+Barrault%2C+Christophe+Servan%2C+Sadaf+Abdul%2DRauf%2C+Haithem+Afli%2C+and+Kashif+Shah%2E+2011%2E+LIUM%E2%80%99s+SMT+machine+translation+systems+for+WMT+2011%2E+In+Proceedings+of+the+Six)
68. Rico Sennrich. 2011. The UZH system combination system for WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2E+2011%2E+The+UZH+system+combination+system+for+WMT+2011%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
69. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Biennial Conference of the Association for Machine Translation in the Americas (AMTA-2006), Cambridge, Massachusetts. [find](https://scholar.google.com/scholar?q=Matthew+Snover%2C+Bonnie+Dorr%2C+Richard+Schwartz%2C+Linnea+Micciulla%2C+and+John+Makhoul%2E+2006%2E+A+study+of+translation+edit+rate+with+targeted+human+annotation%2E+In+Proceedings+of+the+7th+Biennial+Conference+) [our card](https://md.research.elamur.ai/free/papers/snover-2006-study-translation-edit-rate-targeted/)
70. Xingyi Song and Trevor Cohn. 2011. Regression and ranking based optimisation for sentence level MT evaluation. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Xingyi+Song+and+Trevor+Cohn%2E+2011%2E+Regression+and+ranking+based+optimisation+for+sentence+level+MT+evaluation%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
71. Lucia Specia, Dhwaj Raj, and Marco Turchi. 2010. Machine translation evaluation versus quality estimation. Machine Translation, 24(1):39–50. [find](https://scholar.google.com/scholar?q=Lucia+Specia%2C+Dhwaj+Raj%2C+and+Marco+Turchi%2E+2010%2E+Machine+translation+evaluation+versus+quality+estimation%2E+Machine+Translation%2C+24%281%29%3A39%E2%80%9350%2E)
72. Sara Stymne. 2011. Spell checking techniques for replacement of unknown words and data cleaning for Haitian Creole SMS translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Sara+Stymne%2E+2011%2E+Spell+checking+techniques+for+replacement+of+unknown+words+and+data+cleaning+for+Haitian+Creole+SMS+translation%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translat)
73. Joel Tetreault and Martin Chodorow. 2008. The ups and downs of preposition error detection. In Proceedings of COLING, Manchester, UK. [find](https://scholar.google.com/scholar?q=Joel+Tetreault+and+Martin+Chodorow%2E+2008%2E+The+ups+and+downs+of+preposition+error+detection%2E+In+Proceedings+of+COLING%2C+Manchester%2C+UK%2E)
74. Jonathan Weese, Juri Ganitkevitch, Chris Callison-Burch, Matt Post, and Adam Lopez. 2011. Joshua 3.0: Syntax-based machine translation with the Thrax grammar extractor. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Jonathan+Weese%2C+Juri+Ganitkevitch%2C+Chris+Callison%2DBurch%2C+Matt+Post%2C+and+Adam+Lopez%2E+2011%2E+Joshua+3%2E0%3A+Syntax%2Dbased+machine+translation+with+the+Thrax+grammar+extractor%2E+In+Proceedings+of+the+Sixth+Wor)
75. Eric Wehrli, Luka Nerima, and Yves Scherrer. 2009. Deep linguistic multilingual translation and bilingual dictionaries. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 90–94. [find](https://scholar.google.com/scholar?q=Eric+Wehrli%2C+Luka+Nerima%2C+and+Yves+Scherrer%2E+2009%2E+Deep+linguistic+multilingual+translation+and+bilingual+dictionaries%2E+In+Proceedings+of+the+Fourth+Workshop+on+Statistical+Machine+Translation%2C+pages+)
76. Daguang Xu, Yuan Cao, and Damianos Karakos. 2011a. Description of the JHU system combination scheme for WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Daguang+Xu%2C+Yuan+Cao%2C+and+Damianos+Karakos%2E+2011a%2E+Description+of+the+JHU+system+combination+scheme+for+WMT+2011%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
77. Jia Xu, Hans Uszkoreit, Casey Kennington, David Vilar, and Xiaojun Zhang. 2011b. DFKI hybrid machine translation system for WMT 2011 - on the integration of SMT and RBMT. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Jia+Xu%2C+Hans+Uszkoreit%2C+Casey+Kennington%2C+David+Vilar%2C+and+Xiaojun+Zhang%2E+2011b%2E+DFKI+hybrid+machine+translation+system+for+WMT+2011+%2D+on+the+integration+of+SMT+and+RBMT%2E+In+Proceedings+of+the+Sixth+W)
78. Omar F. Zaidan. 2009. Z-MERT: A fully configurable open source tool for minimum error rate training of machine translation systems. The Prague Bulletin of Mathematical Linguistics, 91:79–88. [find](https://scholar.google.com/scholar?q=Omar+F%2E+Zaidan%2E+2009%2E+Z%2DMERT%3A+A+fully+configurable+open+source+tool+for+minimum+error+rate+training+of+machine+translation+systems%2E+The+Prague+Bulletin+of+Mathematical+Linguistics%2C+91%3A79%E2%80%9388%2E)
79. Francisco Zamora-Martinez and Maria Jose Castro-Bleda. 2011. CEU-UPV English-Spanish system for WMT11. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Francisco+Zamora%2DMartinez+and+Maria+Jose+Castro%2DBleda%2E+2011%2E+CEU%2DUPV+English%2DSpanish+system+for+WMT11%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)
80. Daniel Zeman. 2011. Hierarchical phrase-based MT at the Charles University for the WMT 2011 shared task. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Daniel+Zeman%2E+2011%2E+Hierarchical+phrase%2Dbased+MT+at+the+Charles+University+for+the+WMT+2011+shared+task%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E)